Commit graph

93 commits

Author SHA1 Message Date
Jesse Bannon
2dbe426ced
[BUGFIX] Properly handle UIDs that have misbehaving chars that get sanitized (#870)
Should hopefuly fix both:
- https://github.com/jmbannon/ytdl-sub/issues/843
- https://github.com/jmbannon/ytdl-sub/issues/867

Various sites have unique IDs with characters that get sanitized (i.e. `/`, `:`). `ytdl-sub` was not using sanitized UIDs when looking for files, and would throw an error saying download was missing. This should resolve that issue.
2024-01-04 15:01:17 -08:00
Jesse Bannon
731d4e444c
[DOCS] Tooling to auto-generate custom sphinx docs (#849)
Tooling + updating docs
2023-12-28 22:39:59 -08:00
Jesse Bannon
308c76a8f2
[FEATURE] Filter plugins, regex helper script functions (#848)
Adds generic `filter_include` and `filter_exclude` plugins in anticipation for function script usage. Will detail it more with official docs at a later time!
2023-12-20 09:52:12 -08:00
Jesse Bannon
e92b1cd12a
[BACKEND][HUGE] Function Support in variable syntax (#838)
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.

With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.

Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
2023-12-18 16:08:15 -08:00
OmarAhmadyar
8998861b49
[BUGFIX] file_convert output file deleted (#822)
file_convert.py:modify_entry() deletes the converted output file if the output and input file names are identical.  Reordered the file deletion to avoid.

Thanks @OmarAhmadyar !
2023-11-25 00:23:46 -08:00
Jesse Bannon
395691f2b9
[FEATURE] Create output_directory_nfo_tags on first entry instead of last (#809) 2023-11-05 23:10:29 -08:00
Jesse Bannon
f1a2187a17
[FEATURE] Add throttle_protection plugin (#799)
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.

Usage:
```
    throttle_protection:
      sleep_per_download_s:
        min: 2.2
        max: 10.8
      sleep_per_subscription_s:
        min: 9.0
        max: 14.1
      max_downloads_per_subscription:
        min: 10
        max: 36
      subscription_download_probability: 1.0
```
2023-11-05 08:23:12 -08:00
Jesse Bannon
056b111be6
[FEATURE] Prebuilt Music Videos (#787)
Creates standardized prebuilt presets for music videos. Preset names are:

```
"Kodi Music Videos"
"Jellyfin Music Videos"
"Plex Music Videos"
```

Usage can be found in the `examples/` directory
2023-10-28 00:12:22 -07:00
Jesse Bannon
d810291e7c
[BUGFIX] Write music multi-tags correctly (#782)
Music multi-tags (artists, albumartists, etc) were not writing correctly. Fix it by explicitly checking to see if the tag is multi or not in the codebase.
2023-10-25 10:37:24 -07:00
Jesse Bannon
c119e6f644
[BUGFIX] Fix match-filter edge case and date_range.before (#750)
Fixes yet another bug that is match-filter related. The `date_range.before` value would get ignored if you specified any other match-filters. That is fixed by making them `&` behind-the-scenes.
2023-10-03 11:46:42 -07:00
Jesse Bannon
6615e1d1ac
[BACKEND] Always filter out live, post_live, and upcoming videos on download (#742)
Closes Issue: https://github.com/jmbannon/ytdl-sub/issues/729

Add a new field to the match_filter plugin: `download_match_filters`. This set of filters will be applied at the download stage (as opposed to `filters` which get applied at the metadata stage). By default, always set `"!is_live & !is_upcoming & !post_live"` as a download filter. This prevents ytdl-sub from hanging when trying to grab live and upcoming videos.

Docs on match-filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-24 06:24:21 -07:00
Jesse Bannon
3c87dc9f8e
[BUGFIX] Fix date_range plugin as a match-filter (#726)
Bugfix to the prior release. Convert yt-dlp date formats (i.e. `today-2months`) into a date string with format `YYYYMMDD`. This will make it compatible with match-filter syntax
2023-09-20 08:30:29 -07:00
Jesse Bannon
1a118e225a
[BACKEND] Perform match-filters at metadata stage, use match-filters for date_range plugin (#725)
Solved Issue: https://github.com/jmbannon/ytdl-sub/issues/706

From our discussion with yt-dlp folks (https://github.com/yt-dlp/yt-dlp/issues/8108), we learned that best practices for date--range filtering is done using match-filters. We now set the `date_range` plugin values to be a 'breaking' match filter, and any custom ones specified from users in the `match_filters` plugin as non-breaking. This means match-filters can be performed at the metadata stage, which should yield a significant speedup

Relevant docs:
- date_range: https://ytdl-sub.readthedocs.io/en/latest/config.html#date-range
- match_filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-19 11:19:15 -07:00
Jesse Bannon
0b62cc13d0
[DOCS] Mention YAML single vs double quotes in regex docs (#719)
Improve regex plugin docs by mentioning difference between using single-quotes and double-quotes in YAML: https://ytdl-sub.readthedocs.io/en/latest/config.html#regex
2023-09-15 13:54:08 -07:00
Jesse Bannon
a48efdc84c
[FEATURE] format plugin to set yt-dlp format easier (#714)
Makes setting yt-dlp's `format` field easier by giving it its own documented plugin.

Old:
```
my_format_preset:
  ytdl_options:
    format: "best"
```

New:
```
my_format_preset:
  format: "best"
```

The old method will still work, so do not worry about updating configs ASAP. However, the option is available to save a few lines 😉
2023-09-13 00:02:19 -07:00
Jesse Bannon
792da1ba86
[BACKEND] Set merge_output_format to file_convert file type (#699)
Adds a small optimization to potentially avoid an additional remux by setting yt-dlp's merge output format to the desired format in the `file_convert` plugin (https://ytdl-sub.readthedocs.io/en/latest/config.html#file-convert)
2023-09-02 11:18:04 -07:00
Jesse Bannon
da70c725fc
[FEATURE] Support best codec type in audio_extract (#700)
Adds support to the `audio_extract` plugin (https://ytdl-sub.readthedocs.io/en/latest/config.html#audio-extract) to the following to grab the best audio format available:

```
audio_extract:
  codec: "best"
```
2023-09-02 00:50:26 -07:00
Jesse Bannon
5f9b8afc4b
[BUGFIX] Do not hard-fail if the thumbnail fails to download/convert (#675)
* [BUGFIX] Do not fail with thumbnail issues

* helper

* refactor

* dry-run

* regen fixture

* download arhcive txt -> json in docs, fixtures

* return if no download thumb path

* better test

* more test coverage
2023-07-28 13:12:46 -07:00
Jesse Bannon
e3c662bc35
[REFACTOR] Shared plugin code (#664)
* [REFACTOR] Shared plugin code

* arg names

* all plugins have priority

* split abstract method

* lint

* lint
2023-07-25 12:11:50 -07:00
Jesse Bannon
dcd636192c
[REFACTOR] download abstraction overhaul (#663)
* [REFACTOR] `download` abstraction overhaul

* remove plugin options
2023-07-25 00:36:51 -07:00
Jesse Bannon
bb03c91656
[BACKEND] Links to deprecation (#662) 2023-07-25 00:10:54 -07:00
Jesse Bannon
77487aee86
[FEATURE] Simplify `video_tags` (#661)
* [FEATURE] Simplify ``video_tags``

* video tags old

* change for episode

* deprecation notice
2023-07-24 18:18:53 -07:00
Jesse Bannon
cc38db9c7a
[FEATURE] Simplify `music_tags` plugin (#660)
* [FEATURE] Simplify ``music_tags`` plugin

* docs

* more compatible

* check old format better

* better dict instantiate
2023-07-24 18:05:54 -07:00
Jesse Bannon
0cbd2af4d3
[FEATURE] embed_thumbnail plugin (#659)
* [REFACTOR] OptionsValidator + OptionsDictValidator

* more

* [FEATURE] Dedicated embed_thumbnail plugin

* tests

* fix test

* docs

* deprecation notices

* priority to run after file convert

* naming

* fixtures
2023-07-24 12:07:36 -07:00
Jesse Bannon
7e997f304d
[REFACTOR] OptionsValidator + OptionsDictValidator (#657)
* [REFACTOR] OptionsValidator + OptionsDictValidator

* more
2023-07-24 10:12:38 -07:00
Jesse Bannon
acc7bb306b
[DOCS] Clean up match-filter plugin docs (#633) 2023-06-17 13:32:12 -07:00
Jesse Bannon
a993beff0e
[FEATURE] Ability to use overrides in regex plugin (#631)
* [FEATURE] Ability to use overrides in regex plugin

* lint

* more tests

* more docs

* docs
2023-06-17 13:02:35 -07:00
Jesse Bannon
ec447cfbd7
[FEATURE] exclude field in regex plugin (#620)
* [FEATURE] `exclude` field in regex plugin

* tests

* excludes doc
2023-05-22 10:23:00 -07:00
Jesse Bannon
0497f2b749
[FEATURE] Perform match-filter on download, not metadata pull (#612)
* [FEATURE] Apply match-filter on download instead of metadata

* correct ytdl_option ordering

* lint

* test that downloads
2023-05-18 09:52:24 -07:00
Jesse Bannon
38e64a5bd5
[FEATURE] Perform regex at both metadata and post-download (#611)
* [FEATURE] Perform regex at both metadata and post-download

* sadf
2023-05-17 22:33:42 -07:00
Jesse Bannon
1588725bb1
[BUGFIX] Fix Windows split by chapters not reading chapter titles correctly (#591) 2023-04-19 18:32:13 -07:00
Jesse Bannon
70d81bf20f
[BACKEND] Mark entry as deleted if all entries deleted in update (#559)
* [BACKEND] Mark entry as deleted if all entries deleted in update

* make subtitles not found debug
2023-03-21 14:21:02 -07:00
Jesse Bannon
14ddd750d9
[REFACTOR] Shared OptionsValidator, downloader plugins (#550) 2023-03-15 23:06:55 -07:00
Jesse Bannon
e69933d525
[REFACTOR] Split metadata and download into two separate steps (#540) 2023-03-13 22:41:09 -07:00
Jesse Bannon
2d2fff9a40
[FEATURE] Prebuilt presets to include title in their filename (#525) 2023-03-10 10:30:35 -08:00
Jesse Bannon
f9318b9d07
[BUGFIX] Handle truncating file names if too long (#524)
* [BUGFIX] Handle truncating file names if too long

* always import os

* test fix

* fix for windows

* lint
2023-03-10 07:20:43 -08:00
Jesse Bannon
798bbbc62c
[BACKEND] Windows support (#476) 2023-02-28 17:07:03 -08:00
Jesse Bannon
6020f27f12
[BACKEND] Better error message for music_tags recieving a video (#413)
* [BACKEND] Better error message for music_tags recieving a video

* test and better message
2023-01-13 10:25:17 -08:00
Jesse Bannon
c95c0f6aa4
[DOCS] Document additional backslash for regex (#412)
* [DOCS] Document additional backslash for regex

* double-backslash
2023-01-11 22:51:01 -08:00
Jesse Bannon
96e741c71d
[BACKEND] Proactively clean working directory (#379)
* fix playlist last thumbnail move to copy

* playlist working

* channel working

* working directory getting cleaned

* working dir e2e fixture

* cleanup files in any case

* clean up if pre-split returns None
2022-11-30 10:36:13 -08:00
Jesse Bannon
a77885f4de
[FEATURE] Match filters plugin (#365) 2022-11-26 22:32:19 -08:00
Jesse Bannon
2f792f774c
[FEATURE] Add custom ffmpeg post-processing (#364) 2022-11-26 14:56:34 -08:00
Jesse Bannon
ead5620c5c
[FEATURE] Add mp3 id3v2.4 multi-tag support (#356)
* [FEATURE] Add mp3 id3v2.4 multi-tag support

* docs

* fix new docker issue

* update fixture
2022-11-24 00:56:23 -08:00
Jesse Bannon
5d9fe2eaae
[BUGFIX] Set multi-value music tags as a list with single value (#349)
* [BUGFIX] Set multi-value music tags as a list with single value

* update fixtures
2022-11-21 14:11:55 -08:00
Jesse Bannon
ffaf07d628
[FEATURE] Add support to scrape chapters from comments (#347)
* [FEATURE] Add support to scrape chapters from comments

* tested for comments

* remove timestamps file

* change order of docstrings
2022-11-21 11:19:17 -08:00
Jesse Bannon
70c78dc109
[FEATURE] Embed thumbnails into audio files as cover art (#345)
* [FEATURE] Embed thumbnails into audio files as cover art

* lint

* fix ytdl options
2022-11-19 23:36:15 -08:00
Jesse Bannon
20bec0c664
[BUGFIX] Fix split_by_chapter download archive not working (#344) 2022-11-19 23:08:59 -08:00
Jesse Bannon
5705a28842
[FEATURE] Add ytdl-sub view URL to view source variables (#340)
* [FEATURE] Add `ytdl-sub view URL` to view source variables

* update file path
2022-11-19 16:56:45 -08:00
Jesse Bannon
44ec463ab3
[BACKEND] More strict validation for config files (#301) 2022-10-26 00:00:36 -07:00
Jesse Bannon
3e4a995f6e [DOCS] Fix file_convert yaml example 2022-10-22 22:10:55 -07:00