Commit graph

187 commits

Author SHA1 Message Date
Jesse Bannon
e92b1cd12a
[BACKEND][HUGE] Function Support in variable syntax (#838)
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.

With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.

Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
2023-12-18 16:08:15 -08:00
Jesse Bannon
f1a2187a17
[FEATURE] Add throttle_protection plugin (#799)
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.

Usage:
```
    throttle_protection:
      sleep_per_download_s:
        min: 2.2
        max: 10.8
      sleep_per_subscription_s:
        min: 9.0
        max: 14.1
      max_downloads_per_subscription:
        min: 10
        max: 36
      subscription_download_probability: 1.0
```
2023-11-05 08:23:12 -08:00
Jesse Bannon
e9b2214a0a
[DOCS] New README with only subscriptions (#791) 2023-11-02 23:28:55 -07:00
Jesse Bannon
545f186c46
[FEATURE][BUGFIX] Beautiful subscriptions syntax update + fix (#758)
A follow-up to the prior commit which fixes some bugs in the 'beautiful subscriptions' implementation. A proper migration guide will be written in the wiki once it's ready
2023-10-17 16:30:50 -07:00
Jesse Bannon
c255f40348
[FEATURE] Subscription nesting (#747)
Closes Feature Request: https://github.com/jmbannon/ytdl-sub/issues/743

Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:

```
tv_show:
  only_recent:
    [News]:
      "Breaking News": "https://www.youtube.com/@SomeBreakingNews"

  [Tech]:
    "Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
```
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:

```
"Breaking News":
  preset:
    - "tv_show"
    - "only_recent"

  overrides:
    subscription_indent_1: "News"
    subscription_name: "Breaking News"
    subscription_value: "https://www.youtube.com/@SomeBreakingNews"

"Two Minute Papers":
  preset:
    - "tv_show"

  overrides:
    subscription_indent_1: "Tech"
    subscription_name: "Two Minute Papers"     subscription_value: "https://www.youtube.com/@TwoMinutePapers"
```

You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things you want configurable like so in your
parent preset:

```
presets:
  tv_show_name:
    overrides:
      tv_show_name: "{subscription_name}"
      url: "{subscription_value}"
      genre: "{subscription_indent_1}"
```

NOTE!!!
Changing your subscription name from 'legacy' subscriptions will need their download archive file migrated to a new name. A future update will assist in this migration process. Either wait until then or, if you consider yourself a ytdl-sub expert, keep a backup of your subscription file before migrating.
2023-10-02 16:07:48 -07:00
Jesse Bannon
3e41e0f59e
[FEATURE] Ability to add subscription value via config (#744)
`Subscription Value` is the mechanism used to achieve one-liner subscriptions. It works by having the subscription `"Subscription Name": "value"`, and assigning `"value"` to an override variable via setting the subscription value to that override variable's name.

This feature allows you to set subscription value within the config. In the near future, ytdl-sub will strive to make subscription files simpler by hiding most of the "under-the-hood" logic within the config file.
2023-09-27 21:37:36 -07:00
Jesse Bannon
6615e1d1ac
[BACKEND] Always filter out live, post_live, and upcoming videos on download (#742)
Closes Issue: https://github.com/jmbannon/ytdl-sub/issues/729

Add a new field to the match_filter plugin: `download_match_filters`. This set of filters will be applied at the download stage (as opposed to `filters` which get applied at the metadata stage). By default, always set `"!is_live & !is_upcoming & !post_live"` as a download filter. This prevents ytdl-sub from hanging when trying to grab live and upcoming videos.

Docs on match-filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-24 06:24:21 -07:00
Jesse Bannon
a48efdc84c
[FEATURE] format plugin to set yt-dlp format easier (#714)
Makes setting yt-dlp's `format` field easier by giving it its own documented plugin.

Old:
```
my_format_preset:
  ytdl_options:
    format: "best"
```

New:
```
my_format_preset:
  format: "best"
```

The old method will still work, so do not worry about updating configs ASAP. However, the option is available to save a few lines 😉
2023-09-13 00:02:19 -07:00
CatDuck
3b79a7ccae
[FEATURE] Add max_1080p prebuilt preset (#690)
* Fix Plex MP4

No WebM, Best quality, No AV1 either

* Fixed because I am idiot

Forcing Plex to download only MP4 and 1080P. More efficient then downloading Webm and converting. Plex does not direct Play AV1 currently anyway.

* Plex Update

Default will get best video in H265 or H264, if neither are available, it will get next best video. All MP4, no limit on resolution.

Plex_tv_show_av1 will allow for absolute best possible resolution in any codec as long as its MP4, including Av1 which is not currently supported in Plex.

Without AV1 is considered default, for maximum compatibility at this time.

* EBUATK

Incorrect setting on Av1, fixed.

* Removed Av1 Preset

* adding common 1080p MAX preset

max_1080_video_quality:

* removed remux video

Removed remux video
Shortened preset name.

* Update presets.rst

Update presets.rst documentation to reflect the "max_1080P" preset

* remove remux_video

* Title Change for Clarity

* Update presets.rst

---------

Co-authored-by: catduckgnaf <128985786+cellardoor452@users.noreply.github.com>
2023-08-30 11:28:58 -07:00
Jesse Bannon
77487aee86
[FEATURE] Simplify `video_tags` (#661)
* [FEATURE] Simplify ``video_tags``

* video tags old

* change for episode

* deprecation notice
2023-07-24 18:18:53 -07:00
Jesse Bannon
cc38db9c7a
[FEATURE] Simplify `music_tags` plugin (#660)
* [FEATURE] Simplify ``music_tags`` plugin

* docs

* more compatible

* check old format better

* better dict instantiate
2023-07-24 18:05:54 -07:00
Jesse Bannon
0cbd2af4d3
[FEATURE] embed_thumbnail plugin (#659)
* [REFACTOR] OptionsValidator + OptionsDictValidator

* more

* [FEATURE] Dedicated embed_thumbnail plugin

* tests

* fix test

* docs

* deprecation notices

* priority to run after file convert

* naming

* fixtures
2023-07-24 12:07:36 -07:00
Jesse Bannon
7a43b0a90b
[FEATURE] Add __value__ to subscription files to be able to create one-liner subs (#641)
* [FEATURE] Add __value__ to subscription files to create one-liner subs

* docs
2023-06-29 23:57:54 -07:00
Jesse Bannon
a993beff0e
[FEATURE] Ability to use overrides in regex plugin (#631)
* [FEATURE] Ability to use overrides in regex plugin

* lint

* more tests

* more docs

* docs
2023-06-17 13:02:35 -07:00
Jesse Bannon
ac5561cb2e
[FEATURE] Content rating override variable (#626)
* [FEATURE] Content rating override variable

* docs
2023-06-03 07:42:16 -07:00
Jesse Bannon
ec447cfbd7
[FEATURE] exclude field in regex plugin (#620)
* [FEATURE] `exclude` field in regex plugin

* tests

* excludes doc
2023-05-22 10:23:00 -07:00
Jesse Bannon
2f1756dea7
[FEATURE] More tv show collection seasons (#615) 2023-05-19 09:05:51 -07:00
Jesse Bannon
a2f1a2ba94
[FEATURE] Add subscription_name override variable by default (#599)
* [FEATURE] Add `subscription_name` override variable by default

* docs for it

* lint

* only test non-cli video test
2023-04-28 19:36:20 -07:00
Jesse Bannon
2426b700f3
[DOCS] Add unraid to readthedocs install (#580) 2023-04-05 12:34:00 -07:00
Jesse Bannon
1965bc0a40
[FEATURE] Multiple URLs supported for tv_show_by_date preset (#568)
* [BACKEND] Support empty urls for multi_url, add overrides for extra urls in tv_show_by_date preset

* need to check for missing urls in downloader

* use override in test

* docs about multi url

* wording
2023-03-30 09:28:32 -07:00
Jesse Bannon
54b8b76034
[BACKEND] Handle case when collection url is missing from entry (#558)
* [BACKEND] Handle case when collection url is missing from entry

* hide experimental docs
2023-03-21 13:39:41 -07:00
Jesse Bannon
a96fcff55f
[DOCS] passthrough in README (#549) 2023-03-15 16:12:42 -07:00
Jesse Bannon
ec51361728
[DOCS] Docker passthrough install docs (#548) 2023-03-15 16:08:15 -07:00
Jesse Bannon
9c41a17375
[DOCS] Fix download docs (#547) 2023-03-15 15:55:30 -07:00
Jesse Bannon
45244889ae
[FEATURE] chunk_initial_download preset (#532) 2023-03-10 19:18:37 -08:00
Jesse Bannon
f42973cf2f
[FEATURE] download_reverse in URL download strategy (#531)
* [FEATURE] download_reverse in URL download strategy

* doc fix
2023-03-10 16:13:02 -08:00
Jesse Bannon
11f00abaf6
[FEATURE] best_video_quality prebuilt preset (#526)
* [FEATURE] best_video_quality prebuilt preset

* common
2023-03-10 11:48:14 -08:00
Jesse Bannon
2d2fff9a40
[FEATURE] Prebuilt presets to include title in their filename (#525) 2023-03-10 10:30:35 -08:00
Michael Born
a9212bdf84
[DOCS] Fix yaml syntax for initial docker-compose setup (#523)
The provided `docker-compose.yml` config is not valid YAML, due to a missing space in the last line.
2023-03-09 23:19:41 -08:00
Jesse Bannon
fddf3d3088
[FEATURE] Ability to suppress or write transaction log to file (#514) 2023-03-08 13:32:07 -08:00
Jesse Bannon
648027204f
[FEATURE] Persisted subscription logs (#512) 2023-03-08 00:17:11 -08:00
Jesse Bannon
3c6fd0b4e8
[BUGFIX] Fix global options not working after sub (#500)
* [BACKEND] Error if arg after positional arg

* working

* dry-run default

* remove unused code
2023-03-03 11:07:11 -08:00
Jesse Bannon
628c0f1f43
[BACKEND] Add more collection seasons (#498) 2023-03-02 18:33:25 -08:00
Jesse Bannon
5d57eb14b8
[DOCS] Universal cron mod for docker (#495) 2023-03-02 11:24:04 -08:00
Jesse Bannon
6146762571
[DOCS] readthedocs makeover (#493) 2023-03-01 23:45:16 -08:00
Jesse Bannon
fea3f86acf
[DOCS] Improved installation guide (#490) 2023-03-01 19:58:27 -08:00
Jesse Bannon
74c7c3be01
[DOCS] Add link to prebuilt preset definitions (#454) 2023-02-20 12:49:05 -08:00
Jesse Bannon
a77885f4de
[FEATURE] Match filters plugin (#365) 2022-11-26 22:32:19 -08:00
Jesse Bannon
ecaaeb694a
[DOCS] Preset single or list explanation (#328) 2022-11-15 11:13:08 -08:00
Jesse Bannon
93b5d83597
[DOCS] Add deprecation update docs (#317) 2022-11-12 10:34:08 -08:00
Jesse Bannon
10b58264ea
[BACKEND] Update music video example with url download strategy (#312)
* [DOCS] document file __preset__

* [REFACTOR] Use `download` type in examples

* TODO: download strategy docs, validate fixtures, fix playlist test

* playlist fixed, fixtures look good

* download strategy docs, fix tests

* better docs

* lint

* yt fixed

* split chapters fixed

* fix video

* better docs

* lint
2022-11-11 08:54:07 -08:00
Jesse Bannon
7458814a36
[DOCS] document file __preset__ (#311) 2022-11-06 17:00:42 -08:00
Jesse Bannon
4ddaf72ae1
[DOCS] file_convert plugin (#276)
* [DOCS] file_convert plugin

* better docs

* lint
2022-10-07 22:46:07 -07:00
Jesse Bannon
cfc2e916aa
[BACKEND] Include genre override for tv shows (#277) 2022-10-07 22:23:07 -07:00
Jesse Bannon
5b8ee7f2de
[FEATURE] download_index source variable, new tv presets (#271) 2022-10-05 12:08:43 -07:00
Jesse Bannon
5b9a5dae50
[FEATURE] Prebuilt presets (#264) 2022-10-01 22:41:36 -07:00
Jesse Bannon
e2faffd59c
[DOCS] Fix playlist thumbnail docs (#251) 2022-09-28 00:17:18 -07:00
Jesse Bannon
ebdd6c8aec
[BACKEND] Only have single entry type (#248) 2022-09-27 20:17:10 -07:00
Jesse Bannon
3719b14103
fix download strat docs (#235) 2022-09-15 22:13:44 -07:00
Jesse Bannon
3894625479
[BACKEND] Remove split_video download strategy (#232) 2022-09-14 22:03:44 -07:00
Jesse Bannon
c08ea64b5a
[BACKEND] Add generic.collection download strategy, use it for soundcloud, youtube channel/playlist (#230) 2022-09-14 15:36:14 -07:00
Jesse Bannon
968fa47b0f
[DOCS] Update readthedocs to only contain config documentation (#206) 2022-09-01 23:18:51 -07:00
Jesse Bannon
6d84e52467
[FEATURE] Add multiple preset inheritance (#204) 2022-09-01 10:43:07 -07:00
Jesse Bannon
5782445366
[FEATURE] Add ability for downloaders to add new override variables. Add source_title, source_uploader, and source_description for YouTube channels and playlists (#203) 2022-09-01 00:25:01 -07:00
Jesse Bannon
73be77c500
[FEATURE] date_range plugin, supports Override variables (#199) 2022-08-28 11:09:11 -07:00
Jesse Bannon
d79e94e7e6
[BACKEND] Use native xml library, remove dicttoxml (#196) 2022-08-27 15:01:14 -07:00
Jesse Bannon
f8386daadd
[DOCS] Fix nfo plugin names (#187) 2022-08-19 09:18:35 -07:00
Jesse Bannon
790c65859a
[FEATURE] Add split_by_chapters plugin (#184) 2022-08-18 23:24:03 -07:00
Jesse Bannon
2a666ef70b
[FEATURE] Chapters plugin with SponsorBlock support (#178) 2022-08-15 17:01:03 -07:00
Jesse Bannon
871c051aa8
[FEATURE] Audio extract plugin (#177) 2022-08-14 10:10:10 -07:00
Jesse Bannon
9a7a34f3d0
[FEATURE] Video tags plugin to embed metadata into video files (#173) 2022-08-13 09:53:55 -07:00
Jesse Bannon
1f09398ad5
[FEATURE] Subtitles support via Subtitles plugin (#169) 2022-08-12 09:58:08 -07:00
Jesse Bannon
5244b7c939
[DOCS] Clean up source variable readthedocs section (#152) 2022-08-04 17:34:06 -07:00
Jesse Bannon
53adbbcb22
[FEATURE] Regex capture and filtering on source variables plugin (#100)
* actually working

* changed plugin format. Need unit tests, documentation, update e2e to filter

* refactor plugin name, matching now working

* description work in progress, need more intricate testing

* tests looks good, regenerate output

* e2e tested, need more failure tests, no dupe variable names, add docs

* unit tests passing again

* more fail tests

* updated with name, default, better docs

* sanitized vars created
2022-07-16 23:39:46 -07:00
Jesse Bannon
df9f5a7836
[DOCS] Automated download via cron docs (#95)
* [DOCS] Automated download docs, fix -thumb.jpg

* patty
2022-07-08 11:46:47 -07:00
Jesse Bannon
607ad9f9de
[REFACTOR] Move downloaders to separate files (#91) 2022-07-02 21:48:41 -07:00
Jesse Bannon
cb6dc8e034
Add youtube.download_strategy: "merge_playlist" (#77)
* begin

* chapters class, trying to add chapter splitting and general ffmpeg metadata creation

* working split video

* so very close

* test passing, is reproducible

* lint fixed

* fix docs

* track no longer used in example

* if, then del

* multiple values

* github hash
2022-06-26 15:18:07 -07:00
Jesse Bannon
1620c3a42d
youtube.download_strategy split_video (#76)
* youtube.download_strategy split_video

* almost working, need to deal with track_title and other overwrites

* working test

* ci ffmpeg

* sudo -f

* fix code block
2022-06-18 23:12:10 -07:00
Jesse Bannon
ee6706334f
ytdl_options defaults for each download strategy (#72)
* `ytdl_options` defaults for each download strategy

* fix docker, fix defaults getting added, add debugger test

* test playlist as well

* fix with hack

* fix full channel test

* existing
2022-06-14 00:03:01 -07:00
Jesse Bannon
a79fcaa25b
Use URLs instead of IDs for sources (#70)
* youtube video url

* WIP, channel /user/ and /c/ should be valid

* unit tests passing

* soundcloud artist url

* soundcloud username

* break configs with name change

* Update all yamls with urls

* update all docs

* more negative url tests
2022-06-05 16:19:08 -07:00
Jesse Bannon
67104bca43
Move docker install into main readme (#61) 2022-06-02 22:34:09 -07:00
Jesse Bannon
a7524e1348
Simplify the README (#54)
* better readme

* code block ytdl-sub

* update links
2022-05-30 12:51:16 -07:00
Jesse Bannon
a80a834924
Umask support, fix overrides formatter variable error message (#46)
* umask support, fix error message

* fix error location

* make pylint disable area smaller, remove int
2022-05-28 00:28:54 -07:00
Jesse Bannon
7f79323e1f
readthedocs FAQ (#26) 2022-05-15 22:34:28 -07:00
Jesse Bannon
3543101592
simplified readme (#25) 2022-05-15 21:56:43 -07:00
Jesse Bannon
d17f677db3
Add inheritance for presets (#24) 2022-05-15 16:31:34 -07:00
Jesse Bannon
29ef377cc7
Add documentation for plugins + improve docs overall (#23) 2022-05-10 22:22:03 -07:00
Jesse Bannon
ab2df88299
Overhaul docs for readthedocs (#20) 2022-05-06 22:25:29 -07:00
jbannon
cc7664e838 update other things with new src layout 2022-04-26 07:23:01 +00:00
jbannon
bbf99947be docs are time consuming 2022-04-24 06:40:41 +00:00
jbannon
ebf859ebad strip longer name 2022-04-24 04:37:55 +00:00
jbannon
3e29a33968 get doc layout started 2022-04-24 04:30:35 +00:00
jbannon
5eeb117fca variables as mixins for docs 2022-04-24 00:30:53 +00:00
jbannon
7a3144a838 BROKEN STATE - in middle of refactoring for plugin support 2022-04-22 06:58:53 +00:00
jbannon
279de00c2b cleanup source docs 2022-04-17 05:30:25 +00:00
jbannon
142a32a762 docs, do not include built html 2022-04-17 05:06:16 +00:00
jbannon
c78d31118f sphinx working 2022-04-13 07:47:33 +00:00