Jesse Bannon
f2d55fdb23
[REFACTOR] Organize variable definitions ( #860 )
2023-12-29 14:39:19 -08:00
Jesse Bannon
731d4e444c
[DOCS] Tooling to auto-generate custom sphinx docs ( #849 )
...
Tooling + updating docs
2023-12-28 22:39:59 -08:00
Jesse Bannon
02d51d6aec
[BACKEND] Make sibling metadata optional to compute ( #855 )
...
Partial fix to https://github.com/jmbannon/ytdl-sub/issues/853
Makes computing sibling metadata optional, defaulted to False. This prevents excessive memory usage when scraping large channels
2023-12-26 11:57:03 -08:00
Jesse Bannon
e92b1cd12a
[BACKEND][HUGE] Function Support in variable syntax ( #838 )
...
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
2023-12-18 16:08:15 -08:00
Jesse Bannon
e257210e4a
[BACKEND] Shared multi-url helper preset, can now disable grabbing poster/fanart ( #807 )
2023-11-05 16:51:17 -08:00
Jesse Bannon
6615e1d1ac
[BACKEND] Always filter out live, post_live, and upcoming videos on download ( #742 )
...
Closes Issue: https://github.com/jmbannon/ytdl-sub/issues/729
Add a new field to the match_filter plugin: `download_match_filters`. This set of filters will be applied at the download stage (as opposed to `filters` which get applied at the metadata stage). By default, always set `"!is_live & !is_upcoming & !post_live"` as a download filter. This prevents ytdl-sub from hanging when trying to grab live and upcoming videos.
Docs on match-filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-24 06:24:21 -07:00
Jesse Bannon
1a118e225a
[BACKEND] Perform match-filters at metadata stage, use match-filters for date_range plugin ( #725 )
...
Solved Issue: https://github.com/jmbannon/ytdl-sub/issues/706
From our discussion with yt-dlp folks (https://github.com/yt-dlp/yt-dlp/issues/8108 ), we learned that best practices for date--range filtering is done using match-filters. We now set the `date_range` plugin values to be a 'breaking' match filter, and any custom ones specified from users in the `match_filters` plugin as non-breaking. This means match-filters can be performed at the metadata stage, which should yield a significant speedup
Relevant docs:
- date_range: https://ytdl-sub.readthedocs.io/en/latest/config.html#date-range
- match_filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-19 11:19:15 -07:00
Jesse Bannon
5f9b8afc4b
[BUGFIX] Do not hard-fail if the thumbnail fails to download/convert ( #675 )
...
* [BUGFIX] Do not fail with thumbnail issues
* helper
* refactor
* dry-run
* regen fixture
* download arhcive txt -> json in docs, fixtures
* return if no download thumb path
* better test
* more test coverage
2023-07-28 13:12:46 -07:00
Jesse Bannon
a9e6a80111
[REFACTOR] Support simple download ( #671 )
...
* [REFACTOR] Support simple `download`
* fix test
* asdfas
2023-07-26 09:00:00 -07:00
Jesse Bannon
598b1da8a4
[REFACTOR] Purge download_strategy from configs + tests ( #670 )
2023-07-25 20:26:04 -07:00
Jesse Bannon
d24048bb01
[REFACTOR] No more download strategy ( #669 )
...
* [REFACTOR] No more download strategy
* remove more
2023-07-25 19:51:50 -07:00
Jesse Bannon
3c0d05a383
[REFACTOR] Single download options ( #668 )
...
* [REFACTOR] Single download options
* update
* fix unit test msg
2023-07-25 14:41:33 -07:00
Jesse Bannon
bf3c61213b
[REFACTOR] SourcePluginExtension ( #667 )
...
* [REFACTOR] SourcePluginExtension
* param name
* priority to plugin extensions
* info json
2023-07-25 14:04:18 -07:00
Jesse Bannon
1236af270d
[REFACTOR] SourcePlugin ( #666 )
...
* [REFACTOR] SourcePlugin
* param name
2023-07-25 12:37:56 -07:00
Jesse Bannon
e3c662bc35
[REFACTOR] Shared plugin code ( #664 )
...
* [REFACTOR] Shared plugin code
* arg names
* all plugins have priority
* split abstract method
* lint
* lint
2023-07-25 12:11:50 -07:00
Jesse Bannon
dcd636192c
[REFACTOR] download abstraction overhaul ( #663 )
...
* [REFACTOR] `download` abstraction overhaul
* remove plugin options
2023-07-25 00:36:51 -07:00
Jesse Bannon
7e997f304d
[REFACTOR] OptionsValidator + OptionsDictValidator ( #657 )
...
* [REFACTOR] OptionsValidator + OptionsDictValidator
* more
2023-07-24 10:12:38 -07:00
Jesse Bannon
b8d1f1edbb
[FEATURE] Ability to change download archive name ( #643 )
2023-06-30 11:51:52 -07:00
Jesse Bannon
134882a17d
[DOCS] Fix url + multi_url docs ( #635 )
2023-06-17 13:45:22 -07:00
Jesse Bannon
7816c2fe59
[BUGFIX] Add check_formats on download retry ( #623 )
2023-05-30 23:20:40 -07:00
Jesse Bannon
0497f2b749
[FEATURE] Perform match-filter on download, not metadata pull ( #612 )
...
* [FEATURE] Apply match-filter on download instead of metadata
* correct ytdl_option ordering
* lint
* test that downloads
2023-05-18 09:52:24 -07:00
Jesse Bannon
1965bc0a40
[FEATURE] Multiple URLs supported for tv_show_by_date preset ( #568 )
...
* [BACKEND] Support empty urls for multi_url, add overrides for extra urls in tv_show_by_date preset
* need to check for missing urls in downloader
* use override in test
* docs about multi url
* wording
2023-03-30 09:28:32 -07:00
Jesse Bannon
70d81bf20f
[BACKEND] Mark entry as deleted if all entries deleted in update ( #559 )
...
* [BACKEND] Mark entry as deleted if all entries deleted in update
* make subtitles not found debug
2023-03-21 14:21:02 -07:00
Jesse Bannon
54b8b76034
[BACKEND] Handle case when collection url is missing from entry ( #558 )
...
* [BACKEND] Handle case when collection url is missing from entry
* hide experimental docs
2023-03-21 13:39:41 -07:00
Jesse Bannon
7385aac1ee
[FEATURE][EXPERIMENTAL] --update-with-info-json ( #557 )
...
* [FEATURE
] Reformat existing downloads
* working?!?!
* reformat in output dir
* close, need info json enabled by default
* asdfadsfsaf
* lint
* closer
* multi created variables do not exist for update command!
* info json downloader
* [REFACTOR] Shared plugin and download options class
* more refactoring
* lint
* put in initialize plugins
* changes
* [REFACTOR] simplify enhanced download archive
* working!
* in main
* simplified, many url tests working
* ready?
2023-03-21 08:05:08 -07:00
Jesse Bannon
14ddd750d9
[REFACTOR] Shared OptionsValidator, downloader plugins ( #550 )
2023-03-15 23:06:55 -07:00
Jesse Bannon
7c692cda6e
[REFACTOR] BaseDownloader ( #542 )
...
* [REFACTOR] BaseDownloader
* doh
* fix mock
2023-03-14 23:42:50 -07:00
Jesse Bannon
e69933d525
[REFACTOR] Split metadata and download into two separate steps ( #540 )
2023-03-13 22:41:09 -07:00
Jesse Bannon
1922296c31
[REFACTOR] BaseDownloader class ( #539 )
2023-03-13 00:42:32 -07:00
Jesse Bannon
f42973cf2f
[FEATURE] download_reverse in URL download strategy ( #531 )
...
* [FEATURE] download_reverse in URL download strategy
* doc fix
2023-03-10 16:13:02 -08:00
Jesse Bannon
91e9fe7602
[FEATURE] Add summary to console output ( #515 )
2023-03-08 15:34:58 -08:00
Jesse Bannon
5d40d02eb3
[BACKEND] Make break message debug level and more clear ( #496 )
2023-03-02 13:01:01 -08:00
Jesse Bannon
798bbbc62c
[BACKEND] Windows support ( #476 )
2023-02-28 17:07:03 -08:00
Jesse Bannon
4ba41c757c
[BACKEND] More verbose logging when breaking ( #450 )
...
* [BACKEND] More verbose logging when breaking
* make thumbnail id a debugger log
* mock for info log
* info
2023-02-20 10:50:04 -08:00
Jesse Bannon
32f1398d60
[BUGFIX] Fix playlist season thumbnails not appearing ( #416 )
2023-01-13 14:05:56 -08:00
Jesse Bannon
053042b4b9
[BACKEND] Purge youtube + soundcloud download strategies ( #408 )
...
* [BACKEND] Purge youtube + soundcloud download strategies
* update unit tests
* fix missing for ffmpeg;
* try update
2023-01-11 22:05:50 -08:00
Jesse Bannon
96e741c71d
[BACKEND] Proactively clean working directory ( #379 )
...
* fix playlist last thumbnail move to copy
* playlist working
* channel working
* working directory getting cleaned
* working dir e2e fixture
* cleanup files in any case
* clean up if pre-split returns None
2022-11-30 10:36:13 -08:00
Jesse Bannon
8dd57beebc
[BACKEND] Pass deep copy of ytdl_options to yt-dlp ( #376 )
2022-11-27 23:40:44 -08:00
Jesse Bannon
92c9988987
[BACKEND] Reduce memory footprint ( #375 )
...
* [BACKEND] Reduce memory footprint
* entry refactor
2022-11-27 22:38:50 -08:00
Jesse Bannon
001d516c56
[BACKEND] Fetch thumbnails after first downloaded entry ( #368 )
2022-11-27 00:20:14 -08:00
Jesse Bannon
9f5ee153b7
[BACKEND] Add URL thumbnail download timeout ( #367 )
2022-11-26 23:44:40 -08:00
Jesse Bannon
954dc0f3ee
[FEATURE] Add better download progress ( #366 )
2022-11-26 23:33:14 -08:00
Jesse Bannon
ffaf07d628
[FEATURE] Add support to scrape chapters from comments ( #347 )
...
* [FEATURE] Add support to scrape chapters from comments
* tested for comments
* remove timestamps file
* change order of docstrings
2022-11-21 11:19:17 -08:00
Jesse Bannon
978dc36713
[BACKEND] Optimize metadata fetching ( #335 )
...
* [BACKEND] Do not fetch unnecessary metadata when grabbing .info.json files"
* update with more optimized ytdl options
2022-11-16 12:54:18 -08:00
Jesse Bannon
10b58264ea
[BACKEND] Update music video example with url download strategy ( #312 )
...
* [DOCS] document file __preset__
* [REFACTOR] Use `download` type in examples
* TODO: download strategy docs, validate fixtures, fix playlist test
* playlist fixed, fixtures look good
* download strategy docs, fix tests
* better docs
* lint
* yt fixed
* split chapters fixed
* fix video
* better docs
* lint
2022-11-11 08:54:07 -08:00
Jesse Bannon
255805c871
[REFACTOR] Rename source/collection to url/multi_url ( #310 )
2022-11-06 17:00:26 -08:00
Jesse Bannon
da22c9a608
[REFACTOR] Rename generic download source to download ( #309 )
...
* [REFACTOR] Rename `generic` download source to `download`
* no change to download strategy
2022-11-06 16:24:06 -08:00
Jesse Bannon
44ec463ab3
[BACKEND] More strict validation for config files ( #301 )
2022-10-26 00:00:36 -07:00
Jesse Bannon
1de5014c2f
[BUGFIX] Fix collection urls using wrong variables ( #297 )
2022-10-20 21:57:36 -07:00
Jesse Bannon
86c7e7b84e
[BUGFIX] Retry downloading entry thumbnail if missing ( #293 )
2022-10-16 22:25:59 -07:00
