Commit graph

81 commits

Author SHA1 Message Date
Jesse Bannon
b3d298b664
[DOCS] Advanced scripting usage, rename Overrides to Static (#866) 2024-01-01 00:01:46 -08:00
Jesse Bannon
735da8b647
[BUGFIX] Actually fix subscription_name with periods in it (#861)
Prior PR did not actually work - this actually fixes the bug (https://github.com/jmbannon/ytdl-sub/issues/844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name  Beast.

POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
2023-12-29 15:41:33 -08:00
Jesse Bannon
f2d55fdb23
[REFACTOR] Organize variable definitions (#860) 2023-12-29 14:39:19 -08:00
Jesse Bannon
189e897888
[FEATURE] subscription_map value (#859)
To be used later for advanced presets. For subscriptions in the form of (+ as prefix)

```
   + Subscription Name:
     Music Videos:
       - "https://url1.com/..."
     Concerts:
       - "https://url2.com/..."
```

Stores all the contents under the subscription name into the override variable
``subscription_map`` as a Map value. The above example is stored as:

```
   {
     "Music Videos": [
       "https://url1.com/..."
     ],
     "Concerts: [
       "https://url2.com/..."
     ]
   }
```

Advanced scripting is needed to dissect this Map variable into a usable preset
2023-12-29 08:55:12 -08:00
Jesse Bannon
731d4e444c
[DOCS] Tooling to auto-generate custom sphinx docs (#849)
Tooling + updating docs
2023-12-28 22:39:59 -08:00
Jesse Bannon
02d51d6aec
[BACKEND] Make sibling metadata optional to compute (#855)
Partial fix to https://github.com/jmbannon/ytdl-sub/issues/853
Makes computing sibling metadata optional, defaulted to False. This prevents excessive memory usage when scraping large channels
2023-12-26 11:57:03 -08:00
Jesse Bannon
308c76a8f2
[FEATURE] Filter plugins, regex helper script functions (#848)
Adds generic `filter_include` and `filter_exclude` plugins in anticipation for function script usage. Will detail it more with official docs at a later time!
2023-12-20 09:52:12 -08:00
Jesse Bannon
5d1fd0d7c6
[BACKEND] Validate override variable names (#845)
Validate as early as possible whether override names are invalid, to prevent confusing downstream errors
2023-12-18 17:11:25 -08:00
Jesse Bannon
e92b1cd12a
[BACKEND][HUGE] Function Support in variable syntax (#838)
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.

With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.

Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
2023-12-18 16:08:15 -08:00
Jesse Bannon
4d9a30779c
[BUGFIX] Use proper extractor key for download archive (#840)
For less popular sites, ytdl-sub's custom download archive was not working. This was because we weren't using the right extraction key (though it still worked for YouTube, Bandcamp, Soundcloud, and others). This bugfix should now exactly match yt-dlp's and never be a problem again.

Huge thanks to @Svagtlys for root-causing this 🤝
2023-12-14 17:04:03 -08:00
Jesse Bannon
e9b2214a0a
[DOCS] New README with only subscriptions (#791) 2023-11-02 23:28:55 -07:00
Jesse Bannon
1a118e225a
[BACKEND] Perform match-filters at metadata stage, use match-filters for date_range plugin (#725)
Solved Issue: https://github.com/jmbannon/ytdl-sub/issues/706

From our discussion with yt-dlp folks (https://github.com/yt-dlp/yt-dlp/issues/8108), we learned that best practices for date--range filtering is done using match-filters. We now set the `date_range` plugin values to be a 'breaking' match filter, and any custom ones specified from users in the `match_filters` plugin as non-breaking. This means match-filters can be performed at the metadata stage, which should yield a significant speedup

Relevant docs:
- date_range: https://ytdl-sub.readthedocs.io/en/latest/config.html#date-range
- match_filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
2023-09-19 11:19:15 -07:00
Jesse Bannon
5f9b8afc4b
[BUGFIX] Do not hard-fail if the thumbnail fails to download/convert (#675)
* [BUGFIX] Do not fail with thumbnail issues

* helper

* refactor

* dry-run

* regen fixture

* download arhcive txt -> json in docs, fixtures

* return if no download thumb path

* better test

* more test coverage
2023-07-28 13:12:46 -07:00
Jesse Bannon
557854fc83
[FEATURE] release_date source variables (#638)
* [FEATURE] `release_date` source variables

* lint
2023-06-24 23:01:18 -07:00
Jesse Bannon
0497f2b749
[FEATURE] Perform match-filter on download, not metadata pull (#612)
* [FEATURE] Apply match-filter on download instead of metadata

* correct ytdl_option ordering

* lint

* test that downloads
2023-05-18 09:52:24 -07:00
Jesse Bannon
38e64a5bd5
[FEATURE] Perform regex at both metadata and post-download (#611)
* [FEATURE] Perform regex at both metadata and post-download

* sadf
2023-05-17 22:33:42 -07:00
Benjamin Hsieh
14c7865165
[FEATURE] channel_id source variable (#575) 2023-03-31 21:25:13 -07:00
Benjamin Hsieh
e1f2e37858
[FEATURE] creator_sanitized source variable (#574) 2023-03-31 20:47:44 -07:00
Jesse Bannon
fa6eae7423
[FEATURE] creator source variable (#572) 2023-03-31 15:37:06 -07:00
Jesse Bannon
e69933d525
[REFACTOR] Split metadata and download into two separate steps (#540) 2023-03-13 22:41:09 -07:00
Jesse Bannon
0cd8df6e11
[BUGFIX] Fix extractor IDs in yt-dlp download archive (#528) 2023-03-10 12:48:53 -08:00
Jesse Bannon
73094e79bb
[FEATURE] Plex sanitized title and uid. Add into prebuilt preset (#449)
* [FEATURE] Plex sanitized title and uid. Add into prebuilt preset

* update fixtures, add fixture test

* lint
2023-02-20 10:19:44 -08:00
Jesse Bannon
3106861a42
[BACKEND] Additional filtering on root parent search (#445)
* [BACKEND] Additional filtering on root parent search

* update failing test
2023-02-17 23:32:02 -08:00
Jesse Bannon
30c77565bc
[BACKEND] Add playlist_uploader_sanitized var (#414) 2023-01-13 10:47:43 -08:00
Jesse Bannon
beaa34a367
[BUGFIX] Fix entry missing __contains__ function (#392)
* [BUGFIX] Fix missing contains function when checking entries

* remove children func

* Update failing hash
2022-12-13 00:23:16 -08:00
Jesse Bannon
92c9988987
[BACKEND] Reduce memory footprint (#375)
* [BACKEND] Reduce memory footprint

* entry refactor
2022-11-27 22:38:50 -08:00
Jesse Bannon
954dc0f3ee
[FEATURE] Add better download progress (#366) 2022-11-26 23:33:14 -08:00
Jesse Bannon
2f792f774c
[FEATURE] Add custom ffmpeg post-processing (#364) 2022-11-26 14:56:34 -08:00
Jesse Bannon
ffaf07d628
[FEATURE] Add support to scrape chapters from comments (#347)
* [FEATURE] Add support to scrape chapters from comments

* tested for comments

* remove timestamps file

* change order of docstrings
2022-11-21 11:19:17 -08:00
Jesse Bannon
20bec0c664
[BUGFIX] Fix split_by_chapter download archive not working (#344) 2022-11-19 23:08:59 -08:00
Jesse Bannon
5705a28842
[FEATURE] Add ytdl-sub view URL to view source variables (#340)
* [FEATURE] Add `ytdl-sub view URL` to view source variables

* update file path
2022-11-19 16:56:45 -08:00
Jesse Bannon
978dc36713
[BACKEND] Optimize metadata fetching (#335)
* [BACKEND] Do not fetch unnecessary metadata when grabbing .info.json files"

* update with more optimized ytdl options
2022-11-16 12:54:18 -08:00
Jesse Bannon
26b6ed0add
[BUGFIX] Fix multiple top-level parents issue (#325) 2022-11-14 23:01:34 -08:00
Jesse Bannon
86c7e7b84e
[BUGFIX] Retry downloading entry thumbnail if missing (#293) 2022-10-16 22:25:59 -07:00
Jesse Bannon
9883907bf0
[BUGFIX] Use yt-dlp remuxer instead of our own (#275)
* [BUGFIX] Use yt-dlps remuxer instead of our own

* docstring
2022-10-07 17:16:54 -07:00
Jesse Bannon
5b8ee7f2de
[FEATURE] download_index source variable, new tv presets (#271) 2022-10-05 12:08:43 -07:00
Jesse Bannon
5b9a5dae50
[FEATURE] Prebuilt presets (#264) 2022-10-01 22:41:36 -07:00
Jesse Bannon
6d52177104
[BACKEND] upload_date_index source variables (#261)
* [BACKEND] `upload_date_index` source variables

* len
2022-09-30 22:41:06 -07:00
Jesse Bannon
cd9a094e2c
[BACKEND] epoch and playlist_index_reversed variables (#257) 2022-09-29 15:39:53 -07:00
Jesse Bannon
7d29b72010
[BACKEND] uid_sanitized variable (#256) 2022-09-29 00:33:57 -07:00
Jesse Bannon
ebdd6c8aec
[BACKEND] Only have single entry type (#248) 2022-09-27 20:17:10 -07:00
Jesse Bannon
8edca7af36
[BACKEND] Do not default when building parent kwargs (#247) 2022-09-27 18:39:29 -07:00
Jesse Bannon
32577bbad5
[BACKEND] Expose raw playlist_index and playlist_count to all download strategies (#245) 2022-09-27 17:17:56 -07:00
Jesse Bannon
1486acb421
[BACKEND] Parallelize tests, remove track and artist source variables (#246) 2022-09-27 16:51:42 -07:00
Jesse Bannon
3db6a543cf
[CLEANUP] Remove unused functions (#240) 2022-09-17 11:43:33 -07:00
Jesse Bannon
64d4be0a42
[BACKEND] output_directory_nfo_tags to support source vars (#239) 2022-09-17 11:11:37 -07:00
Jesse Bannon
46c14bd842
[BACKEND] uploader variables (#238) 2022-09-17 09:04:49 -07:00
Jesse Bannon
3ace86435d
[BACKEND] Defined kwarg strings. Add playlist/source description variable (#237) 2022-09-17 08:09:09 -07:00
Jesse Bannon
c66e5b72f3
[BACKEND] Add source and playlist variables to every entry (#236) 2022-09-16 22:56:56 -07:00
Jesse Bannon
c08ea64b5a
[BACKEND] Add generic.collection download strategy, use it for soundcloud, youtube channel/playlist (#230) 2022-09-14 15:36:14 -07:00