Subscription file syntax is designed to minimize boiler-plate when authoring new subscriptions.
You can now unpack any subscription using the ``inspect`` sub-command to see its boiler-plate *preset format*.
```
ytdl-sub inspect --config /path/to/config.yaml --match "BBC News" /path/to/subscriptions.yaml
```
This can be utilized for numerous purposes including:
* Ensuring your custom preset is getting applied correctly.
* Figuring out which variables set things like file names, metadata, etc.
* Understanding how subscription syntax translates to preset representation.
The default ``--level`` of inspect will fill in defined variables. Using ``--level original`` will present the subscription's raw layout with no fill.
Expands variable validation to also include support for partial resolution.
In short, this will partially execute script code until it hits unresolved runtime variables, and store that as the new script representation. This functionality will allow for the upcoming `inspect` command, which will show users their script code in action without having to dry-run.
Makes the underlying formatting process better handle native (non-string) types in a more flexible way. 2nd attempt since the last one got reverted due to breaking changes.
Allows for the `download` plugin to take in a script-based array of URLs. Prebuilt presets are now greatly reduced in size by constructing urls as an array versus 100+ separate URL variables.
Eventually, we can completely remove the limit of the number of URLs in a subscription.
Completely rewrite how subscription validation is performed. Optimizes it quite a bit while also adding backend support for the upcoming `dissect` sub-command, where you can resolve any subscription into its 'raw' form for easier debugging when making scripting changes.
Adds support for setting throttle_protection range values using static override variables. `sleep_per_download_s` has additional support for entry variables since it's used per entry.
We can now experiment with scaling this value based on entry attributes, such as duration. Example:
```
throttle_protection:
sleep_per_download_s:
min: >-
{
%mul(5.5, %pow( duration, 0.4 ))
}
max: >-
{
%mul(6.5, %pow( duration, 0.6 ))
}
```
Adds the ability to toggle whether to use `upload_date` or `release_date` in the date_range plugin, which is used to cull older videos.
```
date_range:
type: "upload_date"
```
If using the Only Recent preset, it will be automatically applied based on whether your season/episode ordering uses release date or upload date.
Closes https://github.com/jmbannon/ytdl-sub/issues/1182
Implements https://github.com/jmbannon/ytdl-sub/issues/1182
Adds the ability to change the `keep_files_date_eval` (traditionally upload_date_standardized) to any date, so long as its in the form of YYYY-MM-DD.
Will be able to use this to have episodes get deleted based on their release_date or epoch_date, instead of their upload_date.
Adds the ability to create map and list-based override variables.
For example, you can now create lists like this:
```
overrides:
urls:
- "https://...1"
- "https://...2"
```
which is equivalent to:
```
overrides:
urls: >-
{
[
"https://...1",
"https://...2",
]
}
```
Likewise, maps can now look like:
```
overrides:
music_video_category:
concerts:
- "https://...1"
- "https://...2"
interviews:
- "https://...3"
```
which is equivalent to:
```
overrides:
music_video_category: >-
{
"concerts": [
"https://...1",
"https://...2"
],
"interviews": [
"https://...3"
]
}
```
With the right functions, it was possible for subscription validation to raise a false-positive error. This should hopefully resolve that issue and give validation a small performance increase
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
Adds a new mechanism to limit subscriptions' number of videos to an explicit number (only supported date ranges prior to this).
Example usage:
```
output_options:
keep_max_files: 10
```
will only keep 10 videos at max.
In addition:
- `keep_max_files` is bundled into the `Only Recent` preset and can be used via setting the `only_recent_max_files` override variable (see examples or README for usage).
- Changed the `date_range` variable name to `only_recent_date_range` for more clarity. Usage of `date_range` will continue to work.
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.
Usage:
```
throttle_protection:
sleep_per_download_s:
min: 2.2
max: 10.8
sleep_per_subscription_s:
min: 9.0
max: 14.1
max_downloads_per_subscription:
min: 10
max: 36
subscription_download_probability: 1.0
```
Closes Feature Request: https://github.com/jmbannon/ytdl-sub/issues/743
Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:
```
tv_show:
only_recent:
[News]:
"Breaking News": "https://www.youtube.com/@SomeBreakingNews"
[Tech]:
"Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
```
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:
```
"Breaking News":
preset:
- "tv_show"
- "only_recent"
overrides:
subscription_indent_1: "News"
subscription_name: "Breaking News"
subscription_value: "https://www.youtube.com/@SomeBreakingNews"
"Two Minute Papers":
preset:
- "tv_show"
overrides:
subscription_indent_1: "Tech"
subscription_name: "Two Minute Papers" subscription_value: "https://www.youtube.com/@TwoMinutePapers"
```
You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things you want configurable like so in your
parent preset:
```
presets:
tv_show_name:
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
genre: "{subscription_indent_1}"
```
NOTE!!!
Changing your subscription name from 'legacy' subscriptions will need their download archive file migrated to a new name. A future update will assist in this migration process. Either wait until then or, if you consider yourself a ytdl-sub expert, keep a backup of your subscription file before migrating.
* [FEATURE
] Reformat existing downloads
* working?!?!
* reformat in output dir
* close, need info json enabled by default
* asdfadsfsaf
* lint
* closer
* multi created variables do not exist for update command!
* info json downloader
* [REFACTOR] Shared plugin and download options class
* more refactoring
* lint
* put in initialize plugins
* changes
* [REFACTOR] simplify enhanced download archive
* working!
* in main
* simplified, many url tests working
* ready?
* [BUGFIX] Fix missing contains function when checking entries
* remove children func
* [BACKEND] Bandcamp e2e test, update music example
* WIP
* combine soundcloud and all music into music_audio config
* sc fixtures
* updated tests and examples
* bandcamp e2e test
* regen off
* remove url validators
* use bandcamp artist url for more parent coverage
* lint
