Subscription file syntax is designed to minimize boiler-plate when authoring new subscriptions.
You can now unpack any subscription using the ``inspect`` sub-command to see its boiler-plate *preset format*.
```
ytdl-sub inspect --config /path/to/config.yaml --match "BBC News" /path/to/subscriptions.yaml
```
This can be utilized for numerous purposes including:
* Ensuring your custom preset is getting applied correctly.
* Figuring out which variables set things like file names, metadata, etc.
* Understanding how subscription syntax translates to preset representation.
The default ``--level`` of inspect will fill in defined variables. Using ``--level original`` will present the subscription's raw layout with no fill.
Expands variable validation to also include support for partial resolution.
In short, this will partially execute script code until it hits unresolved runtime variables, and store that as the new script representation. This functionality will allow for the upcoming `inspect` command, which will show users their script code in action without having to dry-run.
Makes the underlying formatting process better handle native (non-string) types in a more flexible way. 2nd attempt since the last one got reverted due to breaking changes.
Allows for the `download` plugin to take in a script-based array of URLs. Prebuilt presets are now greatly reduced in size by constructing urls as an array versus 100+ separate URL variables.
Eventually, we can completely remove the limit of the number of URLs in a subscription.
Completely rewrite how subscription validation is performed. Optimizes it quite a bit while also adding backend support for the upcoming `dissect` sub-command, where you can resolve any subscription into its 'raw' form for easier debugging when making scripting changes.
Disallow override variable names conflicting with plugin names.
May cause backward incompatible issues from `date_range` being an old override variable name for the `Only Recent` preset. Update this variable name to `only_recent_date_range` to resolve. More info in https://ytdl-sub.readthedocs.io/en/latest/deprecation_notices.html
Closes https://github.com/jmbannon/ytdl-sub/issues/386
Adds the ability to preserve mtime via Output Options. Usage:
```
output_options:
preserve_mtime: True
```
Thanks @e1ven for the contribution!
Changes documentation and default configs to use the `/tv_shows` default path shown in the docker compose setup.
Also changes working directory and file lock directory to not be in /tmp, which often causes permission issues.
* docs(logs): Mimick argparse option default format
I find it more readable to use punctuation to separate technical information, such as
required vs optional or default values, from narrative description. Follow argparse's
lead, and put such information in parens following the narrative description.
* docs(usage): Consistent structure, clarifications
* docs(config): Clarify persist_logs behavior/opts
Clarify the `persist_logs:` options per [Discord discussion](https://discord.com/channels/994270357957648404/1409161361853780060/1409602529460879431).
Adds the ability to make thumbnails square, both file and embedded. Usage:
```
my_preset:
square_thumbnail: True
```
Will soon enable this by default for all music-based presets with a toggle to disable.
Huge thanks to @Kentaro1043 for crafting the ffmpeg command for this!
Closes https://github.com/jmbannon/ytdl-sub/issues/383
Gives Emby first class support by creating prebuilt presets for each variant, including
- `Emby TV Show by Date`
- `Emby TV Show Collection`
- `Emby Music Videos`
Emby is identical to Jellyfin except in TV Show Collection, where it requires a `season.nfo`. A new plugin `StaticNfoTags` has been created to create those.
Huge thanks to @Kamaroth92 for the initial code for this feature!
Implements https://github.com/jmbannon/ytdl-sub/issues/1182
Adds the ability to change the `keep_files_date_eval` (traditionally upload_date_standardized) to any date, so long as its in the form of YYYY-MM-DD.
Will be able to use this to have episodes get deleted based on their release_date or epoch_date, instead of their upload_date.
Regex plugin has been removed in favor of scripting. The function [regex_capture_many](https://ytdl-sub.readthedocs.io/en/latest/config_reference/scripting/scripting_functions.html#regex-capture-many) has been created to replicate the plugin's behavior. See the following converted example:
`regex, now deprecated`
```
regex:
from:
title:
match:
- ".*? - (.*)" # Captures 'Song' from 'Emily Hopkins - Some - Song'
capture_group_names:
- "captured_track_title"
capture_group_defaults:
- "{title}"
overrides:
track_title: "{captured_track_title}"
```
`scripting equivalent`
```
overrides:
# Captures 'Song' from 'Emily Hopkins - Some - Song'
captured_track_title: >-
{
%regex_capture_many(
title,
[ ".*? - (.*)" ],
[ title ]
)
}
track_title: "{%array_at(captured_track_title, 1)}"
```
## Motivation:
Regex was a unique plugin that added custom variables based on user input. This made the backend need to support both `overrides` and 'plugin user variables'. Removing `regex` plugin will consolidate this logic into only `overrides`, making it much easier to maintain.
Adds the ability to create map and list-based override variables.
For example, you can now create lists like this:
```
overrides:
urls:
- "https://...1"
- "https://...2"
```
which is equivalent to:
```
overrides:
urls: >-
{
[
"https://...1",
"https://...2",
]
}
```
Likewise, maps can now look like:
```
overrides:
music_video_category:
concerts:
- "https://...1"
- "https://...2"
interviews:
- "https://...3"
```
which is equivalent to:
```
overrides:
music_video_category: >-
{
"concerts": [
"https://...1",
"https://...2"
],
"interviews": [
"https://...3"
]
}
```
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
Adds a the new field `breaking` to the `date_range` plugin, to toggle whether an entry breaks subsequent metadata pulls. This is useful to disable if you are grabbing a playlist that may have videos out of order, but still want to apply a date range to it.
With the right functions, it was possible for subscription validation to raise a false-positive error. This should hopefully resolve that issue and give validation a small performance increase
Reduce memory usage significantly when pulling large channels all at once by
- Not loading entry script until it is being processed
- Deleting the entry memory once fully processed
- Disable yt-dlp from keeping entry info in memory since ytdl-sub reads it from file
Prior PR did not actually work - this actually fixes the bug (https://github.com/jmbannon/ytdl-sub/issues/844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name Beast.
POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
Removes the deprecated `__value__` field in the subscriptions file (different from `__preset__` which is NOT deprecated). This was very short-lived, and most likely not used.
Adds generic `filter_include` and `filter_exclude` plugins in anticipation for function script usage. Will detail it more with official docs at a later time!
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
Adds a new mechanism to limit subscriptions' number of videos to an explicit number (only supported date ranges prior to this).
Example usage:
```
output_options:
keep_max_files: 10
```
will only keep 10 videos at max.
In addition:
- `keep_max_files` is bundled into the `Only Recent` preset and can be used via setting the `only_recent_max_files` override variable (see examples or README for usage).
- Changed the `date_range` variable name to `only_recent_date_range` for more clarity. Usage of `date_range` will continue to work.
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.
Usage:
```
throttle_protection:
sleep_per_download_s:
min: 2.2
max: 10.8
sleep_per_subscription_s:
min: 9.0
max: 14.1
max_downloads_per_subscription:
min: 10
max: 36
subscription_download_probability: 1.0
```
