Subscription file syntax is designed to minimize boiler-plate when authoring new subscriptions.
You can now unpack any subscription using the ``inspect`` sub-command to see its boiler-plate *preset format*.
```
ytdl-sub inspect --config /path/to/config.yaml --match "BBC News" /path/to/subscriptions.yaml
```
This can be utilized for numerous purposes including:
* Ensuring your custom preset is getting applied correctly.
* Figuring out which variables set things like file names, metadata, etc.
* Understanding how subscription syntax translates to preset representation.
The default ``--level`` of inspect will fill in defined variables. Using ``--level original`` will present the subscription's raw layout with no fill.
Expands variable validation to also include support for partial resolution.
In short, this will partially execute script code until it hits unresolved runtime variables, and store that as the new script representation. This functionality will allow for the upcoming `inspect` command, which will show users their script code in action without having to dry-run.
Makes the underlying formatting process better handle native (non-string) types in a more flexible way. 2nd attempt since the last one got reverted due to breaking changes.
Adds the mechanism to partially resolve an entire script (aka every underlying variable in ytdl-sub). This will be used in the upcoming `inspect` sub-command where it will dump a subscription's raw contents at multiple levels of resolution.
Potentially useful to optimize script execution as well.
Allows for the `download` plugin to take in a script-based array of URLs. Prebuilt presets are now greatly reduced in size by constructing urls as an array versus 100+ separate URL variables.
Eventually, we can completely remove the limit of the number of URLs in a subscription.
Completely rewrite how subscription validation is performed. Optimizes it quite a bit while also adding backend support for the upcoming `dissect` sub-command, where you can resolve any subscription into its 'raw' form for easier debugging when making scripting changes.
Disallow override variable names conflicting with plugin names.
May cause backward incompatible issues from `date_range` being an old override variable name for the `Only Recent` preset. Update this variable name to `only_recent_date_range` to resolve. More info in https://ytdl-sub.readthedocs.io/en/latest/deprecation_notices.html
Closes https://github.com/jmbannon/ytdl-sub/issues/386
Adds the ability to preserve mtime via Output Options. Usage:
```
output_options:
preserve_mtime: True
```
Thanks @e1ven for the contribution!
* docs(config): Clarify preset ordering priority
Refs #1276
* docs(architecture): WIP Narrative how it works
Much of this is repetition of the reference docs and some of the changes here should
also be applied here. The goal is to repeat here the parts of the reference docs that
may hang up a user when reading "Getting Started" without sending them off to get lost
in the weeds of the reference docs.
Note that this references a `Quick Start` document which is just an empty placeholder
and is to be written in a subsequent change.
Fixes#1279
* docs(terms): Glossary redundant with architecture
Now that the introduction includes an architectural overview, that section introduces
new users to necessary concepts and associated terminology and makes the terminology
section redundant in that part of the docs.
* docs(quick): Add a rote quick start guide
This is what I came up with when I tried to write instructions requiring as little
understanding as possible. Doing so really did reinforce my impression that this just
shouldn't be done, that significant understanding is required for *any* use of ytdl-sub
and even offering a quick start may be irresponsible. I'm still on the fence, thoughts?
That said, I also think I got some good explanations out of this and a better
understanding of what the next bits of the Getting Started should be. This Quick Start
is currently redundant with other parts of the Getting Started pages. I suspect that
I'll end up repeating and/or reorganizing parts of this Quick Start into the next bits
of the Getting Started. To that end I might argue that this change is a WIP and that
merging should wait. But I could also argue that this is an incremental improvement and
can be merged before that other work. Your call.
* docs(start): Various rST/Sphinx markup issues
Adds the ability to make thumbnails square, both file and embedded. Usage:
```
my_preset:
square_thumbnail: True
```
Will soon enable this by default for all music-based presets with a toggle to disable.
Huge thanks to @Kentaro1043 for crafting the ffmpeg command for this!
Closes https://github.com/jmbannon/ytdl-sub/issues/383
Adds the function `%contains_any`, and list support in a tilda override subscription. The end-goal of these features are to more easily add a title exclude list, like so:
```
__preset__:
filter_exclude:
- "{%contains_any( %lower(title), exclude_title_strings )}"
...
Jellyfin TV Show by Date:
~History Documentaries:
url: "https://..."
exclude_title_strings:
- "trailer"
- "preview"
```
A proper prebuilt preset or built-in functionality will follow this change.
Enhances the music video presets by supporting the following syntax:
`subscriptions.yaml`
```
__preset__:
overrides:
music_video_directory: "/music_videos"
# Choose between Jellyfin/Kodi/Plex Music Videos preset:
# - Plex Music Videos:
# - Jellyfin Music Videos:
# - Kodi Music Videos:
#
"Plex Music Videos":
= Pop: # Sets genre tag to "Pop"
"Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
"Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E"
= Rock:
# Prefixing with '+' puts the subscription into 'map-mode'.
# Music video presets in map-mode support grouping videos into different
# categories, which get set on the album field.
#
# URLs can either be strings, or maps that can overload title, year, date
"+ Guns N' Roses":
Music Videos:
- "https://www.youtube.com/playlist?list=PLOTK54q5K4INNXaHKtmXYr6J7CajWjqeJ"
Concerts:
- title: "Live at The Ritz - New York City"
year: "1988"
url: "https://www.youtube.com/watch?v=OldpIhHPsbs"
- title: "Live at The Hollywood Bowl"
date: "2023-01-11"
url: "https://www.youtube.com/watch?v=Z7hutGlvq9I"
```
The 'map-mode' (denoted by `+`) lets you specify a map of music video categories. Under each category, a URL can be specified as-is or with additional metadata (title, year/date, url).
Music videos and concerts, especially older ones, are typically uploaded by random users with inconsistencies in their titles. This syntax aims to make it easy to specify individual URLs with the ability to overwrite their title and year, and group them by category.
Adds the ability to create map and list-based override variables.
For example, you can now create lists like this:
```
overrides:
urls:
- "https://...1"
- "https://...2"
```
which is equivalent to:
```
overrides:
urls: >-
{
[
"https://...1",
"https://...2",
]
}
```
Likewise, maps can now look like:
```
overrides:
music_video_category:
concerts:
- "https://...1"
- "https://...2"
interviews:
- "https://...3"
```
which is equivalent to:
```
overrides:
music_video_category: >-
{
"concerts": [
"https://...1",
"https://...2"
],
"interviews": [
"https://...3"
]
}
```
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
With the right functions, it was possible for subscription validation to raise a false-positive error. This should hopefully resolve that issue and give validation a small performance increase
Prior PR did not actually work - this actually fixes the bug (https://github.com/jmbannon/ytdl-sub/issues/844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name Beast.
POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
To be used later for advanced presets. For subscriptions in the form of (+ as prefix)
```
+ Subscription Name:
Music Videos:
- "https://url1.com/..."
Concerts:
- "https://url2.com/..."
```
Stores all the contents under the subscription name into the override variable
``subscription_map`` as a Map value. The above example is stored as:
```
{
"Music Videos": [
"https://url1.com/..."
],
"Concerts: [
"https://url2.com/..."
]
}
```
Advanced scripting is needed to dissect this Map variable into a usable preset
Fixes the bug (#844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name Beast.
POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
Removes the deprecated `__value__` field in the subscriptions file (different from `__preset__` which is NOT deprecated). This was very short-lived, and most likely not used.
Partial fix to https://github.com/jmbannon/ytdl-sub/issues/853
Makes computing sibling metadata optional, defaulted to False. This prevents excessive memory usage when scraping large channels
Adds generic `filter_include` and `filter_exclude` plugins in anticipation for function script usage. Will detail it more with official docs at a later time!
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.
Usage:
```
throttle_protection:
sleep_per_download_s:
min: 2.2
max: 10.8
sleep_per_subscription_s:
min: 9.0
max: 14.1
max_downloads_per_subscription:
min: 10
max: 36
subscription_download_probability: 1.0
```
Extends the subscription.yaml syntax at the cost of introducing a breaking change.
# New Syntax
## Mix-n-match presets and indent variables
```
# Can mix/match presets and indent override variables.
# Uses presets TV Show, Only Recent and assigns Kids, TV-Y to subscription_indent_1 and _2
TV Show | = Kids | = TV-Y | Only Recent:
"Jake Trains": "https://..."
```
## Subscriptions with list-value support
```
TV Show | = Kids | = TV-Y | Only Recent:
"Jake Trains":
- "https://url.1..." # Assigns to subscription_value and subscription_value_1
- "https://url.2..." # Assigns to subscription_value_2
```
## Subscriptions with override-keys support
```
TV Show | = Kids | = TV-Y | Only Recent:
"~Jake Trains": # the ~ means "all keys underneath get assigned as override variables"
url: "https://url.1..." # Assigns to url
url2: "https://url.2..." # Assigns to url2
```
# Breaking Changes
In the TV show subscriptions example (https://github.com/jmbannon/ytdl-sub/blob/master/examples/tv_show_subscriptions.yaml), it had
```
TV Show Full Archive:
# Sets "Kids" for genre, "TV-Y" for content rating
= Kids | TV-Y:
"Jake Trains": "https://www.youtube.com/@JakeTrains"
"Kids Toys Play": "https://www.youtube.com/@KidsToysPlayChannel"
```
This must be changed to
```
TV Show Full Archive:
# Sets "Kids" for genre, "TV-Y" for content rating
= Kids | = TV-Y: # Each indent variable assignment must have an = before it
"Jake Trains": "https://www.youtube.com/@JakeTrains"
"Kids Toys Play": "https://www.youtube.com/@KidsToysPlayChannel"
```
Creates standardized prebuilt presets for music videos. Preset names are:
```
"Kodi Music Videos"
"Jellyfin Music Videos"
"Plex Music Videos"
```
Usage can be found in the `examples/` directory
With the new preset nesting support, `__preset__` would have lower precedence which isn't ideal since it's meant to serve as a 'apply to all presets in this file'. Now the order of priority is:
`inherited presets (top-to-bottom) -> preset itself -> nested subscription presets -> __preset__ -> subscription values`
Adds the following prebuilt presets:
```
"Single"
"SoundCloud Discography"
"YouTube Releases"
"YouTube Full Albums"
"Bandcamp"
```
which require no config.yaml to use. Usage examples can be found in the `/examples` directory
A follow-up to the prior commit which fixes some bugs in the 'beautiful subscriptions' implementation. A proper migration guide will be written in the wiki once it's ready
Closes Feature Request: https://github.com/jmbannon/ytdl-sub/issues/743
Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:
```
tv_show:
only_recent:
[News]:
"Breaking News": "https://www.youtube.com/@SomeBreakingNews"
[Tech]:
"Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
```
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:
```
"Breaking News":
preset:
- "tv_show"
- "only_recent"
overrides:
subscription_indent_1: "News"
subscription_name: "Breaking News"
subscription_value: "https://www.youtube.com/@SomeBreakingNews"
"Two Minute Papers":
preset:
- "tv_show"
overrides:
subscription_indent_1: "Tech"
subscription_name: "Two Minute Papers" subscription_value: "https://www.youtube.com/@TwoMinutePapers"
```
You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things you want configurable like so in your
parent preset:
```
presets:
tv_show_name:
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
genre: "{subscription_indent_1}"
```
NOTE!!!
Changing your subscription name from 'legacy' subscriptions will need their download archive file migrated to a new name. A future update will assist in this migration process. Either wait until then or, if you consider yourself a ytdl-sub expert, keep a backup of your subscription file before migrating.
`Subscription Value` is the mechanism used to achieve one-liner subscriptions. It works by having the subscription `"Subscription Name": "value"`, and assigning `"value"` to an override variable via setting the subscription value to that override variable's name.
This feature allows you to set subscription value within the config. In the near future, ytdl-sub will strive to make subscription files simpler by hiding most of the "under-the-hood" logic within the config file.
Makes setting yt-dlp's `format` field easier by giving it its own documented plugin.
Old:
```
my_format_preset:
ytdl_options:
format: "best"
```
New:
```
my_format_preset:
format: "best"
```
The old method will still work, so do not worry about updating configs ASAP. However, the option is available to save a few lines 😉
