Completely rewrite how subscription validation is performed. Optimizes it quite a bit while also adding backend support for the upcoming `dissect` sub-command, where you can resolve any subscription into its 'raw' form for easier debugging when making scripting changes.
Closes https://github.com/jmbannon/ytdl-sub/issues/386
Adds the ability to preserve mtime via Output Options. Usage:
```
output_options:
preserve_mtime: True
```
Thanks @e1ven for the contribution!
Adds the ability to toggle whether to use `upload_date` or `release_date` in the date_range plugin, which is used to cull older videos.
```
date_range:
type: "upload_date"
```
If using the Only Recent preset, it will be automatically applied based on whether your season/episode ordering uses release date or upload date.
Closes https://github.com/jmbannon/ytdl-sub/issues/1182
Implements https://github.com/jmbannon/ytdl-sub/issues/1182
Adds the ability to change the `keep_files_date_eval` (traditionally upload_date_standardized) to any date, so long as its in the form of YYYY-MM-DD.
Will be able to use this to have episodes get deleted based on their release_date or epoch_date, instead of their upload_date.
Adds the function `%contains_any`, and list support in a tilda override subscription. The end-goal of these features are to more easily add a title exclude list, like so:
```
__preset__:
filter_exclude:
- "{%contains_any( %lower(title), exclude_title_strings )}"
...
Jellyfin TV Show by Date:
~History Documentaries:
url: "https://..."
exclude_title_strings:
- "trailer"
- "preview"
```
A proper prebuilt preset or built-in functionality will follow this change.
Enhances the music video presets by supporting the following syntax:
`subscriptions.yaml`
```
__preset__:
overrides:
music_video_directory: "/music_videos"
# Choose between Jellyfin/Kodi/Plex Music Videos preset:
# - Plex Music Videos:
# - Jellyfin Music Videos:
# - Kodi Music Videos:
#
"Plex Music Videos":
= Pop: # Sets genre tag to "Pop"
"Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
"Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E"
= Rock:
# Prefixing with '+' puts the subscription into 'map-mode'.
# Music video presets in map-mode support grouping videos into different
# categories, which get set on the album field.
#
# URLs can either be strings, or maps that can overload title, year, date
"+ Guns N' Roses":
Music Videos:
- "https://www.youtube.com/playlist?list=PLOTK54q5K4INNXaHKtmXYr6J7CajWjqeJ"
Concerts:
- title: "Live at The Ritz - New York City"
year: "1988"
url: "https://www.youtube.com/watch?v=OldpIhHPsbs"
- title: "Live at The Hollywood Bowl"
date: "2023-01-11"
url: "https://www.youtube.com/watch?v=Z7hutGlvq9I"
```
The 'map-mode' (denoted by `+`) lets you specify a map of music video categories. Under each category, a URL can be specified as-is or with additional metadata (title, year/date, url).
Music videos and concerts, especially older ones, are typically uploaded by random users with inconsistencies in their titles. This syntax aims to make it easy to specify individual URLs with the ability to overwrite their title and year, and group them by category.
Attempts to make writes to the download archive safer (https://github.com/jmbannon/ytdl-sub/issues/982).
`ytdl-sub` will now *copy* the download archive from the working directory to the output directory with a temp name, then perform a *move* to store it with its final expected name. This will drastically lower the window of time where the process could die mid-write and corrupt it on the next read.
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
Adds a the new field `breaking` to the `date_range` plugin, to toggle whether an entry breaks subsequent metadata pulls. This is useful to disable if you are grabbing a playlist that may have videos out of order, but still want to apply a date range to it.
With both `--match` and `--dl-override`, it is now possible to more easily experiment with subscriptions without needing to create a separate file.
For example, suppose you are changing some values in your subscription "Rick A" and want to test them out. You can now run:
```
ytdl-sub sub --dry-run --match Rick --dl-override '--ytdl-options.max_downloads 3'
```
This will
1. Dry-run
2. Only run for the subscription "Rick A"
3. Apply setting max_downloads to 3
Reduce memory usage significantly when pulling large channels all at once by
- Not loading entry script until it is being processed
- Deleting the entry memory once fully processed
- Disable yt-dlp from keeping entry info in memory since ytdl-sub reads it from file
Prior PR did not actually work - this actually fixes the bug (https://github.com/jmbannon/ytdl-sub/issues/844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name Beast.
POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
To be used later for advanced presets. For subscriptions in the form of (+ as prefix)
```
+ Subscription Name:
Music Videos:
- "https://url1.com/..."
Concerts:
- "https://url2.com/..."
```
Stores all the contents under the subscription name into the override variable
``subscription_map`` as a Map value. The above example is stored as:
```
{
"Music Videos": [
"https://url1.com/..."
],
"Concerts: [
"https://url2.com/..."
]
}
```
Advanced scripting is needed to dissect this Map variable into a usable preset
Fixes the bug (#844) where subscription names would drop periods from them, i.e. Mr. Beast would have the name Beast.
POTENTIAL BREAKING CHANGE
If you have a subscription with a period in its name prior to this fix, this change will make it so your download archive is not found since it uses the subscription name in the download archive file path. To fix, simply change the download archive JSON's name to have the actual subscription name, and not the dropped-period one.
Removes the deprecated `__value__` field in the subscriptions file (different from `__preset__` which is NOT deprecated). This was very short-lived, and most likely not used.
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
Adds a new mechanism to limit subscriptions' number of videos to an explicit number (only supported date ranges prior to this).
Example usage:
```
output_options:
keep_max_files: 10
```
will only keep 10 videos at max.
In addition:
- `keep_max_files` is bundled into the `Only Recent` preset and can be used via setting the `only_recent_max_files` override variable (see examples or README for usage).
- Changed the `date_range` variable name to `only_recent_date_range` for more clarity. Usage of `date_range` will continue to work.
Extends the subscription.yaml syntax at the cost of introducing a breaking change.
# New Syntax
## Mix-n-match presets and indent variables
```
# Can mix/match presets and indent override variables.
# Uses presets TV Show, Only Recent and assigns Kids, TV-Y to subscription_indent_1 and _2
TV Show | = Kids | = TV-Y | Only Recent:
"Jake Trains": "https://..."
```
## Subscriptions with list-value support
```
TV Show | = Kids | = TV-Y | Only Recent:
"Jake Trains":
- "https://url.1..." # Assigns to subscription_value and subscription_value_1
- "https://url.2..." # Assigns to subscription_value_2
```
## Subscriptions with override-keys support
```
TV Show | = Kids | = TV-Y | Only Recent:
"~Jake Trains": # the ~ means "all keys underneath get assigned as override variables"
url: "https://url.1..." # Assigns to url
url2: "https://url.2..." # Assigns to url2
```
# Breaking Changes
In the TV show subscriptions example (https://github.com/jmbannon/ytdl-sub/blob/master/examples/tv_show_subscriptions.yaml), it had
```
TV Show Full Archive:
# Sets "Kids" for genre, "TV-Y" for content rating
= Kids | TV-Y:
"Jake Trains": "https://www.youtube.com/@JakeTrains"
"Kids Toys Play": "https://www.youtube.com/@KidsToysPlayChannel"
```
This must be changed to
```
TV Show Full Archive:
# Sets "Kids" for genre, "TV-Y" for content rating
= Kids | = TV-Y: # Each indent variable assignment must have an = before it
"Jake Trains": "https://www.youtube.com/@JakeTrains"
"Kids Toys Play": "https://www.youtube.com/@KidsToysPlayChannel"
```
With the new preset nesting support, `__preset__` would have lower precedence which isn't ideal since it's meant to serve as a 'apply to all presets in this file'. Now the order of priority is:
`inherited presets (top-to-bottom) -> preset itself -> nested subscription presets -> __preset__ -> subscription values`
Closes GH Issue #766, partially closes#520
Prior to this release, no other subscriptions would download if a subscription before it had an error. Now, subscriptions will continue to download even if one has an error, and will be reported in the output summary.
A follow-up to the prior commit which fixes some bugs in the 'beautiful subscriptions' implementation. A proper migration guide will be written in the wiki once it's ready
With the recent push to 'beautify subscriptions' (see https://github.com/jmbannon/ytdl-sub/releases/tag/2023.10.02), we need the ability to change subscription names from their legacy form:
```
rick_a:
preset:
- "tv_show"
overrides:
tv_show_name: "Rick A"
url: "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
into:
```
tv_show:
"Rick A": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
This however has implications from ytdl-sub's legacy download archive naming. By default, we write archives to `.ytdl-sub-{subscription_name}-download-archive.json`. If we change the subscription name, the archive will not be found, causing a complete redownload. This new feature gives us the ability to migrate download archives to a new naming schema.
# Migrating to Beautified Subscriptions
## Step 0
BACK UP ALL CONFIG + SUBSCRIPTION FILES!!!!! If something goes wrong, restore your backup and try again and/or ask for help.
## Step 1
Since we know we'll be changing our `subscripion_name` to the value of `tv_show_name`, we can use that in our newly migrated download archive name by setting this within the `tv_show` preset (or whatever your 'base' preset is).
```
presets:
tv_show:
output_options:
migrated_download_archive_name: ".{tv_show_name_sanitized}-download-archive.json"
```
## Step 2
Perform a download as usual, via `ytdl-sub sub ...`. This will load the old archive, and save it into the new archive. You should see `MIGRATION DETECTED` within the logs. Ensure it completes successfully.
Perform another download invocation and ensure you see the `MIGRATION SUCCESSFUL` within the logs.
## Step 3
Now we can set:
```
presets:
tv_show:
output_options:
# rename migrated_download_archive_name to just download_archive_name
download_archive_name: ".{tv_show_name_sanitized}-download-archive.json"
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
```
Our download archives now default to our new format, and we set `tv_show_name` + `url` to use subscription values by default.
## Step 4
We can now beautify our subscription.yaml file to:
```
tv_show:
"Rick A": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
Fixes yet another bug that is match-filter related. The `date_range.before` value would get ignored if you specified any other match-filters. That is fixed by making them `&` behind-the-scenes.
Closes Feature Request: https://github.com/jmbannon/ytdl-sub/issues/743
Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:
```
tv_show:
only_recent:
[News]:
"Breaking News": "https://www.youtube.com/@SomeBreakingNews"
[Tech]:
"Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
```
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:
```
"Breaking News":
preset:
- "tv_show"
- "only_recent"
overrides:
subscription_indent_1: "News"
subscription_name: "Breaking News"
subscription_value: "https://www.youtube.com/@SomeBreakingNews"
"Two Minute Papers":
preset:
- "tv_show"
overrides:
subscription_indent_1: "Tech"
subscription_name: "Two Minute Papers" subscription_value: "https://www.youtube.com/@TwoMinutePapers"
```
You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things you want configurable like so in your
parent preset:
```
presets:
tv_show_name:
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
genre: "{subscription_indent_1}"
```
NOTE!!!
Changing your subscription name from 'legacy' subscriptions will need their download archive file migrated to a new name. A future update will assist in this migration process. Either wait until then or, if you consider yourself a ytdl-sub expert, keep a backup of your subscription file before migrating.
`Subscription Value` is the mechanism used to achieve one-liner subscriptions. It works by having the subscription `"Subscription Name": "value"`, and assigning `"value"` to an override variable via setting the subscription value to that override variable's name.
This feature allows you to set subscription value within the config. In the near future, ytdl-sub will strive to make subscription files simpler by hiding most of the "under-the-hood" logic within the config file.
Closes Issue: https://github.com/jmbannon/ytdl-sub/issues/729
Add a new field to the match_filter plugin: `download_match_filters`. This set of filters will be applied at the download stage (as opposed to `filters` which get applied at the metadata stage). By default, always set `"!is_live & !is_upcoming & !post_live"` as a download filter. This prevents ytdl-sub from hanging when trying to grab live and upcoming videos.
Docs on match-filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
Makes setting yt-dlp's `format` field easier by giving it its own documented plugin.
Old:
```
my_format_preset:
ytdl_options:
format: "best"
```
New:
```
my_format_preset:
format: "best"
```
The old method will still work, so do not worry about updating configs ASAP. However, the option is available to save a few lines 😉
