Allows for the `download` plugin to take in a script-based array of URLs. Prebuilt presets are now greatly reduced in size by constructing urls as an array versus 100+ separate URL variables.
Eventually, we can completely remove the limit of the number of URLs in a subscription.
Disallow override variable names conflicting with plugin names.
May cause backward incompatible issues from `date_range` being an old override variable name for the `Only Recent` preset. Update this variable name to `only_recent_date_range` to resolve. More info in https://ytdl-sub.readthedocs.io/en/latest/deprecation_notices.html
Closes https://github.com/jmbannon/ytdl-sub/issues/1313
Adds `resolution_assert_ignore_titles` variable to be able to ignore specific videos in a subscription where 360p is suspected to be the only option.
Ideally, it would be nice if ytdl-sub could determine whether 360p is actually the only resolution or not automatically, and download if it's true. We need to do more research to determine if this is possible.
In the meantime, the above variable should help manually get around this. Usage:
```
# use tilda mode to set override variables to the subscription
"~My Subscription":
url: "https://youtube.com/@channel"
resolution_assert_ignore_titles:
- "This 360p Video Title"
```
```
Resolution assert is enabled, will fail on low-quality video downloads and presume throttle. Disable using the override variable `enable_resolution_assert: False
```
Spams during the initialization phase. Ideally, we should only print this once the subscription has begun downloading. Perhaps via a `print` plugin? Either way, set this to debug log level.
* docs(presets): Stale copied prebuilt comment
* docs(start): How to configure subscriptions
Revise the subscriptions file Getting Started tutorial from the perspective of a new
user. The idea is that the ref docs describe a component as the code sees it, whereas
the how-to/tutorial/getting-started docs provide a narrative description of the
component from the external perspective of a typical new user.
* docs(start): Document subscription override mode
It's debatable whether this belongs here. Having moved onto my own custom presets, the
only use cases for override mode in my subscriptions file are for those series that also
have TVDB metadata. Those ytdl-sub downloads I integrate into my Sonarr library need
per-subscription overrides such as:
- Sonarr manages years 2020-2023 for one series and ytdl-sub fills in
from there, so set a `sonarr_series_after: "20240101"` override
- each series needs a directory named per Sonarr's configuration with the TVDB ID, the subscription key,
but also a series name prefix without the TVDB ID for each episode file, so set a
`sonarr_series_prefix: "Foo Series (2020)"` override
So are there other use cases for override mode that are more common and less niche than
mine?
* docs(docker): Revert example bind mount volumes
Per [PR feedback](https://github.com/jmbannon/ytdl-sub/pull/1310#discussion_r2306067263).
* docs(preset): Config file top presets vs base
* docs(throttle): resolution/quality throttle asset
I also added the "wait a few hours and try again" hint, sorry I didn't think of this
when reviewing your PR. While doing that I also reformatted the `%concat(...)` arguments
to be more readable to me. Particularly to keep clear what punctuation is about argument
separation as opposed to punctuation meant to be included in the resulting string. Not a
strong preference if you don't like it.
* docs(various): Minor typo and formatting fixes
* docs(preset): Fix presets vs preset in config file
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289960678).
* test(throttle): Match modified assertion text
If it were my test, I'd only make assertions on the spirit of error message text
necessary to confirm it's the right error message and not another.
* docs(throttle): More likely override value
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289976156).
Closes https://github.com/jmbannon/ytdl-sub/issues/1241
YouTube is upping their throttling by serving 360p videos. Most likely to give us and genAI scraping a bad time on purpose.
Resolution assertion will be enabled by default. It will:
- Require > 360 height (pixels)
- Can configure the width by setting `resolution_assert_height_gte: 1080` to require 1080p, or any desired value
- Can disable resolution assert by setting `enable_resolution_assert: False`
Thanks @rpatterson for the fix!
---------
Following [discussion about the previous preset
delay](https://discord.com/channels/994270357957648404/994270357957648408/1405705039649046589),
we weren't able to find a reason for it and there was agreement that the example value
from yt-dlp is more sensible.
While we're at it, also set a default `min:` value that would probably be sensible if it
were to be used in the future.
Finally, capture the reasoning behind the new preset values in comments to make the
reference documentation more helpful to new users in the future.
Adds the prebuilt preset ``Filter Duration``, which can include/exclude media based on its duration.
Supports the following override variables:
* ``filter_duration_min_s``
* ``filter_duration_max_s``
Usage:
```
Plex TV Show by Date | Filter Duration:
= Documentaries:
"~NOVA PBS":
url: "https://www.youtube.com/@novapbs"
filter_duration_min_s: 120 # Only download videos at least 2m long
= Sports:
"~Maple Leafs Highlights":
url: "https://www.youtube.com/@NHL"
filter_duration_max_s: 180 # Only get highlight videos less than 3m long
```
To protect new users, all prebuilt preset (excluding Soundcloud and Bandcamp) will include throttle protection by default.
> How do I disable?
Set the override variable `enable_throttle_protection: False`.
This can be done on a per-subscription basis and/or in the top __preset__ section to apply to all presets:
```
__preset__:
overrides:
enable_throttle_protection: False
```
> What if I already use throttle protection?
Your settings will override whatever values are set in the defaults.
Closes:
- https://github.com/jmbannon/ytdl-sub/issues/1121
- https://github.com/jmbannon/ytdl-sub/issues/1186
Gives Emby first class support by creating prebuilt presets for each variant, including
- `Emby TV Show by Date`
- `Emby TV Show Collection`
- `Emby Music Videos`
Emby is identical to Jellyfin except in TV Show Collection, where it requires a `season.nfo`. A new plugin `StaticNfoTags` has been created to create those.
Huge thanks to @Kamaroth92 for the initial code for this feature!
Adds the ability to toggle whether to use `upload_date` or `release_date` in the date_range plugin, which is used to cull older videos.
```
date_range:
type: "upload_date"
```
If using the Only Recent preset, it will be automatically applied based on whether your season/episode ordering uses release date or upload date.
Closes https://github.com/jmbannon/ytdl-sub/issues/1182
Adds ``Filter Keywords``, a preset that can include or exclude media with any of the listed keywords. Both keywords and title/description are lower-cased before filtering.
Supports the following override variables:
* ``title_include_keywords``
* ``title_exclude_keywords``
* ``description_include_keywords``
* ``description_exclude_keywords``
For best usage, use the `~` tilda subscription mode to set a subscription's list override variables.
Tilda mode allows override variables to be set directly underneath it.
```
Plex TV Show by Date | Filter Keywords:
= Documentaries:
"~NOVA PBS":
url: "https://www.youtube.com/@novapbs"
title_exclude_keywords:
- "preview"
- "trailer"
"~To Catch a Smuggler":
url: "https://www.youtube.com/@NatGeo"
title_include_keywords:
- "To Catch a Smuggler"
```
Regex plugin has been removed in favor of scripting. The function [regex_capture_many](https://ytdl-sub.readthedocs.io/en/latest/config_reference/scripting/scripting_functions.html#regex-capture-many) has been created to replicate the plugin's behavior. See the following converted example:
`regex, now deprecated`
```
regex:
from:
title:
match:
- ".*? - (.*)" # Captures 'Song' from 'Emily Hopkins - Some - Song'
capture_group_names:
- "captured_track_title"
capture_group_defaults:
- "{title}"
overrides:
track_title: "{captured_track_title}"
```
`scripting equivalent`
```
overrides:
# Captures 'Song' from 'Emily Hopkins - Some - Song'
captured_track_title: >-
{
%regex_capture_many(
title,
[ ".*? - (.*)" ],
[ title ]
)
}
track_title: "{%array_at(captured_track_title, 1)}"
```
## Motivation:
Regex was a unique plugin that added custom variables based on user input. This made the backend need to support both `overrides` and 'plugin user variables'. Removing `regex` plugin will consolidate this logic into only `overrides`, making it much easier to maintain.
Enhances the music video presets by supporting the following syntax:
`subscriptions.yaml`
```
__preset__:
overrides:
music_video_directory: "/music_videos"
# Choose between Jellyfin/Kodi/Plex Music Videos preset:
# - Plex Music Videos:
# - Jellyfin Music Videos:
# - Kodi Music Videos:
#
"Plex Music Videos":
= Pop: # Sets genre tag to "Pop"
"Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
"Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E"
= Rock:
# Prefixing with '+' puts the subscription into 'map-mode'.
# Music video presets in map-mode support grouping videos into different
# categories, which get set on the album field.
#
# URLs can either be strings, or maps that can overload title, year, date
"+ Guns N' Roses":
Music Videos:
- "https://www.youtube.com/playlist?list=PLOTK54q5K4INNXaHKtmXYr6J7CajWjqeJ"
Concerts:
- title: "Live at The Ritz - New York City"
year: "1988"
url: "https://www.youtube.com/watch?v=OldpIhHPsbs"
- title: "Live at The Hollywood Bowl"
date: "2023-01-11"
url: "https://www.youtube.com/watch?v=Z7hutGlvq9I"
```
The 'map-mode' (denoted by `+`) lets you specify a map of music video categories. Under each category, a URL can be specified as-is or with additional metadata (title, year/date, url).
Music videos and concerts, especially older ones, are typically uploaded by random users with inconsistencies in their titles. This syntax aims to make it easy to specify individual URLs with the ability to overwrite their title and year, and group them by category.
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
Adds a the new field `breaking` to the `date_range` plugin, to toggle whether an entry breaks subsequent metadata pulls. This is useful to disable if you are grabbing a playlist that may have videos out of order, but still want to apply a date range to it.
Partial fix to https://github.com/jmbannon/ytdl-sub/issues/853
Makes computing sibling metadata optional, defaulted to False. This prevents excessive memory usage when scraping large channels
Adds a new mechanism to limit subscriptions' number of videos to an explicit number (only supported date ranges prior to this).
Example usage:
```
output_options:
keep_max_files: 10
```
will only keep 10 videos at max.
In addition:
- `keep_max_files` is bundled into the `Only Recent` preset and can be used via setting the `only_recent_max_files` override variable (see examples or README for usage).
- Changed the `date_range` variable name to `only_recent_date_range` for more clarity. Usage of `date_range` will continue to work.
Creates standardized prebuilt presets for music videos. Preset names are:
```
"Kodi Music Videos"
"Jellyfin Music Videos"
"Plex Music Videos"
```
Usage can be found in the `examples/` directory
For music presets, adds `track_genre_default` as an override variable (defaults to Unset)
For TV show presets, adds `tv_show_genre_default` and `tv_show_content_rating_default`, which default to 'ytdl-sub' and 'TV-14'
New music prebuilt presets had set `music_directory`, which is bad practice since it's not visible to the user which directory it would use. This forces `music_directory` to be set either in the user's config or subscription file
Adds the following prebuilt presets:
```
"Single"
"SoundCloud Discography"
"YouTube Releases"
"YouTube Full Albums"
"Bandcamp"
```
which require no config.yaml to use. Usage examples can be found in the `/examples` directory
A follow-up to the prior commit which fixes some bugs in the 'beautiful subscriptions' implementation. A proper migration guide will be written in the wiki once it's ready
Adding a default match-filter broke other people's match-filters. This is because each match-filter acts as an OR. I.e.
```
match_filters:
filters:
- "!is_live & !is_upcoming & !post_live" # the one I added
- "original_url!*=/shorts/ & !is_live"
```
Would not filter shorts, because under the hood, yt-dlp evaluates it as `(!is_live & !is_upcoming & !post_live) | (original_url!*=/shorts/ & !is_live)`
Need to rethink how to filter live videos by default 🤔
