* docs(presets): Stale copied prebuilt comment
* docs(start): How to configure subscriptions
Revise the subscriptions file Getting Started tutorial from the perspective of a new
user. The idea is that the ref docs describe a component as the code sees it, whereas
the how-to/tutorial/getting-started docs provide a narrative description of the
component from the external perspective of a typical new user.
* docs(start): Document subscription override mode
It's debatable whether this belongs here. Having moved onto my own custom presets, the
only use cases for override mode in my subscriptions file are for those series that also
have TVDB metadata. Those ytdl-sub downloads I integrate into my Sonarr library need
per-subscription overrides such as:
- Sonarr manages years 2020-2023 for one series and ytdl-sub fills in
from there, so set a `sonarr_series_after: "20240101"` override
- each series needs a directory named per Sonarr's configuration with the TVDB ID, the subscription key,
but also a series name prefix without the TVDB ID for each episode file, so set a
`sonarr_series_prefix: "Foo Series (2020)"` override
So are there other use cases for override mode that are more common and less niche than
mine?
* docs(docker): Revert example bind mount volumes
Per [PR feedback](https://github.com/jmbannon/ytdl-sub/pull/1310#discussion_r2306067263).
Reconcile the various debugging hints throughout the documentation and integrate into a
central debugging page. Includes clarification of the roles of ytdl-sub, yt-dlp, and
external services. Also adds cross references to the relevant bits across the docs.
* docs(logs): Mimick argparse option default format
I find it more readable to use punctuation to separate technical information, such as
required vs optional or default values, from narrative description. Follow argparse's
lead, and put such information in parens following the narrative description.
* docs(usage): Consistent structure, clarifications
* docs(config): Clarify persist_logs behavior/opts
Clarify the `persist_logs:` options per [Discord discussion](https://discord.com/channels/994270357957648404/1409161361853780060/1409602529460879431).
This is another change that touches the generated docs, so after finishing my writing, I
copied the change to `./docs/source/config_reference/scripting/static_variables.rst`
into the corresponding docstring in
`./src/ytdl_sub/entries/variables/override_variables.py` and formatted it as a
docstring. Then I ran:
$ REGENERATE_DOCS=1 tox exec -e "py" -- pytest tests/unit/docgen/test_docgen.py
But all that did was revert the changes to
`./docs/source/config_reference/scripting/static_variables.rst`. IOW, it did *not*
update the rST from the changed docstring.
* docs(preset): Config file top presets vs base
* docs(throttle): resolution/quality throttle asset
I also added the "wait a few hours and try again" hint, sorry I didn't think of this
when reviewing your PR. While doing that I also reformatted the `%concat(...)` arguments
to be more readable to me. Particularly to keep clear what punctuation is about argument
separation as opposed to punctuation meant to be included in the resulting string. Not a
strong preference if you don't like it.
* docs(various): Minor typo and formatting fixes
* docs(preset): Fix presets vs preset in config file
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289960678).
* test(throttle): Match modified assertion text
If it were my test, I'd only make assertions on the spirit of error message text
necessary to confirm it's the right error message and not another.
* docs(throttle): More likely override value
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289976156).
Closes https://github.com/jmbannon/ytdl-sub/issues/1241
YouTube is upping their throttling by serving 360p videos. Most likely to give us and genAI scraping a bad time on purpose.
Resolution assertion will be enabled by default. It will:
- Require > 360 height (pixels)
- Can configure the width by setting `resolution_assert_height_gte: 1080` to require 1080p, or any desired value
- Can disable resolution assert by setting `enable_resolution_assert: False`
* docs(install): Redundant platform-specific mention
I'm guessing this is a remnant from before it got it's own page.
* docs(docker): Clarify container image install
Capturing my changes thus far while I go off to educate myself about rST/Sphinx refs vs
links. As such, this duplicates a reference link in a way I don't like:
:ref:`Automating Downloads <guides/getting_started/automating_downloads:docker and
unraid>`
If you want to fix that or are OK with the duplication, this should be ready to
go. Otherwise, I'll fix and force push once I've learned how to do this correctly.
A matter of opinion in this change, I did away with the tabbed GUI vs headless code
blocks and the separate CPU/GPU passthrough code blocks because I find comments in
example code to be more clear, more readable, and more approachable. This also has the
benefit of putting informative comments in the resulting user's configuration. For
example, imagine a user that learns only later that they require GPU passthrough, their
configuration is already ready to get them started. This comes at the cost some repeated
comments (`environment:` and `deploy:` for GPU passthrough) and the fancy Sphinx
highlighting of relevant lines. I think the trade-off is a net benefit for users.
* docs(sphinx): Fix extlinks LSIO reference usage
* docs(sphinx): Address inline literal warning
Addresses:
./docs/source/config_reference/plugins.rst:848: WARNING: Inline literal start-string
without end-string. [docutils]
* docs(start): Clarify prerequisite tech knowledge
Thanks @rpatterson for the fix!
---------
Following [discussion about the previous preset
delay](https://discord.com/channels/994270357957648404/994270357957648408/1405705039649046589),
we weren't able to find a reason for it and there was agreement that the example value
from yt-dlp is more sensible.
While we're at it, also set a default `min:` value that would probably be sensible if it
were to be used in the future.
Finally, capture the reasoning behind the new preset values in comments to make the
reference documentation more helpful to new users in the future.
Adds the prebuilt preset ``Filter Duration``, which can include/exclude media based on its duration.
Supports the following override variables:
* ``filter_duration_min_s``
* ``filter_duration_max_s``
Usage:
```
Plex TV Show by Date | Filter Duration:
= Documentaries:
"~NOVA PBS":
url: "https://www.youtube.com/@novapbs"
filter_duration_min_s: 120 # Only download videos at least 2m long
= Sports:
"~Maple Leafs Highlights":
url: "https://www.youtube.com/@NHL"
filter_duration_max_s: 180 # Only get highlight videos less than 3m long
```
To protect new users, all prebuilt preset (excluding Soundcloud and Bandcamp) will include throttle protection by default.
> How do I disable?
Set the override variable `enable_throttle_protection: False`.
This can be done on a per-subscription basis and/or in the top __preset__ section to apply to all presets:
```
__preset__:
overrides:
enable_throttle_protection: False
```
> What if I already use throttle protection?
Your settings will override whatever values are set in the defaults.
Closes:
- https://github.com/jmbannon/ytdl-sub/issues/1121
- https://github.com/jmbannon/ytdl-sub/issues/1186
Adds the ability to make thumbnails square, both file and embedded. Usage:
```
my_preset:
square_thumbnail: True
```
Will soon enable this by default for all music-based presets with a toggle to disable.
Huge thanks to @Kentaro1043 for crafting the ffmpeg command for this!
Closes https://github.com/jmbannon/ytdl-sub/issues/383
Gives Emby first class support by creating prebuilt presets for each variant, including
- `Emby TV Show by Date`
- `Emby TV Show Collection`
- `Emby Music Videos`
Emby is identical to Jellyfin except in TV Show Collection, where it requires a `season.nfo`. A new plugin `StaticNfoTags` has been created to create those.
Huge thanks to @Kamaroth92 for the initial code for this feature!
Adds the ability to toggle whether to use `upload_date` or `release_date` in the date_range plugin, which is used to cull older videos.
```
date_range:
type: "upload_date"
```
If using the Only Recent preset, it will be automatically applied based on whether your season/episode ordering uses release date or upload date.
Closes https://github.com/jmbannon/ytdl-sub/issues/1182
Add the ability to print custom messages in override scripts. Adds the functions
- `print`
- `print_if_true`
- `print_if_false`
Will be used to make under-the-hood preset things more verbose.
Implements https://github.com/jmbannon/ytdl-sub/issues/1182
Adds the ability to change the `keep_files_date_eval` (traditionally upload_date_standardized) to any date, so long as its in the form of YYYY-MM-DD.
Will be able to use this to have episodes get deleted based on their release_date or epoch_date, instead of their upload_date.
Not all terminals/logs displayed colors correctly. Adds the ability to suppress any color via `--suppress-colors` cli arg.
Thanks @drewski3420 for the contribution 🎉
Adds ``Filter Keywords``, a preset that can include or exclude media with any of the listed keywords. Both keywords and title/description are lower-cased before filtering.
Supports the following override variables:
* ``title_include_keywords``
* ``title_exclude_keywords``
* ``description_include_keywords``
* ``description_exclude_keywords``
For best usage, use the `~` tilda subscription mode to set a subscription's list override variables.
Tilda mode allows override variables to be set directly underneath it.
```
Plex TV Show by Date | Filter Keywords:
= Documentaries:
"~NOVA PBS":
url: "https://www.youtube.com/@novapbs"
title_exclude_keywords:
- "preview"
- "trailer"
"~To Catch a Smuggler":
url: "https://www.youtube.com/@NatGeo"
title_include_keywords:
- "To Catch a Smuggler"
```
Regex plugin has been removed in favor of scripting. The function [regex_capture_many](https://ytdl-sub.readthedocs.io/en/latest/config_reference/scripting/scripting_functions.html#regex-capture-many) has been created to replicate the plugin's behavior. See the following converted example:
`regex, now deprecated`
```
regex:
from:
title:
match:
- ".*? - (.*)" # Captures 'Song' from 'Emily Hopkins - Some - Song'
capture_group_names:
- "captured_track_title"
capture_group_defaults:
- "{title}"
overrides:
track_title: "{captured_track_title}"
```
`scripting equivalent`
```
overrides:
# Captures 'Song' from 'Emily Hopkins - Some - Song'
captured_track_title: >-
{
%regex_capture_many(
title,
[ ".*? - (.*)" ],
[ title ]
)
}
track_title: "{%array_at(captured_track_title, 1)}"
```
## Motivation:
Regex was a unique plugin that added custom variables based on user input. This made the backend need to support both `overrides` and 'plugin user variables'. Removing `regex` plugin will consolidate this logic into only `overrides`, making it much easier to maintain.
Adds the function `%contains_any`, and list support in a tilda override subscription. The end-goal of these features are to more easily add a title exclude list, like so:
```
__preset__:
filter_exclude:
- "{%contains_any( %lower(title), exclude_title_strings )}"
...
Jellyfin TV Show by Date:
~History Documentaries:
url: "https://..."
exclude_title_strings:
- "trailer"
- "preview"
```
A proper prebuilt preset or built-in functionality will follow this change.
Enhances the music video presets by supporting the following syntax:
`subscriptions.yaml`
```
__preset__:
overrides:
music_video_directory: "/music_videos"
# Choose between Jellyfin/Kodi/Plex Music Videos preset:
# - Plex Music Videos:
# - Jellyfin Music Videos:
# - Kodi Music Videos:
#
"Plex Music Videos":
= Pop: # Sets genre tag to "Pop"
"Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
"Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E"
= Rock:
# Prefixing with '+' puts the subscription into 'map-mode'.
# Music video presets in map-mode support grouping videos into different
# categories, which get set on the album field.
#
# URLs can either be strings, or maps that can overload title, year, date
"+ Guns N' Roses":
Music Videos:
- "https://www.youtube.com/playlist?list=PLOTK54q5K4INNXaHKtmXYr6J7CajWjqeJ"
Concerts:
- title: "Live at The Ritz - New York City"
year: "1988"
url: "https://www.youtube.com/watch?v=OldpIhHPsbs"
- title: "Live at The Hollywood Bowl"
date: "2023-01-11"
url: "https://www.youtube.com/watch?v=Z7hutGlvq9I"
```
The 'map-mode' (denoted by `+`) lets you specify a map of music video categories. Under each category, a URL can be specified as-is or with additional metadata (title, year/date, url).
Music videos and concerts, especially older ones, are typically uploaded by random users with inconsistencies in their titles. This syntax aims to make it easy to specify individual URLs with the ability to overwrite their title and year, and group them by category.
Adds the ability to create map and list-based override variables.
For example, you can now create lists like this:
```
overrides:
urls:
- "https://...1"
- "https://...2"
```
which is equivalent to:
```
overrides:
urls: >-
{
[
"https://...1",
"https://...2",
]
}
```
Likewise, maps can now look like:
```
overrides:
music_video_category:
concerts:
- "https://...1"
- "https://...2"
interviews:
- "https://...3"
```
which is equivalent to:
```
overrides:
music_video_category: >-
{
"concerts": [
"https://...1",
"https://...2"
],
"interviews": [
"https://...3"
]
}
```
Attempts to make writes to the download archive safer (https://github.com/jmbannon/ytdl-sub/issues/982).
`ytdl-sub` will now *copy* the download archive from the working directory to the output directory with a temp name, then perform a *move* to store it with its final expected name. This will drastically lower the window of time where the process could die mid-write and corrupt it on the next read.
This implements %regex_sub built-in function to enhance string-processing capabilities, allowing users to perform substitutes like:
- removing non-ascii characters
- replacing subsequent whitespace characters with a single whitespace
Thanks @tbabej !
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
Fixes https://github.com/jmbannon/ytdl-sub/issues/912
If a required variable (like uid) is missing, ytdl-sub will error. This bugfix prevents sites that do not provide an upload date from erroring.
A recent feature to grab channel artwork for playlists caused downloading channels to misreport the number of entries it was downloading. This change fixes that
Adds a the new field `breaking` to the `date_range` plugin, to toggle whether an entry breaks subsequent metadata pulls. This is useful to disable if you are grabbing a playlist that may have videos out of order, but still want to apply a date range to it.
With the right functions, it was possible for subscription validation to raise a false-positive error. This should hopefully resolve that issue and give validation a small performance increase
Makes it possible to fix https://github.com/jmbannon/ytdl-sub/issues/756
In the case that uploader_url does not equal the input url, keep trying to fetch uploader_url to get all possible metadata that yt-dlp does not provide in a single API call
With both `--match` and `--dl-override`, it is now possible to more easily experiment with subscriptions without needing to create a separate file.
For example, suppose you are changing some values in your subscription "Rick A" and want to test them out. You can now run:
```
ytdl-sub sub --dry-run --match Rick --dl-override '--ytdl-options.max_downloads 3'
```
This will
1. Dry-run
2. Only run for the subscription "Rick A"
3. Apply setting max_downloads to 3
Reduce memory usage significantly when pulling large channels all at once by
- Not loading entry script until it is being processed
- Deleting the entry memory once fully processed
- Disable yt-dlp from keeping entry info in memory since ytdl-sub reads it from file
