- Update Ubuntu-based images to use Python 3.12
- Move hidden files from GUI image's main dir
- Update github workflows to use python 3.12
- Update fixtures
- Minor doc fixes
Regex plugin has been removed in favor of scripting. The function [regex_capture_many](https://ytdl-sub.readthedocs.io/en/latest/config_reference/scripting/scripting_functions.html#regex-capture-many) has been created to replicate the plugin's behavior. See the following converted example:
`regex, now deprecated`
```
regex:
from:
title:
match:
- ".*? - (.*)" # Captures 'Song' from 'Emily Hopkins - Some - Song'
capture_group_names:
- "captured_track_title"
capture_group_defaults:
- "{title}"
overrides:
track_title: "{captured_track_title}"
```
`scripting equivalent`
```
overrides:
# Captures 'Song' from 'Emily Hopkins - Some - Song'
captured_track_title: >-
{
%regex_capture_many(
title,
[ ".*? - (.*)" ],
[ title ]
)
}
track_title: "{%array_at(captured_track_title, 1)}"
```
## Motivation:
Regex was a unique plugin that added custom variables based on user input. This made the backend need to support both `overrides` and 'plugin user variables'. Removing `regex` plugin will consolidate this logic into only `overrides`, making it much easier to maintain.
Adds the ability to create map and list-based override variables.
For example, you can now create lists like this:
```
overrides:
urls:
- "https://...1"
- "https://...2"
```
which is equivalent to:
```
overrides:
urls: >-
{
[
"https://...1",
"https://...2",
]
}
```
Likewise, maps can now look like:
```
overrides:
music_video_category:
concerts:
- "https://...1"
- "https://...2"
interviews:
- "https://...3"
```
which is equivalent to:
```
overrides:
music_video_category: >-
{
"concerts": [
"https://...1",
"https://...2"
],
"interviews": [
"https://...3"
]
}
```
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
- Check to see if a URL is a YouTube playlist URL, if so...
- Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
- Otherwise...
- Set the field to an empty string, which means ytdl-sub will skip it
Adds a the new field `breaking` to the `date_range` plugin, to toggle whether an entry breaks subsequent metadata pulls. This is useful to disable if you are grabbing a playlist that may have videos out of order, but still want to apply a date range to it.
With both `--match` and `--dl-override`, it is now possible to more easily experiment with subscriptions without needing to create a separate file.
For example, suppose you are changing some values in your subscription "Rick A" and want to test them out. You can now run:
```
ytdl-sub sub --dry-run --match Rick --dl-override '--ytdl-options.max_downloads 3'
```
This will
1. Dry-run
2. Only run for the subscription "Rick A"
3. Apply setting max_downloads to 3
Adds generic `filter_include` and `filter_exclude` plugins in anticipation for function script usage. Will detail it more with official docs at a later time!
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.
With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.
Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
For less popular sites, ytdl-sub's custom download archive was not working. This was because we weren't using the right extraction key (though it still worked for YouTube, Bandcamp, Soundcloud, and others). This bugfix should now exactly match yt-dlp's and never be a problem again.
Huge thanks to @Svagtlys for root-causing this 🤝
Adds a new mechanism to limit subscriptions' number of videos to an explicit number (only supported date ranges prior to this).
Example usage:
```
output_options:
keep_max_files: 10
```
will only keep 10 videos at max.
In addition:
- `keep_max_files` is bundled into the `Only Recent` preset and can be used via setting the `only_recent_max_files` override variable (see examples or README for usage).
- Changed the `date_range` variable name to `only_recent_date_range` for more clarity. Usage of `date_range` will continue to work.
Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For range-based values, a random number will be chosen within the range to avoid sleeps looking scripted.
Usage:
```
throttle_protection:
sleep_per_download_s:
min: 2.2
max: 10.8
sleep_per_subscription_s:
min: 9.0
max: 14.1
max_downloads_per_subscription:
min: 10
max: 36
subscription_download_probability: 1.0
```
Creates standardized prebuilt presets for music videos. Preset names are:
```
"Kodi Music Videos"
"Jellyfin Music Videos"
"Plex Music Videos"
```
Usage can be found in the `examples/` directory
Music multi-tags (artists, albumartists, etc) were not writing correctly. Fix it by explicitly checking to see if the tag is multi or not in the codebase.
Adds the following prebuilt presets:
```
"Single"
"SoundCloud Discography"
"YouTube Releases"
"YouTube Full Albums"
"Bandcamp"
```
which require no config.yaml to use. Usage examples can be found in the `/examples` directory
Closes GH Issue #766, partially closes#520
Prior to this release, no other subscriptions would download if a subscription before it had an error. Now, subscriptions will continue to download even if one has an error, and will be reported in the output summary.
The only required field in a config was `working_directory`. This is now optional (defaults to `.ytdl-sub-working-directory`) making it so a config is no longer required. This sets the stage to onboard noob users who will solely use prebuilt presets in a single subscription file
A follow-up to the prior commit which fixes some bugs in the 'beautiful subscriptions' implementation. A proper migration guide will be written in the wiki once it's ready
With the recent push to 'beautify subscriptions' (see https://github.com/jmbannon/ytdl-sub/releases/tag/2023.10.02), we need the ability to change subscription names from their legacy form:
```
rick_a:
preset:
- "tv_show"
overrides:
tv_show_name: "Rick A"
url: "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
into:
```
tv_show:
"Rick A": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
This however has implications from ytdl-sub's legacy download archive naming. By default, we write archives to `.ytdl-sub-{subscription_name}-download-archive.json`. If we change the subscription name, the archive will not be found, causing a complete redownload. This new feature gives us the ability to migrate download archives to a new naming schema.
# Migrating to Beautified Subscriptions
## Step 0
BACK UP ALL CONFIG + SUBSCRIPTION FILES!!!!! If something goes wrong, restore your backup and try again and/or ask for help.
## Step 1
Since we know we'll be changing our `subscripion_name` to the value of `tv_show_name`, we can use that in our newly migrated download archive name by setting this within the `tv_show` preset (or whatever your 'base' preset is).
```
presets:
tv_show:
output_options:
migrated_download_archive_name: ".{tv_show_name_sanitized}-download-archive.json"
```
## Step 2
Perform a download as usual, via `ytdl-sub sub ...`. This will load the old archive, and save it into the new archive. You should see `MIGRATION DETECTED` within the logs. Ensure it completes successfully.
Perform another download invocation and ensure you see the `MIGRATION SUCCESSFUL` within the logs.
## Step 3
Now we can set:
```
presets:
tv_show:
output_options:
# rename migrated_download_archive_name to just download_archive_name
download_archive_name: ".{tv_show_name_sanitized}-download-archive.json"
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
```
Our download archives now default to our new format, and we set `tv_show_name` + `url` to use subscription values by default.
## Step 4
We can now beautify our subscription.yaml file to:
```
tv_show:
"Rick A": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
```
Closes Issue: https://github.com/jmbannon/ytdl-sub/issues/729
Add a new field to the match_filter plugin: `download_match_filters`. This set of filters will be applied at the download stage (as opposed to `filters` which get applied at the metadata stage). By default, always set `"!is_live & !is_upcoming & !post_live"` as a download filter. This prevents ytdl-sub from hanging when trying to grab live and upcoming videos.
Docs on match-filters: https://ytdl-sub.readthedocs.io/en/latest/config.html#match-filters
Makes setting yt-dlp's `format` field easier by giving it its own documented plugin.
Old:
```
my_format_preset:
ytdl_options:
format: "best"
```
New:
```
my_format_preset:
format: "best"
```
The old method will still work, so do not worry about updating configs ASAP. However, the option is available to save a few lines 😉
* [BUGFIX] Do not fail with thumbnail issues
* helper
* refactor
* dry-run
* regen fixture
* download arhcive txt -> json in docs, fixtures
* return if no download thumb path
* better test
* more test coverage
