Expands variable validation to also include support for partial resolution.
In short, this will partially execute script code until it hits unresolved runtime variables, and store that as the new script representation. This functionality will allow for the upcoming `inspect` command, which will show users their script code in action without having to dry-run.
Makes the underlying formatting process better handle native (non-string) types in a more flexible way. 2nd attempt since the last one got reverted due to breaking changes.
Adds the mechanism to partially resolve an entire script (aka every underlying variable in ytdl-sub). This will be used in the upcoming `inspect` sub-command where it will dump a subscription's raw contents at multiple levels of resolution.
Potentially useful to optimize script execution as well.
Allows for the `download` plugin to take in a script-based array of URLs. Prebuilt presets are now greatly reduced in size by constructing urls as an array versus 100+ separate URL variables.
Eventually, we can completely remove the limit of the number of URLs in a subscription.
There was a bug that did not properly check variable dependency when using lambda functions. It could still work depending on order of definitions. This fix should make it work no matter the order.
Completely rewrite how subscription validation is performed. Optimizes it quite a bit while also adding backend support for the upcoming `dissect` sub-command, where you can resolve any subscription into its 'raw' form for easier debugging when making scripting changes.
Disallow override variable names conflicting with plugin names.
May cause backward incompatible issues from `date_range` being an old override variable name for the `Only Recent` preset. Update this variable name to `only_recent_date_range` to resolve. More info in https://ytdl-sub.readthedocs.io/en/latest/deprecation_notices.html
Closes https://github.com/jmbannon/ytdl-sub/issues/386
Adds the ability to preserve mtime via Output Options. Usage:
```
output_options:
preserve_mtime: True
```
Thanks @e1ven for the contribution!
Closes https://github.com/jmbannon/ytdl-sub/issues/1269
Adds the new paramter `languages_required`, which forces an entry to have this set of subtitle files, otherwise it will error.
In addition, better file checking has been added to ensure they exist before attempting to move.
Closes https://github.com/jmbannon/ytdl-sub/issues/1313
Adds `resolution_assert_ignore_titles` variable to be able to ignore specific videos in a subscription where 360p is suspected to be the only option.
Ideally, it would be nice if ytdl-sub could determine whether 360p is actually the only resolution or not automatically, and download if it's true. We need to do more research to determine if this is possible.
In the meantime, the above variable should help manually get around this. Usage:
```
# use tilda mode to set override variables to the subscription
"~My Subscription":
url: "https://youtube.com/@channel"
resolution_assert_ignore_titles:
- "This 360p Video Title"
```
Changes documentation and default configs to use the `/tv_shows` default path shown in the docker compose setup.
Also changes working directory and file lock directory to not be in /tmp, which often causes permission issues.
```
Resolution assert is enabled, will fail on low-quality video downloads and presume throttle. Disable using the override variable `enable_resolution_assert: False
```
Spams during the initialization phase. Ideally, we should only print this once the subscription has begun downloading. Perhaps via a `print` plugin? Either way, set this to debug log level.
Adds support for setting throttle_protection range values using static override variables. `sleep_per_download_s` has additional support for entry variables since it's used per entry.
We can now experiment with scaling this value based on entry attributes, such as duration. Example:
```
throttle_protection:
sleep_per_download_s:
min: >-
{
%mul(5.5, %pow( duration, 0.4 ))
}
max: >-
{
%mul(6.5, %pow( duration, 0.6 ))
}
```
* docs(presets): Stale copied prebuilt comment
* docs(start): How to configure subscriptions
Revise the subscriptions file Getting Started tutorial from the perspective of a new
user. The idea is that the ref docs describe a component as the code sees it, whereas
the how-to/tutorial/getting-started docs provide a narrative description of the
component from the external perspective of a typical new user.
* docs(start): Document subscription override mode
It's debatable whether this belongs here. Having moved onto my own custom presets, the
only use cases for override mode in my subscriptions file are for those series that also
have TVDB metadata. Those ytdl-sub downloads I integrate into my Sonarr library need
per-subscription overrides such as:
- Sonarr manages years 2020-2023 for one series and ytdl-sub fills in
from there, so set a `sonarr_series_after: "20240101"` override
- each series needs a directory named per Sonarr's configuration with the TVDB ID, the subscription key,
but also a series name prefix without the TVDB ID for each episode file, so set a
`sonarr_series_prefix: "Foo Series (2020)"` override
So are there other use cases for override mode that are more common and less niche than
mine?
* docs(docker): Revert example bind mount volumes
Per [PR feedback](https://github.com/jmbannon/ytdl-sub/pull/1310#discussion_r2306067263).
Reconcile the various debugging hints throughout the documentation and integrate into a
central debugging page. Includes clarification of the roles of ytdl-sub, yt-dlp, and
external services. Also adds cross references to the relevant bits across the docs.
* docs(logs): Mimick argparse option default format
I find it more readable to use punctuation to separate technical information, such as
required vs optional or default values, from narrative description. Follow argparse's
lead, and put such information in parens following the narrative description.
* docs(usage): Consistent structure, clarifications
* docs(config): Clarify persist_logs behavior/opts
Clarify the `persist_logs:` options per [Discord discussion](https://discord.com/channels/994270357957648404/1409161361853780060/1409602529460879431).
This is another change that touches the generated docs, so after finishing my writing, I
copied the change to `./docs/source/config_reference/scripting/static_variables.rst`
into the corresponding docstring in
`./src/ytdl_sub/entries/variables/override_variables.py` and formatted it as a
docstring. Then I ran:
$ REGENERATE_DOCS=1 tox exec -e "py" -- pytest tests/unit/docgen/test_docgen.py
But all that did was revert the changes to
`./docs/source/config_reference/scripting/static_variables.rst`. IOW, it did *not*
update the rST from the changed docstring.
* docs(preset): Config file top presets vs base
* docs(throttle): resolution/quality throttle asset
I also added the "wait a few hours and try again" hint, sorry I didn't think of this
when reviewing your PR. While doing that I also reformatted the `%concat(...)` arguments
to be more readable to me. Particularly to keep clear what punctuation is about argument
separation as opposed to punctuation meant to be included in the resulting string. Not a
strong preference if you don't like it.
* docs(various): Minor typo and formatting fixes
* docs(preset): Fix presets vs preset in config file
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289960678).
* test(throttle): Match modified assertion text
If it were my test, I'd only make assertions on the spirit of error message text
necessary to confirm it's the right error message and not another.
* docs(throttle): More likely override value
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289976156).
Closes https://github.com/jmbannon/ytdl-sub/issues/1241
YouTube is upping their throttling by serving 360p videos. Most likely to give us and genAI scraping a bad time on purpose.
Resolution assertion will be enabled by default. It will:
- Require > 360 height (pixels)
- Can configure the width by setting `resolution_assert_height_gte: 1080` to require 1080p, or any desired value
- Can disable resolution assert by setting `enable_resolution_assert: False`
* docs(install): Redundant platform-specific mention
I'm guessing this is a remnant from before it got it's own page.
* docs(docker): Clarify container image install
Capturing my changes thus far while I go off to educate myself about rST/Sphinx refs vs
links. As such, this duplicates a reference link in a way I don't like:
:ref:`Automating Downloads <guides/getting_started/automating_downloads:docker and
unraid>`
If you want to fix that or are OK with the duplication, this should be ready to
go. Otherwise, I'll fix and force push once I've learned how to do this correctly.
A matter of opinion in this change, I did away with the tabbed GUI vs headless code
blocks and the separate CPU/GPU passthrough code blocks because I find comments in
example code to be more clear, more readable, and more approachable. This also has the
benefit of putting informative comments in the resulting user's configuration. For
example, imagine a user that learns only later that they require GPU passthrough, their
configuration is already ready to get them started. This comes at the cost some repeated
comments (`environment:` and `deploy:` for GPU passthrough) and the fancy Sphinx
highlighting of relevant lines. I think the trade-off is a net benefit for users.
* docs(sphinx): Fix extlinks LSIO reference usage
* docs(sphinx): Address inline literal warning
Addresses:
./docs/source/config_reference/plugins.rst:848: WARNING: Inline literal start-string
without end-string. [docutils]
* docs(start): Clarify prerequisite tech knowledge
Thanks @rpatterson for the fix!
---------
Following [discussion about the previous preset
delay](https://discord.com/channels/994270357957648404/994270357957648408/1405705039649046589),
we weren't able to find a reason for it and there was agreement that the example value
from yt-dlp is more sensible.
While we're at it, also set a default `min:` value that would probably be sensible if it
were to be used in the future.
Finally, capture the reasoning behind the new preset values in comments to make the
reference documentation more helpful to new users in the future.
Adds the prebuilt preset ``Filter Duration``, which can include/exclude media based on its duration.
Supports the following override variables:
* ``filter_duration_min_s``
* ``filter_duration_max_s``
Usage:
```
Plex TV Show by Date | Filter Duration:
= Documentaries:
"~NOVA PBS":
url: "https://www.youtube.com/@novapbs"
filter_duration_min_s: 120 # Only download videos at least 2m long
= Sports:
"~Maple Leafs Highlights":
url: "https://www.youtube.com/@NHL"
filter_duration_max_s: 180 # Only get highlight videos less than 3m long
```
To protect new users, all prebuilt preset (excluding Soundcloud and Bandcamp) will include throttle protection by default.
> How do I disable?
Set the override variable `enable_throttle_protection: False`.
This can be done on a per-subscription basis and/or in the top __preset__ section to apply to all presets:
```
__preset__:
overrides:
enable_throttle_protection: False
```
> What if I already use throttle protection?
Your settings will override whatever values are set in the defaults.
Closes:
- https://github.com/jmbannon/ytdl-sub/issues/1121
- https://github.com/jmbannon/ytdl-sub/issues/1186
Adds the ability to make thumbnails square, both file and embedded. Usage:
```
my_preset:
square_thumbnail: True
```
Will soon enable this by default for all music-based presets with a toggle to disable.
Huge thanks to @Kentaro1043 for crafting the ffmpeg command for this!
Closes https://github.com/jmbannon/ytdl-sub/issues/383
