Fixes `UPDATE_YT_DLP_ON_START` when set to 'nightly' as the pip command is missing --break-system-packages. Simplifies dockerfiles by setting this globally via pip config file.
Thanks @Snuffy2 for the contribution!
Closes https://github.com/jmbannon/ytdl-sub/issues/1269
Adds the new paramter `languages_required`, which forces an entry to have this set of subtitle files, otherwise it will error.
In addition, better file checking has been added to ensure they exist before attempting to move.
Closes https://github.com/jmbannon/ytdl-sub/issues/1154
For docker images, add the ability to set the variable `UPDATE_YT_DLP_ON_START`, which will update yt-dlp to the specified version if a new one exists.
Supports ``stable``, ``nightly``, ``master``.
Closes https://github.com/jmbannon/ytdl-sub/issues/1313
Adds `resolution_assert_ignore_titles` variable to be able to ignore specific videos in a subscription where 360p is suspected to be the only option.
Ideally, it would be nice if ytdl-sub could determine whether 360p is actually the only resolution or not automatically, and download if it's true. We need to do more research to determine if this is possible.
In the meantime, the above variable should help manually get around this. Usage:
```
# use tilda mode to set override variables to the subscription
"~My Subscription":
url: "https://youtube.com/@channel"
resolution_assert_ignore_titles:
- "This 360p Video Title"
```
Changes documentation and default configs to use the `/tv_shows` default path shown in the docker compose setup.
Also changes working directory and file lock directory to not be in /tmp, which often causes permission issues.
```
Resolution assert is enabled, will fail on low-quality video downloads and presume throttle. Disable using the override variable `enable_resolution_assert: False
```
Spams during the initialization phase. Ideally, we should only print this once the subscription has begun downloading. Perhaps via a `print` plugin? Either way, set this to debug log level.
* ci(docs): RTD Sphinx warnings false successes
Use the same options on readthedocs.com that we use locally to fail on all warnings,
mostly to better catch stale cross references.
* docs(usage): Yet another stale Sphinx cross ref
I confirmed the previous RTD CI fix in the previous commit by pushing it to a draft PR
before pushing this change and the RTD action failed with the warning this commit
fixes. After pushing this commit, the RTD action succeeded.
* build(docs): Sphinx stale cross refs false success
I figured out why I kept getting warnings for broken Sphinx cross-refs *after* the
changes that caused them have already been merged, changes I know I ran `$ make docs`
for before pushing. The issue is that by default Sphinx only builds changed files for
faster iterations while editing, but it only catches broken cross-refs when it builds
files. So if changing, for example, a section name in one page that is referenced from
another page that you do *not* change, then the warning will be missed until something
changes that other page, such as a pull or rebase.
I considered adding a separate `./Makefile` target for incremental builds
in the inner loop of making changes. But I opted to just remove the `--write-all` CLI
option locally while editing in the inner loop, because the uncommitted change will
remind me to revert it and run a full rebuild before pushing.
Adds support for setting throttle_protection range values using static override variables. `sleep_per_download_s` has additional support for entry variables since it's used per entry.
We can now experiment with scaling this value based on entry attributes, such as duration. Example:
```
throttle_protection:
sleep_per_download_s:
min: >-
{
%mul(5.5, %pow( duration, 0.4 ))
}
max: >-
{
%mul(6.5, %pow( duration, 0.6 ))
}
```
* docs(presets): Stale copied prebuilt comment
* docs(start): How to configure subscriptions
Revise the subscriptions file Getting Started tutorial from the perspective of a new
user. The idea is that the ref docs describe a component as the code sees it, whereas
the how-to/tutorial/getting-started docs provide a narrative description of the
component from the external perspective of a typical new user.
* docs(start): Document subscription override mode
It's debatable whether this belongs here. Having moved onto my own custom presets, the
only use cases for override mode in my subscriptions file are for those series that also
have TVDB metadata. Those ytdl-sub downloads I integrate into my Sonarr library need
per-subscription overrides such as:
- Sonarr manages years 2020-2023 for one series and ytdl-sub fills in
from there, so set a `sonarr_series_after: "20240101"` override
- each series needs a directory named per Sonarr's configuration with the TVDB ID, the subscription key,
but also a series name prefix without the TVDB ID for each episode file, so set a
`sonarr_series_prefix: "Foo Series (2020)"` override
So are there other use cases for override mode that are more common and less niche than
mine?
* docs(docker): Revert example bind mount volumes
Per [PR feedback](https://github.com/jmbannon/ytdl-sub/pull/1310#discussion_r2306067263).
Adds better support to run commands via `docker exec ...` by setting the working directory to be ytdl-sub's working directory.
Thanks @rpatterson for the contribution 😊
Reconcile the various debugging hints throughout the documentation and integrate into a
central debugging page. Includes clarification of the roles of ytdl-sub, yt-dlp, and
external services. Also adds cross references to the relevant bits across the docs.
* docs(logs): Mimick argparse option default format
I find it more readable to use punctuation to separate technical information, such as
required vs optional or default values, from narrative description. Follow argparse's
lead, and put such information in parens following the narrative description.
* docs(usage): Consistent structure, clarifications
* docs(config): Clarify persist_logs behavior/opts
Clarify the `persist_logs:` options per [Discord discussion](https://discord.com/channels/994270357957648404/1409161361853780060/1409602529460879431).
This is another change that touches the generated docs, so after finishing my writing, I
copied the change to `./docs/source/config_reference/scripting/static_variables.rst`
into the corresponding docstring in
`./src/ytdl_sub/entries/variables/override_variables.py` and formatted it as a
docstring. Then I ran:
$ REGENERATE_DOCS=1 tox exec -e "py" -- pytest tests/unit/docgen/test_docgen.py
But all that did was revert the changes to
`./docs/source/config_reference/scripting/static_variables.rst`. IOW, it did *not*
update the rST from the changed docstring.
* docs(preset): Config file top presets vs base
* docs(throttle): resolution/quality throttle asset
I also added the "wait a few hours and try again" hint, sorry I didn't think of this
when reviewing your PR. While doing that I also reformatted the `%concat(...)` arguments
to be more readable to me. Particularly to keep clear what punctuation is about argument
separation as opposed to punctuation meant to be included in the resulting string. Not a
strong preference if you don't like it.
* docs(various): Minor typo and formatting fixes
* docs(preset): Fix presets vs preset in config file
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289960678).
* test(throttle): Match modified assertion text
If it were my test, I'd only make assertions on the spirit of error message text
necessary to confirm it's the right error message and not another.
* docs(throttle): More likely override value
From [feedback](https://github.com/jmbannon/ytdl-sub/pull/1289#discussion_r2289976156).
I've been running into variations in style in the docs I've edited so far. I've been
including changes for consistency in with other commits but in many cases that makes one
part of one doc more consistent with the rest but less consistent with itself. It seems
like a bit of cleanup may be in order.
This change applies the following conventions:
- Section heading heirarchy:
#. ``=`` with overline for document title, IOW first heading
#. ``-``
#. ``~``
#. ``"``
- Two newlines before ``-`` sections for readability
- A newline between every section of any level and the first line of text
- Wrap paragraph lines at 88 characters to match Python's Black
I've only applied these changes to those ``*.rst`` files that aren't generated.
In the future, I might suggest another bulk change to match [the Sphinx conventions for
section
headings](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections).
Closes https://github.com/jmbannon/ytdl-sub/issues/1241
YouTube is upping their throttling by serving 360p videos. Most likely to give us and genAI scraping a bad time on purpose.
Resolution assertion will be enabled by default. It will:
- Require > 360 height (pixels)
- Can configure the width by setting `resolution_assert_height_gte: 1080` to require 1080p, or any desired value
- Can disable resolution assert by setting `enable_resolution_assert: False`
* docs(config): Clarify preset ordering priority
Refs #1276
* docs(architecture): WIP Narrative how it works
Much of this is repetition of the reference docs and some of the changes here should
also be applied here. The goal is to repeat here the parts of the reference docs that
may hang up a user when reading "Getting Started" without sending them off to get lost
in the weeds of the reference docs.
Note that this references a `Quick Start` document which is just an empty placeholder
and is to be written in a subsequent change.
Fixes#1279
* docs(terms): Glossary redundant with architecture
Now that the introduction includes an architectural overview, that section introduces
new users to necessary concepts and associated terminology and makes the terminology
section redundant in that part of the docs.
* docs(quick): Add a rote quick start guide
This is what I came up with when I tried to write instructions requiring as little
understanding as possible. Doing so really did reinforce my impression that this just
shouldn't be done, that significant understanding is required for *any* use of ytdl-sub
and even offering a quick start may be irresponsible. I'm still on the fence, thoughts?
That said, I also think I got some good explanations out of this and a better
understanding of what the next bits of the Getting Started should be. This Quick Start
is currently redundant with other parts of the Getting Started pages. I suspect that
I'll end up repeating and/or reorganizing parts of this Quick Start into the next bits
of the Getting Started. To that end I might argue that this change is a WIP and that
merging should wait. But I could also argue that this is an incremental improvement and
can be merged before that other work. Your call.
* docs(start): Various rST/Sphinx markup issues
* docs(install): Redundant platform-specific mention
I'm guessing this is a remnant from before it got it's own page.
* docs(docker): Clarify container image install
Capturing my changes thus far while I go off to educate myself about rST/Sphinx refs vs
links. As such, this duplicates a reference link in a way I don't like:
:ref:`Automating Downloads <guides/getting_started/automating_downloads:docker and
unraid>`
If you want to fix that or are OK with the duplication, this should be ready to
go. Otherwise, I'll fix and force push once I've learned how to do this correctly.
A matter of opinion in this change, I did away with the tabbed GUI vs headless code
blocks and the separate CPU/GPU passthrough code blocks because I find comments in
example code to be more clear, more readable, and more approachable. This also has the
benefit of putting informative comments in the resulting user's configuration. For
example, imagine a user that learns only later that they require GPU passthrough, their
configuration is already ready to get them started. This comes at the cost some repeated
comments (`environment:` and `deploy:` for GPU passthrough) and the fancy Sphinx
highlighting of relevant lines. I think the trade-off is a net benefit for users.
* docs(sphinx): Fix extlinks LSIO reference usage
* docs(sphinx): Address inline literal warning
Addresses:
./docs/source/config_reference/plugins.rst:848: WARNING: Inline literal start-string
without end-string. [docutils]
* docs(start): Clarify prerequisite tech knowledge
* build(make): Defensive make settings best practice
I've been bitten by silent failures in make many times and these settings have spared me
that many times. I now use them religiously whenever possible.
* build(docs): Missed Sphinx issues, fail on warn
It would be nice to have these failures in CI. Is there a way to get the ReadTheDocs
integration to use these options?
* docs(docker): Clarify where the configuration is
The docs move from installation with Docker straight to modifying the configuration
without detailing where those files are.
* docs(start): Defining comments is discracting
This one is squarely me with my editor's red pen, do what you will. I whole heartedly
agree with this statement about comments and the intention (I assume) to encourage new
users to comment their configurations early and often. I just think this isn't the right
place for it and makes this document less focused and effective.
* docs(start): Less technical YAML key description
* docs(intro): Update and clarify goals/motivation
To help calibrate our collaboration, I went uninhibited with these changes including:
- assumptions about history I don't actually know:
"this project was one of the early entrants"
- description of the yt-dlp/ytdl-sub division of labor I'm not actually certain about
- max opinion, both about this project and the domain in general
- max cheeky/fun writing tone, with the intention of being personable
- max editor's red pen, cut mercilessly
IOW, review carefully and reject early and often to help me submit future changes that
require less review and revision. To that end, give me more rather than less feedback on
this change in particular.
I include an endorsement of Pinchflat because [they do the same for
ytdl-sub](https://github.com/kieraneglin/pinchflat?tab=readme-ov-file#what-it-does). It
would be my next choice if ytdl-sub couldn't meet my needs or vanquished me. More
than that link, in other Reddit comments and the like, the Pinchflat author clearly
expresses preference for ytdl-sub as their own "next option".
