From d801d0c5fc1934e6bc2dc71cd0e45a5f7802b0bc Mon Sep 17 00:00:00 2001 From: Ross Patterson Date: Thu, 21 Aug 2025 00:07:22 -0700 Subject: [PATCH] docs(faq): Misunderstood re-download CLI options https://discord.com/channels/994270357957648404/1102703969266049174/1407956781316964372 --- docs/source/faq/index.rst | 28 +++++++++---------- docs/source/guides/getting_started/index.rst | 18 ++++++++++++ .../guides/getting_started/quick_start.rst | 5 ++-- 3 files changed, 35 insertions(+), 16 deletions(-) diff --git a/docs/source/faq/index.rst b/docs/source/faq/index.rst index 5dd17ae9..58977b02 100644 --- a/docs/source/faq/index.rst +++ b/docs/source/faq/index.rst @@ -182,27 +182,27 @@ need to be removed: the downloaded files in this file and delete that whole entry, from the YouTube ID string to the closing curly braces. Be ware of JSON traling commas. -- Run ``$ ytdl-sub sub`` again with the appropriate CLI overrides: +- Run ``$ ytdl-sub sub`` again with the appropriate CLI plugin options: - Depending on the presets and overrides of the subscriptions whose downloaded files - you've renamed in the steps above, you need to pass some overrides from the command - line to tell ``ytdl-sub`` to change its behavior and re-download those files. + In normal operation, :ref:`yt-dlp minimizes requests and the files considered for + download `. To re-download, those options must be disabled or modified. Disable + :ref:`the 'break_on_existing' option `, set + :ref:`the 'date_range:' plugin `, and :ref:`limit + the subscriptions ` to + download only the files that you've renamed in the steps above. - Most presets tell ``yt-dlp`` not to bother even looking at videos before the most - recently downloaded video :ref:`using 'break_on_existing' - `. Other presets, such as :ref:`the Only Recent - presets `, restrict the videos considered for - download by date range. - - Determine the necessary CLI overrides and re-run, for example: + Set the appropriate dates and subscription name to include only the files you've + renamed, and re-run. For example, if you've renamed all the files from 2024 in the + ``NOVA PBS`` subscription: .. code-block:: shell ytdl-sub sub -o "\ --ytdl_options.break_on_existing False \ - --date_range.after 19700101 \ - --date_range.before 20380119 \ - " + --date_range.after 20240101 \ + --date_range.before 20250101 \ + " --match="NOVA PBS" There is a bug where... diff --git a/docs/source/guides/getting_started/index.rst b/docs/source/guides/getting_started/index.rst index 6e38c249..a0ada76c 100644 --- a/docs/source/guides/getting_started/index.rst +++ b/docs/source/guides/getting_started/index.rst @@ -104,6 +104,24 @@ Users define additional presets in :doc:`their configuration file <./first_confi they then use in most of their subscriptions. Most user-defines presets extend the :doc:`../../prebuilt_presets/index` provided by ``ytdl-sub``. +Minimize the work to only what's necessary +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Throttling and bans are a core problem for any web scraping tool, perhaps even more so +for ``yt-dlp``, and no good actor *wants* to be an onerous burden on a +service. Similarly, many web scraping use cases involve very large sets of data that are +too big to process as a whole for performance. It's important to narrow the amount of +data considered and minimize requests. + +To these ends, most presets tell ``yt-dlp`` not to consider files before the most +recently downloaded file using :ref:`the 'break_on_existing' option +`. Similarly, and particularly for huge channels +or playlists, most users should use either :ref:`an 'Only Recent' preset +` and/or :ref:`the 'Chunk Downloads' preset +` to restrict the number of downloads +considered. + + Caveats ~~~~~~~ diff --git a/docs/source/guides/getting_started/quick_start.rst b/docs/source/guides/getting_started/quick_start.rst index d0c27169..9c0402a9 100644 --- a/docs/source/guides/getting_started/quick_start.rst +++ b/docs/source/guides/getting_started/quick_start.rst @@ -63,8 +63,9 @@ instructions to the letter. Repeat steps #4-7 for each of your subscriptions. Be sure to repeat the preview and review steps for each subscription. In general, move slowly and carefully review - everything. It's best to catch issues early to avoid repeating downloads and to - minimize requests to avoid being throttled or banned by servers. + everything. It's best to catch issues early :ref:`to avoid repeating downloads and to + minimize requests `. #. Automate downloads: