diff --git a/docs/source/faq/index.rst b/docs/source/faq/index.rst index 8f69d8cd..b33a4b55 100644 --- a/docs/source/faq/index.rst +++ b/docs/source/faq/index.rst @@ -154,6 +154,70 @@ suite of :ref:`scripting functions ` to create your own clever scraping mechanisms. +...force ytdl-sub to re-download a file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sometimes users may wish to replace a file already in the archive, for example, if the +current file is a lower resolution than desired, missing subtitles, corrupt, etc.. + +``ytdl-sub`` decides what files have already been downloaded by entries in :ref:`the +download archive file `, +``./.ytdl-sub-...-download-archive.json``, at the top of the subscription/series/show +:ref:`output directory ` in the appropriate +``overrides: / ..._directory:`` library path, *and* the presence of the corresponding +downloaded files under the same path. To force ``ytdl-sub`` to re-download an entry both +need to be removed: + +- Move aside the downloaded files: + + Rename or move the downloaded files, including the associated files with the same + base/stem name, such as ``./*.nfo``, ``./*.info-json``, etc.. + +- Ensure ``ytdl-sub`` is not running and won't run, such as by cron: + + ``ytdl-sub`` loads the ``./.ytdl-sub-...-download-archive.json`` file early, keeps it + in memory, and writes it back out late. If it's running or starts running while you're + modifying that file, then your changes will be overwritten when it exits. + +- Remove the ``./.ytdl-sub-...-download-archive.json`` JSON array item: + + Search for the stem name, the basename without any extension or suffix, common to all + the downloaded files in this file and delete that whole entry, from the YouTube ID + string to the closing curly braces. Be ware of JSON traling commas. + +- Run ``$ ytdl-sub sub`` again with the appropriate CLI plugin options: + + In normal operation, :ref:`yt-dlp minimizes requests and the files considered for + download `. To re-download, those options must be disabled or modified. Disable + :ref:`the 'break_on_existing' option `, set + :ref:`the 'date_range:' plugin `, and :ref:`limit + the subscriptions ` to + download only the files that you've renamed in the steps above. + + Set the appropriate dates and subscription name to include only the files you've + renamed, and re-run. For example, if you've renamed all the files from 2024 in the + ``NOVA PBS`` subscription: + + .. code-block:: shell + + ytdl-sub sub -o "\ + --ytdl_options.break_on_existing False \ + --date_range.after 20240101 \ + --date_range.before 20250101 \ + " --match="NOVA PBS" + +...download a file missing from the archive +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The root causes are unknown, but sometimes even after successful, complete runs, some +files will be missing from the archive. To attempt to download those missing files, +use `the same CLI options as re-downloading a file`_ + +.. _`the same CLI options as re-downloading a file`: + `...force ytdl-sub to re-download a file`_ + + There is a bug where... ----------------------- diff --git a/docs/source/guides/getting_started/index.rst b/docs/source/guides/getting_started/index.rst index f7674108..33ba7126 100644 --- a/docs/source/guides/getting_started/index.rst +++ b/docs/source/guides/getting_started/index.rst @@ -104,6 +104,24 @@ Users define additional presets in :doc:`their configuration file <./first_confi they then use in most of their subscriptions. Most user-defined presets extend the :doc:`../../prebuilt_presets/index` provided by ``ytdl-sub``. +Minimize the work to only what's necessary +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Throttling and bans are a core problem for any web scraping tool, perhaps even more so +for ``yt-dlp``, and no good actor *wants* to be an onerous burden on a +service. Similarly, many web scraping use cases involve very large sets of data that are +too big to process as a whole for performance. It's important to narrow the amount of +data considered and minimize requests. + +To these ends, most presets tell ``yt-dlp`` not to consider files before the most +recently downloaded file using :ref:`the 'break_on_existing' option +`. Similarly, and particularly for huge channels +or playlists, most users should use either :ref:`an 'Only Recent' preset +` and/or :ref:`the 'Chunk Downloads' preset +` to restrict the number of downloads +considered. + + Caveats ~~~~~~~ diff --git a/docs/source/guides/getting_started/quick_start.rst b/docs/source/guides/getting_started/quick_start.rst index d0c27169..9c0402a9 100644 --- a/docs/source/guides/getting_started/quick_start.rst +++ b/docs/source/guides/getting_started/quick_start.rst @@ -63,8 +63,9 @@ instructions to the letter. Repeat steps #4-7 for each of your subscriptions. Be sure to repeat the preview and review steps for each subscription. In general, move slowly and carefully review - everything. It's best to catch issues early to avoid repeating downloads and to - minimize requests to avoid being throttled or banned by servers. + everything. It's best to catch issues early :ref:`to avoid repeating downloads and to + minimize requests `. #. Automate downloads: