[DOCS] How to test subscription downloads (#1318)
Revise the "Getting Started" page for testing subscription downloads.
This commit is contained in:
parent
a7cf361821
commit
72419a3e98
6 changed files with 78 additions and 84 deletions
|
|
@ -172,7 +172,7 @@ need to be removed:
|
||||||
necessary>`. To re-download, those options must be disabled or modified. Disable
|
necessary>`. To re-download, those options must be disabled or modified. Disable
|
||||||
:ref:`the 'break_on_existing' option <config_reference/plugins:ytdl_options>`, set
|
:ref:`the 'break_on_existing' option <config_reference/plugins:ytdl_options>`, set
|
||||||
:ref:`the 'date_range:' plugin <config_reference/plugins:date_range>`, and :ref:`limit
|
:ref:`the 'date_range:' plugin <config_reference/plugins:date_range>`, and :ref:`limit
|
||||||
the subscriptions <guides/getting_started/first_download:faster iteration cycle>` to
|
the subscriptions <guides/getting_started/downloading:preview>` to
|
||||||
download only the files that you've renamed in the steps above.
|
download only the files that you've renamed in the steps above.
|
||||||
|
|
||||||
Set the appropriate dates, :ref:`including a sufficient margin
|
Set the appropriate dates, :ref:`including a sufficient margin
|
||||||
|
|
@ -182,11 +182,11 @@ need to be removed:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
|
|
||||||
ytdl-sub sub -o "\
|
ytdl-sub --match="NOVA PBS" sub -o "\
|
||||||
--ytdl_options.break_on_existing False \
|
--ytdl_options.break_on_existing False \
|
||||||
--date_range.after 20240101 \
|
--date_range.after 20240101 \
|
||||||
--date_range.before 20250101 \
|
--date_range.before 20250101 \
|
||||||
" --match="NOVA PBS"
|
"
|
||||||
|
|
||||||
...download a file missing from the archive
|
...download a file missing from the archive
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
|
||||||
69
docs/source/guides/getting_started/downloading.rst
Normal file
69
docs/source/guides/getting_started/downloading.rst
Normal file
|
|
@ -0,0 +1,69 @@
|
||||||
|
Downloading
|
||||||
|
===========
|
||||||
|
|
||||||
|
Once you've :doc:`defined your subscriptions <./subscriptions>`, it's time to test your
|
||||||
|
configuration and try your first download. As a web scraping tool, :ref:`it's important
|
||||||
|
to minimize the requests sent to external services
|
||||||
|
<guides/getting_started/index:minimize the work to only what's necessary>` to avoid
|
||||||
|
triggering throttling or bans. Further, a full download of even one subscription can
|
||||||
|
take significant time. Test each change to your subscriptions carefully and quickly as
|
||||||
|
follows.
|
||||||
|
|
||||||
|
|
||||||
|
Preview
|
||||||
|
-------
|
||||||
|
|
||||||
|
Preview what ``ytdl-sub`` would do for this subscription. Run the :ref:`'sub'
|
||||||
|
sub-command <usage:subscriptions options>` with CLI options to restrict requests as much
|
||||||
|
as possible:
|
||||||
|
|
||||||
|
- Pull metadata and *simulate* a download without actually downloading any media files
|
||||||
|
using the ``--dry-run`` CLI option.
|
||||||
|
|
||||||
|
- Limit requests by narrowing the run to one subscription by giving a
|
||||||
|
subscription name to the ``--match`` CLI option.
|
||||||
|
|
||||||
|
- Stop after just a few downloads to further minimize requests and make testing faster
|
||||||
|
using the ``max_downloads`` setting from ``yt-dlp``.
|
||||||
|
|
||||||
|
Change to the directory containing your ``./subscriptions.yaml`` file and run with those
|
||||||
|
options:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
cd "/config/ytdl-sub-configs/"
|
||||||
|
ytdl-sub --dry-run --match="NOVA PBS" sub -o '--ytdl_options.max_downloads 3'
|
||||||
|
|
||||||
|
Examine the output carefully, investigate anything that doesn't look right and repeat
|
||||||
|
this step until everything looks right.
|
||||||
|
|
||||||
|
|
||||||
|
Review
|
||||||
|
------
|
||||||
|
|
||||||
|
Review the results of real downloads. Run it again without the ``--dry-run`` option to
|
||||||
|
actually download media and place the files in your library:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ytdl-sub --match="NOVA PBS" sub -o '--ytdl_options.max_downloads 3'
|
||||||
|
|
||||||
|
Examine the output carefully again. Then examine how the resulting downloads work in
|
||||||
|
your library. Repeat with a larger value for ``max_downloads`` and examine the output
|
||||||
|
and downloads again.
|
||||||
|
|
||||||
|
|
||||||
|
Next Steps
|
||||||
|
----------
|
||||||
|
|
||||||
|
Once you're `previewed <preview_>`_ and `reviewed <review_>`_ successful downloads of
|
||||||
|
each of your subscriptions, you're ready to run a full download of all your
|
||||||
|
subscriptions. Run the sub-command without the CLI options you used to limit what
|
||||||
|
``ytdl-sub`` does while testing:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ytdl-sub sub
|
||||||
|
|
||||||
|
If you're ready to let ``ytdl-sub`` run unattended, it's time to :doc:`automate
|
||||||
|
downloads <./automating_downloads>`.
|
||||||
|
|
@ -1,56 +0,0 @@
|
||||||
Initial Download
|
|
||||||
================
|
|
||||||
|
|
||||||
Once you've created :doc:`a subscriptions file <./subscriptions>`, you can perform your
|
|
||||||
first download. Access ``ytdl-sub``, navigate to the directory containing your
|
|
||||||
``subscriptions.yaml`` file.
|
|
||||||
|
|
||||||
|
|
||||||
Dry Run
|
|
||||||
-------
|
|
||||||
|
|
||||||
Performing a dry run is important when applying any change to your subscriptions to
|
|
||||||
ensure output looks as expected. Dry runs will pull metadata to *simulate* a download
|
|
||||||
without actually downloading the media file.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
ytdl-sub --dry-run sub subscriptions.yaml
|
|
||||||
|
|
||||||
|
|
||||||
Faster Iteration Cycle
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
Testing subscriptions can take quite some time to perform a full download. This can be
|
|
||||||
speed up by applying an override via command-line to set max number of downloads.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
ytdl-sub --dry-run sub subscriptions.yaml -o '--ytdl_options.max_downloads 3'
|
|
||||||
|
|
||||||
Having many subscriptions could still make this dry run take a while. A subset of
|
|
||||||
subscriptions can be dry ran using a match.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
:caption: Only run subscriptions that have PBS in their names
|
|
||||||
|
|
||||||
ytdl-sub --dry-run sub subscriptions.yaml -o '--ytdl_options.max_downloads 3' --match PBS
|
|
||||||
|
|
||||||
|
|
||||||
Downloading
|
|
||||||
-----------
|
|
||||||
|
|
||||||
Once the subscriptions file is validated, a download can be performed by omitting the
|
|
||||||
dry run argument.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
ytdl-sub sub subscriptions.yaml
|
|
||||||
|
|
||||||
Multiple subscription file names can be provided to perform a download on all of them. A
|
|
||||||
single file named ``subscriptions.yaml`` does not require a file name specification
|
|
||||||
since it will look for that file name by default, making the following command valid.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
ytdl-sub sub
|
|
||||||
|
|
@ -151,7 +151,7 @@ adding subscriptions <./subscriptions>`.
|
||||||
:hidden:
|
:hidden:
|
||||||
|
|
||||||
subscriptions
|
subscriptions
|
||||||
first_download
|
downloading
|
||||||
automating_downloads
|
automating_downloads
|
||||||
first_config
|
first_config
|
||||||
quick_start
|
quick_start
|
||||||
|
|
|
||||||
|
|
@ -37,31 +37,12 @@ instructions to the letter.
|
||||||
``"https://www.youtube.com/@novapbs"`` value to the URL of the channel or playlist
|
``"https://www.youtube.com/@novapbs"`` value to the URL of the channel or playlist
|
||||||
for this subscription.
|
for this subscription.
|
||||||
|
|
||||||
#. Preview what ``ytdl-sub`` would do for this subscription:
|
#. :ref:`Preview <guides/getting_started/downloading:preview>` and :ref:`Review
|
||||||
|
<guides/getting_started/downloading:review>` the subscription.
|
||||||
Run the :ref:`'sub' sub-command <usage:subscriptions options>` but with the ``max_downloads``
|
|
||||||
setting from ``yt-dlp`` along with the ``--dry-run`` and ``--match`` options from
|
|
||||||
``ytdl-sub`` to minimize requests and prevent actual downloads. Be sure to update the
|
|
||||||
``--match="..."`` value with the subscription name::
|
|
||||||
|
|
||||||
$ ytdl-sub --dry-run sub -o '--ytdl_options.max_downloads 3' --match="NOVA PBS"
|
|
||||||
|
|
||||||
Examine the output carefully.
|
|
||||||
|
|
||||||
#. Review the results of real downloads:
|
|
||||||
|
|
||||||
Run it again without the ``--dry-run`` option to actually download media and place
|
|
||||||
the files in your library::
|
|
||||||
|
|
||||||
$ ytdl-sub sub -o '--ytdl_options.max_downloads 3' --match="NOVA PBS"
|
|
||||||
|
|
||||||
Examine the output carefully, then examine how the downloads work in your
|
|
||||||
library. Repeat with a larger value for ``max_downloads`` and examine the output and
|
|
||||||
downloads again.
|
|
||||||
|
|
||||||
#. Add the rest of your subscriptions:
|
#. Add the rest of your subscriptions:
|
||||||
|
|
||||||
Repeat steps #4-7 for each of your subscriptions. Be sure to repeat the preview and
|
Repeat steps #3-6 for each of your subscriptions. Be sure to repeat the preview and
|
||||||
review steps for each subscription. In general, move slowly and carefully review
|
review steps for each subscription. In general, move slowly and carefully review
|
||||||
everything. It's best to catch issues early :ref:`to avoid repeating downloads and to
|
everything. It's best to catch issues early :ref:`to avoid repeating downloads and to
|
||||||
minimize requests <guides/getting_started/index:minimize the work to only what's
|
minimize requests <guides/getting_started/index:minimize the work to only what's
|
||||||
|
|
|
||||||
|
|
@ -163,5 +163,5 @@ specific to only one subscription and will never be shared with another:
|
||||||
Next Steps
|
Next Steps
|
||||||
----------
|
----------
|
||||||
|
|
||||||
Once you've defined your subscriptions, it's time to :doc:`test the configuration and
|
Once you've defined your subscriptions, it's time to :doc:`test your configuration and
|
||||||
try your first download <./first_download>`.
|
try your first download <./downloading>`.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue