From c7b25d202d8bb130f8b54e0e3e8ec5ec105be8bc Mon Sep 17 00:00:00 2001 From: Qualis Svagtlys <45446348+Svagtlys@users.noreply.github.com> Date: Thu, 21 Dec 2023 11:32:27 -0600 Subject: [PATCH] Add in generated docs, documenting prebuilt_presets --- .../source/code_reference/entry_variables.rst | 442 +++++++++- docs/source/code_reference/index.rst | 4 +- .../code_reference/override_variables.rst | 45 +- docs/source/code_reference/plugins.rst | 799 ++++++++++++++++-- .../code_reference/prebuilt_presets.rst | 87 ++ .../code_reference/scripting_functions.rst | 474 +++++++++++ .../code_reference/subscriptions_yaml.rst | 2 +- docs/source/conf.py | 70 +- .../advanced_configuration.rst | 6 +- .../guides/getting_started/first_config.rst | 4 +- docs/source/guides/getting_started/index.rst | 13 +- .../getting_started/prebuilt_presets.rst | 5 - 12 files changed, 1806 insertions(+), 145 deletions(-) create mode 100644 docs/source/code_reference/prebuilt_presets.rst create mode 100644 docs/source/code_reference/scripting_functions.rst delete mode 100644 docs/source/guides/getting_started/prebuilt_presets.rst diff --git a/docs/source/code_reference/entry_variables.rst b/docs/source/code_reference/entry_variables.rst index d6c0c886..cbd3c0cf 100644 --- a/docs/source/code_reference/entry_variables.rst +++ b/docs/source/code_reference/entry_variables.rst @@ -1,11 +1,439 @@ -=============== Entry Variables =============== -.. .. autoclass:: ytdl_sub.entries.script.variable_definitions.VariableDefinitions -.. :members: -.. :inherited-members: -.. :undoc-members: +Entry Variables +--------------- -.. autoclass:: ytdl_sub.entries.script.variable_definitions.VariableDefinitions - :members: \ No newline at end of file +channel +~~~~~~~ +The channel name if it exists, otherwise returns the uploader. + +channel_id +~~~~~~~~~~ +The channel id if it exists, otherwise returns the entry uploader ID. + +chapters +~~~~~~~~ +Chapters if they exist + +comments +~~~~~~~~ +Comments if they are requested + +creator +~~~~~~~ +The creator name if it exists, otherwise returns the channel. + +description +~~~~~~~~~~~ +The description if it exists. Otherwise, returns an emtpy string. + +duration +~~~~~~~~ +The duration of the entry in seconds + +epoch +~~~~~ +The unix epoch of when the metadata was scraped by yt-dlp. + +epoch_date +~~~~~~~~~~ +The epoch's date, in YYYYMMDD format. + +epoch_hour +~~~~~~~~~~ +The epoch's hour + +ext +~~~ +The downloaded entry's file extension + +extractor +~~~~~~~~~ +The yt-dlp extractor name + +extractor_key +~~~~~~~~~~~~~ +The yt-dlp extractor key + +ie_key +~~~~~~ +The ie_key, used in legacy yt-dlp things as the 'info-extractor key' + +info_json_ext +~~~~~~~~~~~~~ +The "info.json" extension + +requested_subtitles +~~~~~~~~~~~~~~~~~~~ +Subtitles if they are requested and exist + +sponsorblock_chapters +~~~~~~~~~~~~~~~~~~~~~ +Sponsorblock Chapters if they are requested and exist + +thumbnail_ext +~~~~~~~~~~~~~ +The download entry's thumbnail extension. Will always return 'jpg'. Until there is a +need to support other image types, we always convert to jpg. + +title +~~~~~ +The title of the entry. If a title does not exist, returns its unique ID. + +title_sanitized_plex +~~~~~~~~~~~~~~~~~~~~ +The sanitized title with additional sanitizing for Plex. It replaces numbers with +fixed-width numbers so Plex does not recognize them as season or episode numbers. + +uid +~~~ +The entry's unique ID + +uid_sanitized_plex +~~~~~~~~~~~~~~~~~~ +The sanitized uid with additional sanitizing for Plex. Replaces numbers with +fixed-width numbers so Plex does not recognize them as season or episode numbers. + +uploader +~~~~~~~~ +The uploader if it exists, otherwise return the uploader ID. + +uploader_id +~~~~~~~~~~~ +The uploader id if it exists, otherwise return the unique ID. + +uploader_url +~~~~~~~~~~~~ +The uploader url if it exists, otherwise returns the webpage_url. + +webpage_url +~~~~~~~~~~~ +The url to the webpage. + +Metadata Variables +------------------ + +entry_metadata +~~~~~~~~~~~~~~ +The entry's info.json + +playlist_metadata +~~~~~~~~~~~~~~~~~ +Metadata from the playlist (i.e. the parent metadata, like playlist -> entry) + +sibling_metadata +~~~~~~~~~~~~~~~~ +Metadata from any sibling entries that reside in the same playlist as this entry. + +source_metadata +~~~~~~~~~~~~~~~ +Metadata from the source (i.e. the grandparent metadata, like channel -> playlist -> entry) + +Playlist Variables +------------------ + +playlist_count +~~~~~~~~~~~~~~ +Playlist count if it exists, otherwise returns ``1``. + +Note that for channels/playlists, any change (i.e. adding or removing a video) will make +this value change. Use with caution. + +playlist_description +~~~~~~~~~~~~~~~~~~~~ +The playlist description if it exists, otherwise returns the entry's description. + +playlist_index +~~~~~~~~~~~~~~ +Playlist index if it exists, otherwise returns ``1``. + +Note that for channels/playlists, any change (i.e. adding or removing a video) will make +this value change. Use with caution. + +playlist_index_padded +~~~~~~~~~~~~~~~~~~~~~ +playlist_index padded two digits + +playlist_index_padded6 +~~~~~~~~~~~~~~~~~~~~~~ +playlist_index padded six digits. + +playlist_index_reversed +~~~~~~~~~~~~~~~~~~~~~~~ +Playlist index reversed via ``playlist_count - playlist_index + 1`` + +playlist_index_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +playlist_index_reversed padded two digits + +playlist_index_reversed_padded6 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +playlist_index_reversed padded six digits. + +playlist_max_upload_date +~~~~~~~~~~~~~~~~~~~~~~~~ +Max upload_date for all entries in this entry's playlist if it exists, otherwise returns +``upload_date`` + +playlist_max_upload_year +~~~~~~~~~~~~~~~~~~~~~~~~ +Max upload_year for all entries in this entry's playlist if it exists, otherwise returns +``upload_year`` + +playlist_max_upload_year_truncated +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The max playlist truncated upload year for all entries in this entry's playlist if it +exists, otherwise returns ``upload_year_truncated``. + +playlist_title +~~~~~~~~~~~~~~ +Name of its parent playlist/channel if it exists, otherwise returns its title. + +playlist_uid +~~~~~~~~~~~~ +The playlist unique ID if it exists, otherwise return the entry unique ID. + +playlist_uploader +~~~~~~~~~~~~~~~~~ +The playlist uploader if it exists, otherwise return the entry uploader. + +playlist_uploader_id +~~~~~~~~~~~~~~~~~~~~ +The playlist uploader id if it exists, otherwise returns the entry uploader ID. + +playlist_uploader_url +~~~~~~~~~~~~~~~~~~~~~ +The playlist uploader url if it exists, otherwise returns the playlist webpage_url. + +playlist_webpage_url +~~~~~~~~~~~~~~~~~~~~ +The playlist webpage url if it exists. Otherwise, returns the entry webpage url. + +Release Date Variables +---------------------- + +release_date +~~~~~~~~~~~~ +The entry’s release date, in YYYYMMDD format. If not present, return the upload date. + +release_date_standardized +~~~~~~~~~~~~~~~~~~~~~~~~~ +The release date formatted as YYYY-MM-DD + +release_day +~~~~~~~~~~~ +The release day as an integer (no padding). + +release_day_of_year +~~~~~~~~~~~~~~~~~~~ +The day of the year, i.e. February 1st returns ``32`` + +release_day_of_year_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The release day of year, but padded i.e. February 1st returns "032" + +release_day_of_year_reversed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The release day, but reversed using ``{total_days_in_year} + 1 - {release_day}``, +i.e. February 2nd would have release_day_of_year_reversed of ``365 + 1 - 32`` = ``334`` + +release_day_of_year_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed release day of year, but padded i.e. December 31st returns "001" + +release_day_padded +~~~~~~~~~~~~~~~~~~ +The entry's release day padded to two digits, i.e. the fifth returns "05" + +release_day_reversed +~~~~~~~~~~~~~~~~~~~~ +The release day, but reversed using ``{total_days_in_month} + 1 - {release_day}``, +i.e. August 8th would have release_day_reversed of ``31 + 1 - 8`` = ``24`` + +release_day_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed release day, but padded. i.e. August 30th returns "02". + +release_month +~~~~~~~~~~~~~ +The release month as an integer (no padding). + +release_month_padded +~~~~~~~~~~~~~~~~~~~~ +The entry's release month padded to two digits, i.e. March returns "03" + +release_month_reversed +~~~~~~~~~~~~~~~~~~~~~~ +The release month, but reversed +using ``13 - {release_month}``, i.e. March returns ``10`` + +release_month_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed release month, but padded. i.e. November returns "02" + +release_year +~~~~~~~~~~~~ +The entry's release year + +release_year_truncated +~~~~~~~~~~~~~~~~~~~~~~ +The last two digits of the release year, i.e. 22 in 2022 + +release_year_truncated_reversed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The release year truncated, but reversed using ``100 - {release_year_truncated}``, i.e. +2022 returns ``100 - 22`` = ``78`` + +Source Variables +---------------- + +source_count +~~~~~~~~~~~~ +The source count if it exists, otherwise returns the playlist count. + +source_description +~~~~~~~~~~~~~~~~~~ +The source description if it exists, otherwise returns the playlist description. + +source_index +~~~~~~~~~~~~ +Source index if it exists, otherwise returns ``1``. + +It is recommended to not use this unless you know the source will never add new content +(it is easy for this value to change). + +source_index_padded +~~~~~~~~~~~~~~~~~~~ +The source index, padded. + +source_title +~~~~~~~~~~~~ +Name of the source (i.e. channel with multiple playlists) if it exists, otherwise +returns its playlist_title. + +source_uid +~~~~~~~~~~ +The source unique id if it exists, otherwise returns the playlist unique ID. + +source_uploader +~~~~~~~~~~~~~~~ +The source uploader if it exists, otherwise return the playlist_uploader + +source_uploader_id +~~~~~~~~~~~~~~~~~~ +The source uploader id if it exists, otherwise returns the playlist_uploader_id + +source_uploader_url +~~~~~~~~~~~~~~~~~~~ +The source uploader url if it exists, otherwise returns the source webpage_url. + +source_webpage_url +~~~~~~~~~~~~~~~~~~ +The source webpage url if it exists, otherwise returns the playlist webpage url. + +Upload Date Variables +--------------------- + +upload_date +~~~~~~~~~~~ +The entry’s uploaded date, in YYYYMMDD format. If not present, return today’s date. + +upload_date_standardized +~~~~~~~~~~~~~~~~~~~~~~~~ +The uploaded date formatted as YYYY-MM-DD + +upload_day +~~~~~~~~~~ +The upload day as an integer (no padding). + +upload_day_of_year +~~~~~~~~~~~~~~~~~~ +The day of the year, i.e. February 1st returns ``32`` + +upload_day_of_year_padded +~~~~~~~~~~~~~~~~~~~~~~~~~ +The upload day of year, but padded i.e. February 1st returns "032" + +upload_day_of_year_reversed +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The upload day, but reversed using ``{total_days_in_year} + 1 - {upload_day}``, +i.e. February 2nd would have upload_day_of_year_reversed of ``365 + 1 - 32`` = ``334`` + +upload_day_of_year_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed upload day of year, but padded i.e. December 31st returns "001" + +upload_day_padded +~~~~~~~~~~~~~~~~~ +The entry's upload day padded to two digits, i.e. the fifth returns "05" + +upload_day_reversed +~~~~~~~~~~~~~~~~~~~ +The upload day, but reversed using ``{total_days_in_month} + 1 - {upload_day}``, +i.e. August 8th would have upload_day_reversed of ``31 + 1 - 8`` = ``24`` + +upload_day_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed upload day, but padded. i.e. August 30th returns "02". + +upload_month +~~~~~~~~~~~~ +The upload month as an integer (no padding). + +upload_month_padded +~~~~~~~~~~~~~~~~~~~ +The entry's upload month padded to two digits, i.e. March returns "03" + +upload_month_reversed +~~~~~~~~~~~~~~~~~~~~~ +The upload month, but reversed using ``13 - {upload_month}``, i.e. March returns ``10`` + +upload_month_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The reversed upload month, but padded. i.e. November returns "02" + +upload_year +~~~~~~~~~~~ +The entry's upload year + +upload_year_truncated +~~~~~~~~~~~~~~~~~~~~~ +The last two digits of the upload year, i.e. 22 in 2022 + +upload_year_truncated_reversed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The upload year truncated, but reversed using ``100 - {upload_year_truncated}``, i.e. +2022 returns ``100 - 22`` = ``78`` + +Ytdl-Sub Variables +------------------ + +download_index +~~~~~~~~~~~~~~ +The i'th entry downloaded. NOTE that this is fetched dynamically from the download +archive. + +download_index_padded6 +~~~~~~~~~~~~~~~~~~~~~~ +The download_index padded six digits + +upload_date_index +~~~~~~~~~~~~~~~~~ +The i'th entry downloaded with this upload date. + +upload_date_index_padded +~~~~~~~~~~~~~~~~~~~~~~~~ +The upload_date_index padded two digits + +upload_date_index_reversed +~~~~~~~~~~~~~~~~~~~~~~~~~~ +100 - upload_date_index + +upload_date_index_reversed_padded +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The upload_date_index padded two digits + +ytdl_sub_input_url +~~~~~~~~~~~~~~~~~~ +The input URL used in ytdl-sub to create this entry. \ No newline at end of file diff --git a/docs/source/code_reference/index.rst b/docs/source/code_reference/index.rst index 3d75fdbd..ae199305 100644 --- a/docs/source/code_reference/index.rst +++ b/docs/source/code_reference/index.rst @@ -9,4 +9,6 @@ This section contains direct references to the code of ``ytdl-sub`` and informat plugins entry_variables override_variables - config \ No newline at end of file + config + prebuilt_presets + scripting_functions \ No newline at end of file diff --git a/docs/source/code_reference/override_variables.rst b/docs/source/code_reference/override_variables.rst index dfc42649..7cb23e4a 100644 --- a/docs/source/code_reference/override_variables.rst +++ b/docs/source/code_reference/override_variables.rst @@ -1,7 +1,44 @@ -================== Override Variables ================== -.. autoclass:: ytdl_sub.entries.variables.override_variables.OverrideVariables() - :members: - :member-order: bysource \ No newline at end of file +subscription_indent_i +--------------------- +For subscriptions in the form of + +.. code-block:: yaml + + Preset | = Indent Value 1: + = Indent Value 2: + "Subscription Name": "https://..." + +``subscription_indent_1`` and ``subscription_indent_2`` get set to +``Indent Value 1`` and ``Indent Value 2``. + +subscription_name +----------------- +Name of the subscription + +subscription_value +------------------ +For subscriptions in the form of + +.. code-block:: yaml + + "Subscription Name": "https://..." + +``subscription_value`` gets set to ``https://...``. + +subscription_value_i +-------------------- +For subscriptions in the form of + +.. code-block:: yaml + + "Subscription Name": + - "https://url1.com/..." + - "https://url2.com/..." + +``subscription_value_1`` and ``subscription_value_2`` get set to ``https://url1.com/...`` +and ``https://url2.com/...``. Note that ``subscription_value_1`` also gets set to +``subscription_value``. + diff --git a/docs/source/code_reference/plugins.rst b/docs/source/code_reference/plugins.rst index f739b3ef..451a0ed8 100644 --- a/docs/source/code_reference/plugins.rst +++ b/docs/source/code_reference/plugins.rst @@ -1,128 +1,767 @@ -======= Plugins ======= -Plugins are used to perform any type of post-processing to the already downloaded files. audio_extract -''''''''''''' -.. autoclass:: ytdl_sub.plugins.audio_extract.AudioExtractOptions() - :members: - :member-order: bysource - :exclude-members: partial_validate +------------- +Extracts audio from a video file. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + audio_extract: + codec: "mp3" + quality: 128 + +codec +~~~~~ +The codec to output after extracting the audio. Supported codecs are aac, flac, mp3, m4a, +opus, vorbis, wav, and best to grab the best possible format at runtime. + +quality +~~~~~~~ +Optional. Specify ffmpeg audio quality. Insert a value between ``0`` (better) and ``9`` +(worse) for variable bitrate, or a specific bitrate like ``128`` for 128k. chapters -'''''''' -.. autoclass:: ytdl_sub.plugins.chapters.ChaptersOptions() - :members: - :member-order: bysource - :exclude-members: partial_validate +-------- +Embeds chapters to video files if they are present. Additional options to add SponsorBlock +chapters and remove specific ones. Can also remove chapters using regex. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + chapters: + # Embedded Chapter Fields + embed_chapters: True + allow_chapters_from_comments: False + remove_chapters_regex: + - "Intro" + - "Outro" + + # Sponsorblock Fields + sponsorblock_categories: + - "outro" + - "selfpromo" + - "preview" + - "interaction" + - "sponsor" + - "music_offtopic" + - "intro" + remove_sponsorblock_categories: "all" + force_key_frames: False + +allow_chapters_from_comments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. If chapters do not exist in the video/description itself, attempt to scrape +comments to find the chapters. Defaults to False. + +embed_chapters +~~~~~~~~~~~~~~ +Optional. Embed chapters into the file. Defaults to True. + +force_key_frames +~~~~~~~~~~~~~~~~ +Optional. Force keyframes at cuts when removing sections. This is slow due to needing a +re-encode, but the resulting video may have fewer artifacts around the cuts. Defaults to +False. + +remove_chapters_regex +~~~~~~~~~~~~~~~~~~~~~ +Optional. List of regex patterns to match chapter titles against and remove them from the +entry. + +remove_sponsorblock_categories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. List of SponsorBlock categories to remove from the output file. Can only remove +categories that are specified in ``sponsorblock_categories`` or "all", which removes +everything specified in ``sponsorblock_categories``. + +sponsorblock_categories +~~~~~~~~~~~~~~~~~~~~~~~ +Optional. List of SponsorBlock categories to embed as chapters. Supports "sponsor", +"intro", "outro", "selfpromo", "preview", "filler", "interaction", "music_offtopic", +"poi_highlight", or "all" to include all categories. date_range -'''''''''' -.. autoclass:: ytdl_sub.plugins.date_range.DateRangeOptions() - :members: - :member-order: bysource - :exclude-members: partial_validate +---------- +Only download files uploaded within the specified date range. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + date_range: + before: "now" + after: "today-2weeks" + +after +~~~~~ +Optional. Only download videos after this datetime. + +before +~~~~~~ +Optional. Only download videos before this datetime. embed_thumbnail -'''''''''''''''' +--------------- +Whether to embed thumbnails to the audio/video file or not. -.. autoclass:: ytdl_sub.plugins.embed_thumbnail.EmbedThumbnailOptions() +Usage: -------------------------------------------------------------------------------- +.. code-block:: yaml + + presets: + my_example_preset: + embed_thumbnail: True file_convert -'''''''''''' -.. autoclass:: ytdl_sub.plugins.file_convert.FileConvertOptions() - :members: - :member-order: bysource - :exclude-members: partial_validate +------------ +Converts video files from one extension to another. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + file_convert: + convert_to: "mp4" + +Supports custom ffmpeg conversions: + +.. code-block:: yaml + + presets: + my_example_preset: + file_convert: + convert_to: "mkv" + convert_with: "ffmpeg" + ffmpeg_post_process_args: > + -bitexact + -vcodec copy + -acodec copy + -scodec mov_text + +convert_to +~~~~~~~~~~ +Convert to a desired file type. Supports: + +* Video: avi, flv, mkv, mov, mp4, webm +* Audio: aac, flac, mp3, m4a, opus, vorbis, wav + +convert_with +~~~~~~~~~~~~ +Optional. Supports ``yt-dlp`` and ``ffmpeg``. ``yt-dlp`` will convert files within +yt-dlp whereas ``ffmpeg`` specifies it will be converted using a custom command specified +with ``ffmpeg_post_process_args``. Defaults to ``yt-dlp``. + +ffmpeg_post_process_args +~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. ffmpeg args to post-process an entry file with. The args will be inserted in the +form of: + +.. code-block:: bash + + ffmpeg -i input_file.ext {ffmpeg_post_process_args) output_file.output_ext + +The output file will use the extension specified in ``convert_to``. Post-processing args +can still be set with ``convert_with`` set to ``yt-dlp``. format -'''''' -.. autoclass:: ytdl_sub.plugins.format.FormatOptions() +------ +Set ``--format`` to pass into yt-dlp to download a specific format quality. +Uses the same syntax as yt-dlp. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + format: "(bv*[height<=1080]+bestaudio/best[height<=1080])" + +format +~~~~~~ +yt-dlp format, uses same syntax as yt-dlp. match_filters -''''''''''''' -.. autoclass:: ytdl_sub.plugins.match_filters.MatchFiltersOptions() - :members: - :member-order: bysource - :exclude-members: partial_validate +------------- +Set ``--match-filters``` to pass into yt-dlp to filter entries from being downloaded. +Uses the same syntax as yt-dlp. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + match_filters: + filters: "original_url!*=/shorts/" + +Supports one or multiple filters: + +.. code-block:: yaml + + presets: + my_example_preset: + match_filters: + filters: + - "age_limit?100" + # Other common match-filters + # - "original_url!*=/shorts/ & !is_live" + # - "age_limit`_, +the same audio file tagging package used by +`beets `_. +It supports basic tags like ``title``, ``album``, ``artist`` and ``albumartist``. You can find +a full list of tags for various file types in MediaFile's +`source code `_. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + music_tags: + tags: + artist: "{artist}" + album: "{album}" + # Supports id3v2.4 multi-tags + genres: + - "{genre}" + - "ytdl-sub" + albumartists: + - "{artist}" + - "ytdl-sub" + +embed_thumbnail +~~~~~~~~~~~~~~~ +Optional. Whether to embed the thumbnail into the audio file. + +tags +~~~~ +Key, values of tag names, tag values. Supports source and override variables. +Supports lists which will get written to MP3s as id3v2.4 multi-tags. nfo_tags -'''''''' -.. autoclass:: ytdl_sub.plugins.nfo_tags.NfoTagsOptions() - :members: nfo_name, nfo_root, tags, kodi_safe - :member-order: bysource - :exclude-members: partial_validate +-------- +Adds an NFO file for every download file. An NFO file is simply an XML file +with a ``.nfo`` extension. You can add any values into the NFO. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + nfo_tags: + # required + nfo_name: "{title_sanitized}.nfo" + nfo_root: "episodedetails" + tags: + title: "{title}" + season: "{upload_year}" + episode: "{upload_month}{upload_day_padded}" + # optional + kodi_safe: False + +kodi_safe +~~~~~~~~~ +Optional. Kodi does not support > 3-byte unicode characters, which include emojis and some +foreign language characters. Setting this to True will replace those characters with '□'. +Defaults to False. + +nfo_name +~~~~~~~~ +The NFO file name. + +nfo_root +~~~~~~~~ +The root tag of the NFO's XML. In the usage above, it would look like + +.. code-block:: xml + + + + + +tags +~~~~ +Tags within the nfo_root tag. In the usage above, it would look like + +.. code-block:: xml + + + + Awesome Youtube Video + 2022 + 502 + + +Also supports xml attributes and duplicate keys: + +.. code-block:: yaml + + tags: + season: + attributes: + name: "Best Year" + tag: "{upload_year}" + genre: + - "Comedy" + - "Drama" + +Which translates to + +.. code-block:: xml + + 2022 + Comedy + Drama output_directory_nfo_tags -''''''''''''''''''''''''' -.. autoclass:: ytdl_sub.plugins.output_directory_nfo_tags.OutputDirectoryNfoTagsOptions() - :members: nfo_name, nfo_root, tags, kodi_safe - :member-order: bysource - :exclude-members: partial_validate +------------------------- +Adds a single NFO file in the output directory. An NFO file is simply an XML file with a +``.nfo`` extension. It uses the last entry's source variables which can change per download +invocation. Be cautious of which variables you use. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + output_directory_nfo_tags: + # required + nfo_name: "tvshow.nfo" + nfo_root: "tvshow" + tags: + title: "Sweet youtube TV show" + # optional + kodi_safe: False + +kodi_safe +~~~~~~~~~ +Optional. Kodi does not support > 3-byte unicode characters, which include emojis and some +foreign language characters. Setting this to True will replace those characters with '□'. +Defaults to False. + +nfo_name +~~~~~~~~ +The NFO file name. + +nfo_root +~~~~~~~~ +The root tag of the NFO's XML. In the usage above, it would look like + +.. code-block:: xml + + + + + +tags +~~~~ +Tags within the nfo_root tag. In the usage above, it would look like + +.. code-block:: xml + + + + Sweet youtube TV show + + +Also supports xml attributes and duplicate keys: + +.. code-block:: yaml + + tags: + named_season: + - tag: "{source_title}" + attributes: + number: "{collection_index}" + genre: + - "Comedy" + - "Drama" + +Which translates to + +.. code-block:: xml + + Sweet youtube TV show</season> + <genre>Comedy</genre> + <genre>Drama</genre> + +output_options +-------------- +Defines where to output files and thumbnails after all post-processing has completed. + +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + output_options: + # required + output_directory: "/path/to/videos_or_music" + file_name: "{title_sanitized}.{ext}" + # optional + thumbnail_name: "{title_sanitized}.{thumbnail_ext}" + info_json_name: "{title_sanitized}.{info_json_ext}" + download_archive_name: ".ytdl-sub-{subscription_name}-download-archive.json" + migrated_download_archive_name: ".ytdl-sub-{subscription_name_sanitized}-download-archive.json" + maintain_download_archive: True + keep_files_before: now + keep_files_after: 19000101 + +download_archive_name +~~~~~~~~~~~~~~~~~~~~~ +Optional. The file name to store a subscriptions download archive placed relative to +the output directory. Defaults to ``.ytdl-sub-{subscription_name}-download-archive.json`` + +file_name +~~~~~~~~~ +Required. The file name for the media file. This can include directories such as +``"Season {upload_year}/{title}.{ext}"``, and will be placed in the output directory. + +info_json_name +~~~~~~~~~~~~~~ +Optional. The file name for the media's info json file. This can include directories such +as ``"Season {upload_year}/{title}.{info_json_ext}"``, and will be placed in the output +directory. Can be set to empty string or `null` to disable info json writes. + +keep_files_after +~~~~~~~~~~~~~~~~ +Optional. Requires ``maintain_download_archive`` set to True. + +Only keeps files that are uploaded after this datetime. By default, ytdl-sub will keep +files after ``19000101``, which implies all files. Can be used in conjunction with +``keep_max_files``. + +keep_files_before +~~~~~~~~~~~~~~~~~ +Optional. Requires ``maintain_download_archive`` set to True. + +Only keeps files that are uploaded before this datetime. By default, ytdl-sub will keep +files before ``now``, which implies all files. Can be used in conjunction with +``keep_max_files``. + +keep_max_files +~~~~~~~~~~~~~~ +Optional. Requires ``maintain_download_archive`` set to True. + +Only keeps N most recently uploaded videos. If set to <= 0, ``keep_max_files`` will not be +applied. Can be used in conjunction with ``keep_files_before`` and ``keep_files_after``. + +maintain_download_archive +~~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. Maintains a download archive file in the output directory for a subscription. +It is named ``.ytdl-sub-{subscription_name}-download-archive.json``, stored in the +output directory. + +The download archive contains a mapping of ytdl IDs to downloaded files. This is used to +create a ytdl download-archive file when invoking a download on a subscription. This will +prevent ytdl from redownloading media already downloaded. + +Defaults to False. + +migrated_download_archive_name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. Intended to be used if you are migrating a subscription with either a new +subscription name or output directory. It will try to load the archive file using this name +first, and fallback to ``download_archive_name``. It will always save to this file +and remove the original ``download_archive_name``. + +output_directory +~~~~~~~~~~~~~~~~ +Required. The output directory to store all media files downloaded. + +thumbnail_name +~~~~~~~~~~~~~~ +Optional. The file name for the media's thumbnail image. This can include directories such +as ``"Season {upload_year}/{title}.{thumbnail_ext}"``, and will be placed in the output +directory. Can be set to empty string or `null` to disable thumbnail writes. + +overrides +--------- +Optional. This section allows you to define variables that can be used in any string formatter. +For example, if you want your file and thumbnail files to match without copy-pasting a large +format string, you can define something like: + +.. code-block:: yaml + + presets: + my_example_preset: + overrides: + output_directory: "/path/to/media" + custom_file_name: "{upload_date_standardized}.{title_sanitized}" + + # Then use the override variables in the output options + output_options: + output_directory: "{output_directory}" + file_name: "{custom_file_name}.{ext}" + thumbnail_name: "{custom_file_name}.{thumbnail_ext}" + +Override variables can contain explicit values and other variables, including both override +and source variables. + +In addition, any override variable defined will automatically create a ``sanitized`` variable +for use. In the example above, ``output_directory_sanitized`` will exist and perform +sanitization on the value when used. regex -''''' -.. autoclass:: ytdl_sub.plugins.regex.RegexOptions() - :members: skip_if_match_fails +----- +Performs regex matching on an entry's source or override variables. Regex can be used to filter +entries from proceeding with download or capture groups to create new source variables. -.. autoclass:: ytdl_sub.plugins.regex.VariableRegex() - :members: match, capture_group_names, capture_group_defaults, exclude - :member-order: bysource - :exclude-members: partial_validate +NOTE that YAML differentiates between single-quote (``'``) and double-quote (``"``), which can +affect regex. Double-quote implies string literals, i.e. ``"\n"`` is the literal chars ``\n``, +whereas single-quote, ``'\n'`` gets evaluated to a new line. To escape ``\`` when using +single-quote, use ``\\``. This is necessary if you want your regex to be something like +``\d\n`` to match a number and adjacent new-line. It must be written as ``\\d\n``. -------------------------------------------------------------------------------- +If you want to regex-search multiple source variables to create a logical OR effect, you can +create an override variable that contains the concatenation of them, and search that with regex. +For example, creating the override variable ``"title_and_description": "{title} {description}"`` +and using ``title_and_description`` can regex match/exclude from either ``title`` or +``description``. + +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + regex: + # By default, if any match fails and has no defaults, the entry will + # be skipped. If False, ytdl-sub will error and stop all downloads + # from proceeding. + skip_if_match_fails: True + + from: + # For each entry's `title` value... + title: + # Perform this regex match on it to act as a filter. + # This will only download videos with "[Official Video]" in it. Note that we + # double backslash to make YAML happy + match: + - '\\[Official Video\\]' + + # For each entry's `description` value... + description: + # Match with capture groups and defaults. + # This tries to scrape a date from the description and produce new + # source variables + match: + - '([0-9]{4})-([0-9]{2})-([0-9]{2})' + # Exclude any entry where the description contains #short + exclude: + - '#short' + + # Each capture group creates these new source variables, respectively, + # as well a sanitized version, i.e. `captured_upload_year_sanitized` + capture_group_names: + - "captured_upload_year" + - "captured_upload_month" + - "captured_upload_day" + + # And if the string does not match, use these as respective default + # values for the new source variables. + capture_group_defaults: + - "{upload_year}" + - "{upload_month}" + - "{upload_day}" + +skip_if_match_fails +~~~~~~~~~~~~~~~~~~~ +Defaults to True. If True, when any match fails and has no defaults, the entry will be +skipped. If False, ytdl-sub will error and all downloads will not proceed. split_by_chapters -''''''''''''''''' -.. autoclass:: ytdl_sub.plugins.split_by_chapters.SplitByChaptersOptions() - :members: when_no_chapters - :member-order: bysource - :exclude-members: partial_validate +----------------- +Splits a file by chapters into multiple files. Each file becomes its own entry with the +new source variables ``chapter_title``, ``chapter_title_sanitized``, ``chapter_index``, +``chapter_index_padded``, ``chapter_count``. -------------------------------------------------------------------------------- +If a file has no chapters, and ``when_no_chapters`` is set to "pass", then ``chapter_title`` is +set to the entry's title and ``chapter_index``, ``chapter_count`` are both set to 1. + +Note that when using this plugin and performing dry-run, it assumes embedded chapters are being +used with no modifications. + +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + split_by_chapters: + when_no_chapters: "pass" + +when_no_chapters +~~~~~~~~~~~~~~~~ +Behavior to perform when no chapters are present. Supports "pass" (continue processing), +"drop" (exclude it from output), and "error" (stop processing for everything). subtitles -''''''''' -.. autoclass:: ytdl_sub.plugins.subtitles.SubtitleOptions() - :members: subtitles_name, subtitles_type, embed_subtitles, languages, allow_auto_generated_subtitles - :member-order: bysource - :exclude-members: partial_validate +--------- +Defines how to download and store subtitles. Using this plugin creates two new variables: +``lang`` and ``subtitles_ext``. ``lang`` is dynamic since you can download multiple subtitles. +It will set the respective language to the correct subtitle file. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + subtitles: + subtitles_name: "{title_sanitized}.{lang}.{subtitles_ext}" + subtitles_type: "srt" + embed_subtitles: False + languages: "en" # supports list of multiple languages + allow_auto_generated_subtitles: False + +allow_auto_generated_subtitles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Optional. Whether to allow auto generated subtitles. Defaults to False. + +embed_subtitles +~~~~~~~~~~~~~~~ +Optional. Whether to embed the subtitles into the video file. Defaults to False. +NOTE: webm files can only embed "vtt" subtitle types. + +languages +~~~~~~~~~ +Optional. Language code(s) to download for subtitles. Supports a single or list of multiple +language codes. Defaults to "en". + +subtitles_name +~~~~~~~~~~~~~~ +Optional. The file name for the media's subtitles if they are present. This can include +directories such as ``"Season {upload_year}/{title_sanitized}.{lang}.{subtitles_ext}"``, and +will be placed in the output directory. ``lang`` is dynamic since you can download multiple +subtitles. It will set the respective language to the correct subtitle file. + +subtitles_type +~~~~~~~~~~~~~~ +Optional. One of the subtitle file types "srt", "vtt", "ass", "lrc". Defaults to "srt" throttle_protection -''''''''''''''''''' -.. autoclass:: ytdl_sub.plugins.throttle_protection.ThrottleProtectionOptions() - :members: - :member-order: bysource +------------------- +Provides options to make ytdl-sub look more 'human-like' to protect from throttling. For +range-based values, a random number will be chosen within the range to avoid sleeps looking +scripted. -------------------------------------------------------------------------------- +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + throttle_protection: + sleep_per_download_s: + min: 2.2 + max: 10.8 + sleep_per_subscription_s: + min: 9.0 + max: 14.1 + max_downloads_per_subscription: + min: 10 + max: 36 + subscription_download_probability: 1.0 + +max_downloads_per_subscription +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Number of downloads to perform per subscription. + +sleep_per_download_s +~~~~~~~~~~~~~~~~~~~~ +Number in seconds to sleep between each download. Does not include time it takes for +ytdl-sub to perform post-processing. + +sleep_per_subscription_s +~~~~~~~~~~~~~~~~~~~~~~~~ +Number in seconds to sleep between each subscription. + +subscription_download_probability +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Probability to perform any downloads, recomputed for each subscription. This is only +recommended to set if you run ytdl-sub in a cron-job, that way you are statistically +guaranteed over time to eventually download the subscription. video_tags -'''''''''' -.. autoclass:: ytdl_sub.plugins.video_tags.VideoTagsOptions() \ No newline at end of file +---------- +Adds tags to every downloaded video file using ffmpeg ``-metadata key=value`` args. + +Usage: + +.. code-block:: yaml + + presets: + my_example_preset: + video_tags: + title: "{title}" + date: "{upload_date}" + description: "{description}" + +tags +~~~~ +Key/values of tag names/values. Supports source and override variables. + +ytdl_options +------------ +Optional. This section allows you to add any ytdl argument to ytdl-sub's downloader. +The argument names can differ slightly from the command-line argument names. See +`this docstring <https://github.com/yt-dlp/yt-dlp/blob/2022.04.08/yt_dlp/YoutubeDL.py#L197>`_ +for more details. + +ytdl_options should be formatted like: + +.. code-block:: yaml + + presets: + my_example_preset: + ytdl_options: + # Ignore any download related errors and continue + ignoreerrors: True + # Stop downloading additional metadata/videos if it + # exists in your download archive + break_on_existing: True + # Stop downloading additional metadata/videos if it + # is out of your date range + break_on_reject: True + # Path to your YouTube cookies file to download 18+ restricted content + cookiefile: "/path/to/cookies/file.txt" + # Only download this number of videos/audio + max_downloads: 10 + # Download and use English title/description/etc YouTube metadata + extractor_args: + youtube: + lang: + - "en" + + +where each key is a ytdl argument. Include in the example are some popular ytdl_options. diff --git a/docs/source/code_reference/prebuilt_presets.rst b/docs/source/code_reference/prebuilt_presets.rst new file mode 100644 index 00000000..f91778c2 --- /dev/null +++ b/docs/source/code_reference/prebuilt_presets.rst @@ -0,0 +1,87 @@ +================ +Prebuilt Presets +================ +If you are ready to start downloading, see our +`examples directory <https://github.com/jmbannon/ytdl-sub/tree/master/examples>`_ +for ready-to-use configs and subscriptions. Read through them carefully before use. + +Helper Presets +============== + +By themselves, these presets won't get your media downloaded, but they're useful for condensing multi-line modifications to other presets into a single line through :ref:`guides/getting_started/advanced_configuration:preset inheritance`. + +best_video_quality +------------------ + +This preset: + ``1`` Is named ``best_video_quality`` + + ``2`` Searches for the best video and audio formats available to download + + ``4`` Uses the ``mp4`` extension when merging multiple formats into one file + +.. code-block:: yaml + :linenos: + + best_video_quality: + format: "bestvideo+bestaudio/best" + ytdl_options: + merge_output_format: "mp4" + +max_1080p +--------- + +This preset: + ``1`` Is named ``max_1080p`` + + ``2`` Searches for the best video (up to 1080p) and audio formats to download + + ``4`` Uses the ``mp4`` extension when merging multiple formats into one file + + +.. code-block:: yaml + :linenos: + + max_1080p: + format: "(bv*[height<=1080]+bestaudio/best[height<=1080])" + ytdl_options: + merge_output_format: "mp4" + +chunk_initial_download +---------------------- + +This preset: + ``1`` Is named ``chunk_initial_download`` + + ``3`` Downloads up to 20 videos per subscription per run + + ``4`` Reverses the subscription playlist, which starts from the beginning of the playlist in most cases + + ``5`` Will not stop downloading videos when it reaches a video that exists, but will skip the existing videos instead of redownloading them + + ``6`` Will stop downloading videos if a yt-dlp reject occurs + +.. code-block:: yaml + :linenos: + + chunk_initial_download: + ytdl_options: + max_downloads: 20 + playlistreverse: True + break_on_existing: False + break_on_reject: True + +Only Recent +----------- + +This preset: + ``1`` Is named "Only Recent" + + ``3`` Only attempts to download videos uploaded after ``today-{only_recent_date_range}``, where ``{only_recent_date_range}`` is set as an override variable in your subscription or a :ref:`child preset <guides/getting_started/advanced_configuration:preset inheritance>`. + +.. code-block:: yaml + :linenos: + + "Only Recent": + date_range: + after: "today-{only_recent_date_range}" \ No newline at end of file diff --git a/docs/source/code_reference/scripting_functions.rst b/docs/source/code_reference/scripting_functions.rst new file mode 100644 index 00000000..b462acf0 --- /dev/null +++ b/docs/source/code_reference/scripting_functions.rst @@ -0,0 +1,474 @@ + +Scripting Functions +=================== + +Array Functions +--------------- + +array +~~~~~ +``array(maybe_array: AnyArgument) -> Array`` + +Tries to cast an unknown variable type to an Array. + +array_apply +~~~~~~~~~~~ +``array_apply(array: Array, lambda_function: Lambda) -> Array`` + +Apply a lambda function on every element in the Array. + +array_at +~~~~~~~~ +``array_at(array: Array, idx: Integer) -> AnyArgument`` + +Return the element in the Array at index ``idx``. + +array_contains +~~~~~~~~~~~~~~ +``array_contains(array: Array, value: AnyArgument) -> Boolean`` + +Return True if the value exists in the Array. False otherwise. + +array_enumerate +~~~~~~~~~~~~~~~ +``array_enumerate(array: Array, lambda_function: LambdaTwo) -> Array`` + +Apply a lambda function on every element in the Array, where each arg +passed to the lambda function is ``idx, element`` as two separate args. + +array_extend +~~~~~~~~~~~~ +``array_extend(arrays: Array, ...) -> Array`` + +Combine multiple Arrays into a single Array. + +array_flatten +~~~~~~~~~~~~~ +``array_flatten(array: Array) -> Array`` + +Flatten any nested Arrays into a single-dimensional Array. + +array_index +~~~~~~~~~~~ +``array_index(array: Array, value: AnyArgument) -> Integer`` + +Return the index of the value within the Array if it exists. If it does not, it will +throw an error. + +array_product +~~~~~~~~~~~~~ +``array_product(arrays: Array, ...) -> Array`` + +Returns the Cartesian product of elements from different arrays + +array_reduce +~~~~~~~~~~~~ +``array_reduce(array: Array, lambda_reduce_function: LambdaReduce) -> AnyArgument`` + +Apply a reduce function on pairs of elements in the Array, until one element remains. +Executes using the left-most and reduces in the right direction. + +array_reverse +~~~~~~~~~~~~~ +``array_reverse(array: Array) -> Array`` + +Reverse an Array. + +array_size +~~~~~~~~~~ +``array_size(array: Array) -> Integer`` + +Returns the size of an Array. + +array_slice +~~~~~~~~~~~ +``array_slice(array: Array, start: Integer, end: Optional[Integer]) -> Array`` + +Returns the slice of the Array. + +Boolean Functions +----------------- + +and +~~~ +``and(values: AnyArgument, ...) -> Boolean`` + +``and`` operator. Returns True if all values evaluate to True. False otherwise. + +bool +~~~~ +``bool(value: AnyArgument) -> Boolean`` + +Cast any type to a Boolean. + +eq +~~ +``eq(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``==`` operator. Returns True if left == right. False otherwise. + +gt +~~ +``gt(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``>`` operator. Returns True if left > right. False otherwise. + +gte +~~~ +``gte(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``>=`` operator. Returns True if left >= right. False otherwise. + +lt +~~ +``lt(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``<`` operator. Returns True if left < right. False otherwise. + +lte +~~~ +``lte(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``<=`` operator. Returns True if left <= right. False otherwise. + +ne +~~ +``ne(left: AnyArgument, right: AnyArgument) -> Boolean`` + +``!=`` operator. Returns True if left != right. False otherwise. + +not +~~~ +``not(value: Boolean) -> Boolean`` + +``not`` operator. Returns the opposite of value. + +or +~~ +``or(values: AnyArgument, ...) -> Boolean`` + +``or`` operator. Returns True if any value evaluates to True. False otherwise. + +xor +~~~ +``xor(values: AnyArgument, ...) -> Boolean`` + +``^`` operator. Returns True if exactly one value is set to True. False otherwise. + +Conditional Functions +--------------------- + +if +~~ +``if(condition: Boolean, true: ReturnableArgumentA, false: ReturnableArgumentB) -> Union[ReturnableArgumentA, ReturnableArgumentB]`` + +Conditional ``if`` statement that returns the ``true`` or ``false`` parameter +depending on the ``condition`` value. + +if_passthrough +~~~~~~~~~~~~~~ +``if_passthrough(maybe_true_arg: ReturnableArgumentA, else_arg: ReturnableArgumentB) -> Union[ReturnableArgumentA, ReturnableArgumentB]`` + +Conditional ``if`` statement that returns the ``maybe_true_arg`` if it evaluates to True, +otherwise returns ``else_arg``. + +Date Functions +-------------- + +datetime_strftime +~~~~~~~~~~~~~~~~~ +``datetime_strftime(posix_timestamp: Integer, date_format: String) -> String`` + +Converts a posix timestamp to a date using strftime formatting. + +Error Functions +--------------- + +assert +~~~~~~ +``assert(value: ReturnableArgument, assert_message: String) -> ReturnableArgument`` + +Explicitly throw an error with the provided assert message if ``value`` evaluates to False. +If it evaluates to True, it will return ``value``. + +throw +~~~~~ +``throw(error_message: String) -> AnyArgument`` + +Explicitly throw an error with the provided error message. + +Json Functions +-------------- + +from_json +~~~~~~~~~ +``from_json(argument: String) -> AnyArgument`` + +Converts a JSON string into an actual type. + +Map Functions +------------- + +map +~~~ +``map(maybe_mapping: AnyArgument) -> Map`` + +Tries to cast an unknown variable type to a Map. + +map_apply +~~~~~~~~~ +``map_apply(mapping: Map, lambda_function: LambdaTwo) -> Array`` + +Apply a lambda function on the Map, where each arg +passed to the lambda function is ``key, value`` as two separate args. + +map_contains +~~~~~~~~~~~~ +``map_contains(mapping: Map, key: AnyArgument) -> Boolean`` + +Returns True if the key is in the Map. False otherwise. + +map_enumerate +~~~~~~~~~~~~~ +``map_enumerate(mapping: Map, lambda_function: LambdaThree) -> Array`` + +Apply a lambda function on the Map, where each arg +passed to the lambda function is ``idx, key, value`` as three separate args. + +map_get +~~~~~~~ +``map_get(mapping: Map, key: AnyArgument, default: Optional[AnyArgument]) -> AnyArgument`` + +Return ``key``'s value within the Map. If ``key`` does not exist, and ``default`` is +provided, it will return ``default``. Otherwise, will error. + +map_get_non_empty +~~~~~~~~~~~~~~~~~ +``map_get_non_empty(mapping: Map, key: AnyArgument, default: AnyArgument) -> AnyArgument`` + +Return ``key``'s value within the Map. If ``key`` does not exist or is an empty string, +return ``default``. Otherwise, will error. + +map_size +~~~~~~~~ +``map_size(mapping: Map) -> Integer`` + +Returns the size of a Map. + +Numeric Functions +----------------- + +add +~~~ +``add(values: Numeric, ...) -> Numeric`` + +``+`` operator. Returns the sum of all values. + +div +~~~ +``div(left: Numeric, right: Numeric) -> Numeric`` + +``/`` operator. Returns ``left / right``. + +float +~~~~~ +``float(value: AnyArgument) -> Float`` + +Cast to Float. + +int +~~~ +``int(value: AnyArgument) -> Integer`` + +Cast to Integer. + +max +~~~ +``max(values: Numeric, ...) -> Numeric`` + +Returns max of all values. + +min +~~~ +``min(values: Numeric, ...) -> Numeric`` + +Returns min of all values. + +mod +~~~ +``mod(left: Numeric, right: Numeric) -> Numeric`` + +``%`` operator. Returns ``left % right``. + +mul +~~~ +``mul(values: Numeric, ...) -> Numeric`` + +``*`` operator. Returns the product of all values. + +pow +~~~ +``pow(base: Numeric, exponent: Numeric) -> Numeric`` + +``**`` operator. Returns the exponential of the base and exponent value. + +sub +~~~ +``sub(values: Numeric, ...) -> Numeric`` + +``-`` operator. Subtracts all values from left to right. + +Regex Functions +--------------- + +regex_fullmatch +~~~~~~~~~~~~~~~ +``regex_fullmatch(regex: String, string: String) -> Array`` + +Checks for entire string to be a match. If a match exists, returns +the string as the first element of the Array. If there are capture groups, returns each +group as a subsequent element in the Array. + +regex_match +~~~~~~~~~~~ +``regex_match(regex: String, string: String) -> Array`` + +Checks for a match only at the beginning of the string. If a match exists, returns +the string as the first element of the Array. If there are capture groups, returns each +group as a subsequent element in the Array. + +regex_search +~~~~~~~~~~~~ +``regex_search(regex: String, string: String) -> Array`` + +Checks for a match anywhere in the string. If a match exists, returns +the string as the first element of the Array. If there are capture groups, returns each +group as a subsequent element in the Array. + +String Functions +---------------- + +capitalize +~~~~~~~~~~ +``capitalize(string: String) -> String`` + +Capitalize the first character in the string. + +concat +~~~~~~ +``concat(values: String, ...) -> String`` + +Concatenate multiple Strings into a single String. + +lower +~~~~~ +``lower(string: String) -> String`` + +Lower-case the entire String. + +pad +~~~ +``pad(string: String, length: Integer, char: String) -> String`` + +Pads the string to the given length + +pad_zero +~~~~~~~~ +``pad_zero(numeric: Numeric, length: Integer) -> String`` + +Pads a numeric with zeros to the given length + +replace +~~~~~~~ +``replace(string: String, old: String, new: String, count: Optional[Integer]) -> String`` + +Replace the ``old`` part of the String with the ``new``. Optionally only replace it +``count`` number of times. + +slice +~~~~~ +``slice(string: String, start: Integer, end: Optional[Integer]) -> String`` + +Returns the slice of the Array. + +string +~~~~~~ +``string(value: AnyArgument) -> String`` + +Cast to String. + +titlecase +~~~~~~~~~ +``titlecase(string: String) -> String`` + +Capitalize each word in the string. + +upper +~~~~~ +``upper(string: String) -> String`` + +Upper-case the entire String. + +Ytdl-Sub Functions +------------------ + +legacy_bracket_safety +~~~~~~~~~~~~~~~~~~~~~ +``legacy_bracket_safety(value: ReturnableArgument) -> ReturnableArgument`` + +ytdl-sub used to replace brackets ('{', '}') with unicode brackets ('｛', '｝') to not +interfere with its legacy variable scripting system. This function replicates that +behavior. + +sanitize +~~~~~~~~ +``sanitize(value: AnyArgument) -> String`` + +Sanitize a string using yt-dlp's ``sanitize_filename`` method to ensure it's safe to use +for file/directory names on any OS. + +sanitize_plex_episode +~~~~~~~~~~~~~~~~~~~~~ +``sanitize_plex_episode(string: String) -> String`` + +Sanitize a string using ``sanitize`` and replace numerics with their respective fixed-width +numbers. This is used to have Plex avoid scraping numbers like ``4x4`` as the +season and/or episode. + +to_date_metadata +~~~~~~~~~~~~~~~~ +``to_date_metadata(yyyymmdd: String) -> Map`` + +Takes a date in the form of YYYYMMDD and returns a Map containing: + +- date (String, YYYYMMDD) +- date_standardized (String, YYYY-MM-DD) +- year (Integer) +- month (Integer) +- day (Integer) +- year_truncated (String, YY from YY[YY]) +- month_padded (String) +- day_padded (String) +- year_truncated_reversed (Integer, 100 - year_truncated) +- month_reversed (Integer, 13 - month) +- month_reversed_padded (String) +- day_reversed (Integer, total_days_in_month + 1 - day) +- day_reversed_padded (String) +- day_of_year (Integer) +- day_of_year_padded (String, padded 3) +- day_of_year_reversed (Integer, total_days_in_year + 1 - day_of_year) +- day_of_year_reversed_padded (String, padded 3) + +to_native_filepath +~~~~~~~~~~~~~~~~~~ +``to_native_filepath(filepath: String) -> String`` + +Convert any unix-based path separators ('/') with the OS's native +separator. + +truncate_filepath_if_too_long +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +``truncate_filepath_if_too_long(filepath: String) -> String`` + +If a file-path is too long for the OS, this function will truncate it while preserving +the extension. diff --git a/docs/source/code_reference/subscriptions_yaml.rst b/docs/source/code_reference/subscriptions_yaml.rst index fe763a1d..82f65c4f 100644 --- a/docs/source/code_reference/subscriptions_yaml.rst +++ b/docs/source/code_reference/subscriptions_yaml.rst @@ -10,7 +10,7 @@ to define a ``subscription``: something we want to recurrently download such as channel or playlist. The only difference between a ``subscription`` and ``preset`` is that the subscription -must have all required fields and ``{variables}`` defined so it can perform a download. +must have all required fields and ``{override_variables}`` defined so it can perform a download. Below is an example that downloads a YouTube playlist: diff --git a/docs/source/conf.py b/docs/source/conf.py index b4a8f222..d8857931 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -1,9 +1,5 @@ # Configuration file for the Sphinx documentation builder. -import os -import sys -sys.path.insert(0, os.path.abspath('../../src')) - # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html @@ -19,13 +15,12 @@ release = "2023.12.15" # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ - "sphinx.ext.autodoc", - "sphinx.ext.autosectionlabel", - # "sphinx.ext.autosummary", - "sphinx.ext.extlinks", - "sphinx.ext.napoleon", - "sphinx_copybutton", - "sphinx_design", + "sphinx.ext.autodoc", + "sphinx.ext.autosectionlabel", + "sphinx.ext.extlinks", + "sphinx.ext.napoleon", + "sphinx_copybutton", + "sphinx_design", ] templates_path = ["_templates"] @@ -38,25 +33,25 @@ exclude_patterns = [] html_theme = "sphinx_book_theme" html_theme_options = { - "icon_links": [ - { - "name": "GitHub", - "url": "https://github.com/jmbannon/ytdl-sub", - "icon": "fa-brands fa-square-github", - "type": "fontawesome", - }, - { - "name": "Discord", - "url": "https://discord.gg/v8j9RAHb4k", - "icon": "https://img.shields.io/discord/994270357957648404?logo=Discord", - "type": "url", - }, - ], - "announcement": ( - "Migration to <a href='https://ytdl-sub--841.org.readthedocs.build/en/841/config.html#beautifying-subscriptions'>beautiful subscriptions</a> is now live" - ), - "navigation_depth": 10, - "show_toc_level": 10, + "icon_links": [ + { + "name": "GitHub", + "url": "https://github.com/jmbannon/ytdl-sub", + "icon": "fa-brands fa-square-github", + "type": "fontawesome", + }, + { + "name": "Discord", + "url": "https://discord.gg/v8j9RAHb4k", + "icon": "https://img.shields.io/discord/994270357957648404?logo=Discord", + "type": "url", + }, + ], + "announcement": ( + "Migration to <a href='https://ytdl-sub--841.org.readthedocs.build/en/841/config.html#beautifying-subscriptions'>beautiful subscriptions</a> is now live" + ), + "navigation_depth": 10, + "show_toc_level": 10, } html_static_path = ["_static"] @@ -65,8 +60,13 @@ html_static_path = ["_static"] # Make sure the all autosectionlabel targets are unique autosectionlabel_prefix_document = True +suppress_warnings = [ + 'autosectionlabel.*', +] -extlinks = {"yt-dlp": ("https://github.com/yt-dlp/yt-dlp/%s", "yt-dlp%s")} +extlinks = { + "yt-dlp": ("https://github.com/yt-dlp/yt-dlp/%s", "yt-dlp%s"), + } # -- Options for autodoc ---------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#configuration @@ -74,10 +74,10 @@ extlinks = {"yt-dlp": ("https://github.com/yt-dlp/yt-dlp/%s", "yt-dlp%s")} # Automatically extract typehints when specified and place them in # descriptions of the relevant function/method. autodoc_default_options = { - "autodoc_typehints_format": "short", - "autodoc_class_signature": "separated", - "add_module_names": False, - # "add_class_names": False, + "autodoc_typehints_format": "short", + "autodoc_class_signature": "separated", + "add_module_names": False, + # "add_class_names": False, } python_use_unqualified_type_names = True diff --git a/docs/source/guides/getting_started/advanced_configuration.rst b/docs/source/guides/getting_started/advanced_configuration.rst index c3ec8aae..42567bad 100644 --- a/docs/source/guides/getting_started/advanced_configuration.rst +++ b/docs/source/guides/getting_started/advanced_configuration.rst @@ -1,7 +1,7 @@ Advanced Configuration ====================== -If the :doc:`prebuilt presets </guides/getting_started/prebuilt_presets>` aren't suitable for your needs, you may want to set up an advanced configuration. +If the :doc:`prebuilt presets </code_reference/prebuilt_presets>` aren't suitable for your needs, you may want to set up an advanced configuration. Layout of a Config file ----------------------- @@ -17,5 +17,5 @@ The layout of the ``config.yaml`` file is relatively straightforward: plugin1_option1: value1 -Modifying an Existing Preset ----------------------------- +Preset Inheritance +------------------ diff --git a/docs/source/guides/getting_started/first_config.rst b/docs/source/guides/getting_started/first_config.rst index 34126aa9..40c47e32 100644 --- a/docs/source/guides/getting_started/first_config.rst +++ b/docs/source/guides/getting_started/first_config.rst @@ -22,9 +22,9 @@ Your first configuration will look pretty simple: The first two lines in this ``config.yaml`` file are the ``configuration``, and define the ``working_directory``, which is described near the bottom of :ref:`this section <guides/getting_started/index:quick overview of \`\`ytdl-sub\`\`>` -Line 4 begins the definition o your custom ``presets``, with line 5 being the name of your first custom ``preset``. +Line 4 begins the definition of your custom ``presets``, with line 5 being the name of your first custom ``preset``. -Lines 7 and 8 tell ``ytdl-sub`` which :doc:`/guides/getting_started/prebuilt_presets` to expand on; these ``presets`` already indicate that the downloaded files should be: +Lines 7 and 8 tell ``ytdl-sub`` which :doc:`/code_reference/prebuilt_presets` to expand on; these ``presets`` already indicate that the downloaded files should be: - in a format usable by, and with metadata accessible to, Jellyfin - sorted by upload date, and diff --git a/docs/source/guides/getting_started/index.rst b/docs/source/guides/getting_started/index.rst index 63145b5f..a6c7b043 100644 --- a/docs/source/guides/getting_started/index.rst +++ b/docs/source/guides/getting_started/index.rst @@ -3,7 +3,7 @@ Getting Started Now that you've completed your install of ``ytdl-sub``, it's time to get started. This is a 3-step process: -- Create your configuration file (if the :doc:`prebuilt_presets` don't fit your needs) +- Create your configuration file (if the :doc:`/code_reference/prebuilt_presets` don't fit your needs) - Create your subscription file - Automate starting YTDL-Sub @@ -57,19 +57,19 @@ Ready to Start? Now that you have installed ``ytdl-sub``, checked your skills, and gotten a bit of background on how ``ytdl-sub`` functions, read through the articles below to get started: -:doc:`Step 1: Initial Configuration <first_config>` +:doc:`Step 1: Initial Subscriptions <first_sub>` -:doc:`Step 2: Initial Subscriptions <first_sub>` +:doc:`Step 2: Your First Download <first_download>` -:doc:`Step 3: Your First Download <first_download>` +:doc:`Step 3: Automating Downloads <automating_downloads>` -:doc:`Step 4: Automating Downloads <automating_downloads>` +:doc:`Step 4: Initial Configuration <first_config>` :doc:`Advanced Configuration <advanced_configuration>` Other docs that may be of use: -:doc:`prebuilt_presets` +:doc:`/code_reference/prebuilt_presets` :doc:`examples` @@ -83,5 +83,4 @@ Other docs that may be of use: first_download automating_downloads advanced_configuration - prebuilt_presets examples \ No newline at end of file diff --git a/docs/source/guides/getting_started/prebuilt_presets.rst b/docs/source/guides/getting_started/prebuilt_presets.rst deleted file mode 100644 index 57ea0a45..00000000 --- a/docs/source/guides/getting_started/prebuilt_presets.rst +++ /dev/null @@ -1,5 +0,0 @@ -Example Configs -=============== -If you are ready to start downloading, see our -`examples directory <https://github.com/jmbannon/ytdl-sub/tree/master/examples>`_ -for ready-to-use configs and subscriptions. Read through them carefully before use. \ No newline at end of file