diff --git a/README.md b/README.md index 8231c085..4658f182 100644 --- a/README.md +++ b/README.md @@ -33,213 +33,163 @@ maximum flexibility while maintaining simplicity. #### Jellyfin ![jelly_mv](https://user-images.githubusercontent.com/10107080/182677256-43aeb029-0c3f-4648-9fd2-352b9666b262.PNG) -### SoundCloud Albums and Singles -#### MusicBee (any file or tag-based music players) +### SoundCloud Discography +#### Writes proper music-tags via beets API ![sc_mb](https://user-images.githubusercontent.com/10107080/182685415-06adf477-3dd3-475d-bbcd-53b0152b9f0a.PNG) ### Bandcamp Discography -#### Navidrome (any file or tag-based music servers) ![bc_nav](https://user-images.githubusercontent.com/10107080/212503861-1d8748e6-6f6d-4043-b543-84226cd1f662.png) - ## How it Works -`ytdl-sub` uses YAML configs to define a layout for how you want media to look -after it is downloaded. See our +`ytdl-sub` uses YAML files to define subscriptions. Each subscription imports _presets_ that +define how to handle and output media files. `ytdl-sub` comes packaged with many _prebuilt presets_ +that do the work of config-building, so you can start downloading immediately. + +```yaml +# subscriptions.yaml: + +# __preset__ is a place to define global overrides for all subscriptions +__preset__: + overrides: + # Root folder of all ytdl-sub TV Shows + tv_show_directory: "/tv_shows" + + # Root folder of all ytdl-sub Music + music_directory: "/music" + + # Root folder of all ytdl-sub Music Videos + music_video_directory: "/music_videos" + + # For 'Only Recent' preset, only keep vids uploaded in this range + date_range: "2months" + + # Pass any arg directly to yt-dlp's Python API + ytdl_options: + cookiefile: "/config/cookie.txt" + +################################################################### +# Subscriptions nested under this will use the +# `Plex TV Show by Date` preset. +# +# Can choose between: +# - Plex TV Show by Date: +# - Jellyfin TV Show by Date: +# - Kodi TV Show by Date: +Plex TV Show by Date: + + # Sets genre tag to "Documentaries" + = Documentaries: + "NOVA PBS": "https://www.youtube.com/@novapbs" + "National Geographic": "https://www.youtube.com/@NatGeo" + "Cosmos - What If": "https://www.youtube.com/playlist?list=PLZdXRHYAVxTJno6oFF9nLGuwXNGYHmE8U" + + # Sets genre tag to "Kids", "TV-Y" for content rating + = Kids | = TV-Y: + "Jake Trains": "https://www.youtube.com/@JakeTrains" + "Kids Toys Play": "https://www.youtube.com/@KidsToysPlayChannel" + + = Music: + # TV show subscriptions can support multiple urls and store in the same TV Show + "Rick Beato": + - "https://www.youtube.com/@RickBeato" + - "https://www.youtube.com/@rickbeato240" + + # Set genre tag to "News", use `Only Recent` preset to only store videos uploaded recently + = News | Only Recent: + "BBC News": "https://www.youtube.com/@BBCNews" + +################################################################### +# Subscriptions nested under these will use the various prebuilt +# music presets +YouTube Releases: + = Jazz: # Sets genre tag to "Jazz" + "Thelonious Monk": "https://www.youtube.com/@theloniousmonk3870/releases" + +YouTube Full Albums: + = Lofi: + "Game Chops": "https://www.youtube.com/playlist?list=PLBsm_SagFMmdWnCnrNtLjA9kzfrRkto4i" + +SoundCloud Discography: + = Chill Hop: + "UKNOWY": "https://soundcloud.com/uknowymunich" + = Synthwave: + "Lazerdiscs Records": "https://soundcloud.com/lazerdiscsrecords" + "Earmake": "https://soundcloud.com/earmake" + +Bandcamp: + = Lofi: + "Emily Hopkins": "https://emilyharpist.bandcamp.com/" + +################################################################### +# Can choose between: +# - Plex Music Videos: +# - Jellyfin Music Videos: +# - Kodi Music Videos: +"Plex Music Videos": + = Pop: # Sets genre tag to "Pop" + "Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc" + "Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E" +``` + +All of this can be downloaded and ready to import to your favorite player +using the command +```commandline +ytdl-sub sub subscriptions.yaml +``` +See our +[example subscriptions](https://github.com/jmbannon/ytdl-sub/tree/master/examples) +for more detailed examples and use-cases. + +### Output +After `ytdl-sub` runs, the end result will download and format the files into something ready +to be consumed by your favorite media player/server. +``` +tv_shows/ + Jake Trains/ + Season 2021/ + s2021.e031701 - Pattys Day Video-thumb.jpg + s2021.e031701 - Pattys Day Video.mp4 + s2021.e031701 - Pattys Day Video.nfo + s2021.e031702 - Second Pattys Day Video-thumb.jpg + s2021.e031702 - Second Pattys Day Video.mp4 + s2021.e031702 - Second Pattys Day Video.nfo + Season 2022/ + s2022.e122501 - Merry Christmas-thumb.jpg + s2022.e122501 - Merry Christmas.mp4 + s2022.e122501 - Merry Christmas.nfo + poster.jpg + fanart.jpg + tvshow.nfo + +music/ + Artist/ + [2022] Some Single/ + 01 - Some Single.mp3 + folder.jpg + [2023] Latest Album/ + 01 - Track Title.mp3 + 02 - Another Track.mp3 + folder.jpg + +music_videos/ + Elton John/ + Elton John - Rocketman.jpg + Elton John - Rocketman.mp4 +``` + +## Custom Configs +Any part of this process is modifiable by using custom configs. See our [walk-through guide](https://github.com/jmbannon/ytdl-sub/wiki) -on how to get started. Ready-to-use +on how to build your first config from scratch. Ready-to-use [example configurations](https://github.com/jmbannon/ytdl-sub/tree/master/examples) can be found here alongside our [readthedocs](https://ytdl-sub.readthedocs.io/en/latest/config.html#) -for detailed information on config fields. - - -### Config -The `config.yaml` defines how our downloads will look. For this example, let us -download YouTube channels and generate metadata to look like TV shows using -ytdl-sub's prebuilt presets. No additional plugins or programs are needed for -Kodi, Jellyfin, Plex, or Emby to recognize your downloads. This can also be -used to download any yt-dlp supported URL, including YouTube playlists, Bitchute channels, etc. - - -```yaml -# Set the working directory which will be used to stage downloads -# before placing them in your desired output directory. -configuration: - working_directory: '.ytdl-sub-downloads' - -# Presets are where you create 'sub-configs' that can can be -# merged together to dictate what is downloaded, how to format it, -# and what metadata to generate. -presets: - - # Let us create a preset called `only_recent_videos` that will - # only download recent videos in the last 2 months. - only_recent_videos: - - # Use the `date_range` plugin to specify ytdl-sub to only - # download videos after today MINUS {download_range}, which - # is an override variable that we can alter per channel. - date_range: - after: "today-{download_range}" - - # Any yt-dlp argument can be passed via ytdl-sub. Let us set - # yt-dlp's `break_on_reject` to True to stop downloading after - # any video is rejected. Videos will be rejected if they are - # uploaded after our {download_range}. - ytdl_options: - break_on_reject: True - - # Deletes any videos uploaded after {download_range}. - output_options: - keep_files_after: "today-{download_range}" - - # Set the override variable {download_range} to 2months. - # This will serve as our default value. We can override - # this per channel or in a child preset. - overrides: - download_range: "2months" - - #################################################################### - - # Now let us create a preset that downloads videos and formats - # as TV shows. - tv_show: - - # Presets can inherit all attributes from other presets. Our - # `tv_show` preset will inherit these presets built into ytdl-sub. - preset: - # Let us specify all the TV show by date presets to support all - # players. You only need to specify one, but this ensures - # compatibility with all players. - - "kodi_tv_show_by_date" - - "jellyfin_tv_show_by_date" - - "plex_tv_show_by_date" - # Now we choose a preset that defines how our seasons and - # episode numbers look. - - "season_by_year__episode_by_month_day" - - # Set override variables that will be applicable to all downloads - # in main presets. - overrides: - tv_show_directory: "/tv_shows" # Replace with desired directory - -``` - -### Subscriptions -The `subscriptions.yaml` file is where we define content to download using -presets in the `config.yaml`. Each subscription can overwrite any field used -in a preset. -```yaml -# The name of our subscription. Let us create one to download -# ALL of Rick A's videos -rick_a: - # Inherit our `tv_show` preset we made above - preset: - - "tv_show" - - # Set override variables to set the channel URL and the - # name we want to give the TV show. - overrides: - tv_show_name: "Rick A" - url: "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw" - -# Let us make another subscription that will only download Rick A's -# video's in the last 2 weeks. -rick_a_recent: - # Inherit our `tv_show` AND `only_recent_videos` preset - # Bottom-most presets take precedence. - preset: - - "tv_show" - - "only_recent_videos" - - # Set override variables for this subscription. Modify the - # `download_range` to only download and keep 2 weeks' worth - # of videos. - overrides: - tv_show_name: "Rick A" - url: "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw" - download_range: "2weeks" - -``` -The download can now be performed using: -```shell -ytdl-sub sub subscriptions.yaml -``` -To preview what your output files before doing any downloads, you can dry run using: -```shell -ytdl-sub --dry-run sub subscriptions.yaml -``` - -### One-time Download -There are things we will only want to download once and never again. Anything -you can define in a subscription can be defined using CLI arguments. This -example is equivalent to the subscription example above: -```shell -ytdl-sub dl \ - --preset "tv_show" \ - --overrides.tv_show_name "Rick A" \ - --overrides.url "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw" -``` - -#### Download Aliases -In the `config.yaml`, we can define aliases to make `dl` commands shorter. -```yaml -configuration: - dl_aliases: - tv: "--preset tv_show" - name: "--overrides.tv_show_name" - url: "--overrides.url" -``` -The above command can now be shortened to -```shell -ytdl-sub dl \ - --tv \ - --name "Rick A" \ - --url "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw" -``` - -### Output -After `ytdl-sub` runs, the end result will download and format the channel -files into something ready to be consumed by your favorite media player or -server. -``` -/path/to/tv_shows/Rick Aß - /Season 2021 - s2021.e0317 - Pattys Day Video-thumb.jpg - s2021.e0317 - Pattys Day Video.mp4 - s2021.e0317 - Pattys Day Video.nfo - /Season 2022 - s2022.e1225 - Merry Christmas-thumb.jpg - s2022.e1225 - Merry Christmas.mp4 - s2022.e1225 - Merry Christmas.nfo - poster.jpg - fanart.jpg - tvshow.nfo -``` - -### Beyond TV Shows -The above example made heavy-use of `ytdl-sub` prebuilt presets and hides many -features that are offered. `ytdl-sub` strives to support _any_ use case that first requires -a download via yt-dlp. Use `ytdl-sub` to download, format, and convert media for your media -player to recognize downloads as: -- Movies -- TV shows - - From a single channel or playlist - - From multiple channels or playlists - - From individual videos -- Extracted audio as podcasts -- Music videos -- Music, including: - - Individual songs - - Albums - - Discographies +for detailed information on all config fields. ## Installation `ytdl-sub` can be installed on the following platforms. - [Docker Compose](https://ytdl-sub.readthedocs.io/en/latest/install.html#docker-compose_) - - [with CPU passthrough](https://ytdl-sub.readthedocs.io/en/latest/install.html#cpu-passthrough) - - [with GPU passthrough](https://ytdl-sub.readthedocs.io/en/latest/install.html#nvidia-gpu-passthrough) - [Docker CLI](https://ytdl-sub.readthedocs.io/en/latest/install.html#docker) - [Windows](https://ytdl-sub.readthedocs.io/en/latest/install.html#windows) - [Unraid](https://ytdl-sub.readthedocs.io/en/latest/install.html#unraid) @@ -249,6 +199,15 @@ player to recognize downloads as: - [Local Install](https://ytdl-sub.readthedocs.io/en/latest/install.html#local-install) - [Local Docker Build](https://ytdl-sub.readthedocs.io/en/latest/install.html#local-docker-build) +### Docker Installation +Docker installs can be either headless or use the GUI image, which comprises +[LSIO's](https://www.linuxserver.io/) +[code-server](https://hub.docker.com/r/linuxserver/code-server) +Docker image with `ytdl-sub` preinstalled. This is the recommended way to use ``ytdl-sub``. + +![image](https://github.com/jmbannon/ytdl-sub/assets/10107080/c2aac8a1-5443-4345-b438-be4b17187c80) + + ## Contributing There are many ways to contribute, even without coding. Please take a look in our [GitHub Issues](https://github.com/jmbannon/ytdl-sub/issues) to submit a feature request, or diff --git a/docs/config.rst b/docs/config.rst index da379040..ca8aa980 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -423,8 +423,9 @@ Source Variables Override Variables ------------------ -.. autoclass:: ytdl_sub.config.preset_options.OverridesVariables() +.. autoclass:: ytdl_sub.entries.variables.override_variables.OverrideVariables() :members: + :member-order: bysource ------------------------------------------------------------------------------- diff --git a/docs/install.rst b/docs/install.rst index 9b2fbc28..bea0667b 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -3,22 +3,62 @@ Install ``ytdl-sub`` can be installed on the following platforms. .. contents:: - :depth: 2 + :depth: 3 All installations require a 64-bit CPU. 32-bit is not supported. Docker Compose -------------- -The ytdl-sub docker image uses -`Linux Server's `_ -`base alpine image `_ -It looks, feels, and operates like other LinuxServer images. This is the -recommended way to use ytdl-sub. +The ytdl-sub Docker images use +`LSIO-based images `_ +and installs ytdl-sub on top. There are a few flavors to choose from. -The docker image is intended to be used as a console. For automating -``subscriptions.yaml`` downloads to pull new media, see +For automating ``subscriptions.yaml`` downloads to pull new media, see `this guide `_ -on how set up a cron job in the docker container. +on how set up a cron job in any of the docker containers. + +GUI +^^^^ + +The GUI image uses LSIO's +`code-server `_ +for its base image. More info on other code-server environment variables +can be found within its documentation. This is the recommended way to use ``ytdl-sub``. + +After starting, code-server will be running at http://localhost:8443/ + +.. code-block:: yaml + + services: + ytdl-sub: + image: ghcr.io/jmbannon/ytdl-sub-gui:latest + container_name: ytdl-sub + environment: + - PUID=1000 + - PGID=1000 + - TZ=America/Los_Angeles + volumes: + - :/config + - :/tv_shows # optional + - :/movies # optional + - :/music_videos # optional + - :/music # optional + ports: + - 8443:8443 + restart: unless-stopped + +Headless +^^^^^^^^^^ + +The headless image uses LSIO's +`baseimage-alpine `_ +for its base image. With this image, ``ytdl-sub`` is meant to be ran from console +via exec'ing into the image using the command: + +.. code-block:: bash + + docker exec -u abc -it ytdl-sub /bin/bash + .. code-block:: yaml @@ -39,46 +79,51 @@ on how set up a cron job in the docker container. - :/music # optional restart: unless-stopped -CPU Passthrough -^^^^^^^^^^^^^^^^^^^^^^ -For CPU passthrough, you must use the ``ytdl-sub`` Ubuntu version with the following additions: +Passthrough +^^^^^^^^^^^ +For CPU or GPU passthrough, you must use either the GUI image or the headless Ubuntu image +``ghcr.io/jmbannon/ytdl-sub:ubuntu-latest``. + +The docker-compose examples use the GUI image. + +CPU +____ .. code-block:: yaml services: ytdl-sub: - image: ghcr.io/jmbannon/ytdl-sub:ubuntu-latest + image: ghcr.io/jmbannon/ytdl-sub-gui:latest container_name: ytdl-sub environment: - PUID=1000 - PGID=1000 - TZ=America/Los_Angeles - - DOCKER_MODS=linuxserver/mods:universal-cron volumes: - :/config - :/tv_shows # optional - :/movies # optional - :/music_videos # optional - :/music # optional + ports: + - 8443:8443 devices: - /dev/dri:/dev/dri # CPU passthrough restart: unless-stopped -Nvidia GPU Passthrough -^^^^^^^^^^^^^^^^^^^^^^ -For GPU passthrough, you must use the ``ytdl-sub`` Ubuntu version with the following additions: +GPU +____ .. code-block:: yaml services: ytdl-sub: - image: ghcr.io/jmbannon/ytdl-sub:ubuntu-latest + image: ghcr.io/jmbannon/ytdl-sub-gui:latest container_name: ytdl-sub environment: - PUID=1000 - PGID=1000 - TZ=America/Los_Angeles - - DOCKER_MODS=linuxserver/mods:universal-cron - NVIDIA_DRIVER_CAPABILITIES=all # Nvidia ENV args - NVIDIA_VISIBLE_DEVICES=all volumes: @@ -87,6 +132,8 @@ For GPU passthrough, you must use the ``ytdl-sub`` Ubuntu version with the follo - :/movies # optional - :/music_videos # optional - :/music # optional + ports: + - 8443:8443 deploy: resources: reservations: @@ -103,14 +150,14 @@ Docker -e PUID=1000 \ -e PGID=1000 \ -e TZ=America/Los_Angeles \ - -e DOCKER_MODS=linuxserver/mods:universal-cron \ + -p 8443:8443 \ -v :/config \ -v :/tv_shows \ -v :/movies \ -v :/music_videos \ -v :/music \ --restart unless-stopped \ - ghcr.io/jmbannon/ytdl-sub:latest + ghcr.io/jmbannon/ytdl-sub-gui:latest Windows -------------- diff --git a/src/ytdl_sub/config/preset_options.py b/src/ytdl_sub/config/preset_options.py index c159fa4d..ff07ce92 100644 --- a/src/ytdl_sub/config/preset_options.py +++ b/src/ytdl_sub/config/preset_options.py @@ -9,7 +9,7 @@ from yt_dlp.utils import sanitize_filename from ytdl_sub.config.defaults import DEFAULT_DOWNLOAD_ARCHIVE_NAME from ytdl_sub.entries.entry import Entry -from ytdl_sub.subscriptions.utils import SUBSCRIPTION_NAME +from ytdl_sub.entries.variables.override_variables import SUBSCRIPTION_NAME from ytdl_sub.utils.exceptions import ValidationException from ytdl_sub.validators.file_path_validators import OverridesStringFormatterFilePathValidator from ytdl_sub.validators.file_path_validators import StringFormatterFileNameValidator diff --git a/src/ytdl_sub/entries/variables/override_variables.py b/src/ytdl_sub/entries/variables/override_variables.py new file mode 100644 index 00000000..8a885f83 --- /dev/null +++ b/src/ytdl_sub/entries/variables/override_variables.py @@ -0,0 +1,57 @@ +SUBSCRIPTION_NAME = "subscription_name" +SUBSCRIPTION_VALUE = "subscription_value" + + +class OverrideVariables: + @classmethod + def subscription_name(cls) -> str: + """ + Name of the subscription + """ + return SUBSCRIPTION_NAME + + @classmethod + def subscription_value(cls) -> str: + """ + For subscriptions in the form of + + .. code-block:: yaml + + "Subscription Name": "https://..." + + ``subscription_value`` gets set to ``https://...``. + """ + return SUBSCRIPTION_VALUE + + @classmethod + def subscription_indent_i(cls, index: int) -> str: + """ + For subscriptions in the form of + + .. code-block:: yaml + + Preset | = Indent Value 1: + = Indent Value 2: + "Subscription Name": "https://..." + + ``subscription_indent_1`` and ``subscription_indent_2`` get set to + ``Indent Value 1`` and ``Indent Value 2``. + """ + return f"subscription_indent_{index + 1}" + + @classmethod + def subscription_value_i(cls, index: int) -> str: + """ + For subscriptions in the form of + + .. code-block:: yaml + + "Subscription Name": + - "https://url1.com/..." + - "https://url2.com/..." + + ``subscription_value_1`` and ``subscription_value_2`` get set to ``https://url1.com/...`` + and ``https://url2.com/...``. Note that ``subscription_value_1`` also gets set to + ``subscription_value``. + """ + return f"subscription_value_{index + 1}" diff --git a/src/ytdl_sub/subscriptions/subscription_validators.py b/src/ytdl_sub/subscriptions/subscription_validators.py index 007b2b9a..678a9758 100644 --- a/src/ytdl_sub/subscriptions/subscription_validators.py +++ b/src/ytdl_sub/subscriptions/subscription_validators.py @@ -9,10 +9,9 @@ from typing import final from ytdl_sub.config.config_file import ConfigFile from ytdl_sub.config.preset_options import Overrides -from ytdl_sub.subscriptions.utils import SUBSCRIPTION_NAME -from ytdl_sub.subscriptions.utils import SUBSCRIPTION_VALUE -from ytdl_sub.subscriptions.utils import subscription_indent_variable_name -from ytdl_sub.subscriptions.utils import subscription_list_variable_name +from ytdl_sub.entries.variables.override_variables import SUBSCRIPTION_NAME +from ytdl_sub.entries.variables.override_variables import SUBSCRIPTION_VALUE +from ytdl_sub.entries.variables.override_variables import OverrideVariables from ytdl_sub.validators.string_formatter_validators import DictFormatterValidator from ytdl_sub.validators.validators import DictValidator from ytdl_sub.validators.validators import StringListValidator @@ -33,7 +32,7 @@ class SubscriptionOutput(Validator, ABC): indent overrides to merge with the preset dict's overrides """ return { - subscription_indent_variable_name(i): self._indent_overrides[i] + OverrideVariables.subscription_indent_i(i): self._indent_overrides[i] for i in range(len(self._indent_overrides)) } @@ -159,7 +158,9 @@ class SubscriptionListValuesValidator(SubscriptionLeafValidator, StringListValid if idx == 0: self._overrides_to_add[SUBSCRIPTION_VALUE] = list_value.value - self._overrides_to_add[subscription_list_variable_name(index=idx)] = list_value.value + self._overrides_to_add[ + OverrideVariables.subscription_value_i(index=idx) + ] = list_value.value class SubscriptionWithOverridesValidator(SubscriptionLeafValidator, DictFormatterValidator): diff --git a/src/ytdl_sub/subscriptions/utils.py b/src/ytdl_sub/subscriptions/utils.py index 3e2c9a00..b82620a0 100644 --- a/src/ytdl_sub/subscriptions/utils.py +++ b/src/ytdl_sub/subscriptions/utils.py @@ -1,33 +1,2 @@ -SUBSCRIPTION_NAME = "subscription_name" -SUBSCRIPTION_VALUE = "subscription_value" - # Key used in configs, should delete at some point SUBSCRIPTION_VALUE_CONFIG_KEY = "subscription_value" - - -def subscription_indent_variable_name(index: int) -> str: - """ - Parameters - ---------- - index - 0th-based index - - Returns - ------- - subscription_index_i, where i is 1-based index - """ - return f"subscription_indent_{index + 1}" - - -def subscription_list_variable_name(index: int) -> str: - """ - Parameters - ---------- - index - 0th-based index - - Returns - ------- - subscription_value_i, where i is 1-based index - """ - return f"subscription_value_{index + 1}"