danvics/ytdl-sub
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Use URLs instead of IDs for sources (#70)

Browse source 
* youtube video url

* WIP, channel /user/ and /c/ should be valid

* unit tests passing

* soundcloud artist url

* soundcloud username

* break configs with name change

* Update all yamls with urls

* update all docs

* more negative url tests
This commit is contained in:
Jesse Bannon 2022-06-05 16:19:08 -07:00 committed by GitHub
parent e111ae53f5
commit a79fcaa25b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
16 changed files with 429 additions and 283 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
README.md
View file

@ -38,7 +38,7 @@ download, like a YouTube channel to pull new videos. It looks something like:
john_smith_channel:
preset: "yt_channel_as_tv"
youtube:
channel_id: "UCsvn_Po0SmunchJYtttWpOxMg"
channel_url: "https://youtube.com/channe/UCsvn_Po0SmunchJYtttWpOxMg"
overrides:
tv_show_name: "John Smith Vlogs"
```
@ -53,9 +53,9 @@ a one-hit wonder music video. Anything you can define in a subscription can be
defined using CLI arguments. This example is equivalent to the subscription
example above:
```shell
ytdl-sub dl \
--preset "yt_channel_as_tv" \
--youtube.channel_id "UCsvn_Po0SmunchJYtttWpOxMg" \
ytdl-sub dl \
--preset "yt_channel_as_tv" \
--youtube.channel_url "https://youtube.com/channel/UCsvn_Po0SmunchJYtttWpOxMg" \
--overrides.tv_show_name "John Smith Vlogs"
```

113
config.yaml
View file

@ -1,113 +0,0 @@
# ytdl-sub needs a temporary place to download files. Choose that path here
configuration:
working_directory: 'tmp'
presets:
soundcloud_with_id3_tags:
soundcloud:
download_strategy: "albums_and_singles"
skip_premiere_tracks: True
ytdl_options:
format: 'bestaudio[ext=mp3]'
output_options:
output_directory: "/tmp/{artist_sanitized}"
file_name: "[{album_year}] {album_sanitized}/{track_number_padded} - {title_sanitized}.{ext}"
thumbnail_name: "[{album_year}] {album_sanitized}/folder.jpg"
maintain_download_archive: True
music_tags:
tags:
artist: "{artist}"
albumartist: "{artist}"
title: "{title}"
album: "{album}"
track: "{track_number}"
year: "{album_year}"
genre: "{genre}"
yt_music_video:
# A single YouTube video is our source/download strategy. However, this
# can be overwritten to download music videos from a "playlist", as we
# will see in a preset below
youtube:
download_strategy: "video"
# For advanced YTDL users only.
# It is wise to leave ignoreerrors=True to avoid errors for things
# like age-restricted videos in case you have not set your cookie.
ytdl_options:
ignoreerrors: True
# For each video downloaded, set the file and thumbnail name here.
# We set both with {music_video_name}, which is a variable we define in
# the overrides section further below to represent consistent naming format.
output_options:
output_directory: "{music_video_directory}/{artist_sanitized}"
file_name: "{music_video_name}.{ext}"
thumbnail_name: "{music_video_name}.jpg"
# For each video downloaded, add a music video NFO file for it. Populate it
# with tags that Kodi will read and use to display it in the music or music
# videos section.
nfo_tags:
nfo_name: "{music_video_name}.nfo"
nfo_root: "musicvideo"
tags:
artist: "{artist}"
title: "{title}"
album: "Music Videos"
year: "{upload_year}"
# Overrides is a section where we can define our own variables, and use them in
# any other section. We define our music video directory and episode file name
# here, which gets reused above for the video, thumbnail, and NFO file.
overrides:
music_video_directory: "path/to/Music Videos"
music_video_name: "{artist_sanitized} - {title_sanitized}"
# It is not always ideal to download all of an artist's music videos.
# Maybe you only like one song of theirs. We can reuse our preset above
# to download a single video instead.
yt_music_video_playlist:
preset: "yt_music_video"
youtube:
download_strategy: "playlist"
# Setting maintain_download_archive=True is generally a good thing to enable
# with playlists and channels because it will store previously downloaded
# video ids to tell YTDL not to re-download them on a successive invocation.
output_options:
maintain_download_archive: True
# Setting break_on_existing=True is also a good thing anytime you are using
# a download archive, because it will tell the downloader to stop trying
# to download videos that have already been downloaded.
ytdl_options:
break_on_existing: True
yt_channel:
youtube:
download_strategy: "channel"
channel_avatar_path: "poster.jpg"
channel_banner_path: "fanart.jpg"
ytdl_options:
ignoreerrors: True
output_options:
file_name: "{output_file_name}.{ext}"
thumbnail_name: "{output_file_name}.jpg"
nfo_tags:
nfo_name: "{output_file_name}.nfo"
nfo_root: "episodedetails"
tags:
title: "{title}"
season: "{upload_year}"
episode: "{upload_month}{upload_day_padded}"
plot: "{description}"
year: "{upload_year}"
aired: "{upload_date_standardized}"
output_directory_nfo_tags:
nfo_name: "tvshow.nfo"
nfo_root: "tvshow"
tags:
title: "{tv_show_name}"
overrides:
output_file_name: "Season {upload_year}/s{upload_year}.e{upload_month_padded}{upload_day_padded} - {title_sanitized}"

2
docker/root/defaults/subscriptions.yaml
View file

@ -1,7 +1,7 @@
rammstein_music_videos:
preset: "yt_music_video_playlist"
youtube:
playlist_id: "PLVTLbc6i-h_iuhdwUfuPDLFLXG2QQnz-x"
playlist_url: "https://youtube.com/playlist?list=PLVTLbc6i-h_iuhdwUfuPDLFLXG2QQnz-x"
overrides:
artist: "Rammstein"

4
docs/config.rst
View file

@ -190,13 +190,13 @@ Below is an example that downloads YouTube videos:
my_subscription_name:
preset: "playlist_preset_ex"
youtube:
playlist_id: "UCsvn_Po0SmunchJYtttWpOxMg"
playlist_url: "https://youtube.com/playlist?list=UCsvn_Po0SmunchJYtttWpOxMg"
overrides:
playlist_name: "diy-playlist"
Our preset ``playlist_preset_ex`` uses the `YouTube Playlist`_ download strategy, and defines two
custom variables: ``{output_directory}`` and ``{playlist_name}``. The subscription sets
the `parent preset`_ to ``playlist_preset_ex``, and must define the ``playlist_id`` field and
the `parent preset`_ to ``playlist_preset_ex``, and must define the ``playlist_url`` field and
the ``{playlist_name}`` variable since the preset did not.
-------------------------------------------------------------------------------

16
examples/kodi_music_videos_subscriptions.yaml
View file

@ -14,9 +14,9 @@ john_smith:
# config example "yt_music_video", so set that here.
preset: "yt_music_video_playlist"
# Since our preset download strategy is set to 'playlist', set the playlist id.
# Since our preset download strategy is set to 'playlist', set the playlist url
youtube:
playlist_id: "UCsvn_Po0SmunchJYtttWpOxMg"
playlist_url: "https://youtube.com/playlist?list=UCsvn_Po0SmunchJYtttWpOxMg"
# Overrides can be defined per-subscription. If you noticed, we used {artist}
# and {artist_sanitized} in our "yt_music_video" preset. We intended to reserve
@ -39,21 +39,21 @@ john_smith:
# The only difference between this example and the one above is
# - preset
# Use the music video preset, not the playlist preset
# - youtube.video_id
# The id of the video to download
# - youtube.video_url
# The video url to download
#
# Of course, defining yaml configuration to download a single video once
# and never again seems weird. Instead, we can perform this download via
# command:
#
# ytdl-sub dl \
# --preset "yt_music_video" \
# --youtube.video_id "QhY6r6oAErg" \
# ytdl-sub dl \
# --preset "yt_music_video" \
# --youtube.video_url "https://youtube.com/watch?v=QhY6r6oAErg" \
# --overrides.artist "John Smith and the Instrument Players"
#
john_smith_one_hit_wonder:
preset: "yt_music_video"
youtube:
video_id: "QhY6r6oAErg"
video_url: "https://youtube.com/watch?v=QhY6r6oAErg"
overrides:
artist: "John Smith and the Instrument Players"

8
examples/kodi_tv_shows_subscriptions.yaml
View file

@ -13,10 +13,10 @@ john_smith_archive:
# entire YouTube channel is called "yt_channel_as_tv", so set that here.
preset: "yt_channel_as_tv"
# Our download strategy was Youtube channels. Define the channel id here
# Our download strategy was Youtube channels. Define the channel url here
# in our subscription.
youtube:
channel_id: "UCsvn_Po0SmunchJYtttWpOxMg"
channel_url: "https://youtube.com/channel/UCsvn_Po0SmunchJYtttWpOxMg"
# Overrides can be defined per-subscription. If you noticed, we used
# {tv_show_name} and {tv_show_name_sanitized} in our "yt_channel_as_tv"
@ -34,7 +34,7 @@ john_smith_archive:
john_smith_recent_archive:
preset: "yt_channel_as_tv__recent"
youtube:
channel_id: "UCsvn_Po0SmunchJYtttWpOxMg"
channel_url: "https://youtube.com/channel/UCsvn_Po0SmunchJYtttWpOxMg"
overrides:
tv_show_name: "John /\ Smith"
@ -46,6 +46,6 @@ john_smith_recent_archive:
john_smith_rolling_archive:
preset: "yt_channel_as_tv__only_recent"
youtube:
channel_id: "UCsvn_Po0SmunchJYtttWpOxMg"
channel_url: "https://youtube.com/channel/UCsvn_Po0SmunchJYtttWpOxMg"
overrides:
tv_show_name: "John /\ Smith"

6
src/ytdl_sub/config/config_file.py
View file

@ -51,19 +51,19 @@ class ConfigOptions(StrictDictValidator):
configuration:
dl_aliases:
mv: "--preset yt_music_video"
v: "--youtube.video_id"
v: "--youtube.video_url"
Simplifies
.. code-block:: bash
ytdl-sub dl --preset yt_music_video --youtube.video_id a1b2c3
ytdl-sub dl --preset yt_music_video --youtube.video_url youtube.com/watch?v=a1b2c3
to
.. code-block:: bash
ytdl-sub dl --mv --v a1b2c3
ytdl-sub dl --mv --v youtube.com/watch?v=a1b2c3
"""
if self._dl_aliases:
return self._dl_aliases.dict

33
src/ytdl_sub/downloaders/soundcloud_downloader.py
View file

@ -8,8 +8,8 @@ from ytdl_sub.downloaders.downloader import Downloader
from ytdl_sub.downloaders.downloader import DownloaderValidator
from ytdl_sub.entries.soundcloud import SoundcloudAlbum
from ytdl_sub.entries.soundcloud import SoundcloudTrack
from ytdl_sub.validators.url_validator import SoundcloudUsernameUrlValidator
from ytdl_sub.validators.validators import BoolValidator
from ytdl_sub.validators.validators import StringValidator
###############################################################################
# Abstract Soundcloud downloader + options
@ -61,27 +61,22 @@ class SoundcloudDownloader(
}
@classmethod
def artist_url(cls, artist_name: str) -> str:
"""Returns full artist url"""
return f"https://soundcloud.com/{artist_name}"
@classmethod
def artist_albums_url(cls, artist_name: str) -> str:
def artist_albums_url(cls, artist_url: str) -> str:
"""
Returns
-------
Full artist album url
"""
return cls.artist_url(artist_name) + "/albums"
return f"{artist_url}/albums"
@classmethod
def artist_tracks_url(cls, artist_name: str) -> str:
def artist_tracks_url(cls, artist_url: str) -> str:
"""
Returns
-------
Full artist tracks url
"""
return cls.artist_url(artist_name) + "/tracks"
return f"{artist_url}/tracks"
###############################################################################
@ -102,24 +97,26 @@ class SoundcloudAlbumsAndSinglesDownloadOptions(SoundcloudDownloaderOptions):
soundcloud:
# required
download_strategy: "albums_and_singles"
username: soundcloud_username_from_url
url: "soundcloud.com/username"
# optional
skip_premiere_tracks: True
"""
_required_keys = {"username"}
_required_keys = {"url"}
def __init__(self, name, value):
super().__init__(name, value)
self._username = self._validate_key(key="username", validator=StringValidator)
self._url = self._validate_key(
key="url", validator=SoundcloudUsernameUrlValidator
).username_url
@property
def username(self) -> str:
def url(self) -> str:
"""
Required. The Soundcloud username found in the url of their page.
Required. The Soundcloud user's url, i.e. ``soundcloud.com/the_username``
"""
return self._username.value
return self._url
class SoundcloudAlbumsAndSinglesDownloader(
@ -177,8 +174,8 @@ class SoundcloudAlbumsAndSinglesDownloader(
"""
Soundcloud subscription to download albums and tracks as singles.
"""
artist_albums_url = self.artist_albums_url(artist_name=self.download_options.username)
artist_tracks_url = self.artist_tracks_url(artist_name=self.download_options.username)
artist_albums_url = self.artist_albums_url(artist_url=self.download_options.url)
artist_tracks_url = self.artist_tracks_url(artist_url=self.download_options.url)
album_entry_dicts = self.extract_info_via_info_json(url=artist_albums_url)
tracks_entry_dicts = self.extract_info_via_info_json(url=artist_tracks_url)

82
src/ytdl_sub/downloaders/youtube_downloader.py
View file

@ -17,7 +17,9 @@ from ytdl_sub.utils.logger import Logger
from ytdl_sub.utils.thumbnail import convert_url_thumbnail
from ytdl_sub.validators.date_range_validator import DateRangeValidator
from ytdl_sub.validators.string_formatter_validators import OverridesStringFormatterValidator
from ytdl_sub.validators.validators import StringValidator
from ytdl_sub.validators.url_validator import YoutubeChannelUrlValidator
from ytdl_sub.validators.url_validator import YoutubePlaylistUrlValidator
from ytdl_sub.validators.url_validator import YoutubeVideoUrlValidator
logger = Logger.get()
@ -61,49 +63,41 @@ class YoutubeVideoDownloaderOptions(YoutubeDownloaderOptions):
.. code-block:: yaml
presets:
my_example_preset:
example_preset:
youtube:
# required
download_strategy: "video"
video_id: "VMAPTo7RVDo"
video_url: "youtube.com/watch?v=VMAPTo7RVDo"
CLI usage:
.. code-block:: bash
ytdl-sub dl --preset "my_example_preset" --youtube.video_id "VMAPTo7RVDo"
ytdl-sub dl --preset "example_preset" --youtube.video_url "youtube.com/watch?v=VMAPTo7RVDo"
"""
_required_keys = {"video_id"}
_required_keys = {"video_url"}
def __init__(self, name, value):
super().__init__(name, value)
self._video_id = self._validate_key("video_id", StringValidator)
self._video_url = self._validate_key("video_url", YoutubeVideoUrlValidator).video_url
@property
def video_id(self) -> str:
def video_url(self) -> str:
"""
Required. The ID of the video. Looks like the ``VMAPTo7RVDo`` from
``youtube.com/watch?v=VMAPTo7RVDo``.
Required. The url of the video, i.e. ``youtube.com/watch?v=VMAPTo7RVDo``.
"""
return self._video_id.value
return self._video_url
class YoutubeVideoDownloader(YoutubeDownloader[YoutubeVideoDownloaderOptions, YoutubeVideo]):
downloader_options_type = YoutubeVideoDownloaderOptions
downloader_entry_type = YoutubeVideo
@classmethod
def video_url(cls, video_id: str) -> str:
"""Returns full video url"""
return f"https://youtube.com/watch?v={video_id}"
def download(self) -> List[YoutubeVideo]:
"""Download a single Youtube video"""
video_url = self.video_url(video_id=self.download_options.video_id)
entry_dict = self.extract_info(url=video_url)
entry_dict = self.extract_info(url=self.download_options.video_url)
return [YoutubeVideo(entry_dict=entry_dict, working_directory=self.working_directory)]
@ -124,22 +118,24 @@ class YoutubePlaylistDownloaderOptions(YoutubeDownloaderOptions):
youtube:
# required
download_strategy: "playlist"
playlist_id: "UCsvn_Po0SmunchJYtttWpOxMg"
playlist_url: "https://www.youtube.com/playlist?list=UCsvn_Po0SmunchJYtttWpOxMg"
"""
_required_keys = {"playlist_id"}
_required_keys = {"playlist_url"}
def __init__(self, name, value):
super().__init__(name, value)
self._playlist_id = self._validate_key("playlist_id", StringValidator)
self._playlist_url = self._validate_key(
"playlist_url", YoutubePlaylistUrlValidator
).playlist_url
@property
def playlist_id(self) -> str:
def playlist_url(self) -> str:
"""
Required. The playlist's ID. Looks like "UCsvn_Po0SmunchJYtttWpOxMg"
from ``https://www.youtube.com/playlist?list=UCsvn_Po0SmunchJYtttWpOxMg``.
Required. The playlist's url, i.e.
``https://www.youtube.com/playlist?list=UCsvn_Po0SmunchJYtttWpOxMg``.
"""
return self._playlist_id.value
return self._playlist_url
class YoutubePlaylistDownloader(
@ -148,19 +144,13 @@ class YoutubePlaylistDownloader(
downloader_options_type = YoutubePlaylistDownloaderOptions
downloader_entry_type = YoutubePlaylistVideo
@classmethod
def playlist_url(cls, playlist_id: str) -> str:
"""Returns full playlist url"""
return f"https://youtube.com/playlist?list={playlist_id}"
def download(self) -> List[YoutubePlaylistVideo]:
"""
Downloads all videos in a Youtube playlist
"""
playlist_url = self.playlist_url(playlist_id=self.download_options.playlist_id)
playlist_videos: List[YoutubePlaylistVideo] = []
entry_dicts = self.extract_info_via_info_json(url=playlist_url)
entry_dicts = self.extract_info_via_info_json(url=self.download_options.playlist_url)
for entry_dict in entry_dicts:
if entry_dict.get("extractor") == "youtube":
playlist_videos.append(
@ -189,7 +179,7 @@ class YoutubeChannelDownloaderOptions(YoutubeDownloaderOptions, DateRangeValidat
youtube:
# required
download_strategy: "channel"
channel_id: "UCsvn_Po0SmunchJYtttWpOxMg"
channel_url: "UCsvn_Po0SmunchJYtttWpOxMg"
# optional
channel_avatar_path: "poster.jpg"
channel_banner_path: "fanart.jpg"
@ -197,13 +187,15 @@ class YoutubeChannelDownloaderOptions(YoutubeDownloaderOptions, DateRangeValidat
after: "today-2weeks"
"""
_required_keys = {"channel_id"}
_required_keys = {"channel_url"}
_optional_keys = {"before", "after", "channel_avatar_path", "channel_banner_path"}
def __init__(self, name, value):
YoutubeDownloaderOptions.__init__(self, name, value)
DateRangeValidator.__init__(self, name, value)
self._channel_id = self._validate_key("channel_id", StringValidator)
self._channel_url = self._validate_key(
"channel_url", YoutubeChannelUrlValidator
).channel_url
self._channel_avatar_path = self._validate_key_if_present(
"channel_avatar_path", OverridesStringFormatterValidator
)
@ -212,15 +204,13 @@ class YoutubeChannelDownloaderOptions(YoutubeDownloaderOptions, DateRangeValidat
)
@property
def channel_id(self) -> str:
def channel_url(self) -> str:
"""
Required. The channel's ID. Not to be confused with the username. It should look something
like ``UCsvn_Po0SmunchJYOWpOxMg`` from
``https://www.youtube.com/channel/UCsvn_Po0SmunchJYOWpOxMg``. You can get this by opening a
video and clicking on the channel's avatar image to take you to their channel, then check
the url.
Required. The channel's url, i.e.
``https://www.youtube.com/channel/UCsvn_Po0SmunchJYOWpOxMg``. URLs with ``/username`` or
``/c`` are valid to use.
"""
return self._channel_id.value
return self._channel_url
@property
def channel_avatar_path(self) -> Optional[OverridesStringFormatterValidator]:
@ -256,16 +246,10 @@ class YoutubeChannelDownloader(YoutubeDownloader[YoutubeChannelDownloaderOptions
)
self.channel: Optional[YoutubeChannel] = None
@classmethod
def channel_url(cls, channel_id: str) -> str:
"""Returns full channel url"""
return f"https://youtube.com/channel/{channel_id}"
def download(self) -> List[YoutubeVideo]:
"""
Downloads all videos from a channel
"""
channel_url = self.channel_url(channel_id=self.download_options.channel_id)
channel_videos: List[YoutubeVideo] = []
ytdl_options_overrides = {}
@ -275,7 +259,7 @@ class YoutubeChannelDownloader(YoutubeDownloader[YoutubeChannelDownloaderOptions
ytdl_options_overrides["daterange"] = source_date_range
entry_dicts = self.extract_info_via_info_json(
ytdl_options_overrides=ytdl_options_overrides, url=channel_url
ytdl_options_overrides=ytdl_options_overrides, url=self.download_options.channel_url
)
for entry_dict in entry_dicts:

195
src/ytdl_sub/validators/url_validator.py Normal file
View file

@ -0,0 +1,195 @@
from typing import Any
from typing import Optional
from urllib.parse import parse_qs
from urllib.parse import urlparse
from ytdl_sub.validators.validators import StringValidator
class YoutubeVideoUrlValidator(StringValidator):
_expected_value_type_name = "Youtube video url"
@classmethod
def _get_video_id(cls, url: str) -> Optional[str]:
"""
Examples:
- https://youtu.be/SA2iWivDJiE
- https://www.youtube.com/watch?v=_oPAwA_Udwc&feature=feedu
- https://www.youtube.com/embed/SA2iWivDJiE
- https://www.youtube.com/v/SA2iWivDJiE?version=3&hl=en_US
"""
# If the url doesn't contain youtube or youtu.be, assume it is invalid
if "youtube.com" not in url and "youtu.be" not in url:
return None
# If https:// is not present, urlparse will not work
if not url.startswith("https://"):
url = f"https://{url}"
query = urlparse(url)
if query.hostname in ("youtu.be", "www.youtu.be"):
return query.path[1:]
if query.hostname in ("youtube.com", "www.youtube.com"):
if query.path == "/watch":
parsed_q = parse_qs(query.query)
if "v" in parsed_q:
return parsed_q["v"][0]
if query.path[:7] == "/embed/":
return query.path.split("/")[2]
if query.path[:3] == "/v/":
return query.path.split("/")[2]
return None
def __init__(self, name: str, value: Any):
super().__init__(name, value)
self._video_id = self._get_video_id(value)
if not self._video_id:
raise self._validation_exception(f"'{value}' is not a valid Youtube video url.")
@property
def video_url(self) -> str:
"""
Returns
-------
Full video URL
"""
return f"https://youtube.com/watch?v={self._video_id}"
class YoutubePlaylistUrlValidator(StringValidator):
_expected_value_type_name = "Youtube playlist url"
@classmethod
def _get_playlist_id(cls, url: str) -> Optional[str]:
"""
Examples:
- https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc
"""
# If the url doesn't contain youtube, assume it is invalid
if "youtube.com" not in url:
return None
# If https:// is not present, urlparse will not work
if not url.startswith("https://"):
url = f"https://{url}"
query = urlparse(url)
if query.hostname in ("youtube.com", "www.youtube.com"):
if query.path == "/playlist":
parsed_q = parse_qs(query.query)
if "list" in parsed_q:
return parsed_q["list"][0]
return None
def __init__(self, name: str, value: Any):
super().__init__(name, value)
self._playlist_id = self._get_playlist_id(value)
if not self._playlist_id:
raise self._validation_exception(f"'{value}' is not a valid Youtube playlist url.")
@property
def playlist_url(self) -> str:
"""
Returns
-------
Full playlist URL
"""
return f"https://youtube.com/playlist?list={self._playlist_id}"
class YoutubeChannelUrlValidator(StringValidator):
_expected_value_type_name = "Youtube channel url"
@classmethod
def _get_channel_url(cls, url: str) -> Optional[str]:
"""
Examples:
- https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw
- https://www.youtube.com/user/username
- https://www.youtube.com/c/channel_name
"""
# If the url doesn't contain youtube, assume it is invalid
if "youtube.com" not in url:
return None
# If https:// is not present, urlparse will not work
if not url.startswith("https://"):
url = f"https://{url}"
query = urlparse(url)
if query.hostname in ("youtube.com", "www.youtube.com"):
if any(query.path.startswith(qpath) for qpath in ("/channel/", "/user/", "/c/")):
# Ensure there is "/path/id".split('/') = ['', 'path', 'id'] at least three
query_path_split = query.path.split("/")
if len(query_path_split) < 3 or len(query_path_split[2]) == 0:
return None
return f"https://youtube.com{query.path}"
return None
def __init__(self, name: str, value: Any):
super().__init__(name, value)
self._channel_url = self._get_channel_url(value)
if not self._channel_url:
raise self._validation_exception(f"'{value}' is not a valid Youtube channel url.")
@property
def channel_url(self) -> str:
"""
Returns
-------
Full channel URL
"""
return self._channel_url
class SoundcloudUsernameUrlValidator(StringValidator):
_expected_value_type_name = "Soundcloud username url"
@classmethod
def _get_channel_url(cls, url: str) -> Optional[str]:
"""
Examples:
- https://www.soundcloud.com/artist_name
"""
# If the url doesn't contain soundcloud, assume it is invalid
if "soundcloud.com" not in url:
return None
# If https:// is not present, urlparse will not work
if not url.startswith("https://"):
url = f"https://{url}"
query = urlparse(url)
if query.hostname in ("soundcloud.com", "www.soundcloud.com"):
if len(query.path) > 2: # /artist_name
username = query.path[1:].split("/")[0] # strip extra paths or slashes
username = username.split("?")[0] # strip extra arguments
return f"https://soundcloud.com/{username}"
return None
def __init__(self, name: str, value: Any):
super().__init__(name, value)
self._username_url = self._get_channel_url(value)
if not self._username_url:
raise self._validation_exception(f"'{value}' is not a valid Soundcloud username url.")
@property
def username_url(self) -> str:
"""
Returns
-------
Full artist URL
"""
return self._username_url

71
subscriptions.yaml
View file

@ -1,71 +0,0 @@
#ALISON:
# preset: "soundcloud_with_id3_tags"
# soundcloud:
# username: alis_on
# output_options:
# output_directory: "/tmp/ALISON"
# overrides:
# artist: "A.L.I.S.O.N."
# genre: "Synthwave / Electronic"
#
#delorra:
# preset: "soundcloud_with_id3_tags"
# soundcloud:
# username: delorra
# output_options:
# output_directory: "/tmp/DeLorra"
# overrides:
# artist: "DeLorra"
# genre: "Synthwave / Electronic"
#
#bl00dwave:
# preset: "soundcloud_with_id3_tags"
# soundcloud:
# username: bl00dwave
# overrides:
# artist: "bl00dwave"
# genre: "Synthwave / Electronic"
#
tom_petty:
preset: "music_videos"
youtube:
download_strategy: "playlist"
playlist_id: PLoopXDarluPBnuxs4PTC55Sc_2ShAXC0i
output_options:
output_directory: "/tmp/Tom Petty"
overrides:
artist: "Tom Petty and the Heartbreakers"
#
#rammstein:
# preset: "music_videos"
# youtube:
# download_strategy: "playlist"
# playlist_id: "PLVTLbc6i-h_iuhdwUfuPDLFLXG2QQnz-x"
# output_options:
# output_directory: "/tmp/Rammstein"
# overrides:
# artist: "Rammstein"
#gogo_video_test:
# preset: "music_videos"
# youtube:
# download_strategy: "video"
# video_id: "2RHTiXvELNg"
# output_options:
# output_directory: "/tmp/gogos"
# overrides:
# artist: "The Gogos"
#recent_channel_test:
# preset: "yt_channel"
# ytdl_options:
# break_on_reject: True
# youtube:
# channel_id: "UCsvn_Po0SmunchJYOWpOxMg"
# after: today-13days
# output_options:
# output_directory: "/tmp/dunkey"
# keep_files:
# after: today-3days
# overrides:
# tv_show_name: "Youtube: Dunkey"

2
tests/e2e/soundcloud/test_soundcloud_discography.py
View file

@ -27,7 +27,7 @@ def config(config_path):
def subscription_dict(output_directory, subscription_name):
return {
"preset": "sc_discography",
"soundcloud": {"username": "jessebannon"},
"soundcloud": {"url": "https://soundcloud.com/jessebannon"},
# override the output directory with our fixture-generated dir
"output_options": {"output_directory": output_directory + "/{artist_sanitized}"},
# download the worst format so it is fast

2
tests/e2e/youtube/test_channel_as_kodi_tv_show.py
View file

@ -28,7 +28,7 @@ def config(config_path):
def subscription_dict(output_directory, subscription_name):
return {
"preset": "yt_channel_as_tv",
"youtube": {"channel_id": "UCcRSMoQqXc_JrBZRHDFGbqA"},
"youtube": {"channel_url": "https://youtube.com/channel/UCcRSMoQqXc_JrBZRHDFGbqA"},
# override the output directory with our fixture-generated dir
"output_options": {"output_directory": str(Path(output_directory) / subscription_name)},
# download the worst format so it is fast

4
tests/e2e/youtube/test_playlist_as_kodi_music_videos.py
View file

@ -28,7 +28,7 @@ def config(config_path):
def subscription_dict(output_directory, subscription_name):
return {
"preset": "yt_music_video_playlist",
"youtube": {"playlist_id": "PL5BC0FC26BECA5A35"},
"youtube": {"playlist_url": "https://youtube.com/playlist?list=PL5BC0FC26BECA5A35"},
# override the output directory with our fixture-generated dir
"output_options": {"output_directory": output_directory},
# download the worst format so it is fast
@ -93,7 +93,7 @@ def single_video_subscription_dict(subscription_dict):
subscription_dict,
{
"preset": "yt_music_video",
"youtube": {"video_id": "HKTNxEqsN3Q"},
"youtube": {"video_url": "https://youtube.com/watch?v=HKTNxEqsN3Q"},
},
)

16
tests/unit/cli/test_download_args_parser.py
View file

@ -43,25 +43,29 @@ class TestDownloadArgsParser:
"aliases, cmd, expected_sub_dict",
[
(
{"mv": "--preset yt_music_video", "v": "--youtube.video_id"},
{"mv": "--preset yt_music_video", "v": "--youtube.video_url"},
"dl --mv --v 123abc",
{"preset": "yt_music_video", "youtube": {"video_id": "123abc"}},
{"preset": "yt_music_video", "youtube": {"video_url": "123abc"}},
),
(
{
"ch": "--preset yt_channel",
"c": "--youtube.channel_id",
"c": "--youtube.channel_url",
"name": "--overrides.tv_name",
"concert": "--overrides.genre 'Some Genre Name'",
},
"dl --ch --c 123abc --name 'Sweet TV Show' --concert",
"dl --ch --c https://youtube.com/channel/123abc --name 'Sweet TV Show' --concert",
{
"preset": "yt_channel",
"youtube": {"channel_id": "123abc"},
"youtube": {"channel_url": "https://youtube.com/channel/123abc"},
"overrides": {"tv_name": "Sweet TV Show", "genre": "Some Genre Name"},
},
),
(None, "dl --youtube.playlist_id 123abc", {"youtube": {"playlist_id": "123abc"}}),
(
None,
"dl --youtube.playlist_url https://youtube.com/playlist?list=123abc",
{"youtube": {"playlist_url": "https://youtube.com/playlist?list=123abc"}},
),
],
)
def test_successful_args(self, config_options_generator, aliases, cmd, expected_sub_dict):

150
tests/unit/validators/test_url_validator.py Normal file
View file

@ -0,0 +1,150 @@
import re
import pytest
from ytdl_sub.utils.exceptions import ValidationException
from ytdl_sub.validators.url_validator import SoundcloudUsernameUrlValidator
from ytdl_sub.validators.url_validator import YoutubeChannelUrlValidator
from ytdl_sub.validators.url_validator import YoutubePlaylistUrlValidator
from ytdl_sub.validators.url_validator import YoutubeVideoUrlValidator
class TestYoutubeVideoUrlValidator:
@pytest.mark.parametrize(
"url",
[
"youtu.be/dQw4w9WgXcQ",
"www.youtu.be/dQw4w9WgXcQ",
"https://youtu.be/dQw4w9WgXcQ",
"https://www.youtu.be/dQw4w9WgXcQ",
"https://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=feedu",
"https://www.youtube.com/embed/dQw4w9WgXcQ",
"https://www.youtube.com/v/dQw4w9WgXcQ?version=3&amp;hl=en_US",
],
)
def test_youtube_video_url_validator_success(self, url):
video_url = YoutubeVideoUrlValidator(name="unit test", value=url).video_url
assert video_url == "https://youtube.com/watch?v=dQw4w9WgXcQ"
@pytest.mark.parametrize(
"bad_url",
[
"utube.com/watch?v=sdfsadf",
"youtube.com/nope?v=sdfsadf",
"youtube.com",
"youtube.com/watch",
"youtube.com/watch?v=",
"youtu.be/",
"youtu.be",
],
)
def test_youtube_video_url_validator_fail(self, bad_url):
expected_error_msg = f"'{bad_url}' is not a valid Youtube video url."
with pytest.raises(ValidationException, match=re.escape(expected_error_msg)):
YoutubeVideoUrlValidator(name="unit test", value=bad_url)
class TestYoutubePlaylistUrlValidator:
@pytest.mark.parametrize(
"url",
[
"youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
"www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
"https://youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
"https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
],
)
def test_youtube_playlist_url_validator_success(self, url):
playlist_url = YoutubePlaylistUrlValidator(name="unit test", value=url).playlist_url
assert (
playlist_url == "https://youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
)
@pytest.mark.parametrize(
"bad_url",
[
"youpoop.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
"youtube.com/playlistlist=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc",
"youtube.com",
"youtube.com/playlist/asdfsdfsdf",
"youtube.com/playlist?list=",
"youtube.com/playlist",
],
)
def test_youtube_playlist_url_validator_fail(self, bad_url):
expected_error_msg = f"'{bad_url}' is not a valid Youtube playlist url."
with pytest.raises(ValidationException, match=re.escape(expected_error_msg)):
YoutubePlaylistUrlValidator(name="unit test", value=bad_url)
class TestYoutubeChannelUrlValidator:
@pytest.mark.parametrize(
"url, expected_url",
[
(
"https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
"https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
),
(
"https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
"https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
),
(
"youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
"https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
),
(
"www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
"https://youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
),
(
"https://www.youtube.com/c/RickastleyCoUkOfficial",
"https://youtube.com/c/RickastleyCoUkOfficial",
),
(
"https://www.youtube.com/user/videogamedunkey",
"https://youtube.com/user/videogamedunkey",
),
],
)
def test_youtube_channel_url_validator_success(self, url, expected_url):
channel_url = YoutubeChannelUrlValidator(name="unit test", value=url).channel_url
assert channel_url == expected_url
@pytest.mark.parametrize(
"bad_url",
[
"www.youtube.com/cha/UCuAXFkgsw1L7xaCfnd5JJOw",
"www.nopetube.com/channel/asdfdsf",
"www.youtube.com/channel/",
"www.youtube.com/channel",
"www.youtube.com/channell/asdfasdf",
],
)
def test_youtube_channel_url_validator_fail(self, bad_url):
expected_error_msg = f"'{bad_url}' is not a valid Youtube channel url."
with pytest.raises(ValidationException, match=re.escape(expected_error_msg)):
YoutubeChannelUrlValidator(name="unit test", value=bad_url)
class TestSoundcloudUsernameUrlValidator:
@pytest.mark.parametrize(
"url",
[
"soundcloud.com/poop",
"www.soundcloud.com/poop",
"https://soundcloud.com/poop",
"https://www.soundcloud.com/poop",
"https://www.soundcloud.com/poop/albums",
"https://www.soundcloud.com/poop?link=clipboard_share",
],
)
def test_soundcloud_artist_url_validator_success(self, url):
username_url = SoundcloudUsernameUrlValidator(name="unit test", value=url).username_url
assert username_url == "https://soundcloud.com/poop"
@pytest.mark.parametrize("bad_url", ["soundcloud.com", "soundnope.lol", "soundcloud.comm/"])
def test_youtube_playlist_url_validator_fail(self, bad_url):
expected_error_msg = f"'{bad_url}' is not a valid Soundcloud username url."
with pytest.raises(ValidationException, match=re.escape(expected_error_msg)):
SoundcloudUsernameUrlValidator(name="unit test", value=bad_url)