[FEATURE] Subscription nesting (#747)

Closes Feature Request: https://github.com/jmbannon/ytdl-sub/issues/743

Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:

```
tv_show:
  only_recent:
    [News]:
      "Breaking News": "https://www.youtube.com/@SomeBreakingNews"

  [Tech]:
    "Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
```
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:

```
"Breaking News":
  preset:
    - "tv_show"
    - "only_recent"

  overrides:
    subscription_indent_1: "News"
    subscription_name: "Breaking News"
    subscription_value: "https://www.youtube.com/@SomeBreakingNews"

"Two Minute Papers":
  preset:
    - "tv_show"

  overrides:
    subscription_indent_1: "Tech"
    subscription_name: "Two Minute Papers"     subscription_value: "https://www.youtube.com/@TwoMinutePapers"
```

You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things you want configurable like so in your
parent preset:

```
presets:
  tv_show_name:
    overrides:
      tv_show_name: "{subscription_name}"
      url: "{subscription_value}"
      genre: "{subscription_indent_1}"
```

NOTE!!!
Changing your subscription name from 'legacy' subscriptions will need their download archive file migrated to a new name. A future update will assist in this migration process. Either wait until then or, if you consider yourself a ytdl-sub expert, keep a backup of your subscription file before migrating.
This commit is contained in:
Jesse Bannon 2023-10-02 16:07:48 -07:00 committed by GitHub
parent 3e41e0f59e
commit c255f40348
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 455 additions and 77 deletions

View file

@ -29,7 +29,7 @@ and subscriptions.
.. autoclass:: ytdl_sub.config.config_validator.ConfigOptions()
:members:
:member-order: bysource
:exclude-members: persist_logs, experimental
:exclude-members: subscription_value, persist_logs, experimental
persist_logs
""""""""""""
@ -299,8 +299,66 @@ custom variables: ``{output_directory}``, ``{playlist_name}``, and ``{url}``. Th
the `parent preset`_ to ``playlist_preset_ex``, and must define the variables ``{playlist_name}``
and ``{url}`` since the preset did not.
.. _beautifying subscriptions:
Beautifying Subscriptions
^^^^^^^^^^^^^^^^^^^^^^^^^
Subscriptions support using presets as keys, and using keys to set override variables as values.
For example:
.. code-block:: yaml
:caption: subscription.yaml
tv_show:
only_recent:
[News]:
"Breaking News": "https://www.youtube.com/@SomeBreakingNews"
[Tech]:
"Two Minute Papers": "https://www.youtube.com/@TwoMinutePapers"
Will create two subscriptions named "Breaking News" and "Two Minute Papers", equivalent to:
.. code-block:: yaml
"Breaking News":
preset:
- "tv_show"
- "only_recent"
overrides:
subscription_indent_1: "News"
subscription_name: "Breaking News"
subscription_value: "https://www.youtube.com/@SomeBreakingNews"
"Two Minute Papers":
preset:
- "tv_show"
overrides:
subscription_indent_1: "Tech"
subscription_name: "Two Minute Papers"
subscription_value: "https://www.youtube.com/@TwoMinutePapers"
You can provide as many parent presets in the form of keys, and subscription indents as ``[keys]``.
This can drastically simplify subscription definitions by setting things like so in your
parent preset:
.. code-block:: yaml
presets:
tv_show_name:
overrides:
tv_show_name: "{subscription_name}"
url: "{subscription_value}"
genre: "{subscription_indent_1}"
.. _subscription value:
File Preset
^^^^^^^^^^^
NOTE: This is deprecated in favor of using the method in :ref:`beautifying subscriptions`.
You can apply a preset to all subscriptions in the ``subscription.yaml`` file
by using the file-wide ``__preset__``:
@ -318,10 +376,11 @@ by using the file-wide ``__preset__``:
This ``subscription.yaml`` is equivalent to the one above it because all
subscriptions automatically set ``__preset__`` as a `parent preset`_.
.. _subscription value:
Subscription Value
^^^^^^^^^^^^^^^^^^^
NOTE: This is deprecated in favor of using the method in :ref:`beautifying subscriptions`.
With a clever config and use of ``__preset__``, your subscriptions can typically boil
down to a name and url. You can set ``__value__`` to the name of an override variable,
and use the override variable ``subscription_name`` to achieve one-liner subscriptions.

View file

@ -1,6 +1,14 @@
Deprecation Notices
===================
Oct 2023
--------
subscription preset and value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The use of ``__value__`` will go away in Dec 2023 in favor of the method found in
:ref:`beautifying subscriptions`. ``__preset__`` will still be supported for the time being.
July 2023
---------

View file

@ -1,4 +1,5 @@
import copy
from typing import Any
from typing import Dict
from typing import List
from typing import Optional
@ -6,13 +7,17 @@ from typing import Optional
from ytdl_sub.config.config_file import ConfigFile
from ytdl_sub.config.preset import Preset
from ytdl_sub.subscriptions.subscription_download import SubscriptionDownload
from ytdl_sub.subscriptions.subscription_validators import SubscriptionValidator
from ytdl_sub.utils.exceptions import ValidationException
from ytdl_sub.utils.logger import Logger
from ytdl_sub.utils.yaml import load_yaml
from ytdl_sub.validators.validators import LiteralDictValidator
FILE_PRESET_APPLY_KEY = "__preset__"
FILE_SUBSCRIPTION_VALUE_KEY = "__value__"
logger = Logger.get("subscription")
class Subscription(SubscriptionDownload):
@classmethod
@ -68,15 +73,23 @@ class Subscription(SubscriptionDownload):
def _maybe_get_subscription_value(
cls, config: ConfigFile, subscription_dict: Dict
) -> Optional[str]:
subscription_value_key: Optional[str] = config.config_options.subscription_value
if FILE_SUBSCRIPTION_VALUE_KEY in subscription_dict:
if not isinstance(subscription_dict[FILE_SUBSCRIPTION_VALUE_KEY], str):
raise ValidationException(
f"Using {FILE_SUBSCRIPTION_VALUE_KEY} in a subscription"
f"must be a string that corresponds to an override variable"
)
return subscription_dict[FILE_SUBSCRIPTION_VALUE_KEY]
return config.config_options.subscription_value # can be None
subscription_value_key = subscription_dict[FILE_SUBSCRIPTION_VALUE_KEY]
if subscription_value_key is not None:
logger.warning(
"Using %s in a subscription will eventually be deprecated in favor of writing "
"to the override variable `subscription_value`. Please update by Dec 2023.",
FILE_SUBSCRIPTION_VALUE_KEY,
)
return subscription_value_key
@classmethod
def from_file_path(cls, config: ConfigFile, subscription_path: str) -> List["Subscription"]:
@ -121,39 +134,22 @@ class Subscription(SubscriptionDownload):
config = copy.deepcopy(config)
config.presets.dict[FILE_PRESET_APPLY_KEY] = file_preset.dict
for subscription_key, subscription_object in subscription_dict.items():
subscriptions_dict: Dict[str, Any] = {
key: obj
for key, obj in subscription_dict.items()
if key not in [FILE_PRESET_APPLY_KEY, FILE_SUBSCRIPTION_VALUE_KEY]
}
# Skip file preset or value
if subscription_key in [FILE_PRESET_APPLY_KEY, FILE_SUBSCRIPTION_VALUE_KEY]:
continue
# If the subscription obj is just a string, set it to the override variable
# defined in FILE_SUBSCRIPTION_VALUE_KEY
if isinstance(subscription_object, str) and file_subscription_value:
subscription_object = {"overrides": {file_subscription_value: subscription_object}}
elif isinstance(subscription_object, dict):
pass
elif isinstance(subscription_object, str) and not file_subscription_value:
raise ValidationException(
f"Subscription {subscription_key} is a string, but the subscription value "
f"is not set to an override variable"
)
else:
raise ValidationException(
f"Subscription {subscription_key} should be in the form of a preset"
)
# If it has file_preset, inject it as a parent preset
if has_file_preset:
parent_preset = subscription_object.get("preset", [])
# Preset can be a single string
if isinstance(parent_preset, str):
parent_preset = [parent_preset]
# If it's not a string or list, it will fail downstream
if isinstance(parent_preset, list):
subscription_object["preset"] = parent_preset + [FILE_PRESET_APPLY_KEY]
subscriptions_dicts = SubscriptionValidator(
name="",
value=subscriptions_dict,
config=config,
presets=[FILE_PRESET_APPLY_KEY] if has_file_preset else [],
indent_overrides=[],
subscription_value=file_subscription_value,
).subscription_dicts()
for subscription_key, subscription_object in subscriptions_dicts.items():
subscriptions.append(
cls.from_dict(
config=config,

View file

@ -0,0 +1,208 @@
import copy
from abc import ABC
from abc import abstractmethod
from typing import Dict
from typing import List
from typing import Optional
from ytdl_sub.config.config_file import ConfigFile
from ytdl_sub.config.preset_options import Overrides
from ytdl_sub.validators.validators import DictValidator
from ytdl_sub.validators.validators import StringListValidator
from ytdl_sub.validators.validators import StringValidator
from ytdl_sub.validators.validators import Validator
def subscription_indent_variable_name(index: int) -> str:
"""
Parameters
----------
index
0th-based index
Returns
-------
subscription_index_i, where i is 1-based index
"""
return f"subscription_indent_{index + 1}"
def subscription_value_variable_name() -> str:
"""
Returns
-------
The override variable name containing the subscription value if present
"""
return "subscription_value"
def maybe_indent_override_value(value: str) -> Optional[str]:
"""
Returns
-------
Value if it is an overide [Value]. None otherwise.
"""
if value.startswith("[") and value.endswith("]"):
return value[1:-1]
return None
class SubscriptionOutput(Validator, ABC):
def __init__(self, name, value, presets: List[str], indent_overrides: List[str]):
super().__init__(name, value)
self._presets = copy.deepcopy(presets)
self._indent_overrides = copy.deepcopy(indent_overrides)
def _indent_overrides_dict(self) -> Dict[str, str]:
"""
Returns
-------
indent overrides to merge with the preset dict's overrides
"""
return {
subscription_indent_variable_name(i): self._indent_overrides[i]
for i in range(len(self._indent_overrides))
}
@abstractmethod
def subscription_dicts(self) -> Dict[str, Dict]:
"""
Returns
-------
Subscriptions in the form of ``{ subscription_name: preset_dict }``
"""
class SubscriptionPresetDictValidator(SubscriptionOutput, DictValidator):
def __init__(self, name, value, presets: List[str], indent_overrides: List[str]):
super().__init__(name=name, value=value, presets=presets, indent_overrides=indent_overrides)
_ = self._validate_key_if_present(key="preset", validator=StringListValidator, default=[])
_ = self._validate_key_if_present(key="overrides", validator=Overrides, default={})
def subscription_dicts(self) -> Dict[str, Dict]:
output_dict = copy.deepcopy(self._dict)
parent_presets = output_dict.get("preset", [])
# Preset can be a single string
if isinstance(parent_presets, str):
parent_presets = [parent_presets]
output_dict["preset"] = parent_presets + self._presets
output_dict["overrides"] = dict(
output_dict.get("overrides", {}), **self._indent_overrides_dict()
)
return {self._leaf_name: output_dict}
class SubscriptionValueValidator(SubscriptionOutput, StringValidator):
def __init__(
self,
name,
value,
config: ConfigFile,
presets: List[str],
indent_overrides: List[str],
subscription_value: Optional[str],
):
super().__init__(name=name, value=value, presets=presets, indent_overrides=indent_overrides)
if self._leaf_name in config.presets.keys:
raise self._validation_exception(
f"{self._leaf_name} conflicts with an existing preset name and cannot be "
f"used as a subscription name"
)
if subscription_value is None:
raise self._validation_exception(
f"Subscription {self._leaf_name} is a string, but the subscription value "
f"is not set to an override variable"
)
self._subscription_value: str = subscription_value
def subscription_dicts(self) -> Dict[str, Dict]:
return {
self._leaf_name: {
"preset": self._presets,
"overrides": dict(
{
self._subscription_value: self.value,
"subscription_value": self.value,
},
**self._indent_overrides_dict(),
),
}
}
class SubscriptionValidator(SubscriptionOutput):
"""
Top-level subscription validator
"""
def __init__(
self,
name,
value,
config: ConfigFile,
presets: List[str],
indent_overrides: List[str],
subscription_value: Optional[str],
):
super().__init__(name=name, value=value, presets=presets, indent_overrides=indent_overrides)
self._children: List[SubscriptionOutput] = []
for key, obj in value.items():
obj_name = f"{name}.{key}" if name else key
if isinstance(obj, str):
self._children.append(
SubscriptionValueValidator(
name=obj_name,
value=obj,
config=config,
presets=presets,
indent_overrides=indent_overrides,
subscription_value=subscription_value,
)
)
elif isinstance(value, dict):
if key in config.presets.keys:
self._children.append(
SubscriptionValidator(
name=obj_name,
value=obj,
config=config,
presets=presets + [key],
indent_overrides=indent_overrides,
subscription_value=subscription_value,
)
)
elif override_value := maybe_indent_override_value(key):
self._children.append(
SubscriptionValidator(
name=obj_name,
value=obj,
config=config,
presets=presets,
indent_overrides=indent_overrides + [override_value],
subscription_value=subscription_value,
)
)
else:
self._children.append(
SubscriptionPresetDictValidator(
name=obj_name,
value=obj,
presets=presets,
indent_overrides=indent_overrides,
)
)
def subscription_dicts(self) -> Dict[str, Dict]:
subscription_dicts: Dict[str, Dict] = {}
for child in self._children:
subscription_dicts = dict(subscription_dicts, **child.subscription_dicts())
return subscription_dicts

View file

@ -93,6 +93,26 @@ class Validator(ABC):
"""
return validation_exception(self._name, error_message, exception_class)
@final
@property
def _root_name(self) -> str:
"""
Returns
-------
"first" from the first.element.of.the.name
"""
return self._name.split(".")[0]
@final
@property
def _leaf_name(self) -> str:
"""
Returns
-------
"first" from the first.element.of.the.name
"""
return self._name.split(".")[-1]
class ValueValidator(Validator, ABC, Generic[ValueT]):
"""
@ -179,16 +199,6 @@ class DictValidator(Validator):
super().__init__(name, value)
self.__validator_dict: Dict[str, Validator] = {}
@final
@property
def _root_name(self) -> str:
"""
Returns
-------
"first" from the first.element.of.the.name
"""
return self._name.split(".")[0]
@final
@property
def _dict(self) -> dict:

View file

@ -43,6 +43,8 @@ def preset_with_file_preset(youtube_video: Dict, output_options: Dict):
"overrides": {
"test_file_subscription_value": "original",
"test_config_subscription_value": "original",
"subscription_indent_1": "original_1",
"subscription_indent_2": "original_2",
},
},
"test_preset": {
@ -75,6 +77,38 @@ def preset_with_subscription_file_value(preset_with_subscription_value: Dict):
)
@pytest.fixture
def preset_with_subscription_value_nested_presets(preset_with_subscription_value: Dict):
return dict(
preset_with_subscription_value,
**{
"parent_preset_2": {
"parent_preset_1": {"test_2_1": "is_2_1_overwritten"},
"test_1": "is_1_overwritten",
}
},
)
@pytest.fixture
def preset_with_subscription_value_nested_presets_and_indent_variables(
preset_with_subscription_value: Dict,
):
return dict(
preset_with_subscription_value,
**{
"parent_preset_2": {
"[INDENT_1]": {
"parent_preset_1": {"test_2_1": "is_2_1_overwritten"},
"[INDENT_2]": {
"test_1": "is_1_overwritten",
},
}
}
},
)
def test_subscription_file_preset_applies(config_file: ConfigFile, preset_with_file_preset: Dict):
with mock_load_yaml(preset_dict=preset_with_file_preset):
subs = Subscription.from_file_path(config=config_file, subscription_path="mocked")
@ -104,15 +138,12 @@ def test_subscription_file_value_applies(
# Test __value__ worked correctly
value_sub = subs[1]
overrides = value_sub.overrides.dict_with_format_strings
assert value_sub.name == "test_value"
assert (
value_sub.overrides.dict_with_format_strings.get("test_file_subscription_value")
== "is_overwritten"
)
assert (
value_sub.overrides.dict_with_format_strings.get("test_config_subscription_value")
== "original"
)
assert overrides.get("test_file_subscription_value") == "is_overwritten"
assert overrides.get("test_file_subscription_value")
assert overrides.get("subscription_value") == "is_overwritten"
def test_subscription_file_value_applies_sub_file_takes_precedence(
@ -126,16 +157,11 @@ def test_subscription_file_value_applies_sub_file_takes_precedence(
assert len(subs) == 2
# Test __value__ worked correctly
value_sub = subs[1]
assert value_sub.name == "test_value"
assert (
value_sub.overrides.dict_with_format_strings.get("test_file_subscription_value")
== "is_overwritten"
)
assert (
value_sub.overrides.dict_with_format_strings.get("test_config_subscription_value")
== "original"
)
value_sub = subs[1].overrides.dict_with_format_strings
assert value_sub.get("test_file_subscription_value") == "is_overwritten"
assert value_sub.get("test_config_subscription_value") == "original"
assert value_sub.get("subscription_name") == "test_value"
assert value_sub.get("subscription_value") == "is_overwritten"
def test_subscription_file_value_applies_from_config(
@ -148,16 +174,69 @@ def test_subscription_file_value_applies_from_config(
assert len(subs) == 2
# Test __value__ worked correctly from the config
value_sub = subs[1]
assert value_sub.name == "test_value"
assert (
value_sub.overrides.dict_with_format_strings.get("test_file_subscription_value")
== "original"
)
assert (
value_sub.overrides.dict_with_format_strings.get("test_config_subscription_value")
== "is_overwritten"
)
value_sub = subs[1].overrides.dict_with_format_strings
assert value_sub.get("test_file_subscription_value") == "original"
assert value_sub.get("test_config_subscription_value") == "is_overwritten"
assert value_sub.get("subscription_name") == "test_value"
assert value_sub.get("subscription_value") == "is_overwritten"
def test_subscription_file_value_applies_from_config_and_nested(
config_file_with_subscription_value: ConfigFile,
preset_with_subscription_value_nested_presets: Dict,
):
with mock_load_yaml(preset_dict=preset_with_subscription_value_nested_presets):
subs = Subscription.from_file_path(
config=config_file_with_subscription_value, subscription_path="mocked"
)
assert len(subs) == 4
# Test __value__ worked correctly from the config
sub_1 = [sub for sub in subs if sub.name == "test_1"][0].overrides.dict_with_format_strings
sub_2_1 = [sub for sub in subs if sub.name == "test_2_1"][0].overrides.dict_with_format_strings
assert sub_1.get("test_config_subscription_value") == "is_1_overwritten"
assert sub_1.get("subscription_name") == "test_1"
assert sub_1.get("subscription_value") == "is_1_overwritten"
assert sub_2_1.get("test_config_subscription_value") == "is_2_1_overwritten"
assert sub_2_1.get("subscription_name") == "test_2_1"
assert sub_2_1.get("subscription_value") == "is_2_1_overwritten"
def test_subscription_file_value_applies_from_config_and_nested_and_indent_variables(
config_file_with_subscription_value: ConfigFile,
preset_with_subscription_value_nested_presets_and_indent_variables: Dict,
):
with mock_load_yaml(
preset_dict=preset_with_subscription_value_nested_presets_and_indent_variables
):
subs = Subscription.from_file_path(
config=config_file_with_subscription_value, subscription_path="mocked"
)
assert len(subs) == 4
# Test __value__ worked correctly from the config
sub_test_value = [sub for sub in subs if sub.name == "test_value"][
0
].overrides.dict_with_format_strings
sub_1 = [sub for sub in subs if sub.name == "test_1"][0].overrides.dict_with_format_strings
sub_2_1 = [sub for sub in subs if sub.name == "test_2_1"][0].overrides.dict_with_format_strings
assert sub_test_value.get("subscription_indent_1") == "original_1"
assert sub_test_value.get("subscription_indent_2") == "original_2"
assert sub_1.get("test_config_subscription_value") == "is_1_overwritten"
assert sub_1.get("subscription_name") == "test_1"
assert sub_1.get("subscription_value") == "is_1_overwritten"
assert sub_1.get("subscription_indent_1") == "INDENT_1"
assert sub_1.get("subscription_indent_2") == "INDENT_2"
assert sub_2_1.get("test_config_subscription_value") == "is_2_1_overwritten"
assert sub_2_1.get("subscription_name") == "test_2_1"
assert sub_2_1.get("subscription_value") == "is_2_1_overwritten"
assert sub_2_1.get("subscription_indent_1") == "INDENT_1"
assert sub_2_1.get("subscription_indent_2") == "original_2"
def test_subscription_file_bad_value(config_file: ConfigFile):
@ -173,12 +252,30 @@ def test_subscription_file_bad_value(config_file: ConfigFile):
def test_subscription_file_using_value_when_not_defined(config_file: ConfigFile):
with mock_load_yaml(
preset_dict={"sub_name": "single value, __value__ not defined"}
preset_dict={"[INDENTS_IN_ERR_MSG]": {"sub_name": "single value, __value__ not defined"}}
), pytest.raises(
ValidationException,
match=re.escape(
f"Subscription sub_name is a string, but "
f"the subscription value is not set to an override variable"
"Validation error in [INDENTS_IN_ERR_MSG].sub_name: Subscription "
"sub_name is a string, but the subscription value is not set to an override variable"
),
):
_ = Subscription.from_file_path(config=config_file, subscription_path="mocked")
def test_subscription_file_using_conflicting_preset_name(config_file: ConfigFile):
with mock_load_yaml(
preset_dict={
"[INDENTS_IN_ERR_MSG]": {
"[ANOTHER]": {"jellyfin_tv_show_by_date": "single value, __value__ not defined"}
}
}
), pytest.raises(
ValidationException,
match=re.escape(
"Validation error in [INDENTS_IN_ERR_MSG].[ANOTHER].jellyfin_tv_show_by_date: "
"jellyfin_tv_show_by_date conflicts with an existing preset name and cannot be used "
"as a subscription name"
),
):
_ = Subscription.from_file_path(config=config_file, subscription_path="mocked")
@ -187,6 +284,6 @@ def test_subscription_file_using_value_when_not_defined(config_file: ConfigFile)
def test_subscription_file_invalid_form(config_file: ConfigFile):
with mock_load_yaml(preset_dict={"sub_name": 4332}), pytest.raises(
ValidationException,
match=re.escape(f"Subscription sub_name should be in the form of a preset"),
match=re.escape(f"Validation error in sub_name: should be of type object."),
):
_ = Subscription.from_file_path(config=config_file, subscription_path="mocked")