diff --git a/README.md b/README.md index 4feb0a37..a6bb3348 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Web GUI for [yt-dlp](https://github.com/yt-dlp/yt-dlp) with playlist & channel s * Multi-downloads support. * Random beautiful background. `can be disabled`. -* Can Handle live streams. +* Can handle live streams. * Scheduler to queue channels or playlists to be downloaded automatically at a specified time. * Send notification to targets based on selected events. * Support per link `cli options` & `cookies` @@ -26,6 +26,7 @@ Web GUI for [yt-dlp](https://github.com/yt-dlp/yt-dlp) with playlist & channel s * Support for both advanced and basic mode for WebUI. * Bundled tools in container: curl-cffi, ffmpeg, ffprobe, aria2, rtmpdump, mkvtoolsnix, mp4box. * Automatic upcoming live stream re-queue. +* Apply `yt-dlp` options per custom defined conditions. # Run using docker command @@ -54,53 +55,11 @@ services: ``` ```bash -$ mkdir {config,downloads} && docker-compose -f compose.yaml up -d +$ mkdir {config,downloads} && docker compose -f compose.yaml up -d ``` Then you can access the WebUI at `http://localhost:8081`. -# Environment variables - -Certain configuration values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in `compose.yaml` file. - -| Environment Variable | Description | Default | -| ------------------------ | ---------------------------------------------------------------- | ---------------------------------- | -| YTP_OUTPUT_TEMPLATE | The template for the filenames of the downloaded videos | `%(title)s.%(ext)s` | -| YTP_DEFAULT_PRESET | The default preset to use for the download | `default` | -| YTP_INSTANCE_TITLE | The title of the instance | `empty string` | -| YTP_FILE_LOGGING | Whether to log to file | `false` | -| YTP_DOWNLOAD_PATH | Path to where the downloads will be saved | `/downloads` | -| YTP_MAX_WORKERS | How many works to use for downloads | `1` | -| YTP_AUTH_USERNAME | Username for basic authentication | `empty string` | -| YTP_AUTH_PASSWORD | Password for basic authentication | `empty string` | -| YTP_CONSOLE_ENABLED | Whether to enable the console | `false` | -| YTP_REMOVE_FILES | Remove the actual file when clicking the remove button | `false` | -| YTP_CONFIG_PATH | Path to where the queue persistence files will be saved | `/config` | -| YTP_TEMP_PATH | Path where intermediary download files will be saved | `/tmp` | -| YTP_TEMP_KEEP | Whether to keep the Individual video temp directory or remove it | `false` | -| YTP_KEEP_ARCHIVE | Keep history of downloaded videos | `true` | -| YTP_YTDL_DEBUG | Whether to turn debug logging for the internal `yt-dlp` package | `false` | -| YTP_ALLOW_MANIFESTLESS | Allow `yt-dlp` to download unprocessed streams | `false` | -| YTP_HOST | Which IP address to bind to | `0.0.0.0` | -| YTP_PORT | Which port to bind to | `8081` | -| YTP_LOG_LEVEL | Log level | `info` | -| YTP_STREAMER_VCODEC | The video codec to use for in-browser streaming | `libx264` | -| YTP_STREAMER_ACODEC | The audio codec to use for in-browser streaming | `aac` | -| YTP_ACCESS_LOG | Whether to log access to the web server | `true` | -| YTP_DEBUG | Whether to turn on debug mode | `false` | -| YTP_DEBUGPY_PORT | The port to use for the debugpy debugger | `5678` | -| YTP_SOCKET_TIMEOUT | The timeout for the yt-dlp socket connection variable | `30` | -| YTP_EXTRACT_INFO_TIMEOUT | The timeout for extracting video information | `70` | -| YTP_DB_FILE | The path to the SQLite database file | `{config_path}/ytptube.db` | -| YTP_MANUAL_ARCHIVE | The path to the manual archive file | `{config_path}/manual_archive.log` | -| YTP_UI_UPDATE_TITLE | Whether to update the title of the page with the current stats | `true` | -| YTP_PIP_PACKAGES | A space separated list of pip packages to install | `empty string` | -| YTP_PIP_IGNORE_UPDATES | Do not update the custom pip packages | `false` | -| YTP_BASIC_MODE | Whether to run WebUI in basic mode | `false` | -| YTP_PICTURES_BACKENDS | A comma separated list of pictures urls to use. | `empty string` | -| YTP_BROWSER_ENABLED | Whether to enable the file browser | `false` | -| YTP_YTDLP_AUTO_UPDATE | Whether to enable the auto update for yt-dlp | `false` | - # Browser extensions & bookmarklets ## Simple bookmarklet @@ -169,21 +128,28 @@ example.com { # Updating yt-dlp -The engine which powers the actual video downloads in YTPTube is [yt-dlp](https://github.com/yt-dlp/yt-dlp). Since video sites regularly change their layouts, frequent updates of yt-dlp are required to keep up. +The engine which powers the actual video downloads in YTPTube is [yt-dlp](https://github.com/yt-dlp/yt-dlp). Since video +sites regularly change their layouts, frequent updates of yt-dlp are required to keep up. -There's an automatic nightly task YTPTube which looks for a new version of yt-dlp, and if one exists, it will be open PR on the repository. The PR will be reviewed and merged by the maintainer. +We have added `YTP_YTDLP_AUTO_UPDATE` environment, which allows you to automatically update yt-dlp to the latest version. +To get latest version, simply set the environment variable to `true` and restart the container. Now, whenever you restart +the container, it will check for the latest version of yt-dlp and update it if a new version is available. + +We will no longer release new versions of YTPTube for every new version of yt-dlp. # Troubleshooting and submitting issues -Before asking a question or submitting an issue for YTPTube, Please remember that YTPTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp). Any issues you might be experiencing with authentication to video websites, postprocessing, permissions, other `yt-dlp options` configurations which seem not to work, or anything else that concerns the workings of the underlying yt-dlp library, need not be opened on the YTPTube project. In order to debug and troubleshoot them, it's advised to try using the yt-dlp binary directly first, bypassing the UI, and once that is working, importing the options that worked for you into `yt-dlp options` file. +Before asking a question or submitting an issue for YTPTube, Please remember that YTPTube is only a thin WebUI for [yt-dlp](https://github.com/yt-dlp/yt-dlp). +Any issues you might be experiencing with authentication to video websites, postprocessing, permissions, other `yt-dlp options` +configurations which seem not to work, or anything else that concerns the workings of the underlying yt-dlp library, need not be opened on the YTPTube project. -In order to test with the yt-dlp command directly, you can either download it and run it locally, or for a better simulation of its actual conditions, you can run it within the YTPTube container itself. +In order to debug and troubleshoot them, it's advised to try using the yt-dlp binary directly first, +bypassing the UI, and once that is working, importing the options that worked for you into a new `preset` or `ytdlp.cli` file. ## Via HTTP If you have enabled the web terminal via `YTP_CONSOLE_ENABLED` environment variable, simply go to `Terminal` button in -your navbar and directly use the yt-dlp command, the interface is jailed to the `yt-dlp` binary you can't access the -anything else. +your navbar and directly use the yt-dlp command, the interface is jailed to the `yt-dlp` binary you can't access anything else. ## Via CLI @@ -197,35 +163,6 @@ yt-dlp .... Once there, you can use the yt-dlp command freely. -# Building and running locally - -> [!IMPORTANT] -> Make sure you have `nodejs` and `Python 3.11+` installed. - -Follow these steps to build and run YTPTube locally: - -```bash -cd ytptube/ui -# install Vue and build the UI -npm install -npm run build -# install python dependencies -cd .. -# you might not have venv module installed, if so, install it via `pip3 install venv` or your package manager. -python -m venv .venv -source .venv/bin/activate -pip3 install pipenv -pipenv install -# run -python app/main.py -``` - -A Docker image can be built locally (it will build the UI too): - -```bash -docker build . -t ytptube -``` - # ytdlp.cli file The `config/ytdlp.cli`, is a command line options file for `yt-dlp` it will be globally applied to all downloads. @@ -248,7 +185,6 @@ As this is a simple basic authentication, if your browser doesn't show the promp `http://username:password@your_ytptube_url:port` - # Basic mode What does the basic mode do? it hides the the following features from the WebUI. @@ -303,17 +239,6 @@ project and what i wanted to achieve with it. This project is not affiliated with YouTube, yt-dlp, or any other service. It's a personal project that was created to make downloading videos from the internet easier. It's not intended to be used for piracy or any other illegal activities. -# Project plan - -Let me start by saying, i am primarily PHP developer, and i am not familiar with the best practices of python or frontend design. - -There are no project plan, or design document for this project. I started this project as a hobby project, and i am learning -as i go. What you see is the result of me learning while creating this tool. It might not be the best but i like it. - -While i value your feedback, remember this project is a hobby project, and i may not have the time to implement all the -features you might want. I am open to PRs, and i will do my best to review them and merge them if they fit my vision for the -project. - # Social contact If you have short or quick questions, you are free to join my [discord server](https://discord.gg/G3GpVR8xpb) and ask @@ -323,3 +248,45 @@ the question. keep in mind it's solo project, as such it might take me a bit of If you feel like donating and appreciate my work, you can do so by donating to children charity. For example [Make-A-Wish](https://worldwish.org). +# Environment variables + +Certain configuration values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in `compose.yaml` file. + +| Environment Variable | Description | Default | +| ------------------------ | ---------------------------------------------------------------- | ---------------------------------- | +| YTP_OUTPUT_TEMPLATE | The template for the filenames of the downloaded videos | `%(title)s.%(ext)s` | +| YTP_DEFAULT_PRESET | The default preset to use for the download | `default` | +| YTP_INSTANCE_TITLE | The title of the instance | `empty string` | +| YTP_FILE_LOGGING | Whether to log to file | `false` | +| YTP_DOWNLOAD_PATH | Path to where the downloads will be saved | `/downloads` | +| YTP_MAX_WORKERS | How many works to use for downloads | `1` | +| YTP_AUTH_USERNAME | Username for basic authentication | `empty string` | +| YTP_AUTH_PASSWORD | Password for basic authentication | `empty string` | +| YTP_CONSOLE_ENABLED | Whether to enable the console | `false` | +| YTP_REMOVE_FILES | Remove the actual file when clicking the remove button | `false` | +| YTP_CONFIG_PATH | Path to where the queue persistence files will be saved | `/config` | +| YTP_TEMP_PATH | Path where intermediary download files will be saved | `/tmp` | +| YTP_TEMP_KEEP | Whether to keep the Individual video temp directory or remove it | `false` | +| YTP_KEEP_ARCHIVE | Keep history of downloaded videos | `true` | +| YTP_YTDL_DEBUG | Whether to turn debug logging for the internal `yt-dlp` package | `false` | +| YTP_ALLOW_MANIFESTLESS | Allow `yt-dlp` to download unprocessed streams | `false` | +| YTP_HOST | Which IP address to bind to | `0.0.0.0` | +| YTP_PORT | Which port to bind to | `8081` | +| YTP_LOG_LEVEL | Log level | `info` | +| YTP_STREAMER_VCODEC | The video codec to use for in-browser streaming | `libx264` | +| YTP_STREAMER_ACODEC | The audio codec to use for in-browser streaming | `aac` | +| YTP_ACCESS_LOG | Whether to log access to the web server | `true` | +| YTP_DEBUG | Whether to turn on debug mode | `false` | +| YTP_DEBUGPY_PORT | The port to use for the debugpy debugger | `5678` | +| YTP_SOCKET_TIMEOUT | The timeout for the yt-dlp socket connection variable | `30` | +| YTP_EXTRACT_INFO_TIMEOUT | The timeout for extracting video information | `70` | +| YTP_DB_FILE | The path to the SQLite database file | `{config_path}/ytptube.db` | +| YTP_MANUAL_ARCHIVE | The path to the manual archive file | `{config_path}/manual_archive.log` | +| YTP_UI_UPDATE_TITLE | Whether to update the title of the page with the current stats | `true` | +| YTP_PIP_PACKAGES | A space separated list of pip packages to install | `empty string` | +| YTP_PIP_IGNORE_UPDATES | Do not update the custom pip packages | `false` | +| YTP_BASIC_MODE | Whether to run WebUI in basic mode | `false` | +| YTP_PICTURES_BACKENDS | A comma separated list of pictures urls to use. | `empty string` | +| YTP_BROWSER_ENABLED | Whether to enable the file browser | `false` | +| YTP_YTDLP_AUTO_UPDATE | Whether to enable the auto update for yt-dlp | `false` | + diff --git a/app/library/DownloadQueue.py b/app/library/DownloadQueue.py index db3769b7..18c1d11b 100644 --- a/app/library/DownloadQueue.py +++ b/app/library/DownloadQueue.py @@ -15,6 +15,7 @@ import yt_dlp from aiohttp import web from .AsyncPool import AsyncPool +from .conditions import Conditions from .config import Config from .DataStore import DataStore from .Download import Download @@ -227,7 +228,7 @@ class DownloadQueue(metaclass=Singleton): eventType = entry.get("_type") or "video" if "playlist" == eventType: - LOG.info("Processing playlist") + LOG.info(f"Processing playlist '{entry.get('id')}: {entry.get('title')}'.") entries = entry.get("entries", []) playlistCount = int(entry.get("playlist_count", len(entries))) results = [] @@ -479,6 +480,13 @@ class DownloadQueue(metaclass=Singleton): if not entry: return {"status": "error", "msg": "Unable to extract info." + "\n".join(logs)} + if not item.requeued: + condition = Conditions.get_instance().match(info=entry) + if condition is not None: + already.pop() + LOG.info(f"Condition '{condition}' matched for '{item.url}'.") + return await self.add(item=item.new_with(requeued=True, cli=condition.cli),already=already) + end_time = time.perf_counter() - started LOG.debug(f"extract_info: for 'URL: {item.url}' is done in '{end_time:.3f}'. Length: '{len(entry)}/keys'.") except yt_dlp.utils.ExistingVideoReached as exc: diff --git a/app/library/Events.py b/app/library/Events.py index c73ad6e9..b07ab345 100644 --- a/app/library/Events.py +++ b/app/library/Events.py @@ -128,6 +128,9 @@ class Events: PRESETS_UPDATE = "presets_update" SCHEDULE_ADD = "schedule_add" + CONDITIONS_ADD = "conditions_add" + CONDITIONS_UPDATE = "conditions_update" + SUBSCRIBED = "subscribed" UNSUBSCRIBED = "unsubscribed" diff --git a/app/library/HttpAPI.py b/app/library/HttpAPI.py index 6a611938..90677263 100644 --- a/app/library/HttpAPI.py +++ b/app/library/HttpAPI.py @@ -22,6 +22,7 @@ from yt_dlp.cookies import LenientSimpleCookie from .cache import Cache from .common import Common +from .conditions import Condition, Conditions from .config import Config from .DownloadQueue import DownloadQueue from .encoder import Encoder @@ -77,6 +78,7 @@ class HttpAPI(Common): "/notifications", "/changelog", "/logs", + "/conditions", "/browser", "/browser/{path:.*}", ] @@ -771,6 +773,94 @@ class HttpAPI(Common): dumps=self.encoder.encode, ) + @route("GET", "api/conditions") + async def conditions(self, _: Request) -> Response: + """ + Get the conditions + + Args: + _ (Request): The request object. + + Returns: + Response: The response object. + + """ + return web.json_response( + data=Conditions.get_instance().get_all(), + status=web.HTTPOk.status_code, + dumps=self.encoder.encode, + ) + + @route("PUT", "api/conditions") + async def conditions_add(self, request: Request) -> Response: + """ + Save Conditions. + + Args: + request (Request): The request object. + + Returns: + Response: The response object + + """ + data = await request.json() + + if not isinstance(data, list): + return web.json_response( + {"error": "Invalid request body expecting list with dicts."}, + status=web.HTTPBadRequest.status_code, + ) + + items: list = [] + + cls = Conditions.get_instance() + + for item in data: + if not isinstance(item, dict): + return web.json_response( + {"error": "Invalid request body expecting list with dicts."}, + status=web.HTTPBadRequest.status_code, + ) + + if not item.get("name"): + return web.json_response( + {"error": "name is required.", "data": item}, status=web.HTTPBadRequest.status_code + ) + + if not item.get("filter"): + return web.json_response( + {"error": "filter is required.", "data": item}, status=web.HTTPBadRequest.status_code + ) + + if not item.get("cli"): + return web.json_response( + {"error": "CLI arguments is required.", "data": item}, status=web.HTTPBadRequest.status_code + ) + + if not item.get("id", None) or not validate_uuid(item.get("id"), version=4): + item["id"] = str(uuid.uuid4()) + + try: + cls.validate(item) + except ValueError as e: + return web.json_response( + {"error": f"Failed to validate condition '{item.get('name')}'. '{e!s}'"}, + status=web.HTTPBadRequest.status_code, + ) + + items.append(Condition(**item)) + try: + items = cls.save(items=items).load().get_all() + except Exception as e: + LOG.exception(e) + return web.json_response( + {"error": "Failed to save conditions.", "message": str(e)}, + status=web.HTTPInternalServerError.status_code, + ) + + await self._notify.emit(Events.CONDITIONS_UPDATE, data=items) + return web.json_response(data=items, status=web.HTTPOk.status_code, dumps=self.encoder.encode) + @route("GET", "api/presets") async def presets(self, request: Request) -> Response: """ @@ -828,11 +918,6 @@ class HttpAPI(Common): {"error": "name is required.", "data": item}, status=web.HTTPBadRequest.status_code ) - if not item.get("format"): - return web.json_response( - {"error": "format is required.", "data": item}, status=web.HTTPBadRequest.status_code - ) - if not item.get("id", None) or not validate_uuid(item.get("id"), version=4): item["id"] = str(uuid.uuid4()) @@ -846,7 +931,7 @@ class HttpAPI(Common): presets.append(Preset(**item)) try: - presets = cls.save(presets=presets).load().get_all() + presets = cls.save(items=presets).load().get_all() except Exception as e: LOG.exception(e) return web.json_response( @@ -1638,7 +1723,7 @@ class HttpAPI(Common): LOG.error(str(e)) return web.json_response(data={"message": str(e)}, status=web.HTTPInternalServerError.status_code) - url = "https://www.youtube.com/account" + url = "https://www.youtube.com/paid_memberships" try: opts = { @@ -1655,8 +1740,8 @@ class HttpAPI(Common): LOG.debug(f"Checking '{url}' redirection.") response = await client.request(method="GET", url=url, follow_redirects=False) return web.json_response( - data={"message": "Authenticated." if response.status_code == 200 else "Not authenticated."}, - status=200 if response.status_code == 200 else 401, + data={"message": "Authenticated." if 200 == response.status_code else "Not authenticated."}, + status=200 if 200 == response.status_code else 401, ) except Exception as e: LOG.error(f"Failed to request '{url}'. '{e}'.") diff --git a/app/library/HttpSocket.py b/app/library/HttpSocket.py index e12cea1b..77ea1c56 100644 --- a/app/library/HttpSocket.py +++ b/app/library/HttpSocket.py @@ -373,15 +373,16 @@ class HttpSocket(Common): pass @ws_event - async def disconnect(self, sid: str): + async def disconnect(self, sid: str, reason: str): """ Handle client disconnection. Args: sid (str): The session ID of the client. + reason (str): The reason for disconnection. """ - LOG.debug(f"Client '{sid}' disconnected.") + LOG.debug(f"Client '{sid}' disconnected. {reason}") for event in self.subscribers: if sid in self.subscribers[event]: diff --git a/app/library/ItemDTO.py b/app/library/ItemDTO.py index 2619675a..7934aa8f 100644 --- a/app/library/ItemDTO.py +++ b/app/library/ItemDTO.py @@ -36,6 +36,9 @@ class Item: extras: dict = field(default_factory=dict) """Extra data to be added to the download.""" + requeued: bool = False + """If the item has been re-queued already via conditions.""" + def serialize(self) -> dict: return self.__dict__.copy() @@ -155,6 +158,9 @@ class Item: if extras and isinstance(extras, dict) and len(extras) > 0: data["extras"] = extras + if item.get("requeued") and isinstance(item.get("requeued"), bool): + data["requeued"] = item.get("requeued") + cli = item.get("cli") if cli and len(cli) > 2: from .Utils import arg_converter diff --git a/app/library/Presets.py b/app/library/Presets.py index f3670857..6202dfbc 100644 --- a/app/library/Presets.py +++ b/app/library/Presets.py @@ -24,9 +24,6 @@ class Preset: name: str """The name of the preset.""" - format: str - """The format of the preset.""" - folder: str = "" """The default download folder to use if non is given.""" @@ -56,13 +53,13 @@ class Presets(metaclass=Singleton): This class is used to manage the presets. """ - _presets: list[Preset] = [] + _items: list[Preset] = [] """The list of presets.""" _instance = None """The instance of the class.""" - _default_presets: list[Preset] = [] + _default: list[Preset] = [] def __init__(self, file: str | None = None, config: Config | None = None): Presets._instance = self @@ -77,13 +74,14 @@ class Presets(metaclass=Singleton): except Exception: pass - with open(os.path.join(os.path.dirname(__file__), "presets.json")) as f: + default_file = os.path.join(os.path.dirname(__file__), "presets.json") + with open(default_file) as f: for i, preset in enumerate(json.load(f)): try: self.validate(preset) - self._default_presets.append(Preset(**preset)) + self._default.append(Preset(**preset)) except Exception as e: - LOG.error(f"Failed to parse default preset '{i}'. '{e!s}'.") + LOG.error(f"Failed to parse '{default_file}:{i}'. '{e!s}'.") continue def event_handler(_, __): @@ -111,7 +109,7 @@ class Presets(metaclass=Singleton): def attach(self, _: web.Application): """ - Attach the work to the aiohttp application. + Attach the class to the aiohttp application. Args: _ (web.Application): The aiohttp application. @@ -123,12 +121,12 @@ class Presets(metaclass=Singleton): self.load() def get_all(self) -> list[Preset]: - """Return the presets.""" - return self._default_presets + self._presets + """Return the items.""" + return self._default + self._items def load(self) -> "Presets": """ - Load the Presets. + Load the items. Returns: Presets: The current instance. @@ -139,16 +137,15 @@ class Presets(metaclass=Singleton): if not os.path.exists(self._file) or os.path.getsize(self._file) < 10: return self - LOG.info(f"Loading presets from '{self._file}'.") + LOG.info(f"Loading '{self._file}'.") try: with open(self._file) as f: presets = json.load(f) except Exception as e: - LOG.error(f"Failed to parse presets from '{self._file}'. '{e}'.") + LOG.error(f"Failed to parse '{self._file}'. '{e}'.") return self if not presets or len(presets) < 1: - LOG.info(f"No presets were defined in '{self._file}'.") return self need_save = False @@ -160,134 +157,144 @@ class Presets(metaclass=Singleton): need_save = True preset, preset_status = clean_item(preset, keys=("args", "postprocessors")) + if preset.get("format"): + if not preset.get("cli"): + preset.update({"cli": f"--format {preset['format']}"}) + else: + preset["cli"] = f"--format '{preset['format']}'\n" + preset["cli"] + + preset["cli"] = str(preset["cli"]).strip() + + preset.pop("format") + need_save = True + preset = Preset(**preset) if preset_status: need_save = True - self._presets.append(preset) + self._items.append(preset) except Exception as e: - LOG.error(f"Failed to parse preset at list position '{i}'. '{e!s}'.") + LOG.error(f"Failed to parse '{self._file}:{i}'. '{e!s}'.") continue if need_save: - LOG.info("Saving presets due to format, or id change.") - self.save(self._presets) + LOG.info(f"Saving '{self._file}' due to changes.") + self.save(self._items) return self def clear(self) -> "Presets": """ - Clear all presets + Clear all items. Returns: Presets: The current instance. """ - if len(self._presets) < 1: + if len(self._items) < 1: return self - self._presets.clear() + self._items.clear() return self - def validate(self, preset: Preset | dict) -> bool: + def validate(self, item: Preset | dict) -> bool: """ - Validate the preset. + Validate the item. Args: - preset (Preset|dict): The preset to validate. + item (Preset|dict): The item to validate. Returns: - bool: True if the preset is valid, False otherwise. + bool: True if valid + + Raises: + ValueError: If the item is not valid. """ - if not isinstance(preset, dict): - if not isinstance(preset, Preset): - msg = f"Invalid preset type. Was expecting a (Preset|dict), but got '{type(preset).__name__}'." + if not isinstance(item, dict): + if not isinstance(item, Preset): + msg = f"Unexpected '{type(item).__name__}' type was given." raise ValueError(msg) # noqa: TRY004 - preset = preset.serialize() + item = item.serialize() - if not preset.get("id"): + if not item.get("id"): msg = "No id found." raise ValueError(msg) - if not preset.get("name"): + if not item.get("name"): msg = "No name found." raise ValueError(msg) - if not preset.get("format"): - msg = "No format found." - raise ValueError(msg) - - if preset.get("cli"): + if item.get("cli"): try: - arg_converter(args=preset.get("cli")) + arg_converter(args=item.get("cli")) except Exception as e: msg = f"Invalid cli options. '{e!s}'." raise ValueError(msg) from e return True - def save(self, presets: list[Preset | dict]) -> "Presets": + def save(self, items: list[Preset | dict]) -> "Presets": """ - Save the presets. + Save the items. Args: - presets (list[Preset]): The presets to save. + items (list[Preset]): The items to save. Returns: Presets: The current instance. """ - for i, preset in enumerate(presets): + for i, preset in enumerate(items): try: if not isinstance(preset, Preset): preset = Preset(**preset) - presets[i] = preset + items[i] = preset except Exception as e: - LOG.error(f"Failed to save preset '{i}' due to parsing error. '{e!s}'.") + LOG.error(f"Failed to save item '{i}' due to parsing error. '{e!s}'.") continue try: self.validate(preset) except ValueError as e: - LOG.error(f"Failed to validate preset '{i}: {preset.name}'. '{e}'.") + LOG.error(f"Failed to validate item '{i}: {preset.name}'. '{e}'.") continue try: with open(self._file, "w") as f: - json.dump(obj=[preset.serialize() for preset in presets if preset.default is False], fp=f, indent=4) + json.dump(obj=[preset.serialize() for preset in items if preset.default is False], fp=f, indent=4) - LOG.info(f"Presets saved to '{self._file}'.") + LOG.info(f"Saved '{self._file}'.") except Exception as e: - LOG.error(f"Failed to save presets to '{self._file}'. '{e!s}'.") + LOG.error(f"Failed to save '{self._file}'. '{e!s}'.") return self def has(self, id_or_name: str) -> bool: """ - Check if the preset exists by id or name. + Check if the item exists by id or name. Args: - id_or_name (str): The id or name of the preset. + id_or_name (str): The id or name of the item. Returns: - bool: True if the preset exists, False otherwise. + bool: True if exists, False otherwise. """ return self.get(id_or_name) is not None def get(self, id_or_name: str) -> Preset | None: """ - Get the preset by id or name. + Get the item by id or name. Args: - id_or_name (str): The id or name of the preset. + id_or_name (str): The id or name of the item. Returns: - Preset|None: The preset if found, None otherwise. + Preset|None: The item if found, None otherwise. """ if not id_or_name: diff --git a/app/library/YTDLPOpts.py b/app/library/YTDLPOpts.py index 4e4003ba..4c4c1894 100644 --- a/app/library/YTDLPOpts.py +++ b/app/library/YTDLPOpts.py @@ -135,9 +135,6 @@ class YTDLPOpts(metaclass=Singleton): self._preset_opts["cookiefile"] = str(file) - if preset.format: - self._preset_opts["format"] = preset.format - if preset.template: self._preset_opts["outtmpl"] = {"default": preset.template, "chapter": self._config.output_template_chapter} diff --git a/app/library/conditions.py b/app/library/conditions.py new file mode 100644 index 00000000..b91beba1 --- /dev/null +++ b/app/library/conditions.py @@ -0,0 +1,319 @@ +import json +import logging +import os +import uuid +from dataclasses import dataclass, field +from typing import Any + +from aiohttp import web + +from .config import Config +from .encoder import Encoder +from .Events import EventBus, Events +from .Singleton import Singleton +from .Utils import arg_converter + +LOG = logging.getLogger("presets") + + +@dataclass(kw_only=True) +class Condition: + id: str = field(default_factory=lambda: str(uuid.uuid4())) + """The id of the condition.""" + + name: str + """The name of the condition.""" + + filter: str + """The filter to run on info dict.""" + + cli: str + """If matched append this to the download request and re-queue.""" + + def serialize(self) -> dict: + return self.__dict__ + + def json(self) -> str: + return Encoder().encode(self.serialize()) + + def get(self, key: str, default: Any = None) -> Any: + return self.serialize().get(key, default) + + +class Conditions(metaclass=Singleton): + """ + This class is used to manage the download conditions. + """ + + _items: list[Condition] = [] + """The list of items.""" + + _instance = None + """The instance of the class.""" + + def __init__(self, file: str | None = None, config: Config | None = None): + Conditions._instance = self + + config = config or Config.get_instance() + + self._file: str = file or os.path.join(config.config_path, "conditions.json") + + if os.path.exists(self._file) and "600" != oct(os.stat(self._file).st_mode)[-3:]: + try: + os.chmod(self._file, 0o600) + except Exception: + pass + + def event_handler(_, __): + msg = "Not implemented" + raise Exception(msg) + + EventBus.get_instance().subscribe(Events.CONDITIONS_ADD, event_handler, f"{__class__.__name__}.add") + + @staticmethod + def get_instance() -> "Conditions": + """ + Get the instance of the class. + + Returns: + Conditions: The instance of the class + + """ + if not Conditions._instance: + Conditions._instance = Conditions() + + return Conditions._instance + + async def on_shutdown(self, _: web.Application): + pass + + def attach(self, _: web.Application): + """ + Attach the work to the aiohttp application. + + Args: + _ (web.Application): The aiohttp application. + + Returns: + None + + """ + self.load() + + def get_all(self) -> list[Condition]: + """Return the items.""" + return self._items + + def load(self) -> "Conditions": + """ + Load the items. + + Returns: + Conditions: The current instance. + + """ + self.clear() + + if not os.path.exists(self._file) or os.path.getsize(self._file) < 10: + return self + + LOG.info(f"Loading '{self._file}'.") + try: + with open(self._file) as f: + items = json.load(f) + except Exception as e: + LOG.error(f"Failed to parse '{self._file}'. '{e}'.") + return self + + if not items or len(items) < 1: + LOG.debug(f"No items were found in '{self._file}'.") + return self + + need_save = False + + for i, item in enumerate(items): + try: + if "id" not in item: + item["id"] = str(uuid.uuid4()) + need_save = True + + item = Condition(**item) + + self._items.append(item) + except Exception as e: + LOG.error(f"Failed to parse failure condition at list position '{i}'. '{e!s}'.") + continue + + if need_save: + LOG.info("Saving failure conditions due to format, or id change.") + self.save(self._items) + + return self + + def clear(self) -> "Conditions": + """ + Clear all items + + Returns: + conditions: The current instance. + + """ + if len(self._items) < 1: + return self + + self._items.clear() + + return self + + def validate(self, item: Condition | dict) -> bool: + """ + Validate item. + + Args: + item (Condition|dict): The item to validate. + + Returns: + bool: True if valid, False otherwise. + + """ + from yt_dlp.utils import match_str + + if not isinstance(item, dict): + if not isinstance(item, Condition): + msg = f"Unexpected '{type(item).__name__}' item type." + raise ValueError(msg) # noqa: TRY004 + + item = item.serialize() + + if not item.get("id"): + msg = "No id found." + raise ValueError(msg) + + if not item.get("name"): + msg = "No name found." + raise ValueError(msg) + + if not item.get("filter"): + msg = "No filter found." + raise ValueError(msg) + + try: + match_str(item.get("filter"), {}) + except Exception as e: + msg = f"Invalid filter. '{e!s}'." + raise ValueError(msg) from e + + if not item.get("cli"): + msg = "No CLI arguments were found." + raise ValueError(msg) + + try: + arg_converter(args=item.get("cli")) + except Exception as e: + msg = f"Invalid cli options. '{e!s}'." + raise ValueError(msg) from e + + return True + + def save(self, items: list[Condition | dict]) -> "Conditions": + """ + Save the items. + + Args: + items (list[Condition]): The items to save. + + Returns: + Conditions: The current instance. + + """ + for i, item in enumerate(items): + try: + if not isinstance(item, Condition): + item = Condition(**item) + items[i] = item + except Exception as e: + LOG.error(f"Failed to save '{i}' due to parsing error. '{e!s}'.") + continue + + try: + self.validate(item) + except ValueError as e: + LOG.error(f"Failed to validate '{i}: {item.name}'. '{e!s}'.") + continue + + try: + with open(self._file, "w") as f: + json.dump(obj=[item.serialize() for item in items], fp=f, indent=4) + + LOG.info(f"Updated '{self._file}'.") + except Exception as e: + LOG.error(f"Failed to save '{self._file}'. '{e!s}'.") + + return self + + def has(self, id_or_name: str) -> bool: + """ + Check if the item exists by id or name. + + Args: + id_or_name (str): The id or name of the preset. + + Returns: + bool: True if the item exists, False otherwise. + + """ + return self.get(id_or_name) is not None + + def get(self, id_or_name: str) -> Condition | None: + """ + Get the item by id or name. + + Args: + id_or_name (str): The id or name of the preset. + + Returns: + Condition|None: The item if found, None otherwise. + + """ + if not id_or_name: + return None + + for condition in self.get_all(): + if id_or_name not in (condition.id, condition.name): + continue + + return condition + + return None + + def match(self, info: dict) -> Condition | None: + """ + Check if any condition matches the info dict. + + Args: + info (dict): The info dict to check. + + Returns: + Condition|None: The condition if found, None otherwise. + + """ + if len(self._items) < 1 or not info or not isinstance(info, dict) or len(info) < 1: + return None + + from yt_dlp.utils import match_str + + for item in self.get_all(): + if not item.filter: + LOG.error(f"Filter is empty for '{item.name}'.") + continue + + try: + if match_str(item.filter, info): + LOG.debug(f"Matched '{item.name}' with filter '{item.filter}'.") + + return item + except Exception as e: + LOG.error(f"Failed to evaluate '{item.name}'. '{e!s}'.") + continue + + return None diff --git a/app/library/presets.json b/app/library/presets.json index 86c6492c..1ece33bd 100644 --- a/app/library/presets.json +++ b/app/library/presets.json @@ -2,7 +2,6 @@ { "id": "3e163c6c-64eb-4448-924f-814b629b3810", "name": "default", - "format": "default", "folder": "", "template": "", "cookies": "", @@ -12,41 +11,37 @@ { "id": "5bf9c42b-8852-468a-99f5-915622dfba25", "name": "Best video and audio", - "format": "bv+ba/b", "folder": "", "template": "", "cookies": "", - "cli": "", + "cli": "--format 'bv+ba/b'", "default": true }, { "id": "441675ed-b739-40f0-a0b0-1ecfcb9dc48b", "name": "1080p H264/m4a or best available", - "format": "bv[height<=1080][ext=mp4]+ba[ext=m4a]/b[ext=mp4]/b[ext=webm]", "folder": "", "template": "", "cookies": "", - "cli": "-S vcodec:h264", + "cli": "-S vcodec:h264 --format 'bv[height<=1080][ext=mp4]+ba[ext=m4a]/b[ext=mp4]/b[ext=webm]'", "default": true }, { "id": "9719fcc3-4cf2-4d88-b1e4-74dff3dba00e", "name": "720p h264/m4a or best available", - "format": "bv[height<=720][ext=mp4]+ba[ext=m4a]/b[ext=mp4]/b[ext=webm]", "folder": "", "template": "", "cookies": "", - "cli": "-S vcodec:h264", + "cli": "-S vcodec:h264 --format 'bv[height<=720][ext=mp4]+ba[ext=m4a]/b[ext=mp4]/b[ext=webm]'", "default": true }, { "id": "a6fd4b25-2b3e-458d-bb57-b75e41cc4330", "name": "Audio only", - "format": "bestaudio/best", "folder": "", "template": "", "cookies": "", - "cli": "--extract-audio --add-chapters --embed-metadata --embed-thumbnail", + "cli": "--extract-audio --add-chapters --embed-metadata --embed-thumbnail --format 'bestaudio/best'", "default": true } ] diff --git a/app/main.py b/app/main.py index c5178537..14d6cb56 100644 --- a/app/main.py +++ b/app/main.py @@ -9,6 +9,7 @@ from pathlib import Path import caribou import magic from aiohttp import web +from library.conditions import Conditions from library.config import Config from library.DownloadQueue import DownloadQueue from library.Events import EventBus, Events @@ -167,6 +168,7 @@ class Main: Tasks.get_instance().attach(self._app) Presets.get_instance().attach(self._app) Notification.get_instance().attach(self._app) + Conditions.get_instance().attach(self._app) EventBus.get_instance().sync_emit(Events.LOADED, data={"app": self._app}) diff --git a/ui/assets/css/style.css b/ui/assets/css/style.css index db506ea5..48234947 100644 --- a/ui/assets/css/style.css +++ b/ui/assets/css/style.css @@ -313,3 +313,7 @@ hr { .is-justify-self-end { justify-self: end; } + +.is-full-height { + height: 100%; +} diff --git a/ui/components/ConditionForm.vue b/ui/components/ConditionForm.vue new file mode 100644 index 00000000..144940c6 --- /dev/null +++ b/ui/components/ConditionForm.vue @@ -0,0 +1,245 @@ +