diff --git a/README.md b/README.md index 2af4e726..7eb3c710 100644 --- a/README.md +++ b/README.md @@ -25,22 +25,47 @@ Web GUI for [yt-dlp](https://github.com/yt-dlp/yt-dlp) with playlist & channel s * Support for curl_cffi, see [yt-dlp documentation](https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#impersonation) * Support for both advanced and basic mode for WebUI. * Bundled tools in container: curl-cffi, ffmpeg, ffprobe, aria2, rtmpdump, mkvtoolsnix, mp4box. - -# Recommended basic `ytdlp.json` file settings -Your `/config/ytdlp.json` config should include the following basic options for optimal working conditions. +# Breaking changes -```json -{ - "windowsfilenames": true, - "continue_dl": true, - "live_from_start": true, - "format_sort": [ "codec:avc:m4a" ], -} -``` +Starting with versions tagged `*-20250330-*` I have deprecated the JSON yt-dlp config and all of it's related settings. -> [!NOTE] -> Note, the `format_sort`, forces YouTube to use x264 instead of vp9 codec, you can ignore it if you want. i prefer the media in x264. +To be frank, it was pain to manage and hard for users to understand how to map cli options to json options. So, I have +decided to just use the cli options directly. This means that you can now use the same options you would use in the command line +directly in the WebUI, presets, and tasks. + +## `ytdlp.json` vs `ytdlp.cli` + +I have also added a new `ytdlp.cli` file that will be used to store the global options for yt-dlp. This file is located in the `/config` directory +and will be used to store the global options for yt-dlp. This file is not required and presets can fill the gap for most of the +use cases. But, if you want to use the same options for all your downloads, you can use this file to store the global options. + +So, if you have `ytdlp.cli` file it will take priority over the `ytdlp.json` file if both exists. As mitigation, I have implemented +fallback to `ytdlp.json` file if `ytdlp.cli` file is not found. + +## Presets + +I have deprecated the `JSON yt-dlp config` and `JSON yt-dlp Post-Processors` fields, and added new `Command arguments for yt-dlp` +field to the presets. This field will be used to store the command line options for yt-dlp. I have also have added fallback, +if you have the `JSON yt-dlp config/Post-Processors` field set, the logic is, if `Command arguments for yt-dlp` is set +it will take priority over the `JSON yt-dlp config/Post-Processors` field. If not set, it will fallback to the `JSON yt-dlp config/Post-Processors` field. + +If you are adding new presets, the deprecated fields will not show up anymore, they will only show up if actually have content in them +and editing old preset. So, Please migrate your presets to the new format. + +## Tasks + +I have also removed the `JSON yt-dlp config` and replaced it with `Command arguments for yt-dlp` field. Sadly, there is +no fallback, and once you upgrade to any version after `*-20250330-*` your `tasks.json` file will be updated to remove the +`config` key. so, please make sure to backup your `tasks.json` file before upgrading. + +## closing statement + +I know it's a painful breaking change, but for the sake of maintainability and ease of use, I have decided to +to make it happen sooner than later, the `JSON yt-dlp config`, was hard to manage and some features weren't really working +as expected, for example `--match-filter` and `--date` arguments etc. + +So, Starting with `*-2025040*-*` tagged versions, the all the fallbacks and backwards compatibility will be removed. # Run using docker command @@ -240,25 +265,19 @@ A Docker image can be built locally (it will build the UI too): docker build . -t ytptube ``` -# ytdlp.json file +# ytdlp.cli file -The `config/ytdlp.json`, is a json file which can be used to alter the default `yt-dlp` config settings globally. +The `config/ytdlp.cli`, is a command line options file for `yt-dlp` it will be globally applied to all downloads. -We recommend not use this file for options that aren't **truly global**, everything that can be done via the `ytdlp.json` file -can be done via a preset, which only effects the download that uses it. Example of good basic `ytdlp.json` file. +We strongly recommend not use this file for options that aren't **truly global**, everything that can be done via the file +can also be done via the presets which is dynamic can be altered per download. Example of good global options are to be +used for all downloads are: -```json -{ - "windowsfilenames": true, - "continue_dl": true, - "live_from_start": true, - "format_sort": [ "codec:avc:m4a" ], -} +```bash +--continue --windows-filenames --live-from-start ``` -Everything else can be done via the presets, and it's more flexible and easier to manage. You can convert your -own yt-dlp command arguments into a preset using the box found in the presets add page. For reference, The options can be found -at [yt-dlp YoutubeDL.py](https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L214) file. And for the postprocessors at [yt-dlp postprocessor](https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/postprocessor). +Everything else can be done via the presets, and it's more flexible and easier to manage. # Authentication @@ -275,7 +294,7 @@ What does the basic mode do? it hides the the following features from the WebUI. ### Header -It disables everything except the `theme switcher` and `reload` button. +It disables everything except the `settings button` and `reload` button. ### Add form diff --git a/app/library/Download.py b/app/library/Download.py index a3fe0c7d..f05eda47 100644 --- a/app/library/Download.py +++ b/app/library/Download.py @@ -114,8 +114,6 @@ class Download: def _progress_hook(self, data: dict): dataDict = {k: v for k, v in data.items() if k in self._ytdlp_fields} - LOG.debug("Status: %s", data) - if "finished" == data.get("status") and data.get("info_dict", {}).get("filename", None): dataDict["filename"] = data["info_dict"]["filename"] @@ -137,6 +135,12 @@ class Download: self.status_queue.put({"id": self.id, "status": "finished", "filename": filename}) + def post_hooks(self, filename: str | None = None): + if not filename: + return + + self.status_queue.put({"id": self.id, "filename": filename}) + def _download(self): try: params = ( @@ -180,6 +184,7 @@ class Download: { "progress_hooks": [self._progress_hook], "postprocessor_hooks": [self._postprocessor_hook], + "post_hooks": [self.post_hooks], } ) diff --git a/app/library/DownloadQueue.py b/app/library/DownloadQueue.py index 7aac9fba..491910ff 100644 --- a/app/library/DownloadQueue.py +++ b/app/library/DownloadQueue.py @@ -21,7 +21,7 @@ from .Events import EventBus, Events from .ItemDTO import ItemDTO from .Presets import Presets from .Singleton import Singleton -from .Utils import calc_download_path, extract_info, is_downloaded +from .Utils import arg_converter, calc_download_path, extract_info, is_downloaded from .YTDLPOpts import YTDLPOpts LOG = logging.getLogger("DownloadQueue") @@ -177,6 +177,7 @@ class DownloadQueue(metaclass=Singleton): cookies: str = "", template: str = "", extras: dict | None = None, + cli: str = "", already=None, ): """ @@ -190,9 +191,9 @@ class DownloadQueue(metaclass=Singleton): cookies (str): The cookies to use for the download. template (str): The output template to use for the download. extras (dict): The extra information to add to the download. + cli (str): The yt-dlp command line options to use for the download. already (set): The set of already downloaded items. - Returns: dict: The status of the operation. @@ -244,6 +245,7 @@ class DownloadQueue(metaclass=Singleton): cookies=cookies, template=template, extras=extras, + cli=cli, already=already, ) ) @@ -320,6 +322,7 @@ class DownloadQueue(metaclass=Singleton): is_live=is_live, live_in=live_in, options=options, + cli=cli, extras=extras, ) @@ -352,6 +355,7 @@ class DownloadQueue(metaclass=Singleton): cookies=cookies, template=template, extras=extras, + cli=cli, already=already, ) @@ -365,12 +369,39 @@ class DownloadQueue(metaclass=Singleton): config: dict | None = None, cookies: str = "", template: str = "", + cli: str = "", extras: dict | None = None, already=None, ): + """ + Add an item to the download queue. + + Args: + url (str): The url to be added to the queue. + preset (str): The preset to be used for the download. + folder (str): The folder to save the download to. + config (dict): The yt-dlp config to be used for the download. + cookies (str): The cookies to be used for the download. + template (str): The template to be used for the download. + cli (str): The yt-dlp cli options to be used for the download. + extras (dict): Extra data to be added to the download + already (set): Set of already downloaded items. + + Returns: + dict[str, str]: The status of the download. + { "status": "text" } + + """ _preset = Presets.get_instance().get(name=preset) config = config if config else {} + if cli: + try: + config = arg_converter(args=cli, level=True) + except Exception as e: + LOG.error(f"Invalid cli options '{cli}'. {e!s}") + return {"status": "error", "msg": f"Invalid cli options '{cli}'. {e!s}"} + folder = str(folder) if folder else "" if not extras: extras = {} @@ -390,7 +421,7 @@ class DownloadQueue(metaclass=Singleton): cookie_file = os.path.join(self.config.temp_path, f"c_{uuid.uuid4().hex}.txt") LOG.info( - f"Adding 'URL: {url}' to 'Folder: {filePath}' with 'Preset: {preset}' 'Naming: {template}', 'Cookies: {len(cookies)}/chars' 'YTConfig: {config}' 'Extras: {extras}'." + f"Adding 'URL: {url}' to 'Folder: {filePath}' with 'Preset: {preset}' 'Naming: {template}', 'Cookies: {len(cookies)}/chars' 'CLI: {cli}' 'Extras: {extras}'." ) if isinstance(config, str): @@ -486,6 +517,7 @@ class DownloadQueue(metaclass=Singleton): template=template, already=already, extras=extras, + cli=cli, ) async def cancel(self, ids: list[str]) -> dict[str, str]: diff --git a/app/library/HttpAPI.py b/app/library/HttpAPI.py index 98e91349..74387c55 100644 --- a/app/library/HttpAPI.py +++ b/app/library/HttpAPI.py @@ -473,10 +473,9 @@ class HttpAPI(Common): except Exception as e: err = str(e).strip() err = err.split("\n")[-1] if "\n" in err else err - LOG.error(f"Failed to convert args. '{err}'.") - LOG.exception(e) + err = err.replace("main.py: error: ", "").strip().capitalize() return web.json_response( - data={"error": f"Failed to convert args. '{err}'."}, status=web.HTTPBadRequest.status_code + data={"error": f"Failed to command line arguments for yt-dlp. '{err}'."}, status=web.HTTPBadRequest.status_code ) @route("GET", "api/yt-dlp/url/info") diff --git a/app/library/ItemDTO.py b/app/library/ItemDTO.py index a3020545..36bcd3c0 100644 --- a/app/library/ItemDTO.py +++ b/app/library/ItemDTO.py @@ -27,7 +27,6 @@ class ItemDTO: temp_dir: str | None = None status: str | None = None cookies: str | None = None - config: dict = field(default_factory=dict) template: str | None = None template_chapter: str | None = None timestamp: float = field(default_factory=lambda: time.time_ns()) @@ -37,6 +36,7 @@ class ItemDTO: file_size: int | None = None options: dict = field(default_factory=dict) extras: dict = field(default_factory=dict) + cli: str = "" # yt-dlp injected fields. tmpfilename: str | None = None @@ -49,12 +49,13 @@ class ItemDTO: speed: str | None = None eta: str | None = None - # DEPRECATED: These fields are deprecated and will be removed in the future. + # @Deprecated: These fields are deprecated and will be removed in the future. thumbnail: str | None = None ytdlp_cookies: str | None = None ytdlp_config: dict = field(default_factory=dict) output_template: str | None = None output_template_chapter: str | None = None + config: dict = field(default_factory=dict) def serialize(self) -> dict: deprecated: tuple = ( @@ -65,6 +66,7 @@ class ItemDTO: "ytdlp_config", "output_template", "output_template_chapter", + "config", ) if self.thumbnail and "thumbnail" not in self.extras: diff --git a/app/library/Utils.py b/app/library/Utils.py index 5f8acdcd..155f1ceb 100644 --- a/app/library/Utils.py +++ b/app/library/Utils.py @@ -27,6 +27,7 @@ REMOVE_KEYS: list = [ "outtmpl": "-o, --output", "progress_hooks": "--progress_hooks", "postprocessor_hooks": "--postprocessor_hooks", + "post_hooks": "--post_hooks", "download_archive": "--download_archive", }, { @@ -50,6 +51,8 @@ REMOVE_KEYS: list = [ "forcejson": "-j, --dump-json", "print_to_file": "--print-to-file", "cookiesfrombrowser": "--cookies-from-browser", + }, + { "cookiefile": "--cookies", }, ] @@ -505,25 +508,22 @@ def arg_converter( if "postprocessors" in diff: diff["postprocessors"] = [pp for pp in diff["postprocessors"] if pp not in default_opts["postprocessors"]] - if "match_filter" in diff: - import inspect - - matchFilter = inspect.getclosurevars(diff["match_filter"].func).nonlocals["filters"] - if isinstance(matchFilter, set): - diff["match_filter"] = {"filters": list(matchFilter)} - if "_warnings" in diff: diff.pop("_warnings", None) - if level: + if level is True or isinstance(level, int): bad_options = {} - level = len(REMOVE_KEYS) if not isinstance(level, int) else level + if isinstance(level, bool) or not isinstance(level, int): + level = len(REMOVE_KEYS) for i, item in enumerate(REMOVE_KEYS): if i > level: break + bad_options.update(item.items()) + LOG.debug("Removed %i the following options: '%s'.", level, ", ".join(bad_options.values())) + for key in diff.copy(): if key not in bad_options: continue @@ -536,6 +536,13 @@ def arg_converter( if dumps is True: from .encoder import Encoder + if "match_filter" in diff: + import inspect + + matchFilter = inspect.getclosurevars(diff["match_filter"].func).nonlocals["filters"] + if isinstance(matchFilter, set): + diff["match_filter"] = {"filters": list(matchFilter)} + return json.loads(json.dumps(diff, cls=Encoder)) return diff diff --git a/app/library/YTDLPOpts.py b/app/library/YTDLPOpts.py index a85ab6ce..2a87ca91 100644 --- a/app/library/YTDLPOpts.py +++ b/app/library/YTDLPOpts.py @@ -118,6 +118,7 @@ class YTDLPOpts(metaclass=Singleton): "temp": self._config.temp_path, } + # @Deprecated - To be removed in future versions. if not preset.cli: if preset.postprocessors and isinstance(preset.postprocessors, list) and len(preset.postprocessors) > 0: self._preset_opts["postprocessors"] = preset.postprocessors @@ -165,6 +166,7 @@ class YTDLPOpts(metaclass=Singleton): if "format" in data and data["format"] in ["not_set", "default"]: data["format"] = None + # @Deprecated - To be removed in future versions, All the checks below this line are deprecated. if "daterange" in data and isinstance(data["daterange"], dict): from yt_dlp.utils import DateRange diff --git a/app/library/common.py b/app/library/common.py index 1fb524fd..2f26b4a9 100644 --- a/app/library/common.py +++ b/app/library/common.py @@ -28,7 +28,16 @@ class Common: self.default_preset = config.default_preset async def add( - self, url: str, preset: str, folder: str, cookies: str, config: dict, template: str, extras: dict | None = None + self, + url: str, + preset: str, + folder: str, + cookies: str, + # @deprecated: config: dict, + config: dict, + template: str, + extras: dict | None = None, + cli: str = "", ) -> dict[str, str]: """ Add an item to the download queue. @@ -41,6 +50,7 @@ class Common: config (dict): The yt-dlp config to be used for the download. template (str): The template to be used for the download. extras (dict): Extra data to be added to the download + cli (str): The yt-dlp cli options to be used for the download. Returns: dict[str, str]: The status of the download. @@ -57,6 +67,7 @@ class Common: cookies=cookies, config=config if isinstance(config, dict) else {}, template=template, + cli=cli, extras=extras, ) @@ -96,13 +107,14 @@ class Common: raise ValueError(msg) from e cli = item.get("cli") - if cli and len(cli) > 1: + if cli and len(cli) > 2: try: removed_options = [] - config = arg_converter(args=cli, level=True, removed_options=removed_options) + arg_converter(args=cli, level=True, removed_options=removed_options) if len(removed_options) > 0: LOG.warning("Removed the following options '%s'.", ", ".join(removed_options)) + config = {} except Exception as e: msg = f"Failed to parse yt-dlp cli options. {e!s}" raise ValueError(msg) from e @@ -114,5 +126,6 @@ class Common: "cookies": cookies, "config": config if isinstance(config, dict) else {}, "template": template, + "cli": cli if isinstance(cli, str) else "", "extras": extras if isinstance(extras, dict) else {}, } diff --git a/app/library/config.py b/app/library/config.py index 91b837ed..fb4c0ab3 100644 --- a/app/library/config.py +++ b/app/library/config.py @@ -332,7 +332,7 @@ class Config: removed_options = [] self.ytdl_options = arg_converter( args=ytdlp_cli_opts, - level=True, + level=1, removed_options=removed_options, ) diff --git a/ui/components/History.vue b/ui/components/History.vue index 147b4cd3..e14b655b 100644 --- a/ui/components/History.vue +++ b/ui/components/History.vue @@ -626,9 +626,9 @@ const reQueueItem = item => { url: item.url, preset: item.preset, folder: item.folder, - config: item.config, cookies: item.cookies, template: item.template, + cli: item?.cli, extras: extras }) }