danvics/ytptube
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

complete the migration to yt-dlp cli instead of JSON config

Browse source
This commit is contained in:
arabcoders 2025-03-31 22:47:54 +03:00
parent 77de30609b
commit b09bf14acd
10 changed files with 131 additions and 52 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
README.md
View file

@ -25,22 +25,47 @@ Web GUI for [yt-dlp](https://github.com/yt-dlp/yt-dlp) with playlist & channel s
* Support for curl_cffi, see [yt-dlp documentation](https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#impersonation)
* Support for both advanced and basic mode for WebUI.
* Bundled tools in container: curl-cffi, ffmpeg, ffprobe, aria2, rtmpdump, mkvtoolsnix, mp4box.
# Recommended basic `ytdlp.json` file settings
Your `/config/ytdlp.json` config should include the following basic options for optimal working conditions.
# Breaking changes
```json
{
"windowsfilenames": true,
"continue_dl": true,
"live_from_start": true,
"format_sort": [ "codec:avc:m4a" ],
}
```
Starting with versions tagged `*-20250330-*` I have deprecated the JSON yt-dlp config and all of it's related settings.
> [!NOTE]
> Note, the `format_sort`, forces YouTube to use x264 instead of vp9 codec, you can ignore it if you want. i prefer the media in x264.
To be frank, it was pain to manage and hard for users to understand how to map cli options to json options. So, I have
decided to just use the cli options directly. This means that you can now use the same options you would use in the command line
directly in the WebUI, presets, and tasks.
## `ytdlp.json` vs `ytdlp.cli`
I have also added a new `ytdlp.cli` file that will be used to store the global options for yt-dlp. This file is located in the `/config` directory
and will be used to store the global options for yt-dlp. This file is not required and presets can fill the gap for most of the
use cases. But, if you want to use the same options for all your downloads, you can use this file to store the global options.
So, if you have `ytdlp.cli` file it will take priority over the `ytdlp.json` file if both exists. As mitigation, I have implemented
fallback to `ytdlp.json` file if `ytdlp.cli` file is not found.
## Presets
I have deprecated the `JSON yt-dlp config` and `JSON yt-dlp Post-Processors` fields, and added new `Command arguments for yt-dlp`
field to the presets. This field will be used to store the command line options for yt-dlp. I have also have added fallback,
if you have the `JSON yt-dlp config/Post-Processors` field set, the logic is, if `Command arguments for yt-dlp` is set
it will take priority over the `JSON yt-dlp config/Post-Processors` field. If not set, it will fallback to the `JSON yt-dlp config/Post-Processors` field.
If you are adding new presets, the deprecated fields will not show up anymore, they will only show up if actually have content in them
and editing old preset. So, Please migrate your presets to the new format.
## Tasks
I have also removed the `JSON yt-dlp config` and replaced it with `Command arguments for yt-dlp` field. Sadly, there is
no fallback, and once you upgrade to any version after `*-20250330-*` your `tasks.json` file will be updated to remove the
`config` key. so, please make sure to backup your `tasks.json` file before upgrading.
## closing statement
I know it's a painful breaking change, but for the sake of maintainability and ease of use, I have decided to
to make it happen sooner than later, the `JSON yt-dlp config`, was hard to manage and some features weren't really working
as expected, for example `--match-filter` and `--date` arguments etc.
So, Starting with `*-2025040*-*` tagged versions, the all the fallbacks and backwards compatibility will be removed.
# Run using docker command
@ -240,25 +265,19 @@ A Docker image can be built locally (it will build the UI too):
docker build . -t ytptube
```
# ytdlp.json file
# ytdlp.cli file
The `config/ytdlp.json`, is a json file which can be used to alter the default `yt-dlp` config settings globally.
The `config/ytdlp.cli`, is a command line options file for `yt-dlp` it will be globally applied to all downloads.
We recommend not use this file for options that aren't **truly global**, everything that can be done via the `ytdlp.json` file
can be done via a preset, which only effects the download that uses it. Example of good basic `ytdlp.json` file.
We strongly recommend not use this file for options that aren't **truly global**, everything that can be done via the file
can also be done via the presets which is dynamic can be altered per download. Example of good global options are to be
used for all downloads are:
```json
{
"windowsfilenames": true,
"continue_dl": true,
"live_from_start": true,
"format_sort": [ "codec:avc:m4a" ],
}
```bash
--continue --windows-filenames --live-from-start
```
Everything else can be done via the presets, and it's more flexible and easier to manage. You can convert your
own yt-dlp command arguments into a preset using the box found in the presets add page. For reference, The options can be found
at [yt-dlp YoutubeDL.py](https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L214) file. And for the postprocessors at [yt-dlp postprocessor](https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/postprocessor).
Everything else can be done via the presets, and it's more flexible and easier to manage.
# Authentication
@ -275,7 +294,7 @@ What does the basic mode do? it hides the the following features from the WebUI.
### Header
It disables everything except the `theme switcher` and `reload` button.
It disables everything except the `settings button` and `reload` button.
### Add form

9
app/library/Download.py
View file

@ -114,8 +114,6 @@ class Download:
def _progress_hook(self, data: dict):
dataDict = {k: v for k, v in data.items() if k in self._ytdlp_fields}
LOG.debug("Status: %s", data)
if "finished" == data.get("status") and data.get("info_dict", {}).get("filename", None):
dataDict["filename"] = data["info_dict"]["filename"]
@ -137,6 +135,12 @@ class Download:
self.status_queue.put({"id": self.id, "status": "finished", "filename": filename})
def post_hooks(self, filename: str | None = None):
if not filename:
return
self.status_queue.put({"id": self.id, "filename": filename})
def _download(self):
try:
params = (
@ -180,6 +184,7 @@ class Download:
{
"progress_hooks": [self._progress_hook],
"postprocessor_hooks": [self._postprocessor_hook],
"post_hooks": [self.post_hooks],
}
)

38
app/library/DownloadQueue.py
View file

@ -21,7 +21,7 @@ from .Events import EventBus, Events
from .ItemDTO import ItemDTO
from .Presets import Presets
from .Singleton import Singleton
from .Utils import calc_download_path, extract_info, is_downloaded
from .Utils import arg_converter, calc_download_path, extract_info, is_downloaded
from .YTDLPOpts import YTDLPOpts
LOG = logging.getLogger("DownloadQueue")
@ -177,6 +177,7 @@ class DownloadQueue(metaclass=Singleton):
cookies: str = "",
template: str = "",
extras: dict | None = None,
cli: str = "",
already=None,
):
"""
@ -190,9 +191,9 @@ class DownloadQueue(metaclass=Singleton):
cookies (str): The cookies to use for the download.
template (str): The output template to use for the download.
extras (dict): The extra information to add to the download.
cli (str): The yt-dlp command line options to use for the download.
already (set): The set of already downloaded items.
Returns:
dict: The status of the operation.
@ -244,6 +245,7 @@ class DownloadQueue(metaclass=Singleton):
cookies=cookies,
template=template,
extras=extras,
cli=cli,
already=already,
)
)
@ -320,6 +322,7 @@ class DownloadQueue(metaclass=Singleton):
is_live=is_live,
live_in=live_in,
options=options,
cli=cli,
extras=extras,
)
@ -352,6 +355,7 @@ class DownloadQueue(metaclass=Singleton):
cookies=cookies,
template=template,
extras=extras,
cli=cli,
already=already,
)
@ -365,12 +369,39 @@ class DownloadQueue(metaclass=Singleton):
config: dict | None = None,
cookies: str = "",
template: str = "",
cli: str = "",
extras: dict | None = None,
already=None,
):
"""
Add an item to the download queue.
Args:
url (str): The url to be added to the queue.
preset (str): The preset to be used for the download.
folder (str): The folder to save the download to.
config (dict): The yt-dlp config to be used for the download.
cookies (str): The cookies to be used for the download.
template (str): The template to be used for the download.
cli (str): The yt-dlp cli options to be used for the download.
extras (dict): Extra data to be added to the download
already (set): Set of already downloaded items.
Returns:
dict[str, str]: The status of the download.
{ "status": "text" }
"""
_preset = Presets.get_instance().get(name=preset)
config = config if config else {}
if cli:
try:
config = arg_converter(args=cli, level=True)
except Exception as e:
LOG.error(f"Invalid cli options '{cli}'. {e!s}")
return {"status": "error", "msg": f"Invalid cli options '{cli}'. {e!s}"}
folder = str(folder) if folder else ""
if not extras:
extras = {}
@ -390,7 +421,7 @@ class DownloadQueue(metaclass=Singleton):
cookie_file = os.path.join(self.config.temp_path, f"c_{uuid.uuid4().hex}.txt")
LOG.info(
f"Adding 'URL: {url}' to 'Folder: {filePath}' with 'Preset: {preset}' 'Naming: {template}', 'Cookies: {len(cookies)}/chars' 'YTConfig: {config}' 'Extras: {extras}'."
f"Adding 'URL: {url}' to 'Folder: {filePath}' with 'Preset: {preset}' 'Naming: {template}', 'Cookies: {len(cookies)}/chars' 'CLI: {cli}' 'Extras: {extras}'."
)
if isinstance(config, str):
@ -486,6 +517,7 @@ class DownloadQueue(metaclass=Singleton):
template=template,
already=already,
extras=extras,
cli=cli,
)
async def cancel(self, ids: list[str]) -> dict[str, str]:

5
app/library/HttpAPI.py
View file

@ -473,10 +473,9 @@ class HttpAPI(Common):
except Exception as e:
err = str(e).strip()
err = err.split("\n")[-1] if "\n" in err else err
LOG.error(f"Failed to convert args. '{err}'.")
LOG.exception(e)
err = err.replace("main.py: error: ", "").strip().capitalize()
return web.json_response(
data={"error": f"Failed to convert args. '{err}'."}, status=web.HTTPBadRequest.status_code
data={"error": f"Failed to command line arguments for yt-dlp. '{err}'."}, status=web.HTTPBadRequest.status_code
)
@route("GET", "api/yt-dlp/url/info")

6
app/library/ItemDTO.py
View file

@ -27,7 +27,6 @@ class ItemDTO:
temp_dir: str | None = None
status: str | None = None
cookies: str | None = None
config: dict = field(default_factory=dict)
template: str | None = None
template_chapter: str | None = None
timestamp: float = field(default_factory=lambda: time.time_ns())
@ -37,6 +36,7 @@ class ItemDTO:
file_size: int | None = None
options: dict = field(default_factory=dict)
extras: dict = field(default_factory=dict)
cli: str = ""
# yt-dlp injected fields.
tmpfilename: str | None = None
@ -49,12 +49,13 @@ class ItemDTO:
speed: str | None = None
eta: str | None = None
# DEPRECATED: These fields are deprecated and will be removed in the future.
# @Deprecated: These fields are deprecated and will be removed in the future.
thumbnail: str | None = None
ytdlp_cookies: str | None = None
ytdlp_config: dict = field(default_factory=dict)
output_template: str | None = None
output_template_chapter: str | None = None
config: dict = field(default_factory=dict)
def serialize(self) -> dict:
deprecated: tuple = (
@ -65,6 +66,7 @@ class ItemDTO:
"ytdlp_config",
"output_template",
"output_template_chapter",
"config",
)
if self.thumbnail and "thumbnail" not in self.extras:

25
app/library/Utils.py
View file

@ -27,6 +27,7 @@ REMOVE_KEYS: list = [
"outtmpl": "-o, --output",
"progress_hooks": "--progress_hooks",
"postprocessor_hooks": "--postprocessor_hooks",
"post_hooks": "--post_hooks",
"download_archive": "--download_archive",
},
{
@ -50,6 +51,8 @@ REMOVE_KEYS: list = [
"forcejson": "-j, --dump-json",
"print_to_file": "--print-to-file",
"cookiesfrombrowser": "--cookies-from-browser",
},
{
"cookiefile": "--cookies",
},
]
@ -505,25 +508,22 @@ def arg_converter(
if "postprocessors" in diff:
diff["postprocessors"] = [pp for pp in diff["postprocessors"] if pp not in default_opts["postprocessors"]]
if "match_filter" in diff:
import inspect
matchFilter = inspect.getclosurevars(diff["match_filter"].func).nonlocals["filters"]
if isinstance(matchFilter, set):
diff["match_filter"] = {"filters": list(matchFilter)}
if "_warnings" in diff:
diff.pop("_warnings", None)
if level:
if level is True or isinstance(level, int):
bad_options = {}
level = len(REMOVE_KEYS) if not isinstance(level, int) else level
if isinstance(level, bool) or not isinstance(level, int):
level = len(REMOVE_KEYS)
for i, item in enumerate(REMOVE_KEYS):
if i > level:
break
bad_options.update(item.items())
LOG.debug("Removed %i the following options: '%s'.", level, ", ".join(bad_options.values()))
for key in diff.copy():
if key not in bad_options:
continue
@ -536,6 +536,13 @@ def arg_converter(
if dumps is True:
from .encoder import Encoder
if "match_filter" in diff:
import inspect
matchFilter = inspect.getclosurevars(diff["match_filter"].func).nonlocals["filters"]
if isinstance(matchFilter, set):
diff["match_filter"] = {"filters": list(matchFilter)}
return json.loads(json.dumps(diff, cls=Encoder))
return diff

2
app/library/YTDLPOpts.py
View file

@ -118,6 +118,7 @@ class YTDLPOpts(metaclass=Singleton):
"temp": self._config.temp_path,
}
# @Deprecated - To be removed in future versions.
if not preset.cli:
if preset.postprocessors and isinstance(preset.postprocessors, list) and len(preset.postprocessors) > 0:
self._preset_opts["postprocessors"] = preset.postprocessors
@ -165,6 +166,7 @@ class YTDLPOpts(metaclass=Singleton):
if "format" in data and data["format"] in ["not_set", "default"]:
data["format"] = None
# @Deprecated - To be removed in future versions, All the checks below this line are deprecated.
if "daterange" in data and isinstance(data["daterange"], dict):
from yt_dlp.utils import DateRange

19
app/library/common.py
View file

@ -28,7 +28,16 @@ class Common:
self.default_preset = config.default_preset
async def add(
self, url: str, preset: str, folder: str, cookies: str, config: dict, template: str, extras: dict | None = None
self,
url: str,
preset: str,
folder: str,
cookies: str,
# @deprecated: config: dict,
config: dict,
template: str,
extras: dict | None = None,
cli: str = "",
) -> dict[str, str]:
"""
Add an item to the download queue.
@ -41,6 +50,7 @@ class Common:
config (dict): The yt-dlp config to be used for the download.
template (str): The template to be used for the download.
extras (dict): Extra data to be added to the download
cli (str): The yt-dlp cli options to be used for the download.
Returns:
dict[str, str]: The status of the download.
@ -57,6 +67,7 @@ class Common:
cookies=cookies,
config=config if isinstance(config, dict) else {},
template=template,
cli=cli,
extras=extras,
)
@ -96,13 +107,14 @@ class Common:
raise ValueError(msg) from e
cli = item.get("cli")
if cli and len(cli) > 1:
if cli and len(cli) > 2:
try:
removed_options = []
config = arg_converter(args=cli, level=True, removed_options=removed_options)
arg_converter(args=cli, level=True, removed_options=removed_options)
if len(removed_options) > 0:
LOG.warning("Removed the following options '%s'.", ", ".join(removed_options))
config = {}
except Exception as e:
msg = f"Failed to parse yt-dlp cli options. {e!s}"
raise ValueError(msg) from e
@ -114,5 +126,6 @@ class Common:
"cookies": cookies,
"config": config if isinstance(config, dict) else {},
"template": template,
"cli": cli if isinstance(cli, str) else "",
"extras": extras if isinstance(extras, dict) else {},
}

2
app/library/config.py
View file

@ -332,7 +332,7 @@ class Config:
removed_options = []
self.ytdl_options = arg_converter(
args=ytdlp_cli_opts,
level=True,
level=1,
removed_options=removed_options,
)

2
ui/components/History.vue
View file

@ -626,9 +626,9 @@ const reQueueItem = item => {
url: item.url,
preset: item.preset,
folder: item.folder,
config: item.config,
cookies: item.cookies,
template: item.template,
cli: item?.cli,
extras: extras
})
}