Compare commits

..

1 commit

Author SHA1 Message Date
ArabCoders
ca4431423f Update yt-dlp to 2026.6.9 2026-06-10 01:02:03 +00:00
147 changed files with 4188 additions and 5868 deletions

View file

@ -15,7 +15,7 @@ on:
env: env:
BUN_VERSION: latest BUN_VERSION: latest
NODE_VERSION: 22 NODE_VERSION: 20
PYTHON_VERSION: "3.13" PYTHON_VERSION: "3.13"
jobs: jobs:

View file

@ -29,7 +29,7 @@ env:
DOCKERHUB_SLUG: arabcoders/ytptube DOCKERHUB_SLUG: arabcoders/ytptube
GHCR_SLUG: ghcr.io/arabcoders/ytptube GHCR_SLUG: ghcr.io/arabcoders/ytptube
BUN_VERSION: latest BUN_VERSION: latest
NODE_VERSION: 22 NODE_VERSION: 20
PYTHON_VERSION: "3.13" PYTHON_VERSION: "3.13"
jobs: jobs:

View file

@ -39,7 +39,7 @@ jobs:
env: env:
PYTHON_VERSION: "3.13" PYTHON_VERSION: "3.13"
BUN_VERSION: latest BUN_VERSION: latest
NODE_VERSION: 22 NODE_VERSION: 20
TAG_NAME: ${{ github.event.inputs.tag || github.ref_name }} TAG_NAME: ${{ github.event.inputs.tag || github.ref_name }}
steps: steps:

70
API.md
View file

@ -101,7 +101,6 @@ This document describes the available endpoints and their usage. All endpoints r
- [POST /api/notifications/test](#post-apinotificationstest) - [POST /api/notifications/test](#post-apinotificationstest)
- [GET /api/yt-dlp/options](#get-apiyt-dlpoptions) - [GET /api/yt-dlp/options](#get-apiyt-dlpoptions)
- [GET /api/system/configuration](#get-apisystemconfiguration) - [GET /api/system/configuration](#get-apisystemconfiguration)
- [GET /api/system/folders](#get-apisystemfolders)
- [GET /api/system/diagnostics](#get-apisystemdiagnostics) - [GET /api/system/diagnostics](#get-apisystemdiagnostics)
- [GET /api/system/limits](#get-apisystemlimits) - [GET /api/system/limits](#get-apisystemlimits)
- [POST /api/system/terminal](#post-apisystemterminal) - [POST /api/system/terminal](#post-apisystemterminal)
@ -700,16 +699,10 @@ GET /api/history?type=queue&status=pending&order=ASC
This endpoint returns the current state of active downloads from memory. This endpoint returns the current state of active downloads from memory.
**Query Parameters**:
- `limit` (optional): Override the configured queue display limit for this request. `0` means unlimited.
**Response**: **Response**:
```json ```json
{ {
"history_count": 0, // total number of completed items in history "history_count": 0, // total number of completed items in history
"queue_count": 250, // total number of queued items
"queue_loaded": 100, // number of queued items included in this response
"queue_limit": 100, // effective display limit, 0 means unlimited
"queue":{ "queue":{
"id": "abc123", "id": "abc123",
"url": "https://example.com/video", "url": "https://example.com/video",
@ -2635,38 +2628,55 @@ or an error:
--- ---
### GET /api/system/configuration ### GET /api/system/configuration
**Purpose**: Retrieve system configuration. **Purpose**: Retrieve comprehensive system configuration including app settings, presets, download fields, queue status, and folder structure.
**Response**: **Response**:
```json ```json
{ {
"app": {...}, "app": {
"presets": [...], "version": "...",
"dl_fields": [...], "download_path": "/path/to/downloads",
"paused": false, "base_path": "/",
"history_count": 150 ...
},
"presets": [
{
"id": 1,
"name": "default",
"description": "...",
...
} }
``` ],
"dl_fields": [
---
### GET /api/system/folders
**Purpose**: List child directories for a given relative path within the download directory.
**Query Parameters**:
- `path=<relative-path>` (optional, default: root) - Relative path within the download directory.
**Response**:
```json
{ {
"path": "videos", "id": 1,
"folders": ["archive", "shorts", "2024"] "name": "Title",
"field": "title",
"kind": "text",
...
}
],
"paused": false,
"folders": [
{"name": "folder1", "path": "folder1"},
{"name": "folder2", "path": "folder2"}
],
"history_count": 150,
"queue": [
{
"id": "abc123",
"url": "https://example.com/video",
"status": "pending",
...
}
]
} }
``` ```
**Notes**: **Notes**:
- Results are cached server-side for a short time. - This endpoint combines multiple data sources into a single response for efficient initialization
- Non-existent paths return an empty folder list. - The `folders` array includes available download folders up to the configured depth limit
- The `queue` array contains active download items
--- ---
@ -3468,7 +3478,7 @@ Emitted when a download item is moved between queue and history.
"data": { "data": {
"id": "abc123", "id": "abc123",
"from": "queue", "from": "queue",
"to": "history" "to": "done"
} }
} }
``` ```

27
FAQ.md
View file

@ -3,9 +3,6 @@
Certain configuration values can be set via environment variables, using the `-e` parameter on the docker command line, Certain configuration values can be set via environment variables, using the `-e` parameter on the docker command line,
or the `environment:` section in `compose.yaml` file. or the `environment:` section in `compose.yaml` file.
<details>
<summary>Click to expand</summary>
| Environment Variable | Description | Default | | Environment Variable | Description | Default |
| ------------------------------- | ------------------------------------------------------------------- | --------------------- | | ------------------------------- | ------------------------------------------------------------------- | --------------------- |
| TZ | The timezone to use for the application | `(not_set)` | | TZ | The timezone to use for the application | `(not_set)` |
@ -47,10 +44,10 @@ or the `environment:` section in `compose.yaml` file.
| YTP_FLARESOLVERR_CACHE_TTL | The cache TTL (in seconds) for FlareSolverr solutions | `600` | | YTP_FLARESOLVERR_CACHE_TTL | The cache TTL (in seconds) for FlareSolverr solutions | `600` |
| YTP_BASE_PATH | Set this if you are serving YTPTube from sub-folder | `/` | | YTP_BASE_PATH | Set this if you are serving YTPTube from sub-folder | `/` |
| YTP_PREVENT_LIVE_PREMIERE | Prevents the initial youtube premiere stream from being downloaded | `false` | | YTP_PREVENT_LIVE_PREMIERE | Prevents the initial youtube premiere stream from being downloaded | `false` |
| YTP_QUEUE_DISPLAY_LIMIT | Max queued downloads returned to the UI. `0` = unlimited | `100` |
| YTP_LIVE_PREMIERE_BUFFER | buffer time in minutes to add to video duration | `5` | | YTP_LIVE_PREMIERE_BUFFER | buffer time in minutes to add to video duration | `5` |
| YTP_TASKS_HANDLER_TIMER | The cron expression for the tasks handler timer | `15 */1 * * *` | | YTP_TASKS_HANDLER_TIMER | The cron expression for the tasks handler timer | `15 */1 * * *` |
| YTP_TEMP_DISABLED | Disable temp files handling. | `false` | | YTP_TEMP_DISABLED | Disable temp files handling. | `false` |
| YTP_DOWNLOAD_PATH_DEPTH | How many subdirectories to show in auto complete. | `1` |
| YTP_ALLOW_INTERNAL_URLS | Allow requests to internal URLs | `false` | | YTP_ALLOW_INTERNAL_URLS | Allow requests to internal URLs | `false` |
| YTP_SIMPLE_MODE | Switch default interface to Simple mode. | `false` | | YTP_SIMPLE_MODE | Switch default interface to Simple mode. | `false` |
| YTP_STATIC_UI_PATH | Path to custom static UI files. | `(not_set)` | | YTP_STATIC_UI_PATH | Path to custom static UI files. | `(not_set)` |
@ -64,19 +61,19 @@ or the `environment:` section in `compose.yaml` file.
| YTP_THUMB_GENERATE | Enable ffmpeg thumbnail generation when no local thumbnail exists. | `true` | | YTP_THUMB_GENERATE | Enable ffmpeg thumbnail generation when no local thumbnail exists. | `true` |
| YTP_THUMB_SIDECAR | Save generated thumbnails next to media instead of temp cache. | `false` | | YTP_THUMB_SIDECAR | Save generated thumbnails next to media instead of temp cache. | `false` |
| YTP_DISABLE_EXEC | Strip some dangerous yt-dlp options. | `false` | | YTP_DISABLE_EXEC | Strip some dangerous yt-dlp options. | `false` |
</details>
> [!NOTE] > [!NOTE]
> To raise the worker limit for a specific extractor, set an env variable using this format: `YTP_MAX_WORKERS_FOR_<EXTRACTOR_NAME>` > To raise the maximum workers for specific extractor, you need to add a ENV variable that follows the pattern `YTP_MAX_WORKERS_FOR_<EXTRACTOR_NAME>`.
> The extractor name must be uppercase. You can find the extractor name in the download logs. This value cannot be > The extractor name must be in uppercase, to know the extractor name, check the log for the specific extractor used for the download.
> higher than `YTP_MAX_WORKERS`; higher values are ignored. > The limit should not exceed the `YTP_MAX_WORKERS` value as it will be ignored.
>
> `YTP_SIMPLE_MODE=true` only applies when the browser has no saved layout choice yet. Users can still choose a layout in > [!IMPORTANT]
> WebUI Settings. `/?simple=1` forces and saves Simple for that browser. > The env variable `YTP_SIMPLE_MODE` only control what being displayed for first time visitor, the users can still switch between the two modes via the WebUI settings page.
>
> `YTP_AUTO_CLEAR_HISTORY_DAYS` `0` days means no automatic clearing of the download history. lowest value that will ## Notes about YTP_AUTO_CLEAR_HISTORY_DAYS
> trigger the clearing is `1` day. This setting will **NOT** delete the downloaded files, it will only clear the
> history from the database. - `0` days means no automatic clearing of the download history. lowest value that will trigger the clearing is `1` day.
- This setting will **NOT** delete the downloaded files, it will only clear the history from the database.
# Browser extensions & bookmarklets # Browser extensions & bookmarklets

View file

@ -6,16 +6,13 @@ from app.features.conditions.migration import Migration
from app.library.Singleton import Singleton from app.library.Singleton import Singleton
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable, Iterable from collections.abc import AsyncGenerator, Iterable
from contextlib import AbstractAsyncContextManager
from sqlalchemy.engine.result import Result from sqlalchemy.engine.result import Result
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select from sqlalchemy.sql.selectable import Select
SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]]
from sqlalchemy import delete, func, or_, select from sqlalchemy import delete, func, or_, select
from app.features.conditions.models import ConditionModel from app.features.conditions.models import ConditionModel
@ -25,26 +22,10 @@ from app.library.log import get_logger
LOG = get_logger() LOG = get_logger()
def _model_from_payload(payload: dict[str, Any]) -> ConditionModel:
model = ConditionModel()
for key, value in payload.items():
if not hasattr(model, key):
msg = f"'{key}' is an invalid keyword argument for ConditionModel"
raise TypeError(msg)
setattr(model, key, value)
return model
def _coerce_model(payload: ConditionModel | dict[str, Any]) -> ConditionModel:
if isinstance(payload, ConditionModel):
return payload
return _model_from_payload(payload)
class ConditionsRepository(metaclass=Singleton): class ConditionsRepository(metaclass=Singleton):
def __init__(self, session: SessionFactory | None = None) -> None: def __init__(self, session: AsyncGenerator[AsyncSession] | None = None) -> None:
self._migrated = False self._migrated = False
self.session: SessionFactory = session or get_session self.session: AsyncGenerator[AsyncSession] = session or get_session
async def run_migrations(self) -> None: async def run_migrations(self) -> None:
if self._migrated: if self._migrated:
@ -57,7 +38,7 @@ class ConditionsRepository(metaclass=Singleton):
def get_instance() -> ConditionsRepository: def get_instance() -> ConditionsRepository:
return ConditionsRepository() return ConditionsRepository()
async def all(self) -> list[ConditionModel]: async def list(self) -> list[ConditionModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[ConditionModel]] = await session.execute( result: Result[tuple[ConditionModel]] = await session.execute(
select(ConditionModel).order_by(ConditionModel.priority.desc(), ConditionModel.name.asc()) select(ConditionModel).order_by(ConditionModel.priority.desc(), ConditionModel.name.asc())
@ -112,11 +93,11 @@ class ConditionsRepository(metaclass=Singleton):
result: Result[tuple[ConditionModel]] = await session.execute(query.limit(1)) result: Result[tuple[ConditionModel]] = await session.execute(query.limit(1))
return result.scalar_one_or_none() return result.scalar_one_or_none()
async def create(self, payload: ConditionModel | dict[str, Any]) -> ConditionModel: async def create(self, payload: ConditionModel | dict) -> ConditionModel:
async with self.session() as session: async with self.session() as session:
model = _coerce_model(payload) model: ConditionModel = ConditionModel(**payload) if isinstance(payload, dict) else payload
if model.id is not None: if model.id is not None:
model.id = None # ty: ignore model.id = None
if await self.get_by_name(name=model.name) is not None: if await self.get_by_name(name=model.name) is not None:
msg: str = f"Condition with name '{model.name}' already exists." msg: str = f"Condition with name '{model.name}' already exists."
@ -179,11 +160,13 @@ class ConditionsRepository(metaclass=Singleton):
await session.commit() await session.commit()
return model return model
async def replace_all(self, items: Iterable[dict[str, Any] | ConditionModel]) -> list[ConditionModel]: async def replace_all(self, items: Iterable[dict | ConditionModel]) -> list[ConditionModel]:
async with self.session() as session: async with self.session() as session:
try: try:
await session.execute(delete(ConditionModel)) await session.execute(delete(ConditionModel))
models: list[ConditionModel] = [_coerce_model(item) for item in items] models: list[ConditionModel] = [
ConditionModel(**item) if isinstance(item, dict) else item for item in items
]
session.add_all(models) session.add_all(models)
await session.commit() await session.commit()
except Exception: except Exception:

View file

@ -131,12 +131,6 @@ async def conditions_test(request: Request, encoder: Encoder, cache: Cache, conf
status=web.HTTPInternalServerError.status_code, status=web.HTTPInternalServerError.status_code,
) )
if not isinstance(data, dict):
return web.json_response(
data={"error": "Failed to extract video info."},
status=web.HTTPInternalServerError.status_code,
)
try: try:
from app.features.ytdlp.mini_filter import match_str from app.features.ytdlp.mini_filter import match_str

View file

@ -1,10 +1,10 @@
from collections.abc import Iterable from collections.abc import Iterable
from numbers import Number
from aiohttp import web from aiohttp import web
from app.features.conditions.models import ConditionModel from app.features.conditions.models import ConditionModel
from app.features.conditions.repository import ConditionsRepository from app.features.conditions.repository import ConditionsRepository
from app.features.conditions.schemas import Condition
from app.features.ytdlp.mini_filter import match_str from app.features.ytdlp.mini_filter import match_str
from app.library.Events import EventBus, Events from app.library.Events import EventBus, Events
from app.library.log import get_logger from app.library.log import get_logger
@ -13,7 +13,7 @@ from app.library.Singleton import Singleton
LOG = get_logger() LOG = get_logger()
def _ignored_identifiers(ignore_conditions: Iterable[str | int | float] | None) -> tuple[set[str], bool]: def _ignored_identifiers(ignore_conditions: Iterable[str | Number] | None) -> tuple[set[str], bool]:
ignored: set[str] = set() ignored: set[str] = set()
ignore_all = False ignore_all = False
@ -21,7 +21,7 @@ def _ignored_identifiers(ignore_conditions: Iterable[str | int | float] | None)
return ignored, ignore_all return ignored, ignore_all
for value in ignore_conditions: for value in ignore_conditions:
if isinstance(value, bool) or not isinstance(value, (str, int, float)): if isinstance(value, bool) or not isinstance(value, (str, Number)):
continue continue
identifier = str(value).strip() identifier = str(value).strip()
@ -55,7 +55,7 @@ class Conditions(metaclass=Singleton):
EventBus.get_instance().subscribe(Events.STARTED, handle_event, "ConditionsRepository.run_migrations") EventBus.get_instance().subscribe(Events.STARTED, handle_event, "ConditionsRepository.run_migrations")
async def get_all(self) -> list[ConditionModel]: async def get_all(self) -> list[ConditionModel]:
return await self._repo.all() return await self._repo.list()
async def save(self, item: ConditionModel | dict) -> ConditionModel: async def save(self, item: ConditionModel | dict) -> ConditionModel:
""" """
@ -80,7 +80,7 @@ class Conditions(metaclass=Singleton):
if item.id is None or 0 == item.id: if item.id is None or 0 == item.id:
model = await repo.create(item) model = await repo.create(item)
else: else:
model = await repo.update(item.id, Condition.model_validate(item).model_dump()) model = await repo.update(item.id, item.serialize())
except KeyError as exc: except KeyError as exc:
raise ValueError(str(exc)) from exc raise ValueError(str(exc)) from exc
@ -113,15 +113,13 @@ class Conditions(metaclass=Singleton):
repo = self._repo repo = self._repo
return await repo.get(identifier) return await repo.get(identifier)
async def match( async def match(self, info: dict, ignore_conditions: Iterable[str | Number] | None = None) -> ConditionModel | None:
self, info: dict, ignore_conditions: Iterable[str | int | float] | None = None
) -> ConditionModel | None:
""" """
Check if any condition matches the info dict. Check if any condition matches the info dict.
Args: Args:
info (dict): The info dict to check. info (dict): The info dict to check.
ignore_conditions (Iterable[str | int | float] | None): Condition ids or names to skip for this match. ignore_conditions (Iterable[str | Number] | None): Condition ids or names to skip for this match.
Returns: Returns:
Condition|None: The condition if found, None otherwise. Condition|None: The condition if found, None otherwise.
@ -135,7 +133,7 @@ class Conditions(metaclass=Singleton):
return None return None
repo = self._repo repo = self._repo
items: list[ConditionModel] = await repo.all() items: list[ConditionModel] = await repo.list()
if len(items) < 1: if len(items) < 1:
return None return None
@ -189,7 +187,7 @@ class Conditions(metaclass=Singleton):
if not info or not isinstance(info, dict) or len(info) < 1: if not info or not isinstance(info, dict) or len(info) < 1:
return None return None
if not (item := await self.get(str(identifier))) or not item.enabled or not item.filter: if not (item := await self.get(identifier)) or not item.enabled or not item.filter:
return None return None
return item if match_str(item.filter, info) else None return item if match_str(item.filter, info) else None

View file

@ -6,7 +6,6 @@ import pytest
import pytest_asyncio import pytest_asyncio
from types import SimpleNamespace from types import SimpleNamespace
import pytest import pytest
from typing import Any
from aiohttp import web from aiohttp import web
from aiohttp.test_utils import make_mocked_request from aiohttp.test_utils import make_mocked_request
@ -40,13 +39,13 @@ async def repo():
def _json_request(path: str, payload: object) -> web.Request: def _json_request(path: str, payload: object) -> web.Request:
mock_request: Any = make_mocked_request("POST", path) request = make_mocked_request("POST", path)
async def _json() -> object: async def _json() -> object:
return payload return payload
mock_request.json = _json request.json = _json # type: ignore[attr-defined]
return mock_request return request
class TestAllowInternalUrlsScope: class TestAllowInternalUrlsScope:
@ -209,7 +208,7 @@ class TestConditionsRepository:
await repo.create({"name": "A", "filter": "test", "priority": 1}) await repo.create({"name": "A", "filter": "test", "priority": 1})
await repo.create({"name": "C", "filter": "test", "priority": 2}) await repo.create({"name": "C", "filter": "test", "priority": 2})
items = await repo.all() items = await repo.list()
assert items[0].name == "C", "Highest priority should be first" assert items[0].name == "C", "Highest priority should be first"
assert items[1].name == "A", "Same priority sorted alphabetically" assert items[1].name == "A", "Same priority sorted alphabetically"
@ -241,6 +240,6 @@ class TestConditionsRepository:
assert len(result) == 2, "Should create 2 new conditions" assert len(result) == 2, "Should create 2 new conditions"
all_items = await repo.all() all_items = await repo.list()
assert len(all_items) == 2, "Should only have new conditions" assert len(all_items) == 2, "Should only have new conditions"
assert all_items[0].name in ["New 1", "New 2"], "Should only have new items" assert all_items[0].name in ["New 1", "New 2"], "Should only have new items"

View file

@ -3,7 +3,7 @@ from __future__ import annotations
from datetime import UTC, datetime from datetime import UTC, datetime
from sqlalchemy import DateTime as SQLADateTime from sqlalchemy import DateTime as SQLADateTime
from sqlalchemy import Dialect, TypeDecorator from sqlalchemy import TypeDecorator
from sqlalchemy.orm import DeclarativeBase from sqlalchemy.orm import DeclarativeBase
@ -20,18 +20,16 @@ class UTCDateTime(TypeDecorator):
impl = SQLADateTime impl = SQLADateTime
cache_ok = True cache_ok = True
def process_bind_param(self, value: datetime | None, dialect: Dialect) -> datetime | None: def process_bind_param(self, value: datetime | None, _dialect) -> datetime | None:
"""Convert datetime to UTC before storing.""" """Convert datetime to UTC before storing."""
_ = dialect
if value is not None: if value is not None:
if value.tzinfo is None: if value.tzinfo is None:
return value.replace(tzinfo=UTC) return value.replace(tzinfo=UTC)
return value.astimezone(UTC).replace(tzinfo=None) return value.astimezone(UTC).replace(tzinfo=None)
return value return value
def process_result_value(self, value: datetime | None, dialect: Dialect) -> datetime | None: def process_result_value(self, value: datetime | None, _dialect) -> datetime | None:
"""Ensure datetime is timezone-aware (UTC) when loading.""" """Ensure datetime is timezone-aware (UTC) when loading."""
_ = dialect
if value is not None and value.tzinfo is None: if value is not None and value.tzinfo is None:
return value.replace(tzinfo=UTC) return value.replace(tzinfo=UTC)
return value return value

View file

@ -1,6 +1,6 @@
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING, Any, cast from typing import TYPE_CHECKING, Any
from sqlalchemy import delete, func, or_, select from sqlalchemy import delete, func, or_, select
@ -11,39 +11,20 @@ from app.library.log import get_logger
from app.library.Singleton import Singleton from app.library.Singleton import Singleton
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable, Iterable from collections.abc import AsyncGenerator, Iterable
from contextlib import AbstractAsyncContextManager
from sqlalchemy.engine.result import Result from sqlalchemy.engine.result import Result
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select from sqlalchemy.sql.selectable import Select
SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]]
LOG = get_logger() LOG = get_logger()
def _model_from_payload(payload: dict[str, Any]) -> DLFieldModel:
model = DLFieldModel()
for key, value in payload.items():
if not hasattr(model, key):
msg = f"'{key}' is an invalid keyword argument for DLFieldModel"
raise TypeError(msg)
setattr(model, key, value)
return model
def _coerce_model(payload: DLFieldModel | dict[str, Any]) -> DLFieldModel:
if isinstance(payload, DLFieldModel):
return payload
return _model_from_payload(payload)
class DLFieldsRepository(metaclass=Singleton): class DLFieldsRepository(metaclass=Singleton):
def __init__(self, session: SessionFactory | None = None) -> None: def __init__(self, session: AsyncGenerator[AsyncSession] | None = None) -> None:
self._migrated = False self._migrated = False
self.session: SessionFactory = session or get_session self.session: AsyncGenerator[AsyncSession] = session or get_session
async def run_migrations(self) -> None: async def run_migrations(self) -> None:
if self._migrated: if self._migrated:
@ -56,7 +37,7 @@ class DLFieldsRepository(metaclass=Singleton):
def get_instance() -> DLFieldsRepository: def get_instance() -> DLFieldsRepository:
return DLFieldsRepository() return DLFieldsRepository()
async def all(self) -> list[DLFieldModel]: async def list(self) -> list[DLFieldModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[DLFieldModel]] = await session.execute( result: Result[tuple[DLFieldModel]] = await session.execute(
select(DLFieldModel).order_by(DLFieldModel.order.asc(), DLFieldModel.name.asc()) select(DLFieldModel).order_by(DLFieldModel.order.asc(), DLFieldModel.name.asc())
@ -109,9 +90,9 @@ class DLFieldsRepository(metaclass=Singleton):
result: Result[tuple[DLFieldModel]] = await session.execute(query.limit(1)) result: Result[tuple[DLFieldModel]] = await session.execute(query.limit(1))
return result.scalar_one_or_none() return result.scalar_one_or_none()
async def create(self, payload: DLFieldModel | dict[str, Any]) -> DLFieldModel: async def create(self, payload: DLFieldModel | dict) -> DLFieldModel:
async with self.session() as session: async with self.session() as session:
model = _coerce_model(payload) model: DLFieldModel = DLFieldModel(**payload) if isinstance(payload, dict) else payload
if await self.get_by_name(name=model.name) is not None: if await self.get_by_name(name=model.name) is not None:
msg: str = f"DL field with name '{model.name}' already exists." msg: str = f"DL field with name '{model.name}' already exists."
@ -172,16 +153,13 @@ class DLFieldsRepository(metaclass=Singleton):
await session.commit() await session.commit()
return model return model
async def replace_all(self, items: Iterable[dict[str, Any] | DLFieldModel]) -> list[DLFieldModel]: async def replace_all(self, items: Iterable[dict | DLFieldModel]) -> list[DLFieldModel]:
async with self.session() as session: async with self.session() as session:
try: try:
await session.execute(delete(DLFieldModel)) await session.execute(delete(DLFieldModel))
models: list[DLFieldModel] = [] models: list[DLFieldModel] = [
for item in items: DLFieldModel(**item) if isinstance(item, dict) else item for item in items
if isinstance(item, dict): ]
models.append(DLFieldModel(**cast("dict[str, Any]", item)))
else:
models.append(item)
session.add_all(models) session.add_all(models)
await session.commit() await session.commit()
except Exception: except Exception:

View file

@ -33,10 +33,10 @@ class DLFields(metaclass=Singleton):
EventBus.get_instance().subscribe(Events.STARTED, handle_event, "DLFieldsRepository.run_migrations") EventBus.get_instance().subscribe(Events.STARTED, handle_event, "DLFieldsRepository.run_migrations")
async def get_all(self) -> list[DLFieldModel]: async def get_all(self) -> list[DLFieldModel]:
return await self._repo.all() return await self._repo.list()
async def get_all_serialized(self) -> list[dict[str, Any]]: async def get_all_serialized(self) -> list[dict[str, Any]]:
items = await self._repo.all() items = await self._repo.list()
return [DLField.model_validate(item).model_dump() for item in items] return [DLField.model_validate(item).model_dump() for item in items]
async def save(self, item: DLField | dict) -> DLFieldModel: async def save(self, item: DLField | dict) -> DLFieldModel:

View file

@ -204,7 +204,7 @@ class TestDLFieldsRepository:
await repo.create({"name": "a", "description": "a", "field": "--a", "kind": "text", "order": 1}) await repo.create({"name": "a", "description": "a", "field": "--a", "kind": "text", "order": 1})
await repo.create({"name": "c", "description": "c", "field": "--c", "kind": "text", "order": 0}) await repo.create({"name": "c", "description": "c", "field": "--c", "kind": "text", "order": 0})
items = await repo.all() items = await repo.list()
assert items[0].name == "c", "Lowest order should be first" assert items[0].name == "c", "Lowest order should be first"
assert items[1].name == "a", "Same order sorted alphabetically" assert items[1].name == "a", "Same order sorted alphabetically"
@ -243,6 +243,6 @@ class TestDLFieldsRepository:
assert len(result) == 2, "Should create 2 new fields" assert len(result) == 2, "Should create 2 new fields"
all_items = await repo.all() all_items = await repo.list()
assert len(all_items) == 2, "Should only have new fields" assert len(all_items) == 2, "Should only have new fields"
assert {item.name for item in all_items} == {"new_1", "new_2"}, "Should only have new items" assert {item.name for item in all_items} == {"new_1", "new_2"}, "Should only have new items"

View file

@ -11,39 +11,20 @@ from app.library.log import get_logger
from app.library.Singleton import Singleton from app.library.Singleton import Singleton
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import AsyncGenerator
from contextlib import AbstractAsyncContextManager
from sqlalchemy.engine.result import Result from sqlalchemy.engine.result import Result
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select from sqlalchemy.sql.selectable import Select
SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]]
LOG = get_logger() LOG = get_logger()
def _model_from_payload(payload: dict[str, Any]) -> NotificationModel:
model = NotificationModel()
for key, value in payload.items():
if not hasattr(model, key):
msg = f"'{key}' is an invalid keyword argument for NotificationModel"
raise TypeError(msg)
setattr(model, key, value)
return model
def _coerce_model(payload: NotificationModel | dict[str, Any]) -> NotificationModel:
if isinstance(payload, NotificationModel):
return payload
return _model_from_payload(payload)
class NotificationsRepository(metaclass=Singleton): class NotificationsRepository(metaclass=Singleton):
def __init__(self, session: SessionFactory | None = None) -> None: def __init__(self, session: AsyncGenerator[AsyncSession] | None = None) -> None:
self._migrated = False self._migrated = False
self.session: SessionFactory = session or get_session self.session: AsyncGenerator[AsyncSession] = session or get_session
async def run_migrations(self) -> None: async def run_migrations(self) -> None:
if self._migrated: if self._migrated:
@ -56,7 +37,7 @@ class NotificationsRepository(metaclass=Singleton):
def get_instance() -> NotificationsRepository: def get_instance() -> NotificationsRepository:
return NotificationsRepository() return NotificationsRepository()
async def all(self) -> list[NotificationModel]: async def list(self) -> list[NotificationModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[NotificationModel]] = await session.execute( result: Result[tuple[NotificationModel]] = await session.execute(
select(NotificationModel).order_by(NotificationModel.name.asc()) select(NotificationModel).order_by(NotificationModel.name.asc())
@ -111,11 +92,11 @@ class NotificationsRepository(metaclass=Singleton):
result: Result[tuple[NotificationModel]] = await session.execute(query.limit(1)) result: Result[tuple[NotificationModel]] = await session.execute(query.limit(1))
return result.scalar_one_or_none() return result.scalar_one_or_none()
async def create(self, payload: NotificationModel | dict[str, Any]) -> NotificationModel: async def create(self, payload: NotificationModel | dict) -> NotificationModel:
async with self.session() as session: async with self.session() as session:
model = _coerce_model(payload) model: NotificationModel = NotificationModel(**payload) if isinstance(payload, dict) else payload
if model.id is not None: if model.id is not None:
model.id = None # ty: ignore model.id = None
if await self.get_by_name(name=model.name) is not None: if await self.get_by_name(name=model.name) is not None:
msg: str = f"Notification target with name '{model.name}' already exists." msg: str = f"Notification target with name '{model.name}' already exists."

View file

@ -2,7 +2,7 @@ from __future__ import annotations
import asyncio import asyncio
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING, Any, cast from typing import TYPE_CHECKING, Any
from app.features.notifications.models import NotificationModel from app.features.notifications.models import NotificationModel
from app.features.notifications.repository import NotificationsRepository from app.features.notifications.repository import NotificationsRepository
@ -64,10 +64,10 @@ class Notifications(metaclass=Singleton):
await self._repo.run_migrations() await self._repo.run_migrations()
EventBus.get_instance().subscribe(Events.STARTED, handle_event, "NotificationsRepository.run_migrations") EventBus.get_instance().subscribe(Events.STARTED, handle_event, "NotificationsRepository.run_migrations")
EventBus.get_instance().subscribe(NotificationEvents.events(), self.emit, f"{type(self).__name__}.emit") EventBus.get_instance().subscribe(NotificationEvents.events(), self.emit, f"{__class__.__name__}.emit")
async def all(self) -> list[NotificationModel]: async def list(self) -> list[NotificationModel]:
return await self._repo.all() return await self._repo.list()
async def list_paginated(self, page: int, per_page: int) -> tuple[list[NotificationModel], int, int, int]: async def list_paginated(self, page: int, per_page: int) -> tuple[list[NotificationModel], int, int, int]:
return await self._repo.list_paginated(page, per_page) return await self._repo.list_paginated(page, per_page)
@ -92,8 +92,8 @@ class Notifications(metaclass=Singleton):
async def delete(self, identifier: int | str) -> NotificationModel: async def delete(self, identifier: int | str) -> NotificationModel:
return await self._repo.delete(identifier) return await self._repo.delete(identifier)
async def send(self, ev: Event, wait: bool = True) -> list[dict | Awaitable[dict]]: async def send(self, ev: Event, wait: bool = True) -> list[dict] | list[Awaitable[dict]]:
targets = await self._repo.all() targets = await self._repo.list()
if len(targets) < 1: if len(targets) < 1:
return [] return []
@ -126,7 +126,7 @@ class Notifications(metaclass=Singleton):
if wait: if wait:
return await asyncio.gather(*tasks) return await asyncio.gather(*tasks)
return cast("list[dict | Awaitable[dict]]", tasks) return tasks
def emit(self, e: Event, _, **__) -> None: def emit(self, e: Event, _, **__) -> None:
if not NotificationEvents.is_valid(e.event): if not NotificationEvents.is_valid(e.event):
@ -271,7 +271,7 @@ class Notifications(metaclass=Singleton):
if not status: if not status:
msg = "Apprise failed to send notification." msg = "Apprise failed to send notification."
self._raise_apprise_error(msg) raise RuntimeError(msg) # noqa: TRY301
except Exception as exc: except Exception as exc:
LOG.exception( LOG.exception(
"Failed to send Apprise notification for event '%s'.", "Failed to send Apprise notification for event '%s'.",
@ -288,10 +288,6 @@ class Notifications(metaclass=Singleton):
return {} return {}
@staticmethod
def _raise_apprise_error(msg: str) -> None:
raise RuntimeError(msg)
async def _send(self, target: Notification, ev: Event) -> dict: async def _send(self, target: Notification, ev: Event) -> dict:
try: try:
LOG.info("Sending notification event '%s: %s' to '%s'.", ev.event, ev.id, target.name) LOG.info("Sending notification event '%s: %s' to '%s'.", ev.event, ev.id, target.name)

View file

@ -35,7 +35,7 @@ class TestNotificationsRepository:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_list_empty(self, repo): async def test_list_empty(self, repo):
"""List returns empty when no notifications exist.""" """List returns empty when no notifications exist."""
notifications = await repo.all() notifications = await repo.list()
assert notifications == [], "Should return empty list when no notifications exist" assert notifications == [], "Should return empty list when no notifications exist"
@pytest.mark.asyncio @pytest.mark.asyncio

View file

@ -46,7 +46,7 @@ class Migration(FeatureMigration):
return return
inserted = 0 inserted = 0
seen_names: dict[str, int] = {preset_name(preset.name): 1 for preset in await self._repo.all()} seen_names: dict[str, int] = {preset_name(preset.name): 1 for preset in await self._repo.list()}
for index, item in enumerate(items): for index, item in enumerate(items):
if not (normalized := self._normalize(item, index, seen_names)): if not (normalized := self._normalize(item, index, seen_names)):

View file

@ -30,30 +30,6 @@ if TYPE_CHECKING:
LOG = get_logger() LOG = get_logger()
def _payload_data(payload: PresetModel | dict[str, Any]) -> dict[str, Any]:
if isinstance(payload, dict):
data: dict[str, Any] = {}
for key, value in payload.items():
if not isinstance(key, str):
msg = "Preset payload keys must be strings."
raise TypeError(msg)
data[key] = value
return data
return {
"name": payload.name,
"description": payload.description,
"folder": payload.folder,
"template": payload.template,
"cookies": payload.cookies,
"cli": payload.cli,
"default": payload.default,
"priority": payload.priority,
"created_at": payload.created_at,
"updated_at": payload.updated_at,
}
class PresetsRepository(metaclass=Singleton): class PresetsRepository(metaclass=Singleton):
SORT_FIELDS: dict[str, Any] = { SORT_FIELDS: dict[str, Any] = {
"id": PresetModel.id, "id": PresetModel.id,
@ -97,15 +73,15 @@ class PresetsRepository(metaclass=Singleton):
LOG.debug("Refreshing presets cache due to configuration update.") LOG.debug("Refreshing presets cache due to configuration update.")
await self._update_cache() await self._update_cache()
Services.get_instance().add(PresetsRepository.__name__, self) Services.get_instance().add(__class__.__name__, self)
EventBus.get_instance().subscribe( EventBus.get_instance().subscribe(
Events.STARTED, handle_event, f"{PresetsRepository.__name__}.run_migrations" Events.STARTED, handle_event, f"{__class__.__name__}.run_migrations"
).subscribe(Events.CONFIG_UPDATE, handler, "Presets.refresh_cache") ).subscribe(Events.CONFIG_UPDATE, handler, "Presets.refresh_cache")
async def _update_cache(self) -> None: async def _update_cache(self) -> None:
from app.features.presets.service import Presets from app.features.presets.service import Presets
await Presets.get_instance().refresh_cache(await self.all()) await Presets.get_instance().refresh_cache(await self.list())
@staticmethod @staticmethod
def get_instance() -> PresetsRepository: def get_instance() -> PresetsRepository:
@ -121,7 +97,7 @@ class PresetsRepository(metaclass=Singleton):
LOG.error("Default preset '%s' not found, using 'default' preset.", default_name) LOG.error("Default preset '%s' not found, using 'default' preset.", default_name)
config.default_preset = "default" config.default_preset = "default"
async def all(self) -> list[PresetModel]: async def list(self) -> list[PresetModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[PresetModel]] = await session.execute( result: Result[tuple[PresetModel]] = await session.execute(
select(PresetModel).order_by(PresetModel.priority.desc(), PresetModel.name.asc()) select(PresetModel).order_by(PresetModel.priority.desc(), PresetModel.name.asc())
@ -262,9 +238,24 @@ class PresetsRepository(metaclass=Singleton):
result: Result[tuple[PresetModel]] = await session.execute(query.limit(1)) result: Result[tuple[PresetModel]] = await session.execute(query.limit(1))
return result.scalar_one_or_none() return result.scalar_one_or_none()
async def create(self, payload: PresetModel | dict[str, Any]) -> PresetModel: async def create(self, payload: PresetModel | dict) -> PresetModel:
async with self.session() as session: async with self.session() as session:
data = _payload_data(payload) data: dict[str, Any]
if isinstance(payload, dict):
data = dict(payload)
else:
data = {
"name": payload.name,
"description": payload.description,
"folder": payload.folder,
"template": payload.template,
"cookies": payload.cookies,
"cli": payload.cli,
"default": payload.default,
"priority": payload.priority,
"created_at": payload.created_at,
"updated_at": payload.updated_at,
}
data.pop("id", None) data.pop("id", None)
model = PresetModel(**data) model = PresetModel(**data)

View file

@ -12,7 +12,7 @@ class Presets(metaclass=Singleton):
def __init__(self, repo: PresetsRepository | None = None) -> None: def __init__(self, repo: PresetsRepository | None = None) -> None:
self._repo: PresetsRepository = repo or PresetsRepository.get_instance() self._repo: PresetsRepository = repo or PresetsRepository.get_instance()
self._cache: list[tuple[int, str, Preset]] = [] self._cache: list[tuple[int, str, Preset]] = []
Services.get_instance().add(type(self).__name__, self) Services.get_instance().add(__class__.__name__, self)
@staticmethod @staticmethod
def get_instance() -> Presets: def get_instance() -> Presets:

View file

@ -57,7 +57,7 @@ class TestPresetsRepository:
await repo.create({"name": "A", "priority": 1}) await repo.create({"name": "A", "priority": 1})
await repo.create({"name": "C", "priority": 2}) await repo.create({"name": "C", "priority": 2})
items = await repo.all() items = await repo.list()
assert items[0].name == "c", "Highest priority should be first" assert items[0].name == "c", "Highest priority should be first"
assert items[1].name == "a", "Same priority should sort by name" assert items[1].name == "a", "Same priority should sort by name"

View file

@ -10,6 +10,8 @@ import os
import subprocess # qa: ignore import subprocess # qa: ignore
from pathlib import Path from pathlib import Path
import anyio
from app.features.streaming.types import FFProbeError from app.features.streaming.types import FFProbeError
from app.library.log import get_logger from app.library.log import get_logger
from app.library.Utils import timed_lru_cache from app.library.Utils import timed_lru_cache
@ -37,26 +39,21 @@ class FFStream:
def __repr__(self): def __repr__(self):
if "codec_long_name" not in self.__dict__: if "codec_long_name" not in self.__dict__:
self.__dict__["codec_long_name"] = self.__dict__.get("codec_name", "") self.codec_long_name = self.__dict__.get("codec_name", "")
index = self.__dict__.get("index", "?")
codec_type = self.__dict__.get("codec_type", "unknown")
codec_long_name = self.__dict__.get("codec_long_name", "")
if self.is_video(): if self.is_video():
return f"<Stream: #{index} [{codec_type}] {codec_long_name}, {self.__dict__.get('framerate')}, ({self.__dict__.get('width')}x{self.__dict__.get('height')})>" return f"<Stream: #{self.index} [{self.codec_type}] {self.codec_long_name}, {self.framerate}, ({self.width}x{self.height})>"
if self.is_audio(): if self.is_audio():
return ( return (
f"<Stream: #{index} [{codec_type}] {codec_long_name}, " f"<Stream: #{self.index} [{self.codec_type}] {self.codec_long_name}, channels: {self.channels} ({self.channel_layout}), "
f"channels: {self.__dict__.get('channels')} ({self.__dict__.get('channel_layout')}), " "{sample_rate}Hz> "
f"{self.__dict__.get('sample_rate')}Hz> "
) )
if self.is_subtitle() or self.is_attachment(): if self.is_subtitle() or self.is_attachment():
return f"<Stream: #{index} [{codec_type}] {codec_long_name}>" return f"<Stream: #{self.index} [{self.codec_type}] {self.codec_long_name}>"
return f"<Stream: #{index} [{codec_type}]>" return f"<Stream: #{self.index} [{self.codec_type}]>"
def is_audio(self): def is_audio(self):
""" """
@ -255,14 +252,14 @@ async def ffprobe(file: Path | str) -> FFProbeResult:
raise OSError(msg) raise OSError(msg)
try: try:
proc = await asyncio.create_subprocess_exec( async with await anyio.open_file(os.devnull, "w") as tempf:
await asyncio.create_subprocess_exec(
"ffprobe", "ffprobe",
"-h", "-h",
stdout=asyncio.subprocess.DEVNULL, stdout=tempf,
stderr=asyncio.subprocess.DEVNULL, stderr=tempf,
creationflags=subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0, creationflags=subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0,
) )
await proc.wait()
except FileNotFoundError as e: except FileNotFoundError as e:
msg = "ffprobe not found." msg = "ffprobe not found."
raise OSError(msg) from e raise OSError(msg) from e

View file

@ -39,7 +39,7 @@ class M3u8:
m3u8.append("#EXT-X-MEDIA-SEQUENCE:0") m3u8.append("#EXT-X-MEDIA-SEQUENCE:0")
m3u8.append("#EXT-X-PLAYLIST-TYPE:VOD") m3u8.append("#EXT-X-PLAYLIST-TYPE:VOD")
segmentSize: str = f"{self.duration:.6f}" segmentSize: float = f"{self.duration:.6f}"
splits: int = math.ceil(duration / self.duration) splits: int = math.ceil(duration / self.duration)
segmentParams: dict = {} segmentParams: dict = {}

View file

@ -14,7 +14,7 @@ class Playlist:
"The path where files are downloaded." "The path where files are downloaded."
async def make(self, file: Path) -> str: async def make(self, file: Path) -> str:
ref: Path = Path(str(file.relative_to(self.download_path)).strip("/")) ref: str = Path(str(file.relative_to(self.download_path)).strip("/"))
try: try:
ff = await ffprobe(file) ff = await ffprobe(file)

View file

@ -1,7 +1,7 @@
# flake8: noqa: ARG002
from __future__ import annotations from __future__ import annotations
import os import os
import shutil
import subprocess import subprocess
import sys import sys
from functools import lru_cache from functools import lru_cache
@ -16,13 +16,9 @@ def detect_qsv_capabilities() -> dict[str, dict[str, bool]]:
Returns a dict where keys are codec names (e.g. "h264", "hevc", "vp9") and Returns a dict where keys are codec names (e.g. "h264", "hevc", "vp9") and
values are dicts with keys "full" and "lp" booleans. values are dicts with keys "full" and "lp" booleans.
""" """
vainfo = shutil.which("vainfo")
if not vainfo:
return {}
try: try:
result: subprocess.CompletedProcess[str] = subprocess.run( result: subprocess.CompletedProcess[str] = subprocess.run(
[vainfo], ["vainfo"], # noqa: S607
capture_output=True, capture_output=True,
text=True, text=True,
check=False, check=False,
@ -91,13 +87,9 @@ def ffmpeg_encoders() -> set[str]:
""" """
from app.library.config import SUPPORTED_CODECS from app.library.config import SUPPORTED_CODECS
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
return set()
try: try:
result: subprocess.CompletedProcess[str] = subprocess.run( result: subprocess.CompletedProcess[str] = subprocess.run(
[ffmpeg, "-hide_banner", "-loglevel", "error", "-encoders"], ["ffmpeg", "-hide_banner", "-loglevel", "error", "-encoders"], # noqa: S607
capture_output=True, capture_output=True,
text=True, text=True,
check=False, check=False,
@ -157,11 +149,9 @@ class _BaseBuilder:
codec_name: str codec_name: str
def input_args(self, ctx: dict[str, Any] | None = None) -> list[str]: def input_args(self, ctx: dict[str, Any] | None = None) -> list[str]:
_ = ctx
return [] return []
def add_video_args(self, args: list[str], ctx: dict[str, Any] | None = None) -> list[str]: def add_video_args(self, args: list[str], ctx: dict[str, Any] | None = None) -> list[str]:
_ = ctx
return [*args, "-codec:v", self.codec_name] return [*args, "-codec:v", self.codec_name]
@ -322,4 +312,4 @@ def encoder_fallback_chain(codec: str) -> tuple[str, ...]:
"libx264": [], "libx264": [],
} }
return tuple(chains.get(codec, chains["libx264"])) return chains.get(codec, chains["libx264"])

View file

@ -1,6 +1,6 @@
import asyncio import asyncio
import os import os
import subprocess import subprocess # type: ignore
import sys import sys
import tempfile import tempfile
from pathlib import Path from pathlib import Path
@ -178,7 +178,6 @@ class Segments:
) )
try: try:
assert proc.stdout is not None
while True: while True:
chunk: bytes = await proc.stdout.read(1024 * 64) chunk: bytes = await proc.stdout.read(1024 * 64)
if not chunk: if not chunk:
@ -241,9 +240,7 @@ class Segments:
return (wrote_any, rc, client_disconnected_local, stderr_buf.decode("utf-8", errors="ignore").strip()) return (wrote_any, rc, client_disconnected_local, stderr_buf.decode("utf-8", errors="ignore").strip())
async def stream(self, file: Path, resp: web.StreamResponse) -> None: async def stream(self, file: Path, resp: web.StreamResponse) -> None:
codec: str = self.vcodec codec: str = Segments._cached_vcodec if Segments._cache_initialized else self.vcodec
if Segments._cache_initialized and Segments._cached_vcodec:
codec = Segments._cached_vcodec
if not codec or codec not in SUPPORTED_CODECS: if not codec or codec not in SUPPORTED_CODECS:
codec: str = select_encoder(self.vcodec or "") codec: str = select_encoder(self.vcodec or "")

View file

@ -2,7 +2,6 @@ from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any
import anyio import anyio
import pysubs2 import pysubs2
@ -49,8 +48,7 @@ def ms_to_timestamp(ms: int) -> str:
return f"{h:01d}:{m:02d}:{s:02d}.{cs:02d}" return f"{h:01d}:{m:02d}:{s:02d}.{cs:02d}"
_substation_format: Any = SubstationFormat SubstationFormat.ms_to_timestamp = ms_to_timestamp
_substation_format.ms_to_timestamp = ms_to_timestamp
class Subtitle: class Subtitle:

View file

@ -24,8 +24,6 @@ THUMBNAIL_MISS_TTL = 3600.0
_LOCK = asyncio.Lock() _LOCK = asyncio.Lock()
_IN_PROCESS: dict[str, asyncio.Task[Path | None]] = {} _IN_PROCESS: dict[str, asyncio.Task[Path | None]] = {}
_SEM: asyncio.Semaphore | None = None _SEM: asyncio.Semaphore | None = None
_SEM_LIMIT: int | None = None _SEM_LIMIT: int | None = None

View file

@ -4,7 +4,6 @@ from pathlib import Path
from aiohttp import web from aiohttp import web
from aiohttp.web import Request, Response from aiohttp.web import Request, Response
from aiohttp.web_response import StreamResponse
from app.features.streaming.library.m3u8 import M3u8 from app.features.streaming.library.m3u8 import M3u8
from app.features.streaming.library.playlist import Playlist from app.features.streaming.library.playlist import Playlist
@ -33,7 +32,7 @@ async def playlist_create(request: Request, config: Config, app: web.Application
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
if not file: if not file:
return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code)
@ -84,8 +83,8 @@ async def m3u8_create(request: Request, config: Config, app: web.Application) ->
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
mode: str | None = request.match_info.get("mode") mode: str = request.match_info.get("mode")
if mode not in ["video", "subtitle"]: if mode not in ["video", "subtitle"]:
return web.json_response( return web.json_response(
@ -95,14 +94,13 @@ async def m3u8_create(request: Request, config: Config, app: web.Application) ->
if not file: if not file:
return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code)
duration: float | None = None duration = request.query.get("duration", None)
duration_arg = request.query.get("duration", None)
if "subtitle" in mode: if "subtitle" in mode:
if not duration_arg: if not duration:
return web.json_response(data={"error": "duration is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "duration is required."}, status=web.HTTPBadRequest.status_code)
duration = float(duration_arg) duration = float(duration)
base_path: str = config.base_path.rstrip("/") base_path: str = config.base_path.rstrip("/")
@ -126,8 +124,6 @@ async def m3u8_create(request: Request, config: Config, app: web.Application) ->
return web.json_response(data={"error": f"File '{file}' does not exist."}, status=status) return web.json_response(data={"error": f"File '{file}' does not exist."}, status=status)
if "subtitle" in mode: if "subtitle" in mode:
if duration is None:
return web.json_response(data={"error": "duration is required."}, status=web.HTTPBadRequest.status_code)
text = await cls.make_subtitle(file=realFile, duration=duration) text = await cls.make_subtitle(file=realFile, duration=duration)
else: else:
text = await cls.make_stream(file=realFile) text = await cls.make_stream(file=realFile)
@ -153,7 +149,7 @@ async def m3u8_create(request: Request, config: Config, app: web.Application) ->
@route("GET", r"api/player/segments/{segment:\d+}/{file:.*}.ts", "segments_stream") @route("GET", r"api/player/segments/{segment:\d+}/{file:.*}.ts", "segments_stream")
async def segments_stream(request: Request, config: Config, app: web.Application) -> StreamResponse: async def segments_stream(request: Request, config: Config, app: web.Application) -> Response:
""" """
Get the segments. Get the segments.
@ -166,9 +162,9 @@ async def segments_stream(request: Request, config: Config, app: web.Application
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
segment: str | None = request.match_info.get("segment") segment: int = request.match_info.get("segment")
sd: str | None = request.query.get("sd") sd: int = request.query.get("sd")
vc: int = int(request.query.get("vc", 0)) vc: int = int(request.query.get("vc", 0))
ac: int = int(request.query.get("ac", 0)) ac: int = int(request.query.get("ac", 0))
@ -245,7 +241,7 @@ async def subtitles_get(request: Request, config: Config, app: web.Application)
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
if not file: if not file:
return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code)
@ -303,7 +299,7 @@ async def subtitles_manifest_get(request: Request, config: Config, app: web.Appl
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
if not file: if not file:
return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required"}, status=web.HTTPBadRequest.status_code)
@ -358,7 +354,7 @@ async def subtitles_track_get(request: Request, config: Config, app: web.Applica
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
source_format: str | None = request.match_info.get("source_format") source_format: str | None = request.match_info.get("source_format")
if not file: if not file:

View file

@ -4,7 +4,6 @@ from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from aiohttp import web
from app.features.streaming.library.segments import Segments from app.features.streaming.library.segments import Segments
from app.tests.helpers import get_test_system_temp_root from app.tests.helpers import get_test_system_temp_root
@ -149,18 +148,18 @@ class _FakeProc:
self.killed = True self.killed = True
class _FakeResp(web.StreamResponse): class _FakeResp:
def __init__(self, fail_with: BaseException | None = None) -> None: def __init__(self, fail_with: Exception | None = None) -> None:
self.data: bytearray = bytearray() self.data: bytearray = bytearray()
self.eof = False self.eof = False
self._exc = fail_with self._exc = fail_with
async def write(self, data: bytes | bytearray | memoryview, *_args: Any, **_kwargs: Any) -> None: async def write(self, data: bytes) -> None:
if self._exc: if self._exc:
raise self._exc raise self._exc
self.data.extend(data) self.data.extend(data)
async def write_eof(self, *_args: Any, **_kwargs: Any) -> None: async def write_eof(self) -> None:
self.eof = True self.eof = True

View file

@ -2,7 +2,6 @@ from __future__ import annotations
import asyncio import asyncio
from pathlib import Path from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock from unittest.mock import AsyncMock
import pytest import pytest
@ -261,7 +260,7 @@ async def test_retry_no_seek(monkeypatch: pytest.MonkeyPatch) -> None:
output = temp_dir / ".jpg" output = temp_dir / ".jpg"
media.write_text("video") media.write_text("video")
ff_info: Any = FFProbeResult() ff_info = FFProbeResult()
ff_info.metadata = {"duration": "120.0"} ff_info.metadata = {"duration": "120.0"}
ff_info.video = [type("Video", (), {"codec_type": "video", "codec_name": "h264"})()] ff_info.video = [type("Video", (), {"codec_type": "video", "codec_name": "h264"})()]
@ -322,7 +321,7 @@ async def test_limit_wait(monkeypatch: pytest.MonkeyPatch) -> None:
media1.write_text("video") media1.write_text("video")
media2.write_text("video") media2.write_text("video")
ff_info: Any = FFProbeResult() ff_info = FFProbeResult()
ff_info.metadata = {"duration": "60.0"} ff_info.metadata = {"duration": "60.0"}
ff_info.video = [type("Video", (), {"codec_type": "video", "codec_name": "h264"})()] ff_info.video = [type("Video", (), {"codec_type": "video", "codec_name": "h264"})()]
monkeypatch.setattr(thumbnail, "ffprobe", AsyncMock(return_value=ff_info)) monkeypatch.setattr(thumbnail, "ffprobe", AsyncMock(return_value=ff_info))

View file

@ -1,3 +1,4 @@
# flake8: noqa: ARG004
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING, Any
from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskResult from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskResult
@ -11,7 +12,6 @@ if TYPE_CHECKING:
class BaseHandler: class BaseHandler:
@staticmethod @staticmethod
async def can_handle(task: HandleTask) -> bool: async def can_handle(task: HandleTask) -> bool:
_ = task
return False return False
@staticmethod @staticmethod
@ -24,7 +24,6 @@ class BaseHandler:
@staticmethod @staticmethod
def parse(url: str) -> Any | None: def parse(url: str) -> Any | None:
_ = url
return None return None
@staticmethod @staticmethod

View file

@ -64,7 +64,7 @@ class GenericTaskHandler(BaseHandler):
from app.features.tasks.definitions.utils import model_to_schema from app.features.tasks.definitions.utils import model_to_schema
repo = TaskDefinitionsRepository.get_instance() repo = TaskDefinitionsRepository.get_instance()
models = await repo.all() models = await repo.list()
cls._definitions = [model_to_schema(model) for model in models] cls._definitions = [model_to_schema(model) for model in models]
return cls._definitions return cls._definitions
@ -141,8 +141,7 @@ class GenericTaskHandler(BaseHandler):
return False return False
@staticmethod @staticmethod
async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: # noqa: ARG004
_ = config
definition: TaskDefinition | None = await GenericTaskHandler._find_definition(task.url) definition: TaskDefinition | None = await GenericTaskHandler._find_definition(task.url)
if not definition: if not definition:
return TaskFailure(message="No generic task definition matched the provided URL.") return TaskFailure(message="No generic task definition matched the provided URL.")

View file

@ -9,7 +9,6 @@ from app.features.tasks.definitions.results import HandleTask, TaskFailure, Task
from app.features.ytdlp.extractor import fetch_info from app.features.ytdlp.extractor import fetch_info
from app.features.ytdlp.utils import get_archive_id from app.features.ytdlp.utils import get_archive_id
from app.library.cache import Cache from app.library.cache import Cache
from app.library.config import Config
from app.library.log import get_logger from app.library.log import get_logger
from ._base_handler import BaseHandler from ._base_handler import BaseHandler
@ -161,14 +160,12 @@ class RssGenericHandler(BaseHandler):
return feed_url, items, real_count return feed_url, items, real_count
@staticmethod @staticmethod
async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: async def extract(task: HandleTask) -> TaskResult | TaskFailure:
_ = config
""" """
Extract items from an RSS/Atom feed. Extract items from an RSS/Atom feed.
Args: Args:
task (Task): The task containing the feed URL. task (Task): The task containing the feed URL.
config (Config | None): Optional handler configuration.
Returns: Returns:
TaskResult | TaskFailure: Extraction result with parsed items or failure information. TaskResult | TaskFailure: Extraction result with parsed items or failure information.

View file

@ -4,7 +4,6 @@ import httpx
from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult
from app.features.ytdlp.utils import get_archive_id from app.features.ytdlp.utils import get_archive_id
from app.library.config import Config
from app.library.log import get_logger from app.library.log import get_logger
from ._base_handler import BaseHandler from ._base_handler import BaseHandler
@ -178,8 +177,7 @@ class TverHandler(BaseHandler):
return feed_url, items, has_items return feed_url, items, has_items
@staticmethod @staticmethod
async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: async def extract(task: HandleTask) -> TaskResult | TaskFailure:
_ = config
series_id: str | None = TverHandler.parse(task.url) series_id: str | None = TverHandler.parse(task.url)
if not series_id: if not series_id:
return TaskFailure(message="Unrecognized Tver series URL.") return TaskFailure(message="Unrecognized Tver series URL.")
@ -210,11 +208,10 @@ class TverHandler(BaseHandler):
if not (url := entry.get("url")): if not (url := entry.get("url")):
continue continue
archive_id: str | None = entry.get("archive_id") archive_id: str = entry.get("archive_id")
metadata = entry.get("metadata", {}) task_items.append(
if not isinstance(metadata, dict): TaskItem(url=url, title=entry.get("title"), archive_id=archive_id, metadata=entry.get("metadata", {}))
metadata = {} )
task_items.append(TaskItem(url=url, title=entry.get("title"), archive_id=archive_id, metadata=metadata))
return TaskResult(items=task_items, metadata={"feed_url": feed_url, "has_entries": has_items}) return TaskResult(items=task_items, metadata={"feed_url": feed_url, "has_entries": has_items})

View file

@ -6,7 +6,6 @@ import httpx
from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult
from app.features.ytdlp.utils import get_archive_id from app.features.ytdlp.utils import get_archive_id
from app.library.config import Config
from app.library.log import get_logger from app.library.log import get_logger
from ._base_handler import BaseHandler from ._base_handler import BaseHandler
@ -93,8 +92,7 @@ class TwitchHandler(BaseHandler):
return feed_url, items, has_items return feed_url, items, has_items
@staticmethod @staticmethod
async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: async def extract(task: HandleTask) -> TaskResult | TaskFailure:
_ = config
handle_name: str | None = TwitchHandler.parse(task.url) handle_name: str | None = TwitchHandler.parse(task.url)
if not handle_name: if not handle_name:
return TaskFailure(message="Unrecognized Twitch channel URL.") return TaskFailure(message="Unrecognized Twitch channel URL.")
@ -124,7 +122,7 @@ class TwitchHandler(BaseHandler):
if not (url := entry.get("url")): if not (url := entry.get("url")):
continue continue
archive_id: str | None = entry.get("archive_id") archive_id: str = entry.get("archive_id")
task_items.append(TaskItem(url=url, title=entry.get("title"), archive_id=archive_id)) task_items.append(TaskItem(url=url, title=entry.get("title"), archive_id=archive_id))
return TaskResult(items=task_items, metadata={"feed_url": feed_url, "has_entries": has_items}) return TaskResult(items=task_items, metadata={"feed_url": feed_url, "has_entries": has_items})

View file

@ -6,7 +6,6 @@ import httpx
from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult
from app.features.ytdlp.utils import get_archive_id from app.features.ytdlp.utils import get_archive_id
from app.library.config import Config
from app.library.log import get_logger from app.library.log import get_logger
from ._base_handler import BaseHandler from ._base_handler import BaseHandler
@ -104,8 +103,7 @@ class YoutubeHandler(BaseHandler):
return feed_url, items, real_count return feed_url, items, real_count
@staticmethod @staticmethod
async def extract(task: HandleTask, config: Config | None = None) -> TaskResult | TaskFailure: async def extract(task: HandleTask) -> TaskResult | TaskFailure:
_ = config
parsed: dict[str, str] | None = YoutubeHandler.parse(task.url) parsed: dict[str, str] | None = YoutubeHandler.parse(task.url)
if not parsed: if not parsed:
return TaskFailure(message="Unrecognized YouTube channel or playlist URL.") return TaskFailure(message="Unrecognized YouTube channel or playlist URL.")
@ -135,7 +133,7 @@ class YoutubeHandler(BaseHandler):
if not (url := entry.get("url")): if not (url := entry.get("url")):
continue continue
archive_id: str | None = entry.get("archive_id") archive_id: str = entry.get("archive_id")
metadata: dict[str, Any] = {"published": entry.get("published")} metadata: dict[str, Any] = {"published": entry.get("published")}
task_items.append(TaskItem(url=url, title=entry.get("title"), archive_id=archive_id, metadata=metadata)) task_items.append(TaskItem(url=url, title=entry.get("title"), archive_id=archive_id, metadata=metadata))

View file

@ -14,23 +14,20 @@ from app.library.Services import Services
from app.library.Singleton import Singleton from app.library.Singleton import Singleton
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import AsyncGenerator
from contextlib import AbstractAsyncContextManager
from sqlalchemy.engine.result import Result from sqlalchemy.engine.result import Result
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select from sqlalchemy.sql.selectable import Select
SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]]
LOG = get_logger() LOG = get_logger()
class TaskDefinitionsRepository(metaclass=Singleton): class TaskDefinitionsRepository(metaclass=Singleton):
def __init__(self, session: SessionFactory | None = None) -> None: def __init__(self, session: AsyncGenerator[AsyncSession] | None = None) -> None:
self._migrated = False self._migrated = False
self.session: SessionFactory = session or get_session self.session: AsyncGenerator[AsyncSession] = session or get_session
async def run_migrations(self) -> None: async def run_migrations(self) -> None:
if self._migrated: if self._migrated:
@ -54,12 +51,12 @@ class TaskDefinitionsRepository(metaclass=Singleton):
LOG.debug("Refreshing task definitions due to configuration update.") LOG.debug("Refreshing task definitions due to configuration update.")
await GenericTaskHandler.refresh_definitions(force=True) await GenericTaskHandler.refresh_definitions(force=True)
Services.get_instance().add(TaskDefinitionsRepository.__name__, self) Services.get_instance().add(__class__.__name__, self)
EventBus.get_instance().subscribe( EventBus.get_instance().subscribe(
Events.STARTED, handle_event, f"{TaskDefinitionsRepository.__name__}.run_migrations" Events.STARTED, handle_event, f"{__class__.__name__}.run_migrations"
).subscribe(Events.CONFIG_UPDATE, handler, "GenericTaskHandler.refresh_definitions") ).subscribe(Events.CONFIG_UPDATE, handler, "GenericTaskHandler.refresh_definitions")
async def all(self) -> list[TaskDefinitionModel]: async def list(self) -> list[TaskDefinitionModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[TaskDefinitionModel]] = await session.execute( result: Result[tuple[TaskDefinitionModel]] = await session.execute(
select(TaskDefinitionModel).order_by(TaskDefinitionModel.priority.asc(), TaskDefinitionModel.name.asc()) select(TaskDefinitionModel).order_by(TaskDefinitionModel.priority.asc(), TaskDefinitionModel.name.asc())
@ -118,9 +115,9 @@ class TaskDefinitionsRepository(metaclass=Singleton):
async def create(self, payload: dict[str, Any]) -> TaskDefinitionModel: async def create(self, payload: dict[str, Any]) -> TaskDefinitionModel:
async with self.session() as session: async with self.session() as session:
model: TaskDefinitionModel = TaskDefinitionModel(**payload) model: TaskDefinitionModel = TaskDefinitionModel(**payload) if isinstance(payload, dict) else payload
if model.id is not None: if model.id is not None:
model.id = None # ty: ignore model.id = None
if await self.get_by_name(name=model.name) is not None: if await self.get_by_name(name=model.name) is not None:
msg: str = f"Task definition with name '{model.name}' already exists." msg: str = f"Task definition with name '{model.name}' already exists."

View file

@ -48,13 +48,10 @@ class HandleTask(TaskSchema):
if isinstance(ret, tuple): if isinstance(ret, tuple):
return ret return ret
archive_file = ret.get("file") archive_file: Path = ret.get("file")
items = ret.get("items", set()) items: set[str] = ret.get("items", set())
if not isinstance(archive_file, Path) or not isinstance(items, set):
return (False, "Failed to get archive information.")
archive_items = [item for item in items if isinstance(item, str)]
if len(archive_items) < 1 or not archive_add(archive_file, archive_items): if len(items) < 1 or not archive_add(archive_file, list(items)):
return (True, "No new items to mark as downloaded.") return (True, "No new items to mark as downloaded.")
return (True, f"Task '{self.name}' items marked as downloaded.") return (True, f"Task '{self.name}' items marked as downloaded.")
@ -73,13 +70,10 @@ class HandleTask(TaskSchema):
if isinstance(ret, tuple): if isinstance(ret, tuple):
return ret return ret
archive_file = ret.get("file") archive_file: Path = ret.get("file")
items = ret.get("items", set()) items: set[str] = ret.get("items", set())
if not isinstance(archive_file, Path) or not isinstance(items, set):
return (False, "Failed to get archive information.")
archive_items = [item for item in items if isinstance(item, str)]
if len(archive_items) < 1 or not archive_delete(archive_file, archive_items): if len(items) < 1 or not archive_delete(archive_file, list(items)):
return (True, "No items to remove from archive file.") return (True, "No items to remove from archive file.")
return (True, f"Removed '{self.name}' items from archive file.") return (True, f"Removed '{self.name}' items from archive file.")

View file

@ -8,7 +8,6 @@ import random
from datetime import UTC, datetime from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING, Any
from app.features.tasks.definitions.handlers._base_handler import BaseHandler
from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult from app.features.tasks.definitions.results import HandleTask, TaskFailure, TaskItem, TaskResult
from app.features.tasks.models import TaskModel from app.features.tasks.models import TaskModel
from app.features.ytdlp.utils import archive_read from app.features.ytdlp.utils import archive_read
@ -28,7 +27,7 @@ LOG = get_logger()
class TaskHandle: class TaskHandle:
def __init__(self, scheduler: Scheduler, tasks: TasksRepository, config: Config) -> None: def __init__(self, scheduler: Scheduler, tasks: TasksRepository, config: Config) -> None:
self._handlers: list[type[BaseHandler]] = [] self._handlers: list[type] = []
"The available handlers." "The available handlers."
self._repo: TasksRepository = tasks self._repo: TasksRepository = tasks
"The tasks manager." "The tasks manager."
@ -36,7 +35,7 @@ class TaskHandle:
"The scheduler." "The scheduler."
self._config: Config = config self._config: Config = config
"The configuration." "The configuration."
self._task_name: str = f"{TaskHandle.__name__}._dispatcher" self._task_name: str = f"{__class__.__name__}._dispatcher"
"The task name for the scheduler." "The task name for the scheduler."
self._queued: dict[str, set[str]] = {} self._queued: dict[str, set[str]] = {}
"Queued archive IDs per handler." "Queued archive IDs per handler."
@ -46,11 +45,11 @@ class TaskHandle:
EventBus.get_instance().subscribe( EventBus.get_instance().subscribe(
Events.ITEM_ERROR, Events.ITEM_ERROR,
self._handle_item_error, self._handle_item_error,
f"{TaskHandle.__name__}.item_error", f"{__class__.__name__}.item_error",
) )
def load(self) -> None: def load(self) -> None:
self._handlers: list[type[BaseHandler]] = self._discover() self._handlers: list[type] = self._discover()
timer: str = self._config.tasks_handler_timer timer: str = self._config.tasks_handler_timer
try: try:
@ -73,16 +72,16 @@ class TaskHandle:
self._scheduler.add( self._scheduler.add(
timer=timer, timer=timer,
func=lambda: asyncio.create_task(self._dispatcher(), name="task-handler-dispatcher"), func=lambda: asyncio.create_task(self._dispatcher(), name="task-handler-dispatcher"),
id=f"{TaskHandle.__name__}._dispatcher", id=f"{__class__.__name__}._dispatcher",
) )
async def _dispatcher(self): async def _dispatcher(self):
s: dict[str, list[str]] = {"h": [], "d": [], "u": [], "f": []} s: dict[str, list[str]] = {"h": [], "d": [], "u": [], "f": []}
handler_groups: dict[str, list[tuple[HandleTask, type[BaseHandler]]]] = {} handler_groups: dict[str, list[tuple[HandleTask, type]]] = {}
dispatches: list[tuple[HandleTask, type[BaseHandler], asyncio.Task[TaskResult | TaskFailure | None]]] = [] dispatches: list[tuple[HandleTask, type, asyncio.Task[TaskResult | TaskFailure | None]]] = []
tasks: list[TaskModel] = await self._repo.all() tasks: list[TaskModel] = await self._repo.list()
for task_model in tasks: for task_model in tasks:
task: HandleTask = HandleTask.model_validate(task_model) task: HandleTask = HandleTask.model_validate(task_model)
@ -101,7 +100,7 @@ class TaskHandle:
continue continue
try: try:
handler: type[BaseHandler] | None = await self._find_handler(task) handler: type | None = await self._find_handler(task)
if handler is None: if handler is None:
s["u"].append(task.name) s["u"].append(task.name)
continue continue
@ -182,9 +181,7 @@ class TaskHandle:
}, },
) )
async def _dispatch( async def _dispatch(self, task: HandleTask, handler: type, delay: float) -> TaskResult | TaskFailure | None:
self, task: HandleTask, handler: type[BaseHandler], delay: float
) -> TaskResult | TaskFailure | None:
""" """
Dispatch a task after a random delay to avoid rate limiting. Dispatch a task after a random delay to avoid rate limiting.
@ -218,7 +215,7 @@ class TaskHandle:
exc_info=(type(exc), exc, exc.__traceback__), exc_info=(type(exc), exc, exc.__traceback__),
) )
async def _find_handler(self, task: HandleTask) -> type[BaseHandler] | None: async def _find_handler(self, task: HandleTask) -> type | None:
for cls in self._handlers: for cls in self._handlers:
try: try:
if await Services.get_instance().handle_async(handler=cls.can_handle, task=task): if await Services.get_instance().handle_async(handler=cls.can_handle, task=task):
@ -242,10 +239,9 @@ class TaskHandle:
async def dispatch( async def dispatch(
self, self,
task: HandleTask, task: HandleTask,
handler: type[BaseHandler] | None = None, handler: type | None = None,
**kwargs, **kwargs, # noqa: ARG002
) -> TaskResult | TaskFailure | None: ) -> TaskResult | TaskFailure | None:
_ = kwargs
""" """
Dispatch a task to the appropriate handler. Dispatch a task to the appropriate handler.
@ -586,11 +582,11 @@ class TaskHandle:
return TaskResult(items=list(extraction.items), metadata=combined_metadata) return TaskResult(items=list(extraction.items), metadata=combined_metadata)
def _discover(self) -> list[type[BaseHandler]]: def _discover(self) -> list[type]:
"""Discover all available task handlers.""" """Discover all available task handlers."""
import app.features.tasks.definitions.handlers as handlers_pkg import app.features.tasks.definitions.handlers as handlers_pkg
handlers: list[type[BaseHandler]] = [] handlers: list[type] = []
for _, module_name, _ in pkgutil.iter_modules(handlers_pkg.__path__): for _, module_name, _ in pkgutil.iter_modules(handlers_pkg.__path__):
if module_name.startswith("_"): if module_name.startswith("_"):

View file

@ -8,7 +8,6 @@ from app.features.tasks.definitions.results import TaskFailure, TaskResult
from app.features.tasks.definitions.schemas import ( from app.features.tasks.definitions.schemas import (
Definition, Definition,
EngineConfig, EngineConfig,
Parse,
RequestConfig, RequestConfig,
ResponseConfig, ResponseConfig,
TaskDefinition, TaskDefinition,
@ -31,12 +30,10 @@ def test_build_def_payload():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"link": {"type": "css", "expression": ".article a.link::attr(href)"}, "link": {"type": "css", "expression": ".article a.link::attr(href)"},
"title": {"type": "css", "expression": ".article .title", "attribute": "text"}, "title": {"type": "css", "expression": ".article .title", "attribute": "text"},
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(), response=ResponseConfig(),
@ -57,8 +54,7 @@ def test_build_def_container():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"selector": ".cards .card", "selector": ".cards .card",
"fields": { "fields": {
@ -66,8 +62,7 @@ def test_build_def_container():
"title": {"type": "css", "expression": ".card-header a", "attribute": "text"}, "title": {"type": "css", "expression": ".card-header a", "attribute": "text"},
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(), response=ResponseConfig(),
@ -87,8 +82,7 @@ def test_build_def_json():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"type": "jsonpath", "type": "jsonpath",
"selector": "items", "selector": "items",
@ -97,8 +91,7 @@ def test_build_def_json():
"title": {"type": "jsonpath", "expression": "title"}, "title": {"type": "jsonpath", "expression": "title"},
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(type="json"), response=ResponseConfig(type="json"),
@ -119,13 +112,11 @@ def test_parse_items_basic():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"link": {"type": "css", "expression": ".article a.link::attr(href)", "attribute": None}, "link": {"type": "css", "expression": ".article a.link::attr(href)", "attribute": None},
"title": {"type": "css", "expression": ".article .title", "attribute": "text"}, "title": {"type": "css", "expression": ".article .title", "attribute": "text"},
"id": {"type": "css", "expression": ".article", "attribute": "data-id"}, "id": {"type": "css", "expression": ".article", "attribute": "data-id"},
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(), response=ResponseConfig(),
@ -163,8 +154,7 @@ def test_parse_items_cards():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"type": "css", "type": "css",
"selector": ".columns .card", "selector": ".columns .card",
@ -191,8 +181,7 @@ def test_parse_items_cards():
}, },
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(), response=ResponseConfig(),
@ -260,8 +249,7 @@ def test_parse_items_json():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"type": "jsonpath", "type": "jsonpath",
"selector": "entries", "selector": "entries",
@ -271,8 +259,7 @@ def test_parse_items_json():
"id": {"type": "jsonpath", "expression": "id"}, "id": {"type": "jsonpath", "expression": "id"},
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(type="json"), response=ResponseConfig(type="json"),
@ -310,8 +297,7 @@ async def test_generic_task_handler_inspect(monkeypatch):
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"type": "jsonpath", "type": "jsonpath",
"selector": "items", "selector": "items",
@ -320,8 +306,7 @@ async def test_generic_task_handler_inspect(monkeypatch):
"title": {"type": "jsonpath", "expression": "title"}, "title": {"type": "jsonpath", "expression": "title"},
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(type="json"), response=ResponseConfig(type="json"),
@ -366,8 +351,7 @@ def test_parse_items_json_list():
created_at=datetime.now(), created_at=datetime.now(),
updated_at=datetime.now(), updated_at=datetime.now(),
definition=Definition( definition=Definition(
parse=Parse.model_validate( parse={
{
"items": { "items": {
"type": "jsonpath", "type": "jsonpath",
"selector": "[]", "selector": "[]",
@ -376,8 +360,7 @@ def test_parse_items_json_list():
"title": {"type": "jsonpath", "expression": "title"}, "title": {"type": "jsonpath", "expression": "title"},
}, },
} }
} },
),
engine=EngineConfig(), engine=EngineConfig(),
request=RequestConfig(), request=RequestConfig(),
response=ResponseConfig(type="json"), response=ResponseConfig(type="json"),

View file

@ -69,7 +69,7 @@ class TestTaskDefinitionsRepository:
await repo.create(_sample_definition("Alpha", priority=2)) await repo.create(_sample_definition("Alpha", priority=2))
await repo.create(_sample_definition("Beta", priority=1)) await repo.create(_sample_definition("Beta", priority=1))
items = await repo.all() items = await repo.list()
assert len(items) == 2, "Should return two task definitions" assert len(items) == 2, "Should return two task definitions"
assert [item.name for item in items] == ["Beta", "Alpha"], "Should sort by priority then name" assert [item.name for item in items] == ["Beta", "Alpha"], "Should sort by priority then name"

View file

@ -1,17 +1,9 @@
from typing import Any, Literal, overload from typing import Any
from app.features.tasks.definitions.models import TaskDefinitionModel from app.features.tasks.definitions.models import TaskDefinitionModel
from app.features.tasks.definitions.schemas import Definition, TaskDefinition, TaskDefinitionSummary from app.features.tasks.definitions.schemas import Definition, TaskDefinition, TaskDefinitionSummary
@overload
def model_to_schema(model: TaskDefinitionModel, summary: Literal[False] = False) -> TaskDefinition: ...
@overload
def model_to_schema(model: TaskDefinitionModel, summary: Literal[True]) -> TaskDefinitionSummary: ...
def model_to_schema(model: TaskDefinitionModel, summary: bool = False) -> TaskDefinition | TaskDefinitionSummary: def model_to_schema(model: TaskDefinitionModel, summary: bool = False) -> TaskDefinition | TaskDefinitionSummary:
""" """
Convert a TaskDefinitionModel to a TaskDefinition or TaskDefinitionSummary schema. Convert a TaskDefinitionModel to a TaskDefinition or TaskDefinitionSummary schema.

View file

@ -10,39 +10,20 @@ from app.library.log import get_logger
from app.library.Singleton import Singleton from app.library.Singleton import Singleton
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import AsyncGenerator
from contextlib import AbstractAsyncContextManager
from sqlalchemy.engine.result import Result from sqlalchemy.engine.result import Result
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.sql.selectable import Select from sqlalchemy.sql.selectable import Select
SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]]
LOG = get_logger() LOG = get_logger()
def _model_from_payload(payload: dict[str, Any]) -> TaskModel:
model = TaskModel()
for key, value in payload.items():
if not hasattr(model, key):
msg = f"'{key}' is an invalid keyword argument for TaskModel"
raise TypeError(msg)
setattr(model, key, value)
return model
def _coerce_model(payload: TaskModel | dict[str, Any]) -> TaskModel:
if isinstance(payload, TaskModel):
return payload
return _model_from_payload(payload)
class TasksRepository(metaclass=Singleton): class TasksRepository(metaclass=Singleton):
def __init__(self, session: SessionFactory | None = None) -> None: def __init__(self, session: AsyncGenerator[AsyncSession] | None = None) -> None:
self._migrated = False self._migrated = False
self.session: SessionFactory = session or get_session self.session = session or get_session
async def run_migrations(self) -> None: async def run_migrations(self) -> None:
if self._migrated: if self._migrated:
@ -57,7 +38,7 @@ class TasksRepository(metaclass=Singleton):
def get_instance() -> TasksRepository: def get_instance() -> TasksRepository:
return TasksRepository() return TasksRepository()
async def all(self) -> list[TaskModel]: async def list(self) -> list[TaskModel]:
async with self.session() as session: async with self.session() as session:
result: Result[tuple[TaskModel]] = await session.execute(select(TaskModel).order_by(TaskModel.name.asc())) result: Result[tuple[TaskModel]] = await session.execute(select(TaskModel).order_by(TaskModel.name.asc()))
return list(result.scalars().all()) return list(result.scalars().all())
@ -109,7 +90,7 @@ class TasksRepository(metaclass=Singleton):
"""Get all enabled tasks.""" """Get all enabled tasks."""
async with self.session() as session: async with self.session() as session:
result: Result[tuple[TaskModel]] = await session.execute( result: Result[tuple[TaskModel]] = await session.execute(
select(TaskModel).where(TaskModel.enabled).order_by(TaskModel.name.asc()) select(TaskModel).where(TaskModel.enabled == True).order_by(TaskModel.name.asc()) # noqa: E712
) )
return list(result.scalars().all()) return list(result.scalars().all())
@ -121,11 +102,11 @@ class TasksRepository(metaclass=Singleton):
) )
return list(result.scalars().all()) return list(result.scalars().all())
async def create(self, payload: TaskModel | dict[str, Any]) -> TaskModel: async def create(self, payload: TaskModel | dict) -> TaskModel:
async with self.session() as session: async with self.session() as session:
model = _coerce_model(payload) model: TaskModel = TaskModel(**payload) if isinstance(payload, dict) else payload
if model.id is not None: if model.id is not None:
model.id = None # ty: ignore model.id = None
if await self.get_by_name(name=model.name) is not None: if await self.get_by_name(name=model.name) is not None:
msg: str = f"Task with name '{model.name}' already exists." msg: str = f"Task with name '{model.name}' already exists."

View file

@ -705,9 +705,8 @@ async def task_metadata(request: Request, repo: TasksRepository, config: Config,
xml_content += f' <uniqueid type="{NFOMakerPP._escape_text(info.get("id_type"))}" default="true">{NFOMakerPP._escape_text(info.get("id"))}</uniqueid>\n' xml_content += f' <uniqueid type="{NFOMakerPP._escape_text(info.get("id_type"))}" default="true">{NFOMakerPP._escape_text(info.get("id"))}</uniqueid>\n'
if info.get("uploader"): if info.get("uploader"):
xml_content += f" <studio>{NFOMakerPP._escape_text(info.get('uploader'))}</studio>\n" xml_content += f" <studio>{NFOMakerPP._escape_text(info.get('uploader'))}</studio>\n"
tags = info.get("tags", []) if info.get("tags", []):
if isinstance(tags, list): for tag in info.get("tags", []):
for tag in tags:
xml_content += f" <tag>{NFOMakerPP._escape_text(tag)}</tag>\n" xml_content += f" <tag>{NFOMakerPP._escape_text(tag)}</tag>\n"
if info.get("year"): if info.get("year"):
xml_content += f" <year>{info.get('year')}</year>\n" xml_content += f" <year>{info.get('year')}</year>\n"

View file

@ -51,7 +51,7 @@ class Tasks(metaclass=Singleton):
pass pass
async def _load_tasks(self) -> None: async def _load_tasks(self) -> None:
tasks = await self._repo.all() tasks = await self._repo.list()
for task in tasks: for task in tasks:
if not task.timer or not task.enabled: if not task.timer or not task.enabled:
@ -134,11 +134,9 @@ class Tasks(metaclass=Singleton):
task_id = task.id task_id = task.id
task_name = task.name task_name = task.name
try: try:
current_task = await self._repo.get(task_id) if not (task := await self._repo.get(task_id)):
if not current_task:
LOG.info("Task '%s' no longer exists.", task_name, extra={"task_id": task_id, "task_name": task_name}) LOG.info("Task '%s' no longer exists.", task_name, extra={"task_id": task_id, "task_name": task_name})
return return
task = current_task
if not task.enabled: if not task.enabled:
LOG.debug( LOG.debug(

View file

@ -260,7 +260,7 @@ class TestTasksRepository:
await repo.create({"name": "Alice", "url": "https://example.com"}) await repo.create({"name": "Alice", "url": "https://example.com"})
await repo.create({"name": "Bob", "url": "https://example.com"}) await repo.create({"name": "Bob", "url": "https://example.com"})
items = await repo.all() items = await repo.list()
assert items[0].name == "Alice", "Should be alphabetically first" assert items[0].name == "Alice", "Should be alphabetically first"
assert items[1].name == "Bob", "Should be alphabetically second" assert items[1].name == "Bob", "Should be alphabetically second"

View file

@ -1,7 +1,6 @@
from __future__ import annotations from __future__ import annotations
from types import SimpleNamespace from types import SimpleNamespace
from typing import Any
import pytest import pytest
import pytest_asyncio import pytest_asyncio
@ -42,13 +41,13 @@ def patch_get_info(monkeypatch: pytest.MonkeyPatch) -> None:
def _json_request(method: str, path: str, payload: object, *, match_info: dict[str, str] | None = None) -> web.Request: def _json_request(method: str, path: str, payload: object, *, match_info: dict[str, str] | None = None) -> web.Request:
mock_request: Any = make_mocked_request(method, path, match_info=match_info or {}, app=SimpleNamespace()) request = make_mocked_request(method, path, match_info=match_info or {}, app=SimpleNamespace())
async def _json() -> object: async def _json() -> object:
return payload return payload
mock_request.json = _json request.json = _json # type: ignore[attr-defined]
return mock_request return request
class _Notify: class _Notify:
@ -86,7 +85,7 @@ async def test_add_requires_timer_without_handler(repo) -> None:
assert response.status == web.HTTPBadRequest.status_code assert response.status == web.HTTPBadRequest.status_code
assert b"requires a timer" in response.body assert b"requires a timer" in response.body
assert await repo.all() == [] assert await repo.list() == []
@pytest.mark.asyncio @pytest.mark.asyncio
@ -110,7 +109,7 @@ async def test_add_all_or_nothing(repo) -> None:
assert response.status == web.HTTPBadRequest.status_code assert response.status == web.HTTPBadRequest.status_code
assert b"requires a timer" in response.body assert b"requires a timer" in response.body
assert await repo.all() == [] assert await repo.list() == []
@pytest.mark.asyncio @pytest.mark.asyncio
@ -124,7 +123,7 @@ async def test_add_allows_handler_only(repo) -> None:
response = await router.tasks_add(request, repo, Encoder(), _Notify(), _Handler(matched=True)) response = await router.tasks_add(request, repo, Encoder(), _Notify(), _Handler(matched=True))
assert response.status == web.HTTPOk.status_code assert response.status == web.HTTPOk.status_code
items = await repo.all() items = await repo.list()
assert len(items) == 1 assert len(items) == 1
assert items[0].name == "Handler Only" assert items[0].name == "Handler Only"

View file

@ -120,7 +120,7 @@ class ExtractorPool(metaclass=Singleton):
def attach(self, app: web.Application) -> None: def attach(self, app: web.Application) -> None:
"""Attach the extractor pool to the application (no-op for now).""" """Attach the extractor pool to the application (no-op for now)."""
app.on_shutdown.append(self.shutdown) app.on_shutdown.append(self.shutdown)
Services.get_instance().add(type(self).__name__, self) Services.get_instance().add(__class__.__name__, self)
def _ensure_initialized(self, config: ExtractorConfig) -> None: def _ensure_initialized(self, config: ExtractorConfig) -> None:
""" """

View file

@ -286,7 +286,7 @@ async def convert(request: Request) -> Response:
return web.json_response(data={"error": "args param is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "args param is required."}, status=web.HTTPBadRequest.status_code)
try: try:
response: dict[str, Any] = {"opts": {}, "output_template": None, "download_path": None} response = {"opts": {}, "output_template": None, "download_path": None}
data = arg_converter(args, dumps=True) data = arg_converter(args, dumps=True)
@ -384,8 +384,6 @@ async def get_info(request: Request, cache: Cache, config: Config) -> Response:
if cache.has(key) and not request.query.get("force", False): if cache.has(key) and not request.query.get("force", False):
data: Any | None = cache.get(key) data: Any | None = cache.get(key)
if data is None:
data = {}
data["_cached"] = { data["_cached"] = {
"status": "hit", "status": "hit",
"preset": preset, "preset": preset,
@ -395,7 +393,7 @@ async def get_info(request: Request, cache: Cache, config: Config) -> Response:
"ttl_left": data.get("_cached", {}).get("expires", time.time() + 300) - time.time(), "ttl_left": data.get("_cached", {}).get("expires", time.time() + 300) - time.time(),
"expires": data.get("_cached", {}).get("expires", time.time() + 300), "expires": data.get("_cached", {}).get("expires", time.time() + 300),
} }
return web.json_response(text=json.dumps(data, indent=4, default=str), status=web.HTTPOk.status_code) return web.json_response(body=json.dumps(data, indent=4, default=str), status=web.HTTPOk.status_code)
ytdlp_opts: dict = opts.get_all() ytdlp_opts: dict = opts.get_all()
@ -452,7 +450,7 @@ async def get_info(request: Request, cache: Cache, config: Config) -> Response:
cache.set(key=key, value=data, ttl=300) cache.set(key=key, value=data, ttl=300)
return web.json_response(text=json.dumps(data, indent=4, default=str), status=web.HTTPOk.status_code) return web.json_response(body=json.dumps(data, indent=4, default=str), status=web.HTTPOk.status_code)
except Exception as e: except Exception as e:
LOG.exception( LOG.exception(
"Failed to get video info for '%s': %s.", "Failed to get video info for '%s': %s.",
@ -488,7 +486,7 @@ async def get_options() -> Response:
""" """
from app.features.ytdlp.ytdlp import ytdlp_options from app.features.ytdlp.ytdlp import ytdlp_options
return web.json_response(text=json.dumps(ytdlp_options(), indent=4, default=str), status=web.HTTPOk.status_code) return web.json_response(body=json.dumps(ytdlp_options(), indent=4, default=str), status=web.HTTPOk.status_code)
@route("POST", "api/yt-dlp/archive_id/", "get_archive_ids") @route("POST", "api/yt-dlp/archive_id/", "get_archive_ids")
@ -510,11 +508,7 @@ async def get_archive_ids(request: Request, config: Config) -> Response:
response = [] response = []
for i, url in enumerate(data): for i, url in enumerate(data):
dct: dict[str, Any] = {"index": i, "url": url} dct = {"index": i, "url": url}
if not isinstance(url, str):
dct.update({"id": None, "ie_key": None, "archive_id": None, "error": "URL must be a string."})
response.append(dct)
continue
try: try:
await asyncio.to_thread(validate_url, url, config.allow_internal_urls) await asyncio.to_thread(validate_url, url, config.allow_internal_urls)
dct.update(get_archive_id(url)) dct.update(get_archive_id(url))

View file

@ -122,7 +122,6 @@ class TestExtractInfo:
{}, "https://example.com/video", debug=True, capture_logs=logging.WARNING {}, "https://example.com/video", debug=True, capture_logs=logging.WARNING
) )
assert result is not None
assert result["id"] == "test123" assert result["id"] == "test123"
assert logs == ["[generic_browser] Browser fallback warning"] assert logs == ["[generic_browser] Browser fallback warning"]
assert (logging.INFO, "[generic_browser] Using remote browser for https://example.com/video") in seen assert (logging.INFO, "[generic_browser] Using remote browser for https://example.com/video") in seen
@ -156,7 +155,6 @@ class TestExtractInfo:
{}, "https://example.com/video", debug=False, capture_logs=logging.WARNING {}, "https://example.com/video", debug=False, capture_logs=logging.WARNING
) )
assert result is not None
assert result["id"] == "test123" assert result["id"] == "test123"
assert logs == ["[generic_browser] Browser fallback warning"] assert logs == ["[generic_browser] Browser fallback warning"]
assert (logging.INFO, "[generic_browser] Using remote browser for https://example.com/video") in seen assert (logging.INFO, "[generic_browser] Using remote browser for https://example.com/video") in seen

View file

@ -1,6 +1,5 @@
import importlib import importlib
from pathlib import Path from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, Mock, patch from unittest.mock import MagicMock, Mock, patch
import pytest import pytest
@ -220,8 +219,7 @@ class TestYTDLP:
"""Test _delete_downloaded_files skips cleanup when _interrupted is True.""" """Test _delete_downloaded_files skips cleanup when _interrupted is True."""
with patch("app.features.ytdlp.ytdlp.yt_dlp.YoutubeDL.__init__", return_value=None): with patch("app.features.ytdlp.ytdlp.yt_dlp.YoutubeDL.__init__", return_value=None):
ytdlp = YTDLP(params={}) ytdlp = YTDLP(params={})
mock_obj: Any = ytdlp ytdlp.to_screen = Mock()
mock_obj.to_screen = Mock()
# Set interrupted flag # Set interrupted flag
ytdlp._interrupted = True ytdlp._interrupted = True
@ -231,7 +229,7 @@ class TestYTDLP:
# Should not call super method # Should not call super method
mock_super_delete.assert_not_called() mock_super_delete.assert_not_called()
# Should show message # Should show message
mock_obj.to_screen.assert_called_once_with("[info] Cancelled — skipping temp cleanup.") ytdlp.to_screen.assert_called_once_with("[info] Cancelled — skipping temp cleanup.")
# Should return None # Should return None
assert result is None assert result is None
@ -351,14 +349,6 @@ class TestYTDLP:
assert len(result) == 8 assert len(result) == 8
assert result.isalnum() assert result.isalnum()
def test_exec_init(self) -> None:
YTDLP(
params={
"compat_opts": set(),
"postprocessors": [{"key": "Exec", "exec_cmd": "echo %(title)q"}],
}
)
def test_outtmpl_reuses_value(self) -> None: def test_outtmpl_reuses_value(self) -> None:
ytdlp = YTDLP(params={"outtmpl": {"default": "%(title)s"}}) ytdlp = YTDLP(params={"outtmpl": {"default": "%(title)s"}})

View file

@ -776,7 +776,7 @@ class TestYTDLPCli:
from app.features.ytdlp.ytdlp_opts import YTDLPCli from app.features.ytdlp.ytdlp_opts import YTDLPCli
with pytest.raises(ValueError, match="Expected Item instance"): with pytest.raises(ValueError, match="Expected Item instance"):
YTDLPCli(item="not an item") YTDLPCli(item="not an item") # type: ignore
@patch("app.features.presets.service.Presets") @patch("app.features.presets.service.Presets")
@patch("app.features.ytdlp.ytdlp_opts.Config") @patch("app.features.ytdlp.ytdlp_opts.Config")

View file

@ -46,8 +46,7 @@ class TestLogWrapper:
def test_add_target_type_validation(self) -> None: def test_add_target_type_validation(self) -> None:
lw = LogWrapper() lw = LogWrapper()
with pytest.raises(TypeError, match=r"Target must be a logging\.Logger instance or a callable"): with pytest.raises(TypeError, match=r"Target must be a logging\.Logger instance or a callable"):
bad: Any = 123 lw.add_target(123) # type: ignore[arg-type]
lw.add_target(bad)
def test_add_target_names(self) -> None: def test_add_target_names(self) -> None:
lw = LogWrapper() lw = LogWrapper()
@ -129,7 +128,7 @@ class TestExtractYtdlpLogs:
def test_extract_ytdlp_logs_with_filters(self): def test_extract_ytdlp_logs_with_filters(self):
"""Test log extraction with filters.""" """Test log extraction with filters."""
logs = ["INFO: Downloading", "ERROR: Failed", "WARNING: Deprecated"] logs = ["INFO: Downloading", "ERROR: Failed", "WARNING: Deprecated"]
filters: list[str | re.Pattern[str]] = [re.compile(r"ERROR")] filters = [re.compile(r"ERROR")]
result = extract_ytdlp_logs(logs, filters) result = extract_ytdlp_logs(logs, filters)
assert result == ["ERROR: Failed"] assert result == ["ERROR: Failed"]
@ -313,12 +312,9 @@ class TestGetThumbnail:
def test_non_list(self): def test_non_list(self):
"""Test that None is returned for non-list input.""" """Test that None is returned for non-list input."""
bad_none: Any = None assert get_thumbnail(None) is None
bad_str: Any = "not a list" assert get_thumbnail("not a list") is None
bad_dict: Any = {"not": "list"} assert get_thumbnail({"not": "list"}) is None
assert get_thumbnail(bad_none) is None
assert get_thumbnail(bad_str) is None
assert get_thumbnail(bad_dict) is None
def test_thumbnail_preference(self): def test_thumbnail_preference(self):
"""Test that the thumbnail with highest preference is returned.""" """Test that the thumbnail with highest preference is returned."""
@ -353,7 +349,6 @@ class TestGetThumbnail:
] ]
result = get_thumbnail(thumbnails) result = get_thumbnail(thumbnails)
assert result is not None
assert result["url"] == "with_pref.jpg" assert result["url"] == "with_pref.jpg"
def test_all_equal(self): def test_all_equal(self):
@ -370,15 +365,12 @@ class TestGetThumbnail:
class TestGetExtras: class TestGetExtras:
def test_none(self): def test_none(self):
"""Test that empty dict is returned for None input.""" """Test that empty dict is returned for None input."""
bad: Any = None assert get_extras(None) == {}
assert get_extras(bad) == {}
def test_non_dict(self): def test_non_dict(self):
"""Test that empty dict is returned for non-dict input.""" """Test that empty dict is returned for non-dict input."""
bad_str: Any = "not a dict" assert get_extras("not a dict") == {}
bad_list: Any = [] assert get_extras([]) == {}
assert get_extras(bad_str) == {}
assert get_extras(bad_list) == {}
def test_extracts_video_information(self): def test_extracts_video_information(self):
"""Test extracting information from a video entry.""" """Test extracting information from a video entry."""

View file

@ -101,12 +101,12 @@ class LogWrapper:
name (str|None): The name of the logging target. Defaults to None. name (str|None): The name of the logging target. Defaults to None.
""" """
if not isinstance(target, logging.Logger) and not callable(target): if not isinstance(target, logging.Logger | Callable):
msg = "Target must be a logging.Logger instance or a callable." msg = "Target must be a logging.Logger instance or a callable."
raise TypeError(msg) raise TypeError(msg)
if name is None: if name is None:
name = target.name if isinstance(target, logging.Logger) else getattr(target, "__name__", "callable") name = target.name if isinstance(target, logging.Logger) else target.__name__
self.targets.append( self.targets.append(
LogTarget( LogTarget(
@ -131,11 +131,11 @@ class LogWrapper:
if level < target.level: if level < target.level:
continue continue
if isinstance(target.target, logging.Logger): if target.logger:
log_kwargs: dict[str, Any] = {**kwargs} log_kwargs = {**kwargs}
log_kwargs.setdefault("stacklevel", 3) log_kwargs.setdefault("stacklevel", 3)
target.target.log(level, msg, *args, **log_kwargs) target.target.log(level, msg, *args, **log_kwargs)
elif callable(target.target): else:
target.target(level, msg, *args, **kwargs) target.target(level, msg, *args, **kwargs)
def debug(self, msg, *args, **kwargs): def debug(self, msg, *args, **kwargs):
@ -179,17 +179,13 @@ def arg_converter(
create_parser = yt_dlp.options.create_parser create_parser = yt_dlp.options.create_parser
def _default_opts(args: str | list[str]): def _default_opts(args: str):
patched_parser = create_parser() patched_parser = create_parser()
def patched_create_parser():
return patched_parser
try: try:
yt_dlp.options.__dict__["create_parser"] = patched_create_parser yt_dlp.options.create_parser = lambda: patched_parser
return yt_dlp.parse_options(args) return yt_dlp.parse_options(args)
finally: finally:
yt_dlp.options.__dict__["create_parser"] = create_parser yt_dlp.options.create_parser = create_parser
apply_ytdlp_patches() apply_ytdlp_patches()
@ -242,7 +238,7 @@ def arg_converter(
return diff return diff
def extract_ytdlp_logs(logs: list[str], filters: list[str | re.Pattern[str]] | None = None) -> list[str]: def extract_ytdlp_logs(logs: list[str], filters: list[str | re.Pattern] = None) -> list[str]:
""" """
Extract yt-dlp log lines matching built-in filters plus any extras. Extract yt-dlp log lines matching built-in filters plus any extras.
@ -336,7 +332,7 @@ def get_ytdlp(params: dict | None = None) -> YTDLP:
return _DATA.YTDLP_INFO_CLS return _DATA.YTDLP_INFO_CLS
def get_thumbnail(thumbnails: list) -> dict | None: def get_thumbnail(thumbnails: list) -> str | None:
""" """
Extract thumbnail URL from a yt-dlp entry. Extract thumbnail URL from a yt-dlp entry.
@ -344,7 +340,7 @@ def get_thumbnail(thumbnails: list) -> dict | None:
thumbnails (list): The list of thumbnail dictionaries from yt-dlp entry. thumbnails (list): The list of thumbnail dictionaries from yt-dlp entry.
Returns: Returns:
dict | None: The best thumbnail dict if available, otherwise None. str | None: The thumbnail URL if available, otherwise None.
""" """
if not thumbnails or not isinstance(thumbnails, list): if not thumbnails or not isinstance(thumbnails, list):
@ -385,7 +381,7 @@ def get_extras(entry: dict, kind: str = "video") -> dict:
extras[f"playlist_{property}"] = val extras[f"playlist_{property}"] = val
if thumbnail := get_thumbnail(entry.get("thumbnails", [])): if thumbnail := get_thumbnail(entry.get("thumbnails", [])):
extras["thumbnail"] = thumbnail.get("url") if isinstance(thumbnail, dict) else thumbnail extras["thumbnail"] = thumbnail.get("url")
elif thumbnail := entry.get("thumbnail"): elif thumbnail := entry.get("thumbnail"):
extras["thumbnail"] = thumbnail extras["thumbnail"] = thumbnail
@ -443,7 +439,7 @@ def get_archive_id(url: str) -> dict[str, str | None]:
} }
""" """
idDict: dict[str, str | None] = { idDict: dict[str, None] = {
"id": None, "id": None,
"ie_key": None, "ie_key": None,
"archive_id": None, "archive_id": None,

View file

@ -81,9 +81,6 @@ class YTDLP(yt_dlp.YoutubeDL):
except Exception: except Exception:
patched_params = params patched_params = params
self._ytptube_outtmpl_info: dict[str, Any] | None = None
self._ytptube_outtmpl_cache: dict[str, Any] = {}
super().__init__(params=patched_params, auto_init=auto_init) super().__init__(params=patched_params, auto_init=auto_init)
# Restore param and replace upstream archive set with our proxy # Restore param and replace upstream archive set with our proxy
@ -93,6 +90,8 @@ class YTDLP(yt_dlp.YoutubeDL):
except Exception: except Exception:
pass pass
self._ytptube_outtmpl_info: dict[str, Any] | None = None
self._ytptube_outtmpl_cache: dict[str, Any] = {}
self.archive = _ArchiveProxy(orig_file) self.archive = _ArchiveProxy(orig_file)
def _delete_downloaded_files(self, *args, **kwargs) -> None: def _delete_downloaded_files(self, *args, **kwargs) -> None:
@ -106,14 +105,14 @@ class YTDLP(yt_dlp.YoutubeDL):
self._ytptube_outtmpl_info = None self._ytptube_outtmpl_info = None
self._ytptube_outtmpl_cache = {} self._ytptube_outtmpl_cache = {}
def prepare_outtmpl(self, outtmpl, info_dict, sanitize=False, *, _exec=False): def prepare_outtmpl(self, outtmpl, info_dict, sanitize=False):
if self._ytptube_outtmpl_info is not info_dict: if self._ytptube_outtmpl_info is not info_dict:
self._ytptube_outtmpl_info = info_dict self._ytptube_outtmpl_info = info_dict
self._ytptube_outtmpl_cache = {} self._ytptube_outtmpl_cache = {}
outtmpl, enriched = rewrite_outtmpl(outtmpl, info_dict, cache=self._ytptube_outtmpl_cache) outtmpl, enriched = rewrite_outtmpl(outtmpl, info_dict, cache=self._ytptube_outtmpl_cache)
return super().prepare_outtmpl(outtmpl, enriched, sanitize=sanitize, _exec=_exec) return super().prepare_outtmpl(outtmpl, enriched, sanitize=sanitize)
def process_info(self, info_dict): def process_info(self, info_dict):
try: try:

View file

@ -50,12 +50,12 @@ class ARGSMerger:
""" """
return str(self) return str(self)
def as_dict(self) -> list[str]: def as_dict(self) -> dict:
""" """
Get all the options as a shell argument list. Get all the options as a dict.
Returns: Returns:
list[str]: The options as shell arguments. dict: The options as a dict
""" """
return shlex.split(shlex.join(self.args)) return shlex.split(shlex.join(self.args))

View file

@ -29,7 +29,7 @@ class BackgroundWorker(metaclass=Singleton):
"The queue to hold the tasks." "The queue to hold the tasks."
self.running = True self.running = True
"Whether the background worker is running or not." "Whether the background worker is running or not."
self.thread: threading.Thread | None = None self.thread: threading.Thread = None
"The thread that runs the background worker." "The thread that runs the background worker."
@staticmethod @staticmethod
@ -49,7 +49,6 @@ class BackgroundWorker(metaclass=Singleton):
try: try:
LOG.debug("Stopping background worker thread.") LOG.debug("Stopping background worker thread.")
self.queue.put((CloseThread, (), {})) self.queue.put((CloseThread, (), {}))
if self.thread:
self.thread.join(timeout=5) self.thread.join(timeout=5)
LOG.debug("Background worker thread has been shut down.") LOG.debug("Background worker thread has been shut down.")
except Exception as e: except Exception as e:

View file

@ -166,12 +166,6 @@ class DataStore:
def items(self): def items(self):
return self._dict.items() return self._dict.items()
def __contains__(self, key: str) -> bool:
return key in self._dict
def __len__(self) -> int:
return len(self._dict)
async def put(self, value: Download, no_notify: bool = False) -> Download: async def put(self, value: Download, no_notify: bool = False) -> Download:
_ = no_notify _ = no_notify
self._dict[value.info._id] = value self._dict[value.info._id] = value

View file

@ -34,7 +34,6 @@ class Events:
ITEM_ADDED: str = "item_added" ITEM_ADDED: str = "item_added"
ITEM_UPDATED: str = "item_updated" ITEM_UPDATED: str = "item_updated"
ITEM_PROGRESS: str = "item_progress"
ITEM_COMPLETED: str = "item_completed" ITEM_COMPLETED: str = "item_completed"
ITEM_CANCELLED: str = "item_cancelled" ITEM_CANCELLED: str = "item_cancelled"
ITEM_DELETED: str = "item_deleted" ITEM_DELETED: str = "item_deleted"
@ -87,7 +86,6 @@ class Events:
Events.LOG_SUCCESS, Events.LOG_SUCCESS,
Events.ITEM_ADDED, Events.ITEM_ADDED,
Events.ITEM_UPDATED, Events.ITEM_UPDATED,
Events.ITEM_PROGRESS,
Events.ITEM_CANCELLED, Events.ITEM_CANCELLED,
Events.ITEM_DELETED, Events.ITEM_DELETED,
Events.ITEM_BULK_DELETED, Events.ITEM_BULK_DELETED,
@ -105,7 +103,7 @@ class Events:
list: The list of debug events. list: The list of debug events.
""" """
return [Events.ITEM_UPDATED, Events.ITEM_PROGRESS] return [Events.ITEM_UPDATED]
@dataclass(kw_only=True) @dataclass(kw_only=True)

View file

@ -1,17 +1,14 @@
import base64 import base64
import hmac import hmac
import inspect import inspect
from collections.abc import Awaitable
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from pathlib import Path from pathlib import Path
from typing import Any, cast
import anyio import anyio
from aiohttp import web from aiohttp import web
from aiohttp.typedefs import Handler, Middleware from aiohttp.web import Request, RequestHandler, Response
from aiohttp.web import Request, Response
from aiohttp.web_log import AccessLogger from aiohttp.web_log import AccessLogger
from aiohttp.web_request import BaseRequest
from aiohttp.web_response import StreamResponse
from app.library.log import get_logger from app.library.log import get_logger
from app.library.Services import Services from app.library.Services import Services
@ -27,12 +24,12 @@ LOG = get_logger("http")
class HttpAccessLogger(AccessLogger): class HttpAccessLogger(AccessLogger):
def log(self, request: BaseRequest, response: StreamResponse, time: float) -> None: def log(self, request: Request, response: Response, time: float) -> None:
try: try:
fmt_info = self._format_line(request, response, time) fmt_info = self._format_line(request, response, time)
values: list[object] = [] values: list[object] = []
extra: dict[str, Any] = {"elapsed_ms": round(time * 1000.0, 2)} extra: dict[str, object] = {"elapsed_ms": round(time * 1000.0, 2)}
for key, value in fmt_info: for key, value in fmt_info:
values.append(value) values.append(value)
@ -40,7 +37,7 @@ class HttpAccessLogger(AccessLogger):
extra[key] = value extra[key] = value
else: else:
parent, child = key parent, child = key
group: dict[str, Any] = extra.get(parent, {}) # type: ignore[assignment] group = extra.get(parent, {})
if not isinstance(group, dict): if not isinstance(group, dict):
group = {} group = {}
group[child] = value group[child] = value
@ -180,7 +177,7 @@ class HttpAPI:
registered_options.append(route.path) registered_options.append(route.path)
@staticmethod @staticmethod
def basic_auth(username: str, password: str) -> Middleware: def basic_auth(username: str, password: str) -> Awaitable:
""" """
Middleware to handle basic authentication. Middleware to handle basic authentication.
@ -194,7 +191,7 @@ class HttpAPI:
""" """
@web.middleware @web.middleware
async def auth_handler(request: Request, handler: Handler) -> StreamResponse: async def auth_handler(request: Request, handler: RequestHandler) -> Response:
# if OPTIONS request, skip auth # if OPTIONS request, skip auth
if "OPTIONS" == request.method: if "OPTIONS" == request.method:
return await handler(request) return await handler(request)
@ -207,7 +204,7 @@ class HttpAPI:
auth_cookie = request.cookies.get("auth") auth_cookie = request.cookies.get("auth")
if auth_cookie is not None: if auth_cookie is not None:
try: try:
data = decrypt_data(auth_cookie, key=cast("bytes", Config.get_instance().secret_key)) data = decrypt_data(auth_cookie, key=Config.get_instance().secret_key)
if data is not None: if data is not None:
data = base64.b64encode(data.encode()).decode() data = base64.b64encode(data.encode()).decode()
auth_header = f"Basic {data}" auth_header = f"Basic {data}"
@ -253,7 +250,7 @@ class HttpAPI:
}, },
) )
response: StreamResponse = await handler(request) response: Response = await handler(request)
contentType: str | None = response.headers.get("content-type", None) contentType: str | None = response.headers.get("content-type", None)
if contentType and not contentType.startswith("text/html"): if contentType and not contentType.startswith("text/html"):
@ -264,7 +261,7 @@ class HttpAPI:
"auth", "auth",
encrypt_data( encrypt_data(
f"{user_name}:{user_password}", f"{user_name}:{user_password}",
key=cast("bytes", Config.get_instance().secret_key), key=Config.get_instance().secret_key,
), ),
max_age=60 * 60 * 24 * 7, max_age=60 * 60 * 24 * 7,
expires=(datetime.now(UTC) + timedelta(days=7)).strftime("%a, %d %b %Y %H:%M:%S GMT"), expires=(datetime.now(UTC) + timedelta(days=7)).strftime("%a, %d %b %Y %H:%M:%S GMT"),
@ -283,9 +280,9 @@ class HttpAPI:
return auth_handler return auth_handler
@staticmethod @staticmethod
def middle_wares(app: web.Application, base_path: str, download_path: str) -> Middleware: def middle_wares(app: web.Application, base_path: str, download_path: str) -> Awaitable:
@web.middleware @web.middleware
async def middleware_handler(request: Request, handler: Handler) -> StreamResponse: async def middleware_handler(request: Request, handler: RequestHandler) -> Response:
static_path = str(app.router["download_static"].url_for(filename="")) static_path = str(app.router["download_static"].url_for(filename=""))
if request.path.startswith(static_path): if request.path.startswith(static_path):
realFile, status = get_file( realFile, status = get_file(
@ -304,13 +301,12 @@ class HttpAPI:
}, },
) )
response: StreamResponse = await handler(request) response: Response = await handler(request)
contentType: str = str(response.headers.get("content-type", "")) contentType: str = str(response.headers.get("content-type", ""))
if contentType.startswith("text/html") and getattr(response, "_path", None): if contentType.startswith("text/html") and getattr(response, "_path", None):
rewrite_path: str = base_path.rstrip("/") rewrite_path: str = base_path.rstrip("/")
response_path = cast("Any", response)._path async with await anyio.open_file(response._path, "rb") as f:
async with await anyio.open_file(response_path, "rb") as f:
content = await f.read() content = await f.read()
content: str = ( content: str = (
content.decode("utf-8") content.decode("utf-8")
@ -328,13 +324,13 @@ class HttpAPI:
new_response.set_cookie( new_response.set_cookie(
morsel.key, morsel.key,
morsel.value, morsel.value,
expires=morsel.get("expires"), expires=morsel["expires"],
domain=morsel.get("domain"), domain=morsel["domain"],
max_age=morsel.get("max-age"), max_age=morsel["max-age"],
path=morsel.get("path") or "/", path=morsel["path"],
secure=bool(morsel.get("secure")), secure=bool(morsel["secure"]),
httponly=bool(morsel.get("httponly")), httponly=bool(morsel["httponly"]),
samesite=morsel.get("samesite") or None, samesite=morsel["samesite"] or None,
) )
return new_response return new_response
@ -343,7 +339,7 @@ class HttpAPI:
return middleware_handler return middleware_handler
async def _handle(self, handler: Handler, request: Request) -> StreamResponse: async def _handle(self, handler: RequestHandler, request: Request) -> Response:
""" """
Call the handler with the request and return the response. Call the handler with the request and return the response.

View file

@ -104,20 +104,19 @@ class HttpSocket:
} }
) )
self._notify.subscribe("frontend", event_handler, f"{HttpSocket.__name__}.emit") self._notify.subscribe("frontend", event_handler, f"{__class__.__name__}.emit")
@staticmethod @staticmethod
def ws_event(func): def ws_event(func): # type: ignore
""" """
Decorator to mark a method as a socket event. Decorator to mark a method as a socket event.
""" """
@functools.wraps(func) @functools.wraps(func) # type: ignore
async def wrapper(*args, **kwargs): async def wrapper(*args, **kwargs):
return await func(*args, **kwargs) return await func(*args, **kwargs) # type: ignore
wrapper_with_event: Any = wrapper wrapper._ws_event = func.__name__ # type: ignore
wrapper_with_event._ws_event = func.__name__
return wrapper return wrapper
async def on_shutdown(self, _: web.Application): async def on_shutdown(self, _: web.Application):
@ -135,11 +134,9 @@ class HttpSocket:
if not (data and data.data): if not (data and data.data):
return return
queue = Services.get_instance().get("queue") await Services.get_instance().get("queue").add(item=Item.format(data.data))
if queue is not None:
await queue.add(item=Item.format(data.data))
self._notify.subscribe(Events.ADD_URL, event_handler, f"{HttpSocket.__name__}.add") self._notify.subscribe(Events.ADD_URL, event_handler, f"{__class__.__name__}.add")
load_modules(self.rootPath, self.rootPath / "routes" / "socket") load_modules(self.rootPath, self.rootPath / "routes" / "socket")

View file

@ -93,7 +93,7 @@ class Item:
bool: True if the item has extras, False otherwise. bool: True if the item has extras, False otherwise.
""" """
return bool(self.extras and len(self.extras) > 0) return self.extras and len(self.extras) > 0
def add_extras(self, key: str, value: Any) -> None: def add_extras(self, key: str, value: Any) -> None:
""" """
@ -117,7 +117,7 @@ class Item:
bool: True if the item has command options for yt, False otherwise. bool: True if the item has command options for yt, False otherwise.
""" """
return bool(self.cli and len(self.cli) > 2) return self.cli and len(self.cli) > 2
@staticmethod @staticmethod
def _default_preset() -> str: def _default_preset() -> str:
@ -161,7 +161,7 @@ class Item:
Item: The formatted item. Item: The formatted item.
""" """
url = item.get("url") url: str = item.get("url")
if not url or not isinstance(url, str): if not url or not isinstance(url, str):
msg = "url param is required." msg = "url param is required."
@ -173,7 +173,7 @@ class Item:
if len(url) >= 11 and re.fullmatch(r"[A-Za-z0-9_-]{11}", url): if len(url) >= 11 and re.fullmatch(r"[A-Za-z0-9_-]{11}", url):
url = f"https://www.youtube.com/watch?v={url}" url = f"https://www.youtube.com/watch?v={url}"
data: dict[str, Any] = {"url": url} data: dict[str, str] = {"url": url}
preset: str | None = item.get("preset") preset: str | None = item.get("preset")
if preset and isinstance(preset, str) and preset != Item._default_preset(): if preset and isinstance(preset, str) and preset != Item._default_preset():
@ -185,28 +185,24 @@ class Item:
data["preset"] = preset data["preset"] = preset
folder = item.get("folder") if item.get("folder") and isinstance(item.get("folder"), str):
if isinstance(folder, str) and folder: data["folder"] = item.get("folder")
data["folder"] = folder
cookies = item.get("cookies") if item.get("cookies") and isinstance(item.get("cookies"), str):
if isinstance(cookies, str) and cookies: data["cookies"] = item.get("cookies")
data["cookies"] = cookies
template = item.get("template") if item.get("template") and isinstance(item.get("template"), str):
if isinstance(template, str) and template: data["template"] = item.get("template")
data["template"] = template
if isinstance(item.get("auto_start"), bool): if "auto_start" in item and isinstance(item.get("auto_start"), bool):
data["auto_start"] = item["auto_start"] data["auto_start"] = bool(item.get("auto_start"))
extras: dict | None = item.get("extras") extras: dict | None = item.get("extras")
if isinstance(extras, dict) and len(extras) > 0: if extras and isinstance(extras, dict) and len(extras) > 0:
data["extras"] = extras data["extras"] = extras
requeued = item.get("requeued") if item.get("requeued") and isinstance(item.get("requeued"), bool):
if isinstance(requeued, bool): data["requeued"] = item.get("requeued")
data["requeued"] = requeued
cli: str | None = item.get("cli") cli: str | None = item.get("cli")
if cli and len(cli) > 2: if cli and len(cli) > 2:
@ -578,10 +574,8 @@ class ItemDTO:
self.is_archivable = bool(self._archive_file) self.is_archivable = bool(self._archive_file)
if not self.is_archivable: if not self.is_archivable:
self.is_archived = False self.is_archived = False
elif self.archive_id and self._archive_file:
self.is_archived = len(archive_read(self._archive_file, [self.archive_id])) > 0
else: else:
self.is_archived = False self.is_archived = len(archive_read(self._archive_file, [self.archive_id])) > 0
def get_archive_file(self) -> str | None: def get_archive_file(self) -> str | None:
""" """

View file

@ -17,7 +17,7 @@ def parse_version(v: str) -> tuple[int, ...]:
class Packages: class Packages:
def __init__(self, env: str | None, file: str | Path | None, upgrade: bool = False): def __init__(self, env: str | None, file: str | None, upgrade: bool = False):
from_env: list[str] = env.split() if env else [] from_env: list[str] = env.split() if env else []
from_file = [] from_file = []

View file

@ -1,6 +1,4 @@
import asyncio import asyncio
from collections.abc import Callable
from typing import Any
from aiocron import Cron from aiocron import Cron
from aiohttp import web from aiohttp import web
@ -68,7 +66,7 @@ class Scheduler(metaclass=Singleton):
if data and data.data: if data and data.data:
self.add(**data.data) self.add(**data.data)
EventBus.get_instance().subscribe(Events.SCHEDULE_ADD, event_handler, f"{type(self).__name__}.add") EventBus.get_instance().subscribe(Events.SCHEDULE_ADD, event_handler, f"{__class__.__name__}.add")
def get_all(self) -> dict[str, Cron]: def get_all(self) -> dict[str, Cron]:
"""Return the jobs.""" """Return the jobs."""
@ -101,7 +99,7 @@ class Scheduler(metaclass=Singleton):
return id in self._jobs return id in self._jobs
def add( def add(
self, timer: str, func: Callable[..., Any], args: tuple = (), kwargs: dict | None = None, id: str | None = None self, timer: str, func: callable, args: tuple = (), kwargs: dict | None = None, id: str | None = None
) -> str: ) -> str:
""" """
Add a job to the schedule. Add a job to the schedule.

View file

@ -1,5 +1,4 @@
import inspect import inspect
from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from typing import Annotated, Any, TypeVar, get_args, get_origin, get_type_hints from typing import Annotated, Any, TypeVar, get_args, get_origin, get_type_hints
@ -118,7 +117,7 @@ class Services(metaclass=Singleton):
return candidates[0] return candidates[0]
def _build_call_args(self, handler: Callable[..., Any], overrides: dict[str, Any]) -> dict[str, Any]: def _build_call_args(self, handler: callable, overrides: dict[str, Any]) -> dict[str, Any]:
sig: inspect.Signature = inspect.signature(handler) sig: inspect.Signature = inspect.signature(handler)
try: try:
@ -170,10 +169,10 @@ class Services(metaclass=Singleton):
return resolved return resolved
async def handle_async(self, handler: Callable[..., Any], **kwargs) -> Any: async def handle_async(self, handler: callable, **kwargs) -> Any:
resolved: dict[str, Any] = self._build_call_args(handler, kwargs) resolved: dict[str, Any] = self._build_call_args(handler, kwargs)
return await handler(**resolved) return await handler(**resolved)
def handle_sync(self, handler: Callable[..., Any], **kwargs) -> Any: def handle_sync(self, handler: callable, **kwargs) -> Any:
resolved: dict[str, Any] = self._build_call_args(handler, kwargs) resolved: dict[str, Any] = self._build_call_args(handler, kwargs)
return handler(**resolved) return handler(**resolved)

View file

@ -75,7 +75,7 @@ class TerminalSessionManager(metaclass=Singleton):
self._cleanup_job_id = Scheduler.get_instance().add( self._cleanup_job_id = Scheduler.get_instance().add(
timer=CLEANUP_SCHEDULE, timer=CLEANUP_SCHEDULE,
func=self.cleanup, func=self.cleanup,
id=f"{type(self).__name__}.{type(self).cleanup.__name__}", id=f"{__class__.__name__}.{__class__.cleanup.__name__}",
) )
async def on_startup(self, _: web.Application) -> None: async def on_startup(self, _: web.Application) -> None:

View file

@ -80,7 +80,7 @@ class UpdateChecker(metaclass=Singleton):
self._notify.subscribe( self._notify.subscribe(
event=Events.STARTED, event=Events.STARTED,
callback=event_handler, callback=event_handler,
name=f"{UpdateChecker.__name__}.{type(self).attach.__name__}", name=f"{__class__.__name__}.{__class__.attach.__name__}",
) )
self._schedule_check() self._schedule_check()
@ -108,7 +108,7 @@ class UpdateChecker(metaclass=Singleton):
self._job_id = self._scheduler.add( self._job_id = self._scheduler.add(
timer=timer, timer=timer,
func=lambda: asyncio.create_task(self.check_for_updates()), func=lambda: asyncio.create_task(self.check_for_updates()),
id=f"{UpdateChecker.__name__}.{self.check_for_updates.__name__}", id=f"{__class__.__name__}.{self.check_for_updates.__name__}",
) )
async def check_for_updates(self) -> tuple[tuple[str, str | None], tuple[str, str | None]]: async def check_for_updates(self) -> tuple[tuple[str, str | None], tuple[str, str | None]]:

View file

@ -40,8 +40,7 @@ TAG_REGEX: re.Pattern[str] = re.compile(r"%{([^:}]+)(?::([^}]*))?}c")
class FileLogFormatter(logging.Formatter): class FileLogFormatter(logging.Formatter):
def formatTime(self, record, datefmt=None): # noqa: N802 def formatTime(self, record, datefmt=None): # noqa: ARG002, N802
_ = datefmt
return datetime.fromtimestamp(record.created).astimezone().isoformat(timespec="milliseconds") return datetime.fromtimestamp(record.created).astimezone().isoformat(timespec="milliseconds")
@ -79,8 +78,7 @@ class JsonLogFormatter(logging.Formatter):
return json.dumps(data, ensure_ascii=False, default=str) return json.dumps(data, ensure_ascii=False, default=str)
def formatTime(self, record, datefmt=None): # noqa: N802 def formatTime(self, record, datefmt=None): # noqa: ARG002, N802
_ = datefmt
return datetime.fromtimestamp(record.created).astimezone().isoformat(timespec="milliseconds") return datetime.fromtimestamp(record.created).astimezone().isoformat(timespec="milliseconds")
@staticmethod @staticmethod
@ -212,7 +210,7 @@ class JsonLogFormatter(logging.Formatter):
return source return source
def timed_lru_cache(ttl_seconds: float, max_size: int = 128): def timed_lru_cache(ttl_seconds: int, max_size: int = 128):
""" """
Decorator that applies an LRU cache with a time-to-live (TTL) to a function. Decorator that applies an LRU cache with a time-to-live (TTL) to a function.
Supports both synchronous and asynchronous functions. Supports both synchronous and asynchronous functions.
@ -282,9 +280,8 @@ def timed_lru_cache(ttl_seconds: float, max_size: int = 128):
return _CacheInfo(hits=0, misses=len(cache), maxsize=max_size, currsize=len(cache)) return _CacheInfo(hits=0, misses=len(cache), maxsize=max_size, currsize=len(cache))
async_wrapper_cache: Any = async_wrapper async_wrapper.cache_clear = cache_clear
async_wrapper_cache.cache_clear = cache_clear async_wrapper.cache_info = cache_info
async_wrapper_cache.cache_info = cache_info
return async_wrapper return async_wrapper
# For sync functions, use the original implementation # For sync functions, use the original implementation
@ -312,9 +309,8 @@ def timed_lru_cache(ttl_seconds: float, max_size: int = 128):
return result return result
# expose cache_clear, cache_info # expose cache_clear, cache_info
sync_wrapper_cache: Any = sync_wrapper sync_wrapper.cache_clear = cached.cache_clear
sync_wrapper_cache.cache_clear = cached.cache_clear sync_wrapper.cache_info = cached.cache_info
sync_wrapper_cache.cache_info = cached.cache_info
return sync_wrapper return sync_wrapper
return decorator return decorator
@ -385,12 +381,7 @@ def _is_safe_key(key: Any) -> bool:
def merge_dict( def merge_dict(
source: dict, source: dict, destination: dict, max_depth: int = 50, max_list_size: int = 10000, _depth: int = 0, _seen: set = None
destination: dict,
max_depth: int = 50,
max_list_size: int = 10000,
_depth: int = 0,
_seen: set | None = None,
) -> dict: ) -> dict:
""" """
Merge data from source into destination. Merge data from source into destination.
@ -481,7 +472,7 @@ def merge_dict(
return destination_copy return destination_copy
def check_id(file: Path) -> bool | Path: def check_id(file: Path) -> bool | str:
""" """
Check if we are able to get an id from the file name. Check if we are able to get an id from the file name.
if so check if any video file with the same id exists. if so check if any video file with the same id exists.
@ -500,8 +491,6 @@ def check_id(file: Path) -> bool | Path:
return False return False
id: str | None = match.groupdict().get("id") id: str | None = match.groupdict().get("id")
if id is None:
return False
try: try:
if not file.parent.exists(): if not file.parent.exists():
@ -556,9 +545,9 @@ def validate_url(url: str, allow_internal: bool = False) -> bool:
from yarl import URL from yarl import URL
parsed_url = URL(url) parsed_url = URL(url)
except ValueError as exc: except ValueError:
msg = "Invalid URL." msg = "Invalid URL."
raise ValueError(msg) from exc raise ValueError(msg) # noqa: B904
# Check allowed schemes # Check allowed schemes
if parsed_url.scheme not in ["http", "https"]: if parsed_url.scheme not in ["http", "https"]:
@ -607,7 +596,7 @@ def validate_uuid(uuid_str: str, version: int = 4) -> bool:
@timed_lru_cache(ttl_seconds=60, max_size=256) @timed_lru_cache(ttl_seconds=60, max_size=256)
def get_file_sidecar(file: Path | None = None) -> dict[str, list[dict[str, Any]]]: def get_file_sidecar(file: Path | None = None) -> dict[dict]:
""" """
Get sidecar files for the given file. Get sidecar files for the given file.
@ -936,7 +925,7 @@ def get_file(download_path: str | Path, file: str | Path) -> tuple[Path, int]:
download_path = Path(download_path) download_path = Path(download_path)
try: try:
realFile: Path = Path(calc_download_path(base_path=download_path, folder=str(file), create_path=False)) realFile: str = Path(calc_download_path(base_path=download_path, folder=str(file), create_path=False))
if realFile.exists(): if realFile.exists():
return (realFile, 200) return (realFile, 200)
except Exception as e: except Exception as e:
@ -947,11 +936,11 @@ def get_file(download_path: str | Path, file: str | Path) -> tuple[Path, int]:
) )
return (Path(file), 404) return (Path(file), 404)
possibleFile: bool | Path = check_id(file=realFile) possibleFile: bool | str = check_id(file=realFile)
if not isinstance(possibleFile, Path): if not possibleFile:
return (realFile, 404) return (realFile, 404)
return (possibleFile, 302) return (Path(possibleFile), 302)
def encrypt_data(data: str, key: bytes) -> str: def encrypt_data(data: str, key: bytes) -> str:
@ -973,7 +962,7 @@ def encrypt_data(data: str, key: bytes) -> str:
return base64.urlsafe_b64encode(iv + ciphertext + tag).decode() return base64.urlsafe_b64encode(iv + ciphertext + tag).decode()
def decrypt_data(data: str, key: bytes) -> str | None: def decrypt_data(data: str, key: bytes) -> str:
""" """
Decrypts AES-GCM encrypted data Decrypts AES-GCM encrypted data
@ -986,8 +975,8 @@ def decrypt_data(data: str, key: bytes) -> str | None:
""" """
try: try:
raw: bytes = base64.urlsafe_b64decode(data) data = base64.urlsafe_b64decode(data)
iv, ciphertext, tag = raw[:12], raw[12:-16], raw[-16:] iv, ciphertext, tag = data[:12], data[12:-16], data[-16:]
cipher = AES.new(key, AES.MODE_GCM, nonce=iv) cipher = AES.new(key, AES.MODE_GCM, nonce=iv)
plaintext = cipher.decrypt_and_verify(ciphertext, tag) plaintext = cipher.decrypt_and_verify(ciphertext, tag)
return plaintext.decode() return plaintext.decode()
@ -998,7 +987,7 @@ def decrypt_data(data: str, key: bytes) -> str | None:
def get( def get(
data: dict | list, data: dict | list,
path: str | list | None = None, path: str | list | None = None,
default: Any = None, default: any = None,
separator=".", separator=".",
): ):
""" """
@ -1122,7 +1111,7 @@ def get_files(
extra={"requested_dir": dir, "base_path": str(base_path), "exception_type": type(e).__name__}, extra={"requested_dir": dir, "base_path": str(base_path), "exception_type": type(e).__name__},
exc_info=True, exc_info=True,
) )
return [], 0 return []
try: try:
dir_path.relative_to(base_path) dir_path.relative_to(base_path)
@ -1133,7 +1122,7 @@ def get_files(
base_path, base_path,
extra={"path": str(dir_path), "base_path": str(base_path)}, extra={"path": str(dir_path), "base_path": str(base_path)},
) )
return [], 0 return []
if not str(dir_path).startswith(str(base_path)): if not str(dir_path).startswith(str(base_path)):
LOG.warning( LOG.warning(
@ -1142,7 +1131,7 @@ def get_files(
base_path, base_path,
extra={"path": str(dir_path), "base_path": str(base_path)}, extra={"path": str(dir_path), "base_path": str(base_path)},
) )
return [], 0 return []
if not dir_path.is_dir(): if not dir_path.is_dir():
LOG.warning( LOG.warning(
@ -1150,7 +1139,7 @@ def get_files(
dir_path, dir_path,
extra={"path": str(dir_path), "base_path": str(base_path)}, extra={"path": str(dir_path), "base_path": str(base_path)},
) )
return [], 0 return []
contents: list = [] contents: list = []
for file in dir_path.iterdir(): for file in dir_path.iterdir():
@ -1309,7 +1298,7 @@ def load_cookies(file: str | Path) -> tuple[bool, MozillaCookieJar]:
try: try:
from http.cookiejar import MozillaCookieJar from http.cookiejar import MozillaCookieJar
cookies = MozillaCookieJar(str(file), delayload=False, policy=None) cookies = MozillaCookieJar(str(file), None, None)
cookies.load() cookies.load()
return (True, cookies) return (True, cookies)
@ -1488,6 +1477,35 @@ def str_to_dt(time_str: str, now=None) -> datetime:
return dt return dt
def list_folders(path: Path, base: Path, depth_limit: int) -> list[str]:
"""
List all folders relative to a base path, up to a specified depth limit.
Args:
path (Path): The path to start listing folders from.
base (Path): The base path to which the folders should be relative.
depth_limit (int): The maximum depth to traverse from the base path.
Returns:
list[str]: A list of folder paths relative to the base path, up to the specified
"""
if "/" == str(path):
return []
rel_depth: int = len(path.relative_to(base).parts)
if rel_depth > depth_limit:
return []
folders: list[str] = []
for entry in path.iterdir():
if entry.is_dir():
folders.append(str(entry.relative_to(base)))
folders.extend(list_folders(entry, base, depth_limit))
return folders
def get_channel_images(thumbnails: list[dict]) -> dict: def get_channel_images(thumbnails: list[dict]) -> dict:
""" """
Extract channel images from a list of thumbnail dictionaries. Extract channel images from a list of thumbnail dictionaries.

View file

@ -1,20 +1,6 @@
import sys
from typing import Any from typing import Any
def _dict_value(data: dict[Any, Any], key: Any) -> tuple[bool, Any]:
if key not in data or data[key] is None:
return False, None
return True, data[key]
def _dict_delete(data: dict[Any, Any], key: Any) -> bool:
if key not in data:
return False
del data[key]
return True
def get_value(value: Any) -> Any: def get_value(value: Any) -> Any:
""" """
Return the result of calling `value` if it is callable, else return `value`. Return the result of calling `value` if it is callable, else return `value`.
@ -105,19 +91,14 @@ def ag(
return val return val
return get_value(default) return get_value(default)
key = path key = path
if isinstance(array, dict): if isinstance(array, dict) and key in array and array[key] is not None:
found, value = _dict_value(array, key) return array[key]
if found:
return value
if not isinstance(key, str) or separator not in key: if not isinstance(key, str) or separator not in key:
return get_value(default) return get_value(default)
current = array current = array
for segment in key.split(separator): for segment in key.split(separator):
if isinstance(current, dict): if isinstance(current, dict) and segment in current and current[segment] is not None:
found, value = _dict_value(current, segment) current = current[segment]
if not found:
return get_value(default)
current = value
elif isinstance(current, list): elif isinstance(current, list):
try: try:
idx = int(segment) idx = int(segment)
@ -169,8 +150,7 @@ def ag_exists(data: dict[Any, Any] | list[Any] | object, path: str | int, separa
except TypeError: except TypeError:
return False return False
if isinstance(data, dict): if isinstance(data, dict):
found, _ = _dict_value(data, path) if path in data and data[path] is not None:
if found:
return True return True
elif isinstance(data, list) and isinstance(path, int): elif isinstance(data, list) and isinstance(path, int):
return 0 <= path < len(data) and data[path] is not None return 0 <= path < len(data) and data[path] is not None
@ -178,11 +158,8 @@ def ag_exists(data: dict[Any, Any] | list[Any] | object, path: str | int, separa
segments = path_str.split(separator) segments = path_str.split(separator)
current = data current = data
for seg in segments: for seg in segments:
if isinstance(current, dict): if isinstance(current, dict) and seg in current and current[seg] is not None:
found, value = _dict_value(current, seg) current = current[seg]
if not found:
return False
current = value
elif isinstance(current, list): elif isinstance(current, list):
try: try:
idx = int(seg) idx = int(seg)
@ -212,16 +189,17 @@ def ag_delete(
The modified structure. The modified structure.
""" """
if isinstance(path, list | tuple):
for p in path:
ag_delete(data, p, separator)
return data
if not isinstance(data, dict | list): if not isinstance(data, dict | list):
try: try:
data = vars(data) data = vars(data)
except TypeError: except TypeError:
return data # type: ignore return data # type: ignore
if isinstance(path, list | tuple): if isinstance(data, dict) and path in data:
for p in path: del data[path]
ag_delete(data, p, separator)
return data
if isinstance(data, dict) and _dict_delete(data, path):
return data return data
if isinstance(data, list) and isinstance(path, int): if isinstance(data, list) and isinstance(path, int):
if 0 <= path < len(data): if 0 <= path < len(data):
@ -232,12 +210,8 @@ def ag_delete(
last = segments.pop() last = segments.pop()
current = data current = data
for seg in segments: for seg in segments:
if isinstance(current, dict): if isinstance(current, dict) and seg in current:
found, value = _dict_value(current, seg) current = current[seg]
if not found:
current = None
break
current = value
elif isinstance(current, list): elif isinstance(current, list):
try: try:
idx = int(seg) idx = int(seg)
@ -254,8 +228,8 @@ def ag_delete(
break break
if current is None: if current is None:
return data return data
if isinstance(current, dict): if isinstance(current, dict) and last in current:
_dict_delete(current, last) del current[last]
elif isinstance(current, list): elif isinstance(current, list):
try: try:
idx = int(last) idx = int(last)
@ -272,12 +246,10 @@ def run_tests():
assert ag_set(d, "a.c.d", 42) == {"a": {"b": 1, "c": {"d": 42}}} assert ag_set(d, "a.c.d", 42) == {"a": {"b": 1, "c": {"d": 42}}}
def test_ag_set_overwrite_error(): def test_ag_set_overwrite_error():
error = ""
try: try:
ag_set({"a": 1}, "a.b", 2) ag_set({"a": 1}, "a.b", 2)
except RuntimeError as e: except RuntimeError as e:
error = str(e) assert "Cannot set value at path" in str(e) # noqa: PT017
assert "Cannot set value at path" in error
def test_ag_get_value_callable_and_value(): def test_ag_get_value_callable_and_value():
assert get_value(5) == 5 assert get_value(5) == 5
@ -348,9 +320,9 @@ def run_tests():
]: ]:
try: try:
test() test()
sys.stdout.write(f"PASS: {test.__name__}\n") print(f"PASS: {test.__name__}") # noqa: T201
except AssertionError: except AssertionError:
sys.stdout.write(f"FAIL {test.__name__}\n") print(f"FAIL {test.__name__}") # noqa: T201
if __name__ == "__main__": if __name__ == "__main__":

View file

@ -38,7 +38,7 @@ class Cache(metaclass=ThreadSafe):
Scheduler.get_instance().add( Scheduler.get_instance().add(
timer="* * * * *", timer="* * * * *",
func=self.cleanup, func=self.cleanup,
id=f"{type(self).__name__}.{type(self).cleanup.__name__}", id=f"{__class__.__name__}.{__class__.cleanup.__name__}",
) )
def set(self, key: str, value: Any, ttl: float | None = None) -> None: def set(self, key: str, value: Any, ttl: float | None = None) -> None:

View file

@ -3,7 +3,7 @@ from __future__ import annotations
import http.cookiejar import http.cookiejar
from abc import ABC from abc import ABC
from collections.abc import Callable from collections.abc import Callable
from typing import Any, ClassVar, cast from typing import Any, ClassVar
from yt_dlp.networking.common import ( from yt_dlp.networking.common import (
_REQUEST_HANDLERS, _REQUEST_HANDLERS,
@ -223,13 +223,11 @@ class CFSolverRH(RequestHandler, ABC):
try: try:
response: Response = director.send(request) response: Response = director.send(request)
except HTTPError as error: except HTTPError as error:
if error.response and is_cf_challenge( if error.response and is_cf_challenge(getattr(error.response, "status", None), error.response.headers):
getattr(error.response, "status", None), cast("Any", error.response.headers)
):
return self._retry_with_clearance(request, error.response, director) return self._retry_with_clearance(request, error.response, director)
raise raise
if is_cf_challenge(getattr(response, "status", None), cast("Any", response.headers)): if is_cf_challenge(getattr(response, "status", None), response.headers):
return self._retry_with_clearance(request, response, director) return self._retry_with_clearance(request, response, director)
return response return response

View file

@ -1,18 +1,16 @@
# flake8: noqa: S310
from __future__ import annotations from __future__ import annotations
import json import json
import time import time
import urllib.request import urllib.request
from typing import TYPE_CHECKING, Any from typing import Any
from urllib.parse import urlparse from urllib.parse import urlparse
from app.library.log import get_logger from app.library.log import get_logger
from .cache import Cache from .cache import Cache
if TYPE_CHECKING:
from collections.abc import Mapping
CACHE: Cache = Cache() CACHE: Cache = Cache()
LOG = get_logger() LOG = get_logger()
@ -59,7 +57,7 @@ def solver(url: str, cookies: list[dict[str, Any]], user_agent: str | None) -> d
if user_agent: if user_agent:
payload.setdefault("headers", {})["User-Agent"] = user_agent payload.setdefault("headers", {})["User-Agent"] = user_agent
req = urllib.request.Request( # noqa: S310 req = urllib.request.Request(
endpoint, endpoint,
data=json.dumps(payload).encode("utf-8"), data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"}, headers={"Content-Type": "application/json"},
@ -70,7 +68,7 @@ def solver(url: str, cookies: list[dict[str, Any]], user_agent: str | None) -> d
"Solving Cloudflare challenge for '%s' via FlareSolverr.", host, extra={"host": host, "endpoint": endpoint} "Solving Cloudflare challenge for '%s' via FlareSolverr.", host, extra={"host": host, "endpoint": endpoint}
) )
start_time = time.time() start_time = time.time()
with urllib.request.urlopen(req, timeout=float(config.flaresolverr_client_timeout)) as resp: # noqa: S310 with urllib.request.urlopen(req, timeout=float(config.flaresolverr_client_timeout)) as resp:
result = json.loads(resp.read().decode("utf-8")) result = json.loads(resp.read().decode("utf-8"))
if "ok" != result.get("status"): if "ok" != result.get("status"):
@ -100,7 +98,7 @@ def solver(url: str, cookies: list[dict[str, Any]], user_agent: str | None) -> d
return CACHE.get(host) return CACHE.get(host)
def is_cf_challenge(status: int | None, headers: Mapping[str, Any] | None) -> bool: def is_cf_challenge(status: int | None, headers: dict[str, Any] | None) -> bool:
""" """
Determine whether a response indicates a Cloudflare challenge. Determine whether a response indicates a Cloudflare challenge.
""" """

View file

@ -2,7 +2,6 @@ import logging
import multiprocessing import multiprocessing
import os import os
import re import re
import shutil
import sys import sys
import time import time
from logging.handlers import TimedRotatingFileHandler from logging.handlers import TimedRotatingFileHandler
@ -31,7 +30,7 @@ APP_THIRD_PARTY_LOG_LEVELS: tuple[tuple[str, int], ...] = (
if TYPE_CHECKING: if TYPE_CHECKING:
from subprocess import CompletedProcess from subprocess import CompletedProcess
SUPPORTED_CODECS: tuple[str, ...] = ("h264_qsv", "h264_nvenc", "h264_amf", "h264_videotoolbox", "h264_vaapi", "libx264") SUPPORTED_CODECS: tuple[str] = ("h264_qsv", "h264_nvenc", "h264_amf", "h264_videotoolbox", "h264_vaapi", "libx264")
"Supported encoder names in order of preference." "Supported encoder names in order of preference."
@ -48,6 +47,9 @@ class Config(metaclass=Singleton):
download_path: str = "." download_path: str = "."
"""The path to the download directory.""" """The path to the download directory."""
download_path_depth: int = 2
"""How many subdirectories to show in auto complete."""
download_info_expires: int = 10800 download_info_expires: int = 10800
"""How long (in seconds) the download info is valid before it needs to be re-extracted.""" """How long (in seconds) the download info is valid before it needs to be re-extracted."""
@ -180,7 +182,7 @@ class Config(metaclass=Singleton):
file_logging: bool = True file_logging: bool = True
"Enable file logging." "Enable file logging."
secret_key: str | bytes secret_key: str
"The secret key to use for the application." "The secret key to use for the application."
tasks_handler_timer: str = "15 */1 * * *" tasks_handler_timer: str = "15 */1 * * *"
@ -228,9 +230,6 @@ class Config(metaclass=Singleton):
default_pagination: int = 50 default_pagination: int = 50
"""The default number of items per page for pagination.""" """The default number of items per page for pagination."""
queue_display_limit: int = 100
"""Maximum number of queued downloads returned to the UI. 0 means unlimited."""
task_handler_random_delay: float = 60.0 task_handler_random_delay: float = 60.0
"""The maximum random delay in seconds before starting a task handler.""" """The maximum random delay in seconds before starting a task handler."""
@ -283,10 +282,10 @@ class Config(metaclass=Singleton):
"max_workers_per_extractor", "max_workers_per_extractor",
"extract_info_timeout", "extract_info_timeout",
"debugpy_port", "debugpy_port",
"download_path_depth",
"download_info_expires", "download_info_expires",
"auto_clear_history_days", "auto_clear_history_days",
"default_pagination", "default_pagination",
"queue_display_limit",
"extract_info_concurrency", "extract_info_concurrency",
"thumb_concurrency", "thumb_concurrency",
"flaresolverr_max_timeout", "flaresolverr_max_timeout",
@ -347,7 +346,6 @@ class Config(metaclass=Singleton):
"app_build_date", "app_build_date",
"app_branch", "app_branch",
"default_pagination", "default_pagination",
"queue_display_limit",
"check_for_updates", "check_for_updates",
"new_version", "new_version",
"yt_new_version", "yt_new_version",
@ -378,7 +376,7 @@ class Config(metaclass=Singleton):
LOG = get_logger() LOG = get_logger()
self.config_path = os.environ.get("YTP_CONFIG_PATH", None) or str(Path(baseDefaultPath) / "var" / "config") self.config_path = os.environ.get("YTP_CONFIG_PATH", None) or str(Path(baseDefaultPath) / "var" / "config")
envFile: Path = Path(self.config_path) / ".env" envFile: str = Path(self.config_path) / ".env"
if envFile.exists(): if envFile.exists():
LOG.info("Loading environment variables from '%s'.", envFile) LOG.info("Loading environment variables from '%s'.", envFile)
@ -389,7 +387,7 @@ class Config(metaclass=Singleton):
self.download_path = os.environ.get("YTP_DOWNLOAD_PATH", None) or str( self.download_path = os.environ.get("YTP_DOWNLOAD_PATH", None) or str(
Path(baseDefaultPath) / "var" / "downloads" Path(baseDefaultPath) / "var" / "downloads"
) )
self.app_path = str(Path(__file__).parent.parent.absolute()) self.app_path = Path(__file__).parent.parent.absolute()
for k, v in self._get_attributes().items(): for k, v in self._get_attributes().items():
if k.startswith("_") or k in self._manual_vars: if k.startswith("_") or k in self._manual_vars:
@ -502,7 +500,7 @@ class Config(metaclass=Singleton):
handler.setFormatter(formatter) handler.setFormatter(formatter)
logging.getLogger().addHandler(handler) logging.getLogger().addHandler(handler)
key_file: Path = Path(self.config_path) / "secret.key" key_file: str = Path(self.config_path) / "secret.key"
if key_file.exists() and key_file.stat().st_size > 2: if key_file.exists() and key_file.stat().st_size > 2:
with open(key_file, "rb") as f: with open(key_file, "rb") as f:
@ -512,7 +510,7 @@ class Config(metaclass=Singleton):
with open(key_file, "wb") as f: with open(key_file, "wb") as f:
f.write(self.secret_key) f.write(self.secret_key)
self.started = int(time.time()) self.started = time.time()
for _tool, _level in APP_THIRD_PARTY_LOG_LEVELS: for _tool, _level in APP_THIRD_PARTY_LOG_LEVELS:
logging.getLogger(_tool).setLevel(_level) logging.getLogger(_tool).setLevel(_level)
@ -546,7 +544,7 @@ class Config(metaclass=Singleton):
def _get_attributes(self) -> dict: def _get_attributes(self) -> dict:
attrs: dict = {} attrs: dict = {}
vClass: type = self.__class__ vClass: str = self.__class__
for attribute in vClass.__dict__: for attribute in vClass.__dict__:
if attribute.startswith("_"): if attribute.startswith("_"):
@ -602,7 +600,7 @@ class Config(metaclass=Singleton):
dict[str, str]: The replacer variables. dict[str, str]: The replacer variables.
""" """
keys: tuple[str, ...] = ("os_sep", "download_path", "temp_path", "config_path", "archive_file") keys: tuple[str] = ("os_sep", "download_path", "temp_path", "config_path", "archive_file")
return {k: getattr(self, k) for k in keys} return {k: getattr(self, k) for k in keys}
@staticmethod @staticmethod
@ -620,20 +618,15 @@ class Config(metaclass=Singleton):
This is used to set the version to the latest git tag. This is used to set the version to the latest git tag.
""" """
LOG = get_logger() LOG = get_logger()
git_path: Path = Path(__file__).parent / ".." / ".." / ".git" git_path: str = Path(__file__).parent / ".." / ".." / ".git"
if not git_path.exists(): if not git_path.exists():
return return
try: try:
import subprocess import subprocess
git = shutil.which("git")
if not git:
LOG.warning("Git executable was not found.")
return
branch_result: CompletedProcess[str] = subprocess.run( branch_result: CompletedProcess[str] = subprocess.run(
[git, "-c", f"safe.directory={git_path.parent!s}", "rev-parse", "--abbrev-ref", "HEAD"], ["git", "-c", f"safe.directory={git_path.parent!s}", "rev-parse", "--abbrev-ref", "HEAD"], # noqa: S607
cwd=os.path.dirname(git_path), cwd=os.path.dirname(git_path),
capture_output=True, capture_output=True,
text=True, text=True,
@ -651,7 +644,7 @@ class Config(metaclass=Singleton):
return return
commit_result: CompletedProcess[str] = subprocess.run( commit_result: CompletedProcess[str] = subprocess.run(
[git, "-c", f"safe.directory={git_path.parent!s}", "log", "-1", "--format=%ct_%H"], ["git", "-c", f"safe.directory={git_path.parent!s}", "log", "-1", "--format=%ct_%H"], # noqa: S607
cwd=os.path.dirname(git_path), cwd=os.path.dirname(git_path),
capture_output=True, capture_output=True,
text=True, text=True,

View file

@ -39,14 +39,6 @@ if TYPE_CHECKING:
class Download: class Download:
update_task: asyncio.Task | None = None update_task: asyncio.Task | None = None
@staticmethod
def _raise_no_formats(msg: str) -> None:
raise ValueError(msg)
@staticmethod
def _raise_interrupted() -> None:
raise SystemExit(130)
def __init__(self, info: ItemDTO, info_dict: dict | None = None, logs: list[str] | None = None): def __init__(self, info: ItemDTO, info_dict: dict | None = None, logs: list[str] | None = None):
""" """
Initialize download task. Initialize download task.
@ -59,7 +51,7 @@ class Download:
""" """
config: Config = Config.get_instance() config: Config = Config.get_instance()
self.download_dir: str = info.download_dir or "" self.download_dir: str = info.download_dir
self.temp_dir: str | None = info.temp_dir self.temp_dir: str | None = info.temp_dir
self.template: str | None = info.template self.template: str | None = info.template
self.template_chapter: str | None = info.template_chapter self.template_chapter: str | None = info.template_chapter
@ -106,16 +98,6 @@ class Download:
params: dict[str, Any] = {} params: dict[str, Any] = {}
download_skipped = False download_skipped = False
hook_handlers = self._hook_handlers
if hook_handlers is None:
msg = "_hook_handlers must be initialized before _download(). Call start() first."
raise RuntimeError(msg)
status_queue = self.status_queue
if status_queue is None:
msg = "status_queue must be initialized before _download(). Call start() first."
raise RuntimeError(msg)
try: try:
params = ( params = (
self.info.get_ytdlp_opts() self.info.get_ytdlp_opts()
@ -143,9 +125,9 @@ class Download:
params.update( params.update(
{ {
"progress_hooks": [hook_handlers.progress_hook], "progress_hooks": [self._hook_handlers.progress_hook],
"postprocessor_hooks": [hook_handlers.postprocessor_hook], "postprocessor_hooks": [self._hook_handlers.postprocessor_hook],
"post_hooks": [hook_handlers.post_hook], "post_hooks": [self._hook_handlers.post_hook],
} }
) )
@ -155,10 +137,7 @@ class Download:
if self.info.cookies: if self.info.cookies:
try: try:
cookie_file = ( cookie_file = Path(self._temp_manager.temp_path or self.temp_dir) / f"cookie_{self.info._id}.txt"
Path(self._temp_manager.temp_path or self.temp_dir or self.download_dir)
/ f"cookie_{self.info._id}.txt"
)
self.logger.debug( self.logger.debug(
f"Creating cookie file for '{self.info.title}' at '{cookie_file}'.", f"Creating cookie file for '{self.info.title}' at '{cookie_file}'.",
extra={ extra={
@ -194,9 +173,7 @@ class Download:
f"Info dict for '{self.info.url}' marked for re-extraction, ignoring pre-extracted info." f"Info dict for '{self.info.url}' marked for re-extraction, ignoring pre-extracted info."
) )
self.info_dict = None self.info_dict = None
elif self.info_dict.get("extractor_key") in GENERIC_EXTRACTORS and ( elif self.info_dict.get("extractor_key") in GENERIC_EXTRACTORS and self.info.get_preset().default:
(preset := self.info.get_preset()) is not None and preset.default
):
self.logger.debug( self.logger.debug(
"Removing download archive option for generic extractor on '%s'.", "Removing download archive option for generic extractor on '%s'.",
self.info.url, self.info.url,
@ -290,7 +267,7 @@ class Download:
if filtered_logs := extract_ytdlp_logs(self.logs): if filtered_logs := extract_ytdlp_logs(self.logs):
msg += " " + ", ".join(filtered_logs) msg += " " + ", ".join(filtered_logs)
self._raise_no_formats(msg) raise ValueError(msg) # noqa: TRY301
self.logger.info( self.logger.info(
f"Downloading '{self.info.title}' from '{self.info.url}' to '{self.download_dir}'.", f"Downloading '{self.info.title}' from '{self.info.url}' to '{self.download_dir}'.",
@ -350,11 +327,11 @@ class Download:
def mark_cancelled(*_) -> None: def mark_cancelled(*_) -> None:
cls._interrupted = True cls._interrupted = True
cls.to_screen("[info] Interrupt received, exiting cleanly...") cls.to_screen("[info] Interrupt received, exiting cleanly...")
self._raise_interrupted() raise SystemExit(130) # noqa: TRY301
signal.signal(signal.SIGUSR1, mark_cancelled) signal.signal(signal.SIGUSR1, mark_cancelled)
status_queue.put({"id": self.id, "status": "downloading", "download_skipped": download_skipped}) self.status_queue.put({"id": self.id, "status": "downloading", "download_skipped": download_skipped})
if isinstance(self.info_dict, dict) and len(self.info_dict) > 1: if isinstance(self.info_dict, dict) and len(self.info_dict) > 1:
self.logger.debug( self.logger.debug(
@ -409,7 +386,7 @@ class Download:
) )
ret = cls.download(url_list=[self.info.url]) ret = cls.download(url_list=[self.info.url])
status_queue.put( self.status_queue.put(
{ {
"id": self.id, "id": self.id,
"status": "finished" if 0 == ret else "error", "status": "finished" if 0 == ret else "error",
@ -430,7 +407,7 @@ class Download:
} }
}, },
) )
status_queue.put( self.status_queue.put(
{ {
"id": self.id, "id": self.id,
"status": "skip", "status": "skip",
@ -455,7 +432,7 @@ class Download:
} }
}, },
) )
status_queue.put( self.status_queue.put(
{ {
"id": self.id, "id": self.id,
"status": "error", "status": "error",
@ -465,7 +442,7 @@ class Download:
} }
) )
finally: finally:
status_queue.put(Terminator()) self.status_queue.put(Terminator())
if cookie_file and cookie_file.exists(): if cookie_file and cookie_file.exists():
try: try:
cookie_file.unlink() cookie_file.unlink()
@ -522,24 +499,24 @@ class Download:
Process exit code or None Process exit code or None
""" """
status_queue: Any = Config.get_manager().Queue() self.status_queue = Config.get_manager().Queue()
self.status_queue = status_queue
temp_path = self._temp_manager.create_temp_path() if temp_path := self._temp_manager.create_temp_path():
self.info.temp_path = str(temp_path)
self._status_tracker = StatusTracker( self._status_tracker = StatusTracker(
info=self.info, info=self.info,
download_id=self.id, download_id=self.id,
download_dir=self.download_dir, download_dir=self.download_dir,
temp_path=temp_path, temp_path=temp_path,
status_queue=status_queue, status_queue=self.status_queue,
logger=self.logger, logger=self.logger,
debug=self.debug, debug=self.debug,
) )
self._hook_handlers = HookHandlers( self._hook_handlers = HookHandlers(
download_id=self.id, download_id=self.id,
status_queue=status_queue, status_queue=self.status_queue,
logger=self.logger, logger=self.logger,
debug=self.debug, debug=self.debug,
) )
@ -552,8 +529,7 @@ class Download:
EventBus.get_instance().emit(Events.ITEM_UPDATED, data=self.info) EventBus.get_instance().emit(Events.ITEM_UPDATED, data=self.info)
asyncio.create_task(self._status_tracker.progress_update(), name=f"update-{self.id}") asyncio.create_task(self._status_tracker.progress_update(), name=f"update-{self.id}")
proc = self._process_manager.proc ret = await asyncio.get_running_loop().run_in_executor(None, self._process_manager.proc.join)
ret = await asyncio.get_running_loop().run_in_executor(None, proc.join) if proc else None
if self._status_tracker.final_update: if self._status_tracker.final_update:
return ret return ret

View file

@ -4,7 +4,7 @@ import time
import uuid import uuid
from numbers import Number from numbers import Number
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING
import yt_dlp.utils import yt_dlp.utils
@ -67,7 +67,7 @@ async def add_item(
already=None, already=None,
logs: list | None = None, logs: list | None = None,
yt_params: dict | None = None, yt_params: dict | None = None,
) -> dict[str, Any]: ) -> dict[str, str]:
""" """
Route an entry to the appropriate processor based on type. Route an entry to the appropriate processor based on type.
@ -102,7 +102,7 @@ async def add_item(
async def add( async def add(
queue: "DownloadQueue", item: "Item", already: set | None = None, entry: dict | None = None queue: "DownloadQueue", item: "Item", already: set | None = None, entry: dict | None = None
) -> dict[str, Any]: ) -> dict[str, str]:
""" """
Add an item to the download queue. Add an item to the download queue.
@ -204,7 +204,7 @@ async def add(
) )
if archive_file := dlInfo.info.get_ytdlp_opts().get_all().get("download_archive"): if archive_file := dlInfo.info.get_ytdlp_opts().get_all().get("download_archive"):
dlInfo.info.msg = (dlInfo.info.msg or "") + f" Found in archive '{archive_file}'." dlInfo.info.msg += f" Found in archive '{archive_file}'."
await queue.done.put(dlInfo) await queue.done.put(dlInfo)
@ -306,10 +306,7 @@ async def add(
new_archive_id: str | None = None new_archive_id: str | None = None
if entry.get("extractor_key") and entry.get("id"): if entry.get("extractor_key") and entry.get("id"):
_extractor_key = entry.get("extractor_key") new_archive_id: str = f"{entry.get('extractor_key').lower()} {entry.get('id')}"
_entry_id = entry.get("id")
if isinstance(_extractor_key, str) and _entry_id is not None:
new_archive_id = f"{_extractor_key.lower()} {_entry_id!s}"
if new_archive_id != archive_id: if new_archive_id != archive_id:
extra_ids.append(new_archive_id) extra_ids.append(new_archive_id)
@ -339,7 +336,7 @@ async def add(
) )
) )
dlInfo.info.msg = (dlInfo.info.msg or "") + f" Found in archive '{_archive_file}'." dlInfo.info.msg += f" Found in archive '{_archive_file}'."
await queue.done.put(dlInfo) await queue.done.put(dlInfo)
queue._notify.emit( queue._notify.emit(
@ -377,7 +374,7 @@ async def add(
if condition.extras.get("ignore_download", False): if condition.extras.get("ignore_download", False):
extra_msg: str = "" extra_msg: str = ""
if _archive_file and not condition.extras.get("no_archive", False) and archive_id: if _archive_file and not condition.extras.get("no_archive", False):
archive_add(_archive_file, [archive_id]) archive_add(_archive_file, [archive_id])
extra_msg = f" and added to archive file '{_archive_file}'" extra_msg = f" and added to archive file '{_archive_file}'"
@ -388,7 +385,7 @@ async def add(
if not store_type: if not store_type:
dlInfo = Download( dlInfo = Download(
info=ItemDTO( info=ItemDTO(
id=str(entry.get("id") or item.url), id=entry.get("id"),
title=_name, title=_name,
url=item.url, url=item.url,
preset=item.preset, preset=item.preset,
@ -404,9 +401,6 @@ async def add(
LOG.info(log_message) LOG.info(log_message)
queue._notify.emit(Events.LOG_INFO, data={}, title="Ignored Download", message=log_message) queue._notify.emit(Events.LOG_INFO, data={}, title="Ignored Download", message=log_message)
if dlInfo is None:
msg = "Download info is not available."
raise RuntimeError(msg)
queue._notify.emit( queue._notify.emit(
Events.ITEM_MOVED, Events.ITEM_MOVED,
data={"to": "history", "preset": dlInfo.info.preset, "item": dlInfo.info}, data={"to": "history", "preset": dlInfo.info.preset, "item": dlInfo.info},

View file

@ -121,8 +121,8 @@ async def process_playlist(
max_downloads: int = -1 max_downloads: int = -1
ytdlp_opts: dict[str, Any] = item.get_ytdlp_opts().get_all() ytdlp_opts: dict[str, Any] = item.get_ytdlp_opts().get_all()
if isinstance(ytdlp_opts.get("max_downloads"), int): if ytdlp_opts.get("max_downloads") and isinstance(ytdlp_opts.get("max_downloads"), int):
max_downloads = ytdlp_opts["max_downloads"] max_downloads: int = ytdlp_opts.get("max_downloads")
results: list[dict[str, str]] = [] results: list[dict[str, str]] = []
for i, etr in enumerate(entries, start=1): for i, etr in enumerate(entries, start=1):

View file

@ -240,7 +240,7 @@ class PoolManager:
await self.queue.done.put(entry) await self.queue.done.put(entry)
self._notify.emit( self._notify.emit(
Events.ITEM_MOVED, Events.ITEM_MOVED,
data={"from": "queue", "to": "history", "preset": entry.info.preset, "item": entry.info}, data={"to": "history", "preset": entry.info.preset, "item": entry.info},
title=nTitle, title=nTitle,
message=nMessage, message=nMessage,
) )

View file

@ -99,19 +99,15 @@ class ProcessManager:
if not self.running(): if not self.running():
return False return False
proc = self.proc procId: int | None = self.proc.ident
if proc is None:
return False
procId: int | None = proc.ident
try: try:
self.logger.info( self.logger.info(
"Stopping download process PID=%s.", "Stopping download process PID=%s.",
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"process_ident": procId, "process_ident": procId,
"is_live": self.is_live, "is_live": self.is_live,
} }
@ -121,25 +117,25 @@ class ProcessManager:
if self.is_live: if self.is_live:
self.logger.debug( self.logger.debug(
"Requesting graceful live cancellation for PID=%s.", "Requesting graceful live cancellation for PID=%s.",
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"is_live": self.is_live, "is_live": self.is_live,
} }
}, },
) )
self.cancel_event.set() self.cancel_event.set()
if wait_for_process_with_timeout(proc, 10): if wait_for_process_with_timeout(self.proc, 10):
self.logger.debug( self.logger.debug(
"Download process PID=%s stopped gracefully.", "Download process PID=%s stopped gracefully.",
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"is_live": self.is_live, "is_live": self.is_live,
} }
}, },
@ -147,43 +143,43 @@ class ProcessManager:
return True return True
self.logger.warning( self.logger.warning(
"Download process PID=%s did not respond to live cancellation; forcing termination.", "Download process PID=%s did not respond to live cancellation; forcing termination.",
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"is_live": self.is_live, "is_live": self.is_live,
"force": True, "force": True,
} }
}, },
) )
elif proc.pid and "posix" == os.name: elif self.proc.pid and "posix" == os.name:
signal_name = "SIGUSR1" signal_name = "SIGUSR1"
try: try:
self.logger.debug( self.logger.debug(
"Sending %s signal to download process PID=%s.", "Sending %s signal to download process PID=%s.",
signal_name, signal_name,
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"signal": signal_name, "signal": signal_name,
} }
}, },
) )
os.kill(proc.pid, signal.SIGUSR1) os.kill(self.proc.pid, signal.SIGUSR1)
if wait_for_process_with_timeout(proc, 5): if wait_for_process_with_timeout(self.proc, 5):
self.logger.debug( self.logger.debug(
"Download process PID=%s stopped gracefully.", "Download process PID=%s stopped gracefully.",
proc.pid, self.proc.pid,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"signal": signal_name, "signal": signal_name,
} }
}, },
@ -191,12 +187,12 @@ class ProcessManager:
return True return True
self.logger.warning( self.logger.warning(
"Download process PID=%s did not respond to %s; forcing termination.", "Download process PID=%s did not respond to %s; forcing termination.",
proc.pid, self.proc.pid,
signal_name, signal_name,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"signal": signal_name, "signal": signal_name,
"force": True, "force": True,
} }
@ -207,49 +203,51 @@ class ProcessManager:
self.logger.debug( self.logger.debug(
"Failed to send %s signal to download process PID=%s. %s", "Failed to send %s signal to download process PID=%s. %s",
signal_name, signal_name,
proc.pid, self.proc.pid,
e, e,
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid, "process_id": self.proc.pid,
"signal": signal_name, "signal": signal_name,
"exception_type": type(e).__name__, "exception_type": type(e).__name__,
} }
}, },
) )
if proc.is_alive(): if self.proc.is_alive():
self.logger.info( self.logger.info(
"Terminating download process PID=%s.", "Terminating download process PID=%s.",
proc.pid, self.proc.pid,
extra={"download": {"download_id": self.download_id, "process_id": proc.pid, "force": True}}, extra={"download": {"download_id": self.download_id, "process_id": self.proc.pid, "force": True}},
) )
proc.terminate() self.proc.terminate()
if not wait_for_process_with_timeout(proc, 1 if self.is_live else 2): if not wait_for_process_with_timeout(self.proc, 1 if self.is_live else 2):
self.logger.warning( self.logger.warning(
"Download process PID=%s did not terminate; killing forcefully.", "Download process PID=%s did not terminate; killing forcefully.",
proc.pid, self.proc.pid,
extra={"download": {"download_id": self.download_id, "process_id": proc.pid, "force": True}}, extra={
"download": {"download_id": self.download_id, "process_id": self.proc.pid, "force": True}
},
) )
proc.kill() self.proc.kill()
wait_for_process_with_timeout(proc, 1) wait_for_process_with_timeout(self.proc, 1)
self.logger.info( self.logger.info(
"Download process PID=%s stopped.", "Download process PID=%s stopped.",
proc.pid, self.proc.pid,
extra={"download": {"download_id": self.download_id, "process_id": proc.pid}}, extra={"download": {"download_id": self.download_id, "process_id": self.proc.pid}},
) )
return True return True
except Exception as e: except Exception as e:
self.logger.exception( self.logger.exception(
f"Failed to stop download process PID={proc.pid if proc else None}, ident={procId}.", f"Failed to stop download process PID={self.proc.pid}, ident={procId}.",
extra={ extra={
"download": { "download": {
"download_id": self.download_id, "download_id": self.download_id,
"process_id": proc.pid if proc else None, "process_id": self.proc.pid,
"process_ident": procId, "process_ident": procId,
"is_live": self.is_live, "is_live": self.is_live,
"exception_type": type(e).__name__, "exception_type": type(e).__name__,
@ -273,12 +271,11 @@ class ProcessManager:
return False return False
self.cancel_in_progress = True self.cancel_in_progress = True
proc = self.proc procId: int | None = self.proc.ident
procId: int | None = proc.ident if proc else None
if not procId: if not procId:
if proc: if self.proc:
proc.close() self.proc.close()
self.proc = None self.proc = None
self.logger.warning("Attempted to close download process, but it is not running.") self.logger.warning("Attempted to close download process, but it is not running.")
return False return False
@ -296,14 +293,13 @@ class ProcessManager:
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
current = self.proc if self.proc.is_alive():
if current is not None and current.is_alive():
self.logger.debug( self.logger.debug(
"Waiting for download process PID='%s' to close.", "Waiting for download process PID='%s' to close.",
procId, procId,
extra={"download": {"download_id": self.download_id, "process_ident": procId}}, extra={"download": {"download_id": self.download_id, "process_ident": procId}},
) )
await loop.run_in_executor(None, current.join) await loop.run_in_executor(None, self.proc.join)
self.logger.debug( self.logger.debug(
"Download process PID='%s' closed.", "Download process PID='%s' closed.",
procId, procId,

View file

@ -53,7 +53,7 @@ class DownloadQueue(metaclass=Singleton):
async def event_handler(_, __): async def event_handler(_, __):
await self.initialize() await self.initialize()
self._notify.subscribe(Events.STARTED, event_handler, f"{DownloadQueue.__name__}.initialize") self._notify.subscribe(Events.STARTED, event_handler, f"{__class__.__name__}.{__class__.initialize.__name__}")
Scheduler.get_instance().add( Scheduler.get_instance().add(
timer="* * * * *", timer="* * * * *",
@ -320,7 +320,7 @@ class DownloadQueue(metaclass=Singleton):
await self.done.put(item) await self.done.put(item)
self._notify.emit( self._notify.emit(
Events.ITEM_MOVED, Events.ITEM_MOVED,
data={"from": "queue", "to": "history", "preset": item.info.preset, "item": item.info}, data={"to": "history", "preset": item.info.preset, "item": item.info},
title="Download Cancelled", title="Download Cancelled",
message=f"Download '{item.info.title}' has been cancelled.", message=f"Download '{item.info.title}' has been cancelled.",
) )
@ -500,15 +500,16 @@ class DownloadQueue(metaclass=Singleton):
if not ids: if not ids:
return {"deleted": 0} return {"deleted": 0}
items: list[tuple[str, Download]] = await self.done.get_many_by_ids(ids) items = await self.done.get_many_by_ids(ids)
if not items: if not items:
return {"deleted": 0} return {"deleted": 0}
remove_file = False if self.config.remove_files is not True else remove_file if self.config.remove_files is not True:
remove_file = False
removed_files = 0 removed_files = 0
deleted_ids: list[str] = [] deleted_ids: list[str] = []
item_summaries: list[dict] = [] deleted_titles: list[str] = []
for item_id, item in items: for item_id, item in items:
filename: str = "" filename: str = ""
@ -523,8 +524,6 @@ class DownloadQueue(metaclass=Singleton):
"title": item.info.title, "title": item.info.title,
"status": item.info.status, "status": item.info.status,
"remove_file": remove_file, "remove_file": remove_file,
"preset": item.info.preset,
"path": str(p) if (p := item.info.get_file()) else None,
} }
}, },
) )
@ -556,9 +555,7 @@ class DownloadQueue(metaclass=Singleton):
"download_id": item_id, "download_id": item_id,
"item_id": item.info.id, "item_id": item.info.id,
"title": item.info.title, "title": item.info.title,
"preset": item.info.preset,
"filename": file_ref.name, "filename": file_ref.name,
"path": str(file_ref),
} }
}, },
) )
@ -574,8 +571,6 @@ class DownloadQueue(metaclass=Singleton):
"item_id": item.info.id, "item_id": item.info.id,
"title": item.info.title, "title": item.info.title,
"filename": rf.name, "filename": rf.name,
"preset": item.info.preset,
"path": str(rf),
} }
}, },
) )
@ -592,8 +587,6 @@ class DownloadQueue(metaclass=Singleton):
"item_id": item.info.id, "item_id": item.info.id,
"title": item.info.title, "title": item.info.title,
"filename": filename, "filename": filename,
"preset": item.info.preset,
"path": str(rf),
} }
}, },
) )
@ -608,7 +601,6 @@ class DownloadQueue(metaclass=Singleton):
"item_id": item.info.id, "item_id": item.info.id,
"title": item.info.title, "title": item.info.title,
"filename": filename, "filename": filename,
"preset": item.info.preset,
"remove_file": remove_file, "remove_file": remove_file,
"exception_type": type(e).__name__, "exception_type": type(e).__name__,
} }
@ -616,80 +608,63 @@ class DownloadQueue(metaclass=Singleton):
) )
deleted_ids.append(item_id) deleted_ids.append(item_id)
item_summaries.append( deleted_titles.append(item.info.title or item.info.id or item_id)
{
"id": item_id,
"title": item.info.title,
"url": item.info.url,
"preset": item.info.preset,
"path": str(p) if (p := item.info.get_file()) else None,
}
)
deleted_count: int = await self.done.bulk_delete(deleted_ids) deleted_count = await self.done.bulk_delete(deleted_ids)
if deleted_count < 1: if deleted_count < 1:
return {"deleted": 0} return {"deleted": 0}
message: list[str] = [f"Cleared {deleted_count} item{'s' if deleted_count != 1 else ''} from history."] title = "History Removed" if removed_files > 0 else "History Cleared"
message = f"Removed {deleted_count} item{'s' if deleted_count != 1 else ''} from history."
if removed_files > 0: if removed_files > 0:
message.append(f"Also removed {removed_files} local file{'s' if removed_files != 1 else ''}.") message += f" Also removed {removed_files} local file{'s' if removed_files != 1 else ''}."
self._notify.emit( self._notify.emit(
Events.ITEM_BULK_DELETED, Events.ITEM_BULK_DELETED,
data={"count": deleted_count, "removed_files": removed_files, "items": item_summaries}, data={"ids": deleted_ids, "count": deleted_count},
title="History Cleared", title=title,
message=" ".join(message), message=message,
) )
summary = ", ".join(deleted_titles[:5])
if deleted_count > 5:
summary += ", ..."
LOG.info( LOG.info(
"Cleared %d history item(s).", "Cleared %s history item(s), including %s.",
deleted_count, deleted_count,
extra={"deleted_count": deleted_count, "removed_files": removed_files, "items": item_summaries}, summary,
extra={"deleted_count": deleted_count, "removed_files": removed_files, "titles": deleted_titles[:5]},
) )
return {"deleted": deleted_count} return {"deleted": deleted_count}
async def clear_by_status(self, status_filter: str, remove_file: bool = False) -> dict[str, int | str]: async def clear_by_status(self, status_filter: str, remove_file: bool = False) -> dict[str, int | str]:
if not (items := await self.done.get_many_by_status(status_filter)): if self.config.remove_files is not True:
remove_file = False
if not remove_file:
deleted_count = await self.done.bulk_delete_by_status(status_filter)
if deleted_count < 1:
return {"deleted": 0} return {"deleted": 0}
remove_file = False if self.config.remove_files is not True else remove_file
if remove_file:
return await self.clear_bulk([item_id for item_id, _ in items], remove_file=remove_file)
deleted_count: int = await self.done.bulk_delete_by_status(status_filter)
item_summaries: list[dict[str, str | None]] = [
{
"id": item_id,
"title": dto.info.title,
"url": dto.info.url,
"preset": dto.info.preset,
"path": str(p) if (p := dto.info.get_file()) else None,
}
for item_id, dto in items
]
self._notify.emit( self._notify.emit(
Events.ITEM_BULK_DELETED, Events.ITEM_BULK_DELETED,
data={"count": deleted_count, "status": status_filter, "items": item_summaries}, data={"count": deleted_count, "status": status_filter},
title="History Cleared", title="History Cleared",
message=f"Cleared {deleted_count} item(s) from history.", message=f"Cleared {deleted_count} item{'s' if deleted_count != 1 else ''} from history.",
) )
LOG.info( LOG.info(
"Cleared %d history item(s) with status '%s'.", "Cleared %s history item(s) with status '%s'.",
deleted_count, deleted_count,
status_filter, status_filter,
extra={ extra={"deleted_count": deleted_count, "status_filter": status_filter},
"deleted_count": deleted_count,
"status_filter": status_filter,
"items": item_summaries,
},
) )
return {"deleted": deleted_count} return {"deleted": deleted_count}
async def get(self, mode: str = "all") -> dict[str, dict[str, ItemDTO]]: items = await self.done.get_many_by_status(status_filter)
return await self.clear_bulk([item_id for item_id, _ in items], remove_file=remove_file)
async def get(self, mode: str = "all") -> dict[str, list[dict[str, ItemDTO]]]:
""" """
Get the download queue and the download history. Get the download queue and the download history.
@ -725,32 +700,6 @@ class DownloadQueue(metaclass=Singleton):
return items return items
def live_queue(self, limit: int = 0) -> dict[str, int | dict[str, ItemDTO]]:
"""Return a display-limited live queue snapshot with total metadata."""
limit = max(0, int(limit or 0))
items: dict[str, ItemDTO] = {}
active = self.pool.get_active_downloads()
for key, download in active.items():
if key in self.queue:
items[key] = download.info
for key, download in self.queue.items():
if limit > 0 and len(items) >= limit:
break
if key in items:
continue
items[key] = active[key].info if key in active else download.info
return {
"queue": items,
"queue_count": len(self.queue),
"queue_loaded": len(items),
"queue_limit": limit,
}
async def get_item(self, **kwargs) -> tuple["StoreType", Download] | tuple[None, None]: async def get_item(self, **kwargs) -> tuple["StoreType", Download] | tuple[None, None]:
""" """
Get a specific item from the download queue or history. Get a specific item from the download queue or history.
@ -759,7 +708,7 @@ class DownloadQueue(metaclass=Singleton):
**kwargs: The key-value pair to search for. Supported keys are 'id', 'url'. **kwargs: The key-value pair to search for. Supported keys are 'id', 'url'.
Returns: Returns:
tuple[StoreType, Download] | tuple[None, None]: The requested item if found, otherwise None. (StoreType, Download) | None: The requested item if found, otherwise None.
""" """
from app.library.DataStore import StoreType from app.library.DataStore import StoreType

View file

@ -55,37 +55,7 @@ class StatusTracker:
self.final_update = False self.final_update = False
self._terminator_sent: bool = False self._terminator_sent: bool = False
self._candidate_filepath: Path | None = None self._candidate_filepath: Path | None = None
self.update_task: asyncio.Future[Any] | None = None self.update_task: asyncio.Task | None = None
self._last_progress_time: float = 0.0
self._pending_progress: bool = False
self._progress_interval: float = 0.5
def _progress_payload(self) -> dict:
return {
"_id": self.info._id,
"status": self.info.status,
"percent": self.info.percent,
"speed": self.info.speed,
"eta": self.info.eta,
"downloaded_bytes": self.info.downloaded_bytes,
"total_bytes": self.info.total_bytes,
"msg": self.info.msg,
}
def _emit_progress(self) -> None:
now = time.monotonic()
if (now - self._last_progress_time) >= self._progress_interval:
self._notify.emit(Events.ITEM_PROGRESS, data=self._progress_payload())
self._last_progress_time = now
self._pending_progress = False
else:
self._pending_progress = True
def _flush_progress(self) -> None:
if self._pending_progress:
self._notify.emit(Events.ITEM_PROGRESS, data=self._progress_payload())
self._pending_progress = False
self._last_progress_time = time.monotonic()
async def _finalize_file(self, filepath: Path) -> None: async def _finalize_file(self, filepath: Path) -> None:
""" """
@ -214,7 +184,6 @@ class StatusTracker:
self.tmpfilename = status.get("tmpfilename") self.tmpfilename = status.get("tmpfilename")
old_status = self.info.status
self.info.status = status.get("status", self.info.status) self.info.status = status.get("status", self.info.status)
if "download_skipped" in status: if "download_skipped" in status:
self.info.download_skipped = bool(status.get("download_skipped")) self.info.download_skipped = bool(status.get("download_skipped"))
@ -256,7 +225,7 @@ class StatusTracker:
self.info.percent = status["downloaded_bytes"] / total * 100 self.info.percent = status["downloaded_bytes"] / total * 100
except ZeroDivisionError: except ZeroDivisionError:
self.info.percent = 0 self.info.percent = 0
self.info.total_bytes = int(total) self.info.total_bytes = total
self.info.speed = status.get("speed") self.info.speed = status.get("speed")
self.info.eta = status.get("eta") self.info.eta = status.get("eta")
@ -266,11 +235,7 @@ class StatusTracker:
await self._finalize_file(Path(final_name)) await self._finalize_file(Path(final_name))
self.info.status = "finished" self.info.status = "finished"
if self.info.status != old_status or self.final_update:
self._flush_progress()
self._notify.emit(Events.ITEM_UPDATED, data=self.info) self._notify.emit(Events.ITEM_UPDATED, data=self.info)
else:
self._emit_progress()
async def progress_update(self) -> None: async def progress_update(self) -> None:
""" """
@ -280,15 +245,12 @@ class StatusTracker:
""" """
while True: while True:
try: try:
update_task = asyncio.get_running_loop().run_in_executor(None, self.status_queue.get) self.update_task = asyncio.get_running_loop().run_in_executor(None, self.status_queue.get)
self.update_task = asyncio.ensure_future(update_task) status = await self.update_task
status = await update_task
if status is None or isinstance(status, Terminator): if status is None or isinstance(status, Terminator):
self._flush_progress()
return return
await self.process_status_update(status) await self.process_status_update(status)
except (asyncio.CancelledError, OSError, FileNotFoundError, EOFError, BrokenPipeError, ConnectionError): except (asyncio.CancelledError, OSError, FileNotFoundError, EOFError, BrokenPipeError, ConnectionError):
self._flush_progress()
return return
async def drain_queue(self, max_iterations: int = 50) -> None: async def drain_queue(self, max_iterations: int = 50) -> None:
@ -333,8 +295,6 @@ class StatusTracker:
except (queue.Empty, BrokenPipeError, ConnectionRefusedError, EOFError, OSError): except (queue.Empty, BrokenPipeError, ConnectionRefusedError, EOFError, OSError):
continue continue
self._flush_progress()
def cancel_update_task(self) -> None: def cancel_update_task(self) -> None:
"""Cancel the progress update task if it's running.""" """Cancel the progress update task if it's running."""
try: try:

View file

@ -2,7 +2,6 @@ from __future__ import annotations
import asyncio import asyncio
import functools import functools
import importlib.util
import threading import threading
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any, Literal, cast, overload from typing import Any, Literal, cast, overload
@ -99,7 +98,12 @@ def _normalize_proxy(proxy: str | dict[str, str] | None) -> str | None:
@functools.lru_cache(maxsize=1) @functools.lru_cache(maxsize=1)
def _curl_available() -> bool: def _curl_available() -> bool:
return importlib.util.find_spec("httpx_curl_cffi") is not None try:
import httpx_curl_cffi # noqa: F401
return True
except Exception:
return False
def resolve_curl_transport(use_curl: bool = True) -> bool: def resolve_curl_transport(use_curl: bool = True) -> bool:
@ -117,7 +121,7 @@ def _build_async_curl_transport(
from httpx_curl_cffi import AsyncCurlTransport from httpx_curl_cffi import AsyncCurlTransport
return AsyncCurlTransport( return AsyncCurlTransport(
impersonate=cast("Any", curl_impersonate), impersonate=curl_impersonate,
default_headers=curl_default_headers, default_headers=curl_default_headers,
) )
@ -304,7 +308,10 @@ def get_async_client(
curl_default_headers=curl_default_headers, curl_default_headers=curl_default_headers,
) )
client = httpx.AsyncClient( client = httpx.AsyncClient(
transport=_get_transport(enable_cf, is_async=True, transport=transport), transport=cast(
"httpx.AsyncBaseTransport",
_get_transport(enable_cf, is_async=True, transport=transport),
),
proxy=cast("Any", proxy), proxy=cast("Any", proxy),
) )
Globals.SHARED_ASYNC_CLIENTS[key] = client Globals.SHARED_ASYNC_CLIENTS[key] = client
@ -323,7 +330,10 @@ def get_sync_client(
return Globals.SHARED_SYNC_CLIENTS[key] return Globals.SHARED_SYNC_CLIENTS[key]
client = httpx.Client( client = httpx.Client(
transport=_get_transport(enable_cf, is_async=False, transport=None), transport=cast(
"httpx.BaseTransport",
_get_transport(enable_cf, is_async=False, transport=None),
),
proxy=cast("Any", proxy), proxy=cast("Any", proxy),
) )
Globals.SHARED_SYNC_CLIENTS[key] = client Globals.SHARED_SYNC_CLIENTS[key] = client

View file

@ -1,3 +1,4 @@
# flake8: noqa: S608
""" """
This module is a fork of the Caribou library This module is a fork of the Caribou library
(https://pypi.org/project/caribou/) modified to work for our specific use case. (https://pypi.org/project/caribou/) modified to work for our specific use case.
@ -8,11 +9,10 @@ import datetime
import glob import glob
import os.path import os.path
import traceback import traceback
from collections.abc import AsyncIterator, Iterable, Mapping, Sequence from collections.abc import AsyncIterator, Iterable, Sequence
from importlib.machinery import ModuleSpec, SourceFileLoader from importlib.machinery import ModuleSpec, SourceFileLoader
from importlib.util import module_from_spec, spec_from_loader from importlib.util import module_from_spec, spec_from_loader
from pathlib import Path from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Any
from sqlalchemy import text from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncConnection from sqlalchemy.ext.asyncio import AsyncConnection
@ -44,9 +44,7 @@ class InvalidNameError(Error):
@contextlib.asynccontextmanager @contextlib.asynccontextmanager
async def execute( async def execute(conn: AsyncConnection, sql: str, params: Sequence[object] | None = None) -> AsyncIterator:
conn: AsyncConnection, sql: str, params: Mapping[str, Any] | Sequence[Mapping[str, Any]] | None = None
) -> AsyncIterator:
params = {} if params is None else params params = {} if params is None else params
result = await conn.execute(text(sql), params) result = await conn.execute(text(sql), params)
try: try:
@ -128,12 +126,12 @@ class Database:
self.version_table: str = version_table self.version_table: str = version_table
self._owns_connection = isinstance(db_url, str) self._owns_connection = bool(isinstance(db_url, str))
if isinstance(db_url, str): if self._owns_connection:
self.conn: AsyncConnection | None = None self.conn: AsyncConnection | None = None
self.db_url: str = db_url self.db_url: str = db_url
else: else:
self.conn = db_url self.conn: AsyncConnection = db_url
async def __aenter__(self): async def __aenter__(self):
if self._owns_connection and self.conn is None: if self._owns_connection and self.conn is None:
@ -185,8 +183,7 @@ class Database:
async def downgrade(self, migrations: list[Migration], target_version: str | int) -> None: async def downgrade(self, migrations: list[Migration], target_version: str | int) -> None:
assert self.conn assert self.conn
if target_version not in (0, "0"): if target_version not in (0, "0"):
_assert_migration_exists(migrations, str(target_version)) _assert_migration_exists(migrations, target_version)
target = str(target_version)
migrations.sort(key=lambda x: x.get_version(), reverse=True) migrations.sort(key=lambda x: x.get_version(), reverse=True)
database_version: str | None = await self.get_version() database_version: str | None = await self.get_version()
@ -195,7 +192,7 @@ class Database:
current_version: str = migration.get_version() current_version: str = migration.get_version()
if database_version is not None and current_version > database_version: if database_version is not None and current_version > database_version:
continue continue
if current_version <= target: if current_version <= target_version:
break break
await migration.downgrade(self.conn) await migration.downgrade(self.conn)
next_version: str | int = 0 next_version: str | int = 0
@ -213,14 +210,14 @@ class Database:
assert self.conn assert self.conn
if not await self.is_version_controlled(): if not await self.is_version_controlled():
return None return None
sql: str = f"SELECT version FROM {self.version_table}" # noqa: S608 sql: str = f"SELECT version FROM {self.version_table}"
async with execute(self.conn, sql) as result: async with execute(self.conn, sql) as result:
rows = result.fetchall() rows = result.fetchall()
return rows[0][0] if rows else "0" return rows[0][0] if rows else "0"
async def update_version(self, version: str) -> None: async def update_version(self, version: str) -> None:
assert self.conn assert self.conn
sql: str = f"UPDATE {self.version_table} SET version = :version" # noqa: S608 sql: str = f"UPDATE {self.version_table} SET version = :version"
async with transaction(self.conn): async with transaction(self.conn):
await self.conn.execute(text(sql), {"version": version}) await self.conn.execute(text(sql), {"version": version})
@ -229,7 +226,7 @@ class Database:
sql: str = f"""CREATE TABLE IF NOT EXISTS {self.version_table} ( version TEXT ) """ sql: str = f"""CREATE TABLE IF NOT EXISTS {self.version_table} ( version TEXT ) """
async with transaction(self.conn): async with transaction(self.conn):
await self.conn.execute(text(sql)) await self.conn.execute(text(sql))
await self.conn.execute(text(f"INSERT INTO {self.version_table} VALUES (:version)"), {"version": "0"}) # noqa: S608 await self.conn.execute(text(f"INSERT INTO {self.version_table} VALUES (:version)"), {"version": "0"})
def __repr__(self) -> str: def __repr__(self) -> str:
return f'Database("{self.db_url if self._owns_connection else "external_connection"}")' return f'Database("{self.db_url if self._owns_connection else "external_connection"}")'
@ -241,7 +238,7 @@ def _assert_migration_exists(migrations: Iterable[Migration], version: str | int
raise Error(msg) raise Error(msg)
def load_migrations(directory: str | Path) -> list[Migration]: def load_migrations(directory: str) -> list[Migration]:
"""Return the migrations contained in the given directory.""" """Return the migrations contained in the given directory."""
directory = str(directory) directory = str(directory)
if not os.path.exists(directory) or not os.path.isdir(directory): if not os.path.exists(directory) or not os.path.isdir(directory):
@ -250,7 +247,7 @@ def load_migrations(directory: str | Path) -> list[Migration]:
return [Migration(f) for f in glob.glob(os.path.join(directory, "*.py"))] return [Migration(f) for f in glob.glob(os.path.join(directory, "*.py"))]
async def upgrade(db_url: AsyncConnection | str, migration_dir: str | Path, version: str | None = None) -> None: async def upgrade(db_url: AsyncConnection | str, migration_dir: str, version: str | None = None) -> None:
""" """
Upgrade the given database with the migrations contained in the Upgrade the given database with the migrations contained in the
migrations directory. If a version is not specified, upgrade migrations directory. If a version is not specified, upgrade
@ -263,7 +260,7 @@ async def upgrade(db_url: AsyncConnection | str, migration_dir: str | Path, vers
await db.upgrade(load_migrations(migration_dir), version) await db.upgrade(load_migrations(migration_dir), version)
async def downgrade(db_url: str | AsyncConnection, migration_dir: str | Path, version: str) -> None: async def downgrade(db_url: str | AsyncConnection, migration_dir: str, version: str) -> None:
""" """
Downgrade the database to the given version with the migrations Downgrade the database to the given version with the migrations
contained in the given migration directory. contained in the given migration directory.

View file

@ -2,16 +2,14 @@ import asyncio
import contextlib import contextlib
import json import json
import os import os
from collections.abc import Iterable, Mapping from collections.abc import Iterable
from dataclasses import fields from dataclasses import fields
from datetime import UTC, datetime from datetime import UTC, datetime
from email.utils import formatdate from email.utils import formatdate
from typing import Any
from urllib.parse import quote_plus from urllib.parse import quote_plus
from aiohttp import web from aiohttp import web
from sqlalchemy import text from sqlalchemy import text
from sqlalchemy.engine.row import RowMapping
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.ext.asyncio.engine import AsyncConnection from sqlalchemy.ext.asyncio.engine import AsyncConnection
@ -38,13 +36,13 @@ def _memory_db_url(db_path: str) -> str:
return f"sqlite+aiosqlite:///file:{quoted_name}?mode=memory&cache=shared&uri=true" return f"sqlite+aiosqlite:///file:{quoted_name}?mode=memory&cache=shared&uri=true"
def _decode_row(row: Mapping[str, Any] | RowMapping) -> ItemDTO: def _decode_row(row: dict) -> ItemDTO:
row_date = datetime.strptime(row["created_at"], "%Y-%m-%d %H:%M:%S").replace(tzinfo=UTC) row_date = datetime.strptime(row["created_at"], "%Y-%m-%d %H:%M:%S") # noqa: DTZ007
data = json.loads(row["data"]) data = json.loads(row["data"])
data.pop("_id", None) data.pop("_id", None)
item = init_class(ItemDTO, data, ITEM_DTO_FIELDS) item = init_class(ItemDTO, data, ITEM_DTO_FIELDS)
item._id = row["id"] item._id = row["id"]
item.datetime = formatdate(row_date.timestamp()) item.datetime = formatdate(row_date.replace(tzinfo=UTC).timestamp())
return item return item
@ -151,8 +149,8 @@ class SqliteStore(metaclass=ThreadSafe):
return self._sessionmaker return self._sessionmaker
async def fetch_saved(self, type_value: str) -> list[tuple[str, ItemDTO]]: async def fetch_saved(self, type_value: str) -> list[tuple[str, ItemDTO]]:
conn = await self.get_connection() await self.get_connection()
result = await conn.execute( result = await self._conn.execute(
text( text(
'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value ORDER BY "created_at" ASC' 'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value ORDER BY "created_at" ASC'
), ),
@ -165,12 +163,12 @@ class SqliteStore(metaclass=ThreadSafe):
async def exists(self, type_value: str, key: str | None = None, url: str | None = None) -> bool: async def exists(self, type_value: str, key: str | None = None, url: str | None = None) -> bool:
return await self.get(type_value, key=key, url=url) is not None return await self.get(type_value, key=key, url=url) is not None
async def get(self, type_value: str, key: str | None = None, url: str | None = None) -> ItemDTO | None: async def get(self, type_value: str, key: str | None = None, url: str | None = None) -> bool:
if not key and not url: if not key and not url:
msg = "key or url must be provided." msg = "key or url must be provided."
raise KeyError(msg) raise KeyError(msg)
conn = await self.get_connection() await self.get_connection()
clauses: list[str] = [] clauses: list[str] = []
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
@ -188,7 +186,7 @@ class SqliteStore(metaclass=ThreadSafe):
f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND ({where_clause}) LIMIT 1' # noqa: S608 f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND ({where_clause}) LIMIT 1' # noqa: S608
) )
result = await conn.execute(text(query), params) result = await self._conn.execute(text(query), params)
row = result.mappings().first() row = result.mappings().first()
if not row: if not row:
@ -197,8 +195,8 @@ class SqliteStore(metaclass=ThreadSafe):
return _decode_row(row) return _decode_row(row)
async def get_by_id(self, type_value: str, id: str) -> ItemDTO | None: async def get_by_id(self, type_value: str, id: str) -> ItemDTO | None:
conn = await self.get_connection() await self.get_connection()
result = await conn.execute( result = await self._conn.execute(
text('SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND "id" = :id'), text('SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND "id" = :id'),
{"type_value": type_value, "id": id}, {"type_value": type_value, "id": id},
) )
@ -214,24 +212,24 @@ class SqliteStore(metaclass=ThreadSafe):
if not ids_list: if not ids_list:
return [] return []
conn = await self.get_connection() await self.get_connection()
placeholders = ", ".join(f":id_{index}" for index in range(len(ids_list))) placeholders = ", ".join(f":id_{index}" for index in range(len(ids_list)))
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
params.update({f"id_{index}": item_id for index, item_id in enumerate(ids_list)}) params.update({f"id_{index}": item_id for index, item_id in enumerate(ids_list)})
result = await conn.execute( result = await self._conn.execute(
text( text(
f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})' # noqa: S608 f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})' # noqa: S608
), ),
params, params,
) )
rows = list(result.mappings().all()) rows = result.mappings().all()
order = {item_id: index for index, item_id in enumerate(ids_list)} order = {item_id: index for index, item_id in enumerate(ids_list)}
rows.sort(key=lambda row: order.get(row["id"], len(ids_list))) rows.sort(key=lambda row: order.get(row["id"], len(ids_list)))
return [(row["id"], _decode_row(row)) for row in rows] return [(row["id"], _decode_row(row)) for row in rows]
async def get_many_by_status(self, type_value: str, status_filter: str) -> list[tuple[str, ItemDTO]]: async def get_many_by_status(self, type_value: str, status_filter: str) -> list[tuple[str, ItemDTO]]:
conn = await self.get_connection() await self.get_connection()
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
where_clauses = ['"type" = :type_value'] where_clauses = ['"type" = :type_value']
@ -241,7 +239,7 @@ class SqliteStore(metaclass=ThreadSafe):
params.update(status_params) params.update(status_params)
query = f'SELECT "id", "data", "created_at" FROM "history" WHERE {" AND ".join(where_clauses)} ORDER BY "created_at" DESC' # noqa: S608 query = f'SELECT "id", "data", "created_at" FROM "history" WHERE {" AND ".join(where_clauses)} ORDER BY "created_at" DESC' # noqa: S608
result = await conn.execute(text(query), params) result = await self._conn.execute(text(query), params)
rows = result.mappings().all() rows = result.mappings().all()
return [(row["id"], _decode_row(row)) for row in rows] return [(row["id"], _decode_row(row)) for row in rows]
@ -256,7 +254,7 @@ class SqliteStore(metaclass=ThreadSafe):
if not kwargs: if not kwargs:
return None return None
conn = await self.get_connection() await self.get_connection()
clauses: list[str] = [] clauses: list[str] = []
params: dict[str, str | float | int] = {"type_value": type_value} params: dict[str, str | float | int] = {"type_value": type_value}
@ -323,7 +321,7 @@ class SqliteStore(metaclass=ThreadSafe):
where_clause: str = " OR ".join(f"({clause})" for clause in clauses) where_clause: str = " OR ".join(f"({clause})" for clause in clauses)
query: str = f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND ({where_clause}) ORDER BY "created_at" ASC LIMIT 1' # noqa: S608 query: str = f'SELECT "id", "data", "created_at" FROM "history" WHERE "type" = :type_value AND ({where_clause}) ORDER BY "created_at" ASC LIMIT 1' # noqa: S608
result = await conn.execute(text(query), params) result = await self._conn.execute(text(query), params)
row = result.mappings().first() row = result.mappings().first()
if not row: if not row:
@ -334,7 +332,7 @@ class SqliteStore(metaclass=ThreadSafe):
return item if any(matches_condition(k, v, item.__dict__) for k, v in kwargs.items()) else None return item if any(matches_condition(k, v, item.__dict__) for k, v in kwargs.items()) else None
async def count(self, type_value: str, status_filter: str | None = None) -> int: async def count(self, type_value: str, status_filter: str | None = None) -> int:
conn = await self.get_connection() await self.get_connection()
where_clauses: list[str] = ['"type" = :type_value'] where_clauses: list[str] = ['"type" = :type_value']
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
@ -346,7 +344,7 @@ class SqliteStore(metaclass=ThreadSafe):
where_clause: str = " AND ".join(where_clauses) where_clause: str = " AND ".join(where_clauses)
count_query: str = f'SELECT COUNT(*) as count FROM "history" WHERE {where_clause}' # noqa: S608 count_query: str = f'SELECT COUNT(*) as count FROM "history" WHERE {where_clause}' # noqa: S608
result = await conn.execute(text(count_query), params) result = await self._conn.execute(text(count_query), params)
row = result.mappings().first() row = result.mappings().first()
return row["count"] if row else 0 return row["count"] if row else 0
@ -359,7 +357,7 @@ class SqliteStore(metaclass=ThreadSafe):
order: str, order: str,
status_filter: str | None = None, status_filter: str | None = None,
) -> tuple[list[tuple[str, ItemDTO]], int, int, int]: ) -> tuple[list[tuple[str, ItemDTO]], int, int, int]:
conn = await self.get_connection() await self.get_connection()
where_clauses: list[str] = ['"type" = :type_value'] where_clauses: list[str] = ['"type" = :type_value']
params: dict[str, str | int] = {"type_value": type_value} params: dict[str, str | int] = {"type_value": type_value}
@ -371,7 +369,7 @@ class SqliteStore(metaclass=ThreadSafe):
where_clause: str = " AND ".join(where_clauses) where_clause: str = " AND ".join(where_clauses)
count_query: str = f'SELECT COUNT(*) as count FROM "history" WHERE {where_clause}' # noqa: S608 count_query: str = f'SELECT COUNT(*) as count FROM "history" WHERE {where_clause}' # noqa: S608
result = await conn.execute(text(count_query), params) result = await self._conn.execute(text(count_query), params)
count_row = result.mappings().first() count_row = result.mappings().first()
total_items = count_row["count"] if count_row else 0 total_items = count_row["count"] if count_row else 0
@ -386,7 +384,7 @@ class SqliteStore(metaclass=ThreadSafe):
query: str = f'SELECT "id", "data", "created_at" FROM "history" WHERE {where_clause} ORDER BY "created_at" {order} LIMIT :limit OFFSET :offset' # noqa: S608 query: str = f'SELECT "id", "data", "created_at" FROM "history" WHERE {where_clause} ORDER BY "created_at" {order} LIMIT :limit OFFSET :offset' # noqa: S608
result = await conn.execute(text(query), params) result = await self._conn.execute(text(query), params)
rows = result.mappings().all() rows = result.mappings().all()
items = [(row["id"], _decode_row(row)) for row in rows] items = [(row["id"], _decode_row(row)) for row in rows]
@ -398,15 +396,15 @@ class SqliteStore(metaclass=ThreadSafe):
await self._upsert_now(type_value, item) await self._upsert_now(type_value, item)
async def delete(self, type_value: str, key: str) -> None: async def delete(self, type_value: str, key: str) -> None:
conn = await self.get_connection() await self.get_connection()
await conn.execute( await self._conn.execute(
text('DELETE FROM "history" WHERE "type" = :type_value AND "id" = :key'), text('DELETE FROM "history" WHERE "type" = :type_value AND "id" = :key'),
{"type_value": type_value, "key": key}, {"type_value": type_value, "key": key},
) )
await conn.commit() await self._conn.commit()
async def bulk_delete(self, type_value: str, keys: Iterable[str]) -> int: async def bulk_delete(self, type_value: str, keys: Iterable[str]) -> int:
conn = await self.get_connection() await self.get_connection()
keys_list: list[str] = list(keys) keys_list: list[str] = list(keys)
if not keys_list: if not keys_list:
return 0 return 0
@ -414,15 +412,15 @@ class SqliteStore(metaclass=ThreadSafe):
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
params.update({f"key_{i}": key for i, key in enumerate(keys_list)}) params.update({f"key_{i}": key for i, key in enumerate(keys_list)})
result = await conn.execute( result = await self._conn.execute(
text(f'DELETE FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})'), # noqa: S608 text(f'DELETE FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})'), # noqa: S608
params, params,
) )
await conn.commit() await self._conn.commit()
return result.rowcount if result else 0 return result.rowcount if result else 0
async def bulk_delete_by_status(self, type_value: str, status_filter: str) -> int: async def bulk_delete_by_status(self, type_value: str, status_filter: str) -> int:
conn = await self.get_connection() await self.get_connection()
params: dict[str, str] = {"type_value": type_value} params: dict[str, str] = {"type_value": type_value}
where_clauses = ['"type" = :type_value'] where_clauses = ['"type" = :type_value']
@ -431,11 +429,11 @@ class SqliteStore(metaclass=ThreadSafe):
where_clauses.append(status_clause) where_clauses.append(status_clause)
params.update(status_params) params.update(status_params)
result = await conn.execute( result = await self._conn.execute(
text(f'DELETE FROM "history" WHERE {" AND ".join(where_clauses)}'), # noqa: S608 text(f'DELETE FROM "history" WHERE {" AND ".join(where_clauses)}'), # noqa: S608
params, params,
) )
await conn.commit() await self._conn.commit()
return result.rowcount if result else 0 return result.rowcount if result else 0
async def enqueue_upsert(self, type_value: str, item: ItemDTO) -> None: async def enqueue_upsert(self, type_value: str, item: ItemDTO) -> None:
@ -486,9 +484,6 @@ class SqliteStore(metaclass=ThreadSafe):
async def _enqueue(self, op: _Op) -> None: async def _enqueue(self, op: _Op) -> None:
self._ensure_worker() self._ensure_worker()
if self._queue is None:
msg = "Queue was not initialized by _ensure_worker."
raise RuntimeError(msg)
await self._queue.put(op) await self._queue.put(op)
def _ensure_worker(self) -> None: def _ensure_worker(self) -> None:
@ -497,14 +492,10 @@ class SqliteStore(metaclass=ThreadSafe):
self._task = asyncio.create_task(self._writer(), name="sqlite-store-writer") self._task = asyncio.create_task(self._writer(), name="sqlite-store-writer")
async def _writer(self) -> None: async def _writer(self) -> None:
queue = self._queue
if queue is None:
msg = "Writer started before queue was initialized."
raise RuntimeError(msg)
while True: while True:
op = await queue.get() op = await self._queue.get()
if isinstance(op.op, Terminator): if isinstance(op.op, Terminator):
queue.task_done() self._queue.task_done()
break break
try: try:
async with self._lock: async with self._lock:
@ -516,34 +507,34 @@ class SqliteStore(metaclass=ThreadSafe):
extra={"operation": op.op, "type_value": op.type_value, "exception_type": type(ex).__name__}, extra={"operation": op.op, "type_value": op.type_value, "exception_type": type(ex).__name__},
) )
finally: finally:
queue.task_done() self._queue.task_done()
await asyncio.sleep(self._flush_interval) await asyncio.sleep(self._flush_interval)
async def _apply(self, op: _Op) -> None: async def _apply(self, op: _Op) -> None:
conn = await self.get_connection() await self.get_connection()
if op.op == "upsert" and op.item: if op.op == "upsert" and op.item:
await self._upsert_now_conn(conn, op.type_value, op.item) await self._upsert_now_conn(self._conn, op.type_value, op.item)
await conn.commit() await self._conn.commit()
elif op.op == "delete" and op.key: elif op.op == "delete" and op.key:
await conn.execute( await self._conn.execute(
text('DELETE FROM "history" WHERE "type" = :type_value AND "id" = :key'), text('DELETE FROM "history" WHERE "type" = :type_value AND "id" = :key'),
{"type_value": op.type_value, "key": op.key}, {"type_value": op.type_value, "key": op.key},
) )
await conn.commit() await self._conn.commit()
elif op.op == "bulk_delete" and op.keys: elif op.op == "bulk_delete" and op.keys:
placeholders = ",".join(f":key_{i}" for i in range(len(op.keys))) placeholders = ",".join(f":key_{i}" for i in range(len(op.keys)))
params: dict[str, str] = {"type_value": op.type_value} params: dict[str, str] = {"type_value": op.type_value}
params.update({f"key_{i}": key for i, key in enumerate(op.keys)}) params.update({f"key_{i}": key for i, key in enumerate(op.keys)})
await conn.execute( await self._conn.execute(
text(f'DELETE FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})'), # noqa: S608 text(f'DELETE FROM "history" WHERE "type" = :type_value AND "id" IN ({placeholders})'), # noqa: S608
params, params,
) )
await conn.commit() await self._conn.commit()
async def _upsert_now(self, type_value: str, item: ItemDTO) -> None: async def _upsert_now(self, type_value: str, item: ItemDTO) -> None:
conn = await self.get_connection() await self.get_connection()
await self._upsert_now_conn(conn, type_value, item) await self._upsert_now_conn(self._conn, type_value, item)
await conn.commit() await self._conn.commit()
async def _upsert_now_conn(self, conn, type_value: str, item: ItemDTO) -> None: async def _upsert_now_conn(self, conn, type_value: str, item: ItemDTO) -> None:
"""Helper to upsert using an existing connection.""" """Helper to upsert using an existing connection."""
@ -569,7 +560,7 @@ class SqliteStore(metaclass=ThreadSafe):
async def execute_raw(self, query: str, params: dict | tuple | None = None) -> None: async def execute_raw(self, query: str, params: dict | tuple | None = None) -> None:
"""Execute raw SQL query (for testing purposes).""" """Execute raw SQL query (for testing purposes)."""
conn = await self.get_connection() await self.get_connection()
if isinstance(params, tuple): if isinstance(params, tuple):
# Convert positional params to dict for SQLAlchemy # Convert positional params to dict for SQLAlchemy
# Assuming queries use ? placeholders, we need to count them # Assuming queries use ? placeholders, we need to count them
@ -584,16 +575,16 @@ class SqliteStore(metaclass=ThreadSafe):
# Replace ? with :p0, :p1, etc. # Replace ? with :p0, :p1, etc.
for i in range(len(params)): for i in range(len(params)):
query = query.replace("?", f":p{i}", 1) query = query.replace("?", f":p{i}", 1)
await conn.execute(text(query), param_dict) await self._conn.execute(text(query), param_dict)
elif isinstance(params, dict): elif isinstance(params, dict):
await conn.execute(text(query), params) await self._conn.execute(text(query), params)
else: else:
await conn.execute(text(query)) await self._conn.execute(text(query))
await conn.commit() await self._conn.commit()
async def fetch_raw(self, query: str, params: dict | tuple | None = None): async def fetch_raw(self, query: str, params: dict | tuple | None = None):
"""Fetch results from raw SQL query (for testing purposes).""" """Fetch results from raw SQL query (for testing purposes)."""
conn = await self.get_connection() await self.get_connection()
if isinstance(params, tuple): if isinstance(params, tuple):
# Convert positional params to dict for SQLAlchemy # Convert positional params to dict for SQLAlchemy
placeholders = query.count("?") placeholders = query.count("?")
@ -605,11 +596,11 @@ class SqliteStore(metaclass=ThreadSafe):
param_dict = {f"p{i}": params[i] for i in range(len(params))} param_dict = {f"p{i}": params[i] for i in range(len(params))}
for i in range(len(params)): for i in range(len(params)):
query = query.replace("?", f":p{i}", 1) query = query.replace("?", f":p{i}", 1)
result = await conn.execute(text(query), param_dict) result = await self._conn.execute(text(query), param_dict)
elif isinstance(params, dict): elif isinstance(params, dict):
result = await conn.execute(text(query), params) result = await self._conn.execute(text(query), params)
else: else:
result = await conn.execute(text(query)) result = await self._conn.execute(text(query))
return result.mappings().all() return result.mappings().all()
async def get_connection(self) -> AsyncConnection: async def get_connection(self) -> AsyncConnection:

View file

@ -1,14 +1,14 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# flake8: noqa: S310 T201
import argparse import argparse
import http.client
import os import os
import pathlib import pathlib
import socket import socket
import sys import sys
import time import time
import urllib.request
import webbrowser import webbrowser
from urllib.parse import urlsplit
if __name__ == "__main__": if __name__ == "__main__":
from multiprocessing import freeze_support from multiprocessing import freeze_support
@ -26,7 +26,7 @@ APP_ROOT = str((pathlib.Path(__file__).parent / "..").resolve())
if APP_ROOT not in sys.path: if APP_ROOT not in sys.path:
sys.path.insert(0, APP_ROOT) sys.path.insert(0, APP_ROOT)
import platformdirs import platformdirs # type: ignore
def set_env(): def set_env():
@ -55,27 +55,18 @@ def open_browser_when_ready(url: str, timeout: float = 5.0) -> None:
import threading import threading
def wait_then_open(): def wait_then_open():
parsed = urlsplit(url)
path = parsed.path or "/"
deadline = time.time() + timeout deadline = time.time() + timeout
while time.time() < deadline: while time.time() < deadline:
try: try:
conn = http.client.HTTPConnection(parsed.hostname or "127.0.0.1", parsed.port or 80, timeout=1) with urllib.request.urlopen(url, timeout=1) as response:
try: if 200 == response.status:
conn.request("GET", path)
response = conn.getresponse()
if response.status == 200:
os.environ.pop("LD_LIBRARY_PATH", None) os.environ.pop("LD_LIBRARY_PATH", None)
webbrowser.open_new(url) webbrowser.open_new(url)
return return
finally:
conn.close()
except Exception: except Exception:
time.sleep(0.5) time.sleep(0.5)
sys.stderr.write( print(f"Failed to open browser automatically within {timeout} seconds. Please open {url} manually.")
f"Failed to open browser automatically within {timeout} seconds. Please open {url} manually.\n"
)
threading.Thread(target=wait_then_open, daemon=True).start() threading.Thread(target=wait_then_open, daemon=True).start()
@ -124,7 +115,7 @@ def main():
set_env() set_env()
env_file: pathlib.Path = pathlib.Path(os.getenv("YTP_CONFIG_PATH", "")) / ".env" env_file: pathlib.Path = pathlib.Path(os.getenv("YTP_CONFIG_PATH")) / ".env"
port = None port = None
if env_file.exists(): if env_file.exists():

View file

@ -199,16 +199,18 @@ class Main:
) )
HTTP_LOGGER_CLASS = HttpAccessLogger HTTP_LOGGER_CLASS = HttpAccessLogger
web.run_app( run_args = {
self._app, "host": host,
host=host, "port": port,
port=port, "loop": asyncio.get_event_loop(),
loop=asyncio.get_event_loop(), "access_log": HTTP_LOGGER,
access_log=HTTP_LOGGER, "print": started,
print=started, "handle_signals": cb is None,
handle_signals=cb is None, }
access_log_class=HTTP_LOGGER_CLASS if HTTP_LOGGER_CLASS is not None else web.AccessLogger, if HTTP_LOGGER_CLASS is not None:
) run_args["access_log_class"] = HTTP_LOGGER_CLASS
web.run_app(self._app, **run_args)
if __name__ == "__main__": if __name__ == "__main__":

View file

@ -144,7 +144,7 @@ async def serve_static_file(request: Request, config: Config) -> StreamResponse:
else: else:
return web.json_response({"error": "File not found.", "file": path}, status=web.HTTPNotFound.status_code) return web.json_response({"error": "File not found.", "file": path}, status=web.HTTPNotFound.status_code)
response = web.FileResponse( return web.FileResponse(
path=file_path, path=file_path,
headers={ headers={
"Pragma": "public", "Pragma": "public",
@ -154,16 +154,6 @@ async def serve_static_file(request: Request, config: Config) -> StreamResponse:
status=web.HTTPOk.status_code, status=web.HTTPOk.status_code,
) )
if STATIC_STATE.index_file is not None and file_path == STATIC_STATE.index_file:
response.set_cookie(
name="simple_mode",
value="true" if config.simple_mode else "false",
path=config.base_path if "/" != config.base_path else "/",
samesite="Lax",
)
return response
def setup_static_routes(root_path: Path, config: Config) -> None: def setup_static_routes(root_path: Path, config: Config) -> None:
""" """

View file

@ -35,7 +35,7 @@ async def get_ffprobe(request: Request, config: Config, encoder: Encoder, app: w
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
if not file: if not file:
return web.json_response(data={"error": "file is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required."}, status=web.HTTPBadRequest.status_code)
@ -76,7 +76,7 @@ async def get_file_info(request: Request, config: Config, encoder: Encoder, app:
Response: The response object. Response: The response object.
""" """
file: str | None = request.match_info.get("file") file: str = request.match_info.get("file")
if not file: if not file:
return web.json_response(data={"error": "file is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "file is required."}, status=web.HTTPBadRequest.status_code)
@ -137,7 +137,7 @@ async def file_browser(request: Request, config: Config, encoder: Encoder) -> Re
Response: The response object. Response: The response object.
""" """
req_path: str | None = request.match_info.get("path") req_path: str = request.match_info.get("path")
req_path: str = "/" if not req_path else unquote_plus(req_path) req_path: str = "/" if not req_path else unquote_plus(req_path)
raw_req: str = (req_path or "").strip() raw_req: str = (req_path or "").strip()
@ -461,7 +461,7 @@ async def path_actions(request: Request, config: Config, queue: DownloadQueue, n
record(path, ok=False, error=str(e), action=action, extra={"item": params}) record(path, ok=False, error=str(e), action=action, extra={"item": params})
continue continue
else: else:
extra_info: dict[str, Any] = {"new_path": renamed} extra_info: dict[str, Path] = {"new_path": renamed}
if sidecar_renamed: if sidecar_renamed:
extra_info["sidecar_count"] = len(sidecar_renamed) extra_info["sidecar_count"] = len(sidecar_renamed)
record(path, ok=True, action=action, extra=extra_info) record(path, ok=True, action=action, extra=extra_info)
@ -591,7 +591,7 @@ async def path_actions(request: Request, config: Config, queue: DownloadQueue, n
record(path, ok=False, error=str(e), action=action, extra={"item": params}) record(path, ok=False, error=str(e), action=action, extra={"item": params})
continue continue
else: else:
extra_info: dict[str, Any] = {"new_path": moved} extra_info: dict[str, Path] = {"new_path": moved}
if sidecar_moved: if sidecar_moved:
extra_info["sidecar_count"] = len(sidecar_moved) extra_info["sidecar_count"] = len(sidecar_moved)
@ -617,7 +617,7 @@ async def prepare_zip_file(request: Request, config: Config, cache: Cache):
if not json or not isinstance(json, list): if not json or not isinstance(json, list):
return web.json_response({"error": "Invalid parameters."}, status=400) return web.json_response({"error": "Invalid parameters."}, status=400)
files: list[Path] = [] files: list[str] = []
for f in json: for f in json:
if not isinstance(f, str): if not isinstance(f, str):
continue continue
@ -627,10 +627,12 @@ async def prepare_zip_file(request: Request, config: Config, cache: Cache):
continue continue
files.append(ref) files.append(ref)
sc: dict[str, list[dict[str, Any]]] = get_file_sidecar(ref) sc: list[dict] = get_file_sidecar(ref)
if sc: if sc:
for value in sc.values(): for side in sc:
files.extend(scf["file"] for scf in value if isinstance(scf, dict) and "file" in scf) for scf in sc[side]:
if isinstance(scf, dict) and "file" in scf:
files.append(scf["file"]) # noqa: PERF401
if not files: if not files:
return web.json_response({"error": "No valid files."}, status=400) return web.json_response({"error": "No valid files."}, status=400)

View file

@ -95,7 +95,7 @@ async def get_doc(request: Request, config: Config, cache: Cache) -> Response:
cache.set(cache_key, dct, ttl=3600) cache.set(cache_key, dct, ttl=3600)
return web.Response(body=dct["body"], headers=dct["headers"]) return web.Response(**dct)
except Exception: except Exception:
LOG.exception( LOG.exception(
"Failed to request doc '%s' from '%s'.", "Failed to request doc '%s' from '%s'.",

View file

@ -1,5 +1,5 @@
from aiohttp import web from aiohttp import web
from aiohttp.web import Request, Response, StreamResponse from aiohttp.web import Request, Response
from app.library.config import Config from app.library.config import Config
from app.library.log import get_logger from app.library.log import get_logger
@ -10,7 +10,7 @@ LOG = get_logger()
@route(["GET", "HEAD"], "/api/download/{filename:.+}", "download_static") @route(["GET", "HEAD"], "/api/download/{filename:.+}", "download_static")
async def download_file(request: Request, config: Config, app: web.Application) -> StreamResponse: async def download_file(request: Request, config: Config, app: web.Application) -> Response:
""" """
Serve download files. Serve download files.

View file

@ -1,10 +1,9 @@
import asyncio import asyncio
from pathlib import Path from pathlib import Path
from typing import Any from typing import TYPE_CHECKING, Any
from aiohttp import web from aiohttp import web
from aiohttp.web import Request, Response from aiohttp.web import Request, Response
from aiohttp.web_response import StreamResponse
from app.features.presets.schemas import Preset from app.features.presets.schemas import Preset
from app.features.presets.service import Presets from app.features.presets.service import Presets
@ -21,6 +20,10 @@ from app.library.log import get_logger
from app.library.router import route from app.library.router import route
from app.library.Utils import calc_download_path, get_file_sidecar, rename_file from app.library.Utils import calc_download_path, get_file_sidecar, rename_file
if TYPE_CHECKING:
from library.downloads import Download
LOG = get_logger() LOG = get_logger()
@ -120,29 +123,23 @@ async def items_list(request: Request, queue: DownloadQueue, encoder: Encoder, c
@route("GET", "api/history/live", "items_live") @route("GET", "api/history/live", "items_live")
async def items_live(request: Request, queue: DownloadQueue, encoder: Encoder, config: Config) -> Response: async def items_live(queue: DownloadQueue, encoder: Encoder) -> Response:
""" """
Get live queue data Get live queue data
Args: Args:
request (Request): The request object.
queue (DownloadQueue): The download queue instance. queue (DownloadQueue): The download queue instance.
encoder (Encoder): The encoder instance. encoder (Encoder): The encoder instance.
config (Config): The configuration instance.
Returns: Returns:
Response: The response object with live queue items. Response: The response object with live queue items.
""" """
try:
limit = int(request.query.get("limit", config.queue_display_limit))
except ValueError:
return web.json_response( return web.json_response(
data={"error": "limit must be a valid integer."}, status=web.HTTPBadRequest.status_code data={
) "queue": (await queue.get("queue"))["queue"],
"history_count": await queue.done.get_total_count(),
return web.json_response( },
data={**queue.live_queue(limit), "history_count": await queue.done.get_total_count()},
status=web.HTTPOk.status_code, status=web.HTTPOk.status_code,
dumps=encoder.encode, dumps=encoder.encode,
) )
@ -277,7 +274,7 @@ async def item_view(request: Request, queue: DownloadQueue, encoder: Encoder) ->
Response: The response object. Response: The response object.
""" """
id: str | None = request.match_info.get("id") id: str = request.match_info.get("id")
if not id: if not id:
return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code)
@ -311,7 +308,7 @@ async def item_view(request: Request, queue: DownloadQueue, encoder: Encoder) ->
@route(["GET", "HEAD"], r"api/history/{id}/thumbnail", "history.item.thumbnail") @route(["GET", "HEAD"], r"api/history/{id}/thumbnail", "history.item.thumbnail")
async def item_thumbnail(request: Request, queue: DownloadQueue, config: Config) -> StreamResponse: async def item_thumbnail(request: Request, queue: DownloadQueue, config: Config) -> Response:
if not (id := request.match_info.get("id")): if not (id := request.match_info.get("id")):
return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code)
@ -455,7 +452,7 @@ async def item_update(request: Request, queue: DownloadQueue, encoder: Encoder,
Response: The response object. Response: The response object.
""" """
id: str | None = request.match_info.get("id") id: str = request.match_info.get("id")
if not id: if not id:
return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code) return web.json_response(data={"error": "id is required."}, status=web.HTTPBadRequest.status_code)

View file

@ -116,9 +116,8 @@ async def get_background(request: Request, config: Config, cache: Cache) -> Resp
safe_backend: str = _safe_url(backend) safe_backend: str = _safe_url(backend)
await cache.aset(key=CACHE_KEY_BING, value=backend, ttl=3600 * 24) await cache.aset(key=CACHE_KEY_BING, value=backend, ttl=3600 * 24)
else: else:
cached_backend = await cache.aget(CACHE_KEY_BING) backend = await cache.aget(CACHE_KEY_BING)
backend = cached_backend if isinstance(cached_backend, str) else "" safe_backend = _safe_url(backend if isinstance(backend, str) else None)
safe_backend = _safe_url(backend)
if not isinstance(backend, str) or not backend: if not isinstance(backend, str) or not backend:
return web.json_response( return web.json_response(

View file

@ -1,7 +1,6 @@
import asyncio import asyncio
import json import json
import os import os
from collections.abc import Awaitable, Callable
from pathlib import Path from pathlib import Path
from aiohttp import web from aiohttp import web
@ -111,7 +110,7 @@ async def _read_logfile(file: Path, offset: int = 0, limit: int = 50) -> dict:
return {"logs": [], "next_offset": None, "end_is_reached": True} return {"logs": [], "next_offset": None, "end_is_reached": True}
async def _tail_log(file: Path, emitter: Callable[[dict], Awaitable[None]], sleep_time: float = 0.5): async def _tail_log(file: Path, emitter: callable, sleep_time: float = 0.5):
""" """
Live tail. Live tail.

View file

@ -2,7 +2,6 @@ import asyncio
import os import os
import time import time
from pathlib import Path from pathlib import Path
from typing import Any
from aiohttp import web from aiohttp import web
from aiohttp.web import Request, Response from aiohttp.web import Request, Response
@ -21,14 +20,17 @@ from app.library.log import get_logger
from app.library.router import route from app.library.router import route
from app.library.TerminalSessionManager import TerminalSessionConflictError, TerminalSessionManager from app.library.TerminalSessionManager import TerminalSessionConflictError, TerminalSessionManager
from app.library.UpdateChecker import UpdateChecker from app.library.UpdateChecker import UpdateChecker
from app.library.Utils import list_folders
LOG = get_logger() LOG = get_logger()
DIAGNOSTICS_CACHE_KEY = "system:diagnostics"
DIAGNOSTICS_CACHE_TTL = 5.0
@route("GET", "api/system/configuration", "system.configuration") @route("GET", "api/system/configuration", "system.configuration")
async def system_config(queue: DownloadQueue, config: Config, encoder: Encoder) -> Response: async def system_config(queue: DownloadQueue, config: Config, encoder: Encoder) -> Response:
""" """
Get the system configuration. Pause non-active downloads.
Args: Args:
queue (DownloadQueue): The download queue instance. queue (DownloadQueue): The download queue instance.
@ -45,61 +47,19 @@ async def system_config(queue: DownloadQueue, config: Config, encoder: Encoder)
"presets": Presets.get_instance().get_all(), "presets": Presets.get_instance().get_all(),
"dl_fields": await DLFields.get_instance().get_all_serialized(), "dl_fields": await DLFields.get_instance().get_all_serialized(),
"paused": queue.is_paused(), "paused": queue.is_paused(),
"folders": list_folders(
path=Path(config.download_path),
base=Path(config.download_path),
depth_limit=config.download_path_depth - 1,
),
"history_count": await queue.done.get_total_count(), "history_count": await queue.done.get_total_count(),
"queue": (await queue.get("queue"))["queue"],
}, },
status=web.HTTPOk.status_code, status=web.HTTPOk.status_code,
dumps=encoder.encode, dumps=encoder.encode,
) )
@route("GET", "api/system/folders", "system.folders")
async def system_folders(request: Request, config: Config, encoder: Encoder, cache: Cache) -> Response:
"""
List child directories for a given relative path.
Query params:
path: Relative path within the download directory (default: root).
Returns:
Response: The response object.
"""
raw_path: str = request.query.get("path", "").strip().lstrip("/")
base: Path = Path(config.download_path).resolve()
target: Path = (base / raw_path).resolve()
if not target.is_relative_to(base):
return web.json_response(
data={"path": raw_path, "folders": []},
status=web.HTTPOk.status_code,
dumps=encoder.encode,
)
cache_key = f"folders:{target!s}"
if (cached := cache.get(cache_key)) is not None:
return web.json_response(data=cached, status=web.HTTPOk.status_code, dumps=encoder.encode)
folders: list[str] = []
if target.is_dir():
try:
folders.extend(
entry.name
for entry in sorted(target.iterdir())
if entry.is_dir() and entry.resolve().is_relative_to(base)
)
except PermissionError:
pass
resolved_rel: str = str(target.relative_to(base)) if target != base else ""
if resolved_rel == ".":
resolved_rel = ""
data: dict[str, str | list[str]] = {"path": resolved_rel, "folders": folders}
cache.set(cache_key, data, ttl=30.0)
return web.json_response(data=data, status=web.HTTPOk.status_code, dumps=encoder.encode)
@route("POST", "api/system/pause", "system.pause") @route("POST", "api/system/pause", "system.pause")
async def downloads_pause(queue: DownloadQueue, encoder: Encoder, notify: EventBus) -> Response: async def downloads_pause(queue: DownloadQueue, encoder: Encoder, notify: EventBus) -> Response:
""" """
@ -287,7 +247,7 @@ async def system_diagnostics(
LOG.exception("Failed to collect system diagnostics.") LOG.exception("Failed to collect system diagnostics.")
data = diagnostics_error_report(config) data = diagnostics_error_report(config)
else: else:
cache.set(cache_key, data, ttl=60.0) cache.set(cache_key, data, ttl=60)
return web.json_response(data=data, status=web.HTTPOk.status_code, dumps=encoder.encode) return web.json_response(data=data, status=web.HTTPOk.status_code, dumps=encoder.encode)
@ -490,7 +450,7 @@ async def system_limits(queue: DownloadQueue, config: Config, encoder: Encoder)
for download in [*active_non_live, *queued_non_live]: for download in [*active_non_live, *queued_non_live]:
extractor = (download.info.get_extractor() or "unknown").lower() extractor = (download.info.get_extractor() or "unknown").lower()
entry: dict[str, Any] = per_extractor.setdefault( entry = per_extractor.setdefault(
extractor, extractor,
{ {
"name": extractor, "name": extractor,

View file

@ -15,7 +15,7 @@ from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from sqlalchemy import text from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine, create_async_engine from sqlalchemy.ext.asyncio import AsyncConnection, create_async_engine
APP_ROOT = str((Path(__file__).parent / ".." / "..").resolve()) APP_ROOT = str((Path(__file__).parent / ".." / "..").resolve())
if APP_ROOT not in sys.path: if APP_ROOT not in sys.path:
@ -69,7 +69,7 @@ def parse_args() -> argparse.Namespace:
return parser.parse_args() return parser.parse_args()
async def connect_db(db_file: Path) -> tuple[AsyncConnection, AsyncEngine]: async def connect_db(db_file: str) -> AsyncConnection:
"""Create async SQLAlchemy connection.""" """Create async SQLAlchemy connection."""
engine = create_async_engine(f"sqlite+aiosqlite:///{db_file}") engine = create_async_engine(f"sqlite+aiosqlite:///{db_file}")
conn = await engine.connect() conn = await engine.connect()

View file

@ -1,4 +1,3 @@
from typing import Any
from unittest.mock import MagicMock from unittest.mock import MagicMock
import pytest import pytest
@ -92,7 +91,7 @@ class TestAgSet:
def test_ag_set_error_on_non_dict_final(self): def test_ag_set_error_on_non_dict_final(self):
"""Test error when final target is not a dict.""" """Test error when final target is not a dict."""
data: Any = "not_a_dict" data = "not_a_dict"
with pytest.raises(RuntimeError, match="Cannot set value at path 'key'"): with pytest.raises(RuntimeError, match="Cannot set value at path 'key'"):
ag_set(data, "key", "value") ag_set(data, "key", "value")

View file

@ -1,7 +1,6 @@
from __future__ import annotations from __future__ import annotations
import http.cookiejar import http.cookiejar
from typing import Any
from unittest.mock import Mock, patch from unittest.mock import Mock, patch
import pytest import pytest
@ -13,10 +12,10 @@ try:
YTDLP_AVAILABLE = True YTDLP_AVAILABLE = True
except ImportError: except ImportError:
YTDLP_AVAILABLE = False YTDLP_AVAILABLE = False
Request: Any = Mock Request = Mock
Response: Any = Mock Response = Mock
RequestDirector: Any = Mock RequestDirector = Mock
HTTPError: Any = Exception HTTPError = Exception
pytestmark = pytest.mark.skipif(not YTDLP_AVAILABLE, reason="yt-dlp not available") pytestmark = pytest.mark.skipif(not YTDLP_AVAILABLE, reason="yt-dlp not available")

View file

@ -1,6 +1,5 @@
from pathlib import Path from pathlib import Path
from types import SimpleNamespace from types import SimpleNamespace
from typing import Any
from unittest.mock import AsyncMock, Mock, patch from unittest.mock import AsyncMock, Mock, patch
import pytest import pytest
@ -25,7 +24,7 @@ class TestConditionIgnoreMatching:
second = SimpleNamespace(id=2, name="Fallback", enabled=True, filter="duration > 0", priority=10) second = SimpleNamespace(id=2, name="Fallback", enabled=True, filter="duration > 0", priority=10)
service = Conditions.get_instance() service = Conditions.get_instance()
service._repo = Mock(all=AsyncMock(return_value=[first, second])) service._repo = Mock(list=AsyncMock(return_value=[first, second]))
matched = await service.match(info={"duration": 60}, ignore_conditions=["Primary"]) matched = await service.match(info={"duration": 60}, ignore_conditions=["Primary"])
@ -37,7 +36,7 @@ class TestConditionIgnoreMatching:
second = SimpleNamespace(id=124, name="Fallback", enabled=True, filter="duration > 0", priority=10) second = SimpleNamespace(id=124, name="Fallback", enabled=True, filter="duration > 0", priority=10)
service = Conditions.get_instance() service = Conditions.get_instance()
service._repo = Mock(all=AsyncMock(return_value=[first, second])) service._repo = Mock(list=AsyncMock(return_value=[first, second]))
matched = await service.match(info={"duration": 60}, ignore_conditions=["123"]) matched = await service.match(info={"duration": 60}, ignore_conditions=["123"])
@ -49,7 +48,7 @@ class TestConditionIgnoreMatching:
second = SimpleNamespace(id=124, name="Fallback", enabled=True, filter="duration > 0", priority=10) second = SimpleNamespace(id=124, name="Fallback", enabled=True, filter="duration > 0", priority=10)
service = Conditions.get_instance() service = Conditions.get_instance()
service._repo = Mock(all=AsyncMock(return_value=[first, second])) service._repo = Mock(list=AsyncMock(return_value=[first, second]))
matched = await service.match(info={"duration": 60}, ignore_conditions=[123]) matched = await service.match(info={"duration": 60}, ignore_conditions=[123])
@ -60,7 +59,7 @@ class TestConditionIgnoreMatching:
first = SimpleNamespace(id=1, name="Primary", enabled=True, filter="duration > 0", priority=20) first = SimpleNamespace(id=1, name="Primary", enabled=True, filter="duration > 0", priority=20)
service = Conditions.get_instance() service = Conditions.get_instance()
service._repo = Mock(all=AsyncMock(return_value=[first])) service._repo = Mock(list=AsyncMock(return_value=[first]))
matched = await service.match(info={"duration": 60}, ignore_conditions=["*"]) matched = await service.match(info={"duration": 60}, ignore_conditions=["*"])
@ -79,11 +78,10 @@ class TestConditionIgnorePropagation:
preset="default", preset="default",
extras={"ignore_conditions": [" 123 ", "Primary", "", " * "]}, extras={"ignore_conditions": [" 123 ", "Primary", "", " * "]},
) )
mock_item: Any = item item.get_ytdlp_opts = Mock(return_value=Mock(get_all=Mock(return_value={})))
mock_item.get_ytdlp_opts = Mock(return_value=Mock(get_all=Mock(return_value={}))) item.get_archive_id = Mock(return_value=None)
mock_item.get_archive_id = Mock(return_value=None) item.get_archive_file = Mock(return_value=None)
mock_item.get_archive_file = Mock(return_value=None) item.is_archived = Mock(return_value=False)
mock_item.is_archived = Mock(return_value=False)
matcher = Mock(match=AsyncMock(return_value=None)) matcher = Mock(match=AsyncMock(return_value=None))
entry = {"id": "video-1", "duration": 60} entry = {"id": "video-1", "duration": 60}
@ -115,11 +113,10 @@ class TestConditionIgnorePropagation:
preset="default", preset="default",
extras={"ignore_conditions": [123, "Primary", True, " * "]}, extras={"ignore_conditions": [123, "Primary", True, " * "]},
) )
mock_item: Any = item item.get_ytdlp_opts = Mock(return_value=Mock(get_all=Mock(return_value={})))
mock_item.get_ytdlp_opts = Mock(return_value=Mock(get_all=Mock(return_value={}))) item.get_archive_id = Mock(return_value=None)
mock_item.get_archive_id = Mock(return_value=None) item.get_archive_file = Mock(return_value=None)
mock_item.get_archive_file = Mock(return_value=None) item.is_archived = Mock(return_value=False)
mock_item.is_archived = Mock(return_value=False)
matcher = Mock(match=AsyncMock(return_value=None)) matcher = Mock(match=AsyncMock(return_value=None))
entry = {"id": "video-1", "duration": 60} entry = {"id": "video-1", "duration": 60}
@ -138,7 +135,7 @@ class TestConditionIgnorePropagation:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_playlist_keeps_parent_ignore(self) -> None: async def test_playlist_keeps_parent_ignore(self) -> None:
captured: dict[str, Any] = {} captured: dict[str, object] = {}
class FakeItem: class FakeItem:
def __init__(self) -> None: def __init__(self) -> None:
@ -161,8 +158,7 @@ class TestConditionIgnorePropagation:
} }
with patch("app.library.downloads.playlist_processor.ytdlp_reject", return_value=(True, "")): with patch("app.library.downloads.playlist_processor.ytdlp_reject", return_value=(True, "")):
fake: Any = FakeItem() result = await process_playlist(queue=queue, entry=entry, item=FakeItem())
result = await process_playlist(queue=queue, entry=entry, item=fake)
assert result == {"status": "ok"} assert result == {"status": "ok"}
assert captured["url"] == "https://example.com/v/1" assert captured["url"] == "https://example.com/v/1"
@ -213,8 +209,7 @@ class TestConditionIgnorePropagation:
} }
with patch("app.library.downloads.playlist_processor.ytdlp_reject", return_value=(True, "")): with patch("app.library.downloads.playlist_processor.ytdlp_reject", return_value=(True, "")):
fake: Any = FakeItem() result = await process_playlist(queue=queue, entry=entry, item=FakeItem())
result = await process_playlist(queue=queue, entry=entry, item=fake)
assert result == {"status": "ok"} assert result == {"status": "ok"}
assert queue.add.await_count == 2 assert queue.add.await_count == 2

View file

@ -4,13 +4,10 @@ from collections import OrderedDict
from dataclasses import asdict from dataclasses import asdict
from datetime import UTC, datetime from datetime import UTC, datetime
from email.utils import formatdate from email.utils import formatdate
from typing import Any
from unittest.mock import AsyncMock, Mock from unittest.mock import AsyncMock, Mock
import pytest import pytest
from app.library.DataStore import DataStore, StoreType from app.library.DataStore import DataStore, StoreType
from app.library.downloads import Download
from app.library.ItemDTO import ItemDTO from app.library.ItemDTO import ItemDTO
from app.library.operations import Operation from app.library.operations import Operation
from app.library.sqlite_store import SqliteStore from app.library.sqlite_store import SqliteStore
@ -65,18 +62,17 @@ async def make_db(data: int = 0) -> SqliteStore:
return ins return ins
class StubDownload(Download): class StubDownload:
def __init__(self, info: ItemDTO, started: bool = False, cancelled: bool = False) -> None: def __init__(self, info: ItemDTO, started: bool = False, cancelled: bool = False):
self.info = info self.info = info
self.id = info._id self._started: bool = started
self._stub_started: bool = started self._cancelled: bool = cancelled
self._stub_cancelled: bool = cancelled
def started(self) -> bool: def started(self) -> bool:
return self._stub_started return self._started
def is_cancelled(self) -> bool: def is_cancelled(self) -> bool:
return self._stub_cancelled return self._cancelled
async def make_store_async(store_type: StoreType) -> DataStore: async def make_store_async(store_type: StoreType) -> DataStore:
@ -361,8 +357,7 @@ class TestDataStore:
def __init__(self): def __init__(self):
self.info = None self.info = None
broken: Any = BrokenDownload() store._dict["broken"] = BrokenDownload()
store._dict["broken"] = broken
# Should still find the valid item # Should still find the valid item
result = await store.get_item(title="Video 1") result = await store.get_item(title="Video 1")
@ -965,10 +960,10 @@ class TestDataStoreOperations:
item1 = make_item(id="vid1", title="Video 1") item1 = make_item(id="vid1", title="Video 1")
item1._id = "id1" item1._id = "id1"
setattr(item1, "filesize", 1000) item1.filesize = 1000
item2 = make_item(id="vid2", title="Video 2") item2 = make_item(id="vid2", title="Video 2")
item2._id = "id2" item2._id = "id2"
setattr(item2, "filesize", 2000) item2.filesize = 2000
await store.put(StubDownload(info=item1)) await store.put(StubDownload(info=item1))
await store.put(StubDownload(info=item2)) await store.put(StubDownload(info=item2))
@ -991,10 +986,10 @@ class TestDataStoreOperations:
item1 = make_item(id="vid1", title="Video 1") item1 = make_item(id="vid1", title="Video 1")
item1._id = "id1" item1._id = "id1"
setattr(item1, "filesize", 1000) item1.filesize = 1000
item2 = make_item(id="vid2", title="Video 2") item2 = make_item(id="vid2", title="Video 2")
item2._id = "id2" item2._id = "id2"
setattr(item2, "filesize", 2000) item2.filesize = 2000
await store.put(StubDownload(info=item1)) await store.put(StubDownload(info=item1))
await store.put(StubDownload(info=item2)) await store.put(StubDownload(info=item2))
@ -1012,7 +1007,7 @@ class TestDataStoreOperations:
item1 = make_item(id="vid1", title="Video 1") item1 = make_item(id="vid1", title="Video 1")
item1._id = "id1" item1._id = "id1"
setattr(item1, "filesize", 1000) item1.filesize = 1000
await store.put(StubDownload(info=item1)) await store.put(StubDownload(info=item1))
@ -1038,7 +1033,7 @@ class TestDataStoreOperations:
item1 = make_item(id="vid1", title="Video 1") item1 = make_item(id="vid1", title="Video 1")
item1._id = "id1" item1._id = "id1"
setattr(item1, "filesize", 1000) item1.filesize = 1000
await store.put(StubDownload(info=item1)) await store.put(StubDownload(info=item1))
@ -1089,7 +1084,7 @@ class TestDataStoreOperations:
item1 = make_item(id="vid1", title="Video 1") item1 = make_item(id="vid1", title="Video 1")
item1._id = "id1" item1._id = "id1"
setattr(item1, "description", None) item1.description = None
await store.put(StubDownload(info=item1)) await store.put(StubDownload(info=item1))

View file

@ -307,8 +307,7 @@ class TestDownloadFlow:
postprocessor_hook=Mock(), postprocessor_hook=Mock(),
post_hook=Mock(), post_hook=Mock(),
) )
download_any: Any = download.info download.info.get_ytdlp_opts = Mock(
download_any.get_ytdlp_opts = Mock(
return_value=Mock( return_value=Mock(
add=Mock( add=Mock(
return_value=Mock( return_value=Mock(
@ -380,8 +379,7 @@ class TestDownloadFlow:
postprocessor_hook=Mock(), postprocessor_hook=Mock(),
post_hook=Mock(), post_hook=Mock(),
) )
download_any2: Any = download.info download.info.get_ytdlp_opts = Mock(
download_any2.get_ytdlp_opts = Mock(
return_value=Mock(add=Mock(return_value=Mock(get_all=Mock(return_value={})))) return_value=Mock(add=Mock(return_value=Mock(get_all=Mock(return_value={}))))
) )
@ -483,8 +481,7 @@ class TestDownloadFlow:
) )
queue.put(Terminator()) queue.put(Terminator())
download_mock: Any = download download._download = fake_download
download_mock._download = fake_download
class InlineProcess: class InlineProcess:
def __init__(self, target): def __init__(self, target):
@ -581,8 +578,7 @@ class TestDownloadFlow:
queue.put({"id": download.id, "status": "finished", "final_name": str(final_file)}) queue.put({"id": download.id, "status": "finished", "final_name": str(final_file)})
queue.put(Terminator()) queue.put(Terminator())
download_mock: Any = download download._download = fake_download
download_mock._download = fake_download
class InlineProcess: class InlineProcess:
def __init__(self, target): def __init__(self, target):
@ -1032,7 +1028,7 @@ class TestStatusTracker:
"debug": False, "debug": False,
} }
def test_init_sets_attributes(self, mock_config: dict[str, Any]) -> None: def test_init_sets_attributes(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
assert st.id == "test-id", "Should set download ID" assert st.id == "test-id", "Should set download ID"
assert st.info == mock_config["info"], "Should set info reference" assert st.info == mock_config["info"], "Should set info reference"
@ -1040,7 +1036,7 @@ class TestStatusTracker:
assert st.final_update is False, "Should initialize final_update as False" assert st.final_update is False, "Should initialize final_update as False"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_ignores_bad_id(self, mock_config: dict[str, Any]) -> None: async def test_status_ignores_bad_id(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "wrong-id", "status": "downloading"} status = {"id": "wrong-id", "status": "downloading"}
@ -1048,14 +1044,14 @@ class TestStatusTracker:
assert st.info.status != "downloading", "Should not update status for wrong ID" assert st.info.status != "downloading", "Should not update status for wrong ID"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_ignores_short(self, mock_config: dict[str, Any]) -> None: async def test_status_ignores_short(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id"} status = {"id": "test-id"}
await st.process_status_update(status) await st.process_status_update(status)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_process_status_update_sets_status(self, mock_config: dict[str, Any]) -> None: async def test_process_status_update_sets_status(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id", "status": "downloading", "downloaded_bytes": 1000} status = {"id": "test-id", "status": "downloading", "downloaded_bytes": 1000}
@ -1063,7 +1059,7 @@ class TestStatusTracker:
assert st.info.status == "downloading", "Should update info status" assert st.info.status == "downloading", "Should update info status"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_sets_skipped(self, mock_config: dict[str, Any]) -> None: async def test_status_sets_skipped(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id", "status": "downloading", "download_skipped": True} status = {"id": "test-id", "status": "downloading", "download_skipped": True}
@ -1071,7 +1067,7 @@ class TestStatusTracker:
assert st.info.download_skipped is True, "Should update download_skipped from status queue" assert st.info.download_skipped is True, "Should update download_skipped from status queue"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_process_status_update_sets_tmpfilename(self, mock_config: dict[str, Any]) -> None: async def test_process_status_update_sets_tmpfilename(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id", "status": "downloading", "tmpfilename": "/tmp/file.part"} status = {"id": "test-id", "status": "downloading", "tmpfilename": "/tmp/file.part"}
@ -1079,7 +1075,7 @@ class TestStatusTracker:
assert st.tmpfilename == "/tmp/file.part", "Should update tmpfilename" assert st.tmpfilename == "/tmp/file.part", "Should update tmpfilename"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_sets_percent(self, mock_config: dict[str, Any]) -> None: async def test_status_sets_percent(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = { status = {
"id": "test-id", "id": "test-id",
@ -1094,7 +1090,7 @@ class TestStatusTracker:
assert st.info.percent == 50.0, "Should calculate percent correctly" assert st.info.percent == 50.0, "Should calculate percent correctly"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_uses_estimate(self, mock_config: dict[str, Any]) -> None: async def test_status_uses_estimate(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = { status = {
"id": "test-id", "id": "test-id",
@ -1108,7 +1104,7 @@ class TestStatusTracker:
assert st.info.percent == 30.0, "Should calculate percent from estimate" assert st.info.percent == 30.0, "Should calculate percent from estimate"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_percent(self, mock_config: dict[str, Any]) -> None: async def test_status_percent(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = { status = {
"id": "test-id", "id": "test-id",
@ -1121,7 +1117,7 @@ class TestStatusTracker:
assert st.info.percent == 50.0, "Should calculate percent correctly with valid total" assert st.info.percent == 50.0, "Should calculate percent correctly with valid total"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_status_sets_speed_eta(self, mock_config: dict[str, Any]) -> None: async def test_status_sets_speed_eta(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id", "status": "downloading", "speed": 1024000, "eta": 60} status = {"id": "test-id", "status": "downloading", "speed": 1024000, "eta": 60}
@ -1130,7 +1126,7 @@ class TestStatusTracker:
assert st.info.eta == 60, "Should set eta" assert st.info.eta == 60, "Should set eta"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_process_status_update_sets_error(self, mock_config: dict[str, Any]) -> None: async def test_process_status_update_sets_error(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
status = {"id": "test-id", "status": "error", "error": "Download failed"} status = {"id": "test-id", "status": "error", "error": "Download failed"}
@ -1139,7 +1135,7 @@ class TestStatusTracker:
assert st.info.error == "Download failed", "Should set error message" assert st.info.error == "Download failed", "Should set error message"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_process_status_update_sets_final_update(self, tmp_path: Path, mock_config: dict[str, Any]) -> None: async def test_process_status_update_sets_final_update(self, tmp_path: Path, mock_config: dict) -> None:
test_file = tmp_path / "test.mp4" test_file = tmp_path / "test.mp4"
test_file.write_text("test content") test_file.write_text("test content")
@ -1152,20 +1148,20 @@ class TestStatusTracker:
assert st.info.filename == "test.mp4", "Should set relative filename" assert st.info.filename == "test.mp4", "Should set relative filename"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_drain_queue_processes_remaining_updates(self, mock_config: dict[str, Any]) -> None: async def test_drain_queue_processes_remaining_updates(self, mock_config: dict) -> None:
queue = DummyQueue() queue = DummyQueue()
queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 100}) queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 100})
queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 200}) queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 200})
queue.put(Terminator()) queue.put(Terminator())
config: dict[str, Any] = {**mock_config, "status_queue": queue} config = {**mock_config, "status_queue": queue}
st = StatusTracker(**config) st = StatusTracker(**config)
await st.drain_queue(max_iterations=10) await st.drain_queue(max_iterations=10)
assert st.info.downloaded_bytes == 200, "Should process all queued updates" assert st.info.downloaded_bytes == 200, "Should process all queued updates"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_drain_queue_stops_on_final_update(self, tmp_path: Path, mock_config: dict[str, Any]) -> None: async def test_drain_queue_stops_on_final_update(self, tmp_path: Path, mock_config: dict) -> None:
test_file = tmp_path / "test.mp4" test_file = tmp_path / "test.mp4"
test_file.write_text("test content") test_file.write_text("test content")
@ -1173,25 +1169,25 @@ class TestStatusTracker:
queue.put({"id": "test-id", "status": "finished", "final_name": str(test_file)}) queue.put({"id": "test-id", "status": "finished", "final_name": str(test_file)})
queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 999}) queue.put({"id": "test-id", "status": "downloading", "downloaded_bytes": 999})
config: dict[str, Any] = {**mock_config, "status_queue": queue, "download_dir": str(tmp_path)} config = {**mock_config, "status_queue": queue, "download_dir": str(tmp_path)}
st = StatusTracker(**config) st = StatusTracker(**config)
await st.drain_queue(max_iterations=10) await st.drain_queue(max_iterations=10)
assert st.final_update is True, "Should stop draining after final update" assert st.final_update is True, "Should stop draining after final update"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_drain_queue_skips_invalid(self, mock_config: dict[str, Any]) -> None: async def test_drain_queue_skips_invalid(self, mock_config: dict) -> None:
queue = DummyQueue() queue = DummyQueue()
queue.put({"id": "test-id", "status": "downloading"}) queue.put({"id": "test-id", "status": "downloading"})
queue.put(None) queue.put(None)
config: dict[str, Any] = {**mock_config, "status_queue": queue} config = {**mock_config, "status_queue": queue}
st = StatusTracker(**config) st = StatusTracker(**config)
await st.drain_queue(max_iterations=5) await st.drain_queue(max_iterations=5)
assert st.info.status == "downloading", "valid updates should still be processed" assert st.info.status == "downloading", "valid updates should still be processed"
def test_cancel_update_task(self, mock_config: dict[str, Any]) -> None: def test_cancel_update_task(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
st.update_task = Mock() st.update_task = Mock()
st.update_task.done = Mock(return_value=False) st.update_task.done = Mock(return_value=False)
@ -1200,122 +1196,23 @@ class TestStatusTracker:
st.cancel_update_task() st.cancel_update_task()
st.update_task.cancel.assert_called_once() st.update_task.cancel.assert_called_once()
def test_cancel_update_task_noop(self, mock_config: dict[str, Any]) -> None: def test_cancel_update_task_noop(self, mock_config: dict) -> None:
st = StatusTracker(**mock_config) st = StatusTracker(**mock_config)
st.cancel_update_task() st.cancel_update_task()
assert st.update_task is None, "missing tasks should be ignored" assert st.update_task is None, "missing tasks should be ignored"
def test_put_terminator_adds_to_queue(self, mock_config: dict[str, Any]) -> None: def test_put_terminator_adds_to_queue(self, mock_config: dict) -> None:
queue = DummyQueue() queue = DummyQueue()
config: dict[str, Any] = {**mock_config, "status_queue": queue} config = {**mock_config, "status_queue": queue}
st = StatusTracker(**config) st = StatusTracker(**config)
st.put_terminator() st.put_terminator()
assert 1 == len(queue.items), "Should add terminator to queue" assert 1 == len(queue.items), "Should add terminator to queue"
assert isinstance(queue.items[0], Terminator), "Should add Terminator instance" assert isinstance(queue.items[0], Terminator), "Should add Terminator instance"
@pytest.mark.asyncio
async def test_progress_emits_item_progress(self, mock_config: dict[str, Any]) -> None:
st = StatusTracker(**mock_config)
st.info.status = "downloading"
calls: list = []
st._notify = Mock()
st._notify.emit = Mock(side_effect=lambda *a, **kw: calls.append((a, kw)))
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 50, "total_bytes": 100}
)
progress_calls = [c for c in calls if c[0][0] == Events.ITEM_PROGRESS]
updated_calls = [c for c in calls if c[0][0] == Events.ITEM_UPDATED]
assert len(progress_calls) == 1
assert len(updated_calls) == 0
payload = progress_calls[0][1]["data"]
assert payload["_id"] == st.info._id
assert payload["percent"] == 50.0
assert "options" not in payload
@pytest.mark.asyncio
async def test_status_change_emits_item_updated(self, mock_config: dict[str, Any]) -> None:
st = StatusTracker(**mock_config)
st.info.status = "started"
calls: list = []
st._notify = Mock()
st._notify.emit = Mock(side_effect=lambda *a, **kw: calls.append((a, kw)))
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 10, "total_bytes": 100}
)
updated_calls = [c for c in calls if c[0][0] == Events.ITEM_UPDATED]
assert len(updated_calls) == 1
assert updated_calls[0][1]["data"] is st.info
@pytest.mark.asyncio
async def test_progress_throttled(self, mock_config: dict[str, Any]) -> None:
st = StatusTracker(**mock_config)
st.info.status = "downloading"
st._progress_interval = 0.5
calls: list = []
st._notify = Mock()
st._notify.emit = Mock(side_effect=lambda *a, **kw: calls.append((a, kw)))
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 10, "total_bytes": 100}
)
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 20, "total_bytes": 100}
)
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 30, "total_bytes": 100}
)
progress_calls = [c for c in calls if c[0][0] == Events.ITEM_PROGRESS]
assert len(progress_calls) == 1, "Rapid ticks should be throttled to one emission"
assert st._pending_progress is True
@pytest.mark.asyncio
async def test_flush_on_status_change(self, mock_config: dict[str, Any]) -> None:
st = StatusTracker(**mock_config)
st.info.status = "downloading"
st._progress_interval = 10.0
calls: list = []
st._notify = Mock()
st._notify.emit = Mock(side_effect=lambda *a, **kw: calls.append((a, kw)))
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 10, "total_bytes": 100}
)
await st.process_status_update(
{"id": "test-id", "status": "downloading", "downloaded_bytes": 50, "total_bytes": 100}
)
assert st._pending_progress is True
await st.process_status_update({"id": "test-id", "status": "error", "error": "fail"})
progress_calls = [c for c in calls if c[0][0] == Events.ITEM_PROGRESS]
updated_calls = [c for c in calls if c[0][0] == Events.ITEM_UPDATED]
assert len(progress_calls) == 2, "Pending progress should be flushed before status change"
assert len(updated_calls) == 1
assert st._pending_progress is False
class TestQueueManager: class TestQueueManager:
class LiveStore:
def __init__(self, items: dict[str, Download]) -> None:
self._items = items
def items(self):
return self._items.items()
def __contains__(self, key: str) -> bool:
return key in self._items
def __len__(self) -> int:
return len(self._items)
@staticmethod @staticmethod
def _video_queue() -> Mock: def _video_queue() -> Mock:
async def put(item): async def put(item):
@ -1349,45 +1246,6 @@ class TestQueueManager:
auto_start=True, auto_start=True,
) )
@staticmethod
def _any_video_item() -> Any:
return TestQueueManager._video_item()
def test_live_queue_caps_visible_items(self) -> None:
queue_manager = object.__new__(DownloadQueue)
items: dict[str, Any] = {f"id{i}": Mock(info=make_item(id=f"id{i}", title=f"Video {i}")) for i in range(5)}
queue_manager.queue = self.LiveStore(items)
queue_manager.pool = Mock()
queue_manager.pool.get_active_downloads.return_value = {}
snapshot = DownloadQueue.live_queue(queue_manager, limit=2)
queue_view = snapshot["queue"]
assert isinstance(queue_view, dict)
assert list(queue_view.keys()) == ["id0", "id1"]
assert snapshot["queue_count"] == 5
assert snapshot["queue_loaded"] == 2
assert snapshot["queue_limit"] == 2
def test_live_queue_keeps_active(self) -> None:
queue_manager = object.__new__(DownloadQueue)
items: dict[str, Any] = {f"id{i}": Mock(info=make_item(id=f"id{i}", title=f"Video {i}")) for i in range(5)}
queue_manager.queue = self.LiveStore(items)
queue_manager.pool = Mock()
queue_manager.pool.get_active_downloads.return_value = {
"id3": Mock(info=make_item(id="id3", title="Active 3")),
"id4": Mock(info=make_item(id="id4", title="Active 4")),
}
snapshot = DownloadQueue.live_queue(queue_manager, limit=1)
queue_view = snapshot["queue"]
assert isinstance(queue_view, dict)
assert list(queue_view.keys()) == ["id3", "id4"]
assert snapshot["queue_count"] == 5
assert snapshot["queue_loaded"] == 2
assert snapshot["queue_limit"] == 1
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_live_reextracts(self, monkeypatch: pytest.MonkeyPatch) -> None: async def test_live_reextracts(self, monkeypatch: pytest.MonkeyPatch) -> None:
seen: list[dict | None] = [] seen: list[dict | None] = []
@ -1400,7 +1258,7 @@ class TestQueueManager:
result = await add_video( result = await add_video(
queue=self._video_queue(), queue=self._video_queue(),
item=self._any_video_item(), item=self._video_item(),
entry={ entry={
"id": "live-id", "id": "live-id",
"title": "Live stream", "title": "Live stream",
@ -1432,7 +1290,7 @@ class TestQueueManager:
"formats": [{"format_id": "18"}], "formats": [{"format_id": "18"}],
} }
result = await add_video(queue=self._video_queue(), item=self._any_video_item(), entry=entry) result = await add_video(queue=self._video_queue(), item=self._video_item(), entry=entry)
assert result == {"status": "ok"} assert result == {"status": "ok"}
assert seen == [entry] assert seen == [entry]
@ -1557,11 +1415,7 @@ class TestQueueManager:
done_store.bulk_delete.assert_awaited_once_with(["done-id-1", "done-id-2"]) done_store.bulk_delete.assert_awaited_once_with(["done-id-1", "done-id-2"])
queue_manager._notify.emit.assert_called_once() queue_manager._notify.emit.assert_called_once()
assert queue_manager._notify.emit.call_args.args[0] == Events.ITEM_BULK_DELETED assert queue_manager._notify.emit.call_args.args[0] == Events.ITEM_BULK_DELETED
assert queue_manager._notify.emit.call_args.kwargs["data"]["count"] == 2 assert queue_manager._notify.emit.call_args.kwargs["data"] == {"ids": ["done-id-1", "done-id-2"], "count": 2}
assert queue_manager._notify.emit.call_args.kwargs["data"]["removed_files"] == 0
assert len(queue_manager._notify.emit.call_args.kwargs["data"]["items"]) == 2
assert queue_manager._notify.emit.call_args.kwargs["data"]["items"][0]["id"] == "done-id-1"
assert queue_manager._notify.emit.call_args.kwargs["data"]["items"][1]["id"] == "done-id-2"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_clear_status_fetches(self) -> None: async def test_clear_status_fetches(self) -> None:
@ -1569,25 +1423,19 @@ class TestQueueManager:
queue_manager.config = Mock(remove_files=False, download_path="/tmp") queue_manager.config = Mock(remove_files=False, download_path="/tmp")
queue_manager._notify = Mock() queue_manager._notify = Mock()
item = Mock()
item.info = make_item(id="done-id", title="Cleared clip")
done_store = Mock() done_store = Mock()
done_store.bulk_delete_by_status = AsyncMock(return_value=1) done_store.bulk_delete_by_status = AsyncMock(return_value=1)
done_store.get_many_by_status = AsyncMock(return_value=[("done-id", item)]) done_store.get_many_by_status = AsyncMock()
queue_manager.done = done_store queue_manager.done = done_store
result = await DownloadQueue.clear_by_status(queue_manager, "finished", remove_file=False) result = await DownloadQueue.clear_by_status(queue_manager, "finished", remove_file=False)
assert result == {"deleted": 1} assert result == {"deleted": 1}
done_store.get_many_by_status.assert_awaited_once_with("finished")
done_store.bulk_delete_by_status.assert_awaited_once_with("finished") done_store.bulk_delete_by_status.assert_awaited_once_with("finished")
done_store.get_many_by_status.assert_not_called()
queue_manager._notify.emit.assert_called_once() queue_manager._notify.emit.assert_called_once()
assert queue_manager._notify.emit.call_args.args[0] == Events.ITEM_BULK_DELETED assert queue_manager._notify.emit.call_args.args[0] == Events.ITEM_BULK_DELETED
assert queue_manager._notify.emit.call_args.kwargs["data"]["count"] == 1 assert queue_manager._notify.emit.call_args.kwargs["data"] == {"count": 1, "status": "finished"}
assert queue_manager._notify.emit.call_args.kwargs["data"]["status"] == "finished"
assert len(queue_manager._notify.emit.call_args.kwargs["data"]["items"]) == 1
assert queue_manager._notify.emit.call_args.kwargs["data"]["items"][0]["id"] == "done-id"
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_clear_status_files_fetch(self) -> None: async def test_clear_status_files_fetch(self) -> None:

View file

@ -1,7 +1,6 @@
import json import json
from datetime import date from datetime import date
from pathlib import Path from pathlib import Path
from typing import Any
from unittest.mock import MagicMock from unittest.mock import MagicMock
import pytest import pytest
@ -133,8 +132,7 @@ class TestEncoder:
return True return True
return original_isinstance(obj, cls) return original_isinstance(obj, cls)
builtins_any: Any = builtins builtins.isinstance = mock_isinstance
builtins_any.isinstance = mock_isinstance
try: try:
result = self.encoder.default(mock_daterange) result = self.encoder.default(mock_daterange)

Some files were not shown because too many files have changed in this diff Show more