Compare commits
No commits in common. "main" and "2.4.2" have entirely different histories.
975 changed files with 24810 additions and 216381 deletions
|
|
@ -25,13 +25,6 @@ __pycache__/
|
|||
dist/
|
||||
build/
|
||||
|
||||
# Frontend build artifacts and local dependency caches
|
||||
webui/.tanstack/
|
||||
webui/.vite/
|
||||
webui/node_modules/
|
||||
webui/test-results/
|
||||
webui/static/dist/
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
env/
|
||||
|
|
@ -58,20 +51,6 @@ Incomplete/*
|
|||
artist_bubble_snapshots.json
|
||||
.spotify_cache
|
||||
|
||||
# Auto-downloaded ffmpeg binaries — the YouTube client downloads these
|
||||
# into tools/ when system ffmpeg isn't on PATH. The Dockerfile installs
|
||||
# system ffmpeg via apt, so the container never needs the bundled
|
||||
# binaries. If a CI run leaves them in the workspace before the docker
|
||||
# build (e.g. because a test imported web_server which initialized the
|
||||
# YouTube client), they'd otherwise get baked into the image — adding
|
||||
# ~388 MB and getting duplicated again by the chown layer.
|
||||
tools/ffmpeg
|
||||
tools/ffprobe
|
||||
tools/ffmpeg.exe
|
||||
tools/ffprobe.exe
|
||||
tools/*.zip
|
||||
tools/*.tar.xz
|
||||
|
||||
# Documentation
|
||||
*.md
|
||||
README.md
|
||||
|
|
|
|||
4
.github/workflows/docker-publish.yml
vendored
4
.github/workflows/docker-publish.yml
vendored
|
|
@ -9,9 +9,9 @@ on:
|
|||
workflow_dispatch:
|
||||
inputs:
|
||||
version_tag:
|
||||
description: 'Version tag (e.g. 2.8.2)'
|
||||
description: 'Version tag (e.g. 2.4.2)'
|
||||
required: true
|
||||
default: '2.8.2'
|
||||
default: '2.4.2'
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
|
|
|
|||
11
.gitignore
vendored
11
.gitignore
vendored
|
|
@ -12,14 +12,11 @@ __pycache__/
|
|||
|
||||
# User-specific files (auto-created by the app if missing)
|
||||
config/config.json
|
||||
config/youtube_cookies.txt
|
||||
# All app databases are live user data — never commit (music_library, video_library, …)
|
||||
database/*.db
|
||||
database/*.db-shm
|
||||
database/*.db-wal
|
||||
database/*.db.backup_*
|
||||
database/music_library.db
|
||||
database/music_library.db-shm
|
||||
database/music_library.db-wal
|
||||
database/music_library.db.backup_*
|
||||
database/api_call_history.json
|
||||
storage/image_cache/
|
||||
logs/*.log
|
||||
logs/*.log.*
|
||||
|
||||
|
|
|
|||
|
|
@ -1,74 +0,0 @@
|
|||
# discover page — best in class plan (#913 + full generator audit)
|
||||
|
||||
morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
|
||||
|
||||
## what i shipped tonight (done, tested, safe)
|
||||
|
||||
### 1. listening recommendations (#913) — went from BROKEN to best-in-class
|
||||
|
||||
the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
|
||||
|
||||
- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
|
||||
- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
|
||||
- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
|
||||
|
||||
the fix (all in the pure, tested core + thin scan wiring):
|
||||
- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
|
||||
- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
|
||||
|
||||
result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
|
||||
|
||||
### 2. its own row on the discover page
|
||||
|
||||
new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
|
||||
|
||||
**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
|
||||
|
||||
> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
|
||||
|
||||
### 3. Fresh Tape "only 5-10 tracks" — fixed
|
||||
|
||||
root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
|
||||
|
||||
**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
|
||||
|
||||
---
|
||||
|
||||
## best-in-class roadmap for listening recs (next phases — your call)
|
||||
|
||||
these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
|
||||
|
||||
| phase | what | value | risk | notes |
|
||||
|---|---|---|---|---|
|
||||
| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
|
||||
| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
|
||||
| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
|
||||
| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
|
||||
| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
|
||||
|
||||
the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
|
||||
|
||||
---
|
||||
|
||||
## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
|
||||
|
||||
how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
|
||||
|
||||
### curated (built during the scan, then hydrated)
|
||||
- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
|
||||
- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
|
||||
- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
|
||||
|
||||
### discovery-pool generators (live queries)
|
||||
- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
|
||||
- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
|
||||
- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
|
||||
- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
|
||||
- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
|
||||
- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
|
||||
|
||||
### cross-cutting
|
||||
- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
|
||||
- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
|
||||
|
||||
want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.
|
||||
69
Dockerfile
69
Dockerfile
|
|
@ -1,16 +1,6 @@
|
|||
# SoulSync WebUI Dockerfile
|
||||
# Multi-architecture support for AMD64 and ARM64
|
||||
|
||||
FROM node:24-slim AS webui-builder
|
||||
|
||||
WORKDIR /app/webui
|
||||
|
||||
COPY webui/package.json webui/package-lock.json ./
|
||||
RUN npm ci
|
||||
|
||||
COPY webui/ ./
|
||||
RUN npm run build
|
||||
|
||||
# Stage 1: Builder — install Python dependencies with compilation tools
|
||||
FROM python:3.11-slim AS builder
|
||||
|
||||
|
|
@ -29,16 +19,6 @@ COPY requirements.txt .
|
|||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# yt-dlp must track YouTube faster than its stable channel ships — stable can
|
||||
# lag months behind a breaking YouTube change while extraction is broken
|
||||
# ("Requested format is not available"). Build images with the NIGHTLY channel.
|
||||
# COMMIT_SHA is referenced in the RUN so CI's layer cache (cache-from: gha)
|
||||
# busts on every new commit — otherwise this layer could pin a stale "nightly"
|
||||
# for months, silently defeating its purpose.
|
||||
ARG COMMIT_SHA=""
|
||||
RUN echo "yt-dlp nightly for build ${COMMIT_SHA}" && \
|
||||
pip install --no-cache-dir -U --pre "yt-dlp[default]"
|
||||
|
||||
# Stage 2: Runtime — only runtime dependencies, no build tools
|
||||
FROM python:3.11-slim
|
||||
|
||||
|
|
@ -54,61 +34,24 @@ ENV PATH="/opt/venv/bin:$PATH"
|
|||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install runtime-only system dependencies (no gcc/build tools).
|
||||
# unzip is needed by the Deno installer below.
|
||||
# Install runtime-only system dependencies (no gcc/build tools)
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
gosu \
|
||||
ffmpeg \
|
||||
libchromaprint-tools \
|
||||
unzip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Deno — JavaScript runtime for yt-dlp. YouTube gates its downloadable formats
|
||||
# behind JS challenges (nsig); without a JS runtime, yt-dlp's extraction is
|
||||
# deprecated and streams / music-video downloads fail with "Requested format
|
||||
# is not available". Deno is yt-dlp's default-enabled runtime; the official
|
||||
# installer auto-detects amd64/arm64. `deno --version` fails the build early
|
||||
# if the install ever breaks.
|
||||
RUN curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh && \
|
||||
deno --version
|
||||
|
||||
# Create non-root user for security
|
||||
RUN useradd --create-home --shell /bin/bash --uid 1000 soulsync
|
||||
|
||||
# Copy application code with ownership baked in.
|
||||
# Using `COPY --chown` instead of `COPY` + `chown -R /app` avoids an
|
||||
# extra image layer that duplicates the entire /app tree just to flip
|
||||
# ownership bits — Docker layers are immutable, so chown -R rewrites
|
||||
# every file into a new layer. On a clean repo that's small; if any
|
||||
# bulky workspace file slips in (e.g. auto-downloaded ffmpeg binaries
|
||||
# in tools/), it gets counted twice in the image. Cin caught this on
|
||||
# 2026-05-08 — see the .dockerignore comment for the same incident.
|
||||
COPY --chown=soulsync:soulsync . .
|
||||
COPY --chown=soulsync:soulsync --from=webui-builder /app/webui/static/dist /app/webui/static/dist
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Create runtime mount-point directories the app expects to exist.
|
||||
# Create necessary directories with proper permissions
|
||||
# NOTE: /app/data is for database FILES, /app/database is the Python package
|
||||
# NOTE: /app/Staging is required even though most users bind-mount it —
|
||||
# the entrypoint mkdir runs early and is gated by `set -e`, so a missing
|
||||
# pre-baked directory would crash the container into a restart loop on
|
||||
# rootless Docker/Podman where in-container "root" can't write to /app.
|
||||
# Pre-baking the dir here makes the entrypoint mkdir a guaranteed no-op.
|
||||
# NOTE: /app/Stream is the transient single-file streaming cache used by
|
||||
# the basic-search "Play" flow (cleared per use, never persistent). It's
|
||||
# created lazily by `core/streaming/prepare.py` via `os.makedirs`, which
|
||||
# fails silently on rootless Docker where the soulsync UID can't write
|
||||
# to /app — playback then errors out with no obvious cause. Pre-baking
|
||||
# at build time (when the layer is owned by root) avoids that path.
|
||||
# NOTE: /app/storage is the PRIVATE album-bundle staging area for the
|
||||
# torrent / usenet whole-release flow (download_source.album_bundle_staging_path
|
||||
# defaults to 'storage/album_bundle_staging'). Like /app/Stream it's created
|
||||
# lazily at runtime via mkdir(parents=True); without pre-baking it owned by
|
||||
# soulsync, the album-bundle copy fails with "[Errno 13] Permission denied:
|
||||
# 'storage'" because /app itself is root-owned and the soulsync UID can't
|
||||
# create a top-level dir there.
|
||||
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts && \
|
||||
chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts
|
||||
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/MusicVideos /app/scripts && \
|
||||
chown -R soulsync:soulsync /app
|
||||
|
||||
# Create defaults directory and copy template files
|
||||
# These will be used by entrypoint.sh to initialize empty volumes
|
||||
|
|
|
|||
84
README.md
84
README.md
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
> **IMPORTANT**: Configure file sharing in slskd to avoid Soulseek bans. Set up shared folders at `http://localhost:5030/shares`.
|
||||
|
||||
**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
|
||||
**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | [Reddit](https://old.reddit.com/r/ssync/) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -105,21 +105,6 @@ SoulSync bridges streaming services to your music library with automated discove
|
|||
- Catches wrong versions (live, remix, cover) even from streaming API sources
|
||||
- Fail-open design: verification errors never block downloads
|
||||
|
||||
#### AcoustID API key
|
||||
|
||||
AcoustID verification is opt-in. To enable it, request a free API key
|
||||
at <https://acoustid.org/new-application> and paste it into
|
||||
Settings → AcoustID. Without a key, downloads still complete but the
|
||||
verification step is skipped silently.
|
||||
|
||||
If a track was previously tagged by AcoustID but the retag action in
|
||||
the AcoustID Scanner no longer changes anything, see issue #704 — the
|
||||
most common cause is that the file already carries a
|
||||
`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
|
||||
and therefore never overwrites. Removing the cached
|
||||
`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
|
||||
restores the retag.
|
||||
|
||||
### Metadata & Enrichment
|
||||
|
||||
**10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz
|
||||
|
|
@ -293,64 +278,19 @@ The template points at `boulderbadgedad/soulsync:latest` (stable) by default. To
|
|||
```bash
|
||||
git clone https://github.com/Nezreka/SoulSync
|
||||
cd SoulSync
|
||||
python -m pip install -r requirements.txt
|
||||
|
||||
# Build the React WebUI bundle used by the Python server.
|
||||
# Docker does this automatically; Python installs must do it manually.
|
||||
cd webui
|
||||
npm ci
|
||||
npm run build
|
||||
cd ..
|
||||
|
||||
pip install -r requirements.txt
|
||||
gunicorn -c gunicorn.conf.py wsgi:application
|
||||
# Open http://localhost:8008
|
||||
```
|
||||
|
||||
When updating a Python/no-Docker install with `git pull`, rebuild the WebUI before restarting SoulSync:
|
||||
For local development and tests:
|
||||
|
||||
```bash
|
||||
cd webui
|
||||
npm ci
|
||||
npm run build
|
||||
cd ..
|
||||
pip install -r requirements-dev.txt
|
||||
pytest
|
||||
gunicorn -c gunicorn.dev.conf.py wsgi:application
|
||||
```
|
||||
|
||||
If `webui/static/dist/.vite/manifest.json` is missing or stale, React-owned routes and route handoffs may not load correctly.
|
||||
|
||||
**YouTube streaming / music videos** need two extra things on bare-metal installs (Docker bundles both):
|
||||
|
||||
- **Deno** — yt-dlp now requires a JavaScript runtime to unlock YouTube formats. Without it, streams and music-video downloads fail with `Requested format is not available`. Install: `winget install DenoLand.Deno` (Windows) or see [deno.com](https://docs.deno.com/runtime/), then restart SoulSync.
|
||||
- **yt-dlp nightly** — the stable release can lag months behind YouTube changes. If YouTube breaks, update with: `python -m pip install -U --pre "yt-dlp[default]"`
|
||||
|
||||
### Local Development
|
||||
|
||||
This is only for contributors working on the WebUI with hot reload. Normal Python/no-Docker installs should build once with `npm run build` as shown above, then run only Gunicorn.
|
||||
|
||||
For active frontend development, use two terminals so the backend and Vite stay independent:
|
||||
|
||||
1. Backend
|
||||
```bash
|
||||
python -m pip install -r requirements-dev.txt
|
||||
gunicorn -c gunicorn.dev.conf.py wsgi:application
|
||||
```
|
||||
The dev Gunicorn config watches backend files and restarts the Python server when they change.
|
||||
2. Frontend
|
||||
```bash
|
||||
cd webui
|
||||
npm ci
|
||||
npm run dev
|
||||
```
|
||||
Vite hot reloads the React side when you change webui files.
|
||||
|
||||
Run tests separately when needed:
|
||||
|
||||
```bash
|
||||
python -m pytest
|
||||
```
|
||||
|
||||
If you want a convenience launcher, `python dev.py` starts both halves together
|
||||
on any OS. `./dev.sh` remains available as a Unix shell wrapper.
|
||||
|
||||
---
|
||||
|
||||
## Setup Guide
|
||||
|
|
@ -360,7 +300,6 @@ on any OS. `./dev.sh` remains available as a Unix shell wrapper.
|
|||
- **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases)) — required for Soulseek downloads
|
||||
- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended for discovery
|
||||
- **Media Server** (optional): Plex, Jellyfin, or Navidrome
|
||||
- **Deno** (Python/no-Docker installs only): JavaScript runtime required by yt-dlp for YouTube streaming/music videos — `winget install DenoLand.Deno` or [deno.com](https://docs.deno.com/runtime/). Docker images bundle it.
|
||||
- **Deezer ARL token** (optional): For Deezer downloads — get from browser cookies after logging into deezer.com
|
||||
- **Tidal account** (optional): For Tidal downloads — authenticate via device flow in Settings
|
||||
- **Qobuz account** (optional): For Qobuz downloads — email/password login in Settings
|
||||
|
|
@ -474,14 +413,17 @@ SoulSync uses a `dev` → `main` flow:
|
|||
2. Branch off `dev`: `git checkout -b fix/your-change dev`
|
||||
3. Make your changes and commit
|
||||
4. Push and open a PR against **`dev`** (not `main`)
|
||||
5. CI (`build-and-test.yml`) runs ruff lint + compile + `python -m pytest` on your branch — wait for green
|
||||
5. CI (`build-and-test.yml`) runs ruff lint + compile + pytest on your branch — wait for green
|
||||
6. A maintainer reviews and merges
|
||||
|
||||
### Running locally
|
||||
|
||||
Use the [Local Development](#local-development) section above for the full repo-wide setup and the portable dev launcher.
|
||||
|
||||
For web UI work, see [webui/README.md](webui/README.md). It keeps the React-side notes close to the app while this file stays the single place for repo-wide dev instructions.
|
||||
```bash
|
||||
pip install -r requirements-dev.txt
|
||||
python -m ruff check . # must be 0 errors
|
||||
python -m pytest # all tests must pass
|
||||
gunicorn -c gunicorn.dev.conf.py wsgi:application
|
||||
```
|
||||
|
||||
Ruff config lives in `pyproject.toml`. The ruleset is intentionally lenient — it catches real bugs (undefined names, import shadowing, closure-in-loop) without style nits.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +0,0 @@
|
|||
**SoulSync 2.7.9** is out 🎉 a big one.
|
||||
|
||||
🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
|
||||
|
||||
🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
|
||||
|
||||
⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
|
||||
|
||||
🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
|
||||
|
||||
🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
|
||||
|
||||
⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
|
||||
|
||||
🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
|
||||
|
||||
✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
|
||||
|
||||
💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
|
||||
|
||||
🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
|
||||
|
||||
⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
|
||||
|
||||
🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
|
||||
|
||||
🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
|
||||
|
||||
💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
|
||||
|
||||
🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
|
||||
|
||||
⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
|
||||
|
||||
🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
**SoulSync 2.8.2** is out 🎉 a stability + performance release.
|
||||
|
||||
🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
|
||||
|
||||
⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
|
||||
|
||||
📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
|
||||
|
||||
**Status:** design only — no code yet.
|
||||
**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
|
||||
the user's actual files, so the Library Reorganizer, Track Number Repair, and
|
||||
tagging/enrichment all agree on the same release. Today each re-resolves
|
||||
independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
|
||||
Believer=3; standard album mislinked to a deluxe release).
|
||||
|
||||
**Canonical-selection rule (decided):** *match the user's actual files.* The
|
||||
canonical release is the candidate whose track count + per-track durations +
|
||||
titles best fit what's on disk. Self-correcting: picks standard when you own the
|
||||
standard, deluxe when you own the deluxe.
|
||||
|
||||
---
|
||||
|
||||
## Hard requirement: don't disrupt the running app
|
||||
|
||||
Every stage below is **additive and dormant until explicitly consumed**, and
|
||||
every consumer **falls back to today's behavior when no canonical is set**. So:
|
||||
- albums with no resolved canonical behave EXACTLY as they do now;
|
||||
- each stage is independently shippable and reversible;
|
||||
- nothing big-bangs.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
|
||||
|
||||
### Schema (additive, nullable → migration-safe)
|
||||
Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
|
||||
existing column-exists checks; see [[db-schema-review]] migration-safety notes):
|
||||
- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
|
||||
- `canonical_album_id TEXT`
|
||||
- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
|
||||
- `canonical_resolved_at TIMESTAMP`
|
||||
|
||||
All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
|
||||
backfill in this stage. No reads in this stage.
|
||||
|
||||
### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
|
||||
```
|
||||
score_release_against_files(file_tracks, release_tracks) -> float
|
||||
pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
|
||||
```
|
||||
- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
|
||||
- `release_tracks`: a candidate release's tracklist (same shape).
|
||||
- Scoring (tunable weights):
|
||||
- **track-count fit** — exact match strongly preferred; |Δcount| penalized.
|
||||
- **duration alignment** — greedily match each file to its closest release
|
||||
track by duration (within a tolerance, e.g. ±3s); reward coverage.
|
||||
- **title overlap** — token/fuzzy overlap as a tiebreaker.
|
||||
- **graceful degradation** — if a source gives no per-track durations, fall
|
||||
back to count + title only (never crash, never force-pick).
|
||||
- Returns the best candidate + score, or (None, 0) when nothing clears a floor
|
||||
(so we never pin a bad guess — leave it unresolved, consumers fall back).
|
||||
|
||||
### Tests (extreme, like the rest of this codebase)
|
||||
- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
|
||||
- same album, 17 files → picks deluxe.
|
||||
- duration disambiguation when track counts tie (e.g. radio edit vs album).
|
||||
- missing-duration source → count+title fallback still picks sanely.
|
||||
- no candidate clears the floor → (None, 0).
|
||||
- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
|
||||
whichever the files actually match.
|
||||
|
||||
**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
|
||||
them yet. Provably zero behavior change.**
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Resolver populates canonical (writes, still no consumers)
|
||||
|
||||
A function `resolve_canonical_for_album(album_id, db, ...)`:
|
||||
1. Gather on-disk file metadata for the album (durations/titles) via the
|
||||
library's known file paths.
|
||||
2. Gather candidate releases: every source the album has an ID for
|
||||
(spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
|
||||
case — sibling editions discoverable from those. Fetch each tracklist
|
||||
(cached, rate-limited).
|
||||
3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
|
||||
on the album row if it clears the floor.
|
||||
|
||||
Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
|
||||
when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
|
||||
unchanged — this stage only populates the new columns. Reversible: clearing the
|
||||
columns reverts to unresolved.
|
||||
|
||||
Tests: resolver picks the right release for the standard/deluxe fixtures; stores
|
||||
nothing when below floor; idempotent re-resolve.
|
||||
|
||||
Cost note: fetching multiple candidate releases = more API calls. Mitigate via
|
||||
cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
|
||||
progress so it's not silent.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
|
||||
|
||||
In `library_reorganize._resolve_source`: if the album has
|
||||
`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
|
||||
current `get_source_priority` walk. One-line precedence change, fully gated on
|
||||
non-NULL.
|
||||
|
||||
Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
|
||||
to today. Re-run the existing reorganize battery (148 tests) — must stay green.
|
||||
|
||||
**This alone fixes #767-Bug2** (a standard album whose files match the standard
|
||||
release pins the standard, so reorganize stops targeting the deluxe folder).
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Track Number Repair reads canonical (closes #765)
|
||||
|
||||
In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
|
||||
everything) — if the album has a canonical `(source, album_id)`, use it. The
|
||||
existing 6-level cascade stays as the fallback for albums with no canonical
|
||||
(preserves its all-01-album rescue ability — the regression risk we refused to
|
||||
take in the reactive fix).
|
||||
|
||||
Now both tools resolve the SAME release → same track numbers → no contradiction.
|
||||
|
||||
Tests: canonical present → both tools agree (shared-release test); canonical
|
||||
NULL → existing cascade unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Risks & mitigations
|
||||
- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
|
||||
only-on-(re)enrich, progress-logged.
|
||||
- **Sources without per-track durations** → scorer degrades to count+title.
|
||||
- **Schema migration** → additive nullable columns only; idempotent guards.
|
||||
- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
|
||||
stored for inspection/re-resolve; manual override possible later.
|
||||
- **Backward-compat** → every consumer falls back to today's path when NULL, so
|
||||
un-resolved albums (incl. all existing albums until backfilled) are unaffected.
|
||||
|
||||
## Out of scope (for now)
|
||||
- Per-album manual version override UI (can layer on later — the columns support it).
|
||||
- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
|
||||
once they share the canonical.
|
||||
|
||||
## Suggested order to build
|
||||
1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
|
||||
after any stage and the app is consistent (just with fewer consumers wired).
|
||||
|
|
@ -1,134 +0,0 @@
|
|||
# Running SoulSync behind a reverse proxy (nginx / Caddy / Traefik)
|
||||
|
||||
Putting SoulSync behind a reverse proxy lets you serve it over **HTTPS** and — the
|
||||
important part — put **authentication** in front of it before exposing it to the
|
||||
internet. This guide covers the safe setup.
|
||||
|
||||
> **The golden rule:** the safest way to expose *any* self-hosted app publicly is
|
||||
> to require authentication at the proxy (an auth layer), **not** to rely on the
|
||||
> app's own protection. SoulSync's launch PIN is a useful fallback, but it is not
|
||||
> a substitute for a real auth layer on a public instance.
|
||||
|
||||
---
|
||||
|
||||
## 1. Turn on reverse-proxy mode
|
||||
|
||||
By default SoulSync does **not** trust proxy headers (so a direct client can't spoof
|
||||
its IP or pretend the connection is HTTPS). If you're behind a proxy that
|
||||
terminates TLS, turn on **Settings → Security → "Behind a reverse proxy"** and
|
||||
**restart SoulSync** (this option applies at startup).
|
||||
|
||||
When enabled, SoulSync:
|
||||
- trusts `X-Forwarded-For/Proto/Host/Port` from **one** proxy hop (correct client
|
||||
IP, HTTPS detection, redirects),
|
||||
- marks its session cookie `Secure` (HTTPS-only) + `SameSite=Lax`, and
|
||||
- sends conservative security headers (`X-Content-Type-Options: nosniff`,
|
||||
`X-Frame-Options: SAMEORIGIN`, `Strict-Transport-Security`). No CSP is set — tune
|
||||
one at your proxy if you want it.
|
||||
|
||||
**Leave it off if you access SoulSync directly over http:// on your LAN** — turning
|
||||
it on would make the session cookie HTTPS-only and break plain-HTTP access. With it
|
||||
off, none of the above applies and SoulSync behaves exactly as before.
|
||||
|
||||
> The launch PIN is also brute-force limited (10 wrong attempts from an IP → a
|
||||
> short cooldown), regardless of this setting — a correct PIN is never affected.
|
||||
|
||||
Restart SoulSync after changing it.
|
||||
|
||||
---
|
||||
|
||||
## 2. nginx
|
||||
|
||||
SoulSync uses WebSockets (Socket.IO), so the `Upgrade`/`Connection` headers are
|
||||
**required** — without them live updates silently stop working.
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name soulsync.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/soulsync.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/soulsync.example.com/privkey.pem;
|
||||
|
||||
# Large library scans / uploads
|
||||
client_max_body_size 0;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8008;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
|
||||
# Required for Socket.IO / live updates
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
|
||||
proxy_read_timeout 3600s; # long-running scans
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Caddy
|
||||
|
||||
Caddy handles TLS automatically and proxies WebSockets out of the box:
|
||||
|
||||
```caddy
|
||||
soulsync.example.com {
|
||||
reverse_proxy 127.0.0.1:8008
|
||||
}
|
||||
```
|
||||
|
||||
Caddy sets `X-Forwarded-*` for you. (Add an auth provider directive if you want
|
||||
auth at the proxy — see below.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Traefik
|
||||
|
||||
Traefik proxies WebSockets automatically and forwards the headers. Point a router
|
||||
at the SoulSync service on port `8008` with your TLS resolver; no extra WebSocket
|
||||
config is needed.
|
||||
|
||||
---
|
||||
|
||||
## 5. Add authentication in front (recommended for public instances)
|
||||
|
||||
Pick one:
|
||||
|
||||
- **Auth proxy** — [Authelia](https://www.authelia.com/),
|
||||
[Authentik](https://goauthentik.io/), or
|
||||
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). These sit in front
|
||||
of SoulSync and force a login (with 2FA) before any request reaches it. Best
|
||||
option for internet exposure.
|
||||
|
||||
SoulSync can **trust the proxy's authenticated-user header** so the launch PIN is
|
||||
skipped once the proxy has logged you in. Set the header name in **Settings →
|
||||
Security → "Auth proxy user header"** (e.g. `Remote-User`).
|
||||
|
||||
> ⚠️ **Only enable this behind a proxy you control that STRIPS any client-supplied
|
||||
> copy of that header.** Otherwise a direct visitor could send `Remote-User: admin`
|
||||
> and walk straight in. It's **off by default** — an unset header name means
|
||||
> SoulSync ignores the header entirely (a spoofed one does nothing).
|
||||
- **HTTP Basic Auth** — quick and simple (nginx `auth_basic` / Caddy `basicauth`).
|
||||
Better than nothing; weaker than an auth proxy.
|
||||
- **SoulSync launch PIN** — set an admin PIN in Settings. Enforced server-side, so
|
||||
it can't be bypassed by hitting the API directly — but it's a shared PIN, so
|
||||
treat it as a fallback, not your only gate.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Live updates / progress bars don't move** → the WebSocket `Upgrade`/`Connection`
|
||||
headers are missing (nginx) or your proxy is buffering. Check section 2.
|
||||
- **Login won't stick / "session expired"** → you enabled `trust_reverse_proxy` but
|
||||
are reaching SoulSync over plain `http://`. The session cookie is now HTTPS-only;
|
||||
use `https://`, or turn the setting off for direct HTTP access.
|
||||
- **Scans time out** → raise `proxy_read_timeout` / `proxy_send_timeout`.
|
||||
|
|
@ -125,9 +125,6 @@ def register_routes(bp):
|
|||
if not soulseek:
|
||||
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
|
||||
|
||||
current_app.logger.info(
|
||||
f"[CancelTrigger:api.public_cancel] download_id={download_id} username={username}"
|
||||
)
|
||||
ok = run_async(soulseek.cancel_download(download_id, username, remove=True))
|
||||
if ok:
|
||||
return api_success({"message": "Download cancelled."})
|
||||
|
|
|
|||
|
|
@ -368,7 +368,6 @@ def serialize_similar_artist(obj, fields: Optional[Set[str]] = None) -> dict:
|
|||
"source_artist_id": d.get("source_artist_id"),
|
||||
"similar_artist_spotify_id": d.get("similar_artist_spotify_id"),
|
||||
"similar_artist_itunes_id": d.get("similar_artist_itunes_id"),
|
||||
"similar_artist_musicbrainz_id": d.get("similar_artist_musicbrainz_id"),
|
||||
"similar_artist_name": d.get("similar_artist_name"),
|
||||
"similarity_rank": d.get("similarity_rank"),
|
||||
"occurrence_count": d.get("occurrence_count"),
|
||||
|
|
|
|||
|
|
@ -44,8 +44,7 @@
|
|||
},
|
||||
"metadata_enhancement": {
|
||||
"enabled": true,
|
||||
"embed_album_art": true,
|
||||
"single_to_album": false
|
||||
"embed_album_art": true
|
||||
},
|
||||
"file_organization": {
|
||||
"enabled": true,
|
||||
|
|
@ -57,11 +56,6 @@
|
|||
"playlist_path": "$playlist/$artist - $title"
|
||||
}
|
||||
},
|
||||
"import": {
|
||||
"staging_path": "./Staging",
|
||||
"replace_lower_quality": false,
|
||||
"folder_artist_override": true
|
||||
},
|
||||
"lossy_copy": {
|
||||
"enabled": false,
|
||||
"bitrate": "320",
|
||||
|
|
@ -73,4 +67,4 @@
|
|||
"listenbrainz": {
|
||||
"token": "LISTENBRAINZ_TOKEN"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -66,12 +66,6 @@ class ConfigManager:
|
|||
|
||||
self._load_config()
|
||||
|
||||
# Placeholder shipped to the browser in place of a configured secret
|
||||
# (#832 follow-up). The settings UI shows it as masked dots; if it's
|
||||
# round-tripped back on save, ``set()`` treats it as "keep existing" so the
|
||||
# real value is never overwritten by the mask.
|
||||
REDACTED_SENTINEL = '__redacted_unchanged__'
|
||||
|
||||
# Dot-notation paths to sensitive config values that must be encrypted at rest.
|
||||
# Paths pointing to dicts encrypt the entire dict as a JSON blob.
|
||||
_SENSITIVE_PATHS = frozenset({
|
||||
|
|
@ -93,10 +87,6 @@ class ConfigManager:
|
|||
'soulseek.api_key',
|
||||
'deezer_download.arl',
|
||||
'lidarr_download.api_key',
|
||||
'prowlarr.api_key',
|
||||
'torrent_client.password',
|
||||
'usenet_client.api_key',
|
||||
'usenet_client.password',
|
||||
# Enrichment services
|
||||
'listenbrainz.token',
|
||||
'acoustid.api_key',
|
||||
|
|
@ -476,71 +466,14 @@ class ConfigManager:
|
|||
"download_path": "./downloads",
|
||||
"transfer_path": "./Transfer",
|
||||
"max_peer_queue": 0,
|
||||
"download_timeout": 600,
|
||||
# Reddit report (YeloMelo95, Bell Canada): the existing
|
||||
# 35-per-220s sliding-window cap allows all 35 searches in
|
||||
# rapid succession before throttling — that burst trips ISP
|
||||
# anti-abuse. This knob forces a min gap between consecutive
|
||||
# searches even when the window cap isn't hit. 0 = disabled
|
||||
# (preserves prior behavior).
|
||||
"search_min_delay_seconds": 0,
|
||||
"download_timeout": 600
|
||||
},
|
||||
"download_source": {
|
||||
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid", "torrent", "usenet"
|
||||
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid"
|
||||
"hybrid_primary": "soulseek", # Legacy: primary source for hybrid mode
|
||||
"hybrid_secondary": "youtube", # Legacy: fallback source for hybrid mode
|
||||
"hybrid_order": [], # Ordered list of sources for hybrid mode (overrides primary/secondary)
|
||||
"stream_source": "youtube", # Options: "youtube" (instant, default), "active" (use download source; falls back to youtube if soulseek)
|
||||
# Album-bundle (torrent / usenet single-source) poll tuning.
|
||||
# Downloader is polled every N seconds until the release
|
||||
# lands; whole job aborts at the timeout. Defaults match
|
||||
# the previous hard-coded constants. Users on slow private
|
||||
# trackers / large box sets can extend the timeout without
|
||||
# editing source.
|
||||
"album_bundle_poll_interval_seconds": 2.0,
|
||||
"album_bundle_timeout_seconds": 6 * 60 * 60, # 6 hours
|
||||
# Stalled-torrent handling (noldevin): abandon a torrent that
|
||||
# makes zero download progress for this long (dead magnet
|
||||
# stuck on "downloading metadata", no seeders) instead of
|
||||
# holding the worker for the full album timeout. 0 disables.
|
||||
"torrent_stall_timeout_seconds": 10 * 60, # 10 minutes
|
||||
# What to do when a torrent stalls: "abandon" (remove it +
|
||||
# its partial data, fail the download so the next source can
|
||||
# try) or "pause" (pause in the client, leave for the user).
|
||||
"torrent_stall_action": "abandon",
|
||||
# Where THIS container can read completed torrent/usenet
|
||||
# downloads (#857). The downloader (qBit/SAB) reports a save
|
||||
# path from inside ITS OWN container — often a category folder
|
||||
# like /data/downloads/music — which may be mounted at a
|
||||
# different point here. Set these to the in-container path(s)
|
||||
# where SoulSync sees those finished downloads; the resolver
|
||||
# then finds the release by name under them. Empty = fall back
|
||||
# to the soulseek download/transfer dirs (the shared-volume
|
||||
# default). See core.download_plugins.album_bundle.resolve_reported_save_path.
|
||||
"torrent_download_path": "",
|
||||
"usenet_download_path": "",
|
||||
# Explicit remote→local prefix mappings for non-shared / oddly
|
||||
# mounted layouts (Sonarr/Radarr "Remote Path Mapping" style):
|
||||
# a list of {"from": "<client path>", "to": "<soulsync path>"}.
|
||||
# Tried before the basename fallback above.
|
||||
"usenet_path_mappings": [],
|
||||
},
|
||||
"post_processing": {
|
||||
# When a download is quarantined (AcoustID mismatch, integrity /
|
||||
# duration failure), retry the next-best candidate instead of
|
||||
# failing outright. Default ON (PR #801's documented default —
|
||||
# the monitor reads this with inline default True; this template
|
||||
# said False, so fresh installs silently shipped with the retry
|
||||
# engine off while existing configs got it on. CI caught the
|
||||
# split: its fresh default config failed all 7 requeue tests).
|
||||
"retry_next_candidate_on_mismatch": True,
|
||||
# Opt-in exhaustive retry: budget retries PER SOURCE so every
|
||||
# source (Soulseek, then HiFi/Tidal/…) gets its own attempts
|
||||
# before the track gives up. Default off (single global cap).
|
||||
"retry_exhaustive": False,
|
||||
# Retries per search query per source in exhaustive mode. The
|
||||
# per-source budget is query_count × this value.
|
||||
"retries_per_query": 5,
|
||||
},
|
||||
"tidal_download": {
|
||||
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
|
||||
|
|
@ -579,37 +512,6 @@ class ConfigManager:
|
|||
"quality_profile": "Any",
|
||||
"cleanup_after_import": True,
|
||||
},
|
||||
# Prowlarr — indexer aggregator. Feeds the torrent / usenet
|
||||
# download plugins. Not a standalone source.
|
||||
"prowlarr": {
|
||||
"url": "",
|
||||
"api_key": "",
|
||||
# Comma-separated list of indexer IDs to limit searches to.
|
||||
# Empty = search all enabled indexers.
|
||||
"indexer_ids": "",
|
||||
},
|
||||
# Torrent client — receives .torrent / magnet URIs from the
|
||||
# torrent download plugin. ``type`` picks which adapter to
|
||||
# instantiate (qbittorrent | transmission | deluge).
|
||||
"torrent_client": {
|
||||
"type": "qbittorrent",
|
||||
"url": "",
|
||||
"username": "",
|
||||
"password": "",
|
||||
"category": "soulsync",
|
||||
"save_path": "",
|
||||
},
|
||||
# Usenet client — receives .nzb URLs / payloads. ``type``
|
||||
# picks the adapter (sabnzbd | nzbget). SABnzbd uses an
|
||||
# API key; NZBGet uses username + password.
|
||||
"usenet_client": {
|
||||
"type": "sabnzbd",
|
||||
"url": "",
|
||||
"api_key": "",
|
||||
"username": "",
|
||||
"password": "",
|
||||
"category": "soulsync",
|
||||
},
|
||||
"soundcloud_download": {
|
||||
# Anonymous-only for now — SoundCloud Go+ OAuth tier could be
|
||||
# added later, with credentials living under a "session" subkey
|
||||
|
|
@ -643,43 +545,16 @@ class ConfigManager:
|
|||
"path": os.environ.get('DATABASE_PATH', 'database/music_library.db'),
|
||||
"max_workers": 5
|
||||
},
|
||||
"image_cache": {
|
||||
"enabled": True,
|
||||
"path": "storage/image_cache",
|
||||
"ttl_seconds": 2592000,
|
||||
"failed_ttl_seconds": 21600,
|
||||
"max_download_mb": 15
|
||||
},
|
||||
"metadata_enhancement": {
|
||||
"enabled": True,
|
||||
"embed_album_art": True,
|
||||
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"],
|
||||
# Ordered preferred cover-art sources (empty = use the
|
||||
# download's own art, i.e. today's behavior). Resolved + walked
|
||||
# with fallback by core/metadata/art_sources.py.
|
||||
"album_art_order": [],
|
||||
# Minimum cover-art resolution (shortest side, px). A preferred
|
||||
# source whose art is smaller is skipped so the next source is
|
||||
# tried — stops a low-res Cover Art Archive upload from winning.
|
||||
# 0 disables the size gate.
|
||||
"min_art_size": 1000,
|
||||
# When a track matches a SINGLE release, look up the parent ALBUM
|
||||
# that contains it and tag it as that album, so it groups with its
|
||||
# album-mates and gets the album cover (not the single's). Off by
|
||||
# default — it's an extra per-import metadata lookup.
|
||||
"single_to_album": False
|
||||
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"]
|
||||
},
|
||||
"musicbrainz": {
|
||||
"embed_tags": True
|
||||
},
|
||||
"playlist_sync": {
|
||||
"create_backup": True,
|
||||
# How a re-sync writes to the server playlist:
|
||||
# replace — delete + recreate (default; today's behavior)
|
||||
# reconcile — edit in place (add/remove delta), preserving the
|
||||
# playlist's custom image, description, and identity (#792)
|
||||
# append — only add new tracks, never remove
|
||||
"mode": "replace"
|
||||
"create_backup": True
|
||||
},
|
||||
"settings": {
|
||||
"audio_quality": "flac"
|
||||
|
|
@ -705,35 +580,12 @@ class ConfigManager:
|
|||
},
|
||||
"import": {
|
||||
"staging_path": "./Staging",
|
||||
# Master toggle for quality-filtering on import. On by default:
|
||||
# downloaded files that don't meet the quality profile are
|
||||
# quarantined instead of imported (same gate the download
|
||||
# pipeline uses). Off → import everything regardless of quality;
|
||||
# the library Quality Upgrade Scanner still flags them.
|
||||
"quality_filter_enabled": True,
|
||||
"replace_lower_quality": False,
|
||||
# Use the top Staging folder as the artist (Artist/Album layouts,
|
||||
# mixtapes). On by default to preserve the long-standing import
|
||||
# behaviour for existing users. Turn OFF if you stage a mixed pile
|
||||
# of songs under one container folder, otherwise that folder's name
|
||||
# overrides every metadata-identified artist (the "soulsync" case).
|
||||
"folder_artist_override": True
|
||||
"replace_lower_quality": False
|
||||
},
|
||||
"m3u_export": {
|
||||
"enabled": False,
|
||||
"entry_base_path": ""
|
||||
},
|
||||
"playlists": {
|
||||
# Where "Organize by playlist" materializes playlist folders.
|
||||
# MUST be a separate root from the music library so the media
|
||||
# server (and the maintenance jobs) never scan it — otherwise the
|
||||
# same track would show up twice. Mapped separately for Docker.
|
||||
"materialize_path": "./Playlists",
|
||||
# "symlink" (relative links, ~zero disk) or "copy" (real
|
||||
# duplicates for FAT/USB/DAPs that can't follow links). Symlink
|
||||
# auto-falls back to copy when the filesystem can't link.
|
||||
"materialize_mode": "symlink"
|
||||
},
|
||||
"youtube": {
|
||||
"cookies_browser": "", # "", "chrome", "firefox", "edge", "brave", "opera", "safari"
|
||||
"download_delay": 3, # seconds between sequential downloads
|
||||
|
|
@ -842,40 +694,7 @@ class ConfigManager:
|
|||
|
||||
return value
|
||||
|
||||
def redacted_config(self) -> Dict[str, Any]:
|
||||
"""Deep copy of the live config with every sensitive value masked.
|
||||
|
||||
Used for ``GET /api/settings`` so decrypted secrets never reach the
|
||||
browser (#832 follow-up). A *set* secret becomes ``REDACTED_SENTINEL``
|
||||
(the UI renders it as masked dots); an unset one stays empty so the UI
|
||||
can show "not configured". Dict-valued secrets (OAuth sessions) collapse
|
||||
to the sentinel too — the UI has no field for them anyway. The matching
|
||||
guard in ``set()`` turns a round-tripped sentinel back into a no-op.
|
||||
"""
|
||||
import copy
|
||||
data = copy.deepcopy(self.config_data)
|
||||
for path in self._SENSITIVE_PATHS:
|
||||
keys = path.split('.')
|
||||
parent = data
|
||||
for k in keys[:-1]:
|
||||
if isinstance(parent, dict) and k in parent:
|
||||
parent = parent[k]
|
||||
else:
|
||||
parent = None
|
||||
break
|
||||
if not isinstance(parent, dict):
|
||||
continue
|
||||
leaf = keys[-1]
|
||||
if leaf in parent and parent[leaf] not in (None, '', {}, [], 0, False):
|
||||
parent[leaf] = self.REDACTED_SENTINEL
|
||||
return data
|
||||
|
||||
def set(self, key: str, value: Any):
|
||||
# The UI round-trips REDACTED_SENTINEL for any secret the user didn't
|
||||
# touch — never let the mask overwrite the real value (#832 follow-up).
|
||||
if value == self.REDACTED_SENTINEL and key in self._SENSITIVE_PATHS:
|
||||
return
|
||||
|
||||
keys = key.split('.')
|
||||
config = self.config_data
|
||||
|
||||
|
|
@ -887,20 +706,6 @@ class ConfigManager:
|
|||
config[keys[-1]] = value
|
||||
self._save_config()
|
||||
|
||||
def resolve_secret(self, key: str, posted: Any) -> str:
|
||||
"""Resolve a secret value coming back from the settings UI.
|
||||
|
||||
The UI renders a saved-but-untouched secret as the REDACTED_SENTINEL (shown
|
||||
masked); empty or that sentinel means "use the stored value", while a real
|
||||
string is a genuine new secret. A connection-test endpoint should test the
|
||||
EFFECTIVE secret, not the mask — otherwise testing a saved-but-untouched
|
||||
token sends the sentinel and the source rejects it (#870)."""
|
||||
if isinstance(posted, str):
|
||||
posted = posted.strip()
|
||||
if not posted or posted == self.REDACTED_SENTINEL:
|
||||
return self.get(key, '') or ''
|
||||
return posted
|
||||
|
||||
def get_spotify_config(self) -> Dict[str, str]:
|
||||
return self.get('spotify', {})
|
||||
|
||||
|
|
|
|||
|
|
@ -282,9 +282,8 @@ class AcoustIDClient:
|
|||
|
||||
def test_api_key(self) -> Tuple[bool, str]:
|
||||
"""
|
||||
Validate the API key with a direct AcoustID lookup call. An invalid key
|
||||
is reported as invalid (error code 4); any other error means the key was
|
||||
accepted.
|
||||
Validate the API key by fingerprinting a real audio file and looking it up.
|
||||
Falls back to a direct API call if no audio files are available.
|
||||
|
||||
Returns:
|
||||
Tuple of (success, message)
|
||||
|
|
@ -295,12 +294,24 @@ class AcoustIDClient:
|
|||
import requests
|
||||
|
||||
try:
|
||||
# Authoritative key check: a direct API lookup with a dummy
|
||||
# fingerprint. AcoustID validates the client key first, so an
|
||||
# invalid key returns error code 4 regardless of the fingerprint.
|
||||
# (The previous real-file path trusted "no exception = valid", but
|
||||
# fingerprint_and_lookup swallows the invalid-key error and returns
|
||||
# None — so it reported broken keys as valid. #756-adjacent.)
|
||||
# Try to find a real audio file to fingerprint for an end-to-end test
|
||||
test_file = self._find_test_audio_file()
|
||||
|
||||
if test_file and CHROMAPRINT_AVAILABLE:
|
||||
logger.info(f"Testing API key with real audio file: {test_file}")
|
||||
try:
|
||||
result = self.fingerprint_and_lookup(test_file)
|
||||
# If we get here without exception, the API key is valid
|
||||
# (invalid keys raise or return error before results)
|
||||
return True, "AcoustID API key is valid"
|
||||
except Exception as e:
|
||||
error_str = str(e).lower()
|
||||
if 'invalid' in error_str and 'api' in error_str:
|
||||
return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
|
||||
# Fingerprint/lookup failed for non-key reasons, fall through to direct test
|
||||
logger.warning(f"Real file test failed ({e}), trying direct API call")
|
||||
|
||||
# Fallback: direct API call with minimal fingerprint
|
||||
url = 'https://api.acoustid.org/v2/lookup'
|
||||
params = {
|
||||
'client': self.api_key,
|
||||
|
|
@ -315,6 +326,7 @@ class AcoustIDClient:
|
|||
if data.get('status') == 'error':
|
||||
error = data.get('error', {})
|
||||
error_code = error.get('code', 0)
|
||||
error_msg = error.get('message', 'Unknown error')
|
||||
|
||||
# Error code 4 is specifically "invalid API key"
|
||||
if error_code == 4:
|
||||
|
|
@ -334,33 +346,33 @@ class AcoustIDClient:
|
|||
logger.error(f"Error testing AcoustID API key: {e}")
|
||||
return False, f"Error: {str(e)}"
|
||||
|
||||
def lookup_with_status(self, audio_file: str) -> Dict[str, Any]:
|
||||
"""Fingerprint + AcoustID lookup returning a STRUCTURED result.
|
||||
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Generate fingerprint and look up recording in AcoustID.
|
||||
|
||||
Unlike fingerprint_and_lookup() (which collapses every outcome into
|
||||
dict-or-None), this distinguishes a genuine no-match from an actual
|
||||
error — an invalid API key, rate limit, missing chromaprint, or a
|
||||
fingerprint failure. That distinction is what lets the UI show "AcoustID
|
||||
Error" (something is broken — fix it) instead of a benign-looking
|
||||
"Skipped" that silently hides a dead key.
|
||||
This is the main method - combines fingerprinting and lookup in one call.
|
||||
|
||||
Returns dict with:
|
||||
'status': 'ok' | 'no_match' | 'error' | 'no_backend'
|
||||
| 'fingerprint_error' | 'unsupported' | 'unavailable'
|
||||
| 'not_found'
|
||||
'recordings': list (meaningful only for 'ok')
|
||||
'best_score': float
|
||||
'recording_mbids': list
|
||||
'error': human-readable detail for any non-'ok' status
|
||||
'invalid_key': bool (True when the API specifically rejected the key)
|
||||
Args:
|
||||
audio_file: Path to the audio file
|
||||
|
||||
Returns:
|
||||
Dict with:
|
||||
'recordings': list of dicts with 'mbid', 'title', 'artist', 'score'
|
||||
'best_score': float (highest score across all results)
|
||||
'recording_mbids': list of unique MBIDs (for backward compat)
|
||||
Or None on error.
|
||||
"""
|
||||
if not ACOUSTID_AVAILABLE:
|
||||
return {'status': 'unavailable', 'recordings': [], 'error': 'pyacoustid library not installed'}
|
||||
logger.debug("Cannot lookup: pyacoustid not available")
|
||||
return None
|
||||
|
||||
if not self.api_key:
|
||||
return {'status': 'unavailable', 'recordings': [], 'error': 'No AcoustID API key configured'}
|
||||
logger.debug("Cannot lookup: no API key")
|
||||
return None
|
||||
|
||||
if not os.path.isfile(audio_file):
|
||||
logger.warning(f"Cannot lookup: file not found: {audio_file}")
|
||||
return {'status': 'not_found', 'recordings': [], 'error': f'File not found: {audio_file}'}
|
||||
return None
|
||||
|
||||
# Check channel count — chromaprint crashes (SIGABRT) on >2 channel files (e.g. 5.1 surround)
|
||||
try:
|
||||
|
|
@ -370,8 +382,7 @@ class AcoustIDClient:
|
|||
channels = getattr(mf.info, 'channels', 2)
|
||||
if channels and channels > 2:
|
||||
logger.warning(f"Skipping AcoustID: file has {channels} channels (surround audio): {audio_file}")
|
||||
return {'status': 'unsupported', 'recordings': [],
|
||||
'error': f'{channels}-channel (surround) audio not supported by chromaprint'}
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"Could not check channel count, proceeding anyway: {e}")
|
||||
|
||||
|
|
@ -381,12 +392,17 @@ class AcoustIDClient:
|
|||
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "NOT SET"
|
||||
logger.info(f"Fingerprinting and looking up: {audio_file} (API key: {api_key_preview})")
|
||||
|
||||
# Use match() which handles fingerprinting + lookup + parsing
|
||||
logger.debug("Running acoustid.match()...")
|
||||
recordings = []
|
||||
seen_mbids = set()
|
||||
best_score = 0.0
|
||||
|
||||
for result in acoustid.match(self.api_key, audio_file, parse=True):
|
||||
for result in acoustid.match(
|
||||
self.api_key,
|
||||
audio_file,
|
||||
parse=True
|
||||
):
|
||||
# match() with parse=True returns (score, recording_id, title, artist)
|
||||
if not isinstance(result, tuple) or len(result) < 2:
|
||||
logger.warning(f"Unexpected result format: {result}")
|
||||
|
|
@ -404,57 +420,45 @@ class AcoustIDClient:
|
|||
|
||||
if recording_id and recording_id not in seen_mbids:
|
||||
seen_mbids.add(recording_id)
|
||||
recordings.append({'mbid': recording_id, 'title': title, 'artist': artist, 'score': score})
|
||||
recordings.append({
|
||||
'mbid': recording_id,
|
||||
'title': title,
|
||||
'artist': artist,
|
||||
'score': score,
|
||||
})
|
||||
logger.debug(f"Found match: {title} by {artist} (MBID: {recording_id}, score: {score})")
|
||||
|
||||
if not recordings:
|
||||
logger.info(f"No AcoustID matches found for: {audio_file}")
|
||||
return {'status': 'no_match', 'recordings': [], 'best_score': best_score,
|
||||
'recording_mbids': [], 'error': 'Track not found in AcoustID database'}
|
||||
return None
|
||||
|
||||
logger.info(f"AcoustID found {len(recordings)} recording(s) (best score: {best_score:.2f})")
|
||||
return {'status': 'ok', 'recordings': recordings, 'best_score': best_score,
|
||||
'recording_mbids': list(seen_mbids)}
|
||||
return {
|
||||
'recordings': recordings,
|
||||
'best_score': best_score,
|
||||
'recording_mbids': list(seen_mbids),
|
||||
}
|
||||
|
||||
except acoustid.NoBackendError:
|
||||
logger.error("Chromaprint library not found and fpcalc not available")
|
||||
return {'status': 'no_backend', 'recordings': [],
|
||||
'error': 'Chromaprint/fpcalc not installed (install libchromaprint1)'}
|
||||
return None
|
||||
except acoustid.FingerprintGenerationError as e:
|
||||
logger.warning(f"Failed to fingerprint {audio_file}: {e}")
|
||||
return {'status': 'fingerprint_error', 'recordings': [], 'error': f'Could not fingerprint file: {e}'}
|
||||
return None
|
||||
except acoustid.WebServiceError as e:
|
||||
# Log more details about the API error
|
||||
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "???"
|
||||
logger.warning(f"AcoustID API error (key: {api_key_preview}): {e}")
|
||||
# Check for common errors
|
||||
error_str = str(e).lower()
|
||||
# Old pyacoustid reports an invalid key as the bare "status: error"
|
||||
# (it drops the detail), so treat that as an invalid-key signal too.
|
||||
invalid = ('invalid' in error_str or 'unknown' in error_str or 'status: error' in error_str)
|
||||
if invalid:
|
||||
logger.error("AcoustID API key appears to be invalid — check your AcoustID settings")
|
||||
if 'invalid' in error_str or 'unknown' in error_str:
|
||||
logger.error("API key appears to be invalid - check your AcoustID settings")
|
||||
elif 'rate' in error_str or 'limit' in error_str:
|
||||
logger.warning("Rate limited by AcoustID — will retry later")
|
||||
return {'status': 'error', 'recordings': [], 'invalid_key': invalid,
|
||||
'error': f'AcoustID API error: {e}'}
|
||||
logger.warning("Rate limited by AcoustID - will retry later")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error in AcoustID lookup: {e}", exc_info=True)
|
||||
return {'status': 'error', 'recordings': [], 'error': f'Unexpected error: {e}'}
|
||||
|
||||
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
|
||||
"""Legacy dict-or-None lookup. Returns the recordings dict on a confirmed
|
||||
match, else None. Kept for callers that only need "did we identify it"
|
||||
(library scanner, auto-import). Callers that must report WHY a lookup
|
||||
didn't match (verification badge, key test) should use
|
||||
``lookup_with_status`` so an error isn't mistaken for a no-match.
|
||||
"""
|
||||
res = self.lookup_with_status(audio_file)
|
||||
if res.get('status') == 'ok':
|
||||
return {
|
||||
'recordings': res['recordings'],
|
||||
'best_score': res.get('best_score', 0.0),
|
||||
'recording_mbids': res.get('recording_mbids', []),
|
||||
}
|
||||
return None
|
||||
return None
|
||||
|
||||
def refresh_config(self):
|
||||
"""Refresh cached config values (call after settings change)."""
|
||||
|
|
|
|||
|
|
@ -16,17 +16,14 @@ from enum import Enum
|
|||
from utils.logging_config import get_logger
|
||||
from core.acoustid_client import AcoustIDClient
|
||||
from core.matching_engine import MusicMatchingEngine
|
||||
from core.matching.version_mismatch import is_acceptable_version_mismatch
|
||||
from core.matching.script_compat import is_cross_script_mismatch
|
||||
from core.musicbrainz_client import MusicBrainzClient
|
||||
|
||||
logger = get_logger("acoustid.verification")
|
||||
|
||||
# Thresholds — single definition lives in the shared core; re-exported here so
|
||||
# existing importers keep working and the values can't drift between paths.
|
||||
from core.matching.audio_verification import ( # noqa: E402
|
||||
MIN_ACOUSTID_SCORE, TITLE_MATCH_THRESHOLD, ARTIST_MATCH_THRESHOLD,
|
||||
)
|
||||
# Thresholds
|
||||
MIN_ACOUSTID_SCORE = 0.80 # Minimum AcoustID fingerprint score to trust
|
||||
TITLE_MATCH_THRESHOLD = 0.70 # Title similarity needed to consider a match
|
||||
ARTIST_MATCH_THRESHOLD = 0.60 # Artist similarity needed to consider a match
|
||||
|
||||
# Single matching-engine instance so version detection reuses the same patterns
|
||||
# used by the pre-download Soulseek matcher (remix / live / acoustic /
|
||||
|
|
@ -52,45 +49,82 @@ class VerificationResult(Enum):
|
|||
"""Possible outcomes of audio verification."""
|
||||
PASS = "pass" # Title/artist match - file is correct
|
||||
FAIL = "fail" # Title/artist mismatch - wrong file downloaded
|
||||
SKIP = "skip" # Genuinely couldn't verify (no match in DB) - continue normally
|
||||
SKIP = "skip" # Could not verify (error or unavailable) - continue normally
|
||||
DISABLED = "disabled" # Verification not enabled
|
||||
ERROR = "error" # Lookup errored (invalid key / rate limit / no backend) - continue, but flag it
|
||||
|
||||
|
||||
# normalize() + similarity() + the alias-aware comparison now live in the shared
|
||||
# decision core (core/matching/audio_verification.py) so import-time verification
|
||||
# and the library scan share ONE definition — the <>-strip fix, CJK handling and
|
||||
# thresholds can't drift apart again. Names kept (`_normalize` etc.) for existing
|
||||
# importers/tests.
|
||||
from core.matching.audio_verification import ( # noqa: E402
|
||||
normalize as _normalize,
|
||||
similarity as _similarity,
|
||||
_alias_aware_artist_sim,
|
||||
_find_best_title_artist_match as _core_find_best_title_artist_match,
|
||||
evaluate as _core_evaluate,
|
||||
Decision as _CoreDecision,
|
||||
)
|
||||
def _normalize(text: str) -> str:
|
||||
"""Normalize a string for comparison: lowercase, strip parentheticals, punctuation."""
|
||||
if not text:
|
||||
return ""
|
||||
s = text.lower().strip()
|
||||
# Remove ALL parenthetical suffixes — these are metadata annotations, not core title
|
||||
# Covers: (Live), (Remastered), (Parody of ...), (from "..." Soundtrack), (feat. ...), etc.
|
||||
s = re.sub(r'\s*\([^)]*\)', '', s)
|
||||
# Remove ALL square bracket suffixes: [Live], [Remastered], [Deluxe], etc.
|
||||
s = re.sub(r'\s*\[[^\]]*\]', '', s)
|
||||
# Remove trailing featuring info not in parentheses: "feat. ...", "ft. ...", "featuring ..."
|
||||
s = re.sub(r'\s+(?:feat\.?|ft\.?|featuring)\s+.*$', '', s, flags=re.IGNORECASE)
|
||||
# Remove dash-separated version tags: "- Vocal", "- Instrumental", "- Acoustic", etc.
|
||||
s = re.sub(r'\s*-\s*(?:vocal|instrumental|acoustic|live|remix|cover|clean|explicit|radio\s*edit|original\s*mix|extended\s*mix|club\s*mix)\s*$', '', s, flags=re.IGNORECASE)
|
||||
# Remove soundtrack/source subtitles: ' - From "..." Soundtrack', ' - from the film ...'
|
||||
s = re.sub(r'\s*-\s*from\s+.+$', '', s, flags=re.IGNORECASE)
|
||||
# Remove non-alphanumeric except spaces
|
||||
s = re.sub(r'[^\w\s]', '', s)
|
||||
# Collapse whitespace
|
||||
s = re.sub(r'\s+', ' ', s).strip()
|
||||
return s
|
||||
|
||||
|
||||
def _find_best_title_artist_match(recordings, expected_title, expected_artist,
|
||||
expected_artist_aliases=None):
|
||||
"""Back-compat wrapper around the shared core matcher (keeps the
|
||||
``expected_artist_aliases`` kwarg name for existing callers/tests)."""
|
||||
return _core_find_best_title_artist_match(
|
||||
recordings, expected_title, expected_artist, expected_artist_aliases,
|
||||
)
|
||||
def _similarity(a: str, b: str) -> float:
|
||||
"""Calculate similarity between two strings (0.0-1.0) after normalization."""
|
||||
na = _normalize(a)
|
||||
nb = _normalize(b)
|
||||
if not na or not nb:
|
||||
return 0.0
|
||||
if na == nb:
|
||||
return 1.0
|
||||
return SequenceMatcher(None, na, nb).ratio()
|
||||
|
||||
|
||||
def _find_best_title_artist_match(
|
||||
recordings: List[Dict[str, Any]],
|
||||
expected_title: str,
|
||||
expected_artist: str,
|
||||
) -> Tuple[Optional[Dict], float, float]:
|
||||
"""
|
||||
Find the AcoustID recording that best matches expected title/artist.
|
||||
|
||||
Returns:
|
||||
(best_recording, title_similarity, artist_similarity)
|
||||
"""
|
||||
best_rec = None
|
||||
best_title_sim = 0.0
|
||||
best_artist_sim = 0.0
|
||||
best_combined = 0.0
|
||||
|
||||
for rec in recordings:
|
||||
title = rec.get('title') or ''
|
||||
artist = rec.get('artist') or ''
|
||||
|
||||
title_sim = _similarity(expected_title, title)
|
||||
artist_sim = _similarity(expected_artist, artist)
|
||||
# Weight title higher since that's the primary identifier
|
||||
combined = (title_sim * 0.6) + (artist_sim * 0.4)
|
||||
|
||||
if combined > best_combined:
|
||||
best_combined = combined
|
||||
best_rec = rec
|
||||
best_title_sim = title_sim
|
||||
best_artist_sim = artist_sim
|
||||
|
||||
return best_rec, best_title_sim, best_artist_sim
|
||||
|
||||
|
||||
# Shared MusicBrainz client for enrichment lookups
|
||||
_mb_client = None
|
||||
_mb_client_lock = threading.Lock()
|
||||
|
||||
# Shared MusicBrainzService for alias lookups (issue #442). Service
|
||||
# layer wraps the raw client + adds caching + DB access — all of which
|
||||
# the alias resolution chain (library DB → cache → live MB) needs.
|
||||
_mb_service = None
|
||||
_mb_service_lock = threading.Lock()
|
||||
|
||||
MAX_MB_ENRICHMENT_LOOKUPS = 3
|
||||
|
||||
|
||||
|
|
@ -104,42 +138,6 @@ def _get_mb_client() -> MusicBrainzClient:
|
|||
return _mb_client
|
||||
|
||||
|
||||
def _get_mb_service():
|
||||
"""Get or create a shared MusicBrainzService instance.
|
||||
|
||||
Used by the alias-resolution chain in `verify_audio_file`. Lazy
|
||||
init so importing this module doesn't trigger a DB connection on
|
||||
paths that never run AcoustID verification (test runs, dry runs).
|
||||
"""
|
||||
global _mb_service
|
||||
if _mb_service is None:
|
||||
with _mb_service_lock:
|
||||
if _mb_service is None:
|
||||
from core.musicbrainz_service import MusicBrainzService
|
||||
from database.music_database import get_database
|
||||
_mb_service = MusicBrainzService(get_database())
|
||||
return _mb_service
|
||||
|
||||
|
||||
def _resolve_expected_artist_aliases(expected_artist_name: str) -> List[str]:
|
||||
"""Look up alternate-spelling aliases for the expected artist.
|
||||
|
||||
Issue #442 — bridges cross-script artist comparisons (Japanese
|
||||
kanji ↔ romanized, Cyrillic ↔ Latin, etc.) without forcing the
|
||||
verifier to know about the resolution chain. Best-effort: any
|
||||
failure (no MB service, network down, no library DB) returns
|
||||
empty list so verification falls back to the prior direct
|
||||
similarity check.
|
||||
"""
|
||||
if not expected_artist_name:
|
||||
return []
|
||||
try:
|
||||
return _get_mb_service().lookup_artist_aliases(expected_artist_name)
|
||||
except Exception as e:
|
||||
logger.debug("alias lookup failed for %r: %s", expected_artist_name, e)
|
||||
return []
|
||||
|
||||
|
||||
def _enrich_recordings_from_musicbrainz(
|
||||
recordings: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
|
|
@ -265,33 +263,18 @@ class AcoustIDVerification:
|
|||
logger.debug(f"AcoustID verification skipped: {reason}")
|
||||
return VerificationResult.SKIP, reason
|
||||
|
||||
# Step 2: Fingerprint and lookup in AcoustID (structured so an
|
||||
# actual error — invalid key / rate limit / no chromaprint — is
|
||||
# reported distinctly from a genuine no-match, instead of both
|
||||
# silently surfacing as "Skipped").
|
||||
# Step 2: Fingerprint and lookup in AcoustID
|
||||
logger.info(f"Fingerprinting and looking up: {audio_file_path}")
|
||||
lookup = self.acoustid_client.lookup_with_status(audio_file_path) or {}
|
||||
status = lookup.get('status')
|
||||
# Infer status by content when absent (a caller/stub that returned
|
||||
# just recordings): recordings => matched, none => no match.
|
||||
if status is None:
|
||||
status = 'ok' if lookup.get('recordings') else 'no_match'
|
||||
acoustid_result = self.acoustid_client.fingerprint_and_lookup(audio_file_path)
|
||||
|
||||
if status in ('error', 'no_backend', 'fingerprint_error', 'unavailable'):
|
||||
# Something is broken (not the track's fault) — never quarantine
|
||||
# on this; surface it so the user can fix it.
|
||||
return VerificationResult.ERROR, lookup.get('error', 'AcoustID lookup failed')
|
||||
if not acoustid_result:
|
||||
return VerificationResult.SKIP, "Track not found in AcoustID database"
|
||||
|
||||
if status != 'ok':
|
||||
# no_match / unsupported / not_found — genuinely could not verify.
|
||||
return VerificationResult.SKIP, lookup.get('error', 'No match in AcoustID database')
|
||||
|
||||
acoustid_result = lookup
|
||||
recordings = acoustid_result.get('recordings', [])
|
||||
best_score = acoustid_result.get('best_score', 0)
|
||||
|
||||
if not recordings:
|
||||
return VerificationResult.SKIP, "No match in AcoustID database"
|
||||
return VerificationResult.SKIP, "AcoustID returned no recordings"
|
||||
|
||||
logger.debug(
|
||||
f"AcoustID returned {len(recordings)} recording(s) "
|
||||
|
|
@ -307,55 +290,187 @@ class AcoustIDVerification:
|
|||
# Enrich recordings that are missing title/artist via MusicBrainz lookup
|
||||
recordings = _enrich_recordings_from_musicbrainz(recordings)
|
||||
|
||||
# Issue #442 — alias resolution is LAZY. We pass a memoising
|
||||
# thunk to the artist-comparison sites; it only fires the
|
||||
# multi-tier lookup (library DB → cache → live MB) when
|
||||
# direct artist similarity falls below threshold. Verifications
|
||||
# where the direct match already passes (the common case for
|
||||
# same-script artist names) never trigger any lookup work,
|
||||
# so the fix doesn't add a per-verification DB query for the
|
||||
# happy path. When the thunk DOES fire, the result is cached
|
||||
# in the closure so the 3 comparison sites within one
|
||||
# verification share a single resolution pass.
|
||||
_alias_cache: Dict[str, Any] = {}
|
||||
|
||||
def _aliases_provider() -> List[str]:
|
||||
if 'value' not in _alias_cache:
|
||||
resolved = _resolve_expected_artist_aliases(expected_artist_name)
|
||||
_alias_cache['value'] = resolved
|
||||
if resolved:
|
||||
logger.debug(
|
||||
"Resolved %d aliases for expected artist '%s'",
|
||||
len(resolved), expected_artist_name,
|
||||
)
|
||||
return _alias_cache['value']
|
||||
|
||||
# Steps 4-5: delegate the PASS/SKIP/FAIL decision to the shared core
|
||||
# (core/matching/audio_verification.evaluate) so import verification
|
||||
# and the library scan apply identical logic.
|
||||
outcome = _core_evaluate(
|
||||
expected_track_name, expected_artist_name, recordings,
|
||||
fingerprint_score=best_score,
|
||||
aliases_provider=_aliases_provider,
|
||||
# Step 4: Find best title/artist match among AcoustID results
|
||||
best_rec, title_sim, artist_sim = _find_best_title_artist_match(
|
||||
recordings, expected_track_name, expected_artist_name
|
||||
)
|
||||
|
||||
if not best_rec:
|
||||
return VerificationResult.SKIP, "No recordings with title/artist info"
|
||||
|
||||
matched_title = best_rec.get('title', '?')
|
||||
matched_artist = best_rec.get('artist', '?')
|
||||
|
||||
logger.info(
|
||||
"Best match: '%s' by '%s' (title_sim=%.2f, artist_sim=%.2f) -> %s",
|
||||
outcome.matched_title, outcome.matched_artist,
|
||||
outcome.title_sim, outcome.artist_sim, outcome.decision.value,
|
||||
f"Best match: '{matched_title}' by '{matched_artist}' "
|
||||
f"(title_sim={title_sim:.2f}, artist_sim={artist_sim:.2f})"
|
||||
)
|
||||
_decision_map = {
|
||||
_CoreDecision.PASS: VerificationResult.PASS,
|
||||
_CoreDecision.SKIP: VerificationResult.SKIP,
|
||||
_CoreDecision.FAIL: VerificationResult.FAIL,
|
||||
}
|
||||
result = _decision_map[outcome.decision]
|
||||
if result == VerificationResult.PASS:
|
||||
logger.info("AcoustID verification PASSED - %s", outcome.reason)
|
||||
elif result == VerificationResult.FAIL:
|
||||
logger.warning("AcoustID verification FAILED - %s", outcome.reason)
|
||||
else:
|
||||
logger.info("AcoustID verification SKIPPED - %s", outcome.reason)
|
||||
return result, outcome.reason
|
||||
|
||||
# Step 4b: Version-mismatch gate.
|
||||
#
|
||||
# The ``_normalize`` step deliberately strips parentheticals and
|
||||
# version tags ("(Instrumental)", "- Live", etc) so that legit
|
||||
# name variations don't fail the title-similarity comparison.
|
||||
# That same stripping made it impossible to tell a vocal track
|
||||
# apart from its instrumental: "In My Feelings" and "In My
|
||||
# Feelings (Instrumental)" both normalize to "in my feelings",
|
||||
# the title sim ends up 1.0, and the file passes verification
|
||||
# even though it's the wrong cut.
|
||||
#
|
||||
# Detect the version on each side BEFORE normalization runs.
|
||||
# If the expected track and the AcoustID-matched recording
|
||||
# disagree on version (one is original, the other is
|
||||
# instrumental / live / remix / acoustic / etc), reject — the
|
||||
# fingerprint identified a real song but it's not the one the
|
||||
# caller asked for.
|
||||
expected_version = _detect_title_version(expected_track_name)
|
||||
matched_version = _detect_title_version(matched_title)
|
||||
if expected_version != matched_version:
|
||||
msg = (
|
||||
f"Version mismatch: expected '{expected_track_name}' ({expected_version}) "
|
||||
f"but file is '{matched_title}' ({matched_version})"
|
||||
)
|
||||
logger.warning(f"AcoustID verification FAILED (version mismatch) - {msg}")
|
||||
return VerificationResult.FAIL, msg
|
||||
|
||||
# Step 5: Decide pass/fail based on similarity
|
||||
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim >= ARTIST_MATCH_THRESHOLD:
|
||||
msg = (
|
||||
f"Audio verified: '{matched_title}' by '{matched_artist}' "
|
||||
f"matches expected '{expected_track_name}' by '{expected_artist_name}' "
|
||||
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
|
||||
)
|
||||
logger.info(f"AcoustID verification PASSED - {msg}")
|
||||
return VerificationResult.PASS, msg
|
||||
|
||||
# Title matches but artist doesn't — could be a cover/collab OR a
|
||||
# genuinely different track with the same name. Distinguish the
|
||||
# two by checking whether the expected artist appears anywhere in
|
||||
# AcoustID's returned recordings.
|
||||
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim < ARTIST_MATCH_THRESHOLD:
|
||||
# First: if the expected artist is present in ANY recording's
|
||||
# metadata for this fingerprint, it's likely the right track
|
||||
# (AcoustID's "best" match just picked the wrong variant).
|
||||
for rec in recordings:
|
||||
if _similarity(expected_artist_name, rec.get('artist', '')) >= ARTIST_MATCH_THRESHOLD:
|
||||
msg = (
|
||||
f"Audio verified: found '{expected_track_name}' by '{expected_artist_name}' "
|
||||
f"in AcoustID results"
|
||||
)
|
||||
logger.info(f"AcoustID verification PASSED (secondary match) - {msg}")
|
||||
return VerificationResult.PASS, msg
|
||||
|
||||
# Expected artist wasn't found anywhere. Decide between:
|
||||
# - FAIL: clear mismatch, e.g. "Tom Walker" (sim ~0.2) when
|
||||
# expecting "Maduk" — different song with same name
|
||||
# - SKIP: ambiguous, e.g. collab / alt credit / formatting
|
||||
# difference (sim 0.3-0.6)
|
||||
#
|
||||
# The 0.3 cutoff catches hard mismatches while preserving the
|
||||
# benefit of the doubt for borderline artist formatting.
|
||||
CLEAR_MISMATCH_THRESHOLD = 0.3
|
||||
if artist_sim < CLEAR_MISMATCH_THRESHOLD:
|
||||
msg = (
|
||||
f"Audio mismatch: file identified as '{matched_title}' by '{matched_artist}', "
|
||||
f"expected '{expected_track_name}' by '{expected_artist_name}' "
|
||||
f"(title={title_sim:.0%}, artist={artist_sim:.0%}) — "
|
||||
f"expected artist not found in any AcoustID recording"
|
||||
)
|
||||
logger.warning(f"AcoustID verification FAILED (clear artist mismatch) - {msg}")
|
||||
return VerificationResult.FAIL, msg
|
||||
|
||||
msg = (
|
||||
f"Title matches but artist unclear: "
|
||||
f"AcoustID='{matched_title}' by '{matched_artist}', "
|
||||
f"expected '{expected_track_name}' by '{expected_artist_name}' "
|
||||
f"(artist_sim={artist_sim:.0%} — ambiguous, could be cover/collab)"
|
||||
)
|
||||
logger.info(f"AcoustID verification SKIPPED - {msg}")
|
||||
return VerificationResult.SKIP, msg
|
||||
|
||||
# Title doesn't match — check ALL recordings for any title/artist match
|
||||
# (the best combined match might not be the right one if there are many results)
|
||||
# Skip recordings whose version (instrumental/live/etc) disagrees with
|
||||
# what the caller asked for — the version mismatch above checked
|
||||
# only the best recording, but a wrong-version variant could still
|
||||
# win this fallback scan if its bare title matched.
|
||||
for rec in recordings:
|
||||
t = rec.get('title') or ''
|
||||
a = rec.get('artist') or ''
|
||||
if _detect_title_version(t) != expected_version:
|
||||
continue
|
||||
if (_similarity(expected_track_name, t) >= TITLE_MATCH_THRESHOLD and
|
||||
_similarity(expected_artist_name, a) >= ARTIST_MATCH_THRESHOLD):
|
||||
msg = (
|
||||
f"Audio verified: found '{t}' by '{a}' in AcoustID results "
|
||||
f"matching expected '{expected_track_name}' by '{expected_artist_name}'"
|
||||
)
|
||||
logger.info(f"AcoustID verification PASSED (scan match) - {msg}")
|
||||
return VerificationResult.PASS, msg
|
||||
|
||||
# No match found — but if fingerprint score is very high (≥0.95)
|
||||
# AND we have evidence the mismatch is a language/script case
|
||||
# (rather than two genuinely different songs by the same artist),
|
||||
# skip rather than quarantine a correct file. Two routes:
|
||||
#
|
||||
# (a) Either side of the comparison contains non-ASCII characters
|
||||
# — strong signal of transliteration / kanji↔roman cases.
|
||||
# Artist must still be a strong match to use this path.
|
||||
# (b) Both title AND artist similarity are very high (the song
|
||||
# is recognizably the same with minor punctuation / casing
|
||||
# differences that fell below the strict match thresholds).
|
||||
#
|
||||
# The OLD logic was ``title_sim >= 0.55 OR artist_sim >= match``.
|
||||
# That fired for English-vs-English songs by the same artist that
|
||||
# share NO actual content — e.g. "R.O.T.C (Interlude)" by
|
||||
# Kendrick Lamar getting accepted as "Rich (Interlude)" by
|
||||
# Kendrick Lamar because the artist matched perfectly and
|
||||
# "interlude" was shared in both titles. Reported by user when
|
||||
# downloading Mr. Morale: three tracks (Rich Interlude, Savior
|
||||
# Interlude, Savior) all received the wrong R.O.T.C audio file
|
||||
# because of this leak.
|
||||
top = recordings[0]
|
||||
top_title = top.get('title', '?') or ''
|
||||
top_artist = top.get('artist', '?') or ''
|
||||
has_non_ascii = (
|
||||
any(ord(c) > 127 for c in (expected_track_name or ''))
|
||||
or any(ord(c) > 127 for c in top_title)
|
||||
)
|
||||
language_script_skip = (
|
||||
best_score >= 0.95
|
||||
and has_non_ascii
|
||||
and artist_sim >= ARTIST_MATCH_THRESHOLD
|
||||
)
|
||||
high_confidence_strong_match_skip = (
|
||||
best_score >= 0.95
|
||||
and title_sim >= 0.80
|
||||
and artist_sim >= ARTIST_MATCH_THRESHOLD
|
||||
)
|
||||
if language_script_skip or high_confidence_strong_match_skip:
|
||||
reason = (
|
||||
"likely same song in different language/script"
|
||||
if language_script_skip
|
||||
else "title/artist match within tolerance"
|
||||
)
|
||||
msg = (
|
||||
f"Title/artist mismatch but fingerprint confidence very high ({best_score:.2f}): "
|
||||
f"AcoustID='{top_title}' by '{top_artist}', "
|
||||
f"expected '{expected_track_name}' by '{expected_artist_name}' — "
|
||||
f"{reason}"
|
||||
)
|
||||
logger.info(f"AcoustID verification SKIPPED (high confidence) - {msg}")
|
||||
return VerificationResult.SKIP, msg
|
||||
|
||||
# Low fingerprint score + no metadata match — file is likely wrong
|
||||
# `top`, `top_title`, `top_artist` already resolved above for the
|
||||
# skip-eligibility check.
|
||||
msg = (
|
||||
f"Audio mismatch: file identified as '{top_title}' by '{top_artist}', "
|
||||
f"expected '{expected_track_name}' by '{expected_artist_name}' "
|
||||
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
|
||||
)
|
||||
logger.warning(f"AcoustID verification FAILED - {msg}")
|
||||
return VerificationResult.FAIL, msg
|
||||
|
||||
except Exception as e:
|
||||
# Any unexpected error -> SKIP (fail open)
|
||||
|
|
|
|||
|
|
@ -1,796 +0,0 @@
|
|||
"""Amazon Music metadata client backed by a T2Tunes proxy instance.
|
||||
|
||||
T2Tunes exposes the Amazon Music catalog (search, album metadata, stream info)
|
||||
through a simple REST API. This module wraps those endpoints and normalises the
|
||||
responses into the same Track / Artist / Album dataclass shape used by
|
||||
DeezerClient and iTunesClient.
|
||||
|
||||
Endpoints used:
|
||||
GET /api/status
|
||||
GET /api/amazon-music/search
|
||||
GET /api/amazon-music/metadata
|
||||
GET /api/amazon-music/media-from-asin
|
||||
|
||||
Config keys (all optional — fall back to public defaults):
|
||||
amazon.base_url T2Tunes instance URL (default: https://t2tunes.site)
|
||||
amazon.country ISO-3166 country code (default: US)
|
||||
amazon.preferred_codec Preferred audio codec (default: flac)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, Iterator, List, Optional
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import requests
|
||||
|
||||
from config.settings import config_manager
|
||||
from core.api_call_tracker import api_call_tracker
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("amazon_client")
|
||||
|
||||
# Strips featuring credits like "Artist feat. X", "Artist ft. Y" so artist
|
||||
# deduplication works on the primary artist name only.
|
||||
_FEAT_RE = re.compile(r'\s+(?:feat(?:uring)?\.?|ft\.?)\s+.*', re.IGNORECASE)
|
||||
|
||||
# Strips the Explicit marker — explicit is treated as the default version.
|
||||
# Clean/Edited/Censored stay in the name so users can distinguish them.
|
||||
_EDITION_RE = re.compile(r'\s*[\[\(]explicit[\]\)]', re.IGNORECASE)
|
||||
|
||||
|
||||
def _primary_artist(name: str) -> str:
|
||||
return _FEAT_RE.sub('', name).strip()
|
||||
|
||||
|
||||
def _strip_edition(name: str) -> str:
|
||||
return _EDITION_RE.sub('', name).strip()
|
||||
|
||||
|
||||
def _unslugify(name: str) -> str:
|
||||
"""Convert a slug-form artist ID (e.g. 'kendrick_lamar') to a search name."""
|
||||
return name.replace('_', ' ')
|
||||
|
||||
|
||||
DEFAULT_BASE_URL = "https://t2tunes.site"
|
||||
DEFAULT_COUNTRY = "US"
|
||||
DEFAULT_CODEC = "flac"
|
||||
MIN_API_INTERVAL = 0.5 # seconds — T2Tunes has no published rate limit
|
||||
|
||||
_last_api_call: float = 0.0
|
||||
_api_call_lock = threading.Lock()
|
||||
|
||||
_META_CACHE_TTL = 300 # seconds
|
||||
_meta_cache: Dict[str, tuple] = {} # asin -> (fetched_at, meta_dict)
|
||||
_meta_cache_lock = threading.Lock()
|
||||
|
||||
|
||||
class AmazonClientError(RuntimeError):
|
||||
"""Raised on unrecoverable T2Tunes API errors.
|
||||
|
||||
Carries the HTTP ``status_code`` when the failure was an HTTP error, so
|
||||
callers (the worker's outage detection) can tell a source outage (5xx) from
|
||||
a per-item miss without parsing the message.
|
||||
"""
|
||||
|
||||
def __init__(self, *args, status_code=None):
|
||||
super().__init__(*args)
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dataclasses — field layout matches DeezerClient / iTunesClient exactly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class Track:
|
||||
id: str # Amazon track ASIN
|
||||
name: str
|
||||
artists: List[str]
|
||||
album: str
|
||||
duration_ms: int
|
||||
popularity: int
|
||||
preview_url: Optional[str] = None
|
||||
external_urls: Optional[Dict[str, str]] = None
|
||||
image_url: Optional[str] = None
|
||||
release_date: Optional[str] = None
|
||||
track_number: Optional[int] = None
|
||||
disc_number: Optional[int] = None
|
||||
album_type: Optional[str] = None
|
||||
total_tracks: Optional[int] = None
|
||||
isrc: Optional[str] = None
|
||||
|
||||
@classmethod
|
||||
def from_search_hit(cls, doc: Dict[str, Any]) -> "Track":
|
||||
return cls(
|
||||
id=str(doc.get("asin") or ""),
|
||||
name=str(doc.get("title") or ""),
|
||||
artists=[str(doc.get("artistName") or "Unknown Artist")],
|
||||
album=str(doc.get("albumName") or ""),
|
||||
duration_ms=int(doc.get("duration") or 0) * 1000,
|
||||
popularity=0,
|
||||
isrc=str(doc.get("isrc") or "") or None,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_stream_info(
|
||||
cls,
|
||||
stream: "T2TunesStreamInfo",
|
||||
album_meta: Optional[Dict[str, Any]] = None,
|
||||
) -> "Track":
|
||||
album = album_meta or {}
|
||||
return cls(
|
||||
id=stream.asin,
|
||||
name=stream.title,
|
||||
artists=[stream.artist] if stream.artist else ["Unknown Artist"],
|
||||
album=stream.album,
|
||||
duration_ms=0,
|
||||
popularity=0,
|
||||
image_url=album.get("image"),
|
||||
release_date=album.get("release_date"),
|
||||
total_tracks=album.get("trackCount"),
|
||||
isrc=stream.isrc or None,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Artist:
|
||||
id: str # Slugified artist name — T2Tunes exposes no artist IDs
|
||||
name: str
|
||||
popularity: int
|
||||
genres: List[str]
|
||||
followers: int
|
||||
image_url: Optional[str] = None
|
||||
external_urls: Optional[Dict[str, str]] = None
|
||||
|
||||
@classmethod
|
||||
def from_name(cls, name: str) -> "Artist":
|
||||
slug = name.lower().replace(" ", "_")
|
||||
return cls(id=slug, name=name, popularity=0, genres=[], followers=0)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Album:
|
||||
id: str # Amazon album ASIN
|
||||
name: str
|
||||
artists: List[str]
|
||||
release_date: str
|
||||
total_tracks: int
|
||||
album_type: str
|
||||
image_url: Optional[str] = None
|
||||
external_urls: Optional[Dict[str, str]] = None
|
||||
explicit: Optional[bool] = None
|
||||
|
||||
@classmethod
|
||||
def from_search_hit(cls, doc: Dict[str, Any]) -> "Album":
|
||||
return cls(
|
||||
id=str(doc.get("albumAsin") or doc.get("asin") or ""),
|
||||
name=str(doc.get("albumName") or doc.get("title") or ""),
|
||||
artists=[str(doc.get("artistName") or "Unknown Artist")],
|
||||
release_date="",
|
||||
total_tracks=0,
|
||||
album_type="album",
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_metadata(cls, album_meta: Dict[str, Any], asin: str = "") -> "Album":
|
||||
return cls(
|
||||
id=str(album_meta.get("asin") or asin or ""),
|
||||
name=str(album_meta.get("title") or ""),
|
||||
artists=[str(album_meta.get("artistName") or "Unknown Artist")],
|
||||
release_date=str(album_meta.get("release_date") or album_meta.get("releaseDate") or ""),
|
||||
total_tracks=int(album_meta.get("trackCount") or 0),
|
||||
album_type="album",
|
||||
image_url=album_meta.get("image"),
|
||||
explicit=album_meta.get("explicit"),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal dataclasses for raw T2Tunes response parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class T2TunesSearchItem:
|
||||
asin: str
|
||||
title: str
|
||||
artist_name: str
|
||||
item_type: str
|
||||
album_name: str = ""
|
||||
album_asin: str = ""
|
||||
duration_seconds: int = 0
|
||||
isrc: str = ""
|
||||
|
||||
@property
|
||||
def is_album(self) -> bool:
|
||||
return "album" in self.item_type.lower()
|
||||
|
||||
@property
|
||||
def is_track(self) -> bool:
|
||||
return "track" in self.item_type.lower()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class T2TunesStreamInfo:
|
||||
asin: str
|
||||
streamable: bool
|
||||
codec: str
|
||||
format: str
|
||||
sample_rate: Optional[int]
|
||||
stream_url: str
|
||||
decryption_key: Optional[str] # hex-encoded AES key; None when stream is clear
|
||||
title: str = ""
|
||||
artist: str = ""
|
||||
album: str = ""
|
||||
isrc: str = ""
|
||||
cover_url: str = ""
|
||||
track_number: Optional[int] = None
|
||||
disc_number: Optional[int] = None
|
||||
genre: str = ""
|
||||
label: str = ""
|
||||
date: str = ""
|
||||
|
||||
@property
|
||||
def has_decryption_key(self) -> bool:
|
||||
return bool(self.decryption_key)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rate-limit enforcement
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _rate_limit() -> None:
|
||||
global _last_api_call
|
||||
with _api_call_lock:
|
||||
elapsed = time.monotonic() - _last_api_call
|
||||
if elapsed < MIN_API_INTERVAL:
|
||||
time.sleep(MIN_API_INTERVAL - elapsed)
|
||||
_last_api_call = time.monotonic()
|
||||
api_call_tracker.record_call("amazon")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main client
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class AmazonClient:
|
||||
"""T2Tunes-backed Amazon Music metadata and stream-info client."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: Optional[str] = None,
|
||||
country: Optional[str] = None,
|
||||
preferred_codec: Optional[str] = None,
|
||||
timeout: int = 30,
|
||||
session: Optional[Any] = None,
|
||||
) -> None:
|
||||
self.base_url = (base_url or config_manager.get("amazon.base_url", DEFAULT_BASE_URL)).rstrip("/")
|
||||
self.country = (country or config_manager.get("amazon.country", DEFAULT_COUNTRY)).upper()
|
||||
self.preferred_codec = (
|
||||
preferred_codec or config_manager.get("amazon.preferred_codec", DEFAULT_CODEC)
|
||||
).lower()
|
||||
self.timeout = timeout
|
||||
self.session: Any = session or requests.Session()
|
||||
if isinstance(self.session, requests.Session):
|
||||
self.session.headers.update({
|
||||
"Accept": "application/json",
|
||||
"User-Agent": "SoulSync/1.0",
|
||||
"Referer": self.base_url,
|
||||
})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def reload_config(self) -> None:
|
||||
self.base_url = config_manager.get("amazon.base_url", DEFAULT_BASE_URL).rstrip("/")
|
||||
self.country = config_manager.get("amazon.country", DEFAULT_COUNTRY).upper()
|
||||
self.preferred_codec = config_manager.get("amazon.preferred_codec", DEFAULT_CODEC).lower()
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
"""Return True when the T2Tunes instance reports Amazon Music as up."""
|
||||
try:
|
||||
return str(self.status().get("amazonMusic", "")).lower() == "up"
|
||||
except AmazonClientError:
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Low-level API wrappers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def status(self) -> Dict[str, Any]:
|
||||
return self._get_json("/api/status")
|
||||
|
||||
def search_raw(self, query: str, *, types: str = "track,album") -> List[T2TunesSearchItem]:
|
||||
data = self._get_json(
|
||||
"/api/amazon-music/search",
|
||||
params={"query": query, "types": types, "country": self.country},
|
||||
)
|
||||
return list(self._iter_search_items(data))
|
||||
|
||||
def album_metadata(self, asin: str) -> Dict[str, Any]:
|
||||
return self._get_json(
|
||||
"/api/amazon-music/metadata",
|
||||
params={"asin": asin, "country": self.country},
|
||||
)
|
||||
|
||||
def media_from_asin(self, asin: str, codec: Optional[str] = None) -> List[T2TunesStreamInfo]:
|
||||
effective_codec = (codec or self.preferred_codec).lower()
|
||||
data = self._get_json(
|
||||
"/api/amazon-music/media-from-asin",
|
||||
params={"asin": asin, "country": self.country, "codec": effective_codec},
|
||||
)
|
||||
if isinstance(data, list):
|
||||
return [self._parse_stream_info(item) for item in data if isinstance(item, dict)]
|
||||
if isinstance(data, dict):
|
||||
return [self._parse_stream_info(data)]
|
||||
raise AmazonClientError(f"Unexpected media-from-asin response type: {type(data).__name__}")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Metadata interface — mirrors DeezerClient / iTunesClient signatures
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
|
||||
_rate_limit()
|
||||
items = self.search_raw(query, types="track")
|
||||
track_pairs: List[tuple] = [] # (Track, album_asin)
|
||||
seen_album_asins: List[str] = []
|
||||
for item in items:
|
||||
if not item.is_track:
|
||||
continue
|
||||
track = Track.from_search_hit({
|
||||
"asin": item.asin,
|
||||
"title": _strip_edition(item.title),
|
||||
"artistName": _primary_artist(item.artist_name),
|
||||
"albumName": _strip_edition(item.album_name),
|
||||
"albumAsin": item.album_asin,
|
||||
"duration": item.duration_seconds,
|
||||
"isrc": item.isrc,
|
||||
})
|
||||
track_pairs.append((track, item.album_asin))
|
||||
if item.album_asin and item.album_asin not in seen_album_asins:
|
||||
seen_album_asins.append(item.album_asin)
|
||||
if len(track_pairs) >= limit:
|
||||
break
|
||||
album_metas = self._fetch_album_metas(seen_album_asins[:5])
|
||||
tracks: List[Track] = []
|
||||
for track, album_asin in track_pairs:
|
||||
if album_asin and album_asin in album_metas:
|
||||
meta = album_metas[album_asin]
|
||||
track.image_url = meta.get("image")
|
||||
track.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
|
||||
track.total_tracks = meta.get("trackCount")
|
||||
tracks.append(track)
|
||||
return tracks
|
||||
|
||||
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
|
||||
_rate_limit()
|
||||
items = self.search_raw(query, types="track")
|
||||
seen: Dict[str, Artist] = {}
|
||||
artist_album_asin: Dict[str, str] = {} # artist name → first album ASIN seen
|
||||
for item in items:
|
||||
name = _primary_artist(item.artist_name)
|
||||
if not name:
|
||||
continue
|
||||
if name not in seen:
|
||||
seen[name] = Artist.from_name(name)
|
||||
if name not in artist_album_asin and item.album_asin:
|
||||
artist_album_asin[name] = item.album_asin
|
||||
if len(seen) >= limit:
|
||||
break
|
||||
# T2Tunes has no artist images — use an album cover as stand-in.
|
||||
unique_asins = list({v for v in artist_album_asin.values()})[:5]
|
||||
album_metas = self._fetch_album_metas(unique_asins)
|
||||
for name, artist in seen.items():
|
||||
asin = artist_album_asin.get(name)
|
||||
if asin and asin in album_metas:
|
||||
artist.image_url = album_metas[asin].get("image")
|
||||
return list(seen.values())
|
||||
|
||||
def search_albums(self, query: str, limit: int = 20) -> List[Album]:
|
||||
_rate_limit()
|
||||
items = self.search_raw(query, types="track")
|
||||
album_candidates: List[tuple] = [] # (Album, asin)
|
||||
seen_keys: set = set()
|
||||
for item in items:
|
||||
album_asin = item.album_asin
|
||||
raw_name = item.album_name
|
||||
if not raw_name:
|
||||
continue
|
||||
display_name = _strip_edition(raw_name)
|
||||
artist = _primary_artist(item.artist_name)
|
||||
# Collapse Explicit/Clean variants: same normalised name + artist = same album
|
||||
dedup_key = (display_name.lower(), artist.lower())
|
||||
if dedup_key in seen_keys:
|
||||
continue
|
||||
seen_keys.add(dedup_key)
|
||||
album = Album.from_search_hit({
|
||||
"albumAsin": album_asin,
|
||||
"albumName": display_name,
|
||||
"artistName": artist,
|
||||
})
|
||||
album_candidates.append((album, album_asin))
|
||||
if len(album_candidates) >= limit:
|
||||
break
|
||||
album_metas = self._fetch_album_metas([a for _, a in album_candidates[:10]])
|
||||
albums: List[Album] = []
|
||||
for album, asin in album_candidates:
|
||||
if asin in album_metas:
|
||||
meta = album_metas[asin]
|
||||
album.image_url = meta.get("image")
|
||||
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
|
||||
album.total_tracks = int(meta.get("trackCount") or 0)
|
||||
albums.append(album)
|
||||
return albums
|
||||
|
||||
def get_track_details(self, asin: str) -> Optional[Dict[str, Any]]:
|
||||
"""Return a Spotify-compatible dict for a single track ASIN."""
|
||||
_rate_limit()
|
||||
try:
|
||||
streams = self.media_from_asin(asin)
|
||||
except AmazonClientError:
|
||||
return None
|
||||
if not streams:
|
||||
return None
|
||||
s = streams[0]
|
||||
|
||||
album_data: Dict[str, Any] = {}
|
||||
try:
|
||||
meta = self.album_metadata(asin)
|
||||
albums = meta.get("albumList")
|
||||
if isinstance(albums, list) and albums and isinstance(albums[0], dict):
|
||||
album_data = albums[0]
|
||||
except AmazonClientError:
|
||||
pass
|
||||
|
||||
return {
|
||||
"id": s.asin,
|
||||
"name": _strip_edition(s.title),
|
||||
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
|
||||
"album": {
|
||||
"id": album_data.get("asin", ""),
|
||||
"name": _strip_edition(s.album),
|
||||
"images": [{"url": album_data["image"]}] if album_data.get("image") else [],
|
||||
"release_date": album_data.get("release_date") or album_data.get("releaseDate") or s.date or "",
|
||||
"total_tracks": album_data.get("trackCount", 0),
|
||||
},
|
||||
"duration_ms": 0,
|
||||
"popularity": 0,
|
||||
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
|
||||
"track_number": s.track_number,
|
||||
"disc_number": s.disc_number,
|
||||
"isrc": s.isrc,
|
||||
"is_album_track": True,
|
||||
"raw_data": {
|
||||
"codec": s.codec,
|
||||
"format": s.format,
|
||||
"sample_rate": s.sample_rate,
|
||||
"streamable": s.streamable,
|
||||
"has_decryption_key": s.has_decryption_key,
|
||||
},
|
||||
}
|
||||
|
||||
def get_album(self, asin: str, include_tracks: bool = True) -> Optional[Dict[str, Any]]:
|
||||
"""Return a Spotify-compatible album dict."""
|
||||
_rate_limit()
|
||||
try:
|
||||
meta = self.album_metadata(asin)
|
||||
except AmazonClientError:
|
||||
return None
|
||||
|
||||
albums = meta.get("albumList")
|
||||
if not isinstance(albums, list) or not albums:
|
||||
return None
|
||||
album = albums[0] if isinstance(albums[0], dict) else {}
|
||||
|
||||
result: Dict[str, Any] = {
|
||||
"id": asin,
|
||||
"name": _strip_edition(album.get("title", "")),
|
||||
"artists": [{"name": _primary_artist(album.get("artistName", "")), "id": ""}],
|
||||
"release_date": album.get("release_date") or album.get("releaseDate") or "",
|
||||
"total_tracks": album.get("trackCount", 0),
|
||||
"album_type": "album",
|
||||
"images": [{"url": album["image"]}] if album.get("image") else [],
|
||||
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
|
||||
"label": album.get("label", ""),
|
||||
}
|
||||
if include_tracks:
|
||||
tracks_data = self.get_album_tracks(asin) or {
|
||||
"items": [], "total": 0, "limit": 50, "next": None,
|
||||
}
|
||||
result["tracks"] = tracks_data
|
||||
# Backfill release_date from stream tags when album metadata lacks it.
|
||||
if not result["release_date"]:
|
||||
items = tracks_data.get("items") or []
|
||||
for item in items:
|
||||
rd = item.get("release_date") or ""
|
||||
if rd and len(rd) >= 4:
|
||||
result["release_date"] = rd
|
||||
break
|
||||
return result
|
||||
|
||||
def get_album_tracks(self, asin: str) -> Optional[Dict[str, Any]]:
|
||||
"""Return album tracks in Spotify pagination format."""
|
||||
_rate_limit()
|
||||
try:
|
||||
streams = self.media_from_asin(asin)
|
||||
except AmazonClientError:
|
||||
return None
|
||||
|
||||
# media_from_asin has no duration — enrich from search results which do.
|
||||
duration_map: Dict[str, int] = {} # track asin → duration_ms
|
||||
if streams:
|
||||
album_name = _strip_edition(streams[0].album)
|
||||
artist_name = _primary_artist(streams[0].artist)
|
||||
try:
|
||||
search_items = self.search_raw(
|
||||
f"{album_name} {artist_name}", types="track"
|
||||
)
|
||||
for item in search_items:
|
||||
if item.album_asin == asin and item.duration_seconds:
|
||||
duration_map[item.asin] = item.duration_seconds * 1000
|
||||
except Exception as exc:
|
||||
logger.debug("Duration backfill failed for ASIN %s: %s", asin, exc)
|
||||
|
||||
items = [
|
||||
{
|
||||
"id": s.asin,
|
||||
"name": _strip_edition(s.title),
|
||||
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
|
||||
"duration_ms": duration_map.get(s.asin, 0),
|
||||
"track_number": s.track_number if s.track_number is not None else idx + 1,
|
||||
"disc_number": s.disc_number,
|
||||
"release_date": s.date or "",
|
||||
"isrc": s.isrc,
|
||||
}
|
||||
for idx, s in enumerate(streams)
|
||||
]
|
||||
return {"items": items, "total": len(items), "limit": 50, "next": None}
|
||||
|
||||
def get_artist(self, artist_name: str) -> Optional[Dict[str, Any]]:
|
||||
"""Return a Spotify-compatible artist dict inferred from search results."""
|
||||
_rate_limit()
|
||||
search_name = _unslugify(artist_name)
|
||||
try:
|
||||
items = self.search_raw(search_name, types="track")
|
||||
except AmazonClientError:
|
||||
return None
|
||||
name_lower = search_name.lower()
|
||||
match = next(
|
||||
(i for i in items if _primary_artist(i.artist_name).lower() == name_lower),
|
||||
next((i for i in items if name_lower in _primary_artist(i.artist_name).lower()), None),
|
||||
)
|
||||
if not match:
|
||||
return None
|
||||
return {
|
||||
"id": match.artist_name.lower().replace(" ", "_"),
|
||||
"name": match.artist_name,
|
||||
"genres": [],
|
||||
"popularity": 0,
|
||||
"followers": {"total": 0},
|
||||
"images": [],
|
||||
"external_urls": {},
|
||||
}
|
||||
|
||||
def get_artist_albums(
|
||||
self,
|
||||
artist_name: str,
|
||||
album_type: str = "album,single",
|
||||
limit: int = 200,
|
||||
) -> List[Album]:
|
||||
"""Return albums for an artist inferred from search results."""
|
||||
_rate_limit()
|
||||
search_name = _unslugify(artist_name)
|
||||
try:
|
||||
items = self.search_raw(f"{search_name} album", types="track")
|
||||
except AmazonClientError:
|
||||
return []
|
||||
album_candidates: List[tuple] = [] # (Album, asin)
|
||||
seen_asins: set = set()
|
||||
name_lower = search_name.lower()
|
||||
for item in items:
|
||||
if not item.is_album:
|
||||
continue
|
||||
if _primary_artist(item.artist_name).lower() != name_lower:
|
||||
continue
|
||||
album_asin = item.album_asin or item.asin
|
||||
if album_asin in seen_asins:
|
||||
continue
|
||||
seen_asins.add(album_asin)
|
||||
album = Album.from_search_hit({
|
||||
"albumAsin": album_asin,
|
||||
"albumName": _strip_edition(item.album_name or item.title),
|
||||
"artistName": _primary_artist(item.artist_name),
|
||||
})
|
||||
album_candidates.append((album, album_asin))
|
||||
if len(album_candidates) >= limit:
|
||||
break
|
||||
|
||||
# Fetch metadata for art, release_date, track_count, and type inference.
|
||||
# No explicit cap here — search_raw naturally bounds results to ~20 items,
|
||||
# so album_candidates is already small.
|
||||
asins_to_fetch = [a for _, a in album_candidates]
|
||||
metas = self._fetch_album_metas(asins_to_fetch)
|
||||
|
||||
albums: List[Album] = []
|
||||
for album, asin in album_candidates:
|
||||
meta = metas.get(asin, {})
|
||||
if meta:
|
||||
album.image_url = meta.get("image")
|
||||
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
|
||||
total = int(meta.get("trackCount") or 0)
|
||||
album.total_tracks = total
|
||||
# T2Tunes doesn't expose release type — infer from track count.
|
||||
# 1-track releases are singles; keep default "album" otherwise.
|
||||
if total == 1:
|
||||
album.album_type = "single"
|
||||
albums.append(album)
|
||||
return albums
|
||||
|
||||
def get_track_features(self, track_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Not available from Amazon Music — returns None for compatibility."""
|
||||
return None
|
||||
|
||||
def _get_artist_image_from_albums(self, artist_id: str) -> Optional[str]:
|
||||
"""Return an album cover as artist image stand-in (T2Tunes has no artist images)."""
|
||||
search_name = _unslugify(artist_id)
|
||||
try:
|
||||
items = self.search_raw(search_name, types="track")
|
||||
except AmazonClientError:
|
||||
return None
|
||||
name_lower = search_name.lower()
|
||||
for item in items:
|
||||
if _primary_artist(item.artist_name).lower() != name_lower:
|
||||
continue
|
||||
asin = item.album_asin or item.asin
|
||||
if not asin:
|
||||
continue
|
||||
metas = self._fetch_album_metas([asin])
|
||||
if asin in metas and metas[asin].get("image"):
|
||||
return metas[asin]["image"]
|
||||
return None
|
||||
|
||||
# ==================== Interface Aliases (match DeezerClient method names) ====================
|
||||
get_album_metadata = get_album
|
||||
get_artist_info = get_artist
|
||||
get_artist_albums_list = get_artist_albums
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _fetch_album_metas(self, asins: List[str]) -> Dict[str, Dict[str, Any]]:
|
||||
"""Parallel-fetch album metadata for up to N ASINs. Returns {asin: albumList[0]}."""
|
||||
if not asins:
|
||||
return {}
|
||||
metas: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
def _fetch(asin: str) -> None:
|
||||
now = time.monotonic()
|
||||
with _meta_cache_lock:
|
||||
entry = _meta_cache.get(asin)
|
||||
if entry and (now - entry[0]) < _META_CACHE_TTL:
|
||||
metas[asin] = entry[1]
|
||||
return
|
||||
_rate_limit()
|
||||
try:
|
||||
raw = self.album_metadata(asin)
|
||||
lst = raw.get("albumList")
|
||||
if isinstance(lst, list) and lst and isinstance(lst[0], dict):
|
||||
meta = lst[0]
|
||||
with _meta_cache_lock:
|
||||
_meta_cache[asin] = (time.monotonic(), meta)
|
||||
metas[asin] = meta
|
||||
except Exception as exc:
|
||||
logger.debug("Album metadata fetch failed for ASIN %s: %s", asin, exc)
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
with ThreadPoolExecutor(max_workers=min(len(asins), 5)) as pool:
|
||||
list(pool.map(_fetch, asins))
|
||||
return metas
|
||||
|
||||
def _get_json(self, path: str, params: Optional[Dict[str, Any]] = None) -> Any:
|
||||
url = urljoin(f"{self.base_url}/", path.lstrip("/"))
|
||||
last_exc: Optional[Exception] = None
|
||||
for attempt in range(3):
|
||||
if attempt:
|
||||
time.sleep(1.0 * attempt)
|
||||
try:
|
||||
resp = self.session.get(url, params=params, timeout=self.timeout)
|
||||
resp.raise_for_status()
|
||||
except requests.HTTPError as exc:
|
||||
body = exc.response.text[:500].replace("\n", " ")
|
||||
# T2Tunes returns 400 for transient Amazon-side failures — retry those.
|
||||
if exc.response.status_code == 400 and "Failed to search" in exc.response.text:
|
||||
logger.debug("T2Tunes transient 400 on attempt %d, retrying: %s", attempt + 1, url)
|
||||
last_exc = AmazonClientError(
|
||||
f"HTTP {exc.response.status_code} for {url} — body: {body!r}"
|
||||
)
|
||||
continue
|
||||
raise AmazonClientError(
|
||||
f"HTTP {exc.response.status_code} for {url} — body: {body!r}",
|
||||
status_code=exc.response.status_code,
|
||||
) from exc
|
||||
except requests.RequestException as exc:
|
||||
raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
|
||||
try:
|
||||
return resp.json()
|
||||
except ValueError as exc:
|
||||
preview = resp.text[:200].replace("\n", " ")
|
||||
raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
|
||||
raise last_exc # type: ignore[misc]
|
||||
|
||||
@staticmethod
|
||||
def _iter_search_items(response: Any) -> Iterator[T2TunesSearchItem]:
|
||||
if not isinstance(response, dict):
|
||||
raise AmazonClientError(
|
||||
f"Unexpected search response type: {type(response).__name__}"
|
||||
)
|
||||
for result in response.get("results") or []:
|
||||
if not isinstance(result, dict):
|
||||
continue
|
||||
for hit in result.get("hits") or []:
|
||||
if not isinstance(hit, dict):
|
||||
continue
|
||||
doc = hit.get("document")
|
||||
if not isinstance(doc, dict):
|
||||
continue
|
||||
asin = str(doc.get("asin") or "")
|
||||
if not asin:
|
||||
continue
|
||||
yield T2TunesSearchItem(
|
||||
asin=asin,
|
||||
title=str(doc.get("title") or ""),
|
||||
artist_name=str(doc.get("artistName") or ""),
|
||||
item_type=str(doc.get("__type") or ""),
|
||||
album_name=str(doc.get("albumName") or ""),
|
||||
album_asin=str(doc.get("albumAsin") or ""),
|
||||
duration_seconds=int(doc.get("duration") or 0),
|
||||
isrc=str(doc.get("isrc") or ""),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_stream_info(item: Dict[str, Any]) -> T2TunesStreamInfo:
|
||||
stream_info = item.get("streamInfo") if isinstance(item.get("streamInfo"), dict) else {}
|
||||
tags = item.get("tags") if isinstance(item.get("tags"), dict) else {}
|
||||
# T2Tunes API has a typo: "stremeable" in some responses
|
||||
streamable = item.get("streamable")
|
||||
if streamable is None:
|
||||
streamable = item.get("stremeable")
|
||||
raw_key = item.get("decryptionKey")
|
||||
decryption_key = str(raw_key) if raw_key else None
|
||||
|
||||
def _int_tag(key: str) -> Optional[int]:
|
||||
v = tags.get(key)
|
||||
try:
|
||||
return int(v) if v is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
return T2TunesStreamInfo(
|
||||
asin=str(item.get("asin") or ""),
|
||||
streamable=bool(streamable),
|
||||
codec=str(stream_info.get("codec") or ""),
|
||||
format=str(stream_info.get("format") or ""),
|
||||
sample_rate=(
|
||||
stream_info.get("sampleRate")
|
||||
if isinstance(stream_info.get("sampleRate"), int)
|
||||
else None
|
||||
),
|
||||
stream_url=str(stream_info.get("streamUrl") or ""),
|
||||
decryption_key=decryption_key,
|
||||
title=str(tags.get("title") or ""),
|
||||
artist=str(tags.get("artist") or ""),
|
||||
album=str(tags.get("album") or ""),
|
||||
isrc=str(tags.get("isrc") or ""),
|
||||
cover_url=str(item.get("coverUrl") or ""),
|
||||
track_number=_int_tag("trackNumber"),
|
||||
disc_number=_int_tag("discNumber"),
|
||||
genre=str(tags.get("genre") or ""),
|
||||
label=str(tags.get("label") or ""),
|
||||
date=str(tags.get("date") or ""),
|
||||
)
|
||||
|
|
@ -1,466 +0,0 @@
|
|||
"""Amazon Music download source plugin backed by a T2Tunes proxy.
|
||||
|
||||
NOT yet wired into the app registry — validated in isolation only.
|
||||
See tests/tools/test_amazon_download_client.py.
|
||||
|
||||
Filename encoding (the DownloadSourcePlugin dispatch contract):
|
||||
"{asin}||{display_name}"
|
||||
e.g. "B09XYZ1234||Kendrick Lamar - Not Like Us"
|
||||
|
||||
Codec preference order: FLAC → Opus → EAC3.
|
||||
|
||||
Download flow (from Tubifarry reference implementation):
|
||||
1. GET stream_url → encrypted bytes on disk
|
||||
2. FFmpeg -decryption_key <hex> to decrypt in place
|
||||
3. Embed metadata tags (handled by the app's standard post-processing)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
import requests as http_requests
|
||||
|
||||
from config.settings import config_manager
|
||||
from core.amazon_client import AmazonClient, AmazonClientError
|
||||
from core.download_plugins.base import DownloadSourcePlugin
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.quality.model import AudioQuality
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("amazon_download_client")
|
||||
|
||||
# Quality / codec helpers
|
||||
CODEC_PREFERENCE = ["flac", "opus", "eac3"]
|
||||
|
||||
_CODEC_EXTENSIONS: Dict[str, str] = {
|
||||
"flac": "flac",
|
||||
"ogg_vorbis": "ogg",
|
||||
"opus": "opus",
|
||||
"eac3": "eac3",
|
||||
"mp4": "m4a",
|
||||
"aac": "m4a",
|
||||
"mp3": "mp3",
|
||||
}
|
||||
|
||||
MIN_AUDIO_BYTES = 512 * 1024 # 512 KB — anything smaller is a broken stream
|
||||
|
||||
|
||||
def _codec_key(codec: str) -> str:
|
||||
return codec.lower().replace("-", "_").replace(" ", "_")
|
||||
|
||||
|
||||
def _file_extension(codec: str) -> str:
|
||||
return _CODEC_EXTENSIONS.get(_codec_key(codec), "bin")
|
||||
|
||||
|
||||
def _quality_label(codec: str, sample_rate: Optional[int] = None) -> str:
|
||||
ck = _codec_key(codec)
|
||||
if ck == "flac":
|
||||
if sample_rate and sample_rate > 48000:
|
||||
return "Hi-Res"
|
||||
return "Lossless"
|
||||
return "Lossy"
|
||||
|
||||
|
||||
class AmazonDownloadClient(DownloadSourcePlugin):
|
||||
"""DownloadSourcePlugin — Amazon Music via T2Tunes proxy."""
|
||||
|
||||
def __init__(self, download_path: Optional[str] = None) -> None:
|
||||
if download_path is None:
|
||||
download_path = config_manager.get("soulseek.download_path", "./downloads")
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
|
||||
self._quality = quality_tier_for_source("amazon", default="flac")
|
||||
self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
|
||||
|
||||
self._client = AmazonClient(preferred_codec=self._quality)
|
||||
self.session = http_requests.Session()
|
||||
self.session.headers.update({
|
||||
"User-Agent": "SoulSync/1.0",
|
||||
"Accept": "*/*",
|
||||
})
|
||||
|
||||
self._engine: Any = None
|
||||
self.shutdown_check: Any = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# DownloadSourcePlugin — lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def set_engine(self, engine) -> None:
|
||||
"""Engine callback — wires the central thread worker + state store."""
|
||||
self._engine = engine
|
||||
|
||||
def set_shutdown_check(self, check_callable) -> None:
|
||||
self.shutdown_check = check_callable
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
# T2Tunes has a public default instance; no credentials required.
|
||||
# Return True unconditionally so the source shows as available.
|
||||
return True
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
try:
|
||||
return self._client.is_authenticated()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# DownloadSourcePlugin — search
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query: str,
|
||||
timeout: Optional[int] = None,
|
||||
progress_callback: Any = None,
|
||||
) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
try:
|
||||
items = self._client.search_raw(query, types="track,album")
|
||||
except AmazonClientError as exc:
|
||||
logger.warning(f"Amazon search failed for {query!r}: {exc}")
|
||||
return [], []
|
||||
|
||||
track_results: List[TrackResult] = []
|
||||
album_map: Dict[str, AlbumResult] = {}
|
||||
album_order: List[str] = []
|
||||
preferred = self._client.preferred_codec
|
||||
# Search results only carry the codec (real sample_rate arrives at
|
||||
# stream time). Claim the format honestly — FLAC for the lossless
|
||||
# codec, lossy otherwise — so audio_quality derives a real format
|
||||
# instead of the display label ("Lossless"), and the post-download
|
||||
# probe pins the actual sample_rate/bit_depth.
|
||||
amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
|
||||
|
||||
for item in items:
|
||||
quality = _quality_label(preferred)
|
||||
if item.is_track:
|
||||
tr = TrackResult(
|
||||
username="amazon",
|
||||
filename=f"{item.asin}||{item.artist_name} - {item.title}",
|
||||
size=0,
|
||||
bitrate=None,
|
||||
duration=item.duration_seconds * 1000 if item.duration_seconds else None,
|
||||
quality=quality,
|
||||
free_upload_slots=999,
|
||||
upload_speed=999_999,
|
||||
queue_length=0,
|
||||
artist=item.artist_name,
|
||||
title=item.title,
|
||||
album=item.album_name,
|
||||
_source_metadata={
|
||||
"asin": item.asin,
|
||||
"album_asin": item.album_asin,
|
||||
"isrc": item.isrc,
|
||||
},
|
||||
)
|
||||
tr.set_quality(amazon_q)
|
||||
track_results.append(tr)
|
||||
elif item.is_album:
|
||||
album_asin = item.album_asin or item.asin
|
||||
if album_asin not in album_map:
|
||||
placeholder = TrackResult(
|
||||
username="amazon",
|
||||
filename=f"{item.asin}||{item.artist_name} - {item.title}",
|
||||
size=0,
|
||||
bitrate=None,
|
||||
duration=None,
|
||||
quality=quality,
|
||||
free_upload_slots=999,
|
||||
upload_speed=999_999,
|
||||
queue_length=0,
|
||||
artist=item.artist_name,
|
||||
title=item.title,
|
||||
album=item.album_name,
|
||||
)
|
||||
placeholder.set_quality(amazon_q)
|
||||
album_map[album_asin] = AlbumResult(
|
||||
username="amazon",
|
||||
album_path=album_asin,
|
||||
album_title=item.album_name or item.title,
|
||||
artist=item.artist_name,
|
||||
track_count=0,
|
||||
total_size=0,
|
||||
tracks=[placeholder],
|
||||
dominant_quality=quality,
|
||||
)
|
||||
album_order.append(album_asin)
|
||||
|
||||
return track_results, [album_map[k] for k in album_order]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# DownloadSourcePlugin — download dispatch
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def download(
|
||||
self,
|
||||
username: str,
|
||||
filename: str,
|
||||
file_size: int = 0,
|
||||
) -> Optional[str]:
|
||||
if "||" not in filename:
|
||||
logger.error(f"Invalid Amazon filename format (no '||'): {filename!r}")
|
||||
return None
|
||||
if self._engine is None:
|
||||
raise RuntimeError(
|
||||
"AmazonDownloadClient._engine is not set — "
|
||||
"client not connected to download infrastructure"
|
||||
)
|
||||
asin, display_name = filename.split("||", 1)
|
||||
asin = asin.strip()
|
||||
display_name = display_name.strip()
|
||||
return self._engine.worker.dispatch(
|
||||
source_name="amazon",
|
||||
target_id=asin,
|
||||
display_name=display_name,
|
||||
original_filename=filename,
|
||||
impl_callable=self._download_sync,
|
||||
)
|
||||
|
||||
def _download_sync(
|
||||
self,
|
||||
download_id: str,
|
||||
target_id: str,
|
||||
display_name: str,
|
||||
) -> Optional[str]:
|
||||
asin = str(target_id)
|
||||
codecs = CODEC_PREFERENCE if self._allow_fallback else [self._quality]
|
||||
for codec in codecs:
|
||||
try:
|
||||
streams = self._client.media_from_asin(asin, codec=codec)
|
||||
except AmazonClientError as exc:
|
||||
logger.warning(f"media_from_asin({asin!r}, {codec}) failed: {exc}")
|
||||
continue
|
||||
|
||||
stream = next(
|
||||
(s for s in streams if s.streamable and s.stream_url),
|
||||
None,
|
||||
)
|
||||
if not stream:
|
||||
logger.debug(f"No streamable result for {asin} at codec={codec}")
|
||||
continue
|
||||
|
||||
ext = _file_extension(stream.codec or codec)
|
||||
safe = "".join(
|
||||
ch if ch.isalnum() or ch in " -_." else "_"
|
||||
for ch in display_name
|
||||
)[:80]
|
||||
# T2Tunes always serves audio in an encrypted MP4 container.
|
||||
# The codec extension (.flac, .opus, .eac3) is only for the
|
||||
# final decrypted output.
|
||||
enc_ext = "mp4" if stream.decryption_key else ext
|
||||
enc_path = self._unique_path(self.download_path / f"{safe}.enc.{enc_ext}")
|
||||
out_path = self._unique_path(self.download_path / f"{safe}.{ext}")
|
||||
|
||||
if self._engine is not None:
|
||||
self._engine.update_record(
|
||||
"amazon", download_id, {"state": "downloading", "progress": 0.0}
|
||||
)
|
||||
try:
|
||||
downloaded = self._stream_to_file(stream.stream_url, enc_path, download_id)
|
||||
except Exception as exc:
|
||||
logger.warning(f"Stream download failed for {asin} ({codec}): {exc}")
|
||||
enc_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
if downloaded < MIN_AUDIO_BYTES:
|
||||
logger.warning(
|
||||
f"File too small ({downloaded} B) for {asin} at {codec} — trying next"
|
||||
)
|
||||
enc_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
if stream.decryption_key:
|
||||
if self._engine is not None:
|
||||
self._engine.update_record(
|
||||
"amazon", download_id, {"state": "decrypting", "progress": 1.0}
|
||||
)
|
||||
try:
|
||||
self._decrypt_with_ffmpeg(enc_path, out_path, stream.decryption_key)
|
||||
enc_path.unlink(missing_ok=True)
|
||||
except Exception as exc:
|
||||
logger.error(f"Decryption failed for {asin} ({codec}): {exc}")
|
||||
enc_path.unlink(missing_ok=True)
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
else:
|
||||
enc_path.rename(out_path)
|
||||
|
||||
final_size = out_path.stat().st_size
|
||||
logger.info(
|
||||
f"Amazon download complete ({codec}): {out_path} "
|
||||
f"({final_size / (1024 * 1024):.1f} MB)"
|
||||
)
|
||||
# Sync size == transferred so the download monitor's bytes-incomplete
|
||||
# guard doesn't block post-processing. The throttled updates in
|
||||
# _stream_to_file leave transferred < size after the last 0.5s tick;
|
||||
# other streaming clients avoid this by not tracking bytes at all
|
||||
# (size stays 0, the guard is skipped). Writing the final output size
|
||||
# here restores parity.
|
||||
if self._engine is not None:
|
||||
self._engine.update_record(
|
||||
"amazon", download_id, {'size': final_size, 'transferred': final_size}
|
||||
)
|
||||
return str(out_path)
|
||||
|
||||
logger.error(f"All codec tiers exhausted for '{display_name}' ({asin})")
|
||||
return None
|
||||
|
||||
def _decrypt_with_ffmpeg(
|
||||
self, enc_path: Path, out_path: Path, hex_key: str
|
||||
) -> None:
|
||||
"""Decrypt a T2Tunes encrypted audio file using FFmpeg -decryption_key.
|
||||
|
||||
T2Tunes uses CENC (Common Encryption) for DRM-protected tracks.
|
||||
FFmpeg supports decryption via the -decryption_key flag when the
|
||||
key is provided in hex.
|
||||
"""
|
||||
ffmpeg = shutil.which("ffmpeg")
|
||||
if not ffmpeg:
|
||||
tools_dir = Path(__file__).parent.parent / "tools"
|
||||
candidate = tools_dir / ("ffmpeg.exe" if os.name == "nt" else "ffmpeg")
|
||||
if candidate.exists():
|
||||
ffmpeg = str(candidate)
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"ffmpeg is required for Amazon Music decryption. "
|
||||
"Install ffmpeg and ensure it is on your PATH."
|
||||
)
|
||||
cmd = [
|
||||
ffmpeg,
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-decryption_key", hex_key,
|
||||
"-i", str(enc_path),
|
||||
"-map", "0:a:0", # extract first audio stream (FLAC/Opus/EAC3 inside MP4)
|
||||
"-c", "copy",
|
||||
str(out_path),
|
||||
]
|
||||
logger.debug(f"Decrypting {enc_path.name} → {out_path.name}")
|
||||
result = subprocess.run(cmd, capture_output=True)
|
||||
if result.returncode != 0:
|
||||
stderr = result.stderr.decode("utf-8", errors="replace").strip()
|
||||
raise RuntimeError(f"FFmpeg decryption failed (exit {result.returncode}): {stderr}")
|
||||
|
||||
def _stream_to_file(self, url: str, out_path: Path, download_id: str) -> int:
|
||||
resp = self.session.get(url, stream=True, timeout=60)
|
||||
resp.raise_for_status()
|
||||
|
||||
total = int(resp.headers.get("content-length", 0))
|
||||
downloaded = 0
|
||||
last_report = time.monotonic()
|
||||
shutdown_triggered = False
|
||||
|
||||
with out_path.open("wb") as fh:
|
||||
for chunk in resp.iter_content(chunk_size=64 * 1024):
|
||||
if not chunk:
|
||||
continue
|
||||
if self.shutdown_check and self.shutdown_check():
|
||||
shutdown_triggered = True
|
||||
break
|
||||
fh.write(chunk)
|
||||
downloaded += len(chunk)
|
||||
now = time.monotonic()
|
||||
if self._engine and now - last_report >= 0.5:
|
||||
self._engine.update_record(
|
||||
"amazon",
|
||||
download_id,
|
||||
{
|
||||
"transferred": downloaded,
|
||||
"size": total,
|
||||
"progress": downloaded / total if total else 0.0,
|
||||
},
|
||||
)
|
||||
last_report = now
|
||||
|
||||
if shutdown_triggered:
|
||||
out_path.unlink(missing_ok=True)
|
||||
raise RuntimeError("Shutdown requested mid-download")
|
||||
|
||||
return downloaded
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# DownloadSourcePlugin — status interface
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def get_all_downloads(self) -> List[DownloadStatus]:
|
||||
if self._engine is None:
|
||||
return []
|
||||
return [
|
||||
self._record_to_status(record)
|
||||
for record in self._engine.iter_records_for_source('amazon')
|
||||
]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
|
||||
if self._engine is None:
|
||||
return None
|
||||
record = self._engine.get_record('amazon', download_id)
|
||||
return self._record_to_status(record) if record is not None else None
|
||||
|
||||
async def cancel_download(
|
||||
self,
|
||||
download_id: str,
|
||||
username: Optional[str] = None,
|
||||
remove: bool = False,
|
||||
) -> bool:
|
||||
if self._engine is None:
|
||||
return False
|
||||
if self._engine.get_record('amazon', download_id) is None:
|
||||
return False
|
||||
self._engine.update_record('amazon', download_id, {'state': 'Cancelled'})
|
||||
if remove:
|
||||
self._engine.remove_record('amazon', download_id)
|
||||
return True
|
||||
|
||||
async def clear_all_completed_downloads(self) -> bool:
|
||||
if self._engine is None:
|
||||
return True
|
||||
terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
|
||||
for record in list(self._engine.iter_records_for_source('amazon')):
|
||||
if record.get('state') in terminal:
|
||||
self._engine.remove_record('amazon', record['id'])
|
||||
return True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _unique_path(path: Path) -> Path:
|
||||
if not path.exists():
|
||||
return path
|
||||
stem, suffix = path.stem, path.suffix
|
||||
for i in range(1, 100):
|
||||
candidate = path.with_name(f"{stem} ({i}){suffix}")
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
return path.with_name(f"{stem}_{uuid.uuid4().hex[:8]}{suffix}")
|
||||
|
||||
@staticmethod
|
||||
def _record_to_status(rec: Dict[str, Any]) -> DownloadStatus:
|
||||
return DownloadStatus(
|
||||
id=str(rec.get('id', '')),
|
||||
filename=str(rec.get('filename', '')),
|
||||
username='amazon',
|
||||
state=str(rec.get('state', 'queued')),
|
||||
progress=float(rec.get('progress', 0.0)),
|
||||
size=int(rec.get('size', 0)),
|
||||
transferred=int(rec.get('transferred', 0)),
|
||||
speed=int(rec.get('speed', 0)),
|
||||
time_remaining=rec.get('time_remaining'),
|
||||
file_path=rec.get('file_path'),
|
||||
)
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
"""Amazon enrichment outage detection + back-off — pure, importable, testable.
|
||||
|
||||
The Amazon worker enriches via a public T2Tunes proxy instance. When that
|
||||
instance is down (HTTP 5xx, "Amazon Music API is not initialized", or an
|
||||
unreachable host), the worker must NOT treat every album as an individual
|
||||
failure: doing so floods the logs with an error per item, churns network + DB
|
||||
continuously, and permanently marks the whole library ``error`` (which the
|
||||
retry tiers never re-attempt) for what is really a transient outage.
|
||||
|
||||
Instead it recognizes "the whole source is down", leaves the item untouched so
|
||||
it's retried once the instance recovers, and backs off hard. These two pure
|
||||
helpers carry that logic so it can be unit-tested without the worker, the DB,
|
||||
or the network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# HTTP statuses that mean "the source/proxy is unhealthy", not "no match".
|
||||
_OUTAGE_STATUS = {500, 502, 503, 504}
|
||||
|
||||
# Substrings (lower-cased) in an error message that indicate a source outage
|
||||
# rather than a per-item miss: proxy not ready, gateway errors, the host being
|
||||
# unreachable, or an error page returned instead of JSON.
|
||||
_OUTAGE_PHRASES = (
|
||||
"not initialized", "not configured", "service unavailable",
|
||||
"bad gateway", "gateway time", "request failed", "response not json",
|
||||
"max retries", "connection", "timed out", "temporarily unavailable",
|
||||
)
|
||||
|
||||
# Back-off schedule while the source is down.
|
||||
_NORMAL_DELAY = 2 # seconds between items when healthy
|
||||
_OUTAGE_BASE = 30 # first back-off step
|
||||
_OUTAGE_CAP = 1800 # 30 minutes max
|
||||
|
||||
|
||||
def is_source_outage(exc: Exception) -> bool:
|
||||
"""True when ``exc`` indicates the Amazon source/proxy is down (transient,
|
||||
whole-source), as opposed to a normal per-item error.
|
||||
|
||||
Robust to how the error is surfaced: an explicit ``status_code`` attribute,
|
||||
an ``HTTP <code>`` prefix in the message, or an outage phrase (covers
|
||||
connection failures and non-JSON error pages that carry no status code)."""
|
||||
code = getattr(exc, "status_code", None)
|
||||
if isinstance(code, int) and code in _OUTAGE_STATUS:
|
||||
return True
|
||||
msg = str(exc).lower()
|
||||
m = re.search(r"http\s+(\d{3})", msg)
|
||||
if m and int(m.group(1)) in _OUTAGE_STATUS:
|
||||
return True
|
||||
return any(p in msg for p in _OUTAGE_PHRASES)
|
||||
|
||||
|
||||
def next_poll_delay_seconds(outage_streak: int) -> int:
|
||||
"""Seconds to wait before the next item. Normal cadence when healthy;
|
||||
escalating back-off (30s, 60s, 120s, … capped at 30 min) the longer the
|
||||
source has been down, so a dead instance can't flood logs/CPU/DB."""
|
||||
if outage_streak <= 0:
|
||||
return _NORMAL_DELAY
|
||||
return min(_OUTAGE_BASE * (2 ** min(outage_streak - 1, 6)), _OUTAGE_CAP)
|
||||
|
|
@ -1,645 +0,0 @@
|
|||
import re
|
||||
import threading
|
||||
import time
|
||||
from difflib import SequenceMatcher
|
||||
from typing import Optional, Dict, Any
|
||||
from datetime import datetime, timedelta
|
||||
from utils.logging_config import get_logger
|
||||
from database.music_database import MusicDatabase
|
||||
from core.amazon_client import AmazonClient
|
||||
from core.worker_utils import interruptible_sleep, set_album_api_track_count
|
||||
from core.enrichment.manual_match_honoring import honor_stored_match
|
||||
from core.amazon_outage import is_source_outage, next_poll_delay_seconds
|
||||
|
||||
logger = get_logger("amazon_worker")
|
||||
|
||||
|
||||
class AmazonWorker:
|
||||
"""Background worker for enriching library artists, albums, and tracks with Amazon Music metadata."""
|
||||
|
||||
def __init__(self, database: MusicDatabase):
|
||||
self.db = database
|
||||
self.client = AmazonClient()
|
||||
self._amazon_schema_checked = False
|
||||
|
||||
self.running = False
|
||||
self.paused = False
|
||||
self.should_stop = False
|
||||
self.thread = None
|
||||
self._stop_event = threading.Event()
|
||||
|
||||
self.current_item = None
|
||||
|
||||
self.stats = {
|
||||
'matched': 0,
|
||||
'not_found': 0,
|
||||
'pending': 0,
|
||||
'errors': 0,
|
||||
}
|
||||
|
||||
self.retry_days = 30
|
||||
self.name_similarity_threshold = 0.80
|
||||
|
||||
# Source-outage circuit breaker: counts consecutive whole-source
|
||||
# failures (proxy down / "not initialized" / unreachable) so the loop
|
||||
# backs off instead of grinding the whole library item-by-item.
|
||||
self._outage_streak = 0
|
||||
|
||||
logger.info("Amazon background worker initialized")
|
||||
|
||||
def _ensure_amazon_schema(self, cursor) -> None:
|
||||
"""Ensure upgraded installs have the Amazon enrichment columns.
|
||||
|
||||
MusicDatabase normally runs this migration during startup, but the
|
||||
worker should still be defensive because it is the code path that
|
||||
repeatedly queries these columns in the background.
|
||||
"""
|
||||
if self._amazon_schema_checked:
|
||||
return
|
||||
|
||||
table_columns = {
|
||||
'artists': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
|
||||
'albums': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
|
||||
'tracks': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
|
||||
}
|
||||
for table, columns in table_columns.items():
|
||||
cursor.execute(f"PRAGMA table_info({table})")
|
||||
existing = {row[1] for row in cursor.fetchall()}
|
||||
for column in columns:
|
||||
if column not in existing:
|
||||
column_type = 'TIMESTAMP' if column == 'amazon_last_attempted' else 'TEXT'
|
||||
cursor.execute(f"ALTER TABLE {table} ADD COLUMN {column} {column_type}")
|
||||
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_id ON artists (amazon_id)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_status ON artists (amazon_match_status)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_id ON albums (amazon_id)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_status ON albums (amazon_match_status)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_id ON tracks (amazon_id)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_status ON tracks (amazon_match_status)")
|
||||
cursor.connection.commit()
|
||||
self._amazon_schema_checked = True
|
||||
|
||||
def start(self):
|
||||
if self.running:
|
||||
logger.warning("Worker already running")
|
||||
return
|
||||
|
||||
self.running = True
|
||||
self.should_stop = False
|
||||
self._stop_event.clear()
|
||||
self.thread = threading.Thread(target=self._run, daemon=True)
|
||||
self.thread.start()
|
||||
logger.info("Amazon background worker started")
|
||||
|
||||
def stop(self):
|
||||
if not self.running:
|
||||
return
|
||||
|
||||
logger.info("Stopping Amazon worker...")
|
||||
self.should_stop = True
|
||||
self.running = False
|
||||
self._stop_event.set()
|
||||
|
||||
if self.thread:
|
||||
self.thread.join(timeout=1)
|
||||
|
||||
logger.info("Amazon worker stopped")
|
||||
|
||||
def pause(self):
|
||||
if not self.running:
|
||||
logger.warning("Worker not running, cannot pause")
|
||||
return
|
||||
self.paused = True
|
||||
logger.info("Amazon worker paused")
|
||||
|
||||
def resume(self):
|
||||
if not self.running:
|
||||
logger.warning("Worker not running, start it first")
|
||||
return
|
||||
self.paused = False
|
||||
logger.info("Amazon worker resumed")
|
||||
|
||||
def get_stats(self) -> Dict[str, Any]:
|
||||
self.stats['pending'] = self._count_pending_items()
|
||||
progress = self._get_progress_breakdown()
|
||||
is_actually_running = self.running and (self.thread is not None and self.thread.is_alive())
|
||||
is_idle = is_actually_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
|
||||
return {
|
||||
'enabled': True,
|
||||
'running': is_actually_running and not self.paused,
|
||||
'paused': self.paused,
|
||||
'idle': is_idle,
|
||||
'current_item': self.current_item,
|
||||
'stats': self.stats.copy(),
|
||||
'progress': progress,
|
||||
}
|
||||
|
||||
def _run(self):
|
||||
logger.info("Amazon worker thread started")
|
||||
while not self.should_stop:
|
||||
try:
|
||||
if self.paused:
|
||||
interruptible_sleep(self._stop_event, 1)
|
||||
continue
|
||||
|
||||
self.current_item = None
|
||||
item = self._get_next_item()
|
||||
|
||||
if not item:
|
||||
logger.debug("No pending items, sleeping...")
|
||||
interruptible_sleep(self._stop_event, 10)
|
||||
continue
|
||||
|
||||
self.current_item = item
|
||||
item_id = item.get('id') or item.get('artist_id') or item.get('album_id')
|
||||
if item_id is None:
|
||||
logger.warning(f"Skipping {item.get('type', 'unknown')} with NULL id: {item.get('name', '?')}")
|
||||
continue
|
||||
|
||||
self._process_item(item)
|
||||
# Normal 2s cadence when healthy; escalating back-off (up to
|
||||
# 30 min) while the source is in an outage streak.
|
||||
interruptible_sleep(self._stop_event, next_poll_delay_seconds(self._outage_streak))
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in worker loop: {e}")
|
||||
interruptible_sleep(self._stop_event, 5)
|
||||
|
||||
logger.info("Amazon worker thread finished")
|
||||
|
||||
def _get_next_item(self) -> Optional[Dict[str, Any]]:
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
self._ensure_amazon_schema(cursor)
|
||||
|
||||
# Pinned-group override (Manage Enrichment Workers): process one
|
||||
# entity type first, then fall through to the normal chain. Unset or
|
||||
# exhausted ⇒ default artist→album→track order, unchanged.
|
||||
from core.worker_utils import read_enrichment_priority, priority_pending_item
|
||||
_prio = read_enrichment_priority('amazon')
|
||||
if _prio:
|
||||
_pi = priority_pending_item(cursor, 'amazon', _prio)
|
||||
if _pi:
|
||||
return _pi
|
||||
|
||||
# Priority 1: Unattempted artists
|
||||
cursor.execute("""
|
||||
SELECT id, name FROM artists
|
||||
WHERE amazon_match_status IS NULL AND id IS NOT NULL
|
||||
ORDER BY id ASC LIMIT 1
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'artist', 'id': row[0], 'name': row[1]}
|
||||
|
||||
# Priority 2: Unattempted albums
|
||||
cursor.execute("""
|
||||
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
|
||||
FROM albums a
|
||||
JOIN artists ar ON a.artist_id = ar.id
|
||||
WHERE a.amazon_match_status IS NULL AND a.id IS NOT NULL
|
||||
ORDER BY a.id ASC LIMIT 1
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
|
||||
|
||||
# Priority 3: Unattempted tracks
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
|
||||
FROM tracks t
|
||||
JOIN artists ar ON t.artist_id = ar.id
|
||||
WHERE t.amazon_match_status IS NULL AND t.id IS NOT NULL
|
||||
ORDER BY t.id ASC LIMIT 1
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
|
||||
|
||||
not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
|
||||
|
||||
# Priority 4: Retry not_found artists
|
||||
cursor.execute("""
|
||||
SELECT id, name FROM artists
|
||||
WHERE amazon_match_status = 'not_found' AND amazon_last_attempted < ?
|
||||
ORDER BY amazon_last_attempted ASC LIMIT 1
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
|
||||
return {'type': 'artist', 'id': row[0], 'name': row[1]}
|
||||
|
||||
# Priority 5: Retry not_found albums
|
||||
cursor.execute("""
|
||||
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
|
||||
FROM albums a
|
||||
JOIN artists ar ON a.artist_id = ar.id
|
||||
WHERE a.amazon_match_status = 'not_found' AND a.amazon_last_attempted < ?
|
||||
ORDER BY a.amazon_last_attempted ASC LIMIT 1
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
|
||||
|
||||
# Priority 6: Retry not_found tracks
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
|
||||
FROM tracks t
|
||||
JOIN artists ar ON t.artist_id = ar.id
|
||||
WHERE t.amazon_match_status = 'not_found' AND t.amazon_last_attempted < ?
|
||||
ORDER BY t.amazon_last_attempted ASC LIMIT 1
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting next item: {e}")
|
||||
return None
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _normalize_name(self, name: str) -> str:
|
||||
name = name.lower().strip()
|
||||
name = re.sub(r'\s+[-–—]\s+.*$', '', name)
|
||||
name = re.sub(r'\s*\(.*?\)\s*', ' ', name)
|
||||
name = re.sub(r'[^\w\s]', '', name)
|
||||
name = re.sub(r'\s+', ' ', name).strip()
|
||||
return name
|
||||
|
||||
def _name_matches(self, query_name: str, result_name: str) -> bool:
|
||||
norm_query = self._normalize_name(query_name)
|
||||
norm_result = self._normalize_name(result_name)
|
||||
similarity = SequenceMatcher(None, norm_query, norm_result).ratio()
|
||||
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
|
||||
return similarity >= self.name_similarity_threshold
|
||||
|
||||
def _process_item(self, item: Dict[str, Any]):
|
||||
try:
|
||||
item_type = item['type']
|
||||
item_id = item['id']
|
||||
item_name = item['name']
|
||||
logger.debug(f"Processing {item_type} #{item_id}: {item_name}")
|
||||
|
||||
if item_type == 'artist':
|
||||
self._process_artist(item_id, item_name)
|
||||
elif item_type == 'album':
|
||||
self._process_album(item_id, item_name, item.get('artist', ''), item)
|
||||
elif item_type == 'track':
|
||||
self._process_track(item_id, item_name, item.get('artist', ''), item)
|
||||
|
||||
# The source answered (match or not_found) — clear any outage streak.
|
||||
if self._outage_streak:
|
||||
logger.info("Amazon source recovered after %d outage(s), resuming",
|
||||
self._outage_streak)
|
||||
self._outage_streak = 0
|
||||
|
||||
except Exception as e:
|
||||
if is_source_outage(e):
|
||||
# The whole source is down (proxy 5xx / "not initialized" /
|
||||
# unreachable). Do NOT mark the item 'error' — that would burn
|
||||
# the entire library to a state the retry tiers never re-attempt
|
||||
# for a transient outage. Leave it untouched so it's retried once
|
||||
# the instance recovers, and let the loop back off. Log once per
|
||||
# streak to avoid flooding.
|
||||
self._outage_streak += 1
|
||||
if self._outage_streak == 1:
|
||||
logger.warning("Amazon source unavailable — pausing enrichment "
|
||||
"until it recovers: %s", e)
|
||||
else:
|
||||
logger.debug("Amazon source still unavailable (streak=%d): %s",
|
||||
self._outage_streak, e)
|
||||
return
|
||||
# A non-outage error means the source actually answered (e.g. a
|
||||
# 404/parse error on a real response), so the outage is over —
|
||||
# clear the streak and handle this as a normal per-item error.
|
||||
self._outage_streak = 0
|
||||
logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
|
||||
self.stats['errors'] += 1
|
||||
try:
|
||||
self._mark_status(item['type'], item['id'], 'error')
|
||||
except Exception as e2:
|
||||
logger.error(f"Error updating item status: {e2}")
|
||||
|
||||
def _get_existing_id(self, entity_type: str, entity_id: int) -> Optional[str]:
|
||||
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
|
||||
table = table_map.get(entity_type)
|
||||
if not table:
|
||||
return None
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(f"SELECT amazon_id FROM {table} WHERE id = ?", (entity_id,))
|
||||
row = cursor.fetchone()
|
||||
return row[0] if row and row[0] else None
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _process_artist(self, artist_id: int, artist_name: str):
|
||||
existing_id = self._get_existing_id('artist', artist_id)
|
||||
if existing_id:
|
||||
logger.debug(f"Preserving existing Amazon ID for artist '{artist_name}': {existing_id}")
|
||||
return
|
||||
|
||||
results = self.client.search_artists(artist_name, limit=5)
|
||||
if results:
|
||||
result = results[0]
|
||||
if self._name_matches(artist_name, result.name):
|
||||
self._update_artist(artist_id, result)
|
||||
self.stats['matched'] += 1
|
||||
logger.info(f"Matched artist '{artist_name}' -> Amazon ID: {result.id}")
|
||||
else:
|
||||
self._mark_status('artist', artist_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result.name}')")
|
||||
else:
|
||||
self._mark_status('artist', artist_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"No match for artist '{artist_name}'")
|
||||
|
||||
def _refresh_album_via_stored_id(self, album_id, stored_id, api_data):
|
||||
self._update_album(album_id, api_data, stored_id)
|
||||
|
||||
def _refresh_track_via_stored_id(self, track_id, stored_id, api_data):
|
||||
self._update_track(track_id, api_data, stored_id)
|
||||
|
||||
def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
|
||||
if honor_stored_match(
|
||||
db=self.db, entity_table='albums', entity_id=album_id,
|
||||
id_column='amazon_id',
|
||||
client_fetch_fn=lambda asin: self.client.get_album(asin, include_tracks=False),
|
||||
on_match_fn=self._refresh_album_via_stored_id,
|
||||
log_prefix='Amazon',
|
||||
):
|
||||
self.stats['matched'] += 1
|
||||
return
|
||||
|
||||
query = f"{artist_name} {album_name}"
|
||||
results = self.client.search_albums(query, limit=10)
|
||||
if results:
|
||||
result = results[0]
|
||||
if self._name_matches(album_name, result.name):
|
||||
full_album = None
|
||||
if result.id:
|
||||
try:
|
||||
full_album = self.client.get_album(result.id, include_tracks=False)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch full album '{album_name}' (ASIN: {result.id}): {e}")
|
||||
|
||||
if full_album is None:
|
||||
self._mark_status('album', album_id, 'error')
|
||||
self.stats['errors'] += 1
|
||||
logger.warning(f"Album '{album_name}' matched but full details unavailable, will retry")
|
||||
return
|
||||
|
||||
self._update_album(album_id, full_album, result.id)
|
||||
self.stats['matched'] += 1
|
||||
logger.info(f"Matched album '{album_name}' -> Amazon ASIN: {result.id}")
|
||||
else:
|
||||
self._mark_status('album', album_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"Name mismatch for album '{album_name}' (got '{result.name}')")
|
||||
else:
|
||||
self._mark_status('album', album_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"No match for album '{album_name}'")
|
||||
|
||||
def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
|
||||
if honor_stored_match(
|
||||
db=self.db, entity_table='tracks', entity_id=track_id,
|
||||
id_column='amazon_id',
|
||||
client_fetch_fn=self.client.get_track_details,
|
||||
on_match_fn=self._refresh_track_via_stored_id,
|
||||
log_prefix='Amazon',
|
||||
):
|
||||
self.stats['matched'] += 1
|
||||
return
|
||||
|
||||
query = f"{artist_name} {track_name}"
|
||||
results = self.client.search_tracks(query, limit=10)
|
||||
if results:
|
||||
result = results[0]
|
||||
if self._name_matches(track_name, result.name):
|
||||
full_track = None
|
||||
if result.id:
|
||||
try:
|
||||
full_track = self.client.get_track_details(result.id)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch full track '{track_name}' (ASIN: {result.id}): {e}")
|
||||
|
||||
if full_track is None:
|
||||
self._mark_status('track', track_id, 'error')
|
||||
self.stats['errors'] += 1
|
||||
logger.warning(f"Track '{track_name}' matched but full details unavailable, will retry")
|
||||
return
|
||||
|
||||
self._update_track(track_id, full_track, result.id)
|
||||
self.stats['matched'] += 1
|
||||
logger.info(f"Matched track '{track_name}' -> Amazon ASIN: {result.id}")
|
||||
else:
|
||||
self._mark_status('track', track_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"Name mismatch for track '{track_name}' (got '{result.name}')")
|
||||
else:
|
||||
self._mark_status('track', track_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"No match for track '{track_name}'")
|
||||
|
||||
def _update_artist(self, artist_id: int, result):
|
||||
"""Store Amazon metadata for an artist. ``result`` is an Artist dataclass."""
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE artists SET
|
||||
amazon_id = ?,
|
||||
amazon_match_status = 'matched',
|
||||
amazon_last_attempted = CURRENT_TIMESTAMP,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (str(result.id), artist_id))
|
||||
|
||||
# Backfill thumb_url from album cover stand-in when artist has no image
|
||||
image_url = result.image_url
|
||||
if not image_url:
|
||||
try:
|
||||
image_url = self.client._get_artist_image_from_albums(result.id)
|
||||
except Exception as exc:
|
||||
logger.debug("Artist image from albums failed for %s: %s", result.id, exc)
|
||||
if image_url:
|
||||
cursor.execute("""
|
||||
UPDATE artists SET thumb_url = ?
|
||||
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
|
||||
""", (image_url, artist_id))
|
||||
|
||||
conn.commit()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating artist #{artist_id} with Amazon data: {e}")
|
||||
raise
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _update_album(self, album_id: int, full_data: Dict[str, Any], asin: str):
|
||||
"""Store Amazon metadata for an album. ``full_data`` is a get_album() dict."""
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE albums SET
|
||||
amazon_id = ?,
|
||||
amazon_match_status = 'matched',
|
||||
amazon_last_attempted = CURRENT_TIMESTAMP,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (asin, album_id))
|
||||
|
||||
# Backfill label when missing
|
||||
label = full_data.get('label')
|
||||
if label:
|
||||
cursor.execute("""
|
||||
UPDATE albums SET label = ?
|
||||
WHERE id = ? AND (label IS NULL OR label = '')
|
||||
""", (label, album_id))
|
||||
|
||||
# Backfill thumb_url
|
||||
images = full_data.get('images') or []
|
||||
thumb_url = images[0].get('url') if images else None
|
||||
if thumb_url:
|
||||
cursor.execute("""
|
||||
UPDATE albums SET thumb_url = ?
|
||||
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
|
||||
""", (thumb_url, album_id))
|
||||
|
||||
# Cache authoritative track count for completeness repair
|
||||
total_tracks = full_data.get('total_tracks') or (
|
||||
full_data.get('tracks', {}).get('total') if isinstance(full_data.get('tracks'), dict) else None
|
||||
)
|
||||
set_album_api_track_count(cursor, album_id, total_tracks)
|
||||
|
||||
conn.commit()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating album #{album_id} with Amazon data: {e}")
|
||||
raise
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _update_track(self, track_id: int, full_data: Dict[str, Any], asin: str):
|
||||
"""Store Amazon metadata for a track. ``full_data`` is a get_track_details() dict."""
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE tracks SET
|
||||
amazon_id = ?,
|
||||
amazon_match_status = 'matched',
|
||||
amazon_last_attempted = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (asin, track_id))
|
||||
|
||||
conn.commit()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating track #{track_id} with Amazon data: {e}")
|
||||
raise
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _mark_status(self, entity_type: str, entity_id: int, status: str):
|
||||
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
|
||||
table = table_map.get(entity_type)
|
||||
if not table:
|
||||
logger.error(f"Unknown entity type: {entity_type}")
|
||||
return
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(f"""
|
||||
UPDATE {table} SET
|
||||
amazon_match_status = ?,
|
||||
amazon_last_attempted = CURRENT_TIMESTAMP,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (status, entity_id))
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
logger.error(f"Error marking {entity_type} #{entity_id} status: {e}")
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _count_pending_items(self) -> int:
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
self._ensure_amazon_schema(cursor)
|
||||
cursor.execute("""
|
||||
SELECT
|
||||
(SELECT COUNT(*) FROM artists WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
|
||||
(SELECT COUNT(*) FROM albums WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
|
||||
(SELECT COUNT(*) FROM tracks WHERE amazon_match_status IS NULL AND id IS NOT NULL)
|
||||
AS pending
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
return row[0] if row else 0
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting pending items: {e}")
|
||||
return 0
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _get_progress_breakdown(self) -> Dict[str, Dict[str, int]]:
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
self._ensure_amazon_schema(cursor)
|
||||
progress = {}
|
||||
|
||||
for table in ('artists', 'albums', 'tracks'):
|
||||
cursor.execute(f"""
|
||||
SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN amazon_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
|
||||
FROM {table}
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
total, processed = row[0], row[1] or 0
|
||||
progress[table] = {
|
||||
'matched': processed,
|
||||
'total': total,
|
||||
'percent': int((processed / total * 100) if total > 0 else 0),
|
||||
}
|
||||
|
||||
return progress
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting progress breakdown: {e}")
|
||||
return {}
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
|
@ -30,7 +30,6 @@ RATE_LIMITS = {
|
|||
'tidal': 120, # MIN_API_INTERVAL=0.5s → ~120/min
|
||||
'qobuz': 60, # Variable throttle, ~60/min estimate
|
||||
'discogs': 60, # MIN_API_INTERVAL=1.0s with auth → ~60/min
|
||||
'amazon': 120, # MIN_API_INTERVAL=0.5s → ~120/min (T2Tunes proxy)
|
||||
}
|
||||
|
||||
# Display names for UI
|
||||
|
|
@ -45,13 +44,12 @@ SERVICE_LABELS = {
|
|||
'tidal': 'Tidal',
|
||||
'qobuz': 'Qobuz',
|
||||
'discogs': 'Discogs',
|
||||
'amazon': 'Amazon Music',
|
||||
}
|
||||
|
||||
# Display order
|
||||
SERVICE_ORDER = [
|
||||
'spotify', 'itunes', 'deezer', 'lastfm', 'genius',
|
||||
'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs', 'amazon',
|
||||
'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs',
|
||||
]
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,238 +0,0 @@
|
|||
"""Archive extraction + audio-file discovery for torrent / usenet downloads.
|
||||
|
||||
The torrent and usenet download plugins need a uniform way to:
|
||||
|
||||
1. Walk the downloader's save directory and find every audio file in it.
|
||||
2. If the directory contains an archive (``.zip`` / ``.rar`` / ``.tar`` /
|
||||
``.7z``), extract it first so the audio files inside become walkable.
|
||||
|
||||
This module is intentionally narrow — no matching, no tagging, no
|
||||
import. The download plugin layer composes this with the existing
|
||||
post-processing / matching pipeline. Lidarr does NOT use this module:
|
||||
Lidarr extracts archives in its own import step before SoulSync sees
|
||||
the files at all. Usenet downloaders (SABnzbd, NZBGet) also auto-
|
||||
extract by default. Torrents are the main case where SoulSync may
|
||||
need to do the extract step itself — most music torrents ship loose,
|
||||
but some bundle the album in a ``.rar`` archive.
|
||||
|
||||
``rarfile`` is an optional dependency. If it isn't installed, archives
|
||||
with ``.rar`` content are skipped with a single warning rather than
|
||||
crashing the download.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tarfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("archive_pipeline")
|
||||
|
||||
|
||||
# Same audio-extension set as ``core/imports/file_ops.py`` ``quality_tiers``.
|
||||
# Keep them in sync — if a new format is added to file_ops, add it here too
|
||||
# or the walker will skip it and the download plugin will mark the download
|
||||
# failed even when files arrived.
|
||||
AUDIO_EXTENSIONS = frozenset([
|
||||
# lossless
|
||||
'.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
|
||||
# high lossy
|
||||
'.opus', '.ogg',
|
||||
# standard lossy
|
||||
'.m4a', '.aac',
|
||||
# low lossy
|
||||
'.mp3', '.wma',
|
||||
])
|
||||
|
||||
ARCHIVE_EXTENSIONS = frozenset(['.zip', '.rar', '.tar', '.tar.gz', '.tgz', '.7z'])
|
||||
|
||||
|
||||
def is_archive(path: Path) -> bool:
|
||||
"""True if the file extension looks like a supported archive.
|
||||
|
||||
Compound extensions (``.tar.gz``, ``.tar.bz2``) are detected by
|
||||
checking the last two suffixes joined together — Path.suffix
|
||||
only returns the final suffix.
|
||||
"""
|
||||
if not path.is_file():
|
||||
return False
|
||||
name = path.name.lower()
|
||||
if name.endswith(('.tar.gz', '.tar.bz2', '.tar.xz')):
|
||||
return True
|
||||
return path.suffix.lower() in ARCHIVE_EXTENSIONS
|
||||
|
||||
|
||||
def walk_audio_files(directory: Path) -> List[Path]:
|
||||
"""Recursively scan ``directory`` for audio files. Returns
|
||||
a sorted list of absolute paths. Empty list if the directory
|
||||
doesn't exist or contains no audio.
|
||||
"""
|
||||
if not directory or not directory.exists() or not directory.is_dir():
|
||||
return []
|
||||
out: List[Path] = []
|
||||
for child in directory.rglob('*'):
|
||||
if not child.is_file():
|
||||
continue
|
||||
if child.suffix.lower() in AUDIO_EXTENSIONS:
|
||||
out.append(child.resolve())
|
||||
out.sort()
|
||||
return out
|
||||
|
||||
|
||||
def find_archives_in_dir(directory: Path) -> List[Path]:
|
||||
"""Find every archive file directly inside ``directory`` (one
|
||||
level deep — torrents normally put the archive at the root of
|
||||
their folder; we don't search nested dirs to avoid extracting
|
||||
something we shouldn't).
|
||||
"""
|
||||
if not directory or not directory.exists() or not directory.is_dir():
|
||||
return []
|
||||
return sorted(p for p in directory.iterdir() if is_archive(p))
|
||||
|
||||
|
||||
def extract_archive(archive_path: Path, extract_to: Optional[Path] = None) -> Optional[Path]:
|
||||
"""Extract a single archive in-place (or to ``extract_to`` if
|
||||
given). Returns the directory the archive was extracted into,
|
||||
or ``None`` on failure.
|
||||
|
||||
Supports ``.zip``, ``.tar``/``.tar.gz``/``.tar.bz2``/``.tar.xz``,
|
||||
and ``.rar`` (only when the optional ``rarfile`` library is
|
||||
installed). ``.7z`` is recognised but extraction requires
|
||||
``py7zr``; without it, the call logs and returns None.
|
||||
"""
|
||||
if not archive_path or not archive_path.exists():
|
||||
logger.warning("archive_pipeline: %s does not exist", archive_path)
|
||||
return None
|
||||
dest = extract_to or archive_path.parent
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
name = archive_path.name.lower()
|
||||
try:
|
||||
if name.endswith('.zip'):
|
||||
with zipfile.ZipFile(archive_path) as zf:
|
||||
_safe_extract_zip(zf, dest)
|
||||
return dest
|
||||
if name.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
|
||||
with tarfile.open(archive_path) as tf:
|
||||
_safe_extract_tar(tf, dest)
|
||||
return dest
|
||||
if name.endswith('.rar'):
|
||||
return _extract_rar(archive_path, dest)
|
||||
if name.endswith('.7z'):
|
||||
return _extract_7z(archive_path, dest)
|
||||
except (zipfile.BadZipFile, tarfile.TarError, OSError) as e:
|
||||
logger.error("archive_pipeline: failed to extract %s: %s", archive_path, e)
|
||||
return None
|
||||
logger.warning("archive_pipeline: unknown archive type for %s", archive_path)
|
||||
return None
|
||||
|
||||
|
||||
def extract_all_in_dir(directory: Path) -> List[Path]:
|
||||
"""Find every archive in ``directory`` and extract each in place.
|
||||
Returns the list of directories archives were extracted into
|
||||
(usually all the same — ``directory`` itself). Archives that
|
||||
failed to extract are skipped silently after a warning.
|
||||
"""
|
||||
out: List[Path] = []
|
||||
for archive in find_archives_in_dir(directory):
|
||||
result = extract_archive(archive)
|
||||
if result is not None:
|
||||
out.append(result)
|
||||
return out
|
||||
|
||||
|
||||
def collect_audio_after_extraction(directory: Path) -> List[Path]:
|
||||
"""One-shot helper for the download plugins: extract any archives
|
||||
in the directory, then return the walked audio file list. This is
|
||||
the common pattern — torrent / usenet plugin gets a save_path,
|
||||
calls this, hands the resulting files to the matching pipeline.
|
||||
"""
|
||||
extract_all_in_dir(directory)
|
||||
return walk_audio_files(directory)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Safety helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _safe_extract_zip(zf: zipfile.ZipFile, dest: Path) -> None:
|
||||
"""Extract a zipfile after rejecting any member whose resolved
|
||||
path escapes ``dest`` (path traversal protection).
|
||||
"""
|
||||
dest = dest.resolve()
|
||||
for member in zf.namelist():
|
||||
target = (dest / member).resolve()
|
||||
if dest not in target.parents and target != dest:
|
||||
logger.error("archive_pipeline: refusing path-traversal member %r", member)
|
||||
return
|
||||
zf.extractall(dest)
|
||||
|
||||
|
||||
def _safe_extract_tar(tf: tarfile.TarFile, dest: Path) -> None:
|
||||
"""Same path-traversal protection for tarfiles."""
|
||||
dest = dest.resolve()
|
||||
for member in tf.getmembers():
|
||||
target = (dest / member.name).resolve()
|
||||
if dest not in target.parents and target != dest:
|
||||
logger.error("archive_pipeline: refusing path-traversal member %r", member.name)
|
||||
return
|
||||
# ``filter='data'`` is the Python 3.12+ safe extractor; fall back
|
||||
# to the legacy call on older runtimes.
|
||||
try:
|
||||
tf.extractall(dest, filter='data') # type: ignore[call-arg]
|
||||
except TypeError:
|
||||
tf.extractall(dest)
|
||||
|
||||
|
||||
def _extract_rar(archive_path: Path, dest: Path) -> Optional[Path]:
|
||||
try:
|
||||
import rarfile # type: ignore[import-untyped]
|
||||
except ImportError:
|
||||
logger.warning(
|
||||
"archive_pipeline: cannot extract %s — rarfile library not installed. "
|
||||
"Install with: pip install rarfile (and ensure unrar is on PATH).",
|
||||
archive_path,
|
||||
)
|
||||
return None
|
||||
try:
|
||||
with rarfile.RarFile(archive_path) as rf:
|
||||
dest_resolved = dest.resolve()
|
||||
for name in rf.namelist():
|
||||
target = (dest_resolved / name).resolve()
|
||||
if dest_resolved not in target.parents and target != dest_resolved:
|
||||
logger.error("archive_pipeline: refusing path-traversal rar member %r", name)
|
||||
return None
|
||||
rf.extractall(dest)
|
||||
return dest
|
||||
except Exception as e:
|
||||
logger.error("archive_pipeline: rar extract failed for %s: %s", archive_path, e)
|
||||
return None
|
||||
|
||||
|
||||
def _extract_7z(archive_path: Path, dest: Path) -> Optional[Path]:
|
||||
try:
|
||||
import py7zr # type: ignore[import-untyped]
|
||||
except ImportError:
|
||||
logger.warning(
|
||||
"archive_pipeline: cannot extract %s — py7zr library not installed. "
|
||||
"Install with: pip install py7zr.",
|
||||
archive_path,
|
||||
)
|
||||
return None
|
||||
try:
|
||||
with py7zr.SevenZipFile(archive_path, 'r') as sz:
|
||||
dest_resolved = dest.resolve()
|
||||
for name in sz.getnames():
|
||||
target = (dest_resolved / name).resolve()
|
||||
if dest_resolved not in target.parents and target != dest_resolved:
|
||||
logger.error("archive_pipeline: refusing path-traversal 7z member %r", name)
|
||||
return None
|
||||
sz.extractall(path=dest)
|
||||
return dest
|
||||
except Exception as e:
|
||||
logger.error("archive_pipeline: 7z extract failed for %s: %s", archive_path, e)
|
||||
return None
|
||||
|
|
@ -43,7 +43,6 @@ def build_source_only_artist_detail(
|
|||
deezer_client: Optional[Any] = None,
|
||||
itunes_client: Optional[Any] = None,
|
||||
discogs_client: Optional[Any] = None,
|
||||
amazon_client: Optional[Any] = None,
|
||||
lastfm_api_key: Optional[str] = None,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Build the artist-detail payload for a source-only artist.
|
||||
|
|
@ -52,7 +51,7 @@ def build_source_only_artist_detail(
|
|||
``jsonify`` or equivalent. Status is 200 on success, 404 when the
|
||||
source's discography lookup returned no releases.
|
||||
"""
|
||||
resolved_name = (artist_name or "").strip()
|
||||
resolved_name = (artist_name or artist_id or "").strip()
|
||||
|
||||
# 1. Image URL via the same helper /api/artist/<id>/image uses.
|
||||
image_url: Optional[str] = None
|
||||
|
|
@ -68,8 +67,6 @@ def build_source_only_artist_detail(
|
|||
if source == "spotify" and spotify_client is not None:
|
||||
sp_artist = spotify_client.get_artist(artist_id, allow_fallback=False)
|
||||
if sp_artist:
|
||||
if not artist_name and sp_artist.get("name"):
|
||||
resolved_name = sp_artist["name"]
|
||||
source_genres = sp_artist.get("genres") or []
|
||||
source_followers = (sp_artist.get("followers") or {}).get("total")
|
||||
if not image_url and sp_artist.get("images"):
|
||||
|
|
@ -77,41 +74,16 @@ def build_source_only_artist_detail(
|
|||
elif source == "deezer" and deezer_client is not None:
|
||||
dz_artist = deezer_client.get_artist_info(artist_id)
|
||||
if dz_artist:
|
||||
if not artist_name and dz_artist.get("name"):
|
||||
resolved_name = dz_artist["name"]
|
||||
source_genres = dz_artist.get("genres") or []
|
||||
source_followers = (dz_artist.get("followers") or {}).get("total")
|
||||
elif source == "itunes" and itunes_client is not None:
|
||||
it_artist = itunes_client.get_artist(artist_id)
|
||||
if it_artist:
|
||||
if not artist_name and it_artist.get("name"):
|
||||
resolved_name = it_artist["name"]
|
||||
source_genres = it_artist.get("genres") or []
|
||||
elif source == "discogs" and discogs_client is not None:
|
||||
dc_artist = discogs_client.get_artist(artist_id)
|
||||
if dc_artist:
|
||||
if not artist_name and dc_artist.get("name"):
|
||||
resolved_name = dc_artist["name"]
|
||||
source_genres = dc_artist.get("genres") or []
|
||||
elif source == "amazon" and amazon_client is not None:
|
||||
az_artist = amazon_client.get_artist(resolved_name or artist_id)
|
||||
if az_artist:
|
||||
if not artist_name and az_artist.get("name"):
|
||||
resolved_name = az_artist["name"]
|
||||
source_genres = az_artist.get("genres") or []
|
||||
if not image_url and az_artist.get("images"):
|
||||
image_url = az_artist["images"][0].get("url")
|
||||
elif source == "musicbrainz":
|
||||
try:
|
||||
from core.musicbrainz_search import MusicBrainzSearchClient
|
||||
mb = MusicBrainzSearchClient()
|
||||
mb_artist = mb.get_artist(artist_id)
|
||||
if mb_artist:
|
||||
if not artist_name and mb_artist.get("name"):
|
||||
resolved_name = mb_artist["name"]
|
||||
source_genres = mb_artist.get("genres") or []
|
||||
except Exception as e:
|
||||
logger.debug(f"MusicBrainz artist info lookup failed for {artist_id}: {e}")
|
||||
except Exception as e:
|
||||
logger.debug(f"Source-side artist info lookup failed for {source}:{artist_id}: {e}")
|
||||
|
||||
|
|
@ -154,11 +126,7 @@ def build_source_only_artist_detail(
|
|||
allow_fallback=True,
|
||||
skip_cache=False,
|
||||
max_pages=0,
|
||||
# Match the Download Discography endpoint cap (200).
|
||||
# Spotify already paginates all; Deezer / iTunes / Discogs /
|
||||
# Hydrabase clamp at the outer limit. 200 covers prolific
|
||||
# catalogues without exceeding iTunes/Discogs internal caps.
|
||||
limit=200,
|
||||
limit=50,
|
||||
artist_source_ids={source: artist_id},
|
||||
dedup_variants=False,
|
||||
),
|
||||
|
|
|
|||
|
|
@ -21,24 +21,21 @@ from __future__ import annotations
|
|||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from core.source_ids import id_column as _artist_id_column
|
||||
|
||||
logger = logging.getLogger("artist_source_lookup")
|
||||
|
||||
|
||||
SOURCE_ONLY_ARTIST_SOURCES = frozenset({
|
||||
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
|
||||
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz",
|
||||
})
|
||||
|
||||
|
||||
# The per-source column on the ``artists`` table, derived from the canonical
|
||||
# source-ID registry (the single source of truth). Values are unchanged from the
|
||||
# previous hardcoded map — this just stops duplicating that knowledge here.
|
||||
SOURCE_ID_FIELD = {
|
||||
source: _artist_id_column(source, "artist")
|
||||
for source in (
|
||||
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
|
||||
)
|
||||
"spotify": "spotify_artist_id",
|
||||
"itunes": "itunes_artist_id",
|
||||
"deezer": "deezer_id",
|
||||
"discogs": "discogs_id",
|
||||
"hydrabase": "soul_id",
|
||||
"musicbrainz": "musicbrainz_id",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -53,11 +50,7 @@ def find_library_artist_for_source(
|
|||
|
||||
Lookup order:
|
||||
1. Direct match on the source-specific ID column (server-agnostic — any
|
||||
library record with the right external ID is a hit). If that id is
|
||||
stamped on MORE than one library artist, the mapping is corrupt /
|
||||
ambiguous (e.g. an enrichment bug wrote one Deezer id onto several
|
||||
artists) — we refuse to guess and fall through, so the caller can
|
||||
show the source artist directly instead of an arbitrary wrong one.
|
||||
library record with the right external ID is a hit).
|
||||
2. Case-insensitive name match within ``active_server`` (defaults to the
|
||||
active media server when not provided), so we don't jump the user
|
||||
across server contexts on a name collision.
|
||||
|
|
@ -71,23 +64,13 @@ def find_library_artist_for_source(
|
|||
try:
|
||||
with database._get_connection() as conn:
|
||||
cursor = conn.cursor()
|
||||
# LIMIT 2 so we can tell a unique match from an ambiguous one.
|
||||
cursor.execute(
|
||||
f"SELECT id FROM artists WHERE {column} = ? LIMIT 2",
|
||||
f"SELECT id, name FROM artists WHERE {column} = ? LIMIT 1",
|
||||
(str(source_artist_id),),
|
||||
)
|
||||
rows = cursor.fetchall()
|
||||
if len(rows) == 1:
|
||||
return rows[0][0]
|
||||
if len(rows) > 1:
|
||||
# Same source id on multiple artists — corrupt mapping. Don't
|
||||
# upgrade on the id; fall through to the name match (and, if
|
||||
# that misses, let the caller render the source artist).
|
||||
logger.warning(
|
||||
f"Source id {source}:{source_artist_id} maps to "
|
||||
f"{len(rows)}+ library artists — ambiguous, skipping "
|
||||
f"id-based library upgrade"
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return row[0]
|
||||
|
||||
if artist_name and active_server:
|
||||
cursor.execute(
|
||||
|
|
|
|||
|
|
@ -12,7 +12,6 @@ from core.metadata.registry import (
|
|||
get_deezer_client,
|
||||
get_discogs_client,
|
||||
get_itunes_client,
|
||||
get_musicbrainz_client,
|
||||
get_spotify_client,
|
||||
)
|
||||
|
||||
|
|
@ -34,11 +33,6 @@ def _get_discogs_client(token=None):
|
|||
return get_discogs_client(token)
|
||||
|
||||
|
||||
def _get_musicbrainz_client():
|
||||
"""Mirror of web_server._get_musicbrainz_client — delegates to registry."""
|
||||
return get_musicbrainz_client()
|
||||
|
||||
|
||||
class _SpotifyClientProxy:
|
||||
"""Resolves the global Spotify client lazily so a Spotify re-auth that
|
||||
rebinds the cached client in core.metadata.registry is visible to the
|
||||
|
|
@ -71,7 +65,6 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
|
|||
'itunes': 'itunes_artist_id',
|
||||
'deezer': 'deezer_artist_id',
|
||||
'discogs': 'discogs_artist_id',
|
||||
'musicbrainz': 'musicbrainz_artist_id',
|
||||
}
|
||||
id_cols = list(source_cols.values())
|
||||
|
||||
|
|
@ -110,10 +103,6 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
|
|||
search_clients['discogs'] = dc
|
||||
except Exception as e:
|
||||
logger.debug("discogs client init failed: %s", e)
|
||||
try:
|
||||
search_clients['musicbrainz'] = _get_musicbrainz_client()
|
||||
except Exception as e:
|
||||
logger.debug("musicbrainz client init failed: %s", e)
|
||||
|
||||
# Reuse watchlist scanner's fuzzy matching logic
|
||||
from core.watchlist_scanner import WatchlistScanner
|
||||
|
|
@ -245,7 +234,7 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
|
|||
# Determine best active source/ID — prefer Spotify, then iTunes, Deezer, Discogs
|
||||
resolved_source = None
|
||||
resolved_id = None
|
||||
for src in ('spotify', 'itunes', 'deezer', 'discogs', 'musicbrainz'):
|
||||
for src in ('spotify', 'itunes', 'deezer', 'discogs'):
|
||||
col = source_cols[src]
|
||||
if col in harvested_ids:
|
||||
resolved_source = src
|
||||
|
|
|
|||
|
|
@ -20,14 +20,10 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
|
||||
def get_current_profile_id() -> int:
|
||||
"""Mirror of web_server.get_current_profile_id — uses Flask g.
|
||||
|
||||
Catches RuntimeError too because reading `g` outside a request
|
||||
context raises that (not AttributeError) — happens when this is
|
||||
called from background threads (sync, automation, scanners)."""
|
||||
"""Mirror of web_server.get_current_profile_id — uses Flask g."""
|
||||
try:
|
||||
return g.profile_id
|
||||
except (AttributeError, RuntimeError):
|
||||
except AttributeError:
|
||||
return 1
|
||||
|
||||
|
||||
|
|
@ -140,7 +136,7 @@ def get_artist_map_data():
|
|||
placeholders = ','.join(['?'] * len(watchlist_ids))
|
||||
cursor.execute(f"""
|
||||
SELECT source_artist_id, similar_artist_name, similar_artist_spotify_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id,
|
||||
similarity_rank, occurrence_count, image_url, genres, popularity
|
||||
FROM similar_artists
|
||||
WHERE profile_id = ? AND source_artist_id IN ({placeholders})
|
||||
|
|
@ -173,7 +169,6 @@ def get_artist_map_data():
|
|||
'spotify_id': r.get('similar_artist_spotify_id') or '',
|
||||
'itunes_id': r.get('similar_artist_itunes_id') or '',
|
||||
'deezer_id': r.get('similar_artist_deezer_id') or '',
|
||||
'musicbrainz_id': r.get('similar_artist_musicbrainz_id') or '',
|
||||
'rank': r.get('similarity_rank', 5),
|
||||
'occurrence': r.get('occurrence_count', 1),
|
||||
'popularity': r.get('popularity', 0),
|
||||
|
|
@ -246,7 +241,7 @@ def get_artist_map_data():
|
|||
}
|
||||
|
||||
# Apply cache data to nodes
|
||||
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
|
||||
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
|
||||
for n in nodes:
|
||||
nn = _norm(n['name'])
|
||||
cached = cache_by_name.get(nn)
|
||||
|
|
@ -263,7 +258,7 @@ def get_artist_map_data():
|
|||
break
|
||||
# Backfill genres if missing
|
||||
if not n.get('genres') or len(n.get('genres', [])) == 0:
|
||||
for source in ('spotify', 'deezer', 'itunes', 'discogs', 'musicbrainz'):
|
||||
for source in ('spotify', 'deezer', 'itunes', 'discogs'):
|
||||
if source in cached and cached[source].get('genres'):
|
||||
n['genres'] = cached[source]['genres'][:5]
|
||||
break
|
||||
|
|
@ -370,14 +365,14 @@ def get_artist_map_genres():
|
|||
def _norm(n):
|
||||
return (n or '').lower().strip()
|
||||
|
||||
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, musicbrainz_id=None, source='unknown', popularity=0):
|
||||
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, source='unknown', popularity=0):
|
||||
n = _norm(name)
|
||||
if not n or len(n) < 2:
|
||||
return
|
||||
if n not in artists_by_name:
|
||||
artists_by_name[n] = {
|
||||
'name': name, 'image_url': '', 'genres': set(),
|
||||
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': '',
|
||||
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '',
|
||||
'sources': set(), 'popularity': 0
|
||||
}
|
||||
a = artists_by_name[n]
|
||||
|
|
@ -395,8 +390,6 @@ def get_artist_map_genres():
|
|||
a['deezer_id'] = str(deezer_id)
|
||||
if discogs_id and not a['discogs_id']:
|
||||
a['discogs_id'] = str(discogs_id)
|
||||
if musicbrainz_id and not a['musicbrainz_id']:
|
||||
a['musicbrainz_id'] = str(musicbrainz_id)
|
||||
if popularity > a['popularity']:
|
||||
a['popularity'] = popularity
|
||||
a['sources'].add(source)
|
||||
|
|
@ -413,14 +406,14 @@ def get_artist_map_genres():
|
|||
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
|
||||
except Exception as e:
|
||||
logger.debug("cache artist genres parse failed: %s", e)
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
|
||||
kwargs = {src_map.get(r['source'], 'spotify_id'): r['entity_id']}
|
||||
_add(r['name'], image_url=r['image_url'], genres=genres, source='cache', popularity=r['popularity'] or 0, **kwargs)
|
||||
|
||||
# 2. Similar artists
|
||||
cursor.execute("""
|
||||
SELECT similar_artist_name, similar_artist_spotify_id, similar_artist_itunes_id,
|
||||
similar_artist_deezer_id, similar_artist_musicbrainz_id, image_url, genres, popularity
|
||||
similar_artist_deezer_id, image_url, genres, popularity
|
||||
FROM similar_artists WHERE profile_id = ?
|
||||
""", (profile_id,))
|
||||
for r in cursor.fetchall():
|
||||
|
|
@ -432,9 +425,7 @@ def get_artist_map_genres():
|
|||
logger.debug("similar artist genres parse failed: %s", e)
|
||||
_add(r['similar_artist_name'], image_url=r['image_url'], genres=genres,
|
||||
spotify_id=r['similar_artist_spotify_id'], itunes_id=r['similar_artist_itunes_id'],
|
||||
deezer_id=r['similar_artist_deezer_id'],
|
||||
musicbrainz_id=r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else None,
|
||||
source='similar', popularity=r['popularity'] or 0)
|
||||
deezer_id=r['similar_artist_deezer_id'], source='similar', popularity=r['popularity'] or 0)
|
||||
|
||||
# 3. Watchlist artists
|
||||
cursor.execute("""
|
||||
|
|
@ -488,7 +479,6 @@ def get_artist_map_genres():
|
|||
'itunes_id': a['itunes_id'],
|
||||
'deezer_id': a['deezer_id'],
|
||||
'discogs_id': a['discogs_id'],
|
||||
'musicbrainz_id': a['musicbrainz_id'],
|
||||
'popularity': a['popularity'],
|
||||
'type': 'watchlist' if 'watchlist' in a['sources'] else 'similar',
|
||||
})
|
||||
|
|
@ -632,7 +622,7 @@ def get_artist_map_explore():
|
|||
# Find the center artist
|
||||
center_name = artist_name
|
||||
center_image = ''
|
||||
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': ''}
|
||||
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': ''}
|
||||
center_genres = []
|
||||
|
||||
# Search metadata cache for the center artist
|
||||
|
|
@ -654,7 +644,7 @@ def get_artist_map_explore():
|
|||
center_name = row['name']
|
||||
if row['image_url'] and row['image_url'].startswith('http'):
|
||||
center_image = row['image_url']
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
|
||||
k = src_map.get(row['source'], 'spotify_id')
|
||||
center_ids[k] = row['entity_id']
|
||||
if row['genres']:
|
||||
|
|
@ -665,14 +655,14 @@ def get_artist_map_explore():
|
|||
|
||||
# Check watchlist + library if not in cache
|
||||
if not artist_found and not artist_id:
|
||||
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id, musicbrainz_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
|
||||
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
|
||||
wr = cursor.fetchone()
|
||||
if wr:
|
||||
artist_found = True
|
||||
center_name = wr['artist_name']
|
||||
if wr['image_url'] and str(wr['image_url']).startswith('http'):
|
||||
center_image = wr['image_url']
|
||||
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id'), ('musicbrainz_id', 'musicbrainz_artist_id')]:
|
||||
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id')]:
|
||||
if wr[col]:
|
||||
center_ids[k] = str(wr[col])
|
||||
else:
|
||||
|
|
@ -723,7 +713,7 @@ def get_artist_map_explore():
|
|||
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
|
||||
""", (center_name,))
|
||||
for r in cursor.fetchall():
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
|
||||
k = src_map.get(r['source'], 'spotify_id')
|
||||
if not center_ids.get(k):
|
||||
center_ids[k] = r['entity_id']
|
||||
|
|
@ -752,7 +742,7 @@ def get_artist_map_explore():
|
|||
placeholders = ','.join(['?'] * len(id_values))
|
||||
cursor.execute(f"""
|
||||
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id,
|
||||
image_url, genres, popularity, similarity_rank
|
||||
FROM similar_artists
|
||||
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
|
||||
|
|
@ -763,7 +753,7 @@ def get_artist_map_explore():
|
|||
# Also search by name (the center artist might be a watchlist source)
|
||||
cursor.execute("""
|
||||
SELECT DISTINCT sa.similar_artist_name, sa.similar_artist_spotify_id,
|
||||
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id, sa.similar_artist_musicbrainz_id,
|
||||
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id,
|
||||
sa.image_url, sa.genres, sa.popularity, sa.similarity_rank
|
||||
FROM similar_artists sa
|
||||
JOIN watchlist_artists wa ON sa.source_artist_id = COALESCE(wa.spotify_artist_id, wa.itunes_artist_id, CAST(wa.id AS TEXT))
|
||||
|
|
@ -795,8 +785,7 @@ def get_artist_map_explore():
|
|||
image_url=sa.get('image_url'),
|
||||
genres=sa.get('genres'),
|
||||
popularity=sa.get('popularity', 0),
|
||||
similar_artist_deezer_id=sa.get('deezer_id'),
|
||||
similar_artist_musicbrainz_id=sa.get('musicbrainz_id'),
|
||||
similar_artist_deezer_id=sa.get('deezer_id')
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("similar artist insert failed: %s", e)
|
||||
|
|
@ -805,7 +794,7 @@ def get_artist_map_explore():
|
|||
placeholders = ','.join(['?'] * len(id_values))
|
||||
cursor.execute(f"""
|
||||
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id,
|
||||
image_url, genres, popularity, similarity_rank
|
||||
FROM similar_artists
|
||||
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
|
||||
|
|
@ -816,7 +805,7 @@ def get_artist_map_explore():
|
|||
# Fallback: query by name-based source ID
|
||||
cursor.execute("""
|
||||
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
|
||||
similar_artist_itunes_id, similar_artist_deezer_id,
|
||||
image_url, genres, popularity, similarity_rank
|
||||
FROM similar_artists
|
||||
WHERE source_artist_id = ? AND profile_id = ?
|
||||
|
|
@ -848,7 +837,6 @@ def get_artist_map_explore():
|
|||
'spotify_id': r['similar_artist_spotify_id'] or '',
|
||||
'itunes_id': r['similar_artist_itunes_id'] or '',
|
||||
'deezer_id': r['similar_artist_deezer_id'] or '',
|
||||
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
|
||||
'discogs_id': '',
|
||||
'popularity': r['popularity'] or 0,
|
||||
'rank': r['similarity_rank'] or 5,
|
||||
|
|
@ -869,8 +857,7 @@ def get_artist_map_explore():
|
|||
cursor.execute(f"""
|
||||
SELECT DISTINCT source_artist_id, similar_artist_name,
|
||||
similar_artist_spotify_id, similar_artist_itunes_id,
|
||||
similar_artist_deezer_id, similar_artist_musicbrainz_id,
|
||||
image_url, genres, popularity, similarity_rank
|
||||
similar_artist_deezer_id, image_url, genres, popularity, similarity_rank
|
||||
FROM similar_artists
|
||||
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
|
||||
ORDER BY similarity_rank ASC
|
||||
|
|
@ -911,7 +898,6 @@ def get_artist_map_explore():
|
|||
'spotify_id': r['similar_artist_spotify_id'] or '',
|
||||
'itunes_id': r['similar_artist_itunes_id'] or '',
|
||||
'deezer_id': r['similar_artist_deezer_id'] or '',
|
||||
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
|
||||
'discogs_id': '',
|
||||
'popularity': r['popularity'] or 0,
|
||||
'rank': r['similarity_rank'] or 5,
|
||||
|
|
@ -945,7 +931,7 @@ def get_artist_map_explore():
|
|||
except Exception as e:
|
||||
logger.debug("explorer node genres parse failed: %s", e)
|
||||
# Harvest missing IDs from cache
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
|
||||
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
|
||||
k = src_map.get(cr['source'])
|
||||
if k and not n.get(k):
|
||||
n[k] = cr['entity_id']
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ from datetime import datetime, timedelta
|
|||
from utils.logging_config import get_logger
|
||||
from database.music_database import MusicDatabase
|
||||
from core.audiodb_client import AudioDBClient
|
||||
from core.worker_utils import accept_artist_match, interruptible_sleep
|
||||
from core.worker_utils import interruptible_sleep
|
||||
|
||||
logger = get_logger("audiodb_worker")
|
||||
|
||||
|
|
@ -162,16 +162,6 @@ class AudioDBWorker:
|
|||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Pinned-group override (Manage Enrichment Workers): process one
|
||||
# entity type first, then fall through to the normal chain. Unset or
|
||||
# exhausted ⇒ default artist→album→track order, unchanged.
|
||||
from core.worker_utils import read_enrichment_priority, priority_pending_item
|
||||
_prio = read_enrichment_priority('audiodb')
|
||||
if _prio:
|
||||
_pi = priority_pending_item(cursor, 'audiodb', _prio)
|
||||
if _pi:
|
||||
return _pi
|
||||
|
||||
# Priority 1: Unattempted artists
|
||||
cursor.execute("""
|
||||
SELECT id, name
|
||||
|
|
@ -210,45 +200,42 @@ class AudioDBWorker:
|
|||
if row:
|
||||
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
|
||||
|
||||
# Priority 4: Retry 'not_found' OR 'error' artists after retry_days.
|
||||
# 'error' status covers transient AudioDB outages (timeouts, 500s)
|
||||
# that the issue-#553 fix marks rather than leaving NULL — without
|
||||
# this retry path those rows would stay errored forever.
|
||||
retry_cutoff = datetime.now() - timedelta(days=self.retry_days)
|
||||
# Priority 4: Retry 'not_found' artists after retry_days
|
||||
not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
|
||||
cursor.execute("""
|
||||
SELECT id, name
|
||||
FROM artists
|
||||
WHERE audiodb_match_status IN ('not_found', 'error') AND audiodb_last_attempted < ?
|
||||
WHERE audiodb_match_status = 'not_found' AND audiodb_last_attempted < ?
|
||||
ORDER BY audiodb_last_attempted ASC
|
||||
LIMIT 1
|
||||
""", (retry_cutoff,))
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
|
||||
return {'type': 'artist', 'id': row[0], 'name': row[1]}
|
||||
|
||||
# Priority 5: Retry 'not_found' OR 'error' albums
|
||||
# Priority 5: Retry 'not_found' albums
|
||||
cursor.execute("""
|
||||
SELECT a.id, a.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
|
||||
FROM albums a
|
||||
JOIN artists ar ON a.artist_id = ar.id
|
||||
WHERE a.audiodb_match_status IN ('not_found', 'error') AND a.audiodb_last_attempted < ?
|
||||
WHERE a.audiodb_match_status = 'not_found' AND a.audiodb_last_attempted < ?
|
||||
ORDER BY a.audiodb_last_attempted ASC
|
||||
LIMIT 1
|
||||
""", (retry_cutoff,))
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
|
||||
|
||||
# Priority 6: Retry 'not_found' OR 'error' tracks
|
||||
# Priority 6: Retry 'not_found' tracks
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
|
||||
FROM tracks t
|
||||
JOIN artists ar ON t.artist_id = ar.id
|
||||
WHERE t.audiodb_match_status IN ('not_found', 'error') AND t.audiodb_last_attempted < ?
|
||||
WHERE t.audiodb_match_status = 'not_found' AND t.audiodb_last_attempted < ?
|
||||
ORDER BY t.audiodb_last_attempted ASC
|
||||
LIMIT 1
|
||||
""", (retry_cutoff,))
|
||||
""", (not_found_cutoff,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
|
||||
|
|
@ -273,13 +260,8 @@ class AudioDBWorker:
|
|||
|
||||
def _verify_artist_id(self, item: Dict[str, Any], result: Dict[str, Any]) -> bool:
|
||||
"""Verify that the result's artist ID matches the parent artist's stored AudioDB ID.
|
||||
|
||||
If mismatched, the album/track search is more specific (uses artist+title),
|
||||
so we trust it and correct the parent artist's audiodb_id — BUT only when
|
||||
the result's artist *name* matches our parent artist. Without that guard,
|
||||
a collaboration/compilation (a track our library credits to one artist
|
||||
that lives on another artist's album) would stamp the wrong AudioDB id
|
||||
onto our artist. See the Deezer fix for the full write-up."""
|
||||
so we trust it and correct the parent artist's audiodb_id."""
|
||||
parent_audiodb_id = item.get('artist_audiodb_id')
|
||||
if not parent_audiodb_id:
|
||||
return True
|
||||
|
|
@ -289,18 +271,6 @@ class AudioDBWorker:
|
|||
return True
|
||||
|
||||
if str(result_artist_id) != str(parent_audiodb_id):
|
||||
parent_name = item.get('artist') or ''
|
||||
result_artist_name = result.get('strArtist') or ''
|
||||
if (result_artist_name and parent_name
|
||||
and not self._name_matches(parent_name, result_artist_name)):
|
||||
logger.info(
|
||||
f"Skipping artist-ID correction from {item['type']} "
|
||||
f"'{item['name']}': result artist '{result_artist_name}' "
|
||||
f"≠ parent '{parent_name}' (collab/compilation, not a "
|
||||
f"correction)"
|
||||
)
|
||||
return True
|
||||
|
||||
logger.info(
|
||||
f"Artist ID correction from {item['type']} '{item['name']}': "
|
||||
f"updating parent artist AudioDB ID from {parent_audiodb_id} to {result_artist_id}"
|
||||
|
|
@ -404,40 +374,22 @@ class AudioDBWorker:
|
|||
return
|
||||
except Exception as e:
|
||||
logger.warning(f"Direct lookup failed for existing AudioDB ID {existing_id}: {e}")
|
||||
# Direct lookup returned no metadata (None) or raised — don't
|
||||
# fall through to the name-search path below, which could
|
||||
# overwrite a manually-matched audiodb_id with a wrong guess.
|
||||
# Mark status='error' so the queue's NULL-status filter stops
|
||||
# re-picking this row on every tick (issue #553: AudioDB
|
||||
# `track.php` timeouts caused infinite enrichment loops as
|
||||
# the row was repeatedly picked + re-attempted because it
|
||||
# never left the NULL state). The error-retry priority block
|
||||
# in `_get_next_item` re-attempts after `retry_days` so
|
||||
# transient AudioDB outages still recover automatically.
|
||||
self._mark_status(item_type, item_id, 'error')
|
||||
self.stats['errors'] += 1
|
||||
logger.debug(
|
||||
f"Preserving manual match for {item_type} '{item_name}' "
|
||||
f"(AudioDB ID: {existing_id}); marked error pending retry"
|
||||
)
|
||||
# Direct lookup failed — don't overwrite manual match
|
||||
logger.debug(f"Preserving manual match for {item_type} '{item_name}' (AudioDB ID: {existing_id})")
|
||||
return
|
||||
|
||||
if item_type == 'artist':
|
||||
result = self.client.search_artist(item_name)
|
||||
if result:
|
||||
result_name = result.get('strArtist', '')
|
||||
ok, reason = accept_artist_match(
|
||||
self.db, 'audiodb_id', result.get('idArtist'), item_id,
|
||||
item_name, result_name,
|
||||
)
|
||||
if ok:
|
||||
if self._name_matches(item_name, result_name):
|
||||
self._update_artist(item_id, result)
|
||||
self.stats['matched'] += 1
|
||||
logger.info(f"Matched artist '{item_name}' -> AudioDB ID: {result.get('idArtist')}")
|
||||
else:
|
||||
self._mark_status('artist', item_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"Artist '{item_name}' not matched: {reason}")
|
||||
logger.debug(f"Name mismatch for artist '{item_name}' (got '{result_name}')")
|
||||
else:
|
||||
self._mark_status('artist', item_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,11 +1,7 @@
|
|||
"""Automation API + progress + handlers package.
|
||||
"""Automation API + progress tracking helpers package.
|
||||
|
||||
Lifted from web_server.py:
|
||||
- `/api/automations/*` route helpers → `api.py`
|
||||
- block library used by the trigger/action UI → `blocks.py`
|
||||
- progress tracker (init / update / finish) → `progress.py`
|
||||
- cross-handler signal bus → `signals.py`
|
||||
- per-action handler functions → `handlers/` subpackage (with
|
||||
`deps.py` defining the dependency-injection surface so handlers
|
||||
stay testable in isolation)
|
||||
Lifted from web_server.py /api/automations/* routes and progress
|
||||
emitters. The action handler registration (`_register_automation_handlers`)
|
||||
stays in web_server.py because each handler closure is tightly coupled
|
||||
to other application features.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -172,11 +172,9 @@ def create_automation(
|
|||
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
|
||||
|
||||
group_name = data.get('group_name') or None
|
||||
owned_by = data.get('owned_by') or None
|
||||
auto_id = database.create_automation(
|
||||
name, trigger_type, trigger_config, action_type, action_config,
|
||||
profile_id, notify_type, notify_config, then_actions_json, group_name,
|
||||
owned_by=owned_by,
|
||||
)
|
||||
if auto_id is None:
|
||||
return {'error': 'Failed to create automation'}, 500
|
||||
|
|
@ -219,8 +217,6 @@ def update_automation(
|
|||
update_fields['notify_config'] = json.dumps(data['notify_config'])
|
||||
if 'group_name' in data:
|
||||
update_fields['group_name'] = data['group_name'] or None
|
||||
if 'owned_by' in data:
|
||||
update_fields['owned_by'] = data['owned_by'] or None
|
||||
|
||||
if not update_fields:
|
||||
return {'error': 'No fields to update'}, 400
|
||||
|
|
@ -229,12 +225,6 @@ def update_automation(
|
|||
if cycle_path:
|
||||
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
|
||||
|
||||
# Schedule-shape changes must invalidate the stored next_run so the
|
||||
# scheduler recomputes it; otherwise restart-survival logic keeps the
|
||||
# leftover timestamp from the previous interval.
|
||||
if {'trigger_type', 'trigger_config'} & update_fields.keys():
|
||||
update_fields['next_run'] = None
|
||||
|
||||
success = database.update_automation(automation_id, **update_fields)
|
||||
if not success:
|
||||
return {'error': 'Automation not found'}, 404
|
||||
|
|
|
|||
|
|
@ -146,15 +146,6 @@ ACTIONS: list[dict] = [
|
|||
{"key": "all", "type": "checkbox", "label": "Process all mirrored playlists", "default": False},
|
||||
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
|
||||
]},
|
||||
{"type": "personalized_pipeline", "label": "Personalized Playlist Pipeline", "icon": "sparkles",
|
||||
"description": "Sync personalized / discover-page playlists (Hidden Gems, Time Machine, Fresh Tape, etc.) to your media server + queue missing tracks for download.",
|
||||
"available": True,
|
||||
"config_fields": [
|
||||
{"key": "kinds", "type": "personalized_playlist_select", "label": "Playlists to sync",
|
||||
"description": "Multi-select: Hidden Gems, Discovery Shuffle, Time Machine (per decade), Genre playlists, Fresh Tape, The Archives, Seasonal Mix (per season)"},
|
||||
{"key": "refresh_first", "type": "checkbox", "label": "Refresh playlists before sync (regenerate snapshots)", "default": False},
|
||||
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
|
||||
]},
|
||||
{"type": "notify_only", "label": "Notify Only", "icon": "bell", "description": "No action — just send notification", "available": True},
|
||||
# Phase 3 actions
|
||||
{"type": "start_database_update", "label": "Update Database", "icon": "database",
|
||||
|
|
@ -171,7 +162,12 @@ ACTIONS: list[dict] = [
|
|||
{"type": "update_discovery_pool", "label": "Update Discovery", "icon": "compass",
|
||||
"description": "Refresh discovery pool with new tracks", "available": True},
|
||||
{"type": "start_quality_scan", "label": "Run Quality Scan", "icon": "bar-chart",
|
||||
"description": "Run the Quality Upgrade Finder (scope is set in Library Maintenance)", "available": True},
|
||||
"description": "Scan for low-quality audio files", "available": True,
|
||||
"config_fields": [
|
||||
{"key": "scope", "type": "select", "label": "Scope",
|
||||
"options": [{"value": "watchlist", "label": "Watchlist Artists"}, {"value": "library", "label": "Full Library"}],
|
||||
"default": "watchlist"}
|
||||
]},
|
||||
{"type": "backup_database", "label": "Backup Database", "icon": "save",
|
||||
"description": "Create timestamped database backup", "available": True},
|
||||
{"type": "refresh_beatport_cache", "label": "Refresh Beatport Cache", "icon": "music",
|
||||
|
|
|
|||
|
|
@ -1,163 +0,0 @@
|
|||
"""Dependency-injection surface for automation handlers.
|
||||
|
||||
Each handler in ``core.automation.handlers`` is a top-level pure
|
||||
function that accepts ``(config: dict, deps: AutomationDeps)`` instead
|
||||
of reaching for module-level globals in ``web_server``. The deps
|
||||
namespace bundles every callable, client, and mutable-state container
|
||||
the handlers need.
|
||||
|
||||
Construction happens once at app startup in ``web_server.py``:
|
||||
|
||||
from core.automation.deps import AutomationDeps, AutomationState
|
||||
state = AutomationState()
|
||||
deps = AutomationDeps(
|
||||
engine=automation_engine,
|
||||
state=state,
|
||||
get_database=get_database,
|
||||
spotify_client=spotify_client,
|
||||
...
|
||||
)
|
||||
register_all(deps)
|
||||
|
||||
Tests construct a fake ``AutomationDeps`` with stub callables — every
|
||||
handler is then exercisable without spinning up Flask, the DB, or
|
||||
the real media-server clients.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class AutomationState:
|
||||
"""Mutable flags shared across handler invocations.
|
||||
|
||||
Pre-refactor each was a ``global`` or ``nonlocal`` variable inside
|
||||
the registration closure. Lifted here so handlers + their guards
|
||||
can read/write a single object instead of importing globals.
|
||||
|
||||
All mutations should hold ``lock``; the helper methods below do
|
||||
so for the common get/set patterns.
|
||||
"""
|
||||
|
||||
scan_library_automation_id: Optional[str] = None
|
||||
db_update_automation_id: Optional[str] = None
|
||||
pipeline_running: bool = False
|
||||
lock: threading.Lock = field(default_factory=threading.Lock)
|
||||
|
||||
def is_scan_library_active(self) -> bool:
|
||||
with self.lock:
|
||||
return self.scan_library_automation_id is not None
|
||||
|
||||
def is_pipeline_running(self) -> bool:
|
||||
with self.lock:
|
||||
return self.pipeline_running
|
||||
|
||||
def try_start_pipeline(self) -> bool:
|
||||
"""Atomically mark the shared playlist pipeline as running."""
|
||||
with self.lock:
|
||||
if self.pipeline_running:
|
||||
return False
|
||||
self.pipeline_running = True
|
||||
return True
|
||||
|
||||
def set_scan_library_id(self, automation_id: Optional[str]) -> None:
|
||||
with self.lock:
|
||||
self.scan_library_automation_id = automation_id
|
||||
|
||||
def set_pipeline_running(self, value: bool) -> None:
|
||||
with self.lock:
|
||||
self.pipeline_running = value
|
||||
|
||||
|
||||
@dataclass
|
||||
class AutomationDeps:
|
||||
"""Bundle of every callable + client an automation handler may need.
|
||||
|
||||
Add fields as new handlers are extracted. Every field is required
|
||||
at construction (no defaults) so a missing dep fails loudly at
|
||||
startup, not silently mid-handler.
|
||||
"""
|
||||
|
||||
# --- Engine + shared state ---
|
||||
engine: Any # AutomationEngine instance
|
||||
state: AutomationState
|
||||
config_manager: Any # config.settings.ConfigManager singleton
|
||||
update_progress: Callable[..., None] # _update_automation_progress
|
||||
logger: Any # module-level logger from utils.logging_config
|
||||
|
||||
# --- Service clients (each may be None depending on user config) ---
|
||||
get_database: Callable[[], Any] # late-binding so tests don't need DB
|
||||
spotify_client: Any
|
||||
tidal_client: Any
|
||||
web_scan_manager: Any
|
||||
|
||||
# --- Background-task entry points ---
|
||||
process_wishlist_automatically: Callable[..., Any]
|
||||
process_watchlist_scan_automatically: Callable[..., Any]
|
||||
is_wishlist_actually_processing: Callable[[], bool]
|
||||
is_watchlist_actually_scanning: Callable[[], bool]
|
||||
get_watchlist_scan_state: Callable[[], dict] # accessor returns the live mutable dict
|
||||
|
||||
# --- Playlist pipeline entry points ---
|
||||
run_playlist_discovery_worker: Callable[..., Any]
|
||||
run_sync_task: Callable[..., Any]
|
||||
run_playlist_organize_download: Callable[..., Dict[str, Any]]
|
||||
missing_download_executor: Any
|
||||
load_sync_status_file: Callable[[], dict]
|
||||
get_deezer_client: Callable[[], Any]
|
||||
parse_youtube_playlist: Callable[[str], Any]
|
||||
get_sync_states: Callable[[], dict] # accessor returns the live dict shared with the sync UI
|
||||
|
||||
# --- Database update + quality scanner (shared state + executors) ---
|
||||
set_db_update_automation_id: Callable[[Optional[str]], None] # syncs the legacy `_db_update_automation_id` global so the live DB-update progress callbacks (which still read the global directly) keep firing for the active automation
|
||||
get_db_update_state: Callable[[], dict]
|
||||
db_update_lock: Any # threading.Lock
|
||||
db_update_executor: Any # ThreadPoolExecutor
|
||||
run_db_update_task: Callable[..., Any]
|
||||
run_deep_scan_task: Callable[..., Any]
|
||||
get_duplicate_cleaner_state: Callable[[], dict]
|
||||
duplicate_cleaner_lock: Any
|
||||
duplicate_cleaner_executor: Any
|
||||
run_duplicate_cleaner: Callable[..., Any]
|
||||
# Triggers a "Run Now" of a library-maintenance repair job by id (e.g.
|
||||
# 'quality_upgrade'). Returns truthy if the job was queued. Replaces the old
|
||||
# standalone quality-scanner executor/state (the scanner is now a repair job).
|
||||
run_repair_job_now: Callable[[str], Any]
|
||||
|
||||
# --- Download orchestrator + queue accessors ---
|
||||
download_orchestrator: Any
|
||||
run_async: Callable[..., Any]
|
||||
tasks_lock: Any
|
||||
get_download_batches: Callable[[], dict]
|
||||
get_download_tasks: Callable[[], dict]
|
||||
sweep_empty_download_directories: Callable[[], int]
|
||||
get_staging_path: Callable[[], str]
|
||||
|
||||
# --- Maintenance helpers ---
|
||||
docker_resolve_path: Callable[[str], str]
|
||||
get_current_profile_id: Callable[[], int]
|
||||
get_watchlist_scanner: Callable[[Any], Any]
|
||||
get_app: Callable[[], Any] # Flask app for test_client (beatport refresh)
|
||||
get_beatport_data_cache: Callable[[], dict]
|
||||
|
||||
# --- Progress + history callbacks (used by register_all to wire
|
||||
# the engine's progress callback hooks). ---
|
||||
init_automation_progress: Callable[..., Any]
|
||||
record_progress_history: Callable[..., Any]
|
||||
|
||||
# --- Personalized playlist pipeline ---
|
||||
# Lazy builder so the pipeline handler can construct a fresh
|
||||
# PersonalizedPlaylistManager per run (cheap accessors inside,
|
||||
# no caching needed yet).
|
||||
build_personalized_manager: Callable[[], Any]
|
||||
|
||||
# --- Unified PlaylistSource registry ---
|
||||
# Optional so test fixtures that don't exercise refresh_mirrored
|
||||
# can keep their existing scaffolding. Production wiring in
|
||||
# ``web_server.py`` always populates it via
|
||||
# ``core.playlists.sources.bootstrap.build_playlist_source_registry``.
|
||||
playlist_source_registry: Optional[Any] = None
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
"""Per-action automation handlers.
|
||||
|
||||
Each module in this subpackage exposes one top-level handler function
|
||||
(or a small cluster of related handlers) of the form::
|
||||
|
||||
def auto_<action_name>(config: dict, deps: AutomationDeps) -> dict
|
||||
|
||||
The ``register_all`` helper in :mod:`registration` wires every handler
|
||||
to the engine in one place. ``web_server.py`` calls
|
||||
``register_all(deps)`` once at startup.
|
||||
"""
|
||||
|
||||
from core.automation.handlers.process_wishlist import auto_process_wishlist
|
||||
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
|
||||
from core.automation.handlers.scan_library import auto_scan_library
|
||||
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
|
||||
from core.automation.handlers.sync_playlist import auto_sync_playlist
|
||||
from core.automation.handlers.discover_playlist import auto_discover_playlist
|
||||
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
|
||||
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
|
||||
from core.automation.handlers.database_update import auto_start_database_update, auto_deep_scan_library
|
||||
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
|
||||
from core.automation.handlers.quality_scanner import auto_start_quality_scan
|
||||
from core.automation.handlers.maintenance import (
|
||||
auto_clear_quarantine,
|
||||
auto_cleanup_wishlist,
|
||||
auto_update_discovery_pool,
|
||||
auto_backup_database,
|
||||
auto_refresh_beatport_cache,
|
||||
)
|
||||
from core.automation.handlers.download_cleanup import (
|
||||
auto_clean_search_history,
|
||||
auto_clean_completed_downloads,
|
||||
auto_full_cleanup,
|
||||
)
|
||||
from core.automation.handlers.run_script import auto_run_script
|
||||
from core.automation.handlers.search_and_download import auto_search_and_download
|
||||
from core.automation.handlers.registration import register_all
|
||||
|
||||
__all__ = [
|
||||
'auto_process_wishlist',
|
||||
'auto_scan_watchlist',
|
||||
'auto_scan_library',
|
||||
'auto_refresh_mirrored',
|
||||
'auto_sync_playlist',
|
||||
'auto_discover_playlist',
|
||||
'auto_playlist_pipeline',
|
||||
'auto_personalized_pipeline',
|
||||
'auto_start_database_update',
|
||||
'auto_deep_scan_library',
|
||||
'auto_run_duplicate_cleaner',
|
||||
'auto_start_quality_scan',
|
||||
'auto_clear_quarantine',
|
||||
'auto_cleanup_wishlist',
|
||||
'auto_update_discovery_pool',
|
||||
'auto_backup_database',
|
||||
'auto_refresh_beatport_cache',
|
||||
'auto_clean_search_history',
|
||||
'auto_clean_completed_downloads',
|
||||
'auto_full_cleanup',
|
||||
'auto_run_script',
|
||||
'auto_search_and_download',
|
||||
'register_all',
|
||||
]
|
||||
|
|
@ -1,259 +0,0 @@
|
|||
"""Shared helpers between mirrored + personalized playlist pipelines.
|
||||
|
||||
Both pipelines end in the same shape:
|
||||
1. SYNC each playlist to the active media server.
|
||||
2. WISHLIST: trigger the wishlist processor for missing tracks.
|
||||
|
||||
The differing prefix (mirrored = REFRESH external sources + DISCOVER
|
||||
metadata; personalized = SNAPSHOT manager-backed playlists) is owned
|
||||
by each pipeline. This module owns the SYNC + WISHLIST tail so both
|
||||
pipelines stay consistent + DRY.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
# Per-playlist sync poll cap (mirrored side already used this).
|
||||
_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS = 600
|
||||
# Sync-status final-state markers.
|
||||
_SYNC_TERMINAL_STATUSES = ('finished', 'complete', 'error', 'failed')
|
||||
|
||||
|
||||
def run_sync_and_wishlist(
|
||||
deps: AutomationDeps,
|
||||
automation_id: Optional[str],
|
||||
playlists: List[Dict[str, Any]],
|
||||
*,
|
||||
sync_one_fn: Callable[[Dict[str, Any]], Dict[str, Any]],
|
||||
sync_id_for_fn: Callable[[Dict[str, Any]], str],
|
||||
skip_wishlist: bool = False,
|
||||
progress_start: int = 56,
|
||||
progress_end: int = 85,
|
||||
sync_phase_label: str = 'Phase: Syncing to server...',
|
||||
sync_phase_start_log: str = 'Sync',
|
||||
wishlist_phase_label: str = 'Phase: Processing wishlist...',
|
||||
wishlist_phase_start_log: str = 'Wishlist',
|
||||
) -> Dict[str, int]:
|
||||
"""Run the SYNC + WISHLIST tail of a playlist pipeline.
|
||||
|
||||
The caller supplies:
|
||||
- ``playlists``: list of playlist payload dicts. Each must have at
|
||||
least a ``name`` key (used in progress logs). The shape beyond
|
||||
``name`` is opaque to the helper — ``sync_one_fn`` receives the
|
||||
payload and returns a sync_result dict.
|
||||
- ``sync_one_fn(payload) -> sync_result``: launches sync for one
|
||||
playlist. Result dict must carry ``status`` ∈ ``('started',
|
||||
'skipped', 'error')`` and may carry ``reason``.
|
||||
- ``sync_id_for_fn(payload) -> str``: returns the sync-state key
|
||||
the helper polls on (so we can wait for the background sync
|
||||
thread to complete + read the matched_tracks count).
|
||||
|
||||
Returns ``{'synced': int, 'skipped': int, 'errors': int,
|
||||
'wishlist_queued': int}`` so the caller can stitch it into its
|
||||
final status.
|
||||
"""
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=progress_start,
|
||||
phase=sync_phase_label,
|
||||
log_line=sync_phase_start_log,
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
total_synced = 0
|
||||
total_skipped = 0
|
||||
sync_errors = 0
|
||||
sync_states = deps.get_sync_states()
|
||||
n_playlists = max(1, len(playlists))
|
||||
progress_span = max(1, progress_end - progress_start - 1)
|
||||
|
||||
for pl_idx, pl in enumerate(playlists):
|
||||
pl_name = pl.get('name', '')
|
||||
sync_result = sync_one_fn(pl)
|
||||
sync_status = sync_result.get('status', '')
|
||||
|
||||
if sync_status == 'started':
|
||||
sync_id = sync_id_for_fn(pl)
|
||||
sync_poll_start = time.time()
|
||||
timed_out = True
|
||||
while time.time() - sync_poll_start < _SYNC_PER_PLAYLIST_TIMEOUT_SECONDS:
|
||||
if (sync_id in sync_states
|
||||
and sync_states[sync_id].get('status') in _SYNC_TERMINAL_STATUSES):
|
||||
timed_out = False
|
||||
break
|
||||
time.sleep(2)
|
||||
elapsed = int(time.time() - sync_poll_start)
|
||||
sub_progress = progress_start + 1 + ((pl_idx + 1) / n_playlists) * progress_span
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=min(int(sub_progress), progress_end - 1),
|
||||
phase=f'{sync_phase_label.rstrip(".")} — "{pl_name}" ({elapsed}s)',
|
||||
)
|
||||
|
||||
ss = sync_states.get(sync_id, {})
|
||||
final_status = ss.get('status')
|
||||
if timed_out:
|
||||
sync_errors += 1
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Sync timed out "{pl_name}" after {_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS}s',
|
||||
log_type='error',
|
||||
)
|
||||
continue
|
||||
if final_status in ('error', 'failed'):
|
||||
sync_errors += 1
|
||||
reason = ss.get('error') or ss.get('reason') or 'background sync failed'
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Sync error "{pl_name}": {reason}',
|
||||
log_type='error',
|
||||
)
|
||||
continue
|
||||
|
||||
ss_result = ss.get('result', ss.get('progress', {}))
|
||||
matched = ss_result.get('matched_tracks', 0) if isinstance(ss_result, dict) else 0
|
||||
total_synced += int(matched) if matched else 0
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Synced "{pl_name}": {matched} tracks matched',
|
||||
log_type='success',
|
||||
)
|
||||
|
||||
elif sync_status == 'skipped':
|
||||
total_skipped += 1
|
||||
reason = sync_result.get('reason', 'unchanged')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Skipped "{pl_name}": {reason}',
|
||||
log_type='skip',
|
||||
)
|
||||
elif sync_status == 'error':
|
||||
sync_errors += 1
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Sync error "{pl_name}": {sync_result.get("reason", "unknown")}',
|
||||
log_type='error',
|
||||
)
|
||||
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=progress_end,
|
||||
phase=f'{sync_phase_label.rstrip(".")} complete',
|
||||
log_line=f'Sync done: {total_synced} matched, {total_skipped} skipped, {sync_errors} errors',
|
||||
log_type='success' if sync_errors == 0 else 'warning',
|
||||
)
|
||||
|
||||
organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
|
||||
organize_started = 0
|
||||
if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
|
||||
for pl in organize_playlists:
|
||||
pl_id = pl.get('id')
|
||||
if not pl_id:
|
||||
continue
|
||||
pl_name = pl.get('name', '')
|
||||
try:
|
||||
org_result = deps.run_playlist_organize_download(
|
||||
mirrored_playlist_id=int(pl_id),
|
||||
automation_id=automation_id,
|
||||
)
|
||||
if org_result.get('status') == 'started':
|
||||
organize_started += 1
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Organize download started for "{pl_name}"',
|
||||
log_type='success',
|
||||
)
|
||||
elif org_result.get('status') == 'skipped':
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
|
||||
log_type='skip',
|
||||
)
|
||||
except Exception as org_err: # noqa: BLE001
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Organize download error for "{pl_name}": {org_err}',
|
||||
log_type='warning',
|
||||
)
|
||||
|
||||
all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
|
||||
effective_skip_wishlist = skip_wishlist or all_organize
|
||||
|
||||
wishlist_queued = run_wishlist_phase(
|
||||
deps, automation_id,
|
||||
skip=effective_skip_wishlist,
|
||||
progress_pct=progress_end + 1,
|
||||
wishlist_phase_label=wishlist_phase_label,
|
||||
wishlist_phase_start_log=wishlist_phase_start_log,
|
||||
)
|
||||
|
||||
return {
|
||||
'synced': total_synced,
|
||||
'skipped': total_skipped,
|
||||
'errors': sync_errors,
|
||||
'wishlist_queued': wishlist_queued,
|
||||
'organize_downloads_started': organize_started,
|
||||
}
|
||||
|
||||
|
||||
def run_wishlist_phase(
|
||||
deps: AutomationDeps,
|
||||
automation_id: Optional[str],
|
||||
*,
|
||||
skip: bool,
|
||||
progress_pct: int,
|
||||
wishlist_phase_label: str = 'Phase: Processing wishlist...',
|
||||
wishlist_phase_start_log: str = 'Wishlist',
|
||||
) -> int:
|
||||
"""Trigger the wishlist processor unless skipped or already running.
|
||||
|
||||
Returns 1 when the processor was triggered, 0 otherwise. Errors are
|
||||
logged but never raised — wishlist failure should not abort the
|
||||
pipeline."""
|
||||
if skip:
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=progress_pct,
|
||||
log_line=f'{wishlist_phase_start_log}: skipped (disabled)',
|
||||
log_type='skip',
|
||||
)
|
||||
return 0
|
||||
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=progress_pct,
|
||||
phase=wishlist_phase_label,
|
||||
log_line=wishlist_phase_start_log,
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
try:
|
||||
if not deps.is_wishlist_actually_processing():
|
||||
deps.process_wishlist_automatically(automation_id=None)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line='Wishlist processing triggered',
|
||||
log_type='success',
|
||||
)
|
||||
return 1
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line='Wishlist already running — skipped',
|
||||
log_type='skip',
|
||||
)
|
||||
return 0
|
||||
except Exception as e: # noqa: BLE001 — wishlist failure must never abort pipeline
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Wishlist error: {e}',
|
||||
log_type='warning',
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
__all__ = ['run_sync_and_wishlist', 'run_wishlist_phase']
|
||||
|
|
@ -1,197 +0,0 @@
|
|||
"""Automation handlers: ``start_database_update`` and
|
||||
``deep_scan_library`` actions.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_start_database_update`` and ``_auto_deep_scan_library``
|
||||
closures). Both share the same ``db_update_state`` / executor / lock
|
||||
infrastructure -- the only difference is which task they submit
|
||||
(``run_db_update_task`` vs ``run_deep_scan_task``).
|
||||
|
||||
Pattern: pre-set state to running, submit task to executor, then
|
||||
poll the state dict until it transitions away from ``running``.
|
||||
Stall-detection emits a warning every 10 minutes when progress
|
||||
hasn't budged. 2-hour outer timeout caps the worst case.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
# Time out on STALL (no progress), not total runtime: a large library can scan
|
||||
# for many hours while progressing fine — a hard total cap would falsely mark a
|
||||
# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
|
||||
# up when progress hasn't moved for a long stretch, with a generous absolute
|
||||
# backstop against a truly stuck monitor loop.
|
||||
_STALL_WARNING_SECONDS = 600 # warn after 10 min with no progress (repeats)
|
||||
_STALL_TIMEOUT_SECONDS = 1800 # 30 min with no progress at all = genuinely stalled
|
||||
_ABSOLUTE_CAP_SECONDS = 86400 # 24h hard backstop (runaway-loop guard only)
|
||||
_POLL_INTERVAL_SECONDS = 3
|
||||
_INITIAL_DELAY_SECONDS = 1
|
||||
|
||||
|
||||
def scan_wait_action(
|
||||
*,
|
||||
status: str,
|
||||
idle_seconds: float,
|
||||
total_seconds: float,
|
||||
stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
|
||||
stall_warn_s: float = _STALL_WARNING_SECONDS,
|
||||
abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
|
||||
) -> str:
|
||||
"""Decide what the monitor loop should do on a poll tick (pure/testable).
|
||||
|
||||
``idle_seconds`` is time since progress last changed; ``total_seconds`` is
|
||||
time since the wait began. Returns one of:
|
||||
``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
|
||||
too long → give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
|
||||
(stalled long enough to warn but not give up), or ``'continue'``.
|
||||
|
||||
Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
|
||||
it never hits ``stall_timeout`` no matter how long the whole scan takes.
|
||||
"""
|
||||
if status != 'running':
|
||||
return 'finished'
|
||||
if total_seconds >= abs_cap_s:
|
||||
return 'abs_timeout'
|
||||
if idle_seconds >= stall_timeout_s:
|
||||
return 'stall_timeout'
|
||||
if idle_seconds >= stall_warn_s:
|
||||
return 'warn'
|
||||
return 'continue'
|
||||
|
||||
|
||||
def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run a full or incremental DB update via ``run_db_update_task``."""
|
||||
return _run_with_progress(
|
||||
config, deps,
|
||||
task=deps.run_db_update_task,
|
||||
task_args=(config.get('full_refresh', False), deps.config_manager.get_active_media_server()),
|
||||
initial_phase='Initializing...',
|
||||
stall_label='Database update',
|
||||
finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
|
||||
timeout_label='Database update timed out after 24 hours',
|
||||
)
|
||||
|
||||
|
||||
def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run a deep library scan via ``run_deep_scan_task``."""
|
||||
return _run_with_progress(
|
||||
config, deps,
|
||||
task=deps.run_deep_scan_task,
|
||||
task_args=(deps.config_manager.get_active_media_server(),),
|
||||
initial_phase='Deep scan: Initializing...',
|
||||
stall_label='Deep scan',
|
||||
finished_extras=lambda: {},
|
||||
timeout_label='Deep scan timed out after 24 hours',
|
||||
)
|
||||
|
||||
|
||||
def _run_with_progress(
|
||||
config: Dict[str, Any],
|
||||
deps: AutomationDeps,
|
||||
*,
|
||||
task,
|
||||
task_args: tuple,
|
||||
initial_phase: str,
|
||||
stall_label: str,
|
||||
finished_extras,
|
||||
timeout_label: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Shared poll-and-wait body for both DB-update handlers."""
|
||||
automation_id = config.get('_automation_id')
|
||||
state = deps.get_db_update_state()
|
||||
if state.get('status') == 'running':
|
||||
return {'status': 'skipped', 'reason': 'Database update already running'}
|
||||
deps.state.db_update_automation_id = automation_id
|
||||
# Sync legacy module global so the DB-update progress callbacks
|
||||
# (still living in web_server.py) emit against this automation.
|
||||
deps.set_db_update_automation_id(automation_id)
|
||||
|
||||
with deps.db_update_lock:
|
||||
state.update({
|
||||
'status': 'running', 'phase': initial_phase,
|
||||
'progress': 0, 'current_item': '', 'processed': 0, 'total': 0,
|
||||
'error_message': '',
|
||||
})
|
||||
deps.db_update_executor.submit(task, *task_args)
|
||||
|
||||
# Monitor progress (callbacks handle card updates, we just block until done).
|
||||
# We time out on STALL, not total runtime: ``processed`` advances on every
|
||||
# artist, so an actively-progressing scan keeps resetting the idle clock and
|
||||
# is never falsely failed no matter how long the whole library takes.
|
||||
time.sleep(_INITIAL_DELAY_SECONDS)
|
||||
poll_start = time.time()
|
||||
last_progress_time = time.time()
|
||||
# Any of these advancing means the scan is alive. current_item (the artist
|
||||
# being processed) changes every artist even when the rounded progress %
|
||||
# holds steady, so it guards against a false stall during slow stretches.
|
||||
last_progress_val = (0, 0, '')
|
||||
last_warn_time = 0.0
|
||||
outcome = 'finished'
|
||||
while True:
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
now = time.time()
|
||||
with deps.db_update_lock:
|
||||
current_status = state.get('status', 'idle')
|
||||
current_val = (state.get('processed', 0), state.get('progress', 0),
|
||||
state.get('current_item', ''))
|
||||
if current_val != last_progress_val:
|
||||
last_progress_val = current_val
|
||||
last_progress_time = now
|
||||
|
||||
action = scan_wait_action(
|
||||
status=current_status,
|
||||
idle_seconds=now - last_progress_time,
|
||||
total_seconds=now - poll_start,
|
||||
)
|
||||
if action in ('finished', 'stall_timeout', 'abs_timeout'):
|
||||
outcome = action
|
||||
break
|
||||
if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
|
||||
idle_min = int((now - last_progress_time) / 60)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
|
||||
log_type='warning',
|
||||
)
|
||||
last_warn_time = now
|
||||
|
||||
if outcome == 'stall_timeout':
|
||||
deps.update_progress(
|
||||
automation_id, status='error', phase='Stalled',
|
||||
log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
|
||||
if outcome == 'abs_timeout':
|
||||
deps.update_progress(
|
||||
automation_id, status='error',
|
||||
phase='Timed out', log_line=timeout_label, log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
|
||||
|
||||
# Finished/error callback already updated the card — return matching status.
|
||||
with deps.db_update_lock:
|
||||
final_status = state.get('status', 'unknown')
|
||||
if final_status == 'error':
|
||||
return {
|
||||
'status': 'error',
|
||||
'reason': state.get('error_message', 'Unknown error'),
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
with deps.db_update_lock:
|
||||
stats = {
|
||||
'status': 'completed', '_manages_own_progress': True,
|
||||
'artists': state.get('total', 0),
|
||||
'albums': state.get('total_albums', 0),
|
||||
'tracks': state.get('total_tracks', 0),
|
||||
'removed_artists': state.get('removed_artists', 0),
|
||||
'removed_albums': state.get('removed_albums', 0),
|
||||
'removed_tracks': state.get('removed_tracks', 0),
|
||||
}
|
||||
stats.update(finished_extras())
|
||||
return stats
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
"""Automation handler: ``discover_playlist`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_discover_playlist`` closure). Kicks off background discovery
|
||||
of official Spotify / iTunes metadata for mirrored playlist tracks.
|
||||
The worker runs in a daemon thread and emits its own progress; this
|
||||
handler returns immediately after launching it (``_manages_own_progress``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_discover_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Discover official Spotify/iTunes metadata for mirrored
|
||||
playlist tracks. Runs the worker in a background thread."""
|
||||
db = deps.get_database()
|
||||
playlist_id = config.get('playlist_id')
|
||||
discover_all = config.get('all', False)
|
||||
|
||||
if discover_all:
|
||||
playlists = db.get_mirrored_playlists()
|
||||
elif playlist_id:
|
||||
p = db.get_mirrored_playlist(int(playlist_id))
|
||||
playlists = [p] if p else []
|
||||
else:
|
||||
return {'status': 'error', 'reason': 'No playlist specified'}
|
||||
|
||||
if not playlists:
|
||||
return {'status': 'error', 'reason': 'No playlists found'}
|
||||
|
||||
threading.Thread(
|
||||
target=deps.run_playlist_discovery_worker,
|
||||
args=(playlists, config.get('_automation_id')),
|
||||
daemon=True,
|
||||
name='auto-discover-playlist',
|
||||
).start()
|
||||
names = ', '.join(p['name'] for p in playlists[:3])
|
||||
return {
|
||||
'status': 'started',
|
||||
'playlist_count': str(len(playlists)),
|
||||
'playlists': names,
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
|
|
@ -1,267 +0,0 @@
|
|||
"""Automation handlers: download-queue cleanup actions.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers``:
|
||||
- ``clean_search_history`` → :func:`auto_clean_search_history`
|
||||
- ``clean_completed_downloads`` → :func:`auto_clean_completed_downloads`
|
||||
- ``full_cleanup`` → :func:`auto_full_cleanup`
|
||||
|
||||
All three share the download-orchestrator + tasks_lock /
|
||||
download_batches / download_tasks accessors. ``full_cleanup`` is a
|
||||
multi-step orchestration that pulls in quarantine purge + staging
|
||||
sweep on top of the queue cleanup -- kept as one big handler since
|
||||
its phases share state-detection logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil as _shutil
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
# ─── clean_search_history ────────────────────────────────────────────
|
||||
|
||||
|
||||
def auto_clean_search_history(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Remove old searches from Soulseek when configured."""
|
||||
automation_id = config.get('_automation_id')
|
||||
# Skip if soulseek is not the active download source or in hybrid order.
|
||||
dl_mode = deps.config_manager.get('download_source.mode', 'hybrid')
|
||||
hybrid_order = deps.config_manager.get(
|
||||
'download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'],
|
||||
)
|
||||
soulseek_active = (
|
||||
dl_mode == 'soulseek'
|
||||
or (dl_mode == 'hybrid' and 'soulseek' in hybrid_order)
|
||||
)
|
||||
# Reach the underlying SoulseekClient via the orchestrator's
|
||||
# generic accessor.
|
||||
slskd = deps.download_orchestrator.client('soulseek') if deps.download_orchestrator else None
|
||||
if not soulseek_active or not slskd or not slskd.base_url:
|
||||
deps.update_progress(automation_id, log_line='Soulseek not active — skipped', log_type='skip')
|
||||
return {'status': 'skipped'}
|
||||
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Auto-clear disabled in settings', log_type='skip',
|
||||
)
|
||||
return {'status': 'skipped'}
|
||||
try:
|
||||
success = deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
|
||||
keep_searches=50, trigger_threshold=200,
|
||||
))
|
||||
if success:
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line='Search history maintenance completed',
|
||||
log_type='success',
|
||||
)
|
||||
return {'status': 'completed'}
|
||||
else:
|
||||
deps.update_progress(automation_id, log_line='No cleanup needed', log_type='skip')
|
||||
return {'status': 'completed'}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise
|
||||
return {'status': 'error', 'error': str(e)}
|
||||
|
||||
|
||||
# ─── clean_completed_downloads ───────────────────────────────────────
|
||||
|
||||
|
||||
def auto_clean_completed_downloads(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Clear completed downloads + sweep empty download directories.
|
||||
Skips when active batches or post-processing is in flight."""
|
||||
automation_id = config.get('_automation_id')
|
||||
try:
|
||||
has_active_batches = False
|
||||
has_post_processing = False
|
||||
with deps.tasks_lock:
|
||||
batches = deps.get_download_batches()
|
||||
for batch_data in batches.values():
|
||||
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
|
||||
has_active_batches = True
|
||||
break
|
||||
if not has_active_batches:
|
||||
tasks = deps.get_download_tasks()
|
||||
for task_data in tasks.values():
|
||||
if task_data.get('status') == 'post_processing':
|
||||
has_post_processing = True
|
||||
break
|
||||
|
||||
if has_active_batches:
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Skipped — downloads active', log_type='skip',
|
||||
)
|
||||
return {'status': 'completed'}
|
||||
|
||||
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
|
||||
if not has_post_processing:
|
||||
deps.sweep_empty_download_directories()
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Download cleanup completed', log_type='success',
|
||||
)
|
||||
return {'status': 'completed'}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise
|
||||
return {'status': 'error', 'reason': str(e)}
|
||||
|
||||
|
||||
# ─── full_cleanup ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def auto_full_cleanup(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run all cleanup tasks: quarantine purge → download queue clear
|
||||
→ empty-dir sweep → staging sweep → search history."""
|
||||
automation_id = config.get('_automation_id')
|
||||
steps = []
|
||||
|
||||
# --- 1. Clear quarantine ---
|
||||
deps.update_progress(automation_id, phase='Clearing quarantine...', progress=0)
|
||||
quarantine_path = os.path.join(
|
||||
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
|
||||
'ss_quarantine',
|
||||
)
|
||||
q_removed = 0
|
||||
if os.path.exists(quarantine_path):
|
||||
for f in os.listdir(quarantine_path):
|
||||
fp = os.path.join(quarantine_path, f)
|
||||
try:
|
||||
if os.path.isfile(fp):
|
||||
os.remove(fp)
|
||||
q_removed += 1
|
||||
elif os.path.isdir(fp):
|
||||
_shutil.rmtree(fp)
|
||||
q_removed += 1
|
||||
except Exception as e: # noqa: BLE001 — best-effort purge
|
||||
deps.logger.debug("quarantine entry purge failed: %s", e)
|
||||
steps.append(f'Quarantine: removed {q_removed} items')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Quarantine: removed {q_removed} items',
|
||||
log_type='success' if q_removed else 'info',
|
||||
)
|
||||
|
||||
# --- 2. Clear completed/errored/cancelled downloads from Soulseek queue ---
|
||||
deps.update_progress(automation_id, phase='Clearing download queue...', progress=20)
|
||||
has_active_batches = False
|
||||
has_post_processing = False
|
||||
with deps.tasks_lock:
|
||||
batches = deps.get_download_batches()
|
||||
for batch_data in batches.values():
|
||||
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
|
||||
has_active_batches = True
|
||||
break
|
||||
if not has_active_batches:
|
||||
tasks = deps.get_download_tasks()
|
||||
for task_data in tasks.values():
|
||||
if task_data.get('status') == 'post_processing':
|
||||
has_post_processing = True
|
||||
break
|
||||
if has_active_batches:
|
||||
steps.append('Download queue: skipped (active batches)')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line='Download queue: skipped (active batches)',
|
||||
log_type='skip',
|
||||
)
|
||||
else:
|
||||
try:
|
||||
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
|
||||
steps.append('Download queue: cleared')
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Download queue: cleared', log_type='success',
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 — per-step best-effort
|
||||
steps.append(f'Download queue: error ({e})')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Download queue: error ({e})',
|
||||
log_type='error',
|
||||
)
|
||||
|
||||
# --- 3. Sweep empty download directories ---
|
||||
deps.update_progress(automation_id, phase='Sweeping empty directories...', progress=40)
|
||||
if has_active_batches or has_post_processing:
|
||||
reason = 'active batches' if has_active_batches else 'post-processing active'
|
||||
steps.append(f'Empty directories: skipped ({reason})')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Empty directories: skipped ({reason})',
|
||||
log_type='skip',
|
||||
)
|
||||
else:
|
||||
dirs_removed = deps.sweep_empty_download_directories()
|
||||
steps.append(f'Empty directories: removed {dirs_removed}')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Empty directories: removed {dirs_removed}',
|
||||
log_type='success' if dirs_removed else 'info',
|
||||
)
|
||||
|
||||
# --- 4. Sweep empty staging directories ---
|
||||
deps.update_progress(automation_id, phase='Sweeping import folder...', progress=60)
|
||||
staging_path = deps.get_staging_path()
|
||||
s_removed = 0
|
||||
if os.path.isdir(staging_path):
|
||||
for dirpath, _dirnames, _filenames in os.walk(staging_path, topdown=False):
|
||||
if os.path.normpath(dirpath) == os.path.normpath(staging_path):
|
||||
continue
|
||||
try:
|
||||
entries = os.listdir(dirpath)
|
||||
except OSError:
|
||||
continue
|
||||
visible = [e for e in entries if not e.startswith('.')]
|
||||
if not visible:
|
||||
for hidden in entries:
|
||||
try:
|
||||
os.remove(os.path.join(dirpath, hidden))
|
||||
except Exception as e: # noqa: BLE001 — best-effort
|
||||
deps.logger.debug("hidden file cleanup failed: %s", e)
|
||||
try:
|
||||
os.rmdir(dirpath)
|
||||
s_removed += 1
|
||||
except OSError:
|
||||
pass
|
||||
steps.append(f'Staging: removed {s_removed} empty directories')
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Staging: removed {s_removed} empty directories',
|
||||
log_type='success' if s_removed else 'info',
|
||||
)
|
||||
|
||||
# --- 5. Clean search history (if enabled) ---
|
||||
deps.update_progress(automation_id, phase='Cleaning search history...', progress=80)
|
||||
try:
|
||||
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
|
||||
steps.append('Search cleanup: disabled in settings')
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Search cleanup: disabled in settings', log_type='skip',
|
||||
)
|
||||
else:
|
||||
deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
|
||||
keep_searches=50, trigger_threshold=200,
|
||||
))
|
||||
steps.append('Search history: cleaned')
|
||||
deps.update_progress(
|
||||
automation_id, log_line='Search history: cleaned', log_type='success',
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 — per-step best-effort
|
||||
steps.append(f'Search history: error ({e})')
|
||||
deps.update_progress(
|
||||
automation_id, log_line=f'Search history: error ({e})', log_type='error',
|
||||
)
|
||||
|
||||
total_removed = q_removed + s_removed
|
||||
deps.update_progress(
|
||||
automation_id, status='finished', progress=100,
|
||||
phase='Complete',
|
||||
log_line=f'Full cleanup complete — {total_removed} items removed',
|
||||
log_type='success',
|
||||
)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'quarantine_removed': str(q_removed),
|
||||
'staging_removed': str(s_removed),
|
||||
'total_removed': str(total_removed),
|
||||
'steps': steps,
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
"""Automation handler: ``run_duplicate_cleaner`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_run_duplicate_cleaner`` closure). Submits the duplicate
|
||||
cleaner to its executor, then polls the shared state dict until
|
||||
the worker transitions away from ``running``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
_TIMEOUT_SECONDS = 7200 # 2 hours
|
||||
_POLL_INTERVAL_SECONDS = 3
|
||||
_INITIAL_DELAY_SECONDS = 1
|
||||
|
||||
|
||||
def auto_run_duplicate_cleaner(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Kick off the duplicate cleaner and report final stats."""
|
||||
automation_id = config.get('_automation_id')
|
||||
state = deps.get_duplicate_cleaner_state()
|
||||
if state.get('status') == 'running':
|
||||
return {'status': 'skipped', 'reason': 'Duplicate cleaner already running'}
|
||||
|
||||
# Pre-set status before submit so the polling loop doesn't see a
|
||||
# stale 'finished' from a previous run.
|
||||
with deps.duplicate_cleaner_lock:
|
||||
state['status'] = 'running'
|
||||
deps.duplicate_cleaner_executor.submit(deps.run_duplicate_cleaner)
|
||||
deps.update_progress(automation_id, log_line='Duplicate cleaner started', log_type='info')
|
||||
|
||||
# Monitor progress (max 2 hours).
|
||||
time.sleep(_INITIAL_DELAY_SECONDS)
|
||||
poll_start = time.time()
|
||||
while time.time() - poll_start < _TIMEOUT_SECONDS:
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
current_status = state.get('status', 'idle')
|
||||
if current_status not in ('running',):
|
||||
break
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
phase=state.get('phase', 'Scanning...'),
|
||||
progress=state.get('progress', 0),
|
||||
processed=state.get('files_scanned', 0),
|
||||
total=state.get('total_files', 0),
|
||||
)
|
||||
else:
|
||||
# 2-hour timeout reached.
|
||||
deps.update_progress(
|
||||
automation_id, status='error',
|
||||
phase='Timed out',
|
||||
log_line='Duplicate cleaner timed out after 2 hours',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
|
||||
|
||||
# Check actual exit status (could be 'finished' or 'error').
|
||||
final_status = state.get('status', 'idle')
|
||||
if final_status == 'error':
|
||||
err = state.get('error_message', 'Unknown error')
|
||||
deps.update_progress(
|
||||
automation_id, status='error', progress=100,
|
||||
phase='Error', log_line=err, log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': err, '_manages_own_progress': True}
|
||||
|
||||
dupes = state.get('duplicates_found', 0)
|
||||
removed = state.get('deleted', 0)
|
||||
space_freed = state.get('space_freed', 0)
|
||||
scanned = state.get('files_scanned', 0)
|
||||
deps.update_progress(
|
||||
automation_id, status='finished', progress=100,
|
||||
phase='Complete',
|
||||
log_line=f'Found {dupes} duplicates, removed {removed} files',
|
||||
log_type='success',
|
||||
)
|
||||
return {
|
||||
'status': 'completed', '_manages_own_progress': True,
|
||||
'files_scanned': scanned,
|
||||
'duplicates_found': dupes,
|
||||
'files_deleted': removed,
|
||||
'space_freed_mb': round(space_freed / (1024 * 1024), 1),
|
||||
}
|
||||
|
|
@ -1,221 +0,0 @@
|
|||
"""Automation handlers: short maintenance actions.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers``:
|
||||
- ``clear_quarantine`` → :func:`auto_clear_quarantine`
|
||||
- ``cleanup_wishlist`` → :func:`auto_cleanup_wishlist`
|
||||
- ``update_discovery_pool`` → :func:`auto_update_discovery_pool`
|
||||
- ``backup_database`` → :func:`auto_backup_database`
|
||||
- ``refresh_beatport_cache`` → :func:`auto_refresh_beatport_cache`
|
||||
|
||||
Each is a thin wrapper around an existing service / helper. Grouped
|
||||
in one module because every body is short and they share no state
|
||||
between them — splitting into per-handler files would just add
|
||||
import noise.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import glob as _glob
|
||||
import os
|
||||
import shutil as _shutil
|
||||
import sqlite3
|
||||
import time
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
# ─── clear_quarantine ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def auto_clear_quarantine(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Purge every file/folder under the configured ss_quarantine path."""
|
||||
automation_id = config.get('_automation_id')
|
||||
quarantine_path = os.path.join(
|
||||
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
|
||||
'ss_quarantine',
|
||||
)
|
||||
if not os.path.exists(quarantine_path):
|
||||
deps.update_progress(automation_id, log_line='No quarantine folder found', log_type='info')
|
||||
return {'status': 'completed', 'removed': '0'}
|
||||
removed = 0
|
||||
for f in os.listdir(quarantine_path):
|
||||
fp = os.path.join(quarantine_path, f)
|
||||
try:
|
||||
if os.path.isfile(fp):
|
||||
os.remove(fp)
|
||||
removed += 1
|
||||
elif os.path.isdir(fp):
|
||||
_shutil.rmtree(fp)
|
||||
removed += 1
|
||||
except Exception as e: # noqa: BLE001 — best-effort purge
|
||||
deps.logger.debug("quarantine entry purge failed: %s", e)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Removed {removed} quarantined items',
|
||||
log_type='success' if removed > 0 else 'info',
|
||||
)
|
||||
return {'status': 'completed', 'removed': str(removed)}
|
||||
|
||||
|
||||
# ─── cleanup_wishlist ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def auto_cleanup_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Drop duplicate entries from the wishlist for the active profile."""
|
||||
automation_id = config.get('_automation_id')
|
||||
db = deps.get_database()
|
||||
removed = db.remove_wishlist_duplicates(deps.get_current_profile_id())
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Removed {removed or 0} duplicate wishlist entries',
|
||||
log_type='success' if removed else 'info',
|
||||
)
|
||||
return {'status': 'completed', 'removed': str(removed or 0)}
|
||||
|
||||
|
||||
# ─── update_discovery_pool ───────────────────────────────────────────
|
||||
|
||||
|
||||
def auto_update_discovery_pool(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run an incremental refresh of the discovery pool via the
|
||||
watchlist scanner."""
|
||||
automation_id = config.get('_automation_id')
|
||||
try:
|
||||
scanner = deps.get_watchlist_scanner(deps.spotify_client)
|
||||
deps.update_progress(automation_id, log_line='Updating discovery pool...', log_type='info')
|
||||
scanner.update_discovery_pool_incremental(deps.get_current_profile_id())
|
||||
deps.update_progress(
|
||||
automation_id, status='finished', progress=100,
|
||||
phase='Complete', log_line='Discovery pool updated', log_type='success',
|
||||
)
|
||||
return {'status': 'completed', '_manages_own_progress': True}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise
|
||||
deps.update_progress(
|
||||
automation_id, status='error',
|
||||
phase='Error', log_line=str(e), log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': str(e), '_manages_own_progress': True}
|
||||
|
||||
|
||||
# ─── backup_database ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
_MAX_BACKUPS = 5
|
||||
|
||||
|
||||
def auto_backup_database(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Create a hot SQLite backup, then prune old backups so only the
|
||||
newest ``_MAX_BACKUPS`` remain."""
|
||||
automation_id = config.get('_automation_id')
|
||||
db_path = os.environ.get('DATABASE_PATH', 'database/music_library.db')
|
||||
if not os.path.exists(db_path):
|
||||
return {'status': 'error', 'reason': 'Database file not found'}
|
||||
|
||||
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
backup_path = f"{db_path}.backup_{timestamp}"
|
||||
# safe_backup verifies source + result integrity, so an automated backup
|
||||
# can never silently snapshot a corrupt DB (the incident where every
|
||||
# rolling backup faithfully copied the corruption).
|
||||
from core.db_integrity import DBIntegrityError, safe_backup, prune_backups
|
||||
try:
|
||||
safe_backup(db_path, backup_path)
|
||||
except DBIntegrityError as integ:
|
||||
deps.logger.error("Auto-backup refused — DB integrity check failed: %s", integ)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Backup SKIPPED — database failed integrity check: {integ}',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': f'Database integrity check failed: {integ}'}
|
||||
size_mb = round(os.path.getsize(backup_path) / (1024 * 1024), 1)
|
||||
|
||||
# Rolling cleanup — never evict the most-recent verified-healthy backup.
|
||||
existing = list(_glob.glob(f"{db_path}.backup_*"))
|
||||
for removed in prune_backups(existing, _MAX_BACKUPS):
|
||||
try:
|
||||
os.remove(removed)
|
||||
except Exception as e: # noqa: BLE001 — best-effort cleanup
|
||||
deps.logger.debug("rolling backup cleanup failed: %s", e)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Backup created: {size_mb}MB ({os.path.basename(backup_path)})',
|
||||
log_type='success',
|
||||
)
|
||||
return {'status': 'completed', 'backup_path': backup_path, 'size_mb': str(size_mb)}
|
||||
|
||||
|
||||
# ─── refresh_beatport_cache ──────────────────────────────────────────
|
||||
|
||||
|
||||
_BEATPORT_SECTIONS = (
|
||||
('hero_tracks', '/api/beatport/hero-tracks', 'Hero Tracks'),
|
||||
('new_releases', '/api/beatport/new-releases', 'New Releases'),
|
||||
('featured_charts', '/api/beatport/featured-charts', 'Featured Charts'),
|
||||
('dj_charts', '/api/beatport/dj-charts', 'DJ Charts'),
|
||||
('top_10_lists', '/api/beatport/homepage/top-10-lists', 'Top 10 Lists'),
|
||||
('top_10_releases', '/api/beatport/homepage/top-10-releases-cards', 'Top 10 Releases'),
|
||||
('hype_picks', '/api/beatport/hype-picks', 'Hype Picks'),
|
||||
)
|
||||
|
||||
|
||||
def auto_refresh_beatport_cache(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Refresh Beatport homepage cache by calling each endpoint internally
|
||||
via Flask's ``test_client``. Invalidates the homepage cache first
|
||||
so endpoints re-scrape rather than returning stale data."""
|
||||
automation_id = config.get('_automation_id')
|
||||
cache = deps.get_beatport_data_cache()
|
||||
# Invalidate all homepage cache timestamps so endpoints re-scrape.
|
||||
with cache['cache_lock']:
|
||||
for key in cache['homepage']:
|
||||
cache['homepage'][key]['timestamp'] = 0
|
||||
cache['homepage'][key]['data'] = None
|
||||
|
||||
refreshed = 0
|
||||
errors = []
|
||||
app = deps.get_app()
|
||||
with app.test_client() as client:
|
||||
for idx, (_, endpoint, label) in enumerate(_BEATPORT_SECTIONS):
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=(idx / len(_BEATPORT_SECTIONS)) * 100,
|
||||
phase=f'Scraping: {label}',
|
||||
current_item=label,
|
||||
)
|
||||
try:
|
||||
resp = client.get(endpoint)
|
||||
if resp.status_code == 200:
|
||||
refreshed += 1
|
||||
deps.update_progress(
|
||||
automation_id, log_line=f'{label}: cached', log_type='success',
|
||||
)
|
||||
else:
|
||||
errors.append(label)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'{label}: HTTP {resp.status_code}',
|
||||
log_type='error',
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 — per-section best-effort
|
||||
errors.append(label)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'{label}: {str(e)}',
|
||||
log_type='error',
|
||||
)
|
||||
if idx < len(_BEATPORT_SECTIONS) - 1:
|
||||
time.sleep(2)
|
||||
|
||||
deps.update_progress(
|
||||
automation_id, status='finished', progress=100,
|
||||
phase='Complete',
|
||||
log_line=f'Refreshed {refreshed}/{len(_BEATPORT_SECTIONS)} sections',
|
||||
log_type='success',
|
||||
)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'refreshed': str(refreshed),
|
||||
'errors': str(len(errors)),
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
|
|
@ -1,356 +0,0 @@
|
|||
"""Personalized Playlist Pipeline automation handler.
|
||||
|
||||
Sibling to ``auto_playlist_pipeline`` (mirrored). Where the mirrored
|
||||
pipeline runs REFRESH external sources → DISCOVER metadata → SYNC →
|
||||
WISHLIST, the personalized pipeline is simpler:
|
||||
|
||||
SNAPSHOT → SYNC → WISHLIST
|
||||
|
||||
SNAPSHOT reads the persisted track list from
|
||||
``PersonalizedPlaylistManager``. When ``refresh_first=True`` (config),
|
||||
each playlist is refreshed BEFORE syncing — useful when the user
|
||||
wants the cron to capture a fresh-each-run view (e.g. "give me a new
|
||||
Hidden Gems set every night"). Default is to sync the existing
|
||||
snapshot, on the assumption the user / a separate cron has already
|
||||
refreshed when they wanted to.
|
||||
|
||||
Config schema:
|
||||
{
|
||||
'kinds': [
|
||||
{'kind': 'hidden_gems'},
|
||||
{'kind': 'time_machine', 'variant': '1980s'},
|
||||
{'kind': 'seasonal_mix', 'variant': 'halloween'},
|
||||
...
|
||||
],
|
||||
'refresh_first': bool, # default false
|
||||
'skip_wishlist': bool, # default false
|
||||
}
|
||||
|
||||
Each kind dict has at minimum ``kind``; ``variant`` is required for
|
||||
kinds that need it (time_machine, genre_playlist, daily_mix,
|
||||
seasonal_mix). Singleton kinds (hidden_gems, discovery_shuffle,
|
||||
popular_picks, fresh_tape, archives) ignore variant.
|
||||
|
||||
Pipeline-running flag (``deps.state.pipeline_running``) is shared
|
||||
with the mirrored pipeline so the two can't overlap. (One sync
|
||||
queue, one wishlist worker — overlapping triggers would step on
|
||||
each other.)"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
|
||||
|
||||
|
||||
# Sync state key prefix so personalized syncs don't collide with
|
||||
# mirrored ones (`auto_mirror_<id>`).
|
||||
_SYNC_ID_PREFIX = 'auto_personalized'
|
||||
|
||||
|
||||
def auto_personalized_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run SNAPSHOT → SYNC → WISHLIST for selected personalized playlists."""
|
||||
deps.state.set_pipeline_running(True)
|
||||
automation_id = config.get('_automation_id')
|
||||
pipeline_start = time.time()
|
||||
|
||||
try:
|
||||
kinds_config = config.get('kinds') or []
|
||||
if not isinstance(kinds_config, list) or not kinds_config:
|
||||
deps.state.set_pipeline_running(False)
|
||||
return {
|
||||
'status': 'error',
|
||||
'error': 'No personalized playlist kinds selected',
|
||||
}
|
||||
|
||||
refresh_first = bool(config.get('refresh_first', False))
|
||||
skip_wishlist = bool(config.get('skip_wishlist', False))
|
||||
|
||||
manager = deps.build_personalized_manager()
|
||||
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=2,
|
||||
phase=f'Personalized pipeline: {len(kinds_config)} playlist(s)',
|
||||
log_line=f'Starting pipeline for {len(kinds_config)} playlist(s)',
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
# ── PHASE 1: SNAPSHOT (optionally refresh) ──────────────────
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=3,
|
||||
phase='Phase 1/2: Loading snapshots...' if not refresh_first
|
||||
else 'Phase 1/2: Refreshing snapshots...',
|
||||
log_line='Phase 1: Snapshot' + (' (with refresh)' if refresh_first else ''),
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
profile_id = deps.get_current_profile_id()
|
||||
playload_payloads = _build_payloads_for_kinds(
|
||||
deps, manager, kinds_config, profile_id,
|
||||
automation_id=automation_id,
|
||||
refresh_first=refresh_first,
|
||||
)
|
||||
|
||||
if not playload_payloads:
|
||||
deps.state.set_pipeline_running(False)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='finished', progress=100,
|
||||
phase='No playlists to sync',
|
||||
log_line='No personalized playlists had tracks to sync',
|
||||
log_type='warning',
|
||||
)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'_manages_own_progress': True,
|
||||
'playlists_synced': '0',
|
||||
'tracks_synced': '0',
|
||||
'duration_seconds': str(int(time.time() - pipeline_start)),
|
||||
}
|
||||
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=50,
|
||||
phase='Phase 1/2: Snapshot complete',
|
||||
log_line=f'Phase 1 done: {len(playload_payloads)} playlist(s) ready to sync',
|
||||
log_type='success',
|
||||
)
|
||||
|
||||
# ── PHASE 2: SYNC + WISHLIST (shared helper) ────────────────
|
||||
sync_summary = run_sync_and_wishlist(
|
||||
deps,
|
||||
automation_id,
|
||||
playload_payloads,
|
||||
sync_one_fn=lambda pl: _sync_personalized_playlist(deps, pl),
|
||||
sync_id_for_fn=lambda pl: pl['sync_id'],
|
||||
skip_wishlist=skip_wishlist,
|
||||
progress_start=51,
|
||||
progress_end=90,
|
||||
sync_phase_label='Phase 2/2: Syncing to server...',
|
||||
sync_phase_start_log='Phase 2: Sync',
|
||||
wishlist_phase_label='Phase 2/2: Processing wishlist...',
|
||||
wishlist_phase_start_log='Wishlist',
|
||||
)
|
||||
|
||||
# ── COMPLETE ────────────────────────────────────────────────
|
||||
duration = int(time.time() - pipeline_start)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='finished', progress=100,
|
||||
phase='Pipeline complete',
|
||||
log_line=f'Personalized pipeline finished in {duration // 60}m {duration % 60}s',
|
||||
log_type='success',
|
||||
)
|
||||
|
||||
deps.state.set_pipeline_running(False)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'_manages_own_progress': True,
|
||||
'playlists_synced': str(len(playload_payloads)),
|
||||
'tracks_synced': str(sync_summary['synced']),
|
||||
'sync_skipped': str(sync_summary['skipped']),
|
||||
'wishlist_queued': str(sync_summary['wishlist_queued']),
|
||||
'duration_seconds': str(duration),
|
||||
}
|
||||
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise into engine
|
||||
deps.state.set_pipeline_running(False)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='error', progress=100,
|
||||
phase='Pipeline error',
|
||||
log_line=f'Personalized pipeline failed: {e}',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
|
||||
|
||||
|
||||
def _build_payloads_for_kinds(
|
||||
deps: AutomationDeps,
|
||||
manager: Any,
|
||||
kinds_config: List[Dict[str, Any]],
|
||||
profile_id: int,
|
||||
*,
|
||||
automation_id: Optional[str],
|
||||
refresh_first: bool,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Resolve each requested kind+variant into a sync-payload dict.
|
||||
|
||||
Each payload has: ``{'name', 'kind', 'variant', 'tracks_json',
|
||||
'image_url', 'sync_id'}``. Playlists with no tracks (e.g. a
|
||||
seasonal mix that hasn't been populated yet) are omitted from
|
||||
the result so the sync loop doesn't waste time on empty pushes.
|
||||
"""
|
||||
payloads: List[Dict[str, Any]] = []
|
||||
for entry in kinds_config:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
kind = entry.get('kind')
|
||||
variant = entry.get('variant') or ''
|
||||
if not kind:
|
||||
continue
|
||||
|
||||
try:
|
||||
# Refresh when ANY of:
|
||||
# - explicit user flag (cron use case: regenerate each run)
|
||||
# - snapshot marked stale by upstream data refresher
|
||||
# - playlist was never generated yet (auto-created by
|
||||
# ensure_playlist; track_count=0, last_generated_at=NULL).
|
||||
# Without this branch, a first-run pipeline reads the
|
||||
# empty snapshot and silently skips — user picks a kind,
|
||||
# hits run, gets "No tracks to sync" with no clue why.
|
||||
if refresh_first:
|
||||
record = manager.refresh_playlist(kind, variant, profile_id)
|
||||
else:
|
||||
existing = manager.ensure_playlist(kind, variant, profile_id)
|
||||
needs_first_gen = existing.last_generated_at is None
|
||||
if existing.is_stale or needs_first_gen:
|
||||
record = manager.refresh_playlist(kind, variant, profile_id)
|
||||
else:
|
||||
record = existing
|
||||
except Exception as exc: # noqa: BLE001 — log + continue with next kind
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Skipping {kind}{("/" + variant) if variant else ""}: {exc}',
|
||||
log_type='warning',
|
||||
)
|
||||
continue
|
||||
|
||||
tracks = manager.get_playlist_tracks(record.id)
|
||||
if not tracks:
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'No tracks in {record.name} — skipping sync',
|
||||
log_type='skip',
|
||||
)
|
||||
continue
|
||||
|
||||
tracks_json = [_track_to_sync_shape(t) for t in tracks]
|
||||
payloads.append({
|
||||
'name': record.name,
|
||||
'kind': record.kind,
|
||||
'variant': record.variant,
|
||||
'tracks_json': tracks_json,
|
||||
'image_url': '', # personalized playlists don't have a cover image yet
|
||||
'sync_id': f'{_SYNC_ID_PREFIX}_{record.kind}_{record.variant or "_"}',
|
||||
})
|
||||
return payloads
|
||||
|
||||
|
||||
def _track_to_sync_shape(track: Any) -> Dict[str, Any]:
|
||||
"""Convert a personalized.types.Track into the dict shape
|
||||
`_run_sync_task` expects. Mirrors what the mirrored pipeline
|
||||
builds from extra_data.matched_data, preserving enriched metadata
|
||||
from personalized snapshots when available."""
|
||||
primary_id = track.spotify_track_id or track.itunes_track_id or track.deezer_track_id or ''
|
||||
rich_data = _coerce_track_data_json(getattr(track, 'track_data_json', None))
|
||||
|
||||
if not rich_data:
|
||||
album = {'name': track.album_name or ''}
|
||||
cover_url = getattr(track, 'album_cover_url', None)
|
||||
if cover_url:
|
||||
album['images'] = [{'url': cover_url}]
|
||||
return {
|
||||
'name': track.track_name,
|
||||
'artists': [{'name': track.artist_name}],
|
||||
'album': album,
|
||||
'duration_ms': int(track.duration_ms or 0),
|
||||
'id': primary_id,
|
||||
}
|
||||
|
||||
payload = dict(rich_data)
|
||||
cover_url = (
|
||||
getattr(track, 'album_cover_url', None)
|
||||
or payload.get('album_cover_url')
|
||||
or payload.get('image_url')
|
||||
)
|
||||
payload['id'] = payload.get('id') or primary_id
|
||||
payload['name'] = payload.get('name') or track.track_name
|
||||
payload['artists'] = _normalize_artists(payload.get('artists'), track.artist_name)
|
||||
payload['album'] = _normalize_album(payload.get('album'), track, cover_url=cover_url)
|
||||
payload['duration_ms'] = int(payload.get('duration_ms') or track.duration_ms or 0)
|
||||
if 'popularity' not in payload and getattr(track, 'popularity', None) is not None:
|
||||
payload['popularity'] = int(track.popularity or 0)
|
||||
|
||||
if cover_url and not payload.get('image_url'):
|
||||
payload['image_url'] = cover_url
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def _coerce_track_data_json(value: Any) -> Dict[str, Any]:
|
||||
if isinstance(value, dict):
|
||||
return value
|
||||
if isinstance(value, str) and value.strip():
|
||||
try:
|
||||
loaded = json.loads(value)
|
||||
except (TypeError, ValueError):
|
||||
return {}
|
||||
return loaded if isinstance(loaded, dict) else {}
|
||||
return {}
|
||||
|
||||
|
||||
def _normalize_artists(artists: Any, fallback_artist: str) -> List[Dict[str, Any]]:
|
||||
if not artists:
|
||||
return [{'name': fallback_artist or 'Unknown Artist'}]
|
||||
if isinstance(artists, list):
|
||||
normalized = []
|
||||
for artist in artists:
|
||||
if isinstance(artist, dict):
|
||||
normalized.append(artist if artist.get('name') else {'name': str(artist)})
|
||||
elif isinstance(artist, str):
|
||||
normalized.append({'name': artist})
|
||||
else:
|
||||
normalized.append({'name': str(artist)})
|
||||
return normalized or [{'name': fallback_artist or 'Unknown Artist'}]
|
||||
if isinstance(artists, dict):
|
||||
return [artists if artists.get('name') else {'name': str(artists)}]
|
||||
return [{'name': str(artists)}]
|
||||
|
||||
|
||||
def _normalize_album(album: Any, track: Any, cover_url: Optional[str] = None) -> Dict[str, Any]:
|
||||
if isinstance(album, dict):
|
||||
normalized = dict(album)
|
||||
else:
|
||||
normalized = {'name': str(album) if album else (track.album_name or '')}
|
||||
|
||||
normalized['name'] = normalized.get('name') or track.album_name or ''
|
||||
images = normalized.get('images')
|
||||
if not isinstance(images, list):
|
||||
images = []
|
||||
if cover_url and not images:
|
||||
images = [{'url': cover_url}]
|
||||
if images:
|
||||
normalized['images'] = images
|
||||
return normalized
|
||||
|
||||
|
||||
def _sync_personalized_playlist(deps: AutomationDeps, payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Launch a personalized playlist sync via _run_sync_task on a
|
||||
daemon thread + return immediately with status='started'.
|
||||
|
||||
Mirrors the mirrored ``auto_sync_playlist`` return contract so the
|
||||
shared helper can poll on ``sync_states[sync_id]`` and aggregate
|
||||
results identically."""
|
||||
sync_id = payload['sync_id']
|
||||
name = payload['name']
|
||||
tracks_json = payload['tracks_json']
|
||||
profile_id = deps.get_current_profile_id()
|
||||
|
||||
threading.Thread(
|
||||
target=deps.run_sync_task,
|
||||
args=(sync_id, name, tracks_json, None, profile_id, payload.get('image_url', '')),
|
||||
daemon=True,
|
||||
name=f'auto-personalized-{sync_id}',
|
||||
).start()
|
||||
return {
|
||||
'status': 'started',
|
||||
'playlist_name': name,
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
"""Automation adapter for the mirrored playlist pipeline.
|
||||
|
||||
The actual all-in-one playlist lifecycle lives in
|
||||
``core.playlists.pipeline`` so it can be reused by non-automation UI actions.
|
||||
This module only wires automation-specific dependencies and handlers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
|
||||
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
|
||||
from core.automation.handlers.sync_playlist import auto_sync_playlist
|
||||
from core.playlists.pipeline import run_mirrored_playlist_pipeline
|
||||
|
||||
|
||||
def auto_playlist_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run REFRESH -> DISCOVER -> SYNC -> WISHLIST for mirrored playlists."""
|
||||
return run_mirrored_playlist_pipeline(
|
||||
config,
|
||||
deps,
|
||||
refresh_fn=auto_refresh_mirrored,
|
||||
sync_one_fn=auto_sync_playlist,
|
||||
sync_and_wishlist_fn=run_sync_and_wishlist,
|
||||
)
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
"""Automation handler: ``process_wishlist`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_process_wishlist`` closure). Wishlist processing is async —
|
||||
the helper submits a batch to an executor and returns immediately;
|
||||
per-track stats arrive later via batch-completion callbacks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_process_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Kick off the wishlist processor for an automation trigger.
|
||||
|
||||
Returns immediately after submission; the wishlist worker emits
|
||||
per-batch progress via its own callbacks. We only report
|
||||
``status: completed`` to mark the trigger fired successfully.
|
||||
"""
|
||||
try:
|
||||
deps.process_wishlist_automatically(automation_id=config.get('_automation_id'))
|
||||
return {'status': 'completed'}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
|
||||
return {'status': 'error', 'error': str(e)}
|
||||
|
|
@ -1,89 +0,0 @@
|
|||
"""Progress + history callbacks the automation engine invokes around
|
||||
each handler run.
|
||||
|
||||
Lifted from the closures at the bottom of
|
||||
``web_server._register_automation_handlers``:
|
||||
- ``_progress_init`` → :func:`progress_init`
|
||||
- ``_progress_finish`` → :func:`progress_finish`
|
||||
- ``_record_automation_history`` → :func:`record_history`
|
||||
- ``_on_library_scan_completed`` → :func:`on_library_scan_completed`
|
||||
|
||||
The engine accepts four callables via
|
||||
``register_progress_callbacks(init, finish, update, history)``;
|
||||
``registration.register_all`` wires these here. The
|
||||
``library_scan_completed`` callback is registered separately on the
|
||||
``web_scan_manager`` (when one is available) -- see
|
||||
``register_library_scan_completed_emitter``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def progress_init(aid: Any, name: str, action_type: str, deps: AutomationDeps) -> None:
|
||||
"""Initialize per-automation progress state when the engine starts
|
||||
a handler. Thin wrapper so the engine receives a closure that
|
||||
delegates into the live progress tracker."""
|
||||
deps.init_automation_progress(aid, name, action_type)
|
||||
|
||||
|
||||
def progress_finish(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
|
||||
"""Emit the final progress update when a handler returns.
|
||||
|
||||
Skipped for handlers that manage their own progress lifecycle
|
||||
(they call ``update_progress(status='finished')`` themselves and
|
||||
set ``_manages_own_progress: True`` in the returned dict).
|
||||
Otherwise translates the handler's status into a finished/error
|
||||
progress emit with a status-appropriate phase + log line.
|
||||
"""
|
||||
if result.get('_manages_own_progress'):
|
||||
return
|
||||
result_status = result.get('status', '')
|
||||
status = 'error' if result_status == 'error' else 'finished'
|
||||
msg = result.get('error', result.get('reason', result_status or 'done'))
|
||||
deps.update_progress(
|
||||
aid,
|
||||
status=status,
|
||||
progress=100,
|
||||
phase='Error' if status == 'error' else 'Complete',
|
||||
log_line=msg,
|
||||
log_type='error' if status == 'error' else 'success',
|
||||
)
|
||||
|
||||
|
||||
def record_history(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
|
||||
"""Capture progress state into run history before the engine's
|
||||
cleanup pass clears it. Thin wrapper so the engine sees a stable
|
||||
callable."""
|
||||
deps.record_progress_history(aid, result, deps.get_database())
|
||||
|
||||
|
||||
def on_library_scan_completed(deps: AutomationDeps) -> None:
|
||||
"""Emit the ``library_scan_completed`` automation event with the
|
||||
active media-server type. Replaces the hard-coded
|
||||
``scan_completion_callback → trigger_automatic_database_update``
|
||||
chain so any automation can listen for scan completion as a
|
||||
trigger."""
|
||||
if not deps.engine:
|
||||
return
|
||||
server_type = (
|
||||
getattr(deps.web_scan_manager, '_current_server_type', None)
|
||||
or 'unknown'
|
||||
)
|
||||
deps.engine.emit('library_scan_completed', {
|
||||
'server_type': server_type,
|
||||
})
|
||||
|
||||
|
||||
def register_library_scan_completed_emitter(deps: AutomationDeps) -> None:
|
||||
"""Wire :func:`on_library_scan_completed` to the
|
||||
``web_scan_manager``'s scan-completion callback list. No-op when
|
||||
no scan manager is configured (e.g. headless / test contexts)."""
|
||||
if not deps.web_scan_manager:
|
||||
return
|
||||
deps.web_scan_manager.add_scan_completion_callback(
|
||||
lambda: on_library_scan_completed(deps),
|
||||
)
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""Automation handler: ``start_quality_scan`` action.
|
||||
|
||||
The quality scanner was redesigned from an auto-acting tool into the
|
||||
``quality_upgrade`` library-maintenance repair job (findings-based, reviewed
|
||||
before anything is wishlisted). This action now simply triggers a "Run Now" of
|
||||
that job; its progress and findings surface in Library Maintenance. The action
|
||||
name is kept so existing automation rules keep working.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_start_quality_scan(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
automation_id = config.get('_automation_id')
|
||||
|
||||
triggered = deps.run_repair_job_now('quality_upgrade')
|
||||
if not triggered:
|
||||
deps.update_progress(
|
||||
automation_id, status='error', phase='Unavailable',
|
||||
log_line='Quality Upgrade job could not be triggered (library worker unavailable)',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': 'library worker unavailable',
|
||||
'_manages_own_progress': True}
|
||||
|
||||
deps.update_progress(
|
||||
automation_id, status='finished', progress=100, phase='Triggered',
|
||||
log_line='Quality Upgrade scan queued — findings appear in Library Maintenance',
|
||||
log_type='success',
|
||||
)
|
||||
return {'status': 'completed', 'triggered': True, '_manages_own_progress': True}
|
||||
|
|
@ -1,341 +0,0 @@
|
|||
"""Automation handler: ``refresh_mirrored`` action.
|
||||
|
||||
Re-pulls track lists from each mirrored playlist's source via the
|
||||
unified ``PlaylistSourceRegistry`` (Phase 1 of the Discover-to-Sync
|
||||
unification). The pre-extraction handler had ~190 lines of per-source
|
||||
if/elif branches; this version delegates to the adapter for each
|
||||
source, leaving the handler responsible only for:
|
||||
|
||||
- filtering sources that can't be refreshed (``file``, ``beatport``),
|
||||
- extracting upstream URLs from the stored ``description`` for URL-
|
||||
backed sources (``spotify_public``, ``youtube``),
|
||||
- the Spotify-public → authenticated-Spotify fallback (uses the
|
||||
``spotify`` adapter when the user is signed in so the mirror keeps
|
||||
album art),
|
||||
- the Tidal-not-authenticated skip log type (vs error),
|
||||
- preserving existing per-track ``extra_data`` on tracks that survive
|
||||
the refresh, and
|
||||
- emitting the ``playlist_changed`` automation event when the track
|
||||
set actually shifts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
from core.playlists.source_refs import require_refresh_url
|
||||
from core.playlists.sources import PlaylistDetail, to_mirror_track_dict
|
||||
from core.playlists.sources.base import (
|
||||
SOURCE_SPOTIFY,
|
||||
SOURCE_SPOTIFY_PUBLIC,
|
||||
SOURCE_TIDAL,
|
||||
SOURCE_YOUTUBE,
|
||||
)
|
||||
|
||||
|
||||
# Sources that store the upstream URL in ``description`` (because their
|
||||
# ``source_playlist_id`` is a deterministic hash, not the native ID).
|
||||
# The refresh path has to recover the URL before calling the adapter.
|
||||
_URL_BACKED_SOURCES = {SOURCE_SPOTIFY_PUBLIC, SOURCE_YOUTUBE}
|
||||
|
||||
|
||||
def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Refresh mirrored playlist(s) from source.
|
||||
|
||||
Returns ``{'status': 'completed', 'refreshed': '<int>',
|
||||
'errors': '<int>'}`` on success (counts stringified to match the
|
||||
automation engine's stat-rendering convention).
|
||||
"""
|
||||
db = deps.get_database()
|
||||
playlist_id = config.get('playlist_id')
|
||||
refresh_all = config.get('all', False)
|
||||
auto_id = config.get('_automation_id')
|
||||
# Pipeline runs (``run_mirrored_playlist_pipeline``) set this flag
|
||||
# because Phase 2 of the pipeline already runs the playlist
|
||||
# discovery worker — the same matching engine ``_maybe_discover``
|
||||
# would call here. Running both means LB tracks discover twice
|
||||
# AND the refresh-side discovery blocks 5+ minutes with no
|
||||
# progress emission, leaving the UI stuck on "Refreshing:" until
|
||||
# the loop returns. Standalone callers (Sync page, registration
|
||||
# action) leave it False so LB tracks still get matched_data on
|
||||
# refresh.
|
||||
skip_discovery = bool(config.get('skip_discovery', False))
|
||||
|
||||
if refresh_all:
|
||||
playlists = db.get_mirrored_playlists()
|
||||
elif playlist_id:
|
||||
p = db.get_mirrored_playlist(int(playlist_id))
|
||||
playlists = [p] if p else []
|
||||
else:
|
||||
return {'status': 'error', 'reason': 'No playlist specified'}
|
||||
|
||||
# Filter out sources that can't be refreshed (no external API).
|
||||
playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
|
||||
|
||||
refreshed = 0
|
||||
errors: List[str] = []
|
||||
for idx, pl in enumerate(playlists):
|
||||
try:
|
||||
source = pl.get('source', '')
|
||||
source_id = pl.get('source_playlist_id', '')
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
progress=(idx / max(1, len(playlists))) * 100,
|
||||
phase=f'Refreshing: "{pl.get("name", "")}"',
|
||||
current_item=pl.get('name', ''),
|
||||
)
|
||||
|
||||
detail = _fetch_detail(source, source_id, pl, deps, auto_id)
|
||||
if detail is None:
|
||||
# _fetch_detail already logged the specific failure;
|
||||
# mark the playlist as a generic refresh error so the
|
||||
# automation result tally matches the legacy handler.
|
||||
errors.append(f"{pl.get('name', '?')}: no tracks returned from source")
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'Refresh failed: "{pl.get("name", "")}" - no tracks returned from source',
|
||||
log_type='error',
|
||||
)
|
||||
continue
|
||||
|
||||
# Sources that return MB-metadata-only tracks (LB, Last.fm)
|
||||
# mark them ``needs_discovery=True``. Hand them to the
|
||||
# adapter's matcher so the resulting mirror rows carry
|
||||
# provider IDs + matched_data, ready for the sync pipeline.
|
||||
#
|
||||
# Pipeline runs skip this because Phase 2's discovery
|
||||
# worker handles it with proper progress emission — see
|
||||
# ``skip_discovery`` resolution at the top of this fn.
|
||||
detail_tracks = (
|
||||
detail.tracks if skip_discovery
|
||||
else _maybe_discover(detail.tracks, source, deps)
|
||||
)
|
||||
|
||||
tracks = [to_mirror_track_dict(t) for t in detail_tracks]
|
||||
refreshed += _commit_refresh(pl, source, source_id, tracks, db, deps, auto_id)
|
||||
except _SkipPlaylist:
|
||||
# Source-specific soft-skip (e.g. Tidal not authenticated).
|
||||
# Logging was already emitted; do not count as error.
|
||||
continue
|
||||
except Exception as e:
|
||||
errors.append(f"{pl.get('name', '?')}: {str(e)}")
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'Error: {pl.get("name", "?")} — {str(e)}',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'completed', 'refreshed': str(refreshed), 'errors': str(len(errors))}
|
||||
|
||||
|
||||
def _maybe_discover(
|
||||
tracks: List[Any],
|
||||
source: str,
|
||||
deps: AutomationDeps,
|
||||
) -> List[Any]:
|
||||
"""Run the adapter's ``discover_tracks`` when any track needs it.
|
||||
|
||||
Most sources are no-ops here (their tracks have provider IDs
|
||||
already). LB / Last.fm are the ones that actually do work."""
|
||||
if not tracks:
|
||||
return tracks
|
||||
if not any(getattr(t, "needs_discovery", False) for t in tracks):
|
||||
return tracks
|
||||
registry = deps.playlist_source_registry
|
||||
if registry is None:
|
||||
return tracks
|
||||
adapter = registry.get_source(source)
|
||||
if adapter is None:
|
||||
return tracks
|
||||
try:
|
||||
return adapter.discover_tracks(tracks)
|
||||
except Exception as exc:
|
||||
deps.logger.warning(f"{source} discover_tracks failed: {exc}")
|
||||
return tracks
|
||||
|
||||
|
||||
class _SkipPlaylist(Exception):
|
||||
"""Internal sentinel: source-specific soft-skip (e.g. not authed).
|
||||
|
||||
The per-playlist loop catches it specifically so the skip isn't
|
||||
counted in the error tally — matches the pre-extraction behavior
|
||||
where ``continue`` was used inline."""
|
||||
|
||||
|
||||
def _fetch_detail(
|
||||
source: str,
|
||||
source_id: str,
|
||||
pl: Dict[str, Any],
|
||||
deps: AutomationDeps,
|
||||
auto_id: Optional[str],
|
||||
) -> Optional[PlaylistDetail]:
|
||||
"""Resolve the playlist's tracks through the registry.
|
||||
|
||||
Handler-level branches (URL extraction, Spotify-public→authed
|
||||
fallback, Tidal not-authed skip) live here; everything else
|
||||
delegates to the adapter."""
|
||||
registry = deps.playlist_source_registry
|
||||
if registry is None:
|
||||
return None
|
||||
|
||||
# URL-backed sources: pull the upstream URL out of `description`.
|
||||
playlist_input = source_id
|
||||
if source in _URL_BACKED_SOURCES:
|
||||
# ``require_refresh_url`` raises ValueError on missing URL.
|
||||
# The outer try/except in the loop catches it and reports as
|
||||
# an error — matching the pre-extraction behavior.
|
||||
playlist_input = require_refresh_url(
|
||||
source, pl.get('description', ''), pl.get('name', '')
|
||||
)
|
||||
|
||||
# Spotify-public refresh: prefer the authenticated Spotify API
|
||||
# when the user is signed in. Better album art, matches the
|
||||
# pre-extraction handler. Falls through to the public scraper on
|
||||
# auth failure or non-playlist URL types (e.g. album URLs).
|
||||
if source == SOURCE_SPOTIFY_PUBLIC:
|
||||
detail = _try_spotify_authed_for_public(playlist_input, deps)
|
||||
if detail is not None:
|
||||
return detail
|
||||
|
||||
# Tidal not-authed: soft-skip with a 'skip' log line, not an error.
|
||||
if source == SOURCE_TIDAL:
|
||||
tidal_source = registry.get_source(SOURCE_TIDAL)
|
||||
if tidal_source is None or not tidal_source.is_authenticated():
|
||||
deps.logger.warning(
|
||||
f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'"
|
||||
)
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
|
||||
log_type='skip',
|
||||
)
|
||||
raise _SkipPlaylist
|
||||
|
||||
adapter = registry.get_source(source)
|
||||
if adapter is None:
|
||||
return None
|
||||
try:
|
||||
return adapter.refresh_playlist(playlist_input)
|
||||
except Exception as exc:
|
||||
deps.logger.warning(
|
||||
f"{source} playlist refresh failed for {playlist_input}: {exc}"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _try_spotify_authed_for_public(
|
||||
spotify_url: str, deps: AutomationDeps
|
||||
) -> Optional[PlaylistDetail]:
|
||||
"""Best-effort: use the authenticated Spotify adapter on a public URL.
|
||||
|
||||
Returns ``None`` to signal "fall through to the public-scraper
|
||||
adapter" — never raises. Only applies to ``playlist``-type URLs;
|
||||
album URLs fall through unconditionally."""
|
||||
if not spotify_url:
|
||||
return None
|
||||
spotify_client = deps.spotify_client
|
||||
if spotify_client is None or not spotify_client.is_spotify_authenticated():
|
||||
return None
|
||||
try:
|
||||
from core.spotify_public_scraper import parse_spotify_url
|
||||
parsed = parse_spotify_url(spotify_url)
|
||||
except Exception:
|
||||
return None
|
||||
if not parsed or parsed.get('type') != 'playlist':
|
||||
return None
|
||||
adapter = deps.playlist_source_registry.get_source(SOURCE_SPOTIFY)
|
||||
if adapter is None:
|
||||
return None
|
||||
try:
|
||||
return adapter.refresh_playlist(parsed['id'])
|
||||
except Exception as exc:
|
||||
deps.logger.debug(f"Spotify authed fallback for public mirror failed: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
def _commit_refresh(
|
||||
pl: Dict[str, Any],
|
||||
source: str,
|
||||
source_id: str,
|
||||
tracks: List[Dict[str, Any]],
|
||||
db: Any,
|
||||
deps: AutomationDeps,
|
||||
auto_id: Optional[str],
|
||||
) -> int:
|
||||
"""Persist the refreshed track list + emit playlist_changed when delta.
|
||||
|
||||
Returns 1 when a refresh successfully landed, 0 otherwise. The
|
||||
caller is responsible for incrementing the running tally."""
|
||||
old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
|
||||
old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
|
||||
new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
|
||||
|
||||
# Preserve existing extra_data (matched_data + discovery state)
|
||||
# for tracks that still exist in the refreshed snapshot, unless
|
||||
# the adapter already provided fresh extra_data for that track.
|
||||
old_extra_map = (
|
||||
db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
|
||||
)
|
||||
for t in tracks:
|
||||
sid = t.get('source_track_id', '')
|
||||
if sid and sid in old_extra_map and 'extra_data' not in t:
|
||||
t['extra_data'] = old_extra_map[sid]
|
||||
|
||||
db.mirror_playlist(
|
||||
source=source,
|
||||
source_playlist_id=source_id,
|
||||
name=pl['name'],
|
||||
tracks=tracks,
|
||||
profile_id=pl.get('profile_id', 1),
|
||||
description=pl.get('description'),
|
||||
owner=pl.get('owner'),
|
||||
image_url=pl.get('image_url'),
|
||||
)
|
||||
|
||||
# Membership just changed — if this playlist is organize-by-playlist, rebuild
|
||||
# its folder (with prune) so a track that LEFT the playlist has its symlink
|
||||
# cleaned up now. Gated to organized playlists, non-fatal — never disturbs
|
||||
# the refresh. (Additions are handled by the post-download reconcile.)
|
||||
try:
|
||||
from core.playlists.materialize_service import rebuild_mirrored_playlist_if_organized
|
||||
rebuild_mirrored_playlist_if_organized(
|
||||
db, deps.config_manager, pl.get('id'), profile_id=pl.get('profile_id', 1)
|
||||
)
|
||||
except Exception as _mat_err:
|
||||
deps.logger.debug(f"[Playlist Folder] mirror-refresh cleanup skipped: {_mat_err}")
|
||||
|
||||
if old_ids != new_ids:
|
||||
added = len(new_ids - old_ids)
|
||||
removed = len(old_ids - new_ids)
|
||||
deps.logger.info(
|
||||
f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
|
||||
f"{added} added, {removed} removed (old={len(old_ids)}, new={len(new_ids)})"
|
||||
)
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'"{pl.get("name", "")}" — {added} added, {removed} removed',
|
||||
log_type='success',
|
||||
)
|
||||
try:
|
||||
if deps.engine:
|
||||
deps.engine.emit('playlist_changed', {
|
||||
'playlist_name': pl.get('name', ''),
|
||||
'playlist_id': str(pl.get('id', '')),
|
||||
'old_count': str(len(old_ids)),
|
||||
'new_count': str(len(new_ids)),
|
||||
'added': str(added),
|
||||
'removed': str(removed),
|
||||
})
|
||||
except Exception as e:
|
||||
deps.logger.debug("playlist_synced automation emit failed: %s", e)
|
||||
else:
|
||||
deps.logger.warning(
|
||||
f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})"
|
||||
)
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'No changes: "{pl.get("name", "")}"',
|
||||
log_type='skip',
|
||||
)
|
||||
|
||||
return 1
|
||||
|
|
@ -1,184 +0,0 @@
|
|||
"""One-stop registration of every extracted automation handler.
|
||||
|
||||
``web_server`` builds the deps once at startup and calls
|
||||
:func:`register_all` here. Each new handler module gets one line in
|
||||
this file when it lands.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
from core.automation.handlers.process_wishlist import auto_process_wishlist
|
||||
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
|
||||
from core.automation.handlers.scan_library import auto_scan_library
|
||||
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
|
||||
from core.automation.handlers.sync_playlist import auto_sync_playlist
|
||||
from core.automation.handlers.discover_playlist import auto_discover_playlist
|
||||
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
|
||||
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
|
||||
from core.automation.handlers.database_update import (
|
||||
auto_start_database_update, auto_deep_scan_library,
|
||||
)
|
||||
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
|
||||
from core.automation.handlers.quality_scanner import auto_start_quality_scan
|
||||
from core.automation.handlers.maintenance import (
|
||||
auto_clear_quarantine,
|
||||
auto_cleanup_wishlist,
|
||||
auto_update_discovery_pool,
|
||||
auto_backup_database,
|
||||
auto_refresh_beatport_cache,
|
||||
)
|
||||
from core.automation.handlers.download_cleanup import (
|
||||
auto_clean_search_history,
|
||||
auto_clean_completed_downloads,
|
||||
auto_full_cleanup,
|
||||
)
|
||||
from core.automation.handlers.run_script import auto_run_script
|
||||
from core.automation.handlers.search_and_download import auto_search_and_download
|
||||
from core.automation.handlers.progress_callbacks import (
|
||||
progress_init,
|
||||
progress_finish,
|
||||
record_history,
|
||||
register_library_scan_completed_emitter,
|
||||
)
|
||||
|
||||
|
||||
def register_all(deps: AutomationDeps) -> None:
|
||||
"""Wire every extracted handler to the engine.
|
||||
|
||||
Each ``register_action_handler`` call binds the action name (the
|
||||
string the trigger uses to look up its action) to a thin lambda
|
||||
that injects ``deps`` and forwards the engine-supplied config.
|
||||
Guards stay alongside their handler so duplicate-run prevention
|
||||
behaves identically to the pre-extraction code.
|
||||
"""
|
||||
engine = deps.engine
|
||||
|
||||
# Self-guards prevent duplicate runs of the SAME operation, but
|
||||
# different operations can run concurrently — wishlist downloads
|
||||
# use bandwidth, watchlist scans use API calls, library scans use
|
||||
# media-server CPU. Different resources, no contention.
|
||||
engine.register_action_handler(
|
||||
'process_wishlist',
|
||||
lambda config: auto_process_wishlist(config, deps),
|
||||
guard_fn=deps.is_wishlist_actually_processing,
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'scan_watchlist',
|
||||
lambda config: auto_scan_watchlist(config, deps),
|
||||
guard_fn=deps.is_watchlist_actually_scanning,
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'scan_library',
|
||||
lambda config: auto_scan_library(config, deps),
|
||||
deps.state.is_scan_library_active,
|
||||
)
|
||||
|
||||
# Playlist lifecycle handlers. The pipeline composes refresh +
|
||||
# sync + discover (it imports them directly), so all four ship
|
||||
# together. The pipeline guard prevents an in-flight pipeline
|
||||
# from being re-triggered mid-run.
|
||||
engine.register_action_handler(
|
||||
'refresh_mirrored',
|
||||
lambda config: auto_refresh_mirrored(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'sync_playlist',
|
||||
lambda config: auto_sync_playlist(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'discover_playlist',
|
||||
lambda config: auto_discover_playlist(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'playlist_pipeline',
|
||||
lambda config: auto_playlist_pipeline(config, deps),
|
||||
deps.state.is_pipeline_running,
|
||||
)
|
||||
# Personalized pipeline shares the pipeline_running flag with the
|
||||
# mirrored pipeline so the two can't overlap (single sync queue,
|
||||
# single wishlist worker).
|
||||
engine.register_action_handler(
|
||||
'personalized_pipeline',
|
||||
lambda config: auto_personalized_pipeline(config, deps),
|
||||
deps.state.is_pipeline_running,
|
||||
)
|
||||
|
||||
# Database update + deep scan share the db_update_state guard —
|
||||
# only one operation can mutate that state at a time.
|
||||
engine.register_action_handler(
|
||||
'start_database_update',
|
||||
lambda config: auto_start_database_update(config, deps),
|
||||
lambda: deps.get_db_update_state().get('status') == 'running',
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'deep_scan_library',
|
||||
lambda config: auto_deep_scan_library(config, deps),
|
||||
lambda: deps.get_db_update_state().get('status') == 'running',
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'run_duplicate_cleaner',
|
||||
lambda config: auto_run_duplicate_cleaner(config, deps),
|
||||
lambda: deps.get_duplicate_cleaner_state().get('status') == 'running',
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'clear_quarantine',
|
||||
lambda config: auto_clear_quarantine(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'cleanup_wishlist',
|
||||
lambda config: auto_cleanup_wishlist(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'update_discovery_pool',
|
||||
lambda config: auto_update_discovery_pool(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'start_quality_scan',
|
||||
lambda config: auto_start_quality_scan(config, deps),
|
||||
lambda: False, # repair worker dedupes Run-Now requests itself
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'backup_database',
|
||||
lambda config: auto_backup_database(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'refresh_beatport_cache',
|
||||
lambda config: auto_refresh_beatport_cache(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'clean_search_history',
|
||||
lambda config: auto_clean_search_history(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'clean_completed_downloads',
|
||||
lambda config: auto_clean_completed_downloads(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'full_cleanup',
|
||||
lambda config: auto_full_cleanup(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'run_script',
|
||||
lambda config: auto_run_script(config, deps),
|
||||
)
|
||||
engine.register_action_handler(
|
||||
'search_and_download',
|
||||
lambda config: auto_search_and_download(config, deps),
|
||||
)
|
||||
|
||||
# Progress + history callbacks: the engine invokes these around
|
||||
# each handler run. Lift the closures from
|
||||
# `web_server._register_automation_handlers` into thin lambdas
|
||||
# that delegate into the extracted top-level functions.
|
||||
engine.register_progress_callbacks(
|
||||
lambda aid, name, action_type: progress_init(aid, name, action_type, deps),
|
||||
lambda aid, result: progress_finish(aid, result, deps),
|
||||
deps.update_progress,
|
||||
lambda aid, result: record_history(aid, result, deps),
|
||||
)
|
||||
|
||||
# `library_scan_completed` event: when the media-server scan
|
||||
# manager finishes a scan, emit the event so any automation can
|
||||
# trigger off it. No-op when no scan manager is configured.
|
||||
register_library_scan_completed_emitter(deps)
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
"""Automation handler: ``run_script`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_run_script`` closure). Runs a user-provided shell or Python
|
||||
script from the configured scripts directory with bounded timeout +
|
||||
captured stdout/stderr. Path-traversal guard ensures users can't
|
||||
escape the scripts directory.
|
||||
|
||||
Environment variables exposed to the script:
|
||||
- ``SOULSYNC_EVENT``: triggering event type (when fired by an event)
|
||||
- ``SOULSYNC_AUTOMATION``: automation name
|
||||
- ``SOULSYNC_SCRIPTS_DIR``: absolute path to the scripts dir
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess as _sp
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
_MAX_TIMEOUT_SECONDS = 300 # Hard cap on user-supplied timeout config.
|
||||
|
||||
|
||||
def auto_run_script(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
script_name = config.get('script_name', '')
|
||||
timeout = min(int(config.get('timeout', 60)), _MAX_TIMEOUT_SECONDS)
|
||||
automation_id = config.get('_automation_id')
|
||||
|
||||
if not script_name:
|
||||
return {'status': 'error', 'error': 'No script selected'}
|
||||
|
||||
scripts_dir = deps.docker_resolve_path(deps.config_manager.get('scripts.path', './scripts'))
|
||||
if not scripts_dir or not os.path.isdir(scripts_dir):
|
||||
os.makedirs(scripts_dir, exist_ok=True)
|
||||
return {
|
||||
'status': 'error',
|
||||
'error': 'Scripts directory is empty. Add scripts to the scripts/ folder.',
|
||||
}
|
||||
|
||||
script_path = os.path.join(scripts_dir, script_name)
|
||||
script_path = os.path.realpath(script_path)
|
||||
|
||||
# Security: block path traversal — script must resolve under
|
||||
# the scripts dir, no symlinks/.. tricks allowed out.
|
||||
if not script_path.startswith(os.path.realpath(scripts_dir)):
|
||||
return {'status': 'error', 'error': 'Script path traversal blocked'}
|
||||
|
||||
if not os.path.isfile(script_path):
|
||||
return {'status': 'error', 'error': f'Script not found: {script_name}'}
|
||||
|
||||
deps.update_progress(automation_id, phase=f'Running {script_name}...', progress=10)
|
||||
|
||||
# Build environment with SoulSync context.
|
||||
env = os.environ.copy()
|
||||
event_data = config.get('_event_data') or {}
|
||||
env['SOULSYNC_EVENT'] = str(event_data.get('type', ''))
|
||||
env['SOULSYNC_AUTOMATION'] = config.get('_automation_name', '')
|
||||
env['SOULSYNC_SCRIPTS_DIR'] = scripts_dir
|
||||
|
||||
try:
|
||||
# Determine how to run the script.
|
||||
if script_path.endswith('.py'):
|
||||
cmd = ['python', script_path]
|
||||
elif script_path.endswith('.sh'):
|
||||
cmd = ['bash', script_path]
|
||||
else:
|
||||
cmd = [script_path]
|
||||
|
||||
result = _sp.run(
|
||||
cmd,
|
||||
capture_output=True, text=True, timeout=timeout,
|
||||
cwd=scripts_dir, env=env,
|
||||
)
|
||||
|
||||
deps.update_progress(automation_id, phase='Script completed', progress=100)
|
||||
|
||||
stdout = result.stdout[:2000] if result.stdout else ''
|
||||
stderr = result.stderr[:1000] if result.stderr else ''
|
||||
|
||||
if result.returncode == 0:
|
||||
deps.logger.info(f"Script '{script_name}' completed (exit 0)")
|
||||
else:
|
||||
deps.logger.warning(f"Script '{script_name}' exited with code {result.returncode}")
|
||||
|
||||
return {
|
||||
'status': 'completed' if result.returncode == 0 else 'error',
|
||||
'exit_code': str(result.returncode),
|
||||
'stdout': stdout,
|
||||
'stderr': stderr,
|
||||
'script': script_name,
|
||||
}
|
||||
except _sp.TimeoutExpired:
|
||||
deps.update_progress(automation_id, phase='Script timed out', progress=100)
|
||||
return {
|
||||
'status': 'error',
|
||||
'error': f'Script timed out after {timeout}s',
|
||||
'script': script_name,
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise
|
||||
return {'status': 'error', 'error': str(e), 'script': script_name}
|
||||
|
|
@ -1,158 +0,0 @@
|
|||
"""Automation handler: ``scan_library`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_scan_library`` closure). The handler triggers a media-server
|
||||
scan via ``web_scan_manager``, then polls the manager's status until
|
||||
the scan completes (or a 30-minute timeout fires). Progress phases
|
||||
are emitted via :func:`AutomationDeps.update_progress` so the
|
||||
trigger card stays current throughout the run.
|
||||
|
||||
The handler manages its own progress reporting (it sets
|
||||
``_manages_own_progress: True`` in the result) so the engine doesn't
|
||||
overwrite the live phase string with a generic 'completed' label.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
# Outer poll cap — covers extreme worst case (long Plex scans on
|
||||
# huge libraries). Past this point we surface a clear timeout error
|
||||
# so users notice rather than letting the trigger hang forever.
|
||||
_SCAN_TIMEOUT_SECONDS = 1800
|
||||
|
||||
# Per-phase poll intervals.
|
||||
_POLL_SCHEDULED_SECONDS = 2
|
||||
_POLL_SCANNING_SECONDS = 5
|
||||
_POLL_UNKNOWN_SECONDS = 2
|
||||
|
||||
# Progress percentage waypoints.
|
||||
_PROGRESS_SCHEDULED_MAX = 14
|
||||
_PROGRESS_SCAN_START = 15
|
||||
_PROGRESS_SCAN_MAX = 95
|
||||
|
||||
|
||||
def auto_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run a media-server library scan and stream progress to the
|
||||
trigger card.
|
||||
|
||||
Returns one of:
|
||||
- ``{'status': 'completed', '_manages_own_progress': True, ...}``
|
||||
- ``{'status': 'skipped', 'reason': 'Scan already being tracked'}``
|
||||
- ``{'status': 'error', 'reason': '...', '_manages_own_progress': True}``
|
||||
"""
|
||||
automation_id = config.get('_automation_id')
|
||||
|
||||
if not deps.web_scan_manager:
|
||||
return {'status': 'error', 'reason': 'Scan manager not available'}
|
||||
|
||||
# If another automation is already tracking the scan, just forward
|
||||
# the request — the original tracker keeps emitting progress.
|
||||
if deps.state.is_scan_library_active():
|
||||
deps.web_scan_manager.request_scan('Automation trigger (additional batch)')
|
||||
return {'status': 'skipped', 'reason': 'Scan already being tracked'}
|
||||
|
||||
deps.state.set_scan_library_id(automation_id)
|
||||
|
||||
try:
|
||||
result = deps.web_scan_manager.request_scan('Automation trigger')
|
||||
scan_status_val = result.get('status', 'unknown')
|
||||
|
||||
if scan_status_val == 'queued':
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line='Scan already in progress — waiting for completion',
|
||||
log_type='info',
|
||||
)
|
||||
else:
|
||||
delay = result.get('delay_seconds', 60)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Scan scheduled (debounce: {delay}s)',
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
# Unified polling loop — handles debounce → scanning → idle.
|
||||
poll_start = time.time()
|
||||
scan_started = (scan_status_val == 'queued')
|
||||
while time.time() - poll_start < _SCAN_TIMEOUT_SECONDS:
|
||||
status = deps.web_scan_manager.get_scan_status()
|
||||
st = status.get('status')
|
||||
|
||||
if st == 'idle':
|
||||
break # Scan completed (or finished before we polled)
|
||||
|
||||
if st == 'scheduled':
|
||||
elapsed = int(time.time() - poll_start)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
phase=f'Waiting for scan to start... ({elapsed}s)',
|
||||
progress=min(int(elapsed / 60 * 10), _PROGRESS_SCHEDULED_MAX),
|
||||
)
|
||||
time.sleep(_POLL_SCHEDULED_SECONDS)
|
||||
continue
|
||||
|
||||
if st == 'scanning':
|
||||
if not scan_started:
|
||||
scan_started = True
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
progress=_PROGRESS_SCAN_START,
|
||||
log_line='Scan triggered on media server',
|
||||
log_type='success',
|
||||
)
|
||||
elapsed = status.get('elapsed_seconds', 0)
|
||||
max_time = status.get('max_time_seconds', 300)
|
||||
pct = min(_PROGRESS_SCAN_START + int(elapsed / max_time * 80), _PROGRESS_SCAN_MAX)
|
||||
mins, secs = divmod(elapsed, 60)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
phase=f'Library scan in progress... ({mins}m {secs}s)',
|
||||
progress=pct,
|
||||
)
|
||||
time.sleep(_POLL_SCANNING_SECONDS)
|
||||
continue
|
||||
|
||||
time.sleep(_POLL_UNKNOWN_SECONDS)
|
||||
else:
|
||||
# 30-min timeout reached
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='error',
|
||||
phase='Timed out',
|
||||
log_line='Library scan timed out after 30 minutes',
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
|
||||
|
||||
elapsed = round(time.time() - poll_start, 1)
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='finished',
|
||||
progress=100,
|
||||
phase='Complete',
|
||||
log_line='Library scan completed',
|
||||
log_type='success',
|
||||
)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'_manages_own_progress': True,
|
||||
'scan_duration_seconds': elapsed,
|
||||
}
|
||||
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
status='error',
|
||||
phase='Error',
|
||||
log_line=str(e),
|
||||
log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
|
||||
|
||||
finally:
|
||||
deps.state.set_scan_library_id(None)
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
"""Automation handler: ``scan_watchlist`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_scan_watchlist`` closure). The watchlist scanner returns
|
||||
summary stats for the trigger card only when a fresh scan actually
|
||||
ran — detected by snapshotting ``id(state_dict)`` before/after, since
|
||||
the live processor reassigns the dict on each new scan.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_scan_watchlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Run a watchlist scan when the automation triggers.
|
||||
|
||||
Pre-scan we capture ``id(watchlist_scan_state)`` so we can tell
|
||||
afterwards whether the worker ran (and reassigned the state dict)
|
||||
or short-circuited (kept the same dict). Only fresh scans report
|
||||
summary stats — repeat triggers without an intervening run return
|
||||
a bare ``completed``.
|
||||
"""
|
||||
try:
|
||||
pre_state = deps.get_watchlist_scan_state()
|
||||
pre_state_id = id(pre_state)
|
||||
deps.process_watchlist_scan_automatically(
|
||||
automation_id=config.get('_automation_id'),
|
||||
profile_id=config.get('_profile_id'),
|
||||
)
|
||||
post_state = deps.get_watchlist_scan_state()
|
||||
# Fresh scan = state dict was reassigned mid-run.
|
||||
if id(post_state) != pre_state_id:
|
||||
summary = post_state.get('summary', {}) if isinstance(post_state, dict) else {}
|
||||
return {
|
||||
'status': 'completed',
|
||||
'artists_scanned': summary.get('total_artists', 0),
|
||||
'successful_scans': summary.get('successful_scans', 0),
|
||||
'new_tracks_found': summary.get('new_tracks_found', 0),
|
||||
'tracks_added_to_wishlist': summary.get('tracks_added_to_wishlist', 0),
|
||||
}
|
||||
return {'status': 'completed'}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
|
||||
return {'status': 'error', 'error': str(e)}
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
"""Automation handler: ``search_and_download`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_search_and_download`` closure). Searches for a track by
|
||||
name/artist string and dispatches the best match through the
|
||||
download orchestrator. Query can come from the trigger config
|
||||
(direct value) or from event data (e.g. webhook payload).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_search_and_download(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
automation_id = config.get('_automation_id')
|
||||
query = config.get('query', '').strip()
|
||||
# Event-triggered: pull query from event data (e.g. webhook_received).
|
||||
if not query:
|
||||
event_data = config.get('_event_data', {})
|
||||
query = (event_data.get('query', '') or '').strip()
|
||||
if not query:
|
||||
if automation_id:
|
||||
deps.update_progress(
|
||||
automation_id, log_line='No search query provided', log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'error': 'No search query provided'}
|
||||
try:
|
||||
if automation_id:
|
||||
deps.update_progress(
|
||||
automation_id, phase='Searching',
|
||||
log_line=f'Searching: {query}', log_type='info',
|
||||
)
|
||||
result = deps.run_async(deps.download_orchestrator.search_and_download_best(query))
|
||||
if result:
|
||||
if automation_id:
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'Download started for: {query}',
|
||||
log_type='success',
|
||||
)
|
||||
return {'status': 'completed', 'query': query, 'download_id': result}
|
||||
if automation_id:
|
||||
deps.update_progress(
|
||||
automation_id,
|
||||
log_line=f'No match found for: {query}',
|
||||
log_type='warning',
|
||||
)
|
||||
return {'status': 'not_found', 'query': query, 'error': 'No match found'}
|
||||
except Exception as e: # noqa: BLE001 — automation handlers must never raise
|
||||
if automation_id:
|
||||
deps.update_progress(
|
||||
automation_id, log_line=f'Error: {e}', log_type='error',
|
||||
)
|
||||
return {'status': 'error', 'query': query, 'error': str(e)}
|
||||
|
|
@ -1,240 +0,0 @@
|
|||
"""Automation handler: ``sync_playlist`` action.
|
||||
|
||||
Lifted from ``web_server._register_automation_handlers`` (the
|
||||
``_auto_sync_playlist`` closure). Syncs a mirrored playlist to the
|
||||
configured media server, using discovered metadata when available
|
||||
and skipping undiscovered tracks. When triggered on a schedule with
|
||||
no track changes since the last sync, short-circuits with
|
||||
``status: skipped`` (saves Plex / Jellyfin / Navidrome from
|
||||
needless rewrites)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import threading
|
||||
from typing import Any, Dict
|
||||
|
||||
from core.automation.deps import AutomationDeps
|
||||
|
||||
|
||||
def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
|
||||
"""Sync a mirrored playlist to the active media server.
|
||||
|
||||
Behavior:
|
||||
- Tracks with discovered metadata (extra_data.discovered + matched_data)
|
||||
are routed via the official metadata.
|
||||
- Tracks with a Spotify hint (real Spotify ID from the embed
|
||||
scraper) are included so they can still hit Soulseek + the
|
||||
wishlist.
|
||||
- Tracks with neither are counted as ``skipped_tracks``.
|
||||
- Empty result → ``status: skipped`` with the skipped count.
|
||||
- Same track set as last sync (matched_tracks unchanged) →
|
||||
``status: skipped`` (no-op).
|
||||
- Otherwise spawns a daemon thread running ``run_sync_task`` and
|
||||
returns ``status: started`` with ``_manages_own_progress: True``.
|
||||
"""
|
||||
auto_id = config.get('_automation_id')
|
||||
playlist_id = config.get('playlist_id')
|
||||
if not playlist_id:
|
||||
return {'status': 'error', 'reason': 'No playlist specified'}
|
||||
|
||||
db = deps.get_database()
|
||||
pl = db.get_mirrored_playlist(int(playlist_id))
|
||||
if not pl:
|
||||
return {'status': 'error', 'reason': 'Playlist not found'}
|
||||
|
||||
tracks = db.get_mirrored_playlist_tracks(int(playlist_id))
|
||||
if not tracks:
|
||||
return {'status': 'error', 'reason': 'No tracks in playlist'}
|
||||
|
||||
# Convert mirrored tracks to format expected by run_sync_task.
|
||||
# Use discovered metadata when available, fall back to Spotify
|
||||
# hint or raw playlist fields when not.
|
||||
tracks_json = []
|
||||
skipped_count = 0
|
||||
|
||||
for t in tracks:
|
||||
# Parse extra_data for discovery info.
|
||||
extra = {}
|
||||
if t.get('extra_data'):
|
||||
try:
|
||||
extra = json.loads(t['extra_data']) if isinstance(t['extra_data'], str) else t['extra_data']
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
|
||||
if extra.get('discovered') and extra.get('matched_data'):
|
||||
# Use official discovered metadata.
|
||||
md = extra['matched_data']
|
||||
album_raw = md.get('album', '')
|
||||
album_obj = album_raw if isinstance(album_raw, dict) else {'name': album_raw or ''}
|
||||
_track_entry = {
|
||||
'name': md.get('name', ''),
|
||||
'artists': md.get('artists', [{'name': t.get('artist_name', '')}]),
|
||||
'album': album_obj,
|
||||
'duration_ms': md.get('duration_ms', 0),
|
||||
'id': md.get('id', ''),
|
||||
}
|
||||
if md.get('track_number'):
|
||||
_track_entry['track_number'] = md['track_number']
|
||||
if md.get('disc_number'):
|
||||
_track_entry['disc_number'] = md['disc_number']
|
||||
tracks_json.append(_track_entry)
|
||||
else:
|
||||
# NOT discovered — try to include using available metadata so
|
||||
# the track can still be searched on Soulseek and added to
|
||||
# wishlist. Without this, failed discovery blocks the entire
|
||||
# download pipeline.
|
||||
#
|
||||
# Priority: spotify_hint (has real Spotify ID from embed
|
||||
# scraper) > raw playlist fields (only if source_track_id
|
||||
# is valid).
|
||||
hint = extra.get('spotify_hint', {})
|
||||
# Build album object with cover art from the mirrored playlist track.
|
||||
track_image = (t.get('image_url') or '').strip()
|
||||
album_obj = {
|
||||
'name': (t.get('album_name') or '').strip(),
|
||||
'images': [{'url': track_image, 'height': 300, 'width': 300}] if track_image else [],
|
||||
}
|
||||
|
||||
if hint.get('id') and hint.get('name'):
|
||||
# spotify_hint has proper Spotify track ID + metadata from embed scraper.
|
||||
hint_artists = hint.get('artists', [])
|
||||
if hint_artists and isinstance(hint_artists[0], str):
|
||||
hint_artists = [{'name': a} for a in hint_artists]
|
||||
elif hint_artists and isinstance(hint_artists[0], dict):
|
||||
pass # Already in correct format
|
||||
else:
|
||||
hint_artists = [{'name': t.get('artist_name', '')}]
|
||||
tracks_json.append({
|
||||
'name': hint['name'],
|
||||
'artists': hint_artists,
|
||||
'album': album_obj,
|
||||
'duration_ms': t.get('duration_ms', 0),
|
||||
'id': hint['id'],
|
||||
})
|
||||
elif t.get('source_track_id') and (t.get('track_name') or '').strip():
|
||||
# Has a valid source ID and track name — usable for wishlist.
|
||||
tracks_json.append({
|
||||
'name': t['track_name'].strip(),
|
||||
'artists': [{'name': (t.get('artist_name') or '').strip() or 'Unknown Artist'}],
|
||||
'album': album_obj,
|
||||
'duration_ms': t.get('duration_ms', 0),
|
||||
'id': t['source_track_id'],
|
||||
})
|
||||
else:
|
||||
skipped_count += 1 # No usable ID or name — truly can't process.
|
||||
|
||||
if not tracks_json:
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'No discovered tracks — {skipped_count} need discovery first',
|
||||
log_type='skip',
|
||||
)
|
||||
return {
|
||||
'status': 'skipped',
|
||||
'reason': f'No discovered tracks to sync ({skipped_count} tracks need discovery first)',
|
||||
'skipped_tracks': str(skipped_count),
|
||||
}
|
||||
|
||||
# Preflight: hash the track list and compare against last sync.
|
||||
# Skip if the exact same set of tracks was already synced and
|
||||
# everything matched (no-op preserves Plex / Jellyfin / Navidrome
|
||||
# from needless rewrites).
|
||||
track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
|
||||
tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
|
||||
|
||||
sync_id_key = f"auto_mirror_{playlist_id}"
|
||||
# Full mirror identity (every source_track_id on the playlist). tracks_hash
|
||||
# only covers tracks_json — if a new mirror row is skipped (no discovery /
|
||||
# no source id), tracks_hash stays identical to the pre-add sync and we
|
||||
# used to no-op with "unchanged" while the new song never hit wishlist.
|
||||
mirror_ids_str = ','.join(
|
||||
sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
|
||||
)
|
||||
mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
|
||||
|
||||
event_data = config.get('_event_data') or {}
|
||||
try:
|
||||
tracks_added = int(event_data.get('added') or 0)
|
||||
except (TypeError, ValueError):
|
||||
tracks_added = 0
|
||||
force_sync = tracks_added > 0 or skipped_count > 0
|
||||
|
||||
try:
|
||||
sync_statuses = deps.load_sync_status_file()
|
||||
last_status = sync_statuses.get(sync_id_key, {})
|
||||
last_hash = last_status.get('tracks_hash', '')
|
||||
last_mirror_hash = last_status.get('mirror_tracks_hash', '')
|
||||
last_matched = last_status.get('matched_tracks', -1)
|
||||
|
||||
mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
|
||||
if (
|
||||
not force_sync
|
||||
and not mirror_changed
|
||||
and last_hash == tracks_hash
|
||||
and last_matched >= len(tracks_json)
|
||||
):
|
||||
# Exact same tracks, all matched last time — nothing to do.
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=f'All {len(tracks_json)} tracks unchanged since last sync — skipping',
|
||||
log_type='skip',
|
||||
)
|
||||
return {
|
||||
'status': 'skipped',
|
||||
'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
|
||||
}
|
||||
if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line=(
|
||||
f'Forcing sync: playlist changed ({tracks_added} added) or '
|
||||
f'{skipped_count} track(s) need discovery'
|
||||
),
|
||||
log_type='info',
|
||||
)
|
||||
elif mirror_changed:
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
log_line='Mirror track list changed — running sync',
|
||||
log_type='info',
|
||||
)
|
||||
except Exception as e:
|
||||
deps.logger.debug("mirror sync last-status read: %s", e)
|
||||
|
||||
# Sync under the user's custom alias when set, else the upstream name (#865
|
||||
# follow-up). The server-side playlist is named with this.
|
||||
from core.playlists.naming import effective_mirrored_name
|
||||
sync_name = effective_mirrored_name(pl) or pl.get('name') or 'Playlist'
|
||||
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
progress=50,
|
||||
phase=f'Syncing "{sync_name}"',
|
||||
log_line=f'{len(tracks_json)} discovered, {skipped_count} skipped',
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
sync_id = f"auto_mirror_{playlist_id}"
|
||||
deps.update_progress(
|
||||
auto_id,
|
||||
progress=90,
|
||||
log_line=f'Starting sync: {len(tracks_json)} tracks',
|
||||
log_type='success',
|
||||
)
|
||||
skip_wishlist_add = bool(pl.get('organize_by_playlist'))
|
||||
threading.Thread(
|
||||
target=deps.run_sync_task,
|
||||
args=(sync_id, sync_name, tracks_json, auto_id, 1, pl.get('image_url', '')),
|
||||
kwargs={'skip_wishlist_add': skip_wishlist_add},
|
||||
daemon=True,
|
||||
name=f'auto-sync-{playlist_id}',
|
||||
).start()
|
||||
return {
|
||||
'status': 'started',
|
||||
'playlist_name': sync_name,
|
||||
'discovered_tracks': str(len(tracks_json)),
|
||||
'skipped_tracks': str(skipped_count),
|
||||
'_manages_own_progress': True,
|
||||
}
|
||||
|
|
@ -1,321 +0,0 @@
|
|||
"""Pure functions for computing the next-run datetime of a scheduled
|
||||
automation trigger.
|
||||
|
||||
The Auto-Sync schedule board currently exposes interval-based scheduling
|
||||
(``every N hours``) backed by ``trigger_type='schedule'``. The
|
||||
automation engine ALSO supports ``daily_time`` and ``weekly_time``
|
||||
triggers via separate ``_setup_*_trigger`` methods inline on the engine
|
||||
class. None of that logic is currently testable in isolation — the
|
||||
engine's ``_finish_run`` reaches for ``datetime.now()``, threads it
|
||||
through ``_next_weekly_occurrence``, and writes the result to the DB,
|
||||
all on the same call.
|
||||
|
||||
This module lifts the "given a trigger config, what's the next run?"
|
||||
question out of the engine into a pure function:
|
||||
|
||||
next_run_at(trigger_type, trigger_config, now_utc, default_tz)
|
||||
-> Optional[datetime]
|
||||
|
||||
That means:
|
||||
- ``now_utc`` is INJECTED, not pulled from the system clock. Tests
|
||||
freeze time without monkeypatching ``datetime.now``.
|
||||
- ``default_tz`` is INJECTED. Daily / weekly / monthly schedules are
|
||||
inherently in the USER'S timezone (cron "every Monday at 9am" is
|
||||
not UTC), and the historic engine implicitly used the server's
|
||||
local tz via naive ``datetime.now()``. That broke for users on a
|
||||
different tz than their server. The pure function takes the tz
|
||||
explicitly so the caller controls it.
|
||||
- Returns an aware UTC ``datetime`` ready to serialise to the DB's
|
||||
``next_run`` string column, or ``None`` for unrecognised /
|
||||
event-based triggers (engine should not store a next_run for those).
|
||||
|
||||
PR 1 of the schedule-types feature ships ONLY this module + tests.
|
||||
The engine continues to compute next_run via its existing inline
|
||||
helpers; PR 2 collapses those into a single ``next_run_at`` call.
|
||||
Net behavior is identical until the engine is wired through — this
|
||||
PR is pure plumbing.
|
||||
|
||||
Schedule types supported here:
|
||||
|
||||
- ``schedule`` (interval): ``{interval: N, unit: 'minutes'|'hours'|'days'}``
|
||||
— adds the interval to ``now_utc``; no tz needed.
|
||||
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — runs every day at
|
||||
the given local time in the given timezone. ``tz`` falls back to
|
||||
``default_tz`` when absent.
|
||||
- ``weekly_time``: ``{time: 'HH:MM', days: ['mon','wed',...], tz: '<IANA>'}``
|
||||
— runs on the matching weekday(s) at the given local time. Empty
|
||||
``days`` list means "every day" (matches the engine's existing
|
||||
fallback in ``_next_weekly_occurrence``).
|
||||
- ``monthly_time``: ``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}``
|
||||
— runs on the given day each month. Days that don't exist in a
|
||||
given month (Feb 30, Apr 31) clamp to the LAST valid day of that
|
||||
month rather than skipping the run entirely; missing a whole
|
||||
month silently because the schedule was over-eager is worse than
|
||||
running a day early.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Dict, Optional
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("automation.schedule")
|
||||
|
||||
|
||||
# Unknown-tz names already warned about in this process — avoids
|
||||
# spamming the log on every poll cycle for the same misconfigured row.
|
||||
_UNKNOWN_TZ_WARNED: set = set()
|
||||
|
||||
|
||||
# Weekday abbreviation → ``datetime.weekday()`` index (Mon=0..Sun=6).
|
||||
# Mirrors the engine's existing ``_next_weekly_occurrence`` mapping so
|
||||
# schedules created against either implementation accept the same
|
||||
# ``days`` strings.
|
||||
_WEEKDAY_MAP = {
|
||||
'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6,
|
||||
}
|
||||
|
||||
# Interval multipliers — kept aligned with the engine's existing
|
||||
# ``_calc_delay_seconds`` in ``core/automation_engine.py``. Adding
|
||||
# entries here without also updating the engine would silently drift:
|
||||
# this function would honour the new unit while the live engine path
|
||||
# defaults it to hours. Keep the maps in sync until PR 2 collapses the
|
||||
# engine through this function.
|
||||
_INTERVAL_MULTIPLIERS = {
|
||||
'minutes': 60,
|
||||
'hours': 60 * 60,
|
||||
'days': 60 * 60 * 24,
|
||||
}
|
||||
|
||||
|
||||
def next_run_at(
|
||||
trigger_type: str,
|
||||
trigger_config: Dict[str, Any],
|
||||
now_utc: datetime,
|
||||
default_tz: str = 'UTC',
|
||||
) -> Optional[datetime]:
|
||||
"""Compute the next-run timestamp (UTC, aware) for a scheduled
|
||||
trigger. Returns ``None`` for unrecognised types or event-based
|
||||
triggers — callers should not write a next_run for those.
|
||||
|
||||
See module docstring for supported trigger types + config shapes.
|
||||
"""
|
||||
if not isinstance(trigger_config, dict):
|
||||
trigger_config = {}
|
||||
|
||||
if trigger_type == 'schedule':
|
||||
return _next_interval(trigger_config, now_utc)
|
||||
if trigger_type == 'daily_time':
|
||||
return _next_daily(trigger_config, now_utc, default_tz)
|
||||
if trigger_type == 'weekly_time':
|
||||
return _next_weekly(trigger_config, now_utc, default_tz)
|
||||
if trigger_type == 'monthly_time':
|
||||
return _next_monthly(trigger_config, now_utc, default_tz)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Interval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _next_interval(config: Dict[str, Any], now_utc: datetime) -> datetime:
|
||||
"""``{interval: N, unit: 'hours'}`` → ``now_utc + N hours``.
|
||||
|
||||
Mirrors the engine's existing ``_calc_delay_seconds``. Unit defaults
|
||||
to ``hours`` for backward compat with legacy DB rows that pre-date
|
||||
the unit field being mandatory; interval defaults to 1 so a fully
|
||||
empty config doesn't divide-by-zero or schedule for the past."""
|
||||
try:
|
||||
interval = max(int(config.get('interval', 1)), 1)
|
||||
except (TypeError, ValueError):
|
||||
interval = 1
|
||||
unit = config.get('unit') or 'hours'
|
||||
seconds = interval * _INTERVAL_MULTIPLIERS.get(unit, _INTERVAL_MULTIPLIERS['hours'])
|
||||
return _ensure_utc(now_utc) + timedelta(seconds=seconds)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Daily
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _next_daily(
|
||||
config: Dict[str, Any], now_utc: datetime, default_tz: str,
|
||||
) -> datetime:
|
||||
"""``{time: 'HH:MM', tz: '<IANA>'}`` → next occurrence of that
|
||||
wall-clock time in the user's timezone, expressed as aware UTC.
|
||||
|
||||
DST-aware via ``zoneinfo``: when the local time falls during a
|
||||
spring-forward gap, the ``replace`` lands on a non-existent
|
||||
instant; ``zoneinfo`` resolves that to the gap's later side
|
||||
(e.g. 02:30 on the DST-forward day becomes 03:30 local). Tests
|
||||
pin both spring-forward and fall-back behaviour."""
|
||||
tz = _resolve_tz(config.get('tz') or default_tz)
|
||||
hour, minute = _parse_hhmm(config.get('time'))
|
||||
now_local = _ensure_utc(now_utc).astimezone(tz)
|
||||
target_local = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if target_local <= now_local:
|
||||
target_local = target_local + timedelta(days=1)
|
||||
return target_local.astimezone(timezone.utc)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Weekly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _next_weekly(
|
||||
config: Dict[str, Any], now_utc: datetime, default_tz: str,
|
||||
) -> datetime:
|
||||
"""``{time: 'HH:MM', days: ['mon',...], tz: '<IANA>'}`` → next
|
||||
occurrence of that wall-clock time on any of the listed weekdays
|
||||
in the user's timezone.
|
||||
|
||||
Empty ``days`` list ≡ every day, matching the engine's existing
|
||||
fallback. Unrecognised day abbreviations are silently dropped
|
||||
(an empty result-set then triggers the every-day fallback)."""
|
||||
tz = _resolve_tz(config.get('tz') or default_tz)
|
||||
hour, minute = _parse_hhmm(config.get('time'))
|
||||
days = _parse_weekdays(config.get('days'))
|
||||
|
||||
now_local = _ensure_utc(now_utc).astimezone(tz)
|
||||
# Scan today + next 7 days; the matching day with a future
|
||||
# local time wins. 8-day scan is enough to handle the case where
|
||||
# today already passed the time AND today is the only allowed
|
||||
# weekday (next occurrence is exactly one week out).
|
||||
for offset in range(8):
|
||||
candidate = now_local + timedelta(days=offset)
|
||||
if candidate.weekday() not in days:
|
||||
continue
|
||||
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if target > now_local:
|
||||
return target.astimezone(timezone.utc)
|
||||
# Shouldn't reach: 8-day scan always finds a hit when ``days``
|
||||
# is non-empty. Defensive fallback: next week, same weekday as today.
|
||||
fallback = (now_local + timedelta(days=7)).replace(
|
||||
hour=hour, minute=minute, second=0, microsecond=0,
|
||||
)
|
||||
return fallback.astimezone(timezone.utc)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Monthly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _next_monthly(
|
||||
config: Dict[str, Any], now_utc: datetime, default_tz: str,
|
||||
) -> datetime:
|
||||
"""``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}`` → next
|
||||
occurrence in the user's timezone.
|
||||
|
||||
``day_of_month`` is clamped to ``[1, 31]``. When the target day
|
||||
doesn't exist in a given month (Feb 30, Apr 31), the schedule
|
||||
falls back to the LAST valid day of that month — running a day
|
||||
or two early in short months is less surprising than skipping
|
||||
a month entirely. This matches the convention every cron
|
||||
implementation in the wild settled on."""
|
||||
tz = _resolve_tz(config.get('tz') or default_tz)
|
||||
hour, minute = _parse_hhmm(config.get('time'))
|
||||
raw_day = config.get('day_of_month', 1)
|
||||
try:
|
||||
target_day = max(1, min(31, int(raw_day)))
|
||||
except (TypeError, ValueError):
|
||||
target_day = 1
|
||||
|
||||
now_local = _ensure_utc(now_utc).astimezone(tz)
|
||||
# Try this month first; if the target day has already passed
|
||||
# (or doesn't exist this month and the clamped day is in the
|
||||
# past), advance to next month. Loop bounded to 12 iterations
|
||||
# so a pathologically broken config can't infinite-loop us.
|
||||
year, month = now_local.year, now_local.month
|
||||
for _ in range(12):
|
||||
day = min(target_day, _days_in_month(year, month))
|
||||
target = now_local.replace(
|
||||
year=year, month=month, day=day,
|
||||
hour=hour, minute=minute, second=0, microsecond=0,
|
||||
)
|
||||
if target > now_local:
|
||||
return target.astimezone(timezone.utc)
|
||||
# Roll to next month.
|
||||
if month == 12:
|
||||
year, month = year + 1, 1
|
||||
else:
|
||||
month += 1
|
||||
# Defensive — should be unreachable.
|
||||
return (now_local + timedelta(days=30)).astimezone(timezone.utc)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ensure_utc(dt: datetime) -> datetime:
|
||||
"""Coerce a possibly-naive datetime to aware UTC. Naive inputs
|
||||
are assumed UTC (matches the convention the engine uses when
|
||||
parsing the DB ``next_run`` column)."""
|
||||
if dt.tzinfo is None:
|
||||
return dt.replace(tzinfo=timezone.utc)
|
||||
return dt.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def _resolve_tz(name: Optional[str]):
|
||||
"""Look up an IANA tz by name. Falls back to UTC when the name is
|
||||
unknown — ``ZoneInfoNotFoundError`` is the symptom of either a
|
||||
typo in the tz string or ``tzdata`` missing on the host. Logged
|
||||
once per unknown name so the user can see WHY their schedule
|
||||
isn't running in the timezone they configured."""
|
||||
if not name:
|
||||
return timezone.utc
|
||||
try:
|
||||
return ZoneInfo(name)
|
||||
except ZoneInfoNotFoundError:
|
||||
if name not in _UNKNOWN_TZ_WARNED:
|
||||
_UNKNOWN_TZ_WARNED.add(name)
|
||||
logger.warning(
|
||||
"Unknown timezone %r — schedule will run against UTC. "
|
||||
"Check the spelling (IANA format like 'America/Los_Angeles') "
|
||||
"or install the `tzdata` package on minimal hosts.",
|
||||
name,
|
||||
)
|
||||
return timezone.utc
|
||||
|
||||
|
||||
def _parse_hhmm(time_str: Optional[str]) -> tuple:
|
||||
"""Parse ``HH:MM`` → ``(hour, minute)``. Defaults to 00:00 on
|
||||
garbage input — same defensive shape as the engine's existing
|
||||
daily/weekly time parsing."""
|
||||
if not isinstance(time_str, str):
|
||||
return 0, 0
|
||||
try:
|
||||
h, m = time_str.split(':', 1)
|
||||
return max(0, min(23, int(h))), max(0, min(59, int(m)))
|
||||
except (ValueError, AttributeError):
|
||||
return 0, 0
|
||||
|
||||
|
||||
def _parse_weekdays(days) -> set:
|
||||
"""``['mon', 'wed']`` → ``{0, 2}``. Empty / missing / all-invalid
|
||||
list returns ``set(range(7))`` ("every day"), matching the
|
||||
engine's existing ``_next_weekly_occurrence`` fallback."""
|
||||
if not isinstance(days, (list, tuple)):
|
||||
return set(range(7))
|
||||
parsed = {_WEEKDAY_MAP[d.lower()] for d in days
|
||||
if isinstance(d, str) and d.lower() in _WEEKDAY_MAP}
|
||||
return parsed or set(range(7))
|
||||
|
||||
|
||||
def _days_in_month(year: int, month: int) -> int:
|
||||
"""Last calendar day of ``year-month``. Stdlib-only — no calendar
|
||||
module import needed; cycle through the 12 months."""
|
||||
if month == 12:
|
||||
next_first = datetime(year + 1, 1, 1)
|
||||
else:
|
||||
next_first = datetime(year, month + 1, 1)
|
||||
last_day = next_first - timedelta(days=1)
|
||||
return last_day.day
|
||||
|
|
@ -18,14 +18,10 @@ import time
|
|||
import threading
|
||||
import requests
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
from core.automation.schedule import next_run_at
|
||||
|
||||
logger = get_logger("automation_engine")
|
||||
|
||||
|
||||
def _utcnow():
|
||||
"""Return current UTC time as timezone-aware datetime."""
|
||||
return datetime.now(timezone.utc)
|
||||
|
|
@ -38,42 +34,6 @@ def _utc_after(seconds):
|
|||
"""Return UTC time N seconds from now as naive string for DB storage."""
|
||||
return (datetime.now(timezone.utc) + timedelta(seconds=seconds)).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
|
||||
def _dt_to_db_str(dt: datetime) -> str:
|
||||
"""Convert an aware-UTC datetime to the naive-UTC string the DB
|
||||
``next_run`` column stores. Centralised so a tz mistake here
|
||||
surfaces in one place, not scattered through every caller of
|
||||
``next_run_at``."""
|
||||
if dt.tzinfo is None:
|
||||
return dt.strftime('%Y-%m-%d %H:%M:%S')
|
||||
return dt.astimezone(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
|
||||
def _resolve_system_default_tz() -> str:
|
||||
"""Return the IANA tz name the engine uses when a schedule's
|
||||
trigger config doesn't carry an explicit ``tz`` field.
|
||||
|
||||
Existing daily / weekly rows pre-date the ``tz`` field — the
|
||||
historic engine computed delays from naive ``datetime.now()``,
|
||||
which is implicitly the server's local timezone. Falling back to
|
||||
that same tz here preserves "every Monday at 09:00" running at
|
||||
09:00 server local for rows that already exist in the DB.
|
||||
Without ``tzlocal`` installed (or a system without a discoverable
|
||||
tz), falls back to UTC."""
|
||||
try:
|
||||
import tzlocal
|
||||
return tzlocal.get_localzone_name() or 'UTC'
|
||||
except Exception:
|
||||
return 'UTC'
|
||||
|
||||
|
||||
# Server-local tz cached at import time. Re-reading per-call is
|
||||
# pointless: the host's timezone doesn't change while the process is
|
||||
# running. Tests that need a different default tz inject it through
|
||||
# the engine's ``_default_tz`` attribute or via the
|
||||
# ``automation.default_timezone`` config key.
|
||||
_SYSTEM_DEFAULT_TZ = _resolve_system_default_tz()
|
||||
|
||||
SYSTEM_AUTOMATIONS = [
|
||||
{
|
||||
'name': 'Auto-Process Wishlist',
|
||||
|
|
@ -174,25 +134,11 @@ class AutomationEngine:
|
|||
self._max_chain_depth = 5
|
||||
self._signal_cooldown_seconds = 10
|
||||
|
||||
# Default tz used when a schedule's ``trigger_config`` doesn't
|
||||
# carry an explicit ``tz`` field — preserves historic behaviour
|
||||
# for daily / weekly rows created before the field existed
|
||||
# (engine used naive ``datetime.now()`` = server local). Reads
|
||||
# from the ``automation.default_timezone`` config key first to
|
||||
# let users override without touching env vars; falls back to
|
||||
# the system-detected local tz.
|
||||
try:
|
||||
from config.settings import config_manager
|
||||
self._default_tz = (config_manager.get('automation.default_timezone', '') or _SYSTEM_DEFAULT_TZ)
|
||||
except Exception:
|
||||
self._default_tz = _SYSTEM_DEFAULT_TZ
|
||||
|
||||
# Trigger registry: type → setup function (schedule only — events use emit())
|
||||
self._trigger_handlers = {
|
||||
'schedule': self._setup_schedule_trigger,
|
||||
'daily_time': self._setup_daily_time_trigger,
|
||||
'weekly_time': self._setup_weekly_time_trigger,
|
||||
'monthly_time': self._setup_monthly_time_trigger,
|
||||
}
|
||||
|
||||
# --- Action Handler Registration ---
|
||||
|
|
@ -635,11 +581,6 @@ class AutomationEngine:
|
|||
action_config['_automation_name'] = auto.get('name', '')
|
||||
if profile_id is not None:
|
||||
action_config['_profile_id'] = profile_id
|
||||
# The profile this run acts AS: an explicit trigger profile, else the
|
||||
# automation's owner, else admin. System + admin automations are
|
||||
# profile 1, so this is a no-op for them — only non-admin-owned
|
||||
# automations gain their correct identity in the background.
|
||||
_effective_profile_id = profile_id if profile_id is not None else (auto.get('profile_id') or 1)
|
||||
|
||||
# Action delay (skipped for manual run_now)
|
||||
delay_minutes = action_config.get('delay', 0)
|
||||
|
|
@ -676,7 +617,7 @@ class AutomationEngine:
|
|||
self._progress_finish_fn(automation_id, result)
|
||||
except Exception as e:
|
||||
logger.debug("scheduled progress finish (skipped): %s", e)
|
||||
self._finish_run(auto, automation_id, result, error=None, retry_delay_seconds=300)
|
||||
self._finish_run(auto, automation_id, result, error=None)
|
||||
return
|
||||
|
||||
# Initialize progress tracking (skip if already done during delay)
|
||||
|
|
@ -686,14 +627,9 @@ class AutomationEngine:
|
|||
except Exception as e:
|
||||
logger.debug("scheduled progress init: %s", e)
|
||||
|
||||
# Execute the action under the owner's profile so get_current_profile_id()
|
||||
# (and the per-profile clients it resolves) act as the automation's owner
|
||||
# in the background, not admin. Reset in finally so a pooled thread can't
|
||||
# leak the override to the next job.
|
||||
# Execute the action
|
||||
error = None
|
||||
result = {}
|
||||
from core.profile_context import set_background_profile, reset_background_profile
|
||||
_bg_token = set_background_profile(_effective_profile_id)
|
||||
try:
|
||||
result = handler_info['handler'](action_config) or {}
|
||||
logger.info(f"Automation '{auto['name']}' (id={automation_id}) executed: {result.get('status', 'ok')}")
|
||||
|
|
@ -712,8 +648,6 @@ class AutomationEngine:
|
|||
error = str(e)
|
||||
result = {'status': 'error', 'error': error}
|
||||
logger.error(f"Automation '{auto['name']}' (id={automation_id}) failed: {e}")
|
||||
finally:
|
||||
reset_background_profile(_bg_token)
|
||||
|
||||
# Finalize progress tracking
|
||||
if self._progress_finish_fn:
|
||||
|
|
@ -730,7 +664,7 @@ class AutomationEngine:
|
|||
|
||||
self._finish_run(auto, automation_id, result, error)
|
||||
|
||||
def _finish_run(self, auto, automation_id, result, error, retry_delay_seconds=None):
|
||||
def _finish_run(self, auto, automation_id, result, error):
|
||||
"""Update DB with run stats and reschedule."""
|
||||
next_run_str = None
|
||||
trigger_type = auto.get('trigger_type', '')
|
||||
|
|
@ -738,23 +672,23 @@ class AutomationEngine:
|
|||
if trigger_type in self._trigger_handlers:
|
||||
try:
|
||||
trigger_config = json.loads(auto.get('trigger_config') or '{}')
|
||||
if retry_delay_seconds:
|
||||
next_run_str = _utc_after(retry_delay_seconds)
|
||||
if trigger_type == 'daily_time':
|
||||
# Next run is tomorrow at the configured time (compute delay from local time, store as UTC)
|
||||
time_str = trigger_config.get('time', '00:00')
|
||||
hour, minute = map(int, time_str.split(':'))
|
||||
now_local = datetime.now()
|
||||
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
|
||||
next_run_str = _utc_after((target - now_local).total_seconds())
|
||||
elif trigger_type == 'weekly_time':
|
||||
time_str = trigger_config.get('time', '00:00')
|
||||
hour, minute = map(int, time_str.split(':'))
|
||||
now_local = datetime.now()
|
||||
target = self._next_weekly_occurrence(hour, minute, trigger_config.get('days', []))
|
||||
next_run_str = _utc_after((target - now_local).total_seconds())
|
||||
else:
|
||||
# Single integration point with ``next_run_at``. The
|
||||
# helper handles every trigger type the engine
|
||||
# supports (interval / daily / weekly / monthly) and
|
||||
# returns aware-UTC; ``_dt_to_db_str`` normalises to
|
||||
# the naive-UTC string the DB column stores. Tests
|
||||
# injecting a different ``now_utc`` patch this same
|
||||
# path — no scattered ``datetime.now()`` calls left.
|
||||
next_run_dt = next_run_at(
|
||||
trigger_type, trigger_config,
|
||||
now_utc=_utcnow(),
|
||||
default_tz=self._default_tz,
|
||||
)
|
||||
if next_run_dt is not None:
|
||||
next_run_str = _dt_to_db_str(next_run_dt)
|
||||
delay = self._calc_delay_seconds(trigger_config)
|
||||
if delay:
|
||||
next_run_str = _utc_after(delay)
|
||||
except Exception as e:
|
||||
logger.debug("next run calc failed: %s", e)
|
||||
|
||||
|
|
@ -828,72 +762,22 @@ class AutomationEngine:
|
|||
logger.debug(f"Scheduled automation {automation_id} in {delay:.0f}s")
|
||||
|
||||
def _setup_daily_time_trigger(self, automation_id, config):
|
||||
"""Config: ``{"time": "03:00", "tz": "<IANA>"}`` — runs daily
|
||||
at the specified local time. Tz defaults to ``self._default_tz``
|
||||
when absent."""
|
||||
self._setup_timed_trigger(automation_id, 'daily_time', config,
|
||||
label=f"Daily at {config.get('time', '00:00')}")
|
||||
"""Config: {"time": "03:00"} — runs daily at the specified local time."""
|
||||
time_str = config.get('time', '00:00')
|
||||
try:
|
||||
hour, minute = map(int, time_str.split(':'))
|
||||
except (ValueError, AttributeError):
|
||||
hour, minute = 0, 0
|
||||
|
||||
def _setup_weekly_time_trigger(self, automation_id, config):
|
||||
"""Config: ``{"time": "03:00", "days": ["mon","wed","fri"], "tz": "<IANA>"}``."""
|
||||
day_names = ', '.join(config.get('days') or []) or 'every day'
|
||||
self._setup_timed_trigger(automation_id, 'weekly_time', config,
|
||||
label=f"Weekly {config.get('time', '00:00')} on {day_names}")
|
||||
now_local = datetime.now()
|
||||
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if target <= now_local:
|
||||
target += timedelta(days=1)
|
||||
|
||||
def _setup_monthly_time_trigger(self, automation_id, config):
|
||||
"""Config: ``{"time": "09:00", "day_of_month": 15, "tz": "<IANA>"}``.
|
||||
delay = (target - now_local).total_seconds()
|
||||
|
||||
Day clamped to [1, 31]; months too short for the target day
|
||||
clamp to the last valid day (Feb 31 → Feb 28 / Feb 29 leap
|
||||
year) per standard cron convention — see
|
||||
``core.automation.schedule._next_monthly`` for the rule."""
|
||||
day = config.get('day_of_month', 1)
|
||||
self._setup_timed_trigger(automation_id, 'monthly_time', config,
|
||||
label=f"Monthly {config.get('time', '00:00')} on day {day}")
|
||||
|
||||
def _setup_timed_trigger(self, automation_id, trigger_type, config, *, label):
|
||||
"""Shared setup for daily / weekly / monthly time triggers.
|
||||
|
||||
All three flow through the same skeleton: compute next-run
|
||||
via ``next_run_at``, persist to DB, arm a ``threading.Timer``
|
||||
that fires the automation when the delay elapses. Lifting
|
||||
these out of three near-identical methods means there's one
|
||||
place to fix when (e.g.) timer rearm semantics need a tweak.
|
||||
|
||||
Honours an existing future ``next_run`` row in the DB —
|
||||
prevents losing a hand-edited next_run when the engine
|
||||
reschedules at startup. Same guard as the interval path."""
|
||||
target_dt = next_run_at(
|
||||
trigger_type, config or {},
|
||||
now_utc=_utcnow(),
|
||||
default_tz=self._default_tz,
|
||||
)
|
||||
if target_dt is None:
|
||||
logger.warning(
|
||||
f"Skip scheduling automation {automation_id}: next_run_at returned "
|
||||
f"None for {trigger_type!r}",
|
||||
)
|
||||
return
|
||||
|
||||
delay = max(0.0, (target_dt - _utcnow()).total_seconds())
|
||||
|
||||
# If the DB already carries a future next_run, prefer it —
|
||||
# matches the interval-path behaviour and lets manual edits
|
||||
# or pending retries survive a process restart.
|
||||
auto = self.db.get_automation(automation_id)
|
||||
if auto and auto.get('next_run'):
|
||||
try:
|
||||
existing = datetime.strptime(
|
||||
auto['next_run'], '%Y-%m-%d %H:%M:%S',
|
||||
).replace(tzinfo=timezone.utc)
|
||||
remaining = (existing - _utcnow()).total_seconds()
|
||||
if remaining > 0:
|
||||
delay = remaining
|
||||
target_dt = existing
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
self.db.update_automation(automation_id, next_run=_dt_to_db_str(target_dt))
|
||||
next_run_str = _utc_after(delay)
|
||||
self.db.update_automation(automation_id, next_run=next_run_str)
|
||||
|
||||
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
|
||||
timer.daemon = True
|
||||
|
|
@ -902,7 +786,48 @@ class AutomationEngine:
|
|||
with self._lock:
|
||||
self._timers[automation_id] = timer
|
||||
|
||||
logger.debug(f"{label} automation {automation_id} scheduled (in {delay:.0f}s)")
|
||||
logger.debug(f"Daily automation {automation_id} scheduled for {time_str} (in {delay:.0f}s)")
|
||||
|
||||
def _setup_weekly_time_trigger(self, automation_id, config):
|
||||
"""Config: {"time": "03:00", "days": ["mon", "wed", "fri"]}"""
|
||||
time_str = config.get('time', '00:00')
|
||||
try:
|
||||
hour, minute = map(int, time_str.split(':'))
|
||||
except (ValueError, AttributeError):
|
||||
hour, minute = 0, 0
|
||||
|
||||
target = self._next_weekly_occurrence(hour, minute, config.get('days', []))
|
||||
delay = (target - datetime.now()).total_seconds()
|
||||
|
||||
next_run_str = _utc_after(delay)
|
||||
self.db.update_automation(automation_id, next_run=next_run_str)
|
||||
|
||||
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
|
||||
timer.daemon = True
|
||||
timer.start()
|
||||
|
||||
with self._lock:
|
||||
self._timers[automation_id] = timer
|
||||
|
||||
day_names = ', '.join(config.get('days', [])) or 'every day'
|
||||
logger.debug(f"Weekly automation {automation_id} scheduled for {time_str} on {day_names} (in {delay:.0f}s)")
|
||||
|
||||
def _next_weekly_occurrence(self, hour, minute, days):
|
||||
"""Find the next datetime matching one of the given weekday abbreviations."""
|
||||
day_map = {'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6}
|
||||
allowed = {day_map[d] for d in days if d in day_map}
|
||||
if not allowed:
|
||||
allowed = set(range(7)) # no days selected = every day
|
||||
|
||||
now = datetime.now()
|
||||
for offset in range(8): # check today + next 7 days
|
||||
candidate = now + timedelta(days=offset)
|
||||
if candidate.weekday() in allowed:
|
||||
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if target > now:
|
||||
return target
|
||||
# Fallback: tomorrow (shouldn't happen with 8-day scan)
|
||||
return now.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
|
||||
|
||||
# --- Then Actions (notifications + signals) ---
|
||||
|
||||
|
|
@ -1072,8 +997,6 @@ class AutomationEngine:
|
|||
"""Send message via Telegram Bot API."""
|
||||
bot_token = config.get('bot_token', '').strip()
|
||||
chat_id = config.get('chat_id', '').strip()
|
||||
thread_id = config.get('thread_id', '').strip()
|
||||
|
||||
if not bot_token or not chat_id:
|
||||
raise ValueError("Bot token and chat ID are required for Telegram")
|
||||
|
||||
|
|
@ -1082,16 +1005,9 @@ class AutomationEngine:
|
|||
for key, value in variables.items():
|
||||
message = message.replace('{' + key + '}', value)
|
||||
|
||||
payload = {"chat_id": chat_id, "text": message, "parse_mode": "HTML"}
|
||||
if thread_id:
|
||||
try:
|
||||
payload["message_thread_id"] = int(thread_id)
|
||||
except ValueError:
|
||||
pass # invalid — fall back to main chat
|
||||
|
||||
resp = requests.post(
|
||||
f'https://api.telegram.org/bot{bot_token}/sendMessage',
|
||||
json=payload,
|
||||
json={"chat_id": chat_id, "text": message, "parse_mode": "HTML"},
|
||||
timeout=10,
|
||||
)
|
||||
data = resp.json() if resp.status_code == 200 else {}
|
||||
|
|
|
|||
|
|
@ -1,37 +0,0 @@
|
|||
"""Artist / album / track blocklist (the "proper" blacklist).
|
||||
|
||||
Distinct from ``download_blacklist`` (which skips one bad source file from one
|
||||
Soulseek peer — untouched here). This blocklist bans an ARTIST, ALBUM, or
|
||||
TRACK from being acquired, keyed by metadata-source IDs (Spotify / iTunes /
|
||||
Deezer / MusicBrainz) so a ban survives a source switch.
|
||||
|
||||
Phase 1 enforces at the single ``add_to_wishlist`` chokepoint: every
|
||||
auto-acquisition path (watchlist, discography backfill, repair, manual
|
||||
wishlist add) funnels through it, so one guard covers them all.
|
||||
|
||||
- ``matching`` — the pure decision core (no DB, no I/O): build an index from
|
||||
blocklist rows, ask whether a candidate is blocked, with artist→album→track
|
||||
cascade.
|
||||
"""
|
||||
|
||||
from core.blocklist.matching import (
|
||||
ENTITY_ALBUM,
|
||||
ENTITY_ARTIST,
|
||||
ENTITY_TRACK,
|
||||
ENTITY_TYPES,
|
||||
SOURCE_ID_FIELDS,
|
||||
BlocklistIndex,
|
||||
build_index,
|
||||
candidate_block_reason,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ENTITY_ARTIST",
|
||||
"ENTITY_ALBUM",
|
||||
"ENTITY_TRACK",
|
||||
"ENTITY_TYPES",
|
||||
"SOURCE_ID_FIELDS",
|
||||
"BlocklistIndex",
|
||||
"build_index",
|
||||
"candidate_block_reason",
|
||||
]
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
"""Cross-source ID backfill for blocklist entries.
|
||||
|
||||
When a user blocks an item, the modal gives us the ID for ONE source (the one
|
||||
they searched). For the ban to survive a source switch, we resolve the OTHER
|
||||
sources' IDs too — matching the blocked artist/album/track by name on each
|
||||
source and taking a confident hit.
|
||||
|
||||
The resolution is kept pure + injected so it tests without a network: callers
|
||||
pass a ``resolvers`` map ``{source: fn(entity_type, name, parent_name) -> id |
|
||||
None}``. ``core/blocklist/runtime.py`` wires the real metadata clients.
|
||||
|
||||
Honest about fragility (acknowledged in design): artist matching is reliable,
|
||||
album/track cross-source matching is best-effort (editions, common titles), so
|
||||
a resolver returning None just leaves that source unmatched — the artist
|
||||
name-fallback in matching.py covers artist gaps; album/track gaps mean that
|
||||
ban only applies on sources where an ID resolved.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
from core.blocklist.matching import SOURCE_ID_FIELDS
|
||||
|
||||
|
||||
def resolve_missing_ids(
|
||||
entry: Dict[str, Any],
|
||||
resolvers: Dict[str, Callable[..., Optional[str]]],
|
||||
) -> Dict[str, str]:
|
||||
"""Return ``{id_column: resolved_id}`` for the sources currently missing an
|
||||
ID on ``entry``. Never raises — a resolver that errors is skipped."""
|
||||
out: Dict[str, str] = {}
|
||||
entity_type = entry.get("entity_type")
|
||||
name = entry.get("name")
|
||||
parent = entry.get("parent_name")
|
||||
if not entity_type or not name:
|
||||
return out
|
||||
for source, col in SOURCE_ID_FIELDS.items():
|
||||
if entry.get(col):
|
||||
continue # already known
|
||||
fn = resolvers.get(source)
|
||||
if not fn:
|
||||
continue
|
||||
try:
|
||||
rid = fn(entity_type, name, parent)
|
||||
except Exception:
|
||||
rid = None
|
||||
if rid:
|
||||
out[col] = str(rid)
|
||||
return out
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
"""Pure blocklist matching — no DB, no I/O, fully unit-testable.
|
||||
|
||||
The brain of the blocklist: given the stored blocklist rows and a candidate
|
||||
track being considered for the wishlist, decide whether it's blocked.
|
||||
|
||||
Design decisions (per Boulder):
|
||||
- **ID-keyed.** Each row carries the candidate's IDs in up to four metadata
|
||||
sources. A candidate is matched against the SAME source it came in on
|
||||
(the wishlist payload carries active-source IDs), so a Deezer-numeric id
|
||||
can't collide with an iTunes-numeric id of a different entity.
|
||||
- **Cascade.** Blocking an artist blocks their albums + tracks; blocking an
|
||||
album blocks its tracks. The candidate carries its own artist/album/track
|
||||
IDs, so the check walks track → album → artist and blocks on the first hit.
|
||||
- **Name fallback for ARTISTS only.** A blocked artist also matches by
|
||||
case-folded name — this covers the window before the background ID-backfill
|
||||
has resolved the active source's id. Albums/tracks do NOT fall back to name
|
||||
(common titles like "Greatest Hits" would false-positive across artists);
|
||||
they rely on IDs, which backfill fills in.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
|
||||
|
||||
ENTITY_ARTIST = "artist"
|
||||
ENTITY_ALBUM = "album"
|
||||
ENTITY_TRACK = "track"
|
||||
ENTITY_TYPES = (ENTITY_ARTIST, ENTITY_ALBUM, ENTITY_TRACK)
|
||||
|
||||
# Blocklist-row column → the metadata source it belongs to.
|
||||
SOURCE_ID_FIELDS = {
|
||||
"spotify": "spotify_id",
|
||||
"itunes": "itunes_id",
|
||||
"deezer": "deezer_id",
|
||||
"musicbrainz": "musicbrainz_id",
|
||||
}
|
||||
|
||||
|
||||
def _norm(text: Any) -> str:
|
||||
return str(text or "").strip().casefold()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _TypeIndex:
|
||||
# per-source set of blocked ids, plus a case-folded name set (artists only)
|
||||
ids: Dict[str, Set[str]] = field(default_factory=lambda: {s: set() for s in SOURCE_ID_FIELDS})
|
||||
names: Set[str] = field(default_factory=set)
|
||||
|
||||
def hit(self, source: Optional[str], entity_id: Any, name: Any, use_name: bool) -> bool:
|
||||
if entity_id and source in self.ids and str(entity_id) in self.ids[source]:
|
||||
return True
|
||||
if use_name and name and _norm(name) in self.names:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
@dataclass
|
||||
class BlocklistIndex:
|
||||
"""Membership index built once per scan from the blocklist rows."""
|
||||
artists: _TypeIndex = field(default_factory=_TypeIndex)
|
||||
albums: _TypeIndex = field(default_factory=_TypeIndex)
|
||||
tracks: _TypeIndex = field(default_factory=_TypeIndex)
|
||||
|
||||
@property
|
||||
def is_empty(self) -> bool:
|
||||
for ti in (self.artists, self.albums, self.tracks):
|
||||
if ti.names or any(ti.ids.values()):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def build_index(rows: Iterable[Dict[str, Any]]) -> BlocklistIndex:
|
||||
"""Build a BlocklistIndex from blocklist DB rows.
|
||||
|
||||
Each row needs ``entity_type``, ``name``, and the source id columns
|
||||
(``spotify_id`` / ``itunes_id`` / ``deezer_id`` / ``musicbrainz_id``).
|
||||
Unknown entity types are ignored."""
|
||||
idx = BlocklistIndex()
|
||||
by_type = {ENTITY_ARTIST: idx.artists, ENTITY_ALBUM: idx.albums, ENTITY_TRACK: idx.tracks}
|
||||
for row in rows or []:
|
||||
ti = by_type.get((row.get("entity_type") or "").strip().lower())
|
||||
if ti is None:
|
||||
continue
|
||||
for source, col in SOURCE_ID_FIELDS.items():
|
||||
val = row.get(col)
|
||||
if val:
|
||||
ti.ids[source].add(str(val))
|
||||
name = row.get("name")
|
||||
if name:
|
||||
ti.names.add(_norm(name))
|
||||
return idx
|
||||
|
||||
|
||||
def candidate_block_reason(
|
||||
index: BlocklistIndex,
|
||||
*,
|
||||
source: Optional[str],
|
||||
track_id: Any = None,
|
||||
track_name: Any = None,
|
||||
album_id: Any = None,
|
||||
album_name: Any = None,
|
||||
artists: Optional[List[Dict[str, Any]]] = None,
|
||||
) -> Optional[Tuple[str, str]]:
|
||||
"""Return ``(entity_type, label)`` for the first cascade hit, else None.
|
||||
|
||||
``source`` is the metadata source the candidate IDs came from (the wishlist
|
||||
payload's provider). ``artists`` is a list of ``{'id', 'name'}`` dicts.
|
||||
Order matters only for the returned reason — any hit blocks."""
|
||||
if index.is_empty:
|
||||
return None
|
||||
|
||||
# Track level — id only (names too ambiguous to ban across artists).
|
||||
if index.tracks.hit(source, track_id, track_name, use_name=False):
|
||||
return (ENTITY_TRACK, str(track_name or track_id or "track"))
|
||||
|
||||
# Album level — id only.
|
||||
if index.albums.hit(source, album_id, album_name, use_name=False):
|
||||
return (ENTITY_ALBUM, str(album_name or album_id or "album"))
|
||||
|
||||
# Artist level — id OR case-folded name (safe + covers the backfill window).
|
||||
for artist in artists or []:
|
||||
a_id = artist.get("id") if isinstance(artist, dict) else None
|
||||
a_name = artist.get("name") if isinstance(artist, dict) else artist
|
||||
if index.artists.hit(source, a_id, a_name, use_name=True):
|
||||
return (ENTITY_ARTIST, str(a_name or a_id or "artist"))
|
||||
|
||||
return None
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
"""Wire real metadata clients to the blocklist backfill resolvers.
|
||||
|
||||
Resolves a blocked item's ID on each metadata source by searching that source
|
||||
for the name and taking a confidently name-matched hit. Confidence = exact
|
||||
significant-token match (drops articles/punctuation) so we never hang a wrong
|
||||
ID on an entry. Albums/tracks additionally require the parent artist to match
|
||||
when both sides expose one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("blocklist.runtime")
|
||||
|
||||
_STOP = {"the", "a", "an", "feat", "ft", "featuring", "with"}
|
||||
|
||||
|
||||
def _tokens(text: Any) -> frozenset:
|
||||
words = re.sub(r"[^a-z0-9]+", " ", str(text or "").lower()).split()
|
||||
return frozenset(w for w in words if w not in _STOP)
|
||||
|
||||
|
||||
def _name_of(obj: Any) -> str:
|
||||
if isinstance(obj, dict):
|
||||
return str(obj.get("name") or obj.get("title") or "")
|
||||
return str(getattr(obj, "name", None) or getattr(obj, "title", None) or "")
|
||||
|
||||
|
||||
def _id_of(obj: Any) -> Optional[str]:
|
||||
val = obj.get("id") if isinstance(obj, dict) else getattr(obj, "id", None)
|
||||
return str(val) if val else None
|
||||
|
||||
|
||||
def _confident(result_name: str, want_name: str) -> bool:
|
||||
rt, wt = _tokens(result_name), _tokens(want_name)
|
||||
return bool(rt) and rt == wt
|
||||
|
||||
|
||||
def _make_resolver(source: str) -> Callable[..., Optional[str]]:
|
||||
def resolve(entity_type: str, name: str, parent_name: Optional[str] = None) -> Optional[str]:
|
||||
from core.metadata.registry import get_client_for_source
|
||||
client = get_client_for_source(source)
|
||||
if not client:
|
||||
return None
|
||||
method = {
|
||||
"artist": "search_artists",
|
||||
"album": "search_albums",
|
||||
"track": "search_tracks",
|
||||
}.get(entity_type)
|
||||
fn = getattr(client, method, None) if method else None
|
||||
if not fn:
|
||||
return None
|
||||
try:
|
||||
results = fn(name, limit=5) or []
|
||||
except Exception as e:
|
||||
logger.debug("%s %s search failed for %r: %s", source, entity_type, name, e)
|
||||
return None
|
||||
for r in results:
|
||||
if not _confident(_name_of(r), name):
|
||||
continue
|
||||
# For album/track, also require the artist to line up when known.
|
||||
if entity_type in ("album", "track") and parent_name:
|
||||
artists = (r.get("artists") if isinstance(r, dict) else getattr(r, "artists", None)) or []
|
||||
cand_artists = " ".join(
|
||||
a.get("name", "") if isinstance(a, dict) else str(a) for a in artists)
|
||||
if _tokens(parent_name) and not (_tokens(parent_name) & _tokens(cand_artists)):
|
||||
continue
|
||||
rid = _id_of(r)
|
||||
if rid:
|
||||
return rid
|
||||
return None
|
||||
|
||||
return resolve
|
||||
|
||||
|
||||
def build_resolvers() -> Dict[str, Callable[..., Optional[str]]]:
|
||||
"""Source→resolver map for core.blocklist.backfill.resolve_missing_ids."""
|
||||
return {s: _make_resolver(s) for s in ("spotify", "itunes", "deezer", "musicbrainz")}
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
"""Boot-phase guard for non-blocking container startup.
|
||||
|
||||
While the gunicorn worker is importing ``web_server`` (module-level client and
|
||||
worker initialization), external provider API probes must not block startup.
|
||||
Network validation is deferred until ``mark_boot_complete()`` runs at the end
|
||||
of that import pass.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
|
||||
_boot_lock = threading.Lock()
|
||||
_boot_active = True
|
||||
|
||||
|
||||
def is_boot_phase() -> bool:
|
||||
"""Return True while module import must avoid blocking provider API calls."""
|
||||
with _boot_lock:
|
||||
return _boot_active
|
||||
|
||||
|
||||
def mark_boot_complete() -> None:
|
||||
"""End the boot phase — provider clients may perform network probes again."""
|
||||
global _boot_active
|
||||
with _boot_lock:
|
||||
_boot_active = False
|
||||
|
|
@ -82,16 +82,7 @@ def run_service_test(service, test_config):
|
|||
if temp_client.is_spotify_authenticated():
|
||||
return True, "Spotify connection successful!"
|
||||
else:
|
||||
# Spotify-Free (no-auth) metadata path: officially unauthenticated,
|
||||
# but the no-creds source is selected and available. Report it as the
|
||||
# working source rather than the generic Deezer/Discogs/iTunes fallback.
|
||||
try:
|
||||
spotify_free_available = temp_client.is_spotify_metadata_available()
|
||||
except Exception:
|
||||
spotify_free_available = False
|
||||
if spotify_free_available:
|
||||
return True, "Spotify (no-auth) connection successful!"
|
||||
# Using a different fallback metadata source
|
||||
# Using fallback metadata source
|
||||
fb_src = _get_metadata_fallback_source()
|
||||
fallback_name = 'Deezer' if fb_src == 'deezer' else 'Discogs' if fb_src == 'discogs' else 'iTunes'
|
||||
if spotify_configured:
|
||||
|
|
@ -200,12 +191,6 @@ def run_service_test(service, test_config):
|
|||
'tidal': "Tidal download source ready.",
|
||||
'qobuz': "Qobuz download source ready.",
|
||||
'hifi': "HiFi download source ready.",
|
||||
'deezer_dl': "Deezer download source ready.",
|
||||
'amazon': "Amazon download source ready.",
|
||||
'lidarr': "Lidarr download source ready.",
|
||||
'soundcloud': "SoundCloud download source ready.",
|
||||
'torrent': "Torrent download source ready.",
|
||||
'usenet': "Usenet download source ready.",
|
||||
'hybrid': "Download sources ready (Hybrid mode)."
|
||||
}
|
||||
message = mode_messages.get(download_mode, "Download source connected.")
|
||||
|
|
@ -218,12 +203,6 @@ def run_service_test(service, test_config):
|
|||
'tidal': "Tidal download source not available. Check authentication.",
|
||||
'qobuz': "Qobuz download source not available. Check authentication.",
|
||||
'hifi': "HiFi download source not available. Public API instances may be down.",
|
||||
'deezer_dl': "Deezer download source not available. Check authentication.",
|
||||
'amazon': "Amazon download source not available.",
|
||||
'lidarr': "Lidarr download source not available. Check Lidarr URL and API key.",
|
||||
'soundcloud': "SoundCloud download source not available.",
|
||||
'torrent': "Torrent download source not available. Check Prowlarr and torrent client settings.",
|
||||
'usenet': "Usenet download source not available. Check Prowlarr and usenet client settings.",
|
||||
'hybrid': "Could not connect to download sources. Check configuration."
|
||||
}
|
||||
error = mode_errors.get(download_mode, "Download source connection failed.")
|
||||
|
|
@ -326,61 +305,6 @@ def run_service_test(service, test_config):
|
|||
return False, "Invalid Genius access token."
|
||||
except Exception as e:
|
||||
return False, f"Genius connection error: {str(e)}"
|
||||
elif service == "usenet_client":
|
||||
client_type = (config_manager.get('usenet_client.type', '') or '').strip().lower()
|
||||
url = config_manager.get('usenet_client.url', '')
|
||||
if not url:
|
||||
return False, "Usenet client URL is required."
|
||||
if not client_type:
|
||||
return False, "Pick a usenet client (SABnzbd or NZBGet)."
|
||||
try:
|
||||
from core.usenet_clients import adapter_for_type as _usenet_adapter_for_type
|
||||
adapter = _usenet_adapter_for_type(client_type)
|
||||
if adapter is None:
|
||||
return False, f"Unknown usenet client type: {client_type}"
|
||||
if not adapter.is_configured():
|
||||
if client_type == "sabnzbd":
|
||||
return False, "SABnzbd needs both URL and API key."
|
||||
return False, "NZBGet needs URL, username, and password."
|
||||
if run_async(adapter.check_connection()):
|
||||
return True, f"Connected to {client_type}"
|
||||
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
|
||||
except Exception as e:
|
||||
return False, f"Usenet client connection error: {str(e)}"
|
||||
elif service == "torrent_client":
|
||||
client_type = (config_manager.get('torrent_client.type', '') or '').strip().lower()
|
||||
url = config_manager.get('torrent_client.url', '')
|
||||
if not url:
|
||||
return False, "Torrent client URL is required."
|
||||
if not client_type:
|
||||
return False, "Pick a torrent client (qBittorrent, Transmission, or Deluge)."
|
||||
try:
|
||||
from core.torrent_clients import adapter_for_type
|
||||
adapter = adapter_for_type(client_type)
|
||||
if adapter is None:
|
||||
return False, f"Unknown torrent client type: {client_type}"
|
||||
if not adapter.is_configured():
|
||||
return False, "Torrent client missing required credentials."
|
||||
if run_async(adapter.check_connection()):
|
||||
return True, f"Connected to {client_type}"
|
||||
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
|
||||
except Exception as e:
|
||||
return False, f"Torrent client connection error: {str(e)}"
|
||||
elif service == "prowlarr":
|
||||
url = config_manager.get('prowlarr.url', '')
|
||||
api_key = config_manager.get('prowlarr.api_key', '')
|
||||
if not url or not api_key:
|
||||
return False, "Prowlarr URL and API key are required."
|
||||
try:
|
||||
import requests as _req
|
||||
resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
|
||||
headers={'X-Api-Key': api_key}, timeout=10)
|
||||
if resp.ok:
|
||||
version = resp.json().get('version', '?')
|
||||
return True, f"Connected to Prowlarr v{version}"
|
||||
return False, f"Prowlarr returned HTTP {resp.status_code}"
|
||||
except Exception as e:
|
||||
return False, f"Prowlarr connection error: {str(e)}"
|
||||
elif service == "lidarr" or service == "lidarr_download":
|
||||
url = config_manager.get('lidarr_download.url', '')
|
||||
api_key = config_manager.get('lidarr_download.api_key', '')
|
||||
|
|
@ -455,16 +379,6 @@ def run_service_test(service, test_config):
|
|||
return False, "Hydrabase not connected. Configure URL + API key and click Connect."
|
||||
except Exception as e:
|
||||
return False, f"Hydrabase connection error: {str(e)}"
|
||||
elif service == "musicbrainz":
|
||||
try:
|
||||
from core.metadata.registry import get_musicbrainz_client
|
||||
mb = get_musicbrainz_client()
|
||||
results = mb.search_artists("radiohead", limit=1)
|
||||
if results:
|
||||
return True, "MusicBrainz reachable"
|
||||
return False, "MusicBrainz returned no results — may be rate-limited or unreachable."
|
||||
except Exception as e:
|
||||
return False, f"MusicBrainz connection error: {str(e)}"
|
||||
elif service == "soundcloud":
|
||||
# Anonymous SoundCloud has no auth, so "test" really means
|
||||
# "is yt-dlp installed and can it reach SoundCloud right now."
|
||||
|
|
|
|||
|
|
@ -1,85 +0,0 @@
|
|||
"""Named, switchable service-credential sets — pure logic (Phase 0 foundation).
|
||||
|
||||
Today every auth service (Spotify, Tidal, Deezer, Qobuz, Plex, Jellyfin,
|
||||
Navidrome) holds ONE credential set in config, and clients are global singletons
|
||||
built from that single slot. This module is the groundwork for letting an admin
|
||||
save MULTIPLE named credential sets per service ("pills") that each profile can
|
||||
switch between, without anyone but the admin creating them.
|
||||
|
||||
Kept PURE — service registry, payload validation, and active-set selection,
|
||||
free of DB/Flask so it's unit-testable. Encrypted storage lives in MusicDatabase
|
||||
(service_credentials / profile_service_credentials tables); runtime client
|
||||
resolution + UI come in later phases. Nothing here changes existing behaviour;
|
||||
it's dormant capability until wired.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Services that support multiple named credential sets, mapped to the payload
|
||||
# keys that MUST be present for a set to be usable. Extra keys (OAuth tokens,
|
||||
# redirect URIs, quality prefs) are allowed and preserved — these are only the
|
||||
# minimum required to validate a set the admin is saving.
|
||||
SERVICE_CREDENTIAL_SCHEMA = {
|
||||
'spotify': ('client_id', 'client_secret'),
|
||||
'tidal': ('access_token', 'refresh_token'),
|
||||
'deezer': ('arl',),
|
||||
'qobuz': ('user_auth_token',),
|
||||
'plex': ('base_url', 'token'),
|
||||
'jellyfin': ('base_url', 'api_key'),
|
||||
'navidrome': ('base_url', 'username', 'password'),
|
||||
}
|
||||
|
||||
SUPPORTED_SERVICES = frozenset(SERVICE_CREDENTIAL_SCHEMA)
|
||||
|
||||
|
||||
def is_supported_service(service: str) -> bool:
|
||||
"""True when the service supports named credential sets."""
|
||||
return service in SERVICE_CREDENTIAL_SCHEMA
|
||||
|
||||
|
||||
def validate_credential_payload(service: str, payload):
|
||||
"""Return ``(ok, missing_keys)`` for a credential set.
|
||||
|
||||
Valid when every required key for the service is present and truthy. An
|
||||
unknown service is invalid with no missing list (caller should reject it
|
||||
as unsupported, not as "incomplete").
|
||||
"""
|
||||
required = SERVICE_CREDENTIAL_SCHEMA.get(service)
|
||||
if required is None:
|
||||
return False, []
|
||||
if not isinstance(payload, dict):
|
||||
return False, list(required)
|
||||
|
||||
def _present(v):
|
||||
# Whitespace-only strings count as missing — they'd otherwise save a
|
||||
# blank secret that fails confusingly at the real service later.
|
||||
return bool(v.strip()) if isinstance(v, str) else bool(v)
|
||||
|
||||
missing = [k for k in required if not _present(payload.get(k))]
|
||||
return (not missing), missing
|
||||
|
||||
|
||||
def pick_active_credential(credentials, selected_id):
|
||||
"""From ``credentials`` (a list of dicts each carrying ``id``), return the
|
||||
one whose id == ``selected_id``.
|
||||
|
||||
Returns None when there's no selection OR the selected id isn't present —
|
||||
i.e. a stale pointer whose credential set was deleted. The caller then
|
||||
falls back to the global/admin default, so a deleted set never breaks a
|
||||
profile. Pure + stale-safe.
|
||||
"""
|
||||
if not selected_id:
|
||||
return None
|
||||
for cred in credentials or []:
|
||||
if cred.get('id') == selected_id:
|
||||
return cred
|
||||
return None
|
||||
|
||||
|
||||
__all__ = [
|
||||
'SERVICE_CREDENTIAL_SCHEMA',
|
||||
'SUPPORTED_SERVICES',
|
||||
'is_supported_service',
|
||||
'validate_credential_payload',
|
||||
'pick_active_credential',
|
||||
]
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
"""Stall detection for the database-update job.
|
||||
|
||||
The DB updater keeps a single in-memory state dict whose ``status`` is set to
|
||||
``running`` at start and only flipped to ``finished``/``error`` by the worker's
|
||||
completion/error callbacks. If the worker thread hangs — e.g. a media-server API
|
||||
call with no timeout, a DB lock — those callbacks never fire, so ``status`` stays
|
||||
``running`` forever and the UI shows a frozen progress bar with no way to recover
|
||||
(GitHub #859).
|
||||
|
||||
This module is the single, *pure* decision for "is a running job stalled?". It
|
||||
takes the state dict plus the current wall-clock time and a timeout, and answers
|
||||
yes/no — no DB, no globals, no clock of its own. That keeps it unit-testable and
|
||||
lets the watchdog wiring in web_server.py stay a thin call. The job carries a
|
||||
``last_progress_at`` epoch timestamp that the start path and every progress/phase
|
||||
callback bump; staleness is simply "running, and that timestamp is older than the
|
||||
timeout".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Mapping
|
||||
|
||||
# 5 minutes with zero forward progress = presumed hung. A healthy scan ticks
|
||||
# progress (per-artist) far more often than this even for large libraries, so
|
||||
# the timeout won't false-positive a slow-but-working run.
|
||||
DEFAULT_STALL_TIMEOUT_SECONDS = 300
|
||||
|
||||
|
||||
def is_db_update_stalled(
|
||||
state: Mapping[str, Any],
|
||||
now: float,
|
||||
timeout_seconds: float = DEFAULT_STALL_TIMEOUT_SECONDS,
|
||||
) -> bool:
|
||||
"""Return True when the job is ``running`` but has made no progress within
|
||||
``timeout_seconds``.
|
||||
|
||||
Conservative by design — it only ever reports a stall it can prove:
|
||||
- Only a ``running`` job can stall (idle/finished/error never do).
|
||||
- With no usable ``last_progress_at`` timestamp we cannot judge, so we return
|
||||
False rather than risk killing a job we have no clock for.
|
||||
- A non-positive timeout is treated as "disabled" (never stalls).
|
||||
"""
|
||||
if not isinstance(state, Mapping):
|
||||
return False
|
||||
if state.get("status") != "running":
|
||||
return False
|
||||
if timeout_seconds is None or timeout_seconds <= 0:
|
||||
return False
|
||||
last = state.get("last_progress_at")
|
||||
if not last:
|
||||
return False
|
||||
try:
|
||||
elapsed = float(now) - float(last)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
return elapsed >= float(timeout_seconds)
|
||||
|
||||
|
||||
def stalled_error_message(state: Mapping[str, Any], now: float) -> str:
|
||||
"""Build a clear, human-facing message for a stalled job, including how long
|
||||
it has been silent and the phase it died in."""
|
||||
last = state.get("last_progress_at") if isinstance(state, Mapping) else None
|
||||
phase = state.get("phase") if isinstance(state, Mapping) else None
|
||||
try:
|
||||
secs = int(float(now) - float(last)) if last else 0
|
||||
except (TypeError, ValueError):
|
||||
secs = 0
|
||||
msg = "Update appears stuck — no progress"
|
||||
if secs > 0:
|
||||
msg += f" for {secs}s"
|
||||
if phase:
|
||||
msg += f" (last phase: {phase})"
|
||||
msg += (". The worker may be hung on the media server. Start a new update "
|
||||
"to try again, or restart SoulSync if it keeps stalling.")
|
||||
return msg
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_STALL_TIMEOUT_SECONDS",
|
||||
"is_db_update_stalled",
|
||||
"stalled_error_message",
|
||||
]
|
||||
|
|
@ -41,19 +41,7 @@ class DatabaseUpdateWorker:
|
|||
self.database_path = database_path
|
||||
self.full_refresh = full_refresh
|
||||
self.should_stop = False
|
||||
|
||||
# Track ids of rows newly INSERTED this run (not updates). The web
|
||||
# layer reads this to gap-fill embedded provider IDs for the new files
|
||||
# (auto-reconcile), so newly-added music contributes its
|
||||
# Spotify/MusicBrainz/etc. ids without a manual backfill.
|
||||
self._new_track_ids = set()
|
||||
|
||||
# Optional callback(worker) run as the FINAL scan phase, immediately
|
||||
# before the 'finished' signal — so the auto-reconcile is inside the
|
||||
# scan's running window (automations/UI treat it as a normal phase and
|
||||
# wait for it). Injected by the web layer (which owns path resolution).
|
||||
self.post_scan_hook = None
|
||||
|
||||
|
||||
# Statistics tracking
|
||||
self.processed_artists = 0
|
||||
self.processed_albums = 0
|
||||
|
|
@ -91,26 +79,7 @@ class DatabaseUpdateWorker:
|
|||
callback(*args)
|
||||
except Exception as e:
|
||||
logger.error(f"Error in callback for {signal_name}: {e}")
|
||||
|
||||
def _emit_finished(self, *args):
|
||||
"""Run the post-scan hook (auto-reconcile) as the final phase, THEN
|
||||
emit 'finished'.
|
||||
|
||||
Running the hook before 'finished' keeps the scan's status at
|
||||
'running' through the reconcile, so every caller (automations that
|
||||
poll for completion, the dashboard card, the Tools page) treats it as
|
||||
a normal scan phase and waits for it — rather than seeing 'finished'
|
||||
and missing the tail. Best-effort: a hook failure never blocks the
|
||||
completion signal.
|
||||
"""
|
||||
if self.post_scan_hook:
|
||||
try:
|
||||
self.post_scan_hook(self)
|
||||
except Exception as e:
|
||||
logger.warning(f"post-scan hook failed (non-fatal): {e}")
|
||||
self._emit_signal('finished', *args)
|
||||
|
||||
|
||||
|
||||
def connect_callback(self, signal_name: str, callback: Callable):
|
||||
"""Connect a callback for progress notifications."""
|
||||
self.callbacks.setdefault(signal_name, []).append(callback)
|
||||
|
|
@ -177,7 +146,7 @@ class DatabaseUpdateWorker:
|
|||
logger.info(f"Merged {merged} duplicate artists")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not merge duplicate artists: {e}")
|
||||
self._emit_finished(0, 0, 0, 0, 0)
|
||||
self._emit_signal('finished', 0, 0, 0, 0, 0)
|
||||
return
|
||||
logger.info(f"Incremental update: Found {len(artists_to_process)} artists to process")
|
||||
|
||||
|
|
@ -261,7 +230,7 @@ class DatabaseUpdateWorker:
|
|||
self.removed_tracks = removal.get('tracks_removed', 0) if removal else 0
|
||||
|
||||
# Emit final results
|
||||
self._emit_finished(
|
||||
self._emit_signal('finished',
|
||||
self.processed_artists,
|
||||
self.processed_albums,
|
||||
self.processed_tracks,
|
||||
|
|
@ -362,7 +331,7 @@ class DatabaseUpdateWorker:
|
|||
f"{self.processed_albums} albums, {self.processed_tracks} new tracks, "
|
||||
f"{stale_removed} stale tracks removed")
|
||||
|
||||
self._emit_finished(
|
||||
self._emit_signal('finished',
|
||||
self.processed_artists,
|
||||
self.processed_albums,
|
||||
self.processed_tracks,
|
||||
|
|
@ -911,8 +880,6 @@ class DatabaseUpdateWorker:
|
|||
track_success = self.database.insert_or_update_media_track(track, album_id, artist_id, server_source=self.server_type)
|
||||
if track_success:
|
||||
total_processed_tracks += 1
|
||||
if track_success == 'inserted':
|
||||
self._new_track_ids.add(str(track.ratingKey))
|
||||
logger.debug(f"Processed new track: {track.title}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
|
||||
|
|
@ -1377,8 +1344,6 @@ class DatabaseUpdateWorker:
|
|||
skipped_count += 1
|
||||
elif track_success:
|
||||
track_count += 1
|
||||
if track_success == 'inserted':
|
||||
self._new_track_ids.add(track_id_str)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
|
||||
|
||||
|
|
|
|||
|
|
@ -1,153 +0,0 @@
|
|||
"""SQLite integrity + safe-backup helpers.
|
||||
|
||||
Born out of a real incident: a WAL-mode DB got corrupted (most likely an
|
||||
interrupted write during a hard restart), and because the backup routine
|
||||
(a) never checked integrity and (b) rotated the oldest backup out by mtime,
|
||||
every rolling backup ended up being a faithful copy of the already-corrupt
|
||||
file — so when recovery was needed, all snapshots were poisoned.
|
||||
|
||||
This module makes that impossible:
|
||||
|
||||
* ``quick_check(path)`` / ``is_healthy(path)`` — fast read-only integrity probe.
|
||||
* ``safe_backup(...)`` — verifies the SOURCE is healthy before copying, uses the
|
||||
SQLite Online Backup API, then verifies the RESULT. A corrupt source never
|
||||
produces (or keeps) a backup.
|
||||
* ``prune_backups(...)`` — rotation that NEVER deletes the most recent
|
||||
*verified-healthy* backup, even to honor the max-count, so a run of bad
|
||||
backups can't evict your last good one.
|
||||
|
||||
Pure-ish: only touches sqlite3 + the filesystem paths it's given; no Flask, no
|
||||
app globals. Unit-testable with real (and deliberately-corrupted) temp DBs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("db_integrity")
|
||||
|
||||
|
||||
def _close_quietly(conn) -> None:
|
||||
"""Best-effort close; a failure to close during cleanup must not mask the
|
||||
real error we're handling, but we log it rather than swallow silently."""
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
conn.close()
|
||||
except Exception as e: # noqa: BLE001 — cleanup path, real error already in flight
|
||||
logger.debug("db_integrity: connection close failed: %s", e)
|
||||
|
||||
|
||||
class DBIntegrityError(Exception):
|
||||
"""Raised when a database fails its integrity check."""
|
||||
|
||||
|
||||
def quick_check(db_path: str, *, timeout: float = 30.0) -> str:
|
||||
"""Run ``PRAGMA quick_check`` read-only and return its first result row.
|
||||
|
||||
Returns ``'ok'`` for a healthy DB, otherwise the first error line. Raises
|
||||
``DBIntegrityError`` if the file can't even be opened/read (malformed
|
||||
header, I/O error) — i.e. unambiguously bad.
|
||||
"""
|
||||
if not os.path.exists(db_path):
|
||||
raise DBIntegrityError(f"Database file not found: {db_path}")
|
||||
conn = None
|
||||
try:
|
||||
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=timeout)
|
||||
row = conn.execute("PRAGMA quick_check(1)").fetchone()
|
||||
return (row[0] if row else "no result")
|
||||
except sqlite3.DatabaseError as e:
|
||||
# malformed header / disk image malformed / disk I/O error
|
||||
raise DBIntegrityError(f"{db_path}: {e}") from e
|
||||
finally:
|
||||
_close_quietly(conn)
|
||||
|
||||
|
||||
def is_healthy(db_path: str, *, timeout: float = 30.0) -> bool:
|
||||
"""True iff the DB opens and ``quick_check`` reports 'ok'. Never raises."""
|
||||
try:
|
||||
return quick_check(db_path, timeout=timeout) == "ok"
|
||||
except DBIntegrityError:
|
||||
return False
|
||||
|
||||
|
||||
def safe_backup(src_path: str, dst_path: str, *, verify_source: bool = True,
|
||||
verify_result: bool = True) -> None:
|
||||
"""Back up ``src_path`` to ``dst_path`` via the SQLite Online Backup API,
|
||||
refusing to produce a backup from (or keep a backup of) a corrupt DB.
|
||||
|
||||
Raises ``DBIntegrityError`` and removes any partial ``dst_path`` when the
|
||||
source is unhealthy (``verify_source``) or the produced backup fails its
|
||||
own check (``verify_result``). On success ``dst_path`` is a verified-good
|
||||
copy.
|
||||
"""
|
||||
if verify_source and not is_healthy(src_path):
|
||||
# Don't immortalize corruption — surface it so the caller can alert
|
||||
# and, crucially, NOT rotate out the existing good backups.
|
||||
raise DBIntegrityError(
|
||||
f"Refusing to back up: source database failed integrity check ({src_path})"
|
||||
)
|
||||
|
||||
src = dst = None
|
||||
try:
|
||||
src = sqlite3.connect(src_path)
|
||||
dst = sqlite3.connect(dst_path)
|
||||
src.backup(dst)
|
||||
finally:
|
||||
_close_quietly(dst)
|
||||
_close_quietly(src)
|
||||
|
||||
if verify_result and not is_healthy(dst_path):
|
||||
# The copy itself came out bad — discard it rather than keep a dud.
|
||||
try:
|
||||
os.remove(dst_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise DBIntegrityError(
|
||||
f"Backup produced a corrupt file and was discarded ({dst_path})"
|
||||
)
|
||||
|
||||
|
||||
def prune_backups(backup_paths, max_keep: int,
|
||||
health_check=is_healthy) -> list:
|
||||
"""Decide which backups to delete to honor ``max_keep`` WITHOUT ever
|
||||
deleting the most-recent verified-healthy backup.
|
||||
|
||||
``backup_paths`` is an iterable of paths; order does not matter (we sort by
|
||||
mtime). Returns the list of paths that SHOULD be deleted (does not delete
|
||||
them — the caller does the IO, so this stays pure/testable).
|
||||
|
||||
Rule: oldest-first deletion until <= max_keep, but the single newest
|
||||
*healthy* backup is protected and never selected for deletion. So even if
|
||||
the newest few backups are corrupt, the last good snapshot survives.
|
||||
"""
|
||||
paths = [p for p in backup_paths]
|
||||
# Newest first.
|
||||
paths.sort(key=lambda p: _safe_mtime(p), reverse=True)
|
||||
|
||||
# Find the newest healthy backup — the one we must never drop.
|
||||
protected: Optional[str] = None
|
||||
for p in paths:
|
||||
if health_check(p):
|
||||
protected = p
|
||||
break
|
||||
|
||||
if len(paths) <= max_keep:
|
||||
return []
|
||||
|
||||
# Delete oldest-first beyond max_keep, but skip the protected one.
|
||||
deletable = [p for p in paths if p != protected]
|
||||
# oldest first among deletable
|
||||
deletable.sort(key=lambda p: _safe_mtime(p))
|
||||
num_to_delete = len(paths) - max_keep
|
||||
return deletable[:num_to_delete]
|
||||
|
||||
|
||||
def _safe_mtime(path: str) -> float:
|
||||
try:
|
||||
return os.path.getmtime(path)
|
||||
except OSError:
|
||||
return 0.0
|
||||
|
|
@ -14,12 +14,7 @@ from pathlib import Path
|
|||
from flask import jsonify, request
|
||||
|
||||
from config.settings import config_manager
|
||||
from core.metadata.registry import (
|
||||
get_spotify_client,
|
||||
get_primary_source,
|
||||
is_hydrabase_enabled,
|
||||
)
|
||||
from core.metadata.status import get_spotify_status
|
||||
from core.metadata.registry import get_spotify_client
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -188,47 +183,21 @@ def get_debug_info():
|
|||
info['paths']['music_videos_path'] = music_videos_path
|
||||
info['paths']['music_videos_path_exists'] = os.path.isdir(music_videos_path)
|
||||
|
||||
# Services. `_status_cache` only carries 'media_server' and 'soulseek'
|
||||
# (no 'spotify' key) so anything we used to read from `spotify_cache`
|
||||
# silently defaulted to the missing-value fallback — that's the
|
||||
# "music_source: unknown" bug. Spotify status now comes from the
|
||||
# canonical `get_spotify_status` accessor; primary metadata source
|
||||
# comes from `get_primary_source` (which already accounts for the
|
||||
# auth-fallback chain — Spotify drops back to Deezer when not
|
||||
# authenticated).
|
||||
# Services from status cache
|
||||
spotify_cache = _status_cache.get('spotify', {})
|
||||
media_server_cache = _status_cache.get('media_server', {})
|
||||
soulseek_cache = _status_cache.get('soulseek', {})
|
||||
spotify_status = _safe_check(lambda: get_spotify_status(spotify_client=spotify_client), default={})
|
||||
if not isinstance(spotify_status, dict):
|
||||
spotify_status = {}
|
||||
info['services'] = {
|
||||
'music_source': _safe_check(get_primary_source, default='unknown') or 'unknown',
|
||||
'spotify_connected': bool(spotify_status.get('connected', False)),
|
||||
'spotify_rate_limited': bool(spotify_status.get('rate_limited', False)),
|
||||
'music_source': spotify_cache.get('source', 'unknown'),
|
||||
'spotify_connected': spotify_cache.get('connected', False),
|
||||
'spotify_rate_limited': spotify_cache.get('rate_limited', False),
|
||||
'media_server_type': media_server_cache.get('type', 'none'),
|
||||
'media_server_connected': media_server_cache.get('connected', False),
|
||||
'soulseek_connected': soulseek_cache.get('connected', False),
|
||||
'download_source': config_manager.get('download_source.mode', 'hybrid'),
|
||||
'tidal_connected': _safe_check(lambda: bool(tidal_client and tidal_client.is_authenticated())),
|
||||
'qobuz_connected': _safe_check(lambda: bool(qobuz_enrichment_worker and qobuz_enrichment_worker.client and qobuz_enrichment_worker.client.is_authenticated())),
|
||||
'hydrabase_connected': _safe_check(is_hydrabase_enabled),
|
||||
# YouTube is URL-based via yt-dlp — no auth, always reachable as
|
||||
# long as the binary is installed. Surfaced so the debug dump
|
||||
# documents that YouTube is one of the available download sources
|
||||
# rather than implying it doesn't exist.
|
||||
'youtube_available': True,
|
||||
}
|
||||
# HiFi instance count — separate from connection status because each
|
||||
# instance is its own independent endpoint with its own auth state.
|
||||
info['services']['hifi_instance_count'] = _safe_check(
|
||||
lambda: len(get_database().get_hifi_instances()), default=0
|
||||
)
|
||||
# Always-available public metadata sources (no auth, no per-user
|
||||
# connection state). Listed so the debug dump reflects the full
|
||||
# metadata surface SoulSync queries from, not just the auth-gated ones.
|
||||
info['services']['always_available_metadata_sources'] = [
|
||||
'deezer', 'itunes', 'musicbrainz',
|
||||
]
|
||||
|
||||
# Enrichment workers
|
||||
workers = {}
|
||||
|
|
@ -331,11 +300,7 @@ def get_debug_info():
|
|||
# API rate monitor — current calls/min, 24h totals, peaks, rate limit events
|
||||
try:
|
||||
from core.api_call_tracker import api_call_tracker
|
||||
# `get_spotify_status` is already imported at module level. A
|
||||
# local re-import here would make Python treat the name as a
|
||||
# function-scoped local for the WHOLE body, breaking the lambda
|
||||
# at the top of get_debug_info that closes over the module-level
|
||||
# binding (Python 3.12 NameError on free variables).
|
||||
from core.metadata.status import get_spotify_status
|
||||
rates = api_call_tracker.get_all_rates()
|
||||
info['api_rates'] = rates
|
||||
# Rich 24h debug summary with peaks, totals, per-endpoint breakdown, events
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@ from typing import Dict, List, Optional, Any
|
|||
from functools import wraps
|
||||
from dataclasses import dataclass
|
||||
from utils.logging_config import get_logger
|
||||
from core.metadata.artist_album_cache import get_cached_artist_album_items, store_artist_album_items
|
||||
from core.metadata.cache import get_metadata_cache
|
||||
|
||||
logger = get_logger("deezer_client")
|
||||
|
|
@ -46,119 +45,6 @@ def rate_limited(func):
|
|||
return wrapper
|
||||
|
||||
|
||||
# Pattern matches Deezer's CDN cover/picture URL: a numeric width-x-height
|
||||
# segment in the path (e.g. ``/1000x1000-000000-80-0-0.jpg``). Captures
|
||||
# both halves so the replacement can use a single dimension and preserve
|
||||
# the rest of the path verbatim.
|
||||
_DEEZER_CDN_SIZE_PATTERN = re.compile(r'/(\d+)x(\d+)-')
|
||||
|
||||
# Maximum size Deezer's CDN serves before returning 403. Verified
|
||||
# empirically against multiple albums — 1900 works reliably, 2000+
|
||||
# returns Forbidden. CDN serves the source-native size when it's
|
||||
# smaller than requested, so asking for 1900 is safe even on albums
|
||||
# whose source upload was lower-res (no upscaling, just same bytes).
|
||||
_DEEZER_MAX_COVER_SIZE = 1900
|
||||
|
||||
|
||||
def _upgrade_deezer_cover_url(url: str, target_size: int = _DEEZER_MAX_COVER_SIZE) -> str:
|
||||
"""Rewrite a Deezer CDN cover/picture URL to request a larger size.
|
||||
|
||||
Deezer's API returns ``cover_xl`` / ``picture_xl`` URLs at
|
||||
1000×1000, but the underlying CDN serves up to 1900×1900 by
|
||||
rewriting the size segment in the URL path. This helper does the
|
||||
rewrite — same idea as ``_upgrade_spotify_image_url`` in
|
||||
``spotify_client`` and the ``mzstatic.com`` size-replacement in
|
||||
``download_cover_art``.
|
||||
|
||||
Defensive on every input shape:
|
||||
- Empty / None URL → returned as-is
|
||||
- Non-Deezer URL (no ``dzcdn`` host, no size segment) → returned as-is
|
||||
- Already at or above target size → returned as-is (no point rewriting)
|
||||
|
||||
The CDN returns the source-native image bytes when source < target,
|
||||
so asking for 1900 on an album whose source was uploaded at 600
|
||||
just returns the 600-pixel image — no upscaling, no failure.
|
||||
"""
|
||||
if not url or 'dzcdn' not in url:
|
||||
return url
|
||||
match = _DEEZER_CDN_SIZE_PATTERN.search(url)
|
||||
if not match:
|
||||
return url
|
||||
current = int(match.group(1))
|
||||
if current >= target_size:
|
||||
return url
|
||||
return _DEEZER_CDN_SIZE_PATTERN.sub(f'/{target_size}x{target_size}-', url, count=1)
|
||||
|
||||
|
||||
def _is_full_track_payload(payload: Optional[Dict[str, Any]]) -> bool:
|
||||
"""Distinguish a full `/track/<id>` cache hit from partial album-tracks data.
|
||||
|
||||
Three Deezer endpoints feed the per-track cache:
|
||||
- `/track/<id>` — full record, includes both `track_position` AND
|
||||
`contributors` (the multi-artist list the contributors-upgrade
|
||||
path reads).
|
||||
- `/album/<id>/tracks` — partial; includes `track_position` but
|
||||
omits `contributors`.
|
||||
- `/search/track` — minimal; lacks `track_position`.
|
||||
|
||||
Pre-fix `get_track_details` only checked `track_position`, so
|
||||
partial album-tracks payloads were treated as full hits and the
|
||||
contributors-upgrade silently fell back to single-artist tagging
|
||||
whenever an album had been fetched before its individual tracks
|
||||
were post-processed (issue #588).
|
||||
|
||||
`contributors` key presence is the load-bearing distinction —
|
||||
`[]` is a valid value for genuinely single-artist tracks fetched
|
||||
via the per-track endpoint, so test for key membership not
|
||||
truthiness.
|
||||
"""
|
||||
if not isinstance(payload, dict):
|
||||
return False
|
||||
return 'track_position' in payload and 'contributors' in payload
|
||||
|
||||
|
||||
def resolve_album_track_positions(session, base_url, album_ids, cache=None, sleep_s=0.2):
|
||||
"""Build ``{str(track_id): track_position}`` for a set of Deezer album ids.
|
||||
|
||||
Deezer PLAYLIST and SEARCH track objects (and even the album object's embedded
|
||||
``tracks.data``) omit ``track_position`` — only ``/album/<id>/tracks`` and
|
||||
``/track/<id>`` carry it. So numbering playlist tracks by their playlist index
|
||||
silently poisons the real album track number, which then rides onto the
|
||||
downloaded file's tag. This resolves the authoritative position per album
|
||||
(cache-first, best-effort — a failed album just isn't in the map)."""
|
||||
import time as _time
|
||||
positions: Dict[str, int] = {}
|
||||
for aid in album_ids:
|
||||
aid = str(aid)
|
||||
at_list = None
|
||||
if cache:
|
||||
try:
|
||||
ct = cache.get_entity('deezer', 'album_tracks', aid)
|
||||
if ct and ct.get('data'):
|
||||
at_list = ct['data']
|
||||
except Exception: # noqa: BLE001 - cache is best-effort
|
||||
at_list = None
|
||||
if at_list is None:
|
||||
try:
|
||||
if sleep_s:
|
||||
_time.sleep(sleep_s) # respect Deezer rate limits
|
||||
r = session.get(f"{base_url}/album/{aid}/tracks", params={'limit': 500}, timeout=10)
|
||||
if getattr(r, 'ok', False):
|
||||
at_list = (r.json() or {}).get('data', [])
|
||||
if cache and at_list is not None:
|
||||
try:
|
||||
cache.store_entity('deezer', 'album_tracks', aid, {'data': at_list})
|
||||
except Exception as _cache_err: # noqa: BLE001
|
||||
logger.debug("album_tracks cache store failed for %s: %s", aid, _cache_err)
|
||||
except Exception: # noqa: BLE001 - never let metadata resolution break the fetch
|
||||
at_list = None
|
||||
for at in (at_list or []):
|
||||
tp = at.get('track_position')
|
||||
if at.get('id') and tp:
|
||||
positions[str(at['id'])] = tp
|
||||
return positions
|
||||
|
||||
|
||||
# ==================== Dataclasses (match iTunesClient / SpotifyClient format) ====================
|
||||
|
||||
@dataclass
|
||||
|
|
@ -282,7 +168,6 @@ class Album:
|
|||
album_type: str
|
||||
image_url: Optional[str] = None
|
||||
external_urls: Optional[Dict[str, str]] = None
|
||||
explicit: Optional[bool] = None
|
||||
|
||||
@classmethod
|
||||
def from_deezer_album(cls, album_data: Dict[str, Any]) -> 'Album':
|
||||
|
|
@ -314,8 +199,7 @@ class Album:
|
|||
total_tracks=album_data.get('nb_tracks', 0),
|
||||
album_type=album_type,
|
||||
image_url=image_url,
|
||||
external_urls=external_urls if external_urls else None,
|
||||
explicit=bool(album_data.get('explicit_lyrics', False)),
|
||||
external_urls=external_urls if external_urls else None
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -414,78 +298,10 @@ class DeezerClient:
|
|||
# can serve as a drop-in fallback metadata source in SpotifyClient.
|
||||
|
||||
@rate_limited
|
||||
def search_tracks(
|
||||
self,
|
||||
query: str = '',
|
||||
limit: int = 20,
|
||||
*,
|
||||
track: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
album: Optional[str] = None,
|
||||
) -> List[Track]:
|
||||
"""Search for tracks — returns Track dataclass list (metadata source interface).
|
||||
|
||||
Two call modes:
|
||||
|
||||
1. **Free-text** (`query='Foreigner Dirty White Boy'`) — legacy
|
||||
shape, passes the string straight to Deezer's `q` param.
|
||||
Same behaviour as before, kept for backward compat.
|
||||
|
||||
2. **Field-scoped** (`track='Dirty White Boy', artist='Foreigner'`) —
|
||||
builds Deezer's advanced search syntax (`track:"X" artist:"Y"`).
|
||||
Massively tighter relevance than the free-text path because
|
||||
the API matches each term in the right field instead of
|
||||
anywhere across title / lyrics / artist / album / contributors.
|
||||
Without this, the Deezer ranking buries the canonical track
|
||||
under karaoke / cover / "originally performed by" variants
|
||||
— see issue #534.
|
||||
|
||||
Field-scoped form is used whenever ``track`` or ``artist`` is
|
||||
provided. ``query`` is ignored in that case (the field params
|
||||
are authoritative). When both are missing, falls through to
|
||||
``query``. The cache key is the constructed query string in
|
||||
either case so the two paths share entries naturally.
|
||||
"""
|
||||
# Build the actual API query — advanced syntax when callers pass
|
||||
# field hints, raw query otherwise.
|
||||
used_advanced = bool(track or artist or album)
|
||||
if used_advanced:
|
||||
api_query = self._build_advanced_query(track=track, artist=artist, album=album)
|
||||
else:
|
||||
api_query = query
|
||||
|
||||
if not api_query:
|
||||
return []
|
||||
|
||||
tracks = self._search_tracks_with_query(api_query, limit)
|
||||
|
||||
# Safety net: Deezer's advanced syntax is `artist:"X"`-style
|
||||
# substring match, but in practice it's brittle on artist name
|
||||
# variants ("Foreigner [US]", "The Foreigner", etc.) and on
|
||||
# tracks indexed under non-canonical title spellings. When the
|
||||
# advanced query returns nothing, fall back to a free-text join
|
||||
# so the user sees the prior (less-relevant but non-empty) result
|
||||
# set rather than "No matches" — same behaviour as pre-fix for
|
||||
# this edge case. Caller-side rerank still tightens the result.
|
||||
if not tracks and used_advanced:
|
||||
fallback_parts = [p for p in (track, artist, album) if p]
|
||||
fallback_query = ' '.join(fallback_parts)
|
||||
if fallback_query and fallback_query != api_query:
|
||||
logger.debug(
|
||||
"[Deezer] Advanced query returned 0 results, falling back "
|
||||
"to free-text: %r → %r", api_query, fallback_query,
|
||||
)
|
||||
tracks = self._search_tracks_with_query(fallback_query, limit)
|
||||
|
||||
return tracks
|
||||
|
||||
def _search_tracks_with_query(self, api_query: str, limit: int) -> List[Track]:
|
||||
"""Cache-aware single API call. Pulled out so the
|
||||
``search_tracks`` orchestration can call this twice (advanced
|
||||
query → free-text fallback) without duplicating the cache +
|
||||
parse + store dance."""
|
||||
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
|
||||
"""Search for tracks — returns Track dataclass list (metadata source interface)"""
|
||||
cache = get_metadata_cache()
|
||||
cached_results = cache.get_search_results('deezer', 'track', api_query, limit)
|
||||
cached_results = cache.get_search_results('deezer', 'track', query, limit)
|
||||
if cached_results is not None:
|
||||
tracks = []
|
||||
for raw in cached_results:
|
||||
|
|
@ -496,54 +312,25 @@ class DeezerClient:
|
|||
if tracks:
|
||||
return tracks
|
||||
|
||||
data = self._api_get('search/track', {'q': api_query, 'limit': min(limit, 100)})
|
||||
data = self._api_get('search/track', {'q': query, 'limit': min(limit, 100)})
|
||||
if not data or 'data' not in data:
|
||||
return []
|
||||
|
||||
tracks = []
|
||||
raw_items = []
|
||||
for track_data in data['data']:
|
||||
track_obj = Track.from_deezer_track(track_data)
|
||||
tracks.append(track_obj)
|
||||
track = Track.from_deezer_track(track_data)
|
||||
tracks.append(track)
|
||||
raw_items.append(track_data)
|
||||
|
||||
entries = [(str(td.get('id', '')), td) for td in raw_items if td.get('id')]
|
||||
if entries:
|
||||
cache.store_entities_bulk('deezer', 'track', entries)
|
||||
cache.store_search_results('deezer', 'track', api_query, limit,
|
||||
cache.store_search_results('deezer', 'track', query, limit,
|
||||
[str(td.get('id', '')) for td in raw_items if td.get('id')])
|
||||
|
||||
return tracks
|
||||
|
||||
@staticmethod
|
||||
def _build_advanced_query(
|
||||
*,
|
||||
track: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
album: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Compose Deezer's advanced search syntax from field hints.
|
||||
|
||||
Per Deezer's docs:
|
||||
https://developers.deezer.com/api/search
|
||||
|
||||
q=track:"X" artist:"Y" album:"Z"
|
||||
|
||||
Quotes around each value preserve multi-word phrases. Empty
|
||||
fields are skipped. Embedded double-quotes get stripped (no
|
||||
escape mechanism in Deezer's syntax) — rare in practice, but
|
||||
a search for `O"Hara` would otherwise produce a malformed
|
||||
query.
|
||||
"""
|
||||
parts = []
|
||||
if track:
|
||||
parts.append(f'track:"{track.replace(chr(34), "")}"')
|
||||
if artist:
|
||||
parts.append(f'artist:"{artist.replace(chr(34), "")}"')
|
||||
if album:
|
||||
parts.append(f'album:"{album.replace(chr(34), "")}"')
|
||||
return ' '.join(parts)
|
||||
|
||||
@rate_limited
|
||||
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
|
||||
"""Search for artists — returns Artist dataclass list (metadata source interface)"""
|
||||
|
|
@ -616,9 +403,14 @@ class DeezerClient:
|
|||
"""Get detailed track info — returns Spotify-compatible dict (metadata source interface)"""
|
||||
cache = get_metadata_cache()
|
||||
cached = cache.get_entity('deezer', 'track', str(track_id))
|
||||
if cached and cached.get('title') and _is_full_track_payload(cached):
|
||||
return self._build_enhanced_track(cached)
|
||||
# Otherwise fall through to fetch full data from API
|
||||
if cached and cached.get('title'):
|
||||
# Search results are cached with minimal data (no track_position).
|
||||
# Only use cache if it has track_position — the key field from /track/{id}.
|
||||
# Search results include 'isrc' and 'release_date' but NOT track_position,
|
||||
# so those fields alone are not sufficient to distinguish full from partial data.
|
||||
if 'track_position' in cached:
|
||||
return self._build_enhanced_track(cached)
|
||||
# Otherwise fall through to fetch full data from API
|
||||
|
||||
data = self._api_get(f'track/{track_id}')
|
||||
if not data:
|
||||
|
|
@ -916,41 +708,17 @@ class DeezerClient:
|
|||
|
||||
Matches iTunesClient.get_artist_albums() interface.
|
||||
Paginates through all results up to the requested limit."""
|
||||
cache = get_metadata_cache()
|
||||
cached_items = get_cached_artist_album_items(cache, 'deezer', artist_id, album_type=album_type, limit=limit)
|
||||
if cached_items:
|
||||
try:
|
||||
requested_types = [t.strip() for t in album_type.split(',')]
|
||||
cached_albums = []
|
||||
for album_data in cached_items:
|
||||
album = Album.from_deezer_album(album_data)
|
||||
if album_type != 'album,single':
|
||||
if album.album_type not in requested_types:
|
||||
if not (album.album_type == 'ep' and 'single' in requested_types):
|
||||
continue
|
||||
cached_albums.append(album)
|
||||
return cached_albums[:limit]
|
||||
except Exception as e:
|
||||
logger.debug("Deezer artist albums cache reuse failed: %s", e)
|
||||
|
||||
albums = []
|
||||
all_raw = []
|
||||
requested_types = [t.strip() for t in album_type.split(',')]
|
||||
offset = 0
|
||||
page_size = 100 # Deezer API max per request
|
||||
complete = True # cleared if pagination breaks on a transient/malformed error
|
||||
|
||||
while offset < limit:
|
||||
fetch_limit = min(page_size, limit - offset)
|
||||
data = self._api_get(f'artist/{artist_id}/albums', {'limit': fetch_limit, 'index': offset})
|
||||
if not data or 'data' not in data:
|
||||
# Malformed/transient response mid-pagination — what we have is a
|
||||
# PARTIAL discography. Don't cache it as the full list (mirrors the
|
||||
# Spotify truncated-fetch guard). #853 follow-up.
|
||||
complete = False
|
||||
if not data or 'data' not in data or len(data['data']) == 0:
|
||||
break
|
||||
if len(data['data']) == 0:
|
||||
break # No more albums — a clean end of pagination.
|
||||
|
||||
for album_data in data['data']:
|
||||
all_raw.append(album_data)
|
||||
|
|
@ -967,6 +735,7 @@ class DeezerClient:
|
|||
break # Last page
|
||||
offset += len(data['data'])
|
||||
|
||||
cache = get_metadata_cache()
|
||||
# Deezer's /artist/{id}/albums endpoint doesn't include artist info on each album.
|
||||
# Inject it so cached album entities have artist_name for discover page display.
|
||||
artist_stub = None
|
||||
|
|
@ -980,11 +749,6 @@ class DeezerClient:
|
|||
entries.append((str(ad['id']), ad))
|
||||
if entries:
|
||||
cache.store_entities_bulk('deezer', 'album', entries, skip_if_exists=True)
|
||||
# Only cache the artist→album-LIST when pagination finished cleanly; a
|
||||
# partial list would otherwise serve an incomplete discography until TTL.
|
||||
# (Individual album entities above are complete, so they cache regardless.)
|
||||
if complete:
|
||||
store_artist_album_items(cache, 'deezer', artist_id, all_raw, album_type=album_type, limit=limit)
|
||||
|
||||
logger.info(f"Retrieved {len(albums)} albums for artist {artist_id}")
|
||||
return albums[:limit]
|
||||
|
|
@ -1399,16 +1163,6 @@ class DeezerClient:
|
|||
|
||||
raw_tracks.extend(page_tracks)
|
||||
|
||||
# Real album track positions — playlist tracks don't carry track_position,
|
||||
# so numbering by playlist index would poison the downloaded file's tag.
|
||||
album_ids = {str(t.get('album', {}).get('id')) for t in raw_tracks if t.get('album', {}).get('id')}
|
||||
try:
|
||||
from core.metadata.cache import get_metadata_cache
|
||||
_cache = get_metadata_cache()
|
||||
except Exception:
|
||||
_cache = None
|
||||
track_positions = resolve_album_track_positions(self.session, self.BASE_URL, album_ids, _cache)
|
||||
|
||||
# Normalize tracks
|
||||
tracks: List[Dict[str, Any]] = []
|
||||
for i, t in enumerate(raw_tracks, start=1):
|
||||
|
|
@ -1420,8 +1174,7 @@ class DeezerClient:
|
|||
'artists': [artist_name],
|
||||
'album': t.get('album', {}).get('title', ''),
|
||||
'duration_ms': t.get('duration', 0) * 1000,
|
||||
# REAL album position; the playlist index is a last resort only.
|
||||
'track_number': track_positions.get(str(t.get('id'))) or i,
|
||||
'track_number': i,
|
||||
})
|
||||
|
||||
result = {
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ from typing import Any, Dict, List, Optional, Tuple
|
|||
import requests
|
||||
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.quality.source_map import quality_from_deezer, quality_tier_for_source
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("deezer_download")
|
||||
|
|
@ -93,10 +92,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
if download_path is None:
|
||||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Engine reference is populated by set_engine() at registration
|
||||
# time. None until orchestrator wires the registry.
|
||||
|
|
@ -121,20 +117,14 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
self._license_token = None
|
||||
self._user_data = None
|
||||
self._authenticated = False
|
||||
self._pending_arl: Optional[str] = None
|
||||
|
||||
# Quality preference
|
||||
self._quality = quality_tier_for_source('deezer', default='flac')
|
||||
self._quality = config_manager.get('deezer_download.quality', 'flac')
|
||||
|
||||
# Try to authenticate on init if ARL is configured
|
||||
arl = config_manager.get('deezer_download.arl', '')
|
||||
if arl:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if is_boot_phase():
|
||||
self._pending_arl = arl
|
||||
logger.debug("Deezer ARL present — authentication deferred until after boot")
|
||||
else:
|
||||
self._authenticate(arl)
|
||||
self._authenticate(arl)
|
||||
|
||||
logger.info(f"Deezer download client initialized (download path: {self.download_path})")
|
||||
|
||||
|
|
@ -233,66 +223,12 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
return self._authenticated
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
if self._pending_arl and not self._authenticated:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if not is_boot_phase():
|
||||
self._authenticate(self._pending_arl)
|
||||
self._pending_arl = None
|
||||
return self._authenticated
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.is_available)
|
||||
|
||||
# ─── Playlist export (#945) ──────────────────────────────────
|
||||
#
|
||||
# UNOFFICIAL: rides the private gw-light gateway with the ARL session already used
|
||||
# for downloads. Deezer shut their public developer API, so this is the only write
|
||||
# path — and it's fragile by nature (breaks when Deezer changes internals).
|
||||
|
||||
def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
|
||||
public=False, description=""):
|
||||
"""Create a Deezer playlist (or append to an existing one) from a mirrored
|
||||
playlist's tracks. ``track_ids`` are stored ``deezer_id`` values per library track.
|
||||
``existing_id`` set → add to that playlist (idempotent re-export reuses the stored
|
||||
target); unset → create a new one. Returns
|
||||
``{success, playlist_id, url, added, error}``."""
|
||||
if not self._authenticated:
|
||||
return {"success": False, "error": "Deezer is not connected (ARL)"}
|
||||
song_ids = [str(t) for t in (track_ids or []) if t]
|
||||
if not song_ids:
|
||||
return {"success": False, "error": "No matching Deezer tracks to export"}
|
||||
try:
|
||||
songs = [[sid, i] for i, sid in enumerate(song_ids)]
|
||||
if existing_id:
|
||||
res = self._gw_call("playlist.addSongs",
|
||||
{"playlist_id": int(existing_id), "songs": songs})
|
||||
if res is None:
|
||||
return {"success": False, "error": "Deezer rejected the playlist update"}
|
||||
playlist_id = existing_id
|
||||
else:
|
||||
res = self._gw_call("playlist.create", {
|
||||
"title": name, "description": description,
|
||||
"is_public": bool(public), "songs": songs,
|
||||
})
|
||||
if res is None:
|
||||
return {"success": False, "error": "Deezer rejected the playlist create"}
|
||||
# gw 'playlist.create' returns the new playlist id (int) as `results`.
|
||||
if isinstance(res, dict):
|
||||
playlist_id = res.get("PLAYLIST_ID") or res.get("id")
|
||||
else:
|
||||
playlist_id = res
|
||||
if not playlist_id:
|
||||
return {"success": False, "error": "Deezer did not return a playlist id"}
|
||||
return {
|
||||
"success": True,
|
||||
"playlist_id": str(playlist_id),
|
||||
"url": f"https://www.deezer.com/playlist/{playlist_id}",
|
||||
"added": len(song_ids),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e)}
|
||||
|
||||
def reconnect(self, arl: str = None) -> bool:
|
||||
"""Re-authenticate with a new or existing ARL."""
|
||||
if arl is None:
|
||||
|
|
@ -481,12 +417,6 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
if aid:
|
||||
album_ids.add(str(aid))
|
||||
album_release_dates = {}
|
||||
# Deezer PLAYLIST tracks do NOT carry `track_position` (only `/track/<id>`
|
||||
# and `/album/<id>/tracks` do), so numbering them by their playlist index
|
||||
# poisons the real album track number — which then rides into the wishlist
|
||||
# and onto the downloaded file's tag (e.g. 'Apologize' tagged track 1 instead
|
||||
# of 16). Resolve the REAL position from each album's track list (cache-first).
|
||||
track_positions: Dict[str, int] = {} # str(track_id) -> album track_position
|
||||
try:
|
||||
from core.metadata.cache import get_metadata_cache
|
||||
cache = get_metadata_cache()
|
||||
|
|
@ -499,32 +429,24 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
cached = cache.get_entity('deezer', 'album', aid)
|
||||
if cached and cached.get('release_date'):
|
||||
album_release_dates[aid] = cached['release_date']
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.debug("cache get_entity album release_date: %s", e)
|
||||
# Cache miss — fetch from API
|
||||
if aid not in album_release_dates:
|
||||
try:
|
||||
time.sleep(0.3) # Respect rate limits
|
||||
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
|
||||
if a_resp.ok:
|
||||
a_data = a_resp.json()
|
||||
album_release_dates[aid] = a_data.get('release_date', '')
|
||||
# Store in metadata cache for future use
|
||||
if cache:
|
||||
try:
|
||||
cache.store_entity('deezer', 'album', aid, a_data)
|
||||
except Exception as e:
|
||||
logger.debug("cache store_entity album release_date: %s", e)
|
||||
except Exception as e:
|
||||
logger.debug("fetch deezer album release_date %s: %s", aid, e)
|
||||
# Real album track positions (separate endpoint — playlist tracks AND the
|
||||
# album object's embedded tracks both omit track_position). Cache-first.
|
||||
try:
|
||||
from core.deezer_client import resolve_album_track_positions
|
||||
track_positions = resolve_album_track_positions(
|
||||
self._session, 'https://api.deezer.com', album_ids, cache)
|
||||
except Exception as e:
|
||||
logger.debug("resolve deezer album track positions: %s", e)
|
||||
try:
|
||||
time.sleep(0.3) # Respect rate limits
|
||||
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
|
||||
if a_resp.ok:
|
||||
a_data = a_resp.json()
|
||||
album_release_dates[aid] = a_data.get('release_date', '')
|
||||
# Store in metadata cache for future use
|
||||
if cache:
|
||||
try:
|
||||
cache.store_entity('deezer', 'album', aid, a_data)
|
||||
except Exception as e:
|
||||
logger.debug("cache store_entity album release_date: %s", e)
|
||||
except Exception as e:
|
||||
logger.debug("fetch deezer album release_date %s: %s", aid, e)
|
||||
|
||||
tracks = []
|
||||
for i, t in enumerate(raw_tracks, start=1):
|
||||
|
|
@ -545,9 +467,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
'id': album_id,
|
||||
},
|
||||
'duration_ms': t.get('duration', 0) * 1000,
|
||||
# REAL album position (resolved above); the playlist index is a last
|
||||
# resort only when the album lookup failed, never the default.
|
||||
'track_number': track_positions.get(str(t.get('id'))) or t.get('track_position') or i,
|
||||
'track_number': i,
|
||||
})
|
||||
|
||||
return {
|
||||
|
|
@ -661,7 +581,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
bitrate = 128
|
||||
quality = 'mp3'
|
||||
|
||||
tr = TrackResult(
|
||||
results.append(TrackResult(
|
||||
username='deezer_dl',
|
||||
filename=f"{track_id}||{artist} - {title}",
|
||||
size=est_size,
|
||||
|
|
@ -675,10 +595,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
title=title,
|
||||
album=album,
|
||||
track_number=item.get('track_position'),
|
||||
)
|
||||
# Stamp CD-quality FLAC (16/44.1) so lossless ranks correctly.
|
||||
tr.set_quality(quality_from_deezer(self._quality))
|
||||
results.append(tr)
|
||||
))
|
||||
|
||||
logger.info(f"Deezer search for '{query}' returned {len(results)} results")
|
||||
return results, []
|
||||
|
|
|
|||
|
|
@ -8,15 +8,7 @@ from datetime import datetime, timedelta
|
|||
from utils.logging_config import get_logger
|
||||
from database.music_database import MusicDatabase
|
||||
from core.deezer_client import DeezerClient
|
||||
from core.worker_utils import (
|
||||
accept_artist_match,
|
||||
artist_name_matches,
|
||||
interruptible_sleep,
|
||||
owned_album_titles,
|
||||
pick_artist_by_catalog,
|
||||
release_titles,
|
||||
set_album_api_track_count,
|
||||
)
|
||||
from core.worker_utils import interruptible_sleep, set_album_api_track_count
|
||||
from core.enrichment.manual_match_honoring import honor_stored_match
|
||||
|
||||
logger = get_logger("deezer_worker")
|
||||
|
|
@ -171,16 +163,6 @@ class DeezerWorker:
|
|||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Pinned-group override (Manage Enrichment Workers): process one
|
||||
# entity type first, then fall through to the normal chain. Unset or
|
||||
# exhausted ⇒ default artist→album→track order, unchanged.
|
||||
from core.worker_utils import read_enrichment_priority, priority_pending_item
|
||||
_prio = read_enrichment_priority('deezer')
|
||||
if _prio:
|
||||
_pi = priority_pending_item(cursor, 'deezer', _prio)
|
||||
if _pi:
|
||||
return _pi
|
||||
|
||||
# Priority 1: Unattempted artists
|
||||
cursor.execute("""
|
||||
SELECT id, name
|
||||
|
|
@ -286,19 +268,10 @@ class DeezerWorker:
|
|||
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
|
||||
return similarity >= self.name_similarity_threshold
|
||||
|
||||
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id,
|
||||
result_artist_name: Optional[str] = None) -> bool:
|
||||
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id) -> bool:
|
||||
"""Verify that the result's artist ID matches the parent artist's stored Deezer ID.
|
||||
|
||||
If mismatched, the album/track search is more specific (uses artist+title),
|
||||
so we trust it and correct the parent artist's deezer_id — BUT only when
|
||||
the result's artist *name* actually matches our parent artist. Without
|
||||
that guard, a collaboration or compilation track (e.g. a track our
|
||||
library credits to Jorja Smith that lives on Kendrick Lamar's curated
|
||||
"Black Panther" album) would search up to an album whose Deezer primary
|
||||
artist is someone else (Kendrick), and we'd stamp that wrong Deezer ID
|
||||
onto our artist — corrupting it (and causing duplicate ids shared across
|
||||
unrelated artists)."""
|
||||
so we trust it and correct the parent artist's deezer_id."""
|
||||
parent_deezer_id = item.get('artist_deezer_id')
|
||||
if not parent_deezer_id:
|
||||
return True
|
||||
|
|
@ -307,20 +280,6 @@ class DeezerWorker:
|
|||
return True
|
||||
|
||||
if str(result_artist_id) != str(parent_deezer_id):
|
||||
# Guard: only correct when the album/track's primary artist is the
|
||||
# SAME artist by name. A mismatch means it's a collab/compilation,
|
||||
# not a stale-id correction.
|
||||
parent_name = item.get('artist') or ''
|
||||
if (result_artist_name and parent_name
|
||||
and not self._name_matches(parent_name, result_artist_name)):
|
||||
logger.info(
|
||||
f"Skipping artist-ID correction from {item['type']} "
|
||||
f"'{item['name']}': result artist '{result_artist_name}' "
|
||||
f"≠ parent '{parent_name}' (collab/compilation, not a "
|
||||
f"correction)"
|
||||
)
|
||||
return True
|
||||
|
||||
logger.info(
|
||||
f"Artist ID correction from {item['type']} '{item['name']}': "
|
||||
f"updating parent artist Deezer ID from {parent_deezer_id} to {result_artist_id}"
|
||||
|
|
@ -409,34 +368,17 @@ class DeezerWorker:
|
|||
logger.debug(f"Preserving existing Deezer ID for artist '{artist_name}': {existing_id}")
|
||||
return
|
||||
|
||||
# Multi-candidate search (was single search_artist) so same-name artists
|
||||
# can be disambiguated: gate by name, then pick the one whose catalog
|
||||
# overlaps the albums this library owns.
|
||||
results = self.client.search_artists(artist_name, limit=5)
|
||||
gated = [a for a in (results or []) if artist_name_matches(artist_name, getattr(a, 'name', ''))]
|
||||
chosen, _overlap = pick_artist_by_catalog(
|
||||
gated,
|
||||
owned_album_titles(self.db, artist_id),
|
||||
lambda a: release_titles(self.client.get_artist_albums_list(a.id)),
|
||||
)
|
||||
|
||||
# search_artists returns lean Artist objects; fetch the full dict (same
|
||||
# shape the old search_artist returned) for storage.
|
||||
result = self.client.get_artist_info(chosen.id) if chosen else None
|
||||
result = self.client.search_artist(artist_name)
|
||||
if result:
|
||||
result_name = result.get('name', '')
|
||||
ok, reason = accept_artist_match(
|
||||
self.db, 'deezer_id', result.get('id'), artist_id,
|
||||
artist_name, result_name,
|
||||
)
|
||||
if ok:
|
||||
if self._name_matches(artist_name, result_name):
|
||||
self._update_artist(artist_id, result)
|
||||
self.stats['matched'] += 1
|
||||
logger.info(f"Matched artist '{artist_name}' -> Deezer ID: {result.get('id')}")
|
||||
else:
|
||||
self._mark_status('artist', artist_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
logger.debug(f"Artist '{artist_name}' not matched: {reason}")
|
||||
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result_name}')")
|
||||
else:
|
||||
self._mark_status('artist', artist_id, 'not_found')
|
||||
self.stats['not_found'] += 1
|
||||
|
|
@ -478,8 +420,7 @@ class DeezerWorker:
|
|||
# Verify artist ID
|
||||
result_artist = result.get('artist', {})
|
||||
result_artist_id = result_artist.get('id') if result_artist else None
|
||||
result_artist_name = result_artist.get('name') if result_artist else None
|
||||
self._verify_artist_id(item, result_artist_id, result_artist_name)
|
||||
self._verify_artist_id(item, result_artist_id)
|
||||
|
||||
# Fetch full album details for label, genres, explicit
|
||||
deezer_album_id = result.get('id')
|
||||
|
|
@ -530,8 +471,7 @@ class DeezerWorker:
|
|||
# Verify artist ID
|
||||
result_artist = result.get('artist', {})
|
||||
result_artist_id = result_artist.get('id') if result_artist else None
|
||||
result_artist_name = result_artist.get('name') if result_artist else None
|
||||
self._verify_artist_id(item, result_artist_id, result_artist_name)
|
||||
self._verify_artist_id(item, result_artist_id)
|
||||
|
||||
# Fetch full track details for BPM
|
||||
deezer_track_id = result.get('id')
|
||||
|
|
|
|||
|
|
@ -1,136 +0,0 @@
|
|||
"""On-demand memory-growth diagnostic (issue #802: ~0.7 MiB/s RSS growth).
|
||||
|
||||
Wraps ``tracemalloc`` so a user seeing runaway memory can capture WHERE the
|
||||
allocations come from instead of us guessing:
|
||||
|
||||
1. start_tracking() — begins tracing + stores a baseline snapshot
|
||||
2. ...reproduce the growth for a few minutes...
|
||||
3. report() — top allocation sites, with the DELTA since baseline
|
||||
(the delta is the leak; absolute sizes are mostly
|
||||
startup noise)
|
||||
4. stop_tracking() — ends tracing, frees trace memory
|
||||
|
||||
Opt-in by design: tracemalloc costs CPU and memory while active (it shadows
|
||||
every allocation), so it must never run by default. The Flask endpoints that
|
||||
expose this live in web_server (GET /api/debug/memory/...) so a user can drive
|
||||
the whole flow from a browser.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
import tracemalloc
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("diagnostics.memory")
|
||||
|
||||
_baseline: Optional[tracemalloc.Snapshot] = None
|
||||
_started_at: Optional[float] = None
|
||||
|
||||
# Allocation-site traces this deep give useful "who called it" context without
|
||||
# pathological overhead.
|
||||
_TRACE_FRAMES = 15
|
||||
|
||||
|
||||
def is_tracking() -> bool:
|
||||
return tracemalloc.is_tracing()
|
||||
|
||||
|
||||
def start_tracking() -> Dict[str, Any]:
|
||||
"""Begin tracing and store the baseline snapshot. Idempotent."""
|
||||
global _baseline, _started_at
|
||||
if tracemalloc.is_tracing():
|
||||
return {"tracking": True, "already_running": True, "started_at": _started_at}
|
||||
tracemalloc.start(_TRACE_FRAMES)
|
||||
_baseline = tracemalloc.take_snapshot()
|
||||
_started_at = time.time()
|
||||
logger.info("Memory tracking started (tracemalloc, %d frames)", _TRACE_FRAMES)
|
||||
return {"tracking": True, "already_running": False, "started_at": _started_at}
|
||||
|
||||
|
||||
def stop_tracking() -> Dict[str, Any]:
|
||||
"""End tracing and free the trace bookkeeping."""
|
||||
global _baseline, _started_at
|
||||
was = tracemalloc.is_tracing()
|
||||
if was:
|
||||
tracemalloc.stop()
|
||||
logger.info("Memory tracking stopped")
|
||||
_baseline = None
|
||||
_started_at = None
|
||||
return {"tracking": False, "was_tracking": was}
|
||||
|
||||
|
||||
def _rss_mb() -> Optional[float]:
|
||||
"""Process RSS in MiB, best-effort (psutil, then /proc fallback)."""
|
||||
try:
|
||||
import psutil
|
||||
return round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 1)
|
||||
except Exception: # noqa: S110 — RSS is optional context; fall through to /proc
|
||||
pass
|
||||
try:
|
||||
with open("/proc/self/status", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
if line.startswith("VmRSS:"):
|
||||
return round(int(line.split()[1]) / 1024, 1)
|
||||
except Exception: # noqa: S110 — no /proc on this platform; RSS stays None
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def format_stat(stat: Any) -> Dict[str, Any]:
|
||||
"""Project one tracemalloc StatisticDiff/Statistic into a plain dict.
|
||||
Duck-typed (reads size/count/size_diff/count_diff/traceback) so it's
|
||||
unit-testable without real snapshots."""
|
||||
tb = getattr(stat, "traceback", None)
|
||||
frames: List[str] = []
|
||||
if tb:
|
||||
# Most-recent-call-last reads naturally top-down in a report.
|
||||
for frame in list(tb)[-3:]:
|
||||
frames.append(f"{frame.filename}:{frame.lineno}")
|
||||
return {
|
||||
"location": frames[-1] if frames else "?",
|
||||
"trace": frames,
|
||||
"size_mb": round(getattr(stat, "size", 0) / (1024 * 1024), 3),
|
||||
"size_diff_mb": round(getattr(stat, "size_diff", 0) / (1024 * 1024), 3),
|
||||
"count": getattr(stat, "count", 0),
|
||||
"count_diff": getattr(stat, "count_diff", 0),
|
||||
}
|
||||
|
||||
|
||||
def report(top: int = 25) -> Dict[str, Any]:
|
||||
"""Current snapshot vs the start_tracking() baseline: the top allocation
|
||||
sites by GROWTH (size_diff). Includes traced totals + process RSS so the
|
||||
user can see how much of the real growth tracemalloc accounts for."""
|
||||
if not tracemalloc.is_tracing():
|
||||
return {
|
||||
"tracking": False,
|
||||
"rss_mb": _rss_mb(),
|
||||
"hint": "Start with /api/debug/memory/start, reproduce the growth "
|
||||
"for a few minutes, then call this again.",
|
||||
}
|
||||
snapshot = tracemalloc.take_snapshot()
|
||||
# Filter the tracer's own bookkeeping out of the picture.
|
||||
snapshot = snapshot.filter_traces((
|
||||
tracemalloc.Filter(False, tracemalloc.__file__),
|
||||
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
|
||||
))
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
|
||||
if _baseline is not None:
|
||||
stats = snapshot.compare_to(_baseline, "traceback")
|
||||
stats.sort(key=lambda s: s.size_diff, reverse=True)
|
||||
else:
|
||||
stats = snapshot.statistics("traceback")
|
||||
|
||||
return {
|
||||
"tracking": True,
|
||||
"started_at": _started_at,
|
||||
"elapsed_seconds": round(time.time() - _started_at, 1) if _started_at else None,
|
||||
"traced_current_mb": round(current / (1024 * 1024), 1),
|
||||
"traced_peak_mb": round(peak / (1024 * 1024), 1),
|
||||
"rss_mb": _rss_mb(),
|
||||
"top_growth": [format_stat(s) for s in stats[:top]],
|
||||
}
|
||||
|
|
@ -12,7 +12,6 @@ import re
|
|||
import time
|
||||
import threading
|
||||
import requests
|
||||
from core.metadata.artist_album_cache import get_cached_artist_album_payload, store_artist_album_items
|
||||
from core.metadata.cache import get_metadata_cache
|
||||
from typing import List, Dict, Any, Optional
|
||||
from dataclasses import dataclass
|
||||
|
|
@ -62,69 +61,6 @@ def rate_limited(func):
|
|||
return wrapper
|
||||
|
||||
|
||||
# Discogs disambiguates duplicate artist names two ways: numeric `(N)`
|
||||
# suffix on the older convention (e.g. "Bullet (2)") and a trailing `*`
|
||||
# on the newer convention (e.g. "John Smith*"). Both are presentation-
|
||||
# only — the underlying canonical name is what users expect to see in
|
||||
# search results, on cards, and especially in import paths (otherwise
|
||||
# library folders end up named `Foo*` on disk). Strip both at every
|
||||
# point a Discogs payload becomes a name string.
|
||||
_DISCOGS_DISAMBIG_RE = re.compile(r'(?:\s*\(\d+\))?\s*\*+\s*$|\s*\(\d+\)\s*$')
|
||||
|
||||
|
||||
def _clean_discogs_artist_name(name: Optional[str]) -> str:
|
||||
"""Strip Discogs disambiguation suffixes — both `(N)` and trailing
|
||||
`*` — from an artist name. Returns '' for None / empty input."""
|
||||
if not name:
|
||||
return ''
|
||||
return _DISCOGS_DISAMBIG_RE.sub('', name).strip()
|
||||
|
||||
|
||||
# --- Discogs album ID typing -------------------------------------------------
|
||||
# Discogs has two album object types — masters (/masters/{id}) and releases
|
||||
# (/releases/{id}) — whose numeric IDs share one space, so release N and master
|
||||
# N are DIFFERENT albums. A bare numeric ID is therefore ambiguous. We tag the
|
||||
# type into the ID string ('m12345' / 'r12345') at the point we parse it, so the
|
||||
# correct endpoint can be chosen later without guessing. (Artist IDs are a single
|
||||
# namespace and stay untagged.)
|
||||
|
||||
def _discogs_album_kind(data: Dict[str, Any]) -> str:
|
||||
"""Classify a Discogs album payload as 'master' or 'release'.
|
||||
|
||||
Search results and artist-discography items carry an explicit ``type``;
|
||||
full detail responses don't, but only master detail has ``main_release``."""
|
||||
t = (data.get('type') or '').lower()
|
||||
if t in ('master', 'release'):
|
||||
return t
|
||||
return 'master' if 'main_release' in data else 'release'
|
||||
|
||||
|
||||
def _tag_discogs_album_id(raw_id: Any, kind: str) -> str:
|
||||
"""``'12345'`` + ``'master'`` -> ``'m12345'``; empty input -> ``''``."""
|
||||
s = str(raw_id or '').strip()
|
||||
if not s:
|
||||
return ''
|
||||
return f"{'m' if kind == 'master' else 'r'}{s}"
|
||||
|
||||
|
||||
def _discogs_album_endpoints(album_id: Any) -> List[str]:
|
||||
"""Map a (possibly tagged) album ID to the API path(s) to try, in order.
|
||||
|
||||
``'m12345'`` -> ``['/masters/12345']``
|
||||
``'r12345'`` -> ``['/releases/12345']``
|
||||
``'12345'`` (legacy untagged) -> ``['/releases/12345', '/masters/12345']``
|
||||
|
||||
Legacy bare IDs are tried release-first because stored IDs originate
|
||||
overwhelmingly from search / manual-match / collection sync (all releases);
|
||||
this also self-heals pre-fix bad matches. Returns ``[]`` for unusable input."""
|
||||
s = str(album_id or '').strip()
|
||||
if len(s) > 1 and s[0] in ('m', 'r') and s[1:].isdigit():
|
||||
return [f"/{'masters' if s[0] == 'm' else 'releases'}/{s[1:]}"]
|
||||
if s.isdigit():
|
||||
return [f'/releases/{s}', f'/masters/{s}']
|
||||
return []
|
||||
|
||||
|
||||
# --- Shared dataclasses (same shape as iTunes/Deezer/Spotify) ---
|
||||
|
||||
@dataclass
|
||||
|
|
@ -181,10 +117,9 @@ class Track:
|
|||
# Artists from track-level or release-level
|
||||
track_artists = []
|
||||
if track_data.get('artists'):
|
||||
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in track_data['artists'] if a.get('name')]
|
||||
track_artists = [a.get('name', '') for a in track_data['artists'] if a.get('name')]
|
||||
if not track_artists and release.get('artists'):
|
||||
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release['artists'] if a.get('name')]
|
||||
track_artists = [a for a in track_artists if a]
|
||||
track_artists = [a.get('name', '') for a in release['artists'] if a.get('name')]
|
||||
if not track_artists:
|
||||
track_artists = ['Unknown Artist']
|
||||
|
||||
|
|
@ -255,9 +190,7 @@ class Artist:
|
|||
|
||||
return cls(
|
||||
id=str(artist_data.get('id', '')),
|
||||
name=_clean_discogs_artist_name(
|
||||
artist_data.get('name', artist_data.get('title', ''))
|
||||
),
|
||||
name=artist_data.get('name', artist_data.get('title', '')),
|
||||
popularity=0,
|
||||
genres=[],
|
||||
followers=0,
|
||||
|
|
@ -283,15 +216,14 @@ class Album:
|
|||
artists = []
|
||||
title = release_data.get('title', '')
|
||||
if release_data.get('artists'):
|
||||
artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release_data['artists'] if a.get('name')]
|
||||
artists = [a.get('name', '') for a in release_data['artists'] if a.get('name')]
|
||||
elif release_data.get('artist'):
|
||||
artists = [_clean_discogs_artist_name(release_data['artist'])]
|
||||
artists = [release_data['artist']]
|
||||
elif ' - ' in title:
|
||||
# Search results: "Radiohead - OK Computer" → artist="Radiohead", title="OK Computer"
|
||||
parts = title.split(' - ', 1)
|
||||
artists = [_clean_discogs_artist_name(parts[0])]
|
||||
artists = [parts[0].strip()]
|
||||
title = parts[1].strip()
|
||||
artists = [a for a in artists if a]
|
||||
if not artists:
|
||||
artists = ['Unknown Artist']
|
||||
|
||||
|
|
@ -350,7 +282,7 @@ class Album:
|
|||
external_urls['discogs_api'] = release_data['resource_url']
|
||||
|
||||
return cls(
|
||||
id=_tag_discogs_album_id(release_data.get('id', ''), _discogs_album_kind(release_data)),
|
||||
id=str(release_data.get('id', '')),
|
||||
name=title,
|
||||
artists=artists,
|
||||
release_date=release_date,
|
||||
|
|
@ -512,8 +444,8 @@ class DiscogsClient:
|
|||
artist_name = ''
|
||||
if artists and isinstance(artists[0], dict):
|
||||
artist_name = (artists[0].get('name') or '').strip()
|
||||
# Strip Discogs disambiguation suffixes — both `(N)` and `*`.
|
||||
artist_name = _clean_discogs_artist_name(artist_name)
|
||||
# Strip trailing "(N)" disambiguation suffix Discogs adds.
|
||||
artist_name = re.sub(r'\s*\(\d+\)$', '', artist_name)
|
||||
if not title or not artist_name:
|
||||
continue
|
||||
|
||||
|
|
@ -689,13 +621,10 @@ class DiscogsClient:
|
|||
if cached and cached.get('title'):
|
||||
data = cached
|
||||
else:
|
||||
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
|
||||
data = None
|
||||
for path in _discogs_album_endpoints(release_id):
|
||||
data = self._api_get(path)
|
||||
if data and data.get('title'):
|
||||
break
|
||||
data = None
|
||||
# Try as master first (artist discography returns master IDs)
|
||||
data = self._api_get(f'/masters/{release_id}')
|
||||
if not data or not data.get('title'):
|
||||
data = self._api_get(f'/releases/{release_id}')
|
||||
if not data:
|
||||
return None
|
||||
cache.store_entity('discogs', 'album', release_id, data)
|
||||
|
|
@ -729,45 +658,22 @@ class DiscogsClient:
|
|||
|
||||
def get_artist_albums(self, artist_id: str, album_type: str = 'album,single', limit: int = 50) -> List[Album]:
|
||||
"""Get releases by an artist. Prefers master releases, filters features."""
|
||||
cache = get_metadata_cache()
|
||||
cached_payload = get_cached_artist_album_payload(cache, 'discogs', artist_id, album_type=album_type, limit=limit)
|
||||
releases = cached_payload.get('_releases') if cached_payload else None
|
||||
artist_name = ''
|
||||
if cached_payload:
|
||||
artist_name = str(cached_payload.get('artist_name') or '').lower()
|
||||
# First get the artist name for feature filtering
|
||||
artist_data = self._api_get(f'/artists/{artist_id}')
|
||||
artist_name = artist_data.get('name', '').lower() if artist_data else ''
|
||||
|
||||
if not isinstance(releases, list) or not releases:
|
||||
# First get the artist name for feature filtering. Strip Discogs
|
||||
# disambiguation suffix so feature-vs-primary matching below
|
||||
# compares against the canonical name, not "Beyoncé*".
|
||||
artist_data = self._api_get(f'/artists/{artist_id}')
|
||||
artist_name = _clean_discogs_artist_name(
|
||||
artist_data.get('name', '') if artist_data else ''
|
||||
).lower()
|
||||
|
||||
data = self._api_get(f'/artists/{artist_id}/releases', {
|
||||
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
|
||||
})
|
||||
if not data or not data.get('releases'):
|
||||
return []
|
||||
releases = data.get('releases') or []
|
||||
store_artist_album_items(
|
||||
cache,
|
||||
'discogs',
|
||||
artist_id,
|
||||
releases,
|
||||
album_type=album_type,
|
||||
limit=limit,
|
||||
items_field='_releases',
|
||||
extra_fields={'artist_name': artist_name},
|
||||
)
|
||||
data = self._api_get(f'/artists/{artist_id}/releases', {
|
||||
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
|
||||
})
|
||||
if not data or not data.get('releases'):
|
||||
return []
|
||||
|
||||
# Separate masters from individual releases — prefer masters (canonical versions)
|
||||
masters = []
|
||||
releases_no_master = []
|
||||
master_titles = set()
|
||||
|
||||
for item in releases:
|
||||
for item in data['releases']:
|
||||
# Skip non-main roles
|
||||
role = item.get('role', 'Main').lower()
|
||||
if role not in ('main', ''):
|
||||
|
|
@ -835,13 +741,10 @@ class DiscogsClient:
|
|||
if cached:
|
||||
return cached
|
||||
|
||||
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
|
||||
data = None
|
||||
for path in _discogs_album_endpoints(release_id):
|
||||
data = self._api_get(path)
|
||||
if data and data.get('tracklist'):
|
||||
break
|
||||
data = None
|
||||
# Try as master first (master IDs are used in artist discography)
|
||||
data = self._api_get(f'/masters/{release_id}')
|
||||
if not data or not data.get('tracklist'):
|
||||
data = self._api_get(f'/releases/{release_id}')
|
||||
if not data or not data.get('tracklist'):
|
||||
return None
|
||||
|
||||
|
|
@ -853,7 +756,7 @@ class DiscogsClient:
|
|||
image_url = (primary or images[0]).get('uri')
|
||||
|
||||
album_info = {
|
||||
'id': str(release_id),
|
||||
'id': str(data.get('id', release_id)),
|
||||
'name': data.get('title', ''),
|
||||
'images': [{'url': image_url, 'height': 600, 'width': 600}] if image_url else [],
|
||||
'release_date': str(data.get('year', '')) if data.get('year') else '',
|
||||
|
|
@ -862,11 +765,7 @@ class DiscogsClient:
|
|||
# Get artists
|
||||
artists_list = []
|
||||
if data.get('artists'):
|
||||
artists_list = [
|
||||
{'name': _clean_discogs_artist_name(a.get('name', ''))}
|
||||
for a in data['artists']
|
||||
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
|
||||
]
|
||||
artists_list = [{'name': a.get('name', '')} for a in data['artists'] if a.get('name')]
|
||||
if not artists_list:
|
||||
artists_list = [{'name': 'Unknown Artist'}]
|
||||
|
||||
|
|
@ -895,11 +794,7 @@ class DiscogsClient:
|
|||
# Per-track artists or fall back to release artists
|
||||
track_artists = artists_list
|
||||
if t.get('artists'):
|
||||
track_artists = [
|
||||
{'name': _clean_discogs_artist_name(a.get('name', ''))}
|
||||
for a in t['artists']
|
||||
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
|
||||
]
|
||||
track_artists = [{'name': a.get('name', '')} for a in t['artists'] if a.get('name')]
|
||||
|
||||
tracks.append({
|
||||
'id': f"{release_id}_t{track_num}",
|
||||
|
|
@ -942,13 +837,9 @@ class DiscogsClient:
|
|||
cached = cache.get_entity('discogs', 'album', str(release_id))
|
||||
if cached and cached.get('title'):
|
||||
return cached
|
||||
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
|
||||
data = None
|
||||
for path in _discogs_album_endpoints(release_id):
|
||||
data = self._api_get(path)
|
||||
if data and data.get('title'):
|
||||
break
|
||||
data = None
|
||||
data = self._api_get(f'/masters/{release_id}')
|
||||
if not data or not data.get('title'):
|
||||
data = self._api_get(f'/releases/{release_id}')
|
||||
if data:
|
||||
cache.store_entity('discogs', 'album', str(release_id), data)
|
||||
return data
|
||||
|
|
|
|||
|
|
@ -17,8 +17,8 @@ from typing import Optional, Dict, Any
|
|||
from datetime import datetime, timedelta
|
||||
from utils.logging_config import get_logger
|
||||
from database.music_database import MusicDatabase
|
||||
from core.discogs_client import DiscogsClient, _discogs_album_kind, _tag_discogs_album_id
|
||||
from core.worker_utils import accept_artist_match, interruptible_sleep, set_album_api_track_count
|
||||
from core.discogs_client import DiscogsClient
|
||||
from core.worker_utils import interruptible_sleep, set_album_api_track_count
|
||||
|
||||
logger = get_logger("discogs_worker")
|
||||
|
||||
|
|
@ -174,16 +174,6 @@ class DiscogsWorker:
|
|||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Pinned-group override (Manage Enrichment Workers): process one
|
||||
# entity type first, then fall through to the normal chain. Discogs
|
||||
# has no track endpoint, so only artist/album are honored.
|
||||
from core.worker_utils import read_enrichment_priority, priority_pending_item
|
||||
_prio = read_enrichment_priority('discogs')
|
||||
if _prio in ('artist', 'album'):
|
||||
_pi = priority_pending_item(cursor, 'discogs', _prio)
|
||||
if _pi:
|
||||
return _pi
|
||||
|
||||
# Priority 1: Unattempted artists
|
||||
cursor.execute("""
|
||||
SELECT id, name FROM artists
|
||||
|
|
@ -332,13 +322,9 @@ class DiscogsWorker:
|
|||
self.stats['not_found'] += 1
|
||||
return
|
||||
|
||||
# Find best match by name similarity (skipping ids already claimed by
|
||||
# a differently-named artist, so we don't create a shared/duplicate id).
|
||||
# Find best match by name similarity
|
||||
for result in results:
|
||||
ok, reason = accept_artist_match(
|
||||
self.db, 'discogs_id', result.id, artist_id, artist_name, result.name,
|
||||
)
|
||||
if ok:
|
||||
if self._name_matches(artist_name, result.name):
|
||||
# Fetch full artist detail (uses cache)
|
||||
data = self.client._fetch_and_cache_artist(result.id)
|
||||
if data:
|
||||
|
|
@ -436,9 +422,7 @@ class DiscogsWorker:
|
|||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Tag the ID with its Discogs type so later re-fetches hit the right
|
||||
# endpoint (master vs release share one numeric space).
|
||||
discogs_id = _tag_discogs_album_id(data.get('id', ''), _discogs_album_kind(data))
|
||||
discogs_id = str(data.get('id', ''))
|
||||
genres = json.dumps(data.get('genres', []))
|
||||
styles = json.dumps(data.get('styles', []))
|
||||
labels = data.get('labels', [])
|
||||
|
|
|
|||
|
|
@ -1,767 +0,0 @@
|
|||
"""Generic, source-agnostic helpers for the playlist-discovery route layer.
|
||||
|
||||
The discovery/sync endpoints in ``web_server.py`` were copy-pasted once per
|
||||
source (Tidal, Deezer, Qobuz, Spotify-public, iTunes-link, YouTube,
|
||||
ListenBrainz, Beatport). The per-source copies differ only by a source label
|
||||
string and which ``<source>_discovery_states`` global they read. This module
|
||||
lifts the source-agnostic pieces into importable, unit-testable helpers so the
|
||||
route functions become thin wrappers — exactly preserving behavior (1:1).
|
||||
|
||||
Each helper is lifted verbatim from its web_server.py counterpart; any
|
||||
per-source quirk that genuinely differs (e.g. Beatport's distinct result
|
||||
shape) is intentionally NOT routed through here and stays in its own function.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("discovery.endpoints")
|
||||
|
||||
|
||||
def convert_results_to_spotify_tracks(
|
||||
discovery_results: List[Dict[str, Any]],
|
||||
source_label: str,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Convert a source's discovery results into the Spotify-track dicts the
|
||||
sync pipeline expects.
|
||||
|
||||
Lifted verbatim from the per-source ``convert_<source>_results_to_spotify_tracks``
|
||||
functions (and the already-generic ``_convert_link_results_to_spotify_tracks``),
|
||||
which were byte-identical apart from the ``source_label`` used in the log
|
||||
line. Two input shapes are supported, matching the originals exactly:
|
||||
|
||||
- ``spotify_data`` (manual-fix shape): copied through, preserving optional
|
||||
``track_number`` / ``disc_number``.
|
||||
- ``spotify_track`` + ``status_class == 'found'`` (auto-discovery shape):
|
||||
rebuilt from the flat ``spotify_*`` fields.
|
||||
|
||||
Any result matching neither shape is skipped, identical to the originals.
|
||||
|
||||
NOTE: Beatport deliberately does NOT use this — its converter coerces
|
||||
artist objects to strings and emits a different track shape (``source``
|
||||
field, album dict), so it keeps its own implementation.
|
||||
"""
|
||||
spotify_tracks: List[Dict[str, Any]] = []
|
||||
|
||||
for result in discovery_results:
|
||||
# Support both data formats: spotify_data (manual fixes) and individual
|
||||
# fields (automatic discovery).
|
||||
if result.get('spotify_data'):
|
||||
spotify_data = result['spotify_data']
|
||||
track = {
|
||||
'id': spotify_data['id'],
|
||||
'name': spotify_data['name'],
|
||||
'artists': spotify_data['artists'],
|
||||
'album': spotify_data['album'],
|
||||
'duration_ms': spotify_data.get('duration_ms', 0),
|
||||
}
|
||||
if spotify_data.get('track_number'):
|
||||
track['track_number'] = spotify_data['track_number']
|
||||
if spotify_data.get('disc_number'):
|
||||
track['disc_number'] = spotify_data['disc_number']
|
||||
spotify_tracks.append(track)
|
||||
elif result.get('spotify_track') and result.get('status_class') == 'found':
|
||||
spotify_tracks.append({
|
||||
'id': result.get('spotify_id', 'unknown'),
|
||||
'name': result.get('spotify_track', 'Unknown Track'),
|
||||
'artists': [result.get('spotify_artist', 'Unknown Artist')] if result.get('spotify_artist') else ['Unknown Artist'],
|
||||
'album': result.get('spotify_album', 'Unknown Album'),
|
||||
'duration_ms': 0,
|
||||
})
|
||||
|
||||
logger.info(f"Converted {len(spotify_tracks)} {source_label} matches to Spotify tracks for sync")
|
||||
return spotify_tracks
|
||||
|
||||
|
||||
def cancel_sync(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
label: str,
|
||||
not_found_message: str,
|
||||
sync_lock: Any,
|
||||
sync_states: Dict[str, Any],
|
||||
active_sync_workers: Dict[str, Any],
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Cancel an in-progress sync for one discovery playlist.
|
||||
|
||||
1:1 lift of the byte-identical ``cancel_<source>_sync`` bodies (Tidal,
|
||||
Deezer, Qobuz, Spotify-Public, iTunes-Link, YouTube, ListenBrainz). The
|
||||
caller passes the already-resolved state key (ListenBrainz transforms it
|
||||
via ``_lb_state_key`` first), the source ``label``, the exact 404 message
|
||||
(iTunes-Link uses "iTunes Link not found", not "... playlist not found"),
|
||||
and the shared sync infrastructure (so this stays free of web_server
|
||||
globals / Flask).
|
||||
|
||||
Returns ``(payload_dict, status_code)``; the caller wraps in ``jsonify``.
|
||||
|
||||
Beatport is NOT routed here — it cancels a stored ``sync_future`` and
|
||||
returns a different payload.
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
# Idempotent: the live discovery state is gone (a restart wiped the
|
||||
# in-memory state, or it was already cancelled). Cancelling a sync
|
||||
# that isn't running is a no-op SUCCESS, not a 404 — otherwise a
|
||||
# mirrored playlist (e.g. a ListenBrainz weekly) whose state vanished
|
||||
# is permanently wedged with "playlist not found" and can never be
|
||||
# re-synced or dismissed (#702).
|
||||
return {"success": True, "message": f"No active {label} sync to cancel"}, 200
|
||||
|
||||
state = states[key]
|
||||
state['last_accessed'] = time.time()
|
||||
sync_playlist_id = state.get('sync_playlist_id')
|
||||
|
||||
if sync_playlist_id:
|
||||
with sync_lock:
|
||||
sync_states[sync_playlist_id] = {"status": "cancelled"}
|
||||
if sync_playlist_id in active_sync_workers:
|
||||
del active_sync_workers[sync_playlist_id]
|
||||
|
||||
state['phase'] = 'discovered'
|
||||
state['sync_playlist_id'] = None
|
||||
state['sync_progress'] = {}
|
||||
|
||||
return {"success": True, "message": f"{label} sync cancelled"}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error cancelling {label} sync: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def delete_playlist_state(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
label: str,
|
||||
not_found_message: str,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Delete a discovery playlist's state entry, cancelling any active
|
||||
discovery first.
|
||||
|
||||
1:1 lift of the byte-identical ``delete_<source>_playlist`` bodies
|
||||
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
|
||||
|
||||
The iTunes-Link / YouTube / ListenBrainz / Beatport deletes intentionally
|
||||
keep their own bodies — they differ in success message, info-log wording,
|
||||
name extraction, and/or key transform.
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
state = states[key]
|
||||
if 'discovery_future' in state and state['discovery_future']:
|
||||
state['discovery_future'].cancel()
|
||||
del states[key]
|
||||
|
||||
logger.info(f"Deleted {label} playlist state: {key}")
|
||||
return {"success": True, "message": "Playlist deleted"}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting {label} playlist: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
# --- playlist-name accessors -------------------------------------------------
|
||||
# The per-source sync-status handlers read the display name three different
|
||||
# ways. Each is reproduced verbatim so the 1:1 behavior (including which ones
|
||||
# raise vs. fall back to 'Unknown Playlist') is preserved.
|
||||
|
||||
def playlist_name_attr_or_unknown(state: Dict[str, Any]) -> str:
|
||||
"""Tidal: playlist is an object — use ``.name`` or 'Unknown Playlist'."""
|
||||
pl = state.get('playlist')
|
||||
return pl.name if pl and hasattr(pl, 'name') else 'Unknown Playlist'
|
||||
|
||||
|
||||
def playlist_name_strict(state: Dict[str, Any]) -> str:
|
||||
"""Deezer / Qobuz / Spotify-Public / iTunes-Link: strict dict access —
|
||||
raises (→ 500) if 'playlist' is missing, exactly like the originals."""
|
||||
return state['playlist']['name']
|
||||
|
||||
|
||||
def playlist_name_safe(state: Dict[str, Any]) -> str:
|
||||
"""YouTube / ListenBrainz: safe dict access, defaulting to 'Unknown
|
||||
Playlist'."""
|
||||
return state.get('playlist', {}).get('name', 'Unknown Playlist')
|
||||
|
||||
|
||||
def playlist_name_obj(state: Dict[str, Any]) -> str:
|
||||
"""Tidal start-sync: playlist is an object — strict ``.name`` (raises if
|
||||
absent, exactly like the original)."""
|
||||
return state['playlist'].name
|
||||
|
||||
|
||||
def playlist_image_obj(state: Dict[str, Any]) -> str:
|
||||
"""Tidal: ``getattr(playlist, 'image_url', '')`` (object attribute)."""
|
||||
return getattr(state['playlist'], 'image_url', '')
|
||||
|
||||
|
||||
def playlist_image_dict(state: Dict[str, Any]) -> str:
|
||||
"""Deezer/Qobuz/Spotify-Public/YouTube: ``playlist.get('image_url', '')``
|
||||
(dict access)."""
|
||||
return state['playlist'].get('image_url', '')
|
||||
|
||||
|
||||
def get_sync_status(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
not_found_message: str,
|
||||
error_label: str,
|
||||
activity_subject: str,
|
||||
playlist_name_getter,
|
||||
sync_lock: Any,
|
||||
sync_states: Dict[str, Any],
|
||||
add_activity_item,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Report sync status for one discovery playlist, posting an activity-feed
|
||||
item when the sync finishes or errors.
|
||||
|
||||
1:1 lift of the ``get_<source>_sync_status`` bodies (Tidal, Deezer, Qobuz,
|
||||
Spotify-Public, iTunes-Link, YouTube, ListenBrainz). Per-source variation
|
||||
is captured by the parameters:
|
||||
|
||||
- ``not_found_message`` — the 404 string (iTunes-Link drops "playlist").
|
||||
- ``error_label`` — used in the except log ("Error getting <X> sync status").
|
||||
- ``activity_subject`` — the activity-feed prefix; note Spotify-Public uses
|
||||
"Spotify Link playlist" while its error_label is "Spotify Public".
|
||||
- ``playlist_name_getter`` — one of the accessors above (attr/strict/safe);
|
||||
the strict one can raise, matching the originals (→ 500). The state's
|
||||
phase/sync_progress are mutated BEFORE the name is read, so a raising
|
||||
getter leaves the same partial mutation the original did.
|
||||
|
||||
Beatport is NOT routed here — it returns a different payload (``status``
|
||||
not ``sync_status``, includes ``sync_id``, no lock, ``chart`` key).
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
state = states[key]
|
||||
state['last_accessed'] = time.time()
|
||||
sync_playlist_id = state.get('sync_playlist_id')
|
||||
|
||||
if not sync_playlist_id:
|
||||
return {"error": "No sync in progress"}, 404
|
||||
|
||||
with sync_lock:
|
||||
sync_state = sync_states.get(sync_playlist_id, {})
|
||||
|
||||
response = {
|
||||
'phase': state['phase'],
|
||||
'sync_status': sync_state.get('status', 'unknown'),
|
||||
'progress': sync_state.get('progress', {}),
|
||||
'complete': sync_state.get('status') == 'finished',
|
||||
'error': sync_state.get('error'),
|
||||
}
|
||||
|
||||
if sync_state.get('status') == 'finished':
|
||||
state['phase'] = 'sync_complete'
|
||||
state['sync_progress'] = sync_state.get('progress', {})
|
||||
playlist_name = playlist_name_getter(state)
|
||||
add_activity_item("", "Sync Complete", f"{activity_subject} '{playlist_name}' synced successfully", "Now")
|
||||
elif sync_state.get('status') == 'error':
|
||||
state['phase'] = 'discovered'
|
||||
playlist_name = playlist_name_getter(state)
|
||||
add_activity_item("", "Sync Failed", f"{activity_subject} '{playlist_name}' sync failed", "Now")
|
||||
|
||||
return response, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting {error_label} sync status: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def get_discovery_status(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
not_found_message: str,
|
||||
error_label: str,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Report real-time discovery progress/results for one playlist.
|
||||
|
||||
1:1 lift of the byte-identical ``get_<source>_discovery_status`` bodies.
|
||||
Unlike sync-status, this shape is identical for ALL eight sources —
|
||||
Beatport included — so it folds in too. Only the 404 message
|
||||
(".../discovery not found" vs ".../playlist not found" vs "Beatport chart
|
||||
not found") and the except-log label vary, both passed in. The caller
|
||||
resolves the key (ListenBrainz via ``_lb_state_key``).
|
||||
|
||||
Returns ``(payload, status_code)``.
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
state = states[key]
|
||||
state['last_accessed'] = time.time()
|
||||
|
||||
return {
|
||||
'phase': state['phase'],
|
||||
'status': state['status'],
|
||||
'progress': state['discovery_progress'],
|
||||
'spotify_matches': state['spotify_matches'],
|
||||
'spotify_total': state['spotify_total'],
|
||||
'results': state['discovery_results'],
|
||||
'complete': state['phase'] == 'discovered',
|
||||
}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting {error_label} discovery status: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def reset_playlist(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
label: str,
|
||||
not_found_message: str,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Reset a discovery playlist back to the 'fresh' phase, clearing all
|
||||
discovery/sync data while preserving the original playlist payload.
|
||||
|
||||
1:1 lift of the byte-identical ``reset_<source>_playlist`` bodies
|
||||
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
|
||||
|
||||
NOT folded in (genuinely divergent): YouTube (status -> 'parsed', no
|
||||
download_process_id, logs the playlist name, "reset to fresh state"),
|
||||
ListenBrainz (status -> 'cached', logs playlist title, returns
|
||||
{"phase": "fresh"}), iTunes-Link (uses state.update, no info log, distinct
|
||||
message). Those keep their own bodies.
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
state = states[key]
|
||||
if 'discovery_future' in state and state['discovery_future']:
|
||||
state['discovery_future'].cancel()
|
||||
|
||||
state['phase'] = 'fresh'
|
||||
state['status'] = 'fresh'
|
||||
state['discovery_results'] = []
|
||||
state['discovery_progress'] = 0
|
||||
state['spotify_matches'] = 0
|
||||
state['sync_playlist_id'] = None
|
||||
state['converted_spotify_playlist_id'] = None
|
||||
state['download_process_id'] = None
|
||||
state['sync_progress'] = {}
|
||||
state['discovery_future'] = None
|
||||
state['last_accessed'] = time.time()
|
||||
|
||||
logger.info(f"Reset {label} playlist to fresh: {key}")
|
||||
return {"success": True, "message": "Playlist reset to fresh phase"}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error resetting {label} playlist: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def get_playlist_states(
|
||||
states: Dict[str, Any],
|
||||
*,
|
||||
error_label: str,
|
||||
info_log_label: str = None,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Return all stored discovery states for a source as a list for frontend
|
||||
card hydration (``{"states": [...]}``).
|
||||
|
||||
1:1 lift of the ``get_<source>_playlist_states`` bodies (Tidal, Deezer,
|
||||
Qobuz, Spotify-Public, iTunes-Link), which build the same per-entry dict.
|
||||
iTunes-Link is the only one without the "Returning N ..." info log, so
|
||||
``info_log_label`` is optional (pass None to suppress it, as iTunes did).
|
||||
|
||||
NOT folded in: the YouTube/ListenBrainz ``get_all_*_playlists`` endpoints —
|
||||
they return ``{"playlists": [...]}`` (different key + fields: url/created_at,
|
||||
no discovery_results) and filter mirrored/profile-scoped entries.
|
||||
"""
|
||||
try:
|
||||
result = []
|
||||
current_time = time.time()
|
||||
|
||||
for key, state in states.items():
|
||||
state['last_accessed'] = current_time
|
||||
result.append({
|
||||
'playlist_id': key,
|
||||
'phase': state['phase'],
|
||||
'status': state['status'],
|
||||
'discovery_progress': state['discovery_progress'],
|
||||
'spotify_matches': state['spotify_matches'],
|
||||
'spotify_total': state['spotify_total'],
|
||||
'discovery_results': state['discovery_results'],
|
||||
'converted_spotify_playlist_id': state.get('converted_spotify_playlist_id'),
|
||||
'download_process_id': state.get('download_process_id'),
|
||||
'last_accessed': state['last_accessed'],
|
||||
})
|
||||
|
||||
if info_log_label:
|
||||
logger.info(f"Returning {len(result)} stored {info_log_label} playlist states for hydration")
|
||||
return {"states": result}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting {error_label} playlist states: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def save_bubble_snapshot(
|
||||
get_json,
|
||||
*,
|
||||
payload_key: str,
|
||||
no_data_error: str,
|
||||
snapshot_kind: str,
|
||||
success_noun: str,
|
||||
log_subject: str,
|
||||
log_noun: str,
|
||||
get_database,
|
||||
get_current_profile_id,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Persist a bubble/download snapshot for cross-refresh hydration.
|
||||
|
||||
1:1 lift of the four structurally-identical snapshot endpoints
|
||||
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles),
|
||||
which differ only by:
|
||||
|
||||
- ``payload_key`` ('downloads' for discover, 'bubbles' for the rest) and
|
||||
its ``no_data_error`` message.
|
||||
- ``snapshot_kind`` — the db.save_bubble_snapshot category.
|
||||
- ``success_noun`` — fills "Snapshot saved with N <noun>".
|
||||
- ``log_subject`` / ``log_noun`` — the info ("Saved <subject>: N <noun>")
|
||||
and except ("Error saving <subject>") log lines.
|
||||
|
||||
Returns ``(payload, status_code)``. ``get_json`` is invoked inside the try
|
||||
like the original ``request.json``.
|
||||
"""
|
||||
try:
|
||||
from datetime import datetime
|
||||
|
||||
data = get_json()
|
||||
if not data or payload_key not in data:
|
||||
return {'success': False, 'error': no_data_error}, 400
|
||||
|
||||
items = data[payload_key]
|
||||
|
||||
db = get_database()
|
||||
db.save_bubble_snapshot(snapshot_kind, items, profile_id=get_current_profile_id())
|
||||
|
||||
count = len(items)
|
||||
logger.info(f"Saved {log_subject}: {count} {log_noun}")
|
||||
|
||||
return {
|
||||
'success': True,
|
||||
'message': f'Snapshot saved with {count} {success_noun}',
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving {log_subject}: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return {'success': False, 'error': str(e)}, 500
|
||||
|
||||
|
||||
def update_playlist_phase(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
get_json,
|
||||
*,
|
||||
not_found_message: str,
|
||||
error_label: str,
|
||||
valid_phases: List[str],
|
||||
apply_extra_fields: bool,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Update a discovery playlist's phase (used when the modal closes, e.g. to
|
||||
reset download_complete -> discovered).
|
||||
|
||||
1:1 lift of the ``update_<source>_playlist_phase`` bodies for the five
|
||||
sources with the identical validation + full-message response (Tidal,
|
||||
Deezer, Qobuz, Spotify-Public, YouTube). Per-source params:
|
||||
|
||||
- ``valid_phases`` — YouTube's list additionally includes 'parsed'.
|
||||
- ``apply_extra_fields`` — Deezer/Qobuz/Spotify-Public also persist
|
||||
download_process_id / converted_spotify_playlist_id from the body;
|
||||
Tidal/YouTube do NOT (so pass False to keep them 1:1).
|
||||
- ``not_found_message`` / ``error_label``; ``get_json`` invoked inside the
|
||||
try like the original ``request.get_json()``.
|
||||
|
||||
Returns ``(payload, status_code)``.
|
||||
|
||||
NOT folded in: iTunes-Link — it uses ``data.get('phase')`` (no separate
|
||||
"Phase not provided" 400) and returns a no-message payload.
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
data = get_json()
|
||||
if not data or 'phase' not in data:
|
||||
return {"error": "Phase not provided"}, 400
|
||||
|
||||
new_phase = data['phase']
|
||||
if new_phase not in valid_phases:
|
||||
return {"error": f"Invalid phase. Must be one of: {', '.join(valid_phases)}"}, 400
|
||||
|
||||
state = states[key]
|
||||
old_phase = state.get('phase', 'unknown')
|
||||
state['phase'] = new_phase
|
||||
state['last_accessed'] = time.time()
|
||||
|
||||
if apply_extra_fields:
|
||||
if 'download_process_id' in data:
|
||||
state['download_process_id'] = data['download_process_id']
|
||||
if 'converted_spotify_playlist_id' in data:
|
||||
state['converted_spotify_playlist_id'] = data['converted_spotify_playlist_id']
|
||||
|
||||
logger.info(f"Updated {error_label} playlist {key} phase: {old_phase} → {new_phase}")
|
||||
return {"success": True, "message": f"Phase updated to {new_phase}", "old_phase": old_phase, "new_phase": new_phase}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating {error_label} playlist phase: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
||||
|
||||
def first_artist_str_or_obj(original_track: Dict[str, Any]) -> str:
|
||||
"""Tidal: first artist from an artists list that may hold strings OR
|
||||
objects ({'name': ...}); '' when empty."""
|
||||
artists = original_track.get('artists', [])
|
||||
if artists:
|
||||
return artists[0] if isinstance(artists[0], str) else artists[0].get('name', '')
|
||||
return ''
|
||||
|
||||
|
||||
def first_artist_plain(original_track: Dict[str, Any]) -> str:
|
||||
"""Deezer/Qobuz/Spotify-Public: first artist assuming a list of strings;
|
||||
'' when empty."""
|
||||
artists = original_track.get('artists', [])
|
||||
return artists[0] if artists else ''
|
||||
|
||||
|
||||
def update_discovery_match(
|
||||
states: Dict[str, Any],
|
||||
get_json,
|
||||
*,
|
||||
source_log_label: str,
|
||||
error_label: str,
|
||||
original_track_key: str,
|
||||
original_artist_getter,
|
||||
join_artist_names,
|
||||
extract_artist_name,
|
||||
build_fix_modal_spotify_data,
|
||||
get_discovery_cache_key,
|
||||
get_database,
|
||||
get_active_discovery_source,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Apply a manually-selected Spotify track to a discovery result (the
|
||||
fix-modal flow) and persist it to the discovery cache.
|
||||
|
||||
1:1 lift of the ``update_<source>_discovery_match`` bodies for the four
|
||||
sources with the identical structure (Tidal, Deezer, Qobuz, Spotify-Public).
|
||||
Per-source pieces are params:
|
||||
|
||||
- ``source_log_label`` (lowercase, e.g. "tidal") for the "Manual match
|
||||
updated: ..." line; ``error_label`` for the except log.
|
||||
- ``original_track_key`` — the raw-source track key on the result
|
||||
('tidal_track', 'deezer_track', ...).
|
||||
- ``original_artist_getter`` — Tidal handles string-or-object artists
|
||||
(``first_artist_str_or_obj``); the rest assume strings
|
||||
(``first_artist_plain``).
|
||||
- the web_server helpers (join/extract artist, build_fix_modal_spotify_data,
|
||||
cache-key, get_database, active-discovery-source) are injected so this
|
||||
stays free of those globals.
|
||||
- ``get_json`` is called INSIDE the try (like the original's
|
||||
``request.get_json()``) so a malformed body yields the same 500.
|
||||
|
||||
Returns ``(payload, status_code)``.
|
||||
|
||||
NOT folded in: iTunes-Link (saves spotify_data directly via a different
|
||||
cache signature), YouTube (multi-key original_track fallback), ListenBrainz
|
||||
(entirely different unmatch-capable structure, no cache write), Beatport.
|
||||
"""
|
||||
try:
|
||||
data = get_json()
|
||||
identifier = data.get('identifier')
|
||||
track_index = data.get('track_index')
|
||||
spotify_track = data.get('spotify_track')
|
||||
|
||||
if not identifier or track_index is None or not spotify_track:
|
||||
return {'error': 'Missing required fields'}, 400
|
||||
|
||||
state = states.get(identifier)
|
||||
result = None
|
||||
if state:
|
||||
if track_index >= len(state['discovery_results']):
|
||||
return {'error': 'Invalid track index'}, 400
|
||||
|
||||
result = state['discovery_results'][track_index]
|
||||
old_status = result.get('status')
|
||||
|
||||
result['status'] = 'Found'
|
||||
result['status_class'] = 'found'
|
||||
result['spotify_track'] = spotify_track['name']
|
||||
result['spotify_artist'] = join_artist_names(spotify_track['artists']) if isinstance(spotify_track['artists'], list) else extract_artist_name(spotify_track['artists'])
|
||||
result['spotify_album'] = spotify_track['album']
|
||||
result['spotify_id'] = spotify_track['id']
|
||||
|
||||
duration_ms = spotify_track.get('duration_ms', 0)
|
||||
if duration_ms:
|
||||
minutes = duration_ms // 60000
|
||||
seconds = (duration_ms % 60000) // 1000
|
||||
result['duration'] = f"{minutes}:{seconds:02d}"
|
||||
else:
|
||||
result['duration'] = '0:00'
|
||||
|
||||
result['spotify_data'] = build_fix_modal_spotify_data(spotify_track)
|
||||
result['wing_it_fallback'] = False
|
||||
result['manual_match'] = True
|
||||
|
||||
if old_status != 'found' and old_status != 'Found':
|
||||
state['spotify_matches'] = state.get('spotify_matches', 0) + 1
|
||||
|
||||
logger.info(f"Manual match updated: {source_log_label} - {identifier} - track {track_index}")
|
||||
logger.info(f" → {result['spotify_artist']} - {result['spotify_track']}")
|
||||
|
||||
original_track = result.get(original_track_key, {})
|
||||
original_name = original_track.get('name', spotify_track['name'])
|
||||
original_artist = original_artist_getter(original_track)
|
||||
else:
|
||||
# #843: the in-memory discovery state can be gone — a server restart,
|
||||
# or an imported playlist that wasn't discovered in THIS process —
|
||||
# while the card is still shown from persisted data. The DURABLE part
|
||||
# of a manual fix (writing the match to the discovery cache so future
|
||||
# syncs resolve it) doesn't need the in-memory state, only the original
|
||||
# track's name + artist, which the client now sends. Fall back to those
|
||||
# instead of 404ing the fix into uselessness.
|
||||
original_name = (data.get('original_name') or '').strip()
|
||||
original_artist = (data.get('original_artist') or '').strip()
|
||||
if not original_name and not original_artist:
|
||||
return {'error': 'Discovery state not found'}, 404
|
||||
if not original_name:
|
||||
original_name = spotify_track['name']
|
||||
# Key the cache by the FIRST artist — every in-memory + sync path uses
|
||||
# artists[0], but the client may send a joined "A, B, C" string. Without
|
||||
# this, a multi-artist track would save under a key the sync never looks
|
||||
# up (full string ≠ first artist), so the fix would silently never apply.
|
||||
if original_artist:
|
||||
original_artist = original_artist.split(',')[0].strip()
|
||||
logger.info(
|
||||
f"Manual match (no in-memory state) → discovery cache: "
|
||||
f"{source_log_label} - {identifier} - '{original_name}' by '{original_artist}'"
|
||||
)
|
||||
|
||||
try:
|
||||
cache_key = get_discovery_cache_key(original_name, original_artist)
|
||||
artists_list = spotify_track['artists']
|
||||
if isinstance(artists_list, list):
|
||||
artists_list = [a if isinstance(a, str) else a.get('name', '') for a in artists_list]
|
||||
image_url = spotify_track.get('image_url') or ''
|
||||
album_raw = spotify_track.get('album', '')
|
||||
if isinstance(album_raw, dict):
|
||||
album_obj = dict(album_raw)
|
||||
if image_url and not album_obj.get('image_url'):
|
||||
album_obj['image_url'] = image_url
|
||||
if image_url and not album_obj.get('images'):
|
||||
album_obj['images'] = [{'url': image_url}]
|
||||
else:
|
||||
album_obj = {'name': album_raw or ''}
|
||||
if image_url:
|
||||
album_obj['image_url'] = image_url
|
||||
album_obj['images'] = [{'url': image_url}]
|
||||
|
||||
matched_data = {
|
||||
'id': spotify_track['id'],
|
||||
'name': spotify_track['name'],
|
||||
'artists': artists_list,
|
||||
'album': album_obj,
|
||||
'duration_ms': spotify_track.get('duration_ms', 0),
|
||||
'image_url': image_url,
|
||||
'source': 'spotify',
|
||||
}
|
||||
cache_db = get_database()
|
||||
cache_db.save_discovery_cache_match(
|
||||
cache_key[0], cache_key[1], get_active_discovery_source(), 1.0, matched_data,
|
||||
original_name, original_artist
|
||||
)
|
||||
logger.info(f"Manual fix saved to discovery cache: {original_name} by {original_artist}")
|
||||
except Exception as cache_err:
|
||||
logger.error(f"Error saving manual fix to discovery cache: {cache_err}")
|
||||
|
||||
return {'success': True, 'result': result}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating {error_label} discovery match: {e}")
|
||||
return {'error': str(e)}, 500
|
||||
|
||||
|
||||
def start_sync(
|
||||
states: Dict[str, Any],
|
||||
key: str,
|
||||
*,
|
||||
sync_id_prefix: str,
|
||||
not_found_message: str,
|
||||
not_ready_message: str,
|
||||
convert_fn,
|
||||
playlist_name_getter,
|
||||
playlist_image_getter,
|
||||
activity_label: str,
|
||||
error_label: str,
|
||||
sync_lock: Any,
|
||||
sync_states: Dict[str, Any],
|
||||
active_sync_workers: Dict[str, Any],
|
||||
submit_sync_task,
|
||||
add_activity_item,
|
||||
) -> Tuple[Dict[str, Any], int]:
|
||||
"""Kick off a playlist sync from a source's discovered Spotify matches.
|
||||
|
||||
1:1 lift of the ``start_<source>_sync`` bodies for the five sources with
|
||||
the identical flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube). The
|
||||
per-source pieces are parameters:
|
||||
|
||||
- ``sync_id_prefix`` — the ``f"{prefix}_{key}"`` sync id.
|
||||
- ``convert_fn`` — the source's discovery->spotify-tracks converter.
|
||||
- ``playlist_name_getter`` / ``playlist_image_getter`` — Tidal reads an
|
||||
object (``.name`` / ``getattr``), the rest read a dict; lifted as the
|
||||
``playlist_name_obj``/``playlist_image_obj`` vs ``playlist_name_strict``/
|
||||
``playlist_image_dict`` accessors.
|
||||
- ``activity_label`` vs ``error_label`` — these DIFFER for Spotify-Public:
|
||||
activity says "Spotify Link Sync Started" while logs say "Spotify Public".
|
||||
- ``submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks,
|
||||
playlist_image_url) -> Future`` — wraps sync_executor/_run_sync_task/
|
||||
get_current_profile_id so this stays free of those globals.
|
||||
|
||||
Returns ``(payload, status_code)``.
|
||||
|
||||
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
|
||||
task without an image arg), Beatport (extra debug logging, 'chart' key).
|
||||
"""
|
||||
try:
|
||||
if key not in states:
|
||||
return {"error": not_found_message}, 404
|
||||
|
||||
state = states[key]
|
||||
state['last_accessed'] = time.time()
|
||||
|
||||
if state['phase'] not in ['discovered', 'sync_complete', 'download_complete']:
|
||||
return {"error": not_ready_message}, 400
|
||||
|
||||
spotify_tracks = convert_fn(state['discovery_results'])
|
||||
if not spotify_tracks:
|
||||
return {"error": "No Spotify matches found for sync"}, 400
|
||||
|
||||
sync_playlist_id = f"{sync_id_prefix}_{key}"
|
||||
playlist_name = playlist_name_getter(state)
|
||||
|
||||
add_activity_item("", f"{activity_label} Sync Started", f"'{playlist_name}' - {len(spotify_tracks)} tracks", "Now")
|
||||
|
||||
state['phase'] = 'syncing'
|
||||
state['sync_playlist_id'] = sync_playlist_id
|
||||
state['sync_progress'] = {}
|
||||
|
||||
with sync_lock:
|
||||
sync_states[sync_playlist_id] = {"status": "starting", "progress": {}}
|
||||
|
||||
playlist_image_url = playlist_image_getter(state)
|
||||
future = submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks, playlist_image_url)
|
||||
active_sync_workers[sync_playlist_id] = future
|
||||
|
||||
logger.info(f"Started {error_label} sync for: {playlist_name} ({len(spotify_tracks)} tracks)")
|
||||
return {"success": True, "sync_playlist_id": sync_playlist_id}, 200
|
||||
except Exception as e:
|
||||
logger.error(f"Error starting {error_label} sync: {e}")
|
||||
return {"error": str(e)}, 500
|
||||
|
|
@ -17,14 +17,10 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
|
||||
def get_current_profile_id() -> int:
|
||||
"""Mirror of web_server.get_current_profile_id — uses Flask g.
|
||||
|
||||
Catches RuntimeError too because reading `g` outside a request
|
||||
context raises that (not AttributeError) — happens when this is
|
||||
called from background threads (sync, automation, scanners)."""
|
||||
"""Mirror of web_server.get_current_profile_id — uses Flask g."""
|
||||
try:
|
||||
return g.profile_id
|
||||
except (AttributeError, RuntimeError):
|
||||
except AttributeError:
|
||||
return 1
|
||||
|
||||
|
||||
|
|
@ -96,8 +92,6 @@ def get_discover_hero():
|
|||
artist_id = artist.spotify_artist_id
|
||||
elif active_source == 'deezer':
|
||||
artist_id = getattr(artist, 'deezer_artist_id', None) or artist.itunes_artist_id
|
||||
elif active_source == 'musicbrainz':
|
||||
artist_id = getattr(artist, 'musicbrainz_artist_id', None) or artist.itunes_artist_id
|
||||
else:
|
||||
artist_id = artist.itunes_artist_id
|
||||
if not artist_id:
|
||||
|
|
@ -127,7 +121,7 @@ def get_discover_hero():
|
|||
valid_artists = list(similar_artists)
|
||||
|
||||
# FALLBACK: If no valid artists for fallback source, try to resolve IDs on-the-fly
|
||||
if active_source in ('itunes', 'deezer', 'musicbrainz') and not valid_artists:
|
||||
if active_source in ('itunes', 'deezer') and not valid_artists:
|
||||
logger.warning(f"[{active_source} Fallback] No artists with {active_source} IDs found, attempting on-the-fly resolution for {len(similar_artists)} artists")
|
||||
resolved_count = 0
|
||||
for artist in similar_artists:
|
||||
|
|
@ -137,20 +131,13 @@ def get_discover_hero():
|
|||
continue
|
||||
# Try to resolve ID by name
|
||||
try:
|
||||
resolve_client = itunes_client
|
||||
if active_source == 'musicbrainz':
|
||||
from core.metadata.registry import get_musicbrainz_client
|
||||
resolve_client = get_musicbrainz_client()
|
||||
search_results = resolve_client.search_artists(artist.similar_artist_name, limit=1)
|
||||
search_results = itunes_client.search_artists(artist.similar_artist_name, limit=1)
|
||||
if search_results and len(search_results) > 0:
|
||||
resolved_id = search_results[0].id
|
||||
# Cache the resolved ID for future use
|
||||
if active_source == 'deezer':
|
||||
database.update_similar_artist_deezer_id(artist.id, resolved_id)
|
||||
artist.similar_artist_deezer_id = resolved_id
|
||||
elif active_source == 'musicbrainz':
|
||||
database.update_similar_artist_musicbrainz_id(artist.id, resolved_id)
|
||||
artist.similar_artist_musicbrainz_id = resolved_id
|
||||
else:
|
||||
database.update_similar_artist_itunes_id(artist.id, resolved_id)
|
||||
artist.similar_artist_itunes_id = resolved_id
|
||||
|
|
@ -182,15 +169,12 @@ def get_discover_hero():
|
|||
artist_id = artist.similar_artist_spotify_id or artist.similar_artist_itunes_id
|
||||
elif active_source == 'deezer':
|
||||
artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
|
||||
elif active_source == 'musicbrainz':
|
||||
artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
|
||||
else:
|
||||
artist_id = artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
|
||||
|
||||
artist_data = {
|
||||
"spotify_artist_id": artist.similar_artist_spotify_id,
|
||||
"itunes_artist_id": artist.similar_artist_itunes_id,
|
||||
"musicbrainz_artist_id": getattr(artist, 'similar_artist_musicbrainz_id', None),
|
||||
"artist_id": artist_id,
|
||||
"artist_name": artist.similar_artist_name,
|
||||
"occurrence_count": artist.occurrence_count,
|
||||
|
|
@ -219,19 +203,11 @@ def get_discover_hero():
|
|||
artist.id, artist_data.get('image_url'),
|
||||
artist_data.get('genres'), artist_data.get('popularity')
|
||||
)
|
||||
elif active_source in ('itunes', 'deezer', 'musicbrainz'):
|
||||
if active_source == 'deezer':
|
||||
fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id
|
||||
fetch_client = itunes_client
|
||||
elif active_source == 'musicbrainz':
|
||||
fb_artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None)
|
||||
from core.metadata.registry import get_musicbrainz_client
|
||||
fetch_client = get_musicbrainz_client()
|
||||
else:
|
||||
fb_artist_id = artist.similar_artist_itunes_id
|
||||
fetch_client = itunes_client
|
||||
elif active_source in ('itunes', 'deezer'):
|
||||
fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) if active_source == 'deezer' else None
|
||||
fb_artist_id = fb_artist_id or artist.similar_artist_itunes_id
|
||||
if fb_artist_id:
|
||||
fb_artist_data = fetch_client.get_artist(fb_artist_id)
|
||||
fb_artist_data = itunes_client.get_artist(fb_artist_id)
|
||||
if fb_artist_data:
|
||||
artist_data['artist_name'] = fb_artist_data.get('name', artist.similar_artist_name)
|
||||
artist_data['image_url'] = fb_artist_data.get('images', [{}])[0].get('url') if fb_artist_data.get('images') else None
|
||||
|
|
|
|||
|
|
@ -1,340 +0,0 @@
|
|||
"""Listening-driven recommendation core (#913).
|
||||
|
||||
PURE, side-effect-free ranking that turns "the artists you listen to most" plus
|
||||
"who's similar to each" into:
|
||||
|
||||
1. a consensus-ranked list of artists you'd probably love but don't own, and
|
||||
2. an aggregated candidate-track list for a generated playlist.
|
||||
|
||||
No DB / network / config here. The caller (the watchlist scanner) supplies the
|
||||
seeds (top-played artists), the ``similar_artists`` rows per seed, and the
|
||||
owned-artist set, then fetches top tracks for the winners. Keeping the decision
|
||||
logic in one pure place makes it fully unit-testable without the live stack and
|
||||
keeps the scan wiring thin — and additive, so it can't disturb existing flows.
|
||||
|
||||
Scoring rationale (the "best in class" bit): a recommended artist's score is
|
||||
``Σ over the seeds that recommend it of (seed_weight × similarity)``. That single
|
||||
sum rewards all three signals at once — **consensus** (an artist endorsed by many
|
||||
of your seeds accumulates more terms), your **play weight** (heavier seeds push
|
||||
harder), and **similarity strength** — instead of a flat "appears in N lists".
|
||||
``seed_count`` is exposed separately for display ("because you like A, B, C") and
|
||||
as the adventurousness dial's lever (``min_seed_count``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Sequence, Set
|
||||
|
||||
|
||||
def _norm(name: object) -> str:
|
||||
return str(name or "").strip().lower()
|
||||
|
||||
|
||||
def _positive_float(value: object, default: float = 1.0) -> float:
|
||||
try:
|
||||
f = float(value) # type: ignore[arg-type]
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return f if f > 0 else default
|
||||
|
||||
|
||||
def _get(row: object, attr: str):
|
||||
"""Read a field from a dataclass row or a dict row."""
|
||||
if isinstance(row, dict):
|
||||
return row.get(attr)
|
||||
return getattr(row, attr, None)
|
||||
|
||||
|
||||
def choose_mix_fetch_source(active_source: object, active_can_fetch: bool) -> str:
|
||||
"""Pick which source to fetch the "Listening Mix" top tracks from.
|
||||
|
||||
The mix is a list of (artist, title) pairs acquired via Soulseek, so the fetch source need
|
||||
NOT match the user's active metadata source. Use the active source when it can fetch top
|
||||
tracks itself (Spotify/Deezer); otherwise fall back to Deezer, whose public ``artist/{id}/top``
|
||||
needs no auth and is available to every user — so iTunes / Discogs / MusicBrainz users still
|
||||
get a full mix without switching sources. Pure.
|
||||
"""
|
||||
if str(active_source or "").lower() in ("spotify", "deezer") and active_can_fetch:
|
||||
return str(active_source).lower()
|
||||
return "deezer"
|
||||
|
||||
|
||||
def names_match(a: object, b: object) -> bool:
|
||||
"""Strict artist-name equality after stripping case + non-alphanumerics.
|
||||
|
||||
Used to verify a name-search result before fetching that artist's top tracks, so the
|
||||
"Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact
|
||||
alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell".
|
||||
Pure.
|
||||
"""
|
||||
def _alnum(x: object) -> str:
|
||||
return "".join(ch for ch in str(x or "").lower() if ch.isalnum())
|
||||
na, nb = _alnum(a), _alnum(b)
|
||||
return bool(na) and na == nb
|
||||
|
||||
|
||||
def similarity_from_rank(rank: object, max_rank: int = 10) -> float:
|
||||
"""Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight.
|
||||
|
||||
SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest
|
||||
match). The ranker multiplies this into the score so a seed's *closest* matches count
|
||||
for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0,
|
||||
rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A
|
||||
missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure.
|
||||
"""
|
||||
try:
|
||||
r = int(rank)
|
||||
except (TypeError, ValueError):
|
||||
return 1.0
|
||||
floor = round(1.0 / max_rank, 4)
|
||||
if r <= 1:
|
||||
return 1.0
|
||||
if r >= max_rank:
|
||||
return floor
|
||||
return round((max_rank - r + 1) / max_rank, 4)
|
||||
|
||||
|
||||
def build_recency_weighted_seeds(
|
||||
top_artists: Sequence[dict],
|
||||
recent_play_counts: Optional[Dict[str, float]] = None,
|
||||
*,
|
||||
recency_factor: float = 1.5,
|
||||
) -> List[dict]:
|
||||
"""Blend lifetime + recent play counts into seed weights — "what you're into NOW".
|
||||
|
||||
``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a
|
||||
lot *recently* outranks one you played a lot years ago, so the recommendations track
|
||||
your current taste instead of your all-time history. ``recency_factor`` is the dial
|
||||
(0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`.
|
||||
Pure — the caller supplies both play-count maps from the listening history.
|
||||
"""
|
||||
recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()}
|
||||
out: List[dict] = []
|
||||
for a in top_artists or ():
|
||||
name = str(a.get("name") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0)))
|
||||
boost = recency_factor * recent.get(_norm(name), 0.0)
|
||||
out.append({"name": name, "weight": lifetime + boost})
|
||||
return out
|
||||
|
||||
|
||||
def group_similars_by_seed(
|
||||
seeds: Sequence[dict],
|
||||
similar_rows: Sequence,
|
||||
id_to_name: Dict[str, str],
|
||||
*,
|
||||
source_id_attr: str = "source_artist_id",
|
||||
similar_name_attr: str = "similar_artist_name",
|
||||
rank_attr: Optional[str] = None,
|
||||
) -> Dict[str, List[dict]]:
|
||||
"""Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``.
|
||||
|
||||
The stored rows key the similar artist by the SEED's source id (``source_artist_id``),
|
||||
not its name, so :func:`rank_recommended_artists` can't consume them directly. This
|
||||
resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id:
|
||||
artist_name}`` for the library, built by the caller) and keeps only rows that resolve
|
||||
to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O.
|
||||
|
||||
``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that
|
||||
is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id.
|
||||
When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank`
|
||||
and carried as ``score`` so closer matches weigh more; without it every similar comes out
|
||||
score-less (the ranker then treats similarity as 1.0 — original behavior).
|
||||
"""
|
||||
seed_names = {_norm(s.get("name")) for s in seeds}
|
||||
seed_names.discard("")
|
||||
id_to_norm = {str(k): _norm(v) for k, v in (id_to_name or {}).items()}
|
||||
|
||||
out: Dict[str, List[dict]] = {}
|
||||
for row in similar_rows or ():
|
||||
seed_name = id_to_norm.get(str(_get(row, source_id_attr) or ""), "")
|
||||
if not seed_name or seed_name not in seed_names:
|
||||
continue
|
||||
sim_name = str(_get(row, similar_name_attr) or "").strip()
|
||||
if not sim_name:
|
||||
continue
|
||||
entry = {"name": sim_name}
|
||||
if rank_attr is not None:
|
||||
entry["score"] = similarity_from_rank(_get(row, rank_attr))
|
||||
out.setdefault(seed_name, []).append(entry)
|
||||
return out
|
||||
|
||||
|
||||
@dataclass
|
||||
class RecommendedArtist:
|
||||
"""One artist recommended from your listening, with the why."""
|
||||
name: str # display name (first-seen casing)
|
||||
score: float # Σ seed_weight × similarity
|
||||
seed_count: int # distinct seeds endorsing it (consensus)
|
||||
seeds: List[str] = field(default_factory=list) # display names of those seeds
|
||||
|
||||
|
||||
def rank_recommended_artists(
|
||||
seeds: Sequence[dict],
|
||||
similars_by_seed: Dict[str, Sequence[dict]],
|
||||
owned_artist_names: Optional[Set[str]] = None,
|
||||
*,
|
||||
limit: int = 30,
|
||||
min_seed_count: int = 1,
|
||||
) -> List[RecommendedArtist]:
|
||||
"""Rank artists similar to your most-played by consensus + play weight + similarity.
|
||||
|
||||
Args:
|
||||
seeds: ``[{'name': str, 'weight': float}]`` — your top-played artists.
|
||||
``weight`` (play count or any positive number) defaults to 1.0.
|
||||
similars_by_seed: ``{seed_name_lower: [{'name': str, 'score': float}]}`` — the
|
||||
similar-artist rows for each seed. ``score`` is optional (defaults 1.0).
|
||||
owned_artist_names: lowercased names already in the library — excluded so the
|
||||
result is artists you DON'T have. The seeds themselves are always excluded.
|
||||
limit: max results.
|
||||
min_seed_count: drop recommendations endorsed by fewer than N seeds — the
|
||||
adventurousness dial's "Safer" end raises this for higher-confidence picks.
|
||||
|
||||
Returns up to ``limit`` :class:`RecommendedArtist`, highest score first.
|
||||
"""
|
||||
owned = {_norm(a) for a in (owned_artist_names or set())}
|
||||
seed_norms = {_norm(s.get("name")) for s in seeds}
|
||||
seed_norms.discard("")
|
||||
exclude = owned | seed_norms
|
||||
|
||||
acc: Dict[str, dict] = {}
|
||||
for seed in seeds:
|
||||
s_name = _norm(seed.get("name"))
|
||||
if not s_name:
|
||||
continue
|
||||
s_display = str(seed.get("name") or "").strip()
|
||||
weight = _positive_float(seed.get("weight", 1.0))
|
||||
for sim in similars_by_seed.get(s_name, ()) or ():
|
||||
a_norm = _norm(sim.get("name"))
|
||||
if not a_norm or a_norm in exclude:
|
||||
continue
|
||||
sim_score = _positive_float(sim.get("score", 1.0))
|
||||
row = acc.setdefault(
|
||||
a_norm, {"name": str(sim.get("name") or "").strip(), "score": 0.0, "seeds": {}}
|
||||
)
|
||||
row["score"] += weight * sim_score
|
||||
row["seeds"].setdefault(s_name, s_display) # one seed counts once
|
||||
|
||||
out: List[RecommendedArtist] = []
|
||||
floor = max(1, int(min_seed_count))
|
||||
for row in acc.values():
|
||||
seed_count = len(row["seeds"])
|
||||
if seed_count < floor:
|
||||
continue
|
||||
out.append(RecommendedArtist(
|
||||
name=row["name"],
|
||||
score=round(row["score"], 6),
|
||||
seed_count=seed_count,
|
||||
seeds=list(row["seeds"].values()),
|
||||
))
|
||||
out.sort(key=lambda r: (-r.score, -r.seed_count, r.name.lower()))
|
||||
return out[:limit]
|
||||
|
||||
|
||||
def aggregate_candidate_tracks(
|
||||
recommended_artists: Sequence[RecommendedArtist],
|
||||
top_tracks_by_artist: Dict[str, Sequence[dict]],
|
||||
owned_track_keys: Optional[Set] = None,
|
||||
*,
|
||||
per_artist: int = 3,
|
||||
limit: int = 50,
|
||||
exclude_owned: bool = True,
|
||||
) -> List[dict]:
|
||||
"""Build the candidate track list for the generated playlist.
|
||||
|
||||
Takes the top ``per_artist`` tracks from each recommended artist **in artist-rank
|
||||
order**, dedups by ``(artist, title)``, optionally drops owned tracks (the
|
||||
"discovery" flavor) and caps at ``limit``. Each returned track dict is the source
|
||||
track plus ``_seed_artist`` (which recommended artist it came from).
|
||||
|
||||
Args:
|
||||
recommended_artists: ranked output of :func:`rank_recommended_artists`.
|
||||
top_tracks_by_artist: ``{artist_name_lower: [track_dict, ...]}`` — fetched by
|
||||
the caller (Last.fm / source top tracks), NOT limited to a curated pool.
|
||||
owned_track_keys: set of ``(artist_lower, title_lower)`` already in the library.
|
||||
exclude_owned: drop tracks in ``owned_track_keys`` (discovery flavor). Set False
|
||||
for a "replay" playlist of tracks you already own.
|
||||
"""
|
||||
owned = owned_track_keys or set()
|
||||
seen: Set = set()
|
||||
out: List[dict] = []
|
||||
for art in recommended_artists:
|
||||
tracks = top_tracks_by_artist.get(_norm(art.name), ()) or ()
|
||||
taken = 0
|
||||
for t in tracks:
|
||||
if taken >= per_artist:
|
||||
break
|
||||
title = str(t.get("name") or t.get("title") or "").strip()
|
||||
if not title:
|
||||
continue
|
||||
key = (_norm(art.name), _norm(title))
|
||||
if key in seen:
|
||||
continue
|
||||
if exclude_owned and key in owned:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append({**t, "_seed_artist": art.name})
|
||||
taken += 1
|
||||
if len(out) >= limit:
|
||||
break
|
||||
return out[:limit]
|
||||
|
||||
|
||||
def to_mix_track(track: object, source: str) -> Optional[dict]:
|
||||
"""Shape one source "top tracks" API dict into the flat dict the Discover compact
|
||||
playlist row renders + syncs (the "Listening Mix" #913 playlist).
|
||||
|
||||
Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the
|
||||
same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``).
|
||||
This flattens that into the renderer's field names (``track_name/artist_name/album_name/
|
||||
album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and
|
||||
sets the source-specific id field. Returns None for anything without a usable id/title so
|
||||
the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can
|
||||
dedup by title. Pure — no I/O.
|
||||
"""
|
||||
if not isinstance(track, dict):
|
||||
return None
|
||||
tid = track.get("id")
|
||||
name = str(track.get("name") or "").strip()
|
||||
if not tid or not name:
|
||||
return None
|
||||
artists = track.get("artists") or []
|
||||
artist_name = ""
|
||||
if artists and isinstance(artists[0], dict):
|
||||
artist_name = str(artists[0].get("name") or "").strip()
|
||||
album = track.get("album") if isinstance(track.get("album"), dict) else {}
|
||||
album_name = str(album.get("name") or "").strip()
|
||||
images = album.get("images") or []
|
||||
cover = images[0].get("url") if images and isinstance(images[0], dict) else None
|
||||
out = {
|
||||
"track_id": str(tid),
|
||||
"name": name, # for aggregate_candidate_tracks dedup
|
||||
"track_name": name, # for the renderer
|
||||
"artist_name": artist_name,
|
||||
"album_name": album_name,
|
||||
"album_cover_url": cover,
|
||||
"duration_ms": track.get("duration_ms") or 0,
|
||||
"track_data_json": track, # full payload for sync/download
|
||||
"source": source,
|
||||
}
|
||||
id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id",
|
||||
"itunes": "itunes_track_id"}.get(source)
|
||||
if id_field:
|
||||
out[id_field] = str(tid)
|
||||
return out
|
||||
|
||||
|
||||
__all__ = [
|
||||
"RecommendedArtist",
|
||||
"choose_mix_fetch_source",
|
||||
"names_match",
|
||||
"similarity_from_rank",
|
||||
"build_recency_weighted_seeds",
|
||||
"to_mix_track",
|
||||
"group_similars_by_seed",
|
||||
"rank_recommended_artists",
|
||||
"aggregate_candidate_tracks",
|
||||
]
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
"""Helpers for Fix-popup manual match persistence.
|
||||
|
||||
When the user manually fixes a mirrored-playlist discovery via the Fix
|
||||
popup, two questions land at the web_server route layer that are easier
|
||||
to test in isolation:
|
||||
|
||||
1. *Which metadata source did the manual match come from?* — the popup
|
||||
cascade queries the user's primary source first, then Spotify /
|
||||
Deezer / iTunes / MusicBrainz as fallbacks; each search endpoint
|
||||
stamps `source` on its rows but the MBID-paste lookup uses a lean
|
||||
flat shape that doesn't carry it. `derive_manual_match_provider`
|
||||
collapses the fallback chain into a single string.
|
||||
|
||||
2. *Should the discovery layer re-run for this track when the current
|
||||
active provider differs from the cached one?* — re-running silently
|
||||
overwrites the user's deliberate pick with whatever the auto-search
|
||||
ranks first, so manual matches are exempt regardless of provider
|
||||
drift. `is_drifted_for_redo` encapsulates the decision.
|
||||
|
||||
3. *Should the Playlist Pipeline pre-scan (re)discover this track at all?*
|
||||
— `should_rediscover` encapsulates that gate, with the manual match
|
||||
checked FIRST so a leftover Wing It flag can't override the user's pick.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
def derive_manual_match_provider(
|
||||
payload_track: Dict[str, Any],
|
||||
active_provider: Optional[str],
|
||||
) -> str:
|
||||
"""Return the provider string to stamp on a manually-fixed match.
|
||||
|
||||
Resolution order:
|
||||
1. ``payload_track['source']`` — every *_search_tracks endpoint
|
||||
sets this; the MBID-paste path doesn't.
|
||||
2. ``active_provider`` — what the user has configured as their
|
||||
primary discovery source.
|
||||
3. ``'spotify'`` — last-ditch default matching the historic
|
||||
hardcode (so behaviour is identical when both upstream
|
||||
signals are absent).
|
||||
"""
|
||||
if not isinstance(payload_track, dict):
|
||||
payload_track = {}
|
||||
source = payload_track.get('source')
|
||||
if source:
|
||||
return source
|
||||
if active_provider:
|
||||
return active_provider
|
||||
return 'spotify'
|
||||
|
||||
|
||||
def is_drifted_for_redo(
|
||||
extra_data: Optional[Dict[str, Any]],
|
||||
active_provider: Optional[str],
|
||||
) -> bool:
|
||||
"""Return True when a cached discovery entry should be treated as
|
||||
stale because the user's active provider has changed since it was
|
||||
cached AND the entry isn't a manual match.
|
||||
|
||||
Manual matches are *always* considered fresh: re-running discovery
|
||||
against the current source would overwrite the user's deliberate
|
||||
pick with whatever auto-search ranks first. The first Playlist
|
||||
Pipeline run after a manual fix used to clobber it for exactly
|
||||
this reason — the check lives here now so it's pinned by tests.
|
||||
"""
|
||||
if not isinstance(extra_data, dict):
|
||||
return False
|
||||
if extra_data.get('manual_match'):
|
||||
return False
|
||||
cached_provider = extra_data.get('provider', 'spotify')
|
||||
return cached_provider != active_provider
|
||||
|
||||
|
||||
def should_rediscover(extra_data: Optional[Dict[str, Any]]) -> bool:
|
||||
"""Return True when a mirrored track needs (re)discovery, False to skip it.
|
||||
|
||||
This is the gate the Playlist Pipeline pre-scan runs over every mirrored
|
||||
track before discovering. The **ordering is the fix**: a manual match is
|
||||
authoritative and is checked FIRST.
|
||||
|
||||
``extra_data`` is *merged* on save (see ``update_mirrored_track_extra_data``),
|
||||
so a track that was a Wing It stub and is then manually fixed still carries
|
||||
``wing_it_fallback: True`` alongside the new ``manual_match: True``. The old
|
||||
pre-scan tested ``wing_it_fallback`` before ``manual_match``, so the stale
|
||||
flag won and the pipeline re-discovered the track — silently reverting the
|
||||
user's pick to Wing It. Checking ``manual_match`` first makes the fix stick.
|
||||
|
||||
Decision order:
|
||||
* manual_match -> skip (authoritative; never re-discover)
|
||||
* wing_it_fallback -> redo (stub — keep trying for a real match)
|
||||
* discovered + complete -> skip (full metadata already stored)
|
||||
* discovered + incomplete -> redo (backfill track_number / album fields)
|
||||
* unmatched_by_user -> skip (user deliberately removed the match)
|
||||
* never discovered -> redo (first-time discovery)
|
||||
"""
|
||||
extra = extra_data if isinstance(extra_data, dict) else {}
|
||||
|
||||
if extra.get('discovered'):
|
||||
if extra.get('manual_match'):
|
||||
return False
|
||||
if extra.get('wing_it_fallback'):
|
||||
return True
|
||||
# Otherwise re-discover only when the stored match is missing the
|
||||
# enriched fields (track_number + release_date/album.id) that older
|
||||
# discoveries dropped via the Track dataclass.
|
||||
matched = extra.get('matched_data')
|
||||
matched = matched if isinstance(matched, dict) else {}
|
||||
album = matched.get('album')
|
||||
album = album if isinstance(album, dict) else {}
|
||||
has_track_num = matched.get('track_number')
|
||||
has_release = album.get('release_date')
|
||||
has_album_id = album.get('id')
|
||||
return not (has_track_num and (has_release or has_album_id))
|
||||
|
||||
if extra.get('unmatched_by_user'):
|
||||
return False
|
||||
return True
|
||||
|
|
@ -1,134 +0,0 @@
|
|||
"""Pure helper for matching raw MusicBrainz-metadata tracks against
|
||||
Spotify / iTunes.
|
||||
|
||||
Used by the PlaylistSource adapters whose ``get_playlist`` returns
|
||||
tracks with ``needs_discovery=True`` (ListenBrainz, Last.fm radio).
|
||||
Phase 1b ships Strategy 1 only (matching-engine queries → search →
|
||||
score → pick best ≥0.9). The richer multi-strategy +
|
||||
discovery-cache flow stays in
|
||||
``core.discovery.listenbrainz.run_listenbrainz_discovery_worker``
|
||||
for the Discover-page state-machine UI; this helper is the slimmer
|
||||
version used by the auto-refresh pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MBMatchDeps:
|
||||
"""Bundle of primitives the matcher needs.
|
||||
|
||||
Wired up at bootstrap. Tests pass stub callables / clients."""
|
||||
|
||||
matching_engine: Any
|
||||
score_candidates: Callable[..., Any]
|
||||
spotify_client_getter: Callable[[], Any]
|
||||
itunes_client_getter: Callable[[], Any]
|
||||
prefer_spotify_getter: Callable[[], bool]
|
||||
min_confidence: float = 0.9
|
||||
|
||||
|
||||
def match_mb_track(
|
||||
track: Dict[str, Any], deps: MBMatchDeps
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""Try to match a single MB-metadata track.
|
||||
|
||||
Input shape:
|
||||
``{'track_name', 'artist_name', 'album_name', 'duration_ms'}``
|
||||
|
||||
Returns the matched_data dict (Spotify/iTunes track projection)
|
||||
or ``None`` when no candidate cleared the confidence threshold.
|
||||
"""
|
||||
title = track.get("track_name") or ""
|
||||
artist = track.get("artist_name") or ""
|
||||
album = track.get("album_name") or ""
|
||||
duration_ms = int(track.get("duration_ms") or 0)
|
||||
if not title or not artist:
|
||||
return None
|
||||
|
||||
spotify_client = deps.spotify_client_getter()
|
||||
itunes_client = deps.itunes_client_getter()
|
||||
use_spotify = bool(
|
||||
deps.prefer_spotify_getter()
|
||||
and spotify_client is not None
|
||||
and getattr(spotify_client, "is_spotify_authenticated", lambda: False)()
|
||||
)
|
||||
if not use_spotify and itunes_client is None:
|
||||
return None
|
||||
|
||||
# Strategy 1 — matching-engine query generation.
|
||||
try:
|
||||
temp_track = type("_TempTrack", (), {
|
||||
"name": title,
|
||||
"artists": [artist],
|
||||
"album": album or None,
|
||||
})()
|
||||
queries = deps.matching_engine.generate_download_queries(temp_track)
|
||||
except Exception as exc:
|
||||
logger.debug(f"matching_engine query-gen failed: {exc}")
|
||||
queries = [f"{artist} {title}", title]
|
||||
|
||||
best_match: Any = None
|
||||
best_confidence = 0.0
|
||||
for query in queries:
|
||||
try:
|
||||
if use_spotify:
|
||||
results = spotify_client.search_tracks(query, limit=10)
|
||||
else:
|
||||
results = itunes_client.search_tracks(query, limit=10)
|
||||
except Exception as exc:
|
||||
logger.debug(f"search failed for query={query!r}: {exc}")
|
||||
continue
|
||||
if not results:
|
||||
continue
|
||||
try:
|
||||
match, confidence, _ = deps.score_candidates(
|
||||
title, artist, duration_ms, results
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug(f"score_candidates failed: {exc}")
|
||||
continue
|
||||
if match and confidence > best_confidence and confidence >= deps.min_confidence:
|
||||
best_match = match
|
||||
best_confidence = confidence
|
||||
if best_confidence >= deps.min_confidence:
|
||||
break
|
||||
|
||||
if not best_match:
|
||||
return None
|
||||
|
||||
provider = "spotify" if use_spotify else "itunes"
|
||||
image_url = getattr(best_match, "image_url", None) or ""
|
||||
album_data: Dict[str, Any] = {
|
||||
"name": getattr(best_match, "album", "") or "",
|
||||
}
|
||||
if image_url:
|
||||
album_data["images"] = [{"url": image_url}]
|
||||
return {
|
||||
"id": getattr(best_match, "id", "") or "",
|
||||
"name": getattr(best_match, "name", "") or "",
|
||||
"artists": list(getattr(best_match, "artists", []) or []),
|
||||
"album": album_data,
|
||||
"duration_ms": int(getattr(best_match, "duration_ms", 0) or 0),
|
||||
"image_url": image_url,
|
||||
"source": provider,
|
||||
"_provider": provider,
|
||||
"_confidence": float(best_confidence),
|
||||
}
|
||||
|
||||
|
||||
def match_mb_tracks(
|
||||
tracks: List[Dict[str, Any]], deps: MBMatchDeps
|
||||
) -> List[Optional[Dict[str, Any]]]:
|
||||
"""Vectorized variant — runs ``match_mb_track`` per track.
|
||||
|
||||
Phase 1b is sequential. If profiling shows it's too slow on big
|
||||
LB playlists, this becomes the natural spot to thread-pool the
|
||||
per-track searches."""
|
||||
return [match_mb_track(t, deps) for t in tracks]
|
||||
|
|
@ -37,35 +37,9 @@ import time
|
|||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
from core.discovery.manual_match import should_rediscover
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _canonical_best_score(deps, title, artist, duration_ms, results):
|
||||
"""Score search results against the source track, trying the canonicalized
|
||||
title/artist too and keeping the better confidence (#785).
|
||||
|
||||
YouTube playlists have their "Artist - Title" / channel decoration stripped
|
||||
at ingest, but file/CSV-imported playlists keep raw titles — so a track
|
||||
titled "Arctic Monkeys - Do I Wanna Know?" scored verbatim against the
|
||||
library's "Do I Wanna Know?" never matched. canonical_source_track is
|
||||
conservative (only strips an "<artist> - " prefix when it equals the
|
||||
artist), so this can only ADD a better candidate, never weaken a match.
|
||||
Returns (match, confidence)."""
|
||||
match, confidence, _ = deps.discovery_score_candidates(title, artist, duration_ms, results)
|
||||
try:
|
||||
from core.text.source_title import canonical_source_track
|
||||
canon_title, canon_artist = canonical_source_track(title or '', artist or '')
|
||||
except Exception:
|
||||
return match, confidence
|
||||
if (canon_title, canon_artist) != (title, artist):
|
||||
alt_match, alt_conf, _ = deps.discovery_score_candidates(canon_title, canon_artist, duration_ms, results)
|
||||
if alt_match and alt_conf > confidence:
|
||||
return alt_match, alt_conf
|
||||
return match, confidence
|
||||
|
||||
|
||||
@dataclass
|
||||
class PlaylistDiscoveryDeps:
|
||||
"""Bundle of cross-cutting deps the playlist discovery worker needs."""
|
||||
|
|
@ -147,14 +121,31 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
existing_extra = json.loads(track['extra_data']) if isinstance(track['extra_data'], str) else track['extra_data']
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
# `should_rediscover` is the single source of truth for this
|
||||
# gate (manual match checked FIRST so a stale Wing It flag can't
|
||||
# revert a user's deliberate fix — see its docstring).
|
||||
if should_rediscover(existing_extra):
|
||||
undiscovered_tracks.append(track)
|
||||
else:
|
||||
if existing_extra.get('discovered'):
|
||||
if existing_extra.get('wing_it_fallback'):
|
||||
# Wing It stub — always re-attempt to find a real match
|
||||
undiscovered_tracks.append(track)
|
||||
else:
|
||||
# Check if matched_data is complete — old discoveries may be missing
|
||||
# track_number/release_date due to the Track dataclass stripping them.
|
||||
# Re-discover these so the enriched pipeline fills in the gaps.
|
||||
md = existing_extra.get('matched_data', {})
|
||||
album = md.get('album', {})
|
||||
has_track_num = md.get('track_number')
|
||||
has_release = album.get('release_date') if isinstance(album, dict) else None
|
||||
has_album_id = album.get('id') if isinstance(album, dict) else None
|
||||
if has_track_num and (has_release or has_album_id):
|
||||
pl_skipped += 1
|
||||
total_skipped += 1
|
||||
else:
|
||||
# Incomplete discovery — re-discover to get full metadata
|
||||
undiscovered_tracks.append(track)
|
||||
elif existing_extra.get('unmatched_by_user'):
|
||||
# User explicitly removed this match — respect their choice
|
||||
pl_skipped += 1
|
||||
total_skipped += 1
|
||||
else:
|
||||
undiscovered_tracks.append(track)
|
||||
|
||||
if pl_skipped > 0:
|
||||
deps.update_automation_progress(automation_id,
|
||||
|
|
@ -219,20 +210,6 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
except Exception:
|
||||
search_queries = [f"{artist_name} {track_name}", track_name]
|
||||
|
||||
# #785: file/CSV playlists keep raw "Artist - Title" titles, so the
|
||||
# queries above search for the artist prefix too. Also search the
|
||||
# canonicalized title so the right candidates are actually returned
|
||||
# (the scorer best-of then matches them).
|
||||
try:
|
||||
from core.text.source_title import canonical_source_track
|
||||
_cq_title, _cq_artist = canonical_source_track(track_name, artist_name)
|
||||
if (_cq_title, _cq_artist) != (track_name, artist_name):
|
||||
for _q in (f"{_cq_artist} {_cq_title}", _cq_title):
|
||||
if _q not in search_queries:
|
||||
search_queries.append(_q)
|
||||
except Exception as _cq_err:
|
||||
logger.debug("canonical search-query add failed: %s", _cq_err)
|
||||
|
||||
# Step 3: Search and score
|
||||
best_match = None
|
||||
best_confidence = 0.0
|
||||
|
|
@ -247,8 +224,8 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
if not results:
|
||||
continue
|
||||
|
||||
match, confidence = _canonical_best_score(
|
||||
deps, track_name, artist_name, duration_ms, results
|
||||
match, confidence, _ = deps.discovery_score_candidates(
|
||||
track_name, artist_name, duration_ms, results
|
||||
)
|
||||
|
||||
if match and confidence > best_confidence:
|
||||
|
|
@ -269,8 +246,8 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
else:
|
||||
extended = itunes_client_instance.search_tracks(query, limit=50)
|
||||
if extended:
|
||||
match, confidence = _canonical_best_score(
|
||||
deps, track_name, artist_name, duration_ms, extended
|
||||
match, confidence, _ = deps.discovery_score_candidates(
|
||||
track_name, artist_name, duration_ms, extended
|
||||
)
|
||||
if match and confidence > best_confidence:
|
||||
best_confidence = confidence
|
||||
|
|
@ -316,18 +293,6 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
except Exception as e:
|
||||
logger.debug("metadata cache lookup for album enrichment failed: %s", e)
|
||||
|
||||
# Always include ``track_number`` / ``disc_number``
|
||||
# in the matched payload — None when unknown rather
|
||||
# than omitting the key. Downstream consumers
|
||||
# (``ensure_wishlist_track_format``, post-process
|
||||
# pipeline) check for None to know "look this up
|
||||
# somewhere else"; an absent key was indistinguishable
|
||||
# from "value is 1" after older payload helpers
|
||||
# silently filled the default. Pre-fix Deezer-sourced
|
||||
# matches always omitted the key (Deezer's track shape
|
||||
# uses ``track_position`` and the cache lookup at
|
||||
# line 304 reads ``track_number`` literally so it
|
||||
# returns None for Deezer rows).
|
||||
matched_data = {
|
||||
'id': best_match.id if hasattr(best_match, 'id') else '',
|
||||
'name': best_match.name if hasattr(best_match, 'name') else '',
|
||||
|
|
@ -336,13 +301,11 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
|
|||
'duration_ms': best_match.duration_ms if hasattr(best_match, 'duration_ms') else 0,
|
||||
'image_url': match_image,
|
||||
'source': discovery_source,
|
||||
'track_number': track_number if track_number else (
|
||||
getattr(best_match, 'track_number', None)
|
||||
),
|
||||
'disc_number': disc_number if disc_number else (
|
||||
getattr(best_match, 'disc_number', None)
|
||||
),
|
||||
}
|
||||
if track_number:
|
||||
matched_data['track_number'] = track_number
|
||||
if disc_number:
|
||||
matched_data['disc_number'] = disc_number
|
||||
|
||||
extra_data = {
|
||||
'discovered': True,
|
||||
|
|
|
|||
|
|
@ -1,299 +0,0 @@
|
|||
"""Background worker for Qobuz playlist discovery.
|
||||
|
||||
`run_qobuz_discovery_worker(playlist_id, deps)` is the function the
|
||||
Qobuz discovery start-endpoint submits to its executor to match each
|
||||
Qobuz playlist track against Spotify (preferred) or the configured
|
||||
fallback metadata source (iTunes / Deezer / Discogs / MusicBrainz).
|
||||
|
||||
Mirrors `core/discovery/deezer.py` exactly — Qobuz playlists arrive as
|
||||
dicts (not dataclasses) from `core/qobuz_client.py:get_playlist`, so
|
||||
this worker uses dict-style access on track data and wraps each entry
|
||||
in a SimpleNamespace before handing it to the shared
|
||||
`_search_spotify_for_tidal_track` helper.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
import types
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class QobuzDiscoveryDeps:
|
||||
"""Bundle of cross-cutting deps the Qobuz discovery worker needs."""
|
||||
qobuz_discovery_states: dict
|
||||
spotify_client: Any
|
||||
pause_enrichment_workers: Callable[[str], dict]
|
||||
resume_enrichment_workers: Callable[[dict, str], None]
|
||||
get_active_discovery_source: Callable[[], str]
|
||||
get_metadata_fallback_client: Callable[[], Any]
|
||||
get_discovery_cache_key: Callable
|
||||
get_database: Callable[[], Any]
|
||||
validate_discovery_cache_artist: Callable
|
||||
search_spotify_for_tidal_track: Callable
|
||||
build_discovery_wing_it_stub: Callable
|
||||
add_activity_item: Callable
|
||||
sync_discovery_results_to_mirrored: Callable
|
||||
|
||||
|
||||
def run_qobuz_discovery_worker(playlist_id, deps: QobuzDiscoveryDeps):
|
||||
"""Background worker for Qobuz discovery process (Spotify preferred, fallback metadata source)."""
|
||||
_ew_state = {}
|
||||
try:
|
||||
_ew_state = deps.pause_enrichment_workers('Qobuz discovery')
|
||||
state = deps.qobuz_discovery_states[playlist_id]
|
||||
playlist = state['playlist']
|
||||
|
||||
# Determine which provider to use
|
||||
discovery_source = deps.get_active_discovery_source()
|
||||
use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
|
||||
|
||||
# Initialize fallback client if needed
|
||||
itunes_client_instance = None
|
||||
if not use_spotify:
|
||||
itunes_client_instance = deps.get_metadata_fallback_client()
|
||||
|
||||
logger.info(f"Starting Qobuz discovery for: {playlist['name']} (using {discovery_source.upper()})")
|
||||
|
||||
# Store discovery source in state for frontend
|
||||
state['discovery_source'] = discovery_source
|
||||
|
||||
successful_discoveries = 0
|
||||
tracks = playlist['tracks']
|
||||
|
||||
for i, qobuz_track in enumerate(tracks):
|
||||
if state.get('cancelled', False):
|
||||
break
|
||||
|
||||
try:
|
||||
track_name = qobuz_track['name']
|
||||
track_artists = qobuz_track['artists']
|
||||
track_id = qobuz_track['id']
|
||||
track_album = qobuz_track.get('album', '')
|
||||
track_duration_ms = qobuz_track.get('duration_ms', 0)
|
||||
|
||||
logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
|
||||
|
||||
# Check discovery cache first
|
||||
cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
|
||||
try:
|
||||
cache_db = deps.get_database()
|
||||
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
|
||||
if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
|
||||
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
|
||||
cached_artists = cached_match.get('artists', [])
|
||||
if cached_artists:
|
||||
cached_artist_str = ', '.join(
|
||||
a if isinstance(a, str) else a.get('name', '') for a in cached_artists
|
||||
)
|
||||
else:
|
||||
cached_artist_str = ''
|
||||
cached_album = cached_match.get('album', '')
|
||||
if isinstance(cached_album, dict):
|
||||
cached_album = cached_album.get('name', '')
|
||||
|
||||
result = {
|
||||
'qobuz_track': {
|
||||
'id': track_id,
|
||||
'name': track_name,
|
||||
'artists': track_artists or [],
|
||||
'album': track_album,
|
||||
'duration_ms': track_duration_ms,
|
||||
},
|
||||
'spotify_data': cached_match,
|
||||
'match_data': cached_match,
|
||||
'status': 'Found',
|
||||
'status_class': 'found',
|
||||
'spotify_track': cached_match.get('name', ''),
|
||||
'spotify_artist': cached_artist_str,
|
||||
'spotify_album': cached_album,
|
||||
'spotify_id': cached_match.get('id', ''),
|
||||
'discovery_source': discovery_source,
|
||||
'index': i
|
||||
}
|
||||
successful_discoveries += 1
|
||||
state['spotify_matches'] = successful_discoveries
|
||||
state['discovery_results'].append(result)
|
||||
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
|
||||
continue
|
||||
except Exception as cache_err:
|
||||
logger.error(f"Cache lookup error: {cache_err}")
|
||||
|
||||
# SimpleNamespace duck-type for _search_spotify_for_tidal_track
|
||||
track_ns = types.SimpleNamespace(
|
||||
id=track_id,
|
||||
name=track_name,
|
||||
artists=track_artists,
|
||||
album=track_album,
|
||||
duration_ms=track_duration_ms
|
||||
)
|
||||
|
||||
track_result = deps.search_spotify_for_tidal_track(
|
||||
track_ns,
|
||||
use_spotify=use_spotify,
|
||||
itunes_client=itunes_client_instance
|
||||
)
|
||||
|
||||
result = {
|
||||
'qobuz_track': {
|
||||
'id': track_id,
|
||||
'name': track_name,
|
||||
'artists': track_artists or [],
|
||||
'album': track_album,
|
||||
'duration_ms': track_duration_ms,
|
||||
},
|
||||
'spotify_data': None,
|
||||
'match_data': None,
|
||||
'status': 'Not Found',
|
||||
'status_class': 'not-found',
|
||||
'spotify_track': '',
|
||||
'spotify_artist': '',
|
||||
'spotify_album': '',
|
||||
'discovery_source': discovery_source
|
||||
}
|
||||
|
||||
match_confidence = 0.0
|
||||
|
||||
if use_spotify and isinstance(track_result, tuple):
|
||||
track_obj, raw_track_data, match_confidence = track_result
|
||||
album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
|
||||
if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
|
||||
album_obj['name'] = track_obj.album
|
||||
elif not album_obj and track_obj.album:
|
||||
album_obj = {'name': track_obj.album}
|
||||
if isinstance(album_obj, dict) and not album_obj.get('release_date'):
|
||||
album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
|
||||
_album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
|
||||
_image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
|
||||
|
||||
match_data = {
|
||||
'id': track_obj.id,
|
||||
'name': track_obj.name,
|
||||
'artists': track_obj.artists,
|
||||
'album': album_obj,
|
||||
'duration_ms': track_obj.duration_ms,
|
||||
'external_urls': track_obj.external_urls,
|
||||
'image_url': _image_url,
|
||||
'source': 'spotify'
|
||||
}
|
||||
if raw_track_data and raw_track_data.get('track_number'):
|
||||
match_data['track_number'] = raw_track_data['track_number']
|
||||
if raw_track_data and raw_track_data.get('disc_number'):
|
||||
match_data['disc_number'] = raw_track_data['disc_number']
|
||||
result['spotify_data'] = match_data
|
||||
result['match_data'] = match_data
|
||||
result['status'] = 'Found'
|
||||
result['status_class'] = 'found'
|
||||
result['spotify_track'] = track_obj.name
|
||||
result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
|
||||
result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
|
||||
result['spotify_id'] = track_obj.id
|
||||
result['confidence'] = match_confidence
|
||||
successful_discoveries += 1
|
||||
state['spotify_matches'] = successful_discoveries
|
||||
|
||||
elif not use_spotify and track_result and isinstance(track_result, dict):
|
||||
match_confidence = track_result.pop('confidence', 0.80)
|
||||
match_data = track_result
|
||||
match_data['source'] = discovery_source
|
||||
_fb_album = match_data.get('album', {})
|
||||
_fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
|
||||
if _fb_images and 'image_url' not in match_data:
|
||||
match_data['image_url'] = _fb_images[0].get('url', '')
|
||||
result['spotify_data'] = match_data
|
||||
result['match_data'] = match_data
|
||||
result['status'] = 'Found'
|
||||
result['status_class'] = 'found'
|
||||
result['spotify_track'] = match_data.get('name', '')
|
||||
itunes_artists = match_data.get('artists', [])
|
||||
result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
|
||||
result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
|
||||
result['spotify_id'] = match_data.get('id', '')
|
||||
result['confidence'] = match_confidence
|
||||
successful_discoveries += 1
|
||||
state['spotify_matches'] = successful_discoveries
|
||||
|
||||
# Save to discovery cache if match found
|
||||
if result['status_class'] == 'found' and result.get('match_data'):
|
||||
try:
|
||||
cache_db = deps.get_database()
|
||||
cache_db.save_discovery_cache_match(
|
||||
cache_key[0], cache_key[1], discovery_source, match_confidence,
|
||||
result['match_data'], track_name,
|
||||
track_artists[0] if track_artists else ''
|
||||
)
|
||||
logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
|
||||
except Exception as cache_err:
|
||||
logger.error(f"Cache save error: {cache_err}")
|
||||
|
||||
# Auto Wing It fallback for unmatched tracks
|
||||
if result['status_class'] == 'not-found':
|
||||
qobuz_t = result.get('qobuz_track', {})
|
||||
stub = deps.build_discovery_wing_it_stub(
|
||||
qobuz_t.get('name', ''),
|
||||
', '.join(qobuz_t.get('artists', [])),
|
||||
qobuz_t.get('duration_ms', 0)
|
||||
)
|
||||
result['status'] = 'Wing It'
|
||||
result['status_class'] = 'wing-it'
|
||||
result['spotify_data'] = stub
|
||||
result['match_data'] = stub
|
||||
result['spotify_track'] = qobuz_t.get('name', '')
|
||||
result['spotify_artist'] = ', '.join(qobuz_t.get('artists', []))
|
||||
result['wing_it_fallback'] = True
|
||||
result['confidence'] = 0
|
||||
successful_discoveries += 1
|
||||
state['spotify_matches'] = successful_discoveries
|
||||
state['wing_it_count'] = state.get('wing_it_count', 0) + 1
|
||||
|
||||
result['index'] = i
|
||||
state['discovery_results'].append(result)
|
||||
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
|
||||
|
||||
time.sleep(0.1)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing track {i+1}: {e}")
|
||||
result = {
|
||||
'qobuz_track': {
|
||||
'name': qobuz_track.get('name', 'Unknown'),
|
||||
'artists': qobuz_track.get('artists', []),
|
||||
},
|
||||
'spotify_data': None,
|
||||
'match_data': None,
|
||||
'status': 'Error',
|
||||
'status_class': 'error',
|
||||
'spotify_track': '',
|
||||
'spotify_artist': '',
|
||||
'spotify_album': '',
|
||||
'error': str(e),
|
||||
'discovery_source': discovery_source,
|
||||
'index': i
|
||||
}
|
||||
state['discovery_results'].append(result)
|
||||
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
|
||||
|
||||
# Mark as complete
|
||||
state['phase'] = 'discovered'
|
||||
state['status'] = 'discovered'
|
||||
state['discovery_progress'] = 100
|
||||
|
||||
source_label = discovery_source.upper()
|
||||
deps.add_activity_item("", f"Qobuz Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
|
||||
|
||||
logger.info(f"Qobuz discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
|
||||
|
||||
deps.sync_discovery_results_to_mirrored('qobuz', playlist_id, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in Qobuz discovery worker: {e}")
|
||||
if playlist_id in deps.qobuz_discovery_states:
|
||||
deps.qobuz_discovery_states[playlist_id]['phase'] = 'error'
|
||||
deps.qobuz_discovery_states[playlist_id]['status'] = f'error: {str(e)}'
|
||||
finally:
|
||||
deps.resume_enrichment_workers(_ew_state, 'Qobuz discovery')
|
||||
|
|
@ -1,31 +1,44 @@
|
|||
"""Shared metadata match + result-normalization helpers for quality matching.
|
||||
"""Background worker for the library quality scanner.
|
||||
|
||||
These were the matching guts of the old auto-acting quality-scanner worker (now
|
||||
removed — quality scanning is the ``quality_upgrade`` library-maintenance repair
|
||||
job in ``core/repair_jobs/quality_upgrade.py``). They're kept here as a single
|
||||
source of truth and imported by that job:
|
||||
`run_quality_scanner(scope, profile_id, deps)` is the function the
|
||||
quality-scanner endpoint kicks off in a thread to scan the library
|
||||
for low-quality tracks (below the user's configured quality profile)
|
||||
and add provider matches to the wishlist:
|
||||
|
||||
- ``_search_tracks_for_source`` — query one metadata source's ``search_tracks``.
|
||||
- ``_normalize_track_match`` / ``_normalize_track_album`` / ``_normalize_track_artists``
|
||||
— turn a provider track into the wishlist-ready dict (typed Album converters
|
||||
with legacy duck-typed fallback).
|
||||
- ``_track_name`` / ``_track_artist_names`` / ``_extract_lookup_value`` — accessors.
|
||||
1. Reset scanner state, load quality profile + minimum acceptable tier.
|
||||
2. Load tracks from DB based on scope:
|
||||
- 'watchlist' → tracks for watchlisted artists only.
|
||||
- other → all library tracks.
|
||||
3. For each track:
|
||||
- Stop-request gate (state['status'] != 'running').
|
||||
- Quality-tier check via _get_quality_tier_from_extension(file_path).
|
||||
- Skip tracks meeting standards (tier_num <= min_acceptable_tier).
|
||||
- For low-quality tracks: matching_engine search query gen, score
|
||||
candidates against the configured metadata source priority
|
||||
(artist + title similarity, album-type bonus), pick best match >=
|
||||
0.7 confidence.
|
||||
- On match: add normalized track data to wishlist via
|
||||
`wishlist_service.add_track_to_wishlist` with
|
||||
source_type='quality_scanner' and a source_context that captures
|
||||
original file_path, format tier, bitrate, and match confidence.
|
||||
4. After all tracks: status='finished', progress=100, activity feed
|
||||
entry, emit `quality_scan_completed` event for automation engine.
|
||||
5. On critical exception: status='error', error message captured.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
from core.metadata.registry import get_client_for_source
|
||||
from core.metadata.registry import get_client_for_source, get_primary_source, get_source_priority
|
||||
from core.metadata.types import Album
|
||||
from core.wishlist.payloads import ensure_wishlist_track_format
|
||||
|
||||
# Use the project logger namespace ("soulsync.*") so the scanner's progress and
|
||||
# diagnostics actually surface in the app log — plain getLogger(__name__) lands
|
||||
# under "core.discovery.quality_scanner", which the app log view doesn't show.
|
||||
from utils.logging_config import get_logger
|
||||
logger = get_logger("discovery.quality_scanner")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Per-source typed converter dispatch — same registry pattern as
|
||||
|
|
@ -43,6 +56,16 @@ _TYPED_ALBUM_CONVERTERS: Dict[str, Callable[[Dict[str, Any]], Album]] = {
|
|||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class QualityScannerDeps:
|
||||
"""Bundle of cross-cutting deps the quality scanner needs."""
|
||||
quality_scanner_state: dict
|
||||
quality_scanner_lock: Any # threading.Lock
|
||||
QUALITY_TIERS: dict
|
||||
matching_engine: Any
|
||||
automation_engine: Any
|
||||
get_quality_tier_from_extension: Callable
|
||||
add_activity_item: Callable
|
||||
|
||||
|
||||
def _extract_lookup_value(value: Any, *names: str, default: Any = None) -> Any:
|
||||
|
|
@ -277,3 +300,363 @@ def _search_tracks_for_source(source: str, query: str, limit: int = 5, client: A
|
|||
except Exception as exc:
|
||||
logger.debug("Could not search %s for %s: %s", source, query, exc)
|
||||
return []
|
||||
|
||||
|
||||
def run_quality_scanner(scope='watchlist', profile_id=1, deps: QualityScannerDeps = None):
|
||||
"""Main quality scanner worker function"""
|
||||
from core.wishlist_service import get_wishlist_service
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
try:
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["status"] = "running"
|
||||
deps.quality_scanner_state["phase"] = "Initializing scan..."
|
||||
deps.quality_scanner_state["progress"] = 0
|
||||
deps.quality_scanner_state["processed"] = 0
|
||||
deps.quality_scanner_state["total"] = 0
|
||||
deps.quality_scanner_state["quality_met"] = 0
|
||||
deps.quality_scanner_state["low_quality"] = 0
|
||||
deps.quality_scanner_state["matched"] = 0
|
||||
deps.quality_scanner_state["results"] = []
|
||||
deps.quality_scanner_state["error_message"] = ""
|
||||
|
||||
logger.info(f"[Quality Scanner] Starting scan with scope: {scope}")
|
||||
|
||||
# Get database instance
|
||||
db = MusicDatabase()
|
||||
|
||||
# Get quality profile to determine preferred quality
|
||||
quality_profile = db.get_quality_profile()
|
||||
preferred_qualities = quality_profile.get('qualities', {})
|
||||
|
||||
# Determine minimum acceptable tier based on enabled qualities
|
||||
min_acceptable_tier = 999
|
||||
for quality_name, quality_config in preferred_qualities.items():
|
||||
if quality_config.get('enabled', False):
|
||||
# Map quality profile names to tier names
|
||||
tier_map = {
|
||||
'flac': 'lossless',
|
||||
'mp3_320': 'low_lossy',
|
||||
'mp3_256': 'low_lossy',
|
||||
'mp3_192': 'low_lossy'
|
||||
}
|
||||
tier_name = tier_map.get(quality_name)
|
||||
if tier_name:
|
||||
tier_num = deps.QUALITY_TIERS[tier_name]['tier']
|
||||
min_acceptable_tier = min(min_acceptable_tier, tier_num)
|
||||
|
||||
logger.info(f"[Quality Scanner] Minimum acceptable tier: {min_acceptable_tier}")
|
||||
|
||||
# Get tracks to scan based on scope
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["phase"] = "Loading tracks from database..."
|
||||
|
||||
if scope == 'watchlist':
|
||||
# Get watchlist artists
|
||||
watchlist_artists = db.get_watchlist_artists(profile_id=profile_id)
|
||||
if not watchlist_artists:
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["status"] = "finished"
|
||||
deps.quality_scanner_state["phase"] = "No watchlist artists found"
|
||||
deps.quality_scanner_state["error_message"] = "Please add artists to watchlist first"
|
||||
logger.warning("[Quality Scanner] No watchlist artists found")
|
||||
return
|
||||
|
||||
# Get artist names from watchlist
|
||||
artist_names = [artist.artist_name for artist in watchlist_artists]
|
||||
logger.info(f"[Quality Scanner] Scanning {len(artist_names)} watchlist artists")
|
||||
|
||||
# Get all tracks for these artists by name
|
||||
conn = db._get_connection()
|
||||
placeholders = ','.join(['?' for _ in artist_names])
|
||||
tracks_to_scan = conn.execute(
|
||||
f"SELECT t.id, t.title, t.artist_id, t.album_id, t.file_path, t.bitrate, a.name as artist_name, al.title as album_title "
|
||||
f"FROM tracks t "
|
||||
f"JOIN artists a ON t.artist_id = a.id "
|
||||
f"JOIN albums al ON t.album_id = al.id "
|
||||
f"WHERE a.name IN ({placeholders}) AND t.file_path IS NOT NULL",
|
||||
artist_names
|
||||
).fetchall()
|
||||
conn.close()
|
||||
else:
|
||||
# Scan all library tracks
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["phase"] = "Loading all library tracks..."
|
||||
|
||||
conn = db._get_connection()
|
||||
tracks_to_scan = conn.execute(
|
||||
"SELECT t.id, t.title, t.artist_id, t.album_id, t.file_path, t.bitrate, a.name as artist_name, al.title as album_title "
|
||||
"FROM tracks t "
|
||||
"JOIN artists a ON t.artist_id = a.id "
|
||||
"JOIN albums al ON t.album_id = al.id "
|
||||
"WHERE t.file_path IS NOT NULL"
|
||||
).fetchall()
|
||||
conn.close()
|
||||
|
||||
total_tracks = len(tracks_to_scan)
|
||||
logger.info(f"[Quality Scanner] Found {total_tracks} tracks to scan")
|
||||
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["total"] = total_tracks
|
||||
deps.quality_scanner_state["phase"] = f"Scanning {total_tracks} tracks..."
|
||||
|
||||
source_priority = get_source_priority(get_primary_source())
|
||||
if not source_priority:
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["status"] = "error"
|
||||
deps.quality_scanner_state["phase"] = "No metadata provider available"
|
||||
deps.quality_scanner_state["error_message"] = "No metadata provider is available for quality scanning"
|
||||
logger.info("[Quality Scanner] No metadata provider available")
|
||||
return
|
||||
|
||||
logger.info("[Quality Scanner] Using metadata source priority: %s", source_priority)
|
||||
|
||||
wishlist_service = get_wishlist_service()
|
||||
add_to_wishlist = getattr(wishlist_service, 'add_track_to_wishlist', None)
|
||||
if add_to_wishlist is None:
|
||||
add_to_wishlist = getattr(wishlist_service, 'add_spotify_track_to_wishlist', None)
|
||||
if add_to_wishlist is None:
|
||||
raise AttributeError("Wishlist service does not expose an add-to-wishlist method")
|
||||
|
||||
# Scan each track
|
||||
for idx, track_row in enumerate(tracks_to_scan, 1):
|
||||
# Check for stop request
|
||||
if deps.quality_scanner_state.get('status') != 'running':
|
||||
logger.info(f"[Quality Scanner] Stop requested, halting at track {idx}/{total_tracks}")
|
||||
break
|
||||
|
||||
try:
|
||||
track_id, title, artist_id, album_id, file_path, bitrate, artist_name, album_title = track_row
|
||||
|
||||
# Check quality tier
|
||||
tier_name, tier_num = deps.get_quality_tier_from_extension(file_path)
|
||||
|
||||
# Update progress
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["processed"] = idx
|
||||
deps.quality_scanner_state["progress"] = (idx / total_tracks) * 100
|
||||
deps.quality_scanner_state["phase"] = f"Scanning: {artist_name} - {title}"
|
||||
|
||||
# Check if meets quality standards
|
||||
if tier_num <= min_acceptable_tier:
|
||||
# Quality met
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["quality_met"] += 1
|
||||
continue
|
||||
|
||||
# Low quality track found
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["low_quality"] += 1
|
||||
|
||||
logger.info(f"[Quality Scanner] Low quality: {artist_name} - {title} ({tier_name}, {file_path})")
|
||||
|
||||
# Attempt to match using the active metadata provider
|
||||
matched = False
|
||||
matched_track_data = None
|
||||
best_source = None
|
||||
attempted_any_provider = False
|
||||
|
||||
try:
|
||||
# Generate search queries using matching engine
|
||||
temp_track = type('TempTrack', (), {
|
||||
'name': title,
|
||||
'artists': [artist_name],
|
||||
'album': album_title
|
||||
})()
|
||||
|
||||
search_queries = deps.matching_engine.generate_download_queries(temp_track)
|
||||
logger.info(f"[Quality Scanner] Generated {len(search_queries)} search queries for {artist_name} - {title}")
|
||||
|
||||
# Find best match using confidence scoring
|
||||
best_match = None
|
||||
best_confidence = 0.0
|
||||
min_confidence = 0.7 # Match existing standard
|
||||
|
||||
for _query_idx, search_query in enumerate(search_queries):
|
||||
try:
|
||||
for source in source_priority:
|
||||
client = get_client_for_source(source)
|
||||
if not client or not hasattr(client, 'search_tracks'):
|
||||
continue
|
||||
|
||||
attempted_any_provider = True
|
||||
provider_matches = _search_tracks_for_source(source, search_query, limit=5, client=client)
|
||||
time.sleep(0.5) # Rate limit metadata API calls
|
||||
|
||||
if not provider_matches:
|
||||
continue
|
||||
|
||||
# Score each result using matching engine
|
||||
for provider_track in provider_matches:
|
||||
try:
|
||||
# Calculate artist confidence
|
||||
artist_confidence = 0.0
|
||||
provider_artists = _track_artist_names(provider_track)
|
||||
if provider_artists:
|
||||
for result_artist in provider_artists:
|
||||
artist_sim = deps.matching_engine.similarity_score(
|
||||
deps.matching_engine.normalize_string(artist_name),
|
||||
deps.matching_engine.normalize_string(result_artist)
|
||||
)
|
||||
artist_confidence = max(artist_confidence, artist_sim)
|
||||
|
||||
# Calculate title confidence
|
||||
title_confidence = deps.matching_engine.similarity_score(
|
||||
deps.matching_engine.normalize_string(title),
|
||||
deps.matching_engine.normalize_string(_track_name(provider_track))
|
||||
)
|
||||
|
||||
# Combined confidence (50% artist + 50% title)
|
||||
combined_confidence = (artist_confidence * 0.5 + title_confidence * 0.5)
|
||||
|
||||
# Small bonus for album tracks over singles
|
||||
_at = _extract_lookup_value(provider_track, 'album_type', default='') or ''
|
||||
if _at == 'album':
|
||||
combined_confidence += 0.02
|
||||
elif _at == 'ep':
|
||||
combined_confidence += 0.01
|
||||
|
||||
candidate_artist = provider_artists[0] if provider_artists else 'Unknown Artist'
|
||||
candidate_name = _track_name(provider_track)
|
||||
logger.info(
|
||||
f"[Quality Scanner] Candidate ({source}): '{candidate_artist}' - "
|
||||
f"'{candidate_name}' (confidence: {combined_confidence:.3f})"
|
||||
)
|
||||
|
||||
# Update best match if this is better
|
||||
if combined_confidence > best_confidence and combined_confidence >= min_confidence:
|
||||
best_confidence = combined_confidence
|
||||
best_match = provider_track
|
||||
best_source = source
|
||||
logger.info(
|
||||
f"[Quality Scanner] New best match ({source}): {candidate_artist} - "
|
||||
f"{candidate_name} (confidence: {combined_confidence:.3f})"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[Quality Scanner] Error scoring result: {e}")
|
||||
continue
|
||||
|
||||
# If we found a very high confidence match, stop searching this query
|
||||
if best_confidence >= 0.9:
|
||||
logger.info(f"[Quality Scanner] High confidence match found ({best_confidence:.3f}), stopping search")
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f"[Quality Scanner] Error searching with query '{search_query}': {e}")
|
||||
continue
|
||||
|
||||
if not attempted_any_provider:
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["status"] = "error"
|
||||
deps.quality_scanner_state["phase"] = "No metadata provider available"
|
||||
deps.quality_scanner_state["error_message"] = "No metadata provider is available for quality scanning"
|
||||
logger.info("[Quality Scanner] No metadata provider available")
|
||||
return
|
||||
|
||||
# Process best match
|
||||
if best_match:
|
||||
matched = True
|
||||
final_artist = _track_artist_names(best_match)[0] if _track_artist_names(best_match) else 'Unknown Artist'
|
||||
final_name = _track_name(best_match)
|
||||
final_source = best_source or 'metadata'
|
||||
logger.info(
|
||||
f"[Quality Scanner] Final match ({final_source}): {final_artist} - "
|
||||
f"{final_name} (confidence: {best_confidence:.3f})"
|
||||
)
|
||||
|
||||
# Build normalized track data for wishlist
|
||||
matched_track_data = _normalize_track_match(best_match, final_source)
|
||||
|
||||
# Add to wishlist
|
||||
source_context = {
|
||||
'quality_scanner': True,
|
||||
'original_file_path': file_path,
|
||||
'original_format': tier_name,
|
||||
'original_bitrate': bitrate,
|
||||
'match_confidence': best_confidence,
|
||||
'scan_date': datetime.now().isoformat()
|
||||
}
|
||||
|
||||
success = add_to_wishlist(
|
||||
track_data=matched_track_data,
|
||||
failure_reason=f"Low quality - {tier_name.replace('_', ' ').title()} format",
|
||||
source_type='quality_scanner',
|
||||
source_context=source_context,
|
||||
profile_id=profile_id
|
||||
)
|
||||
|
||||
if success:
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["matched"] += 1
|
||||
logger.info(f"[Quality Scanner] Matched and added to wishlist: {artist_name} - {title}")
|
||||
else:
|
||||
logger.error(f"[Quality Scanner] Failed to add to wishlist: {artist_name} - {title}")
|
||||
else:
|
||||
logger.warning(
|
||||
f"[Quality Scanner] No suitable metadata match found "
|
||||
f"(best confidence: {best_confidence:.3f}, required: {min_confidence:.3f})"
|
||||
)
|
||||
|
||||
except Exception as matching_error:
|
||||
logger.error(f"[Quality Scanner] Matching error for {artist_name} - {title}: {matching_error}")
|
||||
|
||||
# Store result
|
||||
result_entry = {
|
||||
'track_id': track_id,
|
||||
'title': title,
|
||||
'artist': artist_name,
|
||||
'album': album_title,
|
||||
'file_path': file_path,
|
||||
'current_format': tier_name,
|
||||
'bitrate': bitrate,
|
||||
'matched': matched,
|
||||
'match_id': matched_track_data['id'] if matched_track_data else None,
|
||||
'provider': best_source if matched else None,
|
||||
'spotify_id': matched_track_data['id'] if matched_track_data else None,
|
||||
}
|
||||
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["results"].append(result_entry)
|
||||
|
||||
if not matched:
|
||||
logger.warning(f"[Quality Scanner] No metadata match found for: {artist_name} - {title}")
|
||||
|
||||
except Exception as track_error:
|
||||
logger.error(f"[Quality Scanner] Error processing track: {track_error}")
|
||||
continue
|
||||
|
||||
# Scan complete (don't overwrite if already stopped by user)
|
||||
with deps.quality_scanner_lock:
|
||||
was_stopped = deps.quality_scanner_state["status"] != "running"
|
||||
deps.quality_scanner_state["status"] = "finished"
|
||||
deps.quality_scanner_state["progress"] = 100
|
||||
if not was_stopped:
|
||||
deps.quality_scanner_state["phase"] = "Scan complete"
|
||||
|
||||
logger.info(f"[Quality Scanner] Scan {'stopped' if was_stopped else 'complete'}: {deps.quality_scanner_state['processed']} processed, "
|
||||
f"{deps.quality_scanner_state['low_quality']} low quality, {deps.quality_scanner_state['matched']} matched to metadata providers")
|
||||
|
||||
# Add activity
|
||||
deps.add_activity_item("", "Quality Scan Complete",
|
||||
f"{deps.quality_scanner_state['matched']} tracks added to wishlist", "Now")
|
||||
|
||||
try:
|
||||
if deps.automation_engine:
|
||||
deps.automation_engine.emit('quality_scan_completed', {
|
||||
'quality_met': str(deps.quality_scanner_state.get('quality_met', 0)),
|
||||
'low_quality': str(deps.quality_scanner_state.get('low_quality', 0)),
|
||||
'total_scanned': str(deps.quality_scanner_state.get('processed', 0)),
|
||||
})
|
||||
except Exception as e:
|
||||
logger.debug("emit quality_scan_completed failed: %s", e)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[Quality Scanner] Critical error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
with deps.quality_scanner_lock:
|
||||
deps.quality_scanner_state["status"] = "error"
|
||||
deps.quality_scanner_state["error_message"] = str(e)
|
||||
deps.quality_scanner_state["phase"] = f"Error: {str(e)}"
|
||||
|
|
|
|||
|
|
@ -55,17 +55,13 @@ class SpotifyPublicDiscoveryDeps:
|
|||
search_spotify_for_tidal_track: Callable
|
||||
build_discovery_wing_it_stub: Callable
|
||||
add_activity_item: Callable
|
||||
source_label: str = "Spotify Public"
|
||||
activity_label: str = "Spotify Link"
|
||||
original_track_key: str = "spotify_public_track"
|
||||
|
||||
|
||||
def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDeps):
|
||||
"""Background worker for Spotify Public discovery process (Spotify preferred, iTunes fallback)"""
|
||||
_ew_state = {}
|
||||
try:
|
||||
worker_label = f"{deps.source_label} discovery"
|
||||
_ew_state = deps.pause_enrichment_workers(worker_label)
|
||||
_ew_state = deps.pause_enrichment_workers('Spotify Public discovery')
|
||||
state = deps.spotify_public_discovery_states[url_hash]
|
||||
playlist = state['playlist']
|
||||
|
||||
|
|
@ -78,7 +74,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
if not use_spotify:
|
||||
itunes_client_instance = deps.get_metadata_fallback_client()
|
||||
|
||||
logger.info(f"Starting {deps.source_label} discovery for: {playlist['name']} (using {discovery_source.upper()})")
|
||||
logger.info(f"Starting Spotify Public discovery for: {playlist['name']} (using {discovery_source.upper()})")
|
||||
|
||||
# Store discovery source in state for frontend
|
||||
state['discovery_source'] = discovery_source
|
||||
|
|
@ -130,7 +126,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
cached_album = cached_album.get('name', '')
|
||||
|
||||
result = {
|
||||
deps.original_track_key: {
|
||||
'spotify_public_track': {
|
||||
'id': track_id,
|
||||
'name': track_name,
|
||||
'artists': track_artists or [],
|
||||
|
|
@ -174,7 +170,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
|
||||
# Create result entry
|
||||
result = {
|
||||
deps.original_track_key: {
|
||||
'spotify_public_track': {
|
||||
'id': track_id,
|
||||
'name': track_name,
|
||||
'artists': track_artists or [],
|
||||
|
|
@ -274,7 +270,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
|
||||
# Auto Wing It fallback for unmatched tracks
|
||||
if result['status_class'] == 'not-found':
|
||||
sp_t = result.get(deps.original_track_key, {})
|
||||
sp_t = result.get('spotify_public_track', {})
|
||||
stub = deps.build_discovery_wing_it_stub(
|
||||
sp_t.get('name', ''),
|
||||
', '.join(sp_t.get('artists', [])),
|
||||
|
|
@ -303,7 +299,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
logger.error(f"Error processing track {i+1}: {e}")
|
||||
# Add error result
|
||||
result = {
|
||||
deps.original_track_key: {
|
||||
'spotify_public_track': {
|
||||
'name': sp_track.get('name', 'Unknown'),
|
||||
'artists': sp_track.get('artists', []),
|
||||
},
|
||||
|
|
@ -328,14 +324,14 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
|
|||
|
||||
# Add activity for discovery completion
|
||||
source_label = discovery_source.upper()
|
||||
deps.add_activity_item("", f"{deps.activity_label} Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
|
||||
deps.add_activity_item("", f"Spotify Link Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
|
||||
|
||||
logger.info(f"{deps.source_label} discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
|
||||
logger.info(f"Spotify Public discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in {deps.source_label} discovery worker: {e}")
|
||||
logger.error(f"Error in Spotify Public discovery worker: {e}")
|
||||
if url_hash in deps.spotify_public_discovery_states:
|
||||
deps.spotify_public_discovery_states[url_hash]['phase'] = 'error'
|
||||
deps.spotify_public_discovery_states[url_hash]['status'] = f'error: {str(e)}'
|
||||
finally:
|
||||
deps.resume_enrichment_workers(_ew_state, f"{deps.source_label} discovery")
|
||||
deps.resume_enrichment_workers(_ew_state, 'Spotify Public discovery')
|
||||
|
|
|
|||
|
|
@ -47,229 +47,9 @@ class SyncDeps:
|
|||
update_and_save_sync_status: Callable
|
||||
sync_states: dict
|
||||
sync_lock: Any # threading.Lock
|
||||
# Optional: post-sync download follow-up for mirrored-playlist automations.
|
||||
process_wishlist_automatically: Callable[..., Any] | None = None
|
||||
run_playlist_organize_download: Callable[..., Any] | None = None
|
||||
is_wishlist_actually_processing: Callable[[], bool] | None = None
|
||||
|
||||
|
||||
def _post_sync_automation_followup(
|
||||
deps: SyncDeps,
|
||||
*,
|
||||
automation_id: str,
|
||||
playlist_id: str,
|
||||
skip_wishlist_add: bool,
|
||||
result: Any,
|
||||
) -> None:
|
||||
"""Queue downloads after an automation sync finishes.
|
||||
|
||||
Sync Playlist runs in a background thread and returns immediately, so a
|
||||
separate scheduled "Process Wishlist" action often runs on an empty wishlist.
|
||||
Organize-by-playlist skips sync-time wishlist adds and expects a folder
|
||||
download batch instead — that only ran in Playlist Pipeline before this hook.
|
||||
"""
|
||||
if not automation_id or not str(playlist_id).startswith('auto_mirror_'):
|
||||
return
|
||||
try:
|
||||
mirrored_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
|
||||
except ValueError:
|
||||
return
|
||||
|
||||
failed = int(getattr(result, 'failed_tracks', 0) or 0)
|
||||
wishlist_added = int(getattr(result, 'wishlist_added_count', 0) or 0)
|
||||
|
||||
if skip_wishlist_add:
|
||||
org_fn = deps.run_playlist_organize_download
|
||||
if failed <= 0:
|
||||
return
|
||||
if not org_fn:
|
||||
logger.warning(
|
||||
"Organize-by-playlist sync left %s missing tracks but organize download is unavailable",
|
||||
failed,
|
||||
)
|
||||
deps.update_automation_progress(
|
||||
automation_id,
|
||||
log_line=f'{failed} missing — enable Playlist Pipeline or disable Organize by Playlist',
|
||||
log_type='warning',
|
||||
)
|
||||
return
|
||||
org_result = org_fn(mirrored_playlist_id=mirrored_id, automation_id=automation_id)
|
||||
status = org_result.get('status', 'unknown') if isinstance(org_result, dict) else 'unknown'
|
||||
reason = org_result.get('reason', '') if isinstance(org_result, dict) else ''
|
||||
log_type = 'success' if status == 'started' else 'warning'
|
||||
detail = f' ({reason})' if reason and status != 'started' else ''
|
||||
deps.update_automation_progress(
|
||||
automation_id,
|
||||
log_line=f'Organize download {status} for {failed} missing track(s){detail}',
|
||||
log_type=log_type,
|
||||
)
|
||||
return
|
||||
|
||||
if wishlist_added <= 0:
|
||||
if failed > 0:
|
||||
deps.update_automation_progress(
|
||||
automation_id,
|
||||
log_line=f'{failed} missing but none added to wishlist — check logs',
|
||||
log_type='warning',
|
||||
)
|
||||
return
|
||||
|
||||
proc_fn = deps.process_wishlist_automatically
|
||||
if not proc_fn:
|
||||
return
|
||||
is_busy = deps.is_wishlist_actually_processing
|
||||
if is_busy and is_busy():
|
||||
deps.update_automation_progress(
|
||||
automation_id,
|
||||
log_line=f'Added {wishlist_added} to wishlist; download worker already running',
|
||||
log_type='info',
|
||||
)
|
||||
return
|
||||
proc_fn(automation_id=automation_id)
|
||||
deps.update_automation_progress(
|
||||
automation_id,
|
||||
log_line=f'Started wishlist download for {wishlist_added} track(s)',
|
||||
log_type='success',
|
||||
)
|
||||
|
||||
|
||||
async def _database_only_find_track(spotify_track, candidate_pool=None):
|
||||
"""Database-only track matcher used when no media server is connected.
|
||||
|
||||
Patched onto sync_service._find_track_in_media_server. Accepts
|
||||
``candidate_pool`` for interface parity with the real matcher (sync_service
|
||||
calls it with candidate_pool=...); the DB path queries the library directly
|
||||
via check_track_exists, so it doesn't need the per-artist candidate cache —
|
||||
but it MUST accept the kwarg or sync raises "unexpected keyword argument
|
||||
'candidate_pool'". Module-level (not a nested closure) so it's importable
|
||||
and unit-tested.
|
||||
"""
|
||||
logger.info(f"Database-only search for: '{spotify_track.name}' by {spotify_track.artists}")
|
||||
try:
|
||||
from database.music_database import MusicDatabase
|
||||
from config.settings import config_manager
|
||||
|
||||
db = MusicDatabase()
|
||||
active_server = config_manager.get_active_media_server()
|
||||
original_title = spotify_track.name
|
||||
spotify_id = getattr(spotify_track, 'id', '') or ''
|
||||
|
||||
# --- Sync match cache fast-path ---
|
||||
if spotify_id:
|
||||
try:
|
||||
cached = db.read_sync_match_cache(spotify_id, active_server)
|
||||
if cached:
|
||||
db_track_check = db.get_track_by_id(cached['server_track_id'])
|
||||
if db_track_check:
|
||||
class DatabaseTrackCached:
|
||||
def __init__(self, db_t):
|
||||
self.ratingKey = db_t.id
|
||||
self.title = db_t.title
|
||||
self.id = db_t.id
|
||||
logger.debug(f"Sync cache hit: '{original_title}' → server track {cached['server_track_id']}")
|
||||
return DatabaseTrackCached(db_track_check), cached['confidence']
|
||||
logger.warning(f"Sync cache stale for '{original_title}' — track gone")
|
||||
except Exception as e:
|
||||
logger.debug("sync match cache fast-path failed: %s", e)
|
||||
# --- End cache fast-path ---
|
||||
|
||||
# Durable manual library match (#787) — survives a library rescan (the
|
||||
# sync_match_cache above does not), so a user's Find & Add pairing keeps
|
||||
# sticking across auto-syncs instead of being re-matched from scratch (#895
|
||||
# follow-up). Self-heals a stale library id via the stored file path.
|
||||
if spotify_id:
|
||||
try:
|
||||
from core.artists.map import get_current_profile_id
|
||||
m = db.find_manual_library_match_by_source_track_id(
|
||||
get_current_profile_id(), str(spotify_id), active_server)
|
||||
if m:
|
||||
lib_id = m.get('library_track_id')
|
||||
dt = db.get_track_by_id(lib_id) if lib_id is not None else None
|
||||
if not dt and m.get('library_file_path'):
|
||||
new_id = db.find_track_id_by_file_path(m['library_file_path'])
|
||||
dt = db.get_track_by_id(new_id) if new_id else None
|
||||
if dt:
|
||||
class DatabaseTrackDurable:
|
||||
def __init__(self, db_t):
|
||||
self.ratingKey = db_t.id
|
||||
self.title = db_t.title
|
||||
self.id = db_t.id
|
||||
logger.debug(f"Durable manual match hit: '{original_title}' → {lib_id}")
|
||||
return DatabaseTrackDurable(dt), 1.0
|
||||
except Exception as e:
|
||||
logger.debug("durable manual match fast-path failed: %s", e)
|
||||
|
||||
# Try each artist
|
||||
for artist in spotify_track.artists:
|
||||
if isinstance(artist, str):
|
||||
artist_name = artist
|
||||
elif isinstance(artist, dict) and 'name' in artist:
|
||||
artist_name = artist['name']
|
||||
else:
|
||||
artist_name = str(artist)
|
||||
|
||||
db_track, confidence = db.check_track_exists(
|
||||
original_title, artist_name,
|
||||
confidence_threshold=0.80,
|
||||
server_source=active_server
|
||||
)
|
||||
|
||||
if not (db_track and confidence >= 0.80):
|
||||
# #785: file/CSV playlists keep raw "Artist - Title" titles (unlike
|
||||
# YouTube, cleaned at ingest), which don't match the clean library
|
||||
# title. Retry with the canonical form (best-of, conservative).
|
||||
try:
|
||||
from core.text.source_title import canonical_source_track
|
||||
_canon_title, _canon_artist = canonical_source_track(original_title, artist_name)
|
||||
if (_canon_title, _canon_artist) != (original_title, artist_name):
|
||||
_alt_track, _alt_conf = db.check_track_exists(
|
||||
_canon_title, _canon_artist,
|
||||
confidence_threshold=0.80, server_source=active_server)
|
||||
if _alt_track and _alt_conf > confidence:
|
||||
db_track, confidence = _alt_track, _alt_conf
|
||||
except Exception as _canon_err:
|
||||
logger.debug("canonical retry failed: %s", _canon_err)
|
||||
|
||||
if db_track and confidence >= 0.80:
|
||||
logger.info(f"Database match: '{db_track.title}' (confidence: {confidence:.2f})")
|
||||
if spotify_id:
|
||||
try:
|
||||
from core.matching_engine import MusicMatchingEngine
|
||||
me = MusicMatchingEngine()
|
||||
db.save_sync_match_cache(
|
||||
spotify_id, me.clean_title(original_title), me.clean_artist(artist_name),
|
||||
active_server, db_track.id, db_track.title, confidence
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("save sync match cache failed: %s", e)
|
||||
|
||||
class DatabaseTrackMock:
|
||||
def __init__(self, db_track):
|
||||
self.ratingKey = db_track.id
|
||||
self.title = db_track.title
|
||||
self.id = db_track.id
|
||||
|
||||
return DatabaseTrackMock(db_track), confidence
|
||||
|
||||
logger.warning(f"No database match found for: '{original_title}'")
|
||||
return None, 0.0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Database search error: {e}")
|
||||
return None, 0.0
|
||||
|
||||
|
||||
def run_sync_task(
|
||||
playlist_id,
|
||||
playlist_name,
|
||||
tracks_json,
|
||||
automation_id=None,
|
||||
profile_id=1,
|
||||
playlist_image_url='',
|
||||
deps: SyncDeps = None,
|
||||
sync_mode: str = 'replace',
|
||||
skip_wishlist_add: bool = False,
|
||||
):
|
||||
def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, profile_id=1, playlist_image_url='', deps: SyncDeps = None):
|
||||
"""The actual sync function that runs in the background thread."""
|
||||
sync_states = deps.sync_states
|
||||
sync_lock = deps.sync_lock
|
||||
|
|
@ -307,11 +87,35 @@ def run_sync_task(
|
|||
# This avoids needing to re-fetch it from Spotify
|
||||
logger.info("Converting JSON tracks to SpotifyTrack objects...")
|
||||
|
||||
# Store original track data with full album objects (for wishlist with cover art).
|
||||
# Shared with the sync-detail "re-add to wishlist" action so both build the
|
||||
# IDENTICAL payload (album→dict + images, artists→dicts). Copy-safe.
|
||||
from core.sync.wishlist_readd import build_original_tracks_map
|
||||
original_tracks_map = build_original_tracks_map(tracks_json)
|
||||
# Store original track data with full album objects (for wishlist with cover art)
|
||||
# Normalize formats for wishlist: album must be dict {'name': ...}, artists must be [{'name': ...}]
|
||||
# Important: copy data — don't mutate tracks_json since SpotifyTrack expects List[str] artists
|
||||
original_tracks_map = {}
|
||||
for t in tracks_json:
|
||||
track_id = t.get('id', '')
|
||||
if track_id:
|
||||
normalized = dict(t)
|
||||
# Normalize album to dict format, preserving images and metadata
|
||||
raw_album = normalized.get('album', '')
|
||||
if isinstance(raw_album, str):
|
||||
normalized['album'] = {
|
||||
'name': raw_album or normalized.get('name', 'Unknown Album'),
|
||||
'images': [], 'album_type': 'single', 'total_tracks': 1, 'release_date': ''
|
||||
}
|
||||
elif not isinstance(raw_album, dict):
|
||||
normalized['album'] = {
|
||||
'name': str(raw_album) if raw_album else normalized.get('name', 'Unknown Album'),
|
||||
'images': [], 'album_type': 'single', 'total_tracks': 1, 'release_date': ''
|
||||
}
|
||||
else:
|
||||
# Dict — ensure required keys exist
|
||||
raw_album.setdefault('name', 'Unknown Album')
|
||||
raw_album.setdefault('images', [])
|
||||
# Normalize artists to list of dicts
|
||||
raw_artists = normalized.get('artists', [])
|
||||
if raw_artists and isinstance(raw_artists[0], str):
|
||||
normalized['artists'] = [{'name': a} for a in raw_artists]
|
||||
original_tracks_map[track_id] = normalized
|
||||
|
||||
tracks = []
|
||||
for i, t in enumerate(tracks_json):
|
||||
|
|
@ -456,9 +260,89 @@ def run_sync_task(
|
|||
if media_client is None or not media_client.is_connected():
|
||||
logger.info("Media client not connected - patching sync service for database-only matching")
|
||||
|
||||
# Patch the matcher to the module-level database-only implementation
|
||||
# (importable + unit-tested; accepts candidate_pool for parity).
|
||||
sync_service._find_track_in_media_server = _database_only_find_track
|
||||
# Store original method
|
||||
original_find_track = sync_service._find_track_in_media_server
|
||||
|
||||
# Create database-only replacement method
|
||||
async def database_only_find_track(spotify_track):
|
||||
logger.info(f"Database-only search for: '{spotify_track.name}' by {spotify_track.artists}")
|
||||
try:
|
||||
from database.music_database import MusicDatabase
|
||||
from config.settings import config_manager
|
||||
|
||||
db = MusicDatabase()
|
||||
active_server = config_manager.get_active_media_server()
|
||||
original_title = spotify_track.name
|
||||
spotify_id = getattr(spotify_track, 'id', '') or ''
|
||||
|
||||
# --- Sync match cache fast-path ---
|
||||
if spotify_id:
|
||||
try:
|
||||
cached = db.read_sync_match_cache(spotify_id, active_server)
|
||||
if cached:
|
||||
db_track_check = db.get_track_by_id(cached['server_track_id'])
|
||||
if db_track_check:
|
||||
class DatabaseTrackCached:
|
||||
def __init__(self, db_t):
|
||||
self.ratingKey = db_t.id
|
||||
self.title = db_t.title
|
||||
self.id = db_t.id
|
||||
logger.debug(f"Sync cache hit: '{original_title}' → server track {cached['server_track_id']}")
|
||||
return DatabaseTrackCached(db_track_check), cached['confidence']
|
||||
logger.warning(f"Sync cache stale for '{original_title}' — track gone")
|
||||
except Exception as e:
|
||||
logger.debug("sync match cache fast-path failed: %s", e)
|
||||
# --- End cache fast-path ---
|
||||
|
||||
# Try each artist (same logic as original)
|
||||
for artist in spotify_track.artists:
|
||||
# Extract artist name from both string and dict formats
|
||||
if isinstance(artist, str):
|
||||
artist_name = artist
|
||||
elif isinstance(artist, dict) and 'name' in artist:
|
||||
artist_name = artist['name']
|
||||
else:
|
||||
artist_name = str(artist)
|
||||
|
||||
db_track, confidence = db.check_track_exists(
|
||||
original_title, artist_name,
|
||||
confidence_threshold=0.80,
|
||||
server_source=active_server
|
||||
)
|
||||
|
||||
if db_track and confidence >= 0.80:
|
||||
logger.info(f"Database match: '{db_track.title}' (confidence: {confidence:.2f})")
|
||||
|
||||
# Save to sync match cache
|
||||
if spotify_id:
|
||||
try:
|
||||
from core.matching_engine import MusicMatchingEngine
|
||||
me = MusicMatchingEngine()
|
||||
db.save_sync_match_cache(
|
||||
spotify_id, me.clean_title(original_title), me.clean_artist(artist_name),
|
||||
active_server, db_track.id, db_track.title, confidence
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("save sync match cache failed: %s", e)
|
||||
|
||||
# Create mock track object for playlist creation
|
||||
class DatabaseTrackMock:
|
||||
def __init__(self, db_track):
|
||||
self.ratingKey = db_track.id
|
||||
self.title = db_track.title
|
||||
self.id = db_track.id
|
||||
|
||||
return DatabaseTrackMock(db_track), confidence
|
||||
|
||||
logger.warning(f"No database match found for: '{original_title}'")
|
||||
return None, 0.0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Database search error: {e}")
|
||||
return None, 0.0
|
||||
|
||||
# Patch the method
|
||||
sync_service._find_track_in_media_server = database_only_find_track
|
||||
logger.info("Patched sync service to use database-only matching")
|
||||
|
||||
sync_start_time = time.time()
|
||||
|
|
@ -472,17 +356,10 @@ def run_sync_task(
|
|||
# Wing It mode — skip wishlist for unmatched tracks
|
||||
with sync_lock:
|
||||
is_wing_it = sync_states.get(playlist_id, {}).get('wing_it', False)
|
||||
sync_service._skip_unmatched_wishlist = is_wing_it or skip_wishlist_add
|
||||
sync_service._skip_wishlist = is_wing_it
|
||||
if skip_wishlist_add:
|
||||
logger.info(
|
||||
"[Organize by Playlist] Skipping sync-time wishlist for '%s' — "
|
||||
"organize download + batch failure handling cover missing tracks",
|
||||
playlist_name,
|
||||
)
|
||||
|
||||
# Run the sync (this is a blocking call within this thread)
|
||||
result = deps.run_async(sync_service.sync_playlist(playlist, download_missing=False, profile_id=profile_id, sync_mode=sync_mode))
|
||||
result = deps.run_async(sync_service.sync_playlist(playlist, download_missing=False, profile_id=profile_id))
|
||||
|
||||
# Clear progress callback immediately to prevent race condition where a
|
||||
# late-firing progress callback overwrites the "finished" state below
|
||||
|
|
@ -520,19 +397,10 @@ def run_sync_task(
|
|||
}
|
||||
logger.info(f"Sync finished for {playlist_id} - state updated")
|
||||
|
||||
# Set playlist poster image if available (Plex, Jellyfin, Emby).
|
||||
# Don't log the URL itself — it may carry an auth token (Plex
|
||||
# X-Plex-Token / Jellyfin X-Emby-Token / Subsonic auth) that we
|
||||
# don't want persisted to app.log.
|
||||
# Set playlist poster image if available (Plex, Jellyfin, Emby)
|
||||
_synced = getattr(result, 'synced_tracks', 0)
|
||||
logger.info(f"[PLAYLIST IMAGE] has_image={bool(playlist_image_url)}, synced_tracks={_synced}")
|
||||
# Modes that edit a playlist in place (reconcile #792, append #811) must
|
||||
# NOT push the source image — doing so re-clobbers a user's custom poster
|
||||
# every sync, the exact bug these modes exist to avoid. Only the
|
||||
# destructive 'replace' (recreate-from-scratch) pushes the image.
|
||||
if sync_mode in ('reconcile', 'append'):
|
||||
logger.info(f"[PLAYLIST IMAGE] {sync_mode} mode — preserving existing playlist image")
|
||||
elif playlist_image_url and _synced > 0:
|
||||
logger.info(f"[PLAYLIST IMAGE] image_url={playlist_image_url!r}, synced_tracks={_synced}")
|
||||
if playlist_image_url and _synced > 0:
|
||||
try:
|
||||
active_server = deps.config_manager.get_active_media_server()
|
||||
logger.info(f"[PLAYLIST IMAGE] active_server={active_server}")
|
||||
|
|
@ -584,21 +452,9 @@ def run_sync_task(
|
|||
matched = getattr(result, 'matched_tracks', 0)
|
||||
total = getattr(result, 'total_tracks', 0)
|
||||
failed = getattr(result, 'failed_tracks', 0)
|
||||
wishlist_added = getattr(result, 'wishlist_added_count', 0) or 0
|
||||
deps.update_automation_progress(automation_id, status='finished', progress=100,
|
||||
phase='Sync complete',
|
||||
log_line=(
|
||||
f'Done: {matched}/{total} in library, {failed} missing'
|
||||
+ (f', {wishlist_added} added to wishlist' if wishlist_added else '')
|
||||
),
|
||||
log_type='success')
|
||||
_post_sync_automation_followup(
|
||||
deps,
|
||||
automation_id=automation_id,
|
||||
playlist_id=playlist_id,
|
||||
skip_wishlist_add=skip_wishlist_add,
|
||||
result=result,
|
||||
)
|
||||
log_line=f'Done: {matched}/{total} matched, {failed} failed', log_type='success')
|
||||
|
||||
# Emit playlist_synced event for automation engine
|
||||
try:
|
||||
|
|
@ -617,28 +473,12 @@ def run_sync_task(
|
|||
import hashlib as _hl
|
||||
_track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
|
||||
_tracks_hash = _hl.md5(_track_ids_str.encode()).hexdigest()
|
||||
_mirror_tracks_hash = None
|
||||
if str(playlist_id).startswith('auto_mirror_'):
|
||||
try:
|
||||
_mp_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
|
||||
from database.music_database import MusicDatabase
|
||||
_mtracks = MusicDatabase().get_mirrored_playlist_tracks(_mp_id)
|
||||
_mids = ','.join(
|
||||
sorted(t.get('source_track_id', '') or '' for t in _mtracks if t.get('source_track_id'))
|
||||
)
|
||||
_mirror_tracks_hash = _hl.md5(_mids.encode()).hexdigest() if _mids else ''
|
||||
except Exception as e:
|
||||
logger.debug("mirror_tracks_hash for sync status: %s", e)
|
||||
snapshot_id = getattr(playlist, 'snapshot_id', None)
|
||||
_status_kwargs = dict(
|
||||
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id,
|
||||
matched_tracks=getattr(result, 'matched_tracks', 0),
|
||||
total_tracks=getattr(result, 'total_tracks', 0),
|
||||
discovered_tracks=len(tracks_json),
|
||||
tracks_hash=_tracks_hash,
|
||||
)
|
||||
if _mirror_tracks_hash is not None:
|
||||
_status_kwargs['mirror_tracks_hash'] = _mirror_tracks_hash
|
||||
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id, **_status_kwargs)
|
||||
tracks_hash=_tracks_hash)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"SYNC FAILED for {playlist_id}: {e}")
|
||||
|
|
|
|||
|
|
@ -32,26 +32,6 @@ from typing import Any, Callable
|
|||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_UNKNOWN_ARTIST = 'Unknown Artist'
|
||||
|
||||
|
||||
def resolve_display_artist(yt_artist: str, matched_artist: str) -> str:
|
||||
"""The artist to show in the 'YT Artist' column (#909).
|
||||
|
||||
YouTube's flat playlist data carries no artist, so a track starts as
|
||||
"Unknown Artist" and only gains a real name if per-video recovery succeeds.
|
||||
When recovery comes up empty but the track still matched confidently, show
|
||||
the matched artist instead of a misleading "Unknown Artist". Returns the
|
||||
original ``yt_artist`` whenever it's already a real name (recovery worked) or
|
||||
when there's no matched artist to fall back to — purely a display choice, the
|
||||
match itself is unaffected.
|
||||
"""
|
||||
current = (yt_artist or '').strip()
|
||||
if current and current != _UNKNOWN_ARTIST:
|
||||
return current # recovery already gave a real name — keep it
|
||||
fallback = (matched_artist or '').strip()
|
||||
return fallback or _UNKNOWN_ARTIST # backfill from the match, else honest Unknown
|
||||
|
||||
|
||||
@dataclass
|
||||
class YoutubeDiscoveryDeps:
|
||||
|
|
@ -72,10 +52,6 @@ class YoutubeDiscoveryDeps:
|
|||
build_discovery_wing_it_stub: Callable
|
||||
get_database: Callable[[], Any]
|
||||
add_activity_item: Callable
|
||||
# Recover a YouTube track's artist from its own video page when flat playlist
|
||||
# extraction left it "Unknown Artist" (#863). Takes a video id, returns a raw
|
||||
# artist string or ''. Optional — discovery still works without it.
|
||||
recover_youtube_artist: Callable[[str], str] = None
|
||||
|
||||
|
||||
def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
|
||||
|
|
@ -118,30 +94,6 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
|
|||
cleaned_title = track['name']
|
||||
cleaned_artist = track['artists'][0] if track['artists'] else 'Unknown Artist'
|
||||
|
||||
# Recover the artist from the track's own video page if flat
|
||||
# playlist extraction left it Unknown (#863). Done here, in the
|
||||
# background worker, rather than in the parse request (which would
|
||||
# block for minutes on a big playlist). Per-track cost is hidden
|
||||
# behind the discovery progress bar; the recovered artist makes the
|
||||
# match below actually find the song.
|
||||
if cleaned_artist == 'Unknown Artist' and track.get('id'):
|
||||
if not deps.recover_youtube_artist:
|
||||
logger.warning("[YT Discovery] artist recovery unavailable (dep not wired) "
|
||||
"— '%s' stays Unknown", cleaned_title)
|
||||
else:
|
||||
try:
|
||||
_rec = deps.recover_youtube_artist(track['id'])
|
||||
except Exception as _rec_err:
|
||||
logger.warning(f"[YT Discovery] artist recovery raised for {track.get('id')}: {_rec_err}")
|
||||
_rec = ''
|
||||
if _rec and _rec != 'Unknown Artist':
|
||||
logger.info(f"[YT Discovery] recovered artist '{_rec}' for '{cleaned_title}' ({track['id']})")
|
||||
cleaned_artist = _rec
|
||||
track['artists'] = [_rec] # persist so retries/UI see it
|
||||
else:
|
||||
logger.info(f"[YT Discovery] artist recovery returned nothing for "
|
||||
f"'{cleaned_title}' ({track['id']}) — leaving Unknown")
|
||||
|
||||
logger.info(f"Searching {discovery_source} for: '{cleaned_artist}' - '{cleaned_title}'")
|
||||
|
||||
# Check discovery cache first
|
||||
|
|
@ -151,15 +103,14 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
|
|||
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
|
||||
if cached_match and deps.validate_discovery_cache_artist(cleaned_artist, cached_match):
|
||||
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {cleaned_artist} - {cleaned_title}")
|
||||
_match_artist = deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else ''
|
||||
result = {
|
||||
'index': i,
|
||||
'yt_track': cleaned_title,
|
||||
'yt_artist': resolve_display_artist(cleaned_artist, _match_artist),
|
||||
'yt_artist': cleaned_artist,
|
||||
'status': 'Found',
|
||||
'status_class': 'found',
|
||||
'spotify_track': cached_match.get('name', ''),
|
||||
'spotify_artist': _match_artist,
|
||||
'spotify_artist': deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else '',
|
||||
'spotify_album': cached_match.get('album', {}).get('name', '') if isinstance(cached_match.get('album'), dict) else cached_match.get('album', ''),
|
||||
'duration': f"{int(track['duration_ms']) // 60000}:{(int(track['duration_ms']) % 60000) // 1000:02d}" if track['duration_ms'] else '0:00',
|
||||
'discovery_source': discovery_source,
|
||||
|
|
@ -286,17 +237,15 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
|
|||
best_confidence = confidence
|
||||
logger.info(f"Strategy 4 YouTube match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
|
||||
|
||||
# Create result entry. yt_artist falls back to the matched artist when
|
||||
# YouTube/recovery left it "Unknown Artist" but we matched confidently (#909).
|
||||
_match_artist = deps.extract_artist_name(matched_track.artists[0]) if matched_track else ''
|
||||
# Create result entry
|
||||
result = {
|
||||
'index': i,
|
||||
'yt_track': cleaned_title,
|
||||
'yt_artist': resolve_display_artist(cleaned_artist, _match_artist),
|
||||
'yt_artist': cleaned_artist,
|
||||
'status': 'Found' if matched_track else 'Not Found',
|
||||
'status_class': 'found' if matched_track else 'not-found',
|
||||
'spotify_track': matched_track.name if matched_track else '',
|
||||
'spotify_artist': _match_artist,
|
||||
'spotify_artist': deps.extract_artist_name(matched_track.artists[0]) if matched_track else '',
|
||||
'spotify_album': matched_track.album if matched_track else '',
|
||||
'duration': f"{int(track['duration_ms']) // 60000}:{(int(track['duration_ms']) % 60000) // 1000:02d}" if track['duration_ms'] else '0:00',
|
||||
'discovery_source': discovery_source,
|
||||
|
|
|
|||
|
|
@ -25,7 +25,6 @@ big-bang switchover.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
from typing import Any, Dict, Iterator, List, Optional, Tuple
|
||||
|
||||
|
|
@ -392,16 +391,6 @@ class DownloadEngine:
|
|||
(tracks, albums) tuple, or ``([], [])`` when every source
|
||||
in the chain is exhausted.
|
||||
|
||||
Priority mode is deliberately quality-AGNOSTIC at search time — source
|
||||
order is king and the first source that returns any tracks wins, exactly
|
||||
matching pre-quality-system behaviour byte-for-byte (#896 review #3).
|
||||
Quality-gating the priority path would deprioritise e.g. a soulseek
|
||||
mp3 whose bitrate slskd omitted (``bitrate=None`` → "unsatisfied"),
|
||||
changing which source wins and adding latency for users who never opted
|
||||
in. Cross-source quality pooling is the job of best_quality mode
|
||||
(``search_all_sources``); final per-result ranking still happens in the
|
||||
orchestrator's match/quality filter. RAW tracks are returned.
|
||||
|
||||
Replaces orchestrator's hand-rolled hybrid search loop. The
|
||||
chain is ordered (most-preferred first).
|
||||
"""
|
||||
|
|
@ -417,10 +406,9 @@ class DownloadEngine:
|
|||
try:
|
||||
logger.info(f"Trying {source_name} (priority {i+1}): {query}")
|
||||
tracks, albums = await plugin.search(query, timeout, progress_callback)
|
||||
if not tracks:
|
||||
continue
|
||||
logger.info(f"{source_name} found {len(tracks)} tracks")
|
||||
return (tracks, albums)
|
||||
if tracks:
|
||||
logger.info(f"{source_name} found {len(tracks)} tracks")
|
||||
return (tracks, albums)
|
||||
except Exception as e:
|
||||
logger.warning(f"{source_name} search failed: {e}")
|
||||
|
||||
|
|
@ -430,75 +418,6 @@ class DownloadEngine:
|
|||
)
|
||||
return ([], [])
|
||||
|
||||
async def search_all_sources(self, query: str, source_chain,
|
||||
timeout=None, progress_callback=None,
|
||||
exclude_sources=None):
|
||||
"""Best-quality mode: pool RAW tracks from EVERY configured source in
|
||||
``source_chain`` instead of stopping at the first satisfying one.
|
||||
|
||||
Unlike :meth:`search_with_fallback`, no source short-circuits the
|
||||
search — the caller (orchestrator/worker) ranks the combined pool
|
||||
best→worst by actual audio quality. ``exclude_sources`` drops sources
|
||||
whose per-source retry budget is already spent (so their candidates
|
||||
never re-enter the pool). Unconfigured / unregistered / raising sources
|
||||
are skipped exactly like the fallback path. Returns
|
||||
``(combined_tracks, combined_albums)``.
|
||||
"""
|
||||
excluded = {s.lower() for s in (exclude_sources or []) if s}
|
||||
pooled_tracks = []
|
||||
pooled_albums = []
|
||||
# Per-source contribution for an honest pool log — e.g. a release-level
|
||||
# source like usenet/torrent that returns nothing for a track-title
|
||||
# query should read "usenet=0", not silently hide behind the chain name.
|
||||
contributions = []
|
||||
|
||||
# Decide which sources to actually query, recording why the rest were
|
||||
# skipped. Searches then run CONCURRENTLY so the pool waits only for the
|
||||
# slowest source (e.g. usenet/Prowlarr, which can be slow) rather than
|
||||
# the sum of every source's latency.
|
||||
to_search = [] # (source_name, plugin)
|
||||
for source_name in source_chain:
|
||||
if source_name.lower() in excluded:
|
||||
contributions.append(f"{source_name}=excluded")
|
||||
continue
|
||||
plugin = self._plugins.get(source_name)
|
||||
if plugin is None:
|
||||
logger.info(f"Skipping {source_name} (not available)")
|
||||
contributions.append(f"{source_name}=unavailable")
|
||||
continue
|
||||
if hasattr(plugin, 'is_configured') and not plugin.is_configured():
|
||||
logger.info(f"Skipping {source_name} (not configured)")
|
||||
contributions.append(f"{source_name}=unconfigured")
|
||||
continue
|
||||
to_search.append((source_name, plugin))
|
||||
|
||||
async def _one(plugin):
|
||||
return await plugin.search(query, timeout, progress_callback)
|
||||
|
||||
results = await asyncio.gather(
|
||||
*[_one(plugin) for _, plugin in to_search],
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
for (source_name, _), result in zip(to_search, results, strict=True):
|
||||
if isinstance(result, Exception):
|
||||
logger.warning(f"{source_name} search failed: {result}")
|
||||
contributions.append(f"{source_name}=error")
|
||||
continue
|
||||
tracks, albums = result
|
||||
n = len(tracks) if tracks else 0
|
||||
if tracks:
|
||||
pooled_tracks.extend(tracks)
|
||||
if albums:
|
||||
pooled_albums.extend(albums)
|
||||
contributions.append(f"{source_name}={n}")
|
||||
|
||||
logger.info(
|
||||
"Best-quality pool: %d candidates [%s] for: %s",
|
||||
len(pooled_tracks), ', '.join(contributions), query,
|
||||
)
|
||||
return (pooled_tracks, pooled_albums)
|
||||
|
||||
async def download_with_fallback(self, username: str, filename: str,
|
||||
file_size: int, source_chain) -> Optional[str]:
|
||||
"""Try each source in ``source_chain`` until one accepts the
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ from config.settings import config_manager
|
|||
from core.download_engine import DownloadEngine
|
||||
from core.download_plugins.registry import DownloadPluginRegistry, build_default_registry
|
||||
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
|
||||
from core.quality.selection import load_search_mode
|
||||
|
||||
logger = get_logger("download_orchestrator")
|
||||
|
||||
|
|
@ -103,34 +102,7 @@ class DownloadOrchestrator:
|
|||
deezer_dl = self.client('deezer_dl')
|
||||
if deezer_arl and deezer_dl:
|
||||
deezer_dl.reconnect(deezer_arl)
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
deezer_dl._quality = quality_tier_for_source('deezer', default='flac')
|
||||
|
||||
# Reload Amazon quality preference (T2Tunes needs no reconnect — public proxy)
|
||||
amazon = self.client('amazon')
|
||||
if amazon:
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
quality = quality_tier_for_source('amazon', default='flac')
|
||||
amazon._quality = quality
|
||||
amazon._allow_fallback = config_manager.get('amazon_download.allow_fallback', True)
|
||||
if hasattr(amazon, '_client') and amazon._client:
|
||||
amazon._client.preferred_codec = quality
|
||||
|
||||
# Let registry-backed plugins refresh any config they cache at
|
||||
# construction time. This covers Prowlarr-backed torrent / usenet
|
||||
# clients without rebuilding the registry and losing active downloads.
|
||||
for name, client in self.registry.all_plugins():
|
||||
if not hasattr(client, 'reload_settings'):
|
||||
continue
|
||||
try:
|
||||
client.reload_settings()
|
||||
logger.info("%s client settings reloaded", self.registry.display_name(name))
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"%s client settings reload failed: %s",
|
||||
self.registry.display_name(name),
|
||||
exc,
|
||||
)
|
||||
deezer_dl._quality = config_manager.get('deezer_download.quality', 'flac')
|
||||
|
||||
# Reload download path for all clients that cache it.
|
||||
# Soulseek owns the path config and is reloaded above; every
|
||||
|
|
@ -143,10 +115,7 @@ class DownloadOrchestrator:
|
|||
continue
|
||||
if hasattr(client, 'download_path') and client.download_path != new_path:
|
||||
client.download_path = new_path
|
||||
try:
|
||||
client.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {new_path}: {e}")
|
||||
client.download_path.mkdir(parents=True, exist_ok=True)
|
||||
# YouTube also caches path in yt-dlp opts
|
||||
if hasattr(client, 'download_opts') and 'outtmpl' in client.download_opts:
|
||||
client.download_opts['outtmpl'] = str(new_path / '%(title)s.%(ext)s')
|
||||
|
|
@ -308,25 +277,11 @@ class DownloadOrchestrator:
|
|||
chain = ['soulseek']
|
||||
return chain
|
||||
|
||||
async def search(self, query: str, timeout: int = None, progress_callback=None,
|
||||
exclude_sources=None) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
async def search(self, query: str, timeout: int = None, progress_callback=None) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
"""Search for tracks using configured source(s). Single-source
|
||||
modes route directly; hybrid mode delegates to
|
||||
``engine.search_with_fallback`` which tries the chain in order.
|
||||
|
||||
``exclude_sources`` (optional) is an iterable of source names
|
||||
the caller wants filtered out of the hybrid chain. Used by the
|
||||
per-track download worker to skip torrent / usenet for album-
|
||||
context batches — those sources are release-level and don't
|
||||
score meaningfully on per-track titles; the album-bundle flow
|
||||
on the master worker handles them separately when they're the
|
||||
single active source."""
|
||||
``engine.search_with_fallback`` which tries the chain in order."""
|
||||
if self.mode != 'hybrid':
|
||||
# Single-source mode is opt-in; honour the user's choice even
|
||||
# if it's torrent/usenet on an album batch (the master worker
|
||||
# routes those through the album-bundle flow before per-track
|
||||
# tasks ever fire, so search() being called here would be a
|
||||
# non-album / wishlist / basic-search use case).
|
||||
client = self._client(self.mode)
|
||||
if not client:
|
||||
logger.error(f"{self.registry.display_name(self.mode)} client not available (failed to initialize)")
|
||||
|
|
@ -335,24 +290,6 @@ class DownloadOrchestrator:
|
|||
return await client.search(query, timeout, progress_callback)
|
||||
|
||||
chain = self._resolve_source_chain()
|
||||
if exclude_sources:
|
||||
blocked = {s.lower() for s in exclude_sources if s}
|
||||
filtered = [s for s in chain if s.lower() not in blocked]
|
||||
if filtered != chain:
|
||||
logger.info(
|
||||
"Hybrid search: excluding %s for this query (chain %s -> %s)",
|
||||
sorted(blocked & {s.lower() for s in chain}),
|
||||
" → ".join(chain), " → ".join(filtered) if filtered else "(empty)",
|
||||
)
|
||||
chain = filtered
|
||||
if not chain:
|
||||
logger.warning("Hybrid search exhausted: no eligible sources after exclusion filter")
|
||||
return [], []
|
||||
if load_search_mode() == 'best_quality':
|
||||
logger.info(f"Best-quality search ({' → '.join(chain)}): {query}")
|
||||
return await self.engine.search_all_sources(
|
||||
query, chain, timeout, progress_callback,
|
||||
)
|
||||
logger.info(f"Hybrid search ({' → '.join(chain)}): {query}")
|
||||
return await self.engine.search_with_fallback(query, chain, timeout, progress_callback)
|
||||
|
||||
|
|
@ -382,7 +319,7 @@ class DownloadOrchestrator:
|
|||
return None
|
||||
|
||||
# 2. Filter and validate results
|
||||
_streaming_sources = ('youtube', 'tidal', 'qobuz', 'hifi', 'deezer_dl', 'lidarr', 'soundcloud', 'amazon')
|
||||
_streaming_sources = ('youtube', 'tidal', 'qobuz', 'hifi', 'deezer_dl', 'lidarr', 'soundcloud')
|
||||
is_streaming = tracks[0].username in _streaming_sources if tracks else False
|
||||
|
||||
if is_streaming and expected_track:
|
||||
|
|
@ -423,17 +360,9 @@ class DownloadOrchestrator:
|
|||
|
||||
if scored:
|
||||
scored.sort(key=lambda x: x._match_confidence, reverse=True)
|
||||
# Match filter done (right track); now prefer the best quality
|
||||
# among the confidence-passing survivors so streaming isn't
|
||||
# quality-blind like Soulseek already isn't. Stable ranking
|
||||
# keeps confidence order within an equal quality tier; the
|
||||
# `or scored` fail-safe never leaves us with nothing to try.
|
||||
from core.quality.selection import rank_for_profile
|
||||
ranked, _ = rank_for_profile(scored)
|
||||
filtered_results = ranked or scored
|
||||
filtered_results = scored
|
||||
logger.info(f"Streaming validation: {len(scored)}/{len(tracks)} passed "
|
||||
f"(best: {scored[0]._match_confidence:.2f}, "
|
||||
f"quality pick: {filtered_results[0].audio_quality.label()})")
|
||||
f"(best: {scored[0]._match_confidence:.2f})")
|
||||
else:
|
||||
logger.warning(f"No streaming results passed validation for: {query}")
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -1,766 +0,0 @@
|
|||
"""Shared helpers for the album-bundle download flow.
|
||||
|
||||
The torrent and usenet download plugins both implement a
|
||||
``download_album_to_staging`` method that searches Prowlarr for a
|
||||
whole release, hands it to the active downloader, walks the
|
||||
resulting audio files, and copies them into the staging folder. The
|
||||
two implementations share the same release-picker heuristic and the
|
||||
same staging-path collision logic.
|
||||
|
||||
Pulled out of ``core/download_plugins/torrent.py`` so the usenet
|
||||
plugin doesn't have to import private helpers from a sibling
|
||||
plugin (Cin's "no leaky module boundaries" standard).
|
||||
|
||||
Also exposes ``atomic_copy_to_staging`` — the audio file is copied
|
||||
to a ``.tmp.<random>`` sidecar first and atomically renamed onto its
|
||||
final extension. The Auto-Import worker filters by audio extension
|
||||
so the in-flight ``.tmp`` file is never picked up mid-copy, closing
|
||||
the race between the album-bundle copy loop and Auto-Import's
|
||||
folder scan.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
import unicodedata
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Iterable, Optional
|
||||
|
||||
from config.settings import config_manager
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("download_plugins.album_bundle")
|
||||
|
||||
# Minimum album-title relevance a Prowlarr candidate must clear to be eligible
|
||||
# for an album-bundle download (#730). Prowlarr returns broad fuzzy matches — a
|
||||
# "Heroes" search also returns other Bowie albums — so without this gate the
|
||||
# most-popular result wins regardless of whether it's the right album. Below
|
||||
# this floor we refuse the bundle and let the caller fall back to per-track.
|
||||
_ALBUM_TITLE_RELEVANCE_FLOOR = 0.6
|
||||
|
||||
|
||||
# Album-pick size floor / ceiling. Single-track torrents (~10 MB)
|
||||
# are rejected when bigger candidates exist; anything past 3 GB is
|
||||
# treated as suspicious (multi-disc box-set + scans + extras).
|
||||
ALBUM_PICK_MIN_BYTES = 40 * 1024 * 1024
|
||||
ALBUM_PICK_MAX_BYTES = 3 * 1024 * 1024 * 1024
|
||||
|
||||
|
||||
# Quality-score weights for the album-pick heuristic. Mirrors the
|
||||
# tier order in ``core/imports/file_ops.py``'s ``quality_tiers`` —
|
||||
# higher number = preferred.
|
||||
_QUALITY_SCORE = {'flac': 4, 'ogg': 3, 'aac': 2, 'mp3': 1}
|
||||
|
||||
|
||||
# Default poll cadence + timeout for the album-download poll loop.
|
||||
# Both are overridable through config so users with slow trackers
|
||||
# / large box-sets can extend the deadline without editing code.
|
||||
DEFAULT_POLL_INTERVAL_SECONDS = 2.0
|
||||
DEFAULT_POLL_TIMEOUT_SECONDS = 6 * 60 * 60
|
||||
|
||||
|
||||
def get_poll_interval() -> float:
|
||||
"""Return the per-poll sleep duration (seconds). Configurable via
|
||||
``download_source.album_bundle_poll_interval_seconds``."""
|
||||
raw = config_manager.get('download_source.album_bundle_poll_interval_seconds',
|
||||
DEFAULT_POLL_INTERVAL_SECONDS)
|
||||
try:
|
||||
value = float(raw)
|
||||
if value > 0:
|
||||
return value
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return DEFAULT_POLL_INTERVAL_SECONDS
|
||||
|
||||
|
||||
def get_poll_timeout() -> float:
|
||||
"""Return the total deadline for an album-bundle download
|
||||
(seconds). Configurable via
|
||||
``download_source.album_bundle_timeout_seconds``."""
|
||||
raw = config_manager.get('download_source.album_bundle_timeout_seconds',
|
||||
DEFAULT_POLL_TIMEOUT_SECONDS)
|
||||
try:
|
||||
value = float(raw)
|
||||
if value > 0:
|
||||
return value
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return DEFAULT_POLL_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def quality_score(title: str, quality_guess) -> int:
|
||||
"""Map a release title's inferred quality to a sortable integer.
|
||||
|
||||
``quality_guess`` is the function from each plugin that maps a
|
||||
title string to a quality string ('flac' / 'mp3' / etc.) — passed
|
||||
in so this module doesn't have to import either plugin and risk
|
||||
a circular import."""
|
||||
return _QUALITY_SCORE.get(quality_guess(title) or '', 0)
|
||||
|
||||
|
||||
def _normalize_release_text(text: str) -> str:
|
||||
"""Lowercase, fold accents (Björk -> bjork), strip punctuation to spaces.
|
||||
|
||||
NFKD-decompose then drop combining marks so accented characters fold to
|
||||
their base letter instead of fragmenting (the naive approach turned
|
||||
'Björk' into 'bj rk'). Collapses runs of whitespace.
|
||||
"""
|
||||
if not text:
|
||||
return ""
|
||||
decomposed = unicodedata.normalize("NFKD", text)
|
||||
stripped = "".join(c for c in decomposed if not unicodedata.combining(c))
|
||||
lowered = stripped.lower()
|
||||
# Punctuation -> space (so "heroes" matches "heroes:" / "heroes -"),
|
||||
# then collapse whitespace.
|
||||
cleaned = re.sub(r"[^a-z0-9]+", " ", lowered)
|
||||
return re.sub(r"\s+", " ", cleaned).strip()
|
||||
|
||||
|
||||
# Edition / format / qualifier words that appear in stored album names or
|
||||
# release titles but say nothing about WHICH album it is. Stripped before
|
||||
# scoring so "Currents" matches "Currents (Deluxe)" and "Heroes" matches
|
||||
# "Heroes (2017 Remaster)" — the #730 fix must not reject the RIGHT album just
|
||||
# because the DB name carries an edition suffix the torrent title lacks.
|
||||
_ALBUM_NOISE_WORDS = frozenset({
|
||||
"deluxe", "edition", "remaster", "remastered", "remasters", "remix",
|
||||
"expanded", "anniversary", "bonus", "version", "explicit", "clean",
|
||||
"reissue", "special", "limited", "collectors", "collector", "the",
|
||||
"ep", "lp", "album", "single", "disc", "cd", "vol", "volume",
|
||||
"flac", "mp3", "aac", "ogg", "wav", "alac", "m4a", "320", "256", "192",
|
||||
"web", "vinyl", "hi", "res", "hires", "24bit", "16bit", "original",
|
||||
"soundtrack", "ost",
|
||||
})
|
||||
|
||||
|
||||
def _significant_words(normalized: str) -> list:
|
||||
"""Words that actually identify an album: drop pure-digit tokens (years,
|
||||
bitrates) and edition/format noise. Keeps at least the raw words if the
|
||||
filter would empty it (e.g. an album literally named '1989' or 'Deluxe')."""
|
||||
words = [w for w in normalized.split()
|
||||
if w not in _ALBUM_NOISE_WORDS and not w.isdigit()]
|
||||
return words or normalized.split()
|
||||
|
||||
|
||||
def album_title_relevance(candidate_title: str, album_name: str) -> float:
|
||||
"""How well a release title matches the requested album, 0.0–1.0.
|
||||
|
||||
Scores the fraction of the album's SIGNIFICANT words (edition/format/year
|
||||
noise removed) that appear as whole words in the candidate title.
|
||||
Word-boundary, not substring, so "Heroes" does NOT match "Superheroes" and
|
||||
a different album sharing no significant words scores 0 — while "Currents"
|
||||
still matches "Currents (Deluxe)" and "Heroes" matches the "2017 Remaster".
|
||||
|
||||
Returns 1.0 when there's no album name to check (can't gate on nothing —
|
||||
preserves old behavior for callers that don't pass a title).
|
||||
"""
|
||||
norm_album = _normalize_release_text(album_name)
|
||||
if not norm_album:
|
||||
return 1.0
|
||||
norm_title = _normalize_release_text(candidate_title)
|
||||
if not norm_title:
|
||||
return 0.0
|
||||
album_words = _significant_words(norm_album)
|
||||
title_words = set(norm_title.split())
|
||||
if not album_words:
|
||||
return 1.0
|
||||
matched = sum(1 for w in album_words if w in title_words)
|
||||
coverage = matched / len(album_words)
|
||||
# Full-phrase bonus (idea from contributor PR #731): when the album's core
|
||||
# phrase appears intact in the title, we're highly confident it's the right
|
||||
# release even if token-coverage is dragged down by a long multi-word name.
|
||||
# MUST be word-boundary anchored, NOT a raw substring — a naive
|
||||
# `phrase in norm_title` lets "heroes" match "superheroes" and reintroduces
|
||||
# the exact wrong-album bug #730 fixes (PR #731's version has this flaw).
|
||||
core_phrase = " ".join(album_words)
|
||||
if core_phrase and re.search(rf"(?:^| ){re.escape(core_phrase)}(?: |$)", norm_title):
|
||||
coverage = max(coverage, 0.9)
|
||||
return coverage
|
||||
|
||||
|
||||
def pick_best_album_release(candidates, quality_guess,
|
||||
album_name: str = "") -> Optional[object]:
|
||||
"""Pick the single best torrent / NZB for an album-bundle download.
|
||||
|
||||
Heuristic, in priority order:
|
||||
0. Album-TITLE relevance gate (#730): drop candidates whose title doesn't
|
||||
sufficiently match the requested album. Prowlarr returns broad fuzzy
|
||||
matches, so without this the most-popular result wins even when it's a
|
||||
different album. When ``album_name`` is given and NOTHING clears the
|
||||
relevance floor, return None — the caller then falls back to per-track
|
||||
rather than downloading a confident mismatch.
|
||||
1. Reasonable album-ish size (40 MB – 3 GB) — drops single-track
|
||||
releases that snuck in and quarantines suspicious giants.
|
||||
2. Higher seeders > lower (dead torrents = dead downloads).
|
||||
Usenet releases use ``grabs`` as a popularity proxy when
|
||||
seeders is None.
|
||||
3. Higher quality (FLAC > AAC > MP3) inferred from title.
|
||||
4. Larger size as tiebreaker (often = higher bitrate).
|
||||
"""
|
||||
if not candidates:
|
||||
return None
|
||||
|
||||
# 0. Title-relevance gate. Only applied when we know the album name; with
|
||||
# no name we can't judge relevance, so we don't gate (old behavior).
|
||||
if album_name:
|
||||
relevant = [
|
||||
c for c in candidates
|
||||
if album_title_relevance(c.title or "", album_name) >= _ALBUM_TITLE_RELEVANCE_FLOOR
|
||||
]
|
||||
if not relevant:
|
||||
logger.warning(
|
||||
"[Album Bundle] No candidate cleared the title-relevance floor "
|
||||
"for '%s' (%d candidates rejected as wrong album) — refusing the "
|
||||
"bundle so the caller falls back to per-track.",
|
||||
album_name, len(candidates),
|
||||
)
|
||||
return None
|
||||
candidates = relevant
|
||||
|
||||
sized = [c for c in candidates
|
||||
if ALBUM_PICK_MIN_BYTES <= (c.size or 0) <= ALBUM_PICK_MAX_BYTES]
|
||||
pool = sized or list(candidates)
|
||||
if not pool:
|
||||
return None
|
||||
|
||||
def _score(c) -> tuple:
|
||||
seeders = c.seeders if c.seeders is not None else (c.grabs or 0)
|
||||
return (seeders, quality_score(c.title or '', quality_guess), c.size or 0)
|
||||
|
||||
return max(pool, key=_score)
|
||||
|
||||
|
||||
def unique_staging_path(staging_dir: Path, src: Path) -> Path:
|
||||
"""Return a destination path inside ``staging_dir`` that doesn't
|
||||
collide with an existing file. Appends ``_1``, ``_2``, ... before
|
||||
the extension when needed; gives up after 1000 candidates and
|
||||
returns the unsuffixed path so the caller will overwrite (better
|
||||
than infinite loop or crash)."""
|
||||
dest = staging_dir / src.name
|
||||
if not dest.exists():
|
||||
return dest
|
||||
stem = dest.stem
|
||||
suffix = dest.suffix
|
||||
for i in range(1, 1000):
|
||||
candidate = staging_dir / f"{stem}_{i}{suffix}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
return dest
|
||||
|
||||
|
||||
def atomic_copy_to_staging(src: Path, dest: Path) -> bool:
|
||||
"""Copy ``src`` to ``dest`` without exposing a partial file to
|
||||
folder scanners.
|
||||
|
||||
The Auto-Import worker filters by audio extension when scanning
|
||||
Staging — see ``AUDIO_EXTENSIONS`` in ``core/auto_import_worker.py``.
|
||||
Naming the in-flight file ``<dest>.tmp.<random>`` keeps it
|
||||
invisible until the rename atomically swings it to its final
|
||||
extension. ``os.replace`` (used by ``Path.rename`` on Python 3.x)
|
||||
is atomic on the same filesystem, so Auto-Import either sees the
|
||||
file at its final name (complete) or doesn't see it at all
|
||||
(in flight).
|
||||
|
||||
Returns True on success, False on copy / rename failure. Caller
|
||||
is expected to log the failure case so we don't double-log here.
|
||||
"""
|
||||
tmp = dest.with_name(f"{dest.name}.tmp.{uuid.uuid4().hex[:8]}")
|
||||
try:
|
||||
shutil.copy2(src, tmp)
|
||||
except Exception:
|
||||
# Best-effort cleanup of the partial file. If unlink fails
|
||||
# (locked, permissions) we leave it — Auto-Import ignores it
|
||||
# anyway because of the .tmp extension.
|
||||
try:
|
||||
if tmp.exists():
|
||||
tmp.unlink()
|
||||
except Exception as cleanup_exc:
|
||||
logger.debug("album_bundle tmp cleanup failed: %s", cleanup_exc)
|
||||
raise
|
||||
try:
|
||||
tmp.replace(dest)
|
||||
return True
|
||||
except Exception:
|
||||
try:
|
||||
tmp.unlink(missing_ok=True)
|
||||
except Exception as cleanup_exc:
|
||||
logger.debug("album_bundle tmp cleanup failed: %s", cleanup_exc)
|
||||
raise
|
||||
|
||||
|
||||
# Number of consecutive None-status reads tolerated before treating the
|
||||
# job as gone. Sized for the SAB queue→history transition window: SAB
|
||||
# removes the slot from the queue before adding it to history, and on a
|
||||
# busy server (par2 verify + unrar) that window can be several poll
|
||||
# intervals. At the default 2s interval, 5 retries = ~10s of tolerance
|
||||
# before we give up and emit a terminal failure. Override via
|
||||
# ``download_source.album_bundle_transient_miss_threshold`` for users
|
||||
# whose servers need more headroom (very large multi-disc box sets,
|
||||
# slow disks, etc.).
|
||||
DEFAULT_TRANSIENT_MISS_THRESHOLD = 5
|
||||
|
||||
|
||||
def get_transient_miss_threshold() -> int:
|
||||
"""Return the configured transient-miss threshold for poll loops."""
|
||||
raw = config_manager.get('download_source.album_bundle_transient_miss_threshold',
|
||||
DEFAULT_TRANSIENT_MISS_THRESHOLD)
|
||||
try:
|
||||
value = int(raw)
|
||||
if value > 0:
|
||||
return value
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return DEFAULT_TRANSIENT_MISS_THRESHOLD
|
||||
|
||||
|
||||
# How long to keep polling after the client reports terminal success
|
||||
# but hasn't yet exposed a final save_path. Distinct from the
|
||||
# transient-miss threshold because the two model different things:
|
||||
# a transient miss is "the job vanished — fail fast (~10s) so a deleted
|
||||
# job doesn't hang"; a completed-no-path read is "the download SUCCEEDED
|
||||
# and the files are on disk — SAB just hasn't finished writing the
|
||||
# ``storage`` field." The #706 fix reused the 5-poll (~10s) miss window
|
||||
# here, but #721's own report shows SAB can take 2+ minutes (or, on some
|
||||
# versions, never expose ``storage`` at all) — so a 10s window false-fails
|
||||
# a download that actually completed. Expressed in SECONDS (converted to
|
||||
# a poll count against the live interval) so it's interval-independent.
|
||||
# Override via ``download_source.album_bundle_completed_no_path_seconds``.
|
||||
DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS = 120.0
|
||||
|
||||
|
||||
def get_completed_no_path_window_seconds() -> float:
|
||||
"""Return the completed-but-no-save_path tolerance window (seconds)."""
|
||||
raw = config_manager.get('download_source.album_bundle_completed_no_path_seconds',
|
||||
DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS)
|
||||
try:
|
||||
value = float(raw)
|
||||
if value > 0:
|
||||
return value
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS
|
||||
|
||||
|
||||
class TransientMissCounter:
|
||||
"""Bounded retry counter for adapter status reads.
|
||||
|
||||
Both the album-bundle poll (in ``poll_album_download``) and the
|
||||
per-track download threads in ``usenet.py`` / ``torrent.py`` need
|
||||
the same "tolerate N consecutive missing or unmapped reads before
|
||||
declaring the job gone" logic. Lifted into one class so the rule
|
||||
is in one place and unit-testable in isolation — the per-track
|
||||
paths used to carry inline counters that mirrored this logic by
|
||||
hand, which is exactly the kind of duplication that drifts."""
|
||||
|
||||
def __init__(self, threshold: Optional[int] = None) -> None:
|
||||
self.threshold = threshold if threshold is not None else get_transient_miss_threshold()
|
||||
self.misses = 0
|
||||
|
||||
def record_miss(self) -> bool:
|
||||
"""Bump the miss counter. Returns True when the counter has
|
||||
reached the threshold (caller should give up)."""
|
||||
self.misses += 1
|
||||
return self.misses >= self.threshold
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Successful read — reset the counter back to zero."""
|
||||
self.misses = 0
|
||||
|
||||
|
||||
def poll_album_download(
|
||||
*,
|
||||
get_status: Callable[[], Optional[Any]],
|
||||
title: str,
|
||||
emit: Callable[..., None],
|
||||
complete_states: frozenset,
|
||||
failed_states: frozenset = frozenset(['failed']),
|
||||
is_shutdown: Optional[Callable[[], bool]] = None,
|
||||
transient_miss_threshold: int = DEFAULT_TRANSIENT_MISS_THRESHOLD,
|
||||
completed_no_path_threshold: Optional[int] = None,
|
||||
poll_interval: Optional[float] = None,
|
||||
timeout: Optional[float] = None,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
monotonic: Callable[[], float] = time.monotonic,
|
||||
log_prefix: str = '[album_bundle]',
|
||||
) -> Optional[str]:
|
||||
"""Drive the per-poll status loop for an album-bundle download.
|
||||
|
||||
Lifted out of ``UsenetDownloadPlugin._poll_album_download`` and the
|
||||
sibling torrent method so the loop is testable in isolation and so
|
||||
both plugins share the same exit semantics.
|
||||
|
||||
Contract:
|
||||
- ``get_status()`` returns the adapter status object for the bound
|
||||
job, ``None`` when the client doesn't know about the job
|
||||
currently (transient or terminal — disambiguated by retry count).
|
||||
- ``emit(state, **fields)`` is the plugin's progress callback —
|
||||
this function calls it on EVERY successful poll with
|
||||
``state='downloading'`` and ALWAYS calls it once more with
|
||||
``state='failed'`` before returning ``None`` on any failure path,
|
||||
so the UI doesn't freeze on the last 'downloading' emit.
|
||||
- ``complete_states`` is the adapter's terminal-success set
|
||||
('completed' alone for usenet; 'seeding' + 'completed' for
|
||||
torrent because seeding-but-files-on-disk also counts).
|
||||
- ``failed_states`` is the explicit-failure set. The adapter-level
|
||||
'error' (unmapped state default) is intentionally NOT in here —
|
||||
that's treated as a transient miss because a real SAB / NZBGet
|
||||
/ qBit never returns a literal 'error' state on a healthy job;
|
||||
it's only our default fallback for unknown queue strings. Real
|
||||
example: SAB's 'Pp' post-processing state was unmapped → became
|
||||
'error' → poll infinite-looped until the 6-hour timeout.
|
||||
- ``transient_miss_threshold`` is the number of consecutive None /
|
||||
'error' reads tolerated before declaring the job gone. Sized for
|
||||
the SAB queue→history gap window (~10s) — a vanished job should
|
||||
fail fast.
|
||||
- ``completed_no_path_threshold`` is a SEPARATE, longer window for
|
||||
the "client says complete but no save_path yet" case. The download
|
||||
already succeeded, so this defaults to ~120s (configurable via
|
||||
``download_source.album_bundle_completed_no_path_seconds``) instead
|
||||
of reusing the 10s miss window — #721 showed SAB can take 2+ minutes
|
||||
to write ``storage``. When the window is exhausted the loop falls
|
||||
back to the adapter's ``incomplete_path`` (the on-disk in-progress
|
||||
dir) if present, and only emits terminal ``failed`` when there's no
|
||||
path of any kind to scan.
|
||||
|
||||
Returns the adapter's reported save_path (or, as a last resort, its
|
||||
``incomplete_path``) on terminal success, or ``None`` on any failure
|
||||
(timeout / disappeared / explicit failed / shutdown). On every
|
||||
failure path emits ``'failed'`` once with an ``error`` field
|
||||
describing why.
|
||||
"""
|
||||
interval = poll_interval if poll_interval is not None else get_poll_interval()
|
||||
deadline = monotonic() + (timeout if timeout is not None else get_poll_timeout())
|
||||
last_save_path: Optional[str] = None
|
||||
last_incomplete_path: Optional[str] = None
|
||||
misses = TransientMissCounter(transient_miss_threshold)
|
||||
# Separate counter for "client reports terminal-success state but no
|
||||
# save_path field has landed yet." SAB History flips ``status`` to
|
||||
# 'Completed' a few seconds before its post-processing pipeline
|
||||
# writes the final ``storage`` field — see issue #721 (Forty Licks
|
||||
# stuck at 61%): SAB shows Completed in the UI, but
|
||||
# ``_parse_history_slot`` returns ``save_path=None`` for those few
|
||||
# seconds because ``storage`` isn't populated yet. Pre-fix the
|
||||
# poll returned ``None`` on the first such read, the bundle
|
||||
# plugin marked the batch failed, but the UI still displayed the
|
||||
# last ``downloading`` progress emit.
|
||||
#
|
||||
# This window is intentionally LONGER than the transient-miss window:
|
||||
# the download already SUCCEEDED, so being patient here is cheap and
|
||||
# correct, whereas the original 5-poll (~10s) reuse false-failed real
|
||||
# completions (#721 reported SAB taking 2+ minutes). Default ~120s,
|
||||
# converted from seconds to a poll count against the live interval.
|
||||
if completed_no_path_threshold is None:
|
||||
completed_no_path_threshold = max(
|
||||
transient_miss_threshold,
|
||||
int(get_completed_no_path_window_seconds() / max(interval, 0.001)) or 1,
|
||||
)
|
||||
completed_no_path_misses = TransientMissCounter(completed_no_path_threshold)
|
||||
|
||||
def _fail(reason: str) -> None:
|
||||
try:
|
||||
emit('failed', release=title, error=reason)
|
||||
except Exception as cb_exc:
|
||||
logger.debug("%s terminal emit failed: %s", log_prefix, cb_exc)
|
||||
|
||||
# Heartbeat so the otherwise-silent download loop is diagnosable.
|
||||
# The loop emits progress to the UI on every poll but logs nothing
|
||||
# during normal operation — which made the #721 "stuck at N%" reports
|
||||
# impossible to triage from logs alone (we couldn't tell if the poll
|
||||
# was alive, what state SAB returned, or whether it had wedged). Log
|
||||
# the raw adapter read at most once per heartbeat interval.
|
||||
HEARTBEAT_SECONDS = 30.0
|
||||
last_heartbeat = monotonic()
|
||||
poll_count = 0
|
||||
|
||||
while monotonic() < deadline:
|
||||
if is_shutdown and is_shutdown():
|
||||
# Shutdown is a clean exit — don't paint failure on the UI;
|
||||
# the app is going away anyway.
|
||||
return None
|
||||
|
||||
try:
|
||||
status = get_status()
|
||||
except Exception as e:
|
||||
logger.warning("%s Poll error: %s", log_prefix, e)
|
||||
status = None
|
||||
|
||||
poll_count += 1
|
||||
now = monotonic()
|
||||
if now - last_heartbeat >= HEARTBEAT_SECONDS:
|
||||
last_heartbeat = now
|
||||
if status is None:
|
||||
logger.info("%s '%s' poll #%d: client returned no status (miss %d/%d)",
|
||||
log_prefix, title, poll_count, misses.misses, misses.threshold)
|
||||
else:
|
||||
logger.info(
|
||||
"%s '%s' poll #%d: state=%r progress=%.2f save_path=%r",
|
||||
log_prefix, title, poll_count,
|
||||
getattr(status, 'state', None), getattr(status, 'progress', 0.0) or 0.0,
|
||||
getattr(status, 'save_path', None),
|
||||
)
|
||||
|
||||
if status is None:
|
||||
if misses.record_miss():
|
||||
logger.error(
|
||||
"%s '%s' missing from client for %d consecutive polls — giving up",
|
||||
log_prefix, title, misses.misses,
|
||||
)
|
||||
_fail('Disappeared from client (no status after retries)')
|
||||
return None
|
||||
sleep(interval)
|
||||
continue
|
||||
|
||||
# Reset the miss counter only when the adapter returned a state
|
||||
# we actually recognise. The default-fallback 'error' is treated
|
||||
# as a continuing transient miss below, so it must NOT reset
|
||||
# here — otherwise a persistently-unmapped state loops forever.
|
||||
if status.state != 'error':
|
||||
misses.reset()
|
||||
|
||||
emit('downloading', progress=status.progress, downloaded=status.downloaded,
|
||||
speed=status.download_speed)
|
||||
if status.save_path:
|
||||
last_save_path = status.save_path
|
||||
# Remember the in-progress dir too — never used on a normal
|
||||
# completion, only as the last-resort fallback below when the
|
||||
# final save_path provably never lands.
|
||||
incomplete_path = getattr(status, 'incomplete_path', None)
|
||||
if incomplete_path:
|
||||
last_incomplete_path = incomplete_path
|
||||
|
||||
if status.state in complete_states:
|
||||
if last_save_path:
|
||||
completed_no_path_misses.reset()
|
||||
return last_save_path
|
||||
# Terminal-success state but no save_path landed yet.
|
||||
# SAB History flips ``Completed`` a few seconds before
|
||||
# ``storage`` is populated — give the adapter a generous
|
||||
# window before declaring this a hard failure. Without this
|
||||
# tolerance, every TAR / unrar-bearing usenet release
|
||||
# would race the path-write window and randomly fail.
|
||||
if completed_no_path_misses.record_miss():
|
||||
# Last resort before failing: SAB finished and the files
|
||||
# are physically on disk (#721), but the final ``storage``
|
||||
# field never landed. Fall back to the in-progress dir so
|
||||
# the bundle can still scan + stage the audio, rather than
|
||||
# leaving the user stuck with a completed-in-SAB download
|
||||
# that SoulSync never imports.
|
||||
if last_incomplete_path:
|
||||
logger.warning(
|
||||
"%s '%s' completed on the client but never exposed a final "
|
||||
"save_path after %d polls — falling back to the in-progress "
|
||||
"path %r as a last resort. If staging fails, the SAB job "
|
||||
"likely needs its post-process move to finish first.",
|
||||
log_prefix, title, completed_no_path_misses.misses,
|
||||
last_incomplete_path,
|
||||
)
|
||||
return last_incomplete_path
|
||||
logger.error(
|
||||
"%s '%s' reported terminal success but no save_path landed "
|
||||
"after %d consecutive polls — bundle cannot stage. Adapter "
|
||||
"may need new history-slot fallback fields (storage / path "
|
||||
"/ download_path / dirname). Last status: state=%r progress=%r",
|
||||
log_prefix, title, completed_no_path_misses.misses,
|
||||
status.state, status.progress,
|
||||
)
|
||||
_fail('Client reported success but never provided a save_path')
|
||||
return None
|
||||
logger.info(
|
||||
"%s '%s' is %s on the client but save_path not yet set — "
|
||||
"retrying (poll %d/%d)",
|
||||
log_prefix, title, status.state,
|
||||
completed_no_path_misses.misses, completed_no_path_misses.threshold,
|
||||
)
|
||||
sleep(interval)
|
||||
continue
|
||||
if status.state in failed_states:
|
||||
error = getattr(status, 'error', None) or 'Client reported failure'
|
||||
logger.error("%s '%s' failed: %s", log_prefix, title, error)
|
||||
_fail(error)
|
||||
return None
|
||||
if status.state == 'error':
|
||||
# Unmapped adapter state — see contract docstring. Warn so
|
||||
# we hear about new states the adapter map needs to grow
|
||||
# without breaking the user's download. The miss counter
|
||||
# was intentionally NOT reset above for this branch.
|
||||
logger.warning(
|
||||
"%s '%s' returned unmapped state — treating as transient",
|
||||
log_prefix, title,
|
||||
)
|
||||
if misses.record_miss():
|
||||
_fail('Client returned unmapped state repeatedly')
|
||||
return None
|
||||
|
||||
sleep(interval)
|
||||
|
||||
logger.error("%s '%s' timed out", log_prefix, title)
|
||||
_fail('Download timed out')
|
||||
return None
|
||||
|
||||
|
||||
def _candidate_download_roots(config_get: Callable[..., Any]) -> list:
|
||||
"""Directories where THIS process can read finished downloads — used by
|
||||
``resolve_reported_save_path`` for the basename fallback.
|
||||
|
||||
Order matters: most-specific usenet/torrent roots first, then the
|
||||
general Soulseek download / transfer dirs, which in the standard
|
||||
shared-volume arr setup are bind-mounted to the very directory the
|
||||
usenet client writes its completed downloads into. Relative values
|
||||
(e.g. ``./downloads``) resolve against the process CWD — the
|
||||
container's ``/app`` — which is exactly where those mounts live.
|
||||
"""
|
||||
roots: list = []
|
||||
for key in (
|
||||
'download_source.usenet_download_path',
|
||||
'usenet_client.completed_path',
|
||||
'usenet_client.download_path',
|
||||
'download_source.torrent_download_path',
|
||||
'soulseek.download_path',
|
||||
'soulseek.transfer_path',
|
||||
):
|
||||
value = config_get(key, None)
|
||||
if value:
|
||||
roots.append(str(value))
|
||||
seen: set = set()
|
||||
out: list = []
|
||||
for root in roots:
|
||||
if root not in seen:
|
||||
seen.add(root)
|
||||
out.append(root)
|
||||
return out
|
||||
|
||||
|
||||
def resolve_reported_save_path(
|
||||
reported_path: Optional[str],
|
||||
config_get: Optional[Callable[..., Any]] = None,
|
||||
) -> Optional[str]:
|
||||
"""Translate a downloader-reported save_path into one THIS process can read.
|
||||
|
||||
Usenet / torrent clients report paths from inside THEIR OWN container
|
||||
(e.g. SAB hands back ``/data/downloads/music/<album>``); SoulSync often
|
||||
mounts the very same files at a different point (``/app/downloads/<album>``).
|
||||
Feeding the client's path straight to the audio walker then yields
|
||||
"No audio files found" even though the files are physically present —
|
||||
the classic arr-stack remote-path mismatch.
|
||||
|
||||
Resolution order:
|
||||
1. The reported path verbatim, if it's a readable directory here
|
||||
(deployments that mirror the client's mount paths).
|
||||
2. Explicit prefix mappings from ``download_source.usenet_path_mappings``
|
||||
— a list of ``{"from": "...", "to": "..."}`` (Sonarr/Radarr-style
|
||||
remote path mapping) for non-shared / oddly-mounted layouts.
|
||||
3. Basename fallback: a same-named folder under a known SoulSync
|
||||
download root. Zero-config for the standard shared-volume setup —
|
||||
the album folder shows up under SoulSync's own ``./downloads``
|
||||
mount with the same name the client reported.
|
||||
|
||||
Returns the best resolved path, or ``reported_path`` unchanged when
|
||||
nothing better is found (so the caller's existing "no audio" error still
|
||||
surfaces, with both paths logged).
|
||||
"""
|
||||
if not reported_path:
|
||||
return reported_path
|
||||
if config_get is None:
|
||||
config_get = config_manager.get
|
||||
|
||||
def _is_dir(candidate) -> bool:
|
||||
try:
|
||||
return Path(candidate).is_dir()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
# 1. Reported path is directly readable — mounts already line up.
|
||||
if _is_dir(reported_path):
|
||||
return reported_path
|
||||
|
||||
normalized = str(reported_path).replace('\\', '/')
|
||||
|
||||
# 2. Explicit prefix mappings (remote-path-mapping escape hatch).
|
||||
mappings = config_get('download_source.usenet_path_mappings', None) or []
|
||||
if isinstance(mappings, (list, tuple)):
|
||||
for mapping in mappings:
|
||||
if not isinstance(mapping, dict):
|
||||
continue
|
||||
frm = str(mapping.get('from') or '').replace('\\', '/').rstrip('/')
|
||||
to = str(mapping.get('to') or '')
|
||||
if not frm or not to:
|
||||
continue
|
||||
if normalized == frm or normalized.startswith(frm + '/'):
|
||||
rest = normalized[len(frm):].lstrip('/')
|
||||
candidate = str(Path(to) / rest) if rest else to
|
||||
if _is_dir(candidate):
|
||||
return candidate
|
||||
|
||||
# 3. Basename fallback under known download roots — covers the standard
|
||||
# shared-volume layout with zero configuration.
|
||||
basename = Path(normalized).name
|
||||
if basename:
|
||||
for root in _candidate_download_roots(config_get):
|
||||
candidate = Path(root) / basename
|
||||
if _is_dir(candidate):
|
||||
return str(candidate)
|
||||
|
||||
return reported_path
|
||||
|
||||
|
||||
def copy_audio_files_atomically(
|
||||
sources: Iterable[Path], staging_dir: Path, remove_source: bool = False,
|
||||
) -> list:
|
||||
"""Convenience wrapper: pick a non-colliding staging path for
|
||||
each source, copy via ``atomic_copy_to_staging``. Returns the
|
||||
list of final destination paths (as strings). Files that fail
|
||||
to copy are logged and skipped; the caller decides what to do
|
||||
with a partial result.
|
||||
|
||||
``remove_source=True`` deletes each source AFTER it copies
|
||||
successfully — used by the Soulseek bundle path so slskd's
|
||||
completed downloads don't pile up in its download folder (#796).
|
||||
Kept False for torrent/usenet, whose clients must retain the
|
||||
originals (seeding / client-managed)."""
|
||||
staging_dir.mkdir(parents=True, exist_ok=True)
|
||||
out: list = []
|
||||
for src in sources:
|
||||
dest = unique_staging_path(staging_dir, src)
|
||||
try:
|
||||
atomic_copy_to_staging(src, dest)
|
||||
out.append(str(dest))
|
||||
if remove_source:
|
||||
# Only after a verified copy — never lose data on a failed stage.
|
||||
try:
|
||||
Path(src).unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.debug("[album_bundle] Could not remove staged source %s: %s", src, e)
|
||||
except Exception as e:
|
||||
logger.warning("[album_bundle] Failed to stage %s -> %s: %s", src, dest, e)
|
||||
return out
|
||||
|
||||
|
||||
# Re-export so callers don't have to remember which module owns
|
||||
# what. The ``time`` import is kept so plugins can ``from
|
||||
# core.download_plugins.album_bundle import time`` if they want to,
|
||||
# avoiding a second std-lib import line for a single use.
|
||||
__all__ = [
|
||||
"ALBUM_PICK_MIN_BYTES",
|
||||
"ALBUM_PICK_MAX_BYTES",
|
||||
"DEFAULT_POLL_INTERVAL_SECONDS",
|
||||
"DEFAULT_POLL_TIMEOUT_SECONDS",
|
||||
"DEFAULT_TRANSIENT_MISS_THRESHOLD",
|
||||
"DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS",
|
||||
"TransientMissCounter",
|
||||
"atomic_copy_to_staging",
|
||||
"copy_audio_files_atomically",
|
||||
"get_completed_no_path_window_seconds",
|
||||
"get_poll_interval",
|
||||
"get_poll_timeout",
|
||||
"get_transient_miss_threshold",
|
||||
"resolve_reported_save_path",
|
||||
"pick_best_album_release",
|
||||
"poll_album_download",
|
||||
"quality_score",
|
||||
"time",
|
||||
"unique_staging_path",
|
||||
]
|
||||
|
|
@ -38,10 +38,7 @@ from core.download_plugins.base import DownloadSourcePlugin
|
|||
# than the legacy module-top imports here. Importing everything at
|
||||
# registry-load time pins the bindings the same way the legacy
|
||||
# orchestrator did.
|
||||
from core.amazon_download_client import AmazonDownloadClient
|
||||
from core.deezer_download_client import DeezerDownloadClient
|
||||
from core.download_plugins.torrent import TorrentDownloadPlugin
|
||||
from core.download_plugins.usenet import UsenetDownloadPlugin
|
||||
from core.hifi_client import HiFiClient
|
||||
from core.lidarr_download_client import LidarrDownloadClient
|
||||
from core.qobuz_client import QobuzClient
|
||||
|
|
@ -179,7 +176,6 @@ def build_default_registry() -> DownloadPluginRegistry:
|
|||
"""
|
||||
registry = DownloadPluginRegistry()
|
||||
|
||||
registry.register(PluginSpec(name='amazon', factory=AmazonDownloadClient, display_name='Amazon Music'))
|
||||
registry.register(PluginSpec(name='soulseek', factory=SoulseekClient, display_name='Soulseek'))
|
||||
registry.register(PluginSpec(name='youtube', factory=YouTubeClient, display_name='YouTube'))
|
||||
registry.register(PluginSpec(name='tidal', factory=TidalDownloadClient, display_name='Tidal'))
|
||||
|
|
@ -192,7 +188,5 @@ def build_default_registry() -> DownloadPluginRegistry:
|
|||
aliases=('deezer_dl',)))
|
||||
registry.register(PluginSpec(name='lidarr', factory=LidarrDownloadClient, display_name='Lidarr'))
|
||||
registry.register(PluginSpec(name='soundcloud',factory=SoundcloudClient, display_name='SoundCloud'))
|
||||
registry.register(PluginSpec(name='torrent', factory=TorrentDownloadPlugin, display_name='Torrent (Prowlarr)'))
|
||||
registry.register(PluginSpec(name='usenet', factory=UsenetDownloadPlugin, display_name='Usenet (Prowlarr)'))
|
||||
|
||||
return registry
|
||||
|
|
|
|||
|
|
@ -1,764 +0,0 @@
|
|||
"""TorrentDownloadPlugin — composes Prowlarr search + torrent client
|
||||
adapter + archive_pipeline into a uniform download source.
|
||||
|
||||
Two flows:
|
||||
|
||||
**Per-track flow** (basic search, single-track wishlist) —
|
||||
1. ``search(query)`` calls ``ProwlarrClient.search`` filtered to
|
||||
``protocol='torrent'`` results, projects releases into
|
||||
``TrackResult`` / ``AlbumResult`` shaped objects the existing
|
||||
search UI already understands. Encodes the indexer's
|
||||
``downloadUrl`` (or magnet URI) into the filename so
|
||||
``download()`` can recover it.
|
||||
2. ``download(username, filename, ...)`` decodes the URL, asks the
|
||||
active torrent adapter (qBittorrent, Transmission, or Deluge per
|
||||
user's settings) to add it, spawns a background thread that
|
||||
polls the adapter for completion.
|
||||
3. On completion the thread walks the adapter-reported save path
|
||||
via ``archive_pipeline.collect_audio_after_extraction`` and
|
||||
exposes the full audio-file list. Post-processing can then pick
|
||||
the requested track from a completed release instead of importing
|
||||
the first file blindly.
|
||||
|
||||
**Album-bundle flow** (album-context batch downloads — wired in
|
||||
``core/downloads/master.py``) —
|
||||
4. ``download_album_to_staging(album, artist, staging_dir)`` does
|
||||
ONE Prowlarr search for the whole release, picks the best
|
||||
torrent (prefers FLAC, decent seeders, reasonable size),
|
||||
downloads it, extracts archives if needed, copies every audio
|
||||
file into the staging directory. The existing per-track
|
||||
``try_staging_match`` flow then finds + imports each track by
|
||||
fuzzy title match against the staged files. Per-track Prowlarr
|
||||
queries never fire — track titles like "Luther (with SZA)"
|
||||
would match album torrents like "GNX (2024) [FLAC]" at near-
|
||||
zero confidence and break the per-track dispatch.
|
||||
|
||||
Limitations:
|
||||
- ``save_path`` is the torrent client's view of the disk. If
|
||||
SoulSync runs on a different host than qBit / Trans / Deluge,
|
||||
the post-processing pipeline can't see those files. The plugin
|
||||
works fine for the all-on-one-box case (most users); remote
|
||||
setups will need a future sync step (rclone / SMB / Docker
|
||||
bind mount).
|
||||
- Track-level metadata isn't available until after download.
|
||||
Search results carry only the release title + indexer metadata;
|
||||
individual track names are populated when the matching pipeline
|
||||
walks the extracted audio files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from config.settings import config_manager
|
||||
from core.archive_pipeline import collect_audio_after_extraction
|
||||
from core.download_plugins.album_bundle import (
|
||||
TransientMissCounter,
|
||||
copy_audio_files_atomically,
|
||||
get_poll_interval,
|
||||
get_poll_timeout,
|
||||
pick_best_album_release,
|
||||
poll_album_download,
|
||||
resolve_reported_save_path,
|
||||
)
|
||||
from core.download_plugins.base import DownloadSourcePlugin
|
||||
from core.download_plugins.torrent_stall import (
|
||||
StallTracker,
|
||||
get_stall_action,
|
||||
get_stall_timeout,
|
||||
)
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.prowlarr_client import (
|
||||
DEFAULT_MUSIC_CATEGORIES,
|
||||
ProwlarrClient,
|
||||
ProwlarrSearchResult,
|
||||
)
|
||||
from core.torrent_clients import get_active_adapter as get_active_torrent_adapter
|
||||
from utils.async_helpers import run_async
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("download_plugins.torrent")
|
||||
|
||||
|
||||
# Separator used to encode the download URL inside the filename
|
||||
# field. Same convention Lidarr / YouTube use for embedding their
|
||||
# own opaque identifiers — ``<download_url>||<display>``.
|
||||
_FILENAME_SEP = '||'
|
||||
|
||||
# Adapter states that count as the download being on-disk and
|
||||
# safe to walk. ``seeding`` and ``completed`` both mean the
|
||||
# bits are there; the user can pause seeding manually if they
|
||||
# don't want to keep sharing.
|
||||
_COMPLETE_STATES = frozenset(['seeding', 'completed'])
|
||||
|
||||
# Poll cadence / timeout — both pull from config via the shared
|
||||
# album_bundle helpers so users can extend the deadline for slow
|
||||
# trackers without editing source. Kept as module aliases so the
|
||||
# per-track flow at the bottom of this file can still import them
|
||||
# under the legacy names without re-reading config every loop.
|
||||
_POLL_TIMEOUT_SECONDS = get_poll_timeout()
|
||||
_POLL_INTERVAL_SECONDS = get_poll_interval()
|
||||
|
||||
|
||||
class TorrentDownloadPlugin(DownloadSourcePlugin):
|
||||
"""Torrent download source backed by Prowlarr + an active
|
||||
torrent client adapter."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._prowlarr = ProwlarrClient()
|
||||
# Track every download we've kicked off. Keyed by our own
|
||||
# uuid — NOT the adapter's hash — because the orchestrator
|
||||
# owns the lifecycle and we need a stable id even before
|
||||
# the adapter has assigned one.
|
||||
self.active_downloads: Dict[str, Dict[str, Any]] = {}
|
||||
self._lock = threading.Lock()
|
||||
self.shutdown_check = None
|
||||
|
||||
def set_shutdown_check(self, check_callable):
|
||||
self.shutdown_check = check_callable
|
||||
|
||||
def reload_settings(self) -> None:
|
||||
self._prowlarr.reload_settings()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
if not self._prowlarr.is_configured():
|
||||
return False
|
||||
adapter = get_active_torrent_adapter()
|
||||
return bool(adapter and adapter.is_configured())
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
if not self._prowlarr.is_configured():
|
||||
return False
|
||||
adapter = get_active_torrent_adapter()
|
||||
if not adapter or not adapter.is_configured():
|
||||
return False
|
||||
# Probe both sides. A torrent download is useless if either
|
||||
# the indexer or the downloader is unreachable.
|
||||
prowlarr_ok = await self._prowlarr.check_connection()
|
||||
if not prowlarr_ok:
|
||||
return False
|
||||
return await adapter.check_connection()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Search
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query: str,
|
||||
timeout: Optional[int] = None,
|
||||
progress_callback=None,
|
||||
) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
if not self._prowlarr.is_configured():
|
||||
return ([], [])
|
||||
try:
|
||||
indexer_ids = _parse_indexer_id_filter()
|
||||
results = await self._prowlarr.search(
|
||||
query,
|
||||
categories=DEFAULT_MUSIC_CATEGORIES,
|
||||
indexer_ids=indexer_ids,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("Torrent plugin search failed: %s", e)
|
||||
return ([], [])
|
||||
return self._project_results(results)
|
||||
|
||||
def _project_results(
|
||||
self, results: List[ProwlarrSearchResult]
|
||||
) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
"""Turn Prowlarr releases into TrackResult / AlbumResult
|
||||
shaped objects. One TrackResult + one AlbumResult per
|
||||
release — Prowlarr search hits are at the release level,
|
||||
not the track level, so we can't synthesise track listings
|
||||
without downloading the actual torrent."""
|
||||
tracks: List[TrackResult] = []
|
||||
albums: List[AlbumResult] = []
|
||||
for result in results:
|
||||
if result.protocol != 'torrent':
|
||||
continue
|
||||
download_url = result.magnet_uri or result.download_url
|
||||
if not download_url:
|
||||
continue
|
||||
filename = f"{download_url}{_FILENAME_SEP}{result.title}"
|
||||
quality = _guess_quality_from_title(result.title)
|
||||
parsed_artist, parsed_title = _parse_release_title(result.title)
|
||||
tr = TrackResult(
|
||||
username='torrent',
|
||||
filename=filename,
|
||||
size=result.size,
|
||||
bitrate=None,
|
||||
duration=None,
|
||||
quality=quality,
|
||||
# Torrent results don't have per-uploader slot / queue
|
||||
# data the way Soulseek does. Fill with neutral values
|
||||
# so the quality_score doesn't punish them artificially.
|
||||
free_upload_slots=max(1, result.seeders or 0),
|
||||
upload_speed=0,
|
||||
queue_length=0,
|
||||
# Pre-fill artist + title so TrackResult.__post_init__
|
||||
# doesn't auto-parse the filename — our filename starts
|
||||
# with the indexer download URL, which would otherwise
|
||||
# show up as "by download?apikey=..." in the UI.
|
||||
artist=parsed_artist or result.indexer_name or 'Torrent',
|
||||
title=parsed_title or result.title,
|
||||
album=parsed_title or None,
|
||||
track_number=None,
|
||||
_source_metadata={
|
||||
'indexer': result.indexer_name,
|
||||
'indexer_id': result.indexer_id,
|
||||
'seeders': result.seeders,
|
||||
'leechers': result.leechers,
|
||||
'grabs': result.grabs,
|
||||
'protocol': 'torrent',
|
||||
},
|
||||
)
|
||||
tracks.append(tr)
|
||||
albums.append(AlbumResult(
|
||||
username='torrent',
|
||||
album_path=f"torrent/{result.guid}",
|
||||
album_title=parsed_title or result.title,
|
||||
artist=parsed_artist or None,
|
||||
track_count=1, # unknown until download finishes
|
||||
total_size=result.size,
|
||||
tracks=[tr],
|
||||
dominant_quality=quality,
|
||||
year=None,
|
||||
))
|
||||
return tracks, albums
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Download
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def download(
|
||||
self,
|
||||
username: str,
|
||||
filename: str,
|
||||
file_size: int = 0,
|
||||
) -> Optional[str]:
|
||||
if not self.is_configured():
|
||||
return None
|
||||
download_url, display_name = _decode_filename(filename)
|
||||
if not download_url:
|
||||
logger.error("Torrent download missing URL in filename: %r", filename)
|
||||
return None
|
||||
|
||||
download_id = str(uuid.uuid4())
|
||||
with self._lock:
|
||||
self.active_downloads[download_id] = {
|
||||
'id': download_id,
|
||||
'filename': filename,
|
||||
'username': 'torrent',
|
||||
'display_name': display_name,
|
||||
'state': 'Initializing',
|
||||
'progress': 0.0,
|
||||
'size': file_size,
|
||||
'transferred': 0,
|
||||
'speed': 0,
|
||||
'file_path': None,
|
||||
'audio_files': [],
|
||||
'torrent_hash': None,
|
||||
'error': None,
|
||||
}
|
||||
|
||||
thread = threading.Thread(
|
||||
target=self._download_thread,
|
||||
args=(download_id, download_url, display_name),
|
||||
daemon=True,
|
||||
name=f'torrent-dl-{download_id[:8]}',
|
||||
)
|
||||
thread.start()
|
||||
return download_id
|
||||
|
||||
def _download_thread(self, download_id: str, download_url: str, display_name: str) -> None:
|
||||
"""Background worker: hand the URL to the active adapter,
|
||||
poll until done, then walk the resulting directory."""
|
||||
adapter = get_active_torrent_adapter()
|
||||
if adapter is None or not adapter.is_configured():
|
||||
self._mark_error(download_id, "No torrent client configured")
|
||||
return
|
||||
|
||||
try:
|
||||
torrent_hash = run_async(adapter.add_torrent(download_url))
|
||||
except Exception as e:
|
||||
self._mark_error(download_id, f"add_torrent failed: {e}")
|
||||
return
|
||||
if not torrent_hash:
|
||||
self._mark_error(download_id, "Torrent client refused the URL")
|
||||
return
|
||||
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['torrent_hash'] = torrent_hash
|
||||
row['state'] = 'InProgress, Downloading'
|
||||
|
||||
deadline = time.monotonic() + _POLL_TIMEOUT_SECONDS
|
||||
last_save_path: Optional[str] = None
|
||||
# Tolerate transient None reads — covers network blips. Torrent
|
||||
# adapters don't have an SAB-style queue→history transition,
|
||||
# but the same tolerance keeps a one-off connection failure
|
||||
# from killing an otherwise-healthy download.
|
||||
misses = TransientMissCounter()
|
||||
# Stalled-torrent handling (noldevin): give up early on a torrent
|
||||
# making zero progress (dead magnet stuck on metadata, no seeders)
|
||||
# instead of holding this worker for the full album deadline. Read
|
||||
# per-download so a settings change applies to in-flight torrents.
|
||||
stall = StallTracker(get_stall_timeout())
|
||||
while time.monotonic() < deadline:
|
||||
if self.shutdown_check and self.shutdown_check():
|
||||
return
|
||||
try:
|
||||
status = run_async(adapter.get_status(torrent_hash))
|
||||
except Exception as e:
|
||||
logger.warning("Torrent poll error for %s: %s", torrent_hash, e)
|
||||
status = None
|
||||
|
||||
if status is None:
|
||||
if misses.record_miss():
|
||||
self._mark_error(
|
||||
download_id,
|
||||
f"Torrent disappeared from client (no status after {misses.threshold} polls)",
|
||||
)
|
||||
return
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
continue
|
||||
|
||||
misses.reset()
|
||||
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['progress'] = status.progress * 100.0
|
||||
row['transferred'] = status.downloaded
|
||||
row['speed'] = status.download_speed
|
||||
row['size'] = status.size or row.get('size', 0)
|
||||
row['state'] = _adapter_state_to_display(status.state)
|
||||
row['error'] = status.error
|
||||
if status.save_path:
|
||||
last_save_path = status.save_path
|
||||
|
||||
if status.state in _COMPLETE_STATES:
|
||||
self._finalize_download(download_id, last_save_path)
|
||||
return
|
||||
if status.state == 'error':
|
||||
# Clean the dead torrent out of the client, or it's left orphaned
|
||||
# (active in qbit, untracked here) and re-grabbed as a duplicate.
|
||||
self._cleanup_torrent(torrent_hash, get_stall_action())
|
||||
self._mark_error(download_id, status.error or "Torrent client reported error")
|
||||
return
|
||||
|
||||
if stall.is_stalled(status.downloaded, status.state, time.monotonic(),
|
||||
size=status.size):
|
||||
self._handle_stalled(download_id, torrent_hash, get_stall_action())
|
||||
return
|
||||
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
|
||||
# Deadline reached. One last status check closes the race where the
|
||||
# torrent completed during the final poll interval — finalize it instead
|
||||
# of deleting a just-finished download's files. Otherwise clean it out of
|
||||
# the client, or it sits orphaned in qbit (e.g. a metadata-stuck magnet
|
||||
# that escaped the stall timer) and gets re-grabbed as a duplicate.
|
||||
try:
|
||||
final = run_async(adapter.get_status(torrent_hash))
|
||||
except Exception:
|
||||
final = None
|
||||
if final is not None and final.state in _COMPLETE_STATES:
|
||||
self._finalize_download(download_id, final.save_path or last_save_path)
|
||||
return
|
||||
self._cleanup_torrent(torrent_hash, get_stall_action())
|
||||
self._mark_error(download_id, "Torrent download timed out")
|
||||
|
||||
def _cleanup_torrent(self, torrent_hash: str, action: str) -> None:
|
||||
"""Remove (abandon) or pause a dead/stalled/timed-out torrent in the
|
||||
client so it isn't left ORPHANED — active in qbit but no longer tracked
|
||||
here, which makes SoulSync re-grab the same dead torrent as a duplicate
|
||||
on the next attempt (noldevin). Best-effort: a client error is logged,
|
||||
not raised, so the download still fails cleanly."""
|
||||
adapter = get_active_torrent_adapter()
|
||||
if adapter is None or not torrent_hash:
|
||||
return
|
||||
try:
|
||||
if action == "pause":
|
||||
run_async(adapter.pause(torrent_hash))
|
||||
else:
|
||||
# delete_files: a stalled/failed torrent's partial data is junk
|
||||
# (often just a metadata stub) — don't leave it on disk.
|
||||
run_async(adapter.remove(torrent_hash, delete_files=True))
|
||||
except Exception as e:
|
||||
logger.warning("Torrent cleanup (%s) on %s failed: %s",
|
||||
action, torrent_hash[:8] if torrent_hash else "?", e)
|
||||
|
||||
def _handle_stalled(self, download_id: str, torrent_hash: str, action: str) -> None:
|
||||
"""A torrent made no progress past the stall timeout. Abandon it
|
||||
(remove from client + delete its partial data) or pause it for the
|
||||
user, then fail the download so the worker frees up."""
|
||||
timeout_min = round(get_stall_timeout() / 60, 1)
|
||||
self._cleanup_torrent(torrent_hash, action)
|
||||
verb = "paused" if action == "pause" else "removed"
|
||||
self._mark_error(
|
||||
download_id,
|
||||
f"Torrent stalled (no progress for {timeout_min} min) — {verb}",
|
||||
)
|
||||
|
||||
def _finalize_download(self, download_id: str, save_path: Optional[str]) -> None:
|
||||
"""Adapter said complete. Walk the directory + pick the
|
||||
first audio file as the canonical ``file_path``."""
|
||||
if not save_path:
|
||||
self._mark_error(download_id, "Torrent completed but no save_path reported")
|
||||
return
|
||||
# Resolve the client-reported path to one this process can read
|
||||
# (the client may report its own container's mount). See
|
||||
# ``resolve_reported_save_path``.
|
||||
local_path = resolve_reported_save_path(save_path)
|
||||
if local_path != save_path:
|
||||
logger.info("Torrent %s: resolved client path %r -> %r",
|
||||
download_id[:8], save_path, local_path)
|
||||
try:
|
||||
audio_files = collect_audio_after_extraction(Path(local_path))
|
||||
except Exception as e:
|
||||
self._mark_error(download_id, f"Post-extract walk failed: {e}")
|
||||
return
|
||||
if not audio_files:
|
||||
suffix = f" (resolved: {local_path})" if local_path != save_path else ""
|
||||
self._mark_error(download_id, f"No audio files found in {save_path}{suffix}")
|
||||
return
|
||||
primary = audio_files[0]
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Completed, Succeeded'
|
||||
row['progress'] = 100.0
|
||||
row['file_path'] = str(primary)
|
||||
row['audio_files'] = [str(path) for path in audio_files]
|
||||
logger.info("Torrent download complete: %s -> %s (%d audio files)",
|
||||
download_id[:8], primary.name, len(audio_files))
|
||||
|
||||
def _mark_error(self, download_id: str, message: str) -> None:
|
||||
logger.error("Torrent download %s failed: %s", download_id[:8], message)
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Completed, Errored'
|
||||
row['error'] = message
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Status / lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def get_all_downloads(self) -> List[DownloadStatus]:
|
||||
with self._lock:
|
||||
rows = list(self.active_downloads.values())
|
||||
return [_row_to_status(r) for r in rows]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_status(row)
|
||||
|
||||
async def cancel_download(
|
||||
self,
|
||||
download_id: str,
|
||||
username: Optional[str] = None,
|
||||
remove: bool = False,
|
||||
) -> bool:
|
||||
adapter = get_active_torrent_adapter()
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
torrent_hash = row.get('torrent_hash') if row else None
|
||||
if adapter and torrent_hash:
|
||||
try:
|
||||
await adapter.remove(torrent_hash, delete_files=remove)
|
||||
except Exception as e:
|
||||
logger.warning("Torrent cancel via adapter failed: %s", e)
|
||||
with self._lock:
|
||||
if remove:
|
||||
self.active_downloads.pop(download_id, None)
|
||||
else:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Cancelled'
|
||||
return True
|
||||
|
||||
async def clear_all_completed_downloads(self) -> bool:
|
||||
with self._lock:
|
||||
for did in list(self.active_downloads.keys()):
|
||||
state = self.active_downloads[did].get('state', '')
|
||||
if state.startswith('Completed') or state == 'Cancelled':
|
||||
self.active_downloads.pop(did, None)
|
||||
return True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Album-bundle flow
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def download_album_to_staging(
|
||||
self,
|
||||
album_name: str,
|
||||
artist_name: str,
|
||||
staging_dir: str,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""One-shot album download: search Prowlarr for the whole
|
||||
release, pick the best torrent, fetch it, extract if needed,
|
||||
copy every audio file into ``staging_dir`` so the existing
|
||||
``try_staging_match`` flow can hand each track off to the
|
||||
post-processing pipeline.
|
||||
|
||||
``progress_callback`` is called with a dict on each state
|
||||
change so the batch UI can show download progress without
|
||||
waiting for the whole thing.
|
||||
|
||||
Returns ``{'success': bool, 'files': [paths], 'error': str|None}``.
|
||||
"""
|
||||
result: Dict[str, Any] = {'success': False, 'files': [], 'error': None}
|
||||
if not self.is_configured():
|
||||
result['error'] = 'Torrent source not configured'
|
||||
return result
|
||||
|
||||
adapter = get_active_torrent_adapter()
|
||||
if adapter is None or not adapter.is_configured():
|
||||
result['error'] = 'No active torrent client'
|
||||
return result
|
||||
|
||||
def _emit(state: str, **extra) -> None:
|
||||
if progress_callback:
|
||||
payload = {'state': state, **extra}
|
||||
try:
|
||||
progress_callback(payload)
|
||||
except Exception as cb_exc:
|
||||
logger.debug("[Torrent album] progress callback failed: %s", cb_exc)
|
||||
|
||||
# Phase 1: search Prowlarr for the album.
|
||||
query = f"{artist_name} {album_name}".strip()
|
||||
_emit('searching', query=query)
|
||||
try:
|
||||
search_results = run_async(self._prowlarr.search(
|
||||
query, categories=DEFAULT_MUSIC_CATEGORIES,
|
||||
indexer_ids=_parse_indexer_id_filter(),
|
||||
))
|
||||
except Exception as e:
|
||||
result['error'] = f'Prowlarr search failed: {e}'
|
||||
return result
|
||||
|
||||
candidates = [r for r in search_results
|
||||
if r.protocol == 'torrent' and (r.magnet_uri or r.download_url)]
|
||||
if not candidates:
|
||||
# Album isn't available on this source. Mark the failure as
|
||||
# fallback-eligible so the dispatch returns to the per-track flow
|
||||
# instead of hard-failing the batch — in hybrid mode that lets the
|
||||
# next configured source take over. Without this flag a torrent-first
|
||||
# hybrid would get stuck at "searching" forever when Prowlarr
|
||||
# returns nothing, never trying the other sources.
|
||||
result['error'] = f'No torrent results found for "{query}"'
|
||||
result['fallback'] = True
|
||||
return result
|
||||
|
||||
picked = pick_best_album_release(
|
||||
candidates, _guess_quality_from_title, album_name=album_name,
|
||||
)
|
||||
if picked is None:
|
||||
# No candidate matched the requested album (or none passed filtering).
|
||||
# Fall back to the per-track flow rather than downloading a wrong
|
||||
# album (#730) — per-track searches each track individually.
|
||||
result['error'] = 'No torrent candidate matched the requested album'
|
||||
result['fallback'] = True
|
||||
return result
|
||||
|
||||
download_url = picked.magnet_uri or picked.download_url
|
||||
logger.info("[Torrent album] Picked '%s' (size=%.1fMB seeders=%s indexer=%s)",
|
||||
picked.title, picked.size / 1_048_576, picked.seeders, picked.indexer_name)
|
||||
_emit('queued', release=picked.title, size=picked.size, seeders=picked.seeders)
|
||||
|
||||
# Phase 2: hand to adapter.
|
||||
try:
|
||||
torrent_id = run_async(adapter.add_torrent(download_url))
|
||||
except Exception as e:
|
||||
result['error'] = f'Torrent client refused the release: {e}'
|
||||
return result
|
||||
if not torrent_id:
|
||||
result['error'] = 'Torrent client refused the release'
|
||||
return result
|
||||
|
||||
# Phase 3: poll until complete. The lifted helper handles
|
||||
# transient missing windows (uncommon for torrents — adapters
|
||||
# don't have a queue→history transition like SAB — but the
|
||||
# same path also catches network blips that would otherwise
|
||||
# take down the whole download) and always emits a terminal
|
||||
# 'failed' state on failure paths so the UI doesn't freeze on
|
||||
# the last 'downloading' emit.
|
||||
_emit('downloading', release=picked.title)
|
||||
save_path = poll_album_download(
|
||||
get_status=lambda: run_async(adapter.get_status(torrent_id)),
|
||||
title=picked.title,
|
||||
emit=_emit,
|
||||
# Torrent adapters flip to 'seeding' on completion (files
|
||||
# on disk, share-ratio progress) — both states count as
|
||||
# terminal success.
|
||||
complete_states=frozenset(['seeding', 'completed']),
|
||||
# qBit / Transmission / Deluge surface a real 'error'
|
||||
# state when the torrent itself errors (tracker, missing
|
||||
# files, etc.). That's distinct from the unmapped-state
|
||||
# default-'error' fallback the helper treats as transient.
|
||||
failed_states=frozenset(['error']),
|
||||
is_shutdown=self.shutdown_check,
|
||||
log_prefix='[Torrent album]',
|
||||
)
|
||||
if save_path is None:
|
||||
# poll_album_download already emitted terminal 'failed'.
|
||||
result['error'] = 'Torrent download failed or timed out'
|
||||
return result
|
||||
|
||||
# Phase 4: extract + walk + copy to staging.
|
||||
_emit('staging', release=picked.title)
|
||||
# Resolve the client-reported path to one this process can read.
|
||||
local_path = resolve_reported_save_path(save_path)
|
||||
if local_path != save_path:
|
||||
logger.info("[Torrent album] Resolved client path %r -> %r", save_path, local_path)
|
||||
try:
|
||||
audio_files = collect_audio_after_extraction(Path(local_path))
|
||||
except Exception as e:
|
||||
result['error'] = f'Failed to walk audio files: {e}'
|
||||
return result
|
||||
if not audio_files:
|
||||
suffix = f' (resolved: {local_path})' if local_path != save_path else ''
|
||||
result['error'] = f'No audio files found in {save_path}{suffix}'
|
||||
return result
|
||||
|
||||
copied = copy_audio_files_atomically(audio_files, Path(staging_dir))
|
||||
if not copied:
|
||||
result['error'] = 'No audio files copied to staging'
|
||||
return result
|
||||
logger.info("[Torrent album] Staged %d audio files for '%s'", len(copied), album_name)
|
||||
_emit('staged', count=len(copied))
|
||||
result['success'] = True
|
||||
result['files'] = copied
|
||||
return result
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level helpers (pure functions — easy to unit-test)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _decode_filename(filename: str) -> Tuple[Optional[str], str]:
|
||||
"""Pull the encoded download URL out of the ``filename`` string.
|
||||
Returns ``(url, display_name)``. ``url`` is None when the string
|
||||
has no separator."""
|
||||
if not filename or _FILENAME_SEP not in filename:
|
||||
return (None, filename or '')
|
||||
url, display = filename.split(_FILENAME_SEP, 1)
|
||||
return (url, display)
|
||||
|
||||
|
||||
def _parse_release_title(title: str) -> Tuple[str, str]:
|
||||
"""Split a release title into ``(artist, title)`` using the
|
||||
``Artist - Title`` / ``Artist - Album`` convention almost every
|
||||
indexer follows. Returns ``('', title)`` when no dash is found.
|
||||
|
||||
Without this, ``TrackResult.__post_init__`` runs the bare
|
||||
filename through ``parse_filename_metadata`` — and our filename
|
||||
starts with the indexer's download URL, so the auto-parser
|
||||
extracts garbage like ``download?apikey=...`` as the artist
|
||||
and shows it in the search-result UI's "by" line. Pre-filling
|
||||
the artist field short-circuits the auto-parse.
|
||||
"""
|
||||
if not title:
|
||||
return ('', '')
|
||||
# Strip common quality / format tags so the dash split doesn't
|
||||
# eat them — "Artist - Album [FLAC] (2020)" → "Artist", "Album".
|
||||
cleaned = re.sub(r'\s*[\[\(][^\]\)]*[\]\)]\s*$', '', title.strip())
|
||||
# Look for the FIRST " - " (or "-" surrounded by content). Some
|
||||
# release titles have multiple dashes (subtitle dashes); the
|
||||
# first split is the artist/work boundary.
|
||||
parts = re.split(r'\s+-\s+|\s+-(?=\S)|(?<=\S)-\s+', cleaned, maxsplit=1)
|
||||
if len(parts) == 2:
|
||||
artist = parts[0].strip()
|
||||
rest = parts[1].strip()
|
||||
# Reject obvious non-artist prefixes (URLs, hashes, single
|
||||
# punctuation) so we don't propagate garbage.
|
||||
if artist and not re.match(r'^https?:|^[a-f0-9]{32,}$', artist):
|
||||
return (artist, rest or cleaned)
|
||||
return ('', cleaned)
|
||||
|
||||
|
||||
def _guess_quality_from_title(title: str) -> str:
|
||||
"""Read the quality hint from a release title — most music
|
||||
torrents put the encoding right in the name (FLAC, MP3 320,
|
||||
etc.). Falls back to ``'mp3'`` so quality_score doesn't crash."""
|
||||
if not title:
|
||||
return 'mp3'
|
||||
lower = title.lower()
|
||||
if 'flac' in lower:
|
||||
return 'flac'
|
||||
if re.search(r'\b24[\s-]?bit\b', lower) or 'hi-?res' in lower:
|
||||
return 'flac'
|
||||
if 'aac' in lower:
|
||||
return 'aac'
|
||||
if 'ogg' in lower:
|
||||
return 'ogg'
|
||||
return 'mp3'
|
||||
|
||||
|
||||
def _parse_indexer_id_filter() -> List[int]:
|
||||
"""Read the comma-separated indexer-ID allowlist from config.
|
||||
Empty list = search every enabled indexer."""
|
||||
raw = (config_manager.get('prowlarr.indexer_ids', '') or '').strip()
|
||||
if not raw:
|
||||
return []
|
||||
out: List[int] = []
|
||||
for chunk in raw.split(','):
|
||||
chunk = chunk.strip()
|
||||
if not chunk:
|
||||
continue
|
||||
try:
|
||||
out.append(int(chunk))
|
||||
except ValueError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def _adapter_state_to_display(state: str) -> str:
|
||||
"""Translate the adapter-uniform state strings into the
|
||||
``'InProgress, Downloading'`` / ``'Completed, Succeeded'``
|
||||
style the existing UI expects (matches Soulseek + Lidarr)."""
|
||||
mapping = {
|
||||
'queued': 'Queued',
|
||||
'downloading': 'InProgress, Downloading',
|
||||
'stalled': 'InProgress, Stalled',
|
||||
'seeding': 'Completed, Succeeded',
|
||||
'completed': 'Completed, Succeeded',
|
||||
'paused': 'Paused',
|
||||
'error': 'Completed, Errored',
|
||||
}
|
||||
return mapping.get(state, state.title())
|
||||
|
||||
|
||||
def _row_to_status(row: Dict[str, Any]) -> DownloadStatus:
|
||||
return DownloadStatus(
|
||||
id=row['id'],
|
||||
filename=row['filename'],
|
||||
username=row['username'],
|
||||
state=row.get('state', 'Unknown'),
|
||||
progress=float(row.get('progress', 0.0)),
|
||||
size=int(row.get('size', 0)),
|
||||
transferred=int(row.get('transferred', 0)),
|
||||
speed=int(row.get('speed', 0)),
|
||||
time_remaining=None,
|
||||
file_path=row.get('file_path'),
|
||||
audio_files=row.get('audio_files') or None,
|
||||
)
|
||||
|
|
@ -1,131 +0,0 @@
|
|||
"""Stalled-torrent detection + policy (noldevin's request).
|
||||
|
||||
A torrent can sit forever making zero progress — most commonly stuck
|
||||
"downloading metadata" on a magnet with no peers, but also a dead swarm
|
||||
mid-download. The torrent poll loop would just burn the full 6-hour album
|
||||
timeout on it. This module decides, from the live status stream, when a
|
||||
torrent has been stalled too long, and what to do about it.
|
||||
|
||||
Design split, kept testable:
|
||||
- ``StallTracker`` is the pure decision core — feed it each poll's
|
||||
``(downloaded, state, now)`` and it answers "stalled too long?" using a
|
||||
monotonic clock passed in (no time import, no I/O). Progress = bytes
|
||||
moved since the last poll; any forward movement resets the stall clock.
|
||||
Terminal/healthy-but-idle states (seeding, completed, paused) never count
|
||||
as stalled — only states where the torrent is *supposed* to be working.
|
||||
- ``get_stall_timeout`` / ``get_stall_action`` read the two settings.
|
||||
|
||||
A timeout of 0 disables stall handling entirely (back to the old behavior:
|
||||
ride the full poll deadline).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from config.settings import config_manager
|
||||
|
||||
# 0 = disabled. 10 minutes is long enough to ride out a slow metadata fetch
|
||||
# or a brief peer drought, short enough to give up on a truly dead magnet
|
||||
# instead of holding a worker for 6 hours.
|
||||
DEFAULT_STALL_TIMEOUT_SECONDS = 10 * 60
|
||||
|
||||
# What to do when a torrent stalls past the timeout:
|
||||
# 'abandon' — remove it from the client (and its partial data) + fail the
|
||||
# download so the worker is freed and the next source can try.
|
||||
# 'pause' — pause it in the client + fail the download, leaving the
|
||||
# torrent for the user to inspect/resume manually.
|
||||
_VALID_ACTIONS = ("abandon", "pause")
|
||||
DEFAULT_STALL_ACTION = "abandon"
|
||||
|
||||
# States where the torrent is meant to be making download progress, so a
|
||||
# lack of it counts toward the stall clock. Mirrors the adapter-uniform set
|
||||
# in core/torrent_clients/base.py. Notably EXCLUDES seeding/completed (done)
|
||||
# and paused (the user's own choice) — neither is a stall.
|
||||
STALLABLE_STATES = frozenset(("queued", "downloading", "stalled", "error"))
|
||||
|
||||
|
||||
def get_stall_timeout() -> float:
|
||||
"""Seconds of zero progress before a torrent is considered stalled.
|
||||
0 (or invalid/negative) disables stall handling."""
|
||||
raw = config_manager.get("download_source.torrent_stall_timeout_seconds",
|
||||
DEFAULT_STALL_TIMEOUT_SECONDS)
|
||||
try:
|
||||
value = float(raw)
|
||||
if value >= 0:
|
||||
return value
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return DEFAULT_STALL_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def get_stall_action() -> str:
|
||||
"""What to do with a stalled torrent: 'abandon' (default) or 'pause'."""
|
||||
raw = config_manager.get("download_source.torrent_stall_action",
|
||||
DEFAULT_STALL_ACTION)
|
||||
action = str(raw or "").strip().lower()
|
||||
return action if action in _VALID_ACTIONS else DEFAULT_STALL_ACTION
|
||||
|
||||
|
||||
class StallTracker:
|
||||
"""Tracks one torrent's forward progress across polls.
|
||||
|
||||
Pure + clock-injected so it tests without sleeping. ``timeout`` <= 0
|
||||
disables it (``is_stalled`` always returns False)."""
|
||||
|
||||
def __init__(self, timeout_seconds: float):
|
||||
self.timeout = float(timeout_seconds or 0)
|
||||
self._last_downloaded = -1 # -1 = first observation
|
||||
self._had_metadata = None # None = first observation; else size>0?
|
||||
self._progress_since = None # monotonic time of last forward movement
|
||||
|
||||
def is_stalled(self, downloaded: int, state: str, now: float,
|
||||
size: int = None) -> bool:
|
||||
"""Record this poll's observation; return True iff the torrent has gone
|
||||
``timeout`` seconds with no real forward progress while in a working state.
|
||||
|
||||
``downloaded`` is cumulative payload bytes; ``state`` is the adapter-uniform
|
||||
state; ``now`` is a monotonic timestamp; ``size`` is the torrent's total
|
||||
size in bytes (0/None while still fetching metadata).
|
||||
|
||||
Metadata-phase fix (#852-adjacent torrent report): a magnet stuck
|
||||
"downloading metadata" reports ``size==0`` and a ``downloaded`` byte
|
||||
counter that still ticks up from DHT/peer-protocol overhead even though it
|
||||
makes no actual progress. Treating those bumps as progress reset the stall
|
||||
clock forever, so a dead magnet never timed out. Now the byte counter only
|
||||
counts once metadata is in (``size>0``); during the metadata phase the only
|
||||
thing that counts as progress is *obtaining* the metadata, so a torrent
|
||||
that can't even do that within the timeout is correctly flagged stalled.
|
||||
"""
|
||||
if self.timeout <= 0:
|
||||
return False
|
||||
|
||||
downloaded = int(downloaded or 0)
|
||||
# size is None when the caller doesn't track it (assume metadata present —
|
||||
# the old byte-progress behavior); an explicit size==0 is the metadata
|
||||
# phase (metaDL), where the byte counter is unreliable noise.
|
||||
has_metadata = size is None or int(size) > 0
|
||||
|
||||
# Real forward progress: first sighting, metadata just arrived, or (only
|
||||
# once we have metadata) more payload bytes. Byte bumps during the
|
||||
# metadata phase are protocol noise and do NOT count.
|
||||
progressed = (
|
||||
self._had_metadata is None # first poll
|
||||
or (has_metadata and not self._had_metadata) # got metadata
|
||||
or (has_metadata and downloaded > self._last_downloaded) # more payload
|
||||
)
|
||||
self._had_metadata = has_metadata
|
||||
self._last_downloaded = downloaded
|
||||
|
||||
if progressed:
|
||||
self._progress_since = now
|
||||
return False
|
||||
|
||||
# Not in a working state → not a stall (seeding/paused/completed).
|
||||
if state not in STALLABLE_STATES:
|
||||
self._progress_since = now # don't accrue stall time while idle-by-design
|
||||
return False
|
||||
|
||||
if self._progress_since is None:
|
||||
self._progress_since = now
|
||||
return False
|
||||
|
||||
return (now - self._progress_since) >= self.timeout
|
||||
|
|
@ -13,11 +13,10 @@ import from a neutral package per Cin's contract-first standard.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from core.imports.filename import parse_filename_metadata
|
||||
from core.quality.model import AudioQuality
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -33,36 +32,6 @@ class SearchResult:
|
|||
upload_speed: int
|
||||
queue_length: int
|
||||
result_type: str = "track" # "track" or "album"
|
||||
# Rich quality metadata — populated by sources that provide it.
|
||||
# None means "unknown", not "absent".
|
||||
sample_rate: Optional[int] = None # Hz (e.g. 44100, 96000, 192000)
|
||||
bit_depth: Optional[int] = None # bits per sample (16, 24)
|
||||
|
||||
@property
|
||||
def audio_quality(self) -> AudioQuality:
|
||||
"""Unified quality descriptor derived from this result's fields."""
|
||||
return AudioQuality(
|
||||
format=self.quality.lower() if self.quality else 'unknown',
|
||||
bitrate=self.bitrate,
|
||||
sample_rate=self.sample_rate,
|
||||
bit_depth=self.bit_depth,
|
||||
)
|
||||
|
||||
def set_quality(self, aq: AudioQuality) -> None:
|
||||
"""Merge a mapped :class:`AudioQuality` onto this result's fields.
|
||||
|
||||
Used by streaming sources to stamp their claimed tier (Tidal/HiFi
|
||||
tier strings, Qobuz API values, …) so ``audio_quality`` ranks
|
||||
correctly. Mapper-provided fields win; a ``None`` from the mapper
|
||||
leaves any already-reported value (e.g. a probed bitrate) intact.
|
||||
"""
|
||||
self.quality = aq.format
|
||||
if aq.bitrate is not None:
|
||||
self.bitrate = aq.bitrate
|
||||
if aq.sample_rate is not None:
|
||||
self.sample_rate = aq.sample_rate
|
||||
if aq.bit_depth is not None:
|
||||
self.bit_depth = aq.bit_depth
|
||||
|
||||
@property
|
||||
def quality_score(self) -> float:
|
||||
|
|
@ -158,19 +127,6 @@ class AlbumResult:
|
|||
queue_length: int = 0
|
||||
result_type: str = "album"
|
||||
|
||||
@property
|
||||
def audio_quality(self) -> AudioQuality:
|
||||
"""Unified quality descriptor derived from dominant track quality."""
|
||||
sample_rates = [t.sample_rate for t in self.tracks if t.sample_rate]
|
||||
bit_depths = [t.bit_depth for t in self.tracks if t.bit_depth]
|
||||
bitrates = [t.bitrate for t in self.tracks if t.bitrate]
|
||||
return AudioQuality(
|
||||
format=self.dominant_quality.lower() if self.dominant_quality else 'unknown',
|
||||
bitrate=max(bitrates) if bitrates else None,
|
||||
sample_rate=max(sample_rates) if sample_rates else None,
|
||||
bit_depth=max(bit_depths) if bit_depths else None,
|
||||
)
|
||||
|
||||
@property
|
||||
def quality_score(self) -> float:
|
||||
"""Calculate album quality score based on dominant quality and track count"""
|
||||
|
|
@ -241,4 +197,3 @@ class DownloadStatus:
|
|||
speed: int
|
||||
time_remaining: Optional[int] = None
|
||||
file_path: Optional[str] = None
|
||||
audio_files: Optional[List[str]] = None
|
||||
|
|
|
|||
|
|
@ -1,543 +0,0 @@
|
|||
"""UsenetDownloadPlugin — composes Prowlarr search + usenet client
|
||||
adapter + archive_pipeline into a uniform download source.
|
||||
|
||||
Mirrors ``TorrentDownloadPlugin`` in shape and lifecycle (see that
|
||||
module's docstring for the full pipeline rationale). Differences:
|
||||
|
||||
- Search filters Prowlarr results to ``protocol='usenet'``.
|
||||
- ``add_nzb`` replaces ``add_torrent``; for NZBs we usually have
|
||||
a direct HTTP URL the indexer exposes via Prowlarr.
|
||||
- Usenet clients (SABnzbd, NZBGet) typically auto-extract during
|
||||
post-processing, so ``archive_pipeline.collect_audio_after_extraction``
|
||||
usually has nothing to extract and just walks loose files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from core.archive_pipeline import collect_audio_after_extraction
|
||||
from core.download_plugins.album_bundle import (
|
||||
TransientMissCounter,
|
||||
copy_audio_files_atomically,
|
||||
get_completed_no_path_window_seconds,
|
||||
pick_best_album_release,
|
||||
poll_album_download,
|
||||
resolve_reported_save_path,
|
||||
)
|
||||
from core.download_plugins.base import DownloadSourcePlugin
|
||||
from core.download_plugins.torrent import (
|
||||
_adapter_state_to_display,
|
||||
_decode_filename,
|
||||
_guess_quality_from_title,
|
||||
_parse_indexer_id_filter,
|
||||
_parse_release_title,
|
||||
_row_to_status,
|
||||
_COMPLETE_STATES,
|
||||
_FILENAME_SEP,
|
||||
_POLL_INTERVAL_SECONDS,
|
||||
_POLL_TIMEOUT_SECONDS,
|
||||
)
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.prowlarr_client import (
|
||||
DEFAULT_MUSIC_CATEGORIES,
|
||||
ProwlarrClient,
|
||||
ProwlarrSearchResult,
|
||||
)
|
||||
from core.usenet_clients import get_active_adapter as get_active_usenet_adapter
|
||||
from utils.async_helpers import run_async
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("download_plugins.usenet")
|
||||
|
||||
|
||||
class UsenetDownloadPlugin(DownloadSourcePlugin):
|
||||
"""Usenet download source backed by Prowlarr + an active usenet
|
||||
client adapter (SABnzbd or NZBGet)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._prowlarr = ProwlarrClient()
|
||||
self.active_downloads: Dict[str, Dict[str, Any]] = {}
|
||||
self._lock = threading.Lock()
|
||||
self.shutdown_check = None
|
||||
|
||||
def set_shutdown_check(self, check_callable):
|
||||
self.shutdown_check = check_callable
|
||||
|
||||
def reload_settings(self) -> None:
|
||||
self._prowlarr.reload_settings()
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
if not self._prowlarr.is_configured():
|
||||
return False
|
||||
adapter = get_active_usenet_adapter()
|
||||
return bool(adapter and adapter.is_configured())
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
if not self._prowlarr.is_configured():
|
||||
return False
|
||||
adapter = get_active_usenet_adapter()
|
||||
if not adapter or not adapter.is_configured():
|
||||
return False
|
||||
if not await self._prowlarr.check_connection():
|
||||
return False
|
||||
return await adapter.check_connection()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Search
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query: str,
|
||||
timeout: Optional[int] = None,
|
||||
progress_callback=None,
|
||||
) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
if not self._prowlarr.is_configured():
|
||||
return ([], [])
|
||||
try:
|
||||
indexer_ids = _parse_indexer_id_filter()
|
||||
results = await self._prowlarr.search(
|
||||
query,
|
||||
categories=DEFAULT_MUSIC_CATEGORIES,
|
||||
indexer_ids=indexer_ids,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("Usenet plugin search failed: %s", e)
|
||||
return ([], [])
|
||||
return self._project_results(results)
|
||||
|
||||
def _project_results(
|
||||
self, results: List[ProwlarrSearchResult]
|
||||
) -> Tuple[List[TrackResult], List[AlbumResult]]:
|
||||
tracks: List[TrackResult] = []
|
||||
albums: List[AlbumResult] = []
|
||||
for result in results:
|
||||
if result.protocol != 'usenet':
|
||||
continue
|
||||
if not result.download_url:
|
||||
continue
|
||||
filename = f"{result.download_url}{_FILENAME_SEP}{result.title}"
|
||||
quality = _guess_quality_from_title(result.title)
|
||||
parsed_artist, parsed_title = _parse_release_title(result.title)
|
||||
tr = TrackResult(
|
||||
username='usenet',
|
||||
filename=filename,
|
||||
size=result.size,
|
||||
bitrate=None,
|
||||
duration=None,
|
||||
quality=quality,
|
||||
# Usenet doesn't expose per-uploader concurrency the way
|
||||
# Soulseek does; fill in neutral non-punishing values.
|
||||
free_upload_slots=1,
|
||||
upload_speed=0,
|
||||
queue_length=0,
|
||||
# Pre-fill artist + title so TrackResult.__post_init__
|
||||
# doesn't auto-parse the filename — same URL-in-filename
|
||||
# gotcha as the torrent plugin.
|
||||
artist=parsed_artist or result.indexer_name or 'Usenet',
|
||||
title=parsed_title or result.title,
|
||||
album=parsed_title or None,
|
||||
track_number=None,
|
||||
_source_metadata={
|
||||
'indexer': result.indexer_name,
|
||||
'indexer_id': result.indexer_id,
|
||||
'grabs': result.grabs,
|
||||
'protocol': 'usenet',
|
||||
},
|
||||
)
|
||||
tracks.append(tr)
|
||||
albums.append(AlbumResult(
|
||||
username='usenet',
|
||||
album_path=f"usenet/{result.guid}",
|
||||
album_title=parsed_title or result.title,
|
||||
artist=parsed_artist or None,
|
||||
track_count=1,
|
||||
total_size=result.size,
|
||||
tracks=[tr],
|
||||
dominant_quality=quality,
|
||||
year=None,
|
||||
))
|
||||
return tracks, albums
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Download
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def download(
|
||||
self,
|
||||
username: str,
|
||||
filename: str,
|
||||
file_size: int = 0,
|
||||
) -> Optional[str]:
|
||||
if not self.is_configured():
|
||||
return None
|
||||
nzb_url, display_name = _decode_filename(filename)
|
||||
if not nzb_url:
|
||||
logger.error("Usenet download missing URL in filename: %r", filename)
|
||||
return None
|
||||
|
||||
download_id = str(uuid.uuid4())
|
||||
with self._lock:
|
||||
self.active_downloads[download_id] = {
|
||||
'id': download_id,
|
||||
'filename': filename,
|
||||
'username': 'usenet',
|
||||
'display_name': display_name,
|
||||
'state': 'Initializing',
|
||||
'progress': 0.0,
|
||||
'size': file_size,
|
||||
'transferred': 0,
|
||||
'speed': 0,
|
||||
'file_path': None,
|
||||
'audio_files': [],
|
||||
'job_id': None,
|
||||
'error': None,
|
||||
}
|
||||
|
||||
thread = threading.Thread(
|
||||
target=self._download_thread,
|
||||
args=(download_id, nzb_url),
|
||||
daemon=True,
|
||||
name=f'usenet-dl-{download_id[:8]}',
|
||||
)
|
||||
thread.start()
|
||||
return download_id
|
||||
|
||||
def _download_thread(self, download_id: str, nzb_url: str) -> None:
|
||||
adapter = get_active_usenet_adapter()
|
||||
if adapter is None or not adapter.is_configured():
|
||||
self._mark_error(download_id, "No usenet client configured")
|
||||
return
|
||||
|
||||
try:
|
||||
job_id = run_async(adapter.add_nzb(nzb_url))
|
||||
except Exception as e:
|
||||
self._mark_error(download_id, f"add_nzb failed: {e}")
|
||||
return
|
||||
if not job_id:
|
||||
self._mark_error(download_id, "Usenet client refused the NZB")
|
||||
return
|
||||
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['job_id'] = job_id
|
||||
row['state'] = 'InProgress, Downloading'
|
||||
|
||||
deadline = time.monotonic() + _POLL_TIMEOUT_SECONDS
|
||||
last_save_path: Optional[str] = None
|
||||
last_incomplete_path: Optional[str] = None
|
||||
# Tolerate transient None / unmapped 'error' reads — SAB
|
||||
# removes a job from the queue before adding it to history,
|
||||
# and on busy servers that gap spans several polls. See
|
||||
# ``album_bundle.TransientMissCounter`` for the shared rule.
|
||||
misses = TransientMissCounter()
|
||||
# Separate, LONGER window for "SAB says completed but hasn't
|
||||
# written the final save_path yet" — the per-track sibling of the
|
||||
# bundle fix (#721). Without this the thread called
|
||||
# ``_finalize_download(None)`` on the first Completed-no-path read
|
||||
# and errored a download that actually succeeded in SAB. Default
|
||||
# ~120s, converted to a poll count against the live interval.
|
||||
completed_no_path_misses = TransientMissCounter(
|
||||
max(misses.threshold,
|
||||
int(get_completed_no_path_window_seconds() / max(_POLL_INTERVAL_SECONDS, 0.001)) or 1)
|
||||
)
|
||||
while time.monotonic() < deadline:
|
||||
if self.shutdown_check and self.shutdown_check():
|
||||
return
|
||||
try:
|
||||
status = run_async(adapter.get_status(job_id))
|
||||
except Exception as e:
|
||||
logger.warning("Usenet poll error for %s: %s", job_id, e)
|
||||
status = None
|
||||
|
||||
if status is None:
|
||||
if misses.record_miss():
|
||||
self._mark_error(
|
||||
download_id,
|
||||
f"Usenet job disappeared from client (no status after {misses.threshold} polls)",
|
||||
)
|
||||
return
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
continue
|
||||
|
||||
if status.state != 'error':
|
||||
misses.reset()
|
||||
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['progress'] = status.progress * 100.0
|
||||
row['transferred'] = status.downloaded
|
||||
row['speed'] = status.download_speed
|
||||
row['size'] = status.size or row.get('size', 0)
|
||||
row['state'] = _adapter_state_to_display(status.state)
|
||||
row['error'] = status.error
|
||||
if status.save_path:
|
||||
last_save_path = status.save_path
|
||||
incomplete_path = getattr(status, 'incomplete_path', None)
|
||||
if incomplete_path:
|
||||
last_incomplete_path = incomplete_path
|
||||
|
||||
if status.state in _COMPLETE_STATES:
|
||||
if last_save_path:
|
||||
self._finalize_download(download_id, last_save_path)
|
||||
return
|
||||
# Completed but no final save_path yet — SAB flips
|
||||
# History to 'Completed' before writing ``storage``.
|
||||
# Wait out the (longer) completed-no-path window rather
|
||||
# than erroring a download that actually succeeded.
|
||||
if completed_no_path_misses.record_miss():
|
||||
if last_incomplete_path:
|
||||
logger.warning(
|
||||
"Usenet %s: '%s' completed but no final save_path after "
|
||||
"%d polls — falling back to in-progress path %r",
|
||||
download_id[:8], job_id, completed_no_path_misses.misses,
|
||||
last_incomplete_path,
|
||||
)
|
||||
self._finalize_download(download_id, last_incomplete_path)
|
||||
return
|
||||
self._mark_error(
|
||||
download_id,
|
||||
"Usenet job completed but client never reported a save_path",
|
||||
)
|
||||
return
|
||||
logger.info(
|
||||
"Usenet %s: '%s' completed on client but save_path not yet set — "
|
||||
"retrying (poll %d/%d)",
|
||||
download_id[:8], job_id,
|
||||
completed_no_path_misses.misses, completed_no_path_misses.threshold,
|
||||
)
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
continue
|
||||
if status.state == 'failed':
|
||||
self._mark_error(download_id, status.error or "Usenet client reported failure")
|
||||
return
|
||||
if status.state == 'error':
|
||||
logger.warning(
|
||||
"Usenet poll: '%s' returned unmapped state — treating as transient",
|
||||
job_id,
|
||||
)
|
||||
if misses.record_miss():
|
||||
self._mark_error(
|
||||
download_id,
|
||||
"Usenet client returned unmapped state repeatedly",
|
||||
)
|
||||
return
|
||||
|
||||
time.sleep(_POLL_INTERVAL_SECONDS)
|
||||
|
||||
self._mark_error(download_id, "Usenet download timed out")
|
||||
|
||||
def _finalize_download(self, download_id: str, save_path: Optional[str]) -> None:
|
||||
if not save_path:
|
||||
self._mark_error(download_id, "Usenet job completed but no save_path reported")
|
||||
return
|
||||
# Translate the client-reported path to one THIS process can read
|
||||
# (SAB reports its own container path; SoulSync may see the same
|
||||
# files at a different mount). See ``resolve_reported_save_path``.
|
||||
local_path = resolve_reported_save_path(save_path)
|
||||
if local_path != save_path:
|
||||
logger.info("Usenet %s: resolved client path %r -> %r",
|
||||
download_id[:8], save_path, local_path)
|
||||
try:
|
||||
audio_files = collect_audio_after_extraction(Path(local_path))
|
||||
except Exception as e:
|
||||
self._mark_error(download_id, f"Post-extract walk failed: {e}")
|
||||
return
|
||||
if not audio_files:
|
||||
suffix = f" (resolved: {local_path})" if local_path != save_path else ""
|
||||
self._mark_error(download_id, f"No audio files found in {save_path}{suffix}")
|
||||
return
|
||||
primary = audio_files[0]
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Completed, Succeeded'
|
||||
row['progress'] = 100.0
|
||||
row['file_path'] = str(primary)
|
||||
row['audio_files'] = [str(path) for path in audio_files]
|
||||
logger.info("Usenet download complete: %s -> %s (%d audio files)",
|
||||
download_id[:8], primary.name, len(audio_files))
|
||||
|
||||
def _mark_error(self, download_id: str, message: str) -> None:
|
||||
logger.error("Usenet download %s failed: %s", download_id[:8], message)
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Completed, Errored'
|
||||
row['error'] = message
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Status / lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def get_all_downloads(self) -> List[DownloadStatus]:
|
||||
with self._lock:
|
||||
rows = list(self.active_downloads.values())
|
||||
return [_row_to_status(r) for r in rows]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_status(row)
|
||||
|
||||
async def cancel_download(
|
||||
self,
|
||||
download_id: str,
|
||||
username: Optional[str] = None,
|
||||
remove: bool = False,
|
||||
) -> bool:
|
||||
adapter = get_active_usenet_adapter()
|
||||
with self._lock:
|
||||
row = self.active_downloads.get(download_id)
|
||||
job_id = row.get('job_id') if row else None
|
||||
if adapter and job_id:
|
||||
try:
|
||||
await adapter.remove(job_id, delete_files=remove)
|
||||
except Exception as e:
|
||||
logger.warning("Usenet cancel via adapter failed: %s", e)
|
||||
with self._lock:
|
||||
if remove:
|
||||
self.active_downloads.pop(download_id, None)
|
||||
else:
|
||||
row = self.active_downloads.get(download_id)
|
||||
if row is not None:
|
||||
row['state'] = 'Cancelled'
|
||||
return True
|
||||
|
||||
async def clear_all_completed_downloads(self) -> bool:
|
||||
with self._lock:
|
||||
for did in list(self.active_downloads.keys()):
|
||||
state = self.active_downloads[did].get('state', '')
|
||||
if state.startswith('Completed') or state == 'Cancelled':
|
||||
self.active_downloads.pop(did, None)
|
||||
return True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Album-bundle flow
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def download_album_to_staging(
|
||||
self,
|
||||
album_name: str,
|
||||
artist_name: str,
|
||||
staging_dir: str,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Usenet sibling of ``TorrentDownloadPlugin.download_album_to_staging``.
|
||||
See that method's docstring for the contract."""
|
||||
result: Dict[str, Any] = {'success': False, 'files': [], 'error': None}
|
||||
if not self.is_configured():
|
||||
result['error'] = 'Usenet source not configured'
|
||||
return result
|
||||
|
||||
adapter = get_active_usenet_adapter()
|
||||
if adapter is None or not adapter.is_configured():
|
||||
result['error'] = 'No active usenet client'
|
||||
return result
|
||||
|
||||
def _emit(state: str, **extra) -> None:
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback({'state': state, **extra})
|
||||
except Exception as cb_exc:
|
||||
logger.debug("[Usenet album] progress callback failed: %s", cb_exc)
|
||||
|
||||
query = f"{artist_name} {album_name}".strip()
|
||||
_emit('searching', query=query)
|
||||
try:
|
||||
search_results = run_async(self._prowlarr.search(
|
||||
query, categories=DEFAULT_MUSIC_CATEGORIES,
|
||||
indexer_ids=_parse_indexer_id_filter(),
|
||||
))
|
||||
except Exception as e:
|
||||
result['error'] = f'Prowlarr search failed: {e}'
|
||||
return result
|
||||
|
||||
candidates = [r for r in search_results
|
||||
if r.protocol == 'usenet' and r.download_url]
|
||||
if not candidates:
|
||||
# Album isn't available on this source — fall back to the per-track
|
||||
# flow (next configured source in hybrid mode) rather than hard-
|
||||
# failing the whole batch. Mirrors the torrent plugin + soulseek's
|
||||
# default fallback contract.
|
||||
result['error'] = f'No usenet results found for "{query}"'
|
||||
result['fallback'] = True
|
||||
return result
|
||||
|
||||
picked = pick_best_album_release(
|
||||
candidates, _guess_quality_from_title, album_name=album_name,
|
||||
)
|
||||
if picked is None:
|
||||
# No candidate matched the requested album (or none passed filtering).
|
||||
# Fall back to per-track rather than grabbing a wrong album (#730).
|
||||
result['error'] = 'No NZB candidate matched the requested album'
|
||||
result['fallback'] = True
|
||||
return result
|
||||
|
||||
logger.info("[Usenet album] Picked '%s' (size=%.1fMB grabs=%s indexer=%s)",
|
||||
picked.title, picked.size / 1_048_576, picked.grabs, picked.indexer_name)
|
||||
_emit('queued', release=picked.title, size=picked.size, grabs=picked.grabs)
|
||||
|
||||
try:
|
||||
job_id = run_async(adapter.add_nzb(picked.download_url))
|
||||
except Exception as e:
|
||||
result['error'] = f'Usenet client refused the NZB: {e}'
|
||||
return result
|
||||
if not job_id:
|
||||
result['error'] = 'Usenet client refused the NZB'
|
||||
return result
|
||||
|
||||
_emit('downloading', release=picked.title)
|
||||
save_path = poll_album_download(
|
||||
get_status=lambda: run_async(adapter.get_status(job_id)),
|
||||
title=picked.title,
|
||||
emit=_emit,
|
||||
# Usenet completes into history as 'completed'; no 'seeding'
|
||||
# equivalent. Failed is explicit on history failures.
|
||||
complete_states=frozenset(['completed']),
|
||||
failed_states=frozenset(['failed']),
|
||||
is_shutdown=self.shutdown_check,
|
||||
log_prefix='[Usenet album]',
|
||||
)
|
||||
if save_path is None:
|
||||
# poll_album_download already emitted the terminal 'failed'
|
||||
# state on every failure path (timeout / disappeared /
|
||||
# explicit failure / unmapped). UI is unstuck either way.
|
||||
result['error'] = 'Usenet download failed or timed out'
|
||||
return result
|
||||
|
||||
_emit('staging', release=picked.title)
|
||||
# SAB reports its own container path; SoulSync may mount the same
|
||||
# files elsewhere. Resolve to a locally-readable path before walking.
|
||||
local_path = resolve_reported_save_path(save_path)
|
||||
if local_path != save_path:
|
||||
logger.info("[Usenet album] Resolved client path %r -> %r", save_path, local_path)
|
||||
try:
|
||||
audio_files = collect_audio_after_extraction(Path(local_path))
|
||||
except Exception as e:
|
||||
result['error'] = f'Failed to walk audio files: {e}'
|
||||
return result
|
||||
if not audio_files:
|
||||
suffix = f' (resolved: {local_path})' if local_path != save_path else ''
|
||||
result['error'] = f'No audio files found in {save_path}{suffix}'
|
||||
return result
|
||||
|
||||
copied = copy_audio_files_atomically(audio_files, Path(staging_dir))
|
||||
if not copied:
|
||||
result['error'] = 'No audio files copied to staging'
|
||||
return result
|
||||
logger.info("[Usenet album] Staged %d audio files for '%s'", len(copied), album_name)
|
||||
_emit('staged', count=len(copied))
|
||||
result['success'] = True
|
||||
result['files'] = copied
|
||||
return result
|
||||
|
||||
|
|
@ -1,233 +0,0 @@
|
|||
"""Album-bundle dispatch for torrent / usenet single-source downloads.
|
||||
|
||||
Lifted from ``run_full_missing_tracks_process`` so the master
|
||||
worker doesn't carry a 90-line inline branch and so the gate logic
|
||||
can be unit-tested in isolation.
|
||||
|
||||
The gate fires only when ALL conditions hold:
|
||||
|
||||
- Batch is an album-context download (``is_album_download`` flag).
|
||||
- Active download source is ``torrent``, ``usenet``, or ``soulseek``.
|
||||
In hybrid mode the caller may pass the first configured source as a
|
||||
source override; later hybrid sources stay per-track to preserve fallback.
|
||||
- Both album-name and artist-name are populated in batch context.
|
||||
- The resolved plugin exposes ``download_album_to_staging``.
|
||||
|
||||
When the gate engages it runs the plugin synchronously (the master
|
||||
worker is already on a thread-pool executor) and mirrors the
|
||||
plugin's lifecycle payloads into the batch state so the Downloads
|
||||
page can render meaningful progress before per-track tasks exist.
|
||||
|
||||
Return semantics: ``True`` means the gate handled the batch — the
|
||||
master worker should stop and not run per-track analysis. ``False``
|
||||
means the gate didn't engage (or engaged-and-fell-back) — caller
|
||||
continues the normal per-track flow.
|
||||
|
||||
The ``BatchStateAccess`` Protocol exists so this module doesn't
|
||||
import ``download_batches`` from runtime_state directly. The
|
||||
caller (master worker) injects accessors so this module stays
|
||||
testable without touching live runtime state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional, Protocol
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
# Use the project logger factory so these lines land in app.log under the
|
||||
# ``soulsync.*`` namespace the file handler captures. Plain
|
||||
# ``logging.getLogger(__name__)`` logs to the console only (the file
|
||||
# handler is attached to the ``soulsync`` logger), which is why
|
||||
# ``[Album Bundle] flow failed`` showed up in the terminal but never in
|
||||
# app.log during the #721 triage.
|
||||
logger = get_logger("downloads.album_bundle_dispatch")
|
||||
|
||||
|
||||
class BatchStateAccess(Protocol):
|
||||
"""Narrow shim around the batch-state dict ops the dispatch needs.
|
||||
|
||||
Two methods to keep the surface small:
|
||||
- ``update_fields(batch_id, fields)`` — atomic merge into the
|
||||
batch dict under tasks_lock.
|
||||
- ``mark_failed(batch_id, error)`` — convenience for the failure
|
||||
path (sets phase + error + album_bundle_state in one shot).
|
||||
"""
|
||||
|
||||
def update_fields(self, batch_id: str, fields: dict) -> None: ...
|
||||
|
||||
def mark_failed(self, batch_id: str, error: str) -> None: ...
|
||||
|
||||
|
||||
# Fields the album-bundle progress callback may carry. Anything in
|
||||
# this set gets mirrored onto the batch row as ``album_bundle_<key>``
|
||||
# so the Downloads page can render it without coupling to the
|
||||
# specific payload shape.
|
||||
_MIRRORED_KEYS = ('progress', 'release', 'speed', 'downloaded',
|
||||
'size', 'seeders', 'grabs', 'count', 'failed')
|
||||
|
||||
|
||||
def is_eligible(
|
||||
*,
|
||||
mode: str,
|
||||
is_album: bool,
|
||||
album_name: str,
|
||||
artist_name: str,
|
||||
) -> bool:
|
||||
"""Pure predicate: does this batch even qualify for the album
|
||||
flow? Separate from the resolution+run step so tests can pin
|
||||
the gate logic without standing up a plugin."""
|
||||
if not is_album:
|
||||
return False
|
||||
if (mode or '').lower() not in ('torrent', 'usenet', 'soulseek'):
|
||||
return False
|
||||
if not (album_name or '').strip():
|
||||
return False
|
||||
if not (artist_name or '').strip():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def try_dispatch(
|
||||
*,
|
||||
batch_id: str,
|
||||
is_album: bool,
|
||||
album_context: Optional[dict],
|
||||
artist_context: Optional[dict],
|
||||
config_get: Callable[..., Any],
|
||||
plugin_resolver: Callable[[str], Optional[Any]],
|
||||
state: BatchStateAccess,
|
||||
source_override: Optional[str] = None,
|
||||
plugin_kwargs: Optional[dict] = None,
|
||||
) -> bool:
|
||||
"""Attempt the album-bundle flow. Returns ``True`` iff the
|
||||
master worker should return early (gate engaged and completed
|
||||
— success OR failure). ``False`` means fall through to the
|
||||
normal per-track flow.
|
||||
|
||||
``config_get`` is a callable shaped like ``config_manager.get``;
|
||||
``plugin_resolver`` resolves a source-name string to an
|
||||
initialised plugin instance (or None); ``state`` is the
|
||||
BatchStateAccess shim. Injecting these keeps the module
|
||||
dependency-light + unit-testable.
|
||||
"""
|
||||
mode = (source_override or config_get('download_source.mode', 'soulseek') or 'soulseek').lower()
|
||||
album_name = (album_context or {}).get('name') or ''
|
||||
artist_name = (artist_context or {}).get('name') or ''
|
||||
|
||||
if not is_eligible(mode=mode, is_album=is_album,
|
||||
album_name=album_name, artist_name=artist_name):
|
||||
return False
|
||||
|
||||
album_name = album_name.strip()
|
||||
artist_name = artist_name.strip()
|
||||
|
||||
plugin = None
|
||||
try:
|
||||
plugin = plugin_resolver(mode)
|
||||
except Exception as exc:
|
||||
logger.warning("[Album Bundle] Could not resolve %s plugin: %s", mode, exc)
|
||||
|
||||
if plugin is None or not hasattr(plugin, 'download_album_to_staging'):
|
||||
logger.warning(
|
||||
"[Album Bundle] Gate matched but plugin / context unavailable "
|
||||
"(mode=%s album=%r artist=%r plugin=%s) — falling back to per-track flow",
|
||||
mode, album_name, artist_name,
|
||||
type(plugin).__name__ if plugin else None,
|
||||
)
|
||||
return False
|
||||
|
||||
staging_root = config_get(
|
||||
'download_source.album_bundle_staging_path',
|
||||
'storage/album_bundle_staging',
|
||||
) or 'storage/album_bundle_staging'
|
||||
staging_dir = str(Path(staging_root) / _safe_batch_dirname(batch_id))
|
||||
logger.info(
|
||||
"[Album Bundle] Engaging %s album flow for '%s' by '%s' -> %s",
|
||||
mode, album_name, artist_name, staging_dir,
|
||||
)
|
||||
state.update_fields(batch_id, {
|
||||
'phase': 'album_downloading',
|
||||
'album_bundle_state': 'searching',
|
||||
'album_bundle_source': mode,
|
||||
'album_bundle_staging_path': staging_dir,
|
||||
'album_bundle_private_staging': True,
|
||||
})
|
||||
|
||||
def _emit(payload):
|
||||
"""Mirror plugin lifecycle into batch state for UI rendering."""
|
||||
try:
|
||||
fields = {'album_bundle_state': payload.get('state', '')}
|
||||
for key in _MIRRORED_KEYS:
|
||||
if key in payload:
|
||||
fields[f'album_bundle_{key}'] = payload[key]
|
||||
state.update_fields(batch_id, fields)
|
||||
except Exception as exc:
|
||||
logger.debug("[Album Bundle] emit failed: %s", exc)
|
||||
|
||||
try:
|
||||
outcome = plugin.download_album_to_staging(
|
||||
album_name, artist_name, staging_dir, _emit,
|
||||
**(plugin_kwargs or {}),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("[Album Bundle] %s plugin raised: %s", mode, exc)
|
||||
# An OSError means an I/O step failed after the source already had the
|
||||
# album — most importantly the staging dir not being writable (#760),
|
||||
# but also any transient filesystem error. Treat it as fallback-eligible
|
||||
# so we return to the per-track flow instead of hard-failing the whole
|
||||
# batch (the #715 symptom: files download, then the batch fails).
|
||||
# Programming errors (TypeError, KeyError, …) are NOT OSError and stay
|
||||
# terminal, so genuine bugs still fail loudly. (requests' network
|
||||
# exceptions also subclass OSError, but plugins normally catch those
|
||||
# internally and return an outcome rather than raising; if one does
|
||||
# surface here, falling back to per-track is still the safe choice.)
|
||||
is_io_failure = isinstance(exc, OSError)
|
||||
outcome = {
|
||||
'success': False,
|
||||
'error': f'Plugin error: {exc}',
|
||||
'fallback': is_io_failure,
|
||||
}
|
||||
|
||||
if not outcome.get('success'):
|
||||
err = outcome.get('error', 'Album bundle download failed')
|
||||
if outcome.get('fallback'):
|
||||
logger.warning(
|
||||
"[Album Bundle] %s flow could not commit for '%s': %s — falling back to per-track flow",
|
||||
mode, album_name, err,
|
||||
)
|
||||
state.update_fields(batch_id, {
|
||||
'phase': 'analysis',
|
||||
'album_bundle_state': 'fallback',
|
||||
'album_bundle_error': err,
|
||||
'album_bundle_private_staging': False,
|
||||
'album_bundle_staging_path': None,
|
||||
})
|
||||
return False
|
||||
logger.error("[Album Bundle] %s flow failed for '%s': %s",
|
||||
mode, album_name, err)
|
||||
state.mark_failed(batch_id, err)
|
||||
return True
|
||||
|
||||
logger.info(
|
||||
"[Album Bundle] %s staged %d files for '%s' — handing off to per-track staging matcher",
|
||||
mode, len(outcome.get('files', [])), album_name,
|
||||
)
|
||||
state.update_fields(batch_id, {
|
||||
'phase': 'analysis',
|
||||
'album_bundle_state': 'staged',
|
||||
'album_bundle_partial': bool(outcome.get('partial')),
|
||||
'album_bundle_expected_count': outcome.get('expected_count'),
|
||||
'album_bundle_completed_count': outcome.get('completed_count', len(outcome.get('files', []))),
|
||||
})
|
||||
# Engaged-and-succeeded: we DON'T early-return because the
|
||||
# per-track flow needs to run to create + complete the per-track
|
||||
# task rows. Those tasks will hit try_staging_match and pull the
|
||||
# files we just staged.
|
||||
return False
|
||||
|
||||
|
||||
def _safe_batch_dirname(batch_id: str) -> str:
|
||||
safe = ''.join(ch if ch.isalnum() or ch in ('-', '_') else '_' for ch in str(batch_id or 'batch'))
|
||||
return safe or 'batch'
|
||||
|
|
@ -45,9 +45,6 @@ _TERMINAL_STATUSES = {
|
|||
def cancel_single_download(download_orchestrator, run_async: Callable,
|
||||
download_id: str, username: str) -> bool:
|
||||
"""Cancel one specific slskd download (with `remove=True`)."""
|
||||
logger.info(
|
||||
f"[CancelTrigger:api.manual_cancel_single] download_id={download_id} username={username}"
|
||||
)
|
||||
return run_async(download_orchestrator.cancel_download(download_id, username, remove=True))
|
||||
|
||||
|
||||
|
|
@ -83,26 +80,9 @@ def clear_completed_local() -> int:
|
|||
"""
|
||||
cleared = 0
|
||||
with tasks_lock:
|
||||
# Protect tasks belonging to a still-active batch. A batch is "active"
|
||||
# while any of its queued tasks is non-terminal (still searching /
|
||||
# downloading / queued / post-processing). Pruning a batch's completed
|
||||
# or failed tasks mid-run would yank them out of the Downloads page —
|
||||
# and failed/cancelled rows aren't recoverable from library_history —
|
||||
# so the user would never see them until the batch ended. Keep the whole
|
||||
# active batch intact; it gets cleaned by a later run once it finishes.
|
||||
protected_task_ids: set = set()
|
||||
for batch in download_batches.values():
|
||||
queue = batch.get('queue', []) if isinstance(batch, dict) else []
|
||||
batch_active = any(
|
||||
download_tasks.get(tid, {}).get('status') not in _TERMINAL_STATUSES
|
||||
for tid in queue if tid in download_tasks
|
||||
)
|
||||
if batch_active:
|
||||
protected_task_ids.update(queue)
|
||||
|
||||
task_ids_to_remove = [
|
||||
tid for tid, task in download_tasks.items()
|
||||
if task.get('status') in _TERMINAL_STATUSES and tid not in protected_task_ids
|
||||
if task.get('status') in _TERMINAL_STATUSES
|
||||
]
|
||||
for tid in task_ids_to_remove:
|
||||
del download_tasks[tid]
|
||||
|
|
|
|||
|
|
@ -36,7 +36,6 @@ import os
|
|||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
from core.downloads.track_metadata_backfill import hydrate_download_metadata
|
||||
from core.runtime_state import (
|
||||
download_tasks,
|
||||
matched_context_lock,
|
||||
|
|
@ -47,55 +46,6 @@ from core.runtime_state import (
|
|||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _priority_sort_key(r):
|
||||
"""Today's confidence-first key: never download a high-quality WRONG file."""
|
||||
return (
|
||||
getattr(r, 'confidence', 0) or 0,
|
||||
getattr(r, 'quality_score', 0) or 0,
|
||||
getattr(r, 'upload_speed', 0) or 0,
|
||||
-(getattr(r, 'queue_length', 0) or 0),
|
||||
getattr(r, 'free_upload_slots', 0) or 0,
|
||||
getattr(r, 'size', 0) or 0,
|
||||
)
|
||||
|
||||
|
||||
def _quality_first_sort_key(r, targets):
|
||||
"""Best-quality key: the user's profile quality rank dominates; all the
|
||||
priority-mode signals (confidence, speed, …) become tiebreakers.
|
||||
|
||||
Every candidate reaching this point already passed match filtering, so it
|
||||
is "correct enough" — ordering by quality among correct candidates is safe.
|
||||
Candidates with no usable quality info, or that match no target, sort last
|
||||
(never dropped). Lower target index = better target, so it's negated to fit
|
||||
the descending (reverse=True) sort.
|
||||
"""
|
||||
from core.quality.model import rank_candidate
|
||||
|
||||
aq = getattr(r, 'audio_quality', None)
|
||||
if aq is None or not targets:
|
||||
target_idx, tier = (len(targets) if targets else 0), 0.0
|
||||
else:
|
||||
try:
|
||||
target_idx, tier = rank_candidate(aq, targets)
|
||||
except Exception:
|
||||
target_idx, tier = len(targets), 0.0
|
||||
return (-target_idx, tier) + _priority_sort_key(r)
|
||||
|
||||
|
||||
def order_candidates(candidates, *, quality_first=False, targets=None):
|
||||
"""Return *candidates* ordered best-first for the download walk.
|
||||
|
||||
``quality_first=False`` (priority mode) → confidence-first, byte-for-byte
|
||||
today's behaviour. ``quality_first=True`` (best-quality mode) → the user's
|
||||
profile quality rank dominates, confidence/peer signals break ties.
|
||||
"""
|
||||
if quality_first:
|
||||
key = lambda r: _quality_first_sort_key(r, targets or [])
|
||||
else:
|
||||
key = _priority_sort_key
|
||||
return sorted(candidates, key=key, reverse=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CandidatesDeps:
|
||||
"""Bundle of cross-cutting deps the candidate-fallback logic needs."""
|
||||
|
|
@ -108,37 +58,19 @@ class CandidatesDeps:
|
|||
on_download_completed: Callable
|
||||
|
||||
|
||||
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
||||
deps: CandidatesDeps = None, *,
|
||||
quality_first=False, quality_targets=None):
|
||||
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None, deps: CandidatesDeps = None):
|
||||
"""
|
||||
Attempts to download with fallback candidate logic (matches GUI's retry_parallel_download_with_fallback).
|
||||
Returns True if successful, False if all candidates fail.
|
||||
|
||||
``quality_first`` (best-quality search mode) orders the walk by the user's
|
||||
profile quality rank instead of confidence-first; ``quality_targets`` is the
|
||||
profile target list used for that ranking. Defaults preserve priority-mode
|
||||
behaviour exactly.
|
||||
"""
|
||||
# Sort candidates. Priority mode: confidence-first, then peer quality —
|
||||
# upstream Soulseek validation already considers peer speed/slots/queue when
|
||||
# scores are close; preserve that signal instead of flattening ties back to
|
||||
# arbitrary slskd response order. Best-quality mode: profile quality rank
|
||||
# dominates (all candidates here already passed match filtering).
|
||||
candidates = order_candidates(
|
||||
candidates, quality_first=quality_first, targets=quality_targets,
|
||||
)
|
||||
# Sort candidates by confidence (best first)
|
||||
candidates.sort(key=lambda r: r.confidence, reverse=True)
|
||||
|
||||
with tasks_lock:
|
||||
task = download_tasks.get(task_id)
|
||||
if not task:
|
||||
return False
|
||||
used_sources = task.get('used_sources', set())
|
||||
# User-initiated manual picks (candidates modal) bypass quarantine
|
||||
# gates downstream. The user already accepted the risk by choosing
|
||||
# the file; we trust their selection over AcoustID disagreement so
|
||||
# repeated manual picks don't loop back into quarantine.
|
||||
user_manual_pick = bool(task.get('_user_manual_pick', False))
|
||||
|
||||
# Try each candidate until one succeeds (like GUI's fallback logic)
|
||||
for candidate_index, candidate in enumerate(candidates):
|
||||
|
|
@ -254,21 +186,6 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
|||
'artists': _fallback_album_artists
|
||||
}
|
||||
|
||||
# #915: parity with Reorganize / manual Enrich. If the album context is lean
|
||||
# (no release_date) and the user's PRIMARY metadata source isn't Spotify, hydrate
|
||||
# it from that source — the same place a reorganize reads — so the download's
|
||||
# $year folder, release_date and album_type match instead of dropping the year /
|
||||
# defaulting to YYYY-01-01 and forcing a manual reorganize afterwards.
|
||||
try:
|
||||
from core.downloads.track_metadata_backfill import backfill_album_context_from_source
|
||||
from core.metadata import registry as _meta_registry
|
||||
from core.metadata.album_tracks import get_album_for_source as _get_album_for_source
|
||||
backfill_album_context_from_source(
|
||||
spotify_album_context, _meta_registry.get_primary_source(), _get_album_for_source,
|
||||
)
|
||||
except Exception as _bf_err: # noqa: BLE001 — never let backfill break a download
|
||||
logger.debug("[Context] primary-source album backfill skipped: %s", _bf_err)
|
||||
|
||||
download_payload = candidate.__dict__
|
||||
|
||||
username = download_payload.get('username')
|
||||
|
|
@ -312,30 +229,55 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
|||
enhanced_payload['artists'] = [{'name': artist} for artist in track.artists] if track.artists else []
|
||||
logger.info(f"[Context] Using clean Spotify metadata - Album: '{track.album}', Title: '{track.name}'")
|
||||
|
||||
# Resolve track_number / disc_number and hydrate
|
||||
# lean album context. Extracted to
|
||||
# track_metadata_backfill.hydrate_download_metadata
|
||||
# — see that module for the precedence chain.
|
||||
# Why the extract: the inline pre-fix coupled
|
||||
# album-backfill to the "track_number missing"
|
||||
# branch. When wishlist payloads carried a poisoned
|
||||
# default-1 track_number (older routes.py used
|
||||
# ``.get('track_number', 1)``) the API call short-
|
||||
# circuited and the lean album_context (no
|
||||
# release_date / total_tracks for Deezer-sourced
|
||||
# discovery matches) survived untouched, producing
|
||||
# folders without a year subfolder.
|
||||
resolved = hydrate_download_metadata(
|
||||
track, track_info, spotify_album_context, deps.spotify_client,
|
||||
)
|
||||
if resolved.track_number is not None:
|
||||
enhanced_payload['track_number'] = resolved.track_number
|
||||
enhanced_payload['disc_number'] = resolved.disc_number
|
||||
logger.info(
|
||||
f"[Context] Added track_number from {resolved.source}: "
|
||||
f"{resolved.track_number}, disc_number: {resolved.disc_number}"
|
||||
)
|
||||
else:
|
||||
# Get track_number and disc_number — prefer track data we already have,
|
||||
# fall back to detailed API call only if needed
|
||||
got_track_number = False
|
||||
|
||||
# 1. Try track_info (from frontend, has album track data)
|
||||
tn = track_info.get('track_number', 0) if isinstance(track_info, dict) else 0
|
||||
dn = track_info.get('disc_number', 1) if isinstance(track_info, dict) else 1
|
||||
if tn and tn > 0:
|
||||
enhanced_payload['track_number'] = tn
|
||||
enhanced_payload['disc_number'] = dn
|
||||
got_track_number = True
|
||||
logger.info(f"[Context] Added track_number from track_info: {tn}, disc_number: {dn}")
|
||||
|
||||
# 2. Try the track object itself (from album tracks response)
|
||||
if not got_track_number and hasattr(track, 'track_number') and track.track_number:
|
||||
enhanced_payload['track_number'] = track.track_number
|
||||
enhanced_payload['disc_number'] = getattr(track, 'disc_number', 1) or 1
|
||||
got_track_number = True
|
||||
logger.info(f"[Context] Added track_number from track object: {track.track_number}, disc_number: {enhanced_payload['disc_number']}")
|
||||
|
||||
# 3. Last resort — fetch from metadata source API
|
||||
if not got_track_number and hasattr(track, 'id') and track.id:
|
||||
try:
|
||||
detailed_track = deps.spotify_client.get_track_details(track.id)
|
||||
if detailed_track and detailed_track.get('track_number'):
|
||||
enhanced_payload['track_number'] = detailed_track['track_number']
|
||||
enhanced_payload['disc_number'] = detailed_track.get('disc_number', 1)
|
||||
got_track_number = True
|
||||
logger.info(f"[Context] Added track_number from API: {detailed_track['track_number']}, disc_number: {enhanced_payload['disc_number']}")
|
||||
|
||||
# Backfill album metadata from detailed track when context
|
||||
# has incomplete data (missing release_date, total_tracks, etc.)
|
||||
if isinstance(detailed_track.get('album'), dict):
|
||||
dt_album = detailed_track['album']
|
||||
if not spotify_album_context.get('release_date') and dt_album.get('release_date'):
|
||||
spotify_album_context['release_date'] = dt_album['release_date']
|
||||
logger.info(f"[Context] Backfilled release_date from API: {dt_album['release_date']}")
|
||||
if not spotify_album_context.get('album_type') and dt_album.get('album_type'):
|
||||
spotify_album_context['album_type'] = dt_album['album_type']
|
||||
if not spotify_album_context.get('total_tracks') and dt_album.get('total_tracks'):
|
||||
spotify_album_context['total_tracks'] = dt_album['total_tracks']
|
||||
if not spotify_album_context.get('id') and dt_album.get('id'):
|
||||
spotify_album_context['id'] = dt_album['id']
|
||||
if not spotify_album_context.get('image_url') and dt_album.get('images'):
|
||||
spotify_album_context['image_url'] = dt_album['images'][0].get('url', '')
|
||||
except Exception as e:
|
||||
logger.error(f"[Context] API track details failed: {e}")
|
||||
|
||||
if not got_track_number:
|
||||
enhanced_payload.setdefault('track_number', 0)
|
||||
enhanced_payload.setdefault('disc_number', 1)
|
||||
logger.warning("[Context] No track_number found from any source")
|
||||
|
|
@ -376,33 +318,9 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
|||
"track_info": track_info, # Add track_info for playlist folder mode
|
||||
"_download_username": username, # Source username for AcoustID skip logic
|
||||
}
|
||||
if user_manual_pick:
|
||||
# The user explicitly picked this candidate via the
|
||||
# candidates modal — trust their metadata judgement
|
||||
# over AcoustID disagreement so manual picks don't
|
||||
# loop back into quarantine. Integrity + bit-depth
|
||||
# gates still run because those check the new file's
|
||||
# actual condition, not its identity.
|
||||
matched_downloads_context[context_key]['_skip_quarantine_check'] = 'acoustid'
|
||||
matched_downloads_context[context_key]['_user_manual_pick'] = True
|
||||
logger.info(
|
||||
"[Context] User manual pick — bypassing AcoustID for "
|
||||
"task=%s username=%s filename=%s",
|
||||
task_id, username, os.path.basename(filename),
|
||||
)
|
||||
elif track_info and track_info.get('_skip_acoustid'):
|
||||
# Issue #797 — the album-download request had the
|
||||
# per-request "Skip AcoustID verification" toggle on.
|
||||
# Bypass only the AcoustID gate (same as a manual
|
||||
# pick); integrity + bit-depth still run.
|
||||
matched_downloads_context[context_key]['_skip_quarantine_check'] = 'acoustid'
|
||||
logger.info(
|
||||
"[Context] Skip-AcoustID toggle — bypassing AcoustID for "
|
||||
"task=%s filename=%s",
|
||||
task_id, os.path.basename(filename),
|
||||
)
|
||||
|
||||
logger.info(f"[Context] Set is_album_download: {is_album_context} (has clean data: {has_clean_spotify_data})")
|
||||
logger.debug(f"[Debug] Context creation - track_info: {track_info is not None}, playlist_folder_mode: {track_info.get('_playlist_folder_mode', False) if track_info else False}")
|
||||
|
||||
# Update task with successful download info
|
||||
with tasks_lock:
|
||||
|
|
@ -412,10 +330,6 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
|||
logger.warning(f"[Modal Worker] Task {task_id} cancelled after download {download_id} started - attempting to cancel download")
|
||||
# Try to cancel the download immediately
|
||||
try:
|
||||
logger.info(
|
||||
f"[CancelTrigger:candidates.worker_cancelled_during_download] "
|
||||
f"download_id={download_id} username={username} task_id={task_id}"
|
||||
)
|
||||
deps.run_async(deps.download_orchestrator.cancel_download(download_id, username, remove=True))
|
||||
logger.warning(f"Successfully cancelled active download {download_id}")
|
||||
except Exception as cancel_error:
|
||||
|
|
|
|||
|
|
@ -1,295 +0,0 @@
|
|||
"""Robust completed-download file finder.
|
||||
|
||||
Walks a download directory (and optional transfer directory) to find
|
||||
the local file matching an API-reported remote filename. Handles:
|
||||
|
||||
- Arbitrary subdirectory layouts (slskd flat / username-prefixed /
|
||||
remote-tree-preserved). Pre-extract callers in the Soulseek
|
||||
album-bundle path tried three hard-coded candidate paths and
|
||||
silently failed on layouts that didn't match — see issue #715
|
||||
(Billy Ocean album task fails after slskd finishes downloading
|
||||
release). The per-track flow already used a recursive walk and
|
||||
worked; the bundle path didn't, so users on common slskd configs
|
||||
with username-prefixed downloads saw bundles time out 22 minutes
|
||||
after slskd reported every transfer Completed.
|
||||
- slskd dedup suffix ``_<10-or-more-digit-timestamp>`` appended when
|
||||
a file with the same basename already exists.
|
||||
- YouTube / Tidal encoded filename format ``id||title`` — the
|
||||
``||`` half is the human title and used for matching.
|
||||
- Multiple files sharing a basename — disambiguates by counting
|
||||
how many remote-path directory components appear in the local
|
||||
path.
|
||||
|
||||
This was lifted verbatim from ``web_server._find_completed_file_robust``
|
||||
so both the per-track download poll AND the Soulseek album-bundle
|
||||
poll go through one finder. Pre-extract the bundle path probed
|
||||
three hard-coded candidates only, which is why bundle downloads
|
||||
on slskd setups with username-prefixed download dirs silently
|
||||
timed out (#715).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from difflib import SequenceMatcher
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from unidecode import unidecode
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
AUDIO_EXTENSIONS = frozenset({
|
||||
'.mp3', '.flac', '.m4a', '.aac', '.ogg', '.opus', '.wav', '.wma',
|
||||
'.alac', '.aiff', '.aif', '.dsf', '.dff', '.ape',
|
||||
})
|
||||
|
||||
# slskd appends a 10+ digit timestamp suffix when a file with the
|
||||
# same basename already exists. Match-strip those so we still
|
||||
# resolve the transfer.
|
||||
_SLSKD_DEDUP_SUFFIX = re.compile(r'_\d{10,}$')
|
||||
|
||||
# AcoustID-quarantined files live under this dirname and must be
|
||||
# skipped — they are known-wrong matches the verifier rejected.
|
||||
_QUARANTINE_DIRNAME = 'ss_quarantine'
|
||||
|
||||
# Confidence floor for accepting a fuzzy basename match. Anything
|
||||
# below this is treated as "no match" so we don't drag in unrelated
|
||||
# files.
|
||||
_FUZZY_THRESHOLD = 0.85
|
||||
|
||||
|
||||
def _is_audio_candidate(path: str) -> bool:
|
||||
return os.path.splitext(str(path or ''))[1].lower() in AUDIO_EXTENSIONS
|
||||
|
||||
|
||||
def _normalize_for_finding(text: str) -> str:
|
||||
"""Match-engine-style normalisation for fuzzy filename comparison.
|
||||
|
||||
Lowercases, transliterates unicode, drops bracketed content
|
||||
("(Remastered 2016)", "[FLAC]"), strips punctuation, collapses
|
||||
whitespace. Mirrors ``matching_engine.py``'s text normaliser so
|
||||
finder + matcher agree on equivalence.
|
||||
"""
|
||||
if not text:
|
||||
return ""
|
||||
text = unidecode(text).lower()
|
||||
text = re.sub(r'[._/]', ' ', text)
|
||||
# Strip ONLY balanced bracket pairs (tags like "[FLAC]", "(Remastered 2016)").
|
||||
# The old combined pattern r'[\[\(].*?[\]\)]' allowed MISMATCHED delimiters, so a
|
||||
# lone unbalanced '[' — slskd reports "[34 - You & Me (Flume Remix)" but saves the
|
||||
# file as "34 - You & Me (Flume Remix)" — matched from that '[' all the way to the
|
||||
# next ')', eating the entire title and collapsing the search target to "flac". The
|
||||
# file then scored 0.40 against the real on-disk name and was reported "not found"
|
||||
# despite sitting right there. Per-delimiter pairs can't over-consume; a stray
|
||||
# unbalanced bracket simply survives to the alphanumeric strip below.
|
||||
text = re.sub(r'\[[^\]]*\]', '', text)
|
||||
text = re.sub(r'\([^)]*\)', '', text)
|
||||
text = re.sub(r'[^a-z0-9\s-]', '', text)
|
||||
return ' '.join(text.split()).strip()
|
||||
|
||||
|
||||
def _extract_basename(api_filename: str) -> str:
|
||||
"""Cross-platform rightmost-separator split for a real remote PATH.
|
||||
|
||||
A YouTube/Tidal/Qobuz ``id||title`` encoded filename is handled by
|
||||
returning the title VERBATIM: the title is not a filesystem path, so a '/'
|
||||
in it (e.g. the Sawano track ``YouSeeBIGGIRL/T:T``) is part of the name and
|
||||
must NOT be split on (issue #835)."""
|
||||
if not api_filename:
|
||||
return ""
|
||||
if '||' in api_filename:
|
||||
_id, title = api_filename.split('||', 1)
|
||||
return title
|
||||
last_slash = max(api_filename.rfind('/'), api_filename.rfind('\\'))
|
||||
return api_filename[last_slash + 1:] if last_slash != -1 else api_filename
|
||||
|
||||
|
||||
def _api_dir_parts(api_filename: str) -> list[str]:
|
||||
"""Lowercased remote-path directory components, sans the
|
||||
filename itself. Used to disambiguate when several local files
|
||||
share the same basename — the one whose path mirrors the
|
||||
remote folder structure wins."""
|
||||
if not api_filename:
|
||||
return []
|
||||
normalized = api_filename.replace('\\', '/')
|
||||
return [p.lower() for p in normalized.split('/')[:-1] if p]
|
||||
|
||||
|
||||
def _path_matches_api_dirs(file_path: str, api_dirs: list[str]) -> bool:
|
||||
"""``True`` iff every remote directory component appears as a
|
||||
path part of the local file. Cheap "is this file on a sibling
|
||||
tree" check."""
|
||||
if not api_dirs:
|
||||
return False
|
||||
path_parts = set(p.lower() for p in file_path.replace('\\', '/').split('/'))
|
||||
return all(d in path_parts for d in api_dirs)
|
||||
|
||||
|
||||
def _search_in_directory(
|
||||
search_dir: str,
|
||||
location_name: str,
|
||||
target_basename: str,
|
||||
normalized_target: str,
|
||||
api_dirs: list[str],
|
||||
) -> Tuple[Optional[str], float]:
|
||||
"""Walk ``search_dir`` once, return (best_match, similarity).
|
||||
|
||||
Priority order, highest first:
|
||||
1. Exact basename match with directory-structure confirmation
|
||||
2. Exact basename match without disambiguation (when no api_dirs)
|
||||
3. slskd-dedup-suffix basename match (same two tiers as exact)
|
||||
4. Best fuzzy basename match above ``_FUZZY_THRESHOLD``
|
||||
"""
|
||||
best_fuzzy_path: Optional[str] = None
|
||||
highest_fuzzy_similarity = 0.0
|
||||
exact_matches: list[str] = []
|
||||
|
||||
for root, dirs, files in os.walk(search_dir):
|
||||
# Strip quarantine subdir from the walk in place — these
|
||||
# files are known-bad and matching them would re-poison the
|
||||
# post-process pipeline.
|
||||
dirs[:] = [d for d in dirs if d != _QUARANTINE_DIRNAME]
|
||||
|
||||
for filename in files:
|
||||
file_path = os.path.join(root, filename)
|
||||
if not _is_audio_candidate(file_path):
|
||||
continue
|
||||
|
||||
# Tier 1 + 2: exact basename match.
|
||||
if filename == target_basename:
|
||||
if api_dirs and _path_matches_api_dirs(file_path, api_dirs):
|
||||
logger.info(
|
||||
"Found path-confirmed match in %s: %s",
|
||||
location_name, file_path,
|
||||
)
|
||||
return file_path, 1.0
|
||||
if not api_dirs:
|
||||
logger.info(
|
||||
"Found exact match in %s: %s",
|
||||
location_name, file_path,
|
||||
)
|
||||
return file_path, 1.0
|
||||
exact_matches.append(file_path)
|
||||
continue
|
||||
|
||||
# Tier 3: slskd dedup suffix.
|
||||
stem, ext = os.path.splitext(filename)
|
||||
stripped_stem = _SLSKD_DEDUP_SUFFIX.sub('', stem)
|
||||
if stripped_stem != stem and stripped_stem + ext == target_basename:
|
||||
if api_dirs and _path_matches_api_dirs(file_path, api_dirs):
|
||||
logger.info(
|
||||
"Found path-confirmed dedup match in %s: %s",
|
||||
location_name, file_path,
|
||||
)
|
||||
return file_path, 1.0
|
||||
if not api_dirs:
|
||||
logger.info(
|
||||
"Found dedup-suffix match in %s: %s",
|
||||
location_name, file_path,
|
||||
)
|
||||
return file_path, 1.0
|
||||
exact_matches.append(file_path)
|
||||
continue
|
||||
|
||||
# Tier 4: fuzzy basename match. Cheaper than path-walking
|
||||
# the whole tree a second time, so always compute and
|
||||
# keep the best one as a fallback.
|
||||
normalized_file = _normalize_for_finding(filename)
|
||||
similarity = SequenceMatcher(
|
||||
None, normalized_target, normalized_file,
|
||||
).ratio()
|
||||
if similarity > highest_fuzzy_similarity:
|
||||
highest_fuzzy_similarity = similarity
|
||||
best_fuzzy_path = file_path
|
||||
|
||||
if exact_matches:
|
||||
if len(exact_matches) == 1:
|
||||
logger.info(
|
||||
"Found exact match in %s: %s",
|
||||
location_name, exact_matches[0],
|
||||
)
|
||||
return exact_matches[0], 1.0
|
||||
# Multiple basename collisions — pick the one whose path
|
||||
# carries the most of the remote directory tree (album
|
||||
# folder etc.). Breaks ties deterministically.
|
||||
best = exact_matches[0]
|
||||
best_score = -1
|
||||
for m in exact_matches:
|
||||
m_parts = set(p.lower() for p in m.replace('\\', '/').split('/'))
|
||||
score = sum(1 for d in api_dirs if d in m_parts)
|
||||
if score > best_score:
|
||||
best_score = score
|
||||
best = m
|
||||
logger.info(
|
||||
"Found %d files named '%s' in %s, picked best path match: %s",
|
||||
len(exact_matches), target_basename, location_name, best,
|
||||
)
|
||||
return best, 1.0
|
||||
|
||||
return best_fuzzy_path, highest_fuzzy_similarity
|
||||
|
||||
|
||||
def find_completed_audio_file(
|
||||
download_dir: str,
|
||||
api_filename: str,
|
||||
transfer_dir: Optional[str] = None,
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""Locate a completed download's local file via recursive walk.
|
||||
|
||||
Tries the downloads tree first; if nothing above the fuzzy
|
||||
threshold lands AND a transfer_dir was passed, tries that too.
|
||||
|
||||
Returns ``(file_path, location)`` where ``location`` is
|
||||
``'downloads'`` / ``'transfer'`` / ``None``. Both elements are
|
||||
``None`` when the file isn't found anywhere — callers should
|
||||
treat that as "not yet" (still mid-write) or "lost".
|
||||
"""
|
||||
# YouTube / Tidal / Qobuz encoded filenames carry the id ahead of ``||``.
|
||||
# The title half is NOT a filesystem path: a '/' in it (e.g. the Sawano
|
||||
# track ``YouSeeBIGGIRL/T:T``) is part of the title, so it must NOT be
|
||||
# basename-split or read as a remote directory component — doing so
|
||||
# truncated the search target to ``T:T`` and the real file was never found,
|
||||
# quarantining valid downloads (issue #835). Real remote paths (Soulseek)
|
||||
# still get basename + dir-component extraction.
|
||||
encoded_title = None
|
||||
if api_filename and '||' in api_filename:
|
||||
_id, encoded_title = api_filename.split('||', 1)
|
||||
if encoded_title is not None:
|
||||
target_basename = encoded_title
|
||||
api_dirs = []
|
||||
else:
|
||||
target_basename = _extract_basename(api_filename)
|
||||
api_dirs = _api_dir_parts(api_filename)
|
||||
normalized_target = _normalize_for_finding(target_basename)
|
||||
|
||||
best_dl_path, dl_sim = _search_in_directory(
|
||||
download_dir, 'downloads', target_basename, normalized_target, api_dirs,
|
||||
)
|
||||
|
||||
if dl_sim > _FUZZY_THRESHOLD:
|
||||
if dl_sim < 1.0:
|
||||
logger.info(
|
||||
"Found fuzzy match in downloads (%.2f): %s",
|
||||
dl_sim, best_dl_path,
|
||||
)
|
||||
return (best_dl_path, 'downloads')
|
||||
|
||||
if transfer_dir and os.path.exists(transfer_dir):
|
||||
best_tx_path, tx_sim = _search_in_directory(
|
||||
transfer_dir, 'transfer', target_basename, normalized_target, api_dirs,
|
||||
)
|
||||
if tx_sim > _FUZZY_THRESHOLD:
|
||||
if tx_sim < 1.0:
|
||||
logger.info(
|
||||
"Found fuzzy match in transfer (%.2f): %s",
|
||||
tx_sim, best_tx_path,
|
||||
)
|
||||
return (best_tx_path, 'transfer')
|
||||
|
||||
return (None, None)
|
||||
|
||||
|
||||
__all__ = ['find_completed_audio_file', 'AUDIO_EXTENSIONS']
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
"""Match a file back to its download-history row when its path has drifted (#934).
|
||||
|
||||
``library_history.file_path`` is frozen at import time, but the file moves afterward
|
||||
(media-server import, library reorganize) and ``tracks.file_path`` — what the AcoustID
|
||||
scanner reads — no longer equals it. Matching on the exact path alone then fails twice:
|
||||
the verification status never reaches the history row (verified tracks read "unverified"),
|
||||
and a fresh ``acoustid_scan`` row gets inserted every run (thousands of duplicates).
|
||||
|
||||
This module picks the canonical history row by exact path first, then by FILENAME guarded
|
||||
by a title check — so a shared filename ("01 - Intro.flac") can never heal the wrong song.
|
||||
Pure (no DB) so the matching rules are unit-testable; the caller does the SQL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Iterable, Optional, Sequence, Tuple
|
||||
|
||||
|
||||
def _norm_title(value) -> str:
|
||||
"""Alphanumeric-only lowercase form, so "Song (Remaster)" vs "song remaster"
|
||||
style drift between the download tag and the media-server tag still agrees."""
|
||||
return ''.join(ch for ch in str(value or '').lower() if ch.isalnum())
|
||||
|
||||
|
||||
def like_filename_filter(basename: str) -> str:
|
||||
"""A ``LIKE ... ESCAPE '\\'`` pattern that coarsely matches rows whose path ends
|
||||
in ``basename``. Escapes the LIKE metacharacters (``%`` ``_`` ``\\``) — filenames
|
||||
routinely contain underscores. Callers MUST still confirm with an exact basename
|
||||
compare (``pick_history_row`` does), since ``'%name'`` also matches ``'xname'``."""
|
||||
esc = basename.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_')
|
||||
return '%' + esc
|
||||
|
||||
|
||||
def pick_history_row(candidates: Sequence[Tuple], *, current_paths: Iterable[str],
|
||||
basename: str, title: str) -> Optional[int]:
|
||||
"""Return the id of the history row to update for this file, or None.
|
||||
|
||||
``candidates``: ``(id, file_path, title, download_source)`` rows the DB pre-filtered
|
||||
(exact path or filename LIKE). A row matches when its path equals the current path OR
|
||||
its filename matches AND its title agrees — the title guard prevents a shared filename
|
||||
("01 - Intro.flac") from healing a different song's row. Among matches a REAL download
|
||||
row is preferred over a synthetic ``acoustid_scan`` row, so the scanner heals the
|
||||
genuine record and the caller can delete the synthetic duplicate. None when nothing
|
||||
matches safely (caller then inserts a fresh row — the "file SoulSync never downloaded"
|
||||
intent)."""
|
||||
paths = {p for p in current_paths if p}
|
||||
want = _norm_title(title)
|
||||
matches: list = [] # (id, is_exact, is_real)
|
||||
for cid, cpath, ctitle, csource in candidates:
|
||||
is_real = csource != 'acoustid_scan'
|
||||
if cpath and cpath in paths:
|
||||
matches.append((cid, True, is_real))
|
||||
elif (basename and cpath and os.path.basename(cpath) == basename
|
||||
and (not want or not _norm_title(ctitle) or _norm_title(ctitle) == want)):
|
||||
matches.append((cid, False, is_real))
|
||||
if not matches:
|
||||
return None
|
||||
# Prefer a REAL download row over a synthetic acoustid_scan row; within that, prefer an
|
||||
# exact-path match over a filename match. Stable, so ties keep DB order (first/oldest id).
|
||||
matches.sort(key=lambda m: (m[2], m[1]), reverse=True)
|
||||
return matches[0][0]
|
||||
|
|
@ -26,12 +26,9 @@ Lifted verbatim from web_server.py. Dependencies injected via
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
import traceback
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from core.downloads.history import record_sync_history_completion
|
||||
|
|
@ -45,151 +42,6 @@ from core.runtime_state import (
|
|||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# A task that has been in 'post_processing' longer than this is treated as stuck.
|
||||
# Post-processing (AcoustID + quality + import) is serialized, so a large batch
|
||||
# legitimately backs up — keep this generous so genuinely-slow imports aren't
|
||||
# cut off mid-flight (the old 5-min cutoff falsely "completed" queued tasks).
|
||||
_POST_PROCESSING_STUCK_TIMEOUT = 1800 # 30 minutes
|
||||
|
||||
|
||||
def _resolve_stuck_post_processing_status(task: dict) -> str:
|
||||
"""Decide the terminal status for a task stuck in post_processing.
|
||||
|
||||
Only call it 'completed' if the import actually produced a file on disk
|
||||
(``final_file_path`` is set at the end of successful post-processing). Without
|
||||
a real file, force-completing is a lie — the task shows as a downloaded track
|
||||
that isn't anywhere. Mark those 'failed' so they're retryable and honest.
|
||||
"""
|
||||
final_path = task.get('final_file_path')
|
||||
if final_path and os.path.exists(final_path):
|
||||
return 'completed'
|
||||
return 'failed'
|
||||
|
||||
|
||||
def _safe_batch_dirname(batch_id: str) -> str:
|
||||
safe = ''.join(ch if ch.isalnum() or ch in ('-', '_') else '_' for ch in str(batch_id or 'batch'))
|
||||
return safe or 'batch'
|
||||
|
||||
|
||||
_ALBUM_BUNDLE_CLEANED_SOURCES = ('soulseek', 'torrent', 'usenet')
|
||||
|
||||
|
||||
def _cleanup_private_album_bundle_staging(batch_id: str, batch: dict) -> None:
|
||||
"""Best-effort cleanup for album-bundle private staging copies.
|
||||
|
||||
Fires when a batch reaches a terminal state. SoulSync's per-batch
|
||||
copy lives at ``storage/album_bundle_staging/<batch_id>/`` —
|
||||
safe to remove because by the time the batch is "complete" the
|
||||
per-track workers have already claimed their files out of staging
|
||||
via ``try_staging_match`` and moved them to the Transfer dir.
|
||||
|
||||
Pre-fix this only ran for torrent / usenet bundles because the
|
||||
comment was "slskd keeps its own completed folders" — but the
|
||||
Soulseek bundle path ALSO copies files into the private staging
|
||||
dir (``soulseek_client.py:1599``), so slskd bundle copies were
|
||||
leaking forever. Coverage extended to all three bundle-capable
|
||||
sources.
|
||||
"""
|
||||
if not batch.get('album_bundle_private_staging'):
|
||||
return
|
||||
if (batch.get('album_bundle_source') or '').lower() not in _ALBUM_BUNDLE_CLEANED_SOURCES:
|
||||
return
|
||||
|
||||
staging_path = batch.get('album_bundle_staging_path')
|
||||
if not staging_path:
|
||||
return
|
||||
|
||||
path = Path(staging_path)
|
||||
expected_name = _safe_batch_dirname(batch_id)
|
||||
if path.name != expected_name:
|
||||
logger.warning(
|
||||
"[Album Bundle] Refusing to clean private staging path with unexpected name: %s",
|
||||
staging_path,
|
||||
)
|
||||
return
|
||||
if not path.exists():
|
||||
return
|
||||
if not path.is_dir():
|
||||
logger.warning("[Album Bundle] Refusing to clean non-directory staging path: %s", staging_path)
|
||||
return
|
||||
|
||||
try:
|
||||
shutil.rmtree(path)
|
||||
logger.info("[Album Bundle] Cleaned private staging folder for batch %s: %s", batch_id, staging_path)
|
||||
except Exception as exc:
|
||||
logger.warning("[Album Bundle] Could not clean private staging folder %s: %s", staging_path, exc)
|
||||
|
||||
|
||||
def sweep_orphan_album_bundle_staging(
|
||||
staging_root: str,
|
||||
*,
|
||||
active_batch_ids: Optional[set] = None,
|
||||
) -> int:
|
||||
"""Remove orphan per-batch dirs from album-bundle staging.
|
||||
|
||||
An orphan is a ``<staging_root>/<dirname>`` subdir whose ``dirname``
|
||||
matches no batch_id in the current ``download_batches`` runtime
|
||||
state. Happens when:
|
||||
|
||||
- The app crashed mid-bundle (cleanup never fired).
|
||||
- A batch errored on a non-completion code path.
|
||||
- A pre-extension Soulseek bundle (where cleanup was gated to
|
||||
torrent/usenet) left a copy behind.
|
||||
|
||||
Intended to run ONCE at server startup, before any new batch can
|
||||
register an active staging dir. That guarantees ``active_batch_ids``
|
||||
is genuinely empty / pre-existing; we don't race a starting batch.
|
||||
|
||||
Returns the count of dirs removed. Safe-by-design:
|
||||
- Only touches subdirs of the configured staging root.
|
||||
- Each candidate goes through the same ``_safe_batch_dirname``
|
||||
name-guard as the per-batch cleanup, so escape-via-symlink
|
||||
isn't possible.
|
||||
- Refuses to act on non-directories.
|
||||
- ``shutil.rmtree`` errors are logged, not raised — sweep must
|
||||
not crash app startup over a permission glitch.
|
||||
"""
|
||||
if not staging_root:
|
||||
return 0
|
||||
root = Path(staging_root)
|
||||
if not root.exists() or not root.is_dir():
|
||||
return 0
|
||||
|
||||
active = active_batch_ids if active_batch_ids is not None else set()
|
||||
# Normalize active batch ids to their on-disk dirname form so the
|
||||
# set lookup matches what's actually on disk.
|
||||
active_dirnames = {_safe_batch_dirname(bid) for bid in active if bid}
|
||||
|
||||
removed = 0
|
||||
try:
|
||||
entries = list(root.iterdir())
|
||||
except OSError as exc:
|
||||
logger.warning("[Album Bundle Sweep] Could not list staging root %s: %s", staging_root, exc)
|
||||
return 0
|
||||
|
||||
for entry in entries:
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
# The directory name MUST match what _safe_batch_dirname
|
||||
# produces — anything else was hand-created and we leave it
|
||||
# alone. Defensive against stray dirs the user might have
|
||||
# placed in the staging root.
|
||||
if entry.name != _safe_batch_dirname(entry.name):
|
||||
continue
|
||||
if entry.name in active_dirnames:
|
||||
continue
|
||||
try:
|
||||
shutil.rmtree(entry)
|
||||
removed += 1
|
||||
logger.info("[Album Bundle Sweep] Removed orphan staging dir: %s", entry)
|
||||
except OSError as exc:
|
||||
logger.warning("[Album Bundle Sweep] Could not remove orphan staging dir %s: %s", entry, exc)
|
||||
|
||||
if removed:
|
||||
logger.info("[Album Bundle Sweep] Cleaned %d orphan staging dir(s) under %s", removed, staging_root)
|
||||
return removed
|
||||
|
||||
|
||||
@dataclass
|
||||
class LifecycleDeps:
|
||||
"""Bundle of cross-cutting deps the batch lifecycle needs."""
|
||||
|
|
@ -321,7 +173,7 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
|
|||
|
||||
# Guard against double-calling: track which tasks have already been completed
|
||||
# This prevents active_count from being decremented multiple times for the same task
|
||||
# (e.g. status polling and post-processing both observe the same terminal task)
|
||||
# (e.g. monitor detects completion AND post-processing calls this again)
|
||||
# NOTE: On duplicate calls, we skip decrement/tracking but STILL check batch completion.
|
||||
# This is critical because the first call may see the task in 'post_processing' (not finished),
|
||||
# and the second call (from post-processing worker) arrives after the task is truly 'completed'.
|
||||
|
|
@ -457,15 +309,9 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
|
|||
retrying_count += 1
|
||||
elif task_status == 'post_processing':
|
||||
task_age = current_time - task.get('status_change_time', current_time)
|
||||
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
|
||||
new_status = _resolve_stuck_post_processing_status(task)
|
||||
if new_status == 'completed':
|
||||
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
|
||||
task['status'] = 'completed'
|
||||
else:
|
||||
logger.warning(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
|
||||
task['status'] = 'failed'
|
||||
task['error_message'] = 'Post-processing timed out without producing a file'
|
||||
if task_age > 300: # 5 minutes (post-processing should be fast)
|
||||
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
|
||||
task['status'] = 'completed' # Assume it worked if file verification is taking too long
|
||||
finished_count += 1
|
||||
else:
|
||||
retrying_count += 1
|
||||
|
|
@ -552,7 +398,6 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
|
|||
|
||||
logger.info(f"[Batch Manager] Batch {batch_id} complete - stopping monitor")
|
||||
deps.download_monitor.stop_monitoring(batch_id)
|
||||
_cleanup_private_album_bundle_staging(batch_id, batch)
|
||||
|
||||
# M3U REGENERATION: Regenerate M3U with real library paths now that
|
||||
# all post-processing (tagging, moving, DB writes) is complete.
|
||||
|
|
@ -577,24 +422,6 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
|
|||
except Exception as m3u_err:
|
||||
logger.error(f"[M3U] Error regenerating M3U on batch complete: {m3u_err}")
|
||||
|
||||
# PLAYLIST MATERIALIZE: one path-independent reconcile — drop this
|
||||
# batch's newly-resolved tracks into the right Playlists/<name>/
|
||||
# folders. Covers an organize-by-playlist download AND a late
|
||||
# wishlist arrival (via each track's playlist provenance). Built
|
||||
# from the batch's own captured paths — non-fatal, derived view.
|
||||
try:
|
||||
from core.playlists.materialize_service import reconcile_batch_playlists
|
||||
from database.music_database import MusicDatabase
|
||||
for _pl_name, _mat in reconcile_batch_playlists(MusicDatabase(), batch, download_tasks, deps.config_manager):
|
||||
logger.info(
|
||||
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}': "
|
||||
f"{_mat.linked} linked, {_mat.copied} copied, "
|
||||
f"{_mat.unchanged} unchanged, {_mat.removed_stale} stale removed"
|
||||
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
|
||||
)
|
||||
except Exception as _mat_err:
|
||||
logger.error(f"[Playlist Folder] Materialize failed (non-fatal): {_mat_err}")
|
||||
|
||||
# REPAIR: Scan all album folders from this batch for track number issues
|
||||
if deps.repair_worker:
|
||||
deps.repair_worker.process_batch(batch_id)
|
||||
|
|
@ -695,15 +522,9 @@ def check_batch_completion_v2(batch_id: str, deps: LifecycleDeps) -> Optional[bo
|
|||
retrying_count += 1
|
||||
elif task_status == 'post_processing':
|
||||
task_age = current_time - task.get('status_change_time', current_time)
|
||||
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
|
||||
new_status = _resolve_stuck_post_processing_status(task)
|
||||
if new_status == 'completed':
|
||||
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
|
||||
task['status'] = 'completed'
|
||||
else:
|
||||
logger.warning(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
|
||||
task['status'] = 'failed'
|
||||
task['error_message'] = 'Post-processing timed out without producing a file'
|
||||
if task_age > 300: # 5 minutes (post-processing should be fast)
|
||||
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
|
||||
task['status'] = 'completed' # Assume it worked if file verification is taking too long
|
||||
finished_count += 1
|
||||
else:
|
||||
retrying_count += 1
|
||||
|
|
@ -785,24 +606,6 @@ def check_batch_completion_v2(batch_id: str, deps: LifecycleDeps) -> Optional[bo
|
|||
|
||||
logger.info(f"[Completion Check V2] Batch {batch_id} complete - stopping monitor")
|
||||
deps.download_monitor.stop_monitoring(batch_id)
|
||||
_cleanup_private_album_bundle_staging(batch_id, batch)
|
||||
|
||||
# PLAYLIST MATERIALIZE: same reconcile as the primary completion path
|
||||
# (on_download_completed). Monitor-detected downloads complete via THIS
|
||||
# V2 path, so the reconcile must run here too or playlist folders never
|
||||
# get built for them. Path-independent, non-fatal, derived view.
|
||||
try:
|
||||
from core.playlists.materialize_service import reconcile_batch_playlists
|
||||
from database.music_database import MusicDatabase
|
||||
for _pl_name, _mat in reconcile_batch_playlists(MusicDatabase(), batch, download_tasks, deps.config_manager):
|
||||
logger.info(
|
||||
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}': "
|
||||
f"{_mat.linked} linked, {_mat.copied} copied, "
|
||||
f"{_mat.unchanged} unchanged, {_mat.removed_stale} stale removed"
|
||||
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
|
||||
)
|
||||
except Exception as _mat_err:
|
||||
logger.error(f"[Playlist Folder] Materialize failed (non-fatal): {_mat_err}")
|
||||
|
||||
# REPAIR: Scan all album folders from this batch for track number issues
|
||||
if deps.repair_worker:
|
||||
|
|
|
|||
|
|
@ -31,244 +31,13 @@ import re
|
|||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from difflib import SequenceMatcher
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from core.downloads import album_bundle_dispatch as _album_bundle_dispatch
|
||||
from core.runtime_state import download_batches, download_tasks, tasks_lock
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
_ALBUM_PREFLIGHT_MIN_SCORE = 0.62
|
||||
_EDITION_WORDS = {
|
||||
'deluxe', 'expanded', 'anniversary', 'special', 'platinum', 'bonus',
|
||||
'remaster', 'remastered', 'edition', 'version',
|
||||
}
|
||||
_VARIANT_WORDS = {
|
||||
'remix', 'rmx', 'acapella', 'a cappella', 'instrumental', 'karaoke',
|
||||
'live', 'demo', 'extended',
|
||||
}
|
||||
_ALBUM_BUNDLE_SOURCES = frozenset(('torrent', 'usenet', 'soulseek'))
|
||||
|
||||
|
||||
def _norm_text(value: Any) -> str:
|
||||
text = str(value or '').lower()
|
||||
text = re.sub(r'[_./\\|()[\]{}:;,+]', ' ', text)
|
||||
text = re.sub(r'[^a-z0-9\s-]', '', text)
|
||||
text = re.sub(r'\s+', ' ', text).strip()
|
||||
return text
|
||||
|
||||
|
||||
def _similarity(left: Any, right: Any) -> float:
|
||||
a = _norm_text(left)
|
||||
b = _norm_text(right)
|
||||
if not a or not b:
|
||||
return 0.0
|
||||
if a == b:
|
||||
return 1.0
|
||||
if a in b or b in a:
|
||||
return min(len(a), len(b)) / max(len(a), len(b))
|
||||
return SequenceMatcher(None, a, b).ratio()
|
||||
|
||||
|
||||
def _track_title_from_candidate(candidate: Any) -> str:
|
||||
title = getattr(candidate, 'title', None)
|
||||
if title:
|
||||
return str(title)
|
||||
filename = getattr(candidate, 'filename', '') or ''
|
||||
stem = Path(filename.replace('\\', '/')).stem
|
||||
stem = re.sub(r'^\s*(?:disc\s*)?\d+[-_.\s]+', '', stem, flags=re.IGNORECASE)
|
||||
return stem
|
||||
|
||||
|
||||
def _track_number_from_track(track_data: dict) -> int:
|
||||
value = track_data.get('track_number') or track_data.get('trackNumber') or 0
|
||||
try:
|
||||
return int(str(value).split('/')[0])
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _track_number_from_candidate(candidate: Any) -> int:
|
||||
value = getattr(candidate, 'track_number', None) or 0
|
||||
try:
|
||||
return int(str(value).split('/')[0])
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _folder_variant_penalty(expected_album_name: str, folder_text: str) -> float:
|
||||
expected = _norm_text(expected_album_name)
|
||||
folder = _norm_text(folder_text)
|
||||
if not folder:
|
||||
return 0.0
|
||||
|
||||
penalty = 0.0
|
||||
for word in _VARIANT_WORDS:
|
||||
if word in folder and word not in expected:
|
||||
penalty += 0.12
|
||||
for word in _EDITION_WORDS:
|
||||
if word in folder and word not in expected:
|
||||
penalty += 0.06
|
||||
return min(penalty, 0.30)
|
||||
|
||||
|
||||
def _source_quality_score(source: Any) -> float:
|
||||
score = getattr(source, 'quality_score', None)
|
||||
if callable(score):
|
||||
try:
|
||||
return float(score())
|
||||
except Exception:
|
||||
return 0.0
|
||||
try:
|
||||
return float(score or 0.0)
|
||||
except (TypeError, ValueError):
|
||||
return 0.0
|
||||
|
||||
|
||||
def _album_context_richness(album_ctx: dict) -> int:
|
||||
if not isinstance(album_ctx, dict):
|
||||
return 0
|
||||
fields = ('id', 'name', 'release_date', 'total_tracks', 'album_type')
|
||||
score = sum(1 for field in fields if album_ctx.get(field))
|
||||
images = album_ctx.get('images')
|
||||
if images:
|
||||
score += 1
|
||||
artists = album_ctx.get('artists')
|
||||
if artists:
|
||||
score += 1
|
||||
return score
|
||||
|
||||
|
||||
def _score_album_folder(album_result: Any, album_context: dict, artist_context: dict,
|
||||
tracks_json: list[dict], filtered_track_count: int) -> float:
|
||||
"""Score one slskd folder as a whole release, not as isolated tracks."""
|
||||
expected_album = str((album_context or {}).get('name') or '')
|
||||
expected_artist = str((artist_context or {}).get('name') or '')
|
||||
expected_count = int((album_context or {}).get('total_tracks') or len(tracks_json) or 0)
|
||||
expected_year = str((album_context or {}).get('release_date') or '')[:4]
|
||||
|
||||
folder_text = ' '.join(
|
||||
str(getattr(album_result, attr, '') or '')
|
||||
for attr in ('album_title', 'album_path')
|
||||
)
|
||||
album_score = max(
|
||||
_similarity(expected_album, getattr(album_result, 'album_title', '')),
|
||||
_similarity(expected_album, getattr(album_result, 'album_path', '')),
|
||||
)
|
||||
artist_score = max(
|
||||
_similarity(expected_artist, getattr(album_result, 'artist', '')),
|
||||
_similarity(expected_artist, getattr(album_result, 'album_path', '')),
|
||||
)
|
||||
|
||||
actual_count = int(getattr(album_result, 'track_count', 0) or len(getattr(album_result, 'tracks', []) or []))
|
||||
if expected_count > 0 and actual_count > 0:
|
||||
diff = abs(actual_count - expected_count)
|
||||
if diff == 0:
|
||||
count_score = 1.0
|
||||
elif diff <= 2:
|
||||
count_score = 0.75
|
||||
elif diff <= 5:
|
||||
count_score = 0.35
|
||||
else:
|
||||
count_score = 0.0
|
||||
else:
|
||||
count_score = 0.4
|
||||
|
||||
candidate_tracks = list(getattr(album_result, 'tracks', []) or [])
|
||||
matched = 0
|
||||
expected_tracks = [
|
||||
(track_data, _norm_text(track_data.get('name', '')))
|
||||
for track_data in tracks_json
|
||||
if track_data.get('name')
|
||||
]
|
||||
for track_data, expected_title in expected_tracks:
|
||||
expected_number = _track_number_from_track(track_data)
|
||||
best = 0.0
|
||||
for candidate in candidate_tracks:
|
||||
cand_title = _norm_text(_track_title_from_candidate(candidate))
|
||||
title_sim = _similarity(expected_title, cand_title)
|
||||
cand_number = _track_number_from_candidate(candidate)
|
||||
if expected_number and cand_number and expected_number == cand_number:
|
||||
title_sim = min(1.0, title_sim + 0.12)
|
||||
best = max(best, title_sim)
|
||||
if best >= 0.72:
|
||||
matched += 1
|
||||
coverage_score = matched / max(1, len(expected_tracks))
|
||||
|
||||
year_score = 0.5
|
||||
folder_year = str(getattr(album_result, 'year', '') or '')
|
||||
if expected_year and folder_year:
|
||||
year_score = 1.0 if expected_year == folder_year else 0.2
|
||||
elif expected_year and expected_year in _norm_text(folder_text):
|
||||
year_score = 1.0
|
||||
|
||||
quality_count_score = min(1.0, filtered_track_count / max(1, expected_count or actual_count or 1))
|
||||
peer_score = _source_quality_score(album_result)
|
||||
penalty = _folder_variant_penalty(expected_album, folder_text)
|
||||
|
||||
score = (
|
||||
album_score * 0.24
|
||||
+ artist_score * 0.16
|
||||
+ count_score * 0.16
|
||||
+ coverage_score * 0.28
|
||||
+ year_score * 0.06
|
||||
+ quality_count_score * 0.06
|
||||
+ peer_score * 0.04
|
||||
- penalty
|
||||
)
|
||||
return max(0.0, min(score, 1.0))
|
||||
|
||||
|
||||
def _resolve_soulseek_client(download_orchestrator: Any) -> Any:
|
||||
if hasattr(download_orchestrator, 'client'):
|
||||
try:
|
||||
client = download_orchestrator.client('soulseek')
|
||||
if client:
|
||||
return client
|
||||
except Exception as exc:
|
||||
logger.debug("Soulseek client lookup through orchestrator failed: %s", exc)
|
||||
return getattr(download_orchestrator, 'soulseek', download_orchestrator)
|
||||
|
||||
|
||||
def _soulseek_album_preflight_enabled(config_manager: Any) -> bool:
|
||||
mode = config_manager.get('download_source.mode', 'hybrid')
|
||||
if mode == 'soulseek':
|
||||
return True
|
||||
if mode != 'hybrid':
|
||||
return False
|
||||
order = config_manager.get('download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'])
|
||||
if order:
|
||||
return order[0] == 'soulseek'
|
||||
primary = config_manager.get('download_source.hybrid_primary', '')
|
||||
return primary == 'soulseek'
|
||||
|
||||
|
||||
def _resolve_album_bundle_source(config_manager: Any) -> str:
|
||||
"""Return the album-bundle source for this batch.
|
||||
|
||||
In single-source mode, the active source may own the whole album if
|
||||
it supports album bundles. In hybrid mode, only the first source in
|
||||
the configured order may claim the whole album; later sources remain
|
||||
per-track fallback.
|
||||
"""
|
||||
mode = (config_manager.get('download_source.mode', 'soulseek') or 'soulseek').lower()
|
||||
if mode in _ALBUM_BUNDLE_SOURCES:
|
||||
return mode
|
||||
if mode != 'hybrid':
|
||||
return ''
|
||||
|
||||
order = config_manager.get('download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'])
|
||||
first = ''
|
||||
if order:
|
||||
first = str(order[0] or '').lower()
|
||||
else:
|
||||
first = str(config_manager.get('download_source.hybrid_primary', '') or '').lower()
|
||||
return first if first in _ALBUM_BUNDLE_SOURCES else ''
|
||||
|
||||
|
||||
@dataclass
|
||||
class MasterDeps:
|
||||
"""Bundle of cross-cutting deps the master worker needs."""
|
||||
|
|
@ -295,72 +64,7 @@ class MasterDeps:
|
|||
reset_wishlist_auto_processing: Callable[[], None]
|
||||
|
||||
|
||||
class _BatchStateAccessImpl:
|
||||
"""Concrete ``BatchStateAccess`` for the runtime ``download_batches``
|
||||
dict — wraps the lock + the existing-batch check so the album-
|
||||
bundle dispatcher stays decoupled from runtime_state."""
|
||||
|
||||
def update_fields(self, batch_id: str, fields: dict) -> None:
|
||||
with tasks_lock:
|
||||
row = download_batches.get(batch_id)
|
||||
if row is not None:
|
||||
row.update(fields)
|
||||
|
||||
def mark_failed(self, batch_id: str, error: str) -> None:
|
||||
with tasks_lock:
|
||||
row = download_batches.get(batch_id)
|
||||
if row is not None:
|
||||
row['phase'] = 'failed'
|
||||
row['error'] = error
|
||||
row['album_bundle_state'] = 'failed'
|
||||
|
||||
|
||||
# Task states that mean a batch still has work in flight. While ANY of a batch's
|
||||
# tasks is in one of these, a serialized album-pool worker keeps its slot.
|
||||
_NON_TERMINAL_TASK_STATUSES = ('pending', 'queued', 'searching', 'downloading', 'post_processing')
|
||||
|
||||
|
||||
def _wait_for_batch_drain(batch_id: str, poll_seconds: float = 1.5,
|
||||
max_wait_seconds: float = 3600.0) -> None:
|
||||
"""Block until every task in ``batch_id`` reaches a terminal state (the batch
|
||||
is fully drained), the batch is removed, shutdown is requested, or a safety
|
||||
cap elapses.
|
||||
|
||||
Used to make the dedicated album-bundle pool actually SERIALIZE albums: the
|
||||
worker holds its pool slot for the album's whole lifetime instead of
|
||||
returning the instant downloads are started. That stops every album from
|
||||
dumping its tracks into the shared download pool at once (Sokhi: "searching
|
||||
for way too many tracks at once"). It's a PASSIVE wait — the downloads are
|
||||
driven by the monitor + completion callbacks on other threads, so this never
|
||||
drives the work and can't deadlock; worst case the cap releases the slot and
|
||||
the downloads simply finish in the background."""
|
||||
from core.downloads import monitor as _monitor
|
||||
start = time.time()
|
||||
while True:
|
||||
if getattr(_monitor, 'IS_SHUTTING_DOWN', False):
|
||||
return
|
||||
with tasks_lock:
|
||||
batch = download_batches.get(batch_id)
|
||||
if not batch:
|
||||
return
|
||||
queue = list(batch.get('queue', ()) or ())
|
||||
still_working = any(
|
||||
download_tasks.get(t, {}).get('status') in _NON_TERMINAL_TASK_STATUSES
|
||||
for t in queue
|
||||
)
|
||||
if not still_working:
|
||||
return
|
||||
if time.time() - start > max_wait_seconds:
|
||||
logger.warning(
|
||||
"[Album Serialize] batch %s not drained after %.0fs — releasing the "
|
||||
"album-pool slot (its downloads continue in the background)",
|
||||
batch_id, max_wait_seconds)
|
||||
return
|
||||
time.sleep(poll_seconds)
|
||||
|
||||
|
||||
def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: MasterDeps,
|
||||
serialize: bool = False):
|
||||
def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: MasterDeps):
|
||||
"""
|
||||
A master worker that handles the entire missing tracks process:
|
||||
1. Runs the analysis.
|
||||
|
|
@ -375,64 +79,21 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
download_batches[batch_id]['analysis_processed'] = 0
|
||||
|
||||
from database.music_database import MusicDatabase
|
||||
from core.library import manual_library_match as _mlm
|
||||
db = MusicDatabase()
|
||||
active_server = deps.config_manager.get_active_media_server()
|
||||
analysis_results = []
|
||||
|
||||
# Get force download flag and album context from batch
|
||||
force_download_all = False
|
||||
ignore_manual_matches = False
|
||||
batch_album_context = None
|
||||
batch_artist_context = None
|
||||
batch_is_album = False
|
||||
batch_profile_id = 1
|
||||
batch_source = 'spotify'
|
||||
batch_playlist_folder_mode = False
|
||||
batch_playlist_name = 'Unknown Playlist'
|
||||
batch_playlist_id = playlist_id
|
||||
batch_source_playlist_ref = ''
|
||||
# Issue #797 — per-request "Skip AcoustID verification" toggle from
|
||||
# the album-download modal. When set, every track in this batch
|
||||
# bypasses the AcoustID quarantine gate (the user has chosen to
|
||||
# trust the metadata over fingerprint disagreement — useful for
|
||||
# non-English artists whose native-script metadata AcoustID can't
|
||||
# reconcile with the romanized request).
|
||||
batch_skip_acoustid = False
|
||||
with tasks_lock:
|
||||
if batch_id in download_batches:
|
||||
force_download_all = download_batches[batch_id].get('force_download_all', False)
|
||||
ignore_manual_matches = download_batches[batch_id].get('ignore_manual_matches', False)
|
||||
batch_is_album = download_batches[batch_id].get('is_album_download', False)
|
||||
batch_album_context = download_batches[batch_id].get('album_context')
|
||||
batch_artist_context = download_batches[batch_id].get('artist_context')
|
||||
batch_profile_id = download_batches[batch_id].get('profile_id', 1) or 1
|
||||
batch_source = download_batches[batch_id].get('batch_source', 'spotify') or 'spotify'
|
||||
batch_playlist_folder_mode = download_batches[batch_id].get('playlist_folder_mode', False)
|
||||
batch_playlist_name = download_batches[batch_id].get('playlist_name', 'Unknown Playlist')
|
||||
batch_playlist_id = download_batches[batch_id].get('playlist_id', playlist_id)
|
||||
batch_source_playlist_ref = (
|
||||
download_batches[batch_id].get('source_playlist_ref') or ''
|
||||
).strip()
|
||||
batch_skip_acoustid = bool(download_batches[batch_id].get('skip_acoustid', False))
|
||||
|
||||
from core.downloads.playlist_folder import (
|
||||
resolve_playlist_folder_mode_for_batch,
|
||||
track_exists_in_playlist_folder_from_track_data,
|
||||
)
|
||||
effective_playlist_folder_mode, effective_playlist_name = resolve_playlist_folder_mode_for_batch(
|
||||
db,
|
||||
playlist_id=str(batch_playlist_id),
|
||||
playlist_name=batch_playlist_name,
|
||||
batch_playlist_folder_mode=batch_playlist_folder_mode,
|
||||
profile_id=batch_profile_id,
|
||||
source=batch_source,
|
||||
)
|
||||
if effective_playlist_folder_mode and not batch_playlist_folder_mode:
|
||||
with tasks_lock:
|
||||
if batch_id in download_batches:
|
||||
download_batches[batch_id]['playlist_folder_mode'] = True
|
||||
download_batches[batch_id]['playlist_name'] = effective_playlist_name
|
||||
|
||||
if force_download_all:
|
||||
logger.warning(f"[Force Download] Force download mode enabled for batch {batch_id} - treating all tracks as missing")
|
||||
|
|
@ -505,55 +166,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
track_name = track_data.get('name', '')
|
||||
artists = track_data.get('artists', [])
|
||||
found, confidence = False, 0.0
|
||||
# Additive payload: the owned library track (DatabaseTrack) when this
|
||||
# item is found in the library, so downstream (playlist materialization)
|
||||
# knows WHERE the real file is without re-matching. None when not owned.
|
||||
matched_track = None
|
||||
|
||||
# Manual library matches are authoritative unless the user explicitly
|
||||
# requested a force re-download from the normal download modal.
|
||||
_stid = track_data.get('spotify_track_id') or track_data.get('source_track_id') or track_data.get('id', '')
|
||||
_manual_match = (
|
||||
_mlm.get_match_for_track(db, batch_profile_id, track_data, default_source=batch_source)
|
||||
if (not ignore_manual_matches and _stid) else None
|
||||
)
|
||||
if _manual_match:
|
||||
logger.info(f"[Manual Match] '{track_name}' already matched in library — skipping download")
|
||||
try:
|
||||
deps.check_and_remove_track_from_wishlist_by_metadata(track_data)
|
||||
except Exception as _wl_err:
|
||||
logger.debug(f"[Manual Match] Wishlist removal attempt failed: {_wl_err}")
|
||||
analysis_results.append({
|
||||
'track_index': track_index,
|
||||
'track': track_data,
|
||||
'found': True,
|
||||
'confidence': 1.0,
|
||||
'match_reason': 'manual_library_match',
|
||||
'matched_file_path': _manual_match.get('library_file_path'),
|
||||
'matched_track_id': _manual_match.get('library_track_id'),
|
||||
})
|
||||
continue
|
||||
|
||||
if effective_playlist_folder_mode and not force_download_all:
|
||||
if track_exists_in_playlist_folder_from_track_data(
|
||||
effective_playlist_name,
|
||||
track_data,
|
||||
):
|
||||
logger.info(
|
||||
f"[Playlist Folder] '{track_name}' already on disk in playlist folder — skipping download"
|
||||
)
|
||||
try:
|
||||
deps.check_and_remove_track_from_wishlist_by_metadata(track_data)
|
||||
except Exception as _wl_err:
|
||||
logger.debug(f"[Playlist Folder] Wishlist removal attempt failed: {_wl_err}")
|
||||
analysis_results.append({
|
||||
'track_index': track_index,
|
||||
'track': track_data,
|
||||
'found': True,
|
||||
'confidence': 1.0,
|
||||
'match_reason': 'playlist_folder_file',
|
||||
})
|
||||
continue
|
||||
|
||||
# Skip database check if force download is enabled
|
||||
if force_download_all:
|
||||
|
|
@ -562,42 +174,18 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
elif album_tracks_map:
|
||||
# Album-scoped matching: check against known album tracks first
|
||||
track_name_lower = track_name.lower().strip()
|
||||
# Issue #589 — strip suffixes that just repeat the album
|
||||
# context (e.g. "Shy Away (MTV Unplugged Live)" on a
|
||||
# "MTV Unplugged" album → "Shy Away") so album-owned
|
||||
# tracks don't false-miss when the local DB stored the
|
||||
# base title. Only fires inside the album-confirmed
|
||||
# scope; global matching elsewhere is unchanged.
|
||||
from core.matching.album_context_title import strip_redundant_album_suffix
|
||||
_album_name_for_strip = (batch_album_context or {}).get('name', '')
|
||||
_normalized_source_title = strip_redundant_album_suffix(
|
||||
track_name, _album_name_for_strip
|
||||
).lower().strip()
|
||||
# Direct title match (try both raw and normalized)
|
||||
# Direct title match
|
||||
if track_name_lower in album_tracks_map:
|
||||
found, confidence = True, 1.0
|
||||
matched_track = album_tracks_map[track_name_lower]
|
||||
elif _normalized_source_title and _normalized_source_title in album_tracks_map:
|
||||
found, confidence = True, 1.0
|
||||
matched_track = album_tracks_map[_normalized_source_title]
|
||||
else:
|
||||
# Fuzzy match against album tracks using string similarity.
|
||||
# Compare BOTH the raw and normalized source titles —
|
||||
# whichever scores higher wins. Preserves strict
|
||||
# matching when the album doesn't imply version
|
||||
# context (helper returns the input unchanged).
|
||||
# Fuzzy match against album tracks using string similarity
|
||||
best_sim = 0.0
|
||||
best_track = None
|
||||
for db_title_lower, _db_track in album_tracks_map.items():
|
||||
sim_raw = db._string_similarity(track_name_lower, db_title_lower)
|
||||
sim_norm = db._string_similarity(_normalized_source_title, db_title_lower) if _normalized_source_title else 0.0
|
||||
sim = max(sim_raw, sim_norm)
|
||||
sim = db._string_similarity(track_name_lower, db_title_lower)
|
||||
if sim > best_sim:
|
||||
best_sim = sim
|
||||
best_track = _db_track
|
||||
if best_sim >= 0.7:
|
||||
found, confidence = True, best_sim
|
||||
matched_track = best_track
|
||||
else:
|
||||
# Fall back to global per-track search for this track
|
||||
# When allow_duplicates is on for album downloads, skip global
|
||||
|
|
@ -618,7 +206,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
)
|
||||
if db_track and track_confidence >= 0.7:
|
||||
found, confidence = True, track_confidence
|
||||
matched_track = db_track
|
||||
break
|
||||
elif allow_duplicates and batch_is_album:
|
||||
# Allow duplicates + album download + album not in DB yet → treat all as missing
|
||||
|
|
@ -638,15 +225,10 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
)
|
||||
if db_track and track_confidence >= 0.7:
|
||||
found, confidence = True, track_confidence
|
||||
matched_track = db_track
|
||||
break
|
||||
|
||||
analysis_results.append({
|
||||
'track_index': track_index, 'track': track_data, 'found': found, 'confidence': confidence,
|
||||
# Additive: real on-disk location of the owned track (None when not
|
||||
# owned), so playlist materialization links the right file.
|
||||
'matched_file_path': getattr(matched_track, 'file_path', None),
|
||||
'matched_track_id': getattr(matched_track, 'id', None),
|
||||
'track_index': track_index, 'track': track_data, 'found': found, 'confidence': confidence
|
||||
})
|
||||
|
||||
# WISHLIST REMOVAL: If track is found in database, check if it should be removed from wishlist
|
||||
|
|
@ -672,34 +254,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
if skipped > 0:
|
||||
logger.warning(f"[Content Filter] Filtered out {skipped} explicit track(s) from download queue")
|
||||
|
||||
# Blocklist (Phase 2a): drop banned artists/albums/tracks before queueing,
|
||||
# so a blocked item can't slip in via playlist sync / album download /
|
||||
# discography. Same ID-cascade brain as the wishlist guard (Phase 1) —
|
||||
# the only other auto-acquisition path. Skipped when the user confirmed
|
||||
# "download anyway" at the modal (Phase 2b override).
|
||||
_ignore_blocklist = False
|
||||
with tasks_lock:
|
||||
if batch_id in download_batches:
|
||||
_ignore_blocklist = download_batches[batch_id].get('ignore_blocklist', False)
|
||||
if not _ignore_blocklist:
|
||||
try:
|
||||
_bl_before = len(missing_tracks)
|
||||
_bl_kept = []
|
||||
for res in missing_tracks:
|
||||
reason = db.blocklist_reason_for_track(
|
||||
batch_profile_id, res.get('track', {}), source=batch_source)
|
||||
if reason:
|
||||
logger.info("[Blocklist] Skipping %s '%s' from download queue (%s blocked)",
|
||||
reason[0], res.get('track', {}).get('name', '?'), reason[0])
|
||||
else:
|
||||
_bl_kept.append(res)
|
||||
if len(_bl_kept) != _bl_before:
|
||||
logger.info("[Blocklist] Filtered out %d blocklisted track(s) from download queue",
|
||||
_bl_before - len(_bl_kept))
|
||||
missing_tracks = _bl_kept
|
||||
except Exception as _bl_err:
|
||||
logger.debug("blocklist queue filter skipped: %s", _bl_err)
|
||||
|
||||
with tasks_lock:
|
||||
if batch_id in download_batches:
|
||||
download_batches[batch_id]['analysis_results'] = analysis_results
|
||||
|
|
@ -785,41 +339,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
logger.warning("[Auto-Wishlist] No missing tracks found - calling auto-completion handler to toggle cycle and reschedule")
|
||||
deps.missing_download_executor.submit(deps.process_failed_tracks_to_wishlist_exact_with_auto_completion, batch_id)
|
||||
|
||||
# Organize-by-playlist with NOTHING to download (every track already
|
||||
# owned): the batch never enters the download/lifecycle path, so build
|
||||
# the playlist folder here from the owned files the analysis matched.
|
||||
# Gated + non-fatal; runs once after analysis, not in the per-track loop.
|
||||
if effective_playlist_folder_mode:
|
||||
try:
|
||||
from core.playlists.materialize_service import reconcile_batch_playlists
|
||||
from database.music_database import MusicDatabase as _MDB
|
||||
_batch = download_batches.get(batch_id)
|
||||
if _batch is not None:
|
||||
# We KNOW the intent is organize-by-playlist here (the gate
|
||||
# above). The line-431 sync only writes the dict field when
|
||||
# effective and NOT batch_playlist_folder_mode, so when the
|
||||
# toggle itself drove it the dict field can still be falsy —
|
||||
# which makes reconcile build no batch ref. Make the dict
|
||||
# authoritative so reconcile sees the batch's own playlist.
|
||||
_batch['playlist_folder_mode'] = True
|
||||
if effective_playlist_name:
|
||||
_batch['playlist_name'] = effective_playlist_name
|
||||
_results = reconcile_batch_playlists(_MDB(), _batch, download_tasks, deps.config_manager)
|
||||
if not _results:
|
||||
logger.info(
|
||||
f"[Playlist Folder] All-owned: nothing rebuilt for "
|
||||
f"ref={_batch.get('source_playlist_ref') or _batch.get('playlist_id')} "
|
||||
f"source={_batch.get('batch_source')}"
|
||||
)
|
||||
for _pl_name, _mat in _results:
|
||||
logger.info(
|
||||
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}' (all owned): "
|
||||
f"{_mat.linked} linked, {_mat.copied} copied, {_mat.removed_stale} stale removed"
|
||||
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
|
||||
)
|
||||
except Exception as _mat_err:
|
||||
logger.error(f"[Playlist Folder] All-owned materialize failed (non-fatal): {_mat_err}")
|
||||
|
||||
return
|
||||
|
||||
logger.warning(f" transitioning batch {batch_id} to download phase with {len(missing_tracks)} tracks.")
|
||||
|
|
@ -831,36 +350,20 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
batch_album_context = batch.get('album_context')
|
||||
batch_artist_context = batch.get('artist_context')
|
||||
batch_is_album = batch.get('is_album_download', False)
|
||||
batch_private_album_bundle = bool(batch.get('album_bundle_private_staging'))
|
||||
batch_playlist_folder_mode = batch.get('playlist_folder_mode', False)
|
||||
batch_playlist_name = batch.get('playlist_name', 'Unknown Playlist')
|
||||
|
||||
# Album-bundle sources download a whole release into private staging,
|
||||
# then the normal per-track workers claim those staged files. Run this
|
||||
# only after analysis has found missing tracks; otherwise an already
|
||||
# owned album would still trigger a release download.
|
||||
_bundle_state = _BatchStateAccessImpl()
|
||||
_album_bundle_source = _resolve_album_bundle_source(deps.config_manager)
|
||||
if _album_bundle_source and _album_bundle_source != 'soulseek':
|
||||
if _album_bundle_dispatch.try_dispatch(
|
||||
batch_id=batch_id,
|
||||
is_album=batch_is_album,
|
||||
album_context=batch_album_context,
|
||||
artist_context=batch_artist_context,
|
||||
config_get=deps.config_manager.get,
|
||||
plugin_resolver=deps.download_orchestrator.client,
|
||||
state=_bundle_state,
|
||||
source_override=_album_bundle_source,
|
||||
):
|
||||
return
|
||||
|
||||
# === ALBUM PRE-FLIGHT: Search for complete album folder before track-by-track ===
|
||||
# Only run pre-flight when Soulseek is the download source (or hybrid with soulseek)
|
||||
preflight_source = None
|
||||
preflight_tracks = None
|
||||
soulseek_is_source = _soulseek_album_preflight_enabled(deps.config_manager)
|
||||
if (batch_is_album and batch_album_context and batch_artist_context
|
||||
and soulseek_is_source and not batch_private_album_bundle):
|
||||
dl_source_mode = deps.config_manager.get('download_source.mode', 'hybrid')
|
||||
_dl_hybrid_order = deps.config_manager.get('download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'])
|
||||
_dl_hybrid_first = _dl_hybrid_order[0] if _dl_hybrid_order else deps.config_manager.get('download_source.hybrid_primary', 'hifi')
|
||||
soulseek_is_source = dl_source_mode == 'soulseek' or (
|
||||
dl_source_mode == 'hybrid' and _dl_hybrid_first == 'soulseek'
|
||||
)
|
||||
if batch_is_album and batch_album_context and batch_artist_context and soulseek_is_source:
|
||||
artist_name = batch_artist_context.get('name', '')
|
||||
album_name = batch_album_context.get('name', '')
|
||||
if artist_name and album_name:
|
||||
|
|
@ -869,7 +372,7 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
_sr.info(f"[Album Pre-flight] Searching for '{artist_name} {album_name}'")
|
||||
logger.info(f"[Album Pre-flight] Searching Soulseek for complete album: '{artist_name} - {album_name}'")
|
||||
|
||||
slsk = _resolve_soulseek_client(deps.download_orchestrator)
|
||||
slsk = deps.download_orchestrator.client('soulseek') if hasattr(deps.download_orchestrator, 'client') else deps.download_orchestrator
|
||||
|
||||
# Try multiple query variations (banned keywords in artist/album name can return 0 results)
|
||||
album_queries = [f"{artist_name} {album_name}"]
|
||||
|
|
@ -883,57 +386,29 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
|
||||
album_results = []
|
||||
track_results = []
|
||||
album_results_by_source = {}
|
||||
for aq in album_queries:
|
||||
_sr.info(f"[Album Pre-flight] Trying query: '{aq}'")
|
||||
track_results, album_results = deps.run_async(slsk.search(aq, timeout=30))
|
||||
if album_results:
|
||||
_sr.info(f"[Album Pre-flight] Found {len(album_results)} album results with query: '{aq}'")
|
||||
for ar in album_results:
|
||||
key = (getattr(ar, 'username', ''), getattr(ar, 'album_path', ''))
|
||||
if key[0] and key[1] and key not in album_results_by_source:
|
||||
album_results_by_source[key] = ar
|
||||
else:
|
||||
_sr.info(f"[Album Pre-flight] No album results for query: '{aq}'")
|
||||
break
|
||||
_sr.info(f"[Album Pre-flight] No album results for query: '{aq}'")
|
||||
|
||||
album_results = list(album_results_by_source.values())
|
||||
if album_results:
|
||||
# Score complete folders as releases before falling back to per-track search.
|
||||
scored_albums = []
|
||||
# Filter by quality preference
|
||||
quality_filtered = []
|
||||
for ar in album_results:
|
||||
filtered_tracks = slsk.filter_results_by_quality_preference(ar.tracks)
|
||||
if filtered_tracks:
|
||||
folder_score = _score_album_folder(
|
||||
ar,
|
||||
batch_album_context,
|
||||
batch_artist_context,
|
||||
tracks_json,
|
||||
len(filtered_tracks),
|
||||
)
|
||||
scored_albums.append((ar, len(filtered_tracks), folder_score))
|
||||
_sr.info(
|
||||
f"[Album Pre-flight] Candidate {ar.username}:{ar.album_path} "
|
||||
f"score={folder_score:.3f}, tracks={ar.track_count}, "
|
||||
f"quality_tracks={len(filtered_tracks)}"
|
||||
)
|
||||
quality_filtered.append((ar, len(filtered_tracks)))
|
||||
|
||||
best_album = None
|
||||
best_score = 0.0
|
||||
if scored_albums:
|
||||
scored_albums.sort(key=lambda x: (x[2], x[1], x[0].quality_score), reverse=True)
|
||||
best_album, _best_filtered_count, best_score = scored_albums[0]
|
||||
if best_score < _ALBUM_PREFLIGHT_MIN_SCORE:
|
||||
_sr.info(
|
||||
f"[Album Pre-flight] Best folder score {best_score:.3f} below "
|
||||
f"threshold {_ALBUM_PREFLIGHT_MIN_SCORE:.2f}; falling back"
|
||||
)
|
||||
logger.warning("[Album Pre-flight] No Soulseek folder passed album-level validation")
|
||||
best_album = None
|
||||
|
||||
if best_album:
|
||||
if quality_filtered:
|
||||
# Sort by track count (most complete album first), then quality score
|
||||
quality_filtered.sort(key=lambda x: (x[1], x[0].quality_score), reverse=True)
|
||||
best_album = quality_filtered[0][0]
|
||||
|
||||
_sr.info(f"[Album Pre-flight] Best album result: {best_album.username}:{best_album.album_path} "
|
||||
f"({best_album.track_count} tracks, quality={best_album.dominant_quality}, score={best_score:.3f})")
|
||||
f"({best_album.track_count} tracks, quality={best_album.dominant_quality})")
|
||||
logger.info(f"[Album Pre-flight] Found album folder: {best_album.username} — "
|
||||
f"{best_album.track_count} tracks ({best_album.dominant_quality})")
|
||||
|
||||
|
|
@ -962,7 +437,7 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
}
|
||||
preflight_tracks = best_album.tracks
|
||||
logger.info(f"[Album Pre-flight] Using {len(best_album.tracks)} tracks from search results (browse unavailable)")
|
||||
elif not scored_albums:
|
||||
else:
|
||||
_sr.info("[Album Pre-flight] No album results passed quality filter")
|
||||
logger.warning("[Album Pre-flight] No album results matched quality preferences")
|
||||
else:
|
||||
|
|
@ -973,44 +448,13 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
logger.error(f"[Album Pre-flight] Search failed (non-fatal, falling back to track-by-track): {preflight_err}")
|
||||
deps.source_reuse_logger.info(f"[Album Pre-flight] Exception: {preflight_err}")
|
||||
|
||||
# Soulseek album bundles run after analysis so an already-owned
|
||||
# album does not get downloaded just because the source supports a
|
||||
# whole-folder flow. When preflight selected a folder, pass that
|
||||
# exact source into the bundle downloader so we keep the richer
|
||||
# tracklist-aware scoring instead of doing a weaker second pick.
|
||||
_bundle_state = _BatchStateAccessImpl()
|
||||
_album_bundle_source = _resolve_album_bundle_source(deps.config_manager)
|
||||
if _album_bundle_source == 'soulseek':
|
||||
if _album_bundle_dispatch.try_dispatch(
|
||||
batch_id=batch_id,
|
||||
is_album=batch_is_album,
|
||||
album_context=batch_album_context,
|
||||
artist_context=batch_artist_context,
|
||||
config_get=deps.config_manager.get,
|
||||
plugin_resolver=deps.download_orchestrator.client,
|
||||
state=_bundle_state,
|
||||
source_override=_album_bundle_source,
|
||||
plugin_kwargs={
|
||||
'preferred_source': preflight_source,
|
||||
'preferred_tracks': preflight_tracks,
|
||||
} if preflight_source and preflight_tracks else None,
|
||||
):
|
||||
return
|
||||
|
||||
with tasks_lock:
|
||||
if batch_id not in download_batches: return
|
||||
|
||||
download_batches[batch_id]['phase'] = 'downloading'
|
||||
|
||||
# Store album pre-flight results on batch for source reuse
|
||||
# unless the Soulseek album-bundle path already staged a private
|
||||
# release. Task workers check source reuse before staging match, so
|
||||
# preloading here would make the staged happy path re-download.
|
||||
if (
|
||||
preflight_source
|
||||
and preflight_tracks
|
||||
and not download_batches[batch_id].get('album_bundle_private_staging')
|
||||
):
|
||||
if preflight_source and preflight_tracks:
|
||||
download_batches[batch_id]['last_good_source'] = preflight_source
|
||||
download_batches[batch_id]['source_folder_tracks'] = preflight_tracks
|
||||
download_batches[batch_id]['failed_sources'] = set()
|
||||
|
|
@ -1020,7 +464,7 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
# Use ALL tracks (tracks_json), not just missing ones, to correctly detect multi-disc
|
||||
# even when only one disc has missing tracks
|
||||
if batch_is_album and batch_album_context:
|
||||
total_discs = max((t.get('disc_number') or 1 for t in tracks_json), default=1)
|
||||
total_discs = max((t.get('disc_number', 1) for t in tracks_json), default=1)
|
||||
batch_album_context['total_discs'] = total_discs
|
||||
if total_discs > 1:
|
||||
logger.info(f"[Multi-Disc] Detected {total_discs} discs for album '{batch_album_context.get('name')}'")
|
||||
|
|
@ -1029,7 +473,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
# Wishlist tracks aren't batch_is_album but each track has disc_number in spotify_data
|
||||
wishlist_album_disc_counts = {}
|
||||
wishlist_album_artist_map = {} # album_id -> resolved artist context (consistent per album)
|
||||
wishlist_album_context_map = {} # album_id -> richest shared album context
|
||||
if playlist_id == 'wishlist':
|
||||
import json as _json
|
||||
# First pass: collect disc_number and resolve ONE artist per album
|
||||
|
|
@ -1045,15 +488,11 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
# Fallback album key: use album name when ID is missing (e.g. mirrored playlist tracks)
|
||||
if not album_id and isinstance(album_val, dict) and album_val.get('name'):
|
||||
album_id = f"_name_{album_val['name'].lower().strip()}"
|
||||
disc_num = sp_data.get('disc_number') or t.get('disc_number') or 1
|
||||
disc_num = sp_data.get('disc_number', t.get('disc_number', 1))
|
||||
if album_id:
|
||||
wishlist_album_disc_counts[album_id] = max(
|
||||
wishlist_album_disc_counts.get(album_id, 1), disc_num
|
||||
)
|
||||
if isinstance(album_val, dict):
|
||||
existing_album_ctx = wishlist_album_context_map.get(album_id, {})
|
||||
if _album_context_richness(album_val) > _album_context_richness(existing_album_ctx):
|
||||
wishlist_album_context_map[album_id] = dict(album_val)
|
||||
# Resolve album-level artist once per album (first track wins)
|
||||
if album_id not in wishlist_album_artist_map:
|
||||
_wl_source = t.get('source_info') or {}
|
||||
|
|
@ -1144,20 +583,17 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
_fb_name = track_info.get('artist', '')
|
||||
artist_ctx = {'name': _fb_name or 'Unknown Artist'}
|
||||
|
||||
# Construct a shared album context from the richest track in
|
||||
# this album group so release_date/year and artwork do not
|
||||
# vary per track and split folders.
|
||||
# Construct minimal album context
|
||||
# Ensure images are preserved (important for artwork)
|
||||
album_id = s_album.get('id', 'wishlist_album')
|
||||
shared_album = wishlist_album_context_map.get(album_id_for_lookup, s_album)
|
||||
album_ctx = {
|
||||
'id': album_id,
|
||||
'name': shared_album.get('name') or s_album.get('name'),
|
||||
'release_date': shared_album.get('release_date', ''),
|
||||
'total_tracks': shared_album.get('total_tracks') or s_album.get('total_tracks', 1),
|
||||
'total_discs': wishlist_album_disc_counts.get(album_id_for_lookup, 1),
|
||||
'album_type': shared_album.get('album_type') or s_album.get('album_type', 'album'),
|
||||
'images': shared_album.get('images') or s_album.get('images', []),
|
||||
'artists': shared_album.get('artists') or s_album.get('artists', []),
|
||||
'name': s_album.get('name'),
|
||||
'release_date': s_album.get('release_date', ''),
|
||||
'total_tracks': s_album.get('total_tracks', 1),
|
||||
'total_discs': wishlist_album_disc_counts.get(album_id, 1),
|
||||
'album_type': s_album.get('album_type', 'album'),
|
||||
'images': s_album.get('images', []) # Pass images array directly
|
||||
}
|
||||
|
||||
track_info['_explicit_album_context'] = album_ctx
|
||||
|
|
@ -1166,81 +602,13 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
logger.info(f"[Wishlist] Added album context for: '{track_info.get('name')}' -> '{album_ctx['name']}'")
|
||||
|
||||
|
||||
# Issue #797 — propagate the batch-level "skip AcoustID"
|
||||
# toggle onto each track so the per-track download context
|
||||
# (built in core/downloads/candidates.py) can set the
|
||||
# AcoustID quarantine bypass. Mirrors the _playlist_folder_mode
|
||||
# threading pattern below.
|
||||
if batch_skip_acoustid:
|
||||
track_info['_skip_acoustid'] = True
|
||||
|
||||
# Add playlist folder mode flag for sync page playlists and wishlist
|
||||
# tracks tied to a mirrored playlist with organize_by_playlist enabled.
|
||||
task_pl_folder_mode = batch_playlist_folder_mode
|
||||
task_pl_name = batch_playlist_name
|
||||
if not task_pl_folder_mode and playlist_id == 'wishlist':
|
||||
wl_source = track_info.get('source_info') or {}
|
||||
if isinstance(wl_source, str):
|
||||
try:
|
||||
wl_source = json.loads(wl_source)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
wl_source = {}
|
||||
wl_pl_ref = wl_source.get('playlist_id')
|
||||
wl_pl_name = wl_source.get('playlist_name')
|
||||
wl_pl_source = wl_source.get('source') or 'spotify'
|
||||
if wl_pl_ref and hasattr(db, 'resolve_mirrored_playlist'):
|
||||
wl_mirrored = db.resolve_mirrored_playlist(
|
||||
wl_pl_ref,
|
||||
profile_id=batch_profile_id,
|
||||
default_source=wl_pl_source,
|
||||
)
|
||||
if wl_mirrored and wl_mirrored.get('organize_by_playlist'):
|
||||
task_pl_folder_mode = True
|
||||
task_pl_name = wl_pl_name or wl_mirrored.get('name') or batch_playlist_name
|
||||
if task_pl_folder_mode:
|
||||
# Organize-by-playlist now imports each track NORMALLY into the
|
||||
# Artist/Album library (i.e. exactly what a normal download does)
|
||||
# — the playlist folder is built as links/copies AFTER the batch
|
||||
# from the real library files. So we deliberately DON'T set
|
||||
# `_playlist_folder_mode` (which routed the real file into a flat
|
||||
# Music/<playlist>/ dump). We keep `_playlist_name` + source_info
|
||||
# — they're download provenance (core/downloads/origin.py).
|
||||
track_info['_playlist_name'] = task_pl_name
|
||||
if batch_source_playlist_ref:
|
||||
track_info['source_info'] = {
|
||||
'playlist_id': batch_source_playlist_ref,
|
||||
'playlist_name': task_pl_name,
|
||||
'source': batch_source,
|
||||
}
|
||||
logger.info(
|
||||
f"[Task Creation] Organize-by-playlist (normal import + "
|
||||
f"materialize after batch): {track_info.get('name')} → {task_pl_name}"
|
||||
)
|
||||
# Add playlist folder mode flag for sync page playlists
|
||||
if batch_playlist_folder_mode:
|
||||
track_info['_playlist_folder_mode'] = True
|
||||
track_info['_playlist_name'] = batch_playlist_name
|
||||
logger.info(f"[Task Creation] Added playlist folder mode for: {track_info.get('name')} → {batch_playlist_name}")
|
||||
else:
|
||||
logger.debug(
|
||||
f"[Debug] Task Creation - playlist folder mode NOT enabled for: "
|
||||
f"{track_info.get('name')}"
|
||||
)
|
||||
|
||||
# Download-origin provenance: stamp what TRIGGERED this download
|
||||
# so the history chokepoint can record it (origin-history modal).
|
||||
# Wishlist rows already ride their source_info in track_info
|
||||
# (watchlist_artist_name / playlist_name — the deriver reads
|
||||
# those directly); this stamp covers DIRECT playlist batches,
|
||||
# where the playlist context otherwise only survives in
|
||||
# folder mode.
|
||||
if '_dl_origin' not in track_info and batch_source_playlist_ref and batch_playlist_name:
|
||||
_prov_si = track_info.get('source_info') or {}
|
||||
if isinstance(_prov_si, str):
|
||||
try:
|
||||
_prov_si = json.loads(_prov_si)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
_prov_si = {}
|
||||
if not _prov_si.get('watchlist_artist_name'):
|
||||
track_info['_dl_origin'] = 'playlist'
|
||||
track_info['_dl_origin_context'] = (
|
||||
_prov_si.get('playlist_name') or batch_playlist_name
|
||||
)
|
||||
logger.debug(f"[Debug] Task Creation - playlist folder mode NOT enabled for: {track_info.get('name')}")
|
||||
|
||||
download_tasks[task_id] = {
|
||||
'status': 'pending', 'track_info': track_info,
|
||||
|
|
@ -1255,16 +623,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
|
|||
deps.download_monitor.start_monitoring(batch_id)
|
||||
deps.start_next_batch_of_downloads(batch_id)
|
||||
|
||||
# Album-bundle batches run on the dedicated album pool and pass
|
||||
# serialize=True: hold this pool slot until the album finishes so only a
|
||||
# few albums are ever in flight at once, instead of every album batch
|
||||
# immediately starting and flooding the shared download pool with
|
||||
# 'searching' tracks (#740 / Sokhi). The residual + playlist + manual
|
||||
# paths run on the shared download pool and DON'T serialize (blocking
|
||||
# there would steal an actual download worker).
|
||||
if serialize:
|
||||
_wait_for_batch_drain(batch_id)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Master worker for batch {batch_id} failed: {e}")
|
||||
import traceback
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue