danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Compare commits

base: danvics:main
Branches Tags
danvics:main
danvics:dev
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65
..
compare: danvics:2.5.5
Branches Tags
danvics:dev
danvics:main
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65

No commits in common. "main" and "2.5.5" have entirely different histories.
main ... 2.5.5

847 changed files with 19436 additions and 154527 deletions
Show stats Expand all files Collapse all files

4
.github/workflows/docker-publish.yml vendored
View file

@ -9,9 +9,9 @@ on:
workflow_dispatch:
inputs:
version_tag:
description: 'Version tag (e.g. 2.8.2)'
description: 'Version tag (e.g. 2.5.4)'
required: true
default: '2.8.2'
default: '2.5.4'
jobs:
build-and-push:

11
.gitignore vendored
View file

@ -12,14 +12,11 @@ __pycache__/
# User-specific files (auto-created by the app if missing)
config/config.json
config/youtube_cookies.txt
# All app databases are live user data — never commit (music_library, video_library, …)
database/*.db
database/*.db-shm
database/*.db-wal
database/*.db.backup_*
database/music_library.db
database/music_library.db-shm
database/music_library.db-wal
database/music_library.db.backup_*
database/api_call_history.json
storage/image_cache/
logs/*.log
logs/*.log.*

74
DISCOVER_BEST_IN_CLASS_PLAN.md
View file

@ -1,74 +0,0 @@
# discover page — best in class plan (#913 + full generator audit)
morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
## what i shipped tonight (done, tested, safe)
### 1. listening recommendations (#913) — went from BROKEN to best-in-class
the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
the fix (all in the pure, tested core + thin scan wiring):
- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
### 2. its own row on the discover page
new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
### 3. Fresh Tape "only 5-10 tracks" — fixed
root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
---
## best-in-class roadmap for listening recs (next phases — your call)
these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
| phase | what | value | risk | notes |
|---|---|---|---|---|
| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
---
## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
### curated (built during the scan, then hydrated)
- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
### discovery-pool generators (live queries)
- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
### cross-cutting
- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.

40
Dockerfile
View file

@ -29,16 +29,6 @@ COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# yt-dlp must track YouTube faster than its stable channel ships — stable can
# lag months behind a breaking YouTube change while extraction is broken
# ("Requested format is not available"). Build images with the NIGHTLY channel.
# COMMIT_SHA is referenced in the RUN so CI's layer cache (cache-from: gha)
# busts on every new commit — otherwise this layer could pin a stale "nightly"
# for months, silently defeating its purpose.
ARG COMMIT_SHA=""
RUN echo "yt-dlp nightly for build ${COMMIT_SHA}" && \
pip install --no-cache-dir -U --pre "yt-dlp[default]"
# Stage 2: Runtime — only runtime dependencies, no build tools
FROM python:3.11-slim
@ -54,25 +44,14 @@ ENV PATH="/opt/venv/bin:$PATH"
# Set working directory
WORKDIR /app
# Install runtime-only system dependencies (no gcc/build tools).
# unzip is needed by the Deno installer below.
# Install runtime-only system dependencies (no gcc/build tools)
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
gosu \
ffmpeg \
libchromaprint-tools \
unzip \
&& rm -rf /var/lib/apt/lists/*
# Deno — JavaScript runtime for yt-dlp. YouTube gates its downloadable formats
# behind JS challenges (nsig); without a JS runtime, yt-dlp's extraction is
# deprecated and streams / music-video downloads fail with "Requested format
# is not available". Deno is yt-dlp's default-enabled runtime; the official
# installer auto-detects amd64/arm64. `deno --version` fails the build early
# if the install ever breaks.
RUN curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh && \
deno --version
# Create non-root user for security
RUN useradd --create-home --shell /bin/bash --uid 1000 soulsync
@ -94,21 +73,8 @@ COPY --chown=soulsync:soulsync --from=webui-builder /app/webui/static/dist /app/
# pre-baked directory would crash the container into a restart loop on
# rootless Docker/Podman where in-container "root" can't write to /app.
# Pre-baking the dir here makes the entrypoint mkdir a guaranteed no-op.
# NOTE: /app/Stream is the transient single-file streaming cache used by
# the basic-search "Play" flow (cleared per use, never persistent). It's
# created lazily by `core/streaming/prepare.py` via `os.makedirs`, which
# fails silently on rootless Docker where the soulsync UID can't write
# to /app — playback then errors out with no obvious cause. Pre-baking
# at build time (when the layer is owned by root) avoids that path.
# NOTE: /app/storage is the PRIVATE album-bundle staging area for the
# torrent / usenet whole-release flow (download_source.album_bundle_staging_path
# defaults to 'storage/album_bundle_staging'). Like /app/Stream it's created
# lazily at runtime via mkdir(parents=True); without pre-baking it owned by
# soulsync, the album-bundle copy fails with "[Errno 13] Permission denied:
# 'storage'" because /app itself is root-owned and the soulsync UID can't
# create a top-level dir there.
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts && \
chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/MusicVideos /app/scripts && \
chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/MusicVideos /app/scripts
# Create defaults directory and copy template files
# These will be used by entrypoint.sh to initialize empty volumes

44
README.md
View file

@ -105,21 +105,6 @@ SoulSync bridges streaming services to your music library with automated discove
- Catches wrong versions (live, remix, cover) even from streaming API sources
- Fail-open design: verification errors never block downloads
#### AcoustID API key
AcoustID verification is opt-in. To enable it, request a free API key
at <https://acoustid.org/new-application> and paste it into
Settings → AcoustID. Without a key, downloads still complete but the
verification step is skipped silently.
If a track was previously tagged by AcoustID but the retag action in
the AcoustID Scanner no longer changes anything, see issue #704 — the
most common cause is that the file already carries a
`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
and therefore never overwrites. Removing the cached
`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
restores the retag.
### Metadata & Enrichment
**10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz
@ -294,39 +279,13 @@ The template points at `boulderbadgedad/soulsync:latest` (stable) by default. To
git clone https://github.com/Nezreka/SoulSync
cd SoulSync
python -m pip install -r requirements.txt
# Build the React WebUI bundle used by the Python server.
# Docker does this automatically; Python installs must do it manually.
cd webui
npm ci
npm run build
cd ..
gunicorn -c gunicorn.conf.py wsgi:application
# Open http://localhost:8008
```
When updating a Python/no-Docker install with `git pull`, rebuild the WebUI before restarting SoulSync:
```bash
cd webui
npm ci
npm run build
cd ..
```
If `webui/static/dist/.vite/manifest.json` is missing or stale, React-owned routes and route handoffs may not load correctly.
**YouTube streaming / music videos** need two extra things on bare-metal installs (Docker bundles both):
- **Deno** — yt-dlp now requires a JavaScript runtime to unlock YouTube formats. Without it, streams and music-video downloads fail with `Requested format is not available`. Install: `winget install DenoLand.Deno` (Windows) or see [deno.com](https://docs.deno.com/runtime/), then restart SoulSync.
- **yt-dlp nightly** — the stable release can lag months behind YouTube changes. If YouTube breaks, update with: `python -m pip install -U --pre "yt-dlp[default]"`
### Local Development
This is only for contributors working on the WebUI with hot reload. Normal Python/no-Docker installs should build once with `npm run build` as shown above, then run only Gunicorn.
For active frontend development, use two terminals so the backend and Vite stay independent:
Use two terminals so the backend and Vite stay independent:
1. Backend
```bash
@ -360,7 +319,6 @@ on any OS. `./dev.sh` remains available as a Unix shell wrapper.
- **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases)) — required for Soulseek downloads
- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended for discovery
- **Media Server** (optional): Plex, Jellyfin, or Navidrome
- **Deno** (Python/no-Docker installs only): JavaScript runtime required by yt-dlp for YouTube streaming/music videos — `winget install DenoLand.Deno` or [deno.com](https://docs.deno.com/runtime/). Docker images bundle it.
- **Deezer ARL token** (optional): For Deezer downloads — get from browser cookies after logging into deezer.com
- **Tidal account** (optional): For Tidal downloads — authenticate via device flow in Settings
- **Qobuz account** (optional): For Qobuz downloads — email/password login in Settings

13
RELEASE_2.7.9_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.7.9** is out 🎉 a big one.
🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶

13
RELEASE_2.8.0_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
enjoy! 🎶

15
RELEASE_2.8.1_discord.md
View file

@ -1,15 +0,0 @@
**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
enjoy! 🎶

9
RELEASE_2.8.2_discord.md
View file

@ -1,9 +0,0 @@
**SoulSync 2.8.2** is out 🎉 a stability + performance release.
🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
enjoy! 🎶

144
SPEC_canonical_album_version.md
View file

@ -1,144 +0,0 @@
# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
**Status:** design only — no code yet.
**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
the user's actual files, so the Library Reorganizer, Track Number Repair, and
tagging/enrichment all agree on the same release. Today each re-resolves
independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
Believer=3; standard album mislinked to a deluxe release).
**Canonical-selection rule (decided):** *match the user's actual files.* The
canonical release is the candidate whose track count + per-track durations +
titles best fit what's on disk. Self-correcting: picks standard when you own the
standard, deluxe when you own the deluxe.
---
## Hard requirement: don't disrupt the running app
Every stage below is **additive and dormant until explicitly consumed**, and
every consumer **falls back to today's behavior when no canonical is set**. So:
- albums with no resolved canonical behave EXACTLY as they do now;
- each stage is independently shippable and reversible;
- nothing big-bangs.
---
## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
### Schema (additive, nullable → migration-safe)
Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
existing column-exists checks; see [[db-schema-review]] migration-safety notes):
- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
- `canonical_album_id TEXT`
- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
- `canonical_resolved_at TIMESTAMP`
All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
backfill in this stage. No reads in this stage.
### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
```
score_release_against_files(file_tracks, release_tracks) -> float
pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
```
- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
- `release_tracks`: a candidate release's tracklist (same shape).
- Scoring (tunable weights):
- **track-count fit** — exact match strongly preferred; |Δcount| penalized.
- **duration alignment** — greedily match each file to its closest release
track by duration (within a tolerance, e.g. ±3s); reward coverage.
- **title overlap** — token/fuzzy overlap as a tiebreaker.
- **graceful degradation** — if a source gives no per-track durations, fall
back to count + title only (never crash, never force-pick).
- Returns the best candidate + score, or (None, 0) when nothing clears a floor
(so we never pin a bad guess — leave it unresolved, consumers fall back).
### Tests (extreme, like the rest of this codebase)
- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
- same album, 17 files → picks deluxe.
- duration disambiguation when track counts tie (e.g. radio edit vs album).
- missing-duration source → count+title fallback still picks sanely.
- no candidate clears the floor → (None, 0).
- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
whichever the files actually match.
**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
them yet. Provably zero behavior change.**
---
## Stage 2 — Resolver populates canonical (writes, still no consumers)
A function `resolve_canonical_for_album(album_id, db, ...)`:
1. Gather on-disk file metadata for the album (durations/titles) via the
library's known file paths.
2. Gather candidate releases: every source the album has an ID for
(spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
case — sibling editions discoverable from those. Fetch each tracklist
(cached, rate-limited).
3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
on the album row if it clears the floor.
Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
unchanged — this stage only populates the new columns. Reversible: clearing the
columns reverts to unresolved.
Tests: resolver picks the right release for the standard/deluxe fixtures; stores
nothing when below floor; idempotent re-resolve.
Cost note: fetching multiple candidate releases = more API calls. Mitigate via
cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
progress so it's not silent.
---
## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
In `library_reorganize._resolve_source`: if the album has
`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
current `get_source_priority` walk. One-line precedence change, fully gated on
non-NULL.
Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
to today. Re-run the existing reorganize battery (148 tests) — must stay green.
**This alone fixes #767-Bug2** (a standard album whose files match the standard
release pins the standard, so reorganize stops targeting the deluxe folder).
---
## Stage 4 — Track Number Repair reads canonical (closes #765)
In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
everything) — if the album has a canonical `(source, album_id)`, use it. The
existing 6-level cascade stays as the fallback for albums with no canonical
(preserves its all-01-album rescue ability — the regression risk we refused to
take in the reactive fix).
Now both tools resolve the SAME release → same track numbers → no contradiction.
Tests: canonical present → both tools agree (shared-release test); canonical
NULL → existing cascade unchanged.
---
## Risks & mitigations
- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
only-on-(re)enrich, progress-logged.
- **Sources without per-track durations** → scorer degrades to count+title.
- **Schema migration** → additive nullable columns only; idempotent guards.
- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
stored for inspection/re-resolve; manual override possible later.
- **Backward-compat** → every consumer falls back to today's path when NULL, so
un-resolved albums (incl. all existing albums until backfilled) are unaffected.
## Out of scope (for now)
- Per-album manual version override UI (can layer on later — the columns support it).
- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
once they share the canonical.
## Suggested order to build
1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
after any stage and the app is consistent (just with fewer consumers wired).

134
Support/REVERSE-PROXY.md
View file

@ -1,134 +0,0 @@
# Running SoulSync behind a reverse proxy (nginx / Caddy / Traefik)
Putting SoulSync behind a reverse proxy lets you serve it over **HTTPS** and — the
important part — put **authentication** in front of it before exposing it to the
internet. This guide covers the safe setup.
> **The golden rule:** the safest way to expose *any* self-hosted app publicly is
> to require authentication at the proxy (an auth layer), **not** to rely on the
> app's own protection. SoulSync's launch PIN is a useful fallback, but it is not
> a substitute for a real auth layer on a public instance.
---
## 1. Turn on reverse-proxy mode
By default SoulSync does **not** trust proxy headers (so a direct client can't spoof
its IP or pretend the connection is HTTPS). If you're behind a proxy that
terminates TLS, turn on **Settings → Security → "Behind a reverse proxy"** and
**restart SoulSync** (this option applies at startup).
When enabled, SoulSync:
- trusts `X-Forwarded-For/Proto/Host/Port` from **one** proxy hop (correct client
IP, HTTPS detection, redirects),
- marks its session cookie `Secure` (HTTPS-only) + `SameSite=Lax`, and
- sends conservative security headers (`X-Content-Type-Options: nosniff`,
`X-Frame-Options: SAMEORIGIN`, `Strict-Transport-Security`). No CSP is set — tune
one at your proxy if you want it.
**Leave it off if you access SoulSync directly over http:// on your LAN** — turning
it on would make the session cookie HTTPS-only and break plain-HTTP access. With it
off, none of the above applies and SoulSync behaves exactly as before.
> The launch PIN is also brute-force limited (10 wrong attempts from an IP → a
> short cooldown), regardless of this setting — a correct PIN is never affected.
Restart SoulSync after changing it.
---
## 2. nginx
SoulSync uses WebSockets (Socket.IO), so the `Upgrade`/`Connection` headers are
**required** — without them live updates silently stop working.
```nginx
server {
listen 443 ssl;
server_name soulsync.example.com;
ssl_certificate /etc/letsencrypt/live/soulsync.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/soulsync.example.com/privkey.pem;
# Large library scans / uploads
client_max_body_size 0;
location / {
proxy_pass http://127.0.0.1:8008;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
# Required for Socket.IO / live updates
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 3600s; # long-running scans
proxy_send_timeout 3600s;
}
}
```
---
## 3. Caddy
Caddy handles TLS automatically and proxies WebSockets out of the box:
```caddy
soulsync.example.com {
reverse_proxy 127.0.0.1:8008
}
```
Caddy sets `X-Forwarded-*` for you. (Add an auth provider directive if you want
auth at the proxy — see below.)
---
## 4. Traefik
Traefik proxies WebSockets automatically and forwards the headers. Point a router
at the SoulSync service on port `8008` with your TLS resolver; no extra WebSocket
config is needed.
---
## 5. Add authentication in front (recommended for public instances)
Pick one:
- **Auth proxy** — [Authelia](https://www.authelia.com/),
[Authentik](https://goauthentik.io/), or
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). These sit in front
of SoulSync and force a login (with 2FA) before any request reaches it. Best
option for internet exposure.
SoulSync can **trust the proxy's authenticated-user header** so the launch PIN is
skipped once the proxy has logged you in. Set the header name in **Settings →
Security → "Auth proxy user header"** (e.g. `Remote-User`).
> ⚠️ **Only enable this behind a proxy you control that STRIPS any client-supplied
> copy of that header.** Otherwise a direct visitor could send `Remote-User: admin`
> and walk straight in. It's **off by default** — an unset header name means
> SoulSync ignores the header entirely (a spoofed one does nothing).
- **HTTP Basic Auth** — quick and simple (nginx `auth_basic` / Caddy `basicauth`).
Better than nothing; weaker than an auth proxy.
- **SoulSync launch PIN** — set an admin PIN in Settings. Enforced server-side, so
it can't be bypassed by hitting the API directly — but it's a shared PIN, so
treat it as a fallback, not your only gate.
---
## Troubleshooting
- **Live updates / progress bars don't move** → the WebSocket `Upgrade`/`Connection`
headers are missing (nginx) or your proxy is buffering. Check section 2.
- **Login won't stick / "session expired"** → you enabled `trust_reverse_proxy` but
are reaching SoulSync over plain `http://`. The session cookie is now HTTPS-only;
use `https://`, or turn the setting off for direct HTTP access.
- **Scans time out** → raise `proxy_read_timeout` / `proxy_send_timeout`.

3
api/downloads.py
View file

@ -125,9 +125,6 @@ def register_routes(bp):
if not soulseek:
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
current_app.logger.info(
f"[CancelTrigger:api.public_cancel] download_id={download_id} username={username}"
)
ok = run_async(soulseek.cancel_download(download_id, username, remove=True))
if ok:
return api_success({"message": "Download cancelled."})

1
api/serializers.py
View file

@ -368,7 +368,6 @@ def serialize_similar_artist(obj, fields: Optional[Set[str]] = None) -> dict:
"source_artist_id": d.get("source_artist_id"),
"similar_artist_spotify_id": d.get("similar_artist_spotify_id"),
"similar_artist_itunes_id": d.get("similar_artist_itunes_id"),
"similar_artist_musicbrainz_id": d.get("similar_artist_musicbrainz_id"),
"similar_artist_name": d.get("similar_artist_name"),
"similarity_rank": d.get("similarity_rank"),
"occurrence_count": d.get("occurrence_count"),

10
config/config.example.json
View file

@ -44,8 +44,7 @@
},
"metadata_enhancement": {
"enabled": true,
"embed_album_art": true,
"single_to_album": false
"embed_album_art": true
},
"file_organization": {
"enabled": true,
@ -57,11 +56,6 @@
"playlist_path": "$playlist/$artist - $title"
}
},
"import": {
"staging_path": "./Staging",
"replace_lower_quality": false,
"folder_artist_override": true
},
"lossy_copy": {
"enabled": false,
"bitrate": "320",
@ -73,4 +67,4 @@
"listenbrainz": {
"token": "LISTENBRAINZ_TOKEN"
}
}
}

196
config/settings.py
View file

@ -66,12 +66,6 @@ class ConfigManager:
self._load_config()
# Placeholder shipped to the browser in place of a configured secret
# (#832 follow-up). The settings UI shows it as masked dots; if it's
# round-tripped back on save, ``set()`` treats it as "keep existing" so the
# real value is never overwritten by the mask.
REDACTED_SENTINEL = '__redacted_unchanged__'
# Dot-notation paths to sensitive config values that must be encrypted at rest.
# Paths pointing to dicts encrypt the entire dict as a JSON blob.
_SENSITIVE_PATHS = frozenset({
@ -93,10 +87,6 @@ class ConfigManager:
'soulseek.api_key',
'deezer_download.arl',
'lidarr_download.api_key',
'prowlarr.api_key',
'torrent_client.password',
'usenet_client.api_key',
'usenet_client.password',
# Enrichment services
'listenbrainz.token',
'acoustid.api_key',
@ -486,61 +476,11 @@ class ConfigManager:
"search_min_delay_seconds": 0,
},
"download_source": {
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid", "torrent", "usenet"
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid"
"hybrid_primary": "soulseek", # Legacy: primary source for hybrid mode
"hybrid_secondary": "youtube", # Legacy: fallback source for hybrid mode
"hybrid_order": [], # Ordered list of sources for hybrid mode (overrides primary/secondary)
"stream_source": "youtube", # Options: "youtube" (instant, default), "active" (use download source; falls back to youtube if soulseek)
# Album-bundle (torrent / usenet single-source) poll tuning.
# Downloader is polled every N seconds until the release
# lands; whole job aborts at the timeout. Defaults match
# the previous hard-coded constants. Users on slow private
# trackers / large box sets can extend the timeout without
# editing source.
"album_bundle_poll_interval_seconds": 2.0,
"album_bundle_timeout_seconds": 6 * 60 * 60, # 6 hours
# Stalled-torrent handling (noldevin): abandon a torrent that
# makes zero download progress for this long (dead magnet
# stuck on "downloading metadata", no seeders) instead of
# holding the worker for the full album timeout. 0 disables.
"torrent_stall_timeout_seconds": 10 * 60, # 10 minutes
# What to do when a torrent stalls: "abandon" (remove it +
# its partial data, fail the download so the next source can
# try) or "pause" (pause in the client, leave for the user).
"torrent_stall_action": "abandon",
# Where THIS container can read completed torrent/usenet
# downloads (#857). The downloader (qBit/SAB) reports a save
# path from inside ITS OWN container — often a category folder
# like /data/downloads/music — which may be mounted at a
# different point here. Set these to the in-container path(s)
# where SoulSync sees those finished downloads; the resolver
# then finds the release by name under them. Empty = fall back
# to the soulseek download/transfer dirs (the shared-volume
# default). See core.download_plugins.album_bundle.resolve_reported_save_path.
"torrent_download_path": "",
"usenet_download_path": "",
# Explicit remote→local prefix mappings for non-shared / oddly
# mounted layouts (Sonarr/Radarr "Remote Path Mapping" style):
# a list of {"from": "<client path>", "to": "<soulsync path>"}.
# Tried before the basename fallback above.
"usenet_path_mappings": [],
},
"post_processing": {
# When a download is quarantined (AcoustID mismatch, integrity /
# duration failure), retry the next-best candidate instead of
# failing outright. Default ON (PR #801's documented default —
# the monitor reads this with inline default True; this template
# said False, so fresh installs silently shipped with the retry
# engine off while existing configs got it on. CI caught the
# split: its fresh default config failed all 7 requeue tests).
"retry_next_candidate_on_mismatch": True,
# Opt-in exhaustive retry: budget retries PER SOURCE so every
# source (Soulseek, then HiFi/Tidal/…) gets its own attempts
# before the track gives up. Default off (single global cap).
"retry_exhaustive": False,
# Retries per search query per source in exhaustive mode. The
# per-source budget is query_count × this value.
"retries_per_query": 5,
},
"tidal_download": {
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
@ -579,37 +519,6 @@ class ConfigManager:
"quality_profile": "Any",
"cleanup_after_import": True,
},
# Prowlarr — indexer aggregator. Feeds the torrent / usenet
# download plugins. Not a standalone source.
"prowlarr": {
"url": "",
"api_key": "",
# Comma-separated list of indexer IDs to limit searches to.
# Empty = search all enabled indexers.
"indexer_ids": "",
},
# Torrent client — receives .torrent / magnet URIs from the
# torrent download plugin. ``type`` picks which adapter to
# instantiate (qbittorrent | transmission | deluge).
"torrent_client": {
"type": "qbittorrent",
"url": "",
"username": "",
"password": "",
"category": "soulsync",
"save_path": "",
},
# Usenet client — receives .nzb URLs / payloads. ``type``
# picks the adapter (sabnzbd | nzbget). SABnzbd uses an
# API key; NZBGet uses username + password.
"usenet_client": {
"type": "sabnzbd",
"url": "",
"api_key": "",
"username": "",
"password": "",
"category": "soulsync",
},
"soundcloud_download": {
# Anonymous-only for now — SoundCloud Go+ OAuth tier could be
# added later, with credentials living under a "session" subkey
@ -643,43 +552,16 @@ class ConfigManager:
"path": os.environ.get('DATABASE_PATH', 'database/music_library.db'),
"max_workers": 5
},
"image_cache": {
"enabled": True,
"path": "storage/image_cache",
"ttl_seconds": 2592000,
"failed_ttl_seconds": 21600,
"max_download_mb": 15
},
"metadata_enhancement": {
"enabled": True,
"embed_album_art": True,
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"],
# Ordered preferred cover-art sources (empty = use the
# download's own art, i.e. today's behavior). Resolved + walked
# with fallback by core/metadata/art_sources.py.
"album_art_order": [],
# Minimum cover-art resolution (shortest side, px). A preferred
# source whose art is smaller is skipped so the next source is
# tried — stops a low-res Cover Art Archive upload from winning.
# 0 disables the size gate.
"min_art_size": 1000,
# When a track matches a SINGLE release, look up the parent ALBUM
# that contains it and tag it as that album, so it groups with its
# album-mates and gets the album cover (not the single's). Off by
# default — it's an extra per-import metadata lookup.
"single_to_album": False
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"]
},
"musicbrainz": {
"embed_tags": True
},
"playlist_sync": {
"create_backup": True,
# How a re-sync writes to the server playlist:
# replace — delete + recreate (default; today's behavior)
# reconcile — edit in place (add/remove delta), preserving the
# playlist's custom image, description, and identity (#792)
# append — only add new tracks, never remove
"mode": "replace"
"create_backup": True
},
"settings": {
"audio_quality": "flac"
@ -705,35 +587,12 @@ class ConfigManager:
},
"import": {
"staging_path": "./Staging",
# Master toggle for quality-filtering on import. On by default:
# downloaded files that don't meet the quality profile are
# quarantined instead of imported (same gate the download
# pipeline uses). Off → import everything regardless of quality;
# the library Quality Upgrade Scanner still flags them.
"quality_filter_enabled": True,
"replace_lower_quality": False,
# Use the top Staging folder as the artist (Artist/Album layouts,
# mixtapes). On by default to preserve the long-standing import
# behaviour for existing users. Turn OFF if you stage a mixed pile
# of songs under one container folder, otherwise that folder's name
# overrides every metadata-identified artist (the "soulsync" case).
"folder_artist_override": True
"replace_lower_quality": False
},
"m3u_export": {
"enabled": False,
"entry_base_path": ""
},
"playlists": {
# Where "Organize by playlist" materializes playlist folders.
# MUST be a separate root from the music library so the media
# server (and the maintenance jobs) never scan it — otherwise the
# same track would show up twice. Mapped separately for Docker.
"materialize_path": "./Playlists",
# "symlink" (relative links, ~zero disk) or "copy" (real
# duplicates for FAT/USB/DAPs that can't follow links). Symlink
# auto-falls back to copy when the filesystem can't link.
"materialize_mode": "symlink"
},
"youtube": {
"cookies_browser": "", # "", "chrome", "firefox", "edge", "brave", "opera", "safari"
"download_delay": 3, # seconds between sequential downloads
@ -842,40 +701,7 @@ class ConfigManager:
return value
def redacted_config(self) -> Dict[str, Any]:
"""Deep copy of the live config with every sensitive value masked.
Used for ``GET /api/settings`` so decrypted secrets never reach the
browser (#832 follow-up). A *set* secret becomes ``REDACTED_SENTINEL``
(the UI renders it as masked dots); an unset one stays empty so the UI
can show "not configured". Dict-valued secrets (OAuth sessions) collapse
to the sentinel too — the UI has no field for them anyway. The matching
guard in ``set()`` turns a round-tripped sentinel back into a no-op.
"""
import copy
data = copy.deepcopy(self.config_data)
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
parent = data
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf in parent and parent[leaf] not in (None, '', {}, [], 0, False):
parent[leaf] = self.REDACTED_SENTINEL
return data
def set(self, key: str, value: Any):
# The UI round-trips REDACTED_SENTINEL for any secret the user didn't
# touch — never let the mask overwrite the real value (#832 follow-up).
if value == self.REDACTED_SENTINEL and key in self._SENSITIVE_PATHS:
return
keys = key.split('.')
config = self.config_data
@ -887,20 +713,6 @@ class ConfigManager:
config[keys[-1]] = value
self._save_config()
def resolve_secret(self, key: str, posted: Any) -> str:
"""Resolve a secret value coming back from the settings UI.
The UI renders a saved-but-untouched secret as the REDACTED_SENTINEL (shown
masked); empty or that sentinel means "use the stored value", while a real
string is a genuine new secret. A connection-test endpoint should test the
EFFECTIVE secret, not the mask — otherwise testing a saved-but-untouched
token sends the sentinel and the source rejects it (#870)."""
if isinstance(posted, str):
posted = posted.strip()
if not posted or posted == self.REDACTED_SENTINEL:
return self.get(key, '') or ''
return posted
def get_spotify_config(self) -> Dict[str, str]:
return self.get('spotify', {})

134
core/acoustid_client.py
View file

@ -282,9 +282,8 @@ class AcoustIDClient:
def test_api_key(self) -> Tuple[bool, str]:
"""
Validate the API key with a direct AcoustID lookup call. An invalid key
is reported as invalid (error code 4); any other error means the key was
accepted.
Validate the API key by fingerprinting a real audio file and looking it up.
Falls back to a direct API call if no audio files are available.
Returns:
Tuple of (success, message)
@ -295,12 +294,24 @@ class AcoustIDClient:
import requests
try:
# Authoritative key check: a direct API lookup with a dummy
# fingerprint. AcoustID validates the client key first, so an
# invalid key returns error code 4 regardless of the fingerprint.
# (The previous real-file path trusted "no exception = valid", but
# fingerprint_and_lookup swallows the invalid-key error and returns
# None — so it reported broken keys as valid. #756-adjacent.)
# Try to find a real audio file to fingerprint for an end-to-end test
test_file = self._find_test_audio_file()
if test_file and CHROMAPRINT_AVAILABLE:
logger.info(f"Testing API key with real audio file: {test_file}")
try:
result = self.fingerprint_and_lookup(test_file)
# If we get here without exception, the API key is valid
# (invalid keys raise or return error before results)
return True, "AcoustID API key is valid"
except Exception as e:
error_str = str(e).lower()
if 'invalid' in error_str and 'api' in error_str:
return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
# Fingerprint/lookup failed for non-key reasons, fall through to direct test
logger.warning(f"Real file test failed ({e}), trying direct API call")
# Fallback: direct API call with minimal fingerprint
url = 'https://api.acoustid.org/v2/lookup'
params = {
'client': self.api_key,
@ -315,6 +326,7 @@ class AcoustIDClient:
if data.get('status') == 'error':
error = data.get('error', {})
error_code = error.get('code', 0)
error_msg = error.get('message', 'Unknown error')
# Error code 4 is specifically "invalid API key"
if error_code == 4:
@ -334,33 +346,33 @@ class AcoustIDClient:
logger.error(f"Error testing AcoustID API key: {e}")
return False, f"Error: {str(e)}"
def lookup_with_status(self, audio_file: str) -> Dict[str, Any]:
"""Fingerprint + AcoustID lookup returning a STRUCTURED result.
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
"""
Generate fingerprint and look up recording in AcoustID.
Unlike fingerprint_and_lookup() (which collapses every outcome into
dict-or-None), this distinguishes a genuine no-match from an actual
error — an invalid API key, rate limit, missing chromaprint, or a
fingerprint failure. That distinction is what lets the UI show "AcoustID
Error" (something is broken — fix it) instead of a benign-looking
"Skipped" that silently hides a dead key.
This is the main method - combines fingerprinting and lookup in one call.
Returns dict with:
'status': 'ok' | 'no_match' | 'error' | 'no_backend'
| 'fingerprint_error' | 'unsupported' | 'unavailable'
| 'not_found'
'recordings': list (meaningful only for 'ok')
'best_score': float
'recording_mbids': list
'error': human-readable detail for any non-'ok' status
'invalid_key': bool (True when the API specifically rejected the key)
Args:
audio_file: Path to the audio file
Returns:
Dict with:
'recordings': list of dicts with 'mbid', 'title', 'artist', 'score'
'best_score': float (highest score across all results)
'recording_mbids': list of unique MBIDs (for backward compat)
Or None on error.
"""
if not ACOUSTID_AVAILABLE:
return {'status': 'unavailable', 'recordings': [], 'error': 'pyacoustid library not installed'}
logger.debug("Cannot lookup: pyacoustid not available")
return None
if not self.api_key:
return {'status': 'unavailable', 'recordings': [], 'error': 'No AcoustID API key configured'}
logger.debug("Cannot lookup: no API key")
return None
if not os.path.isfile(audio_file):
logger.warning(f"Cannot lookup: file not found: {audio_file}")
return {'status': 'not_found', 'recordings': [], 'error': f'File not found: {audio_file}'}
return None
# Check channel count — chromaprint crashes (SIGABRT) on >2 channel files (e.g. 5.1 surround)
try:
@ -370,8 +382,7 @@ class AcoustIDClient:
channels = getattr(mf.info, 'channels', 2)
if channels and channels > 2:
logger.warning(f"Skipping AcoustID: file has {channels} channels (surround audio): {audio_file}")
return {'status': 'unsupported', 'recordings': [],
'error': f'{channels}-channel (surround) audio not supported by chromaprint'}
return None
except Exception as e:
logger.debug(f"Could not check channel count, proceeding anyway: {e}")
@ -381,12 +392,17 @@ class AcoustIDClient:
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "NOT SET"
logger.info(f"Fingerprinting and looking up: {audio_file} (API key: {api_key_preview})")
# Use match() which handles fingerprinting + lookup + parsing
logger.debug("Running acoustid.match()...")
recordings = []
seen_mbids = set()
best_score = 0.0
for result in acoustid.match(self.api_key, audio_file, parse=True):
for result in acoustid.match(
self.api_key,
audio_file,
parse=True
):
# match() with parse=True returns (score, recording_id, title, artist)
if not isinstance(result, tuple) or len(result) < 2:
logger.warning(f"Unexpected result format: {result}")
@ -404,57 +420,45 @@ class AcoustIDClient:
if recording_id and recording_id not in seen_mbids:
seen_mbids.add(recording_id)
recordings.append({'mbid': recording_id, 'title': title, 'artist': artist, 'score': score})
recordings.append({
'mbid': recording_id,
'title': title,
'artist': artist,
'score': score,
})
logger.debug(f"Found match: {title} by {artist} (MBID: {recording_id}, score: {score})")
if not recordings:
logger.info(f"No AcoustID matches found for: {audio_file}")
return {'status': 'no_match', 'recordings': [], 'best_score': best_score,
'recording_mbids': [], 'error': 'Track not found in AcoustID database'}
return None
logger.info(f"AcoustID found {len(recordings)} recording(s) (best score: {best_score:.2f})")
return {'status': 'ok', 'recordings': recordings, 'best_score': best_score,
'recording_mbids': list(seen_mbids)}
return {
'recordings': recordings,
'best_score': best_score,
'recording_mbids': list(seen_mbids),
}
except acoustid.NoBackendError:
logger.error("Chromaprint library not found and fpcalc not available")
return {'status': 'no_backend', 'recordings': [],
'error': 'Chromaprint/fpcalc not installed (install libchromaprint1)'}
return None
except acoustid.FingerprintGenerationError as e:
logger.warning(f"Failed to fingerprint {audio_file}: {e}")
return {'status': 'fingerprint_error', 'recordings': [], 'error': f'Could not fingerprint file: {e}'}
return None
except acoustid.WebServiceError as e:
# Log more details about the API error
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "???"
logger.warning(f"AcoustID API error (key: {api_key_preview}): {e}")
# Check for common errors
error_str = str(e).lower()
# Old pyacoustid reports an invalid key as the bare "status: error"
# (it drops the detail), so treat that as an invalid-key signal too.
invalid = ('invalid' in error_str or 'unknown' in error_str or 'status: error' in error_str)
if invalid:
logger.error("AcoustID API key appears to be invalid — check your AcoustID settings")
if 'invalid' in error_str or 'unknown' in error_str:
logger.error("API key appears to be invalid - check your AcoustID settings")
elif 'rate' in error_str or 'limit' in error_str:
logger.warning("Rate limited by AcoustID — will retry later")
return {'status': 'error', 'recordings': [], 'invalid_key': invalid,
'error': f'AcoustID API error: {e}'}
logger.warning("Rate limited by AcoustID - will retry later")
return None
except Exception as e:
logger.error(f"Unexpected error in AcoustID lookup: {e}", exc_info=True)
return {'status': 'error', 'recordings': [], 'error': f'Unexpected error: {e}'}
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
"""Legacy dict-or-None lookup. Returns the recordings dict on a confirmed
match, else None. Kept for callers that only need "did we identify it"
(library scanner, auto-import). Callers that must report WHY a lookup
didn't match (verification badge, key test) should use
``lookup_with_status`` so an error isn't mistaken for a no-match.
"""
res = self.lookup_with_status(audio_file)
if res.get('status') == 'ok':
return {
'recordings': res['recordings'],
'best_score': res.get('best_score', 0.0),
'recording_mbids': res.get('recording_mbids', []),
}
return None
return None
def refresh_config(self):
"""Refresh cached config values (call after settings change)."""

446
core/acoustid_verification.py
View file

@ -17,16 +17,14 @@ from utils.logging_config import get_logger
from core.acoustid_client import AcoustIDClient
from core.matching_engine import MusicMatchingEngine
from core.matching.version_mismatch import is_acceptable_version_mismatch
from core.matching.script_compat import is_cross_script_mismatch
from core.musicbrainz_client import MusicBrainzClient
logger = get_logger("acoustid.verification")
# Thresholds — single definition lives in the shared core; re-exported here so
# existing importers keep working and the values can't drift between paths.
from core.matching.audio_verification import ( # noqa: E402
MIN_ACOUSTID_SCORE, TITLE_MATCH_THRESHOLD, ARTIST_MATCH_THRESHOLD,
)
# Thresholds
MIN_ACOUSTID_SCORE = 0.80 # Minimum AcoustID fingerprint score to trust
TITLE_MATCH_THRESHOLD = 0.70 # Title similarity needed to consider a match
ARTIST_MATCH_THRESHOLD = 0.60 # Artist similarity needed to consider a match
# Single matching-engine instance so version detection reuses the same patterns
# used by the pre-download Soulseek matcher (remix / live / acoustic /
@ -52,34 +50,170 @@ class VerificationResult(Enum):
"""Possible outcomes of audio verification."""
PASS = "pass" # Title/artist match - file is correct
FAIL = "fail" # Title/artist mismatch - wrong file downloaded
SKIP = "skip" # Genuinely couldn't verify (no match in DB) - continue normally
SKIP = "skip" # Could not verify (error or unavailable) - continue normally
DISABLED = "disabled" # Verification not enabled
ERROR = "error" # Lookup errored (invalid key / rate limit / no backend) - continue, but flag it
# normalize() + similarity() + the alias-aware comparison now live in the shared
# decision core (core/matching/audio_verification.py) so import-time verification
# and the library scan share ONE definition — the <>-strip fix, CJK handling and
# thresholds can't drift apart again. Names kept (`_normalize` etc.) for existing
# importers/tests.
from core.matching.audio_verification import ( # noqa: E402
normalize as _normalize,
similarity as _similarity,
_alias_aware_artist_sim,
_find_best_title_artist_match as _core_find_best_title_artist_match,
evaluate as _core_evaluate,
Decision as _CoreDecision,
)
def _normalize(text: str) -> str:
"""Normalize a string for comparison: lowercase, strip parentheticals, punctuation."""
if not text:
return ""
s = text.lower().strip()
# Remove ALL parenthetical suffixes — these are metadata annotations, not core title
# Covers: (Live), (Remastered), (Parody of ...), (from "..." Soundtrack), (feat. ...), etc.
s = re.sub(r'\s*\([^)]*\)', '', s)
# Remove ALL square bracket suffixes: [Live], [Remastered], [Deluxe], etc.
s = re.sub(r'\s*\[[^\]]*\]', '', s)
# Remove trailing featuring info not in parentheses: "feat. ...", "ft. ...", "featuring ..."
s = re.sub(r'\s+(?:feat\.?|ft\.?|featuring)\s+.*$', '', s, flags=re.IGNORECASE)
# Remove dash-separated version tags: "- Vocal", "- Instrumental", "- Acoustic", etc.
s = re.sub(r'\s*-\s*(?:vocal|instrumental|acoustic|live|remix|cover|clean|explicit|radio\s*edit|original\s*mix|extended\s*mix|club\s*mix)\s*$', '', s, flags=re.IGNORECASE)
# Remove soundtrack/source subtitles: ' - From "..." Soundtrack', ' - from the film ...'
s = re.sub(r'\s*-\s*from\s+.+$', '', s, flags=re.IGNORECASE)
# Remove non-alphanumeric except spaces
s = re.sub(r'[^\w\s]', '', s)
# Collapse whitespace
s = re.sub(r'\s+', ' ', s).strip()
return s
def _find_best_title_artist_match(recordings, expected_title, expected_artist,
expected_artist_aliases=None):
"""Back-compat wrapper around the shared core matcher (keeps the
``expected_artist_aliases`` kwarg name for existing callers/tests)."""
return _core_find_best_title_artist_match(
recordings, expected_title, expected_artist, expected_artist_aliases,
def _similarity(a: str, b: str) -> float:
"""Calculate similarity between two strings (0.0-1.0) after normalization."""
na = _normalize(a)
nb = _normalize(b)
if not na or not nb:
return 0.0
if na == nb:
return 1.0
return SequenceMatcher(None, na, nb).ratio()
def _alias_aware_artist_sim(
expected_artist: str,
actual_artist: str,
aliases: Optional[Any] = None,
) -> float:
"""Best artist-similarity across (expected, *aliases) vs actual.
Issue #442 — when expected and actual are in different scripts
(e.g. `Hiroyuki Sawano` vs `澤野弘之`), raw `_similarity` scores
near 0% even though MusicBrainz aliases bridge them. Routes
through the pure helper so the verifier inherits one shared
contract.
Returns the highest score across all candidates so existing
threshold checks (>= ARTIST_MATCH_THRESHOLD) keep their
semantics. When `aliases` is None or empty, behaves identically
to the prior raw `_similarity(expected, actual)` call.
`aliases` accepts two shapes:
- **Iterable** (list/tuple/set of strings): used directly. Used
by tests that already know the aliases.
- **Callable**: invoked LAZILY only when direct similarity
falls below the threshold. Lets the verifier pass a memoizing
thunk that resolves aliases (DB / cache / live MB) only when
needed. Verifications where the direct match already passes
never trigger the lookup chain — no wasted DB query for the
happy path.
Diagnostic logging: emits an INFO line whenever an alias rescues
a comparison that direct similarity would have failed. Lets
future bug reports trace which alias triggered which PASS
decision (e.g. "this file passed because alias `澤野弘之` matched
the file's artist tag").
"""
from core.matching.artist_aliases import artist_names_match
direct = _similarity(expected_artist, actual_artist)
# Fast path — direct match already passes the threshold OR caller
# supplied no aliases handle. Avoids any lookup work.
if aliases is None:
return direct
if direct >= ARTIST_MATCH_THRESHOLD:
return direct
# Resolve the iterable. Callable provider invoked NOW (lazily —
# the caller can memoize the result across multiple invocations
# within one verify_audio_file call).
resolved = aliases() if callable(aliases) else aliases
if not resolved:
return direct
_matched, score = artist_names_match(
expected_artist,
actual_artist,
aliases=resolved,
threshold=ARTIST_MATCH_THRESHOLD,
similarity=_similarity,
)
# Diagnostic — alias rescued a comparison that direct would
# have failed. Worth logging at INFO since it's a user-visible
# decision (file PASS instead of FAIL). One line per rescue
# within a single verify call.
if score >= ARTIST_MATCH_THRESHOLD and direct < ARTIST_MATCH_THRESHOLD:
from core.matching.artist_aliases import best_alias_match
winner, _ = best_alias_match(
expected_artist, actual_artist, resolved, similarity=_similarity,
)
logger.info(
"Artist alias rescued comparison: expected=%r vs actual=%r "
"(direct sim=%.2f, alias %r → score=%.2f)",
expected_artist, actual_artist, direct, winner, score,
)
return score
def _find_best_title_artist_match(
recordings: List[Dict[str, Any]],
expected_title: str,
expected_artist: str,
expected_artist_aliases: Optional[Any] = None,
) -> Tuple[Optional[Dict], float, float]:
"""
Find the AcoustID recording that best matches expected title/artist.
Issue #442 — `expected_artist_aliases` (when supplied) is the
list of alternate spellings for `expected_artist` (Japanese
kanji, Cyrillic, etc.). Accepts either:
- An iterable of alias strings (used eagerly), or
- A callable returning the list (resolved lazily — only fires
when at least one recording fails direct artist similarity).
Each recording's artist is scored against (expected, *aliases)
and the best score wins. When the list is empty/omitted/None,
behavior is identical to the prior raw similarity comparison.
Returns:
(best_recording, title_similarity, artist_similarity)
"""
best_rec = None
best_title_sim = 0.0
best_artist_sim = 0.0
best_combined = 0.0
for rec in recordings:
title = rec.get('title') or ''
artist = rec.get('artist') or ''
title_sim = _similarity(expected_title, title)
artist_sim = _alias_aware_artist_sim(
expected_artist, artist, expected_artist_aliases,
)
# Weight title higher since that's the primary identifier
combined = (title_sim * 0.6) + (artist_sim * 0.4)
if combined > best_combined:
best_combined = combined
best_rec = rec
best_title_sim = title_sim
best_artist_sim = artist_sim
return best_rec, best_title_sim, best_artist_sim
# Shared MusicBrainz client for enrichment lookups
_mb_client = None
@ -265,33 +399,18 @@ class AcoustIDVerification:
logger.debug(f"AcoustID verification skipped: {reason}")
return VerificationResult.SKIP, reason
# Step 2: Fingerprint and lookup in AcoustID (structured so an
# actual error — invalid key / rate limit / no chromaprint — is
# reported distinctly from a genuine no-match, instead of both
# silently surfacing as "Skipped").
# Step 2: Fingerprint and lookup in AcoustID
logger.info(f"Fingerprinting and looking up: {audio_file_path}")
lookup = self.acoustid_client.lookup_with_status(audio_file_path) or {}
status = lookup.get('status')
# Infer status by content when absent (a caller/stub that returned
# just recordings): recordings => matched, none => no match.
if status is None:
status = 'ok' if lookup.get('recordings') else 'no_match'
acoustid_result = self.acoustid_client.fingerprint_and_lookup(audio_file_path)
if status in ('error', 'no_backend', 'fingerprint_error', 'unavailable'):
# Something is broken (not the track's fault) — never quarantine
# on this; surface it so the user can fix it.
return VerificationResult.ERROR, lookup.get('error', 'AcoustID lookup failed')
if not acoustid_result:
return VerificationResult.SKIP, "Track not found in AcoustID database"
if status != 'ok':
# no_match / unsupported / not_found — genuinely could not verify.
return VerificationResult.SKIP, lookup.get('error', 'No match in AcoustID database')
acoustid_result = lookup
recordings = acoustid_result.get('recordings', [])
best_score = acoustid_result.get('best_score', 0)
if not recordings:
return VerificationResult.SKIP, "No match in AcoustID database"
return VerificationResult.SKIP, "AcoustID returned no recordings"
logger.debug(
f"AcoustID returned {len(recordings)} recording(s) "
@ -330,32 +449,219 @@ class AcoustIDVerification:
)
return _alias_cache['value']
# Steps 4-5: delegate the PASS/SKIP/FAIL decision to the shared core
# (core/matching/audio_verification.evaluate) so import verification
# and the library scan apply identical logic.
outcome = _core_evaluate(
expected_track_name, expected_artist_name, recordings,
fingerprint_score=best_score,
aliases_provider=_aliases_provider,
# Step 4: Find best title/artist match among AcoustID results
best_rec, title_sim, artist_sim = _find_best_title_artist_match(
recordings, expected_track_name, expected_artist_name,
expected_artist_aliases=_aliases_provider,
)
if not best_rec:
return VerificationResult.SKIP, "No recordings with title/artist info"
matched_title = best_rec.get('title', '?')
matched_artist = best_rec.get('artist', '?')
logger.info(
"Best match: '%s' by '%s' (title_sim=%.2f, artist_sim=%.2f) -> %s",
outcome.matched_title, outcome.matched_artist,
outcome.title_sim, outcome.artist_sim, outcome.decision.value,
f"Best match: '{matched_title}' by '{matched_artist}' "
f"(title_sim={title_sim:.2f}, artist_sim={artist_sim:.2f})"
)
_decision_map = {
_CoreDecision.PASS: VerificationResult.PASS,
_CoreDecision.SKIP: VerificationResult.SKIP,
_CoreDecision.FAIL: VerificationResult.FAIL,
}
result = _decision_map[outcome.decision]
if result == VerificationResult.PASS:
logger.info("AcoustID verification PASSED - %s", outcome.reason)
elif result == VerificationResult.FAIL:
logger.warning("AcoustID verification FAILED - %s", outcome.reason)
else:
logger.info("AcoustID verification SKIPPED - %s", outcome.reason)
return result, outcome.reason
# Step 4b: Version-mismatch gate.
#
# The ``_normalize`` step deliberately strips parentheticals and
# version tags ("(Instrumental)", "- Live", etc) so that legit
# name variations don't fail the title-similarity comparison.
# That same stripping made it impossible to tell a vocal track
# apart from its instrumental: "In My Feelings" and "In My
# Feelings (Instrumental)" both normalize to "in my feelings",
# the title sim ends up 1.0, and the file passes verification
# even though it's the wrong cut.
#
# Detect the version on each side BEFORE normalization runs.
# If the expected track and the AcoustID-matched recording
# disagree on version (one is original, the other is
# instrumental / live / remix / acoustic / etc), reject — the
# fingerprint identified a real song but it's not the one the
# caller asked for.
expected_version = _detect_title_version(expected_track_name)
matched_version = _detect_title_version(matched_title)
if expected_version != matched_version:
# Issue #607 (AfonsoG6): MusicBrainz often stores live
# recordings with bare titles ("Clarity") while the
# release entry carries the venue annotation ("Clarity
# (Live at Blossom Music Center, ...)"). The fingerprint
# correctly identifies the LIVE recording; only the
# title text is bare. Helper accepts the one-sided bare
# case when fingerprint + bare-title + artist all agree.
# Two-sided version mismatches (live vs remix etc) stay
# strict — those are genuinely different recordings.
if is_acceptable_version_mismatch(
expected_version, matched_version,
fingerprint_score=best_score,
title_similarity=title_sim,
artist_similarity=artist_sim,
):
logger.info(
f"AcoustID version annotation differs (expected={expected_version}, "
f"matched={matched_version}) but fingerprint+title+artist all match — "
f"accepting (likely MB metadata gap on a live/version-annotated recording)"
)
else:
msg = (
f"Version mismatch: expected '{expected_track_name}' ({expected_version}) "
f"but file is '{matched_title}' ({matched_version})"
)
logger.warning(f"AcoustID verification FAILED (version mismatch) - {msg}")
return VerificationResult.FAIL, msg
# Step 5: Decide pass/fail based on similarity
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim >= ARTIST_MATCH_THRESHOLD:
msg = (
f"Audio verified: '{matched_title}' by '{matched_artist}' "
f"matches expected '{expected_track_name}' by '{expected_artist_name}' "
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
)
logger.info(f"AcoustID verification PASSED - {msg}")
return VerificationResult.PASS, msg
# Title matches but artist doesn't — could be a cover/collab OR a
# genuinely different track with the same name. Distinguish the
# two by checking whether the expected artist appears anywhere in
# AcoustID's returned recordings.
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim < ARTIST_MATCH_THRESHOLD:
# First: if the expected artist is present in ANY recording's
# metadata for this fingerprint, it's likely the right track
# (AcoustID's "best" match just picked the wrong variant).
for rec in recordings:
rec_artist = rec.get('artist', '')
if _alias_aware_artist_sim(
expected_artist_name, rec_artist, _aliases_provider,
) >= ARTIST_MATCH_THRESHOLD:
msg = (
f"Audio verified: found '{expected_track_name}' by '{expected_artist_name}' "
f"in AcoustID results"
)
logger.info(f"AcoustID verification PASSED (secondary match) - {msg}")
return VerificationResult.PASS, msg
# Expected artist wasn't found anywhere. Decide between:
# - FAIL: clear mismatch, e.g. "Tom Walker" (sim ~0.2) when
# expecting "Maduk" — different song with same name
# - SKIP: ambiguous, e.g. collab / alt credit / formatting
# difference (sim 0.3-0.6)
#
# The 0.3 cutoff catches hard mismatches while preserving the
# benefit of the doubt for borderline artist formatting.
CLEAR_MISMATCH_THRESHOLD = 0.3
if artist_sim < CLEAR_MISMATCH_THRESHOLD:
msg = (
f"Audio mismatch: file identified as '{matched_title}' by '{matched_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' "
f"(title={title_sim:.0%}, artist={artist_sim:.0%}) — "
f"expected artist not found in any AcoustID recording"
)
logger.warning(f"AcoustID verification FAILED (clear artist mismatch) - {msg}")
return VerificationResult.FAIL, msg
msg = (
f"Title matches but artist unclear: "
f"AcoustID='{matched_title}' by '{matched_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' "
f"(artist_sim={artist_sim:.0%} — ambiguous, could be cover/collab)"
)
logger.info(f"AcoustID verification SKIPPED - {msg}")
return VerificationResult.SKIP, msg
# Title doesn't match — check ALL recordings for any title/artist match
# (the best combined match might not be the right one if there are many results)
# Skip recordings whose version (instrumental/live/etc) disagrees with
# what the caller asked for — the version mismatch above checked
# only the best recording, but a wrong-version variant could still
# win this fallback scan if its bare title matched.
for rec in recordings:
t = rec.get('title') or ''
a = rec.get('artist') or ''
if _detect_title_version(t) != expected_version:
continue
if (_similarity(expected_track_name, t) >= TITLE_MATCH_THRESHOLD and
_alias_aware_artist_sim(
expected_artist_name, a, _aliases_provider,
) >= ARTIST_MATCH_THRESHOLD):
msg = (
f"Audio verified: found '{t}' by '{a}' in AcoustID results "
f"matching expected '{expected_track_name}' by '{expected_artist_name}'"
)
logger.info(f"AcoustID verification PASSED (scan match) - {msg}")
return VerificationResult.PASS, msg
# No match found — but if fingerprint score is very high (≥0.95)
# AND we have evidence the mismatch is a language/script case
# (rather than two genuinely different songs by the same artist),
# skip rather than quarantine a correct file. Two routes:
#
# (a) Either side of the comparison contains non-ASCII characters
# — strong signal of transliteration / kanji↔roman cases.
# Artist must still be a strong match to use this path.
# (b) Both title AND artist similarity are very high (the song
# is recognizably the same with minor punctuation / casing
# differences that fell below the strict match thresholds).
#
# The OLD logic was ``title_sim >= 0.55 OR artist_sim >= match``.
# That fired for English-vs-English songs by the same artist that
# share NO actual content — e.g. "R.O.T.C (Interlude)" by
# Kendrick Lamar getting accepted as "Rich (Interlude)" by
# Kendrick Lamar because the artist matched perfectly and
# "interlude" was shared in both titles. Reported by user when
# downloading Mr. Morale: three tracks (Rich Interlude, Savior
# Interlude, Savior) all received the wrong R.O.T.C audio file
# because of this leak.
# Use the BEST matching recording's strings here (not
# `recordings[0]`) so the failure message reports the same
# candidate the title/artist similarity scores came from.
# Issue #607 (AfonsoG6) example 1: the prior code mixed
# `recordings[0]`'s strings (which can be empty) with
# `best_rec`'s scores, producing nonsense reasons like
# "file identified as '' by '' (artist=100%)" when a later
# recording in the list scored well on artist.
display_title = matched_title or '?'
display_artist = matched_artist or '?'
has_non_ascii = (
any(ord(c) > 127 for c in (expected_track_name or ''))
or any(ord(c) > 127 for c in display_title)
)
language_script_skip = (
best_score >= 0.95
and has_non_ascii
and artist_sim >= ARTIST_MATCH_THRESHOLD
)
high_confidence_strong_match_skip = (
best_score >= 0.95
and title_sim >= 0.80
and artist_sim >= ARTIST_MATCH_THRESHOLD
)
if language_script_skip or high_confidence_strong_match_skip:
reason = (
"likely same song in different language/script"
if language_script_skip
else "title/artist match within tolerance"
)
msg = (
f"Title/artist mismatch but fingerprint confidence very high ({best_score:.2f}): "
f"AcoustID='{display_title}' by '{display_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' — "
f"{reason}"
)
logger.info(f"AcoustID verification SKIPPED (high confidence) - {msg}")
return VerificationResult.SKIP, msg
# Low fingerprint score + no metadata match — file is likely wrong.
msg = (
f"Audio mismatch: file identified as '{display_title}' by '{display_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' "
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
)
logger.warning(f"AcoustID verification FAILED - {msg}")
return VerificationResult.FAIL, msg
except Exception as e:
# Any unexpected error -> SKIP (fail open)

77
core/amazon_client.py
View file

@ -70,16 +70,7 @@ _meta_cache_lock = threading.Lock()
class AmazonClientError(RuntimeError):
"""Raised on unrecoverable T2Tunes API errors.
Carries the HTTP ``status_code`` when the failure was an HTTP error, so
callers (the worker's outage detection) can tell a source outage (5xx) from
a per-item miss without parsing the message.
"""
def __init__(self, *args, status_code=None):
super().__init__(*args)
self.status_code = status_code
"""Raised on unrecoverable T2Tunes API errors."""
# ---------------------------------------------------------------------------
@ -336,7 +327,7 @@ class AmazonClient:
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
_rate_limit()
items = self.search_raw(query, types="track")
items = self.search_raw(query, types="track,album")
track_pairs: List[tuple] = [] # (Track, album_asin)
seen_album_asins: List[str] = []
for item in items:
@ -369,7 +360,7 @@ class AmazonClient:
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
_rate_limit()
items = self.search_raw(query, types="track")
items = self.search_raw(query, types="track,album")
seen: Dict[str, Artist] = {}
artist_album_asin: Dict[str, str] = {} # artist name → first album ASIN seen
for item in items:
@ -393,14 +384,14 @@ class AmazonClient:
def search_albums(self, query: str, limit: int = 20) -> List[Album]:
_rate_limit()
items = self.search_raw(query, types="track")
items = self.search_raw(query, types="track,album")
album_candidates: List[tuple] = [] # (Album, asin)
seen_keys: set = set()
for item in items:
album_asin = item.album_asin
raw_name = item.album_name
if not raw_name:
if not item.is_album:
continue
album_asin = item.album_asin or item.asin
raw_name = item.album_name or item.title
display_name = _strip_edition(raw_name)
artist = _primary_artist(item.artist_name)
# Collapse Explicit/Clean variants: same normalised name + artist = same album
@ -528,7 +519,7 @@ class AmazonClient:
artist_name = _primary_artist(streams[0].artist)
try:
search_items = self.search_raw(
f"{album_name} {artist_name}", types="track"
f"{album_name} {artist_name}", types="track,album"
)
for item in search_items:
if item.album_asin == asin and item.duration_seconds:
@ -542,12 +533,12 @@ class AmazonClient:
"name": _strip_edition(s.title),
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
"duration_ms": duration_map.get(s.asin, 0),
"track_number": s.track_number if s.track_number is not None else idx + 1,
"track_number": s.track_number,
"disc_number": s.disc_number,
"release_date": s.date or "",
"isrc": s.isrc,
}
for idx, s in enumerate(streams)
for s in streams
]
return {"items": items, "total": len(items), "limit": 50, "next": None}
@ -556,7 +547,7 @@ class AmazonClient:
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(search_name, types="track")
items = self.search_raw(search_name, types="track,album")
except AmazonClientError:
return None
name_lower = search_name.lower()
@ -586,7 +577,7 @@ class AmazonClient:
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(f"{search_name} album", types="track")
items = self.search_raw(f"{search_name} album", types="track,album")
except AmazonClientError:
return []
album_candidates: List[tuple] = [] # (Album, asin)
@ -639,7 +630,7 @@ class AmazonClient:
"""Return an album cover as artist image stand-in (T2Tunes has no artist images)."""
search_name = _unslugify(artist_id)
try:
items = self.search_raw(search_name, types="track")
items = self.search_raw(search_name, types="track,album")
except AmazonClientError:
return None
name_lower = search_name.lower()
@ -695,34 +686,20 @@ class AmazonClient:
def _get_json(self, path: str, params: Optional[Dict[str, Any]] = None) -> Any:
url = urljoin(f"{self.base_url}/", path.lstrip("/"))
last_exc: Optional[Exception] = None
for attempt in range(3):
if attempt:
time.sleep(1.0 * attempt)
try:
resp = self.session.get(url, params=params, timeout=self.timeout)
resp.raise_for_status()
except requests.HTTPError as exc:
body = exc.response.text[:500].replace("\n", " ")
# T2Tunes returns 400 for transient Amazon-side failures — retry those.
if exc.response.status_code == 400 and "Failed to search" in exc.response.text:
logger.debug("T2Tunes transient 400 on attempt %d, retrying: %s", attempt + 1, url)
last_exc = AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}"
)
continue
raise AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}",
status_code=exc.response.status_code,
) from exc
except requests.RequestException as exc:
raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
try:
return resp.json()
except ValueError as exc:
preview = resp.text[:200].replace("\n", " ")
raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
raise last_exc # type: ignore[misc]
try:
resp = self.session.get(url, params=params, timeout=self.timeout)
resp.raise_for_status()
except requests.HTTPError as exc:
raise AmazonClientError(
f"HTTP {exc.response.status_code} for {url}"
) from exc
except requests.RequestException as exc:
raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
try:
return resp.json()
except ValueError as exc:
preview = resp.text[:200].replace("\n", " ")
raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
@staticmethod
def _iter_search_items(response: Any) -> Iterator[T2TunesSearchItem]:

22
core/amazon_download_client.py
View file

@ -32,8 +32,6 @@ from config.settings import config_manager
from core.amazon_client import AmazonClient, AmazonClientError
from core.download_plugins.base import DownloadSourcePlugin
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.quality.model import AudioQuality
from core.quality.source_map import quality_tier_for_source
from utils.logging_config import get_logger
logger = get_logger("amazon_download_client")
@ -78,12 +76,9 @@ class AmazonDownloadClient(DownloadSourcePlugin):
if download_path is None:
download_path = config_manager.get("soulseek.download_path", "./downloads")
self.download_path = Path(download_path)
try:
self.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {self.download_path}: {e}")
self.download_path.mkdir(parents=True, exist_ok=True)
self._quality = quality_tier_for_source("amazon", default="flac")
self._quality = config_manager.get("amazon_download.quality", "flac")
self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
self._client = AmazonClient(preferred_codec=self._quality)
@ -138,17 +133,11 @@ class AmazonDownloadClient(DownloadSourcePlugin):
album_map: Dict[str, AlbumResult] = {}
album_order: List[str] = []
preferred = self._client.preferred_codec
# Search results only carry the codec (real sample_rate arrives at
# stream time). Claim the format honestly — FLAC for the lossless
# codec, lossy otherwise — so audio_quality derives a real format
# instead of the display label ("Lossless"), and the post-download
# probe pins the actual sample_rate/bit_depth.
amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
for item in items:
quality = _quality_label(preferred)
if item.is_track:
tr = TrackResult(
track_results.append(TrackResult(
username="amazon",
filename=f"{item.asin}||{item.artist_name} - {item.title}",
size=0,
@ -166,9 +155,7 @@ class AmazonDownloadClient(DownloadSourcePlugin):
"album_asin": item.album_asin,
"isrc": item.isrc,
},
)
tr.set_quality(amazon_q)
track_results.append(tr)
))
elif item.is_album:
album_asin = item.album_asin or item.asin
if album_asin not in album_map:
@ -186,7 +173,6 @@ class AmazonDownloadClient(DownloadSourcePlugin):
title=item.title,
album=item.album_name,
)
placeholder.set_quality(amazon_q)
album_map[album_asin] = AlbumResult(
username="amazon",
album_path=album_asin,

61
core/amazon_outage.py
View file

@ -1,61 +0,0 @@
"""Amazon enrichment outage detection + back-off — pure, importable, testable.
The Amazon worker enriches via a public T2Tunes proxy instance. When that
instance is down (HTTP 5xx, "Amazon Music API is not initialized", or an
unreachable host), the worker must NOT treat every album as an individual
failure: doing so floods the logs with an error per item, churns network + DB
continuously, and permanently marks the whole library ``error`` (which the
retry tiers never re-attempt) for what is really a transient outage.
Instead it recognizes "the whole source is down", leaves the item untouched so
it's retried once the instance recovers, and backs off hard. These two pure
helpers carry that logic so it can be unit-tested without the worker, the DB,
or the network.
"""
from __future__ import annotations
import re
# HTTP statuses that mean "the source/proxy is unhealthy", not "no match".
_OUTAGE_STATUS = {500, 502, 503, 504}
# Substrings (lower-cased) in an error message that indicate a source outage
# rather than a per-item miss: proxy not ready, gateway errors, the host being
# unreachable, or an error page returned instead of JSON.
_OUTAGE_PHRASES = (
"not initialized", "not configured", "service unavailable",
"bad gateway", "gateway time", "request failed", "response not json",
"max retries", "connection", "timed out", "temporarily unavailable",
)
# Back-off schedule while the source is down.
_NORMAL_DELAY = 2 # seconds between items when healthy
_OUTAGE_BASE = 30 # first back-off step
_OUTAGE_CAP = 1800 # 30 minutes max
def is_source_outage(exc: Exception) -> bool:
"""True when ``exc`` indicates the Amazon source/proxy is down (transient,
whole-source), as opposed to a normal per-item error.
Robust to how the error is surfaced: an explicit ``status_code`` attribute,
an ``HTTP <code>`` prefix in the message, or an outage phrase (covers
connection failures and non-JSON error pages that carry no status code)."""
code = getattr(exc, "status_code", None)
if isinstance(code, int) and code in _OUTAGE_STATUS:
return True
msg = str(exc).lower()
m = re.search(r"http\s+(\d{3})", msg)
if m and int(m.group(1)) in _OUTAGE_STATUS:
return True
return any(p in msg for p in _OUTAGE_PHRASES)
def next_poll_delay_seconds(outage_streak: int) -> int:
"""Seconds to wait before the next item. Normal cadence when healthy;
escalating back-off (30s, 60s, 120s, … capped at 30 min) the longer the
source has been down, so a dead instance can't flood logs/CPU/DB."""
if outage_streak <= 0:
return _NORMAL_DELAY
return min(_OUTAGE_BASE * (2 ** min(outage_streak - 1, 6)), _OUTAGE_CAP)

81
core/amazon_worker.py
View file

@ -9,7 +9,6 @@ from database.music_database import MusicDatabase
from core.amazon_client import AmazonClient
from core.worker_utils import interruptible_sleep, set_album_api_track_count
from core.enrichment.manual_match_honoring import honor_stored_match
from core.amazon_outage import is_source_outage, next_poll_delay_seconds
logger = get_logger("amazon_worker")
@ -20,7 +19,6 @@ class AmazonWorker:
def __init__(self, database: MusicDatabase):
self.db = database
self.client = AmazonClient()
self._amazon_schema_checked = False
self.running = False
self.paused = False
@ -40,45 +38,8 @@ class AmazonWorker:
self.retry_days = 30
self.name_similarity_threshold = 0.80
# Source-outage circuit breaker: counts consecutive whole-source
# failures (proxy down / "not initialized" / unreachable) so the loop
# backs off instead of grinding the whole library item-by-item.
self._outage_streak = 0
logger.info("Amazon background worker initialized")
def _ensure_amazon_schema(self, cursor) -> None:
"""Ensure upgraded installs have the Amazon enrichment columns.
MusicDatabase normally runs this migration during startup, but the
worker should still be defensive because it is the code path that
repeatedly queries these columns in the background.
"""
if self._amazon_schema_checked:
return
table_columns = {
'artists': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'albums': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'tracks': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
}
for table, columns in table_columns.items():
cursor.execute(f"PRAGMA table_info({table})")
existing = {row[1] for row in cursor.fetchall()}
for column in columns:
if column not in existing:
column_type = 'TIMESTAMP' if column == 'amazon_last_attempted' else 'TEXT'
cursor.execute(f"ALTER TABLE {table} ADD COLUMN {column} {column_type}")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_id ON artists (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_status ON artists (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_id ON albums (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_status ON albums (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_id ON tracks (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_status ON tracks (amazon_match_status)")
cursor.connection.commit()
self._amazon_schema_checked = True
def start(self):
if self.running:
logger.warning("Worker already running")
@ -157,9 +118,7 @@ class AmazonWorker:
continue
self._process_item(item)
# Normal 2s cadence when healthy; escalating back-off (up to
# 30 min) while the source is in an outage streak.
interruptible_sleep(self._stop_event, next_poll_delay_seconds(self._outage_streak))
interruptible_sleep(self._stop_event, 2)
except Exception as e:
logger.error(f"Error in worker loop: {e}")
@ -172,17 +131,6 @@ class AmazonWorker:
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('amazon')
if _prio:
_pi = priority_pending_item(cursor, 'amazon', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
@ -293,32 +241,7 @@ class AmazonWorker:
elif item_type == 'track':
self._process_track(item_id, item_name, item.get('artist', ''), item)
# The source answered (match or not_found) — clear any outage streak.
if self._outage_streak:
logger.info("Amazon source recovered after %d outage(s), resuming",
self._outage_streak)
self._outage_streak = 0
except Exception as e:
if is_source_outage(e):
# The whole source is down (proxy 5xx / "not initialized" /
# unreachable). Do NOT mark the item 'error' — that would burn
# the entire library to a state the retry tiers never re-attempt
# for a transient outage. Leave it untouched so it's retried once
# the instance recovers, and let the loop back off. Log once per
# streak to avoid flooding.
self._outage_streak += 1
if self._outage_streak == 1:
logger.warning("Amazon source unavailable — pausing enrichment "
"until it recovers: %s", e)
else:
logger.debug("Amazon source still unavailable (streak=%d): %s",
self._outage_streak, e)
return
# A non-outage error means the source actually answered (e.g. a
# 404/parse error on a real response), so the outage is over —
# clear the streak and handle this as a normal per-item error.
self._outage_streak = 0
logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
self.stats['errors'] += 1
try:
@ -594,7 +517,6 @@ class AmazonWorker:
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
cursor.execute("""
SELECT
(SELECT COUNT(*) FROM artists WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
@ -616,7 +538,6 @@ class AmazonWorker:
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
progress = {}
for table in ('artists', 'albums', 'tracks'):

238
core/archive_pipeline.py
View file

@ -1,238 +0,0 @@
"""Archive extraction + audio-file discovery for torrent / usenet downloads.
The torrent and usenet download plugins need a uniform way to:
1. Walk the downloader's save directory and find every audio file in it.
2. If the directory contains an archive (``.zip`` / ``.rar`` / ``.tar`` /
``.7z``), extract it first so the audio files inside become walkable.
This module is intentionally narrow — no matching, no tagging, no
import. The download plugin layer composes this with the existing
post-processing / matching pipeline. Lidarr does NOT use this module:
Lidarr extracts archives in its own import step before SoulSync sees
the files at all. Usenet downloaders (SABnzbd, NZBGet) also auto-
extract by default. Torrents are the main case where SoulSync may
need to do the extract step itself — most music torrents ship loose,
but some bundle the album in a ``.rar`` archive.
``rarfile`` is an optional dependency. If it isn't installed, archives
with ``.rar`` content are skipped with a single warning rather than
crashing the download.
"""
from __future__ import annotations
import tarfile
import zipfile
from pathlib import Path
from typing import List, Optional
from utils.logging_config import get_logger
logger = get_logger("archive_pipeline")
# Same audio-extension set as ``core/imports/file_ops.py`` ``quality_tiers``.
# Keep them in sync — if a new format is added to file_ops, add it here too
# or the walker will skip it and the download plugin will mark the download
# failed even when files arrived.
AUDIO_EXTENSIONS = frozenset([
# lossless
'.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
# high lossy
'.opus', '.ogg',
# standard lossy
'.m4a', '.aac',
# low lossy
'.mp3', '.wma',
])
ARCHIVE_EXTENSIONS = frozenset(['.zip', '.rar', '.tar', '.tar.gz', '.tgz', '.7z'])
def is_archive(path: Path) -> bool:
"""True if the file extension looks like a supported archive.
Compound extensions (``.tar.gz``, ``.tar.bz2``) are detected by
checking the last two suffixes joined together — Path.suffix
only returns the final suffix.
"""
if not path.is_file():
return False
name = path.name.lower()
if name.endswith(('.tar.gz', '.tar.bz2', '.tar.xz')):
return True
return path.suffix.lower() in ARCHIVE_EXTENSIONS
def walk_audio_files(directory: Path) -> List[Path]:
"""Recursively scan ``directory`` for audio files. Returns
a sorted list of absolute paths. Empty list if the directory
doesn't exist or contains no audio.
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
out: List[Path] = []
for child in directory.rglob('*'):
if not child.is_file():
continue
if child.suffix.lower() in AUDIO_EXTENSIONS:
out.append(child.resolve())
out.sort()
return out
def find_archives_in_dir(directory: Path) -> List[Path]:
"""Find every archive file directly inside ``directory`` (one
level deep — torrents normally put the archive at the root of
their folder; we don't search nested dirs to avoid extracting
something we shouldn't).
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
return sorted(p for p in directory.iterdir() if is_archive(p))
def extract_archive(archive_path: Path, extract_to: Optional[Path] = None) -> Optional[Path]:
"""Extract a single archive in-place (or to ``extract_to`` if
given). Returns the directory the archive was extracted into,
or ``None`` on failure.
Supports ``.zip``, ``.tar``/``.tar.gz``/``.tar.bz2``/``.tar.xz``,
and ``.rar`` (only when the optional ``rarfile`` library is
installed). ``.7z`` is recognised but extraction requires
``py7zr``; without it, the call logs and returns None.
"""
if not archive_path or not archive_path.exists():
logger.warning("archive_pipeline: %s does not exist", archive_path)
return None
dest = extract_to or archive_path.parent
dest.mkdir(parents=True, exist_ok=True)
name = archive_path.name.lower()
try:
if name.endswith('.zip'):
with zipfile.ZipFile(archive_path) as zf:
_safe_extract_zip(zf, dest)
return dest
if name.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
with tarfile.open(archive_path) as tf:
_safe_extract_tar(tf, dest)
return dest
if name.endswith('.rar'):
return _extract_rar(archive_path, dest)
if name.endswith('.7z'):
return _extract_7z(archive_path, dest)
except (zipfile.BadZipFile, tarfile.TarError, OSError) as e:
logger.error("archive_pipeline: failed to extract %s: %s", archive_path, e)
return None
logger.warning("archive_pipeline: unknown archive type for %s", archive_path)
return None
def extract_all_in_dir(directory: Path) -> List[Path]:
"""Find every archive in ``directory`` and extract each in place.
Returns the list of directories archives were extracted into
(usually all the same — ``directory`` itself). Archives that
failed to extract are skipped silently after a warning.
"""
out: List[Path] = []
for archive in find_archives_in_dir(directory):
result = extract_archive(archive)
if result is not None:
out.append(result)
return out
def collect_audio_after_extraction(directory: Path) -> List[Path]:
"""One-shot helper for the download plugins: extract any archives
in the directory, then return the walked audio file list. This is
the common pattern — torrent / usenet plugin gets a save_path,
calls this, hands the resulting files to the matching pipeline.
"""
extract_all_in_dir(directory)
return walk_audio_files(directory)
# ---------------------------------------------------------------------------
# Safety helpers
# ---------------------------------------------------------------------------
def _safe_extract_zip(zf: zipfile.ZipFile, dest: Path) -> None:
"""Extract a zipfile after rejecting any member whose resolved
path escapes ``dest`` (path traversal protection).
"""
dest = dest.resolve()
for member in zf.namelist():
target = (dest / member).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member)
return
zf.extractall(dest)
def _safe_extract_tar(tf: tarfile.TarFile, dest: Path) -> None:
"""Same path-traversal protection for tarfiles."""
dest = dest.resolve()
for member in tf.getmembers():
target = (dest / member.name).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member.name)
return
# ``filter='data'`` is the Python 3.12+ safe extractor; fall back
# to the legacy call on older runtimes.
try:
tf.extractall(dest, filter='data') # type: ignore[call-arg]
except TypeError:
tf.extractall(dest)
def _extract_rar(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import rarfile # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — rarfile library not installed. "
"Install with: pip install rarfile (and ensure unrar is on PATH).",
archive_path,
)
return None
try:
with rarfile.RarFile(archive_path) as rf:
dest_resolved = dest.resolve()
for name in rf.namelist():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal rar member %r", name)
return None
rf.extractall(dest)
return dest
except Exception as e:
logger.error("archive_pipeline: rar extract failed for %s: %s", archive_path, e)
return None
def _extract_7z(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import py7zr # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — py7zr library not installed. "
"Install with: pip install py7zr.",
archive_path,
)
return None
try:
with py7zr.SevenZipFile(archive_path, 'r') as sz:
dest_resolved = dest.resolve()
for name in sz.getnames():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal 7z member %r", name)
return None
sz.extractall(path=dest)
return dest
except Exception as e:
logger.error("archive_pipeline: 7z extract failed for %s: %s", archive_path, e)
return None

23
core/artist_source_detail.py
View file

@ -52,7 +52,7 @@ def build_source_only_artist_detail(
``jsonify`` or equivalent. Status is 200 on success, 404 when the
source's discography lookup returned no releases.
"""
resolved_name = (artist_name or "").strip()
resolved_name = (artist_name or artist_id or "").strip()
# 1. Image URL via the same helper /api/artist/<id>/image uses.
image_url: Optional[str] = None
@ -68,8 +68,6 @@ def build_source_only_artist_detail(
if source == "spotify" and spotify_client is not None:
sp_artist = spotify_client.get_artist(artist_id, allow_fallback=False)
if sp_artist:
if not artist_name and sp_artist.get("name"):
resolved_name = sp_artist["name"]
source_genres = sp_artist.get("genres") or []
source_followers = (sp_artist.get("followers") or {}).get("total")
if not image_url and sp_artist.get("images"):
@ -77,41 +75,22 @@ def build_source_only_artist_detail(
elif source == "deezer" and deezer_client is not None:
dz_artist = deezer_client.get_artist_info(artist_id)
if dz_artist:
if not artist_name and dz_artist.get("name"):
resolved_name = dz_artist["name"]
source_genres = dz_artist.get("genres") or []
source_followers = (dz_artist.get("followers") or {}).get("total")
elif source == "itunes" and itunes_client is not None:
it_artist = itunes_client.get_artist(artist_id)
if it_artist:
if not artist_name and it_artist.get("name"):
resolved_name = it_artist["name"]
source_genres = it_artist.get("genres") or []
elif source == "discogs" and discogs_client is not None:
dc_artist = discogs_client.get_artist(artist_id)
if dc_artist:
if not artist_name and dc_artist.get("name"):
resolved_name = dc_artist["name"]
source_genres = dc_artist.get("genres") or []
elif source == "amazon" and amazon_client is not None:
az_artist = amazon_client.get_artist(resolved_name or artist_id)
if az_artist:
if not artist_name and az_artist.get("name"):
resolved_name = az_artist["name"]
source_genres = az_artist.get("genres") or []
if not image_url and az_artist.get("images"):
image_url = az_artist["images"][0].get("url")
elif source == "musicbrainz":
try:
from core.musicbrainz_search import MusicBrainzSearchClient
mb = MusicBrainzSearchClient()
mb_artist = mb.get_artist(artist_id)
if mb_artist:
if not artist_name and mb_artist.get("name"):
resolved_name = mb_artist["name"]
source_genres = mb_artist.get("genres") or []
except Exception as e:
logger.debug(f"MusicBrainz artist info lookup failed for {artist_id}: {e}")
except Exception as e:
logger.debug(f"Source-side artist info lookup failed for {source}:{artist_id}: {e}")

40
core/artist_source_lookup.py
View file

@ -21,8 +21,6 @@ from __future__ import annotations
import logging
from typing import Optional
from core.source_ids import id_column as _artist_id_column
logger = logging.getLogger("artist_source_lookup")
@ -31,14 +29,14 @@ SOURCE_ONLY_ARTIST_SOURCES = frozenset({
})
# The per-source column on the ``artists`` table, derived from the canonical
# source-ID registry (the single source of truth). Values are unchanged from the
# previous hardcoded map — this just stops duplicating that knowledge here.
SOURCE_ID_FIELD = {
source: _artist_id_column(source, "artist")
for source in (
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
)
"spotify": "spotify_artist_id",
"itunes": "itunes_artist_id",
"deezer": "deezer_id",
"discogs": "discogs_id",
"hydrabase": "soul_id",
"musicbrainz": "musicbrainz_id",
"amazon": "amazon_id",
}
@ -53,11 +51,7 @@ def find_library_artist_for_source(
Lookup order:
1. Direct match on the source-specific ID column (server-agnostic — any
library record with the right external ID is a hit). If that id is
stamped on MORE than one library artist, the mapping is corrupt /
ambiguous (e.g. an enrichment bug wrote one Deezer id onto several
artists) — we refuse to guess and fall through, so the caller can
show the source artist directly instead of an arbitrary wrong one.
library record with the right external ID is a hit).
2. Case-insensitive name match within ``active_server`` (defaults to the
active media server when not provided), so we don't jump the user
across server contexts on a name collision.
@ -71,23 +65,13 @@ def find_library_artist_for_source(
try:
with database._get_connection() as conn:
cursor = conn.cursor()
# LIMIT 2 so we can tell a unique match from an ambiguous one.
cursor.execute(
f"SELECT id FROM artists WHERE {column} = ? LIMIT 2",
f"SELECT id, name FROM artists WHERE {column} = ? LIMIT 1",
(str(source_artist_id),),
)
rows = cursor.fetchall()
if len(rows) == 1:
return rows[0][0]
if len(rows) > 1:
# Same source id on multiple artists — corrupt mapping. Don't
# upgrade on the id; fall through to the name match (and, if
# that misses, let the caller render the source artist).
logger.warning(
f"Source id {source}:{source_artist_id} maps to "
f"{len(rows)}+ library artists — ambiguous, skipping "
f"id-based library upgrade"
)
row = cursor.fetchone()
if row:
return row[0]
if artist_name and active_server:
cursor.execute(

13
core/artists/liked_match.py
View file

@ -12,7 +12,6 @@ from core.metadata.registry import (
get_deezer_client,
get_discogs_client,
get_itunes_client,
get_musicbrainz_client,
get_spotify_client,
)
@ -34,11 +33,6 @@ def _get_discogs_client(token=None):
return get_discogs_client(token)
def _get_musicbrainz_client():
"""Mirror of web_server._get_musicbrainz_client — delegates to registry."""
return get_musicbrainz_client()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily so a Spotify re-auth that
rebinds the cached client in core.metadata.registry is visible to the
@ -71,7 +65,6 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
'itunes': 'itunes_artist_id',
'deezer': 'deezer_artist_id',
'discogs': 'discogs_artist_id',
'musicbrainz': 'musicbrainz_artist_id',
}
id_cols = list(source_cols.values())
@ -110,10 +103,6 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
search_clients['discogs'] = dc
except Exception as e:
logger.debug("discogs client init failed: %s", e)
try:
search_clients['musicbrainz'] = _get_musicbrainz_client()
except Exception as e:
logger.debug("musicbrainz client init failed: %s", e)
# Reuse watchlist scanner's fuzzy matching logic
from core.watchlist_scanner import WatchlistScanner
@ -245,7 +234,7 @@ def _match_liked_artists_to_all_sources(database, profile_id: int):
# Determine best active source/ID — prefer Spotify, then iTunes, Deezer, Discogs
resolved_source = None
resolved_id = None
for src in ('spotify', 'itunes', 'deezer', 'discogs', 'musicbrainz'):
for src in ('spotify', 'itunes', 'deezer', 'discogs'):
col = source_cols[src]
if col in harvested_ids:
resolved_source = src

50
core/artists/map.py
View file

@ -140,7 +140,7 @@ def get_artist_map_data():
placeholders = ','.join(['?'] * len(watchlist_ids))
cursor.execute(f"""
SELECT source_artist_id, similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similar_artist_itunes_id, similar_artist_deezer_id,
similarity_rank, occurrence_count, image_url, genres, popularity
FROM similar_artists
WHERE profile_id = ? AND source_artist_id IN ({placeholders})
@ -173,7 +173,6 @@ def get_artist_map_data():
'spotify_id': r.get('similar_artist_spotify_id') or '',
'itunes_id': r.get('similar_artist_itunes_id') or '',
'deezer_id': r.get('similar_artist_deezer_id') or '',
'musicbrainz_id': r.get('similar_artist_musicbrainz_id') or '',
'rank': r.get('similarity_rank', 5),
'occurrence': r.get('occurrence_count', 1),
'popularity': r.get('popularity', 0),
@ -246,7 +245,7 @@ def get_artist_map_data():
}
# Apply cache data to nodes
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
for n in nodes:
nn = _norm(n['name'])
cached = cache_by_name.get(nn)
@ -263,7 +262,7 @@ def get_artist_map_data():
break
# Backfill genres if missing
if not n.get('genres') or len(n.get('genres', [])) == 0:
for source in ('spotify', 'deezer', 'itunes', 'discogs', 'musicbrainz'):
for source in ('spotify', 'deezer', 'itunes', 'discogs'):
if source in cached and cached[source].get('genres'):
n['genres'] = cached[source]['genres'][:5]
break
@ -370,14 +369,14 @@ def get_artist_map_genres():
def _norm(n):
return (n or '').lower().strip()
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, musicbrainz_id=None, source='unknown', popularity=0):
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, source='unknown', popularity=0):
n = _norm(name)
if not n or len(n) < 2:
return
if n not in artists_by_name:
artists_by_name[n] = {
'name': name, 'image_url': '', 'genres': set(),
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': '',
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '',
'sources': set(), 'popularity': 0
}
a = artists_by_name[n]
@ -395,8 +394,6 @@ def get_artist_map_genres():
a['deezer_id'] = str(deezer_id)
if discogs_id and not a['discogs_id']:
a['discogs_id'] = str(discogs_id)
if musicbrainz_id and not a['musicbrainz_id']:
a['musicbrainz_id'] = str(musicbrainz_id)
if popularity > a['popularity']:
a['popularity'] = popularity
a['sources'].add(source)
@ -413,14 +410,14 @@ def get_artist_map_genres():
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("cache artist genres parse failed: %s", e)
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
kwargs = {src_map.get(r['source'], 'spotify_id'): r['entity_id']}
_add(r['name'], image_url=r['image_url'], genres=genres, source='cache', popularity=r['popularity'] or 0, **kwargs)
# 2. Similar artists
cursor.execute("""
SELECT similar_artist_name, similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id, image_url, genres, popularity
similar_artist_deezer_id, image_url, genres, popularity
FROM similar_artists WHERE profile_id = ?
""", (profile_id,))
for r in cursor.fetchall():
@ -432,9 +429,7 @@ def get_artist_map_genres():
logger.debug("similar artist genres parse failed: %s", e)
_add(r['similar_artist_name'], image_url=r['image_url'], genres=genres,
spotify_id=r['similar_artist_spotify_id'], itunes_id=r['similar_artist_itunes_id'],
deezer_id=r['similar_artist_deezer_id'],
musicbrainz_id=r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else None,
source='similar', popularity=r['popularity'] or 0)
deezer_id=r['similar_artist_deezer_id'], source='similar', popularity=r['popularity'] or 0)
# 3. Watchlist artists
cursor.execute("""
@ -488,7 +483,6 @@ def get_artist_map_genres():
'itunes_id': a['itunes_id'],
'deezer_id': a['deezer_id'],
'discogs_id': a['discogs_id'],
'musicbrainz_id': a['musicbrainz_id'],
'popularity': a['popularity'],
'type': 'watchlist' if 'watchlist' in a['sources'] else 'similar',
})
@ -632,7 +626,7 @@ def get_artist_map_explore():
# Find the center artist
center_name = artist_name
center_image = ''
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': ''}
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': ''}
center_genres = []
# Search metadata cache for the center artist
@ -654,7 +648,7 @@ def get_artist_map_explore():
center_name = row['name']
if row['image_url'] and row['image_url'].startswith('http'):
center_image = row['image_url']
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
k = src_map.get(row['source'], 'spotify_id')
center_ids[k] = row['entity_id']
if row['genres']:
@ -665,14 +659,14 @@ def get_artist_map_explore():
# Check watchlist + library if not in cache
if not artist_found and not artist_id:
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id, musicbrainz_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
wr = cursor.fetchone()
if wr:
artist_found = True
center_name = wr['artist_name']
if wr['image_url'] and str(wr['image_url']).startswith('http'):
center_image = wr['image_url']
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id'), ('musicbrainz_id', 'musicbrainz_artist_id')]:
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id')]:
if wr[col]:
center_ids[k] = str(wr[col])
else:
@ -723,7 +717,7 @@ def get_artist_map_explore():
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
""", (center_name,))
for r in cursor.fetchall():
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
k = src_map.get(r['source'], 'spotify_id')
if not center_ids.get(k):
center_ids[k] = r['entity_id']
@ -752,7 +746,7 @@ def get_artist_map_explore():
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similar_artist_itunes_id, similar_artist_deezer_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
@ -763,7 +757,7 @@ def get_artist_map_explore():
# Also search by name (the center artist might be a watchlist source)
cursor.execute("""
SELECT DISTINCT sa.similar_artist_name, sa.similar_artist_spotify_id,
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id, sa.similar_artist_musicbrainz_id,
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id,
sa.image_url, sa.genres, sa.popularity, sa.similarity_rank
FROM similar_artists sa
JOIN watchlist_artists wa ON sa.source_artist_id = COALESCE(wa.spotify_artist_id, wa.itunes_artist_id, CAST(wa.id AS TEXT))
@ -795,8 +789,7 @@ def get_artist_map_explore():
image_url=sa.get('image_url'),
genres=sa.get('genres'),
popularity=sa.get('popularity', 0),
similar_artist_deezer_id=sa.get('deezer_id'),
similar_artist_musicbrainz_id=sa.get('musicbrainz_id'),
similar_artist_deezer_id=sa.get('deezer_id')
)
except Exception as e:
logger.debug("similar artist insert failed: %s", e)
@ -805,7 +798,7 @@ def get_artist_map_explore():
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similar_artist_itunes_id, similar_artist_deezer_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
@ -816,7 +809,7 @@ def get_artist_map_explore():
# Fallback: query by name-based source ID
cursor.execute("""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similar_artist_itunes_id, similar_artist_deezer_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id = ? AND profile_id = ?
@ -848,7 +841,6 @@ def get_artist_map_explore():
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
@ -869,8 +861,7 @@ def get_artist_map_explore():
cursor.execute(f"""
SELECT DISTINCT source_artist_id, similar_artist_name,
similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
similar_artist_deezer_id, image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
@ -911,7 +902,6 @@ def get_artist_map_explore():
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
@ -945,7 +935,7 @@ def get_artist_map_explore():
except Exception as e:
logger.debug("explorer node genres parse failed: %s", e)
# Harvest missing IDs from cache
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id'}
k = src_map.get(cr['source'])
if k and not n.get(k):
n[k] = cr['entity_id']

39
core/audiodb_worker.py
View file

@ -8,7 +8,7 @@ from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.audiodb_client import AudioDBClient
from core.worker_utils import accept_artist_match, interruptible_sleep
from core.worker_utils import interruptible_sleep
logger = get_logger("audiodb_worker")
@ -162,16 +162,6 @@ class AudioDBWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('audiodb')
if _prio:
_pi = priority_pending_item(cursor, 'audiodb', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name
@ -273,13 +263,8 @@ class AudioDBWorker:
def _verify_artist_id(self, item: Dict[str, Any], result: Dict[str, Any]) -> bool:
"""Verify that the result's artist ID matches the parent artist's stored AudioDB ID.
If mismatched, the album/track search is more specific (uses artist+title),
so we trust it and correct the parent artist's audiodb_id — BUT only when
the result's artist *name* matches our parent artist. Without that guard,
a collaboration/compilation (a track our library credits to one artist
that lives on another artist's album) would stamp the wrong AudioDB id
onto our artist. See the Deezer fix for the full write-up."""
so we trust it and correct the parent artist's audiodb_id."""
parent_audiodb_id = item.get('artist_audiodb_id')
if not parent_audiodb_id:
return True
@ -289,18 +274,6 @@ class AudioDBWorker:
return True
if str(result_artist_id) != str(parent_audiodb_id):
parent_name = item.get('artist') or ''
result_artist_name = result.get('strArtist') or ''
if (result_artist_name and parent_name
and not self._name_matches(parent_name, result_artist_name)):
logger.info(
f"Skipping artist-ID correction from {item['type']} "
f"'{item['name']}': result artist '{result_artist_name}' "
f"≠ parent '{parent_name}' (collab/compilation, not a "
f"correction)"
)
return True
logger.info(
f"Artist ID correction from {item['type']} '{item['name']}': "
f"updating parent artist AudioDB ID from {parent_audiodb_id} to {result_artist_id}"
@ -426,18 +399,14 @@ class AudioDBWorker:
result = self.client.search_artist(item_name)
if result:
result_name = result.get('strArtist', '')
ok, reason = accept_artist_match(
self.db, 'audiodb_id', result.get('idArtist'), item_id,
item_name, result_name,
)
if ok:
if self._name_matches(item_name, result_name):
self._update_artist(item_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{item_name}' -> AudioDB ID: {result.get('idArtist')}")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Artist '{item_name}' not matched: {reason}")
logger.debug(f"Name mismatch for artist '{item_name}' (got '{result_name}')")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1

146
core/auto_import_worker.py
View file

@ -21,7 +21,6 @@ from datetime import datetime
from difflib import SequenceMatcher
from typing import Any, Callable, Dict, List, Optional
from core.imports.folder_artist import resolve_folder_artist
from utils.logging_config import get_logger
logger = get_logger("auto_import")
@ -660,13 +659,8 @@ class AutoImportWorker:
auto_process = self._config_manager.get('auto_import.auto_process', True)
try:
# Phase 3: Identify.
# Re-identify (#889): if the user designated this exact file's release in
# the Re-identify modal, a hint short-circuits the guessing — we match
# straight against the chosen album. No hint → byte-identical to before.
rematch_hint, identification = self._resolve_rematch_hint(candidate)
if identification is None:
identification = self._identify_folder(candidate)
# Phase 3: Identify
identification = self._identify_folder(candidate)
if not identification:
self._record_result(candidate, 'needs_identification', 0.0,
error_message='Could not identify album from tags, folder name, or fingerprint')
@ -695,10 +689,7 @@ class AutoImportWorker:
high_conf_matches = [m for m in match_result.get('matches', []) if m['confidence'] >= 0.8]
has_strong_individual_matches = len(high_conf_matches) > 0
# A re-identify is an explicit user choice — let it auto-process like a
# strong match (still gated on the global auto_process preference).
if (confidence >= threshold or has_strong_individual_matches
or rematch_hint is not None) and auto_process:
if (confidence >= threshold or has_strong_individual_matches) and auto_process:
# Phase 5: Auto-process — insert an in-progress row
# so the UI sees the import the moment it starts,
# then update it with the final status when done.
@ -717,13 +708,6 @@ class AutoImportWorker:
confidence = max(confidence, effective_conf)
if success:
self._bump_stat('auto_processed')
# Re-identify (#889): only NOW that the new home exists do we
# consume the hint and (if replace was chosen) delete the old
# row + file — so a failed import never loses the original. Pass
# the landing paths so we never delete a file the re-import landed
# at the SAME place (picking the release it's already in).
if rematch_hint is not None:
self._finalize_rematch_hint(rematch_hint, getattr(candidate, '_reid_final_paths', None))
else:
self._bump_stat('failed')
@ -1018,75 +1002,6 @@ class AutoImportWorker:
except Exception:
return False
# ── Re-identify hints (#889) ──
def _resolve_rematch_hint(self, candidate: 'FolderCandidate'):
"""If this staged file carries a user-designated re-identify hint, return
``(hint, identification)`` so matching skips the guessing tiers; otherwise
``(None, None)`` and the caller falls back to normal identification.
Fail-safe: ANY error (no table, DB hiccup) returns ``(None, None)`` so a
re-identify problem can never break ordinary auto-import. Only single-file
candidates are eligible — a re-identify always stages exactly one track."""
try:
files = candidate.audio_files or []
if len(files) != 1:
return None, None
from core.imports.rematch_hints import (
build_identification_from_hint,
find_hint_for_file,
quick_file_signature,
)
file_path = files[0]
sig = quick_file_signature(file_path)
conn = self.database._get_connection()
try:
cursor = conn.cursor()
hint = find_hint_for_file(cursor, file_path, sig)
finally:
conn.close()
if hint is None:
return None, None
logger.info("[Auto-Import] Re-identify hint for %s → %s '%s' (%s)",
candidate.name, hint.album_type or 'release',
hint.album_name or '?', hint.source)
return hint, build_identification_from_hint(hint)
except Exception as e:
logger.debug("[Auto-Import] rematch-hint lookup skipped: %s", e)
return None, None
def _finalize_rematch_hint(self, hint, new_paths=None) -> None:
"""Post-success: delete the replaced library row + file (if the user chose
replace) and consume the hint so it's single-use. ``new_paths`` are where the
re-import landed — passed through so the same-home guard never deletes a file
the import wrote at the old location. Best-effort — a cleanup failure is
logged, never raised, since the re-import already succeeded."""
try:
from core.imports.rematch_hints import consume_hint, delete_replaced_track
def _resolve_old(stored):
# The old row's path is a STORED path (Docker/media-server view) — map
# it to a file this process can actually unlink, same as everywhere else.
try:
from core.library.path_resolver import resolve_library_file_path
return resolve_library_file_path(stored, config_manager=getattr(self, '_config_manager', None))
except Exception:
return None
conn = self.database._get_connection()
try:
cursor = conn.cursor()
removed = delete_replaced_track(cursor, hint.replace_track_id,
resolve_fn=_resolve_old, new_paths=new_paths)
consume_hint(cursor, hint.id)
conn.commit()
finally:
conn.close()
if removed:
logger.info("[Auto-Import] Re-identify replaced old track — removed %s", removed)
except Exception as e:
logger.warning("[Auto-Import] rematch-hint finalize failed (import still OK): %s", e)
# ── Identification ──
def _identify_folder(self, candidate: FolderCandidate) -> Optional[Dict]:
@ -1515,11 +1430,8 @@ class AutoImportWorker:
def _match_tracks(self, candidate: FolderCandidate, identification: Dict) -> Optional[Dict]:
"""Match staging files to the identified album's tracklist."""
# Singles: no album tracklist to match against — the file IS the match.
# force_album_match (set by a re-identify hint) overrides this: even a lone
# staged file is matched INTO the chosen album, so it inherits the album's
# year / track number / art instead of the bare singles stub (#889).
if not identification.get('force_album_match') and (candidate.is_single or identification.get('is_single')):
# Singles: no album tracklist to match against — the file IS the match
if candidate.is_single or identification.get('is_single'):
conf = identification.get('identification_confidence', 0.7)
track_data = {
'name': identification.get('track_name', identification.get('album_name', '')),
@ -1664,18 +1576,31 @@ class AutoImportWorker:
album_name = identification.get('album_name', 'Unknown')
image_url = identification.get('image_url', '')
# Parent folder artist override via import.folder_artist_override.
# Default on to preserve the legacy Artist/Album staging behavior.
# Users who stage mixed piles under one container folder can turn it off
# to keep the metadata-identified artist.
# Parent folder artist override: if the staging folder structure is
# Artist/Albums/AlbumName or Artist/AlbumName, use the parent folder
# as the artist name when the tag-extracted artist looks wrong.
# This handles mixtapes/compilations where embedded tags have DJ names.
try:
if self._config_manager.get('import.folder_artist_override', True):
staging_root = self._resolve_staging_path() or self.staging_path
rel_path = os.path.relpath(candidate.path, staging_root)
folder_artist = resolve_folder_artist(rel_path, artist_name, enabled=True)
if folder_artist:
logger.info(f"[Auto-Import] Parent folder artist '{folder_artist}' differs from tag artist '{artist_name}' — using folder artist")
artist_name = folder_artist
staging_root = self._resolve_staging_path() or self.staging_path
rel_path = os.path.relpath(candidate.path, staging_root)
parts = [p for p in rel_path.replace('\\', '/').split('/') if p]
# parts[0] = artist folder, parts[1] = album or category subfolder, etc.
# Only attempt override if there's at least 2 levels (artist/album)
folder_artist = None
if len(parts) >= 2:
_category_names = {'albums', 'singles', 'eps', 'compilations', 'mixtapes',
'discography', 'music', 'downloads'}
if len(parts) >= 3 and parts[1].lower() in _category_names:
# Artist/Albums/AlbumFolder → parts[0] is artist
folder_artist = parts[0]
elif parts[0].lower() not in _category_names:
# Artist/AlbumFolder → parts[0] is artist
folder_artist = parts[0]
if folder_artist and folder_artist.lower() != artist_name.lower():
logger.info(f"[Auto-Import] Parent folder artist '{folder_artist}' differs from tag artist '{artist_name}' — using folder artist")
artist_name = folder_artist
except Exception as e:
logger.debug("folder artist override failed: %s", e)
release_date = identification.get('release_date', '') or album_data.get('release_date', '')
@ -1687,7 +1612,6 @@ class AutoImportWorker:
processed = 0
errors = []
reid_final_paths = [] # #889: where the pipeline landed each file (same-home guard)
all_matches = list(match_result.get('matches', []))
# Album total duration — sum of every matched track's duration.
@ -1868,11 +1792,6 @@ class AutoImportWorker:
self._process_callback(context_key, context, file_path)
processed += 1
# Capture where the pipeline actually landed the file (#889 same-home
# guard) — the pipeline writes it back into the mutable context.
_landed = context.get('_final_processed_path')
if _landed:
reid_final_paths.append(_landed)
logger.info(f"[Auto-Import] Processed: {track_number}. {track_name}")
except Exception as e:
@ -1896,13 +1815,6 @@ class AutoImportWorker:
except Exception as e:
logger.debug("automation emit failed: %s", e)
# Stash landing paths on the candidate so _finalize_rematch_hint can avoid
# deleting a file the re-import landed at the SAME place (#889).
try:
candidate._reid_final_paths = reid_final_paths
except Exception as e:
logger.debug("could not stash reid final paths: %s", e)
return processed > 0
# ── Database ──

10
core/automation/api.py
View file

@ -172,11 +172,9 @@ def create_automation(
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
group_name = data.get('group_name') or None
owned_by = data.get('owned_by') or None
auto_id = database.create_automation(
name, trigger_type, trigger_config, action_type, action_config,
profile_id, notify_type, notify_config, then_actions_json, group_name,
owned_by=owned_by,
)
if auto_id is None:
return {'error': 'Failed to create automation'}, 500
@ -219,8 +217,6 @@ def update_automation(
update_fields['notify_config'] = json.dumps(data['notify_config'])
if 'group_name' in data:
update_fields['group_name'] = data['group_name'] or None
if 'owned_by' in data:
update_fields['owned_by'] = data['owned_by'] or None
if not update_fields:
return {'error': 'No fields to update'}, 400
@ -229,12 +225,6 @@ def update_automation(
if cycle_path:
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
# Schedule-shape changes must invalidate the stored next_run so the
# scheduler recomputes it; otherwise restart-survival logic keeps the
# leftover timestamp from the previous interval.
if {'trigger_type', 'trigger_config'} & update_fields.keys():
update_fields['next_run'] = None
success = database.update_automation(automation_id, **update_fields)
if not success:
return {'error': 'Automation not found'}, 404

7
core/automation/blocks.py
View file

@ -171,7 +171,12 @@ ACTIONS: list[dict] = [
{"type": "update_discovery_pool", "label": "Update Discovery", "icon": "compass",
"description": "Refresh discovery pool with new tracks", "available": True},
{"type": "start_quality_scan", "label": "Run Quality Scan", "icon": "bar-chart",
"description": "Run the Quality Upgrade Finder (scope is set in Library Maintenance)", "available": True},
"description": "Scan for low-quality audio files", "available": True,
"config_fields": [
{"key": "scope", "type": "select", "label": "Scope",
"options": [{"value": "watchlist", "label": "Watchlist Artists"}, {"value": "library", "label": "Full Library"}],
"default": "watchlist"}
]},
{"type": "backup_database", "label": "Backup Database", "icon": "save",
"description": "Create timestamped database backup", "available": True},
{"type": "refresh_beatport_cache", "label": "Refresh Beatport Cache", "icon": "music",

27
core/automation/deps.py
View file

@ -28,7 +28,7 @@ from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, Optional
from typing import Any, Callable, Optional
@dataclass
@ -56,14 +56,6 @@ class AutomationState:
with self.lock:
return self.pipeline_running
def try_start_pipeline(self) -> bool:
"""Atomically mark the shared playlist pipeline as running."""
with self.lock:
if self.pipeline_running:
return False
self.pipeline_running = True
return True
def set_scan_library_id(self, automation_id: Optional[str]) -> None:
with self.lock:
self.scan_library_automation_id = automation_id
@ -105,8 +97,6 @@ class AutomationDeps:
# --- Playlist pipeline entry points ---
run_playlist_discovery_worker: Callable[..., Any]
run_sync_task: Callable[..., Any]
run_playlist_organize_download: Callable[..., Dict[str, Any]]
missing_download_executor: Any
load_sync_status_file: Callable[[], dict]
get_deezer_client: Callable[[], Any]
parse_youtube_playlist: Callable[[str], Any]
@ -123,10 +113,10 @@ class AutomationDeps:
duplicate_cleaner_lock: Any
duplicate_cleaner_executor: Any
run_duplicate_cleaner: Callable[..., Any]
# Triggers a "Run Now" of a library-maintenance repair job by id (e.g.
# 'quality_upgrade'). Returns truthy if the job was queued. Replaces the old
# standalone quality-scanner executor/state (the scanner is now a repair job).
run_repair_job_now: Callable[[str], Any]
get_quality_scanner_state: Callable[[], dict]
quality_scanner_lock: Any
quality_scanner_executor: Any
run_quality_scanner: Callable[..., Any]
# --- Download orchestrator + queue accessors ---
download_orchestrator: Any
@ -154,10 +144,3 @@ class AutomationDeps:
# PersonalizedPlaylistManager per run (cheap accessors inside,
# no caching needed yet).
build_personalized_manager: Callable[[], Any]
# --- Unified PlaylistSource registry ---
# Optional so test fixtures that don't exercise refresh_mirrored
# can keep their existing scaffolding. Production wiring in
# ``web_server.py`` always populates it via
# ``core.playlists.sources.bootstrap.build_playlist_source_registry``.
playlist_source_registry: Optional[Any] = None

60
core/automation/handlers/_pipeline_shared.py
View file

@ -80,11 +80,9 @@ def run_sync_and_wishlist(
if sync_status == 'started':
sync_id = sync_id_for_fn(pl)
sync_poll_start = time.time()
timed_out = True
while time.time() - sync_poll_start < _SYNC_PER_PLAYLIST_TIMEOUT_SECONDS:
if (sync_id in sync_states
and sync_states[sync_id].get('status') in _SYNC_TERMINAL_STATUSES):
timed_out = False
break
time.sleep(2)
elapsed = int(time.time() - sync_poll_start)
@ -96,25 +94,6 @@ def run_sync_and_wishlist(
)
ss = sync_states.get(sync_id, {})
final_status = ss.get('status')
if timed_out:
sync_errors += 1
deps.update_progress(
automation_id,
log_line=f'Sync timed out "{pl_name}" after {_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS}s',
log_type='error',
)
continue
if final_status in ('error', 'failed'):
sync_errors += 1
reason = ss.get('error') or ss.get('reason') or 'background sync failed'
deps.update_progress(
automation_id,
log_line=f'Sync error "{pl_name}": {reason}',
log_type='error',
)
continue
ss_result = ss.get('result', ss.get('progress', {}))
matched = ss_result.get('matched_tracks', 0) if isinstance(ss_result, dict) else 0
total_synced += int(matched) if matched else 0
@ -148,45 +127,9 @@ def run_sync_and_wishlist(
log_type='success' if sync_errors == 0 else 'warning',
)
organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
organize_started = 0
if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
for pl in organize_playlists:
pl_id = pl.get('id')
if not pl_id:
continue
pl_name = pl.get('name', '')
try:
org_result = deps.run_playlist_organize_download(
mirrored_playlist_id=int(pl_id),
automation_id=automation_id,
)
if org_result.get('status') == 'started':
organize_started += 1
deps.update_progress(
automation_id,
log_line=f'Organize download started for "{pl_name}"',
log_type='success',
)
elif org_result.get('status') == 'skipped':
deps.update_progress(
automation_id,
log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
log_type='skip',
)
except Exception as org_err: # noqa: BLE001
deps.update_progress(
automation_id,
log_line=f'Organize download error for "{pl_name}": {org_err}',
log_type='warning',
)
all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
effective_skip_wishlist = skip_wishlist or all_organize
wishlist_queued = run_wishlist_phase(
deps, automation_id,
skip=effective_skip_wishlist,
skip=skip_wishlist,
progress_pct=progress_end + 1,
wishlist_phase_label=wishlist_phase_label,
wishlist_phase_start_log=wishlist_phase_start_log,
@ -197,7 +140,6 @@ def run_sync_and_wishlist(
'skipped': total_skipped,
'errors': sync_errors,
'wishlist_queued': wishlist_queued,
'organize_downloads_started': organize_started,
}

95
core/automation/handlers/database_update.py
View file

@ -21,49 +21,12 @@ from typing import Any, Dict
from core.automation.deps import AutomationDeps
# Time out on STALL (no progress), not total runtime: a large library can scan
# for many hours while progressing fine — a hard total cap would falsely mark a
# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
# up when progress hasn't moved for a long stretch, with a generous absolute
# backstop against a truly stuck monitor loop.
_STALL_WARNING_SECONDS = 600 # warn after 10 min with no progress (repeats)
_STALL_TIMEOUT_SECONDS = 1800 # 30 min with no progress at all = genuinely stalled
_ABSOLUTE_CAP_SECONDS = 86400 # 24h hard backstop (runaway-loop guard only)
_TIMEOUT_SECONDS = 7200 # 2 hours — covers the worst large-library case
_STALL_WARNING_SECONDS = 600 # 10 minutes without progress = stall
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def scan_wait_action(
*,
status: str,
idle_seconds: float,
total_seconds: float,
stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
stall_warn_s: float = _STALL_WARNING_SECONDS,
abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
) -> str:
"""Decide what the monitor loop should do on a poll tick (pure/testable).
``idle_seconds`` is time since progress last changed; ``total_seconds`` is
time since the wait began. Returns one of:
``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
too long → give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
(stalled long enough to warn but not give up), or ``'continue'``.
Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
it never hits ``stall_timeout`` no matter how long the whole scan takes.
"""
if status != 'running':
return 'finished'
if total_seconds >= abs_cap_s:
return 'abs_timeout'
if idle_seconds >= stall_timeout_s:
return 'stall_timeout'
if idle_seconds >= stall_warn_s:
return 'warn'
return 'continue'
def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a full or incremental DB update via ``run_db_update_task``."""
return _run_with_progress(
@ -73,7 +36,7 @@ def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) ->
initial_phase='Initializing...',
stall_label='Database update',
finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
timeout_label='Database update timed out after 24 hours',
timeout_label='Database update timed out after 2 hours',
)
@ -86,7 +49,7 @@ def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict
initial_phase='Deep scan: Initializing...',
stall_label='Deep scan',
finished_extras=lambda: {},
timeout_label='Deep scan timed out after 24 hours',
timeout_label='Deep scan timed out after 2 hours',
)
@ -120,54 +83,30 @@ def _run_with_progress(
deps.db_update_executor.submit(task, *task_args)
# Monitor progress (callbacks handle card updates, we just block until done).
# We time out on STALL, not total runtime: ``processed`` advances on every
# artist, so an actively-progressing scan keeps resetting the idle clock and
# is never falsely failed no matter how long the whole library takes.
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
last_progress_time = time.time()
# Any of these advancing means the scan is alive. current_item (the artist
# being processed) changes every artist even when the rounded progress %
# holds steady, so it guards against a false stall during slow stretches.
last_progress_val = (0, 0, '')
last_warn_time = 0.0
outcome = 'finished'
while True:
last_progress_val = 0
while time.time() - poll_start < _TIMEOUT_SECONDS:
time.sleep(_POLL_INTERVAL_SECONDS)
now = time.time()
with deps.db_update_lock:
current_status = state.get('status', 'idle')
current_val = (state.get('processed', 0), state.get('progress', 0),
state.get('current_item', ''))
if current_val != last_progress_val:
last_progress_val = current_val
last_progress_time = now
action = scan_wait_action(
status=current_status,
idle_seconds=now - last_progress_time,
total_seconds=now - poll_start,
)
if action in ('finished', 'stall_timeout', 'abs_timeout'):
outcome = action
current_progress = state.get('progress', 0)
if current_status != 'running':
break
if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
idle_min = int((now - last_progress_time) / 60)
# Stall detection — if no progress change in 10 minutes, warn.
if current_progress != last_progress_val:
last_progress_val = current_progress
last_progress_time = time.time()
elif time.time() - last_progress_time > _STALL_WARNING_SECONDS:
deps.update_progress(
automation_id,
log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
log_line=f'{stall_label} appears stalled — waiting...',
log_type='warning',
)
last_warn_time = now
if outcome == 'stall_timeout':
deps.update_progress(
automation_id, status='error', phase='Stalled',
log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
log_type='error',
)
return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
if outcome == 'abs_timeout':
last_progress_time = time.time() # Reset so warning repeats every 10 min.
else:
# 2-hour timeout reached.
deps.update_progress(
automation_id, status='error',
phase='Timed out', log_line=timeout_label, log_type='error',

28
core/automation/handlers/maintenance.py
View file

@ -115,27 +115,19 @@ def auto_backup_database(config: Dict[str, Any], deps: AutomationDeps) -> Dict[s
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
backup_path = f"{db_path}.backup_{timestamp}"
# safe_backup verifies source + result integrity, so an automated backup
# can never silently snapshot a corrupt DB (the incident where every
# rolling backup faithfully copied the corruption).
from core.db_integrity import DBIntegrityError, safe_backup, prune_backups
try:
safe_backup(db_path, backup_path)
except DBIntegrityError as integ:
deps.logger.error("Auto-backup refused — DB integrity check failed: %s", integ)
deps.update_progress(
automation_id,
log_line=f'Backup SKIPPED — database failed integrity check: {integ}',
log_type='error',
)
return {'status': 'error', 'reason': f'Database integrity check failed: {integ}'}
# Use SQLite backup API for a safe hot-copy of an active database.
src = sqlite3.connect(db_path)
dst = sqlite3.connect(backup_path)
src.backup(dst)
dst.close()
src.close()
size_mb = round(os.path.getsize(backup_path) / (1024 * 1024), 1)
# Rolling cleanup — never evict the most-recent verified-healthy backup.
existing = list(_glob.glob(f"{db_path}.backup_*"))
for removed in prune_backups(existing, _MAX_BACKUPS):
# Rolling cleanup — keep only the newest N backups.
existing = sorted(_glob.glob(f"{db_path}.backup_*"), key=os.path.getmtime)
while len(existing) > _MAX_BACKUPS:
try:
os.remove(removed)
os.remove(existing.pop(0))
except Exception as e: # noqa: BLE001 — best-effort cleanup
deps.logger.debug("rolling backup cleanup failed: %s", e)
deps.update_progress(

226
core/automation/handlers/playlist_pipeline.py
View file

@ -1,27 +1,227 @@
"""Automation adapter for the mirrored playlist pipeline.
"""Automation handler: ``playlist_pipeline`` action.
The actual all-in-one playlist lifecycle lives in
``core.playlists.pipeline`` so it can be reused by non-automation UI actions.
This module only wires automation-specific dependencies and handlers.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_playlist_pipeline`` closure). Runs the full playlist
lifecycle in a single trigger:
Phase 1: REFRESH -- pull fresh track lists from sources
Phase 2: DISCOVER -- look up official Spotify/iTunes metadata
Phase 3: SYNC -- push the result to the active media server
Phase 4: WISHLIST -- queue any missing tracks for download
Each phase emits its own progress range so the trigger card shows
useful per-phase percentages instead of "loading...". Phase 4 is
optional via ``skip_wishlist`` config.
Composition: this handler invokes ``auto_refresh_mirrored`` and
``auto_sync_playlist`` directly (passing ``_automation_id: None`` so
the sub-handlers don't hijack pipeline progress) instead of going
through the engine — keeps the four phases observable as one
trigger from the user's perspective. Pipeline-level guard
(``state.pipeline_running``) prevents overlapping runs.
"""
from __future__ import annotations
import threading
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.playlists.pipeline import run_mirrored_playlist_pipeline
# Per-playlist sync poll cap inside Phase 3.
# Discovery poll cap inside Phase 2.
_DISCOVERY_TIMEOUT_SECONDS = 3600
def auto_playlist_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run REFRESH -> DISCOVER -> SYNC -> WISHLIST for mirrored playlists."""
return run_mirrored_playlist_pipeline(
config,
deps,
refresh_fn=auto_refresh_mirrored,
sync_one_fn=auto_sync_playlist,
sync_and_wishlist_fn=run_sync_and_wishlist,
)
"""Run REFRESH → DISCOVER → SYNC → WISHLIST in sequence.
Sets / clears ``deps.state.pipeline_running`` around the whole
run so the registration guard can short-circuit overlapping
triggers.
"""
deps.state.set_pipeline_running(True)
automation_id = config.get('_automation_id')
pipeline_start = time.time()
try:
db = deps.get_database()
playlist_id = config.get('playlist_id')
process_all = config.get('all', False)
skip_wishlist = config.get('skip_wishlist', False)
# Resolve playlists.
if process_all:
playlists = db.get_mirrored_playlists()
elif playlist_id:
p = db.get_mirrored_playlist(int(playlist_id))
playlists = [p] if p else []
else:
deps.state.set_pipeline_running(False)
return {'status': 'error', 'error': 'No playlist specified'}
playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
if not playlists:
deps.state.set_pipeline_running(False)
return {'status': 'error', 'error': 'No refreshable playlists found'}
pl_names = ', '.join(p.get('name', '?') for p in playlists[:3])
if len(playlists) > 3:
pl_names += f' (+{len(playlists) - 3} more)'
deps.update_progress(
automation_id,
progress=2,
phase=f'Pipeline: {len(playlists)} playlist(s)',
log_line=f'Starting pipeline for: {pl_names}',
log_type='info',
)
# ── PHASE 1: REFRESH ──────────────────────────────────────────
deps.update_progress(
automation_id,
progress=3,
phase='Phase 1/4: Refreshing playlists...',
log_line='Phase 1: Refresh',
log_type='info',
)
refresh_config = dict(config)
refresh_config['_automation_id'] = None # Don't let sub-handler hijack pipeline progress.
refresh_result = auto_refresh_mirrored(refresh_config, deps)
refreshed = int(refresh_result.get('refreshed', 0))
refresh_errors = int(refresh_result.get('errors', 0))
deps.update_progress(
automation_id,
progress=25,
phase='Phase 1/4: Refresh complete',
log_line=f'Phase 1 done: {refreshed} refreshed, {refresh_errors} errors',
log_type='success' if refresh_errors == 0 else 'warning',
)
# ── PHASE 2: DISCOVER ─────────────────────────────────────────
deps.update_progress(
automation_id,
progress=26,
phase='Phase 2/4: Discovering metadata...',
log_line='Phase 2: Discover',
log_type='info',
)
# Reload playlists (refresh may have updated them).
if process_all:
disc_playlists = db.get_mirrored_playlists()
else:
disc_playlists = [db.get_mirrored_playlist(int(playlist_id))]
disc_playlists = [p for p in disc_playlists if p]
# Run discovery in a thread and wait for it.
disc_done = threading.Event()
def _disc_wrapper(pls):
try:
# The worker updates automation_progress internally,
# but we pass None so it doesn't conflict with our
# pipeline progress.
deps.run_playlist_discovery_worker(pls, automation_id=None)
except Exception as e:
deps.logger.error(f"[Pipeline] Discovery error: {e}")
finally:
disc_done.set()
threading.Thread(
target=_disc_wrapper, args=(disc_playlists,),
daemon=True, name='pipeline-discover',
).start()
# Poll for completion with progress updates.
poll_start = time.time()
while not disc_done.wait(timeout=3):
elapsed = int(time.time() - poll_start)
deps.update_progress(
automation_id,
progress=min(26 + elapsed // 4, 54),
phase=f'Phase 2/4: Discovering... ({elapsed}s)',
)
if elapsed > _DISCOVERY_TIMEOUT_SECONDS:
deps.update_progress(
automation_id,
log_line='Discovery timed out after 1 hour',
log_type='warning',
)
break
deps.update_progress(
automation_id,
progress=55,
phase='Phase 2/4: Discovery complete',
log_line='Phase 2 done: discovery complete',
log_type='success',
)
# ── PHASE 3 + 4: SYNC + WISHLIST (delegated to shared helper) ──
# Each mirrored playlist payload only needs `id` + `name` for
# the helper; `auto_sync_playlist` reads the rest from the
# mirrored DB by id.
sync_summary = run_sync_and_wishlist(
deps,
automation_id,
[pl for pl in playlists if pl.get('id')],
sync_one_fn=lambda pl: auto_sync_playlist(
{'playlist_id': str(pl['id']), '_automation_id': None},
deps,
),
sync_id_for_fn=lambda pl: f"auto_mirror_{pl['id']}",
skip_wishlist=skip_wishlist,
progress_start=56,
progress_end=85,
sync_phase_label='Phase 3/4: Syncing to server...',
sync_phase_start_log='Phase 3: Sync',
wishlist_phase_label='Phase 4/4: Processing wishlist...',
wishlist_phase_start_log='Phase 4: Wishlist',
)
total_synced = sync_summary['synced']
total_skipped = sync_summary['skipped']
sync_errors = sync_summary['errors']
wishlist_queued = sync_summary['wishlist_queued']
# ── COMPLETE ──────────────────────────────────────────────────
duration = int(time.time() - pipeline_start)
deps.update_progress(
automation_id,
status='finished',
progress=100,
phase='Pipeline complete',
log_line=f'Pipeline finished in {duration // 60}m {duration % 60}s',
log_type='success',
)
deps.state.set_pipeline_running(False)
return {
'status': 'completed',
'_manages_own_progress': True,
'playlists_refreshed': str(refreshed),
'tracks_discovered': 'completed',
'tracks_synced': str(total_synced),
'sync_skipped': str(total_skipped),
'wishlist_queued': str(wishlist_queued),
'duration_seconds': str(duration),
}
except Exception as e:
deps.state.set_pipeline_running(False)
deps.update_progress(
automation_id,
status='error',
progress=100,
phase='Pipeline error',
log_line=f'Pipeline failed: {e}',
log_type='error',
)
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}

76
core/automation/handlers/quality_scanner.py
View file

@ -1,35 +1,83 @@
"""Automation handler: ``start_quality_scan`` action.
The quality scanner was redesigned from an auto-acting tool into the
``quality_upgrade`` library-maintenance repair job (findings-based, reviewed
before anything is wishlisted). This action now simply triggers a "Run Now" of
that job; its progress and findings surface in Library Maintenance. The action
name is kept so existing automation rules keep working.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_start_quality_scan`` closure). Submits the quality scanner
to its executor with the configured scope (default: ``watchlist``)
then polls the shared state dict.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
_TIMEOUT_SECONDS = 7200 # 2 hours
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def auto_start_quality_scan(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
automation_id = config.get('_automation_id')
state = deps.get_quality_scanner_state()
if state.get('status') == 'running':
return {'status': 'skipped', 'reason': 'Quality scan already running'}
triggered = deps.run_repair_job_now('quality_upgrade')
if not triggered:
scope = config.get('scope', 'watchlist')
# Pre-set status before submit so the polling loop doesn't see a
# stale 'finished' from a previous run.
with deps.quality_scanner_lock:
state['status'] = 'running'
deps.quality_scanner_executor.submit(deps.run_quality_scanner, scope, deps.get_current_profile_id())
deps.update_progress(
automation_id, log_line=f'Quality scan started (scope: {scope})', log_type='info',
)
# Monitor progress (max 2 hours).
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
while time.time() - poll_start < _TIMEOUT_SECONDS:
time.sleep(_POLL_INTERVAL_SECONDS)
current_status = state.get('status', 'idle')
if current_status not in ('running',):
break
deps.update_progress(
automation_id, status='error', phase='Unavailable',
log_line='Quality Upgrade job could not be triggered (library worker unavailable)',
automation_id,
phase=state.get('phase', 'Scanning...'),
progress=state.get('progress', 0),
processed=state.get('processed', 0),
total=state.get('total', 0),
)
else:
deps.update_progress(
automation_id, status='error',
phase='Timed out', log_line='Quality scan timed out after 2 hours',
log_type='error',
)
return {'status': 'error', 'reason': 'library worker unavailable',
'_manages_own_progress': True}
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
final_status = state.get('status', 'idle')
if final_status == 'error':
err = state.get('error_message', 'Unknown error')
deps.update_progress(
automation_id, status='error', progress=100,
phase='Error', log_line=err, log_type='error',
)
return {'status': 'error', 'reason': err, '_manages_own_progress': True}
issues = state.get('low_quality', 0)
deps.update_progress(
automation_id, status='finished', progress=100, phase='Triggered',
log_line='Quality Upgrade scan queued — findings appear in Library Maintenance',
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Quality scan complete — {issues} issues found',
log_type='success',
)
return {'status': 'completed', 'triggered': True, '_manages_own_progress': True}
return {
'status': 'completed', 'scope': scope, '_manages_own_progress': True,
'tracks_scanned': state.get('processed', 0),
'quality_met': state.get('quality_met', 0),
'low_quality': issues,
'matched': state.get('matched', 0),
}

537
core/automation/handlers/refresh_mirrored.py
View file

@ -1,43 +1,23 @@
"""Automation handler: ``refresh_mirrored`` action.
Re-pulls track lists from each mirrored playlist's source via the
unified ``PlaylistSourceRegistry`` (Phase 1 of the Discover-to-Sync
unification). The pre-extraction handler had ~190 lines of per-source
if/elif branches; this version delegates to the adapter for each
source, leaving the handler responsible only for:
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_refresh_mirrored`` closure). Re-pulls track lists from each
mirrored playlist's source (Spotify / Tidal / Deezer / YouTube),
updates the local mirror DB, and emits a ``playlist_changed``
automation event when the track set actually shifts.
- filtering sources that can't be refreshed (``file``, ``beatport``),
- extracting upstream URLs from the stored ``description`` for URL-
backed sources (``spotify_public``, ``youtube``),
- the Spotify-public → authenticated-Spotify fallback (uses the
``spotify`` adapter when the user is signed in so the mirror keeps
album art),
- the Tidal-not-authenticated skip log type (vs error),
- preserving existing per-track ``extra_data`` on tracks that survive
the refresh, and
- emitting the ``playlist_changed`` automation event when the track
set actually shifts.
Source-specific branches (Spotify auth + public-embed fallback,
``spotify_public`` URL→ID resolution, Deezer / Tidal / YouTube)
remain identical to the pre-extraction closure — this is a
mechanical lift, not a redesign.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
import json
from typing import Any, Dict
from core.automation.deps import AutomationDeps
from core.playlists.source_refs import require_refresh_url
from core.playlists.sources import PlaylistDetail, to_mirror_track_dict
from core.playlists.sources.base import (
SOURCE_SPOTIFY,
SOURCE_SPOTIFY_PUBLIC,
SOURCE_TIDAL,
SOURCE_YOUTUBE,
)
# Sources that store the upstream URL in ``description`` (because their
# ``source_playlist_id`` is a deterministic hash, not the native ID).
# The refresh path has to recover the URL before calling the adapter.
_URL_BACKED_SOURCES = {SOURCE_SPOTIFY_PUBLIC, SOURCE_YOUTUBE}
def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
@ -51,16 +31,6 @@ def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[
playlist_id = config.get('playlist_id')
refresh_all = config.get('all', False)
auto_id = config.get('_automation_id')
# Pipeline runs (``run_mirrored_playlist_pipeline``) set this flag
# because Phase 2 of the pipeline already runs the playlist
# discovery worker — the same matching engine ``_maybe_discover``
# would call here. Running both means LB tracks discover twice
# AND the refresh-side discovery blocks 5+ minutes with no
# progress emission, leaving the UI stuck on "Refreshing:" until
# the loop returns. Standalone callers (Sync page, registration
# action) leave it False so LB tracks still get matched_data on
# refresh.
skip_discovery = bool(config.get('skip_discovery', False))
if refresh_all:
playlists = db.get_mirrored_playlists()
@ -74,7 +44,7 @@ def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[
playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
refreshed = 0
errors: List[str] = []
errors = []
for idx, pl in enumerate(playlists):
try:
source = pl.get('source', '')
@ -85,39 +55,249 @@ def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[
phase=f'Refreshing: "{pl.get("name", "")}"',
current_item=pl.get('name', ''),
)
tracks = None
detail = _fetch_detail(source, source_id, pl, deps, auto_id)
if detail is None:
# _fetch_detail already logged the specific failure;
# mark the playlist as a generic refresh error so the
# automation result tally matches the legacy handler.
errors.append(f"{pl.get('name', '?')}: no tracks returned from source")
deps.update_progress(
auto_id,
log_line=f'Refresh failed: "{pl.get("name", "")}" - no tracks returned from source',
log_type='error',
if source == 'spotify':
# Try authenticated API first, fall back to public embed scraper.
if deps.spotify_client and deps.spotify_client.is_spotify_authenticated():
playlist_obj = deps.spotify_client.get_playlist_by_id(source_id)
if playlist_obj and playlist_obj.tracks:
tracks = []
for t in playlist_obj.tracks:
artist_name = t.artists[0] if t.artists else ''
track_dict = {
'track_name': t.name or '',
'artist_name': str(artist_name),
'album_name': t.album or '',
'duration_ms': t.duration_ms or 0,
'source_track_id': t.id or '',
}
# Spotify data IS official — auto-mark as discovered.
if t.id:
_album_obj = {'name': t.album or ''}
if getattr(t, 'image_url', None):
_album_obj['images'] = [{'url': t.image_url, 'height': 600, 'width': 600}]
track_dict['extra_data'] = json.dumps({
'discovered': True,
'provider': 'spotify',
'confidence': 1.0,
'matched_data': {
'id': t.id,
'name': t.name or '',
'artists': [{'name': str(a)} for a in (t.artists or [])],
'album': _album_obj,
'duration_ms': t.duration_ms or 0,
'image_url': getattr(t, 'image_url', None),
}
})
tracks.append(track_dict)
# Fallback: public embed scraper (no auth needed).
if tracks is None:
try:
from core.spotify_public_scraper import scrape_spotify_embed
embed_data = scrape_spotify_embed('playlist', source_id)
if embed_data and not embed_data.get('error') and embed_data.get('tracks'):
embed_album = embed_data.get('name', '') if embed_data.get('type') == 'album' else ''
tracks = []
for t in embed_data['tracks']:
artist_names = [a['name'] for a in t.get('artists', [])]
artist_name = artist_names[0] if artist_names else ''
track_dict = {
'track_name': t.get('name', ''),
'artist_name': artist_name,
'album_name': embed_album,
'duration_ms': t.get('duration_ms', 0),
'source_track_id': t.get('id', ''),
}
# Store Spotify track ID hint but don't mark discovered —
# Discover step needs to run for proper album art.
if t.get('id'):
track_dict['extra_data'] = json.dumps({
'discovered': False,
'spotify_hint': {
'id': t['id'],
'name': t.get('name', ''),
'artists': t.get('artists', []),
}
})
tracks.append(track_dict)
except Exception as e:
deps.logger.warning(f"Spotify public scraper fallback failed for {source_id}: {e}")
elif source == 'spotify_public':
# source_playlist_id is an MD5 hash; extract actual Spotify ID from stored description (URL).
try:
from core.spotify_public_scraper import parse_spotify_url, scrape_spotify_embed
spotify_url = pl.get('description', '')
parsed = parse_spotify_url(spotify_url) if spotify_url else None
# If Spotify is authenticated, use the full API (auto-discovers with album art).
if (parsed and parsed.get('type') == 'playlist'
and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()):
playlist_obj = deps.spotify_client.get_playlist_by_id(parsed['id'])
if playlist_obj and playlist_obj.tracks:
tracks = []
for t in playlist_obj.tracks:
artist_name = t.artists[0] if t.artists else ''
track_dict = {
'track_name': t.name or '',
'artist_name': str(artist_name),
'album_name': t.album or '',
'duration_ms': t.duration_ms or 0,
'source_track_id': t.id or '',
}
if t.id:
_album_obj = {'name': t.album or ''}
if getattr(t, 'image_url', None):
_album_obj['images'] = [{'url': t.image_url, 'height': 600, 'width': 600}]
track_dict['extra_data'] = json.dumps({
'discovered': True,
'provider': 'spotify',
'confidence': 1.0,
'matched_data': {
'id': t.id,
'name': t.name or '',
'artists': [{'name': str(a)} for a in (t.artists or [])],
'album': _album_obj,
'duration_ms': t.duration_ms or 0,
'image_url': getattr(t, 'image_url', None),
}
})
tracks.append(track_dict)
# Fallback: public embed scraper (no auth or album-type URL).
if tracks is None and parsed:
embed_data = scrape_spotify_embed(parsed['type'], parsed['id'])
if embed_data and not embed_data.get('error') and embed_data.get('tracks'):
embed_album = embed_data.get('name', '') if embed_data.get('type') == 'album' else ''
tracks = []
for t in embed_data['tracks']:
artist_names = [a['name'] for a in t.get('artists', [])]
artist_name = artist_names[0] if artist_names else ''
tracks.append({
'track_name': t.get('name', ''),
'artist_name': artist_name,
'album_name': embed_album,
'duration_ms': t.get('duration_ms', 0),
'source_track_id': t.get('id', ''),
})
# No extra_data — let preservation code keep existing discovery data.
except Exception as e:
deps.logger.warning(f"Spotify public playlist refresh failed for {source_id}: {e}")
elif source == 'deezer':
try:
deezer = deps.get_deezer_client()
playlist_data = deezer.get_playlist(source_id)
if playlist_data and playlist_data.get('tracks'):
tracks = []
for t in playlist_data['tracks']:
artist_name = t['artists'][0] if t.get('artists') else ''
tracks.append({
'track_name': t.get('name', ''),
'artist_name': str(artist_name),
'album_name': t.get('album', ''),
'duration_ms': t.get('duration_ms', 0),
'source_track_id': str(t.get('id', '')),
})
except Exception as e:
deps.logger.warning(f"Deezer playlist refresh failed for {source_id}: {e}")
elif source == 'tidal':
if not deps.tidal_client or not deps.tidal_client.is_authenticated():
deps.logger.warning(f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'")
deps.update_progress(
auto_id,
log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
log_type='skip',
)
continue
full_playlist = deps.tidal_client.get_playlist(source_id)
if full_playlist and full_playlist.tracks:
tracks = []
for t in full_playlist.tracks:
artist_name = t.artists[0] if t.artists else ''
tracks.append({
'track_name': t.name or '',
'artist_name': str(artist_name),
'album_name': t.album or '',
'duration_ms': t.duration_ms or 0,
'source_track_id': t.id or '',
})
elif source == 'youtube':
# source_playlist_id is now a deterministic hash; use stored description (original URL) for refresh.
yt_url = pl.get('description', '') or f"https://www.youtube.com/playlist?list={source_id}"
playlist_data = deps.parse_youtube_playlist(yt_url)
if playlist_data and playlist_data.get('tracks'):
tracks = []
for t in playlist_data['tracks']:
artist_name = t['artists'][0] if t.get('artists') else ''
tracks.append({
'track_name': t.get('name', ''),
'artist_name': str(artist_name),
'album_name': '',
'duration_ms': t.get('duration_ms', 0),
'source_track_id': t.get('id', ''),
})
if tracks is not None:
# Compare old vs new track IDs to detect changes.
old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
# Preserve existing discovery extra_data for tracks that still exist.
old_extra_map = db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
for t in tracks:
sid = t.get('source_track_id', '')
if sid and sid in old_extra_map and 'extra_data' not in t:
t['extra_data'] = old_extra_map[sid]
db.mirror_playlist(
source=source,
source_playlist_id=source_id,
name=pl['name'],
tracks=tracks,
profile_id=pl.get('profile_id', 1),
owner=pl.get('owner'),
image_url=pl.get('image_url'),
)
continue
refreshed += 1
# Sources that return MB-metadata-only tracks (LB, Last.fm)
# mark them ``needs_discovery=True``. Hand them to the
# adapter's matcher so the resulting mirror rows carry
# provider IDs + matched_data, ready for the sync pipeline.
#
# Pipeline runs skip this because Phase 2's discovery
# worker handles it with proper progress emission — see
# ``skip_discovery`` resolution at the top of this fn.
detail_tracks = (
detail.tracks if skip_discovery
else _maybe_discover(detail.tracks, source, deps)
)
tracks = [to_mirror_track_dict(t) for t in detail_tracks]
refreshed += _commit_refresh(pl, source, source_id, tracks, db, deps, auto_id)
except _SkipPlaylist:
# Source-specific soft-skip (e.g. Tidal not authenticated).
# Logging was already emitted; do not count as error.
continue
# Emit playlist_changed if tracks actually changed.
if old_ids != new_ids:
added_count = len(new_ids - old_ids)
removed_count = len(old_ids - new_ids)
deps.logger.info(
f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
f"{added_count} added, {removed_count} removed (old={len(old_ids)}, new={len(new_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'"{pl.get("name", "")}" — {added_count} added, {removed_count} removed',
log_type='success',
)
try:
if deps.engine:
deps.engine.emit('playlist_changed', {
'playlist_name': pl.get('name', ''),
'playlist_id': str(pl.get('id', '')),
'old_count': str(len(old_ids)),
'new_count': str(len(new_ids)),
'added': str(added_count),
'removed': str(removed_count),
})
except Exception as e:
deps.logger.debug("playlist_synced automation emit failed: %s", e)
else:
deps.logger.warning(f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})")
deps.update_progress(
auto_id,
log_line=f'No changes: "{pl.get("name", "")}"',
log_type='skip',
)
except Exception as e:
errors.append(f"{pl.get('name', '?')}: {str(e)}")
deps.update_progress(
@ -126,216 +306,3 @@ def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[
log_type='error',
)
return {'status': 'completed', 'refreshed': str(refreshed), 'errors': str(len(errors))}
def _maybe_discover(
tracks: List[Any],
source: str,
deps: AutomationDeps,
) -> List[Any]:
"""Run the adapter's ``discover_tracks`` when any track needs it.
Most sources are no-ops here (their tracks have provider IDs
already). LB / Last.fm are the ones that actually do work."""
if not tracks:
return tracks
if not any(getattr(t, "needs_discovery", False) for t in tracks):
return tracks
registry = deps.playlist_source_registry
if registry is None:
return tracks
adapter = registry.get_source(source)
if adapter is None:
return tracks
try:
return adapter.discover_tracks(tracks)
except Exception as exc:
deps.logger.warning(f"{source} discover_tracks failed: {exc}")
return tracks
class _SkipPlaylist(Exception):
"""Internal sentinel: source-specific soft-skip (e.g. not authed).
The per-playlist loop catches it specifically so the skip isn't
counted in the error tally — matches the pre-extraction behavior
where ``continue`` was used inline."""
def _fetch_detail(
source: str,
source_id: str,
pl: Dict[str, Any],
deps: AutomationDeps,
auto_id: Optional[str],
) -> Optional[PlaylistDetail]:
"""Resolve the playlist's tracks through the registry.
Handler-level branches (URL extraction, Spotify-public→authed
fallback, Tidal not-authed skip) live here; everything else
delegates to the adapter."""
registry = deps.playlist_source_registry
if registry is None:
return None
# URL-backed sources: pull the upstream URL out of `description`.
playlist_input = source_id
if source in _URL_BACKED_SOURCES:
# ``require_refresh_url`` raises ValueError on missing URL.
# The outer try/except in the loop catches it and reports as
# an error — matching the pre-extraction behavior.
playlist_input = require_refresh_url(
source, pl.get('description', ''), pl.get('name', '')
)
# Spotify-public refresh: prefer the authenticated Spotify API
# when the user is signed in. Better album art, matches the
# pre-extraction handler. Falls through to the public scraper on
# auth failure or non-playlist URL types (e.g. album URLs).
if source == SOURCE_SPOTIFY_PUBLIC:
detail = _try_spotify_authed_for_public(playlist_input, deps)
if detail is not None:
return detail
# Tidal not-authed: soft-skip with a 'skip' log line, not an error.
if source == SOURCE_TIDAL:
tidal_source = registry.get_source(SOURCE_TIDAL)
if tidal_source is None or not tidal_source.is_authenticated():
deps.logger.warning(
f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'"
)
deps.update_progress(
auto_id,
log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
log_type='skip',
)
raise _SkipPlaylist
adapter = registry.get_source(source)
if adapter is None:
return None
try:
return adapter.refresh_playlist(playlist_input)
except Exception as exc:
deps.logger.warning(
f"{source} playlist refresh failed for {playlist_input}: {exc}"
)
return None
def _try_spotify_authed_for_public(
spotify_url: str, deps: AutomationDeps
) -> Optional[PlaylistDetail]:
"""Best-effort: use the authenticated Spotify adapter on a public URL.
Returns ``None`` to signal "fall through to the public-scraper
adapter" — never raises. Only applies to ``playlist``-type URLs;
album URLs fall through unconditionally."""
if not spotify_url:
return None
spotify_client = deps.spotify_client
if spotify_client is None or not spotify_client.is_spotify_authenticated():
return None
try:
from core.spotify_public_scraper import parse_spotify_url
parsed = parse_spotify_url(spotify_url)
except Exception:
return None
if not parsed or parsed.get('type') != 'playlist':
return None
adapter = deps.playlist_source_registry.get_source(SOURCE_SPOTIFY)
if adapter is None:
return None
try:
return adapter.refresh_playlist(parsed['id'])
except Exception as exc:
deps.logger.debug(f"Spotify authed fallback for public mirror failed: {exc}")
return None
def _commit_refresh(
pl: Dict[str, Any],
source: str,
source_id: str,
tracks: List[Dict[str, Any]],
db: Any,
deps: AutomationDeps,
auto_id: Optional[str],
) -> int:
"""Persist the refreshed track list + emit playlist_changed when delta.
Returns 1 when a refresh successfully landed, 0 otherwise. The
caller is responsible for incrementing the running tally."""
old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
# Preserve existing extra_data (matched_data + discovery state)
# for tracks that still exist in the refreshed snapshot, unless
# the adapter already provided fresh extra_data for that track.
old_extra_map = (
db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
)
for t in tracks:
sid = t.get('source_track_id', '')
if sid and sid in old_extra_map and 'extra_data' not in t:
t['extra_data'] = old_extra_map[sid]
db.mirror_playlist(
source=source,
source_playlist_id=source_id,
name=pl['name'],
tracks=tracks,
profile_id=pl.get('profile_id', 1),
description=pl.get('description'),
owner=pl.get('owner'),
image_url=pl.get('image_url'),
)
# Membership just changed — if this playlist is organize-by-playlist, rebuild
# its folder (with prune) so a track that LEFT the playlist has its symlink
# cleaned up now. Gated to organized playlists, non-fatal — never disturbs
# the refresh. (Additions are handled by the post-download reconcile.)
try:
from core.playlists.materialize_service import rebuild_mirrored_playlist_if_organized
rebuild_mirrored_playlist_if_organized(
db, deps.config_manager, pl.get('id'), profile_id=pl.get('profile_id', 1)
)
except Exception as _mat_err:
deps.logger.debug(f"[Playlist Folder] mirror-refresh cleanup skipped: {_mat_err}")
if old_ids != new_ids:
added = len(new_ids - old_ids)
removed = len(old_ids - new_ids)
deps.logger.info(
f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
f"{added} added, {removed} removed (old={len(old_ids)}, new={len(new_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'"{pl.get("name", "")}" — {added} added, {removed} removed',
log_type='success',
)
try:
if deps.engine:
deps.engine.emit('playlist_changed', {
'playlist_name': pl.get('name', ''),
'playlist_id': str(pl.get('id', '')),
'old_count': str(len(old_ids)),
'new_count': str(len(new_ids)),
'added': str(added),
'removed': str(removed),
})
except Exception as e:
deps.logger.debug("playlist_synced automation emit failed: %s", e)
else:
deps.logger.warning(
f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'No changes: "{pl.get("name", "")}"',
log_type='skip',
)
return 1

2
core/automation/handlers/registration.py
View file

@ -136,7 +136,7 @@ def register_all(deps: AutomationDeps) -> None:
engine.register_action_handler(
'start_quality_scan',
lambda config: auto_start_quality_scan(config, deps),
lambda: False, # repair worker dedupes Run-Now requests itself
lambda: deps.get_quality_scanner_state().get('status') == 'running',
)
engine.register_action_handler(
'backup_database',

53
core/automation/handlers/sync_playlist.py
View file

@ -145,36 +145,13 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
sync_id_key = f"auto_mirror_{playlist_id}"
# Full mirror identity (every source_track_id on the playlist). tracks_hash
# only covers tracks_json — if a new mirror row is skipped (no discovery /
# no source id), tracks_hash stays identical to the pre-add sync and we
# used to no-op with "unchanged" while the new song never hit wishlist.
mirror_ids_str = ','.join(
sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
)
mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
event_data = config.get('_event_data') or {}
try:
tracks_added = int(event_data.get('added') or 0)
except (TypeError, ValueError):
tracks_added = 0
force_sync = tracks_added > 0 or skipped_count > 0
try:
sync_statuses = deps.load_sync_status_file()
last_status = sync_statuses.get(sync_id_key, {})
last_hash = last_status.get('tracks_hash', '')
last_mirror_hash = last_status.get('mirror_tracks_hash', '')
last_matched = last_status.get('matched_tracks', -1)
mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
if (
not force_sync
and not mirror_changed
and last_hash == tracks_hash
and last_matched >= len(tracks_json)
):
if last_hash == tracks_hash and last_matched >= len(tracks_json):
# Exact same tracks, all matched last time — nothing to do.
deps.update_progress(
auto_id,
@ -185,33 +162,13 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
'status': 'skipped',
'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
}
if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
deps.update_progress(
auto_id,
log_line=(
f'Forcing sync: playlist changed ({tracks_added} added) or '
f'{skipped_count} track(s) need discovery'
),
log_type='info',
)
elif mirror_changed:
deps.update_progress(
auto_id,
log_line='Mirror track list changed — running sync',
log_type='info',
)
except Exception as e:
deps.logger.debug("mirror sync last-status read: %s", e)
# Sync under the user's custom alias when set, else the upstream name (#865
# follow-up). The server-side playlist is named with this.
from core.playlists.naming import effective_mirrored_name
sync_name = effective_mirrored_name(pl) or pl.get('name') or 'Playlist'
deps.update_progress(
auto_id,
progress=50,
phase=f'Syncing "{sync_name}"',
phase=f'Syncing "{pl["name"]}"',
log_line=f'{len(tracks_json)} discovered, {skipped_count} skipped',
log_type='info',
)
@ -223,17 +180,15 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
log_line=f'Starting sync: {len(tracks_json)} tracks',
log_type='success',
)
skip_wishlist_add = bool(pl.get('organize_by_playlist'))
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, sync_name, tracks_json, auto_id, 1, pl.get('image_url', '')),
kwargs={'skip_wishlist_add': skip_wishlist_add},
args=(sync_id, pl['name'], tracks_json, auto_id, 1, pl.get('image_url', '')),
daemon=True,
name=f'auto-sync-{playlist_id}',
).start()
return {
'status': 'started',
'playlist_name': sync_name,
'playlist_name': pl['name'],
'discovered_tracks': str(len(tracks_json)),
'skipped_tracks': str(skipped_count),
'_manages_own_progress': True,

321
core/automation/schedule.py
View file

@ -1,321 +0,0 @@
"""Pure functions for computing the next-run datetime of a scheduled
automation trigger.
The Auto-Sync schedule board currently exposes interval-based scheduling
(``every N hours``) backed by ``trigger_type='schedule'``. The
automation engine ALSO supports ``daily_time`` and ``weekly_time``
triggers via separate ``_setup_*_trigger`` methods inline on the engine
class. None of that logic is currently testable in isolation — the
engine's ``_finish_run`` reaches for ``datetime.now()``, threads it
through ``_next_weekly_occurrence``, and writes the result to the DB,
all on the same call.
This module lifts the "given a trigger config, what's the next run?"
question out of the engine into a pure function:
next_run_at(trigger_type, trigger_config, now_utc, default_tz)
-> Optional[datetime]
That means:
- ``now_utc`` is INJECTED, not pulled from the system clock. Tests
freeze time without monkeypatching ``datetime.now``.
- ``default_tz`` is INJECTED. Daily / weekly / monthly schedules are
inherently in the USER'S timezone (cron "every Monday at 9am" is
not UTC), and the historic engine implicitly used the server's
local tz via naive ``datetime.now()``. That broke for users on a
different tz than their server. The pure function takes the tz
explicitly so the caller controls it.
- Returns an aware UTC ``datetime`` ready to serialise to the DB's
``next_run`` string column, or ``None`` for unrecognised /
event-based triggers (engine should not store a next_run for those).
PR 1 of the schedule-types feature ships ONLY this module + tests.
The engine continues to compute next_run via its existing inline
helpers; PR 2 collapses those into a single ``next_run_at`` call.
Net behavior is identical until the engine is wired through — this
PR is pure plumbing.
Schedule types supported here:
- ``schedule`` (interval): ``{interval: N, unit: 'minutes'|'hours'|'days'}``
— adds the interval to ``now_utc``; no tz needed.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — runs every day at
the given local time in the given timezone. ``tz`` falls back to
``default_tz`` when absent.
- ``weekly_time``: ``{time: 'HH:MM', days: ['mon','wed',...], tz: '<IANA>'}``
— runs on the matching weekday(s) at the given local time. Empty
``days`` list means "every day" (matches the engine's existing
fallback in ``_next_weekly_occurrence``).
- ``monthly_time``: ``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}``
— runs on the given day each month. Days that don't exist in a
given month (Feb 30, Apr 31) clamp to the LAST valid day of that
month rather than skipping the run entirely; missing a whole
month silently because the schedule was over-eager is worse than
running a day early.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, Optional
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from utils.logging_config import get_logger
logger = get_logger("automation.schedule")
# Unknown-tz names already warned about in this process — avoids
# spamming the log on every poll cycle for the same misconfigured row.
_UNKNOWN_TZ_WARNED: set = set()
# Weekday abbreviation → ``datetime.weekday()`` index (Mon=0..Sun=6).
# Mirrors the engine's existing ``_next_weekly_occurrence`` mapping so
# schedules created against either implementation accept the same
# ``days`` strings.
_WEEKDAY_MAP = {
'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6,
}
# Interval multipliers — kept aligned with the engine's existing
# ``_calc_delay_seconds`` in ``core/automation_engine.py``. Adding
# entries here without also updating the engine would silently drift:
# this function would honour the new unit while the live engine path
# defaults it to hours. Keep the maps in sync until PR 2 collapses the
# engine through this function.
_INTERVAL_MULTIPLIERS = {
'minutes': 60,
'hours': 60 * 60,
'days': 60 * 60 * 24,
}
def next_run_at(
trigger_type: str,
trigger_config: Dict[str, Any],
now_utc: datetime,
default_tz: str = 'UTC',
) -> Optional[datetime]:
"""Compute the next-run timestamp (UTC, aware) for a scheduled
trigger. Returns ``None`` for unrecognised types or event-based
triggers — callers should not write a next_run for those.
See module docstring for supported trigger types + config shapes.
"""
if not isinstance(trigger_config, dict):
trigger_config = {}
if trigger_type == 'schedule':
return _next_interval(trigger_config, now_utc)
if trigger_type == 'daily_time':
return _next_daily(trigger_config, now_utc, default_tz)
if trigger_type == 'weekly_time':
return _next_weekly(trigger_config, now_utc, default_tz)
if trigger_type == 'monthly_time':
return _next_monthly(trigger_config, now_utc, default_tz)
return None
# ---------------------------------------------------------------------------
# Interval
# ---------------------------------------------------------------------------
def _next_interval(config: Dict[str, Any], now_utc: datetime) -> datetime:
"""``{interval: N, unit: 'hours'}`` → ``now_utc + N hours``.
Mirrors the engine's existing ``_calc_delay_seconds``. Unit defaults
to ``hours`` for backward compat with legacy DB rows that pre-date
the unit field being mandatory; interval defaults to 1 so a fully
empty config doesn't divide-by-zero or schedule for the past."""
try:
interval = max(int(config.get('interval', 1)), 1)
except (TypeError, ValueError):
interval = 1
unit = config.get('unit') or 'hours'
seconds = interval * _INTERVAL_MULTIPLIERS.get(unit, _INTERVAL_MULTIPLIERS['hours'])
return _ensure_utc(now_utc) + timedelta(seconds=seconds)
# ---------------------------------------------------------------------------
# Daily
# ---------------------------------------------------------------------------
def _next_daily(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', tz: '<IANA>'}`` → next occurrence of that
wall-clock time in the user's timezone, expressed as aware UTC.
DST-aware via ``zoneinfo``: when the local time falls during a
spring-forward gap, the ``replace`` lands on a non-existent
instant; ``zoneinfo`` resolves that to the gap's later side
(e.g. 02:30 on the DST-forward day becomes 03:30 local). Tests
pin both spring-forward and fall-back behaviour."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
now_local = _ensure_utc(now_utc).astimezone(tz)
target_local = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target_local <= now_local:
target_local = target_local + timedelta(days=1)
return target_local.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Weekly
# ---------------------------------------------------------------------------
def _next_weekly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', days: ['mon',...], tz: '<IANA>'}`` → next
occurrence of that wall-clock time on any of the listed weekdays
in the user's timezone.
Empty ``days`` list ≡ every day, matching the engine's existing
fallback. Unrecognised day abbreviations are silently dropped
(an empty result-set then triggers the every-day fallback)."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
days = _parse_weekdays(config.get('days'))
now_local = _ensure_utc(now_utc).astimezone(tz)
# Scan today + next 7 days; the matching day with a future
# local time wins. 8-day scan is enough to handle the case where
# today already passed the time AND today is the only allowed
# weekday (next occurrence is exactly one week out).
for offset in range(8):
candidate = now_local + timedelta(days=offset)
if candidate.weekday() not in days:
continue
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target > now_local:
return target.astimezone(timezone.utc)
# Shouldn't reach: 8-day scan always finds a hit when ``days``
# is non-empty. Defensive fallback: next week, same weekday as today.
fallback = (now_local + timedelta(days=7)).replace(
hour=hour, minute=minute, second=0, microsecond=0,
)
return fallback.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Monthly
# ---------------------------------------------------------------------------
def _next_monthly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}`` → next
occurrence in the user's timezone.
``day_of_month`` is clamped to ``[1, 31]``. When the target day
doesn't exist in a given month (Feb 30, Apr 31), the schedule
falls back to the LAST valid day of that month — running a day
or two early in short months is less surprising than skipping
a month entirely. This matches the convention every cron
implementation in the wild settled on."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
raw_day = config.get('day_of_month', 1)
try:
target_day = max(1, min(31, int(raw_day)))
except (TypeError, ValueError):
target_day = 1
now_local = _ensure_utc(now_utc).astimezone(tz)
# Try this month first; if the target day has already passed
# (or doesn't exist this month and the clamped day is in the
# past), advance to next month. Loop bounded to 12 iterations
# so a pathologically broken config can't infinite-loop us.
year, month = now_local.year, now_local.month
for _ in range(12):
day = min(target_day, _days_in_month(year, month))
target = now_local.replace(
year=year, month=month, day=day,
hour=hour, minute=minute, second=0, microsecond=0,
)
if target > now_local:
return target.astimezone(timezone.utc)
# Roll to next month.
if month == 12:
year, month = year + 1, 1
else:
month += 1
# Defensive — should be unreachable.
return (now_local + timedelta(days=30)).astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _ensure_utc(dt: datetime) -> datetime:
"""Coerce a possibly-naive datetime to aware UTC. Naive inputs
are assumed UTC (matches the convention the engine uses when
parsing the DB ``next_run`` column)."""
if dt.tzinfo is None:
return dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def _resolve_tz(name: Optional[str]):
"""Look up an IANA tz by name. Falls back to UTC when the name is
unknown — ``ZoneInfoNotFoundError`` is the symptom of either a
typo in the tz string or ``tzdata`` missing on the host. Logged
once per unknown name so the user can see WHY their schedule
isn't running in the timezone they configured."""
if not name:
return timezone.utc
try:
return ZoneInfo(name)
except ZoneInfoNotFoundError:
if name not in _UNKNOWN_TZ_WARNED:
_UNKNOWN_TZ_WARNED.add(name)
logger.warning(
"Unknown timezone %r — schedule will run against UTC. "
"Check the spelling (IANA format like 'America/Los_Angeles') "
"or install the `tzdata` package on minimal hosts.",
name,
)
return timezone.utc
def _parse_hhmm(time_str: Optional[str]) -> tuple:
"""Parse ``HH:MM`` → ``(hour, minute)``. Defaults to 00:00 on
garbage input — same defensive shape as the engine's existing
daily/weekly time parsing."""
if not isinstance(time_str, str):
return 0, 0
try:
h, m = time_str.split(':', 1)
return max(0, min(23, int(h))), max(0, min(59, int(m)))
except (ValueError, AttributeError):
return 0, 0
def _parse_weekdays(days) -> set:
"""``['mon', 'wed']`` → ``{0, 2}``. Empty / missing / all-invalid
list returns ``set(range(7))`` ("every day"), matching the
engine's existing ``_next_weekly_occurrence`` fallback."""
if not isinstance(days, (list, tuple)):
return set(range(7))
parsed = {_WEEKDAY_MAP[d.lower()] for d in days
if isinstance(d, str) and d.lower() in _WEEKDAY_MAP}
return parsed or set(range(7))
def _days_in_month(year: int, month: int) -> int:
"""Last calendar day of ``year-month``. Stdlib-only — no calendar
module import needed; cycle through the 12 months."""
if month == 12:
next_first = datetime(year + 1, 1, 1)
else:
next_first = datetime(year, month + 1, 1)
last_day = next_first - timedelta(days=1)
return last_day.day

234
core/automation_engine.py
View file

@ -18,14 +18,10 @@ import time
import threading
import requests
from datetime import datetime, timedelta, timezone
from typing import Optional
from utils.logging_config import get_logger
from core.automation.schedule import next_run_at
logger = get_logger("automation_engine")
def _utcnow():
"""Return current UTC time as timezone-aware datetime."""
return datetime.now(timezone.utc)
@ -38,42 +34,6 @@ def _utc_after(seconds):
"""Return UTC time N seconds from now as naive string for DB storage."""
return (datetime.now(timezone.utc) + timedelta(seconds=seconds)).strftime('%Y-%m-%d %H:%M:%S')
def _dt_to_db_str(dt: datetime) -> str:
"""Convert an aware-UTC datetime to the naive-UTC string the DB
``next_run`` column stores. Centralised so a tz mistake here
surfaces in one place, not scattered through every caller of
``next_run_at``."""
if dt.tzinfo is None:
return dt.strftime('%Y-%m-%d %H:%M:%S')
return dt.astimezone(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')
def _resolve_system_default_tz() -> str:
"""Return the IANA tz name the engine uses when a schedule's
trigger config doesn't carry an explicit ``tz`` field.
Existing daily / weekly rows pre-date the ``tz`` field — the
historic engine computed delays from naive ``datetime.now()``,
which is implicitly the server's local timezone. Falling back to
that same tz here preserves "every Monday at 09:00" running at
09:00 server local for rows that already exist in the DB.
Without ``tzlocal`` installed (or a system without a discoverable
tz), falls back to UTC."""
try:
import tzlocal
return tzlocal.get_localzone_name() or 'UTC'
except Exception:
return 'UTC'
# Server-local tz cached at import time. Re-reading per-call is
# pointless: the host's timezone doesn't change while the process is
# running. Tests that need a different default tz inject it through
# the engine's ``_default_tz`` attribute or via the
# ``automation.default_timezone`` config key.
_SYSTEM_DEFAULT_TZ = _resolve_system_default_tz()
SYSTEM_AUTOMATIONS = [
{
'name': 'Auto-Process Wishlist',
@ -174,25 +134,11 @@ class AutomationEngine:
self._max_chain_depth = 5
self._signal_cooldown_seconds = 10
# Default tz used when a schedule's ``trigger_config`` doesn't
# carry an explicit ``tz`` field — preserves historic behaviour
# for daily / weekly rows created before the field existed
# (engine used naive ``datetime.now()`` = server local). Reads
# from the ``automation.default_timezone`` config key first to
# let users override without touching env vars; falls back to
# the system-detected local tz.
try:
from config.settings import config_manager
self._default_tz = (config_manager.get('automation.default_timezone', '') or _SYSTEM_DEFAULT_TZ)
except Exception:
self._default_tz = _SYSTEM_DEFAULT_TZ
# Trigger registry: type → setup function (schedule only — events use emit())
self._trigger_handlers = {
'schedule': self._setup_schedule_trigger,
'daily_time': self._setup_daily_time_trigger,
'weekly_time': self._setup_weekly_time_trigger,
'monthly_time': self._setup_monthly_time_trigger,
}
# --- Action Handler Registration ---
@ -635,11 +581,6 @@ class AutomationEngine:
action_config['_automation_name'] = auto.get('name', '')
if profile_id is not None:
action_config['_profile_id'] = profile_id
# The profile this run acts AS: an explicit trigger profile, else the
# automation's owner, else admin. System + admin automations are
# profile 1, so this is a no-op for them — only non-admin-owned
# automations gain their correct identity in the background.
_effective_profile_id = profile_id if profile_id is not None else (auto.get('profile_id') or 1)
# Action delay (skipped for manual run_now)
delay_minutes = action_config.get('delay', 0)
@ -676,7 +617,7 @@ class AutomationEngine:
self._progress_finish_fn(automation_id, result)
except Exception as e:
logger.debug("scheduled progress finish (skipped): %s", e)
self._finish_run(auto, automation_id, result, error=None, retry_delay_seconds=300)
self._finish_run(auto, automation_id, result, error=None)
return
# Initialize progress tracking (skip if already done during delay)
@ -686,14 +627,9 @@ class AutomationEngine:
except Exception as e:
logger.debug("scheduled progress init: %s", e)
# Execute the action under the owner's profile so get_current_profile_id()
# (and the per-profile clients it resolves) act as the automation's owner
# in the background, not admin. Reset in finally so a pooled thread can't
# leak the override to the next job.
# Execute the action
error = None
result = {}
from core.profile_context import set_background_profile, reset_background_profile
_bg_token = set_background_profile(_effective_profile_id)
try:
result = handler_info['handler'](action_config) or {}
logger.info(f"Automation '{auto['name']}' (id={automation_id}) executed: {result.get('status', 'ok')}")
@ -712,8 +648,6 @@ class AutomationEngine:
error = str(e)
result = {'status': 'error', 'error': error}
logger.error(f"Automation '{auto['name']}' (id={automation_id}) failed: {e}")
finally:
reset_background_profile(_bg_token)
# Finalize progress tracking
if self._progress_finish_fn:
@ -730,7 +664,7 @@ class AutomationEngine:
self._finish_run(auto, automation_id, result, error)
def _finish_run(self, auto, automation_id, result, error, retry_delay_seconds=None):
def _finish_run(self, auto, automation_id, result, error):
"""Update DB with run stats and reschedule."""
next_run_str = None
trigger_type = auto.get('trigger_type', '')
@ -738,23 +672,23 @@ class AutomationEngine:
if trigger_type in self._trigger_handlers:
try:
trigger_config = json.loads(auto.get('trigger_config') or '{}')
if retry_delay_seconds:
next_run_str = _utc_after(retry_delay_seconds)
if trigger_type == 'daily_time':
# Next run is tomorrow at the configured time (compute delay from local time, store as UTC)
time_str = trigger_config.get('time', '00:00')
hour, minute = map(int, time_str.split(':'))
now_local = datetime.now()
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
next_run_str = _utc_after((target - now_local).total_seconds())
elif trigger_type == 'weekly_time':
time_str = trigger_config.get('time', '00:00')
hour, minute = map(int, time_str.split(':'))
now_local = datetime.now()
target = self._next_weekly_occurrence(hour, minute, trigger_config.get('days', []))
next_run_str = _utc_after((target - now_local).total_seconds())
else:
# Single integration point with ``next_run_at``. The
# helper handles every trigger type the engine
# supports (interval / daily / weekly / monthly) and
# returns aware-UTC; ``_dt_to_db_str`` normalises to
# the naive-UTC string the DB column stores. Tests
# injecting a different ``now_utc`` patch this same
# path — no scattered ``datetime.now()`` calls left.
next_run_dt = next_run_at(
trigger_type, trigger_config,
now_utc=_utcnow(),
default_tz=self._default_tz,
)
if next_run_dt is not None:
next_run_str = _dt_to_db_str(next_run_dt)
delay = self._calc_delay_seconds(trigger_config)
if delay:
next_run_str = _utc_after(delay)
except Exception as e:
logger.debug("next run calc failed: %s", e)
@ -828,72 +762,22 @@ class AutomationEngine:
logger.debug(f"Scheduled automation {automation_id} in {delay:.0f}s")
def _setup_daily_time_trigger(self, automation_id, config):
"""Config: ``{"time": "03:00", "tz": "<IANA>"}`` — runs daily
at the specified local time. Tz defaults to ``self._default_tz``
when absent."""
self._setup_timed_trigger(automation_id, 'daily_time', config,
label=f"Daily at {config.get('time', '00:00')}")
"""Config: {"time": "03:00"} — runs daily at the specified local time."""
time_str = config.get('time', '00:00')
try:
hour, minute = map(int, time_str.split(':'))
except (ValueError, AttributeError):
hour, minute = 0, 0
def _setup_weekly_time_trigger(self, automation_id, config):
"""Config: ``{"time": "03:00", "days": ["mon","wed","fri"], "tz": "<IANA>"}``."""
day_names = ', '.join(config.get('days') or []) or 'every day'
self._setup_timed_trigger(automation_id, 'weekly_time', config,
label=f"Weekly {config.get('time', '00:00')} on {day_names}")
now_local = datetime.now()
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target <= now_local:
target += timedelta(days=1)
def _setup_monthly_time_trigger(self, automation_id, config):
"""Config: ``{"time": "09:00", "day_of_month": 15, "tz": "<IANA>"}``.
delay = (target - now_local).total_seconds()
Day clamped to [1, 31]; months too short for the target day
clamp to the last valid day (Feb 31 → Feb 28 / Feb 29 leap
year) per standard cron convention — see
``core.automation.schedule._next_monthly`` for the rule."""
day = config.get('day_of_month', 1)
self._setup_timed_trigger(automation_id, 'monthly_time', config,
label=f"Monthly {config.get('time', '00:00')} on day {day}")
def _setup_timed_trigger(self, automation_id, trigger_type, config, *, label):
"""Shared setup for daily / weekly / monthly time triggers.
All three flow through the same skeleton: compute next-run
via ``next_run_at``, persist to DB, arm a ``threading.Timer``
that fires the automation when the delay elapses. Lifting
these out of three near-identical methods means there's one
place to fix when (e.g.) timer rearm semantics need a tweak.
Honours an existing future ``next_run`` row in the DB —
prevents losing a hand-edited next_run when the engine
reschedules at startup. Same guard as the interval path."""
target_dt = next_run_at(
trigger_type, config or {},
now_utc=_utcnow(),
default_tz=self._default_tz,
)
if target_dt is None:
logger.warning(
f"Skip scheduling automation {automation_id}: next_run_at returned "
f"None for {trigger_type!r}",
)
return
delay = max(0.0, (target_dt - _utcnow()).total_seconds())
# If the DB already carries a future next_run, prefer it —
# matches the interval-path behaviour and lets manual edits
# or pending retries survive a process restart.
auto = self.db.get_automation(automation_id)
if auto and auto.get('next_run'):
try:
existing = datetime.strptime(
auto['next_run'], '%Y-%m-%d %H:%M:%S',
).replace(tzinfo=timezone.utc)
remaining = (existing - _utcnow()).total_seconds()
if remaining > 0:
delay = remaining
target_dt = existing
except (ValueError, TypeError):
pass
self.db.update_automation(automation_id, next_run=_dt_to_db_str(target_dt))
next_run_str = _utc_after(delay)
self.db.update_automation(automation_id, next_run=next_run_str)
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
timer.daemon = True
@ -902,7 +786,48 @@ class AutomationEngine:
with self._lock:
self._timers[automation_id] = timer
logger.debug(f"{label} automation {automation_id} scheduled (in {delay:.0f}s)")
logger.debug(f"Daily automation {automation_id} scheduled for {time_str} (in {delay:.0f}s)")
def _setup_weekly_time_trigger(self, automation_id, config):
"""Config: {"time": "03:00", "days": ["mon", "wed", "fri"]}"""
time_str = config.get('time', '00:00')
try:
hour, minute = map(int, time_str.split(':'))
except (ValueError, AttributeError):
hour, minute = 0, 0
target = self._next_weekly_occurrence(hour, minute, config.get('days', []))
delay = (target - datetime.now()).total_seconds()
next_run_str = _utc_after(delay)
self.db.update_automation(automation_id, next_run=next_run_str)
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
timer.daemon = True
timer.start()
with self._lock:
self._timers[automation_id] = timer
day_names = ', '.join(config.get('days', [])) or 'every day'
logger.debug(f"Weekly automation {automation_id} scheduled for {time_str} on {day_names} (in {delay:.0f}s)")
def _next_weekly_occurrence(self, hour, minute, days):
"""Find the next datetime matching one of the given weekday abbreviations."""
day_map = {'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6}
allowed = {day_map[d] for d in days if d in day_map}
if not allowed:
allowed = set(range(7)) # no days selected = every day
now = datetime.now()
for offset in range(8): # check today + next 7 days
candidate = now + timedelta(days=offset)
if candidate.weekday() in allowed:
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target > now:
return target
# Fallback: tomorrow (shouldn't happen with 8-day scan)
return now.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
# --- Then Actions (notifications + signals) ---
@ -1072,8 +997,6 @@ class AutomationEngine:
"""Send message via Telegram Bot API."""
bot_token = config.get('bot_token', '').strip()
chat_id = config.get('chat_id', '').strip()
thread_id = config.get('thread_id', '').strip()
if not bot_token or not chat_id:
raise ValueError("Bot token and chat ID are required for Telegram")
@ -1082,16 +1005,9 @@ class AutomationEngine:
for key, value in variables.items():
message = message.replace('{' + key + '}', value)
payload = {"chat_id": chat_id, "text": message, "parse_mode": "HTML"}
if thread_id:
try:
payload["message_thread_id"] = int(thread_id)
except ValueError:
pass # invalid — fall back to main chat
resp = requests.post(
f'https://api.telegram.org/bot{bot_token}/sendMessage',
json=payload,
json={"chat_id": chat_id, "text": message, "parse_mode": "HTML"},
timeout=10,
)
data = resp.json() if resp.status_code == 200 else {}

37
core/blocklist/__init__.py
View file

@ -1,37 +0,0 @@
"""Artist / album / track blocklist (the "proper" blacklist).
Distinct from ``download_blacklist`` (which skips one bad source file from one
Soulseek peer — untouched here). This blocklist bans an ARTIST, ALBUM, or
TRACK from being acquired, keyed by metadata-source IDs (Spotify / iTunes /
Deezer / MusicBrainz) so a ban survives a source switch.
Phase 1 enforces at the single ``add_to_wishlist`` chokepoint: every
auto-acquisition path (watchlist, discography backfill, repair, manual
wishlist add) funnels through it, so one guard covers them all.
- ``matching`` — the pure decision core (no DB, no I/O): build an index from
blocklist rows, ask whether a candidate is blocked, with artist→album→track
cascade.
"""
from core.blocklist.matching import (
ENTITY_ALBUM,
ENTITY_ARTIST,
ENTITY_TRACK,
ENTITY_TYPES,
SOURCE_ID_FIELDS,
BlocklistIndex,
build_index,
candidate_block_reason,
)
__all__ = [
"ENTITY_ARTIST",
"ENTITY_ALBUM",
"ENTITY_TRACK",
"ENTITY_TYPES",
"SOURCE_ID_FIELDS",
"BlocklistIndex",
"build_index",
"candidate_block_reason",
]

50
core/blocklist/backfill.py
View file

@ -1,50 +0,0 @@
"""Cross-source ID backfill for blocklist entries.
When a user blocks an item, the modal gives us the ID for ONE source (the one
they searched). For the ban to survive a source switch, we resolve the OTHER
sources' IDs too — matching the blocked artist/album/track by name on each
source and taking a confident hit.
The resolution is kept pure + injected so it tests without a network: callers
pass a ``resolvers`` map ``{source: fn(entity_type, name, parent_name) -> id |
None}``. ``core/blocklist/runtime.py`` wires the real metadata clients.
Honest about fragility (acknowledged in design): artist matching is reliable,
album/track cross-source matching is best-effort (editions, common titles), so
a resolver returning None just leaves that source unmatched — the artist
name-fallback in matching.py covers artist gaps; album/track gaps mean that
ban only applies on sources where an ID resolved.
"""
from __future__ import annotations
from typing import Any, Callable, Dict, Optional
from core.blocklist.matching import SOURCE_ID_FIELDS
def resolve_missing_ids(
entry: Dict[str, Any],
resolvers: Dict[str, Callable[..., Optional[str]]],
) -> Dict[str, str]:
"""Return ``{id_column: resolved_id}`` for the sources currently missing an
ID on ``entry``. Never raises — a resolver that errors is skipped."""
out: Dict[str, str] = {}
entity_type = entry.get("entity_type")
name = entry.get("name")
parent = entry.get("parent_name")
if not entity_type or not name:
return out
for source, col in SOURCE_ID_FIELDS.items():
if entry.get(col):
continue # already known
fn = resolvers.get(source)
if not fn:
continue
try:
rid = fn(entity_type, name, parent)
except Exception:
rid = None
if rid:
out[col] = str(rid)
return out

128
core/blocklist/matching.py
View file

@ -1,128 +0,0 @@
"""Pure blocklist matching — no DB, no I/O, fully unit-testable.
The brain of the blocklist: given the stored blocklist rows and a candidate
track being considered for the wishlist, decide whether it's blocked.
Design decisions (per Boulder):
- **ID-keyed.** Each row carries the candidate's IDs in up to four metadata
sources. A candidate is matched against the SAME source it came in on
(the wishlist payload carries active-source IDs), so a Deezer-numeric id
can't collide with an iTunes-numeric id of a different entity.
- **Cascade.** Blocking an artist blocks their albums + tracks; blocking an
album blocks its tracks. The candidate carries its own artist/album/track
IDs, so the check walks track → album → artist and blocks on the first hit.
- **Name fallback for ARTISTS only.** A blocked artist also matches by
case-folded name — this covers the window before the background ID-backfill
has resolved the active source's id. Albums/tracks do NOT fall back to name
(common titles like "Greatest Hits" would false-positive across artists);
they rely on IDs, which backfill fills in.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
ENTITY_ARTIST = "artist"
ENTITY_ALBUM = "album"
ENTITY_TRACK = "track"
ENTITY_TYPES = (ENTITY_ARTIST, ENTITY_ALBUM, ENTITY_TRACK)
# Blocklist-row column → the metadata source it belongs to.
SOURCE_ID_FIELDS = {
"spotify": "spotify_id",
"itunes": "itunes_id",
"deezer": "deezer_id",
"musicbrainz": "musicbrainz_id",
}
def _norm(text: Any) -> str:
return str(text or "").strip().casefold()
@dataclass
class _TypeIndex:
# per-source set of blocked ids, plus a case-folded name set (artists only)
ids: Dict[str, Set[str]] = field(default_factory=lambda: {s: set() for s in SOURCE_ID_FIELDS})
names: Set[str] = field(default_factory=set)
def hit(self, source: Optional[str], entity_id: Any, name: Any, use_name: bool) -> bool:
if entity_id and source in self.ids and str(entity_id) in self.ids[source]:
return True
if use_name and name and _norm(name) in self.names:
return True
return False
@dataclass
class BlocklistIndex:
"""Membership index built once per scan from the blocklist rows."""
artists: _TypeIndex = field(default_factory=_TypeIndex)
albums: _TypeIndex = field(default_factory=_TypeIndex)
tracks: _TypeIndex = field(default_factory=_TypeIndex)
@property
def is_empty(self) -> bool:
for ti in (self.artists, self.albums, self.tracks):
if ti.names or any(ti.ids.values()):
return False
return True
def build_index(rows: Iterable[Dict[str, Any]]) -> BlocklistIndex:
"""Build a BlocklistIndex from blocklist DB rows.
Each row needs ``entity_type``, ``name``, and the source id columns
(``spotify_id`` / ``itunes_id`` / ``deezer_id`` / ``musicbrainz_id``).
Unknown entity types are ignored."""
idx = BlocklistIndex()
by_type = {ENTITY_ARTIST: idx.artists, ENTITY_ALBUM: idx.albums, ENTITY_TRACK: idx.tracks}
for row in rows or []:
ti = by_type.get((row.get("entity_type") or "").strip().lower())
if ti is None:
continue
for source, col in SOURCE_ID_FIELDS.items():
val = row.get(col)
if val:
ti.ids[source].add(str(val))
name = row.get("name")
if name:
ti.names.add(_norm(name))
return idx
def candidate_block_reason(
index: BlocklistIndex,
*,
source: Optional[str],
track_id: Any = None,
track_name: Any = None,
album_id: Any = None,
album_name: Any = None,
artists: Optional[List[Dict[str, Any]]] = None,
) -> Optional[Tuple[str, str]]:
"""Return ``(entity_type, label)`` for the first cascade hit, else None.
``source`` is the metadata source the candidate IDs came from (the wishlist
payload's provider). ``artists`` is a list of ``{'id', 'name'}`` dicts.
Order matters only for the returned reason — any hit blocks."""
if index.is_empty:
return None
# Track level — id only (names too ambiguous to ban across artists).
if index.tracks.hit(source, track_id, track_name, use_name=False):
return (ENTITY_TRACK, str(track_name or track_id or "track"))
# Album level — id only.
if index.albums.hit(source, album_id, album_name, use_name=False):
return (ENTITY_ALBUM, str(album_name or album_id or "album"))
# Artist level — id OR case-folded name (safe + covers the backfill window).
for artist in artists or []:
a_id = artist.get("id") if isinstance(artist, dict) else None
a_name = artist.get("name") if isinstance(artist, dict) else artist
if index.artists.hit(source, a_id, a_name, use_name=True):
return (ENTITY_ARTIST, str(a_name or a_id or "artist"))
return None

82
core/blocklist/runtime.py
View file

@ -1,82 +0,0 @@
"""Wire real metadata clients to the blocklist backfill resolvers.
Resolves a blocked item's ID on each metadata source by searching that source
for the name and taking a confidently name-matched hit. Confidence = exact
significant-token match (drops articles/punctuation) so we never hang a wrong
ID on an entry. Albums/tracks additionally require the parent artist to match
when both sides expose one.
"""
from __future__ import annotations
import re
from typing import Any, Callable, Dict, Optional
from utils.logging_config import get_logger
logger = get_logger("blocklist.runtime")
_STOP = {"the", "a", "an", "feat", "ft", "featuring", "with"}
def _tokens(text: Any) -> frozenset:
words = re.sub(r"[^a-z0-9]+", " ", str(text or "").lower()).split()
return frozenset(w for w in words if w not in _STOP)
def _name_of(obj: Any) -> str:
if isinstance(obj, dict):
return str(obj.get("name") or obj.get("title") or "")
return str(getattr(obj, "name", None) or getattr(obj, "title", None) or "")
def _id_of(obj: Any) -> Optional[str]:
val = obj.get("id") if isinstance(obj, dict) else getattr(obj, "id", None)
return str(val) if val else None
def _confident(result_name: str, want_name: str) -> bool:
rt, wt = _tokens(result_name), _tokens(want_name)
return bool(rt) and rt == wt
def _make_resolver(source: str) -> Callable[..., Optional[str]]:
def resolve(entity_type: str, name: str, parent_name: Optional[str] = None) -> Optional[str]:
from core.metadata.registry import get_client_for_source
client = get_client_for_source(source)
if not client:
return None
method = {
"artist": "search_artists",
"album": "search_albums",
"track": "search_tracks",
}.get(entity_type)
fn = getattr(client, method, None) if method else None
if not fn:
return None
try:
results = fn(name, limit=5) or []
except Exception as e:
logger.debug("%s %s search failed for %r: %s", source, entity_type, name, e)
return None
for r in results:
if not _confident(_name_of(r), name):
continue
# For album/track, also require the artist to line up when known.
if entity_type in ("album", "track") and parent_name:
artists = (r.get("artists") if isinstance(r, dict) else getattr(r, "artists", None)) or []
cand_artists = " ".join(
a.get("name", "") if isinstance(a, dict) else str(a) for a in artists)
if _tokens(parent_name) and not (_tokens(parent_name) & _tokens(cand_artists)):
continue
rid = _id_of(r)
if rid:
return rid
return None
return resolve
def build_resolvers() -> Dict[str, Callable[..., Optional[str]]]:
"""Source→resolver map for core.blocklist.backfill.resolve_missing_ids."""
return {s: _make_resolver(s) for s in ("spotify", "itunes", "deezer", "musicbrainz")}

27
core/boot_phase.py
View file

@ -1,27 +0,0 @@
"""Boot-phase guard for non-blocking container startup.
While the gunicorn worker is importing ``web_server`` (module-level client and
worker initialization), external provider API probes must not block startup.
Network validation is deferred until ``mark_boot_complete()`` runs at the end
of that import pass.
"""
from __future__ import annotations
import threading
_boot_lock = threading.Lock()
_boot_active = True
def is_boot_phase() -> bool:
"""Return True while module import must avoid blocking provider API calls."""
with _boot_lock:
return _boot_active
def mark_boot_complete() -> None:
"""End the boot phase — provider clients may perform network probes again."""
global _boot_active
with _boot_lock:
_boot_active = False

88
core/connection_test.py
View file

@ -82,16 +82,7 @@ def run_service_test(service, test_config):
if temp_client.is_spotify_authenticated():
return True, "Spotify connection successful!"
else:
# Spotify-Free (no-auth) metadata path: officially unauthenticated,
# but the no-creds source is selected and available. Report it as the
# working source rather than the generic Deezer/Discogs/iTunes fallback.
try:
spotify_free_available = temp_client.is_spotify_metadata_available()
except Exception:
spotify_free_available = False
if spotify_free_available:
return True, "Spotify (no-auth) connection successful!"
# Using a different fallback metadata source
# Using fallback metadata source
fb_src = _get_metadata_fallback_source()
fallback_name = 'Deezer' if fb_src == 'deezer' else 'Discogs' if fb_src == 'discogs' else 'iTunes'
if spotify_configured:
@ -200,12 +191,6 @@ def run_service_test(service, test_config):
'tidal': "Tidal download source ready.",
'qobuz': "Qobuz download source ready.",
'hifi': "HiFi download source ready.",
'deezer_dl': "Deezer download source ready.",
'amazon': "Amazon download source ready.",
'lidarr': "Lidarr download source ready.",
'soundcloud': "SoundCloud download source ready.",
'torrent': "Torrent download source ready.",
'usenet': "Usenet download source ready.",
'hybrid': "Download sources ready (Hybrid mode)."
}
message = mode_messages.get(download_mode, "Download source connected.")
@ -218,12 +203,6 @@ def run_service_test(service, test_config):
'tidal': "Tidal download source not available. Check authentication.",
'qobuz': "Qobuz download source not available. Check authentication.",
'hifi': "HiFi download source not available. Public API instances may be down.",
'deezer_dl': "Deezer download source not available. Check authentication.",
'amazon': "Amazon download source not available.",
'lidarr': "Lidarr download source not available. Check Lidarr URL and API key.",
'soundcloud': "SoundCloud download source not available.",
'torrent': "Torrent download source not available. Check Prowlarr and torrent client settings.",
'usenet': "Usenet download source not available. Check Prowlarr and usenet client settings.",
'hybrid': "Could not connect to download sources. Check configuration."
}
error = mode_errors.get(download_mode, "Download source connection failed.")
@ -326,61 +305,6 @@ def run_service_test(service, test_config):
return False, "Invalid Genius access token."
except Exception as e:
return False, f"Genius connection error: {str(e)}"
elif service == "usenet_client":
client_type = (config_manager.get('usenet_client.type', '') or '').strip().lower()
url = config_manager.get('usenet_client.url', '')
if not url:
return False, "Usenet client URL is required."
if not client_type:
return False, "Pick a usenet client (SABnzbd or NZBGet)."
try:
from core.usenet_clients import adapter_for_type as _usenet_adapter_for_type
adapter = _usenet_adapter_for_type(client_type)
if adapter is None:
return False, f"Unknown usenet client type: {client_type}"
if not adapter.is_configured():
if client_type == "sabnzbd":
return False, "SABnzbd needs both URL and API key."
return False, "NZBGet needs URL, username, and password."
if run_async(adapter.check_connection()):
return True, f"Connected to {client_type}"
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
except Exception as e:
return False, f"Usenet client connection error: {str(e)}"
elif service == "torrent_client":
client_type = (config_manager.get('torrent_client.type', '') or '').strip().lower()
url = config_manager.get('torrent_client.url', '')
if not url:
return False, "Torrent client URL is required."
if not client_type:
return False, "Pick a torrent client (qBittorrent, Transmission, or Deluge)."
try:
from core.torrent_clients import adapter_for_type
adapter = adapter_for_type(client_type)
if adapter is None:
return False, f"Unknown torrent client type: {client_type}"
if not adapter.is_configured():
return False, "Torrent client missing required credentials."
if run_async(adapter.check_connection()):
return True, f"Connected to {client_type}"
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
except Exception as e:
return False, f"Torrent client connection error: {str(e)}"
elif service == "prowlarr":
url = config_manager.get('prowlarr.url', '')
api_key = config_manager.get('prowlarr.api_key', '')
if not url or not api_key:
return False, "Prowlarr URL and API key are required."
try:
import requests as _req
resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
headers={'X-Api-Key': api_key}, timeout=10)
if resp.ok:
version = resp.json().get('version', '?')
return True, f"Connected to Prowlarr v{version}"
return False, f"Prowlarr returned HTTP {resp.status_code}"
except Exception as e:
return False, f"Prowlarr connection error: {str(e)}"
elif service == "lidarr" or service == "lidarr_download":
url = config_manager.get('lidarr_download.url', '')
api_key = config_manager.get('lidarr_download.api_key', '')
@ -455,16 +379,6 @@ def run_service_test(service, test_config):
return False, "Hydrabase not connected. Configure URL + API key and click Connect."
except Exception as e:
return False, f"Hydrabase connection error: {str(e)}"
elif service == "musicbrainz":
try:
from core.metadata.registry import get_musicbrainz_client
mb = get_musicbrainz_client()
results = mb.search_artists("radiohead", limit=1)
if results:
return True, "MusicBrainz reachable"
return False, "MusicBrainz returned no results — may be rate-limited or unreachable."
except Exception as e:
return False, f"MusicBrainz connection error: {str(e)}"
elif service == "soundcloud":
# Anonymous SoundCloud has no auth, so "test" really means
# "is yt-dlp installed and can it reach SoundCloud right now."

0
core/credentials/__init__.py
View file

85
core/credentials/store.py
View file

@ -1,85 +0,0 @@
"""Named, switchable service-credential sets — pure logic (Phase 0 foundation).
Today every auth service (Spotify, Tidal, Deezer, Qobuz, Plex, Jellyfin,
Navidrome) holds ONE credential set in config, and clients are global singletons
built from that single slot. This module is the groundwork for letting an admin
save MULTIPLE named credential sets per service ("pills") that each profile can
switch between, without anyone but the admin creating them.
Kept PURE — service registry, payload validation, and active-set selection,
free of DB/Flask so it's unit-testable. Encrypted storage lives in MusicDatabase
(service_credentials / profile_service_credentials tables); runtime client
resolution + UI come in later phases. Nothing here changes existing behaviour;
it's dormant capability until wired.
"""
from __future__ import annotations
# Services that support multiple named credential sets, mapped to the payload
# keys that MUST be present for a set to be usable. Extra keys (OAuth tokens,
# redirect URIs, quality prefs) are allowed and preserved — these are only the
# minimum required to validate a set the admin is saving.
SERVICE_CREDENTIAL_SCHEMA = {
'spotify': ('client_id', 'client_secret'),
'tidal': ('access_token', 'refresh_token'),
'deezer': ('arl',),
'qobuz': ('user_auth_token',),
'plex': ('base_url', 'token'),
'jellyfin': ('base_url', 'api_key'),
'navidrome': ('base_url', 'username', 'password'),
}
SUPPORTED_SERVICES = frozenset(SERVICE_CREDENTIAL_SCHEMA)
def is_supported_service(service: str) -> bool:
"""True when the service supports named credential sets."""
return service in SERVICE_CREDENTIAL_SCHEMA
def validate_credential_payload(service: str, payload):
"""Return ``(ok, missing_keys)`` for a credential set.
Valid when every required key for the service is present and truthy. An
unknown service is invalid with no missing list (caller should reject it
as unsupported, not as "incomplete").
"""
required = SERVICE_CREDENTIAL_SCHEMA.get(service)
if required is None:
return False, []
if not isinstance(payload, dict):
return False, list(required)
def _present(v):
# Whitespace-only strings count as missing — they'd otherwise save a
# blank secret that fails confusingly at the real service later.
return bool(v.strip()) if isinstance(v, str) else bool(v)
missing = [k for k in required if not _present(payload.get(k))]
return (not missing), missing
def pick_active_credential(credentials, selected_id):
"""From ``credentials`` (a list of dicts each carrying ``id``), return the
one whose id == ``selected_id``.
Returns None when there's no selection OR the selected id isn't present —
i.e. a stale pointer whose credential set was deleted. The caller then
falls back to the global/admin default, so a deleted set never breaks a
profile. Pure + stale-safe.
"""
if not selected_id:
return None
for cred in credentials or []:
if cred.get('id') == selected_id:
return cred
return None
__all__ = [
'SERVICE_CREDENTIAL_SCHEMA',
'SUPPORTED_SERVICES',
'is_supported_service',
'validate_credential_payload',
'pick_active_credential',
]

82
core/database_update_health.py
View file

@ -1,82 +0,0 @@
"""Stall detection for the database-update job.
The DB updater keeps a single in-memory state dict whose ``status`` is set to
``running`` at start and only flipped to ``finished``/``error`` by the worker's
completion/error callbacks. If the worker thread hangs — e.g. a media-server API
call with no timeout, a DB lock — those callbacks never fire, so ``status`` stays
``running`` forever and the UI shows a frozen progress bar with no way to recover
(GitHub #859).
This module is the single, *pure* decision for "is a running job stalled?". It
takes the state dict plus the current wall-clock time and a timeout, and answers
yes/no — no DB, no globals, no clock of its own. That keeps it unit-testable and
lets the watchdog wiring in web_server.py stay a thin call. The job carries a
``last_progress_at`` epoch timestamp that the start path and every progress/phase
callback bump; staleness is simply "running, and that timestamp is older than the
timeout".
"""
from __future__ import annotations
from typing import Any, Mapping
# 5 minutes with zero forward progress = presumed hung. A healthy scan ticks
# progress (per-artist) far more often than this even for large libraries, so
# the timeout won't false-positive a slow-but-working run.
DEFAULT_STALL_TIMEOUT_SECONDS = 300
def is_db_update_stalled(
state: Mapping[str, Any],
now: float,
timeout_seconds: float = DEFAULT_STALL_TIMEOUT_SECONDS,
) -> bool:
"""Return True when the job is ``running`` but has made no progress within
``timeout_seconds``.
Conservative by design — it only ever reports a stall it can prove:
- Only a ``running`` job can stall (idle/finished/error never do).
- With no usable ``last_progress_at`` timestamp we cannot judge, so we return
False rather than risk killing a job we have no clock for.
- A non-positive timeout is treated as "disabled" (never stalls).
"""
if not isinstance(state, Mapping):
return False
if state.get("status") != "running":
return False
if timeout_seconds is None or timeout_seconds <= 0:
return False
last = state.get("last_progress_at")
if not last:
return False
try:
elapsed = float(now) - float(last)
except (TypeError, ValueError):
return False
return elapsed >= float(timeout_seconds)
def stalled_error_message(state: Mapping[str, Any], now: float) -> str:
"""Build a clear, human-facing message for a stalled job, including how long
it has been silent and the phase it died in."""
last = state.get("last_progress_at") if isinstance(state, Mapping) else None
phase = state.get("phase") if isinstance(state, Mapping) else None
try:
secs = int(float(now) - float(last)) if last else 0
except (TypeError, ValueError):
secs = 0
msg = "Update appears stuck — no progress"
if secs > 0:
msg += f" for {secs}s"
if phase:
msg += f" (last phase: {phase})"
msg += (". The worker may be hung on the media server. Start a new update "
"to try again, or restart SoulSync if it keeps stalling.")
return msg
__all__ = [
"DEFAULT_STALL_TIMEOUT_SECONDS",
"is_db_update_stalled",
"stalled_error_message",
]

45
core/database_update_worker.py
View file

@ -41,19 +41,7 @@ class DatabaseUpdateWorker:
self.database_path = database_path
self.full_refresh = full_refresh
self.should_stop = False
# Track ids of rows newly INSERTED this run (not updates). The web
# layer reads this to gap-fill embedded provider IDs for the new files
# (auto-reconcile), so newly-added music contributes its
# Spotify/MusicBrainz/etc. ids without a manual backfill.
self._new_track_ids = set()
# Optional callback(worker) run as the FINAL scan phase, immediately
# before the 'finished' signal — so the auto-reconcile is inside the
# scan's running window (automations/UI treat it as a normal phase and
# wait for it). Injected by the web layer (which owns path resolution).
self.post_scan_hook = None
# Statistics tracking
self.processed_artists = 0
self.processed_albums = 0
@ -91,26 +79,7 @@ class DatabaseUpdateWorker:
callback(*args)
except Exception as e:
logger.error(f"Error in callback for {signal_name}: {e}")
def _emit_finished(self, *args):
"""Run the post-scan hook (auto-reconcile) as the final phase, THEN
emit 'finished'.
Running the hook before 'finished' keeps the scan's status at
'running' through the reconcile, so every caller (automations that
poll for completion, the dashboard card, the Tools page) treats it as
a normal scan phase and waits for it — rather than seeing 'finished'
and missing the tail. Best-effort: a hook failure never blocks the
completion signal.
"""
if self.post_scan_hook:
try:
self.post_scan_hook(self)
except Exception as e:
logger.warning(f"post-scan hook failed (non-fatal): {e}")
self._emit_signal('finished', *args)
def connect_callback(self, signal_name: str, callback: Callable):
"""Connect a callback for progress notifications."""
self.callbacks.setdefault(signal_name, []).append(callback)
@ -177,7 +146,7 @@ class DatabaseUpdateWorker:
logger.info(f"Merged {merged} duplicate artists")
except Exception as e:
logger.warning(f"Could not merge duplicate artists: {e}")
self._emit_finished(0, 0, 0, 0, 0)
self._emit_signal('finished', 0, 0, 0, 0, 0)
return
logger.info(f"Incremental update: Found {len(artists_to_process)} artists to process")
@ -261,7 +230,7 @@ class DatabaseUpdateWorker:
self.removed_tracks = removal.get('tracks_removed', 0) if removal else 0
# Emit final results
self._emit_finished(
self._emit_signal('finished',
self.processed_artists,
self.processed_albums,
self.processed_tracks,
@ -362,7 +331,7 @@ class DatabaseUpdateWorker:
f"{self.processed_albums} albums, {self.processed_tracks} new tracks, "
f"{stale_removed} stale tracks removed")
self._emit_finished(
self._emit_signal('finished',
self.processed_artists,
self.processed_albums,
self.processed_tracks,
@ -911,8 +880,6 @@ class DatabaseUpdateWorker:
track_success = self.database.insert_or_update_media_track(track, album_id, artist_id, server_source=self.server_type)
if track_success:
total_processed_tracks += 1
if track_success == 'inserted':
self._new_track_ids.add(str(track.ratingKey))
logger.debug(f"Processed new track: {track.title}")
except Exception as e:
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
@ -1377,8 +1344,6 @@ class DatabaseUpdateWorker:
skipped_count += 1
elif track_success:
track_count += 1
if track_success == 'inserted':
self._new_track_ids.add(track_id_str)
except Exception as e:
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")

153
core/db_integrity.py
View file

@ -1,153 +0,0 @@
"""SQLite integrity + safe-backup helpers.
Born out of a real incident: a WAL-mode DB got corrupted (most likely an
interrupted write during a hard restart), and because the backup routine
(a) never checked integrity and (b) rotated the oldest backup out by mtime,
every rolling backup ended up being a faithful copy of the already-corrupt
file — so when recovery was needed, all snapshots were poisoned.
This module makes that impossible:
* ``quick_check(path)`` / ``is_healthy(path)`` — fast read-only integrity probe.
* ``safe_backup(...)`` — verifies the SOURCE is healthy before copying, uses the
SQLite Online Backup API, then verifies the RESULT. A corrupt source never
produces (or keeps) a backup.
* ``prune_backups(...)`` — rotation that NEVER deletes the most recent
*verified-healthy* backup, even to honor the max-count, so a run of bad
backups can't evict your last good one.
Pure-ish: only touches sqlite3 + the filesystem paths it's given; no Flask, no
app globals. Unit-testable with real (and deliberately-corrupted) temp DBs.
"""
from __future__ import annotations
import logging
import os
import sqlite3
from typing import Optional
logger = logging.getLogger("db_integrity")
def _close_quietly(conn) -> None:
"""Best-effort close; a failure to close during cleanup must not mask the
real error we're handling, but we log it rather than swallow silently."""
if conn is None:
return
try:
conn.close()
except Exception as e: # noqa: BLE001 — cleanup path, real error already in flight
logger.debug("db_integrity: connection close failed: %s", e)
class DBIntegrityError(Exception):
"""Raised when a database fails its integrity check."""
def quick_check(db_path: str, *, timeout: float = 30.0) -> str:
"""Run ``PRAGMA quick_check`` read-only and return its first result row.
Returns ``'ok'`` for a healthy DB, otherwise the first error line. Raises
``DBIntegrityError`` if the file can't even be opened/read (malformed
header, I/O error) — i.e. unambiguously bad.
"""
if not os.path.exists(db_path):
raise DBIntegrityError(f"Database file not found: {db_path}")
conn = None
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=timeout)
row = conn.execute("PRAGMA quick_check(1)").fetchone()
return (row[0] if row else "no result")
except sqlite3.DatabaseError as e:
# malformed header / disk image malformed / disk I/O error
raise DBIntegrityError(f"{db_path}: {e}") from e
finally:
_close_quietly(conn)
def is_healthy(db_path: str, *, timeout: float = 30.0) -> bool:
"""True iff the DB opens and ``quick_check`` reports 'ok'. Never raises."""
try:
return quick_check(db_path, timeout=timeout) == "ok"
except DBIntegrityError:
return False
def safe_backup(src_path: str, dst_path: str, *, verify_source: bool = True,
verify_result: bool = True) -> None:
"""Back up ``src_path`` to ``dst_path`` via the SQLite Online Backup API,
refusing to produce a backup from (or keep a backup of) a corrupt DB.
Raises ``DBIntegrityError`` and removes any partial ``dst_path`` when the
source is unhealthy (``verify_source``) or the produced backup fails its
own check (``verify_result``). On success ``dst_path`` is a verified-good
copy.
"""
if verify_source and not is_healthy(src_path):
# Don't immortalize corruption — surface it so the caller can alert
# and, crucially, NOT rotate out the existing good backups.
raise DBIntegrityError(
f"Refusing to back up: source database failed integrity check ({src_path})"
)
src = dst = None
try:
src = sqlite3.connect(src_path)
dst = sqlite3.connect(dst_path)
src.backup(dst)
finally:
_close_quietly(dst)
_close_quietly(src)
if verify_result and not is_healthy(dst_path):
# The copy itself came out bad — discard it rather than keep a dud.
try:
os.remove(dst_path)
except OSError:
pass
raise DBIntegrityError(
f"Backup produced a corrupt file and was discarded ({dst_path})"
)
def prune_backups(backup_paths, max_keep: int,
health_check=is_healthy) -> list:
"""Decide which backups to delete to honor ``max_keep`` WITHOUT ever
deleting the most-recent verified-healthy backup.
``backup_paths`` is an iterable of paths; order does not matter (we sort by
mtime). Returns the list of paths that SHOULD be deleted (does not delete
them — the caller does the IO, so this stays pure/testable).
Rule: oldest-first deletion until <= max_keep, but the single newest
*healthy* backup is protected and never selected for deletion. So even if
the newest few backups are corrupt, the last good snapshot survives.
"""
paths = [p for p in backup_paths]
# Newest first.
paths.sort(key=lambda p: _safe_mtime(p), reverse=True)
# Find the newest healthy backup — the one we must never drop.
protected: Optional[str] = None
for p in paths:
if health_check(p):
protected = p
break
if len(paths) <= max_keep:
return []
# Delete oldest-first beyond max_keep, but skip the protected one.
deletable = [p for p in paths if p != protected]
# oldest first among deletable
deletable.sort(key=lambda p: _safe_mtime(p))
num_to_delete = len(paths) - max_keep
return deletable[:num_to_delete]
def _safe_mtime(path: str) -> float:
try:
return os.path.getmtime(path)
except OSError:
return 0.0

88
core/deezer_client.py
View file

@ -6,7 +6,6 @@ from typing import Dict, List, Optional, Any
from functools import wraps
from dataclasses import dataclass
from utils.logging_config import get_logger
from core.metadata.artist_album_cache import get_cached_artist_album_items, store_artist_album_items
from core.metadata.cache import get_metadata_cache
logger = get_logger("deezer_client")
@ -117,48 +116,6 @@ def _is_full_track_payload(payload: Optional[Dict[str, Any]]) -> bool:
return 'track_position' in payload and 'contributors' in payload
def resolve_album_track_positions(session, base_url, album_ids, cache=None, sleep_s=0.2):
"""Build ``{str(track_id): track_position}`` for a set of Deezer album ids.
Deezer PLAYLIST and SEARCH track objects (and even the album object's embedded
``tracks.data``) omit ``track_position`` — only ``/album/<id>/tracks`` and
``/track/<id>`` carry it. So numbering playlist tracks by their playlist index
silently poisons the real album track number, which then rides onto the
downloaded file's tag. This resolves the authoritative position per album
(cache-first, best-effort — a failed album just isn't in the map)."""
import time as _time
positions: Dict[str, int] = {}
for aid in album_ids:
aid = str(aid)
at_list = None
if cache:
try:
ct = cache.get_entity('deezer', 'album_tracks', aid)
if ct and ct.get('data'):
at_list = ct['data']
except Exception: # noqa: BLE001 - cache is best-effort
at_list = None
if at_list is None:
try:
if sleep_s:
_time.sleep(sleep_s) # respect Deezer rate limits
r = session.get(f"{base_url}/album/{aid}/tracks", params={'limit': 500}, timeout=10)
if getattr(r, 'ok', False):
at_list = (r.json() or {}).get('data', [])
if cache and at_list is not None:
try:
cache.store_entity('deezer', 'album_tracks', aid, {'data': at_list})
except Exception as _cache_err: # noqa: BLE001
logger.debug("album_tracks cache store failed for %s: %s", aid, _cache_err)
except Exception: # noqa: BLE001 - never let metadata resolution break the fetch
at_list = None
for at in (at_list or []):
tp = at.get('track_position')
if at.get('id') and tp:
positions[str(at['id'])] = tp
return positions
# ==================== Dataclasses (match iTunesClient / SpotifyClient format) ====================
@dataclass
@ -916,41 +873,17 @@ class DeezerClient:
Matches iTunesClient.get_artist_albums() interface.
Paginates through all results up to the requested limit."""
cache = get_metadata_cache()
cached_items = get_cached_artist_album_items(cache, 'deezer', artist_id, album_type=album_type, limit=limit)
if cached_items:
try:
requested_types = [t.strip() for t in album_type.split(',')]
cached_albums = []
for album_data in cached_items:
album = Album.from_deezer_album(album_data)
if album_type != 'album,single':
if album.album_type not in requested_types:
if not (album.album_type == 'ep' and 'single' in requested_types):
continue
cached_albums.append(album)
return cached_albums[:limit]
except Exception as e:
logger.debug("Deezer artist albums cache reuse failed: %s", e)
albums = []
all_raw = []
requested_types = [t.strip() for t in album_type.split(',')]
offset = 0
page_size = 100 # Deezer API max per request
complete = True # cleared if pagination breaks on a transient/malformed error
while offset < limit:
fetch_limit = min(page_size, limit - offset)
data = self._api_get(f'artist/{artist_id}/albums', {'limit': fetch_limit, 'index': offset})
if not data or 'data' not in data:
# Malformed/transient response mid-pagination — what we have is a
# PARTIAL discography. Don't cache it as the full list (mirrors the
# Spotify truncated-fetch guard). #853 follow-up.
complete = False
if not data or 'data' not in data or len(data['data']) == 0:
break
if len(data['data']) == 0:
break # No more albums — a clean end of pagination.
for album_data in data['data']:
all_raw.append(album_data)
@ -967,6 +900,7 @@ class DeezerClient:
break # Last page
offset += len(data['data'])
cache = get_metadata_cache()
# Deezer's /artist/{id}/albums endpoint doesn't include artist info on each album.
# Inject it so cached album entities have artist_name for discover page display.
artist_stub = None
@ -980,11 +914,6 @@ class DeezerClient:
entries.append((str(ad['id']), ad))
if entries:
cache.store_entities_bulk('deezer', 'album', entries, skip_if_exists=True)
# Only cache the artist→album-LIST when pagination finished cleanly; a
# partial list would otherwise serve an incomplete discography until TTL.
# (Individual album entities above are complete, so they cache regardless.)
if complete:
store_artist_album_items(cache, 'deezer', artist_id, all_raw, album_type=album_type, limit=limit)
logger.info(f"Retrieved {len(albums)} albums for artist {artist_id}")
return albums[:limit]
@ -1399,16 +1328,6 @@ class DeezerClient:
raw_tracks.extend(page_tracks)
# Real album track positions — playlist tracks don't carry track_position,
# so numbering by playlist index would poison the downloaded file's tag.
album_ids = {str(t.get('album', {}).get('id')) for t in raw_tracks if t.get('album', {}).get('id')}
try:
from core.metadata.cache import get_metadata_cache
_cache = get_metadata_cache()
except Exception:
_cache = None
track_positions = resolve_album_track_positions(self.session, self.BASE_URL, album_ids, _cache)
# Normalize tracks
tracks: List[Dict[str, Any]] = []
for i, t in enumerate(raw_tracks, start=1):
@ -1420,8 +1339,7 @@ class DeezerClient:
'artists': [artist_name],
'album': t.get('album', {}).get('title', ''),
'duration_ms': t.get('duration', 0) * 1000,
# REAL album position; the playlist index is a last resort only.
'track_number': track_positions.get(str(t.get('id'))) or i,
'track_number': i,
})
result = {

125
core/deezer_download_client.py
View file

@ -21,7 +21,6 @@ from typing import Any, Dict, List, Optional, Tuple
import requests
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.quality.source_map import quality_from_deezer, quality_tier_for_source
from utils.logging_config import get_logger
logger = get_logger("deezer_download")
@ -93,10 +92,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if download_path is None:
download_path = config_manager.get('soulseek.download_path', './downloads')
self.download_path = Path(download_path)
try:
self.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {self.download_path}: {e}")
self.download_path.mkdir(parents=True, exist_ok=True)
# Engine reference is populated by set_engine() at registration
# time. None until orchestrator wires the registry.
@ -121,20 +117,14 @@ class DeezerDownloadClient(DownloadSourcePlugin):
self._license_token = None
self._user_data = None
self._authenticated = False
self._pending_arl: Optional[str] = None
# Quality preference
self._quality = quality_tier_for_source('deezer', default='flac')
self._quality = config_manager.get('deezer_download.quality', 'flac')
# Try to authenticate on init if ARL is configured
arl = config_manager.get('deezer_download.arl', '')
if arl:
from core.boot_phase import is_boot_phase
if is_boot_phase():
self._pending_arl = arl
logger.debug("Deezer ARL present — authentication deferred until after boot")
else:
self._authenticate(arl)
self._authenticate(arl)
logger.info(f"Deezer download client initialized (download path: {self.download_path})")
@ -233,66 +223,12 @@ class DeezerDownloadClient(DownloadSourcePlugin):
return self._authenticated
def is_authenticated(self) -> bool:
if self._pending_arl and not self._authenticated:
from core.boot_phase import is_boot_phase
if not is_boot_phase():
self._authenticate(self._pending_arl)
self._pending_arl = None
return self._authenticated
async def check_connection(self) -> bool:
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.is_available)
# ─── Playlist export (#945) ──────────────────────────────────
#
# UNOFFICIAL: rides the private gw-light gateway with the ARL session already used
# for downloads. Deezer shut their public developer API, so this is the only write
# path — and it's fragile by nature (breaks when Deezer changes internals).
def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
public=False, description=""):
"""Create a Deezer playlist (or append to an existing one) from a mirrored
playlist's tracks. ``track_ids`` are stored ``deezer_id`` values per library track.
``existing_id`` set → add to that playlist (idempotent re-export reuses the stored
target); unset → create a new one. Returns
``{success, playlist_id, url, added, error}``."""
if not self._authenticated:
return {"success": False, "error": "Deezer is not connected (ARL)"}
song_ids = [str(t) for t in (track_ids or []) if t]
if not song_ids:
return {"success": False, "error": "No matching Deezer tracks to export"}
try:
songs = [[sid, i] for i, sid in enumerate(song_ids)]
if existing_id:
res = self._gw_call("playlist.addSongs",
{"playlist_id": int(existing_id), "songs": songs})
if res is None:
return {"success": False, "error": "Deezer rejected the playlist update"}
playlist_id = existing_id
else:
res = self._gw_call("playlist.create", {
"title": name, "description": description,
"is_public": bool(public), "songs": songs,
})
if res is None:
return {"success": False, "error": "Deezer rejected the playlist create"}
# gw 'playlist.create' returns the new playlist id (int) as `results`.
if isinstance(res, dict):
playlist_id = res.get("PLAYLIST_ID") or res.get("id")
else:
playlist_id = res
if not playlist_id:
return {"success": False, "error": "Deezer did not return a playlist id"}
return {
"success": True,
"playlist_id": str(playlist_id),
"url": f"https://www.deezer.com/playlist/{playlist_id}",
"added": len(song_ids),
}
except Exception as e:
return {"success": False, "error": str(e)}
def reconnect(self, arl: str = None) -> bool:
"""Re-authenticate with a new or existing ARL."""
if arl is None:
@ -481,12 +417,6 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if aid:
album_ids.add(str(aid))
album_release_dates = {}
# Deezer PLAYLIST tracks do NOT carry `track_position` (only `/track/<id>`
# and `/album/<id>/tracks` do), so numbering them by their playlist index
# poisons the real album track number — which then rides into the wishlist
# and onto the downloaded file's tag (e.g. 'Apologize' tagged track 1 instead
# of 16). Resolve the REAL position from each album's track list (cache-first).
track_positions: Dict[str, int] = {} # str(track_id) -> album track_position
try:
from core.metadata.cache import get_metadata_cache
cache = get_metadata_cache()
@ -499,32 +429,24 @@ class DeezerDownloadClient(DownloadSourcePlugin):
cached = cache.get_entity('deezer', 'album', aid)
if cached and cached.get('release_date'):
album_release_dates[aid] = cached['release_date']
continue
except Exception as e:
logger.debug("cache get_entity album release_date: %s", e)
# Cache miss — fetch from API
if aid not in album_release_dates:
try:
time.sleep(0.3) # Respect rate limits
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
if a_resp.ok:
a_data = a_resp.json()
album_release_dates[aid] = a_data.get('release_date', '')
# Store in metadata cache for future use
if cache:
try:
cache.store_entity('deezer', 'album', aid, a_data)
except Exception as e:
logger.debug("cache store_entity album release_date: %s", e)
except Exception as e:
logger.debug("fetch deezer album release_date %s: %s", aid, e)
# Real album track positions (separate endpoint — playlist tracks AND the
# album object's embedded tracks both omit track_position). Cache-first.
try:
from core.deezer_client import resolve_album_track_positions
track_positions = resolve_album_track_positions(
self._session, 'https://api.deezer.com', album_ids, cache)
except Exception as e:
logger.debug("resolve deezer album track positions: %s", e)
try:
time.sleep(0.3) # Respect rate limits
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
if a_resp.ok:
a_data = a_resp.json()
album_release_dates[aid] = a_data.get('release_date', '')
# Store in metadata cache for future use
if cache:
try:
cache.store_entity('deezer', 'album', aid, a_data)
except Exception as e:
logger.debug("cache store_entity album release_date: %s", e)
except Exception as e:
logger.debug("fetch deezer album release_date %s: %s", aid, e)
tracks = []
for i, t in enumerate(raw_tracks, start=1):
@ -545,9 +467,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
'id': album_id,
},
'duration_ms': t.get('duration', 0) * 1000,
# REAL album position (resolved above); the playlist index is a last
# resort only when the album lookup failed, never the default.
'track_number': track_positions.get(str(t.get('id'))) or t.get('track_position') or i,
'track_number': i,
})
return {
@ -661,7 +581,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
bitrate = 128
quality = 'mp3'
tr = TrackResult(
results.append(TrackResult(
username='deezer_dl',
filename=f"{track_id}||{artist} - {title}",
size=est_size,
@ -675,10 +595,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
title=title,
album=album,
track_number=item.get('track_position'),
)
# Stamp CD-quality FLAC (16/44.1) so lossless ranks correctly.
tr.set_quality(quality_from_deezer(self._quality))
results.append(tr)
))
logger.info(f"Deezer search for '{query}' returned {len(results)} results")
return results, []

76
core/deezer_worker.py
View file

@ -8,15 +8,7 @@ from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.deezer_client import DeezerClient
from core.worker_utils import (
accept_artist_match,
artist_name_matches,
interruptible_sleep,
owned_album_titles,
pick_artist_by_catalog,
release_titles,
set_album_api_track_count,
)
from core.worker_utils import interruptible_sleep, set_album_api_track_count
from core.enrichment.manual_match_honoring import honor_stored_match
logger = get_logger("deezer_worker")
@ -171,16 +163,6 @@ class DeezerWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('deezer')
if _prio:
_pi = priority_pending_item(cursor, 'deezer', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name
@ -286,19 +268,10 @@ class DeezerWorker:
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
return similarity >= self.name_similarity_threshold
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id,
result_artist_name: Optional[str] = None) -> bool:
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id) -> bool:
"""Verify that the result's artist ID matches the parent artist's stored Deezer ID.
If mismatched, the album/track search is more specific (uses artist+title),
so we trust it and correct the parent artist's deezer_id — BUT only when
the result's artist *name* actually matches our parent artist. Without
that guard, a collaboration or compilation track (e.g. a track our
library credits to Jorja Smith that lives on Kendrick Lamar's curated
"Black Panther" album) would search up to an album whose Deezer primary
artist is someone else (Kendrick), and we'd stamp that wrong Deezer ID
onto our artist — corrupting it (and causing duplicate ids shared across
unrelated artists)."""
so we trust it and correct the parent artist's deezer_id."""
parent_deezer_id = item.get('artist_deezer_id')
if not parent_deezer_id:
return True
@ -307,20 +280,6 @@ class DeezerWorker:
return True
if str(result_artist_id) != str(parent_deezer_id):
# Guard: only correct when the album/track's primary artist is the
# SAME artist by name. A mismatch means it's a collab/compilation,
# not a stale-id correction.
parent_name = item.get('artist') or ''
if (result_artist_name and parent_name
and not self._name_matches(parent_name, result_artist_name)):
logger.info(
f"Skipping artist-ID correction from {item['type']} "
f"'{item['name']}': result artist '{result_artist_name}' "
f"≠ parent '{parent_name}' (collab/compilation, not a "
f"correction)"
)
return True
logger.info(
f"Artist ID correction from {item['type']} '{item['name']}': "
f"updating parent artist Deezer ID from {parent_deezer_id} to {result_artist_id}"
@ -409,34 +368,17 @@ class DeezerWorker:
logger.debug(f"Preserving existing Deezer ID for artist '{artist_name}': {existing_id}")
return
# Multi-candidate search (was single search_artist) so same-name artists
# can be disambiguated: gate by name, then pick the one whose catalog
# overlaps the albums this library owns.
results = self.client.search_artists(artist_name, limit=5)
gated = [a for a in (results or []) if artist_name_matches(artist_name, getattr(a, 'name', ''))]
chosen, _overlap = pick_artist_by_catalog(
gated,
owned_album_titles(self.db, artist_id),
lambda a: release_titles(self.client.get_artist_albums_list(a.id)),
)
# search_artists returns lean Artist objects; fetch the full dict (same
# shape the old search_artist returned) for storage.
result = self.client.get_artist_info(chosen.id) if chosen else None
result = self.client.search_artist(artist_name)
if result:
result_name = result.get('name', '')
ok, reason = accept_artist_match(
self.db, 'deezer_id', result.get('id'), artist_id,
artist_name, result_name,
)
if ok:
if self._name_matches(artist_name, result_name):
self._update_artist(artist_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{artist_name}' -> Deezer ID: {result.get('id')}")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Artist '{artist_name}' not matched: {reason}")
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result_name}')")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
@ -478,8 +420,7 @@ class DeezerWorker:
# Verify artist ID
result_artist = result.get('artist', {})
result_artist_id = result_artist.get('id') if result_artist else None
result_artist_name = result_artist.get('name') if result_artist else None
self._verify_artist_id(item, result_artist_id, result_artist_name)
self._verify_artist_id(item, result_artist_id)
# Fetch full album details for label, genres, explicit
deezer_album_id = result.get('id')
@ -530,8 +471,7 @@ class DeezerWorker:
# Verify artist ID
result_artist = result.get('artist', {})
result_artist_id = result_artist.get('id') if result_artist else None
result_artist_name = result_artist.get('name') if result_artist else None
self._verify_artist_id(item, result_artist_id, result_artist_name)
self._verify_artist_id(item, result_artist_id)
# Fetch full track details for BPM
deezer_track_id = result.get('id')

0
core/diagnostics/__init__.py
View file

136
core/diagnostics/memory_tracker.py
View file

@ -1,136 +0,0 @@
"""On-demand memory-growth diagnostic (issue #802: ~0.7 MiB/s RSS growth).
Wraps ``tracemalloc`` so a user seeing runaway memory can capture WHERE the
allocations come from instead of us guessing:
1. start_tracking() — begins tracing + stores a baseline snapshot
2. ...reproduce the growth for a few minutes...
3. report() — top allocation sites, with the DELTA since baseline
(the delta is the leak; absolute sizes are mostly
startup noise)
4. stop_tracking() — ends tracing, frees trace memory
Opt-in by design: tracemalloc costs CPU and memory while active (it shadows
every allocation), so it must never run by default. The Flask endpoints that
expose this live in web_server (GET /api/debug/memory/...) so a user can drive
the whole flow from a browser.
"""
from __future__ import annotations
import os
import time
import tracemalloc
from typing import Any, Dict, List, Optional
from utils.logging_config import get_logger
logger = get_logger("diagnostics.memory")
_baseline: Optional[tracemalloc.Snapshot] = None
_started_at: Optional[float] = None
# Allocation-site traces this deep give useful "who called it" context without
# pathological overhead.
_TRACE_FRAMES = 15
def is_tracking() -> bool:
return tracemalloc.is_tracing()
def start_tracking() -> Dict[str, Any]:
"""Begin tracing and store the baseline snapshot. Idempotent."""
global _baseline, _started_at
if tracemalloc.is_tracing():
return {"tracking": True, "already_running": True, "started_at": _started_at}
tracemalloc.start(_TRACE_FRAMES)
_baseline = tracemalloc.take_snapshot()
_started_at = time.time()
logger.info("Memory tracking started (tracemalloc, %d frames)", _TRACE_FRAMES)
return {"tracking": True, "already_running": False, "started_at": _started_at}
def stop_tracking() -> Dict[str, Any]:
"""End tracing and free the trace bookkeeping."""
global _baseline, _started_at
was = tracemalloc.is_tracing()
if was:
tracemalloc.stop()
logger.info("Memory tracking stopped")
_baseline = None
_started_at = None
return {"tracking": False, "was_tracking": was}
def _rss_mb() -> Optional[float]:
"""Process RSS in MiB, best-effort (psutil, then /proc fallback)."""
try:
import psutil
return round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 1)
except Exception: # noqa: S110 — RSS is optional context; fall through to /proc
pass
try:
with open("/proc/self/status", encoding="utf-8") as fh:
for line in fh:
if line.startswith("VmRSS:"):
return round(int(line.split()[1]) / 1024, 1)
except Exception: # noqa: S110 — no /proc on this platform; RSS stays None
pass
return None
def format_stat(stat: Any) -> Dict[str, Any]:
"""Project one tracemalloc StatisticDiff/Statistic into a plain dict.
Duck-typed (reads size/count/size_diff/count_diff/traceback) so it's
unit-testable without real snapshots."""
tb = getattr(stat, "traceback", None)
frames: List[str] = []
if tb:
# Most-recent-call-last reads naturally top-down in a report.
for frame in list(tb)[-3:]:
frames.append(f"{frame.filename}:{frame.lineno}")
return {
"location": frames[-1] if frames else "?",
"trace": frames,
"size_mb": round(getattr(stat, "size", 0) / (1024 * 1024), 3),
"size_diff_mb": round(getattr(stat, "size_diff", 0) / (1024 * 1024), 3),
"count": getattr(stat, "count", 0),
"count_diff": getattr(stat, "count_diff", 0),
}
def report(top: int = 25) -> Dict[str, Any]:
"""Current snapshot vs the start_tracking() baseline: the top allocation
sites by GROWTH (size_diff). Includes traced totals + process RSS so the
user can see how much of the real growth tracemalloc accounts for."""
if not tracemalloc.is_tracing():
return {
"tracking": False,
"rss_mb": _rss_mb(),
"hint": "Start with /api/debug/memory/start, reproduce the growth "
"for a few minutes, then call this again.",
}
snapshot = tracemalloc.take_snapshot()
# Filter the tracer's own bookkeeping out of the picture.
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, tracemalloc.__file__),
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
))
current, peak = tracemalloc.get_traced_memory()
if _baseline is not None:
stats = snapshot.compare_to(_baseline, "traceback")
stats.sort(key=lambda s: s.size_diff, reverse=True)
else:
stats = snapshot.statistics("traceback")
return {
"tracking": True,
"started_at": _started_at,
"elapsed_seconds": round(time.time() - _started_at, 1) if _started_at else None,
"traced_current_mb": round(current / (1024 * 1024), 1),
"traced_peak_mb": round(peak / (1024 * 1024), 1),
"rss_mb": _rss_mb(),
"top_growth": [format_stat(s) for s in stats[:top]],
}

173
core/discogs_client.py
View file

@ -12,7 +12,6 @@ import re
import time
import threading
import requests
from core.metadata.artist_album_cache import get_cached_artist_album_payload, store_artist_album_items
from core.metadata.cache import get_metadata_cache
from typing import List, Dict, Any, Optional
from dataclasses import dataclass
@ -62,69 +61,6 @@ def rate_limited(func):
return wrapper
# Discogs disambiguates duplicate artist names two ways: numeric `(N)`
# suffix on the older convention (e.g. "Bullet (2)") and a trailing `*`
# on the newer convention (e.g. "John Smith*"). Both are presentation-
# only — the underlying canonical name is what users expect to see in
# search results, on cards, and especially in import paths (otherwise
# library folders end up named `Foo*` on disk). Strip both at every
# point a Discogs payload becomes a name string.
_DISCOGS_DISAMBIG_RE = re.compile(r'(?:\s*\(\d+\))?\s*\*+\s*$|\s*\(\d+\)\s*$')
def _clean_discogs_artist_name(name: Optional[str]) -> str:
"""Strip Discogs disambiguation suffixes — both `(N)` and trailing
`*` — from an artist name. Returns '' for None / empty input."""
if not name:
return ''
return _DISCOGS_DISAMBIG_RE.sub('', name).strip()
# --- Discogs album ID typing -------------------------------------------------
# Discogs has two album object types — masters (/masters/{id}) and releases
# (/releases/{id}) — whose numeric IDs share one space, so release N and master
# N are DIFFERENT albums. A bare numeric ID is therefore ambiguous. We tag the
# type into the ID string ('m12345' / 'r12345') at the point we parse it, so the
# correct endpoint can be chosen later without guessing. (Artist IDs are a single
# namespace and stay untagged.)
def _discogs_album_kind(data: Dict[str, Any]) -> str:
"""Classify a Discogs album payload as 'master' or 'release'.
Search results and artist-discography items carry an explicit ``type``;
full detail responses don't, but only master detail has ``main_release``."""
t = (data.get('type') or '').lower()
if t in ('master', 'release'):
return t
return 'master' if 'main_release' in data else 'release'
def _tag_discogs_album_id(raw_id: Any, kind: str) -> str:
"""``'12345'`` + ``'master'`` -> ``'m12345'``; empty input -> ``''``."""
s = str(raw_id or '').strip()
if not s:
return ''
return f"{'m' if kind == 'master' else 'r'}{s}"
def _discogs_album_endpoints(album_id: Any) -> List[str]:
"""Map a (possibly tagged) album ID to the API path(s) to try, in order.
``'m12345'`` -> ``['/masters/12345']``
``'r12345'`` -> ``['/releases/12345']``
``'12345'`` (legacy untagged) -> ``['/releases/12345', '/masters/12345']``
Legacy bare IDs are tried release-first because stored IDs originate
overwhelmingly from search / manual-match / collection sync (all releases);
this also self-heals pre-fix bad matches. Returns ``[]`` for unusable input."""
s = str(album_id or '').strip()
if len(s) > 1 and s[0] in ('m', 'r') and s[1:].isdigit():
return [f"/{'masters' if s[0] == 'm' else 'releases'}/{s[1:]}"]
if s.isdigit():
return [f'/releases/{s}', f'/masters/{s}']
return []
# --- Shared dataclasses (same shape as iTunes/Deezer/Spotify) ---
@dataclass
@ -181,10 +117,9 @@ class Track:
# Artists from track-level or release-level
track_artists = []
if track_data.get('artists'):
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in track_data['artists'] if a.get('name')]
track_artists = [a.get('name', '') for a in track_data['artists'] if a.get('name')]
if not track_artists and release.get('artists'):
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release['artists'] if a.get('name')]
track_artists = [a for a in track_artists if a]
track_artists = [a.get('name', '') for a in release['artists'] if a.get('name')]
if not track_artists:
track_artists = ['Unknown Artist']
@ -255,9 +190,7 @@ class Artist:
return cls(
id=str(artist_data.get('id', '')),
name=_clean_discogs_artist_name(
artist_data.get('name', artist_data.get('title', ''))
),
name=artist_data.get('name', artist_data.get('title', '')),
popularity=0,
genres=[],
followers=0,
@ -283,15 +216,14 @@ class Album:
artists = []
title = release_data.get('title', '')
if release_data.get('artists'):
artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release_data['artists'] if a.get('name')]
artists = [a.get('name', '') for a in release_data['artists'] if a.get('name')]
elif release_data.get('artist'):
artists = [_clean_discogs_artist_name(release_data['artist'])]
artists = [release_data['artist']]
elif ' - ' in title:
# Search results: "Radiohead - OK Computer" → artist="Radiohead", title="OK Computer"
parts = title.split(' - ', 1)
artists = [_clean_discogs_artist_name(parts[0])]
artists = [parts[0].strip()]
title = parts[1].strip()
artists = [a for a in artists if a]
if not artists:
artists = ['Unknown Artist']
@ -350,7 +282,7 @@ class Album:
external_urls['discogs_api'] = release_data['resource_url']
return cls(
id=_tag_discogs_album_id(release_data.get('id', ''), _discogs_album_kind(release_data)),
id=str(release_data.get('id', '')),
name=title,
artists=artists,
release_date=release_date,
@ -512,8 +444,8 @@ class DiscogsClient:
artist_name = ''
if artists and isinstance(artists[0], dict):
artist_name = (artists[0].get('name') or '').strip()
# Strip Discogs disambiguation suffixes — both `(N)` and `*`.
artist_name = _clean_discogs_artist_name(artist_name)
# Strip trailing "(N)" disambiguation suffix Discogs adds.
artist_name = re.sub(r'\s*\(\d+\)$', '', artist_name)
if not title or not artist_name:
continue
@ -689,13 +621,10 @@ class DiscogsClient:
if cached and cached.get('title'):
data = cached
else:
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('title'):
break
data = None
# Try as master first (artist discography returns master IDs)
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('title'):
data = self._api_get(f'/releases/{release_id}')
if not data:
return None
cache.store_entity('discogs', 'album', release_id, data)
@ -729,45 +658,22 @@ class DiscogsClient:
def get_artist_albums(self, artist_id: str, album_type: str = 'album,single', limit: int = 50) -> List[Album]:
"""Get releases by an artist. Prefers master releases, filters features."""
cache = get_metadata_cache()
cached_payload = get_cached_artist_album_payload(cache, 'discogs', artist_id, album_type=album_type, limit=limit)
releases = cached_payload.get('_releases') if cached_payload else None
artist_name = ''
if cached_payload:
artist_name = str(cached_payload.get('artist_name') or '').lower()
# First get the artist name for feature filtering
artist_data = self._api_get(f'/artists/{artist_id}')
artist_name = artist_data.get('name', '').lower() if artist_data else ''
if not isinstance(releases, list) or not releases:
# First get the artist name for feature filtering. Strip Discogs
# disambiguation suffix so feature-vs-primary matching below
# compares against the canonical name, not "Beyoncé*".
artist_data = self._api_get(f'/artists/{artist_id}')
artist_name = _clean_discogs_artist_name(
artist_data.get('name', '') if artist_data else ''
).lower()
data = self._api_get(f'/artists/{artist_id}/releases', {
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
})
if not data or not data.get('releases'):
return []
releases = data.get('releases') or []
store_artist_album_items(
cache,
'discogs',
artist_id,
releases,
album_type=album_type,
limit=limit,
items_field='_releases',
extra_fields={'artist_name': artist_name},
)
data = self._api_get(f'/artists/{artist_id}/releases', {
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
})
if not data or not data.get('releases'):
return []
# Separate masters from individual releases — prefer masters (canonical versions)
masters = []
releases_no_master = []
master_titles = set()
for item in releases:
for item in data['releases']:
# Skip non-main roles
role = item.get('role', 'Main').lower()
if role not in ('main', ''):
@ -835,13 +741,10 @@ class DiscogsClient:
if cached:
return cached
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('tracklist'):
break
data = None
# Try as master first (master IDs are used in artist discography)
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('tracklist'):
data = self._api_get(f'/releases/{release_id}')
if not data or not data.get('tracklist'):
return None
@ -853,7 +756,7 @@ class DiscogsClient:
image_url = (primary or images[0]).get('uri')
album_info = {
'id': str(release_id),
'id': str(data.get('id', release_id)),
'name': data.get('title', ''),
'images': [{'url': image_url, 'height': 600, 'width': 600}] if image_url else [],
'release_date': str(data.get('year', '')) if data.get('year') else '',
@ -862,11 +765,7 @@ class DiscogsClient:
# Get artists
artists_list = []
if data.get('artists'):
artists_list = [
{'name': _clean_discogs_artist_name(a.get('name', ''))}
for a in data['artists']
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
]
artists_list = [{'name': a.get('name', '')} for a in data['artists'] if a.get('name')]
if not artists_list:
artists_list = [{'name': 'Unknown Artist'}]
@ -895,11 +794,7 @@ class DiscogsClient:
# Per-track artists or fall back to release artists
track_artists = artists_list
if t.get('artists'):
track_artists = [
{'name': _clean_discogs_artist_name(a.get('name', ''))}
for a in t['artists']
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
]
track_artists = [{'name': a.get('name', '')} for a in t['artists'] if a.get('name')]
tracks.append({
'id': f"{release_id}_t{track_num}",
@ -942,13 +837,9 @@ class DiscogsClient:
cached = cache.get_entity('discogs', 'album', str(release_id))
if cached and cached.get('title'):
return cached
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('title'):
break
data = None
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('title'):
data = self._api_get(f'/releases/{release_id}')
if data:
cache.store_entity('discogs', 'album', str(release_id), data)
return data

26
core/discogs_worker.py
View file

@ -17,8 +17,8 @@ from typing import Optional, Dict, Any
from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.discogs_client import DiscogsClient, _discogs_album_kind, _tag_discogs_album_id
from core.worker_utils import accept_artist_match, interruptible_sleep, set_album_api_track_count
from core.discogs_client import DiscogsClient
from core.worker_utils import interruptible_sleep, set_album_api_track_count
logger = get_logger("discogs_worker")
@ -174,16 +174,6 @@ class DiscogsWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Discogs
# has no track endpoint, so only artist/album are honored.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('discogs')
if _prio in ('artist', 'album'):
_pi = priority_pending_item(cursor, 'discogs', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists
@ -332,13 +322,9 @@ class DiscogsWorker:
self.stats['not_found'] += 1
return
# Find best match by name similarity (skipping ids already claimed by
# a differently-named artist, so we don't create a shared/duplicate id).
# Find best match by name similarity
for result in results:
ok, reason = accept_artist_match(
self.db, 'discogs_id', result.id, artist_id, artist_name, result.name,
)
if ok:
if self._name_matches(artist_name, result.name):
# Fetch full artist detail (uses cache)
data = self.client._fetch_and_cache_artist(result.id)
if data:
@ -436,9 +422,7 @@ class DiscogsWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Tag the ID with its Discogs type so later re-fetches hit the right
# endpoint (master vs release share one numeric space).
discogs_id = _tag_discogs_album_id(data.get('id', ''), _discogs_album_kind(data))
discogs_id = str(data.get('id', ''))
genres = json.dumps(data.get('genres', []))
styles = json.dumps(data.get('styles', []))
labels = data.get('labels', [])

767
core/discovery/endpoints.py
View file

@ -1,767 +0,0 @@
"""Generic, source-agnostic helpers for the playlist-discovery route layer.
The discovery/sync endpoints in ``web_server.py`` were copy-pasted once per
source (Tidal, Deezer, Qobuz, Spotify-public, iTunes-link, YouTube,
ListenBrainz, Beatport). The per-source copies differ only by a source label
string and which ``<source>_discovery_states`` global they read. This module
lifts the source-agnostic pieces into importable, unit-testable helpers so the
route functions become thin wrappers — exactly preserving behavior (1:1).
Each helper is lifted verbatim from its web_server.py counterpart; any
per-source quirk that genuinely differs (e.g. Beatport's distinct result
shape) is intentionally NOT routed through here and stays in its own function.
"""
from __future__ import annotations
import time
from typing import Any, Dict, List, Tuple
from utils.logging_config import get_logger
logger = get_logger("discovery.endpoints")
def convert_results_to_spotify_tracks(
discovery_results: List[Dict[str, Any]],
source_label: str,
) -> List[Dict[str, Any]]:
"""Convert a source's discovery results into the Spotify-track dicts the
sync pipeline expects.
Lifted verbatim from the per-source ``convert_<source>_results_to_spotify_tracks``
functions (and the already-generic ``_convert_link_results_to_spotify_tracks``),
which were byte-identical apart from the ``source_label`` used in the log
line. Two input shapes are supported, matching the originals exactly:
- ``spotify_data`` (manual-fix shape): copied through, preserving optional
``track_number`` / ``disc_number``.
- ``spotify_track`` + ``status_class == 'found'`` (auto-discovery shape):
rebuilt from the flat ``spotify_*`` fields.
Any result matching neither shape is skipped, identical to the originals.
NOTE: Beatport deliberately does NOT use this — its converter coerces
artist objects to strings and emits a different track shape (``source``
field, album dict), so it keeps its own implementation.
"""
spotify_tracks: List[Dict[str, Any]] = []
for result in discovery_results:
# Support both data formats: spotify_data (manual fixes) and individual
# fields (automatic discovery).
if result.get('spotify_data'):
spotify_data = result['spotify_data']
track = {
'id': spotify_data['id'],
'name': spotify_data['name'],
'artists': spotify_data['artists'],
'album': spotify_data['album'],
'duration_ms': spotify_data.get('duration_ms', 0),
}
if spotify_data.get('track_number'):
track['track_number'] = spotify_data['track_number']
if spotify_data.get('disc_number'):
track['disc_number'] = spotify_data['disc_number']
spotify_tracks.append(track)
elif result.get('spotify_track') and result.get('status_class') == 'found':
spotify_tracks.append({
'id': result.get('spotify_id', 'unknown'),
'name': result.get('spotify_track', 'Unknown Track'),
'artists': [result.get('spotify_artist', 'Unknown Artist')] if result.get('spotify_artist') else ['Unknown Artist'],
'album': result.get('spotify_album', 'Unknown Album'),
'duration_ms': 0,
})
logger.info(f"Converted {len(spotify_tracks)} {source_label} matches to Spotify tracks for sync")
return spotify_tracks
def cancel_sync(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
sync_lock: Any,
sync_states: Dict[str, Any],
active_sync_workers: Dict[str, Any],
) -> Tuple[Dict[str, Any], int]:
"""Cancel an in-progress sync for one discovery playlist.
1:1 lift of the byte-identical ``cancel_<source>_sync`` bodies (Tidal,
Deezer, Qobuz, Spotify-Public, iTunes-Link, YouTube, ListenBrainz). The
caller passes the already-resolved state key (ListenBrainz transforms it
via ``_lb_state_key`` first), the source ``label``, the exact 404 message
(iTunes-Link uses "iTunes Link not found", not "... playlist not found"),
and the shared sync infrastructure (so this stays free of web_server
globals / Flask).
Returns ``(payload_dict, status_code)``; the caller wraps in ``jsonify``.
Beatport is NOT routed here — it cancels a stored ``sync_future`` and
returns a different payload.
"""
try:
if key not in states:
# Idempotent: the live discovery state is gone (a restart wiped the
# in-memory state, or it was already cancelled). Cancelling a sync
# that isn't running is a no-op SUCCESS, not a 404 — otherwise a
# mirrored playlist (e.g. a ListenBrainz weekly) whose state vanished
# is permanently wedged with "playlist not found" and can never be
# re-synced or dismissed (#702).
return {"success": True, "message": f"No active {label} sync to cancel"}, 200
state = states[key]
state['last_accessed'] = time.time()
sync_playlist_id = state.get('sync_playlist_id')
if sync_playlist_id:
with sync_lock:
sync_states[sync_playlist_id] = {"status": "cancelled"}
if sync_playlist_id in active_sync_workers:
del active_sync_workers[sync_playlist_id]
state['phase'] = 'discovered'
state['sync_playlist_id'] = None
state['sync_progress'] = {}
return {"success": True, "message": f"{label} sync cancelled"}, 200
except Exception as e:
logger.error(f"Error cancelling {label} sync: {e}")
return {"error": str(e)}, 500
def delete_playlist_state(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
) -> Tuple[Dict[str, Any], int]:
"""Delete a discovery playlist's state entry, cancelling any active
discovery first.
1:1 lift of the byte-identical ``delete_<source>_playlist`` bodies
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
The iTunes-Link / YouTube / ListenBrainz / Beatport deletes intentionally
keep their own bodies — they differ in success message, info-log wording,
name extraction, and/or key transform.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
if 'discovery_future' in state and state['discovery_future']:
state['discovery_future'].cancel()
del states[key]
logger.info(f"Deleted {label} playlist state: {key}")
return {"success": True, "message": "Playlist deleted"}, 200
except Exception as e:
logger.error(f"Error deleting {label} playlist: {e}")
return {"error": str(e)}, 500
# --- playlist-name accessors -------------------------------------------------
# The per-source sync-status handlers read the display name three different
# ways. Each is reproduced verbatim so the 1:1 behavior (including which ones
# raise vs. fall back to 'Unknown Playlist') is preserved.
def playlist_name_attr_or_unknown(state: Dict[str, Any]) -> str:
"""Tidal: playlist is an object — use ``.name`` or 'Unknown Playlist'."""
pl = state.get('playlist')
return pl.name if pl and hasattr(pl, 'name') else 'Unknown Playlist'
def playlist_name_strict(state: Dict[str, Any]) -> str:
"""Deezer / Qobuz / Spotify-Public / iTunes-Link: strict dict access —
raises (→ 500) if 'playlist' is missing, exactly like the originals."""
return state['playlist']['name']
def playlist_name_safe(state: Dict[str, Any]) -> str:
"""YouTube / ListenBrainz: safe dict access, defaulting to 'Unknown
Playlist'."""
return state.get('playlist', {}).get('name', 'Unknown Playlist')
def playlist_name_obj(state: Dict[str, Any]) -> str:
"""Tidal start-sync: playlist is an object — strict ``.name`` (raises if
absent, exactly like the original)."""
return state['playlist'].name
def playlist_image_obj(state: Dict[str, Any]) -> str:
"""Tidal: ``getattr(playlist, 'image_url', '')`` (object attribute)."""
return getattr(state['playlist'], 'image_url', '')
def playlist_image_dict(state: Dict[str, Any]) -> str:
"""Deezer/Qobuz/Spotify-Public/YouTube: ``playlist.get('image_url', '')``
(dict access)."""
return state['playlist'].get('image_url', '')
def get_sync_status(
states: Dict[str, Any],
key: str,
*,
not_found_message: str,
error_label: str,
activity_subject: str,
playlist_name_getter,
sync_lock: Any,
sync_states: Dict[str, Any],
add_activity_item,
) -> Tuple[Dict[str, Any], int]:
"""Report sync status for one discovery playlist, posting an activity-feed
item when the sync finishes or errors.
1:1 lift of the ``get_<source>_sync_status`` bodies (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz). Per-source variation
is captured by the parameters:
- ``not_found_message`` — the 404 string (iTunes-Link drops "playlist").
- ``error_label`` — used in the except log ("Error getting <X> sync status").
- ``activity_subject`` — the activity-feed prefix; note Spotify-Public uses
"Spotify Link playlist" while its error_label is "Spotify Public".
- ``playlist_name_getter`` — one of the accessors above (attr/strict/safe);
the strict one can raise, matching the originals (→ 500). The state's
phase/sync_progress are mutated BEFORE the name is read, so a raising
getter leaves the same partial mutation the original did.
Beatport is NOT routed here — it returns a different payload (``status``
not ``sync_status``, includes ``sync_id``, no lock, ``chart`` key).
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
sync_playlist_id = state.get('sync_playlist_id')
if not sync_playlist_id:
return {"error": "No sync in progress"}, 404
with sync_lock:
sync_state = sync_states.get(sync_playlist_id, {})
response = {
'phase': state['phase'],
'sync_status': sync_state.get('status', 'unknown'),
'progress': sync_state.get('progress', {}),
'complete': sync_state.get('status') == 'finished',
'error': sync_state.get('error'),
}
if sync_state.get('status') == 'finished':
state['phase'] = 'sync_complete'
state['sync_progress'] = sync_state.get('progress', {})
playlist_name = playlist_name_getter(state)
add_activity_item("", "Sync Complete", f"{activity_subject} '{playlist_name}' synced successfully", "Now")
elif sync_state.get('status') == 'error':
state['phase'] = 'discovered'
playlist_name = playlist_name_getter(state)
add_activity_item("", "Sync Failed", f"{activity_subject} '{playlist_name}' sync failed", "Now")
return response, 200
except Exception as e:
logger.error(f"Error getting {error_label} sync status: {e}")
return {"error": str(e)}, 500
def get_discovery_status(
states: Dict[str, Any],
key: str,
*,
not_found_message: str,
error_label: str,
) -> Tuple[Dict[str, Any], int]:
"""Report real-time discovery progress/results for one playlist.
1:1 lift of the byte-identical ``get_<source>_discovery_status`` bodies.
Unlike sync-status, this shape is identical for ALL eight sources —
Beatport included — so it folds in too. Only the 404 message
(".../discovery not found" vs ".../playlist not found" vs "Beatport chart
not found") and the except-log label vary, both passed in. The caller
resolves the key (ListenBrainz via ``_lb_state_key``).
Returns ``(payload, status_code)``.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
return {
'phase': state['phase'],
'status': state['status'],
'progress': state['discovery_progress'],
'spotify_matches': state['spotify_matches'],
'spotify_total': state['spotify_total'],
'results': state['discovery_results'],
'complete': state['phase'] == 'discovered',
}, 200
except Exception as e:
logger.error(f"Error getting {error_label} discovery status: {e}")
return {"error": str(e)}, 500
def reset_playlist(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
) -> Tuple[Dict[str, Any], int]:
"""Reset a discovery playlist back to the 'fresh' phase, clearing all
discovery/sync data while preserving the original playlist payload.
1:1 lift of the byte-identical ``reset_<source>_playlist`` bodies
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
NOT folded in (genuinely divergent): YouTube (status -> 'parsed', no
download_process_id, logs the playlist name, "reset to fresh state"),
ListenBrainz (status -> 'cached', logs playlist title, returns
{"phase": "fresh"}), iTunes-Link (uses state.update, no info log, distinct
message). Those keep their own bodies.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
if 'discovery_future' in state and state['discovery_future']:
state['discovery_future'].cancel()
state['phase'] = 'fresh'
state['status'] = 'fresh'
state['discovery_results'] = []
state['discovery_progress'] = 0
state['spotify_matches'] = 0
state['sync_playlist_id'] = None
state['converted_spotify_playlist_id'] = None
state['download_process_id'] = None
state['sync_progress'] = {}
state['discovery_future'] = None
state['last_accessed'] = time.time()
logger.info(f"Reset {label} playlist to fresh: {key}")
return {"success": True, "message": "Playlist reset to fresh phase"}, 200
except Exception as e:
logger.error(f"Error resetting {label} playlist: {e}")
return {"error": str(e)}, 500
def get_playlist_states(
states: Dict[str, Any],
*,
error_label: str,
info_log_label: str = None,
) -> Tuple[Dict[str, Any], int]:
"""Return all stored discovery states for a source as a list for frontend
card hydration (``{"states": [...]}``).
1:1 lift of the ``get_<source>_playlist_states`` bodies (Tidal, Deezer,
Qobuz, Spotify-Public, iTunes-Link), which build the same per-entry dict.
iTunes-Link is the only one without the "Returning N ..." info log, so
``info_log_label`` is optional (pass None to suppress it, as iTunes did).
NOT folded in: the YouTube/ListenBrainz ``get_all_*_playlists`` endpoints —
they return ``{"playlists": [...]}`` (different key + fields: url/created_at,
no discovery_results) and filter mirrored/profile-scoped entries.
"""
try:
result = []
current_time = time.time()
for key, state in states.items():
state['last_accessed'] = current_time
result.append({
'playlist_id': key,
'phase': state['phase'],
'status': state['status'],
'discovery_progress': state['discovery_progress'],
'spotify_matches': state['spotify_matches'],
'spotify_total': state['spotify_total'],
'discovery_results': state['discovery_results'],
'converted_spotify_playlist_id': state.get('converted_spotify_playlist_id'),
'download_process_id': state.get('download_process_id'),
'last_accessed': state['last_accessed'],
})
if info_log_label:
logger.info(f"Returning {len(result)} stored {info_log_label} playlist states for hydration")
return {"states": result}, 200
except Exception as e:
logger.error(f"Error getting {error_label} playlist states: {e}")
return {"error": str(e)}, 500
def save_bubble_snapshot(
get_json,
*,
payload_key: str,
no_data_error: str,
snapshot_kind: str,
success_noun: str,
log_subject: str,
log_noun: str,
get_database,
get_current_profile_id,
) -> Tuple[Dict[str, Any], int]:
"""Persist a bubble/download snapshot for cross-refresh hydration.
1:1 lift of the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles),
which differ only by:
- ``payload_key`` ('downloads' for discover, 'bubbles' for the rest) and
its ``no_data_error`` message.
- ``snapshot_kind`` — the db.save_bubble_snapshot category.
- ``success_noun`` — fills "Snapshot saved with N <noun>".
- ``log_subject`` / ``log_noun`` — the info ("Saved <subject>: N <noun>")
and except ("Error saving <subject>") log lines.
Returns ``(payload, status_code)``. ``get_json`` is invoked inside the try
like the original ``request.json``.
"""
try:
from datetime import datetime
data = get_json()
if not data or payload_key not in data:
return {'success': False, 'error': no_data_error}, 400
items = data[payload_key]
db = get_database()
db.save_bubble_snapshot(snapshot_kind, items, profile_id=get_current_profile_id())
count = len(items)
logger.info(f"Saved {log_subject}: {count} {log_noun}")
return {
'success': True,
'message': f'Snapshot saved with {count} {success_noun}',
'timestamp': datetime.now().isoformat(),
}, 200
except Exception as e:
logger.error(f"Error saving {log_subject}: {e}")
import traceback
traceback.print_exc()
return {'success': False, 'error': str(e)}, 500
def update_playlist_phase(
states: Dict[str, Any],
key: str,
get_json,
*,
not_found_message: str,
error_label: str,
valid_phases: List[str],
apply_extra_fields: bool,
) -> Tuple[Dict[str, Any], int]:
"""Update a discovery playlist's phase (used when the modal closes, e.g. to
reset download_complete -> discovered).
1:1 lift of the ``update_<source>_playlist_phase`` bodies for the five
sources with the identical validation + full-message response (Tidal,
Deezer, Qobuz, Spotify-Public, YouTube). Per-source params:
- ``valid_phases`` — YouTube's list additionally includes 'parsed'.
- ``apply_extra_fields`` — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body;
Tidal/YouTube do NOT (so pass False to keep them 1:1).
- ``not_found_message`` / ``error_label``; ``get_json`` invoked inside the
try like the original ``request.get_json()``.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link — it uses ``data.get('phase')`` (no separate
"Phase not provided" 400) and returns a no-message payload.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
data = get_json()
if not data or 'phase' not in data:
return {"error": "Phase not provided"}, 400
new_phase = data['phase']
if new_phase not in valid_phases:
return {"error": f"Invalid phase. Must be one of: {', '.join(valid_phases)}"}, 400
state = states[key]
old_phase = state.get('phase', 'unknown')
state['phase'] = new_phase
state['last_accessed'] = time.time()
if apply_extra_fields:
if 'download_process_id' in data:
state['download_process_id'] = data['download_process_id']
if 'converted_spotify_playlist_id' in data:
state['converted_spotify_playlist_id'] = data['converted_spotify_playlist_id']
logger.info(f"Updated {error_label} playlist {key} phase: {old_phase} → {new_phase}")
return {"success": True, "message": f"Phase updated to {new_phase}", "old_phase": old_phase, "new_phase": new_phase}, 200
except Exception as e:
logger.error(f"Error updating {error_label} playlist phase: {e}")
return {"error": str(e)}, 500
def first_artist_str_or_obj(original_track: Dict[str, Any]) -> str:
"""Tidal: first artist from an artists list that may hold strings OR
objects ({'name': ...}); '' when empty."""
artists = original_track.get('artists', [])
if artists:
return artists[0] if isinstance(artists[0], str) else artists[0].get('name', '')
return ''
def first_artist_plain(original_track: Dict[str, Any]) -> str:
"""Deezer/Qobuz/Spotify-Public: first artist assuming a list of strings;
'' when empty."""
artists = original_track.get('artists', [])
return artists[0] if artists else ''
def update_discovery_match(
states: Dict[str, Any],
get_json,
*,
source_log_label: str,
error_label: str,
original_track_key: str,
original_artist_getter,
join_artist_names,
extract_artist_name,
build_fix_modal_spotify_data,
get_discovery_cache_key,
get_database,
get_active_discovery_source,
) -> Tuple[Dict[str, Any], int]:
"""Apply a manually-selected Spotify track to a discovery result (the
fix-modal flow) and persist it to the discovery cache.
1:1 lift of the ``update_<source>_discovery_match`` bodies for the four
sources with the identical structure (Tidal, Deezer, Qobuz, Spotify-Public).
Per-source pieces are params:
- ``source_log_label`` (lowercase, e.g. "tidal") for the "Manual match
updated: ..." line; ``error_label`` for the except log.
- ``original_track_key`` — the raw-source track key on the result
('tidal_track', 'deezer_track', ...).
- ``original_artist_getter`` — Tidal handles string-or-object artists
(``first_artist_str_or_obj``); the rest assume strings
(``first_artist_plain``).
- the web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) are injected so this
stays free of those globals.
- ``get_json`` is called INSIDE the try (like the original's
``request.get_json()``) so a malformed body yields the same 500.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link (saves spotify_data directly via a different
cache signature), YouTube (multi-key original_track fallback), ListenBrainz
(entirely different unmatch-capable structure, no cache write), Beatport.
"""
try:
data = get_json()
identifier = data.get('identifier')
track_index = data.get('track_index')
spotify_track = data.get('spotify_track')
if not identifier or track_index is None or not spotify_track:
return {'error': 'Missing required fields'}, 400
state = states.get(identifier)
result = None
if state:
if track_index >= len(state['discovery_results']):
return {'error': 'Invalid track index'}, 400
result = state['discovery_results'][track_index]
old_status = result.get('status')
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = spotify_track['name']
result['spotify_artist'] = join_artist_names(spotify_track['artists']) if isinstance(spotify_track['artists'], list) else extract_artist_name(spotify_track['artists'])
result['spotify_album'] = spotify_track['album']
result['spotify_id'] = spotify_track['id']
duration_ms = spotify_track.get('duration_ms', 0)
if duration_ms:
minutes = duration_ms // 60000
seconds = (duration_ms % 60000) // 1000
result['duration'] = f"{minutes}:{seconds:02d}"
else:
result['duration'] = '0:00'
result['spotify_data'] = build_fix_modal_spotify_data(spotify_track)
result['wing_it_fallback'] = False
result['manual_match'] = True
if old_status != 'found' and old_status != 'Found':
state['spotify_matches'] = state.get('spotify_matches', 0) + 1
logger.info(f"Manual match updated: {source_log_label} - {identifier} - track {track_index}")
logger.info(f" → {result['spotify_artist']} - {result['spotify_track']}")
original_track = result.get(original_track_key, {})
original_name = original_track.get('name', spotify_track['name'])
original_artist = original_artist_getter(original_track)
else:
# #843: the in-memory discovery state can be gone — a server restart,
# or an imported playlist that wasn't discovered in THIS process —
# while the card is still shown from persisted data. The DURABLE part
# of a manual fix (writing the match to the discovery cache so future
# syncs resolve it) doesn't need the in-memory state, only the original
# track's name + artist, which the client now sends. Fall back to those
# instead of 404ing the fix into uselessness.
original_name = (data.get('original_name') or '').strip()
original_artist = (data.get('original_artist') or '').strip()
if not original_name and not original_artist:
return {'error': 'Discovery state not found'}, 404
if not original_name:
original_name = spotify_track['name']
# Key the cache by the FIRST artist — every in-memory + sync path uses
# artists[0], but the client may send a joined "A, B, C" string. Without
# this, a multi-artist track would save under a key the sync never looks
# up (full string ≠ first artist), so the fix would silently never apply.
if original_artist:
original_artist = original_artist.split(',')[0].strip()
logger.info(
f"Manual match (no in-memory state) → discovery cache: "
f"{source_log_label} - {identifier} - '{original_name}' by '{original_artist}'"
)
try:
cache_key = get_discovery_cache_key(original_name, original_artist)
artists_list = spotify_track['artists']
if isinstance(artists_list, list):
artists_list = [a if isinstance(a, str) else a.get('name', '') for a in artists_list]
image_url = spotify_track.get('image_url') or ''
album_raw = spotify_track.get('album', '')
if isinstance(album_raw, dict):
album_obj = dict(album_raw)
if image_url and not album_obj.get('image_url'):
album_obj['image_url'] = image_url
if image_url and not album_obj.get('images'):
album_obj['images'] = [{'url': image_url}]
else:
album_obj = {'name': album_raw or ''}
if image_url:
album_obj['image_url'] = image_url
album_obj['images'] = [{'url': image_url}]
matched_data = {
'id': spotify_track['id'],
'name': spotify_track['name'],
'artists': artists_list,
'album': album_obj,
'duration_ms': spotify_track.get('duration_ms', 0),
'image_url': image_url,
'source': 'spotify',
}
cache_db = get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], get_active_discovery_source(), 1.0, matched_data,
original_name, original_artist
)
logger.info(f"Manual fix saved to discovery cache: {original_name} by {original_artist}")
except Exception as cache_err:
logger.error(f"Error saving manual fix to discovery cache: {cache_err}")
return {'success': True, 'result': result}, 200
except Exception as e:
logger.error(f"Error updating {error_label} discovery match: {e}")
return {'error': str(e)}, 500
def start_sync(
states: Dict[str, Any],
key: str,
*,
sync_id_prefix: str,
not_found_message: str,
not_ready_message: str,
convert_fn,
playlist_name_getter,
playlist_image_getter,
activity_label: str,
error_label: str,
sync_lock: Any,
sync_states: Dict[str, Any],
active_sync_workers: Dict[str, Any],
submit_sync_task,
add_activity_item,
) -> Tuple[Dict[str, Any], int]:
"""Kick off a playlist sync from a source's discovered Spotify matches.
1:1 lift of the ``start_<source>_sync`` bodies for the five sources with
the identical flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube). The
per-source pieces are parameters:
- ``sync_id_prefix`` — the ``f"{prefix}_{key}"`` sync id.
- ``convert_fn`` — the source's discovery->spotify-tracks converter.
- ``playlist_name_getter`` / ``playlist_image_getter`` — Tidal reads an
object (``.name`` / ``getattr``), the rest read a dict; lifted as the
``playlist_name_obj``/``playlist_image_obj`` vs ``playlist_name_strict``/
``playlist_image_dict`` accessors.
- ``activity_label`` vs ``error_label`` — these DIFFER for Spotify-Public:
activity says "Spotify Link Sync Started" while logs say "Spotify Public".
- ``submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks,
playlist_image_url) -> Future`` — wraps sync_executor/_run_sync_task/
get_current_profile_id so this stays free of those globals.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task without an image arg), Beatport (extra debug logging, 'chart' key).
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
if state['phase'] not in ['discovered', 'sync_complete', 'download_complete']:
return {"error": not_ready_message}, 400
spotify_tracks = convert_fn(state['discovery_results'])
if not spotify_tracks:
return {"error": "No Spotify matches found for sync"}, 400
sync_playlist_id = f"{sync_id_prefix}_{key}"
playlist_name = playlist_name_getter(state)
add_activity_item("", f"{activity_label} Sync Started", f"'{playlist_name}' - {len(spotify_tracks)} tracks", "Now")
state['phase'] = 'syncing'
state['sync_playlist_id'] = sync_playlist_id
state['sync_progress'] = {}
with sync_lock:
sync_states[sync_playlist_id] = {"status": "starting", "progress": {}}
playlist_image_url = playlist_image_getter(state)
future = submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks, playlist_image_url)
active_sync_workers[sync_playlist_id] = future
logger.info(f"Started {error_label} sync for: {playlist_name} ({len(spotify_tracks)} tracks)")
return {"success": True, "sync_playlist_id": sync_playlist_id}, 200
except Exception as e:
logger.error(f"Error starting {error_label} sync: {e}")
return {"error": str(e)}, 500

32
core/discovery/hero.py
View file

@ -96,8 +96,6 @@ def get_discover_hero():
artist_id = artist.spotify_artist_id
elif active_source == 'deezer':
artist_id = getattr(artist, 'deezer_artist_id', None) or artist.itunes_artist_id
elif active_source == 'musicbrainz':
artist_id = getattr(artist, 'musicbrainz_artist_id', None) or artist.itunes_artist_id
else:
artist_id = artist.itunes_artist_id
if not artist_id:
@ -127,7 +125,7 @@ def get_discover_hero():
valid_artists = list(similar_artists)
# FALLBACK: If no valid artists for fallback source, try to resolve IDs on-the-fly
if active_source in ('itunes', 'deezer', 'musicbrainz') and not valid_artists:
if active_source in ('itunes', 'deezer') and not valid_artists:
logger.warning(f"[{active_source} Fallback] No artists with {active_source} IDs found, attempting on-the-fly resolution for {len(similar_artists)} artists")
resolved_count = 0
for artist in similar_artists:
@ -137,20 +135,13 @@ def get_discover_hero():
continue
# Try to resolve ID by name
try:
resolve_client = itunes_client
if active_source == 'musicbrainz':
from core.metadata.registry import get_musicbrainz_client
resolve_client = get_musicbrainz_client()
search_results = resolve_client.search_artists(artist.similar_artist_name, limit=1)
search_results = itunes_client.search_artists(artist.similar_artist_name, limit=1)
if search_results and len(search_results) > 0:
resolved_id = search_results[0].id
# Cache the resolved ID for future use
if active_source == 'deezer':
database.update_similar_artist_deezer_id(artist.id, resolved_id)
artist.similar_artist_deezer_id = resolved_id
elif active_source == 'musicbrainz':
database.update_similar_artist_musicbrainz_id(artist.id, resolved_id)
artist.similar_artist_musicbrainz_id = resolved_id
else:
database.update_similar_artist_itunes_id(artist.id, resolved_id)
artist.similar_artist_itunes_id = resolved_id
@ -182,15 +173,12 @@ def get_discover_hero():
artist_id = artist.similar_artist_spotify_id or artist.similar_artist_itunes_id
elif active_source == 'deezer':
artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
elif active_source == 'musicbrainz':
artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
else:
artist_id = artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
artist_data = {
"spotify_artist_id": artist.similar_artist_spotify_id,
"itunes_artist_id": artist.similar_artist_itunes_id,
"musicbrainz_artist_id": getattr(artist, 'similar_artist_musicbrainz_id', None),
"artist_id": artist_id,
"artist_name": artist.similar_artist_name,
"occurrence_count": artist.occurrence_count,
@ -219,19 +207,11 @@ def get_discover_hero():
artist.id, artist_data.get('image_url'),
artist_data.get('genres'), artist_data.get('popularity')
)
elif active_source in ('itunes', 'deezer', 'musicbrainz'):
if active_source == 'deezer':
fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id
fetch_client = itunes_client
elif active_source == 'musicbrainz':
fb_artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None)
from core.metadata.registry import get_musicbrainz_client
fetch_client = get_musicbrainz_client()
else:
fb_artist_id = artist.similar_artist_itunes_id
fetch_client = itunes_client
elif active_source in ('itunes', 'deezer'):
fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) if active_source == 'deezer' else None
fb_artist_id = fb_artist_id or artist.similar_artist_itunes_id
if fb_artist_id:
fb_artist_data = fetch_client.get_artist(fb_artist_id)
fb_artist_data = itunes_client.get_artist(fb_artist_id)
if fb_artist_data:
artist_data['artist_name'] = fb_artist_data.get('name', artist.similar_artist_name)
artist_data['image_url'] = fb_artist_data.get('images', [{}])[0].get('url') if fb_artist_data.get('images') else None

340
core/discovery/listening_recommendations.py
View file

@ -1,340 +0,0 @@
"""Listening-driven recommendation core (#913).
PURE, side-effect-free ranking that turns "the artists you listen to most" plus
"who's similar to each" into:
1. a consensus-ranked list of artists you'd probably love but don't own, and
2. an aggregated candidate-track list for a generated playlist.
No DB / network / config here. The caller (the watchlist scanner) supplies the
seeds (top-played artists), the ``similar_artists`` rows per seed, and the
owned-artist set, then fetches top tracks for the winners. Keeping the decision
logic in one pure place makes it fully unit-testable without the live stack and
keeps the scan wiring thin — and additive, so it can't disturb existing flows.
Scoring rationale (the "best in class" bit): a recommended artist's score is
``Σ over the seeds that recommend it of (seed_weight × similarity)``. That single
sum rewards all three signals at once — **consensus** (an artist endorsed by many
of your seeds accumulates more terms), your **play weight** (heavier seeds push
harder), and **similarity strength** — instead of a flat "appears in N lists".
``seed_count`` is exposed separately for display ("because you like A, B, C") and
as the adventurousness dial's lever (``min_seed_count``).
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Sequence, Set
def _norm(name: object) -> str:
return str(name or "").strip().lower()
def _positive_float(value: object, default: float = 1.0) -> float:
try:
f = float(value) # type: ignore[arg-type]
except (TypeError, ValueError):
return default
return f if f > 0 else default
def _get(row: object, attr: str):
"""Read a field from a dataclass row or a dict row."""
if isinstance(row, dict):
return row.get(attr)
return getattr(row, attr, None)
def choose_mix_fetch_source(active_source: object, active_can_fetch: bool) -> str:
"""Pick which source to fetch the "Listening Mix" top tracks from.
The mix is a list of (artist, title) pairs acquired via Soulseek, so the fetch source need
NOT match the user's active metadata source. Use the active source when it can fetch top
tracks itself (Spotify/Deezer); otherwise fall back to Deezer, whose public ``artist/{id}/top``
needs no auth and is available to every user — so iTunes / Discogs / MusicBrainz users still
get a full mix without switching sources. Pure.
"""
if str(active_source or "").lower() in ("spotify", "deezer") and active_can_fetch:
return str(active_source).lower()
return "deezer"
def names_match(a: object, b: object) -> bool:
"""Strict artist-name equality after stripping case + non-alphanumerics.
Used to verify a name-search result before fetching that artist's top tracks, so the
"Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact
alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell".
Pure.
"""
def _alnum(x: object) -> str:
return "".join(ch for ch in str(x or "").lower() if ch.isalnum())
na, nb = _alnum(a), _alnum(b)
return bool(na) and na == nb
def similarity_from_rank(rank: object, max_rank: int = 10) -> float:
"""Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight.
SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest
match). The ranker multiplies this into the score so a seed's *closest* matches count
for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0,
rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A
missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure.
"""
try:
r = int(rank)
except (TypeError, ValueError):
return 1.0
floor = round(1.0 / max_rank, 4)
if r <= 1:
return 1.0
if r >= max_rank:
return floor
return round((max_rank - r + 1) / max_rank, 4)
def build_recency_weighted_seeds(
top_artists: Sequence[dict],
recent_play_counts: Optional[Dict[str, float]] = None,
*,
recency_factor: float = 1.5,
) -> List[dict]:
"""Blend lifetime + recent play counts into seed weights — "what you're into NOW".
``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a
lot *recently* outranks one you played a lot years ago, so the recommendations track
your current taste instead of your all-time history. ``recency_factor`` is the dial
(0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`.
Pure — the caller supplies both play-count maps from the listening history.
"""
recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()}
out: List[dict] = []
for a in top_artists or ():
name = str(a.get("name") or "").strip()
if not name:
continue
lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0)))
boost = recency_factor * recent.get(_norm(name), 0.0)
out.append({"name": name, "weight": lifetime + boost})
return out
def group_similars_by_seed(
seeds: Sequence[dict],
similar_rows: Sequence,
id_to_name: Dict[str, str],
*,
source_id_attr: str = "source_artist_id",
similar_name_attr: str = "similar_artist_name",
rank_attr: Optional[str] = None,
) -> Dict[str, List[dict]]:
"""Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``.
The stored rows key the similar artist by the SEED's source id (``source_artist_id``),
not its name, so :func:`rank_recommended_artists` can't consume them directly. This
resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id:
artist_name}`` for the library, built by the caller) and keeps only rows that resolve
to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O.
``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that
is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id.
When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank`
and carried as ``score`` so closer matches weigh more; without it every similar comes out
score-less (the ranker then treats similarity as 1.0 — original behavior).
"""
seed_names = {_norm(s.get("name")) for s in seeds}
seed_names.discard("")
id_to_norm = {str(k): _norm(v) for k, v in (id_to_name or {}).items()}
out: Dict[str, List[dict]] = {}
for row in similar_rows or ():
seed_name = id_to_norm.get(str(_get(row, source_id_attr) or ""), "")
if not seed_name or seed_name not in seed_names:
continue
sim_name = str(_get(row, similar_name_attr) or "").strip()
if not sim_name:
continue
entry = {"name": sim_name}
if rank_attr is not None:
entry["score"] = similarity_from_rank(_get(row, rank_attr))
out.setdefault(seed_name, []).append(entry)
return out
@dataclass
class RecommendedArtist:
"""One artist recommended from your listening, with the why."""
name: str # display name (first-seen casing)
score: float # Σ seed_weight × similarity
seed_count: int # distinct seeds endorsing it (consensus)
seeds: List[str] = field(default_factory=list) # display names of those seeds
def rank_recommended_artists(
seeds: Sequence[dict],
similars_by_seed: Dict[str, Sequence[dict]],
owned_artist_names: Optional[Set[str]] = None,
*,
limit: int = 30,
min_seed_count: int = 1,
) -> List[RecommendedArtist]:
"""Rank artists similar to your most-played by consensus + play weight + similarity.
Args:
seeds: ``[{'name': str, 'weight': float}]`` — your top-played artists.
``weight`` (play count or any positive number) defaults to 1.0.
similars_by_seed: ``{seed_name_lower: [{'name': str, 'score': float}]}`` — the
similar-artist rows for each seed. ``score`` is optional (defaults 1.0).
owned_artist_names: lowercased names already in the library — excluded so the
result is artists you DON'T have. The seeds themselves are always excluded.
limit: max results.
min_seed_count: drop recommendations endorsed by fewer than N seeds — the
adventurousness dial's "Safer" end raises this for higher-confidence picks.
Returns up to ``limit`` :class:`RecommendedArtist`, highest score first.
"""
owned = {_norm(a) for a in (owned_artist_names or set())}
seed_norms = {_norm(s.get("name")) for s in seeds}
seed_norms.discard("")
exclude = owned | seed_norms
acc: Dict[str, dict] = {}
for seed in seeds:
s_name = _norm(seed.get("name"))
if not s_name:
continue
s_display = str(seed.get("name") or "").strip()
weight = _positive_float(seed.get("weight", 1.0))
for sim in similars_by_seed.get(s_name, ()) or ():
a_norm = _norm(sim.get("name"))
if not a_norm or a_norm in exclude:
continue
sim_score = _positive_float(sim.get("score", 1.0))
row = acc.setdefault(
a_norm, {"name": str(sim.get("name") or "").strip(), "score": 0.0, "seeds": {}}
)
row["score"] += weight * sim_score
row["seeds"].setdefault(s_name, s_display) # one seed counts once
out: List[RecommendedArtist] = []
floor = max(1, int(min_seed_count))
for row in acc.values():
seed_count = len(row["seeds"])
if seed_count < floor:
continue
out.append(RecommendedArtist(
name=row["name"],
score=round(row["score"], 6),
seed_count=seed_count,
seeds=list(row["seeds"].values()),
))
out.sort(key=lambda r: (-r.score, -r.seed_count, r.name.lower()))
return out[:limit]
def aggregate_candidate_tracks(
recommended_artists: Sequence[RecommendedArtist],
top_tracks_by_artist: Dict[str, Sequence[dict]],
owned_track_keys: Optional[Set] = None,
*,
per_artist: int = 3,
limit: int = 50,
exclude_owned: bool = True,
) -> List[dict]:
"""Build the candidate track list for the generated playlist.
Takes the top ``per_artist`` tracks from each recommended artist **in artist-rank
order**, dedups by ``(artist, title)``, optionally drops owned tracks (the
"discovery" flavor) and caps at ``limit``. Each returned track dict is the source
track plus ``_seed_artist`` (which recommended artist it came from).
Args:
recommended_artists: ranked output of :func:`rank_recommended_artists`.
top_tracks_by_artist: ``{artist_name_lower: [track_dict, ...]}`` — fetched by
the caller (Last.fm / source top tracks), NOT limited to a curated pool.
owned_track_keys: set of ``(artist_lower, title_lower)`` already in the library.
exclude_owned: drop tracks in ``owned_track_keys`` (discovery flavor). Set False
for a "replay" playlist of tracks you already own.
"""
owned = owned_track_keys or set()
seen: Set = set()
out: List[dict] = []
for art in recommended_artists:
tracks = top_tracks_by_artist.get(_norm(art.name), ()) or ()
taken = 0
for t in tracks:
if taken >= per_artist:
break
title = str(t.get("name") or t.get("title") or "").strip()
if not title:
continue
key = (_norm(art.name), _norm(title))
if key in seen:
continue
if exclude_owned and key in owned:
continue
seen.add(key)
out.append({**t, "_seed_artist": art.name})
taken += 1
if len(out) >= limit:
break
return out[:limit]
def to_mix_track(track: object, source: str) -> Optional[dict]:
"""Shape one source "top tracks" API dict into the flat dict the Discover compact
playlist row renders + syncs (the "Listening Mix" #913 playlist).
Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the
same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``).
This flattens that into the renderer's field names (``track_name/artist_name/album_name/
album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and
sets the source-specific id field. Returns None for anything without a usable id/title so
the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can
dedup by title. Pure — no I/O.
"""
if not isinstance(track, dict):
return None
tid = track.get("id")
name = str(track.get("name") or "").strip()
if not tid or not name:
return None
artists = track.get("artists") or []
artist_name = ""
if artists and isinstance(artists[0], dict):
artist_name = str(artists[0].get("name") or "").strip()
album = track.get("album") if isinstance(track.get("album"), dict) else {}
album_name = str(album.get("name") or "").strip()
images = album.get("images") or []
cover = images[0].get("url") if images and isinstance(images[0], dict) else None
out = {
"track_id": str(tid),
"name": name, # for aggregate_candidate_tracks dedup
"track_name": name, # for the renderer
"artist_name": artist_name,
"album_name": album_name,
"album_cover_url": cover,
"duration_ms": track.get("duration_ms") or 0,
"track_data_json": track, # full payload for sync/download
"source": source,
}
id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id",
"itunes": "itunes_track_id"}.get(source)
if id_field:
out[id_field] = str(tid)
return out
__all__ = [
"RecommendedArtist",
"choose_mix_fetch_source",
"names_match",
"similarity_from_rank",
"build_recency_weighted_seeds",
"to_mix_track",
"group_similars_by_seed",
"rank_recommended_artists",
"aggregate_candidate_tracks",
]

120
core/discovery/manual_match.py
View file

@ -1,120 +0,0 @@
"""Helpers for Fix-popup manual match persistence.
When the user manually fixes a mirrored-playlist discovery via the Fix
popup, two questions land at the web_server route layer that are easier
to test in isolation:
1. *Which metadata source did the manual match come from?* — the popup
cascade queries the user's primary source first, then Spotify /
Deezer / iTunes / MusicBrainz as fallbacks; each search endpoint
stamps `source` on its rows but the MBID-paste lookup uses a lean
flat shape that doesn't carry it. `derive_manual_match_provider`
collapses the fallback chain into a single string.
2. *Should the discovery layer re-run for this track when the current
active provider differs from the cached one?* — re-running silently
overwrites the user's deliberate pick with whatever the auto-search
ranks first, so manual matches are exempt regardless of provider
drift. `is_drifted_for_redo` encapsulates the decision.
3. *Should the Playlist Pipeline pre-scan (re)discover this track at all?*
— `should_rediscover` encapsulates that gate, with the manual match
checked FIRST so a leftover Wing It flag can't override the user's pick.
"""
from __future__ import annotations
from typing import Any, Dict, Optional
def derive_manual_match_provider(
payload_track: Dict[str, Any],
active_provider: Optional[str],
) -> str:
"""Return the provider string to stamp on a manually-fixed match.
Resolution order:
1. ``payload_track['source']`` — every *_search_tracks endpoint
sets this; the MBID-paste path doesn't.
2. ``active_provider`` — what the user has configured as their
primary discovery source.
3. ``'spotify'`` — last-ditch default matching the historic
hardcode (so behaviour is identical when both upstream
signals are absent).
"""
if not isinstance(payload_track, dict):
payload_track = {}
source = payload_track.get('source')
if source:
return source
if active_provider:
return active_provider
return 'spotify'
def is_drifted_for_redo(
extra_data: Optional[Dict[str, Any]],
active_provider: Optional[str],
) -> bool:
"""Return True when a cached discovery entry should be treated as
stale because the user's active provider has changed since it was
cached AND the entry isn't a manual match.
Manual matches are *always* considered fresh: re-running discovery
against the current source would overwrite the user's deliberate
pick with whatever auto-search ranks first. The first Playlist
Pipeline run after a manual fix used to clobber it for exactly
this reason — the check lives here now so it's pinned by tests.
"""
if not isinstance(extra_data, dict):
return False
if extra_data.get('manual_match'):
return False
cached_provider = extra_data.get('provider', 'spotify')
return cached_provider != active_provider
def should_rediscover(extra_data: Optional[Dict[str, Any]]) -> bool:
"""Return True when a mirrored track needs (re)discovery, False to skip it.
This is the gate the Playlist Pipeline pre-scan runs over every mirrored
track before discovering. The **ordering is the fix**: a manual match is
authoritative and is checked FIRST.
``extra_data`` is *merged* on save (see ``update_mirrored_track_extra_data``),
so a track that was a Wing It stub and is then manually fixed still carries
``wing_it_fallback: True`` alongside the new ``manual_match: True``. The old
pre-scan tested ``wing_it_fallback`` before ``manual_match``, so the stale
flag won and the pipeline re-discovered the track — silently reverting the
user's pick to Wing It. Checking ``manual_match`` first makes the fix stick.
Decision order:
* manual_match -> skip (authoritative; never re-discover)
* wing_it_fallback -> redo (stub — keep trying for a real match)
* discovered + complete -> skip (full metadata already stored)
* discovered + incomplete -> redo (backfill track_number / album fields)
* unmatched_by_user -> skip (user deliberately removed the match)
* never discovered -> redo (first-time discovery)
"""
extra = extra_data if isinstance(extra_data, dict) else {}
if extra.get('discovered'):
if extra.get('manual_match'):
return False
if extra.get('wing_it_fallback'):
return True
# Otherwise re-discover only when the stored match is missing the
# enriched fields (track_number + release_date/album.id) that older
# discoveries dropped via the Track dataclass.
matched = extra.get('matched_data')
matched = matched if isinstance(matched, dict) else {}
album = matched.get('album')
album = album if isinstance(album, dict) else {}
has_track_num = matched.get('track_number')
has_release = album.get('release_date')
has_album_id = album.get('id')
return not (has_track_num and (has_release or has_album_id))
if extra.get('unmatched_by_user'):
return False
return True

134
core/discovery/matching.py
View file

@ -1,134 +0,0 @@
"""Pure helper for matching raw MusicBrainz-metadata tracks against
Spotify / iTunes.
Used by the PlaylistSource adapters whose ``get_playlist`` returns
tracks with ``needs_discovery=True`` (ListenBrainz, Last.fm radio).
Phase 1b ships Strategy 1 only (matching-engine queries → search →
score → pick best ≥0.9). The richer multi-strategy +
discovery-cache flow stays in
``core.discovery.listenbrainz.run_listenbrainz_discovery_worker``
for the Discover-page state-machine UI; this helper is the slimmer
version used by the auto-refresh pipeline.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any, Callable, Dict, List, Optional
logger = logging.getLogger(__name__)
@dataclass
class MBMatchDeps:
"""Bundle of primitives the matcher needs.
Wired up at bootstrap. Tests pass stub callables / clients."""
matching_engine: Any
score_candidates: Callable[..., Any]
spotify_client_getter: Callable[[], Any]
itunes_client_getter: Callable[[], Any]
prefer_spotify_getter: Callable[[], bool]
min_confidence: float = 0.9
def match_mb_track(
track: Dict[str, Any], deps: MBMatchDeps
) -> Optional[Dict[str, Any]]:
"""Try to match a single MB-metadata track.
Input shape:
``{'track_name', 'artist_name', 'album_name', 'duration_ms'}``
Returns the matched_data dict (Spotify/iTunes track projection)
or ``None`` when no candidate cleared the confidence threshold.
"""
title = track.get("track_name") or ""
artist = track.get("artist_name") or ""
album = track.get("album_name") or ""
duration_ms = int(track.get("duration_ms") or 0)
if not title or not artist:
return None
spotify_client = deps.spotify_client_getter()
itunes_client = deps.itunes_client_getter()
use_spotify = bool(
deps.prefer_spotify_getter()
and spotify_client is not None
and getattr(spotify_client, "is_spotify_authenticated", lambda: False)()
)
if not use_spotify and itunes_client is None:
return None
# Strategy 1 — matching-engine query generation.
try:
temp_track = type("_TempTrack", (), {
"name": title,
"artists": [artist],
"album": album or None,
})()
queries = deps.matching_engine.generate_download_queries(temp_track)
except Exception as exc:
logger.debug(f"matching_engine query-gen failed: {exc}")
queries = [f"{artist} {title}", title]
best_match: Any = None
best_confidence = 0.0
for query in queries:
try:
if use_spotify:
results = spotify_client.search_tracks(query, limit=10)
else:
results = itunes_client.search_tracks(query, limit=10)
except Exception as exc:
logger.debug(f"search failed for query={query!r}: {exc}")
continue
if not results:
continue
try:
match, confidence, _ = deps.score_candidates(
title, artist, duration_ms, results
)
except Exception as exc:
logger.debug(f"score_candidates failed: {exc}")
continue
if match and confidence > best_confidence and confidence >= deps.min_confidence:
best_match = match
best_confidence = confidence
if best_confidence >= deps.min_confidence:
break
if not best_match:
return None
provider = "spotify" if use_spotify else "itunes"
image_url = getattr(best_match, "image_url", None) or ""
album_data: Dict[str, Any] = {
"name": getattr(best_match, "album", "") or "",
}
if image_url:
album_data["images"] = [{"url": image_url}]
return {
"id": getattr(best_match, "id", "") or "",
"name": getattr(best_match, "name", "") or "",
"artists": list(getattr(best_match, "artists", []) or []),
"album": album_data,
"duration_ms": int(getattr(best_match, "duration_ms", 0) or 0),
"image_url": image_url,
"source": provider,
"_provider": provider,
"_confidence": float(best_confidence),
}
def match_mb_tracks(
tracks: List[Dict[str, Any]], deps: MBMatchDeps
) -> List[Optional[Dict[str, Any]]]:
"""Vectorized variant — runs ``match_mb_track`` per track.
Phase 1b is sequential. If profiling shows it's too slow on big
LB playlists, this becomes the natural spot to thread-pool the
per-track searches."""
return [match_mb_track(t, deps) for t in tracks]

99
core/discovery/playlist.py
View file

@ -37,35 +37,9 @@ import time
from dataclasses import dataclass
from typing import Any, Callable
from core.discovery.manual_match import should_rediscover
logger = logging.getLogger(__name__)
def _canonical_best_score(deps, title, artist, duration_ms, results):
"""Score search results against the source track, trying the canonicalized
title/artist too and keeping the better confidence (#785).
YouTube playlists have their "Artist - Title" / channel decoration stripped
at ingest, but file/CSV-imported playlists keep raw titles — so a track
titled "Arctic Monkeys - Do I Wanna Know?" scored verbatim against the
library's "Do I Wanna Know?" never matched. canonical_source_track is
conservative (only strips an "<artist> - " prefix when it equals the
artist), so this can only ADD a better candidate, never weaken a match.
Returns (match, confidence)."""
match, confidence, _ = deps.discovery_score_candidates(title, artist, duration_ms, results)
try:
from core.text.source_title import canonical_source_track
canon_title, canon_artist = canonical_source_track(title or '', artist or '')
except Exception:
return match, confidence
if (canon_title, canon_artist) != (title, artist):
alt_match, alt_conf, _ = deps.discovery_score_candidates(canon_title, canon_artist, duration_ms, results)
if alt_match and alt_conf > confidence:
return alt_match, alt_conf
return match, confidence
@dataclass
class PlaylistDiscoveryDeps:
"""Bundle of cross-cutting deps the playlist discovery worker needs."""
@ -147,14 +121,31 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
existing_extra = json.loads(track['extra_data']) if isinstance(track['extra_data'], str) else track['extra_data']
except (json.JSONDecodeError, TypeError):
pass
# `should_rediscover` is the single source of truth for this
# gate (manual match checked FIRST so a stale Wing It flag can't
# revert a user's deliberate fix — see its docstring).
if should_rediscover(existing_extra):
undiscovered_tracks.append(track)
else:
if existing_extra.get('discovered'):
if existing_extra.get('wing_it_fallback'):
# Wing It stub — always re-attempt to find a real match
undiscovered_tracks.append(track)
else:
# Check if matched_data is complete — old discoveries may be missing
# track_number/release_date due to the Track dataclass stripping them.
# Re-discover these so the enriched pipeline fills in the gaps.
md = existing_extra.get('matched_data', {})
album = md.get('album', {})
has_track_num = md.get('track_number')
has_release = album.get('release_date') if isinstance(album, dict) else None
has_album_id = album.get('id') if isinstance(album, dict) else None
if has_track_num and (has_release or has_album_id):
pl_skipped += 1
total_skipped += 1
else:
# Incomplete discovery — re-discover to get full metadata
undiscovered_tracks.append(track)
elif existing_extra.get('unmatched_by_user'):
# User explicitly removed this match — respect their choice
pl_skipped += 1
total_skipped += 1
else:
undiscovered_tracks.append(track)
if pl_skipped > 0:
deps.update_automation_progress(automation_id,
@ -219,20 +210,6 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
except Exception:
search_queries = [f"{artist_name} {track_name}", track_name]
# #785: file/CSV playlists keep raw "Artist - Title" titles, so the
# queries above search for the artist prefix too. Also search the
# canonicalized title so the right candidates are actually returned
# (the scorer best-of then matches them).
try:
from core.text.source_title import canonical_source_track
_cq_title, _cq_artist = canonical_source_track(track_name, artist_name)
if (_cq_title, _cq_artist) != (track_name, artist_name):
for _q in (f"{_cq_artist} {_cq_title}", _cq_title):
if _q not in search_queries:
search_queries.append(_q)
except Exception as _cq_err:
logger.debug("canonical search-query add failed: %s", _cq_err)
# Step 3: Search and score
best_match = None
best_confidence = 0.0
@ -247,8 +224,8 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
if not results:
continue
match, confidence = _canonical_best_score(
deps, track_name, artist_name, duration_ms, results
match, confidence, _ = deps.discovery_score_candidates(
track_name, artist_name, duration_ms, results
)
if match and confidence > best_confidence:
@ -269,8 +246,8 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
else:
extended = itunes_client_instance.search_tracks(query, limit=50)
if extended:
match, confidence = _canonical_best_score(
deps, track_name, artist_name, duration_ms, extended
match, confidence, _ = deps.discovery_score_candidates(
track_name, artist_name, duration_ms, extended
)
if match and confidence > best_confidence:
best_confidence = confidence
@ -316,18 +293,6 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
except Exception as e:
logger.debug("metadata cache lookup for album enrichment failed: %s", e)
# Always include ``track_number`` / ``disc_number``
# in the matched payload — None when unknown rather
# than omitting the key. Downstream consumers
# (``ensure_wishlist_track_format``, post-process
# pipeline) check for None to know "look this up
# somewhere else"; an absent key was indistinguishable
# from "value is 1" after older payload helpers
# silently filled the default. Pre-fix Deezer-sourced
# matches always omitted the key (Deezer's track shape
# uses ``track_position`` and the cache lookup at
# line 304 reads ``track_number`` literally so it
# returns None for Deezer rows).
matched_data = {
'id': best_match.id if hasattr(best_match, 'id') else '',
'name': best_match.name if hasattr(best_match, 'name') else '',
@ -336,13 +301,11 @@ def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistD
'duration_ms': best_match.duration_ms if hasattr(best_match, 'duration_ms') else 0,
'image_url': match_image,
'source': discovery_source,
'track_number': track_number if track_number else (
getattr(best_match, 'track_number', None)
),
'disc_number': disc_number if disc_number else (
getattr(best_match, 'disc_number', None)
),
}
if track_number:
matched_data['track_number'] = track_number
if disc_number:
matched_data['disc_number'] = disc_number
extra_data = {
'discovered': True,

299
core/discovery/qobuz.py
View file

@ -1,299 +0,0 @@
"""Background worker for Qobuz playlist discovery.
`run_qobuz_discovery_worker(playlist_id, deps)` is the function the
Qobuz discovery start-endpoint submits to its executor to match each
Qobuz playlist track against Spotify (preferred) or the configured
fallback metadata source (iTunes / Deezer / Discogs / MusicBrainz).
Mirrors `core/discovery/deezer.py` exactly — Qobuz playlists arrive as
dicts (not dataclasses) from `core/qobuz_client.py:get_playlist`, so
this worker uses dict-style access on track data and wraps each entry
in a SimpleNamespace before handing it to the shared
`_search_spotify_for_tidal_track` helper.
"""
from __future__ import annotations
import logging
import time
import types
from dataclasses import dataclass
from typing import Any, Callable
logger = logging.getLogger(__name__)
@dataclass
class QobuzDiscoveryDeps:
"""Bundle of cross-cutting deps the Qobuz discovery worker needs."""
qobuz_discovery_states: dict
spotify_client: Any
pause_enrichment_workers: Callable[[str], dict]
resume_enrichment_workers: Callable[[dict, str], None]
get_active_discovery_source: Callable[[], str]
get_metadata_fallback_client: Callable[[], Any]
get_discovery_cache_key: Callable
get_database: Callable[[], Any]
validate_discovery_cache_artist: Callable
search_spotify_for_tidal_track: Callable
build_discovery_wing_it_stub: Callable
add_activity_item: Callable
sync_discovery_results_to_mirrored: Callable
def run_qobuz_discovery_worker(playlist_id, deps: QobuzDiscoveryDeps):
"""Background worker for Qobuz discovery process (Spotify preferred, fallback metadata source)."""
_ew_state = {}
try:
_ew_state = deps.pause_enrichment_workers('Qobuz discovery')
state = deps.qobuz_discovery_states[playlist_id]
playlist = state['playlist']
# Determine which provider to use
discovery_source = deps.get_active_discovery_source()
use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
# Initialize fallback client if needed
itunes_client_instance = None
if not use_spotify:
itunes_client_instance = deps.get_metadata_fallback_client()
logger.info(f"Starting Qobuz discovery for: {playlist['name']} (using {discovery_source.upper()})")
# Store discovery source in state for frontend
state['discovery_source'] = discovery_source
successful_discoveries = 0
tracks = playlist['tracks']
for i, qobuz_track in enumerate(tracks):
if state.get('cancelled', False):
break
try:
track_name = qobuz_track['name']
track_artists = qobuz_track['artists']
track_id = qobuz_track['id']
track_album = qobuz_track.get('album', '')
track_duration_ms = qobuz_track.get('duration_ms', 0)
logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
# Check discovery cache first
cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
try:
cache_db = deps.get_database()
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
cached_artists = cached_match.get('artists', [])
if cached_artists:
cached_artist_str = ', '.join(
a if isinstance(a, str) else a.get('name', '') for a in cached_artists
)
else:
cached_artist_str = ''
cached_album = cached_match.get('album', '')
if isinstance(cached_album, dict):
cached_album = cached_album.get('name', '')
result = {
'qobuz_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
'album': track_album,
'duration_ms': track_duration_ms,
},
'spotify_data': cached_match,
'match_data': cached_match,
'status': 'Found',
'status_class': 'found',
'spotify_track': cached_match.get('name', ''),
'spotify_artist': cached_artist_str,
'spotify_album': cached_album,
'spotify_id': cached_match.get('id', ''),
'discovery_source': discovery_source,
'index': i
}
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
continue
except Exception as cache_err:
logger.error(f"Cache lookup error: {cache_err}")
# SimpleNamespace duck-type for _search_spotify_for_tidal_track
track_ns = types.SimpleNamespace(
id=track_id,
name=track_name,
artists=track_artists,
album=track_album,
duration_ms=track_duration_ms
)
track_result = deps.search_spotify_for_tidal_track(
track_ns,
use_spotify=use_spotify,
itunes_client=itunes_client_instance
)
result = {
'qobuz_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
'album': track_album,
'duration_ms': track_duration_ms,
},
'spotify_data': None,
'match_data': None,
'status': 'Not Found',
'status_class': 'not-found',
'spotify_track': '',
'spotify_artist': '',
'spotify_album': '',
'discovery_source': discovery_source
}
match_confidence = 0.0
if use_spotify and isinstance(track_result, tuple):
track_obj, raw_track_data, match_confidence = track_result
album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
album_obj['name'] = track_obj.album
elif not album_obj and track_obj.album:
album_obj = {'name': track_obj.album}
if isinstance(album_obj, dict) and not album_obj.get('release_date'):
album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
_album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
_image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
match_data = {
'id': track_obj.id,
'name': track_obj.name,
'artists': track_obj.artists,
'album': album_obj,
'duration_ms': track_obj.duration_ms,
'external_urls': track_obj.external_urls,
'image_url': _image_url,
'source': 'spotify'
}
if raw_track_data and raw_track_data.get('track_number'):
match_data['track_number'] = raw_track_data['track_number']
if raw_track_data and raw_track_data.get('disc_number'):
match_data['disc_number'] = raw_track_data['disc_number']
result['spotify_data'] = match_data
result['match_data'] = match_data
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = track_obj.name
result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
result['spotify_id'] = track_obj.id
result['confidence'] = match_confidence
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
elif not use_spotify and track_result and isinstance(track_result, dict):
match_confidence = track_result.pop('confidence', 0.80)
match_data = track_result
match_data['source'] = discovery_source
_fb_album = match_data.get('album', {})
_fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
if _fb_images and 'image_url' not in match_data:
match_data['image_url'] = _fb_images[0].get('url', '')
result['spotify_data'] = match_data
result['match_data'] = match_data
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = match_data.get('name', '')
itunes_artists = match_data.get('artists', [])
result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
result['spotify_id'] = match_data.get('id', '')
result['confidence'] = match_confidence
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
# Save to discovery cache if match found
if result['status_class'] == 'found' and result.get('match_data'):
try:
cache_db = deps.get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], discovery_source, match_confidence,
result['match_data'], track_name,
track_artists[0] if track_artists else ''
)
logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
except Exception as cache_err:
logger.error(f"Cache save error: {cache_err}")
# Auto Wing It fallback for unmatched tracks
if result['status_class'] == 'not-found':
qobuz_t = result.get('qobuz_track', {})
stub = deps.build_discovery_wing_it_stub(
qobuz_t.get('name', ''),
', '.join(qobuz_t.get('artists', [])),
qobuz_t.get('duration_ms', 0)
)
result['status'] = 'Wing It'
result['status_class'] = 'wing-it'
result['spotify_data'] = stub
result['match_data'] = stub
result['spotify_track'] = qobuz_t.get('name', '')
result['spotify_artist'] = ', '.join(qobuz_t.get('artists', []))
result['wing_it_fallback'] = True
result['confidence'] = 0
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
state['wing_it_count'] = state.get('wing_it_count', 0) + 1
result['index'] = i
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
time.sleep(0.1)
except Exception as e:
logger.error(f"Error processing track {i+1}: {e}")
result = {
'qobuz_track': {
'name': qobuz_track.get('name', 'Unknown'),
'artists': qobuz_track.get('artists', []),
},
'spotify_data': None,
'match_data': None,
'status': 'Error',
'status_class': 'error',
'spotify_track': '',
'spotify_artist': '',
'spotify_album': '',
'error': str(e),
'discovery_source': discovery_source,
'index': i
}
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
# Mark as complete
state['phase'] = 'discovered'
state['status'] = 'discovered'
state['discovery_progress'] = 100
source_label = discovery_source.upper()
deps.add_activity_item("", f"Qobuz Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
logger.info(f"Qobuz discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
deps.sync_discovery_results_to_mirrored('qobuz', playlist_id, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
except Exception as e:
logger.error(f"Error in Qobuz discovery worker: {e}")
if playlist_id in deps.qobuz_discovery_states:
deps.qobuz_discovery_states[playlist_id]['phase'] = 'error'
deps.qobuz_discovery_states[playlist_id]['status'] = f'error: {str(e)}'
finally:
deps.resume_enrichment_workers(_ew_state, 'Qobuz discovery')

415
core/discovery/quality_scanner.py
View file

@ -1,31 +1,44 @@
"""Shared metadata match + result-normalization helpers for quality matching.
"""Background worker for the library quality scanner.
These were the matching guts of the old auto-acting quality-scanner worker (now
removed — quality scanning is the ``quality_upgrade`` library-maintenance repair
job in ``core/repair_jobs/quality_upgrade.py``). They're kept here as a single
source of truth and imported by that job:
`run_quality_scanner(scope, profile_id, deps)` is the function the
quality-scanner endpoint kicks off in a thread to scan the library
for low-quality tracks (below the user's configured quality profile)
and add provider matches to the wishlist:
- ``_search_tracks_for_source`` — query one metadata source's ``search_tracks``.
- ``_normalize_track_match`` / ``_normalize_track_album`` / ``_normalize_track_artists``
— turn a provider track into the wishlist-ready dict (typed Album converters
with legacy duck-typed fallback).
- ``_track_name`` / ``_track_artist_names`` / ``_extract_lookup_value`` — accessors.
1. Reset scanner state, load quality profile + minimum acceptable tier.
2. Load tracks from DB based on scope:
- 'watchlist' → tracks for watchlisted artists only.
- other → all library tracks.
3. For each track:
- Stop-request gate (state['status'] != 'running').
- Quality-tier check via _get_quality_tier_from_extension(file_path).
- Skip tracks meeting standards (tier_num <= min_acceptable_tier).
- For low-quality tracks: matching_engine search query gen, score
candidates against the configured metadata source priority
(artist + title similarity, album-type bonus), pick best match >=
0.7 confidence.
- On match: add normalized track data to wishlist via
`wishlist_service.add_track_to_wishlist` with
source_type='quality_scanner' and a source_context that captures
original file_path, format tier, bitrate, and match confidence.
4. After all tracks: status='finished', progress=100, activity feed
entry, emit `quality_scan_completed` event for automation engine.
5. On critical exception: status='error', error message captured.
"""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Callable, Dict, Optional
from core.metadata.registry import get_client_for_source
from core.metadata.registry import get_client_for_source, get_primary_source, get_source_priority
from core.metadata.types import Album
from core.wishlist.payloads import ensure_wishlist_track_format
# Use the project logger namespace ("soulsync.*") so the scanner's progress and
# diagnostics actually surface in the app log — plain getLogger(__name__) lands
# under "core.discovery.quality_scanner", which the app log view doesn't show.
from utils.logging_config import get_logger
logger = get_logger("discovery.quality_scanner")
logger = logging.getLogger(__name__)
# Per-source typed converter dispatch — same registry pattern as
@ -43,6 +56,16 @@ _TYPED_ALBUM_CONVERTERS: Dict[str, Callable[[Dict[str, Any]], Album]] = {
}
@dataclass
class QualityScannerDeps:
"""Bundle of cross-cutting deps the quality scanner needs."""
quality_scanner_state: dict
quality_scanner_lock: Any # threading.Lock
QUALITY_TIERS: dict
matching_engine: Any
automation_engine: Any
get_quality_tier_from_extension: Callable
add_activity_item: Callable
def _extract_lookup_value(value: Any, *names: str, default: Any = None) -> Any:
@ -277,3 +300,363 @@ def _search_tracks_for_source(source: str, query: str, limit: int = 5, client: A
except Exception as exc:
logger.debug("Could not search %s for %s: %s", source, query, exc)
return []
def run_quality_scanner(scope='watchlist', profile_id=1, deps: QualityScannerDeps = None):
"""Main quality scanner worker function"""
from core.wishlist_service import get_wishlist_service
from database.music_database import MusicDatabase
try:
with deps.quality_scanner_lock:
deps.quality_scanner_state["status"] = "running"
deps.quality_scanner_state["phase"] = "Initializing scan..."
deps.quality_scanner_state["progress"] = 0
deps.quality_scanner_state["processed"] = 0
deps.quality_scanner_state["total"] = 0
deps.quality_scanner_state["quality_met"] = 0
deps.quality_scanner_state["low_quality"] = 0
deps.quality_scanner_state["matched"] = 0
deps.quality_scanner_state["results"] = []
deps.quality_scanner_state["error_message"] = ""
logger.info(f"[Quality Scanner] Starting scan with scope: {scope}")
# Get database instance
db = MusicDatabase()
# Get quality profile to determine preferred quality
quality_profile = db.get_quality_profile()
preferred_qualities = quality_profile.get('qualities', {})
# Determine minimum acceptable tier based on enabled qualities
min_acceptable_tier = 999
for quality_name, quality_config in preferred_qualities.items():
if quality_config.get('enabled', False):
# Map quality profile names to tier names
tier_map = {
'flac': 'lossless',
'mp3_320': 'low_lossy',
'mp3_256': 'low_lossy',
'mp3_192': 'low_lossy'
}
tier_name = tier_map.get(quality_name)
if tier_name:
tier_num = deps.QUALITY_TIERS[tier_name]['tier']
min_acceptable_tier = min(min_acceptable_tier, tier_num)
logger.info(f"[Quality Scanner] Minimum acceptable tier: {min_acceptable_tier}")
# Get tracks to scan based on scope
with deps.quality_scanner_lock:
deps.quality_scanner_state["phase"] = "Loading tracks from database..."
if scope == 'watchlist':
# Get watchlist artists
watchlist_artists = db.get_watchlist_artists(profile_id=profile_id)
if not watchlist_artists:
with deps.quality_scanner_lock:
deps.quality_scanner_state["status"] = "finished"
deps.quality_scanner_state["phase"] = "No watchlist artists found"
deps.quality_scanner_state["error_message"] = "Please add artists to watchlist first"
logger.warning("[Quality Scanner] No watchlist artists found")
return
# Get artist names from watchlist
artist_names = [artist.artist_name for artist in watchlist_artists]
logger.info(f"[Quality Scanner] Scanning {len(artist_names)} watchlist artists")
# Get all tracks for these artists by name
conn = db._get_connection()
placeholders = ','.join(['?' for _ in artist_names])
tracks_to_scan = conn.execute(
f"SELECT t.id, t.title, t.artist_id, t.album_id, t.file_path, t.bitrate, a.name as artist_name, al.title as album_title "
f"FROM tracks t "
f"JOIN artists a ON t.artist_id = a.id "
f"JOIN albums al ON t.album_id = al.id "
f"WHERE a.name IN ({placeholders}) AND t.file_path IS NOT NULL",
artist_names
).fetchall()
conn.close()
else:
# Scan all library tracks
with deps.quality_scanner_lock:
deps.quality_scanner_state["phase"] = "Loading all library tracks..."
conn = db._get_connection()
tracks_to_scan = conn.execute(
"SELECT t.id, t.title, t.artist_id, t.album_id, t.file_path, t.bitrate, a.name as artist_name, al.title as album_title "
"FROM tracks t "
"JOIN artists a ON t.artist_id = a.id "
"JOIN albums al ON t.album_id = al.id "
"WHERE t.file_path IS NOT NULL"
).fetchall()
conn.close()
total_tracks = len(tracks_to_scan)
logger.info(f"[Quality Scanner] Found {total_tracks} tracks to scan")
with deps.quality_scanner_lock:
deps.quality_scanner_state["total"] = total_tracks
deps.quality_scanner_state["phase"] = f"Scanning {total_tracks} tracks..."
source_priority = get_source_priority(get_primary_source())
if not source_priority:
with deps.quality_scanner_lock:
deps.quality_scanner_state["status"] = "error"
deps.quality_scanner_state["phase"] = "No metadata provider available"
deps.quality_scanner_state["error_message"] = "No metadata provider is available for quality scanning"
logger.info("[Quality Scanner] No metadata provider available")
return
logger.info("[Quality Scanner] Using metadata source priority: %s", source_priority)
wishlist_service = get_wishlist_service()
add_to_wishlist = getattr(wishlist_service, 'add_track_to_wishlist', None)
if add_to_wishlist is None:
add_to_wishlist = getattr(wishlist_service, 'add_spotify_track_to_wishlist', None)
if add_to_wishlist is None:
raise AttributeError("Wishlist service does not expose an add-to-wishlist method")
# Scan each track
for idx, track_row in enumerate(tracks_to_scan, 1):
# Check for stop request
if deps.quality_scanner_state.get('status') != 'running':
logger.info(f"[Quality Scanner] Stop requested, halting at track {idx}/{total_tracks}")
break
try:
track_id, title, artist_id, album_id, file_path, bitrate, artist_name, album_title = track_row
# Check quality tier
tier_name, tier_num = deps.get_quality_tier_from_extension(file_path)
# Update progress
with deps.quality_scanner_lock:
deps.quality_scanner_state["processed"] = idx
deps.quality_scanner_state["progress"] = (idx / total_tracks) * 100
deps.quality_scanner_state["phase"] = f"Scanning: {artist_name} - {title}"
# Check if meets quality standards
if tier_num <= min_acceptable_tier:
# Quality met
with deps.quality_scanner_lock:
deps.quality_scanner_state["quality_met"] += 1
continue
# Low quality track found
with deps.quality_scanner_lock:
deps.quality_scanner_state["low_quality"] += 1
logger.info(f"[Quality Scanner] Low quality: {artist_name} - {title} ({tier_name}, {file_path})")
# Attempt to match using the active metadata provider
matched = False
matched_track_data = None
best_source = None
attempted_any_provider = False
try:
# Generate search queries using matching engine
temp_track = type('TempTrack', (), {
'name': title,
'artists': [artist_name],
'album': album_title
})()
search_queries = deps.matching_engine.generate_download_queries(temp_track)
logger.info(f"[Quality Scanner] Generated {len(search_queries)} search queries for {artist_name} - {title}")
# Find best match using confidence scoring
best_match = None
best_confidence = 0.0
min_confidence = 0.7 # Match existing standard
for _query_idx, search_query in enumerate(search_queries):
try:
for source in source_priority:
client = get_client_for_source(source)
if not client or not hasattr(client, 'search_tracks'):
continue
attempted_any_provider = True
provider_matches = _search_tracks_for_source(source, search_query, limit=5, client=client)
time.sleep(0.5) # Rate limit metadata API calls
if not provider_matches:
continue
# Score each result using matching engine
for provider_track in provider_matches:
try:
# Calculate artist confidence
artist_confidence = 0.0
provider_artists = _track_artist_names(provider_track)
if provider_artists:
for result_artist in provider_artists:
artist_sim = deps.matching_engine.similarity_score(
deps.matching_engine.normalize_string(artist_name),
deps.matching_engine.normalize_string(result_artist)
)
artist_confidence = max(artist_confidence, artist_sim)
# Calculate title confidence
title_confidence = deps.matching_engine.similarity_score(
deps.matching_engine.normalize_string(title),
deps.matching_engine.normalize_string(_track_name(provider_track))
)
# Combined confidence (50% artist + 50% title)
combined_confidence = (artist_confidence * 0.5 + title_confidence * 0.5)
# Small bonus for album tracks over singles
_at = _extract_lookup_value(provider_track, 'album_type', default='') or ''
if _at == 'album':
combined_confidence += 0.02
elif _at == 'ep':
combined_confidence += 0.01
candidate_artist = provider_artists[0] if provider_artists else 'Unknown Artist'
candidate_name = _track_name(provider_track)
logger.info(
f"[Quality Scanner] Candidate ({source}): '{candidate_artist}' - "
f"'{candidate_name}' (confidence: {combined_confidence:.3f})"
)
# Update best match if this is better
if combined_confidence > best_confidence and combined_confidence >= min_confidence:
best_confidence = combined_confidence
best_match = provider_track
best_source = source
logger.info(
f"[Quality Scanner] New best match ({source}): {candidate_artist} - "
f"{candidate_name} (confidence: {combined_confidence:.3f})"
)
except Exception as e:
logger.error(f"[Quality Scanner] Error scoring result: {e}")
continue
# If we found a very high confidence match, stop searching this query
if best_confidence >= 0.9:
logger.info(f"[Quality Scanner] High confidence match found ({best_confidence:.3f}), stopping search")
break
except Exception as e:
logger.debug(f"[Quality Scanner] Error searching with query '{search_query}': {e}")
continue
if not attempted_any_provider:
with deps.quality_scanner_lock:
deps.quality_scanner_state["status"] = "error"
deps.quality_scanner_state["phase"] = "No metadata provider available"
deps.quality_scanner_state["error_message"] = "No metadata provider is available for quality scanning"
logger.info("[Quality Scanner] No metadata provider available")
return
# Process best match
if best_match:
matched = True
final_artist = _track_artist_names(best_match)[0] if _track_artist_names(best_match) else 'Unknown Artist'
final_name = _track_name(best_match)
final_source = best_source or 'metadata'
logger.info(
f"[Quality Scanner] Final match ({final_source}): {final_artist} - "
f"{final_name} (confidence: {best_confidence:.3f})"
)
# Build normalized track data for wishlist
matched_track_data = _normalize_track_match(best_match, final_source)
# Add to wishlist
source_context = {
'quality_scanner': True,
'original_file_path': file_path,
'original_format': tier_name,
'original_bitrate': bitrate,
'match_confidence': best_confidence,
'scan_date': datetime.now().isoformat()
}
success = add_to_wishlist(
track_data=matched_track_data,
failure_reason=f"Low quality - {tier_name.replace('_', ' ').title()} format",
source_type='quality_scanner',
source_context=source_context,
profile_id=profile_id
)
if success:
with deps.quality_scanner_lock:
deps.quality_scanner_state["matched"] += 1
logger.info(f"[Quality Scanner] Matched and added to wishlist: {artist_name} - {title}")
else:
logger.error(f"[Quality Scanner] Failed to add to wishlist: {artist_name} - {title}")
else:
logger.warning(
f"[Quality Scanner] No suitable metadata match found "
f"(best confidence: {best_confidence:.3f}, required: {min_confidence:.3f})"
)
except Exception as matching_error:
logger.error(f"[Quality Scanner] Matching error for {artist_name} - {title}: {matching_error}")
# Store result
result_entry = {
'track_id': track_id,
'title': title,
'artist': artist_name,
'album': album_title,
'file_path': file_path,
'current_format': tier_name,
'bitrate': bitrate,
'matched': matched,
'match_id': matched_track_data['id'] if matched_track_data else None,
'provider': best_source if matched else None,
'spotify_id': matched_track_data['id'] if matched_track_data else None,
}
with deps.quality_scanner_lock:
deps.quality_scanner_state["results"].append(result_entry)
if not matched:
logger.warning(f"[Quality Scanner] No metadata match found for: {artist_name} - {title}")
except Exception as track_error:
logger.error(f"[Quality Scanner] Error processing track: {track_error}")
continue
# Scan complete (don't overwrite if already stopped by user)
with deps.quality_scanner_lock:
was_stopped = deps.quality_scanner_state["status"] != "running"
deps.quality_scanner_state["status"] = "finished"
deps.quality_scanner_state["progress"] = 100
if not was_stopped:
deps.quality_scanner_state["phase"] = "Scan complete"
logger.info(f"[Quality Scanner] Scan {'stopped' if was_stopped else 'complete'}: {deps.quality_scanner_state['processed']} processed, "
f"{deps.quality_scanner_state['low_quality']} low quality, {deps.quality_scanner_state['matched']} matched to metadata providers")
# Add activity
deps.add_activity_item("", "Quality Scan Complete",
f"{deps.quality_scanner_state['matched']} tracks added to wishlist", "Now")
try:
if deps.automation_engine:
deps.automation_engine.emit('quality_scan_completed', {
'quality_met': str(deps.quality_scanner_state.get('quality_met', 0)),
'low_quality': str(deps.quality_scanner_state.get('low_quality', 0)),
'total_scanned': str(deps.quality_scanner_state.get('processed', 0)),
})
except Exception as e:
logger.debug("emit quality_scan_completed failed: %s", e)
except Exception as e:
logger.error(f"[Quality Scanner] Critical error: {e}")
import traceback
traceback.print_exc()
with deps.quality_scanner_lock:
deps.quality_scanner_state["status"] = "error"
deps.quality_scanner_state["error_message"] = str(e)
deps.quality_scanner_state["phase"] = f"Error: {str(e)}"

24
core/discovery/spotify_public.py
View file

@ -55,17 +55,13 @@ class SpotifyPublicDiscoveryDeps:
search_spotify_for_tidal_track: Callable
build_discovery_wing_it_stub: Callable
add_activity_item: Callable
source_label: str = "Spotify Public"
activity_label: str = "Spotify Link"
original_track_key: str = "spotify_public_track"
def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDeps):
"""Background worker for Spotify Public discovery process (Spotify preferred, iTunes fallback)"""
_ew_state = {}
try:
worker_label = f"{deps.source_label} discovery"
_ew_state = deps.pause_enrichment_workers(worker_label)
_ew_state = deps.pause_enrichment_workers('Spotify Public discovery')
state = deps.spotify_public_discovery_states[url_hash]
playlist = state['playlist']
@ -78,7 +74,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
if not use_spotify:
itunes_client_instance = deps.get_metadata_fallback_client()
logger.info(f"Starting {deps.source_label} discovery for: {playlist['name']} (using {discovery_source.upper()})")
logger.info(f"Starting Spotify Public discovery for: {playlist['name']} (using {discovery_source.upper()})")
# Store discovery source in state for frontend
state['discovery_source'] = discovery_source
@ -130,7 +126,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
cached_album = cached_album.get('name', '')
result = {
deps.original_track_key: {
'spotify_public_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
@ -174,7 +170,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
# Create result entry
result = {
deps.original_track_key: {
'spotify_public_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
@ -274,7 +270,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
# Auto Wing It fallback for unmatched tracks
if result['status_class'] == 'not-found':
sp_t = result.get(deps.original_track_key, {})
sp_t = result.get('spotify_public_track', {})
stub = deps.build_discovery_wing_it_stub(
sp_t.get('name', ''),
', '.join(sp_t.get('artists', [])),
@ -303,7 +299,7 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
logger.error(f"Error processing track {i+1}: {e}")
# Add error result
result = {
deps.original_track_key: {
'spotify_public_track': {
'name': sp_track.get('name', 'Unknown'),
'artists': sp_track.get('artists', []),
},
@ -328,14 +324,14 @@ def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDe
# Add activity for discovery completion
source_label = discovery_source.upper()
deps.add_activity_item("", f"{deps.activity_label} Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
deps.add_activity_item("", f"Spotify Link Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
logger.info(f"{deps.source_label} discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
logger.info(f"Spotify Public discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
except Exception as e:
logger.error(f"Error in {deps.source_label} discovery worker: {e}")
logger.error(f"Error in Spotify Public discovery worker: {e}")
if url_hash in deps.spotify_public_discovery_states:
deps.spotify_public_discovery_states[url_hash]['phase'] = 'error'
deps.spotify_public_discovery_states[url_hash]['status'] = f'error: {str(e)}'
finally:
deps.resume_enrichment_workers(_ew_state, f"{deps.source_label} discovery")
deps.resume_enrichment_workers(_ew_state, 'Spotify Public discovery')

391
core/discovery/sync.py
View file

@ -47,229 +47,9 @@ class SyncDeps:
update_and_save_sync_status: Callable
sync_states: dict
sync_lock: Any # threading.Lock
# Optional: post-sync download follow-up for mirrored-playlist automations.
process_wishlist_automatically: Callable[..., Any] | None = None
run_playlist_organize_download: Callable[..., Any] | None = None
is_wishlist_actually_processing: Callable[[], bool] | None = None
def _post_sync_automation_followup(
deps: SyncDeps,
*,
automation_id: str,
playlist_id: str,
skip_wishlist_add: bool,
result: Any,
) -> None:
"""Queue downloads after an automation sync finishes.
Sync Playlist runs in a background thread and returns immediately, so a
separate scheduled "Process Wishlist" action often runs on an empty wishlist.
Organize-by-playlist skips sync-time wishlist adds and expects a folder
download batch instead — that only ran in Playlist Pipeline before this hook.
"""
if not automation_id or not str(playlist_id).startswith('auto_mirror_'):
return
try:
mirrored_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
except ValueError:
return
failed = int(getattr(result, 'failed_tracks', 0) or 0)
wishlist_added = int(getattr(result, 'wishlist_added_count', 0) or 0)
if skip_wishlist_add:
org_fn = deps.run_playlist_organize_download
if failed <= 0:
return
if not org_fn:
logger.warning(
"Organize-by-playlist sync left %s missing tracks but organize download is unavailable",
failed,
)
deps.update_automation_progress(
automation_id,
log_line=f'{failed} missing — enable Playlist Pipeline or disable Organize by Playlist',
log_type='warning',
)
return
org_result = org_fn(mirrored_playlist_id=mirrored_id, automation_id=automation_id)
status = org_result.get('status', 'unknown') if isinstance(org_result, dict) else 'unknown'
reason = org_result.get('reason', '') if isinstance(org_result, dict) else ''
log_type = 'success' if status == 'started' else 'warning'
detail = f' ({reason})' if reason and status != 'started' else ''
deps.update_automation_progress(
automation_id,
log_line=f'Organize download {status} for {failed} missing track(s){detail}',
log_type=log_type,
)
return
if wishlist_added <= 0:
if failed > 0:
deps.update_automation_progress(
automation_id,
log_line=f'{failed} missing but none added to wishlist — check logs',
log_type='warning',
)
return
proc_fn = deps.process_wishlist_automatically
if not proc_fn:
return
is_busy = deps.is_wishlist_actually_processing
if is_busy and is_busy():
deps.update_automation_progress(
automation_id,
log_line=f'Added {wishlist_added} to wishlist; download worker already running',
log_type='info',
)
return
proc_fn(automation_id=automation_id)
deps.update_automation_progress(
automation_id,
log_line=f'Started wishlist download for {wishlist_added} track(s)',
log_type='success',
)
async def _database_only_find_track(spotify_track, candidate_pool=None):
"""Database-only track matcher used when no media server is connected.
Patched onto sync_service._find_track_in_media_server. Accepts
``candidate_pool`` for interface parity with the real matcher (sync_service
calls it with candidate_pool=...); the DB path queries the library directly
via check_track_exists, so it doesn't need the per-artist candidate cache —
but it MUST accept the kwarg or sync raises "unexpected keyword argument
'candidate_pool'". Module-level (not a nested closure) so it's importable
and unit-tested.
"""
logger.info(f"Database-only search for: '{spotify_track.name}' by {spotify_track.artists}")
try:
from database.music_database import MusicDatabase
from config.settings import config_manager
db = MusicDatabase()
active_server = config_manager.get_active_media_server()
original_title = spotify_track.name
spotify_id = getattr(spotify_track, 'id', '') or ''
# --- Sync match cache fast-path ---
if spotify_id:
try:
cached = db.read_sync_match_cache(spotify_id, active_server)
if cached:
db_track_check = db.get_track_by_id(cached['server_track_id'])
if db_track_check:
class DatabaseTrackCached:
def __init__(self, db_t):
self.ratingKey = db_t.id
self.title = db_t.title
self.id = db_t.id
logger.debug(f"Sync cache hit: '{original_title}' → server track {cached['server_track_id']}")
return DatabaseTrackCached(db_track_check), cached['confidence']
logger.warning(f"Sync cache stale for '{original_title}' — track gone")
except Exception as e:
logger.debug("sync match cache fast-path failed: %s", e)
# --- End cache fast-path ---
# Durable manual library match (#787) — survives a library rescan (the
# sync_match_cache above does not), so a user's Find & Add pairing keeps
# sticking across auto-syncs instead of being re-matched from scratch (#895
# follow-up). Self-heals a stale library id via the stored file path.
if spotify_id:
try:
from core.artists.map import get_current_profile_id
m = db.find_manual_library_match_by_source_track_id(
get_current_profile_id(), str(spotify_id), active_server)
if m:
lib_id = m.get('library_track_id')
dt = db.get_track_by_id(lib_id) if lib_id is not None else None
if not dt and m.get('library_file_path'):
new_id = db.find_track_id_by_file_path(m['library_file_path'])
dt = db.get_track_by_id(new_id) if new_id else None
if dt:
class DatabaseTrackDurable:
def __init__(self, db_t):
self.ratingKey = db_t.id
self.title = db_t.title
self.id = db_t.id
logger.debug(f"Durable manual match hit: '{original_title}' → {lib_id}")
return DatabaseTrackDurable(dt), 1.0
except Exception as e:
logger.debug("durable manual match fast-path failed: %s", e)
# Try each artist
for artist in spotify_track.artists:
if isinstance(artist, str):
artist_name = artist
elif isinstance(artist, dict) and 'name' in artist:
artist_name = artist['name']
else:
artist_name = str(artist)
db_track, confidence = db.check_track_exists(
original_title, artist_name,
confidence_threshold=0.80,
server_source=active_server
)
if not (db_track and confidence >= 0.80):
# #785: file/CSV playlists keep raw "Artist - Title" titles (unlike
# YouTube, cleaned at ingest), which don't match the clean library
# title. Retry with the canonical form (best-of, conservative).
try:
from core.text.source_title import canonical_source_track
_canon_title, _canon_artist = canonical_source_track(original_title, artist_name)
if (_canon_title, _canon_artist) != (original_title, artist_name):
_alt_track, _alt_conf = db.check_track_exists(
_canon_title, _canon_artist,
confidence_threshold=0.80, server_source=active_server)
if _alt_track and _alt_conf > confidence:
db_track, confidence = _alt_track, _alt_conf
except Exception as _canon_err:
logger.debug("canonical retry failed: %s", _canon_err)
if db_track and confidence >= 0.80:
logger.info(f"Database match: '{db_track.title}' (confidence: {confidence:.2f})")
if spotify_id:
try:
from core.matching_engine import MusicMatchingEngine
me = MusicMatchingEngine()
db.save_sync_match_cache(
spotify_id, me.clean_title(original_title), me.clean_artist(artist_name),
active_server, db_track.id, db_track.title, confidence
)
except Exception as e:
logger.debug("save sync match cache failed: %s", e)
class DatabaseTrackMock:
def __init__(self, db_track):
self.ratingKey = db_track.id
self.title = db_track.title
self.id = db_track.id
return DatabaseTrackMock(db_track), confidence
logger.warning(f"No database match found for: '{original_title}'")
return None, 0.0
except Exception as e:
logger.error(f"Database search error: {e}")
return None, 0.0
def run_sync_task(
playlist_id,
playlist_name,
tracks_json,
automation_id=None,
profile_id=1,
playlist_image_url='',
deps: SyncDeps = None,
sync_mode: str = 'replace',
skip_wishlist_add: bool = False,
):
def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, profile_id=1, playlist_image_url='', deps: SyncDeps = None, sync_mode: str = 'replace'):
"""The actual sync function that runs in the background thread."""
sync_states = deps.sync_states
sync_lock = deps.sync_lock
@ -307,11 +87,35 @@ def run_sync_task(
# This avoids needing to re-fetch it from Spotify
logger.info("Converting JSON tracks to SpotifyTrack objects...")
# Store original track data with full album objects (for wishlist with cover art).
# Shared with the sync-detail "re-add to wishlist" action so both build the
# IDENTICAL payload (album→dict + images, artists→dicts). Copy-safe.
from core.sync.wishlist_readd import build_original_tracks_map
original_tracks_map = build_original_tracks_map(tracks_json)
# Store original track data with full album objects (for wishlist with cover art)
# Normalize formats for wishlist: album must be dict {'name': ...}, artists must be [{'name': ...}]
# Important: copy data — don't mutate tracks_json since SpotifyTrack expects List[str] artists
original_tracks_map = {}
for t in tracks_json:
track_id = t.get('id', '')
if track_id:
normalized = dict(t)
# Normalize album to dict format, preserving images and metadata
raw_album = normalized.get('album', '')
if isinstance(raw_album, str):
normalized['album'] = {
'name': raw_album or normalized.get('name', 'Unknown Album'),
'images': [], 'album_type': 'single', 'total_tracks': 1, 'release_date': ''
}
elif not isinstance(raw_album, dict):
normalized['album'] = {
'name': str(raw_album) if raw_album else normalized.get('name', 'Unknown Album'),
'images': [], 'album_type': 'single', 'total_tracks': 1, 'release_date': ''
}
else:
# Dict — ensure required keys exist
raw_album.setdefault('name', 'Unknown Album')
raw_album.setdefault('images', [])
# Normalize artists to list of dicts
raw_artists = normalized.get('artists', [])
if raw_artists and isinstance(raw_artists[0], str):
normalized['artists'] = [{'name': a} for a in raw_artists]
original_tracks_map[track_id] = normalized
tracks = []
for i, t in enumerate(tracks_json):
@ -456,9 +260,89 @@ def run_sync_task(
if media_client is None or not media_client.is_connected():
logger.info("Media client not connected - patching sync service for database-only matching")
# Patch the matcher to the module-level database-only implementation
# (importable + unit-tested; accepts candidate_pool for parity).
sync_service._find_track_in_media_server = _database_only_find_track
# Store original method
original_find_track = sync_service._find_track_in_media_server
# Create database-only replacement method
async def database_only_find_track(spotify_track):
logger.info(f"Database-only search for: '{spotify_track.name}' by {spotify_track.artists}")
try:
from database.music_database import MusicDatabase
from config.settings import config_manager
db = MusicDatabase()
active_server = config_manager.get_active_media_server()
original_title = spotify_track.name
spotify_id = getattr(spotify_track, 'id', '') or ''
# --- Sync match cache fast-path ---
if spotify_id:
try:
cached = db.read_sync_match_cache(spotify_id, active_server)
if cached:
db_track_check = db.get_track_by_id(cached['server_track_id'])
if db_track_check:
class DatabaseTrackCached:
def __init__(self, db_t):
self.ratingKey = db_t.id
self.title = db_t.title
self.id = db_t.id
logger.debug(f"Sync cache hit: '{original_title}' → server track {cached['server_track_id']}")
return DatabaseTrackCached(db_track_check), cached['confidence']
logger.warning(f"Sync cache stale for '{original_title}' — track gone")
except Exception as e:
logger.debug("sync match cache fast-path failed: %s", e)
# --- End cache fast-path ---
# Try each artist (same logic as original)
for artist in spotify_track.artists:
# Extract artist name from both string and dict formats
if isinstance(artist, str):
artist_name = artist
elif isinstance(artist, dict) and 'name' in artist:
artist_name = artist['name']
else:
artist_name = str(artist)
db_track, confidence = db.check_track_exists(
original_title, artist_name,
confidence_threshold=0.80,
server_source=active_server
)
if db_track and confidence >= 0.80:
logger.info(f"Database match: '{db_track.title}' (confidence: {confidence:.2f})")
# Save to sync match cache
if spotify_id:
try:
from core.matching_engine import MusicMatchingEngine
me = MusicMatchingEngine()
db.save_sync_match_cache(
spotify_id, me.clean_title(original_title), me.clean_artist(artist_name),
active_server, db_track.id, db_track.title, confidence
)
except Exception as e:
logger.debug("save sync match cache failed: %s", e)
# Create mock track object for playlist creation
class DatabaseTrackMock:
def __init__(self, db_track):
self.ratingKey = db_track.id
self.title = db_track.title
self.id = db_track.id
return DatabaseTrackMock(db_track), confidence
logger.warning(f"No database match found for: '{original_title}'")
return None, 0.0
except Exception as e:
logger.error(f"Database search error: {e}")
return None, 0.0
# Patch the method
sync_service._find_track_in_media_server = database_only_find_track
logger.info("Patched sync service to use database-only matching")
sync_start_time = time.time()
@ -472,14 +356,7 @@ def run_sync_task(
# Wing It mode — skip wishlist for unmatched tracks
with sync_lock:
is_wing_it = sync_states.get(playlist_id, {}).get('wing_it', False)
sync_service._skip_unmatched_wishlist = is_wing_it or skip_wishlist_add
sync_service._skip_wishlist = is_wing_it
if skip_wishlist_add:
logger.info(
"[Organize by Playlist] Skipping sync-time wishlist for '%s' — "
"organize download + batch failure handling cover missing tracks",
playlist_name,
)
# Run the sync (this is a blocking call within this thread)
result = deps.run_async(sync_service.sync_playlist(playlist, download_missing=False, profile_id=profile_id, sync_mode=sync_mode))
@ -526,13 +403,7 @@ def run_sync_task(
# don't want persisted to app.log.
_synced = getattr(result, 'synced_tracks', 0)
logger.info(f"[PLAYLIST IMAGE] has_image={bool(playlist_image_url)}, synced_tracks={_synced}")
# Modes that edit a playlist in place (reconcile #792, append #811) must
# NOT push the source image — doing so re-clobbers a user's custom poster
# every sync, the exact bug these modes exist to avoid. Only the
# destructive 'replace' (recreate-from-scratch) pushes the image.
if sync_mode in ('reconcile', 'append'):
logger.info(f"[PLAYLIST IMAGE] {sync_mode} mode — preserving existing playlist image")
elif playlist_image_url and _synced > 0:
if playlist_image_url and _synced > 0:
try:
active_server = deps.config_manager.get_active_media_server()
logger.info(f"[PLAYLIST IMAGE] active_server={active_server}")
@ -584,21 +455,9 @@ def run_sync_task(
matched = getattr(result, 'matched_tracks', 0)
total = getattr(result, 'total_tracks', 0)
failed = getattr(result, 'failed_tracks', 0)
wishlist_added = getattr(result, 'wishlist_added_count', 0) or 0
deps.update_automation_progress(automation_id, status='finished', progress=100,
phase='Sync complete',
log_line=(
f'Done: {matched}/{total} in library, {failed} missing'
+ (f', {wishlist_added} added to wishlist' if wishlist_added else '')
),
log_type='success')
_post_sync_automation_followup(
deps,
automation_id=automation_id,
playlist_id=playlist_id,
skip_wishlist_add=skip_wishlist_add,
result=result,
)
log_line=f'Done: {matched}/{total} matched, {failed} failed', log_type='success')
# Emit playlist_synced event for automation engine
try:
@ -617,28 +476,12 @@ def run_sync_task(
import hashlib as _hl
_track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
_tracks_hash = _hl.md5(_track_ids_str.encode()).hexdigest()
_mirror_tracks_hash = None
if str(playlist_id).startswith('auto_mirror_'):
try:
_mp_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
from database.music_database import MusicDatabase
_mtracks = MusicDatabase().get_mirrored_playlist_tracks(_mp_id)
_mids = ','.join(
sorted(t.get('source_track_id', '') or '' for t in _mtracks if t.get('source_track_id'))
)
_mirror_tracks_hash = _hl.md5(_mids.encode()).hexdigest() if _mids else ''
except Exception as e:
logger.debug("mirror_tracks_hash for sync status: %s", e)
snapshot_id = getattr(playlist, 'snapshot_id', None)
_status_kwargs = dict(
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id,
matched_tracks=getattr(result, 'matched_tracks', 0),
total_tracks=getattr(result, 'total_tracks', 0),
discovered_tracks=len(tracks_json),
tracks_hash=_tracks_hash,
)
if _mirror_tracks_hash is not None:
_status_kwargs['mirror_tracks_hash'] = _mirror_tracks_hash
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id, **_status_kwargs)
tracks_hash=_tracks_hash)
except Exception as e:
logger.error(f"SYNC FAILED for {playlist_id}: {e}")

61
core/discovery/youtube.py
View file

@ -32,26 +32,6 @@ from typing import Any, Callable
logger = logging.getLogger(__name__)
_UNKNOWN_ARTIST = 'Unknown Artist'
def resolve_display_artist(yt_artist: str, matched_artist: str) -> str:
"""The artist to show in the 'YT Artist' column (#909).
YouTube's flat playlist data carries no artist, so a track starts as
"Unknown Artist" and only gains a real name if per-video recovery succeeds.
When recovery comes up empty but the track still matched confidently, show
the matched artist instead of a misleading "Unknown Artist". Returns the
original ``yt_artist`` whenever it's already a real name (recovery worked) or
when there's no matched artist to fall back to — purely a display choice, the
match itself is unaffected.
"""
current = (yt_artist or '').strip()
if current and current != _UNKNOWN_ARTIST:
return current # recovery already gave a real name — keep it
fallback = (matched_artist or '').strip()
return fallback or _UNKNOWN_ARTIST # backfill from the match, else honest Unknown
@dataclass
class YoutubeDiscoveryDeps:
@ -72,10 +52,6 @@ class YoutubeDiscoveryDeps:
build_discovery_wing_it_stub: Callable
get_database: Callable[[], Any]
add_activity_item: Callable
# Recover a YouTube track's artist from its own video page when flat playlist
# extraction left it "Unknown Artist" (#863). Takes a video id, returns a raw
# artist string or ''. Optional — discovery still works without it.
recover_youtube_artist: Callable[[str], str] = None
def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
@ -118,30 +94,6 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
cleaned_title = track['name']
cleaned_artist = track['artists'][0] if track['artists'] else 'Unknown Artist'
# Recover the artist from the track's own video page if flat
# playlist extraction left it Unknown (#863). Done here, in the
# background worker, rather than in the parse request (which would
# block for minutes on a big playlist). Per-track cost is hidden
# behind the discovery progress bar; the recovered artist makes the
# match below actually find the song.
if cleaned_artist == 'Unknown Artist' and track.get('id'):
if not deps.recover_youtube_artist:
logger.warning("[YT Discovery] artist recovery unavailable (dep not wired) "
"— '%s' stays Unknown", cleaned_title)
else:
try:
_rec = deps.recover_youtube_artist(track['id'])
except Exception as _rec_err:
logger.warning(f"[YT Discovery] artist recovery raised for {track.get('id')}: {_rec_err}")
_rec = ''
if _rec and _rec != 'Unknown Artist':
logger.info(f"[YT Discovery] recovered artist '{_rec}' for '{cleaned_title}' ({track['id']})")
cleaned_artist = _rec
track['artists'] = [_rec] # persist so retries/UI see it
else:
logger.info(f"[YT Discovery] artist recovery returned nothing for "
f"'{cleaned_title}' ({track['id']}) — leaving Unknown")
logger.info(f"Searching {discovery_source} for: '{cleaned_artist}' - '{cleaned_title}'")
# Check discovery cache first
@ -151,15 +103,14 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
if cached_match and deps.validate_discovery_cache_artist(cleaned_artist, cached_match):
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {cleaned_artist} - {cleaned_title}")
_match_artist = deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else ''
result = {
'index': i,
'yt_track': cleaned_title,
'yt_artist': resolve_display_artist(cleaned_artist, _match_artist),
'yt_artist': cleaned_artist,
'status': 'Found',
'status_class': 'found',
'spotify_track': cached_match.get('name', ''),
'spotify_artist': _match_artist,
'spotify_artist': deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else '',
'spotify_album': cached_match.get('album', {}).get('name', '') if isinstance(cached_match.get('album'), dict) else cached_match.get('album', ''),
'duration': f"{int(track['duration_ms']) // 60000}:{(int(track['duration_ms']) % 60000) // 1000:02d}" if track['duration_ms'] else '0:00',
'discovery_source': discovery_source,
@ -286,17 +237,15 @@ def run_youtube_discovery_worker(url_hash, deps: YoutubeDiscoveryDeps):
best_confidence = confidence
logger.info(f"Strategy 4 YouTube match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
# Create result entry. yt_artist falls back to the matched artist when
# YouTube/recovery left it "Unknown Artist" but we matched confidently (#909).
_match_artist = deps.extract_artist_name(matched_track.artists[0]) if matched_track else ''
# Create result entry
result = {
'index': i,
'yt_track': cleaned_title,
'yt_artist': resolve_display_artist(cleaned_artist, _match_artist),
'yt_artist': cleaned_artist,
'status': 'Found' if matched_track else 'Not Found',
'status_class': 'found' if matched_track else 'not-found',
'spotify_track': matched_track.name if matched_track else '',
'spotify_artist': _match_artist,
'spotify_artist': deps.extract_artist_name(matched_track.artists[0]) if matched_track else '',
'spotify_album': matched_track.album if matched_track else '',
'duration': f"{int(track['duration_ms']) // 60000}:{(int(track['duration_ms']) % 60000) // 1000:02d}" if track['duration_ms'] else '0:00',
'discovery_source': discovery_source,

87
core/download_engine/engine.py
View file

@ -25,7 +25,6 @@ big-bang switchover.
from __future__ import annotations
import asyncio
import threading
from typing import Any, Dict, Iterator, List, Optional, Tuple
@ -392,16 +391,6 @@ class DownloadEngine:
(tracks, albums) tuple, or ``([], [])`` when every source
in the chain is exhausted.
Priority mode is deliberately quality-AGNOSTIC at search time — source
order is king and the first source that returns any tracks wins, exactly
matching pre-quality-system behaviour byte-for-byte (#896 review #3).
Quality-gating the priority path would deprioritise e.g. a soulseek
mp3 whose bitrate slskd omitted (``bitrate=None`` → "unsatisfied"),
changing which source wins and adding latency for users who never opted
in. Cross-source quality pooling is the job of best_quality mode
(``search_all_sources``); final per-result ranking still happens in the
orchestrator's match/quality filter. RAW tracks are returned.
Replaces orchestrator's hand-rolled hybrid search loop. The
chain is ordered (most-preferred first).
"""
@ -417,10 +406,9 @@ class DownloadEngine:
try:
logger.info(f"Trying {source_name} (priority {i+1}): {query}")
tracks, albums = await plugin.search(query, timeout, progress_callback)
if not tracks:
continue
logger.info(f"{source_name} found {len(tracks)} tracks")
return (tracks, albums)
if tracks:
logger.info(f"{source_name} found {len(tracks)} tracks")
return (tracks, albums)
except Exception as e:
logger.warning(f"{source_name} search failed: {e}")
@ -430,75 +418,6 @@ class DownloadEngine:
)
return ([], [])
async def search_all_sources(self, query: str, source_chain,
timeout=None, progress_callback=None,
exclude_sources=None):
"""Best-quality mode: pool RAW tracks from EVERY configured source in
``source_chain`` instead of stopping at the first satisfying one.
Unlike :meth:`search_with_fallback`, no source short-circuits the
search — the caller (orchestrator/worker) ranks the combined pool
best→worst by actual audio quality. ``exclude_sources`` drops sources
whose per-source retry budget is already spent (so their candidates
never re-enter the pool). Unconfigured / unregistered / raising sources
are skipped exactly like the fallback path. Returns
``(combined_tracks, combined_albums)``.
"""
excluded = {s.lower() for s in (exclude_sources or []) if s}
pooled_tracks = []
pooled_albums = []
# Per-source contribution for an honest pool log — e.g. a release-level
# source like usenet/torrent that returns nothing for a track-title
# query should read "usenet=0", not silently hide behind the chain name.
contributions = []
# Decide which sources to actually query, recording why the rest were
# skipped. Searches then run CONCURRENTLY so the pool waits only for the
# slowest source (e.g. usenet/Prowlarr, which can be slow) rather than
# the sum of every source's latency.
to_search = [] # (source_name, plugin)
for source_name in source_chain:
if source_name.lower() in excluded:
contributions.append(f"{source_name}=excluded")
continue
plugin = self._plugins.get(source_name)
if plugin is None:
logger.info(f"Skipping {source_name} (not available)")
contributions.append(f"{source_name}=unavailable")
continue
if hasattr(plugin, 'is_configured') and not plugin.is_configured():
logger.info(f"Skipping {source_name} (not configured)")
contributions.append(f"{source_name}=unconfigured")
continue
to_search.append((source_name, plugin))
async def _one(plugin):
return await plugin.search(query, timeout, progress_callback)
results = await asyncio.gather(
*[_one(plugin) for _, plugin in to_search],
return_exceptions=True,
)
for (source_name, _), result in zip(to_search, results, strict=True):
if isinstance(result, Exception):
logger.warning(f"{source_name} search failed: {result}")
contributions.append(f"{source_name}=error")
continue
tracks, albums = result
n = len(tracks) if tracks else 0
if tracks:
pooled_tracks.extend(tracks)
if albums:
pooled_albums.extend(albums)
contributions.append(f"{source_name}={n}")
logger.info(
"Best-quality pool: %d candidates [%s] for: %s",
len(pooled_tracks), ', '.join(contributions), query,
)
return (pooled_tracks, pooled_albums)
async def download_with_fallback(self, username: str, filename: str,
file_size: int, source_chain) -> Optional[str]:
"""Try each source in ``source_chain`` until one accepts the

76
core/download_orchestrator.py
View file

@ -28,7 +28,6 @@ from config.settings import config_manager
from core.download_engine import DownloadEngine
from core.download_plugins.registry import DownloadPluginRegistry, build_default_registry
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
from core.quality.selection import load_search_mode
logger = get_logger("download_orchestrator")
@ -103,35 +102,17 @@ class DownloadOrchestrator:
deezer_dl = self.client('deezer_dl')
if deezer_arl and deezer_dl:
deezer_dl.reconnect(deezer_arl)
from core.quality.source_map import quality_tier_for_source
deezer_dl._quality = quality_tier_for_source('deezer', default='flac')
deezer_dl._quality = config_manager.get('deezer_download.quality', 'flac')
# Reload Amazon quality preference (T2Tunes needs no reconnect — public proxy)
amazon = self.client('amazon')
if amazon:
from core.quality.source_map import quality_tier_for_source
quality = quality_tier_for_source('amazon', default='flac')
quality = config_manager.get('amazon_download.quality', 'flac')
amazon._quality = quality
amazon._allow_fallback = config_manager.get('amazon_download.allow_fallback', True)
if hasattr(amazon, '_client') and amazon._client:
amazon._client.preferred_codec = quality
# Let registry-backed plugins refresh any config they cache at
# construction time. This covers Prowlarr-backed torrent / usenet
# clients without rebuilding the registry and losing active downloads.
for name, client in self.registry.all_plugins():
if not hasattr(client, 'reload_settings'):
continue
try:
client.reload_settings()
logger.info("%s client settings reloaded", self.registry.display_name(name))
except Exception as exc:
logger.warning(
"%s client settings reload failed: %s",
self.registry.display_name(name),
exc,
)
# Reload download path for all clients that cache it.
# Soulseek owns the path config and is reloaded above; every
# other source mirrors that path so files all land in one
@ -143,10 +124,7 @@ class DownloadOrchestrator:
continue
if hasattr(client, 'download_path') and client.download_path != new_path:
client.download_path = new_path
try:
client.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {new_path}: {e}")
client.download_path.mkdir(parents=True, exist_ok=True)
# YouTube also caches path in yt-dlp opts
if hasattr(client, 'download_opts') and 'outtmpl' in client.download_opts:
client.download_opts['outtmpl'] = str(new_path / '%(title)s.%(ext)s')
@ -308,25 +286,11 @@ class DownloadOrchestrator:
chain = ['soulseek']
return chain
async def search(self, query: str, timeout: int = None, progress_callback=None,
exclude_sources=None) -> Tuple[List[TrackResult], List[AlbumResult]]:
async def search(self, query: str, timeout: int = None, progress_callback=None) -> Tuple[List[TrackResult], List[AlbumResult]]:
"""Search for tracks using configured source(s). Single-source
modes route directly; hybrid mode delegates to
``engine.search_with_fallback`` which tries the chain in order.
``exclude_sources`` (optional) is an iterable of source names
the caller wants filtered out of the hybrid chain. Used by the
per-track download worker to skip torrent / usenet for album-
context batches — those sources are release-level and don't
score meaningfully on per-track titles; the album-bundle flow
on the master worker handles them separately when they're the
single active source."""
``engine.search_with_fallback`` which tries the chain in order."""
if self.mode != 'hybrid':
# Single-source mode is opt-in; honour the user's choice even
# if it's torrent/usenet on an album batch (the master worker
# routes those through the album-bundle flow before per-track
# tasks ever fire, so search() being called here would be a
# non-album / wishlist / basic-search use case).
client = self._client(self.mode)
if not client:
logger.error(f"{self.registry.display_name(self.mode)} client not available (failed to initialize)")
@ -335,24 +299,6 @@ class DownloadOrchestrator:
return await client.search(query, timeout, progress_callback)
chain = self._resolve_source_chain()
if exclude_sources:
blocked = {s.lower() for s in exclude_sources if s}
filtered = [s for s in chain if s.lower() not in blocked]
if filtered != chain:
logger.info(
"Hybrid search: excluding %s for this query (chain %s -> %s)",
sorted(blocked & {s.lower() for s in chain}),
" → ".join(chain), " → ".join(filtered) if filtered else "(empty)",
)
chain = filtered
if not chain:
logger.warning("Hybrid search exhausted: no eligible sources after exclusion filter")
return [], []
if load_search_mode() == 'best_quality':
logger.info(f"Best-quality search ({' → '.join(chain)}): {query}")
return await self.engine.search_all_sources(
query, chain, timeout, progress_callback,
)
logger.info(f"Hybrid search ({' → '.join(chain)}): {query}")
return await self.engine.search_with_fallback(query, chain, timeout, progress_callback)
@ -423,17 +369,9 @@ class DownloadOrchestrator:
if scored:
scored.sort(key=lambda x: x._match_confidence, reverse=True)
# Match filter done (right track); now prefer the best quality
# among the confidence-passing survivors so streaming isn't
# quality-blind like Soulseek already isn't. Stable ranking
# keeps confidence order within an equal quality tier; the
# `or scored` fail-safe never leaves us with nothing to try.
from core.quality.selection import rank_for_profile
ranked, _ = rank_for_profile(scored)
filtered_results = ranked or scored
filtered_results = scored
logger.info(f"Streaming validation: {len(scored)}/{len(tracks)} passed "
f"(best: {scored[0]._match_confidence:.2f}, "
f"quality pick: {filtered_results[0].audio_quality.label()})")
f"(best: {scored[0]._match_confidence:.2f})")
else:
logger.warning(f"No streaming results passed validation for: {query}")
return None

766
core/download_plugins/album_bundle.py
View file

@ -1,766 +0,0 @@
"""Shared helpers for the album-bundle download flow.
The torrent and usenet download plugins both implement a
``download_album_to_staging`` method that searches Prowlarr for a
whole release, hands it to the active downloader, walks the
resulting audio files, and copies them into the staging folder. The
two implementations share the same release-picker heuristic and the
same staging-path collision logic.
Pulled out of ``core/download_plugins/torrent.py`` so the usenet
plugin doesn't have to import private helpers from a sibling
plugin (Cin's "no leaky module boundaries" standard).
Also exposes ``atomic_copy_to_staging`` — the audio file is copied
to a ``.tmp.<random>`` sidecar first and atomically renamed onto its
final extension. The Auto-Import worker filters by audio extension
so the in-flight ``.tmp`` file is never picked up mid-copy, closing
the race between the album-bundle copy loop and Auto-Import's
folder scan.
"""
from __future__ import annotations
import re
import shutil
import time
import unicodedata
import uuid
from pathlib import Path
from typing import Any, Callable, Iterable, Optional
from config.settings import config_manager
from utils.logging_config import get_logger
logger = get_logger("download_plugins.album_bundle")
# Minimum album-title relevance a Prowlarr candidate must clear to be eligible
# for an album-bundle download (#730). Prowlarr returns broad fuzzy matches — a
# "Heroes" search also returns other Bowie albums — so without this gate the
# most-popular result wins regardless of whether it's the right album. Below
# this floor we refuse the bundle and let the caller fall back to per-track.
_ALBUM_TITLE_RELEVANCE_FLOOR = 0.6
# Album-pick size floor / ceiling. Single-track torrents (~10 MB)
# are rejected when bigger candidates exist; anything past 3 GB is
# treated as suspicious (multi-disc box-set + scans + extras).
ALBUM_PICK_MIN_BYTES = 40 * 1024 * 1024
ALBUM_PICK_MAX_BYTES = 3 * 1024 * 1024 * 1024
# Quality-score weights for the album-pick heuristic. Mirrors the
# tier order in ``core/imports/file_ops.py``'s ``quality_tiers`` —
# higher number = preferred.
_QUALITY_SCORE = {'flac': 4, 'ogg': 3, 'aac': 2, 'mp3': 1}
# Default poll cadence + timeout for the album-download poll loop.
# Both are overridable through config so users with slow trackers
# / large box-sets can extend the deadline without editing code.
DEFAULT_POLL_INTERVAL_SECONDS = 2.0
DEFAULT_POLL_TIMEOUT_SECONDS = 6 * 60 * 60
def get_poll_interval() -> float:
"""Return the per-poll sleep duration (seconds). Configurable via
``download_source.album_bundle_poll_interval_seconds``."""
raw = config_manager.get('download_source.album_bundle_poll_interval_seconds',
DEFAULT_POLL_INTERVAL_SECONDS)
try:
value = float(raw)
if value > 0:
return value
except (TypeError, ValueError):
pass
return DEFAULT_POLL_INTERVAL_SECONDS
def get_poll_timeout() -> float:
"""Return the total deadline for an album-bundle download
(seconds). Configurable via
``download_source.album_bundle_timeout_seconds``."""
raw = config_manager.get('download_source.album_bundle_timeout_seconds',
DEFAULT_POLL_TIMEOUT_SECONDS)
try:
value = float(raw)
if value > 0:
return value
except (TypeError, ValueError):
pass
return DEFAULT_POLL_TIMEOUT_SECONDS
def quality_score(title: str, quality_guess) -> int:
"""Map a release title's inferred quality to a sortable integer.
``quality_guess`` is the function from each plugin that maps a
title string to a quality string ('flac' / 'mp3' / etc.) — passed
in so this module doesn't have to import either plugin and risk
a circular import."""
return _QUALITY_SCORE.get(quality_guess(title) or '', 0)
def _normalize_release_text(text: str) -> str:
"""Lowercase, fold accents (Björk -> bjork), strip punctuation to spaces.
NFKD-decompose then drop combining marks so accented characters fold to
their base letter instead of fragmenting (the naive approach turned
'Björk' into 'bj rk'). Collapses runs of whitespace.
"""
if not text:
return ""
decomposed = unicodedata.normalize("NFKD", text)
stripped = "".join(c for c in decomposed if not unicodedata.combining(c))
lowered = stripped.lower()
# Punctuation -> space (so "heroes" matches "heroes:" / "heroes -"),
# then collapse whitespace.
cleaned = re.sub(r"[^a-z0-9]+", " ", lowered)
return re.sub(r"\s+", " ", cleaned).strip()
# Edition / format / qualifier words that appear in stored album names or
# release titles but say nothing about WHICH album it is. Stripped before
# scoring so "Currents" matches "Currents (Deluxe)" and "Heroes" matches
# "Heroes (2017 Remaster)" — the #730 fix must not reject the RIGHT album just
# because the DB name carries an edition suffix the torrent title lacks.
_ALBUM_NOISE_WORDS = frozenset({
"deluxe", "edition", "remaster", "remastered", "remasters", "remix",
"expanded", "anniversary", "bonus", "version", "explicit", "clean",
"reissue", "special", "limited", "collectors", "collector", "the",
"ep", "lp", "album", "single", "disc", "cd", "vol", "volume",
"flac", "mp3", "aac", "ogg", "wav", "alac", "m4a", "320", "256", "192",
"web", "vinyl", "hi", "res", "hires", "24bit", "16bit", "original",
"soundtrack", "ost",
})
def _significant_words(normalized: str) -> list:
"""Words that actually identify an album: drop pure-digit tokens (years,
bitrates) and edition/format noise. Keeps at least the raw words if the
filter would empty it (e.g. an album literally named '1989' or 'Deluxe')."""
words = [w for w in normalized.split()
if w not in _ALBUM_NOISE_WORDS and not w.isdigit()]
return words or normalized.split()
def album_title_relevance(candidate_title: str, album_name: str) -> float:
"""How well a release title matches the requested album, 0.0–1.0.
Scores the fraction of the album's SIGNIFICANT words (edition/format/year
noise removed) that appear as whole words in the candidate title.
Word-boundary, not substring, so "Heroes" does NOT match "Superheroes" and
a different album sharing no significant words scores 0 — while "Currents"
still matches "Currents (Deluxe)" and "Heroes" matches the "2017 Remaster".
Returns 1.0 when there's no album name to check (can't gate on nothing —
preserves old behavior for callers that don't pass a title).
"""
norm_album = _normalize_release_text(album_name)
if not norm_album:
return 1.0
norm_title = _normalize_release_text(candidate_title)
if not norm_title:
return 0.0
album_words = _significant_words(norm_album)
title_words = set(norm_title.split())
if not album_words:
return 1.0
matched = sum(1 for w in album_words if w in title_words)
coverage = matched / len(album_words)
# Full-phrase bonus (idea from contributor PR #731): when the album's core
# phrase appears intact in the title, we're highly confident it's the right
# release even if token-coverage is dragged down by a long multi-word name.
# MUST be word-boundary anchored, NOT a raw substring — a naive
# `phrase in norm_title` lets "heroes" match "superheroes" and reintroduces
# the exact wrong-album bug #730 fixes (PR #731's version has this flaw).
core_phrase = " ".join(album_words)
if core_phrase and re.search(rf"(?:^| ){re.escape(core_phrase)}(?: |$)", norm_title):
coverage = max(coverage, 0.9)
return coverage
def pick_best_album_release(candidates, quality_guess,
album_name: str = "") -> Optional[object]:
"""Pick the single best torrent / NZB for an album-bundle download.
Heuristic, in priority order:
0. Album-TITLE relevance gate (#730): drop candidates whose title doesn't
sufficiently match the requested album. Prowlarr returns broad fuzzy
matches, so without this the most-popular result wins even when it's a
different album. When ``album_name`` is given and NOTHING clears the
relevance floor, return None — the caller then falls back to per-track
rather than downloading a confident mismatch.
1. Reasonable album-ish size (40 MB – 3 GB) — drops single-track
releases that snuck in and quarantines suspicious giants.
2. Higher seeders > lower (dead torrents = dead downloads).
Usenet releases use ``grabs`` as a popularity proxy when
seeders is None.
3. Higher quality (FLAC > AAC > MP3) inferred from title.
4. Larger size as tiebreaker (often = higher bitrate).
"""
if not candidates:
return None
# 0. Title-relevance gate. Only applied when we know the album name; with
# no name we can't judge relevance, so we don't gate (old behavior).
if album_name:
relevant = [
c for c in candidates
if album_title_relevance(c.title or "", album_name) >= _ALBUM_TITLE_RELEVANCE_FLOOR
]
if not relevant:
logger.warning(
"[Album Bundle] No candidate cleared the title-relevance floor "
"for '%s' (%d candidates rejected as wrong album) — refusing the "
"bundle so the caller falls back to per-track.",
album_name, len(candidates),
)
return None
candidates = relevant
sized = [c for c in candidates
if ALBUM_PICK_MIN_BYTES <= (c.size or 0) <= ALBUM_PICK_MAX_BYTES]
pool = sized or list(candidates)
if not pool:
return None
def _score(c) -> tuple:
seeders = c.seeders if c.seeders is not None else (c.grabs or 0)
return (seeders, quality_score(c.title or '', quality_guess), c.size or 0)
return max(pool, key=_score)
def unique_staging_path(staging_dir: Path, src: Path) -> Path:
"""Return a destination path inside ``staging_dir`` that doesn't
collide with an existing file. Appends ``_1``, ``_2``, ... before
the extension when needed; gives up after 1000 candidates and
returns the unsuffixed path so the caller will overwrite (better
than infinite loop or crash)."""
dest = staging_dir / src.name
if not dest.exists():
return dest
stem = dest.stem
suffix = dest.suffix
for i in range(1, 1000):
candidate = staging_dir / f"{stem}_{i}{suffix}"
if not candidate.exists():
return candidate
return dest
def atomic_copy_to_staging(src: Path, dest: Path) -> bool:
"""Copy ``src`` to ``dest`` without exposing a partial file to
folder scanners.
The Auto-Import worker filters by audio extension when scanning
Staging — see ``AUDIO_EXTENSIONS`` in ``core/auto_import_worker.py``.
Naming the in-flight file ``<dest>.tmp.<random>`` keeps it
invisible until the rename atomically swings it to its final
extension. ``os.replace`` (used by ``Path.rename`` on Python 3.x)
is atomic on the same filesystem, so Auto-Import either sees the
file at its final name (complete) or doesn't see it at all
(in flight).
Returns True on success, False on copy / rename failure. Caller
is expected to log the failure case so we don't double-log here.
"""
tmp = dest.with_name(f"{dest.name}.tmp.{uuid.uuid4().hex[:8]}")
try:
shutil.copy2(src, tmp)
except Exception:
# Best-effort cleanup of the partial file. If unlink fails
# (locked, permissions) we leave it — Auto-Import ignores it
# anyway because of the .tmp extension.
try:
if tmp.exists():
tmp.unlink()
except Exception as cleanup_exc:
logger.debug("album_bundle tmp cleanup failed: %s", cleanup_exc)
raise
try:
tmp.replace(dest)
return True
except Exception:
try:
tmp.unlink(missing_ok=True)
except Exception as cleanup_exc:
logger.debug("album_bundle tmp cleanup failed: %s", cleanup_exc)
raise
# Number of consecutive None-status reads tolerated before treating the
# job as gone. Sized for the SAB queue→history transition window: SAB
# removes the slot from the queue before adding it to history, and on a
# busy server (par2 verify + unrar) that window can be several poll
# intervals. At the default 2s interval, 5 retries = ~10s of tolerance
# before we give up and emit a terminal failure. Override via
# ``download_source.album_bundle_transient_miss_threshold`` for users
# whose servers need more headroom (very large multi-disc box sets,
# slow disks, etc.).
DEFAULT_TRANSIENT_MISS_THRESHOLD = 5
def get_transient_miss_threshold() -> int:
"""Return the configured transient-miss threshold for poll loops."""
raw = config_manager.get('download_source.album_bundle_transient_miss_threshold',
DEFAULT_TRANSIENT_MISS_THRESHOLD)
try:
value = int(raw)
if value > 0:
return value
except (TypeError, ValueError):
pass
return DEFAULT_TRANSIENT_MISS_THRESHOLD
# How long to keep polling after the client reports terminal success
# but hasn't yet exposed a final save_path. Distinct from the
# transient-miss threshold because the two model different things:
# a transient miss is "the job vanished — fail fast (~10s) so a deleted
# job doesn't hang"; a completed-no-path read is "the download SUCCEEDED
# and the files are on disk — SAB just hasn't finished writing the
# ``storage`` field." The #706 fix reused the 5-poll (~10s) miss window
# here, but #721's own report shows SAB can take 2+ minutes (or, on some
# versions, never expose ``storage`` at all) — so a 10s window false-fails
# a download that actually completed. Expressed in SECONDS (converted to
# a poll count against the live interval) so it's interval-independent.
# Override via ``download_source.album_bundle_completed_no_path_seconds``.
DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS = 120.0
def get_completed_no_path_window_seconds() -> float:
"""Return the completed-but-no-save_path tolerance window (seconds)."""
raw = config_manager.get('download_source.album_bundle_completed_no_path_seconds',
DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS)
try:
value = float(raw)
if value > 0:
return value
except (TypeError, ValueError):
pass
return DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS
class TransientMissCounter:
"""Bounded retry counter for adapter status reads.
Both the album-bundle poll (in ``poll_album_download``) and the
per-track download threads in ``usenet.py`` / ``torrent.py`` need
the same "tolerate N consecutive missing or unmapped reads before
declaring the job gone" logic. Lifted into one class so the rule
is in one place and unit-testable in isolation — the per-track
paths used to carry inline counters that mirrored this logic by
hand, which is exactly the kind of duplication that drifts."""
def __init__(self, threshold: Optional[int] = None) -> None:
self.threshold = threshold if threshold is not None else get_transient_miss_threshold()
self.misses = 0
def record_miss(self) -> bool:
"""Bump the miss counter. Returns True when the counter has
reached the threshold (caller should give up)."""
self.misses += 1
return self.misses >= self.threshold
def reset(self) -> None:
"""Successful read — reset the counter back to zero."""
self.misses = 0
def poll_album_download(
*,
get_status: Callable[[], Optional[Any]],
title: str,
emit: Callable[..., None],
complete_states: frozenset,
failed_states: frozenset = frozenset(['failed']),
is_shutdown: Optional[Callable[[], bool]] = None,
transient_miss_threshold: int = DEFAULT_TRANSIENT_MISS_THRESHOLD,
completed_no_path_threshold: Optional[int] = None,
poll_interval: Optional[float] = None,
timeout: Optional[float] = None,
sleep: Callable[[float], None] = time.sleep,
monotonic: Callable[[], float] = time.monotonic,
log_prefix: str = '[album_bundle]',
) -> Optional[str]:
"""Drive the per-poll status loop for an album-bundle download.
Lifted out of ``UsenetDownloadPlugin._poll_album_download`` and the
sibling torrent method so the loop is testable in isolation and so
both plugins share the same exit semantics.
Contract:
- ``get_status()`` returns the adapter status object for the bound
job, ``None`` when the client doesn't know about the job
currently (transient or terminal — disambiguated by retry count).
- ``emit(state, **fields)`` is the plugin's progress callback —
this function calls it on EVERY successful poll with
``state='downloading'`` and ALWAYS calls it once more with
``state='failed'`` before returning ``None`` on any failure path,
so the UI doesn't freeze on the last 'downloading' emit.
- ``complete_states`` is the adapter's terminal-success set
('completed' alone for usenet; 'seeding' + 'completed' for
torrent because seeding-but-files-on-disk also counts).
- ``failed_states`` is the explicit-failure set. The adapter-level
'error' (unmapped state default) is intentionally NOT in here —
that's treated as a transient miss because a real SAB / NZBGet
/ qBit never returns a literal 'error' state on a healthy job;
it's only our default fallback for unknown queue strings. Real
example: SAB's 'Pp' post-processing state was unmapped → became
'error' → poll infinite-looped until the 6-hour timeout.
- ``transient_miss_threshold`` is the number of consecutive None /
'error' reads tolerated before declaring the job gone. Sized for
the SAB queue→history gap window (~10s) — a vanished job should
fail fast.
- ``completed_no_path_threshold`` is a SEPARATE, longer window for
the "client says complete but no save_path yet" case. The download
already succeeded, so this defaults to ~120s (configurable via
``download_source.album_bundle_completed_no_path_seconds``) instead
of reusing the 10s miss window — #721 showed SAB can take 2+ minutes
to write ``storage``. When the window is exhausted the loop falls
back to the adapter's ``incomplete_path`` (the on-disk in-progress
dir) if present, and only emits terminal ``failed`` when there's no
path of any kind to scan.
Returns the adapter's reported save_path (or, as a last resort, its
``incomplete_path``) on terminal success, or ``None`` on any failure
(timeout / disappeared / explicit failed / shutdown). On every
failure path emits ``'failed'`` once with an ``error`` field
describing why.
"""
interval = poll_interval if poll_interval is not None else get_poll_interval()
deadline = monotonic() + (timeout if timeout is not None else get_poll_timeout())
last_save_path: Optional[str] = None
last_incomplete_path: Optional[str] = None
misses = TransientMissCounter(transient_miss_threshold)
# Separate counter for "client reports terminal-success state but no
# save_path field has landed yet." SAB History flips ``status`` to
# 'Completed' a few seconds before its post-processing pipeline
# writes the final ``storage`` field — see issue #721 (Forty Licks
# stuck at 61%): SAB shows Completed in the UI, but
# ``_parse_history_slot`` returns ``save_path=None`` for those few
# seconds because ``storage`` isn't populated yet. Pre-fix the
# poll returned ``None`` on the first such read, the bundle
# plugin marked the batch failed, but the UI still displayed the
# last ``downloading`` progress emit.
#
# This window is intentionally LONGER than the transient-miss window:
# the download already SUCCEEDED, so being patient here is cheap and
# correct, whereas the original 5-poll (~10s) reuse false-failed real
# completions (#721 reported SAB taking 2+ minutes). Default ~120s,
# converted from seconds to a poll count against the live interval.
if completed_no_path_threshold is None:
completed_no_path_threshold = max(
transient_miss_threshold,
int(get_completed_no_path_window_seconds() / max(interval, 0.001)) or 1,
)
completed_no_path_misses = TransientMissCounter(completed_no_path_threshold)
def _fail(reason: str) -> None:
try:
emit('failed', release=title, error=reason)
except Exception as cb_exc:
logger.debug("%s terminal emit failed: %s", log_prefix, cb_exc)
# Heartbeat so the otherwise-silent download loop is diagnosable.
# The loop emits progress to the UI on every poll but logs nothing
# during normal operation — which made the #721 "stuck at N%" reports
# impossible to triage from logs alone (we couldn't tell if the poll
# was alive, what state SAB returned, or whether it had wedged). Log
# the raw adapter read at most once per heartbeat interval.
HEARTBEAT_SECONDS = 30.0
last_heartbeat = monotonic()
poll_count = 0
while monotonic() < deadline:
if is_shutdown and is_shutdown():
# Shutdown is a clean exit — don't paint failure on the UI;
# the app is going away anyway.
return None
try:
status = get_status()
except Exception as e:
logger.warning("%s Poll error: %s", log_prefix, e)
status = None
poll_count += 1
now = monotonic()
if now - last_heartbeat >= HEARTBEAT_SECONDS:
last_heartbeat = now
if status is None:
logger.info("%s '%s' poll #%d: client returned no status (miss %d/%d)",
log_prefix, title, poll_count, misses.misses, misses.threshold)
else:
logger.info(
"%s '%s' poll #%d: state=%r progress=%.2f save_path=%r",
log_prefix, title, poll_count,
getattr(status, 'state', None), getattr(status, 'progress', 0.0) or 0.0,
getattr(status, 'save_path', None),
)
if status is None:
if misses.record_miss():
logger.error(
"%s '%s' missing from client for %d consecutive polls — giving up",
log_prefix, title, misses.misses,
)
_fail('Disappeared from client (no status after retries)')
return None
sleep(interval)
continue
# Reset the miss counter only when the adapter returned a state
# we actually recognise. The default-fallback 'error' is treated
# as a continuing transient miss below, so it must NOT reset
# here — otherwise a persistently-unmapped state loops forever.
if status.state != 'error':
misses.reset()
emit('downloading', progress=status.progress, downloaded=status.downloaded,
speed=status.download_speed)
if status.save_path:
last_save_path = status.save_path
# Remember the in-progress dir too — never used on a normal
# completion, only as the last-resort fallback below when the
# final save_path provably never lands.
incomplete_path = getattr(status, 'incomplete_path', None)
if incomplete_path:
last_incomplete_path = incomplete_path
if status.state in complete_states:
if last_save_path:
completed_no_path_misses.reset()
return last_save_path
# Terminal-success state but no save_path landed yet.
# SAB History flips ``Completed`` a few seconds before
# ``storage`` is populated — give the adapter a generous
# window before declaring this a hard failure. Without this
# tolerance, every TAR / unrar-bearing usenet release
# would race the path-write window and randomly fail.
if completed_no_path_misses.record_miss():
# Last resort before failing: SAB finished and the files
# are physically on disk (#721), but the final ``storage``
# field never landed. Fall back to the in-progress dir so
# the bundle can still scan + stage the audio, rather than
# leaving the user stuck with a completed-in-SAB download
# that SoulSync never imports.
if last_incomplete_path:
logger.warning(
"%s '%s' completed on the client but never exposed a final "
"save_path after %d polls — falling back to the in-progress "
"path %r as a last resort. If staging fails, the SAB job "
"likely needs its post-process move to finish first.",
log_prefix, title, completed_no_path_misses.misses,
last_incomplete_path,
)
return last_incomplete_path
logger.error(
"%s '%s' reported terminal success but no save_path landed "
"after %d consecutive polls — bundle cannot stage. Adapter "
"may need new history-slot fallback fields (storage / path "
"/ download_path / dirname). Last status: state=%r progress=%r",
log_prefix, title, completed_no_path_misses.misses,
status.state, status.progress,
)
_fail('Client reported success but never provided a save_path')
return None
logger.info(
"%s '%s' is %s on the client but save_path not yet set — "
"retrying (poll %d/%d)",
log_prefix, title, status.state,
completed_no_path_misses.misses, completed_no_path_misses.threshold,
)
sleep(interval)
continue
if status.state in failed_states:
error = getattr(status, 'error', None) or 'Client reported failure'
logger.error("%s '%s' failed: %s", log_prefix, title, error)
_fail(error)
return None
if status.state == 'error':
# Unmapped adapter state — see contract docstring. Warn so
# we hear about new states the adapter map needs to grow
# without breaking the user's download. The miss counter
# was intentionally NOT reset above for this branch.
logger.warning(
"%s '%s' returned unmapped state — treating as transient",
log_prefix, title,
)
if misses.record_miss():
_fail('Client returned unmapped state repeatedly')
return None
sleep(interval)
logger.error("%s '%s' timed out", log_prefix, title)
_fail('Download timed out')
return None
def _candidate_download_roots(config_get: Callable[..., Any]) -> list:
"""Directories where THIS process can read finished downloads — used by
``resolve_reported_save_path`` for the basename fallback.
Order matters: most-specific usenet/torrent roots first, then the
general Soulseek download / transfer dirs, which in the standard
shared-volume arr setup are bind-mounted to the very directory the
usenet client writes its completed downloads into. Relative values
(e.g. ``./downloads``) resolve against the process CWD — the
container's ``/app`` — which is exactly where those mounts live.
"""
roots: list = []
for key in (
'download_source.usenet_download_path',
'usenet_client.completed_path',
'usenet_client.download_path',
'download_source.torrent_download_path',
'soulseek.download_path',
'soulseek.transfer_path',
):
value = config_get(key, None)
if value:
roots.append(str(value))
seen: set = set()
out: list = []
for root in roots:
if root not in seen:
seen.add(root)
out.append(root)
return out
def resolve_reported_save_path(
reported_path: Optional[str],
config_get: Optional[Callable[..., Any]] = None,
) -> Optional[str]:
"""Translate a downloader-reported save_path into one THIS process can read.
Usenet / torrent clients report paths from inside THEIR OWN container
(e.g. SAB hands back ``/data/downloads/music/<album>``); SoulSync often
mounts the very same files at a different point (``/app/downloads/<album>``).
Feeding the client's path straight to the audio walker then yields
"No audio files found" even though the files are physically present —
the classic arr-stack remote-path mismatch.
Resolution order:
1. The reported path verbatim, if it's a readable directory here
(deployments that mirror the client's mount paths).
2. Explicit prefix mappings from ``download_source.usenet_path_mappings``
— a list of ``{"from": "...", "to": "..."}`` (Sonarr/Radarr-style
remote path mapping) for non-shared / oddly-mounted layouts.
3. Basename fallback: a same-named folder under a known SoulSync
download root. Zero-config for the standard shared-volume setup —
the album folder shows up under SoulSync's own ``./downloads``
mount with the same name the client reported.
Returns the best resolved path, or ``reported_path`` unchanged when
nothing better is found (so the caller's existing "no audio" error still
surfaces, with both paths logged).
"""
if not reported_path:
return reported_path
if config_get is None:
config_get = config_manager.get
def _is_dir(candidate) -> bool:
try:
return Path(candidate).is_dir()
except OSError:
return False
# 1. Reported path is directly readable — mounts already line up.
if _is_dir(reported_path):
return reported_path
normalized = str(reported_path).replace('\\', '/')
# 2. Explicit prefix mappings (remote-path-mapping escape hatch).
mappings = config_get('download_source.usenet_path_mappings', None) or []
if isinstance(mappings, (list, tuple)):
for mapping in mappings:
if not isinstance(mapping, dict):
continue
frm = str(mapping.get('from') or '').replace('\\', '/').rstrip('/')
to = str(mapping.get('to') or '')
if not frm or not to:
continue
if normalized == frm or normalized.startswith(frm + '/'):
rest = normalized[len(frm):].lstrip('/')
candidate = str(Path(to) / rest) if rest else to
if _is_dir(candidate):
return candidate
# 3. Basename fallback under known download roots — covers the standard
# shared-volume layout with zero configuration.
basename = Path(normalized).name
if basename:
for root in _candidate_download_roots(config_get):
candidate = Path(root) / basename
if _is_dir(candidate):
return str(candidate)
return reported_path
def copy_audio_files_atomically(
sources: Iterable[Path], staging_dir: Path, remove_source: bool = False,
) -> list:
"""Convenience wrapper: pick a non-colliding staging path for
each source, copy via ``atomic_copy_to_staging``. Returns the
list of final destination paths (as strings). Files that fail
to copy are logged and skipped; the caller decides what to do
with a partial result.
``remove_source=True`` deletes each source AFTER it copies
successfully — used by the Soulseek bundle path so slskd's
completed downloads don't pile up in its download folder (#796).
Kept False for torrent/usenet, whose clients must retain the
originals (seeding / client-managed)."""
staging_dir.mkdir(parents=True, exist_ok=True)
out: list = []
for src in sources:
dest = unique_staging_path(staging_dir, src)
try:
atomic_copy_to_staging(src, dest)
out.append(str(dest))
if remove_source:
# Only after a verified copy — never lose data on a failed stage.
try:
Path(src).unlink()
except FileNotFoundError:
pass
except Exception as e:
logger.debug("[album_bundle] Could not remove staged source %s: %s", src, e)
except Exception as e:
logger.warning("[album_bundle] Failed to stage %s -> %s: %s", src, dest, e)
return out
# Re-export so callers don't have to remember which module owns
# what. The ``time`` import is kept so plugins can ``from
# core.download_plugins.album_bundle import time`` if they want to,
# avoiding a second std-lib import line for a single use.
__all__ = [
"ALBUM_PICK_MIN_BYTES",
"ALBUM_PICK_MAX_BYTES",
"DEFAULT_POLL_INTERVAL_SECONDS",
"DEFAULT_POLL_TIMEOUT_SECONDS",
"DEFAULT_TRANSIENT_MISS_THRESHOLD",
"DEFAULT_COMPLETED_NO_PATH_WINDOW_SECONDS",
"TransientMissCounter",
"atomic_copy_to_staging",
"copy_audio_files_atomically",
"get_completed_no_path_window_seconds",
"get_poll_interval",
"get_poll_timeout",
"get_transient_miss_threshold",
"resolve_reported_save_path",
"pick_best_album_release",
"poll_album_download",
"quality_score",
"time",
"unique_staging_path",
]

4
core/download_plugins/registry.py
View file

@ -40,8 +40,6 @@ from core.download_plugins.base import DownloadSourcePlugin
# orchestrator did.
from core.amazon_download_client import AmazonDownloadClient
from core.deezer_download_client import DeezerDownloadClient
from core.download_plugins.torrent import TorrentDownloadPlugin
from core.download_plugins.usenet import UsenetDownloadPlugin
from core.hifi_client import HiFiClient
from core.lidarr_download_client import LidarrDownloadClient
from core.qobuz_client import QobuzClient
@ -192,7 +190,5 @@ def build_default_registry() -> DownloadPluginRegistry:
aliases=('deezer_dl',)))
registry.register(PluginSpec(name='lidarr', factory=LidarrDownloadClient, display_name='Lidarr'))
registry.register(PluginSpec(name='soundcloud',factory=SoundcloudClient, display_name='SoundCloud'))
registry.register(PluginSpec(name='torrent', factory=TorrentDownloadPlugin, display_name='Torrent (Prowlarr)'))
registry.register(PluginSpec(name='usenet', factory=UsenetDownloadPlugin, display_name='Usenet (Prowlarr)'))
return registry

764
core/download_plugins/torrent.py
View file

@ -1,764 +0,0 @@
"""TorrentDownloadPlugin — composes Prowlarr search + torrent client
adapter + archive_pipeline into a uniform download source.
Two flows:
**Per-track flow** (basic search, single-track wishlist) —
1. ``search(query)`` calls ``ProwlarrClient.search`` filtered to
``protocol='torrent'`` results, projects releases into
``TrackResult`` / ``AlbumResult`` shaped objects the existing
search UI already understands. Encodes the indexer's
``downloadUrl`` (or magnet URI) into the filename so
``download()`` can recover it.
2. ``download(username, filename, ...)`` decodes the URL, asks the
active torrent adapter (qBittorrent, Transmission, or Deluge per
user's settings) to add it, spawns a background thread that
polls the adapter for completion.
3. On completion the thread walks the adapter-reported save path
via ``archive_pipeline.collect_audio_after_extraction`` and
exposes the full audio-file list. Post-processing can then pick
the requested track from a completed release instead of importing
the first file blindly.
**Album-bundle flow** (album-context batch downloads — wired in
``core/downloads/master.py``) —
4. ``download_album_to_staging(album, artist, staging_dir)`` does
ONE Prowlarr search for the whole release, picks the best
torrent (prefers FLAC, decent seeders, reasonable size),
downloads it, extracts archives if needed, copies every audio
file into the staging directory. The existing per-track
``try_staging_match`` flow then finds + imports each track by
fuzzy title match against the staged files. Per-track Prowlarr
queries never fire — track titles like "Luther (with SZA)"
would match album torrents like "GNX (2024) [FLAC]" at near-
zero confidence and break the per-track dispatch.
Limitations:
- ``save_path`` is the torrent client's view of the disk. If
SoulSync runs on a different host than qBit / Trans / Deluge,
the post-processing pipeline can't see those files. The plugin
works fine for the all-on-one-box case (most users); remote
setups will need a future sync step (rclone / SMB / Docker
bind mount).
- Track-level metadata isn't available until after download.
Search results carry only the release title + indexer metadata;
individual track names are populated when the matching pipeline
walks the extracted audio files.
"""
from __future__ import annotations
import asyncio
import re
import threading
import time
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from config.settings import config_manager
from core.archive_pipeline import collect_audio_after_extraction
from core.download_plugins.album_bundle import (
TransientMissCounter,
copy_audio_files_atomically,
get_poll_interval,
get_poll_timeout,
pick_best_album_release,
poll_album_download,
resolve_reported_save_path,
)
from core.download_plugins.base import DownloadSourcePlugin
from core.download_plugins.torrent_stall import (
StallTracker,
get_stall_action,
get_stall_timeout,
)
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.prowlarr_client import (
DEFAULT_MUSIC_CATEGORIES,
ProwlarrClient,
ProwlarrSearchResult,
)
from core.torrent_clients import get_active_adapter as get_active_torrent_adapter
from utils.async_helpers import run_async
from utils.logging_config import get_logger
logger = get_logger("download_plugins.torrent")
# Separator used to encode the download URL inside the filename
# field. Same convention Lidarr / YouTube use for embedding their
# own opaque identifiers — ``<download_url>||<display>``.
_FILENAME_SEP = '||'
# Adapter states that count as the download being on-disk and
# safe to walk. ``seeding`` and ``completed`` both mean the
# bits are there; the user can pause seeding manually if they
# don't want to keep sharing.
_COMPLETE_STATES = frozenset(['seeding', 'completed'])
# Poll cadence / timeout — both pull from config via the shared
# album_bundle helpers so users can extend the deadline for slow
# trackers without editing source. Kept as module aliases so the
# per-track flow at the bottom of this file can still import them
# under the legacy names without re-reading config every loop.
_POLL_TIMEOUT_SECONDS = get_poll_timeout()
_POLL_INTERVAL_SECONDS = get_poll_interval()
class TorrentDownloadPlugin(DownloadSourcePlugin):
"""Torrent download source backed by Prowlarr + an active
torrent client adapter."""
def __init__(self) -> None:
self._prowlarr = ProwlarrClient()
# Track every download we've kicked off. Keyed by our own
# uuid — NOT the adapter's hash — because the orchestrator
# owns the lifecycle and we need a stable id even before
# the adapter has assigned one.
self.active_downloads: Dict[str, Dict[str, Any]] = {}
self._lock = threading.Lock()
self.shutdown_check = None
def set_shutdown_check(self, check_callable):
self.shutdown_check = check_callable
def reload_settings(self) -> None:
self._prowlarr.reload_settings()
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def is_configured(self) -> bool:
if not self._prowlarr.is_configured():
return False
adapter = get_active_torrent_adapter()
return bool(adapter and adapter.is_configured())
async def check_connection(self) -> bool:
if not self._prowlarr.is_configured():
return False
adapter = get_active_torrent_adapter()
if not adapter or not adapter.is_configured():
return False
# Probe both sides. A torrent download is useless if either
# the indexer or the downloader is unreachable.
prowlarr_ok = await self._prowlarr.check_connection()
if not prowlarr_ok:
return False
return await adapter.check_connection()
# ------------------------------------------------------------------
# Search
# ------------------------------------------------------------------
async def search(
self,
query: str,
timeout: Optional[int] = None,
progress_callback=None,
) -> Tuple[List[TrackResult], List[AlbumResult]]:
if not self._prowlarr.is_configured():
return ([], [])
try:
indexer_ids = _parse_indexer_id_filter()
results = await self._prowlarr.search(
query,
categories=DEFAULT_MUSIC_CATEGORIES,
indexer_ids=indexer_ids,
)
except Exception as e:
logger.error("Torrent plugin search failed: %s", e)
return ([], [])
return self._project_results(results)
def _project_results(
self, results: List[ProwlarrSearchResult]
) -> Tuple[List[TrackResult], List[AlbumResult]]:
"""Turn Prowlarr releases into TrackResult / AlbumResult
shaped objects. One TrackResult + one AlbumResult per
release — Prowlarr search hits are at the release level,
not the track level, so we can't synthesise track listings
without downloading the actual torrent."""
tracks: List[TrackResult] = []
albums: List[AlbumResult] = []
for result in results:
if result.protocol != 'torrent':
continue
download_url = result.magnet_uri or result.download_url
if not download_url:
continue
filename = f"{download_url}{_FILENAME_SEP}{result.title}"
quality = _guess_quality_from_title(result.title)
parsed_artist, parsed_title = _parse_release_title(result.title)
tr = TrackResult(
username='torrent',
filename=filename,
size=result.size,
bitrate=None,
duration=None,
quality=quality,
# Torrent results don't have per-uploader slot / queue
# data the way Soulseek does. Fill with neutral values
# so the quality_score doesn't punish them artificially.
free_upload_slots=max(1, result.seeders or 0),
upload_speed=0,
queue_length=0,
# Pre-fill artist + title so TrackResult.__post_init__
# doesn't auto-parse the filename — our filename starts
# with the indexer download URL, which would otherwise
# show up as "by download?apikey=..." in the UI.
artist=parsed_artist or result.indexer_name or 'Torrent',
title=parsed_title or result.title,
album=parsed_title or None,
track_number=None,
_source_metadata={
'indexer': result.indexer_name,
'indexer_id': result.indexer_id,
'seeders': result.seeders,
'leechers': result.leechers,
'grabs': result.grabs,
'protocol': 'torrent',
},
)
tracks.append(tr)
albums.append(AlbumResult(
username='torrent',
album_path=f"torrent/{result.guid}",
album_title=parsed_title or result.title,
artist=parsed_artist or None,
track_count=1, # unknown until download finishes
total_size=result.size,
tracks=[tr],
dominant_quality=quality,
year=None,
))
return tracks, albums
# ------------------------------------------------------------------
# Download
# ------------------------------------------------------------------
async def download(
self,
username: str,
filename: str,
file_size: int = 0,
) -> Optional[str]:
if not self.is_configured():
return None
download_url, display_name = _decode_filename(filename)
if not download_url:
logger.error("Torrent download missing URL in filename: %r", filename)
return None
download_id = str(uuid.uuid4())
with self._lock:
self.active_downloads[download_id] = {
'id': download_id,
'filename': filename,
'username': 'torrent',
'display_name': display_name,
'state': 'Initializing',
'progress': 0.0,
'size': file_size,
'transferred': 0,
'speed': 0,
'file_path': None,
'audio_files': [],
'torrent_hash': None,
'error': None,
}
thread = threading.Thread(
target=self._download_thread,
args=(download_id, download_url, display_name),
daemon=True,
name=f'torrent-dl-{download_id[:8]}',
)
thread.start()
return download_id
def _download_thread(self, download_id: str, download_url: str, display_name: str) -> None:
"""Background worker: hand the URL to the active adapter,
poll until done, then walk the resulting directory."""
adapter = get_active_torrent_adapter()
if adapter is None or not adapter.is_configured():
self._mark_error(download_id, "No torrent client configured")
return
try:
torrent_hash = run_async(adapter.add_torrent(download_url))
except Exception as e:
self._mark_error(download_id, f"add_torrent failed: {e}")
return
if not torrent_hash:
self._mark_error(download_id, "Torrent client refused the URL")
return
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['torrent_hash'] = torrent_hash
row['state'] = 'InProgress, Downloading'
deadline = time.monotonic() + _POLL_TIMEOUT_SECONDS
last_save_path: Optional[str] = None
# Tolerate transient None reads — covers network blips. Torrent
# adapters don't have an SAB-style queue→history transition,
# but the same tolerance keeps a one-off connection failure
# from killing an otherwise-healthy download.
misses = TransientMissCounter()
# Stalled-torrent handling (noldevin): give up early on a torrent
# making zero progress (dead magnet stuck on metadata, no seeders)
# instead of holding this worker for the full album deadline. Read
# per-download so a settings change applies to in-flight torrents.
stall = StallTracker(get_stall_timeout())
while time.monotonic() < deadline:
if self.shutdown_check and self.shutdown_check():
return
try:
status = run_async(adapter.get_status(torrent_hash))
except Exception as e:
logger.warning("Torrent poll error for %s: %s", torrent_hash, e)
status = None
if status is None:
if misses.record_miss():
self._mark_error(
download_id,
f"Torrent disappeared from client (no status after {misses.threshold} polls)",
)
return
time.sleep(_POLL_INTERVAL_SECONDS)
continue
misses.reset()
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['progress'] = status.progress * 100.0
row['transferred'] = status.downloaded
row['speed'] = status.download_speed
row['size'] = status.size or row.get('size', 0)
row['state'] = _adapter_state_to_display(status.state)
row['error'] = status.error
if status.save_path:
last_save_path = status.save_path
if status.state in _COMPLETE_STATES:
self._finalize_download(download_id, last_save_path)
return
if status.state == 'error':
# Clean the dead torrent out of the client, or it's left orphaned
# (active in qbit, untracked here) and re-grabbed as a duplicate.
self._cleanup_torrent(torrent_hash, get_stall_action())
self._mark_error(download_id, status.error or "Torrent client reported error")
return
if stall.is_stalled(status.downloaded, status.state, time.monotonic(),
size=status.size):
self._handle_stalled(download_id, torrent_hash, get_stall_action())
return
time.sleep(_POLL_INTERVAL_SECONDS)
# Deadline reached. One last status check closes the race where the
# torrent completed during the final poll interval — finalize it instead
# of deleting a just-finished download's files. Otherwise clean it out of
# the client, or it sits orphaned in qbit (e.g. a metadata-stuck magnet
# that escaped the stall timer) and gets re-grabbed as a duplicate.
try:
final = run_async(adapter.get_status(torrent_hash))
except Exception:
final = None
if final is not None and final.state in _COMPLETE_STATES:
self._finalize_download(download_id, final.save_path or last_save_path)
return
self._cleanup_torrent(torrent_hash, get_stall_action())
self._mark_error(download_id, "Torrent download timed out")
def _cleanup_torrent(self, torrent_hash: str, action: str) -> None:
"""Remove (abandon) or pause a dead/stalled/timed-out torrent in the
client so it isn't left ORPHANED — active in qbit but no longer tracked
here, which makes SoulSync re-grab the same dead torrent as a duplicate
on the next attempt (noldevin). Best-effort: a client error is logged,
not raised, so the download still fails cleanly."""
adapter = get_active_torrent_adapter()
if adapter is None or not torrent_hash:
return
try:
if action == "pause":
run_async(adapter.pause(torrent_hash))
else:
# delete_files: a stalled/failed torrent's partial data is junk
# (often just a metadata stub) — don't leave it on disk.
run_async(adapter.remove(torrent_hash, delete_files=True))
except Exception as e:
logger.warning("Torrent cleanup (%s) on %s failed: %s",
action, torrent_hash[:8] if torrent_hash else "?", e)
def _handle_stalled(self, download_id: str, torrent_hash: str, action: str) -> None:
"""A torrent made no progress past the stall timeout. Abandon it
(remove from client + delete its partial data) or pause it for the
user, then fail the download so the worker frees up."""
timeout_min = round(get_stall_timeout() / 60, 1)
self._cleanup_torrent(torrent_hash, action)
verb = "paused" if action == "pause" else "removed"
self._mark_error(
download_id,
f"Torrent stalled (no progress for {timeout_min} min) — {verb}",
)
def _finalize_download(self, download_id: str, save_path: Optional[str]) -> None:
"""Adapter said complete. Walk the directory + pick the
first audio file as the canonical ``file_path``."""
if not save_path:
self._mark_error(download_id, "Torrent completed but no save_path reported")
return
# Resolve the client-reported path to one this process can read
# (the client may report its own container's mount). See
# ``resolve_reported_save_path``.
local_path = resolve_reported_save_path(save_path)
if local_path != save_path:
logger.info("Torrent %s: resolved client path %r -> %r",
download_id[:8], save_path, local_path)
try:
audio_files = collect_audio_after_extraction(Path(local_path))
except Exception as e:
self._mark_error(download_id, f"Post-extract walk failed: {e}")
return
if not audio_files:
suffix = f" (resolved: {local_path})" if local_path != save_path else ""
self._mark_error(download_id, f"No audio files found in {save_path}{suffix}")
return
primary = audio_files[0]
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Completed, Succeeded'
row['progress'] = 100.0
row['file_path'] = str(primary)
row['audio_files'] = [str(path) for path in audio_files]
logger.info("Torrent download complete: %s -> %s (%d audio files)",
download_id[:8], primary.name, len(audio_files))
def _mark_error(self, download_id: str, message: str) -> None:
logger.error("Torrent download %s failed: %s", download_id[:8], message)
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Completed, Errored'
row['error'] = message
# ------------------------------------------------------------------
# Status / lifecycle
# ------------------------------------------------------------------
async def get_all_downloads(self) -> List[DownloadStatus]:
with self._lock:
rows = list(self.active_downloads.values())
return [_row_to_status(r) for r in rows]
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
with self._lock:
row = self.active_downloads.get(download_id)
if row is None:
return None
return _row_to_status(row)
async def cancel_download(
self,
download_id: str,
username: Optional[str] = None,
remove: bool = False,
) -> bool:
adapter = get_active_torrent_adapter()
with self._lock:
row = self.active_downloads.get(download_id)
torrent_hash = row.get('torrent_hash') if row else None
if adapter and torrent_hash:
try:
await adapter.remove(torrent_hash, delete_files=remove)
except Exception as e:
logger.warning("Torrent cancel via adapter failed: %s", e)
with self._lock:
if remove:
self.active_downloads.pop(download_id, None)
else:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Cancelled'
return True
async def clear_all_completed_downloads(self) -> bool:
with self._lock:
for did in list(self.active_downloads.keys()):
state = self.active_downloads[did].get('state', '')
if state.startswith('Completed') or state == 'Cancelled':
self.active_downloads.pop(did, None)
return True
# ------------------------------------------------------------------
# Album-bundle flow
# ------------------------------------------------------------------
def download_album_to_staging(
self,
album_name: str,
artist_name: str,
staging_dir: str,
progress_callback=None,
) -> Dict[str, Any]:
"""One-shot album download: search Prowlarr for the whole
release, pick the best torrent, fetch it, extract if needed,
copy every audio file into ``staging_dir`` so the existing
``try_staging_match`` flow can hand each track off to the
post-processing pipeline.
``progress_callback`` is called with a dict on each state
change so the batch UI can show download progress without
waiting for the whole thing.
Returns ``{'success': bool, 'files': [paths], 'error': str|None}``.
"""
result: Dict[str, Any] = {'success': False, 'files': [], 'error': None}
if not self.is_configured():
result['error'] = 'Torrent source not configured'
return result
adapter = get_active_torrent_adapter()
if adapter is None or not adapter.is_configured():
result['error'] = 'No active torrent client'
return result
def _emit(state: str, **extra) -> None:
if progress_callback:
payload = {'state': state, **extra}
try:
progress_callback(payload)
except Exception as cb_exc:
logger.debug("[Torrent album] progress callback failed: %s", cb_exc)
# Phase 1: search Prowlarr for the album.
query = f"{artist_name} {album_name}".strip()
_emit('searching', query=query)
try:
search_results = run_async(self._prowlarr.search(
query, categories=DEFAULT_MUSIC_CATEGORIES,
indexer_ids=_parse_indexer_id_filter(),
))
except Exception as e:
result['error'] = f'Prowlarr search failed: {e}'
return result
candidates = [r for r in search_results
if r.protocol == 'torrent' and (r.magnet_uri or r.download_url)]
if not candidates:
# Album isn't available on this source. Mark the failure as
# fallback-eligible so the dispatch returns to the per-track flow
# instead of hard-failing the batch — in hybrid mode that lets the
# next configured source take over. Without this flag a torrent-first
# hybrid would get stuck at "searching" forever when Prowlarr
# returns nothing, never trying the other sources.
result['error'] = f'No torrent results found for "{query}"'
result['fallback'] = True
return result
picked = pick_best_album_release(
candidates, _guess_quality_from_title, album_name=album_name,
)
if picked is None:
# No candidate matched the requested album (or none passed filtering).
# Fall back to the per-track flow rather than downloading a wrong
# album (#730) — per-track searches each track individually.
result['error'] = 'No torrent candidate matched the requested album'
result['fallback'] = True
return result
download_url = picked.magnet_uri or picked.download_url
logger.info("[Torrent album] Picked '%s' (size=%.1fMB seeders=%s indexer=%s)",
picked.title, picked.size / 1_048_576, picked.seeders, picked.indexer_name)
_emit('queued', release=picked.title, size=picked.size, seeders=picked.seeders)
# Phase 2: hand to adapter.
try:
torrent_id = run_async(adapter.add_torrent(download_url))
except Exception as e:
result['error'] = f'Torrent client refused the release: {e}'
return result
if not torrent_id:
result['error'] = 'Torrent client refused the release'
return result
# Phase 3: poll until complete. The lifted helper handles
# transient missing windows (uncommon for torrents — adapters
# don't have a queue→history transition like SAB — but the
# same path also catches network blips that would otherwise
# take down the whole download) and always emits a terminal
# 'failed' state on failure paths so the UI doesn't freeze on
# the last 'downloading' emit.
_emit('downloading', release=picked.title)
save_path = poll_album_download(
get_status=lambda: run_async(adapter.get_status(torrent_id)),
title=picked.title,
emit=_emit,
# Torrent adapters flip to 'seeding' on completion (files
# on disk, share-ratio progress) — both states count as
# terminal success.
complete_states=frozenset(['seeding', 'completed']),
# qBit / Transmission / Deluge surface a real 'error'
# state when the torrent itself errors (tracker, missing
# files, etc.). That's distinct from the unmapped-state
# default-'error' fallback the helper treats as transient.
failed_states=frozenset(['error']),
is_shutdown=self.shutdown_check,
log_prefix='[Torrent album]',
)
if save_path is None:
# poll_album_download already emitted terminal 'failed'.
result['error'] = 'Torrent download failed or timed out'
return result
# Phase 4: extract + walk + copy to staging.
_emit('staging', release=picked.title)
# Resolve the client-reported path to one this process can read.
local_path = resolve_reported_save_path(save_path)
if local_path != save_path:
logger.info("[Torrent album] Resolved client path %r -> %r", save_path, local_path)
try:
audio_files = collect_audio_after_extraction(Path(local_path))
except Exception as e:
result['error'] = f'Failed to walk audio files: {e}'
return result
if not audio_files:
suffix = f' (resolved: {local_path})' if local_path != save_path else ''
result['error'] = f'No audio files found in {save_path}{suffix}'
return result
copied = copy_audio_files_atomically(audio_files, Path(staging_dir))
if not copied:
result['error'] = 'No audio files copied to staging'
return result
logger.info("[Torrent album] Staged %d audio files for '%s'", len(copied), album_name)
_emit('staged', count=len(copied))
result['success'] = True
result['files'] = copied
return result
# ---------------------------------------------------------------------------
# Module-level helpers (pure functions — easy to unit-test)
# ---------------------------------------------------------------------------
def _decode_filename(filename: str) -> Tuple[Optional[str], str]:
"""Pull the encoded download URL out of the ``filename`` string.
Returns ``(url, display_name)``. ``url`` is None when the string
has no separator."""
if not filename or _FILENAME_SEP not in filename:
return (None, filename or '')
url, display = filename.split(_FILENAME_SEP, 1)
return (url, display)
def _parse_release_title(title: str) -> Tuple[str, str]:
"""Split a release title into ``(artist, title)`` using the
``Artist - Title`` / ``Artist - Album`` convention almost every
indexer follows. Returns ``('', title)`` when no dash is found.
Without this, ``TrackResult.__post_init__`` runs the bare
filename through ``parse_filename_metadata`` — and our filename
starts with the indexer's download URL, so the auto-parser
extracts garbage like ``download?apikey=...`` as the artist
and shows it in the search-result UI's "by" line. Pre-filling
the artist field short-circuits the auto-parse.
"""
if not title:
return ('', '')
# Strip common quality / format tags so the dash split doesn't
# eat them — "Artist - Album [FLAC] (2020)" → "Artist", "Album".
cleaned = re.sub(r'\s*[\[\(][^\]\)]*[\]\)]\s*$', '', title.strip())
# Look for the FIRST " - " (or "-" surrounded by content). Some
# release titles have multiple dashes (subtitle dashes); the
# first split is the artist/work boundary.
parts = re.split(r'\s+-\s+|\s+-(?=\S)|(?<=\S)-\s+', cleaned, maxsplit=1)
if len(parts) == 2:
artist = parts[0].strip()
rest = parts[1].strip()
# Reject obvious non-artist prefixes (URLs, hashes, single
# punctuation) so we don't propagate garbage.
if artist and not re.match(r'^https?:|^[a-f0-9]{32,}$', artist):
return (artist, rest or cleaned)
return ('', cleaned)
def _guess_quality_from_title(title: str) -> str:
"""Read the quality hint from a release title — most music
torrents put the encoding right in the name (FLAC, MP3 320,
etc.). Falls back to ``'mp3'`` so quality_score doesn't crash."""
if not title:
return 'mp3'
lower = title.lower()
if 'flac' in lower:
return 'flac'
if re.search(r'\b24[\s-]?bit\b', lower) or 'hi-?res' in lower:
return 'flac'
if 'aac' in lower:
return 'aac'
if 'ogg' in lower:
return 'ogg'
return 'mp3'
def _parse_indexer_id_filter() -> List[int]:
"""Read the comma-separated indexer-ID allowlist from config.
Empty list = search every enabled indexer."""
raw = (config_manager.get('prowlarr.indexer_ids', '') or '').strip()
if not raw:
return []
out: List[int] = []
for chunk in raw.split(','):
chunk = chunk.strip()
if not chunk:
continue
try:
out.append(int(chunk))
except ValueError:
continue
return out
def _adapter_state_to_display(state: str) -> str:
"""Translate the adapter-uniform state strings into the
``'InProgress, Downloading'`` / ``'Completed, Succeeded'``
style the existing UI expects (matches Soulseek + Lidarr)."""
mapping = {
'queued': 'Queued',
'downloading': 'InProgress, Downloading',
'stalled': 'InProgress, Stalled',
'seeding': 'Completed, Succeeded',
'completed': 'Completed, Succeeded',
'paused': 'Paused',
'error': 'Completed, Errored',
}
return mapping.get(state, state.title())
def _row_to_status(row: Dict[str, Any]) -> DownloadStatus:
return DownloadStatus(
id=row['id'],
filename=row['filename'],
username=row['username'],
state=row.get('state', 'Unknown'),
progress=float(row.get('progress', 0.0)),
size=int(row.get('size', 0)),
transferred=int(row.get('transferred', 0)),
speed=int(row.get('speed', 0)),
time_remaining=None,
file_path=row.get('file_path'),
audio_files=row.get('audio_files') or None,
)

131
core/download_plugins/torrent_stall.py
View file

@ -1,131 +0,0 @@
"""Stalled-torrent detection + policy (noldevin's request).
A torrent can sit forever making zero progress — most commonly stuck
"downloading metadata" on a magnet with no peers, but also a dead swarm
mid-download. The torrent poll loop would just burn the full 6-hour album
timeout on it. This module decides, from the live status stream, when a
torrent has been stalled too long, and what to do about it.
Design split, kept testable:
- ``StallTracker`` is the pure decision core — feed it each poll's
``(downloaded, state, now)`` and it answers "stalled too long?" using a
monotonic clock passed in (no time import, no I/O). Progress = bytes
moved since the last poll; any forward movement resets the stall clock.
Terminal/healthy-but-idle states (seeding, completed, paused) never count
as stalled — only states where the torrent is *supposed* to be working.
- ``get_stall_timeout`` / ``get_stall_action`` read the two settings.
A timeout of 0 disables stall handling entirely (back to the old behavior:
ride the full poll deadline).
"""
from __future__ import annotations
from config.settings import config_manager
# 0 = disabled. 10 minutes is long enough to ride out a slow metadata fetch
# or a brief peer drought, short enough to give up on a truly dead magnet
# instead of holding a worker for 6 hours.
DEFAULT_STALL_TIMEOUT_SECONDS = 10 * 60
# What to do when a torrent stalls past the timeout:
# 'abandon' — remove it from the client (and its partial data) + fail the
# download so the worker is freed and the next source can try.
# 'pause' — pause it in the client + fail the download, leaving the
# torrent for the user to inspect/resume manually.
_VALID_ACTIONS = ("abandon", "pause")
DEFAULT_STALL_ACTION = "abandon"
# States where the torrent is meant to be making download progress, so a
# lack of it counts toward the stall clock. Mirrors the adapter-uniform set
# in core/torrent_clients/base.py. Notably EXCLUDES seeding/completed (done)
# and paused (the user's own choice) — neither is a stall.
STALLABLE_STATES = frozenset(("queued", "downloading", "stalled", "error"))
def get_stall_timeout() -> float:
"""Seconds of zero progress before a torrent is considered stalled.
0 (or invalid/negative) disables stall handling."""
raw = config_manager.get("download_source.torrent_stall_timeout_seconds",
DEFAULT_STALL_TIMEOUT_SECONDS)
try:
value = float(raw)
if value >= 0:
return value
except (TypeError, ValueError):
pass
return DEFAULT_STALL_TIMEOUT_SECONDS
def get_stall_action() -> str:
"""What to do with a stalled torrent: 'abandon' (default) or 'pause'."""
raw = config_manager.get("download_source.torrent_stall_action",
DEFAULT_STALL_ACTION)
action = str(raw or "").strip().lower()
return action if action in _VALID_ACTIONS else DEFAULT_STALL_ACTION
class StallTracker:
"""Tracks one torrent's forward progress across polls.
Pure + clock-injected so it tests without sleeping. ``timeout`` <= 0
disables it (``is_stalled`` always returns False)."""
def __init__(self, timeout_seconds: float):
self.timeout = float(timeout_seconds or 0)
self._last_downloaded = -1 # -1 = first observation
self._had_metadata = None # None = first observation; else size>0?
self._progress_since = None # monotonic time of last forward movement
def is_stalled(self, downloaded: int, state: str, now: float,
size: int = None) -> bool:
"""Record this poll's observation; return True iff the torrent has gone
``timeout`` seconds with no real forward progress while in a working state.
``downloaded`` is cumulative payload bytes; ``state`` is the adapter-uniform
state; ``now`` is a monotonic timestamp; ``size`` is the torrent's total
size in bytes (0/None while still fetching metadata).
Metadata-phase fix (#852-adjacent torrent report): a magnet stuck
"downloading metadata" reports ``size==0`` and a ``downloaded`` byte
counter that still ticks up from DHT/peer-protocol overhead even though it
makes no actual progress. Treating those bumps as progress reset the stall
clock forever, so a dead magnet never timed out. Now the byte counter only
counts once metadata is in (``size>0``); during the metadata phase the only
thing that counts as progress is *obtaining* the metadata, so a torrent
that can't even do that within the timeout is correctly flagged stalled.
"""
if self.timeout <= 0:
return False
downloaded = int(downloaded or 0)
# size is None when the caller doesn't track it (assume metadata present —
# the old byte-progress behavior); an explicit size==0 is the metadata
# phase (metaDL), where the byte counter is unreliable noise.
has_metadata = size is None or int(size) > 0
# Real forward progress: first sighting, metadata just arrived, or (only
# once we have metadata) more payload bytes. Byte bumps during the
# metadata phase are protocol noise and do NOT count.
progressed = (
self._had_metadata is None # first poll
or (has_metadata and not self._had_metadata) # got metadata
or (has_metadata and downloaded > self._last_downloaded) # more payload
)
self._had_metadata = has_metadata
self._last_downloaded = downloaded
if progressed:
self._progress_since = now
return False
# Not in a working state → not a stall (seeding/paused/completed).
if state not in STALLABLE_STATES:
self._progress_since = now # don't accrue stall time while idle-by-design
return False
if self._progress_since is None:
self._progress_since = now
return False
return (now - self._progress_since) >= self.timeout

47
core/download_plugins/types.py
View file

@ -13,11 +13,10 @@ import from a neutral package per Cin's contract-first standard.
from __future__ import annotations
from dataclasses import dataclass, field
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
from core.imports.filename import parse_filename_metadata
from core.quality.model import AudioQuality
@dataclass
@ -33,36 +32,6 @@ class SearchResult:
upload_speed: int
queue_length: int
result_type: str = "track" # "track" or "album"
# Rich quality metadata — populated by sources that provide it.
# None means "unknown", not "absent".
sample_rate: Optional[int] = None # Hz (e.g. 44100, 96000, 192000)
bit_depth: Optional[int] = None # bits per sample (16, 24)
@property
def audio_quality(self) -> AudioQuality:
"""Unified quality descriptor derived from this result's fields."""
return AudioQuality(
format=self.quality.lower() if self.quality else 'unknown',
bitrate=self.bitrate,
sample_rate=self.sample_rate,
bit_depth=self.bit_depth,
)
def set_quality(self, aq: AudioQuality) -> None:
"""Merge a mapped :class:`AudioQuality` onto this result's fields.
Used by streaming sources to stamp their claimed tier (Tidal/HiFi
tier strings, Qobuz API values, …) so ``audio_quality`` ranks
correctly. Mapper-provided fields win; a ``None`` from the mapper
leaves any already-reported value (e.g. a probed bitrate) intact.
"""
self.quality = aq.format
if aq.bitrate is not None:
self.bitrate = aq.bitrate
if aq.sample_rate is not None:
self.sample_rate = aq.sample_rate
if aq.bit_depth is not None:
self.bit_depth = aq.bit_depth
@property
def quality_score(self) -> float:
@ -158,19 +127,6 @@ class AlbumResult:
queue_length: int = 0
result_type: str = "album"
@property
def audio_quality(self) -> AudioQuality:
"""Unified quality descriptor derived from dominant track quality."""
sample_rates = [t.sample_rate for t in self.tracks if t.sample_rate]
bit_depths = [t.bit_depth for t in self.tracks if t.bit_depth]
bitrates = [t.bitrate for t in self.tracks if t.bitrate]
return AudioQuality(
format=self.dominant_quality.lower() if self.dominant_quality else 'unknown',
bitrate=max(bitrates) if bitrates else None,
sample_rate=max(sample_rates) if sample_rates else None,
bit_depth=max(bit_depths) if bit_depths else None,
)
@property
def quality_score(self) -> float:
"""Calculate album quality score based on dominant quality and track count"""
@ -241,4 +197,3 @@ class DownloadStatus:
speed: int
time_remaining: Optional[int] = None
file_path: Optional[str] = None
audio_files: Optional[List[str]] = None

543
core/download_plugins/usenet.py
View file

@ -1,543 +0,0 @@
"""UsenetDownloadPlugin — composes Prowlarr search + usenet client
adapter + archive_pipeline into a uniform download source.
Mirrors ``TorrentDownloadPlugin`` in shape and lifecycle (see that
module's docstring for the full pipeline rationale). Differences:
- Search filters Prowlarr results to ``protocol='usenet'``.
- ``add_nzb`` replaces ``add_torrent``; for NZBs we usually have
a direct HTTP URL the indexer exposes via Prowlarr.
- Usenet clients (SABnzbd, NZBGet) typically auto-extract during
post-processing, so ``archive_pipeline.collect_audio_after_extraction``
usually has nothing to extract and just walks loose files.
"""
from __future__ import annotations
import threading
import time
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from core.archive_pipeline import collect_audio_after_extraction
from core.download_plugins.album_bundle import (
TransientMissCounter,
copy_audio_files_atomically,
get_completed_no_path_window_seconds,
pick_best_album_release,
poll_album_download,
resolve_reported_save_path,
)
from core.download_plugins.base import DownloadSourcePlugin
from core.download_plugins.torrent import (
_adapter_state_to_display,
_decode_filename,
_guess_quality_from_title,
_parse_indexer_id_filter,
_parse_release_title,
_row_to_status,
_COMPLETE_STATES,
_FILENAME_SEP,
_POLL_INTERVAL_SECONDS,
_POLL_TIMEOUT_SECONDS,
)
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.prowlarr_client import (
DEFAULT_MUSIC_CATEGORIES,
ProwlarrClient,
ProwlarrSearchResult,
)
from core.usenet_clients import get_active_adapter as get_active_usenet_adapter
from utils.async_helpers import run_async
from utils.logging_config import get_logger
logger = get_logger("download_plugins.usenet")
class UsenetDownloadPlugin(DownloadSourcePlugin):
"""Usenet download source backed by Prowlarr + an active usenet
client adapter (SABnzbd or NZBGet)."""
def __init__(self) -> None:
self._prowlarr = ProwlarrClient()
self.active_downloads: Dict[str, Dict[str, Any]] = {}
self._lock = threading.Lock()
self.shutdown_check = None
def set_shutdown_check(self, check_callable):
self.shutdown_check = check_callable
def reload_settings(self) -> None:
self._prowlarr.reload_settings()
def is_configured(self) -> bool:
if not self._prowlarr.is_configured():
return False
adapter = get_active_usenet_adapter()
return bool(adapter and adapter.is_configured())
async def check_connection(self) -> bool:
if not self._prowlarr.is_configured():
return False
adapter = get_active_usenet_adapter()
if not adapter or not adapter.is_configured():
return False
if not await self._prowlarr.check_connection():
return False
return await adapter.check_connection()
# ------------------------------------------------------------------
# Search
# ------------------------------------------------------------------
async def search(
self,
query: str,
timeout: Optional[int] = None,
progress_callback=None,
) -> Tuple[List[TrackResult], List[AlbumResult]]:
if not self._prowlarr.is_configured():
return ([], [])
try:
indexer_ids = _parse_indexer_id_filter()
results = await self._prowlarr.search(
query,
categories=DEFAULT_MUSIC_CATEGORIES,
indexer_ids=indexer_ids,
)
except Exception as e:
logger.error("Usenet plugin search failed: %s", e)
return ([], [])
return self._project_results(results)
def _project_results(
self, results: List[ProwlarrSearchResult]
) -> Tuple[List[TrackResult], List[AlbumResult]]:
tracks: List[TrackResult] = []
albums: List[AlbumResult] = []
for result in results:
if result.protocol != 'usenet':
continue
if not result.download_url:
continue
filename = f"{result.download_url}{_FILENAME_SEP}{result.title}"
quality = _guess_quality_from_title(result.title)
parsed_artist, parsed_title = _parse_release_title(result.title)
tr = TrackResult(
username='usenet',
filename=filename,
size=result.size,
bitrate=None,
duration=None,
quality=quality,
# Usenet doesn't expose per-uploader concurrency the way
# Soulseek does; fill in neutral non-punishing values.
free_upload_slots=1,
upload_speed=0,
queue_length=0,
# Pre-fill artist + title so TrackResult.__post_init__
# doesn't auto-parse the filename — same URL-in-filename
# gotcha as the torrent plugin.
artist=parsed_artist or result.indexer_name or 'Usenet',
title=parsed_title or result.title,
album=parsed_title or None,
track_number=None,
_source_metadata={
'indexer': result.indexer_name,
'indexer_id': result.indexer_id,
'grabs': result.grabs,
'protocol': 'usenet',
},
)
tracks.append(tr)
albums.append(AlbumResult(
username='usenet',
album_path=f"usenet/{result.guid}",
album_title=parsed_title or result.title,
artist=parsed_artist or None,
track_count=1,
total_size=result.size,
tracks=[tr],
dominant_quality=quality,
year=None,
))
return tracks, albums
# ------------------------------------------------------------------
# Download
# ------------------------------------------------------------------
async def download(
self,
username: str,
filename: str,
file_size: int = 0,
) -> Optional[str]:
if not self.is_configured():
return None
nzb_url, display_name = _decode_filename(filename)
if not nzb_url:
logger.error("Usenet download missing URL in filename: %r", filename)
return None
download_id = str(uuid.uuid4())
with self._lock:
self.active_downloads[download_id] = {
'id': download_id,
'filename': filename,
'username': 'usenet',
'display_name': display_name,
'state': 'Initializing',
'progress': 0.0,
'size': file_size,
'transferred': 0,
'speed': 0,
'file_path': None,
'audio_files': [],
'job_id': None,
'error': None,
}
thread = threading.Thread(
target=self._download_thread,
args=(download_id, nzb_url),
daemon=True,
name=f'usenet-dl-{download_id[:8]}',
)
thread.start()
return download_id
def _download_thread(self, download_id: str, nzb_url: str) -> None:
adapter = get_active_usenet_adapter()
if adapter is None or not adapter.is_configured():
self._mark_error(download_id, "No usenet client configured")
return
try:
job_id = run_async(adapter.add_nzb(nzb_url))
except Exception as e:
self._mark_error(download_id, f"add_nzb failed: {e}")
return
if not job_id:
self._mark_error(download_id, "Usenet client refused the NZB")
return
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['job_id'] = job_id
row['state'] = 'InProgress, Downloading'
deadline = time.monotonic() + _POLL_TIMEOUT_SECONDS
last_save_path: Optional[str] = None
last_incomplete_path: Optional[str] = None
# Tolerate transient None / unmapped 'error' reads — SAB
# removes a job from the queue before adding it to history,
# and on busy servers that gap spans several polls. See
# ``album_bundle.TransientMissCounter`` for the shared rule.
misses = TransientMissCounter()
# Separate, LONGER window for "SAB says completed but hasn't
# written the final save_path yet" — the per-track sibling of the
# bundle fix (#721). Without this the thread called
# ``_finalize_download(None)`` on the first Completed-no-path read
# and errored a download that actually succeeded in SAB. Default
# ~120s, converted to a poll count against the live interval.
completed_no_path_misses = TransientMissCounter(
max(misses.threshold,
int(get_completed_no_path_window_seconds() / max(_POLL_INTERVAL_SECONDS, 0.001)) or 1)
)
while time.monotonic() < deadline:
if self.shutdown_check and self.shutdown_check():
return
try:
status = run_async(adapter.get_status(job_id))
except Exception as e:
logger.warning("Usenet poll error for %s: %s", job_id, e)
status = None
if status is None:
if misses.record_miss():
self._mark_error(
download_id,
f"Usenet job disappeared from client (no status after {misses.threshold} polls)",
)
return
time.sleep(_POLL_INTERVAL_SECONDS)
continue
if status.state != 'error':
misses.reset()
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['progress'] = status.progress * 100.0
row['transferred'] = status.downloaded
row['speed'] = status.download_speed
row['size'] = status.size or row.get('size', 0)
row['state'] = _adapter_state_to_display(status.state)
row['error'] = status.error
if status.save_path:
last_save_path = status.save_path
incomplete_path = getattr(status, 'incomplete_path', None)
if incomplete_path:
last_incomplete_path = incomplete_path
if status.state in _COMPLETE_STATES:
if last_save_path:
self._finalize_download(download_id, last_save_path)
return
# Completed but no final save_path yet — SAB flips
# History to 'Completed' before writing ``storage``.
# Wait out the (longer) completed-no-path window rather
# than erroring a download that actually succeeded.
if completed_no_path_misses.record_miss():
if last_incomplete_path:
logger.warning(
"Usenet %s: '%s' completed but no final save_path after "
"%d polls — falling back to in-progress path %r",
download_id[:8], job_id, completed_no_path_misses.misses,
last_incomplete_path,
)
self._finalize_download(download_id, last_incomplete_path)
return
self._mark_error(
download_id,
"Usenet job completed but client never reported a save_path",
)
return
logger.info(
"Usenet %s: '%s' completed on client but save_path not yet set — "
"retrying (poll %d/%d)",
download_id[:8], job_id,
completed_no_path_misses.misses, completed_no_path_misses.threshold,
)
time.sleep(_POLL_INTERVAL_SECONDS)
continue
if status.state == 'failed':
self._mark_error(download_id, status.error or "Usenet client reported failure")
return
if status.state == 'error':
logger.warning(
"Usenet poll: '%s' returned unmapped state — treating as transient",
job_id,
)
if misses.record_miss():
self._mark_error(
download_id,
"Usenet client returned unmapped state repeatedly",
)
return
time.sleep(_POLL_INTERVAL_SECONDS)
self._mark_error(download_id, "Usenet download timed out")
def _finalize_download(self, download_id: str, save_path: Optional[str]) -> None:
if not save_path:
self._mark_error(download_id, "Usenet job completed but no save_path reported")
return
# Translate the client-reported path to one THIS process can read
# (SAB reports its own container path; SoulSync may see the same
# files at a different mount). See ``resolve_reported_save_path``.
local_path = resolve_reported_save_path(save_path)
if local_path != save_path:
logger.info("Usenet %s: resolved client path %r -> %r",
download_id[:8], save_path, local_path)
try:
audio_files = collect_audio_after_extraction(Path(local_path))
except Exception as e:
self._mark_error(download_id, f"Post-extract walk failed: {e}")
return
if not audio_files:
suffix = f" (resolved: {local_path})" if local_path != save_path else ""
self._mark_error(download_id, f"No audio files found in {save_path}{suffix}")
return
primary = audio_files[0]
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Completed, Succeeded'
row['progress'] = 100.0
row['file_path'] = str(primary)
row['audio_files'] = [str(path) for path in audio_files]
logger.info("Usenet download complete: %s -> %s (%d audio files)",
download_id[:8], primary.name, len(audio_files))
def _mark_error(self, download_id: str, message: str) -> None:
logger.error("Usenet download %s failed: %s", download_id[:8], message)
with self._lock:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Completed, Errored'
row['error'] = message
# ------------------------------------------------------------------
# Status / lifecycle
# ------------------------------------------------------------------
async def get_all_downloads(self) -> List[DownloadStatus]:
with self._lock:
rows = list(self.active_downloads.values())
return [_row_to_status(r) for r in rows]
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
with self._lock:
row = self.active_downloads.get(download_id)
if row is None:
return None
return _row_to_status(row)
async def cancel_download(
self,
download_id: str,
username: Optional[str] = None,
remove: bool = False,
) -> bool:
adapter = get_active_usenet_adapter()
with self._lock:
row = self.active_downloads.get(download_id)
job_id = row.get('job_id') if row else None
if adapter and job_id:
try:
await adapter.remove(job_id, delete_files=remove)
except Exception as e:
logger.warning("Usenet cancel via adapter failed: %s", e)
with self._lock:
if remove:
self.active_downloads.pop(download_id, None)
else:
row = self.active_downloads.get(download_id)
if row is not None:
row['state'] = 'Cancelled'
return True
async def clear_all_completed_downloads(self) -> bool:
with self._lock:
for did in list(self.active_downloads.keys()):
state = self.active_downloads[did].get('state', '')
if state.startswith('Completed') or state == 'Cancelled':
self.active_downloads.pop(did, None)
return True
# ------------------------------------------------------------------
# Album-bundle flow
# ------------------------------------------------------------------
def download_album_to_staging(
self,
album_name: str,
artist_name: str,
staging_dir: str,
progress_callback=None,
) -> Dict[str, Any]:
"""Usenet sibling of ``TorrentDownloadPlugin.download_album_to_staging``.
See that method's docstring for the contract."""
result: Dict[str, Any] = {'success': False, 'files': [], 'error': None}
if not self.is_configured():
result['error'] = 'Usenet source not configured'
return result
adapter = get_active_usenet_adapter()
if adapter is None or not adapter.is_configured():
result['error'] = 'No active usenet client'
return result
def _emit(state: str, **extra) -> None:
if progress_callback:
try:
progress_callback({'state': state, **extra})
except Exception as cb_exc:
logger.debug("[Usenet album] progress callback failed: %s", cb_exc)
query = f"{artist_name} {album_name}".strip()
_emit('searching', query=query)
try:
search_results = run_async(self._prowlarr.search(
query, categories=DEFAULT_MUSIC_CATEGORIES,
indexer_ids=_parse_indexer_id_filter(),
))
except Exception as e:
result['error'] = f'Prowlarr search failed: {e}'
return result
candidates = [r for r in search_results
if r.protocol == 'usenet' and r.download_url]
if not candidates:
# Album isn't available on this source — fall back to the per-track
# flow (next configured source in hybrid mode) rather than hard-
# failing the whole batch. Mirrors the torrent plugin + soulseek's
# default fallback contract.
result['error'] = f'No usenet results found for "{query}"'
result['fallback'] = True
return result
picked = pick_best_album_release(
candidates, _guess_quality_from_title, album_name=album_name,
)
if picked is None:
# No candidate matched the requested album (or none passed filtering).
# Fall back to per-track rather than grabbing a wrong album (#730).
result['error'] = 'No NZB candidate matched the requested album'
result['fallback'] = True
return result
logger.info("[Usenet album] Picked '%s' (size=%.1fMB grabs=%s indexer=%s)",
picked.title, picked.size / 1_048_576, picked.grabs, picked.indexer_name)
_emit('queued', release=picked.title, size=picked.size, grabs=picked.grabs)
try:
job_id = run_async(adapter.add_nzb(picked.download_url))
except Exception as e:
result['error'] = f'Usenet client refused the NZB: {e}'
return result
if not job_id:
result['error'] = 'Usenet client refused the NZB'
return result
_emit('downloading', release=picked.title)
save_path = poll_album_download(
get_status=lambda: run_async(adapter.get_status(job_id)),
title=picked.title,
emit=_emit,
# Usenet completes into history as 'completed'; no 'seeding'
# equivalent. Failed is explicit on history failures.
complete_states=frozenset(['completed']),
failed_states=frozenset(['failed']),
is_shutdown=self.shutdown_check,
log_prefix='[Usenet album]',
)
if save_path is None:
# poll_album_download already emitted the terminal 'failed'
# state on every failure path (timeout / disappeared /
# explicit failure / unmapped). UI is unstuck either way.
result['error'] = 'Usenet download failed or timed out'
return result
_emit('staging', release=picked.title)
# SAB reports its own container path; SoulSync may mount the same
# files elsewhere. Resolve to a locally-readable path before walking.
local_path = resolve_reported_save_path(save_path)
if local_path != save_path:
logger.info("[Usenet album] Resolved client path %r -> %r", save_path, local_path)
try:
audio_files = collect_audio_after_extraction(Path(local_path))
except Exception as e:
result['error'] = f'Failed to walk audio files: {e}'
return result
if not audio_files:
suffix = f' (resolved: {local_path})' if local_path != save_path else ''
result['error'] = f'No audio files found in {save_path}{suffix}'
return result
copied = copy_audio_files_atomically(audio_files, Path(staging_dir))
if not copied:
result['error'] = 'No audio files copied to staging'
return result
logger.info("[Usenet album] Staged %d audio files for '%s'", len(copied), album_name)
_emit('staged', count=len(copied))
result['success'] = True
result['files'] = copied
return result

233
core/downloads/album_bundle_dispatch.py
View file

@ -1,233 +0,0 @@
"""Album-bundle dispatch for torrent / usenet single-source downloads.
Lifted from ``run_full_missing_tracks_process`` so the master
worker doesn't carry a 90-line inline branch and so the gate logic
can be unit-tested in isolation.
The gate fires only when ALL conditions hold:
- Batch is an album-context download (``is_album_download`` flag).
- Active download source is ``torrent``, ``usenet``, or ``soulseek``.
In hybrid mode the caller may pass the first configured source as a
source override; later hybrid sources stay per-track to preserve fallback.
- Both album-name and artist-name are populated in batch context.
- The resolved plugin exposes ``download_album_to_staging``.
When the gate engages it runs the plugin synchronously (the master
worker is already on a thread-pool executor) and mirrors the
plugin's lifecycle payloads into the batch state so the Downloads
page can render meaningful progress before per-track tasks exist.
Return semantics: ``True`` means the gate handled the batch — the
master worker should stop and not run per-track analysis. ``False``
means the gate didn't engage (or engaged-and-fell-back) — caller
continues the normal per-track flow.
The ``BatchStateAccess`` Protocol exists so this module doesn't
import ``download_batches`` from runtime_state directly. The
caller (master worker) injects accessors so this module stays
testable without touching live runtime state.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any, Callable, Optional, Protocol
from utils.logging_config import get_logger
# Use the project logger factory so these lines land in app.log under the
# ``soulsync.*`` namespace the file handler captures. Plain
# ``logging.getLogger(__name__)`` logs to the console only (the file
# handler is attached to the ``soulsync`` logger), which is why
# ``[Album Bundle] flow failed`` showed up in the terminal but never in
# app.log during the #721 triage.
logger = get_logger("downloads.album_bundle_dispatch")
class BatchStateAccess(Protocol):
"""Narrow shim around the batch-state dict ops the dispatch needs.
Two methods to keep the surface small:
- ``update_fields(batch_id, fields)`` — atomic merge into the
batch dict under tasks_lock.
- ``mark_failed(batch_id, error)`` — convenience for the failure
path (sets phase + error + album_bundle_state in one shot).
"""
def update_fields(self, batch_id: str, fields: dict) -> None: ...
def mark_failed(self, batch_id: str, error: str) -> None: ...
# Fields the album-bundle progress callback may carry. Anything in
# this set gets mirrored onto the batch row as ``album_bundle_<key>``
# so the Downloads page can render it without coupling to the
# specific payload shape.
_MIRRORED_KEYS = ('progress', 'release', 'speed', 'downloaded',
'size', 'seeders', 'grabs', 'count', 'failed')
def is_eligible(
*,
mode: str,
is_album: bool,
album_name: str,
artist_name: str,
) -> bool:
"""Pure predicate: does this batch even qualify for the album
flow? Separate from the resolution+run step so tests can pin
the gate logic without standing up a plugin."""
if not is_album:
return False
if (mode or '').lower() not in ('torrent', 'usenet', 'soulseek'):
return False
if not (album_name or '').strip():
return False
if not (artist_name or '').strip():
return False
return True
def try_dispatch(
*,
batch_id: str,
is_album: bool,
album_context: Optional[dict],
artist_context: Optional[dict],
config_get: Callable[..., Any],
plugin_resolver: Callable[[str], Optional[Any]],
state: BatchStateAccess,
source_override: Optional[str] = None,
plugin_kwargs: Optional[dict] = None,
) -> bool:
"""Attempt the album-bundle flow. Returns ``True`` iff the
master worker should return early (gate engaged and completed
— success OR failure). ``False`` means fall through to the
normal per-track flow.
``config_get`` is a callable shaped like ``config_manager.get``;
``plugin_resolver`` resolves a source-name string to an
initialised plugin instance (or None); ``state`` is the
BatchStateAccess shim. Injecting these keeps the module
dependency-light + unit-testable.
"""
mode = (source_override or config_get('download_source.mode', 'soulseek') or 'soulseek').lower()
album_name = (album_context or {}).get('name') or ''
artist_name = (artist_context or {}).get('name') or ''
if not is_eligible(mode=mode, is_album=is_album,
album_name=album_name, artist_name=artist_name):
return False
album_name = album_name.strip()
artist_name = artist_name.strip()
plugin = None
try:
plugin = plugin_resolver(mode)
except Exception as exc:
logger.warning("[Album Bundle] Could not resolve %s plugin: %s", mode, exc)
if plugin is None or not hasattr(plugin, 'download_album_to_staging'):
logger.warning(
"[Album Bundle] Gate matched but plugin / context unavailable "
"(mode=%s album=%r artist=%r plugin=%s) — falling back to per-track flow",
mode, album_name, artist_name,
type(plugin).__name__ if plugin else None,
)
return False
staging_root = config_get(
'download_source.album_bundle_staging_path',
'storage/album_bundle_staging',
) or 'storage/album_bundle_staging'
staging_dir = str(Path(staging_root) / _safe_batch_dirname(batch_id))
logger.info(
"[Album Bundle] Engaging %s album flow for '%s' by '%s' -> %s",
mode, album_name, artist_name, staging_dir,
)
state.update_fields(batch_id, {
'phase': 'album_downloading',
'album_bundle_state': 'searching',
'album_bundle_source': mode,
'album_bundle_staging_path': staging_dir,
'album_bundle_private_staging': True,
})
def _emit(payload):
"""Mirror plugin lifecycle into batch state for UI rendering."""
try:
fields = {'album_bundle_state': payload.get('state', '')}
for key in _MIRRORED_KEYS:
if key in payload:
fields[f'album_bundle_{key}'] = payload[key]
state.update_fields(batch_id, fields)
except Exception as exc:
logger.debug("[Album Bundle] emit failed: %s", exc)
try:
outcome = plugin.download_album_to_staging(
album_name, artist_name, staging_dir, _emit,
**(plugin_kwargs or {}),
)
except Exception as exc:
logger.exception("[Album Bundle] %s plugin raised: %s", mode, exc)
# An OSError means an I/O step failed after the source already had the
# album — most importantly the staging dir not being writable (#760),
# but also any transient filesystem error. Treat it as fallback-eligible
# so we return to the per-track flow instead of hard-failing the whole
# batch (the #715 symptom: files download, then the batch fails).
# Programming errors (TypeError, KeyError, …) are NOT OSError and stay
# terminal, so genuine bugs still fail loudly. (requests' network
# exceptions also subclass OSError, but plugins normally catch those
# internally and return an outcome rather than raising; if one does
# surface here, falling back to per-track is still the safe choice.)
is_io_failure = isinstance(exc, OSError)
outcome = {
'success': False,
'error': f'Plugin error: {exc}',
'fallback': is_io_failure,
}
if not outcome.get('success'):
err = outcome.get('error', 'Album bundle download failed')
if outcome.get('fallback'):
logger.warning(
"[Album Bundle] %s flow could not commit for '%s': %s — falling back to per-track flow",
mode, album_name, err,
)
state.update_fields(batch_id, {
'phase': 'analysis',
'album_bundle_state': 'fallback',
'album_bundle_error': err,
'album_bundle_private_staging': False,
'album_bundle_staging_path': None,
})
return False
logger.error("[Album Bundle] %s flow failed for '%s': %s",
mode, album_name, err)
state.mark_failed(batch_id, err)
return True
logger.info(
"[Album Bundle] %s staged %d files for '%s' — handing off to per-track staging matcher",
mode, len(outcome.get('files', [])), album_name,
)
state.update_fields(batch_id, {
'phase': 'analysis',
'album_bundle_state': 'staged',
'album_bundle_partial': bool(outcome.get('partial')),
'album_bundle_expected_count': outcome.get('expected_count'),
'album_bundle_completed_count': outcome.get('completed_count', len(outcome.get('files', []))),
})
# Engaged-and-succeeded: we DON'T early-return because the
# per-track flow needs to run to create + complete the per-track
# task rows. Those tasks will hit try_staging_match and pull the
# files we just staged.
return False
def _safe_batch_dirname(batch_id: str) -> str:
safe = ''.join(ch if ch.isalnum() or ch in ('-', '_') else '_' for ch in str(batch_id or 'batch'))
return safe or 'batch'

22
core/downloads/cancel.py
View file

@ -45,9 +45,6 @@ _TERMINAL_STATUSES = {
def cancel_single_download(download_orchestrator, run_async: Callable,
download_id: str, username: str) -> bool:
"""Cancel one specific slskd download (with `remove=True`)."""
logger.info(
f"[CancelTrigger:api.manual_cancel_single] download_id={download_id} username={username}"
)
return run_async(download_orchestrator.cancel_download(download_id, username, remove=True))
@ -83,26 +80,9 @@ def clear_completed_local() -> int:
"""
cleared = 0
with tasks_lock:
# Protect tasks belonging to a still-active batch. A batch is "active"
# while any of its queued tasks is non-terminal (still searching /
# downloading / queued / post-processing). Pruning a batch's completed
# or failed tasks mid-run would yank them out of the Downloads page —
# and failed/cancelled rows aren't recoverable from library_history —
# so the user would never see them until the batch ended. Keep the whole
# active batch intact; it gets cleaned by a later run once it finishes.
protected_task_ids: set = set()
for batch in download_batches.values():
queue = batch.get('queue', []) if isinstance(batch, dict) else []
batch_active = any(
download_tasks.get(tid, {}).get('status') not in _TERMINAL_STATUSES
for tid in queue if tid in download_tasks
)
if batch_active:
protected_task_ids.update(queue)
task_ids_to_remove = [
tid for tid, task in download_tasks.items()
if task.get('status') in _TERMINAL_STATUSES and tid not in protected_task_ids
if task.get('status') in _TERMINAL_STATUSES
]
for tid in task_ids_to_remove:
del download_tasks[tid]

203
core/downloads/candidates.py
View file

@ -36,7 +36,6 @@ import os
from dataclasses import dataclass
from typing import Any, Callable
from core.downloads.track_metadata_backfill import hydrate_download_metadata
from core.runtime_state import (
download_tasks,
matched_context_lock,
@ -47,55 +46,6 @@ from core.runtime_state import (
logger = logging.getLogger(__name__)
def _priority_sort_key(r):
"""Today's confidence-first key: never download a high-quality WRONG file."""
return (
getattr(r, 'confidence', 0) or 0,
getattr(r, 'quality_score', 0) or 0,
getattr(r, 'upload_speed', 0) or 0,
-(getattr(r, 'queue_length', 0) or 0),
getattr(r, 'free_upload_slots', 0) or 0,
getattr(r, 'size', 0) or 0,
)
def _quality_first_sort_key(r, targets):
"""Best-quality key: the user's profile quality rank dominates; all the
priority-mode signals (confidence, speed, …) become tiebreakers.
Every candidate reaching this point already passed match filtering, so it
is "correct enough" — ordering by quality among correct candidates is safe.
Candidates with no usable quality info, or that match no target, sort last
(never dropped). Lower target index = better target, so it's negated to fit
the descending (reverse=True) sort.
"""
from core.quality.model import rank_candidate
aq = getattr(r, 'audio_quality', None)
if aq is None or not targets:
target_idx, tier = (len(targets) if targets else 0), 0.0
else:
try:
target_idx, tier = rank_candidate(aq, targets)
except Exception:
target_idx, tier = len(targets), 0.0
return (-target_idx, tier) + _priority_sort_key(r)
def order_candidates(candidates, *, quality_first=False, targets=None):
"""Return *candidates* ordered best-first for the download walk.
``quality_first=False`` (priority mode) → confidence-first, byte-for-byte
today's behaviour. ``quality_first=True`` (best-quality mode) → the user's
profile quality rank dominates, confidence/peer signals break ties.
"""
if quality_first:
key = lambda r: _quality_first_sort_key(r, targets or [])
else:
key = _priority_sort_key
return sorted(candidates, key=key, reverse=True)
@dataclass
class CandidatesDeps:
"""Bundle of cross-cutting deps the candidate-fallback logic needs."""
@ -108,25 +58,25 @@ class CandidatesDeps:
on_download_completed: Callable
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
deps: CandidatesDeps = None, *,
quality_first=False, quality_targets=None):
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None, deps: CandidatesDeps = None):
"""
Attempts to download with fallback candidate logic (matches GUI's retry_parallel_download_with_fallback).
Returns True if successful, False if all candidates fail.
``quality_first`` (best-quality search mode) orders the walk by the user's
profile quality rank instead of confidence-first; ``quality_targets`` is the
profile target list used for that ranking. Defaults preserve priority-mode
behaviour exactly.
"""
# Sort candidates. Priority mode: confidence-first, then peer quality —
# upstream Soulseek validation already considers peer speed/slots/queue when
# scores are close; preserve that signal instead of flattening ties back to
# arbitrary slskd response order. Best-quality mode: profile quality rank
# dominates (all candidates here already passed match filtering).
candidates = order_candidates(
candidates, quality_first=quality_first, targets=quality_targets,
# Sort candidates by match confidence first, then peer quality. Upstream
# Soulseek validation already considers peer speed/slots/queue when scores
# are close; preserve that signal here instead of flattening ties back to
# arbitrary slskd response order.
candidates.sort(
key=lambda r: (
getattr(r, 'confidence', 0) or 0,
getattr(r, 'quality_score', 0) or 0,
getattr(r, 'upload_speed', 0) or 0,
-(getattr(r, 'queue_length', 0) or 0),
getattr(r, 'free_upload_slots', 0) or 0,
getattr(r, 'size', 0) or 0,
),
reverse=True,
)
with tasks_lock:
@ -134,11 +84,6 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
if not task:
return False
used_sources = task.get('used_sources', set())
# User-initiated manual picks (candidates modal) bypass quarantine
# gates downstream. The user already accepted the risk by choosing
# the file; we trust their selection over AcoustID disagreement so
# repeated manual picks don't loop back into quarantine.
user_manual_pick = bool(task.get('_user_manual_pick', False))
# Try each candidate until one succeeds (like GUI's fallback logic)
for candidate_index, candidate in enumerate(candidates):
@ -254,21 +199,6 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
'artists': _fallback_album_artists
}
# #915: parity with Reorganize / manual Enrich. If the album context is lean
# (no release_date) and the user's PRIMARY metadata source isn't Spotify, hydrate
# it from that source — the same place a reorganize reads — so the download's
# $year folder, release_date and album_type match instead of dropping the year /
# defaulting to YYYY-01-01 and forcing a manual reorganize afterwards.
try:
from core.downloads.track_metadata_backfill import backfill_album_context_from_source
from core.metadata import registry as _meta_registry
from core.metadata.album_tracks import get_album_for_source as _get_album_for_source
backfill_album_context_from_source(
spotify_album_context, _meta_registry.get_primary_source(), _get_album_for_source,
)
except Exception as _bf_err: # noqa: BLE001 — never let backfill break a download
logger.debug("[Context] primary-source album backfill skipped: %s", _bf_err)
download_payload = candidate.__dict__
username = download_payload.get('username')
@ -312,30 +242,55 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
enhanced_payload['artists'] = [{'name': artist} for artist in track.artists] if track.artists else []
logger.info(f"[Context] Using clean Spotify metadata - Album: '{track.album}', Title: '{track.name}'")
# Resolve track_number / disc_number and hydrate
# lean album context. Extracted to
# track_metadata_backfill.hydrate_download_metadata
# — see that module for the precedence chain.
# Why the extract: the inline pre-fix coupled
# album-backfill to the "track_number missing"
# branch. When wishlist payloads carried a poisoned
# default-1 track_number (older routes.py used
# ``.get('track_number', 1)``) the API call short-
# circuited and the lean album_context (no
# release_date / total_tracks for Deezer-sourced
# discovery matches) survived untouched, producing
# folders without a year subfolder.
resolved = hydrate_download_metadata(
track, track_info, spotify_album_context, deps.spotify_client,
)
if resolved.track_number is not None:
enhanced_payload['track_number'] = resolved.track_number
enhanced_payload['disc_number'] = resolved.disc_number
logger.info(
f"[Context] Added track_number from {resolved.source}: "
f"{resolved.track_number}, disc_number: {resolved.disc_number}"
)
else:
# Get track_number and disc_number — prefer track data we already have,
# fall back to detailed API call only if needed
got_track_number = False
# 1. Try track_info (from frontend, has album track data)
tn = track_info.get('track_number', 0) if isinstance(track_info, dict) else 0
dn = (track_info.get('disc_number') or 1) if isinstance(track_info, dict) else 1
if tn and tn > 0:
enhanced_payload['track_number'] = tn
enhanced_payload['disc_number'] = dn
got_track_number = True
logger.info(f"[Context] Added track_number from track_info: {tn}, disc_number: {dn}")
# 2. Try the track object itself (from album tracks response)
if not got_track_number and hasattr(track, 'track_number') and track.track_number:
enhanced_payload['track_number'] = track.track_number
enhanced_payload['disc_number'] = getattr(track, 'disc_number', 1) or 1
got_track_number = True
logger.info(f"[Context] Added track_number from track object: {track.track_number}, disc_number: {enhanced_payload['disc_number']}")
# 3. Last resort — fetch from metadata source API
if not got_track_number and hasattr(track, 'id') and track.id:
try:
detailed_track = deps.spotify_client.get_track_details(track.id)
if detailed_track and detailed_track.get('track_number'):
enhanced_payload['track_number'] = detailed_track['track_number']
enhanced_payload['disc_number'] = detailed_track.get('disc_number') or 1
got_track_number = True
logger.info(f"[Context] Added track_number from API: {detailed_track['track_number']}, disc_number: {enhanced_payload['disc_number']}")
# Backfill album metadata from detailed track when context
# has incomplete data (missing release_date, total_tracks, etc.)
if isinstance(detailed_track.get('album'), dict):
dt_album = detailed_track['album']
if not spotify_album_context.get('release_date') and dt_album.get('release_date'):
spotify_album_context['release_date'] = dt_album['release_date']
logger.info(f"[Context] Backfilled release_date from API: {dt_album['release_date']}")
if not spotify_album_context.get('album_type') and dt_album.get('album_type'):
spotify_album_context['album_type'] = dt_album['album_type']
if not spotify_album_context.get('total_tracks') and dt_album.get('total_tracks'):
spotify_album_context['total_tracks'] = dt_album['total_tracks']
if not spotify_album_context.get('id') and dt_album.get('id'):
spotify_album_context['id'] = dt_album['id']
if not spotify_album_context.get('image_url') and dt_album.get('images'):
spotify_album_context['image_url'] = dt_album['images'][0].get('url', '')
except Exception as e:
logger.error(f"[Context] API track details failed: {e}")
if not got_track_number:
enhanced_payload.setdefault('track_number', 0)
enhanced_payload.setdefault('disc_number', 1)
logger.warning("[Context] No track_number found from any source")
@ -376,33 +331,9 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
"track_info": track_info, # Add track_info for playlist folder mode
"_download_username": username, # Source username for AcoustID skip logic
}
if user_manual_pick:
# The user explicitly picked this candidate via the
# candidates modal — trust their metadata judgement
# over AcoustID disagreement so manual picks don't
# loop back into quarantine. Integrity + bit-depth
# gates still run because those check the new file's
# actual condition, not its identity.
matched_downloads_context[context_key]['_skip_quarantine_check'] = 'acoustid'
matched_downloads_context[context_key]['_user_manual_pick'] = True
logger.info(
"[Context] User manual pick — bypassing AcoustID for "
"task=%s username=%s filename=%s",
task_id, username, os.path.basename(filename),
)
elif track_info and track_info.get('_skip_acoustid'):
# Issue #797 — the album-download request had the
# per-request "Skip AcoustID verification" toggle on.
# Bypass only the AcoustID gate (same as a manual
# pick); integrity + bit-depth still run.
matched_downloads_context[context_key]['_skip_quarantine_check'] = 'acoustid'
logger.info(
"[Context] Skip-AcoustID toggle — bypassing AcoustID for "
"task=%s filename=%s",
task_id, os.path.basename(filename),
)
logger.info(f"[Context] Set is_album_download: {is_album_context} (has clean data: {has_clean_spotify_data})")
logger.debug(f"[Debug] Context creation - track_info: {track_info is not None}, playlist_folder_mode: {track_info.get('_playlist_folder_mode', False) if track_info else False}")
# Update task with successful download info
with tasks_lock:
@ -412,10 +343,6 @@ def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
logger.warning(f"[Modal Worker] Task {task_id} cancelled after download {download_id} started - attempting to cancel download")
# Try to cancel the download immediately
try:
logger.info(
f"[CancelTrigger:candidates.worker_cancelled_during_download] "
f"download_id={download_id} username={username} task_id={task_id}"
)
deps.run_async(deps.download_orchestrator.cancel_download(download_id, username, remove=True))
logger.warning(f"Successfully cancelled active download {download_id}")
except Exception as cancel_error:

295
core/downloads/file_finder.py
View file

@ -1,295 +0,0 @@
"""Robust completed-download file finder.
Walks a download directory (and optional transfer directory) to find
the local file matching an API-reported remote filename. Handles:
- Arbitrary subdirectory layouts (slskd flat / username-prefixed /
remote-tree-preserved). Pre-extract callers in the Soulseek
album-bundle path tried three hard-coded candidate paths and
silently failed on layouts that didn't match — see issue #715
(Billy Ocean album task fails after slskd finishes downloading
release). The per-track flow already used a recursive walk and
worked; the bundle path didn't, so users on common slskd configs
with username-prefixed downloads saw bundles time out 22 minutes
after slskd reported every transfer Completed.
- slskd dedup suffix ``_<10-or-more-digit-timestamp>`` appended when
a file with the same basename already exists.
- YouTube / Tidal encoded filename format ``id||title`` — the
``||`` half is the human title and used for matching.
- Multiple files sharing a basename — disambiguates by counting
how many remote-path directory components appear in the local
path.
This was lifted verbatim from ``web_server._find_completed_file_robust``
so both the per-track download poll AND the Soulseek album-bundle
poll go through one finder. Pre-extract the bundle path probed
three hard-coded candidates only, which is why bundle downloads
on slskd setups with username-prefixed download dirs silently
timed out (#715).
"""
from __future__ import annotations
import logging
import os
import re
from difflib import SequenceMatcher
from typing import Optional, Tuple
from unidecode import unidecode
logger = logging.getLogger(__name__)
AUDIO_EXTENSIONS = frozenset({
'.mp3', '.flac', '.m4a', '.aac', '.ogg', '.opus', '.wav', '.wma',
'.alac', '.aiff', '.aif', '.dsf', '.dff', '.ape',
})
# slskd appends a 10+ digit timestamp suffix when a file with the
# same basename already exists. Match-strip those so we still
# resolve the transfer.
_SLSKD_DEDUP_SUFFIX = re.compile(r'_\d{10,}$')
# AcoustID-quarantined files live under this dirname and must be
# skipped — they are known-wrong matches the verifier rejected.
_QUARANTINE_DIRNAME = 'ss_quarantine'
# Confidence floor for accepting a fuzzy basename match. Anything
# below this is treated as "no match" so we don't drag in unrelated
# files.
_FUZZY_THRESHOLD = 0.85
def _is_audio_candidate(path: str) -> bool:
return os.path.splitext(str(path or ''))[1].lower() in AUDIO_EXTENSIONS
def _normalize_for_finding(text: str) -> str:
"""Match-engine-style normalisation for fuzzy filename comparison.
Lowercases, transliterates unicode, drops bracketed content
("(Remastered 2016)", "[FLAC]"), strips punctuation, collapses
whitespace. Mirrors ``matching_engine.py``'s text normaliser so
finder + matcher agree on equivalence.
"""
if not text:
return ""
text = unidecode(text).lower()
text = re.sub(r'[._/]', ' ', text)
# Strip ONLY balanced bracket pairs (tags like "[FLAC]", "(Remastered 2016)").
# The old combined pattern r'[\[\(].*?[\]\)]' allowed MISMATCHED delimiters, so a
# lone unbalanced '[' — slskd reports "[34 - You & Me (Flume Remix)" but saves the
# file as "34 - You & Me (Flume Remix)" — matched from that '[' all the way to the
# next ')', eating the entire title and collapsing the search target to "flac". The
# file then scored 0.40 against the real on-disk name and was reported "not found"
# despite sitting right there. Per-delimiter pairs can't over-consume; a stray
# unbalanced bracket simply survives to the alphanumeric strip below.
text = re.sub(r'\[[^\]]*\]', '', text)
text = re.sub(r'\([^)]*\)', '', text)
text = re.sub(r'[^a-z0-9\s-]', '', text)
return ' '.join(text.split()).strip()
def _extract_basename(api_filename: str) -> str:
"""Cross-platform rightmost-separator split for a real remote PATH.
A YouTube/Tidal/Qobuz ``id||title`` encoded filename is handled by
returning the title VERBATIM: the title is not a filesystem path, so a '/'
in it (e.g. the Sawano track ``YouSeeBIGGIRL/T:T``) is part of the name and
must NOT be split on (issue #835)."""
if not api_filename:
return ""
if '||' in api_filename:
_id, title = api_filename.split('||', 1)
return title
last_slash = max(api_filename.rfind('/'), api_filename.rfind('\\'))
return api_filename[last_slash + 1:] if last_slash != -1 else api_filename
def _api_dir_parts(api_filename: str) -> list[str]:
"""Lowercased remote-path directory components, sans the
filename itself. Used to disambiguate when several local files
share the same basename — the one whose path mirrors the
remote folder structure wins."""
if not api_filename:
return []
normalized = api_filename.replace('\\', '/')
return [p.lower() for p in normalized.split('/')[:-1] if p]
def _path_matches_api_dirs(file_path: str, api_dirs: list[str]) -> bool:
"""``True`` iff every remote directory component appears as a
path part of the local file. Cheap "is this file on a sibling
tree" check."""
if not api_dirs:
return False
path_parts = set(p.lower() for p in file_path.replace('\\', '/').split('/'))
return all(d in path_parts for d in api_dirs)
def _search_in_directory(
search_dir: str,
location_name: str,
target_basename: str,
normalized_target: str,
api_dirs: list[str],
) -> Tuple[Optional[str], float]:
"""Walk ``search_dir`` once, return (best_match, similarity).
Priority order, highest first:
1. Exact basename match with directory-structure confirmation
2. Exact basename match without disambiguation (when no api_dirs)
3. slskd-dedup-suffix basename match (same two tiers as exact)
4. Best fuzzy basename match above ``_FUZZY_THRESHOLD``
"""
best_fuzzy_path: Optional[str] = None
highest_fuzzy_similarity = 0.0
exact_matches: list[str] = []
for root, dirs, files in os.walk(search_dir):
# Strip quarantine subdir from the walk in place — these
# files are known-bad and matching them would re-poison the
# post-process pipeline.
dirs[:] = [d for d in dirs if d != _QUARANTINE_DIRNAME]
for filename in files:
file_path = os.path.join(root, filename)
if not _is_audio_candidate(file_path):
continue
# Tier 1 + 2: exact basename match.
if filename == target_basename:
if api_dirs and _path_matches_api_dirs(file_path, api_dirs):
logger.info(
"Found path-confirmed match in %s: %s",
location_name, file_path,
)
return file_path, 1.0
if not api_dirs:
logger.info(
"Found exact match in %s: %s",
location_name, file_path,
)
return file_path, 1.0
exact_matches.append(file_path)
continue
# Tier 3: slskd dedup suffix.
stem, ext = os.path.splitext(filename)
stripped_stem = _SLSKD_DEDUP_SUFFIX.sub('', stem)
if stripped_stem != stem and stripped_stem + ext == target_basename:
if api_dirs and _path_matches_api_dirs(file_path, api_dirs):
logger.info(
"Found path-confirmed dedup match in %s: %s",
location_name, file_path,
)
return file_path, 1.0
if not api_dirs:
logger.info(
"Found dedup-suffix match in %s: %s",
location_name, file_path,
)
return file_path, 1.0
exact_matches.append(file_path)
continue
# Tier 4: fuzzy basename match. Cheaper than path-walking
# the whole tree a second time, so always compute and
# keep the best one as a fallback.
normalized_file = _normalize_for_finding(filename)
similarity = SequenceMatcher(
None, normalized_target, normalized_file,
).ratio()
if similarity > highest_fuzzy_similarity:
highest_fuzzy_similarity = similarity
best_fuzzy_path = file_path
if exact_matches:
if len(exact_matches) == 1:
logger.info(
"Found exact match in %s: %s",
location_name, exact_matches[0],
)
return exact_matches[0], 1.0
# Multiple basename collisions — pick the one whose path
# carries the most of the remote directory tree (album
# folder etc.). Breaks ties deterministically.
best = exact_matches[0]
best_score = -1
for m in exact_matches:
m_parts = set(p.lower() for p in m.replace('\\', '/').split('/'))
score = sum(1 for d in api_dirs if d in m_parts)
if score > best_score:
best_score = score
best = m
logger.info(
"Found %d files named '%s' in %s, picked best path match: %s",
len(exact_matches), target_basename, location_name, best,
)
return best, 1.0
return best_fuzzy_path, highest_fuzzy_similarity
def find_completed_audio_file(
download_dir: str,
api_filename: str,
transfer_dir: Optional[str] = None,
) -> Tuple[Optional[str], Optional[str]]:
"""Locate a completed download's local file via recursive walk.
Tries the downloads tree first; if nothing above the fuzzy
threshold lands AND a transfer_dir was passed, tries that too.
Returns ``(file_path, location)`` where ``location`` is
``'downloads'`` / ``'transfer'`` / ``None``. Both elements are
``None`` when the file isn't found anywhere — callers should
treat that as "not yet" (still mid-write) or "lost".
"""
# YouTube / Tidal / Qobuz encoded filenames carry the id ahead of ``||``.
# The title half is NOT a filesystem path: a '/' in it (e.g. the Sawano
# track ``YouSeeBIGGIRL/T:T``) is part of the title, so it must NOT be
# basename-split or read as a remote directory component — doing so
# truncated the search target to ``T:T`` and the real file was never found,
# quarantining valid downloads (issue #835). Real remote paths (Soulseek)
# still get basename + dir-component extraction.
encoded_title = None
if api_filename and '||' in api_filename:
_id, encoded_title = api_filename.split('||', 1)
if encoded_title is not None:
target_basename = encoded_title
api_dirs = []
else:
target_basename = _extract_basename(api_filename)
api_dirs = _api_dir_parts(api_filename)
normalized_target = _normalize_for_finding(target_basename)
best_dl_path, dl_sim = _search_in_directory(
download_dir, 'downloads', target_basename, normalized_target, api_dirs,
)
if dl_sim > _FUZZY_THRESHOLD:
if dl_sim < 1.0:
logger.info(
"Found fuzzy match in downloads (%.2f): %s",
dl_sim, best_dl_path,
)
return (best_dl_path, 'downloads')
if transfer_dir and os.path.exists(transfer_dir):
best_tx_path, tx_sim = _search_in_directory(
transfer_dir, 'transfer', target_basename, normalized_target, api_dirs,
)
if tx_sim > _FUZZY_THRESHOLD:
if tx_sim < 1.0:
logger.info(
"Found fuzzy match in transfer (%.2f): %s",
tx_sim, best_tx_path,
)
return (best_tx_path, 'transfer')
return (None, None)
__all__ = ['find_completed_audio_file', 'AUDIO_EXTENSIONS']

62
core/downloads/history_match.py
View file

@ -1,62 +0,0 @@
"""Match a file back to its download-history row when its path has drifted (#934).
``library_history.file_path`` is frozen at import time, but the file moves afterward
(media-server import, library reorganize) and ``tracks.file_path`` — what the AcoustID
scanner reads — no longer equals it. Matching on the exact path alone then fails twice:
the verification status never reaches the history row (verified tracks read "unverified"),
and a fresh ``acoustid_scan`` row gets inserted every run (thousands of duplicates).
This module picks the canonical history row by exact path first, then by FILENAME guarded
by a title check — so a shared filename ("01 - Intro.flac") can never heal the wrong song.
Pure (no DB) so the matching rules are unit-testable; the caller does the SQL.
"""
from __future__ import annotations
import os
from typing import Iterable, Optional, Sequence, Tuple
def _norm_title(value) -> str:
"""Alphanumeric-only lowercase form, so "Song (Remaster)" vs "song remaster"
style drift between the download tag and the media-server tag still agrees."""
return ''.join(ch for ch in str(value or '').lower() if ch.isalnum())
def like_filename_filter(basename: str) -> str:
"""A ``LIKE ... ESCAPE '\\'`` pattern that coarsely matches rows whose path ends
in ``basename``. Escapes the LIKE metacharacters (``%`` ``_`` ``\\``) — filenames
routinely contain underscores. Callers MUST still confirm with an exact basename
compare (``pick_history_row`` does), since ``'%name'`` also matches ``'xname'``."""
esc = basename.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_')
return '%' + esc
def pick_history_row(candidates: Sequence[Tuple], *, current_paths: Iterable[str],
basename: str, title: str) -> Optional[int]:
"""Return the id of the history row to update for this file, or None.
``candidates``: ``(id, file_path, title, download_source)`` rows the DB pre-filtered
(exact path or filename LIKE). A row matches when its path equals the current path OR
its filename matches AND its title agrees — the title guard prevents a shared filename
("01 - Intro.flac") from healing a different song's row. Among matches a REAL download
row is preferred over a synthetic ``acoustid_scan`` row, so the scanner heals the
genuine record and the caller can delete the synthetic duplicate. None when nothing
matches safely (caller then inserts a fresh row — the "file SoulSync never downloaded"
intent)."""
paths = {p for p in current_paths if p}
want = _norm_title(title)
matches: list = [] # (id, is_exact, is_real)
for cid, cpath, ctitle, csource in candidates:
is_real = csource != 'acoustid_scan'
if cpath and cpath in paths:
matches.append((cid, True, is_real))
elif (basename and cpath and os.path.basename(cpath) == basename
and (not want or not _norm_title(ctitle) or _norm_title(ctitle) == want)):
matches.append((cid, False, is_real))
if not matches:
return None
# Prefer a REAL download row over a synthetic acoustid_scan row; within that, prefer an
# exact-path match over a filename match. Stable, so ties keep DB order (first/oldest id).
matches.sort(key=lambda m: (m[2], m[1]), reverse=True)
return matches[0][0]

211
core/downloads/lifecycle.py
View file

@ -26,12 +26,9 @@ Lifted verbatim from web_server.py. Dependencies injected via
from __future__ import annotations
import logging
import os
import shutil
import time
import traceback
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Optional
from core.downloads.history import record_sync_history_completion
@ -45,151 +42,6 @@ from core.runtime_state import (
logger = logging.getLogger(__name__)
# A task that has been in 'post_processing' longer than this is treated as stuck.
# Post-processing (AcoustID + quality + import) is serialized, so a large batch
# legitimately backs up — keep this generous so genuinely-slow imports aren't
# cut off mid-flight (the old 5-min cutoff falsely "completed" queued tasks).
_POST_PROCESSING_STUCK_TIMEOUT = 1800 # 30 minutes
def _resolve_stuck_post_processing_status(task: dict) -> str:
"""Decide the terminal status for a task stuck in post_processing.
Only call it 'completed' if the import actually produced a file on disk
(``final_file_path`` is set at the end of successful post-processing). Without
a real file, force-completing is a lie — the task shows as a downloaded track
that isn't anywhere. Mark those 'failed' so they're retryable and honest.
"""
final_path = task.get('final_file_path')
if final_path and os.path.exists(final_path):
return 'completed'
return 'failed'
def _safe_batch_dirname(batch_id: str) -> str:
safe = ''.join(ch if ch.isalnum() or ch in ('-', '_') else '_' for ch in str(batch_id or 'batch'))
return safe or 'batch'
_ALBUM_BUNDLE_CLEANED_SOURCES = ('soulseek', 'torrent', 'usenet')
def _cleanup_private_album_bundle_staging(batch_id: str, batch: dict) -> None:
"""Best-effort cleanup for album-bundle private staging copies.
Fires when a batch reaches a terminal state. SoulSync's per-batch
copy lives at ``storage/album_bundle_staging/<batch_id>/`` —
safe to remove because by the time the batch is "complete" the
per-track workers have already claimed their files out of staging
via ``try_staging_match`` and moved them to the Transfer dir.
Pre-fix this only ran for torrent / usenet bundles because the
comment was "slskd keeps its own completed folders" — but the
Soulseek bundle path ALSO copies files into the private staging
dir (``soulseek_client.py:1599``), so slskd bundle copies were
leaking forever. Coverage extended to all three bundle-capable
sources.
"""
if not batch.get('album_bundle_private_staging'):
return
if (batch.get('album_bundle_source') or '').lower() not in _ALBUM_BUNDLE_CLEANED_SOURCES:
return
staging_path = batch.get('album_bundle_staging_path')
if not staging_path:
return
path = Path(staging_path)
expected_name = _safe_batch_dirname(batch_id)
if path.name != expected_name:
logger.warning(
"[Album Bundle] Refusing to clean private staging path with unexpected name: %s",
staging_path,
)
return
if not path.exists():
return
if not path.is_dir():
logger.warning("[Album Bundle] Refusing to clean non-directory staging path: %s", staging_path)
return
try:
shutil.rmtree(path)
logger.info("[Album Bundle] Cleaned private staging folder for batch %s: %s", batch_id, staging_path)
except Exception as exc:
logger.warning("[Album Bundle] Could not clean private staging folder %s: %s", staging_path, exc)
def sweep_orphan_album_bundle_staging(
staging_root: str,
*,
active_batch_ids: Optional[set] = None,
) -> int:
"""Remove orphan per-batch dirs from album-bundle staging.
An orphan is a ``<staging_root>/<dirname>`` subdir whose ``dirname``
matches no batch_id in the current ``download_batches`` runtime
state. Happens when:
- The app crashed mid-bundle (cleanup never fired).
- A batch errored on a non-completion code path.
- A pre-extension Soulseek bundle (where cleanup was gated to
torrent/usenet) left a copy behind.
Intended to run ONCE at server startup, before any new batch can
register an active staging dir. That guarantees ``active_batch_ids``
is genuinely empty / pre-existing; we don't race a starting batch.
Returns the count of dirs removed. Safe-by-design:
- Only touches subdirs of the configured staging root.
- Each candidate goes through the same ``_safe_batch_dirname``
name-guard as the per-batch cleanup, so escape-via-symlink
isn't possible.
- Refuses to act on non-directories.
- ``shutil.rmtree`` errors are logged, not raised — sweep must
not crash app startup over a permission glitch.
"""
if not staging_root:
return 0
root = Path(staging_root)
if not root.exists() or not root.is_dir():
return 0
active = active_batch_ids if active_batch_ids is not None else set()
# Normalize active batch ids to their on-disk dirname form so the
# set lookup matches what's actually on disk.
active_dirnames = {_safe_batch_dirname(bid) for bid in active if bid}
removed = 0
try:
entries = list(root.iterdir())
except OSError as exc:
logger.warning("[Album Bundle Sweep] Could not list staging root %s: %s", staging_root, exc)
return 0
for entry in entries:
if not entry.is_dir():
continue
# The directory name MUST match what _safe_batch_dirname
# produces — anything else was hand-created and we leave it
# alone. Defensive against stray dirs the user might have
# placed in the staging root.
if entry.name != _safe_batch_dirname(entry.name):
continue
if entry.name in active_dirnames:
continue
try:
shutil.rmtree(entry)
removed += 1
logger.info("[Album Bundle Sweep] Removed orphan staging dir: %s", entry)
except OSError as exc:
logger.warning("[Album Bundle Sweep] Could not remove orphan staging dir %s: %s", entry, exc)
if removed:
logger.info("[Album Bundle Sweep] Cleaned %d orphan staging dir(s) under %s", removed, staging_root)
return removed
@dataclass
class LifecycleDeps:
"""Bundle of cross-cutting deps the batch lifecycle needs."""
@ -321,7 +173,7 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
# Guard against double-calling: track which tasks have already been completed
# This prevents active_count from being decremented multiple times for the same task
# (e.g. status polling and post-processing both observe the same terminal task)
# (e.g. monitor detects completion AND post-processing calls this again)
# NOTE: On duplicate calls, we skip decrement/tracking but STILL check batch completion.
# This is critical because the first call may see the task in 'post_processing' (not finished),
# and the second call (from post-processing worker) arrives after the task is truly 'completed'.
@ -457,15 +309,9 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
retrying_count += 1
elif task_status == 'post_processing':
task_age = current_time - task.get('status_change_time', current_time)
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
new_status = _resolve_stuck_post_processing_status(task)
if new_status == 'completed':
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
task['status'] = 'completed'
else:
logger.warning(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
task['status'] = 'failed'
task['error_message'] = 'Post-processing timed out without producing a file'
if task_age > 300: # 5 minutes (post-processing should be fast)
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
task['status'] = 'completed' # Assume it worked if file verification is taking too long
finished_count += 1
else:
retrying_count += 1
@ -552,7 +398,6 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
logger.info(f"[Batch Manager] Batch {batch_id} complete - stopping monitor")
deps.download_monitor.stop_monitoring(batch_id)
_cleanup_private_album_bundle_staging(batch_id, batch)
# M3U REGENERATION: Regenerate M3U with real library paths now that
# all post-processing (tagging, moving, DB writes) is complete.
@ -577,24 +422,6 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
except Exception as m3u_err:
logger.error(f"[M3U] Error regenerating M3U on batch complete: {m3u_err}")
# PLAYLIST MATERIALIZE: one path-independent reconcile — drop this
# batch's newly-resolved tracks into the right Playlists/<name>/
# folders. Covers an organize-by-playlist download AND a late
# wishlist arrival (via each track's playlist provenance). Built
# from the batch's own captured paths — non-fatal, derived view.
try:
from core.playlists.materialize_service import reconcile_batch_playlists
from database.music_database import MusicDatabase
for _pl_name, _mat in reconcile_batch_playlists(MusicDatabase(), batch, download_tasks, deps.config_manager):
logger.info(
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}': "
f"{_mat.linked} linked, {_mat.copied} copied, "
f"{_mat.unchanged} unchanged, {_mat.removed_stale} stale removed"
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
)
except Exception as _mat_err:
logger.error(f"[Playlist Folder] Materialize failed (non-fatal): {_mat_err}")
# REPAIR: Scan all album folders from this batch for track number issues
if deps.repair_worker:
deps.repair_worker.process_batch(batch_id)
@ -695,15 +522,9 @@ def check_batch_completion_v2(batch_id: str, deps: LifecycleDeps) -> Optional[bo
retrying_count += 1
elif task_status == 'post_processing':
task_age = current_time - task.get('status_change_time', current_time)
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
new_status = _resolve_stuck_post_processing_status(task)
if new_status == 'completed':
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
task['status'] = 'completed'
else:
logger.warning(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
task['status'] = 'failed'
task['error_message'] = 'Post-processing timed out without producing a file'
if task_age > 300: # 5 minutes (post-processing should be fast)
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
task['status'] = 'completed' # Assume it worked if file verification is taking too long
finished_count += 1
else:
retrying_count += 1
@ -785,24 +606,6 @@ def check_batch_completion_v2(batch_id: str, deps: LifecycleDeps) -> Optional[bo
logger.info(f"[Completion Check V2] Batch {batch_id} complete - stopping monitor")
deps.download_monitor.stop_monitoring(batch_id)
_cleanup_private_album_bundle_staging(batch_id, batch)
# PLAYLIST MATERIALIZE: same reconcile as the primary completion path
# (on_download_completed). Monitor-detected downloads complete via THIS
# V2 path, so the reconcile must run here too or playlist folders never
# get built for them. Path-independent, non-fatal, derived view.
try:
from core.playlists.materialize_service import reconcile_batch_playlists
from database.music_database import MusicDatabase
for _pl_name, _mat in reconcile_batch_playlists(MusicDatabase(), batch, download_tasks, deps.config_manager):
logger.info(
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}': "
f"{_mat.linked} linked, {_mat.copied} copied, "
f"{_mat.unchanged} unchanged, {_mat.removed_stale} stale removed"
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
)
except Exception as _mat_err:
logger.error(f"[Playlist Folder] Materialize failed (non-fatal): {_mat_err}")
# REPAIR: Scan all album folders from this batch for track number issues
if deps.repair_worker:

385
core/downloads/master.py
View file

@ -35,7 +35,6 @@ from difflib import SequenceMatcher
from pathlib import Path
from typing import Any, Callable
from core.downloads import album_bundle_dispatch as _album_bundle_dispatch
from core.runtime_state import download_batches, download_tasks, tasks_lock
logger = logging.getLogger(__name__)
@ -50,7 +49,6 @@ _VARIANT_WORDS = {
'remix', 'rmx', 'acapella', 'a cappella', 'instrumental', 'karaoke',
'live', 'demo', 'extended',
}
_ALBUM_BUNDLE_SOURCES = frozenset(('torrent', 'usenet', 'soulseek'))
def _norm_text(value: Any) -> str:
@ -246,29 +244,6 @@ def _soulseek_album_preflight_enabled(config_manager: Any) -> bool:
return primary == 'soulseek'
def _resolve_album_bundle_source(config_manager: Any) -> str:
"""Return the album-bundle source for this batch.
In single-source mode, the active source may own the whole album if
it supports album bundles. In hybrid mode, only the first source in
the configured order may claim the whole album; later sources remain
per-track fallback.
"""
mode = (config_manager.get('download_source.mode', 'soulseek') or 'soulseek').lower()
if mode in _ALBUM_BUNDLE_SOURCES:
return mode
if mode != 'hybrid':
return ''
order = config_manager.get('download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'])
first = ''
if order:
first = str(order[0] or '').lower()
else:
first = str(config_manager.get('download_source.hybrid_primary', '') or '').lower()
return first if first in _ALBUM_BUNDLE_SOURCES else ''
@dataclass
class MasterDeps:
"""Bundle of cross-cutting deps the master worker needs."""
@ -295,72 +270,7 @@ class MasterDeps:
reset_wishlist_auto_processing: Callable[[], None]
class _BatchStateAccessImpl:
"""Concrete ``BatchStateAccess`` for the runtime ``download_batches``
dict — wraps the lock + the existing-batch check so the album-
bundle dispatcher stays decoupled from runtime_state."""
def update_fields(self, batch_id: str, fields: dict) -> None:
with tasks_lock:
row = download_batches.get(batch_id)
if row is not None:
row.update(fields)
def mark_failed(self, batch_id: str, error: str) -> None:
with tasks_lock:
row = download_batches.get(batch_id)
if row is not None:
row['phase'] = 'failed'
row['error'] = error
row['album_bundle_state'] = 'failed'
# Task states that mean a batch still has work in flight. While ANY of a batch's
# tasks is in one of these, a serialized album-pool worker keeps its slot.
_NON_TERMINAL_TASK_STATUSES = ('pending', 'queued', 'searching', 'downloading', 'post_processing')
def _wait_for_batch_drain(batch_id: str, poll_seconds: float = 1.5,
max_wait_seconds: float = 3600.0) -> None:
"""Block until every task in ``batch_id`` reaches a terminal state (the batch
is fully drained), the batch is removed, shutdown is requested, or a safety
cap elapses.
Used to make the dedicated album-bundle pool actually SERIALIZE albums: the
worker holds its pool slot for the album's whole lifetime instead of
returning the instant downloads are started. That stops every album from
dumping its tracks into the shared download pool at once (Sokhi: "searching
for way too many tracks at once"). It's a PASSIVE wait — the downloads are
driven by the monitor + completion callbacks on other threads, so this never
drives the work and can't deadlock; worst case the cap releases the slot and
the downloads simply finish in the background."""
from core.downloads import monitor as _monitor
start = time.time()
while True:
if getattr(_monitor, 'IS_SHUTTING_DOWN', False):
return
with tasks_lock:
batch = download_batches.get(batch_id)
if not batch:
return
queue = list(batch.get('queue', ()) or ())
still_working = any(
download_tasks.get(t, {}).get('status') in _NON_TERMINAL_TASK_STATUSES
for t in queue
)
if not still_working:
return
if time.time() - start > max_wait_seconds:
logger.warning(
"[Album Serialize] batch %s not drained after %.0fs — releasing the "
"album-pool slot (its downloads continue in the background)",
batch_id, max_wait_seconds)
return
time.sleep(poll_seconds)
def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: MasterDeps,
serialize: bool = False):
def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: MasterDeps):
"""
A master worker that handles the entire missing tracks process:
1. Runs the analysis.
@ -388,17 +298,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
batch_is_album = False
batch_profile_id = 1
batch_source = 'spotify'
batch_playlist_folder_mode = False
batch_playlist_name = 'Unknown Playlist'
batch_playlist_id = playlist_id
batch_source_playlist_ref = ''
# Issue #797 — per-request "Skip AcoustID verification" toggle from
# the album-download modal. When set, every track in this batch
# bypasses the AcoustID quarantine gate (the user has chosen to
# trust the metadata over fingerprint disagreement — useful for
# non-English artists whose native-script metadata AcoustID can't
# reconcile with the romanized request).
batch_skip_acoustid = False
with tasks_lock:
if batch_id in download_batches:
force_download_all = download_batches[batch_id].get('force_download_all', False)
@ -408,31 +307,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
batch_artist_context = download_batches[batch_id].get('artist_context')
batch_profile_id = download_batches[batch_id].get('profile_id', 1) or 1
batch_source = download_batches[batch_id].get('batch_source', 'spotify') or 'spotify'
batch_playlist_folder_mode = download_batches[batch_id].get('playlist_folder_mode', False)
batch_playlist_name = download_batches[batch_id].get('playlist_name', 'Unknown Playlist')
batch_playlist_id = download_batches[batch_id].get('playlist_id', playlist_id)
batch_source_playlist_ref = (
download_batches[batch_id].get('source_playlist_ref') or ''
).strip()
batch_skip_acoustid = bool(download_batches[batch_id].get('skip_acoustid', False))
from core.downloads.playlist_folder import (
resolve_playlist_folder_mode_for_batch,
track_exists_in_playlist_folder_from_track_data,
)
effective_playlist_folder_mode, effective_playlist_name = resolve_playlist_folder_mode_for_batch(
db,
playlist_id=str(batch_playlist_id),
playlist_name=batch_playlist_name,
batch_playlist_folder_mode=batch_playlist_folder_mode,
profile_id=batch_profile_id,
source=batch_source,
)
if effective_playlist_folder_mode and not batch_playlist_folder_mode:
with tasks_lock:
if batch_id in download_batches:
download_batches[batch_id]['playlist_folder_mode'] = True
download_batches[batch_id]['playlist_name'] = effective_playlist_name
if force_download_all:
logger.warning(f"[Force Download] Force download mode enabled for batch {batch_id} - treating all tracks as missing")
@ -505,19 +379,11 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
track_name = track_data.get('name', '')
artists = track_data.get('artists', [])
found, confidence = False, 0.0
# Additive payload: the owned library track (DatabaseTrack) when this
# item is found in the library, so downstream (playlist materialization)
# knows WHERE the real file is without re-matching. None when not owned.
matched_track = None
# Manual library matches are authoritative unless the user explicitly
# requested a force re-download from the normal download modal.
_stid = track_data.get('spotify_track_id') or track_data.get('source_track_id') or track_data.get('id', '')
_manual_match = (
_mlm.get_match_for_track(db, batch_profile_id, track_data, default_source=batch_source)
if (not ignore_manual_matches and _stid) else None
)
if _manual_match:
_stid = track_data.get('spotify_track_id') or track_data.get('id', '')
if not ignore_manual_matches and _stid and _mlm.get_match(db, batch_profile_id, batch_source, _stid):
logger.info(f"[Manual Match] '{track_name}' already matched in library — skipping download")
try:
deps.check_and_remove_track_from_wishlist_by_metadata(track_data)
@ -529,32 +395,9 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
'found': True,
'confidence': 1.0,
'match_reason': 'manual_library_match',
'matched_file_path': _manual_match.get('library_file_path'),
'matched_track_id': _manual_match.get('library_track_id'),
})
continue
if effective_playlist_folder_mode and not force_download_all:
if track_exists_in_playlist_folder_from_track_data(
effective_playlist_name,
track_data,
):
logger.info(
f"[Playlist Folder] '{track_name}' already on disk in playlist folder — skipping download"
)
try:
deps.check_and_remove_track_from_wishlist_by_metadata(track_data)
except Exception as _wl_err:
logger.debug(f"[Playlist Folder] Wishlist removal attempt failed: {_wl_err}")
analysis_results.append({
'track_index': track_index,
'track': track_data,
'found': True,
'confidence': 1.0,
'match_reason': 'playlist_folder_file',
})
continue
# Skip database check if force download is enabled
if force_download_all:
logger.warning(f"[Force Download] Skipping database check for '{track_name}' - treating as missing")
@ -576,10 +419,8 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
# Direct title match (try both raw and normalized)
if track_name_lower in album_tracks_map:
found, confidence = True, 1.0
matched_track = album_tracks_map[track_name_lower]
elif _normalized_source_title and _normalized_source_title in album_tracks_map:
found, confidence = True, 1.0
matched_track = album_tracks_map[_normalized_source_title]
else:
# Fuzzy match against album tracks using string similarity.
# Compare BOTH the raw and normalized source titles —
@ -587,17 +428,14 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
# matching when the album doesn't imply version
# context (helper returns the input unchanged).
best_sim = 0.0
best_track = None
for db_title_lower, _db_track in album_tracks_map.items():
sim_raw = db._string_similarity(track_name_lower, db_title_lower)
sim_norm = db._string_similarity(_normalized_source_title, db_title_lower) if _normalized_source_title else 0.0
sim = max(sim_raw, sim_norm)
if sim > best_sim:
best_sim = sim
best_track = _db_track
if best_sim >= 0.7:
found, confidence = True, best_sim
matched_track = best_track
else:
# Fall back to global per-track search for this track
# When allow_duplicates is on for album downloads, skip global
@ -618,7 +456,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
)
if db_track and track_confidence >= 0.7:
found, confidence = True, track_confidence
matched_track = db_track
break
elif allow_duplicates and batch_is_album:
# Allow duplicates + album download + album not in DB yet → treat all as missing
@ -638,15 +475,10 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
)
if db_track and track_confidence >= 0.7:
found, confidence = True, track_confidence
matched_track = db_track
break
analysis_results.append({
'track_index': track_index, 'track': track_data, 'found': found, 'confidence': confidence,
# Additive: real on-disk location of the owned track (None when not
# owned), so playlist materialization links the right file.
'matched_file_path': getattr(matched_track, 'file_path', None),
'matched_track_id': getattr(matched_track, 'id', None),
'track_index': track_index, 'track': track_data, 'found': found, 'confidence': confidence
})
# WISHLIST REMOVAL: If track is found in database, check if it should be removed from wishlist
@ -672,34 +504,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
if skipped > 0:
logger.warning(f"[Content Filter] Filtered out {skipped} explicit track(s) from download queue")
# Blocklist (Phase 2a): drop banned artists/albums/tracks before queueing,
# so a blocked item can't slip in via playlist sync / album download /
# discography. Same ID-cascade brain as the wishlist guard (Phase 1) —
# the only other auto-acquisition path. Skipped when the user confirmed
# "download anyway" at the modal (Phase 2b override).
_ignore_blocklist = False
with tasks_lock:
if batch_id in download_batches:
_ignore_blocklist = download_batches[batch_id].get('ignore_blocklist', False)
if not _ignore_blocklist:
try:
_bl_before = len(missing_tracks)
_bl_kept = []
for res in missing_tracks:
reason = db.blocklist_reason_for_track(
batch_profile_id, res.get('track', {}), source=batch_source)
if reason:
logger.info("[Blocklist] Skipping %s '%s' from download queue (%s blocked)",
reason[0], res.get('track', {}).get('name', '?'), reason[0])
else:
_bl_kept.append(res)
if len(_bl_kept) != _bl_before:
logger.info("[Blocklist] Filtered out %d blocklisted track(s) from download queue",
_bl_before - len(_bl_kept))
missing_tracks = _bl_kept
except Exception as _bl_err:
logger.debug("blocklist queue filter skipped: %s", _bl_err)
with tasks_lock:
if batch_id in download_batches:
download_batches[batch_id]['analysis_results'] = analysis_results
@ -785,41 +589,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
logger.warning("[Auto-Wishlist] No missing tracks found - calling auto-completion handler to toggle cycle and reschedule")
deps.missing_download_executor.submit(deps.process_failed_tracks_to_wishlist_exact_with_auto_completion, batch_id)
# Organize-by-playlist with NOTHING to download (every track already
# owned): the batch never enters the download/lifecycle path, so build
# the playlist folder here from the owned files the analysis matched.
# Gated + non-fatal; runs once after analysis, not in the per-track loop.
if effective_playlist_folder_mode:
try:
from core.playlists.materialize_service import reconcile_batch_playlists
from database.music_database import MusicDatabase as _MDB
_batch = download_batches.get(batch_id)
if _batch is not None:
# We KNOW the intent is organize-by-playlist here (the gate
# above). The line-431 sync only writes the dict field when
# effective and NOT batch_playlist_folder_mode, so when the
# toggle itself drove it the dict field can still be falsy —
# which makes reconcile build no batch ref. Make the dict
# authoritative so reconcile sees the batch's own playlist.
_batch['playlist_folder_mode'] = True
if effective_playlist_name:
_batch['playlist_name'] = effective_playlist_name
_results = reconcile_batch_playlists(_MDB(), _batch, download_tasks, deps.config_manager)
if not _results:
logger.info(
f"[Playlist Folder] All-owned: nothing rebuilt for "
f"ref={_batch.get('source_playlist_ref') or _batch.get('playlist_id')} "
f"source={_batch.get('batch_source')}"
)
for _pl_name, _mat in _results:
logger.info(
f"[Playlist Folder] Rebuilt '{_mat.playlist_dir}' (all owned): "
f"{_mat.linked} linked, {_mat.copied} copied, {_mat.removed_stale} stale removed"
+ (" (symlinks unsupported here → copied)" if _mat.fellback else "")
)
except Exception as _mat_err:
logger.error(f"[Playlist Folder] All-owned materialize failed (non-fatal): {_mat_err}")
return
logger.warning(f" transitioning batch {batch_id} to download phase with {len(missing_tracks)} tracks.")
@ -831,36 +600,15 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
batch_album_context = batch.get('album_context')
batch_artist_context = batch.get('artist_context')
batch_is_album = batch.get('is_album_download', False)
batch_private_album_bundle = bool(batch.get('album_bundle_private_staging'))
batch_playlist_folder_mode = batch.get('playlist_folder_mode', False)
batch_playlist_name = batch.get('playlist_name', 'Unknown Playlist')
# Album-bundle sources download a whole release into private staging,
# then the normal per-track workers claim those staged files. Run this
# only after analysis has found missing tracks; otherwise an already
# owned album would still trigger a release download.
_bundle_state = _BatchStateAccessImpl()
_album_bundle_source = _resolve_album_bundle_source(deps.config_manager)
if _album_bundle_source and _album_bundle_source != 'soulseek':
if _album_bundle_dispatch.try_dispatch(
batch_id=batch_id,
is_album=batch_is_album,
album_context=batch_album_context,
artist_context=batch_artist_context,
config_get=deps.config_manager.get,
plugin_resolver=deps.download_orchestrator.client,
state=_bundle_state,
source_override=_album_bundle_source,
):
return
# === ALBUM PRE-FLIGHT: Search for complete album folder before track-by-track ===
# Only run pre-flight when Soulseek is the download source (or hybrid with soulseek)
preflight_source = None
preflight_tracks = None
soulseek_is_source = _soulseek_album_preflight_enabled(deps.config_manager)
if (batch_is_album and batch_album_context and batch_artist_context
and soulseek_is_source and not batch_private_album_bundle):
if batch_is_album and batch_album_context and batch_artist_context and soulseek_is_source:
artist_name = batch_artist_context.get('name', '')
album_name = batch_album_context.get('name', '')
if artist_name and album_name:
@ -973,44 +721,13 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
logger.error(f"[Album Pre-flight] Search failed (non-fatal, falling back to track-by-track): {preflight_err}")
deps.source_reuse_logger.info(f"[Album Pre-flight] Exception: {preflight_err}")
# Soulseek album bundles run after analysis so an already-owned
# album does not get downloaded just because the source supports a
# whole-folder flow. When preflight selected a folder, pass that
# exact source into the bundle downloader so we keep the richer
# tracklist-aware scoring instead of doing a weaker second pick.
_bundle_state = _BatchStateAccessImpl()
_album_bundle_source = _resolve_album_bundle_source(deps.config_manager)
if _album_bundle_source == 'soulseek':
if _album_bundle_dispatch.try_dispatch(
batch_id=batch_id,
is_album=batch_is_album,
album_context=batch_album_context,
artist_context=batch_artist_context,
config_get=deps.config_manager.get,
plugin_resolver=deps.download_orchestrator.client,
state=_bundle_state,
source_override=_album_bundle_source,
plugin_kwargs={
'preferred_source': preflight_source,
'preferred_tracks': preflight_tracks,
} if preflight_source and preflight_tracks else None,
):
return
with tasks_lock:
if batch_id not in download_batches: return
download_batches[batch_id]['phase'] = 'downloading'
# Store album pre-flight results on batch for source reuse
# unless the Soulseek album-bundle path already staged a private
# release. Task workers check source reuse before staging match, so
# preloading here would make the staged happy path re-download.
if (
preflight_source
and preflight_tracks
and not download_batches[batch_id].get('album_bundle_private_staging')
):
if preflight_source and preflight_tracks:
download_batches[batch_id]['last_good_source'] = preflight_source
download_batches[batch_id]['source_folder_tracks'] = preflight_tracks
download_batches[batch_id]['failed_sources'] = set()
@ -1166,81 +883,13 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
logger.info(f"[Wishlist] Added album context for: '{track_info.get('name')}' -> '{album_ctx['name']}'")
# Issue #797 — propagate the batch-level "skip AcoustID"
# toggle onto each track so the per-track download context
# (built in core/downloads/candidates.py) can set the
# AcoustID quarantine bypass. Mirrors the _playlist_folder_mode
# threading pattern below.
if batch_skip_acoustid:
track_info['_skip_acoustid'] = True
# Add playlist folder mode flag for sync page playlists and wishlist
# tracks tied to a mirrored playlist with organize_by_playlist enabled.
task_pl_folder_mode = batch_playlist_folder_mode
task_pl_name = batch_playlist_name
if not task_pl_folder_mode and playlist_id == 'wishlist':
wl_source = track_info.get('source_info') or {}
if isinstance(wl_source, str):
try:
wl_source = json.loads(wl_source)
except (json.JSONDecodeError, TypeError):
wl_source = {}
wl_pl_ref = wl_source.get('playlist_id')
wl_pl_name = wl_source.get('playlist_name')
wl_pl_source = wl_source.get('source') or 'spotify'
if wl_pl_ref and hasattr(db, 'resolve_mirrored_playlist'):
wl_mirrored = db.resolve_mirrored_playlist(
wl_pl_ref,
profile_id=batch_profile_id,
default_source=wl_pl_source,
)
if wl_mirrored and wl_mirrored.get('organize_by_playlist'):
task_pl_folder_mode = True
task_pl_name = wl_pl_name or wl_mirrored.get('name') or batch_playlist_name
if task_pl_folder_mode:
# Organize-by-playlist now imports each track NORMALLY into the
# Artist/Album library (i.e. exactly what a normal download does)
# — the playlist folder is built as links/copies AFTER the batch
# from the real library files. So we deliberately DON'T set
# `_playlist_folder_mode` (which routed the real file into a flat
# Music/<playlist>/ dump). We keep `_playlist_name` + source_info
# — they're download provenance (core/downloads/origin.py).
track_info['_playlist_name'] = task_pl_name
if batch_source_playlist_ref:
track_info['source_info'] = {
'playlist_id': batch_source_playlist_ref,
'playlist_name': task_pl_name,
'source': batch_source,
}
logger.info(
f"[Task Creation] Organize-by-playlist (normal import + "
f"materialize after batch): {track_info.get('name')} → {task_pl_name}"
)
# Add playlist folder mode flag for sync page playlists
if batch_playlist_folder_mode:
track_info['_playlist_folder_mode'] = True
track_info['_playlist_name'] = batch_playlist_name
logger.info(f"[Task Creation] Added playlist folder mode for: {track_info.get('name')} → {batch_playlist_name}")
else:
logger.debug(
f"[Debug] Task Creation - playlist folder mode NOT enabled for: "
f"{track_info.get('name')}"
)
# Download-origin provenance: stamp what TRIGGERED this download
# so the history chokepoint can record it (origin-history modal).
# Wishlist rows already ride their source_info in track_info
# (watchlist_artist_name / playlist_name — the deriver reads
# those directly); this stamp covers DIRECT playlist batches,
# where the playlist context otherwise only survives in
# folder mode.
if '_dl_origin' not in track_info and batch_source_playlist_ref and batch_playlist_name:
_prov_si = track_info.get('source_info') or {}
if isinstance(_prov_si, str):
try:
_prov_si = json.loads(_prov_si)
except (json.JSONDecodeError, TypeError):
_prov_si = {}
if not _prov_si.get('watchlist_artist_name'):
track_info['_dl_origin'] = 'playlist'
track_info['_dl_origin_context'] = (
_prov_si.get('playlist_name') or batch_playlist_name
)
logger.debug(f"[Debug] Task Creation - playlist folder mode NOT enabled for: {track_info.get('name')}")
download_tasks[task_id] = {
'status': 'pending', 'track_info': track_info,
@ -1255,16 +904,6 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
deps.download_monitor.start_monitoring(batch_id)
deps.start_next_batch_of_downloads(batch_id)
# Album-bundle batches run on the dedicated album pool and pass
# serialize=True: hold this pool slot until the album finishes so only a
# few albums are ever in flight at once, instead of every album batch
# immediately starting and flooding the shared download pool with
# 'searching' tracks (#740 / Sokhi). The residual + playlist + manual
# paths run on the shared download pool and DON'T serialize (blocking
# there would steal an actual download worker).
if serialize:
_wait_for_batch_drain(batch_id)
except Exception as e:
logger.error(f"Master worker for batch {batch_id} failed: {e}")
import traceback

403
core/downloads/monitor.py
View file

@ -5,6 +5,7 @@ The class body is byte-identical to the original. Module-level globals
helpers and orchestrator handles. ``IS_SHUTTING_DOWN`` is a module-level
flag mirrored from web_server's own flag in ``_shutdown_runtime_components``.
"""
import logging
import threading
import time
@ -17,10 +18,8 @@ from core.runtime_state import (
tasks_lock,
)
from utils.async_helpers import run_async
from utils.logging_config import get_logger
# Project logger factory so these lines reach app.log (soulsync.* namespace).
logger = get_logger("downloads.monitor")
logger = logging.getLogger(__name__)
# Mirrored from web_server.IS_SHUTTING_DOWN via _shutdown_runtime_components.
IS_SHUTTING_DOWN = False
@ -35,254 +34,6 @@ _start_next_batch_of_downloads = None
_orphaned_download_keys = None
missing_download_executor = None
download_orchestrator = None
_RELEASE_SOURCE_NAMES = frozenset(('torrent', 'usenet'))
# Hard ceiling on automatic next-candidate retries after a download was
# quarantined (AcoustID mismatch / integrity / duration). The natural
# terminator is used_sources exhaustion — once every candidate the worker can
# find has been tried, attempt_download_with_candidates returns False and the
# worker reports a clean failure. This cap is a safety net against a pathological
# quarantine→retry→quarantine loop (e.g. a source that keeps returning fresh
# wrong files).
#
# Default (non-exhaustive) mode uses this single global cap. The opt-in
# exhaustive mode (post_processing.retry_exhaustive) instead budgets retries
# PER SOURCE — see requeue_quarantined_task_for_retry.
MAX_QUARANTINE_RETRIES = 5
# Absolute runaway guard for exhaustive mode. Per-source budgets are already
# finite (query_count × retries_per_query, and Soulseek peers all collapse to
# one 'soulseek' bucket), but this ceiling caps the TOTAL retries across every
# source so a misbehaving source-resolution can never loop forever.
MAX_TOTAL_QUARANTINE_RETRIES = 100
# Streaming plugins report their source name as the download's "username"
# (see download_orchestrator._streaming_sources). Soulseek uses the peer name
# instead, so anything not in this set is bucketed under 'soulseek' for the
# per-source retry budget.
_STREAMING_SOURCE_NAMES = frozenset((
'youtube', 'tidal', 'qobuz', 'hifi', 'deezer_dl', 'lidarr', 'soundcloud', 'amazon',
))
def _resolve_download_source(username):
"""Map a download's username to its logical source for per-source budgeting.
Streaming sources use the source name as username; Soulseek uses the peer
name, so every Soulseek peer collapses to a single 'soulseek' bucket.
"""
if username and username in _STREAMING_SOURCE_NAMES:
return username
return 'soulseek'
def _remaining_fallback_sources(exhausted):
"""Sources in the configured hybrid chain that haven't exhausted their
per-source budget yet.
When a source spends its whole budget (exhaustive mode), the task switches
to the next source instead of failing — but only if there *is* another
source. Single-source mode has nothing to fall back to, so this returns
empty there (and when the orchestrator isn't wired). The returned list
drives both the give-up decision here and the worker's search-exclusion on
the next attempt (see task_worker: exhausted_download_sources).
"""
orch = download_orchestrator
if orch is None or getattr(orch, 'mode', None) != 'hybrid':
return []
chain = getattr(orch, 'hybrid_order', None) or []
blocked = {str(s).lower() for s in exhausted}
return [s for s in chain if str(s).lower() not in blocked]
def _download_id_key(download_id):
return f"download_id::{download_id}" if download_id else None
def requeue_quarantined_task_for_retry(task_id, batch_id, trigger):
"""Re-queue a task whose download was just quarantined so the worker tries
the NEXT best candidate instead of failing outright.
Called from the post-processing verification wrapper when AcoustID
verification or the integrity/duration check quarantines a file. It mirrors
the monitor's transfer-error retry path: mark the bad source as used, clear
the stale download identity, reset the task to ``searching`` and resubmit
the download worker. Because ``used_sources`` is preserved across the
re-run, the worker skips the quarantined source and picks the next-best
candidate (see ``attempt_download_with_candidates``).
Returns True if a retry was queued — the caller must then NOT mark the task
failed or notify batch completion, since the task is going around again.
Returns False when no retry is possible (retry engine unwired, manual pick,
cancelled, or retry budget exhausted); the caller falls through to its
existing failure handling.
"""
# Opt-out escape hatch — default on. Lets users restore the old
# quarantine-and-fail behaviour without a code change.
if not config_manager.get('post_processing.retry_next_candidate_on_mismatch', True):
return False
# Retry engine not wired (e.g. manual-import path that never started a
# download worker). Nothing to re-run.
if missing_download_executor is None or _download_track_worker is None:
return False
with tasks_lock:
task = download_tasks.get(task_id)
if not task:
return False
# The user explicitly picked this candidate via the candidates modal —
# honour their choice rather than silently swapping in another file.
# (Matches the monitor's transfer-retry guards.)
if task.get('_user_manual_pick'):
return False
if task.get('status') == 'cancelled':
return False
username = task.get('username')
filename = task.get('filename')
# No source identity means this wasn't a worker-dispatched download we
# can retry — without the "{username}_{filename}" key we can't flag the
# bad source as used, so a re-run could re-pick the same file and loop.
# Bail and let the caller fail it normally.
if not username or not filename:
return False
total_count = task.get('quarantine_retry_count', 0)
if config_manager.get('post_processing.retry_exhaustive', False):
# Exhaustive mode: a SEPARATE budget per source. The budget scales
# with the track's own query count (the worker generates a variable
# number of search queries per track) × the configured retries per
# query. Soulseek candidates are walked first (one per retry), then
# the worker's hybrid fallback moves to the next source — each source
# spending its own budget. The natural terminator (used_sources
# exhaustion → worker clean-fail) still ends most tracks well before
# any budget is reached; the budget is the per-source safety ceiling.
source = _resolve_download_source(username)
retries_per_query = config_manager.get('post_processing.retries_per_query', 5)
try:
retries_per_query = int(retries_per_query)
except (TypeError, ValueError):
retries_per_query = 5
if retries_per_query < 1:
retries_per_query = 1
query_count = task.get('query_count') or 1
if query_count < 1:
query_count = 1
budget = query_count * retries_per_query
counts = task.get('quarantine_retry_counts_by_source')
if not isinstance(counts, dict):
counts = {}
source_count = counts.get(source, 0)
if source_count >= budget:
# This source spent its whole budget. Rather than fail the
# track outright, mark the source exhausted and fall through to
# the next source in the hybrid chain (the worker excludes
# exhausted sources from its next search). Only give up once no
# fallback source remains — or the absolute ceiling trips.
exhausted = set(task.get('exhausted_download_sources') or ())
exhausted.add(source)
remaining = _remaining_fallback_sources(exhausted)
if not remaining:
logger.warning(
f"[Retry:{trigger}] Task {task_id} exhausted its retry "
f"budget for source '{source}' ({source_count}/{budget}) "
f"and no fallback source remains — giving up, marking failed"
)
return False
if total_count >= MAX_TOTAL_QUARANTINE_RETRIES:
logger.warning(
f"[Retry:{trigger}] Task {task_id} hit the absolute retry "
f"ceiling ({MAX_TOTAL_QUARANTINE_RETRIES}) — giving up, "
f"marking failed"
)
return False
task['exhausted_download_sources'] = exhausted
# Don't push this source's counter past its budget — it's done.
# The next source starts spending its own fresh budget when its
# first candidate fails verification.
attempt_desc = (
f"source '{source}' budget spent ({source_count}/{budget}) "
f"— switching sources (remaining: {', '.join(remaining)})"
)
else:
if total_count >= MAX_TOTAL_QUARANTINE_RETRIES:
logger.warning(
f"[Retry:{trigger}] Task {task_id} hit the absolute retry "
f"ceiling ({MAX_TOTAL_QUARANTINE_RETRIES}) — giving up, "
f"marking failed"
)
return False
counts[source] = source_count + 1
task['quarantine_retry_counts_by_source'] = counts
attempt_desc = f"source '{source}' {source_count + 1}/{budget}"
else:
# Default mode: a single global cap, conservative and predictable.
if total_count >= MAX_QUARANTINE_RETRIES:
logger.warning(
f"[Retry:{trigger}] Task {task_id} hit the quarantine-retry cap "
f"({MAX_QUARANTINE_RETRIES}) — giving up, marking failed"
)
return False
attempt_desc = f"{total_count + 1}/{MAX_QUARANTINE_RETRIES}"
# Mark the quarantined source as used so the re-run won't pick it again.
# Uses the same "{username}_{filename}" key the worker dedups against.
used_sources = task.get('used_sources', set())
used_sources.add(f"{username}_{filename}")
task['used_sources'] = used_sources
task['quarantine_retry_count'] = total_count + 1
# Flag the re-run as a quarantine retry so the worker walks the
# already-found candidates (cached-first) before re-searching — the
# connection was fine, the content was just wrong. Dead-connection /
# stuck retries (handled elsewhere in the monitor) deliberately do NOT
# set this, so they re-search fresh.
task['_quarantine_retry'] = True
# Drop the stale download identity + the prior attempt's quarantine link.
task.pop('download_id', None)
task.pop('username', None)
task.pop('filename', None)
task.pop('quarantine_entry_id', None)
task['status'] = 'searching'
task['status_change_time'] = time.time()
# Surface the retry progress to the UI ("attempt 2/5" next to the
# status while the task goes around again). Cleared implicitly on
# completion (UI only renders it for active/queued states).
task['retry_info'] = attempt_desc
task['retry_trigger'] = trigger
logger.info(
f"[Retry:{trigger}] Re-queuing task {task_id} for next-best candidate "
f"(attempt {attempt_desc})"
)
missing_download_executor.submit(_download_track_worker, task_id, batch_id)
return True
def _is_release_task(task):
ti = task.get('track_info') if isinstance(task.get('track_info'), dict) else {}
username = task.get('username') or ti.get('username')
return username in _RELEASE_SOURCE_NAMES
def _lookup_live_info(task, live_transfers_lookup):
ti = task.get('track_info') if isinstance(task.get('track_info'), dict) else {}
download_id = task.get('download_id')
if _is_release_task(task):
by_id = live_transfers_lookup.get(_download_id_key(download_id))
if by_id:
return by_id
task_filename = task.get('filename') or ti.get('filename')
task_username = task.get('username') or ti.get('username')
if not task_filename or not task_username:
return None
return live_transfers_lookup.get(_make_context_key(task_username, task_filename))
def init(
@ -388,64 +139,55 @@ class WebUIDownloadMonitor:
for task_id in download_batches[batch_id].get('queue', []):
task = download_tasks.get(task_id)
if not task:
continue
release_recoverable = (
_is_release_task(task)
and task.get('download_id')
and task.get('status') in ['failed', 'not_found']
)
if task['status'] not in ['downloading', 'queued'] and not release_recoverable:
if not task or task['status'] not in ['downloading', 'queued']:
continue
# Check for timeouts and errors - retries handled directly in _should_retry_task
# If _should_retry_task returns True, it means retries were exhausted
retry_exhausted = False
if not release_recoverable:
retry_exhausted = self._should_retry_task(task_id, task, live_transfers_lookup, current_time, deferred_ops)
retry_exhausted = self._should_retry_task(task_id, task, live_transfers_lookup, current_time, deferred_ops)
# Collect exhausted tasks to handle outside lock (prevents deadlock)
if retry_exhausted:
exhausted_tasks.append((batch_id, task_id))
# ENHANCED: Check for successful completions (especially YouTube).
# Release-style sources can report a completed audio file
# name that differs from the original indexer URL/title
# stored on the task, so prefer the stable download_id.
live_info = _lookup_live_info(task, live_transfers_lookup)
# ENHANCED: Check for successful completions (especially YouTube)
task_filename = task.get('filename') or task.get('track_info', {}).get('filename')
task_username = task.get('username') or task.get('track_info', {}).get('username')
if live_info:
state = live_info.get('state', '')
# Trigger post-processing if download is completed successfully
# slskd uses compound states like 'Completed, Succeeded' - use substring matching
# Must exclude error states first (matching _build_batch_status_data's prioritized checking)
has_error = ('Errored' in state or 'Failed' in state or 'Rejected' in state or 'TimedOut' in state)
has_completion = ('Completed' in state or 'Succeeded' in state)
# Verify bytes actually transferred before trusting state string.
# slskd can report "Completed" before the full file is flushed to disk,
# or on connection drops that leave a partial file.
if has_completion and not has_error:
expected_size = live_info.get('size', 0)
transferred = live_info.get('bytesTransferred', 0)
if expected_size > 0 and transferred < expected_size:
if not task.get('_incomplete_warned'):
logger.debug("Monitor: %s state=%s but bytes incomplete (%s/%s) - waiting", task_id, state, transferred, expected_size)
task['_incomplete_warned'] = True
continue
if has_completion and not has_error and (
task['status'] == 'downloading' or release_recoverable
):
task.pop('_incomplete_warned', None)
# CRITICAL FIX: Transition to 'post_processing' HERE so downloads
# don't depend on browser polling to trigger post-processing.
# Previously, post-processing was only submitted by _build_batch_status_data
# (called from browser-polled endpoints), meaning closing the browser
# left tasks stuck in 'downloading' forever.
task['status'] = 'post_processing'
task['status_change_time'] = current_time
logger.info(f"Monitor detected completed download for {task_id} ({state}) - submitting post-processing")
# Collect for handling outside the lock to prevent deadlock.
# _on_download_completed acquires tasks_lock which is non-reentrant.
completed_tasks.append((batch_id, task_id))
if task_filename and task_username:
lookup_key = _make_context_key(task_username, task_filename)
live_info = live_transfers_lookup.get(lookup_key)
if live_info:
state = live_info.get('state', '')
# Trigger post-processing if download is completed successfully
# slskd uses compound states like 'Completed, Succeeded' - use substring matching
# Must exclude error states first (matching _build_batch_status_data's prioritized checking)
has_error = ('Errored' in state or 'Failed' in state or 'Rejected' in state or 'TimedOut' in state)
has_completion = ('Completed' in state or 'Succeeded' in state)
# Verify bytes actually transferred before trusting state string.
# slskd can report "Completed" before the full file is flushed to disk,
# or on connection drops that leave a partial file.
if has_completion and not has_error:
expected_size = live_info.get('size', 0)
transferred = live_info.get('bytesTransferred', 0)
if expected_size > 0 and transferred < expected_size:
if not task.get('_incomplete_warned'):
logger.debug(f"Monitor: {task_id} state={state} but bytes incomplete ({transferred}/{expected_size}) — waiting")
task['_incomplete_warned'] = True
continue
if has_completion and not has_error and task['status'] == 'downloading':
task.pop('_incomplete_warned', None)
# CRITICAL FIX: Transition to 'post_processing' HERE so downloads
# don't depend on browser polling to trigger post-processing.
# Previously, post-processing was only submitted by _build_batch_status_data
# (called from browser-polled endpoints), meaning closing the browser
# left tasks stuck in 'downloading' forever.
task['status'] = 'post_processing'
task['status_change_time'] = current_time
logger.info(f"Monitor detected completed download for {task_id} ({state}) - submitting post-processing")
# Collect for handling outside the lock to prevent deadlock.
# _on_download_completed acquires tasks_lock which is non-reentrant.
completed_tasks.append((batch_id, task_id))
# ---- All work below runs WITHOUT tasks_lock held ----
if globals().get('IS_SHUTTING_DOWN', False) or not self.monitoring:
@ -455,21 +197,8 @@ class WebUIDownloadMonitor:
for op in deferred_ops:
try:
if op[0] == 'cancel_download':
# Issue #648 diagnostic — `op` now carries a trigger
# label (4-tuple, was 3-tuple) so the next log dump
# tells us WHICH path in `_should_retry_task` is
# firing for users seeing "Tidal downloads failed to
# start" mass-cancels. Label format pinned in commit
# message for grep-ability.
if len(op) >= 4:
_, download_id, username, trigger = op[0], op[1], op[2], op[3]
else:
_, download_id, username = op
trigger = 'unlabeled'
logger.info(
f"[CancelTrigger:monitor.{trigger}] download_id={download_id} "
f"username={username}"
)
_, download_id, username = op
logger.debug(f"[Deferred] Cancelling download: {download_id} from {username}")
run_async(download_orchestrator.cancel_download(download_id, username, remove=True))
logger.debug(f"[Deferred] Successfully cancelled download {download_id}")
elif op[0] == 'cleanup_orphan':
@ -485,28 +214,18 @@ class WebUIDownloadMonitor:
except Exception as e:
logger.error(f"[Deferred] Error executing deferred operation {op[0]}: {e}")
# Handle completed transfers outside the lock. The transfer engine's
# "complete" state only means the remote download finished; the
# post-processing worker still has to find, verify, tag, and move the
# file before it can report real batch success or failure.
# Handle completed downloads outside the lock to prevent deadlock
# (_on_download_completed acquires tasks_lock internally)
for batch_id, task_id in completed_tasks:
try:
# Submit post-processing worker (file move, tagging, AcoustID verification)
# This makes batch downloads fully independent of browser polling.
logger.info(f"[Monitor] Submitting post-processing worker for task {task_id}")
missing_download_executor.submit(_run_post_processing_worker, task_id, batch_id)
# Chain to next download in the batch queue
_on_download_completed(batch_id, task_id, success=True)
except Exception as e:
logger.error(f"[Monitor] Error handling completed task {task_id}: {e}")
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['status'] = 'failed'
download_tasks[task_id]['error_message'] = f'Post-processing could not be scheduled: {e}'
try:
_on_download_completed(batch_id, task_id, success=False)
except Exception as completion_error:
logger.error(
f"[Monitor] Error marking failed post-processing submit for task {task_id}: {completion_error}"
)
# Handle exhausted retry tasks outside the lock to prevent deadlock
for batch_id, task_id in exhausted_tasks:
try:
@ -566,7 +285,7 @@ class WebUIDownloadMonitor:
for download in all_downloads:
key = _make_context_key(download.username, download.filename)
# Convert DownloadStatus to transfer dict format for monitor compatibility
transfer_row = {
live_transfers[key] = {
'id': download.id,
'filename': download.filename,
'username': download.username,
@ -576,10 +295,6 @@ class WebUIDownloadMonitor:
'bytesTransferred': download.transferred,
'averageSpeed': download.speed,
}
live_transfers[key] = transfer_row
id_key = _download_id_key(download.id)
if id_key:
live_transfers[id_key] = transfer_row
except Exception as yt_error:
logger.error(f"Monitor: Could not fetch streaming source downloads: {yt_error}")
@ -614,7 +329,7 @@ class WebUIDownloadMonitor:
return False
lookup_key = _make_context_key(task_username, task_filename)
live_info = _lookup_live_info(task, live_transfers_lookup)
live_info = live_transfers_lookup.get(lookup_key)
if not live_info:
# User-initiated manual pick — skip auto-retry. The status
@ -638,8 +353,7 @@ class WebUIDownloadMonitor:
# Defer slskd cancel to outside the lock
if task_username and download_id:
deferred_ops.append(('cancel_download', download_id, task_username,
'not_in_live_transfers_90s'))
deferred_ops.append(('cancel_download', download_id, task_username))
# Mark current source as used (full filename to match worker format)
if task_username and task_filename:
@ -710,8 +424,7 @@ class WebUIDownloadMonitor:
# Defer slskd cancel to outside the lock
if username and download_id:
deferred_ops.append(('cancel_download', download_id, username,
'errored_state_retry'))
deferred_ops.append(('cancel_download', download_id, username))
# Mark current source as used to prevent retry loops
# CRITICAL: Use full filename (not basename) to match worker's source_key format
@ -760,8 +473,7 @@ class WebUIDownloadMonitor:
# used_sources keys are formatted as "{username}_{filename}", so startswith is exact.
is_tidal = any(s.startswith('tidal_') for s in tried_sources)
if is_tidal:
from core.quality.source_map import quality_tier_for_source
tidal_quality = quality_tier_for_source('tidal', default='lossless')
tidal_quality = config_manager.get('tidal_download.quality', 'lossless')
allow_fb = config_manager.get('tidal_download.allow_fallback', True)
if tidal_quality == 'hires' and not allow_fb:
task['error_message'] = (
@ -815,8 +527,7 @@ class WebUIDownloadMonitor:
# Defer slskd cancel to outside the lock
if username and download_id:
deferred_ops.append(('cancel_download', download_id, username,
'queued_state_timeout'))
deferred_ops.append(('cancel_download', download_id, username))
# UNIFIED RETRY LOGIC: Handle timeout retry exactly like error retry
# Mark current source as used to prevent retry loops
@ -904,8 +615,7 @@ class WebUIDownloadMonitor:
# Defer slskd cancel to outside the lock
if username and download_id:
deferred_ops.append(('cancel_download', download_id, username,
'stuck_at_0pct_timeout'))
deferred_ops.append(('cancel_download', download_id, username))
# UNIFIED RETRY LOGIC: Handle 0% timeout retry exactly like error retry
# Mark current source as used to prevent retry loops
@ -994,8 +704,7 @@ class WebUIDownloadMonitor:
download_id = task.get('download_id')
if username and download_id:
deferred_ops.append(('cancel_download', download_id, username,
'unknown_state_no_progress_timeout'))
deferred_ops.append(('cancel_download', download_id, username))
if username and filename:
used_sources = task.get('used_sources', set())

71
core/downloads/origin.py
View file

@ -1,71 +0,0 @@
"""Download-origin provenance: what TRIGGERED a download.
The library history records which SERVICE a file came from (Soulseek,
YouTube, ...) but not WHY it was downloaded — a watchlist scan, a playlist
sync, or a manual click. The origin-history modal (watchlist page / sync
page) answers that, so the trigger must be derived once, at the history
chokepoint (``record_library_history_download``), from the post-process
context.
Signals, in priority order:
1. explicit ``track_info._dl_origin`` / ``_dl_origin_context`` stamps
(set at batch-task creation in core/downloads/master.py)
2. wishlist provenance riding in ``track_info.source_info`` — watchlist
items carry ``watchlist_artist_name``, playlist items ``playlist_name``
3. the playlist-folder-mode ``_playlist_name`` thread
Anything unmatched derives ``(None, '')`` — manual/other downloads are
intentionally not classified.
"""
from __future__ import annotations
import json
from typing import Any, Dict, Optional, Tuple
ORIGIN_WATCHLIST = "watchlist"
ORIGIN_PLAYLIST = "playlist"
VALID_ORIGINS = (ORIGIN_WATCHLIST, ORIGIN_PLAYLIST)
def _parse_source_info(raw: Any) -> Dict[str, Any]:
if isinstance(raw, dict):
return raw
if isinstance(raw, str) and raw:
try:
parsed = json.loads(raw)
return parsed if isinstance(parsed, dict) else {}
except (json.JSONDecodeError, TypeError):
return {}
return {}
def derive_download_origin(context: Dict[str, Any]) -> Tuple[Optional[str], str]:
"""Return ``(origin, origin_context)`` for a completed download.
``origin`` is 'watchlist' / 'playlist' / None; ``origin_context`` is the
human label (watchlist artist name / playlist name). Never raises."""
try:
ti = context.get("track_info") or {}
if not isinstance(ti, dict):
return None, ""
si = _parse_source_info(ti.get("source_info"))
# 1. Explicit stamp wins.
origin = ti.get("_dl_origin")
if origin in VALID_ORIGINS:
return origin, str(ti.get("_dl_origin_context") or "")
# 2. Wishlist provenance riding in source_info.
if si.get("watchlist_artist_name"):
return ORIGIN_WATCHLIST, str(si["watchlist_artist_name"])
if si.get("playlist_name"):
return ORIGIN_PLAYLIST, str(si["playlist_name"])
# 3. Playlist-folder-mode thread.
if ti.get("_playlist_name"):
return ORIGIN_PLAYLIST, str(ti["_playlist_name"])
return None, ""
except Exception:
return None, ""

55
core/downloads/orphan_history.py
View file

@ -1,55 +0,0 @@
"""Identify dead review-queue history rows whose file is gone (#934 follow-up).
The Unverified/Quarantine review queue is fed from ``library_history`` — an
append-only log that is never pruned. When a file is deleted, replaced, or
re-downloaded elsewhere, its old ``unverified`` row lingers forever and can
never be healed (there's no file left to confirm). Those are *orphans*.
This decides which rows are orphans, given a ``resolve(row) -> path | None``
the caller wires to the real filesystem lookup. Pure (no DB, no filesystem) so
the rules — including the safety gate — are unit-testable.
Safety gate: a filesystem check mass-false-positives when the library mount is
down (every file looks missing). So if EVERY reviewed file is unreachable and
there are enough rows to judge, we flag it ``suspicious`` and the caller refuses
to delete — better to clean nothing than to wipe a healthy log during an outage.
"""
from __future__ import annotations
from typing import Any, Callable, Sequence
def find_orphan_history_ids(
rows: Sequence[dict],
resolve: Callable[[dict], Any],
*,
min_for_safety: int = 5,
deletable: Callable[[dict], bool] | None = None,
) -> dict:
"""Return ``{'orphan_ids', 'checked', 'suspicious'}``.
A row is an orphan when it has a non-empty ``file_path`` but ``resolve`` can
find no file for it. ``suspicious`` is True when every checked row is
missing and there are at least ``min_for_safety`` of them — the mount-down
signature; the caller should refuse to delete in that case.
``deletable`` (optional) protects rows from removal WITHOUT weakening the
safety gate: a protected row still counts toward ``checked`` and the
all-missing signal (so e.g. a few unverified orphans can't be swept during a
mount outage just because protected rows were filtered out first), but it
never appears in ``orphan_ids``. Default: every missing row is deletable.
"""
orphan_ids = []
checked = 0
missing = 0
for row in rows:
if not str((row.get('file_path') or '')).strip():
continue
checked += 1
if resolve(row) is None:
missing += 1
if deletable is None or deletable(row):
orphan_ids.append(row.get('id'))
suspicious = checked >= min_for_safety and missing == checked
return {'orphan_ids': orphan_ids, 'checked': checked, 'suspicious': suspicious}

128
core/downloads/playlist_folder.py
View file

@ -1,128 +0,0 @@
"""Playlist-folder layout helpers for download analysis and existence checks."""
from __future__ import annotations
import os
from typing import Any, Dict, List, Optional
from core.downloads.file_finder import AUDIO_EXTENSIONS
from core.imports.paths import (
_get_config_manager,
docker_resolve_path,
get_file_path_from_template,
sanitize_filename,
)
def _first_artist_name(artists: Any) -> str:
if not artists:
return ''
first = artists[0]
if isinstance(first, dict):
return str(first.get('name', '') or '').strip()
return str(first).strip()
def candidate_playlist_folder_paths(
playlist_name: str,
artist: str,
title: str,
) -> List[str]:
"""Return absolute candidate paths for a track in playlist-folder layout."""
if not playlist_name or not title:
return []
artist_name = (artist or 'Unknown Artist').strip()
track_name = title.strip()
transfer_dir = docker_resolve_path(
_get_config_manager().get('soulseek.transfer_path', './Transfer')
)
template_context = {
'artist': artist_name,
'albumartist': artist_name,
'album': track_name,
'title': track_name,
'playlist_name': playlist_name,
'track_number': 1,
'disc_number': 1,
'year': '',
'quality': '',
'albumtype': '',
'_artists_list': [{'name': artist_name}],
}
candidates: List[str] = []
folder_path, filename_base = get_file_path_from_template(template_context, 'playlist_path')
if folder_path and filename_base:
base = os.path.join(transfer_dir, folder_path, filename_base)
for ext in AUDIO_EXTENSIONS:
candidates.append(base + ext)
else:
playlist_name_sanitized = sanitize_filename(playlist_name)
playlist_dir = os.path.join(transfer_dir, playlist_name_sanitized)
artist_name_sanitized = sanitize_filename(artist_name)
track_name_sanitized = sanitize_filename(track_name)
stem = f'{artist_name_sanitized} - {track_name_sanitized}'
for ext in AUDIO_EXTENSIONS:
candidates.append(os.path.join(playlist_dir, stem + ext))
return candidates
def track_exists_in_playlist_folder(
playlist_name: str,
artist: str,
title: str,
) -> bool:
"""Return True if any audio file exists at the playlist-folder path for this track."""
for path in candidate_playlist_folder_paths(playlist_name, artist, title):
if os.path.isfile(path):
return True
return False
def track_exists_in_playlist_folder_from_track_data(
playlist_name: str,
track_data: Dict[str, Any],
) -> bool:
"""Check playlist-folder existence using Spotify-style track payload."""
title = track_data.get('name', '') or track_data.get('track_name', '')
artist = _first_artist_name(track_data.get('artists', []))
if not artist:
artist = str(track_data.get('artist_name', '') or '').strip()
return track_exists_in_playlist_folder(playlist_name, artist, title)
def resolve_playlist_folder_mode_for_batch(
db: Any,
*,
playlist_id: str,
playlist_name: str,
batch_playlist_folder_mode: bool,
profile_id: int = 1,
source: str = 'spotify',
) -> tuple[bool, str]:
"""Merge batch flag with persisted mirrored-playlist preference."""
if batch_playlist_folder_mode:
return True, playlist_name
if not hasattr(db, 'resolve_mirrored_playlist'):
return False, playlist_name
# Pass the batch's source so numeric upstream ids (e.g. Deezer) resolve by
# source instead of colliding with the mirrored-playlists primary key.
mirrored = db.resolve_mirrored_playlist(
playlist_id, profile_id=profile_id, default_source=source or 'spotify'
)
if mirrored and mirrored.get('organize_by_playlist'):
return True, mirrored.get('name') or playlist_name
return False, playlist_name
__all__ = [
'candidate_playlist_folder_paths',
'track_exists_in_playlist_folder',
'track_exists_in_playlist_folder_from_track_data',
'resolve_playlist_folder_mode_for_batch',
]

187
core/downloads/post_processing.py
View file

@ -19,11 +19,9 @@ from __future__ import annotations
import logging
import os
import shutil
import time
import traceback
from dataclasses import dataclass
from difflib import SequenceMatcher
from pathlib import Path
from typing import Any, Callable, Optional
@ -36,7 +34,7 @@ from core.imports.context import (
get_import_original_search,
normalize_import_context,
)
from core.imports.filename import extract_track_number_from_filename, parse_filename_metadata
from core.imports.filename import extract_track_number_from_filename
from core.metadata import enrichment as metadata_enrichment
from core.runtime_state import (
download_tasks,
@ -48,85 +46,6 @@ from core.runtime_state import (
logger = logging.getLogger(__name__)
_AUDIO_EXTENSIONS = {
'.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
'.opus', '.ogg', '.m4a', '.aac', '.mp3', '.wma',
}
def _is_audio_file(path: str) -> bool:
return Path(str(path or '')).suffix.lower() in _AUDIO_EXTENSIONS
def _reject_non_audio_found_file(found_file: Optional[str], file_location: Optional[str]) -> tuple[Optional[str], Optional[str]]:
if found_file and not _is_audio_file(found_file):
logger.warning(
"[Post-Processing] Ignoring non-audio candidate found during file search: %s",
found_file,
)
return None, None
return found_file, file_location
def _normalize_match_text(value: str) -> str:
return ''.join(ch.lower() for ch in str(value or '') if ch.isalnum())
def _release_audio_match_score(path: str, expected_title: str, expected_artist: str) -> float:
parsed = parse_filename_metadata(path)
parsed_title = parsed.get('title') or Path(path).stem
parsed_artist = parsed.get('artist') or ''
expected_title_norm = _normalize_match_text(expected_title)
parsed_title_norm = _normalize_match_text(parsed_title)
if expected_title_norm and (
expected_title_norm in parsed_title_norm or parsed_title_norm in expected_title_norm
):
title_score = 1.0
else:
title_score = SequenceMatcher(
None,
expected_title_norm,
parsed_title_norm,
).ratio()
if expected_artist and parsed_artist:
artist_score = SequenceMatcher(
None,
_normalize_match_text(expected_artist),
_normalize_match_text(parsed_artist),
).ratio()
return (title_score * 0.75) + (artist_score * 0.25)
return title_score
def _track_title_from_task(track_info: Any, context: Optional[dict]) -> str:
if isinstance(track_info, dict):
title = track_info.get('name') or track_info.get('title')
if title:
return str(title)
return get_import_clean_title(context or {}, default='')
def _copy_release_audio_to_transfer(source_path: str, transfer_dir: str) -> Optional[str]:
try:
src = Path(source_path)
if not src.exists() or not src.is_file():
return None
dest_dir = Path(transfer_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
dest = dest_dir / src.name
if dest.exists():
stem, suffix = src.stem, src.suffix
counter = 1
while dest.exists():
dest = dest_dir / f"{stem}_release_{counter}{suffix}"
counter += 1
shutil.copy2(src, dest)
return str(dest)
except Exception as exc:
logger.warning("[Post-Processing] Could not copy release audio to transfer: %s", exc)
return None
@dataclass
class PostProcessDeps:
"""Bundle of dependencies the post-processing worker needs.
@ -171,21 +90,6 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
logger.info(f"[Post-Processing] Task {task_id} already completed by stream processor, skipping verification")
return
# RACE GUARD: the monitor sets status -> 'post_processing' immediately
# before submitting this worker. If the status is now anything else, the
# browser-poll post-processor already took ownership of this task — e.g.
# it quarantined the file and requeued the next-best candidate (status
# -> 'searching', source identity cleared). Bail WITHOUT marking failed
# or notifying batch completion: otherwise we clobber that in-flight
# retry with a bogus "missing file or source information" failure while a
# parallel attempt is importing the song.
if task['status'] != 'post_processing':
logger.info(
f"[Post-Processing] Task {task_id} no longer in 'post_processing' "
f"(now '{task['status']}') — another path took over, skipping"
)
return
# Extract file information for verification
track_info = task.get('track_info', {})
task_filename = task.get('filename') or track_info.get('filename')
@ -286,71 +190,14 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
# CRITICAL FIX: For YouTube downloads, the filename in task is 'id||title' (metadata),
# but the actual file on disk is 'Title.mp3'. We must ask the client for the real path.
# Torrent/usenet also use opaque "url||display" filenames. Their completed
# job may contain a full release, so choose the best matching audio file
# and copy it to transfer before importing instead of moving the client's
# original completed download.
if (task.get('username') == 'youtube' or '||' in str(task_filename)) and not found_file:
logger.info(f"[Post-Processing] Detected engine-backed download task: {task_id}")
logger.info(f"[Post-Processing] Detected YouTube download task: {task_id}")
try:
# Query the download orchestrator for the status which contains the real file path
# CRITICAL FIX: Use the actual download_id designated by the client, not the internal task_id
actual_download_id = task.get('download_id') or task_id
status = deps.run_async(deps.download_orchestrator.get_download_status(actual_download_id))
if status and task.get('username') in ('torrent', 'usenet'):
audio_files = list(getattr(status, 'audio_files', None) or [])
if not audio_files and getattr(status, 'file_path', None):
audio_files = [status.file_path]
expected_title = _track_title_from_task(track_info, context)
expected_artist = ''
if isinstance(track_info, dict):
artists = track_info.get('artists', [])
if isinstance(artists, list) and artists:
first_artist = artists[0]
expected_artist = first_artist.get('name', '') if isinstance(first_artist, dict) else str(first_artist)
elif isinstance(artists, str):
expected_artist = artists
if not expected_artist and context:
artist_ctx = get_import_context_artist(context)
expected_artist = artist_ctx.get('name', '') if isinstance(artist_ctx, dict) else ''
scored_files = [
(_release_audio_match_score(path, expected_title, expected_artist), path)
for path in audio_files
if _is_audio_file(path)
]
scored_files.sort(reverse=True)
if scored_files:
best_score, best_path = scored_files[0]
logger.info(
"[Post-Processing] Best %s release file for '%s': %s (score %.2f)",
task.get('username'), expected_title, best_path, best_score,
)
if best_score >= 0.80:
copied_path = _copy_release_audio_to_transfer(best_path, transfer_dir)
if copied_path:
found_file = copied_path
file_location = 'download'
logger.info(
"[Post-Processing] Copied matched %s release file to transfer: %s",
task.get('username'), copied_path,
)
else:
logger.warning(
"[Post-Processing] No %s release file met match threshold for '%s' (best %.2f)",
task.get('username'), expected_title, best_score,
)
if not found_file:
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['status'] = 'failed'
download_tasks[task_id]['error_message'] = (
f"No matching audio file found in {task.get('username')} release"
)
deps.on_download_completed(batch_id, task_id, False)
return
if status and status.file_path and not found_file and task.get('username') not in ('torrent', 'usenet'):
if status and status.file_path:
real_path = status.file_path
if os.path.exists(real_path):
# Determine if it's in download or transfer directory
@ -380,12 +227,11 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
logger.info(f"[Post-Processing] Resolved actual YouTube filename: {found_file} (Location: {file_location})")
else:
logger.warning(f"[Post-Processing] Engine status reported path but file missing: {real_path}")
logger.warning(f"[Post-Processing] YouTube status reported path but file missing: {real_path}")
else:
if not found_file:
logger.warning(f"[Post-Processing] Engine status returned no file_path for task {task_id}")
logger.warning(f"[Post-Processing] YouTube status returned no file_path for task {task_id}")
except Exception as e:
logger.error(f"[Post-Processing] Failed to retrieve engine task status: {e}")
logger.error(f"[Post-Processing] Failed to retrieve YouTube task status: {e}")
_file_search_max_retries = 5
for retry_count in range(_file_search_max_retries):
@ -411,7 +257,6 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
# Strategy 1: Try with original filename in both downloads and transfer
logger.info("[Post-Processing] Strategy 1: Searching with original filename...")
found_file, file_location = deps.find_completed_file(download_dir, task_filename, transfer_dir)
found_file, file_location = _reject_non_audio_found_file(found_file, file_location)
if found_file:
logger.info(f"[Post-Processing] Strategy 1 SUCCESS: Found file with original filename in {file_location}: {found_file}")
@ -424,11 +269,7 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
found_result = deps.find_completed_file(transfer_dir, expected_final_filename)
if found_result and found_result[0]:
found_file, file_location = found_result[0], 'transfer'
found_file, file_location = _reject_non_audio_found_file(found_file, file_location)
if found_file:
logger.info(f"[Post-Processing] Strategy 2 SUCCESS: Found file with expected final filename: {found_file}")
else:
logger.error("[Post-Processing] Strategy 2 FAILED: Expected final filename resolved to a non-audio file")
logger.info(f"[Post-Processing] Strategy 2 SUCCESS: Found file with expected final filename: {found_file}")
else:
logger.error("[Post-Processing] Strategy 2 FAILED: Expected final filename not found in transfer folder")
elif not expected_final_filename:
@ -453,19 +294,7 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
logger.error(f"[Post-Processing] Task {task_id} was completed by stream processor - not marking as failed")
return
download_tasks[task_id]['status'] = 'failed'
# slskd reported the transfer complete, but the finder never located
# the file under the configured download folder. Name the folder we
# searched and the two real causes — "still being written" (timing)
# or "SoulSync's download path doesn't match slskd's" (the classic
# standalone config mismatch) — so the user can self-diagnose instead
# of getting an opaque "not found". (Discord: Shdjfgatdif.)
_searched_name = os.path.basename((task_filename or '').replace('\\', '/')) or task_filename
download_tasks[task_id]['error_message'] = (
f"slskd reported '{_searched_name}' downloaded, but it never appeared "
f"under the download folder ({download_dir}) after {_file_search_max_retries} "
f"checks. Either it's still being written, or SoulSync's download path "
f"doesn't match slskd's download directory — they must point at the same folder."
)
download_tasks[task_id]['error_message'] = f'File not found on disk after {_file_search_max_retries} search attempts. Expected: {os.path.basename(task_filename)}'
deps.on_download_completed(batch_id, task_id, False)
return

254
core/downloads/staging.py
View file

@ -34,7 +34,6 @@ from __future__ import annotations
import logging
import os
import re
from dataclasses import dataclass
from typing import Any, Callable
@ -51,101 +50,6 @@ from core.runtime_state import (
logger = logging.getLogger(__name__)
def _coerce_positive_int(value: Any, default: int = 0) -> int:
try:
coerced = int(str(value).split('/')[0])
except (TypeError, ValueError):
return default
return coerced if coerced > 0 else default
def _extract_explicit_track_number(filename: str) -> int:
"""Extract a track number only when the filename visibly carries one."""
basename = os.path.splitext(os.path.basename(str(filename or '')))[0].strip()
if not basename:
return 0
match = re.match(r"^\d[\-\.](\d{1,2})\s*[\-\.]\s*", basename)
if match:
num = int(match.group(1))
return num if 1 <= num <= 99 else 0
match = re.match(r"^\(?(\d{1,3})\)?\s*[\-\.)\]]\s*", basename)
if match:
num = int(match.group(1))
return num if 1 <= num <= 999 else 0
return 0
def _extract_release_filename_title(stem: str) -> str:
"""Extract the trailing title segment from a release-style filename stem.
Slskd album bundles often arrive untagged with stems like
'Artist - Album - 03 - Title' or '03 - Title'. The full stem is too
noisy to fuzzy-match against a clean Spotify title, so when we
detect a bare track-number segment between ' - ' delimiters we
return the trailing segments as an extra match candidate.
Returns '' when no clear track-number signal is present so we don't
accidentally extract tails from real song titles that legitimately
contain ' - ' (e.g. 'Hold Me - Live').
"""
if not stem:
return ''
parts = [p.strip() for p in stem.split(' - ') if p.strip()]
if len(parts) < 2:
return ''
for i, part in enumerate(parts):
if re.fullmatch(r'\d{1,3}', part) and i < len(parts) - 1:
return ' - '.join(parts[i + 1:]).strip()
return ''
def _staging_title_variants(title: Any, normalize: Callable[[str], str]) -> list[str]:
"""Return conservative title variants for release-file matching.
Torrent / usenet release files often encode featured artists in the
filename/title while streaming metadata keeps them in the artist credit.
Strip only feature/bonus noise here; keep version words like remix,
extended, live, acoustic, etc. so distinct recordings do not collapse.
"""
raw = str(title or '').strip()
if not raw:
return []
compacted_separators = re.sub(r'[_]+', ' ', raw)
compacted_separators = re.sub(r'\s+', ' ', compacted_separators).strip()
without_feat = re.sub(
r'\s*[\(\[]\s*(?:feat\.?|ft\.?|featuring)\s+[^)\]]*[\)\]]',
'',
compacted_separators,
flags=re.IGNORECASE,
)
without_feat = re.sub(
r'\s+(?:feat\.?|ft\.?|featuring)\s+.*$',
'',
without_feat,
flags=re.IGNORECASE,
)
without_bonus = re.sub(
r'\s*[\(\[]\s*bonus\s+track\s*[\)\]]',
'',
without_feat,
flags=re.IGNORECASE,
)
release_tail = _extract_release_filename_title(compacted_separators)
variants: list[str] = []
for candidate in (raw, compacted_separators, without_feat, without_bonus, release_tail):
normalized = normalize(candidate)
if normalized and normalized not in variants:
variants.append(normalized)
return variants
@dataclass
class StagingDeps:
"""Bundle of cross-cutting deps the staging-match helper needs."""
@ -154,12 +58,6 @@ class StagingDeps:
get_staging_file_cache: Callable[[str], list]
docker_resolve_path: Callable[[str], str]
post_process_matched_download_with_verification: Callable
# Optional batch-field accessor. Returns ``download_batches[batch_id].get(field)``
# when the runtime state is available, ``None`` otherwise. Injected so
# this module doesn't have to import from runtime_state directly —
# keeps the dep surface explicit and the function unit-testable
# without a live batch dict.
get_batch_field: Callable[[str, str], Any] = None # type: ignore[assignment]
def try_staging_match(task_id, batch_id, track, deps: StagingDeps):
@ -167,71 +65,35 @@ def try_staging_match(task_id, batch_id, track, deps: StagingDeps):
Returns True if a match was found and the file was moved to the transfer folder.
Returns False to fall through to normal download.
Every silent-False exit point logs at INFO with the rejection reason
so #706 / #708-class "track staged but never imported, ends up
re-added to wishlist" loops can be diagnosed from app.log without
a re-instrumentation round-trip. Per-candidate skips log at DEBUG
so the noise stays out of INFO unless explicitly turned up.
"""
track_title = (track.name or '').strip() if hasattr(track, 'name') else ''
track_artist = (track.artists[0] if (hasattr(track, 'artists') and track.artists) else '').strip()
# Compact identifier for the log lines below so a multi-batch
# wishlist run can be greppable per-track.
_track_label = f"'{track_title}' by '{track_artist}'"
staging_files = deps.get_staging_file_cache(batch_id or task_id)
if not staging_files:
logger.info(
"[Staging] No match attempted for %s — staging cache empty for batch %s",
_track_label, batch_id or task_id,
)
return False
track_title = track.name or ''
track_artist = track.artists[0] if track.artists else ''
if not track_title:
logger.info(
"[Staging] No match attempted for task %s — track has empty title",
task_id,
)
return False
from difflib import SequenceMatcher
normalize = deps.matching_engine.normalize_string
norm_title = normalize(track_title)
norm_artist = normalize(track_artist)
title_variants = _staging_title_variants(track_title, normalize) or [norm_title]
best_match = None
best_score = 0.0
# Track per-candidate scoring so the rejection log can show the
# near-miss that DID exist (useful when title-sim is 0.79 — one
# point below the threshold).
candidate_scores: list = []
for sf in staging_files:
sf_title_variants = _staging_title_variants(sf['title'], normalize)
sf_norm_title = normalize(sf['title'])
sf_norm_artist = normalize(sf['artist'])
if not sf_title_variants:
logger.debug(
"[Staging] Skip candidate %s — no usable title variants",
os.path.basename(sf.get('full_path', '?')),
)
if not sf_norm_title:
continue
# Title similarity (primary)
title_sim = max(
SequenceMatcher(None, expected, candidate).ratio()
for expected in title_variants
for candidate in sf_title_variants
)
title_sim = SequenceMatcher(None, norm_title, sf_norm_title).ratio()
if title_sim < 0.80:
logger.debug(
"[Staging] Skip candidate %s — title_sim=%.2f below 0.80 threshold (%s vs %s)",
os.path.basename(sf.get('full_path', '?')),
title_sim, norm_title, '|'.join(sf_title_variants[:3]),
)
candidate_scores.append((sf, title_sim, None, 0.0))
continue
# Artist similarity (secondary)
@ -252,37 +114,12 @@ def try_staging_match(task_id, batch_id, track, deps: StagingDeps):
else:
combined = (title_sim * 0.80) + (artist_sim * 0.20)
candidate_scores.append((sf, title_sim, artist_sim, combined))
if combined > best_score:
best_score = combined
best_match = sf
# Require high confidence to avoid false positives
if not best_match or best_score < 0.75:
# Log the rejection with the best near-miss so we can see why
# the staged files didn't claim this wishlist track. Pre-fix
# this returned False silently and the loop "download album,
# stage files, never claim them, re-add to wishlist" was
# impossible to debug from logs alone.
if candidate_scores:
near_miss = max(candidate_scores, key=lambda c: c[3])
sf, title_sim, artist_sim, combined = near_miss
logger.info(
"[Staging] No match for %s in batch %s — best candidate %s "
"(title_sim=%.2f, artist_sim=%s, combined=%.2f) below 0.75 threshold",
_track_label, batch_id or task_id,
os.path.basename(sf.get('full_path', '?')),
title_sim,
f"{artist_sim:.2f}" if artist_sim is not None else 'n/a',
combined,
)
else:
logger.info(
"[Staging] No match for %s in batch %s — %d staging files "
"but none had usable title variants",
_track_label, batch_id or task_id, len(staging_files),
)
return False
logger.info(f"[Staging] Match found for '{track_title}' by '{track_artist}': "
@ -304,51 +141,20 @@ def try_staging_match(task_id, batch_id, track, deps: StagingDeps):
shutil.copy2(best_match['full_path'], dest_path)
logger.info(f"[Staging] Copied to transfer: {dest_path}")
# Mark task as completed with staging context.
# If the batch was populated by the torrent / usenet album-bundle
# flow, prefer that provenance label over generic 'staging' so the
# download history reflects the real source. The accessor is
# injected via StagingDeps so this module doesn't reach into
# runtime_state directly (see deps.get_batch_field docstring).
_provenance_override = None
if batch_id and deps.get_batch_field is not None:
try:
_provenance_override = deps.get_batch_field(batch_id, 'album_bundle_source')
except Exception as _exc:
logger.debug("get_batch_field failed: %s", _exc)
_provenance_username = _provenance_override or 'staging'
_private_album_bundle_staging = False
if batch_id and deps.get_batch_field is not None:
try:
_private_album_bundle_staging = bool(
deps.get_batch_field(batch_id, 'album_bundle_private_staging')
)
except Exception as _exc:
logger.debug("get_batch_field failed: %s", _exc)
# Mark task as completed with staging context
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['status'] = 'post_processing'
download_tasks[task_id]['filename'] = dest_path
download_tasks[task_id]['username'] = _provenance_username
download_tasks[task_id]['username'] = 'staging'
download_tasks[task_id]['staging_match'] = True
if _private_album_bundle_staging:
try:
os.remove(best_match['full_path'])
logger.debug("[Staging] Removed private album-bundle staging file: %s", best_match['full_path'])
except FileNotFoundError:
pass
except Exception as _exc:
logger.debug("[Staging] Could not remove private album-bundle staging file: %s", _exc)
# Run post-processing (tagging, AcoustID verification, path building)
context_key = f"staging_{task_id}"
with tasks_lock:
track_info = download_tasks.get(task_id, {}).get('track_info', {})
if not isinstance(track_info, dict):
track_info = {}
else:
track_info = dict(track_info)
# Build spotify_artist / spotify_album context so post-processing can apply
# the path template. Without these, _post_process_matched_download returns
@ -417,50 +223,20 @@ def try_staging_match(task_id, batch_id, track, deps: StagingDeps):
)
has_clean_data = bool(track_title and track_artist and track_album_name)
file_track_number = (
_coerce_positive_int(best_match.get('track_number'), 0) or
_extract_explicit_track_number(best_match.get('full_path', ''))
track_number = (
track_info.get('track_number', 0) or
getattr(track, 'track_number', 0) or 0
)
disc_number = (
track_info.get('disc_number', 1) or
getattr(track, 'disc_number', 1) or 1
)
file_disc_number = _coerce_positive_int(best_match.get('disc_number'), 0)
if _private_album_bundle_staging:
track_number = (
file_track_number or
_coerce_positive_int(track_info.get('track_number'), 0) or
_coerce_positive_int(track_info.get('trackNumber'), 0) or
_coerce_positive_int(getattr(track, 'track_number', 0), 0) or
1
)
disc_number = (
file_disc_number or
_coerce_positive_int(track_info.get('disc_number'), 0) or
_coerce_positive_int(track_info.get('discNumber'), 0) or
_coerce_positive_int(getattr(track, 'disc_number', 0), 0) or
1
)
else:
track_number = (
_coerce_positive_int(track_info.get('track_number'), 0) or
_coerce_positive_int(track_info.get('trackNumber'), 0) or
_coerce_positive_int(getattr(track, 'track_number', 0), 0) or
file_track_number
)
disc_number = (
_coerce_positive_int(track_info.get('disc_number'), 0) or
_coerce_positive_int(track_info.get('discNumber'), 0) or
_coerce_positive_int(getattr(track, 'disc_number', 0), 0) or
file_disc_number or
1
)
track_info['track_number'] = track_number
track_info['disc_number'] = disc_number
context = {
'track_info': track_info,
'spotify_artist': spotify_artist_ctx,
'spotify_album': spotify_album_ctx,
'original_search_result': {
'username': _provenance_username,
'filename': best_match.get('full_path', ''),
'title': track_title,
'artist': track_artist,
'spotify_clean_title': track_title,

375
core/downloads/status.py
View file

@ -20,10 +20,10 @@ are passed via `StatusDeps` so the module is web_server-import-free.
from __future__ import annotations
import logging
import threading
import time
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Callable, Optional
from core.runtime_state import (
@ -31,19 +31,8 @@ from core.runtime_state import (
download_tasks,
tasks_lock,
)
from utils.logging_config import get_logger
# Project logger factory so these lines reach app.log (soulsync.* namespace).
logger = get_logger("downloads.status")
# #836 backstop: how long an slskd error state (Rejected/Failed/Errored/TimedOut)
# may persist on a non-manual task before the status formatter gives up on the
# retry monitor and marks it failed. The monitor's own retry window is ~15s
# (3 × 5s); this is well beyond it so a healthy retry always wins, and it only
# fires when the monitor genuinely can't make progress (e.g. a rejected transfer
# with no other source) — which otherwise hangs the task at 'downloading 0%'
# forever and blocks the whole batch from completing.
ERROR_STATE_TERMINAL_GRACE_SECONDS = 60
logger = logging.getLogger(__name__)
def _schedule_completion_callback(deps, batch_id: str, task_id: str, success: bool) -> None:
@ -93,20 +82,13 @@ class StatusDeps:
download_orchestrator: Any = None
run_async: Optional[Callable] = None
on_download_completed: Optional[Callable[[str, str, bool], None]] = None
get_persistent_download_history: Optional[Callable[[int], list[dict]]] = None
# Returns ALL library_history rows with verification_status in
# ('unverified', 'force_imported') — no recency limit, so historical
# entries are never buried by the general history tail cap.
get_unverified_download_history: Optional[Callable[[], list[dict]]] = None
# Streaming sources the engine fallback applies to. Soulseek goes through
# slskd's live_transfers path and must NOT hit the engine fallback.
_STREAMING_SOURCE_NAMES = frozenset((
'youtube', 'tidal', 'qobuz', 'hifi', 'deezer_dl', 'lidarr', 'soundcloud', 'amazon',
'torrent', 'usenet',
))
_RELEASE_SOURCE_NAMES = frozenset(('torrent', 'usenet'))
# Keep these in sync with the engine plugins' state strings.
_ENGINE_FAILURE_STATES = ('Errored', 'Failed', 'Rejected', 'TimedOut', 'Aborted')
@ -154,12 +136,21 @@ def _apply_engine_state_fallback(
Mutates ``task`` in place (status / error_message) the same way the
Soulseek branch does, so the next status poll sees the new state.
Submits post-processing on terminal success. Manual-pick failures
are completed here; automatic failures keep the existing retry path
in charge.
Submits post-processing on terminal success and fires
``on_download_completed`` on terminal failure to free the worker
slot.
"""
if deps.download_orchestrator is None or deps.run_async is None:
return
if task.get('status') in ('completed', 'failed', 'cancelled', 'not_found', 'post_processing'):
return
# Scope this fallback to user-initiated manual picks. Auto attempts
# already flow through the live_transfers_lookup IF branch (the engine
# pre-populates non-Soulseek records via get_all_downloads), and on
# failure the monitor's existing retry path picks the next candidate.
# Marking auto attempts failed here would short-circuit that fallback.
if not task.get('_user_manual_pick'):
return
download_id = task.get('download_id')
if not download_id:
return
@ -167,17 +158,6 @@ def _apply_engine_state_fallback(
username = task.get('username') or ti.get('username')
if username not in _STREAMING_SOURCE_NAMES:
return
manual_pick = bool(task.get('_user_manual_pick'))
if not manual_pick and username not in _RELEASE_SOURCE_NAMES:
return
release_source = username in _RELEASE_SOURCE_NAMES
terminal_status = task.get('status') in ('completed', 'failed', 'cancelled', 'not_found', 'post_processing')
if terminal_status and not (
release_source
and task.get('status') in ('failed', 'not_found')
and download_id
):
return
try:
record = deps.run_async(
@ -209,14 +189,6 @@ def _apply_engine_state_fallback(
return
if any(s in state_str for s in _ENGINE_FAILURE_STATES):
if not manual_pick:
task_status['status'] = task.get('status', 'downloading')
task_status['progress'] = _engine_progress_pct(record)
logger.info(
"[Engine Fallback] Task %s engine reports '%s' for auto %s attempt; leaving retry handling to monitor",
task_id, state_str, username,
)
return
if task['status'] != 'failed':
task['status'] = 'failed'
err = getattr(record, 'error_message', None) or getattr(record, 'error', None) or ''
@ -275,19 +247,8 @@ def build_batch_status_data(batch_id: str, batch: dict, live_transfers_lookup: d
"playlist_id": batch.get('playlist_id'), # Include playlist_id for rehydration
"playlist_name": batch.get('playlist_name'), # Include playlist_name for reference
}
album_bundle = _build_album_bundle_status(batch)
if album_bundle:
response_data['album_bundle'] = album_bundle
if response_data["phase"] in ('analysis', 'queued'):
# Surface analysis_progress for both phases — ``queued`` rows
# haven't started analysis yet (processed=0) but the UI still
# needs ``total`` so it can render "Queued — N tracks". The
# master worker flips phase from ``queued`` to ``analysis`` on
# entry (see ``core/downloads/master.py:328``); the wishlist
# submission sites pre-populate ``analysis_total`` so the
# queued state isn't shape-mismatched against the analysis
# state for downstream consumers.
if response_data["phase"] == 'analysis':
response_data['analysis_progress'] = {
'total': batch.get('analysis_total', 0),
'processed': batch.get('analysis_processed', 0),
@ -351,29 +312,11 @@ def build_batch_status_data(batch_id: str, batch: dict, live_transfers_lookup: d
'ui_state': task.get('ui_state', 'normal'), # normal|cancelling|cancelled
'playlist_id': task.get('playlist_id'), # For V2 system identification
'error_message': task.get('error_message'), # Surface failure reasons to UI
'quarantine_entry_id': task.get('quarantine_entry_id'),
'has_candidates': bool(task.get('cached_candidates')), # Whether search found results (for clickable review)
# 'verified' / 'unverified' / 'force_imported' — set by the
# import pipeline once post-processing finishes.
'verification_status': task.get('verification_status'),
# "2/5" while the quarantine-retry engine walks candidates.
'retry_info': task.get('retry_info'),
'retry_trigger': task.get('retry_trigger'),
}
_ti = task.get('track_info') if isinstance(task.get('track_info'), dict) else {}
task_filename = task.get('filename') or _ti.get('filename')
task_username = task.get('username') or _ti.get('username')
if (
task_username in _RELEASE_SOURCE_NAMES
and task['status'] not in ['completed', 'cancelled', 'post_processing']
and task.get('download_id')
):
_apply_engine_state_fallback(
task_id, task, task_status, batch_id, deps,
)
batch_tasks.append(task_status)
continue
if task_filename and task_username:
lookup_key = deps.make_context_key(task_username, task_filename)
@ -413,59 +356,17 @@ def build_batch_status_data(batch_id: str, batch: dict, live_transfers_lookup: d
# release the lock.
_schedule_completion_callback(deps, batch_id, task_id, False)
else:
# Normally the retry monitor picks up an errored state and
# retries within ~15s. But if it can't make progress — e.g. an
# slskd transfer rejected with no other source — the task would
# otherwise sit at 'downloading 0%' forever, spam an ERROR every
# poll, AND block its batch from ever completing (#836: a rejected
# wishlist track, or rejected tracks in an album download).
#
# Backstop: measure how long the ERROR state has persisted (not
# how long the task has downloaded, so a slow-but-healthy transfer
# isn't failed). Once it exceeds the monitor's retry window with no
# resolution, mark the task failed so the worker frees and the
# batch can finish. A working retry transitions the task out of the
# error state first, clearing the timer below — so the healthy path
# never hits this.
# A monitor retry transitions the task (newer
# status_change_time), which restarts the window so each
# error EPISODE gets a fresh grace. If the monitor never
# transitions it (the stuck case), the window keeps growing.
err_since = task.get('_error_state_since')
if err_since is None or task.get('status_change_time', 0) > err_since:
task['_error_state_since'] = err_since = current_time
task.pop('_error_state_logged', None)
error_age = current_time - err_since
# UNIFIED ERROR HANDLING: Let monitor handle errors for consistency
# Monitor will detect errored state and trigger retry within 5 seconds
logger.error(f"Task {task_id} API shows error state: {state_str} - letting monitor handle retry")
if error_age > ERROR_STATE_TERMINAL_GRACE_SECONDS:
err_msg = live_info.get('errorMessage') or live_info.get('error') or ''
task['status'] = 'failed'
task['error_message'] = (
str(err_msg) if err_msg
else f'Download failed (state: {state_str})'
)
task_status['status'] = 'failed'
task_status['error_message'] = task['error_message']
logger.warning(
f"Task {task_id} stuck in error state '{state_str}' for "
f"{error_age:.0f}s with no retry progress — marking failed (#836)"
)
_schedule_completion_callback(deps, batch_id, task_id, False)
# Keep task in current status (downloading/queued) so monitor can detect error
# Don't mark as failed here - let the unified retry system handle it
if task['status'] in ['searching', 'downloading', 'queued']:
task_status['status'] = task['status'] # Keep current status for monitor
else:
# Within the retry window — keep current status so the monitor
# can act. Log once per episode, not every poll, to stop the
# 2-second ERROR spam the reporter saw.
if not task.get('_error_state_logged'):
logger.warning(
f"Task {task_id} API shows error state: {state_str} "
f"- letting monitor handle retry"
)
task['_error_state_logged'] = True
if task['status'] in ['searching', 'downloading', 'queued']:
task_status['status'] = task['status'] # Keep current status for monitor
else:
task_status['status'] = 'downloading' # Default to downloading for error detection
task['status'] = 'downloading'
task_status['status'] = 'downloading' # Default to downloading for error detection
task['status'] = 'downloading'
elif 'Completed' in state_str or 'Succeeded' in state_str:
# Verify bytes actually transferred before trusting state string
expected_size = live_info.get('size', 0)
@ -546,16 +447,7 @@ def build_single_batch_status(batch_id: str, deps: StatusDeps) -> tuple[Optional
def build_batched_status(requested_batch_ids: list, deps: StatusDeps) -> dict:
"""For /api/download_status/batch. Returns the full response dict (always 200).
When a requested batch carries a ``wishlist_run_id`` (Phase 1c.2.1
per-album split), the response merges in every sibling sub-batch
of the same run via ``merge_wishlist_run_status``. The merged view
lands keyed under the originally-requested ``batch_id`` so the
frontend modal (which polls one batch id) sees every sibling's
tasks + progress without needing to know about the split."""
from core.downloads.wishlist_aggregator import merge_wishlist_run_status
"""For /api/download_status/batch. Returns the full response dict (always 200)."""
live_transfers_lookup = deps.get_cached_transfer_data()
response: dict[str, Any] = {"batches": {}}
@ -568,49 +460,11 @@ def build_batched_status(requested_batch_ids: list, deps: StatusDeps) -> dict:
else:
target_batches = download_batches.copy()
# Pre-index sibling batch ids by wishlist_run_id so the per-
# batch loop below can find them in O(1). Snapshot under the
# held lock; subsequent dict mutations don't matter for this
# build.
run_id_to_batch_ids: dict[str, list[str]] = {}
for bid, batch_row in download_batches.items():
run_id = (batch_row or {}).get('wishlist_run_id') if isinstance(batch_row, dict) else None
if run_id:
run_id_to_batch_ids.setdefault(str(run_id), []).append(bid)
for batch_id, batch in target_batches.items():
try:
primary_status = build_batch_status_data(
response["batches"][batch_id] = build_batch_status_data(
batch_id, batch, live_transfers_lookup, deps,
)
# Wishlist-run merge — kicks in only when this batch
# has a run_id AND at least one sibling exists. Falls
# through to legacy single-batch shape otherwise.
run_id = batch.get('wishlist_run_id') if isinstance(batch, dict) else None
sibling_ids = run_id_to_batch_ids.get(str(run_id), []) if run_id else []
if run_id and len(sibling_ids) > 1:
sibling_statuses = []
for sib_id in sibling_ids:
if sib_id == batch_id:
continue
sib_batch = download_batches.get(sib_id)
if not isinstance(sib_batch, dict):
continue
try:
sibling_statuses.append(
build_batch_status_data(
sib_id, sib_batch, live_transfers_lookup, deps,
)
)
except Exception as sib_err:
logger.warning(
f"[Wishlist Run] Sibling status build failed for {sib_id}: {sib_err}"
)
merged = merge_wishlist_run_status(primary_status, sibling_statuses)
response["batches"][batch_id] = merged
else:
response["batches"][batch_id] = primary_status
except Exception as batch_error:
logger.error(f"Error processing batch {batch_id}: {batch_error}")
response["batches"][batch_id] = {"error": str(batch_error)}
@ -655,70 +509,6 @@ _STATUS_PRIORITY = {
'not_found': 6, 'failed': 7, 'cancelled': 8,
}
_PERSISTENT_HISTORY_TAIL_LIMIT = 50
def _normalize_identity_part(value: Any) -> str:
return str(value or '').strip().casefold()
def _download_identity(title: Any, artist: Any, album: Any) -> tuple[str, str, str]:
return (
_normalize_identity_part(title),
_normalize_identity_part(artist),
_normalize_identity_part(album),
)
def _history_timestamp(value: Any) -> float:
if not value:
return 0.0
if isinstance(value, (int, float)):
return float(value)
text = str(value).strip()
if not text:
return 0.0
try:
# SQLite CURRENT_TIMESTAMP uses "YYYY-MM-DD HH:MM:SS".
return datetime.fromisoformat(text.replace('Z', '+00:00')).timestamp()
except ValueError:
try:
return datetime.strptime(text[:19], '%Y-%m-%d %H:%M:%S').timestamp()
except ValueError:
return 0.0
def _build_history_download_item(entry: dict) -> dict:
history_id = entry.get('id') or entry.get('history_id') or ''
title = entry.get('title') or entry.get('source_track_title') or ''
artist = entry.get('artist_name') or entry.get('source_artist') or ''
album = entry.get('album_name') or ''
created_at = entry.get('created_at') or entry.get('completed_at') or ''
source = entry.get('download_source') or ''
return {
'task_id': f'history-{history_id}' if history_id else f'history-{title}-{created_at}',
'title': title,
'artist': artist,
'album': album,
'artwork': entry.get('thumb_url') or '',
'status': 'completed',
'progress': 100,
'error': None,
'batch_id': '',
'batch_name': source,
'batch_source': source,
'playlist_id': '',
'track_index': 0,
'batch_total': 1,
'timestamp': _history_timestamp(created_at),
'created_at': created_at,
'priority': _STATUS_PRIORITY['completed'],
'quality': entry.get('quality') or '',
'file_path': entry.get('file_path') or '',
'verification_status': entry.get('verification_status'),
'is_persistent_history': True,
}
def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
"""Flat list of every task across batches, sorted active-first then by recency.
@ -726,7 +516,6 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
Powers /api/downloads/all for the centralized Downloads page.
"""
items = []
live_identities = set()
with tasks_lock:
for task_id, task in download_tasks.items():
track_info = task.get('track_info') or {}
@ -773,7 +562,6 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
artwork = images[0].get('url', '') if isinstance(images[0], dict) else str(images[0])
status = task.get('status', 'queued')
live_identities.add(_download_identity(title, artist, album))
# Determine download progress percentage
progress = 0
if status == 'completed':
@ -799,16 +587,6 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
'status': status,
'progress': progress,
'error': task.get('error_message'),
'verification_status': task.get('verification_status'),
# library_history row id (set at import) so the Unverified review
# queue can act on a still-live completed task before it becomes
# a persistent-history row.
'history_id': task.get('history_id'),
# Real probed audio quality (mutagen-read from the actual file),
# surfaced so the Downloads page can show what was downloaded.
'quality': task.get('quality') or '',
'retry_info': task.get('retry_info'),
'retry_trigger': task.get('retry_trigger'),
'batch_id': batch_id,
'batch_name': batch.get('playlist_name') or batch.get('album_name') or '',
'batch_source': batch.get('source_page') or batch.get('initiated_from') or '',
@ -820,63 +598,9 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
'batch_total': len(batch.get('queue', [])),
'timestamp': task.get('status_change_time', 0),
'priority': _STATUS_PRIORITY.get(status, 9),
'is_persistent_history': False,
})
# --- Unverified history (unconditional, no limit) ---
# Always load every library_history row that still needs human confirmation
# (verification_status IN ('unverified', 'force_imported')). This is NOT
# gated on len(items) < limit so that historical entries from past batches
# are visible even during a large active batch that would otherwise exhaust
# the limit before the history tail is read. Dedup against live tasks by
# identity so a track currently in post-processing isn't shown twice.
if deps.get_unverified_download_history is not None:
try:
unverified_entries = deps.get_unverified_download_history() or []
except Exception as exc:
logger.debug("[Downloads] unverified history lookup failed: %s", exc)
unverified_entries = []
for entry in unverified_entries:
item = _build_history_download_item(entry)
identity = _download_identity(item.get('title'), item.get('artist'), item.get('album'))
if identity in live_identities:
continue
items.append(item)
live_identities.add(identity)
# --- General recent-history tail (capped, recency-ordered) ---
# Fills in the completed/verified tail so the full Downloads list looks
# populated. Gated on len(items) < limit so a busy batch doesn't trigger
# an extra DB round-trip when we're already at capacity.
if deps.get_persistent_download_history is not None and len(items) < limit:
history_limit = min(limit - len(items), _PERSISTENT_HISTORY_TAIL_LIMIT)
try:
history_entries = deps.get_persistent_download_history(history_limit) or []
except Exception as exc:
logger.debug("[Downloads] persistent history lookup failed: %s", exc)
history_entries = []
appended_history = 0
for entry in history_entries:
if len(items) >= limit or appended_history >= history_limit:
break
item = _build_history_download_item(entry)
identity = _download_identity(item.get('title'), item.get('artist'), item.get('album'))
if identity in live_identities:
continue
items.append(item)
live_identities.add(identity)
appended_history += 1
# Sort: active first (by priority), then by timestamp desc within each group.
# NOTE: the array order is presentation-only — the Downloads page filters
# client-side per tab. What matters is that EVERY live task is present: an
# earlier `items[:limit]` truncation (active-first) starved completed/failed/
# unverified rows off the end during a busy batch, so those tabs stayed empty
# until the batch drained. `limit` now bounds only the persistent-history
# tail (handled above); live in-memory tasks are always returned in full
# (they're already bounded by the 5-min cleanup automation).
# Sort: active first (by priority), then by timestamp desc within each group
items.sort(key=lambda x: (x['priority'], -x['timestamp']))
# Build batch summaries for the batch context panel
@ -885,7 +609,7 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
for bid, batch in download_batches.items():
queue = batch.get('queue', [])
statuses = [download_tasks[tid]['status'] for tid in queue if tid in download_tasks]
summary = {
batch_summaries.append({
'batch_id': bid,
'playlist_id': batch.get('playlist_id', ''),
'batch_name': batch.get('playlist_name') or batch.get('album_name') or '',
@ -896,51 +620,12 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
'failed': sum(1 for s in statuses if s in ('failed', 'not_found', 'cancelled')),
'active': sum(1 for s in statuses if s in ('downloading', 'searching', 'post_processing')),
'queued': sum(1 for s in statuses if s in ('queued', 'pending')),
}
album_bundle = _build_album_bundle_status(batch)
if album_bundle:
summary['album_bundle'] = album_bundle
batch_summaries.append(summary)
})
return {
'success': True,
'downloads': items,
'downloads': items[:limit],
'total': len(items),
'batches': batch_summaries,
'timestamp': time.time(),
}
def _build_album_bundle_status(batch: dict) -> dict:
"""Return public batch-level status for torrent/Usenet album-bundle work."""
state = batch.get('album_bundle_state')
if not state:
return {}
status = {
'state': state,
'source': batch.get('album_bundle_source'),
'release': batch.get('album_bundle_release'),
'progress': batch.get('album_bundle_progress'),
'progress_percent': _album_bundle_progress_percent(
batch.get('album_bundle_progress')
),
'speed': batch.get('album_bundle_speed'),
'downloaded': batch.get('album_bundle_downloaded'),
'size': batch.get('album_bundle_size'),
'seeders': batch.get('album_bundle_seeders'),
'grabs': batch.get('album_bundle_grabs'),
'count': batch.get('album_bundle_count'),
}
return {key: value for key, value in status.items() if value is not None}
def _album_bundle_progress_percent(value: Any) -> int:
try:
progress = float(value)
except (TypeError, ValueError):
return 0
if progress <= 1:
progress *= 100
return max(0, min(100, int(round(progress))))

333
core/downloads/task_worker.py
View file

@ -19,161 +19,16 @@ a large web_server.py helper that will get its own lift in subsequent PRs.
from __future__ import annotations
import logging
import re
import traceback
from dataclasses import dataclass
from typing import Any, Callable, Optional
from core.runtime_state import download_batches, download_tasks, tasks_lock
from core.runtime_state import download_tasks, tasks_lock
from core.spotify_client import Track as SpotifyTrack
from utils.logging_config import get_logger
# Must live under the soulsync.* namespace — handlers only attach there. The
# old bare getLogger(__name__) ("core.downloads.task_worker") had no handler,
# so the entire [Modal Worker] story — search queries, retry walks, candidate
# decisions — never reached app.log.
logger = get_logger("downloads.task_worker")
def _resolve_worker_source(username):
"""Logical source bucket for a candidate's username (Soulseek peers all
collapse to 'soulseek'; streaming sources keep their name). Mirrors the
monitor's resolver — imported lazily to avoid an import cycle."""
try:
from core.downloads.monitor import _resolve_download_source
return _resolve_download_source(username)
except Exception:
return 'soulseek'
def _cand_user_file(candidate):
"""Read (username, filename) from a candidate that may be a TrackResult
object or a plain dict (tests / cached raw rows)."""
if isinstance(candidate, dict):
return candidate.get('username'), candidate.get('filename')
return getattr(candidate, 'username', None), getattr(candidate, 'filename', None)
def _candidate_ordering():
"""Return ``(quality_first, targets)`` for the active search mode + toggle.
The candidate walk is ordered by the user's profile quality rank
(best→worst) instead of confidence-first when EITHER:
- best-quality search mode is active (always quality-first), OR
- priority mode and the ``rank_candidates_by_quality`` toggle is on
(opt-in; default off keeps the byte-for-byte confidence-first walk).
Quality-first ordering also makes the version-mismatch force-import pick
the highest-quality candidate, because that fallback accepts the
first-tried (= best-ordered) quarantined entry.
Fails closed to confidence-first ordering on any error so a profile/DB
hiccup never blocks a download. See
docs/superpowers/specs/2026-06-14-best-quality-search-mode-design.md.
"""
try:
from core.quality.selection import (
load_search_mode,
load_profile_targets,
load_rank_candidates_by_quality,
)
if load_search_mode() == 'best_quality' or load_rank_candidates_by_quality():
targets, _ = load_profile_targets()
return True, targets
except Exception as exc:
logger.debug("[Modal Worker] quality ordering unavailable: %s", exc)
return False, None
def _try_cached_candidates(task_id, batch_id, track, deps):
"""Quarantine-retry fast path: attempt the already-found candidates before
re-searching anything.
When a verified-bad file is re-queued, the connection was fine (the file
downloaded, it was just the wrong/broken content) — so the next-best pick is
almost always already sitting in ``cached_candidates``. Walk those (skipping
sources already tried or budget-exhausted) and hand them to the normal
download path. Returns True if a download was started; False to fall through
to a fresh search (which only happens for a not-yet-searched source).
"""
with tasks_lock:
task = download_tasks.get(task_id)
if not task:
return False
cached = list(task.get('cached_candidates') or [])
used = set(task.get('used_sources') or ())
exhausted = {str(s).lower() for s in (task.get('exhausted_download_sources') or ())}
remaining = []
for c in cached:
uname, fname = _cand_user_file(c)
if not uname or not fname:
continue
if f"{uname}_{fname}" in used:
continue
if _resolve_worker_source(uname).lower() in exhausted:
continue
remaining.append(c)
if not remaining:
return False
logger.info(
f"[Modal Worker] Quarantine retry: trying {len(remaining)} cached "
f"candidate(s) before re-searching (task {task_id})"
)
_qf, _qt = _candidate_ordering()
return deps.attempt_download_with_candidates(
task_id, remaining, track, batch_id, quality_first=_qf, quality_targets=_qt,
)
def _private_album_bundle_staging_miss_reason(batch_id: Optional[str], deps: Any) -> Optional[str]:
"""Return a user-facing miss reason when per-track search should stop.
Torrent / usenet album batches first download one private staged release,
then each track claims the matching staged file. If that claim fails after
the release is already staged, falling through to the normal per-track
search only retries release-level sources N times and can keep re-adding
the same torrent/NZB. For those two sources we treat the staged release as
authoritative for this pass.
Soulseek is deliberately NOT short-circuited. A Soulseek album bundle stages
whichever single folder scored best, and ``album_bundle_partial`` only
reflects whether the files found IN that folder downloaded — not whether the
folder actually contained every track the album needs. So a track the album
needs but that wasn't in the chosen folder would otherwise be marked
not_found with no fallback (#743). Unlike torrent/usenet, Soulseek per-track
search is a genuine per-file network search — it doesn't re-add a release —
so letting these misses fall through to the normal per-track flow (and, in
hybrid mode, onward to the next source) is correct and cheap.
"""
if not batch_id:
return None
batch = download_batches.get(batch_id)
if not isinstance(batch, dict):
return None
source = (batch.get('album_bundle_source') or '').lower()
mode = (getattr(deps.download_orchestrator, 'mode', '') or '').lower()
hybrid_first = ''
if mode == 'hybrid':
order = getattr(deps.download_orchestrator, 'hybrid_order', None) or []
if order:
hybrid_first = str(order[0] or '').lower()
else:
hybrid_first = str(getattr(deps.download_orchestrator, 'hybrid_primary', '') or '').lower()
if (
batch.get('album_bundle_private_staging')
and batch.get('album_bundle_state') == 'staged'
and not batch.get('album_bundle_partial')
and source in ('torrent', 'usenet')
and (mode == source or (mode == 'hybrid' and hybrid_first == source))
):
return f'Track was not found in the staged {source} album release'
return None
logger = logging.getLogger(__name__)
@dataclass
@ -189,7 +44,6 @@ class TaskWorkerDeps:
attempt_download_with_candidates: Callable # (task_id, candidates, track, batch_id) -> bool
on_download_completed: Callable # (batch_id, task_id, success) -> None
recover_worker_slot: Callable # (batch_id, task_id) -> None
try_version_mismatch_fallback: Optional[Callable] = None # (title, artist, task_id, batch_id) -> bool
def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorkerDeps) -> None:
@ -274,21 +128,6 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
# === STAGING CHECK: Check staging folder for existing file before searching ===
if deps.try_staging_match(task_id, batch_id, track):
return
staging_miss_reason = _private_album_bundle_staging_miss_reason(batch_id, deps)
if staging_miss_reason:
logger.warning(
"[Modal Worker] %s for '%s'; skipping redundant per-track %s search",
staging_miss_reason,
track.name,
getattr(deps.download_orchestrator, 'mode', 'release-source'),
)
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['status'] = 'not_found'
download_tasks[task_id]['error_message'] = staging_miss_reason
if batch_id:
deps.on_download_completed(batch_id, task_id, False)
return
# Initialize task state tracking (like GUI's parallel_search_tracking)
with tasks_lock:
@ -304,51 +143,10 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
download_tasks[task_id]['used_sources'] = set()
# Else: keep existing used_sources to avoid retrying same failed hosts
# Cached-first quarantine retry. The monitor sets ``_quarantine_retry``
# when a verified-bad file is re-queued; in that case we walk the
# already-found candidates before re-searching (the connection was fine,
# just the content was wrong). A NON-quarantine entry (fresh download, or
# the monitor's dead-connection/stuck retry) instead starts a new search
# generation: clear the searched-source memory so each source can be
# searched fresh again.
with tasks_lock:
_t = download_tasks.get(task_id, {})
is_quarantine_retry = bool(_t.pop('_quarantine_retry', False))
if not is_quarantine_retry:
_t.pop('searched_queries', None)
if is_quarantine_retry and _try_cached_candidates(task_id, batch_id, track, deps):
with tasks_lock:
used_filename = download_tasks.get(task_id, {}).get('filename')
used_username = download_tasks.get(task_id, {}).get('username')
if used_filename and used_username:
deps.store_batch_source(batch_id, used_username, used_filename)
return
# 1. Generate multiple search queries (like GUI's generate_smart_search_queries)
artist_name = track.artists[0] if track.artists else None
track_name = track.name
release_queries = []
try:
_download_mode = (getattr(deps.download_orchestrator, 'mode', '') or '').lower()
_track_album = (getattr(track, 'album', '') or '').strip()
_track_title = (getattr(track, 'name', '') or '').strip()
_track_artists = list(getattr(track, 'artists', []) or [])
_first_artist = _track_artists[0] if _track_artists else ''
_primary_artist = (
(_first_artist.get('name', '') if isinstance(_first_artist, dict) else str(_first_artist))
or ''
).strip()
if (
_download_mode in ('torrent', 'usenet')
and _primary_artist
and _track_album
and _track_album.lower() not in ('unknown album', _track_title.lower())
):
release_queries.append(f"{_primary_artist} {_track_album}".strip())
except Exception as _release_query_exc:
logger.debug("[Modal Worker] release query hint failed: %s", _release_query_exc)
# Start with matching engine queries
search_queries = deps.matching_engine.generate_download_queries(track)
@ -377,14 +175,8 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
if cleaned_name and cleaned_name.lower() != track_name.lower():
legacy_queries.append(cleaned_name.strip())
# Combine enhanced queries with legacy fallbacks.
#
# Torrent / usenet can use full album releases as a fallback for
# single-track requests, but trying the album release first makes
# playlist batches download whole albums before checking whether a
# track-shaped release exists. Keep release queries last so singles
# stay light when the indexer has a direct result.
all_queries = search_queries + legacy_queries + release_queries
# Combine enhanced queries with legacy fallbacks
all_queries = search_queries + legacy_queries
# Remove duplicates while preserving order
unique_queries = []
@ -395,45 +187,12 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
seen.add(query.lower())
search_queries = unique_queries
# Expose the query count so the quarantine-retry budget (exhaustive mode)
# can size each source's budget as query_count × retries_per_query.
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['query_count'] = len(search_queries)
logger.info(f"[Modal Worker] Generated {len(search_queries)} smart search queries for '{track.name}': {search_queries}")
logger.info(f"[Modal Worker] About to start search loop for task {task_id} (track: '{track.name}')")
# Best-quality search mode: the orchestrator already pooled candidates
# across every source for each query, so order the candidate walk by the
# user's profile quality rank (best→worst). Computed once per task.
_best_quality, _quality_targets = _candidate_ordering()
# 2. Sequential Query Search (matches GUI's start_search_worker_parallel logic)
search_diagnostics = [] # Track what happened per query for detailed error messages
all_raw_results = [] # Collect raw results across queries for candidate review modal
# Sources whose per-source quarantine-retry budget is spent (exhaustive
# mode). The monitor sets this when a source gives up; we exclude those
# sources from the hybrid search so the chain falls through to the next
# source instead of re-fetching the same exhausted one (e.g. Soulseek
# keeps returning fresh wrong peers — once its budget is gone, switch to
# HiFi/Tidal/…). See monitor.requeue_quarantined_task_for_retry.
#
# On a quarantine retry we do NOT exclude a source just because it was
# searched once: the first run only ran ONE query before starting a
# download, so the later queries (e.g. "artist + album") have never hit
# that source yet and may surface the correct upload. Instead we remember
# which QUERIES already ran (``searched_queries``) and skip re-running
# only those — their candidates are walked via the cached-first path
# above. The not-yet-searched queries still search the same source, so
# every query is exhausted per source before the chain switches sources.
# Fresh / dead-connection runs cleared searched_queries above, so they
# search everything again.
with tasks_lock:
_t = download_tasks.get(task_id, {})
_exhausted_sources = [str(s) for s in (_t.get('exhausted_download_sources') or ())]
_searched_queries = (
set(_t.get('searched_queries') or ()) if is_quarantine_retry else set()
)
for query_index, query in enumerate(search_queries):
# Cancellation check before each query
with tasks_lock:
@ -446,49 +205,12 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
return
download_tasks[task_id]['current_query_index'] = query_index
# Cached-first: a query already run last generation has its candidates
# sitting in cache (walked above) — re-searching it is the wasteful
# repeat the cached-first design removes. Skip it; the not-yet-run
# queries below still search this source.
if is_quarantine_retry and query in _searched_queries:
logger.debug(
f"[Modal Worker] Skipping already-searched query '{query}' "
f"(candidates served from cache) for task {task_id}"
)
continue
logger.debug(f"[Modal Worker] Query {query_index + 1}/{len(search_queries)}: '{query}'")
logger.debug(f"About to call soulseek search for task {task_id}")
try:
# Hybrid + album-context batches must skip torrent / usenet during
# the per-track loop — they're release-level sources, can't match
# individual tracks meaningfully, and album-bundle handling only
# fires in single-source mode (see core/downloads/master.py). The
# exclusion lets the hybrid chain fall through to per-track-
# compatible sources (soulseek / streaming) instead of attempting
# N redundant Prowlarr searches that all download the same album
# torrent and rely on the auto-import sweep to clean up.
_exclude_for_hybrid_album = None
try:
_batch_is_album = False
if batch_id:
from core.runtime_state import download_batches as _db
_b = _db.get(batch_id)
if isinstance(_b, dict):
_batch_is_album = bool(_b.get('is_album_download'))
if _batch_is_album and getattr(deps.download_orchestrator, 'mode', '') == 'hybrid':
_exclude_for_hybrid_album = ['torrent', 'usenet']
except Exception as _exc_filter_err:
logger.debug("[Modal Worker] album-source-exclusion check failed: %s", _exc_filter_err)
# Fold in budget-exhausted sources (per-source quarantine retry).
_exclude_sources = list(_exhausted_sources)
if _exclude_for_hybrid_album:
_exclude_sources.extend(_exclude_for_hybrid_album)
# Perform search with timeout
tracks_result, _ = deps.run_async(deps.download_orchestrator.search(
query, timeout=30, exclude_sources=_exclude_sources or None,
))
tracks_result, _ = deps.run_async(deps.download_orchestrator.search(query, timeout=30))
logger.debug(f"Search completed for task {task_id}, got {len(tracks_result) if tracks_result else 0} results")
# CRITICAL: Check cancellation immediately after search returns
@ -496,16 +218,6 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
if task_id not in download_tasks:
logger.info(f"[Modal Worker] Task {task_id} was deleted after search returned")
return
# Remember this query ran so a later quarantine retry skips
# re-searching it (its candidates are walked via cached-first).
# Recorded regardless of result count: re-running a query is
# deterministic, so a query that returned nothing won't return
# anything new next time either.
_sq = download_tasks[task_id].get('searched_queries')
if not isinstance(_sq, set):
_sq = set()
_sq.add(query)
download_tasks[task_id]['searched_queries'] = _sq
if download_tasks[task_id]['status'] == 'cancelled':
logger.warning(f"[Modal Worker] Task {task_id} cancelled after search returned - ignoring results")
# Don't call _on_download_completed for cancelled tasks as it can stop monitoring
@ -528,16 +240,11 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
logger.warning(f"[Modal Worker] Task {task_id} cancelled before processing candidates")
# Don't call _on_download_completed for cancelled tasks as it can stop monitoring
return
# Store candidates for retry fallback (like GUI). A
# later quarantine retry walks these via cached-first
# and skips re-searching this query (searched_queries).
# Store candidates for retry fallback (like GUI)
download_tasks[task_id]['cached_candidates'] = candidates
# Try to download with these candidates
success = deps.attempt_download_with_candidates(
task_id, candidates, track, batch_id,
quality_first=_best_quality, quality_targets=_quality_targets,
)
success = deps.attempt_download_with_candidates(task_id, candidates, track, batch_id)
if success:
# Download initiated successfully - let the download monitoring system handle completion
if batch_id:
@ -571,10 +278,7 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
# === HYBRID FALLBACK: If primary source failed, try remaining sources directly ===
# The orchestrator's hybrid search stops at the first source with results, even if
# those results all fail quality filtering. Try remaining sources individually.
#
# Best-quality mode already searched EVERY source per query (the pool), so this
# block would only re-search the same sources — skip it there.
if not _best_quality and getattr(deps.download_orchestrator, 'mode', '') == 'hybrid':
if getattr(deps.download_orchestrator, 'mode', '') == 'hybrid':
try:
orch = deps.download_orchestrator
hybrid_order = getattr(orch, 'hybrid_order', None) or []
@ -598,12 +302,7 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
# (which was definitely tried). If the first was skipped (unconfigured),
# the orchestrator would have tried the second — but trying it again is
# harmless (streaming sources return fast).
_exhausted_lower = {s.lower() for s in _exhausted_sources}
remaining_sources = [
s for s in hybrid_order[1:]
if s in source_clients and source_clients[s]
and s.lower() not in _exhausted_lower
]
remaining_sources = [s for s in hybrid_order[1:] if s in source_clients and source_clients[s]]
if remaining_sources:
logger.warning(f"[Hybrid Fallback] Primary source had no valid matches. Trying fallback sources: {remaining_sources}")
@ -622,9 +321,6 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
fb_candidates = deps.get_valid_candidates(fb_results, track, fb_query)
if fb_candidates:
logger.warning(f"[Hybrid Fallback] {fallback_source} found {len(fb_candidates)} valid candidates!")
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['cached_candidates'] = fb_candidates
success = deps.attempt_download_with_candidates(task_id, fb_candidates, track, batch_id)
if success:
return
@ -639,15 +335,6 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
# If we get here, all search queries and hybrid fallbacks failed
logger.warning(f"[Modal Worker] No valid candidates found for '{track.name}' after trying all {len(search_queries)} queries.")
# Last-resort: quarantine retry with no new candidates — the retry search
# exhausted all sources. If the setting is enabled, accept the best
# already-quarantined candidate rather than leaving the track missing.
if is_quarantine_retry and deps.try_version_mismatch_fallback:
_fallback_artist = track.artists[0] if track.artists else ''
if deps.try_version_mismatch_fallback(track.name, _fallback_artist, task_id, batch_id):
return # fallback re-dispatched; batch completion handled by reprocess thread
with tasks_lock:
if task_id in download_tasks:
download_tasks[task_id]['status'] = 'not_found'

106
core/downloads/track_detail.py
View file

@ -1,106 +0,0 @@
"""Assemble a per-track detail view for the download-modal "track detail" modal.
Merges a live download task with its ``library_history`` record (the same data
the Download History cards render) into one dict the frontend modal consumes.
Kept pure + importable so the merge + status classification are unit-tested
without Flask or the DB; the web endpoint is thin glue around build_track_detail.
"""
from __future__ import annotations
from typing import Any, Dict, Optional
# error_message substrings that mean "quarantined" (file recoverable) rather
# than a plain failure. Mirrors the download-modal status renderer.
_QUARANTINE_KEYWORDS = (
'integrity check failed',
'bit depth filter',
'verification failed',
'quarantin',
)
def classify_status_kind(status: str, error_message: str = '') -> str:
"""Map a raw task status to a UI 'kind' that drives the modal layout:
completed / quarantined / failed / not_found / in_progress.
"""
s = (status or '').lower()
if s == 'completed':
return 'completed'
if s in ('failed', 'cancelled'):
em = (error_message or '').lower()
if any(k in em for k in _QUARANTINE_KEYWORDS):
return 'quarantined'
return 'failed'
if s == 'not_found':
return 'not_found'
return 'in_progress'
def _first_artist(track_info: Dict[str, Any]) -> str:
artists = track_info.get('artists') or []
if isinstance(artists, list) and artists:
first = artists[0]
if isinstance(first, dict):
return (first.get('name') or '').strip()
return str(first).strip()
return (track_info.get('artist') or track_info.get('artist_name') or '').strip()
def _album_name(track_info: Dict[str, Any]) -> str:
album = track_info.get('album')
if isinstance(album, dict):
return (album.get('name') or '').strip()
return (album or '').strip() if isinstance(album, str) else ''
def build_track_detail(task: Dict[str, Any], history: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
"""Merge a download task (+ optional library_history row) into one detail dict.
The task supplies live status/source/reason/quarantine id; the history row
(when found) supplies the durable provenance — final file path, quality,
AcoustID verdict, source, and the expected-vs-downloaded comparison.
"""
ti = task.get('track_info') if isinstance(task.get('track_info'), dict) else {}
status = task.get('status', '') or ''
kind = classify_status_kind(status, task.get('error_message', '') or '')
detail: Dict[str, Any] = {
'task_id': task.get('task_id') or task.get('id') or '',
'status': status,
'status_kind': kind,
'title': (ti.get('name') or '').strip(),
'artist': _first_artist(ti),
'album': _album_name(ti),
'source': (task.get('username') or '').strip(),
'reason': (task.get('error_message') or '').strip(),
'quarantine_entry_id': task.get('quarantine_entry_id') or '',
'file_path': (task.get('filename') or '').strip(),
'quality': '',
'acoustid_result': '',
'thumb_url': '',
'expected': {},
'downloaded': {},
}
if history:
detail['file_path'] = (history.get('file_path') or detail['file_path'])
detail['quality'] = history.get('quality') or ''
detail['acoustid_result'] = history.get('acoustid_result') or ''
detail['source'] = history.get('download_source') or detail['source']
detail['thumb_url'] = history.get('thumb_url') or ''
detail['downloaded'] = {
'title': history.get('title') or '',
'artist': history.get('artist_name') or '',
'album': history.get('album_name') or '',
}
detail['expected'] = {
'title': history.get('source_track_title') or '',
'artist': history.get('source_artist') or '',
}
# Fall back to history values when the task had none.
detail['title'] = detail['title'] or detail['downloaded']['title']
detail['artist'] = detail['artist'] or detail['downloaded']['artist']
detail['album'] = detail['album'] or detail['downloaded']['album']
return detail

136
core/downloads/track_link.py
View file

@ -1,136 +0,0 @@
"""Recognize a pasted streaming-source track link in the manual download
search (#813).
A user pastes e.g. ``https://tidal.com/track/434945950/u`` instead of typing a
query, to grab the exact version. We only recognize sources that download by
track ID (Tidal, Qobuz) — the manual search then resolves the link to that
track and runs the source's own search so the result is a normal, downloadable
candidate (no hand-built download encoding).
Pure + import-safe: parsing only, no network.
"""
from __future__ import annotations
import re
from typing import Any, List, Optional, Tuple
from urllib.parse import urlparse
def linked_track_id(track: Any) -> str:
"""The source track id stamped on a search result, read from
``_source_metadata['track_id']`` — the field every ID-downloadable source
(Tidal, Qobuz) records. Empty string when absent. ``TrackResult`` has no
top-level ``id``, so callers must NOT use ``getattr(t, 'id')`` (that always
missed and left the pasted-link bubble a silent no-op — #932)."""
meta = getattr(track, '_source_metadata', None)
if not isinstance(meta, dict):
return ''
return str(meta.get('track_id') or '')
def bubble_linked_track_first(tracks: List[Any], link_track_id: str) -> List[Any]:
"""Float the result whose source id matches a pasted link to the top so the
user sees the EXACT track they linked, not a fuzzy text-search lookalike
(#813/#932). Stable + a graceful no-op when no result carries the id."""
if not link_track_id or not tracks:
return tracks
target = str(link_track_id)
return sorted(tracks, key=lambda t: linked_track_id(t) != target)
def inject_linked_track_first(
tracks: List[Any], linked_result: Any, link_track_id: str
) -> List[Any]:
"""Put the EXACT linked track first.
When ``linked_result`` is the track fetched directly by id, prepend it and
drop any search duplicate of it — so an obscure track a text search never
surfaced is still present and downloadable (#932). When it's None (the source
can't fetch one), fall back to bubbling a matching search result. Pure."""
if not link_track_id:
return tracks
target = str(link_track_id)
if linked_result is not None:
return [linked_result] + [t for t in tracks if linked_track_id(t) != target]
return bubble_linked_track_first(tracks, target)
# host substring → download source id. Only ID-downloadable streaming sources.
_HOSTS = (
('tidal.com', 'tidal'),
('qobuz.com', 'qobuz'),
)
def parse_download_track_link(raw: str) -> Optional[Tuple[str, str]]:
"""Parse a pasted Tidal/Qobuz track URL into ``(source, track_id)``.
Returns None when the input isn't a recognized track link (so the caller
falls back to a normal text search). Handles the common URL shapes:
``tidal.com/track/<id>[/u]``, ``listen.tidal.com/track/<id>``,
``tidal.com/browse/track/<id>``, ``open.qobuz.com/track/<id>``,
``play.qobuz.com/track/<id>`` — with or without the scheme.
"""
raw = (raw or '').strip()
if not raw:
return None
lowered = raw.lower()
if '://' not in raw and not any(h in lowered for h, _ in _HOSTS):
return None # not even a URL we care about
url = raw if '://' in raw else f'https://{raw}'
parsed = urlparse(url)
host = (parsed.netloc or '').lower()
source = next((sid for h, sid in _HOSTS if h in host), None)
if not source:
return None
segs = [s for s in (parsed.path or '').split('/') if s]
for i, seg in enumerate(segs):
if seg.lower() == 'track' and i + 1 < len(segs):
m = re.match(r'(\d+)', segs[i + 1]) # id may carry a slug/suffix
if m:
return (source, m.group(1))
return None
def _first_artist_name(value: Any) -> str:
"""First artist name from a list of {'name': ...}/strings, or a single
{'name': ...}/string."""
if isinstance(value, list):
value = value[0] if value else None
if isinstance(value, dict):
return str(value.get('name') or '')
return str(value or '')
def query_from_track_payload(source: str, raw: Any) -> Optional[str]:
"""Build a clean ``"artist title"`` search query from a source ``get_track``
payload — pure, so the per-source shape parsing is unit-testable without a
live client.
- Tidal: attributes dict (``title`` + optional ``version`` + maybe
``artists``/``artist``). The version is appended so a remix link searches
for the remix.
- Qobuz: track dict (``title`` + ``performer``/``album.artist``).
"""
if not isinstance(raw, dict):
return None
title = (raw.get('title') or '').strip()
artist = ''
if source == 'tidal':
version = (raw.get('version') or '').strip()
if version and version.lower() not in title.lower():
title = f"{title} ({version})" if title else version
artist = _first_artist_name(raw.get('artists') or raw.get('artist'))
elif source == 'qobuz':
artist = _first_artist_name(raw.get('performer'))
if not artist:
album = raw.get('album') if isinstance(raw.get('album'), dict) else {}
artist = _first_artist_name(album.get('artist'))
query = f"{artist} {title}".strip()
return query or (title or None)

226
core/downloads/track_metadata_backfill.py
View file

@ -1,226 +0,0 @@
"""Track-position resolution + album-context hydration helper.
Lifted out of ``core/downloads/candidates.py`` to break a quiet
regression. The pre-extract code fetched detailed track data from
Spotify only when ``track_number`` was missing — the same API call
*also* backfilled the lean ``spotify_album_context`` (release_date,
total_tracks, image_url) but only as a side-effect of the
track_number branch. When a wishlist row carried a poisoned
``track_number=1`` (older payload helpers defaulted missing values
to 1), the conditional short-circuited, the API call never fired,
and the album context stayed lean — producing folders without a
year subfolder for residual per-track wishlist downloads.
The fix splits the two concerns: ``track_number`` resolution
follows its precedence chain (track_info → track object → API),
but album hydration runs whenever ``spotify_album_context`` is
missing any of release_date / total_tracks regardless of whether
``track_number`` was already known. A single API call still serves
both — the side-effect coupling is gone but the network cost
isn't paid twice.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, NamedTuple, Optional
logger = logging.getLogger(__name__)
class ResolvedTrackMetadata(NamedTuple):
"""Result of ``hydrate_download_metadata``.
``track_number`` is ``None`` when every source — track_info,
track object, API — failed to produce a positive integer. The
caller (candidates.py) treats ``None`` as "fall back to the
setdefault(0)" path so the existing 0-floor sentinel behaviour
is preserved.
"""
track_number: Optional[int]
disc_number: int
source: str # 'track_info' | 'track_object' | 'api' | 'none'
def _positive_int(value: Any) -> Optional[int]:
if isinstance(value, bool):
return None
if isinstance(value, int):
return value if value > 0 else None
if isinstance(value, str) and value.isdigit():
coerced = int(value)
return coerced if coerced > 0 else None
return None
def _album_is_lean(album_context: Any) -> bool:
"""An album context is "lean" when it lacks release_date OR
total_tracks. Both fields are required downstream:
``release_date`` drives the year folder, ``total_tracks`` drives
the TRCK tag denominator + the album-vs-single classification
in ``build_import_album_info``."""
if not isinstance(album_context, dict):
return True
if not album_context.get('release_date'):
return True
if not album_context.get('total_tracks'):
return True
return False
def _backfill_album_context(
album_context: Dict[str, Any], detailed_track: Dict[str, Any]
) -> None:
"""Copy missing fields from ``detailed_track['album']`` into the
in-place album_context dict. Existing values are preserved —
never overwritten — because the caller's context may carry
fields the API doesn't (e.g. enhanced search shapes can include
artists arrays the bare track endpoint omits)."""
dt_album = detailed_track.get('album')
if not isinstance(dt_album, dict) or not isinstance(album_context, dict):
return
for key in ('release_date', 'album_type', 'total_tracks', 'id'):
if not album_context.get(key) and dt_album.get(key):
album_context[key] = dt_album[key]
if not album_context.get('image_url'):
images = dt_album.get('images')
if isinstance(images, list) and images:
first = images[0]
if isinstance(first, dict) and first.get('url'):
album_context['image_url'] = first['url']
# Placeholder album ids used when no real source album id is known — never queryable.
_SENTINEL_ALBUM_IDS = {'explicit_album', 'from_sync_modal', ''}
def backfill_album_context_from_source(
album_context: Dict[str, Any],
primary_source: Optional[str],
get_album_for_source_fn: Any,
) -> bool:
"""Hydrate a lean album context from the user's PRIMARY metadata source (#915).
Post-processing's only album backfill (:func:`hydrate_download_metadata`) goes through
``spotify_client.get_track_details`` — Spotify-only. An iTunes/Deezer-primary user's
download therefore kept a lean context (no ``release_date``), so the path dropped the
``$year`` and the date defaulted to ``YYYY-01-01`` — until they ran a Reorganize, which
reads the full album from the PRIMARY source. This closes that gap by doing the same:
fetch the full album from the primary source and backfill, so a download's pathing/tags
match what a later reorganize would produce.
``get_album_for_source_fn(source, album_id)`` is injected (the real one is
``core.metadata.album_tracks.get_album_for_source``) so this stays pure + testable.
No-op when: the context is already complete; the primary source is spotify (the existing
track-details path covers it); or no real source album id is present. Returns True when
it filled anything. Never raises — a backfill failure must not break a download.
"""
if not isinstance(album_context, dict) or not _album_is_lean(album_context):
return False
if not primary_source or primary_source == 'spotify':
return False
album_id = album_context.get('id')
if not album_id or str(album_id) in _SENTINEL_ALBUM_IDS:
return False
try:
album = get_album_for_source_fn(primary_source, str(album_id))
except Exception as e: # noqa: BLE001 — defensive: never let backfill break a download
logger.warning("[Context] primary-source (%s) album backfill failed: %s", primary_source, e)
return False
if not isinstance(album, dict):
return False
before = album_context.get('release_date')
_backfill_album_context(album_context, {'album': album})
if album_context.get('release_date') and album_context.get('release_date') != before:
logger.info(
"[Context] Hydrated lean album context from primary source %s "
"(release_date=%r, total_tracks=%r)",
primary_source, album_context.get('release_date'), album_context.get('total_tracks'),
)
return True
def hydrate_download_metadata(
track: Any,
track_info: Any,
spotify_album_context: Dict[str, Any],
spotify_client: Any,
) -> ResolvedTrackMetadata:
"""Resolve track position and hydrate lean album context.
Steps:
1. ``track_info['track_number']`` when positive
2. ``track.track_number`` when truthy
3. ``spotify_client.get_track_details(track.id)`` — fires when
EITHER track_number unresolved OR album_context lean. The
same call serves both concerns; only one round-trip per task.
``spotify_album_context`` is mutated in place when API returns
richer data. Returns the resolved track_number / disc_number /
source. ``track_number=None`` signals "no usable value found";
the caller decides whether to floor it to 1 or leave 0.
"""
ti = track_info if isinstance(track_info, dict) else {}
# Step 1: track_info top-level — wishlist + frontend payloads.
tn = _positive_int(ti.get('track_number'))
if tn is not None:
dn = _positive_int(ti.get('disc_number')) or 1
source = 'track_info'
else:
tn = None
dn = 1
source = 'none'
# Step 2: track object — Spotify Track dataclass from search.
if tn is None:
track_tn = getattr(track, 'track_number', None)
coerced = _positive_int(track_tn)
if coerced is not None:
tn = coerced
dn = _positive_int(getattr(track, 'disc_number', None)) or 1
source = 'track_object'
needs_api_for_tn = tn is None
needs_api_for_album = _album_is_lean(spotify_album_context)
track_id = getattr(track, 'id', None)
if (needs_api_for_tn or needs_api_for_album) and track_id:
try:
detailed = spotify_client.get_track_details(track_id)
except Exception as e: # noqa: BLE001 — defensive log + continue
logger.error("[Context] API track details failed: %s", e)
detailed = None
if isinstance(detailed, dict):
if needs_api_for_tn:
api_tn = _positive_int(detailed.get('track_number'))
if api_tn is not None:
tn = api_tn
dn = _positive_int(detailed.get('disc_number')) or 1
source = 'api'
logger.info(
"[Context] Resolved track_number=%d disc_number=%d from API",
tn, dn,
)
if needs_api_for_album and isinstance(spotify_album_context, dict):
_backfill_album_context(spotify_album_context, detailed)
logger.info(
"[Context] Backfilled album context from API "
"(release_date=%r, total_tracks=%r)",
spotify_album_context.get('release_date'),
spotify_album_context.get('total_tracks'),
)
return ResolvedTrackMetadata(track_number=tn, disc_number=dn, source=source)
__all__ = [
'ResolvedTrackMetadata',
'hydrate_download_metadata',
'backfill_album_context_from_source',
]

114
core/downloads/validation.py
View file

@ -9,7 +9,6 @@ import logging
import re
from config.settings import config_manager
from core.imports.file_integrity import resolve_duration_tolerance
logger = logging.getLogger(__name__)
@ -25,20 +24,6 @@ def init(matching_engine_obj, download_orchestrator_obj):
download_orchestrator = download_orchestrator_obj
def _torrent_usenet_artist_is_fallback(result):
"""True when a release result has no parsed artist, only indexer filler."""
if getattr(result, 'username', None) not in ('torrent', 'usenet'):
return False
artist = (getattr(result, 'artist', None) or '').strip()
if not artist:
return True
metadata = getattr(result, '_source_metadata', None) or {}
indexer = str(metadata.get('indexer') or '').strip()
if artist.lower() in ('torrent', 'usenet'):
return True
return bool(indexer and artist.lower() == indexer.lower())
def filter_soundcloud_previews(results, expected_track):
"""Drop SoundCloud preview snippets so they never reach the cache,
the modal, or the auto-download attempt.
@ -76,24 +61,6 @@ def filter_soundcloud_previews(results, expected_track):
return [r for r in results if not _is_preview(r)]
def _duration_tolerance_seconds(expected_duration_ms):
override = resolve_duration_tolerance(
config_manager.get('post_processing.duration_tolerance_seconds', 0)
)
if override is not None:
return override
expected_seconds = expected_duration_ms / 1000.0
return 5.0 if expected_seconds > 600.0 else 3.0
def _duration_mismatch_exceeds_integrity_tolerance(expected_duration_ms, candidate_duration_ms):
if not expected_duration_ms or not candidate_duration_ms:
return False
tolerance = _duration_tolerance_seconds(expected_duration_ms)
drift = abs((candidate_duration_ms / 1000.0) - (expected_duration_ms / 1000.0))
return drift > tolerance
def get_valid_candidates(results, spotify_track, query):
"""
This function is a direct port from sync.py. It scores and filters
@ -111,12 +78,8 @@ def get_valid_candidates(results, spotify_track, query):
return []
# Streaming sources (YouTube, Tidal, Qobuz, HiFi, Deezer, SoundCloud) return structured API results
# with proper artist/title metadata — score using the same matching engine as Soulseek.
# Torrent / usenet results also belong here: their filename field is a download URL, not
# a slskd-style ``Artist/Album/Track.flac`` path, so the Soulseek matcher would extract
# garbage segments from it. Routing them through the streaming path means score_track_match
# reads ``r.title`` and ``r.artist`` directly (which the torrent/usenet projections pre-fill).
_streaming_sources = ("youtube", "tidal", "qobuz", "hifi", "deezer_dl", "soundcloud", "amazon", "torrent", "usenet")
# with proper artist/title metadata — score using the same matching engine as Soulseek
_streaming_sources = ("youtube", "tidal", "qobuz", "hifi", "deezer_dl", "soundcloud", "amazon")
if results[0].username in _streaming_sources:
source_label = results[0].username.replace('_dl', '').title()
expected_artists = spotify_track.artists if spotify_track else []
@ -135,70 +98,17 @@ def get_valid_candidates(results, spotify_track, query):
expected_is_version = any(kw in expected_title_lower for kw in _version_keywords)
scored = []
_strict_duration_sources = {'tidal', 'qobuz', 'hifi', 'deezer_dl', 'amazon'}
for r in results:
if (
r.username in _strict_duration_sources
and _duration_mismatch_exceeds_integrity_tolerance(expected_duration, r.duration or 0)
):
logger.info(
"[%s] Rejecting candidate due to duration mismatch before download: "
"expected %.1fs, candidate %.1fs",
source_label,
expected_duration / 1000.0,
(r.duration or 0) / 1000.0,
)
continue
# Score using matching engine's generic scorer (same weights as Soulseek).
# Torrent/usenet release projections sometimes only have the indexer name
# in the artist field when a title did not parse as "Artist - Release".
# Treat that as unknown artist, not as a real mismatch.
has_only_fallback_artist = _torrent_usenet_artist_is_fallback(r)
candidate_artists = [] if has_only_fallback_artist else ([r.artist] if r.artist else [])
# Score using matching engine's generic scorer (same weights as Soulseek)
confidence, match_type = matching_engine.score_track_match(
source_title=expected_title,
source_artists=expected_artists,
source_duration_ms=expected_duration,
candidate_title=r.title or '',
candidate_artists=candidate_artists,
candidate_artists=[r.artist] if r.artist else [],
candidate_duration_ms=r.duration or 0,
)
# Album-name fallback for torrent / usenet per-track results.
#
# When this fallback runs: hybrid mode + non-album batch (single
# track wishlist / playlist of singles). Album-context batches
# never reach here — the album-bundle gate in
# core/downloads/album_bundle_dispatch.py engages the bulk-
# download flow in single-source mode, and the hybrid chain
# filter in core/downloads/task_worker.py strips torrent /
# usenet from album batches in hybrid mode. What's left is the
# single-track-in-hybrid case where a user is searching for one
# track and the only torrent / usenet result is the album that
# contains it.
#
# Without this fallback, "Luther (with SZA)" against a
# candidate titled "GNX (2024) [FLAC]" scores ~0 on track-title
# alone — even though the album torrent does in fact contain
# the wanted track. Scoring the candidate title against the
# wanted track's ALBUM name and taking the max gives album-
# level releases a fair shot. The Auto-Import sweep then picks
# the right file out of the downloaded album folder.
expected_album = getattr(spotify_track, 'album', None) if spotify_track else None
if r.username in ('torrent', 'usenet') and expected_album:
album_conf, _ = matching_engine.score_track_match(
source_title=expected_album,
source_artists=expected_artists,
source_duration_ms=0, # albums don't have one duration
candidate_title=r.title or '',
candidate_artists=candidate_artists,
candidate_duration_ms=0,
)
if album_conf > confidence:
confidence = album_conf
match_type = 'album_release'
# Version detection penalty — reject live/remix/acoustic when expecting original
r_title_lower = (r.title or '').lower()
is_wrong_version = False
@ -219,10 +129,8 @@ def get_valid_candidates(results, spotify_track, query):
# Artist gate — streaming APIs (Tidal/Qobuz/HiFi/Deezer) have reliable metadata,
# so "My Will" by "B. Starr" should never match expected "B小町".
# YouTube stays excluded because video-title parsing is unreliable.
# Torrent/usenet must also pass this gate so title-only matches
# from the wrong artist do not get downloaded.
if r.username != 'youtube' and not has_only_fallback_artist:
# Skip for YouTube — artist is parsed from video titles and often unreliable.
if r.username != 'youtube':
from difflib import SequenceMatcher
import re as _re
_cand_artist_raw = r.artist or ''
@ -254,16 +162,6 @@ def get_valid_candidates(results, spotify_track, query):
# so falling to SequenceMatcher means the strings are genuinely
# different. 0.5 gives a safer buffer without blocking real
# matches that would have scored above 0.85 anyway.
if r.username in ('torrent', 'usenet') and _best_artist < 0.5:
logger.info(
"[%s] Rejecting candidate due to artist mismatch: "
"expected=%s candidate=%r title=%r",
source_label,
list(expected_artists),
_cand_artist_raw,
r.title or '',
)
continue
if _best_artist < 0.5 and confidence < 0.85:
continue

198
core/downloads/wishlist_aggregator.py
View file

@ -1,198 +0,0 @@
"""Merge sibling download_batches statuses into one view for the
wishlist-run model.
When the wishlist runs are split into per-album sub-batches
(Phase 1c.2.1), the frontend modal polls the ORIGINAL batch id
allocated by ``start_manual_wishlist_download_batch`` /
``process_wishlist_automatically``. That batch id is now just one
sibling among N. Without merging, the modal goes blank after the
first sibling finishes because subsequent siblings live under
fresh batch ids the modal never learned about.
This module is the merge layer: pure function, no IO, no runtime
state. ``build_batched_status`` in ``core/downloads/status.py``
calls into it when a requested batch has ``wishlist_run_id`` set
and at least one sibling exists.
Design notes:
- ``track_index`` re-indexed to a global 0..N-1 across the merged
results so the modal's ``data-track-index`` DOM keys don't
collide between siblings (each sibling locally starts at 0).
Tasks reference their analysis result via track_index, so the
remap is applied to tasks too.
- ``task_id`` is a uuid per task — no collision concern across
siblings.
- Phase aggregation surfaces the MOST advanced live phase so the
modal renders task rows the moment any sibling reaches the task
stage. The earlier "least-complete" rule made the modal appear
frozen during parallel album_downloading because the task table
is phase-gated to ``downloading``/``complete``/``error``. Sticky
``error`` so failures don't get hidden by a running sibling.
- ``album_bundle`` is picked from whichever sibling currently has
an active bundle download — gives the user a useful progress
bar even when the primary sibling is past its bundle stage.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
_ACTIVE_BUNDLE_STATES = frozenset({
'searching',
'downloading',
'downloading_release',
'staging',
})
def _aggregate_phases(phases: List[str]) -> str:
"""Pick the merged phase for a multi-sibling wishlist run.
Surfaces the MOST advanced live phase across the run so the
modal renders task progress the moment any sibling reaches
the task stage. Earlier ("least complete") aggregation hid
downloading tasks behind the bundle progress UI when any
sibling was still in ``album_downloading``, so the modal
appeared frozen for the entire duration of the slowest
sibling's bundle phase.
Rules:
- ``error`` is sticky — if any sibling errored, surface error
so the user notices the failure even mid-run.
- ``downloading``/``complete`` (the phases that produce a
task table on the modal side) WIN over earlier phases.
Any sibling at this stage means "show tasks now". All-
complete returns ``complete``; mixed complete + downloading
stays ``downloading`` so the modal keeps polling.
- ``album_downloading`` wins over ``analysis`` only when no
sibling has reached the task stage.
- ``analysis`` is the last resort.
"""
phases = [p for p in phases if p]
if not phases:
return 'unknown'
if 'error' in phases:
return 'error'
if all(p == 'complete' for p in phases):
return 'complete'
if any(p in ('downloading', 'complete') for p in phases):
return 'downloading'
if 'album_downloading' in phases:
return 'album_downloading'
if 'analysis' in phases:
return 'analysis'
return phases[0]
def _pick_active_album_bundle(statuses: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
"""Return the album_bundle of whichever sibling is currently
staging or downloading. Falls back to the first non-empty
bundle when nothing is active (so a completed bundle still
shows up vs. a totally empty progress bar)."""
fallback = None
for s in statuses:
bundle = s.get('album_bundle')
if not bundle:
continue
if fallback is None:
fallback = bundle
state = (bundle.get('state') or '').lower()
if state in _ACTIVE_BUNDLE_STATES:
return bundle
return fallback
def merge_wishlist_run_status(
primary: Dict[str, Any],
siblings: List[Dict[str, Any]],
) -> Dict[str, Any]:
"""Return a status dict that merges ``siblings`` into ``primary``.
Empty ``siblings`` is the legacy single-batch case — primary
is returned unchanged.
The returned dict has the same shape as a single-batch status
response from ``build_batch_status_data`` so the frontend
modal needs no changes to consume it. Tracks and tasks are
re-indexed globally; phase + progress + active_count
aggregated across the run.
"""
if not siblings:
return primary
all_statuses = [primary] + list(siblings)
# Phase aggregation.
merged_phase = _aggregate_phases([s.get('phase', '') for s in all_statuses])
# Analysis progress — sum across siblings.
total = 0
processed = 0
has_progress = False
for s in all_statuses:
ap = s.get('analysis_progress')
if isinstance(ap, dict):
total += int(ap.get('total') or 0)
processed += int(ap.get('processed') or 0)
has_progress = True
# Analysis results — concat + re-index. Build a (batch_obj_id,
# old_track_index) -> new_track_index map so tasks can be
# re-indexed consistently.
merged_results: List[Dict[str, Any]] = []
track_index_remap: Dict[tuple, int] = {}
next_index = 0
for s in all_statuses:
batch_ref = id(s)
for r in (s.get('analysis_results') or []):
old_idx = int(r.get('track_index') or 0)
track_index_remap[(batch_ref, old_idx)] = next_index
new_r = dict(r)
new_r['track_index'] = next_index
merged_results.append(new_r)
next_index += 1
# Tasks — concat + re-index using the remap above. Tasks
# without a remapped entry keep their original track_index
# (defensive — shouldn't happen if analysis_results is
# consistent with the task list).
merged_tasks: List[Dict[str, Any]] = []
for s in all_statuses:
batch_ref = id(s)
for t in (s.get('tasks') or []):
old_idx = int(t.get('track_index') or 0)
new_t = dict(t)
new_t['track_index'] = track_index_remap.get((batch_ref, old_idx), old_idx)
merged_tasks.append(new_t)
merged_tasks.sort(key=lambda x: x.get('track_index', 0))
# Album bundle — pick the active sibling's, fall back to first
# bundle present, omit if none.
merged_bundle = _pick_active_album_bundle(all_statuses)
# Worker accounting — sum active_count across siblings so the
# modal's overall download progress display reflects total
# in-flight work; max_concurrent stays from primary as
# representative.
active_total = sum(int(s.get('active_count') or 0) for s in all_statuses)
merged = dict(primary) # keeps playlist_id, playlist_name, error, etc.
merged['phase'] = merged_phase
if has_progress:
merged['analysis_progress'] = {'total': total, 'processed': processed}
merged['analysis_results'] = merged_results
if merged_tasks or 'tasks' in primary:
merged['tasks'] = merged_tasks
if merged_bundle:
merged['album_bundle'] = merged_bundle
elif 'album_bundle' in primary:
merged['album_bundle'] = primary['album_bundle']
merged['active_count'] = active_total
return merged
__all__ = ['merge_wishlist_run_status']

10
core/downloads/wishlist_failed.py
View file

@ -19,7 +19,6 @@ from core.wishlist.processing import (
build_wishlist_source_context as _build_wishlist_source_context,
recover_uncaptured_failed_tracks as _recover_uncaptured_failed_tracks,
remove_completed_tracks_from_wishlist as _remove_completed_tracks_from_wishlist,
resolve_wishlist_source_type_for_batch as _resolve_wishlist_source_type_for_batch,
)
from core.wishlist.resolution import (
check_and_remove_from_wishlist as _check_and_remove_from_wishlist,
@ -108,11 +107,10 @@ def _process_failed_tracks_to_wishlist_exact(batch_id):
if permanently_failed_tracks:
try:
wishlist_service = get_wishlist_service()
# Create source_context identical to sync.py
source_context = _build_wishlist_source_context(batch)
batch_source_type = _resolve_wishlist_source_type_for_batch(batch)
# Process each failed track (matching sync.py's loop) with safety limit
max_failed_tracks = min(len(permanently_failed_tracks), 50) # Safety limit
wing_it_skipped = 0
@ -131,10 +129,10 @@ def _process_failed_tracks_to_wishlist_exact(batch_id):
continue
logger.error(f"[Wishlist Processing] Adding track {i+1}/{max_failed_tracks}: {track_name}")
success = wishlist_service.add_failed_track_from_modal(
track_info=failed_track_info,
source_type=batch_source_type,
source_type='playlist',
source_context=source_context,
profile_id=batch.get('profile_id', 1)
)

148
core/enrichment/api.py
View file

@ -18,14 +18,9 @@ from __future__ import annotations
from typing import Any, Callable, Optional
from flask import Blueprint, jsonify, request
from flask import Blueprint, jsonify
from core.enrichment.services import EnrichmentService, get_service
from core.enrichment.unmatched import (
SERVICE_ENTITY_SUPPORT,
UnmatchedQueryError,
supported_entity_types,
)
from utils.logging_config import get_logger
@ -35,33 +30,25 @@ logger = get_logger("enrichment.api")
# Hooks the host wires up so the blueprint can persist pause state and
# clean up auto-pause / yield-override sets without circular imports.
_config_set: Optional[Callable[[str, Any], None]] = None
_config_get: Optional[Callable[[str, Any], Any]] = None
_auto_paused_discard: Optional[Callable[[str], None]] = None
_yield_override_add: Optional[Callable[[str], None]] = None
_db_getter: Optional[Callable[[], Any]] = None
def configure(
*,
config_set: Optional[Callable[[str, Any], None]] = None,
config_get: Optional[Callable[[str, Any], Any]] = None,
auto_paused_discard: Optional[Callable[[str], None]] = None,
yield_override_add: Optional[Callable[[str], None]] = None,
db_getter: Optional[Callable[[], Any]] = None,
) -> None:
"""Wire host-side mutators that the generic routes call after pause/resume.
Each is optional — pass None for hosts that don't have a corresponding
mechanism (e.g. tests). ``db_getter`` returns the live ``MusicDatabase``
for the unmatched-browser routes; ``config_get``/``config_set`` read and
write the per-worker priority override.
mechanism (e.g. tests).
"""
global _config_set, _config_get, _auto_paused_discard, _yield_override_add, _db_getter
global _config_set, _auto_paused_discard, _yield_override_add
_config_set = config_set
_config_get = config_get
_auto_paused_discard = auto_paused_discard
_yield_override_add = yield_override_add
_db_getter = db_getter
def _persist_paused(service: EnrichmentService, paused: bool) -> None:
@ -166,133 +153,4 @@ def create_blueprint() -> Blueprint:
logger.error("Error resuming %s worker: %s", service.id, e)
return jsonify({'error': str(e)}), 500
@bp.route('/api/enrichment/<service_id>/breakdown', methods=['GET'])
def enrichment_breakdown(service_id: str):
"""matched / not_found / pending tallies per entity type for the modal."""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
try:
db = _db_getter()
breakdown = {
entity: db.get_enrichment_breakdown(service_id, entity)
for entity in supported_entity_types(service_id)
}
return jsonify({'service': service_id, 'breakdown': breakdown}), 200
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error building %s enrichment breakdown: %s", service_id, e)
return jsonify({'error': str(e)}), 500
@bp.route('/api/enrichment/<service_id>/unmatched', methods=['GET'])
def enrichment_unmatched(service_id: str):
"""Paginated list of items this source hasn't matched (for manual match).
Query params: ``entity_type`` (artist|album|track), ``status``
(not_found|pending|unmatched), ``q`` (name search), ``limit``, ``offset``.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
entity_type = (request.args.get('entity_type') or 'artist').strip()
status = (request.args.get('status') or 'not_found').strip()
query = (request.args.get('q') or '').strip() or None
try:
limit = int(request.args.get('limit', 50))
offset = int(request.args.get('offset', 0))
except (TypeError, ValueError):
return jsonify({'error': 'limit/offset must be integers'}), 400
try:
result = _db_getter().get_enrichment_unmatched(
service_id, entity_type, status, query, limit, offset
)
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error listing %s unmatched %ss: %s", service_id, entity_type, e)
return jsonify({'error': str(e)}), 500
result.update({
'service': service_id,
'entity_type': entity_type,
'status': status,
'limit': limit,
'offset': offset,
'entity_types': list(supported_entity_types(service_id)),
})
return jsonify(result), 200
@bp.route('/api/enrichment/<service_id>/retry', methods=['POST'])
def enrichment_retry(service_id: str):
"""Re-queue item(s) so the worker re-attempts them.
Body: ``entity_type`` (artist|album|track), ``scope`` (item|failed),
``entity_id`` (required when scope='item'). 'failed' re-queues every
not_found item of that entity type.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
data = request.get_json(silent=True) or {}
entity_type = (data.get('entity_type') or 'artist').strip()
scope = (data.get('scope') or 'item').strip()
entity_id = data.get('entity_id')
try:
count = _db_getter().reset_enrichment(service_id, entity_type, scope, entity_id)
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error re-queuing %s %s (%s): %s", service_id, entity_type, scope, e)
return jsonify({'error': str(e)}), 500
return jsonify({'success': True, 'reset': count, 'service': service_id,
'entity_type': entity_type, 'scope': scope}), 200
@bp.route('/api/enrichment/<service_id>/priority', methods=['GET'])
def enrichment_get_priority(service_id: str):
"""Return the pinned 'process this group first' entity for a worker."""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
priority = ''
if _config_get is not None:
try:
priority = (_config_get(f'{service_id}_enrichment_priority', '') or '').strip().lower()
except Exception as e:
logger.debug("reading %s priority: %s", service_id, e)
if priority not in supported_entity_types(service_id):
priority = ''
return jsonify({'service': service_id, 'priority': priority,
'entity_types': list(supported_entity_types(service_id))}), 200
@bp.route('/api/enrichment/<service_id>/priority', methods=['POST'])
def enrichment_set_priority(service_id: str):
"""Pin (or clear) the entity type the worker should process first.
Body: ``entity`` = 'artist'|'album'|'track' to pin, or '' / null / 'none'
to clear. Must be an entity type the source actually enriches.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _config_set is None:
return jsonify({'error': 'config unavailable'}), 503
data = request.get_json(silent=True) or {}
entity = (data.get('entity') or '').strip().lower()
if entity in ('none', 'clear'):
entity = ''
if entity and entity not in supported_entity_types(service_id):
return jsonify({'error': f'{service_id} does not enrich {entity!r}'}), 400
try:
_config_set(f'{service_id}_enrichment_priority', entity)
except Exception as e:
logger.error("setting %s priority: %s", service_id, e)
return jsonify({'error': str(e)}), 500
logger.info("%s enrichment priority set to %r via UI", service_id, entity or '(none)')
return jsonify({'success': True, 'service': service_id, 'priority': entity}), 200
return bp

Some files were not shown because too many files have changed in this diff Show more