Merge pull request #951 from Nezreka/dev

Dev

.dockerignore

@@ -25,6 +25,13 @@ __pycache__/
 dist/
 build/
 
+# Frontend build artifacts and local dependency caches
+webui/.tanstack/
+webui/.vite/
+webui/node_modules/
+webui/test-results/
+webui/static/dist/
+
 # Virtual environments
 venv/
 env/
@@ -51,6 +58,20 @@ Incomplete/*
 artist_bubble_snapshots.json
 .spotify_cache
 
+# Auto-downloaded ffmpeg binaries — the YouTube client downloads these
+# into tools/ when system ffmpeg isn't on PATH. The Dockerfile installs
+# system ffmpeg via apt, so the container never needs the bundled
+# binaries. If a CI run leaves them in the workspace before the docker
+# build (e.g. because a test imported web_server which initialized the
+# YouTube client), they'd otherwise get baked into the image — adding
+# ~388 MB and getting duplicated again by the chown layer.
+tools/ffmpeg
+tools/ffprobe
+tools/ffmpeg.exe
+tools/ffprobe.exe
+tools/*.zip
+tools/*.tar.xz
+
 # Documentation
 *.md
 README.md

.github/workflows/docker-publish.yml

@@ -9,9 +9,9 @@ on:
   workflow_dispatch:
     inputs:
       version_tag:
-        description: 'Version tag (e.g. 1.6, 1.7)'
+        description: 'Version tag (e.g. 2.8.2)'
         required: true
-        default: '2.3'
+        default: '2.8.2'
 
 jobs:
   build-and-push:

.gitignore

@@ -12,11 +12,14 @@ __pycache__/
 
 # User-specific files (auto-created by the app if missing)
 config/config.json
-database/music_library.db
-database/music_library.db-shm
-database/music_library.db-wal
-database/music_library.db.backup_*
+config/youtube_cookies.txt
+# All app databases are live user data — never commit (music_library, video_library, …)
+database/*.db
+database/*.db-shm
+database/*.db-wal
+database/*.db.backup_*
 database/api_call_history.json
+storage/image_cache/
 logs/*.log
 logs/*.log.*
 

DISCOVER_BEST_IN_CLASS_PLAN.md

@@ -0,0 +1,74 @@
+# discover page — best in class plan (#913 + full generator audit)
+
+morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
+
+## what i shipped tonight (done, tested, safe)
+
+### 1. listening recommendations (#913) — went from BROKEN to best-in-class
+
+the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
+
+- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
+- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
+- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
+
+the fix (all in the pure, tested core + thin scan wiring):
+- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
+- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
+
+result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
+
+### 2. its own row on the discover page
+
+new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
+
+**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
+
+> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
+
+### 3. Fresh Tape "only 5-10 tracks" — fixed
+
+root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
+
+**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
+
+---
+
+## best-in-class roadmap for listening recs (next phases — your call)
+
+these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
+
+| phase | what | value | risk | notes |
+|---|---|---|---|---|
+| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
+| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
+| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
+| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
+| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
+
+the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
+
+---
+
+## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
+
+how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
+
+### curated (built during the scan, then hydrated)
+- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
+- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
+- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
+
+### discovery-pool generators (live queries)
+- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
+- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
+- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
+- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
+- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
+- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
+
+### cross-cutting
+- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
+- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
+
+want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.

Dockerfile

@@ -1,6 +1,16 @@
 # SoulSync WebUI Dockerfile
 # Multi-architecture support for AMD64 and ARM64
 
+FROM node:24-slim AS webui-builder
+
+WORKDIR /app/webui
+
+COPY webui/package.json webui/package-lock.json ./
+RUN npm ci
+
+COPY webui/ ./
+RUN npm run build
+
 # Stage 1: Builder — install Python dependencies with compilation tools
 FROM python:3.11-slim AS builder
 
@@ -19,6 +29,16 @@ COPY requirements.txt .
 RUN pip install --no-cache-dir --upgrade pip && \
     pip install --no-cache-dir -r requirements.txt
 
+# yt-dlp must track YouTube faster than its stable channel ships — stable can
+# lag months behind a breaking YouTube change while extraction is broken
+# ("Requested format is not available"). Build images with the NIGHTLY channel.
+# COMMIT_SHA is referenced in the RUN so CI's layer cache (cache-from: gha)
+# busts on every new commit — otherwise this layer could pin a stale "nightly"
+# for months, silently defeating its purpose.
+ARG COMMIT_SHA=""
+RUN echo "yt-dlp nightly for build ${COMMIT_SHA}" && \
+    pip install --no-cache-dir -U --pre "yt-dlp[default]"
+
 # Stage 2: Runtime — only runtime dependencies, no build tools
 FROM python:3.11-slim
 
@@ -34,24 +54,61 @@ ENV PATH="/opt/venv/bin:$PATH"
 # Set working directory
 WORKDIR /app
 
-# Install runtime-only system dependencies (no gcc/build tools)
+# Install runtime-only system dependencies (no gcc/build tools).
+# unzip is needed by the Deno installer below.
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     gosu \
     ffmpeg \
     libchromaprint-tools \
+    unzip \
     && rm -rf /var/lib/apt/lists/*
 
+# Deno — JavaScript runtime for yt-dlp. YouTube gates its downloadable formats
+# behind JS challenges (nsig); without a JS runtime, yt-dlp's extraction is
+# deprecated and streams / music-video downloads fail with "Requested format
+# is not available". Deno is yt-dlp's default-enabled runtime; the official
+# installer auto-detects amd64/arm64. `deno --version` fails the build early
+# if the install ever breaks.
+RUN curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh && \
+    deno --version
+
 # Create non-root user for security
 RUN useradd --create-home --shell /bin/bash --uid 1000 soulsync
 
-# Copy application code
-COPY . .
+# Copy application code with ownership baked in.
+# Using `COPY --chown` instead of `COPY` + `chown -R /app` avoids an
+# extra image layer that duplicates the entire /app tree just to flip
+# ownership bits — Docker layers are immutable, so chown -R rewrites
+# every file into a new layer. On a clean repo that's small; if any
+# bulky workspace file slips in (e.g. auto-downloaded ffmpeg binaries
+# in tools/), it gets counted twice in the image. Cin caught this on
+# 2026-05-08 — see the .dockerignore comment for the same incident.
+COPY --chown=soulsync:soulsync . .
+COPY --chown=soulsync:soulsync --from=webui-builder /app/webui/static/dist /app/webui/static/dist
 
-# Create necessary directories with proper permissions
+# Create runtime mount-point directories the app expects to exist.
 # NOTE: /app/data is for database FILES, /app/database is the Python package
-RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/MusicVideos /app/scripts && \
-    chown -R soulsync:soulsync /app
+# NOTE: /app/Staging is required even though most users bind-mount it —
+# the entrypoint mkdir runs early and is gated by `set -e`, so a missing
+# pre-baked directory would crash the container into a restart loop on
+# rootless Docker/Podman where in-container "root" can't write to /app.
+# Pre-baking the dir here makes the entrypoint mkdir a guaranteed no-op.
+# NOTE: /app/Stream is the transient single-file streaming cache used by
+# the basic-search "Play" flow (cleared per use, never persistent). It's
+# created lazily by `core/streaming/prepare.py` via `os.makedirs`, which
+# fails silently on rootless Docker where the soulsync UID can't write
+# to /app — playback then errors out with no obvious cause. Pre-baking
+# at build time (when the layer is owned by root) avoids that path.
+# NOTE: /app/storage is the PRIVATE album-bundle staging area for the
+# torrent / usenet whole-release flow (download_source.album_bundle_staging_path
+# defaults to 'storage/album_bundle_staging'). Like /app/Stream it's created
+# lazily at runtime via mkdir(parents=True); without pre-baking it owned by
+# soulsync, the album-bundle copy fails with "[Errno 13] Permission denied:
+# 'storage'" because /app itself is root-owned and the soulsync UID can't
+# create a top-level dir there.
+RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts && \
+    chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts
 
 # Create defaults directory and copy template files
 # These will be used by entrypoint.sh to initialize empty volumes

README.md

@@ -8,7 +8,7 @@
 
 > **IMPORTANT**: Configure file sharing in slskd to avoid Soulseek bans. Set up shared folders at `http://localhost:5030/shares`.
 
-**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | [Reddit](https://old.reddit.com/r/ssync/) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
+**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
 
 ---
 
@@ -105,6 +105,21 @@ SoulSync bridges streaming services to your music library with automated discove
 - Catches wrong versions (live, remix, cover) even from streaming API sources
 - Fail-open design: verification errors never block downloads
 
+#### AcoustID API key
+
+AcoustID verification is opt-in. To enable it, request a free API key
+at <https://acoustid.org/new-application> and paste it into
+Settings → AcoustID. Without a key, downloads still complete but the
+verification step is skipped silently.
+
+If a track was previously tagged by AcoustID but the retag action in
+the AcoustID Scanner no longer changes anything, see issue #704 — the
+most common cause is that the file already carries a
+`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
+and therefore never overwrites. Removing the cached
+`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
+restores the retag.
+
 ### Metadata & Enrichment
 
 **10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz
@@ -278,19 +293,64 @@ The template points at `boulderbadgedad/soulsync:latest` (stable) by default. To
 ```bash
 git clone https://github.com/Nezreka/SoulSync
 cd SoulSync
-pip install -r requirements.txt
+python -m pip install -r requirements.txt
+
+# Build the React WebUI bundle used by the Python server.
+# Docker does this automatically; Python installs must do it manually.
+cd webui
+npm ci
+npm run build
+cd ..
+
 gunicorn -c gunicorn.conf.py wsgi:application
 # Open http://localhost:8008
 ```
 
-For local development and tests:
+When updating a Python/no-Docker install with `git pull`, rebuild the WebUI before restarting SoulSync:
 
 ```bash
-pip install -r requirements-dev.txt
-pytest
-gunicorn -c gunicorn.dev.conf.py wsgi:application
+cd webui
+npm ci
+npm run build
+cd ..
 ```
 
+If `webui/static/dist/.vite/manifest.json` is missing or stale, React-owned routes and route handoffs may not load correctly.
+
+**YouTube streaming / music videos** need two extra things on bare-metal installs (Docker bundles both):
+
+- **Deno** — yt-dlp now requires a JavaScript runtime to unlock YouTube formats. Without it, streams and music-video downloads fail with `Requested format is not available`. Install: `winget install DenoLand.Deno` (Windows) or see [deno.com](https://docs.deno.com/runtime/), then restart SoulSync.
+- **yt-dlp nightly** — the stable release can lag months behind YouTube changes. If YouTube breaks, update with: `python -m pip install -U --pre "yt-dlp[default]"`
+
+### Local Development
+
+This is only for contributors working on the WebUI with hot reload. Normal Python/no-Docker installs should build once with `npm run build` as shown above, then run only Gunicorn.
+
+For active frontend development, use two terminals so the backend and Vite stay independent:
+
+1. Backend
+   ```bash
+   python -m pip install -r requirements-dev.txt
+   gunicorn -c gunicorn.dev.conf.py wsgi:application
+   ```
+   The dev Gunicorn config watches backend files and restarts the Python server when they change.
+2. Frontend
+   ```bash
+   cd webui
+   npm ci
+   npm run dev
+   ```
+   Vite hot reloads the React side when you change webui files.
+
+Run tests separately when needed:
+
+```bash
+python -m pytest
+```
+
+If you want a convenience launcher, `python dev.py` starts both halves together
+on any OS. `./dev.sh` remains available as a Unix shell wrapper.
+
 ---
 
 ## Setup Guide
@@ -300,6 +360,7 @@ gunicorn -c gunicorn.dev.conf.py wsgi:application
 - **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases)) — required for Soulseek downloads
 - **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended for discovery
 - **Media Server** (optional): Plex, Jellyfin, or Navidrome
+- **Deno** (Python/no-Docker installs only): JavaScript runtime required by yt-dlp for YouTube streaming/music videos — `winget install DenoLand.Deno` or [deno.com](https://docs.deno.com/runtime/). Docker images bundle it.
 - **Deezer ARL token** (optional): For Deezer downloads — get from browser cookies after logging into deezer.com
 - **Tidal account** (optional): For Tidal downloads — authenticate via device flow in Settings
 - **Qobuz account** (optional): For Qobuz downloads — email/password login in Settings
@@ -413,17 +474,14 @@ SoulSync uses a `dev` → `main` flow:
 2. Branch off `dev`: `git checkout -b fix/your-change dev`
 3. Make your changes and commit
 4. Push and open a PR against **`dev`** (not `main`)
-5. CI (`build-and-test.yml`) runs ruff lint + compile + pytest on your branch — wait for green
+5. CI (`build-and-test.yml`) runs ruff lint + compile + `python -m pytest` on your branch — wait for green
 6. A maintainer reviews and merges
 
 ### Running locally
 
-```bash
-pip install -r requirements-dev.txt
-python -m ruff check .       # must be 0 errors
-python -m pytest             # all tests must pass
-gunicorn -c gunicorn.dev.conf.py wsgi:application
-```
+Use the [Local Development](#local-development) section above for the full repo-wide setup and the portable dev launcher.
+
+For web UI work, see [webui/README.md](webui/README.md). It keeps the React-side notes close to the app while this file stays the single place for repo-wide dev instructions.
 
 Ruff config lives in `pyproject.toml`. The ruleset is intentionally lenient — it catches real bugs (undefined names, import shadowing, closure-in-loop) without style nits.
 

RELEASE_2.7.9_discord.md

@@ -0,0 +1,13 @@
+**SoulSync 2.7.9** is out 🎉 a big one.
+
+🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
+
+🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
+
+⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
+
+🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
+
+🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
+
+⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶

RELEASE_2.8.0_discord.md

@@ -0,0 +1,13 @@
+**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
+
+🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
+
+✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
+
+💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
+
+🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
+
+⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
+
+enjoy! 🎶

RELEASE_2.8.1_discord.md

@@ -0,0 +1,15 @@
+**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
+
+🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
+
+🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
+
+💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
+
+🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
+
+⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
+
+🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
+
+enjoy! 🎶

RELEASE_2.8.2_discord.md

@@ -0,0 +1,9 @@
+**SoulSync 2.8.2** is out 🎉 a stability + performance release.
+
+🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
+
+⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
+
+📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
+
+enjoy! 🎶

SPEC_canonical_album_version.md

@@ -0,0 +1,144 @@
+# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
+
+**Status:** design only — no code yet.
+**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
+the user's actual files, so the Library Reorganizer, Track Number Repair, and
+tagging/enrichment all agree on the same release. Today each re-resolves
+independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
+Believer=3; standard album mislinked to a deluxe release).
+
+**Canonical-selection rule (decided):** *match the user's actual files.* The
+canonical release is the candidate whose track count + per-track durations +
+titles best fit what's on disk. Self-correcting: picks standard when you own the
+standard, deluxe when you own the deluxe.
+
+---
+
+## Hard requirement: don't disrupt the running app
+
+Every stage below is **additive and dormant until explicitly consumed**, and
+every consumer **falls back to today's behavior when no canonical is set**. So:
+- albums with no resolved canonical behave EXACTLY as they do now;
+- each stage is independently shippable and reversible;
+- nothing big-bangs.
+
+---
+
+## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
+
+### Schema (additive, nullable → migration-safe)
+Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
+existing column-exists checks; see [[db-schema-review]] migration-safety notes):
+- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
+- `canonical_album_id TEXT`
+- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
+- `canonical_resolved_at TIMESTAMP`
+
+All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
+backfill in this stage. No reads in this stage.
+
+### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
+```
+score_release_against_files(file_tracks, release_tracks) -> float
+pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
+```
+- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
+- `release_tracks`: a candidate release's tracklist (same shape).
+- Scoring (tunable weights):
+  - **track-count fit** — exact match strongly preferred; |Δcount| penalized.
+  - **duration alignment** — greedily match each file to its closest release
+    track by duration (within a tolerance, e.g. ±3s); reward coverage.
+  - **title overlap** — token/fuzzy overlap as a tiebreaker.
+  - **graceful degradation** — if a source gives no per-track durations, fall
+    back to count + title only (never crash, never force-pick).
+- Returns the best candidate + score, or (None, 0) when nothing clears a floor
+  (so we never pin a bad guess — leave it unresolved, consumers fall back).
+
+### Tests (extreme, like the rest of this codebase)
+- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
+- same album, 17 files → picks deluxe.
+- duration disambiguation when track counts tie (e.g. radio edit vs album).
+- missing-duration source → count+title fallback still picks sanely.
+- no candidate clears the floor → (None, 0).
+- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
+  whichever the files actually match.
+
+**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
+them yet. Provably zero behavior change.**
+
+---
+
+## Stage 2 — Resolver populates canonical (writes, still no consumers)
+
+A function `resolve_canonical_for_album(album_id, db, ...)`:
+1. Gather on-disk file metadata for the album (durations/titles) via the
+   library's known file paths.
+2. Gather candidate releases: every source the album has an ID for
+   (spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
+   case — sibling editions discoverable from those. Fetch each tracklist
+   (cached, rate-limited).
+3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
+   on the album row if it clears the floor.
+
+Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
+when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
+unchanged — this stage only populates the new columns. Reversible: clearing the
+columns reverts to unresolved.
+
+Tests: resolver picks the right release for the standard/deluxe fixtures; stores
+nothing when below floor; idempotent re-resolve.
+
+Cost note: fetching multiple candidate releases = more API calls. Mitigate via
+cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
+progress so it's not silent.
+
+---
+
+## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
+
+In `library_reorganize._resolve_source`: if the album has
+`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
+current `get_source_priority` walk. One-line precedence change, fully gated on
+non-NULL.
+
+Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
+to today. Re-run the existing reorganize battery (148 tests) — must stay green.
+
+**This alone fixes #767-Bug2** (a standard album whose files match the standard
+release pins the standard, so reorganize stops targeting the deluxe folder).
+
+---
+
+## Stage 4 — Track Number Repair reads canonical (closes #765)
+
+In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
+everything) — if the album has a canonical `(source, album_id)`, use it. The
+existing 6-level cascade stays as the fallback for albums with no canonical
+(preserves its all-01-album rescue ability — the regression risk we refused to
+take in the reactive fix).
+
+Now both tools resolve the SAME release → same track numbers → no contradiction.
+
+Tests: canonical present → both tools agree (shared-release test); canonical
+NULL → existing cascade unchanged.
+
+---
+
+## Risks & mitigations
+- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
+  only-on-(re)enrich, progress-logged.
+- **Sources without per-track durations** → scorer degrades to count+title.
+- **Schema migration** → additive nullable columns only; idempotent guards.
+- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
+  stored for inspection/re-resolve; manual override possible later.
+- **Backward-compat** → every consumer falls back to today's path when NULL, so
+  un-resolved albums (incl. all existing albums until backfilled) are unaffected.
+
+## Out of scope (for now)
+- Per-album manual version override UI (can layer on later — the columns support it).
+- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
+  once they share the canonical.
+
+## Suggested order to build
+1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
+after any stage and the app is consistent (just with fewer consumers wired).

Support/REVERSE-PROXY.md

@@ -0,0 +1,134 @@
+# Running SoulSync behind a reverse proxy (nginx / Caddy / Traefik)
+
+Putting SoulSync behind a reverse proxy lets you serve it over **HTTPS** and — the
+important part — put **authentication** in front of it before exposing it to the
+internet. This guide covers the safe setup.
+
+> **The golden rule:** the safest way to expose *any* self-hosted app publicly is
+> to require authentication at the proxy (an auth layer), **not** to rely on the
+> app's own protection. SoulSync's launch PIN is a useful fallback, but it is not
+> a substitute for a real auth layer on a public instance.
+
+---
+
+## 1. Turn on reverse-proxy mode
+
+By default SoulSync does **not** trust proxy headers (so a direct client can't spoof
+its IP or pretend the connection is HTTPS). If you're behind a proxy that
+terminates TLS, turn on **Settings → Security → "Behind a reverse proxy"** and
+**restart SoulSync** (this option applies at startup).
+
+When enabled, SoulSync:
+- trusts `X-Forwarded-For/Proto/Host/Port` from **one** proxy hop (correct client
+  IP, HTTPS detection, redirects),
+- marks its session cookie `Secure` (HTTPS-only) + `SameSite=Lax`, and
+- sends conservative security headers (`X-Content-Type-Options: nosniff`,
+  `X-Frame-Options: SAMEORIGIN`, `Strict-Transport-Security`). No CSP is set — tune
+  one at your proxy if you want it.
+
+**Leave it off if you access SoulSync directly over http:// on your LAN** — turning
+it on would make the session cookie HTTPS-only and break plain-HTTP access. With it
+off, none of the above applies and SoulSync behaves exactly as before.
+
+> The launch PIN is also brute-force limited (10 wrong attempts from an IP → a
+> short cooldown), regardless of this setting — a correct PIN is never affected.
+
+Restart SoulSync after changing it.
+
+---
+
+## 2. nginx
+
+SoulSync uses WebSockets (Socket.IO), so the `Upgrade`/`Connection` headers are
+**required** — without them live updates silently stop working.
+
+```nginx
+server {
+    listen 443 ssl;
+    server_name soulsync.example.com;
+
+    ssl_certificate     /etc/letsencrypt/live/soulsync.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/soulsync.example.com/privkey.pem;
+
+    # Large library scans / uploads
+    client_max_body_size 0;
+
+    location / {
+        proxy_pass http://127.0.0.1:8008;
+
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host  $host;
+
+        # Required for Socket.IO / live updates
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade    $http_upgrade;
+        proxy_set_header Connection "upgrade";
+
+        proxy_read_timeout 3600s;   # long-running scans
+        proxy_send_timeout 3600s;
+    }
+}
+```
+
+---
+
+## 3. Caddy
+
+Caddy handles TLS automatically and proxies WebSockets out of the box:
+
+```caddy
+soulsync.example.com {
+    reverse_proxy 127.0.0.1:8008
+}
+```
+
+Caddy sets `X-Forwarded-*` for you. (Add an auth provider directive if you want
+auth at the proxy — see below.)
+
+---
+
+## 4. Traefik
+
+Traefik proxies WebSockets automatically and forwards the headers. Point a router
+at the SoulSync service on port `8008` with your TLS resolver; no extra WebSocket
+config is needed.
+
+---
+
+## 5. Add authentication in front (recommended for public instances)
+
+Pick one:
+
+- **Auth proxy** — [Authelia](https://www.authelia.com/),
+  [Authentik](https://goauthentik.io/), or
+  [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). These sit in front
+  of SoulSync and force a login (with 2FA) before any request reaches it. Best
+  option for internet exposure.
+
+  SoulSync can **trust the proxy's authenticated-user header** so the launch PIN is
+  skipped once the proxy has logged you in. Set the header name in **Settings →
+  Security → "Auth proxy user header"** (e.g. `Remote-User`).
+
+  > ⚠️ **Only enable this behind a proxy you control that STRIPS any client-supplied
+  > copy of that header.** Otherwise a direct visitor could send `Remote-User: admin`
+  > and walk straight in. It's **off by default** — an unset header name means
+  > SoulSync ignores the header entirely (a spoofed one does nothing).
+- **HTTP Basic Auth** — quick and simple (nginx `auth_basic` / Caddy `basicauth`).
+  Better than nothing; weaker than an auth proxy.
+- **SoulSync launch PIN** — set an admin PIN in Settings. Enforced server-side, so
+  it can't be bypassed by hitting the API directly — but it's a shared PIN, so
+  treat it as a fallback, not your only gate.
+
+---
+
+## Troubleshooting
+
+- **Live updates / progress bars don't move** → the WebSocket `Upgrade`/`Connection`
+  headers are missing (nginx) or your proxy is buffering. Check section 2.
+- **Login won't stick / "session expired"** → you enabled `trust_reverse_proxy` but
+  are reaching SoulSync over plain `http://`. The session cookie is now HTTPS-only;
+  use `https://`, or turn the setting off for direct HTTP access.
+- **Scans time out** → raise `proxy_read_timeout` / `proxy_send_timeout`.

api/downloads.py

@@ -5,6 +5,7 @@ Download management endpoints — list, cancel active downloads.
 from flask import request, current_app
 from .auth import require_api_key
 from .helpers import api_success, api_error
+from core.runtime_state import download_tasks, tasks_lock
 
 
 def _serialize_download(task_id, task):
@@ -53,8 +54,6 @@ def register_routes(bp):
         descending so newest/in-flight tasks appear first.
         """
         try:
-            from web_server import download_tasks, tasks_lock
-
             # Parse pagination params
             try:
                 limit = int(request.args.get("limit", 100))
@@ -122,10 +121,13 @@ def register_routes(bp):
 
         try:
             from utils.async_helpers import run_async
-            soulseek = current_app.soulsync.get("soulseek_client")
+            soulseek = current_app.soulsync.get("download_orchestrator")
             if not soulseek:
                 return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
 
+            current_app.logger.info(
+                f"[CancelTrigger:api.public_cancel] download_id={download_id} username={username}"
+            )
             ok = run_async(soulseek.cancel_download(download_id, username, remove=True))
             if ok:
                 return api_success({"message": "Download cancelled."})
@@ -139,7 +141,7 @@ def register_routes(bp):
         """Cancel all active downloads and clear completed ones."""
         try:
             from utils.async_helpers import run_async
-            soulseek = current_app.soulsync.get("soulseek_client")
+            soulseek = current_app.soulsync.get("download_orchestrator")
             if not soulseek:
                 return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
 

api/request.py

@@ -104,7 +104,7 @@ def _run_search_and_download(request_id, query, notify_url):
             if request_id in _pending_requests:
                 _pending_requests[request_id]['status'] = 'searching'
 
-        soulseek = current_app._get_current_object().soulsync.get('soulseek_client')
+        soulseek = current_app._get_current_object().soulsync.get('download_orchestrator')
         if not soulseek:
             with _requests_lock:
                 if request_id in _pending_requests:

api/search.py

@@ -2,10 +2,14 @@
 Search endpoints — search external sources (Spotify, iTunes, Hydrabase).
 """
 
+import logging
+
 from flask import request, current_app
 from .auth import require_api_key
 from .helpers import api_success, api_error
 
+logger = logging.getLogger(__name__)
+
 
 def register_routes(bp):
 
@@ -38,8 +42,8 @@ def register_routes(bp):
                         if hydra_results:
                             tracks = [_serialize_track(t) for t in hydra_results]
                             return api_success({"tracks": tracks, "source": "hydrabase"})
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("hydrabase search failed: %s", e)
 
             spotify = ctx.get("spotify_client")
             from core.metadata_service import get_primary_source, get_primary_client

api/serializers.py

@@ -287,12 +287,12 @@ def serialize_watchlist_artist(obj, fields: Optional[Set[str]] = None) -> dict:
 def serialize_wishlist_track(obj, fields: Optional[Set[str]] = None) -> dict:
     """Standardized wishlist track serialization."""
     d = _to_dict(obj)
-    spotify_data = d.get("spotify_data", {})
-    if isinstance(spotify_data, str):
+    track_data = d.get("track_data", d.get("spotify_data", {}))
+    if isinstance(track_data, str):
         try:
-            spotify_data = json.loads(spotify_data)
+            track_data = json.loads(track_data)
         except (json.JSONDecodeError, TypeError):
-            spotify_data = {}
+            track_data = {}
 
     source_info = d.get("source_info")
     if isinstance(source_info, str):
@@ -303,17 +303,23 @@ def serialize_wishlist_track(obj, fields: Optional[Set[str]] = None) -> dict:
 
     result = {
         "id": d.get("id"),
+        "track_id": d.get("track_id") or d.get("spotify_track_id") or d.get("id"),
         "spotify_track_id": d.get("spotify_track_id"),
-        "track_name": spotify_data.get("name", "Unknown") if isinstance(spotify_data, dict) else "Unknown",
+        "track_name": (
+            track_data.get("name", "Unknown") if isinstance(track_data, dict) else d.get("track_name", "Unknown")
+        ),
         "artist_name": ", ".join(
-            a.get("name", "") for a in spotify_data.get("artists", [])
-        ) if isinstance(spotify_data, dict) and isinstance(spotify_data.get("artists"), list) else "",
+            a.get("name", "") if isinstance(a, dict) else str(a)
+            for a in track_data.get("artists", [])
+        ) if isinstance(track_data, dict) and isinstance(track_data.get("artists"), list) else "",
         "album_name": (
-            spotify_data.get("album", {}).get("name")
-            if isinstance(spotify_data, dict) and isinstance(spotify_data.get("album"), dict)
+            track_data.get("album", {}).get("name")
+            if isinstance(track_data, dict) and isinstance(track_data.get("album"), dict)
             else None
         ),
-        "spotify_data": spotify_data,
+        "track_data": track_data,
+        "spotify_data": track_data,
+        "provider": track_data.get("provider") if isinstance(track_data, dict) else d.get("provider"),
         "failure_reason": d.get("failure_reason"),
         "retry_count": d.get("retry_count", 0),
         "last_attempted": _isoformat(d.get("last_attempted")),
@@ -362,6 +368,7 @@ def serialize_similar_artist(obj, fields: Optional[Set[str]] = None) -> dict:
         "source_artist_id": d.get("source_artist_id"),
         "similar_artist_spotify_id": d.get("similar_artist_spotify_id"),
         "similar_artist_itunes_id": d.get("similar_artist_itunes_id"),
+        "similar_artist_musicbrainz_id": d.get("similar_artist_musicbrainz_id"),
         "similar_artist_name": d.get("similar_artist_name"),
         "similarity_rank": d.get("similarity_rank"),
         "occurrence_count": d.get("occurrence_count"),

api/system.py

@@ -2,12 +2,15 @@
 System endpoints — status, activity feed, stats.
 """
 
+import logging
 import time
 
 from flask import current_app
 from .auth import require_api_key
 from .helpers import api_success, api_error
 
+logger = logging.getLogger(__name__)
+
 
 def register_routes(bp):
 
@@ -26,7 +29,7 @@ def register_routes(bp):
             spotify = ctx.get("spotify_client")
             spotify_ok = bool(spotify and spotify.is_authenticated())
 
-            soulseek = ctx.get("soulseek_client")
+            soulseek = ctx.get("download_orchestrator")
             soulseek_ok = bool(soulseek)
 
             hydrabase = ctx.get("hydrabase_client")
@@ -35,8 +38,8 @@ def register_routes(bp):
                 try:
                     ws, _ = hydrabase.get_ws_and_lock()
                     hydrabase_ok = ws is not None and ws.connected
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("hydrabase status probe failed: %s", e)
 
             return api_success({
                 "uptime": f"{hours}h {minutes}m {seconds}s",
@@ -55,7 +58,7 @@ def register_routes(bp):
     def system_activity():
         """Recent activity feed."""
         try:
-            from web_server import activity_feed
+            from core.runtime_state import activity_feed
             items = list(activity_feed) if activity_feed else []
             return api_success({"activities": items})
         except Exception as e:
@@ -74,7 +77,7 @@ def register_routes(bp):
             # Active download count
             download_count = 0
             try:
-                from web_server import download_tasks, tasks_lock
+                from core.runtime_state import download_tasks, tasks_lock
                 with tasks_lock:
                     download_count = sum(
                         1 for t in download_tasks.values()

api/wishlist.py

@@ -45,16 +45,16 @@ def register_routes(bp):
     def add_to_wishlist():
         """Add a track to the wishlist.
 
-        Body: {"spotify_track_data": {...}, "failure_reason": "...", "source_type": "..."}
+        Body: {"track_data": {...}, "failure_reason": "...", "source_type": "..."}
         """
         body = request.get_json(silent=True) or {}
-        track_data = body.get("spotify_track_data")
+        track_data = body.get("track_data") or body.get("spotify_track_data")
         reason = body.get("failure_reason", "Added via API")
         source_type = body.get("source_type", "api")
         profile_id = parse_profile_id(request)
 
         if not track_data:
-            return api_error("BAD_REQUEST", "Missing 'spotify_track_data' in body.", 400)
+            return api_error("BAD_REQUEST", "Missing 'track_data' in body.", 400)
 
         try:
             from database.music_database import get_database
@@ -74,7 +74,7 @@ def register_routes(bp):
     @bp.route("/wishlist/<track_id>", methods=["DELETE"])
     @require_api_key
     def remove_from_wishlist(track_id):
-        """Remove a track from the wishlist by its Spotify track ID."""
+        """Remove a track from the wishlist by its track ID."""
         profile_id = parse_profile_id(request)
         try:
             from database.music_database import get_database

beatport_unified_scraper.py

@@ -2612,7 +2612,7 @@ class BeatportUnifiedScraper:
                     result['title'] = title
 
         except Exception as e:
-            pass  # Silently handle URL extraction errors
+            logger.debug("URL slug extraction: %s", e)
 
         return result
 
@@ -3186,8 +3186,8 @@ class BeatportUnifiedScraper:
                     except Exception:
                         continue
 
-        except Exception:
-            pass
+        except Exception as e:
+            logger.debug("parse release tracks failed: %s", e)
 
         return tracks
 

config/config.example.json

@@ -44,7 +44,8 @@
   },
   "metadata_enhancement": {
     "enabled": true,
-    "embed_album_art": true
+    "embed_album_art": true,
+    "single_to_album": false
   },
   "file_organization": {
     "enabled": true,
@@ -56,6 +57,11 @@
       "playlist_path": "$playlist/$artist - $title"
     }
   },
+  "import": {
+    "staging_path": "./Staging",
+    "replace_lower_quality": false,
+    "folder_artist_override": true
+  },
   "lossy_copy": {
     "enabled": false,
     "bitrate": "320",
@@ -67,4 +73,4 @@
   "listenbrainz": {
     "token": "LISTENBRAINZ_TOKEN"
   }
-}
+}

config/settings.py

@@ -2,6 +2,7 @@ import copy
 import json
 import os
 import sqlite3
+import time
 from typing import Dict, Any, Optional
 from cryptography.fernet import Fernet, InvalidToken
 from pathlib import Path
@@ -65,6 +66,12 @@ class ConfigManager:
         
         self._load_config()
 
+    # Placeholder shipped to the browser in place of a configured secret
+    # (#832 follow-up). The settings UI shows it as masked dots; if it's
+    # round-tripped back on save, ``set()`` treats it as "keep existing" so the
+    # real value is never overwritten by the mask.
+    REDACTED_SENTINEL = '__redacted_unchanged__'
+
     # Dot-notation paths to sensitive config values that must be encrypted at rest.
     # Paths pointing to dicts encrypt the entire dict as a JSON blob.
     _SENSITIVE_PATHS = frozenset({
@@ -86,6 +93,10 @@ class ConfigManager:
         'soulseek.api_key',
         'deezer_download.arl',
         'lidarr_download.api_key',
+        'prowlarr.api_key',
+        'torrent_client.password',
+        'usenet_client.api_key',
+        'usenet_client.password',
         # Enrichment services
         'listenbrainz.token',
         'acoustid.api_key',
@@ -220,7 +231,7 @@ class ConfigManager:
         """Re-save config to encrypt any plaintext sensitive values still in the DB."""
         try:
             # Read raw DB content to check if any sensitive value is still plaintext
-            conn = sqlite3.connect(str(self.database_path))
+            conn = self._connect_db()
             cursor = conn.cursor()
             cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
             row = cursor.fetchone()
@@ -257,6 +268,19 @@ class ConfigManager:
         except Exception as e:
             logger.warning(f"Could not migrate encryption: {e}")
 
+    def _connect_db(self) -> sqlite3.Connection:
+        """Open a configured SQLite connection for the config DB.
+
+        Centralizes pragma setup so every connection gets WAL mode,
+        a 30s busy timeout, and synchronous=NORMAL (the safe pairing
+        with WAL that avoids unnecessary fsyncs on slow disks).
+        """
+        conn = sqlite3.connect(str(self.database_path), timeout=30.0)
+        conn.execute("PRAGMA journal_mode=WAL")
+        conn.execute("PRAGMA busy_timeout=30000")
+        conn.execute("PRAGMA synchronous=NORMAL")
+        return conn
+
     def _ensure_database_exists(self):
         """Ensure database file and metadata table exist"""
         try:
@@ -264,8 +288,7 @@ class ConfigManager:
             self.database_path.parent.mkdir(parents=True, exist_ok=True)
 
             # Connect to database (creates file if it doesn't exist)
-            conn = sqlite3.connect(str(self.database_path), timeout=30.0)
-            conn.execute("PRAGMA journal_mode=WAL")
+            conn = self._connect_db()
             cursor = conn.cursor()
 
             # Create metadata table if it doesn't exist
@@ -287,8 +310,7 @@ class ConfigManager:
         try:
             self._ensure_database_exists()
 
-            conn = sqlite3.connect(str(self.database_path), timeout=30.0)
-            conn.execute("PRAGMA journal_mode=WAL")
+            conn = self._connect_db()
             cursor = conn.cursor()
             cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
             row = cursor.fetchone()
@@ -314,8 +336,7 @@ class ConfigManager:
         conn = None
         try:
             self._ensure_database_exists()
-            conn = sqlite3.connect(str(self.database_path), timeout=30.0)
-            conn.execute("PRAGMA journal_mode=WAL")
+            conn = self._connect_db()
             cursor = conn.cursor()
             cursor.execute("SELECT value FROM metadata WHERE key = 'log_level'")
             row = cursor.fetchone()
@@ -362,7 +383,13 @@ class ConfigManager:
         return config_data
 
     def _save_to_database(self, config_data: Dict[str, Any]) -> bool:
-        """Save configuration to database, encrypting sensitive values."""
+        """Save configuration to database, encrypting sensitive values.
+
+        Returns ``True`` on success. Transient ``database is locked``
+        failures are logged at DEBUG so the caller's retry loop owns the
+        user-visible error message — otherwise every retry would spam
+        ERROR-level logs even when the next attempt succeeds.
+        """
         conn = None
         try:
             self._ensure_database_exists()
@@ -370,9 +397,7 @@ class ConfigManager:
             # Encrypt sensitive values before writing (original dict is untouched)
             encrypted_data = self._encrypt_sensitive(config_data)
 
-            # Use longer timeout (30s) to handle contention from enrichment workers
-            conn = sqlite3.connect(str(self.database_path), timeout=30.0)
-            conn.execute("PRAGMA journal_mode=WAL")
+            conn = self._connect_db()
             cursor = conn.cursor()
 
             config_json = json.dumps(encrypted_data, indent=2)
@@ -384,6 +409,16 @@ class ConfigManager:
             conn.commit()
             return True
 
+        except sqlite3.OperationalError as e:
+            # SQLite raises OperationalError("database is locked") when the
+            # busy_timeout expires while another writer holds the lock.
+            # Log at DEBUG so the caller can decide whether the final
+            # outcome warrants an ERROR-level message.
+            if "locked" in str(e).lower():
+                logger.debug(f"Config DB locked, will retry: {e}")
+            else:
+                logger.error(f"Could not save config to database: {e}")
+            return False
         except Exception as e:
             logger.error(f"Could not save config to database: {e}")
             return False
@@ -441,14 +476,71 @@ class ConfigManager:
                 "download_path": "./downloads",
                 "transfer_path": "./Transfer",
                 "max_peer_queue": 0,
-                "download_timeout": 600
+                "download_timeout": 600,
+                # Reddit report (YeloMelo95, Bell Canada): the existing
+                # 35-per-220s sliding-window cap allows all 35 searches in
+                # rapid succession before throttling — that burst trips ISP
+                # anti-abuse. This knob forces a min gap between consecutive
+                # searches even when the window cap isn't hit. 0 = disabled
+                # (preserves prior behavior).
+                "search_min_delay_seconds": 0,
             },
             "download_source": {
-                "mode": "soulseek",  # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid"
+                "mode": "soulseek",  # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid", "torrent", "usenet"
                 "hybrid_primary": "soulseek",  # Legacy: primary source for hybrid mode
                 "hybrid_secondary": "youtube",  # Legacy: fallback source for hybrid mode
                 "hybrid_order": [],  # Ordered list of sources for hybrid mode (overrides primary/secondary)
                 "stream_source": "youtube",  # Options: "youtube" (instant, default), "active" (use download source; falls back to youtube if soulseek)
+                # Album-bundle (torrent / usenet single-source) poll tuning.
+                # Downloader is polled every N seconds until the release
+                # lands; whole job aborts at the timeout. Defaults match
+                # the previous hard-coded constants. Users on slow private
+                # trackers / large box sets can extend the timeout without
+                # editing source.
+                "album_bundle_poll_interval_seconds": 2.0,
+                "album_bundle_timeout_seconds": 6 * 60 * 60,    # 6 hours
+                # Stalled-torrent handling (noldevin): abandon a torrent that
+                # makes zero download progress for this long (dead magnet
+                # stuck on "downloading metadata", no seeders) instead of
+                # holding the worker for the full album timeout. 0 disables.
+                "torrent_stall_timeout_seconds": 10 * 60,       # 10 minutes
+                # What to do when a torrent stalls: "abandon" (remove it +
+                # its partial data, fail the download so the next source can
+                # try) or "pause" (pause in the client, leave for the user).
+                "torrent_stall_action": "abandon",
+                # Where THIS container can read completed torrent/usenet
+                # downloads (#857). The downloader (qBit/SAB) reports a save
+                # path from inside ITS OWN container — often a category folder
+                # like /data/downloads/music — which may be mounted at a
+                # different point here. Set these to the in-container path(s)
+                # where SoulSync sees those finished downloads; the resolver
+                # then finds the release by name under them. Empty = fall back
+                # to the soulseek download/transfer dirs (the shared-volume
+                # default). See core.download_plugins.album_bundle.resolve_reported_save_path.
+                "torrent_download_path": "",
+                "usenet_download_path": "",
+                # Explicit remote→local prefix mappings for non-shared / oddly
+                # mounted layouts (Sonarr/Radarr "Remote Path Mapping" style):
+                # a list of {"from": "<client path>", "to": "<soulsync path>"}.
+                # Tried before the basename fallback above.
+                "usenet_path_mappings": [],
+            },
+            "post_processing": {
+                # When a download is quarantined (AcoustID mismatch, integrity /
+                # duration failure), retry the next-best candidate instead of
+                # failing outright. Default ON (PR #801's documented default —
+                # the monitor reads this with inline default True; this template
+                # said False, so fresh installs silently shipped with the retry
+                # engine off while existing configs got it on. CI caught the
+                # split: its fresh default config failed all 7 requeue tests).
+                "retry_next_candidate_on_mismatch": True,
+                # Opt-in exhaustive retry: budget retries PER SOURCE so every
+                # source (Soulseek, then HiFi/Tidal/…) gets its own attempts
+                # before the track gives up. Default off (single global cap).
+                "retry_exhaustive": False,
+                # Retries per search query per source in exhaustive mode. The
+                # per-source budget is query_count × this value.
+                "retries_per_query": 5,
             },
             "tidal_download": {
                 "quality": "lossless",  # Options: "low", "high", "lossless", "hires"
@@ -470,6 +562,16 @@ class ConfigManager:
             "hifi_download": {
                 "quality": "lossless",  # Options: "low", "high", "lossless", "hires"
             },
+            "hifi": {
+                "embed_tags": True,
+                "tags": {
+                    "track_id": True,
+                    "artist_id": True,
+                    "isrc": True,
+                    "bpm": True,
+                    "copyright": True,
+                }
+            },
             "lidarr_download": {
                 "url": "",
                 "api_key": "",
@@ -477,6 +579,44 @@ class ConfigManager:
                 "quality_profile": "Any",
                 "cleanup_after_import": True,
             },
+            # Prowlarr — indexer aggregator. Feeds the torrent / usenet
+            # download plugins. Not a standalone source.
+            "prowlarr": {
+                "url": "",
+                "api_key": "",
+                # Comma-separated list of indexer IDs to limit searches to.
+                # Empty = search all enabled indexers.
+                "indexer_ids": "",
+            },
+            # Torrent client — receives .torrent / magnet URIs from the
+            # torrent download plugin. ``type`` picks which adapter to
+            # instantiate (qbittorrent | transmission | deluge).
+            "torrent_client": {
+                "type": "qbittorrent",
+                "url": "",
+                "username": "",
+                "password": "",
+                "category": "soulsync",
+                "save_path": "",
+            },
+            # Usenet client — receives .nzb URLs / payloads. ``type``
+            # picks the adapter (sabnzbd | nzbget). SABnzbd uses an
+            # API key; NZBGet uses username + password.
+            "usenet_client": {
+                "type": "sabnzbd",
+                "url": "",
+                "api_key": "",
+                "username": "",
+                "password": "",
+                "category": "soulsync",
+            },
+            "soundcloud_download": {
+                # Anonymous-only for now — SoundCloud Go+ OAuth tier could be
+                # added later, with credentials living under a "session" subkey
+                # alongside Tidal/Qobuz. No quality knob: anonymous SoundCloud
+                # caps at the upload's transcoding (typically 128 kbps MP3 or
+                # AAC). yt-dlp resolves bestaudio at download time.
+            },
             "listenbrainz": {
                 "base_url": "",
                 "token": "",
@@ -503,16 +643,43 @@ class ConfigManager:
                 "path": os.environ.get('DATABASE_PATH', 'database/music_library.db'),
                 "max_workers": 5
             },
+            "image_cache": {
+                "enabled": True,
+                "path": "storage/image_cache",
+                "ttl_seconds": 2592000,
+                "failed_ttl_seconds": 21600,
+                "max_download_mb": 15
+            },
             "metadata_enhancement": {
                 "enabled": True,
                 "embed_album_art": True,
-                "post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"]
+                "post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"],
+                # Ordered preferred cover-art sources (empty = use the
+                # download's own art, i.e. today's behavior). Resolved + walked
+                # with fallback by core/metadata/art_sources.py.
+                "album_art_order": [],
+                # Minimum cover-art resolution (shortest side, px). A preferred
+                # source whose art is smaller is skipped so the next source is
+                # tried — stops a low-res Cover Art Archive upload from winning.
+                # 0 disables the size gate.
+                "min_art_size": 1000,
+                # When a track matches a SINGLE release, look up the parent ALBUM
+                # that contains it and tag it as that album, so it groups with its
+                # album-mates and gets the album cover (not the single's). Off by
+                # default — it's an extra per-import metadata lookup.
+                "single_to_album": False
             },
             "musicbrainz": {
                 "embed_tags": True
             },
             "playlist_sync": {
-                "create_backup": True
+                "create_backup": True,
+                # How a re-sync writes to the server playlist:
+                #   replace   — delete + recreate (default; today's behavior)
+                #   reconcile — edit in place (add/remove delta), preserving the
+                #               playlist's custom image, description, and identity (#792)
+                #   append    — only add new tracks, never remove
+                "mode": "replace"
             },
             "settings": {
                 "audio_quality": "flac"
@@ -538,12 +705,35 @@ class ConfigManager:
             },
             "import": {
                 "staging_path": "./Staging",
-                "replace_lower_quality": False
+                # Master toggle for quality-filtering on import. On by default:
+                # downloaded files that don't meet the quality profile are
+                # quarantined instead of imported (same gate the download
+                # pipeline uses). Off → import everything regardless of quality;
+                # the library Quality Upgrade Scanner still flags them.
+                "quality_filter_enabled": True,
+                "replace_lower_quality": False,
+                # Use the top Staging folder as the artist (Artist/Album layouts,
+                # mixtapes). On by default to preserve the long-standing import
+                # behaviour for existing users. Turn OFF if you stage a mixed pile
+                # of songs under one container folder, otherwise that folder's name
+                # overrides every metadata-identified artist (the "soulsync" case).
+                "folder_artist_override": True
             },
             "m3u_export": {
                 "enabled": False,
                 "entry_base_path": ""
             },
+            "playlists": {
+                # Where "Organize by playlist" materializes playlist folders.
+                # MUST be a separate root from the music library so the media
+                # server (and the maintenance jobs) never scan it — otherwise the
+                # same track would show up twice. Mapped separately for Docker.
+                "materialize_path": "./Playlists",
+                # "symlink" (relative links, ~zero disk) or "copy" (real
+                # duplicates for FAT/USB/DAPs that can't follow links). Symlink
+                # auto-falls back to copy when the filesystem can't link.
+                "materialize_mode": "symlink"
+            },
             "youtube": {
                 "cookies_browser": "",      # "", "chrome", "firefox", "edge", "brave", "opera", "safari"
                 "download_delay": 3,        # seconds between sequential downloads
@@ -607,25 +797,38 @@ class ConfigManager:
         self.config_data = self._apply_log_level_overrides(config_data)
 
     def _save_config(self):
-        """Save configuration to database with retry on lock."""
-        success = self._save_to_database(self.config_data)
+        """Save configuration to database with exponential-backoff retry on lock.
 
-        if not success:
-            # Retry once after a brief wait (handles transient lock contention)
-            import time
-            time.sleep(1)
-            success = self._save_to_database(self.config_data)
+        Spread retries over ~7 seconds so a long-held writer (enrichment
+        worker batch insert, library scan commit, etc.) on a slow disk
+        has time to release the lock before we fall back to the JSON
+        file. The single 1s retry that used to live here gave up too
+        early on HDD-backed Docker volumes.
+        """
+        # Cumulative delay across attempts: 0.2 + 0.5 + 1.0 + 2.0 + 4.0 = 7.7s
+        # plus the 30s busy_timeout that already runs inside each attempt.
+        retry_delays = [0.2, 0.5, 1.0, 2.0, 4.0]
+        if self._save_to_database(self.config_data):
+            return
 
-        if not success:
-            # Fallback: Try to save to config.json if database fails
-            logger.warning("Database save failed - attempting file fallback")
-            try:
-                self.config_path.parent.mkdir(parents=True, exist_ok=True)
-                with open(self.config_path, 'w') as f:
-                    json.dump(self.config_data, f, indent=2)
-                logger.info("Configuration saved to config.json as fallback")
-            except Exception as e:
-                logger.error(f"Failed to save configuration: {e}")
+        for delay in retry_delays:
+            time.sleep(delay)
+            if self._save_to_database(self.config_data):
+                return
+
+        # All retries exhausted — fall back to config.json so the user
+        # doesn't lose their settings, then log a single error.
+        logger.error(
+            f"Config DB save failed after {len(retry_delays) + 1} attempts (database is locked) — "
+            "falling back to config.json"
+        )
+        try:
+            self.config_path.parent.mkdir(parents=True, exist_ok=True)
+            with open(self.config_path, 'w') as f:
+                json.dump(self.config_data, f, indent=2)
+            logger.warning("Configuration saved to config.json as fallback")
+        except Exception as e:
+            logger.error(f"Failed to save configuration: {e}")
 
     def get(self, key: str, default: Any = None) -> Any:
         keys = key.split('.')
@@ -639,7 +842,40 @@ class ConfigManager:
 
         return value
 
+    def redacted_config(self) -> Dict[str, Any]:
+        """Deep copy of the live config with every sensitive value masked.
+
+        Used for ``GET /api/settings`` so decrypted secrets never reach the
+        browser (#832 follow-up). A *set* secret becomes ``REDACTED_SENTINEL``
+        (the UI renders it as masked dots); an unset one stays empty so the UI
+        can show "not configured". Dict-valued secrets (OAuth sessions) collapse
+        to the sentinel too — the UI has no field for them anyway. The matching
+        guard in ``set()`` turns a round-tripped sentinel back into a no-op.
+        """
+        import copy
+        data = copy.deepcopy(self.config_data)
+        for path in self._SENSITIVE_PATHS:
+            keys = path.split('.')
+            parent = data
+            for k in keys[:-1]:
+                if isinstance(parent, dict) and k in parent:
+                    parent = parent[k]
+                else:
+                    parent = None
+                    break
+            if not isinstance(parent, dict):
+                continue
+            leaf = keys[-1]
+            if leaf in parent and parent[leaf] not in (None, '', {}, [], 0, False):
+                parent[leaf] = self.REDACTED_SENTINEL
+        return data
+
     def set(self, key: str, value: Any):
+        # The UI round-trips REDACTED_SENTINEL for any secret the user didn't
+        # touch — never let the mask overwrite the real value (#832 follow-up).
+        if value == self.REDACTED_SENTINEL and key in self._SENSITIVE_PATHS:
+            return
+
         keys = key.split('.')
         config = self.config_data
 
@@ -651,6 +887,20 @@ class ConfigManager:
         config[keys[-1]] = value
         self._save_config()
 
+    def resolve_secret(self, key: str, posted: Any) -> str:
+        """Resolve a secret value coming back from the settings UI.
+
+        The UI renders a saved-but-untouched secret as the REDACTED_SENTINEL (shown
+        masked); empty or that sentinel means "use the stored value", while a real
+        string is a genuine new secret. A connection-test endpoint should test the
+        EFFECTIVE secret, not the mask — otherwise testing a saved-but-untouched
+        token sends the sentinel and the source rejects it (#870)."""
+        if isinstance(posted, str):
+            posted = posted.strip()
+        if not posted or posted == self.REDACTED_SENTINEL:
+            return self.get(key, '') or ''
+        return posted
+
     def get_spotify_config(self) -> Dict[str, str]:
         return self.get('spotify', {})
 

core/acoustid_client.py

@@ -282,8 +282,9 @@ class AcoustIDClient:
 
     def test_api_key(self) -> Tuple[bool, str]:
         """
-        Validate the API key by fingerprinting a real audio file and looking it up.
-        Falls back to a direct API call if no audio files are available.
+        Validate the API key with a direct AcoustID lookup call. An invalid key
+        is reported as invalid (error code 4); any other error means the key was
+        accepted.
 
         Returns:
             Tuple of (success, message)
@@ -294,24 +295,12 @@ class AcoustIDClient:
         import requests
 
         try:
-            # Try to find a real audio file to fingerprint for an end-to-end test
-            test_file = self._find_test_audio_file()
-
-            if test_file and CHROMAPRINT_AVAILABLE:
-                logger.info(f"Testing API key with real audio file: {test_file}")
-                try:
-                    result = self.fingerprint_and_lookup(test_file)
-                    # If we get here without exception, the API key is valid
-                    # (invalid keys raise or return error before results)
-                    return True, "AcoustID API key is valid"
-                except Exception as e:
-                    error_str = str(e).lower()
-                    if 'invalid' in error_str and 'api' in error_str:
-                        return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
-                    # Fingerprint/lookup failed for non-key reasons, fall through to direct test
-                    logger.warning(f"Real file test failed ({e}), trying direct API call")
-
-            # Fallback: direct API call with minimal fingerprint
+            # Authoritative key check: a direct API lookup with a dummy
+            # fingerprint. AcoustID validates the client key first, so an
+            # invalid key returns error code 4 regardless of the fingerprint.
+            # (The previous real-file path trusted "no exception = valid", but
+            # fingerprint_and_lookup swallows the invalid-key error and returns
+            # None — so it reported broken keys as valid. #756-adjacent.)
             url = 'https://api.acoustid.org/v2/lookup'
             params = {
                 'client': self.api_key,
@@ -326,7 +315,6 @@ class AcoustIDClient:
             if data.get('status') == 'error':
                 error = data.get('error', {})
                 error_code = error.get('code', 0)
-                error_msg = error.get('message', 'Unknown error')
 
                 # Error code 4 is specifically "invalid API key"
                 if error_code == 4:
@@ -346,33 +334,33 @@ class AcoustIDClient:
             logger.error(f"Error testing AcoustID API key: {e}")
             return False, f"Error: {str(e)}"
 
-    def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
-        """
-        Generate fingerprint and look up recording in AcoustID.
+    def lookup_with_status(self, audio_file: str) -> Dict[str, Any]:
+        """Fingerprint + AcoustID lookup returning a STRUCTURED result.
 
-        This is the main method - combines fingerprinting and lookup in one call.
+        Unlike fingerprint_and_lookup() (which collapses every outcome into
+        dict-or-None), this distinguishes a genuine no-match from an actual
+        error — an invalid API key, rate limit, missing chromaprint, or a
+        fingerprint failure. That distinction is what lets the UI show "AcoustID
+        Error" (something is broken — fix it) instead of a benign-looking
+        "Skipped" that silently hides a dead key.
 
-        Args:
-            audio_file: Path to the audio file
-
-        Returns:
-            Dict with:
-                'recordings': list of dicts with 'mbid', 'title', 'artist', 'score'
-                'best_score': float (highest score across all results)
-                'recording_mbids': list of unique MBIDs (for backward compat)
-            Or None on error.
+        Returns dict with:
+            'status':   'ok' | 'no_match' | 'error' | 'no_backend'
+                        | 'fingerprint_error' | 'unsupported' | 'unavailable'
+                        | 'not_found'
+            'recordings': list (meaningful only for 'ok')
+            'best_score': float
+            'recording_mbids': list
+            'error':    human-readable detail for any non-'ok' status
+            'invalid_key': bool (True when the API specifically rejected the key)
         """
         if not ACOUSTID_AVAILABLE:
-            logger.debug("Cannot lookup: pyacoustid not available")
-            return None
-
+            return {'status': 'unavailable', 'recordings': [], 'error': 'pyacoustid library not installed'}
         if not self.api_key:
-            logger.debug("Cannot lookup: no API key")
-            return None
-
+            return {'status': 'unavailable', 'recordings': [], 'error': 'No AcoustID API key configured'}
         if not os.path.isfile(audio_file):
             logger.warning(f"Cannot lookup: file not found: {audio_file}")
-            return None
+            return {'status': 'not_found', 'recordings': [], 'error': f'File not found: {audio_file}'}
 
         # Check channel count — chromaprint crashes (SIGABRT) on >2 channel files (e.g. 5.1 surround)
         try:
@@ -382,7 +370,8 @@ class AcoustIDClient:
                 channels = getattr(mf.info, 'channels', 2)
                 if channels and channels > 2:
                     logger.warning(f"Skipping AcoustID: file has {channels} channels (surround audio): {audio_file}")
-                    return None
+                    return {'status': 'unsupported', 'recordings': [],
+                            'error': f'{channels}-channel (surround) audio not supported by chromaprint'}
         except Exception as e:
             logger.debug(f"Could not check channel count, proceeding anyway: {e}")
 
@@ -392,17 +381,12 @@ class AcoustIDClient:
             api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "NOT SET"
             logger.info(f"Fingerprinting and looking up: {audio_file} (API key: {api_key_preview})")
 
-            # Use match() which handles fingerprinting + lookup + parsing
             logger.debug("Running acoustid.match()...")
             recordings = []
             seen_mbids = set()
             best_score = 0.0
 
-            for result in acoustid.match(
-                self.api_key,
-                audio_file,
-                parse=True
-            ):
+            for result in acoustid.match(self.api_key, audio_file, parse=True):
                 # match() with parse=True returns (score, recording_id, title, artist)
                 if not isinstance(result, tuple) or len(result) < 2:
                     logger.warning(f"Unexpected result format: {result}")
@@ -420,45 +404,57 @@ class AcoustIDClient:
 
                 if recording_id and recording_id not in seen_mbids:
                     seen_mbids.add(recording_id)
-                    recordings.append({
-                        'mbid': recording_id,
-                        'title': title,
-                        'artist': artist,
-                        'score': score,
-                    })
+                    recordings.append({'mbid': recording_id, 'title': title, 'artist': artist, 'score': score})
                     logger.debug(f"Found match: {title} by {artist} (MBID: {recording_id}, score: {score})")
 
             if not recordings:
                 logger.info(f"No AcoustID matches found for: {audio_file}")
-                return None
+                return {'status': 'no_match', 'recordings': [], 'best_score': best_score,
+                        'recording_mbids': [], 'error': 'Track not found in AcoustID database'}
 
             logger.info(f"AcoustID found {len(recordings)} recording(s) (best score: {best_score:.2f})")
-            return {
-                'recordings': recordings,
-                'best_score': best_score,
-                'recording_mbids': list(seen_mbids),
-            }
+            return {'status': 'ok', 'recordings': recordings, 'best_score': best_score,
+                    'recording_mbids': list(seen_mbids)}
 
         except acoustid.NoBackendError:
             logger.error("Chromaprint library not found and fpcalc not available")
-            return None
+            return {'status': 'no_backend', 'recordings': [],
+                    'error': 'Chromaprint/fpcalc not installed (install libchromaprint1)'}
         except acoustid.FingerprintGenerationError as e:
             logger.warning(f"Failed to fingerprint {audio_file}: {e}")
-            return None
+            return {'status': 'fingerprint_error', 'recordings': [], 'error': f'Could not fingerprint file: {e}'}
         except acoustid.WebServiceError as e:
-            # Log more details about the API error
             api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "???"
             logger.warning(f"AcoustID API error (key: {api_key_preview}): {e}")
-            # Check for common errors
             error_str = str(e).lower()
-            if 'invalid' in error_str or 'unknown' in error_str:
-                logger.error("API key appears to be invalid - check your AcoustID settings")
+            # Old pyacoustid reports an invalid key as the bare "status: error"
+            # (it drops the detail), so treat that as an invalid-key signal too.
+            invalid = ('invalid' in error_str or 'unknown' in error_str or 'status: error' in error_str)
+            if invalid:
+                logger.error("AcoustID API key appears to be invalid — check your AcoustID settings")
             elif 'rate' in error_str or 'limit' in error_str:
-                logger.warning("Rate limited by AcoustID - will retry later")
-            return None
+                logger.warning("Rate limited by AcoustID — will retry later")
+            return {'status': 'error', 'recordings': [], 'invalid_key': invalid,
+                    'error': f'AcoustID API error: {e}'}
         except Exception as e:
             logger.error(f"Unexpected error in AcoustID lookup: {e}", exc_info=True)
-            return None
+            return {'status': 'error', 'recordings': [], 'error': f'Unexpected error: {e}'}
+
+    def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
+        """Legacy dict-or-None lookup. Returns the recordings dict on a confirmed
+        match, else None. Kept for callers that only need "did we identify it"
+        (library scanner, auto-import). Callers that must report WHY a lookup
+        didn't match (verification badge, key test) should use
+        ``lookup_with_status`` so an error isn't mistaken for a no-match.
+        """
+        res = self.lookup_with_status(audio_file)
+        if res.get('status') == 'ok':
+            return {
+                'recordings': res['recordings'],
+                'best_score': res.get('best_score', 0.0),
+                'recording_mbids': res.get('recording_mbids', []),
+            }
+        return None
 
     def refresh_config(self):
         """Refresh cached config values (call after settings change)."""

core/acoustid_verification.py

@@ -15,96 +15,82 @@ from typing import Optional, Dict, Any, Tuple, List
 from enum import Enum
 from utils.logging_config import get_logger
 from core.acoustid_client import AcoustIDClient
+from core.matching_engine import MusicMatchingEngine
+from core.matching.version_mismatch import is_acceptable_version_mismatch
+from core.matching.script_compat import is_cross_script_mismatch
 from core.musicbrainz_client import MusicBrainzClient
 
 logger = get_logger("acoustid.verification")
 
-# Thresholds
-MIN_ACOUSTID_SCORE = 0.80       # Minimum AcoustID fingerprint score to trust
-TITLE_MATCH_THRESHOLD = 0.70    # Title similarity needed to consider a match
-ARTIST_MATCH_THRESHOLD = 0.60   # Artist similarity needed to consider a match
+# Thresholds — single definition lives in the shared core; re-exported here so
+# existing importers keep working and the values can't drift between paths.
+from core.matching.audio_verification import (  # noqa: E402
+    MIN_ACOUSTID_SCORE, TITLE_MATCH_THRESHOLD, ARTIST_MATCH_THRESHOLD,
+)
+
+# Single matching-engine instance so version detection reuses the same patterns
+# used by the pre-download Soulseek matcher (remix / live / acoustic /
+# instrumental / etc). detect_version_type doesn't use self state, so one
+# shared instance is fine.
+_match_engine_for_version = MusicMatchingEngine()
+
+
+def _detect_title_version(title: str) -> str:
+    """Return version label for a track title.
+
+    Returns ``'original'`` when no version marker is detected, otherwise one
+    of the labels produced by ``MusicMatchingEngine.detect_version_type``
+    (``'instrumental'``, ``'live'``, ``'acoustic'``, ``'remix'``, etc).
+    """
+    if not title:
+        return 'original'
+    version_type, _ = _match_engine_for_version.detect_version_type(title)
+    return version_type
 
 
 class VerificationResult(Enum):
     """Possible outcomes of audio verification."""
     PASS = "pass"       # Title/artist match - file is correct
     FAIL = "fail"       # Title/artist mismatch - wrong file downloaded
-    SKIP = "skip"       # Could not verify (error or unavailable) - continue normally
+    SKIP = "skip"       # Genuinely couldn't verify (no match in DB) - continue normally
     DISABLED = "disabled"  # Verification not enabled
+    ERROR = "error"     # Lookup errored (invalid key / rate limit / no backend) - continue, but flag it
 
 
-def _normalize(text: str) -> str:
-    """Normalize a string for comparison: lowercase, strip parentheticals, punctuation."""
-    if not text:
-        return ""
-    s = text.lower().strip()
-    # Remove ALL parenthetical suffixes — these are metadata annotations, not core title
-    # Covers: (Live), (Remastered), (Parody of ...), (from "..." Soundtrack), (feat. ...), etc.
-    s = re.sub(r'\s*\([^)]*\)', '', s)
-    # Remove ALL square bracket suffixes: [Live], [Remastered], [Deluxe], etc.
-    s = re.sub(r'\s*\[[^\]]*\]', '', s)
-    # Remove trailing featuring info not in parentheses: "feat. ...", "ft. ...", "featuring ..."
-    s = re.sub(r'\s+(?:feat\.?|ft\.?|featuring)\s+.*$', '', s, flags=re.IGNORECASE)
-    # Remove dash-separated version tags: "- Vocal", "- Instrumental", "- Acoustic", etc.
-    s = re.sub(r'\s*-\s*(?:vocal|instrumental|acoustic|live|remix|cover|clean|explicit|radio\s*edit|original\s*mix|extended\s*mix|club\s*mix)\s*$', '', s, flags=re.IGNORECASE)
-    # Remove soundtrack/source subtitles: ' - From "..." Soundtrack', ' - from the film ...'
-    s = re.sub(r'\s*-\s*from\s+.+$', '', s, flags=re.IGNORECASE)
-    # Remove non-alphanumeric except spaces
-    s = re.sub(r'[^\w\s]', '', s)
-    # Collapse whitespace
-    s = re.sub(r'\s+', ' ', s).strip()
-    return s
+# normalize() + similarity() + the alias-aware comparison now live in the shared
+# decision core (core/matching/audio_verification.py) so import-time verification
+# and the library scan share ONE definition — the <>-strip fix, CJK handling and
+# thresholds can't drift apart again. Names kept (`_normalize` etc.) for existing
+# importers/tests.
+from core.matching.audio_verification import (  # noqa: E402
+    normalize as _normalize,
+    similarity as _similarity,
+    _alias_aware_artist_sim,
+    _find_best_title_artist_match as _core_find_best_title_artist_match,
+    evaluate as _core_evaluate,
+    Decision as _CoreDecision,
+)
 
 
-def _similarity(a: str, b: str) -> float:
-    """Calculate similarity between two strings (0.0-1.0) after normalization."""
-    na = _normalize(a)
-    nb = _normalize(b)
-    if not na or not nb:
-        return 0.0
-    if na == nb:
-        return 1.0
-    return SequenceMatcher(None, na, nb).ratio()
-
-
-def _find_best_title_artist_match(
-    recordings: List[Dict[str, Any]],
-    expected_title: str,
-    expected_artist: str,
-) -> Tuple[Optional[Dict], float, float]:
-    """
-    Find the AcoustID recording that best matches expected title/artist.
-
-    Returns:
-        (best_recording, title_similarity, artist_similarity)
-    """
-    best_rec = None
-    best_title_sim = 0.0
-    best_artist_sim = 0.0
-    best_combined = 0.0
-
-    for rec in recordings:
-        title = rec.get('title') or ''
-        artist = rec.get('artist') or ''
-
-        title_sim = _similarity(expected_title, title)
-        artist_sim = _similarity(expected_artist, artist)
-        # Weight title higher since that's the primary identifier
-        combined = (title_sim * 0.6) + (artist_sim * 0.4)
-
-        if combined > best_combined:
-            best_combined = combined
-            best_rec = rec
-            best_title_sim = title_sim
-            best_artist_sim = artist_sim
-
-    return best_rec, best_title_sim, best_artist_sim
+def _find_best_title_artist_match(recordings, expected_title, expected_artist,
+                                  expected_artist_aliases=None):
+    """Back-compat wrapper around the shared core matcher (keeps the
+    ``expected_artist_aliases`` kwarg name for existing callers/tests)."""
+    return _core_find_best_title_artist_match(
+        recordings, expected_title, expected_artist, expected_artist_aliases,
+    )
 
 
 # Shared MusicBrainz client for enrichment lookups
 _mb_client = None
 _mb_client_lock = threading.Lock()
 
+# Shared MusicBrainzService for alias lookups (issue #442). Service
+# layer wraps the raw client + adds caching + DB access — all of which
+# the alias resolution chain (library DB → cache → live MB) needs.
+_mb_service = None
+_mb_service_lock = threading.Lock()
+
 MAX_MB_ENRICHMENT_LOOKUPS = 3
 
 
@@ -118,6 +104,42 @@ def _get_mb_client() -> MusicBrainzClient:
     return _mb_client
 
 
+def _get_mb_service():
+    """Get or create a shared MusicBrainzService instance.
+
+    Used by the alias-resolution chain in `verify_audio_file`. Lazy
+    init so importing this module doesn't trigger a DB connection on
+    paths that never run AcoustID verification (test runs, dry runs).
+    """
+    global _mb_service
+    if _mb_service is None:
+        with _mb_service_lock:
+            if _mb_service is None:
+                from core.musicbrainz_service import MusicBrainzService
+                from database.music_database import get_database
+                _mb_service = MusicBrainzService(get_database())
+    return _mb_service
+
+
+def _resolve_expected_artist_aliases(expected_artist_name: str) -> List[str]:
+    """Look up alternate-spelling aliases for the expected artist.
+
+    Issue #442 — bridges cross-script artist comparisons (Japanese
+    kanji ↔ romanized, Cyrillic ↔ Latin, etc.) without forcing the
+    verifier to know about the resolution chain. Best-effort: any
+    failure (no MB service, network down, no library DB) returns
+    empty list so verification falls back to the prior direct
+    similarity check.
+    """
+    if not expected_artist_name:
+        return []
+    try:
+        return _get_mb_service().lookup_artist_aliases(expected_artist_name)
+    except Exception as e:
+        logger.debug("alias lookup failed for %r: %s", expected_artist_name, e)
+        return []
+
+
 def _enrich_recordings_from_musicbrainz(
     recordings: List[Dict[str, Any]],
 ) -> List[Dict[str, Any]]:
@@ -243,18 +265,33 @@ class AcoustIDVerification:
                 logger.debug(f"AcoustID verification skipped: {reason}")
                 return VerificationResult.SKIP, reason
 
-            # Step 2: Fingerprint and lookup in AcoustID
+            # Step 2: Fingerprint and lookup in AcoustID (structured so an
+            # actual error — invalid key / rate limit / no chromaprint — is
+            # reported distinctly from a genuine no-match, instead of both
+            # silently surfacing as "Skipped").
             logger.info(f"Fingerprinting and looking up: {audio_file_path}")
-            acoustid_result = self.acoustid_client.fingerprint_and_lookup(audio_file_path)
+            lookup = self.acoustid_client.lookup_with_status(audio_file_path) or {}
+            status = lookup.get('status')
+            # Infer status by content when absent (a caller/stub that returned
+            # just recordings): recordings => matched, none => no match.
+            if status is None:
+                status = 'ok' if lookup.get('recordings') else 'no_match'
 
-            if not acoustid_result:
-                return VerificationResult.SKIP, "Track not found in AcoustID database"
+            if status in ('error', 'no_backend', 'fingerprint_error', 'unavailable'):
+                # Something is broken (not the track's fault) — never quarantine
+                # on this; surface it so the user can fix it.
+                return VerificationResult.ERROR, lookup.get('error', 'AcoustID lookup failed')
 
+            if status != 'ok':
+                # no_match / unsupported / not_found — genuinely could not verify.
+                return VerificationResult.SKIP, lookup.get('error', 'No match in AcoustID database')
+
+            acoustid_result = lookup
             recordings = acoustid_result.get('recordings', [])
             best_score = acoustid_result.get('best_score', 0)
 
             if not recordings:
-                return VerificationResult.SKIP, "AcoustID returned no recordings"
+                return VerificationResult.SKIP, "No match in AcoustID database"
 
             logger.debug(
                 f"AcoustID returned {len(recordings)} recording(s) "
@@ -270,120 +307,55 @@ class AcoustIDVerification:
             # Enrich recordings that are missing title/artist via MusicBrainz lookup
             recordings = _enrich_recordings_from_musicbrainz(recordings)
 
-            # Step 4: Find best title/artist match among AcoustID results
-            best_rec, title_sim, artist_sim = _find_best_title_artist_match(
-                recordings, expected_track_name, expected_artist_name
-            )
+            # Issue #442 — alias resolution is LAZY. We pass a memoising
+            # thunk to the artist-comparison sites; it only fires the
+            # multi-tier lookup (library DB → cache → live MB) when
+            # direct artist similarity falls below threshold. Verifications
+            # where the direct match already passes (the common case for
+            # same-script artist names) never trigger any lookup work,
+            # so the fix doesn't add a per-verification DB query for the
+            # happy path. When the thunk DOES fire, the result is cached
+            # in the closure so the 3 comparison sites within one
+            # verification share a single resolution pass.
+            _alias_cache: Dict[str, Any] = {}
 
-            if not best_rec:
-                return VerificationResult.SKIP, "No recordings with title/artist info"
-
-            matched_title = best_rec.get('title', '?')
-            matched_artist = best_rec.get('artist', '?')
-
-            logger.info(
-                f"Best match: '{matched_title}' by '{matched_artist}' "
-                f"(title_sim={title_sim:.2f}, artist_sim={artist_sim:.2f})"
-            )
-
-            # Step 5: Decide pass/fail based on similarity
-            if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim >= ARTIST_MATCH_THRESHOLD:
-                msg = (
-                    f"Audio verified: '{matched_title}' by '{matched_artist}' "
-                    f"matches expected '{expected_track_name}' by '{expected_artist_name}' "
-                    f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
-                )
-                logger.info(f"AcoustID verification PASSED - {msg}")
-                return VerificationResult.PASS, msg
-
-            # Title matches but artist doesn't — could be a cover/collab OR a
-            # genuinely different track with the same name. Distinguish the
-            # two by checking whether the expected artist appears anywhere in
-            # AcoustID's returned recordings.
-            if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim < ARTIST_MATCH_THRESHOLD:
-                # First: if the expected artist is present in ANY recording's
-                # metadata for this fingerprint, it's likely the right track
-                # (AcoustID's "best" match just picked the wrong variant).
-                for rec in recordings:
-                    if _similarity(expected_artist_name, rec.get('artist', '')) >= ARTIST_MATCH_THRESHOLD:
-                        msg = (
-                            f"Audio verified: found '{expected_track_name}' by '{expected_artist_name}' "
-                            f"in AcoustID results"
+            def _aliases_provider() -> List[str]:
+                if 'value' not in _alias_cache:
+                    resolved = _resolve_expected_artist_aliases(expected_artist_name)
+                    _alias_cache['value'] = resolved
+                    if resolved:
+                        logger.debug(
+                            "Resolved %d aliases for expected artist '%s'",
+                            len(resolved), expected_artist_name,
                         )
-                        logger.info(f"AcoustID verification PASSED (secondary match) - {msg}")
-                        return VerificationResult.PASS, msg
+                return _alias_cache['value']
 
-                # Expected artist wasn't found anywhere. Decide between:
-                #   - FAIL: clear mismatch, e.g. "Tom Walker" (sim ~0.2) when
-                #     expecting "Maduk" — different song with same name
-                #   - SKIP: ambiguous, e.g. collab / alt credit / formatting
-                #     difference (sim 0.3-0.6)
-                #
-                # The 0.3 cutoff catches hard mismatches while preserving the
-                # benefit of the doubt for borderline artist formatting.
-                CLEAR_MISMATCH_THRESHOLD = 0.3
-                if artist_sim < CLEAR_MISMATCH_THRESHOLD:
-                    msg = (
-                        f"Audio mismatch: file identified as '{matched_title}' by '{matched_artist}', "
-                        f"expected '{expected_track_name}' by '{expected_artist_name}' "
-                        f"(title={title_sim:.0%}, artist={artist_sim:.0%}) — "
-                        f"expected artist not found in any AcoustID recording"
-                    )
-                    logger.warning(f"AcoustID verification FAILED (clear artist mismatch) - {msg}")
-                    return VerificationResult.FAIL, msg
-
-                msg = (
-                    f"Title matches but artist unclear: "
-                    f"AcoustID='{matched_title}' by '{matched_artist}', "
-                    f"expected '{expected_track_name}' by '{expected_artist_name}' "
-                    f"(artist_sim={artist_sim:.0%} — ambiguous, could be cover/collab)"
-                )
-                logger.info(f"AcoustID verification SKIPPED - {msg}")
-                return VerificationResult.SKIP, msg
-
-            # Title doesn't match — check ALL recordings for any title/artist match
-            # (the best combined match might not be the right one if there are many results)
-            for rec in recordings:
-                t = rec.get('title') or ''
-                a = rec.get('artist') or ''
-                if (_similarity(expected_track_name, t) >= TITLE_MATCH_THRESHOLD and
-                        _similarity(expected_artist_name, a) >= ARTIST_MATCH_THRESHOLD):
-                    msg = (
-                        f"Audio verified: found '{t}' by '{a}' in AcoustID results "
-                        f"matching expected '{expected_track_name}' by '{expected_artist_name}'"
-                    )
-                    logger.info(f"AcoustID verification PASSED (scan match) - {msg}")
-                    return VerificationResult.PASS, msg
-
-            # No match found — but if fingerprint score is very high (≥0.95)
-            # AND there's partial similarity in title or artist, the mismatch is
-            # likely a language/script difference (e.g. Japanese kanji vs English).
-            # Skip rather than quarantine a correct file.
-            # But if both title AND artist similarity are very low, the download
-            # source gave us a completely wrong file — fail it.
-            if best_score >= 0.95 and (title_sim >= 0.55 or artist_sim >= ARTIST_MATCH_THRESHOLD):
-                top = recordings[0]
-                msg = (
-                    f"Title/artist mismatch but fingerprint confidence very high ({best_score:.2f}): "
-                    f"AcoustID='{top.get('title', '?')}' by '{top.get('artist', '?')}', "
-                    f"expected '{expected_track_name}' by '{expected_artist_name}' — "
-                    f"likely same song in different language/script"
-                )
-                logger.info(f"AcoustID verification SKIPPED (high confidence) - {msg}")
-                return VerificationResult.SKIP, msg
-
-            # Low fingerprint score + no metadata match — file is likely wrong
-            top = recordings[0]
-            top_title = top.get('title', '?')
-            top_artist = top.get('artist', '?')
-
-            msg = (
-                f"Audio mismatch: file identified as '{top_title}' by '{top_artist}', "
-                f"expected '{expected_track_name}' by '{expected_artist_name}' "
-                f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
+            # Steps 4-5: delegate the PASS/SKIP/FAIL decision to the shared core
+            # (core/matching/audio_verification.evaluate) so import verification
+            # and the library scan apply identical logic.
+            outcome = _core_evaluate(
+                expected_track_name, expected_artist_name, recordings,
+                fingerprint_score=best_score,
+                aliases_provider=_aliases_provider,
             )
-            logger.warning(f"AcoustID verification FAILED - {msg}")
-            return VerificationResult.FAIL, msg
+            logger.info(
+                "Best match: '%s' by '%s' (title_sim=%.2f, artist_sim=%.2f) -> %s",
+                outcome.matched_title, outcome.matched_artist,
+                outcome.title_sim, outcome.artist_sim, outcome.decision.value,
+            )
+            _decision_map = {
+                _CoreDecision.PASS: VerificationResult.PASS,
+                _CoreDecision.SKIP: VerificationResult.SKIP,
+                _CoreDecision.FAIL: VerificationResult.FAIL,
+            }
+            result = _decision_map[outcome.decision]
+            if result == VerificationResult.PASS:
+                logger.info("AcoustID verification PASSED - %s", outcome.reason)
+            elif result == VerificationResult.FAIL:
+                logger.warning("AcoustID verification FAILED - %s", outcome.reason)
+            else:
+                logger.info("AcoustID verification SKIPPED - %s", outcome.reason)
+            return result, outcome.reason
 
         except Exception as e:
             # Any unexpected error -> SKIP (fail open)

core/album_consistency.py

@@ -195,8 +195,8 @@ def _find_best_release(album_name, artist_name, track_count, mb_service):
                     sr_id = sr.get('id', '')
                     if sr_id and sr_id not in candidate_mbids:
                         candidate_mbids.append(sr_id)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("search_release fallback failed: %s", e)
 
         if not candidate_mbids:
             logger.info(f"No MB release found for '{album_name}' by '{artist_name}'")

core/amazon_client.py

@@ -0,0 +1,796 @@
+"""Amazon Music metadata client backed by a T2Tunes proxy instance.
+
+T2Tunes exposes the Amazon Music catalog (search, album metadata, stream info)
+through a simple REST API. This module wraps those endpoints and normalises the
+responses into the same Track / Artist / Album dataclass shape used by
+DeezerClient and iTunesClient.
+
+Endpoints used:
+    GET /api/status
+    GET /api/amazon-music/search
+    GET /api/amazon-music/metadata
+    GET /api/amazon-music/media-from-asin
+
+Config keys (all optional — fall back to public defaults):
+    amazon.base_url        T2Tunes instance URL  (default: https://t2tunes.site)
+    amazon.country         ISO-3166 country code (default: US)
+    amazon.preferred_codec Preferred audio codec (default: flac)
+"""
+
+from __future__ import annotations
+
+import re
+import threading
+import time
+from dataclasses import dataclass
+from typing import Any, Dict, Iterator, List, Optional
+from urllib.parse import urljoin
+
+import requests
+
+from config.settings import config_manager
+from core.api_call_tracker import api_call_tracker
+from utils.logging_config import get_logger
+
+logger = get_logger("amazon_client")
+
+# Strips featuring credits like "Artist feat. X", "Artist ft. Y" so artist
+# deduplication works on the primary artist name only.
+_FEAT_RE = re.compile(r'\s+(?:feat(?:uring)?\.?|ft\.?)\s+.*', re.IGNORECASE)
+
+# Strips the Explicit marker — explicit is treated as the default version.
+# Clean/Edited/Censored stay in the name so users can distinguish them.
+_EDITION_RE = re.compile(r'\s*[\[\(]explicit[\]\)]', re.IGNORECASE)
+
+
+def _primary_artist(name: str) -> str:
+    return _FEAT_RE.sub('', name).strip()
+
+
+def _strip_edition(name: str) -> str:
+    return _EDITION_RE.sub('', name).strip()
+
+
+def _unslugify(name: str) -> str:
+    """Convert a slug-form artist ID (e.g. 'kendrick_lamar') to a search name."""
+    return name.replace('_', ' ')
+
+
+DEFAULT_BASE_URL = "https://t2tunes.site"
+DEFAULT_COUNTRY = "US"
+DEFAULT_CODEC = "flac"
+MIN_API_INTERVAL = 0.5  # seconds — T2Tunes has no published rate limit
+
+_last_api_call: float = 0.0
+_api_call_lock = threading.Lock()
+
+_META_CACHE_TTL = 300  # seconds
+_meta_cache: Dict[str, tuple] = {}  # asin -> (fetched_at, meta_dict)
+_meta_cache_lock = threading.Lock()
+
+
+class AmazonClientError(RuntimeError):
+    """Raised on unrecoverable T2Tunes API errors.
+
+    Carries the HTTP ``status_code`` when the failure was an HTTP error, so
+    callers (the worker's outage detection) can tell a source outage (5xx) from
+    a per-item miss without parsing the message.
+    """
+
+    def __init__(self, *args, status_code=None):
+        super().__init__(*args)
+        self.status_code = status_code
+
+
+# ---------------------------------------------------------------------------
+# Dataclasses — field layout matches DeezerClient / iTunesClient exactly
+# ---------------------------------------------------------------------------
+
+@dataclass
+class Track:
+    id: str           # Amazon track ASIN
+    name: str
+    artists: List[str]
+    album: str
+    duration_ms: int
+    popularity: int
+    preview_url: Optional[str] = None
+    external_urls: Optional[Dict[str, str]] = None
+    image_url: Optional[str] = None
+    release_date: Optional[str] = None
+    track_number: Optional[int] = None
+    disc_number: Optional[int] = None
+    album_type: Optional[str] = None
+    total_tracks: Optional[int] = None
+    isrc: Optional[str] = None
+
+    @classmethod
+    def from_search_hit(cls, doc: Dict[str, Any]) -> "Track":
+        return cls(
+            id=str(doc.get("asin") or ""),
+            name=str(doc.get("title") or ""),
+            artists=[str(doc.get("artistName") or "Unknown Artist")],
+            album=str(doc.get("albumName") or ""),
+            duration_ms=int(doc.get("duration") or 0) * 1000,
+            popularity=0,
+            isrc=str(doc.get("isrc") or "") or None,
+        )
+
+    @classmethod
+    def from_stream_info(
+        cls,
+        stream: "T2TunesStreamInfo",
+        album_meta: Optional[Dict[str, Any]] = None,
+    ) -> "Track":
+        album = album_meta or {}
+        return cls(
+            id=stream.asin,
+            name=stream.title,
+            artists=[stream.artist] if stream.artist else ["Unknown Artist"],
+            album=stream.album,
+            duration_ms=0,
+            popularity=0,
+            image_url=album.get("image"),
+            release_date=album.get("release_date"),
+            total_tracks=album.get("trackCount"),
+            isrc=stream.isrc or None,
+        )
+
+
+@dataclass
+class Artist:
+    id: str           # Slugified artist name — T2Tunes exposes no artist IDs
+    name: str
+    popularity: int
+    genres: List[str]
+    followers: int
+    image_url: Optional[str] = None
+    external_urls: Optional[Dict[str, str]] = None
+
+    @classmethod
+    def from_name(cls, name: str) -> "Artist":
+        slug = name.lower().replace(" ", "_")
+        return cls(id=slug, name=name, popularity=0, genres=[], followers=0)
+
+
+@dataclass
+class Album:
+    id: str           # Amazon album ASIN
+    name: str
+    artists: List[str]
+    release_date: str
+    total_tracks: int
+    album_type: str
+    image_url: Optional[str] = None
+    external_urls: Optional[Dict[str, str]] = None
+    explicit: Optional[bool] = None
+
+    @classmethod
+    def from_search_hit(cls, doc: Dict[str, Any]) -> "Album":
+        return cls(
+            id=str(doc.get("albumAsin") or doc.get("asin") or ""),
+            name=str(doc.get("albumName") or doc.get("title") or ""),
+            artists=[str(doc.get("artistName") or "Unknown Artist")],
+            release_date="",
+            total_tracks=0,
+            album_type="album",
+        )
+
+    @classmethod
+    def from_metadata(cls, album_meta: Dict[str, Any], asin: str = "") -> "Album":
+        return cls(
+            id=str(album_meta.get("asin") or asin or ""),
+            name=str(album_meta.get("title") or ""),
+            artists=[str(album_meta.get("artistName") or "Unknown Artist")],
+            release_date=str(album_meta.get("release_date") or album_meta.get("releaseDate") or ""),
+            total_tracks=int(album_meta.get("trackCount") or 0),
+            album_type="album",
+            image_url=album_meta.get("image"),
+            explicit=album_meta.get("explicit"),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Internal dataclasses for raw T2Tunes response parsing
+# ---------------------------------------------------------------------------
+
+@dataclass(frozen=True)
+class T2TunesSearchItem:
+    asin: str
+    title: str
+    artist_name: str
+    item_type: str
+    album_name: str = ""
+    album_asin: str = ""
+    duration_seconds: int = 0
+    isrc: str = ""
+
+    @property
+    def is_album(self) -> bool:
+        return "album" in self.item_type.lower()
+
+    @property
+    def is_track(self) -> bool:
+        return "track" in self.item_type.lower()
+
+
+@dataclass(frozen=True)
+class T2TunesStreamInfo:
+    asin: str
+    streamable: bool
+    codec: str
+    format: str
+    sample_rate: Optional[int]
+    stream_url: str
+    decryption_key: Optional[str]   # hex-encoded AES key; None when stream is clear
+    title: str = ""
+    artist: str = ""
+    album: str = ""
+    isrc: str = ""
+    cover_url: str = ""
+    track_number: Optional[int] = None
+    disc_number: Optional[int] = None
+    genre: str = ""
+    label: str = ""
+    date: str = ""
+
+    @property
+    def has_decryption_key(self) -> bool:
+        return bool(self.decryption_key)
+
+
+# ---------------------------------------------------------------------------
+# Rate-limit enforcement
+# ---------------------------------------------------------------------------
+
+def _rate_limit() -> None:
+    global _last_api_call
+    with _api_call_lock:
+        elapsed = time.monotonic() - _last_api_call
+        if elapsed < MIN_API_INTERVAL:
+            time.sleep(MIN_API_INTERVAL - elapsed)
+        _last_api_call = time.monotonic()
+    api_call_tracker.record_call("amazon")
+
+
+# ---------------------------------------------------------------------------
+# Main client
+# ---------------------------------------------------------------------------
+
+class AmazonClient:
+    """T2Tunes-backed Amazon Music metadata and stream-info client."""
+
+    def __init__(
+        self,
+        base_url: Optional[str] = None,
+        country: Optional[str] = None,
+        preferred_codec: Optional[str] = None,
+        timeout: int = 30,
+        session: Optional[Any] = None,
+    ) -> None:
+        self.base_url = (base_url or config_manager.get("amazon.base_url", DEFAULT_BASE_URL)).rstrip("/")
+        self.country = (country or config_manager.get("amazon.country", DEFAULT_COUNTRY)).upper()
+        self.preferred_codec = (
+            preferred_codec or config_manager.get("amazon.preferred_codec", DEFAULT_CODEC)
+        ).lower()
+        self.timeout = timeout
+        self.session: Any = session or requests.Session()
+        if isinstance(self.session, requests.Session):
+            self.session.headers.update({
+                "Accept": "application/json",
+                "User-Agent": "SoulSync/1.0",
+                "Referer": self.base_url,
+            })
+
+    # ------------------------------------------------------------------
+    # Lifecycle
+    # ------------------------------------------------------------------
+
+    def reload_config(self) -> None:
+        self.base_url = config_manager.get("amazon.base_url", DEFAULT_BASE_URL).rstrip("/")
+        self.country = config_manager.get("amazon.country", DEFAULT_COUNTRY).upper()
+        self.preferred_codec = config_manager.get("amazon.preferred_codec", DEFAULT_CODEC).lower()
+
+    def is_authenticated(self) -> bool:
+        """Return True when the T2Tunes instance reports Amazon Music as up."""
+        try:
+            return str(self.status().get("amazonMusic", "")).lower() == "up"
+        except AmazonClientError:
+            return False
+
+    # ------------------------------------------------------------------
+    # Low-level API wrappers
+    # ------------------------------------------------------------------
+
+    def status(self) -> Dict[str, Any]:
+        return self._get_json("/api/status")
+
+    def search_raw(self, query: str, *, types: str = "track,album") -> List[T2TunesSearchItem]:
+        data = self._get_json(
+            "/api/amazon-music/search",
+            params={"query": query, "types": types, "country": self.country},
+        )
+        return list(self._iter_search_items(data))
+
+    def album_metadata(self, asin: str) -> Dict[str, Any]:
+        return self._get_json(
+            "/api/amazon-music/metadata",
+            params={"asin": asin, "country": self.country},
+        )
+
+    def media_from_asin(self, asin: str, codec: Optional[str] = None) -> List[T2TunesStreamInfo]:
+        effective_codec = (codec or self.preferred_codec).lower()
+        data = self._get_json(
+            "/api/amazon-music/media-from-asin",
+            params={"asin": asin, "country": self.country, "codec": effective_codec},
+        )
+        if isinstance(data, list):
+            return [self._parse_stream_info(item) for item in data if isinstance(item, dict)]
+        if isinstance(data, dict):
+            return [self._parse_stream_info(data)]
+        raise AmazonClientError(f"Unexpected media-from-asin response type: {type(data).__name__}")
+
+    # ------------------------------------------------------------------
+    # Metadata interface — mirrors DeezerClient / iTunesClient signatures
+    # ------------------------------------------------------------------
+
+    def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
+        _rate_limit()
+        items = self.search_raw(query, types="track")
+        track_pairs: List[tuple] = []   # (Track, album_asin)
+        seen_album_asins: List[str] = []
+        for item in items:
+            if not item.is_track:
+                continue
+            track = Track.from_search_hit({
+                "asin": item.asin,
+                "title": _strip_edition(item.title),
+                "artistName": _primary_artist(item.artist_name),
+                "albumName": _strip_edition(item.album_name),
+                "albumAsin": item.album_asin,
+                "duration": item.duration_seconds,
+                "isrc": item.isrc,
+            })
+            track_pairs.append((track, item.album_asin))
+            if item.album_asin and item.album_asin not in seen_album_asins:
+                seen_album_asins.append(item.album_asin)
+            if len(track_pairs) >= limit:
+                break
+        album_metas = self._fetch_album_metas(seen_album_asins[:5])
+        tracks: List[Track] = []
+        for track, album_asin in track_pairs:
+            if album_asin and album_asin in album_metas:
+                meta = album_metas[album_asin]
+                track.image_url = meta.get("image")
+                track.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
+                track.total_tracks = meta.get("trackCount")
+            tracks.append(track)
+        return tracks
+
+    def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
+        _rate_limit()
+        items = self.search_raw(query, types="track")
+        seen: Dict[str, Artist] = {}
+        artist_album_asin: Dict[str, str] = {}  # artist name → first album ASIN seen
+        for item in items:
+            name = _primary_artist(item.artist_name)
+            if not name:
+                continue
+            if name not in seen:
+                seen[name] = Artist.from_name(name)
+            if name not in artist_album_asin and item.album_asin:
+                artist_album_asin[name] = item.album_asin
+            if len(seen) >= limit:
+                break
+        # T2Tunes has no artist images — use an album cover as stand-in.
+        unique_asins = list({v for v in artist_album_asin.values()})[:5]
+        album_metas = self._fetch_album_metas(unique_asins)
+        for name, artist in seen.items():
+            asin = artist_album_asin.get(name)
+            if asin and asin in album_metas:
+                artist.image_url = album_metas[asin].get("image")
+        return list(seen.values())
+
+    def search_albums(self, query: str, limit: int = 20) -> List[Album]:
+        _rate_limit()
+        items = self.search_raw(query, types="track")
+        album_candidates: List[tuple] = []  # (Album, asin)
+        seen_keys: set = set()
+        for item in items:
+            album_asin = item.album_asin
+            raw_name = item.album_name
+            if not raw_name:
+                continue
+            display_name = _strip_edition(raw_name)
+            artist = _primary_artist(item.artist_name)
+            # Collapse Explicit/Clean variants: same normalised name + artist = same album
+            dedup_key = (display_name.lower(), artist.lower())
+            if dedup_key in seen_keys:
+                continue
+            seen_keys.add(dedup_key)
+            album = Album.from_search_hit({
+                "albumAsin": album_asin,
+                "albumName": display_name,
+                "artistName": artist,
+            })
+            album_candidates.append((album, album_asin))
+            if len(album_candidates) >= limit:
+                break
+        album_metas = self._fetch_album_metas([a for _, a in album_candidates[:10]])
+        albums: List[Album] = []
+        for album, asin in album_candidates:
+            if asin in album_metas:
+                meta = album_metas[asin]
+                album.image_url = meta.get("image")
+                album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
+                album.total_tracks = int(meta.get("trackCount") or 0)
+            albums.append(album)
+        return albums
+
+    def get_track_details(self, asin: str) -> Optional[Dict[str, Any]]:
+        """Return a Spotify-compatible dict for a single track ASIN."""
+        _rate_limit()
+        try:
+            streams = self.media_from_asin(asin)
+        except AmazonClientError:
+            return None
+        if not streams:
+            return None
+        s = streams[0]
+
+        album_data: Dict[str, Any] = {}
+        try:
+            meta = self.album_metadata(asin)
+            albums = meta.get("albumList")
+            if isinstance(albums, list) and albums and isinstance(albums[0], dict):
+                album_data = albums[0]
+        except AmazonClientError:
+            pass
+
+        return {
+            "id": s.asin,
+            "name": _strip_edition(s.title),
+            "artists": [{"name": _primary_artist(s.artist), "id": ""}],
+            "album": {
+                "id": album_data.get("asin", ""),
+                "name": _strip_edition(s.album),
+                "images": [{"url": album_data["image"]}] if album_data.get("image") else [],
+                "release_date": album_data.get("release_date") or album_data.get("releaseDate") or s.date or "",
+                "total_tracks": album_data.get("trackCount", 0),
+            },
+            "duration_ms": 0,
+            "popularity": 0,
+            "external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
+            "track_number": s.track_number,
+            "disc_number": s.disc_number,
+            "isrc": s.isrc,
+            "is_album_track": True,
+            "raw_data": {
+                "codec": s.codec,
+                "format": s.format,
+                "sample_rate": s.sample_rate,
+                "streamable": s.streamable,
+                "has_decryption_key": s.has_decryption_key,
+            },
+        }
+
+    def get_album(self, asin: str, include_tracks: bool = True) -> Optional[Dict[str, Any]]:
+        """Return a Spotify-compatible album dict."""
+        _rate_limit()
+        try:
+            meta = self.album_metadata(asin)
+        except AmazonClientError:
+            return None
+
+        albums = meta.get("albumList")
+        if not isinstance(albums, list) or not albums:
+            return None
+        album = albums[0] if isinstance(albums[0], dict) else {}
+
+        result: Dict[str, Any] = {
+            "id": asin,
+            "name": _strip_edition(album.get("title", "")),
+            "artists": [{"name": _primary_artist(album.get("artistName", "")), "id": ""}],
+            "release_date": album.get("release_date") or album.get("releaseDate") or "",
+            "total_tracks": album.get("trackCount", 0),
+            "album_type": "album",
+            "images": [{"url": album["image"]}] if album.get("image") else [],
+            "external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
+            "label": album.get("label", ""),
+        }
+        if include_tracks:
+            tracks_data = self.get_album_tracks(asin) or {
+                "items": [], "total": 0, "limit": 50, "next": None,
+            }
+            result["tracks"] = tracks_data
+            # Backfill release_date from stream tags when album metadata lacks it.
+            if not result["release_date"]:
+                items = tracks_data.get("items") or []
+                for item in items:
+                    rd = item.get("release_date") or ""
+                    if rd and len(rd) >= 4:
+                        result["release_date"] = rd
+                        break
+        return result
+
+    def get_album_tracks(self, asin: str) -> Optional[Dict[str, Any]]:
+        """Return album tracks in Spotify pagination format."""
+        _rate_limit()
+        try:
+            streams = self.media_from_asin(asin)
+        except AmazonClientError:
+            return None
+
+        # media_from_asin has no duration — enrich from search results which do.
+        duration_map: Dict[str, int] = {}  # track asin → duration_ms
+        if streams:
+            album_name = _strip_edition(streams[0].album)
+            artist_name = _primary_artist(streams[0].artist)
+            try:
+                search_items = self.search_raw(
+                    f"{album_name} {artist_name}", types="track"
+                )
+                for item in search_items:
+                    if item.album_asin == asin and item.duration_seconds:
+                        duration_map[item.asin] = item.duration_seconds * 1000
+            except Exception as exc:
+                logger.debug("Duration backfill failed for ASIN %s: %s", asin, exc)
+
+        items = [
+            {
+                "id": s.asin,
+                "name": _strip_edition(s.title),
+                "artists": [{"name": _primary_artist(s.artist), "id": ""}],
+                "duration_ms": duration_map.get(s.asin, 0),
+                "track_number": s.track_number if s.track_number is not None else idx + 1,
+                "disc_number": s.disc_number,
+                "release_date": s.date or "",
+                "isrc": s.isrc,
+            }
+            for idx, s in enumerate(streams)
+        ]
+        return {"items": items, "total": len(items), "limit": 50, "next": None}
+
+    def get_artist(self, artist_name: str) -> Optional[Dict[str, Any]]:
+        """Return a Spotify-compatible artist dict inferred from search results."""
+        _rate_limit()
+        search_name = _unslugify(artist_name)
+        try:
+            items = self.search_raw(search_name, types="track")
+        except AmazonClientError:
+            return None
+        name_lower = search_name.lower()
+        match = next(
+            (i for i in items if _primary_artist(i.artist_name).lower() == name_lower),
+            next((i for i in items if name_lower in _primary_artist(i.artist_name).lower()), None),
+        )
+        if not match:
+            return None
+        return {
+            "id": match.artist_name.lower().replace(" ", "_"),
+            "name": match.artist_name,
+            "genres": [],
+            "popularity": 0,
+            "followers": {"total": 0},
+            "images": [],
+            "external_urls": {},
+        }
+
+    def get_artist_albums(
+        self,
+        artist_name: str,
+        album_type: str = "album,single",
+        limit: int = 200,
+    ) -> List[Album]:
+        """Return albums for an artist inferred from search results."""
+        _rate_limit()
+        search_name = _unslugify(artist_name)
+        try:
+            items = self.search_raw(f"{search_name} album", types="track")
+        except AmazonClientError:
+            return []
+        album_candidates: List[tuple] = []  # (Album, asin)
+        seen_asins: set = set()
+        name_lower = search_name.lower()
+        for item in items:
+            if not item.is_album:
+                continue
+            if _primary_artist(item.artist_name).lower() != name_lower:
+                continue
+            album_asin = item.album_asin or item.asin
+            if album_asin in seen_asins:
+                continue
+            seen_asins.add(album_asin)
+            album = Album.from_search_hit({
+                "albumAsin": album_asin,
+                "albumName": _strip_edition(item.album_name or item.title),
+                "artistName": _primary_artist(item.artist_name),
+            })
+            album_candidates.append((album, album_asin))
+            if len(album_candidates) >= limit:
+                break
+
+        # Fetch metadata for art, release_date, track_count, and type inference.
+        # No explicit cap here — search_raw naturally bounds results to ~20 items,
+        # so album_candidates is already small.
+        asins_to_fetch = [a for _, a in album_candidates]
+        metas = self._fetch_album_metas(asins_to_fetch)
+
+        albums: List[Album] = []
+        for album, asin in album_candidates:
+            meta = metas.get(asin, {})
+            if meta:
+                album.image_url = meta.get("image")
+                album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
+                total = int(meta.get("trackCount") or 0)
+                album.total_tracks = total
+                # T2Tunes doesn't expose release type — infer from track count.
+                # 1-track releases are singles; keep default "album" otherwise.
+                if total == 1:
+                    album.album_type = "single"
+            albums.append(album)
+        return albums
+
+    def get_track_features(self, track_id: str) -> Optional[Dict[str, Any]]:
+        """Not available from Amazon Music — returns None for compatibility."""
+        return None
+
+    def _get_artist_image_from_albums(self, artist_id: str) -> Optional[str]:
+        """Return an album cover as artist image stand-in (T2Tunes has no artist images)."""
+        search_name = _unslugify(artist_id)
+        try:
+            items = self.search_raw(search_name, types="track")
+        except AmazonClientError:
+            return None
+        name_lower = search_name.lower()
+        for item in items:
+            if _primary_artist(item.artist_name).lower() != name_lower:
+                continue
+            asin = item.album_asin or item.asin
+            if not asin:
+                continue
+            metas = self._fetch_album_metas([asin])
+            if asin in metas and metas[asin].get("image"):
+                return metas[asin]["image"]
+        return None
+
+    # ==================== Interface Aliases (match DeezerClient method names) ====================
+    get_album_metadata = get_album
+    get_artist_info = get_artist
+    get_artist_albums_list = get_artist_albums
+
+    # ------------------------------------------------------------------
+    # Private helpers
+    # ------------------------------------------------------------------
+
+    def _fetch_album_metas(self, asins: List[str]) -> Dict[str, Dict[str, Any]]:
+        """Parallel-fetch album metadata for up to N ASINs. Returns {asin: albumList[0]}."""
+        if not asins:
+            return {}
+        metas: Dict[str, Dict[str, Any]] = {}
+
+        def _fetch(asin: str) -> None:
+            now = time.monotonic()
+            with _meta_cache_lock:
+                entry = _meta_cache.get(asin)
+                if entry and (now - entry[0]) < _META_CACHE_TTL:
+                    metas[asin] = entry[1]
+                    return
+            _rate_limit()
+            try:
+                raw = self.album_metadata(asin)
+                lst = raw.get("albumList")
+                if isinstance(lst, list) and lst and isinstance(lst[0], dict):
+                    meta = lst[0]
+                    with _meta_cache_lock:
+                        _meta_cache[asin] = (time.monotonic(), meta)
+                    metas[asin] = meta
+            except Exception as exc:
+                logger.debug("Album metadata fetch failed for ASIN %s: %s", asin, exc)
+
+        from concurrent.futures import ThreadPoolExecutor
+        with ThreadPoolExecutor(max_workers=min(len(asins), 5)) as pool:
+            list(pool.map(_fetch, asins))
+        return metas
+
+    def _get_json(self, path: str, params: Optional[Dict[str, Any]] = None) -> Any:
+        url = urljoin(f"{self.base_url}/", path.lstrip("/"))
+        last_exc: Optional[Exception] = None
+        for attempt in range(3):
+            if attempt:
+                time.sleep(1.0 * attempt)
+            try:
+                resp = self.session.get(url, params=params, timeout=self.timeout)
+                resp.raise_for_status()
+            except requests.HTTPError as exc:
+                body = exc.response.text[:500].replace("\n", " ")
+                # T2Tunes returns 400 for transient Amazon-side failures — retry those.
+                if exc.response.status_code == 400 and "Failed to search" in exc.response.text:
+                    logger.debug("T2Tunes transient 400 on attempt %d, retrying: %s", attempt + 1, url)
+                    last_exc = AmazonClientError(
+                        f"HTTP {exc.response.status_code} for {url} — body: {body!r}"
+                    )
+                    continue
+                raise AmazonClientError(
+                    f"HTTP {exc.response.status_code} for {url} — body: {body!r}",
+                    status_code=exc.response.status_code,
+                ) from exc
+            except requests.RequestException as exc:
+                raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
+            try:
+                return resp.json()
+            except ValueError as exc:
+                preview = resp.text[:200].replace("\n", " ")
+                raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
+        raise last_exc  # type: ignore[misc]
+
+    @staticmethod
+    def _iter_search_items(response: Any) -> Iterator[T2TunesSearchItem]:
+        if not isinstance(response, dict):
+            raise AmazonClientError(
+                f"Unexpected search response type: {type(response).__name__}"
+            )
+        for result in response.get("results") or []:
+            if not isinstance(result, dict):
+                continue
+            for hit in result.get("hits") or []:
+                if not isinstance(hit, dict):
+                    continue
+                doc = hit.get("document")
+                if not isinstance(doc, dict):
+                    continue
+                asin = str(doc.get("asin") or "")
+                if not asin:
+                    continue
+                yield T2TunesSearchItem(
+                    asin=asin,
+                    title=str(doc.get("title") or ""),
+                    artist_name=str(doc.get("artistName") or ""),
+                    item_type=str(doc.get("__type") or ""),
+                    album_name=str(doc.get("albumName") or ""),
+                    album_asin=str(doc.get("albumAsin") or ""),
+                    duration_seconds=int(doc.get("duration") or 0),
+                    isrc=str(doc.get("isrc") or ""),
+                )
+
+    @staticmethod
+    def _parse_stream_info(item: Dict[str, Any]) -> T2TunesStreamInfo:
+        stream_info = item.get("streamInfo") if isinstance(item.get("streamInfo"), dict) else {}
+        tags = item.get("tags") if isinstance(item.get("tags"), dict) else {}
+        # T2Tunes API has a typo: "stremeable" in some responses
+        streamable = item.get("streamable")
+        if streamable is None:
+            streamable = item.get("stremeable")
+        raw_key = item.get("decryptionKey")
+        decryption_key = str(raw_key) if raw_key else None
+
+        def _int_tag(key: str) -> Optional[int]:
+            v = tags.get(key)
+            try:
+                return int(v) if v is not None else None
+            except (TypeError, ValueError):
+                return None
+
+        return T2TunesStreamInfo(
+            asin=str(item.get("asin") or ""),
+            streamable=bool(streamable),
+            codec=str(stream_info.get("codec") or ""),
+            format=str(stream_info.get("format") or ""),
+            sample_rate=(
+                stream_info.get("sampleRate")
+                if isinstance(stream_info.get("sampleRate"), int)
+                else None
+            ),
+            stream_url=str(stream_info.get("streamUrl") or ""),
+            decryption_key=decryption_key,
+            title=str(tags.get("title") or ""),
+            artist=str(tags.get("artist") or ""),
+            album=str(tags.get("album") or ""),
+            isrc=str(tags.get("isrc") or ""),
+            cover_url=str(item.get("coverUrl") or ""),
+            track_number=_int_tag("trackNumber"),
+            disc_number=_int_tag("discNumber"),
+            genre=str(tags.get("genre") or ""),
+            label=str(tags.get("label") or ""),
+            date=str(tags.get("date") or ""),
+        )

core/amazon_download_client.py

@@ -0,0 +1,466 @@
+"""Amazon Music download source plugin backed by a T2Tunes proxy.
+
+NOT yet wired into the app registry — validated in isolation only.
+See tests/tools/test_amazon_download_client.py.
+
+Filename encoding (the DownloadSourcePlugin dispatch contract):
+    "{asin}||{display_name}"
+    e.g. "B09XYZ1234||Kendrick Lamar - Not Like Us"
+
+Codec preference order: FLAC → Opus → EAC3.
+
+Download flow (from Tubifarry reference implementation):
+    1. GET stream_url → encrypted bytes on disk
+    2. FFmpeg -decryption_key <hex> to decrypt in place
+    3. Embed metadata tags (handled by the app's standard post-processing)
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import threading
+import time
+import uuid
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+import requests as http_requests
+
+from config.settings import config_manager
+from core.amazon_client import AmazonClient, AmazonClientError
+from core.download_plugins.base import DownloadSourcePlugin
+from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
+from core.quality.model import AudioQuality
+from core.quality.source_map import quality_tier_for_source
+from utils.logging_config import get_logger
+
+logger = get_logger("amazon_download_client")
+
+# Quality / codec helpers
+CODEC_PREFERENCE = ["flac", "opus", "eac3"]
+
+_CODEC_EXTENSIONS: Dict[str, str] = {
+    "flac": "flac",
+    "ogg_vorbis": "ogg",
+    "opus": "opus",
+    "eac3": "eac3",
+    "mp4": "m4a",
+    "aac": "m4a",
+    "mp3": "mp3",
+}
+
+MIN_AUDIO_BYTES = 512 * 1024  # 512 KB — anything smaller is a broken stream
+
+
+def _codec_key(codec: str) -> str:
+    return codec.lower().replace("-", "_").replace(" ", "_")
+
+
+def _file_extension(codec: str) -> str:
+    return _CODEC_EXTENSIONS.get(_codec_key(codec), "bin")
+
+
+def _quality_label(codec: str, sample_rate: Optional[int] = None) -> str:
+    ck = _codec_key(codec)
+    if ck == "flac":
+        if sample_rate and sample_rate > 48000:
+            return "Hi-Res"
+        return "Lossless"
+    return "Lossy"
+
+
+class AmazonDownloadClient(DownloadSourcePlugin):
+    """DownloadSourcePlugin — Amazon Music via T2Tunes proxy."""
+
+    def __init__(self, download_path: Optional[str] = None) -> None:
+        if download_path is None:
+            download_path = config_manager.get("soulseek.download_path", "./downloads")
+        self.download_path = Path(download_path)
+        try:
+            self.download_path.mkdir(parents=True, exist_ok=True)
+        except OSError as e:
+            logger.warning(f"Could not verify download path {self.download_path}: {e}")
+
+        self._quality = quality_tier_for_source("amazon", default="flac")
+        self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
+
+        self._client = AmazonClient(preferred_codec=self._quality)
+        self.session = http_requests.Session()
+        self.session.headers.update({
+            "User-Agent": "SoulSync/1.0",
+            "Accept": "*/*",
+        })
+
+        self._engine: Any = None
+        self.shutdown_check: Any = None
+
+    # ------------------------------------------------------------------
+    # DownloadSourcePlugin — lifecycle
+    # ------------------------------------------------------------------
+
+    def set_engine(self, engine) -> None:
+        """Engine callback — wires the central thread worker + state store."""
+        self._engine = engine
+
+    def set_shutdown_check(self, check_callable) -> None:
+        self.shutdown_check = check_callable
+
+    def is_configured(self) -> bool:
+        # T2Tunes has a public default instance; no credentials required.
+        # Return True unconditionally so the source shows as available.
+        return True
+
+    async def check_connection(self) -> bool:
+        try:
+            return self._client.is_authenticated()
+        except Exception:
+            return False
+
+    # ------------------------------------------------------------------
+    # DownloadSourcePlugin — search
+    # ------------------------------------------------------------------
+
+    async def search(
+        self,
+        query: str,
+        timeout: Optional[int] = None,
+        progress_callback: Any = None,
+    ) -> Tuple[List[TrackResult], List[AlbumResult]]:
+        try:
+            items = self._client.search_raw(query, types="track,album")
+        except AmazonClientError as exc:
+            logger.warning(f"Amazon search failed for {query!r}: {exc}")
+            return [], []
+
+        track_results: List[TrackResult] = []
+        album_map: Dict[str, AlbumResult] = {}
+        album_order: List[str] = []
+        preferred = self._client.preferred_codec
+        # Search results only carry the codec (real sample_rate arrives at
+        # stream time). Claim the format honestly — FLAC for the lossless
+        # codec, lossy otherwise — so audio_quality derives a real format
+        # instead of the display label ("Lossless"), and the post-download
+        # probe pins the actual sample_rate/bit_depth.
+        amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
+
+        for item in items:
+            quality = _quality_label(preferred)
+            if item.is_track:
+                tr = TrackResult(
+                    username="amazon",
+                    filename=f"{item.asin}||{item.artist_name} - {item.title}",
+                    size=0,
+                    bitrate=None,
+                    duration=item.duration_seconds * 1000 if item.duration_seconds else None,
+                    quality=quality,
+                    free_upload_slots=999,
+                    upload_speed=999_999,
+                    queue_length=0,
+                    artist=item.artist_name,
+                    title=item.title,
+                    album=item.album_name,
+                    _source_metadata={
+                        "asin": item.asin,
+                        "album_asin": item.album_asin,
+                        "isrc": item.isrc,
+                    },
+                )
+                tr.set_quality(amazon_q)
+                track_results.append(tr)
+            elif item.is_album:
+                album_asin = item.album_asin or item.asin
+                if album_asin not in album_map:
+                    placeholder = TrackResult(
+                        username="amazon",
+                        filename=f"{item.asin}||{item.artist_name} - {item.title}",
+                        size=0,
+                        bitrate=None,
+                        duration=None,
+                        quality=quality,
+                        free_upload_slots=999,
+                        upload_speed=999_999,
+                        queue_length=0,
+                        artist=item.artist_name,
+                        title=item.title,
+                        album=item.album_name,
+                    )
+                    placeholder.set_quality(amazon_q)
+                    album_map[album_asin] = AlbumResult(
+                        username="amazon",
+                        album_path=album_asin,
+                        album_title=item.album_name or item.title,
+                        artist=item.artist_name,
+                        track_count=0,
+                        total_size=0,
+                        tracks=[placeholder],
+                        dominant_quality=quality,
+                    )
+                    album_order.append(album_asin)
+
+        return track_results, [album_map[k] for k in album_order]
+
+    # ------------------------------------------------------------------
+    # DownloadSourcePlugin — download dispatch
+    # ------------------------------------------------------------------
+
+    async def download(
+        self,
+        username: str,
+        filename: str,
+        file_size: int = 0,
+    ) -> Optional[str]:
+        if "||" not in filename:
+            logger.error(f"Invalid Amazon filename format (no '||'): {filename!r}")
+            return None
+        if self._engine is None:
+            raise RuntimeError(
+                "AmazonDownloadClient._engine is not set — "
+                "client not connected to download infrastructure"
+            )
+        asin, display_name = filename.split("||", 1)
+        asin = asin.strip()
+        display_name = display_name.strip()
+        return self._engine.worker.dispatch(
+            source_name="amazon",
+            target_id=asin,
+            display_name=display_name,
+            original_filename=filename,
+            impl_callable=self._download_sync,
+        )
+
+    def _download_sync(
+        self,
+        download_id: str,
+        target_id: str,
+        display_name: str,
+    ) -> Optional[str]:
+        asin = str(target_id)
+        codecs = CODEC_PREFERENCE if self._allow_fallback else [self._quality]
+        for codec in codecs:
+            try:
+                streams = self._client.media_from_asin(asin, codec=codec)
+            except AmazonClientError as exc:
+                logger.warning(f"media_from_asin({asin!r}, {codec}) failed: {exc}")
+                continue
+
+            stream = next(
+                (s for s in streams if s.streamable and s.stream_url),
+                None,
+            )
+            if not stream:
+                logger.debug(f"No streamable result for {asin} at codec={codec}")
+                continue
+
+            ext = _file_extension(stream.codec or codec)
+            safe = "".join(
+                ch if ch.isalnum() or ch in " -_." else "_"
+                for ch in display_name
+            )[:80]
+            # T2Tunes always serves audio in an encrypted MP4 container.
+            # The codec extension (.flac, .opus, .eac3) is only for the
+            # final decrypted output.
+            enc_ext = "mp4" if stream.decryption_key else ext
+            enc_path = self._unique_path(self.download_path / f"{safe}.enc.{enc_ext}")
+            out_path = self._unique_path(self.download_path / f"{safe}.{ext}")
+
+            if self._engine is not None:
+                self._engine.update_record(
+                    "amazon", download_id, {"state": "downloading", "progress": 0.0}
+                )
+            try:
+                downloaded = self._stream_to_file(stream.stream_url, enc_path, download_id)
+            except Exception as exc:
+                logger.warning(f"Stream download failed for {asin} ({codec}): {exc}")
+                enc_path.unlink(missing_ok=True)
+                continue
+
+            if downloaded < MIN_AUDIO_BYTES:
+                logger.warning(
+                    f"File too small ({downloaded} B) for {asin} at {codec} — trying next"
+                )
+                enc_path.unlink(missing_ok=True)
+                continue
+
+            if stream.decryption_key:
+                if self._engine is not None:
+                    self._engine.update_record(
+                        "amazon", download_id, {"state": "decrypting", "progress": 1.0}
+                    )
+                try:
+                    self._decrypt_with_ffmpeg(enc_path, out_path, stream.decryption_key)
+                    enc_path.unlink(missing_ok=True)
+                except Exception as exc:
+                    logger.error(f"Decryption failed for {asin} ({codec}): {exc}")
+                    enc_path.unlink(missing_ok=True)
+                    out_path.unlink(missing_ok=True)
+                    continue
+            else:
+                enc_path.rename(out_path)
+
+            final_size = out_path.stat().st_size
+            logger.info(
+                f"Amazon download complete ({codec}): {out_path} "
+                f"({final_size / (1024 * 1024):.1f} MB)"
+            )
+            # Sync size == transferred so the download monitor's bytes-incomplete
+            # guard doesn't block post-processing. The throttled updates in
+            # _stream_to_file leave transferred < size after the last 0.5s tick;
+            # other streaming clients avoid this by not tracking bytes at all
+            # (size stays 0, the guard is skipped). Writing the final output size
+            # here restores parity.
+            if self._engine is not None:
+                self._engine.update_record(
+                    "amazon", download_id, {'size': final_size, 'transferred': final_size}
+                )
+            return str(out_path)
+
+        logger.error(f"All codec tiers exhausted for '{display_name}' ({asin})")
+        return None
+
+    def _decrypt_with_ffmpeg(
+        self, enc_path: Path, out_path: Path, hex_key: str
+    ) -> None:
+        """Decrypt a T2Tunes encrypted audio file using FFmpeg -decryption_key.
+
+        T2Tunes uses CENC (Common Encryption) for DRM-protected tracks.
+        FFmpeg supports decryption via the -decryption_key flag when the
+        key is provided in hex.
+        """
+        ffmpeg = shutil.which("ffmpeg")
+        if not ffmpeg:
+            tools_dir = Path(__file__).parent.parent / "tools"
+            candidate = tools_dir / ("ffmpeg.exe" if os.name == "nt" else "ffmpeg")
+            if candidate.exists():
+                ffmpeg = str(candidate)
+            else:
+                raise RuntimeError(
+                    "ffmpeg is required for Amazon Music decryption. "
+                    "Install ffmpeg and ensure it is on your PATH."
+                )
+        cmd = [
+            ffmpeg,
+            "-y",
+            "-hide_banner",
+            "-loglevel", "error",
+            "-decryption_key", hex_key,
+            "-i", str(enc_path),
+            "-map", "0:a:0",   # extract first audio stream (FLAC/Opus/EAC3 inside MP4)
+            "-c", "copy",
+            str(out_path),
+        ]
+        logger.debug(f"Decrypting {enc_path.name} → {out_path.name}")
+        result = subprocess.run(cmd, capture_output=True)
+        if result.returncode != 0:
+            stderr = result.stderr.decode("utf-8", errors="replace").strip()
+            raise RuntimeError(f"FFmpeg decryption failed (exit {result.returncode}): {stderr}")
+
+    def _stream_to_file(self, url: str, out_path: Path, download_id: str) -> int:
+        resp = self.session.get(url, stream=True, timeout=60)
+        resp.raise_for_status()
+
+        total = int(resp.headers.get("content-length", 0))
+        downloaded = 0
+        last_report = time.monotonic()
+        shutdown_triggered = False
+
+        with out_path.open("wb") as fh:
+            for chunk in resp.iter_content(chunk_size=64 * 1024):
+                if not chunk:
+                    continue
+                if self.shutdown_check and self.shutdown_check():
+                    shutdown_triggered = True
+                    break
+                fh.write(chunk)
+                downloaded += len(chunk)
+                now = time.monotonic()
+                if self._engine and now - last_report >= 0.5:
+                    self._engine.update_record(
+                        "amazon",
+                        download_id,
+                        {
+                            "transferred": downloaded,
+                            "size": total,
+                            "progress": downloaded / total if total else 0.0,
+                        },
+                    )
+                    last_report = now
+
+        if shutdown_triggered:
+            out_path.unlink(missing_ok=True)
+            raise RuntimeError("Shutdown requested mid-download")
+
+        return downloaded
+
+    # ------------------------------------------------------------------
+    # DownloadSourcePlugin — status interface
+    # ------------------------------------------------------------------
+
+    async def get_all_downloads(self) -> List[DownloadStatus]:
+        if self._engine is None:
+            return []
+        return [
+            self._record_to_status(record)
+            for record in self._engine.iter_records_for_source('amazon')
+        ]
+
+    async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
+        if self._engine is None:
+            return None
+        record = self._engine.get_record('amazon', download_id)
+        return self._record_to_status(record) if record is not None else None
+
+    async def cancel_download(
+        self,
+        download_id: str,
+        username: Optional[str] = None,
+        remove: bool = False,
+    ) -> bool:
+        if self._engine is None:
+            return False
+        if self._engine.get_record('amazon', download_id) is None:
+            return False
+        self._engine.update_record('amazon', download_id, {'state': 'Cancelled'})
+        if remove:
+            self._engine.remove_record('amazon', download_id)
+        return True
+
+    async def clear_all_completed_downloads(self) -> bool:
+        if self._engine is None:
+            return True
+        terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
+        for record in list(self._engine.iter_records_for_source('amazon')):
+            if record.get('state') in terminal:
+                self._engine.remove_record('amazon', record['id'])
+        return True
+
+    # ------------------------------------------------------------------
+    # Private helpers
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _unique_path(path: Path) -> Path:
+        if not path.exists():
+            return path
+        stem, suffix = path.stem, path.suffix
+        for i in range(1, 100):
+            candidate = path.with_name(f"{stem} ({i}){suffix}")
+            if not candidate.exists():
+                return candidate
+        return path.with_name(f"{stem}_{uuid.uuid4().hex[:8]}{suffix}")
+
+    @staticmethod
+    def _record_to_status(rec: Dict[str, Any]) -> DownloadStatus:
+        return DownloadStatus(
+            id=str(rec.get('id', '')),
+            filename=str(rec.get('filename', '')),
+            username='amazon',
+            state=str(rec.get('state', 'queued')),
+            progress=float(rec.get('progress', 0.0)),
+            size=int(rec.get('size', 0)),
+            transferred=int(rec.get('transferred', 0)),
+            speed=int(rec.get('speed', 0)),
+            time_remaining=rec.get('time_remaining'),
+            file_path=rec.get('file_path'),
+        )

core/amazon_outage.py

@@ -0,0 +1,61 @@
+"""Amazon enrichment outage detection + back-off — pure, importable, testable.
+
+The Amazon worker enriches via a public T2Tunes proxy instance. When that
+instance is down (HTTP 5xx, "Amazon Music API is not initialized", or an
+unreachable host), the worker must NOT treat every album as an individual
+failure: doing so floods the logs with an error per item, churns network + DB
+continuously, and permanently marks the whole library ``error`` (which the
+retry tiers never re-attempt) for what is really a transient outage.
+
+Instead it recognizes "the whole source is down", leaves the item untouched so
+it's retried once the instance recovers, and backs off hard. These two pure
+helpers carry that logic so it can be unit-tested without the worker, the DB,
+or the network.
+"""
+
+from __future__ import annotations
+
+import re
+
+# HTTP statuses that mean "the source/proxy is unhealthy", not "no match".
+_OUTAGE_STATUS = {500, 502, 503, 504}
+
+# Substrings (lower-cased) in an error message that indicate a source outage
+# rather than a per-item miss: proxy not ready, gateway errors, the host being
+# unreachable, or an error page returned instead of JSON.
+_OUTAGE_PHRASES = (
+    "not initialized", "not configured", "service unavailable",
+    "bad gateway", "gateway time", "request failed", "response not json",
+    "max retries", "connection", "timed out", "temporarily unavailable",
+)
+
+# Back-off schedule while the source is down.
+_NORMAL_DELAY = 2          # seconds between items when healthy
+_OUTAGE_BASE = 30          # first back-off step
+_OUTAGE_CAP = 1800         # 30 minutes max
+
+
+def is_source_outage(exc: Exception) -> bool:
+    """True when ``exc`` indicates the Amazon source/proxy is down (transient,
+    whole-source), as opposed to a normal per-item error.
+
+    Robust to how the error is surfaced: an explicit ``status_code`` attribute,
+    an ``HTTP <code>`` prefix in the message, or an outage phrase (covers
+    connection failures and non-JSON error pages that carry no status code)."""
+    code = getattr(exc, "status_code", None)
+    if isinstance(code, int) and code in _OUTAGE_STATUS:
+        return True
+    msg = str(exc).lower()
+    m = re.search(r"http\s+(\d{3})", msg)
+    if m and int(m.group(1)) in _OUTAGE_STATUS:
+        return True
+    return any(p in msg for p in _OUTAGE_PHRASES)
+
+
+def next_poll_delay_seconds(outage_streak: int) -> int:
+    """Seconds to wait before the next item. Normal cadence when healthy;
+    escalating back-off (30s, 60s, 120s, … capped at 30 min) the longer the
+    source has been down, so a dead instance can't flood logs/CPU/DB."""
+    if outage_streak <= 0:
+        return _NORMAL_DELAY
+    return min(_OUTAGE_BASE * (2 ** min(outage_streak - 1, 6)), _OUTAGE_CAP)

core/amazon_worker.py

@@ -0,0 +1,645 @@
+import re
+import threading
+import time
+from difflib import SequenceMatcher
+from typing import Optional, Dict, Any
+from datetime import datetime, timedelta
+from utils.logging_config import get_logger
+from database.music_database import MusicDatabase
+from core.amazon_client import AmazonClient
+from core.worker_utils import interruptible_sleep, set_album_api_track_count
+from core.enrichment.manual_match_honoring import honor_stored_match
+from core.amazon_outage import is_source_outage, next_poll_delay_seconds
+
+logger = get_logger("amazon_worker")
+
+
+class AmazonWorker:
+    """Background worker for enriching library artists, albums, and tracks with Amazon Music metadata."""
+
+    def __init__(self, database: MusicDatabase):
+        self.db = database
+        self.client = AmazonClient()
+        self._amazon_schema_checked = False
+
+        self.running = False
+        self.paused = False
+        self.should_stop = False
+        self.thread = None
+        self._stop_event = threading.Event()
+
+        self.current_item = None
+
+        self.stats = {
+            'matched': 0,
+            'not_found': 0,
+            'pending': 0,
+            'errors': 0,
+        }
+
+        self.retry_days = 30
+        self.name_similarity_threshold = 0.80
+
+        # Source-outage circuit breaker: counts consecutive whole-source
+        # failures (proxy down / "not initialized" / unreachable) so the loop
+        # backs off instead of grinding the whole library item-by-item.
+        self._outage_streak = 0
+
+        logger.info("Amazon background worker initialized")
+
+    def _ensure_amazon_schema(self, cursor) -> None:
+        """Ensure upgraded installs have the Amazon enrichment columns.
+
+        MusicDatabase normally runs this migration during startup, but the
+        worker should still be defensive because it is the code path that
+        repeatedly queries these columns in the background.
+        """
+        if self._amazon_schema_checked:
+            return
+
+        table_columns = {
+            'artists': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
+            'albums': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
+            'tracks': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
+        }
+        for table, columns in table_columns.items():
+            cursor.execute(f"PRAGMA table_info({table})")
+            existing = {row[1] for row in cursor.fetchall()}
+            for column in columns:
+                if column not in existing:
+                    column_type = 'TIMESTAMP' if column == 'amazon_last_attempted' else 'TEXT'
+                    cursor.execute(f"ALTER TABLE {table} ADD COLUMN {column} {column_type}")
+
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_id ON artists (amazon_id)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_status ON artists (amazon_match_status)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_id ON albums (amazon_id)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_status ON albums (amazon_match_status)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_id ON tracks (amazon_id)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_status ON tracks (amazon_match_status)")
+        cursor.connection.commit()
+        self._amazon_schema_checked = True
+
+    def start(self):
+        if self.running:
+            logger.warning("Worker already running")
+            return
+
+        self.running = True
+        self.should_stop = False
+        self._stop_event.clear()
+        self.thread = threading.Thread(target=self._run, daemon=True)
+        self.thread.start()
+        logger.info("Amazon background worker started")
+
+    def stop(self):
+        if not self.running:
+            return
+
+        logger.info("Stopping Amazon worker...")
+        self.should_stop = True
+        self.running = False
+        self._stop_event.set()
+
+        if self.thread:
+            self.thread.join(timeout=1)
+
+        logger.info("Amazon worker stopped")
+
+    def pause(self):
+        if not self.running:
+            logger.warning("Worker not running, cannot pause")
+            return
+        self.paused = True
+        logger.info("Amazon worker paused")
+
+    def resume(self):
+        if not self.running:
+            logger.warning("Worker not running, start it first")
+            return
+        self.paused = False
+        logger.info("Amazon worker resumed")
+
+    def get_stats(self) -> Dict[str, Any]:
+        self.stats['pending'] = self._count_pending_items()
+        progress = self._get_progress_breakdown()
+        is_actually_running = self.running and (self.thread is not None and self.thread.is_alive())
+        is_idle = is_actually_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
+        return {
+            'enabled': True,
+            'running': is_actually_running and not self.paused,
+            'paused': self.paused,
+            'idle': is_idle,
+            'current_item': self.current_item,
+            'stats': self.stats.copy(),
+            'progress': progress,
+        }
+
+    def _run(self):
+        logger.info("Amazon worker thread started")
+        while not self.should_stop:
+            try:
+                if self.paused:
+                    interruptible_sleep(self._stop_event, 1)
+                    continue
+
+                self.current_item = None
+                item = self._get_next_item()
+
+                if not item:
+                    logger.debug("No pending items, sleeping...")
+                    interruptible_sleep(self._stop_event, 10)
+                    continue
+
+                self.current_item = item
+                item_id = item.get('id') or item.get('artist_id') or item.get('album_id')
+                if item_id is None:
+                    logger.warning(f"Skipping {item.get('type', 'unknown')} with NULL id: {item.get('name', '?')}")
+                    continue
+
+                self._process_item(item)
+                # Normal 2s cadence when healthy; escalating back-off (up to
+                # 30 min) while the source is in an outage streak.
+                interruptible_sleep(self._stop_event, next_poll_delay_seconds(self._outage_streak))
+
+            except Exception as e:
+                logger.error(f"Error in worker loop: {e}")
+                interruptible_sleep(self._stop_event, 5)
+
+        logger.info("Amazon worker thread finished")
+
+    def _get_next_item(self) -> Optional[Dict[str, Any]]:
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+            self._ensure_amazon_schema(cursor)
+
+            # Pinned-group override (Manage Enrichment Workers): process one
+            # entity type first, then fall through to the normal chain. Unset or
+            # exhausted ⇒ default artist→album→track order, unchanged.
+            from core.worker_utils import read_enrichment_priority, priority_pending_item
+            _prio = read_enrichment_priority('amazon')
+            if _prio:
+                _pi = priority_pending_item(cursor, 'amazon', _prio)
+                if _pi:
+                    return _pi
+
+            # Priority 1: Unattempted artists
+            cursor.execute("""
+                SELECT id, name FROM artists
+                WHERE amazon_match_status IS NULL AND id IS NOT NULL
+                ORDER BY id ASC LIMIT 1
+            """)
+            row = cursor.fetchone()
+            if row:
+                return {'type': 'artist', 'id': row[0], 'name': row[1]}
+
+            # Priority 2: Unattempted albums
+            cursor.execute("""
+                SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
+                FROM albums a
+                JOIN artists ar ON a.artist_id = ar.id
+                WHERE a.amazon_match_status IS NULL AND a.id IS NOT NULL
+                ORDER BY a.id ASC LIMIT 1
+            """)
+            row = cursor.fetchone()
+            if row:
+                return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
+
+            # Priority 3: Unattempted tracks
+            cursor.execute("""
+                SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
+                FROM tracks t
+                JOIN artists ar ON t.artist_id = ar.id
+                WHERE t.amazon_match_status IS NULL AND t.id IS NOT NULL
+                ORDER BY t.id ASC LIMIT 1
+            """)
+            row = cursor.fetchone()
+            if row:
+                return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
+
+            not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
+
+            # Priority 4: Retry not_found artists
+            cursor.execute("""
+                SELECT id, name FROM artists
+                WHERE amazon_match_status = 'not_found' AND amazon_last_attempted < ?
+                ORDER BY amazon_last_attempted ASC LIMIT 1
+            """, (not_found_cutoff,))
+            row = cursor.fetchone()
+            if row:
+                logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
+                return {'type': 'artist', 'id': row[0], 'name': row[1]}
+
+            # Priority 5: Retry not_found albums
+            cursor.execute("""
+                SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
+                FROM albums a
+                JOIN artists ar ON a.artist_id = ar.id
+                WHERE a.amazon_match_status = 'not_found' AND a.amazon_last_attempted < ?
+                ORDER BY a.amazon_last_attempted ASC LIMIT 1
+            """, (not_found_cutoff,))
+            row = cursor.fetchone()
+            if row:
+                return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
+
+            # Priority 6: Retry not_found tracks
+            cursor.execute("""
+                SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
+                FROM tracks t
+                JOIN artists ar ON t.artist_id = ar.id
+                WHERE t.amazon_match_status = 'not_found' AND t.amazon_last_attempted < ?
+                ORDER BY t.amazon_last_attempted ASC LIMIT 1
+            """, (not_found_cutoff,))
+            row = cursor.fetchone()
+            if row:
+                return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
+
+            return None
+
+        except Exception as e:
+            logger.error(f"Error getting next item: {e}")
+            return None
+        finally:
+            if conn:
+                conn.close()
+
+    def _normalize_name(self, name: str) -> str:
+        name = name.lower().strip()
+        name = re.sub(r'\s+[-–—]\s+.*$', '', name)
+        name = re.sub(r'\s*\(.*?\)\s*', ' ', name)
+        name = re.sub(r'[^\w\s]', '', name)
+        name = re.sub(r'\s+', ' ', name).strip()
+        return name
+
+    def _name_matches(self, query_name: str, result_name: str) -> bool:
+        norm_query = self._normalize_name(query_name)
+        norm_result = self._normalize_name(result_name)
+        similarity = SequenceMatcher(None, norm_query, norm_result).ratio()
+        logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
+        return similarity >= self.name_similarity_threshold
+
+    def _process_item(self, item: Dict[str, Any]):
+        try:
+            item_type = item['type']
+            item_id = item['id']
+            item_name = item['name']
+            logger.debug(f"Processing {item_type} #{item_id}: {item_name}")
+
+            if item_type == 'artist':
+                self._process_artist(item_id, item_name)
+            elif item_type == 'album':
+                self._process_album(item_id, item_name, item.get('artist', ''), item)
+            elif item_type == 'track':
+                self._process_track(item_id, item_name, item.get('artist', ''), item)
+
+            # The source answered (match or not_found) — clear any outage streak.
+            if self._outage_streak:
+                logger.info("Amazon source recovered after %d outage(s), resuming",
+                            self._outage_streak)
+                self._outage_streak = 0
+
+        except Exception as e:
+            if is_source_outage(e):
+                # The whole source is down (proxy 5xx / "not initialized" /
+                # unreachable). Do NOT mark the item 'error' — that would burn
+                # the entire library to a state the retry tiers never re-attempt
+                # for a transient outage. Leave it untouched so it's retried once
+                # the instance recovers, and let the loop back off. Log once per
+                # streak to avoid flooding.
+                self._outage_streak += 1
+                if self._outage_streak == 1:
+                    logger.warning("Amazon source unavailable — pausing enrichment "
+                                   "until it recovers: %s", e)
+                else:
+                    logger.debug("Amazon source still unavailable (streak=%d): %s",
+                                 self._outage_streak, e)
+                return
+            # A non-outage error means the source actually answered (e.g. a
+            # 404/parse error on a real response), so the outage is over —
+            # clear the streak and handle this as a normal per-item error.
+            self._outage_streak = 0
+            logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
+            self.stats['errors'] += 1
+            try:
+                self._mark_status(item['type'], item['id'], 'error')
+            except Exception as e2:
+                logger.error(f"Error updating item status: {e2}")
+
+    def _get_existing_id(self, entity_type: str, entity_id: int) -> Optional[str]:
+        table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
+        table = table_map.get(entity_type)
+        if not table:
+            return None
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+            cursor.execute(f"SELECT amazon_id FROM {table} WHERE id = ?", (entity_id,))
+            row = cursor.fetchone()
+            return row[0] if row and row[0] else None
+        except Exception:
+            return None
+        finally:
+            if conn:
+                conn.close()
+
+    def _process_artist(self, artist_id: int, artist_name: str):
+        existing_id = self._get_existing_id('artist', artist_id)
+        if existing_id:
+            logger.debug(f"Preserving existing Amazon ID for artist '{artist_name}': {existing_id}")
+            return
+
+        results = self.client.search_artists(artist_name, limit=5)
+        if results:
+            result = results[0]
+            if self._name_matches(artist_name, result.name):
+                self._update_artist(artist_id, result)
+                self.stats['matched'] += 1
+                logger.info(f"Matched artist '{artist_name}' -> Amazon ID: {result.id}")
+            else:
+                self._mark_status('artist', artist_id, 'not_found')
+                self.stats['not_found'] += 1
+                logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result.name}')")
+        else:
+            self._mark_status('artist', artist_id, 'not_found')
+            self.stats['not_found'] += 1
+            logger.debug(f"No match for artist '{artist_name}'")
+
+    def _refresh_album_via_stored_id(self, album_id, stored_id, api_data):
+        self._update_album(album_id, api_data, stored_id)
+
+    def _refresh_track_via_stored_id(self, track_id, stored_id, api_data):
+        self._update_track(track_id, api_data, stored_id)
+
+    def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
+        if honor_stored_match(
+            db=self.db, entity_table='albums', entity_id=album_id,
+            id_column='amazon_id',
+            client_fetch_fn=lambda asin: self.client.get_album(asin, include_tracks=False),
+            on_match_fn=self._refresh_album_via_stored_id,
+            log_prefix='Amazon',
+        ):
+            self.stats['matched'] += 1
+            return
+
+        query = f"{artist_name} {album_name}"
+        results = self.client.search_albums(query, limit=10)
+        if results:
+            result = results[0]
+            if self._name_matches(album_name, result.name):
+                full_album = None
+                if result.id:
+                    try:
+                        full_album = self.client.get_album(result.id, include_tracks=False)
+                    except Exception as e:
+                        logger.warning(f"Failed to fetch full album '{album_name}' (ASIN: {result.id}): {e}")
+
+                if full_album is None:
+                    self._mark_status('album', album_id, 'error')
+                    self.stats['errors'] += 1
+                    logger.warning(f"Album '{album_name}' matched but full details unavailable, will retry")
+                    return
+
+                self._update_album(album_id, full_album, result.id)
+                self.stats['matched'] += 1
+                logger.info(f"Matched album '{album_name}' -> Amazon ASIN: {result.id}")
+            else:
+                self._mark_status('album', album_id, 'not_found')
+                self.stats['not_found'] += 1
+                logger.debug(f"Name mismatch for album '{album_name}' (got '{result.name}')")
+        else:
+            self._mark_status('album', album_id, 'not_found')
+            self.stats['not_found'] += 1
+            logger.debug(f"No match for album '{album_name}'")
+
+    def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
+        if honor_stored_match(
+            db=self.db, entity_table='tracks', entity_id=track_id,
+            id_column='amazon_id',
+            client_fetch_fn=self.client.get_track_details,
+            on_match_fn=self._refresh_track_via_stored_id,
+            log_prefix='Amazon',
+        ):
+            self.stats['matched'] += 1
+            return
+
+        query = f"{artist_name} {track_name}"
+        results = self.client.search_tracks(query, limit=10)
+        if results:
+            result = results[0]
+            if self._name_matches(track_name, result.name):
+                full_track = None
+                if result.id:
+                    try:
+                        full_track = self.client.get_track_details(result.id)
+                    except Exception as e:
+                        logger.warning(f"Failed to fetch full track '{track_name}' (ASIN: {result.id}): {e}")
+
+                if full_track is None:
+                    self._mark_status('track', track_id, 'error')
+                    self.stats['errors'] += 1
+                    logger.warning(f"Track '{track_name}' matched but full details unavailable, will retry")
+                    return
+
+                self._update_track(track_id, full_track, result.id)
+                self.stats['matched'] += 1
+                logger.info(f"Matched track '{track_name}' -> Amazon ASIN: {result.id}")
+            else:
+                self._mark_status('track', track_id, 'not_found')
+                self.stats['not_found'] += 1
+                logger.debug(f"Name mismatch for track '{track_name}' (got '{result.name}')")
+        else:
+            self._mark_status('track', track_id, 'not_found')
+            self.stats['not_found'] += 1
+            logger.debug(f"No match for track '{track_name}'")
+
+    def _update_artist(self, artist_id: int, result):
+        """Store Amazon metadata for an artist. ``result`` is an Artist dataclass."""
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+
+            cursor.execute("""
+                UPDATE artists SET
+                    amazon_id = ?,
+                    amazon_match_status = 'matched',
+                    amazon_last_attempted = CURRENT_TIMESTAMP,
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = ?
+            """, (str(result.id), artist_id))
+
+            # Backfill thumb_url from album cover stand-in when artist has no image
+            image_url = result.image_url
+            if not image_url:
+                try:
+                    image_url = self.client._get_artist_image_from_albums(result.id)
+                except Exception as exc:
+                    logger.debug("Artist image from albums failed for %s: %s", result.id, exc)
+            if image_url:
+                cursor.execute("""
+                    UPDATE artists SET thumb_url = ?
+                    WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
+                """, (image_url, artist_id))
+
+            conn.commit()
+
+        except Exception as e:
+            logger.error(f"Error updating artist #{artist_id} with Amazon data: {e}")
+            raise
+        finally:
+            if conn:
+                conn.close()
+
+    def _update_album(self, album_id: int, full_data: Dict[str, Any], asin: str):
+        """Store Amazon metadata for an album. ``full_data`` is a get_album() dict."""
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+
+            cursor.execute("""
+                UPDATE albums SET
+                    amazon_id = ?,
+                    amazon_match_status = 'matched',
+                    amazon_last_attempted = CURRENT_TIMESTAMP,
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = ?
+            """, (asin, album_id))
+
+            # Backfill label when missing
+            label = full_data.get('label')
+            if label:
+                cursor.execute("""
+                    UPDATE albums SET label = ?
+                    WHERE id = ? AND (label IS NULL OR label = '')
+                """, (label, album_id))
+
+            # Backfill thumb_url
+            images = full_data.get('images') or []
+            thumb_url = images[0].get('url') if images else None
+            if thumb_url:
+                cursor.execute("""
+                    UPDATE albums SET thumb_url = ?
+                    WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
+                """, (thumb_url, album_id))
+
+            # Cache authoritative track count for completeness repair
+            total_tracks = full_data.get('total_tracks') or (
+                full_data.get('tracks', {}).get('total') if isinstance(full_data.get('tracks'), dict) else None
+            )
+            set_album_api_track_count(cursor, album_id, total_tracks)
+
+            conn.commit()
+
+        except Exception as e:
+            logger.error(f"Error updating album #{album_id} with Amazon data: {e}")
+            raise
+        finally:
+            if conn:
+                conn.close()
+
+    def _update_track(self, track_id: int, full_data: Dict[str, Any], asin: str):
+        """Store Amazon metadata for a track. ``full_data`` is a get_track_details() dict."""
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+
+            cursor.execute("""
+                UPDATE tracks SET
+                    amazon_id = ?,
+                    amazon_match_status = 'matched',
+                    amazon_last_attempted = CURRENT_TIMESTAMP
+                WHERE id = ?
+            """, (asin, track_id))
+
+            conn.commit()
+
+        except Exception as e:
+            logger.error(f"Error updating track #{track_id} with Amazon data: {e}")
+            raise
+        finally:
+            if conn:
+                conn.close()
+
+    def _mark_status(self, entity_type: str, entity_id: int, status: str):
+        table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
+        table = table_map.get(entity_type)
+        if not table:
+            logger.error(f"Unknown entity type: {entity_type}")
+            return
+
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+            cursor.execute(f"""
+                UPDATE {table} SET
+                    amazon_match_status = ?,
+                    amazon_last_attempted = CURRENT_TIMESTAMP,
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = ?
+            """, (status, entity_id))
+            conn.commit()
+        except Exception as e:
+            logger.error(f"Error marking {entity_type} #{entity_id} status: {e}")
+        finally:
+            if conn:
+                conn.close()
+
+    def _count_pending_items(self) -> int:
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+            self._ensure_amazon_schema(cursor)
+            cursor.execute("""
+                SELECT
+                    (SELECT COUNT(*) FROM artists WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
+                    (SELECT COUNT(*) FROM albums  WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
+                    (SELECT COUNT(*) FROM tracks  WHERE amazon_match_status IS NULL AND id IS NOT NULL)
+                AS pending
+            """)
+            row = cursor.fetchone()
+            return row[0] if row else 0
+        except Exception as e:
+            logger.error(f"Error counting pending items: {e}")
+            return 0
+        finally:
+            if conn:
+                conn.close()
+
+    def _get_progress_breakdown(self) -> Dict[str, Dict[str, int]]:
+        conn = None
+        try:
+            conn = self.db._get_connection()
+            cursor = conn.cursor()
+            self._ensure_amazon_schema(cursor)
+            progress = {}
+
+            for table in ('artists', 'albums', 'tracks'):
+                cursor.execute(f"""
+                    SELECT
+                        COUNT(*) AS total,
+                        SUM(CASE WHEN amazon_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
+                    FROM {table}
+                """)
+                row = cursor.fetchone()
+                if row:
+                    total, processed = row[0], row[1] or 0
+                    progress[table] = {
+                        'matched': processed,
+                        'total': total,
+                        'percent': int((processed / total * 100) if total > 0 else 0),
+                    }
+
+            return progress
+
+        except Exception as e:
+            logger.error(f"Error getting progress breakdown: {e}")
+            return {}
+        finally:
+            if conn:
+                conn.close()

core/api_call_tracker.py

@@ -30,6 +30,7 @@ RATE_LIMITS = {
     'tidal': 120,         # MIN_API_INTERVAL=0.5s → ~120/min
     'qobuz': 60,          # Variable throttle, ~60/min estimate
     'discogs': 60,        # MIN_API_INTERVAL=1.0s with auth → ~60/min
+    'amazon': 120,        # MIN_API_INTERVAL=0.5s → ~120/min (T2Tunes proxy)
 }
 
 # Display names for UI
@@ -44,12 +45,13 @@ SERVICE_LABELS = {
     'tidal': 'Tidal',
     'qobuz': 'Qobuz',
     'discogs': 'Discogs',
+    'amazon': 'Amazon Music',
 }
 
 # Display order
 SERVICE_ORDER = [
     'spotify', 'itunes', 'deezer', 'lastfm', 'genius',
-    'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs',
+    'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs', 'amazon',
 ]
 
 
@@ -302,8 +304,8 @@ class ApiCallTracker:
             try:
                 if os.path.exists(_PERSIST_PATH + '.tmp'):
                     os.remove(_PERSIST_PATH + '.tmp')
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("remove stale tmp file failed: %s", e)
 
     def _load(self):
         """Restore 24h minute history from disk. Called on init."""

core/archive_pipeline.py

@@ -0,0 +1,238 @@
+"""Archive extraction + audio-file discovery for torrent / usenet downloads.
+
+The torrent and usenet download plugins need a uniform way to:
+
+1. Walk the downloader's save directory and find every audio file in it.
+2. If the directory contains an archive (``.zip`` / ``.rar`` / ``.tar`` /
+   ``.7z``), extract it first so the audio files inside become walkable.
+
+This module is intentionally narrow — no matching, no tagging, no
+import. The download plugin layer composes this with the existing
+post-processing / matching pipeline. Lidarr does NOT use this module:
+Lidarr extracts archives in its own import step before SoulSync sees
+the files at all. Usenet downloaders (SABnzbd, NZBGet) also auto-
+extract by default. Torrents are the main case where SoulSync may
+need to do the extract step itself — most music torrents ship loose,
+but some bundle the album in a ``.rar`` archive.
+
+``rarfile`` is an optional dependency. If it isn't installed, archives
+with ``.rar`` content are skipped with a single warning rather than
+crashing the download.
+"""
+
+from __future__ import annotations
+
+import tarfile
+import zipfile
+from pathlib import Path
+from typing import List, Optional
+
+from utils.logging_config import get_logger
+
+logger = get_logger("archive_pipeline")
+
+
+# Same audio-extension set as ``core/imports/file_ops.py`` ``quality_tiers``.
+# Keep them in sync — if a new format is added to file_ops, add it here too
+# or the walker will skip it and the download plugin will mark the download
+# failed even when files arrived.
+AUDIO_EXTENSIONS = frozenset([
+    # lossless
+    '.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
+    # high lossy
+    '.opus', '.ogg',
+    # standard lossy
+    '.m4a', '.aac',
+    # low lossy
+    '.mp3', '.wma',
+])
+
+ARCHIVE_EXTENSIONS = frozenset(['.zip', '.rar', '.tar', '.tar.gz', '.tgz', '.7z'])
+
+
+def is_archive(path: Path) -> bool:
+    """True if the file extension looks like a supported archive.
+
+    Compound extensions (``.tar.gz``, ``.tar.bz2``) are detected by
+    checking the last two suffixes joined together — Path.suffix
+    only returns the final suffix.
+    """
+    if not path.is_file():
+        return False
+    name = path.name.lower()
+    if name.endswith(('.tar.gz', '.tar.bz2', '.tar.xz')):
+        return True
+    return path.suffix.lower() in ARCHIVE_EXTENSIONS
+
+
+def walk_audio_files(directory: Path) -> List[Path]:
+    """Recursively scan ``directory`` for audio files. Returns
+    a sorted list of absolute paths. Empty list if the directory
+    doesn't exist or contains no audio.
+    """
+    if not directory or not directory.exists() or not directory.is_dir():
+        return []
+    out: List[Path] = []
+    for child in directory.rglob('*'):
+        if not child.is_file():
+            continue
+        if child.suffix.lower() in AUDIO_EXTENSIONS:
+            out.append(child.resolve())
+    out.sort()
+    return out
+
+
+def find_archives_in_dir(directory: Path) -> List[Path]:
+    """Find every archive file directly inside ``directory`` (one
+    level deep — torrents normally put the archive at the root of
+    their folder; we don't search nested dirs to avoid extracting
+    something we shouldn't).
+    """
+    if not directory or not directory.exists() or not directory.is_dir():
+        return []
+    return sorted(p for p in directory.iterdir() if is_archive(p))
+
+
+def extract_archive(archive_path: Path, extract_to: Optional[Path] = None) -> Optional[Path]:
+    """Extract a single archive in-place (or to ``extract_to`` if
+    given). Returns the directory the archive was extracted into,
+    or ``None`` on failure.
+
+    Supports ``.zip``, ``.tar``/``.tar.gz``/``.tar.bz2``/``.tar.xz``,
+    and ``.rar`` (only when the optional ``rarfile`` library is
+    installed). ``.7z`` is recognised but extraction requires
+    ``py7zr``; without it, the call logs and returns None.
+    """
+    if not archive_path or not archive_path.exists():
+        logger.warning("archive_pipeline: %s does not exist", archive_path)
+        return None
+    dest = extract_to or archive_path.parent
+    dest.mkdir(parents=True, exist_ok=True)
+
+    name = archive_path.name.lower()
+    try:
+        if name.endswith('.zip'):
+            with zipfile.ZipFile(archive_path) as zf:
+                _safe_extract_zip(zf, dest)
+            return dest
+        if name.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
+            with tarfile.open(archive_path) as tf:
+                _safe_extract_tar(tf, dest)
+            return dest
+        if name.endswith('.rar'):
+            return _extract_rar(archive_path, dest)
+        if name.endswith('.7z'):
+            return _extract_7z(archive_path, dest)
+    except (zipfile.BadZipFile, tarfile.TarError, OSError) as e:
+        logger.error("archive_pipeline: failed to extract %s: %s", archive_path, e)
+        return None
+    logger.warning("archive_pipeline: unknown archive type for %s", archive_path)
+    return None
+
+
+def extract_all_in_dir(directory: Path) -> List[Path]:
+    """Find every archive in ``directory`` and extract each in place.
+    Returns the list of directories archives were extracted into
+    (usually all the same — ``directory`` itself). Archives that
+    failed to extract are skipped silently after a warning.
+    """
+    out: List[Path] = []
+    for archive in find_archives_in_dir(directory):
+        result = extract_archive(archive)
+        if result is not None:
+            out.append(result)
+    return out
+
+
+def collect_audio_after_extraction(directory: Path) -> List[Path]:
+    """One-shot helper for the download plugins: extract any archives
+    in the directory, then return the walked audio file list. This is
+    the common pattern — torrent / usenet plugin gets a save_path,
+    calls this, hands the resulting files to the matching pipeline.
+    """
+    extract_all_in_dir(directory)
+    return walk_audio_files(directory)
+
+
+# ---------------------------------------------------------------------------
+# Safety helpers
+# ---------------------------------------------------------------------------
+
+
+def _safe_extract_zip(zf: zipfile.ZipFile, dest: Path) -> None:
+    """Extract a zipfile after rejecting any member whose resolved
+    path escapes ``dest`` (path traversal protection).
+    """
+    dest = dest.resolve()
+    for member in zf.namelist():
+        target = (dest / member).resolve()
+        if dest not in target.parents and target != dest:
+            logger.error("archive_pipeline: refusing path-traversal member %r", member)
+            return
+    zf.extractall(dest)
+
+
+def _safe_extract_tar(tf: tarfile.TarFile, dest: Path) -> None:
+    """Same path-traversal protection for tarfiles."""
+    dest = dest.resolve()
+    for member in tf.getmembers():
+        target = (dest / member.name).resolve()
+        if dest not in target.parents and target != dest:
+            logger.error("archive_pipeline: refusing path-traversal member %r", member.name)
+            return
+    # ``filter='data'`` is the Python 3.12+ safe extractor; fall back
+    # to the legacy call on older runtimes.
+    try:
+        tf.extractall(dest, filter='data')  # type: ignore[call-arg]
+    except TypeError:
+        tf.extractall(dest)
+
+
+def _extract_rar(archive_path: Path, dest: Path) -> Optional[Path]:
+    try:
+        import rarfile  # type: ignore[import-untyped]
+    except ImportError:
+        logger.warning(
+            "archive_pipeline: cannot extract %s — rarfile library not installed. "
+            "Install with: pip install rarfile (and ensure unrar is on PATH).",
+            archive_path,
+        )
+        return None
+    try:
+        with rarfile.RarFile(archive_path) as rf:
+            dest_resolved = dest.resolve()
+            for name in rf.namelist():
+                target = (dest_resolved / name).resolve()
+                if dest_resolved not in target.parents and target != dest_resolved:
+                    logger.error("archive_pipeline: refusing path-traversal rar member %r", name)
+                    return None
+            rf.extractall(dest)
+        return dest
+    except Exception as e:
+        logger.error("archive_pipeline: rar extract failed for %s: %s", archive_path, e)
+        return None
+
+
+def _extract_7z(archive_path: Path, dest: Path) -> Optional[Path]:
+    try:
+        import py7zr  # type: ignore[import-untyped]
+    except ImportError:
+        logger.warning(
+            "archive_pipeline: cannot extract %s — py7zr library not installed. "
+            "Install with: pip install py7zr.",
+            archive_path,
+        )
+        return None
+    try:
+        with py7zr.SevenZipFile(archive_path, 'r') as sz:
+            dest_resolved = dest.resolve()
+            for name in sz.getnames():
+                target = (dest_resolved / name).resolve()
+                if dest_resolved not in target.parents and target != dest_resolved:
+                    logger.error("archive_pipeline: refusing path-traversal 7z member %r", name)
+                    return None
+            sz.extractall(path=dest)
+        return dest
+    except Exception as e:
+        logger.error("archive_pipeline: 7z extract failed for %s: %s", archive_path, e)
+        return None

core/artist_source_detail.py

@@ -9,7 +9,7 @@ Used by ``/api/artist-detail/<id>`` when the URL is called with a ``source``
 query parameter and the library DB lookup misses. Enriches the response with
 whatever metadata we can pull on demand:
 
-  * Image URL (via ``metadata_service.get_artist_image_url``)
+  * Image URL (via ``core.metadata.artist_image.get_artist_image_url``)
   * Source-specific artist info — genres + follower count from the named
     source's ``get_artist`` / ``get_artist_info`` helper
   * Last.fm bio + listeners + playcount + URL (by artist name)
@@ -27,6 +27,9 @@ import logging
 from typing import Any, Dict, Optional, Tuple
 
 from core.artist_source_lookup import SOURCE_ID_FIELD
+from core.metadata import artist_image as metadata_artist_image
+from core.metadata import discography as metadata_discography
+from core.metadata.lookup import MetadataLookupOptions
 
 logger = logging.getLogger("artist_source_detail")
 
@@ -40,6 +43,7 @@ def build_source_only_artist_detail(
     deezer_client: Optional[Any] = None,
     itunes_client: Optional[Any] = None,
     discogs_client: Optional[Any] = None,
+    amazon_client: Optional[Any] = None,
     lastfm_api_key: Optional[str] = None,
 ) -> Tuple[Dict[str, Any], int]:
     """Build the artist-detail payload for a source-only artist.
@@ -48,21 +52,12 @@ def build_source_only_artist_detail(
     ``jsonify`` or equivalent. Status is 200 on success, 404 when the
     source's discography lookup returned no releases.
     """
-    # Deferred import — keeps the top-level module importable in test rigs
-    # that stub out only what they need (same pattern `_find_library_artist`
-    # uses).
-    from core.metadata_service import (
-        MetadataLookupOptions,
-        get_artist_detail_discography,
-        get_artist_image_url,
-    )
-
-    resolved_name = (artist_name or artist_id or "").strip()
+    resolved_name = (artist_name or "").strip()
 
     # 1. Image URL via the same helper /api/artist/<id>/image uses.
     image_url: Optional[str] = None
     try:
-        image_url = get_artist_image_url(artist_id, source_override=source)
+        image_url = metadata_artist_image.get_artist_image_url(artist_id, source_override=source)
     except Exception as e:
         logger.debug(f"Artist image lookup failed for {source}:{artist_id}: {e}")
 
@@ -73,6 +68,8 @@ def build_source_only_artist_detail(
         if source == "spotify" and spotify_client is not None:
             sp_artist = spotify_client.get_artist(artist_id, allow_fallback=False)
             if sp_artist:
+                if not artist_name and sp_artist.get("name"):
+                    resolved_name = sp_artist["name"]
                 source_genres = sp_artist.get("genres") or []
                 source_followers = (sp_artist.get("followers") or {}).get("total")
                 if not image_url and sp_artist.get("images"):
@@ -80,16 +77,41 @@ def build_source_only_artist_detail(
         elif source == "deezer" and deezer_client is not None:
             dz_artist = deezer_client.get_artist_info(artist_id)
             if dz_artist:
+                if not artist_name and dz_artist.get("name"):
+                    resolved_name = dz_artist["name"]
                 source_genres = dz_artist.get("genres") or []
                 source_followers = (dz_artist.get("followers") or {}).get("total")
         elif source == "itunes" and itunes_client is not None:
             it_artist = itunes_client.get_artist(artist_id)
             if it_artist:
+                if not artist_name and it_artist.get("name"):
+                    resolved_name = it_artist["name"]
                 source_genres = it_artist.get("genres") or []
         elif source == "discogs" and discogs_client is not None:
             dc_artist = discogs_client.get_artist(artist_id)
             if dc_artist:
+                if not artist_name and dc_artist.get("name"):
+                    resolved_name = dc_artist["name"]
                 source_genres = dc_artist.get("genres") or []
+        elif source == "amazon" and amazon_client is not None:
+            az_artist = amazon_client.get_artist(resolved_name or artist_id)
+            if az_artist:
+                if not artist_name and az_artist.get("name"):
+                    resolved_name = az_artist["name"]
+                source_genres = az_artist.get("genres") or []
+                if not image_url and az_artist.get("images"):
+                    image_url = az_artist["images"][0].get("url")
+        elif source == "musicbrainz":
+            try:
+                from core.musicbrainz_search import MusicBrainzSearchClient
+                mb = MusicBrainzSearchClient()
+                mb_artist = mb.get_artist(artist_id)
+                if mb_artist:
+                    if not artist_name and mb_artist.get("name"):
+                        resolved_name = mb_artist["name"]
+                    source_genres = mb_artist.get("genres") or []
+            except Exception as e:
+                logger.debug(f"MusicBrainz artist info lookup failed for {artist_id}: {e}")
     except Exception as e:
         logger.debug(f"Source-side artist info lookup failed for {source}:{artist_id}: {e}")
 
@@ -124,7 +146,7 @@ def build_source_only_artist_detail(
     # 4. Discography from the specified source. Skip variant dedup so the
     #    page shows every release the source returns — matches the inline
     #    Artists-page behaviour that this view was modelled after.
-    discography_result = get_artist_detail_discography(
+    discography_result = metadata_discography.get_artist_detail_discography(
         artist_id,
         artist_name=resolved_name or artist_id,
         options=MetadataLookupOptions(
@@ -132,7 +154,11 @@ def build_source_only_artist_detail(
             allow_fallback=True,
             skip_cache=False,
             max_pages=0,
-            limit=50,
+            # Match the Download Discography endpoint cap (200).
+            # Spotify already paginates all; Deezer / iTunes / Discogs /
+            # Hydrabase clamp at the outer limit. 200 covers prolific
+            # catalogues without exceeding iTunes/Discogs internal caps.
+            limit=200,
             artist_source_ids={source: artist_id},
             dedup_variants=False,
         ),

core/artist_source_lookup.py

@@ -21,21 +21,24 @@ from __future__ import annotations
 import logging
 from typing import Optional
 
+from core.source_ids import id_column as _artist_id_column
+
 logger = logging.getLogger("artist_source_lookup")
 
 
 SOURCE_ONLY_ARTIST_SOURCES = frozenset({
-    "spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz",
+    "spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
 })
 
 
+# The per-source column on the ``artists`` table, derived from the canonical
+# source-ID registry (the single source of truth). Values are unchanged from the
+# previous hardcoded map — this just stops duplicating that knowledge here.
 SOURCE_ID_FIELD = {
-    "spotify": "spotify_artist_id",
-    "itunes": "itunes_artist_id",
-    "deezer": "deezer_id",
-    "discogs": "discogs_id",
-    "hydrabase": "soul_id",
-    "musicbrainz": "musicbrainz_id",
+    source: _artist_id_column(source, "artist")
+    for source in (
+        "spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
+    )
 }
 
 
@@ -50,7 +53,11 @@ def find_library_artist_for_source(
 
     Lookup order:
       1. Direct match on the source-specific ID column (server-agnostic — any
-         library record with the right external ID is a hit).
+         library record with the right external ID is a hit). If that id is
+         stamped on MORE than one library artist, the mapping is corrupt /
+         ambiguous (e.g. an enrichment bug wrote one Deezer id onto several
+         artists) — we refuse to guess and fall through, so the caller can
+         show the source artist directly instead of an arbitrary wrong one.
       2. Case-insensitive name match within ``active_server`` (defaults to the
          active media server when not provided), so we don't jump the user
          across server contexts on a name collision.
@@ -64,13 +71,23 @@ def find_library_artist_for_source(
     try:
         with database._get_connection() as conn:
             cursor = conn.cursor()
+            # LIMIT 2 so we can tell a unique match from an ambiguous one.
             cursor.execute(
-                f"SELECT id, name FROM artists WHERE {column} = ? LIMIT 1",
+                f"SELECT id FROM artists WHERE {column} = ? LIMIT 2",
                 (str(source_artist_id),),
             )
-            row = cursor.fetchone()
-            if row:
-                return row[0]
+            rows = cursor.fetchall()
+            if len(rows) == 1:
+                return rows[0][0]
+            if len(rows) > 1:
+                # Same source id on multiple artists — corrupt mapping. Don't
+                # upgrade on the id; fall through to the name match (and, if
+                # that misses, let the caller render the source artist).
+                logger.warning(
+                    f"Source id {source}:{source_artist_id} maps to "
+                    f"{len(rows)}+ library artists — ambiguous, skipping "
+                    f"id-based library upgrade"
+                )
 
             if artist_name and active_server:
                 cursor.execute(

core/artists/liked_match.py

@@ -0,0 +1,324 @@
+"""Liked-artist multi-source matching — lifted from web_server.py.
+
+Both function bodies are byte-identical to the originals. The
+``spotify_client`` proxy + ``_get_*_client`` shims let the bodies resolve
+their original names without any modification.
+"""
+import logging
+import time
+
+from config.settings import config_manager
+from core.metadata.registry import (
+    get_deezer_client,
+    get_discogs_client,
+    get_itunes_client,
+    get_musicbrainz_client,
+    get_spotify_client,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def _get_itunes_client():
+    """Mirror of web_server._get_itunes_client — delegates to registry."""
+    return get_itunes_client()
+
+
+def _get_deezer_client():
+    """Mirror of web_server._get_deezer_client — delegates to registry."""
+    return get_deezer_client()
+
+
+def _get_discogs_client(token=None):
+    """Mirror of web_server._get_discogs_client — delegates to registry."""
+    return get_discogs_client(token)
+
+
+def _get_musicbrainz_client():
+    """Mirror of web_server._get_musicbrainz_client — delegates to registry."""
+    return get_musicbrainz_client()
+
+
+class _SpotifyClientProxy:
+    """Resolves the global Spotify client lazily so a Spotify re-auth that
+    rebinds the cached client in core.metadata.registry is visible to the
+    lifted bodies."""
+
+    def __getattr__(self, name):
+        client = get_spotify_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        return get_spotify_client() is not None
+
+
+spotify_client = _SpotifyClientProxy()
+
+
+def _match_liked_artists_to_all_sources(database, profile_id: int):
+    """Match pending liked artists to ALL metadata sources (Spotify, iTunes, Deezer, Discogs).
+    Uses the same matching pattern as the watchlist scanner: DB-first, then API search
+    with fuzzy name matching. Stores all resolved IDs so source switching works instantly."""
+    pending = database.get_liked_artists_pending_match(profile_id, limit=200)
+    if not pending:
+        return
+
+    # Source → column mapping
+    source_cols = {
+        'spotify': 'spotify_artist_id',
+        'itunes': 'itunes_artist_id',
+        'deezer': 'deezer_artist_id',
+        'discogs': 'discogs_artist_id',
+        'musicbrainz': 'musicbrainz_artist_id',
+    }
+    id_cols = list(source_cols.values())
+
+    # Reject known placeholder images and local server paths
+    _placeholder_hashes = {'2a96cbd8b46e442fc41c2b86b821562f'}
+    def _valid_image(url):
+        if not url or not url.strip():
+            return None
+        if any(ph in url for ph in _placeholder_hashes):
+            return None
+        # Reject local media server paths (Plex/Jellyfin) — not loadable in browser
+        if url.startswith('/') or url.startswith('\\'):
+            return None
+        if not url.startswith('http'):
+            return None
+        return url
+
+    # Build search clients for each source
+    from core.deezer_client import DeezerClient
+    search_clients = {}
+    if spotify_client and spotify_client.is_spotify_authenticated():
+        search_clients['spotify'] = spotify_client
+    try:
+        search_clients['itunes'] = _get_itunes_client()
+    except Exception as e:
+        logger.debug("itunes client init failed: %s", e)
+    try:
+        search_clients['deezer'] = _get_deezer_client()
+    except Exception as e:
+        logger.debug("deezer client init failed: %s", e)
+    try:
+        dc = _get_discogs_client()
+        # Only use Discogs if token is configured
+        from config.settings import config_manager as _cm
+        if _cm.get('discogs.token', ''):
+            search_clients['discogs'] = dc
+    except Exception as e:
+        logger.debug("discogs client init failed: %s", e)
+    try:
+        search_clients['musicbrainz'] = _get_musicbrainz_client()
+    except Exception as e:
+        logger.debug("musicbrainz client init failed: %s", e)
+
+    # Reuse watchlist scanner's fuzzy matching logic
+    from core.watchlist_scanner import WatchlistScanner
+    _normalize = WatchlistScanner._normalize_artist_name
+
+    def _best_match(results, artist_name):
+        """Pick best match from search results using name similarity (same as watchlist scanner)."""
+        if not results:
+            return None
+        # Exact normalized match
+        for r in results:
+            if _normalize(r.name) == _normalize(artist_name):
+                return r
+        # Fuzzy scoring
+        best = None
+        best_sim = 0
+        for r in results:
+            # Simple normalized comparison
+            n1 = _normalize(artist_name)
+            n2 = _normalize(r.name)
+            if n1 == n2:
+                return r
+            # Levenshtein-style similarity
+            max_len = max(len(n1), len(n2))
+            if max_len == 0:
+                continue
+            distance = sum(1 for a, b in zip(n1, n2, strict=False) if a != b) + abs(len(n1) - len(n2))
+            sim = (max_len - distance) / max_len
+            if sim > best_sim:
+                best_sim = sim
+                best = r
+        if best and best_sim >= 0.85:
+            return best
+        return None
+
+    api_calls = 0
+    matched = 0
+
+    for entry in pending:
+        name = entry['artist_name']
+        pool_id = entry['id']
+        harvested_ids = {}
+        best_image = None
+
+        # Pre-load existing IDs from the entry itself
+        for col in id_cols:
+            if entry.get(col):
+                harvested_ids[col] = entry[col]
+
+        # --- DB STRATEGIES (free, no API calls) ---
+
+        # 1. Library artists table
+        try:
+            conn = database._get_connection()
+            cursor = conn.cursor()
+            cursor.execute("SELECT * FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (name,))
+            row = cursor.fetchone()
+            if row:
+                r = dict(row)
+                for col in id_cols:
+                    if r.get(col) and col not in harvested_ids:
+                        harvested_ids[col] = str(r[col])
+                if _valid_image(r.get('thumb_url')):
+                    best_image = r['thumb_url']
+        except Exception as e:
+            logger.debug("library artist lookup failed: %s", e)
+
+        # 2. Watchlist artists
+        try:
+            conn = database._get_connection()
+            cursor = conn.cursor()
+            cursor.execute(
+                "SELECT * FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE AND profile_id = ? LIMIT 1",
+                (name, profile_id)
+            )
+            row = cursor.fetchone()
+            if row:
+                wl = dict(row)
+                for col in id_cols:
+                    if wl.get(col) and col not in harvested_ids:
+                        harvested_ids[col] = str(wl[col])
+                if _valid_image(wl.get('image_url')) and not best_image:
+                    best_image = wl['image_url']
+        except Exception as e:
+            logger.debug("watchlist artist lookup failed: %s", e)
+
+        # 3. Metadata cache (all sources)
+        try:
+            conn = database._get_connection()
+            cursor = conn.cursor()
+            cursor.execute(
+                "SELECT entity_id, source, image_url FROM metadata_cache_entities WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE",
+                (name,)
+            )
+            for row in cursor.fetchall():
+                col = source_cols.get(row['source'])
+                if col and col not in harvested_ids:
+                    harvested_ids[col] = row['entity_id']
+                if _valid_image(row['image_url']) and not best_image:
+                    best_image = row['image_url']
+        except Exception as e:
+            logger.debug("metadata cache lookup failed: %s", e)
+
+        # --- API STRATEGIES (search each missing source) ---
+        # Same pattern as watchlist scanner's _backfill_missing_ids
+        for source, col in source_cols.items():
+            if col in harvested_ids:
+                continue  # Already have this source's ID
+            client = search_clients.get(source)
+            if not client:
+                continue
+            if api_calls >= 200:  # Hard cap per refresh cycle
+                break
+            try:
+                results = client.search_artists(name, limit=5)
+                best = _best_match(results, name)
+                if best:
+                    harvested_ids[col] = best.id
+                    if hasattr(best, 'image_url') and _valid_image(best.image_url) and not best_image:
+                        best_image = best.image_url
+                api_calls += 1
+                time.sleep(0.4)  # Rate limit breathing room
+            except Exception as e:
+                logger.debug(f"[Your Artists] {source} search failed for '{name}': {e}")
+                api_calls += 1
+
+        # Save all harvested IDs
+        if harvested_ids:
+            # Determine best active source/ID — prefer Spotify, then iTunes, Deezer, Discogs
+            resolved_source = None
+            resolved_id = None
+            for src in ('spotify', 'itunes', 'deezer', 'discogs', 'musicbrainz'):
+                col = source_cols[src]
+                if col in harvested_ids:
+                    resolved_source = src
+                    resolved_id = harvested_ids[col]
+                    break
+
+            database.update_liked_artist_match(
+                pool_id, active_source=resolved_source, active_source_id=resolved_id,
+                image_url=best_image, all_ids=harvested_ids
+            )
+            matched += 1
+
+    database.sync_liked_artists_watchlist_flags(profile_id)
+    logger.info(f"[Your Artists] Matched {matched}/{len(pending)} artists to {len(search_clients)} sources ({api_calls} API calls)")
+
+    # Image backfill: fetch images for matched artists that have IDs but no image
+    _backfill_liked_artist_images(database, profile_id, search_clients)
+
+
+def _backfill_liked_artist_images(database, profile_id: int, search_clients: dict):
+    """Fetch images for matched artists missing artwork using their stored source IDs."""
+    try:
+        conn = database._get_connection()
+        cursor = conn.cursor()
+        cursor.execute("""
+            SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id
+            FROM liked_artists_pool
+            WHERE profile_id = ? AND match_status = 'matched'
+              AND (image_url IS NULL OR image_url = ''
+                   OR image_url LIKE '%2a96cbd8b46e442fc41c2b86b821562f%'
+                   OR image_url NOT LIKE 'http%')
+            LIMIT 100
+        """, (profile_id,))
+        rows = cursor.fetchall()
+        if not rows:
+            return
+
+        logger.info(f"[Your Artists] Backfilling images for {len(rows)} artists...")
+        filled = 0
+
+        for row in rows:
+            r = dict(row)
+            image_url = None
+
+            # Try Spotify artist lookup (has best images)
+            if r.get('spotify_artist_id') and 'spotify' in search_clients:
+                try:
+                    sp = search_clients['spotify']
+                    if hasattr(sp, 'sp') and sp.sp:
+                        artist_data = sp.sp.artist(r['spotify_artist_id'])
+                        if artist_data and artist_data.get('images'):
+                            image_url = artist_data['images'][0]['url']
+                except Exception as e:
+                    logger.debug("spotify artist image fetch failed: %s", e)
+
+            # Try Deezer (direct image URL from ID)
+            if not image_url and r.get('deezer_artist_id'):
+                image_url = f"https://api.deezer.com/artist/{r['deezer_artist_id']}/image?size=big"
+
+            if image_url:
+                try:
+                    cursor2 = conn.cursor()
+                    cursor2.execute(
+                        "UPDATE liked_artists_pool SET image_url = ? WHERE id = ?",
+                        (image_url, r['id'])
+                    )
+                    filled += 1
+                except Exception as e:
+                    logger.debug("liked artist image update failed: %s", e)
+                time.sleep(0.3)
+
+        conn.commit()
+        if filled:
+            logger.info(f"[Your Artists] Backfilled {filled}/{len(rows)} artist images")
+    except Exception as e:
+        logger.debug(f"[Your Artists] Image backfill error: {e}")

core/artists/map.py

@@ -0,0 +1,994 @@
+"""Artist Map endpoints — lifted from web_server.py.
+
+The four route bodies (``get_artist_map_data``, ``get_artist_map_genre_list``,
+``get_artist_map_genres``, ``get_artist_map_explore``) plus their cache helpers
+and the artist-map cache are byte-identical to the originals. Module-level
+shims for ``get_current_profile_id``, ``_get_itunes_client``, and the
+``spotify_client`` proxy let the bodies resolve their original names without
+modification.
+"""
+import json
+import logging
+import time
+
+from flask import g, jsonify, request
+
+from database.music_database import get_database
+from core.metadata.registry import get_itunes_client, get_spotify_client
+
+logger = logging.getLogger(__name__)
+
+
+def get_current_profile_id() -> int:
+    """Mirror of web_server.get_current_profile_id — uses Flask g.
+
+    Catches RuntimeError too because reading `g` outside a request
+    context raises that (not AttributeError) — happens when this is
+    called from background threads (sync, automation, scanners)."""
+    try:
+        return g.profile_id
+    except (AttributeError, RuntimeError):
+        return 1
+
+
+def _get_itunes_client():
+    """Mirror of web_server._get_itunes_client — delegates to registry."""
+    return get_itunes_client()
+
+
+class _SpotifyClientProxy:
+    """Resolves the global Spotify client lazily so a Spotify re-auth that
+    rebinds the cached client in core.metadata.registry is visible to the
+    lifted route bodies."""
+
+    def __getattr__(self, name):
+        client = get_spotify_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        return get_spotify_client() is not None
+
+
+spotify_client = _SpotifyClientProxy()
+
+
+# Artist Map data cache — avoids re-querying 4+ tables on every request
+# Keys: 'watchlist_{profile}', 'genres_{profile}', 'genre_list'
+# Values: {'data': <json-ready dict>, 'ts': <timestamp>}
+_artist_map_cache = {}
+_ARTIST_MAP_CACHE_TTL = 300  # 5 minutes
+
+
+def _artmap_cache_get(key):
+    """Get cached artist map data if still fresh."""
+    entry = _artist_map_cache.get(key)
+    if entry and (time.time() - entry['ts']) < _ARTIST_MAP_CACHE_TTL:
+        return entry['data']
+    return None
+
+
+def _artmap_cache_set(key, data):
+    """Store artist map data in cache."""
+    _artist_map_cache[key] = {'data': data, 'ts': time.time()}
+
+
+def _artmap_cache_invalidate(profile_id=None):
+    """Invalidate artist map cache (call after watchlist changes, scans, etc.)."""
+    if profile_id:
+        _artist_map_cache.pop(f'watchlist_{profile_id}', None)
+        _artist_map_cache.pop(f'genres_{profile_id}', None)
+    _artist_map_cache.pop('genre_list', None)
+
+
+def get_artist_map_data():
+    """Get watchlist artists + their similar artists for the force-directed artist map."""
+    try:
+        database = get_database()
+        profile_id = get_current_profile_id()
+
+        cached = _artmap_cache_get(f'watchlist_{profile_id}')
+        if cached:
+            return jsonify(cached)
+
+        # Get all watchlist artists
+        conn = database._get_connection()
+        cursor = conn.cursor()
+        cursor.execute("""
+            SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
+                   discogs_artist_id, image_url
+            FROM watchlist_artists WHERE profile_id = ?
+        """, (profile_id,))
+        watchlist_rows = cursor.fetchall()
+
+        nodes = []  # {id, name, image_url, type: 'watchlist'|'similar', genres, size}
+        edges = []  # {source, target, weight}
+        seen_names = {}  # normalized_name → node index
+
+        def _norm(name):
+            return (name or '').lower().strip()
+
+        # Add watchlist artists as anchor nodes
+        for wa in watchlist_rows:
+            w = dict(wa)
+            norm = _norm(w['artist_name'])
+            if norm in seen_names:
+                continue
+            idx = len(nodes)
+            seen_names[norm] = idx
+            # Get image — prefer HTTP URLs
+            img = w.get('image_url', '') or ''
+            if img and not img.startswith('http'):
+                img = ''
+            nodes.append({
+                'id': idx,
+                'name': w['artist_name'],
+                'image_url': img,
+                'type': 'watchlist',
+                'genres': [],
+                'spotify_id': w.get('spotify_artist_id') or '',
+                'itunes_id': w.get('itunes_artist_id') or '',
+                'deezer_id': w.get('deezer_artist_id') or '',
+                'discogs_id': w.get('discogs_artist_id') or '',
+                'source_db_id': str(w['id']),
+            })
+
+        # Get all similar artists for all watchlist artists
+        watchlist_ids = [dict(wa)['spotify_artist_id'] or dict(wa)['itunes_artist_id'] or str(dict(wa)['id']) for wa in watchlist_rows]
+        if watchlist_ids:
+            placeholders = ','.join(['?'] * len(watchlist_ids))
+            cursor.execute(f"""
+                SELECT source_artist_id, similar_artist_name, similar_artist_spotify_id,
+                       similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
+                       similarity_rank, occurrence_count, image_url, genres, popularity
+                FROM similar_artists
+                WHERE profile_id = ? AND source_artist_id IN ({placeholders})
+                ORDER BY similarity_rank ASC
+            """, [profile_id] + watchlist_ids)
+
+            for row in cursor.fetchall():
+                r = dict(row)
+                sim_norm = _norm(r['similar_artist_name'])
+
+                # Find or create similar artist node
+                if sim_norm not in seen_names:
+                    idx = len(nodes)
+                    seen_names[sim_norm] = idx
+                    img = r.get('image_url', '') or ''
+                    if img and not img.startswith('http'):
+                        img = ''
+                    genres = []
+                    if r.get('genres'):
+                        try:
+                            genres = json.loads(r['genres'])
+                        except Exception as e:
+                            logger.debug("similar node genres parse failed: %s", e)
+                    nodes.append({
+                        'id': idx,
+                        'name': r['similar_artist_name'],
+                        'image_url': img,
+                        'type': 'similar',
+                        'genres': genres,
+                        'spotify_id': r.get('similar_artist_spotify_id') or '',
+                        'itunes_id': r.get('similar_artist_itunes_id') or '',
+                        'deezer_id': r.get('similar_artist_deezer_id') or '',
+                        'musicbrainz_id': r.get('similar_artist_musicbrainz_id') or '',
+                        'rank': r.get('similarity_rank', 5),
+                        'occurrence': r.get('occurrence_count', 1),
+                        'popularity': r.get('popularity', 0),
+                    })
+
+                sim_idx = seen_names[sim_norm]
+
+                # Find the watchlist node that sourced this similar artist
+                source_norm = None
+                for wa in watchlist_rows:
+                    w = dict(wa)
+                    sid = w.get('spotify_artist_id') or w.get('itunes_artist_id') or str(w['id'])
+                    if sid == r['source_artist_id']:
+                        source_norm = _norm(w['artist_name'])
+                        break
+
+                if source_norm and source_norm in seen_names:
+                    source_idx = seen_names[source_norm]
+                    # Weight: inverse of rank (rank 1 = strongest connection)
+                    weight = max(1, 11 - (r.get('similarity_rank', 5)))
+                    edges.append({
+                        'source': source_idx,
+                        'target': sim_idx,
+                        'weight': weight,
+                    })
+
+        # Also check if any similar artists ARE watchlist artists (cross-links)
+        # These create extra connections between watchlist nodes
+        for i, node in enumerate(nodes):
+            if node['type'] == 'similar':
+                # Check if this similar artist is also a watchlist artist
+                for j, wnode in enumerate(nodes):
+                    if wnode['type'] == 'watchlist' and i != j:
+                        if _norm(node['name']) == _norm(wnode['name']):
+                            # Merge: upgrade the similar node to watchlist
+                            node['type'] = 'watchlist'
+                            break
+
+        # ── Backfill from metadata cache: batch-lookup all node names across all sources ──
+        # Single query to get ALL cached artist entries matching ANY node name
+        try:
+            all_names = list(set(_norm(n['name']) for n in nodes if n.get('name')))
+            if all_names:
+                # Build case-insensitive IN clause via temp matching
+                # Lightweight query — no raw_json (can be huge)
+                cursor.execute("""
+                    SELECT entity_id, source, name, image_url, genres, popularity
+                    FROM metadata_cache_entities
+                    WHERE entity_type = 'artist'
+                """)
+                cache_rows = cursor.fetchall()
+
+                # Index cache by normalized name → {source: {id, image_url, genres}}
+                cache_by_name = {}
+                for cr in cache_rows:
+                    cn = _norm(cr['name'] or '')
+                    if cn not in cache_by_name:
+                        cache_by_name[cn] = {}
+                    source = cr['source']
+                    genres = []
+                    if cr['genres']:
+                        try:
+                            genres = json.loads(cr['genres']) if isinstance(cr['genres'], str) else []
+                        except Exception as e:
+                            logger.debug("backfill cache genres parse failed: %s", e)
+                    cache_by_name[cn][source] = {
+                        'id': cr['entity_id'],
+                        'image_url': cr['image_url'] or '',
+                        'genres': genres,
+                    }
+
+                # Apply cache data to nodes
+                source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
+                for n in nodes:
+                    nn = _norm(n['name'])
+                    cached = cache_by_name.get(nn)
+                    if not cached:
+                        continue
+                    for source, field in source_id_map.items():
+                        if not n.get(field) and source in cached:
+                            n[field] = cached[source]['id']
+                    # Backfill image if missing or local path
+                    if not n.get('image_url') or not n['image_url'].startswith('http'):
+                        for source in ('spotify', 'deezer', 'itunes'):
+                            if source in cached and cached[source].get('image_url', '').startswith('http'):
+                                n['image_url'] = cached[source]['image_url']
+                                break
+                    # Backfill genres if missing
+                    if not n.get('genres') or len(n.get('genres', [])) == 0:
+                        for source in ('spotify', 'deezer', 'itunes', 'discogs', 'musicbrainz'):
+                            if source in cached and cached[source].get('genres'):
+                                n['genres'] = cached[source]['genres'][:5]
+                                break
+                # Deezer direct URL fallback
+                for n in nodes:
+                    if not n.get('image_url') or not n['image_url'].startswith('http'):
+                        if n.get('deezer_id'):
+                            n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
+
+                # Album art fallback (iTunes artists have no artist images)
+                _album_art = {}
+                try:
+                    cursor.execute("""
+                        SELECT artist_name, image_url FROM metadata_cache_entities
+                        WHERE entity_type = 'album' AND image_url LIKE 'http%'
+                          AND artist_name IS NOT NULL AND artist_name != ''
+                    """)
+                    for r in cursor.fetchall():
+                        an = _norm(r['artist_name'])
+                        if an and an not in _album_art:
+                            _album_art[an] = r['image_url']
+                except Exception as e:
+                    logger.debug("artist map album-art cache build failed: %s", e)
+                for n in nodes:
+                    if not n.get('image_url') or not n['image_url'].startswith('http'):
+                        nn = _norm(n['name'])
+                        if nn in _album_art:
+                            n['image_url'] = _album_art[nn]
+
+        except Exception as cache_err:
+            logger.debug(f"Artist map cache backfill error: {cache_err}")
+
+        result = {
+            'success': True,
+            'nodes': nodes,
+            'edges': edges,
+            'watchlist_count': sum(1 for n in nodes if n['type'] == 'watchlist'),
+            'similar_count': sum(1 for n in nodes if n['type'] == 'similar'),
+        }
+        _artmap_cache_set(f'watchlist_{profile_id}', result)
+        return jsonify(result)
+    except Exception as e:
+        logger.error(f"Error getting artist map data: {e}")
+        import traceback
+        traceback.print_exc()
+        return jsonify({"success": False, "error": str(e)}), 500
+
+
+def get_artist_map_genre_list():
+    """Lightweight endpoint — just genre names + counts for the picker. No node data."""
+    try:
+        cached = _artmap_cache_get('genre_list')
+        if cached:
+            return jsonify(cached)
+
+        database = get_database()
+        conn = database._get_connection()
+        cursor = conn.cursor()
+
+        # Fast query: just count artists per genre from cache
+        genre_counts = {}
+        cursor.execute("""
+            SELECT genres FROM metadata_cache_entities
+            WHERE entity_type = 'artist' AND genres IS NOT NULL AND genres != '' AND genres != '[]'
+        """)
+        for r in cursor.fetchall():
+            try:
+                for g in json.loads(r['genres']):
+                    if g and isinstance(g, str):
+                        gl = g.lower().strip()
+                        genre_counts[gl] = genre_counts.get(gl, 0) + 1
+            except Exception as e:
+                logger.debug("genre count row parse failed: %s", e)
+
+        # Sort by count descending
+        sorted_genres = sorted(genre_counts.items(), key=lambda x: -x[1])
+
+        result = {
+            'success': True,
+            'genres': [{'name': g, 'count': c} for g, c in sorted_genres],
+            'total': len(sorted_genres)
+        }
+        _artmap_cache_set('genre_list', result)
+        return jsonify(result)
+    except Exception as e:
+        return jsonify({"success": False, "error": str(e)}), 500
+
+
+def get_artist_map_genres():
+    """Get ALL artists from every data source, grouped by genre for the genre map."""
+    try:
+        database = get_database()
+        profile_id = get_current_profile_id()
+
+        cached = _artmap_cache_get(f'genres_{profile_id}')
+        if cached:
+            return jsonify(cached)
+
+        conn = database._get_connection()
+        cursor = conn.cursor()
+
+        artists_by_name = {}  # normalized_name → {name, image, genres[], sources, ids}
+
+        def _norm(n):
+            return (n or '').lower().strip()
+
+        def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, musicbrainz_id=None, source='unknown', popularity=0):
+            n = _norm(name)
+            if not n or len(n) < 2:
+                return
+            if n not in artists_by_name:
+                artists_by_name[n] = {
+                    'name': name, 'image_url': '', 'genres': set(),
+                    'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': '',
+                    'sources': set(), 'popularity': 0
+                }
+            a = artists_by_name[n]
+            if image_url and image_url.startswith('http') and not a['image_url']:
+                a['image_url'] = image_url
+            if genres:
+                for g in (genres if isinstance(genres, list) else []):
+                    if g and isinstance(g, str):
+                        a['genres'].add(g.lower().strip())
+            if spotify_id and not a['spotify_id']:
+                a['spotify_id'] = str(spotify_id)
+            if itunes_id and not a['itunes_id']:
+                a['itunes_id'] = str(itunes_id)
+            if deezer_id and not a['deezer_id']:
+                a['deezer_id'] = str(deezer_id)
+            if discogs_id and not a['discogs_id']:
+                a['discogs_id'] = str(discogs_id)
+            if musicbrainz_id and not a['musicbrainz_id']:
+                a['musicbrainz_id'] = str(musicbrainz_id)
+            if popularity > a['popularity']:
+                a['popularity'] = popularity
+            a['sources'].add(source)
+
+        # 1. Metadata cache — biggest source
+        cursor.execute("""
+            SELECT name, entity_id, source, image_url, genres, popularity
+            FROM metadata_cache_entities WHERE entity_type = 'artist'
+        """)
+        for r in cursor.fetchall():
+            genres = []
+            if r['genres']:
+                try:
+                    genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                except Exception as e:
+                    logger.debug("cache artist genres parse failed: %s", e)
+            src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
+            kwargs = {src_map.get(r['source'], 'spotify_id'): r['entity_id']}
+            _add(r['name'], image_url=r['image_url'], genres=genres, source='cache', popularity=r['popularity'] or 0, **kwargs)
+
+        # 2. Similar artists
+        cursor.execute("""
+            SELECT similar_artist_name, similar_artist_spotify_id, similar_artist_itunes_id,
+                   similar_artist_deezer_id, similar_artist_musicbrainz_id, image_url, genres, popularity
+            FROM similar_artists WHERE profile_id = ?
+        """, (profile_id,))
+        for r in cursor.fetchall():
+            genres = []
+            if r['genres']:
+                try:
+                    genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                except Exception as e:
+                    logger.debug("similar artist genres parse failed: %s", e)
+            _add(r['similar_artist_name'], image_url=r['image_url'], genres=genres,
+                 spotify_id=r['similar_artist_spotify_id'], itunes_id=r['similar_artist_itunes_id'],
+                 deezer_id=r['similar_artist_deezer_id'],
+                 musicbrainz_id=r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else None,
+                 source='similar', popularity=r['popularity'] or 0)
+
+        # 3. Watchlist artists
+        cursor.execute("""
+            SELECT artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
+                   discogs_artist_id, image_url
+            FROM watchlist_artists WHERE profile_id = ?
+        """, (profile_id,))
+        for r in cursor.fetchall():
+            _add(r['artist_name'], image_url=r['image_url'],
+                 spotify_id=r['spotify_artist_id'], itunes_id=r['itunes_artist_id'],
+                 deezer_id=r['deezer_artist_id'], discogs_id=r['discogs_artist_id'], source='watchlist')
+
+        # 4. Library artists
+        cursor.execute("SELECT name, thumb_url, genres FROM artists")
+        for r in cursor.fetchall():
+            genres = []
+            if r['genres']:
+                try:
+                    genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                except Exception as e:
+                    logger.debug("library artist genres parse failed: %s", e)
+            img = r['thumb_url'] if r['thumb_url'] and r['thumb_url'].startswith('http') else None
+            _add(r['name'], image_url=img, genres=genres, source='library')
+
+        # Filter: only include artists that have at least one genre
+        genre_artists = {k: v for k, v in artists_by_name.items() if v['genres']}
+
+        # Build genre → artists map
+        genre_map = {}  # genre_name → [artist_keys]
+        for key, a in genre_artists.items():
+            for g in a['genres']:
+                if g not in genre_map:
+                    genre_map[g] = []
+                genre_map[g].append(key)
+
+        # Sort genres by artist count, take top genres
+        sorted_genres = sorted(genre_map.items(), key=lambda x: -len(x[1]))
+
+        # Build nodes
+        nodes = []
+        node_idx = {}
+        for key, a in genre_artists.items():
+            idx = len(nodes)
+            node_idx[key] = idx
+            nodes.append({
+                'id': idx,
+                'name': a['name'],
+                'image_url': a['image_url'],
+                'genres': list(a['genres'])[:5],
+                'spotify_id': a['spotify_id'],
+                'itunes_id': a['itunes_id'],
+                'deezer_id': a['deezer_id'],
+                'discogs_id': a['discogs_id'],
+                'musicbrainz_id': a['musicbrainz_id'],
+                'popularity': a['popularity'],
+                'type': 'watchlist' if 'watchlist' in a['sources'] else 'similar',
+            })
+
+        # Build genre clusters — allow artists in multiple genres
+        top_genres = sorted_genres[:40]
+
+        # Sort genres by co-occurrence so related genres are adjacent in the list.
+        # This makes the spiral layout place related genres near each other.
+        if len(top_genres) > 2:
+            genre_sets = {g: set(keys) for g, keys in top_genres}
+            ordered = [top_genres[0][0]]  # Start with biggest genre
+            remaining = {g for g, _ in top_genres[1:]}
+            while remaining:
+                last = ordered[-1]
+                last_set = genre_sets.get(last, set())
+                # Find most similar remaining genre (highest artist overlap)
+                best = None
+                best_overlap = -1
+                for g in remaining:
+                    overlap = len(last_set & genre_sets.get(g, set()))
+                    if overlap > best_overlap:
+                        best_overlap = overlap
+                        best = g
+                ordered.append(best)
+                remaining.remove(best)
+            # Rebuild top_genres in the ordered sequence
+            genre_dict = dict(top_genres)
+            top_genres = [(g, genre_dict[g]) for g in ordered if g in genre_dict]
+
+        genres_out = []
+        for genre, artist_keys in top_genres:
+            genres_out.append({
+                'name': genre,
+                'count': len(artist_keys),
+                'artist_ids': [node_idx[k] for k in artist_keys if k in node_idx],
+            })
+
+        # Image cleanup + multi-source fallback
+        # Build two lookups: name→image_url AND name→deezer_entity_id
+        _img_cache = {}
+        _deezer_id_cache = {}
+        _album_art_cache = {}  # artist_name → album image (iTunes fallback)
+        try:
+            # Artist images + Deezer IDs
+            cursor.execute("""
+                SELECT name, entity_id, source, image_url FROM metadata_cache_entities
+                WHERE entity_type = 'artist'
+                  AND ((image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%')
+                       OR source = 'deezer')
+            """)
+            for r in cursor.fetchall():
+                nn = (r['name'] or '').lower().strip()
+                if not nn:
+                    continue
+                if r['image_url'] and r['image_url'].startswith('http') and nn not in _img_cache:
+                    _img_cache[nn] = r['image_url']
+                if r['source'] == 'deezer' and r['entity_id'] and nn not in _deezer_id_cache:
+                    _deezer_id_cache[nn] = r['entity_id']
+
+            # Album art by artist name (for iTunes artists with no artist image)
+            cursor.execute("""
+                SELECT artist_name, image_url FROM metadata_cache_entities
+                WHERE entity_type = 'album'
+                  AND image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%'
+                  AND artist_name IS NOT NULL AND artist_name != ''
+            """)
+            for r in cursor.fetchall():
+                nn = (r['artist_name'] or '').lower().strip()
+                if nn and nn not in _album_art_cache:
+                    _album_art_cache[nn] = r['image_url']
+        except Exception as e:
+            logger.debug("genre map cache build failed: %s", e)
+
+        for n in nodes:
+            img = n.get('image_url', '')
+            if img in ('None', 'null', '') or (img and not img.startswith('http')):
+                n['image_url'] = ''
+            nn = n['name'].lower().strip()
+            if not n['image_url']:
+                # Try cache image by name
+                n['image_url'] = _img_cache.get(nn, '')
+            if not n['image_url'] and n.get('deezer_id'):
+                n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
+            if not n['image_url']:
+                # Try Deezer ID from cache by name
+                did = _deezer_id_cache.get(nn)
+                if did:
+                    n['deezer_id'] = did
+                    n['image_url'] = f"https://api.deezer.com/artist/{did}/image?size=big"
+            if not n['image_url']:
+                # Try album art by artist name (iTunes artists have no artist images)
+                n['image_url'] = _album_art_cache.get(nn, '')
+
+        _img_count = sum(1 for n in nodes if n.get('image_url'))
+        _deezer_count = sum(1 for n in nodes if n.get('image_url', '').startswith('https://api.deezer'))
+        _none_count = sum(1 for n in nodes if not n.get('image_url'))
+        logger.info(f"[Genre Map] {len(nodes)} artists, {len(sorted_genres)} genres")
+        logger.warning(f"[Genre Map] Images: {_img_count} have URLs, {_deezer_count} Deezer fallback, {_none_count} missing")
+        if _none_count > 0:
+            samples = [n['name'] for n in nodes if not n.get('image_url')][:5]
+            logger.warning(f"[Genre Map] Missing image samples: {samples}")
+
+        result = {
+            'success': True,
+            'nodes': nodes,
+            'genres': genres_out,
+            'total_artists': len(nodes),
+            'total_genres': len(sorted_genres),
+        }
+        _artmap_cache_set(f'genres_{profile_id}', result)
+        return jsonify(result)
+    except Exception as e:
+        logger.error(f"Error getting genre map data: {e}")
+        import traceback
+        traceback.print_exc()
+        return jsonify({"success": False, "error": str(e)}), 500
+
+
+def get_artist_map_explore():
+    """Build an exploration map outward from a single artist."""
+    try:
+        artist_name = request.args.get('name', '').strip()
+        artist_id = request.args.get('id', '').strip()
+
+        if not artist_name and not artist_id:
+            return jsonify({"success": False, "error": "Provide artist name or id"}), 400
+
+        database = get_database()
+        profile_id = get_current_profile_id()
+        conn = database._get_connection()
+        cursor = conn.cursor()
+
+        def _norm(n):
+            return (n or '').lower().strip()
+
+        nodes = []
+        edges = []
+        seen = {}  # norm_name → node index
+
+        # Find the center artist
+        center_name = artist_name
+        center_image = ''
+        center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': ''}
+        center_genres = []
+
+        # Search metadata cache for the center artist
+        if artist_id:
+            cursor.execute("""
+                SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
+                WHERE entity_type = 'artist' AND entity_id = ? LIMIT 1
+            """, (artist_id,))
+        else:
+            cursor.execute("""
+                SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
+                WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE LIMIT 1
+            """, (artist_name,))
+
+        row = cursor.fetchone()
+        artist_found = False
+        if row:
+            artist_found = True
+            center_name = row['name']
+            if row['image_url'] and row['image_url'].startswith('http'):
+                center_image = row['image_url']
+            src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
+            k = src_map.get(row['source'], 'spotify_id')
+            center_ids[k] = row['entity_id']
+            if row['genres']:
+                try:
+                    center_genres = json.loads(row['genres']) if isinstance(row['genres'], str) else []
+                except Exception as e:
+                    logger.debug("initial center genres parse failed: %s", e)
+
+        # Check watchlist + library if not in cache
+        if not artist_found and not artist_id:
+            cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id, musicbrainz_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
+            wr = cursor.fetchone()
+            if wr:
+                artist_found = True
+                center_name = wr['artist_name']
+                if wr['image_url'] and str(wr['image_url']).startswith('http'):
+                    center_image = wr['image_url']
+                for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id'), ('musicbrainz_id', 'musicbrainz_artist_id')]:
+                    if wr[col]:
+                        center_ids[k] = str(wr[col])
+            else:
+                cursor.execute("SELECT name, thumb_url FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
+                lr = cursor.fetchone()
+                if lr:
+                    artist_found = True
+                    center_name = lr['name']
+                    if lr['thumb_url'] and str(lr['thumb_url']).startswith('http'):
+                        center_image = lr['thumb_url']
+
+        # If not found locally, validate via metadata API search
+        if not artist_found and not artist_id:
+            try:
+                api_match = None
+                if spotify_client and spotify_client.is_spotify_authenticated():
+                    results = spotify_client.search_artists(artist_name, limit=1)
+                    if results and len(results) > 0:
+                        sa = results[0]
+                        if sa.name.lower().strip() == artist_name.lower().strip() or \
+                           artist_name.lower().strip() in sa.name.lower().strip():
+                            api_match = sa
+                            center_name = sa.name
+                            center_ids['spotify_id'] = sa.id
+                            center_image = sa.image_url if hasattr(sa, 'image_url') else ''
+                            center_genres = sa.genres if hasattr(sa, 'genres') else []
+                            artist_found = True
+                if not artist_found:
+                    ic = _get_itunes_client()
+                    results = ic.search_artists(artist_name, limit=1)
+                    if results and len(results) > 0:
+                        ia = results[0]
+                        if ia.name.lower().strip() == artist_name.lower().strip() or \
+                           artist_name.lower().strip() in ia.name.lower().strip():
+                            center_name = ia.name
+                            center_ids['itunes_id'] = str(ia.id)
+                            center_image = ia.image_url if hasattr(ia, 'image_url') else ''
+                            artist_found = True
+            except Exception as e:
+                logger.debug(f"[Artist Explorer] API validation failed for '{artist_name}': {e}")
+
+        if not artist_found:
+            return jsonify({"success": False, "error": f"Artist '{artist_name}' not found"}), 404
+
+        # Also check cache for other source IDs
+        cursor.execute("""
+            SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
+            WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
+        """, (center_name,))
+        for r in cursor.fetchall():
+            src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
+            k = src_map.get(r['source'], 'spotify_id')
+            if not center_ids.get(k):
+                center_ids[k] = r['entity_id']
+            if r['image_url'] and r['image_url'].startswith('http') and not center_image:
+                center_image = r['image_url']
+            if r['genres'] and not center_genres:
+                try:
+                    center_genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                except Exception as e:
+                    logger.debug("center genres parse failed: %s", e)
+
+        # Add center node
+        center_idx = 0
+        seen[_norm(center_name)] = center_idx
+        nodes.append({
+            'id': 0, 'name': center_name, 'image_url': center_image,
+            'type': 'center', 'genres': center_genres[:5],
+            **center_ids, 'ring': 0
+        })
+
+        # Ring 1: Direct similar artists from similar_artists table
+        # Search by all known IDs
+        id_values = [v for v in center_ids.values() if v]
+        ring1_artists = []
+        if id_values:
+            placeholders = ','.join(['?'] * len(id_values))
+            cursor.execute(f"""
+                SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
+                       similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
+                       image_url, genres, popularity, similarity_rank
+                FROM similar_artists
+                WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
+                ORDER BY similarity_rank ASC
+            """, id_values + [profile_id])
+            ring1_artists = cursor.fetchall()
+
+        # Also search by name (the center artist might be a watchlist source)
+        cursor.execute("""
+            SELECT DISTINCT sa.similar_artist_name, sa.similar_artist_spotify_id,
+                   sa.similar_artist_itunes_id, sa.similar_artist_deezer_id, sa.similar_artist_musicbrainz_id,
+                   sa.image_url, sa.genres, sa.popularity, sa.similarity_rank
+            FROM similar_artists sa
+            JOIN watchlist_artists wa ON sa.source_artist_id = COALESCE(wa.spotify_artist_id, wa.itunes_artist_id, CAST(wa.id AS TEXT))
+            WHERE wa.artist_name = ? COLLATE NOCASE AND sa.profile_id = ?
+            ORDER BY sa.similarity_rank ASC
+        """, (center_name, profile_id))
+        ring1_artists.extend(cursor.fetchall())
+
+        # If no similar artists in DB, fetch from MusicMap on-the-fly
+        if not ring1_artists:
+            try:
+                logger.debug(f"[Artist Explorer] No stored similar artists for '{center_name}', fetching from MusicMap...")
+                from core.watchlist_scanner import WatchlistScanner
+                scanner = WatchlistScanner(spotify_client=spotify_client) if spotify_client else None
+                if scanner:
+                    similar = scanner._fetch_similar_artists_from_musicmap(center_name, limit=15)
+                    if similar:
+                        source_artist_id = center_ids.get('spotify_id') or center_ids.get('itunes_id') or center_name
+                        # Store in DB for future use
+                        for rank, sa in enumerate(similar, 1):
+                            try:
+                                database.add_or_update_similar_artist(
+                                    source_artist_id=source_artist_id,
+                                    similar_artist_name=sa['name'],
+                                    similar_artist_spotify_id=sa.get('spotify_id'),
+                                    similar_artist_itunes_id=sa.get('itunes_id'),
+                                    similarity_rank=rank,
+                                    profile_id=profile_id,
+                                    image_url=sa.get('image_url'),
+                                    genres=sa.get('genres'),
+                                    popularity=sa.get('popularity', 0),
+                                    similar_artist_deezer_id=sa.get('deezer_id'),
+                                    similar_artist_musicbrainz_id=sa.get('musicbrainz_id'),
+                                )
+                            except Exception as e:
+                                logger.debug("similar artist insert failed: %s", e)
+                        # Re-query from DB to get consistent format
+                        if id_values:
+                            placeholders = ','.join(['?'] * len(id_values))
+                            cursor.execute(f"""
+                                SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
+                                       similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
+                                       image_url, genres, popularity, similarity_rank
+                                FROM similar_artists
+                                WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
+                                ORDER BY similarity_rank ASC
+                            """, id_values + [profile_id])
+                            ring1_artists = cursor.fetchall()
+                        if not ring1_artists:
+                            # Fallback: query by name-based source ID
+                            cursor.execute("""
+                                SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
+                                       similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
+                                       image_url, genres, popularity, similarity_rank
+                                FROM similar_artists
+                                WHERE source_artist_id = ? AND profile_id = ?
+                                ORDER BY similarity_rank ASC
+                            """, (source_artist_id, profile_id))
+                            ring1_artists = cursor.fetchall()
+                        logger.debug(f"[Artist Explorer] Fetched {len(ring1_artists)} similar artists from MusicMap for '{center_name}'")
+                        _artmap_cache_invalidate(profile_id)  # New similar artists added
+            except Exception as e:
+                logger.debug(f"[Artist Explorer] MusicMap fetch failed for '{center_name}': {e}")
+
+        # Deduplicate ring 1
+        for r in ring1_artists:
+            nn = _norm(r['similar_artist_name'])
+            if nn in seen:
+                continue
+            idx = len(nodes)
+            seen[nn] = idx
+            genres = []
+            if r['genres']:
+                try:
+                    genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                except Exception as e:
+                    logger.debug("ring1 genres parse failed: %s", e)
+            img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
+            nodes.append({
+                'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
+                'type': 'ring1', 'genres': genres[:5],
+                'spotify_id': r['similar_artist_spotify_id'] or '',
+                'itunes_id': r['similar_artist_itunes_id'] or '',
+                'deezer_id': r['similar_artist_deezer_id'] or '',
+                'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
+                'discogs_id': '',
+                'popularity': r['popularity'] or 0,
+                'rank': r['similarity_rank'] or 5,
+                'ring': 1,
+            })
+            weight = max(1, 11 - (r['similarity_rank'] or 5))
+            edges.append({'source': center_idx, 'target': idx, 'weight': weight})
+
+        # Ring 2: Similar artists of ring 1 artists (from similar_artists table)
+        ring1_ids = []
+        for n in nodes[1:]:  # skip center
+            for sid in [n.get('spotify_id'), n.get('itunes_id')]:
+                if sid:
+                    ring1_ids.append(sid)
+
+        if ring1_ids:
+            placeholders = ','.join(['?'] * len(ring1_ids))
+            cursor.execute(f"""
+                SELECT DISTINCT source_artist_id, similar_artist_name,
+                       similar_artist_spotify_id, similar_artist_itunes_id,
+                       similar_artist_deezer_id, similar_artist_musicbrainz_id,
+                       image_url, genres, popularity, similarity_rank
+                FROM similar_artists
+                WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
+                ORDER BY similarity_rank ASC
+            """, ring1_ids + [profile_id])
+
+            for r in cursor.fetchall():
+                nn = _norm(r['similar_artist_name'])
+                if nn in seen:
+                    # Create edge to existing node if not center
+                    existing_idx = seen[nn]
+                    # Find the ring1 node that sourced this
+                    source_norm = None
+                    for n in nodes[1:]:
+                        for sid in [n.get('spotify_id'), n.get('itunes_id')]:
+                            if sid == r['source_artist_id']:
+                                source_norm = _norm(n['name'])
+                                break
+                        if source_norm:
+                            break
+                    if source_norm and source_norm in seen and existing_idx != seen[source_norm]:
+                        edges.append({'source': seen[source_norm], 'target': existing_idx, 'weight': 3})
+                    continue
+
+                idx = len(nodes)
+                if idx >= 500:  # Cap at 500 nodes for performance
+                    break
+                seen[nn] = idx
+                genres = []
+                if r['genres']:
+                    try:
+                        genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
+                    except Exception as e:
+                        logger.debug("ring2 genres parse failed: %s", e)
+                img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
+                nodes.append({
+                    'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
+                    'type': 'ring2', 'genres': genres[:5],
+                    'spotify_id': r['similar_artist_spotify_id'] or '',
+                    'itunes_id': r['similar_artist_itunes_id'] or '',
+                    'deezer_id': r['similar_artist_deezer_id'] or '',
+                    'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
+                    'discogs_id': '',
+                    'popularity': r['popularity'] or 0,
+                    'rank': r['similarity_rank'] or 5,
+                    'ring': 2,
+                })
+                # Find the ring1 source
+                for n in nodes[1:]:
+                    for sid in [n.get('spotify_id'), n.get('itunes_id')]:
+                        if sid == r['source_artist_id']:
+                            edges.append({'source': n['id'], 'target': idx, 'weight': max(1, 11 - (r['similarity_rank'] or 5))})
+                            break
+
+        # Backfill images/genres from ALL cache sources + Deezer fallback
+        for n in nodes:
+            # Clean up string "None" stored as image URL
+            if n['image_url'] in ('None', 'null', ''):
+                n['image_url'] = ''
+            if n['image_url'] and n['genres']:
+                continue
+            # Check all cache entries for this artist (multiple sources)
+            cursor.execute("""
+                SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
+                WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
+            """, (n['name'],))
+            for cr in cursor.fetchall():
+                if not n['image_url'] and cr['image_url'] and cr['image_url'].startswith('http'):
+                    n['image_url'] = cr['image_url']
+                if not n['genres'] and cr['genres']:
+                    try:
+                        n['genres'] = json.loads(cr['genres'])[:5] if isinstance(cr['genres'], str) else []
+                    except Exception as e:
+                        logger.debug("explorer node genres parse failed: %s", e)
+                # Harvest missing IDs from cache
+                src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
+                k = src_map.get(cr['source'])
+                if k and not n.get(k):
+                    n[k] = cr['entity_id']
+
+            # Deezer image fallback — construct URL directly from ID
+            if not n['image_url'] and n.get('deezer_id'):
+                n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
+
+            # Spotify image fallback — try API if authenticated
+            if not n['image_url'] and n.get('spotify_id'):
+                try:
+                    if spotify_client and spotify_client.is_spotify_authenticated():
+                        from core.api_call_tracker import api_call_tracker
+                        api_call_tracker.record_call('spotify', endpoint='artist')
+                        artist_data = spotify_client.sp.artist(n['spotify_id'])
+                        if artist_data and artist_data.get('images'):
+                            n['image_url'] = artist_data['images'][0]['url']
+                            if not n['genres'] and artist_data.get('genres'):
+                                n['genres'] = artist_data['genres'][:5]
+                except Exception as e:
+                    logger.debug("spotify artist image fallback failed: %s", e)
+
+            # Album art fallback (iTunes artists have no artist images)
+            if not n['image_url']:
+                cursor.execute("""
+                    SELECT image_url FROM metadata_cache_entities
+                    WHERE entity_type = 'album' AND image_url LIKE 'http%'
+                      AND artist_name = ? COLLATE NOCASE LIMIT 1
+                """, (n['name'],))
+                alb = cursor.fetchone()
+                if alb:
+                    n['image_url'] = alb['image_url']
+
+        logger.info(f"[Artist Explorer] Center: {center_name}, Ring 1: {sum(1 for n in nodes if n.get('ring')==1)}, Ring 2: {sum(1 for n in nodes if n.get('ring')==2)}, Edges: {len(edges)}")
+
+        return jsonify({
+            'success': True,
+            'nodes': nodes,
+            'edges': edges,
+            'center': center_name,
+        })
+    except Exception as e:
+        logger.error(f"Error getting artist explorer data: {e}")
+        import traceback
+        traceback.print_exc()
+        return jsonify({"success": False, "error": str(e)}), 500

core/artists/quality.py

@@ -0,0 +1,496 @@
+"""Artist quality enhancement helper.
+
+`enhance_artist_quality(artist_id, track_ids, deps)` is the route-handler
+body for the `/api/library/artist/<artist_id>/enhance` endpoint. It walks
+the user's selected tracks, finds the best metadata match against the
+configured primary source, and queues high-quality re-downloads on the
+wishlist with `source_type='enhance'`.
+
+Per-track flow:
+
+1. Resolve the existing track via the artist's full detail map (built up
+   front from `database.get_artist_full_detail`).
+2. Read current quality tier from the file extension.
+3. Build `matched_track_data` for the wishlist entry, in priority order:
+   - **Direct lookup using stored source IDs** — for every source the
+     user has configured, if the library track has the corresponding
+     stored ID (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
+     `soul_id`), call `client.get_track_details(stored_id)` and convert
+     the result to the wishlist payload. First success wins; the user's
+     configured primary source is tried first. Mirrors what Download
+     Discography does — stable IDs straight to the source's API, no
+     fuzzy text matching.
+   - **Multi-source parallel text search fallback** — if no stored ID
+     resolved, run the shared `core.metadata.multi_source_search`
+     against every configured source in parallel and pick the best
+     cross-source match (auto-accept threshold 0.7).
+4. Validate the match has non-empty title, album, and artists. Reject
+   matches with empty fields — those propagated as
+   "unknown artist - unknown album - unknown track" wishlist entries
+   pre-fix because the wishlist payload normalizer's truthy-check
+   passthrough accepted dicts with empty string fields.
+5. Add to wishlist via `wishlist_service.add_spotify_track_to_wishlist`
+   with `source_type='enhance'` and a `source_context` carrying the
+   original file path, format tier, bitrate, and artist name.
+6. Tally `enhanced_count` / `failed_count` / per-track failure reasons.
+
+The flow originally had Spotify-only logic with an iTunes search-only
+fallback. Two failure modes drove the rewrite:
+
+- Users with neither Spotify nor Deezer connected got silent failures
+  ("unknown artist - unknown album - unknown track" wishlist entries)
+  because iTunes's text search returned junk matches with empty fields
+  that cleared the 0.7 confidence threshold.
+- Library tracks with messy tags ("Title (Live)", featured artists in
+  the artist field, etc.) failed fuzzy text search even when a perfect
+  stored ID was available — Download Discography had no such problem
+  because it resolves albums by stable ID.
+
+Direct-lookup-via-stored-ID matches the Download Discography contract
+for every source where we have an ID column. Text search is only the
+fallback now.
+
+Returns `(payload_dict, http_status_code)` so the route wrapper can
+`jsonify()` and return.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+from typing import Any, Callable, Optional
+
+from utils.logging_config import get_logger
+
+logger = get_logger('artists.quality')
+
+
+@dataclass
+class ArtistQualityDeps:
+    """Bundle of cross-cutting deps the artist quality enhancement needs."""
+    matching_engine: Any
+    get_database: Callable[[], Any]
+    get_wishlist_service: Callable[[], Any]
+    get_current_profile_id: Callable[[], int]
+    get_quality_tier_from_extension: Callable
+    # Returns ``[(source_name, client), ...]`` for every metadata source
+    # the user has configured. Powers both the direct-lookup fast path
+    # (resolves stored source IDs straight from each source's API,
+    # like Download Discography) and the multi-source parallel text
+    # search fallback (shared with Track Redownload via
+    # ``core.metadata.multi_source_search``).
+    get_metadata_search_sources: Callable[[], list]
+
+
+def _has_complete_metadata(payload: Optional[dict]) -> bool:
+    """Reject matches with empty / missing core fields. Pre-fix, iTunes
+    returned matches that cleared the 0.7 confidence threshold while
+    having empty artist / album / title — those propagated as junk
+    wishlist entries displayed as 'unknown artist - unknown album -
+    unknown track'."""
+    if not payload:
+        return False
+    if not (payload.get('name') or '').strip():
+        return False
+    artists = payload.get('artists') or []
+    has_artist = any(
+        (a.get('name') or '').strip() if isinstance(a, dict) else (a or '').strip()
+        for a in artists
+    )
+    if not has_artist:
+        return False
+    album = payload.get('album') or {}
+    if isinstance(album, dict):
+        if not (album.get('name') or '').strip():
+            return False
+    elif not (album or '').strip():
+        return False
+    return True
+
+
+def _build_payload_from_track(track_obj) -> dict:
+    """Build a Spotify-shaped wishlist payload from any metadata source's
+    Track-shaped object (Spotify Track, iTunes Track, Deezer Track,
+    Discogs Track — they all have the same .id / .name / .artists /
+    .album / .duration_ms / etc shape because each client mimics
+    Spotify's surface).
+
+    The wishlist's downstream pipeline expects Spotify shape; this helper
+    is the single place that knows how to produce it. Replaces the
+    duplicated payload construction that used to live in the Spotify
+    search path AND the iTunes fallback path.
+
+    Does NOT substitute defaults for missing artists / album / title —
+    ``_has_complete_metadata`` rejects empty matches downstream so the
+    user sees a clear failure instead of a junk wishlist entry with
+    fabricated values.
+    """
+    image_url = getattr(track_obj, 'image_url', '') or ''
+    album_images = (
+        [{'url': image_url, 'height': 600, 'width': 600}]
+        if image_url else []
+    )
+    artist_names = list(getattr(track_obj, 'artists', None) or [])
+    return {
+        'id': getattr(track_obj, 'id', ''),
+        'name': getattr(track_obj, 'name', '') or '',
+        'artists': [{'name': a} for a in artist_names],
+        'album': {
+            'name': getattr(track_obj, 'album', '') or '',
+            'artists': [{'name': a} for a in artist_names],
+            'album_type': getattr(track_obj, 'album_type', None) or 'album',
+            'images': album_images,
+            'release_date': getattr(track_obj, 'release_date', '') or '',
+            'total_tracks': 1,
+        },
+        'duration_ms': getattr(track_obj, 'duration_ms', 0) or 0,
+        'track_number': getattr(track_obj, 'track_number', None) or 1,
+        'disc_number': getattr(track_obj, 'disc_number', None) or 1,
+        'popularity': getattr(track_obj, 'popularity', None) or 0,
+        'preview_url': getattr(track_obj, 'preview_url', None),
+        'external_urls': getattr(track_obj, 'external_urls', None) or {},
+    }
+
+
+# Map metadata source name → DB column on the ``tracks`` table that
+# stores that source's native track ID. Used to drive the direct-lookup
+# fast path: when a library track has a stored ID for source X and the
+# user has source X configured, skip fuzzy text search and resolve
+# straight from X's API. Mirrors what Download Discography does — stable
+# IDs all the way, no fuzzy text matching.
+#
+# Discogs is release-based and has no per-track ID column; not listed
+# here, so direct lookup never tries Discogs (search-fallback still
+# runs for Discogs as one of the parallel sources).
+_STORED_ID_COLUMNS = {
+    'spotify': 'spotify_track_id',
+    'deezer': 'deezer_id',
+    'itunes': 'itunes_track_id',
+    'hydrabase': 'soul_id',
+}
+
+
+def _enhanced_to_wishlist_payload(enhanced: dict,
+                                   fallback_title: str,
+                                   fallback_artist: str,
+                                   fallback_album: str) -> Optional[dict]:
+    """Convert a ``get_track_details`` enhanced-shape dict to the
+    Spotify-shape wishlist payload.
+
+    Every metadata source's ``get_track_details`` returns the same
+    "enhanced" intermediate shape (top-level ``id``, ``name``,
+    ``artists`` as a list of strings, ``album.artists`` as strings),
+    documented and pinned across spotify_client / itunes_client /
+    deezer_client / hydrabase_client. The wishlist downstream expects
+    Spotify's native shape (``artists`` as ``[{'name': ...}]``), so
+    this helper does the conversion in one place.
+
+    Spotify's ``raw_data`` field is already in wishlist shape (the
+    raw Spotify API response), so we return it as-is when detected,
+    preserving full ``album.images`` and ``external_urls`` that the
+    enhanced top-level fields drop. Other sources' ``raw_data`` is
+    in source-native shape and gets ignored.
+    """
+    if not enhanced:
+        return None
+    raw = enhanced.get('raw_data')
+    if isinstance(raw, dict):
+        raw_artists = raw.get('artists')
+        if (isinstance(raw_artists, list) and raw_artists
+                and isinstance(raw_artists[0], dict)):
+            return raw
+
+    artists = enhanced.get('artists') or [fallback_artist]
+    album_data = enhanced.get('album') or {}
+    album_artists = album_data.get('artists') or artists
+
+    def _to_dict_artists(seq):
+        return [a if isinstance(a, dict) else {'name': a} for a in seq]
+
+    image_url = enhanced.get('image_url') or ''
+    album_images_field = album_data.get('images')
+    if isinstance(album_images_field, list) and album_images_field:
+        album_images = album_images_field
+    elif image_url:
+        album_images = [{'url': image_url, 'height': 600, 'width': 600}]
+    else:
+        album_images = []
+
+    return {
+        'id': str(enhanced.get('id', '')),
+        'name': enhanced.get('name') or fallback_title,
+        'artists': _to_dict_artists(artists),
+        'album': {
+            'id': str(album_data.get('id', '')),
+            'name': album_data.get('name') or fallback_album,
+            'album_type': album_data.get('album_type', 'album'),
+            'release_date': album_data.get('release_date', ''),
+            'total_tracks': album_data.get('total_tracks', 1),
+            'artists': _to_dict_artists(album_artists),
+            'images': album_images,
+        },
+        'duration_ms': enhanced.get('duration_ms', 0),
+        'track_number': enhanced.get('track_number', 1),
+        'disc_number': enhanced.get('disc_number', 1),
+        'popularity': enhanced.get('popularity', 0),
+        'preview_url': enhanced.get('preview_url'),
+        'external_urls': enhanced.get('external_urls', {}),
+    }
+
+
+def _try_direct_lookup_all_sources(track: dict,
+                                    sources: list,
+                                    preferred_source: Optional[str],
+                                    title: str,
+                                    artist_name: str,
+                                    album_title: str
+                                    ) -> tuple:
+    """Try direct ID-based lookup on every source where the library
+    track has a stored ID. Returns ``(payload, source_name)`` on first
+    success, or ``(None, None)`` if no source has a stored ID with a
+    successful lookup.
+
+    Mirrors what Download Discography does — stable IDs straight to the
+    source's API, no fuzzy text matching. Avoids the failure mode where
+    library text tags don't match the source's canonical title (the
+    Discord report case: track tagged "Title (Live)" and source has
+    "Title" → fuzzy search misses, but stored ID resolves directly).
+
+    Preferred source attempted first when present in ``sources``,
+    typically the user's configured primary metadata source — so a
+    Deezer-primary user gets Deezer art / album shape on the wishlist
+    entry instead of whichever source happened to have a stored ID
+    first in iteration order.
+    """
+    def _priority(entry):
+        name = entry[0]
+        return 0 if name == preferred_source else 1
+    ordered = sorted(sources, key=_priority)
+
+    for source_name, client in ordered:
+        column = _STORED_ID_COLUMNS.get(source_name)
+        if not column:
+            continue
+        stored_id = track.get(column)
+        if not stored_id:
+            continue
+        if not hasattr(client, 'get_track_details'):
+            continue
+        try:
+            enhanced = client.get_track_details(str(stored_id))
+        except Exception as exc:
+            logger.error(
+                f"[Enhance] {source_name} direct lookup failed for "
+                f"ID {stored_id}: {exc}"
+            )
+            continue
+        if not enhanced:
+            continue
+        payload = _enhanced_to_wishlist_payload(
+            enhanced, title, artist_name, album_title,
+        )
+        if _has_complete_metadata(payload):
+            logger.info(
+                f"[Enhance] Direct lookup matched: {source_name} "
+                f"ID {stored_id} → '{payload.get('name')}'"
+            )
+            return payload, source_name
+
+    return None, None
+
+
+# Minimum match-score threshold for accepting a search-fallback match
+# without user confirmation. Mirrors the legacy threshold the enhance
+# flow has always used.
+_AUTO_ACCEPT_SCORE_THRESHOLD = 0.7
+
+
+def enhance_artist_quality(artist_id, track_ids, deps: ArtistQualityDeps):
+    """Add selected tracks to wishlist for quality enhancement re-download.
+
+    Per-track flow:
+
+    1. **Direct lookup using stored source IDs** (mirrors what Download
+       Discography does — stable IDs straight to the source's API, no
+       fuzzy text matching). For each source the user has configured,
+       if the library track has the corresponding stored ID
+       (``spotify_track_id`` / ``deezer_id`` / ``itunes_track_id`` /
+       ``soul_id``), call ``client.get_track_details(stored_id)`` and
+       convert to wishlist payload. First success wins; preferred
+       source (user's configured primary) tried first.
+
+    2. **Multi-source parallel text search fallback** (via the shared
+       ``core.metadata.multi_source_search`` module — same code path
+       Track Redownload uses) for tracks with no stored IDs / lookup
+       misses.
+
+    3. **Validation**: reject matches with empty title / album / artists
+       so the user sees a clear failure instead of an "unknown artist"
+       wishlist entry.
+
+    Pre-refactor: only Spotify had a direct-lookup fast path; everything
+    else went through fuzzy text search. Discogs / Hydrabase / Deezer-
+    primary users got far worse coverage than Download Discography
+    despite both flows asking the same question.
+    """
+    from core.metadata.multi_source_search import TrackQuery, search_all_sources
+    from core.metadata.registry import get_primary_source
+
+    try:
+        if not track_ids:
+            return {"success": False, "error": "No track IDs provided"}, 400
+
+        database = deps.get_database()
+        wishlist_service = deps.get_wishlist_service()
+        profile_id = deps.get_current_profile_id()
+
+        # Get artist info
+        artist_result = database.get_artist_full_detail(artist_id)
+        if not artist_result.get('success'):
+            return {"success": False, "error": "Artist not found"}, 404
+
+        artist_name = artist_result.get('artist', {}).get('name', 'Unknown Artist')
+
+        # Build lookup of all tracks for this artist
+        track_lookup = {}
+        for album in artist_result.get('albums', []):
+            album_title = album.get('title', '')
+            for track in album.get('tracks', []):
+                tid = str(track.get('id', ''))
+                track['_album_title'] = album_title
+                track['_album_id'] = album.get('id')
+                track_lookup[tid] = track
+
+        # Resolve every configured metadata source up front.
+        search_sources = deps.get_metadata_search_sources()
+
+        # User's configured primary source — direct-lookup tries this
+        # first so Deezer-primary users get Deezer payloads on the
+        # wishlist entry (correct cover art / album shape) even when
+        # other sources also have stored IDs for the same track.
+        try:
+            preferred_source = get_primary_source()
+        except Exception:
+            preferred_source = None
+
+        enhanced_count = 0
+        failed_count = 0
+        failed_tracks = []
+
+        for track_id in track_ids:
+            track_id_str = str(track_id)
+            track = track_lookup.get(track_id_str)
+            if not track:
+                failed_count += 1
+                failed_tracks.append({'track_id': track_id, 'reason': 'Track not found'})
+                continue
+
+            file_path = track.get('file_path')
+            if not file_path:
+                failed_count += 1
+                failed_tracks.append({'track_id': track_id, 'reason': 'No file path'})
+                continue
+
+            tier_name, tier_num = deps.get_quality_tier_from_extension(file_path)
+            title = track.get('title', '') or ''
+            if not title.strip():
+                title = os.path.splitext(os.path.basename(file_path))[0]
+            album_title = track.get('_album_title', '')
+
+            matched_track_data = None
+            chosen_source = None
+
+            # 1. Direct lookup via every stored source ID — like Download
+            # Discography. Stable IDs, no fuzzy text matching.
+            if search_sources:
+                matched_track_data, chosen_source = _try_direct_lookup_all_sources(
+                    track, search_sources, preferred_source,
+                    title, artist_name, album_title,
+                )
+
+            # 2. Multi-source parallel text search fallback — for tracks
+            # with no stored IDs / lookup misses.
+            if not matched_track_data and search_sources:
+                try:
+                    track_query = TrackQuery(
+                        title=title,
+                        artist=artist_name,
+                        album=album_title,
+                        duration_ms=track.get('duration', 0) or 0,
+                        spotify_track_id=track.get('spotify_track_id'),
+                        deezer_id=track.get('deezer_id'),
+                    )
+                    multi_result = search_all_sources(track_query, search_sources)
+                    if multi_result.best_match and multi_result.best_match['score'] >= _AUTO_ACCEPT_SCORE_THRESHOLD:
+                        chosen_source = multi_result.best_match['source']
+                        best_track_obj = multi_result.best_track()
+                        if best_track_obj:
+                            matched_track_data = _build_payload_from_track(best_track_obj)
+                except Exception as exc:
+                    logger.error(f"[Enhance] Multi-source search failed for {title}: {exc}")
+
+            # 3. Reject matches with empty / missing core fields.
+            if not _has_complete_metadata(matched_track_data):
+                if matched_track_data:
+                    logger.warning(
+                        f"[Enhance] {chosen_source} match for '{title}' rejected — "
+                        f"empty title / album / artists (would render as 'unknown')"
+                    )
+                matched_track_data = None
+
+            if not matched_track_data:
+                failed_count += 1
+                source_list = ', '.join(name for name, _ in (search_sources or []))
+                if not source_list:
+                    reason = (
+                        'No metadata source configured — connect Spotify / '
+                        'iTunes / Deezer / Discogs / Hydrabase to enable enhance'
+                    )
+                else:
+                    reason = (
+                        f'No usable match across {source_list} — '
+                        f'try connecting an additional metadata source'
+                    )
+                failed_tracks.append({
+                    'track_id': track_id,
+                    'title': title,
+                    'reason': reason,
+                })
+                continue
+
+            # Add to wishlist with enhance source
+            source_context = {
+                'enhance': True,
+                'original_file_path': file_path,
+                'original_format': tier_name,
+                'original_bitrate': track.get('bitrate'),
+                'original_tier': tier_num,
+                'artist_name': artist_name,
+            }
+
+            success = wishlist_service.add_spotify_track_to_wishlist(
+                spotify_track_data=matched_track_data,
+                failure_reason=f"Quality enhance - upgrading from {tier_name.replace('_', ' ').title()}",
+                source_type='enhance',
+                source_context=source_context,
+                profile_id=profile_id
+            )
+
+            if success:
+                enhanced_count += 1
+                logger.info(f"[Enhance] Queued for upgrade: {artist_name} - {title} ({tier_name})")
+            else:
+                failed_count += 1
+                failed_tracks.append({'track_id': track_id, 'title': title, 'reason': 'Wishlist add failed'})
+
+        return {
+            'success': True,
+            'enhanced_count': enhanced_count,
+            'failed_count': failed_count,
+            'failed_tracks': failed_tracks
+        }, 200
+    except Exception as e:
+        logger.error(f"[Enhance] {e}")
+        import traceback
+        traceback.print_exc()
+        return {"success": False, "error": str(e)}, 500

core/audiodb_worker.py

@@ -8,7 +8,7 @@ from datetime import datetime, timedelta
 from utils.logging_config import get_logger
 from database.music_database import MusicDatabase
 from core.audiodb_client import AudioDBClient
-from core.worker_utils import interruptible_sleep
+from core.worker_utils import accept_artist_match, interruptible_sleep
 
 logger = get_logger("audiodb_worker")
 
@@ -140,8 +140,8 @@ class AudioDBWorker:
                         itype = item.get('type', '')
                         table = 'artists' if 'artist' in itype else ('albums' if 'album' in itype else 'tracks')
                         # Can't mark status without an ID — just skip
-                    except Exception:
-                        pass
+                    except Exception as e:
+                        logger.debug("null id table resolve failed: %s", e)
                     continue
 
 
@@ -162,6 +162,16 @@ class AudioDBWorker:
             conn = self.db._get_connection()
             cursor = conn.cursor()
 
+            # Pinned-group override (Manage Enrichment Workers): process one
+            # entity type first, then fall through to the normal chain. Unset or
+            # exhausted ⇒ default artist→album→track order, unchanged.
+            from core.worker_utils import read_enrichment_priority, priority_pending_item
+            _prio = read_enrichment_priority('audiodb')
+            if _prio:
+                _pi = priority_pending_item(cursor, 'audiodb', _prio)
+                if _pi:
+                    return _pi
+
             # Priority 1: Unattempted artists
             cursor.execute("""
                 SELECT id, name
@@ -200,42 +210,45 @@ class AudioDBWorker:
             if row:
                 return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
 
-            # Priority 4: Retry 'not_found' artists after retry_days
-            not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
+            # Priority 4: Retry 'not_found' OR 'error' artists after retry_days.
+            # 'error' status covers transient AudioDB outages (timeouts, 500s)
+            # that the issue-#553 fix marks rather than leaving NULL — without
+            # this retry path those rows would stay errored forever.
+            retry_cutoff = datetime.now() - timedelta(days=self.retry_days)
             cursor.execute("""
                 SELECT id, name
                 FROM artists
-                WHERE audiodb_match_status = 'not_found' AND audiodb_last_attempted < ?
+                WHERE audiodb_match_status IN ('not_found', 'error') AND audiodb_last_attempted < ?
                 ORDER BY audiodb_last_attempted ASC
                 LIMIT 1
-            """, (not_found_cutoff,))
+            """, (retry_cutoff,))
             row = cursor.fetchone()
             if row:
                 logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
                 return {'type': 'artist', 'id': row[0], 'name': row[1]}
 
-            # Priority 5: Retry 'not_found' albums
+            # Priority 5: Retry 'not_found' OR 'error' albums
             cursor.execute("""
                 SELECT a.id, a.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
                 FROM albums a
                 JOIN artists ar ON a.artist_id = ar.id
-                WHERE a.audiodb_match_status = 'not_found' AND a.audiodb_last_attempted < ?
+                WHERE a.audiodb_match_status IN ('not_found', 'error') AND a.audiodb_last_attempted < ?
                 ORDER BY a.audiodb_last_attempted ASC
                 LIMIT 1
-            """, (not_found_cutoff,))
+            """, (retry_cutoff,))
             row = cursor.fetchone()
             if row:
                 return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
 
-            # Priority 6: Retry 'not_found' tracks
+            # Priority 6: Retry 'not_found' OR 'error' tracks
             cursor.execute("""
                 SELECT t.id, t.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
                 FROM tracks t
                 JOIN artists ar ON t.artist_id = ar.id
-                WHERE t.audiodb_match_status = 'not_found' AND t.audiodb_last_attempted < ?
+                WHERE t.audiodb_match_status IN ('not_found', 'error') AND t.audiodb_last_attempted < ?
                 ORDER BY t.audiodb_last_attempted ASC
                 LIMIT 1
-            """, (not_found_cutoff,))
+            """, (retry_cutoff,))
             row = cursor.fetchone()
             if row:
                 return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
@@ -260,8 +273,13 @@ class AudioDBWorker:
 
     def _verify_artist_id(self, item: Dict[str, Any], result: Dict[str, Any]) -> bool:
         """Verify that the result's artist ID matches the parent artist's stored AudioDB ID.
+
         If mismatched, the album/track search is more specific (uses artist+title),
-        so we trust it and correct the parent artist's audiodb_id."""
+        so we trust it and correct the parent artist's audiodb_id — BUT only when
+        the result's artist *name* matches our parent artist. Without that guard,
+        a collaboration/compilation (a track our library credits to one artist
+        that lives on another artist's album) would stamp the wrong AudioDB id
+        onto our artist. See the Deezer fix for the full write-up."""
         parent_audiodb_id = item.get('artist_audiodb_id')
         if not parent_audiodb_id:
             return True
@@ -271,6 +289,18 @@ class AudioDBWorker:
             return True
 
         if str(result_artist_id) != str(parent_audiodb_id):
+            parent_name = item.get('artist') or ''
+            result_artist_name = result.get('strArtist') or ''
+            if (result_artist_name and parent_name
+                    and not self._name_matches(parent_name, result_artist_name)):
+                logger.info(
+                    f"Skipping artist-ID correction from {item['type']} "
+                    f"'{item['name']}': result artist '{result_artist_name}' "
+                    f"≠ parent '{parent_name}' (collab/compilation, not a "
+                    f"correction)"
+                )
+                return True
+
             logger.info(
                 f"Artist ID correction from {item['type']} '{item['name']}': "
                 f"updating parent artist AudioDB ID from {parent_audiodb_id} to {result_artist_id}"
@@ -374,22 +404,40 @@ class AudioDBWorker:
                             return
                     except Exception as e:
                         logger.warning(f"Direct lookup failed for existing AudioDB ID {existing_id}: {e}")
-                    # Direct lookup failed — don't overwrite manual match
-                    logger.debug(f"Preserving manual match for {item_type} '{item_name}' (AudioDB ID: {existing_id})")
+                    # Direct lookup returned no metadata (None) or raised — don't
+                    # fall through to the name-search path below, which could
+                    # overwrite a manually-matched audiodb_id with a wrong guess.
+                    # Mark status='error' so the queue's NULL-status filter stops
+                    # re-picking this row on every tick (issue #553: AudioDB
+                    # `track.php` timeouts caused infinite enrichment loops as
+                    # the row was repeatedly picked + re-attempted because it
+                    # never left the NULL state). The error-retry priority block
+                    # in `_get_next_item` re-attempts after `retry_days` so
+                    # transient AudioDB outages still recover automatically.
+                    self._mark_status(item_type, item_id, 'error')
+                    self.stats['errors'] += 1
+                    logger.debug(
+                        f"Preserving manual match for {item_type} '{item_name}' "
+                        f"(AudioDB ID: {existing_id}); marked error pending retry"
+                    )
                     return
 
             if item_type == 'artist':
                 result = self.client.search_artist(item_name)
                 if result:
                     result_name = result.get('strArtist', '')
-                    if self._name_matches(item_name, result_name):
+                    ok, reason = accept_artist_match(
+                        self.db, 'audiodb_id', result.get('idArtist'), item_id,
+                        item_name, result_name,
+                    )
+                    if ok:
                         self._update_artist(item_id, result)
                         self.stats['matched'] += 1
                         logger.info(f"Matched artist '{item_name}' -> AudioDB ID: {result.get('idArtist')}")
                     else:
                         self._mark_status('artist', item_id, 'not_found')
                         self.stats['not_found'] += 1
-                        logger.debug(f"Name mismatch for artist '{item_name}' (got '{result_name}')")
+                        logger.debug(f"Artist '{item_name}' not matched: {reason}")
                 else:
                     self._mark_status('artist', item_id, 'not_found')
                     self.stats['not_found'] += 1

core/automation/__init__.py

@@ -0,0 +1,11 @@
+"""Automation API + progress + handlers package.
+
+Lifted from web_server.py:
+  - `/api/automations/*` route helpers → `api.py`
+  - block library used by the trigger/action UI → `blocks.py`
+  - progress tracker (init / update / finish) → `progress.py`
+  - cross-handler signal bus → `signals.py`
+  - per-action handler functions → `handlers/` subpackage (with
+    `deps.py` defining the dependency-injection surface so handlers
+    stay testable in isolation)
+"""

core/automation/api.py

@@ -0,0 +1,377 @@
+"""Automation REST API helpers.
+
+CRUD + run + progress + history logic for /api/automations/* routes.
+Each function takes the deps it needs (database, automation_engine,
+profile_id) so the route layer is left as pure HTTP shuffling.
+
+Out of scope:
+- /api/automations/blocks — static JSON + one call into signals.py;
+  stays inline in web_server.py for now.
+- /api/test/automation — touches scan manager + media clients +
+  config_manager; stays inline.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from typing import Any, Optional
+
+logger = logging.getLogger(__name__)
+
+
+# ---------------------------------------------------------------------------
+# Hydration helpers — convert raw DB rows to API-friendly dicts
+# ---------------------------------------------------------------------------
+
+_JSON_FIELDS = ('trigger_config', 'action_config', 'notify_config', 'last_result')
+_JSON_DEFAULT_DICT = {'trigger_config', 'action_config', 'notify_config'}
+
+
+def _hydrate_automation(auto: dict) -> dict:
+    """Parse JSON columns and backfill `then_actions` from legacy notify_*."""
+    for field in _JSON_FIELDS:
+        try:
+            auto[field] = json.loads(auto[field]) if isinstance(auto[field], str) else auto[field]
+        except (json.JSONDecodeError, TypeError):
+            auto[field] = {} if field in _JSON_DEFAULT_DICT else None
+
+    try:
+        raw = auto.get('then_actions')
+        auto['then_actions'] = json.loads(raw or '[]') if isinstance(raw, str) else (raw or [])
+    except (json.JSONDecodeError, TypeError):
+        auto['then_actions'] = []
+
+    if not auto['then_actions'] and auto.get('notify_type'):
+        auto['then_actions'] = [{
+            'type': auto['notify_type'],
+            'config': auto.get('notify_config', {}),
+        }]
+    return auto
+
+
+# ---------------------------------------------------------------------------
+# Signal cycle detection
+# ---------------------------------------------------------------------------
+
+def _has_signal_concern(trigger_type: str, then_actions: list[dict]) -> bool:
+    return trigger_type == 'signal_received' or any(
+        t.get('type') == 'fire_signal' for t in then_actions
+    )
+
+
+def _check_create_cycle(
+    automation_engine,
+    database,
+    profile_id: int,
+    trigger_type: str,
+    trigger_config_json: str,
+    then_actions_json: str,
+    then_actions: list[dict],
+) -> Optional[str]:
+    """Return cycle path string if creating this automation would loop, else None."""
+    if not automation_engine or not _has_signal_concern(trigger_type, then_actions):
+        return None
+    all_autos = database.get_automations(profile_id)
+    test_auto = {
+        'trigger_type': trigger_type,
+        'trigger_config': trigger_config_json,
+        'then_actions': then_actions_json,
+        'enabled': True,
+    }
+    all_autos.append(test_auto)
+    cycle = automation_engine.detect_signal_cycles(all_autos)
+    if cycle:
+        return ' → '.join(cycle)
+    return None
+
+
+def _check_update_cycle(
+    automation_engine,
+    database,
+    automation_id: int,
+    data: dict,
+) -> Optional[str]:
+    """Return cycle path string if updating this automation would loop, else None."""
+    if not automation_engine:
+        return None
+    trigger_type = data.get('trigger_type', '')
+    then_actions = data.get('then_actions', [])
+    if not _has_signal_concern(trigger_type, then_actions):
+        return None
+
+    all_autos = database.get_automations()
+    test_autos = []
+    for a in all_autos:
+        if a['id'] == automation_id:
+            merged = dict(a)
+            if 'trigger_type' in data:
+                merged['trigger_type'] = data['trigger_type']
+            if 'trigger_config' in data:
+                merged['trigger_config'] = json.dumps(data['trigger_config'])
+            if 'then_actions' in data:
+                merged['then_actions'] = json.dumps(data['then_actions'])
+            merged['enabled'] = True
+            test_autos.append(merged)
+        else:
+            test_autos.append(a)
+    cycle = automation_engine.detect_signal_cycles(test_autos)
+    if cycle:
+        return ' → '.join(cycle)
+    return None
+
+
+# ---------------------------------------------------------------------------
+# CRUD helpers — return (response_dict, http_status)
+# ---------------------------------------------------------------------------
+
+def list_automations(database, profile_id: int) -> list[dict]:
+    """All automations for the profile, with JSON columns parsed."""
+    automations = database.get_automations(profile_id)
+    return [_hydrate_automation(a) for a in automations]
+
+
+def get_automation(database, automation_id: int) -> Optional[dict]:
+    """One automation, hydrated. Returns None if not found."""
+    auto = database.get_automation(automation_id)
+    if not auto:
+        return None
+    return _hydrate_automation(auto)
+
+
+def create_automation(
+    database,
+    automation_engine,
+    profile_id: int,
+    data: dict,
+) -> tuple[dict, int]:
+    """Create + schedule an automation. Returns (response_body, http_status)."""
+    name = (data.get('name') or '').strip()
+    if not name:
+        return {'error': 'Name is required'}, 400
+
+    trigger_type = data.get('trigger_type', 'schedule')
+    trigger_config = json.dumps(data.get('trigger_config', {}))
+    action_type = data.get('action_type', 'process_wishlist')
+    action_config = json.dumps(data.get('action_config', {}))
+    then_actions = data.get('then_actions', [])
+    then_actions_json = json.dumps(then_actions)
+
+    if then_actions:
+        notify_type = then_actions[0].get('type')
+        notify_config = json.dumps(then_actions[0].get('config', {}))
+    else:
+        notify_type = data.get('notify_type') or None
+        notify_config = json.dumps(data.get('notify_config', {})) if notify_type else '{}'
+
+    cycle_path = _check_create_cycle(
+        automation_engine, database, profile_id,
+        trigger_type, trigger_config, then_actions_json, then_actions,
+    )
+    if cycle_path:
+        return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
+
+    group_name = data.get('group_name') or None
+    owned_by = data.get('owned_by') or None
+    auto_id = database.create_automation(
+        name, trigger_type, trigger_config, action_type, action_config,
+        profile_id, notify_type, notify_config, then_actions_json, group_name,
+        owned_by=owned_by,
+    )
+    if auto_id is None:
+        return {'error': 'Failed to create automation'}, 500
+
+    if automation_engine:
+        automation_engine.schedule_automation(auto_id)
+    return {'success': True, 'id': auto_id}, 200
+
+
+def update_automation(
+    database,
+    automation_engine,
+    automation_id: int,
+    data: dict,
+) -> tuple[dict, int]:
+    """Update + reschedule an automation. Returns (response_body, http_status)."""
+    update_fields: dict[str, Any] = {}
+    if 'name' in data:
+        update_fields['name'] = data['name'].strip()
+    if 'trigger_type' in data:
+        update_fields['trigger_type'] = data['trigger_type']
+    if 'trigger_config' in data:
+        update_fields['trigger_config'] = json.dumps(data['trigger_config'])
+    if 'action_type' in data:
+        update_fields['action_type'] = data['action_type']
+    if 'action_config' in data:
+        update_fields['action_config'] = json.dumps(data['action_config'])
+    if 'then_actions' in data:
+        then_actions = data['then_actions']
+        update_fields['then_actions'] = json.dumps(then_actions)
+        if then_actions:
+            update_fields['notify_type'] = then_actions[0].get('type')
+            update_fields['notify_config'] = json.dumps(then_actions[0].get('config', {}))
+        else:
+            update_fields['notify_type'] = None
+            update_fields['notify_config'] = '{}'
+    elif 'notify_type' in data:
+        update_fields['notify_type'] = data['notify_type'] or None
+    if 'notify_config' in data and 'then_actions' not in data:
+        update_fields['notify_config'] = json.dumps(data['notify_config'])
+    if 'group_name' in data:
+        update_fields['group_name'] = data['group_name'] or None
+    if 'owned_by' in data:
+        update_fields['owned_by'] = data['owned_by'] or None
+
+    if not update_fields:
+        return {'error': 'No fields to update'}, 400
+
+    cycle_path = _check_update_cycle(automation_engine, database, automation_id, data)
+    if cycle_path:
+        return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
+
+    # Schedule-shape changes must invalidate the stored next_run so the
+    # scheduler recomputes it; otherwise restart-survival logic keeps the
+    # leftover timestamp from the previous interval.
+    if {'trigger_type', 'trigger_config'} & update_fields.keys():
+        update_fields['next_run'] = None
+
+    success = database.update_automation(automation_id, **update_fields)
+    if not success:
+        return {'error': 'Automation not found'}, 404
+
+    if automation_engine:
+        auto = database.get_automation(automation_id)
+        if auto and auto.get('enabled'):
+            automation_engine.schedule_automation(automation_id)
+        else:
+            automation_engine.cancel_automation(automation_id)
+    return {'success': True}, 200
+
+
+def batch_update_group(database, automation_ids: list, group_name: Optional[str]) -> tuple[dict, int]:
+    """Move/rename a set of automations into a single group (or ungroup)."""
+    if not automation_ids or not isinstance(automation_ids, list):
+        return {'error': 'automation_ids must be a non-empty list'}, 400
+    try:
+        automation_ids = [int(aid) for aid in automation_ids]
+    except (ValueError, TypeError):
+        return {'error': 'automation_ids must contain integers'}, 400
+
+    updated = database.batch_update_group(automation_ids, group_name)
+    return {'success': True, 'updated': updated}, 200
+
+
+def bulk_toggle(
+    database,
+    automation_engine,
+    automation_ids: list,
+    enabled: bool,
+) -> tuple[dict, int]:
+    """Bulk enable/disable a set of automations + reschedule each affected."""
+    if not automation_ids or not isinstance(automation_ids, list):
+        return {'error': 'automation_ids must be a non-empty list'}, 400
+    try:
+        automation_ids = [int(aid) for aid in automation_ids]
+    except (ValueError, TypeError):
+        return {'error': 'automation_ids must contain integers'}, 400
+
+    updated = database.bulk_set_enabled(automation_ids, bool(enabled))
+
+    if automation_engine and updated > 0:
+        for aid in automation_ids:
+            auto = database.get_automation(aid)
+            if auto:
+                if auto.get('enabled'):
+                    automation_engine.schedule_automation(auto)
+                else:
+                    automation_engine.cancel_automation(aid)
+    return {'success': True, 'updated': updated}, 200
+
+
+def delete_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
+    """Delete an automation. System automations are protected."""
+    auto = database.get_automation(automation_id)
+    if auto and auto.get('is_system'):
+        return {'error': 'System automations cannot be deleted'}, 403
+    if automation_engine:
+        automation_engine.cancel_automation(automation_id)
+    success = database.delete_automation(automation_id)
+    if not success:
+        return {'error': 'Automation not found'}, 404
+    return {'success': True}, 200
+
+
+def duplicate_automation(
+    database,
+    automation_engine,
+    profile_id: int,
+    automation_id: int,
+) -> tuple[dict, int]:
+    """Duplicate an automation. System automations are protected."""
+    auto = database.get_automation(automation_id)
+    if not auto:
+        return {'error': 'Automation not found'}, 404
+    if auto.get('is_system'):
+        return {'error': 'System automations cannot be duplicated'}, 403
+    new_id = database.create_automation(
+        name=f"{auto['name']} (Copy)",
+        trigger_type=auto['trigger_type'],
+        trigger_config=auto.get('trigger_config', '{}'),
+        action_type=auto['action_type'],
+        action_config=auto.get('action_config', '{}'),
+        profile_id=profile_id,
+        notify_type=auto.get('notify_type'),
+        notify_config=auto.get('notify_config', '{}'),
+        then_actions=auto.get('then_actions', '[]'),
+        group_name=auto.get('group_name'),
+    )
+    if new_id is None:
+        return {'error': 'Failed to duplicate automation'}, 500
+    if automation_engine:
+        automation_engine.schedule_automation(new_id)
+    return {'success': True, 'id': new_id}, 200
+
+
+def toggle_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
+    """Toggle an automation's enabled state + reschedule/cancel."""
+    success = database.toggle_automation(automation_id)
+    if not success:
+        return {'error': 'Automation not found'}, 404
+
+    if automation_engine:
+        auto = database.get_automation(automation_id)
+        if auto and auto.get('enabled'):
+            automation_engine.schedule_automation(automation_id)
+        else:
+            automation_engine.cancel_automation(automation_id)
+    return {'success': True}, 200
+
+
+def run_automation(automation_engine, automation_id: int, profile_id: int) -> tuple[dict, int]:
+    """Manually trigger an automation."""
+    if not automation_engine:
+        return {'error': 'Automation engine not available'}, 500
+    success = automation_engine.run_now(automation_id, profile_id=profile_id)
+    if not success:
+        return {'error': 'Automation not found'}, 404
+    return {'success': True}, 200
+
+
+def get_history(database, automation_id: int, *, limit: int, offset: int) -> dict:
+    """Run-history page for an automation, with log_lines/result_json parsed."""
+    data = database.get_automation_run_history(automation_id, limit=limit, offset=offset)
+    for entry in data.get('history', []):
+        if entry.get('log_lines'):
+            try:
+                entry['log_lines'] = json.loads(entry['log_lines'])
+            except (json.JSONDecodeError, TypeError):
+                entry['log_lines'] = []
+        else:
+            entry['log_lines'] = []
+        if entry.get('result_json'):
+            try:
+                entry['result_json'] = json.loads(entry['result_json'])
+            except (json.JSONDecodeError, TypeError):
+                pass
+    data['automation_id'] = automation_id
+    return data

core/automation/blocks.py

@@ -0,0 +1,219 @@
+"""Static block definitions for the automation builder UI.
+
+Returned verbatim by `/api/automations/blocks` (with `known_signals`
+injected by the route from `signals.collect_known_signals`).
+
+Three top-level lists:
+- `TRIGGERS` — WHEN blocks: schedule, daily/weekly time, app started,
+  event triggers (track_downloaded, batch_complete, etc.), signal_received,
+  webhook_received.
+- `ACTIONS` — DO blocks: process_wishlist, scan_library, etc.
+- `NOTIFICATIONS` — THEN blocks: discord/pushbullet/telegram/webhook,
+  plus fire_signal and run_script then-actions.
+"""
+
+from __future__ import annotations
+
+TRIGGERS: list[dict] = [
+    {"type": "schedule", "label": "Schedule", "icon": "clock", "description": "Run on a timer interval", "available": True,
+     "config_fields": [
+         {"key": "interval", "type": "number", "label": "Every", "default": 6, "min": 1},
+         {"key": "unit", "type": "select", "label": "Unit",
+          "options": [{"value": "minutes", "label": "Minutes"}, {"value": "hours", "label": "Hours"}, {"value": "days", "label": "Days"}],
+          "default": "hours"}
+     ]},
+    {"type": "daily_time", "label": "Daily Time", "icon": "clock", "description": "Run every day at a specific time", "available": True,
+     "config_fields": [
+         {"key": "time", "type": "time", "label": "At", "default": "03:00"}
+     ]},
+    {"type": "weekly_time", "label": "Weekly Schedule", "icon": "calendar", "description": "Run on specific days of the week at a set time", "available": True,
+     "config_fields": [
+         {"key": "time", "type": "time", "label": "At", "default": "03:00"},
+         {"key": "days", "type": "multi_select", "label": "Days",
+          "options": [{"value": "mon", "label": "Mon"}, {"value": "tue", "label": "Tue"}, {"value": "wed", "label": "Wed"},
+                      {"value": "thu", "label": "Thu"}, {"value": "fri", "label": "Fri"}, {"value": "sat", "label": "Sat"}, {"value": "sun", "label": "Sun"}]}
+     ]},
+    {"type": "app_started", "label": "App Started", "icon": "power", "description": "When SoulSync starts up", "available": True},
+    {"type": "track_downloaded", "label": "Track Downloaded", "icon": "download", "description": "When a track finishes downloading", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["artist", "title", "album", "quality"],
+     "variables": ["artist", "title", "album", "quality"]},
+    {"type": "batch_complete", "label": "Batch Complete", "icon": "check-circle", "description": "When an album/playlist download finishes", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["playlist_name"],
+     "variables": ["playlist_name", "total_tracks", "completed_tracks", "failed_tracks"]},
+    {"type": "watchlist_new_release", "label": "New Release Found", "icon": "bell", "description": "When watchlist detects new music", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["artist"],
+     "variables": ["artist", "new_tracks", "added_to_wishlist"]},
+    {"type": "playlist_synced", "label": "Playlist Synced", "icon": "refresh", "description": "When a playlist sync completes", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["playlist_name"],
+     "variables": ["playlist_name", "total_tracks", "matched_tracks", "synced_tracks", "failed_tracks"]},
+    {"type": "playlist_changed", "label": "Playlist Changed", "icon": "edit", "description": "When a mirrored playlist detects track changes from source", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["playlist_name"],
+     "variables": ["playlist_name", "old_count", "new_count", "added", "removed"]},
+    {"type": "discovery_completed", "label": "Discovery Complete", "icon": "search", "description": "When playlist track discovery finishes", "available": True,
+     "has_conditions": True,
+     "condition_fields": ["playlist_name"],
+     "variables": ["playlist_name", "total_tracks", "discovered_count", "failed_count", "skipped_count"]},
+    # Phase 3 triggers
+    {"type": "wishlist_processing_completed", "label": "Wishlist Processed", "icon": "check-circle",
+     "description": "When auto-wishlist processing finishes", "available": True,
+     "variables": ["tracks_processed", "tracks_found", "tracks_failed"]},
+    {"type": "watchlist_scan_completed", "label": "Watchlist Scan Done", "icon": "check-circle",
+     "description": "When watchlist scan finishes", "available": True,
+     "variables": ["artists_scanned", "new_tracks_found", "tracks_added"]},
+    {"type": "database_update_completed", "label": "Database Updated", "icon": "database",
+     "description": "When library database refresh finishes", "available": True,
+     "variables": ["total_artists", "total_albums", "total_tracks"]},
+    {"type": "library_scan_completed", "label": "Library Scan Done", "icon": "hard-drive",
+     "description": "When media library scan finishes", "available": True,
+     "variables": ["server_type"]},
+    {"type": "download_failed", "label": "Download Failed", "icon": "x-circle",
+     "description": "When a track permanently fails to download", "available": True,
+     "has_conditions": True, "condition_fields": ["artist", "title", "reason"],
+     "variables": ["artist", "title", "reason"]},
+    {"type": "download_quarantined", "label": "File Quarantined", "icon": "alert-triangle",
+     "description": "When AcoustID verification fails", "available": True,
+     "has_conditions": True, "condition_fields": ["artist", "title"],
+     "variables": ["artist", "title", "reason"]},
+    {"type": "wishlist_item_added", "label": "Wishlist Item Added", "icon": "plus-circle",
+     "description": "When a track is added to wishlist", "available": True,
+     "has_conditions": True, "condition_fields": ["artist", "title"],
+     "variables": ["artist", "title", "reason"]},
+    {"type": "watchlist_artist_added", "label": "Artist Watched", "icon": "user-plus",
+     "description": "When an artist is added to watchlist", "available": True,
+     "has_conditions": True, "condition_fields": ["artist"],
+     "variables": ["artist", "artist_id"]},
+    {"type": "watchlist_artist_removed", "label": "Artist Unwatched", "icon": "user-minus",
+     "description": "When an artist is removed from watchlist", "available": True,
+     "has_conditions": True, "condition_fields": ["artist"],
+     "variables": ["artist", "artist_id"]},
+    {"type": "import_completed", "label": "Import Complete", "icon": "upload",
+     "description": "When album/track import finishes", "available": True,
+     "has_conditions": True, "condition_fields": ["artist", "album_name"],
+     "variables": ["track_count", "album_name", "artist"]},
+    {"type": "mirrored_playlist_created", "label": "Playlist Mirrored", "icon": "copy",
+     "description": "When a new playlist is mirrored", "available": True,
+     "has_conditions": True, "condition_fields": ["playlist_name", "source"],
+     "variables": ["playlist_name", "source", "track_count"]},
+    {"type": "quality_scan_completed", "label": "Quality Scan Done", "icon": "bar-chart",
+     "description": "When quality scan finishes", "available": True,
+     "variables": ["quality_met", "low_quality", "total_scanned"]},
+    {"type": "duplicate_scan_completed", "label": "Duplicate Scan Done", "icon": "layers",
+     "description": "When duplicate cleaner finishes", "available": True,
+     "variables": ["files_scanned", "duplicates_found", "space_freed"]},
+    # Signal trigger
+    {"type": "signal_received", "label": "Signal Received", "icon": "zap",
+     "description": "When another automation fires a named signal", "available": True,
+     "config_fields": [
+         {"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
+     ],
+     "variables": ["signal_name"]},
+    # Webhook trigger
+    {"type": "webhook_received", "label": "Webhook Received", "icon": "globe",
+     "description": "When an external API request is received (POST /api/v1/request)", "available": True,
+     "variables": ["query", "request_id", "source"]},
+]
+
+
+ACTIONS: list[dict] = [
+    {"type": "process_wishlist", "label": "Process Wishlist", "icon": "list", "description": "Retry failed downloads from wishlist", "available": True,
+     "config_fields": [{"key": "category", "type": "select", "label": "Category", "options": [{"value": "all", "label": "All"}, {"value": "albums", "label": "Albums"}, {"value": "singles", "label": "Singles"}], "default": "all"}]},
+    {"type": "scan_watchlist", "label": "Scan Watchlist", "icon": "eye", "description": "Check watched artists for new releases", "available": True},
+    {"type": "scan_library", "label": "Scan Library", "icon": "refresh", "description": "Trigger media server library scan", "available": True},
+    {"type": "refresh_mirrored", "label": "Refresh Mirrored Playlist", "icon": "copy", "description": "Re-fetch playlist from source and update mirror", "available": True,
+     "config_fields": [
+         {"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
+         {"key": "all", "type": "checkbox", "label": "Refresh all mirrored playlists", "default": False}
+     ]},
+    {"type": "sync_playlist", "label": "Sync Playlist", "icon": "sync", "description": "Sync mirrored playlist to media server", "available": True,
+     "config_fields": [
+         {"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"}
+     ]},
+    {"type": "discover_playlist", "label": "Discover Playlist", "icon": "search", "description": "Find official Spotify/iTunes metadata for mirrored playlist tracks", "available": True,
+     "config_fields": [
+         {"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
+         {"key": "all", "type": "checkbox", "label": "Discover all mirrored playlists", "default": False}
+     ]},
+    {"type": "playlist_pipeline", "label": "Playlist Pipeline", "icon": "rocket",
+     "description": "Full lifecycle: refresh → discover → sync → download missing. One automation for the entire flow.",
+     "available": True,
+     "config_fields": [
+         {"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
+         {"key": "all", "type": "checkbox", "label": "Process all mirrored playlists", "default": False},
+         {"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
+     ]},
+    {"type": "personalized_pipeline", "label": "Personalized Playlist Pipeline", "icon": "sparkles",
+     "description": "Sync personalized / discover-page playlists (Hidden Gems, Time Machine, Fresh Tape, etc.) to your media server + queue missing tracks for download.",
+     "available": True,
+     "config_fields": [
+         {"key": "kinds", "type": "personalized_playlist_select", "label": "Playlists to sync",
+          "description": "Multi-select: Hidden Gems, Discovery Shuffle, Time Machine (per decade), Genre playlists, Fresh Tape, The Archives, Seasonal Mix (per season)"},
+         {"key": "refresh_first", "type": "checkbox", "label": "Refresh playlists before sync (regenerate snapshots)", "default": False},
+         {"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
+     ]},
+    {"type": "notify_only", "label": "Notify Only", "icon": "bell", "description": "No action — just send notification", "available": True},
+    # Phase 3 actions
+    {"type": "start_database_update", "label": "Update Database", "icon": "database",
+     "description": "Trigger library database refresh", "available": True,
+     "config_fields": [
+         {"key": "full_refresh", "type": "checkbox", "label": "Full refresh (slower)", "default": False}
+     ]},
+    {"type": "run_duplicate_cleaner", "label": "Run Duplicate Cleaner", "icon": "layers",
+     "description": "Scan for and remove duplicate files", "available": True},
+    {"type": "clear_quarantine", "label": "Clear Quarantine", "icon": "trash",
+     "description": "Delete all quarantined files", "available": True},
+    {"type": "cleanup_wishlist", "label": "Clean Up Wishlist", "icon": "filter",
+     "description": "Remove duplicate/owned tracks from wishlist", "available": True},
+    {"type": "update_discovery_pool", "label": "Update Discovery", "icon": "compass",
+     "description": "Refresh discovery pool with new tracks", "available": True},
+    {"type": "start_quality_scan", "label": "Run Quality Scan", "icon": "bar-chart",
+     "description": "Run the Quality Upgrade Finder (scope is set in Library Maintenance)", "available": True},
+    {"type": "backup_database", "label": "Backup Database", "icon": "save",
+     "description": "Create timestamped database backup", "available": True},
+    {"type": "refresh_beatport_cache", "label": "Refresh Beatport Cache", "icon": "music",
+     "description": "Scrape Beatport homepage and warm the cache", "available": True},
+    {"type": "clean_search_history", "label": "Clean Search History", "icon": "trash-2",
+     "description": "Remove old searches from Soulseek", "available": True},
+    {"type": "clean_completed_downloads", "label": "Clean Completed Downloads", "icon": "check-square",
+     "description": "Clear completed downloads and empty directories", "available": True},
+    {"type": "full_cleanup", "label": "Full Cleanup", "icon": "trash",
+     "description": "Clear quarantine, download queue, import folder, and search history in one sweep", "available": True},
+    {"type": "deep_scan_library", "label": "Deep Scan Library", "icon": "search",
+     "description": "Full library comparison without losing enrichment data", "available": True},
+    {"type": "run_script", "label": "Run Script", "icon": "terminal",
+     "description": "Execute a script from the scripts folder", "available": True},
+    {"type": "search_and_download", "label": "Search & Download", "icon": "download",
+     "description": "Search for a track and download the best match", "available": True,
+     "config_fields": [
+         {"key": "query", "type": "text", "label": "Search Query",
+          "placeholder": "Artist - Track (leave empty to use trigger's query)"}
+     ]},
+]
+
+
+NOTIFICATIONS: list[dict] = [
+    {"type": "discord_webhook", "label": "Discord Webhook", "icon": "message", "description": "Send a Discord notification", "available": True,
+     "variables": ["time", "name", "run_count", "status"]},
+    {"type": "pushbullet", "label": "Pushbullet", "icon": "push", "description": "Push notification to phone/desktop", "available": True,
+     "variables": ["time", "name", "run_count", "status"]},
+    {"type": "telegram", "label": "Telegram", "icon": "message", "description": "Send a Telegram message", "available": True,
+     "variables": ["time", "name", "run_count", "status"]},
+    {"type": "webhook", "label": "Webhook (POST)", "icon": "globe", "description": "Send a POST request to any URL", "available": True,
+     "variables": ["time", "name", "run_count", "status"]},
+    # Signal fire action
+    {"type": "fire_signal", "label": "Fire Signal", "icon": "zap",
+     "description": "Fire a signal that other automations can listen for", "available": True,
+     "config_fields": [
+         {"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
+     ]},
+    # Run script then-action
+    {"type": "run_script", "label": "Run Script", "icon": "terminal",
+     "description": "Execute a script after the action completes", "available": True,
+     "config_fields": [
+         {"key": "script_name", "type": "script_select", "label": "Script"}
+     ]},
+]

core/automation/deps.py

@@ -0,0 +1,163 @@
+"""Dependency-injection surface for automation handlers.
+
+Each handler in ``core.automation.handlers`` is a top-level pure
+function that accepts ``(config: dict, deps: AutomationDeps)`` instead
+of reaching for module-level globals in ``web_server``. The deps
+namespace bundles every callable, client, and mutable-state container
+the handlers need.
+
+Construction happens once at app startup in ``web_server.py``:
+
+    from core.automation.deps import AutomationDeps, AutomationState
+    state = AutomationState()
+    deps = AutomationDeps(
+        engine=automation_engine,
+        state=state,
+        get_database=get_database,
+        spotify_client=spotify_client,
+        ...
+    )
+    register_all(deps)
+
+Tests construct a fake ``AutomationDeps`` with stub callables — every
+handler is then exercisable without spinning up Flask, the DB, or
+the real media-server clients.
+"""
+
+from __future__ import annotations
+
+import threading
+from dataclasses import dataclass, field
+from typing import Any, Callable, Dict, Optional
+
+
+@dataclass
+class AutomationState:
+    """Mutable flags shared across handler invocations.
+
+    Pre-refactor each was a ``global`` or ``nonlocal`` variable inside
+    the registration closure. Lifted here so handlers + their guards
+    can read/write a single object instead of importing globals.
+
+    All mutations should hold ``lock``; the helper methods below do
+    so for the common get/set patterns.
+    """
+
+    scan_library_automation_id: Optional[str] = None
+    db_update_automation_id: Optional[str] = None
+    pipeline_running: bool = False
+    lock: threading.Lock = field(default_factory=threading.Lock)
+
+    def is_scan_library_active(self) -> bool:
+        with self.lock:
+            return self.scan_library_automation_id is not None
+
+    def is_pipeline_running(self) -> bool:
+        with self.lock:
+            return self.pipeline_running
+
+    def try_start_pipeline(self) -> bool:
+        """Atomically mark the shared playlist pipeline as running."""
+        with self.lock:
+            if self.pipeline_running:
+                return False
+            self.pipeline_running = True
+            return True
+
+    def set_scan_library_id(self, automation_id: Optional[str]) -> None:
+        with self.lock:
+            self.scan_library_automation_id = automation_id
+
+    def set_pipeline_running(self, value: bool) -> None:
+        with self.lock:
+            self.pipeline_running = value
+
+
+@dataclass
+class AutomationDeps:
+    """Bundle of every callable + client an automation handler may need.
+
+    Add fields as new handlers are extracted. Every field is required
+    at construction (no defaults) so a missing dep fails loudly at
+    startup, not silently mid-handler.
+    """
+
+    # --- Engine + shared state ---
+    engine: Any                                          # AutomationEngine instance
+    state: AutomationState
+    config_manager: Any                                  # config.settings.ConfigManager singleton
+    update_progress: Callable[..., None]                 # _update_automation_progress
+    logger: Any                                          # module-level logger from utils.logging_config
+
+    # --- Service clients (each may be None depending on user config) ---
+    get_database: Callable[[], Any]                      # late-binding so tests don't need DB
+    spotify_client: Any
+    tidal_client: Any
+    web_scan_manager: Any
+
+    # --- Background-task entry points ---
+    process_wishlist_automatically: Callable[..., Any]
+    process_watchlist_scan_automatically: Callable[..., Any]
+    is_wishlist_actually_processing: Callable[[], bool]
+    is_watchlist_actually_scanning: Callable[[], bool]
+    get_watchlist_scan_state: Callable[[], dict]         # accessor returns the live mutable dict
+
+    # --- Playlist pipeline entry points ---
+    run_playlist_discovery_worker: Callable[..., Any]
+    run_sync_task: Callable[..., Any]
+    run_playlist_organize_download: Callable[..., Dict[str, Any]]
+    missing_download_executor: Any
+    load_sync_status_file: Callable[[], dict]
+    get_deezer_client: Callable[[], Any]
+    parse_youtube_playlist: Callable[[str], Any]
+    get_sync_states: Callable[[], dict]                  # accessor returns the live dict shared with the sync UI
+
+    # --- Database update + quality scanner (shared state + executors) ---
+    set_db_update_automation_id: Callable[[Optional[str]], None]  # syncs the legacy `_db_update_automation_id` global so the live DB-update progress callbacks (which still read the global directly) keep firing for the active automation
+    get_db_update_state: Callable[[], dict]
+    db_update_lock: Any                                  # threading.Lock
+    db_update_executor: Any                              # ThreadPoolExecutor
+    run_db_update_task: Callable[..., Any]
+    run_deep_scan_task: Callable[..., Any]
+    get_duplicate_cleaner_state: Callable[[], dict]
+    duplicate_cleaner_lock: Any
+    duplicate_cleaner_executor: Any
+    run_duplicate_cleaner: Callable[..., Any]
+    # Triggers a "Run Now" of a library-maintenance repair job by id (e.g.
+    # 'quality_upgrade'). Returns truthy if the job was queued. Replaces the old
+    # standalone quality-scanner executor/state (the scanner is now a repair job).
+    run_repair_job_now: Callable[[str], Any]
+
+    # --- Download orchestrator + queue accessors ---
+    download_orchestrator: Any
+    run_async: Callable[..., Any]
+    tasks_lock: Any
+    get_download_batches: Callable[[], dict]
+    get_download_tasks: Callable[[], dict]
+    sweep_empty_download_directories: Callable[[], int]
+    get_staging_path: Callable[[], str]
+
+    # --- Maintenance helpers ---
+    docker_resolve_path: Callable[[str], str]
+    get_current_profile_id: Callable[[], int]
+    get_watchlist_scanner: Callable[[Any], Any]
+    get_app: Callable[[], Any]                           # Flask app for test_client (beatport refresh)
+    get_beatport_data_cache: Callable[[], dict]
+
+    # --- Progress + history callbacks (used by register_all to wire
+    # the engine's progress callback hooks). ---
+    init_automation_progress: Callable[..., Any]
+    record_progress_history: Callable[..., Any]
+
+    # --- Personalized playlist pipeline ---
+    # Lazy builder so the pipeline handler can construct a fresh
+    # PersonalizedPlaylistManager per run (cheap accessors inside,
+    # no caching needed yet).
+    build_personalized_manager: Callable[[], Any]
+
+    # --- Unified PlaylistSource registry ---
+    # Optional so test fixtures that don't exercise refresh_mirrored
+    # can keep their existing scaffolding. Production wiring in
+    # ``web_server.py`` always populates it via
+    # ``core.playlists.sources.bootstrap.build_playlist_source_registry``.
+    playlist_source_registry: Optional[Any] = None

core/automation/handlers/__init__.py

@@ -0,0 +1,64 @@
+"""Per-action automation handlers.
+
+Each module in this subpackage exposes one top-level handler function
+(or a small cluster of related handlers) of the form::
+
+    def auto_<action_name>(config: dict, deps: AutomationDeps) -> dict
+
+The ``register_all`` helper in :mod:`registration` wires every handler
+to the engine in one place. ``web_server.py`` calls
+``register_all(deps)`` once at startup.
+"""
+
+from core.automation.handlers.process_wishlist import auto_process_wishlist
+from core.automation.handlers.scan_watchlist import auto_scan_watchlist
+from core.automation.handlers.scan_library import auto_scan_library
+from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
+from core.automation.handlers.sync_playlist import auto_sync_playlist
+from core.automation.handlers.discover_playlist import auto_discover_playlist
+from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
+from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
+from core.automation.handlers.database_update import auto_start_database_update, auto_deep_scan_library
+from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
+from core.automation.handlers.quality_scanner import auto_start_quality_scan
+from core.automation.handlers.maintenance import (
+    auto_clear_quarantine,
+    auto_cleanup_wishlist,
+    auto_update_discovery_pool,
+    auto_backup_database,
+    auto_refresh_beatport_cache,
+)
+from core.automation.handlers.download_cleanup import (
+    auto_clean_search_history,
+    auto_clean_completed_downloads,
+    auto_full_cleanup,
+)
+from core.automation.handlers.run_script import auto_run_script
+from core.automation.handlers.search_and_download import auto_search_and_download
+from core.automation.handlers.registration import register_all
+
+__all__ = [
+    'auto_process_wishlist',
+    'auto_scan_watchlist',
+    'auto_scan_library',
+    'auto_refresh_mirrored',
+    'auto_sync_playlist',
+    'auto_discover_playlist',
+    'auto_playlist_pipeline',
+    'auto_personalized_pipeline',
+    'auto_start_database_update',
+    'auto_deep_scan_library',
+    'auto_run_duplicate_cleaner',
+    'auto_start_quality_scan',
+    'auto_clear_quarantine',
+    'auto_cleanup_wishlist',
+    'auto_update_discovery_pool',
+    'auto_backup_database',
+    'auto_refresh_beatport_cache',
+    'auto_clean_search_history',
+    'auto_clean_completed_downloads',
+    'auto_full_cleanup',
+    'auto_run_script',
+    'auto_search_and_download',
+    'register_all',
+]

core/automation/handlers/_pipeline_shared.py

@@ -0,0 +1,259 @@
+"""Shared helpers between mirrored + personalized playlist pipelines.
+
+Both pipelines end in the same shape:
+1. SYNC each playlist to the active media server.
+2. WISHLIST: trigger the wishlist processor for missing tracks.
+
+The differing prefix (mirrored = REFRESH external sources + DISCOVER
+metadata; personalized = SNAPSHOT manager-backed playlists) is owned
+by each pipeline. This module owns the SYNC + WISHLIST tail so both
+pipelines stay consistent + DRY.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Callable, Dict, List, Optional
+
+from core.automation.deps import AutomationDeps
+
+
+# Per-playlist sync poll cap (mirrored side already used this).
+_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS = 600
+# Sync-status final-state markers.
+_SYNC_TERMINAL_STATUSES = ('finished', 'complete', 'error', 'failed')
+
+
+def run_sync_and_wishlist(
+    deps: AutomationDeps,
+    automation_id: Optional[str],
+    playlists: List[Dict[str, Any]],
+    *,
+    sync_one_fn: Callable[[Dict[str, Any]], Dict[str, Any]],
+    sync_id_for_fn: Callable[[Dict[str, Any]], str],
+    skip_wishlist: bool = False,
+    progress_start: int = 56,
+    progress_end: int = 85,
+    sync_phase_label: str = 'Phase: Syncing to server...',
+    sync_phase_start_log: str = 'Sync',
+    wishlist_phase_label: str = 'Phase: Processing wishlist...',
+    wishlist_phase_start_log: str = 'Wishlist',
+) -> Dict[str, int]:
+    """Run the SYNC + WISHLIST tail of a playlist pipeline.
+
+    The caller supplies:
+    - ``playlists``: list of playlist payload dicts. Each must have at
+      least a ``name`` key (used in progress logs). The shape beyond
+      ``name`` is opaque to the helper — ``sync_one_fn`` receives the
+      payload and returns a sync_result dict.
+    - ``sync_one_fn(payload) -> sync_result``: launches sync for one
+      playlist. Result dict must carry ``status`` ∈ ``('started',
+      'skipped', 'error')`` and may carry ``reason``.
+    - ``sync_id_for_fn(payload) -> str``: returns the sync-state key
+      the helper polls on (so we can wait for the background sync
+      thread to complete + read the matched_tracks count).
+
+    Returns ``{'synced': int, 'skipped': int, 'errors': int,
+    'wishlist_queued': int}`` so the caller can stitch it into its
+    final status.
+    """
+    deps.update_progress(
+        automation_id,
+        progress=progress_start,
+        phase=sync_phase_label,
+        log_line=sync_phase_start_log,
+        log_type='info',
+    )
+
+    total_synced = 0
+    total_skipped = 0
+    sync_errors = 0
+    sync_states = deps.get_sync_states()
+    n_playlists = max(1, len(playlists))
+    progress_span = max(1, progress_end - progress_start - 1)
+
+    for pl_idx, pl in enumerate(playlists):
+        pl_name = pl.get('name', '')
+        sync_result = sync_one_fn(pl)
+        sync_status = sync_result.get('status', '')
+
+        if sync_status == 'started':
+            sync_id = sync_id_for_fn(pl)
+            sync_poll_start = time.time()
+            timed_out = True
+            while time.time() - sync_poll_start < _SYNC_PER_PLAYLIST_TIMEOUT_SECONDS:
+                if (sync_id in sync_states
+                        and sync_states[sync_id].get('status') in _SYNC_TERMINAL_STATUSES):
+                    timed_out = False
+                    break
+                time.sleep(2)
+                elapsed = int(time.time() - sync_poll_start)
+                sub_progress = progress_start + 1 + ((pl_idx + 1) / n_playlists) * progress_span
+                deps.update_progress(
+                    automation_id,
+                    progress=min(int(sub_progress), progress_end - 1),
+                    phase=f'{sync_phase_label.rstrip(".")} — "{pl_name}" ({elapsed}s)',
+                )
+
+            ss = sync_states.get(sync_id, {})
+            final_status = ss.get('status')
+            if timed_out:
+                sync_errors += 1
+                deps.update_progress(
+                    automation_id,
+                    log_line=f'Sync timed out "{pl_name}" after {_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS}s',
+                    log_type='error',
+                )
+                continue
+            if final_status in ('error', 'failed'):
+                sync_errors += 1
+                reason = ss.get('error') or ss.get('reason') or 'background sync failed'
+                deps.update_progress(
+                    automation_id,
+                    log_line=f'Sync error "{pl_name}": {reason}',
+                    log_type='error',
+                )
+                continue
+
+            ss_result = ss.get('result', ss.get('progress', {}))
+            matched = ss_result.get('matched_tracks', 0) if isinstance(ss_result, dict) else 0
+            total_synced += int(matched) if matched else 0
+            deps.update_progress(
+                automation_id,
+                log_line=f'Synced "{pl_name}": {matched} tracks matched',
+                log_type='success',
+            )
+
+        elif sync_status == 'skipped':
+            total_skipped += 1
+            reason = sync_result.get('reason', 'unchanged')
+            deps.update_progress(
+                automation_id,
+                log_line=f'Skipped "{pl_name}": {reason}',
+                log_type='skip',
+            )
+        elif sync_status == 'error':
+            sync_errors += 1
+            deps.update_progress(
+                automation_id,
+                log_line=f'Sync error "{pl_name}": {sync_result.get("reason", "unknown")}',
+                log_type='error',
+            )
+
+    deps.update_progress(
+        automation_id,
+        progress=progress_end,
+        phase=f'{sync_phase_label.rstrip(".")} complete',
+        log_line=f'Sync done: {total_synced} matched, {total_skipped} skipped, {sync_errors} errors',
+        log_type='success' if sync_errors == 0 else 'warning',
+    )
+
+    organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
+    organize_started = 0
+    if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
+        for pl in organize_playlists:
+            pl_id = pl.get('id')
+            if not pl_id:
+                continue
+            pl_name = pl.get('name', '')
+            try:
+                org_result = deps.run_playlist_organize_download(
+                    mirrored_playlist_id=int(pl_id),
+                    automation_id=automation_id,
+                )
+                if org_result.get('status') == 'started':
+                    organize_started += 1
+                    deps.update_progress(
+                        automation_id,
+                        log_line=f'Organize download started for "{pl_name}"',
+                        log_type='success',
+                    )
+                elif org_result.get('status') == 'skipped':
+                    deps.update_progress(
+                        automation_id,
+                        log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
+                        log_type='skip',
+                    )
+            except Exception as org_err:  # noqa: BLE001
+                deps.update_progress(
+                    automation_id,
+                    log_line=f'Organize download error for "{pl_name}": {org_err}',
+                    log_type='warning',
+                )
+
+    all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
+    effective_skip_wishlist = skip_wishlist or all_organize
+
+    wishlist_queued = run_wishlist_phase(
+        deps, automation_id,
+        skip=effective_skip_wishlist,
+        progress_pct=progress_end + 1,
+        wishlist_phase_label=wishlist_phase_label,
+        wishlist_phase_start_log=wishlist_phase_start_log,
+    )
+
+    return {
+        'synced': total_synced,
+        'skipped': total_skipped,
+        'errors': sync_errors,
+        'wishlist_queued': wishlist_queued,
+        'organize_downloads_started': organize_started,
+    }
+
+
+def run_wishlist_phase(
+    deps: AutomationDeps,
+    automation_id: Optional[str],
+    *,
+    skip: bool,
+    progress_pct: int,
+    wishlist_phase_label: str = 'Phase: Processing wishlist...',
+    wishlist_phase_start_log: str = 'Wishlist',
+) -> int:
+    """Trigger the wishlist processor unless skipped or already running.
+
+    Returns 1 when the processor was triggered, 0 otherwise. Errors are
+    logged but never raised — wishlist failure should not abort the
+    pipeline."""
+    if skip:
+        deps.update_progress(
+            automation_id,
+            progress=progress_pct,
+            log_line=f'{wishlist_phase_start_log}: skipped (disabled)',
+            log_type='skip',
+        )
+        return 0
+
+    deps.update_progress(
+        automation_id,
+        progress=progress_pct,
+        phase=wishlist_phase_label,
+        log_line=wishlist_phase_start_log,
+        log_type='info',
+    )
+
+    try:
+        if not deps.is_wishlist_actually_processing():
+            deps.process_wishlist_automatically(automation_id=None)
+            deps.update_progress(
+                automation_id,
+                log_line='Wishlist processing triggered',
+                log_type='success',
+            )
+            return 1
+        deps.update_progress(
+            automation_id,
+            log_line='Wishlist already running — skipped',
+            log_type='skip',
+        )
+        return 0
+    except Exception as e:  # noqa: BLE001 — wishlist failure must never abort pipeline
+        deps.update_progress(
+            automation_id,
+            log_line=f'Wishlist error: {e}',
+            log_type='warning',
+        )
+        return 0
+
+
+__all__ = ['run_sync_and_wishlist', 'run_wishlist_phase']

core/automation/handlers/database_update.py

@@ -0,0 +1,197 @@
+"""Automation handlers: ``start_database_update`` and
+``deep_scan_library`` actions.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_start_database_update`` and ``_auto_deep_scan_library``
+closures). Both share the same ``db_update_state`` / executor / lock
+infrastructure -- the only difference is which task they submit
+(``run_db_update_task`` vs ``run_deep_scan_task``).
+
+Pattern: pre-set state to running, submit task to executor, then
+poll the state dict until it transitions away from ``running``.
+Stall-detection emits a warning every 10 minutes when progress
+hasn't budged. 2-hour outer timeout caps the worst case.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+# Time out on STALL (no progress), not total runtime: a large library can scan
+# for many hours while progressing fine — a hard total cap would falsely mark a
+# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
+# up when progress hasn't moved for a long stretch, with a generous absolute
+# backstop against a truly stuck monitor loop.
+_STALL_WARNING_SECONDS = 600     # warn after 10 min with no progress (repeats)
+_STALL_TIMEOUT_SECONDS = 1800    # 30 min with no progress at all = genuinely stalled
+_ABSOLUTE_CAP_SECONDS = 86400    # 24h hard backstop (runaway-loop guard only)
+_POLL_INTERVAL_SECONDS = 3
+_INITIAL_DELAY_SECONDS = 1
+
+
+def scan_wait_action(
+    *,
+    status: str,
+    idle_seconds: float,
+    total_seconds: float,
+    stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
+    stall_warn_s: float = _STALL_WARNING_SECONDS,
+    abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
+) -> str:
+    """Decide what the monitor loop should do on a poll tick (pure/testable).
+
+    ``idle_seconds`` is time since progress last changed; ``total_seconds`` is
+    time since the wait began. Returns one of:
+    ``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
+    too long → give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
+    (stalled long enough to warn but not give up), or ``'continue'``.
+
+    Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
+    it never hits ``stall_timeout`` no matter how long the whole scan takes.
+    """
+    if status != 'running':
+        return 'finished'
+    if total_seconds >= abs_cap_s:
+        return 'abs_timeout'
+    if idle_seconds >= stall_timeout_s:
+        return 'stall_timeout'
+    if idle_seconds >= stall_warn_s:
+        return 'warn'
+    return 'continue'
+
+
+def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run a full or incremental DB update via ``run_db_update_task``."""
+    return _run_with_progress(
+        config, deps,
+        task=deps.run_db_update_task,
+        task_args=(config.get('full_refresh', False), deps.config_manager.get_active_media_server()),
+        initial_phase='Initializing...',
+        stall_label='Database update',
+        finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
+        timeout_label='Database update timed out after 24 hours',
+    )
+
+
+def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run a deep library scan via ``run_deep_scan_task``."""
+    return _run_with_progress(
+        config, deps,
+        task=deps.run_deep_scan_task,
+        task_args=(deps.config_manager.get_active_media_server(),),
+        initial_phase='Deep scan: Initializing...',
+        stall_label='Deep scan',
+        finished_extras=lambda: {},
+        timeout_label='Deep scan timed out after 24 hours',
+    )
+
+
+def _run_with_progress(
+    config: Dict[str, Any],
+    deps: AutomationDeps,
+    *,
+    task,
+    task_args: tuple,
+    initial_phase: str,
+    stall_label: str,
+    finished_extras,
+    timeout_label: str,
+) -> Dict[str, Any]:
+    """Shared poll-and-wait body for both DB-update handlers."""
+    automation_id = config.get('_automation_id')
+    state = deps.get_db_update_state()
+    if state.get('status') == 'running':
+        return {'status': 'skipped', 'reason': 'Database update already running'}
+    deps.state.db_update_automation_id = automation_id
+    # Sync legacy module global so the DB-update progress callbacks
+    # (still living in web_server.py) emit against this automation.
+    deps.set_db_update_automation_id(automation_id)
+
+    with deps.db_update_lock:
+        state.update({
+            'status': 'running', 'phase': initial_phase,
+            'progress': 0, 'current_item': '', 'processed': 0, 'total': 0,
+            'error_message': '',
+        })
+    deps.db_update_executor.submit(task, *task_args)
+
+    # Monitor progress (callbacks handle card updates, we just block until done).
+    # We time out on STALL, not total runtime: ``processed`` advances on every
+    # artist, so an actively-progressing scan keeps resetting the idle clock and
+    # is never falsely failed no matter how long the whole library takes.
+    time.sleep(_INITIAL_DELAY_SECONDS)
+    poll_start = time.time()
+    last_progress_time = time.time()
+    # Any of these advancing means the scan is alive. current_item (the artist
+    # being processed) changes every artist even when the rounded progress %
+    # holds steady, so it guards against a false stall during slow stretches.
+    last_progress_val = (0, 0, '')
+    last_warn_time = 0.0
+    outcome = 'finished'
+    while True:
+        time.sleep(_POLL_INTERVAL_SECONDS)
+        now = time.time()
+        with deps.db_update_lock:
+            current_status = state.get('status', 'idle')
+            current_val = (state.get('processed', 0), state.get('progress', 0),
+                           state.get('current_item', ''))
+        if current_val != last_progress_val:
+            last_progress_val = current_val
+            last_progress_time = now
+
+        action = scan_wait_action(
+            status=current_status,
+            idle_seconds=now - last_progress_time,
+            total_seconds=now - poll_start,
+        )
+        if action in ('finished', 'stall_timeout', 'abs_timeout'):
+            outcome = action
+            break
+        if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
+            idle_min = int((now - last_progress_time) / 60)
+            deps.update_progress(
+                automation_id,
+                log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
+                log_type='warning',
+            )
+            last_warn_time = now
+
+    if outcome == 'stall_timeout':
+        deps.update_progress(
+            automation_id, status='error', phase='Stalled',
+            log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
+            log_type='error',
+        )
+        return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
+    if outcome == 'abs_timeout':
+        deps.update_progress(
+            automation_id, status='error',
+            phase='Timed out', log_line=timeout_label, log_type='error',
+        )
+        return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
+
+    # Finished/error callback already updated the card — return matching status.
+    with deps.db_update_lock:
+        final_status = state.get('status', 'unknown')
+    if final_status == 'error':
+        return {
+            'status': 'error',
+            'reason': state.get('error_message', 'Unknown error'),
+            '_manages_own_progress': True,
+        }
+    with deps.db_update_lock:
+        stats = {
+            'status': 'completed', '_manages_own_progress': True,
+            'artists': state.get('total', 0),
+            'albums': state.get('total_albums', 0),
+            'tracks': state.get('total_tracks', 0),
+            'removed_artists': state.get('removed_artists', 0),
+            'removed_albums': state.get('removed_albums', 0),
+            'removed_tracks': state.get('removed_tracks', 0),
+        }
+    stats.update(finished_extras())
+    return stats

core/automation/handlers/discover_playlist.py

@@ -0,0 +1,48 @@
+"""Automation handler: ``discover_playlist`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_discover_playlist`` closure). Kicks off background discovery
+of official Spotify / iTunes metadata for mirrored playlist tracks.
+The worker runs in a daemon thread and emits its own progress; this
+handler returns immediately after launching it (``_manages_own_progress``).
+"""
+
+from __future__ import annotations
+
+import threading
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_discover_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Discover official Spotify/iTunes metadata for mirrored
+    playlist tracks. Runs the worker in a background thread."""
+    db = deps.get_database()
+    playlist_id = config.get('playlist_id')
+    discover_all = config.get('all', False)
+
+    if discover_all:
+        playlists = db.get_mirrored_playlists()
+    elif playlist_id:
+        p = db.get_mirrored_playlist(int(playlist_id))
+        playlists = [p] if p else []
+    else:
+        return {'status': 'error', 'reason': 'No playlist specified'}
+
+    if not playlists:
+        return {'status': 'error', 'reason': 'No playlists found'}
+
+    threading.Thread(
+        target=deps.run_playlist_discovery_worker,
+        args=(playlists, config.get('_automation_id')),
+        daemon=True,
+        name='auto-discover-playlist',
+    ).start()
+    names = ', '.join(p['name'] for p in playlists[:3])
+    return {
+        'status': 'started',
+        'playlist_count': str(len(playlists)),
+        'playlists': names,
+        '_manages_own_progress': True,
+    }

core/automation/handlers/download_cleanup.py

@@ -0,0 +1,267 @@
+"""Automation handlers: download-queue cleanup actions.
+
+Lifted from ``web_server._register_automation_handlers``:
+- ``clean_search_history``       → :func:`auto_clean_search_history`
+- ``clean_completed_downloads``  → :func:`auto_clean_completed_downloads`
+- ``full_cleanup``               → :func:`auto_full_cleanup`
+
+All three share the download-orchestrator + tasks_lock /
+download_batches / download_tasks accessors. ``full_cleanup`` is a
+multi-step orchestration that pulls in quarantine purge + staging
+sweep on top of the queue cleanup -- kept as one big handler since
+its phases share state-detection logic.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil as _shutil
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+# ─── clean_search_history ────────────────────────────────────────────
+
+
+def auto_clean_search_history(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Remove old searches from Soulseek when configured."""
+    automation_id = config.get('_automation_id')
+    # Skip if soulseek is not the active download source or in hybrid order.
+    dl_mode = deps.config_manager.get('download_source.mode', 'hybrid')
+    hybrid_order = deps.config_manager.get(
+        'download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'],
+    )
+    soulseek_active = (
+        dl_mode == 'soulseek'
+        or (dl_mode == 'hybrid' and 'soulseek' in hybrid_order)
+    )
+    # Reach the underlying SoulseekClient via the orchestrator's
+    # generic accessor.
+    slskd = deps.download_orchestrator.client('soulseek') if deps.download_orchestrator else None
+    if not soulseek_active or not slskd or not slskd.base_url:
+        deps.update_progress(automation_id, log_line='Soulseek not active — skipped', log_type='skip')
+        return {'status': 'skipped'}
+    if not deps.config_manager.get('soulseek.auto_clear_searches', True):
+        deps.update_progress(
+            automation_id, log_line='Auto-clear disabled in settings', log_type='skip',
+        )
+        return {'status': 'skipped'}
+    try:
+        success = deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
+            keep_searches=50, trigger_threshold=200,
+        ))
+        if success:
+            deps.update_progress(
+                automation_id,
+                log_line='Search history maintenance completed',
+                log_type='success',
+            )
+            return {'status': 'completed'}
+        else:
+            deps.update_progress(automation_id, log_line='No cleanup needed', log_type='skip')
+            return {'status': 'completed'}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise
+        return {'status': 'error', 'error': str(e)}
+
+
+# ─── clean_completed_downloads ───────────────────────────────────────
+
+
+def auto_clean_completed_downloads(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Clear completed downloads + sweep empty download directories.
+    Skips when active batches or post-processing is in flight."""
+    automation_id = config.get('_automation_id')
+    try:
+        has_active_batches = False
+        has_post_processing = False
+        with deps.tasks_lock:
+            batches = deps.get_download_batches()
+            for batch_data in batches.values():
+                if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
+                    has_active_batches = True
+                    break
+            if not has_active_batches:
+                tasks = deps.get_download_tasks()
+                for task_data in tasks.values():
+                    if task_data.get('status') == 'post_processing':
+                        has_post_processing = True
+                        break
+
+        if has_active_batches:
+            deps.update_progress(
+                automation_id, log_line='Skipped — downloads active', log_type='skip',
+            )
+            return {'status': 'completed'}
+
+        deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
+        if not has_post_processing:
+            deps.sweep_empty_download_directories()
+        deps.update_progress(
+            automation_id, log_line='Download cleanup completed', log_type='success',
+        )
+        return {'status': 'completed'}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise
+        return {'status': 'error', 'reason': str(e)}
+
+
+# ─── full_cleanup ────────────────────────────────────────────────────
+
+
+def auto_full_cleanup(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run all cleanup tasks: quarantine purge → download queue clear
+    → empty-dir sweep → staging sweep → search history."""
+    automation_id = config.get('_automation_id')
+    steps = []
+
+    # --- 1. Clear quarantine ---
+    deps.update_progress(automation_id, phase='Clearing quarantine...', progress=0)
+    quarantine_path = os.path.join(
+        deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
+        'ss_quarantine',
+    )
+    q_removed = 0
+    if os.path.exists(quarantine_path):
+        for f in os.listdir(quarantine_path):
+            fp = os.path.join(quarantine_path, f)
+            try:
+                if os.path.isfile(fp):
+                    os.remove(fp)
+                    q_removed += 1
+                elif os.path.isdir(fp):
+                    _shutil.rmtree(fp)
+                    q_removed += 1
+            except Exception as e:  # noqa: BLE001 — best-effort purge
+                deps.logger.debug("quarantine entry purge failed: %s", e)
+    steps.append(f'Quarantine: removed {q_removed} items')
+    deps.update_progress(
+        automation_id,
+        log_line=f'Quarantine: removed {q_removed} items',
+        log_type='success' if q_removed else 'info',
+    )
+
+    # --- 2. Clear completed/errored/cancelled downloads from Soulseek queue ---
+    deps.update_progress(automation_id, phase='Clearing download queue...', progress=20)
+    has_active_batches = False
+    has_post_processing = False
+    with deps.tasks_lock:
+        batches = deps.get_download_batches()
+        for batch_data in batches.values():
+            if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
+                has_active_batches = True
+                break
+        if not has_active_batches:
+            tasks = deps.get_download_tasks()
+            for task_data in tasks.values():
+                if task_data.get('status') == 'post_processing':
+                    has_post_processing = True
+                    break
+    if has_active_batches:
+        steps.append('Download queue: skipped (active batches)')
+        deps.update_progress(
+            automation_id,
+            log_line='Download queue: skipped (active batches)',
+            log_type='skip',
+        )
+    else:
+        try:
+            deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
+            steps.append('Download queue: cleared')
+            deps.update_progress(
+                automation_id, log_line='Download queue: cleared', log_type='success',
+            )
+        except Exception as e:  # noqa: BLE001 — per-step best-effort
+            steps.append(f'Download queue: error ({e})')
+            deps.update_progress(
+                automation_id,
+                log_line=f'Download queue: error ({e})',
+                log_type='error',
+            )
+
+    # --- 3. Sweep empty download directories ---
+    deps.update_progress(automation_id, phase='Sweeping empty directories...', progress=40)
+    if has_active_batches or has_post_processing:
+        reason = 'active batches' if has_active_batches else 'post-processing active'
+        steps.append(f'Empty directories: skipped ({reason})')
+        deps.update_progress(
+            automation_id,
+            log_line=f'Empty directories: skipped ({reason})',
+            log_type='skip',
+        )
+    else:
+        dirs_removed = deps.sweep_empty_download_directories()
+        steps.append(f'Empty directories: removed {dirs_removed}')
+        deps.update_progress(
+            automation_id,
+            log_line=f'Empty directories: removed {dirs_removed}',
+            log_type='success' if dirs_removed else 'info',
+        )
+
+    # --- 4. Sweep empty staging directories ---
+    deps.update_progress(automation_id, phase='Sweeping import folder...', progress=60)
+    staging_path = deps.get_staging_path()
+    s_removed = 0
+    if os.path.isdir(staging_path):
+        for dirpath, _dirnames, _filenames in os.walk(staging_path, topdown=False):
+            if os.path.normpath(dirpath) == os.path.normpath(staging_path):
+                continue
+            try:
+                entries = os.listdir(dirpath)
+            except OSError:
+                continue
+            visible = [e for e in entries if not e.startswith('.')]
+            if not visible:
+                for hidden in entries:
+                    try:
+                        os.remove(os.path.join(dirpath, hidden))
+                    except Exception as e:  # noqa: BLE001 — best-effort
+                        deps.logger.debug("hidden file cleanup failed: %s", e)
+                try:
+                    os.rmdir(dirpath)
+                    s_removed += 1
+                except OSError:
+                    pass
+    steps.append(f'Staging: removed {s_removed} empty directories')
+    deps.update_progress(
+        automation_id,
+        log_line=f'Staging: removed {s_removed} empty directories',
+        log_type='success' if s_removed else 'info',
+    )
+
+    # --- 5. Clean search history (if enabled) ---
+    deps.update_progress(automation_id, phase='Cleaning search history...', progress=80)
+    try:
+        if not deps.config_manager.get('soulseek.auto_clear_searches', True):
+            steps.append('Search cleanup: disabled in settings')
+            deps.update_progress(
+                automation_id, log_line='Search cleanup: disabled in settings', log_type='skip',
+            )
+        else:
+            deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
+                keep_searches=50, trigger_threshold=200,
+            ))
+            steps.append('Search history: cleaned')
+            deps.update_progress(
+                automation_id, log_line='Search history: cleaned', log_type='success',
+            )
+    except Exception as e:  # noqa: BLE001 — per-step best-effort
+        steps.append(f'Search history: error ({e})')
+        deps.update_progress(
+            automation_id, log_line=f'Search history: error ({e})', log_type='error',
+        )
+
+    total_removed = q_removed + s_removed
+    deps.update_progress(
+        automation_id, status='finished', progress=100,
+        phase='Complete',
+        log_line=f'Full cleanup complete — {total_removed} items removed',
+        log_type='success',
+    )
+    return {
+        'status': 'completed',
+        'quarantine_removed': str(q_removed),
+        'staging_removed': str(s_removed),
+        'total_removed': str(total_removed),
+        'steps': steps,
+        '_manages_own_progress': True,
+    }

core/automation/handlers/duplicate_cleaner.py

@@ -0,0 +1,87 @@
+"""Automation handler: ``run_duplicate_cleaner`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_run_duplicate_cleaner`` closure). Submits the duplicate
+cleaner to its executor, then polls the shared state dict until
+the worker transitions away from ``running``.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+_TIMEOUT_SECONDS = 7200          # 2 hours
+_POLL_INTERVAL_SECONDS = 3
+_INITIAL_DELAY_SECONDS = 1
+
+
+def auto_run_duplicate_cleaner(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Kick off the duplicate cleaner and report final stats."""
+    automation_id = config.get('_automation_id')
+    state = deps.get_duplicate_cleaner_state()
+    if state.get('status') == 'running':
+        return {'status': 'skipped', 'reason': 'Duplicate cleaner already running'}
+
+    # Pre-set status before submit so the polling loop doesn't see a
+    # stale 'finished' from a previous run.
+    with deps.duplicate_cleaner_lock:
+        state['status'] = 'running'
+    deps.duplicate_cleaner_executor.submit(deps.run_duplicate_cleaner)
+    deps.update_progress(automation_id, log_line='Duplicate cleaner started', log_type='info')
+
+    # Monitor progress (max 2 hours).
+    time.sleep(_INITIAL_DELAY_SECONDS)
+    poll_start = time.time()
+    while time.time() - poll_start < _TIMEOUT_SECONDS:
+        time.sleep(_POLL_INTERVAL_SECONDS)
+        current_status = state.get('status', 'idle')
+        if current_status not in ('running',):
+            break
+        deps.update_progress(
+            automation_id,
+            phase=state.get('phase', 'Scanning...'),
+            progress=state.get('progress', 0),
+            processed=state.get('files_scanned', 0),
+            total=state.get('total_files', 0),
+        )
+    else:
+        # 2-hour timeout reached.
+        deps.update_progress(
+            automation_id, status='error',
+            phase='Timed out',
+            log_line='Duplicate cleaner timed out after 2 hours',
+            log_type='error',
+        )
+        return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
+
+    # Check actual exit status (could be 'finished' or 'error').
+    final_status = state.get('status', 'idle')
+    if final_status == 'error':
+        err = state.get('error_message', 'Unknown error')
+        deps.update_progress(
+            automation_id, status='error', progress=100,
+            phase='Error', log_line=err, log_type='error',
+        )
+        return {'status': 'error', 'reason': err, '_manages_own_progress': True}
+
+    dupes = state.get('duplicates_found', 0)
+    removed = state.get('deleted', 0)
+    space_freed = state.get('space_freed', 0)
+    scanned = state.get('files_scanned', 0)
+    deps.update_progress(
+        automation_id, status='finished', progress=100,
+        phase='Complete',
+        log_line=f'Found {dupes} duplicates, removed {removed} files',
+        log_type='success',
+    )
+    return {
+        'status': 'completed', '_manages_own_progress': True,
+        'files_scanned': scanned,
+        'duplicates_found': dupes,
+        'files_deleted': removed,
+        'space_freed_mb': round(space_freed / (1024 * 1024), 1),
+    }

core/automation/handlers/maintenance.py

@@ -0,0 +1,221 @@
+"""Automation handlers: short maintenance actions.
+
+Lifted from ``web_server._register_automation_handlers``:
+- ``clear_quarantine``       → :func:`auto_clear_quarantine`
+- ``cleanup_wishlist``       → :func:`auto_cleanup_wishlist`
+- ``update_discovery_pool``  → :func:`auto_update_discovery_pool`
+- ``backup_database``        → :func:`auto_backup_database`
+- ``refresh_beatport_cache`` → :func:`auto_refresh_beatport_cache`
+
+Each is a thin wrapper around an existing service / helper. Grouped
+in one module because every body is short and they share no state
+between them — splitting into per-handler files would just add
+import noise.
+"""
+
+from __future__ import annotations
+
+import glob as _glob
+import os
+import shutil as _shutil
+import sqlite3
+import time
+from datetime import datetime
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+# ─── clear_quarantine ────────────────────────────────────────────────
+
+
+def auto_clear_quarantine(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Purge every file/folder under the configured ss_quarantine path."""
+    automation_id = config.get('_automation_id')
+    quarantine_path = os.path.join(
+        deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
+        'ss_quarantine',
+    )
+    if not os.path.exists(quarantine_path):
+        deps.update_progress(automation_id, log_line='No quarantine folder found', log_type='info')
+        return {'status': 'completed', 'removed': '0'}
+    removed = 0
+    for f in os.listdir(quarantine_path):
+        fp = os.path.join(quarantine_path, f)
+        try:
+            if os.path.isfile(fp):
+                os.remove(fp)
+                removed += 1
+            elif os.path.isdir(fp):
+                _shutil.rmtree(fp)
+                removed += 1
+        except Exception as e:  # noqa: BLE001 — best-effort purge
+            deps.logger.debug("quarantine entry purge failed: %s", e)
+    deps.update_progress(
+        automation_id,
+        log_line=f'Removed {removed} quarantined items',
+        log_type='success' if removed > 0 else 'info',
+    )
+    return {'status': 'completed', 'removed': str(removed)}
+
+
+# ─── cleanup_wishlist ────────────────────────────────────────────────
+
+
+def auto_cleanup_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Drop duplicate entries from the wishlist for the active profile."""
+    automation_id = config.get('_automation_id')
+    db = deps.get_database()
+    removed = db.remove_wishlist_duplicates(deps.get_current_profile_id())
+    deps.update_progress(
+        automation_id,
+        log_line=f'Removed {removed or 0} duplicate wishlist entries',
+        log_type='success' if removed else 'info',
+    )
+    return {'status': 'completed', 'removed': str(removed or 0)}
+
+
+# ─── update_discovery_pool ───────────────────────────────────────────
+
+
+def auto_update_discovery_pool(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run an incremental refresh of the discovery pool via the
+    watchlist scanner."""
+    automation_id = config.get('_automation_id')
+    try:
+        scanner = deps.get_watchlist_scanner(deps.spotify_client)
+        deps.update_progress(automation_id, log_line='Updating discovery pool...', log_type='info')
+        scanner.update_discovery_pool_incremental(deps.get_current_profile_id())
+        deps.update_progress(
+            automation_id, status='finished', progress=100,
+            phase='Complete', log_line='Discovery pool updated', log_type='success',
+        )
+        return {'status': 'completed', '_manages_own_progress': True}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise
+        deps.update_progress(
+            automation_id, status='error',
+            phase='Error', log_line=str(e), log_type='error',
+        )
+        return {'status': 'error', 'reason': str(e), '_manages_own_progress': True}
+
+
+# ─── backup_database ─────────────────────────────────────────────────
+
+
+_MAX_BACKUPS = 5
+
+
+def auto_backup_database(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Create a hot SQLite backup, then prune old backups so only the
+    newest ``_MAX_BACKUPS`` remain."""
+    automation_id = config.get('_automation_id')
+    db_path = os.environ.get('DATABASE_PATH', 'database/music_library.db')
+    if not os.path.exists(db_path):
+        return {'status': 'error', 'reason': 'Database file not found'}
+
+    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+    backup_path = f"{db_path}.backup_{timestamp}"
+    # safe_backup verifies source + result integrity, so an automated backup
+    # can never silently snapshot a corrupt DB (the incident where every
+    # rolling backup faithfully copied the corruption).
+    from core.db_integrity import DBIntegrityError, safe_backup, prune_backups
+    try:
+        safe_backup(db_path, backup_path)
+    except DBIntegrityError as integ:
+        deps.logger.error("Auto-backup refused — DB integrity check failed: %s", integ)
+        deps.update_progress(
+            automation_id,
+            log_line=f'Backup SKIPPED — database failed integrity check: {integ}',
+            log_type='error',
+        )
+        return {'status': 'error', 'reason': f'Database integrity check failed: {integ}'}
+    size_mb = round(os.path.getsize(backup_path) / (1024 * 1024), 1)
+
+    # Rolling cleanup — never evict the most-recent verified-healthy backup.
+    existing = list(_glob.glob(f"{db_path}.backup_*"))
+    for removed in prune_backups(existing, _MAX_BACKUPS):
+        try:
+            os.remove(removed)
+        except Exception as e:  # noqa: BLE001 — best-effort cleanup
+            deps.logger.debug("rolling backup cleanup failed: %s", e)
+    deps.update_progress(
+        automation_id,
+        log_line=f'Backup created: {size_mb}MB ({os.path.basename(backup_path)})',
+        log_type='success',
+    )
+    return {'status': 'completed', 'backup_path': backup_path, 'size_mb': str(size_mb)}
+
+
+# ─── refresh_beatport_cache ──────────────────────────────────────────
+
+
+_BEATPORT_SECTIONS = (
+    ('hero_tracks', '/api/beatport/hero-tracks', 'Hero Tracks'),
+    ('new_releases', '/api/beatport/new-releases', 'New Releases'),
+    ('featured_charts', '/api/beatport/featured-charts', 'Featured Charts'),
+    ('dj_charts', '/api/beatport/dj-charts', 'DJ Charts'),
+    ('top_10_lists', '/api/beatport/homepage/top-10-lists', 'Top 10 Lists'),
+    ('top_10_releases', '/api/beatport/homepage/top-10-releases-cards', 'Top 10 Releases'),
+    ('hype_picks', '/api/beatport/hype-picks', 'Hype Picks'),
+)
+
+
+def auto_refresh_beatport_cache(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Refresh Beatport homepage cache by calling each endpoint internally
+    via Flask's ``test_client``. Invalidates the homepage cache first
+    so endpoints re-scrape rather than returning stale data."""
+    automation_id = config.get('_automation_id')
+    cache = deps.get_beatport_data_cache()
+    # Invalidate all homepage cache timestamps so endpoints re-scrape.
+    with cache['cache_lock']:
+        for key in cache['homepage']:
+            cache['homepage'][key]['timestamp'] = 0
+            cache['homepage'][key]['data'] = None
+
+    refreshed = 0
+    errors = []
+    app = deps.get_app()
+    with app.test_client() as client:
+        for idx, (_, endpoint, label) in enumerate(_BEATPORT_SECTIONS):
+            deps.update_progress(
+                automation_id,
+                progress=(idx / len(_BEATPORT_SECTIONS)) * 100,
+                phase=f'Scraping: {label}',
+                current_item=label,
+            )
+            try:
+                resp = client.get(endpoint)
+                if resp.status_code == 200:
+                    refreshed += 1
+                    deps.update_progress(
+                        automation_id, log_line=f'{label}: cached', log_type='success',
+                    )
+                else:
+                    errors.append(label)
+                    deps.update_progress(
+                        automation_id,
+                        log_line=f'{label}: HTTP {resp.status_code}',
+                        log_type='error',
+                    )
+            except Exception as e:  # noqa: BLE001 — per-section best-effort
+                errors.append(label)
+                deps.update_progress(
+                    automation_id,
+                    log_line=f'{label}: {str(e)}',
+                    log_type='error',
+                )
+            if idx < len(_BEATPORT_SECTIONS) - 1:
+                time.sleep(2)
+
+    deps.update_progress(
+        automation_id, status='finished', progress=100,
+        phase='Complete',
+        log_line=f'Refreshed {refreshed}/{len(_BEATPORT_SECTIONS)} sections',
+        log_type='success',
+    )
+    return {
+        'status': 'completed',
+        'refreshed': str(refreshed),
+        'errors': str(len(errors)),
+        '_manages_own_progress': True,
+    }

core/automation/handlers/personalized_pipeline.py

@@ -0,0 +1,356 @@
+"""Personalized Playlist Pipeline automation handler.
+
+Sibling to ``auto_playlist_pipeline`` (mirrored). Where the mirrored
+pipeline runs REFRESH external sources → DISCOVER metadata → SYNC →
+WISHLIST, the personalized pipeline is simpler:
+
+    SNAPSHOT → SYNC → WISHLIST
+
+SNAPSHOT reads the persisted track list from
+``PersonalizedPlaylistManager``. When ``refresh_first=True`` (config),
+each playlist is refreshed BEFORE syncing — useful when the user
+wants the cron to capture a fresh-each-run view (e.g. "give me a new
+Hidden Gems set every night"). Default is to sync the existing
+snapshot, on the assumption the user / a separate cron has already
+refreshed when they wanted to.
+
+Config schema:
+    {
+        'kinds': [
+            {'kind': 'hidden_gems'},
+            {'kind': 'time_machine', 'variant': '1980s'},
+            {'kind': 'seasonal_mix', 'variant': 'halloween'},
+            ...
+        ],
+        'refresh_first': bool,    # default false
+        'skip_wishlist': bool,    # default false
+    }
+
+Each kind dict has at minimum ``kind``; ``variant`` is required for
+kinds that need it (time_machine, genre_playlist, daily_mix,
+seasonal_mix). Singleton kinds (hidden_gems, discovery_shuffle,
+popular_picks, fresh_tape, archives) ignore variant.
+
+Pipeline-running flag (``deps.state.pipeline_running``) is shared
+with the mirrored pipeline so the two can't overlap. (One sync
+queue, one wishlist worker — overlapping triggers would step on
+each other.)"""
+
+from __future__ import annotations
+
+import json
+import threading
+import time
+from typing import Any, Dict, List, Optional
+
+from core.automation.deps import AutomationDeps
+from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
+
+
+# Sync state key prefix so personalized syncs don't collide with
+# mirrored ones (`auto_mirror_<id>`).
+_SYNC_ID_PREFIX = 'auto_personalized'
+
+
+def auto_personalized_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run SNAPSHOT → SYNC → WISHLIST for selected personalized playlists."""
+    deps.state.set_pipeline_running(True)
+    automation_id = config.get('_automation_id')
+    pipeline_start = time.time()
+
+    try:
+        kinds_config = config.get('kinds') or []
+        if not isinstance(kinds_config, list) or not kinds_config:
+            deps.state.set_pipeline_running(False)
+            return {
+                'status': 'error',
+                'error': 'No personalized playlist kinds selected',
+            }
+
+        refresh_first = bool(config.get('refresh_first', False))
+        skip_wishlist = bool(config.get('skip_wishlist', False))
+
+        manager = deps.build_personalized_manager()
+
+        deps.update_progress(
+            automation_id,
+            progress=2,
+            phase=f'Personalized pipeline: {len(kinds_config)} playlist(s)',
+            log_line=f'Starting pipeline for {len(kinds_config)} playlist(s)',
+            log_type='info',
+        )
+
+        # ── PHASE 1: SNAPSHOT (optionally refresh) ──────────────────
+        deps.update_progress(
+            automation_id,
+            progress=3,
+            phase='Phase 1/2: Loading snapshots...' if not refresh_first
+                  else 'Phase 1/2: Refreshing snapshots...',
+            log_line='Phase 1: Snapshot' + (' (with refresh)' if refresh_first else ''),
+            log_type='info',
+        )
+
+        profile_id = deps.get_current_profile_id()
+        playload_payloads = _build_payloads_for_kinds(
+            deps, manager, kinds_config, profile_id,
+            automation_id=automation_id,
+            refresh_first=refresh_first,
+        )
+
+        if not playload_payloads:
+            deps.state.set_pipeline_running(False)
+            deps.update_progress(
+                automation_id,
+                status='finished', progress=100,
+                phase='No playlists to sync',
+                log_line='No personalized playlists had tracks to sync',
+                log_type='warning',
+            )
+            return {
+                'status': 'completed',
+                '_manages_own_progress': True,
+                'playlists_synced': '0',
+                'tracks_synced': '0',
+                'duration_seconds': str(int(time.time() - pipeline_start)),
+            }
+
+        deps.update_progress(
+            automation_id,
+            progress=50,
+            phase='Phase 1/2: Snapshot complete',
+            log_line=f'Phase 1 done: {len(playload_payloads)} playlist(s) ready to sync',
+            log_type='success',
+        )
+
+        # ── PHASE 2: SYNC + WISHLIST (shared helper) ────────────────
+        sync_summary = run_sync_and_wishlist(
+            deps,
+            automation_id,
+            playload_payloads,
+            sync_one_fn=lambda pl: _sync_personalized_playlist(deps, pl),
+            sync_id_for_fn=lambda pl: pl['sync_id'],
+            skip_wishlist=skip_wishlist,
+            progress_start=51,
+            progress_end=90,
+            sync_phase_label='Phase 2/2: Syncing to server...',
+            sync_phase_start_log='Phase 2: Sync',
+            wishlist_phase_label='Phase 2/2: Processing wishlist...',
+            wishlist_phase_start_log='Wishlist',
+        )
+
+        # ── COMPLETE ────────────────────────────────────────────────
+        duration = int(time.time() - pipeline_start)
+        deps.update_progress(
+            automation_id,
+            status='finished', progress=100,
+            phase='Pipeline complete',
+            log_line=f'Personalized pipeline finished in {duration // 60}m {duration % 60}s',
+            log_type='success',
+        )
+
+        deps.state.set_pipeline_running(False)
+        return {
+            'status': 'completed',
+            '_manages_own_progress': True,
+            'playlists_synced': str(len(playload_payloads)),
+            'tracks_synced': str(sync_summary['synced']),
+            'sync_skipped': str(sync_summary['skipped']),
+            'wishlist_queued': str(sync_summary['wishlist_queued']),
+            'duration_seconds': str(duration),
+        }
+
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise into engine
+        deps.state.set_pipeline_running(False)
+        deps.update_progress(
+            automation_id,
+            status='error', progress=100,
+            phase='Pipeline error',
+            log_line=f'Personalized pipeline failed: {e}',
+            log_type='error',
+        )
+        return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
+
+
+def _build_payloads_for_kinds(
+    deps: AutomationDeps,
+    manager: Any,
+    kinds_config: List[Dict[str, Any]],
+    profile_id: int,
+    *,
+    automation_id: Optional[str],
+    refresh_first: bool,
+) -> List[Dict[str, Any]]:
+    """Resolve each requested kind+variant into a sync-payload dict.
+
+    Each payload has: ``{'name', 'kind', 'variant', 'tracks_json',
+    'image_url', 'sync_id'}``. Playlists with no tracks (e.g. a
+    seasonal mix that hasn't been populated yet) are omitted from
+    the result so the sync loop doesn't waste time on empty pushes.
+    """
+    payloads: List[Dict[str, Any]] = []
+    for entry in kinds_config:
+        if not isinstance(entry, dict):
+            continue
+        kind = entry.get('kind')
+        variant = entry.get('variant') or ''
+        if not kind:
+            continue
+
+        try:
+            # Refresh when ANY of:
+            #   - explicit user flag (cron use case: regenerate each run)
+            #   - snapshot marked stale by upstream data refresher
+            #   - playlist was never generated yet (auto-created by
+            #     ensure_playlist; track_count=0, last_generated_at=NULL).
+            #     Without this branch, a first-run pipeline reads the
+            #     empty snapshot and silently skips — user picks a kind,
+            #     hits run, gets "No tracks to sync" with no clue why.
+            if refresh_first:
+                record = manager.refresh_playlist(kind, variant, profile_id)
+            else:
+                existing = manager.ensure_playlist(kind, variant, profile_id)
+                needs_first_gen = existing.last_generated_at is None
+                if existing.is_stale or needs_first_gen:
+                    record = manager.refresh_playlist(kind, variant, profile_id)
+                else:
+                    record = existing
+        except Exception as exc:  # noqa: BLE001 — log + continue with next kind
+            deps.update_progress(
+                automation_id,
+                log_line=f'Skipping {kind}{("/" + variant) if variant else ""}: {exc}',
+                log_type='warning',
+            )
+            continue
+
+        tracks = manager.get_playlist_tracks(record.id)
+        if not tracks:
+            deps.update_progress(
+                automation_id,
+                log_line=f'No tracks in {record.name} — skipping sync',
+                log_type='skip',
+            )
+            continue
+
+        tracks_json = [_track_to_sync_shape(t) for t in tracks]
+        payloads.append({
+            'name': record.name,
+            'kind': record.kind,
+            'variant': record.variant,
+            'tracks_json': tracks_json,
+            'image_url': '',  # personalized playlists don't have a cover image yet
+            'sync_id': f'{_SYNC_ID_PREFIX}_{record.kind}_{record.variant or "_"}',
+        })
+    return payloads
+
+
+def _track_to_sync_shape(track: Any) -> Dict[str, Any]:
+    """Convert a personalized.types.Track into the dict shape
+    `_run_sync_task` expects. Mirrors what the mirrored pipeline
+    builds from extra_data.matched_data, preserving enriched metadata
+    from personalized snapshots when available."""
+    primary_id = track.spotify_track_id or track.itunes_track_id or track.deezer_track_id or ''
+    rich_data = _coerce_track_data_json(getattr(track, 'track_data_json', None))
+
+    if not rich_data:
+        album = {'name': track.album_name or ''}
+        cover_url = getattr(track, 'album_cover_url', None)
+        if cover_url:
+            album['images'] = [{'url': cover_url}]
+        return {
+            'name': track.track_name,
+            'artists': [{'name': track.artist_name}],
+            'album': album,
+            'duration_ms': int(track.duration_ms or 0),
+            'id': primary_id,
+        }
+
+    payload = dict(rich_data)
+    cover_url = (
+        getattr(track, 'album_cover_url', None)
+        or payload.get('album_cover_url')
+        or payload.get('image_url')
+    )
+    payload['id'] = payload.get('id') or primary_id
+    payload['name'] = payload.get('name') or track.track_name
+    payload['artists'] = _normalize_artists(payload.get('artists'), track.artist_name)
+    payload['album'] = _normalize_album(payload.get('album'), track, cover_url=cover_url)
+    payload['duration_ms'] = int(payload.get('duration_ms') or track.duration_ms or 0)
+    if 'popularity' not in payload and getattr(track, 'popularity', None) is not None:
+        payload['popularity'] = int(track.popularity or 0)
+
+    if cover_url and not payload.get('image_url'):
+        payload['image_url'] = cover_url
+
+    return payload
+
+
+def _coerce_track_data_json(value: Any) -> Dict[str, Any]:
+    if isinstance(value, dict):
+        return value
+    if isinstance(value, str) and value.strip():
+        try:
+            loaded = json.loads(value)
+        except (TypeError, ValueError):
+            return {}
+        return loaded if isinstance(loaded, dict) else {}
+    return {}
+
+
+def _normalize_artists(artists: Any, fallback_artist: str) -> List[Dict[str, Any]]:
+    if not artists:
+        return [{'name': fallback_artist or 'Unknown Artist'}]
+    if isinstance(artists, list):
+        normalized = []
+        for artist in artists:
+            if isinstance(artist, dict):
+                normalized.append(artist if artist.get('name') else {'name': str(artist)})
+            elif isinstance(artist, str):
+                normalized.append({'name': artist})
+            else:
+                normalized.append({'name': str(artist)})
+        return normalized or [{'name': fallback_artist or 'Unknown Artist'}]
+    if isinstance(artists, dict):
+        return [artists if artists.get('name') else {'name': str(artists)}]
+    return [{'name': str(artists)}]
+
+
+def _normalize_album(album: Any, track: Any, cover_url: Optional[str] = None) -> Dict[str, Any]:
+    if isinstance(album, dict):
+        normalized = dict(album)
+    else:
+        normalized = {'name': str(album) if album else (track.album_name or '')}
+
+    normalized['name'] = normalized.get('name') or track.album_name or ''
+    images = normalized.get('images')
+    if not isinstance(images, list):
+        images = []
+    if cover_url and not images:
+        images = [{'url': cover_url}]
+    if images:
+        normalized['images'] = images
+    return normalized
+
+
+def _sync_personalized_playlist(deps: AutomationDeps, payload: Dict[str, Any]) -> Dict[str, Any]:
+    """Launch a personalized playlist sync via _run_sync_task on a
+    daemon thread + return immediately with status='started'.
+
+    Mirrors the mirrored ``auto_sync_playlist`` return contract so the
+    shared helper can poll on ``sync_states[sync_id]`` and aggregate
+    results identically."""
+    sync_id = payload['sync_id']
+    name = payload['name']
+    tracks_json = payload['tracks_json']
+    profile_id = deps.get_current_profile_id()
+
+    threading.Thread(
+        target=deps.run_sync_task,
+        args=(sync_id, name, tracks_json, None, profile_id, payload.get('image_url', '')),
+        daemon=True,
+        name=f'auto-personalized-{sync_id}',
+    ).start()
+    return {
+        'status': 'started',
+        'playlist_name': name,
+        '_manages_own_progress': True,
+    }

core/automation/handlers/playlist_pipeline.py

@@ -0,0 +1,27 @@
+"""Automation adapter for the mirrored playlist pipeline.
+
+The actual all-in-one playlist lifecycle lives in
+``core.playlists.pipeline`` so it can be reused by non-automation UI actions.
+This module only wires automation-specific dependencies and handlers.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
+from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
+from core.automation.handlers.sync_playlist import auto_sync_playlist
+from core.playlists.pipeline import run_mirrored_playlist_pipeline
+
+
+def auto_playlist_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run REFRESH -> DISCOVER -> SYNC -> WISHLIST for mirrored playlists."""
+    return run_mirrored_playlist_pipeline(
+        config,
+        deps,
+        refresh_fn=auto_refresh_mirrored,
+        sync_one_fn=auto_sync_playlist,
+        sync_and_wishlist_fn=run_sync_and_wishlist,
+    )

core/automation/handlers/process_wishlist.py

@@ -0,0 +1,27 @@
+"""Automation handler: ``process_wishlist`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_process_wishlist`` closure). Wishlist processing is async —
+the helper submits a batch to an executor and returns immediately;
+per-track stats arrive later via batch-completion callbacks.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_process_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Kick off the wishlist processor for an automation trigger.
+
+    Returns immediately after submission; the wishlist worker emits
+    per-batch progress via its own callbacks. We only report
+    ``status: completed`` to mark the trigger fired successfully.
+    """
+    try:
+        deps.process_wishlist_automatically(automation_id=config.get('_automation_id'))
+        return {'status': 'completed'}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise into the engine
+        return {'status': 'error', 'error': str(e)}

core/automation/handlers/progress_callbacks.py

@@ -0,0 +1,89 @@
+"""Progress + history callbacks the automation engine invokes around
+each handler run.
+
+Lifted from the closures at the bottom of
+``web_server._register_automation_handlers``:
+- ``_progress_init``               → :func:`progress_init`
+- ``_progress_finish``             → :func:`progress_finish`
+- ``_record_automation_history``   → :func:`record_history`
+- ``_on_library_scan_completed``   → :func:`on_library_scan_completed`
+
+The engine accepts four callables via
+``register_progress_callbacks(init, finish, update, history)``;
+``registration.register_all`` wires these here. The
+``library_scan_completed`` callback is registered separately on the
+``web_scan_manager`` (when one is available) -- see
+``register_library_scan_completed_emitter``.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def progress_init(aid: Any, name: str, action_type: str, deps: AutomationDeps) -> None:
+    """Initialize per-automation progress state when the engine starts
+    a handler. Thin wrapper so the engine receives a closure that
+    delegates into the live progress tracker."""
+    deps.init_automation_progress(aid, name, action_type)
+
+
+def progress_finish(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
+    """Emit the final progress update when a handler returns.
+
+    Skipped for handlers that manage their own progress lifecycle
+    (they call ``update_progress(status='finished')`` themselves and
+    set ``_manages_own_progress: True`` in the returned dict).
+    Otherwise translates the handler's status into a finished/error
+    progress emit with a status-appropriate phase + log line.
+    """
+    if result.get('_manages_own_progress'):
+        return
+    result_status = result.get('status', '')
+    status = 'error' if result_status == 'error' else 'finished'
+    msg = result.get('error', result.get('reason', result_status or 'done'))
+    deps.update_progress(
+        aid,
+        status=status,
+        progress=100,
+        phase='Error' if status == 'error' else 'Complete',
+        log_line=msg,
+        log_type='error' if status == 'error' else 'success',
+    )
+
+
+def record_history(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
+    """Capture progress state into run history before the engine's
+    cleanup pass clears it. Thin wrapper so the engine sees a stable
+    callable."""
+    deps.record_progress_history(aid, result, deps.get_database())
+
+
+def on_library_scan_completed(deps: AutomationDeps) -> None:
+    """Emit the ``library_scan_completed`` automation event with the
+    active media-server type. Replaces the hard-coded
+    ``scan_completion_callback → trigger_automatic_database_update``
+    chain so any automation can listen for scan completion as a
+    trigger."""
+    if not deps.engine:
+        return
+    server_type = (
+        getattr(deps.web_scan_manager, '_current_server_type', None)
+        or 'unknown'
+    )
+    deps.engine.emit('library_scan_completed', {
+        'server_type': server_type,
+    })
+
+
+def register_library_scan_completed_emitter(deps: AutomationDeps) -> None:
+    """Wire :func:`on_library_scan_completed` to the
+    ``web_scan_manager``'s scan-completion callback list. No-op when
+    no scan manager is configured (e.g. headless / test contexts)."""
+    if not deps.web_scan_manager:
+        return
+    deps.web_scan_manager.add_scan_completion_callback(
+        lambda: on_library_scan_completed(deps),
+    )

core/automation/handlers/quality_scanner.py

@@ -0,0 +1,35 @@
+"""Automation handler: ``start_quality_scan`` action.
+
+The quality scanner was redesigned from an auto-acting tool into the
+``quality_upgrade`` library-maintenance repair job (findings-based, reviewed
+before anything is wishlisted). This action now simply triggers a "Run Now" of
+that job; its progress and findings surface in Library Maintenance. The action
+name is kept so existing automation rules keep working.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_start_quality_scan(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    automation_id = config.get('_automation_id')
+
+    triggered = deps.run_repair_job_now('quality_upgrade')
+    if not triggered:
+        deps.update_progress(
+            automation_id, status='error', phase='Unavailable',
+            log_line='Quality Upgrade job could not be triggered (library worker unavailable)',
+            log_type='error',
+        )
+        return {'status': 'error', 'reason': 'library worker unavailable',
+                '_manages_own_progress': True}
+
+    deps.update_progress(
+        automation_id, status='finished', progress=100, phase='Triggered',
+        log_line='Quality Upgrade scan queued — findings appear in Library Maintenance',
+        log_type='success',
+    )
+    return {'status': 'completed', 'triggered': True, '_manages_own_progress': True}

core/automation/handlers/refresh_mirrored.py

@@ -0,0 +1,341 @@
+"""Automation handler: ``refresh_mirrored`` action.
+
+Re-pulls track lists from each mirrored playlist's source via the
+unified ``PlaylistSourceRegistry`` (Phase 1 of the Discover-to-Sync
+unification). The pre-extraction handler had ~190 lines of per-source
+if/elif branches; this version delegates to the adapter for each
+source, leaving the handler responsible only for:
+
+- filtering sources that can't be refreshed (``file``, ``beatport``),
+- extracting upstream URLs from the stored ``description`` for URL-
+  backed sources (``spotify_public``, ``youtube``),
+- the Spotify-public → authenticated-Spotify fallback (uses the
+  ``spotify`` adapter when the user is signed in so the mirror keeps
+  album art),
+- the Tidal-not-authenticated skip log type (vs error),
+- preserving existing per-track ``extra_data`` on tracks that survive
+  the refresh, and
+- emitting the ``playlist_changed`` automation event when the track
+  set actually shifts.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from core.automation.deps import AutomationDeps
+from core.playlists.source_refs import require_refresh_url
+from core.playlists.sources import PlaylistDetail, to_mirror_track_dict
+from core.playlists.sources.base import (
+    SOURCE_SPOTIFY,
+    SOURCE_SPOTIFY_PUBLIC,
+    SOURCE_TIDAL,
+    SOURCE_YOUTUBE,
+)
+
+
+# Sources that store the upstream URL in ``description`` (because their
+# ``source_playlist_id`` is a deterministic hash, not the native ID).
+# The refresh path has to recover the URL before calling the adapter.
+_URL_BACKED_SOURCES = {SOURCE_SPOTIFY_PUBLIC, SOURCE_YOUTUBE}
+
+
+def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Refresh mirrored playlist(s) from source.
+
+    Returns ``{'status': 'completed', 'refreshed': '<int>',
+    'errors': '<int>'}`` on success (counts stringified to match the
+    automation engine's stat-rendering convention).
+    """
+    db = deps.get_database()
+    playlist_id = config.get('playlist_id')
+    refresh_all = config.get('all', False)
+    auto_id = config.get('_automation_id')
+    # Pipeline runs (``run_mirrored_playlist_pipeline``) set this flag
+    # because Phase 2 of the pipeline already runs the playlist
+    # discovery worker — the same matching engine ``_maybe_discover``
+    # would call here. Running both means LB tracks discover twice
+    # AND the refresh-side discovery blocks 5+ minutes with no
+    # progress emission, leaving the UI stuck on "Refreshing:" until
+    # the loop returns. Standalone callers (Sync page, registration
+    # action) leave it False so LB tracks still get matched_data on
+    # refresh.
+    skip_discovery = bool(config.get('skip_discovery', False))
+
+    if refresh_all:
+        playlists = db.get_mirrored_playlists()
+    elif playlist_id:
+        p = db.get_mirrored_playlist(int(playlist_id))
+        playlists = [p] if p else []
+    else:
+        return {'status': 'error', 'reason': 'No playlist specified'}
+
+    # Filter out sources that can't be refreshed (no external API).
+    playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
+
+    refreshed = 0
+    errors: List[str] = []
+    for idx, pl in enumerate(playlists):
+        try:
+            source = pl.get('source', '')
+            source_id = pl.get('source_playlist_id', '')
+            deps.update_progress(
+                auto_id,
+                progress=(idx / max(1, len(playlists))) * 100,
+                phase=f'Refreshing: "{pl.get("name", "")}"',
+                current_item=pl.get('name', ''),
+            )
+
+            detail = _fetch_detail(source, source_id, pl, deps, auto_id)
+            if detail is None:
+                # _fetch_detail already logged the specific failure;
+                # mark the playlist as a generic refresh error so the
+                # automation result tally matches the legacy handler.
+                errors.append(f"{pl.get('name', '?')}: no tracks returned from source")
+                deps.update_progress(
+                    auto_id,
+                    log_line=f'Refresh failed: "{pl.get("name", "")}" - no tracks returned from source',
+                    log_type='error',
+                )
+                continue
+
+            # Sources that return MB-metadata-only tracks (LB, Last.fm)
+            # mark them ``needs_discovery=True``. Hand them to the
+            # adapter's matcher so the resulting mirror rows carry
+            # provider IDs + matched_data, ready for the sync pipeline.
+            #
+            # Pipeline runs skip this because Phase 2's discovery
+            # worker handles it with proper progress emission — see
+            # ``skip_discovery`` resolution at the top of this fn.
+            detail_tracks = (
+                detail.tracks if skip_discovery
+                else _maybe_discover(detail.tracks, source, deps)
+            )
+
+            tracks = [to_mirror_track_dict(t) for t in detail_tracks]
+            refreshed += _commit_refresh(pl, source, source_id, tracks, db, deps, auto_id)
+        except _SkipPlaylist:
+            # Source-specific soft-skip (e.g. Tidal not authenticated).
+            # Logging was already emitted; do not count as error.
+            continue
+        except Exception as e:
+            errors.append(f"{pl.get('name', '?')}: {str(e)}")
+            deps.update_progress(
+                auto_id,
+                log_line=f'Error: {pl.get("name", "?")} — {str(e)}',
+                log_type='error',
+            )
+    return {'status': 'completed', 'refreshed': str(refreshed), 'errors': str(len(errors))}
+
+
+def _maybe_discover(
+    tracks: List[Any],
+    source: str,
+    deps: AutomationDeps,
+) -> List[Any]:
+    """Run the adapter's ``discover_tracks`` when any track needs it.
+
+    Most sources are no-ops here (their tracks have provider IDs
+    already). LB / Last.fm are the ones that actually do work."""
+    if not tracks:
+        return tracks
+    if not any(getattr(t, "needs_discovery", False) for t in tracks):
+        return tracks
+    registry = deps.playlist_source_registry
+    if registry is None:
+        return tracks
+    adapter = registry.get_source(source)
+    if adapter is None:
+        return tracks
+    try:
+        return adapter.discover_tracks(tracks)
+    except Exception as exc:
+        deps.logger.warning(f"{source} discover_tracks failed: {exc}")
+        return tracks
+
+
+class _SkipPlaylist(Exception):
+    """Internal sentinel: source-specific soft-skip (e.g. not authed).
+
+    The per-playlist loop catches it specifically so the skip isn't
+    counted in the error tally — matches the pre-extraction behavior
+    where ``continue`` was used inline."""
+
+
+def _fetch_detail(
+    source: str,
+    source_id: str,
+    pl: Dict[str, Any],
+    deps: AutomationDeps,
+    auto_id: Optional[str],
+) -> Optional[PlaylistDetail]:
+    """Resolve the playlist's tracks through the registry.
+
+    Handler-level branches (URL extraction, Spotify-public→authed
+    fallback, Tidal not-authed skip) live here; everything else
+    delegates to the adapter."""
+    registry = deps.playlist_source_registry
+    if registry is None:
+        return None
+
+    # URL-backed sources: pull the upstream URL out of `description`.
+    playlist_input = source_id
+    if source in _URL_BACKED_SOURCES:
+        # ``require_refresh_url`` raises ValueError on missing URL.
+        # The outer try/except in the loop catches it and reports as
+        # an error — matching the pre-extraction behavior.
+        playlist_input = require_refresh_url(
+            source, pl.get('description', ''), pl.get('name', '')
+        )
+
+    # Spotify-public refresh: prefer the authenticated Spotify API
+    # when the user is signed in. Better album art, matches the
+    # pre-extraction handler. Falls through to the public scraper on
+    # auth failure or non-playlist URL types (e.g. album URLs).
+    if source == SOURCE_SPOTIFY_PUBLIC:
+        detail = _try_spotify_authed_for_public(playlist_input, deps)
+        if detail is not None:
+            return detail
+
+    # Tidal not-authed: soft-skip with a 'skip' log line, not an error.
+    if source == SOURCE_TIDAL:
+        tidal_source = registry.get_source(SOURCE_TIDAL)
+        if tidal_source is None or not tidal_source.is_authenticated():
+            deps.logger.warning(
+                f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'"
+            )
+            deps.update_progress(
+                auto_id,
+                log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
+                log_type='skip',
+            )
+            raise _SkipPlaylist
+
+    adapter = registry.get_source(source)
+    if adapter is None:
+        return None
+    try:
+        return adapter.refresh_playlist(playlist_input)
+    except Exception as exc:
+        deps.logger.warning(
+            f"{source} playlist refresh failed for {playlist_input}: {exc}"
+        )
+        return None
+
+
+def _try_spotify_authed_for_public(
+    spotify_url: str, deps: AutomationDeps
+) -> Optional[PlaylistDetail]:
+    """Best-effort: use the authenticated Spotify adapter on a public URL.
+
+    Returns ``None`` to signal "fall through to the public-scraper
+    adapter" — never raises. Only applies to ``playlist``-type URLs;
+    album URLs fall through unconditionally."""
+    if not spotify_url:
+        return None
+    spotify_client = deps.spotify_client
+    if spotify_client is None or not spotify_client.is_spotify_authenticated():
+        return None
+    try:
+        from core.spotify_public_scraper import parse_spotify_url
+        parsed = parse_spotify_url(spotify_url)
+    except Exception:
+        return None
+    if not parsed or parsed.get('type') != 'playlist':
+        return None
+    adapter = deps.playlist_source_registry.get_source(SOURCE_SPOTIFY)
+    if adapter is None:
+        return None
+    try:
+        return adapter.refresh_playlist(parsed['id'])
+    except Exception as exc:
+        deps.logger.debug(f"Spotify authed fallback for public mirror failed: {exc}")
+        return None
+
+
+def _commit_refresh(
+    pl: Dict[str, Any],
+    source: str,
+    source_id: str,
+    tracks: List[Dict[str, Any]],
+    db: Any,
+    deps: AutomationDeps,
+    auto_id: Optional[str],
+) -> int:
+    """Persist the refreshed track list + emit playlist_changed when delta.
+
+    Returns 1 when a refresh successfully landed, 0 otherwise. The
+    caller is responsible for incrementing the running tally."""
+    old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
+    old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
+    new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
+
+    # Preserve existing extra_data (matched_data + discovery state)
+    # for tracks that still exist in the refreshed snapshot, unless
+    # the adapter already provided fresh extra_data for that track.
+    old_extra_map = (
+        db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
+    )
+    for t in tracks:
+        sid = t.get('source_track_id', '')
+        if sid and sid in old_extra_map and 'extra_data' not in t:
+            t['extra_data'] = old_extra_map[sid]
+
+    db.mirror_playlist(
+        source=source,
+        source_playlist_id=source_id,
+        name=pl['name'],
+        tracks=tracks,
+        profile_id=pl.get('profile_id', 1),
+        description=pl.get('description'),
+        owner=pl.get('owner'),
+        image_url=pl.get('image_url'),
+    )
+
+    # Membership just changed — if this playlist is organize-by-playlist, rebuild
+    # its folder (with prune) so a track that LEFT the playlist has its symlink
+    # cleaned up now. Gated to organized playlists, non-fatal — never disturbs
+    # the refresh. (Additions are handled by the post-download reconcile.)
+    try:
+        from core.playlists.materialize_service import rebuild_mirrored_playlist_if_organized
+        rebuild_mirrored_playlist_if_organized(
+            db, deps.config_manager, pl.get('id'), profile_id=pl.get('profile_id', 1)
+        )
+    except Exception as _mat_err:
+        deps.logger.debug(f"[Playlist Folder] mirror-refresh cleanup skipped: {_mat_err}")
+
+    if old_ids != new_ids:
+        added = len(new_ids - old_ids)
+        removed = len(old_ids - new_ids)
+        deps.logger.info(
+            f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
+            f"{added} added, {removed} removed (old={len(old_ids)}, new={len(new_ids)})"
+        )
+        deps.update_progress(
+            auto_id,
+            log_line=f'"{pl.get("name", "")}" — {added} added, {removed} removed',
+            log_type='success',
+        )
+        try:
+            if deps.engine:
+                deps.engine.emit('playlist_changed', {
+                    'playlist_name': pl.get('name', ''),
+                    'playlist_id': str(pl.get('id', '')),
+                    'old_count': str(len(old_ids)),
+                    'new_count': str(len(new_ids)),
+                    'added': str(added),
+                    'removed': str(removed),
+                })
+        except Exception as e:
+            deps.logger.debug("playlist_synced automation emit failed: %s", e)
+    else:
+        deps.logger.warning(
+            f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})"
+        )
+        deps.update_progress(
+            auto_id,
+            log_line=f'No changes: "{pl.get("name", "")}"',
+            log_type='skip',
+        )
+
+    return 1

core/automation/handlers/registration.py

@@ -0,0 +1,184 @@
+"""One-stop registration of every extracted automation handler.
+
+``web_server`` builds the deps once at startup and calls
+:func:`register_all` here. Each new handler module gets one line in
+this file when it lands.
+"""
+
+from __future__ import annotations
+
+from core.automation.deps import AutomationDeps
+from core.automation.handlers.process_wishlist import auto_process_wishlist
+from core.automation.handlers.scan_watchlist import auto_scan_watchlist
+from core.automation.handlers.scan_library import auto_scan_library
+from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
+from core.automation.handlers.sync_playlist import auto_sync_playlist
+from core.automation.handlers.discover_playlist import auto_discover_playlist
+from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
+from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
+from core.automation.handlers.database_update import (
+    auto_start_database_update, auto_deep_scan_library,
+)
+from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
+from core.automation.handlers.quality_scanner import auto_start_quality_scan
+from core.automation.handlers.maintenance import (
+    auto_clear_quarantine,
+    auto_cleanup_wishlist,
+    auto_update_discovery_pool,
+    auto_backup_database,
+    auto_refresh_beatport_cache,
+)
+from core.automation.handlers.download_cleanup import (
+    auto_clean_search_history,
+    auto_clean_completed_downloads,
+    auto_full_cleanup,
+)
+from core.automation.handlers.run_script import auto_run_script
+from core.automation.handlers.search_and_download import auto_search_and_download
+from core.automation.handlers.progress_callbacks import (
+    progress_init,
+    progress_finish,
+    record_history,
+    register_library_scan_completed_emitter,
+)
+
+
+def register_all(deps: AutomationDeps) -> None:
+    """Wire every extracted handler to the engine.
+
+    Each ``register_action_handler`` call binds the action name (the
+    string the trigger uses to look up its action) to a thin lambda
+    that injects ``deps`` and forwards the engine-supplied config.
+    Guards stay alongside their handler so duplicate-run prevention
+    behaves identically to the pre-extraction code.
+    """
+    engine = deps.engine
+
+    # Self-guards prevent duplicate runs of the SAME operation, but
+    # different operations can run concurrently — wishlist downloads
+    # use bandwidth, watchlist scans use API calls, library scans use
+    # media-server CPU. Different resources, no contention.
+    engine.register_action_handler(
+        'process_wishlist',
+        lambda config: auto_process_wishlist(config, deps),
+        guard_fn=deps.is_wishlist_actually_processing,
+    )
+    engine.register_action_handler(
+        'scan_watchlist',
+        lambda config: auto_scan_watchlist(config, deps),
+        guard_fn=deps.is_watchlist_actually_scanning,
+    )
+    engine.register_action_handler(
+        'scan_library',
+        lambda config: auto_scan_library(config, deps),
+        deps.state.is_scan_library_active,
+    )
+
+    # Playlist lifecycle handlers. The pipeline composes refresh +
+    # sync + discover (it imports them directly), so all four ship
+    # together. The pipeline guard prevents an in-flight pipeline
+    # from being re-triggered mid-run.
+    engine.register_action_handler(
+        'refresh_mirrored',
+        lambda config: auto_refresh_mirrored(config, deps),
+    )
+    engine.register_action_handler(
+        'sync_playlist',
+        lambda config: auto_sync_playlist(config, deps),
+    )
+    engine.register_action_handler(
+        'discover_playlist',
+        lambda config: auto_discover_playlist(config, deps),
+    )
+    engine.register_action_handler(
+        'playlist_pipeline',
+        lambda config: auto_playlist_pipeline(config, deps),
+        deps.state.is_pipeline_running,
+    )
+    # Personalized pipeline shares the pipeline_running flag with the
+    # mirrored pipeline so the two can't overlap (single sync queue,
+    # single wishlist worker).
+    engine.register_action_handler(
+        'personalized_pipeline',
+        lambda config: auto_personalized_pipeline(config, deps),
+        deps.state.is_pipeline_running,
+    )
+
+    # Database update + deep scan share the db_update_state guard —
+    # only one operation can mutate that state at a time.
+    engine.register_action_handler(
+        'start_database_update',
+        lambda config: auto_start_database_update(config, deps),
+        lambda: deps.get_db_update_state().get('status') == 'running',
+    )
+    engine.register_action_handler(
+        'deep_scan_library',
+        lambda config: auto_deep_scan_library(config, deps),
+        lambda: deps.get_db_update_state().get('status') == 'running',
+    )
+    engine.register_action_handler(
+        'run_duplicate_cleaner',
+        lambda config: auto_run_duplicate_cleaner(config, deps),
+        lambda: deps.get_duplicate_cleaner_state().get('status') == 'running',
+    )
+    engine.register_action_handler(
+        'clear_quarantine',
+        lambda config: auto_clear_quarantine(config, deps),
+    )
+    engine.register_action_handler(
+        'cleanup_wishlist',
+        lambda config: auto_cleanup_wishlist(config, deps),
+    )
+    engine.register_action_handler(
+        'update_discovery_pool',
+        lambda config: auto_update_discovery_pool(config, deps),
+    )
+    engine.register_action_handler(
+        'start_quality_scan',
+        lambda config: auto_start_quality_scan(config, deps),
+        lambda: False,  # repair worker dedupes Run-Now requests itself
+    )
+    engine.register_action_handler(
+        'backup_database',
+        lambda config: auto_backup_database(config, deps),
+    )
+    engine.register_action_handler(
+        'refresh_beatport_cache',
+        lambda config: auto_refresh_beatport_cache(config, deps),
+    )
+    engine.register_action_handler(
+        'clean_search_history',
+        lambda config: auto_clean_search_history(config, deps),
+    )
+    engine.register_action_handler(
+        'clean_completed_downloads',
+        lambda config: auto_clean_completed_downloads(config, deps),
+    )
+    engine.register_action_handler(
+        'full_cleanup',
+        lambda config: auto_full_cleanup(config, deps),
+    )
+    engine.register_action_handler(
+        'run_script',
+        lambda config: auto_run_script(config, deps),
+    )
+    engine.register_action_handler(
+        'search_and_download',
+        lambda config: auto_search_and_download(config, deps),
+    )
+
+    # Progress + history callbacks: the engine invokes these around
+    # each handler run. Lift the closures from
+    # `web_server._register_automation_handlers` into thin lambdas
+    # that delegate into the extracted top-level functions.
+    engine.register_progress_callbacks(
+        lambda aid, name, action_type: progress_init(aid, name, action_type, deps),
+        lambda aid, result: progress_finish(aid, result, deps),
+        deps.update_progress,
+        lambda aid, result: record_history(aid, result, deps),
+    )
+
+    # `library_scan_completed` event: when the media-server scan
+    # manager finishes a scan, emit the event so any automation can
+    # trigger off it. No-op when no scan manager is configured.
+    register_library_scan_completed_emitter(deps)

core/automation/handlers/run_script.py

@@ -0,0 +1,103 @@
+"""Automation handler: ``run_script`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_run_script`` closure). Runs a user-provided shell or Python
+script from the configured scripts directory with bounded timeout +
+captured stdout/stderr. Path-traversal guard ensures users can't
+escape the scripts directory.
+
+Environment variables exposed to the script:
+- ``SOULSYNC_EVENT``: triggering event type (when fired by an event)
+- ``SOULSYNC_AUTOMATION``: automation name
+- ``SOULSYNC_SCRIPTS_DIR``: absolute path to the scripts dir
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess as _sp
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+_MAX_TIMEOUT_SECONDS = 300  # Hard cap on user-supplied timeout config.
+
+
+def auto_run_script(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    script_name = config.get('script_name', '')
+    timeout = min(int(config.get('timeout', 60)), _MAX_TIMEOUT_SECONDS)
+    automation_id = config.get('_automation_id')
+
+    if not script_name:
+        return {'status': 'error', 'error': 'No script selected'}
+
+    scripts_dir = deps.docker_resolve_path(deps.config_manager.get('scripts.path', './scripts'))
+    if not scripts_dir or not os.path.isdir(scripts_dir):
+        os.makedirs(scripts_dir, exist_ok=True)
+        return {
+            'status': 'error',
+            'error': 'Scripts directory is empty. Add scripts to the scripts/ folder.',
+        }
+
+    script_path = os.path.join(scripts_dir, script_name)
+    script_path = os.path.realpath(script_path)
+
+    # Security: block path traversal — script must resolve under
+    # the scripts dir, no symlinks/.. tricks allowed out.
+    if not script_path.startswith(os.path.realpath(scripts_dir)):
+        return {'status': 'error', 'error': 'Script path traversal blocked'}
+
+    if not os.path.isfile(script_path):
+        return {'status': 'error', 'error': f'Script not found: {script_name}'}
+
+    deps.update_progress(automation_id, phase=f'Running {script_name}...', progress=10)
+
+    # Build environment with SoulSync context.
+    env = os.environ.copy()
+    event_data = config.get('_event_data') or {}
+    env['SOULSYNC_EVENT'] = str(event_data.get('type', ''))
+    env['SOULSYNC_AUTOMATION'] = config.get('_automation_name', '')
+    env['SOULSYNC_SCRIPTS_DIR'] = scripts_dir
+
+    try:
+        # Determine how to run the script.
+        if script_path.endswith('.py'):
+            cmd = ['python', script_path]
+        elif script_path.endswith('.sh'):
+            cmd = ['bash', script_path]
+        else:
+            cmd = [script_path]
+
+        result = _sp.run(
+            cmd,
+            capture_output=True, text=True, timeout=timeout,
+            cwd=scripts_dir, env=env,
+        )
+
+        deps.update_progress(automation_id, phase='Script completed', progress=100)
+
+        stdout = result.stdout[:2000] if result.stdout else ''
+        stderr = result.stderr[:1000] if result.stderr else ''
+
+        if result.returncode == 0:
+            deps.logger.info(f"Script '{script_name}' completed (exit 0)")
+        else:
+            deps.logger.warning(f"Script '{script_name}' exited with code {result.returncode}")
+
+        return {
+            'status': 'completed' if result.returncode == 0 else 'error',
+            'exit_code': str(result.returncode),
+            'stdout': stdout,
+            'stderr': stderr,
+            'script': script_name,
+        }
+    except _sp.TimeoutExpired:
+        deps.update_progress(automation_id, phase='Script timed out', progress=100)
+        return {
+            'status': 'error',
+            'error': f'Script timed out after {timeout}s',
+            'script': script_name,
+        }
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise
+        return {'status': 'error', 'error': str(e), 'script': script_name}

core/automation/handlers/scan_library.py

@@ -0,0 +1,158 @@
+"""Automation handler: ``scan_library`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_scan_library`` closure). The handler triggers a media-server
+scan via ``web_scan_manager``, then polls the manager's status until
+the scan completes (or a 30-minute timeout fires). Progress phases
+are emitted via :func:`AutomationDeps.update_progress` so the
+trigger card stays current throughout the run.
+
+The handler manages its own progress reporting (it sets
+``_manages_own_progress: True`` in the result) so the engine doesn't
+overwrite the live phase string with a generic 'completed' label.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+# Outer poll cap — covers extreme worst case (long Plex scans on
+# huge libraries). Past this point we surface a clear timeout error
+# so users notice rather than letting the trigger hang forever.
+_SCAN_TIMEOUT_SECONDS = 1800
+
+# Per-phase poll intervals.
+_POLL_SCHEDULED_SECONDS = 2
+_POLL_SCANNING_SECONDS = 5
+_POLL_UNKNOWN_SECONDS = 2
+
+# Progress percentage waypoints.
+_PROGRESS_SCHEDULED_MAX = 14
+_PROGRESS_SCAN_START = 15
+_PROGRESS_SCAN_MAX = 95
+
+
+def auto_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run a media-server library scan and stream progress to the
+    trigger card.
+
+    Returns one of:
+      - ``{'status': 'completed', '_manages_own_progress': True, ...}``
+      - ``{'status': 'skipped', 'reason': 'Scan already being tracked'}``
+      - ``{'status': 'error', 'reason': '...', '_manages_own_progress': True}``
+    """
+    automation_id = config.get('_automation_id')
+
+    if not deps.web_scan_manager:
+        return {'status': 'error', 'reason': 'Scan manager not available'}
+
+    # If another automation is already tracking the scan, just forward
+    # the request — the original tracker keeps emitting progress.
+    if deps.state.is_scan_library_active():
+        deps.web_scan_manager.request_scan('Automation trigger (additional batch)')
+        return {'status': 'skipped', 'reason': 'Scan already being tracked'}
+
+    deps.state.set_scan_library_id(automation_id)
+
+    try:
+        result = deps.web_scan_manager.request_scan('Automation trigger')
+        scan_status_val = result.get('status', 'unknown')
+
+        if scan_status_val == 'queued':
+            deps.update_progress(
+                automation_id,
+                log_line='Scan already in progress — waiting for completion',
+                log_type='info',
+            )
+        else:
+            delay = result.get('delay_seconds', 60)
+            deps.update_progress(
+                automation_id,
+                log_line=f'Scan scheduled (debounce: {delay}s)',
+                log_type='info',
+            )
+
+        # Unified polling loop — handles debounce → scanning → idle.
+        poll_start = time.time()
+        scan_started = (scan_status_val == 'queued')
+        while time.time() - poll_start < _SCAN_TIMEOUT_SECONDS:
+            status = deps.web_scan_manager.get_scan_status()
+            st = status.get('status')
+
+            if st == 'idle':
+                break  # Scan completed (or finished before we polled)
+
+            if st == 'scheduled':
+                elapsed = int(time.time() - poll_start)
+                deps.update_progress(
+                    automation_id,
+                    phase=f'Waiting for scan to start... ({elapsed}s)',
+                    progress=min(int(elapsed / 60 * 10), _PROGRESS_SCHEDULED_MAX),
+                )
+                time.sleep(_POLL_SCHEDULED_SECONDS)
+                continue
+
+            if st == 'scanning':
+                if not scan_started:
+                    scan_started = True
+                    deps.update_progress(
+                        automation_id,
+                        progress=_PROGRESS_SCAN_START,
+                        log_line='Scan triggered on media server',
+                        log_type='success',
+                    )
+                elapsed = status.get('elapsed_seconds', 0)
+                max_time = status.get('max_time_seconds', 300)
+                pct = min(_PROGRESS_SCAN_START + int(elapsed / max_time * 80), _PROGRESS_SCAN_MAX)
+                mins, secs = divmod(elapsed, 60)
+                deps.update_progress(
+                    automation_id,
+                    phase=f'Library scan in progress... ({mins}m {secs}s)',
+                    progress=pct,
+                )
+                time.sleep(_POLL_SCANNING_SECONDS)
+                continue
+
+            time.sleep(_POLL_UNKNOWN_SECONDS)
+        else:
+            # 30-min timeout reached
+            deps.update_progress(
+                automation_id,
+                status='error',
+                phase='Timed out',
+                log_line='Library scan timed out after 30 minutes',
+                log_type='error',
+            )
+            return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
+
+        elapsed = round(time.time() - poll_start, 1)
+        deps.update_progress(
+            automation_id,
+            status='finished',
+            progress=100,
+            phase='Complete',
+            log_line='Library scan completed',
+            log_type='success',
+        )
+        return {
+            'status': 'completed',
+            '_manages_own_progress': True,
+            'scan_duration_seconds': elapsed,
+        }
+
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise into the engine
+        deps.update_progress(
+            automation_id,
+            status='error',
+            phase='Error',
+            log_line=str(e),
+            log_type='error',
+        )
+        return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
+
+    finally:
+        deps.state.set_scan_library_id(None)

core/automation/handlers/scan_watchlist.py

@@ -0,0 +1,46 @@
+"""Automation handler: ``scan_watchlist`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_scan_watchlist`` closure). The watchlist scanner returns
+summary stats for the trigger card only when a fresh scan actually
+ran — detected by snapshotting ``id(state_dict)`` before/after, since
+the live processor reassigns the dict on each new scan.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_scan_watchlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Run a watchlist scan when the automation triggers.
+
+    Pre-scan we capture ``id(watchlist_scan_state)`` so we can tell
+    afterwards whether the worker ran (and reassigned the state dict)
+    or short-circuited (kept the same dict). Only fresh scans report
+    summary stats — repeat triggers without an intervening run return
+    a bare ``completed``.
+    """
+    try:
+        pre_state = deps.get_watchlist_scan_state()
+        pre_state_id = id(pre_state)
+        deps.process_watchlist_scan_automatically(
+            automation_id=config.get('_automation_id'),
+            profile_id=config.get('_profile_id'),
+        )
+        post_state = deps.get_watchlist_scan_state()
+        # Fresh scan = state dict was reassigned mid-run.
+        if id(post_state) != pre_state_id:
+            summary = post_state.get('summary', {}) if isinstance(post_state, dict) else {}
+            return {
+                'status': 'completed',
+                'artists_scanned': summary.get('total_artists', 0),
+                'successful_scans': summary.get('successful_scans', 0),
+                'new_tracks_found': summary.get('new_tracks_found', 0),
+                'tracks_added_to_wishlist': summary.get('tracks_added_to_wishlist', 0),
+            }
+        return {'status': 'completed'}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise into the engine
+        return {'status': 'error', 'error': str(e)}

core/automation/handlers/search_and_download.py

@@ -0,0 +1,57 @@
+"""Automation handler: ``search_and_download`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_search_and_download`` closure). Searches for a track by
+name/artist string and dispatches the best match through the
+download orchestrator. Query can come from the trigger config
+(direct value) or from event data (e.g. webhook payload).
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_search_and_download(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    automation_id = config.get('_automation_id')
+    query = config.get('query', '').strip()
+    # Event-triggered: pull query from event data (e.g. webhook_received).
+    if not query:
+        event_data = config.get('_event_data', {})
+        query = (event_data.get('query', '') or '').strip()
+    if not query:
+        if automation_id:
+            deps.update_progress(
+                automation_id, log_line='No search query provided', log_type='error',
+            )
+        return {'status': 'error', 'error': 'No search query provided'}
+    try:
+        if automation_id:
+            deps.update_progress(
+                automation_id, phase='Searching',
+                log_line=f'Searching: {query}', log_type='info',
+            )
+        result = deps.run_async(deps.download_orchestrator.search_and_download_best(query))
+        if result:
+            if automation_id:
+                deps.update_progress(
+                    automation_id,
+                    log_line=f'Download started for: {query}',
+                    log_type='success',
+                )
+            return {'status': 'completed', 'query': query, 'download_id': result}
+        if automation_id:
+            deps.update_progress(
+                automation_id,
+                log_line=f'No match found for: {query}',
+                log_type='warning',
+            )
+        return {'status': 'not_found', 'query': query, 'error': 'No match found'}
+    except Exception as e:  # noqa: BLE001 — automation handlers must never raise
+        if automation_id:
+            deps.update_progress(
+                automation_id, log_line=f'Error: {e}', log_type='error',
+            )
+        return {'status': 'error', 'query': query, 'error': str(e)}

core/automation/handlers/sync_playlist.py

@@ -0,0 +1,240 @@
+"""Automation handler: ``sync_playlist`` action.
+
+Lifted from ``web_server._register_automation_handlers`` (the
+``_auto_sync_playlist`` closure). Syncs a mirrored playlist to the
+configured media server, using discovered metadata when available
+and skipping undiscovered tracks. When triggered on a schedule with
+no track changes since the last sync, short-circuits with
+``status: skipped`` (saves Plex / Jellyfin / Navidrome from
+needless rewrites)."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import threading
+from typing import Any, Dict
+
+from core.automation.deps import AutomationDeps
+
+
+def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
+    """Sync a mirrored playlist to the active media server.
+
+    Behavior:
+    - Tracks with discovered metadata (extra_data.discovered + matched_data)
+      are routed via the official metadata.
+    - Tracks with a Spotify hint (real Spotify ID from the embed
+      scraper) are included so they can still hit Soulseek + the
+      wishlist.
+    - Tracks with neither are counted as ``skipped_tracks``.
+    - Empty result → ``status: skipped`` with the skipped count.
+    - Same track set as last sync (matched_tracks unchanged) →
+      ``status: skipped`` (no-op).
+    - Otherwise spawns a daemon thread running ``run_sync_task`` and
+      returns ``status: started`` with ``_manages_own_progress: True``.
+    """
+    auto_id = config.get('_automation_id')
+    playlist_id = config.get('playlist_id')
+    if not playlist_id:
+        return {'status': 'error', 'reason': 'No playlist specified'}
+
+    db = deps.get_database()
+    pl = db.get_mirrored_playlist(int(playlist_id))
+    if not pl:
+        return {'status': 'error', 'reason': 'Playlist not found'}
+
+    tracks = db.get_mirrored_playlist_tracks(int(playlist_id))
+    if not tracks:
+        return {'status': 'error', 'reason': 'No tracks in playlist'}
+
+    # Convert mirrored tracks to format expected by run_sync_task.
+    # Use discovered metadata when available, fall back to Spotify
+    # hint or raw playlist fields when not.
+    tracks_json = []
+    skipped_count = 0
+
+    for t in tracks:
+        # Parse extra_data for discovery info.
+        extra = {}
+        if t.get('extra_data'):
+            try:
+                extra = json.loads(t['extra_data']) if isinstance(t['extra_data'], str) else t['extra_data']
+            except (json.JSONDecodeError, TypeError):
+                pass
+
+        if extra.get('discovered') and extra.get('matched_data'):
+            # Use official discovered metadata.
+            md = extra['matched_data']
+            album_raw = md.get('album', '')
+            album_obj = album_raw if isinstance(album_raw, dict) else {'name': album_raw or ''}
+            _track_entry = {
+                'name': md.get('name', ''),
+                'artists': md.get('artists', [{'name': t.get('artist_name', '')}]),
+                'album': album_obj,
+                'duration_ms': md.get('duration_ms', 0),
+                'id': md.get('id', ''),
+            }
+            if md.get('track_number'):
+                _track_entry['track_number'] = md['track_number']
+            if md.get('disc_number'):
+                _track_entry['disc_number'] = md['disc_number']
+            tracks_json.append(_track_entry)
+        else:
+            # NOT discovered — try to include using available metadata so
+            # the track can still be searched on Soulseek and added to
+            # wishlist. Without this, failed discovery blocks the entire
+            # download pipeline.
+            #
+            # Priority: spotify_hint (has real Spotify ID from embed
+            # scraper) > raw playlist fields (only if source_track_id
+            # is valid).
+            hint = extra.get('spotify_hint', {})
+            # Build album object with cover art from the mirrored playlist track.
+            track_image = (t.get('image_url') or '').strip()
+            album_obj = {
+                'name': (t.get('album_name') or '').strip(),
+                'images': [{'url': track_image, 'height': 300, 'width': 300}] if track_image else [],
+            }
+
+            if hint.get('id') and hint.get('name'):
+                # spotify_hint has proper Spotify track ID + metadata from embed scraper.
+                hint_artists = hint.get('artists', [])
+                if hint_artists and isinstance(hint_artists[0], str):
+                    hint_artists = [{'name': a} for a in hint_artists]
+                elif hint_artists and isinstance(hint_artists[0], dict):
+                    pass  # Already in correct format
+                else:
+                    hint_artists = [{'name': t.get('artist_name', '')}]
+                tracks_json.append({
+                    'name': hint['name'],
+                    'artists': hint_artists,
+                    'album': album_obj,
+                    'duration_ms': t.get('duration_ms', 0),
+                    'id': hint['id'],
+                })
+            elif t.get('source_track_id') and (t.get('track_name') or '').strip():
+                # Has a valid source ID and track name — usable for wishlist.
+                tracks_json.append({
+                    'name': t['track_name'].strip(),
+                    'artists': [{'name': (t.get('artist_name') or '').strip() or 'Unknown Artist'}],
+                    'album': album_obj,
+                    'duration_ms': t.get('duration_ms', 0),
+                    'id': t['source_track_id'],
+                })
+            else:
+                skipped_count += 1  # No usable ID or name — truly can't process.
+
+    if not tracks_json:
+        deps.update_progress(
+            auto_id,
+            log_line=f'No discovered tracks — {skipped_count} need discovery first',
+            log_type='skip',
+        )
+        return {
+            'status': 'skipped',
+            'reason': f'No discovered tracks to sync ({skipped_count} tracks need discovery first)',
+            'skipped_tracks': str(skipped_count),
+        }
+
+    # Preflight: hash the track list and compare against last sync.
+    # Skip if the exact same set of tracks was already synced and
+    # everything matched (no-op preserves Plex / Jellyfin / Navidrome
+    # from needless rewrites).
+    track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
+    tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
+
+    sync_id_key = f"auto_mirror_{playlist_id}"
+    # Full mirror identity (every source_track_id on the playlist). tracks_hash
+    # only covers tracks_json — if a new mirror row is skipped (no discovery /
+    # no source id), tracks_hash stays identical to the pre-add sync and we
+    # used to no-op with "unchanged" while the new song never hit wishlist.
+    mirror_ids_str = ','.join(
+        sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
+    )
+    mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
+
+    event_data = config.get('_event_data') or {}
+    try:
+        tracks_added = int(event_data.get('added') or 0)
+    except (TypeError, ValueError):
+        tracks_added = 0
+    force_sync = tracks_added > 0 or skipped_count > 0
+
+    try:
+        sync_statuses = deps.load_sync_status_file()
+        last_status = sync_statuses.get(sync_id_key, {})
+        last_hash = last_status.get('tracks_hash', '')
+        last_mirror_hash = last_status.get('mirror_tracks_hash', '')
+        last_matched = last_status.get('matched_tracks', -1)
+
+        mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
+        if (
+            not force_sync
+            and not mirror_changed
+            and last_hash == tracks_hash
+            and last_matched >= len(tracks_json)
+        ):
+            # Exact same tracks, all matched last time — nothing to do.
+            deps.update_progress(
+                auto_id,
+                log_line=f'All {len(tracks_json)} tracks unchanged since last sync — skipping',
+                log_type='skip',
+            )
+            return {
+                'status': 'skipped',
+                'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
+            }
+        if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
+            deps.update_progress(
+                auto_id,
+                log_line=(
+                    f'Forcing sync: playlist changed ({tracks_added} added) or '
+                    f'{skipped_count} track(s) need discovery'
+                ),
+                log_type='info',
+            )
+        elif mirror_changed:
+            deps.update_progress(
+                auto_id,
+                log_line='Mirror track list changed — running sync',
+                log_type='info',
+            )
+    except Exception as e:
+        deps.logger.debug("mirror sync last-status read: %s", e)
+
+    # Sync under the user's custom alias when set, else the upstream name (#865
+    # follow-up). The server-side playlist is named with this.
+    from core.playlists.naming import effective_mirrored_name
+    sync_name = effective_mirrored_name(pl) or pl.get('name') or 'Playlist'
+
+    deps.update_progress(
+        auto_id,
+        progress=50,
+        phase=f'Syncing "{sync_name}"',
+        log_line=f'{len(tracks_json)} discovered, {skipped_count} skipped',
+        log_type='info',
+    )
+
+    sync_id = f"auto_mirror_{playlist_id}"
+    deps.update_progress(
+        auto_id,
+        progress=90,
+        log_line=f'Starting sync: {len(tracks_json)} tracks',
+        log_type='success',
+    )
+    skip_wishlist_add = bool(pl.get('organize_by_playlist'))
+    threading.Thread(
+        target=deps.run_sync_task,
+        args=(sync_id, sync_name, tracks_json, auto_id, 1, pl.get('image_url', '')),
+        kwargs={'skip_wishlist_add': skip_wishlist_add},
+        daemon=True,
+        name=f'auto-sync-{playlist_id}',
+    ).start()
+    return {
+        'status': 'started',
+        'playlist_name': sync_name,
+        'discovered_tracks': str(len(tracks_json)),
+        'skipped_tracks': str(skipped_count),
+        '_manages_own_progress': True,
+    }

core/automation/progress.py

@@ -0,0 +1,216 @@
+"""Automation progress tracking.
+
+Owns the in-memory progress state dict that backs both
+`/api/automations/progress` polling and the WebSocket
+`automation:progress` push emitter. State is per-automation, capped at
+50 log entries each, and finished/error states are reaped 60s after
+they finish so the frontend has a window to show the final state.
+
+Functions are written so the route layer / engine callbacks can pass
+their own socketio emitter, db handle, and shutdown flag. The progress
+state dict (`progress_states`) and its lock (`progress_lock`) are
+module-level so all callers share one view — same as the original
+web_server.py globals.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import threading
+from datetime import datetime, timezone
+from typing import Any, Callable, Optional
+
+logger = logging.getLogger(__name__)
+
+# Shared mutable state — module globals so every caller (routes, engine
+# progress callbacks, emit loop) sees the same dict. Mirrors the original
+# `automation_progress_states` / `automation_progress_lock` in web_server.
+progress_states: dict[int, dict] = {}
+progress_lock = threading.Lock()
+
+
+def init_progress(automation_id: int, automation_name: str, action_type: str) -> None:
+    """Initialize progress state when an automation starts running."""
+    with progress_lock:
+        progress_states[automation_id] = {
+            'status': 'running',
+            'action_type': action_type,
+            'progress': 0,
+            'phase': 'Starting...',
+            'current_item': '',
+            'processed': 0,
+            'total': 0,
+            'log': [{'type': 'info', 'text': f'Starting {automation_name}'}],
+            'started_at': datetime.now(timezone.utc).isoformat(),
+            'finished_at': None,
+        }
+
+
+def update_progress(
+    automation_id: Optional[int],
+    *,
+    socketio_emit: Optional[Callable[[str, Any], None]] = None,
+    **kwargs,
+) -> None:
+    """Update progress state from handler threads. Thread-safe.
+
+    `socketio_emit` lets callers wire in the live socketio.emit so that
+    finished/error transitions push immediately without waiting for the
+    1s emitter loop. Falls back to no-op if not provided.
+    """
+    if automation_id is None:
+        return
+    with progress_lock:
+        state = progress_states.get(automation_id)
+        if not state:
+            return
+        for k, v in kwargs.items():
+            if k == 'log_line':
+                state['log'].append({'type': kwargs.get('log_type', 'info'), 'text': v})
+                if len(state['log']) > 50:
+                    state['log'] = state['log'][-50:]
+            elif k != 'log_type':
+                state[k] = v
+        if kwargs.get('status') in ('finished', 'error'):
+            state['finished_at'] = datetime.now(timezone.utc).isoformat()
+            if socketio_emit is not None:
+                try:
+                    socketio_emit('automation:progress', {str(automation_id): dict(state)})
+                except Exception as e:
+                    logger.debug("socketio progress emit: %s", e)
+
+
+def get_running_progress() -> dict[str, dict]:
+    """Snapshot of running/finished/error states for the polling endpoint."""
+    with progress_lock:
+        result: dict[str, dict] = {}
+        for aid, state in progress_states.items():
+            if state['status'] in ('running', 'finished', 'error'):
+                cp = dict(state)
+                cp['log'] = list(state['log'])
+                result[str(aid)] = cp
+    return result
+
+
+def record_history(
+    automation_id: int,
+    result: dict,
+    database,
+) -> None:
+    """Capture progress state into run history before cleanup clears it.
+
+    `database` is passed in so the function works without a `get_database()`
+    global.
+    """
+    try:
+        with progress_lock:
+            state = progress_states.get(automation_id)
+            if state:
+                started_at = state.get('started_at')
+                finished_at = state.get('finished_at') or datetime.now(timezone.utc).isoformat()
+                log_entries = list(state.get('log', []))
+            else:
+                started_at = datetime.now(timezone.utc).isoformat()
+                finished_at = datetime.now(timezone.utc).isoformat()
+                log_entries = []
+
+        duration = None
+        if started_at and finished_at:
+            try:
+                t0 = datetime.fromisoformat(started_at)
+                t1 = datetime.fromisoformat(finished_at)
+                duration = (t1 - t0).total_seconds()
+            except Exception as e:
+                logger.debug("duration parse: %s", e)
+
+        r_status = result.get('status', 'completed') if result else 'completed'
+        if r_status == 'error':
+            status = 'error'
+        elif r_status == 'skipped':
+            status = 'skipped'
+        elif r_status == 'timeout':
+            status = 'timeout'
+        else:
+            status = 'completed'
+
+        summary = None
+        for entry in reversed(log_entries):
+            if entry.get('type') in ('success', 'error'):
+                summary = entry.get('text', '')
+                break
+        if not summary and log_entries:
+            summary = log_entries[-1].get('text', '')
+        if not summary and result:
+            summary = result.get('reason') or result.get('error') or result.get('status', '')
+
+        result_json = json.dumps({k: v for k, v in result.items() if not k.startswith('_')}) if result else None
+        log_json = json.dumps(log_entries) if log_entries else None
+
+        database.insert_automation_run_history(
+            automation_id=automation_id,
+            started_at=started_at,
+            finished_at=finished_at,
+            duration_seconds=duration,
+            status=status,
+            summary=summary,
+            result_json=result_json,
+            log_lines=log_json,
+        )
+    except Exception as e:
+        logger.error(f"Error recording automation history for {automation_id}: {e}")
+
+
+def emit_progress_loop(
+    socketio,
+    *,
+    is_shutting_down: Callable[[], bool],
+    poll_interval: float = 1.0,
+    timeout_seconds: int = 7200,
+    cleanup_after_seconds: int = 60,
+) -> None:
+    """Push `automation:progress` events for active automations.
+
+    Long-running loop — caller wires this into a socketio background task.
+    - Times out zombie running states after `timeout_seconds` (default 2h).
+    - Reaps finished/error states `cleanup_after_seconds` after finish so the
+      frontend has a final-state window before they disappear.
+    """
+    while not is_shutting_down():
+        socketio.sleep(poll_interval)
+        try:
+            with progress_lock:
+                active: dict[str, dict] = {}
+                stale: list[int] = []
+                now = datetime.now()
+                for aid, state in progress_states.items():
+                    if state['status'] == 'running':
+                        try:
+                            started = datetime.fromisoformat(state.get('started_at', ''))
+                            if (now - started).total_seconds() > timeout_seconds:
+                                state['status'] = 'error'
+                                state['phase'] = 'Timed out'
+                                state['finished_at'] = now.isoformat()
+                                state['log'].append({'type': 'error', 'text': f'Timed out after {timeout_seconds // 3600} hours'})
+                                cp = dict(state)
+                                cp['log'] = list(state['log'])
+                                active[str(aid)] = cp
+                                continue
+                        except (ValueError, TypeError):
+                            pass
+                        cp = dict(state)
+                        cp['log'] = list(state['log'])
+                        active[str(aid)] = cp
+                    elif state['status'] in ('finished', 'error') and state.get('finished_at'):
+                        try:
+                            finished_time = datetime.fromisoformat(state['finished_at'])
+                            if (now - finished_time).total_seconds() > cleanup_after_seconds:
+                                stale.append(aid)
+                        except (ValueError, TypeError):
+                            stale.append(aid)
+                for aid in stale:
+                    del progress_states[aid]
+            if active:
+                socketio.emit('automation:progress', active)
+        except Exception as e:
+            logger.debug(f"Error emitting automation progress: {e}")

core/automation/schedule.py

@@ -0,0 +1,321 @@
+"""Pure functions for computing the next-run datetime of a scheduled
+automation trigger.
+
+The Auto-Sync schedule board currently exposes interval-based scheduling
+(``every N hours``) backed by ``trigger_type='schedule'``. The
+automation engine ALSO supports ``daily_time`` and ``weekly_time``
+triggers via separate ``_setup_*_trigger`` methods inline on the engine
+class. None of that logic is currently testable in isolation — the
+engine's ``_finish_run`` reaches for ``datetime.now()``, threads it
+through ``_next_weekly_occurrence``, and writes the result to the DB,
+all on the same call.
+
+This module lifts the "given a trigger config, what's the next run?"
+question out of the engine into a pure function:
+
+    next_run_at(trigger_type, trigger_config, now_utc, default_tz)
+        -> Optional[datetime]
+
+That means:
+- ``now_utc`` is INJECTED, not pulled from the system clock. Tests
+  freeze time without monkeypatching ``datetime.now``.
+- ``default_tz`` is INJECTED. Daily / weekly / monthly schedules are
+  inherently in the USER'S timezone (cron "every Monday at 9am" is
+  not UTC), and the historic engine implicitly used the server's
+  local tz via naive ``datetime.now()``. That broke for users on a
+  different tz than their server. The pure function takes the tz
+  explicitly so the caller controls it.
+- Returns an aware UTC ``datetime`` ready to serialise to the DB's
+  ``next_run`` string column, or ``None`` for unrecognised /
+  event-based triggers (engine should not store a next_run for those).
+
+PR 1 of the schedule-types feature ships ONLY this module + tests.
+The engine continues to compute next_run via its existing inline
+helpers; PR 2 collapses those into a single ``next_run_at`` call.
+Net behavior is identical until the engine is wired through — this
+PR is pure plumbing.
+
+Schedule types supported here:
+
+- ``schedule`` (interval): ``{interval: N, unit: 'minutes'|'hours'|'days'}``
+  — adds the interval to ``now_utc``; no tz needed.
+- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — runs every day at
+  the given local time in the given timezone. ``tz`` falls back to
+  ``default_tz`` when absent.
+- ``weekly_time``: ``{time: 'HH:MM', days: ['mon','wed',...], tz: '<IANA>'}``
+  — runs on the matching weekday(s) at the given local time. Empty
+  ``days`` list means "every day" (matches the engine's existing
+  fallback in ``_next_weekly_occurrence``).
+- ``monthly_time``: ``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}``
+  — runs on the given day each month. Days that don't exist in a
+  given month (Feb 30, Apr 31) clamp to the LAST valid day of that
+  month rather than skipping the run entirely; missing a whole
+  month silently because the schedule was over-eager is worse than
+  running a day early.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timedelta, timezone
+from typing import Any, Dict, Optional
+from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
+
+from utils.logging_config import get_logger
+
+logger = get_logger("automation.schedule")
+
+
+# Unknown-tz names already warned about in this process — avoids
+# spamming the log on every poll cycle for the same misconfigured row.
+_UNKNOWN_TZ_WARNED: set = set()
+
+
+# Weekday abbreviation → ``datetime.weekday()`` index (Mon=0..Sun=6).
+# Mirrors the engine's existing ``_next_weekly_occurrence`` mapping so
+# schedules created against either implementation accept the same
+# ``days`` strings.
+_WEEKDAY_MAP = {
+    'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6,
+}
+
+# Interval multipliers — kept aligned with the engine's existing
+# ``_calc_delay_seconds`` in ``core/automation_engine.py``. Adding
+# entries here without also updating the engine would silently drift:
+# this function would honour the new unit while the live engine path
+# defaults it to hours. Keep the maps in sync until PR 2 collapses the
+# engine through this function.
+_INTERVAL_MULTIPLIERS = {
+    'minutes': 60,
+    'hours':   60 * 60,
+    'days':    60 * 60 * 24,
+}
+
+
+def next_run_at(
+    trigger_type: str,
+    trigger_config: Dict[str, Any],
+    now_utc: datetime,
+    default_tz: str = 'UTC',
+) -> Optional[datetime]:
+    """Compute the next-run timestamp (UTC, aware) for a scheduled
+    trigger. Returns ``None`` for unrecognised types or event-based
+    triggers — callers should not write a next_run for those.
+
+    See module docstring for supported trigger types + config shapes.
+    """
+    if not isinstance(trigger_config, dict):
+        trigger_config = {}
+
+    if trigger_type == 'schedule':
+        return _next_interval(trigger_config, now_utc)
+    if trigger_type == 'daily_time':
+        return _next_daily(trigger_config, now_utc, default_tz)
+    if trigger_type == 'weekly_time':
+        return _next_weekly(trigger_config, now_utc, default_tz)
+    if trigger_type == 'monthly_time':
+        return _next_monthly(trigger_config, now_utc, default_tz)
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Interval
+# ---------------------------------------------------------------------------
+
+
+def _next_interval(config: Dict[str, Any], now_utc: datetime) -> datetime:
+    """``{interval: N, unit: 'hours'}`` → ``now_utc + N hours``.
+
+    Mirrors the engine's existing ``_calc_delay_seconds``. Unit defaults
+    to ``hours`` for backward compat with legacy DB rows that pre-date
+    the unit field being mandatory; interval defaults to 1 so a fully
+    empty config doesn't divide-by-zero or schedule for the past."""
+    try:
+        interval = max(int(config.get('interval', 1)), 1)
+    except (TypeError, ValueError):
+        interval = 1
+    unit = config.get('unit') or 'hours'
+    seconds = interval * _INTERVAL_MULTIPLIERS.get(unit, _INTERVAL_MULTIPLIERS['hours'])
+    return _ensure_utc(now_utc) + timedelta(seconds=seconds)
+
+
+# ---------------------------------------------------------------------------
+# Daily
+# ---------------------------------------------------------------------------
+
+
+def _next_daily(
+    config: Dict[str, Any], now_utc: datetime, default_tz: str,
+) -> datetime:
+    """``{time: 'HH:MM', tz: '<IANA>'}`` → next occurrence of that
+    wall-clock time in the user's timezone, expressed as aware UTC.
+
+    DST-aware via ``zoneinfo``: when the local time falls during a
+    spring-forward gap, the ``replace`` lands on a non-existent
+    instant; ``zoneinfo`` resolves that to the gap's later side
+    (e.g. 02:30 on the DST-forward day becomes 03:30 local). Tests
+    pin both spring-forward and fall-back behaviour."""
+    tz = _resolve_tz(config.get('tz') or default_tz)
+    hour, minute = _parse_hhmm(config.get('time'))
+    now_local = _ensure_utc(now_utc).astimezone(tz)
+    target_local = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
+    if target_local <= now_local:
+        target_local = target_local + timedelta(days=1)
+    return target_local.astimezone(timezone.utc)
+
+
+# ---------------------------------------------------------------------------
+# Weekly
+# ---------------------------------------------------------------------------
+
+
+def _next_weekly(
+    config: Dict[str, Any], now_utc: datetime, default_tz: str,
+) -> datetime:
+    """``{time: 'HH:MM', days: ['mon',...], tz: '<IANA>'}`` → next
+    occurrence of that wall-clock time on any of the listed weekdays
+    in the user's timezone.
+
+    Empty ``days`` list ≡ every day, matching the engine's existing
+    fallback. Unrecognised day abbreviations are silently dropped
+    (an empty result-set then triggers the every-day fallback)."""
+    tz = _resolve_tz(config.get('tz') or default_tz)
+    hour, minute = _parse_hhmm(config.get('time'))
+    days = _parse_weekdays(config.get('days'))
+
+    now_local = _ensure_utc(now_utc).astimezone(tz)
+    # Scan today + next 7 days; the matching day with a future
+    # local time wins. 8-day scan is enough to handle the case where
+    # today already passed the time AND today is the only allowed
+    # weekday (next occurrence is exactly one week out).
+    for offset in range(8):
+        candidate = now_local + timedelta(days=offset)
+        if candidate.weekday() not in days:
+            continue
+        target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
+        if target > now_local:
+            return target.astimezone(timezone.utc)
+    # Shouldn't reach: 8-day scan always finds a hit when ``days``
+    # is non-empty. Defensive fallback: next week, same weekday as today.
+    fallback = (now_local + timedelta(days=7)).replace(
+        hour=hour, minute=minute, second=0, microsecond=0,
+    )
+    return fallback.astimezone(timezone.utc)
+
+
+# ---------------------------------------------------------------------------
+# Monthly
+# ---------------------------------------------------------------------------
+
+
+def _next_monthly(
+    config: Dict[str, Any], now_utc: datetime, default_tz: str,
+) -> datetime:
+    """``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}`` → next
+    occurrence in the user's timezone.
+
+    ``day_of_month`` is clamped to ``[1, 31]``. When the target day
+    doesn't exist in a given month (Feb 30, Apr 31), the schedule
+    falls back to the LAST valid day of that month — running a day
+    or two early in short months is less surprising than skipping
+    a month entirely. This matches the convention every cron
+    implementation in the wild settled on."""
+    tz = _resolve_tz(config.get('tz') or default_tz)
+    hour, minute = _parse_hhmm(config.get('time'))
+    raw_day = config.get('day_of_month', 1)
+    try:
+        target_day = max(1, min(31, int(raw_day)))
+    except (TypeError, ValueError):
+        target_day = 1
+
+    now_local = _ensure_utc(now_utc).astimezone(tz)
+    # Try this month first; if the target day has already passed
+    # (or doesn't exist this month and the clamped day is in the
+    # past), advance to next month. Loop bounded to 12 iterations
+    # so a pathologically broken config can't infinite-loop us.
+    year, month = now_local.year, now_local.month
+    for _ in range(12):
+        day = min(target_day, _days_in_month(year, month))
+        target = now_local.replace(
+            year=year, month=month, day=day,
+            hour=hour, minute=minute, second=0, microsecond=0,
+        )
+        if target > now_local:
+            return target.astimezone(timezone.utc)
+        # Roll to next month.
+        if month == 12:
+            year, month = year + 1, 1
+        else:
+            month += 1
+    # Defensive — should be unreachable.
+    return (now_local + timedelta(days=30)).astimezone(timezone.utc)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _ensure_utc(dt: datetime) -> datetime:
+    """Coerce a possibly-naive datetime to aware UTC. Naive inputs
+    are assumed UTC (matches the convention the engine uses when
+    parsing the DB ``next_run`` column)."""
+    if dt.tzinfo is None:
+        return dt.replace(tzinfo=timezone.utc)
+    return dt.astimezone(timezone.utc)
+
+
+def _resolve_tz(name: Optional[str]):
+    """Look up an IANA tz by name. Falls back to UTC when the name is
+    unknown — ``ZoneInfoNotFoundError`` is the symptom of either a
+    typo in the tz string or ``tzdata`` missing on the host. Logged
+    once per unknown name so the user can see WHY their schedule
+    isn't running in the timezone they configured."""
+    if not name:
+        return timezone.utc
+    try:
+        return ZoneInfo(name)
+    except ZoneInfoNotFoundError:
+        if name not in _UNKNOWN_TZ_WARNED:
+            _UNKNOWN_TZ_WARNED.add(name)
+            logger.warning(
+                "Unknown timezone %r — schedule will run against UTC. "
+                "Check the spelling (IANA format like 'America/Los_Angeles') "
+                "or install the `tzdata` package on minimal hosts.",
+                name,
+            )
+        return timezone.utc
+
+
+def _parse_hhmm(time_str: Optional[str]) -> tuple:
+    """Parse ``HH:MM`` → ``(hour, minute)``. Defaults to 00:00 on
+    garbage input — same defensive shape as the engine's existing
+    daily/weekly time parsing."""
+    if not isinstance(time_str, str):
+        return 0, 0
+    try:
+        h, m = time_str.split(':', 1)
+        return max(0, min(23, int(h))), max(0, min(59, int(m)))
+    except (ValueError, AttributeError):
+        return 0, 0
+
+
+def _parse_weekdays(days) -> set:
+    """``['mon', 'wed']`` → ``{0, 2}``. Empty / missing / all-invalid
+    list returns ``set(range(7))`` ("every day"), matching the
+    engine's existing ``_next_weekly_occurrence`` fallback."""
+    if not isinstance(days, (list, tuple)):
+        return set(range(7))
+    parsed = {_WEEKDAY_MAP[d.lower()] for d in days
+              if isinstance(d, str) and d.lower() in _WEEKDAY_MAP}
+    return parsed or set(range(7))
+
+
+def _days_in_month(year: int, month: int) -> int:
+    """Last calendar day of ``year-month``. Stdlib-only — no calendar
+    module import needed; cycle through the 12 months."""
+    if month == 12:
+        next_first = datetime(year + 1, 1, 1)
+    else:
+        next_first = datetime(year, month + 1, 1)
+    last_day = next_first - timedelta(days=1)
+    return last_day.day

core/automation/signals.py

@@ -0,0 +1,46 @@
+"""Automation signal helpers — name collection for autocomplete.
+
+Signal cycle detection itself lives in core/automation_engine.py
+(`detect_signal_cycles`); this module just enumerates known signal
+names from the saved automation set so the builder UI can autocomplete.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+def collect_known_signals(database) -> list[str]:
+    """Return sorted, deduped signal names referenced by any saved automation.
+
+    Walks every automation and pulls signal names from both the
+    `signal_received` trigger config and any `fire_signal` then-actions.
+    Errors at every layer are swallowed — the autocomplete is best-effort.
+    """
+    signals: set[str] = set()
+    try:
+        for auto in database.get_automations():
+            if auto.get('trigger_type') == 'signal_received':
+                try:
+                    tc = json.loads(auto.get('trigger_config') or '{}')
+                    sig = tc.get('signal_name', '').strip()
+                    if sig:
+                        signals.add(sig)
+                except (json.JSONDecodeError, TypeError):
+                    pass
+
+            try:
+                ta = json.loads(auto.get('then_actions') or '[]')
+                for item in ta:
+                    if item.get('type') == 'fire_signal':
+                        sig = item.get('config', {}).get('signal_name', '').strip()
+                        if sig:
+                            signals.add(sig)
+            except (json.JSONDecodeError, TypeError):
+                pass
+    except Exception as e:
+        logger.debug("collect known signals failed: %s", e)
+    return sorted(signals)

core/automation_engine.py

@@ -18,10 +18,14 @@ import time
 import threading
 import requests
 from datetime import datetime, timedelta, timezone
+from typing import Optional
 from utils.logging_config import get_logger
 
+from core.automation.schedule import next_run_at
+
 logger = get_logger("automation_engine")
 
+
 def _utcnow():
     """Return current UTC time as timezone-aware datetime."""
     return datetime.now(timezone.utc)
@@ -34,6 +38,42 @@ def _utc_after(seconds):
     """Return UTC time N seconds from now as naive string for DB storage."""
     return (datetime.now(timezone.utc) + timedelta(seconds=seconds)).strftime('%Y-%m-%d %H:%M:%S')
 
+
+def _dt_to_db_str(dt: datetime) -> str:
+    """Convert an aware-UTC datetime to the naive-UTC string the DB
+    ``next_run`` column stores. Centralised so a tz mistake here
+    surfaces in one place, not scattered through every caller of
+    ``next_run_at``."""
+    if dt.tzinfo is None:
+        return dt.strftime('%Y-%m-%d %H:%M:%S')
+    return dt.astimezone(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')
+
+
+def _resolve_system_default_tz() -> str:
+    """Return the IANA tz name the engine uses when a schedule's
+    trigger config doesn't carry an explicit ``tz`` field.
+
+    Existing daily / weekly rows pre-date the ``tz`` field — the
+    historic engine computed delays from naive ``datetime.now()``,
+    which is implicitly the server's local timezone. Falling back to
+    that same tz here preserves "every Monday at 09:00" running at
+    09:00 server local for rows that already exist in the DB.
+    Without ``tzlocal`` installed (or a system without a discoverable
+    tz), falls back to UTC."""
+    try:
+        import tzlocal
+        return tzlocal.get_localzone_name() or 'UTC'
+    except Exception:
+        return 'UTC'
+
+
+# Server-local tz cached at import time. Re-reading per-call is
+# pointless: the host's timezone doesn't change while the process is
+# running. Tests that need a different default tz inject it through
+# the engine's ``_default_tz`` attribute or via the
+# ``automation.default_timezone`` config key.
+_SYSTEM_DEFAULT_TZ = _resolve_system_default_tz()
+
 SYSTEM_AUTOMATIONS = [
     {
         'name': 'Auto-Process Wishlist',
@@ -134,11 +174,25 @@ class AutomationEngine:
         self._max_chain_depth = 5
         self._signal_cooldown_seconds = 10
 
+        # Default tz used when a schedule's ``trigger_config`` doesn't
+        # carry an explicit ``tz`` field — preserves historic behaviour
+        # for daily / weekly rows created before the field existed
+        # (engine used naive ``datetime.now()`` = server local). Reads
+        # from the ``automation.default_timezone`` config key first to
+        # let users override without touching env vars; falls back to
+        # the system-detected local tz.
+        try:
+            from config.settings import config_manager
+            self._default_tz = (config_manager.get('automation.default_timezone', '') or _SYSTEM_DEFAULT_TZ)
+        except Exception:
+            self._default_tz = _SYSTEM_DEFAULT_TZ
+
         # Trigger registry: type → setup function (schedule only — events use emit())
         self._trigger_handlers = {
             'schedule': self._setup_schedule_trigger,
             'daily_time': self._setup_daily_time_trigger,
             'weekly_time': self._setup_weekly_time_trigger,
+            'monthly_time': self._setup_monthly_time_trigger,
         }
 
     # --- Action Handler Registration ---
@@ -455,8 +509,10 @@ class AutomationEngine:
         if delay_minutes and delay_minutes > 0:
             # Initialize progress BEFORE delay so card glows during wait
             if self._progress_init_fn:
-                try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
-                except Exception: pass
+                try:
+                    self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
+                except Exception as e:
+                    logger.debug("event progress init (delay): %s", e)
                 _delay_already_inited = True
 
             delay_seconds = int(delay_minutes) * 60
@@ -486,13 +542,17 @@ class AutomationEngine:
                     logger.info(f"Event automation '{auto.get('name')}' skipped — {action_type} busy")
                     # If progress was initialized during delay, finalize it
                     if _delay_already_inited and self._progress_finish_fn:
-                        try: self._progress_finish_fn(automation_id, result)
-                        except Exception: pass
+                        try:
+                            self._progress_finish_fn(automation_id, result)
+                        except Exception as e:
+                            logger.debug("event progress finish (skipped): %s", e)
                 else:
                     # Initialize progress tracking (skip if already done during delay)
                     if not _delay_already_inited and self._progress_init_fn:
-                        try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
-                        except Exception: pass
+                        try:
+                            self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
+                        except Exception as e:
+                            logger.debug("event progress init: %s", e)
                     try:
                         result = handler_info['handler'](action_config) or {}
                         logger.info(f"Event automation '{auto.get('name')}' executed: {result.get('status', 'ok')}")
@@ -501,8 +561,10 @@ class AutomationEngine:
                         logger.error(f"Event automation '{auto.get('name')}' action failed: {e}")
                     # Finalize progress tracking
                     if self._progress_finish_fn:
-                        try: self._progress_finish_fn(automation_id, result)
-                        except Exception: pass
+                        try:
+                            self._progress_finish_fn(automation_id, result)
+                        except Exception as e:
+                            logger.debug("event progress finish: %s", e)
 
         # Merge event data into result for then-action variables
         merged = {**event_data, **result}
@@ -515,14 +577,24 @@ class AutomationEngine:
 
         # Update run stats (no reschedule — event triggers don't use timers)
         last_result = json.dumps({k: v for k, v in merged.items() if not k.startswith('_')})
-        error = result.get('error') if result.get('status') == 'error' else None
+        # Surface every failure mode to last_error: handlers in this codebase use
+        # 'error', 'reason', or 'message' interchangeably when returning gracefully.
+        if result.get('status') == 'error':
+            error = (
+                result.get('error')
+                or result.get('reason')
+                or result.get('message')
+                or 'Handler reported failure'
+            )
+        else:
+            error = None
         self.db.update_automation_run(automation_id, error=error, last_result=last_result)
 
         if self._history_record_fn:
             try:
                 self._history_record_fn(automation_id, result)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("history record failed: %s", e)
 
     # --- Schedule Execution (timer-based) ---
 
@@ -563,6 +635,11 @@ class AutomationEngine:
         action_config['_automation_name'] = auto.get('name', '')
         if profile_id is not None:
             action_config['_profile_id'] = profile_id
+        # The profile this run acts AS: an explicit trigger profile, else the
+        # automation's owner, else admin. System + admin automations are
+        # profile 1, so this is a no-op for them — only non-admin-owned
+        # automations gain their correct identity in the background.
+        _effective_profile_id = profile_id if profile_id is not None else (auto.get('profile_id') or 1)
 
         # Action delay (skipped for manual run_now)
         delay_minutes = action_config.get('delay', 0)
@@ -570,8 +647,10 @@ class AutomationEngine:
         if not skip_delay and delay_minutes and delay_minutes > 0:
             # Initialize progress BEFORE delay so card glows during wait
             if self._progress_init_fn:
-                try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
-                except Exception: pass
+                try:
+                    self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
+                except Exception as e:
+                    logger.debug("scheduled progress init (delay): %s", e)
                 _delay_already_inited = True
 
             delay_seconds = int(delay_minutes) * 60
@@ -593,31 +672,55 @@ class AutomationEngine:
             logger.info(f"Automation '{auto['name']}' skipped — {action_type} already running")
             # If progress was initialized during delay, finalize it
             if _delay_already_inited and self._progress_finish_fn:
-                try: self._progress_finish_fn(automation_id, result)
-                except Exception: pass
-            self._finish_run(auto, automation_id, result, error=None)
+                try:
+                    self._progress_finish_fn(automation_id, result)
+                except Exception as e:
+                    logger.debug("scheduled progress finish (skipped): %s", e)
+            self._finish_run(auto, automation_id, result, error=None, retry_delay_seconds=300)
             return
 
         # Initialize progress tracking (skip if already done during delay)
         if not _delay_already_inited and self._progress_init_fn:
-            try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
-            except Exception: pass
+            try:
+                self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
+            except Exception as e:
+                logger.debug("scheduled progress init: %s", e)
 
-        # Execute the action
+        # Execute the action under the owner's profile so get_current_profile_id()
+        # (and the per-profile clients it resolves) act as the automation's owner
+        # in the background, not admin. Reset in finally so a pooled thread can't
+        # leak the override to the next job.
         error = None
         result = {}
+        from core.profile_context import set_background_profile, reset_background_profile
+        _bg_token = set_background_profile(_effective_profile_id)
         try:
             result = handler_info['handler'](action_config) or {}
             logger.info(f"Automation '{auto['name']}' (id={automation_id}) executed: {result.get('status', 'ok')}")
+            # Handlers may signal failure by RETURNING {'status': 'error', ...} instead of
+            # raising. Surface that to the DB so `last_error` reflects every failure mode,
+            # not just uncaught exceptions. Falls back through ('error', 'reason', 'message')
+            # because handlers in this codebase aren't consistent about which key they set.
+            if result.get('status') == 'error':
+                error = (
+                    result.get('error')
+                    or result.get('reason')
+                    or result.get('message')
+                    or 'Handler reported failure'
+                )
         except Exception as e:
             error = str(e)
             result = {'status': 'error', 'error': error}
             logger.error(f"Automation '{auto['name']}' (id={automation_id}) failed: {e}")
+        finally:
+            reset_background_profile(_bg_token)
 
         # Finalize progress tracking
         if self._progress_finish_fn:
-            try: self._progress_finish_fn(automation_id, result)
-            except Exception: pass
+            try:
+                self._progress_finish_fn(automation_id, result)
+            except Exception as e:
+                logger.debug("scheduled progress finish: %s", e)
 
         # Execute then-actions (notifications + fire_signal)
         try:
@@ -627,7 +730,7 @@ class AutomationEngine:
 
         self._finish_run(auto, automation_id, result, error)
 
-    def _finish_run(self, auto, automation_id, result, error):
+    def _finish_run(self, auto, automation_id, result, error, retry_delay_seconds=None):
         """Update DB with run stats and reschedule."""
         next_run_str = None
         trigger_type = auto.get('trigger_type', '')
@@ -635,25 +738,25 @@ class AutomationEngine:
         if trigger_type in self._trigger_handlers:
             try:
                 trigger_config = json.loads(auto.get('trigger_config') or '{}')
-                if trigger_type == 'daily_time':
-                    # Next run is tomorrow at the configured time (compute delay from local time, store as UTC)
-                    time_str = trigger_config.get('time', '00:00')
-                    hour, minute = map(int, time_str.split(':'))
-                    now_local = datetime.now()
-                    target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
-                    next_run_str = _utc_after((target - now_local).total_seconds())
-                elif trigger_type == 'weekly_time':
-                    time_str = trigger_config.get('time', '00:00')
-                    hour, minute = map(int, time_str.split(':'))
-                    now_local = datetime.now()
-                    target = self._next_weekly_occurrence(hour, minute, trigger_config.get('days', []))
-                    next_run_str = _utc_after((target - now_local).total_seconds())
+                if retry_delay_seconds:
+                    next_run_str = _utc_after(retry_delay_seconds)
                 else:
-                    delay = self._calc_delay_seconds(trigger_config)
-                    if delay:
-                        next_run_str = _utc_after(delay)
-            except Exception:
-                pass
+                    # Single integration point with ``next_run_at``. The
+                    # helper handles every trigger type the engine
+                    # supports (interval / daily / weekly / monthly) and
+                    # returns aware-UTC; ``_dt_to_db_str`` normalises to
+                    # the naive-UTC string the DB column stores. Tests
+                    # injecting a different ``now_utc`` patch this same
+                    # path — no scattered ``datetime.now()`` calls left.
+                    next_run_dt = next_run_at(
+                        trigger_type, trigger_config,
+                        now_utc=_utcnow(),
+                        default_tz=self._default_tz,
+                    )
+                    if next_run_dt is not None:
+                        next_run_str = _dt_to_db_str(next_run_dt)
+            except Exception as e:
+                logger.debug("next run calc failed: %s", e)
 
         last_result = json.dumps(result) if result else None
         self.db.update_automation_run(automation_id, next_run=next_run_str, error=error, last_result=last_result)
@@ -661,8 +764,8 @@ class AutomationEngine:
         if self._history_record_fn:
             try:
                 self._history_record_fn(automation_id, result)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("history record failed: %s", e)
 
         if self._running:
             self.schedule_automation(automation_id)
@@ -725,45 +828,72 @@ class AutomationEngine:
         logger.debug(f"Scheduled automation {automation_id} in {delay:.0f}s")
 
     def _setup_daily_time_trigger(self, automation_id, config):
-        """Config: {"time": "03:00"}  — runs daily at the specified local time."""
-        time_str = config.get('time', '00:00')
-        try:
-            hour, minute = map(int, time_str.split(':'))
-        except (ValueError, AttributeError):
-            hour, minute = 0, 0
-
-        now_local = datetime.now()
-        target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
-        if target <= now_local:
-            target += timedelta(days=1)
-
-        delay = (target - now_local).total_seconds()
-
-        next_run_str = _utc_after(delay)
-        self.db.update_automation(automation_id, next_run=next_run_str)
-
-        timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
-        timer.daemon = True
-        timer.start()
-
-        with self._lock:
-            self._timers[automation_id] = timer
-
-        logger.debug(f"Daily automation {automation_id} scheduled for {time_str} (in {delay:.0f}s)")
+        """Config: ``{"time": "03:00", "tz": "<IANA>"}`` — runs daily
+        at the specified local time. Tz defaults to ``self._default_tz``
+        when absent."""
+        self._setup_timed_trigger(automation_id, 'daily_time', config,
+                                  label=f"Daily at {config.get('time', '00:00')}")
 
     def _setup_weekly_time_trigger(self, automation_id, config):
-        """Config: {"time": "03:00", "days": ["mon", "wed", "fri"]}"""
-        time_str = config.get('time', '00:00')
-        try:
-            hour, minute = map(int, time_str.split(':'))
-        except (ValueError, AttributeError):
-            hour, minute = 0, 0
+        """Config: ``{"time": "03:00", "days": ["mon","wed","fri"], "tz": "<IANA>"}``."""
+        day_names = ', '.join(config.get('days') or []) or 'every day'
+        self._setup_timed_trigger(automation_id, 'weekly_time', config,
+                                  label=f"Weekly {config.get('time', '00:00')} on {day_names}")
 
-        target = self._next_weekly_occurrence(hour, minute, config.get('days', []))
-        delay = (target - datetime.now()).total_seconds()
+    def _setup_monthly_time_trigger(self, automation_id, config):
+        """Config: ``{"time": "09:00", "day_of_month": 15, "tz": "<IANA>"}``.
 
-        next_run_str = _utc_after(delay)
-        self.db.update_automation(automation_id, next_run=next_run_str)
+        Day clamped to [1, 31]; months too short for the target day
+        clamp to the last valid day (Feb 31 → Feb 28 / Feb 29 leap
+        year) per standard cron convention — see
+        ``core.automation.schedule._next_monthly`` for the rule."""
+        day = config.get('day_of_month', 1)
+        self._setup_timed_trigger(automation_id, 'monthly_time', config,
+                                  label=f"Monthly {config.get('time', '00:00')} on day {day}")
+
+    def _setup_timed_trigger(self, automation_id, trigger_type, config, *, label):
+        """Shared setup for daily / weekly / monthly time triggers.
+
+        All three flow through the same skeleton: compute next-run
+        via ``next_run_at``, persist to DB, arm a ``threading.Timer``
+        that fires the automation when the delay elapses. Lifting
+        these out of three near-identical methods means there's one
+        place to fix when (e.g.) timer rearm semantics need a tweak.
+
+        Honours an existing future ``next_run`` row in the DB —
+        prevents losing a hand-edited next_run when the engine
+        reschedules at startup. Same guard as the interval path."""
+        target_dt = next_run_at(
+            trigger_type, config or {},
+            now_utc=_utcnow(),
+            default_tz=self._default_tz,
+        )
+        if target_dt is None:
+            logger.warning(
+                f"Skip scheduling automation {automation_id}: next_run_at returned "
+                f"None for {trigger_type!r}",
+            )
+            return
+
+        delay = max(0.0, (target_dt - _utcnow()).total_seconds())
+
+        # If the DB already carries a future next_run, prefer it —
+        # matches the interval-path behaviour and lets manual edits
+        # or pending retries survive a process restart.
+        auto = self.db.get_automation(automation_id)
+        if auto and auto.get('next_run'):
+            try:
+                existing = datetime.strptime(
+                    auto['next_run'], '%Y-%m-%d %H:%M:%S',
+                ).replace(tzinfo=timezone.utc)
+                remaining = (existing - _utcnow()).total_seconds()
+                if remaining > 0:
+                    delay = remaining
+                    target_dt = existing
+            except (ValueError, TypeError):
+                pass
+
+        self.db.update_automation(automation_id, next_run=_dt_to_db_str(target_dt))
 
         timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
         timer.daemon = True
@@ -772,25 +902,7 @@ class AutomationEngine:
         with self._lock:
             self._timers[automation_id] = timer
 
-        day_names = ', '.join(config.get('days', [])) or 'every day'
-        logger.debug(f"Weekly automation {automation_id} scheduled for {time_str} on {day_names} (in {delay:.0f}s)")
-
-    def _next_weekly_occurrence(self, hour, minute, days):
-        """Find the next datetime matching one of the given weekday abbreviations."""
-        day_map = {'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6}
-        allowed = {day_map[d] for d in days if d in day_map}
-        if not allowed:
-            allowed = set(range(7))  # no days selected = every day
-
-        now = datetime.now()
-        for offset in range(8):  # check today + next 7 days
-            candidate = now + timedelta(days=offset)
-            if candidate.weekday() in allowed:
-                target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
-                if target > now:
-                    return target
-        # Fallback: tomorrow (shouldn't happen with 8-day scan)
-        return now.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
+        logger.debug(f"{label} automation {automation_id} scheduled (in {delay:.0f}s)")
 
     # --- Then Actions (notifications + signals) ---
 
@@ -960,6 +1072,8 @@ class AutomationEngine:
         """Send message via Telegram Bot API."""
         bot_token = config.get('bot_token', '').strip()
         chat_id = config.get('chat_id', '').strip()
+        thread_id = config.get('thread_id', '').strip()
+        
         if not bot_token or not chat_id:
             raise ValueError("Bot token and chat ID are required for Telegram")
 
@@ -968,9 +1082,16 @@ class AutomationEngine:
         for key, value in variables.items():
             message = message.replace('{' + key + '}', value)
 
+        payload = {"chat_id": chat_id, "text": message, "parse_mode": "HTML"}
+        if thread_id:
+            try:
+                payload["message_thread_id"] = int(thread_id)
+            except ValueError:
+                pass  # invalid — fall back to main chat
+
         resp = requests.post(
             f'https://api.telegram.org/bot{bot_token}/sendMessage',
-            json={"chat_id": chat_id, "text": message, "parse_mode": "HTML"},
+            json=payload,
             timeout=10,
         )
         data = resp.json() if resp.status_code == 200 else {}

core/blocklist/__init__.py

@@ -0,0 +1,37 @@
+"""Artist / album / track blocklist (the "proper" blacklist).
+
+Distinct from ``download_blacklist`` (which skips one bad source file from one
+Soulseek peer — untouched here). This blocklist bans an ARTIST, ALBUM, or
+TRACK from being acquired, keyed by metadata-source IDs (Spotify / iTunes /
+Deezer / MusicBrainz) so a ban survives a source switch.
+
+Phase 1 enforces at the single ``add_to_wishlist`` chokepoint: every
+auto-acquisition path (watchlist, discography backfill, repair, manual
+wishlist add) funnels through it, so one guard covers them all.
+
+- ``matching`` — the pure decision core (no DB, no I/O): build an index from
+  blocklist rows, ask whether a candidate is blocked, with artist→album→track
+  cascade.
+"""
+
+from core.blocklist.matching import (
+    ENTITY_ALBUM,
+    ENTITY_ARTIST,
+    ENTITY_TRACK,
+    ENTITY_TYPES,
+    SOURCE_ID_FIELDS,
+    BlocklistIndex,
+    build_index,
+    candidate_block_reason,
+)
+
+__all__ = [
+    "ENTITY_ARTIST",
+    "ENTITY_ALBUM",
+    "ENTITY_TRACK",
+    "ENTITY_TYPES",
+    "SOURCE_ID_FIELDS",
+    "BlocklistIndex",
+    "build_index",
+    "candidate_block_reason",
+]

core/blocklist/backfill.py

@@ -0,0 +1,50 @@
+"""Cross-source ID backfill for blocklist entries.
+
+When a user blocks an item, the modal gives us the ID for ONE source (the one
+they searched). For the ban to survive a source switch, we resolve the OTHER
+sources' IDs too — matching the blocked artist/album/track by name on each
+source and taking a confident hit.
+
+The resolution is kept pure + injected so it tests without a network: callers
+pass a ``resolvers`` map ``{source: fn(entity_type, name, parent_name) -> id |
+None}``. ``core/blocklist/runtime.py`` wires the real metadata clients.
+
+Honest about fragility (acknowledged in design): artist matching is reliable,
+album/track cross-source matching is best-effort (editions, common titles), so
+a resolver returning None just leaves that source unmatched — the artist
+name-fallback in matching.py covers artist gaps; album/track gaps mean that
+ban only applies on sources where an ID resolved.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Callable, Dict, Optional
+
+from core.blocklist.matching import SOURCE_ID_FIELDS
+
+
+def resolve_missing_ids(
+    entry: Dict[str, Any],
+    resolvers: Dict[str, Callable[..., Optional[str]]],
+) -> Dict[str, str]:
+    """Return ``{id_column: resolved_id}`` for the sources currently missing an
+    ID on ``entry``. Never raises — a resolver that errors is skipped."""
+    out: Dict[str, str] = {}
+    entity_type = entry.get("entity_type")
+    name = entry.get("name")
+    parent = entry.get("parent_name")
+    if not entity_type or not name:
+        return out
+    for source, col in SOURCE_ID_FIELDS.items():
+        if entry.get(col):
+            continue  # already known
+        fn = resolvers.get(source)
+        if not fn:
+            continue
+        try:
+            rid = fn(entity_type, name, parent)
+        except Exception:
+            rid = None
+        if rid:
+            out[col] = str(rid)
+    return out

core/blocklist/matching.py

@@ -0,0 +1,128 @@
+"""Pure blocklist matching — no DB, no I/O, fully unit-testable.
+
+The brain of the blocklist: given the stored blocklist rows and a candidate
+track being considered for the wishlist, decide whether it's blocked.
+
+Design decisions (per Boulder):
+- **ID-keyed.** Each row carries the candidate's IDs in up to four metadata
+  sources. A candidate is matched against the SAME source it came in on
+  (the wishlist payload carries active-source IDs), so a Deezer-numeric id
+  can't collide with an iTunes-numeric id of a different entity.
+- **Cascade.** Blocking an artist blocks their albums + tracks; blocking an
+  album blocks its tracks. The candidate carries its own artist/album/track
+  IDs, so the check walks track → album → artist and blocks on the first hit.
+- **Name fallback for ARTISTS only.** A blocked artist also matches by
+  case-folded name — this covers the window before the background ID-backfill
+  has resolved the active source's id. Albums/tracks do NOT fall back to name
+  (common titles like "Greatest Hits" would false-positive across artists);
+  they rely on IDs, which backfill fills in.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
+
+ENTITY_ARTIST = "artist"
+ENTITY_ALBUM = "album"
+ENTITY_TRACK = "track"
+ENTITY_TYPES = (ENTITY_ARTIST, ENTITY_ALBUM, ENTITY_TRACK)
+
+# Blocklist-row column → the metadata source it belongs to.
+SOURCE_ID_FIELDS = {
+    "spotify": "spotify_id",
+    "itunes": "itunes_id",
+    "deezer": "deezer_id",
+    "musicbrainz": "musicbrainz_id",
+}
+
+
+def _norm(text: Any) -> str:
+    return str(text or "").strip().casefold()
+
+
+@dataclass
+class _TypeIndex:
+    # per-source set of blocked ids, plus a case-folded name set (artists only)
+    ids: Dict[str, Set[str]] = field(default_factory=lambda: {s: set() for s in SOURCE_ID_FIELDS})
+    names: Set[str] = field(default_factory=set)
+
+    def hit(self, source: Optional[str], entity_id: Any, name: Any, use_name: bool) -> bool:
+        if entity_id and source in self.ids and str(entity_id) in self.ids[source]:
+            return True
+        if use_name and name and _norm(name) in self.names:
+            return True
+        return False
+
+
+@dataclass
+class BlocklistIndex:
+    """Membership index built once per scan from the blocklist rows."""
+    artists: _TypeIndex = field(default_factory=_TypeIndex)
+    albums: _TypeIndex = field(default_factory=_TypeIndex)
+    tracks: _TypeIndex = field(default_factory=_TypeIndex)
+
+    @property
+    def is_empty(self) -> bool:
+        for ti in (self.artists, self.albums, self.tracks):
+            if ti.names or any(ti.ids.values()):
+                return False
+        return True
+
+
+def build_index(rows: Iterable[Dict[str, Any]]) -> BlocklistIndex:
+    """Build a BlocklistIndex from blocklist DB rows.
+
+    Each row needs ``entity_type``, ``name``, and the source id columns
+    (``spotify_id`` / ``itunes_id`` / ``deezer_id`` / ``musicbrainz_id``).
+    Unknown entity types are ignored."""
+    idx = BlocklistIndex()
+    by_type = {ENTITY_ARTIST: idx.artists, ENTITY_ALBUM: idx.albums, ENTITY_TRACK: idx.tracks}
+    for row in rows or []:
+        ti = by_type.get((row.get("entity_type") or "").strip().lower())
+        if ti is None:
+            continue
+        for source, col in SOURCE_ID_FIELDS.items():
+            val = row.get(col)
+            if val:
+                ti.ids[source].add(str(val))
+        name = row.get("name")
+        if name:
+            ti.names.add(_norm(name))
+    return idx
+
+
+def candidate_block_reason(
+    index: BlocklistIndex,
+    *,
+    source: Optional[str],
+    track_id: Any = None,
+    track_name: Any = None,
+    album_id: Any = None,
+    album_name: Any = None,
+    artists: Optional[List[Dict[str, Any]]] = None,
+) -> Optional[Tuple[str, str]]:
+    """Return ``(entity_type, label)`` for the first cascade hit, else None.
+
+    ``source`` is the metadata source the candidate IDs came from (the wishlist
+    payload's provider). ``artists`` is a list of ``{'id', 'name'}`` dicts.
+    Order matters only for the returned reason — any hit blocks."""
+    if index.is_empty:
+        return None
+
+    # Track level — id only (names too ambiguous to ban across artists).
+    if index.tracks.hit(source, track_id, track_name, use_name=False):
+        return (ENTITY_TRACK, str(track_name or track_id or "track"))
+
+    # Album level — id only.
+    if index.albums.hit(source, album_id, album_name, use_name=False):
+        return (ENTITY_ALBUM, str(album_name or album_id or "album"))
+
+    # Artist level — id OR case-folded name (safe + covers the backfill window).
+    for artist in artists or []:
+        a_id = artist.get("id") if isinstance(artist, dict) else None
+        a_name = artist.get("name") if isinstance(artist, dict) else artist
+        if index.artists.hit(source, a_id, a_name, use_name=True):
+            return (ENTITY_ARTIST, str(a_name or a_id or "artist"))
+
+    return None

core/blocklist/runtime.py

@@ -0,0 +1,82 @@
+"""Wire real metadata clients to the blocklist backfill resolvers.
+
+Resolves a blocked item's ID on each metadata source by searching that source
+for the name and taking a confidently name-matched hit. Confidence = exact
+significant-token match (drops articles/punctuation) so we never hang a wrong
+ID on an entry. Albums/tracks additionally require the parent artist to match
+when both sides expose one.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Callable, Dict, Optional
+
+from utils.logging_config import get_logger
+
+logger = get_logger("blocklist.runtime")
+
+_STOP = {"the", "a", "an", "feat", "ft", "featuring", "with"}
+
+
+def _tokens(text: Any) -> frozenset:
+    words = re.sub(r"[^a-z0-9]+", " ", str(text or "").lower()).split()
+    return frozenset(w for w in words if w not in _STOP)
+
+
+def _name_of(obj: Any) -> str:
+    if isinstance(obj, dict):
+        return str(obj.get("name") or obj.get("title") or "")
+    return str(getattr(obj, "name", None) or getattr(obj, "title", None) or "")
+
+
+def _id_of(obj: Any) -> Optional[str]:
+    val = obj.get("id") if isinstance(obj, dict) else getattr(obj, "id", None)
+    return str(val) if val else None
+
+
+def _confident(result_name: str, want_name: str) -> bool:
+    rt, wt = _tokens(result_name), _tokens(want_name)
+    return bool(rt) and rt == wt
+
+
+def _make_resolver(source: str) -> Callable[..., Optional[str]]:
+    def resolve(entity_type: str, name: str, parent_name: Optional[str] = None) -> Optional[str]:
+        from core.metadata.registry import get_client_for_source
+        client = get_client_for_source(source)
+        if not client:
+            return None
+        method = {
+            "artist": "search_artists",
+            "album": "search_albums",
+            "track": "search_tracks",
+        }.get(entity_type)
+        fn = getattr(client, method, None) if method else None
+        if not fn:
+            return None
+        try:
+            results = fn(name, limit=5) or []
+        except Exception as e:
+            logger.debug("%s %s search failed for %r: %s", source, entity_type, name, e)
+            return None
+        for r in results:
+            if not _confident(_name_of(r), name):
+                continue
+            # For album/track, also require the artist to line up when known.
+            if entity_type in ("album", "track") and parent_name:
+                artists = (r.get("artists") if isinstance(r, dict) else getattr(r, "artists", None)) or []
+                cand_artists = " ".join(
+                    a.get("name", "") if isinstance(a, dict) else str(a) for a in artists)
+                if _tokens(parent_name) and not (_tokens(parent_name) & _tokens(cand_artists)):
+                    continue
+            rid = _id_of(r)
+            if rid:
+                return rid
+        return None
+
+    return resolve
+
+
+def build_resolvers() -> Dict[str, Callable[..., Optional[str]]]:
+    """Source→resolver map for core.blocklist.backfill.resolve_missing_ids."""
+    return {s: _make_resolver(s) for s in ("spotify", "itunes", "deezer", "musicbrainz")}

core/boot_phase.py

@@ -0,0 +1,27 @@
+"""Boot-phase guard for non-blocking container startup.
+
+While the gunicorn worker is importing ``web_server`` (module-level client and
+worker initialization), external provider API probes must not block startup.
+Network validation is deferred until ``mark_boot_complete()`` runs at the end
+of that import pass.
+"""
+
+from __future__ import annotations
+
+import threading
+
+_boot_lock = threading.Lock()
+_boot_active = True
+
+
+def is_boot_phase() -> bool:
+    """Return True while module import must avoid blocking provider API calls."""
+    with _boot_lock:
+        return _boot_active
+
+
+def mark_boot_complete() -> None:
+    """End the boot phase — provider clients may perform network probes again."""
+    global _boot_active
+    with _boot_lock:
+        _boot_active = False

core/connection_detect.py

@@ -0,0 +1,257 @@
+"""Network detection — lifted from web_server.py.
+
+Body is byte-identical to the original. Pure stdlib + requests, no
+web_server-specific globals or runtime state.
+"""
+import ipaddress
+import logging
+import platform
+import socket
+import subprocess
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+import requests
+
+logger = logging.getLogger(__name__)
+
+
+def run_detection(server_type):
+    """
+    Performs comprehensive network detection for a given server type (plex, jellyfin, slskd).
+    This implements the same scanning logic as the GUI's detection threads.
+    """
+    logger.info(f"Running comprehensive detection for {server_type}...")
+    
+    def get_network_info():
+        """Get comprehensive network information with subnet detection"""
+        try:
+            # Get local IP using socket method
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+            s.connect(("8.8.8.8", 80))
+            local_ip = s.getsockname()[0]
+            s.close()
+            
+            # Try to get actual subnet mask
+            try:
+                if platform.system() == "Windows":
+                    # Windows: Use netsh to get subnet info
+                    result = subprocess.run(['netsh', 'interface', 'ip', 'show', 'config'], 
+                                          capture_output=True, text=True, timeout=3)
+                    # Parse output for subnet mask (simplified)
+                    subnet_mask = "255.255.255.0"  # Default fallback
+                else:
+                    # Linux/Mac: Try to parse network interfaces
+                    result = subprocess.run(['ip', 'route', 'show'], 
+                                          capture_output=True, text=True, timeout=3)
+                    subnet_mask = "255.255.255.0"  # Default fallback
+            except:
+                subnet_mask = "255.255.255.0"  # Default /24
+            
+            # Calculate network range
+            network = ipaddress.IPv4Network(f"{local_ip}/{subnet_mask}", strict=False)
+            return str(network.network_address), str(network.netmask), local_ip, network
+            
+        except Exception as e:
+            # Fallback to original method
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+            s.connect(("8.8.8.8", 80))
+            local_ip = s.getsockname()[0]
+            s.close()
+            
+            # Default to /24 network
+            network = ipaddress.IPv4Network(f"{local_ip}/24", strict=False)
+            return str(network.network_address), "255.255.255.0", local_ip, network
+
+    def test_plex_server(ip, port=32400):
+        """Test if a Plex server is running at the given IP and port"""
+        try:
+            url = f"http://{ip}:{port}/web/index.html"
+            response = requests.get(url, timeout=2, allow_redirects=True)
+            
+            # Check for Plex-specific indicators
+            if response.status_code == 200:
+                # Check if it's actually Plex
+                if 'plex' in response.text.lower() or 'X-Plex' in str(response.headers):
+                    return f"http://{ip}:{port}"
+                    
+                # Also try the API endpoint
+                api_url = f"http://{ip}:{port}/identity"
+                api_response = requests.get(api_url, timeout=1)
+                if api_response.status_code == 200 and 'MediaContainer' in api_response.text:
+                    return f"http://{ip}:{port}"
+
+        except Exception as e:
+            logger.debug("plex probe %s: %s", ip, e)
+        return None
+
+    def test_jellyfin_server(ip, port=8096):
+        """Test if a Jellyfin server is running at the given IP and port"""
+        try:
+            # Try the system info endpoint first
+            url = f"http://{ip}:{port}/System/Info"
+            response = requests.get(url, timeout=2, allow_redirects=True)
+            
+            if response.status_code == 200:
+                # Check if response contains Jellyfin-specific content
+                if 'jellyfin' in response.text.lower() or 'ServerName' in response.text:
+                    return f"http://{ip}:{port}"
+            
+            # Also try the web interface
+            web_url = f"http://{ip}:{port}/web/index.html"
+            web_response = requests.get(web_url, timeout=1)
+            if web_response.status_code == 200 and 'jellyfin' in web_response.text.lower():
+                return f"http://{ip}:{port}"
+
+        except Exception as e:
+            logger.debug("jellyfin probe %s: %s", ip, e)
+        return None
+
+    def test_slskd_server(ip, port=5030):
+        """Test if a slskd server is running at the given IP and port"""
+        try:
+            # slskd specific API endpoint
+            url = f"http://{ip}:{port}/api/v0/session"
+            response = requests.get(url, timeout=2)
+            
+            # slskd returns 401 when not authenticated, which is still a valid response
+            if response.status_code in [200, 401]:
+                return f"http://{ip}:{port}"
+
+        except Exception as e:
+            logger.debug("slskd probe %s: %s", ip, e)
+        return None
+
+    def test_navidrome_server(ip, port=4533):
+        """Test if a Navidrome server is running at the given IP and port"""
+        try:
+            # Try Navidrome's ping endpoint (part of Subsonic API)
+            url = f"http://{ip}:{port}/rest/ping"
+            response = requests.get(url, timeout=2, params={
+                'u': 'test',  # Dummy username for ping test
+                'v': '1.16.1',  # API version
+                'c': 'soulsync',  # Client name
+                'f': 'json'  # Response format
+            })
+
+            # Navidrome should respond even with invalid credentials for ping
+            if response.status_code in [200, 401, 403]:
+                try:
+                    data = response.json()
+                    # Check for Subsonic/Navidrome API response structure
+                    if 'subsonic-response' in data:
+                        return f"http://{ip}:{port}"
+                except Exception as e:
+                    logger.debug("navidrome json parse: %s", e)
+
+            # Also try the web interface
+            web_url = f"http://{ip}:{port}/"
+            web_response = requests.get(web_url, timeout=2)
+            if web_response.status_code == 200 and 'navidrome' in web_response.text.lower():
+                return f"http://{ip}:{port}"
+
+        except Exception as e:
+            logger.debug("navidrome probe %s: %s", ip, e)
+        return None
+
+    try:
+        network_addr, netmask, local_ip, network = get_network_info()
+        
+        # Select the appropriate test function
+        test_functions = {
+            'plex': test_plex_server,
+            'jellyfin': test_jellyfin_server,
+            'navidrome': test_navidrome_server,
+            'slskd': test_slskd_server
+        }
+        
+        test_func = test_functions.get(server_type)
+        if not test_func:
+            return None
+        
+        # Priority 1: Test localhost first
+        logger.debug(f"Testing localhost for {server_type}...")
+        localhost_result = test_func("localhost")
+        if localhost_result:
+            logger.info(f"Found {server_type} at localhost!")
+            return localhost_result
+        
+        # Priority 1.5: In Docker, try Docker host IP
+        import os
+        if os.path.exists('/.dockerenv'):
+            logger.info(f"Docker detected, testing Docker host for {server_type}...")
+            try:
+                # Try host.docker.internal (Windows/Mac)
+                host_result = test_func("host.docker.internal")
+                if host_result:
+                    logger.info(f"Found {server_type} at Docker host!")
+                    return host_result.replace("host.docker.internal", "localhost")  # Convert back to localhost for config
+                
+                # Try Docker bridge gateway (Linux)
+                gateway_result = test_func("172.17.0.1")
+                if gateway_result:
+                    logger.info(f"Found {server_type} at Docker gateway!")
+                    return gateway_result.replace("172.17.0.1", "localhost")  # Convert back to localhost for config
+            except Exception as e:
+                logger.error(f"Docker host detection failed: {e}")
+        
+        # Priority 2: Test local IP
+        logger.debug(f"Testing local IP {local_ip} for {server_type}...")
+        local_result = test_func(local_ip)
+        if local_result:
+            logger.info(f"Found {server_type} at {local_ip}!")
+            return local_result
+        
+        # Priority 3: Test common IPs (router gateway, etc.)
+        common_ips = [
+            local_ip.rsplit('.', 1)[0] + '.1',  # Typical gateway
+            local_ip.rsplit('.', 1)[0] + '.2',  # Alternative gateway
+            local_ip.rsplit('.', 1)[0] + '.100', # Common static IP
+        ]
+        
+        logger.debug(f"Testing common IPs for {server_type}...")
+        for ip in common_ips:
+            logger.info(f"  Checking {ip}...")
+            result = test_func(ip)
+            if result:
+                logger.info(f"Found {server_type} at {ip}!")
+                return result
+        
+        # Priority 4: Scan the network range (limited to reasonable size)
+        network_hosts = list(network.hosts())
+        if len(network_hosts) > 50:
+            # Limit scan to reasonable size for performance
+            step = max(1, len(network_hosts) // 50)
+            network_hosts = network_hosts[::step]
+        
+        logger.debug(f"Scanning network range for {server_type} ({len(network_hosts)} hosts)...")
+        
+        # Use ThreadPoolExecutor for concurrent scanning (limited for web context)
+        with ThreadPoolExecutor(max_workers=5) as executor:
+            # Submit all tasks
+            future_to_ip = {executor.submit(test_func, str(ip)): str(ip) 
+                           for ip in network_hosts}
+            
+            try:
+                for future in as_completed(future_to_ip):
+                    ip = future_to_ip[future]
+                    try:
+                        result = future.result()
+                        if result:
+                            logger.info(f"Found {server_type} at {ip}!")
+                            # Cancel all pending futures before returning
+                            for f in future_to_ip:
+                                if not f.done():
+                                    f.cancel()
+                            return result
+                    except Exception as e:
+                        logger.error(f"Error testing {ip}: {e}")
+                        continue
+            except Exception as e:
+                logger.error(f"Error in concurrent scanning: {e}")
+        
+        logger.warning(f"No {server_type} server found on network")
+        return None
+        
+    except Exception as e:
+        logger.error(f"Error during {server_type} detection: {e}")
+        return None

core/connection_test.py

@@ -0,0 +1,502 @@
+"""Service connection test — lifted from web_server.py.
+
+The function body is byte-identical to the original. download_orchestrator,
+qobuz_enrichment_worker, hydrabase_client, docker_resolve_url, and
+docker_resolve_path are injected at runtime because they live in
+web_server.py and are constructed there.
+"""
+import logging
+import os
+
+import requests
+
+from config.settings import config_manager
+from core.jellyfin_client import JellyfinClient
+from core.metadata.registry import get_primary_source
+from core.plex_client import PlexClient
+from core.spotify_client import SpotifyClient
+from core.tidal_client import TidalClient
+from utils.async_helpers import run_async
+
+logger = logging.getLogger(__name__)
+
+
+def _get_metadata_fallback_source():
+    """Mirror of web_server._get_metadata_fallback_source — delegates to registry."""
+    return get_primary_source()
+
+
+# Injected at runtime via init().
+download_orchestrator = None
+qobuz_enrichment_worker = None
+hydrabase_client = None
+docker_resolve_url = None
+docker_resolve_path = None
+
+
+def init(
+    download_orchestrator_obj,
+    qobuz_worker,
+    hydrabase_client_obj,
+    docker_resolve_url_fn,
+    docker_resolve_path_fn,
+):
+    """Bind web_server-side helpers/globals so the lifted body can resolve them."""
+    global download_orchestrator, qobuz_enrichment_worker, hydrabase_client
+    global docker_resolve_url, docker_resolve_path
+    download_orchestrator = download_orchestrator_obj
+    qobuz_enrichment_worker = qobuz_worker
+    hydrabase_client = hydrabase_client_obj
+    docker_resolve_url = docker_resolve_url_fn
+    docker_resolve_path = docker_resolve_path_fn
+
+
+def run_service_test(service, test_config):
+    """
+    Performs the actual connection test for a given service.
+    This logic is adapted from your ServiceTestThread.
+    It temporarily modifies the config, runs the test, then restores the config.
+    """
+    original_config = {}
+    try:
+        # 1. Save original config for the specific service
+        original_config = config_manager.get(service, {})
+
+        # 2. Temporarily set the new config for the test (with Docker URL resolution)
+        for key, value in test_config.items():
+            # Apply Docker URL resolution for URL/URI fields
+            if isinstance(value, str) and ('url' in key.lower() or 'uri' in key.lower()):
+                value = docker_resolve_url(value)
+            config_manager.set(f"{service}.{key}", value)
+
+        # 3. Run the test with the temporary config
+        if service == "spotify":
+            temp_client = SpotifyClient()
+            
+            # Check if Spotify credentials are configured
+            spotify_config = config_manager.get('spotify', {})
+            spotify_configured = bool(spotify_config.get('client_id') and spotify_config.get('client_secret'))
+            
+            if temp_client.is_authenticated():
+                 # Determine which source is active
+                 if temp_client.is_spotify_authenticated():
+                     return True, "Spotify connection successful!"
+                 else:
+                     # Spotify-Free (no-auth) metadata path: officially unauthenticated,
+                     # but the no-creds source is selected and available. Report it as the
+                     # working source rather than the generic Deezer/Discogs/iTunes fallback.
+                     try:
+                         spotify_free_available = temp_client.is_spotify_metadata_available()
+                     except Exception:
+                         spotify_free_available = False
+                     if spotify_free_available:
+                         return True, "Spotify (no-auth) connection successful!"
+                     # Using a different fallback metadata source
+                     fb_src = _get_metadata_fallback_source()
+                     fallback_name = 'Deezer' if fb_src == 'deezer' else 'Discogs' if fb_src == 'discogs' else 'iTunes'
+                     if spotify_configured:
+                         return True, f"{fallback_name} connection successful! (Spotify configured but not authenticated)"
+                     else:
+                         return True, f"{fallback_name} connection successful! (Spotify not configured)"
+            else:
+                 return False, "Music service authentication failed. Check credentials and complete OAuth flow in browser if prompted."
+        elif service == "tidal":
+            temp_client = TidalClient()
+            if temp_client.is_authenticated():
+                user_info = temp_client.get_user_info()
+                username = user_info.get('display_name', 'Tidal User') if user_info else 'Tidal User'
+                return True, f"Tidal connection successful! Connected as: {username}"
+            else:
+                return False, "Tidal authentication failed. Please use the 'Authenticate' button and complete the flow in your browser."
+        elif service == "plex":
+            temp_client = PlexClient()
+            if temp_client.is_connected():
+                return True, f"Successfully connected to Plex server: {temp_client.server.friendlyName}"
+            else:
+                return False, "Could not connect to Plex. Check URL and Token."
+        elif service == "jellyfin":
+            temp_client = JellyfinClient()
+            if temp_client.is_connected():
+                # FIX: Check if server_info exists before accessing it.
+                server_name = "Unknown Server"
+                if hasattr(temp_client, 'server_info') and temp_client.server_info:
+                    server_name = temp_client.server_info.get('ServerName', 'Unknown Server')
+                return True, f"Successfully connected to Jellyfin server: {server_name}"
+            else:
+                return False, "Could not connect to Jellyfin. Check URL and API Key."
+        elif service == "navidrome":
+            # Test Navidrome connection using Subsonic API
+            base_url = test_config.get('base_url', '')
+            username = test_config.get('username', '')
+            password = test_config.get('password', '')
+
+            if not all([base_url, username, password]):
+                return False, "Missing Navidrome URL, username, or password."
+
+            try:
+                import hashlib
+                import random
+                import string
+
+                # Generate salt and token for Subsonic API authentication
+                salt = ''.join(random.choices(string.ascii_letters + string.digits, k=6))
+                token = hashlib.md5((password + salt).encode()).hexdigest()
+
+                # Test ping endpoint
+                url = f"{base_url.rstrip('/')}/rest/ping"
+                response = requests.get(url, params={
+                    'u': username,
+                    't': token,
+                    's': salt,
+                    'v': '1.16.1',
+                    'c': 'soulsync',
+                    'f': 'json'
+                }, timeout=5)
+
+                if response.status_code == 200:
+                    data = response.json()
+                    if data.get('subsonic-response', {}).get('status') == 'ok':
+                        server_version = data.get('subsonic-response', {}).get('version', 'Unknown')
+                        return True, f"Successfully connected to Navidrome server (v{server_version})"
+                    else:
+                        error = data.get('subsonic-response', {}).get('error', {})
+                        return False, f"Navidrome authentication failed: {error.get('message', 'Unknown error')}"
+                else:
+                    return False, f"Could not connect to Navidrome server (HTTP {response.status_code})"
+
+            except Exception as e:
+                return False, f"Navidrome connection error: {str(e)}"
+        elif service == "soulsync":
+            transfer_path = docker_resolve_path(config_manager.get('soulseek.transfer_path', './Transfer'))
+            if os.path.isdir(transfer_path):
+                # Quick check — count a few audio files to confirm it's a music folder
+                audio_exts = {'.mp3', '.flac', '.ogg', '.opus', '.m4a', '.aac', '.wav'}
+                count = 0
+                found_enough = False
+                for _root, _dirs, files in os.walk(transfer_path):
+                    for f in files:
+                        if os.path.splitext(f)[1].lower() in audio_exts:
+                            count += 1
+                            if count >= 10:
+                                found_enough = True
+                                break
+                    if found_enough:
+                        break
+                return True, f"SoulSync standalone ready! Output folder: {transfer_path}" + (f" ({count}+ audio files)" if count > 0 else " (empty)")
+            else:
+                return False, f"Output folder not found: {transfer_path}"
+        elif service == "soulseek":
+            if download_orchestrator is None:
+                return False, "Download orchestrator failed to initialize. Check server logs for startup errors."
+
+            # Test the orchestrator's configured download source (not just Soulseek)
+            download_mode = config_manager.get('download_source.mode', 'hybrid')
+
+            if run_async(download_orchestrator.check_connection()):
+                # Success message based on active mode
+                mode_messages = {
+                    'soulseek': "Successfully connected to Soulseek network via slskd.",
+                    'youtube': "YouTube download source ready.",
+                    'tidal': "Tidal download source ready.",
+                    'qobuz': "Qobuz download source ready.",
+                    'hifi': "HiFi download source ready.",
+                    'deezer_dl': "Deezer download source ready.",
+                    'amazon': "Amazon download source ready.",
+                    'lidarr': "Lidarr download source ready.",
+                    'soundcloud': "SoundCloud download source ready.",
+                    'torrent': "Torrent download source ready.",
+                    'usenet': "Usenet download source ready.",
+                    'hybrid': "Download sources ready (Hybrid mode)."
+                }
+                message = mode_messages.get(download_mode, "Download source connected.")
+                return True, message
+            else:
+                # Failure message based on active mode
+                mode_errors = {
+                    'soulseek': "slskd is not connected to the Soulseek network. Check slskd status and credentials.",
+                    'youtube': "YouTube download source not available.",
+                    'tidal': "Tidal download source not available. Check authentication.",
+                    'qobuz': "Qobuz download source not available. Check authentication.",
+                    'hifi': "HiFi download source not available. Public API instances may be down.",
+                    'deezer_dl': "Deezer download source not available. Check authentication.",
+                    'amazon': "Amazon download source not available.",
+                    'lidarr': "Lidarr download source not available. Check Lidarr URL and API key.",
+                    'soundcloud': "SoundCloud download source not available.",
+                    'torrent': "Torrent download source not available. Check Prowlarr and torrent client settings.",
+                    'usenet': "Usenet download source not available. Check Prowlarr and usenet client settings.",
+                    'hybrid': "Could not connect to download sources. Check configuration."
+                }
+                error = mode_errors.get(download_mode, "Download source connection failed.")
+                return False, error
+        elif service == "listenbrainz":
+            token = test_config.get('token', '')
+
+            if not token:
+                return False, "Missing ListenBrainz user token."
+
+            try:
+                # Test ListenBrainz API by validating the token
+                custom_base = test_config.get('base_url', '').rstrip('/')
+                if custom_base:
+                    if not custom_base.endswith('/1'):
+                        custom_base += '/1'
+                    lb_api_base = custom_base
+                else:
+                    lb_api_base = "https://api.listenbrainz.org/1"
+                url = f"{lb_api_base}/validate-token"
+                headers = {
+                    'Authorization': f'Token {token}'
+                }
+                response = requests.get(url, headers=headers, timeout=5)
+
+                if response.status_code == 200:
+                    data = response.json()
+                    if data.get('valid'):
+                        username = data.get('user_name', 'Unknown')
+                        return True, f"Successfully connected to ListenBrainz! Connected as: {username}"
+                    else:
+                        return False, "Invalid ListenBrainz token."
+                elif response.status_code == 401:
+                    return False, "Invalid ListenBrainz token (unauthorized)."
+                else:
+                    return False, f"Could not connect to ListenBrainz (HTTP {response.status_code})"
+
+            except Exception as e:
+                return False, f"ListenBrainz connection error: {str(e)}"
+        elif service == "acoustid":
+            api_key = test_config.get('api_key', '')
+
+            if not api_key:
+                return False, "Missing AcoustID API key."
+
+            try:
+                from core.acoustid_client import AcoustIDClient, CHROMAPRINT_AVAILABLE, ACOUSTID_AVAILABLE, FPCALC_PATH
+
+                if not ACOUSTID_AVAILABLE:
+                    return False, "pyacoustid library not installed. Run: pip install pyacoustid"
+
+                client = AcoustIDClient()
+
+                # Override the cached API key with the test config key
+                client._api_key = api_key
+
+                # Check chromaprint/fpcalc availability
+                if CHROMAPRINT_AVAILABLE and FPCALC_PATH:
+                    fingerprint_status = f"fpcalc ready: {FPCALC_PATH}"
+                elif CHROMAPRINT_AVAILABLE:
+                    fingerprint_status = "Fingerprint backend available"
+                else:
+                    fingerprint_status = "fpcalc not found (will auto-download on first use)"
+
+                # Validate API key with test request
+                success, message = client.test_api_key()
+                if success:
+                    return True, f"AcoustID API key is valid! {fingerprint_status}"
+                else:
+                    return False, f"{message}. {fingerprint_status}"
+            except Exception as e:
+                return False, f"AcoustID test error: {str(e)}"
+        elif service == "lastfm":
+            api_key = test_config.get('api_key', '')
+
+            if not api_key:
+                return False, "Missing Last.fm API key."
+
+            try:
+                from core.lastfm_client import LastFMClient
+                client = LastFMClient(api_key=api_key)
+                if client.validate_api_key():
+                    return True, "Successfully connected to Last.fm!"
+                else:
+                    return False, "Invalid Last.fm API key."
+            except Exception as e:
+                return False, f"Last.fm connection error: {str(e)}"
+        elif service == "genius":
+            access_token = test_config.get('access_token', '')
+
+            if not access_token:
+                return False, "Missing Genius access token."
+
+            try:
+                from core.genius_client import GeniusClient
+                client = GeniusClient(access_token=access_token)
+                if client.validate_token():
+                    return True, "Successfully connected to Genius!"
+                else:
+                    return False, "Invalid Genius access token."
+            except Exception as e:
+                return False, f"Genius connection error: {str(e)}"
+        elif service == "usenet_client":
+            client_type = (config_manager.get('usenet_client.type', '') or '').strip().lower()
+            url = config_manager.get('usenet_client.url', '')
+            if not url:
+                return False, "Usenet client URL is required."
+            if not client_type:
+                return False, "Pick a usenet client (SABnzbd or NZBGet)."
+            try:
+                from core.usenet_clients import adapter_for_type as _usenet_adapter_for_type
+                adapter = _usenet_adapter_for_type(client_type)
+                if adapter is None:
+                    return False, f"Unknown usenet client type: {client_type}"
+                if not adapter.is_configured():
+                    if client_type == "sabnzbd":
+                        return False, "SABnzbd needs both URL and API key."
+                    return False, "NZBGet needs URL, username, and password."
+                if run_async(adapter.check_connection()):
+                    return True, f"Connected to {client_type}"
+                return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
+            except Exception as e:
+                return False, f"Usenet client connection error: {str(e)}"
+        elif service == "torrent_client":
+            client_type = (config_manager.get('torrent_client.type', '') or '').strip().lower()
+            url = config_manager.get('torrent_client.url', '')
+            if not url:
+                return False, "Torrent client URL is required."
+            if not client_type:
+                return False, "Pick a torrent client (qBittorrent, Transmission, or Deluge)."
+            try:
+                from core.torrent_clients import adapter_for_type
+                adapter = adapter_for_type(client_type)
+                if adapter is None:
+                    return False, f"Unknown torrent client type: {client_type}"
+                if not adapter.is_configured():
+                    return False, "Torrent client missing required credentials."
+                if run_async(adapter.check_connection()):
+                    return True, f"Connected to {client_type}"
+                return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
+            except Exception as e:
+                return False, f"Torrent client connection error: {str(e)}"
+        elif service == "prowlarr":
+            url = config_manager.get('prowlarr.url', '')
+            api_key = config_manager.get('prowlarr.api_key', '')
+            if not url or not api_key:
+                return False, "Prowlarr URL and API key are required."
+            try:
+                import requests as _req
+                resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
+                                headers={'X-Api-Key': api_key}, timeout=10)
+                if resp.ok:
+                    version = resp.json().get('version', '?')
+                    return True, f"Connected to Prowlarr v{version}"
+                return False, f"Prowlarr returned HTTP {resp.status_code}"
+            except Exception as e:
+                return False, f"Prowlarr connection error: {str(e)}"
+        elif service == "lidarr" or service == "lidarr_download":
+            url = config_manager.get('lidarr_download.url', '')
+            api_key = config_manager.get('lidarr_download.api_key', '')
+            if not url or not api_key:
+                return False, "Lidarr URL and API key are required."
+            try:
+                import requests as _req
+                resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
+                                headers={'X-Api-Key': api_key}, timeout=10)
+                if resp.ok:
+                    version = resp.json().get('version', '?')
+                    return True, f"Connected to Lidarr v{version}"
+                return False, f"Lidarr returned HTTP {resp.status_code}"
+            except Exception as e:
+                return False, f"Lidarr connection error: {str(e)}"
+        elif service == "itunes":
+            # Public API — just confirm we can reach it with a cheap search
+            try:
+                storefront = config_manager.get('itunes.storefront', 'US') or 'US'
+                resp = requests.get(
+                    'https://itunes.apple.com/search',
+                    params={'term': 'beatles', 'limit': 1, 'country': storefront, 'media': 'music'},
+                    timeout=5,
+                )
+                if resp.ok and resp.json().get('resultCount', 0) >= 0:
+                    return True, f"iTunes Search API reachable (storefront: {storefront})"
+                return False, f"iTunes returned HTTP {resp.status_code}"
+            except Exception as e:
+                return False, f"iTunes connection error: {str(e)}"
+        elif service == "deezer":
+            # Public API — anon search works without credentials
+            try:
+                resp = requests.get(
+                    'https://api.deezer.com/search/artist',
+                    params={'q': 'beatles', 'limit': 1},
+                    timeout=5,
+                )
+                if resp.ok and isinstance(resp.json(), dict):
+                    return True, "Deezer Public API reachable"
+                return False, f"Deezer returned HTTP {resp.status_code}"
+            except Exception as e:
+                return False, f"Deezer connection error: {str(e)}"
+        elif service == "discogs":
+            token = test_config.get('token', '') or config_manager.get('discogs.token', '')
+            if not token:
+                return False, "Missing Discogs personal token."
+            try:
+                resp = requests.get(
+                    'https://api.discogs.com/database/search',
+                    params={'q': 'beatles', 'per_page': 1},
+                    headers={'Authorization': f'Discogs token={token}', 'User-Agent': 'SoulSync/1.0'},
+                    timeout=10,
+                )
+                if resp.ok:
+                    return True, "Discogs API reachable with provided token"
+                if resp.status_code == 401:
+                    return False, "Discogs token rejected (HTTP 401)"
+                return False, f"Discogs returned HTTP {resp.status_code}"
+            except Exception as e:
+                return False, f"Discogs connection error: {str(e)}"
+        elif service == "qobuz":
+            try:
+                if qobuz_enrichment_worker and qobuz_enrichment_worker.client and qobuz_enrichment_worker.client.is_authenticated():
+                    return True, "Qobuz client authenticated"
+                return False, "Qobuz not authenticated. Provide email/password or user auth token."
+            except Exception as e:
+                return False, f"Qobuz connection error: {str(e)}"
+        elif service == "hydrabase":
+            try:
+                if hydrabase_client and hydrabase_client.is_connected():
+                    return True, "Hydrabase connected"
+                return False, "Hydrabase not connected. Configure URL + API key and click Connect."
+            except Exception as e:
+                return False, f"Hydrabase connection error: {str(e)}"
+        elif service == "musicbrainz":
+            try:
+                from core.metadata.registry import get_musicbrainz_client
+                mb = get_musicbrainz_client()
+                results = mb.search_artists("radiohead", limit=1)
+                if results:
+                    return True, "MusicBrainz reachable"
+                return False, "MusicBrainz returned no results — may be rate-limited or unreachable."
+            except Exception as e:
+                return False, f"MusicBrainz connection error: {str(e)}"
+        elif service == "soundcloud":
+            # Anonymous SoundCloud has no auth, so "test" really means
+            # "is yt-dlp installed and can it reach SoundCloud right now."
+            # This mirrors the /api/soundcloud/status check.
+            try:
+                from core.soundcloud_client import SoundcloudClient
+                sc = SoundcloudClient()
+                if not sc.is_available():
+                    return False, "SoundCloud unavailable — yt-dlp not installed."
+                # Run a tiny live probe via asyncio so the dashboard test
+                # gives a meaningful pass/fail.
+                import asyncio
+                reachable = asyncio.new_event_loop().run_until_complete(sc.check_connection())
+                if reachable:
+                    return True, "SoundCloud reachable (anonymous)"
+                return False, "SoundCloud unreachable — search probe failed. Try again."
+            except Exception as e:
+                return False, f"SoundCloud connection error: {str(e)}"
+        return False, "Unknown service."
+    except AttributeError as e:
+        # This specifically catches the error you reported for Jellyfin
+        if "'JellyfinClient' object has no attribute 'server_info'" in str(e):
+            return False, "Connection failed. Please check your Jellyfin URL and API Key."
+        else:
+            return False, f"An unexpected error occurred: {e}"
+    except Exception as e:
+        import traceback
+        traceback.print_exc()
+        return False, str(e)
+    finally:
+        # 4. CRITICAL: Restore the original config
+        if original_config:
+            for key, value in original_config.items():
+                config_manager.set(f"{service}.{key}", value)
+            logger.debug(f"Restored original config for '{service}' after test.")

core/credentials/store.py

@@ -0,0 +1,85 @@
+"""Named, switchable service-credential sets — pure logic (Phase 0 foundation).
+
+Today every auth service (Spotify, Tidal, Deezer, Qobuz, Plex, Jellyfin,
+Navidrome) holds ONE credential set in config, and clients are global singletons
+built from that single slot. This module is the groundwork for letting an admin
+save MULTIPLE named credential sets per service ("pills") that each profile can
+switch between, without anyone but the admin creating them.
+
+Kept PURE — service registry, payload validation, and active-set selection,
+free of DB/Flask so it's unit-testable. Encrypted storage lives in MusicDatabase
+(service_credentials / profile_service_credentials tables); runtime client
+resolution + UI come in later phases. Nothing here changes existing behaviour;
+it's dormant capability until wired.
+"""
+
+from __future__ import annotations
+
+# Services that support multiple named credential sets, mapped to the payload
+# keys that MUST be present for a set to be usable. Extra keys (OAuth tokens,
+# redirect URIs, quality prefs) are allowed and preserved — these are only the
+# minimum required to validate a set the admin is saving.
+SERVICE_CREDENTIAL_SCHEMA = {
+    'spotify':   ('client_id', 'client_secret'),
+    'tidal':     ('access_token', 'refresh_token'),
+    'deezer':    ('arl',),
+    'qobuz':     ('user_auth_token',),
+    'plex':      ('base_url', 'token'),
+    'jellyfin':  ('base_url', 'api_key'),
+    'navidrome': ('base_url', 'username', 'password'),
+}
+
+SUPPORTED_SERVICES = frozenset(SERVICE_CREDENTIAL_SCHEMA)
+
+
+def is_supported_service(service: str) -> bool:
+    """True when the service supports named credential sets."""
+    return service in SERVICE_CREDENTIAL_SCHEMA
+
+
+def validate_credential_payload(service: str, payload):
+    """Return ``(ok, missing_keys)`` for a credential set.
+
+    Valid when every required key for the service is present and truthy. An
+    unknown service is invalid with no missing list (caller should reject it
+    as unsupported, not as "incomplete").
+    """
+    required = SERVICE_CREDENTIAL_SCHEMA.get(service)
+    if required is None:
+        return False, []
+    if not isinstance(payload, dict):
+        return False, list(required)
+
+    def _present(v):
+        # Whitespace-only strings count as missing — they'd otherwise save a
+        # blank secret that fails confusingly at the real service later.
+        return bool(v.strip()) if isinstance(v, str) else bool(v)
+
+    missing = [k for k in required if not _present(payload.get(k))]
+    return (not missing), missing
+
+
+def pick_active_credential(credentials, selected_id):
+    """From ``credentials`` (a list of dicts each carrying ``id``), return the
+    one whose id == ``selected_id``.
+
+    Returns None when there's no selection OR the selected id isn't present —
+    i.e. a stale pointer whose credential set was deleted. The caller then
+    falls back to the global/admin default, so a deleted set never breaks a
+    profile. Pure + stale-safe.
+    """
+    if not selected_id:
+        return None
+    for cred in credentials or []:
+        if cred.get('id') == selected_id:
+            return cred
+    return None
+
+
+__all__ = [
+    'SERVICE_CREDENTIAL_SCHEMA',
+    'SUPPORTED_SERVICES',
+    'is_supported_service',
+    'validate_credential_payload',
+    'pick_active_credential',
+]

core/database_update_health.py

@@ -0,0 +1,82 @@
+"""Stall detection for the database-update job.
+
+The DB updater keeps a single in-memory state dict whose ``status`` is set to
+``running`` at start and only flipped to ``finished``/``error`` by the worker's
+completion/error callbacks. If the worker thread hangs — e.g. a media-server API
+call with no timeout, a DB lock — those callbacks never fire, so ``status`` stays
+``running`` forever and the UI shows a frozen progress bar with no way to recover
+(GitHub #859).
+
+This module is the single, *pure* decision for "is a running job stalled?". It
+takes the state dict plus the current wall-clock time and a timeout, and answers
+yes/no — no DB, no globals, no clock of its own. That keeps it unit-testable and
+lets the watchdog wiring in web_server.py stay a thin call. The job carries a
+``last_progress_at`` epoch timestamp that the start path and every progress/phase
+callback bump; staleness is simply "running, and that timestamp is older than the
+timeout".
+"""
+
+from __future__ import annotations
+
+from typing import Any, Mapping
+
+# 5 minutes with zero forward progress = presumed hung. A healthy scan ticks
+# progress (per-artist) far more often than this even for large libraries, so
+# the timeout won't false-positive a slow-but-working run.
+DEFAULT_STALL_TIMEOUT_SECONDS = 300
+
+
+def is_db_update_stalled(
+    state: Mapping[str, Any],
+    now: float,
+    timeout_seconds: float = DEFAULT_STALL_TIMEOUT_SECONDS,
+) -> bool:
+    """Return True when the job is ``running`` but has made no progress within
+    ``timeout_seconds``.
+
+    Conservative by design — it only ever reports a stall it can prove:
+    - Only a ``running`` job can stall (idle/finished/error never do).
+    - With no usable ``last_progress_at`` timestamp we cannot judge, so we return
+      False rather than risk killing a job we have no clock for.
+    - A non-positive timeout is treated as "disabled" (never stalls).
+    """
+    if not isinstance(state, Mapping):
+        return False
+    if state.get("status") != "running":
+        return False
+    if timeout_seconds is None or timeout_seconds <= 0:
+        return False
+    last = state.get("last_progress_at")
+    if not last:
+        return False
+    try:
+        elapsed = float(now) - float(last)
+    except (TypeError, ValueError):
+        return False
+    return elapsed >= float(timeout_seconds)
+
+
+def stalled_error_message(state: Mapping[str, Any], now: float) -> str:
+    """Build a clear, human-facing message for a stalled job, including how long
+    it has been silent and the phase it died in."""
+    last = state.get("last_progress_at") if isinstance(state, Mapping) else None
+    phase = state.get("phase") if isinstance(state, Mapping) else None
+    try:
+        secs = int(float(now) - float(last)) if last else 0
+    except (TypeError, ValueError):
+        secs = 0
+    msg = "Update appears stuck — no progress"
+    if secs > 0:
+        msg += f" for {secs}s"
+    if phase:
+        msg += f" (last phase: {phase})"
+    msg += (". The worker may be hung on the media server. Start a new update "
+            "to try again, or restart SoulSync if it keeps stalling.")
+    return msg
+
+
+__all__ = [
+    "DEFAULT_STALL_TIMEOUT_SECONDS",
+    "is_db_update_stalled",
+    "stalled_error_message",
+]

core/database_update_worker.py

@@ -41,7 +41,19 @@ class DatabaseUpdateWorker:
         self.database_path = database_path
         self.full_refresh = full_refresh
         self.should_stop = False
-        
+
+        # Track ids of rows newly INSERTED this run (not updates). The web
+        # layer reads this to gap-fill embedded provider IDs for the new files
+        # (auto-reconcile), so newly-added music contributes its
+        # Spotify/MusicBrainz/etc. ids without a manual backfill.
+        self._new_track_ids = set()
+
+        # Optional callback(worker) run as the FINAL scan phase, immediately
+        # before the 'finished' signal — so the auto-reconcile is inside the
+        # scan's running window (automations/UI treat it as a normal phase and
+        # wait for it). Injected by the web layer (which owns path resolution).
+        self.post_scan_hook = None
+
         # Statistics tracking
         self.processed_artists = 0
         self.processed_albums = 0
@@ -79,7 +91,26 @@ class DatabaseUpdateWorker:
                 callback(*args)
             except Exception as e:
                 logger.error(f"Error in callback for {signal_name}: {e}")
-    
+
+    def _emit_finished(self, *args):
+        """Run the post-scan hook (auto-reconcile) as the final phase, THEN
+        emit 'finished'.
+
+        Running the hook before 'finished' keeps the scan's status at
+        'running' through the reconcile, so every caller (automations that
+        poll for completion, the dashboard card, the Tools page) treats it as
+        a normal scan phase and waits for it — rather than seeing 'finished'
+        and missing the tail. Best-effort: a hook failure never blocks the
+        completion signal.
+        """
+        if self.post_scan_hook:
+            try:
+                self.post_scan_hook(self)
+            except Exception as e:
+                logger.warning(f"post-scan hook failed (non-fatal): {e}")
+        self._emit_signal('finished', *args)
+
+
     def connect_callback(self, signal_name: str, callback: Callable):
         """Connect a callback for progress notifications."""
         self.callbacks.setdefault(signal_name, []).append(callback)
@@ -146,7 +177,7 @@ class DatabaseUpdateWorker:
                                 logger.info(f"Merged {merged} duplicate artists")
                         except Exception as e:
                             logger.warning(f"Could not merge duplicate artists: {e}")
-                    self._emit_signal('finished', 0, 0, 0, 0, 0)
+                    self._emit_finished(0, 0, 0, 0, 0)
                     return
                 logger.info(f"Incremental update: Found {len(artists_to_process)} artists to process")
             
@@ -230,7 +261,7 @@ class DatabaseUpdateWorker:
             self.removed_tracks = removal.get('tracks_removed', 0) if removal else 0
 
             # Emit final results
-            self._emit_signal('finished',
+            self._emit_finished(
                 self.processed_artists,
                 self.processed_albums,
                 self.processed_tracks,
@@ -331,7 +362,7 @@ class DatabaseUpdateWorker:
                         f"{self.processed_albums} albums, {self.processed_tracks} new tracks, "
                         f"{stale_removed} stale tracks removed")
 
-            self._emit_signal('finished',
+            self._emit_finished(
                 self.processed_artists,
                 self.processed_albums,
                 self.processed_tracks,
@@ -407,9 +438,14 @@ class DatabaseUpdateWorker:
                 logger.error(f"Could not connect to {self.server_type} server — check URL, credentials, and network (Docker users: use container name or host.docker.internal instead of host IP)")
                 return []
             
-            # Check for music library (Plex-specific check)
-            if self.server_type == "plex" and not self.media_client.music_library:
-                logger.error("No music library found in Plex")
+            # Check for music library (Plex-specific check). Routes
+            # through ``is_fully_configured`` so all-libraries mode (in
+            # which ``music_library`` is None but ``_all_libraries_mode``
+            # is True) counts as configured. Pre-fix this bailed out on
+            # the bare music_library None check, silently aborting the
+            # deep scan for any all-libraries-mode user.
+            if self.server_type == "plex" and not self.media_client.is_fully_configured():
+                logger.error("No music library configured in Plex")
                 return []
             
             # Check if database has enough content for incremental updates (server-specific)
@@ -875,6 +911,8 @@ class DatabaseUpdateWorker:
                                             track_success = self.database.insert_or_update_media_track(track, album_id, artist_id, server_source=self.server_type)
                                             if track_success:
                                                 total_processed_tracks += 1
+                                                if track_success == 'inserted':
+                                                    self._new_track_ids.add(str(track.ratingKey))
                                                 logger.debug(f"Processed new track: {track.title}")
                                         except Exception as e:
                                             logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
@@ -1046,8 +1084,8 @@ class DatabaseUpdateWorker:
                             batch + [self.server_type])
                         cascade_album_ids.update(row[0] for row in cursor.fetchall())
                     removed_album_ids -= cascade_album_ids
-            except Exception:
-                pass  # If this optimization fails, double-delete is harmless
+            except Exception as e:
+                logger.debug("cascade album cleanup optimization: %s", e)
 
         if not removed_artist_ids and not removed_album_ids:
             logger.info("Removal detection: no stale content found")
@@ -1084,24 +1122,31 @@ class DatabaseUpdateWorker:
             return []
     
     def _get_recent_albums_plex(self) -> List:
-        """Get recently added and updated albums from Plex"""
+        """Get recently added and updated albums from Plex.
+
+        Routes through ``PlexClient.get_recently_added_albums`` and
+        ``get_recently_updated_albums`` so the all-libraries mode union
+        works (pre-fix this reached ``self.media_client.music_library.X``
+        directly which crashed when music_library is None in all-
+        libraries mode).
+        """
         all_recent_content = []
-        
+
         try:
-            # Get recently added albums (up to 400 to catch more recent content)  
-            try:
-                recently_added = self.media_client.music_library.recentlyAdded(libtype='album', maxresults=400)
+            # Get recently added albums (up to 400 to catch more recent content)
+            recently_added = self.media_client.get_recently_added_albums(maxresults=400, libtype='album')
+            if recently_added:
                 all_recent_content.extend(recently_added)
                 logger.info(f"Found {len(recently_added)} recently added albums")
-            except:
-                # Fallback to general recently added
-                recently_added = self.media_client.music_library.recentlyAdded(maxresults=400)
-                all_recent_content.extend(recently_added)
-                logger.info(f"Found {len(recently_added)} recently added items (mixed types)")
-            
+            else:
+                # Fallback to mixed-type recents.
+                recently_added = self.media_client.get_recently_added_albums(maxresults=400, libtype=None)
+                all_recent_content.extend(recently_added or [])
+                logger.info(f"Found {len(recently_added or [])} recently added items (mixed types)")
+
             # Get recently updated albums (catches metadata corrections)
             try:
-                recently_updated = self.media_client.music_library.search(sort='updatedAt:desc', libtype='album', limit=400)
+                recently_updated = self.media_client.get_recently_updated_albums(limit=400)
                 # Remove duplicates (items that are both recently added and updated)
                 added_keys = {getattr(item, 'ratingKey', None) for item in all_recent_content}
                 unique_updated = [item for item in recently_updated if getattr(item, 'ratingKey', None) not in added_keys]
@@ -1332,6 +1377,8 @@ class DatabaseUpdateWorker:
                                             skipped_count += 1
                                         elif track_success:
                                             track_count += 1
+                                            if track_success == 'inserted':
+                                                self._new_track_ids.add(track_id_str)
                                     except Exception as e:
                                         logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
 

core/db_integrity.py

@@ -0,0 +1,153 @@
+"""SQLite integrity + safe-backup helpers.
+
+Born out of a real incident: a WAL-mode DB got corrupted (most likely an
+interrupted write during a hard restart), and because the backup routine
+(a) never checked integrity and (b) rotated the oldest backup out by mtime,
+every rolling backup ended up being a faithful copy of the already-corrupt
+file — so when recovery was needed, all snapshots were poisoned.
+
+This module makes that impossible:
+
+* ``quick_check(path)`` / ``is_healthy(path)`` — fast read-only integrity probe.
+* ``safe_backup(...)`` — verifies the SOURCE is healthy before copying, uses the
+  SQLite Online Backup API, then verifies the RESULT. A corrupt source never
+  produces (or keeps) a backup.
+* ``prune_backups(...)`` — rotation that NEVER deletes the most recent
+  *verified-healthy* backup, even to honor the max-count, so a run of bad
+  backups can't evict your last good one.
+
+Pure-ish: only touches sqlite3 + the filesystem paths it's given; no Flask, no
+app globals. Unit-testable with real (and deliberately-corrupted) temp DBs.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import sqlite3
+from typing import Optional
+
+logger = logging.getLogger("db_integrity")
+
+
+def _close_quietly(conn) -> None:
+    """Best-effort close; a failure to close during cleanup must not mask the
+    real error we're handling, but we log it rather than swallow silently."""
+    if conn is None:
+        return
+    try:
+        conn.close()
+    except Exception as e:  # noqa: BLE001 — cleanup path, real error already in flight
+        logger.debug("db_integrity: connection close failed: %s", e)
+
+
+class DBIntegrityError(Exception):
+    """Raised when a database fails its integrity check."""
+
+
+def quick_check(db_path: str, *, timeout: float = 30.0) -> str:
+    """Run ``PRAGMA quick_check`` read-only and return its first result row.
+
+    Returns ``'ok'`` for a healthy DB, otherwise the first error line. Raises
+    ``DBIntegrityError`` if the file can't even be opened/read (malformed
+    header, I/O error) — i.e. unambiguously bad.
+    """
+    if not os.path.exists(db_path):
+        raise DBIntegrityError(f"Database file not found: {db_path}")
+    conn = None
+    try:
+        conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=timeout)
+        row = conn.execute("PRAGMA quick_check(1)").fetchone()
+        return (row[0] if row else "no result")
+    except sqlite3.DatabaseError as e:
+        # malformed header / disk image malformed / disk I/O error
+        raise DBIntegrityError(f"{db_path}: {e}") from e
+    finally:
+        _close_quietly(conn)
+
+
+def is_healthy(db_path: str, *, timeout: float = 30.0) -> bool:
+    """True iff the DB opens and ``quick_check`` reports 'ok'. Never raises."""
+    try:
+        return quick_check(db_path, timeout=timeout) == "ok"
+    except DBIntegrityError:
+        return False
+
+
+def safe_backup(src_path: str, dst_path: str, *, verify_source: bool = True,
+                verify_result: bool = True) -> None:
+    """Back up ``src_path`` to ``dst_path`` via the SQLite Online Backup API,
+    refusing to produce a backup from (or keep a backup of) a corrupt DB.
+
+    Raises ``DBIntegrityError`` and removes any partial ``dst_path`` when the
+    source is unhealthy (``verify_source``) or the produced backup fails its
+    own check (``verify_result``). On success ``dst_path`` is a verified-good
+    copy.
+    """
+    if verify_source and not is_healthy(src_path):
+        # Don't immortalize corruption — surface it so the caller can alert
+        # and, crucially, NOT rotate out the existing good backups.
+        raise DBIntegrityError(
+            f"Refusing to back up: source database failed integrity check ({src_path})"
+        )
+
+    src = dst = None
+    try:
+        src = sqlite3.connect(src_path)
+        dst = sqlite3.connect(dst_path)
+        src.backup(dst)
+    finally:
+        _close_quietly(dst)
+        _close_quietly(src)
+
+    if verify_result and not is_healthy(dst_path):
+        # The copy itself came out bad — discard it rather than keep a dud.
+        try:
+            os.remove(dst_path)
+        except OSError:
+            pass
+        raise DBIntegrityError(
+            f"Backup produced a corrupt file and was discarded ({dst_path})"
+        )
+
+
+def prune_backups(backup_paths, max_keep: int,
+                  health_check=is_healthy) -> list:
+    """Decide which backups to delete to honor ``max_keep`` WITHOUT ever
+    deleting the most-recent verified-healthy backup.
+
+    ``backup_paths`` is an iterable of paths; order does not matter (we sort by
+    mtime). Returns the list of paths that SHOULD be deleted (does not delete
+    them — the caller does the IO, so this stays pure/testable).
+
+    Rule: oldest-first deletion until <= max_keep, but the single newest
+    *healthy* backup is protected and never selected for deletion. So even if
+    the newest few backups are corrupt, the last good snapshot survives.
+    """
+    paths = [p for p in backup_paths]
+    # Newest first.
+    paths.sort(key=lambda p: _safe_mtime(p), reverse=True)
+
+    # Find the newest healthy backup — the one we must never drop.
+    protected: Optional[str] = None
+    for p in paths:
+        if health_check(p):
+            protected = p
+            break
+
+    if len(paths) <= max_keep:
+        return []
+
+    # Delete oldest-first beyond max_keep, but skip the protected one.
+    deletable = [p for p in paths if p != protected]
+    # oldest first among deletable
+    deletable.sort(key=lambda p: _safe_mtime(p))
+    num_to_delete = len(paths) - max_keep
+    return deletable[:num_to_delete]
+
+
+def _safe_mtime(path: str) -> float:
+    try:
+        return os.path.getmtime(path)
+    except OSError:
+        return 0.0

core/debug_info.py

@@ -0,0 +1,413 @@
+"""Debug info endpoint — lifted from web_server.py.
+
+The function bodies are byte-identical to the originals. Module-level
+shims for ``spotify_client`` and ``tidal_client`` (proxies that resolve
+through the metadata registry / runtime client registry) plus injected
+state dicts and helpers let the bodies resolve their original names
+without modification.
+"""
+import logging
+import os
+import platform
+from pathlib import Path
+
+from flask import jsonify, request
+
+from config.settings import config_manager
+from core.metadata.registry import (
+    get_spotify_client,
+    get_primary_source,
+    is_hydrabase_enabled,
+)
+from core.metadata.status import get_spotify_status
+
+logger = logging.getLogger(__name__)
+
+
+class _SpotifyClientProxy:
+    """Resolves the global Spotify client lazily through core.metadata.registry."""
+
+    def __getattr__(self, name):
+        client = get_spotify_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        return get_spotify_client() is not None
+
+
+class _TidalClientProxy:
+    """Resolves the global Tidal client lazily via an injected getter so a
+    Tidal re-auth that rebinds web_server.tidal_client is visible here."""
+
+    def __getattr__(self, name):
+        if _get_tidal_client is None:
+            raise AttributeError(name)
+        client = _get_tidal_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        if _get_tidal_client is None:
+            return False
+        return _get_tidal_client() is not None
+
+
+spotify_client = _SpotifyClientProxy()
+tidal_client = _TidalClientProxy()
+_get_tidal_client = None  # injected via init()
+
+
+# Injected at runtime via init().
+SOULSYNC_VERSION = None
+_DIRECT_RUN = None
+_status_cache = None
+qobuz_enrichment_worker = None
+download_batches = None
+sync_states = None
+youtube_playlist_states = None
+tidal_discovery_states = None
+download_orchestrator = None
+_log_path = None
+_log_dir = None
+app = None
+get_database = None
+
+
+def init(
+    soulsync_version,
+    direct_run,
+    status_cache,
+    qobuz_worker,
+    download_batches_dict,
+    sync_states_dict,
+    youtube_playlist_states_dict,
+    tidal_discovery_states_dict,
+    download_orchestrator_obj,
+    log_path,
+    log_dir,
+    flask_app,
+    get_database_fn,
+    tidal_client_getter,
+):
+    """Bind shared state/helpers from web_server."""
+    global SOULSYNC_VERSION, _DIRECT_RUN, _status_cache, qobuz_enrichment_worker
+    global download_batches, sync_states, youtube_playlist_states
+    global tidal_discovery_states, download_orchestrator, _log_path, _log_dir
+    global app, get_database, _get_tidal_client
+    SOULSYNC_VERSION = soulsync_version
+    _DIRECT_RUN = direct_run
+    _status_cache = status_cache
+    qobuz_enrichment_worker = qobuz_worker
+    download_batches = download_batches_dict
+    sync_states = sync_states_dict
+    youtube_playlist_states = youtube_playlist_states_dict
+    tidal_discovery_states = tidal_discovery_states_dict
+    download_orchestrator = download_orchestrator_obj
+    _log_path = log_path
+    _log_dir = log_dir
+    app = flask_app
+    get_database = get_database_fn
+    _get_tidal_client = tidal_client_getter
+
+
+def _safe_check(fn, default=False):
+    """Safely evaluate a check function, returning default on any error."""
+    try:
+        return fn()
+    except Exception:
+        return default
+
+
+def get_debug_info():
+    """Collect system diagnostics for troubleshooting support requests."""
+    import sys
+    import psutil
+    import time
+    from datetime import timedelta
+
+    log_lines = request.args.get('lines', 20, type=int)
+    log_lines = max(10, min(log_lines, 500))
+    log_source = request.args.get('log', 'app')
+
+    info = {}
+
+    # App info
+    info['version'] = SOULSYNC_VERSION
+    info['os'] = f"{platform.system()} {platform.release()}"
+    info['python'] = sys.version.split()[0]
+    info['docker'] = os.path.exists('/.dockerenv')
+    info['runner'] = 'gunicorn' if not _DIRECT_RUN else 'direct (python web_server.py)'
+
+    # ffmpeg version
+    try:
+        import subprocess
+        result = subprocess.run(['ffmpeg', '-version'], capture_output=True, text=True, timeout=5)
+        first_line = result.stdout.split('\n')[0] if result.stdout else ''
+        # e.g. "ffmpeg version 6.1.1 Copyright ..."
+        info['ffmpeg'] = first_line.split('Copyright')[0].replace('ffmpeg version', '').strip() if first_line else 'installed (version unknown)'
+    except FileNotFoundError:
+        info['ffmpeg'] = 'NOT INSTALLED'
+    except Exception:
+        info['ffmpeg'] = 'unknown'
+
+    # Uptime
+    start_time = getattr(app, 'start_time', time.time())
+    uptime_seconds = time.time() - start_time
+    info['uptime'] = str(timedelta(seconds=int(uptime_seconds)))
+
+    # Paths
+    download_path = config_manager.get('soulseek.download_path', './downloads')
+    transfer_folder = config_manager.get('soulseek.transfer_path', './Transfer')
+    staging_folder = config_manager.get('import.staging_path', '')
+    info['paths'] = {
+        'download_path': download_path,
+        'download_path_exists': os.path.isdir(download_path) if download_path else False,
+        'download_path_writable': os.access(download_path, os.W_OK) if download_path and os.path.isdir(download_path) else False,
+        'transfer_folder': transfer_folder,
+        'transfer_folder_exists': os.path.isdir(transfer_folder) if transfer_folder else False,
+        'transfer_folder_writable': os.access(transfer_folder, os.W_OK) if transfer_folder and os.path.isdir(transfer_folder) else False,
+        'staging_folder': staging_folder,
+        'staging_folder_exists': os.path.isdir(staging_folder) if staging_folder else False,
+    }
+    # Music library paths (Settings > Library)
+    music_paths = config_manager.get('library.music_paths', [])
+    if isinstance(music_paths, list) and music_paths:
+        info['paths']['music_library_paths'] = []
+        for p in music_paths:
+            if p and isinstance(p, str):
+                info['paths']['music_library_paths'].append({
+                    'path': p,
+                    'exists': os.path.isdir(p),
+                })
+    # Music videos directory
+    music_videos_path = config_manager.get('library.music_videos_path', '')
+    if music_videos_path:
+        info['paths']['music_videos_path'] = music_videos_path
+        info['paths']['music_videos_path_exists'] = os.path.isdir(music_videos_path)
+
+    # Services. `_status_cache` only carries 'media_server' and 'soulseek'
+    # (no 'spotify' key) so anything we used to read from `spotify_cache`
+    # silently defaulted to the missing-value fallback — that's the
+    # "music_source: unknown" bug. Spotify status now comes from the
+    # canonical `get_spotify_status` accessor; primary metadata source
+    # comes from `get_primary_source` (which already accounts for the
+    # auth-fallback chain — Spotify drops back to Deezer when not
+    # authenticated).
+    media_server_cache = _status_cache.get('media_server', {})
+    soulseek_cache = _status_cache.get('soulseek', {})
+    spotify_status = _safe_check(lambda: get_spotify_status(spotify_client=spotify_client), default={})
+    if not isinstance(spotify_status, dict):
+        spotify_status = {}
+    info['services'] = {
+        'music_source': _safe_check(get_primary_source, default='unknown') or 'unknown',
+        'spotify_connected': bool(spotify_status.get('connected', False)),
+        'spotify_rate_limited': bool(spotify_status.get('rate_limited', False)),
+        'media_server_type': media_server_cache.get('type', 'none'),
+        'media_server_connected': media_server_cache.get('connected', False),
+        'soulseek_connected': soulseek_cache.get('connected', False),
+        'download_source': config_manager.get('download_source.mode', 'hybrid'),
+        'tidal_connected': _safe_check(lambda: bool(tidal_client and tidal_client.is_authenticated())),
+        'qobuz_connected': _safe_check(lambda: bool(qobuz_enrichment_worker and qobuz_enrichment_worker.client and qobuz_enrichment_worker.client.is_authenticated())),
+        'hydrabase_connected': _safe_check(is_hydrabase_enabled),
+        # YouTube is URL-based via yt-dlp — no auth, always reachable as
+        # long as the binary is installed. Surfaced so the debug dump
+        # documents that YouTube is one of the available download sources
+        # rather than implying it doesn't exist.
+        'youtube_available': True,
+    }
+    # HiFi instance count — separate from connection status because each
+    # instance is its own independent endpoint with its own auth state.
+    info['services']['hifi_instance_count'] = _safe_check(
+        lambda: len(get_database().get_hifi_instances()), default=0
+    )
+    # Always-available public metadata sources (no auth, no per-user
+    # connection state). Listed so the debug dump reflects the full
+    # metadata surface SoulSync queries from, not just the auth-gated ones.
+    info['services']['always_available_metadata_sources'] = [
+        'deezer', 'itunes', 'musicbrainz',
+    ]
+
+    # Enrichment workers
+    workers = {}
+    worker_names = ['musicbrainz', 'audiodb', 'deezer', 'spotify', 'itunes', 'lastfm', 'genius', 'discogs', 'tidal', 'qobuz']
+    for name in worker_names:
+        paused_key = f'{name}_enrichment_paused'
+        workers[name] = 'paused' if config_manager.get(paused_key, False) else 'active'
+    info['enrichment_workers'] = workers
+
+    # Library stats — use same method as dashboard (filters by active server)
+    try:
+        db = get_database()
+        lib_stats = db.get_database_info_for_server()
+        info['library'] = {
+            'artists': lib_stats.get('artists', 0),
+            'albums': lib_stats.get('albums', 0),
+            'tracks': lib_stats.get('tracks', 0),
+        }
+    except Exception:
+        info['library'] = {'artists': 0, 'albums': 0, 'tracks': 0}
+
+    # Watchlist count
+    try:
+        db = get_database()
+        info['watchlist_count'] = db.get_watchlist_count()
+    except Exception:
+        info['watchlist_count'] = 0
+
+    # Wishlist pending count
+    try:
+        db = get_database()
+        info['wishlist_count'] = db.get_wishlist_count()
+    except Exception:
+        info['wishlist_count'] = 0
+
+    # Automation count
+    try:
+        db = get_database()
+        automations = db.get_automations()
+        info['automations'] = {
+            'total': len(automations),
+            'enabled': len([a for a in automations if a.get('enabled', False)]),
+        }
+    except Exception:
+        info['automations'] = {'total': 0, 'enabled': 0}
+
+    # Active downloads & syncs (use list() snapshots to avoid RuntimeError from concurrent mutation)
+    try:
+        active_downloads = len([bid for bid, bd in list(download_batches.items()) if bd.get('phase') == 'downloading'])
+    except Exception:
+        active_downloads = 0
+    active_syncs = 0
+    try:
+        for _pid, ss in list(sync_states.items()):
+            if ss.get('status') == 'syncing':
+                active_syncs += 1
+        for _uh, st in list(youtube_playlist_states.items()):
+            if st.get('phase') == 'syncing':
+                active_syncs += 1
+        for _pid, st in list(tidal_discovery_states.items()):
+            if st.get('phase') == 'syncing':
+                active_syncs += 1
+    except Exception as e:
+        logger.debug("count active syncs failed: %s", e)
+    info['active_downloads'] = active_downloads
+    info['active_syncs'] = active_syncs
+
+    # Config settings relevant to troubleshooting
+    source_mode = config_manager.get('download_source.mode', 'hybrid')
+    info['config'] = {
+        'source_mode': source_mode,
+        'quality_profile': config_manager.get('download_source.quality_profile', 'default'),
+        'organization_template': config_manager.get('organization.folder_template', ''),
+        'post_processing_enabled': config_manager.get('post_processing.enabled', True),
+        'acoustid_enabled': bool(config_manager.get('acoustid.api_key', '')),
+        'auto_scan_enabled': config_manager.get('watchlist.auto_scan', False),
+        'm3u_export_enabled': config_manager.get('m3u.enabled', False),
+        'log_level': config_manager.get('logging.level', 'INFO'),
+        'primary_metadata_source': config_manager.get('metadata.fallback_source', 'deezer'),
+        'lossy_copy_enabled': config_manager.get('post_processing.lossy_copy.enabled', False),
+        'lossy_copy_format': config_manager.get('post_processing.lossy_copy.format', 'mp3'),
+        'lossy_copy_bitrate': config_manager.get('post_processing.lossy_copy.bitrate', 320),
+        'allow_duplicate_tracks': config_manager.get('library.allow_duplicate_tracks', False),
+        'replace_lower_quality': config_manager.get('import.replace_lower_quality', False),
+        'auto_import_enabled': config_manager.get('import.auto_import_enabled', False),
+    }
+    # Hybrid source priority order
+    if source_mode == 'hybrid':
+        info['config']['hybrid_sources'] = config_manager.get('download_source.hybrid_order', [])
+    # Discogs connection status
+    info['services']['discogs_connected'] = bool(config_manager.get('discogs.token', ''))
+
+    # Download client init failures
+    info['download_client_failures'] = []
+    if download_orchestrator and hasattr(download_orchestrator, '_init_failures'):
+        info['download_client_failures'] = download_orchestrator._init_failures
+    elif not download_orchestrator:
+        info['download_client_failures'] = ['ALL (orchestrator failed to initialize)']
+
+    # API rate monitor — current calls/min, 24h totals, peaks, rate limit events
+    try:
+        from core.api_call_tracker import api_call_tracker
+        # `get_spotify_status` is already imported at module level. A
+        # local re-import here would make Python treat the name as a
+        # function-scoped local for the WHOLE body, breaking the lambda
+        # at the top of get_debug_info that closes over the module-level
+        # binding (Python 3.12 NameError on free variables).
+        rates = api_call_tracker.get_all_rates()
+        info['api_rates'] = rates
+        # Rich 24h debug summary with peaks, totals, per-endpoint breakdown, events
+        info['api_debug_summary'] = api_call_tracker.get_debug_summary()
+        # Spotify rate limit details
+        spotify_status = get_spotify_status(spotify_client=spotify_client)
+        rl_info = spotify_status.get('rate_limit')
+        if spotify_status.get('rate_limited') and rl_info:
+            info['spotify_rate_limit'] = {
+                'active': True,
+                'remaining_seconds': rl_info.get('remaining_seconds', 0),
+                'retry_after': rl_info.get('retry_after', 0),
+                'endpoint': rl_info.get('endpoint', ''),
+                'expires_at': rl_info.get('expires_at', ''),
+            }
+        else:
+            info['spotify_rate_limit'] = {'active': False}
+    except Exception:
+        info['api_rates'] = {}
+        info['api_debug_summary'] = {}
+        info['spotify_rate_limit'] = {'active': False}
+
+    # Database size
+    db_path = os.path.join('database', 'music_library.db')
+    if os.path.exists(db_path):
+        db_size_mb = os.path.getsize(db_path) / (1024 * 1024)
+        info['database_size'] = f"{db_size_mb:.1f} MB"
+    else:
+        info['database_size'] = 'not found'
+
+    # Memory & CPU
+    process = psutil.Process(os.getpid())
+    mem = process.memory_info()
+    info['memory_usage'] = f"{mem.rss / (1024 * 1024):.0f} MB"
+    info['system_memory'] = f"{psutil.virtual_memory().percent}%"
+    try:
+        info['cpu_percent'] = f"{process.cpu_percent(interval=0.1):.1f}%"
+    except Exception:
+        info['cpu_percent'] = 'unknown'
+    info['thread_count'] = process.num_threads()
+
+    # Log lines
+    log_map = {
+        'app': Path(_log_path),
+        'acoustid': _log_dir / 'acoustid.log',
+        'post_processing': _log_dir / 'post_processing.log',
+        'source_reuse': _log_dir / 'source_reuse.log',
+    }
+    log_path = log_map.get(log_source, log_map['app'])
+    info['log_source'] = log_source
+    info['log_lines_requested'] = log_lines
+    info['recent_logs'] = []
+    if os.path.exists(log_path):
+        try:
+            with open(log_path, 'r', encoding='utf-8', errors='replace') as f:
+                lines = f.readlines()
+                info['recent_logs'] = [line.rstrip() for line in lines[-log_lines:]]
+        except Exception:
+            info['recent_logs'] = ['(could not read log file)']
+
+    # Available log files
+    info['available_logs'] = []
+    logs_dir = 'logs'
+    if os.path.isdir(logs_dir):
+        for fname in sorted(os.listdir(logs_dir)):
+            if fname.endswith('.log'):
+                fpath = os.path.join(logs_dir, fname)
+                size_kb = os.path.getsize(fpath) / 1024
+                info['available_logs'].append({
+                    'name': fname.replace('.log', ''),
+                    'file': fname,
+                    'size': f"{size_kb:.0f} KB" if size_kb < 1024 else f"{size_kb/1024:.1f} MB",
+                })
+
+    return jsonify(info)

core/deezer_client.py

@@ -6,7 +6,8 @@ from typing import Dict, List, Optional, Any
 from functools import wraps
 from dataclasses import dataclass
 from utils.logging_config import get_logger
-from core.metadata_cache import get_metadata_cache
+from core.metadata.artist_album_cache import get_cached_artist_album_items, store_artist_album_items
+from core.metadata.cache import get_metadata_cache
 
 logger = get_logger("deezer_client")
 
@@ -45,6 +46,119 @@ def rate_limited(func):
     return wrapper
 
 
+# Pattern matches Deezer's CDN cover/picture URL: a numeric width-x-height
+# segment in the path (e.g. ``/1000x1000-000000-80-0-0.jpg``). Captures
+# both halves so the replacement can use a single dimension and preserve
+# the rest of the path verbatim.
+_DEEZER_CDN_SIZE_PATTERN = re.compile(r'/(\d+)x(\d+)-')
+
+# Maximum size Deezer's CDN serves before returning 403. Verified
+# empirically against multiple albums — 1900 works reliably, 2000+
+# returns Forbidden. CDN serves the source-native size when it's
+# smaller than requested, so asking for 1900 is safe even on albums
+# whose source upload was lower-res (no upscaling, just same bytes).
+_DEEZER_MAX_COVER_SIZE = 1900
+
+
+def _upgrade_deezer_cover_url(url: str, target_size: int = _DEEZER_MAX_COVER_SIZE) -> str:
+    """Rewrite a Deezer CDN cover/picture URL to request a larger size.
+
+    Deezer's API returns ``cover_xl`` / ``picture_xl`` URLs at
+    1000×1000, but the underlying CDN serves up to 1900×1900 by
+    rewriting the size segment in the URL path. This helper does the
+    rewrite — same idea as ``_upgrade_spotify_image_url`` in
+    ``spotify_client`` and the ``mzstatic.com`` size-replacement in
+    ``download_cover_art``.
+
+    Defensive on every input shape:
+    - Empty / None URL → returned as-is
+    - Non-Deezer URL (no ``dzcdn`` host, no size segment) → returned as-is
+    - Already at or above target size → returned as-is (no point rewriting)
+
+    The CDN returns the source-native image bytes when source < target,
+    so asking for 1900 on an album whose source was uploaded at 600
+    just returns the 600-pixel image — no upscaling, no failure.
+    """
+    if not url or 'dzcdn' not in url:
+        return url
+    match = _DEEZER_CDN_SIZE_PATTERN.search(url)
+    if not match:
+        return url
+    current = int(match.group(1))
+    if current >= target_size:
+        return url
+    return _DEEZER_CDN_SIZE_PATTERN.sub(f'/{target_size}x{target_size}-', url, count=1)
+
+
+def _is_full_track_payload(payload: Optional[Dict[str, Any]]) -> bool:
+    """Distinguish a full `/track/<id>` cache hit from partial album-tracks data.
+
+    Three Deezer endpoints feed the per-track cache:
+      - `/track/<id>` — full record, includes both `track_position` AND
+        `contributors` (the multi-artist list the contributors-upgrade
+        path reads).
+      - `/album/<id>/tracks` — partial; includes `track_position` but
+        omits `contributors`.
+      - `/search/track` — minimal; lacks `track_position`.
+
+    Pre-fix `get_track_details` only checked `track_position`, so
+    partial album-tracks payloads were treated as full hits and the
+    contributors-upgrade silently fell back to single-artist tagging
+    whenever an album had been fetched before its individual tracks
+    were post-processed (issue #588).
+
+    `contributors` key presence is the load-bearing distinction —
+    `[]` is a valid value for genuinely single-artist tracks fetched
+    via the per-track endpoint, so test for key membership not
+    truthiness.
+    """
+    if not isinstance(payload, dict):
+        return False
+    return 'track_position' in payload and 'contributors' in payload
+
+
+def resolve_album_track_positions(session, base_url, album_ids, cache=None, sleep_s=0.2):
+    """Build ``{str(track_id): track_position}`` for a set of Deezer album ids.
+
+    Deezer PLAYLIST and SEARCH track objects (and even the album object's embedded
+    ``tracks.data``) omit ``track_position`` — only ``/album/<id>/tracks`` and
+    ``/track/<id>`` carry it. So numbering playlist tracks by their playlist index
+    silently poisons the real album track number, which then rides onto the
+    downloaded file's tag. This resolves the authoritative position per album
+    (cache-first, best-effort — a failed album just isn't in the map)."""
+    import time as _time
+    positions: Dict[str, int] = {}
+    for aid in album_ids:
+        aid = str(aid)
+        at_list = None
+        if cache:
+            try:
+                ct = cache.get_entity('deezer', 'album_tracks', aid)
+                if ct and ct.get('data'):
+                    at_list = ct['data']
+            except Exception:   # noqa: BLE001 - cache is best-effort
+                at_list = None
+        if at_list is None:
+            try:
+                if sleep_s:
+                    _time.sleep(sleep_s)   # respect Deezer rate limits
+                r = session.get(f"{base_url}/album/{aid}/tracks", params={'limit': 500}, timeout=10)
+                if getattr(r, 'ok', False):
+                    at_list = (r.json() or {}).get('data', [])
+                    if cache and at_list is not None:
+                        try:
+                            cache.store_entity('deezer', 'album_tracks', aid, {'data': at_list})
+                        except Exception as _cache_err:   # noqa: BLE001
+                            logger.debug("album_tracks cache store failed for %s: %s", aid, _cache_err)
+            except Exception:   # noqa: BLE001 - never let metadata resolution break the fetch
+                at_list = None
+        for at in (at_list or []):
+            tp = at.get('track_position')
+            if at.get('id') and tp:
+                positions[str(at['id'])] = tp
+    return positions
+
+
 # ==================== Dataclasses (match iTunesClient / SpotifyClient format) ====================
 
 @dataclass
@@ -168,6 +282,7 @@ class Album:
     album_type: str
     image_url: Optional[str] = None
     external_urls: Optional[Dict[str, str]] = None
+    explicit: Optional[bool] = None
 
     @classmethod
     def from_deezer_album(cls, album_data: Dict[str, Any]) -> 'Album':
@@ -199,7 +314,8 @@ class Album:
             total_tracks=album_data.get('nb_tracks', 0),
             album_type=album_type,
             image_url=image_url,
-            external_urls=external_urls if external_urls else None
+            external_urls=external_urls if external_urls else None,
+            explicit=bool(album_data.get('explicit_lyrics', False)),
         )
 
 
@@ -298,39 +414,136 @@ class DeezerClient:
     # can serve as a drop-in fallback metadata source in SpotifyClient.
 
     @rate_limited
-    def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
-        """Search for tracks — returns Track dataclass list (metadata source interface)"""
+    def search_tracks(
+        self,
+        query: str = '',
+        limit: int = 20,
+        *,
+        track: Optional[str] = None,
+        artist: Optional[str] = None,
+        album: Optional[str] = None,
+    ) -> List[Track]:
+        """Search for tracks — returns Track dataclass list (metadata source interface).
+
+        Two call modes:
+
+        1. **Free-text** (`query='Foreigner Dirty White Boy'`) — legacy
+           shape, passes the string straight to Deezer's `q` param.
+           Same behaviour as before, kept for backward compat.
+
+        2. **Field-scoped** (`track='Dirty White Boy', artist='Foreigner'`) —
+           builds Deezer's advanced search syntax (`track:"X" artist:"Y"`).
+           Massively tighter relevance than the free-text path because
+           the API matches each term in the right field instead of
+           anywhere across title / lyrics / artist / album / contributors.
+           Without this, the Deezer ranking buries the canonical track
+           under karaoke / cover / "originally performed by" variants
+           — see issue #534.
+
+        Field-scoped form is used whenever ``track`` or ``artist`` is
+        provided. ``query`` is ignored in that case (the field params
+        are authoritative). When both are missing, falls through to
+        ``query``. The cache key is the constructed query string in
+        either case so the two paths share entries naturally.
+        """
+        # Build the actual API query — advanced syntax when callers pass
+        # field hints, raw query otherwise.
+        used_advanced = bool(track or artist or album)
+        if used_advanced:
+            api_query = self._build_advanced_query(track=track, artist=artist, album=album)
+        else:
+            api_query = query
+
+        if not api_query:
+            return []
+
+        tracks = self._search_tracks_with_query(api_query, limit)
+
+        # Safety net: Deezer's advanced syntax is `artist:"X"`-style
+        # substring match, but in practice it's brittle on artist name
+        # variants ("Foreigner [US]", "The Foreigner", etc.) and on
+        # tracks indexed under non-canonical title spellings. When the
+        # advanced query returns nothing, fall back to a free-text join
+        # so the user sees the prior (less-relevant but non-empty) result
+        # set rather than "No matches" — same behaviour as pre-fix for
+        # this edge case. Caller-side rerank still tightens the result.
+        if not tracks and used_advanced:
+            fallback_parts = [p for p in (track, artist, album) if p]
+            fallback_query = ' '.join(fallback_parts)
+            if fallback_query and fallback_query != api_query:
+                logger.debug(
+                    "[Deezer] Advanced query returned 0 results, falling back "
+                    "to free-text: %r → %r", api_query, fallback_query,
+                )
+                tracks = self._search_tracks_with_query(fallback_query, limit)
+
+        return tracks
+
+    def _search_tracks_with_query(self, api_query: str, limit: int) -> List[Track]:
+        """Cache-aware single API call. Pulled out so the
+        ``search_tracks`` orchestration can call this twice (advanced
+        query → free-text fallback) without duplicating the cache +
+        parse + store dance."""
         cache = get_metadata_cache()
-        cached_results = cache.get_search_results('deezer', 'track', query, limit)
+        cached_results = cache.get_search_results('deezer', 'track', api_query, limit)
         if cached_results is not None:
             tracks = []
             for raw in cached_results:
                 try:
                     tracks.append(Track.from_deezer_track(raw))
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("Track.from_deezer_track cache parse: %s", e)
             if tracks:
                 return tracks
 
-        data = self._api_get('search/track', {'q': query, 'limit': min(limit, 100)})
+        data = self._api_get('search/track', {'q': api_query, 'limit': min(limit, 100)})
         if not data or 'data' not in data:
             return []
 
         tracks = []
         raw_items = []
         for track_data in data['data']:
-            track = Track.from_deezer_track(track_data)
-            tracks.append(track)
+            track_obj = Track.from_deezer_track(track_data)
+            tracks.append(track_obj)
             raw_items.append(track_data)
 
         entries = [(str(td.get('id', '')), td) for td in raw_items if td.get('id')]
         if entries:
             cache.store_entities_bulk('deezer', 'track', entries)
-            cache.store_search_results('deezer', 'track', query, limit,
+            cache.store_search_results('deezer', 'track', api_query, limit,
                                        [str(td.get('id', '')) for td in raw_items if td.get('id')])
 
         return tracks
 
+    @staticmethod
+    def _build_advanced_query(
+        *,
+        track: Optional[str] = None,
+        artist: Optional[str] = None,
+        album: Optional[str] = None,
+    ) -> str:
+        """Compose Deezer's advanced search syntax from field hints.
+
+        Per Deezer's docs:
+        https://developers.deezer.com/api/search
+
+            q=track:"X" artist:"Y" album:"Z"
+
+        Quotes around each value preserve multi-word phrases. Empty
+        fields are skipped. Embedded double-quotes get stripped (no
+        escape mechanism in Deezer's syntax) — rare in practice, but
+        a search for `O"Hara` would otherwise produce a malformed
+        query.
+        """
+        parts = []
+        if track:
+            parts.append(f'track:"{track.replace(chr(34), "")}"')
+        if artist:
+            parts.append(f'artist:"{artist.replace(chr(34), "")}"')
+        if album:
+            parts.append(f'album:"{album.replace(chr(34), "")}"')
+        return ' '.join(parts)
+
     @rate_limited
     def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
         """Search for artists — returns Artist dataclass list (metadata source interface)"""
@@ -341,8 +554,8 @@ class DeezerClient:
             for raw in cached_results:
                 try:
                     artists.append(Artist.from_deezer_artist(raw))
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("Artist.from_deezer_artist cache parse: %s", e)
             if artists:
                 return artists
 
@@ -375,8 +588,8 @@ class DeezerClient:
             for raw in cached_results:
                 try:
                     albums.append(Album.from_deezer_album(raw))
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("Album.from_deezer_album cache parse: %s", e)
             if albums:
                 return albums
 
@@ -403,14 +616,9 @@ class DeezerClient:
         """Get detailed track info — returns Spotify-compatible dict (metadata source interface)"""
         cache = get_metadata_cache()
         cached = cache.get_entity('deezer', 'track', str(track_id))
-        if cached and cached.get('title'):
-            # Search results are cached with minimal data (no track_position).
-            # Only use cache if it has track_position — the key field from /track/{id}.
-            # Search results include 'isrc' and 'release_date' but NOT track_position,
-            # so those fields alone are not sufficient to distinguish full from partial data.
-            if 'track_position' in cached:
-                return self._build_enhanced_track(cached)
-            # Otherwise fall through to fetch full data from API
+        if cached and cached.get('title') and _is_full_track_payload(cached):
+            return self._build_enhanced_track(cached)
+        # Otherwise fall through to fetch full data from API
 
         data = self._api_get(f'track/{track_id}')
         if not data:
@@ -599,6 +807,74 @@ class DeezerClient:
 
         return result
 
+    def get_artist_top_tracks(self, artist_id: str, limit: int = 10) -> List[Dict[str, Any]]:
+        """Return the artist's top tracks in Spotify-compatible dict format.
+
+        Wraps Deezer's `/artist/{id}/top?limit=N`. Returns dicts with the same
+        shape Spotify's `artist_top_tracks` produces — id, name, artists, album
+        (with album_type / total_tracks / release_date / images), duration_ms,
+        track_number, disc_number — so callers don't need to branch on source.
+        """
+        if not artist_id:
+            return []
+        try:
+            limit = max(1, min(int(limit or 10), 100))
+        except (TypeError, ValueError):
+            limit = 10
+
+        data = self._api_get(f'artist/{artist_id}/top', {'limit': limit})
+        if not data or 'data' not in data:
+            return []
+
+        tracks = []
+        for track_data in data['data']:
+            if not isinstance(track_data, dict):
+                continue
+            artist_data = track_data.get('artist') or {}
+            album_data = track_data.get('album') or {}
+
+            # Build images list from any cover sizes Deezer returned for the album
+            images = []
+            if isinstance(album_data, dict):
+                for size_key, dim in [('cover_xl', 1000), ('cover_big', 500),
+                                       ('cover_medium', 250), ('cover_small', 56)]:
+                    if album_data.get(size_key):
+                        images.append({'url': album_data[size_key], 'height': dim, 'width': dim})
+
+            # Deezer `/artist/{id}/top` results don't include record_type on the
+            # nested album object; we don't have a track-count to infer from
+            # either. Default 'album' so the path-builder template variable
+            # always has something to substitute (existing behavior elsewhere).
+            album_payload = {
+                'id': str(album_data.get('id', '')) if isinstance(album_data, dict) else '',
+                'name': album_data.get('title', '') if isinstance(album_data, dict) else '',
+                'album_type': 'album',
+                'images': images,
+                'release_date': '',
+                'total_tracks': 0,
+                'artists': [{'name': artist_data.get('name', '')}] if isinstance(artist_data, dict) else [],
+            }
+
+            tracks.append({
+                'id': str(track_data.get('id', '')),
+                'name': track_data.get('title', ''),
+                'artists': [{
+                    'id': str(artist_data.get('id', '')) if isinstance(artist_data, dict) else '',
+                    'name': artist_data.get('name', '') if isinstance(artist_data, dict) else '',
+                }],
+                'album': album_payload,
+                'duration_ms': (track_data.get('duration') or 0) * 1000,  # Deezer is seconds
+                'popularity': track_data.get('rank', 0),
+                'preview_url': track_data.get('preview'),
+                'external_urls': {'deezer': track_data['link']} if track_data.get('link') else {},
+                'track_number': track_data.get('track_position'),
+                'disc_number': track_data.get('disk_number', 1),
+                'explicit': bool(track_data.get('explicit_lyrics', False)),
+                '_source': 'deezer',
+            })
+
+        return tracks
+
     def get_artist_info(self, artist_id: str) -> Optional[Dict[str, Any]]:
         """Get full artist details — returns Spotify-compatible dict (metadata source interface).
 
@@ -640,17 +916,41 @@ class DeezerClient:
 
         Matches iTunesClient.get_artist_albums() interface.
         Paginates through all results up to the requested limit."""
+        cache = get_metadata_cache()
+        cached_items = get_cached_artist_album_items(cache, 'deezer', artist_id, album_type=album_type, limit=limit)
+        if cached_items:
+            try:
+                requested_types = [t.strip() for t in album_type.split(',')]
+                cached_albums = []
+                for album_data in cached_items:
+                    album = Album.from_deezer_album(album_data)
+                    if album_type != 'album,single':
+                        if album.album_type not in requested_types:
+                            if not (album.album_type == 'ep' and 'single' in requested_types):
+                                continue
+                    cached_albums.append(album)
+                return cached_albums[:limit]
+            except Exception as e:
+                logger.debug("Deezer artist albums cache reuse failed: %s", e)
+
         albums = []
         all_raw = []
         requested_types = [t.strip() for t in album_type.split(',')]
         offset = 0
         page_size = 100  # Deezer API max per request
+        complete = True  # cleared if pagination breaks on a transient/malformed error
 
         while offset < limit:
             fetch_limit = min(page_size, limit - offset)
             data = self._api_get(f'artist/{artist_id}/albums', {'limit': fetch_limit, 'index': offset})
-            if not data or 'data' not in data or len(data['data']) == 0:
+            if not data or 'data' not in data:
+                # Malformed/transient response mid-pagination — what we have is a
+                # PARTIAL discography. Don't cache it as the full list (mirrors the
+                # Spotify truncated-fetch guard). #853 follow-up.
+                complete = False
                 break
+            if len(data['data']) == 0:
+                break  # No more albums — a clean end of pagination.
 
             for album_data in data['data']:
                 all_raw.append(album_data)
@@ -667,7 +967,6 @@ class DeezerClient:
                 break  # Last page
             offset += len(data['data'])
 
-        cache = get_metadata_cache()
         # Deezer's /artist/{id}/albums endpoint doesn't include artist info on each album.
         # Inject it so cached album entities have artist_name for discover page display.
         artist_stub = None
@@ -681,6 +980,11 @@ class DeezerClient:
                 entries.append((str(ad['id']), ad))
         if entries:
             cache.store_entities_bulk('deezer', 'album', entries, skip_if_exists=True)
+        # Only cache the artist→album-LIST when pagination finished cleanly; a
+        # partial list would otherwise serve an incomplete discography until TTL.
+        # (Individual album entities above are complete, so they cache regardless.)
+        if complete:
+            store_artist_album_items(cache, 'deezer', artist_id, all_raw, album_type=album_type, limit=limit)
 
         logger.info(f"Retrieved {len(albums)} albums for artist {artist_id}")
         return albums[:limit]
@@ -842,8 +1146,8 @@ class DeezerClient:
                 try:
                     cache = get_metadata_cache()
                     cache.store_entity('deezer', 'artist', str(result.get('id', '')), result)
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("cache store_entity artist search: %s", e)
                 logger.debug(f"Found artist for query: {artist_name}")
                 return result
 
@@ -887,8 +1191,8 @@ class DeezerClient:
                 try:
                     cache = get_metadata_cache()
                     cache.store_entity('deezer', 'album', str(result.get('id', '')), result)
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("cache store_entity album search: %s", e)
                 logger.debug(f"Found album for query: {artist_name} - {album_title}")
                 return result
 
@@ -932,8 +1236,8 @@ class DeezerClient:
                 try:
                     cache = get_metadata_cache()
                     cache.store_entity('deezer', 'track', str(result.get('id', '')), result)
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("cache store_entity track search: %s", e)
                 logger.debug(f"Found track for query: {artist_name} - {track_title}")
                 return result
 
@@ -965,8 +1269,8 @@ class DeezerClient:
                 # Cache hit with full details (has label = was a get_album response, not just search)
                 logger.debug(f"Cache hit for album {album_id}")
                 return cached
-        except Exception:
-            pass
+        except Exception as e:
+            logger.debug("cache get_entity album: %s", e)
 
         try:
             response = self.session.get(
@@ -984,8 +1288,8 @@ class DeezerClient:
             try:
                 cache = get_metadata_cache()
                 cache.store_entity('deezer', 'album', str(album_id), data)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("cache store_entity album full: %s", e)
             logger.debug(f"Got full album details for ID: {album_id}")
             return data
 
@@ -1013,8 +1317,8 @@ class DeezerClient:
             if cached and cached.get('bpm'):
                 logger.debug(f"Cache hit for track {track_id}")
                 return cached
-        except Exception:
-            pass
+        except Exception as e:
+            logger.debug("cache get_entity track: %s", e)
 
         try:
             response = self.session.get(
@@ -1032,8 +1336,8 @@ class DeezerClient:
             try:
                 cache = get_metadata_cache()
                 cache.store_entity('deezer', 'track', str(track_id), data)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("cache store_entity track full: %s", e)
             logger.debug(f"Got full track details for ID: {track_id}")
             return data
 
@@ -1095,6 +1399,16 @@ class DeezerClient:
 
                 raw_tracks.extend(page_tracks)
 
+            # Real album track positions — playlist tracks don't carry track_position,
+            # so numbering by playlist index would poison the downloaded file's tag.
+            album_ids = {str(t.get('album', {}).get('id')) for t in raw_tracks if t.get('album', {}).get('id')}
+            try:
+                from core.metadata.cache import get_metadata_cache
+                _cache = get_metadata_cache()
+            except Exception:
+                _cache = None
+            track_positions = resolve_album_track_positions(self.session, self.BASE_URL, album_ids, _cache)
+
             # Normalize tracks
             tracks: List[Dict[str, Any]] = []
             for i, t in enumerate(raw_tracks, start=1):
@@ -1106,7 +1420,8 @@ class DeezerClient:
                     'artists': [artist_name],
                     'album': t.get('album', {}).get('title', ''),
                     'duration_ms': t.get('duration', 0) * 1000,
-                    'track_number': i,
+                    # REAL album position; the playlist index is a last resort only.
+                    'track_number': track_positions.get(str(t.get('id'))) or i,
                 })
 
             result = {

core/deezer_download_client.py

@@ -20,7 +20,8 @@ from typing import Any, Dict, List, Optional, Tuple
 
 import requests
 
-from core.soulseek_client import AlbumResult, DownloadStatus, TrackResult
+from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
+from core.quality.source_map import quality_from_deezer, quality_tier_for_source
 from utils.logging_config import get_logger
 
 logger = get_logger("deezer_download")
@@ -79,7 +80,10 @@ def _decrypt_chunk(chunk: bytes, key: bytes) -> bytes:
             ) from exc
 
 
-class DeezerDownloadClient:
+from core.download_plugins.base import DownloadSourcePlugin
+
+
+class DeezerDownloadClient(DownloadSourcePlugin):
     """Deezer download client using ARL token authentication."""
 
     def __init__(self, download_path: str = None):
@@ -89,11 +93,14 @@ class DeezerDownloadClient:
         if download_path is None:
             download_path = config_manager.get('soulseek.download_path', './downloads')
         self.download_path = Path(download_path)
-        self.download_path.mkdir(parents=True, exist_ok=True)
+        try:
+            self.download_path.mkdir(parents=True, exist_ok=True)
+        except OSError as e:
+            logger.warning(f"Could not verify download path {self.download_path}: {e}")
 
-        # Download tracking (same pattern as Tidal/Qobuz/HiFi)
-        self.active_downloads: Dict[str, Dict[str, Any]] = {}
-        self._download_lock = threading.Lock()
+        # Engine reference is populated by set_engine() at registration
+        # time. None until orchestrator wires the registry.
+        self._engine = None
 
         # Shutdown check callback (set by web_server)
         self.shutdown_check = None
@@ -114,17 +121,27 @@ class DeezerDownloadClient:
         self._license_token = None
         self._user_data = None
         self._authenticated = False
+        self._pending_arl: Optional[str] = None
 
         # Quality preference
-        self._quality = config_manager.get('deezer_download.quality', 'flac')
+        self._quality = quality_tier_for_source('deezer', default='flac')
 
         # Try to authenticate on init if ARL is configured
         arl = config_manager.get('deezer_download.arl', '')
         if arl:
-            self._authenticate(arl)
+            from core.boot_phase import is_boot_phase
+            if is_boot_phase():
+                self._pending_arl = arl
+                logger.debug("Deezer ARL present — authentication deferred until after boot")
+            else:
+                self._authenticate(arl)
 
         logger.info(f"Deezer download client initialized (download path: {self.download_path})")
 
+    def set_engine(self, engine):
+        """Engine callback — wires the central thread worker + state store."""
+        self._engine = engine
+
     # ─── Authentication ──────────────────────────────────────────
 
     def _authenticate(self, arl: str) -> bool:
@@ -216,12 +233,66 @@ class DeezerDownloadClient:
         return self._authenticated
 
     def is_authenticated(self) -> bool:
+        if self._pending_arl and not self._authenticated:
+            from core.boot_phase import is_boot_phase
+            if not is_boot_phase():
+                self._authenticate(self._pending_arl)
+                self._pending_arl = None
         return self._authenticated
 
     async def check_connection(self) -> bool:
         loop = asyncio.get_event_loop()
         return await loop.run_in_executor(None, self.is_available)
 
+    # ─── Playlist export (#945) ──────────────────────────────────
+    #
+    # UNOFFICIAL: rides the private gw-light gateway with the ARL session already used
+    # for downloads. Deezer shut their public developer API, so this is the only write
+    # path — and it's fragile by nature (breaks when Deezer changes internals).
+
+    def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
+                                  public=False, description=""):
+        """Create a Deezer playlist (or append to an existing one) from a mirrored
+        playlist's tracks. ``track_ids`` are stored ``deezer_id`` values per library track.
+        ``existing_id`` set → add to that playlist (idempotent re-export reuses the stored
+        target); unset → create a new one. Returns
+        ``{success, playlist_id, url, added, error}``."""
+        if not self._authenticated:
+            return {"success": False, "error": "Deezer is not connected (ARL)"}
+        song_ids = [str(t) for t in (track_ids or []) if t]
+        if not song_ids:
+            return {"success": False, "error": "No matching Deezer tracks to export"}
+        try:
+            songs = [[sid, i] for i, sid in enumerate(song_ids)]
+            if existing_id:
+                res = self._gw_call("playlist.addSongs",
+                                    {"playlist_id": int(existing_id), "songs": songs})
+                if res is None:
+                    return {"success": False, "error": "Deezer rejected the playlist update"}
+                playlist_id = existing_id
+            else:
+                res = self._gw_call("playlist.create", {
+                    "title": name, "description": description,
+                    "is_public": bool(public), "songs": songs,
+                })
+                if res is None:
+                    return {"success": False, "error": "Deezer rejected the playlist create"}
+                # gw 'playlist.create' returns the new playlist id (int) as `results`.
+                if isinstance(res, dict):
+                    playlist_id = res.get("PLAYLIST_ID") or res.get("id")
+                else:
+                    playlist_id = res
+                if not playlist_id:
+                    return {"success": False, "error": "Deezer did not return a playlist id"}
+            return {
+                "success": True,
+                "playlist_id": str(playlist_id),
+                "url": f"https://www.deezer.com/playlist/{playlist_id}",
+                "added": len(song_ids),
+            }
+        except Exception as e:
+            return {"success": False, "error": str(e)}
+
     def reconnect(self, arl: str = None) -> bool:
         """Re-authenticate with a new or existing ARL."""
         if arl is None:
@@ -410,8 +481,14 @@ class DeezerDownloadClient:
                 if aid:
                     album_ids.add(str(aid))
             album_release_dates = {}
+            # Deezer PLAYLIST tracks do NOT carry `track_position` (only `/track/<id>`
+            # and `/album/<id>/tracks` do), so numbering them by their playlist index
+            # poisons the real album track number — which then rides into the wishlist
+            # and onto the downloaded file's tag (e.g. 'Apologize' tagged track 1 instead
+            # of 16). Resolve the REAL position from each album's track list (cache-first).
+            track_positions: Dict[str, int] = {}   # str(track_id) -> album track_position
             try:
-                from core.metadata_cache import get_metadata_cache
+                from core.metadata.cache import get_metadata_cache
                 cache = get_metadata_cache()
             except Exception:
                 cache = None
@@ -422,24 +499,32 @@ class DeezerDownloadClient:
                         cached = cache.get_entity('deezer', 'album', aid)
                         if cached and cached.get('release_date'):
                             album_release_dates[aid] = cached['release_date']
-                            continue
-                    except Exception:
-                        pass
+                    except Exception as e:
+                        logger.debug("cache get_entity album release_date: %s", e)
                 # Cache miss — fetch from API
-                try:
-                    time.sleep(0.3)  # Respect rate limits
-                    a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
-                    if a_resp.ok:
-                        a_data = a_resp.json()
-                        album_release_dates[aid] = a_data.get('release_date', '')
-                        # Store in metadata cache for future use
-                        if cache:
-                            try:
-                                cache.store_entity('deezer', 'album', aid, a_data)
-                            except Exception:
-                                pass
-                except Exception:
-                    pass
+                if aid not in album_release_dates:
+                    try:
+                        time.sleep(0.3)  # Respect rate limits
+                        a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
+                        if a_resp.ok:
+                            a_data = a_resp.json()
+                            album_release_dates[aid] = a_data.get('release_date', '')
+                            # Store in metadata cache for future use
+                            if cache:
+                                try:
+                                    cache.store_entity('deezer', 'album', aid, a_data)
+                                except Exception as e:
+                                    logger.debug("cache store_entity album release_date: %s", e)
+                    except Exception as e:
+                        logger.debug("fetch deezer album release_date %s: %s", aid, e)
+            # Real album track positions (separate endpoint — playlist tracks AND the
+            # album object's embedded tracks both omit track_position). Cache-first.
+            try:
+                from core.deezer_client import resolve_album_track_positions
+                track_positions = resolve_album_track_positions(
+                    self._session, 'https://api.deezer.com', album_ids, cache)
+            except Exception as e:
+                logger.debug("resolve deezer album track positions: %s", e)
 
             tracks = []
             for i, t in enumerate(raw_tracks, start=1):
@@ -460,7 +545,9 @@ class DeezerDownloadClient:
                         'id': album_id,
                     },
                     'duration_ms': t.get('duration', 0) * 1000,
-                    'track_number': i,
+                    # REAL album position (resolved above); the playlist index is a last
+                    # resort only when the album lookup failed, never the default.
+                    'track_number': track_positions.get(str(t.get('id'))) or t.get('track_position') or i,
                 })
 
             return {
@@ -574,7 +661,7 @@ class DeezerDownloadClient:
                     bitrate = 128
                     quality = 'mp3'
 
-                results.append(TrackResult(
+                tr = TrackResult(
                     username='deezer_dl',
                     filename=f"{track_id}||{artist} - {title}",
                     size=est_size,
@@ -588,7 +675,10 @@ class DeezerDownloadClient:
                     title=title,
                     album=album,
                     track_number=item.get('track_position'),
-                ))
+                )
+                # Stamp CD-quality FLAC (16/44.1) so lossless ranks correctly.
+                tr.set_quality(quality_from_deezer(self._quality))
+                results.append(tr)
 
             logger.info(f"Deezer search for '{query}' returned {len(results)} results")
             return results, []
@@ -605,87 +695,67 @@ class DeezerDownloadClient:
         if not self._authenticated:
             logger.error("Deezer not authenticated — cannot download")
             return None
+        if self._engine is None:
+            # Raise rather than return None so the orchestrator's
+            # download_with_fallback surfaces a real warning + tries
+            # the next source. Returning None silently dropped the
+            # download with no user feedback (per JohnBaumb).
+            raise RuntimeError("Deezer client has no engine reference — cannot dispatch download")
 
         # Parse filename: "track_id||display_name"
         parts = filename.split('||', 1)
         track_id = parts[0]
         display_name = parts[1] if len(parts) > 1 else f"Track {track_id}"
 
-        download_id = str(uuid.uuid4())
-
-        with self._download_lock:
-            self.active_downloads[download_id] = {
-                'id': download_id,
+        return self._engine.worker.dispatch(
+            source_name='deezer',
+            target_id=track_id,
+            display_name=display_name,
+            original_filename=filename,
+            impl_callable=self._download_sync,
+            extra_record_fields={
                 'track_id': track_id,
                 'display_name': display_name,
-                'filename': filename,
-                'username': 'deezer_dl',
-                'state': 'Initializing',
-                'progress': 0.0,
                 'size': file_size,
-                'transferred': 0,
-                'speed': 0,
-                'file_path': None,
                 'error': None,
-            }
-
-        thread = threading.Thread(
-            target=self._download_thread_worker,
-            args=(download_id, track_id, display_name),
-            daemon=True,
-            name=f'deezer-dl-{track_id}'
+            },
+            # Legacy username slot — frontend status indicators key off
+            # ``deezer_dl``, not the canonical ``deezer``.
+            username_override='deezer_dl',
+            # Diagnostic thread name for multi-thread debugging.
+            thread_name=f'deezer-dl-{track_id}',
         )
-        thread.start()
 
-        logger.info(f"Started Deezer download {download_id}: {display_name}")
-        return download_id
+    def _set_error(self, download_id: str, message: str) -> None:
+        """Helper: set the engine record's `error` slot. No-op if
+        engine isn't wired or record was already removed."""
+        if self._engine is None:
+            return
+        self._engine.update_record('deezer', download_id, {'error': message})
 
-    def _download_thread_worker(self, download_id: str, track_id: str, display_name: str):
-        """Background worker for a single download."""
-        try:
-            result_path = self._download_sync(download_id, track_id, display_name)
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    dl = self.active_downloads[download_id]
-                    if dl['state'] == 'Cancelled':
-                        return
-                    if result_path:
-                        dl['state'] = 'Completed, Succeeded'
-                        dl['progress'] = 100.0
-                        dl['file_path'] = result_path
-                        logger.info(f"Deezer download {download_id} completed: {result_path}")
-                    else:
-                        dl['state'] = 'Errored'
-                        logger.error(f"Deezer download {download_id} failed: {dl.get('error', 'unknown')}")
-        except Exception as e:
-            logger.error(f"Deezer download thread error: {e}")
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['state'] = 'Errored'
-                    self.active_downloads[download_id]['error'] = str(e)
+    def _is_cancelled(self, download_id: str) -> bool:
+        if self._engine is None:
+            return False
+        record = self._engine.get_record('deezer', download_id)
+        return record is not None and record.get('state') == 'Cancelled'
 
     def _download_sync(self, download_id: str, track_id: str, display_name: str) -> Optional[str]:
         """Synchronous download: get URL, download, decrypt, save."""
         # Check for shutdown
         if self.shutdown_check and self.shutdown_check():
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['state'] = 'Aborted'
+            if self._engine is not None:
+                self._engine.update_record('deezer', download_id, {'state': 'Aborted'})
             return None
 
         # Get track data from private API
         track_data = self._get_track_data(track_id)
         if not track_data:
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['error'] = 'Failed to get track data'
+            self._set_error(download_id, 'Failed to get track data')
             return None
 
         track_token = track_data.get('TRACK_TOKEN', '')
         if not track_token:
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['error'] = 'No track token available'
+            self._set_error(download_id, 'No track token available')
             return None
 
         # Determine quality and get media URL with fallback
@@ -695,7 +765,6 @@ class DeezerDownloadClient:
 
         if allow_fallback:
             quality_order = _QUALITY_ORDER.copy()
-            # Start from user's preferred quality
             try:
                 pref_idx = quality_order.index(self._quality)
                 quality_order = quality_order[pref_idx:] + quality_order[:pref_idx]
@@ -712,27 +781,16 @@ class DeezerDownloadClient:
                 break
 
         if not media_url:
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['error'] = 'No media URL available (may require higher subscription tier)'
+            self._set_error(download_id, 'No media URL available (may require higher subscription tier)')
             return None
 
         if actual_quality != self._quality:
             logger.info(f"Quality fallback: {self._quality} → {actual_quality} for {display_name}")
 
-        # Determine file extension
         ext = '.flac' if actual_quality == 'flac' else '.mp3'
-
-        # Sanitize filename
         safe_name = self._sanitize_filename(display_name)
         out_path = str(self.download_path / f"{safe_name}{ext}")
 
-        # Update state
-        with self._download_lock:
-            if download_id in self.active_downloads:
-                dl = self.active_downloads[download_id]
-                dl['state'] = 'InProgress, Downloading'
-
         # Download and decrypt
         try:
             bf_key = _get_blowfish_key(track_id)
@@ -740,9 +798,8 @@ class DeezerDownloadClient:
             resp.raise_for_status()
 
             total_size = int(resp.headers.get('content-length', 0))
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['size'] = total_size
+            if self._engine is not None:
+                self._engine.update_record('deezer', download_id, {'size': total_size})
 
             downloaded = 0
             chunk_index = 0
@@ -755,23 +812,20 @@ class DeezerDownloadClient:
 
                     # Check for cancellation/shutdown
                     if self.shutdown_check and self.shutdown_check():
-                        with self._download_lock:
-                            if download_id in self.active_downloads:
-                                self.active_downloads[download_id]['state'] = 'Aborted'
+                        if self._engine is not None:
+                            self._engine.update_record('deezer', download_id, {'state': 'Aborted'})
                         try:
                             os.remove(out_path)
                         except OSError:
                             pass
                         return None
 
-                    with self._download_lock:
-                        if download_id in self.active_downloads:
-                            if self.active_downloads[download_id]['state'] == 'Cancelled':
-                                try:
-                                    os.remove(out_path)
-                                except OSError:
-                                    pass
-                                return None
+                    if self._is_cancelled(download_id):
+                        try:
+                            os.remove(out_path)
+                        except OSError:
+                            pass
+                        return None
 
                     # Decrypt every 3rd chunk (Deezer's encryption pattern)
                     if chunk_index % 3 == 0 and len(raw_chunk) == _CHUNK_SIZE:
@@ -788,12 +842,12 @@ class DeezerDownloadClient:
                     speed = int(downloaded / elapsed) if elapsed > 0 else 0
                     progress = (downloaded / total_size * 100) if total_size > 0 else 0
 
-                    with self._download_lock:
-                        if download_id in self.active_downloads:
-                            dl = self.active_downloads[download_id]
-                            dl['transferred'] = downloaded
-                            dl['progress'] = min(progress, 99.9)
-                            dl['speed'] = speed
+                    if self._engine is not None:
+                        self._engine.update_record('deezer', download_id, {
+                            'transferred': downloaded,
+                            'progress': min(progress, 99.9),
+                            'speed': speed,
+                        })
 
             # Validate file size
             file_size = os.path.getsize(out_path)
@@ -803,9 +857,7 @@ class DeezerDownloadClient:
                     os.remove(out_path)
                 except OSError:
                     pass
-                with self._download_lock:
-                    if download_id in self.active_downloads:
-                        self.active_downloads[download_id]['error'] = f'File too small ({file_size} bytes)'
+                self._set_error(download_id, f'File too small ({file_size} bytes)')
                 return None
 
             logger.info(f"Deezer download complete: {out_path} ({file_size / 1048576:.1f} MB, {actual_quality})")
@@ -817,59 +869,58 @@ class DeezerDownloadClient:
                 os.remove(out_path)
             except OSError:
                 pass
-            with self._download_lock:
-                if download_id in self.active_downloads:
-                    self.active_downloads[download_id]['error'] = str(e)
+            self._set_error(download_id, str(e))
             return None
 
     # ─── Download Status ─────────────────────────────────────────
 
+    def _record_to_status(self, record: dict) -> DownloadStatus:
+        return DownloadStatus(
+            id=record['id'],
+            filename=record['filename'],
+            username=record['username'],
+            state=record['state'],
+            progress=record['progress'],
+            size=record.get('size', 0),
+            transferred=record.get('transferred', 0),
+            speed=record.get('speed', 0),
+            file_path=record.get('file_path'),
+        )
+
     async def get_all_downloads(self) -> List[DownloadStatus]:
-        """Return all active downloads."""
-        with self._download_lock:
-            return [self._to_status(dl) for dl in self.active_downloads.values()]
+        if self._engine is None:
+            return []
+        return [
+            self._record_to_status(record)
+            for record in self._engine.iter_records_for_source('deezer')
+        ]
 
     async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
-        """Get status of a specific download."""
-        with self._download_lock:
-            dl = self.active_downloads.get(download_id)
-            return self._to_status(dl) if dl else None
+        if self._engine is None:
+            return None
+        record = self._engine.get_record('deezer', download_id)
+        return self._record_to_status(record) if record is not None else None
 
     async def cancel_download(self, download_id: str, username: str = None,
                               remove: bool = False) -> bool:
-        """Cancel a download."""
-        with self._download_lock:
-            dl = self.active_downloads.get(download_id)
-            if not dl:
-                return False
-            dl['state'] = 'Cancelled'
-            if remove:
-                del self.active_downloads[download_id]
+        if self._engine is None:
+            return False
+        if self._engine.get_record('deezer', download_id) is None:
+            return False
+        self._engine.update_record('deezer', download_id, {'state': 'Cancelled'})
+        if remove:
+            self._engine.remove_record('deezer', download_id)
         return True
 
     async def clear_all_completed_downloads(self) -> bool:
-        """Remove all terminal downloads."""
-        terminal_states = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
-        with self._download_lock:
-            to_remove = [k for k, v in self.active_downloads.items() if v['state'] in terminal_states]
-            for k in to_remove:
-                del self.active_downloads[k]
+        if self._engine is None:
+            return True
+        terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
+        for record in list(self._engine.iter_records_for_source('deezer')):
+            if record.get('state') in terminal:
+                self._engine.remove_record('deezer', record['id'])
         return True
 
-    def _to_status(self, dl: dict) -> DownloadStatus:
-        """Convert internal dict to DownloadStatus."""
-        return DownloadStatus(
-            id=dl['id'],
-            filename=dl['filename'],
-            username=dl['username'],
-            state=dl['state'],
-            progress=dl['progress'],
-            size=dl['size'],
-            transferred=dl['transferred'],
-            speed=dl['speed'],
-            file_path=dl.get('file_path'),
-        )
-
     # ─── Utilities ───────────────────────────────────────────────
 
     @staticmethod

core/deezer_worker.py

@@ -8,7 +8,16 @@ from datetime import datetime, timedelta
 from utils.logging_config import get_logger
 from database.music_database import MusicDatabase
 from core.deezer_client import DeezerClient
-from core.worker_utils import interruptible_sleep, set_album_api_track_count
+from core.worker_utils import (
+    accept_artist_match,
+    artist_name_matches,
+    interruptible_sleep,
+    owned_album_titles,
+    pick_artist_by_catalog,
+    release_titles,
+    set_album_api_track_count,
+)
+from core.enrichment.manual_match_honoring import honor_stored_match
 
 logger = get_logger("deezer_worker")
 
@@ -140,8 +149,8 @@ class DeezerWorker:
                         itype = item.get('type', '')
                         table = 'artists' if 'artist' in itype else ('albums' if 'album' in itype else 'tracks')
                         # Can't mark status without an ID — just skip
-                    except Exception:
-                        pass
+                    except Exception as e:
+                        logger.debug("null id table resolve failed: %s", e)
                     continue
 
 
@@ -162,6 +171,16 @@ class DeezerWorker:
             conn = self.db._get_connection()
             cursor = conn.cursor()
 
+            # Pinned-group override (Manage Enrichment Workers): process one
+            # entity type first, then fall through to the normal chain. Unset or
+            # exhausted ⇒ default artist→album→track order, unchanged.
+            from core.worker_utils import read_enrichment_priority, priority_pending_item
+            _prio = read_enrichment_priority('deezer')
+            if _prio:
+                _pi = priority_pending_item(cursor, 'deezer', _prio)
+                if _pi:
+                    return _pi
+
             # Priority 1: Unattempted artists
             cursor.execute("""
                 SELECT id, name
@@ -267,10 +286,19 @@ class DeezerWorker:
         logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
         return similarity >= self.name_similarity_threshold
 
-    def _verify_artist_id(self, item: Dict[str, Any], result_artist_id) -> bool:
+    def _verify_artist_id(self, item: Dict[str, Any], result_artist_id,
+                          result_artist_name: Optional[str] = None) -> bool:
         """Verify that the result's artist ID matches the parent artist's stored Deezer ID.
+
         If mismatched, the album/track search is more specific (uses artist+title),
-        so we trust it and correct the parent artist's deezer_id."""
+        so we trust it and correct the parent artist's deezer_id — BUT only when
+        the result's artist *name* actually matches our parent artist. Without
+        that guard, a collaboration or compilation track (e.g. a track our
+        library credits to Jorja Smith that lives on Kendrick Lamar's curated
+        "Black Panther" album) would search up to an album whose Deezer primary
+        artist is someone else (Kendrick), and we'd stamp that wrong Deezer ID
+        onto our artist — corrupting it (and causing duplicate ids shared across
+        unrelated artists)."""
         parent_deezer_id = item.get('artist_deezer_id')
         if not parent_deezer_id:
             return True
@@ -279,6 +307,20 @@ class DeezerWorker:
             return True
 
         if str(result_artist_id) != str(parent_deezer_id):
+            # Guard: only correct when the album/track's primary artist is the
+            # SAME artist by name. A mismatch means it's a collab/compilation,
+            # not a stale-id correction.
+            parent_name = item.get('artist') or ''
+            if (result_artist_name and parent_name
+                    and not self._name_matches(parent_name, result_artist_name)):
+                logger.info(
+                    f"Skipping artist-ID correction from {item['type']} "
+                    f"'{item['name']}': result artist '{result_artist_name}' "
+                    f"≠ parent '{parent_name}' (collab/compilation, not a "
+                    f"correction)"
+                )
+                return True
+
             logger.info(
                 f"Artist ID correction from {item['type']} '{item['name']}': "
                 f"updating parent artist Deezer ID from {parent_deezer_id} to {result_artist_id}"
@@ -367,27 +409,66 @@ class DeezerWorker:
             logger.debug(f"Preserving existing Deezer ID for artist '{artist_name}': {existing_id}")
             return
 
-        result = self.client.search_artist(artist_name)
+        # Multi-candidate search (was single search_artist) so same-name artists
+        # can be disambiguated: gate by name, then pick the one whose catalog
+        # overlaps the albums this library owns.
+        results = self.client.search_artists(artist_name, limit=5)
+        gated = [a for a in (results or []) if artist_name_matches(artist_name, getattr(a, 'name', ''))]
+        chosen, _overlap = pick_artist_by_catalog(
+            gated,
+            owned_album_titles(self.db, artist_id),
+            lambda a: release_titles(self.client.get_artist_albums_list(a.id)),
+        )
+
+        # search_artists returns lean Artist objects; fetch the full dict (same
+        # shape the old search_artist returned) for storage.
+        result = self.client.get_artist_info(chosen.id) if chosen else None
         if result:
             result_name = result.get('name', '')
-            if self._name_matches(artist_name, result_name):
+            ok, reason = accept_artist_match(
+                self.db, 'deezer_id', result.get('id'), artist_id,
+                artist_name, result_name,
+            )
+            if ok:
                 self._update_artist(artist_id, result)
                 self.stats['matched'] += 1
                 logger.info(f"Matched artist '{artist_name}' -> Deezer ID: {result.get('id')}")
             else:
                 self._mark_status('artist', artist_id, 'not_found')
                 self.stats['not_found'] += 1
-                logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result_name}')")
+                logger.debug(f"Artist '{artist_name}' not matched: {reason}")
         else:
             self._mark_status('artist', artist_id, 'not_found')
             self.stats['not_found'] += 1
             logger.debug(f"No match for artist '{artist_name}'")
 
+    def _refresh_album_via_stored_id(self, album_id, stored_id, full_album_dict):
+        """Issue #501 callback. Stored ID exists → fetched full Deezer
+        album payload. Use it as both args to ``_update_album`` (search-
+        result and full-data shapes overlap on the fields we need —
+        artist verification skipped since manual match presumably
+        already vetted)."""
+        self._update_album(album_id, full_album_dict, full_album_dict)
+
+    def _refresh_track_via_stored_id(self, track_id, stored_id, full_track_dict):
+        """Issue #501 callback for tracks — same pattern as albums."""
+        self._update_track(track_id, full_track_dict, full_track_dict)
+
     def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
         """Process an album: search Deezer, verify, fetch full details, store metadata"""
-        existing_id = self._get_existing_id('album', album_id)
-        if existing_id:
-            logger.debug(f"Preserving existing Deezer ID for album '{album_name}': {existing_id}")
+        # Issue #501: honor manual matches. Pre-fix this method just
+        # SKIPPED when a stored ID was present (preserved the ID but
+        # never refreshed metadata). Now it goes through the full
+        # refresh path via the stored ID, picking up label / genres /
+        # explicit updates without ever overwriting the manual match.
+        if honor_stored_match(
+            db=self.db, entity_table='albums', entity_id=album_id,
+            id_column='deezer_id',
+            client_fetch_fn=self.client.get_album_raw,
+            on_match_fn=self._refresh_album_via_stored_id,
+            log_prefix='Deezer',
+        ):
+            self.stats['matched'] += 1
             return
 
         result = self.client.search_album(artist_name, album_name)
@@ -397,7 +478,8 @@ class DeezerWorker:
                 # Verify artist ID
                 result_artist = result.get('artist', {})
                 result_artist_id = result_artist.get('id') if result_artist else None
-                self._verify_artist_id(item, result_artist_id)
+                result_artist_name = result_artist.get('name') if result_artist else None
+                self._verify_artist_id(item, result_artist_id, result_artist_name)
 
                 # Fetch full album details for label, genres, explicit
                 deezer_album_id = result.get('id')
@@ -430,9 +512,15 @@ class DeezerWorker:
 
     def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
         """Process a track: search Deezer, verify, fetch full details for BPM, store metadata"""
-        existing_id = self._get_existing_id('track', track_id)
-        if existing_id:
-            logger.debug(f"Preserving existing Deezer ID for track '{track_name}': {existing_id}")
+        # Issue #501: honor manual matches (see _process_album).
+        if honor_stored_match(
+            db=self.db, entity_table='tracks', entity_id=track_id,
+            id_column='deezer_id',
+            client_fetch_fn=self.client.get_track_raw,
+            on_match_fn=self._refresh_track_via_stored_id,
+            log_prefix='Deezer',
+        ):
+            self.stats['matched'] += 1
             return
 
         result = self.client.search_track(artist_name, track_name)
@@ -442,7 +530,8 @@ class DeezerWorker:
                 # Verify artist ID
                 result_artist = result.get('artist', {})
                 result_artist_id = result_artist.get('id') if result_artist else None
-                self._verify_artist_id(item, result_artist_id)
+                result_artist_name = result_artist.get('name') if result_artist else None
+                self._verify_artist_id(item, result_artist_id, result_artist_name)
 
                 # Fetch full track details for BPM
                 deezer_track_id = result.get('id')

core/diagnostics/memory_tracker.py

@@ -0,0 +1,136 @@
+"""On-demand memory-growth diagnostic (issue #802: ~0.7 MiB/s RSS growth).
+
+Wraps ``tracemalloc`` so a user seeing runaway memory can capture WHERE the
+allocations come from instead of us guessing:
+
+    1. start_tracking()  — begins tracing + stores a baseline snapshot
+    2. ...reproduce the growth for a few minutes...
+    3. report()          — top allocation sites, with the DELTA since baseline
+                           (the delta is the leak; absolute sizes are mostly
+                           startup noise)
+    4. stop_tracking()   — ends tracing, frees trace memory
+
+Opt-in by design: tracemalloc costs CPU and memory while active (it shadows
+every allocation), so it must never run by default. The Flask endpoints that
+expose this live in web_server (GET /api/debug/memory/...) so a user can drive
+the whole flow from a browser.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+import tracemalloc
+from typing import Any, Dict, List, Optional
+
+from utils.logging_config import get_logger
+
+logger = get_logger("diagnostics.memory")
+
+_baseline: Optional[tracemalloc.Snapshot] = None
+_started_at: Optional[float] = None
+
+# Allocation-site traces this deep give useful "who called it" context without
+# pathological overhead.
+_TRACE_FRAMES = 15
+
+
+def is_tracking() -> bool:
+    return tracemalloc.is_tracing()
+
+
+def start_tracking() -> Dict[str, Any]:
+    """Begin tracing and store the baseline snapshot. Idempotent."""
+    global _baseline, _started_at
+    if tracemalloc.is_tracing():
+        return {"tracking": True, "already_running": True, "started_at": _started_at}
+    tracemalloc.start(_TRACE_FRAMES)
+    _baseline = tracemalloc.take_snapshot()
+    _started_at = time.time()
+    logger.info("Memory tracking started (tracemalloc, %d frames)", _TRACE_FRAMES)
+    return {"tracking": True, "already_running": False, "started_at": _started_at}
+
+
+def stop_tracking() -> Dict[str, Any]:
+    """End tracing and free the trace bookkeeping."""
+    global _baseline, _started_at
+    was = tracemalloc.is_tracing()
+    if was:
+        tracemalloc.stop()
+        logger.info("Memory tracking stopped")
+    _baseline = None
+    _started_at = None
+    return {"tracking": False, "was_tracking": was}
+
+
+def _rss_mb() -> Optional[float]:
+    """Process RSS in MiB, best-effort (psutil, then /proc fallback)."""
+    try:
+        import psutil
+        return round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 1)
+    except Exception:  # noqa: S110 — RSS is optional context; fall through to /proc
+        pass
+    try:
+        with open("/proc/self/status", encoding="utf-8") as fh:
+            for line in fh:
+                if line.startswith("VmRSS:"):
+                    return round(int(line.split()[1]) / 1024, 1)
+    except Exception:  # noqa: S110 — no /proc on this platform; RSS stays None
+        pass
+    return None
+
+
+def format_stat(stat: Any) -> Dict[str, Any]:
+    """Project one tracemalloc StatisticDiff/Statistic into a plain dict.
+    Duck-typed (reads size/count/size_diff/count_diff/traceback) so it's
+    unit-testable without real snapshots."""
+    tb = getattr(stat, "traceback", None)
+    frames: List[str] = []
+    if tb:
+        # Most-recent-call-last reads naturally top-down in a report.
+        for frame in list(tb)[-3:]:
+            frames.append(f"{frame.filename}:{frame.lineno}")
+    return {
+        "location": frames[-1] if frames else "?",
+        "trace": frames,
+        "size_mb": round(getattr(stat, "size", 0) / (1024 * 1024), 3),
+        "size_diff_mb": round(getattr(stat, "size_diff", 0) / (1024 * 1024), 3),
+        "count": getattr(stat, "count", 0),
+        "count_diff": getattr(stat, "count_diff", 0),
+    }
+
+
+def report(top: int = 25) -> Dict[str, Any]:
+    """Current snapshot vs the start_tracking() baseline: the top allocation
+    sites by GROWTH (size_diff). Includes traced totals + process RSS so the
+    user can see how much of the real growth tracemalloc accounts for."""
+    if not tracemalloc.is_tracing():
+        return {
+            "tracking": False,
+            "rss_mb": _rss_mb(),
+            "hint": "Start with /api/debug/memory/start, reproduce the growth "
+                    "for a few minutes, then call this again.",
+        }
+    snapshot = tracemalloc.take_snapshot()
+    # Filter the tracer's own bookkeeping out of the picture.
+    snapshot = snapshot.filter_traces((
+        tracemalloc.Filter(False, tracemalloc.__file__),
+        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
+    ))
+    current, peak = tracemalloc.get_traced_memory()
+
+    if _baseline is not None:
+        stats = snapshot.compare_to(_baseline, "traceback")
+        stats.sort(key=lambda s: s.size_diff, reverse=True)
+    else:
+        stats = snapshot.statistics("traceback")
+
+    return {
+        "tracking": True,
+        "started_at": _started_at,
+        "elapsed_seconds": round(time.time() - _started_at, 1) if _started_at else None,
+        "traced_current_mb": round(current / (1024 * 1024), 1),
+        "traced_peak_mb": round(peak / (1024 * 1024), 1),
+        "rss_mb": _rss_mb(),
+        "top_growth": [format_stat(s) for s in stats[:top]],
+    }

core/discogs_client.py

@@ -12,7 +12,8 @@ import re
 import time
 import threading
 import requests
-from core.metadata_cache import get_metadata_cache
+from core.metadata.artist_album_cache import get_cached_artist_album_payload, store_artist_album_items
+from core.metadata.cache import get_metadata_cache
 from typing import List, Dict, Any, Optional
 from dataclasses import dataclass
 from functools import wraps
@@ -61,6 +62,69 @@ def rate_limited(func):
     return wrapper
 
 
+# Discogs disambiguates duplicate artist names two ways: numeric `(N)`
+# suffix on the older convention (e.g. "Bullet (2)") and a trailing `*`
+# on the newer convention (e.g. "John Smith*"). Both are presentation-
+# only — the underlying canonical name is what users expect to see in
+# search results, on cards, and especially in import paths (otherwise
+# library folders end up named `Foo*` on disk). Strip both at every
+# point a Discogs payload becomes a name string.
+_DISCOGS_DISAMBIG_RE = re.compile(r'(?:\s*\(\d+\))?\s*\*+\s*$|\s*\(\d+\)\s*$')
+
+
+def _clean_discogs_artist_name(name: Optional[str]) -> str:
+    """Strip Discogs disambiguation suffixes — both `(N)` and trailing
+    `*` — from an artist name. Returns '' for None / empty input."""
+    if not name:
+        return ''
+    return _DISCOGS_DISAMBIG_RE.sub('', name).strip()
+
+
+# --- Discogs album ID typing -------------------------------------------------
+# Discogs has two album object types — masters (/masters/{id}) and releases
+# (/releases/{id}) — whose numeric IDs share one space, so release N and master
+# N are DIFFERENT albums. A bare numeric ID is therefore ambiguous. We tag the
+# type into the ID string ('m12345' / 'r12345') at the point we parse it, so the
+# correct endpoint can be chosen later without guessing. (Artist IDs are a single
+# namespace and stay untagged.)
+
+def _discogs_album_kind(data: Dict[str, Any]) -> str:
+    """Classify a Discogs album payload as 'master' or 'release'.
+
+    Search results and artist-discography items carry an explicit ``type``;
+    full detail responses don't, but only master detail has ``main_release``."""
+    t = (data.get('type') or '').lower()
+    if t in ('master', 'release'):
+        return t
+    return 'master' if 'main_release' in data else 'release'
+
+
+def _tag_discogs_album_id(raw_id: Any, kind: str) -> str:
+    """``'12345'`` + ``'master'`` -> ``'m12345'``; empty input -> ``''``."""
+    s = str(raw_id or '').strip()
+    if not s:
+        return ''
+    return f"{'m' if kind == 'master' else 'r'}{s}"
+
+
+def _discogs_album_endpoints(album_id: Any) -> List[str]:
+    """Map a (possibly tagged) album ID to the API path(s) to try, in order.
+
+      ``'m12345'`` -> ``['/masters/12345']``
+      ``'r12345'`` -> ``['/releases/12345']``
+      ``'12345'``  (legacy untagged) -> ``['/releases/12345', '/masters/12345']``
+
+    Legacy bare IDs are tried release-first because stored IDs originate
+    overwhelmingly from search / manual-match / collection sync (all releases);
+    this also self-heals pre-fix bad matches. Returns ``[]`` for unusable input."""
+    s = str(album_id or '').strip()
+    if len(s) > 1 and s[0] in ('m', 'r') and s[1:].isdigit():
+        return [f"/{'masters' if s[0] == 'm' else 'releases'}/{s[1:]}"]
+    if s.isdigit():
+        return [f'/releases/{s}', f'/masters/{s}']
+    return []
+
+
 # --- Shared dataclasses (same shape as iTunes/Deezer/Spotify) ---
 
 @dataclass
@@ -117,9 +181,10 @@ class Track:
         # Artists from track-level or release-level
         track_artists = []
         if track_data.get('artists'):
-            track_artists = [a.get('name', '') for a in track_data['artists'] if a.get('name')]
+            track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in track_data['artists'] if a.get('name')]
         if not track_artists and release.get('artists'):
-            track_artists = [a.get('name', '') for a in release['artists'] if a.get('name')]
+            track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release['artists'] if a.get('name')]
+        track_artists = [a for a in track_artists if a]
         if not track_artists:
             track_artists = ['Unknown Artist']
 
@@ -190,7 +255,9 @@ class Artist:
 
         return cls(
             id=str(artist_data.get('id', '')),
-            name=artist_data.get('name', artist_data.get('title', '')),
+            name=_clean_discogs_artist_name(
+                artist_data.get('name', artist_data.get('title', ''))
+            ),
             popularity=0,
             genres=[],
             followers=0,
@@ -216,14 +283,15 @@ class Album:
         artists = []
         title = release_data.get('title', '')
         if release_data.get('artists'):
-            artists = [a.get('name', '') for a in release_data['artists'] if a.get('name')]
+            artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release_data['artists'] if a.get('name')]
         elif release_data.get('artist'):
-            artists = [release_data['artist']]
+            artists = [_clean_discogs_artist_name(release_data['artist'])]
         elif ' - ' in title:
             # Search results: "Radiohead - OK Computer" → artist="Radiohead", title="OK Computer"
             parts = title.split(' - ', 1)
-            artists = [parts[0].strip()]
+            artists = [_clean_discogs_artist_name(parts[0])]
             title = parts[1].strip()
+        artists = [a for a in artists if a]
         if not artists:
             artists = ['Unknown Artist']
 
@@ -282,7 +350,7 @@ class Album:
             external_urls['discogs_api'] = release_data['resource_url']
 
         return cls(
-            id=str(release_data.get('id', '')),
+            id=_tag_discogs_album_id(release_data.get('id', ''), _discogs_album_kind(release_data)),
             name=title,
             artists=artists,
             release_date=release_date,
@@ -320,8 +388,8 @@ class DiscogsClient:
             try:
                 from config.settings import config_manager
                 self.token = config_manager.get('discogs.token', '')
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("load discogs.token from config: %s", e)
 
         if self.token:
             self.session.headers['Authorization'] = f'Discogs token={self.token}'
@@ -369,6 +437,125 @@ class DiscogsClient:
             logger.error(f"Discogs API error ({endpoint}): {e}")
             return None
 
+    # --- User Collection (powers Your Albums Discogs source) ---
+
+    def get_authenticated_username(self) -> Optional[str]:
+        """Resolve the username for the configured personal token.
+
+        Discogs's `/oauth/identity` endpoint returns the user's
+        username when called with a valid token. Cached on the
+        instance so subsequent calls don't re-hit the API.
+        """
+        if hasattr(self, '_cached_username'):
+            return self._cached_username
+        if not self.is_authenticated():
+            self._cached_username = None
+            return None
+        data = self._api_get('/oauth/identity')
+        username = data.get('username') if data else None
+        self._cached_username = username
+        return username
+
+    def get_user_collection(self, username: Optional[str] = None,
+                            folder_id: int = 0,
+                            per_page: int = 100,
+                            max_pages: int = 50) -> List[Dict[str, Any]]:
+        """Fetch a Discogs user's collection (folder 0 = "All").
+
+        Returns a list of normalized release dicts ready for
+        ``database.upsert_liked_album``:
+            {
+                'album_name': str,
+                'artist_name': str,
+                'release_id': int,        # Discogs release id
+                'image_url': str | None,
+                'release_date': str,      # 'YYYY' (Discogs only stores year)
+                'total_tracks': int,
+            }
+
+        Pagination caps at ``max_pages`` to bound runtime — at 100/page
+        that's 5000 releases, more than enough for typical collections.
+        Authenticated calls only (Discogs collection is private).
+        """
+        if not self.is_authenticated():
+            logger.warning("Discogs collection fetch attempted without token")
+            return []
+
+        if not username:
+            username = self.get_authenticated_username()
+            if not username:
+                logger.warning("Could not resolve Discogs username for token")
+                return []
+
+        results: List[Dict[str, Any]] = []
+        page = 1
+        while page <= max_pages:
+            data = self._api_get(
+                f'/users/{username}/collection/folders/{folder_id}/releases',
+                {'page': page, 'per_page': per_page, 'sort': 'added', 'sort_order': 'desc'},
+            )
+            if not data:
+                break
+
+            releases = data.get('releases', []) or []
+            if not releases:
+                break
+
+            for entry in releases:
+                info = entry.get('basic_information') or {}
+                release_id = entry.get('id') or info.get('id')
+                if not release_id:
+                    continue
+                title = info.get('title') or ''
+                # Discogs `artists` is a list of {name, id, ...}; first is primary.
+                artists = info.get('artists') or []
+                artist_name = ''
+                if artists and isinstance(artists[0], dict):
+                    artist_name = (artists[0].get('name') or '').strip()
+                # Strip Discogs disambiguation suffixes — both `(N)` and `*`.
+                artist_name = _clean_discogs_artist_name(artist_name)
+                if not title or not artist_name:
+                    continue
+
+                # Image URLs: cover_image is the primary, also has thumb.
+                image_url = (info.get('cover_image')
+                             or info.get('thumb')
+                             or '')
+
+                year = info.get('year')
+                release_date = str(year) if year and year > 0 else ''
+
+                results.append({
+                    'album_name': title.strip(),
+                    'artist_name': artist_name,
+                    'release_id': int(release_id),
+                    'image_url': image_url or None,
+                    'release_date': release_date,
+                    'total_tracks': 0,  # Not in basic_information; populated via get_release if needed
+                })
+
+            pagination = data.get('pagination') or {}
+            if page >= int(pagination.get('pages') or 1):
+                break
+            page += 1
+
+        logger.info(f"Discogs collection: fetched {len(results)} releases for {username}")
+        return results
+
+    def get_release(self, release_id: int) -> Optional[Dict[str, Any]]:
+        """Fetch full Discogs release detail including tracklist.
+
+        Returns the raw API response so callers can render rich
+        Discogs context (year, format, label, country, tracklist).
+        """
+        if not release_id:
+            return None
+        try:
+            release_id = int(release_id)
+        except (TypeError, ValueError):
+            return None
+        return self._api_get(f'/releases/{release_id}')
+
     # --- Search Methods (same signatures as iTunes/Deezer) ---
 
     def search_artists(self, query: str, limit: int = 10) -> List[Artist]:
@@ -380,8 +567,8 @@ class DiscogsClient:
             for raw in cached_results:
                 try:
                     artists.append(Artist.from_discogs_artist(raw))
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("Artist.from_discogs_artist cache parse: %s", e)
             if artists:
                 return artists
 
@@ -417,8 +604,8 @@ class DiscogsClient:
             for raw in cached_results:
                 try:
                     albums.append(Album.from_discogs_release(raw))
-                except Exception:
-                    pass
+                except Exception as e:
+                    logger.debug("Album.from_discogs_release cache parse: %s", e)
             if albums:
                 return albums
 
@@ -502,10 +689,13 @@ class DiscogsClient:
         if cached and cached.get('title'):
             data = cached
         else:
-            # Try as master first (artist discography returns master IDs)
-            data = self._api_get(f'/masters/{release_id}')
-            if not data or not data.get('title'):
-                data = self._api_get(f'/releases/{release_id}')
+            # Hit the endpoint that matches the ID's type (tag-driven, no guessing).
+            data = None
+            for path in _discogs_album_endpoints(release_id):
+                data = self._api_get(path)
+                if data and data.get('title'):
+                    break
+                data = None
             if not data:
                 return None
             cache.store_entity('discogs', 'album', release_id, data)
@@ -539,22 +729,45 @@ class DiscogsClient:
 
     def get_artist_albums(self, artist_id: str, album_type: str = 'album,single', limit: int = 50) -> List[Album]:
         """Get releases by an artist. Prefers master releases, filters features."""
-        # First get the artist name for feature filtering
-        artist_data = self._api_get(f'/artists/{artist_id}')
-        artist_name = artist_data.get('name', '').lower() if artist_data else ''
+        cache = get_metadata_cache()
+        cached_payload = get_cached_artist_album_payload(cache, 'discogs', artist_id, album_type=album_type, limit=limit)
+        releases = cached_payload.get('_releases') if cached_payload else None
+        artist_name = ''
+        if cached_payload:
+            artist_name = str(cached_payload.get('artist_name') or '').lower()
 
-        data = self._api_get(f'/artists/{artist_id}/releases', {
-            'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
-        })
-        if not data or not data.get('releases'):
-            return []
+        if not isinstance(releases, list) or not releases:
+            # First get the artist name for feature filtering. Strip Discogs
+            # disambiguation suffix so feature-vs-primary matching below
+            # compares against the canonical name, not "Beyoncé*".
+            artist_data = self._api_get(f'/artists/{artist_id}')
+            artist_name = _clean_discogs_artist_name(
+                artist_data.get('name', '') if artist_data else ''
+            ).lower()
+
+            data = self._api_get(f'/artists/{artist_id}/releases', {
+                'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
+            })
+            if not data or not data.get('releases'):
+                return []
+            releases = data.get('releases') or []
+            store_artist_album_items(
+                cache,
+                'discogs',
+                artist_id,
+                releases,
+                album_type=album_type,
+                limit=limit,
+                items_field='_releases',
+                extra_fields={'artist_name': artist_name},
+            )
 
         # Separate masters from individual releases — prefer masters (canonical versions)
         masters = []
         releases_no_master = []
         master_titles = set()
 
-        for item in data['releases']:
+        for item in releases:
             # Skip non-main roles
             role = item.get('role', 'Main').lower()
             if role not in ('main', ''):
@@ -622,10 +835,13 @@ class DiscogsClient:
         if cached:
             return cached
 
-        # Try as master first (master IDs are used in artist discography)
-        data = self._api_get(f'/masters/{release_id}')
-        if not data or not data.get('tracklist'):
-            data = self._api_get(f'/releases/{release_id}')
+        # Hit the endpoint that matches the ID's type (tag-driven, no guessing).
+        data = None
+        for path in _discogs_album_endpoints(release_id):
+            data = self._api_get(path)
+            if data and data.get('tracklist'):
+                break
+            data = None
         if not data or not data.get('tracklist'):
             return None
 
@@ -637,7 +853,7 @@ class DiscogsClient:
             image_url = (primary or images[0]).get('uri')
 
         album_info = {
-            'id': str(data.get('id', release_id)),
+            'id': str(release_id),
             'name': data.get('title', ''),
             'images': [{'url': image_url, 'height': 600, 'width': 600}] if image_url else [],
             'release_date': str(data.get('year', '')) if data.get('year') else '',
@@ -646,7 +862,11 @@ class DiscogsClient:
         # Get artists
         artists_list = []
         if data.get('artists'):
-            artists_list = [{'name': a.get('name', '')} for a in data['artists'] if a.get('name')]
+            artists_list = [
+                {'name': _clean_discogs_artist_name(a.get('name', ''))}
+                for a in data['artists']
+                if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
+            ]
         if not artists_list:
             artists_list = [{'name': 'Unknown Artist'}]
 
@@ -675,7 +895,11 @@ class DiscogsClient:
             # Per-track artists or fall back to release artists
             track_artists = artists_list
             if t.get('artists'):
-                track_artists = [{'name': a.get('name', '')} for a in t['artists'] if a.get('name')]
+                track_artists = [
+                    {'name': _clean_discogs_artist_name(a.get('name', ''))}
+                    for a in t['artists']
+                    if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
+                ]
 
             tracks.append({
                 'id': f"{release_id}_t{track_num}",
@@ -718,9 +942,13 @@ class DiscogsClient:
         cached = cache.get_entity('discogs', 'album', str(release_id))
         if cached and cached.get('title'):
             return cached
-        data = self._api_get(f'/masters/{release_id}')
-        if not data or not data.get('title'):
-            data = self._api_get(f'/releases/{release_id}')
+        # Hit the endpoint that matches the ID's type (tag-driven, no guessing).
+        data = None
+        for path in _discogs_album_endpoints(release_id):
+            data = self._api_get(path)
+            if data and data.get('title'):
+                break
+            data = None
         if data:
             cache.store_entity('discogs', 'album', str(release_id), data)
         return data

core/discogs_worker.py

@@ -17,8 +17,8 @@ from typing import Optional, Dict, Any
 from datetime import datetime, timedelta
 from utils.logging_config import get_logger
 from database.music_database import MusicDatabase
-from core.discogs_client import DiscogsClient
-from core.worker_utils import interruptible_sleep, set_album_api_track_count
+from core.discogs_client import DiscogsClient, _discogs_album_kind, _tag_discogs_album_id
+from core.worker_utils import accept_artist_match, interruptible_sleep, set_album_api_track_count
 
 logger = get_logger("discogs_worker")
 
@@ -174,6 +174,16 @@ class DiscogsWorker:
             conn = self.db._get_connection()
             cursor = conn.cursor()
 
+            # Pinned-group override (Manage Enrichment Workers): process one
+            # entity type first, then fall through to the normal chain. Discogs
+            # has no track endpoint, so only artist/album are honored.
+            from core.worker_utils import read_enrichment_priority, priority_pending_item
+            _prio = read_enrichment_priority('discogs')
+            if _prio in ('artist', 'album'):
+                _pi = priority_pending_item(cursor, 'discogs', _prio)
+                if _pi:
+                    return _pi
+
             # Priority 1: Unattempted artists
             cursor.execute("""
                 SELECT id, name FROM artists
@@ -298,8 +308,8 @@ class DiscogsWorker:
             self.stats['errors'] += 1
             try:
                 self._mark_status(item['type'], item['id'], 'error')
-            except Exception:
-                pass
+            except Exception as e:
+                logger.debug("mark item status error failed: %s", e)
 
     def _get_existing_id(self, entity_type: str, entity_id) -> Optional[str]:
         """Check if entity already has a discogs_id."""
@@ -322,9 +332,13 @@ class DiscogsWorker:
             self.stats['not_found'] += 1
             return
 
-        # Find best match by name similarity
+        # Find best match by name similarity (skipping ids already claimed by
+        # a differently-named artist, so we don't create a shared/duplicate id).
         for result in results:
-            if self._name_matches(artist_name, result.name):
+            ok, reason = accept_artist_match(
+                self.db, 'discogs_id', result.id, artist_id, artist_name, result.name,
+            )
+            if ok:
                 # Fetch full artist detail (uses cache)
                 data = self.client._fetch_and_cache_artist(result.id)
                 if data:
@@ -422,7 +436,9 @@ class DiscogsWorker:
             conn = self.db._get_connection()
             cursor = conn.cursor()
 
-            discogs_id = str(data.get('id', ''))
+            # Tag the ID with its Discogs type so later re-fetches hit the right
+            # endpoint (master vs release share one numeric space).
+            discogs_id = _tag_discogs_album_id(data.get('id', ''), _discogs_album_kind(data))
             genres = json.dumps(data.get('genres', []))
             styles = json.dumps(data.get('styles', []))
             labels = data.get('labels', [])

core/discovery/beatport.py

@@ -0,0 +1,388 @@
+"""Background worker for Beatport chart discovery.
+
+`run_beatport_discovery_worker(url_hash, deps)` is the function the
+beatport discovery start-endpoint submits to its executor to match each
+Beatport chart track against Spotify (preferred) or iTunes (fallback):
+
+1. Pause enrichment workers (release shared resources).
+2. For each Beatport track:
+   - Cancellation gate (state['phase'] != 'discovering').
+   - Clean Beatport text (artist/title) of common annotations.
+   - Discovery cache lookup; cache hit short-circuits the search and
+     normalizes cached artists from ['str'] → [{'name': 'str'}].
+   - matching_engine search-query generation, with high min_confidence
+     (0.9) to avoid bad matches.
+   - Strategy 1: scored candidates from initial Spotify/iTunes searches.
+   - Strategy 4: extended search with limit=50 if no high-confidence
+     match found.
+   - On Spotify match: format artists as [{'name': str}] objects, pull
+     full album object from raw cache when available.
+   - On iTunes match: format with image_url-derived album.images entry.
+   - Save matched result to discovery cache when confidence >= 0.75.
+   - On miss: Wing It stub stored as 'wing-it' status (success ticked).
+3. After all tracks: phase='discovered', activity feed entry, sync
+   discovery results back to mirrored playlist via
+   `_sync_discovery_results_to_mirrored`.
+4. On error: state['phase']='fresh' + status='error'.
+5. Finally: resume enrichment workers.
+
+Lifted verbatim from web_server.py. Wide dependency surface (Spotify
+and iTunes clients, matching engine, multiple discovery helpers, state
+dict, mirrored sync) all injected via `BeatportDiscoveryDeps`.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any, Callable
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class BeatportDiscoveryDeps:
+    """Bundle of cross-cutting deps the Beatport discovery worker needs."""
+    beatport_chart_states: dict
+    spotify_client: Any
+    matching_engine: Any
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    clean_beatport_text: Callable[[str], str]
+    get_discovery_cache_key: Callable
+    get_database: Callable[[], Any]
+    validate_discovery_cache_artist: Callable
+    spotify_rate_limited: Callable[[], bool]
+    discovery_score_candidates: Callable
+    get_metadata_cache: Callable[[], Any]
+    build_discovery_wing_it_stub: Callable
+    add_activity_item: Callable
+    sync_discovery_results_to_mirrored: Callable
+
+
+def run_beatport_discovery_worker(url_hash, deps: BeatportDiscoveryDeps):
+    """Background worker for Beatport discovery process (Spotify preferred, iTunes fallback)"""
+    _ew_state = {}
+    try:
+        _ew_state = deps.pause_enrichment_workers('Beatport discovery')
+        state = deps.beatport_chart_states[url_hash]
+        chart = state['chart']
+        tracks = chart['tracks']
+
+        # Determine which provider to use
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        # Initialize fallback client if needed
+        itunes_client_instance = None
+        if not use_spotify:
+            itunes_client_instance = deps.get_metadata_fallback_client()
+
+        logger.info(f"Starting {discovery_source.upper()} discovery for {len(tracks)} Beatport tracks...")
+
+        # Store discovery source in state for frontend
+        state['discovery_source'] = discovery_source
+
+        # Process each track for discovery
+        for i, track in enumerate(tracks):
+            try:
+                # Check for cancellation
+                if state.get('phase') != 'discovering':
+                    logger.warning(f"Beatport discovery cancelled (phase changed to '{state.get('phase')}')")
+                    return
+
+                # Update progress
+                state['discovery_progress'] = int((i / len(tracks)) * 100)
+
+                # Get track info from Beatport data (frontend sends 'name' and 'artists' fields)
+                track_title = deps.clean_beatport_text(track.get('name', 'Unknown Title'))
+                track_artists = track.get('artists', ['Unknown Artist'])
+                # Handle artists - could be a list or string
+                if isinstance(track_artists, list):
+                    if len(track_artists) > 0 and isinstance(track_artists[0], str):
+                        # Handle case like ["CID,Taylr Renee"] - split on comma and clean
+                        track_artist = deps.clean_beatport_text(track_artists[0].split(',')[0].strip())
+                    else:
+                        track_artist = deps.clean_beatport_text(track_artists[0] if track_artists else 'Unknown Artist')
+                else:
+                    track_artist = deps.clean_beatport_text(str(track_artists))
+
+                logger.debug(f"Searching {discovery_source.upper()} for: '{track_artist}' - '{track_title}'")
+
+                # Check discovery cache first
+                cache_key = deps.get_discovery_cache_key(track_title, track_artist)
+                try:
+                    cache_db = deps.get_database()
+                    cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(track_artist, cached_match):
+                        logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_artist} - {track_title}")
+                        # Convert artists from ['str'] to [{'name': 'str'}] for Beatport frontend format
+                        beatport_artists = cached_match.get('artists', [])
+                        if beatport_artists and isinstance(beatport_artists[0], str):
+                            cached_match['artists'] = [{'name': a} for a in beatport_artists]
+                        result_entry = {
+                            'index': i,
+                            'beatport_track': {
+                                'title': track_title,
+                                'artist': track_artist
+                            },
+                            'status': 'found',
+                            'status_class': 'found',
+                            'discovery_source': discovery_source,
+                            'spotify_data': cached_match
+                        }
+                        state['spotify_matches'] += 1
+                        state['discovery_results'].append(result_entry)
+                        continue
+                except Exception as cache_err:
+                    logger.error(f"Cache lookup error: {cache_err}")
+
+                # Use matching engine for track matching
+                found_track = None
+                best_confidence = 0.0
+                best_raw_track = None
+                min_confidence = 0.9  # Higher threshold for Beatport to avoid bad matches
+
+                # Generate search queries using matching engine (with fallback)
+                try:
+                    temp_track = type('TempTrack', (), {
+                        'name': track_title,
+                        'artists': [track_artist],
+                        'album': None
+                    })()
+                    search_queries = deps.matching_engine.generate_download_queries(temp_track)
+                    logger.debug(f"Generated {len(search_queries)} search queries using matching engine")
+                except Exception as e:
+                    logger.error(f"Matching engine failed for Beatport, falling back to basic queries: {e}")
+                    if use_spotify:
+                        search_queries = [
+                            f"{track_artist} {track_title}",
+                            f'artist:"{track_artist}" track:"{track_title}"',
+                            f'"{track_artist}" "{track_title}"'
+                        ]
+                    else:
+                        search_queries = [
+                            f"{track_artist} {track_title}",
+                            f"{track_title} {track_artist}",
+                            track_title
+                        ]
+
+                for query_idx, search_query in enumerate(search_queries):
+                    try:
+                        logger.debug(f"Query {query_idx + 1}/{len(search_queries)}: {search_query} ({discovery_source.upper()})")
+
+                        search_results = None
+
+                        if use_spotify and not deps.spotify_rate_limited():
+                            search_results = deps.spotify_client.search_tracks(search_query, limit=10)
+                        else:
+                            search_results = itunes_client_instance.search_tracks(search_query, limit=10)
+
+                        if not search_results:
+                            continue
+
+                        # Score all results using the matching engine
+                        match, confidence, match_idx = deps.discovery_score_candidates(
+                            track_title, track_artist, 0, search_results
+                        )
+
+                        if match and confidence > best_confidence and confidence >= min_confidence:
+                            best_confidence = confidence
+                            found_track = match
+                            if use_spotify and match.id:
+                                _cache = deps.get_metadata_cache()
+                                best_raw_track = _cache.get_entity('spotify', 'track', match.id)
+                            else:
+                                best_raw_track = None
+                            logger.debug(f"New best Beatport match: {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                        if best_confidence >= 0.9:
+                            logger.debug(f"High confidence match found ({best_confidence:.3f}), stopping search")
+                            break
+
+                    except Exception as e:
+                        logger.debug(f"Error in {discovery_source.upper()} search for query '{search_query}': {e}")
+                        continue
+
+                # Strategy 4: Extended search with higher limit (last resort)
+                if not found_track:
+                    logger.debug("Beatport Strategy 4: Extended search with limit=50")
+                    query = f"{track_artist} {track_title}"
+                    if use_spotify:
+                        extended_results = deps.spotify_client.search_tracks(query, limit=50)
+                    else:
+                        extended_results = itunes_client_instance.search_tracks(query, limit=50)
+                    if extended_results:
+                        match, confidence, _ = deps.discovery_score_candidates(
+                            track_title, track_artist, 0, extended_results
+                        )
+                        if match and confidence >= min_confidence:
+                            found_track = match
+                            best_confidence = confidence
+                            logger.debug(f"Strategy 4 Beatport match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                if found_track:
+                    logger.info(f"Final Beatport match: {found_track.artists[0]} - {found_track.name} (confidence: {best_confidence:.3f})")
+                else:
+                    logger.warning(f"No suitable match found (best confidence was {best_confidence:.3f}, required {min_confidence:.3f})")
+
+                # Create result entry
+                result_entry = {
+                    'index': i,  # Add index for frontend table row identification
+                    'beatport_track': {
+                        'title': track_title,
+                        'artist': track_artist
+                    },
+                    'status': 'found' if found_track else 'not_found',
+                    'status_class': 'found' if found_track else 'not-found',
+                    'discovery_source': discovery_source,
+                    'confidence': best_confidence
+                }
+
+                if found_track:
+                    if use_spotify:
+                        # SPOTIFY result formatting
+                        # Debug: show available attributes
+                        logger.debug(f"Spotify track attributes: {dir(found_track)}")
+
+                        # Format artists correctly for frontend compatibility
+                        formatted_artists = []
+                        if isinstance(found_track.artists, list):
+                            # If it's already a list of strings, convert to objects with 'name' property
+                            for artist in found_track.artists:
+                                if isinstance(artist, str):
+                                    formatted_artists.append({'name': artist})
+                                else:
+                                    # If it's already an object, use as-is
+                                    formatted_artists.append(artist)
+                        else:
+                            # Single artist case
+                            formatted_artists = [{'name': str(found_track.artists)}]
+
+                        # Use full album object from raw Spotify data if available
+                        album_data = best_raw_track.get('album', {}) if best_raw_track else {}
+                        if not album_data:
+                            # Fallback to string album name
+                            album_data = {'name': found_track.album, 'album_type': 'album', 'release_date': getattr(found_track, 'release_date', '') or '', 'images': []}
+
+                        result_entry['spotify_data'] = {
+                            'name': found_track.name,
+                            'artists': formatted_artists,  # Now formatted as list of objects with 'name' property
+                            'album': album_data,  # Full album object with images
+                            'id': found_track.id,
+                            'source': 'spotify'
+                        }
+                    else:
+                        # ITUNES result formatting
+                        # Note: iTunes Track dataclass has 'artists' (list) and 'image_url', not 'artist' and 'artwork_url'
+                        result_artists = found_track.artists if hasattr(found_track, 'artists') else []
+                        result_artist = result_artists[0] if result_artists else 'Unknown'
+                        result_name = found_track.name if hasattr(found_track, 'name') else 'Unknown'
+                        album_name = found_track.album if hasattr(found_track, 'album') else 'Unknown Album'
+                        image_url = found_track.image_url if hasattr(found_track, 'image_url') else ''
+                        track_id = found_track.id if hasattr(found_track, 'id') else ''
+
+                        # Format artists as list of objects for frontend compatibility
+                        formatted_artists = [{'name': result_artist}]
+
+                        # Build album data with artwork
+                        album_data = {
+                            'name': album_name,
+                            'album_type': 'album',
+                            'release_date': getattr(found_track, 'release_date', '') or '',
+                            'images': [{'url': image_url, 'height': 300, 'width': 300}] if image_url else []
+                        }
+
+                        result_entry['spotify_data'] = {  # Use same key for frontend compatibility
+                            'name': result_name,
+                            'artists': formatted_artists,
+                            'album': album_data,
+                            'id': track_id,
+                            'source': discovery_source
+                        }
+
+                    state['spotify_matches'] += 1
+
+                    # Save to discovery cache (normalize artists from [{name:str}] to [str] for canonical format)
+                    if best_confidence >= 0.75:
+                        try:
+                            cache_data = dict(result_entry['spotify_data'])
+                            cache_artists = cache_data.get('artists', [])
+                            if cache_artists and isinstance(cache_artists[0], dict):
+                                cache_data['artists'] = [a.get('name', '') for a in cache_artists]
+                            # Extract image URL for discovery pool display
+                            if 'image_url' not in cache_data:
+                                _bp_album = cache_data.get('album', {})
+                                _bp_images = _bp_album.get('images', []) if isinstance(_bp_album, dict) else []
+                                cache_data['image_url'] = _bp_images[0].get('url', '') if _bp_images else ''
+                            cache_db = deps.get_database()
+                            cache_db.save_discovery_cache_match(
+                                cache_key[0], cache_key[1], discovery_source, best_confidence,
+                                cache_data, track_title, track_artist
+                            )
+                            logger.debug(f"CACHE SAVED: {track_artist} - {track_title} (confidence: {best_confidence:.3f})")
+                        except Exception as cache_err:
+                            logger.error(f"Cache save error: {cache_err}")
+
+                # Auto Wing It fallback for unmatched tracks
+                if result_entry.get('status_class') == 'not-found':
+                    bp_t = result_entry.get('beatport_track', {})
+                    stub = deps.build_discovery_wing_it_stub(
+                        bp_t.get('title', ''),
+                        bp_t.get('artist', ''),
+                    )
+                    result_entry['status'] = 'found'
+                    result_entry['status_class'] = 'wing-it'
+                    result_entry['spotify_data'] = stub
+                    result_entry['match_data'] = stub
+                    result_entry['wing_it_fallback'] = True
+                    result_entry['confidence'] = 0
+                    state['spotify_matches'] = state.get('spotify_matches', 0) + 1
+                    state['wing_it_count'] = state.get('wing_it_count', 0) + 1
+
+                state['discovery_results'].append(result_entry)
+
+                # Small delay to avoid rate limiting
+                time.sleep(0.1)
+
+            except Exception as e:
+                logger.error(f"Error processing Beatport track {i}: {e}")
+                # Add error result
+                state['discovery_results'].append({
+                    'index': i,  # Add index for frontend table row identification
+                    'beatport_track': {
+                        'title': track.get('name', 'Unknown'),  # Changed from 'title' to 'name' to match track structure
+                        'artist': track.get('artists', ['Unknown'])[0] if isinstance(track.get('artists'), list) else 'Unknown'
+                    },
+                    'status': 'error',
+                    'status_class': 'error',  # Add status class for CSS styling
+                    'error': str(e),
+                    'discovery_source': discovery_source
+                })
+
+        # Mark discovery as complete
+        state['discovery_progress'] = 100
+        state['phase'] = 'discovered'
+        state['status'] = 'discovered'
+
+        # Add activity for completion
+        chart_name = chart.get('name', 'Unknown Chart')
+        source_label = discovery_source.upper()
+        deps.add_activity_item("", f"Beatport Discovery Complete ({source_label})",
+                         f"'{chart_name}' - {state['spotify_matches']}/{len(tracks)} tracks found", "Now")
+
+        logger.info(f"Beatport discovery complete ({source_label}): {state['spotify_matches']}/{len(tracks)} tracks found")
+
+        # Sync discovery results back to mirrored playlist
+        deps.sync_discovery_results_to_mirrored('beatport', url_hash, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
+
+    except Exception as e:
+        logger.error(f"Error in Beatport discovery worker: {e}")
+        if url_hash in deps.beatport_chart_states:
+            deps.beatport_chart_states[url_hash]['status'] = 'error'
+            deps.beatport_chart_states[url_hash]['phase'] = 'fresh'
+    finally:
+        deps.resume_enrichment_workers(_ew_state, 'Beatport discovery')

core/discovery/deezer.py

@@ -0,0 +1,327 @@
+"""Background worker for Deezer playlist discovery.
+
+`run_deezer_discovery_worker(playlist_id, deps)` is the function the
+Deezer discovery start-endpoint submits to the deezer discovery executor
+to match each Deezer playlist track against Spotify (preferred) or iTunes
+(fallback):
+
+1. Pause enrichment workers (release shared resources).
+2. For each Deezer track:
+   - Cancellation gate (state['cancelled']).
+   - Discovery cache lookup; cache hit short-circuits the search.
+   - SimpleNamespace duck-type → `_search_spotify_for_tidal_track`
+     (shared search helper, returns tuple for Spotify or dict for iTunes).
+   - On match: build `match_data` (Spotify path preserves track_number /
+     disc_number from raw API data, image extracted from album images).
+   - Save to discovery cache.
+   - On miss: Wing It stub created from raw Deezer track data.
+3. After all tracks: phase='discovered', activity feed entry.
+4. Sync discovery results back to mirrored playlist via
+   `_sync_discovery_results_to_mirrored`.
+5. On error: state['phase']='error' + status with error string.
+6. Finally: resume enrichment workers.
+
+Lifted verbatim from web_server.py. Wide dependency surface (Spotify and
+iTunes clients, multiple metadata helpers, state dict, mirrored sync,
+shared tidal search helper) all injected via `DeezerDiscoveryDeps`.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+import types
+from dataclasses import dataclass
+from typing import Any, Callable
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class DeezerDiscoveryDeps:
+    """Bundle of cross-cutting deps the Deezer discovery worker needs."""
+    deezer_discovery_states: dict
+    spotify_client: Any
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    get_discovery_cache_key: Callable
+    get_database: Callable[[], Any]
+    validate_discovery_cache_artist: Callable
+    search_spotify_for_tidal_track: Callable
+    build_discovery_wing_it_stub: Callable
+    add_activity_item: Callable
+    sync_discovery_results_to_mirrored: Callable
+
+
+def run_deezer_discovery_worker(playlist_id, deps: DeezerDiscoveryDeps):
+    """Background worker for Deezer discovery process (Spotify preferred, iTunes fallback)"""
+    _ew_state = {}
+    try:
+        _ew_state = deps.pause_enrichment_workers('Deezer discovery')
+        state = deps.deezer_discovery_states[playlist_id]
+        playlist = state['playlist']
+
+        # Determine which provider to use
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        # Initialize fallback client if needed
+        itunes_client_instance = None
+        if not use_spotify:
+            itunes_client_instance = deps.get_metadata_fallback_client()
+
+        logger.info(f"Starting Deezer discovery for: {playlist['name']} (using {discovery_source.upper()})")
+
+        # Store discovery source in state for frontend
+        state['discovery_source'] = discovery_source
+
+        successful_discoveries = 0
+        tracks = playlist['tracks']
+
+        for i, deezer_track in enumerate(tracks):
+            if state.get('cancelled', False):
+                break
+
+            try:
+                track_name = deezer_track['name']
+                track_artists = deezer_track['artists']
+                track_id = deezer_track['id']
+                track_album = deezer_track.get('album', '')
+                track_duration_ms = deezer_track.get('duration_ms', 0)
+
+                logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
+
+                # Check discovery cache first
+                cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
+                try:
+                    cache_db = deps.get_database()
+                    cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
+                        logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
+                        # Extract display-friendly artist string from cached match
+                        cached_artists = cached_match.get('artists', [])
+                        if cached_artists:
+                            cached_artist_str = ', '.join(
+                                a if isinstance(a, str) else a.get('name', '') for a in cached_artists
+                            )
+                        else:
+                            cached_artist_str = ''
+                        cached_album = cached_match.get('album', '')
+                        if isinstance(cached_album, dict):
+                            cached_album = cached_album.get('name', '')
+
+                        result = {
+                            'deezer_track': {
+                                'id': track_id,
+                                'name': track_name,
+                                'artists': track_artists or [],
+                                'album': track_album,
+                                'duration_ms': track_duration_ms,
+                            },
+                            'spotify_data': cached_match,
+                            'match_data': cached_match,
+                            'status': 'Found',
+                            'status_class': 'found',
+                            'spotify_track': cached_match.get('name', ''),
+                            'spotify_artist': cached_artist_str,
+                            'spotify_album': cached_album,
+                            'spotify_id': cached_match.get('id', ''),
+                            'discovery_source': discovery_source,
+                            'index': i
+                        }
+                        successful_discoveries += 1
+                        state['spotify_matches'] = successful_discoveries
+                        state['discovery_results'].append(result)
+                        state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+                        continue
+                except Exception as cache_err:
+                    logger.error(f"Cache lookup error: {cache_err}")
+
+                # Create a SimpleNamespace duck-type object for _search_spotify_for_tidal_track
+                track_ns = types.SimpleNamespace(
+                    id=track_id,
+                    name=track_name,
+                    artists=track_artists,
+                    album=track_album,
+                    duration_ms=track_duration_ms
+                )
+
+                # Use the search function with appropriate provider
+                track_result = deps.search_spotify_for_tidal_track(
+                    track_ns,
+                    use_spotify=use_spotify,
+                    itunes_client=itunes_client_instance
+                )
+
+                # Create result entry
+                result = {
+                    'deezer_track': {
+                        'id': track_id,
+                        'name': track_name,
+                        'artists': track_artists or [],
+                        'album': track_album,
+                        'duration_ms': track_duration_ms,
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Not Found',
+                    'status_class': 'not-found',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'discovery_source': discovery_source
+                }
+
+                match_confidence = 0.0
+
+                if use_spotify and isinstance(track_result, tuple):
+                    # Spotify: Function returns (Track, raw_data, confidence)
+                    track_obj, raw_track_data, match_confidence = track_result
+                    album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
+                    # Ensure album has a name — fall back to track_obj.album if raw_data was missing
+                    if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
+                        album_obj['name'] = track_obj.album
+                    elif not album_obj and track_obj.album:
+                        album_obj = {'name': track_obj.album}
+                    # Ensure release_date is present (raw Spotify data has it, but fallback may not)
+                    if isinstance(album_obj, dict) and not album_obj.get('release_date'):
+                        album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
+                    # Extract image URL from album data or track object
+                    _album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
+                    _image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
+
+                    match_data = {
+                        'id': track_obj.id,
+                        'name': track_obj.name,
+                        'artists': track_obj.artists,
+                        'album': album_obj,
+                        'duration_ms': track_obj.duration_ms,
+                        'external_urls': track_obj.external_urls,
+                        'image_url': _image_url,
+                        'source': 'spotify'
+                    }
+                    # Preserve track_number/disc_number from raw Spotify API data
+                    if raw_track_data and raw_track_data.get('track_number'):
+                        match_data['track_number'] = raw_track_data['track_number']
+                    if raw_track_data and raw_track_data.get('disc_number'):
+                        match_data['disc_number'] = raw_track_data['disc_number']
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = track_obj.name
+                    result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
+                    result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
+                    result['spotify_id'] = track_obj.id
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                elif not use_spotify and track_result and isinstance(track_result, dict):
+                    # Fallback: Function returns a dict with track data (includes 'confidence' key)
+                    match_confidence = track_result.pop('confidence', 0.80)
+                    match_data = track_result
+                    match_data['source'] = discovery_source
+                    # Extract image URL from album images
+                    _fb_album = match_data.get('album', {})
+                    _fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
+                    if _fb_images and 'image_url' not in match_data:
+                        match_data['image_url'] = _fb_images[0].get('url', '')
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = match_data.get('name', '')
+                    itunes_artists = match_data.get('artists', [])
+                    result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
+                    result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
+                    result['spotify_id'] = match_data.get('id', '')
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                # Save to discovery cache if match found
+                if result['status_class'] == 'found' and result.get('match_data'):
+                    try:
+                        cache_db = deps.get_database()
+                        cache_db.save_discovery_cache_match(
+                            cache_key[0], cache_key[1], discovery_source, match_confidence,
+                            result['match_data'], track_name,
+                            track_artists[0] if track_artists else ''
+                        )
+                        logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
+                    except Exception as cache_err:
+                        logger.error(f"Cache save error: {cache_err}")
+
+                # Auto Wing It fallback for unmatched tracks
+                if result['status_class'] == 'not-found':
+                    deezer_t = result.get('deezer_track', {})
+                    stub = deps.build_discovery_wing_it_stub(
+                        deezer_t.get('name', ''),
+                        ', '.join(deezer_t.get('artists', [])),
+                        deezer_t.get('duration_ms', 0)
+                    )
+                    result['status'] = 'Wing It'
+                    result['status_class'] = 'wing-it'
+                    result['spotify_data'] = stub
+                    result['match_data'] = stub
+                    result['spotify_track'] = deezer_t.get('name', '')
+                    result['spotify_artist'] = ', '.join(deezer_t.get('artists', []))
+                    result['wing_it_fallback'] = True
+                    result['confidence'] = 0
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+                    state['wing_it_count'] = state.get('wing_it_count', 0) + 1
+
+                result['index'] = i
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+                # Add delay between requests
+                time.sleep(0.1)
+
+            except Exception as e:
+                logger.error(f"Error processing track {i+1}: {e}")
+                # Add error result
+                result = {
+                    'deezer_track': {
+                        'name': deezer_track.get('name', 'Unknown'),
+                        'artists': deezer_track.get('artists', []),
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Error',
+                    'status_class': 'error',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'error': str(e),
+                    'discovery_source': discovery_source,
+                    'index': i
+                }
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+        # Mark as complete
+        state['phase'] = 'discovered'
+        state['status'] = 'discovered'
+        state['discovery_progress'] = 100
+
+        # Add activity for discovery completion
+        source_label = discovery_source.upper()
+        deps.add_activity_item("", f"Deezer Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
+
+        logger.info(f"Deezer discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
+
+        # Sync discovery results back to mirrored playlist
+        deps.sync_discovery_results_to_mirrored('deezer', playlist_id, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
+
+    except Exception as e:
+        logger.error(f"Error in Deezer discovery worker: {e}")
+        if playlist_id in deps.deezer_discovery_states:
+            deps.deezer_discovery_states[playlist_id]['phase'] = 'error'
+            deps.deezer_discovery_states[playlist_id]['status'] = f'error: {str(e)}'
+    finally:
+        deps.resume_enrichment_workers(_ew_state, 'Deezer discovery')

core/discovery/endpoints.py

@@ -0,0 +1,767 @@
+"""Generic, source-agnostic helpers for the playlist-discovery route layer.
+
+The discovery/sync endpoints in ``web_server.py`` were copy-pasted once per
+source (Tidal, Deezer, Qobuz, Spotify-public, iTunes-link, YouTube,
+ListenBrainz, Beatport). The per-source copies differ only by a source label
+string and which ``<source>_discovery_states`` global they read. This module
+lifts the source-agnostic pieces into importable, unit-testable helpers so the
+route functions become thin wrappers — exactly preserving behavior (1:1).
+
+Each helper is lifted verbatim from its web_server.py counterpart; any
+per-source quirk that genuinely differs (e.g. Beatport's distinct result
+shape) is intentionally NOT routed through here and stays in its own function.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, Dict, List, Tuple
+
+from utils.logging_config import get_logger
+
+logger = get_logger("discovery.endpoints")
+
+
+def convert_results_to_spotify_tracks(
+    discovery_results: List[Dict[str, Any]],
+    source_label: str,
+) -> List[Dict[str, Any]]:
+    """Convert a source's discovery results into the Spotify-track dicts the
+    sync pipeline expects.
+
+    Lifted verbatim from the per-source ``convert_<source>_results_to_spotify_tracks``
+    functions (and the already-generic ``_convert_link_results_to_spotify_tracks``),
+    which were byte-identical apart from the ``source_label`` used in the log
+    line. Two input shapes are supported, matching the originals exactly:
+
+    - ``spotify_data`` (manual-fix shape): copied through, preserving optional
+      ``track_number`` / ``disc_number``.
+    - ``spotify_track`` + ``status_class == 'found'`` (auto-discovery shape):
+      rebuilt from the flat ``spotify_*`` fields.
+
+    Any result matching neither shape is skipped, identical to the originals.
+
+    NOTE: Beatport deliberately does NOT use this — its converter coerces
+    artist objects to strings and emits a different track shape (``source``
+    field, album dict), so it keeps its own implementation.
+    """
+    spotify_tracks: List[Dict[str, Any]] = []
+
+    for result in discovery_results:
+        # Support both data formats: spotify_data (manual fixes) and individual
+        # fields (automatic discovery).
+        if result.get('spotify_data'):
+            spotify_data = result['spotify_data']
+            track = {
+                'id': spotify_data['id'],
+                'name': spotify_data['name'],
+                'artists': spotify_data['artists'],
+                'album': spotify_data['album'],
+                'duration_ms': spotify_data.get('duration_ms', 0),
+            }
+            if spotify_data.get('track_number'):
+                track['track_number'] = spotify_data['track_number']
+            if spotify_data.get('disc_number'):
+                track['disc_number'] = spotify_data['disc_number']
+            spotify_tracks.append(track)
+        elif result.get('spotify_track') and result.get('status_class') == 'found':
+            spotify_tracks.append({
+                'id': result.get('spotify_id', 'unknown'),
+                'name': result.get('spotify_track', 'Unknown Track'),
+                'artists': [result.get('spotify_artist', 'Unknown Artist')] if result.get('spotify_artist') else ['Unknown Artist'],
+                'album': result.get('spotify_album', 'Unknown Album'),
+                'duration_ms': 0,
+            })
+
+    logger.info(f"Converted {len(spotify_tracks)} {source_label} matches to Spotify tracks for sync")
+    return spotify_tracks
+
+
+def cancel_sync(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    label: str,
+    not_found_message: str,
+    sync_lock: Any,
+    sync_states: Dict[str, Any],
+    active_sync_workers: Dict[str, Any],
+) -> Tuple[Dict[str, Any], int]:
+    """Cancel an in-progress sync for one discovery playlist.
+
+    1:1 lift of the byte-identical ``cancel_<source>_sync`` bodies (Tidal,
+    Deezer, Qobuz, Spotify-Public, iTunes-Link, YouTube, ListenBrainz). The
+    caller passes the already-resolved state key (ListenBrainz transforms it
+    via ``_lb_state_key`` first), the source ``label``, the exact 404 message
+    (iTunes-Link uses "iTunes Link not found", not "... playlist not found"),
+    and the shared sync infrastructure (so this stays free of web_server
+    globals / Flask).
+
+    Returns ``(payload_dict, status_code)``; the caller wraps in ``jsonify``.
+
+    Beatport is NOT routed here — it cancels a stored ``sync_future`` and
+    returns a different payload.
+    """
+    try:
+        if key not in states:
+            # Idempotent: the live discovery state is gone (a restart wiped the
+            # in-memory state, or it was already cancelled). Cancelling a sync
+            # that isn't running is a no-op SUCCESS, not a 404 — otherwise a
+            # mirrored playlist (e.g. a ListenBrainz weekly) whose state vanished
+            # is permanently wedged with "playlist not found" and can never be
+            # re-synced or dismissed (#702).
+            return {"success": True, "message": f"No active {label} sync to cancel"}, 200
+
+        state = states[key]
+        state['last_accessed'] = time.time()
+        sync_playlist_id = state.get('sync_playlist_id')
+
+        if sync_playlist_id:
+            with sync_lock:
+                sync_states[sync_playlist_id] = {"status": "cancelled"}
+            if sync_playlist_id in active_sync_workers:
+                del active_sync_workers[sync_playlist_id]
+
+        state['phase'] = 'discovered'
+        state['sync_playlist_id'] = None
+        state['sync_progress'] = {}
+
+        return {"success": True, "message": f"{label} sync cancelled"}, 200
+    except Exception as e:
+        logger.error(f"Error cancelling {label} sync: {e}")
+        return {"error": str(e)}, 500
+
+
+def delete_playlist_state(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    label: str,
+    not_found_message: str,
+) -> Tuple[Dict[str, Any], int]:
+    """Delete a discovery playlist's state entry, cancelling any active
+    discovery first.
+
+    1:1 lift of the byte-identical ``delete_<source>_playlist`` bodies
+    (Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
+
+    The iTunes-Link / YouTube / ListenBrainz / Beatport deletes intentionally
+    keep their own bodies — they differ in success message, info-log wording,
+    name extraction, and/or key transform.
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        state = states[key]
+        if 'discovery_future' in state and state['discovery_future']:
+            state['discovery_future'].cancel()
+        del states[key]
+
+        logger.info(f"Deleted {label} playlist state: {key}")
+        return {"success": True, "message": "Playlist deleted"}, 200
+    except Exception as e:
+        logger.error(f"Error deleting {label} playlist: {e}")
+        return {"error": str(e)}, 500
+
+
+# --- playlist-name accessors -------------------------------------------------
+# The per-source sync-status handlers read the display name three different
+# ways. Each is reproduced verbatim so the 1:1 behavior (including which ones
+# raise vs. fall back to 'Unknown Playlist') is preserved.
+
+def playlist_name_attr_or_unknown(state: Dict[str, Any]) -> str:
+    """Tidal: playlist is an object — use ``.name`` or 'Unknown Playlist'."""
+    pl = state.get('playlist')
+    return pl.name if pl and hasattr(pl, 'name') else 'Unknown Playlist'
+
+
+def playlist_name_strict(state: Dict[str, Any]) -> str:
+    """Deezer / Qobuz / Spotify-Public / iTunes-Link: strict dict access —
+    raises (→ 500) if 'playlist' is missing, exactly like the originals."""
+    return state['playlist']['name']
+
+
+def playlist_name_safe(state: Dict[str, Any]) -> str:
+    """YouTube / ListenBrainz: safe dict access, defaulting to 'Unknown
+    Playlist'."""
+    return state.get('playlist', {}).get('name', 'Unknown Playlist')
+
+
+def playlist_name_obj(state: Dict[str, Any]) -> str:
+    """Tidal start-sync: playlist is an object — strict ``.name`` (raises if
+    absent, exactly like the original)."""
+    return state['playlist'].name
+
+
+def playlist_image_obj(state: Dict[str, Any]) -> str:
+    """Tidal: ``getattr(playlist, 'image_url', '')`` (object attribute)."""
+    return getattr(state['playlist'], 'image_url', '')
+
+
+def playlist_image_dict(state: Dict[str, Any]) -> str:
+    """Deezer/Qobuz/Spotify-Public/YouTube: ``playlist.get('image_url', '')``
+    (dict access)."""
+    return state['playlist'].get('image_url', '')
+
+
+def get_sync_status(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    not_found_message: str,
+    error_label: str,
+    activity_subject: str,
+    playlist_name_getter,
+    sync_lock: Any,
+    sync_states: Dict[str, Any],
+    add_activity_item,
+) -> Tuple[Dict[str, Any], int]:
+    """Report sync status for one discovery playlist, posting an activity-feed
+    item when the sync finishes or errors.
+
+    1:1 lift of the ``get_<source>_sync_status`` bodies (Tidal, Deezer, Qobuz,
+    Spotify-Public, iTunes-Link, YouTube, ListenBrainz). Per-source variation
+    is captured by the parameters:
+
+    - ``not_found_message`` — the 404 string (iTunes-Link drops "playlist").
+    - ``error_label`` — used in the except log ("Error getting <X> sync status").
+    - ``activity_subject`` — the activity-feed prefix; note Spotify-Public uses
+      "Spotify Link playlist" while its error_label is "Spotify Public".
+    - ``playlist_name_getter`` — one of the accessors above (attr/strict/safe);
+      the strict one can raise, matching the originals (→ 500). The state's
+      phase/sync_progress are mutated BEFORE the name is read, so a raising
+      getter leaves the same partial mutation the original did.
+
+    Beatport is NOT routed here — it returns a different payload (``status``
+    not ``sync_status``, includes ``sync_id``, no lock, ``chart`` key).
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        state = states[key]
+        state['last_accessed'] = time.time()
+        sync_playlist_id = state.get('sync_playlist_id')
+
+        if not sync_playlist_id:
+            return {"error": "No sync in progress"}, 404
+
+        with sync_lock:
+            sync_state = sync_states.get(sync_playlist_id, {})
+
+        response = {
+            'phase': state['phase'],
+            'sync_status': sync_state.get('status', 'unknown'),
+            'progress': sync_state.get('progress', {}),
+            'complete': sync_state.get('status') == 'finished',
+            'error': sync_state.get('error'),
+        }
+
+        if sync_state.get('status') == 'finished':
+            state['phase'] = 'sync_complete'
+            state['sync_progress'] = sync_state.get('progress', {})
+            playlist_name = playlist_name_getter(state)
+            add_activity_item("", "Sync Complete", f"{activity_subject} '{playlist_name}' synced successfully", "Now")
+        elif sync_state.get('status') == 'error':
+            state['phase'] = 'discovered'
+            playlist_name = playlist_name_getter(state)
+            add_activity_item("", "Sync Failed", f"{activity_subject} '{playlist_name}' sync failed", "Now")
+
+        return response, 200
+    except Exception as e:
+        logger.error(f"Error getting {error_label} sync status: {e}")
+        return {"error": str(e)}, 500
+
+
+def get_discovery_status(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    not_found_message: str,
+    error_label: str,
+) -> Tuple[Dict[str, Any], int]:
+    """Report real-time discovery progress/results for one playlist.
+
+    1:1 lift of the byte-identical ``get_<source>_discovery_status`` bodies.
+    Unlike sync-status, this shape is identical for ALL eight sources —
+    Beatport included — so it folds in too. Only the 404 message
+    (".../discovery not found" vs ".../playlist not found" vs "Beatport chart
+    not found") and the except-log label vary, both passed in. The caller
+    resolves the key (ListenBrainz via ``_lb_state_key``).
+
+    Returns ``(payload, status_code)``.
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        state = states[key]
+        state['last_accessed'] = time.time()
+
+        return {
+            'phase': state['phase'],
+            'status': state['status'],
+            'progress': state['discovery_progress'],
+            'spotify_matches': state['spotify_matches'],
+            'spotify_total': state['spotify_total'],
+            'results': state['discovery_results'],
+            'complete': state['phase'] == 'discovered',
+        }, 200
+    except Exception as e:
+        logger.error(f"Error getting {error_label} discovery status: {e}")
+        return {"error": str(e)}, 500
+
+
+def reset_playlist(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    label: str,
+    not_found_message: str,
+) -> Tuple[Dict[str, Any], int]:
+    """Reset a discovery playlist back to the 'fresh' phase, clearing all
+    discovery/sync data while preserving the original playlist payload.
+
+    1:1 lift of the byte-identical ``reset_<source>_playlist`` bodies
+    (Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
+
+    NOT folded in (genuinely divergent): YouTube (status -> 'parsed', no
+    download_process_id, logs the playlist name, "reset to fresh state"),
+    ListenBrainz (status -> 'cached', logs playlist title, returns
+    {"phase": "fresh"}), iTunes-Link (uses state.update, no info log, distinct
+    message). Those keep their own bodies.
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        state = states[key]
+        if 'discovery_future' in state and state['discovery_future']:
+            state['discovery_future'].cancel()
+
+        state['phase'] = 'fresh'
+        state['status'] = 'fresh'
+        state['discovery_results'] = []
+        state['discovery_progress'] = 0
+        state['spotify_matches'] = 0
+        state['sync_playlist_id'] = None
+        state['converted_spotify_playlist_id'] = None
+        state['download_process_id'] = None
+        state['sync_progress'] = {}
+        state['discovery_future'] = None
+        state['last_accessed'] = time.time()
+
+        logger.info(f"Reset {label} playlist to fresh: {key}")
+        return {"success": True, "message": "Playlist reset to fresh phase"}, 200
+    except Exception as e:
+        logger.error(f"Error resetting {label} playlist: {e}")
+        return {"error": str(e)}, 500
+
+
+def get_playlist_states(
+    states: Dict[str, Any],
+    *,
+    error_label: str,
+    info_log_label: str = None,
+) -> Tuple[Dict[str, Any], int]:
+    """Return all stored discovery states for a source as a list for frontend
+    card hydration (``{"states": [...]}``).
+
+    1:1 lift of the ``get_<source>_playlist_states`` bodies (Tidal, Deezer,
+    Qobuz, Spotify-Public, iTunes-Link), which build the same per-entry dict.
+    iTunes-Link is the only one without the "Returning N ..." info log, so
+    ``info_log_label`` is optional (pass None to suppress it, as iTunes did).
+
+    NOT folded in: the YouTube/ListenBrainz ``get_all_*_playlists`` endpoints —
+    they return ``{"playlists": [...]}`` (different key + fields: url/created_at,
+    no discovery_results) and filter mirrored/profile-scoped entries.
+    """
+    try:
+        result = []
+        current_time = time.time()
+
+        for key, state in states.items():
+            state['last_accessed'] = current_time
+            result.append({
+                'playlist_id': key,
+                'phase': state['phase'],
+                'status': state['status'],
+                'discovery_progress': state['discovery_progress'],
+                'spotify_matches': state['spotify_matches'],
+                'spotify_total': state['spotify_total'],
+                'discovery_results': state['discovery_results'],
+                'converted_spotify_playlist_id': state.get('converted_spotify_playlist_id'),
+                'download_process_id': state.get('download_process_id'),
+                'last_accessed': state['last_accessed'],
+            })
+
+        if info_log_label:
+            logger.info(f"Returning {len(result)} stored {info_log_label} playlist states for hydration")
+        return {"states": result}, 200
+    except Exception as e:
+        logger.error(f"Error getting {error_label} playlist states: {e}")
+        return {"error": str(e)}, 500
+
+
+def save_bubble_snapshot(
+    get_json,
+    *,
+    payload_key: str,
+    no_data_error: str,
+    snapshot_kind: str,
+    success_noun: str,
+    log_subject: str,
+    log_noun: str,
+    get_database,
+    get_current_profile_id,
+) -> Tuple[Dict[str, Any], int]:
+    """Persist a bubble/download snapshot for cross-refresh hydration.
+
+    1:1 lift of the four structurally-identical snapshot endpoints
+    (discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles),
+    which differ only by:
+
+    - ``payload_key`` ('downloads' for discover, 'bubbles' for the rest) and
+      its ``no_data_error`` message.
+    - ``snapshot_kind`` — the db.save_bubble_snapshot category.
+    - ``success_noun`` — fills "Snapshot saved with N <noun>".
+    - ``log_subject`` / ``log_noun`` — the info ("Saved <subject>: N <noun>")
+      and except ("Error saving <subject>") log lines.
+
+    Returns ``(payload, status_code)``. ``get_json`` is invoked inside the try
+    like the original ``request.json``.
+    """
+    try:
+        from datetime import datetime
+
+        data = get_json()
+        if not data or payload_key not in data:
+            return {'success': False, 'error': no_data_error}, 400
+
+        items = data[payload_key]
+
+        db = get_database()
+        db.save_bubble_snapshot(snapshot_kind, items, profile_id=get_current_profile_id())
+
+        count = len(items)
+        logger.info(f"Saved {log_subject}: {count} {log_noun}")
+
+        return {
+            'success': True,
+            'message': f'Snapshot saved with {count} {success_noun}',
+            'timestamp': datetime.now().isoformat(),
+        }, 200
+    except Exception as e:
+        logger.error(f"Error saving {log_subject}: {e}")
+        import traceback
+        traceback.print_exc()
+        return {'success': False, 'error': str(e)}, 500
+
+
+def update_playlist_phase(
+    states: Dict[str, Any],
+    key: str,
+    get_json,
+    *,
+    not_found_message: str,
+    error_label: str,
+    valid_phases: List[str],
+    apply_extra_fields: bool,
+) -> Tuple[Dict[str, Any], int]:
+    """Update a discovery playlist's phase (used when the modal closes, e.g. to
+    reset download_complete -> discovered).
+
+    1:1 lift of the ``update_<source>_playlist_phase`` bodies for the five
+    sources with the identical validation + full-message response (Tidal,
+    Deezer, Qobuz, Spotify-Public, YouTube). Per-source params:
+
+    - ``valid_phases`` — YouTube's list additionally includes 'parsed'.
+    - ``apply_extra_fields`` — Deezer/Qobuz/Spotify-Public also persist
+      download_process_id / converted_spotify_playlist_id from the body;
+      Tidal/YouTube do NOT (so pass False to keep them 1:1).
+    - ``not_found_message`` / ``error_label``; ``get_json`` invoked inside the
+      try like the original ``request.get_json()``.
+
+    Returns ``(payload, status_code)``.
+
+    NOT folded in: iTunes-Link — it uses ``data.get('phase')`` (no separate
+    "Phase not provided" 400) and returns a no-message payload.
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        data = get_json()
+        if not data or 'phase' not in data:
+            return {"error": "Phase not provided"}, 400
+
+        new_phase = data['phase']
+        if new_phase not in valid_phases:
+            return {"error": f"Invalid phase. Must be one of: {', '.join(valid_phases)}"}, 400
+
+        state = states[key]
+        old_phase = state.get('phase', 'unknown')
+        state['phase'] = new_phase
+        state['last_accessed'] = time.time()
+
+        if apply_extra_fields:
+            if 'download_process_id' in data:
+                state['download_process_id'] = data['download_process_id']
+            if 'converted_spotify_playlist_id' in data:
+                state['converted_spotify_playlist_id'] = data['converted_spotify_playlist_id']
+
+        logger.info(f"Updated {error_label} playlist {key} phase: {old_phase} → {new_phase}")
+        return {"success": True, "message": f"Phase updated to {new_phase}", "old_phase": old_phase, "new_phase": new_phase}, 200
+    except Exception as e:
+        logger.error(f"Error updating {error_label} playlist phase: {e}")
+        return {"error": str(e)}, 500
+
+
+def first_artist_str_or_obj(original_track: Dict[str, Any]) -> str:
+    """Tidal: first artist from an artists list that may hold strings OR
+    objects ({'name': ...}); '' when empty."""
+    artists = original_track.get('artists', [])
+    if artists:
+        return artists[0] if isinstance(artists[0], str) else artists[0].get('name', '')
+    return ''
+
+
+def first_artist_plain(original_track: Dict[str, Any]) -> str:
+    """Deezer/Qobuz/Spotify-Public: first artist assuming a list of strings;
+    '' when empty."""
+    artists = original_track.get('artists', [])
+    return artists[0] if artists else ''
+
+
+def update_discovery_match(
+    states: Dict[str, Any],
+    get_json,
+    *,
+    source_log_label: str,
+    error_label: str,
+    original_track_key: str,
+    original_artist_getter,
+    join_artist_names,
+    extract_artist_name,
+    build_fix_modal_spotify_data,
+    get_discovery_cache_key,
+    get_database,
+    get_active_discovery_source,
+) -> Tuple[Dict[str, Any], int]:
+    """Apply a manually-selected Spotify track to a discovery result (the
+    fix-modal flow) and persist it to the discovery cache.
+
+    1:1 lift of the ``update_<source>_discovery_match`` bodies for the four
+    sources with the identical structure (Tidal, Deezer, Qobuz, Spotify-Public).
+    Per-source pieces are params:
+
+    - ``source_log_label`` (lowercase, e.g. "tidal") for the "Manual match
+      updated: ..." line; ``error_label`` for the except log.
+    - ``original_track_key`` — the raw-source track key on the result
+      ('tidal_track', 'deezer_track', ...).
+    - ``original_artist_getter`` — Tidal handles string-or-object artists
+      (``first_artist_str_or_obj``); the rest assume strings
+      (``first_artist_plain``).
+    - the web_server helpers (join/extract artist, build_fix_modal_spotify_data,
+      cache-key, get_database, active-discovery-source) are injected so this
+      stays free of those globals.
+    - ``get_json`` is called INSIDE the try (like the original's
+      ``request.get_json()``) so a malformed body yields the same 500.
+
+    Returns ``(payload, status_code)``.
+
+    NOT folded in: iTunes-Link (saves spotify_data directly via a different
+    cache signature), YouTube (multi-key original_track fallback), ListenBrainz
+    (entirely different unmatch-capable structure, no cache write), Beatport.
+    """
+    try:
+        data = get_json()
+        identifier = data.get('identifier')
+        track_index = data.get('track_index')
+        spotify_track = data.get('spotify_track')
+
+        if not identifier or track_index is None or not spotify_track:
+            return {'error': 'Missing required fields'}, 400
+
+        state = states.get(identifier)
+        result = None
+        if state:
+            if track_index >= len(state['discovery_results']):
+                return {'error': 'Invalid track index'}, 400
+
+            result = state['discovery_results'][track_index]
+            old_status = result.get('status')
+
+            result['status'] = 'Found'
+            result['status_class'] = 'found'
+            result['spotify_track'] = spotify_track['name']
+            result['spotify_artist'] = join_artist_names(spotify_track['artists']) if isinstance(spotify_track['artists'], list) else extract_artist_name(spotify_track['artists'])
+            result['spotify_album'] = spotify_track['album']
+            result['spotify_id'] = spotify_track['id']
+
+            duration_ms = spotify_track.get('duration_ms', 0)
+            if duration_ms:
+                minutes = duration_ms // 60000
+                seconds = (duration_ms % 60000) // 1000
+                result['duration'] = f"{minutes}:{seconds:02d}"
+            else:
+                result['duration'] = '0:00'
+
+            result['spotify_data'] = build_fix_modal_spotify_data(spotify_track)
+            result['wing_it_fallback'] = False
+            result['manual_match'] = True
+
+            if old_status != 'found' and old_status != 'Found':
+                state['spotify_matches'] = state.get('spotify_matches', 0) + 1
+
+            logger.info(f"Manual match updated: {source_log_label} - {identifier} - track {track_index}")
+            logger.info(f"   → {result['spotify_artist']} - {result['spotify_track']}")
+
+            original_track = result.get(original_track_key, {})
+            original_name = original_track.get('name', spotify_track['name'])
+            original_artist = original_artist_getter(original_track)
+        else:
+            # #843: the in-memory discovery state can be gone — a server restart,
+            # or an imported playlist that wasn't discovered in THIS process —
+            # while the card is still shown from persisted data. The DURABLE part
+            # of a manual fix (writing the match to the discovery cache so future
+            # syncs resolve it) doesn't need the in-memory state, only the original
+            # track's name + artist, which the client now sends. Fall back to those
+            # instead of 404ing the fix into uselessness.
+            original_name = (data.get('original_name') or '').strip()
+            original_artist = (data.get('original_artist') or '').strip()
+            if not original_name and not original_artist:
+                return {'error': 'Discovery state not found'}, 404
+            if not original_name:
+                original_name = spotify_track['name']
+            # Key the cache by the FIRST artist — every in-memory + sync path uses
+            # artists[0], but the client may send a joined "A, B, C" string. Without
+            # this, a multi-artist track would save under a key the sync never looks
+            # up (full string ≠ first artist), so the fix would silently never apply.
+            if original_artist:
+                original_artist = original_artist.split(',')[0].strip()
+            logger.info(
+                f"Manual match (no in-memory state) → discovery cache: "
+                f"{source_log_label} - {identifier} - '{original_name}' by '{original_artist}'"
+            )
+
+        try:
+            cache_key = get_discovery_cache_key(original_name, original_artist)
+            artists_list = spotify_track['artists']
+            if isinstance(artists_list, list):
+                artists_list = [a if isinstance(a, str) else a.get('name', '') for a in artists_list]
+            image_url = spotify_track.get('image_url') or ''
+            album_raw = spotify_track.get('album', '')
+            if isinstance(album_raw, dict):
+                album_obj = dict(album_raw)
+                if image_url and not album_obj.get('image_url'):
+                    album_obj['image_url'] = image_url
+                if image_url and not album_obj.get('images'):
+                    album_obj['images'] = [{'url': image_url}]
+            else:
+                album_obj = {'name': album_raw or ''}
+                if image_url:
+                    album_obj['image_url'] = image_url
+                    album_obj['images'] = [{'url': image_url}]
+
+            matched_data = {
+                'id': spotify_track['id'],
+                'name': spotify_track['name'],
+                'artists': artists_list,
+                'album': album_obj,
+                'duration_ms': spotify_track.get('duration_ms', 0),
+                'image_url': image_url,
+                'source': 'spotify',
+            }
+            cache_db = get_database()
+            cache_db.save_discovery_cache_match(
+                cache_key[0], cache_key[1], get_active_discovery_source(), 1.0, matched_data,
+                original_name, original_artist
+            )
+            logger.info(f"Manual fix saved to discovery cache: {original_name} by {original_artist}")
+        except Exception as cache_err:
+            logger.error(f"Error saving manual fix to discovery cache: {cache_err}")
+
+        return {'success': True, 'result': result}, 200
+    except Exception as e:
+        logger.error(f"Error updating {error_label} discovery match: {e}")
+        return {'error': str(e)}, 500
+
+
+def start_sync(
+    states: Dict[str, Any],
+    key: str,
+    *,
+    sync_id_prefix: str,
+    not_found_message: str,
+    not_ready_message: str,
+    convert_fn,
+    playlist_name_getter,
+    playlist_image_getter,
+    activity_label: str,
+    error_label: str,
+    sync_lock: Any,
+    sync_states: Dict[str, Any],
+    active_sync_workers: Dict[str, Any],
+    submit_sync_task,
+    add_activity_item,
+) -> Tuple[Dict[str, Any], int]:
+    """Kick off a playlist sync from a source's discovered Spotify matches.
+
+    1:1 lift of the ``start_<source>_sync`` bodies for the five sources with
+    the identical flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube). The
+    per-source pieces are parameters:
+
+    - ``sync_id_prefix`` — the ``f"{prefix}_{key}"`` sync id.
+    - ``convert_fn`` — the source's discovery->spotify-tracks converter.
+    - ``playlist_name_getter`` / ``playlist_image_getter`` — Tidal reads an
+      object (``.name`` / ``getattr``), the rest read a dict; lifted as the
+      ``playlist_name_obj``/``playlist_image_obj`` vs ``playlist_name_strict``/
+      ``playlist_image_dict`` accessors.
+    - ``activity_label`` vs ``error_label`` — these DIFFER for Spotify-Public:
+      activity says "Spotify Link Sync Started" while logs say "Spotify Public".
+    - ``submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks,
+      playlist_image_url) -> Future`` — wraps sync_executor/_run_sync_task/
+      get_current_profile_id so this stays free of those globals.
+
+    Returns ``(payload, status_code)``.
+
+    NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
+    task without an image arg), Beatport (extra debug logging, 'chart' key).
+    """
+    try:
+        if key not in states:
+            return {"error": not_found_message}, 404
+
+        state = states[key]
+        state['last_accessed'] = time.time()
+
+        if state['phase'] not in ['discovered', 'sync_complete', 'download_complete']:
+            return {"error": not_ready_message}, 400
+
+        spotify_tracks = convert_fn(state['discovery_results'])
+        if not spotify_tracks:
+            return {"error": "No Spotify matches found for sync"}, 400
+
+        sync_playlist_id = f"{sync_id_prefix}_{key}"
+        playlist_name = playlist_name_getter(state)
+
+        add_activity_item("", f"{activity_label} Sync Started", f"'{playlist_name}' - {len(spotify_tracks)} tracks", "Now")
+
+        state['phase'] = 'syncing'
+        state['sync_playlist_id'] = sync_playlist_id
+        state['sync_progress'] = {}
+
+        with sync_lock:
+            sync_states[sync_playlist_id] = {"status": "starting", "progress": {}}
+
+        playlist_image_url = playlist_image_getter(state)
+        future = submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks, playlist_image_url)
+        active_sync_workers[sync_playlist_id] = future
+
+        logger.info(f"Started {error_label} sync for: {playlist_name} ({len(spotify_tracks)} tracks)")
+        return {"success": True, "sync_playlist_id": sync_playlist_id}, 200
+    except Exception as e:
+        logger.error(f"Error starting {error_label} sync: {e}")
+        return {"error": str(e)}, 500

core/discovery/hero.py

@@ -0,0 +1,258 @@
+"""Discover Hero endpoint — lifted from web_server.py.
+
+The function body is byte-identical to the original. The
+``spotify_client`` proxy + helper shims let the body resolve its
+original names; the more complex ``_get_metadata_fallback_client``
+is injected via init() because it composes multiple registry helpers
+that web_server.py wires together.
+"""
+import logging
+
+from flask import g, jsonify
+
+from database.music_database import get_database
+from core.metadata.registry import get_primary_source, get_spotify_client
+
+logger = logging.getLogger(__name__)
+
+
+def get_current_profile_id() -> int:
+    """Mirror of web_server.get_current_profile_id — uses Flask g.
+
+    Catches RuntimeError too because reading `g` outside a request
+    context raises that (not AttributeError) — happens when this is
+    called from background threads (sync, automation, scanners)."""
+    try:
+        return g.profile_id
+    except (AttributeError, RuntimeError):
+        return 1
+
+
+def _get_active_discovery_source():
+    """Mirror of web_server._get_active_discovery_source — delegates to registry."""
+    return get_primary_source()
+
+
+class _SpotifyClientProxy:
+    """Resolves the global Spotify client lazily through core.metadata.registry."""
+
+    def __getattr__(self, name):
+        client = get_spotify_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        return get_spotify_client() is not None
+
+
+spotify_client = _SpotifyClientProxy()
+
+
+# Injected at runtime via init().
+_get_metadata_fallback_client = None
+
+
+def init(get_metadata_fallback_client_fn):
+    """Bind web_server's _get_metadata_fallback_client helper."""
+    global _get_metadata_fallback_client
+    _get_metadata_fallback_client = get_metadata_fallback_client_fn
+
+
+def get_discover_hero():
+    """Get featured similar artists for hero slideshow"""
+    try:
+        database = get_database()
+
+        # Determine active source
+        active_source = _get_active_discovery_source()
+        logger.info(f"Discover hero using source: {active_source}")
+
+        # Import fallback client for non-Spotify lookups
+        itunes_client = _get_metadata_fallback_client()
+
+        # Get top similar artists (excluding watchlist, cycled by last_featured)
+        # Fetch more than needed since strict source filtering may drop many
+        pid = get_current_profile_id()
+        logger.info(f"[Discover Hero] Profile ID: {pid}, Active source: {active_source}")
+        similar_artists = database.get_top_similar_artists(limit=200, profile_id=pid, require_source=active_source)
+
+        # FALLBACK: If no similar artists exist, use watchlist artists for Hero section
+        if not similar_artists:
+            logger.warning("[Discover Hero] No similar artists found, falling back to watchlist artists")
+            watchlist_artists = database.get_watchlist_artists(profile_id=pid)
+
+            if not watchlist_artists:
+                return jsonify({"success": True, "artists": [], "source": active_source})
+
+            # Convert watchlist artists to hero format
+            import random
+            shuffled_watchlist = list(watchlist_artists)
+            random.shuffle(shuffled_watchlist)
+
+            hero_artists = []
+            for artist in shuffled_watchlist[:10]:
+                if active_source == 'spotify':
+                    artist_id = artist.spotify_artist_id
+                elif active_source == 'deezer':
+                    artist_id = getattr(artist, 'deezer_artist_id', None) or artist.itunes_artist_id
+                elif active_source == 'musicbrainz':
+                    artist_id = getattr(artist, 'musicbrainz_artist_id', None) or artist.itunes_artist_id
+                else:
+                    artist_id = artist.itunes_artist_id
+                if not artist_id:
+                    continue
+
+                artist_data = {
+                    "spotify_artist_id": artist.spotify_artist_id,
+                    "itunes_artist_id": artist.itunes_artist_id,
+                    "artist_id": artist_id,
+                    "artist_name": artist.artist_name,
+                    "occurrence_count": 1,
+                    "similarity_rank": 1,
+                    "source": active_source,
+                    "is_watchlist": True
+                }
+
+                # Use cached image from watchlist — no API call needed
+                if hasattr(artist, 'image_url') and artist.image_url:
+                    artist_data['image_url'] = artist.image_url
+
+                hero_artists.append(artist_data)
+
+            logger.warning(f"[Discover Hero] Returning {len(hero_artists)} watchlist artists as fallback")
+            return jsonify({"success": True, "artists": hero_artists, "source": active_source, "fallback": "watchlist"})
+
+        # Artists are already filtered by source in SQL — no post-filter needed
+        valid_artists = list(similar_artists)
+
+        # FALLBACK: If no valid artists for fallback source, try to resolve IDs on-the-fly
+        if active_source in ('itunes', 'deezer', 'musicbrainz') and not valid_artists:
+            logger.warning(f"[{active_source} Fallback] No artists with {active_source} IDs found, attempting on-the-fly resolution for {len(similar_artists)} artists")
+            resolved_count = 0
+            for artist in similar_artists:
+                existing_id = getattr(artist, f'similar_artist_{active_source}_id', None) or (artist.similar_artist_itunes_id if active_source == 'itunes' else None)
+                if existing_id:
+                    valid_artists.append(artist)
+                    continue
+                # Try to resolve ID by name
+                try:
+                    resolve_client = itunes_client
+                    if active_source == 'musicbrainz':
+                        from core.metadata.registry import get_musicbrainz_client
+                        resolve_client = get_musicbrainz_client()
+                    search_results = resolve_client.search_artists(artist.similar_artist_name, limit=1)
+                    if search_results and len(search_results) > 0:
+                        resolved_id = search_results[0].id
+                        # Cache the resolved ID for future use
+                        if active_source == 'deezer':
+                            database.update_similar_artist_deezer_id(artist.id, resolved_id)
+                            artist.similar_artist_deezer_id = resolved_id
+                        elif active_source == 'musicbrainz':
+                            database.update_similar_artist_musicbrainz_id(artist.id, resolved_id)
+                            artist.similar_artist_musicbrainz_id = resolved_id
+                        else:
+                            database.update_similar_artist_itunes_id(artist.id, resolved_id)
+                            artist.similar_artist_itunes_id = resolved_id
+                        valid_artists.append(artist)
+                        resolved_count += 1
+                        logger.info(f"  [Resolved] {artist.similar_artist_name} -> {active_source} ID: {resolved_id}")
+                except Exception as resolve_err:
+                    logger.error(f"  [Failed] Could not resolve {active_source} ID for {artist.similar_artist_name}: {resolve_err}")
+                # Stop after 10 successful resolutions to avoid rate limiting
+                if len(valid_artists) >= 10:
+                    break
+            logger.warning(f"[{active_source} Fallback] Resolved {resolved_count} artists with IDs")
+
+        logger.info(f"[Discover Hero] Found {len(valid_artists)} valid artists for source: {active_source}")
+
+        # Filter out blacklisted artists
+        blacklisted = database.get_discovery_blacklist_names()
+        if blacklisted:
+            valid_artists = [a for a in valid_artists if a.similar_artist_name.lower() not in blacklisted]
+
+        # Take top 10 (already ordered by least-recently-featured, then quality)
+        similar_artists = valid_artists[:10]
+
+        # Convert to JSON format — use cached metadata, only fetch from API if missing
+        hero_artists = []
+        for artist in similar_artists:
+            # Use the ID for the active source, falling back to the other if needed
+            if active_source == 'spotify':
+                artist_id = artist.similar_artist_spotify_id or artist.similar_artist_itunes_id
+            elif active_source == 'deezer':
+                artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
+            elif active_source == 'musicbrainz':
+                artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
+            else:
+                artist_id = artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
+
+            artist_data = {
+                "spotify_artist_id": artist.similar_artist_spotify_id,
+                "itunes_artist_id": artist.similar_artist_itunes_id,
+                "musicbrainz_artist_id": getattr(artist, 'similar_artist_musicbrainz_id', None),
+                "artist_id": artist_id,
+                "artist_name": artist.similar_artist_name,
+                "occurrence_count": artist.occurrence_count,
+                "similarity_rank": artist.similarity_rank,
+                "source": active_source
+            }
+
+            # Use cached metadata if available
+            if artist.image_url:
+                artist_data['image_url'] = artist.image_url
+                artist_data['genres'] = artist.genres or []
+                artist_data['popularity'] = artist.popularity or 0
+            else:
+                # No cached metadata — fetch from API and cache for next time
+                try:
+                    if active_source == 'spotify' and artist.similar_artist_spotify_id:
+                        if spotify_client and spotify_client.is_authenticated():
+                            sp_artist = spotify_client.get_artist(artist.similar_artist_spotify_id)
+                            if sp_artist and sp_artist.get('images'):
+                                artist_data['artist_name'] = sp_artist.get('name', artist.similar_artist_name)
+                                artist_data['image_url'] = sp_artist['images'][0]['url'] if sp_artist['images'] else None
+                                artist_data['genres'] = sp_artist.get('genres', [])
+                                artist_data['popularity'] = sp_artist.get('popularity', 0)
+                                # Cache it
+                                database.update_similar_artist_metadata(
+                                    artist.id, artist_data.get('image_url'),
+                                    artist_data.get('genres'), artist_data.get('popularity')
+                                )
+                    elif active_source in ('itunes', 'deezer', 'musicbrainz'):
+                        if active_source == 'deezer':
+                            fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id
+                            fetch_client = itunes_client
+                        elif active_source == 'musicbrainz':
+                            fb_artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None)
+                            from core.metadata.registry import get_musicbrainz_client
+                            fetch_client = get_musicbrainz_client()
+                        else:
+                            fb_artist_id = artist.similar_artist_itunes_id
+                            fetch_client = itunes_client
+                        if fb_artist_id:
+                            fb_artist_data = fetch_client.get_artist(fb_artist_id)
+                            if fb_artist_data:
+                                artist_data['artist_name'] = fb_artist_data.get('name', artist.similar_artist_name)
+                                artist_data['image_url'] = fb_artist_data.get('images', [{}])[0].get('url') if fb_artist_data.get('images') else None
+                                artist_data['genres'] = fb_artist_data.get('genres', [])
+                                artist_data['popularity'] = fb_artist_data.get('popularity', 0)
+                                # Cache it
+                                database.update_similar_artist_metadata(
+                                    artist.id, artist_data.get('image_url'),
+                                    artist_data.get('genres'), artist_data.get('popularity')
+                                )
+                except Exception as img_err:
+                    logger.error(f"Could not fetch artist image: {img_err}")
+
+            hero_artists.append(artist_data)
+
+        # Mark these artists as featured so they cycle to the back of the queue
+        featured_names = [a["artist_name"] for a in hero_artists]
+        database.mark_artists_featured(featured_names)
+
+        return jsonify({"success": True, "artists": hero_artists, "source": active_source})
+
+    except Exception as e:
+        logger.error(f"Error getting discover hero: {e}")
+        return jsonify({"success": False, "error": str(e)}), 500

core/discovery/listenbrainz.py

@@ -0,0 +1,342 @@
+"""Background worker for ListenBrainz playlist discovery.
+
+`run_listenbrainz_discovery_worker(state_key, deps)` is the function
+the listenbrainz discovery start-endpoint submits to its executor to
+match each ListenBrainz playlist track against Spotify (preferred) or
+iTunes (fallback):
+
+1. Pause enrichment workers (release shared resources).
+2. For each ListenBrainz track:
+   - Cancellation gate (state['phase'] != 'discovering').
+   - Discovery cache lookup; cache hit short-circuits the search.
+   - Strategy 1: matching_engine search queries with confidence scoring.
+   - Strategy 2: swapped artist/title query.
+   - Strategy 3: album-based query (if album_name available).
+   - Strategy 4: extended search with limit=50.
+   - On match → save to discovery cache.
+   - On miss → build Wing It stub from raw source data.
+3. After loop: phase='discovered', activity feed entry.
+4. On error → state['status']='error', phase='fresh'.
+5. Finally: resume enrichment workers.
+
+Lifted verbatim from web_server.py. Wide dependency surface (Spotify
+and iTunes clients, matching engine, multiple metadata helpers, state
+dict, database access) all injected via `ListenbrainzDiscoveryDeps`.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from typing import Any, Callable
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ListenbrainzDiscoveryDeps:
+    """Bundle of cross-cutting deps the ListenBrainz discovery worker needs."""
+    listenbrainz_playlist_states: dict
+    spotify_client: Any
+    matching_engine: Any
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    get_discovery_cache_key: Callable
+    get_database: Callable[[], Any]
+    validate_discovery_cache_artist: Callable
+    extract_artist_name: Callable
+    spotify_rate_limited: Callable[[], bool]
+    discovery_score_candidates: Callable
+    get_metadata_cache: Callable[[], Any]
+    build_discovery_wing_it_stub: Callable
+    add_activity_item: Callable
+
+
+def run_listenbrainz_discovery_worker(state_key, deps: ListenbrainzDiscoveryDeps):
+    """Background worker for ListenBrainz music discovery process (Spotify preferred, iTunes fallback)"""
+    playlist_mbid = state_key.split(':', 1)[1] if ':' in state_key else state_key
+    _ew_state = {}
+    try:
+        _ew_state = deps.pause_enrichment_workers('ListenBrainz discovery')
+        state = deps.listenbrainz_playlist_states[state_key]
+        playlist = state['playlist']
+        tracks = playlist['tracks']
+
+        # Determine which provider to use (Spotify preferred, iTunes fallback)
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        # Get fallback client
+        itunes_client = deps.get_metadata_fallback_client()
+
+        logger.info(f"Starting {discovery_source} discovery for {len(tracks)} ListenBrainz tracks...")
+
+        # Store the discovery source in state
+        state['discovery_source'] = discovery_source
+
+        # Process each track for discovery
+        for i, track in enumerate(tracks):
+            try:
+                # Check for cancellation
+                if state.get('phase') != 'discovering':
+                    logger.warning(f"ListenBrainz discovery cancelled (phase changed to '{state.get('phase')}')")
+                    return
+
+                # Update progress
+                state['discovery_progress'] = int((i / len(tracks)) * 100)
+
+                # Get cleaned track data from ListenBrainz
+                cleaned_title = track['track_name']
+                cleaned_artist = track['artist_name']
+                album_name = track.get('album_name', '')
+                duration_ms = track.get('duration_ms', 0)
+
+                logger.info(f"Searching {discovery_source} for: '{cleaned_artist}' - '{cleaned_title}'")
+
+                # Check discovery cache first
+                cache_key = deps.get_discovery_cache_key(cleaned_title, cleaned_artist)
+                try:
+                    cache_db = deps.get_database()
+                    cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(cleaned_artist, cached_match):
+                        logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {cleaned_artist} - {cleaned_title}")
+                        result = {
+                            'index': i,
+                            'lb_track': cleaned_title,
+                            'lb_artist': cleaned_artist,
+                            'status': 'Found',
+                            'status_class': 'found',
+                            'spotify_track': cached_match.get('name', ''),
+                            'spotify_artist': deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else '',
+                            'spotify_album': cached_match.get('album', {}).get('name', '') if isinstance(cached_match.get('album'), dict) else cached_match.get('album', ''),
+                            'duration': f"{int(duration_ms) // 60000}:{(int(duration_ms) % 60000) // 1000:02d}" if duration_ms else '0:00',
+                            'discovery_source': discovery_source,
+                            'matched_data': cached_match,
+                            'spotify_data': cached_match
+                        }
+                        state['spotify_matches'] += 1
+                        state['discovery_results'].append(result)
+                        continue
+                except Exception as cache_err:
+                    logger.error(f"Cache lookup error: {cache_err}")
+
+                # Try multiple search strategies using matching engine
+                matched_track = None
+                best_confidence = 0.0
+                best_raw_track = None
+                min_confidence = 0.9
+                source_duration = duration_ms or 0
+
+                # Strategy 1: Use matching_engine search queries
+                try:
+                    temp_track = type('TempTrack', (), {
+                        'name': cleaned_title,
+                        'artists': [cleaned_artist],
+                        'album': album_name if album_name else None
+                    })()
+                    search_queries = deps.matching_engine.generate_download_queries(temp_track)
+                    logger.info(f"Generated {len(search_queries)} search queries for ListenBrainz track")
+                except Exception as e:
+                    logger.error(f"Matching engine failed for ListenBrainz, falling back to basic query: {e}")
+                    search_queries = [f"{cleaned_artist} {cleaned_title}", cleaned_title]
+
+                for query_idx, search_query in enumerate(search_queries):
+                    try:
+                        logger.debug(f"ListenBrainz query {query_idx + 1}/{len(search_queries)}: {search_query}")
+
+                        search_results = None
+
+                        if use_spotify and not deps.spotify_rate_limited():
+                            search_results = deps.spotify_client.search_tracks(search_query, limit=10)
+                        else:
+                            search_results = itunes_client.search_tracks(search_query, limit=10)
+
+                        if not search_results:
+                            continue
+
+                        # Score all results using the matching engine
+                        match, confidence, match_idx = deps.discovery_score_candidates(
+                            cleaned_title, cleaned_artist, source_duration, search_results
+                        )
+
+                        if match and confidence > best_confidence and confidence >= min_confidence:
+                            best_confidence = confidence
+                            matched_track = match
+                            if use_spotify and match.id:
+                                _cache = deps.get_metadata_cache()
+                                best_raw_track = _cache.get_entity('spotify', 'track', match.id)
+                            else:
+                                best_raw_track = None
+                            logger.info(f"New best ListenBrainz match: {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                        if best_confidence >= 0.9:
+                            logger.info(f"High confidence ListenBrainz match found ({best_confidence:.3f}), stopping search")
+                            break
+
+                    except Exception as e:
+                        logger.debug(f"Error in ListenBrainz search for query '{search_query}': {e}")
+                        continue
+
+                if matched_track:
+                    logger.info(f"Strategy 1 ListenBrainz match: {matched_track.artists[0]} - {matched_track.name} (confidence: {best_confidence:.3f})")
+
+                # Strategy 2: Swapped search (if first failed) - score results properly
+                if not matched_track:
+                    logger.info("ListenBrainz Strategy 2: Trying swapped search (artist/title reversed)")
+                    if use_spotify:
+                        query = f"artist:{cleaned_title} track:{cleaned_artist}"
+                        fallback_results = deps.spotify_client.search_tracks(query, limit=5)
+                    else:
+                        query = f"{cleaned_title} {cleaned_artist}"
+                        fallback_results = itunes_client.search_tracks(query, limit=5)
+                    if fallback_results:
+                        match, confidence, _ = deps.discovery_score_candidates(
+                            cleaned_title, cleaned_artist, source_duration, fallback_results
+                        )
+                        if match and confidence >= min_confidence:
+                            matched_track = match
+                            best_confidence = confidence
+                            logger.info(f"Strategy 2 ListenBrainz match (swapped): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                # Strategy 3: Album-based search (if still failed and we have album name) - score results properly
+                if not matched_track and album_name:
+                    logger.info(f"ListenBrainz Strategy 3: Trying album-based search: '{cleaned_artist} {album_name} {cleaned_title}'")
+                    if use_spotify:
+                        query = f"artist:{cleaned_artist} album:{album_name} track:{cleaned_title}"
+                        fallback_results = deps.spotify_client.search_tracks(query, limit=5)
+                    else:
+                        query = f"{cleaned_artist} {album_name} {cleaned_title}"
+                        fallback_results = itunes_client.search_tracks(query, limit=5)
+                    if fallback_results:
+                        match, confidence, _ = deps.discovery_score_candidates(
+                            cleaned_title, cleaned_artist, source_duration, fallback_results
+                        )
+                        if match and confidence >= min_confidence:
+                            matched_track = match
+                            best_confidence = confidence
+                            logger.info(f"Strategy 3 ListenBrainz match (album): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                # Strategy 4: Extended search with higher limit (last resort)
+                if not matched_track:
+                    logger.info("ListenBrainz Strategy 4: Extended search with limit=50")
+                    query = f"{cleaned_artist} {cleaned_title}"
+                    if use_spotify:
+                        extended_results = deps.spotify_client.search_tracks(query, limit=50)
+                    else:
+                        extended_results = itunes_client.search_tracks(query, limit=50)
+                    if extended_results:
+                        match, confidence, _ = deps.discovery_score_candidates(
+                            cleaned_title, cleaned_artist, source_duration, extended_results
+                        )
+                        if match and confidence >= min_confidence:
+                            matched_track = match
+                            best_confidence = confidence
+                            logger.info(f"Strategy 4 ListenBrainz match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                # Create result entry
+                result = {
+                    'index': i,
+                    'lb_track': cleaned_title,
+                    'lb_artist': cleaned_artist,
+                    'status': 'Found' if matched_track else 'Not Found',
+                    'status_class': 'found' if matched_track else 'not-found',
+                    'spotify_track': matched_track.name if matched_track else '',
+                    'spotify_artist': deps.extract_artist_name(matched_track.artists[0]) if matched_track else '',
+                    'spotify_album': matched_track.album if matched_track else '',
+                    'duration': f"{int(duration_ms) // 60000}:{(int(duration_ms) % 60000) // 1000:02d}" if duration_ms else '0:00',
+                    'discovery_source': discovery_source,
+                    'confidence': best_confidence
+                }
+
+                if matched_track:
+                    state['spotify_matches'] += 1
+
+                    # Build album data based on provider
+                    if use_spotify and best_raw_track:
+                        album_data = best_raw_track.get('album', {})
+                    else:
+                        album_data = {
+                            'name': matched_track.album,
+                            'album_type': 'album',
+                            'release_date': getattr(matched_track, 'release_date', '') or '',
+                            'images': [{'url': matched_track.image_url}] if hasattr(matched_track, 'image_url') and matched_track.image_url else []
+                        }
+
+                    # Extract image URL for discovery pool display
+                    _yt_album_images = album_data.get('images', [])
+                    _yt_image_url = _yt_album_images[0].get('url', '') if _yt_album_images else (getattr(matched_track, 'image_url', '') or '')
+
+                    result['matched_data'] = {
+                        'id': matched_track.id,
+                        'name': matched_track.name,
+                        'artists': matched_track.artists,
+                        'album': album_data,
+                        'duration_ms': matched_track.duration_ms,
+                        'image_url': _yt_image_url,
+                        'source': discovery_source
+                    }
+                    result['spotify_data'] = result['matched_data']
+
+                    # Save to discovery cache (only high-confidence matches)
+                    if best_confidence >= 0.7:
+                        try:
+                            cache_db = deps.get_database()
+                            cache_db.save_discovery_cache_match(
+                                cache_key[0], cache_key[1], discovery_source, best_confidence,
+                                result['matched_data'], cleaned_title, cleaned_artist
+                            )
+                            logger.info(f"CACHE SAVED: {cleaned_artist} - {cleaned_title} (confidence: {best_confidence:.3f})")
+                        except Exception as cache_err:
+                            logger.error(f"Cache save error: {cache_err}")
+
+                else:
+                    # Auto Wing It fallback — build stub from raw source data
+                    stub = deps.build_discovery_wing_it_stub(cleaned_title, cleaned_artist, duration_ms)
+                    result['status'] = 'Wing It'
+                    result['status_class'] = 'wing-it'
+                    result['spotify_track'] = cleaned_title
+                    result['spotify_artist'] = cleaned_artist
+                    result['spotify_album'] = ''
+                    result['matched_data'] = stub
+                    result['spotify_data'] = stub
+                    result['wing_it_fallback'] = True
+                    state['wing_it_count'] = state.get('wing_it_count', 0) + 1
+
+                state['discovery_results'].append(result)
+
+                logger.info(f"  {'' if matched_track else ''} Track {i+1}/{len(tracks)}: {result['status']}")
+
+            except Exception as e:
+                logger.error(f"Error processing track {i}: {e}")
+                result = {
+                    'index': i,
+                    'lb_track': track['track_name'],
+                    'lb_artist': track['artist_name'],
+                    'status': 'Error',
+                    'status_class': 'error',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'duration': '0:00'
+                }
+                state['discovery_results'].append(result)
+
+        # Complete discovery
+        state['phase'] = 'discovered'
+        state['status'] = 'complete'
+        state['discovery_progress'] = 100
+
+        playlist_name = playlist.get('name') or playlist.get('title') or 'Unknown Playlist'
+        source_label = discovery_source.upper()
+        deps.add_activity_item("", f"ListenBrainz Discovery Complete ({source_label})", f"'{playlist_name}' - {state['spotify_matches']}/{len(tracks)} tracks found", "Now")
+
+        logger.info(f"ListenBrainz discovery complete ({discovery_source}): {state['spotify_matches']}/{len(tracks)} tracks matched")
+
+    except Exception as e:
+        logger.error(f"Error in ListenBrainz discovery worker: {e}")
+        state['status'] = 'error'
+        state['phase'] = 'fresh'
+    finally:
+        deps.resume_enrichment_workers(_ew_state, 'ListenBrainz discovery')

core/discovery/listening_recommendations.py

@@ -0,0 +1,340 @@
+"""Listening-driven recommendation core (#913).
+
+PURE, side-effect-free ranking that turns "the artists you listen to most" plus
+"who's similar to each" into:
+
+  1. a consensus-ranked list of artists you'd probably love but don't own, and
+  2. an aggregated candidate-track list for a generated playlist.
+
+No DB / network / config here. The caller (the watchlist scanner) supplies the
+seeds (top-played artists), the ``similar_artists`` rows per seed, and the
+owned-artist set, then fetches top tracks for the winners. Keeping the decision
+logic in one pure place makes it fully unit-testable without the live stack and
+keeps the scan wiring thin — and additive, so it can't disturb existing flows.
+
+Scoring rationale (the "best in class" bit): a recommended artist's score is
+``Σ over the seeds that recommend it of (seed_weight × similarity)``. That single
+sum rewards all three signals at once — **consensus** (an artist endorsed by many
+of your seeds accumulates more terms), your **play weight** (heavier seeds push
+harder), and **similarity strength** — instead of a flat "appears in N lists".
+``seed_count`` is exposed separately for display ("because you like A, B, C") and
+as the adventurousness dial's lever (``min_seed_count``).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional, Sequence, Set
+
+
+def _norm(name: object) -> str:
+    return str(name or "").strip().lower()
+
+
+def _positive_float(value: object, default: float = 1.0) -> float:
+    try:
+        f = float(value)  # type: ignore[arg-type]
+    except (TypeError, ValueError):
+        return default
+    return f if f > 0 else default
+
+
+def _get(row: object, attr: str):
+    """Read a field from a dataclass row or a dict row."""
+    if isinstance(row, dict):
+        return row.get(attr)
+    return getattr(row, attr, None)
+
+
+def choose_mix_fetch_source(active_source: object, active_can_fetch: bool) -> str:
+    """Pick which source to fetch the "Listening Mix" top tracks from.
+
+    The mix is a list of (artist, title) pairs acquired via Soulseek, so the fetch source need
+    NOT match the user's active metadata source. Use the active source when it can fetch top
+    tracks itself (Spotify/Deezer); otherwise fall back to Deezer, whose public ``artist/{id}/top``
+    needs no auth and is available to every user — so iTunes / Discogs / MusicBrainz users still
+    get a full mix without switching sources. Pure.
+    """
+    if str(active_source or "").lower() in ("spotify", "deezer") and active_can_fetch:
+        return str(active_source).lower()
+    return "deezer"
+
+
+def names_match(a: object, b: object) -> bool:
+    """Strict artist-name equality after stripping case + non-alphanumerics.
+
+    Used to verify a name-search result before fetching that artist's top tracks, so the
+    "Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact
+    alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell".
+    Pure.
+    """
+    def _alnum(x: object) -> str:
+        return "".join(ch for ch in str(x or "").lower() if ch.isalnum())
+    na, nb = _alnum(a), _alnum(b)
+    return bool(na) and na == nb
+
+
+def similarity_from_rank(rank: object, max_rank: int = 10) -> float:
+    """Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight.
+
+    SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest
+    match). The ranker multiplies this into the score so a seed's *closest* matches count
+    for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0,
+    rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A
+    missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure.
+    """
+    try:
+        r = int(rank)
+    except (TypeError, ValueError):
+        return 1.0
+    floor = round(1.0 / max_rank, 4)
+    if r <= 1:
+        return 1.0
+    if r >= max_rank:
+        return floor
+    return round((max_rank - r + 1) / max_rank, 4)
+
+
+def build_recency_weighted_seeds(
+    top_artists: Sequence[dict],
+    recent_play_counts: Optional[Dict[str, float]] = None,
+    *,
+    recency_factor: float = 1.5,
+) -> List[dict]:
+    """Blend lifetime + recent play counts into seed weights — "what you're into NOW".
+
+    ``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a
+    lot *recently* outranks one you played a lot years ago, so the recommendations track
+    your current taste instead of your all-time history. ``recency_factor`` is the dial
+    (0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`.
+    Pure — the caller supplies both play-count maps from the listening history.
+    """
+    recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()}
+    out: List[dict] = []
+    for a in top_artists or ():
+        name = str(a.get("name") or "").strip()
+        if not name:
+            continue
+        lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0)))
+        boost = recency_factor * recent.get(_norm(name), 0.0)
+        out.append({"name": name, "weight": lifetime + boost})
+    return out
+
+
+def group_similars_by_seed(
+    seeds: Sequence[dict],
+    similar_rows: Sequence,
+    id_to_name: Dict[str, str],
+    *,
+    source_id_attr: str = "source_artist_id",
+    similar_name_attr: str = "similar_artist_name",
+    rank_attr: Optional[str] = None,
+) -> Dict[str, List[dict]]:
+    """Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``.
+
+    The stored rows key the similar artist by the SEED's source id (``source_artist_id``),
+    not its name, so :func:`rank_recommended_artists` can't consume them directly. This
+    resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id:
+    artist_name}`` for the library, built by the caller) and keeps only rows that resolve
+    to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O.
+
+    ``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that
+    is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id.
+    When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank`
+    and carried as ``score`` so closer matches weigh more; without it every similar comes out
+    score-less (the ranker then treats similarity as 1.0 — original behavior).
+    """
+    seed_names = {_norm(s.get("name")) for s in seeds}
+    seed_names.discard("")
+    id_to_norm = {str(k): _norm(v) for k, v in (id_to_name or {}).items()}
+
+    out: Dict[str, List[dict]] = {}
+    for row in similar_rows or ():
+        seed_name = id_to_norm.get(str(_get(row, source_id_attr) or ""), "")
+        if not seed_name or seed_name not in seed_names:
+            continue
+        sim_name = str(_get(row, similar_name_attr) or "").strip()
+        if not sim_name:
+            continue
+        entry = {"name": sim_name}
+        if rank_attr is not None:
+            entry["score"] = similarity_from_rank(_get(row, rank_attr))
+        out.setdefault(seed_name, []).append(entry)
+    return out
+
+
+@dataclass
+class RecommendedArtist:
+    """One artist recommended from your listening, with the why."""
+    name: str                                       # display name (first-seen casing)
+    score: float                                    # Σ seed_weight × similarity
+    seed_count: int                                 # distinct seeds endorsing it (consensus)
+    seeds: List[str] = field(default_factory=list)  # display names of those seeds
+
+
+def rank_recommended_artists(
+    seeds: Sequence[dict],
+    similars_by_seed: Dict[str, Sequence[dict]],
+    owned_artist_names: Optional[Set[str]] = None,
+    *,
+    limit: int = 30,
+    min_seed_count: int = 1,
+) -> List[RecommendedArtist]:
+    """Rank artists similar to your most-played by consensus + play weight + similarity.
+
+    Args:
+        seeds: ``[{'name': str, 'weight': float}]`` — your top-played artists.
+            ``weight`` (play count or any positive number) defaults to 1.0.
+        similars_by_seed: ``{seed_name_lower: [{'name': str, 'score': float}]}`` — the
+            similar-artist rows for each seed. ``score`` is optional (defaults 1.0).
+        owned_artist_names: lowercased names already in the library — excluded so the
+            result is artists you DON'T have. The seeds themselves are always excluded.
+        limit: max results.
+        min_seed_count: drop recommendations endorsed by fewer than N seeds — the
+            adventurousness dial's "Safer" end raises this for higher-confidence picks.
+
+    Returns up to ``limit`` :class:`RecommendedArtist`, highest score first.
+    """
+    owned = {_norm(a) for a in (owned_artist_names or set())}
+    seed_norms = {_norm(s.get("name")) for s in seeds}
+    seed_norms.discard("")
+    exclude = owned | seed_norms
+
+    acc: Dict[str, dict] = {}
+    for seed in seeds:
+        s_name = _norm(seed.get("name"))
+        if not s_name:
+            continue
+        s_display = str(seed.get("name") or "").strip()
+        weight = _positive_float(seed.get("weight", 1.0))
+        for sim in similars_by_seed.get(s_name, ()) or ():
+            a_norm = _norm(sim.get("name"))
+            if not a_norm or a_norm in exclude:
+                continue
+            sim_score = _positive_float(sim.get("score", 1.0))
+            row = acc.setdefault(
+                a_norm, {"name": str(sim.get("name") or "").strip(), "score": 0.0, "seeds": {}}
+            )
+            row["score"] += weight * sim_score
+            row["seeds"].setdefault(s_name, s_display)   # one seed counts once
+
+    out: List[RecommendedArtist] = []
+    floor = max(1, int(min_seed_count))
+    for row in acc.values():
+        seed_count = len(row["seeds"])
+        if seed_count < floor:
+            continue
+        out.append(RecommendedArtist(
+            name=row["name"],
+            score=round(row["score"], 6),
+            seed_count=seed_count,
+            seeds=list(row["seeds"].values()),
+        ))
+    out.sort(key=lambda r: (-r.score, -r.seed_count, r.name.lower()))
+    return out[:limit]
+
+
+def aggregate_candidate_tracks(
+    recommended_artists: Sequence[RecommendedArtist],
+    top_tracks_by_artist: Dict[str, Sequence[dict]],
+    owned_track_keys: Optional[Set] = None,
+    *,
+    per_artist: int = 3,
+    limit: int = 50,
+    exclude_owned: bool = True,
+) -> List[dict]:
+    """Build the candidate track list for the generated playlist.
+
+    Takes the top ``per_artist`` tracks from each recommended artist **in artist-rank
+    order**, dedups by ``(artist, title)``, optionally drops owned tracks (the
+    "discovery" flavor) and caps at ``limit``. Each returned track dict is the source
+    track plus ``_seed_artist`` (which recommended artist it came from).
+
+    Args:
+        recommended_artists: ranked output of :func:`rank_recommended_artists`.
+        top_tracks_by_artist: ``{artist_name_lower: [track_dict, ...]}`` — fetched by
+            the caller (Last.fm / source top tracks), NOT limited to a curated pool.
+        owned_track_keys: set of ``(artist_lower, title_lower)`` already in the library.
+        exclude_owned: drop tracks in ``owned_track_keys`` (discovery flavor). Set False
+            for a "replay" playlist of tracks you already own.
+    """
+    owned = owned_track_keys or set()
+    seen: Set = set()
+    out: List[dict] = []
+    for art in recommended_artists:
+        tracks = top_tracks_by_artist.get(_norm(art.name), ()) or ()
+        taken = 0
+        for t in tracks:
+            if taken >= per_artist:
+                break
+            title = str(t.get("name") or t.get("title") or "").strip()
+            if not title:
+                continue
+            key = (_norm(art.name), _norm(title))
+            if key in seen:
+                continue
+            if exclude_owned and key in owned:
+                continue
+            seen.add(key)
+            out.append({**t, "_seed_artist": art.name})
+            taken += 1
+        if len(out) >= limit:
+            break
+    return out[:limit]
+
+
+def to_mix_track(track: object, source: str) -> Optional[dict]:
+    """Shape one source "top tracks" API dict into the flat dict the Discover compact
+    playlist row renders + syncs (the "Listening Mix" #913 playlist).
+
+    Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the
+    same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``).
+    This flattens that into the renderer's field names (``track_name/artist_name/album_name/
+    album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and
+    sets the source-specific id field. Returns None for anything without a usable id/title so
+    the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can
+    dedup by title. Pure — no I/O.
+    """
+    if not isinstance(track, dict):
+        return None
+    tid = track.get("id")
+    name = str(track.get("name") or "").strip()
+    if not tid or not name:
+        return None
+    artists = track.get("artists") or []
+    artist_name = ""
+    if artists and isinstance(artists[0], dict):
+        artist_name = str(artists[0].get("name") or "").strip()
+    album = track.get("album") if isinstance(track.get("album"), dict) else {}
+    album_name = str(album.get("name") or "").strip()
+    images = album.get("images") or []
+    cover = images[0].get("url") if images and isinstance(images[0], dict) else None
+    out = {
+        "track_id": str(tid),
+        "name": name,                  # for aggregate_candidate_tracks dedup
+        "track_name": name,            # for the renderer
+        "artist_name": artist_name,
+        "album_name": album_name,
+        "album_cover_url": cover,
+        "duration_ms": track.get("duration_ms") or 0,
+        "track_data_json": track,      # full payload for sync/download
+        "source": source,
+    }
+    id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id",
+                "itunes": "itunes_track_id"}.get(source)
+    if id_field:
+        out[id_field] = str(tid)
+    return out
+
+
+__all__ = [
+    "RecommendedArtist",
+    "choose_mix_fetch_source",
+    "names_match",
+    "similarity_from_rank",
+    "build_recency_weighted_seeds",
+    "to_mix_track",
+    "group_similars_by_seed",
+    "rank_recommended_artists",
+    "aggregate_candidate_tracks",
+]

core/discovery/manual_match.py

@@ -0,0 +1,120 @@
+"""Helpers for Fix-popup manual match persistence.
+
+When the user manually fixes a mirrored-playlist discovery via the Fix
+popup, two questions land at the web_server route layer that are easier
+to test in isolation:
+
+1. *Which metadata source did the manual match come from?* — the popup
+   cascade queries the user's primary source first, then Spotify /
+   Deezer / iTunes / MusicBrainz as fallbacks; each search endpoint
+   stamps `source` on its rows but the MBID-paste lookup uses a lean
+   flat shape that doesn't carry it. `derive_manual_match_provider`
+   collapses the fallback chain into a single string.
+
+2. *Should the discovery layer re-run for this track when the current
+   active provider differs from the cached one?* — re-running silently
+   overwrites the user's deliberate pick with whatever the auto-search
+   ranks first, so manual matches are exempt regardless of provider
+   drift. `is_drifted_for_redo` encapsulates the decision.
+
+3. *Should the Playlist Pipeline pre-scan (re)discover this track at all?*
+   — `should_rediscover` encapsulates that gate, with the manual match
+   checked FIRST so a leftover Wing It flag can't override the user's pick.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, Optional
+
+
+def derive_manual_match_provider(
+    payload_track: Dict[str, Any],
+    active_provider: Optional[str],
+) -> str:
+    """Return the provider string to stamp on a manually-fixed match.
+
+    Resolution order:
+        1. ``payload_track['source']`` — every *_search_tracks endpoint
+           sets this; the MBID-paste path doesn't.
+        2. ``active_provider`` — what the user has configured as their
+           primary discovery source.
+        3. ``'spotify'`` — last-ditch default matching the historic
+           hardcode (so behaviour is identical when both upstream
+           signals are absent).
+    """
+    if not isinstance(payload_track, dict):
+        payload_track = {}
+    source = payload_track.get('source')
+    if source:
+        return source
+    if active_provider:
+        return active_provider
+    return 'spotify'
+
+
+def is_drifted_for_redo(
+    extra_data: Optional[Dict[str, Any]],
+    active_provider: Optional[str],
+) -> bool:
+    """Return True when a cached discovery entry should be treated as
+    stale because the user's active provider has changed since it was
+    cached AND the entry isn't a manual match.
+
+    Manual matches are *always* considered fresh: re-running discovery
+    against the current source would overwrite the user's deliberate
+    pick with whatever auto-search ranks first. The first Playlist
+    Pipeline run after a manual fix used to clobber it for exactly
+    this reason — the check lives here now so it's pinned by tests.
+    """
+    if not isinstance(extra_data, dict):
+        return False
+    if extra_data.get('manual_match'):
+        return False
+    cached_provider = extra_data.get('provider', 'spotify')
+    return cached_provider != active_provider
+
+
+def should_rediscover(extra_data: Optional[Dict[str, Any]]) -> bool:
+    """Return True when a mirrored track needs (re)discovery, False to skip it.
+
+    This is the gate the Playlist Pipeline pre-scan runs over every mirrored
+    track before discovering. The **ordering is the fix**: a manual match is
+    authoritative and is checked FIRST.
+
+    ``extra_data`` is *merged* on save (see ``update_mirrored_track_extra_data``),
+    so a track that was a Wing It stub and is then manually fixed still carries
+    ``wing_it_fallback: True`` alongside the new ``manual_match: True``. The old
+    pre-scan tested ``wing_it_fallback`` before ``manual_match``, so the stale
+    flag won and the pipeline re-discovered the track — silently reverting the
+    user's pick to Wing It. Checking ``manual_match`` first makes the fix stick.
+
+    Decision order:
+      * manual_match            -> skip   (authoritative; never re-discover)
+      * wing_it_fallback        -> redo   (stub — keep trying for a real match)
+      * discovered + complete   -> skip   (full metadata already stored)
+      * discovered + incomplete -> redo   (backfill track_number / album fields)
+      * unmatched_by_user       -> skip   (user deliberately removed the match)
+      * never discovered        -> redo   (first-time discovery)
+    """
+    extra = extra_data if isinstance(extra_data, dict) else {}
+
+    if extra.get('discovered'):
+        if extra.get('manual_match'):
+            return False
+        if extra.get('wing_it_fallback'):
+            return True
+        # Otherwise re-discover only when the stored match is missing the
+        # enriched fields (track_number + release_date/album.id) that older
+        # discoveries dropped via the Track dataclass.
+        matched = extra.get('matched_data')
+        matched = matched if isinstance(matched, dict) else {}
+        album = matched.get('album')
+        album = album if isinstance(album, dict) else {}
+        has_track_num = matched.get('track_number')
+        has_release = album.get('release_date')
+        has_album_id = album.get('id')
+        return not (has_track_num and (has_release or has_album_id))
+
+    if extra.get('unmatched_by_user'):
+        return False
+    return True

core/discovery/matching.py

@@ -0,0 +1,134 @@
+"""Pure helper for matching raw MusicBrainz-metadata tracks against
+Spotify / iTunes.
+
+Used by the PlaylistSource adapters whose ``get_playlist`` returns
+tracks with ``needs_discovery=True`` (ListenBrainz, Last.fm radio).
+Phase 1b ships Strategy 1 only (matching-engine queries → search →
+score → pick best ≥0.9). The richer multi-strategy +
+discovery-cache flow stays in
+``core.discovery.listenbrainz.run_listenbrainz_discovery_worker``
+for the Discover-page state-machine UI; this helper is the slimmer
+version used by the auto-refresh pipeline.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from typing import Any, Callable, Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class MBMatchDeps:
+    """Bundle of primitives the matcher needs.
+
+    Wired up at bootstrap. Tests pass stub callables / clients."""
+
+    matching_engine: Any
+    score_candidates: Callable[..., Any]
+    spotify_client_getter: Callable[[], Any]
+    itunes_client_getter: Callable[[], Any]
+    prefer_spotify_getter: Callable[[], bool]
+    min_confidence: float = 0.9
+
+
+def match_mb_track(
+    track: Dict[str, Any], deps: MBMatchDeps
+) -> Optional[Dict[str, Any]]:
+    """Try to match a single MB-metadata track.
+
+    Input shape:
+        ``{'track_name', 'artist_name', 'album_name', 'duration_ms'}``
+
+    Returns the matched_data dict (Spotify/iTunes track projection)
+    or ``None`` when no candidate cleared the confidence threshold.
+    """
+    title = track.get("track_name") or ""
+    artist = track.get("artist_name") or ""
+    album = track.get("album_name") or ""
+    duration_ms = int(track.get("duration_ms") or 0)
+    if not title or not artist:
+        return None
+
+    spotify_client = deps.spotify_client_getter()
+    itunes_client = deps.itunes_client_getter()
+    use_spotify = bool(
+        deps.prefer_spotify_getter()
+        and spotify_client is not None
+        and getattr(spotify_client, "is_spotify_authenticated", lambda: False)()
+    )
+    if not use_spotify and itunes_client is None:
+        return None
+
+    # Strategy 1 — matching-engine query generation.
+    try:
+        temp_track = type("_TempTrack", (), {
+            "name": title,
+            "artists": [artist],
+            "album": album or None,
+        })()
+        queries = deps.matching_engine.generate_download_queries(temp_track)
+    except Exception as exc:
+        logger.debug(f"matching_engine query-gen failed: {exc}")
+        queries = [f"{artist} {title}", title]
+
+    best_match: Any = None
+    best_confidence = 0.0
+    for query in queries:
+        try:
+            if use_spotify:
+                results = spotify_client.search_tracks(query, limit=10)
+            else:
+                results = itunes_client.search_tracks(query, limit=10)
+        except Exception as exc:
+            logger.debug(f"search failed for query={query!r}: {exc}")
+            continue
+        if not results:
+            continue
+        try:
+            match, confidence, _ = deps.score_candidates(
+                title, artist, duration_ms, results
+            )
+        except Exception as exc:
+            logger.debug(f"score_candidates failed: {exc}")
+            continue
+        if match and confidence > best_confidence and confidence >= deps.min_confidence:
+            best_match = match
+            best_confidence = confidence
+        if best_confidence >= deps.min_confidence:
+            break
+
+    if not best_match:
+        return None
+
+    provider = "spotify" if use_spotify else "itunes"
+    image_url = getattr(best_match, "image_url", None) or ""
+    album_data: Dict[str, Any] = {
+        "name": getattr(best_match, "album", "") or "",
+    }
+    if image_url:
+        album_data["images"] = [{"url": image_url}]
+    return {
+        "id": getattr(best_match, "id", "") or "",
+        "name": getattr(best_match, "name", "") or "",
+        "artists": list(getattr(best_match, "artists", []) or []),
+        "album": album_data,
+        "duration_ms": int(getattr(best_match, "duration_ms", 0) or 0),
+        "image_url": image_url,
+        "source": provider,
+        "_provider": provider,
+        "_confidence": float(best_confidence),
+    }
+
+
+def match_mb_tracks(
+    tracks: List[Dict[str, Any]], deps: MBMatchDeps
+) -> List[Optional[Dict[str, Any]]]:
+    """Vectorized variant — runs ``match_mb_track`` per track.
+
+    Phase 1b is sequential. If profiling shows it's too slow on big
+    LB playlists, this becomes the natural spot to thread-pool the
+    per-track searches."""
+    return [match_mb_track(t, deps) for t in tracks]

core/discovery/playlist.py

@@ -0,0 +1,423 @@
+"""Background worker for mirrored playlist track discovery.
+
+`run_playlist_discovery_worker(playlists, automation_id, deps)` is the
+function the automation engine schedules to enrich undiscovered mirrored
+playlist tracks with Spotify (preferred) or iTunes (fallback) metadata:
+
+1. Pause enrichment workers and pre-compute total track count for the
+   automation progress card.
+2. For each playlist:
+   - Fast pre-scan separates already-discovered tracks (skipped, unless
+     incomplete metadata or a Wing It stub) from undiscovered ones.
+   - For each undiscovered track:
+     - Cancellation gate.
+     - Discovery cache lookup (with artist validation).
+     - matching_engine search-query generation, then Spotify/iTunes
+       search + scoring across queries.
+     - Extended search fallback (limit=50) if no high-confidence match.
+     - On match → enrich album from metadata cache, build matched_data,
+       store in track.extra_data, save discovery cache entry.
+     - On miss → Wing It stub stored as 'wing_it_fallback' provider.
+3. After all playlists: emit `discovery_completed` event when at least
+   one new track was discovered, mark automation progress 'finished'.
+4. On error → automation progress 'error', traceback printed.
+5. Finally: resume enrichment workers.
+
+Lifted verbatim from web_server.py. Wide dependency surface (Spotify
+and iTunes clients, matching engine, discovery helpers, DB access,
+automation engine, cancellation set) all injected via
+`PlaylistDiscoveryDeps`.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any, Callable
+
+from core.discovery.manual_match import should_rediscover
+
+logger = logging.getLogger(__name__)
+
+
+def _canonical_best_score(deps, title, artist, duration_ms, results):
+    """Score search results against the source track, trying the canonicalized
+    title/artist too and keeping the better confidence (#785).
+
+    YouTube playlists have their "Artist - Title" / channel decoration stripped
+    at ingest, but file/CSV-imported playlists keep raw titles — so a track
+    titled "Arctic Monkeys - Do I Wanna Know?" scored verbatim against the
+    library's "Do I Wanna Know?" never matched. canonical_source_track is
+    conservative (only strips an "<artist> - " prefix when it equals the
+    artist), so this can only ADD a better candidate, never weaken a match.
+    Returns (match, confidence)."""
+    match, confidence, _ = deps.discovery_score_candidates(title, artist, duration_ms, results)
+    try:
+        from core.text.source_title import canonical_source_track
+        canon_title, canon_artist = canonical_source_track(title or '', artist or '')
+    except Exception:
+        return match, confidence
+    if (canon_title, canon_artist) != (title, artist):
+        alt_match, alt_conf, _ = deps.discovery_score_candidates(canon_title, canon_artist, duration_ms, results)
+        if alt_match and alt_conf > confidence:
+            return alt_match, alt_conf
+    return match, confidence
+
+
+@dataclass
+class PlaylistDiscoveryDeps:
+    """Bundle of cross-cutting deps the playlist discovery worker needs."""
+    spotify_client: Any
+    matching_engine: Any
+    automation_engine: Any
+    playlist_discovery_cancelled: set
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    get_metadata_fallback_source: Callable[[], str]
+    update_automation_progress: Callable
+    get_database: Callable[[], Any]
+    get_discovery_cache_key: Callable
+    validate_discovery_cache_artist: Callable
+    discovery_score_candidates: Callable
+    get_metadata_cache: Callable[[], Any]
+    build_discovery_wing_it_stub: Callable
+
+
+def run_playlist_discovery_worker(playlists, automation_id=None, deps: PlaylistDiscoveryDeps = None):
+    """Background worker that discovers Spotify/iTunes metadata for undiscovered
+    mirrored playlist tracks. Stores results in extra_data for use by sync."""
+    _ew_state = {}
+    try:
+        _ew_state = deps.pause_enrichment_workers('mirrored playlist discovery')
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        itunes_client_instance = None
+        if not use_spotify:
+            try:
+                itunes_client_instance = deps.get_metadata_fallback_client()
+            except Exception:
+                logger.warning(f"Neither Spotify nor {deps.get_metadata_fallback_source()} available for discovery")
+                deps.update_automation_progress(automation_id, status='error', progress=100,
+                                            phase='Error', log_line=f'Neither Spotify nor {deps.get_metadata_fallback_source()} available',
+                                            log_type='error')
+                return
+
+        total_discovered = 0
+        total_failed = 0
+        total_skipped = 0
+        total_tracks = 0
+        last_playlist_name = ''
+
+        # Pre-compute grand total for progress tracking
+        grand_total = 0
+        db_init = deps.get_database()
+        for pl in playlists:
+            t = db_init.get_mirrored_playlist_tracks(pl['id'])
+            if t:
+                grand_total += len(t)
+        deps.update_automation_progress(automation_id, total=grand_total)
+
+        for pl in playlists:
+            pl_id = pl['id']
+            pl_name = pl.get('name', '')
+            last_playlist_name = pl_name
+            source = pl.get('source', '')
+
+            db = deps.get_database()
+            tracks = db.get_mirrored_playlist_tracks(pl_id)
+            if not tracks:
+                continue
+
+            logger.info(f"Starting discovery for playlist '{pl_name}' ({len(tracks)} tracks, using {discovery_source.upper()})")
+            deps.update_automation_progress(automation_id, phase=f'Discovering: "{pl_name}"',
+                                         log_line=f'Playlist "{pl_name}" — {len(tracks)} tracks ({discovery_source.upper()})', log_type='info')
+
+            # Fast pre-scan: separate already-discovered from undiscovered
+            undiscovered_tracks = []
+            pl_skipped = 0
+            for track in tracks:
+                existing_extra = {}
+                if track.get('extra_data'):
+                    try:
+                        existing_extra = json.loads(track['extra_data']) if isinstance(track['extra_data'], str) else track['extra_data']
+                    except (json.JSONDecodeError, TypeError):
+                        pass
+                # `should_rediscover` is the single source of truth for this
+                # gate (manual match checked FIRST so a stale Wing It flag can't
+                # revert a user's deliberate fix — see its docstring).
+                if should_rediscover(existing_extra):
+                    undiscovered_tracks.append(track)
+                else:
+                    pl_skipped += 1
+                    total_skipped += 1
+
+            if pl_skipped > 0:
+                deps.update_automation_progress(automation_id,
+                    log_line=f'{pl_skipped} tracks already discovered — skipped', log_type='skip')
+
+            if not undiscovered_tracks:
+                deps.update_automation_progress(automation_id,
+                    progress=((total_skipped + total_discovered + total_failed) / max(1, grand_total)) * 100,
+                    log_line=f'All {len(tracks)} tracks already discovered', log_type='skip')
+                continue
+
+            deps.update_automation_progress(automation_id,
+                log_line=f'{len(undiscovered_tracks)} tracks to discover', log_type='info')
+
+            for i, track in enumerate(undiscovered_tracks):
+                # Check for cancellation
+                if automation_id and automation_id in deps.playlist_discovery_cancelled:
+                    deps.playlist_discovery_cancelled.discard(automation_id)
+                    logger.warning(f"Playlist discovery cancelled (automation {automation_id})")
+                    deps.update_automation_progress(automation_id, status='finished', progress=100,
+                                                 phase='Discovery cancelled',
+                                                 log_line=f'Cancelled: {total_discovered} discovered, {total_failed} failed',
+                                                 log_type='info')
+                    return
+
+                total_tracks += 1
+                track_id = track['id']
+                track_name = track.get('track_name', '')
+                artist_name = track.get('artist_name', '')
+                duration_ms = track.get('duration_ms', 0)
+
+                # Step 1: Check discovery cache
+                cache_key = deps.get_discovery_cache_key(track_name, artist_name)
+                try:
+                    cached_match = db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(artist_name, cached_match):
+                        extra_data = {
+                            'discovered': True,
+                            'provider': discovery_source,
+                            'confidence': cached_match.get('confidence', 0.85),
+                            'matched_data': cached_match,
+                        }
+                        db.update_mirrored_track_extra_data(track_id, extra_data)
+                        total_discovered += 1
+                        logger.info(f"CACHE [{i+1}/{len(undiscovered_tracks)}]: {track_name} → {cached_match.get('name', '?')}")
+                        deps.update_automation_progress(automation_id,
+                            progress=((total_skipped + total_discovered + total_failed) / max(1, grand_total)) * 100,
+                            current_item=track_name,
+                            log_line=f'{track_name} → {cached_match.get("name", "?")} (cache)', log_type='success')
+                        continue
+                except Exception as e:
+                    logger.debug("discovery cache lookup failed: %s", e)
+
+                # Step 2: Generate search queries
+                try:
+                    temp_track = type('TempTrack', (), {
+                        'name': track_name,
+                        'artists': [artist_name],
+                        'album': None
+                    })()
+                    search_queries = deps.matching_engine.generate_download_queries(temp_track)
+                except Exception:
+                    search_queries = [f"{artist_name} {track_name}", track_name]
+
+                # #785: file/CSV playlists keep raw "Artist - Title" titles, so the
+                # queries above search for the artist prefix too. Also search the
+                # canonicalized title so the right candidates are actually returned
+                # (the scorer best-of then matches them).
+                try:
+                    from core.text.source_title import canonical_source_track
+                    _cq_title, _cq_artist = canonical_source_track(track_name, artist_name)
+                    if (_cq_title, _cq_artist) != (track_name, artist_name):
+                        for _q in (f"{_cq_artist} {_cq_title}", _cq_title):
+                            if _q not in search_queries:
+                                search_queries.append(_q)
+                except Exception as _cq_err:
+                    logger.debug("canonical search-query add failed: %s", _cq_err)
+
+                # Step 3: Search and score
+                best_match = None
+                best_confidence = 0.0
+                min_confidence = 0.7
+
+                for search_query in search_queries:
+                    try:
+                        if use_spotify:
+                            results = deps.spotify_client.search_tracks(search_query, limit=10)
+                        else:
+                            results = itunes_client_instance.search_tracks(search_query, limit=10)
+                        if not results:
+                            continue
+
+                        match, confidence = _canonical_best_score(
+                            deps, track_name, artist_name, duration_ms, results
+                        )
+
+                        if match and confidence > best_confidence:
+                            best_confidence = confidence
+                            best_match = match
+
+                        if best_confidence >= 0.9:
+                            break
+                    except Exception:
+                        continue
+
+                # Extended search fallback
+                if not best_match or best_confidence < min_confidence:
+                    try:
+                        query = f"{artist_name} {track_name}"
+                        if use_spotify:
+                            extended = deps.spotify_client.search_tracks(query, limit=50)
+                        else:
+                            extended = itunes_client_instance.search_tracks(query, limit=50)
+                        if extended:
+                            match, confidence = _canonical_best_score(
+                                deps, track_name, artist_name, duration_ms, extended
+                            )
+                            if match and confidence > best_confidence:
+                                best_confidence = confidence
+                                best_match = match
+                    except Exception as e:
+                        logger.debug("extended discovery search failed: %s", e)
+
+                # Step 4: Store results
+                if best_match and best_confidence >= min_confidence:
+                    match_artists = best_match.artists if hasattr(best_match, 'artists') else []
+                    match_image = getattr(best_match, 'image_url', None)
+                    album_name = best_match.album if hasattr(best_match, 'album') else ''
+                    album_obj = {'name': album_name, 'release_date': getattr(best_match, 'release_date', '') or ''}
+                    if match_image:
+                        album_obj['images'] = [{'url': match_image, 'height': 600, 'width': 600}]
+
+                    # Enrich album data from metadata cache — search_tracks() caches the
+                    # raw API response which has full album info (id, images, total_tracks)
+                    # that the Track dataclass strips to just a name string
+                    track_number = None
+                    disc_number = None
+                    if hasattr(best_match, 'id') and best_match.id:
+                        try:
+                            cache = deps.get_metadata_cache()
+                            _raw = cache.get_entity(discovery_source if not use_spotify else 'spotify', 'track', best_match.id)
+                            if _raw and isinstance(_raw.get('album'), dict):
+                                _raw_album = _raw['album']
+                                if _raw_album.get('id'):
+                                    album_obj['id'] = _raw_album['id']
+                                if _raw_album.get('images') and not album_obj.get('images'):
+                                    album_obj['images'] = _raw_album['images']
+                                if _raw_album.get('total_tracks'):
+                                    album_obj['total_tracks'] = _raw_album['total_tracks']
+                                if _raw_album.get('album_type'):
+                                    album_obj['album_type'] = _raw_album['album_type']
+                                if _raw_album.get('release_date') and not album_obj.get('release_date'):
+                                    album_obj['release_date'] = _raw_album['release_date']
+                                if _raw_album.get('artists'):
+                                    album_obj['artists'] = _raw_album['artists']
+                            if _raw:
+                                track_number = _raw.get('track_number')
+                                disc_number = _raw.get('disc_number')
+                        except Exception as e:
+                            logger.debug("metadata cache lookup for album enrichment failed: %s", e)
+
+                    # Always include ``track_number`` / ``disc_number``
+                    # in the matched payload — None when unknown rather
+                    # than omitting the key. Downstream consumers
+                    # (``ensure_wishlist_track_format``, post-process
+                    # pipeline) check for None to know "look this up
+                    # somewhere else"; an absent key was indistinguishable
+                    # from "value is 1" after older payload helpers
+                    # silently filled the default. Pre-fix Deezer-sourced
+                    # matches always omitted the key (Deezer's track shape
+                    # uses ``track_position`` and the cache lookup at
+                    # line 304 reads ``track_number`` literally so it
+                    # returns None for Deezer rows).
+                    matched_data = {
+                        'id': best_match.id if hasattr(best_match, 'id') else '',
+                        'name': best_match.name if hasattr(best_match, 'name') else '',
+                        'artists': [{'name': a} if isinstance(a, str) else a for a in match_artists],
+                        'album': album_obj,
+                        'duration_ms': best_match.duration_ms if hasattr(best_match, 'duration_ms') else 0,
+                        'image_url': match_image,
+                        'source': discovery_source,
+                        'track_number': track_number if track_number else (
+                            getattr(best_match, 'track_number', None)
+                        ),
+                        'disc_number': disc_number if disc_number else (
+                            getattr(best_match, 'disc_number', None)
+                        ),
+                    }
+
+                    extra_data = {
+                        'discovered': True,
+                        'provider': discovery_source,
+                        'confidence': best_confidence,
+                        'matched_data': matched_data,
+                    }
+                    db.update_mirrored_track_extra_data(track_id, extra_data)
+                    total_discovered += 1
+
+                    # Save to discovery cache
+                    try:
+                        db.save_discovery_cache_match(
+                            cache_key[0], cache_key[1], discovery_source,
+                            best_confidence, matched_data,
+                            track_name, artist_name
+                        )
+                    except Exception as e:
+                        logger.debug("save discovery cache match failed: %s", e)
+
+                    logger.info(f"[{i+1}/{len(undiscovered_tracks)}] {track_name} → {matched_data['name']} ({best_confidence:.2f})")
+                    deps.update_automation_progress(automation_id,
+                        progress=((total_skipped + total_discovered + total_failed) / max(1, grand_total)) * 100,
+                        processed=total_discovered + total_failed,
+                        current_item=f'{track_name} - {artist_name}',
+                        log_line=f'{track_name} → {matched_data["name"]} ({best_confidence:.2f})', log_type='success')
+                else:
+                    # Auto Wing It fallback — mark as discovered with stub metadata
+                    stub = deps.build_discovery_wing_it_stub(track_name, artist_name, duration_ms)
+                    extra_data = {
+                        'discovered': True,
+                        'provider': 'wing_it_fallback',
+                        'confidence': 0,
+                        'wing_it_fallback': True,
+                        'matched_data': stub,
+                    }
+                    db.update_mirrored_track_extra_data(track_id, extra_data)
+                    total_discovered += 1
+                    logger.info(f"[{i+1}/{len(undiscovered_tracks)}] Wing It: {track_name} by {artist_name}")
+                    deps.update_automation_progress(automation_id,
+                        progress=((total_skipped + total_discovered + total_failed) / max(1, grand_total)) * 100,
+                        processed=total_discovered + total_failed,
+                        current_item=f'{track_name} - {artist_name}',
+                        log_line=f'{track_name} by {artist_name} → wing it (no API match)', log_type='info')
+
+                time.sleep(0.15)
+
+        # Emit completion event only if new tracks were actually discovered
+        # (no point triggering downstream sync if nothing changed)
+        try:
+            if deps.automation_engine and total_discovered > 0:
+                _disc_pl_id = str(playlists[0]['id']) if len(playlists) == 1 else ''
+                deps.automation_engine.emit('discovery_completed', {
+                    'playlist_name': last_playlist_name if len(playlists) == 1 else f'{len(playlists)} playlists',
+                    'playlist_id': _disc_pl_id,
+                    'total_tracks': str(total_tracks),
+                    'discovered_count': str(total_discovered),
+                    'failed_count': str(total_failed),
+                    'skipped_count': str(total_skipped),
+                })
+        except Exception as e:
+            logger.debug("discovery_completed emit failed: %s", e)
+
+        logger.error(f"Playlist discovery complete: {total_discovered} discovered, {total_failed} failed, {total_skipped} skipped")
+        deps.update_automation_progress(automation_id, status='finished', progress=100,
+                                     phase='Discovery complete',
+                                     log_line=f'Done: {total_discovered} discovered, {total_failed} failed, {total_skipped} skipped',
+                                     log_type='success')
+
+    except Exception as e:
+        logger.error(f"Error in playlist discovery worker: {e}")
+        import traceback
+        traceback.print_exc()
+        deps.update_automation_progress(automation_id, status='error', progress=100,
+                                     phase='Error',
+                                     log_line=f'Error: {str(e)}', log_type='error')
+    finally:
+        deps.resume_enrichment_workers(_ew_state, 'mirrored playlist discovery')

core/discovery/qobuz.py

@@ -0,0 +1,299 @@
+"""Background worker for Qobuz playlist discovery.
+
+`run_qobuz_discovery_worker(playlist_id, deps)` is the function the
+Qobuz discovery start-endpoint submits to its executor to match each
+Qobuz playlist track against Spotify (preferred) or the configured
+fallback metadata source (iTunes / Deezer / Discogs / MusicBrainz).
+
+Mirrors `core/discovery/deezer.py` exactly — Qobuz playlists arrive as
+dicts (not dataclasses) from `core/qobuz_client.py:get_playlist`, so
+this worker uses dict-style access on track data and wraps each entry
+in a SimpleNamespace before handing it to the shared
+`_search_spotify_for_tidal_track` helper.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+import types
+from dataclasses import dataclass
+from typing import Any, Callable
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class QobuzDiscoveryDeps:
+    """Bundle of cross-cutting deps the Qobuz discovery worker needs."""
+    qobuz_discovery_states: dict
+    spotify_client: Any
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    get_discovery_cache_key: Callable
+    get_database: Callable[[], Any]
+    validate_discovery_cache_artist: Callable
+    search_spotify_for_tidal_track: Callable
+    build_discovery_wing_it_stub: Callable
+    add_activity_item: Callable
+    sync_discovery_results_to_mirrored: Callable
+
+
+def run_qobuz_discovery_worker(playlist_id, deps: QobuzDiscoveryDeps):
+    """Background worker for Qobuz discovery process (Spotify preferred, fallback metadata source)."""
+    _ew_state = {}
+    try:
+        _ew_state = deps.pause_enrichment_workers('Qobuz discovery')
+        state = deps.qobuz_discovery_states[playlist_id]
+        playlist = state['playlist']
+
+        # Determine which provider to use
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        # Initialize fallback client if needed
+        itunes_client_instance = None
+        if not use_spotify:
+            itunes_client_instance = deps.get_metadata_fallback_client()
+
+        logger.info(f"Starting Qobuz discovery for: {playlist['name']} (using {discovery_source.upper()})")
+
+        # Store discovery source in state for frontend
+        state['discovery_source'] = discovery_source
+
+        successful_discoveries = 0
+        tracks = playlist['tracks']
+
+        for i, qobuz_track in enumerate(tracks):
+            if state.get('cancelled', False):
+                break
+
+            try:
+                track_name = qobuz_track['name']
+                track_artists = qobuz_track['artists']
+                track_id = qobuz_track['id']
+                track_album = qobuz_track.get('album', '')
+                track_duration_ms = qobuz_track.get('duration_ms', 0)
+
+                logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
+
+                # Check discovery cache first
+                cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
+                try:
+                    cache_db = deps.get_database()
+                    cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
+                        logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
+                        cached_artists = cached_match.get('artists', [])
+                        if cached_artists:
+                            cached_artist_str = ', '.join(
+                                a if isinstance(a, str) else a.get('name', '') for a in cached_artists
+                            )
+                        else:
+                            cached_artist_str = ''
+                        cached_album = cached_match.get('album', '')
+                        if isinstance(cached_album, dict):
+                            cached_album = cached_album.get('name', '')
+
+                        result = {
+                            'qobuz_track': {
+                                'id': track_id,
+                                'name': track_name,
+                                'artists': track_artists or [],
+                                'album': track_album,
+                                'duration_ms': track_duration_ms,
+                            },
+                            'spotify_data': cached_match,
+                            'match_data': cached_match,
+                            'status': 'Found',
+                            'status_class': 'found',
+                            'spotify_track': cached_match.get('name', ''),
+                            'spotify_artist': cached_artist_str,
+                            'spotify_album': cached_album,
+                            'spotify_id': cached_match.get('id', ''),
+                            'discovery_source': discovery_source,
+                            'index': i
+                        }
+                        successful_discoveries += 1
+                        state['spotify_matches'] = successful_discoveries
+                        state['discovery_results'].append(result)
+                        state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+                        continue
+                except Exception as cache_err:
+                    logger.error(f"Cache lookup error: {cache_err}")
+
+                # SimpleNamespace duck-type for _search_spotify_for_tidal_track
+                track_ns = types.SimpleNamespace(
+                    id=track_id,
+                    name=track_name,
+                    artists=track_artists,
+                    album=track_album,
+                    duration_ms=track_duration_ms
+                )
+
+                track_result = deps.search_spotify_for_tidal_track(
+                    track_ns,
+                    use_spotify=use_spotify,
+                    itunes_client=itunes_client_instance
+                )
+
+                result = {
+                    'qobuz_track': {
+                        'id': track_id,
+                        'name': track_name,
+                        'artists': track_artists or [],
+                        'album': track_album,
+                        'duration_ms': track_duration_ms,
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Not Found',
+                    'status_class': 'not-found',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'discovery_source': discovery_source
+                }
+
+                match_confidence = 0.0
+
+                if use_spotify and isinstance(track_result, tuple):
+                    track_obj, raw_track_data, match_confidence = track_result
+                    album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
+                    if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
+                        album_obj['name'] = track_obj.album
+                    elif not album_obj and track_obj.album:
+                        album_obj = {'name': track_obj.album}
+                    if isinstance(album_obj, dict) and not album_obj.get('release_date'):
+                        album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
+                    _album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
+                    _image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
+
+                    match_data = {
+                        'id': track_obj.id,
+                        'name': track_obj.name,
+                        'artists': track_obj.artists,
+                        'album': album_obj,
+                        'duration_ms': track_obj.duration_ms,
+                        'external_urls': track_obj.external_urls,
+                        'image_url': _image_url,
+                        'source': 'spotify'
+                    }
+                    if raw_track_data and raw_track_data.get('track_number'):
+                        match_data['track_number'] = raw_track_data['track_number']
+                    if raw_track_data and raw_track_data.get('disc_number'):
+                        match_data['disc_number'] = raw_track_data['disc_number']
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = track_obj.name
+                    result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
+                    result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
+                    result['spotify_id'] = track_obj.id
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                elif not use_spotify and track_result and isinstance(track_result, dict):
+                    match_confidence = track_result.pop('confidence', 0.80)
+                    match_data = track_result
+                    match_data['source'] = discovery_source
+                    _fb_album = match_data.get('album', {})
+                    _fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
+                    if _fb_images and 'image_url' not in match_data:
+                        match_data['image_url'] = _fb_images[0].get('url', '')
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = match_data.get('name', '')
+                    itunes_artists = match_data.get('artists', [])
+                    result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
+                    result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
+                    result['spotify_id'] = match_data.get('id', '')
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                # Save to discovery cache if match found
+                if result['status_class'] == 'found' and result.get('match_data'):
+                    try:
+                        cache_db = deps.get_database()
+                        cache_db.save_discovery_cache_match(
+                            cache_key[0], cache_key[1], discovery_source, match_confidence,
+                            result['match_data'], track_name,
+                            track_artists[0] if track_artists else ''
+                        )
+                        logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
+                    except Exception as cache_err:
+                        logger.error(f"Cache save error: {cache_err}")
+
+                # Auto Wing It fallback for unmatched tracks
+                if result['status_class'] == 'not-found':
+                    qobuz_t = result.get('qobuz_track', {})
+                    stub = deps.build_discovery_wing_it_stub(
+                        qobuz_t.get('name', ''),
+                        ', '.join(qobuz_t.get('artists', [])),
+                        qobuz_t.get('duration_ms', 0)
+                    )
+                    result['status'] = 'Wing It'
+                    result['status_class'] = 'wing-it'
+                    result['spotify_data'] = stub
+                    result['match_data'] = stub
+                    result['spotify_track'] = qobuz_t.get('name', '')
+                    result['spotify_artist'] = ', '.join(qobuz_t.get('artists', []))
+                    result['wing_it_fallback'] = True
+                    result['confidence'] = 0
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+                    state['wing_it_count'] = state.get('wing_it_count', 0) + 1
+
+                result['index'] = i
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+                time.sleep(0.1)
+
+            except Exception as e:
+                logger.error(f"Error processing track {i+1}: {e}")
+                result = {
+                    'qobuz_track': {
+                        'name': qobuz_track.get('name', 'Unknown'),
+                        'artists': qobuz_track.get('artists', []),
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Error',
+                    'status_class': 'error',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'error': str(e),
+                    'discovery_source': discovery_source,
+                    'index': i
+                }
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+        # Mark as complete
+        state['phase'] = 'discovered'
+        state['status'] = 'discovered'
+        state['discovery_progress'] = 100
+
+        source_label = discovery_source.upper()
+        deps.add_activity_item("", f"Qobuz Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
+
+        logger.info(f"Qobuz discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
+
+        deps.sync_discovery_results_to_mirrored('qobuz', playlist_id, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
+
+    except Exception as e:
+        logger.error(f"Error in Qobuz discovery worker: {e}")
+        if playlist_id in deps.qobuz_discovery_states:
+            deps.qobuz_discovery_states[playlist_id]['phase'] = 'error'
+            deps.qobuz_discovery_states[playlist_id]['status'] = f'error: {str(e)}'
+    finally:
+        deps.resume_enrichment_workers(_ew_state, 'Qobuz discovery')

core/discovery/quality_scanner.py

@@ -0,0 +1,279 @@
+"""Shared metadata match + result-normalization helpers for quality matching.
+
+These were the matching guts of the old auto-acting quality-scanner worker (now
+removed — quality scanning is the ``quality_upgrade`` library-maintenance repair
+job in ``core/repair_jobs/quality_upgrade.py``). They're kept here as a single
+source of truth and imported by that job:
+
+- ``_search_tracks_for_source`` — query one metadata source's ``search_tracks``.
+- ``_normalize_track_match`` / ``_normalize_track_album`` / ``_normalize_track_artists``
+  — turn a provider track into the wishlist-ready dict (typed Album converters
+  with legacy duck-typed fallback).
+- ``_track_name`` / ``_track_artist_names`` / ``_extract_lookup_value`` — accessors.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Callable, Dict, Optional
+
+from core.metadata.registry import get_client_for_source
+from core.metadata.types import Album
+from core.wishlist.payloads import ensure_wishlist_track_format
+
+# Use the project logger namespace ("soulsync.*") so the scanner's progress and
+# diagnostics actually surface in the app log — plain getLogger(__name__) lands
+# under "core.discovery.quality_scanner", which the app log view doesn't show.
+from utils.logging_config import get_logger
+logger = get_logger("discovery.quality_scanner")
+
+
+# Per-source typed converter dispatch — same registry pattern as
+# the metadata builders. Quality-scanner result normalization routes
+# the embedded ``track.album`` blob through Album.from_<source>_dict()
+# when provider is known. Falls back to legacy duck-typed extraction.
+_TYPED_ALBUM_CONVERTERS: Dict[str, Callable[[Dict[str, Any]], Album]] = {
+    'spotify': Album.from_spotify_dict,
+    'itunes': Album.from_itunes_dict,
+    'deezer': Album.from_deezer_dict,
+    'discogs': Album.from_discogs_dict,
+    'musicbrainz': Album.from_musicbrainz_dict,
+    'hydrabase': Album.from_hydrabase_dict,
+    'qobuz': Album.from_qobuz_dict,
+}
+
+
+
+
+def _extract_lookup_value(value: Any, *names: str, default: Any = None) -> Any:
+    if value is None:
+        return default
+    if isinstance(value, (str, bytes)):
+        return value
+
+    for name in names:
+        if isinstance(value, dict):
+            if name in value and value[name] is not None:
+                return value[name]
+        else:
+            candidate = getattr(value, name, None)
+            if candidate is not None:
+                return candidate
+    return default
+
+
+def _normalize_track_artists(track_item: Any) -> list[dict]:
+    artists = _extract_lookup_value(track_item, 'artists', default=[]) or []
+    if isinstance(artists, (str, bytes)):
+        artists = [artists]
+    elif isinstance(artists, dict):
+        artists = [artists]
+    else:
+        try:
+            artists = list(artists)
+        except TypeError:
+            artists = [artists]
+
+    normalized = []
+    for artist in artists:
+        artist_name = _extract_lookup_value(artist, 'name', 'artist_name', 'title')
+        if not artist_name and isinstance(artist, (str, bytes)):
+            artist_name = artist
+        if artist_name:
+            artist_data = {'name': str(artist_name)}
+            artist_images = _normalize_image_entries(_extract_lookup_value(artist, 'images', default=[]))
+            artist_image_url = _extract_lookup_value(artist, 'image_url', 'artist_image_url', default=None)
+            if artist_image_url and not artist_images:
+                artist_images = [{'url': str(artist_image_url)}]
+            if artist_images:
+                artist_data['images'] = artist_images
+                artist_data['image_url'] = artist_images[0].get('url')
+            normalized.append(artist_data)
+
+    if not normalized:
+        normalized.append({'name': 'Unknown Artist'})
+
+    return normalized
+
+
+def _normalize_image_entries(image_value: Any) -> list[dict]:
+    if not image_value:
+        return []
+
+    if isinstance(image_value, dict):
+        image_value = [image_value]
+    elif isinstance(image_value, (str, bytes)):
+        image_value = [image_value]
+    else:
+        try:
+            image_value = list(image_value)
+        except TypeError:
+            return []
+
+    normalized = []
+    seen_urls = set()
+    for image in image_value:
+        if isinstance(image, dict):
+            image_url = image.get('url') or image.get('image_url')
+            if not image_url:
+                continue
+            image_dict = dict(image)
+            image_dict['url'] = str(image_url)
+        elif isinstance(image, (str, bytes)):
+            image_dict = {'url': str(image)}
+        else:
+            continue
+
+        if image_dict['url'] in seen_urls:
+            continue
+
+        seen_urls.add(image_dict['url'])
+        normalized.append(image_dict)
+
+    return normalized
+
+
+def _normalize_track_album(track_item: Any, provider: Optional[str] = None) -> dict:
+    """Normalize a track's embedded album blob into a flat dict.
+
+    When ``provider`` is provided AND maps to a registered typed Album
+    converter, routes through the typed path to seed canonical fields
+    on ``album_data`` before legacy fallback chains fill any gaps.
+    Falls back to legacy duck-typed extraction on unknown provider /
+    non-dict input / typed converter error — same pattern as the
+    metadata builders.
+    """
+    album = _extract_lookup_value(track_item, 'album', default={})
+    if isinstance(album, dict):
+        album_data = dict(album)
+    else:
+        album_data = {
+            'name': _extract_lookup_value(album, 'name', 'title', default=str(album) if album else '') or '',
+            'album_type': _extract_lookup_value(album, 'album_type', default='album') or 'album',
+            'total_tracks': _extract_lookup_value(album, 'total_tracks', 'track_count', default=0) or 0,
+            'release_date': _extract_lookup_value(album, 'release_date', default='') or '',
+        }
+
+    if provider and isinstance(album, dict):
+        converter = _TYPED_ALBUM_CONVERTERS.get(provider.strip().lower())
+        if converter is not None:
+            try:
+                typed_album = converter(album)
+                if typed_album.name:
+                    album_data.setdefault('name', typed_album.name)
+                if typed_album.album_type:
+                    album_data.setdefault('album_type', typed_album.album_type)
+                if typed_album.total_tracks:
+                    album_data.setdefault('total_tracks', typed_album.total_tracks)
+                if typed_album.release_date:
+                    album_data.setdefault('release_date', typed_album.release_date)
+                if typed_album.id:
+                    album_data.setdefault('id', typed_album.id)
+            except Exception as exc:
+                logger.debug(
+                    "Typed album converter failed for provider %s in quality "
+                    "scanner normalize, falling back to legacy: %s", provider, exc,
+                )
+
+    album_data.setdefault('name', _extract_lookup_value(track_item, 'album_name', default='Unknown Album') or 'Unknown Album')
+    album_data.setdefault('album_type', _extract_lookup_value(track_item, 'album_type', default='album') or 'album')
+    album_data.setdefault('total_tracks', _extract_lookup_value(track_item, 'total_tracks', 'track_count', default=0) or 0)
+    album_data.setdefault('release_date', _extract_lookup_value(track_item, 'release_date', default='') or '')
+
+    album_images = _normalize_image_entries(album_data.get('images'))
+    if not album_images and isinstance(album, dict):
+        album_images = _normalize_image_entries(
+            album.get('images')
+            or album.get('image_url')
+            or album.get('album_cover_url')
+            or album.get('cover_url')
+        )
+
+    if not album_images:
+        album_images = _normalize_image_entries(
+            _extract_lookup_value(track_item, 'images', default=None)
+            or _extract_lookup_value(track_item, 'image_url', default=None)
+            or _extract_lookup_value(track_item, 'album_cover_url', default=None)
+            or _extract_lookup_value(track_item, 'cover_url', default=None)
+        )
+
+    if album_images:
+        album_data['images'] = album_images
+        album_data.setdefault('image_url', album_images[0].get('url'))
+    else:
+        album_data['images'] = []
+
+    album_data.setdefault('artists', _normalize_track_artists(track_item))
+    return album_data
+
+
+def _normalize_track_match(track_item: Any, provider: str) -> dict:
+    track_data = {
+        'id': _extract_lookup_value(track_item, 'id', 'track_id', default='') or '',
+        'name': _extract_lookup_value(track_item, 'name', 'title', default='Unknown Track') or 'Unknown Track',
+        'artists': _normalize_track_artists(track_item),
+        'album': _normalize_track_album(track_item, provider=provider),
+        'image_url': _extract_lookup_value(track_item, 'image_url', 'album_cover_url', default=None),
+        'duration_ms': _extract_lookup_value(track_item, 'duration_ms', default=0) or 0,
+        'track_number': _extract_lookup_value(track_item, 'track_number', default=1) or 1,
+        'disc_number': _extract_lookup_value(track_item, 'disc_number', default=1) or 1,
+        'preview_url': _extract_lookup_value(track_item, 'preview_url', default=None),
+        'external_urls': _extract_lookup_value(track_item, 'external_urls', default={}) or {},
+        'popularity': _extract_lookup_value(track_item, 'popularity', default=0) or 0,
+        'provider': provider,
+        'source': provider,
+    }
+    if not track_data['image_url']:
+        album_images = track_data['album'].get('images') if isinstance(track_data['album'], dict) else []
+        if isinstance(album_images, list) and album_images:
+            first_image = album_images[0]
+            if isinstance(first_image, dict):
+                track_data['image_url'] = first_image.get('url')
+    return ensure_wishlist_track_format(track_data)
+
+
+def _track_name(track_item: Any) -> str:
+    return str(_extract_lookup_value(track_item, 'name', 'title', default='Unknown Track') or 'Unknown Track')
+
+
+def _track_artist_names(track_item: Any) -> list[str]:
+    artists = _extract_lookup_value(track_item, 'artists', default=[]) or []
+    if isinstance(artists, (str, bytes)):
+        artists = [artists]
+    elif isinstance(artists, dict):
+        artists = [artists]
+    else:
+        try:
+            artists = list(artists)
+        except TypeError:
+            artists = [artists]
+
+    normalized = []
+    for artist in artists:
+        artist_name = _extract_lookup_value(artist, 'name', 'artist_name', 'title')
+        if not artist_name and isinstance(artist, (str, bytes)):
+            artist_name = artist
+        if artist_name:
+            normalized.append(str(artist_name))
+    return normalized
+
+
+def _search_tracks_for_source(source: str, query: str, limit: int = 5, client: Any = None):
+    if client is None:
+        client = get_client_for_source(source)
+    if not client or not hasattr(client, 'search_tracks'):
+        return []
+
+    try:
+        if source == 'spotify':
+            return client.search_tracks(query, limit=limit, allow_fallback=False) or []
+        return client.search_tracks(query, limit=limit) or []
+    except TypeError:
+        try:
+            return client.search_tracks(query, limit=limit) or []
+        except Exception as exc:
+            logger.debug("Could not search %s for %s: %s", source, query, exc)
+            return []
+    except Exception as exc:
+        logger.debug("Could not search %s for %s: %s", source, query, exc)
+        return []

core/discovery/scoring.py

@@ -0,0 +1,323 @@
+"""Discovery scoring + tidal-track search — lifted from web_server.py.
+
+Both function bodies are byte-identical to the originals. The
+``spotify_client`` proxy and ``_get_metadata_fallback_source`` shim
+let the bodies resolve their original names without modification.
+``matching_engine`` is injected via init() because it is constructed
+in web_server.py and referenced by name throughout the bodies.
+"""
+import logging
+
+from core.metadata.cache import get_metadata_cache
+from core.metadata.registry import get_primary_source, get_spotify_client
+from core.spotify_client import _is_globally_rate_limited as _spotify_rate_limited
+
+logger = logging.getLogger(__name__)
+
+
+def _get_metadata_fallback_source():
+    """Mirror of web_server._get_metadata_fallback_source — delegates to registry."""
+    return get_primary_source()
+
+
+class _SpotifyClientProxy:
+    """Resolves the global Spotify client lazily through core.metadata.registry."""
+
+    def __getattr__(self, name):
+        client = get_spotify_client()
+        if client is None:
+            raise AttributeError(name)
+        return getattr(client, name)
+
+    def __bool__(self):
+        return get_spotify_client() is not None
+
+
+spotify_client = _SpotifyClientProxy()
+
+
+# Injected at runtime via init().
+matching_engine = None
+
+
+def init(matching_engine_obj):
+    """Bind the shared matching engine instance from web_server."""
+    global matching_engine
+    matching_engine = matching_engine_obj
+
+
+def _discovery_score_candidates(source_title, source_artist, source_duration_ms, search_results):
+    """Score search results against a source track using the matching engine.
+
+    Both artist AND title must independently pass minimum similarity floors.
+    This prevents weighted scoring from allowing a perfect artist to carry a
+    garbage title (or vice versa). If either dimension doesn't match, the
+    candidate is rejected — no match is better than a wrong match.
+
+    Args:
+        source_title: The source track title (already cleaned for YouTube, raw for others)
+        source_artist: The source track primary artist
+        source_duration_ms: The source track duration in ms (0 if unknown)
+        search_results: List of Track objects (Spotify or iTunes) from search
+
+    Returns:
+        (best_match, best_confidence, best_index) or (None, 0.0, -1) if no results
+    """
+    best_match = None
+    best_confidence = 0.0
+    best_index = -1
+    min_artist_similarity = 0.5
+    min_title_similarity = 0.5
+
+    source_artist_cleaned = matching_engine.clean_artist(source_artist)
+    source_title_cleaned = matching_engine.clean_title(source_title)
+    source_core_title = matching_engine.get_core_string(source_title)
+
+    for idx, result in enumerate(search_results):
+        try:
+            result_artists = result.artists if hasattr(result, 'artists') and result.artists else []
+            result_name = result.name if hasattr(result, 'name') else ''
+            result_duration = result.duration_ms if hasattr(result, 'duration_ms') else 0
+
+            # Artist floor — both must match, not just the weighted score
+            best_artist_sim = 0.0
+            for cand_artist in result_artists:
+                if not cand_artist:
+                    continue
+                cand_cleaned = matching_engine.clean_artist(cand_artist)
+                cand_normalized = matching_engine.normalize_string(cand_artist)
+                if source_artist_cleaned and source_artist_cleaned in cand_normalized:
+                    best_artist_sim = 1.0
+                    break
+                sim = matching_engine.similarity_score(source_artist_cleaned, cand_cleaned)
+                if sim > best_artist_sim:
+                    best_artist_sim = sim
+
+            if best_artist_sim < min_artist_similarity:
+                continue
+
+            # Title floor — both must match, not just the weighted score
+            cand_title_cleaned = matching_engine.clean_title(result_name)
+            cand_core_title = matching_engine.get_core_string(result_name)
+
+            # Core title exact match bypasses the floor (e.g., "edamame" == "edamame")
+            title_passes = False
+            if source_core_title and cand_core_title and source_core_title == cand_core_title:
+                title_passes = True
+            else:
+                title_sim = matching_engine.similarity_score(source_title_cleaned, cand_title_cleaned)
+                if title_sim >= min_title_similarity:
+                    title_passes = True
+
+            if not title_passes:
+                continue
+
+            # Both floors passed — now do full scoring
+            confidence, match_type = matching_engine.score_track_match(
+                source_title=source_title,
+                source_artists=[source_artist],
+                source_duration_ms=source_duration_ms,
+                candidate_title=result_name,
+                candidate_artists=result_artists,
+                candidate_duration_ms=result_duration
+            )
+
+            if confidence > best_confidence:
+                best_confidence = confidence
+                best_match = result
+                best_index = idx
+
+        except Exception as e:
+            logger.error(f"Error scoring candidate {idx}: {e}")
+            continue
+
+    return best_match, best_confidence, best_index
+
+
+def _search_spotify_for_tidal_track(tidal_track, use_spotify=True, itunes_client=None):
+    """Search Spotify/fallback for a Tidal track using matching_engine for better accuracy
+
+    Args:
+        tidal_track: The Tidal track to search for
+        use_spotify: If True, use Spotify; if False, use fallback source
+        itunes_client: Fallback client instance (required when use_spotify=False)
+
+    Returns:
+        For Spotify: (Track, raw_data, confidence) tuple or None
+        For fallback: dict with track data (includes 'confidence' key) or None
+    """
+    if use_spotify:
+        if not spotify_client or not spotify_client.is_authenticated():
+            return None
+    else:
+        if not itunes_client:
+            return None
+
+    try:
+        # Get track info
+        track_name = tidal_track.name
+        artists = tidal_track.artists or []
+
+        if not artists:
+            return None
+
+        artist_name = artists[0]  # Use primary artist
+        source_duration = getattr(tidal_track, 'duration_ms', 0) or 0
+        source_name = "Spotify" if use_spotify else _get_metadata_fallback_source().capitalize()
+
+        logger.info(f"Tidal track: '{artist_name}' - '{track_name}' (searching {source_name})")
+
+        # Use matching engine to generate search queries (with fallback)
+        try:
+            temp_track = type('TempTrack', (), {
+                'name': track_name,
+                'artists': [artist_name],
+                'album': None
+            })()
+            search_queries = matching_engine.generate_download_queries(temp_track)
+            logger.info(f"Generated {len(search_queries)} search queries for Tidal track")
+        except Exception as e:
+            logger.error(f"Matching engine failed for Tidal, falling back to basic queries: {e}")
+            if use_spotify:
+                search_queries = [
+                    f'track:"{track_name}" artist:"{artist_name}"',
+                    f'"{track_name}" "{artist_name}"',
+                    f'{track_name} {artist_name}'
+                ]
+            else:
+                search_queries = [
+                    f'{artist_name} {track_name}',
+                    f'{track_name} {artist_name}',
+                    track_name
+                ]
+
+        best_match = None
+        best_match_raw = None
+        best_confidence = 0.0
+        min_confidence = 0.9
+
+        for query_idx, search_query in enumerate(search_queries):
+            try:
+                logger.debug(f"Tidal query {query_idx + 1}/{len(search_queries)}: {search_query} ({source_name})")
+
+                if use_spotify and not _spotify_rate_limited():
+                    results = spotify_client.search_tracks(search_query, limit=10)
+                    if not results:
+                        continue
+                else:
+                    results = itunes_client.search_tracks(search_query, limit=10)
+                    if not results:
+                        continue
+
+                # Score all results using the matching engine
+                match, confidence, match_idx = _discovery_score_candidates(
+                    track_name, artist_name, source_duration, results
+                )
+
+                if match and confidence > best_confidence and confidence >= min_confidence:
+                    best_confidence = confidence
+                    best_match = match
+                    if use_spotify and match.id:
+                        _cache = get_metadata_cache()
+                        best_match_raw = _cache.get_entity('spotify', 'track', match.id)
+                    else:
+                        best_match_raw = None
+                    logger.info(f"New best Tidal match: {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+                if best_confidence >= 0.9:
+                    logger.info(f"High confidence Tidal match found ({best_confidence:.3f}), stopping search")
+                    break
+
+            except Exception as e:
+                logger.debug(f"Error in Tidal {source_name} search for query '{search_query}': {e}")
+                continue
+
+        # Strategy 4: Extended search with higher limit (last resort)
+        if not best_match:
+            logger.info("Tidal Strategy 4: Extended search with limit=50")
+            query = f"{artist_name} {track_name}"
+            if use_spotify:
+                extended_results = spotify_client.search_tracks(query, limit=50)
+            else:
+                extended_results = itunes_client.search_tracks(query, limit=50)
+            if extended_results:
+                match, confidence, match_idx = _discovery_score_candidates(
+                    track_name, artist_name, source_duration, extended_results
+                )
+                if match and confidence >= min_confidence:
+                    best_match = match
+                    best_confidence = confidence
+                    logger.info(f"Strategy 4 Tidal match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
+
+        if best_match:
+            if use_spotify:
+                logger.info(f"Final Tidal Spotify match: {best_match.artists[0]} - {best_match.name} (confidence: {best_confidence:.3f})")
+                return (best_match, best_match_raw, best_confidence)
+            else:
+                result_artists = best_match.artists if hasattr(best_match, 'artists') else []
+                result_artist = result_artists[0] if result_artists else 'Unknown'
+                result_name = best_match.name if hasattr(best_match, 'name') else 'Unknown'
+                logger.info(f"Final Tidal {source_name} match: {result_artist} - {result_name} (confidence: {best_confidence:.3f})")
+
+                album_name = best_match.album if hasattr(best_match, 'album') else 'Unknown Album'
+                image_url = best_match.image_url if hasattr(best_match, 'image_url') else ''
+                track_id = best_match.id if hasattr(best_match, 'id') else ''
+                duration_ms = best_match.duration_ms if hasattr(best_match, 'duration_ms') else 0
+
+                # Fetch full track details to get album ID, track_number, etc.
+                # The Track dataclass strips this data — the API has it
+                album_obj = {
+                    'name': album_name,
+                    'album_type': 'album',
+                    'release_date': getattr(best_match, 'release_date', '') or '',
+                    'images': [{'url': image_url, 'height': 300, 'width': 300}] if image_url else []
+                }
+                track_number = None
+                disc_number = None
+                if track_id:
+                    try:
+                        detailed = itunes_client.get_track_details(track_id)
+                        if detailed and isinstance(detailed.get('album'), dict):
+                            dt_album = detailed['album']
+                            if dt_album.get('id'):
+                                album_obj['id'] = dt_album['id']
+                            if dt_album.get('total_tracks'):
+                                album_obj['total_tracks'] = dt_album['total_tracks']
+                            if dt_album.get('release_date') and not album_obj.get('release_date'):
+                                album_obj['release_date'] = dt_album['release_date']
+                            if dt_album.get('album_type'):
+                                album_obj['album_type'] = dt_album['album_type']
+                            if dt_album.get('images') and not album_obj.get('images'):
+                                album_obj['images'] = dt_album['images']
+                            if dt_album.get('artists'):
+                                album_obj['artists'] = dt_album['artists']
+                        if detailed:
+                            track_number = detailed.get('track_number')
+                            disc_number = detailed.get('disc_number')
+                            logger.info(f"[Discovery Enrich] {result_name}: track_number={track_number}, disc={disc_number}")
+                        else:
+                            logger.info(f"[Discovery Enrich] get_track_details returned None for ID {track_id} ({result_name})")
+                    except Exception as _enrich_err:
+                        logger.error(f"[Discovery Enrich] Failed for {result_name} (ID {track_id}): {_enrich_err}")
+
+                result_data = {
+                    'id': track_id,
+                    'name': result_name,
+                    'artists': [result_artist],
+                    'album': album_obj,
+                    'duration_ms': duration_ms,
+                    'source': _get_metadata_fallback_source(),
+                    'confidence': best_confidence
+                }
+                if track_number:
+                    result_data['track_number'] = track_number
+                if disc_number:
+                    result_data['disc_number'] = disc_number
+                return result_data
+        else:
+            logger.warning(f"No suitable Tidal match found (best confidence was {best_confidence:.3f}, required {min_confidence:.3f})")
+            return None
+
+    except Exception as e:
+        logger.error(f"Error searching Spotify for Tidal track: {e}")
+        return None

core/discovery/spotify_public.py

@@ -0,0 +1,341 @@
+"""Background worker for public Spotify-link playlist discovery.
+
+`run_spotify_public_discovery_worker(url_hash, deps)` is the function the
+spotify-public discovery start-endpoint submits to its executor to match
+each public Spotify playlist track against Spotify (preferred) or iTunes
+(fallback):
+
+1. Pause enrichment workers (release shared resources).
+2. For each track:
+   - Cancellation gate (state['cancelled']).
+   - Discovery cache lookup; cache hit short-circuits the search and
+     populates display fields from the cached match.
+   - SimpleNamespace duck-type → `_search_spotify_for_tidal_track`
+     (shared search helper, returns tuple for Spotify or dict for iTunes).
+   - On Spotify match: build `match_data` preserving track_number /
+     disc_number from raw API data, image extracted from album images
+     or track object fallback, release_date filled from track.release_date
+     when album dict is missing it.
+   - On iTunes match: dict result populated as `match_data`, source set
+     to discovery_source, image extracted from album images.
+   - Save matched result to discovery cache.
+   - On miss: Wing It stub stored as 'wing-it' status.
+3. After all tracks: phase='discovered', activity feed entry.
+4. On error: state['phase']='error' + status with error string.
+5. Finally: resume enrichment workers.
+
+Lifted verbatim from web_server.py. Wide dependency surface (Spotify and
+iTunes clients, multiple metadata helpers, state dict, shared tidal
+search helper) all injected via `SpotifyPublicDiscoveryDeps`.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+import types
+from dataclasses import dataclass
+from typing import Any, Callable
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class SpotifyPublicDiscoveryDeps:
+    """Bundle of cross-cutting deps the Spotify Public discovery worker needs."""
+    spotify_public_discovery_states: dict
+    spotify_client: Any
+    pause_enrichment_workers: Callable[[str], dict]
+    resume_enrichment_workers: Callable[[dict, str], None]
+    get_active_discovery_source: Callable[[], str]
+    get_metadata_fallback_client: Callable[[], Any]
+    get_discovery_cache_key: Callable
+    get_database: Callable[[], Any]
+    validate_discovery_cache_artist: Callable
+    search_spotify_for_tidal_track: Callable
+    build_discovery_wing_it_stub: Callable
+    add_activity_item: Callable
+    source_label: str = "Spotify Public"
+    activity_label: str = "Spotify Link"
+    original_track_key: str = "spotify_public_track"
+
+
+def run_spotify_public_discovery_worker(url_hash, deps: SpotifyPublicDiscoveryDeps):
+    """Background worker for Spotify Public discovery process (Spotify preferred, iTunes fallback)"""
+    _ew_state = {}
+    try:
+        worker_label = f"{deps.source_label} discovery"
+        _ew_state = deps.pause_enrichment_workers(worker_label)
+        state = deps.spotify_public_discovery_states[url_hash]
+        playlist = state['playlist']
+
+        # Determine which provider to use — respect user's configured primary source
+        discovery_source = deps.get_active_discovery_source()
+        use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
+
+        # Initialize fallback client if needed
+        itunes_client_instance = None
+        if not use_spotify:
+            itunes_client_instance = deps.get_metadata_fallback_client()
+
+        logger.info(f"Starting {deps.source_label} discovery for: {playlist['name']} (using {discovery_source.upper()})")
+
+        # Store discovery source in state for frontend
+        state['discovery_source'] = discovery_source
+
+        successful_discoveries = 0
+        tracks = playlist['tracks']
+
+        for i, sp_track in enumerate(tracks):
+            if state.get('cancelled', False):
+                break
+
+            try:
+                track_name = sp_track['name']
+                track_artists_raw = sp_track.get('artists', [])
+                # Normalize artists to list of strings
+                track_artists = []
+                for a in track_artists_raw:
+                    if isinstance(a, dict):
+                        track_artists.append(a.get('name', ''))
+                    else:
+                        track_artists.append(str(a))
+                track_id = sp_track.get('id', '')
+                track_album = sp_track.get('album', '')
+                if isinstance(track_album, dict):
+                    track_album_name = track_album.get('name', '')
+                else:
+                    track_album_name = track_album or ''
+                track_duration_ms = sp_track.get('duration_ms', 0)
+
+                logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
+
+                # Check discovery cache first
+                cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
+                try:
+                    cache_db = deps.get_database()
+                    cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
+                    if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
+                        logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
+                        # Extract display-friendly artist string from cached match
+                        cached_artists = cached_match.get('artists', [])
+                        if cached_artists:
+                            cached_artist_str = ', '.join(
+                                a if isinstance(a, str) else a.get('name', '') for a in cached_artists
+                            )
+                        else:
+                            cached_artist_str = ''
+                        cached_album = cached_match.get('album', '')
+                        if isinstance(cached_album, dict):
+                            cached_album = cached_album.get('name', '')
+
+                        result = {
+                            deps.original_track_key: {
+                                'id': track_id,
+                                'name': track_name,
+                                'artists': track_artists or [],
+                                'album': track_album_name,
+                                'duration_ms': track_duration_ms,
+                            },
+                            'spotify_data': cached_match,
+                            'match_data': cached_match,
+                            'status': 'Found',
+                            'status_class': 'found',
+                            'spotify_track': cached_match.get('name', ''),
+                            'spotify_artist': cached_artist_str,
+                            'spotify_album': cached_album,
+                            'spotify_id': cached_match.get('id', ''),
+                            'discovery_source': discovery_source,
+                            'index': i
+                        }
+                        successful_discoveries += 1
+                        state['spotify_matches'] = successful_discoveries
+                        state['discovery_results'].append(result)
+                        state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+                        continue
+                except Exception as cache_err:
+                    logger.error(f"Cache lookup error: {cache_err}")
+
+                # Create a SimpleNamespace duck-type object for _search_spotify_for_tidal_track
+                track_ns = types.SimpleNamespace(
+                    id=track_id,
+                    name=track_name,
+                    artists=track_artists,
+                    album=track_album_name,
+                    duration_ms=track_duration_ms
+                )
+
+                # Use the search function with appropriate provider
+                track_result = deps.search_spotify_for_tidal_track(
+                    track_ns,
+                    use_spotify=use_spotify,
+                    itunes_client=itunes_client_instance
+                )
+
+                # Create result entry
+                result = {
+                    deps.original_track_key: {
+                        'id': track_id,
+                        'name': track_name,
+                        'artists': track_artists or [],
+                        'album': track_album_name,
+                        'duration_ms': track_duration_ms,
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Not Found',
+                    'status_class': 'not-found',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'discovery_source': discovery_source
+                }
+
+                match_confidence = 0.0
+
+                if use_spotify and isinstance(track_result, tuple):
+                    # Spotify: Function returns (Track, raw_data, confidence)
+                    track_obj, raw_track_data, match_confidence = track_result
+                    album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
+                    # Ensure album has a name — fall back to track_obj.album if raw_data was missing
+                    if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
+                        album_obj['name'] = track_obj.album
+                    elif not album_obj and track_obj.album:
+                        album_obj = {'name': track_obj.album}
+                    # Ensure release_date is present (raw Spotify data has it, but fallback may not)
+                    if isinstance(album_obj, dict) and not album_obj.get('release_date'):
+                        album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
+                    # Extract image URL from album data or track object
+                    _album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
+                    _image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
+
+                    match_data = {
+                        'id': track_obj.id,
+                        'name': track_obj.name,
+                        'artists': track_obj.artists,
+                        'album': album_obj,
+                        'duration_ms': track_obj.duration_ms,
+                        'external_urls': track_obj.external_urls,
+                        'image_url': _image_url,
+                        'source': 'spotify'
+                    }
+                    # Preserve track_number/disc_number from raw Spotify API data
+                    if raw_track_data and raw_track_data.get('track_number'):
+                        match_data['track_number'] = raw_track_data['track_number']
+                    if raw_track_data and raw_track_data.get('disc_number'):
+                        match_data['disc_number'] = raw_track_data['disc_number']
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = track_obj.name
+                    result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
+                    result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
+                    result['spotify_id'] = track_obj.id
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                elif not use_spotify and track_result and isinstance(track_result, dict):
+                    # Fallback: Function returns a dict with track data (includes 'confidence' key)
+                    match_confidence = track_result.pop('confidence', 0.80)
+                    match_data = track_result
+                    match_data['source'] = discovery_source
+                    # Extract image URL from album images
+                    _fb_album = match_data.get('album', {})
+                    _fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
+                    if _fb_images and 'image_url' not in match_data:
+                        match_data['image_url'] = _fb_images[0].get('url', '')
+                    result['spotify_data'] = match_data
+                    result['match_data'] = match_data
+                    result['status'] = 'Found'
+                    result['status_class'] = 'found'
+                    result['spotify_track'] = match_data.get('name', '')
+                    itunes_artists = match_data.get('artists', [])
+                    result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
+                    result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
+                    result['spotify_id'] = match_data.get('id', '')
+                    result['confidence'] = match_confidence
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+
+                # Save to discovery cache if match found
+                if result['status_class'] == 'found' and result.get('match_data'):
+                    try:
+                        cache_db = deps.get_database()
+                        cache_db.save_discovery_cache_match(
+                            cache_key[0], cache_key[1], discovery_source, match_confidence,
+                            result['match_data'], track_name,
+                            track_artists[0] if track_artists else ''
+                        )
+                        logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
+                    except Exception as cache_err:
+                        logger.error(f"Cache save error: {cache_err}")
+
+                # Auto Wing It fallback for unmatched tracks
+                if result['status_class'] == 'not-found':
+                    sp_t = result.get(deps.original_track_key, {})
+                    stub = deps.build_discovery_wing_it_stub(
+                        sp_t.get('name', ''),
+                        ', '.join(sp_t.get('artists', [])),
+                        sp_t.get('duration_ms', 0)
+                    )
+                    result['status'] = 'Wing It'
+                    result['status_class'] = 'wing-it'
+                    result['spotify_data'] = stub
+                    result['match_data'] = stub
+                    result['spotify_track'] = sp_t.get('name', '')
+                    result['spotify_artist'] = ', '.join(sp_t.get('artists', []))
+                    result['wing_it_fallback'] = True
+                    result['confidence'] = 0
+                    successful_discoveries += 1
+                    state['spotify_matches'] = successful_discoveries
+                    state['wing_it_count'] = state.get('wing_it_count', 0) + 1
+
+                result['index'] = i
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+                # Add delay between requests
+                time.sleep(0.1)
+
+            except Exception as e:
+                logger.error(f"Error processing track {i+1}: {e}")
+                # Add error result
+                result = {
+                    deps.original_track_key: {
+                        'name': sp_track.get('name', 'Unknown'),
+                        'artists': sp_track.get('artists', []),
+                    },
+                    'spotify_data': None,
+                    'match_data': None,
+                    'status': 'Error',
+                    'status_class': 'error',
+                    'spotify_track': '',
+                    'spotify_artist': '',
+                    'spotify_album': '',
+                    'error': str(e),
+                    'discovery_source': discovery_source,
+                    'index': i
+                }
+                state['discovery_results'].append(result)
+                state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
+
+        # Mark as complete
+        state['phase'] = 'discovered'
+        state['status'] = 'discovered'
+        state['discovery_progress'] = 100
+
+        # Add activity for discovery completion
+        source_label = discovery_source.upper()
+        deps.add_activity_item("", f"{deps.activity_label} Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
+
+        logger.info(f"{deps.source_label} discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
+
+    except Exception as e:
+        logger.error(f"Error in {deps.source_label} discovery worker: {e}")
+        if url_hash in deps.spotify_public_discovery_states:
+            deps.spotify_public_discovery_states[url_hash]['phase'] = 'error'
+            deps.spotify_public_discovery_states[url_hash]['status'] = f'error: {str(e)}'
+    finally:
+        deps.resume_enrichment_workers(_ew_state, f"{deps.source_label} discovery")

core/discovery/sync.py

@@ -0,0 +1,663 @@
+"""Background worker for the playlist sync task.
+
+`run_sync_task(playlist_id, playlist_name, tracks_json, automation_id, profile_id,
+playlist_image_url, deps)` is the function `sync_executor.submit(...)` invokes
+to drive the entire playlist-sync workflow:
+
+1. Convert frontend JSON tracks → SpotifyTrack/SpotifyPlaylist objects.
+2. Normalize artist/album shapes for downstream wishlist parity.
+3. Wire a progress_callback that updates `sync_states` + automation card.
+4. Patch sync_service for database-only fallback when no media server is connected.
+5. `run_async(sync_service.sync_playlist(...))` and capture the result.
+6. Update sync_states to 'finished', push playlist poster image to Plex/Jellyfin/Emby,
+   record sync history (with re-sync vs new-sync branching), emit
+   `playlist_synced` event for automation engine, and persist sync status with a
+   tracks_hash for smart-skip on the next scheduled sync.
+7. On exception → mark error in sync_states + automation; finally clear progress
+   callback + drop `_original_tracks_map` from sync_service.
+
+Lifted verbatim from web_server.py. Wide dependency surface (sync_service,
+sync_states, plex/jellyfin clients, automation engine, multiple helper funcs)
+all injected via `SyncDeps`.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any, Callable
+
+from core.spotify_client import Playlist as SpotifyPlaylist, Track as SpotifyTrack
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class SyncDeps:
+    """Bundle of cross-cutting deps the sync worker needs."""
+    config_manager: Any
+    sync_service: Any
+    media_server_engine: Any
+    automation_engine: Any
+    run_async: Callable[..., Any]
+    record_sync_history_start: Callable
+    update_automation_progress: Callable
+    update_and_save_sync_status: Callable
+    sync_states: dict
+    sync_lock: Any  # threading.Lock
+    # Optional: post-sync download follow-up for mirrored-playlist automations.
+    process_wishlist_automatically: Callable[..., Any] | None = None
+    run_playlist_organize_download: Callable[..., Any] | None = None
+    is_wishlist_actually_processing: Callable[[], bool] | None = None
+
+
+def _post_sync_automation_followup(
+    deps: SyncDeps,
+    *,
+    automation_id: str,
+    playlist_id: str,
+    skip_wishlist_add: bool,
+    result: Any,
+) -> None:
+    """Queue downloads after an automation sync finishes.
+
+    Sync Playlist runs in a background thread and returns immediately, so a
+    separate scheduled "Process Wishlist" action often runs on an empty wishlist.
+    Organize-by-playlist skips sync-time wishlist adds and expects a folder
+    download batch instead — that only ran in Playlist Pipeline before this hook.
+    """
+    if not automation_id or not str(playlist_id).startswith('auto_mirror_'):
+        return
+    try:
+        mirrored_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
+    except ValueError:
+        return
+
+    failed = int(getattr(result, 'failed_tracks', 0) or 0)
+    wishlist_added = int(getattr(result, 'wishlist_added_count', 0) or 0)
+
+    if skip_wishlist_add:
+        org_fn = deps.run_playlist_organize_download
+        if failed <= 0:
+            return
+        if not org_fn:
+            logger.warning(
+                "Organize-by-playlist sync left %s missing tracks but organize download is unavailable",
+                failed,
+            )
+            deps.update_automation_progress(
+                automation_id,
+                log_line=f'{failed} missing — enable Playlist Pipeline or disable Organize by Playlist',
+                log_type='warning',
+            )
+            return
+        org_result = org_fn(mirrored_playlist_id=mirrored_id, automation_id=automation_id)
+        status = org_result.get('status', 'unknown') if isinstance(org_result, dict) else 'unknown'
+        reason = org_result.get('reason', '') if isinstance(org_result, dict) else ''
+        log_type = 'success' if status == 'started' else 'warning'
+        detail = f' ({reason})' if reason and status != 'started' else ''
+        deps.update_automation_progress(
+            automation_id,
+            log_line=f'Organize download {status} for {failed} missing track(s){detail}',
+            log_type=log_type,
+        )
+        return
+
+    if wishlist_added <= 0:
+        if failed > 0:
+            deps.update_automation_progress(
+                automation_id,
+                log_line=f'{failed} missing but none added to wishlist — check logs',
+                log_type='warning',
+            )
+        return
+
+    proc_fn = deps.process_wishlist_automatically
+    if not proc_fn:
+        return
+    is_busy = deps.is_wishlist_actually_processing
+    if is_busy and is_busy():
+        deps.update_automation_progress(
+            automation_id,
+            log_line=f'Added {wishlist_added} to wishlist; download worker already running',
+            log_type='info',
+        )
+        return
+    proc_fn(automation_id=automation_id)
+    deps.update_automation_progress(
+        automation_id,
+        log_line=f'Started wishlist download for {wishlist_added} track(s)',
+        log_type='success',
+    )
+
+
+async def _database_only_find_track(spotify_track, candidate_pool=None):
+    """Database-only track matcher used when no media server is connected.
+
+    Patched onto sync_service._find_track_in_media_server. Accepts
+    ``candidate_pool`` for interface parity with the real matcher (sync_service
+    calls it with candidate_pool=...); the DB path queries the library directly
+    via check_track_exists, so it doesn't need the per-artist candidate cache —
+    but it MUST accept the kwarg or sync raises "unexpected keyword argument
+    'candidate_pool'". Module-level (not a nested closure) so it's importable
+    and unit-tested.
+    """
+    logger.info(f"Database-only search for: '{spotify_track.name}' by {spotify_track.artists}")
+    try:
+        from database.music_database import MusicDatabase
+        from config.settings import config_manager
+
+        db = MusicDatabase()
+        active_server = config_manager.get_active_media_server()
+        original_title = spotify_track.name
+        spotify_id = getattr(spotify_track, 'id', '') or ''
+
+        # --- Sync match cache fast-path ---
+        if spotify_id:
+            try:
+                cached = db.read_sync_match_cache(spotify_id, active_server)
+                if cached:
+                    db_track_check = db.get_track_by_id(cached['server_track_id'])
+                    if db_track_check:
+                        class DatabaseTrackCached:
+                            def __init__(self, db_t):
+                                self.ratingKey = db_t.id
+                                self.title = db_t.title
+                                self.id = db_t.id
+                        logger.debug(f"Sync cache hit: '{original_title}' → server track {cached['server_track_id']}")
+                        return DatabaseTrackCached(db_track_check), cached['confidence']
+                    logger.warning(f"Sync cache stale for '{original_title}' — track gone")
+            except Exception as e:
+                logger.debug("sync match cache fast-path failed: %s", e)
+        # --- End cache fast-path ---
+
+        # Durable manual library match (#787) — survives a library rescan (the
+        # sync_match_cache above does not), so a user's Find & Add pairing keeps
+        # sticking across auto-syncs instead of being re-matched from scratch (#895
+        # follow-up). Self-heals a stale library id via the stored file path.
+        if spotify_id:
+            try:
+                from core.artists.map import get_current_profile_id
+                m = db.find_manual_library_match_by_source_track_id(
+                    get_current_profile_id(), str(spotify_id), active_server)
+                if m:
+                    lib_id = m.get('library_track_id')
+                    dt = db.get_track_by_id(lib_id) if lib_id is not None else None
+                    if not dt and m.get('library_file_path'):
+                        new_id = db.find_track_id_by_file_path(m['library_file_path'])
+                        dt = db.get_track_by_id(new_id) if new_id else None
+                    if dt:
+                        class DatabaseTrackDurable:
+                            def __init__(self, db_t):
+                                self.ratingKey = db_t.id
+                                self.title = db_t.title
+                                self.id = db_t.id
+                        logger.debug(f"Durable manual match hit: '{original_title}' → {lib_id}")
+                        return DatabaseTrackDurable(dt), 1.0
+            except Exception as e:
+                logger.debug("durable manual match fast-path failed: %s", e)
+
+        # Try each artist
+        for artist in spotify_track.artists:
+            if isinstance(artist, str):
+                artist_name = artist
+            elif isinstance(artist, dict) and 'name' in artist:
+                artist_name = artist['name']
+            else:
+                artist_name = str(artist)
+
+            db_track, confidence = db.check_track_exists(
+                original_title, artist_name,
+                confidence_threshold=0.80,
+                server_source=active_server
+            )
+
+            if not (db_track and confidence >= 0.80):
+                # #785: file/CSV playlists keep raw "Artist - Title" titles (unlike
+                # YouTube, cleaned at ingest), which don't match the clean library
+                # title. Retry with the canonical form (best-of, conservative).
+                try:
+                    from core.text.source_title import canonical_source_track
+                    _canon_title, _canon_artist = canonical_source_track(original_title, artist_name)
+                    if (_canon_title, _canon_artist) != (original_title, artist_name):
+                        _alt_track, _alt_conf = db.check_track_exists(
+                            _canon_title, _canon_artist,
+                            confidence_threshold=0.80, server_source=active_server)
+                        if _alt_track and _alt_conf > confidence:
+                            db_track, confidence = _alt_track, _alt_conf
+                except Exception as _canon_err:
+                    logger.debug("canonical retry failed: %s", _canon_err)
+
+            if db_track and confidence >= 0.80:
+                logger.info(f"Database match: '{db_track.title}' (confidence: {confidence:.2f})")
+                if spotify_id:
+                    try:
+                        from core.matching_engine import MusicMatchingEngine
+                        me = MusicMatchingEngine()
+                        db.save_sync_match_cache(
+                            spotify_id, me.clean_title(original_title), me.clean_artist(artist_name),
+                            active_server, db_track.id, db_track.title, confidence
+                        )
+                    except Exception as e:
+                        logger.debug("save sync match cache failed: %s", e)
+
+                class DatabaseTrackMock:
+                    def __init__(self, db_track):
+                        self.ratingKey = db_track.id
+                        self.title = db_track.title
+                        self.id = db_track.id
+
+                return DatabaseTrackMock(db_track), confidence
+
+        logger.warning(f"No database match found for: '{original_title}'")
+        return None, 0.0
+
+    except Exception as e:
+        logger.error(f"Database search error: {e}")
+        return None, 0.0
+
+
+def run_sync_task(
+    playlist_id,
+    playlist_name,
+    tracks_json,
+    automation_id=None,
+    profile_id=1,
+    playlist_image_url='',
+    deps: SyncDeps = None,
+    sync_mode: str = 'replace',
+    skip_wishlist_add: bool = False,
+):
+    """The actual sync function that runs in the background thread."""
+    sync_states = deps.sync_states
+    sync_lock = deps.sync_lock
+    sync_service = deps.sync_service
+
+    task_start_time = time.time()
+    logger.info(f"[TIMING] _run_sync_task STARTED for playlist '{playlist_name}' at {time.strftime('%H:%M:%S')}")
+    logger.info(f"Received {len(tracks_json)} tracks from frontend")
+
+    # Record sync history start (skip for re-syncs triggered from history)
+    _is_resync = playlist_id.startswith('resync_')
+    _resync_entry_id = None
+    sync_batch_id = f"sync_{playlist_id}_{int(time.time())}"
+    if _is_resync:
+        # Extract the original entry ID from resync_{entryId}_{timestamp}
+        try:
+            _resync_entry_id = int(playlist_id.split('_')[1])
+        except (IndexError, ValueError):
+            pass
+    else:
+        deps.record_sync_history_start(
+            batch_id=sync_batch_id,
+            playlist_id=playlist_id,
+            playlist_name=playlist_name,
+            tracks=tracks_json,
+            is_album_download=False,
+            album_context=None,
+            artist_context=None,
+            playlist_folder_mode=False,
+            source_page='sync'
+        )
+
+    try:
+        # Recreate a Playlist object from the JSON data sent by the frontend
+        # This avoids needing to re-fetch it from Spotify
+        logger.info("Converting JSON tracks to SpotifyTrack objects...")
+
+        # Store original track data with full album objects (for wishlist with cover art).
+        # Shared with the sync-detail "re-add to wishlist" action so both build the
+        # IDENTICAL payload (album→dict + images, artists→dicts). Copy-safe.
+        from core.sync.wishlist_readd import build_original_tracks_map
+        original_tracks_map = build_original_tracks_map(tracks_json)
+
+        tracks = []
+        for i, t in enumerate(tracks_json):
+            # Handle album field - extract name if it's a dictionary
+            raw_album = t.get('album', '')
+            if isinstance(raw_album, dict) and 'name' in raw_album:
+                album_name = raw_album['name']
+            elif isinstance(raw_album, str):
+                album_name = raw_album
+            else:
+                album_name = str(raw_album)
+            
+            # Extract image URL from album data if available
+            _track_image = ''
+            if isinstance(raw_album, dict):
+                _imgs = raw_album.get('images', [])
+                if _imgs and isinstance(_imgs, list) and len(_imgs) > 0:
+                    _track_image = _imgs[0].get('url', '') if isinstance(_imgs[0], dict) else ''
+            if not _track_image:
+                _track_image = t.get('image_url', '')
+
+            # Create SpotifyTrack objects with proper default values for missing fields
+            track = SpotifyTrack(
+                id=t.get('id', ''),
+                name=t.get('name', ''),
+                artists=t.get('artists', []),
+                album=album_name,
+                duration_ms=t.get('duration_ms', 0),
+                popularity=t.get('popularity', 0),
+                preview_url=t.get('preview_url'),
+                external_urls=t.get('external_urls'),
+                image_url=_track_image or None
+            )
+            tracks.append(track)
+            if i < 3:  # Log first 3 tracks for debugging
+                logger.info(f"  Track {i+1}: '{track.name}' by {track.artists}")
+        
+        logger.info(f"Created {len(tracks)} SpotifyTrack objects")
+        
+        playlist = SpotifyPlaylist(
+            id=playlist_id, 
+            name=playlist_name, 
+            description=None,  # Not needed for sync
+            owner="web_user",  # Placeholder  
+            public=False,      # Default
+            collaborative=False,  # Default
+            tracks=tracks, 
+            total_tracks=len(tracks)
+        )
+        logger.info(f"Created SpotifyPlaylist object: '{playlist.name}' with {playlist.total_tracks} tracks")
+
+        first_callback_time = [None]  # Use list to allow modification in nested function
+        
+        def progress_callback(progress):
+            """Callback to update the shared state."""
+            if first_callback_time[0] is None:
+                first_callback_time[0] = time.time()
+                first_callback_duration = (first_callback_time[0] - task_start_time) * 1000
+                logger.info(f"⏱️ [TIMING] FIRST progress callback at {time.strftime('%H:%M:%S')} (took {first_callback_duration:.1f}ms from start)")
+
+            logger.info(f"PROGRESS CALLBACK: {progress.current_step} - {progress.current_track}")
+            logger.error(f"   Progress: {progress.progress}% ({progress.matched_tracks}/{progress.total_tracks} matched, {progress.failed_tracks} failed)")
+
+            with sync_lock:
+                sync_states[playlist_id] = {
+                    "status": "syncing",
+                    "progress": progress.__dict__ # Convert dataclass to dict
+                }
+                logger.info(f"   Updated sync_states for {playlist_id}")
+
+            # Update automation progress card
+            if automation_id:
+                step = getattr(progress, 'current_step', '')
+                track = getattr(progress, 'current_track', '')
+                pct = getattr(progress, 'progress', 0)
+                matched = getattr(progress, 'matched_tracks', 0)
+                failed = getattr(progress, 'failed_tracks', 0)
+                total = getattr(progress, 'total_tracks', 0)
+                log_type = 'success' if 'matched' in step.lower() or 'found' in step.lower() else 'info'
+                if 'not found' in step.lower() or 'failed' in step.lower():
+                    log_type = 'error'
+                deps.update_automation_progress(automation_id, progress=pct,
+                    phase=f'Syncing: {step}',
+                    processed=matched + failed, total=total,
+                    current_item=track,
+                    log_line=f'{track} — {step}' if track else step, log_type=log_type)
+                
+    except Exception as setup_error:
+        logger.error(f"SETUP ERROR in _run_sync_task: {setup_error}")
+        import traceback
+        traceback.print_exc()
+        with sync_lock:
+            sync_states[playlist_id] = {
+                "status": "error",
+                "error": f"Setup error: {str(setup_error)}"
+            }
+        if automation_id:
+            deps.update_automation_progress(automation_id, status='error', progress=100,
+                phase='Error', log_line=f'Setup error: {str(setup_error)}', log_type='error')
+        return
+
+    try:
+        logger.info("Setting up sync service...")
+        logger.info(f"   sync_service available: {sync_service is not None}")
+        
+        if sync_service is None:
+            raise Exception("sync_service is None - not initialized properly")
+            
+        # Check sync service components
+        logger.info(f"   spotify_client: {sync_service.spotify_client is not None}")
+        _ms_engine = getattr(sync_service, '_engine', None)
+        logger.info(f"   plex_client: {(_ms_engine.client('plex') if _ms_engine else None) is not None}")
+        logger.info(f"   jellyfin_client: {(_ms_engine.client('jellyfin') if _ms_engine else None) is not None}")
+        
+        # Check media server connection before starting
+        from config.settings import config_manager
+        active_server = config_manager.get_active_media_server()
+        logger.info(f"   Active media server: {active_server}")
+        
+        media_client, server_type = sync_service._get_active_media_client()
+        logger.info(f"   Media client available: {media_client is not None}")
+        
+        if media_client:
+            is_connected = media_client.is_connected()
+            logger.info(f"   Media client connected: {is_connected}")
+        
+        # Check database access
+        try:
+            from database.music_database import MusicDatabase
+            db = MusicDatabase()
+            logger.debug(f"   Database initialized: {db is not None}")
+        except Exception as db_error:
+            logger.error(f"   Database initialization failed: {db_error}")
+        
+        logger.info("Attaching progress callback...")
+        # Attach the progress callback
+        sync_service.set_progress_callback(progress_callback, playlist.name)
+        logger.info(f"Progress callback attached for playlist: {playlist.name}")
+
+        # CRITICAL FIX: Add database-only fallback for web context
+        # If media client is not connected, patch the sync service to use database-only matching
+        if media_client is None or not media_client.is_connected():
+            logger.info("Media client not connected - patching sync service for database-only matching")
+            
+            # Patch the matcher to the module-level database-only implementation
+            # (importable + unit-tested; accepts candidate_pool for parity).
+            sync_service._find_track_in_media_server = _database_only_find_track
+            logger.info("Patched sync service to use database-only matching")
+
+        sync_start_time = time.time()
+        setup_duration = (sync_start_time - task_start_time) * 1000
+        logger.info(f"⏱️ [TIMING] Setup completed at {time.strftime('%H:%M:%S')} (took {setup_duration:.1f}ms)")
+        logger.info("Starting actual sync process with run_async()...")
+
+        # Attach original tracks map to sync_service for wishlist with album images
+        sync_service._original_tracks_map = original_tracks_map
+
+        # Wing It mode — skip wishlist for unmatched tracks
+        with sync_lock:
+            is_wing_it = sync_states.get(playlist_id, {}).get('wing_it', False)
+        sync_service._skip_unmatched_wishlist = is_wing_it or skip_wishlist_add
+        sync_service._skip_wishlist = is_wing_it
+        if skip_wishlist_add:
+            logger.info(
+                "[Organize by Playlist] Skipping sync-time wishlist for '%s' — "
+                "organize download + batch failure handling cover missing tracks",
+                playlist_name,
+            )
+
+        # Run the sync (this is a blocking call within this thread)
+        result = deps.run_async(sync_service.sync_playlist(playlist, download_missing=False, profile_id=profile_id, sync_mode=sync_mode))
+
+        # Clear progress callback immediately to prevent race condition where a
+        # late-firing progress callback overwrites the "finished" state below
+        if sync_service:
+            sync_service.clear_progress_callback(playlist.name)
+
+        sync_duration = (time.time() - sync_start_time) * 1000
+        total_duration = (time.time() - task_start_time) * 1000
+        logger.info(f"⏱️ [TIMING] Sync completed at {time.strftime('%H:%M:%S')} (sync: {sync_duration:.1f}ms, total: {total_duration:.1f}ms)")
+        logger.info(f"Sync process completed! Result type: {type(result)}")
+        logger.info(f"   Result details: matched={getattr(result, 'matched_tracks', 'N/A')}, total={getattr(result, 'total_tracks', 'N/A')}")
+
+        # Update final state on completion
+        # Convert result to JSON-serializable dict (datetime/errors can't be emitted via SocketIO)
+        # Exclude match_details (large) but include a summary of unmatched tracks
+        result_dict = {
+            k: (v.isoformat() if hasattr(v, 'isoformat') else v)
+            for k, v in result.__dict__.items()
+            if k != 'match_details'
+        }
+        # Include unmatched track names so the frontend can show which tracks failed
+        match_details = getattr(result, 'match_details', None)
+        if match_details:
+            unmatched_summary = [
+                {'name': d.get('name', ''), 'artist': d.get('artist', ''), 'image_url': d.get('image_url', '')}
+                for d in match_details if d.get('status') == 'not_found'
+            ]
+            if unmatched_summary:
+                result_dict['unmatched_tracks'] = unmatched_summary
+        with sync_lock:
+            sync_states[playlist_id] = {
+                "status": "finished",
+                "progress": result_dict,
+                "result": result_dict
+            }
+        logger.info(f"Sync finished for {playlist_id} - state updated")
+
+        # Set playlist poster image if available (Plex, Jellyfin, Emby).
+        # Don't log the URL itself — it may carry an auth token (Plex
+        # X-Plex-Token / Jellyfin X-Emby-Token / Subsonic auth) that we
+        # don't want persisted to app.log.
+        _synced = getattr(result, 'synced_tracks', 0)
+        logger.info(f"[PLAYLIST IMAGE] has_image={bool(playlist_image_url)}, synced_tracks={_synced}")
+        # Modes that edit a playlist in place (reconcile #792, append #811) must
+        # NOT push the source image — doing so re-clobbers a user's custom poster
+        # every sync, the exact bug these modes exist to avoid. Only the
+        # destructive 'replace' (recreate-from-scratch) pushes the image.
+        if sync_mode in ('reconcile', 'append'):
+            logger.info(f"[PLAYLIST IMAGE] {sync_mode} mode — preserving existing playlist image")
+        elif playlist_image_url and _synced > 0:
+            try:
+                active_server = deps.config_manager.get_active_media_server()
+                logger.info(f"[PLAYLIST IMAGE] active_server={active_server}")
+                _engine = deps.media_server_engine
+                if active_server == 'plex' and _engine and _engine.client('plex'):
+                    ok = _engine.client('plex').set_playlist_image(playlist_name, playlist_image_url)
+                    logger.info(f"[PLAYLIST IMAGE] Plex upload result: {ok}")
+                elif active_server in ('jellyfin', 'emby') and _engine and _engine.client('jellyfin'):
+                    ok = _engine.client('jellyfin').set_playlist_image(playlist_name, playlist_image_url)
+                    logger.info(f"[PLAYLIST IMAGE] Jellyfin upload result: {ok}")
+                # Navidrome doesn't support custom playlist images
+            except Exception as img_err:
+                logger.error(f"[PLAYLIST IMAGE] Exception: {img_err}")
+
+        # Record sync history completion with per-track data
+        try:
+            matched = getattr(result, 'matched_tracks', 0)
+            failed = getattr(result, 'failed_tracks', 0)
+            synced = getattr(result, 'synced_tracks', 0)
+            db = MusicDatabase()
+            target_batch_id = sync_batch_id
+            if _is_resync and _resync_entry_id:
+                db.refresh_sync_history_entry(_resync_entry_id, matched, synced, failed)
+                # For re-sync, get the batch_id from the original entry
+                try:
+                    entry = db.get_sync_history_entry(_resync_entry_id)
+                    if entry:
+                        target_batch_id = entry.get('batch_id', sync_batch_id)
+                except Exception as e:
+                    logger.debug("resync history lookup failed: %s", e)
+            else:
+                db.update_sync_history_completion(sync_batch_id, matched, synced, failed)
+
+            # Save per-track match details from sync service
+            match_details = getattr(result, 'match_details', None)
+            if match_details:
+                try:
+                    track_results_json = json.dumps(match_details, default=str)
+                    saved = db.update_sync_history_track_results(target_batch_id, track_results_json)
+                    logger.info(f"[Sync History] Saved {len(match_details)} track results for batch {target_batch_id} (saved={saved})")
+                except Exception as json_err:
+                    logger.error(f"[Sync History] Failed to serialize track results: {json_err}")
+            else:
+                logger.warning(f"[Sync History] No match_details on SyncResult for batch {target_batch_id}")
+        except Exception as e:
+            logger.warning(f"Failed to record sync history completion: {e}")
+
+        if automation_id:
+            matched = getattr(result, 'matched_tracks', 0)
+            total = getattr(result, 'total_tracks', 0)
+            failed = getattr(result, 'failed_tracks', 0)
+            wishlist_added = getattr(result, 'wishlist_added_count', 0) or 0
+            deps.update_automation_progress(automation_id, status='finished', progress=100,
+                phase='Sync complete',
+                log_line=(
+                    f'Done: {matched}/{total} in library, {failed} missing'
+                    + (f', {wishlist_added} added to wishlist' if wishlist_added else '')
+                ),
+                log_type='success')
+            _post_sync_automation_followup(
+                deps,
+                automation_id=automation_id,
+                playlist_id=playlist_id,
+                skip_wishlist_add=skip_wishlist_add,
+                result=result,
+            )
+
+        # Emit playlist_synced event for automation engine
+        try:
+            if deps.automation_engine:
+                deps.automation_engine.emit('playlist_synced', {
+                    'playlist_name': playlist_name,
+                    'total_tracks': str(getattr(result, 'total_tracks', 0)),
+                    'matched_tracks': str(getattr(result, 'matched_tracks', 0)),
+                    'synced_tracks': str(getattr(result, 'synced_tracks', 0)),
+                    'failed_tracks': str(getattr(result, 'failed_tracks', 0)),
+                })
+        except Exception as e:
+            logger.debug("playlist_synced emit failed: %s", e)
+
+        # Save sync status with match counts and track hash for smart-skip on next scheduled sync
+        import hashlib as _hl
+        _track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
+        _tracks_hash = _hl.md5(_track_ids_str.encode()).hexdigest()
+        _mirror_tracks_hash = None
+        if str(playlist_id).startswith('auto_mirror_'):
+            try:
+                _mp_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
+                from database.music_database import MusicDatabase
+                _mtracks = MusicDatabase().get_mirrored_playlist_tracks(_mp_id)
+                _mids = ','.join(
+                    sorted(t.get('source_track_id', '') or '' for t in _mtracks if t.get('source_track_id'))
+                )
+                _mirror_tracks_hash = _hl.md5(_mids.encode()).hexdigest() if _mids else ''
+            except Exception as e:
+                logger.debug("mirror_tracks_hash for sync status: %s", e)
+        snapshot_id = getattr(playlist, 'snapshot_id', None)
+        _status_kwargs = dict(
+            matched_tracks=getattr(result, 'matched_tracks', 0),
+            total_tracks=getattr(result, 'total_tracks', 0),
+            discovered_tracks=len(tracks_json),
+            tracks_hash=_tracks_hash,
+        )
+        if _mirror_tracks_hash is not None:
+            _status_kwargs['mirror_tracks_hash'] = _mirror_tracks_hash
+        deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id, **_status_kwargs)
+
+    except Exception as e:
+        logger.error(f"SYNC FAILED for {playlist_id}: {e}")
+        import traceback
+        traceback.print_exc()
+        with sync_lock:
+            sync_states[playlist_id] = {
+                "status": "error",
+                "error": str(e)
+            }
+        if automation_id:
+            deps.update_automation_progress(automation_id, status='error', progress=100,
+                phase='Error', log_line=f'Sync failed: {str(e)}', log_type='error')
+    finally:
+        logger.info(f"Cleaning up progress callback for {playlist.name}")
+        # Clean up the callback
+        if sync_service:
+            sync_service.clear_progress_callback(playlist.name)
+            # Clean up original tracks map
+            if hasattr(sync_service, '_original_tracks_map'):
+                del sync_service._original_tracks_map
+        logger.info(f"Cleanup completed for {playlist_id}")