Live-API verification revealed advanced-syntax queries hurt more
than they help on this endpoint. Switching the import-modal Deezer
search back to free-text + local rerank.
# What live testing showed
Hit Deezer's public API with both query forms for the issue #534
case (`Dirty White Boy` + `Foreigner`):
**Free-text (`q=Dirty White Boy Foreigner`):**
- Returns 21 results
- Real Foreigner Head Games studio cut at #1
- Live versions at #2-10
- Karaoke / cover variants at #11-15
**Advanced (`q=track:"Dirty White Boy" artist:"Foreigner"`):**
- Returns 12 results
- "(2008 Remaster)" at #1 — canonical Head Games cut MISSING from
top 8 entirely
- Live + alt-album versions follow
Advanced syntax DOES filter karaoke at the API level (none in the
12-result set vs. 5 at positions 11-15 in free-text), but it has
its own ranking bias that surfaces remasters / "Best Of" cuts
ahead of the canonical recording. Net regression for the user-
facing goal.
# Fix
1. Endpoint reverts to free-text query with local rerank applied.
2. Local rerank gains "remaster" / "remastered" / "reissue"
patterns under VARIANT_TAG_PATTERNS (soft 0.4× penalty — user
may want them but they shouldn't outrank the original).
3. Client kwarg support (`track=` / `artist=` / `album=`) preserved
for future opt-in callers (e.g. exact-match flows where API-
level filtering matters more than ranking).
# Verified end-to-end against live Deezer API
Re-ran the exact #534 case through the live API + new rerank.
Top 15 results post-rerank:
1. Dirty White Boy — Foreigner — Head Games ← REAL CUT AT TOP
2-10. Various Live versions
11-15. Karaoke / cover / tribute variants ← BURIED
Real Foreigner Head Games studio cut at #1, exactly the user's
ask.
# Tests
- `test_relevance.py` — variant tag patterns extended; existing
tests still pass (50 tests).
- `test_search_match_endpoints.py::test_joins_track_and_artist_into_free_text_query`
— replaces `test_passes_track_and_artist_as_kwargs`; verifies
endpoint sends free-text join, NOT field-scoped kwargs (the
prior test asserted the wrong direction now).
- Karaoke-burying assertion at the endpoint still pins the
user-visible behaviour.
- Client kwarg path tests untouched (still pin advanced-syntax
construction for future opt-in callers).
# Verification
- 75 relevance + endpoint + query tests pass
- 2445 full suite passes
- Ruff clean
- Live Deezer API shows real cut at #1 post-rerank
Defensive followup to the relevance fix. Deezer's advanced search
syntax (`artist:"X"`) is documented as substring match, but in
practice it's brittle on artist name variants ("Foreigner [US]",
"The Foreigner") and on tracks indexed under non-canonical title
spellings. When the advanced query returns nothing, we'd previously
land at "No matches" — a regression vs. pre-fix behaviour where
free-text would have returned a less-relevant but non-empty set.
Fix: when the advanced query returns 0 results AND the caller used
field-scoped kwargs, fall back to a free-text join of the same
kwargs and re-query. Caller-side rerank still tightens whatever the
fallback returns, so the worst-case post-fix behaviour is the
pre-fix behaviour — never strictly worse.
Pulled the cache + parse + store dance into a private helper
(`_search_tracks_with_query`) so the orchestration can call it
twice (advanced → fallback) without code duplication. Single API
call when the advanced query has results — no wasted requests.
Diagnostic logger.debug fires when the fallback triggers so we can
see in production whether it's happening (and to which queries).
# Tests added (4)
- `test_falls_back_to_free_text_when_advanced_empty` — advanced
query returns 0, free-text returns hits; client returns the
free-text hits + both API calls fire.
- `test_no_fallback_when_advanced_query_has_results` — single hit
on advanced query → no second API call.
- `test_no_fallback_when_legacy_free_text_call` — legacy callers
already exhausted the only path; empty result is final.
- `test_no_fallback_when_query_unchanged` — empty kwargs path
doesn't trigger the fallback branch (used_advanced=False).
# Existing tests updated
The 4 prior `TestSearchTracksQueryWiring` + `TestSearchTracksCacheKey`
tests were stubbing `_api_get` to return empty `{'data': []}` and
asserting `assert_called_once`. With the new fallback, those stubs
trigger a second API call and the assertions break — even though
the FIRST call construction is what the tests cared about. Updated
the stubs to return one fake hit so the fallback doesn't fire, and
switched to `call_args_list[0]` for first-call inspection.
# Verification
- 18/18 deezer query tests pass (14 prior + 4 new)
- 2445 full suite passes (+4 from prior commit)
- Ruff clean
# Background
User reported (#534) that the import-modal "Search for Match" dialog
returned irrelevant results when Deezer was the metadata source.
Searching `Dirty White Boy` + `Foreigner` returned 5+ karaoke /
"originally performed by" / "in the style of" / "re-recorded" /
tribute-band results ranked above the actual Foreigner studio cut
from Head Games. User had to scroll past the junk every time, or
fall back to iTunes search which is much slower.
# Root cause — two layers
1. **Endpoint joined `track + artist` into free-text query.**
`/api/deezer/search_tracks` was passing `q=Dirty White Boy Foreigner`
to Deezer's `/search/track` API. Deezer fuzzy-matches that
string across title / lyrics / artist / album / contributors and
orders by global popularity — anything that appears across many
compilations outranks the canonical recording.
2. **No local rerank.** None of the search-modal endpoints applied
any post-filtering. Deezer's API order shipped straight to the
user.
# Fix — same architectural shape Cin would build
## Layer 1: field-scoped query at the client boundary
`core/deezer_client.py::search_tracks()` now accepts optional
`track`, `artist`, `album` kwargs. When provided, builds Deezer's
advanced search syntax: `q=track:"X" artist:"Y" album:"Z"`. Massive
relevance improvement because each term matches the right field
instead of fuzzy-matching everywhere.
Backward compat preserved: legacy free-text `query=` callers still
work unchanged. Field-scoped path takes precedence when both are
provided. Empty input fast-fails without an API call. Embedded
double-quotes stripped (Deezer's syntax has no escape mechanism).
## Layer 2: provider-neutral relevance reranker
New `core/metadata/relevance.py` module — pure-function rerank over
the canonical `Track` dataclass. Composable scoring:
- **Cover/karaoke patterns** (multiplier 0.05, effectively buries):
matches "karaoke", "originally performed by", "in the style of",
"made famous by", "tribute", "vocal version", "backing track",
"cover version", "re-recorded", "cover by", etc. across title,
album, AND artist fields. Catches the screenshot's exact junk:
artist credits like "Pop Music Workshop" / "The Karaoke Channel"
/ "Foreigner Tribute Band".
- **Variant tags** (multiplier 0.4): live / acoustic / demo /
instrumental / remix / radio edit / club mix etc. — softer
penalty since the user MAY want them. Skipped entirely when the
expected_title contains the same tag (so searching
"Track (Live)" still ranks Live versions first).
- **Exact artist boost** (multiplier 1.5): primary artist exactly
matches expected_artist after normalisation. Single strongest
signal for "this is the canonical recording".
- **Title + artist similarity** via SequenceMatcher (parentheticals
+ punctuation stripped before comparison).
- **Album-type weighting**: album=1.0 > single/ep=0.85 > compilation=0.7.
Compilations are more likely tribute / karaoke repackages.
Each component is a standalone function so tests pin them
individually without standing up the full pipeline.
## Wired at three search-modal endpoints
- `/api/deezer/search_tracks` — uses both layers (field-scoped
query + rerank).
- `/api/itunes/search_tracks` — uses rerank only (iTunes API has
no advanced-syntax search, but karaoke / cover variants still
leak through and need the local penalty).
- `/api/spotify/search_tracks` — already builds field-scoped
`track:X artist:Y` query; rerank added as the consistency safety
net so all three sources behave the same from the user's
perspective.
Other Deezer call sites (matching engine, watchlist scanner,
auto-import single-track ID) deliberately not touched in this PR
— they have their own elaborate scoring pipelines tuned to their
specific contexts and aren't surfacing the user-reported issue.
Per Cin: "don't refactor beyond what the task requires."
# Tests
71 new tests across 3 files:
- `tests/metadata/test_relevance.py` (50 tests) — every scoring
component pinned individually + the issue #534 screenshot
reproduced as a regression test (real Foreigner cut wins after
rerank, karaoke variants drop to bottom).
- `tests/metadata/test_deezer_search_query.py` (14 tests) —
advanced-syntax query construction, field-scoped wiring at the
client boundary, free-text path unchanged, kwargs win when
ambiguous, limit clamping, cache key consistency.
- `tests/imports/test_search_match_endpoints.py` (7 tests) —
end-to-end through Flask test client: Deezer endpoint passes
kwargs not joined query; karaoke buried at bottom for all three
sources; legacy query param still works without rerank.
# Verification
- 2441 full suite passes (+71 from baseline 2370)
- 0 failures (the prior watchdog flake fix held)
- Ruff clean across all changed files
- JS parses clean (`node -c webui/static/helper.js`)
# Architectural standards followed
- **Logic at the right boundary.** Query construction lives in the
client (every caller benefits from one change). Rerank lives in
a neutral module (`core/metadata/relevance.py`) over the
canonical `Track` dataclass — works for any source, not Deezer-
specific.
- **Explicit > implicit.** Every scoring rule has its own named
function. Pattern tables are module-level constants tests can
introspect.
- **Scope discipline.** Audited every Deezer search call site;
fixed the user-reported one + the consistent siblings. Did NOT
speculatively normalise every Deezer call across the codebase.
- **Backward compat.** Free-text `query=` callers untouched. Kwargs
added to existing client method signature with safe defaults.
- **Tests pin contract at correct boundary.** Pure-function rerank
tests don't mock anything; client-query tests stub at `_api_get`;
endpoint tests run through the real Flask app.
Two pre-existing parity gaps in `record_soulsync_library_entry` that
the prior parity commits left untouched. Both close real holes
between auto-import writes and what the soulsync_client deep scan
would have produced.
# Gap 1: Album duration was the first-imported track's duration
`record_soulsync_library_entry` is called once per track. The album
INSERT only fires for the FIRST track of a new album (subsequent
tracks find the album row already exists). The INSERT was passing
`duration_ms` — `track_info["duration_ms"]` — as the album's
`duration` column. That's the duration of one track, not the album
total. Compare to `SoulSyncAlbum.duration` in soulsync_client which
is `sum(t.duration for t in self._tracks)`.
Fix:
- Worker computes `album_total_duration_ms = sum(...)` across every
matched track and threads it onto context as
`album.duration_ms`.
- side_effects reads that value (or falls back to the per-track
duration for legacy non-auto-import callers) and writes it as the
album row's `duration`.
# Gap 2: Re-imports of the same artist/album were insert-only
When the SELECT-by-id or SELECT-by-name found an existing soulsync
artist or album row, the function skipped completely — no UPDATE
path. Meant: artist genres / thumb / source-id reflected ONLY
whatever the FIRST imported album supplied, never refreshing as
more albums by that artist landed. Ten more imports later, the
artist row still held whatever the first random import wrote.
Conservative fix: when an existing row matches, run an UPDATE that
fills only the columns whose current value is NULL or empty. Never
overwrites populated values — protects manual edits +
enrichment-worker writes the same way the scanner UPDATE path
preserves enrichment columns.
Implementation note: the empty-check happens in Python, NOT SQL.
Initial pass tried `COALESCE(NULLIF(col, ''), NULLIF(col, 0), ?)`
but SQLite's `NULLIF(text_col, 0)` returns the original text value
instead of NULL — different types, no coercion. So the SQL-only
conditional was unreliable on text columns. New helper does
`SELECT cols FROM table WHERE id`, compares each column in Python,
and emits UPDATE clauses only for the ones that need filling.
Allowlist defense: f-string column names go through
`_SOULSYNC_FILLABLE_COLUMNS` validation before interpolation.
Misuse adding new columns without an allowlist update fails closed
(logger.debug + skip).
# Tests added (4)
- `test_album_duration_uses_album_total_not_single_track` —
album with single-track context carrying explicit
`album.duration_ms = 2_500_000` writes 2_500_000 to the album row,
not the per-track 200_000 fallback.
- `test_re_import_fills_empty_artist_fields` — first import lands
artist with empty thumb + empty genres; second import for same
artist with thumb + genres present updates the existing row.
- `test_re_import_does_not_clobber_populated_artist_fields` —
first import writes rich genres + thumb; second import with
worse / different metadata leaves the existing row untouched.
- `test_re_import_fills_empty_source_id_when_missing` — first
import had no source artist ID; second import does — fills the
empty `spotify_artist_id` column on the existing row.
# Verification
- 10/10 side-effects tests pass (including 4 new + 4 from prior
parity commit + 2 history/provenance)
- 217 imports tests pass (no regression)
- 2369 full suite passes (+4 from prior, +22 PR-total from baseline 2347)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated)
- Ruff clean
Cin pre-review followup. Two small parity gaps the prior commits left
open:
# 1. Genre tags land on the standalone artists row
`soulsync_client._scan_transfer` aggregates the GENRE tag across every
track in an album and surfaces it on `SoulSyncAlbum.genres` (which the
DatabaseUpdateWorker writes to the artists+albums row). Auto-import
was hardcoding `'spotify_artist': {'genres': []}` so the imported
artists row landed with empty genres — felt hollow compared to a
Plex/Jellyfin scan, which both pull genres from their respective APIs.
Fix:
- `_read_file_tags` now reads the GENRE tag (mutagen easy mode handles
MP3/FLAC/M4A consistently; some files carry multiple genres so it's
always returned as a list).
- `_process_matches` aggregates genres from each matched file's tags
into a deduped insertion-order list. Dedup is case-insensitive but
preserves original casing — so "Hip-Hop, Rap, Trap" reads naturally
in the JSON column instead of "hip-hop, rap, trap".
- Worker context's `spotify_artist['genres']` carries the aggregated
list, which `record_soulsync_library_entry` already filters via
`core.genre_filter.filter_genres` and writes to the artists row.
# 2. Defensive str() cast for ISRC + MBID
`_build_album_track_entry` already coerces ISRC + MBID to string today
(via `str(isrc) if isrc else ''`). But if a future metadata-source
client returns int / None for either ID, the worker would propagate
the wrong type and side_effects.py's `.strip()` would AttributeError.
Cheap insurance: explicit `str()` cast in the worker before assignment
to track_info. Future-proofs against client drift.
# Tests added (3, in test_auto_import_context_shape.py):
- `test_context_aggregates_genres_from_track_tags` — multi-file
album with overlapping genre lists produces deduped, insertion-
ordered, original-case-preserved result. Stubs `_read_file_tags`
with monkeypatch so we don't need real audio.
- `test_context_genres_empty_when_no_tags` — files without GENRE
tag → empty list. Standalone library write handles gracefully
(genres column stays empty / NULL).
- `test_context_isrc_mbid_coerced_to_string` — hostile types
(int 12345678, None, int 999) coerced to safe strings before
reaching track_info.
# Verification
- 14/14 context-shape tests pass (11 prior + 3 new)
- 213 imports tests pass (no regression)
- 2365 full suite passes (+3 from prior, +18 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
Cin pre-review followup to the standalone library parity commit. The
prior commit fixed `spotify_artist['id']` from the wrong copy-paste
value (`identification['album_id']`) to read from
`identification['artist_id']`, but the identification dict produced
by `_search_metadata_source` and `_search_single_track` never set
`artist_id` — both extracted artist NAME from the search response
and discarded the source ID sitting right next to it. Net effect of
the prior commit: artists row source-id stayed NULL, just for a more
honest reason than before.
Now properly extracted:
- `_search_metadata_source` reads `best_result.artists[0]['id']`
alongside the artist name and returns it on the identification dict
as `artist_id`.
- `_search_single_track` does the same for single-track identification.
- `_identify_single`'s tag-based-confidence path forwards
`result.get('artist_id')` so the artist source-id propagates even
when high-confidence local tags override the search result's name.
Result: identification dict now carries `artist_id` whenever the
metadata source returned an artist with an ID. The worker context
already plumbs it onto `spotify_artist['id']` and
`spotify_album['artists'][0]['id']`, so the standalone library write
finally populates `<source>_artist_id` on the artists row.
Tests added (3, in `test_auto_import_context_shape.py`):
- `test_context_artist_id_uses_identification_artist_id` — when the
identification dict carries `artist_id`, context propagates it
onto `spotify_artist['id']` AND
`spotify_album['artists'][0]['id']`. Pins that the prior copy-
paste bug (artist['id'] = album_id) doesn't return.
- `test_context_artist_id_is_empty_when_identification_missing_it` —
fallback case (filename-only identification): context gets empty
string, NOT album_id. Honest failure mode.
- `test_search_metadata_source_extracts_artist_id_from_dict_artist`
— black-box test of `_search_metadata_source`: feed it a
spotify-shaped result with `artists[0]['id']` and verify
identification dict carries it forward.
Verification:
- 11/11 context-shape tests pass (8 prior + 3 new)
- 210 imports tests pass (no regression)
- 2362 full suite passes (+3 from prior commit, +15 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
# Background
SoulSync standalone is meant to be a full replacement for Plex /
Jellyfin / Navidrome — files imported via auto-import (or any other
import path) should land in the database with the same field richness
a media-server scan would write. They weren't.
# Gaps fixed
The auto-import worker built a context dict for each track and handed
it to `_post_process_matched_download` (the same callback the regular
download flow uses). That dict was missing three things downstream
needed:
1. **No `source` field anywhere.** `record_soulsync_library_entry`
reads `get_import_source(context)` to pick the source-aware ID
columns (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
etc.) on the artists / albums / tracks rows. With no source, the
resolver returned an empty string → `get_library_source_id_columns("")`
returned an empty dict → the `UPDATE tracks SET <source>_id = ?`
blocks were silently skipped. Result: every auto-imported track
landed with NULL on every source-id column. Watchlist scans
(which match by stable source IDs to detect "this track is already
in library") couldn't recognise these rows and would re-download
them on the next pass.
2. **No `_download_username='auto_import'`.** Both
`record_library_history_download` and `record_download_provenance`
default to "Soulseek" when no `username` is in the context. Every
staging-folder import was being labelled as a Soulseek download
in library history + provenance — false signal in the UI.
3. **No per-recording IDs (`isrc`, `musicbrainz_recording_id`) on
track_info.** The Navidrome scanner already writes
`musicbrainz_recording_id` directly to the tracks row when present.
Picard-tagged libraries always carry MBID; metadata sources
(Spotify via MusicBrainz enrichment, Deezer, etc.) carry ISRC.
Auto-import had access to both via the metadata-source response
but didn't propagate them — so the soulsync row went in with
NULL on both columns.
# Changes
**`core/auto_import_worker.py` — `_process_matches`:**
- Top-level `'source': source` (from `identification['source']`)
- `'_download_username': 'auto_import'`
- `track_info['isrc']`, `track_info['musicbrainz_recording_id']` —
pulled from the per-track payload returned by the metadata source
- `track_info['album_id']` — back-reference so source-aware ID
resolution works on sources whose API nests album under
`track.album.id` rather than `track.album_id`
- `spotify_artist['id']` now correctly carries the artist's source ID
(was `identification['album_id']`, a copy-paste bug from the
original implementation that made artist-id resolution fall back
to fuzzy matching)
- `spotify_album['artists'][0]['id']` carries artist source ID for
the same resolution path
**`core/imports/side_effects.py`:**
- `record_library_history_download` source_map: add
`"auto_import": "Auto-Import"` — tags imported tracks correctly
- `record_download_provenance` source_service: add
`"auto_import": "auto_import"` — provenance shows real source
- `record_soulsync_library_entry` track INSERT: now includes
`musicbrainz_recording_id` + `isrc` columns (matches
`insert_or_update_media_track`'s shape for Navidrome /
Plex / Jellyfin scans). Both default to NULL when not present.
# Behavior preserved
- Files still land in the same library template path (no path-build
change)
- Other media-server flows (Plex / Jellyfin / Navidrome users)
unaffected — `record_soulsync_library_entry` still gates on
`get_active_media_server() == "soulsync"`. Auto-import on those
servers continues to drop the file in the library folder + emits
`batch_complete` for the scan-trigger automation, same as before.
- Direct downloads (search → Download button) unaffected — they
already passed `source` + `username` correctly.
# Tests added
`tests/imports/test_auto_import_context_shape.py` (8 tests, new file):
- Worker context carries `source` for every metadata source
(parametrised across spotify / deezer / itunes / discogs)
- `_download_username='auto_import'` set unconditionally
- ISRC + MBID propagate from track payload to track_info when present
- ISRC + MBID default to empty string when absent (downstream
normalises to NULL at write time)
- track_info includes album-id back-reference
`tests/imports/test_import_side_effects.py` (4 new tests + 2 schema
column adds):
- `record_soulsync_library_entry` writes mbid + isrc columns when
present in track_info
- Deezer source maps to deezer_id column (regression case for
source-aware column resolver)
- `record_library_history_download` labels `_download_username=
'auto_import'` as "Auto-Import" not "Soulseek"
- `record_download_provenance` registers source_service as
"auto_import" not "soulseek"
# Verification
- 8/8 new context-shape tests pass
- 6/6 side-effects tests pass (4 new + 2 existing)
- 207 imports tests pass
- 2359 full suite passes (+12 from baseline 2347, no regressions)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated to this change)
- Ruff clean
# Concurrency model
Pre-refactor concurrency was emergent + unbounded:
- The worker's `_run` thread called `_scan_cycle` every 60s,
processing candidates synchronously in a for-loop.
- The `/api/auto-import/scan-now` endpoint spawned a fresh
`threading.Thread(target=_scan_cycle)` per click — extra parallel
scan cycles on top of the timer.
- Multiple "Scan Now" clicks during in-flight processing → multiple
threads racing on `_processing_paths` / `_folder_snapshots` state,
no upper bound on concurrent scanners.
- `stop()` didn't wait for in-flight processing — could leave file
moves / tag writes / DB inserts mid-flight.
Refactor to the pattern Cin uses elsewhere (`missing_download_executor`,
`sync_executor`, `import_singles_executor` all use
`ThreadPoolExecutor(max_workers=3, thread_name_prefix=...)`):
- **One scan thread** — both timer + manual triggers go through
`trigger_scan()`, gated by a non-blocking `_scan_lock`. Duplicate
triggers no-op instead of stacking parallel scanners.
- **Bounded executor** — `ThreadPoolExecutor` (default 3 workers,
configurable via `auto_import.max_workers`) runs per-candidate
work. Each candidate runs to completion in its own pool thread;
up to N candidates run in parallel.
- `_scan_and_submit()` is fast — just enumeration + executor submit,
returns immediately, doesn't block on per-candidate work.
- `_process_one_candidate(candidate)` holds the per-candidate logic
identical to the old for-loop body, lifted into a method so the
pool can run multiple instances concurrently.
- `_submitted_hashes` set + lock dedupes candidates across the
timer + manual triggers so a candidate already queued / running
doesn't get re-submitted.
- `stop()` calls `executor.shutdown(wait=True)` — clean shutdown,
no orphaned file ops.
# Per-candidate UI state isolation
The executor refactor opened two concurrency holes that the old
sequential model masked. Both fixed in this commit:
1. **Scalar UI fields stomped across pool workers.** Pre-refactor
`_current_folder` / `_current_status` / `_current_track_*` were
safe under the sequential model — only one candidate processed
at a time, so the fields tracked the in-flight one. With three
pool workers writing the same fields, the polling UI saw garbage
like "Processing AlbumA, track 7/14: SongFromAlbumB".
Replaced with `_active_imports: Dict[hash, _ActiveImport]` keyed
on folder_hash, gated by `_active_lock`. Each pool worker owns
its own entry. Helpers `_register_active` / `_update_active` /
`_unregister_active` / `_snapshot_active` are the only API.
2. **Stats counters not thread-safe.** `self._stats[k] += 1` is
read-modify-write — under load, parallel pool workers drop
increments. New `_stats_lock` + `_bump_stat()` helper wraps every
mutation. `get_status()` reads under the same lock and returns
a copy.
# Endpoint change
`/api/auto-import/scan-now` no longer spawns its own scan thread —
calls `auto_import_worker.trigger_scan()` (which routes through the
shared lock + executor). Multiple clicks while a scan is in flight
no-op deterministically. Endpoint still wraps the call in a daemon
thread so the HTTP response returns immediately even if the staging
walk is slow.
# Backward compat
The scalar `_current_folder` / `_current_status` / `_current_track_*`
fields are preserved as **read-only properties** that resolve to the
FIRST active import. The existing `get_status()` payload still
includes those fields populated from the first entry — single-import
UIs (and the test fixture) keep working unchanged. New
`active_imports` array exposes the full multi-candidate state for
parallel-aware UIs.
# Behavior preserved
- Per-candidate identify / match / process logic byte-identical
- Live-progress state preserved (per candidate now)
- Stability gate / already-processed dedup preserved
- `_record_in_progress` / `_finalize_result` UI rows preserved
- Tag-based loose-file grouping unchanged
# Behavior changes
- Multiple albums process IN PARALLEL up to `max_workers`
- "Scan Now" while scan in progress no-ops (was: spawned another)
- `stop()` waits for in-flight pool work via `shutdown(wait=True)`
- Auto-import card now lists each in-flight album (one line per
active import) instead of a single shared progress line
# UI
`webui/static/stats-automations.js`:
- Progress widget reads `active_imports` array, renders one line
per in-flight album with per-candidate status / track index
- Falls back to the legacy summary line when payload doesn't
carry `active_imports` (older backend)
- Per-row "live processing" lookup now matches by `folder_hash`
through the array instead of by `folder_name` against scalars
# Tests added (`tests/imports/test_auto_import_executor.py`)
- Pool config: default max_workers=3, configurable via constructor
+ via `auto_import.max_workers` config, floors at 1
- Scan lock: 5 concurrent `trigger_scan()` calls run only 1 scan
while lock held; releases properly so subsequent triggers run
- Executor dispatch: 5 candidates → 5 process calls via the pool
- Bounded parallelism: max_workers=3 caps at 3 concurrent;
max_workers=2 caps at 2
- Cross-trigger dedup: candidate submitted in scan A doesn't get
re-submitted by scan B while still in-flight
- Graceful shutdown: `stop()` blocks until in-flight pool work
finishes
- Per-candidate state isolation: 2 parallel workers updating their
own candidate state don't interfere — each candidate's
track_index / track_name / folder_name reads back exactly as
written for that hash
- `get_status()` returns coherent `active_imports` array with
one entry per in-flight candidate; aggregate top-level
`current_status` is 'processing' when any entry is processing
- Unregister removes only that candidate, others stay visible
- Stats counter thread-safety: 1000 parallel bumps land at 1000
(the read-modify-write race regresses without the lock)
- `get_status()` stats snapshot is a copy, not a live reference
# Verification
- 17 new tests pass (executor + state isolation)
- 2347 full suite passes (1 pre-existing flaky test —
`test_watchdog_warns_about_stuck_workers` — passes in isolation,
unrelated)
- Ruff clean
User report: all 6 staging candidates failing with "Could not match
tracks to album tracklist" despite identification correctly resolving
each album. 18 properly-tagged Chris Brown F.A.M.E. tracks, 21
properly-tagged Mr. Morale tracks, etc. — every match attempt
rejected by the duration sanity gate.
Root cause: I had Deezer in `_SECONDS_DURATION_SOURCES`, assuming
Deezer's `duration` field was raw seconds (which the API returns).
But `DeezerClient.get_album_tracks` already converts seconds → ms
INTERNALLY (`'duration_ms': item.get('duration', 0) * 1000`) before
the value reaches the matcher. My helper saw `source='deezer'` →
multiplied by 1000 again → 255000 ms became 255,000,000 ms (70 hours).
Every track-file pair failed the gate by a factor of 1000×.
Diagnostic chain that got me there:
1. Added `[Album Matching] No matches: X files, Y tracks, Z
duration-rejected, W below threshold` summary log so future "0
matches" reports surface the rejection reason.
2. Fixed the helper's logger from `logging.getLogger(__name__)` (which
resolves outside the soulsync handler tree → invisible in app.log)
to `get_logger("imports.album_matching")` (under the namespace the
file handler watches).
3. Added per-rejection-type diagnostic showing actual file vs track
duration values + raw track keys + source.
That third diagnostic surfaced `track 'United In Grief' resolved=255000000
(raw duration_ms=255000, raw duration=None, source='deezer')` —
making the bug obvious.
Fixes:
- Moved Deezer from `_SECONDS_DURATION_SOURCES` to
`_MS_DURATION_SOURCES`. Comment documents WHY (the client converts
before returning) so a future reader doesn't "fix" the
classification back the wrong way.
- Bumped `DURATION_TOLERANCE_MS` from 3000 → 10000 (3s → 10s) to
match Picard ~7s / Beets ~10-15s / Plex ~10s industry baselines.
3s was a defensive copy of the post-download integrity check
threshold but that's a different problem (catching truncated
downloads, not identifying recordings across remasters/encodings).
- `_track_duration_ms` magnitude heuristic kept as fallback for
unknown / missing source (mocked test data without `source` field).
- Added `Match aborted` warnings at the three earlier silent return
points in `_match_tracks` (no client, no album_data, no tracks)
so future "Could not match" reports show WHICH step bailed.
- Added per-run diagnostic in `match_files_to_tracks` that logs the
first duration rejection's actual values — surfaces unit mismatches
+ drift problems without spamming N×M lines per run.
Test changes:
- `test_deezer_seconds_duration_converted_to_ms` renamed +
rewritten as `test_deezer_already_normalised_to_ms_by_client`
to pin the actual contract (matcher receives ms from the Deezer
client, takes as-is).
- `test_track_duration_source_aware_dispatch` updated — Deezer test
case now uses ms input + expects ms output.
- New `test_raw_deezer_seconds_falls_back_to_magnitude_heuristic`
pins the rare edge case where raw Deezer items WITHOUT `source`
reach the matcher (no client conversion path) — heuristic catches
it.
Verification:
- 179 import tests pass after changes
- Live test: all 6 user staging candidates now matching at 95-100%
confidence
- Multi-disc Mr. Morale lands with proper Disc 1 / Disc 2 / Disc 3
folder structure
- Picard-tagged libraries hit MBID fast paths (verified earlier)
- Tracks process in parallel via the existing scan-now thread spawn
(next commit refactors this to a proper bounded executor)
User reported nothing happening on a chaotic staging root despite
6 candidates being detected. Logs showed "Processing folder" for 3
of 6 — the other 3 were silently skipped.
Root cause:
The previous commit (`a9a6168`) introduced loose-file grouping —
multiple `FolderCandidate` objects can now share a `path` (each
album group at the staging root has the same parent directory but
its own audio_files + folder_hash). But two pieces of dedup
machinery still keyed on `path`:
- `_processing_hashes` (was `_processing_paths`) — runtime set of
in-flight candidates. Path-keyed → first sibling marks the path,
second + third siblings hit "already in flight" and skip.
- `_folder_snapshots` — mtime cache for stability check. Path-keyed
→ siblings overwrite each other's mtimes, stability check returns
unreliable results for whichever sibling lost the write race.
Both kept track of an attribute that was previously unique-per-path
(one candidate per directory) but my refactor broke that
invariant without updating the dedup keys. Net effect: only the
first candidate per directory ever got processed in a chaotic-root
scenario.
Fix:
- Renamed `_processing_paths` → `_processing_hashes` set, keyed on
`candidate.folder_hash`. Hash is unique per candidate by
construction (different audio_files lists hash differently).
- `_folder_snapshots` retyped + rekeyed to `folder_hash`. Siblings
no longer overwrite each other's mtime tracking.
- Both touched in lockstep — comments document why path-keyed
dedup breaks for sibling candidates.
Test added (`test_sibling_candidates_have_unique_folder_hashes`):
verifies 3-album loose root produces 3 candidates with distinct
folder_hashes. If a future change breaks the invariant, the test
fails before the silent-skip regression ships.
Verification:
- 178 imports tests pass (8 new this commit + 170 pre-existing
this branch)
- Ruff clean
- Still scoped to import flow
Two related bugs in `AutoImportWorker._scan_directory` surfaced
during real-world testing of the chaotic-staging case (user dropped
loose tracks from multiple albums at staging root, alongside
intact album subfolders):
Bug 1 — Loose files bundled into one fake "album"
When loose audio files existed at a level, the scanner built ONE
FolderCandidate from all of them regardless of their album tags.
On a chaotic staging root with tracks from 3+ different albums,
the identifier picked the most-common album tag and the matcher
left every other album's tracks unmatched (or mis-attributed via
filename + position guessing).
Bug 2 — Subfolders silently ignored when root has loose files
The scanner only recursed into non-disc subfolders when there were
NO loose files at the parent level. So a layout like:
Staging/
loose1.flac (processed via the loose-files path)
Other Album Folder/ (silently ignored — never scanned)
would skip the album subfolders entirely. Common pattern when a
user moves a few tracks out of an album folder while leaving the
rest of the parent album folder intact, OR when other album
folders sit alongside a partially-extracted album.
Fix:
`_build_loose_file_candidates` (new method) reads each loose file's
`album` tag and groups by normalised album name. Each group becomes
its own FolderCandidate so a chaotic staging root produces one
candidate per album — identifier + matcher run cleanly per album.
Untagged loose files become individual single candidates. Disc
folders at the same level attach to whichever loose-file group's
album tag matches the disc-folder tracks; standalone disc folders
(no matching loose group) get their own multi-disc candidate.
The scanner now ALSO always recurses into non-disc subdirectories,
even when the current level has loose files. So album subfolders
sitting beside loose tracks get processed independently in their
own recursive scan.
Behavior preservation:
- Single-album loose-files staging (every file shares one album tag,
no parallel disc folders) → one FolderCandidate, identical to
pre-fix behavior. Pinned by `test_single_album_loose_files_still_one_candidate`.
- Disc-only directory (no loose files, only Disc 1/Disc 2 subdirs)
→ one multi-disc FolderCandidate, identical to pre-fix. Pinned
by `test_disc_only_directory_still_works`.
7 new tests in `tests/imports/test_auto_import_scanner_grouping.py`:
- Multiple-album loose root → multiple candidates
- Untagged loose files → individual singles
- Single-album loose-files regression guard
- Subfolders recursed even when root has loose files
- Disc folder attaches to matching loose group by album tag
- Disc folder with no matching loose group → standalone candidate
- Disc-only directory regression guard
All write real FLACs via mutagen + exercise `_scan_directory`
end-to-end (no mocking the tag reader — proves the production
read path works).
Verification:
- 7 new tests pass
- 2328 full suite passes (+7 new), 1 pre-existing flaky timing test
unrelated to this PR
- Ruff clean
- All changes still scoped to import flow — download flow byte-
identical
Cin-pass on the MBID/ISRC fast-paths + duration-gate work.
Three small but real gaps closed.
Gap 1 — Real-file tag reader integration test
(tests/imports/test_auto_import_tag_reader_real_files.py, 6 tests):
The matcher unit tests use dict fixtures, which prove the algorithm
handles the right shapes once tags are read. They DON'T prove the tag
reader itself extracts the right values from real files. Mutagen's
easy-mode key normalisation (across FLAC / MP3 / M4A) is the exact
spot a future mutagen version could silently drift and break the
fast paths in production while every unit test stays green.
These tests write real FLAC files via mutagen (using the same
`_make_minimal_flac` pattern from `test_album_mbid_consistency.py`)
and assert `_read_file_tags` extracts:
- Picard's `MUSICBRAINZ_TRACKID` (lowercase normalisation in reader)
- `ISRC` (uppercase normalisation in reader; matcher strips
formatting at compare time)
- "track/total" parsing (TRACKNUMBER='5/12' → 5)
- Duration via `audio.info.length` from synthesised STREAMINFO
- Graceful empty-default return for tagless files
- Graceful empty-default return for invalid audio (not a crash)
Acknowledged gap (carried forward): MP3 + M4A integration coverage
not added — mutagen docs say easy-mode normalisation is identical
across all three formats, but only FLAC is pinned here. Followup
candidate.
Gap 2 — Source-aware duration dispatch
(core/imports/album_matching.py, 4 tests in test_album_matching_exact_id.py):
The previous `_track_duration_ms` helper used a magnitude heuristic
("anything below 30000 is seconds, convert × 1000") to decide
whether a track's duration was in seconds or ms. That worked for
typical tracks but had a real edge case: an actual sub-30-second
Spotify track (intros, interludes, skits) would be detected as
seconds and converted to 8.5 hours, breaking the duration sanity
gate.
Replaced with deterministic source-aware dispatch:
- Spotify / iTunes / Qobuz / HiFi / Hydrabase → ms (canonical)
- Deezer / Discogs / MusicBrainz → seconds, × 1000
- Tidal classified as ms (album-tracks endpoint convention; flagged
in code comment as needing real-world verification — defensive
if wrong)
- Magnitude heuristic kept as fallback for unknown / missing source
(mocked test data without source field)
Tests pin all four paths: confirmed-ms source, confirmed-seconds
source, unknown source falls back to heuristic, and the regression
case (sub-30s real track on a known-ms source — must not be
× 1000-converted).
Gap 3 — Cross-disc consolation rationale
(tests/imports/test_album_matching_helper.py, 1 test):
The `CROSS_DISC_POSITION_WEIGHT = 0.05` magic number had no test
proving it was load-bearing. Anyone could have set it to 0 thinking
"strict matching is better" without realising it would silently
break a real scenario.
New test (`test_cross_disc_consolation_is_load_bearing_for_imperfect_titles`)
constructs the exact case the consolation exists for: file has the
right title spelling but the metadata source returns a slightly-
different version (e.g. "Auntie Diaries" file vs "Auntie Diaries
(Remix)" track), AND the file's disc tag is wrong while the track
number agrees. Title sim ~0.78 × 0.45 = ~0.35 (below
MATCH_THRESHOLD 0.4). Without the 5% consolation → file goes
unmatched. With it → ~0.40, just clears.
The test doesn't justify "why 0.05 specifically" — that's still a
tuned knob, not a measured value. But it forces a deliberate
decision if someone wants to drop it: failing this test gives them
the "you broke imperfect-title cross-disc matching" message
explicitly.
Verification:
- 10 new tests across 3 files, all pass
- 35 album-matching tests total now (including pre-existing 17 +
18 fast-path)
- Full suite: 2321 passed, 1 pre-existing flaky timing test
(`test_watchdog_warns_about_stuck_workers` — passes in isolation,
fails only in full-suite runs, unrelated to this PR)
- Ruff clean
- All changes still scoped to import flow — download flow byte-
identical (verified by grep on every changed file)
Brings the auto-import matcher to picard / beets / roon parity by
reaching for the existing AcoustID-grade infrastructure (typed Album
foundation, integrity check thresholds) and layering id-based exact
matches on top of the fuzzy scorer. Picard-tagged libraries now land
every track with full confidence on the first pass.
Three layered phases in `core/imports/album_matching.match_files_to_tracks`:
1. **MBID exact match** — file has `musicbrainz_trackid` tag, source
returns the same id → instant pair, full confidence, no fuzzy
scoring. Picard's primary identifier; per-recording.
2. **ISRC exact match** — file has `isrc` tag, source returns the same
id → same fast-path, slightly lower priority than mbid (isrc can
be shared across remasters). Both ids normalised before compare
(uppercase + strip dashes/spaces for isrc, lowercase for mbid).
3. **Duration sanity gate** — files in the fuzzy phase whose audio
length differs from the candidate track's duration by more than
`DURATION_TOLERANCE_MS` (3s, matching the post-download integrity
check) are rejected before scoring runs. Defends against the
cross-disc / cross-release / wrong-edit problem the integrity
check used to catch only AFTER the file had already been moved +
tagged + db-inserted.
Tag reader (`_read_file_tags`) extended:
- Reads `isrc` (uppercased, strip / / spaces normalisation deferred
to matcher)
- Reads `musicbrainz_trackid` as `mbid` (lowercased)
- Reads `audio.info.length` and converts to `duration_ms` to match
the metadata-source convention
Metadata-source layer (`_build_album_track_entry`) extended:
- Propagates `isrc` from top-level OR `external_ids.isrc` (spotify
shape — would otherwise be stripped before reaching the matcher)
- Propagates `musicbrainz_id` from top-level OR `external_ids.mbid`
/ `external_ids.musicbrainz`
- Without this layer, fast paths would silently never fire in
production even though unit tests pass — pinned by
`test_album_track_entry_propagates_isrc_and_mbid_from_source`
18 new tests in `tests/imports/test_album_matching_exact_id.py`:
- Direct: `find_exact_id_matches` with mbid, isrc, isrc normalisation,
mbid > isrc priority, spotify-shape `external_ids.isrc`, no-id
empty result, file-used-at-most-once
- Direct: `duration_sanity_ok` within / outside tolerance, missing
durations defer
- End-to-end via `match_files_to_tracks`: mbid match short-circuits
fuzzy scoring, id-matched files excluded from fuzzy phase, duration
gate rejects wrong-disc collisions in fuzzy phase, normal matches
pass through the gate, missing durations fall through, deezer
seconds-vs-ms conversion, full picard-tagged 10-track album via
mbid only
- Production-shape: `_build_album_track_entry` propagates isrc + mbid
from spotify-shape (`external_ids.isrc`) AND itunes-shape (top-
level `isrc`)
Verification:
- 35 album-matching tests pass total (17 helper + 18 fast-path)
- 23 multi-disc tests still pass after the extension (additive)
- Full suite: 2311 passed (+18 new), 1 pre-existing flaky timing test
failure (`test_watchdog_warns_about_stuck_workers` — passes in
isolation, fails only in full-suite runs, unrelated to this PR)
- Ruff clean
For users:
- Picard / Beets / Mp3Tag-tagged libraries (anyone who's organised
their music) get instant perfect-confidence matches every time.
- Soulseek-tagged downloads (which usually carry isrc when sourced
via metadata-aware soulseekers) get the fast path too.
- Naively-named files with no useful tags fall through to the
improved fuzzy + duration-gated path — same correctness as before
for the common case, much harder for the matcher to confidently
pair the wrong file.
- One step closer to standalone-DB feature parity with plex /
jellyfin / navidrome scanners. Acoustid fingerprint fallback
(for files with NO useful tags AND no MBID/ISRC) is the next
followup PR.
Cin-pass on the #524 + multi-disc fixes. Pre-merge polish.
Lifts: `core/imports/album_matching.py`
`AutoImportWorker._match_tracks` was a 100+-line method buried in a
1400-line class. Testing it required monkey-patching `_read_file_tags`
+ mocking the metadata client just to exercise the matching algorithm.
Per Cin's "lift logic out of monolithic classes" pattern (same shape
as the album-info builders / discography / quality scanner lifts),
moved the dedup + scoring into `core/imports/album_matching.py` as
pure functions over already-fetched data.
Helper exposes:
- Constants for every match weight (TITLE_WEIGHT, ARTIST_WEIGHT,
POSITION_WEIGHT, NEAR_POSITION_WEIGHT, CROSS_DISC_POSITION_WEIGHT,
ALBUM_WEIGHT, MATCH_THRESHOLD). Magic numbers killed.
- `dedupe_files_by_position(audio_files, file_tags, *, quality_rank)` —
position-keyed quality dedup.
- `score_file_against_track(file_path, file_tags, track, *,
target_album, similarity)` — pure per-(file, track) scorer.
- `match_files_to_tracks(audio_files, file_tags, tracks, *,
target_album, similarity, quality_rank)` — full matching with
greedy best-per-track + first-come-first-serve over deduped files.
Worker shrinks from 100 lines of inline algorithm to 8 lines that
fetch tags + delegate to the helper.
Tests added (26 new across 3 files):
`tests/imports/test_album_matching_helper.py` (19 tests):
- Constants pin: weights sum to 1.0, threshold above position-only
- `dedupe_files_by_position`: quality wins, cross-disc preserved,
tag-less files passed through, first-wins on equal quality
- `score_file_against_track`: perfect-agreement = 1.0, position
needs both disc+track, near-position only same-disc, missing
artist tags handled, disc field aliases (Spotify/Deezer/iTunes),
filename fallback when title tag missing
- `match_files_to_tracks`: happy path, file used at-most-once,
below-threshold left unmatched
- Edge case Cin would flag: tag-less file with strong filename title
matches multi-disc album track via title alone (perfect-name
scenario works); tag-less file with weak filename title against
multi-disc API correctly stays unmatched (the behavior delta from
the disc-aware fix — pinned so future readers see it's intentional)
`tests/test_import_album_match_endpoint.py` (3 tests):
- Backend warning fires when source missing from match POST
- No warning fires on the legit path (catches noisy-warning regression)
- Endpoint actually forwards source/name/artist to the payload
builder (catches "logging the right warning but doing the wrong
lookup" regression)
`tests/test_import_page_album_lookup_pattern.py` (4 tests):
- Source-text guard for the import-page #524 fix in stats-automations.js.
Until the file is modularized enough for a behavioral JS test (under
the existing tests/static/*.mjs pattern), regex-based assertions pin:
the `_albumLookup` field exists, the click handler reads from it,
both card renderers populate it before emitting onclick, and the
cache stores `source` per entry. Caveat documented in the test
module docstring.
Verification:
- All 26 new tests pass.
- Existing multi-disc tests (test_auto_import_multi_disc_matching.py)
still pass after the lift — proves the helper is behavior-equivalent
to the inline implementation it replaced.
- Full suite: 2293 passed, 1 flaky-timing failure
(test_library_reorganize_orchestrator.py::test_watchdog_warns_about_stuck_workers
— passes in isolation, fails only in full-suite runs, pre-existing,
unrelated to this PR).
- Ruff clean.
Notes for the reviewer:
- The frontend stats-automations.js JS test is structural-only.
Behavioral JS testing for that file requires modularizing the
~7k-line monolith first — out of scope for this fix.
- The cross-disc 5% consolation bonus is a small behavior change for
users with weak/missing tag info on multi-disc albums. Pinned
explicitly in `test_tagless_file_with_weak_title_unmatched_in_multidisc`
so the trade-off is visible: correct multi-disc matching wins over
optimistic position-only matching that produced wrong-disc files.
Caught while live-testing the #524 fix with kendrick lamar
mr morale & the big steppers (3 discs). User dropped discs 1+2
loose in staging root + disc 3 in its own folder, every file
perfectly tagged with disc_number/track_number/title — only 9
tracks ended up in the library, the rest got integrity-rejected
and quarantined.
Two related bugs in `AutoImportWorker._match_tracks`:
1. **Quality dedup keyed on track_number alone.** The dedup loop
kept `seen_track_nums[track_number] = file` and dropped any later
file with the same number, treating it as a quality duplicate.
On a multi-disc release where every disc has tracks 1..N, that
collapses the album to one disc's worth of files BEFORE the
matcher runs. User's 18 loose disc-1+disc-2 files reduced to 9
before any title/disc info was even consulted.
2. **Match scoring ignored disc_number.** The 30% track-number bonus
fired whenever `ft[track_number] == track_num` regardless of disc.
File with tag (disc=2, track=6, "Auntie Diaries", 281s) got the
full bonus matching API track (disc=1, track=6, "Rich Interlude",
103s) — wrong file → wrong destination → integrity check correctly
rejected and quarantined the file. Same for tracks 7, 8, 9.
Fix:
- Dedup keys on `(disc_number, track_number)` tuples — multi-disc
files with parallel numbering all survive.
- Match scoring's 30% bonus only when BOTH disc AND track agree.
Cross-disc same-track-number collisions get a small 5% consolation
bonus so title similarity has to carry the match (covers cases
where tag disc info is missing or wrong).
- API track disc_number read from `disc_number` (Spotify) /
`disk_number` (Deezer) / `discNumber` (iTunes) defaulting to 1.
4 new pinning tests in `tests/imports/test_auto_import_multi_disc_matching.py`:
- 18-file 2-disc regression case (dedup preserves all)
- (disc=2, track=6) file matches API (disc=2, track=6) track, not
the disc-1 same-numbered track
- Single-disc albums still match normally (no regression)
- Quality dedup within a single (disc, track) position still picks
higher-quality format (.flac over .mp3)
Verification:
- 2268 full pytest suite passes (+4 new), 1 skipped, 0 failed
- Ruff clean
Same branch as the #524 fix because both surfaced from the same
import session — easier reviewer context if they ship together.
Ruff F811 — `sys` is already imported at module top (line 13). The
local `import sys` inside `_auto_download_disabled` shadowed it
needlessly. Caught by CI ruff check on the dev-nightly workflow.
Previous commit split _check_ffmpeg into a side-effect-free
_locate_ffmpeg + the original auto-download _check_ffmpeg, and moved
__init__ to call _locate_ffmpeg. That alone wasn't enough — caught
the gap during a deeper audit:
is_configured() → is_available() → _check_ffmpeg() (with download)
The orchestrator registry, download engine, and the orchestrator's
own configured_clients() all probe is_configured() polymorphically at
boot. So when tests import web_server, the registry probes
youtube.is_configured() → is_available() → _check_ffmpeg() →
DOWNLOAD. My __init__ change didn't help because the registry boot
fires the same code path right after.
Real fix: gate the download branch inside _check_ffmpeg itself.
Returns False (and logs a warning) when running under pytest or when
SOULSYNC_NO_FFMPEG_DOWNLOAD=1. End users on a fresh install still get
auto-download on first real YouTube use (gate is off in production).
Container is unaffected (system ffmpeg via apt is found on PATH, the
download branch never runs).
Three detection paths in _auto_download_disabled():
- SOULSYNC_NO_FFMPEG_DOWNLOAD env var (explicit opt-out for CI /
build steps that want to disable outside pytest)
- PYTEST_CURRENT_TEST env var (set by pytest per-test — covers
in-test-body call path)
- 'pytest' in sys.modules (covers calls fired during pytest collection
/ import phase, BEFORE the per-test env var is set — which is
exactly when registry.py probes is_configured() at web_server
import time)
Verified by inspecting tools/ after a full suite run — empty (was
~388 MB after a single test_tidal_auth_instructions.py run before
the gate). Container behavior unchanged: shutil.which('ffmpeg')
returns /usr/bin/ffmpeg from the apt-installed package, so the
download branch is never reached anyway.
5 new pinning tests:
- pytest-in-sys.modules detection works
- PYTEST_CURRENT_TEST env detection works
- SOULSYNC_NO_FFMPEG_DOWNLOAD env detection works
- _check_ffmpeg returns False (no urlretrieve, no tools/ dir created)
when gate is on and ffmpeg is missing — pinned by trapping
urlretrieve to AssertionError so a regression blows up loud
- _locate_ffmpeg never triggers download or creates tools/ —
pinned by trapping both urlretrieve AND Path.mkdir on tools-prefixed
paths
2264 passed (+5), 1 skipped, 0 failed.
kettui reported the dev image roughly doubled in size after a recent
nightly build. codex investigation traced it back to:
1. nightly workflow runs `python -m pytest` before docker build
2. one of the new tests imports web_server (test_tidal_auth_instructions.py)
3. importing web_server constructs YouTubeClient
4. YouTubeClient.__init__ called _check_ffmpeg() — which auto-downloads
a ~388 MB ffmpeg/ffprobe bundle into ./tools/ when system ffmpeg
isn't on PATH (CI runner doesn't have it)
5. .dockerignore didn't exclude tools/ffmpeg or tools/ffprobe
6. docker `COPY . .` shipped the binaries
7. the immediately-following `chown -R /app` rewrote every file into
a new layer — so the 388 MB payload got counted twice in image
size
three fixes:
1. .dockerignore — block the auto-downloaded binaries even if they
leak into the workspace (tools/ffmpeg, tools/ffprobe, .exe variants,
.zip and .tar.xz download archives). Defense-in-depth so a future
regression in the test/import path can't bloat the image again.
2. youtube_client — split _check_ffmpeg into a side-effect-free
_locate_ffmpeg (pure existence check) and the original auto-
download _check_ffmpeg. __init__ now calls _locate_ffmpeg + logs
a warning when missing instead of triggering download. is_available()
and the actual download dispatch paths still call _check_ffmpeg —
so end users still get auto-download on first YouTube use, but
`import web_server` doesn't drag a 388 MB binary into the workspace.
3. Dockerfile — replaced `COPY . .` + `chown -R /app` with
`COPY --chown=soulsync:soulsync . .` + a scoped chown on just the
runtime mount-point dirs. eliminates the layer that duplicated
the entire /app tree just to flip ownership bits, so even legit
workspace content isn't double-counted in the image.
Combined effect: image size returns to baseline + future ffmpeg leaks
can't bloat it. Inside the container nothing changes — the Dockerfile
already installs system ffmpeg via apt, so YouTube downloads find it
on PATH on first use and the auto-download path never fires.
2259 passed, 1 skipped, 0 failed.
Three follow-on fixes to the manual-search candidates modal once people
started actually using it:
1. NDJSON streaming. Manual search waited for every source to return
before showing anything. Now streams one event per source as each
completes — header line, source_results per source, done terminator.
Frontend appends rows incrementally via response.body.getReader().
2. Manual picks no longer auto-retry on failure. New _user_manual_pick
flag set on the task in /download-candidate. Both monitor retry
paths (not-in-live-transfers stuck + Errored state) bail on the
flag. Surfaces the failure to the user instead of silently picking
a different candidate via fresh search.
3. Non-Soulseek manual picks (youtube/tidal/qobuz/hifi/deezer/
soundcloud/lidarr) no longer stuck at "downloading 0%" forever. The
live_transfers IF branch now marks manual-pick tasks failed
directly when the engine reports Errored, instead of deferring to
the monitor (which bails on manual picks). Engine fallback in else
branch covers the rare race where the orchestrator's pre-populated
transfer lookup is missing the entry.
Plus a deadlock fix discovered along the way: the new failure path
synchronously called on_download_completed while holding tasks_lock,
which itself re-acquires the same Lock — non-reentrant
threading.Lock self-deadlocked the polling thread. While wedged, every
other endpoint that needed the lock (including /candidates → other
failed rows couldn't open modals) hung waiting. Moved completion
callbacks onto a daemon thread so the lock releases first.
Plus failed/not_found/cancelled rows are now ALWAYS clickable (not
just when the auto-search cached candidates) — the modal carries the
manual search bar, which is the user's recourse for empty results.
Plus manual download worker now runs on a dedicated thread instead of
competing with the batch's 3-worker missing_download_executor pool —
saturated batches no longer queue manual picks indefinitely.
All scoped to manual picks via the _user_manual_pick flag — auto
attempt flow byte-identical to before. Engine fallback gated on the
flag too so auto attempts in the else branch keep the original
do-nothing behavior (safety valve handles the stuck-forever case).
Also dropped _handle_failed_download from web_server.py — defined
but had no callers (dead code).
17 new unit tests pin the gate behavior:
- engine fallback: Errored/Cancelled/Succeeded/InProgress transitions,
manual-pick gate, terminal-state skip, soulseek skip, missing
download_id skip, engine returning None, orchestrator exception
- monitor: manual-pick skips not-in-live-transfers retry + Errored
retry
- IF-branch end-to-end: Errored marks failed, "Completed, Errored"
hits failure branch, auto attempts defer to monitor
Manual-search endpoint tests rewritten for NDJSON: 11 cases (validation,
single-source dispatch, parallel "all" dispatch, one-event-per-source
streaming shape, unconfigured-source skip + reject, header metadata,
per-source exception isolation).
Full suite 2259 passed, 1 skipped.
Four selection-quality fixes on the SoulSync-made discover playlists.
None change public method signatures; all are tightenings on what's
already there.
(1) Diversity for Hidden Gems + Discovery Shuffle
Both used to be `RANDOM() LIMIT N` with no diversity. Could return
50 tracks from one artist or 20 from one album if the discovery
pool happened to be skewed. Both now over-fetch 3x and run the
existing `_apply_diversity_filter`:
- Hidden Gems: max 2 per album, 3 per artist
- Discovery Shuffle: max 2 per album, 2 per artist (tighter — shuffle
should feel maximally varied)
(2) Source-aware popularity thresholds
`popularity >= 60` for "Popular Picks" and `popularity < 40` for
"Hidden Gems" was Spotify-shaped (0-100 scale). Deezer writes its
`rank` value into that column (often six-digit integers); iTunes
writes nothing meaningful. For Deezer-primary users:
- Popular Picks pulled essentially everything (rank >= 60 = all)
- Hidden Gems pulled essentially nothing (rank < 40 = none)
New `_get_popularity_thresholds(source)` helper returns per-source
values:
- Spotify: (60, 40) — the existing 0-100 scale
- Deezer: (500_000, 100_000) — ballpark from real rank values
- iTunes / unknown: (None, None) — skip the popularity filter
entirely, fall back to random + diversity
`get_popular_picks` and `get_hidden_gems` now consult the helper.
When threshold is None they skip the popularity SQL filter. Diversity
+ ID gate still apply.
(3) Push genre keyword filter into SQL
`get_genre_playlist` used to fetch `limit=1_000_000` rows into Python
then run a substring keyword filter on `artist_genres`. Bad on big
discovery pools.
Now the keyword OR chain is generated as SQL placeholders:
AND (artist_genres LIKE ? OR artist_genres LIKE ? OR ...)
Each placeholder gets `f'%{keyword.lower()}%'` via `extra_params`.
`fetch_limit` drops back to `limit * 10`. `_genre_matches` Python
helper deleted (only intra-file caller; verified via grep).
Parent-genre expansion via `GENRE_MAPPING` preserved — keywords list
feeds the LIKE chain unchanged.
(4) Filter out tracks already in library
Discovery pool can include tracks the user already owns. Hidden Gems
/ Shuffle / Popular Picks shouldn't surface those.
`_select_discovery_tracks` gained `exclude_owned: bool = True`
parameter. When True, adds a correlated NOT EXISTS subquery against
the `tracks` table covering all 3 source IDs:
AND NOT EXISTS (
SELECT 1 FROM tracks t WHERE
(t.spotify_track_id IS NOT NULL AND t.spotify_track_id = discovery_pool.spotify_track_id)
OR (t.itunes_track_id IS NOT NULL AND t.itunes_track_id = discovery_pool.itunes_track_id)
OR (t.deezer_id IS NOT NULL AND t.deezer_id = discovery_pool.deezer_track_id)
)
Note column-name asymmetry: tracks.deezer_id vs
discovery_pool.deezer_track_id. Inline comment marks the trap. All
5 public discovery methods automatically benefit (default True).
Seasonal Playlist doesn't go through the helper so it's unaffected
(curated content, dedup is wrong intent there).
Tests
12 new tests in `tests/test_personalized_playlists_id_gate.py` (27
total in the file):
- Hidden Gems + Discovery Shuffle apply diversity (cap proven by
inserting 10 same-artist + same-album rows and asserting return
count ≤ per-album cap)
- Popularity thresholds: Spotify (60, 40), Deezer larger scale,
iTunes None / None
- Popular Picks skips threshold filter when None
- Genre playlist pushes filter to SQL (parent + child genre expansion)
- Owned-track exclusion: filtered when match, kept when no match,
opt-out flag works
- Deezer column-name asymmetry pinned (regression footgun)
Test fixture re-added the minimal `tracks` table (4 columns: id,
spotify_track_id, itunes_track_id, deezer_id) — only what the new
NOT EXISTS subquery needs to join. Plus `insert_library_track`
helper.
Verification
- 27/27 in this test file pass (15 prior + 12 new)
- 2232/2232 full suite green
- ruff clean
LOC delta:
- core/personalized_playlists.py: 1030 → 1101 (+71)
- tests/test_personalized_playlists_id_gate.py: 352 → 616 (+264)
The original gate baked into `_select_discovery_tracks` only checked
Spotify + iTunes:
AND (spotify_track_id IS NOT NULL OR itunes_track_id IS NOT NULL)
For Deezer-primary users, discovery_pool rows have populated
`deezer_track_id` but NULL Spotify + NULL iTunes IDs. The gate
filtered every row out — Time Machine, Genre Browser, Hidden Gems,
Discovery Shuffle, Popular Picks all rendered "no tracks found" for
every tab on every Deezer-primary install.
Extended the gate to include `deezer_track_id` and added that column
to the standard SELECT column tuple. `_build_track_dict` already
exposed `deezer_track_id` in its output shape, so frontend rendering
needed no changes.
Regression pinned via new test
`test_discovery_helper_accepts_deezer_only_id_rows` — inserts a row
with NULL Spotify + NULL iTunes but a populated `deezer_track_id`
and asserts it survives the gate.
2220/2220 full suite green.
Owner decision: not worth shipping. The four library-driven personalized
sections were stubbed returning [] for ages because their schema
prereqs didn't exist; the prior commit re-enabled them by routing
through a new `_select_library_tracks` helper. Owner reviewed and chose
to delete the sections entirely instead.
Removed everywhere:
- `core/personalized_playlists.py` — `get_recently_added`,
`get_top_tracks`, `get_forgotten_favorites`, `get_familiar_favorites`
+ the `_select_library_tracks` helper (no other callers; verified
via grep).
- `web_server.py` — 4 route handlers
(`/api/discover/personalized/recently-added`, `top-tracks`,
`forgotten-favorites`, `familiar-favorites`).
- `webui/index.html` — 4 `<div class="discover-section">` blocks
(`#personalized-recently-added`, `#personalized-top-tracks`,
`#personalized-forgotten-favorites`,
`#personalized-familiar-favorites`).
- `webui/static/discover.js` — 4 load functions
(`loadPersonalizedRecentlyAdded`, `loadPersonalizedTopTracks`,
`loadPersonalizedForgottenFavorites`, `loadFamiliarFavorites`),
plus their entries in `loadDiscoverPage`'s Promise.all, plus
4 module-level state vars + 6 dead branches across
`openDownloadModalForDiscoverPlaylist` / `startDiscoverPlaylistSync`
and the sync-progress / rehydrate dispatchers.
- `webui/static/helper.js` — 4 tooltip / docs entries.
- `webui/static/sync-spotify.js` — 1 stale rehydrate dispatcher
branch (`discover_familiar_favorites`) caught during the global
grep pass.
- `tests/test_personalized_playlists_id_gate.py` — 3 library-method
tests + the test infrastructure that supported them
(`tracks` schema, `insert_library_track` helper). Documentation
header updated to reflect the deletion.
Net: -527 / +2 lines across 7 files.
What stays:
- Daily Mixes (also in personalized package, intentionally paused —
separate decision).
- Popular Picks + Hidden Gems + Discovery Shuffle (alive, not
affected by this deletion).
- All 14 tests in the personalized-playlists test file still pass.
- The PersonalizedPlaylistsService lift from the prior commit
(`_select_discovery_tracks` etc) — those are still in active use
by the surviving discovery_pool methods.
DISCOVER_TRACK_SELECTION_REVIEW.md at repo root contains historical
references to the four deleted endpoints. Treated as historical
context (same policy as WHATS_NEW), left alone.
2219/2219 full suite green (was 2222 - 3 deleted tests = 2219).
JS parses clean, ruff clean.
User-facing bug found in the discover-page audit: multiple sections
(hidden gems, discovery shuffle, popular picks, decade browser,
genre browser) had no `WHERE (spotify_track_id IS NOT NULL OR
itunes_track_id IS NOT NULL ...)` gate. Tracks with no source IDs
in the discovery pool got displayed, the user clicked download, the
download silently failed because there was nothing to look up.
Lift + gate
`PersonalizedPlaylistsService` had 5 selection methods that all shared
the same shape — connect to DB, run a SELECT against `discovery_pool`
with different WHERE clauses, optionally apply diversity, return
list of track dicts. ~366 lines of business logic, ~55% of which was
repeated boilerplate.
Three new private helpers consolidate everything:
- `_select_discovery_tracks(*, source, extra_where, extra_params,
order_by, fetch_limit, extra_columns)` — shared SELECT against
`discovery_pool`. The mandatory ID gate is hard-coded into the
WHERE clause: no opt-out flag, every method inherits it for free.
Plus the source filter and the blacklist filter — same shape every
selector needs.
- `_apply_diversity_filter(tracks, *, max_per_album, max_per_artist,
limit)` — per-album / per-artist cap loop, returns trimmed list.
Lifted from the inline duplicates in decade / genre / popular_picks.
- `_compute_adaptive_diversity_limits(tracks, *, relaxed=False)` —
step-function tiers based on unique-artist count. `relaxed=True`
gives the slightly looser limits the genre playlist used vs the
decade playlist.
Re-enable 4 library methods
`get_recently_added`, `get_top_tracks`, `get_forgotten_favorites`,
`get_familiar_favorites` were all stubs (`return []`) because they
predated the schema columns they need. Schema now has them:
`tracks.created_at`, `tracks.play_count`, `tracks.last_played`, and
the source ID columns added in earlier work.
New `_select_library_tracks(*, where_clause, params, order_by, limit)`
helper mirrors the discovery selector but targets the `tracks` table
joined against `albums` + `artists`. Mandatory ID gate lives in the
helper too: every library method automatically rejects rows where
spotify_track_id, itunes_track_id, deezer_id,
musicbrainz_recording_id, AND audiodb_id are all NULL.
Selection rules:
- `get_recently_added` — ORDER BY created_at DESC
- `get_top_tracks` — WHERE play_count > 0 ORDER BY play_count DESC
- `get_forgotten_favorites` — WHERE play_count > 5 AND last_played
< (now - 90 days) ORDER BY play_count DESC
- `get_familiar_favorites` — WHERE play_count BETWEEN 3 AND 15
Tests
`tests/test_personalized_playlists_id_gate.py` — 17 tests pinning:
- `_select_discovery_tracks` filters NULL-id rows, honors source +
blacklist + extra_where
- `_apply_diversity_filter` caps per-album + per-artist + stops at
limit
- `_compute_adaptive_diversity_limits` returns the right tier for
unique-artist count + relaxed flag
- All 5 discovery methods (decade, popular_picks, hidden_gems,
discovery_shuffle, genre is exercised via the helper) reject
NULL-id rows
- All 4 library methods reject NULL-id rows + honor their
play-count rules
Behavior preserved
Same diversity tiers, same over-fetch multipliers (10x for decade /
genre, 3x for popular_picks), same `popularity DESC, RANDOM()`
ordering, same `popularity >= 60` / `< 40` thresholds, same
blacklist filter. Public method signatures unchanged — `web_server.py`
needs zero edits.
Net file: 1089 → ~1170 LOC (helpers + docstrings), but actual
business logic across the 9 methods went from ~418 lines down to
~195 (-53%).
2222/2222 full suite green (was 2205 + 17 new). Ruff clean.
Closes#513 (s66jones).
The artist detail page already showed a "Popular on Last.fm" sidebar —
list of an artist's top tracks by playcount, with a play button per row
but no download action. Issue #513 wanted a way to grab those tracks
the same way zotify let users grab "top X songs" without pulling the
full discography.
Pulls from the configured primary metadata source (Spotify
`artist_top_tracks`, Deezer `/artist/{id}/top`) when available, falls
back to the existing Last.fm display-only mode for sources that don't
expose popularity ranking (iTunes / Discogs / MusicBrainz). Source
label in the section title shifts to match.
Each row gets a hover-revealed download button that wishlists the
single track via the existing /api/add-album-to-wishlist endpoint
(preserves the track's real album metadata, so the wishlist worker
later places the file in its proper album folder).
A "Download All" footer button opens the standard download modal in
PLAYLIST context, not album context — the virtual playlist_id is
`top_tracks_<source>_<artistId>` which doesn't match any of the
album-prefix checks in `startMissingTracksProcess` (downloads.js).
That keeps `is_album_download=false`, so the master worker doesn't
inject a wrapper context as `_explicit_album_context`. Each track
downloads using its own real album metadata, files land in proper
per-album folders on disk (not a fake "Top Tracks" folder).
Backend additions:
- `SpotifyClient.get_artist_top_tracks(artist_id, country, limit)` —
wraps `spotipy.artist_top_tracks`, returns up to 10 tracks for the
market (Spotify's API cap). UI-side limit trim only.
- `DeezerClient.get_artist_top_tracks(artist_id, limit)` — wraps
`/artist/{id}/top?limit=N`, converts Deezer's raw shape to the same
Spotify-compatible dict layout (id, name, artists, album with
album_type / total_tracks / images, duration_ms, track_number,
disc_number) so downstream code doesn't branch on source.
- `GET /api/artist/<id>/top-tracks` — dispatches to whichever client
matches the primary source. Resolves per-source artist IDs from the
DB row first (matching what /discography already does) so a Spotify
ID in the URL still works when Deezer is primary, and vice versa.
Returns `{success, source, tracks, resolved_artist_id}` on hit;
`{success: False, reason: 'unsupported_source' | 'spotify_not_authenticated'
| 'deezer_unavailable' | 'no_tracks_found'}` on miss so the frontend
can decide whether to fall through to Last.fm.
Frontend:
- `_loadArtistTopTracks` tries the metadata source first, falls
through to the legacy `/api/artist/0/lastfm-top-tracks` call if the
source can't deliver. Section title and per-row UI shift based on
which source answered.
- New per-row `.hero-top-track-download` button (hover-revealed).
- New `.hero-top-tracks-download-all` footer button — only visible
when metadata-source mode rendered the list (Last.fm fallback hides
it since rows have no track IDs to download).
Tests: 10 new tests pin the client methods —
- Spotify: returns track list, honors UI limit cap, returns empty when
unauthed / artist_id missing / API throws.
- Deezer: shape conversion to Spotify-compatible dict, empty when no
data / artist_id missing, limit clamping at upper bound, default
fallback when limit=0, malformed entries skipped.
The Flask endpoint dispatcher itself isn't covered by the new test
file because importing web_server at test-collection time spins up
worker threads that race with caplog-using tests elsewhere in the
suite (specifically test_library_reorganize_orchestrator). Endpoint
verified manually; the underlying client methods (the load-bearing
logic) are covered.
2204/2204 full suite green (was 2194 + 10 new).
Discord report (corruption [BWC]): downloads coming through as the
instrumental cut when a vocal track was requested. The verification
step's `_normalize` function strips parentheticals and version-suffix
tags ("(Instrumental)", "- Live", etc) so legitimate name variations
don't false-fail the title-similarity check. That also means "In My
Feelings" and "In My Feelings (Instrumental)" both normalize to "in
my feelings", title similarity is 1.0, and the wrong cut passes
verification.
Detect the version label on each side BEFORE normalization runs. If
the expected and matched recordings disagree on version (one is
original, the other is instrumental / live / acoustic / remix /
etc), return FAIL — the fingerprint identified a real song, just
not the version the caller asked for.
Reuses `MusicMatchingEngine.detect_version_type` so the same regex
patterns the pre-download Soulseek matcher applies also drive
post-download verification. No duplicated tables.
Also gates the secondary fallback scan, so a wrong-version variant
sitting in the same fingerprint cluster can't win the loop after
the best match has already been version-rejected.
6 tests pin the behavior:
- instrumental returned for vocal request → FAIL
- vocal returned for instrumental request → FAIL
- live vs acoustic → FAIL
- matching versions on both sides → PASS
- original-to-original happy path → PASS (regression guard)
- secondary scan skips wrong-version recordings → not PASS
2194/2194 full suite green (was 2188 + 6 new).
Catches the silent excepts the awk-based earlier sweeps missed:
- Bare `except:` followed by `pass` (also swallows KeyboardInterrupt
and SystemExit — actively wrong). Upgraded to `except Exception as
e: logger.debug("...: %s", e)`. ~14 sites across connection_detect,
soulseek_client, listenbrainz_manager, watchlist_scanner,
youtube_client, navidrome_client, jellyfin_client, web_server.
- `except Exception:` + pass that the awk pattern missed (e.g.
multi-line or unusual whitespace). ~31 sites across automation_engine,
database_update_worker, music_database, spotify_client, web_server,
others.
- 14 legitimate cleanup sites left silent with explicit `# noqa: S110`
+ comment explaining why (atexit handlers, finally-block conn.close
calls). Logging during shutdown can itself crash because file handles
get torn down before the handler fires.
Also enables `S110` rule in pyproject.toml so this pattern fails CI
going forward — drift fails at PR review instead of at runtime against
a wedged worker thread. Tests path keeps S110 ignored (test fixtures
legitimately use try-except-pass for cleanup).
Adds a WHATS_NEW entry to helper.js summarizing the full #369 sweep.
Verified: `python -m ruff check .` → All checks passed.
Verified: `python -m pytest tests/` → 2188 passed.
Closes#369
Across all background workers (Spotify/Tidal/Deezer/Qobuz/iTunes/
Discogs/Genius/AudioDB/MusicBrainz/Last.fm/SoulID + the metadata-update
worker) and the repair-job scanners. All converted to
`logger.debug("...: %s", e)`.
Two `_e` renames in genius_worker and soulid_worker where outer scope
was already binding `e`. Two finally-block sites in repair_jobs/
library_reorganize.py left silent (conn.close on shutdown path).
Refs #369
These files had silent `except Exception: pass` blocks but no module
logger. Added `import logging` + `logger = logging.getLogger(__name__)`
at the top of each, then replaced the silent excepts with
`logger.debug(...)`.
- core/replaygain.py — 4 sites (id3 txxx + vorbis + mp4 atom reads)
- core/wishlist/presence.py — 3 sites (wishlist row parsing + queries)
- core/runtime_state.py — 1 site (activity toast emit)
- core/automation/signals.py — 1 site (collect known signals)
- core/download_engine/rate_limit.py — 1 site (plugin rate_limit_policy)
- api/system.py — 1 site (hydrabase status probe)
- api/search.py — 1 site (hydrabase search)
Refs #369
Mostly progress-callback try/except (caller-provided fns we don't
control) and best-effort file-move / DB-cleanup paths in the
auto-fixers. All converted to `logger.debug(...)`. No control-flow
changes.
Refs #369
Discord report: Duplicate Detector card said "372 findings" and Cover
Art Filler said "60 findings", but clicking the Findings tab's Pending
filter showed 0. User read it as "findings aren't being created" —
looked like a detector bug.
Actual cause: the badge sourced ``last_run.findings_created``
(historical "found in last scan") without considering current state.
After the user (or bulk-fix automation) resolved or dismissed those
findings, they no longer appeared on the Pending tab — but the badge
kept showing the last-scan number in red urgent styling.
Backend was correct end-to-end: detectors create pending rows,
bulk-fix moves them to resolved, Findings tab filters by status.
Only the badge display lied about current state.
Fix:
- ``RepairWorker._get_pending_count_by_job()`` — single SQL aggregation
returning ``{job_id: pending_count}`` for every job with pending
findings. O(1) lookup per job instead of N round trips.
- ``get_all_job_info()`` calls it once per request and adds
``pending_findings_count`` to each job's API response.
- ``enrichment.js`` job card now branches on the count:
- ``> 0`` → red ``"X pending"`` badge (urgent, action needed)
- ``= 0`` AND last scan found something → muted grey ``"X found in
last scan"`` (historical context, no action needed)
- New CSS class ``.repair-flow-badge.findings-historical`` for the
muted slate color so the two states are visually distinct.
User-visible result with the screenshotted state (372 dup / 60 cover-
art findings, all resolved):
- Before: red "372 findings" / "60 findings" — implied 432 things to
do, but Findings tab showed 0 pending
- After: grey "372 found in last scan" / "60 found in last scan" —
the badge text tells the user the count is historical, no surprise
when Pending is empty
Tests: 3 new tests in ``tests/test_create_finding_dedup_counter.py``
pin the per-job pending count helper:
- returns ``{job_id: count}`` based on status='pending' rows only;
resolved + dismissed rows excluded
- empty dict when no pending findings exist
- gracefully returns ``{}`` on DB error (badge falls back to
historical count via the existing JS ``or 0`` safety)
2188/2188 full suite green. Pure UI/state-display fix — no detector
logic, no backend behavior change.
GitHub issue #499 (@bafoed). Big initial sync of Spotify playlists
worked for 2-3 hours then downloads silently stopped:
- 3 active tasks stuck in "Searching" state, replaced every ~10 min
by different ones
- slskd UI showed no actual searches happening
- Debug log: orphaned-task count grew over time, no jobs executed
- Container restart was the only fix (bought another 2-3 hours)
- Not a rate limit (rates showed 0/min)
Root cause: ``core/soulseek_client.py`` constructed
``aiohttp.ClientSession()`` with no timeout at four sites. When slskd
hung on a request (overloaded, transient network blip, internal
stall), the HTTP call blocked indefinitely — and the worker thread
blocked with it. The download executor only has
``ThreadPoolExecutor(max_workers=3)``, so once 3 worker threads were
wedged on hung calls, no further downloads could start.
Batch-level "stuck detection" (10-minute timer in
``check_batch_completion_v2``) was correctly marking tasks
``not_found`` and trying to start replacements, but the executor pool
was exhausted — replacements queued forever inside the executor with
no thread to run them. Symptom: tasks rotating every ~10 min at the
batch level while the underlying executor stayed wedged.
Fix: bounded ``aiohttp.ClientTimeout`` (total 120s, connect 15s,
sock_read 60s) on every slskd ``ClientSession`` construction. Module-
level constant ``_SLSKD_DEFAULT_TIMEOUT`` so the four sites stay in
lockstep — future sites get the same protection by reusing the
constant.
Why these timeouts are safe:
- Every slskd API call is metadata-level (search submission, status
polls, download enqueue, transfer state queries). None stream
files — slskd handles file transfer via its own peer-to-peer
infrastructure entirely outside our HTTP requests.
- Legitimate metadata calls finish in seconds. 120s ceiling is
~50× the normal latency.
Timeout handling:
- ``asyncio.TimeoutError`` caught explicitly BEFORE the generic
``except Exception`` — surfaces "slskd timed out" specifically in
logs (debuggable instead of buried as "Error making API request").
- Returns None to the caller (same code path as a 5xx response or
any other failure). No new error path; callers already handle
None as "request failed".
- Worker thread unblocks immediately → executor pool stays healthy
→ downloads keep flowing.
Sites updated:
- ``_make_request`` (general /api/v0/ helper, line 152) — used for
every slskd API operation
- ``_make_direct_request`` (non-/api/v0/ helper, line 235)
- ``_explore_api_endpoints`` Swagger fetch (line 1566) — diagnostic
- ``_explore_api_endpoints`` per-endpoint probe (line 1617) —
diagnostic
Tests: 3 new tests in ``tests/downloads/test_soulseek_pinning.py``
pin:
- ``_SLSKD_DEFAULT_TIMEOUT`` is bounded (total set, ≤300s ceiling,
connect ≤60s) — guards against future regressions that drop or
unbound the timeout
- ``_make_request`` returns None on ``asyncio.TimeoutError`` rather
than raising — pins the caller contract
- ``_make_direct_request`` returns None on ``asyncio.TimeoutError``
2185/2185 full suite green.
Closes#499.
GitHub issue #500 (@bafoed). Library Reorganize repair job moved
album tracks to single-template paths because of a fragile
classification heuristic. Concrete symptom: a track at
``Surf Curse/Surf Curse - Nothing Yet (2017)/01 - Christine F.flac``
got proposed for a move to
``Surf Curse/Surf Curse - Christine F/Surf Curse - Christine F.flac``
(single template) instead of staying under the album folder.
Root cause: the job had its own tag-reading + transfer-folder-walk +
template-application implementation. The classification was
``is_album = (group_size > 1)`` where ``group_size`` was the count
of same-album tracks currently sitting in the transfer folder being
scanned. Two failure modes:
- only one track of an album was in the transfer folder (rest already
moved to the library, or not yet downloaded), or
- album tags varied slightly across tracks (e.g. ``"Buds"`` vs
``"Buds (Bonus)"``)
Either case gave a 1-element group → routed through the SINGLE
template → wrong destination.
Rewrite — delegate to the per-album planner the artist-detail
"Reorganize" modal already uses:
- ``core.library_reorganize.preview_album_reorganize`` for path
computation (DB-driven, knows the album has N tracks regardless of
how many sit in transfer; album-vs-single is structurally correct)
- ``core.reorganize_queue.enqueue_many`` for apply mode; the queue
worker dispatches via ``reorganize_album`` which handles file move
+ post-processing + DB update + sidecar through the same code path
the per-album modal uses
Job's per-album loop:
- iterate albums for the active media server only (matches the artist-
detail modal's scope; multi-server users won't have the job touch
the inactive server's files at paths they can't see)
- preview each album, catch exceptions per-album so one bad row
doesn't abort the scan
- branch on planner status:
- ``no_album`` / ``no_tracks`` (race: album deleted mid-scan) →
skip silently
- ``no_source_id`` (album never enriched) → emit ONE album-level
"needs enrichment first" finding (vs N per-track findings cluttering
the UI)
- ``planned`` → filter mismatched tracks (matched + new_path +
not unchanged + file_exists), emit per-track findings (dry-run)
or collect album for bulk enqueue (apply)
- bulk enqueue at end of loop using the queue's correct return-shape
(``{'enqueued': N, 'already_queued': M, 'total': K}``)
What's gone (~500 LOC):
- ``_read_tag_metadata`` / ``_get_audio_quality`` / transfer-folder walk
- ``_load_album_years`` / ``_lookup_years_from_api`` (planner does this)
- ``_apply_path_template`` / ``_build_path_from_template``
- direct ``shutil.move`` + sidecar move logic (queue handles)
- the fragile ``is_album = group_size > 1`` heuristic — structurally gone
- ``move_sidecars`` setting (no longer applicable; queue's post-process
re-downloads cover art at the destination)
What stays:
- dry-run vs apply toggle
- ``file_organization.enabled`` gate
- stop / pause respect
- progress reporting
- findings for the UI
Cleaner separation of concerns:
- this job: DB-known tracks at wrong paths (active server only)
- ``orphan_file_detector``: files on disk with no DB entry
- ``dead_file_cleaner``: DB entries pointing to nonexistent files
Tests: 12 tests in ``tests/test_library_reorganize.py`` pin the
delegation contract — every status branch, every track-filter case,
exception handling, apply-mode enqueue payload, active-server scope,
estimate-scope shape. Three obsolete ``_lookup_years_*`` tests removed
(year handling moved to planner).
Closes#500 (the misclassification half — orphan + dead-file are
downstream sync-gap symptoms, separate concern).
GitHub issue #501 (@Tacobell444). After manually matching an album to
a specific source ID via the match-chip UI, clicking "Enrich" on that
album would fuzzy-search by name and overwrite the manual match with
whatever the search returned — or revert the match status to
``not_found`` if name search missed. Reorganize then read the now-
wrong ID and moved files to the wrong destination.
Root cause was in the per-source enrichment workers'
``_process_*_individual`` methods. Several workers (Spotify, iTunes)
ran search-by-name unconditionally with no check for an existing
stored ID. Others (Deezer, Tidal, Qobuz) skipped on existing-ID but
without refreshing metadata — preserved the ID but didn't actually
honor the user's intent of "use this match to pull fresh data".
Cin-shape lift: same fix needed in 5 workers, so extracted the shared
behavior into ``core/enrichment/manual_match_honoring.py``:
honor_stored_match(
db, entity_table, entity_id, id_column,
client_fetch_fn, on_match_fn, log_prefix,
) -> bool
Per-worker variability (DB column name, client fetch method, response
shape) plugs in via callbacks. Workers call the helper at the top of
``_process_album_individual`` / ``_process_track_individual``; if it
returns True, the manual match was honored and the search-by-name
fallback is skipped. If False (no stored ID, fetch failed, or empty
response), the worker's existing search-by-name flow runs as before.
Workers wired:
- spotify_worker — album + track (was overwriting; now honors)
- itunes_worker — album + track (was overwriting; now honors)
- deezer_worker — album + track (was skip-on-id; now refreshes)
- tidal_worker — album + track (was skip-on-id; now refreshes)
- qobuz_worker — album + track (was skip-on-id; now refreshes)
Workers left alone (already correct):
- discogs_worker — already had inline stored-ID fast path that
refreshes metadata. Same behavior, just inline; refactoring to use
the shared helper would be churn for zero behavior change.
- audiodb_worker — same — inline fast path with full metadata refresh.
- musicbrainz_worker — preserves existing MBID and marks status,
which is the correct behavior for MB (the MBID itself is the match
payload — no separate metadata fetch).
- lastfm_worker / genius_worker — name-based services with no
source-specific IDs to honor. Inherent re-search per call.
Reorganize fixed indirectly — it always honored stored IDs correctly
via ``library_reorganize._extract_source_ids``. The "Reorganize broken"
symptom was downstream of broken Enrich corrupting the stored ID.
Tests:
- ``tests/enrichment/test_manual_match_honoring.py`` — 11 tests
pinning the shared helper contract: stored-ID fast path, no-ID
fallthrough, empty-string treated as no ID, missing row, fetch
exception caught and falls through, fetch returns None falls
through, callback exceptions propagate, configurable table +
column, defensive table-name whitelist.
- Per-worker wiring NOT tested individually — the workers depend
on live DB / client objects that are heavy to mock. The shared
helper's contract is pinned; per-worker call sites are short
enough to verify by code review.
2173/2173 full suite green.
Closes#501.
Followup to the all-libraries-mode commit. Without dedup, a Plex Home
family where two users both have "Drake" in their music libraries
would see "Drake" twice in SoulSync's library list — Plex returns
distinct ratingKeys for each section's copy of the same artist.
Dedup design — applied selectively, NOT everywhere:
- ``_dedupe_artists(artists)``: groups by lowercased title, picks
the canonical entry by ``leafCount`` (more tracks wins). Active
ONLY in all-libraries mode; single-library mode is a no-op fast
path with zero behavior change.
- ``_dedupe_albums(albums)``: same but keys on
(lowercased parentTitle, lowercased title) so two artists with
identically-titled albums (e.g. self-titled releases) stay
separate.
Applied to:
- ``get_all_artists()`` — public listing for the library view
- ``get_library_stats()`` — count matches what user sees in the list
Deliberately NOT applied to:
- ``get_all_artist_ids()`` / ``get_all_album_ids()`` — these feed
removal detection (compare returned ratingKey set against DB-linked
ratingKeys to decide what's been removed). Deduping here would falsely
flag non-canonical ratingKeys as "removed" and prune SoulSync's DB
tracks that are linked to them. Pinned by two CRITICAL tests.
- ``_all_tracks()`` — track count stays raw because the same track
in two sections IS two distinct files / Plex entries, not a logical
duplicate.
- ``_search_general()`` and ``search_tracks`` Stage 1/2 — search
results stay raw so cross-section matches aren't lost. Stage 1
may miss cross-section tracks for the same artist but Stage 2's
server-wide track search catches them.
Logging: when raw vs deduped artist counts differ, ``get_all_artists``
logs both so users can see "Found 4697 artists across all music
sections (4521 unique after cross-section dedup)" — surfaces the
overlap clearly.
Tests: 8 new tests in test_plex_all_libraries.py pin:
- canonical pick by leafCount (artists + albums)
- case-insensitive name match
- single-library no-op path (zero behavior change for those users)
- album dedup keys on (artist, title) so different-artist same-title
albums stay separate
- ``get_all_artists`` listing applies dedup
- ``get_all_artist_ids`` does NOT dedup (CRITICAL — removal detection)
- ``get_all_album_ids`` does NOT dedup (CRITICAL — removal detection)
- ``get_library_stats`` uses deduped counts for artists/albums but
raw count for tracks
Existing pre-stat test updated to use distinct mock instances —
``[MagicMock()] * 5`` creates five references to one mock which now
correctly collapses under dedup.
71/71 media_server tests green, 2162/2162 full suite green.
Honest known limitation acknowledged in WHATS_NEW + version modal:
write-back (genre / poster / metadata updates) targets one
ratingKey at a time — only updates the canonical section's copy of
an artist if it exists in multiple. Other section's copy stays
unchanged. Document and revisit if it matters.
GitHub issue #505 (PopeBruhLXIX): users with multiple Plex music
libraries (e.g. one per Plex Home user, or two folder roots split
across separate library sections) only saw one library inside SoulSync
because the connection settings forced you to pick a single library
section. SoulSync's PlexClient stored exactly one ``self.music_library``
section reference and every read scanned only that one.
This change adds an opt-in "All Libraries (combined)" dropdown option
that flips the client into a server-wide read mode where every read
method (``get_all_artists`` / ``get_all_album_ids`` /
``search_tracks`` / ``get_library_stats`` / etc) dispatches through
``server.library.search(libtype=...)`` instead of querying a single
section. One Plex API call replaces N per-section iterations; Plex
handles the aggregation server-side.
Implementation:
- ``ALL_LIBRARIES_SENTINEL`` (``'__all_libraries__'``) — module-level
constant used as the saved DB preference value when the user picks
the synthetic "All Libraries" entry. Detection is one string compare
in ``_find_music_library`` / ``set_music_library_by_name``. Existing
preferences (real library names) are unaffected.
- ``self._all_libraries_mode`` (private flag) + ``is_all_libraries_mode()``
(public accessor for external callers). When True, ``music_library``
may stay None — ``is_fully_configured()`` recognizes the mode and
still returns True so dispatch sites don't bail.
- New private helpers ``_can_query``, ``_get_music_sections``,
``_all_artists``, ``_all_albums``, ``_all_tracks``, ``_search_general``,
``_search_artists_by_name``. Single dispatch point for the
section-vs-server branch — every read method funnels through them
so future drift fails at one place.
- New public helpers for downstream callers:
- ``get_recently_added_albums(maxresults, libtype)`` — used by
DatabaseUpdateWorker's deep-scan recent-content sweep
- ``get_recently_updated_albums(limit)`` — same
- ``get_music_library_locations()`` — returns folder roots, used
by web_server.py's file-path resolver
- ``trigger_library_scan`` and ``is_library_scanning`` fan out across
every music section in all-libraries mode.
- ``get_available_music_libraries`` prepends a synthetic
``{'title': 'All Libraries (combined)', 'value': sentinel}`` entry
ONLY when more than one music library exists. Single-library users
don't get the extra option. ``value`` field is the canonical
identifier the frontend submits to ``/api/plex/select-music-library``
(real libraries: title; synthetic: sentinel string). Backward-
compatible — entries without ``value`` fall back to ``title``.
Three crash points fixed in downstream consumers (would have failed
during a deep scan after the user picked all-libraries mode):
1. ``database_update_worker.py:411`` — bailed out with "No music
library found in Plex" because ``not self.media_client.music_library``
evaluated True in all-libraries mode (music_library is None there).
Now uses ``is_fully_configured()`` which recognizes the mode.
This was the root cause of the deep scan never starting.
2. ``database_update_worker.py:_get_recent_albums_plex`` — reached
``self.media_client.music_library.recentlyAdded()`` /
``.search()`` directly, AttributeError in all-libraries mode.
Now routes through the new helper methods.
3. ``web_server.py:10947`` (file-path resolver) — accessed
``music_library.locations``; gated on ``music_library`` truthy so
it didn't crash, but silently skipped all-libraries-mode locations.
Now uses ``get_music_library_locations()`` which unions across
sections.
Plus polish:
- ``/api/plex/clear-library`` also resets ``_all_libraries_mode``
so a fresh "select library" flow doesn't inherit stale mode state.
- ``/api/plex/music-libraries`` surfaces "All Libraries (combined)"
as ``current_library`` when in mode (settings UI displays correctly).
- Frontend ``loadPlexMusicLibraries`` uses ``library.value || library.title``
so the sentinel-keyed option submits the sentinel string, not the
human-readable label. Pre-select match handles both paths.
Honest tradeoffs (documented as known limitations):
- Same artist appearing in multiple Plex sections shows as separate
entries in SoulSync (no dedup). Plex returns distinct ratingKeys
for each. Cosmetic; revisit if it bites users.
- Write-back (genre / poster updates) targets one ratingKey at a time
— only updates that section's copy. Other sections' copies stay
unchanged.
- All-libraries mode includes any audiobook library that Plex
classifies as ``type='artist'``. Edge case, opt-in only.
Tests: 21 new tests in tests/media_server/test_plex_all_libraries.py
pin both single-library mode (regression guard) and all-libraries mode
for every refactored method. Existing test_plex_pinning.py fixture
updated to initialize the new flag. 63/63 media_server tests green,
2148/2148 full suite green.
Two fixes.
(1) Discography endpoint now does server-side per-source ID resolution.
When the user clicked Download Discography on a library artist, the
endpoint received whichever artist ID the frontend happened to pick
(spotify_artist_id || itunes_artist_id || deezer_id || library_db_id)
and dispatched it as-is to whichever source it queried. If the picked
ID didn't match the queried source's ID format, the lookup returned
wrong-artist results (numeric ID collisions) or fell back to a fuzzy
name search that picked a wrong artist.
Two reproducible cases:
- 50 Cent's library row had DB id 194687 — coincidentally a real
Deezer artist ID for "Young Hot Rod". When the frontend's
/enhanced fetch silently fell back to the DB id, the backend
sent 194687 to Deezer, and Deezer returned Young Hot Rod's
50 albums in 50 Cent's discography modal.
- Weird Al's library row had a stored Spotify ID. The frontend
sent that to Deezer, which rejected the alphanumeric ID and
fell back to fuzzy name search — which picked The Beatles
somehow, returning 45 Beatles albums.
The mechanism for per-source ID dispatch already exists in
``MetadataLookupOptions.artist_source_ids``, and the watchlist scanner
already uses it; the on-demand discography endpoint just wasn't wired
to it. Fix: when the URL artist_id matches a library row by ANY stored
ID (DB id, spotify_artist_id, itunes_artist_id, deezer_id, or
musicbrainz_id), pull every stored provider ID and pass them as
``artist_source_ids``. Each source gets its OWN stored ID regardless
of which one the URL carries. When the URL ID is a non-library
source-native ID and the row lookup misses entirely, behavior is
identical to before (single-ID dispatch fallback).
Logged the resolved per-source ID dict at INFO so future "wrong artist
showed up" diagnostics are immediately legible in app.log.
(2) Logger namespace fix in core/artists/quality.py and
core/metadata/multi_source_search.py.
Both modules used ``logging.getLogger(__name__)`` which resolves to
``core.artists.quality`` / ``core.metadata.multi_source_search`` —
neither under the ``soulsync`` namespace where the file handler is
wired. Result: every [Enhance], [MultiSourceSearch], and direct-lookup
INFO line was being written to a logger with no handlers and silently
dropped. App log showed the slow-request warning but no diagnostic
detail. Switched both to ``get_logger()`` from utils.logging_config so
the soulsync.* namespace picks them up. Same content, now actually
lands in app.log. Confirmed working in live test:
``[Enhance] Direct lookup matched: deezer ID 1476162252 → 'Desastre'``
No behavior change in any other caller. Empty ``artist_source_ids``
(no library row matched) reaches lookup as ``None`` → identical to
current single-ID dispatch path. Logger fix is pure routing — no
content change.
Followup on the previous Enhance refactor. Multi-source parallel text
search closed the worst case (users with no Spotify/Deezer getting
"unknown artist - unknown album - unknown track" wishlist entries),
but text search itself is still fragile against messy library tags:
"Title (Live)", featured artists in the artist field, etc. Download
Discography never had this problem because it resolves albums by stable
ID, not by name.
Enhance now does the same thing for tracks: for every metadata source
the user has configured, if the library track has the corresponding
stored ID (spotify_track_id / deezer_id / itunes_track_id / soul_id),
call client.get_track_details(stored_id) directly and convert to the
wishlist payload. First success wins. The user's configured primary
source is tried first so a Deezer-primary user gets Deezer payloads on
the wishlist entry (correct cover art / album shape) even when other
sources also have stored IDs for the same track.
Multi-source parallel text search stays as the fallback for tracks
with no stored IDs (e.g. manually imported, never enriched). Empty-
field rejection still gates the wishlist add.
Implementation:
- _STORED_ID_COLUMNS: source name → DB column mapping
(Discogs intentionally omitted — release-based, no per-track IDs)
- _enhanced_to_wishlist_payload: converts the get_track_details
intermediate "enhanced" shape (artists as [str]) to wishlist shape
(artists as [{'name': str}]). Spotify's raw_data is already in
wishlist shape, returned as-is when detected (preserves full
album.images that the enhanced top-level fields drop)
- _try_direct_lookup_all_sources: iterates sources preferred-first,
calls get_track_details on each that has both a stored ID and a
configured client, returns first complete-metadata payload
- spotify_client field removed from ArtistQualityDeps (no longer
used — Spotify direct lookup now flows through the generic
per-source loop using the entry from search_sources)
- _try_upgrade_to_rich_payload removed (was Spotify-only with broken
shape semantics for non-Spotify sources; search-fallback now uses
_build_payload_from_track consistently)
- get_primary_source() consulted to set the per-call preferred source
for direct-lookup priority
Also fixed a stale UI string: the Enhance modal toast read "Matching
tracks to Spotify and adding to wishlist..." regardless of which
sources were actually configured. Now reads "Matching tracks across
metadata sources...".
Tests:
- _build_deps mirrors web_server._resolve_search_sources: passing
spotify=spotify_obj auto-prepends ('spotify', spotify_obj) to
search_sources (Spotify is always added when configured in prod)
- 5 new tests pin the direct-lookup behavior:
- test_direct_lookup_via_deezer_id_skips_text_search
- test_direct_lookup_via_itunes_id_skips_text_search
- test_direct_lookup_prefers_user_primary_source
- test_direct_lookup_falls_through_to_text_search_when_no_stored_ids
- test_direct_lookup_failure_falls_through_to_text_search
- Reframed enhanced-format and search-fallback tests for the new
payload-build path (no album-image side call, search-fallback uses
_build_payload_from_track consistently)
- 22/22 quality tests green, 2133/2133 full suite green.
Track Redownload had been doing parallel multi-source metadata search
across every configured source the whole time; Enhance Quality was
running a single-source primary fallback that returned junk matches
with empty fields when the primary was iTunes (Discord report:
"unknown artist - unknown album - unknown track" wishlist entries
for users with neither Spotify nor Deezer connected).
Lift the redownload search into core/metadata/multi_source_search.py
and point both flows at it. Same scoring, same per-source query
optimization (Deezer's structured artist:/track: form), same
current-match flagging via stored source IDs.
ArtistQualityDeps now takes get_metadata_search_sources (returns
[(name, client), ...] for every configured source) instead of the
single-primary get_metadata_fallback_client + get_metadata_fallback_source.
Spotify direct-lookup stays as a fast-path optimization (only Spotify
exposes get_track_details(id) returning rich raw payload); when it
doesn't fire, the multi-source parallel search picks the cross-source
best match. Empty-field matches still rejected before wishlist add.
Tests: _build_deps helper updated to accept the new search_sources
contract while preserving fallback_client/fallback_source ergonomics.
Reframed tests for the new semantics — direct-lookup is no longer
gated on Spotify being the active primary; failure reason now lists
every searched source. Added a test pinning the no-sources-configured
prompt. 17/17 quality tests green, 2128/2128 full suite green.
Discord report: clicking Enhance Quality on an artist with neither
Spotify nor Deezer connected added tracks to the wishlist as
"unknown artist - unknown album - unknown track".
Root cause was structural. core/artists/quality.py had a hardcoded
Spotify-direct → Spotify-search → iTunes-fallback chain that ignored
the user's configured primary metadata source. When Spotify wasn't
connected, every track fell through to an iTunes-only fallback that
occasionally returned matches with empty fields (cleared the 0.7
confidence threshold but missing artist / album / title). Those
empty strings propagated through the wishlist payload normalizer's
truthy-check passthrough at core/wishlist/payloads.py:77-80 and the
UI rendered them as "Unknown" defaults.
Rewrote the flow source-agnostic:
- ArtistQualityDeps gains get_metadata_fallback_source. Flow resolves
the user's active primary source once up front.
- New _build_payload_from_track helper produces the Spotify-shaped
wishlist payload from any source's Track object — single place
that knows how to construct it (replaces the duplicate construction
in the Spotify-search and iTunes-fallback paths).
- New _search_match helper does generic confidence-scored search
against any client implementing search_tracks(query, limit). Same
0.7 threshold, same album-bonus weighting as before.
- New _has_complete_metadata validator rejects matches with empty
title / album / artists before they reach the wishlist.
- _spotify_direct_lookup kept as a Spotify-only optimization (only
Spotify exposes get_track_details(id) returning rich raw payload);
other sources fall through to search.
- Failure reason now names the active source: "No usable {source}
match — connect another metadata source for better coverage".
Result: Discogs users get a Discogs search. Hydrabase users get a
Hydrabase search. iTunes users get an iTunes search with empty-field
rejection. Spotify keeps its direct-lookup fast path.
6 new tests pin the architectural change:
- Primary-source dispatch routes to the configured client (Discogs,
not Spotify) when Spotify isn't primary
- Spotify direct-lookup is gated on Spotify being the active primary
(skipped when Discogs is configured even if track has spotify_track_id)
- Empty title / album / artists fields all reject the match
- Failure reason names the active source
Going line-by-line through the engine package + boot wiring. Five
small things worth fixing before Cin reads it:
(1) MediaServerEngine class docstring still claimed to be a "single
entry point for cross-server library operations" — but the prior
honesty pass cut all the cross-server dispatch wrappers because they
had no callers. Class is really lookup + small accessors now.
Docstring rewritten to match.
(2) configured_clients() had a dead `not hasattr(client, 'is_connected')`
branch. is_connected is in REQUIRED_METHODS so every client the
registry yields here implements it. Branch removed; comment notes
the reasoning.
(3) types.py imported `datetime` and `Dict` but used neither —
dead imports dropped.
(4) types.py docstring claimed "all four servers" defined an
XTrackInfo dataclass. Actually only Plex / Jellyfin / Navidrome
did; SoulSync uses richer per-track wrappers. Fixed.
(5) web_server.py boot:
- media_server_engine added to the chained `= None` declaration
so it's always defined before the try/except, defending against
the rare path where engine init AND fallback both raise.
- Outer engine init failure logger now uses exc_info=True for full
traceback (boot-time issues are rare but worth diagnosing).
- Nested fallback failure now logs explicitly instead of silently
leaving media_server_engine as None.
Tests: 2121 still pass.
Pre-review audit found premature abstraction + lying docstrings.
Cut what isn't used, made the rest match what's actually shipped.
(1) Engine: dropped 7 cross-server dispatch wrappers that had ZERO
production callers (ensure_connection / get_all_artists /
get_all_album_ids / search_tracks / trigger_library_scan /
is_library_scanning / get_library_stats / get_recently_added_albums).
Every consumer reaches the active client directly via
sync_service._get_active_media_client() or engine.client(name).
Engine surface shrinks to client(name) / active_client() /
active_server / is_connected() (the one wrapper that has callers —
4 dashboard status sites) / configured_clients() / reload_config().
~150 lines deleted, 5 dead-method tests removed.
(2) Contract Protocol body trimmed to match REQUIRED_METHODS exactly
(is_connected, ensure_connection, get_all_artists, get_all_album_ids).
The other 5 methods that were declared in the Protocol
"required" section weren't actually required — Plex doesn't
implement get_recently_added_albums, Jellyfin doesn't implement
search_tracks, SoulSync doesn't implement most of them. Static
contract now matches runtime conformance test. Optional methods
moved to a KNOWN_PER_SERVER_METHODS data-only listing with audited
per-server coverage notes — discoverability without false promises.
(3) Engine module docstring + __init__.py docstring no longer
overclaim "33+ chains collapsed" — only 4 uniform-shape chains
were collapsed; ~18 server-specific chains stay explicit per the
"lift what's truly shared" standard. Phrasing now matches reality.
(4) types.py docstring claimed TrackInfo.from_jellyfin_dict and
TrackInfo.from_navidrome_dict exist as classmethods. They don't —
only from_plex_track / from_plex_playlist do. Jellyfin and Navidrome
construct TrackInfo inline at their call sites today. Docstring
now honest about that + flags the lift as a clean followup.
(5) Engine line 95 comment "backward-compat for source-specific
reaches" was misleading — there is no legacy alternative being
preserved; engine.client(name) IS the canonical access pattern.
Section header rewritten.
Tests: 2121 pass (was 2126; -5 dead-method pin tests).
Three nits I missed in the prior pass:
(1) Two more exception swallows still at logger.debug — the
get_recently_added_albums wrapper and the configured_clients
inner loop. My earlier sed only matched single-line patterns.
Both now log at warning level so a broken Plex / Jellyfin
surfaces in the boot log instead of silently returning [].
(2) registry.py module docstring still claimed it replaced
"33 hand-maintained dispatch sites" — same overclaim I fixed
on the engine docstring. Server-specific chains stay explicit
in web_server.py per the "lift what's truly shared" standard;
the registry just owns name → client lookup. Docstring rewritten
to match reality.
Five tightening passes anticipating Cin / JohnBaumb's review nits:
(1) Engine no longer reaches into ``registry._instances`` private
attr. New public ``MediaServerRegistry.set_instance(name, client)``
method — engine constructor calls it for the ``clients=`` pre-built
case so internal storage stays encapsulated.
(2) Engine module docstring no longer overclaims. Originally said it
"Replaces the historic 33+ if/elif chains" — but only the four
uniform-shape ``is_connected`` chains were collapsed. The 19 chains
that do server-specific work (Plex raw API vs Jellyfin / Navidrome
client methods returning different shapes) stay explicit per the
"lift what's truly shared" standard. Docstring rewritten to say
exactly that.
(3) Per-method exception swallows upgraded from ``logger.debug`` to
``logger.warning``. Returning safe defaults stays the right behavior
for a read-side engine (Plex offline shouldn't crash the app), but
silent debug-level swallowing made debugging hard — JohnBaumb pushed
the download engine to surface real errors. Same treatment here:
default still safe, but the warning tells you Plex is down.
(4) ``_safe_init_media_client`` in web_server.py now logs the
exception type + traceback. Broad ``except Exception`` is still
intentional (any failure means that one server can't be used; the
others stay up) but the boot log now distinguishes config errors
(ConnectionError, AuthenticationError) from import / dependency
failures.
(5) Two new tests pin the encapsulation + fallback contracts:
- ``test_engine_with_empty_clients_dict_is_safe_to_use`` — empty
engine returns safe defaults on every method, doesn't raise.
- ``test_engine_uses_registry_set_instance_not_private_attr`` — spy
on registry.set_instance verifies engine uses the public method.
Plex / Jellyfin / Navidrome each defined a near-identical
XTrackInfo (id / title / artist / album / duration / track_number /
year / rating) and XPlaylistInfo (id / title / description /
duration / leaf_count / tracks). Three classes that grew up by
copy-paste — not a real contract difference.
Lifted both to core/media_server/types.py as canonical TrackInfo +
PlaylistInfo. Per-server constructors live as classmethods on the
unified class (TrackInfo.from_plex_track, PlaylistInfo.from_plex_playlist)
matching the metadata Album.from_X_dict pattern Cin's POC uses.
Heavy plexapi imports stay lazy under TYPE_CHECKING.
- core/plex_client.py / jellyfin_client.py / navidrome_client.py:
per-server XTrackInfo / XPlaylistInfo dataclass definitions
removed; each module now imports TrackInfo + PlaylistInfo from
the neutral package and uses the shared name internally.
- core/matching_engine.py: was annotating callers with PlexTrackInfo
even though sync_service hands it Jellyfin / Navidrome instances
at runtime when those servers are active. Annotation is now the
unified TrackInfo, so signatures match the actual contract.
- services/sync_service.py: same import + annotation update.
Broque Thomas