Adds the user's Tidal favorited tracks ("My Collection" in the Tidal
app) as a virtual playlist alongside their real playlists, mirroring
how Spotify's "Liked Songs" is treated.
Reporter (yug1900) located the working endpoint after the prior
`/v2/favorites?filter[type]=TRACKS` attempt returned empty data —
that endpoint is scoped to collections the third-party app created
itself, not personal favorites. Real endpoint:
GET /v2/userCollectionTracks/me/relationships/items
?countryCode=US&locale=en-US&include=items
Cursor-paginated (20 per page, follow `links.next` with
`page[cursor]=...` until exhausted). Response only carries
track-level attributes — artist + album NAMES come back as
relationship-link stubs, not embedded data.
Implementation:
* Two-phase fetch — `_iter_collection_track_ids` walks the cursor
chain to enumerate every track id (cheap, IDs only), then
`get_collection_tracks` batch-hydrates 20 IDs at a time through
the existing `_get_tracks_batch` helper which already knows how
to `include=artists,albums`. No duplication of the JSON:API
artist/album parse, no new dataclass shape.
* Virtual playlist `tidal-favorites` appended to the end of
`/api/tidal/playlists`. ID intentionally has no colon —
sync-services.js renderer interpolates IDs into CSS selectors
via template literals (`#tidal-card-${p.id} .foo`) and a `:`
would parse as a CSS pseudo-class operator.
* `tidal_client.get_playlist("tidal-favorites")` recognizes the
virtual id and dispatches to the collection path internally, so
every per-id consumer gets it for free: detail endpoint, mirror
auto-refresh automation, "build Spotify discovery from Tidal
playlist" flow.
OAuth scope expansion:
* Added `collection.read` to both OAuth flows (the
`core/tidal_client.py::authenticate` standalone path AND the
`web_server.py::auth_tidal` web flow — they were independent
scope strings that both needed updating).
* Added `prompt=consent` to both flows — without it Tidal silently
returns a token carrying only the ORIGINAL scope set even after
re-authentication, because Tidal treats the existing
authorization as still valid.
* New `disconnect()` method + `POST /api/tidal/disconnect`
endpoint + Disconnect button next to Authenticate in Settings →
Connections → Tidal — required for users whose existing token
predates the scope expansion (forces a clean grant).
Reconnect-needed UI hint:
* `_collection_needs_reconnect` flag set on 401/403 from the
collection endpoint, cleared on next successful walk, NOT set
on 5xx (transient server errors must not falsely tell the user
to reconnect).
* Listing endpoint reads the flag and surfaces a placeholder card
titled "Favorite Tracks (reconnect Tidal to enable)" with a
description pointing at Settings, so the user has something
visible to act on instead of a silently missing row.
Diagnostic logging — collection request URL + response status +
first 300 bytes of body now logged at info level so future "why
is my collection empty" reports can be diagnosed from app.log
without needing live reproduction.
22 new tests pin: cursor walk (full chain, max-ids cap mid-page +
at page boundary), auth gates (no token / 401 / 403 all bail
clean), reconnect-flag lifecycle (set on 401/403, cleared on next
successful walk, NOT set on 5xx), forward-compat type filter
(non-track entries skipped), count helper, batch hydration
delegation + chunking at the 20-per-batch cap, partial-batch
failure containment, virtual-id dispatch (real playlist ids still
flow through the normal path).
Closes#502.
Phase B of foxxify discord report. Pre-#524 manual-import bug left
some albums in the library with `artist=Unknown Artist` and `album.title
= <numeric album_id>`. Reorganize couldn't place them (no usable
metadata source ID) and emitted a generic "run enrichment first" hint
that doesn't apply — enrichment can't fix these rows. The right tool
is the existing `Fix Unknown Artists` repair job (reads file tags,
re-resolves metadata, re-tags + moves files).
Discoverability gap, not a logic gap. Reorganize now detects the bad-
metadata shape (Unknown Artist OR album.title that's a 6+ digit
numeric id) and emits a clear "run the Fix Unknown Artists repair
job" hint at both reason-emit sites (planner + executor). No
duplication of fixer logic.
WHATS_NEW entry covers both Phase A (orphan-format sibling handling,
already committed in d944a16) and Phase B since they ship in the same
PR for the same reporter.
20 new tests pin helpers + reason routing.
Discord report (Skowl): downloaded a compilation album ("High Tea
Music: Vol 1") where every track has a different artist (Eclypse,
Andromedik, T & Sugah, Gourski, etc.) and the AcoustID scanner
flagged every single track as Wrong Song. The file tags had the
correct per-track artist (e.g. "Eclypse" for "City Lights"), but
the scanner compared against the album-level artist ("Andromedik",
the curator). Raw similarity 12% → Wrong Song flag.
# Why the prior multi-value fix didn't help
Foxxify's case (just-merged PR): AcoustID returned multi-value
credit "Okayracer, aldrch & poptropicaslutz!" — primary IS in the
credit. Splitting found it.
Skowl's case: both sides single-value but DIFFERENT artists.
Splitter has nothing to find — Eclypse simply isn't in "Andromedik".
Different bug.
# Cause
Scanner SQL at `core/repair_jobs/acoustid_scanner.py:281` joined
the `artists` table via `tracks.artist_id` which points at the
ALBUM artist (the curator/label-name applied to every row in a
compilation). The `tracks.track_artist` column already holds the
correct per-track artist for compilations — populated by every
server-scan path (Plex `originalTitle`, Jellyfin `ArtistItems`,
Navidrome per-track `artist`) AND the auto-import / direct-download
post-process flow (`record_soulsync_library_entry` writes it when
different from album artist). Scanner just wasn't reading it.
# Fix
```sql
SELECT t.id, t.title,
COALESCE(NULLIF(t.track_artist, ''), ar.name) AS artist,
...
```
Prefers per-track artist when populated, falls back to album artist
for legacy rows / single-artist albums where `track_artist` is NULL.
`NULLIF(t.track_artist, '')` handles the empty-string-instead-of-null
case some legacy rows might have.
# Composes with Foxxify's multi-value fix
For the rare compilation track where AcoustID ALSO returns a
multi-value credit (e.g. compilation track has multiple credited
performers), both paths work together — `track_artist` gives the
correct expected primary, then the helper splits the credit and
finds it.
# Tests added (2)
- `test_load_db_tracks_prefers_track_artist_for_compilation` —
reporter's exact case: track with `track_artist='Eclypse'` AND
`artist_id` pointing at album artist 'Andromedik' resolves to
'Eclypse'. Second track with NULL `track_artist` falls back to
album artist 'Andromedik' (single-artist + legacy compat).
- `test_load_db_tracks_falls_back_when_track_artist_empty_string`
— empty string in `track_artist` (some legacy rows) → NULLIF
returns NULL → COALESCE falls back to album artist.
Both use a real SQLite DB so the COALESCE/NULLIF logic + JOIN
runs against actual schema (SimpleNamespace fakes can't simulate
JOINs).
# Verification
- 6/6 scanner tests pass (2 new + 4 existing)
- 2586 full suite passes (+2 from prior commit)
- Ruff clean
Discord report (Foxxify): the AcoustID scanner repair job flagged
multi-artist tracks as Wrong Song because AcoustID returns the
FULL credit ("Okayracer, aldrch & poptropicaslutz!") while the
library DB carries only the primary artist ("Okayracer"). Raw
SequenceMatcher similarity scored ~43% — well below the 60%
threshold — so the scanner created a finding even though the
audio was correct. User couldn't fix without lowering the global
artist threshold to ~30% (which would let real mismatches through).
# Fix
Extended the shared `core/matching/artist_aliases.py::artist_names_match`
helper (originally lifted for #441) with credit-token splitting.
When the actual artist string contains common separators —
- punctuation: `,` `&` `;` `/` `+`
- keywords (whitespace-bounded): `feat.` `ft.` `featuring` `with`
`vs.` `x`
— the helper splits into individual contributors and checks each
against the expected artist. Primary-in-credit cases now resolve
at 100% instead of 43%.
Two pattern groups because punctuation separators don't need
surrounding whitespace, but keyword separators MUST be
whitespace-bounded — otherwise we'd split artists with `x` /
`with` etc. in their names ("JAY-X" → "JAY-" / "" issue).
Composes with the existing alias path: cross-script multi-artist
credits ("Hiroyuki Sawano" expected, "澤野弘之, FeaturedJp"
actual) work via alias-token-against-credit-token compare.
# Wire-in
Scanner at `core/repair_jobs/acoustid_scanner.py:202` replaces
the raw `SequenceMatcher` call with `artist_names_match`. Pass
RAW artist strings (not pre-normalised by `_normalize`) so the
splitter can recognise separators — `_normalize` strips ALL
punctuation, which destroyed the very tokens the splitter needs.
The AcoustID post-download verifier (`core/acoustid_verification.py`)
already routes through `_alias_aware_artist_sim` which calls the
same helper — gets the multi-value benefit automatically without
a separate wire-in.
# New `split_artist_credit` exported helper
Pure-function helper for callers who want token-level access to
the credit list (debugging, UI, future per-token enrichment). Same
splitter logic, exposed as a top-level function.
# Tests added (14)
`tests/matching/test_artist_aliases.py` (+11):
- `TestSplitArtistCredit` — parametrised across 12 credit-string
formats (comma, ampersand, semicolon, slash, plus, feat./ft./
featuring, with, vs., x, single-token, empty), drops empty
tokens, strips per-token whitespace
- `TestMultiValueCreditMatching` — reporter's exact case
(Okayracer in 3-artist credit → 100%), primary in middle/end of
credit, genuine-mismatch still fails, single-token actual falls
through to direct compare, multi-value composes with aliases,
threshold still respected
`tests/test_acoustid_scanner.py` (+3):
- Reporter's case end-to-end through `_scan_file` — fingerprint
99% / title 100% / multi-artist credit → no finding created
- Genuine artist mismatch still creates finding (no false
suppression of real mismatches)
- `JobResultStub` minimal scaffold for the integration tests
# Verification
- 14 new tests pass (49 helper + 5 scanner total in their files)
- 110 matching + scanner tests pass total
- 2584 full suite passes (+25 from baseline 2559)
- Ruff clean
- Reporter's exact case (Okayracer in `Okayracer, aldrch &
poptropicaslutz!`) now scores 100% match → no Wrong Song flag
Discord report (Tim): downloaded cover art via Deezer metadata
source came out visibly blurry in Navidrome / on phones — large
displays exposed the limited resolution.
# Cause
Deezer's API returns `cover_xl` URLs at 1000×1000. The underlying
CDN actually serves up to 1900×1900 by rewriting the size segment
in the URL path (same trick the iTunes mzstatic + Spotify scdn
upgrades already use). SoulSync wasn't doing the rewrite — every
Deezer-sourced cover got embedded at 1000×1000 regardless of how
much higher resolution the CDN had available.
# Verified empirically
```
$ for size in 1000 1400 1800 1900 2000; do curl -I "...{size}x{size}-..."; done
1000: 200 OK 106 KB
1400: 200 OK 198 KB
1800: 200 OK 331 KB
1900: 200 OK 371 KB
2000: 403 Forbidden
```
1900 is the safe ceiling. Above that the CDN returns 403. CDN
serves source-native bytes when source < target (smaller-source
albums get same bytes whether we ask for 1000 or 1900), so asking
for 1900 universally is safe.
# Fix
New `_upgrade_deezer_cover_url(url, target_size=1900)` helper in
`core/deezer_client.py`. Pure function, mirrors the
`_upgrade_spotify_image_url` pattern that already lives in
`core/spotify_client.py`. Defensive on every input shape:
- Empty / None → returned as-is
- Non-Deezer URL (no `dzcdn`) → returned as-is
- No size segment in URL → returned as-is
- Already at/above target → returned as-is (idempotent, never
downgrades)
Applied at both cover-download sites:
- `core/metadata/artwork.py::download_cover_art` — auto post-process
flow. Mirrors the existing iTunes mzstatic upgrade right above it.
- `core/tag_writer.py::download_cover_art` — enhanced library view's
"Write Tags to File" feature.
# Scope discipline
- Helper applied at the DOWNLOAD boundary, not the source extraction
point in `deezer_client.py`. Means cached entries in the metadata
cache + DB row `image_url` columns keep the original 1000×1000 URL
Deezer's API returned. Future CDN behavior changes only affect the
download path, not stored data.
- Pre-existing `prefer_caa_art` toggle (Settings → Library →
Post-Processing) untouched — orthogonal workaround for users who
want even higher quality (MusicBrainz Cover Art Archive, often
3000×3000+).
- iTunes / Spotify upgrade paths untouched — they already worked.
# Tests added (16)
`tests/metadata/test_deezer_cover_url_upgrade.py`:
- Standard upgrade: default target 1900 on cover URL, alternate
dzcdn host (`e-cdns-images.dzcdn.net` vs `cdn-images.dzcdn.net`),
artist picture URLs (same path pattern), 500×500 source upgrades
too
- Custom target size: smaller target = no-op (never downgrade),
larger target works
- Idempotent: already at/above target returned unchanged
- Defensive on non-Deezer URLs: parametrised across 5 hosts
(Spotify scdn, iTunes mzstatic, MB CAA, Last.fm, random) — all
returned untouched
- Defensive on malformed Deezer URL (no size segment) → returned
as-is
- Empty / None handling
# Verification
- 16/16 helper tests pass
- 560/560 metadata + imports tests pass (no regression)
- 2559 full suite passes
- Ruff clean
# Bug
Plex servers with the music library named anything other than "Music"
(Música, Musique, Musik, Musica, 音乐, موسيقى, etc.) hit this error
after every import cycle:
soulsync.plex_client - ERROR - Failed to trigger library scan
for 'Music': Invalid library section: Music
soulsync.web_scan_manager - ERROR - Failed to initiate PLEX
library scan via web
Side effect: `wishlist.processing` kept reporting "Missing from
media server after sync" for tracks that DID import correctly, so
they got perpetually re-added to the wishlist.
# Root cause
`_find_music_library` correctly auto-detects the music section by
`section.type == 'artist'` and stores it on `self.music_library` —
works for any locale because the type is language-neutral. Read
methods (`get_artists`, etc.) route through `_get_music_sections`
which returns `[self.music_library]`, so they never had the bug.
But `trigger_library_scan` and `is_library_scanning` ignored
`self.music_library` and called
`self.server.library.section(library_name)` directly with the
hardcoded `"Music"` default. `server.library.section('Music')`
raises `NotFound` on any server whose section isn't literally
named "Music".
# Fix
Both methods now prefer `self.music_library` first, fall back to
literal `library_name` lookup only when auto-detection hasn't
populated the cached reference (test fixtures, edge cases).
`is_library_scanning`'s activity-feed match also corrected to
filter by the resolved section's actual title — the prior code
matched `library_name.lower() in activity_title.lower()` which
defaults to "music" and would never match activities for
non-English sections.
`trigger_library_scan`'s success log line now surfaces the actual
section title (`Música`) instead of the unused `library_name`
default ("Music") — confusing when debugging on non-English servers.
# Tests added (13)
`tests/media_server/test_plex_non_english_section_name.py`:
- `test_uses_auto_detected_section_regardless_of_locale` — parametrised
across 6 locale variants (Música, Musique, Musik, Musica, 音乐, موسيقى).
Each verifies trigger_library_scan calls the auto-detected
section's `update()`, NOT a literal-name fallback. Stub raises
AssertionError on `server.library.section()` so a regression that
re-introduces the fallback fails loudly.
- `test_falls_back_to_literal_lookup_when_no_auto_detection` —
backward compat: music_library=None → literal lookup as before.
- `test_explicit_library_name_arg_used_only_when_no_auto_detection` —
auto-detected wins over explicit kwarg when both available.
- `test_logs_correct_section_label_on_success` — log line surfaces
resolved section title.
- 4 symmetric tests for is_library_scanning covering refreshing-attr
check, activity-feed title match, no-match for unrelated sections,
fallback path.
# Verification
- 13 new tests pass
- 84/84 media_server tests pass (no regression in the existing
Plex / Jellyfin / Navidrome suite)
- 2458 full suite passes (+13 from baseline)
- Ruff clean
Live-API verification revealed advanced-syntax queries hurt more
than they help on this endpoint. Switching the import-modal Deezer
search back to free-text + local rerank.
# What live testing showed
Hit Deezer's public API with both query forms for the issue #534
case (`Dirty White Boy` + `Foreigner`):
**Free-text (`q=Dirty White Boy Foreigner`):**
- Returns 21 results
- Real Foreigner Head Games studio cut at #1
- Live versions at #2-10
- Karaoke / cover variants at #11-15
**Advanced (`q=track:"Dirty White Boy" artist:"Foreigner"`):**
- Returns 12 results
- "(2008 Remaster)" at #1 — canonical Head Games cut MISSING from
top 8 entirely
- Live + alt-album versions follow
Advanced syntax DOES filter karaoke at the API level (none in the
12-result set vs. 5 at positions 11-15 in free-text), but it has
its own ranking bias that surfaces remasters / "Best Of" cuts
ahead of the canonical recording. Net regression for the user-
facing goal.
# Fix
1. Endpoint reverts to free-text query with local rerank applied.
2. Local rerank gains "remaster" / "remastered" / "reissue"
patterns under VARIANT_TAG_PATTERNS (soft 0.4× penalty — user
may want them but they shouldn't outrank the original).
3. Client kwarg support (`track=` / `artist=` / `album=`) preserved
for future opt-in callers (e.g. exact-match flows where API-
level filtering matters more than ranking).
# Verified end-to-end against live Deezer API
Re-ran the exact #534 case through the live API + new rerank.
Top 15 results post-rerank:
1. Dirty White Boy — Foreigner — Head Games ← REAL CUT AT TOP
2-10. Various Live versions
11-15. Karaoke / cover / tribute variants ← BURIED
Real Foreigner Head Games studio cut at #1, exactly the user's
ask.
# Tests
- `test_relevance.py` — variant tag patterns extended; existing
tests still pass (50 tests).
- `test_search_match_endpoints.py::test_joins_track_and_artist_into_free_text_query`
— replaces `test_passes_track_and_artist_as_kwargs`; verifies
endpoint sends free-text join, NOT field-scoped kwargs (the
prior test asserted the wrong direction now).
- Karaoke-burying assertion at the endpoint still pins the
user-visible behaviour.
- Client kwarg path tests untouched (still pin advanced-syntax
construction for future opt-in callers).
# Verification
- 75 relevance + endpoint + query tests pass
- 2445 full suite passes
- Ruff clean
- Live Deezer API shows real cut at #1 post-rerank
Defensive followup to the relevance fix. Deezer's advanced search
syntax (`artist:"X"`) is documented as substring match, but in
practice it's brittle on artist name variants ("Foreigner [US]",
"The Foreigner") and on tracks indexed under non-canonical title
spellings. When the advanced query returns nothing, we'd previously
land at "No matches" — a regression vs. pre-fix behaviour where
free-text would have returned a less-relevant but non-empty set.
Fix: when the advanced query returns 0 results AND the caller used
field-scoped kwargs, fall back to a free-text join of the same
kwargs and re-query. Caller-side rerank still tightens whatever the
fallback returns, so the worst-case post-fix behaviour is the
pre-fix behaviour — never strictly worse.
Pulled the cache + parse + store dance into a private helper
(`_search_tracks_with_query`) so the orchestration can call it
twice (advanced → fallback) without code duplication. Single API
call when the advanced query has results — no wasted requests.
Diagnostic logger.debug fires when the fallback triggers so we can
see in production whether it's happening (and to which queries).
# Tests added (4)
- `test_falls_back_to_free_text_when_advanced_empty` — advanced
query returns 0, free-text returns hits; client returns the
free-text hits + both API calls fire.
- `test_no_fallback_when_advanced_query_has_results` — single hit
on advanced query → no second API call.
- `test_no_fallback_when_legacy_free_text_call` — legacy callers
already exhausted the only path; empty result is final.
- `test_no_fallback_when_query_unchanged` — empty kwargs path
doesn't trigger the fallback branch (used_advanced=False).
# Existing tests updated
The 4 prior `TestSearchTracksQueryWiring` + `TestSearchTracksCacheKey`
tests were stubbing `_api_get` to return empty `{'data': []}` and
asserting `assert_called_once`. With the new fallback, those stubs
trigger a second API call and the assertions break — even though
the FIRST call construction is what the tests cared about. Updated
the stubs to return one fake hit so the fallback doesn't fire, and
switched to `call_args_list[0]` for first-call inspection.
# Verification
- 18/18 deezer query tests pass (14 prior + 4 new)
- 2445 full suite passes (+4 from prior commit)
- Ruff clean
# Background
User reported (#534) that the import-modal "Search for Match" dialog
returned irrelevant results when Deezer was the metadata source.
Searching `Dirty White Boy` + `Foreigner` returned 5+ karaoke /
"originally performed by" / "in the style of" / "re-recorded" /
tribute-band results ranked above the actual Foreigner studio cut
from Head Games. User had to scroll past the junk every time, or
fall back to iTunes search which is much slower.
# Root cause — two layers
1. **Endpoint joined `track + artist` into free-text query.**
`/api/deezer/search_tracks` was passing `q=Dirty White Boy Foreigner`
to Deezer's `/search/track` API. Deezer fuzzy-matches that
string across title / lyrics / artist / album / contributors and
orders by global popularity — anything that appears across many
compilations outranks the canonical recording.
2. **No local rerank.** None of the search-modal endpoints applied
any post-filtering. Deezer's API order shipped straight to the
user.
# Fix — same architectural shape Cin would build
## Layer 1: field-scoped query at the client boundary
`core/deezer_client.py::search_tracks()` now accepts optional
`track`, `artist`, `album` kwargs. When provided, builds Deezer's
advanced search syntax: `q=track:"X" artist:"Y" album:"Z"`. Massive
relevance improvement because each term matches the right field
instead of fuzzy-matching everywhere.
Backward compat preserved: legacy free-text `query=` callers still
work unchanged. Field-scoped path takes precedence when both are
provided. Empty input fast-fails without an API call. Embedded
double-quotes stripped (Deezer's syntax has no escape mechanism).
## Layer 2: provider-neutral relevance reranker
New `core/metadata/relevance.py` module — pure-function rerank over
the canonical `Track` dataclass. Composable scoring:
- **Cover/karaoke patterns** (multiplier 0.05, effectively buries):
matches "karaoke", "originally performed by", "in the style of",
"made famous by", "tribute", "vocal version", "backing track",
"cover version", "re-recorded", "cover by", etc. across title,
album, AND artist fields. Catches the screenshot's exact junk:
artist credits like "Pop Music Workshop" / "The Karaoke Channel"
/ "Foreigner Tribute Band".
- **Variant tags** (multiplier 0.4): live / acoustic / demo /
instrumental / remix / radio edit / club mix etc. — softer
penalty since the user MAY want them. Skipped entirely when the
expected_title contains the same tag (so searching
"Track (Live)" still ranks Live versions first).
- **Exact artist boost** (multiplier 1.5): primary artist exactly
matches expected_artist after normalisation. Single strongest
signal for "this is the canonical recording".
- **Title + artist similarity** via SequenceMatcher (parentheticals
+ punctuation stripped before comparison).
- **Album-type weighting**: album=1.0 > single/ep=0.85 > compilation=0.7.
Compilations are more likely tribute / karaoke repackages.
Each component is a standalone function so tests pin them
individually without standing up the full pipeline.
## Wired at three search-modal endpoints
- `/api/deezer/search_tracks` — uses both layers (field-scoped
query + rerank).
- `/api/itunes/search_tracks` — uses rerank only (iTunes API has
no advanced-syntax search, but karaoke / cover variants still
leak through and need the local penalty).
- `/api/spotify/search_tracks` — already builds field-scoped
`track:X artist:Y` query; rerank added as the consistency safety
net so all three sources behave the same from the user's
perspective.
Other Deezer call sites (matching engine, watchlist scanner,
auto-import single-track ID) deliberately not touched in this PR
— they have their own elaborate scoring pipelines tuned to their
specific contexts and aren't surfacing the user-reported issue.
Per Cin: "don't refactor beyond what the task requires."
# Tests
71 new tests across 3 files:
- `tests/metadata/test_relevance.py` (50 tests) — every scoring
component pinned individually + the issue #534 screenshot
reproduced as a regression test (real Foreigner cut wins after
rerank, karaoke variants drop to bottom).
- `tests/metadata/test_deezer_search_query.py` (14 tests) —
advanced-syntax query construction, field-scoped wiring at the
client boundary, free-text path unchanged, kwargs win when
ambiguous, limit clamping, cache key consistency.
- `tests/imports/test_search_match_endpoints.py` (7 tests) —
end-to-end through Flask test client: Deezer endpoint passes
kwargs not joined query; karaoke buried at bottom for all three
sources; legacy query param still works without rerank.
# Verification
- 2441 full suite passes (+71 from baseline 2370)
- 0 failures (the prior watchdog flake fix held)
- Ruff clean across all changed files
- JS parses clean (`node -c webui/static/helper.js`)
# Architectural standards followed
- **Logic at the right boundary.** Query construction lives in the
client (every caller benefits from one change). Rerank lives in
a neutral module (`core/metadata/relevance.py`) over the
canonical `Track` dataclass — works for any source, not Deezer-
specific.
- **Explicit > implicit.** Every scoring rule has its own named
function. Pattern tables are module-level constants tests can
introspect.
- **Scope discipline.** Audited every Deezer search call site;
fixed the user-reported one + the consistent siblings. Did NOT
speculatively normalise every Deezer call across the codebase.
- **Backward compat.** Free-text `query=` callers untouched. Kwargs
added to existing client method signature with safe defaults.
- **Tests pin contract at correct boundary.** Pure-function rerank
tests don't mock anything; client-query tests stub at `_api_get`;
endpoint tests run through the real Flask app.
Two pre-existing parity gaps in `record_soulsync_library_entry` that
the prior parity commits left untouched. Both close real holes
between auto-import writes and what the soulsync_client deep scan
would have produced.
# Gap 1: Album duration was the first-imported track's duration
`record_soulsync_library_entry` is called once per track. The album
INSERT only fires for the FIRST track of a new album (subsequent
tracks find the album row already exists). The INSERT was passing
`duration_ms` — `track_info["duration_ms"]` — as the album's
`duration` column. That's the duration of one track, not the album
total. Compare to `SoulSyncAlbum.duration` in soulsync_client which
is `sum(t.duration for t in self._tracks)`.
Fix:
- Worker computes `album_total_duration_ms = sum(...)` across every
matched track and threads it onto context as
`album.duration_ms`.
- side_effects reads that value (or falls back to the per-track
duration for legacy non-auto-import callers) and writes it as the
album row's `duration`.
# Gap 2: Re-imports of the same artist/album were insert-only
When the SELECT-by-id or SELECT-by-name found an existing soulsync
artist or album row, the function skipped completely — no UPDATE
path. Meant: artist genres / thumb / source-id reflected ONLY
whatever the FIRST imported album supplied, never refreshing as
more albums by that artist landed. Ten more imports later, the
artist row still held whatever the first random import wrote.
Conservative fix: when an existing row matches, run an UPDATE that
fills only the columns whose current value is NULL or empty. Never
overwrites populated values — protects manual edits +
enrichment-worker writes the same way the scanner UPDATE path
preserves enrichment columns.
Implementation note: the empty-check happens in Python, NOT SQL.
Initial pass tried `COALESCE(NULLIF(col, ''), NULLIF(col, 0), ?)`
but SQLite's `NULLIF(text_col, 0)` returns the original text value
instead of NULL — different types, no coercion. So the SQL-only
conditional was unreliable on text columns. New helper does
`SELECT cols FROM table WHERE id`, compares each column in Python,
and emits UPDATE clauses only for the ones that need filling.
Allowlist defense: f-string column names go through
`_SOULSYNC_FILLABLE_COLUMNS` validation before interpolation.
Misuse adding new columns without an allowlist update fails closed
(logger.debug + skip).
# Tests added (4)
- `test_album_duration_uses_album_total_not_single_track` —
album with single-track context carrying explicit
`album.duration_ms = 2_500_000` writes 2_500_000 to the album row,
not the per-track 200_000 fallback.
- `test_re_import_fills_empty_artist_fields` — first import lands
artist with empty thumb + empty genres; second import for same
artist with thumb + genres present updates the existing row.
- `test_re_import_does_not_clobber_populated_artist_fields` —
first import writes rich genres + thumb; second import with
worse / different metadata leaves the existing row untouched.
- `test_re_import_fills_empty_source_id_when_missing` — first
import had no source artist ID; second import does — fills the
empty `spotify_artist_id` column on the existing row.
# Verification
- 10/10 side-effects tests pass (including 4 new + 4 from prior
parity commit + 2 history/provenance)
- 217 imports tests pass (no regression)
- 2369 full suite passes (+4 from prior, +22 PR-total from baseline 2347)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated)
- Ruff clean
Cin pre-review followup. Two small parity gaps the prior commits left
open:
# 1. Genre tags land on the standalone artists row
`soulsync_client._scan_transfer` aggregates the GENRE tag across every
track in an album and surfaces it on `SoulSyncAlbum.genres` (which the
DatabaseUpdateWorker writes to the artists+albums row). Auto-import
was hardcoding `'spotify_artist': {'genres': []}` so the imported
artists row landed with empty genres — felt hollow compared to a
Plex/Jellyfin scan, which both pull genres from their respective APIs.
Fix:
- `_read_file_tags` now reads the GENRE tag (mutagen easy mode handles
MP3/FLAC/M4A consistently; some files carry multiple genres so it's
always returned as a list).
- `_process_matches` aggregates genres from each matched file's tags
into a deduped insertion-order list. Dedup is case-insensitive but
preserves original casing — so "Hip-Hop, Rap, Trap" reads naturally
in the JSON column instead of "hip-hop, rap, trap".
- Worker context's `spotify_artist['genres']` carries the aggregated
list, which `record_soulsync_library_entry` already filters via
`core.genre_filter.filter_genres` and writes to the artists row.
# 2. Defensive str() cast for ISRC + MBID
`_build_album_track_entry` already coerces ISRC + MBID to string today
(via `str(isrc) if isrc else ''`). But if a future metadata-source
client returns int / None for either ID, the worker would propagate
the wrong type and side_effects.py's `.strip()` would AttributeError.
Cheap insurance: explicit `str()` cast in the worker before assignment
to track_info. Future-proofs against client drift.
# Tests added (3, in test_auto_import_context_shape.py):
- `test_context_aggregates_genres_from_track_tags` — multi-file
album with overlapping genre lists produces deduped, insertion-
ordered, original-case-preserved result. Stubs `_read_file_tags`
with monkeypatch so we don't need real audio.
- `test_context_genres_empty_when_no_tags` — files without GENRE
tag → empty list. Standalone library write handles gracefully
(genres column stays empty / NULL).
- `test_context_isrc_mbid_coerced_to_string` — hostile types
(int 12345678, None, int 999) coerced to safe strings before
reaching track_info.
# Verification
- 14/14 context-shape tests pass (11 prior + 3 new)
- 213 imports tests pass (no regression)
- 2365 full suite passes (+3 from prior, +18 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
Cin pre-review followup to the standalone library parity commit. The
prior commit fixed `spotify_artist['id']` from the wrong copy-paste
value (`identification['album_id']`) to read from
`identification['artist_id']`, but the identification dict produced
by `_search_metadata_source` and `_search_single_track` never set
`artist_id` — both extracted artist NAME from the search response
and discarded the source ID sitting right next to it. Net effect of
the prior commit: artists row source-id stayed NULL, just for a more
honest reason than before.
Now properly extracted:
- `_search_metadata_source` reads `best_result.artists[0]['id']`
alongside the artist name and returns it on the identification dict
as `artist_id`.
- `_search_single_track` does the same for single-track identification.
- `_identify_single`'s tag-based-confidence path forwards
`result.get('artist_id')` so the artist source-id propagates even
when high-confidence local tags override the search result's name.
Result: identification dict now carries `artist_id` whenever the
metadata source returned an artist with an ID. The worker context
already plumbs it onto `spotify_artist['id']` and
`spotify_album['artists'][0]['id']`, so the standalone library write
finally populates `<source>_artist_id` on the artists row.
Tests added (3, in `test_auto_import_context_shape.py`):
- `test_context_artist_id_uses_identification_artist_id` — when the
identification dict carries `artist_id`, context propagates it
onto `spotify_artist['id']` AND
`spotify_album['artists'][0]['id']`. Pins that the prior copy-
paste bug (artist['id'] = album_id) doesn't return.
- `test_context_artist_id_is_empty_when_identification_missing_it` —
fallback case (filename-only identification): context gets empty
string, NOT album_id. Honest failure mode.
- `test_search_metadata_source_extracts_artist_id_from_dict_artist`
— black-box test of `_search_metadata_source`: feed it a
spotify-shaped result with `artists[0]['id']` and verify
identification dict carries it forward.
Verification:
- 11/11 context-shape tests pass (8 prior + 3 new)
- 210 imports tests pass (no regression)
- 2362 full suite passes (+3 from prior commit, +15 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
# Background
SoulSync standalone is meant to be a full replacement for Plex /
Jellyfin / Navidrome — files imported via auto-import (or any other
import path) should land in the database with the same field richness
a media-server scan would write. They weren't.
# Gaps fixed
The auto-import worker built a context dict for each track and handed
it to `_post_process_matched_download` (the same callback the regular
download flow uses). That dict was missing three things downstream
needed:
1. **No `source` field anywhere.** `record_soulsync_library_entry`
reads `get_import_source(context)` to pick the source-aware ID
columns (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
etc.) on the artists / albums / tracks rows. With no source, the
resolver returned an empty string → `get_library_source_id_columns("")`
returned an empty dict → the `UPDATE tracks SET <source>_id = ?`
blocks were silently skipped. Result: every auto-imported track
landed with NULL on every source-id column. Watchlist scans
(which match by stable source IDs to detect "this track is already
in library") couldn't recognise these rows and would re-download
them on the next pass.
2. **No `_download_username='auto_import'`.** Both
`record_library_history_download` and `record_download_provenance`
default to "Soulseek" when no `username` is in the context. Every
staging-folder import was being labelled as a Soulseek download
in library history + provenance — false signal in the UI.
3. **No per-recording IDs (`isrc`, `musicbrainz_recording_id`) on
track_info.** The Navidrome scanner already writes
`musicbrainz_recording_id` directly to the tracks row when present.
Picard-tagged libraries always carry MBID; metadata sources
(Spotify via MusicBrainz enrichment, Deezer, etc.) carry ISRC.
Auto-import had access to both via the metadata-source response
but didn't propagate them — so the soulsync row went in with
NULL on both columns.
# Changes
**`core/auto_import_worker.py` — `_process_matches`:**
- Top-level `'source': source` (from `identification['source']`)
- `'_download_username': 'auto_import'`
- `track_info['isrc']`, `track_info['musicbrainz_recording_id']` —
pulled from the per-track payload returned by the metadata source
- `track_info['album_id']` — back-reference so source-aware ID
resolution works on sources whose API nests album under
`track.album.id` rather than `track.album_id`
- `spotify_artist['id']` now correctly carries the artist's source ID
(was `identification['album_id']`, a copy-paste bug from the
original implementation that made artist-id resolution fall back
to fuzzy matching)
- `spotify_album['artists'][0]['id']` carries artist source ID for
the same resolution path
**`core/imports/side_effects.py`:**
- `record_library_history_download` source_map: add
`"auto_import": "Auto-Import"` — tags imported tracks correctly
- `record_download_provenance` source_service: add
`"auto_import": "auto_import"` — provenance shows real source
- `record_soulsync_library_entry` track INSERT: now includes
`musicbrainz_recording_id` + `isrc` columns (matches
`insert_or_update_media_track`'s shape for Navidrome /
Plex / Jellyfin scans). Both default to NULL when not present.
# Behavior preserved
- Files still land in the same library template path (no path-build
change)
- Other media-server flows (Plex / Jellyfin / Navidrome users)
unaffected — `record_soulsync_library_entry` still gates on
`get_active_media_server() == "soulsync"`. Auto-import on those
servers continues to drop the file in the library folder + emits
`batch_complete` for the scan-trigger automation, same as before.
- Direct downloads (search → Download button) unaffected — they
already passed `source` + `username` correctly.
# Tests added
`tests/imports/test_auto_import_context_shape.py` (8 tests, new file):
- Worker context carries `source` for every metadata source
(parametrised across spotify / deezer / itunes / discogs)
- `_download_username='auto_import'` set unconditionally
- ISRC + MBID propagate from track payload to track_info when present
- ISRC + MBID default to empty string when absent (downstream
normalises to NULL at write time)
- track_info includes album-id back-reference
`tests/imports/test_import_side_effects.py` (4 new tests + 2 schema
column adds):
- `record_soulsync_library_entry` writes mbid + isrc columns when
present in track_info
- Deezer source maps to deezer_id column (regression case for
source-aware column resolver)
- `record_library_history_download` labels `_download_username=
'auto_import'` as "Auto-Import" not "Soulseek"
- `record_download_provenance` registers source_service as
"auto_import" not "soulseek"
# Verification
- 8/8 new context-shape tests pass
- 6/6 side-effects tests pass (4 new + 2 existing)
- 207 imports tests pass
- 2359 full suite passes (+12 from baseline 2347, no regressions)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated to this change)
- Ruff clean
Per the semver workflow the version string only bumps at release
time, so the running dev work on the 2.4.3 line should stay listed
under 2.4.3 (not pre-create a 2.4.4 block). Merged the prior
'2.4.4' key's six dev entries into the top of '2.4.3', above the
existing "May 8, 2026 — 2.4.3 release" date marker, with a
"Unreleased — 2.4.3 patch work" date marker so the visual split
between unreleased + released entries is preserved.
`_getLatestWhatsNewVersion` resolves to the current build version
(2.4.3 in `_SOULSYNC_BASE_VERSION`); with the 2.4.4 key gone, the
helper modal now surfaces the dev work alongside the released
entries when the user opens "What's New", instead of being silently
hidden until a future build bump.
The release-time bump remains the canonical step that splits
"unreleased" entries off into their own version block — done as
the last commit on dev before merging dev → main.
No code changes — pure WHATS_NEW reorganisation.
# Concurrency model
Pre-refactor concurrency was emergent + unbounded:
- The worker's `_run` thread called `_scan_cycle` every 60s,
processing candidates synchronously in a for-loop.
- The `/api/auto-import/scan-now` endpoint spawned a fresh
`threading.Thread(target=_scan_cycle)` per click — extra parallel
scan cycles on top of the timer.
- Multiple "Scan Now" clicks during in-flight processing → multiple
threads racing on `_processing_paths` / `_folder_snapshots` state,
no upper bound on concurrent scanners.
- `stop()` didn't wait for in-flight processing — could leave file
moves / tag writes / DB inserts mid-flight.
Refactor to the pattern Cin uses elsewhere (`missing_download_executor`,
`sync_executor`, `import_singles_executor` all use
`ThreadPoolExecutor(max_workers=3, thread_name_prefix=...)`):
- **One scan thread** — both timer + manual triggers go through
`trigger_scan()`, gated by a non-blocking `_scan_lock`. Duplicate
triggers no-op instead of stacking parallel scanners.
- **Bounded executor** — `ThreadPoolExecutor` (default 3 workers,
configurable via `auto_import.max_workers`) runs per-candidate
work. Each candidate runs to completion in its own pool thread;
up to N candidates run in parallel.
- `_scan_and_submit()` is fast — just enumeration + executor submit,
returns immediately, doesn't block on per-candidate work.
- `_process_one_candidate(candidate)` holds the per-candidate logic
identical to the old for-loop body, lifted into a method so the
pool can run multiple instances concurrently.
- `_submitted_hashes` set + lock dedupes candidates across the
timer + manual triggers so a candidate already queued / running
doesn't get re-submitted.
- `stop()` calls `executor.shutdown(wait=True)` — clean shutdown,
no orphaned file ops.
# Per-candidate UI state isolation
The executor refactor opened two concurrency holes that the old
sequential model masked. Both fixed in this commit:
1. **Scalar UI fields stomped across pool workers.** Pre-refactor
`_current_folder` / `_current_status` / `_current_track_*` were
safe under the sequential model — only one candidate processed
at a time, so the fields tracked the in-flight one. With three
pool workers writing the same fields, the polling UI saw garbage
like "Processing AlbumA, track 7/14: SongFromAlbumB".
Replaced with `_active_imports: Dict[hash, _ActiveImport]` keyed
on folder_hash, gated by `_active_lock`. Each pool worker owns
its own entry. Helpers `_register_active` / `_update_active` /
`_unregister_active` / `_snapshot_active` are the only API.
2. **Stats counters not thread-safe.** `self._stats[k] += 1` is
read-modify-write — under load, parallel pool workers drop
increments. New `_stats_lock` + `_bump_stat()` helper wraps every
mutation. `get_status()` reads under the same lock and returns
a copy.
# Endpoint change
`/api/auto-import/scan-now` no longer spawns its own scan thread —
calls `auto_import_worker.trigger_scan()` (which routes through the
shared lock + executor). Multiple clicks while a scan is in flight
no-op deterministically. Endpoint still wraps the call in a daemon
thread so the HTTP response returns immediately even if the staging
walk is slow.
# Backward compat
The scalar `_current_folder` / `_current_status` / `_current_track_*`
fields are preserved as **read-only properties** that resolve to the
FIRST active import. The existing `get_status()` payload still
includes those fields populated from the first entry — single-import
UIs (and the test fixture) keep working unchanged. New
`active_imports` array exposes the full multi-candidate state for
parallel-aware UIs.
# Behavior preserved
- Per-candidate identify / match / process logic byte-identical
- Live-progress state preserved (per candidate now)
- Stability gate / already-processed dedup preserved
- `_record_in_progress` / `_finalize_result` UI rows preserved
- Tag-based loose-file grouping unchanged
# Behavior changes
- Multiple albums process IN PARALLEL up to `max_workers`
- "Scan Now" while scan in progress no-ops (was: spawned another)
- `stop()` waits for in-flight pool work via `shutdown(wait=True)`
- Auto-import card now lists each in-flight album (one line per
active import) instead of a single shared progress line
# UI
`webui/static/stats-automations.js`:
- Progress widget reads `active_imports` array, renders one line
per in-flight album with per-candidate status / track index
- Falls back to the legacy summary line when payload doesn't
carry `active_imports` (older backend)
- Per-row "live processing" lookup now matches by `folder_hash`
through the array instead of by `folder_name` against scalars
# Tests added (`tests/imports/test_auto_import_executor.py`)
- Pool config: default max_workers=3, configurable via constructor
+ via `auto_import.max_workers` config, floors at 1
- Scan lock: 5 concurrent `trigger_scan()` calls run only 1 scan
while lock held; releases properly so subsequent triggers run
- Executor dispatch: 5 candidates → 5 process calls via the pool
- Bounded parallelism: max_workers=3 caps at 3 concurrent;
max_workers=2 caps at 2
- Cross-trigger dedup: candidate submitted in scan A doesn't get
re-submitted by scan B while still in-flight
- Graceful shutdown: `stop()` blocks until in-flight pool work
finishes
- Per-candidate state isolation: 2 parallel workers updating their
own candidate state don't interfere — each candidate's
track_index / track_name / folder_name reads back exactly as
written for that hash
- `get_status()` returns coherent `active_imports` array with
one entry per in-flight candidate; aggregate top-level
`current_status` is 'processing' when any entry is processing
- Unregister removes only that candidate, others stay visible
- Stats counter thread-safety: 1000 parallel bumps land at 1000
(the read-modify-write race regresses without the lock)
- `get_status()` stats snapshot is a copy, not a live reference
# Verification
- 17 new tests pass (executor + state isolation)
- 2347 full suite passes (1 pre-existing flaky test —
`test_watchdog_warns_about_stuck_workers` — passes in isolation,
unrelated)
- Ruff clean
Brings the auto-import matcher to picard / beets / roon parity by
reaching for the existing AcoustID-grade infrastructure (typed Album
foundation, integrity check thresholds) and layering id-based exact
matches on top of the fuzzy scorer. Picard-tagged libraries now land
every track with full confidence on the first pass.
Three layered phases in `core/imports/album_matching.match_files_to_tracks`:
1. **MBID exact match** — file has `musicbrainz_trackid` tag, source
returns the same id → instant pair, full confidence, no fuzzy
scoring. Picard's primary identifier; per-recording.
2. **ISRC exact match** — file has `isrc` tag, source returns the same
id → same fast-path, slightly lower priority than mbid (isrc can
be shared across remasters). Both ids normalised before compare
(uppercase + strip dashes/spaces for isrc, lowercase for mbid).
3. **Duration sanity gate** — files in the fuzzy phase whose audio
length differs from the candidate track's duration by more than
`DURATION_TOLERANCE_MS` (3s, matching the post-download integrity
check) are rejected before scoring runs. Defends against the
cross-disc / cross-release / wrong-edit problem the integrity
check used to catch only AFTER the file had already been moved +
tagged + db-inserted.
Tag reader (`_read_file_tags`) extended:
- Reads `isrc` (uppercased, strip / / spaces normalisation deferred
to matcher)
- Reads `musicbrainz_trackid` as `mbid` (lowercased)
- Reads `audio.info.length` and converts to `duration_ms` to match
the metadata-source convention
Metadata-source layer (`_build_album_track_entry`) extended:
- Propagates `isrc` from top-level OR `external_ids.isrc` (spotify
shape — would otherwise be stripped before reaching the matcher)
- Propagates `musicbrainz_id` from top-level OR `external_ids.mbid`
/ `external_ids.musicbrainz`
- Without this layer, fast paths would silently never fire in
production even though unit tests pass — pinned by
`test_album_track_entry_propagates_isrc_and_mbid_from_source`
18 new tests in `tests/imports/test_album_matching_exact_id.py`:
- Direct: `find_exact_id_matches` with mbid, isrc, isrc normalisation,
mbid > isrc priority, spotify-shape `external_ids.isrc`, no-id
empty result, file-used-at-most-once
- Direct: `duration_sanity_ok` within / outside tolerance, missing
durations defer
- End-to-end via `match_files_to_tracks`: mbid match short-circuits
fuzzy scoring, id-matched files excluded from fuzzy phase, duration
gate rejects wrong-disc collisions in fuzzy phase, normal matches
pass through the gate, missing durations fall through, deezer
seconds-vs-ms conversion, full picard-tagged 10-track album via
mbid only
- Production-shape: `_build_album_track_entry` propagates isrc + mbid
from spotify-shape (`external_ids.isrc`) AND itunes-shape (top-
level `isrc`)
Verification:
- 35 album-matching tests pass total (17 helper + 18 fast-path)
- 23 multi-disc tests still pass after the extension (additive)
- Full suite: 2311 passed (+18 new), 1 pre-existing flaky timing test
failure (`test_watchdog_warns_about_stuck_workers` — passes in
isolation, fails only in full-suite runs, unrelated to this PR)
- Ruff clean
For users:
- Picard / Beets / Mp3Tag-tagged libraries (anyone who's organised
their music) get instant perfect-confidence matches every time.
- Soulseek-tagged downloads (which usually carry isrc when sourced
via metadata-aware soulseekers) get the fast path too.
- Naively-named files with no useful tags fall through to the
improved fuzzy + duration-gated path — same correctness as before
for the common case, much harder for the matcher to confidently
pair the wrong file.
- One step closer to standalone-DB feature parity with plex /
jellyfin / navidrome scanners. Acoustid fingerprint fallback
(for files with NO useful tags AND no MBID/ISRC) is the next
followup PR.
Caught while live-testing the #524 fix with kendrick lamar
mr morale & the big steppers (3 discs). User dropped discs 1+2
loose in staging root + disc 3 in its own folder, every file
perfectly tagged with disc_number/track_number/title — only 9
tracks ended up in the library, the rest got integrity-rejected
and quarantined.
Two related bugs in `AutoImportWorker._match_tracks`:
1. **Quality dedup keyed on track_number alone.** The dedup loop
kept `seen_track_nums[track_number] = file` and dropped any later
file with the same number, treating it as a quality duplicate.
On a multi-disc release where every disc has tracks 1..N, that
collapses the album to one disc's worth of files BEFORE the
matcher runs. User's 18 loose disc-1+disc-2 files reduced to 9
before any title/disc info was even consulted.
2. **Match scoring ignored disc_number.** The 30% track-number bonus
fired whenever `ft[track_number] == track_num` regardless of disc.
File with tag (disc=2, track=6, "Auntie Diaries", 281s) got the
full bonus matching API track (disc=1, track=6, "Rich Interlude",
103s) — wrong file → wrong destination → integrity check correctly
rejected and quarantined the file. Same for tracks 7, 8, 9.
Fix:
- Dedup keys on `(disc_number, track_number)` tuples — multi-disc
files with parallel numbering all survive.
- Match scoring's 30% bonus only when BOTH disc AND track agree.
Cross-disc same-track-number collisions get a small 5% consolation
bonus so title similarity has to carry the match (covers cases
where tag disc info is missing or wrong).
- API track disc_number read from `disc_number` (Spotify) /
`disk_number` (Deezer) / `discNumber` (iTunes) defaulting to 1.
4 new pinning tests in `tests/imports/test_auto_import_multi_disc_matching.py`:
- 18-file 2-disc regression case (dedup preserves all)
- (disc=2, track=6) file matches API (disc=2, track=6) track, not
the disc-1 same-numbered track
- Single-disc albums still match normally (no regression)
- Quality dedup within a single (disc, track) position still picks
higher-quality format (.flac over .mp3)
Verification:
- 2268 full pytest suite passes (+4 new), 1 skipped, 0 failed
- Ruff clean
Same branch as the #524 fix because both surfaced from the same
import session — easier reviewer context if they ship together.
radoslav-orlov reported every imported album landing in the soulsync
standalone library as "Unknown Artist" + the raw 10-digit album id
as the title + 0 tracks. Audit traced it to the click handler in the
import page dropping the source-of-the-album_id on its way to the
backend match endpoint.
Root cause:
`importPageSelectAlbum(albumId)` (the onclick on every suggestion /
search-result card) only passed the album_id string. The full search
response carried `source`, `name`, and `artist` per row — the
backend's `get_artist_album_tracks` needs source so it can route the
lookup to the metadata source the id actually came from. Without it,
the source chain tries each source's `get_album(id)` against an id
shaped for a different source — a Deezer numeric id against
Spotify's id format returns 404, against iTunes's collectionId range
returns 404, etc. — and falls through to the failure-fallback dict
in `get_artist_album_tracks`:
{
'success': False,
'album': {'name': album_name or album_id, 'total_tracks': 0,
'release_date': '', ...}, # no artist field at all
'tracks': [],
}
That broken album dict then flowed through `build_album_import_context`
→ post-processing pipeline → `record_soulsync_library_entry`, writing
"Unknown Artist" + album_id-as-title + 0 tracks rows into the
soulsync standalone library tables.
Why hybrid users hit it most: a Spotify-primary user searching for an
album → search returns the Spotify result PLUS Deezer fallbacks
(via `_search_albums_for_source`'s priority chain). Clicking a Deezer
fallback row then sent only the Deezer id to /album/match without
flagging that source — Spotify-first chain failed against the Deezer
id and the broken fallback got written.
Fix:
Frontend (`webui/static/stats-automations.js`):
- New `importPageState._albumLookup: { albumId: { id, name, artist,
source } }` populated by both card renderers (`_renderSuggestionCard`
+ the search-results render block) before they emit the onclick.
- `importPageSelectAlbum` reads source / name / artist from that
cache and includes them in the match POST body, so the backend
routes to the correct provider's `get_album` on the very first try.
- `_escAttr` applied to album_id in the onclick (defensive — ids
shouldn't contain quotes but `_escAttr` was already being used on
every other field interpolated into onclick attributes).
Backend (`web_server.py:import_album_match`):
- Defensive log warning when source is missing from the request body.
Catches any future regression where another caller (curl /
third-party / new UI flow) drops source again — it'll show up as
a visible warning in app.log instead of silently corrupting the
library.
Verification:
- Full pytest suite: 2264 passed, 1 skipped, 0 failed
- Ruff clean
- JS syntax clean
- Manual repro requires a real user flow (search albums on the
import page → click one → import) which isn't covered by the
existing unit tests; reviewer should verify against issue #524's
steps before merge.
Three follow-on fixes to the manual-search candidates modal once people
started actually using it:
1. NDJSON streaming. Manual search waited for every source to return
before showing anything. Now streams one event per source as each
completes — header line, source_results per source, done terminator.
Frontend appends rows incrementally via response.body.getReader().
2. Manual picks no longer auto-retry on failure. New _user_manual_pick
flag set on the task in /download-candidate. Both monitor retry
paths (not-in-live-transfers stuck + Errored state) bail on the
flag. Surfaces the failure to the user instead of silently picking
a different candidate via fresh search.
3. Non-Soulseek manual picks (youtube/tidal/qobuz/hifi/deezer/
soundcloud/lidarr) no longer stuck at "downloading 0%" forever. The
live_transfers IF branch now marks manual-pick tasks failed
directly when the engine reports Errored, instead of deferring to
the monitor (which bails on manual picks). Engine fallback in else
branch covers the rare race where the orchestrator's pre-populated
transfer lookup is missing the entry.
Plus a deadlock fix discovered along the way: the new failure path
synchronously called on_download_completed while holding tasks_lock,
which itself re-acquires the same Lock — non-reentrant
threading.Lock self-deadlocked the polling thread. While wedged, every
other endpoint that needed the lock (including /candidates → other
failed rows couldn't open modals) hung waiting. Moved completion
callbacks onto a daemon thread so the lock releases first.
Plus failed/not_found/cancelled rows are now ALWAYS clickable (not
just when the auto-search cached candidates) — the modal carries the
manual search bar, which is the user's recourse for empty results.
Plus manual download worker now runs on a dedicated thread instead of
competing with the batch's 3-worker missing_download_executor pool —
saturated batches no longer queue manual picks indefinitely.
All scoped to manual picks via the _user_manual_pick flag — auto
attempt flow byte-identical to before. Engine fallback gated on the
flag too so auto attempts in the else branch keep the original
do-nothing behavior (safety valve handles the stuck-forever case).
Also dropped _handle_failed_download from web_server.py — defined
but had no callers (dead code).
17 new unit tests pin the gate behavior:
- engine fallback: Errored/Cancelled/Succeeded/InProgress transitions,
manual-pick gate, terminal-state skip, soulseek skip, missing
download_id skip, engine returning None, orchestrator exception
- monitor: manual-pick skips not-in-live-transfers retry + Errored
retry
- IF-branch end-to-end: Errored marks failed, "Completed, Errored"
hits failure branch, auto attempts defer to monitor
Manual-search endpoint tests rewritten for NDJSON: 11 cases (validation,
single-source dispatch, parallel "all" dispatch, one-event-per-source
streaming shape, unconfigured-source skip + reject, header metadata,
per-source exception isolation).
Full suite 2259 passed, 1 skipped.
When an auto-download fails or returns "not found" with leftover
candidates, the user can already click the status cell to open a
modal showing those candidates and pick a different one. This adds
a manual search bar to that modal — type any query, hit search,
get a fresh round of results without having to bail out and start
over from the main search page.
Solves the case where the auto-query was bad (featured artist not
in title, parentheticals like "(Remastered 2019)" tripping the
matcher, slight artist-name variants, transliteration) but the
file genuinely exists on the source.
Frontend (downloads.js)
- Added a manual-search section above the existing auto-candidates
table inside the candidates modal.
- Source picker is smart per download mode:
- Single-source mode (soulseek-only / youtube-only / etc) shows
a "Searching X" label, no dropdown.
- Hybrid mode shows a dropdown with "All sources" default + every
configured source. Picking "All" runs parallel searches across
them and tags each result row with its source badge.
- Only configured sources show up; unconfigured are hidden.
- Validation: button disabled until query length >= 2, "Type at
least 2 characters" hint until threshold crosses.
- Loading state on search button while the request is in flight.
- Manual results render in a separate table above the existing
auto-candidates table, using the same row template (file /
quality / size / duration / user / ⬇ button) so the renderer
helper is shared.
- Click ⬇ reuses the existing `downloadCandidate(taskId, candidate,
trackName)` flow — same retry path, same AcoustID verification
when the file lands, no shortcut around the safety net.
- Re-running the search with a different query replaces the
previous manual results.
Backend (web_server.py)
- Extended `GET /api/downloads/task/<id>/candidates` response with:
- `download_mode` (e.g. 'hybrid', 'soulseek')
- `available_sources` (list of configured source IDs + labels)
- `source` field on each candidate (purely additive — frontend
auto-renderer ignores it on legacy code paths, manual-search
renderer uses it for the badge)
- Added `POST /api/downloads/task/<id>/manual-search`:
- Body: `{ query, source: 'all' | <source_id> }`
- Validates query length (>=2 trimmed) → 400
- Validates source against the configured-sources gate → 400
(rejects unconfigured sources even when explicitly named)
- For 'all': parallel `ThreadPoolExecutor` dispatch across every
configured download source, merged results
- For specific source: just that source
- Returns same shape as `/candidates` so the frontend renderer
is reused
- New module-level helpers: `_STREAMING_SOURCE_NAMES`,
`_infer_candidate_source`, `_serialize_candidate`,
`_list_available_download_sources`. The existing `/candidates`
endpoint also goes through `_serialize_candidate` so the source
badge is consistent across both flows.
Behavior preserved
- Existing modal layout / candidates table / ⬇ button are
byte-identical when the user doesn't use manual search.
- `downloadCandidate()` JS function untouched.
- `/candidates` and `/download-candidate` endpoints
backwards-compatible — only NEW fields added, nothing changed
or removed.
Tests
`tests/test_manual_search_endpoint.py` — 10 tests:
- `test_manual_search_validates_query_length`
- `test_manual_search_validates_source` (whitelist gate)
- `test_manual_search_handles_task_not_found` (404)
- `test_manual_search_dispatches_to_configured_source_only`
- `test_manual_search_all_dispatches_parallel`
- `test_manual_search_skips_unconfigured_sources`
- `test_manual_search_rejects_unconfigured_source_explicitly`
- `test_manual_search_returns_same_shape_as_candidates`
- `test_manual_search_single_source_mode_lists_source` (verifies
`available_sources` reflects the active mode)
- `test_manual_search_isolates_per_source_exceptions` (one source
throwing doesn't kill the merged result)
2242/2242 full suite green (was 2232 + 10 new). Ruff clean.
JS parses clean.
- `_SOULSYNC_BASE_VERSION` 2.4.2 → 2.4.3
- helper.js — flip 2.4.3 WHATS_NEW header to "May 8, 2026 — 2.4.3
release"; bump fallback default from 2.4.2 → 2.4.3
- docker-publish.yml — manual-trigger default tag 2.4.2 → 2.4.3
Drive-by — make sidebar version + version-modal subtitle dynamic.
The sidebar version button (`v2.4.1`) and version-modal subtitle
(`Version 2.4.1 — Latest Changes`) were hardcoded text in the HTML.
2.4.2 shipped without these getting bumped — silent drift, easy to
miss at every release.
Added a Flask context_processor that injects `soulsync_version` and
`soulsync_base_version` into every template, then templated the two
hardcoded values:
v{{ soulsync_base_version }}
Version {{ soulsync_base_version }} — Latest Changes
Now bumping `_SOULSYNC_BASE_VERSION` updates the UI everywhere it's
rendered. No more "I forgot to bump the sidebar" at release.
2232/2232 full suite green. Ruff clean. JS parses clean.
Four selection-quality fixes on the SoulSync-made discover playlists.
None change public method signatures; all are tightenings on what's
already there.
(1) Diversity for Hidden Gems + Discovery Shuffle
Both used to be `RANDOM() LIMIT N` with no diversity. Could return
50 tracks from one artist or 20 from one album if the discovery
pool happened to be skewed. Both now over-fetch 3x and run the
existing `_apply_diversity_filter`:
- Hidden Gems: max 2 per album, 3 per artist
- Discovery Shuffle: max 2 per album, 2 per artist (tighter — shuffle
should feel maximally varied)
(2) Source-aware popularity thresholds
`popularity >= 60` for "Popular Picks" and `popularity < 40` for
"Hidden Gems" was Spotify-shaped (0-100 scale). Deezer writes its
`rank` value into that column (often six-digit integers); iTunes
writes nothing meaningful. For Deezer-primary users:
- Popular Picks pulled essentially everything (rank >= 60 = all)
- Hidden Gems pulled essentially nothing (rank < 40 = none)
New `_get_popularity_thresholds(source)` helper returns per-source
values:
- Spotify: (60, 40) — the existing 0-100 scale
- Deezer: (500_000, 100_000) — ballpark from real rank values
- iTunes / unknown: (None, None) — skip the popularity filter
entirely, fall back to random + diversity
`get_popular_picks` and `get_hidden_gems` now consult the helper.
When threshold is None they skip the popularity SQL filter. Diversity
+ ID gate still apply.
(3) Push genre keyword filter into SQL
`get_genre_playlist` used to fetch `limit=1_000_000` rows into Python
then run a substring keyword filter on `artist_genres`. Bad on big
discovery pools.
Now the keyword OR chain is generated as SQL placeholders:
AND (artist_genres LIKE ? OR artist_genres LIKE ? OR ...)
Each placeholder gets `f'%{keyword.lower()}%'` via `extra_params`.
`fetch_limit` drops back to `limit * 10`. `_genre_matches` Python
helper deleted (only intra-file caller; verified via grep).
Parent-genre expansion via `GENRE_MAPPING` preserved — keywords list
feeds the LIKE chain unchanged.
(4) Filter out tracks already in library
Discovery pool can include tracks the user already owns. Hidden Gems
/ Shuffle / Popular Picks shouldn't surface those.
`_select_discovery_tracks` gained `exclude_owned: bool = True`
parameter. When True, adds a correlated NOT EXISTS subquery against
the `tracks` table covering all 3 source IDs:
AND NOT EXISTS (
SELECT 1 FROM tracks t WHERE
(t.spotify_track_id IS NOT NULL AND t.spotify_track_id = discovery_pool.spotify_track_id)
OR (t.itunes_track_id IS NOT NULL AND t.itunes_track_id = discovery_pool.itunes_track_id)
OR (t.deezer_id IS NOT NULL AND t.deezer_id = discovery_pool.deezer_track_id)
)
Note column-name asymmetry: tracks.deezer_id vs
discovery_pool.deezer_track_id. Inline comment marks the trap. All
5 public discovery methods automatically benefit (default True).
Seasonal Playlist doesn't go through the helper so it's unaffected
(curated content, dedup is wrong intent there).
Tests
12 new tests in `tests/test_personalized_playlists_id_gate.py` (27
total in the file):
- Hidden Gems + Discovery Shuffle apply diversity (cap proven by
inserting 10 same-artist + same-album rows and asserting return
count ≤ per-album cap)
- Popularity thresholds: Spotify (60, 40), Deezer larger scale,
iTunes None / None
- Popular Picks skips threshold filter when None
- Genre playlist pushes filter to SQL (parent + child genre expansion)
- Owned-track exclusion: filtered when match, kept when no match,
opt-out flag works
- Deezer column-name asymmetry pinned (regression footgun)
Test fixture re-added the minimal `tracks` table (4 columns: id,
spotify_track_id, itunes_track_id, deezer_id) — only what the new
NOT EXISTS subquery needs to join. Plus `insert_library_track`
helper.
Verification
- 27/27 in this test file pass (15 prior + 12 new)
- 2232/2232 full suite green
- ruff clean
LOC delta:
- core/personalized_playlists.py: 1030 → 1101 (+71)
- tests/test_personalized_playlists_id_gate.py: 352 → 616 (+264)
Owner decision: not worth shipping. The four library-driven personalized
sections were stubbed returning [] for ages because their schema
prereqs didn't exist; the prior commit re-enabled them by routing
through a new `_select_library_tracks` helper. Owner reviewed and chose
to delete the sections entirely instead.
Removed everywhere:
- `core/personalized_playlists.py` — `get_recently_added`,
`get_top_tracks`, `get_forgotten_favorites`, `get_familiar_favorites`
+ the `_select_library_tracks` helper (no other callers; verified
via grep).
- `web_server.py` — 4 route handlers
(`/api/discover/personalized/recently-added`, `top-tracks`,
`forgotten-favorites`, `familiar-favorites`).
- `webui/index.html` — 4 `<div class="discover-section">` blocks
(`#personalized-recently-added`, `#personalized-top-tracks`,
`#personalized-forgotten-favorites`,
`#personalized-familiar-favorites`).
- `webui/static/discover.js` — 4 load functions
(`loadPersonalizedRecentlyAdded`, `loadPersonalizedTopTracks`,
`loadPersonalizedForgottenFavorites`, `loadFamiliarFavorites`),
plus their entries in `loadDiscoverPage`'s Promise.all, plus
4 module-level state vars + 6 dead branches across
`openDownloadModalForDiscoverPlaylist` / `startDiscoverPlaylistSync`
and the sync-progress / rehydrate dispatchers.
- `webui/static/helper.js` — 4 tooltip / docs entries.
- `webui/static/sync-spotify.js` — 1 stale rehydrate dispatcher
branch (`discover_familiar_favorites`) caught during the global
grep pass.
- `tests/test_personalized_playlists_id_gate.py` — 3 library-method
tests + the test infrastructure that supported them
(`tracks` schema, `insert_library_track` helper). Documentation
header updated to reflect the deletion.
Net: -527 / +2 lines across 7 files.
What stays:
- Daily Mixes (also in personalized package, intentionally paused —
separate decision).
- Popular Picks + Hidden Gems + Discovery Shuffle (alive, not
affected by this deletion).
- All 14 tests in the personalized-playlists test file still pass.
- The PersonalizedPlaylistsService lift from the prior commit
(`_select_discovery_tracks` etc) — those are still in active use
by the surviving discovery_pool methods.
DISCOVER_TRACK_SELECTION_REVIEW.md at repo root contains historical
references to the four deleted endpoints. Treated as historical
context (same policy as WHATS_NEW), left alone.
2219/2219 full suite green (was 2222 - 3 deleted tests = 2219).
JS parses clean, ruff clean.
Both tabbed-browser sections — Time Machine ("Decade") and Browse by
Genre — re-implemented the same lifecycle by hand: fetch tabs list,
render the tab strip, attach click handlers, fetch content per tab,
render track list with sync + download action buttons + sync-status
block, handle empty/error/loading states. ~314 lines of identical
boilerplate split across two browsers.
Lifted into one shared `createTabbedBrowserSection(config)` helper.
Each browser is now a thin wrapper:
```js
const ctrl = createTabbedBrowserSection({
id: 'decade-browser',
tabsContainerEl: '#decade-tabs',
contentContainerEl: '#decade-content',
fetchTabs: async () => { ... },
renderTabButton: (tab, isActive) => `<button>...</button>`,
fetchTabContent: async (tab) => { ... },
renderTabContent: (tracks, tab) => `...`,
onTabContentRendered: (tab, contentEl) => { ... },
emptyMessage / errorMessage,
});
```
Migrated:
- `loadDecadeBrowserTabs` 85 → 3 lines
- `loadDecadeTracks` 67 → 3 lines
- `loadGenreBrowserTabs` 92 → 3 lines
- `loadGenreTracks` 70 → 3 lines
Helper: ~125 lines + ~100 lines of per-browser config blocks +
~25 lines of shared `_renderTabbedTrackList` (the two browsers had
byte-identical track-row markup so it lifted cleanly).
Public function names preserved — the four migrated functions stay
on the same signature so existing callers (`loadDiscoverPage`,
refresh buttons, inline handlers) don't change.
Side effects preserved — `decadeTracksCache[year]`, `activeDecade`,
`genreTracksCache[name]`, `activeGenre`, `availableGenres` still
mutated at the same lifecycle moments. The decade-specific
`startDecadeSync(decade)` and genre-specific `startGenreSync(name)`
sync-button handlers stay where they are; they're click handlers
attached to rendered content, not part of the tab lifecycle.
What didn't fit (intentionally left alone):
- `_renderCompactTrackRow` (the existing shared track-row helper) is
NOT used by the tabbed browsers — they had their own template
with a `track_data_json` fallback chain `_renderCompactTrackRow`
doesn't do. Unifying these two would change behavior for
non-tabbed sections, so the tabbed-browser variant lives as
`_renderTabbedTrackList`. Future cleanup could merge them by
giving `_renderCompactTrackRow` an opt-in fallback flag.
- `switchDecadeTab` / `switchGenreTab` still know about cache shape
so they can skip refetch on already-loaded tabs. Keeping that
in the per-browser switch is fine — it's a click handler, not
lifecycle.
Net: 8546 → 8578 LOC on `discover.js` (+32). Helper boilerplate
offsets the line count, but the win is single-source-of-truth, not
raw line reduction.
`node --check` clean. 2222/2222 full suite green.
Three changes tightening the controller before opening the PR.
DROP MAGIC `extractItems` DEFAULTS
Controller used to auto-pull `data.items` / `data.albums` /
`data.artists` / `data.tracks` / `data.results` when no extractor
was supplied. Removed the fallback chain — every section now MUST
provide an explicit `extractItems(data) => array`. Validated at
register-time so misuse fails immediately, not silently on first
load against an endpoint that happened to return two arrays.
Cin standard: explicit > implicit. Magic key-grabbing could pick
the wrong one in edge cases (e.g. an endpoint returning both
`data.albums` and `data.results` would have grabbed albums when
the section actually wanted results).
All 10 existing controller call sites already passed explicit
extractors, so no migration churn — this is purely tightening the
contract for future sections.
REPLACE `renderItems` NULL-RETURN CONVENTION WITH `manualDom: true`
Your Albums and similar sections that delegate to existing renderers
that target a CHILD element of `contentEl` used to signal "leave the
container alone" by returning null/undefined from `renderItems`. That
convention is easy to confuse with an accidental missing-return error.
Replaced with an explicit `manualDom: true` config flag. Renderer is
still called for its side-effects, controller just skips the innerHTML
swap. Clearer intent at the call site. Updated `loadYourAlbums` to
use the new flag.
PIN THE CONTROLLER CONTRACT WITH JS TESTS
Added `tests/static/test_discover_section_controller.mjs` — 32 tests
covering the controller's lifecycle contract:
- Config validation (every required field, mutual exclusivity of
fetchUrl/data, type checks on contentEl)
- Happy-path fetch → parse → render
- Empty state (default empty render, hideWhenEmpty + sectionEl,
success=false treated as empty, custom isSuccess override)
- Stale state (fires when isStale returns true, wins over empty,
custom renderStale override)
- Error state (HTTP non-ok, fetch throws, showErrorToast fires
window.showToast, default off doesn't fire)
- No-fetch `data:` mode (value + function form, doesn't call fetch)
- manualDom mode (skips innerHTML swap, still calls renderer)
- Callable `fetchUrl` (resolved at load time, refresh re-resolves)
- Load coalescing (concurrent loads share one fetch)
- Refresh bypasses coalescing (re-fires fetch every call)
- Hook error containment (throwing renderer/onSuccess hooks don't
crash the controller)
Runs via Node's stable built-in `--test` runner — no package.json,
no jest/vitest dependency, no compile step. Just `node --test`.
Pytest wrapper at `tests/test_discover_section_controller_js.py`
shells out to node and asserts clean exit, so the JS tests fail
the regular pytest sweep if the controller contract drifts.
Skipped gracefully when node isn't available or is < 22.
Closes the "controller is a contract, pin it at the test boundary"
gap that Cin would have flagged on review.
VERIFICATION
- 2205/2205 full pytest suite green (was 2204 + 1 new wrapper)
- 32/32 `node --test` pass on the controller test file directly
- ruff clean
- node --check clean on all touched JS files
Follow-up to the controller migration commits. Closes out the
extension list the per-section migrations surfaced as needed.
CONTROLLER EXTENSIONS
- Callable `fetchUrl: () => string` — resolves the seasonal-playlist
recreate-on-key-change hack from the prior commit.
- No-fetch `data:` mode — value or `() => value`. Lets render-only
sections like Seasonal Albums use the controller without inventing
a fake endpoint. Mutually exclusive with `fetchUrl`; validated up
front so misuse fails at register-time.
- `beforeLoad(ctx)` hook — runs before the spinner shows. Lets
dynamically-inserted sections like Because You Listen To ensure
their `contentEl` exists before the visibility check.
- `onSuccess(data, ctx)` hook — runs after the success gate but
before isEmpty / isStale. Cleaner home for sibling header /
subtitle / button updates than folding them into renderItems.
- `isStale(items, data)` + `onStale(ctx)` + `renderStale(items, data)`
+ `staleMessage` — third render state for "data is empty BUT
upstream is still discovering". Stale wins over empty when both
apply. Default stale UI is the same spinner block used elsewhere.
- `showErrorToast: true` config — opens a global `showToast(...)` in
addition to the in-section error block. Default off; sections that
have no recovery action shouldn't shout at the user.
- `renderItems` returning null/undefined now leaves contentEl
untouched. Lets a renderer do its own DOM manipulation (e.g.
delegating to an existing grid-render fn that targets a child
element) without fighting the controller's innerHTML swap.
MIGRATED THE 2 SKIPPED SECTIONS
- `loadYourAlbums` — uses `isStale`/`onStale`/`renderStale` for the
stale-fetch state, `onSuccess` for the subtitle/filters/download
side-effects, `hideWhenEmpty` + `sectionEl` for the truly-empty
case, `renderItems` returning null since it delegates to the
existing `_renderYourAlbumsGrid` + `_renderYourAlbumsPagination`.
- `loadSeasonalAlbums` — uses no-fetch `data:` mode because the
parent `loadSeasonalContent` already fetched the season payload.
`beforeLoad` updates the sibling title/subtitle text.
ERROR TOASTS ON ALL MIGRATED SECTIONS
Every migrated section now has `showErrorToast: true`. Section load
failures surface a global toast instead of silently spinning forever
or swallowing into console.debug. Same pattern JohnBaumb #369 asked
for at the Python layer, applied at the UI layer.
SHARED SYNC-STATUS BLOCK
Lifted the duplicated decade-tab + genre-tab sync-status HTML
(✓ completed | ⏳ pending | ✗ failed | percentage) into a single
`_renderSyncStatusBlock(idPrefix)` helper. Two call sites now share
one implementation. ListenBrainz playlists keep their own block
because the semantics differ — matching progress (total / matched /
failed) vs download progress.
DEAD-SECTION AUDIT — NONE DEAD
Audited the 13 supposedly-dead hidden sections from
DISCOVER_REVIEW.md. All 13 are alive: gated on user data (discovery
pool, library content, metadata cache) and self-surface when their
data exists via `style.display = 'block'` on the success path. The
review's grep missed the toggle. No deletions made.
DAILY MIXES ORPHAN CALL
Removed the orphaned `loadPersonalizedDailyMixes()` call from
`blockDiscoveryArtist` — Daily Mixes is intentionally paused (its
load call in `loadDiscoverPage` is commented out) so refreshing it
from the post-block hook was a no-op.
2204/2204 full suite green. JS parses clean (`node --check`).
Follow-up to the foundation commit. Drops the hand-rolled
try/catch + spinner injection + empty-state HTML + error-swallow
in seven sections by routing them through
`createDiscoverSectionController`. Each section keeps its existing
public function name + signature so callers, refresh buttons, and
dashboard wiring don't notice the swap.
Migrated:
- `loadDiscoverReleaseRadar` (Fresh Tape)
- `loadDiscoverWeekly` (The Archives)
- `loadDecadeBrowser` (Time Machine intro carousel)
- `loadGenreBrowser` (Browse by Genre intro carousel)
- `loadSeasonalPlaylist` (Seasonal Mix)
- `loadYourArtists`
- `loadBecauseYouListenTo`
Skipped (don't fit the controller's single-fetch / single-render-target
shape):
- `loadYourAlbums` — paginated grid + filters, updates four separate
UI elements (subtitle, filter chips, download button, grid).
- `loadSeasonalAlbums` — receives pre-fetched data from
`loadSeasonalContent`; no fetch URL to satisfy.
Hidden / dead sections (~13 of them — `loadPersonalized*`,
`loadDiscoveryShuffle`, `loadFamiliarFavorites`, `loadCache*`)
untouched in this pass. Separate audit commit will surface or kill
them.
Two side-effects worth noting:
- `loadDecadeBrowser` and `loadGenreBrowser` migrated for
completeness, but neither appears wired into `loadDiscoverPage` or
any inline handler. May be dead code — flagged for the audit pass.
- `loadSeasonalPlaylist` needs a per-load fetch URL (varies by
`currentSeasonKey`); worked around by recreating the controller
when the key changes. Cleaner option: extend the controller to
accept a `fetchUrl: () => string` callable form. Tracked in the
follow-up extension list below.
Controller extension candidates surfaced for follow-up:
- Callable `fetchUrl` (resolves the seasonal playlist
recreate-on-key-change hack)
- Explicit `isStale` / `onStale` hook (so Your Artists doesn't
fold stale handling into renderItems)
- `beforeLoad` / `ensureContentEl` hook (so Because You Listen To
can let the controller own the dynamic container creation)
- No-fetch `data:` mode (so render-only sections like Seasonal
Albums can use the controller too)
- `onSuccess(data)` hook (cleaner home for header / subtitle
side-effects vs folding them into renderItems)
Net: -76 lines in `discover.js` even after adding the per-section
render helpers. 2204/2204 full suite green. JS parses clean.
Every section on the discover page (Recent Releases, Your Artists,
Your Albums, Seasonal Albums, Seasonal Mix, Fresh Tape, The Archives,
Build Playlist, Time Machine, Browse by Genre, ListenBrainz Playlists,
Because You Listen To, plus ~13 hidden sections) currently
re-implements the same lifecycle by hand:
1. show a loading spinner in the carousel container
2. fetch the section's endpoint
3. parse the response, decide if the data is empty
4. either render the items, show an empty-state, or show an error
5. wire post-render handlers (download buttons, hover behavior, etc)
6. maybe expose refresh()
~30 sections worth of duplicated boilerplate, all subtly drifting.
Different empty-state messages. Different error handling (some
`console.debug`, some silently swallowed, some leave the spinner
spinning forever). Different sync-status icons (✓/⏳/✗ vs ♪/✓/✗).
No consistent error toast.
Lifted the lifecycle into a shared `createDiscoverSectionController`
in `webui/static/discover-section-controller.js`. Renderers stay
per-section because section data shapes legitimately differ — album
cards vs artist circles vs playlist tiles vs track rows. The
controller is the wrapper, not a forced visual abstraction.
Foundation contract:
createDiscoverSectionController({
id: 'recent-releases', // for diagnostic logging
contentEl: '#carousel', // selector or Element
fetchUrl: '/api/discover/...',
extractItems: (data) => [...], // pull list from response
renderItems: (items, data, ctx) => '<html>',
onRendered: (ctx) => { ... }, // optional post-render hook
loadingMessage / emptyMessage / errorMessage: copy
sectionEl + hideWhenEmpty: optional whole-section visibility
isSuccess / isEmpty: optional gate overrides
})
Returns `{ load, refresh, destroy, getState }`. Validates config up
front so misuse fails at register-time, not silently on load. Coalesces
concurrent loads (same in-flight promise returned) so a double-click
or repeated trigger doesn't double-fetch. `refresh()` bypasses the
coalesce so the refresh button always re-fires. Errors are logged
(console.debug by default, console.error when verboseErrors=true).
Renderer hook errors are caught + logged so a buggy render callback
can't tear down the controller — keeps the page resilient.
Migrated `Recent Releases` as the proof — simplest album-card shape,
no source-gating, no refresh button. Verified the contract covers it
end-to-end. The legacy `loadDiscoverRecentReleases` entry-point stays
public so existing callers don't change; internally it lazy-builds
the controller and triggers `load()`.
NOT in this commit:
- Other section migrations (one section per follow-up commit, keeps
reviews small + lets us sequence the work)
- Registry-driven section list (so the dead-section audit becomes
registry deletions instead of section-by-section removal)
- Global error toast wrapper
- Per-section "requires X primary source" gate
- Sync-status icon renderer unification
Once every section is on the controller, the discover-page cleanup
work (kill the 13 dead sections, standardize sync-status icons, add
error toasts) becomes single-line registry-level edits instead of
30 separate section-by-section rewrites.
2204/2204 full suite green. JS parses clean (`node --check`). Manual
smoke deferred until follow-up commits — Recent Releases unchanged
on the wire (same endpoint, same payload shape, same render output).
- `web_server.py` — `_SOULSYNC_BASE_VERSION` 2.4.1 → 2.4.2
- `webui/static/helper.js` — flip the 2.4.2 WHATS_NEW header from
"Unreleased — 2.4.2 dev cycle" to "May 7, 2026 — 2.4.2 release"
so the per-version block stops being filtered out by
`_getLatestWhatsNewVersion`. Also bumps the safety-net default
inside that helper from 2.4.1 → 2.4.2.
- `.github/workflows/docker-publish.yml` — manual-trigger default
tag bumped to match.
Drive-by fix: escaped a stray single quote in the `Internal: Download
Engine` 2.4.2 entry that broke `node --check` on the file
(`orchestrator.client('soulseek')` inside a single-quoted desc string
silently terminated the string mid-entry). Pre-existing, unrelated to
the bump but caught while validating JS parse for the release.
VERSION_MODAL_SECTIONS not rotated in this commit — separate
editorial pass.
Closes#513 (s66jones).
The artist detail page already showed a "Popular on Last.fm" sidebar —
list of an artist's top tracks by playcount, with a play button per row
but no download action. Issue #513 wanted a way to grab those tracks
the same way zotify let users grab "top X songs" without pulling the
full discography.
Pulls from the configured primary metadata source (Spotify
`artist_top_tracks`, Deezer `/artist/{id}/top`) when available, falls
back to the existing Last.fm display-only mode for sources that don't
expose popularity ranking (iTunes / Discogs / MusicBrainz). Source
label in the section title shifts to match.
Each row gets a hover-revealed download button that wishlists the
single track via the existing /api/add-album-to-wishlist endpoint
(preserves the track's real album metadata, so the wishlist worker
later places the file in its proper album folder).
A "Download All" footer button opens the standard download modal in
PLAYLIST context, not album context — the virtual playlist_id is
`top_tracks_<source>_<artistId>` which doesn't match any of the
album-prefix checks in `startMissingTracksProcess` (downloads.js).
That keeps `is_album_download=false`, so the master worker doesn't
inject a wrapper context as `_explicit_album_context`. Each track
downloads using its own real album metadata, files land in proper
per-album folders on disk (not a fake "Top Tracks" folder).
Backend additions:
- `SpotifyClient.get_artist_top_tracks(artist_id, country, limit)` —
wraps `spotipy.artist_top_tracks`, returns up to 10 tracks for the
market (Spotify's API cap). UI-side limit trim only.
- `DeezerClient.get_artist_top_tracks(artist_id, limit)` — wraps
`/artist/{id}/top?limit=N`, converts Deezer's raw shape to the same
Spotify-compatible dict layout (id, name, artists, album with
album_type / total_tracks / images, duration_ms, track_number,
disc_number) so downstream code doesn't branch on source.
- `GET /api/artist/<id>/top-tracks` — dispatches to whichever client
matches the primary source. Resolves per-source artist IDs from the
DB row first (matching what /discography already does) so a Spotify
ID in the URL still works when Deezer is primary, and vice versa.
Returns `{success, source, tracks, resolved_artist_id}` on hit;
`{success: False, reason: 'unsupported_source' | 'spotify_not_authenticated'
| 'deezer_unavailable' | 'no_tracks_found'}` on miss so the frontend
can decide whether to fall through to Last.fm.
Frontend:
- `_loadArtistTopTracks` tries the metadata source first, falls
through to the legacy `/api/artist/0/lastfm-top-tracks` call if the
source can't deliver. Section title and per-row UI shift based on
which source answered.
- New per-row `.hero-top-track-download` button (hover-revealed).
- New `.hero-top-tracks-download-all` footer button — only visible
when metadata-source mode rendered the list (Last.fm fallback hides
it since rows have no track IDs to download).
Tests: 10 new tests pin the client methods —
- Spotify: returns track list, honors UI limit cap, returns empty when
unauthed / artist_id missing / API throws.
- Deezer: shape conversion to Spotify-compatible dict, empty when no
data / artist_id missing, limit clamping at upper bound, default
fallback when limit=0, malformed entries skipped.
The Flask endpoint dispatcher itself isn't covered by the new test
file because importing web_server at test-collection time spins up
worker threads that race with caplog-using tests elsewhere in the
suite (specifically test_library_reorganize_orchestrator). Endpoint
verified manually; the underlying client methods (the load-bearing
logic) are covered.
2204/2204 full suite green (was 2194 + 10 new).
Discord report (corruption [BWC]): downloads coming through as the
instrumental cut when a vocal track was requested. The verification
step's `_normalize` function strips parentheticals and version-suffix
tags ("(Instrumental)", "- Live", etc) so legitimate name variations
don't false-fail the title-similarity check. That also means "In My
Feelings" and "In My Feelings (Instrumental)" both normalize to "in
my feelings", title similarity is 1.0, and the wrong cut passes
verification.
Detect the version label on each side BEFORE normalization runs. If
the expected and matched recordings disagree on version (one is
original, the other is instrumental / live / acoustic / remix /
etc), return FAIL — the fingerprint identified a real song, just
not the version the caller asked for.
Reuses `MusicMatchingEngine.detect_version_type` so the same regex
patterns the pre-download Soulseek matcher applies also drive
post-download verification. No duplicated tables.
Also gates the secondary fallback scan, so a wrong-version variant
sitting in the same fingerprint cluster can't win the loop after
the best match has already been version-rejected.
6 tests pin the behavior:
- instrumental returned for vocal request → FAIL
- vocal returned for instrumental request → FAIL
- live vs acoustic → FAIL
- matching versions on both sides → PASS
- original-to-original happy path → PASS (regression guard)
- secondary scan skips wrong-version recordings → not PASS
2194/2194 full suite green (was 2188 + 6 new).
Closes#515 (jaruca).
Search-picker controller in shared-helpers.js resolved the user's
configured primary metadata source by fetching `/api/settings`. That
endpoint is `@admin_only` (it returns full config including
credentials), so non-admin profiles got a 403 and the controller
silently fell back to the hardcoded `'spotify'` default — admin's
chosen source (deezer / itunes / discogs / etc) was ignored on every
non-admin profile, forcing manual reselection each session.
Switched to `/status`, which is public and already exposes the
resolved `metadata_source` for the dashboard. Same value the picker
needs — different endpoint that doesn't gate non-admins.
Admins see no behavior change. Non-admins now see admin's configured
primary source as the default active icon.
Refs #515
Catches the silent excepts the awk-based earlier sweeps missed:
- Bare `except:` followed by `pass` (also swallows KeyboardInterrupt
and SystemExit — actively wrong). Upgraded to `except Exception as
e: logger.debug("...: %s", e)`. ~14 sites across connection_detect,
soulseek_client, listenbrainz_manager, watchlist_scanner,
youtube_client, navidrome_client, jellyfin_client, web_server.
- `except Exception:` + pass that the awk pattern missed (e.g.
multi-line or unusual whitespace). ~31 sites across automation_engine,
database_update_worker, music_database, spotify_client, web_server,
others.
- 14 legitimate cleanup sites left silent with explicit `# noqa: S110`
+ comment explaining why (atexit handlers, finally-block conn.close
calls). Logging during shutdown can itself crash because file handles
get torn down before the handler fires.
Also enables `S110` rule in pyproject.toml so this pattern fails CI
going forward — drift fails at PR review instead of at runtime against
a wedged worker thread. Tests path keeps S110 ignored (test fixtures
legitimately use try-except-pass for cleanup).
Adds a WHATS_NEW entry to helper.js summarizing the full #369 sweep.
Verified: `python -m ruff check .` → All checks passed.
Verified: `python -m pytest tests/` → 2188 passed.
Closes#369
Discord report: Duplicate Detector card said "372 findings" and Cover
Art Filler said "60 findings", but clicking the Findings tab's Pending
filter showed 0. User read it as "findings aren't being created" —
looked like a detector bug.
Actual cause: the badge sourced ``last_run.findings_created``
(historical "found in last scan") without considering current state.
After the user (or bulk-fix automation) resolved or dismissed those
findings, they no longer appeared on the Pending tab — but the badge
kept showing the last-scan number in red urgent styling.
Backend was correct end-to-end: detectors create pending rows,
bulk-fix moves them to resolved, Findings tab filters by status.
Only the badge display lied about current state.
Fix:
- ``RepairWorker._get_pending_count_by_job()`` — single SQL aggregation
returning ``{job_id: pending_count}`` for every job with pending
findings. O(1) lookup per job instead of N round trips.
- ``get_all_job_info()`` calls it once per request and adds
``pending_findings_count`` to each job's API response.
- ``enrichment.js`` job card now branches on the count:
- ``> 0`` → red ``"X pending"`` badge (urgent, action needed)
- ``= 0`` AND last scan found something → muted grey ``"X found in
last scan"`` (historical context, no action needed)
- New CSS class ``.repair-flow-badge.findings-historical`` for the
muted slate color so the two states are visually distinct.
User-visible result with the screenshotted state (372 dup / 60 cover-
art findings, all resolved):
- Before: red "372 findings" / "60 findings" — implied 432 things to
do, but Findings tab showed 0 pending
- After: grey "372 found in last scan" / "60 found in last scan" —
the badge text tells the user the count is historical, no surprise
when Pending is empty
Tests: 3 new tests in ``tests/test_create_finding_dedup_counter.py``
pin the per-job pending count helper:
- returns ``{job_id: count}`` based on status='pending' rows only;
resolved + dismissed rows excluded
- empty dict when no pending findings exist
- gracefully returns ``{}`` on DB error (badge falls back to
historical count via the existing JS ``or 0`` safety)
2188/2188 full suite green. Pure UI/state-display fix — no detector
logic, no backend behavior change.
GitHub issue #499 (@bafoed). Big initial sync of Spotify playlists
worked for 2-3 hours then downloads silently stopped:
- 3 active tasks stuck in "Searching" state, replaced every ~10 min
by different ones
- slskd UI showed no actual searches happening
- Debug log: orphaned-task count grew over time, no jobs executed
- Container restart was the only fix (bought another 2-3 hours)
- Not a rate limit (rates showed 0/min)
Root cause: ``core/soulseek_client.py`` constructed
``aiohttp.ClientSession()`` with no timeout at four sites. When slskd
hung on a request (overloaded, transient network blip, internal
stall), the HTTP call blocked indefinitely — and the worker thread
blocked with it. The download executor only has
``ThreadPoolExecutor(max_workers=3)``, so once 3 worker threads were
wedged on hung calls, no further downloads could start.
Batch-level "stuck detection" (10-minute timer in
``check_batch_completion_v2``) was correctly marking tasks
``not_found`` and trying to start replacements, but the executor pool
was exhausted — replacements queued forever inside the executor with
no thread to run them. Symptom: tasks rotating every ~10 min at the
batch level while the underlying executor stayed wedged.
Fix: bounded ``aiohttp.ClientTimeout`` (total 120s, connect 15s,
sock_read 60s) on every slskd ``ClientSession`` construction. Module-
level constant ``_SLSKD_DEFAULT_TIMEOUT`` so the four sites stay in
lockstep — future sites get the same protection by reusing the
constant.
Why these timeouts are safe:
- Every slskd API call is metadata-level (search submission, status
polls, download enqueue, transfer state queries). None stream
files — slskd handles file transfer via its own peer-to-peer
infrastructure entirely outside our HTTP requests.
- Legitimate metadata calls finish in seconds. 120s ceiling is
~50× the normal latency.
Timeout handling:
- ``asyncio.TimeoutError`` caught explicitly BEFORE the generic
``except Exception`` — surfaces "slskd timed out" specifically in
logs (debuggable instead of buried as "Error making API request").
- Returns None to the caller (same code path as a 5xx response or
any other failure). No new error path; callers already handle
None as "request failed".
- Worker thread unblocks immediately → executor pool stays healthy
→ downloads keep flowing.
Sites updated:
- ``_make_request`` (general /api/v0/ helper, line 152) — used for
every slskd API operation
- ``_make_direct_request`` (non-/api/v0/ helper, line 235)
- ``_explore_api_endpoints`` Swagger fetch (line 1566) — diagnostic
- ``_explore_api_endpoints`` per-endpoint probe (line 1617) —
diagnostic
Tests: 3 new tests in ``tests/downloads/test_soulseek_pinning.py``
pin:
- ``_SLSKD_DEFAULT_TIMEOUT`` is bounded (total set, ≤300s ceiling,
connect ≤60s) — guards against future regressions that drop or
unbound the timeout
- ``_make_request`` returns None on ``asyncio.TimeoutError`` rather
than raising — pins the caller contract
- ``_make_direct_request`` returns None on ``asyncio.TimeoutError``
2185/2185 full suite green.
Closes#499.
GitHub issue #500 (@bafoed). Library Reorganize repair job moved
album tracks to single-template paths because of a fragile
classification heuristic. Concrete symptom: a track at
``Surf Curse/Surf Curse - Nothing Yet (2017)/01 - Christine F.flac``
got proposed for a move to
``Surf Curse/Surf Curse - Christine F/Surf Curse - Christine F.flac``
(single template) instead of staying under the album folder.
Root cause: the job had its own tag-reading + transfer-folder-walk +
template-application implementation. The classification was
``is_album = (group_size > 1)`` where ``group_size`` was the count
of same-album tracks currently sitting in the transfer folder being
scanned. Two failure modes:
- only one track of an album was in the transfer folder (rest already
moved to the library, or not yet downloaded), or
- album tags varied slightly across tracks (e.g. ``"Buds"`` vs
``"Buds (Bonus)"``)
Either case gave a 1-element group → routed through the SINGLE
template → wrong destination.
Rewrite — delegate to the per-album planner the artist-detail
"Reorganize" modal already uses:
- ``core.library_reorganize.preview_album_reorganize`` for path
computation (DB-driven, knows the album has N tracks regardless of
how many sit in transfer; album-vs-single is structurally correct)
- ``core.reorganize_queue.enqueue_many`` for apply mode; the queue
worker dispatches via ``reorganize_album`` which handles file move
+ post-processing + DB update + sidecar through the same code path
the per-album modal uses
Job's per-album loop:
- iterate albums for the active media server only (matches the artist-
detail modal's scope; multi-server users won't have the job touch
the inactive server's files at paths they can't see)
- preview each album, catch exceptions per-album so one bad row
doesn't abort the scan
- branch on planner status:
- ``no_album`` / ``no_tracks`` (race: album deleted mid-scan) →
skip silently
- ``no_source_id`` (album never enriched) → emit ONE album-level
"needs enrichment first" finding (vs N per-track findings cluttering
the UI)
- ``planned`` → filter mismatched tracks (matched + new_path +
not unchanged + file_exists), emit per-track findings (dry-run)
or collect album for bulk enqueue (apply)
- bulk enqueue at end of loop using the queue's correct return-shape
(``{'enqueued': N, 'already_queued': M, 'total': K}``)
What's gone (~500 LOC):
- ``_read_tag_metadata`` / ``_get_audio_quality`` / transfer-folder walk
- ``_load_album_years`` / ``_lookup_years_from_api`` (planner does this)
- ``_apply_path_template`` / ``_build_path_from_template``
- direct ``shutil.move`` + sidecar move logic (queue handles)
- the fragile ``is_album = group_size > 1`` heuristic — structurally gone
- ``move_sidecars`` setting (no longer applicable; queue's post-process
re-downloads cover art at the destination)
What stays:
- dry-run vs apply toggle
- ``file_organization.enabled`` gate
- stop / pause respect
- progress reporting
- findings for the UI
Cleaner separation of concerns:
- this job: DB-known tracks at wrong paths (active server only)
- ``orphan_file_detector``: files on disk with no DB entry
- ``dead_file_cleaner``: DB entries pointing to nonexistent files
Tests: 12 tests in ``tests/test_library_reorganize.py`` pin the
delegation contract — every status branch, every track-filter case,
exception handling, apply-mode enqueue payload, active-server scope,
estimate-scope shape. Three obsolete ``_lookup_years_*`` tests removed
(year handling moved to planner).
Closes#500 (the misclassification half — orphan + dead-file are
downstream sync-gap symptoms, separate concern).
GitHub issue #501 (@Tacobell444). After manually matching an album to
a specific source ID via the match-chip UI, clicking "Enrich" on that
album would fuzzy-search by name and overwrite the manual match with
whatever the search returned — or revert the match status to
``not_found`` if name search missed. Reorganize then read the now-
wrong ID and moved files to the wrong destination.
Root cause was in the per-source enrichment workers'
``_process_*_individual`` methods. Several workers (Spotify, iTunes)
ran search-by-name unconditionally with no check for an existing
stored ID. Others (Deezer, Tidal, Qobuz) skipped on existing-ID but
without refreshing metadata — preserved the ID but didn't actually
honor the user's intent of "use this match to pull fresh data".
Cin-shape lift: same fix needed in 5 workers, so extracted the shared
behavior into ``core/enrichment/manual_match_honoring.py``:
honor_stored_match(
db, entity_table, entity_id, id_column,
client_fetch_fn, on_match_fn, log_prefix,
) -> bool
Per-worker variability (DB column name, client fetch method, response
shape) plugs in via callbacks. Workers call the helper at the top of
``_process_album_individual`` / ``_process_track_individual``; if it
returns True, the manual match was honored and the search-by-name
fallback is skipped. If False (no stored ID, fetch failed, or empty
response), the worker's existing search-by-name flow runs as before.
Workers wired:
- spotify_worker — album + track (was overwriting; now honors)
- itunes_worker — album + track (was overwriting; now honors)
- deezer_worker — album + track (was skip-on-id; now refreshes)
- tidal_worker — album + track (was skip-on-id; now refreshes)
- qobuz_worker — album + track (was skip-on-id; now refreshes)
Workers left alone (already correct):
- discogs_worker — already had inline stored-ID fast path that
refreshes metadata. Same behavior, just inline; refactoring to use
the shared helper would be churn for zero behavior change.
- audiodb_worker — same — inline fast path with full metadata refresh.
- musicbrainz_worker — preserves existing MBID and marks status,
which is the correct behavior for MB (the MBID itself is the match
payload — no separate metadata fetch).
- lastfm_worker / genius_worker — name-based services with no
source-specific IDs to honor. Inherent re-search per call.
Reorganize fixed indirectly — it always honored stored IDs correctly
via ``library_reorganize._extract_source_ids``. The "Reorganize broken"
symptom was downstream of broken Enrich corrupting the stored ID.
Tests:
- ``tests/enrichment/test_manual_match_honoring.py`` — 11 tests
pinning the shared helper contract: stored-ID fast path, no-ID
fallthrough, empty-string treated as no ID, missing row, fetch
exception caught and falls through, fetch returns None falls
through, callback exceptions propagate, configurable table +
column, defensive table-name whitelist.
- Per-worker wiring NOT tested individually — the workers depend
on live DB / client objects that are heavy to mock. The shared
helper's contract is pinned; per-worker call sites are short
enough to verify by code review.
2173/2173 full suite green.
Closes#501.
GitHub issue #503 (@hadshaw21). Adding a HiFi instance via downloader
settings popped up ``no such table: hifi_instances`` even though
"Test Connection" and "Check All Instances" both worked.
Root cause: ``MusicDatabase._initialize_database`` runs every
``CREATE TABLE`` + every migration step inside one sqlite transaction.
Python's sqlite3 module doesn't autocommit DDL by default, so if any
later migration step throws on a user's specific DB shape (e.g. an
old volume from a prior SoulSync version with quirky schema state),
the WHOLE batch rolls back — including the ``hifi_instances`` CREATE
that ran earlier in the function. The user's next boot retries init,
hits the same migration failure, rolls back again. The ``hifi_instances``
table never lands no matter how many restarts.
Fix: defensive lazy-create. New ``_ensure_hifi_instances_table(cursor)``
helper runs ``CREATE TABLE IF NOT EXISTS`` on demand, called immediately
before every CRUD operation that touches ``hifi_instances``:
- ``get_hifi_instances`` / ``get_all_hifi_instances`` (read)
- ``add_hifi_instance`` / ``remove_hifi_instance`` (CRUD)
- ``toggle_hifi_instance`` / ``reorder_hifi_instances`` (CRUD)
- ``seed_hifi_instances`` (defaults seed)
Idempotent — costs one no-op CREATE check when the table is already
present, fully recovers from a broken init state. Read methods now
return empty instead of raising when init failed; write methods work
end-to-end.
Doesn't paper over the underlying init issue (still worth tracking
which migration step breaks for which user DB shapes — separate
concern) but makes HiFi instance management self-healing in the
meantime.
Tests:
- 7 obsolete tests that pinned ``raises sqlite3.OperationalError``
removed — that contract is no longer correct
- 7 new tests pin the lazy-create behavior: every CRUD method works
against a DB that's missing the ``hifi_instances`` table, verifying
the table gets created and the operation completes
2162/2162 full suite green. Pure additive — no behavior change for
users with a healthy DB; affected users get back to working hifi
instance management.
Closes#503.
Followup to the all-libraries-mode commit. Without dedup, a Plex Home
family where two users both have "Drake" in their music libraries
would see "Drake" twice in SoulSync's library list — Plex returns
distinct ratingKeys for each section's copy of the same artist.
Dedup design — applied selectively, NOT everywhere:
- ``_dedupe_artists(artists)``: groups by lowercased title, picks
the canonical entry by ``leafCount`` (more tracks wins). Active
ONLY in all-libraries mode; single-library mode is a no-op fast
path with zero behavior change.
- ``_dedupe_albums(albums)``: same but keys on
(lowercased parentTitle, lowercased title) so two artists with
identically-titled albums (e.g. self-titled releases) stay
separate.
Applied to:
- ``get_all_artists()`` — public listing for the library view
- ``get_library_stats()`` — count matches what user sees in the list
Deliberately NOT applied to:
- ``get_all_artist_ids()`` / ``get_all_album_ids()`` — these feed
removal detection (compare returned ratingKey set against DB-linked
ratingKeys to decide what's been removed). Deduping here would falsely
flag non-canonical ratingKeys as "removed" and prune SoulSync's DB
tracks that are linked to them. Pinned by two CRITICAL tests.
- ``_all_tracks()`` — track count stays raw because the same track
in two sections IS two distinct files / Plex entries, not a logical
duplicate.
- ``_search_general()`` and ``search_tracks`` Stage 1/2 — search
results stay raw so cross-section matches aren't lost. Stage 1
may miss cross-section tracks for the same artist but Stage 2's
server-wide track search catches them.
Logging: when raw vs deduped artist counts differ, ``get_all_artists``
logs both so users can see "Found 4697 artists across all music
sections (4521 unique after cross-section dedup)" — surfaces the
overlap clearly.
Tests: 8 new tests in test_plex_all_libraries.py pin:
- canonical pick by leafCount (artists + albums)
- case-insensitive name match
- single-library no-op path (zero behavior change for those users)
- album dedup keys on (artist, title) so different-artist same-title
albums stay separate
- ``get_all_artists`` listing applies dedup
- ``get_all_artist_ids`` does NOT dedup (CRITICAL — removal detection)
- ``get_all_album_ids`` does NOT dedup (CRITICAL — removal detection)
- ``get_library_stats`` uses deduped counts for artists/albums but
raw count for tracks
Existing pre-stat test updated to use distinct mock instances —
``[MagicMock()] * 5`` creates five references to one mock which now
correctly collapses under dedup.
71/71 media_server tests green, 2162/2162 full suite green.
Honest known limitation acknowledged in WHATS_NEW + version modal:
write-back (genre / poster / metadata updates) targets one
ratingKey at a time — only updates the canonical section's copy of
an artist if it exists in multiple. Other section's copy stays
unchanged. Document and revisit if it matters.
GitHub issue #505 (PopeBruhLXIX): users with multiple Plex music
libraries (e.g. one per Plex Home user, or two folder roots split
across separate library sections) only saw one library inside SoulSync
because the connection settings forced you to pick a single library
section. SoulSync's PlexClient stored exactly one ``self.music_library``
section reference and every read scanned only that one.
This change adds an opt-in "All Libraries (combined)" dropdown option
that flips the client into a server-wide read mode where every read
method (``get_all_artists`` / ``get_all_album_ids`` /
``search_tracks`` / ``get_library_stats`` / etc) dispatches through
``server.library.search(libtype=...)`` instead of querying a single
section. One Plex API call replaces N per-section iterations; Plex
handles the aggregation server-side.
Implementation:
- ``ALL_LIBRARIES_SENTINEL`` (``'__all_libraries__'``) — module-level
constant used as the saved DB preference value when the user picks
the synthetic "All Libraries" entry. Detection is one string compare
in ``_find_music_library`` / ``set_music_library_by_name``. Existing
preferences (real library names) are unaffected.
- ``self._all_libraries_mode`` (private flag) + ``is_all_libraries_mode()``
(public accessor for external callers). When True, ``music_library``
may stay None — ``is_fully_configured()`` recognizes the mode and
still returns True so dispatch sites don't bail.
- New private helpers ``_can_query``, ``_get_music_sections``,
``_all_artists``, ``_all_albums``, ``_all_tracks``, ``_search_general``,
``_search_artists_by_name``. Single dispatch point for the
section-vs-server branch — every read method funnels through them
so future drift fails at one place.
- New public helpers for downstream callers:
- ``get_recently_added_albums(maxresults, libtype)`` — used by
DatabaseUpdateWorker's deep-scan recent-content sweep
- ``get_recently_updated_albums(limit)`` — same
- ``get_music_library_locations()`` — returns folder roots, used
by web_server.py's file-path resolver
- ``trigger_library_scan`` and ``is_library_scanning`` fan out across
every music section in all-libraries mode.
- ``get_available_music_libraries`` prepends a synthetic
``{'title': 'All Libraries (combined)', 'value': sentinel}`` entry
ONLY when more than one music library exists. Single-library users
don't get the extra option. ``value`` field is the canonical
identifier the frontend submits to ``/api/plex/select-music-library``
(real libraries: title; synthetic: sentinel string). Backward-
compatible — entries without ``value`` fall back to ``title``.
Three crash points fixed in downstream consumers (would have failed
during a deep scan after the user picked all-libraries mode):
1. ``database_update_worker.py:411`` — bailed out with "No music
library found in Plex" because ``not self.media_client.music_library``
evaluated True in all-libraries mode (music_library is None there).
Now uses ``is_fully_configured()`` which recognizes the mode.
This was the root cause of the deep scan never starting.
2. ``database_update_worker.py:_get_recent_albums_plex`` — reached
``self.media_client.music_library.recentlyAdded()`` /
``.search()`` directly, AttributeError in all-libraries mode.
Now routes through the new helper methods.
3. ``web_server.py:10947`` (file-path resolver) — accessed
``music_library.locations``; gated on ``music_library`` truthy so
it didn't crash, but silently skipped all-libraries-mode locations.
Now uses ``get_music_library_locations()`` which unions across
sections.
Plus polish:
- ``/api/plex/clear-library`` also resets ``_all_libraries_mode``
so a fresh "select library" flow doesn't inherit stale mode state.
- ``/api/plex/music-libraries`` surfaces "All Libraries (combined)"
as ``current_library`` when in mode (settings UI displays correctly).
- Frontend ``loadPlexMusicLibraries`` uses ``library.value || library.title``
so the sentinel-keyed option submits the sentinel string, not the
human-readable label. Pre-select match handles both paths.
Honest tradeoffs (documented as known limitations):
- Same artist appearing in multiple Plex sections shows as separate
entries in SoulSync (no dedup). Plex returns distinct ratingKeys
for each. Cosmetic; revisit if it bites users.
- Write-back (genre / poster updates) targets one ratingKey at a time
— only updates that section's copy. Other sections' copies stay
unchanged.
- All-libraries mode includes any audiobook library that Plex
classifies as ``type='artist'``. Edge case, opt-in only.
Tests: 21 new tests in tests/media_server/test_plex_all_libraries.py
pin both single-library mode (regression guard) and all-libraries mode
for every refactored method. Existing test_plex_pinning.py fixture
updated to initialize the new flag. 63/63 media_server tests green,
2148/2148 full suite green.
Two fixes.
(1) Discography endpoint now does server-side per-source ID resolution.
When the user clicked Download Discography on a library artist, the
endpoint received whichever artist ID the frontend happened to pick
(spotify_artist_id || itunes_artist_id || deezer_id || library_db_id)
and dispatched it as-is to whichever source it queried. If the picked
ID didn't match the queried source's ID format, the lookup returned
wrong-artist results (numeric ID collisions) or fell back to a fuzzy
name search that picked a wrong artist.
Two reproducible cases:
- 50 Cent's library row had DB id 194687 — coincidentally a real
Deezer artist ID for "Young Hot Rod". When the frontend's
/enhanced fetch silently fell back to the DB id, the backend
sent 194687 to Deezer, and Deezer returned Young Hot Rod's
50 albums in 50 Cent's discography modal.
- Weird Al's library row had a stored Spotify ID. The frontend
sent that to Deezer, which rejected the alphanumeric ID and
fell back to fuzzy name search — which picked The Beatles
somehow, returning 45 Beatles albums.
The mechanism for per-source ID dispatch already exists in
``MetadataLookupOptions.artist_source_ids``, and the watchlist scanner
already uses it; the on-demand discography endpoint just wasn't wired
to it. Fix: when the URL artist_id matches a library row by ANY stored
ID (DB id, spotify_artist_id, itunes_artist_id, deezer_id, or
musicbrainz_id), pull every stored provider ID and pass them as
``artist_source_ids``. Each source gets its OWN stored ID regardless
of which one the URL carries. When the URL ID is a non-library
source-native ID and the row lookup misses entirely, behavior is
identical to before (single-ID dispatch fallback).
Logged the resolved per-source ID dict at INFO so future "wrong artist
showed up" diagnostics are immediately legible in app.log.
(2) Logger namespace fix in core/artists/quality.py and
core/metadata/multi_source_search.py.
Both modules used ``logging.getLogger(__name__)`` which resolves to
``core.artists.quality`` / ``core.metadata.multi_source_search`` —
neither under the ``soulsync`` namespace where the file handler is
wired. Result: every [Enhance], [MultiSourceSearch], and direct-lookup
INFO line was being written to a logger with no handlers and silently
dropped. App log showed the slow-request warning but no diagnostic
detail. Switched both to ``get_logger()`` from utils.logging_config so
the soulsync.* namespace picks them up. Same content, now actually
lands in app.log. Confirmed working in live test:
``[Enhance] Direct lookup matched: deezer ID 1476162252 → 'Desastre'``
No behavior change in any other caller. Empty ``artist_source_ids``
(no library row matched) reaches lookup as ``None`` → identical to
current single-ID dispatch path. Logger fix is pure routing — no
content change.
Followup on the previous Enhance refactor. Multi-source parallel text
search closed the worst case (users with no Spotify/Deezer getting
"unknown artist - unknown album - unknown track" wishlist entries),
but text search itself is still fragile against messy library tags:
"Title (Live)", featured artists in the artist field, etc. Download
Discography never had this problem because it resolves albums by stable
ID, not by name.
Enhance now does the same thing for tracks: for every metadata source
the user has configured, if the library track has the corresponding
stored ID (spotify_track_id / deezer_id / itunes_track_id / soul_id),
call client.get_track_details(stored_id) directly and convert to the
wishlist payload. First success wins. The user's configured primary
source is tried first so a Deezer-primary user gets Deezer payloads on
the wishlist entry (correct cover art / album shape) even when other
sources also have stored IDs for the same track.
Multi-source parallel text search stays as the fallback for tracks
with no stored IDs (e.g. manually imported, never enriched). Empty-
field rejection still gates the wishlist add.
Implementation:
- _STORED_ID_COLUMNS: source name → DB column mapping
(Discogs intentionally omitted — release-based, no per-track IDs)
- _enhanced_to_wishlist_payload: converts the get_track_details
intermediate "enhanced" shape (artists as [str]) to wishlist shape
(artists as [{'name': str}]). Spotify's raw_data is already in
wishlist shape, returned as-is when detected (preserves full
album.images that the enhanced top-level fields drop)
- _try_direct_lookup_all_sources: iterates sources preferred-first,
calls get_track_details on each that has both a stored ID and a
configured client, returns first complete-metadata payload
- spotify_client field removed from ArtistQualityDeps (no longer
used — Spotify direct lookup now flows through the generic
per-source loop using the entry from search_sources)
- _try_upgrade_to_rich_payload removed (was Spotify-only with broken
shape semantics for non-Spotify sources; search-fallback now uses
_build_payload_from_track consistently)
- get_primary_source() consulted to set the per-call preferred source
for direct-lookup priority
Also fixed a stale UI string: the Enhance modal toast read "Matching
tracks to Spotify and adding to wishlist..." regardless of which
sources were actually configured. Now reads "Matching tracks across
metadata sources...".
Tests:
- _build_deps mirrors web_server._resolve_search_sources: passing
spotify=spotify_obj auto-prepends ('spotify', spotify_obj) to
search_sources (Spotify is always added when configured in prod)
- 5 new tests pin the direct-lookup behavior:
- test_direct_lookup_via_deezer_id_skips_text_search
- test_direct_lookup_via_itunes_id_skips_text_search
- test_direct_lookup_prefers_user_primary_source
- test_direct_lookup_falls_through_to_text_search_when_no_stored_ids
- test_direct_lookup_failure_falls_through_to_text_search
- Reframed enhanced-format and search-fallback tests for the new
payload-build path (no album-image side call, search-fallback uses
_build_payload_from_track consistently)
- 22/22 quality tests green, 2133/2133 full suite green.
Track Redownload had been doing parallel multi-source metadata search
across every configured source the whole time; Enhance Quality was
running a single-source primary fallback that returned junk matches
with empty fields when the primary was iTunes (Discord report:
"unknown artist - unknown album - unknown track" wishlist entries
for users with neither Spotify nor Deezer connected).
Lift the redownload search into core/metadata/multi_source_search.py
and point both flows at it. Same scoring, same per-source query
optimization (Deezer's structured artist:/track: form), same
current-match flagging via stored source IDs.
ArtistQualityDeps now takes get_metadata_search_sources (returns
[(name, client), ...] for every configured source) instead of the
single-primary get_metadata_fallback_client + get_metadata_fallback_source.
Spotify direct-lookup stays as a fast-path optimization (only Spotify
exposes get_track_details(id) returning rich raw payload); when it
doesn't fire, the multi-source parallel search picks the cross-source
best match. Empty-field matches still rejected before wishlist add.
Tests: _build_deps helper updated to accept the new search_sources
contract while preserving fallback_client/fallback_source ergonomics.
Reframed tests for the new semantics — direct-lookup is no longer
gated on Spotify being the active primary; failure reason now lists
every searched source. Added a test pinning the no-sources-configured
prompt. 17/17 quality tests green, 2128/2128 full suite green.
Discord report: clicking Enhance Quality on an artist with neither
Spotify nor Deezer connected added tracks to the wishlist as
"unknown artist - unknown album - unknown track".
Root cause was structural. core/artists/quality.py had a hardcoded
Spotify-direct → Spotify-search → iTunes-fallback chain that ignored
the user's configured primary metadata source. When Spotify wasn't
connected, every track fell through to an iTunes-only fallback that
occasionally returned matches with empty fields (cleared the 0.7
confidence threshold but missing artist / album / title). Those
empty strings propagated through the wishlist payload normalizer's
truthy-check passthrough at core/wishlist/payloads.py:77-80 and the
UI rendered them as "Unknown" defaults.
Rewrote the flow source-agnostic:
- ArtistQualityDeps gains get_metadata_fallback_source. Flow resolves
the user's active primary source once up front.
- New _build_payload_from_track helper produces the Spotify-shaped
wishlist payload from any source's Track object — single place
that knows how to construct it (replaces the duplicate construction
in the Spotify-search and iTunes-fallback paths).
- New _search_match helper does generic confidence-scored search
against any client implementing search_tracks(query, limit). Same
0.7 threshold, same album-bonus weighting as before.
- New _has_complete_metadata validator rejects matches with empty
title / album / artists before they reach the wishlist.
- _spotify_direct_lookup kept as a Spotify-only optimization (only
Spotify exposes get_track_details(id) returning rich raw payload);
other sources fall through to search.
- Failure reason now names the active source: "No usable {source}
match — connect another metadata source for better coverage".
Result: Discogs users get a Discogs search. Hydrabase users get a
Hydrabase search. iTunes users get an iTunes search with empty-field
rejection. Spotify keeps its direct-lookup fast path.
6 new tests pin the architectural change:
- Primary-source dispatch routes to the configured client (Discogs,
not Spotify) when Spotify isn't primary
- Spotify direct-lookup is gated on Spotify being the active primary
(skipped when Discogs is configured even if track has spotify_track_id)
- Empty title / album / artists fields all reject the match
- Failure reason names the active source
The "Media Server Engine Foundation" entry was written when the engine
still had safe-default routing wrappers for optional methods. Those
were dropped in the honesty pass. Entry now matches reality:
- Lists the actual engine surface (6 methods: client / active_client /
active_server / is_connected / configured_clients / reload_config)
instead of claiming "uniform safe defaults for optional methods"
- References KNOWN_PER_SERVER_METHODS as the data-only listing
(replaces the old OPTIONAL_METHODS naming)
- Cites real test counts (42 total) instead of the stale 35
- Drops the "33+ dispatch sites" overclaim (was already partial); the
actual framing is "uniform-shape chains lifted, ~18 server-specific
chains stay explicit per the lift-what's-truly-shared standard"
- services/sync_service.py: dropped unused PlexClient / JellyfinClient
/ NavidromeClient class imports. After the engine refactor the
service only needs TrackInfo for type annotations; the class
imports were dead.
- WHATS_NEW: extended the media server engine review-pass entry to
cover the followup commits (Cin-5 per-server global removal +
Gap 1 shared types lift) so the changelog matches the actual
branch state.
Broque Thomas