The sync editor renders server covers as <img src="/api/navidrome/cover/{id}">,
but no Flask route ever served that path — so every Navidrome cover 404'd, on
every album, art or not. The source (left) side then went blank too: a source
row with no native art (e.g. YouTube, which provides none at mirror time) falls
back to borrowing the matched server track's cover — i.e. that same dead route.
So both sides collapsed to nothing.
Fix:
- New NavidromeClient.build_cover_art_url(cover_id) — builds the absolute,
Subsonic-authenticated getCoverArt URL (base_url + token/salt), keeping
credentials server-side. Uses a FIXED cover-art salt so the URL is
deterministic for a given (server, password, cover_id): a rotating salt (as
in _generate_auth_params) would make every request a unique URL → image-cache
miss every time + a dead, never-reused cache row per fetch. Token auth doesn't
require a unique salt, and the password is never exposed (only its salted md5).
- New route /api/navidrome/cover/<cover_id> — resolves that URL and streams the
image through the shared image cache (same pattern as /api/image-proxy), with
a private max-age so the browser caches by the stable route URL.
Effect: server side works for any album that has art in Navidrome; matched
source rows with no native art now borrow the (now-working) server cover.
Unmatched YouTube rows stay blank — no image exists anywhere to show.
Tests: tests/test_navidrome_cover_url.py (8) — URL structure + salted-token auth
(never the raw password), determinism (same id -> same URL so the cache hits;
different id/password -> different URL), optional size, and the not-connected /
no-id / no-credentials guards.
Caveats: not executed against a live Navidrome (no server in CI) — the URL
builder is unit-tested; the route's cache→HTTP→bytes round-trip is read-verified
only. Scope is the sync editor's Navidrome route; Plex/Jellyfin server-cover
branches and any other modals using a different mechanism are untouched.
Three compounding bugs hit tracks whose source metadata is YouTube/streaming-
shaped — title "Artist - Song", artist "Official Artist"/"Artist - Topic"/
"ArtistVEVO" (reported: "Arctic Monkeys - Do I Wanna Know?" by "Official Arctic
Monkeys"). Server-agnostic — affects Plex/Jellyfin/Navidrome, not just the
reporter's Navidrome.
Bug A — the match fails. The confidence scorer and the editor's reconcile both
compared the raw "Artist - Song" title against the library's clean "Song"; the
length-ratio penalty + floor drove it to ~0.18 (NO-MATCH), so the track showed
unmatched while its server copy showed as an orphan "extra". New pure
core/text/source_title.py (clean_source_artist / strip_artist_prefix /
canonical_source_track) strips the channel/video decoration, applied at BOTH
matching seams: services/sync_service._find_track_in_media_server (tries raw
then canonical, keeps the best) and the editor reconcile. Conservative: a title
prefix is stripped only when it equals the artist, so "Self-Titled", "Jay-Z",
and "Marvin Gaye" (by another artist) are untouched, and the canonical form is
an additional best-of candidate so it can only help.
Bug B — manual matches never persisted. get_server_playlist_tracks built the
per-source entry WITHOUT source_track_id, so "Find & add" posted an empty id
and _persist_find_and_add_match returned early. The match reverted to "extra"
on reload and re-adding looped. The editor's 3-pass matcher is now lifted to a
pure, tested core.sync.playlist_reconcile.reconcile_playlist that includes
source_track_id (the frontend at pages-extra.js:1836 already reads + sends it).
Bug C — manual match duplicated + delete wiped all copies. "Find & add" always
inserted, so linking a source to an already-present server track appended a
duplicate (pos 72, 73...); remove filtered out EVERY entry with the target id.
New pure core.sync.playlist_edit (plan_playlist_add: link-don't-duplicate when
the target is already present; remove_one_occurrence: drop a single copy) wired
into the Plex/Jellyfin/Navidrome add + remove branches.
Tests (extreme): tests/test_source_title.py (35), tests/test_playlist_reconcile.py
(11 — incl. the reported case, parity for override/exact/fuzzy/extra, and
duplicate-server handling), tests/test_playlist_edit.py (12). 286 matching/sync
tests still pass.
Caveats: the sync_service change and the add/remove/editor endpoints are
read-verified, not executed against a live media server (none in CI). The pure
cores they call are exhaustively unit-tested; output-shape parity of the
reconcile lift is covered. Delete removes the first matching copy (duplicates
are identical, so harmless).
After an update, installs became unusable: the Amazon enrichment worker runs by
default, the default public T2Tunes proxy (t2tunes.site) was returning
503 'Amazon Music API is not initialized', and the worker treated every album
as an individual error -- logging an ERROR per item, churning network + DB
continuously across the whole library, and marking every row 'error' (a state
the retry tiers never re-attempt, so even after the proxy recovered nothing
re-enriched). The reporter couldn't reach the UI to turn it off.
Two-part fix:
1. Source-outage circuit breaker (core/amazon_outage.py, pure + tested):
- is_source_outage(exc) distinguishes a whole-source outage (HTTP 5xx,
'not initialized', connection failure, non-JSON error page) from a real
per-item miss (404, transient 400, etc.).
- On an outage the worker now leaves the item UNTOUCHED (so it's retried once
the proxy recovers instead of being permanently burned to 'error'), logs
ONCE per streak, and backs off with next_poll_delay_seconds() -- escalating
30s -> 60s -> ... capped at 30 min -- instead of grinding every 2s. It
auto-resumes the normal cadence the moment the source answers (success OR a
non-outage error both clear the streak).
- AmazonClientError now carries status_code so detection doesn't rely on
message parsing.
2. Opt-in by default (web_server.py): amazon_enrichment_paused now defaults to
True. Because enrichment depends on an external public proxy that can be
down, it stays paused unless the user explicitly enables it -- a proxy outage
can no longer take down installs that never opted in. (Behaviour change:
anyone on the old auto-on default is now paused; re-enable in Settings.)
Together: on update the worker is paused -> no flood -> UI accessible; opted-in
users are protected from future outages by the breaker.
Tests: tests/test_amazon_outage.py (12) pin the classifier across every error
surface (incl. the exact 503 'not initialized' case) and the back-off schedule
(monotonic, capped). 157 Amazon tests pass; lint clean.
Note: could not reproduce the exact 'UI fully unreachable' mechanism remotely
(WAL + 8 gthreads shouldn't hard-lock); the fix removes the flood/churn that is
the practical cause and defaults the feature off.
Lets users pick which providers' cover art to use and in what priority,
generalizing the single prefer_caa_art toggle into an ordered, mix-and-match
list (Sokhi's request). Fully opt-in: default album_art_order is [], so every
existing install is byte-for-byte unchanged until the user enables sources.
How it works:
- Per album, walk the user's ordered sources top-to-bottom; the first source
that actually has THIS album's cover wins. A miss falls through to the next;
if all miss, the download's own art is kept (today's default). The worst case
is always exactly the cover you'd get today -- never wrong art, never an
error into the download.
- Connection-gated: a source is only tried when the user is connected to it
(free sources CAA/Deezer/iTunes/AudioDB always; Spotify only when
authenticated). Tidal/Qobuz/HiFi deferred (cover-URL construction + no clean
core accessor -- not shipping unverified extraction).
- Album-match validated: a source's art is used only when the album it returns
matches the requested artist+album (significant-token subset, tolerant of
Deluxe/Remastered/articles/feat./multi-artist). A loose top search hit for a
different record is treated as a miss -> guarantees no wrong-album art.
- The list supersedes the legacy prefer_caa_art toggle: when album_art_order is
non-empty it is the sole authority (add 'caa' to the list to use Cover Art
Archive), and prefer_caa_art is neutralized for both the embedded-tag art and
cover.jpg paths. With an empty list, prefer_caa_art behaves exactly as before.
Implementation:
- core/metadata/art_sources.py: pure resolver -- effective_art_order (config +
legacy back-compat) and resolve_cover_art (ordered walk + fallback,
exception-safe per source). No network/config/DB; fully unit-testable.
- core/metadata/art_lookup.py: availability gating, per-source lookups against
existing clients (Deezer/iTunes/AudioDB/Spotify search + CAA via MBID),
album-match validation, per-album caching, and select_preferred_art_url --
the single gate the pipeline calls (no-op unless an explicit list is set).
- core/metadata/artwork.py: wired into embed_album_art_metadata and
download_cover_art, gated so no configured list == current behavior.
- web_server.py: GET /api/metadata/art-sources (connected sources only).
- config/settings.py: default album_art_order: [].
- webui (index.html + settings.js): reorderable list in Core Features reusing
the hybrid-source-list pattern + real service logos (with emoji fallback);
load/save wired through the existing metadata_enhancement settings flow.
loadArtSourceOrder populates the saved order synchronously (filtered to known
sources, not availability) so a save before the availability fetch resolves,
or a temporarily-disconnected source, can never wipe the saved order.
Tests: 40 unit/seam tests (resolver ordering/fallback/back-compat, availability,
per-source extraction, album-match validation incl. wrong-album/wrong-artist
rejection, caching, exception-safety, the off-by-default gate). Full metadata
suite still green (610 passed) -- the gated integration changes nothing when no
list is configured.
Note: the settings UI (DOM-heavy, not unit-testable in the JS harness) and the
live per-source art-fetch quality are validated by manual testing.
New GET /api/downloads/task/<id>/detail merges the live download task with its
library_history row (the data the Download History cards show) into one payload
the upcoming track-detail modal renders: status kind, title/artist/album,
source, quality, final location, AcoustID verdict, and expected-vs-downloaded.
Assembly + status classification live in core/downloads/track_detail.py as a
pure, importable build_track_detail()/classify_status_kind() (9 unit tests);
the endpoint is thin glue that looks up the matching history row by track.
Accepting a quarantined item re-imported the file correctly, but the download
modal kept showing 'Quarantined'. The re-import ran through the inner pipeline,
which doesn't mark task completion (that's the verification wrapper's job), and
the sidecar context had no task_id anyway (popped before quarantine).
The chooser's Accept now sends the originating task_id, and the endpoint
re-runs the import through the verification wrapper with that task_id (+ batch_id
looked up from the task), so the task is marked completed only after the file is
verified moved — the row flips to Completed on the next poll. Manager-tab
approvals (no task_id, no JSON body — handled via get_json(silent=True)) keep
the original inner-pipeline path.
Also clear has-candidates + the quarantine dataset on every status render so a
row that goes quarantined -> completed doesn't keep a stale chooser attached.
Clicking a quarantined track's status used to open the generic search modal,
identical to a plain failure — no way to review or recover the file. It now
opens a chooser:
- Listen: streams the file in-app via a new /api/quarantine/<id>/stream
endpoint (range-supported; the real audio Content-Type is recovered from the
sidecar since the on-disk file ends in .quarantined).
- Accept & Import: existing /approve (restore + re-import, gates bypassed).
- Search for a different result: the existing candidates modal (old behavior).
Non-quarantine failures (not_found / failed / cancelled) are unchanged — a
single click listener routes by dataset set at render time, so a task that
fails then later quarantines can't end up double-bound.
Also fixes the Accept failure on Windows: the Listen stream holds an open file
handle, so the subsequent restore move hit WinError 32 ('file in use') and the
endpoint mislabeled it 'thin sidecar'. Accept now releases the audio handle
before approving, and approve/recover moves retry briefly on transient OS locks
(_move_with_retry). Accept also auto-falls-back to Recover-to-Staging for
genuinely thin/orphaned sidecars.
Tests: stream-info resolution (sidecar + filename-fallback + missing), and
_move_with_retry success/give-up.
The save endpoint coerced library_track_id with int(), which rejected
every non-numeric id with "Invalid library track id". Library ids are
str(ratingKey) — numeric for Plex but GUIDs/hashes for Navidrome,
Jellyfin, and other Subsonic servers — and are stored in the TEXT
tracks.id column, so the coercion broke manual matching on every
non-Plex server.
Replace the int() coercion with a normalize_library_track_id() helper
that trims and rejects only empty input, passing the opaque string id
straight through. Plex numeric ids are unaffected (SQLite INTEGER
affinity still stores a clean numeric string as an int, so existing
matches are byte-identical) and no schema migration is needed (the
INTEGER column already stores non-numeric ids as text).
Tests: pure-helper cases (numeric/GUID/whitespace/empty) plus a real-DB
round-trip proving a GUID id saves, reads back unchanged, and enriches.
Post-incident hardening. A WAL-mode DB corrupted (most likely an interrupted
write during a hard restart), and the backup routine made it unrecoverable:
it (a) never checked integrity, so src.backup() faithfully copied the corrupt
pages into every rolling backup, and (b) pruned oldest-by-mtime, so each new
corrupt backup evicted the last good one. Result: all snapshots poisoned.
New core/db_integrity.py (pure, unit-tested):
- quick_check()/is_healthy(): fast read-only PRAGMA quick_check probe.
- safe_backup(): verifies the SOURCE is healthy BEFORE the Online-Backup copy
and the RESULT after; refuses + discards rather than save a corrupt copy.
- prune_backups(): rotation that NEVER deletes the most-recent verified-healthy
backup, even to honor max_keep — so a run of bad backups can't drop your last
good snapshot.
Wired into BOTH backup paths (the /api/database/backup endpoint and the
auto_backup_database automation handler) — they now refuse on integrity failure
(409 / error status, existing backups untouched) and prune safely.
Tests: tests/test_db_integrity.py (8) using REAL temp DBs incl. a physically
corrupted one — proves refuse-corrupt-source, discard-corrupt-result, and the
exact incident scenario (newest backups corrupt -> the older healthy one is
protected from pruning). Existing maintenance-handler backup test still green
(29 passed). compile + ruff clean.
NOTE: this prevents silent backup poisoning; it does NOT stop the underlying
corruption. Follow-ups still worth doing: WAL-checkpoint on clean shutdown +
a periodic live-DB integrity alert (so corruption is caught on day 1).
listening_history was populated ONLY from the media server; the web player
recorded nothing. Now a play heard ~10s logs to listening_history AND bumps
tracks.play_count/last_played — so the existing 'recently played' query reflects
actual SoulSync listening, and the Phase-2 smart-radio recency signal gets real
data.
- core/playback/play_log.build_play_event(): pure, DB-agnostic normalizer from
player payload -> listening_history event shape. Caller supplies the
timestamp (stays pure). Composite/streamed ids never become the int
db_track_id; bool ids rejected; missing title -> skip. 9 unit tests.
- MusicDatabase.record_web_player_play(): inserts the history row + increments
play_count/last_played for the library track in one call.
- /api/library/log-play: thin endpoint, server-side timestamp, best-effort
(logging failure never 500s / never affects playback).
- Frontend: npMaybeLogPlay on timeupdate fires once per track at the 10s
threshold (flag reset in setTrackInfo, set-before-fetch so it can't
double-fire), fully fire-and-forget.
Pure builder is unit-tested; the DB write can't run in-sandbox (real DB throws)
so it's a thin straightforward insert+update. JS + web_server parse clean.
Self-audit of the revamp surface found real bugs, now fixed:
- DOUBLE-ADVANCE race: crossfade starts ~6s before track end, but when the
track actually 'ended' fired, onAudioEnded ALSO advanced — two skips.
onAudioEnded now bails when npXfadeActive (crossfade owns the advance).
- STRAY CROSSFADE on manual skip/stop: skipping or stopping mid-fade left the
interval running, firing npFinishCrossfade on top of the manual change, and
left the second <audio> playing. Added npCancelCrossfade() (clears the timer,
tears down the 2nd audio, restores main volume) called at the top of
playQueueItem and in handleStop. The fade interval also self-checks
npXfadeActive each tick. npFinishCrossfade clears all flags cleanly so the
legitimate handoff isn't treated as an abort.
- stream_start: moved 'global stream_background_task' to function top (it was
declared inside an if-block — parsed, but brittle/bad form).
web_server parses; 76 streaming+radio tests pass; JS syntax clean; CSS balance
unchanged from HEAD.
Wires the StreamStateStore (Phase 3a) into the live routes so each browser/
device gets its OWN playback instead of every client sharing one global
stream_state. Fixes the long-standing limit where a second tab/device/listener
would hijack the one playback.
- _stream_session_id(): stable per-browser id stored in the Flask session
cookie (falls back to DEFAULT when no request context — e.g. the socket
broadcast thread — so single-user behavior is identical).
- _current_stream_state(): the StreamSession for the calling browser.
- Routes rewired to the caller's session + its own lock: /api/library/play,
/api/stream/start, /api/stream/status, /stream/audio, /api/stream/stop.
- Background tasks tracked per session (stream_tasks[sid]) instead of one
global Future, so stopping/replacing one listener's stream doesn't cancel
another's. Executor bumped 1 -> 4 workers so concurrent listeners don't
queue behind each other.
- Per-session staging: named sessions stage under Stream/<sid>/Stream so
listeners never clear each other's files; default session keeps flat Stream/.
- /stream/status now returns THIS listener's state, which is what the frontend
polls — so per-listener works without touching the socket broadcast (left
serving the default session, now vestigial for status).
Isolation invariant covered by tests/streaming/test_stream_state_store.py
(distinct sessions independent, default stable). The route-level cookie wiring
+ actual two-client no-collision behavior need live multi-client verification
(can't be tested without booting Flask + separate cookies) — EXPERIMENTAL,
needs Boulder to test with 2 browsers/devices. 33 streaming tests still pass;
web_server parses.
Crossfade was a no-op toggle. Real crossfade needs two tracks audible at once,
but /stream/audio only serves the ONE current track (single global
stream_state). So:
- web_server: extracted the range-serving body of /stream/audio into
_serve_audio_file_with_range, and added /stream/library-audio?path= which
serves an arbitrary LIBRARY file through it. Security: the path is resolved
via _resolve_library_file_path (same validator /api/library/play uses) so it
only serves files inside the configured transfer/download/media-library
dirs — not arbitrary disk.
- frontend: a second hidden <audio> (#audio-player-xfade) preloads the NEXT
library track when the current one is within 6s of ending (crossfade on,
not repeat-one), ramps the two volumes in opposite directions, then hands
off to playQueueItem so all normal now-playing state is set.
Honest limits (documented in code): library→library only (streamed tracks
hard-cut as before); there's a brief silent reload at hand-off because
playQueueItem re-points the single stream_state — the perceived crossfade has
already happened by then. EXPERIMENTAL — needs Boulder's live audio
verification; I can't test audio in-sandbox.
33 streaming tests still pass (stream_audio refactor is behavior-preserving).
Foundation for multi-listener playback. Today web_server.py keeps ONE global
stream_state dict + one lock (web_server.py:747), so the whole server shares a
single 'currently playing' — every tab/device is a remote for the same
playback and two listeners collide. That global is woven through ~22 sites and
isn't unit-testable where it lives.
Lifted into core/streaming/state.py WITHOUT changing behavior:
- StreamSession: one playback's state, dict-compatible (s['k'], s.get,
s.update, 'k' in s) so existing call sites work unchanged, each with its
OWN RLock so distinct sessions never block/clobber each other.
- StreamStateStore: registry of named sessions; lazy + race-safe create;
DEFAULT session reproduces today's exact single-global behavior. Also
drop()/active_ids()/session_ids() for the eventual per-listener wiring.
web_server.py now binds (DEFAULT) and
. Drop-in: every .update()/[k]/.get()/ site behaves identically. _set_stream_state routes a reassign
through session.replace() so the store's session stays the live object (it's
effectively dead — prepare.py only mutates in place — but safe now).
Honest scope: this is the PROVABLE half of Phase 3. The remaining half (3b:
derive a per-browser session id, per-session Stream/ staging, executor
concurrency, disconnect cleanup) is browser-coupled and can't be verified
without driving 2+ live clients — deferred to a live session. The store API is
already shaped for it.
Tests (tests/streaming/, 33 total):
- test_stream_state_store.py (19): session dict-compat, isolation, lazy
create, drop rules, active_ids, concurrent-create race safety.
- test_stream_state_callsite_compat.py (7): every real web_server access
pattern (library/play, stream/start, status, audio guard, stop, prepare
in-place mutation, set->replace) against the exact object web_server binds.
- test_prepare.py +1: real prepare worker drives an actual StreamSession.
76 streaming+radio tests green; ruff clean; web_server.py parses.
Two related bugs from the Slipknot album never finishing.
1) _poll_album_bundle_downloads hung when the peer stalled. The finish check
needs every transfer terminal (completed/failed); the #715 grace only covers
'slskd says Completed but file not on disk'. A transfer stuck InProgress /
Queued, or dropped by slskd, is none of those — so it blocked both the finish
AND the grace exit, and the poll spun to the full ~6h timeout.
Add a bundle-level stall guard: track a progress marker (#terminal transfers,
total bytes across pending). If NOTHING advances for _stall_grace (180s) —
no terminal transition AND no pending byte movement — the peer has stalled;
mark the stuck transfers failed so the existing finish/all-failed checks
resolve the bundle with whatever completed (missing tracks then fall back to
the per-track matcher). Conservative: only trips when EVERYTHING is frozen,
so a slow-but-progressing or still-queued transfer is unaffected.
2) Failed batches lingered in the UI forever ('No tracks loaded'). The
auto-cleanup gate removed only complete/error/cancelled phases — 'failed'
(e.g. an album-bundle hard failure) was missing, so it never aged out. Add
'failed' to the terminal set so it's removed after 5 minutes like the others.
Tests (tests/test_soulseek_album_poll_stall.py): stalled peer → gives up with the
completed subset (not the deadline); progressing bundle not falsely stalled;
all-stalled → empty; dropped transfers stall out; clean finish unaffected.
124 download/soulseek tests pass; ruff clean.
Stage 2: the manual 'Download Wishlist' flow now calls the same
_run_wishlist_cycle engine the auto timer uses, so a manual scan runs the exact
same code path as an auto scan. The old bespoke manual orchestration (build
payloads + SERIAL inline dispatch) is deleted — its grouping/dispatch was a
near-duplicate of auto's that had already drifted.
Behavior changes (all intended, discussed):
- Manual now dispatches album bundles in PARALLEL (album pool) like auto, instead
of serially on one thread. A single cycle='albums' engine call covers the whole
selection (albums bundled, singles/ungroupable -> per-track residual), so no
'both cycles' pass is needed.
- The manual placeholder batch_id is reused as the engine's first sub-batch
(first_batch_id), so the modal's existing poll target stays valid.
- WishlistManualDownloadRuntime gains album_bundle_executor (wired in web_server,
falls back to the shared pool when unset).
- 'Don't start manual while auto is running' is unchanged — the existing route
guard (is_wishlist_actually_processing -> 409) already covers it; no queue added.
NOT touched: process_wishlist_automatically's behavior (proven by test_automation
staying green in Stage 1) and the per-track download mechanics.
test_manual_download.py rewritten to characterize the new behavior (engine
dispatch via the executor, parallel, placeholder reuse, album-context). Full
wishlist suite green (131); wishlist + automation = 392 passed.
A 2.6.3 change (c3b88e69) split the wishlist albums cycle into one batch per
album. Each album batch runs an INLINE-BLOCKING soulseek/torrent/usenet
album-bundle download (album_bundle_dispatch.try_dispatch ->
download_album_to_staging) that holds its worker thread for the whole
search+download. All of these were submitted to the shared 3-worker
missing_download_executor -- the same pool used for per-track downloads AND the
manual 'Download Wishlist' analysis.
So a large Album-Completeness 'Fix all' (-> ~819 wishlist tracks -> ~82 per-album
batches) saturated all 3 workers with blocking album downloads; the manual
wishlist analysis could never get a thread ('Library Analysis' stuck on
Pending), the other ~79 batches sat in phase='queued' forever, and auto-cleanup
(which only evicts terminal-phase batches) never cleared them -> jam until
restart. Fixing batch STATUS would not help: the threads are blocked inside the
download, not waiting on a phase flip.
Fix: add a dedicated bounded album_bundle_executor (max_workers=3) and route the
AUTO per-album bundle batches to it, keeping the shared pool free for analysis /
per-track / the manual wishlist (which always starts now). Hung/slow album
downloads can only delay other album downloads, never the user-facing path.
Additive and decoupled; the submit site falls back to the shared pool when the
album pool isn't wired (older callers / tests) so behavior is unchanged there.
The manual path is untouched (it already runs album bundles serially on a single
thread, by design).
Tests (tests/wishlist/test_automation.py): album sub-batches route to the
dedicated pool while the residual per-track batch stays on the shared pool;
fallback-to-shared-pool when no album pool is wired. Existing auto-processing
tests still green (fallback preserves prior behavior). 707 passed across
wishlist + downloads suites.
The same provider ID is stored under inconsistent column names across tables
(deezer_id vs deezer_artist_id vs album_deezer_id vs similar_artist_deezer_id;
spotify/itunes keep an entity qualifier, others don't; musicbrainz uses three
nouns), so code checks 2-5 name variants everywhere.
Add core/source_ids.py as the single source of truth for (provider, entity) ->
column, with accessors that read an ID from a dict/sqlite3.Row robustly
(canonical column first, then known aliases). NO database columns are renamed —
these are the real names today; the registry just centralizes the knowledge.
Targeted adoption (behavior-identical, verified):
- core/artist_source_lookup.SOURCE_ID_FIELD now derives from the registry
instead of duplicating the mapping (values unchanged).
- web_server.py artist-detail builds artist_source_ids via source_id_map(...)
instead of a hand-rolled per-source .get() dict.
Broader call-site adoption deferred as clearly-scoped follow-up.
Tests: tests/test_source_ids_registry.py (canonical columns, alias fallback,
canonical-preferred, sqlite3.Row, source_id_map, SOURCE_ID_FIELD unchanged).
Existing artist_source_lookup + artist_full_detail suites still green.
Reported by @Yug1900: a Spotify playlist's overview shows the correct count
(e.g. 203) but the sync modal lists only 100, and only 100 would sync.
Root cause in /api/spotify/playlist/<id> (the Spotify-tab track fetch): owned
playlists fetch + paginate fine (sp.next loops over all pages). But when the
authenticated playlist_items call FAILS (intermittently, often a 403 on
followed / not-owned playlists) it fell straight back to the public embed
scraper, which is hard-capped at ~100 tracks and has no pagination — and the
result was returned as if complete. The overview total is correct because it
comes from the playlist *metadata* listing, which succeeds independently.
Fixes (additive — the owned/working path is unchanged):
- Retry the items fetch once before resorting to the embed scraper, so a
transient failure no longer silently truncates a large playlist.
- Guard against silent truncation: compare fetched count against the
playlist's known total; if short (or via the capped embed fallback), log a
warning and return `incomplete: true` + `expected_total` instead of
presenting a partial list as complete.
No behavior change when the fetch succeeds in full (incomplete=false, extra
fields ignored by the frontend). Lets a future UI tweak surface
"got 100 of 203 — retry" rather than silently dropping tracks.
Final cluster: the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles) ->
core.discovery.endpoints.save_bubble_snapshot(...), wired via
_save_source_bubble_snapshot. All four validate a payload key, persist via
db.save_bubble_snapshot(kind, items, profile_id=...), and return a count +
timestamp; they differ only by:
- payload_key ('downloads' for discover, 'bubbles' for the rest) + its
no_data_error message.
- snapshot_kind, success_noun, and the info/except log subject + noun
("downloads"/"artists"/"albums/tracks"/"charts").
get_database / get_current_profile_id injected; get_json (request.json) invoked
inside the try, preserving the original 400/500 behavior incl. traceback dump.
Tests: +5 (missing key 400, None body 400, happy path with kind/profile/count/
timestamp, discover_downloads variant, exception -> 500). Full discovery suite:
210 passed.
web_server.py: -98 lines.
Ninth cluster: update_<source>_playlist_phase for the five sources sharing the
identical validation + full-message response (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube) -> core.discovery.endpoints.update_playlist_phase(...),
wired via _update_source_playlist_phase + the _PHASE_LIST/_PHASE_LIST_YT
constants.
Per-source params:
- valid_phases — YouTube additionally allows 'parsed'.
- apply_extra_fields — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body; Tidal and
YouTube do NOT, so they pass False (kept strictly 1:1 — the generic won't
apply those keys for them even if a caller sent them).
- not_found_message / error_label; get_json invoked inside the try.
NOT folded in: iTunes-Link — uses data.get('phase') (no "Phase not provided"
400) and returns a no-message payload.
Tests: +7 (404, missing-phase 400, invalid 400, happy path with extra-fields
suppressed, extra-fields applied when enabled, YouTube 'parsed' allowed,
exception -> 500). Full discovery suite: 205 passed.
web_server.py: -123 lines.
Eighth cluster, the heavyweights (~110 lines each). The fix-modal
update_<source>_discovery_match for the four sources with the identical
structure (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.update_discovery_match(...), wired via
_update_source_discovery_match. Applies the user-selected Spotify track to the
discovery result (status/artist/album/duration/spotify_data/match-count) and
writes the manual fix to the discovery cache.
Per-source pieces are params:
- source_log_label / error_label.
- original_track_key ('tidal_track' / 'deezer_track' / ...).
- original_artist_getter: Tidal handles string-or-object artists
(first_artist_str_or_obj); the rest assume strings (first_artist_plain).
- web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) injected.
- get_json passed as a callable and invoked INSIDE the try, preserving the
original's "request.get_json() inside try" behavior (malformed body -> 500).
NOT folded in (genuinely divergent): iTunes-Link (saves spotify_data directly
via a different cache signature), YouTube (multi-key original_track fallback),
ListenBrainz (entirely different unmatch-capable structure, no cache write),
Beatport.
Tests: +9 (extractors; 400/404/400 guards; full happy path with result
mutation + duration formatting + match-count + cache-save args; no-increment
when already found; cache error swallowed; get_json raise -> 500). Full
discovery suite: 198 passed.
web_server.py: -400 lines.
Seventh cluster: start_<source>_sync for the five sources with the identical
flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube) ->
core.discovery.endpoints.start_sync(...), wired via _start_source_sync.
Validates phase, converts discovery results, seeds sync state, posts a
"... Sync Started" activity item, and submits to the sync executor. Per-source
pieces are params:
- sync_id_prefix (f"{prefix}_{key}"), not_found/not_ready messages, convert_fn.
- name/image accessors: Tidal reads an object (playlist_name_obj/
playlist_image_obj), the rest a dict (playlist_name_strict/playlist_image_dict).
- activity_label vs error_label DIFFER for Spotify-Public ("Spotify Link
Sync Started" activity, "Spotify Public" logs).
- submit_sync_task glue (_submit_sync_task) closes over sync_executor /
_run_sync_task / get_current_profile_id so the helper stays global-free.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task WITHOUT a playlist_image_url arg), Beatport (extra debug logging, chart).
Tests: +6 (404, not-ready 400, no-matches 400, full happy path with
state/sync-infra/submit/activity assertions, resync phases allowed,
exception -> 500). Full discovery suite: 189 passed.
web_server.py: -172 lines.
Sixth cluster: the bulk-hydration get_<source>_playlist_states endpoints for
the five sources that build the identical per-entry dict + {"states": [...]}
shape (Tidal, Deezer, Qobuz, Spotify-Public, iTunes-Link) ->
core.discovery.endpoints.get_playlist_states(states, *, error_label,
info_log_label=None), wired via _get_source_playlist_states.
iTunes-Link is the only one of the five without the "Returning N stored ..."
info log, so info_log_label is optional (iTunes passes None to suppress it).
NOT folded in: the YouTube/ListenBrainz get_all_*_playlists endpoints. They
return {"playlists": [...]} (different key) with a different field set
(url / created_at / playlist, no discovery_results) and filter out
mirrored_/profile-scoped entries — genuinely divergent, kept as-is.
Tests: +4 (list build + last_accessed bump + exact shape, empty, optional ids
default None, missing-required-field -> 500). Full discovery suite: 183 passed.
web_server.py: -116 lines.
Fifth cluster: reset_<source>_playlist for the four sources with byte-
identical bodies (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.reset_playlist(states, key, *, label,
not_found_message), wired via _reset_source_playlist. Resets phase/status to
'fresh', clears discovery/sync fields, cancels any discovery_future, and
preserves the original playlist payload.
Left with their own bodies (genuinely divergent):
- YouTube: status -> 'parsed' (not 'fresh'), no download_process_id, logs the
playlist name, "reset to fresh state".
- ListenBrainz: status -> 'cached', logs playlist title, returns
{"success": True, "phase": "fresh"} (different payload), _lb_state_key.
- iTunes-Link: state.update(...), no info log, "iTunes Link reset to fresh
phase".
Tests: +4 (404, full clear + playlist preserved + future cancelled, no-future
path, exception -> 500). Full discovery suite: 179 passed.
web_server.py: -100 lines.
Fourth cluster: get_<source>_discovery_status (all eight sources, Beatport
included) -> core.discovery.endpoints.get_discovery_status(states, key, *,
not_found_message, error_label), wired via _get_source_discovery_status.
Unlike sync-status, the discovery-status response shape is byte-identical
across every source (phase/status/progress/spotify_matches/spotify_total/
results/complete), so Beatport folds in here too. Only the 404 string
("... discovery not found" vs "... playlist not found" vs "Beatport chart
not found") and the except-log label vary. ListenBrainz key via _lb_state_key.
NOT touched this cluster: get_*_playlist_state (the sibling endpoints).
Those genuinely diverge per source — different id-key name (playlist_id /
url_hash / playlist_mbid), presence of url / created_at / download_process_id,
Tidal's playlist.__dict__ serialization, and YouTube's strict (non-.get)
field access. Folding them would need a flag pile that wouldn't be a clean
1:1, so they keep their own bodies.
Tests: +4 (404, full response + last_accessed bump, complete=False when not
'discovered', missing-field -> 500). Full discovery suite: 175 passed.
web_server.py: -155 lines.
Third cluster: the get_<source>_sync_status routes (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz) -> core.discovery.
endpoints.get_sync_status(...), wired via _get_source_sync_status glue.
This cluster carries the real per-source quirks, all captured 1:1 as params:
- not_found_message (iTunes-Link uses "iTunes Link not found").
- error_label vs activity_subject — these DIFFER for Spotify-Public: the
activity feed says "Spotify Link playlist ..." while the except log says
"Error getting Spotify Public sync status".
- playlist-name accessor, three styles lifted verbatim as named helpers:
playlist_name_attr_or_unknown (Tidal: object .name), playlist_name_strict
(Deezer/Qobuz/Spotify-Public/iTunes: state['playlist']['name'], can raise),
playlist_name_safe (YouTube/ListenBrainz: .get default). The strict getter
preserves the original's behavior of raising -> 500 AFTER phase/sync_progress
were already mutated.
- ListenBrainz key via _lb_state_key (caller-resolved).
Beatport stays separate (different payload: status not sync_status, sync_id,
no lock, chart key).
Tests: +9 (3 name accessors incl. raise/fallback semantics; status 404s,
running-no-mutation, finished+activity, error+revert+activity, and strict-
getter-missing -> 500 after partial mutation). Full discovery suite: 171 passed.
web_server.py: -244 lines.
First cluster of the per-source playlist-discovery deduplication. The
convert_<source>_results_to_spotify_tracks functions (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube, ListenBrainz) plus the already-generic
_convert_link_results_to_spotify_tracks were byte-identical apart from the
source label used in their log line.
Lift the shared body into core/discovery/endpoints.py as
convert_results_to_spotify_tracks(results, source_label); the 7 web_server
functions become 1-line delegations (names/signatures unchanged, so all
callers and behavior are identical — 1:1).
Beatport is intentionally NOT folded in: its converter coerces artist
objects to strings and emits a different track shape (source field, album
dict), so it keeps its own implementation.
Tests: tests/discovery/test_discovery_endpoints.py (12) pin both input
shapes (manual spotify_data / auto spotify_track+found), optional
track/disc numbers, falsy-0 omission, field defaults, skip-on-neither,
order preservation, if/elif precedence, empty input.
web_server.py: -209 lines. Full discovery suite: 151 passed.
Two things in this commit. Functional download / matched-download
behaviour is untouched — same JS handlers, same routes for the
download actions, same album-expand interaction.
VISUAL REDESIGN
- Glass search-bar card with accent radial wash + focus ring + pill
primary search button
- Source chip row above the search bar (see below)
- Always-visible compact filter pill row (Type / Format / Sort) —
pills carry both ``bs-filter-pill`` (new visual) and ``filter-btn``
(legacy class for ``resetFilters`` + ``applyFiltersAndSort`` in
wishlist-tools.js to keep working)
- Accent-tinted status pill matching the dashboard / auto-sync look
- Album result cards: glass card with accent left-edge stripe,
52px brand-tinted cover icon, chevron expand indicator, pill
action buttons (Download / Matched Album), accent glow on hover
- Track result cards: glass row with accent stripe, 44px icon,
pill action buttons (Stream / Download / Matched Download)
- Multi-disc separators inside expanded album track lists styled
with the accent treatment
- Responsive: action button columns stack vertically below 900px
New CSS lives in a self-contained ``webui/static/basic-search-v2.css``
sheet linked from index.html. Selectors are scoped to
``#basic-search-section`` for any class that already exists in
style.css (``.album-result-card``, ``.album-icon``, ``.track-*``,
etc.); the new ``bs-*`` prefixed classes for the search bar /
filters / source row / status are unscoped because they only exist
in the new markup. ``!important`` is used on the card-level rules
to defeat the original unscoped ``.album-result-card`` etc. rules
in style.css that would otherwise leak heavyweight padding /
box-shadow / 56px icon styles into the new design.
Also removed ``overflow: hidden`` from the original
``.album-result-card`` and ``.track-result-card`` rules in style.css
— those two classes only render in ``downloads.js`` basic search
results (verified via grep, two render sites only), so the
removal can't impact any other UI.
SOURCE PICKER (hybrid mode)
- New ``GET /api/search/sources`` endpoint returns the list of
active sources from the orchestrator's chain (or the single
active source in single-source mode).
- Frontend renders a chip row above the search bar. Click a chip
to target that source for the next search; the chip's brand
accent fills.
- In single-source mode the lone chip is rendered as a dashed-
border label so the user always knows what they're searching
but can't accidentally try to switch to sources that aren't
configured.
- ``/api/search`` accepts an optional ``source`` body param. When
set, ``core/search/basic.py:run_basic_search`` resolves the
client directly via ``orchestrator.client(source)`` and calls
its ``.search()`` instead of going through the hybrid chain.
- Backwards compatible: omitting ``source`` falls through to the
original ``orchestrator.search()`` call exactly as before.
Unknown source names also fall back to the default — typo
protection.
TESTS (5 new + 6 pre-existing = 11 total in test_search_basic.py)
- source param routes to specific client, NOT orchestrator chain
- no source param preserves original orchestrator-default behaviour
- unknown source name falls back to orchestrator default
- ``run_basic_soulseek_search`` backwards-compat alias preserved
- source-targeted path serialises albums + tracks correctly
101 search-suite tests pass.
- _SOULSYNC_BASE_VERSION → 2.6.4
- helper.js: '2.6.4' unreleased → 'May 28, 2026 — 2.6.4 release'
- .github/workflows/docker-publish.yml default version_tag → 2.6.4
- pr_description.md: rewrite for 2.6.4 with #721 as the headline patch,
2.6.3 fixes carried forward unchanged (2.6.3 was bumped on dev but
never reached main / docker, so 2.6.4 is the first release to ship
this batch)
- _SOULSYNC_BASE_VERSION → 2.6.3
- helper.js WHATS_NEW unreleased flag → 'May 27, 2026 — 2.6.3 release'
Note: .github/workflows/docker-publish.yml default version_tag was
also bumped to 2.6.3 locally, but .github is gitignored in this
repo — workflow updates need to land via the GitHub UI separately.
The workflow_dispatch input is overrideable at trigger time
regardless of the default, so this isn't blocking.
Two related leaks in ``storage/album_bundle_staging/<batch_id>/``:
1. **Soulseek bundle cleanup was excluded.** The per-batch cleanup
at the end of a bundle download gated on:
(album_bundle_source or '').lower() in ('torrent', 'usenet')
The comment justified it as "slskd keeps its own completed
folders" — but the Soulseek bundle path ALSO copies completed
files into the private staging dir (``soulseek_client.py:1599``,
``copy_audio_files_atomically(completed, Path(staging_dir))``)
for the per-track workers to claim. Those copies persisted
forever; long-running installs accumulated stale GB. Extended
the cleanup gate's allow-list to include ``soulseek`` so the
per-batch dir is removed on bundle completion — same code path
that already worked for torrent / usenet.
2. **No sweep for orphan dirs.** Any leftover ``<batch_id>``
subdir from a previous-session crash, an errored batch, or a
pre-fix Soulseek bundle stayed on disk forever. Added
``sweep_orphan_album_bundle_staging(staging_root, active_batch_ids)``
that runs ONCE at server startup, before any batch can register
a staging dir. Removes every ``<batch_id>``-shaped subdir
whose id isn't in the active set. Safe by construction:
- Only touches subdirs of the configured staging root.
- Name-shape check (``entry.name == _safe_batch_dirname(entry.name)``)
rejects hand-placed dirs like ``.git`` or stray docs.
- ``shutil.rmtree`` errors log + continue — sweep must not
crash app startup over a permission glitch.
- active_batch_ids normalised through ``_safe_batch_dirname``
so colon-bearing batch_ids match their on-disk form.
Wired into the web_server startup right after the stuck-flags
diagnostic so it fires before anything else touches batches.
Tests
- ``test_downloads_lifecycle.py`` gained one regression test
pinning that Soulseek bundles now have their staging dir
cleaned (sibling to the existing torrent test).
- ``test_album_bundle_staging_sweep.py`` (NEW, 11 tests)
covers: orphan removal with no actives, active dirs preserved,
special-char batch_id normalisation, no-op on missing /empty
/empty-string staging root, non-dir entries skipped, unsafe-
name dirs preserved (.git etc.), partial rmtree failure doesn't
abort the rest, listdir failure returns 0 cleanly, default
None active set, defensive against empty / None entries in
the active set.
488 downloads tests pass.
For users with an existing "clean up old files" automation pointed
at this dir: stop pointing it there if you want — the auto-cleanup
+ startup sweep cover it now. Or leave it as belt-and-suspenders
with a relaxed (1h+) mtime threshold so it can't race a mid-batch
download.
finished the release (#715)
Symptom (user @pavelcreates / @IamGroot60 on 2.6.2):
- Click Download on an album in the search modal
- slskd starts + completes every track of the release
- 22+ minutes after the last completed download, batch flips
to "failed" with no clear log line explaining why
- Per-track Soulseek downloads on the same machine were fine
Root cause: ``core/soulseek_client._resolve_downloaded_album_file``
probed three hard-coded candidate paths to locate each downloaded
file in the slskd download dir:
candidates = [
download_path / remote_filename,
download_path / basename,
download_path / *normalized_path_parts,
]
On the common slskd config ``directories.downloads.username = true``
slskd writes files at ``<download_dir>/<username>/<filename>`` —
none of the three candidates carry a username segment, so the
resolver returned None for every file even though the file was
physically present in a subdir one level deeper. ``_poll_album
_bundle_downloads`` saw 0 completed_paths, kept spinning, and
hit the master deadline (~30 min) before bailing the batch.
Why per-track worked: ``web_server._find_completed_file_robust``
already does a recursive walk-by-basename + path-confirm against
the remote directory components, so any layout slskd writes ends
up resolved. The bundle path didn't go through it.
Fix
- Lifted the robust finder into ``core/downloads/file_finder.py``
as a pure function ``find_completed_audio_file(download_dir,
api_filename, transfer_dir=None) -> (path, location)``. Zero
globals; recursive walk; handles slskd dedup suffix
``_<10+digit-timestamp>``, YouTube / Tidal ``id||title`` encoded
filenames, the AcoustID-quarantine subdir skip, basename
collisions disambiguated by remote-path components, and a
fuzzy-basename fallback above 0.85.
- ``_resolve_downloaded_album_file`` keeps the three-candidate
fast path (cheap probe for the slskd-flat default) but now
delegates to the new helper when none hit, instead of giving up.
- ``_poll_album_bundle_downloads`` tracks "slskd reports
Completed but local resolver returns None" per key. When every
remaining key has been in that state past a 45-second grace
window, the poll exits early with an explicit error pointing at
the likely ``soulseek.download_path`` mismatch instead of
silently spinning until the master deadline.
- ``web_server._find_completed_file_robust`` becomes a thin
delegate so both callers share one finder. Legacy inline impl
kept as ``_find_completed_file_robust_legacy`` for reference;
to be removed next release.
- Fixed misleading ``"(0 tracks, quality=)"`` log on the preflight-
reuse path — was reading attrs off a None ``picked`` object.
Tests (17 new in tests/downloads/test_file_finder.py)
- Flat slskd layout
- Username-prefixed (the #715 case)
- Full remote tree preserved
- Deeply nested username + tree
- File genuinely missing returns None
- Basename collision disambiguated by remote dirs
- Single basename match wins regardless of dirs
- slskd dedup suffix match
- Short ``_<digits>`` (year) not treated as dedup
- AcoustID quarantine subdir skipped
- YouTube / Tidal ``id||title`` encoded filenames
- transfer_dir fallback
- Both dirs miss → (None, None)
- Non-audio files ignored
- Empty api_filename
- Fuzzy match on punctuation variant
- Fuzzy rejects below threshold
475 downloads tests pass after the lift.
Five pre-existing lint errors on dev baseline (all introduced May 25-26
before this branch was cut) were blocking CI on this PR. Cleared as
courtesy fixes so the merge isn't gated on unrelated tech debt:
- web_server.py:22613 — F811 duplicate `urlparse` import inside
`_parse_itunes_link_url` (already imported at module top, line 20).
Removed from the inline `from urllib.parse import parse_qs, urlparse`;
kept `parse_qs` since that one is only used here.
- core/listenbrainz_manager.py:746 — S110 silenced with
`# noqa: S110 — best-effort lookup, delete proceeds either way`.
Matches the existing project convention used in web_server.py:1693,
core/watchlist/auto_scan.py:463, core/library_reorganize.py:548.
- core/playlists/sources/listenbrainz.py:236 — B905 `zip()` without
explicit `strict=`. Added `strict=False` — preserves existing
behaviour where `matched` can legitimately be shorter than
`match_indices` on partial discover failure.
- core/playlists/sources/listenbrainz.py:273 — S110 silenced with
`# noqa: S110 — caller falls back to last cached playlist on
refresh failure`.
- core/playlists/sources/soulsync_discovery.py:105 — S110 silenced
with `# noqa: S110 — manager persists last_generation_error on
failure; surface existing snapshot`. The existing multi-line
comment that already explained the swallow was rolled into the
noqa justification so the rule + reason live on one line.
Ruff `python -m ruff check .` now passes; 664 discovery + metadata
tests still pass.
Self-review pass on the prior three commits — kettui-style cleanup
that should have landed first time.
**Length-preference sort ordering (real bug):**
The `search_tracks_with_artist` stable sort that promoted length-known
recordings ran in `core/musicbrainz_search.py`, but the MB endpoint in
`web_server.py:search_musicbrainz_tracks` runs `rerank_tracks` after
it — which re-sorts by relevance score and dropped the length-pref
ordering down to tiebreaker-only. For canonical-same-song MB duplicates
that all score identically the tiebreaker survived, but the
order-of-operations was wrong.
Moved into `rerank_tracks` itself via a new `prefer_known_duration`
flag. Sort key sits between relevance score and the stable-order
tiebreaker so relevance still wins (length only decides ties, never
overrides a higher-relevance match). The MB endpoint opts in via
`prefer_known_duration=True`; Spotify / iTunes / Deezer callers stay
on the default-off path since their search results always include
length. Pinned with three new `TestRerankTracks` cases:
ties-promote-length, relevance-still-wins, default-off-unchanged.
**Route logic lifted to `core/discovery/manual_match.py`:**
Two pieces lived as inline route logic in `web_server.py` — the
`derive_manual_match_provider` fallback chain (payload.source →
active source → 'spotify') used by `update_youtube_discovery_match`,
and the `is_drifted_for_redo` predicate (cached provider differs from
active AND not manual_match) used by `prepare_mirrored_discovery`.
Per kettui's "extract logic from web_server.py, don't AST-parse it"
standard, both helpers now live in `core/discovery/manual_match.py`
with 12 dedicated unit tests covering fallback resolution order,
non-dict payload defenses, manual_match exemption from drift,
absent-provider legacy default, and edge cases.
Side benefits from the lift:
- `match_source` now derived once before the cache-save try block
instead of being duplicated in try + except (the except block existed
only because the original used `match_source` later — pre-computing
killed the duplication).
- `prepare_mirrored_discovery`'s `has_cached` check now reuses
`is_drifted_for_redo` with inverted polarity instead of restating
the field whitelist inline, so a future schema change only has to
land in one place.
- The mirrored-DB persist block now gates on `matched_data is not None`
to avoid a pre-existing latent NameError if the cache-save block
raised before matched_data construction.
**Enhanced toggle localStorage key now profile-scoped:**
`soulsync-library-view-mode` was global — two admin profiles would
share one preference. Wrapped in `_libraryViewModeKey()` which appends
`:${currentProfile.id}` when a profile is loaded, falls back to the
unsuffixed key otherwise (preserves pre-multi-profile saved values).
Tests:
- 12 new in `tests/discovery/test_manual_match.py` pinning both helpers.
- 3 new in `tests/metadata/test_relevance.py` pinning the
`prefer_known_duration` semantics.
- `test_search_tracks_with_artist_prefers_results_with_known_length`
renamed to `_does_not_resort_by_length` since the sort moved out of
this method. 664 tests pass across discovery + metadata suites.
User reported that manually mapping a mirrored-playlist track via the
Fix popup (either by search or by pasting an MBID) worked end-to-end
once — match saved, library track downloaded — but the next Playlist
Pipeline run flipped the track back to "Provider Changed" and forced
them to re-do the manual map every cycle.
Three independent issues were combining to cause this:
1. Hardcoded `provider: 'spotify'` on manual-fix save
`update_youtube_discovery_match` (the endpoint the Fix popup posts
to, also used by mirrored playlists since the frontend routes
`platform === 'mirrored'` through the YouTube endpoint) always
stamped the cached match as Spotify-provided. The Fix-popup cascade
actually queries the user's primary metadata source first and falls
back to Spotify / Deezer / iTunes / MusicBrainz — so a user on
MusicBrainz primary picking an MB result still had it saved as
`provider: 'spotify'`. The next prepare-discovery call (which
compares cached_provider to the active source) then immediately
classified the match as drifted and pending re-discovery. Fixed by
deriving `match_source` from `spotify_track.get('source')` (every
*_search_tracks endpoint stamps `source` on results) with a fallback
to `_get_active_discovery_source()` for the MBID-paste path (which
uses the lean flat shape that doesn't carry source). `matched_data['source']`
and the mirrored `extra_data['provider']` both now use the derived
value. `match_source` is also recomputed in the cache-save except
handler so the downstream mirrored-DB save still has it.
2. Discovery worker re-queueing manual matches as "incomplete"
`run_playlist_discovery_worker` in `core/discovery/playlist.py`
re-adds any track to `undiscovered_tracks` when its `matched_data`
lacks `track_number` or `album.id` / `album.release_date`. The
check was designed as a legacy-fix backfill for old discoveries
that lost those fields to a Track-dataclass stripping bug. But
manual fixes from the popup are *intentionally* lean — search-
result rows don't include `track_number` (none of the search
endpoints return it), and the MBID-lookup flat shape doesn't
carry `album.id` / `release_date` (the recording lookup returns
only `album.name`). So every manual match looked "incomplete" and
got re-discovered every pipeline run, overwriting the user's pick
with whatever the auto-search ranked first. Manual matches now
short-circuit ahead of the incomplete-data branch.
3. `prepare_mirrored_discovery` ignored the `manual_match` flag
Independent of the provider-stamping fix above, the prepare-
discovery endpoint that powers the mirrored-playlist UI did its
own `cached_provider != current_provider` check and didn't honour
manual_match either. Defence in depth — even if a future code
path stamps the wrong provider on a manual match, the flag now
anchors it as cached. `has_cached` also extended so manual
matches with off-provider stamps still count toward the cached
tally for phase classification.
Tests:
- new `test_manual_match_skipped_even_when_matched_data_incomplete`
in `tests/discovery/test_discovery_playlist.py` pins the worker
short-circuit using a realistic MB-shape matched_data (album dict
without id / release_date, no top-level track_number). 16 existing
tests still green; 848 across discovery / metadata / automation
suites pass.
ListenBrainz publishes "Weekly Jams for X" / "Weekly Exploration
for X" with a fresh MBID every week, and "Top Discoveries of YYYY
for X" / "Top Missed Recordings of YYYY for X" with a fresh MBID
every year. Auto-mirroring those per-period yielded one mirrored-
playlist row per week/year — useless for Auto-Sync schedules
because the underlying LB playlist never updates, only a brand new
playlist replaces it. The user accumulates 100+ dead Weekly Jams
rows per year if they discover regularly.
This commit collapses each family into a single ROLLING mirror
keyed by a synthetic ``source_playlist_id`` (e.g.
``lb_weekly_jams_Nezreka``). Each new period UPSERTs into the same
row, so the user gets one stable Auto-Sync schedule per series
that automatically picks up the latest period's tracks on every
refresh. Non-series LB playlists (user-created, collaborative,
Last.fm radios for a specific seed) continue to mirror under
their per-playlist MBID as before. Per-period LB playlists are
still visible + usable on the LB Sync tab — only the mirror layer
collapses.
- ``core/playlists/lb_series.py`` (new) — series-detect helper
with regex patterns + canonical-name + LIKE-pattern template
for each known LB family. Exposes
``detect_series(title)``, ``is_series_synthetic_id(id)``, and
``list_series_synthetic_ids()`` so both the JS auto-mirror hook
and the LB adapter can speak the same language.
- ``GET /api/listenbrainz/series-detect?title=...`` — thin HTTP
shim around ``detect_series`` so the auto-mirror JS doesn't
duplicate the regex.
- ``ListenBrainzPlaylistSource.get_playlist`` now recognizes
synthetic series ids — it queries the LB cache for the newest
cache row whose title matches the series' LIKE pattern and
resolves to that row's MBID before fetching tracks. The mirror's
meta keeps the synthetic id so refreshes always re-resolve to
the latest period.
- ``_mirrorListenBrainzAfterDiscovery`` (sync-services.js) calls
the new detect endpoint when discovery completes — if a match
comes back it swaps the per-period MBID for the synthetic id +
the canonical name. Existing Last.fm radio routing logic stays
intact (Last.fm radios aren't a series).
- ``ListenBrainzManager._cleanup_per_period_series_mirrors`` —
one-shot consolidation sweeper runs in ``_cleanup_old_playlists``
+ deletes any legacy per-period mirror rows so the consolidated
rolling mirror is the only one left. Idempotent — only matches
per-period titles ("Weekly Jams for ..., week of ...") and never
the canonical rolling-mirror titles ("ListenBrainz Weekly
Jams").
- 11 new tests pin the detector + synthetic-id helpers; 236 total
across adapter + automation + lb-series suites green.
Sibling to the ListenBrainz Sync tab from Phase 1c.1. Last.fm Radio
playlists already live in the same ``listenbrainz_playlists`` table
as LB ones (``playlist_type='lastfm_radio'``) and run through the
same MB-track discovery worker, so this tab is intentionally thin
— list + render + delegate. Card click hands straight off to the
LB Sync-tab click handler since the downstream modal + state
machine are identical.
- ``webui/index.html``: new ``<button data-tab="lastfm-sync">``
+ tab content container between the LB tab and the existing
Import / Mirrored tabs. Plus a ``<script>`` tag for the new
module.
- ``webui/static/sync-lastfm.js`` (new): ``loadLastfmSyncPlaylists``
hits the existing ``/api/discover/listenbrainz/lastfm-radio``
endpoint, ``renderLastfmSyncPlaylists`` mirrors the LB card
shape with a ``📻`` icon + a ``.lastfm-playlist-card`` brand
class, click handler forwards to
``handleListenBrainzSyncCardClick``.
- ``webui/static/sync-listenbrainz.js``: the shared 500ms refresh
loop now iterates LB + Last.fm cards in one pass and treats
either tab as "active" for liveness. No second loop needed.
- ``webui/static/sync-services.js``: new tab-activation branch in
``initializeSyncPage`` mirrors the LB pattern.
- ``webui/static/style.css``: ``.lastfm-icon`` SVG (Last.fm "as"
logo, red), and ``.lastfm-playlist-card`` joins the unified
card selector group with the Last.fm-red accent
(``rgba(213, 16, 7, ...)``).
- ``web_server.py``: the lastfm-radio endpoint now includes
``track_count`` in its JSPF payload (same fix as the LB
endpoints last commit).
- WHATS_NEW entry added under 2.6.3.
Mirrors created from Last.fm radios participate in the same auto-
trim Phase 1c.1's cascade-delete hook does — when the LB manager
rotates a stale ``lastfm_radio`` row out of its 5-most-recent
window, the matching ``source='lastfm'`` mirror row is removed
along with it. Library files stay on disk.
225 tests across adapter + automation suites still green; this
commit adds no Python paths to test.
Two follow-ups to the LB Sync tab work:
1. **Track counts all showed 0.** The
``/api/discover/listenbrainz/*`` endpoints assemble a JSPF-shaped
payload but drop the cached ``track_count`` field from the
underlying ``listenbrainz_playlists`` row — the JSON the frontend
sees only carries ``title`` / ``creator`` / ``annotation`` / an
empty ``track`` array. The Discover-page renderer worked around
it by hard-coding a fallback of 50; the Sync-page renderer had
no such fallback, so every card displayed "0 tracks". Backend
now includes ``track_count`` directly in each playlist payload
(it's already in the cached row) so any frontend can render an
accurate count without resorting to a default. JS still falls
back to ``annotation.track_count`` and then ``track.length`` for
older callers.
2. **LB playlists never landed in Mirrored Playlists.** The
existing ``/api/listenbrainz/sync/start/<mbid>`` endpoint runs
the converted Spotify tracks through ``_run_sync_task`` — i.e.
it pushes them to the user's media server (Plex / Jellyfin /
Navidrome / SoulSync) as a server-side playlist. It does NOT
call ``database.mirror_playlist``. So no ``mirrored_playlists``
row gets created and the playlist can't be picked up by the
Auto-Sync scheduler, can't show up under the Mirrored tab,
doesn't participate in pipeline automations — the whole point
of the Sync-tab unification.
Tidal works because Tidal mirrors on tab load with raw tracks
then enriches via discovery. LB tracks only have provider IDs
*after* discovery, so the equivalent moment for LB is "discovery
complete". Added ``_mirrorListenBrainzAfterDiscovery(mbid)``
that pulls the matched ``spotify_data`` out of
``discovery_results`` and posts to ``/api/mirror-playlist`` via
the existing ``mirrorPlaylist`` helper. Hooked into both the
WebSocket and HTTP-poll completion handlers of
``startListenBrainzDiscoveryPolling``. UPSERT-keyed on (source,
source_playlist_id, profile_id), so re-running discovery is a
safe no-op refresh.
Result: any LB playlist the user discovers (from either the
Discover page or the new Sync tab) now lands in
``mirrored_playlists`` with ``source='listenbrainz'`` + matched
tracks carrying canonical ``extra_data`` JSON, ready for the
Auto-Sync refresh + sync pipeline wired up in Phase 1a + 1b.
Adds ``discover_tracks(tracks) -> List[NormalizedTrack]`` to the
PlaylistSource interface. Sources whose tracks already carry
provider IDs (Spotify, Tidal, Qobuz, YouTube, Deezer, Spotify
public, iTunes link, SoulSync Discovery) inherit a no-op default;
ListenBrainz + Last.fm override to run the matching engine.
This closes the last gap before LB / Last.fm / SoulSync Discovery
can land as Sync-page mirror sources: the refresh handler now
calls ``source.discover_tracks(...)`` whenever a source returns
tracks with ``needs_discovery=True``, so mirrored LB rows arrive
already discovered + ready for the sync pipeline. Previously, LB
playlists ran through a separate state-machine worker tied to the
Discover-page UI, with results stored in ``discovery_cache``
instead of ``mirrored_playlist_tracks.extra_data``.
Changes:
- ``core/playlists/sources/base.py`` — PlaylistSource switches from
Protocol to ABC so a concrete default for ``discover_tracks``
can live on the base class. The four real-work methods stay
``@abstractmethod``; instantiating an adapter that forgets one
fails loudly at construction.
- ``core/discovery/matching.py`` (new) — pure ``match_mb_tracks``
helper that runs Strategy-1-only matching-engine queries against
Spotify (primary) or iTunes (fallback). No state machine, no
discovery-cache writes, no wing-it stub — that richer flow stays
in ``core/discovery/listenbrainz.py`` for the Discover-page UI.
- ``ListenBrainzPlaylistSource`` + ``LastFMPlaylistSource`` take
an optional ``discover_callable`` constructor arg. Last.fm reuses
the LB implementation since the track shape is identical.
- ``bootstrap.build_playlist_source_registry`` accepts a
``discover_callable`` kwarg and wires it into LB + Last.fm
adapters.
- ``web_server.py`` boot constructs the discovery callable from the
existing matching engine + ``_discovery_score_candidates`` +
Spotify / iTunes clients, passes through to the registry.
- ``refresh_mirrored.py`` adds a small ``_maybe_discover`` helper
that calls ``source.discover_tracks(...)`` between fetch and
``to_mirror_track_dict`` projection — only fires when at least
one track has ``needs_discovery=True``, so the normal Spotify /
Tidal / etc. refresh path stays a zero-cost pass-through.
Tests:
- 5 new adapter tests: default no-op pass-through, LB discovery
with mixed matches/misses, LB no-callable fallback, Last.fm
shares the LB implementation, mirror-dict spotify_hint emit.
- 1 new automation test: end-to-end LB refresh with a stub
discover_callable proves the matched_data lands in
``mirror_playlist_tracks.extra_data`` after the registry
refresh + discover hop.
225 tests across adapter + automation suites green.
Phase 1a of the Discover-to-Sync unification. The mirrored-playlist
refresh handler used to branch per-source through a ~190-line
if/elif chain (Spotify, Spotify public, Deezer, Tidal, YouTube).
Each branch hand-built its own ``extra_data`` JSON for the matched-
data block. With every new source we considered for Sync-page mirror
support (ListenBrainz, Last.fm radio, SoulSync Discovery, iTunes
link), that chain would have grown a new elif.
This commit lifts the per-source logic into the existing adapter
layer and collapses the dispatch to a registry lookup:
- ``core/playlists/sources/deezer.py`` — new adapter so the registry
covers every source the refresh handler previously branched on.
- ``core/playlists/sources/bootstrap.py`` — single helper that builds
a populated registry from injected getter callables. Both
``web_server.py`` boot and the automation test fixtures call it,
so the two construction paths can't drift.
- ``core/playlists/sources/base.py`` — ``to_mirror_track_dict``
projection helper centralises the NormalizedTrack → DB-row
conversion (including the discovered/matched_data and
spotify_hint extra_data shapes the downstream sync + wishlist
consumers already expect).
- Spotify adapter now populates ``extra['discovered']`` + an
``extra['matched_data']`` block when fetching via the authed API,
so Spotify mirrors keep landing pre-discovered (matches the
pre-refactor contract pinned by
``test_spotify_refresh_writes_to_db``).
- Spotify-public adapter populates ``extra['spotify_hint']`` so the
discovery worker can skip its search step and jump straight to
enrichment for the known track ID.
- All artist-name fields now project to first-artist-only across
every adapter — matches the pre-refactor mirror_playlist DB shape
(``t.artists[0]``).
``refresh_mirrored.py`` shrinks ~190 → ~80 lines and keeps:
- the file/beatport unrefreshable-source filter,
- URL extraction from ``description`` via ``require_refresh_url``
for spotify_public + youtube,
- the Spotify-public → authed-Spotify fallback when the user is
signed in (handler-level branch, not in any adapter),
- the Tidal-not-authenticated soft-skip log (skip, not error),
- existing-extra_data preservation across refreshes,
- the ``playlist_changed`` automation event emit on track-set delta.
Test scaffolding:
- ``_build_deps`` in ``tests/automation/test_handlers_playlist.py``
now builds a default registry from the passed clients via
``build_playlist_source_registry``, so existing refresh tests
exercise the same path without per-test changes. New tests cover
Tidal-not-authed soft-skip, Deezer refresh writes plain tracks,
YouTube refresh reads URL from description, and Spotify-public
uses authed Spotify when signed in.
- 4 new adapter tests for Deezer projection +
``to_mirror_track_dict`` (minimal track, Spotify matched_data,
Spotify-public spotify_hint).
- ``playlist_source_registry`` field on ``AutomationDeps`` defaults
to ``None`` so the other 5 automation test files (which don't
exercise refresh_mirrored) keep working unchanged.
220 tests across automation + adapter suites green.
New iTunes Link tab between Deezer Link and YouTube. Accepts album,
track, and playlist URLs from music.apple.com / iTunes. Pulls the
tracklist, runs it through the same discovery -> sync -> download
pipeline as the other link tabs.
Apple Music playlists go through amp-api with a Bearer JWT scraped
from the SPA. The legacy meta-tag and inline `"token":"..."` paths
are gone in the current music.apple.com SPA, so the extractor now
walks the page's `<script src>` list (prioritising index/chunk/main
bundles), fetches up to 8 JS bundles, regex-matches JWT-shaped
strings, and base64-decodes each payload to confirm it carries
Apple media-api claims (`root_https_origin`, or `iss + iat + exp`)
before trusting it. Filters out analytics / error-reporter JWTs that
also ship in the bundle.
Tokens are cached at module scope for 6h behind a threading.Lock so
the three-worker discovery executor doesn't thunder-herd Apple on
cold start, and amp-api calls go through a single helper that on
401 invalidates the cache, refetches the page, force-refreshes the
token, and retries the request once. The playlist fetcher memoises
the page HTML for the cache-miss path so we don't refetch it for
every paginated `/tracks` page.
spotify_public discovery worker accepts the new platform shape so
iTunes Link reuses the same matching code path as Deezer Link and
Spotify-public. UI bits live in the sync-services.js iTunes Link
tab, with platform plumbing through wishlist-tools.js for the
multi-source state map.
Extract repeated DB tag payload construction into a new _build_library_tag_db_data(track_data, album_genres) helper and replace in multiple endpoints. The helper builds the metadata dict (title, artist_name, track_artist, album_title, year, genres, track/disc numbers, bpm, track_count, thumb_url) and populates artists_list by splitting track_artist on ';'. Added tests (tests/test_library_tag_payload.py) to verify artists_list creation, genre propagation, thumb_url selection, and fallback behavior when track_artist is missing. This reduces duplication and ensures consistent tag payloads across tag-preview, batch preview, and tag-writing flows.
Update version references for the 2.6.2 release: change the workflow dispatch input description and default to 2.6.2 in .github/workflows/docker-publish.yml, and update the _SOULSYNC_BASE_VERSION constant in web_server.py to 2.6.2 so release metadata and build/version strings reflect the new patch release.
Three problems wrapped into one pass on the Playlist Auto-Sync surface:
1. Visual: the manager modal had its own vibe (radial gradient, pill
tabs, sky-blue chrome) that didn't line up with the rest of the
app. Reworked the modal shell, KPI summary, live pipeline monitor,
tab bar, schedule board sidebar, and column cards to use the
standard SoulSync patterns — gradient `#1a1a1a → #121212`,
accent-tinted 1px border, 20px radius, underline tabs, dense dark
card pattern that Automations + Library pages already use. Modal
now uses near-full screen so there's room for the schedule board
without horizontal scroll pain. Run history cards followed the
same path: slim horizontal row mirroring `.automation-card` plus
an expanded detail that mirrors the Automations run-history modal
(stats-grid + facts row + result pills + log section).
2. Hang: the previous SQL fix for the run-history "in library" count
added `COLLATE NOCASE` on the join columns of `tracks` and
`artists`. SQLite can't use `idx_artists_name` or `idx_tracks_title`
when the comparison collation doesn't match the column collation,
so the join did a full table scan per mirrored playlist track.
~18s per playlist × 30 playlists = `/api/mirrored-playlists` hung
indefinitely and the modal stayed at "Loading schedule…" forever.
Switched the join back to case-sensitive equality (~6ms per
playlist, 3000× faster). Spotify names canonicalize to the same
form as library imports so the recall loss is in the rounding
error of pure case-only mismatches.
3. Slowness: even after the hang fix, each modal open spent ~1.5s
gathering per-playlist status counts. The endpoint looped
`get_mirrored_playlist_status_counts(playlist_id)` per row, which
opened a fresh SQLite connection + PRAGMA setup each time. Added
`get_all_mirrored_playlist_status_counts(profile_id)` which
returns counts for every mirrored playlist owned by the active
profile in 4 batched `GROUP BY` queries over a single connection.
Modal load dropped to ~280ms.
Also fixed: `tracks.artist` reference in `get_mirrored_playlist_status_counts`
that never worked since the schema went relational — the query threw
"no such column", got swallowed by the try/except, and the in-library
count silently defaulted to 0 on every playlist. Rewired to join
through `artists`.
`get_mirrored_playlist_status_counts` (single-playlist) kept for
callers that still want it, but the modal endpoint uses the batched
version.
Persist per-playlist pipeline run snapshots from the shared playlist pipeline, expose a history API, and upgrade the Auto-Sync modal with live pipeline monitoring, Run now controls, and a runs-style history tab.
Expose playlist-native run and status endpoints that reuse the shared mirrored playlist pipeline engine while routing progress into playlist UI state.
Add a Run Pipeline action to mirrored playlist cards and modals with live status polling, and make the shared pipeline lock atomic for manual and scheduled callers.
