Closes the last acquisition gap — user-initiated downloads. A blocklist isn't
a censor, so search + discography stay fully visible; instead the download
ACTION is gated, visibly and overridably:
- Download modal (start-missing-process): an up-front check — if the WHOLE
album or artist being downloaded is blocklisted, return 409 {blocked:true}
with the entity, before starting a batch. The modal shows "X is blocklisted
— download anyway?" and re-POSTs with ignore_blocklist:true on confirm
(threaded onto the batch so the Phase 2a per-track filter skips it).
Scattered single-track bans still fall through to the 2a filter quietly.
- Manual /api/download (search-result download): source-file-centric, so it
matches the blocked ARTIST by name; same 409 + confirm + override. search.js
now sends artist/title so the guard has something to match.
- Precedence confirmed: force-download overrides "already owned", NOT a ban
(the 2a filter runs on the force-expanded missing list).
Frontend: shared confirmBlockedDownload() helper; modal + search callers
handle the blocked response and retry with the override.
Tests: manual download blocked-by-name / unrelated-allowed / override-passes,
and the modal up-front 409 for a blocked album. 8 blocklist API tests pass.
Completes Phase 1 on top of the backend (43c798a7):
- Cross-source backfill: core/blocklist/backfill.py is a pure injected-resolver
core (resolve only missing sources, never raises); core/blocklist/runtime.py
wires the real metadata clients with a confident name-match (exact
significant-token equality; album/track also require the parent artist when
both expose one — no wrong IDs hung on an entry). Resolution runs
synchronously at add time, so a ban is cross-source from the first scan;
the artist name-fallback in matching covers any gap.
- API: GET/POST/DELETE /api/blocklist (profile-scoped) + /api/blocklist/search
(thin wrapper over the manual-match service search on the active source, so
the modal needn't know the source). Add resolves the other sources before
storing.
- Modal (webui/static/blocklist.js): tabbed Artists/Albums/Tracks in the
revamp design language (accent light-edge, pill tabs, debounced search with
spinner + out-of-order guard, per-result Block, "currently blocked" list
with a match-status star and per-row remove). Opened by a new "Blocklist"
button on the watchlist page, next to Download Origins.
Tests: 5 backfill (fill-missing-only, None/exception handling, arg shape) + 4
API (search proxy, add→backfill→list→delete round trip, validation). Modal
registered in the script-split onclick-coverage test; JS syntax-checked.
Self-review of df929dc0 found one gap: the crossfade preloader hits
/stream/library-audio with the file PATH, which 404s for a streamed (not
disk-mounted) Navidrome track — main playback worked, crossfade didn't.
/stream/library-audio now uses the same _build_library_stream_url fallback on
a disk-miss (resolving the song id from the new track_id param, or a DB
lookup by path), and the preloader passes next.id. Crossfade now works for
streamed libraries too.
Review also confirmed (no change needed): /api/stream/status returns only
status/progress/track_info/error_message — the Subsonic token in stream_url
never reaches the browser; it stays server-side and the browser only hits
/stream/audio. Proxy verified live: Range forwarded, 206 + Content-Range/
Accept-Ranges passthrough, body streamed in 64KB chunks, upstream closed.
mlody95pl: Navidrome sync works, playback fails ("Failed to resume playback").
Root cause: SoulSync plays library tracks by reading the file off its OWN
disk (/api/library/play → resolve path → serve bytes). "Report Real Path"
gives the correct path STRING, but that's Navidrome's container path — the
files still have to be mounted into the SoulSync container to open them, and
the user's compose has /music commented out. So disk resolution 404s.
Navidrome is a streaming server, so requiring a disk mirror to play from it is
the real limitation. Now, when a library file isn't on SoulSync's disk and the
active server is Navidrome, playback streams through the server's own Subsonic
/rest/stream API — no mount needed:
- NavidromeClient.build_stream_url(song_id, max_bitrate) — token-authed
/rest/stream URL (mirrors build_cover_art_url; password never exposed).
- /api/library/play: on disk-miss, _build_library_stream_url (Navidrome-only;
uses the song id sent by the player, or a DB lookup by file_path) sets a
session stream_url instead of failing.
- /stream/audio: proxies that stream_url with Range passthrough so HTML5
seeking works, streaming upstream bytes through in 64KB chunks (no full-file
buffering).
- session state gains stream_url; the two library-play callers now send the
track's server id.
Disk playback is unchanged (file_path path still wins when the file resolves),
so Plex/Jellyfin and mounted-Navidrome setups behave exactly as before.
Tests: 7 on the URL builder (auth shape, no-transcode default, maxBitRate,
guards) + 4 on the play-fallback routing (navidrome-only, passed-id vs
DB-lookup, none). 200 navidrome/stream/media-server tests pass.
Follow-up to 603b7a2a (tokens in the config store): /api/settings GET
returns the whole config dict, which would now include the OAuth access +
refresh tokens. The settings UI has no field for them — strip the key from
the response. dict(config_data) is a SHALLOW copy of live state, so the
section is rebuilt rather than popped in place (verified: response clean,
live config intact, token survives a settings save since POST merges
per-key).
Boulder: "Taylor Swift shows only 8 albums, nothing before 2022, no singles,
no EPs" — for every artist (actually: every WATCHLIST artist). Traced live:
get_artist_albums caches its result under an UNQUALIFIED key (no limit/page
info), and the watchlist's new-release probe (limit=5, max_pages=1 — the
April "reduce watchlist API calls ~90%" optimization) stored its truncated
single page in that same slot. The artist detail page reads the cache first,
so a watchlisted artist's page showed only the newest handful of releases —
newest-first, hence "nothing before 2022" — re-poisoned on every scan, with a
30-day TTL. When the source-priority fetch comes back tiny, the page's
fallback path quietly serves it, so the symptom looked like a discography
filter bug. Not related to the #808 matching change (that is a pure max(),
provably additive).
Three pieces:
- get_artist_albums tracks whether the fetch stopped while more pages
existed (truncated) and only caches COMPLETE discographies. Individual
albums keep their opportunistic caching — they're complete entities
regardless of pagination. A small real discography that fits one page
stays cacheable even under max_pages=1.
- MetadataCache.purge_artist_album_lists(): delete the already-poisoned
album-list entries (TTL would have kept them for weeks); lists rebuild
lazily on the next artist-page visit.
- one-time startup purge in web_server, config-guarded
(maintenance.album_cache_purge_v1), mirroring the startup-repair pattern.
Tests: truncated probe never stores the list (but still returns its page),
complete multi-page fetch caches, and a genuinely-small one-page discography
under max_pages=1 still caches. 1087 spotify/cache/watchlist/artist tests
pass.
carlosjfcasero: 'Champagne Supernova (OurVinyl Sessions)' is in the library
but the artist page shows it unowned and wishlist cleanup never removes it.
Measured with the real catalogs: Deezer/iTunes title the TRACK with the
qualifier while the library track is bare (the qualifier lives in the album
title) — and _calculate_track_confidence crushed that pair to ~0.17: the
"clean" titles keep parenthetical words, so the length-ratio penalty treats
'Champagne Supernova' vs 'Champagne Supernova (OurVinyl Sessions)' as
different songs. (Also confirmed: the OurVinyl release is absent from
Deezer's discography for the artist, so the standard page's 25-release list
not showing it is the source catalog, not a bug.)
Fix 1 — core.text.title_match.strip_redundant_context_qualifiers: a
parenthetical qualifier whose text appears (word-bounded) in the db track's
ALBUM title — or in the other title — restates release context and is
stripped for a comparison variant scored with its own length guard. Genuine
version markers keep their penalty: '(Live)' on a studio album appears in no
context and still blocks; '(Live)' on 'Live at Wembley' correctly matches —
owning the live album IS owning the live cut. Wired into
_calculate_track_confidence, so every check_track_exists consumer (wishlist
cleanup, discography dedup, repair jobs) benefits.
Fix 2 — the artist-page ownership endpoint's album gate: when album-aware
narrowing eliminates EVERY library candidate (the source's album naming just
doesn't resemble the library's — 'Jillette Johnson | OurVinyl Sessions' vs
'Champagne Supernova (OurVinyl Sessions)' ~0.5), fall back to artist-wide
title matching instead of declaring everything unowned off a failed
album-NAME comparison.
Tests: 8 — the exact reported pair end-to-end through check_track_exists,
word-boundary containment ('live' in 'alive' doesn't count), version-marker
safety both ways, and prefix songs still blocked. 1125 matching/wishlist/
library tests pass.
User ask: "a modal that lists the tracks downloaded via watchlist" — extended,
as discussed, to playlists too. One modal, two tabs, opened from the Watchlist
page (watchlist tab preselected) and the Sync page (playlists tab) — same
shared-modal-different-entry-points UX as the rest of the app.
The data: library_history recorded which SERVICE a file came from but never
what TRIGGERED it. New origin/origin_context columns (migration + index) are
written once at the import chokepoint via core/downloads/origin.py, a pure
tested deriver that reads, in priority: an explicit _dl_origin stamp (set at
batch-task creation for direct playlist batches, where the playlist context
otherwise only survived in folder mode), the wishlist provenance already
riding in track_info.source_info (watchlist_artist_name / playlist_name —
watchlist_scanner has stamped these for ages), and the folder-mode playlist
thread. Manual downloads stay unclassified by design. History starts from
now — provenance can't be conjured retroactively.
API: GET /api/download-origins?origin=watchlist|playlist (paged) and POST
/api/download-origins/delete — deletes the file on disk (resolved through the
shared container/host path resolver), the matching library track row, and the
history entries; a file that refuses deletion keeps its row and reports the
error instead of lying.
UI: webui/static/origin-history.js — tabbed modal in the revamp design
language (accent light-edge, pill tabs, entry rows reusing the
library-history-entry components), per-row delete + select-all bulk delete
with honest result toasts, empty/loading states, per-tab totals.
Tests: 8 — deriver priority/shapes (incl. the exact watchlist_scanner
source_info shape and JSON-string survival), origin filtering + counts,
row fetch/delete isolation between origins, delete-track-by-path.
Measured during a live album download: ~4m15s per track in post-processing
(normal is ~20s), with the time vanishing silently inside embed_source_ids —
up to 5 MusicBrainz calls per track crawling against a degraded musicbrainz.org
while the MB enrichment worker kept eating the same ~1 req/s per-IP budget.
Only Spotify/Last.fm/Genius were in the yield set; MusicBrainz, Deezer, iTunes,
Discogs etc. kept grinding through downloads.
Policy (new core/enrichment/yield_policy, tested):
- downloads active -> ALL enrichment workers yield (post-processing touches
every metadata source). listening-stats (local-only) and repair
(user-scheduled) intentionally keep running.
- discovery active -> the API-contention five yield (spotify/itunes/deezer/
discogs/hydrabase) — discovery never paused anything before, despite the
pause helper literally defaulting to label='discovery'.
- user overrides and user-paused bookkeeping keep their existing semantics;
the dashboard yield_reason label now says WHICH foreground work caused it.
Observability (the 4-minute silence can never come back):
- every source lookup is timed; >2s logs a warning NAMING the source and
duration (core/metadata/source.py _call_source_lookup)
- the pipeline always logs "Metadata enhancement took X.Xs" per track
7 policy tests (incl. the motivating case: MB yields to downloads, keeps
running during discovery); 277 pipeline/enrichment tests pass.
A user reports ~0.7 MiB/s RSS growth; the one theory offered so far
(connection leak) was debunked, so instead of guessing: measure. New
core/diagnostics/memory_tracker wraps tracemalloc behind three GET endpoints
the user can drive from a browser:
/api/debug/memory/start begin tracing + baseline snapshot (idempotent)
/api/debug/memory/report top allocation sites by GROWTH since the baseline
(?top=N), with traced totals + process RSS so we
can see how much of the real growth tracing
accounts for; 15-frame tracebacks name the caller
/api/debug/memory/stop end tracing, free trace bookkeeping
Opt-in by design — tracemalloc shadows every allocation while active, so it
never runs by default. RSS via psutil with a /proc fallback.
Tests: report-without-tracking returns a hint (not an error); a real
start->hog->report->stop roundtrip attributes a genuine 5MB allocation to the
test file (fun fact encoded in the test: 'x'*1000 constant-folds into ONE
shared string and traces as ~40KB — the hog must allocate at runtime); the
stat formatter is duck-typed and unit-tested.
The tile's liveness was wired to sync:progress / discovery:progress — both
ROOM-scoped (only clients watching a specific playlist receive them), so the
dashboard tile would basically never light. And the scheduled auto-sync runs
as an automation, reporting on automation:progress — the wrong tile.
The 1s sync emitter now also sends an UNSCOPED sync:active heartbeat while any
playlist work is running anywhere: manual per-playlist syncs (sync_states),
the UI-triggered mirrored pipeline (playlist_pipeline_progress_states), and
scheduled auto-sync pipelines (running automations whose action_type is
playlist_pipeline / sync_playlist / refresh_mirrored). Emitted only while
active; the tile's 6s freshness decay handles the off. The dashboard listens
for the heartbeat alongside the (kept) room-scoped signals.
The payload has carried daily_budget {used, limit, exhausted} forever and the
dashboard rendered none of it. The avatar disc now wears a conic progress rim
that fills as the day's real-API budget is spent — green to 70%, amber to 95%,
red after — and flips purple once the worker has bridged to Spotify Free for
the rest of the day (using_free now included in the emit payload). Tooltip
carries the exact used/limit numbers.
Since the per-listener stream sessions refactor (Phase 3b), every browser gets
its own stream session — but the 1s 'tool:stream' socket broadcast still read
the legacy GLOBAL state (the DEFAULT session no real browser uses), so it told
every client "stopped" forever. The frontend skipped HTTP polling whenever the
WebSocket was up, so it only ever saw that wrong broadcast: the backend prep
downloaded the track, moved it into the session's stream folder and sat at
"ready" while the mini player showed nothing. Proxy users whose WebSockets
don't connect fell back to HTTP polling (session-correct) and streamed fine —
which is why this hid so well.
Fix: stream status is inherently per-listener, so stop pretending a global
broadcast can carry it —
- web_server.py: remove the 'tool:stream' emit from the tool-progress loop
(the broadcast thread has no request context; it can only ever see DEFAULT)
- media-player.js: the status poller always polls /api/stream/status (resolves
the caller's own session from the cookie); drop the dead broadcast handler
- core.js: unwire the 'tool:stream' socket listener
Observability fix that made this undebuggable: core/streaming/prepare.py used
getLogger(__name__) — outside the soulsync.* namespace where handlers attach —
so every prep log line (including failures) vanished from app.log. Moved to
get_logger("streaming.prepare") + a regression test locking the namespace.
34 streaming tests pass; ruff clean; web_server compiles; JS syntax-checked.
The existing fallback (pipeline.py:1084) only ran inside
post_process_matched_download_with_verification — i.e. when a file *was*
downloaded and AcoustID retries were fully exhausted. If the retry *search*
itself found zero valid candidates (source returned nothing, or all failed
HiFi validation), the task was marked not_found and the fallback was never
reached, even though the quarantine already held N version-mismatch entries.
Fix: add try_version_mismatch_fallback to TaskWorkerDeps; in the "no valid
candidates" path of task_worker, invoke it before marking not_found when
is_quarantine_retry. Wired in _build_task_worker_deps via a new helper
(_try_version_mismatch_fallback_for_worker) that calls
try_accept_version_mismatch_fallback directly with the track's title and
artist and a reprocess lambda over _post_process_matched_download_with_verification.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Users manually match an album to the regular edition, but enrichment/
repair keeps treating it as the deluxe (missing songs, renumbered tracks).
Root cause: an album has TWO identities — the enrichment match
(spotify_album_id, which manual-match sets and the worker already honors)
and a SEPARATE canonical version pin (canonical_album_id, added by #777).
The canonical pin is what track-number repair / reorganize / missing-track
detection actually read, and library_manual_match never wrote it — so it
was resolved independently and landed on the deluxe edition.
(So #777 did NOT solve #758: it added canonical pinning, but manual
matches didn't write the pin.)
Fix: a manual ALBUM match on a canonical-recognised source now also pins
AND locks the canonical version to the chosen release:
- new canonical_locked column (same migration pattern as the other
canonical cols).
- set_album_canonical(..., locked=False) gains an atomic WHERE-clause
guard: an auto write can't overwrite a locked pin; a manual write
(locked=True) always wins. get_album_canonical exposes `locked`.
- library_manual_match pins canonical for album matches via the pure
should_pin_manual_canonical(entity_type, source).
The auto resolve job already skips already-pinned albums, so the lock is
protected on two fronts; the new guard also covers any future
re-resolution. A new manual match still overrides.
18 tests: the pure gate (+ a sync-invariant test vs _ALBUM_ID_COLUMNS)
and the DB lock seam (auto can't clobber a manual lock; manual overrides;
auto-over-auto still works). Additive — locked defaults False, so the
auto path is unchanged unless a manual lock exists. Full suite clean.
#798 follow-up. The enrichment worker's own loop already bridges to the
no-creds Spotify Free source during a rate-limit ban (its guard checks
is_spotify_metadata_available()). But the resume button's pre-check
(_spotify_resume_pre_check) blocked resume on ANY rate-limit with no
awareness of Free — so a Free-opted-in user who got rate-limited was
locked out of restarting the worker, unable to fall through to the free
API.
Fix: the resume guard now mirrors the worker. Block only when
rate-limited AND nothing can serve (plain auth, no Free) — where resuming
would just sleep out the ban. When Free is available it serves during the
ban (is_spotify_authenticated() is False while banned, so
is_spotify_metadata_available() reports the free source), so resume is
allowed and the worker bridges via Free, then returns to real auth once
the ban lifts. Stays real-API-first; Free is only the bridge.
The rule is pinned in a pure helper should_block_rate_limited_resume()
next to the other gate functions, with 3 tests. Full suite clean (only
pre-existing soundcloud /app env failures remain).
The post-scan reconcile previously ran AFTER the worker's 'finished'
signal, which flips db_update_state status to 'finished'. Automations
wait for a scan by polling that status, so they stopped waiting before
the reconcile ran — and the dashboard/Tools card showed "Completed" then
flipped to "Reading file tags…". For incremental scans this was
invisible (sub-second); for a full refresh it was a real gap (a chained
automation would fire minutes before the IDs were filled).
Fix: the worker now routes completion through _emit_finished(), which
runs self.post_scan_hook (the reconcile) FIRST, then emits 'finished'.
The hook is injected by the web layer (it owns path resolution). So:
- status stays 'running' through the reconcile,
- the reconcile pushes its phase ("Reading file tags for N new tracks…")
and per-track progress through the SAME db_update_state callbacks the
scan already uses — so automations, the dashboard card, and the Tools
page all see it for free and wait for it,
- 'finished' is emitted exactly once, AFTER the reconcile — race-free, no
status blip a poll could catch,
- best-effort: a hook exception never blocks 'finished', so a scan can't
get stranded as perpetually 'running'.
Both scan entry points (_run_database_update_task, _run_deep_scan_task)
set the hook before run()/run_deep_scan(); the redundant post-run calls
are removed.
5 ordering tests pin the contract (hook-before-finished, finished still
fires without a hook, hook exception doesn't block finished, hook gets
the worker). Full suite clean (only pre-existing soundcloud /app env
failures remain).
Extends the manual "Import IDs from File Tags" backfill so newly-scanned
files get their embedded provider IDs pulled into the DB automatically —
no button press needed to keep up with new music.
How it works:
- insert_or_update_media_track now returns 'inserted' / 'updated' / False
(truthy-compatible; existing `if track_success` callers unaffected) so
the scan worker can tell a genuinely new row from an update.
- DatabaseUpdateWorker collects the ids it newly INSERTED this run
(self._new_track_ids) across all insert paths (Plex/Jellyfin/deep).
- After run()/run_deep_scan(), web_server calls _reconcile_after_scan(),
which gap-fills embedded IDs for just those new tracks. Runs as a
post-scan pass (the scan loop itself is untouched/fast — the media
server API never exposes these custom IDs, so the file must be read
once regardless; batching at the end keeps it out of the hot loop and
best-effort so it can never abort a scan). A progress phase ("Reading
file tags for N new tracks…") surfaces the full-refresh tail.
Shared engine:
- New reconcile_library() in core does the paging + lazy parent-map
loading (only loads albums/artists actually referenced — cheap when
scoped to a few new tracks) + per-page commits. BOTH the manual button
and the scan hook call it, so there's one tested orchestration, no
duplication. The backfill job was refactored onto it.
Same hardened safety: gap-fill only, atomically guarded against
overwrite, schema-introspected, idempotent. Scoped to new arrivals for
incremental/deep; full refresh re-inserts everything as new (recovering
the IDs a full-refresh wipe destroys).
+10 reconcile tests (reconcile_library scope/idempotency/progress/stop +
the engine). Full suite clean (only pre-existing soundcloud /app env
failures remain).
Files SoulSync (or MusicBrainz Picard) already tagged carry Spotify /
iTunes / MusicBrainz / Deezer / Tidal / AudioDB / Genius / Last.fm IDs in
their metadata. Enrichment workers gate their queues on
{provider}_match_status IS NULL, so reading those IDs back and gap-filling
the {provider}_id + match_status='matched' columns lets the workers skip
the API lookup entirely — big API savings on an already-tagged library.
New manual job in Tools -> Database & Scanning ("Import IDs from File
Tags"): scans every library file, reads embedded IDs, fills any that are
missing in the DB. Background job + progress card, mirroring the
write-tags-batch pattern.
core/library/embedded_id_reconcile.py (pure + tested):
- plan_reconcile(): gap-fill plan for a track + its album + artist. Only
empty id columns are planned; a disagreeing embedded id is a conflict,
never applied.
- apply_reconcile_plan(): one guarded UPDATE per id column —
WHERE id=? AND (col IS NULL OR col=''). The guard makes the fill atomic:
if an enrichment worker matched the same entity between our read and
this write, the UPDATE affects 0 rows instead of clobbering it. Columns
are introspected so a schema missing a provider's columns is skipped.
- reconcile_track_row(): per-track orchestration (id extraction, plan ->
apply, keeping the in-memory parent maps fresh for sibling tracks).
Job hardening: paged track scan (bounded memory), per-page commits (don't
starve concurrent workers), per-file try/finally (one bad file can't abort
the run), counters from real rowcount.
Scope: 19 column-fills across 8 providers. MB *recording* (track) id is
left out (UFID frame the reader doesn't surface; Vorbis key ambiguous) —
MB album+artist are covered. Amazon/ASIN deliberately excluded (ASIN is a
different namespace than the worker's amazon_id). All target columns
verified against the live schema.
Purely additive: new module, two new endpoints, one new Tools card —
no existing behavior changed. 20 unit tests (incl. the concurrency guard).
Full suite clean (only pre-existing soundcloud /app env failures remain).
AcoustID returns a recording's title/artist in their ORIGINAL script
(e.g. "久石譲" for Joe Hisaishi) while SoulSync's expected metadata is
romanized/English. A correct download then fails verification on two
walls: the title can never clear the 0.70 similarity bar cross-script,
and the only skip path that ignores the title required a near-perfect
0.95 fingerprint plus a resolved alias. Result: every non-English
artist trips it. Two complementary fixes, per the reporter's two ideas.
Graceful fix (automatic):
- New pure core/matching/script_compat.py detects when two strings are
in genuinely different writing systems (CJK/Hangul/Cyrillic/Greek/
Arabic/Hebrew/Thai vs Latin). Accented Latin (Beyoncé, Sigur Rós)
stays Latin — no false trigger.
- acoustid_verification.py: when the EXPECTED artist and the matched
artist span scripts AND the artist is confirmed via the existing
MusicBrainz alias bridge, SKIP instead of quarantine, without the
0.95 floor (the 0.80 trust floor already gates the fingerprint).
- Deliberately narrow: keyed on the ARTIST spanning scripts + being
confirmed. A same-script artist with only a cross-script title keeps
the stricter 0.95 floor, so the #607 wrong-file protection (Kendrick
R.O.T.C, low-fingerprint Japanese-title) is untouched.
Per-request toggle (manual escape hatch):
- New "Skip AcoustID verification" checkbox in the download-missing
modal beside "Force Download All".
- skip_acoustid threads request -> batch -> per-track track_info ->
download context (same path as _playlist_folder_mode), landing on
the existing _skip_quarantine_check='acoustid' bypass. No new
mechanism; only the AcoustID gate is bypassed (integrity/bit-depth
still run).
Tests:
- tests/matching/test_script_compat.py — script-boundary cases.
- test_acoustid_skip_logic.py — Joe Hisaishi SKIPs at 0.85; unconfirmed
cross-script artist still FAILs; same-script low-fingerprint still
FAILs.
- test_downloads_candidates.py — toggle injects the bypass; absent
toggle keeps verification.
Full suite: 5169 passed; only pre-existing soundcloud /app env failures
remain. Zero regressions.
The Settings dropdown reverted 'Spotify Free' because _isMetadataSourceSelectable
reads _lastStatusPayload.spotify.free_installed, but that payload comes from the
WebSocket status:update push (_build_status_payload) — which sent the raw spotify
dict. The availability flags were only added to the GET /status endpoint, so the
frontend never saw free_installed and bounced the selection.
Extract _spotify_status_with_availability() (metadata_available + free_installed)
and use it in BOTH _build_status_payload (WebSocket) and get_status (HTTP poll),
so they can't drift. Now 'Spotify Free' is selectable when SpotipyFree is
installed.
Consistency fix: Spotify Free is now its own entry in the metadata-source
dropdown (alongside Spotify / iTunes / Deezer / MusicBrainz) instead of a
side-toggle. Stored as fallback_source='spotify' + spotify_free=true so all
downstream 'spotify' routing and the spotify_* columns are unchanged.
Refined gate model (no toggle):
- Connected user (has credentials) -> official; bridges to free AUTOMATICALLY
during a rate-limit ban (no opt-in needed).
- No-auth user -> must pick 'Spotify Free' in the dropdown; then free serves.
- Never opted into Spotify (no creds, didn't pick it) -> free never runs, so no
surprise scraping. _free_wanted() = has_credentials OR picked-spotify-free is
the guard.
- AUTHED + healthy -> official always; free never opens.
UI: dropdown gains 'Spotify Free (no credentials)' (selectable when the package
is installed — surfaced via status.free_installed, since selecting it is the
opt-in and can't depend on having selected it); load/save map the dropdown value
to the (fallback_source, spotify_free) pair; old checkbox removed.
Gate model pinned by 6 scenario tests (connected/healthy, connected/ratelimited
bridge, no-auth picked, no-auth not-opted-in, package-missing). 117 tests green.
Surfaces the opt-in Spotify Free source so it's usable end-to-end:
- Settings: 'Enable Spotify Free (no credentials)' toggle that saves
metadata.spotify_free (load + save wired). Clear best-effort/limitations note.
- config-status: adds spotify.metadata_available (configured OR free-available),
keeping the configured flag = has-credentials so the Connections indicator
stays honest. Search source picker shows Spotify when metadata_available.
- status payload: adds spotify.metadata_available; the Settings primary-source
selector now allows picking Spotify when authed OR free-available.
Verified gate composition: OFF by default (no surprise scraping); ON + no auth +
installed -> available & serving; AUTHED -> official always wins (free never
runs); missing package -> gracefully unavailable. JS + integrity + 111 tests green.
A manually-fixed mirrored track silently reverted to 'Wing It' after re-running
discovery. Two compounding causes:
- extra_data is MERGED on save (update_mirrored_track_extra_data), and the
manual-fix DB write (web_server.py) didn't clear the prior wing_it_fallback
flag — so a track fixed after being a Wing It stub kept wing_it_fallback=True.
- the Playlist Pipeline pre-scan checked wing_it_fallback BEFORE manual_match
(if/elif), so the stale flag won: the track was re-discovered and, on a miss,
fell back to Wing It — discarding the user's pick.
Fix: extracted the pre-scan gate into core.discovery.manual_match.should_rediscover
(manual_match checked FIRST = authoritative, regardless of leftover flags), and
the manual-fix write now also clears wing_it_fallback/unmatched_by_user. Behavior
is identical for every other branch — only the manual-vs-wing-it ordering changes.
Tested at the seam incl. the exact regression (wing_it_fallback + manual_match
both set -> skip). 227 discovery tests green.
After the duplicate-id ambiguity guard, an owned artist reached via a source
link (e.g. Kendrick's Deezer link) fell back to the bare source-only view
instead of the rich library view, because the URL path carries no name and the
duplicated id alone can't be matched.
get_artist_detail now resolves the source artist's name (reusing the #775
link-resolver's per-source artist fetch) when the id lookup is ambiguous and no
name was passed, then retries the library upgrade by name. So an owned artist
lands on the full library view; a genuinely-unowned one still renders
source-only (now with its name pre-resolved). Unique ids are unaffected.
Follow-up to the bare-ID footgun: a bare number like 525046 carries no
source and no entity type, so it resolved to whatever album happened to own
that id (a user pasting Kendrick's Deezer artist id got an unrelated album).
Now the resolver accepts provider URLs (and the explicit spotify: URI) only;
a bare/unrecognized string is rejected and the dropdown surfaces a hint to
paste a full link. URL parsing + album/track resolution are unchanged.
New 'Link / ID' input on the Search page: paste a Spotify / Apple Music /
MusicBrainz / Deezer URL (or a bare ID) and it's looked up directly on the
owning source — no fuzzy search, no scoring.
- core/search/by_id.py: source-agnostic parser (URL domain/path or bare-ID
format -> source,kind,id; numeric IDs fan out, first hit wins) + per-source
get-by-id dispatch + adapters projecting each provider's dict onto the
standard album/track card shape.
- /api/enhanced-search/by-id: thin additive route over resolve_identifier.
- Frontend: dedicated input that adopts the resolved source as active and
renders through the existing dropdown + download/import flow.
Purely additive — existing files are insertion-only; the resolver runs only
behind the new route. 29 seam tests cover parsing, shaping, fan-out, and
not-found.
The real path for discovery-modal syncs (Spotify-Public/Tidal/Deezer/Qobuz/
YouTube/iTunes-link/ListenBrainz/Beatport) is _start_source_sync ->
_submit_sync_task, which called _run_sync_task WITHOUT a sync_mode — so it used
the 'replace' default and never consulted the setting. (My earlier fix only
covered /api/sync/start, which these endpoints don't use — hence the log still
showing POST /api/spotify-public/sync/start ... mode: replace.)
_submit_sync_task now resolves the configured default via normalize_sync_mode
and passes it through, so all source syncs honor Settings > Playlist sync mode.
start_playlist_sync validated the resolved mode with 'if sync_mode not in
(replace, append): sync_mode = replace' — a pre-existing clamp two lines below
the config read I added, which silently downgraded a configured 'reconcile' to
'replace'. So config=reconcile resolved correctly then got clobbered, and every
sync ran replace regardless of the setting (and incognito didn't help — it's
backend, not cache).
Replace the hand-rolled clamp with a pure normalize_sync_mode(requested,
configured) helper (VALID_SYNC_MODES includes reconcile) so the resolution is
testable and can't silently drop a mode again. Regression tests cover
reconcile-from-config, request-overrides-config, and unknown->replace.
Replace mode (default) deletes + recreates the server playlist every sync,
which wipes its custom image, description, and identity. Add an opt-in
'reconcile' sync mode that edits the existing playlist in place — adds the
tracks now in the source, removes the ones gone — without destroying the
object, so the user's custom art/description survive.
- Pure planner plan_playlist_reconcile(current, desired) -> {add, remove}.
- Per-client reconcile_playlist: Plex addItems/removeItems on the same object;
Navidrome Subsonic updatePlaylist delta (songIdToAdd / descending
songIndexToRemove); Jellyfin add + remove-by-PlaylistItemId on /Playlists/{id}/Items.
- sync_service: reconcile branch with a replace FALLBACK (if a server's in-place
edit is unavailable/fails, sync still succeeds destructively — logged loudly).
- Default stays 'replace' (no behavior change). New Settings > Playlist sync mode
picker (replace/reconcile/append) backed by playlist_sync.mode; per-request
sync_mode still overrides.
- Reconcile skips the post-sync source-image push so a custom poster isn't
re-clobbered (the bug).
Tests: planner (add/remove/dedupe/order/empty) + reconcile-or-replace dispatch
(success / false-fallback / exception-fallback / no-method). Per-server in-place
API calls need dev validation against real Plex/Jellyfin/Navidrome.
NOTE: opt-in only; default behavior unchanged.
The mlm_save endpoint already fetches+validates the library track, so capture
its file_path and persist it on the manual match. This makes matches created
via the Manual Library Match tool re-resolvable after a rescan re-keys the
track (#787), matching the durability the Find & Add path now has.
Find & Add on the playlist-sync page only wrote sync_match_cache, which is
DELETEd wholesale after every DB scan — so the source->library pairing (and
the user's manual matches) reverted to 'extra'/red-dot on the next shallow
scan. The three match stores (sync_match_cache, manual_library_track_matches,
discovery extra_data) were disconnected and all pointed at tracks.id, which a
rescan re-keys (esp. Jellyfin/Navidrome GUIDs).
Unify the match so it's one durable fact, recorded once, honored everywhere:
- Find & Add also writes a durable manual_library_track_matches row (one-way;
the manual-match tool has no playlist to act on, so no reverse). Carries the
library file path.
- New library_file_path column (idempotent migration) + find_track_id_by_file_path:
re-resolve a stale library_track_id after a rescan re-keys the track, and
self-heal the row.
- The sync compare display's override lookup now falls back to the durable
manual match (resolve_durable_match_server_id) when sync_match_cache misses —
so the pairing persists across a scan instead of reverting to a red dot.
Purely additive: only adds matches when the cache returns nothing.
Tests: durable resolver (valid / stale-reresolve+self-heal / no-match / not-in-
playlist / missing-methods), file_path persistence + find_track_id_by_file_path.
The old per-download Retag Tool was limited (only native-pipeline downloads,
100-group cap, manual per-group) and did the wrong thing — it moved/reorganized
files instead of just tagging. It's superseded by the new Library Re-tag job
(whole-library, in-place) + the enhanced-library 'Write Tags' button.
Removed: the post-download record_retag_download ingestion hook (stops writing
retag_groups on every download), core/library/retag.py, the web_server state +
deps + /api/retag/* endpoints + the tool:retag WebSocket emit, the dashboard
card + both modals (index.html), the core.js socket handler, and the tools-page
wiring + help entry (wishlist-tools.js). Updated the import-pipeline test.
Verified: web_server parses, app + core imports OK, 392 tests pass, no live
references to removed symbols.
Left as inert (harmless) for a careful follow-up sweep: the retag_groups/
retag_tracks tables + their DB CRUD methods (no longer written/read), and the
now-orphaned retag JS helper functions (no entry point/wiring/socket calls them;
interspersed with wishlist functions, so not blind-deleted).
write_tags_to_file wrote the core fields + cover but never the source IDs
(Spotify/iTunes/MusicBrainz) the import post-process embeds. Added a focused
source.embed_known_source_ids() that writes ALREADY-KNOWN ids (from db_data)
via the canonical, Picard-compatible frame writer the import uses
(_write_embedded_metadata) — no API re-fetch, frames correct by construction.
write_tags_to_file now calls it whenever db_data carries id keys.
Fed from both paths: the enhanced-library 'Write Tags' button now carries the
track's spotify/itunes/musicbrainz ids, and the Library Re-tag job stamps the
matched album/track source ids onto each track. So both now write the full tag
set, not a subset.
- Bump _SOULSYNC_BASE_VERSION 2.6.5 -> 2.6.6 (the single source of truth that
propagates to the UI, backups, and the update check).
- Add the 2.6.6 What's New block (qBittorrent 5.2.0 login fix, Cover Art Filler
on-disk detection + file embedding + stricter matching, recommendations
explainability + Discover section, organize-by-playlist #780, nav/scroll perf
#783, dashboard mobile polish).
- Finalize the 2.6.5 block: it shipped in tag 2.6.5 but was left flagged
unreleased (so its notes never displayed) — stripped the flag + dated it per
the file's own release convention.
Adds get_recommendation_sources() — for each recommended similar artist it
resolves the polymorphic similar_artists.source_artist_id back to the display
names of the user's OWN artists (library + watchlist) that list it, by matching
against every provider-id column on both tables. The /api/discover/similar-artists
endpoint now attaches a 'because' array per recommendation so the UI can show
'because you have X, Y, Z' instead of just a count.
Seam tests cover: library + watchlist resolution across different provider-id
columns, dedup + name-sort, max_per cap, orphan source omission, profile scoping.
THE root cause of 'orb frozen, click does nothing visibly': when a socket is
connected, the orbs don't poll — update*Status() bails on socketConnected and
relies on server pushes. similar_artists was missing from BOTH the server emit
loop (_emit_enrichment_status_loop's workers dict) and the client dispatch
(core.js socket.on('enrichment:<id>')), so the orb never received status → never
updated. Clicks DID pause the backend (modal showed paused), but the orb visual
was frozen. Added the worker to the emit loop + the socket.on handler.
- get_stats now reports PERSISTENT counts from the DB (matched/not_found/pending
+ a progress.artists breakdown) instead of in-memory session counters, so the
dashboard orb tooltip and the Manage modal agree (was showing 0 vs 14 after a
restart) and it survives restarts — same approach as the other workers.
- Orb tooltip reads progress.artists ('Artists: 14 / 15 (93%)') like the rest.
- Worker now defaults to ON (running) instead of opt-in-paused; still honors a
saved pause across restarts. It self-paces (~3s/artist) and backs off on
MusicMap outages, so the orb spins/active like the others when there's work.
10 seam tests pass.
SoulSync standalone matches library tracks without Plex fetchItem,
reports missing counts correctly, and skips server playlist writes.
Automation re-syncs when the mirror grows; after sync finishes, starts
organize download (organize-by-playlist) or wishlist processing.
UI: Spotify URL playlist-folder controls, organize toggle layout in the
discovery modal, reload organize preference when reopening Download Missing.
Co-authored-by: Cursor <cursoragent@cursor.com>
Closes the gap where similar artists only existed for WATCHLIST artists: a new
background worker populates them for the whole LIBRARY, slotting into the
existing enrichment-worker pattern (bubble + Manage Enrichment Workers modal,
status/pause/resume, matched/not_found/pending/errors).
Per source-matched library artist → get_musicmap_similar_artists(name, 25)
(the same matcher the artist-detail page uses: fetches MusicMap names, matches
each to the user's source chain — primary + active fallbacks — returns only
matched artists) → store via add_or_update_similar_artist keyed by the artist's
metadata source id, the SAME key the watchlist scanner + artist map use, so the
two cooperate (idempotent upsert + retry_days window).
- core/similar_artists_worker.py: pure seams (pick_source_artist_id,
map_payload_to_store_kwargs, process_artist) + the threaded worker; skips
artists not yet source-matched; classifies not_found vs transient error
(retry after 30d).
- DB migration: similar_artists_match_status / _last_attempted on artists
(mirrors every other source worker's tracking columns).
- Registered in EnrichmentService + instantiated in web_server, DEFAULT-PAUSED
(opt-in) like Amazon — MusicMap is scraped/outage-prone + this is library-wide.
- SERVICE_ENTITY_SUPPORT['similar_artists']=('artist',) so the modal breakdown
('artists with / without similars') + Retry work; manual-match (inapplicable
to a relationship) is gated out via relationship:true.
- 10 seam tests; existing 80 enrichment tests still pass.
Note: keys under profile 1 (single-profile setups); multi-profile is future work.
The on-canvas overlay text can't be copied (and can't be grabbed mid-freeze), so
when perf mode is on ('d'), the frontend now also POSTs the render timings to
/api/discover/artist-map/perf ~1.5x/sec, which logs them as [ARTMAP-PERF] in
app.log. Lets the bottleneck be diagnosed from the server side with no manual
copying.
Persist organize_by_playlist on mirrored playlists and run playlist-folder
downloads from the auto-sync pipeline instead of the global wishlist phase.
Register SoulSync library rows after playlist-folder post-processing, route
failed organize batches to the wishlist correctly, and skip sync-time
unmatched wishlist only when organize download handles retries.
Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL),
re-mirror on refetch, and improve standalone playlist modals (re-analysis,
Open in Mirrored). Add filesystem missing-track detection and tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
The no-auth 'add by link' path scrapes Spotify's embed widget, which only ever
contains ~100 tracks and can't paginate — so big public playlists got
truncated. This adds an in-house anonymous fetch that pulls the FULL list:
- core/spotify_public_api.py: reads the anonymous web-player accessToken Spotify
already embeds in its own open.spotify.com page HTML (no app credentials, and
no rotating TOTP secret for us to maintain), then paginates
/v1/playlists/{id}/tracks 100 at a time until the whole playlist is pulled.
Returns the embed scraper's exact shape. Pure helpers + injected http_get so
it's unit-testable without the network.
- core/spotify_public_scraper.fetch_spotify_public(): tries the full fetch for
playlists; on ANY failure (or for albums) falls back to scrape_spotify_embed.
Worst case == today's behaviour, so the link path can't regress.
- web_server: the link-tab endpoint and the authed flow's last-resort scrape
now both go through fetch_spotify_public.
Scoped entirely to the spotify_public_* (no-auth) path — the authenticated
playlist sync is untouched. 11 tests (token extraction, normalisation,
pagination past 100, and the embed-fallback orchestration).
Caveat: rides Spotify's undocumented page-embedded token — expected to break
when they change their page; it degrades to the embed fallback, never crashes.
Needs a live click-through to confirm the token path works end to end (can't
hit Spotify from the test env).
Per-worker processing-order override + UI polish.
Feature — pin an entity group to enrich first:
- Each worker normally runs artist -> album -> track. A user can pin one
group (artist/album/track) to run first from the modal; the worker keeps
that group first until it's exhausted, then resumes the normal chain.
- core/worker_utils.py: read_enrichment_priority() (reads
<service>_enrichment_priority each loop, live) + priority_pending_item()
(shared, whitelisted query returning the worker's expected item shape;
Spotify/iTunes get album_individual/track_individual via a type map).
- A guarded ~6-line hook at the top of all 11 workers' _get_next_item.
CRITICAL: when nothing is pinned (default) the hook returns immediately,
so default enrichment order is byte-identical to before. Discogs (no track)
and Genius (no album) only honor their supported entities.
- core/enrichment/api.py: GET/POST /api/enrichment/<id>/priority (+ config_get
hook); POST validates the entity against what the source enriches.
- 14 new tests (helper shapes, exhaustion, route get/set/clear/validate).
UI:
- Refined hero header: identity + inline status left, single Pause right,
'now enriching' quiet sub-line; overall coverage % moved into the stats
section ('82% matched · 1,203 of 1,460'). Hero gently pulses while running.
- New processing-order strip: artist→album→track steps showing the live phase
(pulsing 'now'), pinned group ('first' + 📌), and done/remaining; click a
step to pin it, click again for auto.
py_compile clean across all 11 workers; 52 enrichment tests green.
Dashboard 'enrichment bubbles' could pause/hover but offered no way to
*manage* a worker. This adds a full management modal opened from a new
header button, covering all 11 enrichment sources.
Backend (testable core helper + seam tests; no live-DB dependency):
- core/enrichment/unmatched.py: pure, whitelisted SQL builders for the
unmatched browser. service/entity validated against a support map (never
interpolated raw); search + pagination bound as params; tracks join albums
for artwork; limit capped at 200.
- database/music_database.py: get_enrichment_unmatched() +
get_enrichment_breakdown() (the breakdown splits matched/not_found/pending,
which the existing get_stats().progress lumps together).
- core/enrichment/api.py: GET /api/enrichment/<id>/{unmatched,breakdown} on
the existing blueprint + a db_getter hook.
- web_server.py: wire db_getter=get_database.
- tests/enrichment/test_unmatched.py: 19 tests across builders, DB methods,
and Flask routes.
Frontend (vanilla, matches app conventions):
- webui/static/enrichment-manager.js: worker rail with live status + coverage
micro-bars, accent-themed detail panel (hero header, segmented matched/
not_found/pending stat cards, current item, pause/resume), and a searchable
paginated unmatched browser with inline manual match (reusing
search-service + manual-match) and retry (clear-match re-queues).
- Polish: entrance/exit motion, scroll-lock, Escape, refresh control,
flicker-free polling (in-place updates), skeleton loaders, relative
timestamps, per-worker accent theming, real dashboard logos reused at
runtime (with the same invert/circle treatment), responsive rail.
- index.html: header button + script include. style.css: full styling.
Reuses existing pause/resume, status, and manual search+assign endpoints.
Backend tests green (19 new + 11 existing enrichment tests).
