A manually-fixed mirrored track silently reverted to 'Wing It' after re-running
discovery. Two compounding causes:
- extra_data is MERGED on save (update_mirrored_track_extra_data), and the
manual-fix DB write (web_server.py) didn't clear the prior wing_it_fallback
flag — so a track fixed after being a Wing It stub kept wing_it_fallback=True.
- the Playlist Pipeline pre-scan checked wing_it_fallback BEFORE manual_match
(if/elif), so the stale flag won: the track was re-discovered and, on a miss,
fell back to Wing It — discarding the user's pick.
Fix: extracted the pre-scan gate into core.discovery.manual_match.should_rediscover
(manual_match checked FIRST = authoritative, regardless of leftover flags), and
the manual-fix write now also clears wing_it_fallback/unmatched_by_user. Behavior
is identical for every other branch — only the manual-vs-wing-it ordering changes.
Tested at the seam incl. the exact regression (wing_it_fallback + manual_match
both set -> skip). 227 discovery tests green.
Two complementary fixes to stop distinct artists ending up with the same source
id (the near-name collisions: ODESZA/odessa, Blance/Blanke, Lady A/Lady Gaga,
plus MusicBrainz's combined-score weak matches like Grant/Amy Grant):
- core/worker_utils.accept_artist_match() / source_id_conflict(): one shared,
tested gate. Rejects artist matches below 0.85 (stricter than the 0.80 used
for album/track titles, since short artist names false-positive easily) AND
refuses to store a source id a DIFFERENTLY-named artist already holds. A
same-named holder (one act across two media servers) is still allowed.
- Routed every artist-match worker through it: deezer, qobuz, tidal, discogs,
itunes, spotify (its scorer now uses the 0.85 threshold), audiodb, and
musicbrainz (conflict guard only — its matcher is combined-score, so the
guard is the net that catches its weak-name matches).
Centralizing in worker_utils avoids the copy-paste that let the original
album/track overwrite bug live in four workers at once. 17 new gate tests.
core/maintenance/dedupe_source_ids.py + scripts/dedupe_source_ids.py: find
source-id clusters held by differently-named artists (the enrichment-corruption
signature) and clear the id + match-status on those rows so the now-name-checked
workers re-derive each correctly on the next enrichment pass. Same-name
duplicates (one artist across two media servers) are left untouched.
Dry-run by default; --apply to write. 8 seam tests cover detection (corrupt vs
legit), dry-run safety, apply behaviour, and the no-op case.
The blind 'correct the parent artist's source id from an album/track match'
logic was copy-pasted into four enrichment workers; the Deezer fix only covered
one. AudioDB, Qobuz, and Tidal had the identical bug and would corrupt their own
id columns (and re-corrupt after any cleanup).
All three now gate the correction on a name match between the result's artist
and the parent artist (audiodb reads result['strArtist']; qobuz/tidal thread the
result artist name in from their callers, as Deezer does). Regression tests
cover mismatch-skips and match-corrects for each.
Root cause of the duplicate deezer_id corruption: when enriching an album or
track, _verify_artist_id 'corrected' the parent artist's deezer_id to the
search result's primary-artist id whenever they differed — with NO name check.
For a collaboration/compilation track (e.g. one our library credits to Jorja
Smith that lives on Kendrick Lamar's curated 'Black Panther' album), the result
resolves to Kendrick's album, so Kendrick's id (525046) got written onto Jorja,
Vince Staples, SOB X RBE, etc. — many artists ending up with the same id.
Now the correction only fires when the result's primary-artist NAME matches the
parent artist (the album/track-artist path now mirrors _process_artist, which
already name-checks). Mismatches are logged and skipped as collab/compilation.
Note: this prevents new corruption; existing wrong ids in a library aren't
auto-repaired (per-artist enrichment preserves an existing deezer_id).
A pasted Deezer artist link (or any Deezer-source artist click) opened the
wrong artist's header: deezer_id 525046 is stamped on 4 library rows (Kendrick
+ 3 others — an enrichment-corruption bug), and the library-upgrade lookup did
WHERE deezer_id=? LIMIT 1, grabbing an arbitrary row (Jorja Smith) while the
discography loaded fresh from Deezer (Kendrick) — a Frankenstein page.
find_library_artist_for_source now detects when a source id maps to >1 library
artist and refuses to guess: it skips the id-based upgrade (still allowing the
name fallback), so the caller renders the source artist directly — landing on
the correct artist. Unique ids are unaffected (no regression).
The underlying enrichment bug that writes one source id onto multiple artists
is separate and still worth a follow-up.
Spotify/Apple/MusicBrainz/Deezer artist links now resolve via each source's
get-by-id (get_artist / Deezer get_artist_info), shaped to the artist card and
rendered as an artist result that opens the artist detail page through the
existing flow. Album/track link handling is unchanged; bare IDs still rejected.
Follow-up to the bare-ID footgun: a bare number like 525046 carries no
source and no entity type, so it resolved to whatever album happened to own
that id (a user pasting Kendrick's Deezer artist id got an unrelated album).
Now the resolver accepts provider URLs (and the explicit spotify: URI) only;
a bare/unrecognized string is rejected and the dropdown surfaces a hint to
paste a full link. URL parsing + album/track resolution are unchanged.
New 'Link / ID' input on the Search page: paste a Spotify / Apple Music /
MusicBrainz / Deezer URL (or a bare ID) and it's looked up directly on the
owning source — no fuzzy search, no scoring.
- core/search/by_id.py: source-agnostic parser (URL domain/path or bare-ID
format -> source,kind,id; numeric IDs fan out, first hit wins) + per-source
get-by-id dispatch + adapters projecting each provider's dict onto the
standard album/track card shape.
- /api/enhanced-search/by-id: thin additive route over resolve_identifier.
- Frontend: dedicated input that adopts the resolved source as active and
renders through the existing dropdown + download/import flow.
Purely additive — existing files are insertion-only; the resolver runs only
behind the new route. 29 seam tests cover parsing, shaping, fan-out, and
not-found.
The album-bundle path COPIES slskd's completed files into private staging (then
on to the library) but never removed slskd's originals, so they piled up in the
download folder. (copy, not move, is correct for the torrent/usenet bundle paths
— those clients keep seeding — so the shared copier can't just always delete.)
Add an opt-in remove_source to copy_audio_files_atomically that deletes each
source ONLY after it copies successfully (never on a failed stage), and set it
for the Soulseek path only. Torrent/usenet keep their originals.
Tests: keeps source by default / removes when requested / keeps on failed copy.
The scoring best-of only helps if the right candidates were returned. File/CSV
titles ('Artist - Title') made the search query carry the artist prefix; add
canonical-title search queries so the correct tracks are actually found, then
the scorer best-of matches them. Additive (extra queries only when the title
canonicalizes differently).
#768 added canonical_source_track to the live-sync matcher and the playlist
editor reconcile, but NOT to the two paths that actually run for file/CSV
mirrored playlists: the discovery worker (core/discovery/playlist.py) and the
DB-only matcher (core/discovery/sync.py). YouTube playlists are cleaned at
ingest, so they matched; file playlists fed the raw 'Arctic Monkeys - Do I
Wanna Know?' title into search+scoring and never matched the library's clean
'Do I Wanna Know?' → reported missing / shown as 'extra'.
Add a conservative canonical best-of to both: score with the raw title AND the
canonicalized one, keep the better. canonical_source_track only strips an
'<artist> - ' prefix when it equals the artist, so it can only add a candidate.
Tests: _canonical_best_score seam (file-style match / clean title scored once /
keeps original when better).
start_playlist_sync validated the resolved mode with 'if sync_mode not in
(replace, append): sync_mode = replace' — a pre-existing clamp two lines below
the config read I added, which silently downgraded a configured 'reconcile' to
'replace'. So config=reconcile resolved correctly then got clobbered, and every
sync ran replace regardless of the setting (and incognito didn't help — it's
backend, not cache).
Replace the hand-rolled clamp with a pure normalize_sync_mode(requested,
configured) helper (VALID_SYNC_MODES includes reconcile) so the resolution is
testable and can't silently drop a mode again. Regression tests cover
reconcile-from-config, request-overrides-config, and unknown->replace.
Replace mode (default) deletes + recreates the server playlist every sync,
which wipes its custom image, description, and identity. Add an opt-in
'reconcile' sync mode that edits the existing playlist in place — adds the
tracks now in the source, removes the ones gone — without destroying the
object, so the user's custom art/description survive.
- Pure planner plan_playlist_reconcile(current, desired) -> {add, remove}.
- Per-client reconcile_playlist: Plex addItems/removeItems on the same object;
Navidrome Subsonic updatePlaylist delta (songIdToAdd / descending
songIndexToRemove); Jellyfin add + remove-by-PlaylistItemId on /Playlists/{id}/Items.
- sync_service: reconcile branch with a replace FALLBACK (if a server's in-place
edit is unavailable/fails, sync still succeeds destructively — logged loudly).
- Default stays 'replace' (no behavior change). New Settings > Playlist sync mode
picker (replace/reconcile/append) backed by playlist_sync.mode; per-request
sync_mode still overrides.
- Reconcile skips the post-sync source-image push so a custom poster isn't
re-clobbered (the bug).
Tests: planner (add/remove/dedupe/order/empty) + reconcile-or-replace dispatch
(success / false-fallback / exception-fallback / no-method). Per-server in-place
API calls need dev validation against real Plex/Jellyfin/Navidrome.
NOTE: opt-in only; default behavior unchanged.
Consistency follow-up: the Filler picked art via metadata source-priority +
its own prefer_source knob, ignoring metadata_enhancement.album_art_order —
the explicit cover-art source list that post-process embed and the Library
Re-tag job now use. So 'cover art sources' meant two different things.
Prefer select_preferred_art_url (the configured order) first; fall back to the
existing prefer_source / source-priority loop when no order is configured
(the default) — non-breaking, existing behavior unchanged for those users.
Help text updated. Test: configured order wins + skips the source loop.
User feedback (Sokhi): after changing cover-art sources, re-tag should
re-download fresh covers from THEM. The job took cover_url only from the
matched metadata source's album image, ignoring the user's configured
cover-art order. Now prefer select_preferred_art_url (the same
metadata_enhancement.album_art_order the post-process embed honors), falling
back to the source image when no order is configured (non-breaking).
'replace' cover mode already force-refreshes art on every matched album, and
the embed replaces existing art (no duplicate pictures) — so 'replace' + a
configured art order = fresh covers from those sources. Help text updated.
Tests: prefers configured source URL / falls back to source image when unset.
Find & Add on the playlist-sync page only wrote sync_match_cache, which is
DELETEd wholesale after every DB scan — so the source->library pairing (and
the user's manual matches) reverted to 'extra'/red-dot on the next shallow
scan. The three match stores (sync_match_cache, manual_library_track_matches,
discovery extra_data) were disconnected and all pointed at tracks.id, which a
rescan re-keys (esp. Jellyfin/Navidrome GUIDs).
Unify the match so it's one durable fact, recorded once, honored everywhere:
- Find & Add also writes a durable manual_library_track_matches row (one-way;
the manual-match tool has no playlist to act on, so no reverse). Carries the
library file path.
- New library_file_path column (idempotent migration) + find_track_id_by_file_path:
re-resolve a stale library_track_id after a rescan re-keys the track, and
self-heal the row.
- The sync compare display's override lookup now falls back to the durable
manual match (resolve_durable_match_server_id) when sync_match_cache misses —
so the pairing persists across a scan instead of reverting to a red dot.
Purely additive: only adds matches when the cache returns nothing.
Tests: durable resolver (valid / stale-reresolve+self-heal / no-match / not-in-
playlist / missing-methods), file_path persistence + find_track_id_by_file_path.
A bare host like '192.168.1.5:8080' or 'qbittorrent.lan:8080' (no scheme)
is what users naturally type, but requests then raises 'No connection
adapters were found for ...' — it can't pick an http/https adapter, and a
bare host:port even gets misparsed as scheme=host. This surfaced as the
generic 'qbittorrent probe failed' with a 'login error: No connection
adapters were found' in the logs.
Add normalize_client_url() in torrent_clients/base: default a missing scheme
to http:// (+ trim trailing slash), and route all three adapters'
_load_config through it. Transmission normalizes the base before appending
/transmission/rpc.
Tests: normalizer unit cases + per-adapter regression (bare host -> http://).
Note: usenet adapters (sabnzbd/nzbget) share the same pattern and need the
same treatment in a follow-up.
Follow-up hardening to #789. The selection was keyed purely by folder name,
so renaming a music folder in Navidrome silently reverted the scan to all
libraries. Now persist the folder id (stable across renames) as the primary
key alongside the name (kept for display + back-compat), and restore by id
first with a name fallback. Self-heals on reconnect: pre-id installs and
drifted/renamed names get the id + fresh name written back, so the settings
dropdown keeps highlighting the right folder.
Tests: restore-by-id-after-rename (+ name heal), name-fallback self-heals id,
no-drift writes nothing.
The saved music-folder selection was silently dropped on every reconnect.
_setup_client's restore step called the public get_music_folders(), which
starts with ensure_connection() — but we're already inside ensure_connection()
at that point (_is_connecting=True, _connection_attempted not yet set), so the
re-entrant call bailed and returned []. The restore matched nothing,
music_folder_id stayed None, and the per-call musicFolderId filters all
no-op'd → scans imported every library regardless of the user's choice.
Surfaces after any restart or settings save (reload_config resets the state).
Split get_music_folders() into the public method (does the connection check)
and a non-reentrant _fetch_music_folders() seam; the restore now calls the
seam directly (connection is already established + ping succeeded by then).
Regression + seam tests added.
- depth setting (light = core tags + matched source ids; full = same
multi-source enrichment cascade a fresh download gets, run additively
via embed_source_ids). Threaded through scan/finding/auto-apply and the
repair_worker fix handler.
- source now defaults to 'auto' (= your source priority / active source)
instead of blank.
- give native <option> popups a solid dark background (were white-on-white).
- tests for full-depth full_meta payload + enrich invocation + light no-op.
The job was the odd one out — auto_fix=False, no dry_run setting, so it never
showed the 'Dry Run' badge the other jobs do (the badge keys off
settings.dry_run === true). Aligned it to the standard pattern:
- auto_fix=True + dry_run setting defaulting True. Default behavior is unchanged
(findings only, nothing written) AND it now shows the Dry Run badge.
- Turning dry_run off makes the scan auto-apply in place (result.auto_fixed),
no finding — the opt-in 'just retag it' mode.
- Extracted a shared apply_track_plans() used by both the scan auto-apply and
the repair_worker fix handler (handler now resolves Docker paths then
delegates — one code path, no duplication).
Tests: dry_run=False auto-applies + writes + no finding; existing dry-run
finding/skip/apply tests still green. 410 passing.
The old per-download Retag Tool was limited (only native-pipeline downloads,
100-group cap, manual per-group) and did the wrong thing — it moved/reorganized
files instead of just tagging. It's superseded by the new Library Re-tag job
(whole-library, in-place) + the enhanced-library 'Write Tags' button.
Removed: the post-download record_retag_download ingestion hook (stops writing
retag_groups on every download), core/library/retag.py, the web_server state +
deps + /api/retag/* endpoints + the tool:retag WebSocket emit, the dashboard
card + both modals (index.html), the core.js socket handler, and the tools-page
wiring + help entry (wishlist-tools.js). Updated the import-pipeline test.
Verified: web_server parses, app + core imports OK, 392 tests pass, no live
references to removed symbols.
Left as inert (harmless) for a careful follow-up sweep: the retag_groups/
retag_tracks tables + their DB CRUD methods (no longer written/read), and the
now-orphaned retag JS helper functions (no entry point/wiring/socket calls them;
interspersed with wishlist functions, so not blind-deleted).
write_tags_to_file wrote the core fields + cover but never the source IDs
(Spotify/iTunes/MusicBrainz) the import post-process embeds. Added a focused
source.embed_known_source_ids() that writes ALREADY-KNOWN ids (from db_data)
via the canonical, Picard-compatible frame writer the import uses
(_write_embedded_metadata) — no API re-fetch, frames correct by construction.
write_tags_to_file now calls it whenever db_data carries id keys.
Fed from both paths: the enhanced-library 'Write Tags' button now carries the
track's spotify/itunes/musicbrainz ids, and the Library Re-tag job stamps the
matched album/track source ids onto each track. So both now write the full tag
set, not a subset.
New 'Library Re-tag' repair job (default-OFF, opt-in; weekly when enabled):
- Scans every source-matched album (spotify/itunes/deezer/musicbrainz album id),
pulls fresh metadata + tracklist from that source, reads each local track's
current tags, and uses the planner to compute per-field diffs.
- Dry-run by design: scan only CREATES findings — nothing touches a file. Each
finding is highly detailed: per-track old->new for every changed field, the
source used, the mode, a cover-art action, and any unmatched tracks, plus a
summary description. Settings: mode (overwrite | fill_missing), cover_art
(replace | fill_missing | skip), source override.
- Apply handler (_fix_library_retag in repair_worker): writes each track's
planned tags in place via tag_writer.write_tags_to_file (+ batch-embeds cover,
refreshes cover.jpg). Only adds/overwrites planned fields — no moves/renames/
re-match. Resolves Docker paths; read-only/unreachable files counted, never
crash. Media-server-only / unreachable tracks are skipped.
Registered in the job list + fix dispatch. The old per-download Retag Tool is
left untouched alongside this for now.
The testable core for the new library-wide re-tag job. Given a source album's
metadata + tracklist and the library tracks' current file tags, it:
- matches source tracks to library tracks (disc+track number, then title sim),
- computes the per-field diff (old -> new) for the dry-run finding,
- builds the minimal write_tags_to_file payload — only fields that actually
change under the chosen mode (overwrite vs fill-missing), so applying never
touches unrelated/unchanged tags.
No IO/network/DB — 10 unit tests cover matching, both modes, blank-source
fields, and the album-artist/track-count payload mapping.
Verification found a non-additive edge: embed_album_art_metadata uses FLAC
add_picture(), which APPENDS — so applying to an album where some tracks already
had art would have added a duplicate embedded picture. The apply now checks each
file and skips any that already carry art (shared _audio_has_art helper), so it
only ever ADDS art to files missing it. Test covers the skip (no re-embed).
Previously the filler only flagged albums whose DB thumb_url was empty and, on
apply, only updated that DB thumb_url — so albums whose files had no embedded
art and no cover.jpg (but whose DB row had a URL) were never found, and even
'applying' art never touched the files. That's the reported 'doesn't scan all
albums' gap.
New core.metadata.art_apply (reuses the post-processing standard so the user's
album_art_order is honored):
- album_has_art_on_disk(): cheap-first check — folder cover.jpg/folder.jpg
sidecar, then embedded art in a representative track (FLAC/ID3/MP4/Vorbis).
- apply_art_to_album_files(): embeds via embed_album_art_metadata + writes
cover.jpg via download_cover_art; only ADDS art (never rewrites the user's
tags); read-only/unwritable files are skipped + counted, never crash.
Scan now examines every titled album and flags it when art is missing in the DB
OR on disk. Apply embeds into the album's audio files + writes cover.jpg in
addition to the DB thumbnail (media-server-only albums fall back to DB-only).
Tests cover sidecar/embedded detection, the cheap-first short-circuit, and the
apply orchestration (embeds each file + cover.jpg; read-only failures counted).
The title/artist fallback search took results[0]'s artwork unconditionally, so
a loose full-text match returned the wrong album's cover (the 'new sources give
incorrect art' reports). Now it pulls a few results and only accepts one whose
title matches (subset, to allow Deluxe/Remaster) AND whose artist matches
exactly — the artist being the strong guard against wrong covers. Falls back to
an exact title match when a result carries no artist.
The album's own stored source-id path is unchanged (that id is authoritative).
Tests: wrong-artist rejected, skips wrong result for a matching one, + unit
coverage of the matcher (deluxe/feat/stopwords accepted, wrong artist/title
rejected).
qBittorrent 5.2.0 changed /api/v2/auth/login to return HTTP 204 (No Content)
on success instead of HTTP 200 with body 'Ok.'. The adapter required the body
to equal 'Ok.', so every login on 5.2.0+ failed with 'HTTP 204 body=' — the
connection probe and all torrent actions were broken.
Treat login as successful on the SID auth cookie and/or a success body: 'Ok.'
(<=5.1) or an empty HTTP 204 (>=5.2.0). Still reject bad creds, which
qBittorrent reports as HTTP 200 + 'Fails.' (not a 4xx).
Tests: 204-empty -> success, SID-cookie+empty-body -> success, 'Fails.' (even
with a stale cookie) -> failure.
- Stale-cache check (playlistTrackCacheIsStale) compared raw track_count to the
filtered/cached track list, so any playlist with local or unavailable tracks
always looked 'stale' and refetched + re-mirrored on every modal open. Now it
compares the upstream snapshot_id (stored at cache time in the shared fetch
choke point), and returns not-stale when no snapshot is available — explicit
invalidation on refresh still handles real changes.
- organize_download: guard executor.submit so a refused job cleans up the batch
instead of stranding it in 'analysis' (holding a limited analysis slot).
- Removed the dead, deprecated, unused mirrorSpotifyPlaylistTracks.
resolve_mirrored_playlist tried the mirrored-playlists primary key FIRST for
any all-digit ref. Deezer upstream ids are all-numeric, so a Deezer playlist id
was mistaken for the PK and the organize-by-playlist toggle resolved a wrong row
(or nothing) — the toggle silently wouldn't save / 'Open in Mirrored' missed.
Resolve by (source, source_playlist_id) first, fall back to PK only when the
source lookup misses. Thread the batch/wishlist source through the download-path
callers so numeric upstream ids resolve correctly there too. Spotify (base62
ids) is unaffected.
Seam tests: numeric Deezer id resolves by source (not PK), spotify alphanumeric
by source, PK fallback still works, profile-scoped, empty refs -> None.
The worker's WARNING observability proved the '38 errors' were almost all
MusicMap returning 404 (artist has no map page) — a genuine not-found, not a
fetch failure. But iter_musicmap_similar_artist_events flattened every
RequestException to status_code 502, and the worker maps 400/404 -> not_found
/ everything-else -> error, so these inflated the error count.
Surface the real HTTP status from the exception's response (404 stays 404),
falling back to 502 only when there's no response (timeout/connection drop,
which is correctly still an error eligible for retry).
Regression tests: 404 -> 404 (not_found), timeout -> 502 (error), 500 stays
error, plus an end-to-end worker check that a 404 result marks 'not_found'
and stores nothing.
Verified against live data: 1312/1313 stored similars carry a metadata source id,
but 1 slipped through name-only (a match on a source with no id column, e.g.
discogs). Enforce the standard: process_artist now SKIPS any similar whose match
doesn't map to a storable id column (spotify/itunes/deezer/musicbrainz) instead
of writing a useless name-only row. Regression test covers discogs-match + no-id
cases. Now 100% of newly-stored similars are actionable.
The kettui move: 38/79 fetches errored on the first live run, but they were
logged at DEBUG only — invisible in app.log, so the cause (rate-limit vs
no-providers vs bug) is unprovable. process_artist now returns a (status, count,
detail) triple carrying the error reason (status code + message / exception),
and the worker logs the first 15 errors per session at WARNING (rest DEBUG) +
keeps _last_error. No blind pacing tweak — let it run, read the real reason, then
fix the proven cause. Seam tests updated + assert the reason is captured.
- get_stats now reports PERSISTENT counts from the DB (matched/not_found/pending
+ a progress.artists breakdown) instead of in-memory session counters, so the
dashboard orb tooltip and the Manage modal agree (was showing 0 vs 14 after a
restart) and it survives restarts — same approach as the other workers.
- Orb tooltip reads progress.artists ('Artists: 14 / 15 (93%)') like the rest.
- Worker now defaults to ON (running) instead of opt-in-paused; still honors a
saved pause across restarts. It self-paces (~3s/artist) and backs off on
MusicMap outages, so the orb spins/active like the others when there's work.
10 seam tests pass.
SoulSync standalone matches library tracks without Plex fetchItem,
reports missing counts correctly, and skips server playlist writes.
Automation re-syncs when the mirror grows; after sync finishes, starts
organize download (organize-by-playlist) or wishlist processing.
UI: Spotify URL playlist-folder controls, organize toggle layout in the
discovery modal, reload organize preference when reopening Download Missing.
Co-authored-by: Cursor <cursoragent@cursor.com>
Closes the gap where similar artists only existed for WATCHLIST artists: a new
background worker populates them for the whole LIBRARY, slotting into the
existing enrichment-worker pattern (bubble + Manage Enrichment Workers modal,
status/pause/resume, matched/not_found/pending/errors).
Per source-matched library artist → get_musicmap_similar_artists(name, 25)
(the same matcher the artist-detail page uses: fetches MusicMap names, matches
each to the user's source chain — primary + active fallbacks — returns only
matched artists) → store via add_or_update_similar_artist keyed by the artist's
metadata source id, the SAME key the watchlist scanner + artist map use, so the
two cooperate (idempotent upsert + retry_days window).
- core/similar_artists_worker.py: pure seams (pick_source_artist_id,
map_payload_to_store_kwargs, process_artist) + the threaded worker; skips
artists not yet source-matched; classifies not_found vs transient error
(retry after 30d).
- DB migration: similar_artists_match_status / _last_attempted on artists
(mirrors every other source worker's tracking columns).
- Registered in EnrichmentService + instantiated in web_server, DEFAULT-PAUSED
(opt-in) like Amazon — MusicMap is scraped/outage-prone + this is library-wide.
- SERVICE_ENTITY_SUPPORT['similar_artists']=('artist',) so the modal breakdown
('artists with / without similars') + Retry work; manual-match (inapplicable
to a relationship) is gated out via relationship:true.
- 10 seam tests; existing 80 enrichment tests still pass.
Note: keys under profile 1 (single-profile setups); multi-profile is future work.
Persist organize_by_playlist on mirrored playlists and run playlist-folder
downloads from the auto-sync pipeline instead of the global wishlist phase.
Register SoulSync library rows after playlist-folder post-processing, route
failed organize batches to the wishlist correctly, and skip sync-time
unmatched wishlist only when organize download handles retries.
Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL),
re-mirror on refetch, and improve standalone playlist modals (re-analysis,
Open in Mirrored). Add filesystem missing-track detection and tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Verified end-to-end: fetch_public_playlist_full pulled all 236 tracks of the
test playlist via SpotipyFree (the library handles the client-auth that 429'd
the raw approach). Name + tracks correct.
- requirements.txt: declare spotipyFree>=1.1.2 as a normal pip dependency (like
spotDL, also MIT — aggregation, not vendored) + websockets (a transitive dep
SpotipyFree/spotapi needs that pip doesn't pull automatically). Code still
soft-imports + falls back to embed, so it's never a hard runtime requirement.
- meta fetch uses limit=1 (name/owner only) so we don't pull the whole list
twice. 9 tests green.
The in-house anonymous-token path is blocked by Spotify (429 without the web
player's rotating client-auth). Switch the full-fetch to SpotipyFree — the
maintained no-creds spotipy drop-in spotDL uses, which tracks that machinery.
- core/spotify_public_api.fetch_public_playlist_full now uses a SpotipyFree
client (playlist + playlist_items + next), normalising the spotipy-shaped
items to the embed scraper's shape. Injectable client_factory keeps it
unit-testable without the library or network. Dropped the dead in-house
token/pagination code.
- Licensing: SpotipyFree is GPL-3.0, so it is NOT bundled/required (SoulSync is
MIT). Optional, user-installed: the import is soft, and on ImportError (or any
failure) fetch_spotify_public falls back to the embed scraper (~100). So the
shipped project stays cleanly MIT and the link path never regresses.
- requirements.txt: documents it as a commented optional extra
(pip install SpotipyFree) with the GPL/MIT rationale.
- 9 tests: normalisation, pagination past 100, library-missing -> raises (->
fallback), and the embed-fallback orchestration.
Needs a live click-through with SpotipyFree installed to confirm the exact
class/method names match (SpotipyFree.Spotify / playlist / playlist_items).
The full-fetch's logs used a bare module logger that app.log doesn't capture,
so we couldn't see whether the API path succeeded or why it fell back. Route
them to 'soulsync.spotify_public' and log: token found?, embed parsed?, the
API HTTP status on a non-200, and pagination result. Lets us see the exact
failure (e.g. 401 vs 429) on the next link-tab test.
Live debugging the 'shows 100' report:
- The full playlist page no longer embeds an accessToken, and get_access_token
/ server-time now 403/404. The EMBED page (open.spotify.com/embed/playlist/{id})
still ships a usable anonymous token. Was fetching the wrong page -> no token
-> raised -> embed fallback (100). Now reads the embed page for the token.
- Confirmed live: token extraction + embed parse work; the token is accepted by
the Web API (429 rate-limit, not 401). Could not show >100 from here because
the test IP got rate-limited from probing; needs a clean-IP click-through.
While in there, made it more robust against the rate-limiting that's clearly in
play:
- Refactored scrape_spotify_embed -> reusable parse_embed_html.
- fetch_public_playlist_full now does ONE embed fetch for token + name + first
page (no separate metadata call = fewer requests = less 429 surface), then
paginates the API. If the API is unavailable/rate-limited, it keeps the embed
page's tracks (<=100) instead of raising — so the result is always >= today's
behaviour, never worse.
- 12 tests incl. the new API-fails-but-embed-tracks-survive path.
Caveat unchanged: rides Spotify's undocumented embed-page token; degrades to the
embed fallback, never crashes.
The no-auth 'add by link' path scrapes Spotify's embed widget, which only ever
contains ~100 tracks and can't paginate — so big public playlists got
truncated. This adds an in-house anonymous fetch that pulls the FULL list:
- core/spotify_public_api.py: reads the anonymous web-player accessToken Spotify
already embeds in its own open.spotify.com page HTML (no app credentials, and
no rotating TOTP secret for us to maintain), then paginates
/v1/playlists/{id}/tracks 100 at a time until the whole playlist is pulled.
Returns the embed scraper's exact shape. Pure helpers + injected http_get so
it's unit-testable without the network.
- core/spotify_public_scraper.fetch_spotify_public(): tries the full fetch for
playlists; on ANY failure (or for albums) falls back to scrape_spotify_embed.
Worst case == today's behaviour, so the link path can't regress.
- web_server: the link-tab endpoint and the authed flow's last-resort scrape
now both go through fetch_spotify_public.
Scoped entirely to the spotify_public_* (no-auth) path — the authenticated
playlist sync is untouched. 11 tests (token extraction, normalisation,
pagination past 100, and the embed-fallback orchestration).
Caveat: rides Spotify's undocumented page-embedded token — expected to break
when they change their page; it degrades to the embed fallback, never crashes.
Needs a live click-through to confirm the token path works end to end (can't
hit Spotify from the test env).
- #1 Unconfigured-source banner: when a source has enabled=false, show a
notice that browsing works but matches/retries won't run until it's set up.
- #2 Rate-limit detail: when rate_limited, surface 'resumes in ~Xm' (from the
status payload) instead of just a pill.
- #3 Richer rows: unmatched items now show parent context — an album's artist,
a track's album — via a parent expression in the query (+ test).
- #4 Bulk select: per-row checkboxes + a bulk bar to retry several at once
(capped concurrency), reusing the /retry item endpoint.
- #5 Remember last worker: selection persists in localStorage and is restored
on open; openEnrichmentManager(workerId) supports future deep-linking
(bubbles left on their pause-on-click behaviour).
- #6 Keyboard nav: ArrowUp/Down moves focus between rows; actions are native
buttons (Enter/Space) and Escape closes — list isn't poll-refreshed so focus
is stable.
53 enrichment tests green; JS syntax clean.
Per-worker processing-order override + UI polish.
Feature — pin an entity group to enrich first:
- Each worker normally runs artist -> album -> track. A user can pin one
group (artist/album/track) to run first from the modal; the worker keeps
that group first until it's exhausted, then resumes the normal chain.
- core/worker_utils.py: read_enrichment_priority() (reads
<service>_enrichment_priority each loop, live) + priority_pending_item()
(shared, whitelisted query returning the worker's expected item shape;
Spotify/iTunes get album_individual/track_individual via a type map).
- A guarded ~6-line hook at the top of all 11 workers' _get_next_item.
CRITICAL: when nothing is pinned (default) the hook returns immediately,
so default enrichment order is byte-identical to before. Discogs (no track)
and Genius (no album) only honor their supported entities.
- core/enrichment/api.py: GET/POST /api/enrichment/<id>/priority (+ config_get
hook); POST validates the entity against what the source enriches.
- 14 new tests (helper shapes, exhaustion, route get/set/clear/validate).
UI:
- Refined hero header: identity + inline status left, single Pause right,
'now enriching' quiet sub-line; overall coverage % moved into the stats
section ('82% matched · 1,203 of 1,460'). Hero gently pulses while running.
- New processing-order strip: artist→album→track steps showing the live phase
(pulsing 'now'), pinned group ('first' + 📌), and done/remaining; click a
step to pin it, click again for auto.
py_compile clean across all 11 workers; 52 enrichment tests green.
Fixes a correctness bug and adds bulk re-queuing.
- Bug: per-row 'Retry' used clear-match, which sets an item to not_found
with last_attempted=NULL. The worker only retries not_found items where
last_attempted < (now - 30d), and 'NULL < cutoff' is false in SQLite, so
those items were never re-queued. Fixed by resetting match_status to NULL
(pending), which every worker's queue picks up on the next pass.
- New POST /api/enrichment/<id>/retry with scope 'item' | 'failed'
(failed = re-queue every not_found item of an entity type), backed by a
pure whitelisted build_reset_query + MusicDatabase.reset_enrichment().
- UI: per-row Retry now hits /retry; a 'Retry all failed' bulk button appears
when the current entity has not-found items (confirm + count toast); a hint
line explains retry/match/auto-retry behaviour.
- 11 new tests (38 enrichment tests total, all green).
