Commit graph

45 commits

Author SHA1 Message Date
BoulderBadgeDad
2fb142dded jellyfin scan: page the bulk fetch so the no-progress watchdog can't false-stall
Discord (DXP4800 NAS, 7148 tracks): library updates kept dying with "Update appears stuck — no
progress for 300s (last phase: Fetching all tracks in bulk...)". not actually hung.

root cause: the bulk track/album fetch used a single 10000-item page, so a whole library came
back in ONE request that emitted NO progress while in flight. the watchdog (database_update_health)
kills a job with no progress for 300s — so on a slow server that one silent request tripped it even
though it was alive, not stuck. raising the timeout cap only buys the silent request more rope; a
bigger library or slower disk just needs a higher number. the per-batch progress line also only ran
when there was a NEXT page, so a sub-page-size library reported nothing at all.

fix: extract a pure paginate_all_items seam (core/library/bulk_paginate.py) that pages in 1000s and
reports progress after EVERY page — so the watchdog is fed on a cadence set by page size, not library
size, and can't starve mid-fetch no matter how big the library. both Jellyfin bulk loops (tracks +
albums, same defect) now route through it. preserves the failure-shrink resilience (halve to a floor,
then give up). does NOT change what's fetched — same query, fields, items.

note: changes nothing about WHICH tracks come back; only how they're paged + that every page reports.
keep the raised cap on dev as a margin — this is the actual fix. Plex/Navidrome don't share the
pattern (checked). 9 seam tests incl. the watchdog-feed invariant (progress count scales with
N/page_size, never one call for the whole library) + the sub-page regression + failure-shrink.
467 jellyfin/library tests green, ruff clean.
2026-06-28 13:41:38 -07:00
BoulderBadgeDad
ed0a2079cf
Merge pull request #896 from nick2000713/feature/best-quality-search-mode
Global quality system: real-audio verification, best-quality search & quality profiles (please try...not ready to merge)
2026-06-24 20:27:38 -07:00
BoulderBadgeDad
d4e80fdaa0 #915: redownload pulls full album_data from the primary source for iTunes/Deezer too
Second leak of the same class: redownload_start built full album_data (release_date/album_type/
total_tracks) only in the Spotify branch. The iTunes and Deezer branches set just track/disc number
and left album_data lean ({'name': ...}), so single-track redownloads on those sources dropped the
$year — same symptom as #915 in the add/download path.

Fix: both branches now fetch the album via get_album_for_source (cached, source-aware) and build
album_data through the shared _album_data_from_source helper, mirroring the Spotify branch. Falls
back to the lean default if the fetch returns nothing (no regression). get_album is cached on both
iTunes and Deezer, so no extra API cost.

Tests: _album_data_from_source (full build, image-url fallback, defaults). 694 library+downloads
tests green.
2026-06-23 19:21:50 -07:00
BoulderBadgeDad
600a744f7f #917: 'I have this' reuses the album's existing folder year instead of dropping it
The import rebuilds the destination path from album metadata. When the albums row has no year,
release_date is empty, the path template drops $year, and the copied file lands in a NEW yearless
directory instead of the album's existing 'Album (YYYY)' folder. (The code logically forces this:
the year only drops when album.year is empty.)

Fix: when album.year is empty, recover it from a sibling track — its own year column, else a
(YYYY)/[YYYY] in the album folder name — so the rebuilt path matches the existing directory.
No-op when album.year is already set.

Tests: _existing_album_year_from_sibling covers year-column, paren folder, bracket folder, no-signal,
and target-slot exclusion.
2026-06-23 16:18:00 -07:00
nick2000713
63374b32f1 Merge remote-tracking branch 'nezreka/dev' into feature/best-quality-search-mode
# Conflicts:
#	core/hifi_client.py
2026-06-23 11:33:50 +02:00
BoulderBadgeDad
01c51a3c0e #904: guard standalone Deep Scan against relocating a desynced library
Standalone _run_soulsync_deep_scan did a path-only diff (untracked = transfer files
not in the soulsync DB) and shutil.move'd EVERY untracked file to Staging — no guard.
When the DB is empty/out of sync with disk (volume swap, DB reset, external Picard
tag edits) but Transfer holds the real library, that flags the whole library as
untracked and relocates all of it; Phase 5 then deletes the rows, and with Staging
cleanup on the files are gone for good. Reporter lost ~1,500 tracks into Staging.

The stale_guard the orphan detector + media-server deep scan already use (#828, #908)
was never wired into this path. Fix:

- core/library/standalone_scan.py (pure, tested): plan_standalone_deep_scan() diffs
  untracked (separator-normalized) and decides whether the move is safe. Blocks when
  the untracked share is implausibly large (>20 files AND >50% of Transfer — the
  desync signature, via is_implausible_orphan_flood) or when the user marked Transfer
  permanent. A normal batch of new arrivals still moves.
- web_server: consult the planner before Phase 4; on block, move NOTHING, leave files
  in place, and surface a loud warning + activity item. Guard Phase 5 deletes too
  (skip on desync-block or implausible stale share).
- 'Transfer is my permanent library — never move files out' toggle
  (import.transfer_is_permanent) in Settings.
- tests/library/test_standalone_scan.py: seam coverage + the #904 regression
  (empty DB + 1,500 files -> blocked, nothing moved).

No behavior change for in-sync libraries; the guard only trips on the desync pattern.
2026-06-22 17:53:49 -07:00
dev
b761229a00 merge: pull upstream/main (2.7.4) into feature/best-quality-search-mode
- Keep our v3 ranked-targets quality system (filter_and_rank, QualityTarget)
  in soulseek_client.py, settings.js, database presets, and index.html
- Take upstream removal of standalone quality-scanner code:
  QualityScannerDeps + run_quality_scanner moved to repair job
  (core/repair_jobs/quality_upgrade_scanner.py)
- Take upstream AAC-tier addition in database/music_database.py default profile
- Take upstream removal of /api/quality-scanner/* routes from web_server.py
- Remove test_discovery_quality_scanner.py (deleted upstream)
- 47 upstream commits absorbed (2.7.3 + 2.7.4 including re-identify flow,
  dead-folder cleanup, track-number prefix strip, and more)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 16:54:51 +02:00
BoulderBadgeDad
298d825757 #891: clear dead folders left with only cover images / .lrc sidecars
Reorganize leaves a cover.jpg (and other leftovers) behind when it moves an album,
and the Empty Folder Cleaner is too conservative to sweep them. Two complementary
fixes over one shared 'residual file' predicate (core/library/residual_files.py:
junk + cover/scan images + lyric/metadata sidecars), so both features agree on what
a dead folder is.

1. Reorganize (proactive): _delete_album_sidecars now sweeps ALL residual files when
   a source dir has no audio left — previously only a fixed list of cover NAMES, so
   back.jpg / disc.png / .webp survived and kept the folder un-prunable.
2. Empty Folder Cleaner (the request): new opt-in 'Remove Residual Files' setting
   (default off) treats a folder holding only images/sidecars/junk as removable —
   cleans the existing backlog + arbitrary names. Auto-renders as a UI toggle.

Safe by construction: whitelist-only (a booklet.pdf / video / .txt is real content
and kept), reorganize sweep gated on no-audio, cleaner re-checks at apply time.
20 new tests; 272 reorganize/repair/empty-folder green.
2026-06-18 20:07:41 -07:00
dev
973d28f61d fix(path-resolve): CWD-independent base dirs + quality-scan resolve diagnostic
The quality job still resolved 0/18 because the shared resolver kept relative
config paths ("./Transfer") as-is and gated them behind os.path.isdir("./Transfer"),
which only holds when the calling thread's CWD is the app root. The repair
worker thread's CWD isn't guaranteed to be /app, so base_dirs came back empty
and every track was "unresolved".

- _collect_base_dirs now also adds os.path.abspath() of every relative
  candidate, so "./Transfer" → "/app/Transfer" regardless of CWD.
- quality_upgrade_scanner logs a one-shot [QualityResolve] diagnostic on the
  first unresolved track (cwd, transfer_folder + abspath + isdir, base dirs
  tried, abs-join existence) so any remaining mount mismatch is pinpointable
  instead of a silent "all skipped".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 14:39:18 +02:00
dev
4cb1937810 fix(path-resolve): try full relative path first in shared resolver
ROOT CAUSE of the quality scanner's "18/18 could not be probed". The shared
resolver (core/library/path_resolver.py) suffix-walked starting at index 1,
which is correct for absolute media-server paths (/music/Artist/... — index 0
is the empty leading segment) but WRONG for SoulSync's own library, which
stores RELATIVE paths like "Asketa/Another Side/track.flac". Index 0 there is
the artist folder; dropping it meant the resolver joined base/Another Side/...
(no artist) and nothing ever matched — so every library track came back
unresolved and the probe opened a relative path that didn't exist from CWD.

Start the suffix walk at index 0 so the FULL relative path is tried first.
Safe for absolute paths (i=0 yields base//Artist/... which harmlessly misses
and falls through to i=1) and Windows drive parts (E: fails on POSIX, falls
through). Other tools (orphan/fake-lossless detectors) were unaffected because
they os.walk the transfer folder directly and never used this resolver.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 11:40:18 +02:00
BoulderBadgeDad
94a0070fa8 Orphan detector: hard-bail on a mass-orphan flood instead of warn-only
A DB<->filesystem path mismatch (Docker volume change, remount, Music
Paths unset for the container) makes EVERY library file fail to resolve
to a DB track, so the orphan detector flags the whole library as
orphaned. The mass-orphan check only logged a warning and then created
the findings anyway — so a user batch-applying 'move to staging' or
'delete' would relocate or wipe their entire library.

Make it a hard skip (create zero findings) like the dead-file cleaner
and stale-removal paths already do (#828). Centralise the predicate as
is_implausible_orphan_flood() alongside is_implausible_stale_removal()
so the rule lives in one tested place. Small genuine orphan sets still
surface unchanged — only an implausibly large flood (>50% and >20) is
suppressed.

Tests: seam cases for the new predicate + scan-level regressions (mass
mismatch -> 0 findings; small genuine set -> still reported).
2026-06-12 08:15:23 -07:00
BoulderBadgeDad
4d1b9a5639 Artist Sync: guard stale-removal against an unreachable mount + gate it admin-only
The enhanced-tab "Sync" button's stale-removal phase deleted any track whose file
wasn't on disk, with NO guard — so if the music storage was momentarily
unavailable (sleeping NAS, dropped mount, unmounted Docker volume, WSL hiccup),
os.path.exists returned False for EVERY file and one click wiped the whole artist
(tracks + their now-"empty" albums) from the DB. The deep-scan path already had a
50%-stale safety net (#828); this endpoint never got one.

- New core/library/stale_guard.py: is_implausible_stale_removal(missing, total) —
  a tested rule (skip removal when missing > 50% of a >=5-track set), centralised
  so every stale-removal site can share it.
- sync_artist_library: if the guard trips, SKIP removal (delete nothing), return
  removal_skipped + warn; the frontend shows "storage may be offline — skipped"
  instead of silently deleting. Empty-album cleanup now also only runs on the
  non-skipped path and uses `album_id IS NOT NULL` (fixes the NOT IN-with-NULL
  no-op). Frontend also refreshes the view on additions, not just removals.
- @admin_only on the endpoint — it deletes tracks + albums but was ungated, while
  the sibling delete_album endpoint is gated.

Deep scan was already safe (different mechanism: server-diff + its own 50% guard).

Tests: guard unit rules; endpoint skips removal when all files missing (keeps the
tracks), removes only the genuinely-gone few otherwise, and 403s for non-admins.
7 new tests pass.
2026-06-10 19:33:44 -07:00
BoulderBadgeDad
0c1dd6c2a9 Delete: resolve the real on-disk file when DB metadata uses curly quotes (#833)
the-hang-man: tracks with an apostrophe (e.g. "I'm Upset") deleted the DB row
but left the file. The library DB stored the title with U+2019 (the curly form
Spotify/Apple metadata uses) while the file was written to disk with U+0027
(ASCII). _resolve_library_file_path compared the curly path byte-for-byte via
os.path.exists, missed every time, and reported "could not be deleted".

Fix: resolve confusable-tolerantly. New core/library/path_resolve.find_on_disk
descends the path component by component, taking an exact match when present and
otherwise folding a small set of typographic look-alikes (curly vs straight
quotes, en/em dash, ellipsis, nbsp) for the comparison ONLY — it never renames,
just finds the file that's actually there. Exact matches always win per
component, so paths that already resolved are byte-for-byte unaffected. This
also fixes existing mismatched files (no re-import) and every caller of
_resolve_library_file_path (sidecar cleanup, dead-file checks, streaming), not
just delete.

Case is deliberately NOT folded: a case-sensitive dataset (ext4/ZFS) can hold
names differing only by case, and folding could resolve the wrong file. The
reported failure is purely typographic.

Tests: real temp-file fixtures exercising the actual byte mismatch — curly-DB →
ascii-disk resolves, exact still works, confusable in a folder component, exact
wins when both encodings present, genuinely-different name does NOT collide,
missing file → None. 10 new tests; 949 resolver-adjacent tests pass.
2026-06-09 22:28:53 -07:00
BoulderBadgeDad
1d16ac7978 Downloads: reuse an album's existing folder so batches don't split it (#829)
Tacobell444: when tracks land in an album across multiple batches (a wishlist
run, the Album Completeness job, a missed track re-downloaded later), the folder
is rebuilt from API metadata each time — so when $albumtype or $year come back
blank/different on a later batch, the folder NAME changes and the album splits,
forcing a Reorganize.

Fix: build_final_path_for_track now checks whether the album already lives in a
single folder on disk and, if so, drops the new track there instead of a freshly
templated folder. Match (chosen): exact stored Spotify album id first, then a
STRICT >=0.85 name+artist match (vs the 0.7 used elsewhere) — a wrong match here
misplaces a file. New core/library/existing_album_folder.resolve_existing_album_folder
holds the logic; always-on with template fallback.

Safety rails: only returns a folder UNDER the transfer dir (never a read-only
library/NAS mount), only when the album lives in EXACTLY ONE folder (multiple =
disc subfolders, which DatabaseTrack can't disambiguate — those defer to the
template), and any failure falls through to the template path. Added
MusicDatabase.get_album_by_spotify_album_id for the id-first lookup.

Tests: single-folder reuse, no-match, below-threshold, multi-folder defer,
outside-transfer reject, id-first, missing transfer dir, no-files-on-disk.
8 tests; 1556 path/import/download tests pass (only the known soundcloud
failures remain).
2026-06-09 13:47:25 -07:00
BoulderBadgeDad
696119d5ac Expired Download Cleaner: retention-based cleanup of watchlist/playlist downloads (Boulder)
A Library Maintenance job that cleans up downloads tracked by Download Origins
once they pass a per-origin retention window — findings by default, opt-in
auto-delete.

A download is only ever proposed for deletion when ALL hold: older than its
origin's retention, NOT still in an actively-mirrored playlist / watched
artist, and played fewer than the keep-threshold (default 2 → "played more
than once is kept"). Only touches downloads recorded from the Download Origins
feature forward — never pre-existing or manual library.

- core/library/expired_cleanup.py: pure decision core (retention_cutoff,
  is_expired, select_expired) — no DB/clock, fully tested. play_count is the
  reliable listen signal (last_played is often unpopulated, so recency isn't
  used).
- ExpiredDownloadCleanerJob: gathers facts (play_count via a new
  get_origin_cleanup_candidates join; active-mirror via get_mirrored_playlists;
  watch via get_watchlist_artists) and either creates 'expired_download'
  findings or, with auto_delete on, deletes in-scan. Default OFF, both
  retentions default 'off'. Settings auto-render in the Library Maintenance
  panel (same as Cover Art / Lyrics / Re-tag).
- delete_origin_download(): shared delete (resolve path → remove file → drop
  track row → drop history row); a file that won't delete keeps its row +
  reports. Used by auto mode AND the _fix_expired_download apply handler.
- Frontend: type/action ('Delete')/result labels + finding detail render.

Tests: 9 on the pure brain (windows, off, per-origin, protected, play-count
threshold, bad age) + 7 on the job (no-op when off, findings, mirror/watch
protection, auto-delete, delete helper missing/real file). 185 repair/origin
tests pass.
2026-06-07 22:06:56 -07:00
BoulderBadgeDad
8b7609cdb2 Manual match: paste a MusicBrainz ID/URL to match directly (Ashh)
Ashh: the manual-match modal fuzzy-searches a service and shows the top 8.
When the right release isn't in those 8 (common title — their example was
"Idols", which returns 8 unrelated releases and not Yungblud's), there was no
way through. But the user usually already knows the exact MBID.

Now the modal's search box doubles as a direct-ID box. Paste a MusicBrainz
MBID (bare UUID or a musicbrainz.org URL) and SoulSync looks that exact
entity up and shows it as the single result to confirm + Match — no fighting
the search ranking.

- core/library/direct_id.py: pure detector, returns the canonical ID only
  when the text unambiguously IS one (whole-query UUID, or a UUID inside a
  musicbrainz.org URL). "Idols", "Yungblud Idols", a UUID buried in free
  text → None, so normal search is never hijacked.
- _search_service: direct-ID fast path before the fuzzy search —
  get_release (→ get_release_group fallback for albums) / get_artist /
  get_recording. A pasted-but-unresolvable ID falls THROUGH to fuzzy search,
  so a typo can't dead-end the modal.
- UI: MusicBrainz placeholder now says "…or paste a MusicBrainz ID/URL".

Detector is service-keyed so Spotify/iTunes/etc. direct IDs can be added
later; today only MusicBrainz has a confirmable direct lookup, matching the
reporter's ask + screenshot. 9 tests: detector truth table (bare/URL/plain/
buried/other-service) + dispatch (confirmed release, release-group fallback,
unresolvable→fuzzy, plain query skips direct lookup).
2026-06-07 13:04:17 -07:00
BoulderBadgeDad
83c1cd92aa Auto-reconcile embedded IDs for new tracks on library scans
Extends the manual "Import IDs from File Tags" backfill so newly-scanned
files get their embedded provider IDs pulled into the DB automatically —
no button press needed to keep up with new music.

How it works:
- insert_or_update_media_track now returns 'inserted' / 'updated' / False
  (truthy-compatible; existing `if track_success` callers unaffected) so
  the scan worker can tell a genuinely new row from an update.
- DatabaseUpdateWorker collects the ids it newly INSERTED this run
  (self._new_track_ids) across all insert paths (Plex/Jellyfin/deep).
- After run()/run_deep_scan(), web_server calls _reconcile_after_scan(),
  which gap-fills embedded IDs for just those new tracks. Runs as a
  post-scan pass (the scan loop itself is untouched/fast — the media
  server API never exposes these custom IDs, so the file must be read
  once regardless; batching at the end keeps it out of the hot loop and
  best-effort so it can never abort a scan). A progress phase ("Reading
  file tags for N new tracks…") surfaces the full-refresh tail.

Shared engine:
- New reconcile_library() in core does the paging + lazy parent-map
  loading (only loads albums/artists actually referenced — cheap when
  scoped to a few new tracks) + per-page commits. BOTH the manual button
  and the scan hook call it, so there's one tested orchestration, no
  duplication. The backfill job was refactored onto it.

Same hardened safety: gap-fill only, atomically guarded against
overwrite, schema-introspected, idempotent. Scoped to new arrivals for
incremental/deep; full refresh re-inserts everything as new (recovering
the IDs a full-refresh wipe destroys).

+10 reconcile tests (reconcile_library scope/idempotency/progress/stop +
the engine). Full suite clean (only pre-existing soundcloud /app env
failures remain).
2026-06-05 18:31:11 -07:00
BoulderBadgeDad
e6d86dea26 Add "Import IDs from File Tags" backfill — gap-fill provider IDs from embedded tags
Files SoulSync (or MusicBrainz Picard) already tagged carry Spotify /
iTunes / MusicBrainz / Deezer / Tidal / AudioDB / Genius / Last.fm IDs in
their metadata. Enrichment workers gate their queues on
{provider}_match_status IS NULL, so reading those IDs back and gap-filling
the {provider}_id + match_status='matched' columns lets the workers skip
the API lookup entirely — big API savings on an already-tagged library.

New manual job in Tools -> Database & Scanning ("Import IDs from File
Tags"): scans every library file, reads embedded IDs, fills any that are
missing in the DB. Background job + progress card, mirroring the
write-tags-batch pattern.

core/library/embedded_id_reconcile.py (pure + tested):
- plan_reconcile(): gap-fill plan for a track + its album + artist. Only
  empty id columns are planned; a disagreeing embedded id is a conflict,
  never applied.
- apply_reconcile_plan(): one guarded UPDATE per id column —
  WHERE id=? AND (col IS NULL OR col=''). The guard makes the fill atomic:
  if an enrichment worker matched the same entity between our read and
  this write, the UPDATE affects 0 rows instead of clobbering it. Columns
  are introspected so a schema missing a provider's columns is skipped.
- reconcile_track_row(): per-track orchestration (id extraction, plan ->
  apply, keeping the in-memory parent maps fresh for sibling tracks).

Job hardening: paged track scan (bounded memory), per-page commits (don't
starve concurrent workers), per-file try/finally (one bad file can't abort
the run), counters from real rowcount.

Scope: 19 column-fills across 8 providers. MB *recording* (track) id is
left out (UFID frame the reader doesn't surface; Vorbis key ambiguous) —
MB album+artist are covered. Amazon/ASIN deliberately excluded (ASIN is a
different namespace than the worker's amazon_id). All target columns
verified against the live schema.

Purely additive: new module, two new endpoints, one new Tools card —
no existing behavior changed. 20 unit tests (incl. the concurrency guard).
Full suite clean (only pre-existing soundcloud /app env failures remain).
2026-06-05 17:52:29 -07:00
BoulderBadgeDad
d91e6a384d Remove the old Retag Tool (superseded by Library Re-tag job + Write Tags)
The old per-download Retag Tool was limited (only native-pipeline downloads,
100-group cap, manual per-group) and did the wrong thing — it moved/reorganized
files instead of just tagging. It's superseded by the new Library Re-tag job
(whole-library, in-place) + the enhanced-library 'Write Tags' button.

Removed: the post-download record_retag_download ingestion hook (stops writing
retag_groups on every download), core/library/retag.py, the web_server state +
deps + /api/retag/* endpoints + the tool:retag WebSocket emit, the dashboard
card + both modals (index.html), the core.js socket handler, and the tools-page
wiring + help entry (wishlist-tools.js). Updated the import-pipeline test.

Verified: web_server parses, app + core imports OK, 392 tests pass, no live
references to removed symbols.

Left as inert (harmless) for a careful follow-up sweep: the retag_groups/
retag_tracks tables + their DB CRUD methods (no longer written/read), and the
now-orphaned retag JS helper functions (no entry point/wiring/socket calls them;
interspersed with wishlist functions, so not blind-deleted).
2026-06-04 09:33:03 -07:00
BoulderBadgeDad
b0c78c8674 Library re-tag (1/3): pure planner — match source tracklist + per-field tag diff
The testable core for the new library-wide re-tag job. Given a source album's
metadata + tracklist and the library tracks' current file tags, it:
- matches source tracks to library tracks (disc+track number, then title sim),
- computes the per-field diff (old -> new) for the dry-run finding,
- builds the minimal write_tags_to_file payload — only fields that actually
  change under the chosen mode (overwrite vs fill-missing), so applying never
  touches unrelated/unchanged tags.
No IO/network/DB — 10 unit tests cover matching, both modes, blank-source
fields, and the album-artist/track-count payload mapping.
2026-06-04 08:50:40 -07:00
BoulderBadgeDad
28850672a6 Fix: duplicate detector kept lossy over lossless (rank format first)
The Duplicate Detector's 'Keep Best' auto-selection ranked copies by highest
bitrate -> duration -> track number, with no notion of format. A FLAC whose
bitrate the library scan never populated (a common gap) therefore lost to a
282 kbps MP3: 282 > 0, so the MP3 was kept and the FLAC deleted (reported on
Havok 'Prepare For Attack', and again on Kendrick GNX).

Fix: rank by format/lossless tier FIRST, then bitrate, duration, track number.
A lossless file now always beats a lossy one regardless of the recorded
bitrate; bitrate/duration/track# only break ties within the same format.

- core/library/duplicate_keep.py (new): pure, importable pick_duplicate_to_keep
  + duplicate_keep_sort_key + format_rank_for_path (extension rank mirroring
  auto_import_worker._quality_rank: flac=10 ... mp3=5 ... unknown=1).
- core/repair_worker.py: _fix_duplicates auto-pick now calls
  pick_duplicate_to_keep instead of the bitrate-first max().
- webui/static/enrichment.js: the KEEP/REMOVE recommendation mirrors the same
  format-first ranking so the badge matches what the backend will delete.

Parity: Python uses '.ext' keys (os.path.splitext), JS uses 'ext'
(split('.').pop()) -> identical results; both keep the first copy on a full
tie. Verified the only other dedup path (the standalone Duplicate Cleaner
automation, core/library/duplicate_cleaner.py) was already format-priority-first
and correct -- no change needed there.

Tests: tests/test_duplicate_keep.py (11 -- incl. the exact FLAC-with-missing-
bitrate vs 282 kbps MP3 case, format ranking, within-format tie-breakers, and
edge cases). 147 repair/duplicate tests still pass.

Note: why FLAC bitrate is NULL in the DB is a separate library-scan gap;
format-first ranking makes the keep decision correct regardless.
2026-06-01 12:49:34 -07:00
BoulderBadgeDad
ce9ec3f6f4 Manual library match: accept non-numeric library track ids (#754)
The save endpoint coerced library_track_id with int(), which rejected
every non-numeric id with "Invalid library track id". Library ids are
str(ratingKey) — numeric for Plex but GUIDs/hashes for Navidrome,
Jellyfin, and other Subsonic servers — and are stored in the TEXT
tracks.id column, so the coercion broke manual matching on every
non-Plex server.

Replace the int() coercion with a normalize_library_track_id() helper
that trims and rejects only empty input, passing the opaque string id
straight through. Plex numeric ids are unaffected (SQLite INTEGER
affinity still stores a clean numeric string as an int, so existing
matches are byte-identical) and no schema migration is needed (the
INTEGER column already stores non-numeric ids as text).

Tests: pure-helper cases (numeric/GUID/whitespace/empty) plus a real-DB
round-trip proving a GUID id saves, reads back unchanged, and enriches.
2026-05-31 09:11:46 -07:00
Broque Thomas
43ed30b4d2 fix(musicbrainz): user-facing search recall + album-detail 404
Two bugs surfacing on the Fix popup and enhanced-search MB tab:

1. Strict Lucene phrase queries (`recording:"X" AND artist:"Y"`) killed
   recall on user-facing manual search — diacritics ("Bjork" vs canonical
   "Björk"), bracketed suffixes like "(Live)", and any AND-clause
   mismatch returned zero results. Added `strict: bool = True` param to
   `search_release` / `search_recording`; when False, sends a bare query
   joining title + artist so MB hits alias/sortname indexes with
   diacritic folding. `/api/musicbrainz/search` (Fix popup) and
   `core/library/service_search.py` (service tabs) now pass strict=False.
   Enrichment workers stay on strict mode — precision matters there
   because they auto-accept the top hit above a confidence threshold.

2. Every MB album click was silently 404-ing — `_render_release_as_album`
   passed `cover-art-archive` as an MB `inc` param, but it's not a valid
   include for the /release resource (MB rejects with 400). The CAA flags
   come back on every release response by default, so dropping the bad
   include preserves the image-scope picker logic intact.
2026-05-19 15:38:24 -07:00
Broque Thomas
aaf312cd34 Honor manual library matches across source labels
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.

Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
2026-05-18 16:32:38 -07:00
Broque Thomas
0345478361 Skip wishlist adds for manual library matches 2026-05-17 20:50:55 -07:00
Broque Thomas
42f4aa5eac Add manual library track matching 2026-05-17 20:27:05 -07:00
Broque Thomas
f3d5ef6528 Test missing-track existing file imports
Add service-level coverage for the Enhanced Library I Have This flow: copying an existing source file, writing the target album DB row, preserving source audio, inheriting album identity tags, and migrating older track tables that lack disc_number.
2026-05-17 14:18:17 -07:00
Broque Thomas
f9ae0e8d58 Extract missing-track import service
Move the existing-file missing-track import workflow out of web_server.py and into core/library/missing_track_import.py.

Keep the Flask route focused on request wiring and response formatting while the service handles staging copy, post-processing, album identity tag inheritance, DB upsert, and media-server sync.
2026-05-17 14:04:28 -07:00
Broque Thomas
42a833fcb2 Amazon Music: UI badges, enrichment match chips, watchlist linking, metadata cache
- Artist cards, hero section, and enhanced view now show Amazon Music badges
  when amazon_id is populated (AMAZON_LOGO_URL constant, orange #FF9900 brand)
- Enhanced view artist and album match status rows include amazon_match_status
  chip with click-to-rematch via openManualMatchModal
- getServiceUrl: added amazon (album/track ASIN → music.amazon.com) and fixed
  missing discogs entries; serviceLabels adds tidal/qobuz/amazon
- Enhanced view enhanced-artist-id-badges includes amazon_id entry
- DB SELECTs for library artists list and artist detail now return amazon_id;
  both response dicts include the field
- watchlist_artists migration adds amazon_artist_id column
- Watchlist config GET: amazon_artist_id in SELECT/WHERE/response (index 18)
- Watchlist artists list response includes amazon_artist_id
- link-provider endpoint: amazon added to valid_providers and col_map
- _populateLinkedProviderSection: amazonId param + Amazon Music source row
- Watchlist card source badges render Amazon pill (watchlist-source-amazon CSS)
- _openSourceSearch labels map includes amazon
- service_search: amazon_worker injected via init(); _search_service amazon branch
  uses search_artists/albums/tracks, same {id,name,image,extra} return shape
- _SERVICE_ID_COLUMNS: amazon → amazon_id for artist/album/track
- _init_service_search call passes amazon_worker_obj
- amazon_client._fetch_album_metas: 5-minute TTL cache per ASIN — cached hits
  skip _rate_limit() and HTTP call entirely; fixes ~10s artist detail load
- registry.py: removed amazon from METADATA_SOURCE_PRIORITY and
  METADATA_SOURCE_LABELS — T2Tunes has no discography API, cannot serve as a
  primary metadata source; Amazon remains a download source + ASIN enricher
- Settings metadata source dropdown and help text updated accordingly
2026-05-16 22:52:27 -07:00
Broque Thomas
b05ba5d498 Reorganize: optional embedded-tag mode (closes #592)
Adds an opt-in alternative metadata source for reorganize. The
existing API path (query Spotify / iTunes / Deezer / Discogs /
Hydrabase for the canonical tracklist) stays the default and is
unchanged. The new tag mode reads each file's embedded tags as the
source of truth instead -- useful for well-enriched libraries where
API drift can produce inconsistent renames, and avoids API calls
entirely.

- New pure helper `core/library/reorganize_tag_source.py` adapts the
  output of `read_embedded_tags` (the same mutagen path the audit-
  trail modal uses) to the `api_album` / `api_track` shapes that
  `_build_post_process_context` already consumes. Handles ID3-style
  "5/12" track + disc shapes, multi-value Artists tags, year
  normalization across 5 date formats, releasetype canonical tokens,
  multi-artist string splits across 9 separators.
- `plan_album_reorganize` accepts `metadata_source: 'api' | 'tags'`
  (default 'api') and `resolve_file_path_fn`. Tag mode branches into
  a new `_plan_from_tags` that reads each track's file and produces
  per-item `api_album` + `api_track` instead of a shared one.
- `_run_post_process_for_track` accepts a per-item `api_album`
  override so each file's own album metadata flows through post-
  process (not a single shared dict).
- `total_discs` in tag mode honors the `totaldiscs` tag and the
  trailing `/N` of an ID3 `discnumber = "1/2"`. Partial-album
  reorganize still routes into the correct `Disc N/` subfolder when
  the tag knows the total even if not all discs are present locally.
- Bare `discnumber = "1"` no longer poisons `total_discs` -- it
  carries no total signal.
- `reorganize_album` surfaces a tag-mode-specific error when no
  files are readable, instead of the API-mode "run enrichment first"
  message which would mislead in tag mode.
- `QueueItem.metadata_source` field, `enqueue` / `enqueue_many`
  pass-through, runner injects `item.metadata_source` into
  `reorganize_album`.
- `web_server.py` endpoints accept `mode` body param. Falls back to
  the `library.reorganize_metadata_source` config setting, then to
  'api'. Strict allowlist (api / tags) -- anything else falls back.
- Frontend: per-album modal + reorganize-all modal both grow a new
  "Metadata Mode" dropdown above the source picker. Tag mode hides
  the source picker (irrelevant). Choice persisted in localStorage.
  Both preview + execute fetches send `mode` in body.

Tests:
- 49 boundary tests on the pure helper pin every shape: ID3 "5/12",
  multi-artist split, year normalization, releasetype validation,
  total_discs precedence, defensive paths.
- 6 planner-level integration tests pin the wiring: tag-mode with
  good tags, partial-disc with totaldiscs tag, file missing,
  some-match-some-fail, defensive resolve_file_path_fn=None,
  API-mode regression guard.
- All 3171 tests pass; 52 existing reorganize tests unchanged.
2026-05-15 07:56:18 -07:00
Broque Thomas
2f284efa57 Retag now re-embeds LYRICS tag instead of leaving it empty
Discord report (netti93). The download flow runs `enhance_file_metadata`
(clears all tags) then `generate_lrc_file` (writes .lrc sidecar AND
embeds USLT). The retag flow only ran the first half — `enhance_file_metadata`
cleared USLT and there was no follow-up to restore it.

Two coordinated fixes (no new setting per kettui scope discipline —
user described it as "might even be an idea," consistency was the
load-bearing ask).

Fix 1 — retag calls generate_lrc_file after enhance

`core/library/retag.py:execute_retag` now invokes
`deps.generate_lrc_file` right after the `enhance_file_metadata`
call, mirroring the download pipeline. New `generate_lrc_file`
field on `RetagDeps`, defaults to None for backward compat with
any test caller that builds RetagDeps without it. Web_server's
`_build_retag_deps()` factory wires in the real
`core.metadata.lyrics.generate_lrc_file`.

Placement matters — runs BEFORE `safe_move_file` so the helper
sees the audio file at its current path with its existing sidecar
(which retag hasn't moved yet). After the embed, the audio file
gets moved with USLT now present; the sidecar move step that
follows is unaffected.

Fix 2 — create_lrc_file re-embeds from existing sidecar

`core/lyrics_client.py:create_lrc_file` used to early-return True
when an .lrc / .txt sidecar already existed (skipping the LRClib
fetch). For the retag case the sidecar is already there, so the
shortcut hit and USLT was never re-written. Now the helper reads
the existing sidecar and calls `_embed_lyrics` with its content
before returning. Empty / unreadable sidecars short-circuit
silently — defensive, no crash. Download flow unaffected because
no sidecar exists at fetch time.

7 boundary tests pin: existing .lrc triggers re-embed, existing
.txt triggers re-embed, empty sidecar skips embed, unreadable
sidecar swallows error, no sidecar falls through to LRClib (download
path regression guard), RetagDeps.generate_lrc_file field accepted,
field optional for backward compat.

Full suite: 3120 passed.
2026-05-14 15:52:05 -07:00
Broque Thomas
89246a7304 Write artist.jpg to artist folder so Navidrome shows real photos
Closes #572 (rhwc).

Navidrome has no API for setting an artist image — it reads
`artist.jpg` (or `folder.jpg`) from the artist folder during
library scans. SoulSync's `update_artist_poster` for Navidrome
was a no-op, so users only ever saw album-art-derived thumbnails
as the artist photo.

- new "Write Artist Image" button on artist detail page
- POST /api/artist/<id>/write-image-to-disk derives the artist
  folder from any track's resolved file_path (reuses
  _resolve_library_file_path so docker mount translation +
  library.music_paths probes from #558 apply), fetches the photo
  from the configured metadata source priority chain, downloads
  with content-type validation, writes atomically via
  `<filename>.tmp + os.replace`
- when active server is Navidrome, triggers a library scan
  immediately so the file is picked up
- respects existing artist.jpg (frontend prompts before
  overwriting) so user-supplied photos aren't clobbered
- works for plex / jellyfin too as a fallback layer — both
  servers also read artist.jpg from disk

26 tests pin the pure helpers in core/library/artist_image.py:
folder derivation (trailing sep / empty / non-string), URL
picking (missing attr / whitespace / non-string), download
(non-image content-type / 404 / timeout / empty body), atomic
write (replace / temp-cleanup-on-failure / overwrite guard /
missing folder).
2026-05-13 11:48:09 -07:00
Broque Thomas
6ce185491d Add per-download Audit Trail modal to Library History
- new "Audit" button on each download row in the library history
  modal opens a second modal visualizing the download lifecycle as
  an interactive horizontal stepper (request → source → match →
  verify → process → place) with click-to-expand detail cards
- hero header with album art + track title + meta line + status
  pills (source / quality / acoustid result)
- three tabs: Lifecycle / Tags / Lyrics
- Tags tab reads the audio file live via mutagen at audit-open
  time via new GET /api/library/history/<id>/file-tags endpoint;
  file is the single source of truth so background enrichment
  writes (audiodb / lastfm / genius / replaygain / lyrics fetch)
  show up too. flat key/value rows stacked vertically (label-above-
  value) so long MBIDs / URLs / joined genre lists wrap cleanly.
  source IDs grouped per-service into 2-col sub-card grid.
- Lyrics tab renders the full transcript with dimmed timecodes.
- post-processing step infers observable changes from source-vs-
  final state (format conversion, file rename via tag template,
  folder template).
- "Download History" button also added to the Downloads page batch
  panel header so it's reachable outside the dashboard.
- mobile responsive: tabs + stepper scroll horizontally, modal
  goes full-screen, hero stacks below 480px.

19 helper tests pin the mutagen reader: id3 (TIT2/TPE1/TALB + TXXX
+ USLT + APIC), vorbis (FLAC dict + _id/_url passthrough), file
metadata (format / bitrate / duration), defensive paths (empty /
missing file / mutagen returns None / mutagen raises), stringify
edge cases (list / tuple / int / frame-with-text / whitespace).
2026-05-13 09:50:24 -07:00
Broque Thomas
56ae10693b Album Completeness: surface diagnostic when resolver can't find album folder
GitHub issue #558: clicking Auto-Fill / Fix Selected on the Album
Completeness findings page returned a flat "Could not determine album
folder from existing tracks" error with no diagnostic. Reporter is on
Navidrome on Docker — the path resolver in
`core/library/path_resolver.py` couldn't find any of the album's tracks
on disk because Navidrome's Subsonic API doesn't expose filesystem
library paths the way Plex's API does (probed in #476). Default
settings → `library.music_paths` empty → no base directories to probe →
silent None. User had no signal about what to configure.

Not a regression of #476 — that fix targeted Plex auto-discovery and
worked correctly for it. Navidrome was never covered because the
protocol gives the resolver nothing to probe.

Fix scoped to the diagnostic surface, not auto-magic discovery:

- Added `resolve_library_file_path_with_diagnostic` returning
  `(resolved, ResolveAttempt)`. ResolveAttempt records what the resolver
  tried — `raw_path_existed`, `base_dirs_tried`, `had_config_manager`,
  `had_plex_client`. Pure data, no rendering opinions.
- Legacy `resolve_library_file_path` becomes a thin wrapper that
  drops the attempt; every existing call site is unchanged.
- `RepairWorker._fix_incomplete_album` now uses the diagnostic helper
  and renders a multi-part error via `_build_unresolvable_album_folder_error`:
  names the active media server, shows one sample DB-recorded path,
  lists every base directory the resolver actually probed, and points
  the user at Settings → Library → Music Paths as the actionable fix.
- Distinguishes empty-base-dirs vs tried-and-failed cases so the user
  knows whether to add a mount or fix the existing one.
- No auto-probing of common Docker conventions (`/music`, `/media`, etc).
  Speculative — could resolve to wrong dirs on the suffix-walk if a
  conventional path happens to contain a partial collision. User stays
  in control.

12 new tests:
- 7 in `tests/library/test_path_resolver.py`: tuple-shape contract,
  raw-path-existed short-circuit, base-dirs listed even on walk
  failure, had-flags reflect caller inputs, no-base-dirs returns
  None with empty attempt, legacy `resolve_library_file_path`
  delegates correctly across happy / suffix-walk / failure paths.
- 8 in `tests/test_repair_worker_unresolvable_folder_error.py`:
  active server name in error, sample DB path verbatim, base dirs
  listed, empty-base-dirs phrased differently, Settings hint always
  present, defensive against None attempt / missing sample / missing
  config_manager.

Full pytest sweep: 2774 passed.
2026-05-12 14:04:15 -07:00
Broque Thomas
9602d1827c Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites
Catches the silent excepts the awk-based earlier sweeps missed:

- Bare `except:` followed by `pass` (also swallows KeyboardInterrupt
  and SystemExit — actively wrong). Upgraded to `except Exception as
  e: logger.debug("...: %s", e)`. ~14 sites across connection_detect,
  soulseek_client, listenbrainz_manager, watchlist_scanner,
  youtube_client, navidrome_client, jellyfin_client, web_server.
- `except Exception:` + pass that the awk pattern missed (e.g.
  multi-line or unusual whitespace). ~31 sites across automation_engine,
  database_update_worker, music_database, spotify_client, web_server,
  others.
- 14 legitimate cleanup sites left silent with explicit `# noqa: S110`
  + comment explaining why (atexit handlers, finally-block conn.close
  calls). Logging during shutdown can itself crash because file handles
  get torn down before the handler fires.

Also enables `S110` rule in pyproject.toml so this pattern fails CI
going forward — drift fails at PR review instead of at runtime against
a wedged worker thread. Tests path keeps S110 ignored (test fixtures
legitimately use try-except-pass for cleanup).

Adds a WHATS_NEW entry to helper.js summarizing the full #369 sweep.

Verified: `python -m ruff check .` → All checks passed.
Verified: `python -m pytest tests/` → 2188 passed.

Closes #369
2026-05-07 11:16:06 -07:00
Broque Thomas
aa54bed818 Surface silent exceptions across remaining modules — ~70 sites
Final sweep. Covers:
- Downloads: candidates / lifecycle / master / monitor / wishlist_failed
- Metadata: source / registry / cache / common / artwork (+ plex_client)
- Imports: pipeline / resolution / file_ops / paths / guards
- Library: path_resolver / retag / duplicate_cleaner
- Stats / playlists / wishlist / discovery / automation / enrichment
- Misc: hydrabase_client, soulsync_client, tag_writer, debug_info,
  api_call_tracker, album_consistency, beatport_unified_scraper,
  reorganize_runner, seasonal_discovery, lidarr_download_client,
  services/sync_service.py, automation_engine, automation/progress

Two `_e` renames in imports/file_ops.py (outer scope binding `e`).
A few finally-block sites in metadata/album_mbid_cache.py,
library/track_identity.py, listening_stats_worker.py, watchlist/
auto_scan.py left silent — same reason as the rest of the sweep
(logger calls during cleanup paths can themselves raise).

Refs #369
2026-05-07 10:28:58 -07:00
Broque Thomas
d17365296a Lift shared download dataclasses + boot via singleton factory
Two architectural cleanups on top of the download engine refactor.

(1) Shared dataclasses move to neutral plugin package.
TrackResult, AlbumResult, DownloadStatus, SearchResult lived in
core/soulseek_client.py for historical reasons — every other plugin
imported them from the soulseek module just to satisfy the contract,
coupling 8 clients to a sibling source for type imports only. Moved
them to the new core/download_plugins/types.py module and updated all
14 import sites across the deezer/hifi/lidarr/qobuz/soundcloud/tidal/
youtube clients, the engine, matching engine, redownload helper, and
tests. Clean break, no backward-compat re-export.

(2) web_server.py boots the orchestrator via the singleton factory.
After construction it now calls set_download_orchestrator(...) so
get_download_orchestrator() returns the same instance the global
handle points at instead of lazily building a separate orchestrator.
Matches the get_metadata_engine() pattern.
2026-05-05 09:08:39 -07:00
Broque Thomas
d8437c87c6 Fix Album Completeness Auto-Fill on Docker / shared-library setups (#476)
GitHub issue #476 (gabistek, Docker on Arch host): "Auto-Fill" / "Fix
Selected" on the Album Completeness findings page returned
"Could not determine album folder from existing tracks" for every album.
Reproduces on any setup where the media-server library lives outside the
SoulSync transfer/download folders — Docker is the headline case but
native installs that point Plex at a NAS via SMB hit it too.

Root cause: `core/repair_worker.py:_resolve_file_path` only probed the
transfer + download folders. Docker users have their Plex/Jellyfin
library bind-mounted at /music (or similar) — neither configured in
SoulSync. Every existing track got silently treated as missing, so
`album_folder` stayed None and the fix workflow bailed.

The same incomplete logic was duplicated four more times in the
repair_jobs/ modules, all with the same bug. Album Completeness was
just the most user-visible — the same setups were also producing false
"missing file" findings from Dead File Cleaner, silent skips in
MBID Mismatch Detector, etc.

The web server already had the correct logic at
`web_server.py:_resolve_library_file_path` (probes transfer + download
+ Plex-reported library locations + user-configured library.music_paths).
The repair workers had never been updated to match.

Fix:
- New `core/library/path_resolver.py` extracts the union logic into a
  single shared function `resolve_library_file_path()`. Probes (in
  order, deduped): explicit transfer/download kwargs, config-derived
  soulseek.transfer_path/download_path, Plex-reported library
  locations (when a plex_client is passed), user-configured
  library.music_paths. Each defensive: malformed config or a flaky
  Plex client degrades to the dirs that did succeed.
- `core/repair_worker.py:_resolve_file_path` becomes a delegating
  wrapper preserving the legacy signature, with a new `config_manager`
  kwarg. All 15 in-tree call sites updated to thread
  `self._config_manager` through.
- `core/repair_jobs/dead_file_cleaner.py`,
  `mbid_mismatch_detector.py`, and `lossy_converter.py` get the same
  treatment: duplicate function replaced with a thin wrapper, call
  sites pass `context.config_manager`.
- `core/repair_jobs/acoustid_scanner.py` and
  `unknown_artist_fixer.py` (which used to import from repair_worker)
  now call the shared resolver directly with `context.config_manager`.

Side benefit: every other repair job (Dead File Cleaner, MBID
Mismatch Detector, Lossy Converter, AcoustID Scanner, Unknown Artist
Fixer) also stops missing files in the media-server library mount.
Single fix unblocks five user-visible features.

Tests: `tests/library/test_path_resolver.py` — 20 cases covering all
four base-dir sources, suffix-walk algorithm, dedup, defensive paths
(None plex client, malformed config entries, raising config_manager.get,
broken plex attribute access), Docker path translation. Full suite
1677 passed locally.

WHATS_NEW entry under '2.4.2' dev cycle.
2026-05-03 10:11:06 -07:00
Broque Thomas
24c2d75c6d Make extract_external_ids recognize all source-tagging conventions
Smoke-testing the just-merged provenance PR against live logs revealed
the new ID-match block was silently no-opping: no [ExtID Match] /
[Provenance Match] log lines despite the code path being live. Tracing
revealed two related gaps in extract_external_ids' source detection:

1. **Underscore-prefixed key.** Deezer / Discogs / Hydrabase clients
   tag normalized track dicts with ``_source`` (underscore prefix —
   convention used in 8+ places across core/). The extractor only
   looked for ``provider`` and ``source``, so Deezer-sourced tracks
   silently returned no IDs.

2. **No provider field at all.** Spotify and iTunes raw API responses
   carry ``id`` but no provider/source key of any kind. The extractor
   couldn't disambiguate the native ``id``, so Spotify-primary scans
   would have hit the same silent miss once the user switched primary
   sources.

Two-part fix:

- ``extract_external_ids`` now recognizes ``_source`` as another
  candidate provider field.
- New optional ``source_hint`` parameter lets the caller supply the
  configured primary source as a fallback when the track dict has no
  provider field of its own. Track-side provider field still wins
  when present (defensive against a wrong hint).

Watchlist scanner now passes ``get_primary_source()`` as the hint so
both naming conventions (Deezer-style _source, Spotify-style no-tag)
get handled uniformly.

6 new regression tests cover:
- _source recognized for Deezer
- _source recognized for Hydrabase (cross-provider mapping)
- _source recognized for Discogs (no library column — verifies
  graceful no-crash)
- source_hint disambiguates raw tracks for spotify/itunes/deezer
- track-side provider takes precedence over hint
- None hint defaults safely

Full pytest 1630 passed; ruff clean. After this lands and the server
restarts, watchlist scans should produce [ExtID Match] /
[Provenance Match] log lines for tracks already on disk regardless of
which metadata source the user has configured as primary.
2026-05-02 18:26:12 -07:00
Broque Thomas
34ba26f5c8 Persist source IDs at download time + backfill onto tracks on sync
Followup to fix/watchlist-external-id-match. The companion PR closed
the demand side — the watchlist scanner asks for tracks by external IDs
before falling back to fuzzy. But for users on Plex / Jellyfin /
Navidrome the supply side was still broken: tracks.spotify_track_id
(and the other ID columns) only got populated by the asynchronous
enrichment workers, sometimes hours after the file was actually
written. During that window the ID match fell through to fuzzy and
the bug returned.

We were already collecting every ID during post-processing — they
live in the `pp` dict in core/metadata/source.py:embed_source_ids and
get embedded into file tags. We just dropped the in-memory copy
afterwards.

This PR persists them and uses them:

- Schema migration adds spotify_track_id / itunes_track_id /
  deezer_track_id / tidal_track_id / qobuz_track_id /
  musicbrainz_recording_id / audiodb_id / soul_id / isrc columns +
  indexes to the existing track_downloads table (already keyed by
  file_path).
- core/metadata/source.py:embed_source_ids exposes pp["id_tags"] and
  the resolved ISRC back to the import context as _embedded_id_tags
  / _isrc.
- core/imports/side_effects.py:record_download_provenance reads those
  context fields and passes them to db.record_track_download, which
  now accepts the new ID kwargs and persists them.
- New db.get_provenance_by_file_path with exact + basename-suffix
  fallback (handles container mount-root differences between
  download-time path and media-server-reported path).
- New db.backfill_track_external_ids_from_provenance copies IDs
  from track_downloads onto a tracks row idempotently — COALESCE on
  every column preserves any value the enrichment worker already
  wrote (enrichment is more authoritative for late binding).
- database/music_database.py:insert_or_update_media_track (the
  single insertion point used by every Plex / Jellyfin / Navidrome
  sync) calls the backfill immediately after each INSERT/UPDATE.
- New core/library/track_identity.py:find_provenance_by_external_id
  used as a second-tier fallback in watchlist_scanner.is_track_missing
  _from_library — catches the window between download and media-server
  sync. Caller checks os.path.exists on the provenance file_path
  before treating it as "already in library" so a deleted file
  doesn't prevent re-download.

Effect: freshly downloaded files become ID-recognizable to the
watchlist on the very next scan, no enrichment-wait window.

19 regression tests in tests/test_provenance_id_persistence.py:
- Schema migration adds expected columns + indexes
- record_track_download persists every ID kwarg
- record_track_download backward-compat (old kwargs still work)
- get_provenance_by_file_path: exact match, basename fallback for
  mount-root differences, multi-record latest-wins, defensive None
- backfill: copies all IDs, preserves existing via COALESCE,
  no-op when no provenance exists
- find_provenance_by_external_id: per-ID lookup, ISRC cross-bridge,
  OR semantics, latest-wins on multiple matches

Out of scope: backfilling provenance for files downloaded BEFORE
this PR (their track_downloads rows don't carry the new IDs). Those
continue to wait for enrichment. Acceptable — only affects historical
files; new downloads benefit immediately.

Full pytest 1625 passed; ruff clean.
2026-05-02 17:44:10 -07:00
Broque Thomas
ecb8939c80 Match library tracks by external IDs before fuzzy in watchlist scan
Reported case (CAL): a track already on disk got re-downloaded by the
watchlist scanner on every scan. Library DB had stale album metadata
for the file (track tagged on album "Left Alone") while the metadata
source reported it on a different album ("NPC" single). The
title+artist+album fuzzy block correctly said the album names didn't
match and declared the track missing — but the file's stable external
IDs (Spotify ID, ISRC, etc.) unambiguously identified it as the same
recording.

The earlier compilation-album fix (PR #461) handled qualifier drift
("OST" vs "Music From The Motion Picture"). This case is two
genuinely different album names referring to the same song.

Fix: provider-neutral external-ID short-circuit before the fuzzy
block in `is_track_missing_from_library`. Pulls every recognized ID
off the source track (Spotify / iTunes / Deezer / Tidal / Qobuz /
MusicBrainz / AudioDB / Hydrabase / ISRC), runs a single SELECT
against the indexed external-ID columns on the `tracks` table, and
treats any hit as "track exists in library — don't re-download".

If no IDs are available (older imports without enrichment, library
scans that didn't populate external IDs), falls through to the
existing fuzzy logic so the safety net stays intact.

New `core/library/track_identity.py` module with two helpers:
- `extract_external_ids(track)`: handles dict and object-style track
  shapes, direct-field aliases (spotify_id / spotify_track_id /
  SPOTIFY_TRACK_ID), and provider-disambiguated native `id` fields
  (when track has `provider='deezer'` and `id='X'`, treats X as a
  Deezer ID).
- `find_library_track_by_external_id(db, external_ids,
  server_source)`: builds an OR of indexed column matches with
  IS NOT NULL guards, optional server_source filter that also
  passes legacy NULL rows, single-row LIMIT.

ISRC bridges across providers — a library track imported via Deezer
can be matched against a Spotify scan when both sides carry the
same ISRC.

43 regression tests in `tests/test_library_track_identity.py`:
- 9 ID-extraction tests for direct fields (Spotify / iTunes / Deezer /
  ISRC / MBID / AudioDB / Hydrabase)
- 8 ID-extraction tests via the provider field (8 providers + source
  alias + missing-provider-ignored)
- 7 mixed/defensive tests (multiple IDs, object-style, empty strings,
  None track, numeric coercion)
- 8 lookup tests (per-provider + ISRC cross-bridge)
- 3 OR-semantics tests
- 4 server_source filter tests
- 2 ID-column-map sanity tests

Full pytest 1606 passed; ruff clean.
2026-05-02 16:06:59 -07:00
Broque Thomas
b395e33820 Lift redownload_start to core/library/redownload.py
Body byte-identical to the original. Spotify proxy via registry,
iTunes/Deezer client shims wrap registry helpers,
_resolve_library_file_path, _attempt_download_with_candidates, and
missing_download_executor are injected via init() right after
_init_wishlist_failed where all three deps are already defined.

web_server.py: 35239 → 35063 (-176 lines).
2026-04-30 11:27:33 -07:00
Broque Thomas
8299dc211e Lift _run_duplicate_cleaner to core/library/duplicate_cleaner.py
Body byte-identical to the original. The shared state dict, lock,
docker_resolve_path helper, and automation engine are injected via
init() at the lift point, where all four originals are already defined.

web_server.py: 37015 → 36833 (-182 lines).
2026-04-29 20:10:22 -07:00
Broque Thomas
dae7f21265 Lift _search_service to core/library/service_search.py
Lifts _search_service and its _detect_provider helper. Both bodies are
byte-identical to the originals. The nine enrichment worker handles
(spotify/itunes/mb/lastfm/genius/tidal/qobuz/discogs/audiodb) are
injected via init() right after qobuz is constructed, which is the
last worker to come up — and well before Flask starts accepting
requests, so the route handlers never see unbound workers.

web_server.py: 37245 → 37015 (-230 lines).
2026-04-29 18:06:23 -07:00
Broque Thomas
3a6597561a Lift _execute_retag to core/library/retag.py
Pulls the 258-line retag worker out of `web_server.py` into a new
`core/library/` package. Pure 1:1 lift — wrapper keeps the original
entry-point name so the retag-trigger endpoint continues to work
without changes.

What `execute_retag` does:

1. Fetch album + track metadata for the new `album_id` (Spotify or
   iTunes — the Spotify client transparently falls back).
2. Load existing files in the retag group from the DB.
3. Match each existing track to a new Spotify track:
   - Priority 1: same disc + track number.
   - Priority 2: title similarity >= 0.6 (SequenceMatcher).
4. For each matched pair:
   - Re-write metadata tags via `_enhance_file_metadata`.
   - Compute the new path via `_build_final_path_for_track` and move
     the audio file (plus .lrc / .txt sidecars) if the path changes.
   - Drop an orphaned cover.jpg if it's left in an empty directory.
   - Clean up empty parent directories left behind.
   - Download the new cover art into the new album dir.
5. Update the retag group record with new artist / album / image /
   total_tracks / release_date and the appropriate Spotify-or-iTunes
   album ID (numeric → iTunes, alphanumeric → Spotify).
6. Mark the retag state 'finished' (or 'error' on exception).

Strict 1:1 byte parity:
The original mutated `retag_state` as a module global (the function
declared `global retag_state` even though it only mutates in place).
Here `retag_state` is exposed through the `RetagDeps` proxy as a Python
property so the lifted body keeps `name[key] = value` /
`name.update(...)` syntax. The property setter rebinds the
web_server.py reference if the function ever reassigns it (currently
it doesn't, but the setter is wired for parity with the watchlist lift).

Diff vs original after `deps.X` → global X normalization is **zero
differences** apart from the dropped `global retag_state` decl and the
inline `from database.music_database import get_database` (replaced by
deps.get_database()). 258 lines orig = 258 lines lifted, byte-identical
body otherwise.

Dependencies injected via `RetagDeps` (13 fields) — config_manager,
retag_lock, spotify_client, plus 8 callable helpers
(get_audio_quality_string, enhance_file_metadata,
build_final_path_for_track, safe_move_file, cleanup_empty_directories,
download_cover_art, docker_resolve_path, get_database) and 2 property
delegates (_get_retag_state / _set_retag_state).

Tests: 11 new under tests/library/test_retag.py covering setup error
paths (no album data, no album tracks, no existing tracks),
track-number priority match, title-similarity fallback, no-match skip,
missing file skip, file move when path changes, group record update
(spotify vs iTunes ID branching by alphanumeric vs numeric album_id),
multi-disc total_discs computation.

Full suite: 1330 passing (was 1319). Ruff clean.
2026-04-29 09:03:42 -07:00