Commit graph

3616 commits

Author SHA1 Message Date
BoulderBadgeDad
6fa733dc6e Artist Map v2 — fix the real bottleneck: cap the offscreen buffer (76MP → ~12MP)
Perf telemetry from the genre map (2004 nodes) proved it: the offscreen buffer
was 7465×10240 (76 megapixels) — rebuilt in ~979ms on every zoom and blitted at
~150ms/frame (3 fps), with the constellation overlay piling on top. The buffer
renders the WHOLE world, and the size cap was 10240px.

Cap the max buffer dimension to 4096 (MAX_BUFFER_PX). On the dense genre map
that's ~12MP instead of 76MP → ~6x faster rebuild and blit, and more nodes drop
under the LOD dot threshold so the rebuild also draws fewer image-clips. The cap
only binds on large worlds; small watchlist/explorer maps don't reach it and
stay full-resolution.

Tunable; perf overlay ('d' → app.log) stays so we can confirm the new numbers.
2026-06-03 08:03:30 -07:00
BoulderBadgeDad
884c3b5c07 Artist Map perf overlay: also POST timings to app.log (readable server-side)
The on-canvas overlay text can't be copied (and can't be grabbed mid-freeze), so
when perf mode is on ('d'), the frontend now also POSTs the render timings to
/api/discover/artist-map/perf ~1.5x/sec, which logs them as [ARTMAP-PERF] in
app.log. Lets the bottleneck be diagnosed from the server side with no manual
copying.
2026-06-03 07:54:32 -07:00
BoulderBadgeDad
77d6d49069 Artist Map v2 — fix regression + add perf overlay
- REVERT the spatial-grid hit-test I added in Phase 1. It inserted each node
  into every grid cell its bounding box overlaps; the genre map's huge cluster
  nodes span an enormous number of cells, so the first hover/click triggered a
  multi-second synchronous build → 'can't hover or click' freeze. Back to a flat
  O(N) single-pass hit-test (no per-move sort) — sub-ms even for thousands of
  nodes, can't lock up.
- Keep the safe Phase 1 wins (render coalescing, tooltip de-churn, solid-stroke
  connection lines).
- Add a perf overlay toggled with 'd' on the map: shows node/edge counts, the
  offscreen buffer size + scale, zoom, and the last buffer-rebuild + draw times.
  So we can measure the real drag/zoom bottleneck (buffer rebuild) instead of
  optimising blind.

JS clean; 64 integrity tests pass.
2026-06-03 07:34:41 -07:00
BoulderBadgeDad
b85392977c Artist Explorer: search-and-select instead of free text
The Explorer prompt accepted any loose text and explored whatever you typed.
Now it's a proper picker: type -> debounced search of the metadata source
(reuses /api/discover/build-playlist/search-artists — Hydrabase if active,
Spotify if configured, else the active metadata source) -> shows real artist
results with images -> click one to explore that resolved artist. Enter picks
the top match (never explores raw text); Escape/Cancel/backdrop close.

Pure frontend: rebuilds _showArtistMapSearchPrompt() (same Promise<name|null>
contract, so the caller is unchanged), reusing the playlist-builder's search
endpoint + picker styling. No backend change.
2026-06-03 07:12:04 -07:00
BoulderBadgeDad
1b21983ce7 Artist Map v2 — Phase 2: richer real data on hover (connection counts)
Investigating 'each mode is different / not good enough' showed the engine is
already shared across all three modes (watchlist/genre/explore) and already does
LOD rendering, eased camera, and debounced zoom-rebuilds — so the inconsistency
was perception driven mostly by the (now-fixed) lag, not separate engines.

This phase surfaces more real data the map already has: the hover tooltip now
shows each artist's live connection count (computed from the map edges), shown
consistently across all three modes. Cheap (only recomputed when the hovered
artist changes, after Phase 1's de-churn). Additive + safe.

JS syntax clean.
2026-06-02 23:41:56 -07:00
BoulderBadgeDad
14388d4f42 Artist Map v2 — Phase 1: performance engine
Kills the hover/move lag on dense maps. Root causes were in the live
mouse/render path, not the layout:

- Render coalescing: _artMapRender() now just requests a single rAF; the actual
  draw (_artMapDraw) runs at most once per frame. A burst of mousemove/pan/
  animation calls no longer triggers many full-buffer blits per second.
- Tooltip de-churn: only rebuild the tooltip innerHTML (and reload its image)
  when the hovered artist changes; a plain mousemove just repositions. Was
  rebuilding innerHTML + a new <img> every pixel of movement.
- Spatial-grid hit-test: bucket nodes into a coarse world grid and test only the
  cell under the cursor, instead of sorting + scanning every node each move.
  Grid rebuilds only when the node set changes.
- Constellation lines: draw all connection lines as ONE solid-stroke path
  instead of creating a fresh linear-gradient object per line every frame —
  that per-frame gradient churn was the main 'connected lines' lag.

No layout/data/click changes; behaviour identical, just frame-bound. Pure
frontend; JS syntax clean.
2026-06-02 23:37:27 -07:00
BoulderBadgeDad
1ad36963d1
Merge pull request #779 from Nezreka/feature/spotify-public-full-playlist
Feature/spotify public full playlist
2026-06-02 22:57:36 -07:00
BoulderBadgeDad
77b8d7dd1f SpotipyFree integration confirmed working (236 tracks live); deps + meta tweak
- Verified end-to-end: fetch_public_playlist_full pulled all 236 tracks of the
  test playlist via SpotipyFree (the library handles the client-auth that 429'd
  the raw approach). Name + tracks correct.
- requirements.txt: declare spotipyFree>=1.1.2 as a normal pip dependency (like
  spotDL, also MIT — aggregation, not vendored) + websockets (a transitive dep
  SpotipyFree/spotapi needs that pip doesn't pull automatically). Code still
  soft-imports + falls back to embed, so it's never a hard runtime requirement.
- meta fetch uses limit=1 (name/owner only) so we don't pull the whole list
  twice. 9 tests green.
2026-06-02 22:50:04 -07:00
BoulderBadgeDad
06f11dc95a Full public playlists via optional SpotipyFree (no creds), MIT-clean
The in-house anonymous-token path is blocked by Spotify (429 without the web
player's rotating client-auth). Switch the full-fetch to SpotipyFree — the
maintained no-creds spotipy drop-in spotDL uses, which tracks that machinery.

- core/spotify_public_api.fetch_public_playlist_full now uses a SpotipyFree
  client (playlist + playlist_items + next), normalising the spotipy-shaped
  items to the embed scraper's shape. Injectable client_factory keeps it
  unit-testable without the library or network. Dropped the dead in-house
  token/pagination code.
- Licensing: SpotipyFree is GPL-3.0, so it is NOT bundled/required (SoulSync is
  MIT). Optional, user-installed: the import is soft, and on ImportError (or any
  failure) fetch_spotify_public falls back to the embed scraper (~100). So the
  shipped project stays cleanly MIT and the link path never regresses.
- requirements.txt: documents it as a commented optional extra
  (pip install SpotipyFree) with the GPL/MIT rationale.
- 9 tests: normalisation, pagination past 100, library-missing -> raises (->
  fallback), and the embed-fallback orchestration.

Needs a live click-through with SpotipyFree installed to confirm the exact
class/method names match (SpotipyFree.Spotify / playlist / playlist_items).
2026-06-02 22:43:34 -07:00
BoulderBadgeDad
951293c56a Diagnostics: route public-fetch logs to soulsync namespace + log HTTP status
The full-fetch's logs used a bare module logger that app.log doesn't capture,
so we couldn't see whether the API path succeeded or why it fell back. Route
them to 'soulsync.spotify_public' and log: token found?, embed parsed?, the
API HTTP status on a non-200, and pagination result. Lets us see the exact
failure (e.g. 401 vs 429) on the next link-tab test.
2026-06-02 22:09:51 -07:00
BoulderBadgeDad
8b060ee79a Fix: pull anonymous token from the EMBED page; drop meta call; graceful partial
Live debugging the 'shows 100' report:
- The full playlist page no longer embeds an accessToken, and get_access_token
  / server-time now 403/404. The EMBED page (open.spotify.com/embed/playlist/{id})
  still ships a usable anonymous token. Was fetching the wrong page -> no token
  -> raised -> embed fallback (100). Now reads the embed page for the token.
- Confirmed live: token extraction + embed parse work; the token is accepted by
  the Web API (429 rate-limit, not 401). Could not show >100 from here because
  the test IP got rate-limited from probing; needs a clean-IP click-through.

While in there, made it more robust against the rate-limiting that's clearly in
play:
- Refactored scrape_spotify_embed -> reusable parse_embed_html.
- fetch_public_playlist_full now does ONE embed fetch for token + name + first
  page (no separate metadata call = fewer requests = less 429 surface), then
  paginates the API. If the API is unavailable/rate-limited, it keeps the embed
  page's tracks (<=100) instead of raising — so the result is always >= today's
  behaviour, never worse.
- 12 tests incl. the new API-fails-but-embed-tracks-survive path.

Caveat unchanged: rides Spotify's undocumented embed-page token; degrades to the
embed fallback, never crashes.
2026-06-02 21:59:56 -07:00
BoulderBadgeDad
dd7f048386 Full public playlist fetch for the 'Spotify link' path (no creds), embed fallback
The no-auth 'add by link' path scrapes Spotify's embed widget, which only ever
contains ~100 tracks and can't paginate — so big public playlists got
truncated. This adds an in-house anonymous fetch that pulls the FULL list:

- core/spotify_public_api.py: reads the anonymous web-player accessToken Spotify
  already embeds in its own open.spotify.com page HTML (no app credentials, and
  no rotating TOTP secret for us to maintain), then paginates
  /v1/playlists/{id}/tracks 100 at a time until the whole playlist is pulled.
  Returns the embed scraper's exact shape. Pure helpers + injected http_get so
  it's unit-testable without the network.
- core/spotify_public_scraper.fetch_spotify_public(): tries the full fetch for
  playlists; on ANY failure (or for albums) falls back to scrape_spotify_embed.
  Worst case == today's behaviour, so the link path can't regress.
- web_server: the link-tab endpoint and the authed flow's last-resort scrape
  now both go through fetch_spotify_public.

Scoped entirely to the spotify_public_* (no-auth) path — the authenticated
playlist sync is untouched. 11 tests (token extraction, normalisation,
pagination past 100, and the embed-fallback orchestration).

Caveat: rides Spotify's undocumented page-embedded token — expected to break
when they change their page; it degrades to the embed fallback, never crashes.
Needs a live click-through to confirm the token path works end to end (can't
hit Spotify from the test env).
2026-06-02 21:27:06 -07:00
BoulderBadgeDad
73a8882ad7
Merge pull request #778 from Nezreka/feature/enrichment-workers-manager
Feature/enrichment workers manager
2026-06-02 21:09:52 -07:00
BoulderBadgeDad
e1fd1f2489 test: register enrichment-manager.js in script-split-integrity scan
The onclick-coverage guard only scans the split modules + a hardcoded extras
list, so it flagged openEnrichmentManager() (defined in the new, loaded
enrichment-manager.js) as undefined. Add enrichment-manager.js to the scanned
non-split files. The function genuinely exists and is loaded via its script tag.
2026-06-02 21:06:02 -07:00
BoulderBadgeDad
b0d5008673 Global process-first now re-queues failed too (matches per-worker pin)
Global priority previously set order only; per-worker pin also re-queued the
group's failed items. Made global consistent: setting a group globally now also
resets that group's not_found -> pending on every supporting worker, so each
worker sweeps ALL pending + failed of the group before moving on. Toast reports
total re-queued. Workers that don't enrich the group are skipped.
2026-06-02 20:48:39 -07:00
BoulderBadgeDad
3a560bd1bb Enrichment manager: rebuilt header + global process-first + cleaner rows
- Rebuilt the modal header: gradient top bar with a glowing 🧬 icon chip,
  gradient title + subtitle, and styled refresh/close — replaces the flat bar.
- Global 'process first everywhere' control in the header: Artists/Albums/
  Tracks/Auto applies to every worker at once (workers that don't enrich a
  group are skipped via the 400 the endpoint already returns). Sets order only.
- Match rows: replaced the loud accent-gradient artwork placeholder with a
  subtle neutral chip showing the entity glyph; real images layer over it and
  remove themselves on error, so missing/broken art never leaves ragged gaps.
- Removed overflow:hidden from .em-row.

Frontend only; JS syntax clean.
2026-06-02 20:41:05 -07:00
BoulderBadgeDad
f827ddc282 Enrichment manager: merge coverage+order into one card set, fix ordering, richer rows
Addresses three pieces of UI feedback:
- Fix entity order: enrichment coverage was rendering by object-key order
  (albums first). Now sorted canonically artist → album → track via
  _emOrderEntities, used everywhere.
- Combine 'Processing order' and 'Enrichment coverage' into a single set of
  entity cards: each card shows coverage (segmented matched/not_found/pending
  bar + %) AND is the click target to pin that group to enrich first, with
  live 'Now' / pinned 'First 📌' / 'Done' states and per-worker accent. Drops
  the two redundant sections (and the old chain/stats renderers).
- Richer match rows: status stripe down the left edge (red=not found,
  amber=pending), larger rounded artwork with a gradient placeholder, parent
  context (artist/album), and a subtle slide-on-hover.

Frontend only; JS syntax clean.
2026-06-02 20:27:57 -07:00
BoulderBadgeDad
f8afe56642 Enrichment priority: pinning sweeps the whole group; rail shows live phase
Aligns the 'process this group first' behaviour with intent:
- Pinning a group now also re-queues that group's previously-failed
  (not_found -> pending) items, so the worker processes ALL unmatched in the
  group (pending + missing), not just never-tried ones. Safe from loops: each
  is attempted once, still-unmatched return to not_found, and the pending-only
  worker hook won't re-pick them. Toast reports how many were re-queued.
- The left rail now shows each worker's current group while running
  ('Running · albums'), so you can see what every service is on at a glance.

Frontend only; reuses the tested /priority + /retry endpoints.
2026-06-02 19:58:14 -07:00
BoulderBadgeDad
62ee1f8520 Enrichment manager: 6 UX improvements
- #1 Unconfigured-source banner: when a source has enabled=false, show a
  notice that browsing works but matches/retries won't run until it's set up.
- #2 Rate-limit detail: when rate_limited, surface 'resumes in ~Xm' (from the
  status payload) instead of just a pill.
- #3 Richer rows: unmatched items now show parent context — an album's artist,
  a track's album — via a parent expression in the query (+ test).
- #4 Bulk select: per-row checkboxes + a bulk bar to retry several at once
  (capped concurrency), reusing the /retry item endpoint.
- #5 Remember last worker: selection persists in localStorage and is restored
  on open; openEnrichmentManager(workerId) supports future deep-linking
  (bubbles left on their pause-on-click behaviour).
- #6 Keyboard nav: ArrowUp/Down moves focus between rows; actions are native
  buttons (Enter/Space) and Escape closes — list isn't poll-refreshed so focus
  is stable.

53 enrichment tests green; JS syntax clean.
2026-06-02 19:50:57 -07:00
BoulderBadgeDad
e53a157793 Enrichment manager: 'process this group first' + refined hero header
Per-worker processing-order override + UI polish.

Feature — pin an entity group to enrich first:
- Each worker normally runs artist -> album -> track. A user can pin one
  group (artist/album/track) to run first from the modal; the worker keeps
  that group first until it's exhausted, then resumes the normal chain.
- core/worker_utils.py: read_enrichment_priority() (reads
  <service>_enrichment_priority each loop, live) + priority_pending_item()
  (shared, whitelisted query returning the worker's expected item shape;
  Spotify/iTunes get album_individual/track_individual via a type map).
- A guarded ~6-line hook at the top of all 11 workers' _get_next_item.
  CRITICAL: when nothing is pinned (default) the hook returns immediately,
  so default enrichment order is byte-identical to before. Discogs (no track)
  and Genius (no album) only honor their supported entities.
- core/enrichment/api.py: GET/POST /api/enrichment/<id>/priority (+ config_get
  hook); POST validates the entity against what the source enriches.
- 14 new tests (helper shapes, exhaustion, route get/set/clear/validate).

UI:
- Refined hero header: identity + inline status left, single Pause right,
  'now enriching' quiet sub-line; overall coverage % moved into the stats
  section ('82% matched · 1,203 of 1,460'). Hero gently pulses while running.
- New processing-order strip: artist→album→track steps showing the live phase
  (pulsing 'now'), pinned group ('first' + 📌), and done/remaining; click a
  step to pin it, click again for auto.

py_compile clean across all 11 workers; 52 enrichment tests green.
2026-06-02 19:45:04 -07:00
BoulderBadgeDad
fc9a9f1c90 Enrichment manager v2: working retry + bulk retry-all-failed
Fixes a correctness bug and adds bulk re-queuing.

- Bug: per-row 'Retry' used clear-match, which sets an item to not_found
  with last_attempted=NULL. The worker only retries not_found items where
  last_attempted < (now - 30d), and 'NULL < cutoff' is false in SQLite, so
  those items were never re-queued. Fixed by resetting match_status to NULL
  (pending), which every worker's queue picks up on the next pass.
- New POST /api/enrichment/<id>/retry with scope 'item' | 'failed'
  (failed = re-queue every not_found item of an entity type), backed by a
  pure whitelisted build_reset_query + MusicDatabase.reset_enrichment().
- UI: per-row Retry now hits /retry; a 'Retry all failed' bulk button appears
  when the current entity has not-found items (confirm + count toast); a hint
  line explains retry/match/auto-retry behaviour.
- 11 new tests (38 enrichment tests total, all green).
2026-06-02 19:19:39 -07:00
BoulderBadgeDad
0b3c3f656d Add Manage Enrichment Workers modal (v1 + polish)
Dashboard 'enrichment bubbles' could pause/hover but offered no way to
*manage* a worker. This adds a full management modal opened from a new
header button, covering all 11 enrichment sources.

Backend (testable core helper + seam tests; no live-DB dependency):
- core/enrichment/unmatched.py: pure, whitelisted SQL builders for the
  unmatched browser. service/entity validated against a support map (never
  interpolated raw); search + pagination bound as params; tracks join albums
  for artwork; limit capped at 200.
- database/music_database.py: get_enrichment_unmatched() +
  get_enrichment_breakdown() (the breakdown splits matched/not_found/pending,
  which the existing get_stats().progress lumps together).
- core/enrichment/api.py: GET /api/enrichment/<id>/{unmatched,breakdown} on
  the existing blueprint + a db_getter hook.
- web_server.py: wire db_getter=get_database.
- tests/enrichment/test_unmatched.py: 19 tests across builders, DB methods,
  and Flask routes.

Frontend (vanilla, matches app conventions):
- webui/static/enrichment-manager.js: worker rail with live status + coverage
  micro-bars, accent-themed detail panel (hero header, segmented matched/
  not_found/pending stat cards, current item, pause/resume), and a searchable
  paginated unmatched browser with inline manual match (reusing
  search-service + manual-match) and retry (clear-match re-queues).
- Polish: entrance/exit motion, scroll-lock, Escape, refresh control,
  flicker-free polling (in-place updates), skeleton loaders, relative
  timestamps, per-worker accent theming, real dashboard logos reused at
  runtime (with the same invert/circle treatment), responsive rail.
- index.html: header button + script include. style.css: full styling.

Reuses existing pause/resume, status, and manual search+assign endpoints.
Backend tests green (19 new + 11 existing enrichment tests).
2026-06-02 19:06:44 -07:00
BoulderBadgeDad
7956aaac9e Fix #772: manual import progress bar stuck at 0 / 'Failed' on slow imports
Per-track import does heavy synchronous server-side enrichment (metadata,
art, lyrics) that can take 60-90s/track, far longer when external sources
are degraded. The React apiClient (ky) had no timeout, so ky's default 10s
aborted the import-process request client-side even though the server
completed the import (200) and moved the files. The import loop then counted
the aborted call as an error, so the bar stayed at 0 and flipped to 'Failed'
while files imported fine.

Give the two import-process calls (album/process, singles/process) an
explicit 5-min timeout. Scoped to import only -- every other endpoint keeps
the 10s default; bounded, not disabled. Server behavior unchanged.

Adds a test asserting both calls pass the long timeout.
2026-06-02 16:02:26 -07:00
BoulderBadgeDad
e27e436465
Merge pull request #777 from Nezreka/canonical-album-version
Canonical album version
2026-06-02 15:48:19 -07:00
BoulderBadgeDad
eaf74732f9 Canonical: fix ruff lint (B023 loop-bound lambda, S110 bare except-pass)
- B023: default_fetch_tracklist built a per-item lambda closing over the loop
  variable `it`. Replaced with a module-level _item_get(item, key, default)
  helper (takes the item as a param — no closure). Behavior unchanged; the
  dict/object normalization test still passes.
- S110: the two best-effort guards in the canonical job (skip-already-pinned
  read, estimate_scope active-server read) now carry `# noqa: S110 — <reason>`,
  matching the repo's existing convention for intentional swallow-and-continue.

ruff check passes on all canonical files + tests; 30 affected tests green.
2026-06-02 15:42:14 -07:00
BoulderBadgeDad
dfa5204e0a Repair settings: dropdown for fixed-choice settings (canonical source_selection)
The canonical source_selection setting was rendering as a free-text box — easy
to typo an invalid mode. Added a generic choice mechanism so it's a dropdown:

- RepairJob.setting_options: {key: [allowed values]} (default {} — opt-in).
- CanonicalVersionResolveJob declares source_selection's three modes.
- repair_worker.get_all_job_info() includes setting_options in the job payload.
- enrichment.js renders a <select> (options prettified, current value selected)
  for any key listed in setting_options; everything else renders by value type
  as before. The save path already reads <select>.value as a string, so no
  change needed there.

Generic — any future job can get dropdowns the same way. Jobs that don't
declare setting_options are untouched (empty dict -> existing input rendering).

Tests: source_selection exposes the 3 options and its default is one of them.
23 repair-job/worker + canonical tests pass (other jobs unaffected).
2026-06-02 15:33:13 -07:00
BoulderBadgeDad
2fcdfd3145 Canonical findings: include as much (free) data as possible
Per request, pack each finding with everything available WITHOUT extra API
calls (kettui: reuse what's already fetched, read the album row we already
loaded, degrade per-field, keep it tested):

- Pinned release's track titles — already fetched during scoring, so free
  (capped at 60 to bound details_json).
- From the album row (free): year, DB track count, total duration, genres-free
  context, and the album's currently-linked source IDs.
- file_track_titles (your library's titles) for a side-by-side with the release.
- Artist + album thumbs (artist via the guarded lookup) and names.

_describe_pin now renders: "Artist — Album (year)", the fit breakdown, "Currently
linked: … → pinning X", "Beat: <alternatives>", and the release tracklist — so
the card is judge-able at a glance, and the structured fields are in details for
a richer UI.

NOT included (would cost an extra per-album API fetch, left as opt-in): the
*release's* own year/type/cover/URL from get_album_for_source, vs the library's.

Tests: _describe_pin rich-render (year/linked/tracklist), resolver release-titles,
orchestration free-context fields. 94 canonical + reorganize regression pass.
2026-06-02 14:10:02 -07:00
BoulderBadgeDad
03d099fb1d Canonical findings: add artist image (guarded, schema-safe)
Findings now carry artist_thumb_url alongside album_thumb_url (same key the
track-repair findings use, so the findings UI already renders it).

Fetched via a guarded _lookup_artist_thumb() — checks the artists table has a
thumb_url column first and swallows any error — rather than adding ar.thumb_url
to the shared load_album_and_tracks SELECT. The shared-loader approach was
tried first and REVERTED: it crashed reorganize on schemas whose artists table
has no thumb_url column (caught by 40 orchestrator tests). The lookup only runs
for albums that actually resolve, so it adds no cost to the no-source-id
short-circuit majority.

Tests: orchestration test asserts artist_name + album_thumb_url + artist_thumb_url
flow through. 47 canonical + 104 canonical/reorganize regression tests pass.
2026-06-02 14:04:09 -07:00
BoulderBadgeDad
ec8091caad Canonical: richer, judge-able findings (the why behind a pin)
Live-run feedback: "Best-fit release: deezer (665666731), score 1.0" is too thin
to trust/accept. Each finding now explains WHY:

- score_release_detail() exposes the per-signal breakdown (count/duration/title)
  instead of just the blended score.
- resolve_canonical_for_album returns an enriched result: the breakdown,
  file_track_count vs release_track_count, and a `candidates` list of every
  source it scored (so a finding can show what the winner beat).
- resolve_and_store adds album/artist/thumb context from the row it already
  loaded (no extra query). Storage still only reads source/album_id/score.
- The job builds a real description via _describe_pin(), e.g.:
    "Pin deezer release 665666731 (confidence 100%).
     Fit to your library: 11 files vs 11 tracks on this release — track count
     100%, durations 100%, titles 100%.
     Beat: spotify 65% (17 tk)."
  and a clearer title ("Pin deezer as canonical: <artist> — <album>").

Tests: resolver enrichment (breakdown + candidate comparison fields), and
_describe_pin (judge-able text incl. the beaten alternatives, and honest "n/a"
for a missing signal). 42 canonical tests pass.

Note: the description string carries the judge-able info regardless of UI; how
the findings tab renders the extra details keys (thumb image, candidates table)
is still UI-dependent and unverified.
2026-06-02 13:13:37 -07:00
BoulderBadgeDad
57e039e34d Canonical: make source selection a job setting (default active-preferred)
Feedback from the live dry-run: the job was pinning whichever source best fit
the files regardless of which source it was, which was surprising — users
expect it to respect their active metadata source. Made it a per-job setting
instead of a baked-in policy.

source_selection (default 'active_preferred'):
- active_preferred — use the active/primary metadata source's release when the
  album has an ID for it AND it clears the score floor; otherwise fall back to
  the best-fit among the other sources. Respects the configured source but
  self-heals when that link is clearly broken (below floor / no ID).
- active_only — only ever the active source; never considers others.
- best_fit — previous behavior: whichever source matches the files best.

resolve_canonical_for_album gains mode + primary_source; the orchestration
threads the primary source through; the job reads source_selection from its
settings. Note: active_preferred respects the active source as long as it clears
the floor, so it will NOT override a deluxe-vs-standard mismatch on the primary
(#767-Bug2) — that's what best_fit is for; the choice is now the user's.

Tests: per-mode coverage in test_canonical_resolver.py (active_preferred uses
primary when it fits, falls back when primary is below floor, keeps primary even
when another fits better; active_only pins primary / never falls back; best_fit
unchanged), orchestration default-mode test, and the setting default. 39
canonical tests pass.
2026-06-02 12:58:59 -07:00
BoulderBadgeDad
e40b328a94 docs: canonical-album-version design spec
The staged design doc for this branch (#765 + #767-Bug2): the
match-your-files canonical rule, the additive/dormant rollout, and the
stage-by-stage plan the 6 implementation commits followed. Kept on the branch
as its reference; not relevant to dev/main.
2026-06-02 12:41:36 -07:00
BoulderBadgeDad
f9271c0cd8 Canonical album version — backfill job (the opt-in activation)
The populate trigger that turns the (until now dormant) feature on. Until a user
enables and runs this job, no album has a canonical -> both read sides (Stages
3-4) fall back -> zero behavior change. So the whole feature ships safely off.

- core/repair_jobs/canonical_version_resolve.py — "Resolve Canonical Album
  Versions". Iterates the active server's albums, skips ones already pinned, and
  calls the tested resolve_and_store_canonical_for_album per album. Opt-in
  (default_enabled=False) and dry-run-by-default: resolving compares an album's
  candidate releases across sources (metadata-source API calls, once per album),
  so it's deliberately user-triggered. Dry run reports a finding per album it
  would pin; live mode stores. Registered in _JOB_MODULES.
- core/metadata/canonical_resolver.py — resolve_and_store gains store=True; the
  job's dry run passes store=False to resolve-without-writing.

Tests: tests/test_canonical_version_job.py (6) — registered, opt-in + dry-run
defaults, live resolves+stores (auto_fixed), dry run creates findings without
persisting, already-pinned albums skipped. Registry loads all 19 jobs cleanly.
145 tests across the full feature + reorganize/track-repair/DB regression pass.
2026-06-02 11:53:45 -07:00
BoulderBadgeDad
f5752e3dc0 Canonical album version — Stage 4: Track Number Repair prefers canonical (read)
_resolve_album_tracklist gains a Fallback -1: if the album has a pinned
canonical (source, album_id), use it before the existing 6-level cascade — so
Track Number Repair resolves the SAME release the Reorganizer does (Stage 3) and
the two stop contradicting each other (#765, the Spotify-4 vs MusicBrainz-3
conflict).

Gated + additive: the entire existing cascade is untouched for albums without a
canonical, so this job's all-01-album rescue (which relies on the MusicBrainz/
AudioDB fallbacks for albums with no DB source ID) is fully preserved — that's
the regression we explicitly refused to take in a reactive fix.

New helper _lookup_canonical_from_db() mirrors _lookup_album_ids_from_db
(file-path -> track -> album), returns None when no DB / no match / columns
absent / unresolved.

Tests: tests/test_track_repair_canonical.py (4) — returns canonical when pinned,
None when unresolved / file untracked / no DB. Existing track_number_repair
tests still pass (no regression).
2026-06-02 11:47:42 -07:00
BoulderBadgeDad
ecdfde03c6 Canonical album version — Stage 3: Reorganizer prefers pinned canonical (read)
_resolve_source now prefers the album's pinned canonical (source, album_id) when
set, before the source-priority walk. So once an album's canonical is resolved,
reorganize agrees with Track Number Repair (Stage 4) and stops mislabelling a
standard album as deluxe (#767-Bug2).

Gated + side-effect-free: only changes behavior for albums that already carry a
canonical (none do until the populate step runs), an explicit user source pick
(strict_source) still wins over the canonical, and a failed canonical fetch
falls through to today's priority walk. So this stage is behavior-neutral until
canonical is populated.

Tests: tests/test_reorganize_canonical_source.py (4) — canonical preferred over
priority, fetch-failure falls back, strict_source ignores canonical, no-canonical
unchanged. 113 reorganize-orchestrator/tag-source/unknown-artist tests still pass
(no regression).
2026-06-02 11:45:31 -07:00
BoulderBadgeDad
43878b4d3d Canonical album version — Stage 2 (trigger): resolve+store orchestration
Completes Stage 2's populate path. Still dormant — no consumer calls it yet.

- resolve_and_store_canonical_for_album(db, album_id, ...): loads the album's
  source IDs + its tracks' (duration_ms, title) from the DB via the SAME
  loader the Reorganizer uses (load_album_and_tracks + _extract_source_ids), so
  the canonical is chosen over exactly the source IDs the reorganizer sees;
  scores off the DB track rows (the library's view of the files — no per-file
  disk reads), resolves the best fit, and persists it. Returns the stored result
  or None when unresolved.
- default_fetch_tracklist(): production fetcher wrapping
  get_album_tracks_for_source, normalising to {title, track_number, duration_ms}
  (duration best-effort; sec->ms; absent -> scorer leans on count+title).

Design note: chose LAZY resolution (Stages 3-4 consumers call this when they hit
an album with no canonical) over a standalone backfill repair job — no new
scheduling/UI surface, resolves only when a tool actually needs it, and stays
gated (NULL canonical = today's behavior).

Tests: tests/test_canonical_orchestration.py (5) — end-to-end on a real temp DB
(11 files pick the 11-track release over a 17-track deluxe and persist it),
no-source-ids -> None, missing-album -> None, and default_fetch_tracklist
normalization (dict items, seconds->ms) + failure -> None. All canonical +
DB-migration tests green.
2026-06-02 11:42:20 -07:00
BoulderBadgeDad
f37bc34082 Canonical album version — Stage 2 (core): resolver + persistence (dormant)
Turns the Stage-1 scorer into an end-to-end resolver + persists the result.
Still DORMANT — no consumer reads it yet, so zero behavior change.

- core/metadata/canonical_resolver.py — resolve_canonical_for_album(): builds
  candidate releases from the album's per-source IDs (in source-priority order),
  fetches each tracklist via an INJECTED fetch_tracklist (so it's unit-testable
  without live APIs), scores them with pick_canonical_release, and returns the
  best-fit {source, album_id, score}. Skips sources with no id / failed fetch;
  returns None when there are no files, no candidates, or nothing clears the
  confidence floor.
- database/music_database.py — set_album_canonical() / get_album_canonical()
  write/read the Stage-1 columns. get returns None when unresolved, which every
  consumer will treat as "fall back to today's behavior".

Tests: tests/test_canonical_resolver.py (7) — best-fit beats priority, priority
breaks true ties, skips missing-id/failed-fetch sources, None on
no-candidates/no-files/below-floor, score rounding. tests/test_canonical_db.py
(4) — set/get round-trip incl. timestamp, unresolved -> None, overwrite,
missing-album -> False. 34 canonical + DB-migration tests pass.

Remaining for Stage 2 (the trigger): read on-disk file durations/titles for an
album, gather its source IDs, call the resolver, store — wired via a backfill
repair job + an enrichment hook. Then Stages 3-4 wire the Reorganizer and Track
Number Repair to READ the pinned canonical.
2026-06-02 11:36:19 -07:00
BoulderBadgeDad
818c4f0bff Canonical album version — Stage 1: schema + pure scorer (dormant)
First stage of the canonical-album-version fix (#765 + #767-Bug2). Pins ONE
canonical (source, album_id) per album, chosen by best-fit to the user's actual
files, so the Reorganizer, Track Number Repair, and tagging stop re-resolving
independently and contradicting each other.

Ships DORMANT — nothing reads or writes the new data yet, so zero behavior
change. Later stages populate (Stage 2) and consume (Stages 3-4) it.

- core/metadata/canonical_version.py — pure scorer (the testable heart):
  score_release_against_files() rates a candidate release by track-count fit +
  duration alignment (greedy nearest within ±3s) + title overlap, dropping and
  renormalizing missing signals so it never crashes on sparse metadata.
  pick_canonical_release() takes candidates in source-priority order, picks the
  best fit, breaks ties toward the earlier (higher-priority) candidate so the
  choice is DETERMINISTIC — that determinism is what makes every tool agree
  (#765), while count/duration fit picks the right EDITION (#767-Bug2). A
  confidence floor (default 0.5) means a low-confidence guess is never pinned.

- database/music_database.py — additive, nullable columns on albums
  (canonical_source / canonical_album_id / canonical_score /
  canonical_resolved_at), guarded by the existing PRAGMA-table_info pattern.
  NULL = unresolved = every consumer falls back to today's behavior.

Tests: tests/test_canonical_version.py (11) — edition discrimination (11 files
-> standard, 17 -> deluxe), deterministic priority tiebreak, duration
disambiguation on count ties, graceful degradation (no durations / counts only /
fuzzy titles), confidence floor, empty-input safety. tests/test_canonical_
columns_migration.py (4) — fresh DB has the columns, they're nullable w/ NULL
default, migration is idempotent, and it ALTERs them onto an old albums table.
60 DB/schema regression tests still pass.
2026-06-02 11:30:58 -07:00
BoulderBadgeDad
cd9e4abc7c #766 follow-on: source rows borrow their matched server track's cover
A source row with no art of its own (e.g. a YouTube source, which provides
none at mirror time) now borrows the cover from its MATCHED server track, so
both sides of the sync editor show an image.

The endpoint already had a borrow fallback (_server_art_map), but it matched by
an exact normalized "{artist}|{title}" key — so a YouTube-shaped row like
"Arctic Monkeys - Do I Wanna Know?" never matched the library's "Do I Wanna
Know?" and stayed blank even though the server had the cover. This borrow is
keyed off the ACTUAL source<->server pairing the reconcile already computed, so
it works for those rows once #768's canonical matching pairs them.

Done in the pure reconcile_playlist (final pass), so no frontend change is
needed — the editor already renders source_track.image_url. Guarded so it only
fills an EMPTY source image (Spotify/CDN art is never overwritten) and only when
the matched server track actually has a thumb.

Composes with the rest: #766 made the server cover URL work, #768 made the
YouTube row match, this makes the matched source row borrow that cover — so an
artless YouTube row matched to a Navidrome track with art shows on both sides.

Tests: tests/test_playlist_reconcile.py (+4) — artless source borrows the
matched cover; source with its own art keeps it; unmatched source has nothing to
borrow; borrow skipped when the server track has no thumb. 15 reconcile + 59
sync/navidrome tests pass.
2026-06-02 11:08:35 -07:00
BoulderBadgeDad
89b438974f Fix #766: Navidrome album covers blank in the sync editor (+ other modals)
The sync editor renders server covers as <img src="/api/navidrome/cover/{id}">,
but no Flask route ever served that path — so every Navidrome cover 404'd, on
every album, art or not. The source (left) side then went blank too: a source
row with no native art (e.g. YouTube, which provides none at mirror time) falls
back to borrowing the matched server track's cover — i.e. that same dead route.
So both sides collapsed to nothing.

Fix:
- New NavidromeClient.build_cover_art_url(cover_id) — builds the absolute,
  Subsonic-authenticated getCoverArt URL (base_url + token/salt), keeping
  credentials server-side. Uses a FIXED cover-art salt so the URL is
  deterministic for a given (server, password, cover_id): a rotating salt (as
  in _generate_auth_params) would make every request a unique URL → image-cache
  miss every time + a dead, never-reused cache row per fetch. Token auth doesn't
  require a unique salt, and the password is never exposed (only its salted md5).
- New route /api/navidrome/cover/<cover_id> — resolves that URL and streams the
  image through the shared image cache (same pattern as /api/image-proxy), with
  a private max-age so the browser caches by the stable route URL.

Effect: server side works for any album that has art in Navidrome; matched
source rows with no native art now borrow the (now-working) server cover.
Unmatched YouTube rows stay blank — no image exists anywhere to show.

Tests: tests/test_navidrome_cover_url.py (8) — URL structure + salted-token auth
(never the raw password), determinism (same id -> same URL so the cache hits;
different id/password -> different URL), optional size, and the not-connected /
no-id / no-credentials guards.

Caveats: not executed against a live Navidrome (no server in CI) — the URL
builder is unit-tested; the route's cache→HTTP→bytes round-trip is read-verified
only. Scope is the sync editor's Navidrome route; Plex/Jellyfin server-cover
branches and any other modals using a different mechanism are untouched.
2026-06-02 11:01:28 -07:00
BoulderBadgeDad
3b49ac8280 Fix #767: Library Organizer dry run no longer creates folders
The reorganize preview (dry run) was physically creating destination album
folders, littering the library with empty dirs and making "changes" before the
user ever hit Apply.

Cause: preview_album_reorganize calls build_final_path_for_track purely to
COMPUTE the destination path string — but that shared helper has 9 os.makedirs
side effects (it's also the live download/import path builder, where creating
the dir is correct). So computing the preview path created "Lenka (Expanded
Edition)/" on disk.

Fix: build_final_path_for_track gains create_dirs=True; all 9 makedirs now route
through a gated helper. The reorganize PREVIEW passes create_dirs=False, so a
dry run computes the exact destination path with zero filesystem side effects.
Everything else keeps the default True:
- the download/import post-process flow (still writes files into the dir),
- retag,
- the reorganize APPLY path — verified it goes through post_process_fn (the real
  pipeline → build_final_path_for_track with create_dirs=True), so live moves
  still create their destination dirs. The gate only silences the dry run.

Tests: tests/imports/test_import_paths.py — create_dirs=False computes the
correct path (matching the reported "01 - The Show.flac") but writes NOTHING to
disk (not even the Transfer root); create_dirs=True still creates folders; both
yield an identical path. Updated two reorganize-orchestrator test doubles to
accept the new kwarg. 148 reorganize/paths/retag/pipeline tests pass.

Does NOT fix the second half of #767 (Expanded Edition picked over the standard
album). That is NOT a reorganizer bug: the library album row was linked to the
deluxe release at enrichment time (its stored spotify_album_id/itunes_album_id/
deezer_id points at "Lenka (Expanded Edition)"), and the reorganizer faithfully
reorganizes to whatever the album is linked to. The real fix is in album
enrichment's edition preference — tracked separately.
2026-06-02 10:32:06 -07:00
BoulderBadgeDad
bba0836324 Fix #768: playlist sync editor refusing to match certain tracks
Three compounding bugs hit tracks whose source metadata is YouTube/streaming-
shaped — title "Artist - Song", artist "Official Artist"/"Artist - Topic"/
"ArtistVEVO" (reported: "Arctic Monkeys - Do I Wanna Know?" by "Official Arctic
Monkeys"). Server-agnostic — affects Plex/Jellyfin/Navidrome, not just the
reporter's Navidrome.

Bug A — the match fails. The confidence scorer and the editor's reconcile both
compared the raw "Artist - Song" title against the library's clean "Song"; the
length-ratio penalty + floor drove it to ~0.18 (NO-MATCH), so the track showed
unmatched while its server copy showed as an orphan "extra". New pure
core/text/source_title.py (clean_source_artist / strip_artist_prefix /
canonical_source_track) strips the channel/video decoration, applied at BOTH
matching seams: services/sync_service._find_track_in_media_server (tries raw
then canonical, keeps the best) and the editor reconcile. Conservative: a title
prefix is stripped only when it equals the artist, so "Self-Titled", "Jay-Z",
and "Marvin Gaye" (by another artist) are untouched, and the canonical form is
an additional best-of candidate so it can only help.

Bug B — manual matches never persisted. get_server_playlist_tracks built the
per-source entry WITHOUT source_track_id, so "Find & add" posted an empty id
and _persist_find_and_add_match returned early. The match reverted to "extra"
on reload and re-adding looped. The editor's 3-pass matcher is now lifted to a
pure, tested core.sync.playlist_reconcile.reconcile_playlist that includes
source_track_id (the frontend at pages-extra.js:1836 already reads + sends it).

Bug C — manual match duplicated + delete wiped all copies. "Find & add" always
inserted, so linking a source to an already-present server track appended a
duplicate (pos 72, 73...); remove filtered out EVERY entry with the target id.
New pure core.sync.playlist_edit (plan_playlist_add: link-don't-duplicate when
the target is already present; remove_one_occurrence: drop a single copy) wired
into the Plex/Jellyfin/Navidrome add + remove branches.

Tests (extreme): tests/test_source_title.py (35), tests/test_playlist_reconcile.py
(11 — incl. the reported case, parity for override/exact/fuzzy/extra, and
duplicate-server handling), tests/test_playlist_edit.py (12). 286 matching/sync
tests still pass.

Caveats: the sync_service change and the add/remove/editor endpoints are
read-verified, not executed against a live media server (none in CI). The pure
cores they call are exhaustively unit-tested; output-shape parity of the
reconcile lift is covered. Delete removes the first matching copy (duplicates
are identical, so harmless).
2026-06-02 10:16:21 -07:00
BoulderBadgeDad
174513d351 Fix #769: playlist sync matched wrong same-artist track with high confidence
Tracks NOT in the library were matched to a DIFFERENT song by the SAME artist
and reported with high confidence instead of as missing — e.g. "Dani
California" -> "Californication" (Red Hot Chili Peppers), "Under The Bridge"
-> "Around the World".

Root cause: _calculate_track_confidence scores 0.5*title + 0.5*artist. A
same-artist comparison always yields artist = 1.0, so the title score is the
only thing that can tell two of an artist's songs apart — but that score is a
SequenceMatcher CHARACTER ratio, which over-credits unrelated titles that
share a long substring ("californi…" = 0.67) or just a stopword ("the" =
0.62). With the flat 0.5 artist term, anything clearing the weak 0.6 char
floor lands at ~0.81-0.83, well over the 0.7 sync threshold. Reproduced on
dev: both reported pairs score 0.81/0.83.

Fix: new core/text/title_match.py:titles_plausibly_same, called in
_calculate_track_confidence right before the floor. It accepts a pair only
when it's near-identical char-wise (>=0.85, so typos / punctuation / casing
like "Beleive"->"Believe", "HUMBLE."->"Humble" still match) OR the titles
share at least one significant (non-stopword) word. Two different songs by the
same artist share no content word, so they're rejected and the real track is
correctly reported missing. ("the" is a stopword — that's what leaked "Under
The Bridge"/"Around the World".)

Scoped deliberately: the word-overlap test fires ONLY when at least one side
has 2+ content words. For single-word titles there is no other word to share,
so it defers to the existing char floor — otherwise legitimate stylized
spellings ("Grey"/"Gray", "Tonite"/"Tonight", "4ever"/"Forever") would become
new false-negatives. Verified those still match. The few single-word variants
that do score low (Ok/Okay, Thru/Through) were already rejected by the
pre-existing length-ratio penalty, not by this gate.

Both reported false positives now score 0.33/0.31 -> missing. Does NOT address
the harder case of two different same-artist songs that DO share a content
word (e.g. "Believe"/"Believer") — pre-existing and unworsened. Any residual
error fails safe: a false-missing is re-downloaded/wishlisted, vs the old
behavior which silently substituted the wrong song.

Tests: tests/test_title_match_guard.py (14) — pure-guard unit tests + a
13-pair battery driving the REAL _calculate_track_confidence (genuine matches
stay >=0.7, same-artist different songs drop below), plus an explicit
no-regression test for stylized single-word spellings. 292 matching/sync tests
pass.
2026-06-02 09:14:26 -07:00
BoulderBadgeDad
3c15041b88 Fix #764: manual import reported quarantined files as a successful "Done"
The manual-import routes (album + singles) call post_process_matched_download
directly. When the pipeline quarantines a file — integrity / AcoustID / FLAC
bit-depth — or hits the race guard, it sets a context flag and RETURNS
NORMALLY (it only marks the task failed + notifies when there's a task_id,
which manual imports don't have). So the inner pipeline raised no exception,
and routes.py counted `processed += 1` for a file that had just been moved to
ss_quarantine, not the library. Result: the UI shows a green "Done" while the
track silently vanished — exactly the #764 report (Coldplay - Yellow.flac ->
ss_quarantine, but "Done").

The download path already handles this in
post_process_matched_download_with_verification (it reads the same flags and
marks the task failed); only the manual-import routes were missing the check.

Fix: new pure helper import_rejection_reason(context) returns a human-readable
reason for any terminal rejection (_integrity_failure_msg / _acoustid_quarantined
/ _bitdepth_rejected / _race_guard_failed) or None for a clean import. Both
manual-import routes now consult it: album_process reports the track in
`errors` instead of counting it processed; process_single_import_file returns
("error", reason) instead of ("ok", ...). Verified every move_to_quarantine
call site (4, all in pipeline.py) sets one of those flags, so no quarantine
path slips through. This also delivers the "direct display of the error" the
reporter asked for — the reason now surfaces in the response `errors` list.

Does NOT address the reverse symptom ("failed even though it moved correctly")
— not yet root-caused — nor the separate bit-depth hole on the download-path
wrapper.

Tests: tests/imports/test_import_rejection_reason.py (10) — each trigger
detected, falsy flags ignored, deterministic ordering, plus two route-level
tests driving the REAL process_single_import_file (quarantine -> "error";
clean -> "ok").
2026-06-02 08:40:26 -07:00
BoulderBadgeDad
3dfec8a157 Fix #764: import no longer destroys embedded cover art
enhance_file_metadata rebuilds tags from scratch: for FLAC it calls
clear_pictures(), for MP3/MP4 it clears the whole tag block — and it does
this UP FRONT, then saves the file, long before it tries to fetch and embed
the replacement art. So every way the re-embed could come up empty left the
file saved with the original art destroyed and nothing put back:

  - extract_source_metadata returns nothing  -> early save, no embed
  - no album-art URL / art download fails / rejected by the min-size guard
    -> embed_album_art_metadata returns early without adding a picture
  - art embedding disabled in config         -> embed skipped entirely
  - embed raises mid-enrichment               -> file left cleared on disk

This is the "cover art gets corrupted/destroyed during import" half of #764
(continuation of #755); distinct from #750's truncated-cache DISPLAY bug.

Fix: new core/metadata/art_preservation.py snapshots the existing art
(the live Picture / APIC / MP4Cover objects, so they re-apply verbatim)
BEFORE the clear, and restores it before each save IFF the file currently
has none. Wired into all three exit paths in enhance_file_metadata
(no-metadata early return, the final save, and the except handler). The
restore is a strict no-op when art is already present, so the happy path —
new art embedded — is byte-for-byte unchanged: it never clobbers or
duplicates a freshly-embedded cover. embed_album_art_metadata now returns a
bool so the intent (embedded / didn't) is explicit.

Tests:
- tests/test_art_preservation.py (5) — snapshot/restore round-trips through
  real mutagen FLAC + ID3 objects; restore no-ops when new art is present.
- tests/test_enrichment_art_preservation.py (4) — runs the REAL
  enhance_file_metadata over a real FLAC with embedded art and asserts the
  art survives on disk for missing-metadata / failed-embed / embed-raises,
  and is correctly REPLACED (exactly one picture, new bytes) on success.
1019 tests pass across the metadata/enrichment/imports/acoustid suites.
2026-06-02 08:40:05 -07:00
BoulderBadgeDad
de20897f83 Fix: deep-scan / DB-update automation falsely errors on large libraries (stall-based timeout)
The DB-update + deep-scan automation monitor used a hard 2-hour TOTAL cap
(while elapsed < 7200). It tracked progress but only used it to print a stall
warning — the only thing that actually timed out was wall-clock. So a large
library that scans for >2h while progressing fine (reported: 4781 artists) trips
the cap and the automation card flips to 'error: timed out after 2 hours' even
though the scan thread is healthy and still running (the timeout never cancels
it, which is why it keeps progressing in the logs after the 'error').

Time out on STALL, not total runtime:
- 30 min with NO progress -> error ('stalled'); catches a genuinely hung scan.
- 10 min idle -> warning (repeats); unchanged heads-up.
- 24h absolute backstop, purely a runaway-loop guard.
- An actively-progressing scan keeps resetting the idle clock, so it never
  times out no matter how many hours the whole library takes.
- Progress is judged on (processed, progress, current_item) so a slow stretch
  where the rounded % holds steady (but the artist keeps changing) isn't a
  false stall.

The decision is extracted into a pure, testable scan_wait_action(); both the
deep-scan and full-refresh handlers share the monitor loop, so both are fixed.

Tests: tests/test_scan_wait_action.py (9) — headline regression (5h/12h total
but progressing -> 'continue', not timeout), finished/stall-warn/stall-timeout/
abs-cap thresholds, and ordering. 280 automation tests still pass.
2026-06-01 16:30:28 -07:00
BoulderBadgeDad
c8c3789cb9 Album bundle: fall back to per-track on an I/O error, don't hard-fail the batch
Defense-in-depth follow-up to #760. Even with the entrypoint chown fix, if the
album-bundle staging dir ever can't be created/written (permissions, read-only
mount, disk full), the dispatch caught the plugin exception and marked the whole
batch failed — even though the album had already downloaded (the #715 symptom:
'release finishes downloading but the batch fails').

Now an OSError from the plugin is flagged fallback-eligible, so the dispatch
returns to the per-track flow instead of hard-failing. OSError covers the
staging/filesystem failure that motivated this (#760's PermissionError) and, by
Python's IOError==OSError aliasing, any propagated transient I/O error —
falling back is never worse than hard-failing, and per-track is the universal
graceful path. Programming errors (TypeError, KeyError, RuntimeError, …) are
NOT OSError and stay terminal, so genuine bugs still fail loudly — the existing
'plugin exception => failure' contract and its test are preserved.

Test: new test_dispatch_staging_oserror_falls_back_to_per_track (PermissionError
on the staging dir -> result False, phase 'analysis', not failed). Existing
RuntimeError-is-terminal test still passes. 131 album-bundle/plugin tests green.
2026-06-01 13:29:05 -07:00
BoulderBadgeDad
aabf1c0e6a Fix #760: chown /app/storage to PUID on every start (album-bundle staging EACCES)
The album-bundle staging area /app/storage is baked into the image owned by the
build-time soulsync UID. The entrypoint only re-chowned it to the runtime PUID
inside the GATED recursive chown (entrypoint.sh:43), which is skipped whenever
/app/data is already owned correctly — and /app/storage was missing from the
UNCONDITIONAL per-start chown (line 85). So on installs whose PUID differs from
the build UID and whose /app/data is already correct, /app/storage kept its
build ownership and wasn't writable, and the Soulseek album-bundle flow died
with:

  PermissionError: [Errno 13] Permission denied: 'storage/album_bundle_staging'

(/app/Stream was added to the unconditional chown after this exact bug;
/app/storage slipped through.)

Add /app/storage — plus /app/MusicVideos and /app/scripts, which were also
missing — to the unconditional mkdir+chown (lines 84-85) and the writability
audit (line 92), matching the Dockerfile's pre-baked dir list. /app/storage is
now chowned to the runtime PUID on every start regardless of the gated
recursive chown. Verified with bash -n; all four dir lists are now consistent.
2026-06-01 13:19:26 -07:00
BoulderBadgeDad
cea0e365c2 Fix #759: Amazon enrichment floods when its public proxy is down
After an update, installs became unusable: the Amazon enrichment worker runs by
default, the default public T2Tunes proxy (t2tunes.site) was returning
503 'Amazon Music API is not initialized', and the worker treated every album
as an individual error -- logging an ERROR per item, churning network + DB
continuously across the whole library, and marking every row 'error' (a state
the retry tiers never re-attempt, so even after the proxy recovered nothing
re-enriched). The reporter couldn't reach the UI to turn it off.

Two-part fix:

1. Source-outage circuit breaker (core/amazon_outage.py, pure + tested):
   - is_source_outage(exc) distinguishes a whole-source outage (HTTP 5xx,
     'not initialized', connection failure, non-JSON error page) from a real
     per-item miss (404, transient 400, etc.).
   - On an outage the worker now leaves the item UNTOUCHED (so it's retried once
     the proxy recovers instead of being permanently burned to 'error'), logs
     ONCE per streak, and backs off with next_poll_delay_seconds() -- escalating
     30s -> 60s -> ... capped at 30 min -- instead of grinding every 2s. It
     auto-resumes the normal cadence the moment the source answers (success OR a
     non-outage error both clear the streak).
   - AmazonClientError now carries status_code so detection doesn't rely on
     message parsing.

2. Opt-in by default (web_server.py): amazon_enrichment_paused now defaults to
   True. Because enrichment depends on an external public proxy that can be
   down, it stays paused unless the user explicitly enables it -- a proxy outage
   can no longer take down installs that never opted in. (Behaviour change:
   anyone on the old auto-on default is now paused; re-enable in Settings.)

Together: on update the worker is paused -> no flood -> UI accessible; opted-in
users are protected from future outages by the breaker.

Tests: tests/test_amazon_outage.py (12) pin the classifier across every error
surface (incl. the exact 503 'not initialized' case) and the back-off schedule
(monotonic, capped). 157 Amazon tests pass; lint clean.

Note: could not reproduce the exact 'UI fully unreachable' mechanism remotely
(WAL + 8 gthreads shouldn't hard-lock); the fix removes the flood/churn that is
the practical cause and defaults the feature off.
2026-06-01 13:10:51 -07:00
BoulderBadgeDad
28850672a6 Fix: duplicate detector kept lossy over lossless (rank format first)
The Duplicate Detector's 'Keep Best' auto-selection ranked copies by highest
bitrate -> duration -> track number, with no notion of format. A FLAC whose
bitrate the library scan never populated (a common gap) therefore lost to a
282 kbps MP3: 282 > 0, so the MP3 was kept and the FLAC deleted (reported on
Havok 'Prepare For Attack', and again on Kendrick GNX).

Fix: rank by format/lossless tier FIRST, then bitrate, duration, track number.
A lossless file now always beats a lossy one regardless of the recorded
bitrate; bitrate/duration/track# only break ties within the same format.

- core/library/duplicate_keep.py (new): pure, importable pick_duplicate_to_keep
  + duplicate_keep_sort_key + format_rank_for_path (extension rank mirroring
  auto_import_worker._quality_rank: flac=10 ... mp3=5 ... unknown=1).
- core/repair_worker.py: _fix_duplicates auto-pick now calls
  pick_duplicate_to_keep instead of the bitrate-first max().
- webui/static/enrichment.js: the KEEP/REMOVE recommendation mirrors the same
  format-first ranking so the badge matches what the backend will delete.

Parity: Python uses '.ext' keys (os.path.splitext), JS uses 'ext'
(split('.').pop()) -> identical results; both keep the first copy on a full
tie. Verified the only other dedup path (the standalone Duplicate Cleaner
automation, core/library/duplicate_cleaner.py) was already format-priority-first
and correct -- no change needed there.

Tests: tests/test_duplicate_keep.py (11 -- incl. the exact FLAC-with-missing-
bitrate vs 282 kbps MP3 case, format ranking, within-format tie-breakers, and
edge cases). 147 repair/duplicate tests still pass.

Note: why FLAC bitrate is NULL in the DB is a separate library-scan gap;
format-first ranking makes the keep decision correct regardless.
2026-06-01 12:49:34 -07:00
BoulderBadgeDad
b202c176f7 Cover-art sources: skip low-res art (min-resolution guard) + max-res iTunes
Follow-up to the preferred-art feature. Real test runs showed a source could
win on priority while handing back a small cover: Cover Art Archive is
volunteer-uploaded with no size floor, so CAA-first gave a 599x531 (Taylor
Swift) and a 600x600 (Kendrick GNX) -- front-1200 only caps the max, so a
~600px upload stays ~600px -- and Deezer/iTunes lower in the order never got a
turn.

Fix:
- Minimum-resolution guard: artwork._min_size_art_validator builds the
  resolver's validate hook -- it fetches each candidate, caches the bytes (so
  the winner isn't fetched twice), and accepts art only when its shortest side
  >= metadata_enhancement.min_art_size (default 1000px; 0 disables). Art that's
  too small is a miss, so the resolver falls through to the next source instead
  of winning on priority. Unmeasurable images are accepted (don't over-reject;
  fallback is still today's art). Wired into both embed_album_art_metadata and
  download_cover_art.
- iTunes art upgraded to /3000x3000bb/ (was the 600px default) so it
  contributes high-res when it wins.
- select_preferred_art_url gains a validate passthrough to the resolver.
- config default metadata_enhancement.min_art_size: 1000.

Effect: with an order like caa > deezer > spotify > itunes, a ~600px CAA upload
is now skipped and Deezer's ~1900px wins -- consistent big art. (Spotify art
often maxes ~640px, so it's skipped at the 1000 floor in favor of bigger
sources; lower min_art_size to ~640 to allow it.)

Tests: tests/metadata/test_art_min_size.py (6 -- incl. the real 599x531 and
600x600 cases, shortest-side logic, unmeasurable-accept, no-bytes-reject,
0-disables) + iTunes max-res upgrade test. Full metadata suite green (617).
2026-06-01 12:24:51 -07:00