A 2.6.3 change (c3b88e69) split the wishlist albums cycle into one batch per
album. Each album batch runs an INLINE-BLOCKING soulseek/torrent/usenet
album-bundle download (album_bundle_dispatch.try_dispatch ->
download_album_to_staging) that holds its worker thread for the whole
search+download. All of these were submitted to the shared 3-worker
missing_download_executor -- the same pool used for per-track downloads AND the
manual 'Download Wishlist' analysis.
So a large Album-Completeness 'Fix all' (-> ~819 wishlist tracks -> ~82 per-album
batches) saturated all 3 workers with blocking album downloads; the manual
wishlist analysis could never get a thread ('Library Analysis' stuck on
Pending), the other ~79 batches sat in phase='queued' forever, and auto-cleanup
(which only evicts terminal-phase batches) never cleared them -> jam until
restart. Fixing batch STATUS would not help: the threads are blocked inside the
download, not waiting on a phase flip.
Fix: add a dedicated bounded album_bundle_executor (max_workers=3) and route the
AUTO per-album bundle batches to it, keeping the shared pool free for analysis /
per-track / the manual wishlist (which always starts now). Hung/slow album
downloads can only delay other album downloads, never the user-facing path.
Additive and decoupled; the submit site falls back to the shared pool when the
album pool isn't wired (older callers / tests) so behavior is unchanged there.
The manual path is untouched (it already runs album bundles serially on a single
thread, by design).
Tests (tests/wishlist/test_automation.py): album sub-batches route to the
dedicated pool while the residual per-track batch stays on the shared pool;
fallback-to-shared-pool when no album pool is wired. Existing auto-processing
tests still green (fallback preserves prior behavior). 707 passed across
wishlist + downloads suites.
The same provider ID is stored under inconsistent column names across tables
(deezer_id vs deezer_artist_id vs album_deezer_id vs similar_artist_deezer_id;
spotify/itunes keep an entity qualifier, others don't; musicbrainz uses three
nouns), so code checks 2-5 name variants everywhere.
Add core/source_ids.py as the single source of truth for (provider, entity) ->
column, with accessors that read an ID from a dict/sqlite3.Row robustly
(canonical column first, then known aliases). NO database columns are renamed —
these are the real names today; the registry just centralizes the knowledge.
Targeted adoption (behavior-identical, verified):
- core/artist_source_lookup.SOURCE_ID_FIELD now derives from the registry
instead of duplicating the mapping (values unchanged).
- web_server.py artist-detail builds artist_source_ids via source_id_map(...)
instead of a hand-rolled per-source .get() dict.
Broader call-site adoption deferred as clearly-scoped follow-up.
Tests: tests/test_source_ids_registry.py (canonical columns, alias fallback,
canonical-preferred, sqlite3.Row, source_id_map, SOURCE_ID_FIELD unchanged).
Existing artist_source_lookup + artist_full_detail suites still green.
Migration state was scattered across PRAGMA-table_info guards, sentinel marker
tables (_genius_search_fix_applied, ...) and metadata-flag rows
(id_columns_migrated, ...), with no single source of truth and no schema
version — so a half-migrated DB was undetectable.
Add a non-gating backstop: a schema_migrations(name, applied_at) ledger plus a
_sync_migration_ledger pass (runs last in init) that back-fills the ledger from
the existing signals and stamps PRAGMA user_version. ADDITIVE only — existing
migrations keep their own idempotency gates; nothing decides whether a
migration runs based on the ledger or the version. New one-time migrations call
_record_migration (the genres migration already does).
Tests: tests/test_db_migration_ledger.py — table exists, user_version stamped,
record idempotent, genres recorded on fresh init, backfill from flag + marker,
absent signals not recorded.
artists.genres / albums.genres stored EITHER a JSON array (new writes) OR a
legacy comma-separated string (old writes), forcing every reader to
try-JSON-then-split. Add a marker-gated one-time migration
(_normalize_genres_to_json) that rewrites legacy rows to JSON in place,
mirroring the readers' exact parse (JSON list, else comma-split/strip/
drop-empties) so genre VALUES are unchanged — only the storage format.
Per-row diffed (already-canonical rows untouched, no churn) and non-fatal on
error, consistent with the other migrations. Readers still tolerate both
formats, so this breaks nothing; it just removes the dual-format debt.
Tests: tests/test_db_genres_json_normalization.py — CSV->JSON, JSON-unchanged,
whitespace/empties dropped, albums table, legacy-reader-equivalence,
idempotent re-run, marker set on fresh init.
amazon_artist_id is added to watchlist_artists via ALTER (music_database.py
~1732), but both table-rebuild migrations — the spotify_id-nullable fix
(_fix_watchlist_spotify_id_nullable, two CREATE variants) and the
profile-scoped UNIQUE rebuild — recreated the table from a hardcoded column
list that omitted amazon_artist_id. Because shared_cols filters new_cols
against the old table, the column and any stored Amazon artist IDs were
silently dropped on every init (fresh OR upgraded), so Amazon watchlist IDs
never persisted at all.
Fix: add amazon_artist_id to all three rebuild CREATE schemas, both rebuild
new_cols lists, and the base CREATE TABLE (so fresh installs are consistent
and don't rely on the ALTER). Purely additive, column-named inserts + Row
factory mean column position is irrelevant.
Tests (tests/test_db_watchlist_amazon_id_migration.py): drive the real
migrations via MusicDatabase() against a seeded pre-migration temp DB and
assert the column + data survive; differential-proven to FAIL pre-fix.
Continue the design-system unification (kettui UI-consistency item):
migrate the five remaining compact button families onto the shared
.btn .btn--sm primitive + color modifiers, and drop their bespoke base
CSS (net -125 lines of CSS).
- ya-header-btn (Your Albums/Artists, discover.js-injected) -> .btn .btn--sm
.btn--secondary; ya-refresh/ya-settings/ya-viewall co-modifiers kept.
- explorer-action-btn (Playlist Explorer) -> .btn--secondary / .btn--primary.
- repair-bulk-btn -> .btn--secondary / .btn--primary / .btn--warning (fix-all).
- enhanced-bulk-btn (Library bulk bar, library.js-injected) -> .btn--primary/
--secondary/--danger; class kept as a hook for the mobile.css size
override + the .tag-write / .rg-analyze special colors.
- profile-create-btn (init.js-injected) -> .btn .btn--block .btn--primary;
class kept for the scoped .profile-edit-buttons flex:1 rule.
mini-nav-btn deliberately left as a distinct icon-button archetype.
Formalize the compact 'toolbar' button tier as design-system modifiers
(.btn--sm), plus a full-width (.btn--block) and amber caution
(.btn--warning) modifier, so the many smaller per-page buttons can share
the .btn primitive without being forced to the large default size.
First adopter: the Sync page header buttons (.sync-history-btn) now use
.btn .btn--sm .btn--secondary. The class is kept as a JS/onboarding
selector hook; .auto-sync-manager-btn still tints Auto-Sync accent.
The watchlist + wishlist header/overview buttons used a bespoke
.watchlist-action-btn family (different padding/radius/font and white
primary text) instead of the shared .btn design-system primitive.
Migrate all 11 of them to .btn / .btn--primary / .btn--secondary /
.btn--danger so they match the rest of the app, and drop the now-dead
CSS.
The .watchlist-batch-remove-btn / .wishlist-batch-remove-btn hook
classes are kept on the remove buttons (their !important red overrides
compose correctly over .btn--secondary). Static HTML only; no JS-injected
usages, and mobile.css overrides target .playlist-modal-btn, not these.
Reorder the sidebar nav so Downloads sits between Wishlist and
Automations. Mobile nav reuses the same .nav-button elements and the
helper/onboarding references are selector-keyed, so no other changes
are needed.
The Tools-page Database Updater dropdown only offered Incremental and
Full Refresh, even though the backend (/api/database/update with
deep_scan) and the dashboard Deep Scan button already supported a deep
scan. Wire the missing option into the Tools UI:
- Add a "Deep Scan" option to the #db-refresh-type dropdown.
- handleDbUpdateButtonClick now sends { deep_scan: true } for that
option (deep scan takes precedence server-side) and confirms first,
since deep scan removes stale entries — mirroring the dashboard flow.
Frontend-only; the progress/status handler already drives the bar from
the backend phase ("Deep scan: ...") and the help/docs copy already
described all three modes.
Search results live in an overlay dismissed by an outside-click handler
whose allow-list omitted the floating media player. Clicking the mini
player to open the now-playing modal (or clicking inside that modal)
registered as an outside click and tore the results down, forcing a
re-search.
Add the media player containers (#media-player mini bar and
#np-modal-overlay expanded modal) to the dismiss allow-lists in both the
Search page (search.js) and the global search widget (downloads.js),
which share the same outside-click pattern. Additive change: only adds
exceptions, so every existing dismiss case is unchanged.
Reported by CubeComming: importing media keeps the track artist correct
(e.g. Billie Eilish) but changes the album-artist tag ("Albuminterpret") to
"Unknown Artist", breaking grouping in the media server.
Cause: in extract_source_metadata (core/metadata/source.py), album_artist is
seeded from the resolved track artist, then overridden by the album CONTEXT's
first artist. When the album lookup comes back unresolved, that first artist is
the literal "Unknown Artist" placeholder — which is truthy, so it clobbered the
real artist.
Fix: treat "Unknown Artist" (and empty) as a non-value — only let the album
context override the album_artist when it names a real artist. A genuine album
artist (e.g. "Various Artists") still overrides as before.
Tests: tests/metadata/test_album_artist_unknown.py — placeholder doesn't
clobber, real album artist still used, no-album-context falls back to track
artist, empty doesn't clobber. (Pre-existing test_album_mbid_cache.py failures
are an unrelated sandbox DB disk-I/O issue.)
Reported by @Yug1900: a Spotify playlist's overview shows the correct count
(e.g. 203) but the sync modal lists only 100, and only 100 would sync.
Root cause in /api/spotify/playlist/<id> (the Spotify-tab track fetch): owned
playlists fetch + paginate fine (sp.next loops over all pages). But when the
authenticated playlist_items call FAILS (intermittently, often a 403 on
followed / not-owned playlists) it fell straight back to the public embed
scraper, which is hard-capped at ~100 tracks and has no pagination — and the
result was returned as if complete. The overview total is correct because it
comes from the playlist *metadata* listing, which succeeds independently.
Fixes (additive — the owned/working path is unchanged):
- Retry the items fetch once before resorting to the embed scraper, so a
transient failure no longer silently truncates a large playlist.
- Guard against silent truncation: compare fetched count against the
playlist's known total; if short (or via the capped embed fallback), log a
warning and return `incomplete: true` + `expected_total` instead of
presenting a partial list as complete.
No behavior change when the fetch succeeds in full (incomplete=false, extra
fields ignored by the frontend). Lets a future UI tweak surface
"got 100 of 203 — retry" rather than silently dropping tracks.
Low-risk tidy-up for the full-bleed "exception" pages that aren't carded.
Every page already gets a 40px gutter from .page, but the exception pages were
piling on inconsistent extra padding (library +20px, active-downloads +28/32px,
discover/docs +0) — giving accidental 60 / 68-72 / 40 effective gutters.
Drop the redundant container padding on library and active-downloads so the
single .page 40px gutter is the shared, intentional outer spacing across the
full-width exception pages. discover (centered max-width) and docs (sidebar
layout) keep their functional layout; library's mobile padding override is
unaffected.
Standardize the sync page's outer spacing to match the other pages. Like
settings, its .sync-header and .sync-content-area were siblings directly under
.page (no wrapper) — wrap both in a single .page-shell div so it becomes the
floating card with consistent margin/padding. HTML-only change.
Watch: .sync-content-area uses height:95% (grid) — fine against an auto-height
card, but to be confirmed visually (library's full-height grid was the one
that didn't fit a card).
Add the canonical .tab (bordered rounded-pill filter tab) and .card ("glass"
content card) primitives as the documented design-system standard for new
markup and the React pages — modeled on the cleanest existing looks
(watchlist filter pill; dashboard service/stat card).
Deliberately NOT migrating the existing tab/card components onto them: the
current implementations are visually divergent and JS-coupled (active-state
toggled by class name, cards built in JS), so a blind consolidation risks
subtle regressions. These primitives let new/React code be consistent now;
the legacy components migrate when visually verifiable / in React.
Unused classes -> zero visual change to the current UI.
Migrate the wishlist add-to-wishlist modal buttons onto the shared .btn
primitive: primary -> .btn--primary, secondary -> .btn--secondary, the green
download CTA -> new .btn--download modifier. Added a shared .btn.loading state
(amber pulse, reusing the existing pulse-loading keyframe) since
confirm-add-to-wishlist-btn toggles `loading` via JS (wishlist-tools.js).
Removed the dead .wishlist-modal-btn* rules and re-scoped the mobile
full-width override to `.wishlist-modal-actions .btn`.
Start of the button-consolidation pass (kettui's #1). The app had ~236 button
classes / ~8-10 distinct looks with heavy near-duplication.
Introduce a canonical .btn design-system primitive (base + .btn--primary /
.btn--secondary / .btn--danger), modeled on the dominant existing look
(accent-gradient primary, translucent ghost, semantic danger) and built on the
accent CSS vars. New markup and the React pages should use this; existing
per-page button classes will migrate onto it family by family.
First family migrated: the config/settings modal buttons (.config-modal-btn*,
4 static uses, no JS refs) -> .btn .btn--primary / .btn--secondary. Removed the
now-dead .config-modal-btn* rules and re-scoped its mobile full-width override
to `.config-modal-actions .btn`.
Visible change is minor by design (padding 28->24px, gradient direction
normalized). Proof step for sign-off on the .btn look before rolling wider.
Settings was the one flat page with no single wrapper — its .dashboard-header
and .settings-content sat as siblings directly under .page. Wrap both in a
single .page-shell div so the page becomes a floating card with the header
banner at the top, matching the dashboard structure. HTML-only change (no CSS:
.settings-content keeps its minor `0 4px` inner padding).
Library is intentionally NOT converted — its full-height artist grid + A-Z
jump rail overflow a margin:20px card, so it stays flat as a documented
exception (same category as search/discover/active-downloads).
Follow-up to the poll fix, covering the two things that blocked a
successful end-to-end album import once the poll itself stopped
freezing:
1. Staging dir permissions
The album-bundle private staging path defaults to
'storage/album_bundle_staging' -> /app/storage, but /app/storage was
never created or chowned by the image (unlike /app/Staging,
/app/Transfer, etc.), and /app is root-owned. The copy failed with
"[Errno 13] Permission denied: 'storage'" under the non-root soulsync
UID. Added /app/storage to the Dockerfile build-time mkdir+chown and
the entrypoint PUID/PGID chown, exactly like the sibling runtime dirs.
2. Client->local path resolution
Usenet/torrent clients report save paths from inside THEIR OWN
container (e.g. SAB '/data/downloads/music/<album>'); SoulSync often
mounts the same files at a different point ('/app/downloads/<album>').
Feeding the client path straight to the audio walker yields
"No audio files found" though the files are physically present.
New resolve_reported_save_path():
a. use the reported path as-is if readable (mirrored mounts),
b. apply explicit download_source.usenet_path_mappings
({from,to}, Sonarr/Radarr-style) for non-shared layouts,
c. basename fallback under SoulSync's own download roots —
zero-config for the standard shared-volume arr setup.
Wired into both call sites in usenet.py AND torrent.py
(download_album_to_staging + _finalize_download), logging any
translation and including the resolved path in the no-audio error.
Tests: resolver verbatim / explicit-mapping / basename-fallback /
priority / not-found / empty / mapping-miss-then-basename. ruff +
compileall + pytest green (645 in the download suites).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the playlist-explorer page from a flat padded container to the
.page-shell floating card. Drop its bespoke `padding: 24px 32px`; keep the
full-height flex layout (display:flex / column / min-height:100%) since the
explorer fills the viewport.
Visible change by design. Watch: the full-height min-height:100% inside a
margin:20px card may run slightly tall — to be confirmed visually.
First of the "flat -> card" conversions. The automations list view sat
directly on the page background (.automations-container = bare padding) while
its inner .dashboard-header is the same header dashboard uses. Adopt
.page-shell so the page becomes a floating gradient card structurally
identical to the dashboard (page-shell card > dashboard-header > content).
- Drop .automations-container's bespoke `padding: 20px 24px` (card padding now
from .page-shell); keep the class as the mobile/JS hook.
- Add `page-shell` to the container in markup.
Visible change by design (this page was not previously a card). Mobile keeps
its existing .automations-container padding override.
First step of the page-layout-shell standardization (kettui's UI-consistency
point #1). The dashboard, tools, watchlist and wishlist pages each defined a
byte-identical "card" container (padding 28px 24px 30px, margin 20px, gradient
bg, radius 24px, border + border-top, layered shadow) under four different
class names.
Extract that into a single `.page-shell` primitive (modeled on the canonical
dashboard/stats look) and have the four pages adopt it. Each keeps its bespoke
class for page-specific extras and as a JS/mobile hook:
- dashboard-container: keeps display:flex / column / gap:25px
- watchlist/wishlist-page-container: keep position:relative
- tools-page-container: no extras (box now fully from .page-shell)
Zero visual change: computed styles are identical (declarations relocated, not
altered), and mobile.css overrides still target the retained bespoke classes.
Per-page themed headers (watchlist amber, etc.) are intentionally NOT touched.
The class name is intended for reuse by the React pages too, so the primitive
is shared across both stacks.
Next (wave 2): migrate settings / automations / playlist-explorer / library
onto .page-shell, which snaps their slightly-off spacing to canonical.
The earlier #721 fix tolerated a ~10s "completed but no save_path"
window, but the real production stall sits upstream of that: SABnzbd
removes a finished download from the queue and runs par2 verify /
repair / unpack *in History*, exposing the live stage in the slot
`status` ('Verifying' / 'Repairing' / 'Extracting' / 'Moving' / ...)
with `storage` empty until the final move. `_parse_history_slot` mapped
EVERY non-'Failed' status to 'completed', so a still-extracting 1.7 GB
FLAC album looked "completed with no save_path" the instant download hit
100%. The poll burned its completed-no-path budget mid-PP and bailed,
freezing the UI on the last download emit (the stuck-at-99%/100%
signature). SAB then finished fine — which is why the job shows
Completed in History but SoulSync never staged it.
Root fix
- `_parse_history_slot` routes `status` through `_map_state`, so PP
stages stay NON-terminal: the poll keeps waiting (as 'downloading')
for as long as post-processing takes and only a real 'Completed'
flips to terminal success. `save_path` is trusted only on true
completion (mid-PP path fields may point at the incomplete dir).
Supporting / defensive
- `UsenetStatus.incomplete_path`: surfaced separately from save_path
(SAB `incomplete_path`) and used by the poll loops as a LAST RESORT
after the completed-no-path window, to recover the case where
`storage` never lands but the files are physically on disk.
- `poll_album_download`: dedicated, configurable completed-no-path
window (~120s via `download_source.album_bundle_completed_no_path_seconds`)
decoupled from the ~10s transient-miss window; incomplete_path
fallback; a 30s heartbeat log so the previously-silent poll loop is
diagnosable.
- `usenet.py` `_download_thread`: per-track parity — it was erroring
immediately on the first completed-no-path read.
- `album_bundle_dispatch.py` / `status.py` / `monitor.py`: use the
project `get_logger` so download-flow logs land in app.log under the
`soulsync.*` namespace (they were console-only before, which hid the
`[Album Bundle] flow failed` line during triage).
Tests
- PP-history state mapping; end-to-end Hunky Dory PP regression
(download -> Verifying/Extracting in History past both budgets ->
Completed+storage -> success); completed-no-path window +
incomplete_path fallback; per-track thread parity. ruff + compileall +
pytest all green (the only local failures are environmental: missing
tzdata + local tools/ffmpeg.exe, neither present on CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final cluster: the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles) ->
core.discovery.endpoints.save_bubble_snapshot(...), wired via
_save_source_bubble_snapshot. All four validate a payload key, persist via
db.save_bubble_snapshot(kind, items, profile_id=...), and return a count +
timestamp; they differ only by:
- payload_key ('downloads' for discover, 'bubbles' for the rest) + its
no_data_error message.
- snapshot_kind, success_noun, and the info/except log subject + noun
("downloads"/"artists"/"albums/tracks"/"charts").
get_database / get_current_profile_id injected; get_json (request.json) invoked
inside the try, preserving the original 400/500 behavior incl. traceback dump.
Tests: +5 (missing key 400, None body 400, happy path with kind/profile/count/
timestamp, discover_downloads variant, exception -> 500). Full discovery suite:
210 passed.
web_server.py: -98 lines.
Ninth cluster: update_<source>_playlist_phase for the five sources sharing the
identical validation + full-message response (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube) -> core.discovery.endpoints.update_playlist_phase(...),
wired via _update_source_playlist_phase + the _PHASE_LIST/_PHASE_LIST_YT
constants.
Per-source params:
- valid_phases — YouTube additionally allows 'parsed'.
- apply_extra_fields — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body; Tidal and
YouTube do NOT, so they pass False (kept strictly 1:1 — the generic won't
apply those keys for them even if a caller sent them).
- not_found_message / error_label; get_json invoked inside the try.
NOT folded in: iTunes-Link — uses data.get('phase') (no "Phase not provided"
400) and returns a no-message payload.
Tests: +7 (404, missing-phase 400, invalid 400, happy path with extra-fields
suppressed, extra-fields applied when enabled, YouTube 'parsed' allowed,
exception -> 500). Full discovery suite: 205 passed.
web_server.py: -123 lines.
Eighth cluster, the heavyweights (~110 lines each). The fix-modal
update_<source>_discovery_match for the four sources with the identical
structure (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.update_discovery_match(...), wired via
_update_source_discovery_match. Applies the user-selected Spotify track to the
discovery result (status/artist/album/duration/spotify_data/match-count) and
writes the manual fix to the discovery cache.
Per-source pieces are params:
- source_log_label / error_label.
- original_track_key ('tidal_track' / 'deezer_track' / ...).
- original_artist_getter: Tidal handles string-or-object artists
(first_artist_str_or_obj); the rest assume strings (first_artist_plain).
- web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) injected.
- get_json passed as a callable and invoked INSIDE the try, preserving the
original's "request.get_json() inside try" behavior (malformed body -> 500).
NOT folded in (genuinely divergent): iTunes-Link (saves spotify_data directly
via a different cache signature), YouTube (multi-key original_track fallback),
ListenBrainz (entirely different unmatch-capable structure, no cache write),
Beatport.
Tests: +9 (extractors; 400/404/400 guards; full happy path with result
mutation + duration formatting + match-count + cache-save args; no-increment
when already found; cache error swallowed; get_json raise -> 500). Full
discovery suite: 198 passed.
web_server.py: -400 lines.
Seventh cluster: start_<source>_sync for the five sources with the identical
flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube) ->
core.discovery.endpoints.start_sync(...), wired via _start_source_sync.
Validates phase, converts discovery results, seeds sync state, posts a
"... Sync Started" activity item, and submits to the sync executor. Per-source
pieces are params:
- sync_id_prefix (f"{prefix}_{key}"), not_found/not_ready messages, convert_fn.
- name/image accessors: Tidal reads an object (playlist_name_obj/
playlist_image_obj), the rest a dict (playlist_name_strict/playlist_image_dict).
- activity_label vs error_label DIFFER for Spotify-Public ("Spotify Link
Sync Started" activity, "Spotify Public" logs).
- submit_sync_task glue (_submit_sync_task) closes over sync_executor /
_run_sync_task / get_current_profile_id so the helper stays global-free.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task WITHOUT a playlist_image_url arg), Beatport (extra debug logging, chart).
Tests: +6 (404, not-ready 400, no-matches 400, full happy path with
state/sync-infra/submit/activity assertions, resync phases allowed,
exception -> 500). Full discovery suite: 189 passed.
web_server.py: -172 lines.
Sixth cluster: the bulk-hydration get_<source>_playlist_states endpoints for
the five sources that build the identical per-entry dict + {"states": [...]}
shape (Tidal, Deezer, Qobuz, Spotify-Public, iTunes-Link) ->
core.discovery.endpoints.get_playlist_states(states, *, error_label,
info_log_label=None), wired via _get_source_playlist_states.
iTunes-Link is the only one of the five without the "Returning N stored ..."
info log, so info_log_label is optional (iTunes passes None to suppress it).
NOT folded in: the YouTube/ListenBrainz get_all_*_playlists endpoints. They
return {"playlists": [...]} (different key) with a different field set
(url / created_at / playlist, no discovery_results) and filter out
mirrored_/profile-scoped entries — genuinely divergent, kept as-is.
Tests: +4 (list build + last_accessed bump + exact shape, empty, optional ids
default None, missing-required-field -> 500). Full discovery suite: 183 passed.
web_server.py: -116 lines.
Fifth cluster: reset_<source>_playlist for the four sources with byte-
identical bodies (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.reset_playlist(states, key, *, label,
not_found_message), wired via _reset_source_playlist. Resets phase/status to
'fresh', clears discovery/sync fields, cancels any discovery_future, and
preserves the original playlist payload.
Left with their own bodies (genuinely divergent):
- YouTube: status -> 'parsed' (not 'fresh'), no download_process_id, logs the
playlist name, "reset to fresh state".
- ListenBrainz: status -> 'cached', logs playlist title, returns
{"success": True, "phase": "fresh"} (different payload), _lb_state_key.
- iTunes-Link: state.update(...), no info log, "iTunes Link reset to fresh
phase".
Tests: +4 (404, full clear + playlist preserved + future cancelled, no-future
path, exception -> 500). Full discovery suite: 179 passed.
web_server.py: -100 lines.
Fourth cluster: get_<source>_discovery_status (all eight sources, Beatport
included) -> core.discovery.endpoints.get_discovery_status(states, key, *,
not_found_message, error_label), wired via _get_source_discovery_status.
Unlike sync-status, the discovery-status response shape is byte-identical
across every source (phase/status/progress/spotify_matches/spotify_total/
results/complete), so Beatport folds in here too. Only the 404 string
("... discovery not found" vs "... playlist not found" vs "Beatport chart
not found") and the except-log label vary. ListenBrainz key via _lb_state_key.
NOT touched this cluster: get_*_playlist_state (the sibling endpoints).
Those genuinely diverge per source — different id-key name (playlist_id /
url_hash / playlist_mbid), presence of url / created_at / download_process_id,
Tidal's playlist.__dict__ serialization, and YouTube's strict (non-.get)
field access. Folding them would need a flag pile that wouldn't be a clean
1:1, so they keep their own bodies.
Tests: +4 (404, full response + last_accessed bump, complete=False when not
'discovered', missing-field -> 500). Full discovery suite: 175 passed.
web_server.py: -155 lines.
Third cluster: the get_<source>_sync_status routes (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz) -> core.discovery.
endpoints.get_sync_status(...), wired via _get_source_sync_status glue.
This cluster carries the real per-source quirks, all captured 1:1 as params:
- not_found_message (iTunes-Link uses "iTunes Link not found").
- error_label vs activity_subject — these DIFFER for Spotify-Public: the
activity feed says "Spotify Link playlist ..." while the except log says
"Error getting Spotify Public sync status".
- playlist-name accessor, three styles lifted verbatim as named helpers:
playlist_name_attr_or_unknown (Tidal: object .name), playlist_name_strict
(Deezer/Qobuz/Spotify-Public/iTunes: state['playlist']['name'], can raise),
playlist_name_safe (YouTube/ListenBrainz: .get default). The strict getter
preserves the original's behavior of raising -> 500 AFTER phase/sync_progress
were already mutated.
- ListenBrainz key via _lb_state_key (caller-resolved).
Beatport stays separate (different payload: status not sync_status, sync_id,
no lock, chart key).
Tests: +9 (3 name accessors incl. raise/fallback semantics; status 404s,
running-no-mutation, finished+activity, error+revert+activity, and strict-
getter-missing -> 500 after partial mutation). Full discovery suite: 171 passed.
web_server.py: -244 lines.
First cluster of the per-source playlist-discovery deduplication. The
convert_<source>_results_to_spotify_tracks functions (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube, ListenBrainz) plus the already-generic
_convert_link_results_to_spotify_tracks were byte-identical apart from the
source label used in their log line.
Lift the shared body into core/discovery/endpoints.py as
convert_results_to_spotify_tracks(results, source_label); the 7 web_server
functions become 1-line delegations (names/signatures unchanged, so all
callers and behavior are identical — 1:1).
Beatport is intentionally NOT folded in: its converter coerces artist
objects to strings and emits a different track shape (source field, album
dict), so it keeps its own implementation.
Tests: tests/discovery/test_discovery_endpoints.py (12) pin both input
shapes (manual spotify_data / auto spotify_track+found), optional
track/disc numbers, falsy-0 omission, field defaults, skip-on-neither,
order preservation, if/elif precedence, empty input.
web_server.py: -209 lines. Full discovery suite: 151 passed.
Follow-up to the album-art resolution fix. That change upgraded MusicBrainz
Cover Art Archive thumbnails (/front-250) to the bare /front original — but
/front redirects to archive.org, which is unreliable: probing release-group
covers showed intermittent HTTP 500s (same URL 500s one second, serves the
next) and multi-MB originals (2.9 MB seen). The result was the user-reported
flakiness: cover art that "sometimes works, sometimes shows nothing", and a
huge image embedded into every track when it did work.
The sized thumbnails (/front-250, -500, -1200) are served by CAA's own CDN,
not the archive.org redirect — which is why /front-250 (240p) was always
reliable. Upgrade to /front-1200 instead: 1200x1200 is a massive jump from
240p, reliably CDN-served, and a sane ~40 KB instead of multi-MB.
Applied in all three CAA spots for consistency: the _upgrade_art_url helper
(embed + cover.jpg paths) and both prefer_caa ("CCA") blocks, which fetched
the bare /front directly with no fallback — so CCA-on users hit the same
flakiness. _fetch_art_bytes still falls back to the original /front-250 if
/front-1200 is ever refused.
Tests updated to assert the 1200px target, idempotency, and that the bare
/front original is intentionally left untouched.
User report: embedded album art came out ~600x600 while the cover.jpg in
the folder was high-res. The cover.jpg path upgraded the source CDN URL
to its highest resolution, but the tag-embed path fetched the raw URL —
so iTunes art embedded at its 600x600 default, Spotify at 640, Deezer at
1000. The "Write Tags to File" retag path had the same gap (Deezer-only
upgrade), and MusicBrainz art was worse still: every Cover Art Archive
URL is built as the /front-250 thumbnail, so MB-sourced downloads
embedded 250x250.
Factor the resolution upgrade + fetch into two shared helpers in
core/metadata/artwork.py and route every art path through them:
_upgrade_art_url(url) — bump to the source's highest resolution:
- Spotify (i.scdn.co) -> original master (~2000px+)
- iTunes (mzstatic.com) -> 3000x3000
- Deezer (dzcdn) -> 1900x1900
- Cover Art Archive -> /front original (was /front-250)
_fetch_art_bytes(url) — upgrade, fetch, and fall back once to the
original size if the CDN refuses the larger one (non-regressive).
Now consistent across: embed-into-tags (post-process), folder cover.jpg
(post-process), and the enhanced-library "Write Tags to File" retag flow.
The YouTube path already upgraded via Album.from_spotify_album, unchanged.
De-duplicates the per-source upgrade code that was copied across sites
and drops the now-unused urllib import from tag_writer.
Not covered (follow-up): Last.fm / Amazon / Tidal / Qobuz have no
explicit upgrade yet — some already serve full-res, others may hand over
a capped size that passes through unchanged.
Tests: new tests/metadata/test_artwork_resolution.py pins every upgrade
(Spotify 300/640->master, iTunes 100/600->3000, Deezer->1900, CAA
thumbnail->original, unrecognized/empty unchanged) and the fetch
fallback. Updated the two tag_writer fallback tests to patch the network
at its new home in artwork.
The four artist-hero buttons (Artist Radio, Watchlist, Download
Discography, Enhance Quality) had drifted apart visually — different
sizes, weights, radii, hover treatments, and ad-hoc colors. Unify them
on the Download Discography look (the nicest of the set): accent-style
gradient fill, matching border, light-tinted text, compact 7x16 / 12px /
700 sizing, and a translateY(-1px) + colored glow on hover.
To keep them distinguishable, each carries its own hue within that
shared recipe:
- Artist Radio -> violet (#8b5cf6)
- Watchlist -> theme accent (keeps its amber "watching" state)
- Download Discography -> theme accent + shimmer (the primary action)
- Enhance Quality -> cyan (#4fc3f7, its original signature color)
Also:
- Drop the shimmer from the three secondary buttons — four simultaneous
shimmers were distracting; it now marks only the primary action.
- Remove the `margin: 12px 0 4px` on `.discog-download-wrap` (now
`margin: 0; display: inline-flex`) that pushed the discography button
~4px below its siblings in the centered flex row.
- Include Artist Radio in the mobile button sizing override (was missing).
Opening a library artist from a non-library search result (e.g. a
MusicBrainz hit) leaves the artist-detail page holding the source ID —
the MBID — not the integer library PK. The standard /api/artist-detail
route resolves that via find_library_artist_for_source, but the
enhanced-view (`/api/library/artist/<id>/enhanced`) and quality-analysis
endpoints call get_artist_full_detail directly with whatever ID the page
holds. Its lookup was `WHERE id = ?` only, so it 404'd ("Artist with ID
<mbid> not found") and the enhanced view failed to load.
When the direct PK lookup misses, fall back to matching any per-service
ID column, reusing SOURCE_ID_FIELD as the single source of truth so the
resolution covers every source (MusicBrainz, Spotify, Deezer, iTunes,
Discogs, Hydrabase, Amazon), not just MusicBrainz.
Adds 4 isolated DB-method tests: direct PK still works, resolves by
MBID, resolves by Spotify ID, and unknown IDs still 404.
Artist-detail discography from MusicBrainz fetched releases via the
artist lookup (`/artist/<mbid>?inc=release-groups`), which MusicBrainz
hard-caps at 25 embedded release-groups and which ignores the `limit`
param entirely. Prolific artists had ~85% of their catalogue silently
dropped — Kendrick Lamar has 167 release-groups on the site but only the
first 25 ever reached SoulSync. Reported by Sokhi: "a lot of albums are
missing when searching vs what's showing on the site."
Switch `get_artist_albums` to walk the paginated browse endpoint
(`/release-group?artist=<mbid>`, offset loop) — the same pattern the
basic-search path already uses — fetching the full catalogue up to the
caller's limit. No type filter and no studio-only filter here: the
artist-detail page wants every primary/secondary type so its tabs mirror
musicbrainz.org. Verified live: now returns all 167 for Kendrick.
Adds 7 tests covering pagination past the cap, offset advance,
short-page stop, limit cap, cross-page dedup, type->bucket mapping, and
a regression pin asserting the capped inc=release-groups lookup is no
longer the discography source.
Two things in this commit. Functional download / matched-download
behaviour is untouched — same JS handlers, same routes for the
download actions, same album-expand interaction.
VISUAL REDESIGN
- Glass search-bar card with accent radial wash + focus ring + pill
primary search button
- Source chip row above the search bar (see below)
- Always-visible compact filter pill row (Type / Format / Sort) —
pills carry both ``bs-filter-pill`` (new visual) and ``filter-btn``
(legacy class for ``resetFilters`` + ``applyFiltersAndSort`` in
wishlist-tools.js to keep working)
- Accent-tinted status pill matching the dashboard / auto-sync look
- Album result cards: glass card with accent left-edge stripe,
52px brand-tinted cover icon, chevron expand indicator, pill
action buttons (Download / Matched Album), accent glow on hover
- Track result cards: glass row with accent stripe, 44px icon,
pill action buttons (Stream / Download / Matched Download)
- Multi-disc separators inside expanded album track lists styled
with the accent treatment
- Responsive: action button columns stack vertically below 900px
New CSS lives in a self-contained ``webui/static/basic-search-v2.css``
sheet linked from index.html. Selectors are scoped to
``#basic-search-section`` for any class that already exists in
style.css (``.album-result-card``, ``.album-icon``, ``.track-*``,
etc.); the new ``bs-*`` prefixed classes for the search bar /
filters / source row / status are unscoped because they only exist
in the new markup. ``!important`` is used on the card-level rules
to defeat the original unscoped ``.album-result-card`` etc. rules
in style.css that would otherwise leak heavyweight padding /
box-shadow / 56px icon styles into the new design.
Also removed ``overflow: hidden`` from the original
``.album-result-card`` and ``.track-result-card`` rules in style.css
— those two classes only render in ``downloads.js`` basic search
results (verified via grep, two render sites only), so the
removal can't impact any other UI.
SOURCE PICKER (hybrid mode)
- New ``GET /api/search/sources`` endpoint returns the list of
active sources from the orchestrator's chain (or the single
active source in single-source mode).
- Frontend renders a chip row above the search bar. Click a chip
to target that source for the next search; the chip's brand
accent fills.
- In single-source mode the lone chip is rendered as a dashed-
border label so the user always knows what they're searching
but can't accidentally try to switch to sources that aren't
configured.
- ``/api/search`` accepts an optional ``source`` body param. When
set, ``core/search/basic.py:run_basic_search`` resolves the
client directly via ``orchestrator.client(source)`` and calls
its ``.search()`` instead of going through the hybrid chain.
- Backwards compatible: omitting ``source`` falls through to the
original ``orchestrator.search()`` call exactly as before.
Unknown source names also fall back to the default — typo
protection.
TESTS (5 new + 6 pre-existing = 11 total in test_search_basic.py)
- source param routes to specific client, NOT orchestrator chain
- no source param preserves original orchestrator-default behaviour
- unknown source name falls back to orchestrator default
- ``run_basic_soulseek_search`` backwards-compat alias preserved
- source-targeted path serialises albums + tracks correctly
101 search-suite tests pass.
Reporter @Sokhii: downloading the Mushoku Tensei Original
Soundtrack II via Apple Music metadata + Tidal download
produced duplicate library entries — same audio file landed
under multiple track positions in the album view.
Root cause (verified by direct probe + isolated repro):
``MusicMatchingEngine.normalize_string`` correctly skipped
unidecode for CJK text (kanji→pinyin would have produced
gibberish — see the inline comment at line 74-76), but then
ran ``re.sub(r'[^a-z0-9\s$]', '', text)`` which stripped EVERY
CJK character. Every Japanese title normalised to ``''``.
``similarity_score`` has an early-out guard
if not str1 or not str2: return 0.0
so EVERY CJK-vs-CJK title comparison returned 0.000.
Downstream effect: the matcher fell back to duration+artist
alone. For an OST album with 24 tracks all by the same artist
with similar durations, multiple iTunes track queries landed
on the SAME Tidal candidate. SoulSync wrote each download to
a different output filename (per the iTunes track position),
so on disk there were N copies of the same audio under
different track numbers. The user's library showed 34 entries
for an album with 24 actual tracks.
Probed iTunes album 1753240110 directly — 24 distinct tracks,
zero (disc, track_number) collisions, both US + JP storefronts.
So the duplicate origin was definitely downstream of metadata
fetch.
Fix: when CJK is detected upstream, the alphanumeric-strip step
also preserves CJK Unified Ideographs + radicals
(⺀-鿿), Hiragana + Katakana (-ヿ), Halfwidth
/ Fullwidth forms (-), and Hangul syllables
(가-). CJK titles now produce a comparable normalised
form instead of an empty string. ``similarity_score`` works as
intended:
'命の灯火' vs '命の灯火' → 1.000 (was 0.000)
'命の灯火' vs '無職転生' → 0.000 (was 0.000, but now from
actual char comparison
not from the empty-string
guard)
Latin-only normalisation is completely unchanged. ``has_cjk``
is False for Latin input, so both the CJK-lowercase branch AND
the new CJK-preserve strip branch are skipped — Latin titles
go through the original unidecode + lowercase + strip path
verbatim. Tested via 4 regression tests that pin the Latin
baseline (simple, unidecode target, $-preservation, identical
+ different similarity scores).
16 new unit tests in ``tests/test_matching_engine_cjk.py``:
- Kanji / Hiragana / Katakana / Hangul / Chinese all survive
- CJK-only strip still removes Latin punctuation in the
CJK branch
- Mixed Latin + CJK lowercases the Latin half
- Identical CJK titles → 1.0
- Disjoint CJK titles → near 0
- Partially overlapping CJK titles → midrange
- CJK doesn't falsely match unrelated Latin
- 4 Latin-baseline regression pins
- Real-world Mushoku Tensei OST scenario
371 text + imports + new CJK tests pass after the fix.
