Plug the previously-built SoundcloudClient (PR #478, the build-and-verify
phase) into every place a download source needs to appear. Follows the
same wiring contract as Tidal/Qobuz/HiFi/Deezer/Lidarr — orchestrator
routing, hybrid-mode picker, search dispatch, queue/cancel/clear,
provenance + library history, sidebar source label, settings UI all
work plug-and-play.
Backend wiring:
- `core/download_orchestrator.py` — import SoundcloudClient, _safe_init
it at startup, add to _client() lookup, get_source_status(),
check_connection's sources_to_check default, search source_names map,
search_and_download_best _streaming_sources tuple, download
source_map + source_names, and every iteration loop in
reload_settings download-path-update / get_all_downloads /
get_download_status / cancel_download (route + iterate) /
clear_all_completed_downloads / cancel_all_downloads.
- `core/downloads/monitor.py` — added SoundCloud to the per-client
loop that fetches active downloads outside the orchestrator (uses
getattr fallback for older soulseek_client snapshots).
- `core/downloads/task_worker.py` — added SoundCloud (and Lidarr,
which was missing too — bonus fix) to source_clients dict for hybrid
fallback dispatch.
- `core/downloads/validation.py` — added 'soundcloud' to
_streaming_sources so SoundCloud results go through the matching
engine validation path instead of the Soulseek quality-filter path.
- `core/imports/side_effects.py` — three call sites: source_map for
download_source label written to library_history, streaming-source
guard for the `||`-encoded stream_id parsing, and source_service
map for provenance recording. All three now include 'soundcloud'.
- `web_server.py` — five streaming-source detection tuples updated.
New `/api/soundcloud/status` endpoint returns
{available, configured, reachable} mirroring the Deezer/HiFi
status-endpoint pattern; reachability runs a real cheap yt-dlp
search so the settings Test Connection button gives a meaningful
pass/fail signal.
- `config/settings.py` — added empty `soundcloud_download` defaults
block so future tier-2 OAuth (SoundCloud Go+ session) doesn't have
to migrate existing configs.
Frontend:
- `webui/index.html` — new `<option value="soundcloud">` in the
download-source-mode dropdown, SoundCloud added to both hidden
legacy hybrid-source selects, new settings container with info
text + Test Connection button.
- `webui/static/settings.js` — HYBRID_SOURCES entry (with the
SoundCloud cloud SVG icon), _hybridSourceEnabled default,
updateDownloadSourceUI container display, allSources for legacy
hybrid picker, testSoundcloudConnection function (hits the new
status endpoint, color-codes the result), saveSettings
soundcloud_download empty block.
- `webui/static/shared-helpers.js` — sidebar source-name map
includes SoundCloud + Lidarr (Lidarr was also missing, bonus fix).
- `webui/static/helper.js` — WHATS_NEW entry under '2.4.2' dev cycle
describing the user-visible change in the chill terse voice.
Tests:
- `tests/test_download_orchestrator_soundcloud.py` — 14 integration
tests verifying the wiring: client constructed at startup, _client
lookup resolves 'soundcloud', get_source_status includes it,
download dispatcher routes username='soundcloud' to the SoundCloud
client (and unknown usernames still fall back to Soulseek), hybrid
search iterates SoundCloud when in order and skips it cleanly when
unconfigured, get_all_downloads / get_download_status / cancel /
clear walk SoundCloud, soundcloud-only mode dispatches only to
SoundCloud, _streaming_sources tuple in validation includes
'soundcloud'.
- `tests/downloads/test_download_orchestrator.py` — added
`soundcloud` to the test fixture's _build_orchestrator helper so
the new orchestrator attribute doesn't AttributeError in pre-
existing tests that bypass __init__.
Verified:
- Full suite green (1728 passed, 2 deselected for soundcloud_live)
- Ruff clean
- Live SoundCloud-only mode search returns 25 SoundCloud tracks for
"kendrick lamar luther" in <2s, returning properly-shaped
TrackResult objects with username='soundcloud' and dispatch-key
filename ready for the download path.
Out of scope (intentional deferrals):
- SoundCloud Go+ OAuth tier (256 kbps AAC) — anonymous-only for now.
Adding auth later is a settings-page extension, no orchestrator
changes needed.
- Album/playlist support — SoundCloud has playlists but they don't
map to the album model the rest of SoulSync expects. Singles only.
Discord request (Toasti): some tracks (DJ mixes, sets, removed Spotify
content) only live on SoundCloud. Add SoundCloud as an option for the
existing multi-source download dispatch.
This commit only ships the client + tests. Integration into the search
dispatch / settings UI / web_server.py routes is intentionally deferred
to a follow-up PR — the user-requested workflow is build-and-verify
in isolation first, then wire up.
`core/soundcloud_client.py`:
- SoundcloudClient class mirrors the public surface of every other
download client (TidalDownloadClient, QobuzClient, HiFiClient,
DeezerDownloadClient): __init__(download_path), set_shutdown_check,
is_available / is_configured / is_authenticated, async check_connection,
async search returning (List[TrackResult], List[AlbumResult]),
async download returning a download_id, _download_thread_worker /
_download_sync / _update_download_progress, async get_all_downloads /
get_download_status / cancel_download / clear_all_completed_downloads.
- Underlying lib: yt-dlp (already in requirements.txt as 2026.3.17).
- Anonymous-only — public SoundCloud tracks at the cap quality (typically
128 kbps MP3, occasionally 256 kbps AAC depending on the upload).
No FLAC ever; SoundCloud doesn't expose lossless. OAuth tier for
SoundCloud Go+ is documented in the module header as a future tier.
- Returns standard TrackResult / DownloadStatus dataclasses from
core.soulseek_client so downstream matching/post-processing stays
source-agnostic.
- Filename dispatch key encodes track_id + permalink_url + display_name
so the download worker has everything without re-querying.
- Heuristic "Artist - Title" parser handles SoundCloud uploaders'
typical title format; falls back to uploader handle as artist when
the title doesn't have a separator.
- Defensive: search returns empty on bad input, missing yt-dlp, or any
raised exception. Download sync rejects files under 100KB (preview
snippets / broken responses) and cleans them up.
- Cooperative cancellation via shutdown_check inside yt-dlp's
progress_hooks. Cancelled state survives the download thread's
terminal-state assignment.
`tests/test_soundcloud_client.py`:
- 37 unit tests with yt-dlp stubbed: search shape correctness, the
artist/title heuristic, the dispatch-key roundtrip, the download
state machine (success / failure / shutdown / Cancelled-state
preservation), the progress emitter (progress capping, time
remaining), defensive paths (missing yt-dlp, raising yt-dlp,
malformed entries, empty entries), and the cancel/clear ledger
operations.
- 2 live integration tests gated behind `-m soundcloud_live` so CI
doesn't run them by default. Run locally with:
python -m pytest tests/test_soundcloud_client.py -m soundcloud_live -v
- All 37 unit tests pass; both live tests pass against real SoundCloud.
- Verified end-to-end with a real album download (Kendrick GNX, 12/12
tracks, 4-7 MB each, completed under 60s per track).
`pyproject.toml`:
- Register the `soundcloud_live` pytest marker so the unknown-mark
warning is suppressed and the live tests can be cleanly gated.
Not changed: web_server.py, settings UI, search dispatch, matching
engine, WHATS_NEW. Integration is the next PR.
GitHub issue #476 (gabistek, Docker on Arch host): "Auto-Fill" / "Fix
Selected" on the Album Completeness findings page returned
"Could not determine album folder from existing tracks" for every album.
Reproduces on any setup where the media-server library lives outside the
SoulSync transfer/download folders — Docker is the headline case but
native installs that point Plex at a NAS via SMB hit it too.
Root cause: `core/repair_worker.py:_resolve_file_path` only probed the
transfer + download folders. Docker users have their Plex/Jellyfin
library bind-mounted at /music (or similar) — neither configured in
SoulSync. Every existing track got silently treated as missing, so
`album_folder` stayed None and the fix workflow bailed.
The same incomplete logic was duplicated four more times in the
repair_jobs/ modules, all with the same bug. Album Completeness was
just the most user-visible — the same setups were also producing false
"missing file" findings from Dead File Cleaner, silent skips in
MBID Mismatch Detector, etc.
The web server already had the correct logic at
`web_server.py:_resolve_library_file_path` (probes transfer + download
+ Plex-reported library locations + user-configured library.music_paths).
The repair workers had never been updated to match.
Fix:
- New `core/library/path_resolver.py` extracts the union logic into a
single shared function `resolve_library_file_path()`. Probes (in
order, deduped): explicit transfer/download kwargs, config-derived
soulseek.transfer_path/download_path, Plex-reported library
locations (when a plex_client is passed), user-configured
library.music_paths. Each defensive: malformed config or a flaky
Plex client degrades to the dirs that did succeed.
- `core/repair_worker.py:_resolve_file_path` becomes a delegating
wrapper preserving the legacy signature, with a new `config_manager`
kwarg. All 15 in-tree call sites updated to thread
`self._config_manager` through.
- `core/repair_jobs/dead_file_cleaner.py`,
`mbid_mismatch_detector.py`, and `lossy_converter.py` get the same
treatment: duplicate function replaced with a thin wrapper, call
sites pass `context.config_manager`.
- `core/repair_jobs/acoustid_scanner.py` and
`unknown_artist_fixer.py` (which used to import from repair_worker)
now call the shared resolver directly with `context.config_manager`.
Side benefit: every other repair job (Dead File Cleaner, MBID
Mismatch Detector, Lossy Converter, AcoustID Scanner, Unknown Artist
Fixer) also stops missing files in the media-server library mount.
Single fix unblocks five user-visible features.
Tests: `tests/library/test_path_resolver.py` — 20 cases covering all
four base-dir sources, suffix-walk algorithm, dedup, defensive paths
(None plex client, malformed config entries, raising config_manager.get,
broken plex attribute access), Docker path translation. Full suite
1677 passed locally.
WHATS_NEW entry under '2.4.2' dev cycle.
Discord report (fresh.dumbledore [VRN]): slskd sometimes ships broken files
(truncated transfers, corrupt FLAC, wrong file substituted on filename match).
They flowed through post-processing and only surfaced later — Plex/Jellyfin
scan failures, dead-air playback, duplicate detector tripping over the wrong
length. By that point the file was already tagged, copied, mirrored to the
media server, and recorded in provenance.
New module `core/imports/file_integrity.py`:
- `check_audio_integrity(path, expected_duration_ms=None) -> IntegrityResult`
- Three tiered checks, cheapest to most expensive:
1. File size sanity (catches 0-byte stubs and stub transfers)
2. Mutagen parse (catches header damage, wrong-format-with-right-extension)
3. Duration agreement vs. metadata source's expected length, ±3s tolerance
(5s for tracks over 10 minutes — long tracks naturally drift more)
- Returns IntegrityResult with `ok`, human-readable `reason`, and per-check
`checks` dict for debugging
- Never raises; pathological inputs return ok=False with explanation
Pipeline integration in `core/imports/pipeline.py:post_process_matched_download`:
- Hooks between the existing file-stability wait and AcoustID verification
- On failure: quarantine via existing `move_to_quarantine` helper, mark task
failed with descriptive error, clear matched-context, fire
`on_download_completed(success=False)` so the slot is released for retry
- Mirrors the existing AcoustID-failure path so retry behavior stays consistent
- Wrapped in try/except so an unexpected failure inside the check itself
cannot block downloads — logs and continues
This is intentionally tier 1: universal across formats, no external deps.
A future tier could verify FLAC STREAMINFO MD5 by decoding audio (needs
flac binary or libflac wrapper) — skipped for now since tier 1 catches the
dominant Discord-reported cases (truncated, 0-byte, wrong file).
Tests:
- `tests/imports/test_file_integrity.py` — 14 cases covering all three check
tiers, edge cases (zero/negative expected duration, long-track wider
tolerance, caller tolerance override), and the mutagen-unavailable
degradation path
- `tests/imports/test_import_pipeline.py` — two existing tests use 5-byte
fixture files that the new check would reject; they monkeypatch the
integrity check since they're testing plumbing (notification +
metadata_runtime forwarding), not integrity behavior
WHATS_NEW entry under '2.4.2' dev cycle.
User reported (Mushy / generally) that dropping an album into the
staging folder left the auto-import history blank for the entire
processing window — sometimes 5+ minutes for a full album. Pre-
existing UX gap, not caused by the recent context-builder refactor.
Two root causes:
1. ``_record_result`` only fired AFTER ``_process_matches`` returned.
For a 14-track album with ~30s/track post-processing, that meant
~7 minutes of zero rows in auto_import_history → nothing for
``/api/auto-import/results`` to return → empty UI.
2. ``_current_status`` only ever transitioned between 'idle' and
'scanning' — never 'processing'. ``get_status()`` had no per-
track index/name fields, so the UI had no way to render
"Processing track 3/14: Mine" even if it wanted to.
Fix:
- New ``_record_in_progress`` inserts a status='processing' row
up-front (before the per-track loop starts) so the UI sees the
import the moment it begins. Returns the row id.
- New ``_finalize_result`` updates that same row with the final
outcome (completed/failed) when processing finishes. One row per
album, not per track — keeps the history list clean.
- Both share ``_serialize_match_data`` (extracted from the original
``_record_result``) so the in-progress row carries the same match
payload shape the existing review UI already understands.
- ``_process_matches`` updates ``_current_track_index``,
``_current_track_total``, and ``_current_track_name`` BEFORE each
per-track callback fires, so a polling UI sees consistent
"processing N/M: <name>" snapshots.
- ``_scan_cycle`` flips ``_current_status`` to 'processing' before
the per-album loop, resets it + the per-track fields after.
Defensive ``finally`` clears progress even if the inner code path
raised.
- ``get_status()`` exposes the new fields so the UI's existing
/api/auto-import/status polling picks them up.
- Frontend (stats-automations.js): renders the new
``current_status='processing'`` state with track index/total/name
in the existing progress bar element. New 'processing' status
class for styling parity with 'scanning'.
8 regression tests in tests/imports/test_auto_import_live_progress.py:
- get_status surfaces the new fields with sane defaults
- track_index advances 1, 2, 3 during a 3-track loop
- track_total set BEFORE the first callback fires (no '1/0' flicker)
- _record_in_progress writes status='processing' with no
processed_at
- _finalize_result updates the same row to completed +
processed_at, no second insert
- _finalize_result with failed status leaves processed_at NULL
- _finalize_result with row_id=None is a safe no-op
- Per-track fields cleared by _scan_cycle's finally block
Full pytest 1643 passed; ruff clean.
Reported case (Foxxify): Tidal returned error 1002 ("Invalid redirect
URI") on every authentication attempt for users accessing SoulSync
from a network IP. User had ``http://127.0.0.1:8889/tidal/callback``
registered in his Tidal Developer Portal — matching the SoulSync UI
default and docs.
Root cause: the /auth/tidal route at web_server.py:5594-5598 had a
"fallback: dynamically set based on request host" branch that fired
when ``tidal.redirect_uri`` config was empty AND the request didn't
come from localhost. That fallback overrode the TidalClient
constructor's safe default (``http://127.0.0.1:<port>/tidal/callback``)
with a uri built from request.host like
``http://192.168.x.x:8889/tidal/callback``. Tidal compares strings
exactly so this never matched the documented portal registration and
the user got 1002 before the consent screen even rendered.
The trap is the SoulSync settings UI displays the default URI as the
placeholder + "Current Redirect URI" display — but the placeholder
never gets saved to config unless the user explicitly clicks Save.
Most users who follow the docs (register the displayed default with
Tidal, then click Authenticate) hit the empty-config path and the
broken fallback.
Fix: drop the request-host fallback. Empty config falls back to the
constructor default that matches the documented portal registration.
The existing post-auth swap-step in the instructions page below
handles the Docker / remote-access case as designed:
1. SoulSync sends 127.0.0.1:8889 in the authorize URL → matches
portal → Tidal accepts.
2. User authorizes → Tidal redirects browser to 127.0.0.1:8889
(which fails locally — nothing on user's machine listens there).
3. Instructions tell user to swap 127.0.0.1 with the host they're
accessing SoulSync from.
4. Swapped URL hits the container's exposed callback port → auth
completes.
8 regression tests in tests/test_tidal_auth_redirect_uri.py:
- Configured redirect_uri sent verbatim (localhost / custom port /
explicit network IP)
- Empty config falls back to constructor default — NOT request.host
(the actual reported scenario, with explicit assertion message
warning if the bug returns)
- Empty config + localhost access uses the same default (sanity)
Full pytest 1635 passed; ruff clean.
Smoke-testing the just-merged provenance PR against live logs revealed
the new ID-match block was silently no-opping: no [ExtID Match] /
[Provenance Match] log lines despite the code path being live. Tracing
revealed two related gaps in extract_external_ids' source detection:
1. **Underscore-prefixed key.** Deezer / Discogs / Hydrabase clients
tag normalized track dicts with ``_source`` (underscore prefix —
convention used in 8+ places across core/). The extractor only
looked for ``provider`` and ``source``, so Deezer-sourced tracks
silently returned no IDs.
2. **No provider field at all.** Spotify and iTunes raw API responses
carry ``id`` but no provider/source key of any kind. The extractor
couldn't disambiguate the native ``id``, so Spotify-primary scans
would have hit the same silent miss once the user switched primary
sources.
Two-part fix:
- ``extract_external_ids`` now recognizes ``_source`` as another
candidate provider field.
- New optional ``source_hint`` parameter lets the caller supply the
configured primary source as a fallback when the track dict has no
provider field of its own. Track-side provider field still wins
when present (defensive against a wrong hint).
Watchlist scanner now passes ``get_primary_source()`` as the hint so
both naming conventions (Deezer-style _source, Spotify-style no-tag)
get handled uniformly.
6 new regression tests cover:
- _source recognized for Deezer
- _source recognized for Hydrabase (cross-provider mapping)
- _source recognized for Discogs (no library column — verifies
graceful no-crash)
- source_hint disambiguates raw tracks for spotify/itunes/deezer
- track-side provider takes precedence over hint
- None hint defaults safely
Full pytest 1630 passed; ruff clean. After this lands and the server
restarts, watchlist scans should produce [ExtID Match] /
[Provenance Match] log lines for tracks already on disk regardless of
which metadata source the user has configured as primary.
Followup to fix/watchlist-external-id-match. The companion PR closed
the demand side — the watchlist scanner asks for tracks by external IDs
before falling back to fuzzy. But for users on Plex / Jellyfin /
Navidrome the supply side was still broken: tracks.spotify_track_id
(and the other ID columns) only got populated by the asynchronous
enrichment workers, sometimes hours after the file was actually
written. During that window the ID match fell through to fuzzy and
the bug returned.
We were already collecting every ID during post-processing — they
live in the `pp` dict in core/metadata/source.py:embed_source_ids and
get embedded into file tags. We just dropped the in-memory copy
afterwards.
This PR persists them and uses them:
- Schema migration adds spotify_track_id / itunes_track_id /
deezer_track_id / tidal_track_id / qobuz_track_id /
musicbrainz_recording_id / audiodb_id / soul_id / isrc columns +
indexes to the existing track_downloads table (already keyed by
file_path).
- core/metadata/source.py:embed_source_ids exposes pp["id_tags"] and
the resolved ISRC back to the import context as _embedded_id_tags
/ _isrc.
- core/imports/side_effects.py:record_download_provenance reads those
context fields and passes them to db.record_track_download, which
now accepts the new ID kwargs and persists them.
- New db.get_provenance_by_file_path with exact + basename-suffix
fallback (handles container mount-root differences between
download-time path and media-server-reported path).
- New db.backfill_track_external_ids_from_provenance copies IDs
from track_downloads onto a tracks row idempotently — COALESCE on
every column preserves any value the enrichment worker already
wrote (enrichment is more authoritative for late binding).
- database/music_database.py:insert_or_update_media_track (the
single insertion point used by every Plex / Jellyfin / Navidrome
sync) calls the backfill immediately after each INSERT/UPDATE.
- New core/library/track_identity.py:find_provenance_by_external_id
used as a second-tier fallback in watchlist_scanner.is_track_missing
_from_library — catches the window between download and media-server
sync. Caller checks os.path.exists on the provenance file_path
before treating it as "already in library" so a deleted file
doesn't prevent re-download.
Effect: freshly downloaded files become ID-recognizable to the
watchlist on the very next scan, no enrichment-wait window.
19 regression tests in tests/test_provenance_id_persistence.py:
- Schema migration adds expected columns + indexes
- record_track_download persists every ID kwarg
- record_track_download backward-compat (old kwargs still work)
- get_provenance_by_file_path: exact match, basename fallback for
mount-root differences, multi-record latest-wins, defensive None
- backfill: copies all IDs, preserves existing via COALESCE,
no-op when no provenance exists
- find_provenance_by_external_id: per-ID lookup, ISRC cross-bridge,
OR semantics, latest-wins on multiple matches
Out of scope: backfilling provenance for files downloaded BEFORE
this PR (their track_downloads rows don't carry the new IDs). Those
continue to wait for enrichment. Acceptable — only affects historical
files; new downloads benefit immediately.
Full pytest 1625 passed; ruff clean.
Reported case (CAL): a track already on disk got re-downloaded by the
watchlist scanner on every scan. Library DB had stale album metadata
for the file (track tagged on album "Left Alone") while the metadata
source reported it on a different album ("NPC" single). The
title+artist+album fuzzy block correctly said the album names didn't
match and declared the track missing — but the file's stable external
IDs (Spotify ID, ISRC, etc.) unambiguously identified it as the same
recording.
The earlier compilation-album fix (PR #461) handled qualifier drift
("OST" vs "Music From The Motion Picture"). This case is two
genuinely different album names referring to the same song.
Fix: provider-neutral external-ID short-circuit before the fuzzy
block in `is_track_missing_from_library`. Pulls every recognized ID
off the source track (Spotify / iTunes / Deezer / Tidal / Qobuz /
MusicBrainz / AudioDB / Hydrabase / ISRC), runs a single SELECT
against the indexed external-ID columns on the `tracks` table, and
treats any hit as "track exists in library — don't re-download".
If no IDs are available (older imports without enrichment, library
scans that didn't populate external IDs), falls through to the
existing fuzzy logic so the safety net stays intact.
New `core/library/track_identity.py` module with two helpers:
- `extract_external_ids(track)`: handles dict and object-style track
shapes, direct-field aliases (spotify_id / spotify_track_id /
SPOTIFY_TRACK_ID), and provider-disambiguated native `id` fields
(when track has `provider='deezer'` and `id='X'`, treats X as a
Deezer ID).
- `find_library_track_by_external_id(db, external_ids,
server_source)`: builds an OR of indexed column matches with
IS NOT NULL guards, optional server_source filter that also
passes legacy NULL rows, single-row LIMIT.
ISRC bridges across providers — a library track imported via Deezer
can be matched against a Spotify scan when both sides carry the
same ISRC.
43 regression tests in `tests/test_library_track_identity.py`:
- 9 ID-extraction tests for direct fields (Spotify / iTunes / Deezer /
ISRC / MBID / AudioDB / Hydrabase)
- 8 ID-extraction tests via the provider field (8 providers + source
alias + missing-provider-ignored)
- 7 mixed/defensive tests (multiple IDs, object-style, empty strings,
None track, numeric coercion)
- 8 lookup tests (per-provider + ISRC cross-bridge)
- 3 OR-semantics tests
- 4 server_source filter tests
- 2 ID-column-map sanity tests
Full pytest 1606 passed; ruff clean.
Reported case (CAL): with lossy_copy.enabled=True,
lossy_copy.delete_original=True, and codec=mp3, every download left
both the original FLAC AND the converted MP3 in the target folder.
Users opting into a lossy-only library ended up dual-format on
every import.
Root cause: ``core/imports/file_ops.py:create_lossy_copy`` reads
``lossy_copy.codec`` and ``lossy_copy.bitrate`` from config but never
reads ``lossy_copy.delete_original``. The setting is only consulted
by the pre-move source-vanished check at
``core/imports/pipeline.py:651`` (so the pipeline knows to look for
a lossy variant when the FLAC has already moved on), but no code
path actually deletes the source after conversion.
Fix: after ffmpeg returns success and the QUALITY tag is written,
check ``lossy_copy.delete_original`` and ``os.remove`` the original
when enabled. Belt-and-suspenders:
- Same-path guard (``os.path.normpath(out_path) != os.path.normpath(final_path)``)
prevents accidentally wiping the just-converted file if a future
codec choice somehow resolves out_path to the source path.
- ``FileNotFoundError`` is treated as success (concurrent worker /
dedup cleanup got there first).
- Other ``OSError`` (permission denied, locked file) is logged but
doesn't propagate — the conversion already succeeded, the user just
has to clean up the original manually.
Failure paths skip the delete:
- ffmpeg returns non-zero → returns None, original stays
- lossy_copy.enabled=False → early return before conversion runs
- delete_original=False (default) → original stays
7 regression tests cover honored-when-enabled, kept-when-disabled,
default-keep, ffmpeg-failure-path, lossy-disabled-path, racing-delete,
and locked-file paths. Full pytest 1563 passed; ruff clean.
Note: this PR does NOT address the second bug CAL mentioned (track
re-downloaded despite already existing on disk). That symptom is
caused by stale album metadata on the user's existing files — the
library DB has the track tagged on a different album than the
metadata source reports — combined with wishlist.allow_duplicate_tracks
defaulting to True. Same class of issue partially addressed in PR
fix/watchlist-redownload-and-duplicate-detection but compilation-
album drift is the only currently-handled case. Tracking separately.
Discord-reported (Foxxify): logging in to Qobuz via the Connect
button on Settings showed "Connected: <username> (Active)" but
underneath an error said "Qobuz not authenticated...", and the
dashboard indicator stayed yellow. Saving settings or reloading the
tab didn't help.
Root cause: SoulSync runs two QobuzClient instances side by side —
one through soulseek_client.qobuz for the /api/qobuz/auth/* endpoints,
and a second owned by the enrichment worker thread for thread safety.
The login flow only updated the auth-flow instance's in-memory state
(plus persisted to config). The dashboard's "configured" check at
web_server.py:3371 reads
``qobuz_enrichment_worker.client.user_auth_token`` — the WORKER's
instance — which still believed itself unauthenticated. The
connection-test step at core/connection_test.py:370 hits the same
worker instance for the same reason.
Fix: add ``QobuzClient.reload_credentials()`` — a public, network-free
method that re-reads the saved session from config and updates the
instance's in-memory state + session headers. Call it on the
enrichment worker's client immediately after a successful
``/api/qobuz/auth/login``, ``/api/qobuz/auth/token``, or
``/api/qobuz/auth/logout`` so the two instances stay in lockstep
without waiting for the next process restart.
Unlike the existing ``_restore_session()`` this skips the network
probe — the caller has just authenticated, so the token is known
good. A small ``_sync_qobuz_credentials_to_worker()`` helper in
web_server.py wraps the call so all three endpoints share one path.
10 new regression tests cover the populate / clear / partial-config
paths plus the actual two-instance-sync scenario from the bug report.
Full pytest 1555 passed (the one pre-existing flake in
test_tidal_auth_instructions.py is order-dependent and unrelated).
- keep existing /api/image-proxy URLs from being wrapped again
- reuse the shared metadata package instead of duplicating URL logic in web_server.py
- add regression coverage for proxy passthrough and internal URL normalization
- Prefer real Spotify IDs when importing Spotify contexts
- Skip numeric fallback IDs so Deezer values do not leak into spotify_* columns
- Add regressions for import context and SoulSync library writes
- Keep the route test asserting the Spotify album link
- Keep the primary metadata provider snapshot generic and move Spotify auth/rate-limit details into a separate status object.
- Update the websocket fixture and dashboard/settings consumers to read the two buckets independently.
Followup to the enrichment-bubble registry consolidation. The
dashboard polling + click handlers all hit
/api/enrichment/<service>/{status,pause,resume} now, so the 30
hand-rolled per-service routes in web_server.py have zero callers
and can come out:
/api/musicbrainz/{status,pause,resume}
/api/audiodb/{status,pause,resume}
/api/discogs/{status,pause,resume}
/api/deezer/{status,pause,resume}
/api/spotify-enrichment/{status,pause,resume}
/api/itunes-enrichment/{status,pause,resume}
/api/lastfm-enrichment/{status,pause,resume}
/api/genius-enrichment/{status,pause,resume}
/api/tidal-enrichment/{status,pause,resume}
/api/qobuz-enrichment/{status,pause,resume}
Worker init blocks stay (they still construct the workers + persist
pause state). Section comment headers are preserved with a one-line
note pointing readers at the new generic blueprint.
Test fixtures in tests/conftest.py and
tests/metadata/test_enrichment_events.py also updated to use the
new URL paths so they reflect production reality. They were
synthetic stubs that never depended on the production routes —
purely cosmetic alignment.
Net: ~510 lines deleted from web_server.py. Full pytest 1541
passed; ruff clean.
The dashboard's enrichment-status bubbles (MusicBrainz, AudioDB,
Discogs, Deezer, Spotify, iTunes, Last.fm, Genius, Tidal, Qobuz) each
had its own copy-pasted /status, /pause, /resume route in web_server.py
— 30 routes that differed only in the worker reference and a couple
of per-service quirks (Spotify's rate-limit guard, Last.fm/Genius
yield-override behavior, Tidal/Qobuz extra status fields).
Replace them with a registry-driven blueprint:
- core/enrichment/services.py declares an EnrichmentService dataclass
with worker_getter, config_paused_key, pre_resume_check,
auto_pause_token, and extra_status_defaults — all variation captured
as data, no branching on service id.
- core/enrichment/api.py exposes a Flask blueprint with three routes
(/api/enrichment/<service>/{status,pause,resume}). Per-service
quirks are honored via the descriptor: Spotify's rate-limit ban
still returns 429 with `rate_limited: true`, Last.fm/Genius still
drop the auto-pause token and add the yield override, Tidal/Qobuz
still merge `authenticated: false` into the fallback payload.
- web_server.py registers all 10 services after their workers
initialize, wires the host-side hooks (config_manager.set,
_download_auto_paused.discard, _download_yield_override.add), and
registers the blueprint.
- webui/static/enrichment.js polling + click handlers now hit the
generic endpoints. The per-service `update<Service>StatusFromData`
functions are unchanged — they still process the same payload.
This is the cutover step. Old per-service routes are intentionally
left in place as a fallback during the soak period — they currently
have zero callers in the codebase and will be deleted in a follow-up
patch once production has run on the new pipeline for a few days.
27 new tests in tests/test_enrichment_services.py cover the registry
behavior + every quirk path through the generic blueprint (rate-limit
guard, auto-pause token cleanup, persisted-pause config keys, extra
default fields, worker-not-initialized fallback, exceptions). Full
suite 1541 passed; ruff clean.
Self-review of the previous commit found a real false-positive risk in
the new filename-bucket pass: two unrelated songs that happen to share
a canonical filename (e.g. ``Yellow.mp3`` by Coldplay vs by some other
artist) would be grouped because all metadata gates were dropped.
The filename pass now layers a safety net under ``require_metadata_match=False``:
- If both rows carry a duration: must agree within 3 seconds. Same
source download = identical duration; a 3+ second gap means
different recordings.
- Else if both rows carry an artist: relaxed 0.6 similarity check —
catches dedup orphans that share an artist tag while rejecting
strangers-with-same-filename.
- Else (no duration AND at least one artist blank): skip — too little
signal to safely group.
5 additional regression tests cover the false-positive prevention
paths plus the genuine dedup-orphan scenarios that must still be
caught after the safety net.
Two related bugs reported on Discord by Mushy.
1. The watchlist re-downloaded the same OST track up to 7 times.
``is_track_missing_from_library`` compared Spotify's album name and
the media-server scan's album name with a raw SequenceMatcher at a
strict 0.85 threshold. Compilations and soundtracks routinely fail
this — Spotify reports
``"Napoleon Dynamite (Music From The Motion Picture)"`` while the
Plex / Navidrome / Jellyfin tag scan saves it as
``"Napoleon Dynamite OST"``. Raw similarity ≈ 0.49, so the scanner
declared the track missing on every 30-minute scan and added it back
to the wishlist. The wishlist then issued a fresh download. slskd
appended ``_<19-digit-ns-timestamp>`` to each new copy because the
target file already existed, and the user ended up with seven copies
of one song in one folder.
Fix: extract two pure helpers — ``_normalize_album_for_match``
strips qualifier parentheticals (Music From X, OST, Deluxe Edition,
Remastered, Anniversary, etc.) and trailing dash-clauses;
``_albums_likely_match`` checks equality after normalization,
substring containment, and a relaxed 0.6 fuzzy ratio. A volume /
part / disc / standalone-trailing-number guard rejects pairs like
``"Greatest Hits Vol. 1"`` vs ``"Greatest Hits Vol. 2"`` so the
relaxed threshold doesn't introduce false positives on serialized
releases. After this change the Napoleon Dynamite case collapses
to ``"napoleon dynamite" == "napoleon dynamite"`` via the equality
short-circuit and the redownload loop dies.
2. The duplicate detector found only one of the seven dupe files.
The detector buckets tracks by the first 4 chars of their normalized
tag title. Files written by slskd directly into a library folder
often get inconsistent (or blank) tags from the media-server rescan,
so the seven copies were bucketed apart by parsed title and never
compared.
Fix: refactor the per-bucket comparison into ``_scan_bucket``, then
add a second pass — ``_build_filename_buckets`` re-buckets leftover
tracks by canonical filename stem (slskd dedup tail stripped via
``_strip_slskd_dedup_suffix``, same regex the import-cleanup PR uses)
plus extension. Filename agreement is itself strong evidence the
files came from the same source download, so the second pass calls
``_scan_bucket`` with ``require_metadata_match=False`` to skip the
title / artist / cross-album gates. The same-physical-file guard
still runs so bind-mount duplicates aren't flagged.
72 new regression tests across two files cover the album-match
helpers (28 tests including the Napoleon Dynamite scenario, 7 volume
disagreements, 8 positive/negative pairs, 5 defensive cases) and the
new filename-bucket pass (16 tests across bucket construction, scan
integration, and existing title-pass behavior). Full pytest 1509
passed; ruff clean.
Reported by Mushy in Discord.
slskd appends "_<19-digit unix-nanosecond timestamp>" to a downloaded
filename when the destination already contains a same-named file
(concurrent downloads of the same track, partial-file retries after a
connection drop, cancelled-then-redownloaded files, the same track
surfacing in multiple synced playlists). The file-finder code already
recognized the suffix when matching a download to its source — but
after the canonical file moved into the library, the leftover
"_<timestamp>" siblings sat orphaned in the downloads folder forever.
Reported on Discord by Shdjfgatdif.
cleanup_slskd_dedup_siblings() runs at the end of each successful
import (3 safe_move_file sites in pipeline.py) and prunes any
remaining siblings that strip down to the canonical stem with the
same extension. Conservative match (>= 18 trailing digits) keeps
legitimate filenames like "Track 5" and "Album 1995" untouched. Per-
file unlink failures are swallowed so a single locked file doesn't
block the rest.
17 regression tests cover the suffix-strip primitive, orphan removal,
no-op cases, mismatched extensions, subdirectories, and partial-failure
recovery.
Importing web_server fires utils.logging_config.setup_logging at
module-init, which clears + re-installs handlers on the shared
'soulsync' logger and pins its level to the user's configured
value. That mutation leaks across tests in the same pytest process.
This file runs alphabetically before test_library_reorganize_orchestrator,
so the leak broke test_watchdog_warns_about_stuck_workers downstream
— it relies on caplog capturing soulsync.library_reorganize warnings
via root-logger propagation, and the reconfigured logger's new
handler chain swallowed those records before they reached caplog
(caplog.records came back empty even though pytest's live-log
capture clearly showed the warning fired).
Adds an autouse fixture that snapshots the soulsync logger's
handlers, level, and propagate flag before each test in this file
and restores them afterwards. Pollution stays scoped to this file.
tests/test_tidal_auth_instructions.py also imports web_server but
runs alphabetically AFTER test_library_reorganize_orchestrator so
it never tripped this — fix is scoped here, not a project-wide
conftest, so we don't change behaviour for unrelated test files.
- Flatten the Spotify service-status rendering so it shows rate-limit and recovery states explicitly, while otherwise displaying the active metadata provider directly.
- Keep the Spotify auth controls and metadata-source picker aligned with the real session state after authenticate and disconnect flows.
- Return "Unmapped" for unknown metadata source labels instead of implying iTunes.
- Update the metadata registry tests to cover the new label fallback.
- Send Spotify auth completion back to the opener so the settings page refreshes immediately
- Make the local auth flow go straight through to Spotify instead of showing the temporary instruction page
- Keep the remote/docker instruction page available for manual callback setups
- Sync Spotify status, connect/disconnect buttons, and metadata source selection after auth and disconnect
- Keep the disconnect behavior aligned with the active primary metadata source
- Hide the auth button when a Spotify session is active
- Treat disconnect as a session change, not a provider swap
- Share metadata source labels in the registry
- Tighten rate-limit copy around Spotify-specific behavior
Discord-reported (fresh.dumbledore + maintainer ack): the
/api/import/singles/process route iterated staging files through a
plain Python for loop. Per-file work is dominated by metadata
search round-trips (Spotify/iTunes/Deezer/Discogs), so a multi-
track manual import on a typical home network was painfully slow.
Adds a dedicated import_singles_executor (3 workers) alongside the
existing executor pool, and refactors the route to submit every
file at once and aggregate results via as_completed. Worker count
balances throughput against any single provider's per-source rate
limits — the same shape used by missing_download_executor.
Extracts the per-file pipeline into _process_single_import_file
which returns a typed (status, payload) outcome:
- ("ok", final_title) on success
- ("error", message) for missing/malformed input or pipeline failure
The worker wraps its own exceptions so a single bad file can't
crash the batch; the route adds a belt-and-suspenders try/except
around future.result() for any worker-level surprises.
Pipeline thread-safety verified: post_process_matched_download
already serializes per-file via post_process_locks (one lock per
context_key — and each import gets a unique UUID context_key), DB
writes serialize through SQLite's WAL + busy_timeout, metadata
registry uses RLocks, no bare module-level mutable state.
Adds 9 regression tests:
- 4 worker-contract tests (missing file, malformed match, pipeline
exception wrapping, happy-path return shape)
- 2 executor-config tests (worker count, thread name prefix)
- 1 integration test that proves the route actually parallelizes
by checking wall-clock duration is well under sequential cost
- 1 mixed-outcome aggregation test
- 1 worker-crash recovery test
Doesn't address the related "stops on tab close" complaint —
that's a separate request-lifecycle issue that needs job_id +
polling, not just parallelism.
Discord-reported (winecountrygames + fresh.dumbledore): "Import only
makes Albums folder no singles or eps". Users with a
${albumtype}s/$albumartist/... album_path template saw an "Albums"
folder fill up correctly but never any "Singles" or "EPs" folder.
build_import_album_info detected an album using
``total_tracks > 1`` AND ``album_name != track_title``. Spotify
singles fail both — total_tracks is 1 and the album is usually
named after the song. The result was that staging/auto-import
routed singles through single_path, which doesn't honour
$albumtype, so the user's per-type folder layout never applied.
Now also treats the metadata source's explicit release-type
classification ("single", "ep", "compilation") as evidence that
this is an album-shaped release, so it routes through album_path
and the user's $albumtype substitution runs. The default fallback
value "album" is deliberately excluded from this check so
single-track downloads with no real metadata behave exactly as
before.
Adds 10 regression tests covering the reported scenario, EP and
compilation explicit types, and three guards: normal multi-track
albums still detected, default 'album' type falls through, and
empty/unknown types fall through.
Discord-reported: clicking the Tidal "Authenticate" button on a
Docker setup landed users on a remote-access instructions page that
told them their callback URL would look like
http://127.0.0.1:8888/tidal/callback?code=... — Spotify's port,
hardcoded into the Tidal instructions. Users who followed those
instructions literally saved 8888 into their tidal.redirect_uri
setting; that mismatched their Tidal Developer App's registered
:8889 redirect URI and Tidal returned error 1002 (invalid redirect
URI) on every auth attempt.
Pull the port from the actual TidalClient.redirect_uri the OAuth
URL was just built with (urlparse), with the SOULSYNC_TIDAL_CALLBACK_PORT
env var as fallback when the URI can't be parsed. Both the Step 2
example and the Step 3 highlighted URL now reflect whatever Tidal
port the user is actually configured to use.
Adds 3 regression tests covering the reported scenario, custom
callback ports via SOULSYNC_TIDAL_CALLBACK_PORT, and a defensive
fallback when redirect_uri is unparseable. Tests hit the real
/auth/tidal route through Flask's test client and assert the
rendered HTML, so future hardcoded ports get caught immediately.
The /api/library/watchlist-all-unwatched endpoint required the
user's currently active metadata source's ID column on each library
artist. A Spotify-primary user with library artists only matched
against iTunes or Deezer saw them silently skipped — surfacing on
Discord as "Library and Watchlist not syncing correctly". The per-
artist Enhanced View sync sometimes "fixed" them because it triggered
metadata enrichment that occasionally populated the missing Spotify
ID, but couldn't help artists Spotify simply doesn't carry.
Extracts the picker as a standalone helper so it can be tested
directly:
core/watchlist/source_picker.py:pick_artist_id_for_watchlist
Picks the active source first when available, then falls back through
spotify -> itunes -> deezer -> discogs in registration order. Empty
strings count as missing. Numeric IDs are coerced to str so SQLite's
TEXT columns store them in the same form library code reads back.
Returns (None, None) only when the artist has zero source IDs — the
only legitimate skip reason now.
Adds 10 regression tests covering active-source priority for each
supported primary, fallback ordering through every secondary, the
zero-IDs base case, unrecognized active source (e.g. hydrabase still
falls through), empty-string handling, and numeric coercion.
Discord-reported scenario: a single "Super Single" by Artist1 feat.
Artist2 is also on Artist1's "Super Album". When the album is fully
owned, Artist1's discography correctly shows the single as complete,
but Artist2's discography (where the same track also appears as a
single) shows it as missing.
Two layers needed for the fix:
Scanner: the Jellyfin/Emby path was keeping only ArtistItems[0],
which is almost always equal to the album artist — so the
distinguishing per-track credit was silently suppressed. Now joins
every ArtistItems entry with "; " and stores the value when there
are multiple credits OR when the single credit differs from the
album artist. Plex's originalTitle already carries the full multi-
artist tag, so Plex users benefit without needing the scanner change.
Scorer: _calculate_track_confidence now splits track_artist on the
common multi-artist delimiters real-world tags use (",", ";", "&",
"feat.", "ft.", "featuring", "vs.", "x") and scores each piece
independently against the search artist, taking the max along with
the whole-string similarity as the floor. Never reduces a score —
purely additive matching for previously-missed featured-artist
credits.
Adds 12 regression tests covering the reported scenario, primary-
artist back-compat, every delimiter variant (parametrized), no-
regression on exact match, and the scanner storing every ArtistItem.
Existing Jellyfin-scanned rows persist their old single-artist value
until the next library scan rewrites them; Plex rows benefit
immediately on next match without needing a rescan.
Re-enabling the previously-dead album-aware fallback could in
theory leak false positives if its 0.8 album-title floor were
ineffective. Pin the floor with a clearly-mismatched album hint
("Disney Hits" against "Ray of Light") and assert the search
returns no match. Distinct artist names with no shared words so
the main path actually fails through to the fallback (the prior
draft used "Different Artist" / "Real Artist" which both contain
"Artist" and scored above the main path's threshold, never
reaching the fallback at all).
Two bugs surfacing the same user-reported symptom: a Vaiana OST
track ("Where You Are" by Christopher Jackson) wouldn't match against
a Plex/Emby library because the album sits under the album artist
(Lin-Manuel Miranda).
Bug 1: the data was already there but scoring ignored it. The DB
schema has a tracks.track_artist column, the scanner populates it
from Plex's originalTitle and Jellyfin's ArtistItems[0], and the SQL
WHERE clause already searches it — but _rows_to_tracks dropped the
column on its way to the Python object, and _calculate_track_confidence
only scored against the album-artist JOIN. Candidates whose track-
artist matched got returned by the search and then immediately
filtered out by the low confidence score.
Fix: _rows_to_tracks now propagates row['track_artist'] onto the
returned object, and _calculate_track_confidence takes the better of
(album-artist similarity, track-artist similarity) so soundtracks
match through whichever credit the search query carries.
Bug 2: the album-aware fallback path constructed DatabaseTrack with
kwargs the dataclass doesn't accept (artist_name, album_title,
server_source). Every row TypeError'd, the outer except swallowed it
silently, and the fallback never matched anything since the column
was added — invisible because nothing logged it.
Fix: build DatabaseTrack with valid fields and attach the joined
columns afterwards, the same pattern _rows_to_tracks uses.
Adds 6 regression tests covering: track-artist match (the OST case),
album-artist still matches, scorer takes the better of the two,
defensive handling for tracks without track_artist, search-path
attribute propagation, and the previously-dead album-aware fallback.
The "Clean Search History" automation card kept showing a stale
'DownloadOrchestrator' object has no attribute 'base_url' error
even after the underlying handler bug was fixed in 77d20e9. Root
cause is in the engine, not that handler: AutomationEngine only
captured uncaught exceptions into last_error. Handlers that
report failure by RETURNING {'status': 'error', ...} were treated
as successful from the engine's perspective, so subsequent
gracefully-failing runs never updated the row to reflect the
current state.
Both the timer (run_automation) and event (_handle_event_trigger)
paths now extract the error string from a result whose status is
'error', falling through 'error' -> 'reason' -> 'message' -> a
placeholder so last_error is never None on actual failures
regardless of which key the handler chose. Existing behaviour for
raised exceptions and successful runs is preserved.
Also normalizes _auto_clean_search_history's return key from
'reason' to 'error' so older deployed engines that only check
the canonical key still see the failure.
Adds 7 regression tests covering every result shape the engine
might receive.
The fixture used the wrong env var name (SOULSYNC_DB_PATH) when trying
to redirect ConfigManager at a tmp directory. ConfigManager actually
reads DATABASE_PATH (config/settings.py:49), so the test ConfigManager
loaded — and then saved — at the user's real database/music_library.db.
The retry stub in test_lock_errors_during_retries_log_at_debug_not_error
calls the real _save_to_database after its mocked failures, which then
clobbered the encrypted app_config row with the test fixture's stub
payload {"plex": {"base_url": "http://example.test"}}.
Three layers of fix so this can't happen again:
- Use the correct env var (DATABASE_PATH).
- Pin mgr.database_path / mgr.config_path on the instance after
construction, so the test fixture's tmp paths win even if
ConfigManager's resolution logic changes.
- Assert the resolved database_path is rooted under tmp_path before
returning the fixture, so the test refuses to run if it would touch
a non-tmp DB.
When users bind the same host music directory into both SoulSync
(e.g. /app/Transfer) and a media server like Plex (e.g.
/media/Music), both scans add a track row pointing at the same
physical file via different mount paths. The detector previously
flagged those as duplicate groups even though there's only one
file on disk.
New _is_same_physical_file helper filters pairs where:
- The trailing 3 path segments match (filename + album + artist
folder), so they're the same release on disk.
- The leading mount roots actually differ.
- Durations agree within 1s when both rows carry duration data.
Adds 10 regression tests covering the reported scenario plus
edge cases (Windows separators, case differences, missing
durations, sibling-album false-positive guard).
Pin the new save-retry contract so future changes can't silently
re-introduce the spam reported in #434:
- Happy-path saves emit zero ERROR logs.
- Transient locks during retries log at DEBUG, not ERROR.
- Six attempts run before giving up, with the documented backoff
schedule (0.2 + 0.5 + 1.0 + 2.0 + 4.0s).
- Genuine exhaustion logs a single ERROR and writes config.json.
- sqlite3.OperationalError("database is locked") routes to DEBUG;
any other OperationalError still logs ERROR.
- _connect_db() actually applies WAL + busy_timeout + synchronous=NORMAL.
Also moves `import time` from inside _save_config to the module
top so the tests can monkeypatch sleep cleanly.
- normalize album.total_tracks before comparing it in wishlist classification
- avoid mixed-type comparisons when provider payloads serialize track counts as strings
- add regression coverage for numeric strings and invalid values
- carry track-level album art through the quality scanner normalization path
- preserve artist artwork when provider results expose it
- keep album.image_url and album.images populated so the wishlist UI can render the cover consistently
- add a regression test covering provider payloads with image_url on both the track and artist
- search metadata providers in source-priority order for each generated query instead of caching one client for the whole scan
- keep the quality-scanner worker provider-neutral and preserve the no-provider error path
- update the quality-scanner tests and remove the obsolete web_server spotify_client injection
- Switch the download lifecycle over to the neutral wishlist track helper name
- Keep the old Spotify helper as a compatibility alias for older callers
- Store track_data as the primary failed-download wishlist payload key and add regression coverage
- Let the wishlist service accept both track_data and spotify_track_data
- Preserve the backward-compatible wrapper while avoiding the keyword argument crash
- Add a regression test for the alias path
- add neutral wishlist payload helpers while keeping legacy Spotify aliases
- route wishlist removal and classification through generic track data
- keep API and service compatibility for existing callers
Final lift in the web_server.py extraction effort. Pulls two route
handlers + one background worker out of `web_server.py` into new
focused packages:
- `core/streaming/prepare.py` — 258-line stream-prep worker that
downloads a track to the local Stream/ folder for the browser audio
player.
- `core/playlists/explorer.py` — 305-line route handler for
`POST /api/playlist-explorer/build-tree` that streams an NDJSON
discography tree from a mirrored playlist.
What `prepare_stream_task` does:
1. Reset stream state to 'loading' with the new track info.
2. Clear any prior file from Stream/ (only one stream lives there).
3. Spin up a fresh asyncio event loop and `soulseek_client.download()`.
4. Poll progress every 1.5s. Queue timeout 15s; overall 60s.
5. On succeeded + bytes-match: find the file with retry, move into
Stream/, signal slskd completion, mark state 'ready' with file_path.
6. On error/timeout/cancel: state goes to 'error' or 'stopped'.
7. Finally: tear down the event loop cleanly.
What `playlist_explorer_build_tree` does:
1. Validate request, load playlist + tracks from DB.
2. Pick active metadata source (Spotify if authed, else fallback).
3. Group tracks by artist using discovered matched_data when the
provider matches the active source.
4. Stream NDJSON: meta line → one artist line per group → complete line.
5. Per artist: cache check → resolve discography → tag releases with
`in_playlist` flag based on title-similarity match → filter by mode
(`albums` = only matches; `discographies` = full disco).
6. Mark playlist as explored on completion.
Strict 1:1 byte parity:
Both functions exposed their dependencies through proxy patterns
established in earlier lifts (PR4–PR8). For prepare_stream_task,
`stream_state` is a deps property; for the explorer, Flask `request` /
`jsonify` / `Response` are injected via deps so the lifted body keeps
its native syntax. Both lifts verified ZERO diff against the original
after `deps.X` → global X normalization.
258 lines orig = 258 lines lifted (prepare_stream_task).
305 lines orig = 305 lines lifted (explorer).
Bonus cleanup: web_server.py's module-level `import shutil` and
`import glob` were now unused (only `_prepare_stream_task` used them
at module scope; every other reference is via inline `import shutil`
in respective function bodies). Removed both module-level imports —
ruff caught the F811 redefinitions and confirmed they're truly
redundant.
Dependencies for `PrepareStreamDeps` (11 fields):
config_manager, soulseek_client, stream_lock, project_root,
docker_resolve_path, find_streaming_download_in_all_downloads,
find_downloaded_file, extract_filename, cleanup_empty_directories,
plus 2 stream_state property delegates.
Dependencies for `PlaylistExplorerDeps` (9 fields):
Flask request/Response/jsonify, spotify_client, get_database,
get_active_discovery_source, get_metadata_fallback_client,
get_metadata_fallback_source, get_metadata_cache.
Tests: 6 new under tests/streaming/test_prepare.py (state init,
Stream/ folder creation + clearing, download-init failure, completed
+ moved + ready state, partial-bytes incomplete-warning path) plus 9
new under tests/playlists/test_explorer.py (5 validation early-exit
paths, streaming response shape with meta/complete lines, mark-
explored side effect, discovered-artist grouping using matched_data,
provider mismatch falling back to raw artist name).
Full suite: 1355 passing (was 1340). Ruff clean.
End of the web_server.py extraction effort. Started at ~45,000 lines
across PR4–PR8 + this commit; finished around 35,000 lines with the
heavy worker + route logic now living in domain-cohesive packages
under core/. The remaining bulk in web_server.py is route handlers,
service initialization, and the deferred 1530-line
`_register_automation_handlers` (startup-only, marginal lift value).
Pulls the 284-line artist quality enhancement helper out of
`web_server.py` into a new `core/artists/` package. Flask route handler
split: route + request parsing stay in web_server.py, the body lifts to
a pure function returning `(payload_dict, http_status_code)`.
What `enhance_artist_quality` does:
1. Validate request: track_ids must be non-empty, artist must exist.
2. Build a `track_lookup` from `database.get_artist_full_detail` so each
selected track resolves with its album context.
3. Per track:
- Read current quality tier from the file extension.
- Build `matched_track_data` for the wishlist entry, in priority
order:
- Spotify direct lookup via stored `spotify_track_id` (preferred).
Uses raw API data when available; otherwise rebuilds the payload
and pulls album images via a follow-up `get_album` call.
- Spotify search fallback using matching_engine queries with
artist+title similarity scoring (album-type bonus for albums,
smaller bonus for EPs). Stops at first >= 0.9 confidence match.
- iTunes/fallback source search with the same scoring shape.
- Add to wishlist via `wishlist_service.add_spotify_track_to_wishlist`
with `source_type='enhance'` and a `source_context` carrying the
original file path, format tier, bitrate, original_tier, and
artist_name.
- Tally `enhanced_count` / `failed_count` / per-track failure reasons.
4. Return `{success, enhanced_count, failed_count, failed_tracks}` 200.
Dependencies injected via `ArtistQualityDeps` (7 fields) — spotify_client,
matching_engine, get_database, get_wishlist_service,
get_current_profile_id, get_quality_tier_from_extension,
get_metadata_fallback_client.
Diff vs original after `deps.X` → global X normalization is **1 line of
cosmetic drift** — the success return now uses an explicit `(payload, 200)`
tuple to keep all returns shape-consistent for the wrapper. Flask treats
`jsonify(x)` and `(jsonify(x), 200)` identically. 284 lines orig = 285
lines lifted, body otherwise byte-identical.
Tests: 10 new under tests/artists/test_quality.py covering input
validation (empty track_ids, artist not found), Spotify direct lookup
via raw_data, Spotify direct lookup with enhanced format requiring
album image rebuild, Spotify search fallback, iTunes/fallback source
match path, track-not-found and no-file-path failure modes, complete
no-match failure, and source_context payload assertions (enhance flag,
file path, format tier, bitrate, source_type).
Full suite: 1340 passing (was 1330). Ruff clean.
Pulls the 258-line retag worker out of `web_server.py` into a new
`core/library/` package. Pure 1:1 lift — wrapper keeps the original
entry-point name so the retag-trigger endpoint continues to work
without changes.
What `execute_retag` does:
1. Fetch album + track metadata for the new `album_id` (Spotify or
iTunes — the Spotify client transparently falls back).
2. Load existing files in the retag group from the DB.
3. Match each existing track to a new Spotify track:
- Priority 1: same disc + track number.
- Priority 2: title similarity >= 0.6 (SequenceMatcher).
4. For each matched pair:
- Re-write metadata tags via `_enhance_file_metadata`.
- Compute the new path via `_build_final_path_for_track` and move
the audio file (plus .lrc / .txt sidecars) if the path changes.
- Drop an orphaned cover.jpg if it's left in an empty directory.
- Clean up empty parent directories left behind.
- Download the new cover art into the new album dir.
5. Update the retag group record with new artist / album / image /
total_tracks / release_date and the appropriate Spotify-or-iTunes
album ID (numeric → iTunes, alphanumeric → Spotify).
6. Mark the retag state 'finished' (or 'error' on exception).
Strict 1:1 byte parity:
The original mutated `retag_state` as a module global (the function
declared `global retag_state` even though it only mutates in place).
Here `retag_state` is exposed through the `RetagDeps` proxy as a Python
property so the lifted body keeps `name[key] = value` /
`name.update(...)` syntax. The property setter rebinds the
web_server.py reference if the function ever reassigns it (currently
it doesn't, but the setter is wired for parity with the watchlist lift).
Diff vs original after `deps.X` → global X normalization is **zero
differences** apart from the dropped `global retag_state` decl and the
inline `from database.music_database import get_database` (replaced by
deps.get_database()). 258 lines orig = 258 lines lifted, byte-identical
body otherwise.
Dependencies injected via `RetagDeps` (13 fields) — config_manager,
retag_lock, spotify_client, plus 8 callable helpers
(get_audio_quality_string, enhance_file_metadata,
build_final_path_for_track, safe_move_file, cleanup_empty_directories,
download_cover_art, docker_resolve_path, get_database) and 2 property
delegates (_get_retag_state / _set_retag_state).
Tests: 11 new under tests/library/test_retag.py covering setup error
paths (no album data, no album tracks, no existing tracks),
track-number priority match, title-similarity fallback, no-match skip,
missing file skip, file move when path changes, group record update
(spotify vs iTunes ID branching by alphanumeric vs numeric album_id),
multi-disc total_discs computation.
Full suite: 1330 passing (was 1319). Ruff clean.
Pulls the 390-line watchlist auto-scan orchestrator out of `web_server.py`
into a new `core/watchlist/` package. Watchlist (followed-artists scanner
that finds new releases) is a separate domain from kettui's wishlist
(failed-download retry queue), so this lift does not overlap with the
ongoing PR400-style extractions.
What `process_watchlist_scan_automatically` does:
1. Smart stuck-detection guard before acquiring the timer lock —
prevents deadlock when a previous scan flag is dangling past the
2-hour timeout.
2. Inside the timer lock: re-check + set the active scan flag with the
current timestamp.
3. Per-profile expansion (or single-profile when manually triggered):
- Watchlist count check + Spotify auth gate.
- Backfill missing artist images.
4. Initialize a fresh `watchlist_scan_state` dict (the deps property
setter rebinds the web_server.py module-level name so external
sentinel checks via id() comparison still detect the swap).
5. Pause enrichment workers, then call
`WatchlistScanner.scan_watchlist_artists` with a per-event progress
callback that translates scanner events into automation log lines.
6. Post-scan steps (skipped if the scan was cancelled mid-flight):
- Populate discovery pool from similar artists (per-profile).
- Refresh ListenBrainz playlists.
- Update current seasonal playlist (weekly cadence).
- Generate Last.fm radio playlists.
- Sync Spotify library cache.
- Activity feed entry + automation_engine.emit('watchlist_scan_completed').
7. On exception: mark state['status']='error', re-raise so the
automation wrapper records the failure.
8. Finally: resume enrichment workers, clear the scanner's rescan
cutoff, reset the auto-scanning flag.
Strict 1:1 byte parity:
The original mutated `watchlist_auto_scanning`,
`watchlist_auto_scanning_timestamp`, and `watchlist_scan_state` as
module globals (with a leading `global` decl). Here those names are
exposed through the `WatchlistAutoScanDeps` proxy as Python properties
so the lifted body keeps the same `name = value` / `name[key] = value`
shape. Property setters fan writes back to web_server.py via callback
pairs.
Diff vs original after `deps.X` → global X normalization is **zero
differences** apart from the dropped `global` declaration line — Python
doesn't need it once the names are property accesses on the deps object.
390 lines orig = 390 lines lifted, byte-identical body otherwise.
Dependencies injected via `WatchlistAutoScanDeps` (15 fields total) —
Flask app, spotify_client, automation_engine, watchlist_timer_lock, plus
5 callable helpers and 6 property delegate callbacks (paired
get/set for each of the three globals).
Tests: 11 new under tests/watchlist/test_auto_scan.py covering
stuck-detection guard, race-check inside lock, zero-watchlist short-
circuit, unauthenticated Spotify gate, successful scan with all post-
scan steps, automation event emission, activity feed logging,
cancellation mid-scan skipping post-steps, profile-scoped trigger,
flag reset in finally, rescan cutoff clear in finally.
Full suite: 1319 passing (was 1308). Ruff clean.
- let core.metadata.registry own per-profile Spotify client caching
- register the DB-backed profile credentials provider from web_server.py
- invalidate only the affected profile cache entry on save, delete, and auth
- split metadata lookup logic into core/metadata/*
- keep core/metadata_service.py as the legacy barrel
- update tests and artist-detail code to patch concrete modules
Pulls the 201-line staging-folder shortcut out of `web_server.py` into
its own module under the existing `core/downloads/` package. Pure 1:1
lift — wrapper keeps the original entry-point name so the task worker's
existing call site continues to work without changes.
What `try_staging_match` does:
1. Pull the per-batch staging-file cache (one filesystem scan per batch).
2. For each staging entry, compute title + artist similarity using
SequenceMatcher and the matching engine's `normalize_string`. Require
title >= 0.80, then a combined score >= 0.75. The weighting flips
based on whether artist info is available on both sides:
- both have artist: 0.55*title + 0.45*artist
- either side missing artist: 0.80*title + 0.20*artist (lean on title)
3. Copy the matched file to the configured transfer dir (with a
"_staging" suffix when the destination filename already exists, to
avoid overwriting a legitimate prior download).
4. Mark the task as 'post_processing', username='staging',
staging_match=True.
5. Build a synthetic spotify_artist / spotify_album context (mirroring
the modal-worker logic so the file-organization template applies
cleanly) and store it under "staging_<task_id>". Two paths:
- Explicit context branch (track_info has _is_explicit_album_download)
→ real album/artist data copied through.
- Fallback branch → synthesized from track + track_info, with
`is_album_download` heuristically derived (album differs from title
and isn't "Unknown Album").
6. Hand off to `_post_process_matched_download_with_verification` which
does tagging, path building, AcoustID verification, and DB insertion.
Returns True if the staging shortcut won; False to fall through to the
normal Soulseek search path.
Dependencies injected via `StagingDeps` (5 fields) — config_manager,
matching_engine, get_staging_file_cache, docker_resolve_path,
post_process_matched_download_with_verification.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 201 lines orig = 201 lines lifted, byte-identical body
(including all whitespace, comments, log strings, and the inline
`from difflib import SequenceMatcher` / `import shutil` imports inside
the function body).
Tests: 9 new under tests/downloads/test_downloads_staging.py covering
no staging files / no track title / low-confidence match returning
False, exact match copying file + transitioning task state + invoking
post-processing, existing-file rename via `_staging` suffix, explicit
album context branch, fallback context synthesis (with both album-as-
album and album-equals-title cases), and copy failure (missing source
file) returning False.
Full suite: 1308 passing (was 1299). Ruff clean.
Missed worker from the PR5 discovery-workers series — Tidal sits in the
same domain as the deezer / spotify_public / listenbrainz / youtube /
beatport workers that were lifted in PR5b–PR5h, follows the same shape,
shares the same `_search_spotify_for_tidal_track` helper, and was simply
overlooked in the original inventory.
Pure 1:1 lift of the 212-line worker. Wrapper keeps the original
entry-point name so the existing call sites in web_server.py continue
to work without changes.
What `run_tidal_discovery_worker` does:
1. Pause enrichment workers (release shared resources).
2. For each Tidal track:
- Cancellation gate (state['cancelled']).
- Discovery cache lookup; cache hit short-circuits the search.
- SimpleNamespace-style track passed straight to
`_search_spotify_for_tidal_track` (the shared helper used by every
worker in this family).
- On Spotify match: build `match_data` preserving track_number /
disc_number from raw API data, image extracted from album images
or track object fallback, release_date filled from
track.release_date when album dict is missing it.
- On iTunes match: dict result populated as `match_data` with source
set to discovery_source, image extracted from album images.
- Save matched result to discovery cache.
- On miss: Wing It stub stored as 'wing-it' status (success ticked).
3. After all tracks: phase='discovered', activity feed entry, sync
discovery results back to mirrored playlist via
`_sync_discovery_results_to_mirrored` with 'tidal' tag.
4. On error: state['phase']='error' + status with error string.
5. Finally: resume enrichment workers.
Dependencies injected via `TidalDiscoveryDeps` (13 fields) —
tidal_discovery_states, spotify_client, plus 11 callable helpers
(pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, get_discovery_cache_key, get_database,
validate_discovery_cache_artist, search_spotify_for_tidal_track,
build_discovery_wing_it_stub, add_activity_item,
sync_discovery_results_to_mirrored). Same surface as the deezer worker.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 212 lines orig = 212 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 9 new under tests/discovery/test_discovery_tidal.py covering
cache hit short-circuit, Spotify tuple match (track/disc preservation),
iTunes dict match path, Wing It fallback, cancellation, completion
phase update, activity feed entry, mirrored sync invocation, per-track
error handling.
Full suite: 1299 passing (was 1290). Ruff clean.
First lift in the new PR6 batch. Pulls the 312-line candidate-fallback
download dispatcher out of `web_server.py` into a new module under the
existing `core/downloads/` package. Pure 1:1 lift — wrapper keeps the
original entry-point name so all callers (search/match pipeline) work
unchanged.
What `attempt_download_with_candidates` does:
1. Sort candidates by descending confidence.
2. For each candidate:
- Cancellation gates (3 points: top of loop, before download starts,
after download_id is assigned).
- Skip already-tried sources via the per-task `used_sources` set.
- Skip blacklisted sources (user-flagged bad matches).
- Race protection: bail when the task already has an active
download_id.
- `update_task_status('downloading')`, then `soulseek_client.download`.
3. On a successful download_id:
- Build `matched_downloads_context` entry keyed by
`make_context_key(username, filename)`.
- For tracks with clean Spotify metadata, pull track_number /
disc_number from (1) track_info → (2) track object → (3) Spotify
API call. When local album context is incomplete, the API response
backfills release_date / album_type / total_tracks / images / id.
- Set `is_album_download` based on explicit context flag or
heuristic (album differs from title, isn't "Unknown Album").
- Store task/batch IDs and track_info on the context for post-
processing + playlist-folder mode.
4. On a cancellation that wins the race after the download started:
- `cancel_download(...)` to stop the in-flight Soulseek transfer.
- `on_download_completed(batch_id, task_id, success=False)` to free
the worker slot.
5. On exception or download-start failure: reset task status to
'searching', continue to next candidate.
Dependencies injected via `CandidatesDeps` (7 fields) — soulseek_client,
spotify_client, run_async, get_database, update_task_status,
make_context_key, on_download_completed.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 312 lines orig = 312 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 14 new under tests/downloads/test_downloads_candidates.py
covering happy path (first candidate succeeds, confidence ordering),
used_sources dedup, blacklist skip, cancellation gates (cancelled
status, deleted task, active download_id, mid-flight cancel + cleanup
callback), failure paths (all candidates failed, exception during
download falls through to next), context payload (explicit album
context, track_number priority order, API backfill of incomplete album
metadata), and equal-confidence stable order.
Pre-existing behavior documented in tests:
`spotify_album_context['id']` initializes to a non-empty placeholder
'from_sync_modal' in the fallback path, so the API-backfill condition
`if not spotify_album_context.get('id')` never fires for the id field
specifically. Other album fields (release_date, album_type) backfill
fine because they default to empty.
Full suite: 1290 passing (was 1276). Ruff clean.
User report: multi-disc albums on the latest dev had literal "\$cdnum"
in their filenames instead of the expected "CDxx" label, plus a
redundant "Disc N" folder on top of the in-filename label.
Two bugs in core/imports/paths.py:
1. _replace_template_variables (the substitution helper used by every
download path builder) had no handling for \$cdnum or \${cdnum}. The
matching helper in web_server.py and core/repair_jobs/library_reorganize.py
did the substitution; this one didn't, so production downloads passed
the placeholder through unchanged. Added a cdnum_value computation
(CD%02d when total_discs > 1, empty otherwise) plus the corresponding
bracket_map entry and \$cdnum replace before \$track (matches the
ordering in the other path builders).
2. The album-path branch of build_final_path_for_track auto-injected a
"Disc N" folder whenever total_discs > 1, suppressed only when the
template contained \$disc. Templates using \$cdnum (or \${disc} /
\${discnum} / \${cdnum}) got both a "CDxx" label in the filename and
the auto folder. Widened the user_controls_disc check to cover all
the disc-bearing placeholders.
Bonus cleanup along the way:
- Folder-part stripping now drops a leading \$cdnum token (mirrors the
existing \$disc / \$discnum / \$quality strip — defensive against an
empty cdnum landing alone in a folder segment).
- Filename cleanup now strips a leading " - " left behind when \$cdnum
expands to empty on a single-disc album (mirrors the same regex in
library_reorganize.py).
- album_template config access switched from the dotted-path key to the
nested-dict access pattern used by the rest of the function — handles
both production config_manager and the flat _Config used in tests.
Tests: 4 new under tests/imports/test_import_paths.py
- multi-disc cdnum substitution produces "CD02"
- single-disc cdnum collapses to empty
- folder-part containing only \$cdnum is dropped
- build_final_path_for_track with \$cdnum template produces no auto
"Disc N" folder
Full suite: 1276 passing (was 1272). Ruff clean.
`test_demux_flac_uses_tools_dir_fallback` hard-coded `tools_dir / "ffmpeg"`
in its `fake_exists` stub, but `_demux_flac` looks for `ffmpeg.exe` on
Windows (os.name == 'nt'). Result: the fake_exists stub never matched,
the code fell through to the "ffmpeg is required" RuntimeError instead
of the expected "ffmpeg failed" subprocess error, and the test failed
on Windows. Linux CI passed because os.name == 'posix' uses bare
"ffmpeg".
Pick the binary name based on `os.name` to match what `_demux_flac`
actually probes for. Asserts on the matching candidate path.
Tests: 20 passing on Windows (was 19/20). Ruff clean.
Both the auto and manual wishlist download paths called
`remove_tracks_already_in_library` before submitting the batch — a
serial DB lookup per track per artist (~1s/track on a 24-track
wishlist). The batches set `force_download_all=True` which is
explicitly documented as "skip the expensive library check" — the
pre-flight cleanup was contradicting that flag.
Removed the cleanup call from both flows. Kept `remove_wishlist_duplicates`
(fast SQL DELETE) and the standalone `/api/wishlist/cleanup` endpoint
that exposes the library scan as explicit user-triggered maintenance.
Safety check on the trade-off:
- post-processing at `core/imports/pipeline.py:576-624` already handles
re-downloads defensively: existing file with metadata → skip overwrite
+ delete source duplicate, no library corruption.
- Master worker's analysis loop normally removes wishlist entries for
found tracks via `_check_and_remove_track_from_wishlist_by_metadata`,
so stale wishlist entries should be rare in practice.
- Worst case for the rare orphan: one redundant download attempt that
the post-processing layer no-ops on. Bandwidth waste, not data damage.
Tests updated:
- `..._does_not_run_library_cleanup` (renamed from `_skips_enhance_tracks_during_cleanup`)
asserts no DB track-existence checks happen and no wishlist removals
fire — both `enhance` and "owned" tracks reach the master worker.
- `..._marks_batch_complete_when_wishlist_genuinely_empty` (renamed from
`..._after_cleanup`) covers the path where the wishlist starts empty.
Full suite: 1232 passing. Ruff clean.
The manual wishlist download endpoint blocked the request thread on a
slow library-cleanup pass before submitting the batch — for a 24-track
wishlist that's ~50 per-track DB lookups serialised in the request
handler, taking 30+ seconds before the frontend got a response. The
modal sat at "Pending..." with no progress visible the whole time.
Split start_manual_wishlist_download_batch into:
1. SYNC path (request handler):
- Generate batch_id, create download_batches entry with phase=analysis
and analysis_total=0 placeholder.
- Submit a single bg job (`_prepare_and_run_manual_wishlist_batch`) to
the missing-download executor.
- Return 200 with batch_id immediately. Frontend can start polling
/api/active-processes status right away.
2. BG path (executor thread):
- db.remove_wishlist_duplicates (slow-ish, single SQL)
- remove_tracks_already_in_library (the slow one — per-track DB checks)
- wishlist_service.get_wishlist_tracks_for_download
- sanitize + dedupe + filter (track_ids / category)
- Update batch.analysis_total with the real filtered count
- add_activity_item("Wishlist Download Started", ...)
- run_full_missing_tracks_process (master worker)
Edge case: if cleanup empties the wishlist, the bg job marks the batch
phase='complete' with error='No tracks in wishlist' (instead of the old
synchronous 400 response). Frontend status poll picks this up and the
modal can close cleanly.
Tests: existing 2 manual-download tests updated to drive the bg job
explicitly via a new `_run_submitted_bg_job` helper. Added 2 new tests:
- `..._returns_immediately_with_placeholder` — proves the sync path
doesn't trigger any cleanup or master-worker calls; analysis_total=0.
- `..._marks_batch_complete_when_wishlist_empty_after_cleanup` —
cleanup empties the list, master worker never invoked, batch ends
with phase='complete'.
Full suite: 1232 passing (was 1230). Ruff clean.
Final lift in the PR5 discovery-workers series. Pulls the 328-line
library quality scanner out of `web_server.py` into its own focused
module under `core/discovery/`. Pure 1:1 lift — wrapper keeps the
original entry-point name.
What the quality scanner does:
1. Reset scanner state (counters, results), load quality profile +
minimum acceptable tier from QUALITY_TIERS.
2. Load tracks from DB based on scope:
- 'watchlist' → tracks for watchlisted artists only.
- other → all library tracks.
3. For each track:
- Stop-request gate (state['status'] != 'running').
- Quality-tier check via _get_quality_tier_from_extension(file_path).
- Skip tracks meeting standards (tier_num <= min_acceptable_tier).
- For low-quality tracks: matching_engine search query gen, score
candidates against Spotify (artist + title similarity, album-type
bonus), pick best match >= 0.7 confidence.
- On match: add full Spotify track to wishlist via
`wishlist_service.add_spotify_track_to_wishlist` with
source_type='quality_scanner' and a source_context that captures
original file_path, format tier, bitrate, and match confidence.
4. After all tracks: status='finished', progress=100, activity feed
entry, emit `quality_scan_completed` event for automation engine.
5. On critical exception: status='error', error message captured.
Wishlist service interaction is via the public
`add_spotify_track_to_wishlist` API only — no overlap with kettui's
planned `core/wishlist/` package extraction (the import lives inside
the function, exactly as in the original, and will follow whatever
path that package takes).
Dependencies injected via `QualityScannerDeps` (8 fields) —
quality_scanner_state dict, quality_scanner_lock, QUALITY_TIERS
constant, spotify_client, matching_engine, automation_engine, plus 2
callable helpers (get_quality_tier_from_extension, add_activity_item).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 328 lines orig = 328 lines lifted, byte-identical body
(including all whitespace, comments, log strings, and the inline
`from core.wishlist_service import get_wishlist_service` /
`from database.music_database import MusicDatabase` imports at the
top of the function).
Tests: 11 new under tests/discovery/test_discovery_quality_scanner.py
covering state init/reset, no-watchlist-artists short-circuit,
unauthenticated Spotify error, high-quality skip, low-quality search
trigger, match → wishlist add (with full source_context payload),
no-match no-add, mid-loop stop request, completion phase + progress,
automation engine event emission, all-library scope load.
Full suite: 1152 passing (was 1141). Ruff clean.
End of the PR5 series — `web_server.py` lost ~328 lines on this commit
alone; total trim across PR5a–PR5h is ~2,400 lines of discovery worker
code moved into focused `core/discovery/*.py` modules. The remaining
discovery-adjacent worker `_process_watchlist_scan_automatically` was
deliberately deferred to avoid overlap with kettui's planned wishlist
extraction.
Seventh lift in the PR5 discovery-workers series. Pulls the 286-line
ListenBrainz discovery worker out of `web_server.py` into its own
focused module under `core/discovery/`. Pure 1:1 lift — wrapper keeps
the original entry-point name.
What the ListenBrainz discovery worker does:
1. Pause enrichment workers (release shared resources).
2. For each ListenBrainz track:
- Cancellation gate (state['phase'] != 'discovering').
- Discovery cache lookup; cache hit short-circuits the search.
- Strategy 1: matching_engine search queries with confidence scoring
against Spotify (preferred) or iTunes (fallback).
- Strategy 2: swapped artist/title query.
- Strategy 3: album-based query (uses album_name when available —
unique to LB, since YouTube tracks don't have album metadata).
- Strategy 4: extended search with limit=50.
- On match → save to discovery cache with image extracted from album
images or matched_track.image_url fallback.
- On miss → Wing It stub stored as 'wing-it' status.
3. After all tracks: phase='discovered', status='complete', activity feed
entry mentioning 'ListenBrainz Discovery Complete'.
4. On error: state['status']='error', phase='fresh'.
5. Finally: resume enrichment workers.
Dependencies injected via `ListenbrainzDiscoveryDeps` (16 fields) —
listenbrainz_playlist_states, spotify_client, matching_engine, plus 13
callable helpers (pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, get_discovery_cache_key, get_database,
validate_discovery_cache_artist, extract_artist_name,
spotify_rate_limited, discovery_score_candidates, get_metadata_cache,
build_discovery_wing_it_stub, add_activity_item).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 286 lines orig = 286 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Pre-existing bug preserved (not fixed): if `listenbrainz_playlist_states[
state_key]` raises KeyError on entry, the outer except handler tries to
mutate `state` which is unbound → secondary UnboundLocalError. Same bug
in the original (and the YouTube discovery worker). Documented here for
future cleanup but out of scope for the lift.
Tests: 11 new under tests/discovery/test_discovery_listenbrainz.py
covering cache hit short-circuit, Strategy 1 confidence match, Wing It
fallback, iTunes fallback (Spotify unauthenticated and rate-limited),
cancellation (phase change), completion phase update, activity feed
entry, per-track error handling, float duration_ms tolerance (regression
for the :02d format crash fixed earlier), enrichment workers resume on
finally.
Full suite: 1141 passing (was 1130). Ruff clean.
Sixth lift in the PR5 discovery-workers series. Pulls the 323-line
Beatport chart discovery worker out of `web_server.py` into its own
focused module under `core/discovery/`. Pure 1:1 lift — wrapper keeps
the original entry-point name.
What the Beatport discovery worker does:
1. Pause enrichment workers (release shared resources).
2. For each Beatport track:
- Cancellation gate (state['phase'] != 'discovering').
- Clean Beatport text (artist/title) of common annotations via
`clean_beatport_text` helper.
- Single-string artist normalization for "CID,Taylr Renee"-style
entries — split on comma, take the first.
- Discovery cache lookup; cache hit short-circuits the search and
normalizes cached artists from ['str'] → [{'name': 'str'}] to
match the frontend's expected list-of-objects shape.
- matching_engine search-query generation (with high min_confidence
of 0.9 to avoid bad matches).
- Strategy 1: scored candidates from initial Spotify/iTunes searches.
- Strategy 4: extended search with limit=50 if no high-confidence
match found.
- On Spotify match: format artists as [{'name': str}] objects, pull
full album object from raw cache when available, fallback to
reconstructed album dict otherwise.
- On iTunes match: format with image_url-derived album.images entry
(300x300 spec), source set to discovery_source.
- Save matched result to discovery cache when confidence >= 0.75
(note: lower than search threshold; discovery still benefits from
these less-confident matches as user-visible suggestions).
- On miss: Wing It stub stored as 'wing-it' status (success ticked).
3. After all tracks: phase='discovered', activity feed entry, sync
discovery results back to mirrored playlist via
`_sync_discovery_results_to_mirrored` with 'beatport' tag.
4. On error: state['phase']='fresh' + status='error'.
5. Finally: resume enrichment workers.
Dependencies injected via `BeatportDiscoveryDeps` (17 fields) —
beatport_chart_states, spotify_client, matching_engine, plus 14
callable helpers (pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, clean_beatport_text,
get_discovery_cache_key, get_database, validate_discovery_cache_artist,
spotify_rate_limited, discovery_score_candidates, get_metadata_cache,
build_discovery_wing_it_stub, add_activity_item,
sync_discovery_results_to_mirrored).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 323 lines orig = 323 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 12 new under tests/discovery/test_discovery_beatport.py covering
cache hit short-circuit (with cached-artist normalization), Spotify
match formatting (list and string artist inputs), iTunes match
(image_url to album.images), Wing It fallback, cancellation
(phase change), completion phase update, activity feed entry, mirrored
sync invocation, top-level error handler, per-track error handling,
comma-separated artist split.
Full suite: 1130 passing (was 1118). Ruff clean.
Fifth lift in the PR5 discovery-workers series. Pulls the 278-line
public-Spotify-link discovery worker out of `web_server.py` into its
own focused module under `core/discovery/`. Pure 1:1 lift — wrapper
keeps the original entry-point name.
What the Spotify Public discovery worker does:
1. Pause enrichment workers (release shared resources).
2. For each track:
- Cancellation gate (state['cancelled']).
- Normalize artists to plain string list (handles dict + str inputs).
- Discovery cache lookup; cache hit short-circuits the search and
populates display fields from the cached match.
- SimpleNamespace duck-type → `_search_spotify_for_tidal_track`
(shared search helper, returns tuple for Spotify or dict for iTunes).
- On Spotify match: build `match_data` preserving track_number /
disc_number from raw API data; image extracted from album images
or track object fallback; release_date filled from track.release_date
when album dict is missing it.
- On iTunes match: dict result → match_data with source set to
discovery_source; image extracted from album images.
- Save matched result to discovery cache.
- On miss: Wing It stub stored as 'wing-it' status.
3. After all tracks: phase='discovered', activity feed entry.
4. On error: state['phase']='error' + status with error string.
5. Finally: resume enrichment workers.
This worker is structurally close to the Deezer worker (see PR5d) but
intentionally diverges on:
- Track-data field names (`spotify_public_track` vs `deezer_track`).
- Artist normalization (Spotify Public can pass dicts or strings).
- No mirrored-playlist DB writeback (sync is handled separately).
Dependencies injected via `SpotifyPublicDiscoveryDeps` (12 fields) —
spotify_public_discovery_states, spotify_client, plus 10 callable
helpers (pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, get_discovery_cache_key, get_database,
validate_discovery_cache_artist, search_spotify_for_tidal_track,
build_discovery_wing_it_stub, add_activity_item).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 278 lines orig = 278 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 10 new under tests/discovery/test_discovery_spotify_public.py
covering cache hit short-circuit, dict-artist normalization, Spotify
tuple match (track/disc preservation), iTunes dict match path, Wing It
fallback, cancellation, completion phase update, activity feed entry,
top-level error handler, per-track error handling.
Full suite: 1118 passing (was 1108). Ruff clean.
Fourth lift in the PR5 discovery-workers series. Pulls the 270-line
Deezer discovery worker out of `web_server.py` into its own focused
module under `core/discovery/`. Pure 1:1 lift — wrapper keeps the
original entry-point name so the existing call sites continue to work
without changes.
What the Deezer discovery worker does:
1. Pause enrichment workers (release shared resources).
2. For each Deezer track:
- Cancellation gate (state['cancelled']).
- Discovery cache lookup; cache hit short-circuits the search and
populates display fields from the cached match (artist string,
album name).
- SimpleNamespace duck-type → `_search_spotify_for_tidal_track`
(shared search helper, returns tuple for Spotify or dict for iTunes).
- On Spotify match: build `match_data` preserving track_number /
disc_number from raw API data, image extracted from album images
or track object fallback, release_date filled from track.release_date
when album dict is missing it.
- On iTunes match: dict result populated as `match_data`, source set
to discovery_source, image extracted from album images.
- Save matched result to discovery cache.
- On miss: Wing It stub stored as 'wing-it' status.
3. After all tracks: phase='discovered', activity feed entry, sync
discovery results back to mirrored playlist via
`_sync_discovery_results_to_mirrored`.
4. On error: state['phase']='error' + status with error string.
5. Finally: resume enrichment workers.
Dependencies injected via `DeezerDiscoveryDeps` (13 fields) —
deezer_discovery_states dict, spotify_client, plus 11 callable helpers
(pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, get_discovery_cache_key, get_database,
validate_discovery_cache_artist, search_spotify_for_tidal_track,
build_discovery_wing_it_stub, add_activity_item,
sync_discovery_results_to_mirrored).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 270 lines orig = 270 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 10 new under tests/discovery/test_discovery_deezer.py covering
cache hit short-circuit, Spotify tuple match (track/disc number
preservation), iTunes dict match path, Wing It fallback, cancellation,
completion phase update, activity feed entry, mirrored sync invocation,
top-level error handler, per-track error handling.
Full suite: 1108 passing (was 1098). Ruff clean.
Third lift in the PR5 discovery-workers series. Pulls the 323-line
mirrored-playlist discovery worker out of `web_server.py` into its own
focused module under `core/discovery/`. Pure 1:1 lift — wrapper keeps
the original entry-point name so the existing call site
(`_run_playlist_discovery_worker(pls, automation_id=None)` from the
automation engine) continues to work without changes.
What the playlist discovery worker does:
1. Pause enrichment workers (release shared resources).
2. Pre-compute total track count across all playlists for the automation
progress card.
3. For each playlist:
- Fast pre-scan separates already-discovered tracks (skipped, unless
incomplete metadata or a Wing It stub) from undiscovered ones.
- For each undiscovered track:
- Cancellation gate via _playlist_discovery_cancelled set.
- Discovery cache lookup (with artist validation).
- matching_engine search-query generation, then Spotify (preferred)
or iTunes (fallback) search + scoring.
- Extended search fallback (limit=50) if no high-confidence match.
- On match → enrich album from metadata cache (id, images,
total_tracks, album_type, release_date, artists, plus track_number
and disc_number), build matched_data, write to track.extra_data,
save to discovery cache.
- On miss → Wing It stub stored as 'wing_it_fallback' provider.
4. After all playlists: emit `discovery_completed` event when at least
one new track was discovered, mark automation progress 'finished'.
5. On error → automation progress 'error', traceback printed.
6. Finally: resume enrichment workers.
Dependencies injected via `PlaylistDiscoveryDeps` (16 fields) —
spotify_client, matching_engine, automation_engine, the cancellation
set, plus 12 callable helpers (pause/resume enrichment,
get_active_discovery_source, get_metadata_fallback_client/source,
update_automation_progress, get_database, get_discovery_cache_key,
validate_discovery_cache_artist, discovery_score_candidates,
get_metadata_cache, build_discovery_wing_it_stub).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 323 lines orig = 323 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 15 new under tests/discovery/test_discovery_playlist.py covering
empty playlists, no-tracks playlist skip, complete-discovery skip,
incomplete-discovery re-run, Wing It always re-run, unmatched_by_user
respect, cache hit short-circuit, match above threshold (extra_data +
cache save), match below threshold falls to Wing It, iTunes fallback,
neither-provider error path, cancellation, discovery_completed event
emit, no-event on zero-discovered, multi-playlist grand_total
aggregation.
Full suite: 1098 passing (was 1083). Ruff clean.
yt_dlp sometimes returns float `duration_ms` for YouTube tracks. The
discovery workers format the duration with `f"{x // 60000}:{(x % 60000)
// 1000:02d}"` — and `:02d` requires an int. When the duration is a
float, the format string raises:
Unknown format code 'd' for object of type 'float'
Caught when running YouTube discovery on a real playlist (bbno$ tracks)
— every track failed with status='Error'.
Pre-existing bug, surfaced now because of yt_dlp returning float
durations on this playlist. Fixed at all 8 sites by casting through
`int()` before the `// 60000` and `% 60000` operations:
- core/discovery/youtube.py: 2 sites in run_youtube_discovery_worker
(cache hit + main result construction).
- web_server.py L29238/L29372: 2 sites in _run_listenbrainz_discovery_worker.
- web_server.py L40112/L40136/L40161/L40178: 4 sites in the YouTube
retry/pre-discovered results assembly path.
The `if duration_ms` / `if dur` guard already protects against None and 0,
so `int(...)` is only called on truthy numeric values.
Tests: 1 new regression test under tests/discovery/test_discovery_youtube.py
(`test_float_duration_does_not_crash_format`) — passes a float
duration_ms and asserts the worker completes without an error result.
Ruff clean.
Second lift in the PR5 discovery-workers series. Pulls the 332-line
YouTube discovery worker out of `web_server.py` into its own focused
module under `core/discovery/`. Pure 1:1 lift — wrappers keep the
original entry-point name so the two callers
(`youtube_discovery_executor.submit(_run_youtube_discovery_worker, ...)`)
continue to work without changes.
What the YouTube discovery worker does:
1. Pause enrichment workers (release shared resources).
2. For each YouTube playlist track:
- Cancellation check (phase != 'discovering' aborts).
- Discovery cache lookup; cache hit short-circuits the search.
- Strategy 1: matching_engine search queries with confidence scoring
against Spotify (preferred) or iTunes (fallback).
- Strategy 2: swapped artist/title query.
- Strategy 3: raw (untokenized) query.
- Strategy 4: extended search with limit=50.
- On match → save to discovery cache.
- On miss → build Wing It stub from raw source data.
3. After loop: phase='discovered', sort results by index, and for mirrored
playlists write extra_data back to the DB.
4. Activity feed entry with match summary.
5. On error → state['status']='error', phase='fresh'.
6. Finally: resume enrichment workers.
Dependencies injected via `YoutubeDiscoveryDeps` (16 fields) —
youtube_playlist_states, spotify_client, matching_engine, plus 13
callable helpers (pause/resume enrichment, get_active_discovery_source,
get_metadata_fallback_client, discovery cache key/validate, extract
artist name, spotify_rate_limited, discovery_score_candidates,
get_metadata_cache, build_discovery_wing_it_stub, get_database,
add_activity_item).
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 332 lines orig = 332 lines lifted, byte-identical body
(including all whitespace).
Pre-existing bug preserved (not fixed): if `youtube_playlist_states[url_hash]`
raises KeyError on entry, the outer except handler tries to mutate
`state` which is unbound → secondary UnboundLocalError. Same bug in
the original. Documented here for future cleanup but out of scope
for the lift.
Tests: 14 new under tests/discovery/test_discovery_youtube.py covering
cache hit short-circuit, Strategy 1 confidence match, Wing It fallback,
iTunes fallback path (Spotify unauthenticated and rate-limited),
cancellation (phase changed), skip_discovery flag, completion phase
update, activity feed entry, mirrored playlist DB writeback, non-mirrored
no-writeback, enrichment workers pause/resume, error-during-loop resume,
results sorted by index after retry.
Full suite: 1082 passing (was 1068). Ruff clean.
- remove the redundant wishlist-service injection from the runtime wrappers
- keep the package owning its own singleton service access
- simplify the route runtime API and update the wishlist tests to match
Keep the fuzzy-context post-processing test on the fast path by stubbing the retry sleep, matching the other retry-path coverage and avoiding a ~20s pause in the downloads test suite.
- ignore unconfigured backends when clearing completed downloads
- keep the post-download cleanup route best-effort after a successful wishlist run
- add regression coverage for the orchestrator clear step
First lift in the new PR5 discovery-workers series. Pulls the 448-line
playlist sync background worker out of `web_server.py` into its own
focused module under `core/discovery/`. Pure 1:1 lift — wrappers keep
the original entry-point name so the four callers
(`sync_executor.submit(_run_sync_task, ...)`) continue to work without
changes.
What the sync worker does:
1. Convert frontend JSON tracks → SpotifyTrack/SpotifyPlaylist objects.
2. Normalize artist/album shapes for downstream wishlist parity.
3. Wire a progress_callback that updates `sync_states` + automation card.
4. Patch sync_service for database-only fallback when no media server is
connected.
5. `run_async(sync_service.sync_playlist(...))` and capture the result.
6. Update sync_states to 'finished', push playlist poster image to
Plex / Jellyfin / Emby, record sync history (with re-sync vs new-sync
branching), emit `playlist_synced` event for automation engine, and
persist sync status with a tracks_hash for smart-skip on the next
scheduled sync.
7. On exception → mark error in sync_states + automation; finally clear
progress callback + drop `_original_tracks_map` from sync_service.
Dependencies injected via `SyncDeps` (11 fields) — config_manager,
sync_service, plex_client, jellyfin_client, automation_engine, run_async,
record_sync_history_start, update_automation_progress,
update_and_save_sync_status, sync_states dict, sync_lock. The only
structural drift from a pure paste is the top-of-function variable
binding: original used `global sync_states, sync_service`, lifted version
rebinds them as locals from deps (`sync_states = deps.sync_states` etc.)
since the names aren't module-level in the new file. Same behaviour
otherwise — diff against the original after `deps.X` → global X
normalization is **zero differences**.
Tests: 18 new under tests/discovery/test_discovery_sync.py covering
sync history recording (new + resync), setup error path (with and
without automation_id), missing sync_service handling, sync_playlist
exception handling, successful sync state transition, unmatched-tracks
summary, playlist image upload (plex + jellyfin + zero-synced gate),
automation engine emit, automation progress finished call, sync history
DB persistence (completion + match_details), tracks_hash persistence,
and finally-block cleanup (callback clear + map drop).
Full suite: 1068 passing (was 1050). Ruff clean.
Kicks off the PR5 series — 9 discovery workers totaling ~2,400 lines
across `_run_sync_task`, `_run_*_discovery_worker` family,
`_run_quality_scanner`, and `_process_watchlist_scan_automatically`.
Wishlist-related extractions deliberately skipped to avoid overlap with
kettui's planned `core/wishlist/` package.
- add direct service and presence coverage
- pin resolver, processing, route, and payload edge cases
- keep wishlist package extraction safe for future refactors
- extract the remaining wishlist endpoint behavior from web_server.py into core/wishlist/routes.py
- keep web_server.py as a thin Flask adapter around the new route helpers
- add tests that cover wishlist counts, stats, track listing, clear/remove flows, cycle updates, and album-track adds
- add core/wishlist as the home for wishlist payload, resolution, state, processing, reporting, and selection helpers
- move wishlist-specific tests into tests/wishlist alongside the new package layout
- keep web_server.py and the import/search callers as thin adapters for now
Final extraction in the download orchestrator series. Lifts the 586-line
master worker that drives the entire missing-tracks pipeline from
`web_server.py` into `core/downloads/master.py`. Pure 1:1 lift — wrappers
keep the original entry-point name so the three callers
(`missing_download_executor.submit(_run_full_missing_tracks_process, ...)`)
continue to work without changes.
What the master worker does:
1. PHASE 1 ANALYSIS — per-track DB ownership check with album fast path
(lookup album by name+artist, match tracks within it) plus a
MusicBrainz release-cache preflight so per-track post-processing all
uses the same release MBID (prevents Navidrome album splits).
2. Wishlist removal for tracks already in the library.
3. Explicit-content filter.
4. PHASE 2 transition — if nothing missing, mark batch complete, update
per-source playlist phases, kick auto-wishlist completion handler.
5. Soulseek album pre-flight — search for a complete album folder before
falling back to track-by-track search, cache the source for reuse.
6. Wishlist album grouping — derive per-album disc counts and resolve ONE
artist context per album so collab albums don't fold-split.
7. Task creation with explicit album/artist context injection +
playlist-folder-mode flag propagation.
8. Hand off to download_monitor + start_next_batch_of_downloads.
9. Error handler — phase=error, reset YouTube playlist phase to
'discovered', reset auto-wishlist globals on auto-initiated batches.
Dependencies injected via `MasterDeps` (21 fields) — wide surface
covering config, MB caches/locks, soulseek client, source-page state
dicts, multiple callbacks (wishlist removal, explicit filter, executor
+ auto-completion fn, monitor, start_next_batch). The only behaviour
difference from a pure paste is `import traceback` hoisted to module
scope (was inline in the except block) — same behaviour. Trailing
whitespace on two blank lines also got normalized away by the editor;
neither has any runtime effect.
`reset_wishlist_auto_processing` callback wraps the
`global wishlist_auto_processing, wishlist_auto_processing_timestamp`
write + `wishlist_timer_lock` since `global` can't reach back into
web_server.py from a separate module.
Tests: 21 new under tests/downloads/test_downloads_master.py covering
analysis-phase state, force_download_all, found-track wishlist removal,
explicit filter, no-missing complete + per-source state updates, auto
wishlist completion submit, album fast path (direct + fallthrough),
MB preflight (caches both keys, no-mb-worker no-op), task creation
(queue + tasks dict, explicit context for albums, wishlist album
grouping consistency, playlist folder mode), monitor + next-batch
handoff, multi-disc total_discs computation, error handler (phase set,
youtube reset, auto wishlist reset), and batch-removed-mid-flight
defensive path.
Full suite: 1050 passing (was 1029). Ruff clean.
End of the PR4 series — `web_server.py` lost ~590 lines on this commit
alone; total trim across PR4a–PR4h is ~2900 lines of orchestrator code
moved into focused `core/downloads/*.py` modules.
Seventh sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change. ~570 lines moved out of web_server.py.
What moved (lifted as 3 tightly-coupled functions in one module):
- _start_next_batch_of_downloads → start_next_batch_of_downloads
- _on_download_completed → on_download_completed
- _check_batch_completion_v2 → check_batch_completion_v2
Dependencies bundled in `LifecycleDeps` (15+ refs):
- config_manager, automation_engine, download_monitor, repair_worker,
mb_worker (live globals)
- is_shutting_down (lambda over IS_SHUTTING_DOWN flag)
- get_batch_lock (web_server helper for batch_locks dict)
- submit_download_track_worker (lambda wrapping
missing_download_executor.submit + _download_track_worker)
- submit_failed_to_wishlist + submit_failed_to_wishlist_with_auto_completion
(async, used by on_download_completed) AND process_failed_to_wishlist
+ process_failed_to_wishlist_with_auto_completion (sync, used by
check_batch_completion_v2 — direct call matches original v2 behavior;
the non-v2 path always submitted to executor)
- ensure_spotify_track_format, get_track_artist_name,
check_and_remove_from_wishlist, regenerate_batch_m3u (web_server
helpers — large, will lift in follow-up PRs)
- youtube_playlist_states, tidal_discovery_states,
deezer_discovery_states, spotify_public_discovery_states
(per-source playlist state dicts — phase transitions on batch
completion)
Direct imports for already-lifted helpers:
- core.runtime_state.{download_tasks, download_batches, tasks_lock,
add_activity_item}
- core.downloads.history.record_sync_history_completion (PR4a)
- core.album_consistency.run_album_consistency
- core.metadata.common.get_file_lock
Behavior parity verified line-by-line:
- start_next_batch: same batch lock acquisition, same shutdown gate,
same V2-cancelled-task skip, same searching-status-set-before-submit,
same submit-fails-no-ghost-worker semantics
- on_download_completed: same duplicate-call detection (skip decrement
but still check completion), same failed-track tracking with
spotify_track formatting + activity items + automation event emission,
same wishlist removal on success, same active_count decrement, same
stuck-detection (searching > 10min → not_found, post_processing >
5min → completed), same M3U regeneration + repair worker hand-off
+ album consistency pass + wishlist failed-tracks submission
- check_batch_completion_v2: same finished-count tally, same stuck
detection, same already-complete short-circuit returning True, same
per-source playlist phase updates, same album consistency pass,
same DIRECT (sync) wishlist processing call (NOT submit-to-executor
— matches original v2 which called process_* functions directly)
CRITICAL drift caught + fixed during review:
- Initial lift had v2 routing wishlist calls through submit_* deps
(async). Original v2 called process_* directly (sync). Added separate
process_* deps to LifecycleDeps and routed v2 to them. Tests updated.
Two minor defensive additions documented:
- `is_auto_batch = False` initialized before conditional in v2 (Python
scope rules made this unnecessary in original, but explicit is safer)
- Variable rename inside the queue-completion-check loop in
on_download_completed: `task_id` → `queue_task_id` to avoid shadowing
the outer parameter. Log output preserves the same task ID.
Tests: 28 new under tests/downloads/test_downloads_lifecycle.py
covering start-next (early-returns, shutdown gate, max_concurrent,
cancelled-task skip, searching-status-set, submit-failure-no-ghost,
orphan task), on-complete (decrement, duplicate skip, failed/cancelled
tracking, automation emit, wishlist removal, batch completion + emit
+ source phase update, stuck detection, auto vs manual routing),
check-v2 (missing batch, not-complete, complete-marking, already-
complete, auto routing, exception handling).
Full suite: 1029 passing (was 1001). Ruff clean.
Fifth sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _build_batch_status_data → build_batch_status_data
- get_batch_download_status route body → build_single_batch_status
- get_batched_download_statuses route body → build_batched_status
- get_all_downloads_unified route body → build_unified_downloads_response
- Status priority dict → module-level _STATUS_PRIORITY constant
Dependencies bundled in `StatusDeps` dataclass:
- config_manager, docker_resolve_path, find_completed_file,
make_context_key, submit_post_processing (lambda wrapping
missing_download_executor.submit + _run_post_processing_worker),
get_cached_transfer_data
Direct imports from core.runtime_state for download_tasks /
download_batches / tasks_lock (already lifted by kettui).
Behavior parity:
- Same response payload shape across all 3 endpoints
- Same safety-valve mutation: stuck downloading task with file recovered
→ status='post_processing' + submit worker; stuck searching → not_found;
stuck downloading no file → failed
- Same live transfer state mapping (Cancelled/Canceled, Failed/Errored/
Rejected/TimedOut, Completed/Succeeded with byte-mismatch verification,
InProgress, default queued)
- Same intermediate post_processing status promotion + single-shot worker
submission (only when status != 'post_processing')
- Same 'Errored' handling: keeps current status to let monitor retry
- Same 17-key item dict in unified response with same field order
- Same artist/album/artwork normalization (handles string, dict, list,
list-of-dicts, list-of-strings variants)
- Same sort: (priority asc, -timestamp desc)
- Same batch summary aggregation
- Same items[:limit] slicing
- Same logger messages text-for-text
- Same lock scope (single tasks_lock per call) — no new contention
Pre-existing bug preserved (will fix in follow-up PR):
- batched_status `debug_info` block iterates `response["batches"]` and
guards with `if "error" not in batch_status`. Every successful
payload includes `"error": batch.get('error')` (key always present,
value usually None) so the guard is always False and debug_info
never populates in production. Test documents the buggy behavior so
the next PR can flip the check to `batch_status.get('error') is None`.
Tests: 32 new under tests/downloads/test_downloads_status.py covering
phase routing (analysis vs downloading vs unknown), task formatting +
sort + V2 fields, every live transfer state mapping (Cancelled,
Succeeded with full + partial bytes, InProgress, Errored, terminal-
not-overridden), safety valve (stuck searching → not_found, stuck
downloading recovered → post_processing, stuck downloading no file →
failed), all 3 route helpers (single, batched, unified), unified
artist/album/artwork normalization, batch summary aggregation, limit
slicing, plus debug_info bug documentation.
Full suite: 982 passing (was 950). Ruff clean.
Fourth sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change. ~407 lines moved out of web_server.py.
What moved:
- _run_post_processing_worker → run_post_processing_worker
The lifted function is intentionally kept as one ~400-line block to
preserve byte-for-byte parity with the original. Refactoring it into
smaller helpers (context lookup, file search loop, transfer-folder
handler, downloads-folder handler) gets its own follow-up PR.
Dependencies: 9 callbacks bundled in `PostProcessDeps` dataclass.
- config_manager, soulseek_client, run_async (live refs)
- docker_resolve_path, extract_filename, make_context_key
(small utilities still in web_server.py — will lift in a future PR
alongside other shared utilities)
- find_completed_file (file search helper, still in web_server.py)
- enhance_file_metadata, wipe_source_tags (web_server wrappers around
core.metadata.enrichment)
- post_process_with_verification (web_server wrapper around
core.imports.pipeline)
- mark_task_completed (wraps runtime_state.mark_task_completed +
session counter)
- on_download_completed (deferred to PR4g batch lifecycle)
Direct imports for already-lifted helpers (no injection needed):
- core.imports.album_naming.resolve_album_group
- core.imports.context.{get_import_clean_title, get_import_clean_album,
get_import_original_search, get_import_context_artist,
get_import_context_album, normalize_import_context}
- core.imports.filename.extract_track_number_from_filename
- core.metadata.enrichment (re-exported as metadata_enrichment)
- core.runtime_state.{download_tasks, tasks_lock,
matched_downloads_context, matched_context_lock}
Behavior parity:
- Same control flow: missing-task short-circuit → cancelled/completed
short-circuit → missing-filename failure → docker path resolution →
context lookup with fuzzy fallback → expected filename generation →
YouTube special-case path resolution → 5-attempt search loop with
Strategy 1 (original filename in download+transfer) and Strategy 2
(expected final filename in transfer) → file-not-found failure →
transfer-folder handler with metadata enhancement → downloads-folder
handler with full post-process verification
- Same retry count (5), sleep duration (5s), per-attempt logging
- Same album_info dict construction with is_album=True for explicit
album downloads
- Same album grouping skip when context.is_album_download is True
- Same wipe_source_tags fallback when enhancement context missing
- Same matched_downloads_context cleanup on success
- Same exception swallowing at processing-error and critical-error
layers, both setting status='failed' + error_message + calling
on_download_completed(b, t, success=False)
- Every logger message text preserved verbatim (so log filters keep
working)
Tests: 16 new under tests/downloads/test_downloads_post_processing.py
covering missing task, cancelled, already-completed, stream_processed,
missing filename + username, file-not-found-after-retries with sleep
mocked, stream-processor-completes-mid-search, transfer-folder with
metadata enhanced + with no context (wipes tags), downloads-folder
with + without context, processing exception, critical outer
exception, YouTube special path, fuzzy context matching.
Full suite: 950 passing (was 934). Ruff clean.
Third sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _automatic_wishlist_cleanup_after_db_update → cleanup_wishlist_after_db_update
The lifted fn takes config_manager as an arg (so core/downloads/cleanup.py
doesn't need to import web_server). Other deps (wishlist_service,
MusicDatabase, get_database) stay as in-function imports — matches the
original deferred-import pattern.
The single caller in web_server.py (missing_download_executor.submit at
L18028) keeps using the same wrapper name with no signature change.
Behavior parity:
- Same per-profile iteration via get_all_profiles()
- Same essential-field skip (no name / no artists / no spotify_track_id)
- Same artist normalization (string / dict / fallback to str())
- Same 0.7 confidence threshold for db match
- Same break-on-first-artist-match semantics
- Same album extraction (dict.name vs string passthrough)
- Same active_server pulled via config_manager.get_active_media_server()
- Same per-track exception swallowing inside the loops
- Same top-level exception swallow with traceback.print_exc()
- Same logger messages (exact text match for "[Auto Cleanup]" prefix)
Tests: 13 new under tests/downloads/test_downloads_cleanup.py covering
empty wishlist short-circuit, found-in-db removal, missed track stays,
low-confidence skip, missing-fields skip, dict + string artist formats,
break-on-first-match, multi-profile walk, album dict/string handling,
db check failure continuing to next artist, top-level exception swallow,
active server propagation.
Full suite: 934 passing (was 921). Ruff clean.
Second sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- cancel_download (single slskd cancel) → cancel_single_download
- cancel_all_downloads (cancel + clear + sweep) → cancel_all_active
- clear_finished_downloads (slskd clear + sweep) → clear_finished_active
- clear_completed_downloads (local task tracker prune) → clear_completed_local
Slskd-touching helpers take (soulseek_client, run_async, sweep_callback)
explicitly so the route layer wires the live client + the existing
_sweep_empty_download_directories helper. The local-state helper imports
download_tasks/download_batches/batch_locks/tasks_lock straight from
core.runtime_state since those are module-level shared globals.
Prep change: `batch_locks` dict moved from web_server.py global into
core/runtime_state.py alongside the other download globals. web_server.py
re-imports from runtime_state so the ~3 existing call sites in
web_server.py keep resolving without modification. Identity preserved
(same dict across all importers).
Out of scope (deferred to PR4g batch lifecycle):
- cancel_download_task (calls _on_download_completed)
- cancel_task_v2 + _atomic_cancel_task + _find_task_by_playlist_track
(manipulate batch active_count directly, deeply coupled to lifecycle)
Behavior parity:
- Same response shapes + status codes on each route
- Same call order (cancel_all → clear_all_completed → sweep)
- Same conditional sweep on clear_finished (skipped on failure)
- Same sweep ALWAYS runs after cancel_all even if clear_all returns False
(matches original — clear failure was non-fatal in cancel_all path)
- Same TERMINAL_STATUSES set: completed/failed/not_found/cancelled/skipped/
already_owned (lifted to module-level constant)
- Same empty-batch pruning + same batch_locks cleanup
- Same lock acquisition pattern (single tasks_lock)
Tests: 14 new under tests/downloads/test_downloads_cancel.py covering
single cancel, cancel-all happy + failure paths, clear-finished + sweep
gate, local task pruning across all 7 active/terminal states, batch
queue trimming, batch_locks cleanup.
Full suite: 921 passing (was 907). Ruff clean.
First sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _record_sync_history_start → record_sync_history_start
- _record_sync_history_completion → record_sync_history_completion
- _detect_sync_source → detect_sync_source
- Source prefix map → module-level _SOURCE_PREFIX_MAP constant
What stayed:
- web_server.py keeps three thin wrappers (_detect_sync_source,
_record_sync_history_start, _record_sync_history_completion) that
delegate into core/downloads/history.py. ~60 callers of these names
in web_server.py keep resolving without touching every site.
Each lifted function takes `database` as an arg (was
`db = MusicDatabase()` inline). The wrappers construct
`MusicDatabase()` per call to mirror the exact original behavior —
each invocation got a fresh DB connection.
Behavior parity:
- Same SQL UPDATE statement (preserves the in-place update path when
a sync_history entry already exists for the playlist_id)
- Same JSON serialization with ensure_ascii=False
- Same thumb URL extraction order (album_context.images → image_url
→ first track album.images)
- Same per-track result shape (index, name, artist, album, image_url,
duration_ms, source_track_id, status, confidence, matched_track,
download_status)
- Same status mapping (found/not_found, completed/failed)
- Same best-effort exception swallowing (sync history failure must
never break the actual download)
- Reads `download_tasks` from core.runtime_state (already lifted by
kettui in PR378)
Tests: 34 new under tests/downloads/test_downloads_history.py
covering source detection (16 prefixes), start happy paths + thumb
extraction + duplicate-update + DB error swallowing, completion stats
+ per-track results JSON shape + edge cases.
Full suite: 907 passing (was 873). Ruff clean.
User report: SoulSync was only pulling MusicBrainz genres from the
recording (track-level) endpoint. Most MB recordings don't carry genres
at the track level — they live on the release (album) or artist. So
the MB tier was contributing nothing to the genre merge for the
overwhelming majority of tracks.
Fix:
- Added `'genres'` to the release-detail `includes` (was missing).
- After release-detail processing, if pp['mb_genres'] is still empty,
populate from release_detail['genres'] (sorted by count desc).
- If still empty AND artist_mbid is set, fetch artist with
`includes=['genres']` and use those.
No extra API call when the recording (or release) already had genres —
the artist fetch only fires when both upstream tiers came back empty.
The downstream genre merge in _embed_metadata_genres is unchanged; this
just makes the MB feed into it richer.
Tests: 4 new (recording present, recording empty → release, recording
+ release empty → artist, all empty → []). Full suite 873 passing.
Ruff clean.
Reported by @kcaoyef421 in Discord.
The endpoint was returning a 200-line literal dict inline. Moved the
three lists (TRIGGERS, ACTIONS, NOTIFICATIONS) to module-level constants
in core/automation/blocks.py. Route shrinks to 7 lines. Data is now
importable for tests + future docs.
Added 8 shape tests so a typo in the dict (missing 'type', wrong
field type, missing options on a select, etc.) gets caught by CI
instead of breaking the builder UI silently.
The `known_signals` field stays computed at request time via
_collect_known_signals(database) since it's dynamic.
No behavior change. Same response shape. 869 tests passing (was 861).
Ruff clean.
Routes moved to thin parse-args/jsonify handlers; logic now lives in
three focused modules under core/automation/. 436 lines deleted from
web_server.py; 53 added back as wrappers.
Module split:
- core/automation/api.py — CRUD + run + history helpers. Each function
takes (database, automation_engine, ...) explicitly and returns
(response_body, http_status). Includes signal cycle detection
preflight checks for create + update.
- core/automation/progress.py — owns the in-memory progress state dict
+ lock (mirroring the original web_server.py globals as module-level
shared state so all callers see one view), init/update/history
helpers, and the WebSocket emit loop.
- core/automation/signals.py — collect_known_signals for the builder
autocomplete.
Out of scope (deferred):
- _register_automation_handlers — the 23+ action handler closures stay
in web_server.py because each one is tightly coupled to feature-
specific implementations (wishlist, watchlist, library scan, etc.).
- Worker functions (_process_wishlist_automatically, etc.) — belong
with their feature lifts.
- _run_sync_task / _run_playlist_discovery_worker — sync + discovery
PRs.
Behavior preserved 1:1:
- Same route response shapes + status codes
- Same JSON field hydration (trigger_config, action_config,
notify_config, last_result, then_actions)
- Same backward-compat: empty then_actions + notify_type set →
synthesize then_actions from notify_type/notify_config
- Same signal cycle detection behavior on create + update
- Same system-automation protection on delete + duplicate
- Same reschedule/cancel logic on toggle + bulk-toggle + update
- Same progress state shape (status, progress, phase, current_item,
log capped at 50, started_at/finished_at, action_type)
- Same emit-on-finish socketio push from update_progress
- Same emit loop semantics (1s tick, snapshot active states, reap
finished after window)
Pre-existing bugs preserved (will fix in follow-up PRs):
- emit_progress_loop uses naive datetime.now() against tz-aware
started_at/finished_at, so the timeout-zombie check raises
TypeError → caught → never fires, and the cleanup-after-window
check raises → caught → state is reaped on FIRST tick regardless
of the window. Tests document this behavior so the next PR can
flip them to the corrected expectation.
Tests: 72 new under tests/automation/ (signals 10, progress 24,
api 38). Full suite: 861 passing (was 789). Ruff clean.
Three drifts caught in line-by-line review against the pre-lift
web_server.py. All addressed for strict 1:1 behavior parity.
1. /api/enhanced-search/source/<src> now returns plain JSON
`{"artists":[],"albums":[],"tracks":[],"available":false}` (or
`{"videos":[],"available":false}` for youtube_videos) when the
source's client isn't available, matching the original endpoint
contract. Previously streamed an NDJSON `{"type":"done"}` line
instead.
Restructured by splitting the orchestrator into resolve+stream
helpers:
- `resolve_client(source_name, deps)` — already existed, used
for /api/enhanced-search single-source mode
- `resolve_youtube_videos_client(deps)` — new, returns the
soulseek_client.youtube subclient or None
- `stream_metadata_source(source_name, query, client)` — pure
NDJSON generator, caller resolves client first
- `stream_youtube_videos(query, youtube_client, run_async)` —
same shape for the yt-dlp path
The route now decides plain-JSON-vs-stream based on resolution
result, mirroring the original control flow exactly.
2. core/search/library_check.py — reverted the defensive `(x or '')`
and `getattr(plex_client, 'server', None) is not None` patterns
to original byte-for-byte (`x.get('name', '')`,
`plex_client.server`, no try/except around `get_plex_config`).
Lift PR shouldn't change crash semantics; if the original raises
on malformed input, mine should too. Pre-existing edge cases get
their own follow-up PR.
3. core/search/stream.py — same revert: `soulseek_client.youtube`
instead of `getattr(..., 'youtube', None)` etc.
Also removed the module-level `EMPTY_SOURCE` from sources.py and
moved its (per-call) duplicate into _fan_out_response as a local —
the original used a per-request local dict and the identity-check
behavior depends on that. Module-level was a footgun for future
mutations.
789 tests still pass (95 search), ruff clean.
Line-by-line review of the search lift caught one drift: cache.get_cache_key
was coercing falsy provider returns ('', None, 0) to 'unknown' / False.
Original web_server.py only fell back to those sentinels on exception, not
on falsy success values.
Real-world impact: low — get_active_media_server() and get_primary_source()
return non-empty strings in practice. But cache keys are tuples with no
schema enforcement, so any drift here can silently fragment the cache.
Restored 1:1 parity with original semantics.
Added test covering the falsy-success path so this can't drift again.
789 tests pass, ruff clean.
Routes moved to thin parse-args/jsonify handlers; logic now lives in
six focused modules under core/search/. 720 lines deleted from
web_server.py; 109 added back as wrappers; ~700 lines of new core code
plus ~700 lines of tests.
Module split:
- core/search/cache.py — TTL+LRU cache for enhanced-search responses,
keyed by (query, active_server, fallback_source, hydrabase_active,
source_tag) so config changes don't poison stale entries.
- core/search/sources.py — per-kind metadata search (artists/albums/
tracks) and the multi-kind ThreadPoolExecutor that fans them out.
- core/search/library_check.py — library + wishlist presence check
with Plex thumb URL resolution; profile-aware wishlist with legacy
fallback for older DBs missing the profile_id column.
- core/search/stream.py — single-track preview search; effective stream
mode resolution, query-variant generation, retry walk, matching
engine integration.
- core/search/basic.py — flat Soulseek file search, quality-sorted.
- core/search/orchestrator.py — main enhanced-search dispatch
(short-query fast path, single-source bypass, hydrabase-primary fan
out, alternate source list builder), NDJSON streaming generator
for /source/<src>, and the SearchDeps dataclass that bundles the
cross-cutting deps.
Routes pass clients (spotify, hydrabase, hydrabase_worker, soulseek)
and helpers (config_manager, fix_artist_image_url,
_is_hydrabase_active, _get_metadata_fallback_*, _run_background_
comparison, run_async, dev_mode_enabled_provider) into core/search via
a SearchDeps bundle built per-request. fix_artist_image_url stays in
web_server.py because it touches 31 other call sites.
Behavior preserved 1:1:
- Same response shapes (db_artists, spotify_artists, spotify_albums,
spotify_tracks, primary_source, metadata_source, alternate_sources,
source_available)
- Same NDJSON line ordering (artists/albums/tracks as they finish, plus
done marker)
- Same per-kind exception swallowing
- Same hydrabase-worker mirror on dev mode
- Same cache key shape (5-tuple) and TTL/LRU semantics
- Same stream-track effective-mode resolution including the
Soulseek-coerce-to-YouTube edge case
- Same library-check Plex thumb URL rewriting and wishlist fallback
for older DBs
Tests: 94 new (cache TTL/LRU/key, sources happy/partial/all-fail,
library presence with library + wishlist + thumbs, stream effective
mode + query gen + retry, orchestrator client resolution + short
query + single source + fan-out alternates + hydrabase primary +
NDJSON drain). Full suite: 788 passing (was 694).
Ruff clean.
Stats route logic moves into core/stats/queries.py as pure-ish functions
that take dependencies (database, image-url fixer, listening worker) as
arguments. The 13 route handlers in web_server.py shrink to thin
parse-args / jsonify wrappers.
What moved to core/stats/queries.py:
- stats_cached: 3-key metadata cache lookup + image url fix-up
- stats_overview / timeline / genres / library_health / db_storage
- stats_top_artists / top_albums / top_tracks: top-N + DB enrichment
- stats_recent: listening_history readback
- stats_resolve_track: title+artist -> file_path lookup for playback
- listening_stats_sync: spawns daemon thread that runs worker._poll
- listening_stats_status: stats payload, with None-worker fallback shape
No behavior change. Same response shapes, same error handling, same
silent-except on per-row enrichment failure. fix_artist_image_url
stays in web_server.py and is passed through as a callback so we
don't have to lift its config_manager / media-server dependencies in
this PR.
Adds tests/stats/test_stats_queries.py — 27 tests covering happy
paths, edge cases, image-url plumbing, worker glue.
Ruff clean. 694 tests pass (was 667 + 27 new).
- Normalize album import track display handling so queue labels and match rows stay consistent
- Bound MusicBrainz caches and avoid caching transient lookup failures
- Stop swallowing programmer errors in source enrichment helpers
- Restore import config test seams without reintroducing lazy imports
- Guard task completion calls and fix the Windows path test expectation
- Keep file lock tracking from growing without bound
- Cover search_result fallback normalization and ambiguous album detection.
- Add staging metadata, multi-disc path, and MusicBrainz enrichment cases.
- Move the single-track context test next to the imports code it exercises.
- keep single-track import lookup in imports/resolution.py
- normalize simple-download search_result data before wishlist matching
- run wishlist cleanup for simple-download post-processing
- keep source-only artist detail on resolved names and MB short-circuit
- Move the import pipeline runtime factory into core.imports.pipeline
- Move the metadata runtime factory into core.metadata.enrichment
- Keep the web server wiring thin and drop the shared glue module
- Add contract tests that keep the two runtime bundles separate
- Move the metadata and MusicBrainz-related tests into a dedicated tests/metadata subfolder.
- Keep the rest of the suite flat for now.
- Preserve the existing test filenames so the change stays organizational rather than behavioral.
- Pass the live runtime bundle into the shared metadata facade so worker-backed source enrichment can actually run.
- Forward runtime from the import pipeline and web-server wrapper into embed_source_ids.
- Add a regression test that verifies the runtime object reaches the source-ID embedding path.
- Keep existing metadata_cache and metadata_service at the top level for now
- Move the new branch-local metadata helpers under core/metadata
- Share MusicBrainz release cache state from core.metadata.source and update import sites
- Move app-wide task and activity registries out of core/imports
- Share one runtime-state module across the web server, API, and import pipeline
- Keep import-specific helpers focused on context and post-processing
- Move import flow modules into a dedicated package
- Update app and test imports to the new namespace
- Group the import-focused tests under tests/imports
- remove stale wrapper helpers from web_server and metadata_common
- import provider helpers directly in metadata_source
- keep the metadata modules' public surface explicit
- remove runtime from metadata helper APIs where it only carried config, logger, mutagen, and database access
- keep runtime only for the source-ID enrichment path that still needs live worker handles
- add the new metadata helper modules and update the tests to match the slimmer interfaces
- Move filename and staging helpers into their canonical modules
- Extract album naming and grouping from path handling
- Update import and test call sites to the new layout
- Extract the import pipeline, album import, staging, path, file ops, guards, runtime state, side effects, and metadata enrichment out of .
- Canonicalize the refactored import path around and remove legacy , , , and request shapes from the import endpoints.
- Make album and track metadata lookups follow the configured provider priority instead of hard-coding Spotify, while still falling back when needed.
- Update the import routes and frontend payloads to use the new core helpers.
- Add coverage for the extracted helpers and the refactored import flows.
PS. apologies to anyone who might check this commit out - the intention was to start small, but things kinda snowballed out of control at some point since the logic just kept going on and on, and everything kinda had to be changed all at once for it all to make any sense
Six items from a Cin-style line-by-line pass on PR #383:
- resolve_cors_origins: list of non-string entries (`[None, 123]`) now
drops them instead of coercing to junk strings like `'None'`/`'123'`.
- will_reject: backwards-compat shim removed. Production callers always
pass `request.scheme` (Flask-guaranteed); the shim only existed for
tests/non-Flask callers and made the production code path branchier
than necessary. Tests now pass scheme explicitly.
- maybe_log: redundant `if not origin` early-return dropped. will_reject
handles missing origin (engineio's own behavior — server.py:207).
- RejectionLogger.__init__: `int(dedup_cap)` wrapped in try/except so
bad-type input falls back to DEFAULT_DEDUP_CAP instead of raising.
- web_server.py: docstring on the before_request hook explains why the
hook fires on every request (Flask doesn't scope before_request to a
path prefix; the early-return string compare is the cheapest option).
- settings.js: cors-origins URL regex tightened from `[^\s/]+` to
`[^\s/?#]+` so query/fragment chars don't pass validation. Engineio
would silently fail to match those anyway; better to flag at save.
Test changes:
- parametrize gained an explicit `scheme` column (12 cases updated).
- New explicit case: scheme-mismatch rejects (engineio compares full
`{scheme}://{host}` strings).
- `test_will_reject_falls_back_to_host_only_when_no_scheme_info`
deleted — the shim it tested is gone.
- `test_will_reject_honors_x_forwarded_host` now passes scheme info.
Net: -9 production lines, -3 test lines. Production code path is
straight-line. 603 tests pass.
Self-review pass on the security fix uncovered five issues, all fixed
here:
1. will_reject scheme handling. Engineio compares full {scheme}://{host}
strings, not just hostnames. A TLS-terminating proxy can leave the
backend seeing http while the browser's Origin is https — engineio
rejects, but the original predictor said "allow" → no helpful log
line. Added request_scheme + forwarded_proto params, build full
candidate strings to match engineio.
2. EITHER-forwarded-header rule. Engineio adds the forwarded candidate
when EITHER X-Forwarded-Proto OR X-Forwarded-Host is present (it
falls back to HTTP_HOST for the missing one). The original predictor
only added it when forwarded_host was set — false negative for
misconfigs sending only X-Forwarded-Proto. Now mirrors engineio.
3. will_reject incorrectly rejected missing-Origin requests. Engineio
(server.py:207: `if origin: validate`) skips CORS validation when
no Origin header is sent — non-browser clients (curl etc.) are
intentionally permitted. The original code rejected them. Test was
asserting the wrong behavior. Both fixed.
4. RejectionLogger had unbounded dedup set growth. A hostile actor
opening connections from many distinct fake origins would fill
memory unboundedly. Capped at 100 unique origins (configurable);
when cap hit, one overflow notice is emitted and further rejections
are silently dropped until restart.
5. Lock pattern: the overflow log path called logger.warning() while
holding the dedup lock, inconsistent with the normal path. Fixed
to pick the message under the lock and log after release. Critical
section is now minimal and uniform.
Plus polish:
- Stale module docstring fixed (said "empty list" instead of "None").
- settings.js validates each cors_origins line against a URL regex on
save; toasts a one-shot warning if entries are malformed (resolver
silently filters them, but user gets feedback now).
- web_server.py wiring passes request.scheme + X-Forwarded-Proto so
the predictor has full proxy info.
Tests:
- 51 unit tests in tests/test_socketio_cors.py (was 45). New cases:
* scheme comparison (5 cases including TLS-terminating proxies)
* forwarded_proto-alone misconfig
* missing-origin matches engineio (was asserting wrong behavior)
* dedup cap with overflow + reset
* default cap is reasonable (uses public DEFAULT_DEDUP_CAP constant)
Engineio behavior independently verified by reading engineio/server.py
and engineio/base_server.py source. Predictor mirrors both files.
604 tests pass.
Closes#366 (reported by JohnBaumb).
Socket.IO was initialized with `cors_allowed_origins='*'`, accepting
WebSocket connections from any origin. A malicious site could open a
WS to a user's local SoulSync instance and exfiltrate live progress /
toast / activity events.
This commit:
- Defaults to engineio's same-origin behavior (`cors_allowed_origins=None`),
which automatically honors X-Forwarded-Host so reverse proxies that
send that header (Caddy / Traefik by default, properly-configured
Nginx) work transparently.
- Adds a `security.cors_origins` config setting + Settings → Security
textarea where users behind unusual proxies / Electron wrappers /
cross-origin integrations can whitelist their origin. Accepts comma
or newline separated values; `*` on its own line opts back into the
legacy wildcard with a startup-warning log.
- Logs a clear warning the first time engineio rejects each unique
origin, naming the rejected Origin and request Host and pointing
users to the settings field. Without this, engineio silently 403s
the upgrade and the user just sees a half-broken UI with no clue
why. Threadsafe dedup so a hostile origin can't spam logs.
Logic lives in `core/socketio_cors.py` (resolver, rejection
predictor, dedup logger class, startup-status emitter) — pure
functions, no Flask dependency. `web_server.py` adds 23 lines of
wiring and imports.
Important catch during review: my first pass used `cors_allowed_origins=[]`
as the "secure default." Reading engineio's source revealed `[]` actually
means "DISABLE CORS HANDLING" (engineio/server.py:202: `if cors_allowed_origins != []:`)
— identical security to `'*'`. Fixed to use `None` (engineio's actual
same-origin sentinel) and pinned with a regression test that asserts
the resolver never returns `[]` for any input shape.
Tests:
- tests/test_socketio_cors.py — 45 unit tests covering 19 resolver shape
cases (None, empty, whitespace, comma, newline, garbage types, lists),
the `[]`-must-never-be-returned security regression, 12 rejection
prediction cases, X-Forwarded-Host handling, dedup logger behavior,
threadsafe race (8 threads × 50 hammers → exactly 1 warning), and
startup-status emitter outputs.
Frontend:
- Settings → Security gains an "Allowed WebSocket Origins" textarea
with help text explaining same-origin default + when to add a domain
+ the `*` opt-out.
- helper.js — new '2.4.1' WHATS_NEW block (hidden until version bump)
with a chill-voice entry describing the change.
Conftest.py left at `'*'` — test environment, no security concern.
598 tests pass.
Five issues kettui flagged on PR #377:
- Worker race (reorganize_queue.py): _next_queued() picked an item and
released the lock, then re-acquired to flip status='running'. A
cancel() landing in that window marked the item cancelled but the
worker still ran it. Replaced with _claim_next_or_wait() that picks
AND flips under one lock acquisition.
- Wakeup race (reorganize_queue.py): _wakeup.clear() after the empty
check could lose an enqueue's _wakeup.set(), parking a freshly-queued
album for up to 60 seconds. Replaced Lock + Event with a single
threading.Condition; cond.wait() releases and re-acquires atomically
on notify.
- Bulk dedupe (reorganize_queue.py:enqueue_many): looped single-item
enqueue, so a duplicate album_id later in the same batch could slip
through if the worker finished the first copy before the loop
reached the second. Now holds the lock for the whole batch and tracks
a per-batch seen set, so intra-batch duplicates dedupe against each
other and not just pre-existing items.
- Preview button stuck disabled (library.js:loadReorganizePreview):
early returns and thrown errors skipped the re-enable line. Moved
state into a canApply flag committed in finally, so any exit path
lands the button correctly.
- DB helpers swallowing failures (music_database.py): get_album_display_meta
and get_artist_albums_for_reorganize used to catch every Exception
and return None / [], so a real DB outage masqueraded as "album not
found" / "no albums". Now lets exceptions bubble; the route layer
already wraps them as 500.
Tests:
- test_cancel_and_run_are_mutually_exclusive — hammers enqueue+cancel
pairs and asserts the invariant that no successfully-cancelled item
ever ran (catches regressions to the atomic pick).
- test_enqueue_many_dedupes_batch_internal_duplicates — pins the
intra-batch dedupe.
- test_get_album_display_meta_propagates_db_errors and
test_get_artist_albums_for_reorganize_propagates_db_errors — pin
the bubble-up behavior.
Changelog updated in helper.js and version modal.
Replaces the single-slot "one reorganize at a time, return 409 on collision"
model with a per-user FIFO queue. Buttons stay clickable, "Reorganize All"
is one backend call instead of an N-call JS loop, and a status panel mounted
at the top of the artist actions bar shows live progress (active item,
queued count, recent completions) with per-item cancel buttons.
Backend
- core/reorganize_queue.py: singleton queue + worker thread, dedupe-on-
enqueue, cancel rules (queued cancellable, running not), enqueue_many
for bulk operations, progress fan-out via update_active_progress
- core/reorganize_runner.py: factory builds the worker's runner closure
with injected dependencies. Reads config per-call so changing the
download path in Settings takes effect on the next reorganize without
a server restart
- database/music_database.py: get_album_display_meta and
get_artist_albums_for_reorganize — moves the SQL out of route handlers
- web_server.py: thin enqueue/snapshot/cancel/clear endpoints, runner
registration at module load. Old _reorganize_state globals + status
endpoint deleted. Static-asset cache buster (?v=<server-start>)
added so JS/CSS updates ship live without users clearing cache
Frontend
- webui/static/library.js: status panel mount, polling (1.5s when
active, 8s when idle), expand/collapse, per-item cancel, debounced
enhanced-view reload (one reload per artist batch instead of N).
Per-album reorganize button paints with queued/running indicator
and short-circuits to a toast when the album is already in queue
- webui/static/style.css: panel + button styling matching the existing
glass-UI accents
- webui/static/helper.js + version modal: WHATS_NEW entry
Tests (22 new)
- tests/test_reorganize_queue.py (19 tests): FIFO order, dedupe,
per-item source, cancel rules, continue-on-failure, snapshot
shape, progress propagation, bulk enqueue
- tests/test_reorganize_runner.py (4 tests): per-call config reads,
setup-failure summary, dependency injection, progress fan-out
- tests/test_reorganize_db_methods.py (7 tests): SQL JOIN behavior,
ordering, fallback for blank strings, artist isolation
Full suite 549 passed in 27s.
Four changes addressing kettui's PR #377 review comments:
1. **`_finalize_track` no longer over-counts on DB failure (🔴 bug).**
The function previously bailed on DB-update failure but
`_process_one_track` still incremented `summary['moved']`
unconditionally — overstating how many tracks the UI knows are
at their new locations. Fixed by:
- `_finalize_track` now returns ``bool`` (True only when DB row
was updated AND original was dealt with)
- Caller checks the return; on False, records as a failed track
with a clear message ("Track landed at new location but DB
update failed — file is at both old and new paths until library
scan re-indexes")
- Existing `test_db_update_failure_leaves_original_in_place` now
also asserts `moved == 0`, `failed == 1`, and that the error
message names the cause
2. **`executeReorganize` toast no longer says "undefined tracks" (🐛
bug).** `/reorganize` doesn't return `result.total` anymore (the
track count is determined server-side after planning), so the
"Reorganizing undefined tracks..." string was meaningless. Now uses
`result.message` from the backend instead.
3. **`_pollReorganizeStatus` distinguishes completed from skipped
(🟡 risk).** Backend now propagates the orchestrator's status
(`completed` / `no_source_id` / `no_album` / `no_tracks` /
`setup_failed` / `error`) into `_reorganize_state['result_status']`
so the frontend can warn appropriately. Two new helpers:
- `_classifyReorganizeOutcome(state)` — returns 'success' only
when `result_status === 'completed'` AND `failed === 0`;
'warning' otherwise
- `_formatReorganizeResultMessage(state)` — returns a message
specific to the outcome ("Reorganize skipped — album has no
metadata source ID. Run enrichment first." for `no_source_id`,
etc.)
Zero-failure non-completed runs now show as warnings instead of
green checkmarks.
4. **Bulk mode no longer counts skipped albums as succeeded (🟡
risk).** `_executeReorganizeAll`'s loop was treating any HTTP
200 response as success, ignoring the orchestrator's actual
outcome for that album. Fixed by:
- `_waitForReorganizeComplete()` now resolves with the final
state object (was: void)
- Loop checks `finalState.result_status === 'completed'` AND
`finalState.failed === 0` before counting `succeeded++`;
otherwise increments `skipped` (with a per-album warning
toast) or `failed` accordingly
- Final summary toast now reads
"Reorganized N of M albums, K skipped, J failed" and only
shows green when nothing was skipped or failed
All four addressed in a single commit because they form one
coherent UX-correctness fix — the bug bug (#1) and the count-
overstatement bug (#4) both made the user see "everything succeeded"
when reality was different. Together they make the UI honestly
reflect what actually happened.
Files:
- core/library_reorganize.py — `_finalize_track` returns bool,
`_process_one_track` reads it
- web_server.py — `_reorganize_state['result_status']` populated
from orchestrator's summary on success and on exception
- webui/static/library.js — `_classifyReorganizeOutcome` /
`_formatReorganizeResultMessage` helpers, single-album +
bulk-mode flows both consume them
- tests/test_library_reorganize_orchestrator.py — strengthened
the existing DB-failure test to assert moved/failed counts
Credit: kettui — four PR #377 review comments named all of these
precisely with line numbers and severity.
Reported by kettui on PR #374 review: the inline filter that backed
`set_album_api_track_count` only counted rows where `type_ == 'track'`,
but `discogs_client.get_album_tracks` itself accepts both `'track'`
AND empty `type_` as real songs (line 660: `type_ in ('track', '')`).
Releases where Discogs returns some real tracks with an empty `type_`
field would be undercounted, which would silently disagree with the
repair job's fallback `_get_expected_total` path (which calls into
`get_album_tracks_for_source` and therefore uses the client's count).
Extracted the filter into `count_discogs_real_tracks(tracklist)` —
single source of truth for the rule, testable in isolation, and the
worker call site is now a one-liner that names what it's doing. Also
defensive about the input shape: `type_ == None`, missing field, and
empty/None tracklist all handled cleanly.
10 tests pin the behavior:
- empty/missing/None type_ all count as a real track (the kettui case)
- 'heading', 'index', 'sub_track' excluded
- unknown future type strings excluded conservatively
- realistic multi-disc tracklist with mixed shapes counts correctly
- empty/None input returns 0 without raising
Credit: kettui — the PR #374 review comment that flagged this.
Reported on Discord by winecountrygames. The library "Reorganize" tool
had several layered bugs that all traced to the same root cause: the
endpoint reinvented every wheel post-processing already turns — its own
template engine, its own disc-number resolution from file tags, its own
sidecar sweep, its own collision detection — and each had drifted from
the canonical path used by fresh downloads. Reported symptoms:
- 3-disc Aerosmith deluxe collapsed to a flat single-disc layout
- Half the tracks on other albums silently skipped, no error / no count
- Re-runs left empty leftover album folders cluttering the artist dir
Architecture: stop reinventing wheels. Route reorganize through exactly
the same pipeline downloads use. Per-album:
1. Fetch the canonical tracklist from a metadata source (Spotify /
iTunes / Deezer / Discogs / Hydrabase) using the album's stored
source IDs. New `core/library_reorganize.py::plan_album_reorganize`
does this — primary-source-first, fall through priority chain
unless the user picked a specific source in the modal (strict mode).
2. For each local track, find the matching API entry via a scored
candidate matcher. Score components: exact-title (100),
substring-with-length-ratio (40-90), track-number agreement (20).
Hard reject when the two titles have different version
differentiators (Remix vs no-remix means different recordings,
not annotation drift). Below threshold = unmatched, surfaced as
"not in source's tracklist, left in place" rather than silently
mis-routing.
3. Copy the file to a per-album staging directory, build the same
context dict the import flow builds (`spotify_album` /
`track_info` / etc. with `is_album_download=True` so the path
builder enters ALBUM mode, not SINGLE mode), call
`_post_process_matched_download(...)` — same function fresh
downloads use. Post-process handles tagging, multi-disc subfolder
decisions, sidecar regeneration, AcoustID verification.
4. Read `context['_final_processed_path']` to learn where it landed.
Update `tracks.file_path` in the DB BEFORE removing the original
(DB-update failure leaves the file at both locations, recoverable
via library scan; the reverse would orphan the row). Delete
per-track sidecars (post-process recreates them at the new
destination).
3 concurrent workers per album via ThreadPoolExecutor, matching the
download path's per-batch worker count. State mutations all guarded by
a single lock; staging filenames carry a UUID prefix so concurrent
copies of identically-named source files don't overwrite each other.
Source picker in the modal lets the user choose which source to read
the tracklist from. Two endpoints feed it:
- `/api/library/album/<id>/reorganize/sources` — sources for THIS
album that are both authed AND have a stored ID. For the per-
album modal.
- `/api/library/reorganize/sources` — all authed sources globally.
For the bulk "Reorganize All" modal where per-album ID coverage
varies.
When the user picks a specific source, the orchestrator runs in
`strict_source=True` mode (no fallback chain) — picking Spotify means
"use Spotify or fail", not "use Spotify and silently fall back."
Preview endpoint shares the same planning logic as apply via
`preview_album_reorganize` — the destination path comes from the same
`_build_final_path_for_track` post-process uses, so what you see in
the preview is exactly what you get on apply.
Empty destination folders (from earlier failed runs OR from the
current run when post-process creates a dir then fails AcoustID)
get cleaned up after each successful run: walk up to the artist
folder from any successful destination, prune empty album-sibling
folders one level deep. Bounded scope = won't touch unrelated user
dirs.
Web_server.py shrinks by ~450 net lines. The endpoint handler is now
a thin wrapper that builds injected callables (path resolver, post-
process function, DB updater, empty-dir cleaner), spawns a thread
that calls `reorganize_album()`, and returns. All actual logic lives
in `core/library_reorganize.py` where it's unit-testable without
spinning up Flask.
Frontend cleanup: the per-call template input in both reorganize
modals (per-album and bulk) was redundant — the backend always uses
the configured global download template. Removed the input and the
variables-grid reference UI it was for.
39 new unit tests pin every contract:
- source resolution (no_source_id when album has none, fallthrough
chain when primary returns nothing, strict mode bypasses fallback)
- matcher scoring (exact / substring / multi-disc disambiguation /
smart-quote tolerance / dash-vs-parens / bonus-track substring /
Remix-vs-original differentiator rejection / "Real" doesn't false-
match "Real Real Real" / track-number-only no longer fires)
- file safety (DB-update failure leaves original in place, post-
process failure leaves original in place, post-process exception
caught and original preserved, success removes original AND
updates DB in the right order)
- sidecar handling (per-track .lrc/.nfo deleted on success, kept on
failure; album-level cover.jpg/folder.jpg cleaned only when
directory has no remaining audio)
- staging cleanup (recreated between tracks because post-process
nukes it, dir cleaned up on success AND on failure)
- destination-dir prune (empty siblings removed, real album with
files preserved, no recursive sweep)
- source picker (only authed-with-stored-ID sources for per-album,
all authed sources for bulk; strict mode doesn't fall back)
- concurrency (3 workers in flight, state stays consistent under
races, stop_check cuts off pending tasks)
- preview parity (preview produces same destination as apply for
multi-disc; ALBUM mode not SINGLE mode; unmatched/no-path tracks
surfaced with reasons)
Limitations (deliberate punts, NOT in this PR):
- Renamed local titles on multi-disc albums where track_number
also disagrees: matcher returns nothing (track is "not in
source"). Fixable by using duration_ms as a tertiary signal.
- Per-track in-modal source switching with per-album track-count
hints (would need a second API call before opening the modal).
- UI status panel on the artist page during a run — currently
just toasts. Documented as a follow-up PR.
Files:
- core/library_reorganize.py — new module: plan_album_reorganize,
preview_album_reorganize, reorganize_album, available_sources_for_album,
authed_sources, _score_candidate, helpers for staging/post-
processing/finalizing, sidecar + dest-dir cleanup
- core/metadata_service.py — no changes; reused get_album_for_source,
get_album_tracks_for_source, get_source_priority,
get_client_for_source
- web_server.py — three endpoints (preview / apply / sources GETs)
are thin wrappers; -450 net lines
- tests/test_library_reorganize_orchestrator.py — 39 tests covering
every contract above
- webui/static/library.js — source picker UI in both modals; dead
template input + variables-grid removed
- webui/static/style.css — dropdown option styling fix (white-on-
white was unreadable)
Reported on Discord by winecountrygames — his bug report named the
trigger button (Enhanced view → Reorganize All) and both symptoms
(multi-disc collapse, half-album skip), which let the diagnosis go
straight to the architectural problem.
Reported on Discord by Netti93: with Tidal configured for "HiRes only"
and "Allow Quality Fallback" disabled, tracks were still downloading
successfully — as m4a 320kbps files. Some "successful" downloads were
less than half the file size of the same track pulled via Tidarr/tiddl
from the same Tidal account.
Root cause: Tidal's API silently degrades to the best quality your
account + the track + your region permits. Setting
`session.audio_quality = Quality.hi_res_lossless` and calling
`track.get_stream()` on a track that's only available in AAC returns
an AAC stream with no error. The downloader wrote the m4a file to
disk, the ~7MB size sailed past the 100KB stub threshold, and the
download reported success.
The pre-existing "verify quality wasn't silently downgraded" block
only LOGGED a warning when this happened; it did not fail the tier.
Two knock-on effects:
- Users with "HiRes only, no fallback" got m4a files anyway, which
defeats the setting entirely.
- The worker-level fallback chain (hires → lossless → high → low)
couldn't advance past the first tier, because every tier
"succeeded" at whatever Tidal happened to serve.
Fix: after `track.get_stream()`, compare `stream.audio_quality`
against the tier we asked for using a rank-based ordering:
LOW < HIGH < LOSSLESS < HI_RES < HI_RES_LOSSLESS
- Same tier or higher → accept (so the occasional Tidal upgrade
doesn't get rejected just because it's not an exact match).
- Lower tier → reject THIS tier. The loop `continue`s and the next
fallback tier is tried, or the whole download fails honestly
when the user has fallback disabled. The existing final-error
log already has a hint directing users to enable fallback if
they want automatic Lossless substitution.
- Unrecognized `audioQuality` value (e.g. a new Tidal tier we
haven't mapped) → reject conservatively, so the next fallback
tier gets a chance and the diagnostic log names the unknown
value.
Why the rank-based approach instead of strict equality:
Tidal's API doesn't technically promise an exact-tier match on
serving; on tracks that are flagged in its catalog as a higher
tier, it can serve higher than the session setting. Rejecting
higher-than-asked quality would be user-hostile. And the `HI_RES`
(legacy MQA) value — not in tidalapi's modern `Quality` enum but
possibly still present on old catalog entries — needs to rank
below `HI_RES_LOSSLESS`: users asking for true lossless HiRes
should reject MQA since MQA is a lossy format.
tidalapi's `Quality` enum is a `str` subclass whose VALUES (not
member names) match what the Tidal API returns in the
`audioQuality` field (e.g. `Quality.hi_res_lossless.value ==
'HI_RES_LOSSLESS'`, `Quality.low_320k.value == 'HIGH'`). Both
sides of the comparison are coerced to `str` before use, so the
check is robust to whichever tidalapi version exposes the served
quality as an enum or a plain string.
The check is extracted as `_verify_stream_tier(stream, q_info,
q_key) -> (ok, reason)` at module scope — a pure function with no
I/O, unit-tested independently. Ten tests: match, three upgrade
cases (LOSSLESS → HI_RES_LOSSLESS, LOSSLESS → HI_RES, LOW → any
higher), three downgrade cases (the reported HiRes → AAC, HiRes
Lossless → MQA HiRes, Lossless → AAC), one unrecognized-tier case,
and two defensive paths for older tidalapi builds without
`audio_quality` on the stream object and for QUALITY_MAP entries
that lack `tidal_quality` (e.g. tidalapi wasn't importable at
module load). Test stub updated to use uppercase `Quality` values
matching real tidalapi so case-sensitivity regressions get caught.
Also removed the old codec-string-based warning block — the new
tier check is strictly stronger, and keeping the warning around
would just be dead code waiting to drift out of sync.
Deliberately NOT tackling in this PR (documented as follow-ups):
- Bit-depth verification of HiRes FLAC files via mutagen. The
`stream.audio_quality` tier check catches the main "HiRes
requested, got AAC" case; bit-depth would only matter if Tidal
labeled a stream HI_RES_LOSSLESS but served a 16-bit FLAC
(`Stream.bit_depth` isn't reliable for this — tidalapi defaults
missing `bitDepth` fields to 16, so a trust-the-stream check
would spuriously reject valid HiRes whenever Tidal omits the
field). A proper fix runs mutagen post-download to inspect the
actual file, then decides whether to delete + retry the next
tier — a whole new failure mode with design trade-offs that
deserve their own PR. The support logs don't show this
happening.
- The "manual remap still says Not Found" symptom. Might be
downstream of this same bug (silent-AAC "success" hitting a
later rejection), might be a separate task-state issue. Not
guessing without logs from the retry path.
- Quality-aware stub threshold. 100KB is a reasonable floor for
real stub/preview detection and there's no evidence the
universal threshold is misfiring in the wild.
Field-verified status: desk-verified via unit tests and empirical
checks against a live tidalapi import (confirming the `Quality`
enum's str-subclass behavior). Not yet smoke-tested end-to-end
against a real Tidal account with a HiRes-only-no-fallback
setting — Netti93 or anyone else with that config should notice
either the fix working (non-HiRes tracks fail honestly with a
clear log line) or any regression before wider release.
Files:
- core/tidal_download_client.py — new `_verify_stream_tier` helper
and `_QUALITY_RANK` table at module scope, called in the
download loop after the stream is fetched and before any
bandwidth is spent. Removed the old inline codec-based warning
since the new check supersedes it.
- tests/test_tidal_stream_tier_verification.py — ten tests covering
match / upgrade / downgrade / unknown / defensive paths.
- tests/test_tidal_search_shortening.py — fake `Quality` values
brought in line with tidalapi's real values so both files share
a consistent stub regardless of pytest collection order.
- webui/static/helper.js — WHATS_NEW entry under 2.40 describing
the rank-based tier comparison.
Reported on Discord by Netti93 — the "same account works via
Tidarr" comparison narrowed the cause to SoulSync's download path
rather than an account/region issue.
Reported by sassmastawillis: the Album Completeness maintenance job
scans 3127 albums in 0.1 seconds and reports 0 findings — for every
user, regardless of whether their library is actually complete.
Restoring an older DB surfaced 7 correct findings, so the code logic
works; the DB state is what's making everything look complete.
Root cause: `albums.track_count` is only ever written by server-sync
paths — Plex's `leafCount`/`childCount` and SoulSync standalone's
`len(tracks)`. It's the OBSERVED count of tracks SoulSync has indexed,
which is always exactly what `COUNT(tracks)` returns for that album.
The completeness job treated it as the EXPECTED total and compared it
against the observed count. They're equal by construction, so
`actual >= expected` is always true: skip, 0.1s scan, 0 findings.
Fix: new `api_track_count INTEGER` column on `albums`, written only by
metadata-source code paths. Populated in two places so the scan is
fast and the fallback is robust.
1. Enrichment workers — shared helper `set_album_api_track_count`
in `core/worker_utils.py`. Called by each worker's existing
`_update_album` method alongside its other album-column UPDATEs:
- spotify_worker: `album_obj.total_tracks` from the Spotify Album
dataclass (already in hand, zero new API calls)
- itunes_worker: same, from the iTunes Album dataclass
- deezer_worker: `nb_tracks` from full_data, falling back to
search_data when the full lookup didn't run
- discogs_worker: count of tracklist rows where `type_=='track'`
(Discogs tracklists interleave heading and index rows that
shouldn't count as songs)
Helper skips the write on zero/None/negative/non-numeric inputs
so a source lacking track info can't clobber a good value a
different source already wrote. Caller owns the transaction —
helper just queues an UPDATE on the caller's cursor without
committing, so it batches cleanly with each worker's existing
multi-UPDATE pattern.
Hydrabase worker deliberately not touched — it's a P2P mirror
that doesn't write album metadata to the local DB. Hydrabase-
primary users hit the fallback path below.
2. Album Completeness repair job — new `al.api_track_count` column
in the SELECT, read first in the scan loop. On miss (album never
enriched, or enrichment workers haven't run yet on a fresh
install), falls through to the existing `_get_expected_total()`
API lookup and persists the result via the same shared helper
(wrapped in connection/commit management since the repair job
runs outside a worker's batched transaction).
Also removed `al.track_count` from the scan's SELECT — now unused
since the observed count was the whole source of this bug, and
leaving a dead SELECT would invite a future engineer to re-introduce
the same comparison.
Help text on the job card was reworded so it honestly describes
current behavior ("counts cached during normal enrichment are used
when available; otherwise the job queries a metadata source
directly") rather than the old "active provider first, then others
as fallback" phrasing, which doesn't match how the cache actually
fills — any enrichment worker that runs can populate it, and the
last writer wins. Document-only follow-up if this edge case ever
bites in practice: add a `api_track_count_source` column so the
scan can prefer the configured primary source's count over others
(e.g. deluxe vs. standard edition mismatches). Not worth the
complexity today.
For existing users, the first completeness scan after upgrade is
fast to the extent their library is already enriched: the workers
already ran and populated `api_track_count` on their normal schedule.
For brand-new installs, the scan's fallback path handles the cold
start — slower, but correct, and subsequent scans are fast.
Does NOT affect:
- Download / post-processing / wishlist / sync code paths — none
of them read `track_count` for completeness semantics.
- Plex / Jellyfin / Navidrome / standalone sync — still write
`track_count` exactly as before; `api_track_count` is a separate
column they never touch.
- Other repair jobs.
- Any UI path — same finding schema, just correct counts now.
Files:
- database/music_database.py — idempotent migration adding
`api_track_count INTEGER DEFAULT NULL` to the existing album-column
check block.
- core/worker_utils.py — new `set_album_api_track_count` helper with
the documented skip-on-bad-input contract.
- core/spotify_worker.py, itunes_worker.py, deezer_worker.py,
discogs_worker.py — one-liner call from each `_update_album`.
- core/repair_jobs/album_completeness.py — scan uses the cache;
fallback path persists API-lookup results via the shared helper;
help text updated to match actual behavior.
- tests/test_worker_utils_album_track_count.py — 9 tests covering
the helper's write/skip contract + no-commit invariant.
- tests/test_album_completeness_job.py — 2 tests for the repair
job's fallback-path wrapper.
- webui/static/helper.js — WHATS_NEW entry.
Credit: sassmastawillis spotted the bug; the "restored older DB
finds 7 albums" signal pinpointed DB state over code logic and
made the diagnosis tractable.
Typing "michael jackson" returned 7 identical-looking cards because
MusicBrainz has many different PEOPLE sharing a canonical name — the
King of Pop plus a NZ poet, a photographer, a mashup artist, a
didgeridoo player, and more, all scoring 80+ on exact-name match.
All 7 passed the score filter. All 7 rendered with the same
fallback image because iTunes/Deezer only know the famous one.
Fix dedupes by normalized (lowercase, whitespace-trimmed) name before
building Artist dataclasses. Keeps the highest-scoring entry per name,
so the King of Pop (score 100) wins over the others (all score 80-81).
Artists with genuinely different names stay separate — a search for
"the beatles" still surfaces tribute bands if they're above threshold.
Implementation note: fetch `max(limit*3, 10)` from MB instead of
`limit` directly, so the dedup pool is large enough to still return
`limit` distinct artists after collapsing duplicates. Previously the
raw fetch was capped at the caller's limit, which would have left
fewer-than-requested results after dedup for common names.
3 new tests (49 total):
- Dedupe collapses 5 same-named entries to 1 (keeps highest score).
- Dedup key is case-insensitive and whitespace-normalized.
- Dedup preserves distinct names ("The Beatles" vs "The Beatles Revival"
stay separate).
Live-verified: "michael jackson" now returns 1 card, "kendrick lamar"
returns 1 card.
Credit: kettui spotted duplicate Michael Jackson cards in the search UI.
Three bugs from kettui's follow-up review pass on the MusicBrainz
search PR, all fixed in one commit because they share UI context.
1. Missing artist images on MB artist results
MusicBrainz doesn't store artist images directly. My earlier commit
returned `image_url=None` on every artist result and trusted the
frontend's lazy-loader — but the lazy-loader's `/api/artist/<id>/image?
source=musicbrainz` endpoint had no handler for MusicBrainz, so it
silently returned None and the emoji placeholder stayed.
Fix plumbs the artist name through:
- `renderCompactSection` stashes `data-artist-name` on artist cards.
- `search.js` and `downloads.js` lazy-loaders pass `name=<artist>` as a
query param.
- `/api/artist/<id>/image` accepts an optional `name` param.
- `metadata_service.get_artist_image_url` has a new `musicbrainz`
branch: since MB has no artist art, it searches fallback sources
(iTunes/Deezer by configured priority) for the artist name and
returns the first image found.
Verified live — Metallica/Kendrick Lamar/Daft Punk all resolve to
Deezer artist images via the name lookup.
2. total_tracks off-by-one on tracks with a release
`_recording_to_track` initialized `total_tracks = 1` and then summed
media track-counts on top. For an 11-track album, it reported 12. An
adapter-level regression introduced when the recording-projection
helper was extracted during the main MB refactor.
Fix: initialize at 0, sum normally. Standalone recordings with no
release (can happen for uncredited remixes etc.) still report 1 via
an explicit fallback — so the existing "single track" case isn't
broken.
3. "Artist Album Title" queries buried specific albums in the
discography list
Bare-name queries like "The Beatles Abbey Road" used to resolve "The
Beatles" as the artist and then browse their full discography — Abbey
Road was buried alphabetically among 200+ releases instead of being
the top result.
Fix adds a title-hint extractor. When the query starts with the
resolved artist name followed by more words, the trailing portion is
treated as a title hint. Browse results are filtered to those whose
release-group title contains the hint. If the filter matches nothing,
falls back to text-search with the hint as the title (the "keep the
old split-by-whitespace fallback" path kettui called for). If text-
search also misses, shows the full discography rather than nothing.
10 new tests in tests/test_musicbrainz_search.py (46 total):
- Title-hint extractor: basic match, case-insensitive, whitespace
tolerance, bare-artist-no-hint, artist-not-prefix-no-hint, word-
boundary required (no false splits on "Metallicasomething").
- Browse filtering by title hint.
- Text-search fallback when the title hint matches nothing in browse.
- Bare-artist queries return the full discography unfiltered.
- total_tracks for single-release, multi-disc, and no-release cases.
Clicking a MusicBrainz album returned 404 because the browse-based
search path now stores release-GROUP MBIDs in Album.id, but `get_album`
still hit `/ws/2/release/<mbid>` directly. Release-group MBIDs don't
resolve as release MBIDs — MB 404s. User log:
GET /api/spotify/album/b88655ba...?source=musicbrainz → 404
Error fetching release b88655ba...: 404 Client Error
The fix requires a two-step resolution for the new browse path:
1. Look up the release-group with `inc=releases+artist-credits` to get
the list of releases inside (original + reissues + regional + promo
editions). MB release-groups routinely hold 5-20 releases.
2. Pick a representative release: prefer Official status over Promo,
prefer releases with a real tracklist over stubs, then earliest date.
3. Fetch that release's full tracklist via `get_release`.
Two extra seconds at the 1-rps rate limit, but it's on click, not on
search results rendering.
Structure:
- New `MusicBrainzClient.get_release_group(mbid, includes)` method.
- New `_pick_representative_release(releases)` helper encapsulates the
ranking logic.
- Tracklist projection extracted into `_render_release_as_album` so
both paths share the same shape construction.
- `get_album` tries release-group first; falls back to direct release
lookup when the MBID turns out to be a release from the text-search
fallback path.
- Canonical Album.id stays the release-group MBID so a re-fetch with
the same URL hits the same code path idempotently.
3 new tests (now 33 total):
- End-to-end release-group → release resolution with mocked client
- Fallback to direct release lookup when rg lookup misses
- Representative-release picker ranks correctly
Verified against live API with the exact MBID that 404'd for the user
(b88655ba... for DAMN. by Kendrick Lamar): now returns in 1.2s with
the full 14-track listing (BLOOD., DNA., YAH., ELEMENT., FEEL., ...).
Three related fixes to make album/track results look like a real
artist discography instead of a firehose of fan-compiled bootlegs.
1. Drop 'compilation' from the release-group browse primary-type filter.
MB's OR filter (`type=album|ep|single|compilation`) silently breaks
when 'compilation' is included — Metallica drops from 1076 matches
to 82 because `compilation` is a SECONDARY type on MB, not a primary
type. The invalid value corrupts the filter for all types, not just
itself. Now we request `type=album|ep|single` which returns the full
1076; actual compilations (primary=Album + secondary=[Compilation])
are filtered out by the studio-preference logic below.
2. Filter release-groups with non-studio secondary-types
(Live/Compilation/Soundtrack/Remix/Demo/Mixtape/Interview/Audiobook/
Audio drama). For Metallica, the first 100 browse results are 12
studio albums + 83 live bootlegs + 5 compilations — without this
filter the Albums section was dominated by 2019-2021 broadcast
recordings. Falls back to the unfiltered list if filtering leaves
the result set empty (covers live-only niche artists).
3. Sort chronologically ASC by first-release-date. Wikipedia-style
discography ordering — debut album on top, then chronological.
Previous DESC sort put the most recent release on top which, for
prolific artists, meant 2020s material before their classics.
Track side of the same fix:
- Re-orders each recording's `releases` array to put studio releases
first before `_recording_to_track` picks up the first release for
album context. Without this, MB's arbitrary release order often
buried the canonical studio album under random live bootlegs.
- Filters out recordings that only exist on live/compilation release-
groups (keeps the ones with at least one studio release). Falls
back to the full set if the artist has no studio recordings at all.
- Sorts recordings by earliest studio-release year ASC so classic
tracks surface first.
Smoke test against live MB API confirmed:
- Artists: [Metallica score=100]
- Albums: Kill 'Em All (1983) → Ride the Lightning → Master of Puppets
→ ...And Justice for All → Metallica (Black Album) → Load → Reload
→ St. Anger → Death Magnetic → Lulu (2011)
- Tracks: real Metallica recordings (Killing Time, Nothing Else
Matters, Creeping Death, etc.) — a few remastered demos still leak
in where MB metadata quality is thin, but the bulk is correct.
- Total latency: 3.5 seconds.
4 new tests covering the studio filter, live-only fallback, preferred
release ordering, and live-only recording exclusion.
Credit: kettui flagged the poor MB results during PR #371 review.
The previous commit's `browse_artist_recordings` call passed
`inc=releases+artist-credits` — but MusicBrainz's recording browse
endpoint rejects `inc=releases` with HTTP 400. The adapter's error
handler returned an empty list, so the Tracks section stayed empty
even though the fix was supposed to populate it.
Browse without release info is useless for our search UI (tracks
would render with no album), so swap to the fielded Lucene search
`arid:<mbid>` on the `/recording` endpoint. That's the canonical MB
pattern for "find recordings by this artist WITH release context":
- arid: search accepts the artist MBID and returns recordings with
`releases` (release-group, date, media) embedded in each result.
- One API call per lookup, same as browse would have been.
Renamed the method to `search_recordings_by_artist_mbid` so the name
matches its behaviour — it's a search, not a browse. Adapter updated
to call the new name; tests updated to match.
Verified against the live API: Metallica's MBID returns 5 recordings
in ~1.8 seconds (vs the previous 400 error).
26 new unit tests in tests/test_musicbrainz_search.py covering:
- Cover Art URL construction (release + release-group scope, empty MBID,
unknown scope fallback)
- Structured query splitting (hyphen, en-dash, em-dash, bare name, no
false-positive splits on hyphens-inside-words)
- Artist search: score filtering, strict=False call contract, exception
handling, genre extraction from MB tags, mbid/name validation
- Top-artist resolver: memoization by normalized query, sub-threshold
returns None, negative-result caching, empty-query short-circuit
- Album search routing: bare query → browse path, structured query →
text path, no-artist-match falls back to text, text path score filter
- Track search routing: browse path, dedupe-by-title across
live/compilation variants, structured query → text path, text path
score filter
All mock the underlying MusicBrainzClient — no network calls.
Also adds a WHATS_NEW entry under 2.40 explaining the three user-visible
changes: Artists section now populates, album/track results match the
searched artist instead of random title collisions, and search completes
in ~3 seconds instead of 30+.
JohnBaumb's review: "If we're going to refactor the web_server.py soon,
might as well start moving stuff away from web_server.py in our PRs.
_build_source_only_artist_detail, make it a module, it's perfect."
This continues the pattern the prior commit started with the source-ID
lookup helpers: move the pure data-building logic to a side-effect-free
core module, leave a thin wrapper in web_server.py that bridges the
Flask response and the module-global clients.
**core/artist_source_detail.py** — pure function that takes the artist id,
name, and source plus dependency-injected per-source clients (spotify,
deezer, itunes, discogs) and a Last.fm API key. Returns
(payload_dict, http_status) so it isn't coupled to Flask.
**web_server.py wrapper** — builds the client bag from the module globals
(checks Spotify auth, constructs the Discogs client from the configured
token, reads the Last.fm API key) and wraps the core return in jsonify.
147 lines of logic go away from web_server.py; the 24-line wrapper is
purely glue.
**tests/test_artist_source_detail.py** — 21 focused tests covering the
response envelope, the source-specific ID-field stamping for all six
supported sources, the dedup_variants=False contract (the behaviour
that originally motivated the split of MetadataLookupOptions), per-source
genre/follower extraction with safe handling of missing or throwing
clients, and the Last.fm enrichment branch including the no-key and
error-path cases. Runtime 0.26s.
Cin's review note: typing artist_name as plain `str` forced callers
that didn't have a name to pass `""` as a placeholder, which leaks the
parameter's emptiness contract into every call site and reads badly in
tests. Switching to `Optional[str] = None` lets callers omit it.
The function body's `if artist_name and active_server:` check already
handles None and "" identically, so no body changes were needed. Tests
that previously passed `artist_name=""` drop the argument; one new test
covers the omitted-arg path explicitly.
The web_server.py wrapper takes the same default for symmetry.
Cin pointed out that the prior version of test_artist_source_lookup.py
AST-parsed web_server.py to verify a constant and to string-match a
function's response keys. That was a workaround for the fact that
web_server.py can't be imported at test time (it boots Spotify,
Soulseek, Plex, etc.) — the right answer is to move the logic into a
side-effect-free module so it can be imported and tested directly.
This commit:
- adds core/artist_source_lookup.py containing the SOURCE_ID_FIELD
map, the SOURCE_ONLY_ARTIST_SOURCES set, and find_library_artist_for_source
- replaces the inline definitions in web_server.py with imports +
a thin wrapper that injects the active media server
- rewrites the tests to import from the core module directly:
* mapping correctness is now a plain equality assertion
* lookup behaviour is exercised against a real MusicDatabase
* the AST parse and the string-matching contract test class are
gone
- drops the _build_source_only_artist_detail contract test entirely
(the weakest of the four — it was just string-matching the function
body); when that function moves to core/ it can get a real
behavioural test alongside.
Test runtime drops from ~161s to ~5.8s. All 18 tests pass.
Four targeted backend tests for behaviour added during the Search/Artists
unification work:
1. _SOURCE_ID_FIELD mapping is parsed out of web_server.py via AST and
compared against an explicit expectation, so silent renames break the
test instead of silently breaking library-upgrade detection.
2. Every column in _SOURCE_ID_FIELD must exist on the real artists table
after migrations run. This is the schema-vs-query contract that the
`deezer_artist_id` typo would have failed instantly.
3. The two queries from the watchlist-config enrichment path execute
verbatim against a fresh DB — separate ones for the artists table
(deezer_id / discogs_id) and the watchlist_artists join (deezer_artist_id).
Documents the column-name split that caused the original bug.
4. Static contract test for _build_source_only_artist_detail's response
shape: every JSON key the frontend reads (success/artist/discography/
image_url/server_source/genres/lastfm_*) must appear in the function
source, plus the dynamic source-id stamp and the dedup_variants=False
opt-out.
Plus a behavioural test for MetadataLookupOptions.dedup_variants=False
in test_metadata_service_discography.py — proves the flag actually keeps
variant releases that get_artist_detail_discography would otherwise
collapse to a single canonical entry.
The three discovery-pool tests hardcoded release_date strings
("2026-04-01", "2026-04-16") that were checked against a rolling
`datetime.now() - timedelta(days=7)` (or 21-60 day) cutoff in the
scanner. Once the wall clock advanced past the cutoff window the
releases were filtered out and the assertions failed — Python 3.11
Linux CI was already past 2026-04-23 UTC.
Replace the hardcoded values with a module-level
`_RECENT_RELEASE_DATE = now - 2 days` so the fixtures stay inside
every cutoff window regardless of when the suite runs.
Completes the artist-detail unification. Source artists now land on
the same /artist-detail page as library artists (with the source-aware
backend endpoint from earlier this session handling the data fetch).
The inline Artists page is gone — artists.js deleted, #artists-page
HTML block removed, /artists URL aliases to /search.
Source-artist callsites re-migrated from selectArtistForDetail to
navigateToArtistDetail (search results, global widget, download
modal, Discover hero / Your Artists cards / artmap context / genre
deep-dive, watchlist artist detail).
Visual upgrade to standalone hero: added .artist-detail-hero-bg +
.artist-detail-hero-overlay (blurred image bg, dark gradient — same
treatment as the inline page). library.js sets the bg image when
loading an artist.
Library-only UI hidden via CSS for source artists (existing rules
from the previous commit cover Enhanced toggle, Status filter,
completion bars, enrichment coverage, Top Tracks sidebar, Radio /
Enhance buttons).
Final 2 helpers (lazyLoadArtistImages used by wishlist-tools,
showCompletionError used by completion checker) moved from
artists.js into shared-helpers.js. The inline-page candidate set
was dropped from _resolveSimilarArtistsTargets.
init.js: 'artists' alias added at top of navigateToPage (same
pattern as the existing 'downloads' alias). 'case artists:' handler
removed from loadPageData. _getPageFromPath now maps artist-detail
to library as its parent (matches the existing nav highlight at
init.js:2161).
tests/test_script_split_integrity.py: artists.js removed from
SPLIT_MODULES; KNOWN_CROSS_FILE_DUPES updated to point escapeHtml
at shared-helpers.js instead of artists.js. 354/354 tests pass.
Net delta: -1700 lines.
Stays at 2.39. Once you've verified end-to-end (library artist ->
hero looks like inline visual; source artist from Search -> same
page, similar artists works, no 404s; /artists URL -> /search), a
follow-up commit bumps to 2.40 with the full WHATS_NEW entry that's
already prepped.
Part D + E of the deferred cleanup + the final version bump that
publishes the whole Search/Artists unification project.
Deletions:
- webui/static/artists.js (1903 lines) — removed entirely. The 2
remaining externally-referenced helpers (lazyLoadArtistImages +
showCompletionError) moved into shared-helpers.js first.
- webui/index.html — 140-line #artists-page HTML block and the
<script src="artists.js"> tag both removed.
init.js wiring:
- 'case artists:' removed from loadPageData switch (no page to init).
- navigateToPage top-level alias extended: 'artists' → 'search'
(same pattern as the existing 'downloads' → 'search' alias).
Legacy /artists bookmarks land on the unified Search page, the
natural place to find an artist now.
- _getPageFromPath now maps artist-detail → library as its parent
(was artists). Matches the existing library-nav-highlight at
init.js:2161.
Version bump:
- _SOULSYNC_BASE_VERSION 2.39 → 2.40.
- WHATS_NEW entries lose the 'unreleased' scaffolding and gain a
new top entry summarizing the unified artist-detail page + the
final artists.js retirement.
- version-info modal gets a 'Search & Artists Unification' section
at the top.
- The _getLatestWhatsNewVersion filter added during the unreleased-
tracking phase is rolled back — entries now display as soon as
they land in WHATS_NEW, matching the pre-unification behaviour.
Test suite:
- tests/test_script_split_integrity.py SPLIT_MODULES updated:
'artists.js' dropped, 'shared-helpers.js' added. escapeHtml's
cross-file dupe list entry updated to reference shared-helpers.
- 354/354 tests pass.
User-visible result after this commit:
- Sidebar: Search, Downloads, Discover, Library, Wishlist, etc. —
no more Artists entry.
- Click any artist anywhere: lands on the same /artist-detail page.
- Search page has a source dropdown; Soulseek is just another option.
- Legacy /downloads and /artists URLs alias to /search.
- Version button shows v2.3 (Docker major); "What's New" panel
opens to the unification summary.
Closes the project Cin requested in Discord. Future work: source-aware
/api/artist-detail could be extended to fall back through the whole
source priority chain when a specific source is given but returns no
discography. Not needed for the current flows.
Part C of the deferred unification cleanup. The Artists page is no
longer in the sidebar, but its JS file can't be deleted yet because
it houses ~20 general-purpose helpers that other modules depend on
(escapeHtml used in 229 places, service-status polling, image-colour
extraction, download-bubble infrastructure, discography completion
checking, enrichment card rendering).
Moved all non-page-specific code from artists.js into the new
webui/static/shared-helpers.js — pure copy/paste, zero logic change.
Two contiguous blocks extracted:
Block A (lines 1097..1398 of original artists.js): discography
completion suite — checkDiscographyCompletion, handleStreaming-
CompletionUpdate, cacheCompletionData, updateAlbumCompletion-
Overlay, getCompletionStatusText, setAlbumDownloadedStatus,
setAlbumDownloadingStatus.
Block B (lines 2206..EOF of original artists.js): download-bubble
infrastructure (artist + search + Beatport clusters with their
snapshot/hydrate/modal/monitor helpers), openDownloadMissingModal-
ForArtistAlbum, image-colour extractor and dynamic-glow helper,
escapeHtml, service-status polling, renderEnrichmentCards.
Function declarations in a plain <script> tag are auto-global, so all
existing callers continue to resolve without any import/export
changes. Load order in index.html: shared-helpers.js loads right
after core.js (which defines the artistDownloadBubbles / search-
DownloadBubbles / beatportDownloadBubbles globals these helpers use).
Stats:
artists.js: 4638 → 1903 lines (-2735)
shared-helpers.js: new, 2762 lines
No function duplicated between the two files
All 357 tests pass (3 new from split-integrity parametrization)
What's left in artists.js is purely the Artists page — search UI,
detail view, state switching, watchlist button, discography loading.
All of that is reachable only by typing /artists in the URL bar
since the sidebar entry was retired in Phase 4b. Parts D + E will
delete that remainder and the file itself.
Phase 3b of the Search/Artists unification. The Search page's
internal id was 'downloads', which clashed with the actual Downloads
page (id 'active-downloads') and confused anyone reading the code.
Renamed to 'search' across HTML, navigation, DOM selectors, and the
deep-link route list.
Backwards compat: navigateToPage('downloads') aliases to 'search'
at the top of the function; /downloads URL still serves index.html
and the client router resolves the page correctly; profile ACL
checks accept both 'search' and 'downloads' so existing profiles
with 'downloads' in allowed_pages keep working without migration.
Sidebar label unchanged. Zero visual change — pure internal tidy.
Tidal's search engine chokes on long queries with multiple qualifier
words (remix credits, edit labels, bonus-disc markers). User reported
case: "maduk transformations remixed fire away fred v remix" returns 0,
but shortening to "maduk transformations remixed fire away" works.
Behaviour change:
- On a 0-result search, retry with progressively-shortened variants
(capped at 5 total attempts, 100ms pause between).
- Variants (in priority order):
1. strip trailing "(...)" / "[...]"
2. strip all parentheticals/brackets
3-5. drop last 1 / 2 / 3 tokens
6. keep first half of tokens (rounded up)
- Dedupes so identical variants don't re-query.
Safety — qualifier-aware filter:
- Variant keywords (Live / Remix / Acoustic / Extended / Unplugged /
Instrumental / Karaoke / etc.) are extracted from the original query
using word-boundary match so "edit" doesn't match "edition" and
"mix" doesn't match "remixed".
- If the original query carries any qualifiers, fallback results MUST
contain those qualifiers in their track names — otherwise a shortened
query could silently downgrade "Song (Live)" to the studio "Song".
- Tracks that fail the filter are dropped. If no variant produces
qualifier-matching tracks, returns ([], []) — the same outcome as the
original code, so no regression.
Contract preservation:
- Never raises to caller (outer try/except catches orchestration errors).
- Returns ([], []) on any failure path, same as original.
- Original-query successes take the same code path as before — no
behavioural change for queries that already work.
- Defensive guards for None/empty/non-string query (early return).
Logging:
- Preserves original warning/error/info messages for back-compat log
scraping.
- Adds fallback-success INFO log ("Tidal fallback query succeeded: ...")
so successful retries are visible in production logs.
- Adds qualifier-filter INFO/DEBUG logs with kept/total counts.
- Per-attempt exception logs at DEBUG (not ERROR) to avoid noise when
retries succeed.
- Traceback preserved on final failure.
Tests (16 regression tests in tests/test_tidal_search_shortening.py):
- Skowl's reported query reaches his working variant within the cap.
- Paren/bracket stripping priority.
- Short queries produce no variants.
- All variants unique (dedup guard).
- Progressive token drops present for long queries.
- Qualifier extraction is word-bounded (no "edit" in "edition").
- Qualifier extraction is case-insensitive.
- Track name filter requires ALL qualifiers.
- Empty-qualifier list passes every track (original-query behaviour).
All 292 tests pass.
Two bugs reported in issue #320:
1. Auto-watchlist scan bypassed Global Override settings.
scan_watchlist_profile applied _apply_global_watchlist_overrides, but
the scheduled auto-scan called scan_watchlist_artists directly —
bypassing the override. Users who unchecked "Albums" or "Live" under
Watchlist → Global Override still saw full albums and live tracks
added during nightly scans (per-artist defaults, which include
everything, won).
Moved override application into scan_watchlist_artists itself so
every entry point respects it. scan_watchlist_profile now forwards
the apply_global_overrides flag through to avoid double-application.
2. is_live_version (watchlist + discography backfill) and
live_commentary_cleaner's content patterns used bare \blive\b, which
matched verb uses like "What We Live For" by American Authors,
"Live Forever" by Oasis, "Live and Let Die" by Wings.
Tightened the live patterns to require clear recording context:
(Live) / [Live Version] / - Live / Live at|from|in|on|version|
session|recording|performance|album|show|tour|concert|edit|cut|take
/ In Concert / On Stage / Unplugged / Concert.
Locked in 11 regression tests covering the reported false positives
(What We Live For, Live Forever, Living on a Prayer, Live and Let Die)
and the reported true positives (Dimension - Live at Big Day Out,
MTV Unplugged, etc.).
Version bumped to 2.37 with changelog entries.
The _DummyConfigManager stubs in test_metadata_service_musicmap.py and
test_metadata_service_artist_image.py were missing get_active_media_server(),
which the existing test_metadata_service_discography.py dummy provides.
Both files install their dummy via sys.modules["config.settings"] with an
"if not in sys.modules" guard, so whichever test file loads first wins.
When the new files load alphabetically before discography, the limited
dummy persists and later tests hit AttributeError on get_active_media_server.
Adds the same get_active_media_server method to both dummies so all three
test files are equivalent and test ordering no longer affects outcomes.
- Move /api/artist/<artist_id>/image resolution into core.metadata_service.
- Resolve artist artwork through source priority, with explicit source/plugin overrides preserved.
- Keep Spotify call tracking inside the client layer to avoid double counting.
- Update similar-artist lazy loading to pass source context and add service coverage.
- Relocate the streamed MusicMap similar-artist flow out of web_server.py and into core.metadata_service.
- Match similar artists through the configured source-priority chain instead of assuming Spotify first.
- Add iTunes artwork fallback so streamed artist payloads still carry image_url when search results are sparse.
- Cover the new service behavior with tests.
The unknown_artist_fixer was updated to use deezer_id (matching the
actual tracks table column) but the test still passed deezer_track_id
in the track dict, causing the deezer lookup to miss and fall back
to Spotify.
- Move album-track resolution into metadata_service
- Use the configured provider order instead of Spotify-first branching
- Switch the frontend to the unified /api/album/<id>/tracks endpoint
- Add tests for source-priority lookup, DB resolution, and formatting
- pass provider-specific artist ids into the source-priority discography lookup
- stop relying on the local library artist id when querying external metadata
- add a regression test for source-specific artist id resolution
- Stop passing in spotify_id as the id in the UI, use the actual db id instead
- Fixes an issue where albums for another artist would end up being returned for the actual searched artist
- Remove the redundant artist_id filtering code
- Fixes an issue where not-currently-owned albums would be filtered out from the results, even if they were successfully fetched from the configured metadata provider
Coverage for fix 2.1:
TestResolveDbTrackIdsBatch:
- Batch returns the same (title, artist) -> id mapping as the
per-event lookup would have
- Case-insensitive matching preserved
- Empty event list returns an empty dict
- Events without a title are skipped
- A cursor-execute counter proxy confirms 50 events trigger exactly
one SQL query (not 50)
TestMapPlayCountsToDb:
- Returns updates only for server IDs that exist in tracks
- Empty input returns an empty list
- 30 server IDs trigger one batched query
TestEnrichStatsItems:
- Populates image_url / id / artist_id on matching artists, albums,
and tracks; skips rows with no match
- Empty or missing top_* lists are safe
- Three batched queries total (one per section) regardless of the
number of items in each list
Coverage for fix 4.2:
- _cleanup_old_requests evicts entries older than _MAX_REQUEST_AGE
and leaves fresh entries intact; returns the number removed
- Empty map is safe (no error)
- start_cleanup_thread is idempotent (returns False on second call)
- stop_cleanup_thread joins the thread and clears the handle
- The thread actually evicts stale entries on wakeup
- stop signals the thread to exit promptly via the stop event
instead of waiting for the next interval
Covers fix 4.1:
- Default limit (100) applied when no params given
- limit and offset slice correctly without overlap between pages
- status param accepts single or comma-separated values
- Unknown status returns empty list with total=0
- limit is clamped to a max of 500
- Negative or non-integer limit/offset fall back to safe defaults
- Tasks are returned newest-first by status_change_time
Coverage for fix 1.1:
TestBackfillMigration verifies the one-shot migration sets
match_status='matched' for rows that already have a populated
external ID (lastfm_url, musicbrainz_release_id,
musicbrainz_recording_id, tidal_id, qobuz_id) but NULL match_status,
and leaves rows without an ID untouched.
TestGetExistingIdColumnMapping verifies lastfm_worker reads
lastfm_url for all entity types and musicbrainz_worker reads the
correct per-type column (musicbrainz_id / musicbrainz_release_id /
musicbrainz_recording_id).
TestLastFMWorkerMarksMatched / TestTidalWorkerMarksMatched /
TestQobuzWorkerMarksMatched / TestMusicBrainzWorkerMarksMatched
verify each worker's _process_* short-circuit path sets
match_status='matched' (and does not re-call the external API) when
the entity already has an ID populated.
Covers original ordering preservation, partial/full hit thresholds,
empty result_ids, TTL expiration, cache miss behavior, and a
round-trip count assertion confirming 50 entities resolve in a
single SELECT (not 50).
8 test files had _DummyConfigManager missing get_active_media_server(),
causing failures when pytest ran them before the test file that had it.
Whichever file set sys.modules first won, and the incomplete dummy broke
later tests. Also fix script.js read_text() missing encoding='utf-8'
which failed on non-UTF-8 default locales.
- broaden the artist-detail dedup helper to catch trailing parenthetical edition and remaster variants
- keep the legacy hyphenated suffix fallback for older metadata
- add regression coverage for language-specific Edition and remaster cases
- move artist-detail discography resolution onto the shared source-priority metadata service
- keep the variant dedup helper in the UI-facing adapter
- pass the chosen source through completion checks
- add coverage for the new adapter and dedup behavior
Move completion checks into metadata_service and make them follow the configured metadata source priority.
Drop the old test-mode path, remove the web_server wrapper indirection, and keep artist inference on explicit release metadata instead of guessing from a track search.
Add coverage for the source-priority completion behavior and the safer artist-name handling.
Move Hydrabase availability checks into metadata_service so source resolution owns the policy. Keep web_server delegating to the centralized helper and add tests for the enabled/disabled cases.
Move artist discography resolution into core metadata_service, introduce MetadataLookupOptions, and keep web_server focused on request handling. Add focused tests for the new service boundary and preserve current fallback behavior for now.
Repair-worker album fills now generate explicit track IDs when copying rows, instead of relying on SQLite auto-assignment that no longer exists for TEXT primary keys. The unknown-artist fixer now does the same for new artists.
Also add a regression test for the album-fill copy branch and keep the AcoustID scanner resilient to legacy null-ID rows.
Switch similar-artist backfill to the shared provider-priority flow instead of assuming iTunes as the fallback.
Reuse the generic metadata search helpers, keep a compatibility alias for the old helper name, and update the scanner tests to cover the new path.
Add a regression test that verifies backfill walks each available fallback provider and persists the resolved IDs per source.
Shift similar-artist lookup to the shared metadata provider priority flow.
Use generic provider clients for search and metadata extraction instead of
branching on Spotify/iTunes-specific paths.
Add a regression test that verifies MusicMap matching queries the provider
priority list and preserves canonical metadata from the best match.
Make discovery pool population and curated playlists follow the configured metadata source order. Keep Spotify strict where fallback would corrupt source-specific IDs, and trim fan-out with smaller similar-artist samples and page caps. Leave the remaining incremental path for follow-up.
Reduce request volume in the discovery helpers while keeping the source-priority model intact.
- make cache_discovery_recent_albums source-priority aware
- cap Spotify artist-album pagination in the discovery and incremental paths
- reduce the similar-artist sample size for the cache-refresh helper
- keep Spotify strict where fallback would contaminate source-specific IDs
- add regression coverage for source order, strict Spotify lookups, and pagination caps
Watchlist scanner: empty discography (no new releases in lookback) was
treated as API failure, causing "Failed to get artist discography" for
artists like Kendrick Lamar who simply had no recent releases. Now
distinguishes None (API failure → try next source) from [] (success,
no new tracks). Spotify backfill now uses the authenticated client
instance instead of creating a fresh unauthenticated one.
Wishlist nebula: album remove now sends album_name (API updated to
accept album_name as fallback alongside album_id). Track remove
re-renders the nebula after deletion. Toned down processing pulse
animation.
Updated test to verify fallback triggers on API failure (None), not
on empty results.
Make discovery pool population respect provider priority while keeping Spotify strict, and reduce unnecessary request volume in the hot discovery paths.
- keep discovery fan-out source-priority aware
- preserve cache use where freshness is not required
- cap Spotify artist-album pagination in discovery and cache refresh paths
- keep incremental release checks to a single page, since they only need the newest releases
- add regression coverage for provider order, strict Spotify handling, and pagination caps
Resolve Spotify artist matching through the exact Spotify client only, so watchlist ID backfill cannot drift to fallback-provider results. Remove the remaining preemptive provider availability check from the backfill loop.
Move the web watchlist scan core onto the shared metadata source priority so primary provider settings are respected during artist, album, and image resolution.
Add coverage for primary-source-first discography lookup and fallback to later providers when the primary source has no albums.
Bring placeholder tracklist skipping back into the shared watchlist scan path, and centralize the DB-only artist image backfill helper so both web scan entrypoints reuse the same logic.
Move the shared watchlist scan loop into core/watchlist_scanner.py so web_server.py only handles triggers, locks, progress, and post-scan orchestration.
Manual and scheduled watchlist scans now share the same scanner-side core, while the web entrypoints keep profile selection and automation progress updates.
Respect the configured metadata source order when looking up album years, and re-check provider availability during the scan so Spotify can drop out cleanly if it becomes rate-limited.
Cover art lookup now honors an explicit prefer_source first,
falls back to the runtime primary metadata source when unset,
and uses the shared source priority for the remaining fallbacks.
Use the shared metadata source priority when resolving album IDs,
album searches, and tracklists in track number repair.
Keeps Deezer and iTunes ahead of Spotify where configured, while
still allowing the job to fall back through other supported sources.
Unknown artist resolution now uses the shared metadata source priority and only filters to the sources that can actually participate in this job. Deezer and iTunes remain direct lookup sources, while Hydrabase can now join the title-search path when it is the configured priority source.
Album completeness and any other repair job now uses the centralized source/client helpers instead of a worker-local Spotify client or override plumbing
- This keeps source selection aligned with the configured primary provider and removes the last Spotify-only special case from the job path.
This change ultimately is a step towards further centralizing the Spotify client access and the associated `is_spotify_authenticated` check.
- Currently these look-ups are done all over the place in different feature implementations directly, but moving forward, any feature that uses `get_primary_client` or `get_client_for_source` to access the Spotify client, won't have to duplicate any rate-limiting or auth checks as long as these getters are used
Album completeness and downstream repair flow now follow the configured
primary provider first, with Discogs and Hydrabase support added alongside
existing Spotify, iTunes, and Deezer paths.
Keep spotify_track_id for compatibility while preserving source-aware track
IDs for provider-neutral handling.
Stripped 4,200+ emoji characters from print(), logger calls across
39 Python files. Logs are now clean text — easier to grep, more
professional, no encoding issues on terminals without Unicode support.
Seasonal config icons preserved for UI display.
Clients are for the most part being initialized per-request, which leads to a lot of redundant client initialization, as well as noise on the logs, since each client initialization emits a row on the logs, eg. 'Deezer client initialized'
Migrates 38 HTTP polling loops to WebSocket push events across 6 phases: service status, dashboard stats, enrichment workers, tool progress, sync/discovery progress, and scan status. All original HTTP polling is preserved as automatic fallback — if WebSocket is unavailable or disconnects, the app seamlessly reverts to its previous behavior. Includes 162 tests verifying event delivery, data shape, and HTTP parity. Also fixes a copy-paste bug in Beatport sync error cleanup.
