resolve_mirrored_playlist tried the mirrored-playlists primary key FIRST for
any all-digit ref. Deezer upstream ids are all-numeric, so a Deezer playlist id
was mistaken for the PK and the organize-by-playlist toggle resolved a wrong row
(or nothing) — the toggle silently wouldn't save / 'Open in Mirrored' missed.
Resolve by (source, source_playlist_id) first, fall back to PK only when the
source lookup misses. Thread the batch/wishlist source through the download-path
callers so numeric upstream ids resolve correctly there too. Spotify (base62
ids) is unaffected.
Seam tests: numeric Deezer id resolves by source (not PK), spotify alphanumeric
by source, PK fallback still works, profile-scoped, empty refs -> None.
Persist organize_by_playlist on mirrored playlists and run playlist-folder
downloads from the auto-sync pipeline instead of the global wishlist phase.
Register SoulSync library rows after playlist-folder post-processing, route
failed organize batches to the wishlist correctly, and skip sync-time
unmatched wishlist only when organize download handles retries.
Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL),
re-mirror on refetch, and improve standalone playlist modals (re-analysis,
Open in Mirrored). Add filesystem missing-track detection and tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
Defense-in-depth follow-up to #760. Even with the entrypoint chown fix, if the
album-bundle staging dir ever can't be created/written (permissions, read-only
mount, disk full), the dispatch caught the plugin exception and marked the whole
batch failed — even though the album had already downloaded (the #715 symptom:
'release finishes downloading but the batch fails').
Now an OSError from the plugin is flagged fallback-eligible, so the dispatch
returns to the per-track flow instead of hard-failing. OSError covers the
staging/filesystem failure that motivated this (#760's PermissionError) and, by
Python's IOError==OSError aliasing, any propagated transient I/O error —
falling back is never worse than hard-failing, and per-track is the universal
graceful path. Programming errors (TypeError, KeyError, RuntimeError, …) are
NOT OSError and stay terminal, so genuine bugs still fail loudly — the existing
'plugin exception => failure' contract and its test are preserved.
Test: new test_dispatch_staging_oserror_falls_back_to_per_track (PermissionError
on the staging dir -> result False, phase 'analysis', not failed). Existing
RuntimeError-is-terminal test still passes. 131 album-bundle/plugin tests green.
New GET /api/downloads/task/<id>/detail merges the live download task with its
library_history row (the data the Download History cards show) into one payload
the upcoming track-detail modal renders: status kind, title/artist/album,
source, quality, final location, AcoustID verdict, and expected-vs-downloaded.
Assembly + status classification live in core/downloads/track_detail.py as a
pure, importable build_track_detail()/classify_status_kind() (9 unit tests);
the endpoint is thin glue that looks up the matching history row by track.
A Soulseek album bundle stages whichever single folder scored best. If that
folder doesn't contain every track the album needs, the missing tracks were
marked not_found with no fallback — even in hybrid mode where later sources
(Deezer, YouTube, etc.) could fill them. The staging-miss short-circuit fired
for Soulseek because 'soulseek' was lumped into the torrent/usenet source set
when album bundles were added, and album_bundle_partial only reflects whether
the files found IN the folder downloaded, not whether the folder had every
needed track.
Drop 'soulseek' from the short-circuit (keep torrent/usenet). A track not
claimed from the staged Soulseek folder now falls through to the normal
per-track Soulseek search and, in hybrid mode, onward down the configured
chain. Unlike torrent/usenet — where per-track search re-adds the same
release — Soulseek per-track search is a genuine per-file network search, so
this is correct and cheap. Realizes the original author's stated intent
('keep partial bundles from blocking per-track fallback') robustly, since the
partial flag couldn't detect a folder that was simply missing tracks.
Only affects tracks NOT claimed from staging — fully-staged albums claim every
track via try_staging_match and never reach this gate, so working albums are
unchanged. Likely also mitigates #755 (all-album-import failures now fall
through to per-track instead of dying).
Tests: rewrote the two Soulseek staged-miss tests to assert fall-through
(single + hybrid-first); kept the torrent guard; added a usenet guard test.
The earlier #721 fix tolerated a ~10s "completed but no save_path"
window, but the real production stall sits upstream of that: SABnzbd
removes a finished download from the queue and runs par2 verify /
repair / unpack *in History*, exposing the live stage in the slot
`status` ('Verifying' / 'Repairing' / 'Extracting' / 'Moving' / ...)
with `storage` empty until the final move. `_parse_history_slot` mapped
EVERY non-'Failed' status to 'completed', so a still-extracting 1.7 GB
FLAC album looked "completed with no save_path" the instant download hit
100%. The poll burned its completed-no-path budget mid-PP and bailed,
freezing the UI on the last download emit (the stuck-at-99%/100%
signature). SAB then finished fine — which is why the job shows
Completed in History but SoulSync never staged it.
Root fix
- `_parse_history_slot` routes `status` through `_map_state`, so PP
stages stay NON-terminal: the poll keeps waiting (as 'downloading')
for as long as post-processing takes and only a real 'Completed'
flips to terminal success. `save_path` is trusted only on true
completion (mid-PP path fields may point at the incomplete dir).
Supporting / defensive
- `UsenetStatus.incomplete_path`: surfaced separately from save_path
(SAB `incomplete_path`) and used by the poll loops as a LAST RESORT
after the completed-no-path window, to recover the case where
`storage` never lands but the files are physically on disk.
- `poll_album_download`: dedicated, configurable completed-no-path
window (~120s via `download_source.album_bundle_completed_no_path_seconds`)
decoupled from the ~10s transient-miss window; incomplete_path
fallback; a 30s heartbeat log so the previously-silent poll loop is
diagnosable.
- `usenet.py` `_download_thread`: per-track parity — it was erroring
immediately on the first completed-no-path read.
- `album_bundle_dispatch.py` / `status.py` / `monitor.py`: use the
project `get_logger` so download-flow logs land in app.log under the
`soulsync.*` namespace (they were console-only before, which hid the
`[Album Bundle] flow failed` line during triage).
Tests
- PP-history state mapping; end-to-end Hunky Dory PP regression
(download -> Verifying/Extracting in History past both budgets ->
Completed+storage -> success); completed-no-path window +
incomplete_path fallback; per-track thread parity. ruff + compileall +
pytest all green (the only local failures are environmental: missing
tzdata + local tools/ffmpeg.exe, neither present on CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related leaks in ``storage/album_bundle_staging/<batch_id>/``:
1. **Soulseek bundle cleanup was excluded.** The per-batch cleanup
at the end of a bundle download gated on:
(album_bundle_source or '').lower() in ('torrent', 'usenet')
The comment justified it as "slskd keeps its own completed
folders" — but the Soulseek bundle path ALSO copies completed
files into the private staging dir (``soulseek_client.py:1599``,
``copy_audio_files_atomically(completed, Path(staging_dir))``)
for the per-track workers to claim. Those copies persisted
forever; long-running installs accumulated stale GB. Extended
the cleanup gate's allow-list to include ``soulseek`` so the
per-batch dir is removed on bundle completion — same code path
that already worked for torrent / usenet.
2. **No sweep for orphan dirs.** Any leftover ``<batch_id>``
subdir from a previous-session crash, an errored batch, or a
pre-fix Soulseek bundle stayed on disk forever. Added
``sweep_orphan_album_bundle_staging(staging_root, active_batch_ids)``
that runs ONCE at server startup, before any batch can register
a staging dir. Removes every ``<batch_id>``-shaped subdir
whose id isn't in the active set. Safe by construction:
- Only touches subdirs of the configured staging root.
- Name-shape check (``entry.name == _safe_batch_dirname(entry.name)``)
rejects hand-placed dirs like ``.git`` or stray docs.
- ``shutil.rmtree`` errors log + continue — sweep must not
crash app startup over a permission glitch.
- active_batch_ids normalised through ``_safe_batch_dirname``
so colon-bearing batch_ids match their on-disk form.
Wired into the web_server startup right after the stuck-flags
diagnostic so it fires before anything else touches batches.
Tests
- ``test_downloads_lifecycle.py`` gained one regression test
pinning that Soulseek bundles now have their staging dir
cleaned (sibling to the existing torrent test).
- ``test_album_bundle_staging_sweep.py`` (NEW, 11 tests)
covers: orphan removal with no actives, active dirs preserved,
special-char batch_id normalisation, no-op on missing /empty
/empty-string staging root, non-dir entries skipped, unsafe-
name dirs preserved (.git etc.), partial rmtree failure doesn't
abort the rest, listdir failure returns 0 cleanly, default
None active set, defensive against empty / None entries in
the active set.
488 downloads tests pass.
For users with an existing "clean up old files" automation pointed
at this dir: stop pointing it there if you want — the auto-cleanup
+ startup sweep cover it now. Or leave it as belt-and-suspenders
with a relaxed (1h+) mtime threshold so it can't race a mid-batch
download.
finished the release (#715)
Symptom (user @pavelcreates / @IamGroot60 on 2.6.2):
- Click Download on an album in the search modal
- slskd starts + completes every track of the release
- 22+ minutes after the last completed download, batch flips
to "failed" with no clear log line explaining why
- Per-track Soulseek downloads on the same machine were fine
Root cause: ``core/soulseek_client._resolve_downloaded_album_file``
probed three hard-coded candidate paths to locate each downloaded
file in the slskd download dir:
candidates = [
download_path / remote_filename,
download_path / basename,
download_path / *normalized_path_parts,
]
On the common slskd config ``directories.downloads.username = true``
slskd writes files at ``<download_dir>/<username>/<filename>`` —
none of the three candidates carry a username segment, so the
resolver returned None for every file even though the file was
physically present in a subdir one level deeper. ``_poll_album
_bundle_downloads`` saw 0 completed_paths, kept spinning, and
hit the master deadline (~30 min) before bailing the batch.
Why per-track worked: ``web_server._find_completed_file_robust``
already does a recursive walk-by-basename + path-confirm against
the remote directory components, so any layout slskd writes ends
up resolved. The bundle path didn't go through it.
Fix
- Lifted the robust finder into ``core/downloads/file_finder.py``
as a pure function ``find_completed_audio_file(download_dir,
api_filename, transfer_dir=None) -> (path, location)``. Zero
globals; recursive walk; handles slskd dedup suffix
``_<10+digit-timestamp>``, YouTube / Tidal ``id||title`` encoded
filenames, the AcoustID-quarantine subdir skip, basename
collisions disambiguated by remote-path components, and a
fuzzy-basename fallback above 0.85.
- ``_resolve_downloaded_album_file`` keeps the three-candidate
fast path (cheap probe for the slskd-flat default) but now
delegates to the new helper when none hit, instead of giving up.
- ``_poll_album_bundle_downloads`` tracks "slskd reports
Completed but local resolver returns None" per key. When every
remaining key has been in that state past a 45-second grace
window, the poll exits early with an explicit error pointing at
the likely ``soulseek.download_path`` mismatch instead of
silently spinning until the master deadline.
- ``web_server._find_completed_file_robust`` becomes a thin
delegate so both callers share one finder. Legacy inline impl
kept as ``_find_completed_file_robust_legacy`` for reference;
to be removed next release.
- Fixed misleading ``"(0 tracks, quality=)"`` log on the preflight-
reuse path — was reading attrs off a None ``picked`` object.
Tests (17 new in tests/downloads/test_file_finder.py)
- Flat slskd layout
- Username-prefixed (the #715 case)
- Full remote tree preserved
- Deeply nested username + tree
- File genuinely missing returns None
- Basename collision disambiguated by remote dirs
- Single basename match wins regardless of dirs
- slskd dedup suffix match
- Short ``_<digits>`` (year) not treated as dedup
- AcoustID quarantine subdir skipped
- YouTube / Tidal ``id||title`` encoded filenames
- transfer_dir fallback
- Both dirs miss → (None, None)
- Non-audio files ignored
- Empty api_filename
- Fuzzy match on punctuation variant
- Fuzzy rejects below threshold
475 downloads tests pass after the lift.
Residual per-track wishlist downloads (single tracks from different
albums, below the album-bundle threshold) were producing folders
without a year subfolder whenever the wishlist row carried a stale
``track_number=1`` from an older payload default.
Why: ``core/downloads/candidates.py`` had a single API-fetch branch
that served two concerns — resolving the track position AND
hydrating the lean ``spotify_album_context`` (release_date /
total_tracks / cover image) — gated entirely on track_number being
unresolved. When the wishlist row's ``track_number`` happened to
be 1 (a poisoned default rather than a real value), the gate
short-circuited and the album hydration the same call would have
done was skipped. Deezer-sourced discovery matches don't ship
release_date in their search-result album shape, so without the
backfill the folder lost its year.
The two concerns split:
- track_number resolution keeps its track_info → track object →
API precedence chain. track_info defaults still win.
- album hydration runs whenever release_date or total_tracks are
missing, independent of where (or whether) track_number was
resolved.
The single API round-trip still serves both — the cost contract
is preserved. The side-effect coupling is gone.
Lifted into ``core/downloads/track_metadata_backfill.py``
(``hydrate_download_metadata``) so the precedence chain is pinned
in isolation. 24 unit tests cover the precedence chain, the
poisoned-tn=1 regression case, defensive non-dict/None inputs,
the cost guard (API called at most once per invocation), and
disc_number resolution.
Also lands the upstream piece: ``core/wishlist/routes.py:_build_track_data``
no longer defaults ``track_number=1`` / ``disc_number=1`` /
``total_tracks=1`` / ``release_date=''`` when the library-modal add
payload omits them. Missing values now flow through as ``None`` so
the downstream pipeline can detect-and-recover instead of locking
to a fake position.
PR 4 of 4 in the wishlist-album-bundle issue series. UI fix only —
zero behavior change.
User's 26-track wishlist run rendered all 26 sub-batches as
"Analyzing..." simultaneously. Pre-fix the rows were created with
``phase='analysis'`` BEFORE being submitted to ``missing_download_executor``
(max_workers=3 by default), so 23 batches sat in the executor queue
visually identical to the 3 actually running. Misled users into
thinking SoulSync was processing 26 in parallel; really only 3 ever
ran at once with the rest waiting their turn.
Fix:
- Wishlist auto-flow submission sites now create batch rows with
``phase='queued'``.
- The master worker (``core/downloads/master.py:328``) already flipped
phase to ``'analysis'`` as its first action on entry — that
transition becomes the real signal that the executor picked the
batch up.
- ``core/downloads/status.py`` surfaces ``analysis_progress`` for
the ``queued`` phase too so the UI has the track count to render
"Queued — N tracks" instead of an empty card.
- Frontend (``webui/static/pages-extra.js``, ``downloads.js``) renders
"Queued ⏳" for ``phase='queued'`` distinct from the spinner-laden
"Analyzing..." for ``phase='analysis'``.
Scope choices:
- Only the auto-wishlist submission sites flipped this PR
(``core/wishlist/processing.py:860`` album sub-batches +
``core/wishlist/processing.py:907`` residual). The manual-wishlist
sites at ``:451`` and ``:627`` use the same executor + worker, but
those create a caller-allocated batch_id that the frontend polls
immediately — wanted to verify the manual-poll path handles
``queued`` cleanly before flipping those. Trivial follow-up.
- Other submission sites in album_bundle_dispatch / web_server.py /
task_worker.py left untouched — they don't go through the
executor-queue pattern that causes this UI confusion.
Tests:
- Updated ``test_process_wishlist_automatically_creates_batch_for_matching_tracks``
to assert ``phase='queued'`` on creation (was ``'analysis'``); explanatory
comment names the executor-pool reason.
- New ``test_queued_phase_surfaces_analysis_progress_for_ui_count`` in
``tests/downloads/test_downloads_status.py`` pinning the new
``queued ⊂ analysis_progress`` rendering contract.
- 884 tests pass across wishlist + downloads + imports suites.
- Ruff clean on changed Python files; JS syntax OK on changed
webui files.
PR 3 (sibling-completion gate) was investigated and dropped — the
"1/26 finalized" symptom turns out to be downstream of the
staging-match bug (PR 2's instrumentation will catch it on the
user's next reproduction run), not an independent sibling-gate bug.
The gate logic itself is correct.
Pre-fix: ``try_staging_match`` silently returned False on three exit
points (empty cache, no track title, low best-score). Could not
diagnose the "track gets staged via album-bundle but never claimed
→ re-added to wishlist → infinite loop" bug from app.log because the
match-attempt + rejection was invisible.
Now every False exit logs at INFO with enough context to debug from
a single grep:
- ``[Staging] No match attempted for <track> — staging cache empty for batch <id>``
- ``[Staging] No match attempted for task <id> — track has empty title``
- ``[Staging] No match for <track> in batch <id> — best candidate <file> (title_sim=X, artist_sim=Y, combined=Z) below 0.75 threshold``
- ``[Staging] No match for <track> in batch <id> — N staging files but none had usable title variants``
Per-candidate skips (no title variants / title_sim < 0.80) log at
DEBUG so the noise stays out of INFO unless explicitly enabled.
Logs the near-miss candidate score on rejection so a 0.74 (one point
below threshold) surfaces as a different kind of bug than a 0.10
(completely wrong file in staging). Same shape SAB's adapter logs
now use for transient-vs-terminal status calls (PR #717).
Zero behavior change — pure logging. Enables the follow-up commit
that actually fixes the staging-match drop, by giving us real evidence
of WHERE the wishlist tracks are being rejected during the user's
next album-bundle run.
24 staging tests still pass; behavior unchanged.
Commit 1 of 3 in PR 2/4 of the wishlist-album-bundle issue fix
series. See ``memory/feedback_always_build_kettui_grade.md`` for
the instrument-before-blind-fix rule that drove this ordering.
The sibling-merge aggregator from 7f751202 used "least-complete
phase wins", which made the modal appear frozen during parallel
album bundle downloads. The task table is phase-gated to
downloading/complete/error in downloads.js — so whenever any
sibling was still in album_downloading, the merged phase stayed
there and tasks for the sibling that had advanced past its bundle
never rendered. User reported: both albums downloading on slskd,
modal blank until one completes fully.
Flip the rule: surface the most-advanced live phase so the modal
renders task progress as soon as any sibling reaches it. The
all-siblings-in-album_downloading case still surfaces
album_downloading (bundle progress UI is correct there); error
stays sticky.
Updated WHATS_NEW under 2.6.3 to describe the corrected behavior.
Two new tests pin the regression:
- downloading + album_downloading → downloading
- album_downloading + album_downloading → album_downloading
Phase 1c.2.1 splits each wishlist run across multiple
``download_batches`` rows (per-album bundle dispatch). The
download-missing modal opens against the original batch_id
allocated by ``start_manual_wishlist_download_batch`` /
``process_wishlist_automatically``. Pre-fix that batch_id was
just one sibling among N, so the modal went stale as soon as the
primary sub-batch finished — subsequent albums downloaded fine
but no live status reached the UI.
Fix: backend merges every sibling sub-batch's tasks +
analysis_results into the response keyed under the originally-
requested batch_id. Modal sees one unified view of the whole run
without knowing about the split. Frontend untouched.
Architecture (Kettui standards):
- ``core/downloads/wishlist_aggregator.py`` — pure
``merge_wishlist_run_status(primary, siblings)`` helper.
No IO, no runtime state, no globals. Lifted out of
``status.py`` so the merge contract can be pinned via unit
tests without standing up the live ``download_batches`` /
``download_tasks`` state.
- ``core/downloads/status.py``'s ``build_batched_status`` now
pre-indexes ``download_batches`` by ``wishlist_run_id`` inside
the existing ``tasks_lock`` snapshot, then runs the merge
helper whenever a requested batch has a sibling.
Merge rules pinned by 12 tests:
- ``track_index`` re-indexed globally 0..N-1 across the merged
``analysis_results`` so the modal's ``data-track-index`` DOM
keys don't collide between siblings. Tasks' ``track_index``
follows the same remap so the analysis-results ↔ tasks
cross-reference stays intact.
- ``task_id`` is uuid per task — no collision concern.
- Phase: error is sticky; otherwise the LEAST-complete
pre-terminal phase wins (analysis < album_downloading <
downloading). All-complete returns ``complete``; mixed
complete + active returns ``downloading`` so the modal stays
alive until every sibling lands.
- ``album_bundle``: picks whichever sibling currently has an
active bundle download (state in
``{searching, downloading, downloading_release, staging}``).
Falls back to the first non-empty bundle so a completed run
still shows a progress bar.
- ``analysis_progress`` summed across siblings.
- ``active_count`` summed; ``max_concurrent`` keeps primary's
value as the representative.
- ``playlist_id`` + ``playlist_name`` preserved from the primary
(the row the modal originally opened against).
Legacy single-batch wishlist runs (no ``wishlist_run_id`` on the
batch) skip the merge entirely — passthrough. Back-compat by
absence.
1108 tests across downloads + wishlist + automation + imports +
playlist-sources + lb-series suites green. 12 new aggregator
tests pin the merge contract.
Closes the open UX gap from the Phase 1c.2.1 ship — modal now
tracks every sibling sub-batch's progress for the full duration
of the wishlist run.
When a task failed AcoustID verification and got quarantined, opening
the candidates modal and manually picking a different file would just
re-quarantine it. The manual-pick path through
`_attempt_download_with_candidates` ran full post-processing with no
quarantine bypass — so if the alternate file disagreed with AcoustID's
stored metadata too (common for live versions, remasters, regional
title differences, fingerprint coverage gaps) the file landed right
back in quarantine. User got stuck in the loop.
The Approve button on quarantined rows already handles the "I want
this exact file" case via `_skip_quarantine_check='all'`. The
candidates modal handles the "I want a different file" case — same
user intent, opposite direction, but the bypass plumbing didn't carry
through.
`/api/downloads/task/<id>/download-candidate` already sets
`task['_user_manual_pick'] = True`. `attempt_download_with_candidates`
now reads that flag under tasks_lock alongside `used_sources` and,
when set, injects `_skip_quarantine_check='acoustid'` plus
`_user_manual_pick=True` into the stored `matched_downloads_context`
entry. The acoustid-only scope is deliberate: integrity + bit-depth
gates still run because those check the new file's actual condition
(corruption, sample rate) rather than its identity — only the
metadata-mismatch gate is the user-override case.
Auto-search picks (the normal task-worker path) leave the flag unset
and continue to run full AcoustID verification, preserving the
existing safety net for non-user-initiated downloads.
Tests:
- positive: manual-pick task → stored context has
`_skip_quarantine_check='acoustid'` and `_user_manual_pick=True`
- negative: auto-search task → stored context has neither key,
AcoustID still runs as before
Full suite 3976 pass.
Root cause (#700): the Soulseek album-bundle path downloads whole
releases into a private staging dir, then per-track workers claim
those files via the staging-match shortcut. When slskd files arrived
without ID3 tags (common for FLAC rips), the staging cache fell back
to the filename stem as the title — and stems shaped like
"Artist - Album - 03 - Title" could not clear the 0.80 title-
similarity threshold against the clean Spotify track name. Every
track in the album went not_found, the batch ended "failed" in the
Downloads UI with an empty queue, and the bundle-downloaded files
just sat unused in staging.
Fix: in _staging_title_variants, add a trailing-title variant by
extracting the segments after a bare track-number block (e.g. "03")
between " - " delimiters. Conservative — only fires when a clear
digit segment is present, so real song titles with dashes like
"Hold Me - Live" are left intact. Generated as an additional variant
alongside the existing raw/compacted/feat-stripped/bonus-stripped
forms, so behavior on already-matching files is unchanged.
Downstream (#698): the album-bundle staging miss pushed every failed
track to the wishlist labelled as a playlist track, and a couple of
fallback paths in ensure_wishlist_track_format and the slskd-result
reconstruction hardcoded album_type='single' / total_tracks=1 on the
stored album dict. On wishlist requeue the path builder saw
album_type='single' and routed the download through single_path,
dumping the file in the Singles tree even though it belonged to an
album. (Running Reorganize would fix it because the DB album linkage
was still correct, but the file landed in the wrong place first.)
Fixes:
- new resolve_wishlist_source_type_for_batch() returns 'album' for
is_album_download batches; wishlist_failed.py now calls it instead
of hardcoding 'playlist'
- build_wishlist_source_context() threads album_context /
artist_context / is_album_download from the batch into the wishlist
row so future requeue logic has authoritative routing data
- the non-dict-album fallback in ensure_wishlist_track_format and
the slskd-result reconstruction default album_type='album' (and
total_tracks=0 = unknown) instead of lying with 'single'/1; the
existing setdefault chain handles dict-shaped album data unchanged
Tests:
- 2 staging-match tests pin the new tail-extraction behavior against
a realistic untagged slskd stem, plus a negative test that confirms
a dash-in-title without a digit segment still does NOT extract a
variant
- 2 payload tests pin the album_type='album' default for both
fallback paths
- 4 processing tests pin resolve_wishlist_source_type_for_batch()
and the album-context threading in build_wishlist_source_context()
3974 pass; no behavioural change on already-working flows.
Delay torrent and usenet album-bundle dispatch until missing-track analysis confirms there is work to do, matching the Soulseek album flow and avoiding release downloads for already-owned albums.
Clear private album-bundle staging state when a release-level source intentionally falls back to per-track mode so workers can use the normal staging/search path instead of an empty private bundle directory.
Verified by user: focused downloads master tests passed, 2 passed.
Stop appending persistent download history once the unified downloads payload reaches the requested limit. This keeps the Downloads tab history tail bounded even if the history provider returns more rows than requested, while preserving existing live-task total behavior.
Keep album-bundle staging from replacing known per-track album numbers with the filename parser's default when staged files do not expose a real track number. Carry staging tag numbers through the cache, fall back to task metadata for private release staging, and cap hybrid album batches to one worker when Soulseek is first in the source order.
Include a capped recent tail of database-backed download history in the unified Downloads page so completed Deezer and other streaming downloads remain visible after runtime tasks are cleaned up or the container restarts. Use persistent download history for the dashboard finished count, keep live tasks authoritative for active rows, avoid showing the local clear-completed action for persisted history rows, and cover history hydration/deduping/capping in status tests.
Route primary Soulseek album downloads through the album-bundle staging flow, reusing preflight-selected folders when available. In hybrid mode, only the first configured source can claim whole-album bundle behavior so later Soulseek fallback keeps the existing per-track/source-reuse path.
Allow Soulseek album bundles to stage completed tracks when some same-source transfers fail or time out, and keep partial bundles from blocking per-track fallback. Add coverage for dispatcher gating, master flow ordering, task-worker staged-miss behavior, and Soulseek bundle polling.
Adds torrent/usenet as release-oriented download sources with album-bundle staging, live progress reporting, and post-processing that selects the requested audio file from completed releases instead of blindly importing the first file.
Keeps album-bundle behavior gated to single-source torrent/usenet album downloads, excludes release sources from hybrid album per-track searches, and allows hybrid non-album tracks to use release results safely.
Improves staged-release matching for featured/bonus track filenames while preserving version mismatches, records torrent/usenet provenance in library history, and updates service/status UI labels.
Covers the flow with focused lifecycle, status, staging, validation, task worker, post-processing, and import side-effect tests.
Route torrent and Usenet album bundles through private per-batch staging so Auto-Import cannot race public staging or duplicate imports.
Expose album-bundle progress in batch status and render it on the Downloads page while the external client is still downloading.
Tighten release handoff safety by rejecting archive path traversal, ignoring torrent candidates without a usable URL, and skipping Soulseek source reuse for torrent/Usenet batches.
Tests: .venv/bin/python -m pytest tests/downloads/test_downloads_status.py tests/test_album_bundle_dispatch.py tests/downloads/test_downloads_staging.py tests/test_torrent_usenet_plugins.py
Wraps up the code-review refactor pass.
- config/settings.py: ``download_source`` defaults gain
``album_bundle_poll_interval_seconds`` (default 2s) and
``album_bundle_timeout_seconds`` (default 6h, was a hard-coded
``6 * 60 * 60`` magic constant in torrent.py). The plugin reads
these via ``album_bundle.get_poll_interval`` /
``get_poll_timeout`` with safe fallback to the defaults when the
config value is missing / non-numeric. ``mode`` doc-comment
extended to list ``torrent`` and ``usenet``.
- core/downloads/validation.py: comment block above the album-name
fallback rewritten to document when the fallback actually runs
now — single-track hybrid downloads only, because the album-
bundle gate handles single-source mode and the hybrid chain
filter strips torrent / usenet from album batches. Code path
unchanged; just clarifies the contract for the next reader.
- webui/static/helper.js: WHATS_NEW entry summarising the refactor
pass (helper extraction, dispatch lift, staging deps injection,
atomic copy, configurable timeout, test additions).
The /loop of: extract → inject → test was sweep enough to drop the
gate code's coupling to 2-3 modules and put 49 unit tests behind
the new boundaries. Code-review feedback addressed:
1. album_bundle.py extracted ✓
2. Dispatch lifted out of master.py ✓
3. staging.py decoupled from runtime_state ✓
4. Validation fallback scope documented ✓
5. Poll timeout config-driven ✓
6. ``amazon`` provenance owned in a prior commit ✓
7. End-to-end-shaped tests added (test_album_bundle_dispatch.py)
8. Auto-Import race closed via atomic copy ✓
Per code review: the album-bundle provenance override added in an
earlier commit reached into ``core.runtime_state.download_batches``
directly from inside the staging matcher. Sibling modules
shouldn't import each other's globals — the existing StagingDeps
pattern is the canonical way to inject everything else this helper
needs.
- core/downloads/staging.py: new optional ``get_batch_field``
callable on ``StagingDeps`` (defaults to None for backward compat
with any caller that doesn't know about it yet). The inline
``from core.runtime_state import download_batches`` is gone; the
helper now calls ``deps.get_batch_field(batch_id,
'album_bundle_source')`` and falls back to 'staging' when None
is returned. Accessor exceptions are swallowed with a debug log
so a deleted batch mid-process can't break the staging match.
- web_server.py: ``_build_staging_deps`` injects a small
``_staging_get_batch_field`` helper that wraps the tasks_lock +
download_batches dict access. Centralises the lock semantics in
one place — the staging module no longer needs to know about
the lock or the dict.
- tests/test_staging_album_provenance.py: 5 new tests covering the
full matrix — torrent override applied, usenet override applied,
no override falls back to 'staging', missing accessor (default
None) falls back to 'staging', accessor raising falls back to
'staging'. Each test seeds + cleans a synthetic task in
runtime_state so the test doesn't bleed state across the suite.
Per code review: ~90 lines of inline gate logic in
``run_full_missing_tracks_process`` was inflating an already-580-
line worker function and was non-testable in isolation. Lifted to
``core/downloads/album_bundle_dispatch.py`` with two entry points:
- ``is_eligible(mode, is_album, album_name, artist_name)`` — pure
predicate, no side effects, easy to assert against. Splits the
gate decision from the resolution + run step so tests can pin
the gate semantics without standing up a plugin.
- ``try_dispatch(...)`` — full flow. Returns True iff the master
worker should stop (gate fired + failed); False = engaged-and-
succeeded OR didn't engage, both fall through to per-track.
State access is now decoupled from ``runtime_state``:
- New ``BatchStateAccess`` Protocol with two methods
(``update_fields``, ``mark_failed``).
- Concrete impl ``_BatchStateAccessImpl`` lives in master.py and
wraps the tasks_lock + dict ops the original inline code did.
- Injected via parameter so the dispatch module never imports
``download_batches`` / ``tasks_lock`` directly.
Same goes for the plugin resolver and config getter — both
injected, so the dispatcher works against any orchestrator /
config implementation (including the in-test fakes).
Behavior unchanged. The master worker call site is now 11 lines
of boilerplate instead of 90 lines of inline conditional. Plugin
contract (``download_album_to_staging`` return dict shape)
unchanged.
- core/downloads/album_bundle_dispatch.py: new module owning the
gate + execution. ~150 lines including docstrings and the
Protocol definition.
- core/downloads/master.py: gate call site shrunk to a single
``if _album_bundle_dispatch.try_dispatch(...): return``. New
``_BatchStateAccessImpl`` class implements the Protocol against
the existing ``download_batches`` dict + ``tasks_lock`` so the
dispatcher gets injected access instead of importing them.
- tests/test_album_bundle_dispatch.py: 16 new tests covering the
pure predicate (album-required, mode allowlist, name validation,
case insensitivity), the resolver-failure fall-through
(plugin missing, plugin lacks method, resolver raises), the
success path returning False so per-track flows, the failure
path returning True with state.mark_failed called, plugin-raise
treated as a normal failure, whitespace stripping on names,
and progress-callback mirroring lifecycle events into batch
state.
When a user picks Hybrid mode AND downloads an album, the per-track
search loop fires once per track. Torrent / usenet are release-level
sources — Prowlarr returns album torrents, none of which score
meaningfully against an individual track title. Without filtering,
every track triggered a redundant Prowlarr search, qBit rejected
duplicate hashes after the first, and the run only worked at all
because Auto-Import swept Staging behind the scenes. Confusing
logs, wasted searches, brittle timing.
Fix: thread an optional ``exclude_sources`` parameter through
``DownloadOrchestrator.search``. When the per-track worker detects
that the active batch is an album AND mode is hybrid, it passes
``['torrent', 'usenet']`` so the hybrid chain skips them and falls
through to per-track-compatible sources (Soulseek / streaming).
Gate is narrow on purpose:
- Hybrid + album → skip torrent / usenet (THIS fix)
- Single-source torrent / usenet + album → album-bundle flow on
the master worker (already shipped)
- Hybrid + single-track batch (basic search / wishlist / playlist
of singles) → torrent / usenet still tried, validation.py's
album-name fallback gives them a shot
Excluded list logged at INFO when applied so the behavior is
visible in logs ("Hybrid search: excluding ['torrent', 'usenet']
for this query"). Default ``exclude_sources=None`` keeps every
non-task-worker caller (basic search, stream search, search-and-
download-best, automation handlers) on the original code path.
The download history modal was tagging every torrent / usenet
album-bundle download as 'Soulseek FLAC 24bit' because:
- core/imports/side_effects.py's source_service dict didn't have
entries for 'staging', 'torrent', or 'usenet' usernames. The
staging matcher in core/downloads/staging.py sets
download_tasks[task_id]['username'] = 'staging', which fell
through to the dict's default and got recorded as 'soulseek'
in the track download provenance row. Same fate for any
amazon or other source that wasn't whitelisted.
- The album-bundle flow specifically wants to be labeled as
'torrent' or 'usenet' (where the bytes actually came from),
not 'staging' (the intermediate). The plugin already stashes
the source on the batch state as ``album_bundle_source`` for
the Downloads-page status card; provenance recording can
read the same field.
Fixes:
- core/downloads/staging.py: when marking a task post_processing
after a staging match, check the batch's album_bundle_source
override and use that for username instead of 'staging' when
set. Falls back to 'staging' when no override exists
(manual file-drop case).
- core/imports/side_effects.py: source_service map gets entries
for 'staging', 'torrent', 'usenet', and the previously-missing
'amazon' (which was also falling through to 'soulseek').
- webui/static/library.js: the redownload modal's serviceLabels
/ serviceIcons dicts extended to cover lidarr, amazon,
soundcloud, auto_import, staging, torrent, usenet so badges
render the correct name instead of either the raw source_service
string or no badge at all.
- webui/static/wishlist-tools.js: history-source-chip color
palette extended for the new source labels (Torrent sky-blue,
Usenet violet, Staging / Auto-Import neutral grey).
Note: existing tracks in the DB still carry the wrong 'soulseek'
label — only NEW downloads after this fix get the right label.
A future migration could rewrite historical rows but it's
cosmetic and the underlying audio + metadata are correct.
Fixes the core architectural mismatch between indexer-based sources
and the per-track search-and-pick contract every other download
plugin satisfies. Prowlarr returns release-level torrents and NZBs;
searching for "Luther (with SZA)" against the GNX album torrent
scores near-zero on track-title similarity. Per-track candidate
validation rejects every result, every track in the batch flips
to not_found. The album-name fallback added in an earlier commit
papers over it for some cases but doesn't fix the fundamental
behavior: the user wanted the whole album.
New album-bundle flow does what the user actually wanted:
1. Gate fires inside core/downloads/master.py BEFORE the per-track
analysis loop, strictly when the batch has an album context AND
download_source.mode is 'torrent' or 'usenet' (single-source —
hybrid stays per-track to preserve fallback to Soulseek / etc.).
2. Plugin's new download_album_to_staging method searches Prowlarr
ONCE for the album as a whole ('<artist> <album>'), filters to
the right protocol, runs results through _pick_best_album_release.
3. Picker prefers seeded FLAC over low-seeded MP3, drops single-
track torrents that snuck in via the 40 MB size floor (single
tracks are typically ~10 MB), falls back to most-seeded when
every candidate is below the floor.
4. Picked release goes to the active adapter (qBit / Transmission /
Deluge for torrent; SAB / NZBGet for usenet). Polls until
complete with progress mirrored into the batch state so the
Downloads page can show meaningful status.
5. On completion the existing archive_pipeline walks the save dir
(extracting archives if any), every audio file gets copied into
the staging folder via _unique_staging_path so concurrent batches
don't collide.
6. Gate exits, master worker continues into the normal per-track
flow. Each track task hits try_staging_match early in the worker
and finds its file by fuzzy title match — no Prowlarr search
ever fires per-track, no candidate rejection, files flow through
the existing post-processing pipeline (tags, AcoustID, library
import).
Gate is strictly opt-in. Three orthogonal conditions must all hold:
batch_is_album, mode in ('torrent', 'usenet'), and the plugin must
expose download_album_to_staging. Any other source / hybrid mode /
non-album batch flows through the master worker unchanged. The
existing per-track torrent path still works for basic-search
single-track grabs.
- core/download_plugins/torrent.py: download_album_to_staging plus
_pick_best_album_release and _unique_staging_path helpers (shared
with the usenet plugin). _poll_album_download mirrors the existing
poll loop with progress callback emission.
- core/download_plugins/usenet.py: parallel implementation reusing
the picker + staging helpers. Different state set ('failed' vs
'error') from the usenet adapter contract.
- core/downloads/master.py: ~90-line gate right after batch context
loading. Mirrors plugin lifecycle into batch state under
``album_bundle_*`` keys so the Downloads page can render progress
while the torrent/usenet job runs (per-track tasks don't exist
yet during this phase). Failed bundle download fails the batch
with a meaningful error; missing plugin / context falls back to
the per-track flow with a warning.
- tests/test_torrent_usenet_plugins.py: 5 new tests pinning the
album picker preferences (FLAC over MP3 with comparable size +
better seeders, size floor drops singles, fallback when all
small), staging-path collision suffix, and the not-configured
short-circuit.
Live-test bug: Spotify-flow downloads with Torrent Only as the
active source produced 'download_failed' for every track. Searches
hit Prowlarr fine but no candidate ever got picked. Root cause was
in core/downloads/validation.py's get_valid_candidates:
- The streaming-source allowlist for the structured-metadata path
didn't include 'torrent' / 'usenet', so torrent results fell into
the Soulseek matching branch.
- Soulseek matching parses ``candidate.filename`` as a slskd-style
``Artist/Album/Track.flac`` path. Torrent / usenet filenames are
encoded as ``<download_url>||<display_name>`` so the orchestrator
can recover the URL — splitting that string on slashes produced
garbage path segments that never matched the expected artist,
every candidate failed the artist-folder gate, returned [], track
status flipped to 'not_found'.
Fixes:
- _streaming_sources now includes 'torrent' and 'usenet'. They take
the structured-metadata scoring path that reads r.title / r.artist
directly (the projection layer pre-fills both correctly).
- Artist gate skipped for torrent/usenet, same as YouTube. Album-
level releases legitimately don't expose per-track artist — the
projection falls back to the indexer name as the 'artist' field,
which would otherwise fail the gate against every Spotify artist.
- New album-name fallback scoring: for torrent/usenet only, the
candidate title is ALSO scored against the wanted track's
spotify_track.album field, and the max of (track-title score,
album-title score) wins. This makes a candidate titled
"GNX (2024) [FLAC]" match every track on the GNX album rather
than scoring near zero against a specific track title like
"Luther (with SZA)". match_type 'album_release' for visibility.
All 9 existing validation tests still pass.
Add duration tolerance logic and pre-download rejection for structured sources (tidal, qobuz, hifi, deezer_dl, amazon) when candidate duration deviates beyond allowed tolerance. Introduces helper functions _duration_tolerance_seconds and _duration_mismatch_exceeds_integrity_tolerance and uses resolve_duration_tolerance from core.imports.file_integrity. Log and skip candidates that would fail post-processing integrity checks to avoid wasted downloads. Update tests to include matching engine stub and new cases covering rejection and acceptance based on duration tolerance; also adjust imports and test fixtures.
Do not mark a monitored transfer as successful as soon as slskd reports completion. The monitor now only submits the post-processing worker; that worker reports the real success or failure after finding, verifying, and importing the file. If post-processing cannot be scheduled, mark the task failed and release the batch slot. Add a regression test for the premature success path.
Diagnostic-only change for issue Technodude reported: Tidal sync-playlist
downloads getting mass-cancelled mid-flight with no clear cause in the
logs. App.log shows ~91 second gaps between Tidal download start and
cancel — matches the monitor's 90s queue-timeout exactly — but none of
the monitor's WARNING log lines fire, so the trigger is ambiguous
between five `_should_retry_task` paths, three web_server cancel paths,
and the API endpoints.
Added a single `[CancelTrigger:<label>]` INFO log line immediately
before every `download_orchestrator.cancel_download(...)` call so the
next log dump pins down which path is firing.
Labels (grep-able, prefix tells the file, suffix tells the trigger):
monitor.not_in_live_transfers_90s
monitor.errored_state_retry
monitor.queued_state_timeout
monitor.stuck_at_0pct_timeout
monitor.unknown_state_no_progress_timeout
candidates.worker_cancelled_during_download
web.orphan_cleanup
web.cancel_download_task
web.atomic_cancel_v2
api.manual_cancel_single
api.public_cancel
The monitor's `deferred_ops` tuple grew from 3 elements to 4 (added
trigger label as last element). The dispatch loop unpacks both legacy
and new shapes so the change is backward-compatible for any in-flight
ops mid-deploy.
Zero behavior change. 367 download tests still green. WHATS_NEW left
untouched — diagnostic only, not user-facing.
After ship: ask Technodude to re-run the same sync playlist scenario,
attach the new app.log, grep `[CancelTrigger:` lines for the trigger
context, then write the actual fix.
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.
Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
Add a conservative Soulseek album preflight scorer so album downloads choose a coherent slskd folder before per-track enqueue. The scorer compares album title, artist, year, track count, tracklist coverage, peer quality, and penalizes unexpected deluxe/remix/live-style folders.
Preserve hybrid source priority by only running Soulseek album preflight when Soulseek is the selected source or first in the hybrid order. If Soulseek is only a fallback behind another source, the normal hybrid flow is left alone.
Reuse the richest wishlist album context across tracks in the same album group so release date, artwork, album type, and album artist stay consistent for path generation. Also preserve peer-quality tie breakers when attempting equal-confidence candidates.
Tests cover correct-folder selection over larger wrong editions, Soulseek primary vs fallback hybrid behavior, shared wishlist album context, and peer-quality candidate ordering.
AcoustID verification was quarantining every Amazon track because T2Tunes
embeds [Explicit] and [feat. X] in stream tag titles/artists, but AcoustID
returns bare titles — triggering version-mismatch rejection on every track.
- get_track_details: apply _strip_edition to name/album, _primary_artist to
artist; wire s.track_number / s.disc_number instead of hardcoded None
- get_album_tracks: apply _strip_edition to name, _primary_artist to artist
Also fix TypeError crash in album download paths when disc_number is None
(present in dict but explicitly None, so .get('disc_number', 1) returns None):
- master.py run_full_missing_tracks_process: or 1 guard on both max() and disc_num
- candidates.py track_info extraction: or 1 guard on both disc_number reads
- web_server.py enhanced + standard album download max() calls: or 1 guard
`validation.py` had amazon absent from `_streaming_sources`, causing
Amazon TrackResult objects (bitrate=None, size=0) to fall through to
the Soulseek P2P code path and get rejected by
`filter_results_by_quality_preference`. Every album track was marked
not found.
Fix: add 'amazon' to every streaming-source guard tuple/set that was
previously missing it:
- core/downloads/validation.py — primary bug fix (quality-filter bypass)
- core/downloads/status.py — _STREAMING_SOURCE_NAMES frozenset
- core/downloads/task_worker.py — hybrid fallback client map
- core/imports/side_effects.py — || filename→stream-id extraction
- web_server.py — is_streaming_source, transfer list display,
candidate source label, _try_source_reuse, _store_batch_source
- tests/test_download_plugin_conformance.py — registry count + parametrize
Also updates the 2.5.3 What's New entry to drop the stale
"not yet wired" disclaimer.
Closes#589. Tracks from MTV Unplugged / Live At / unplugged albums
consistently failed AcoustID verification with "Version mismatch:
expected (live) but file is (original)". Two upstream bugs fed into
the false positive — the AcoustID gate itself was correctly catching
the wrong file Tidal had selected. Codex diagnosed all three layers,
this fixes the two upstream causes and leaves the verifier alone.
Bug 1 — album-scoped library check false-misses owned albums
`core/downloads/master.py:184` scored "Shy Away (MTV Unplugged Live)"
(source title from playlist) vs "Shy Away" (local DB stored title)
with raw string similarity. Massive length asymmetry → ~0.3 → below
the 0.7 threshold → marked missing. Combined with the
`allow_duplicates and batch_is_album` short-circuit that disables
the global fallback for album downloads, the user's already-owned
album re-triggered every track for download. Explains the screenshot
showing "0 found / 7 missing" on an album the user manually placed.
New pure helper `core/matching/album_context_title.py:strip_redundant_album_suffix`
strips trailing parenthetical / bracket / dash suffixes whose tokens
are fully subsumed by the album context — at least one version
marker (live / unplugged / acoustic / session / concert / tour)
overlapping with the album, and every other token is either a
known marker, a year, a tolerated noise word, or a word from the
album title. Album-context-implied "live" added when the album
mentions unplugged / concert / tour / session.
Wired into the album-confirmed scope ONLY (not global matching).
Compares both raw and normalized source titles per album track and
takes the max similarity, so the helper returning the input
unchanged (when album doesn't imply version context) preserves
the pre-fix behavior.
Bug 2 — Tidal qualifier filter only ran on fallback searches
`core/tidal_download_client.py:345` set `is_fallback = attempt_idx > 0`
and only filtered when `is_fallback and required_qualifiers`. Primary
search returned all results unfiltered, so a query for "Shy Away
(MTV Unplugged Live)" could accept the studio cut if Tidal happened
to rank it first. Now the qualifier filter applies to BOTH primary
and fallback search attempts — log message updated to indicate
which path triggered.
Bug 3 — qualifier check ignored album.name
The legacy `_track_name_contains_qualifiers` only inspected the
track name. For concert / unplugged releases the live signal
typically lives in the album title, not the track title. New
`_track_matches_qualifiers` accepts a track object and inspects
both `track.name` AND `track.album.name`. Legacy helper preserved
to keep its existing test contract.
AcoustID version-mismatch gate at core/acoustid_verification.py
left intact — it correctly catches genuinely-wrong files that slip
through upstream filters. The In My Feelings (Instrumental) test
that pins this behavior continues to pass.
19 tests on the album-context helper covering MTV Unplugged
variants, dash/parens/brackets suffix shapes, year tolerance,
plural-form markers, the implied-live set, anti-regression cases
(instrumental/remix on a studio album must NOT be stripped),
empty/none defensive paths.
13 tests on the Tidal qualifier helper covering legacy
track-name-only behavior preserved, qualifier in track name alone,
qualifier in album name alone (the MTV Unplugged scenario),
multi-qualifier requirements, no-qualifiers always passes,
defensive against missing track.album, word-boundary avoiding
substring false-matches, _extract_qualifiers picking up live +
unplugged from the user's exact reporter query.
Full suite: 3053 passed.
Three follow-on fixes to the manual-search candidates modal once people
started actually using it:
1. NDJSON streaming. Manual search waited for every source to return
before showing anything. Now streams one event per source as each
completes — header line, source_results per source, done terminator.
Frontend appends rows incrementally via response.body.getReader().
2. Manual picks no longer auto-retry on failure. New _user_manual_pick
flag set on the task in /download-candidate. Both monitor retry
paths (not-in-live-transfers stuck + Errored state) bail on the
flag. Surfaces the failure to the user instead of silently picking
a different candidate via fresh search.
3. Non-Soulseek manual picks (youtube/tidal/qobuz/hifi/deezer/
soundcloud/lidarr) no longer stuck at "downloading 0%" forever. The
live_transfers IF branch now marks manual-pick tasks failed
directly when the engine reports Errored, instead of deferring to
the monitor (which bails on manual picks). Engine fallback in else
branch covers the rare race where the orchestrator's pre-populated
transfer lookup is missing the entry.
Plus a deadlock fix discovered along the way: the new failure path
synchronously called on_download_completed while holding tasks_lock,
which itself re-acquires the same Lock — non-reentrant
threading.Lock self-deadlocked the polling thread. While wedged, every
other endpoint that needed the lock (including /candidates → other
failed rows couldn't open modals) hung waiting. Moved completion
callbacks onto a daemon thread so the lock releases first.
Plus failed/not_found/cancelled rows are now ALWAYS clickable (not
just when the auto-search cached candidates) — the modal carries the
manual search bar, which is the user's recourse for empty results.
Plus manual download worker now runs on a dedicated thread instead of
competing with the batch's 3-worker missing_download_executor pool —
saturated batches no longer queue manual picks indefinitely.
All scoped to manual picks via the _user_manual_pick flag — auto
attempt flow byte-identical to before. Engine fallback gated on the
flag too so auto attempts in the else branch keep the original
do-nothing behavior (safety valve handles the stuck-forever case).
Also dropped _handle_failed_download from web_server.py — defined
but had no callers (dead code).
17 new unit tests pin the gate behavior:
- engine fallback: Errored/Cancelled/Succeeded/InProgress transitions,
manual-pick gate, terminal-state skip, soulseek skip, missing
download_id skip, engine returning None, orchestrator exception
- monitor: manual-pick skips not-in-live-transfers retry + Errored
retry
- IF-branch end-to-end: Errored marks failed, "Completed, Errored"
hits failure branch, auto attempts defer to monitor
Manual-search endpoint tests rewritten for NDJSON: 11 cases (validation,
single-source dispatch, parallel "all" dispatch, one-event-per-source
streaming shape, unconfigured-source skip + reject, header metadata,
per-source exception isolation).
Full suite 2259 passed, 1 skipped.
Two findings from JohnBaumb on the engine refactor.
(1) Every download client returned None when self._engine was None,
just logging an error. The orchestrator's download_with_fallback
treated None as "source declined", so the user got no feedback —
download silently disappeared. Now each client raises a RuntimeError
on the engine-not-wired path. download_with_fallback already catches
plugin exceptions, logs a warning, and tries the next source — so
the visible behavior is "real error in logs + fallback to next
source" instead of "silent drop". Six clients touched (deezer, hifi,
qobuz, soundcloud, tidal, youtube). Pinning tests updated to expect
raise.
(2) Monitor's engine.get_all_downloads() walked every plugin
including soulseek, but the same monitor loop already pulled slskd
transfers via the transfers/downloads endpoint a few lines earlier —
soulseek's records were being fetched twice per tick. Same issue in
web_server.py's get_cached_transfer_data path. Added an exclude
parameter to engine.get_all_downloads(); both call sites now pass
('soulseek',). New test pins the exclude semantic.
Also fixed a stray 8-space over-indent on the for-loop body in
get_cached_transfer_data (cosmetic, JohnBaumb flagged the same
pattern in monitor.py earlier).
Three small follow-ups from the Copilot review of the rename PR:
- services/sync_service.py: PlaylistSyncService.__init__'s
download_orchestrator parameter was annotated as SoulseekClient,
which was misleading (the object passed is the DownloadOrchestrator
with .search_and_download_best, .download, etc — not a SoulseekClient).
Switched the import + annotation to DownloadOrchestrator so type
checking + IDE help match reality.
- tests/test_qobuz_credential_sync.py: docstring still referenced the
old soulseek_client global handle; updated to download_orchestrator
to match the rest of the codebase.
- core/downloads/monitor.py: the `for download in all_downloads` body
was over-indented (8 spaces past the for instead of 4) — purely
cosmetic but easy to mis-edit. Re-indented to one level.
A "type beat" is an instrumental track produced in another artist's
style, uploaded to SoundCloud and tagged with that artist's name to
game search ranking. They show up as candidates for major-label
tracks (e.g. "Eminem - Greatest (Kamikaze) Type Beat - Sit Down" for
"Greatest" by Eminem) and have nothing to do with the real song.
Add 'type beat' to the version-keyword list so the scorer applies the
0.4x penalty + flags the result as wrong_version. Currently the
matcher rejects them via low text-similarity scores anyway, but the
explicit keyword makes the rejection deterministic and gives a clear
diagnostic in the logs / modal.
The earlier validation-only filter only ran in the auto-search
scoring path. SoundCloud preview snippets still leaked through:
- The candidate-review modal cached raw search results (pre-validation),
so previews were visible and clickable for manual retry — and the
manual-pick download path bypassed validation entirely, downloading
the preview anyway.
- The not-found raw-results cache stored unfiltered top-20s.
Lift the preview filter into a reusable filter_soundcloud_previews()
helper and apply it at every entry point: validation scoring (still),
modal-cache fallback when validation drops everything, and the
not-found raw-results path. Previews now never reach the cache, the
matcher, or the manual-pick UI. Drops candidates < 35s or below half
the expected duration, gated on expected > 60s so genuine short
tracks still pass. 7 new unit tests pin the helper.
Also fixed a silent regression in core/downloads/task_worker.py's
hybrid-fallback path. Cin-5 dropped the per-source attrs from the
orchestrator (orch.soulseek, orch.youtube, etc.), but the fallback
loop still resolved sources via getattr(orch, '<src>', None) — every
lookup silently returned None, so remaining_sources came back empty
and the fallback never ran. Now uses orch.client(name) like the rest
of the codebase. Updated the test fake to expose client() too — the
old test was passing because the loop was effectively dead.
SoundCloud serves a ~30s preview clip for tracks gated behind Go+
or login (extremely common for major-label uploads — what's actually
on SoundCloud is bootlegs, fan reuploads, type beats, and these
previews). yt-dlp accepts the preview as the download payload, the
post-download integrity check catches the duration mismatch and
quarantines the file, but the user only sees "all candidates failed"
with no obvious explanation.
Filter at validation time when we know expected_duration: drop
SoundCloud candidates whose duration is below half the expected
length OR within ~5s of the 30s preview boundary, gated on
expected being non-trivially long (>60s) so genuinely short tracks
still pass through.
The global handle in web_server.py was named soulseek_client for
historical reasons but the type has long been DownloadOrchestrator,
not SoulseekClient. Renamed the global plus every parameter/attribute
that carried the legacy name.
- web_server.py: global var renamed; all 99 references updated.
- api/, core/downloads/*, core/search/*, core/streaming/*,
services/sync_service.py: parameter names, dataclass fields, and
init() arg names renamed.
- Test fixtures (CandidatesDeps, MasterDeps, SearchDeps, etc.) and
the _build_deps helpers updated accordingly.
The core.soulseek_client module path and SoulseekClient class name
(the actual soulseek-only client) are unchanged — only the orchestrator
handle renamed. Module imports of TrackResult/AlbumResult/DownloadStatus
from core.soulseek_client preserved.
Removed the eight backward-compat attribute aliases on the orchestrator
(soulseek, youtube, tidal, qobuz, hifi, deezer_dl, lidarr, soundcloud).
External callers and the orchestrator's own internals now reach clients
through the generic alias-aware client(name) accessor.
- core/downloads/{master,monitor,validation}.py: migrated to client().
Monitor's per-source aggregation loop replaced with a single
engine.get_all_downloads() call.
- core/search/{orchestrator,stream}.py: migrated; stream.py drops the
hand-built mode-to-client dict.
- web_server.py: migrated /api/deezer/arl-* + tidal client lookup.
- core/download_orchestrator.py: internal self.soulseek /
self.deezer_dl reaches now route through self.client(); attr
assignments dropped from __init__; module docstring updated.
- Test fakes (_FakeSoulseek, _FakeSoulseekWithYT) expose client(name)
instead of stuffing per-source attributes.
- Conformance test re-pinned to the client() accessor contract.
Plug the previously-built SoundcloudClient (PR #478, the build-and-verify
phase) into every place a download source needs to appear. Follows the
same wiring contract as Tidal/Qobuz/HiFi/Deezer/Lidarr — orchestrator
routing, hybrid-mode picker, search dispatch, queue/cancel/clear,
provenance + library history, sidebar source label, settings UI all
work plug-and-play.
Backend wiring:
- `core/download_orchestrator.py` — import SoundcloudClient, _safe_init
it at startup, add to _client() lookup, get_source_status(),
check_connection's sources_to_check default, search source_names map,
search_and_download_best _streaming_sources tuple, download
source_map + source_names, and every iteration loop in
reload_settings download-path-update / get_all_downloads /
get_download_status / cancel_download (route + iterate) /
clear_all_completed_downloads / cancel_all_downloads.
- `core/downloads/monitor.py` — added SoundCloud to the per-client
loop that fetches active downloads outside the orchestrator (uses
getattr fallback for older soulseek_client snapshots).
- `core/downloads/task_worker.py` — added SoundCloud (and Lidarr,
which was missing too — bonus fix) to source_clients dict for hybrid
fallback dispatch.
- `core/downloads/validation.py` — added 'soundcloud' to
_streaming_sources so SoundCloud results go through the matching
engine validation path instead of the Soulseek quality-filter path.
- `core/imports/side_effects.py` — three call sites: source_map for
download_source label written to library_history, streaming-source
guard for the `||`-encoded stream_id parsing, and source_service
map for provenance recording. All three now include 'soundcloud'.
- `web_server.py` — five streaming-source detection tuples updated.
New `/api/soundcloud/status` endpoint returns
{available, configured, reachable} mirroring the Deezer/HiFi
status-endpoint pattern; reachability runs a real cheap yt-dlp
search so the settings Test Connection button gives a meaningful
pass/fail signal.
- `config/settings.py` — added empty `soundcloud_download` defaults
block so future tier-2 OAuth (SoundCloud Go+ session) doesn't have
to migrate existing configs.
Frontend:
- `webui/index.html` — new `<option value="soundcloud">` in the
download-source-mode dropdown, SoundCloud added to both hidden
legacy hybrid-source selects, new settings container with info
text + Test Connection button.
- `webui/static/settings.js` — HYBRID_SOURCES entry (with the
SoundCloud cloud SVG icon), _hybridSourceEnabled default,
updateDownloadSourceUI container display, allSources for legacy
hybrid picker, testSoundcloudConnection function (hits the new
status endpoint, color-codes the result), saveSettings
soundcloud_download empty block.
- `webui/static/shared-helpers.js` — sidebar source-name map
includes SoundCloud + Lidarr (Lidarr was also missing, bonus fix).
- `webui/static/helper.js` — WHATS_NEW entry under '2.4.2' dev cycle
describing the user-visible change in the chill terse voice.
Tests:
- `tests/test_download_orchestrator_soundcloud.py` — 14 integration
tests verifying the wiring: client constructed at startup, _client
lookup resolves 'soundcloud', get_source_status includes it,
download dispatcher routes username='soundcloud' to the SoundCloud
client (and unknown usernames still fall back to Soulseek), hybrid
search iterates SoundCloud when in order and skips it cleanly when
unconfigured, get_all_downloads / get_download_status / cancel /
clear walk SoundCloud, soundcloud-only mode dispatches only to
SoundCloud, _streaming_sources tuple in validation includes
'soundcloud'.
- `tests/downloads/test_download_orchestrator.py` — added
`soundcloud` to the test fixture's _build_orchestrator helper so
the new orchestrator attribute doesn't AttributeError in pre-
existing tests that bypass __init__.
Verified:
- Full suite green (1728 passed, 2 deselected for soundcloud_live)
- Ruff clean
- Live SoundCloud-only mode search returns 25 SoundCloud tracks for
"kendrick lamar luther" in <2s, returning properly-shaped
TrackResult objects with username='soundcloud' and dispatch-key
filename ready for the download path.
Out of scope (intentional deferrals):
- SoundCloud Go+ OAuth tier (256 kbps AAC) — anonymous-only for now.
Adding auth later is a settings-page extension, no orchestrator
changes needed.
- Album/playlist support — SoundCloud has playlists but they don't
map to the album model the rest of SoulSync expects. Singles only.
Body byte-identical to the original. Wishlist helpers come from
core.wishlist.* directly (aliased to the same names the body uses);
runtime state from core.runtime_state. automation_engine,
soulseek_client, and _sweep_empty_download_directories are injected
via init() right after _init_download_validation.
web_server.py: 35408 → 35239 (-169 lines).
