Adds torrent/usenet as release-oriented download sources with album-bundle staging, live progress reporting, and post-processing that selects the requested audio file from completed releases instead of blindly importing the first file.
Keeps album-bundle behavior gated to single-source torrent/usenet album downloads, excludes release sources from hybrid album per-track searches, and allows hybrid non-album tracks to use release results safely.
Improves staged-release matching for featured/bonus track filenames while preserving version mismatches, records torrent/usenet provenance in library history, and updates service/status UI labels.
Covers the flow with focused lifecycle, status, staging, validation, task worker, post-processing, and import side-effect tests.
Route torrent and Usenet album bundles through private per-batch staging so Auto-Import cannot race public staging or duplicate imports.
Expose album-bundle progress in batch status and render it on the Downloads page while the external client is still downloading.
Tighten release handoff safety by rejecting archive path traversal, ignoring torrent candidates without a usable URL, and skipping Soulseek source reuse for torrent/Usenet batches.
Tests: .venv/bin/python -m pytest tests/downloads/test_downloads_status.py tests/test_album_bundle_dispatch.py tests/downloads/test_downloads_staging.py tests/test_torrent_usenet_plugins.py
Wraps up the code-review refactor pass.
- config/settings.py: ``download_source`` defaults gain
``album_bundle_poll_interval_seconds`` (default 2s) and
``album_bundle_timeout_seconds`` (default 6h, was a hard-coded
``6 * 60 * 60`` magic constant in torrent.py). The plugin reads
these via ``album_bundle.get_poll_interval`` /
``get_poll_timeout`` with safe fallback to the defaults when the
config value is missing / non-numeric. ``mode`` doc-comment
extended to list ``torrent`` and ``usenet``.
- core/downloads/validation.py: comment block above the album-name
fallback rewritten to document when the fallback actually runs
now — single-track hybrid downloads only, because the album-
bundle gate handles single-source mode and the hybrid chain
filter strips torrent / usenet from album batches. Code path
unchanged; just clarifies the contract for the next reader.
- webui/static/helper.js: WHATS_NEW entry summarising the refactor
pass (helper extraction, dispatch lift, staging deps injection,
atomic copy, configurable timeout, test additions).
The /loop of: extract → inject → test was sweep enough to drop the
gate code's coupling to 2-3 modules and put 49 unit tests behind
the new boundaries. Code-review feedback addressed:
1. album_bundle.py extracted ✓
2. Dispatch lifted out of master.py ✓
3. staging.py decoupled from runtime_state ✓
4. Validation fallback scope documented ✓
5. Poll timeout config-driven ✓
6. ``amazon`` provenance owned in a prior commit ✓
7. End-to-end-shaped tests added (test_album_bundle_dispatch.py)
8. Auto-Import race closed via atomic copy ✓
Per code review: the album-bundle provenance override added in an
earlier commit reached into ``core.runtime_state.download_batches``
directly from inside the staging matcher. Sibling modules
shouldn't import each other's globals — the existing StagingDeps
pattern is the canonical way to inject everything else this helper
needs.
- core/downloads/staging.py: new optional ``get_batch_field``
callable on ``StagingDeps`` (defaults to None for backward compat
with any caller that doesn't know about it yet). The inline
``from core.runtime_state import download_batches`` is gone; the
helper now calls ``deps.get_batch_field(batch_id,
'album_bundle_source')`` and falls back to 'staging' when None
is returned. Accessor exceptions are swallowed with a debug log
so a deleted batch mid-process can't break the staging match.
- web_server.py: ``_build_staging_deps`` injects a small
``_staging_get_batch_field`` helper that wraps the tasks_lock +
download_batches dict access. Centralises the lock semantics in
one place — the staging module no longer needs to know about
the lock or the dict.
- tests/test_staging_album_provenance.py: 5 new tests covering the
full matrix — torrent override applied, usenet override applied,
no override falls back to 'staging', missing accessor (default
None) falls back to 'staging', accessor raising falls back to
'staging'. Each test seeds + cleans a synthetic task in
runtime_state so the test doesn't bleed state across the suite.
Per code review: ~90 lines of inline gate logic in
``run_full_missing_tracks_process`` was inflating an already-580-
line worker function and was non-testable in isolation. Lifted to
``core/downloads/album_bundle_dispatch.py`` with two entry points:
- ``is_eligible(mode, is_album, album_name, artist_name)`` — pure
predicate, no side effects, easy to assert against. Splits the
gate decision from the resolution + run step so tests can pin
the gate semantics without standing up a plugin.
- ``try_dispatch(...)`` — full flow. Returns True iff the master
worker should stop (gate fired + failed); False = engaged-and-
succeeded OR didn't engage, both fall through to per-track.
State access is now decoupled from ``runtime_state``:
- New ``BatchStateAccess`` Protocol with two methods
(``update_fields``, ``mark_failed``).
- Concrete impl ``_BatchStateAccessImpl`` lives in master.py and
wraps the tasks_lock + dict ops the original inline code did.
- Injected via parameter so the dispatch module never imports
``download_batches`` / ``tasks_lock`` directly.
Same goes for the plugin resolver and config getter — both
injected, so the dispatcher works against any orchestrator /
config implementation (including the in-test fakes).
Behavior unchanged. The master worker call site is now 11 lines
of boilerplate instead of 90 lines of inline conditional. Plugin
contract (``download_album_to_staging`` return dict shape)
unchanged.
- core/downloads/album_bundle_dispatch.py: new module owning the
gate + execution. ~150 lines including docstrings and the
Protocol definition.
- core/downloads/master.py: gate call site shrunk to a single
``if _album_bundle_dispatch.try_dispatch(...): return``. New
``_BatchStateAccessImpl`` class implements the Protocol against
the existing ``download_batches`` dict + ``tasks_lock`` so the
dispatcher gets injected access instead of importing them.
- tests/test_album_bundle_dispatch.py: 16 new tests covering the
pure predicate (album-required, mode allowlist, name validation,
case insensitivity), the resolver-failure fall-through
(plugin missing, plugin lacks method, resolver raises), the
success path returning False so per-track flows, the failure
path returning True with state.mark_failed called, plugin-raise
treated as a normal failure, whitespace stripping on names,
and progress-callback mirroring lifecycle events into batch
state.
When a user picks Hybrid mode AND downloads an album, the per-track
search loop fires once per track. Torrent / usenet are release-level
sources — Prowlarr returns album torrents, none of which score
meaningfully against an individual track title. Without filtering,
every track triggered a redundant Prowlarr search, qBit rejected
duplicate hashes after the first, and the run only worked at all
because Auto-Import swept Staging behind the scenes. Confusing
logs, wasted searches, brittle timing.
Fix: thread an optional ``exclude_sources`` parameter through
``DownloadOrchestrator.search``. When the per-track worker detects
that the active batch is an album AND mode is hybrid, it passes
``['torrent', 'usenet']`` so the hybrid chain skips them and falls
through to per-track-compatible sources (Soulseek / streaming).
Gate is narrow on purpose:
- Hybrid + album → skip torrent / usenet (THIS fix)
- Single-source torrent / usenet + album → album-bundle flow on
the master worker (already shipped)
- Hybrid + single-track batch (basic search / wishlist / playlist
of singles) → torrent / usenet still tried, validation.py's
album-name fallback gives them a shot
Excluded list logged at INFO when applied so the behavior is
visible in logs ("Hybrid search: excluding ['torrent', 'usenet']
for this query"). Default ``exclude_sources=None`` keeps every
non-task-worker caller (basic search, stream search, search-and-
download-best, automation handlers) on the original code path.
The download history modal was tagging every torrent / usenet
album-bundle download as 'Soulseek FLAC 24bit' because:
- core/imports/side_effects.py's source_service dict didn't have
entries for 'staging', 'torrent', or 'usenet' usernames. The
staging matcher in core/downloads/staging.py sets
download_tasks[task_id]['username'] = 'staging', which fell
through to the dict's default and got recorded as 'soulseek'
in the track download provenance row. Same fate for any
amazon or other source that wasn't whitelisted.
- The album-bundle flow specifically wants to be labeled as
'torrent' or 'usenet' (where the bytes actually came from),
not 'staging' (the intermediate). The plugin already stashes
the source on the batch state as ``album_bundle_source`` for
the Downloads-page status card; provenance recording can
read the same field.
Fixes:
- core/downloads/staging.py: when marking a task post_processing
after a staging match, check the batch's album_bundle_source
override and use that for username instead of 'staging' when
set. Falls back to 'staging' when no override exists
(manual file-drop case).
- core/imports/side_effects.py: source_service map gets entries
for 'staging', 'torrent', 'usenet', and the previously-missing
'amazon' (which was also falling through to 'soulseek').
- webui/static/library.js: the redownload modal's serviceLabels
/ serviceIcons dicts extended to cover lidarr, amazon,
soundcloud, auto_import, staging, torrent, usenet so badges
render the correct name instead of either the raw source_service
string or no badge at all.
- webui/static/wishlist-tools.js: history-source-chip color
palette extended for the new source labels (Torrent sky-blue,
Usenet violet, Staging / Auto-Import neutral grey).
Note: existing tracks in the DB still carry the wrong 'soulseek'
label — only NEW downloads after this fix get the right label.
A future migration could rewrite historical rows but it's
cosmetic and the underlying audio + metadata are correct.
Fixes the core architectural mismatch between indexer-based sources
and the per-track search-and-pick contract every other download
plugin satisfies. Prowlarr returns release-level torrents and NZBs;
searching for "Luther (with SZA)" against the GNX album torrent
scores near-zero on track-title similarity. Per-track candidate
validation rejects every result, every track in the batch flips
to not_found. The album-name fallback added in an earlier commit
papers over it for some cases but doesn't fix the fundamental
behavior: the user wanted the whole album.
New album-bundle flow does what the user actually wanted:
1. Gate fires inside core/downloads/master.py BEFORE the per-track
analysis loop, strictly when the batch has an album context AND
download_source.mode is 'torrent' or 'usenet' (single-source —
hybrid stays per-track to preserve fallback to Soulseek / etc.).
2. Plugin's new download_album_to_staging method searches Prowlarr
ONCE for the album as a whole ('<artist> <album>'), filters to
the right protocol, runs results through _pick_best_album_release.
3. Picker prefers seeded FLAC over low-seeded MP3, drops single-
track torrents that snuck in via the 40 MB size floor (single
tracks are typically ~10 MB), falls back to most-seeded when
every candidate is below the floor.
4. Picked release goes to the active adapter (qBit / Transmission /
Deluge for torrent; SAB / NZBGet for usenet). Polls until
complete with progress mirrored into the batch state so the
Downloads page can show meaningful status.
5. On completion the existing archive_pipeline walks the save dir
(extracting archives if any), every audio file gets copied into
the staging folder via _unique_staging_path so concurrent batches
don't collide.
6. Gate exits, master worker continues into the normal per-track
flow. Each track task hits try_staging_match early in the worker
and finds its file by fuzzy title match — no Prowlarr search
ever fires per-track, no candidate rejection, files flow through
the existing post-processing pipeline (tags, AcoustID, library
import).
Gate is strictly opt-in. Three orthogonal conditions must all hold:
batch_is_album, mode in ('torrent', 'usenet'), and the plugin must
expose download_album_to_staging. Any other source / hybrid mode /
non-album batch flows through the master worker unchanged. The
existing per-track torrent path still works for basic-search
single-track grabs.
- core/download_plugins/torrent.py: download_album_to_staging plus
_pick_best_album_release and _unique_staging_path helpers (shared
with the usenet plugin). _poll_album_download mirrors the existing
poll loop with progress callback emission.
- core/download_plugins/usenet.py: parallel implementation reusing
the picker + staging helpers. Different state set ('failed' vs
'error') from the usenet adapter contract.
- core/downloads/master.py: ~90-line gate right after batch context
loading. Mirrors plugin lifecycle into batch state under
``album_bundle_*`` keys so the Downloads page can render progress
while the torrent/usenet job runs (per-track tasks don't exist
yet during this phase). Failed bundle download fails the batch
with a meaningful error; missing plugin / context falls back to
the per-track flow with a warning.
- tests/test_torrent_usenet_plugins.py: 5 new tests pinning the
album picker preferences (FLAC over MP3 with comparable size +
better seeders, size floor drops singles, fallback when all
small), staging-path collision suffix, and the not-configured
short-circuit.
Live-test bug: Spotify-flow downloads with Torrent Only as the
active source produced 'download_failed' for every track. Searches
hit Prowlarr fine but no candidate ever got picked. Root cause was
in core/downloads/validation.py's get_valid_candidates:
- The streaming-source allowlist for the structured-metadata path
didn't include 'torrent' / 'usenet', so torrent results fell into
the Soulseek matching branch.
- Soulseek matching parses ``candidate.filename`` as a slskd-style
``Artist/Album/Track.flac`` path. Torrent / usenet filenames are
encoded as ``<download_url>||<display_name>`` so the orchestrator
can recover the URL — splitting that string on slashes produced
garbage path segments that never matched the expected artist,
every candidate failed the artist-folder gate, returned [], track
status flipped to 'not_found'.
Fixes:
- _streaming_sources now includes 'torrent' and 'usenet'. They take
the structured-metadata scoring path that reads r.title / r.artist
directly (the projection layer pre-fills both correctly).
- Artist gate skipped for torrent/usenet, same as YouTube. Album-
level releases legitimately don't expose per-track artist — the
projection falls back to the indexer name as the 'artist' field,
which would otherwise fail the gate against every Spotify artist.
- New album-name fallback scoring: for torrent/usenet only, the
candidate title is ALSO scored against the wanted track's
spotify_track.album field, and the max of (track-title score,
album-title score) wins. This makes a candidate titled
"GNX (2024) [FLAC]" match every track on the GNX album rather
than scoring near zero against a specific track title like
"Luther (with SZA)". match_type 'album_release' for visibility.
All 9 existing validation tests still pass.
Add duration tolerance logic and pre-download rejection for structured sources (tidal, qobuz, hifi, deezer_dl, amazon) when candidate duration deviates beyond allowed tolerance. Introduces helper functions _duration_tolerance_seconds and _duration_mismatch_exceeds_integrity_tolerance and uses resolve_duration_tolerance from core.imports.file_integrity. Log and skip candidates that would fail post-processing integrity checks to avoid wasted downloads. Update tests to include matching engine stub and new cases covering rejection and acceptance based on duration tolerance; also adjust imports and test fixtures.
Do not mark a monitored transfer as successful as soon as slskd reports completion. The monitor now only submits the post-processing worker; that worker reports the real success or failure after finding, verifying, and importing the file. If post-processing cannot be scheduled, mark the task failed and release the batch slot. Add a regression test for the premature success path.
Diagnostic-only change for issue Technodude reported: Tidal sync-playlist
downloads getting mass-cancelled mid-flight with no clear cause in the
logs. App.log shows ~91 second gaps between Tidal download start and
cancel — matches the monitor's 90s queue-timeout exactly — but none of
the monitor's WARNING log lines fire, so the trigger is ambiguous
between five `_should_retry_task` paths, three web_server cancel paths,
and the API endpoints.
Added a single `[CancelTrigger:<label>]` INFO log line immediately
before every `download_orchestrator.cancel_download(...)` call so the
next log dump pins down which path is firing.
Labels (grep-able, prefix tells the file, suffix tells the trigger):
monitor.not_in_live_transfers_90s
monitor.errored_state_retry
monitor.queued_state_timeout
monitor.stuck_at_0pct_timeout
monitor.unknown_state_no_progress_timeout
candidates.worker_cancelled_during_download
web.orphan_cleanup
web.cancel_download_task
web.atomic_cancel_v2
api.manual_cancel_single
api.public_cancel
The monitor's `deferred_ops` tuple grew from 3 elements to 4 (added
trigger label as last element). The dispatch loop unpacks both legacy
and new shapes so the change is backward-compatible for any in-flight
ops mid-deploy.
Zero behavior change. 367 download tests still green. WHATS_NEW left
untouched — diagnostic only, not user-facing.
After ship: ask Technodude to re-run the same sync playlist scenario,
attach the new app.log, grep `[CancelTrigger:` lines for the trigger
context, then write the actual fix.
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.
Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
Add a conservative Soulseek album preflight scorer so album downloads choose a coherent slskd folder before per-track enqueue. The scorer compares album title, artist, year, track count, tracklist coverage, peer quality, and penalizes unexpected deluxe/remix/live-style folders.
Preserve hybrid source priority by only running Soulseek album preflight when Soulseek is the selected source or first in the hybrid order. If Soulseek is only a fallback behind another source, the normal hybrid flow is left alone.
Reuse the richest wishlist album context across tracks in the same album group so release date, artwork, album type, and album artist stay consistent for path generation. Also preserve peer-quality tie breakers when attempting equal-confidence candidates.
Tests cover correct-folder selection over larger wrong editions, Soulseek primary vs fallback hybrid behavior, shared wishlist album context, and peer-quality candidate ordering.
AcoustID verification was quarantining every Amazon track because T2Tunes
embeds [Explicit] and [feat. X] in stream tag titles/artists, but AcoustID
returns bare titles — triggering version-mismatch rejection on every track.
- get_track_details: apply _strip_edition to name/album, _primary_artist to
artist; wire s.track_number / s.disc_number instead of hardcoded None
- get_album_tracks: apply _strip_edition to name, _primary_artist to artist
Also fix TypeError crash in album download paths when disc_number is None
(present in dict but explicitly None, so .get('disc_number', 1) returns None):
- master.py run_full_missing_tracks_process: or 1 guard on both max() and disc_num
- candidates.py track_info extraction: or 1 guard on both disc_number reads
- web_server.py enhanced + standard album download max() calls: or 1 guard
`validation.py` had amazon absent from `_streaming_sources`, causing
Amazon TrackResult objects (bitrate=None, size=0) to fall through to
the Soulseek P2P code path and get rejected by
`filter_results_by_quality_preference`. Every album track was marked
not found.
Fix: add 'amazon' to every streaming-source guard tuple/set that was
previously missing it:
- core/downloads/validation.py — primary bug fix (quality-filter bypass)
- core/downloads/status.py — _STREAMING_SOURCE_NAMES frozenset
- core/downloads/task_worker.py — hybrid fallback client map
- core/imports/side_effects.py — || filename→stream-id extraction
- web_server.py — is_streaming_source, transfer list display,
candidate source label, _try_source_reuse, _store_batch_source
- tests/test_download_plugin_conformance.py — registry count + parametrize
Also updates the 2.5.3 What's New entry to drop the stale
"not yet wired" disclaimer.
Closes#589. Tracks from MTV Unplugged / Live At / unplugged albums
consistently failed AcoustID verification with "Version mismatch:
expected (live) but file is (original)". Two upstream bugs fed into
the false positive — the AcoustID gate itself was correctly catching
the wrong file Tidal had selected. Codex diagnosed all three layers,
this fixes the two upstream causes and leaves the verifier alone.
Bug 1 — album-scoped library check false-misses owned albums
`core/downloads/master.py:184` scored "Shy Away (MTV Unplugged Live)"
(source title from playlist) vs "Shy Away" (local DB stored title)
with raw string similarity. Massive length asymmetry → ~0.3 → below
the 0.7 threshold → marked missing. Combined with the
`allow_duplicates and batch_is_album` short-circuit that disables
the global fallback for album downloads, the user's already-owned
album re-triggered every track for download. Explains the screenshot
showing "0 found / 7 missing" on an album the user manually placed.
New pure helper `core/matching/album_context_title.py:strip_redundant_album_suffix`
strips trailing parenthetical / bracket / dash suffixes whose tokens
are fully subsumed by the album context — at least one version
marker (live / unplugged / acoustic / session / concert / tour)
overlapping with the album, and every other token is either a
known marker, a year, a tolerated noise word, or a word from the
album title. Album-context-implied "live" added when the album
mentions unplugged / concert / tour / session.
Wired into the album-confirmed scope ONLY (not global matching).
Compares both raw and normalized source titles per album track and
takes the max similarity, so the helper returning the input
unchanged (when album doesn't imply version context) preserves
the pre-fix behavior.
Bug 2 — Tidal qualifier filter only ran on fallback searches
`core/tidal_download_client.py:345` set `is_fallback = attempt_idx > 0`
and only filtered when `is_fallback and required_qualifiers`. Primary
search returned all results unfiltered, so a query for "Shy Away
(MTV Unplugged Live)" could accept the studio cut if Tidal happened
to rank it first. Now the qualifier filter applies to BOTH primary
and fallback search attempts — log message updated to indicate
which path triggered.
Bug 3 — qualifier check ignored album.name
The legacy `_track_name_contains_qualifiers` only inspected the
track name. For concert / unplugged releases the live signal
typically lives in the album title, not the track title. New
`_track_matches_qualifiers` accepts a track object and inspects
both `track.name` AND `track.album.name`. Legacy helper preserved
to keep its existing test contract.
AcoustID version-mismatch gate at core/acoustid_verification.py
left intact — it correctly catches genuinely-wrong files that slip
through upstream filters. The In My Feelings (Instrumental) test
that pins this behavior continues to pass.
19 tests on the album-context helper covering MTV Unplugged
variants, dash/parens/brackets suffix shapes, year tolerance,
plural-form markers, the implied-live set, anti-regression cases
(instrumental/remix on a studio album must NOT be stripped),
empty/none defensive paths.
13 tests on the Tidal qualifier helper covering legacy
track-name-only behavior preserved, qualifier in track name alone,
qualifier in album name alone (the MTV Unplugged scenario),
multi-qualifier requirements, no-qualifiers always passes,
defensive against missing track.album, word-boundary avoiding
substring false-matches, _extract_qualifiers picking up live +
unplugged from the user's exact reporter query.
Full suite: 3053 passed.
Three follow-on fixes to the manual-search candidates modal once people
started actually using it:
1. NDJSON streaming. Manual search waited for every source to return
before showing anything. Now streams one event per source as each
completes — header line, source_results per source, done terminator.
Frontend appends rows incrementally via response.body.getReader().
2. Manual picks no longer auto-retry on failure. New _user_manual_pick
flag set on the task in /download-candidate. Both monitor retry
paths (not-in-live-transfers stuck + Errored state) bail on the
flag. Surfaces the failure to the user instead of silently picking
a different candidate via fresh search.
3. Non-Soulseek manual picks (youtube/tidal/qobuz/hifi/deezer/
soundcloud/lidarr) no longer stuck at "downloading 0%" forever. The
live_transfers IF branch now marks manual-pick tasks failed
directly when the engine reports Errored, instead of deferring to
the monitor (which bails on manual picks). Engine fallback in else
branch covers the rare race where the orchestrator's pre-populated
transfer lookup is missing the entry.
Plus a deadlock fix discovered along the way: the new failure path
synchronously called on_download_completed while holding tasks_lock,
which itself re-acquires the same Lock — non-reentrant
threading.Lock self-deadlocked the polling thread. While wedged, every
other endpoint that needed the lock (including /candidates → other
failed rows couldn't open modals) hung waiting. Moved completion
callbacks onto a daemon thread so the lock releases first.
Plus failed/not_found/cancelled rows are now ALWAYS clickable (not
just when the auto-search cached candidates) — the modal carries the
manual search bar, which is the user's recourse for empty results.
Plus manual download worker now runs on a dedicated thread instead of
competing with the batch's 3-worker missing_download_executor pool —
saturated batches no longer queue manual picks indefinitely.
All scoped to manual picks via the _user_manual_pick flag — auto
attempt flow byte-identical to before. Engine fallback gated on the
flag too so auto attempts in the else branch keep the original
do-nothing behavior (safety valve handles the stuck-forever case).
Also dropped _handle_failed_download from web_server.py — defined
but had no callers (dead code).
17 new unit tests pin the gate behavior:
- engine fallback: Errored/Cancelled/Succeeded/InProgress transitions,
manual-pick gate, terminal-state skip, soulseek skip, missing
download_id skip, engine returning None, orchestrator exception
- monitor: manual-pick skips not-in-live-transfers retry + Errored
retry
- IF-branch end-to-end: Errored marks failed, "Completed, Errored"
hits failure branch, auto attempts defer to monitor
Manual-search endpoint tests rewritten for NDJSON: 11 cases (validation,
single-source dispatch, parallel "all" dispatch, one-event-per-source
streaming shape, unconfigured-source skip + reject, header metadata,
per-source exception isolation).
Full suite 2259 passed, 1 skipped.
Two findings from JohnBaumb on the engine refactor.
(1) Every download client returned None when self._engine was None,
just logging an error. The orchestrator's download_with_fallback
treated None as "source declined", so the user got no feedback —
download silently disappeared. Now each client raises a RuntimeError
on the engine-not-wired path. download_with_fallback already catches
plugin exceptions, logs a warning, and tries the next source — so
the visible behavior is "real error in logs + fallback to next
source" instead of "silent drop". Six clients touched (deezer, hifi,
qobuz, soundcloud, tidal, youtube). Pinning tests updated to expect
raise.
(2) Monitor's engine.get_all_downloads() walked every plugin
including soulseek, but the same monitor loop already pulled slskd
transfers via the transfers/downloads endpoint a few lines earlier —
soulseek's records were being fetched twice per tick. Same issue in
web_server.py's get_cached_transfer_data path. Added an exclude
parameter to engine.get_all_downloads(); both call sites now pass
('soulseek',). New test pins the exclude semantic.
Also fixed a stray 8-space over-indent on the for-loop body in
get_cached_transfer_data (cosmetic, JohnBaumb flagged the same
pattern in monitor.py earlier).
Three small follow-ups from the Copilot review of the rename PR:
- services/sync_service.py: PlaylistSyncService.__init__'s
download_orchestrator parameter was annotated as SoulseekClient,
which was misleading (the object passed is the DownloadOrchestrator
with .search_and_download_best, .download, etc — not a SoulseekClient).
Switched the import + annotation to DownloadOrchestrator so type
checking + IDE help match reality.
- tests/test_qobuz_credential_sync.py: docstring still referenced the
old soulseek_client global handle; updated to download_orchestrator
to match the rest of the codebase.
- core/downloads/monitor.py: the `for download in all_downloads` body
was over-indented (8 spaces past the for instead of 4) — purely
cosmetic but easy to mis-edit. Re-indented to one level.
A "type beat" is an instrumental track produced in another artist's
style, uploaded to SoundCloud and tagged with that artist's name to
game search ranking. They show up as candidates for major-label
tracks (e.g. "Eminem - Greatest (Kamikaze) Type Beat - Sit Down" for
"Greatest" by Eminem) and have nothing to do with the real song.
Add 'type beat' to the version-keyword list so the scorer applies the
0.4x penalty + flags the result as wrong_version. Currently the
matcher rejects them via low text-similarity scores anyway, but the
explicit keyword makes the rejection deterministic and gives a clear
diagnostic in the logs / modal.
The earlier validation-only filter only ran in the auto-search
scoring path. SoundCloud preview snippets still leaked through:
- The candidate-review modal cached raw search results (pre-validation),
so previews were visible and clickable for manual retry — and the
manual-pick download path bypassed validation entirely, downloading
the preview anyway.
- The not-found raw-results cache stored unfiltered top-20s.
Lift the preview filter into a reusable filter_soundcloud_previews()
helper and apply it at every entry point: validation scoring (still),
modal-cache fallback when validation drops everything, and the
not-found raw-results path. Previews now never reach the cache, the
matcher, or the manual-pick UI. Drops candidates < 35s or below half
the expected duration, gated on expected > 60s so genuine short
tracks still pass. 7 new unit tests pin the helper.
Also fixed a silent regression in core/downloads/task_worker.py's
hybrid-fallback path. Cin-5 dropped the per-source attrs from the
orchestrator (orch.soulseek, orch.youtube, etc.), but the fallback
loop still resolved sources via getattr(orch, '<src>', None) — every
lookup silently returned None, so remaining_sources came back empty
and the fallback never ran. Now uses orch.client(name) like the rest
of the codebase. Updated the test fake to expose client() too — the
old test was passing because the loop was effectively dead.
SoundCloud serves a ~30s preview clip for tracks gated behind Go+
or login (extremely common for major-label uploads — what's actually
on SoundCloud is bootlegs, fan reuploads, type beats, and these
previews). yt-dlp accepts the preview as the download payload, the
post-download integrity check catches the duration mismatch and
quarantines the file, but the user only sees "all candidates failed"
with no obvious explanation.
Filter at validation time when we know expected_duration: drop
SoundCloud candidates whose duration is below half the expected
length OR within ~5s of the 30s preview boundary, gated on
expected being non-trivially long (>60s) so genuinely short tracks
still pass through.
The global handle in web_server.py was named soulseek_client for
historical reasons but the type has long been DownloadOrchestrator,
not SoulseekClient. Renamed the global plus every parameter/attribute
that carried the legacy name.
- web_server.py: global var renamed; all 99 references updated.
- api/, core/downloads/*, core/search/*, core/streaming/*,
services/sync_service.py: parameter names, dataclass fields, and
init() arg names renamed.
- Test fixtures (CandidatesDeps, MasterDeps, SearchDeps, etc.) and
the _build_deps helpers updated accordingly.
The core.soulseek_client module path and SoulseekClient class name
(the actual soulseek-only client) are unchanged — only the orchestrator
handle renamed. Module imports of TrackResult/AlbumResult/DownloadStatus
from core.soulseek_client preserved.
Removed the eight backward-compat attribute aliases on the orchestrator
(soulseek, youtube, tidal, qobuz, hifi, deezer_dl, lidarr, soundcloud).
External callers and the orchestrator's own internals now reach clients
through the generic alias-aware client(name) accessor.
- core/downloads/{master,monitor,validation}.py: migrated to client().
Monitor's per-source aggregation loop replaced with a single
engine.get_all_downloads() call.
- core/search/{orchestrator,stream}.py: migrated; stream.py drops the
hand-built mode-to-client dict.
- web_server.py: migrated /api/deezer/arl-* + tidal client lookup.
- core/download_orchestrator.py: internal self.soulseek /
self.deezer_dl reaches now route through self.client(); attr
assignments dropped from __init__; module docstring updated.
- Test fakes (_FakeSoulseek, _FakeSoulseekWithYT) expose client(name)
instead of stuffing per-source attributes.
- Conformance test re-pinned to the client() accessor contract.
Plug the previously-built SoundcloudClient (PR #478, the build-and-verify
phase) into every place a download source needs to appear. Follows the
same wiring contract as Tidal/Qobuz/HiFi/Deezer/Lidarr — orchestrator
routing, hybrid-mode picker, search dispatch, queue/cancel/clear,
provenance + library history, sidebar source label, settings UI all
work plug-and-play.
Backend wiring:
- `core/download_orchestrator.py` — import SoundcloudClient, _safe_init
it at startup, add to _client() lookup, get_source_status(),
check_connection's sources_to_check default, search source_names map,
search_and_download_best _streaming_sources tuple, download
source_map + source_names, and every iteration loop in
reload_settings download-path-update / get_all_downloads /
get_download_status / cancel_download (route + iterate) /
clear_all_completed_downloads / cancel_all_downloads.
- `core/downloads/monitor.py` — added SoundCloud to the per-client
loop that fetches active downloads outside the orchestrator (uses
getattr fallback for older soulseek_client snapshots).
- `core/downloads/task_worker.py` — added SoundCloud (and Lidarr,
which was missing too — bonus fix) to source_clients dict for hybrid
fallback dispatch.
- `core/downloads/validation.py` — added 'soundcloud' to
_streaming_sources so SoundCloud results go through the matching
engine validation path instead of the Soulseek quality-filter path.
- `core/imports/side_effects.py` — three call sites: source_map for
download_source label written to library_history, streaming-source
guard for the `||`-encoded stream_id parsing, and source_service
map for provenance recording. All three now include 'soundcloud'.
- `web_server.py` — five streaming-source detection tuples updated.
New `/api/soundcloud/status` endpoint returns
{available, configured, reachable} mirroring the Deezer/HiFi
status-endpoint pattern; reachability runs a real cheap yt-dlp
search so the settings Test Connection button gives a meaningful
pass/fail signal.
- `config/settings.py` — added empty `soundcloud_download` defaults
block so future tier-2 OAuth (SoundCloud Go+ session) doesn't have
to migrate existing configs.
Frontend:
- `webui/index.html` — new `<option value="soundcloud">` in the
download-source-mode dropdown, SoundCloud added to both hidden
legacy hybrid-source selects, new settings container with info
text + Test Connection button.
- `webui/static/settings.js` — HYBRID_SOURCES entry (with the
SoundCloud cloud SVG icon), _hybridSourceEnabled default,
updateDownloadSourceUI container display, allSources for legacy
hybrid picker, testSoundcloudConnection function (hits the new
status endpoint, color-codes the result), saveSettings
soundcloud_download empty block.
- `webui/static/shared-helpers.js` — sidebar source-name map
includes SoundCloud + Lidarr (Lidarr was also missing, bonus fix).
- `webui/static/helper.js` — WHATS_NEW entry under '2.4.2' dev cycle
describing the user-visible change in the chill terse voice.
Tests:
- `tests/test_download_orchestrator_soundcloud.py` — 14 integration
tests verifying the wiring: client constructed at startup, _client
lookup resolves 'soundcloud', get_source_status includes it,
download dispatcher routes username='soundcloud' to the SoundCloud
client (and unknown usernames still fall back to Soulseek), hybrid
search iterates SoundCloud when in order and skips it cleanly when
unconfigured, get_all_downloads / get_download_status / cancel /
clear walk SoundCloud, soundcloud-only mode dispatches only to
SoundCloud, _streaming_sources tuple in validation includes
'soundcloud'.
- `tests/downloads/test_download_orchestrator.py` — added
`soundcloud` to the test fixture's _build_orchestrator helper so
the new orchestrator attribute doesn't AttributeError in pre-
existing tests that bypass __init__.
Verified:
- Full suite green (1728 passed, 2 deselected for soundcloud_live)
- Ruff clean
- Live SoundCloud-only mode search returns 25 SoundCloud tracks for
"kendrick lamar luther" in <2s, returning properly-shaped
TrackResult objects with username='soundcloud' and dispatch-key
filename ready for the download path.
Out of scope (intentional deferrals):
- SoundCloud Go+ OAuth tier (256 kbps AAC) — anonymous-only for now.
Adding auth later is a settings-page extension, no orchestrator
changes needed.
- Album/playlist support — SoundCloud has playlists but they don't
map to the album model the rest of SoulSync expects. Singles only.
Body byte-identical to the original. Wishlist helpers come from
core.wishlist.* directly (aliased to the same names the body uses);
runtime state from core.runtime_state. automation_engine,
soulseek_client, and _sweep_empty_download_directories are injected
via init() right after _init_download_validation.
web_server.py: 35408 → 35239 (-169 lines).
Body byte-identical to the original. matching_engine and
soulseek_client are injected via init() right after _init_discover_hero
since both originals are constructed early in web_server.py boot
(L598/L610) and never rebound.
web_server.py: 35586 → 35408 (-178 lines).
- Switch the download lifecycle over to the neutral wishlist track helper name
- Keep the old Spotify helper as a compatibility alias for older callers
- Store track_data as the primary failed-download wishlist payload key and add regression coverage
Class body byte-identical to original. Module-level IS_SHUTTING_DOWN
flag is mirrored from web_server's own flag in _shutdown_runtime_components
so the monitor loop still sees shutdown signals at the right moment.
Eight web_server-side helpers (_make_context_key, _on_download_completed,
_run_post_processing_worker, _download_track_worker,
_start_next_batch_of_downloads, _orphaned_download_keys,
missing_download_executor, soulseek_client) are injected via init() after
register_runtime_clients, when all symbols are defined and well before
Flask starts accepting requests.
web_server.py: 38220 → 37501 (-719 lines).
Pulls the 201-line staging-folder shortcut out of `web_server.py` into
its own module under the existing `core/downloads/` package. Pure 1:1
lift — wrapper keeps the original entry-point name so the task worker's
existing call site continues to work without changes.
What `try_staging_match` does:
1. Pull the per-batch staging-file cache (one filesystem scan per batch).
2. For each staging entry, compute title + artist similarity using
SequenceMatcher and the matching engine's `normalize_string`. Require
title >= 0.80, then a combined score >= 0.75. The weighting flips
based on whether artist info is available on both sides:
- both have artist: 0.55*title + 0.45*artist
- either side missing artist: 0.80*title + 0.20*artist (lean on title)
3. Copy the matched file to the configured transfer dir (with a
"_staging" suffix when the destination filename already exists, to
avoid overwriting a legitimate prior download).
4. Mark the task as 'post_processing', username='staging',
staging_match=True.
5. Build a synthetic spotify_artist / spotify_album context (mirroring
the modal-worker logic so the file-organization template applies
cleanly) and store it under "staging_<task_id>". Two paths:
- Explicit context branch (track_info has _is_explicit_album_download)
→ real album/artist data copied through.
- Fallback branch → synthesized from track + track_info, with
`is_album_download` heuristically derived (album differs from title
and isn't "Unknown Album").
6. Hand off to `_post_process_matched_download_with_verification` which
does tagging, path building, AcoustID verification, and DB insertion.
Returns True if the staging shortcut won; False to fall through to the
normal Soulseek search path.
Dependencies injected via `StagingDeps` (5 fields) — config_manager,
matching_engine, get_staging_file_cache, docker_resolve_path,
post_process_matched_download_with_verification.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 201 lines orig = 201 lines lifted, byte-identical body
(including all whitespace, comments, log strings, and the inline
`from difflib import SequenceMatcher` / `import shutil` imports inside
the function body).
Tests: 9 new under tests/downloads/test_downloads_staging.py covering
no staging files / no track title / low-confidence match returning
False, exact match copying file + transitioning task state + invoking
post-processing, existing-file rename via `_staging` suffix, explicit
album context branch, fallback context synthesis (with both album-as-
album and album-equals-title cases), and copy failure (missing source
file) returning False.
Full suite: 1308 passing (was 1299). Ruff clean.
First lift in the new PR6 batch. Pulls the 312-line candidate-fallback
download dispatcher out of `web_server.py` into a new module under the
existing `core/downloads/` package. Pure 1:1 lift — wrapper keeps the
original entry-point name so all callers (search/match pipeline) work
unchanged.
What `attempt_download_with_candidates` does:
1. Sort candidates by descending confidence.
2. For each candidate:
- Cancellation gates (3 points: top of loop, before download starts,
after download_id is assigned).
- Skip already-tried sources via the per-task `used_sources` set.
- Skip blacklisted sources (user-flagged bad matches).
- Race protection: bail when the task already has an active
download_id.
- `update_task_status('downloading')`, then `soulseek_client.download`.
3. On a successful download_id:
- Build `matched_downloads_context` entry keyed by
`make_context_key(username, filename)`.
- For tracks with clean Spotify metadata, pull track_number /
disc_number from (1) track_info → (2) track object → (3) Spotify
API call. When local album context is incomplete, the API response
backfills release_date / album_type / total_tracks / images / id.
- Set `is_album_download` based on explicit context flag or
heuristic (album differs from title, isn't "Unknown Album").
- Store task/batch IDs and track_info on the context for post-
processing + playlist-folder mode.
4. On a cancellation that wins the race after the download started:
- `cancel_download(...)` to stop the in-flight Soulseek transfer.
- `on_download_completed(batch_id, task_id, success=False)` to free
the worker slot.
5. On exception or download-start failure: reset task status to
'searching', continue to next candidate.
Dependencies injected via `CandidatesDeps` (7 fields) — soulseek_client,
spotify_client, run_async, get_database, update_task_status,
make_context_key, on_download_completed.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 312 lines orig = 312 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 14 new under tests/downloads/test_downloads_candidates.py
covering happy path (first candidate succeeds, confidence ordering),
used_sources dedup, blacklist skip, cancellation gates (cancelled
status, deleted task, active download_id, mid-flight cancel + cleanup
callback), failure paths (all candidates failed, exception during
download falls through to next), context payload (explicit album
context, track_number priority order, API backfill of incomplete album
metadata), and equal-confidence stable order.
Pre-existing behavior documented in tests:
`spotify_album_context['id']` initializes to a non-empty placeholder
'from_sync_modal' in the fallback path, so the API-backfill condition
`if not spotify_album_context.get('id')` never fires for the id field
specifically. Other album fields (release_date, album_type) backfill
fine because they default to empty.
Full suite: 1290 passing (was 1276). Ruff clean.
Three drifts caught in line-by-line review against original — all reverted
so the lifted body is now byte-identical to the web_server.py original
(after deps.X → global X normalization):
1. `import traceback` was hoisted to module-level imports. Original imports
it inline inside the except block. Moved back to inline (same behavior,
but strict location parity).
2. Trailing whitespace on the blank line after analysis_results.append.
3. Trailing whitespace on the blank line inside the wishlist except handler.
Diff against original now produces ZERO differences. Tests still 21/21
passing, ruff clean.
Final extraction in the download orchestrator series. Lifts the 586-line
master worker that drives the entire missing-tracks pipeline from
`web_server.py` into `core/downloads/master.py`. Pure 1:1 lift — wrappers
keep the original entry-point name so the three callers
(`missing_download_executor.submit(_run_full_missing_tracks_process, ...)`)
continue to work without changes.
What the master worker does:
1. PHASE 1 ANALYSIS — per-track DB ownership check with album fast path
(lookup album by name+artist, match tracks within it) plus a
MusicBrainz release-cache preflight so per-track post-processing all
uses the same release MBID (prevents Navidrome album splits).
2. Wishlist removal for tracks already in the library.
3. Explicit-content filter.
4. PHASE 2 transition — if nothing missing, mark batch complete, update
per-source playlist phases, kick auto-wishlist completion handler.
5. Soulseek album pre-flight — search for a complete album folder before
falling back to track-by-track search, cache the source for reuse.
6. Wishlist album grouping — derive per-album disc counts and resolve ONE
artist context per album so collab albums don't fold-split.
7. Task creation with explicit album/artist context injection +
playlist-folder-mode flag propagation.
8. Hand off to download_monitor + start_next_batch_of_downloads.
9. Error handler — phase=error, reset YouTube playlist phase to
'discovered', reset auto-wishlist globals on auto-initiated batches.
Dependencies injected via `MasterDeps` (21 fields) — wide surface
covering config, MB caches/locks, soulseek client, source-page state
dicts, multiple callbacks (wishlist removal, explicit filter, executor
+ auto-completion fn, monitor, start_next_batch). The only behaviour
difference from a pure paste is `import traceback` hoisted to module
scope (was inline in the except block) — same behaviour. Trailing
whitespace on two blank lines also got normalized away by the editor;
neither has any runtime effect.
`reset_wishlist_auto_processing` callback wraps the
`global wishlist_auto_processing, wishlist_auto_processing_timestamp`
write + `wishlist_timer_lock` since `global` can't reach back into
web_server.py from a separate module.
Tests: 21 new under tests/downloads/test_downloads_master.py covering
analysis-phase state, force_download_all, found-track wishlist removal,
explicit filter, no-missing complete + per-source state updates, auto
wishlist completion submit, album fast path (direct + fallthrough),
MB preflight (caches both keys, no-mb-worker no-op), task creation
(queue + tasks dict, explicit context for albums, wishlist album
grouping consistency, playlist folder mode), monitor + next-batch
handoff, multi-disc total_discs computation, error handler (phase set,
youtube reset, auto wishlist reset), and batch-removed-mid-flight
defensive path.
Full suite: 1050 passing (was 1029). Ruff clean.
End of the PR4 series — `web_server.py` lost ~590 lines on this commit
alone; total trim across PR4a–PR4h is ~2900 lines of orchestrator code
moved into focused `core/downloads/*.py` modules.
Seventh sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change. ~570 lines moved out of web_server.py.
What moved (lifted as 3 tightly-coupled functions in one module):
- _start_next_batch_of_downloads → start_next_batch_of_downloads
- _on_download_completed → on_download_completed
- _check_batch_completion_v2 → check_batch_completion_v2
Dependencies bundled in `LifecycleDeps` (15+ refs):
- config_manager, automation_engine, download_monitor, repair_worker,
mb_worker (live globals)
- is_shutting_down (lambda over IS_SHUTTING_DOWN flag)
- get_batch_lock (web_server helper for batch_locks dict)
- submit_download_track_worker (lambda wrapping
missing_download_executor.submit + _download_track_worker)
- submit_failed_to_wishlist + submit_failed_to_wishlist_with_auto_completion
(async, used by on_download_completed) AND process_failed_to_wishlist
+ process_failed_to_wishlist_with_auto_completion (sync, used by
check_batch_completion_v2 — direct call matches original v2 behavior;
the non-v2 path always submitted to executor)
- ensure_spotify_track_format, get_track_artist_name,
check_and_remove_from_wishlist, regenerate_batch_m3u (web_server
helpers — large, will lift in follow-up PRs)
- youtube_playlist_states, tidal_discovery_states,
deezer_discovery_states, spotify_public_discovery_states
(per-source playlist state dicts — phase transitions on batch
completion)
Direct imports for already-lifted helpers:
- core.runtime_state.{download_tasks, download_batches, tasks_lock,
add_activity_item}
- core.downloads.history.record_sync_history_completion (PR4a)
- core.album_consistency.run_album_consistency
- core.metadata.common.get_file_lock
Behavior parity verified line-by-line:
- start_next_batch: same batch lock acquisition, same shutdown gate,
same V2-cancelled-task skip, same searching-status-set-before-submit,
same submit-fails-no-ghost-worker semantics
- on_download_completed: same duplicate-call detection (skip decrement
but still check completion), same failed-track tracking with
spotify_track formatting + activity items + automation event emission,
same wishlist removal on success, same active_count decrement, same
stuck-detection (searching > 10min → not_found, post_processing >
5min → completed), same M3U regeneration + repair worker hand-off
+ album consistency pass + wishlist failed-tracks submission
- check_batch_completion_v2: same finished-count tally, same stuck
detection, same already-complete short-circuit returning True, same
per-source playlist phase updates, same album consistency pass,
same DIRECT (sync) wishlist processing call (NOT submit-to-executor
— matches original v2 which called process_* functions directly)
CRITICAL drift caught + fixed during review:
- Initial lift had v2 routing wishlist calls through submit_* deps
(async). Original v2 called process_* directly (sync). Added separate
process_* deps to LifecycleDeps and routed v2 to them. Tests updated.
Two minor defensive additions documented:
- `is_auto_batch = False` initialized before conditional in v2 (Python
scope rules made this unnecessary in original, but explicit is safer)
- Variable rename inside the queue-completion-check loop in
on_download_completed: `task_id` → `queue_task_id` to avoid shadowing
the outer parameter. Log output preserves the same task ID.
Tests: 28 new under tests/downloads/test_downloads_lifecycle.py
covering start-next (early-returns, shutdown gate, max_concurrent,
cancelled-task skip, searching-status-set, submit-failure-no-ghost,
orphan task), on-complete (decrement, duplicate skip, failed/cancelled
tracking, automation emit, wishlist removal, batch completion + emit
+ source phase update, stuck detection, auto vs manual routing),
check-v2 (missing batch, not-complete, complete-marking, already-
complete, auto routing, exception handling).
Full suite: 1029 passing (was 1001). Ruff clean.
Fifth sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _build_batch_status_data → build_batch_status_data
- get_batch_download_status route body → build_single_batch_status
- get_batched_download_statuses route body → build_batched_status
- get_all_downloads_unified route body → build_unified_downloads_response
- Status priority dict → module-level _STATUS_PRIORITY constant
Dependencies bundled in `StatusDeps` dataclass:
- config_manager, docker_resolve_path, find_completed_file,
make_context_key, submit_post_processing (lambda wrapping
missing_download_executor.submit + _run_post_processing_worker),
get_cached_transfer_data
Direct imports from core.runtime_state for download_tasks /
download_batches / tasks_lock (already lifted by kettui).
Behavior parity:
- Same response payload shape across all 3 endpoints
- Same safety-valve mutation: stuck downloading task with file recovered
→ status='post_processing' + submit worker; stuck searching → not_found;
stuck downloading no file → failed
- Same live transfer state mapping (Cancelled/Canceled, Failed/Errored/
Rejected/TimedOut, Completed/Succeeded with byte-mismatch verification,
InProgress, default queued)
- Same intermediate post_processing status promotion + single-shot worker
submission (only when status != 'post_processing')
- Same 'Errored' handling: keeps current status to let monitor retry
- Same 17-key item dict in unified response with same field order
- Same artist/album/artwork normalization (handles string, dict, list,
list-of-dicts, list-of-strings variants)
- Same sort: (priority asc, -timestamp desc)
- Same batch summary aggregation
- Same items[:limit] slicing
- Same logger messages text-for-text
- Same lock scope (single tasks_lock per call) — no new contention
Pre-existing bug preserved (will fix in follow-up PR):
- batched_status `debug_info` block iterates `response["batches"]` and
guards with `if "error" not in batch_status`. Every successful
payload includes `"error": batch.get('error')` (key always present,
value usually None) so the guard is always False and debug_info
never populates in production. Test documents the buggy behavior so
the next PR can flip the check to `batch_status.get('error') is None`.
Tests: 32 new under tests/downloads/test_downloads_status.py covering
phase routing (analysis vs downloading vs unknown), task formatting +
sort + V2 fields, every live transfer state mapping (Cancelled,
Succeeded with full + partial bytes, InProgress, Errored, terminal-
not-overridden), safety valve (stuck searching → not_found, stuck
downloading recovered → post_processing, stuck downloading no file →
failed), all 3 route helpers (single, batched, unified), unified
artist/album/artwork normalization, batch summary aggregation, limit
slicing, plus debug_info bug documentation.
Full suite: 982 passing (was 950). Ruff clean.
Fourth sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change. ~407 lines moved out of web_server.py.
What moved:
- _run_post_processing_worker → run_post_processing_worker
The lifted function is intentionally kept as one ~400-line block to
preserve byte-for-byte parity with the original. Refactoring it into
smaller helpers (context lookup, file search loop, transfer-folder
handler, downloads-folder handler) gets its own follow-up PR.
Dependencies: 9 callbacks bundled in `PostProcessDeps` dataclass.
- config_manager, soulseek_client, run_async (live refs)
- docker_resolve_path, extract_filename, make_context_key
(small utilities still in web_server.py — will lift in a future PR
alongside other shared utilities)
- find_completed_file (file search helper, still in web_server.py)
- enhance_file_metadata, wipe_source_tags (web_server wrappers around
core.metadata.enrichment)
- post_process_with_verification (web_server wrapper around
core.imports.pipeline)
- mark_task_completed (wraps runtime_state.mark_task_completed +
session counter)
- on_download_completed (deferred to PR4g batch lifecycle)
Direct imports for already-lifted helpers (no injection needed):
- core.imports.album_naming.resolve_album_group
- core.imports.context.{get_import_clean_title, get_import_clean_album,
get_import_original_search, get_import_context_artist,
get_import_context_album, normalize_import_context}
- core.imports.filename.extract_track_number_from_filename
- core.metadata.enrichment (re-exported as metadata_enrichment)
- core.runtime_state.{download_tasks, tasks_lock,
matched_downloads_context, matched_context_lock}
Behavior parity:
- Same control flow: missing-task short-circuit → cancelled/completed
short-circuit → missing-filename failure → docker path resolution →
context lookup with fuzzy fallback → expected filename generation →
YouTube special-case path resolution → 5-attempt search loop with
Strategy 1 (original filename in download+transfer) and Strategy 2
(expected final filename in transfer) → file-not-found failure →
transfer-folder handler with metadata enhancement → downloads-folder
handler with full post-process verification
- Same retry count (5), sleep duration (5s), per-attempt logging
- Same album_info dict construction with is_album=True for explicit
album downloads
- Same album grouping skip when context.is_album_download is True
- Same wipe_source_tags fallback when enhancement context missing
- Same matched_downloads_context cleanup on success
- Same exception swallowing at processing-error and critical-error
layers, both setting status='failed' + error_message + calling
on_download_completed(b, t, success=False)
- Every logger message text preserved verbatim (so log filters keep
working)
Tests: 16 new under tests/downloads/test_downloads_post_processing.py
covering missing task, cancelled, already-completed, stream_processed,
missing filename + username, file-not-found-after-retries with sleep
mocked, stream-processor-completes-mid-search, transfer-folder with
metadata enhanced + with no context (wipes tags), downloads-folder
with + without context, processing exception, critical outer
exception, YouTube special path, fuzzy context matching.
Full suite: 950 passing (was 934). Ruff clean.
Third sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _automatic_wishlist_cleanup_after_db_update → cleanup_wishlist_after_db_update
The lifted fn takes config_manager as an arg (so core/downloads/cleanup.py
doesn't need to import web_server). Other deps (wishlist_service,
MusicDatabase, get_database) stay as in-function imports — matches the
original deferred-import pattern.
The single caller in web_server.py (missing_download_executor.submit at
L18028) keeps using the same wrapper name with no signature change.
Behavior parity:
- Same per-profile iteration via get_all_profiles()
- Same essential-field skip (no name / no artists / no spotify_track_id)
- Same artist normalization (string / dict / fallback to str())
- Same 0.7 confidence threshold for db match
- Same break-on-first-artist-match semantics
- Same album extraction (dict.name vs string passthrough)
- Same active_server pulled via config_manager.get_active_media_server()
- Same per-track exception swallowing inside the loops
- Same top-level exception swallow with traceback.print_exc()
- Same logger messages (exact text match for "[Auto Cleanup]" prefix)
Tests: 13 new under tests/downloads/test_downloads_cleanup.py covering
empty wishlist short-circuit, found-in-db removal, missed track stays,
low-confidence skip, missing-fields skip, dict + string artist formats,
break-on-first-match, multi-profile walk, album dict/string handling,
db check failure continuing to next artist, top-level exception swallow,
active server propagation.
Full suite: 934 passing (was 921). Ruff clean.
Second sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- cancel_download (single slskd cancel) → cancel_single_download
- cancel_all_downloads (cancel + clear + sweep) → cancel_all_active
- clear_finished_downloads (slskd clear + sweep) → clear_finished_active
- clear_completed_downloads (local task tracker prune) → clear_completed_local
Slskd-touching helpers take (soulseek_client, run_async, sweep_callback)
explicitly so the route layer wires the live client + the existing
_sweep_empty_download_directories helper. The local-state helper imports
download_tasks/download_batches/batch_locks/tasks_lock straight from
core.runtime_state since those are module-level shared globals.
Prep change: `batch_locks` dict moved from web_server.py global into
core/runtime_state.py alongside the other download globals. web_server.py
re-imports from runtime_state so the ~3 existing call sites in
web_server.py keep resolving without modification. Identity preserved
(same dict across all importers).
Out of scope (deferred to PR4g batch lifecycle):
- cancel_download_task (calls _on_download_completed)
- cancel_task_v2 + _atomic_cancel_task + _find_task_by_playlist_track
(manipulate batch active_count directly, deeply coupled to lifecycle)
Behavior parity:
- Same response shapes + status codes on each route
- Same call order (cancel_all → clear_all_completed → sweep)
- Same conditional sweep on clear_finished (skipped on failure)
- Same sweep ALWAYS runs after cancel_all even if clear_all returns False
(matches original — clear failure was non-fatal in cancel_all path)
- Same TERMINAL_STATUSES set: completed/failed/not_found/cancelled/skipped/
already_owned (lifted to module-level constant)
- Same empty-batch pruning + same batch_locks cleanup
- Same lock acquisition pattern (single tasks_lock)
Tests: 14 new under tests/downloads/test_downloads_cancel.py covering
single cancel, cancel-all happy + failure paths, clear-finished + sweep
gate, local task pruning across all 7 active/terminal states, batch
queue trimming, batch_locks cleanup.
Full suite: 921 passing (was 907). Ruff clean.
First sub-PR in the download orchestrator series. Strict 1:1 lift —
zero behavior change.
What moved:
- _record_sync_history_start → record_sync_history_start
- _record_sync_history_completion → record_sync_history_completion
- _detect_sync_source → detect_sync_source
- Source prefix map → module-level _SOURCE_PREFIX_MAP constant
What stayed:
- web_server.py keeps three thin wrappers (_detect_sync_source,
_record_sync_history_start, _record_sync_history_completion) that
delegate into core/downloads/history.py. ~60 callers of these names
in web_server.py keep resolving without touching every site.
Each lifted function takes `database` as an arg (was
`db = MusicDatabase()` inline). The wrappers construct
`MusicDatabase()` per call to mirror the exact original behavior —
each invocation got a fresh DB connection.
Behavior parity:
- Same SQL UPDATE statement (preserves the in-place update path when
a sync_history entry already exists for the playlist_id)
- Same JSON serialization with ensure_ascii=False
- Same thumb URL extraction order (album_context.images → image_url
→ first track album.images)
- Same per-track result shape (index, name, artist, album, image_url,
duration_ms, source_track_id, status, confidence, matched_track,
download_status)
- Same status mapping (found/not_found, completed/failed)
- Same best-effort exception swallowing (sync history failure must
never break the actual download)
- Reads `download_tasks` from core.runtime_state (already lifted by
kettui in PR378)
Tests: 34 new under tests/downloads/test_downloads_history.py
covering source detection (16 prefixes), start happy paths + thumb
extraction + duplicate-update + DB error swallowing, completion stats
+ per-track results JSON shape + edge cases.
Full suite: 907 passing (was 873). Ruff clean.
