Add fallback support for public hifi-api instances that expose playback through /track/ instead of /trackManifests/. The capability checker now accepts either manifest shape, and downloads can use direct URLs decoded from the legacy base64 manifest.
Tests cover legacy instance capability detection and download-manifest fallback while preserving the newer trackManifests path.
Probe public HiFi instances with the same trackManifests endpoint used by real downloads instead of the legacy /track endpoint. This prevents compatible instances from being falsely labeled search-only in Settings.
Centralize HiFi instance capability checks in HiFiClient and reuse manifest URI parsing with the download path.
Tests cover manifest-based capability detection, no legacy /track probe, and limited instances without a manifest URI.
Adds the PR description for the torrent/usenet release-staging feature and updates drifted tests for the new plugin registry entries and search exclude_sources signature.
Verified with the focused pytest command covering cancellation and default registry source registration.
Adds torrent/usenet as release-oriented download sources with album-bundle staging, live progress reporting, and post-processing that selects the requested audio file from completed releases instead of blindly importing the first file.
Keeps album-bundle behavior gated to single-source torrent/usenet album downloads, excludes release sources from hybrid album per-track searches, and allows hybrid non-album tracks to use release results safely.
Improves staged-release matching for featured/bonus track filenames while preserving version mismatches, records torrent/usenet provenance in library history, and updates service/status UI labels.
Covers the flow with focused lifecycle, status, staging, validation, task worker, post-processing, and import side-effect tests.
Route torrent and Usenet album bundles through private per-batch staging so Auto-Import cannot race public staging or duplicate imports.
Expose album-bundle progress in batch status and render it on the Downloads page while the external client is still downloading.
Tighten release handoff safety by rejecting archive path traversal, ignoring torrent candidates without a usable URL, and skipping Soulseek source reuse for torrent/Usenet batches.
Tests: .venv/bin/python -m pytest tests/downloads/test_downloads_status.py tests/test_album_bundle_dispatch.py tests/downloads/test_downloads_staging.py tests/test_torrent_usenet_plugins.py
Detect JSON decode-like exceptions from Tidal's token endpoint and return a safer, more actionable error message. Adds a _looks_like_json_decode_error helper and special-cases that error in check_device_auth to log the non-JSON response and advise disabling VPN/proxy/network filtering and restarting SoulSync. A test was added to ensure the user-facing message does not leak the raw exception text while still returning an error status. Other errors continue to fall back to the existing behavior.
Add duration tolerance logic and pre-download rejection for structured sources (tidal, qobuz, hifi, deezer_dl, amazon) when candidate duration deviates beyond allowed tolerance. Introduces helper functions _duration_tolerance_seconds and _duration_mismatch_exceeds_integrity_tolerance and uses resolve_duration_tolerance from core.imports.file_integrity. Log and skip candidates that would fail post-processing integrity checks to avoid wasted downloads. Update tests to include matching engine stub and new cases covering rejection and acceptance based on duration tolerance; also adjust imports and test fixtures.
When a file failed AcoustID verification and got quarantined, the next
auto-wishlist cycle would search for the same track, the deterministic
quality picker would re-select the same (uploader, filename) source,
re-download it, and re-quarantine it. Users woke up to hundreds of
duplicate .quarantined entries from a single bad upload — same source
URL repeatedly, byte-for-byte identical files.
Root cause: `SoulseekClient.filter_results_by_quality_preference` ranks
candidates by quality + bitrate density only. Quarantine history wasn't
consulted, so a high-bitrate FLAC upload with a wrong-track AcoustID
fingerprint kept winning the picker against every other candidate.
Fix shape:
- New helper `core/imports/quarantine.py::get_quarantined_source_keys`
reads every quarantine sidecar's `context.original_search_result`
and returns the set of `(username, filename)` tuples for O(1)
membership checks. Sidecars missing the context field (legacy thin
sidecars written pre-Feb 2026, or orphaned files) and corrupt JSON
are skipped silently — defensive against transient FS / encoding
issues.
- `SoulseekClient._drop_quarantined_sources` runs the membership
filter against incoming TrackResults, drops matches, logs a single
INFO line with the skip count. Called first inside
`filter_results_by_quality_preference` so all four callers
(search-and-download, master worker, validation, orchestrator)
benefit transparently.
- Approving or deleting a quarantine entry removes its sidecar, so
the dedup key disappears from the set on the next search — gives
the user a way to opt back in to a previously-quarantined source
without restarting the app.
7 helper tests cover: missing dir, empty dir, well-formed sidecars
collected as tuples, legacy sidecars skipped, empty source fields
skipped (so empty-string keys can't accidentally drop unrelated
results), corrupt JSON tolerated, duplicate quarantines collapse.
5 integration tests pin: clean candidates pass, known-bad candidates
drop, missing quarantine dir returns input unchanged, filesystem
errors swallowed (defensive), full `filter_results_by_quality_preference`
runs the dedup BEFORE the quality picker — so a high-quality
quarantined source can't win on bitrate.
692 existing download + import tests still green. Cosmetic surface
of the fix is invisible — same UX as today when no quarantine entries
exist; loop only kicks in once a sidecar has been written.
Out of scope: bulk-select / multi-delete UI for the quarantine tab —
S-Bryce mentioned this as a separate pain point in the issue, but
it's its own UX work, not a one-commit drive-by.
When slskd_url is configured but the host is unreachable (slskd not
running, wrong port, host.docker.internal not resolving), the frontend's
/api/downloads/status polling fanned out to every download plugin
including Soulseek. soulseek_client._make_request hit a DNS / connect
failure on each poll and logged it at ERROR. Result: one
"Cannot connect to host host.docker.internal:5030" log line every
~2-3 seconds for the entire duration of any download — visible spam
even when the user wasn't using Soulseek at all.
Caught aiohttp.ClientConnectorError explicitly in both _make_request
and _make_direct_request. First failure emits one WARNING with
actionable context (start slskd, or clear soulseek.slskd_url if you
don't use Soulseek). Subsequent failures demote to DEBUG. The
_last_unreachable_logged flag resets on any successful (200/201/204)
response so a later outage warns again — suppression is per-outage,
not per-process-lifetime. Same shape as the existing _last_401_logged
suppression for auth failures.
The architectural gap (status polling fans out to soulseek even when
the user has soulseek disabled in their active download sources) is
intentionally left for a follow-up. The plugin-iteration code lives
in core/download_engine/engine.py and core/download_orchestrator.py;
threading a "skip-when-not-active" gate through every caller is a
bigger refactor than this user-facing log cleanup warrants. The
WARNING-once message tells the user what to do in the meantime.
5 new pinning tests cover the suppression contract: connection error
returns None (not raises), first failure WARNs + sets flag, repeats
stay quiet, successful response resets the flag, _make_direct_request
follows the same pattern, and non-connection exceptions still log at
ERROR so real bugs aren't hidden behind the new suppression.
Manual matches can be created from sync history as mirrored while wishlist and download flows later see the same track as wishlist or a provider source. Add a shared track-level lookup that falls back from exact source/id to source_track_id and title/artist, then use it for wishlist adds, cleanup, and download analysis so mapped tracks are not re-added or redownloaded.
Add coverage for mirrored-source matches being honored by wishlist cleanup and download batches, including the internal wishlist force-download path.
Add a conservative Soulseek album preflight scorer so album downloads choose a coherent slskd folder before per-track enqueue. The scorer compares album title, artist, year, track count, tracklist coverage, peer quality, and penalizes unexpected deluxe/remix/live-style folders.
Preserve hybrid source priority by only running Soulseek album preflight when Soulseek is the selected source or first in the hybrid order. If Soulseek is only a fallback behind another source, the normal hybrid flow is left alone.
Reuse the richest wishlist album context across tracks in the same album group so release date, artwork, album type, and album artist stay consistent for path generation. Also preserve peer-quality tie breakers when attempting equal-confidence candidates.
Tests cover correct-folder selection over larger wrong editions, Soulseek primary vs fallback hybrid behavior, shared wishlist album context, and peer-quality candidate ordering.
GitHub issue #499 (@bafoed). Big initial sync of Spotify playlists
worked for 2-3 hours then downloads silently stopped:
- 3 active tasks stuck in "Searching" state, replaced every ~10 min
by different ones
- slskd UI showed no actual searches happening
- Debug log: orphaned-task count grew over time, no jobs executed
- Container restart was the only fix (bought another 2-3 hours)
- Not a rate limit (rates showed 0/min)
Root cause: ``core/soulseek_client.py`` constructed
``aiohttp.ClientSession()`` with no timeout at four sites. When slskd
hung on a request (overloaded, transient network blip, internal
stall), the HTTP call blocked indefinitely — and the worker thread
blocked with it. The download executor only has
``ThreadPoolExecutor(max_workers=3)``, so once 3 worker threads were
wedged on hung calls, no further downloads could start.
Batch-level "stuck detection" (10-minute timer in
``check_batch_completion_v2``) was correctly marking tasks
``not_found`` and trying to start replacements, but the executor pool
was exhausted — replacements queued forever inside the executor with
no thread to run them. Symptom: tasks rotating every ~10 min at the
batch level while the underlying executor stayed wedged.
Fix: bounded ``aiohttp.ClientTimeout`` (total 120s, connect 15s,
sock_read 60s) on every slskd ``ClientSession`` construction. Module-
level constant ``_SLSKD_DEFAULT_TIMEOUT`` so the four sites stay in
lockstep — future sites get the same protection by reusing the
constant.
Why these timeouts are safe:
- Every slskd API call is metadata-level (search submission, status
polls, download enqueue, transfer state queries). None stream
files — slskd handles file transfer via its own peer-to-peer
infrastructure entirely outside our HTTP requests.
- Legitimate metadata calls finish in seconds. 120s ceiling is
~50× the normal latency.
Timeout handling:
- ``asyncio.TimeoutError`` caught explicitly BEFORE the generic
``except Exception`` — surfaces "slskd timed out" specifically in
logs (debuggable instead of buried as "Error making API request").
- Returns None to the caller (same code path as a 5xx response or
any other failure). No new error path; callers already handle
None as "request failed".
- Worker thread unblocks immediately → executor pool stays healthy
→ downloads keep flowing.
Sites updated:
- ``_make_request`` (general /api/v0/ helper, line 152) — used for
every slskd API operation
- ``_make_direct_request`` (non-/api/v0/ helper, line 235)
- ``_explore_api_endpoints`` Swagger fetch (line 1566) — diagnostic
- ``_explore_api_endpoints`` per-endpoint probe (line 1617) —
diagnostic
Tests: 3 new tests in ``tests/downloads/test_soulseek_pinning.py``
pin:
- ``_SLSKD_DEFAULT_TIMEOUT`` is bounded (total set, ≤300s ceiling,
connect ≤60s) — guards against future regressions that drop or
unbound the timeout
- ``_make_request`` returns None on ``asyncio.TimeoutError`` rather
than raising — pins the caller contract
- ``_make_direct_request`` returns None on ``asyncio.TimeoutError``
2185/2185 full suite green.
Closes#499.
Pinning gaps he flagged after his second pass:
- register_plugin when set_engine() raises: registration must succeed
+ plugin stays in the registry (download() raises later, surfacing
the error to the user via download_with_fallback). Pin so a future
refactor can't accidentally propagate the set_engine exception and
crash boot.
- engine.get_all_downloads exclude actually doesn't invoke the plugin:
ID-only check would pass even if soulseek's get_all was called and
returned []; sentinel proves the plugin isn't touched at all.
- Cancel mid-flight observable from inside _download_sync: existing
tests pin Cancelled-preserve AFTER impl returns, this pins the
contract plugins rely on (engine.get_record reflecting Cancelled
state during the impl thread's polling loop).
- configured_clients() with broken is_configured(): the try/except
guard exists but had no test — broken plugin is silently skipped,
healthy ones still surface.
- Per-source delay independence: YouTube's 3s rate-limit delay must
not block a Tidal download starting in parallel. Companion to the
per-source-locks test.
Two findings from JohnBaumb on the engine refactor.
(1) Every download client returned None when self._engine was None,
just logging an error. The orchestrator's download_with_fallback
treated None as "source declined", so the user got no feedback —
download silently disappeared. Now each client raises a RuntimeError
on the engine-not-wired path. download_with_fallback already catches
plugin exceptions, logs a warning, and tries the next source — so
the visible behavior is "real error in logs + fallback to next
source" instead of "silent drop". Six clients touched (deezer, hifi,
qobuz, soundcloud, tidal, youtube). Pinning tests updated to expect
raise.
(2) Monitor's engine.get_all_downloads() walked every plugin
including soulseek, but the same monitor loop already pulled slskd
transfers via the transfers/downloads endpoint a few lines earlier —
soulseek's records were being fetched twice per tick. Same issue in
web_server.py's get_cached_transfer_data path. Added an exclude
parameter to engine.get_all_downloads(); both call sites now pass
('soulseek',). New test pins the exclude semantic.
Also fixed a stray 8-space over-indent on the for-loop body in
get_cached_transfer_data (cosmetic, JohnBaumb flagged the same
pattern in monitor.py earlier).
Per JohnBaumb: the single state_lock serialized progress callbacks
across every source. Pre-refactor each client owned its own download
lock, so Deezer / YouTube / Tidal workers never blocked each other.
Multi-source concurrent downloads under the unified lock fought for
the same RLock on every progress update.
Replaced the engine-wide state_lock with per-source RLocks. Each
source gets its own lock, lazily created via _source_lock() on first
use (meta-lock guards the create-race). All record mutations
(add/update/update_unless_state/remove/get/iter) take only that
source's lock — Deezer progress updates no longer block Tidal writes.
Cancelled-preserve semantics still hold because cancel + worker
terminal write target the same source, so they share that source's
lock. New test pins lock independence: holding source-A's lock from
one thread does not block a write on source-B from another.
Per JohnBaumb's review: iter_records_for_source() walked every
(source, id) tuple across the entire engine state to filter one
source — O(total_records) instead of O(source_records). Fine in
practice because total active downloads is usually small, but the
shape was wrong.
Switched the engine's _records storage from a single composite-key
dict (Dict[Tuple[str, str], DownloadRecord]) to a nested dict
(Dict[str, Dict[str, DownloadRecord]]). Per-source iteration now
only touches that source's bucket. add/get/update/remove all
adjusted to the nested layout. remove_record drops the empty source
bucket so future iterations don't see stale source keys.
Public surface unchanged. New test pins the empty-bucket-cleanup
behavior.
The earlier validation-only filter only ran in the auto-search
scoring path. SoundCloud preview snippets still leaked through:
- The candidate-review modal cached raw search results (pre-validation),
so previews were visible and clickable for manual retry — and the
manual-pick download path bypassed validation entirely, downloading
the preview anyway.
- The not-found raw-results cache stored unfiltered top-20s.
Lift the preview filter into a reusable filter_soundcloud_previews()
helper and apply it at every entry point: validation scoring (still),
modal-cache fallback when validation drops everything, and the
not-found raw-results path. Previews now never reach the cache, the
matcher, or the manual-pick UI. Drops candidates < 35s or below half
the expected duration, gated on expected > 60s so genuine short
tracks still pass. 7 new unit tests pin the helper.
Also fixed a silent regression in core/downloads/task_worker.py's
hybrid-fallback path. Cin-5 dropped the per-source attrs from the
orchestrator (orch.soulseek, orch.youtube, etc.), but the fallback
loop still resolved sources via getattr(orch, '<src>', None) — every
lookup silently returned None, so remaining_sources came back empty
and the fallback never ran. Now uses orch.client(name) like the rest
of the codebase. Updated the test fake to expose client() too — the
old test was passing because the loop was effectively dead.
The global handle in web_server.py was named soulseek_client for
historical reasons but the type has long been DownloadOrchestrator,
not SoulseekClient. Renamed the global plus every parameter/attribute
that carried the legacy name.
- web_server.py: global var renamed; all 99 references updated.
- api/, core/downloads/*, core/search/*, core/streaming/*,
services/sync_service.py: parameter names, dataclass fields, and
init() arg names renamed.
- Test fixtures (CandidatesDeps, MasterDeps, SearchDeps, etc.) and
the _build_deps helpers updated accordingly.
The core.soulseek_client module path and SoulseekClient class name
(the actual soulseek-only client) are unchanged — only the orchestrator
handle renamed. Module imports of TrackResult/AlbumResult/DownloadStatus
from core.soulseek_client preserved.
Removed the eight backward-compat attribute aliases on the orchestrator
(soulseek, youtube, tidal, qobuz, hifi, deezer_dl, lidarr, soundcloud).
External callers and the orchestrator's own internals now reach clients
through the generic alias-aware client(name) accessor.
- core/downloads/{master,monitor,validation}.py: migrated to client().
Monitor's per-source aggregation loop replaced with a single
engine.get_all_downloads() call.
- core/search/{orchestrator,stream}.py: migrated; stream.py drops the
hand-built mode-to-client dict.
- web_server.py: migrated /api/deezer/arl-* + tidal client lookup.
- core/download_orchestrator.py: internal self.soulseek /
self.deezer_dl reaches now route through self.client(); attr
assignments dropped from __init__; module docstring updated.
- Test fakes (_FakeSoulseek, _FakeSoulseekWithYT) expose client(name)
instead of stuffing per-source attributes.
- Conformance test re-pinned to the client() accessor contract.
Three correctness fixes from kettui's PR review plus the web_server
migration to generic accessors.
- Engine alias map: register_plugin accepts aliases tuple; get_plugin
+ cancel_download resolve through it. Fixes deezer_dl cancels
silently routing to soulseek.
- Orchestrator hybrid_order normalization: _resolve_source_chain
routes raw config names through registry.get_spec() so legacy
deezer_dl entries don't drop deezer from hybrid mode.
- Atomic update_record_unless_state on the engine: holds state_lock
across the check + write. Both _mark_terminal AND the success path
use it now so a Cancelled state set mid-impl can't be clobbered.
- web_server.py: 30 soulseek_client.<source> reaches migrated to
client("<source>"); shutdown-check setup migrated to generic
registry iteration; 4 hifi reload sites use reload_instances('hifi').
- 18 new tests pin every fix.
Cin's review feedback: external callers reach per-source clients
via attribute access (orch.hifi.reload_instances()) — needs
generic accessors so the registry IS the single source of truth.
Adds:
- orch.client(name) — public accessor for a per-source client.
Resolves canonical names (deezer) AND legacy aliases (deezer_dl).
- orch.configured_clients() — returns {name: client} for every
initialized AND is_configured() == True source. Replaces the
6+ if/hasattr/is_configured chain Cin called out:
if hasattr(orch, 'soulseek') and orch.soulseek and \
orch.soulseek.is_configured(): ...
- orch.reload_instances(source=None) — generic dispatch for
source-specific reload calls. Replaces orch.hifi.reload_instances()
with orch.reload_instances('hifi').
- get_download_orchestrator() / set_download_orchestrator()
singleton factory matching Cin's get_metadata_engine pattern in
PR #498. web_server.py can install the orchestrator it builds
at boot so future callers grab via the factory instead of
importing the legacy `soulseek_client` global.
Phase Cin-3/Cin-4 will replace existing call sites; this commit
just provides the surface so those migrations are mechanical.
Suite still green (335 download tests + 6 new generic-accessor
tests).
Three findings from a final review pass:
1. **Worker clobbered Cancelled with Errored when impl returned
None / raised mid-cancel.** The legacy per-client thread workers
each had a guard (``if state != 'Cancelled': state = 'Errored'``);
the shared worker dropped it. Fix: new ``_mark_terminal`` helper
in BackgroundDownloadWorker reads current state before writing
the terminal one and leaves Cancelled alone. SoundCloud test
updated back to the strict Cancelled-only assertion (had been
loosened to accept Errored as a workaround). Two new pinning
tests catch the regression.
2. **Dead code in engine.py.** ``find_record`` and
``iter_all_records`` had no production callers — only tests.
Removed them. Concurrent-add stress test rewritten to use the
per-source iterator that's actually in use.
3. **Silent ``except Exception: pass`` in cross-source query
methods.** Faithful to legacy behavior (one source failing
shouldn't take down aggregation) but Cin's standard is "log
even when you swallow." Each silent-swallow site now logs at
debug level so the source name + exception are inspectable
without adding warning-level noise.
Suite still green (2049 passed).
`engine.search_with_fallback(query, source_chain, ...)` walks the
chain in order, skips unconfigured / unregistered plugins,
swallows per-source exceptions, and returns the first non-empty
(tracks, albums) tuple. Replaces orchestrator's hand-rolled
hybrid search loop.
`engine.download_with_fallback(username, filename, file_size,
source_chain)` falls through the chain when a source returns
None / raises. Username hint promotes a matching source-chain
entry to head of order. NOT yet wired into orchestrator.download
— today's username comes from a search result and represents
the user's explicit source pick, so silently falling through
would override their choice. Engine method is available for
future callers that want fallback semantics
(search_and_download_best, automation).
Orchestrator gains _resolve_source_chain helper that builds
the ordered list (hybrid_order config, falling back to legacy
primary/secondary pair). Orchestrator.search hands chain off
to engine.search_with_fallback for hybrid mode.
8 new tests pin the fallback semantics: chain ordering,
unconfigured-skip, exception-continue, empty-when-exhausted,
username-hint promotion. Suite still green (2050 passed).
YouTubeClient gains rate_limit_policy() that returns a
RateLimitPolicy with the configured download_delay (3s default
from `youtube.download_delay`). Engine reads this at
register_plugin time + applies to engine.worker.
set_engine still re-applies the delay so runtime reload_settings
updates flow through the same pathway. Other sources keep the
default policy (concurrency=1, delay=0) which matches their
current behavior — no migration needed beyond YouTube which is
the only source with a non-default download throttle today.
New pinning test asserts the policy shape (delay=3.0, concurrency=1).
Suite still green (2042 passed).
`core/download_engine/rate_limit.py` introduces a per-source
policy declaration: download_concurrency + download_delay_seconds.
Plugins declare via `RATE_LIMIT_POLICY` class attribute or a
`rate_limit_policy()` method.
Engine applies the declared policy to engine.worker at
register_plugin time — set_concurrency + set_delay get pushed
in automatically. Plugins without a declaration get the
conservative default (1 / 0). The set_engine callback fires
AFTER policy registration so config-driven sources (YouTube
reads user-tunable youtube.download_delay) can override.
Plan doc updated to reflect Phase D skip (search code is 90%
source-specific, not 60% — lifting it would be lossy or
bloated).
Pure additive — no plugin migrated yet. 8 tests pin the
resolution priority + engine wire-up + override semantics.
Suite still green (327 download tests).
Last C-phase migration. Same pattern as C2-C6 — SoundCloud drops
active_downloads + _download_lock + _download_thread_worker.
download() delegates to engine.worker.dispatch with permalink_url
captured in a closure so the impl gets the URL (not the track_id)
yt-dlp needs.
Both progress hooks (HLS-fragmented + byte-based) write to engine
state via update_record. Query/cancel methods read engine state.
Existing test_soundcloud_client.py mass-updated: 16 tests that
reached into client.active_downloads / _download_lock now use
engine.add_record / get_record / update_record via a small
_wire_engine helper. test_download_thread_does_not_clobber_cancelled_state
now accepts either Cancelled or Errored as the final state since
the engine.worker doesn't preserve Cancelled-over-Errored the
way the legacy per-client thread did (potential follow-up: add
that guard uniformly in BackgroundDownloadWorker).
Phase A pinning tests updated. Suite still green (2033 passed).
Same migration pattern as C2-C5. Deezer-specific quirks
preserved through worker overrides:
- username_override='deezer_dl' (legacy slot frontend reads)
- thread_name='deezer-dl-<track_id>' (diagnostic naming)
- track_id stays as STRING (Deezer GW API uses string IDs)
- Extra 'error' slot in record for ARL re-auth failure messages
Mid-download chunk loop's many state mutations (cancellation
checks, progress updates, error capture across multiple failure
modes) all flow through engine.update_record / get_record now.
Added _set_error and _is_cancelled helpers to keep call sites
readable.
Pinning tests updated. Suite still green (319 download tests).
Same pattern as C2/C3/C4. HiFi worker was named _download_worker
(not _thread_worker like the others) — gone now along with the
state dict + lock. Mid-download HLS-segment progress hook
(_update_download_progress) writes to engine state.
Pinning tests updated. Suite still green (318 download tests).
Same pattern as C2 — TidalDownloadClient drops active_downloads
+ _download_lock + _download_thread_worker. download() delegates
to engine.worker.dispatch with _download_sync as the impl.
Source-specific extras (track_id, display_name) merge into the
engine record.
The HLS-segment progress callback (_update_download_progress)
now writes to engine state via engine.update_record instead of
mutating the per-client dict in-place.
Query/cancel methods (get_all_downloads, get_download_status,
cancel_download, clear_all_completed_downloads) now read engine
state via the same accessors as the YouTube migration.
Pinning tests updated to assert engine state. Suite still green
(313 download tests). Behavior preserved end-to-end.
YouTubeClient drops its hand-rolled background thread + state
dict + semaphore + last-download-timestamp. download() now
delegates to engine.worker.dispatch with _download_sync as the
impl callable; YouTube-specific record fields (video_id, url,
title) merge into the engine record via extra_record_fields.
Engine wires itself in via plugin.set_engine(engine) callback
on register_plugin. YouTube uses set_engine to register its
3-second download_delay with worker.set_delay so the rate-limit
gap between successive downloads stays the same.
Query/cancel methods (get_all_downloads, get_download_status,
cancel_download, clear_all_completed_downloads) now read engine
state via engine.iter_records_for_source / get_record /
update_record / remove_record. Net: ~120 LOC of thread+state
boilerplate removed from youtube_client.py.
Phase A pinning tests updated to assert engine state instead of
client.active_downloads — same observable contract (filename
encoding, UUID, record schema with video_id/url/title), new
storage location.
Suite still green (2025 passed). Behavior preserved end-to-end:
YouTube downloads kick off the same way, lifecycle states match,
cancel + clear-completed semantics unchanged.
`BackgroundDownloadWorker` lives on the engine and owns the
boilerplate every streaming download client currently
hand-rolls: thread spawn, per-source semaphore, rate-limit
delay, state lifecycle (Initializing → InProgress → Completed
or Errored), exception capture.
Plugins provide only the atomic download op (`impl_callable`).
Per-source rate-limit policy (concurrency, delay) is configured
on the worker via `set_concurrency` / `set_delay`. Source-
specific record fields merge in via `extra_record_fields` so
existing consumer code that reads `video_id`, `track_id`,
`permalink_url`, etc. keeps working post-migration. Username
slot supports override (Deezer's legacy `'deezer_dl'`).
Phase C1 scope: worker exists. No client migrated yet — C2-C7
migrate sources one at a time, each gated by the Phase A
pinning tests so per-source contract drift fails fast.
10 new tests pin the worker contract: UUID id format, initial
record shape, extra-fields merge, username override, state
transitions on success / impl-returns-None / impl-raises,
semaphore serialization (default + parallel), rate-limit
delay between successive downloads.
Suite still green (308 download tests). Pure additive.
`get_all_downloads`, `get_download_status`, `cancel_download`, and
`clear_all_completed_downloads` on the orchestrator are now thin
pass-throughs to the engine. The plugin-iteration logic lives in
one place (the engine) instead of duplicated across orchestrator
methods.
Source-hint routing semantics preserved verbatim — engine.cancel
treats streaming-source names as direct routes and unknown names
as Soulseek peer usernames, exactly like the legacy orchestrator
did. Per-plugin exceptions still get swallowed defensively.
Test fixture `_build_orchestrator` now constructs an engine and
registers every mock plugin so the helper-built orchestrators
have the same wiring as production.
Suite still green (2012 passed). Zero behavior change for users.
`DownloadEngine` grows async query methods that wrap plugin
iteration: `get_all_downloads` (concatenates every plugin's
active downloads), `get_download_status` (first plugin to
recognize the id wins), `cancel_download` (with source-hint
routing — streaming sources go direct, unknown hints route to
Soulseek as peer username), and `clear_all_completed_downloads`
(skips unconfigured plugins).
Code moved from the orchestrator's hand-iterated loops into the
engine. Orchestrator delegation comes in B3 — for B2 the engine
methods exist but nothing calls them yet.
Per-plugin behavior preserved verbatim (defensive `try ... except`
swallows per-iteration, unconfigured-skip on clear, source-hint
routing semantics). Phase A pinning tests + 8 new engine query
tests catch any drift.
Pure additive — zero behavior change for users.
`core/download_engine/` package with the engine class that will own
cross-source state, threading, search retry, rate-limits, and
fallback chains. Orchestrator constructs an engine and registers
each plugin with it.
Phase B1 scope: skeleton only. Engine stores active_downloads
records keyed by (source, download_id), provides thread-safe
add/update/remove/iterate primitives, and holds plugin references
for later phases. NOT on any code path yet — pure additive
scaffolding so subsequent commits can introduce engine-driven
behavior one piece at a time without a big-bang switchover.
15 new tests pin the engine's state-storage contract: shallow-copy
reads, partial-patch updates, no-op-on-missing semantics,
per-source iteration, id-only find, concurrent-add safety.
Suite still 290 (download subset) green. Zero behavior change.
6 tests pin the Lidarr contract — the special case in the
dispatcher because Lidarr is an ALBUM-grabber not a track-grabber.
Filename format is `album_foreign_id||display` (MusicBrainz album
MBID Lidarr uses for lookups). State dict is SMALLER than streaming
sources (no track_id, no transferred/speed — Lidarr polls its own
queue API for byte-level progress). Thread target signature is
3-arg, no original_filename. Engine refactor's plugin contract
must accommodate album-only sources or Lidarr stays special.
6 tests pin the SoundCloud contract: 3-part filename
`track_id||permalink_url||display_name` (yt-dlp consumes the URL,
not the track_id). Defensive: 2-part filename falls back display
name to track_id; missing url or empty fields return None.
Thread target signature uses URL as the second arg.
6 tests pin the Deezer contract:
- track_id stays as STRING (Deezer GW API uses string IDs).
- username slot is the legacy `'deezer_dl'` (frontend depends on it).
- Auth gate at top of `download()` returns None BEFORE thread spawn.
- Defensive fallback: filename without `||` synthesizes display name.
- Thread is named `deezer-dl-<track_id>` for diagnostics.
- State dict has Deezer-specific `error` slot.
5 tests pin the HiFi contract: int track_id, UUID download_id,
state-dict schema, daemon-thread worker. Note: target method is
`_download_worker` (NOT `_thread_worker` like Tidal/Qobuz) and
worker signature is 3-arg (download_id, track_id, display_name).
Engine refactor's plugin contract must accommodate or normalize.
8 tests pin the Tidal contract: filename encoding (`<int>||display`
where track_id parses as int), UUID download_id format, initial
state-dict schema, daemon-thread spawn semantics, and the
active_downloads → DownloadStatus translation. is_authenticated
false on no-session AND on tidalapi.check_login() exceptions
(orchestrator skip behavior depends on this).
5 tests pin the YouTube download contract: filename encoding
(`video_id||title`), UUID download_id format, initial state-dict
schema, daemon-thread spawn for background work, and the
`_download_thread_worker` target shape. Phase C will replace
the thread spawn with `engine.dispatch_download` — these tests
catch any drift in the per-download record shape that consumers
depend on.
Pure additive — no client code changes.
13 tests pin slskd HTTP API contract: endpoint format
(`transfers/downloads/<username>` POST), payload shape
(slskd web-interface array format), id extraction from dict /
list / fallback responses, and the username-lookup fallback in
cancel_download when no username hint is provided.
Phase A of the download engine refactor — pinning current
behavior of every source BEFORE moving any code so the engine
extraction can't drift the per-source contract. Includes the
plan doc at docs/download-engine-refactor-plan.md.
Pure additive — no client code changes.
Every per-source dispatch site (search, download, get_all_downloads,
get_download_status, cancel_download, clear_all_completed_downloads,
cancel_all_downloads, reload_settings) now iterates
`registry.all_plugins()` instead of hand-maintained client lists.
Backward-compat `self.soulseek` / `self.youtube` / etc. attributes
preserved as registry-resolved aliases — external callers reaching
for source-specific internals (e.g. `orchestrator.soulseek._make_request`)
keep working unchanged.
Adding a new source (Usenet planned) becomes one registry entry +
the new client class — no orchestrator changes.
Plug the previously-built SoundcloudClient (PR #478, the build-and-verify
phase) into every place a download source needs to appear. Follows the
same wiring contract as Tidal/Qobuz/HiFi/Deezer/Lidarr — orchestrator
routing, hybrid-mode picker, search dispatch, queue/cancel/clear,
provenance + library history, sidebar source label, settings UI all
work plug-and-play.
Backend wiring:
- `core/download_orchestrator.py` — import SoundcloudClient, _safe_init
it at startup, add to _client() lookup, get_source_status(),
check_connection's sources_to_check default, search source_names map,
search_and_download_best _streaming_sources tuple, download
source_map + source_names, and every iteration loop in
reload_settings download-path-update / get_all_downloads /
get_download_status / cancel_download (route + iterate) /
clear_all_completed_downloads / cancel_all_downloads.
- `core/downloads/monitor.py` — added SoundCloud to the per-client
loop that fetches active downloads outside the orchestrator (uses
getattr fallback for older soulseek_client snapshots).
- `core/downloads/task_worker.py` — added SoundCloud (and Lidarr,
which was missing too — bonus fix) to source_clients dict for hybrid
fallback dispatch.
- `core/downloads/validation.py` — added 'soundcloud' to
_streaming_sources so SoundCloud results go through the matching
engine validation path instead of the Soulseek quality-filter path.
- `core/imports/side_effects.py` — three call sites: source_map for
download_source label written to library_history, streaming-source
guard for the `||`-encoded stream_id parsing, and source_service
map for provenance recording. All three now include 'soundcloud'.
- `web_server.py` — five streaming-source detection tuples updated.
New `/api/soundcloud/status` endpoint returns
{available, configured, reachable} mirroring the Deezer/HiFi
status-endpoint pattern; reachability runs a real cheap yt-dlp
search so the settings Test Connection button gives a meaningful
pass/fail signal.
- `config/settings.py` — added empty `soundcloud_download` defaults
block so future tier-2 OAuth (SoundCloud Go+ session) doesn't have
to migrate existing configs.
Frontend:
- `webui/index.html` — new `<option value="soundcloud">` in the
download-source-mode dropdown, SoundCloud added to both hidden
legacy hybrid-source selects, new settings container with info
text + Test Connection button.
- `webui/static/settings.js` — HYBRID_SOURCES entry (with the
SoundCloud cloud SVG icon), _hybridSourceEnabled default,
updateDownloadSourceUI container display, allSources for legacy
hybrid picker, testSoundcloudConnection function (hits the new
status endpoint, color-codes the result), saveSettings
soundcloud_download empty block.
- `webui/static/shared-helpers.js` — sidebar source-name map
includes SoundCloud + Lidarr (Lidarr was also missing, bonus fix).
- `webui/static/helper.js` — WHATS_NEW entry under '2.4.2' dev cycle
describing the user-visible change in the chill terse voice.
Tests:
- `tests/test_download_orchestrator_soundcloud.py` — 14 integration
tests verifying the wiring: client constructed at startup, _client
lookup resolves 'soundcloud', get_source_status includes it,
download dispatcher routes username='soundcloud' to the SoundCloud
client (and unknown usernames still fall back to Soulseek), hybrid
search iterates SoundCloud when in order and skips it cleanly when
unconfigured, get_all_downloads / get_download_status / cancel /
clear walk SoundCloud, soundcloud-only mode dispatches only to
SoundCloud, _streaming_sources tuple in validation includes
'soundcloud'.
- `tests/downloads/test_download_orchestrator.py` — added
`soundcloud` to the test fixture's _build_orchestrator helper so
the new orchestrator attribute doesn't AttributeError in pre-
existing tests that bypass __init__.
Verified:
- Full suite green (1728 passed, 2 deselected for soundcloud_live)
- Ruff clean
- Live SoundCloud-only mode search returns 25 SoundCloud tracks for
"kendrick lamar luther" in <2s, returning properly-shaped
TrackResult objects with username='soundcloud' and dispatch-key
filename ready for the download path.
Out of scope (intentional deferrals):
- SoundCloud Go+ OAuth tier (256 kbps AAC) — anonymous-only for now.
Adding auth later is a settings-page extension, no orchestrator
changes needed.
- Album/playlist support — SoundCloud has playlists but they don't
map to the album model the rest of SoulSync expects. Singles only.
- Switch the download lifecycle over to the neutral wishlist track helper name
- Keep the old Spotify helper as a compatibility alias for older callers
- Store track_data as the primary failed-download wishlist payload key and add regression coverage
Pulls the 201-line staging-folder shortcut out of `web_server.py` into
its own module under the existing `core/downloads/` package. Pure 1:1
lift — wrapper keeps the original entry-point name so the task worker's
existing call site continues to work without changes.
What `try_staging_match` does:
1. Pull the per-batch staging-file cache (one filesystem scan per batch).
2. For each staging entry, compute title + artist similarity using
SequenceMatcher and the matching engine's `normalize_string`. Require
title >= 0.80, then a combined score >= 0.75. The weighting flips
based on whether artist info is available on both sides:
- both have artist: 0.55*title + 0.45*artist
- either side missing artist: 0.80*title + 0.20*artist (lean on title)
3. Copy the matched file to the configured transfer dir (with a
"_staging" suffix when the destination filename already exists, to
avoid overwriting a legitimate prior download).
4. Mark the task as 'post_processing', username='staging',
staging_match=True.
5. Build a synthetic spotify_artist / spotify_album context (mirroring
the modal-worker logic so the file-organization template applies
cleanly) and store it under "staging_<task_id>". Two paths:
- Explicit context branch (track_info has _is_explicit_album_download)
→ real album/artist data copied through.
- Fallback branch → synthesized from track + track_info, with
`is_album_download` heuristically derived (album differs from title
and isn't "Unknown Album").
6. Hand off to `_post_process_matched_download_with_verification` which
does tagging, path building, AcoustID verification, and DB insertion.
Returns True if the staging shortcut won; False to fall through to the
normal Soulseek search path.
Dependencies injected via `StagingDeps` (5 fields) — config_manager,
matching_engine, get_staging_file_cache, docker_resolve_path,
post_process_matched_download_with_verification.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 201 lines orig = 201 lines lifted, byte-identical body
(including all whitespace, comments, log strings, and the inline
`from difflib import SequenceMatcher` / `import shutil` imports inside
the function body).
Tests: 9 new under tests/downloads/test_downloads_staging.py covering
no staging files / no track title / low-confidence match returning
False, exact match copying file + transitioning task state + invoking
post-processing, existing-file rename via `_staging` suffix, explicit
album context branch, fallback context synthesis (with both album-as-
album and album-equals-title cases), and copy failure (missing source
file) returning False.
Full suite: 1308 passing (was 1299). Ruff clean.
First lift in the new PR6 batch. Pulls the 312-line candidate-fallback
download dispatcher out of `web_server.py` into a new module under the
existing `core/downloads/` package. Pure 1:1 lift — wrapper keeps the
original entry-point name so all callers (search/match pipeline) work
unchanged.
What `attempt_download_with_candidates` does:
1. Sort candidates by descending confidence.
2. For each candidate:
- Cancellation gates (3 points: top of loop, before download starts,
after download_id is assigned).
- Skip already-tried sources via the per-task `used_sources` set.
- Skip blacklisted sources (user-flagged bad matches).
- Race protection: bail when the task already has an active
download_id.
- `update_task_status('downloading')`, then `soulseek_client.download`.
3. On a successful download_id:
- Build `matched_downloads_context` entry keyed by
`make_context_key(username, filename)`.
- For tracks with clean Spotify metadata, pull track_number /
disc_number from (1) track_info → (2) track object → (3) Spotify
API call. When local album context is incomplete, the API response
backfills release_date / album_type / total_tracks / images / id.
- Set `is_album_download` based on explicit context flag or
heuristic (album differs from title, isn't "Unknown Album").
- Store task/batch IDs and track_info on the context for post-
processing + playlist-folder mode.
4. On a cancellation that wins the race after the download started:
- `cancel_download(...)` to stop the in-flight Soulseek transfer.
- `on_download_completed(batch_id, task_id, success=False)` to free
the worker slot.
5. On exception or download-start failure: reset task status to
'searching', continue to next candidate.
Dependencies injected via `CandidatesDeps` (7 fields) — soulseek_client,
spotify_client, run_async, get_database, update_task_status,
make_context_key, on_download_completed.
Diff vs original after `deps.X` → global X normalization is **zero
differences** — 312 lines orig = 312 lines lifted, byte-identical body
(including all whitespace, comments, log strings).
Tests: 14 new under tests/downloads/test_downloads_candidates.py
covering happy path (first candidate succeeds, confidence ordering),
used_sources dedup, blacklist skip, cancellation gates (cancelled
status, deleted task, active download_id, mid-flight cancel + cleanup
callback), failure paths (all candidates failed, exception during
download falls through to next), context payload (explicit album
context, track_number priority order, API backfill of incomplete album
metadata), and equal-confidence stable order.
Pre-existing behavior documented in tests:
`spotify_album_context['id']` initializes to a non-empty placeholder
'from_sync_modal' in the fallback path, so the API-backfill condition
`if not spotify_album_context.get('id')` never fires for the id field
specifically. Other album fields (release_date, album_type) backfill
fine because they default to empty.
Full suite: 1290 passing (was 1276). Ruff clean.
