noldevin's first torrent was stuck "downloading metadata" — a dead magnet
with no peers. The poll loop would ride the full album deadline (6h default)
on it, holding the worker the whole time, with no built-in escape.
New stall handling, off the existing poll loop:
- core/download_plugins/torrent_stall.py — pure StallTracker (clock injected,
no I/O): forward byte progress resets a stall clock; once a torrent spends
the stall timeout in a working state (queued/downloading/stalled/error)
with zero progress, it's stalled. seeding/completed/paused never count.
Covers the metadata-stuck case (0 bytes, 0 progress) and a dead mid-download
swarm with one rule.
- _handle_stalled: 'abandon' (default) removes the torrent + its partial data
(a metadata stub is junk) and fails the download so the next source can try;
'pause' parks it in the client for the user. Adapter errors are swallowed —
the download still fails cleanly.
- two settings (download_source.torrent_stall_timeout_seconds = 600,
torrent_stall_action = 'abandon'); timeout 0 disables, restoring the old
ride-the-deadline behavior. Config-key driven, matching the existing
album_bundle_* tuning knobs (no UI form, same as those).
Tests: 18 on the tracker + settings (timeout trip, progress reset, idle-state
exemption, pause→resume clock restart, disable, parse tolerance) + 3 on the
plugin action path (abandon removes w/ delete_files, pause pauses, adapter
error survived). 158 torrent-family tests pass.
A torrent-first (or usenet-first) hybrid download would freeze at
"Torrent searching for release 0%" and never move to the next source when
Prowlarr returned no results for the album. Reported by Cezar:
[Album Bundle] torrent flow failed for '...': No torrent results found
Cause: the album-bundle dispatch only returns to the per-track flow (which,
in hybrid mode, tries the next configured source) when the plugin's failure
outcome carries fallback=True; otherwise it marks the batch failed and stops.
Both plugins set fallback=True on their 'results found but none matched the
album' branch, but the adjacent 'no results at all' branch set only an error
and no fallback flag -- so zero results hard-failed while wrong results fell
back. Backwards, and soulseek's plugin already defaults fallback=True for
exactly this reason.
Fix: set fallback=True on the no-results branch in torrent.py and usenet.py.
The dispatch's fallback handling (return False -> per-track flow) was already
correct and is unchanged.
The only consumer of download_album_to_staging is the dispatch, which reads
the result via .get('fallback'), so the change is additive and locally
contained.
Tests: new test_torrent_album_to_staging_no_results_flags_fallback and
test_usenet_album_to_staging_no_results_flags_fallback assert the plugins now
emit fallback=True on an empty search; the existing torrent no-results test is
extended with the same assertion. Existing dispatch tests already pin
fallback=True -> per-track flow. Full downloads/plugins/adapters sweep: 690
passed.
The earlier #721 fix tolerated a ~10s "completed but no save_path"
window, but the real production stall sits upstream of that: SABnzbd
removes a finished download from the queue and runs par2 verify /
repair / unpack *in History*, exposing the live stage in the slot
`status` ('Verifying' / 'Repairing' / 'Extracting' / 'Moving' / ...)
with `storage` empty until the final move. `_parse_history_slot` mapped
EVERY non-'Failed' status to 'completed', so a still-extracting 1.7 GB
FLAC album looked "completed with no save_path" the instant download hit
100%. The poll burned its completed-no-path budget mid-PP and bailed,
freezing the UI on the last download emit (the stuck-at-99%/100%
signature). SAB then finished fine — which is why the job shows
Completed in History but SoulSync never staged it.
Root fix
- `_parse_history_slot` routes `status` through `_map_state`, so PP
stages stay NON-terminal: the poll keeps waiting (as 'downloading')
for as long as post-processing takes and only a real 'Completed'
flips to terminal success. `save_path` is trusted only on true
completion (mid-PP path fields may point at the incomplete dir).
Supporting / defensive
- `UsenetStatus.incomplete_path`: surfaced separately from save_path
(SAB `incomplete_path`) and used by the poll loops as a LAST RESORT
after the completed-no-path window, to recover the case where
`storage` never lands but the files are physically on disk.
- `poll_album_download`: dedicated, configurable completed-no-path
window (~120s via `download_source.album_bundle_completed_no_path_seconds`)
decoupled from the ~10s transient-miss window; incomplete_path
fallback; a 30s heartbeat log so the previously-silent poll loop is
diagnosable.
- `usenet.py` `_download_thread`: per-track parity — it was erroring
immediately on the first completed-no-path read.
- `album_bundle_dispatch.py` / `status.py` / `monitor.py`: use the
project `get_logger` so download-flow logs land in app.log under the
`soulsync.*` namespace (they were console-only before, which hid the
`[Album Bundle] flow failed` line during triage).
Tests
- PP-history state mapping; end-to-end Hunky Dory PP regression
(download -> Verifying/Extracting in History past both budgets ->
Completed+storage -> success); completed-no-path window +
incomplete_path fallback; per-track thread parity. ruff + compileall +
pytest all green (the only local failures are environmental: missing
tzdata + local tools/ffmpeg.exe, neither present on CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refresh registry-backed download plugins when settings are saved so cached Prowlarr clients pick up new indexer credentials immediately. This preserves active download state by reloading existing plugin instances instead of rebuilding the registry.
Add regression coverage for orchestrator reload fanout and the Usenet plugin's cached ProwlarrClient refresh path.
Route torrent and Usenet album bundles through private per-batch staging so Auto-Import cannot race public staging or duplicate imports.
Expose album-bundle progress in batch status and render it on the Downloads page while the external client is still downloading.
Tighten release handoff safety by rejecting archive path traversal, ignoring torrent candidates without a usable URL, and skipping Soulseek source reuse for torrent/Usenet batches.
Tests: .venv/bin/python -m pytest tests/downloads/test_downloads_status.py tests/test_album_bundle_dispatch.py tests/downloads/test_downloads_staging.py tests/test_torrent_usenet_plugins.py
Per code review: the album-bundle helpers (release picker + staging
collision suffix) were defined as private symbols in torrent.py and
imported by usenet.py through ``from core.download_plugins.torrent
import _pick_best_album_release, _unique_staging_path``. Sibling
plugins shouldn't reach into each other's private surface — leaky
module boundary, and the underscore prefix says don't import.
Also addressed two latent issues at the same time:
- The Auto-Import sweep race: my plugin copied audio files into
staging via plain ``shutil.copy2``, which exposes a partial file
at the audio extension for the duration of the copy. The Auto-
Import worker filters by audio extension when scanning Staging
(AUDIO_EXTENSIONS in core/auto_import_worker.py), so a mid-flight
scan could pick up a truncated file. Fix: copy to a
``.tmp.<random>`` sidecar first, then atomically rename via
``Path.replace`` (which is ``os.replace`` — atomic on the same
filesystem). Auto-Import sees the file either at its final name
or not at all.
- The 6-hour poll timeout was a hard-coded magic constant. Users
with slow private trackers or large box sets would silently time
out after 6h. Both the timeout and the poll interval are now
read from config (``download_source.album_bundle_timeout_seconds``
/ ``..._poll_interval_seconds``) with safe fallback to the
existing defaults when unset / non-numeric.
- core/download_plugins/album_bundle.py: new module owns the
shared surface — ``pick_best_album_release`` (with quality_guess
passed in as a parameter to avoid the circular import that would
result from this module trying to know about torrent.py's title
parser), ``unique_staging_path``, ``atomic_copy_to_staging``,
``copy_audio_files_atomically``, ``get_poll_interval``,
``get_poll_timeout``. Module-level size constants and quality
weights live here too. Usenet's grabs-as-popularity-proxy is
built into the picker so both plugins get the right behavior
without divergent local logic.
- core/download_plugins/torrent.py: drops the local helpers + the
hard-coded poll constants, imports from album_bundle. Per-track
download flow still uses module-level ``_POLL_TIMEOUT_SECONDS``
/ ``_POLL_INTERVAL_SECONDS`` aliases (read from config once at
import time, same as before from a per-track perspective).
- core/download_plugins/usenet.py: drops the imports of the
torrent.py private helpers; everything goes through album_bundle
now. Stops the cross-plugin private-import leak that started
this whole refactor.
- tests/test_album_bundle.py: 23 new tests covering the picker
heuristic (empty input, singleton drop, FLAC preference, grabs
fallback for usenet, size-floor / ceiling boundaries), the
collision-suffix logic, the atomic-copy invariant (concurrent
scanner thread asserts it never observes a partial audio file
during five sequential copies), the failure-skip behavior of the
batch copier, and the config-driven poll cadence including
garbage-input fallback.
- tests/test_torrent_usenet_plugins.py: existing picker tests
updated to call the new module-level helpers instead of the
former torrent.py privates.
Fixes the core architectural mismatch between indexer-based sources
and the per-track search-and-pick contract every other download
plugin satisfies. Prowlarr returns release-level torrents and NZBs;
searching for "Luther (with SZA)" against the GNX album torrent
scores near-zero on track-title similarity. Per-track candidate
validation rejects every result, every track in the batch flips
to not_found. The album-name fallback added in an earlier commit
papers over it for some cases but doesn't fix the fundamental
behavior: the user wanted the whole album.
New album-bundle flow does what the user actually wanted:
1. Gate fires inside core/downloads/master.py BEFORE the per-track
analysis loop, strictly when the batch has an album context AND
download_source.mode is 'torrent' or 'usenet' (single-source —
hybrid stays per-track to preserve fallback to Soulseek / etc.).
2. Plugin's new download_album_to_staging method searches Prowlarr
ONCE for the album as a whole ('<artist> <album>'), filters to
the right protocol, runs results through _pick_best_album_release.
3. Picker prefers seeded FLAC over low-seeded MP3, drops single-
track torrents that snuck in via the 40 MB size floor (single
tracks are typically ~10 MB), falls back to most-seeded when
every candidate is below the floor.
4. Picked release goes to the active adapter (qBit / Transmission /
Deluge for torrent; SAB / NZBGet for usenet). Polls until
complete with progress mirrored into the batch state so the
Downloads page can show meaningful status.
5. On completion the existing archive_pipeline walks the save dir
(extracting archives if any), every audio file gets copied into
the staging folder via _unique_staging_path so concurrent batches
don't collide.
6. Gate exits, master worker continues into the normal per-track
flow. Each track task hits try_staging_match early in the worker
and finds its file by fuzzy title match — no Prowlarr search
ever fires per-track, no candidate rejection, files flow through
the existing post-processing pipeline (tags, AcoustID, library
import).
Gate is strictly opt-in. Three orthogonal conditions must all hold:
batch_is_album, mode in ('torrent', 'usenet'), and the plugin must
expose download_album_to_staging. Any other source / hybrid mode /
non-album batch flows through the master worker unchanged. The
existing per-track torrent path still works for basic-search
single-track grabs.
- core/download_plugins/torrent.py: download_album_to_staging plus
_pick_best_album_release and _unique_staging_path helpers (shared
with the usenet plugin). _poll_album_download mirrors the existing
poll loop with progress callback emission.
- core/download_plugins/usenet.py: parallel implementation reusing
the picker + staging helpers. Different state set ('failed' vs
'error') from the usenet adapter contract.
- core/downloads/master.py: ~90-line gate right after batch context
loading. Mirrors plugin lifecycle into batch state under
``album_bundle_*`` keys so the Downloads page can render progress
while the torrent/usenet job runs (per-track tasks don't exist
yet during this phase). Failed bundle download fails the batch
with a meaningful error; missing plugin / context falls back to
the per-track flow with a warning.
- tests/test_torrent_usenet_plugins.py: 5 new tests pinning the
album picker preferences (FLAC over MP3 with comparable size +
better seeders, size floor drops singles, fallback when all
small), staging-path collision suffix, and the not-configured
short-circuit.
Real-world test surfaced the bug — torrent results displayed
'by download?apikey=c15d6f69...&link=...' as the uploader / artist
in the basic search UI. The cause is TrackResult.__post_init__:
when artist is None it runs parse_filename_metadata on the bare
filename, and our filename starts with the indexer's download URL
(needed so download() can recover the URL later). The auto-parser
treats the URL as 'artist' and ships it to the UI.
Fix:
- core/download_plugins/torrent.py: new _parse_release_title()
splits 'Artist - Title' / 'Artist - Album' out of the release
title and strips trailing [FLAC] / (2016) tags. Falls back to
('', cleaned_title) when no dash is found, and explicitly
rejects URL-looking strings as an extra defence. The projection
pre-fills both artist and title on TrackResult, so __post_init__
skips the auto-parse entirely. When the release title has no
dash, artist defaults to the indexer name so the UI shows
'by Indexer' instead of a URL.
- core/download_plugins/usenet.py: imports the new helper and
applies the same fix.
- tests/test_torrent_usenet_plugins.py: 5 tests for the new
helper (dash split, trailing-tag stripping, no-dash fallback,
multiple-dash preservation, URL-prefix rejection). Existing
projection tests updated to assert artist + title come through
parsed correctly, plus a new test pinning the indexer-name
fallback for titles without a dash so the URL-leak regression
can't return.
The payoff for the previous five commits. Two new download
sources slot into the existing DownloadSourcePlugin contract,
backed by Prowlarr (search) + the torrent or usenet client
adapter (transfer) + archive_pipeline (post-extract walk). They
appear in the Download Source dropdown next to Soulseek / Tidal /
Lidarr / etc. and also participate in hybrid mode.
Pipeline (both plugins, mirror shape):
1. search(query) → ProwlarrClient.search filtered to the right
protocol, projected into TrackResult / AlbumResult shapes the
existing search UI already speaks. Filename field encodes the
indexer's download URL (or magnet URI for torrents) so
download() can recover it later.
2. download() → decodes URL, hands it to the active adapter
(qBittorrent / Transmission / Deluge for torrent; SABnzbd /
NZBGet for usenet), spawns a background poll thread that
tracks progress + reports the adapter-reported save_path.
3. On 'seeding' / 'completed' → archive_pipeline walks the save
directory, extracts any archives the downloader didn't
already unpack, picks the first audio file as the canonical
file_path. Matches the Lidarr client's single-track-pick
contract — picking which specific track to import happens in
post-processing.
- core/download_plugins/torrent.py: TorrentDownloadPlugin +
module-level helpers (_decode_filename, _guess_quality_from_title,
_parse_indexer_id_filter, _adapter_state_to_display, _row_to_status).
Uses get_active_torrent_adapter() so a settings change to the
client type takes effect without restart.
- core/download_plugins/usenet.py: UsenetDownloadPlugin —
parallel shape, reuses the torrent module's helpers. Different
enough states (no seeding, no magnet) to warrant its own class
but cheap to keep in lockstep.
- core/download_plugins/registry.py: register 'torrent' and
'usenet' plugins. Per the registry docstring this is the only
wiring point needed — the orchestrator picks them up
automatically via the iteration helpers.
- webui/index.html: 'Torrent Only (via Prowlarr)' + 'Usenet Only
(via Prowlarr)' added to the Download Source dropdown. New
redirect card (#prowlarr-source-redirect) explains that the
actual config lives on the Indexers & Downloaders tab —
shown whenever torrent or usenet is in the active source set.
- webui/static/settings.js: HYBRID_SOURCES gets two new entries
so hybrid mode can pick them up. updateDownloadSourceUI now
toggles the redirect card based on active sources.
- tests/test_torrent_usenet_plugins.py: 23 tests covering pure
helpers (filename encode/decode round-trip incl. magnet URIs,
quality guesser, state mapping), search projection logic
(protocol filter, drops without URLs, magnet-preferred-over-URL,
filename encoding, neutralised soulseek-specific score fields),
is_configured (both prowlarr + adapter required), finalize
(picks first audio file, errors on empty dir / missing save_path),
clear/get_all lifecycle, DownloadSourcePlugin protocol
conformance, and registry membership.
BoulderBadgeDad