Stage 2: the manual 'Download Wishlist' flow now calls the same
_run_wishlist_cycle engine the auto timer uses, so a manual scan runs the exact
same code path as an auto scan. The old bespoke manual orchestration (build
payloads + SERIAL inline dispatch) is deleted — its grouping/dispatch was a
near-duplicate of auto's that had already drifted.
Behavior changes (all intended, discussed):
- Manual now dispatches album bundles in PARALLEL (album pool) like auto, instead
of serially on one thread. A single cycle='albums' engine call covers the whole
selection (albums bundled, singles/ungroupable -> per-track residual), so no
'both cycles' pass is needed.
- The manual placeholder batch_id is reused as the engine's first sub-batch
(first_batch_id), so the modal's existing poll target stays valid.
- WishlistManualDownloadRuntime gains album_bundle_executor (wired in web_server,
falls back to the shared pool when unset).
- 'Don't start manual while auto is running' is unchanged — the existing route
guard (is_wishlist_actually_processing -> 409) already covers it; no queue added.
NOT touched: process_wishlist_automatically's behavior (proven by test_automation
staying green in Stage 1) and the per-track download mechanics.
test_manual_download.py rewritten to characterize the new behavior (engine
dispatch via the executor, parallel, placeholder reuse, album-context). Full
wishlist suite green (131); wishlist + automation = 392 passed.
Real-world wishlist case the original c3b88e69 design missed: user with
26 missing tracks from 26 different albums. Each item used to promote
to its own album-bundle sub-batch (``min_tracks_per_album=1``), which
downloaded the ENTIRE album (5-42 files) to claim one track. Confirmed
in app.log:
- "Licensed To Ill" downloaded 3 times across cycles (3-4 files each)
- "The Understanding" 17 files for 1 wishlist track
- "Alright, Still" 42 files for 1 wishlist track
- ~85% wasted bandwidth, slskd hammered with 26 concurrent searches
PR 1 of a 4-PR fix series — see commit body footer for the other PRs.
Default ``min_tracks_per_album`` 1 → 2. Single-track wishlist items
fall to ``residual_tracks`` → classic per-track batch (already works,
already efficient). Album-bundle kept for the case it was designed
for: user has 2+ tracks missing from the same album.
Override via the new ``wishlist.album_bundle_min_tracks`` config key:
- 1 = previous behaviour (bundle every item)
- 2 = new default
- 3+ = stricter, for users who want bundle only on bigger gaps
Helper ``_resolve_album_bundle_threshold`` lives in
``core/wishlist/processing.py``. Defensive shape mirrors the existing
config-driven knobs (``get_poll_interval`` / ``get_transient_miss_threshold``):
non-numeric, non-positive, or config-manager-raise all fall back to
the safe default. Three test cases pin the fallback chain.
Both wishlist entry points wired through the same helper:
- ``process_wishlist_automatically`` (auto cycle, line 812)
- ``start_manual_wishlist_download_batch`` (manual run, line 539)
Tests:
- ``tests/wishlist/test_album_grouping.py`` — old ``test_default_threshold_promotes_solo_albums`` flipped to ``test_default_threshold_demotes_solo_albums`` with explanatory docstring naming the real-world cause. New ``test_default_threshold_promotes_multi_track_albums`` pins the 2+ promotion. New ``test_explicit_threshold_one_restores_solo_promotion`` pins that the kwarg still works for opt-back-in.
- ``tests/wishlist/test_processing.py`` — 3 new tests for ``_resolve_album_bundle_threshold``: default-when-config-missing, honors-config-override, falls-back-on-garbage.
- ``tests/wishlist/test_automation.py`` — ``test_wishlist_albums_cycle_splits_into_per_album_batches`` updated to use 2+ tracks per album (5 tracks across 2 albums instead of 3 across 2 with 1 solo). ``test_wishlist_albums_cycle_residual_for_orphan_tracks`` updated to include 2 tracks from Album One so it still promotes.
- ``tests/wishlist/test_manual_download.py`` — same shape update for the manual path test.
- ``tests/wishlist/test_album_grouping.py:test_multiple_albums_emit_separate_groups`` updated to reflect new default (alb1 with 2 tracks promotes, alb2 with 1 track goes residual).
- ``tests/wishlist/test_album_grouping.py:test_nested_track_data_payloads_normalized`` pinned with explicit ``min_tracks_per_album=1`` so the test stays focused on payload-shape parsing, not the threshold rule.
114 wishlist tests pass; 866 across wishlist + automation + downloads +
album_bundle + album_bundle_dispatch suites still green. Ruff clean.
Sibling PRs queued in TaskCreate:
- PR 2 — investigate post-process staging-match miss (the second-order
bug that causes the same album to redownload every cycle when the
staging step doesn't claim the requested track).
- PR 3 — fix sibling-completion gate that fires on first sibling
instead of last (log evidence: run a4945c88 finalized 1/26 batches).
- PR 4 — UI distinguish Queued from Analyzing for batches waiting
on the executor (23/26 batches sit at "Analyzing..." while really
queued at max_workers=3).
The Phase-1 fix (commit c3b88e69) only extended the per-album
bundle dispatch to ``process_wishlist_automatically``. The manual
"Run Wishlist Now" path goes through
``_prepare_and_run_manual_wishlist_batch`` instead, so the
behavior didn't change for users who triggered downloads from the
Wishlist tab UI — they still saw N per-track Soulseek searches
when N missing tracks all came from one album.
Caught in a real-app test: user added Katy Perry's PRISM (Deluxe)
to the wishlist + clicked "Download Wishlist" → app log shows
``_prepare_and_run_manual_wishlist_batch:421`` running a single
batch with 16 tracks + per-track searches firing one by one
("katy perry prism deluxe legendary lovers", "katy perry prism
deluxe roar", etc.), no album-bundle dispatch.
Fix:
- ``_prepare_and_run_manual_wishlist_batch`` now runs the same
``group_wishlist_tracks_by_album`` helper after filtering. For
each detected album, it builds a sub-batch with
``is_album_download=True`` + populated album/artist context.
Residual tracks (no resolvable album metadata) land in a single
per-track residual batch.
- The first sub-batch re-uses the caller-allocated ``batch_id``
so the frontend's existing poll against it keeps working;
additional sub-batches get fresh ids materialized into
``download_batches`` so they show up in the Downloads view.
- Sub-batches dispatch serially — each ``run_full_missing_tracks_process``
call blocks until the album-bundle staging + per-track tasks
complete before the next album's bundle search fires.
New test ``test_manual_wishlist_splits_into_per_album_sub_batches``
pins the contract — multi-album wishlist content with
nested-spotify_data shape produces N master-worker calls (one per
album), each batch carries the album_context, first sub-batch
re-uses the original batch_id. 106 wishlist tests + 1099 across
the broader suite green.
Adding 16 Katy Perry PRISM tracks to wishlist + clicking download
should now fire ONE slskd album-bundle search for the release
instead of 16 individual searches.
Both the auto and manual wishlist download paths called
`remove_tracks_already_in_library` before submitting the batch — a
serial DB lookup per track per artist (~1s/track on a 24-track
wishlist). The batches set `force_download_all=True` which is
explicitly documented as "skip the expensive library check" — the
pre-flight cleanup was contradicting that flag.
Removed the cleanup call from both flows. Kept `remove_wishlist_duplicates`
(fast SQL DELETE) and the standalone `/api/wishlist/cleanup` endpoint
that exposes the library scan as explicit user-triggered maintenance.
Safety check on the trade-off:
- post-processing at `core/imports/pipeline.py:576-624` already handles
re-downloads defensively: existing file with metadata → skip overwrite
+ delete source duplicate, no library corruption.
- Master worker's analysis loop normally removes wishlist entries for
found tracks via `_check_and_remove_track_from_wishlist_by_metadata`,
so stale wishlist entries should be rare in practice.
- Worst case for the rare orphan: one redundant download attempt that
the post-processing layer no-ops on. Bandwidth waste, not data damage.
Tests updated:
- `..._does_not_run_library_cleanup` (renamed from `_skips_enhance_tracks_during_cleanup`)
asserts no DB track-existence checks happen and no wishlist removals
fire — both `enhance` and "owned" tracks reach the master worker.
- `..._marks_batch_complete_when_wishlist_genuinely_empty` (renamed from
`..._after_cleanup`) covers the path where the wishlist starts empty.
Full suite: 1232 passing. Ruff clean.
The manual wishlist download endpoint blocked the request thread on a
slow library-cleanup pass before submitting the batch — for a 24-track
wishlist that's ~50 per-track DB lookups serialised in the request
handler, taking 30+ seconds before the frontend got a response. The
modal sat at "Pending..." with no progress visible the whole time.
Split start_manual_wishlist_download_batch into:
1. SYNC path (request handler):
- Generate batch_id, create download_batches entry with phase=analysis
and analysis_total=0 placeholder.
- Submit a single bg job (`_prepare_and_run_manual_wishlist_batch`) to
the missing-download executor.
- Return 200 with batch_id immediately. Frontend can start polling
/api/active-processes status right away.
2. BG path (executor thread):
- db.remove_wishlist_duplicates (slow-ish, single SQL)
- remove_tracks_already_in_library (the slow one — per-track DB checks)
- wishlist_service.get_wishlist_tracks_for_download
- sanitize + dedupe + filter (track_ids / category)
- Update batch.analysis_total with the real filtered count
- add_activity_item("Wishlist Download Started", ...)
- run_full_missing_tracks_process (master worker)
Edge case: if cleanup empties the wishlist, the bg job marks the batch
phase='complete' with error='No tracks in wishlist' (instead of the old
synchronous 400 response). Frontend status poll picks this up and the
modal can close cleanly.
Tests: existing 2 manual-download tests updated to drive the bg job
explicitly via a new `_run_submitted_bg_job` helper. Added 2 new tests:
- `..._returns_immediately_with_placeholder` — proves the sync path
doesn't trigger any cleanup or master-worker calls; analysis_total=0.
- `..._marks_batch_complete_when_wishlist_empty_after_cleanup` —
cleanup empties the list, master worker never invoked, batch ends
with phase='complete'.
Full suite: 1232 passing (was 1230). Ruff clean.
- remove the redundant wishlist-service injection from the runtime wrappers
- keep the package owning its own singleton service access
- simplify the route runtime API and update the wishlist tests to match
- add core/wishlist as the home for wishlist payload, resolution, state, processing, reporting, and selection helpers
- move wishlist-specific tests into tests/wishlist alongside the new package layout
- keep web_server.py and the import/search callers as thin adapters for now
