Foundation for multi-listener playback. Today web_server.py keeps ONE global
stream_state dict + one lock (web_server.py:747), so the whole server shares a
single 'currently playing' — every tab/device is a remote for the same
playback and two listeners collide. That global is woven through ~22 sites and
isn't unit-testable where it lives.
Lifted into core/streaming/state.py WITHOUT changing behavior:
- StreamSession: one playback's state, dict-compatible (s['k'], s.get,
s.update, 'k' in s) so existing call sites work unchanged, each with its
OWN RLock so distinct sessions never block/clobber each other.
- StreamStateStore: registry of named sessions; lazy + race-safe create;
DEFAULT session reproduces today's exact single-global behavior. Also
drop()/active_ids()/session_ids() for the eventual per-listener wiring.
web_server.py now binds (DEFAULT) and
. Drop-in: every .update()/[k]/.get()/ site behaves identically. _set_stream_state routes a reassign
through session.replace() so the store's session stays the live object (it's
effectively dead — prepare.py only mutates in place — but safe now).
Honest scope: this is the PROVABLE half of Phase 3. The remaining half (3b:
derive a per-browser session id, per-session Stream/ staging, executor
concurrency, disconnect cleanup) is browser-coupled and can't be verified
without driving 2+ live clients — deferred to a live session. The store API is
already shaped for it.
Tests (tests/streaming/, 33 total):
- test_stream_state_store.py (19): session dict-compat, isolation, lazy
create, drop rules, active_ids, concurrent-create race safety.
- test_stream_state_callsite_compat.py (7): every real web_server access
pattern (library/play, stream/start, status, audio guard, stop, prepare
in-place mutation, set->replace) against the exact object web_server binds.
- test_prepare.py +1: real prepare worker drives an actual StreamSession.
76 streaming+radio tests green; ruff clean; web_server.py parses.
Replaces radio's pure ORDER BY RANDOM() with weighted ranking. Each tier now
fetches a generous random POOL (4x the needed count, floored) and
core/radio/selection ranks it before the collector keeps the best:
score_candidate = play_count(log-damped, w=1.0)
+ lastfm_playcount(log-damped, w=0.5)
- recently_played penalty(w=2.0)
+ stable per-id jitter(w=1.0, hash-derived so runs vary but
tests stay reproducible)
Modest weights so popularity guides without burying lesser-played tracks, and
jitter keeps radio from being identical every run. All intelligence is in pure
functions (rank_candidates / score_candidate) so it's tunable + unit-testable
without SQL.
Defensive: the DB method probes PRAGMA table_info(tracks) and omits
play_count/lastfm_playcount from the SELECT when absent (older DBs predating
the listening-history migration) — the scorer treats missing signals as 0, so
radio degrades to jitter-only instead of crashing on 'no such column'.
Tests (tests/radio/, 43 total):
- score_candidate / rank_candidates: deterministic unit coverage (popularity
ordering, lastfm contribution, recency penalty, garbage→0, stable jitter).
These CANNOT pass against pre-Phase-2 code.
- DB end-to-end: ranking surfaces the heavily-played track first out of a
decoy pool (wiring proof — probabilistic vs old random, documented honestly);
plus a no-rank-columns DB proving the defensive degrade path.
- All Phase-0a behavioral/refactor-equivalence tests still green.
60 radio + adjacent-DB tests pass; ruff clean.
First step of the stream/player/radio revamp (see revamp_plan.md). The radio
algorithm lived inline inside database.music_database.get_radio_tracks as raw
SQL tangled with selection logic — untestable without a live DB (which also
throws in the dev sandbox). Lifted the pure DECISIONS into core/radio/selection.py:
- parse_tags / merge_tags — JSON-or-CSV tag fields → ordered deduped list
- same_artist_cap — tier-1 30%-floored-at-5 cap
- build_like_conditions — OR-of-LIKEs SQL fragment + params per tier
- RadioCollector — dedup + cap + exclude-set + NOT-IN placeholder/value tracking
The DB method keeps the cursor work and now delegates every decision to these
helpers. Faithful extraction, not a rewrite — behavior unchanged.
This is the kettui foundation move: radio is now unit-testable, so Phase 2
(smart ranking — play-count / recency / feature seeding) becomes 'evolve a
tested function' instead of 'rewrite SQL and pray'.
Tests (tests/radio/):
- test_selection.py (22): unit coverage of every extracted helper
- test_get_radio_tracks_db.py (7): drive the REAL get_radio_tracks against
in-memory sqlite — tier fallback, dedup, exclude, file_path filter.
Behavior-pinned: these 7 pass against BOTH old inline and new extracted
code (refactor-equivalence proof). 52 adjacent DB+radio tests green.
The $year template variable was a blind release_date[:4] slice. When
something upstream poisoned release_date with a non-date value — the album
NAME — that slice emitted garbage: 'Mantras (Deluxe)'[:4] -> 'Mant', so
every download landed in 'Mantras (Deluxe) (Mant) [Album]/' instead of
'(2026)' (Tacobell444's screenshot).
Add _extract_year_from_release_date(): returns the leading 4 chars only
when they're a plausible year (isdigit, 1900 < y <= 2100), else ''. Matches
the guard the codebase already uses in soulid_worker._extract_year. A
non-year resolves to '' and the template's existing empty-() cleanup drops
it, so a poisoned release_date can never write rubbish into the path again.
This is the shared post-process path builder
(core/imports/paths.build_final_path_for_track) that DOWNLOADS, reorganize,
and imports all route through, so the guard covers every surface at once.
Defensive fix only — it stops the SYMPTOM regardless of which upstream
writes the album name into release_date. Pinning that upstream needs the
reporter's metadata source + the release_date value from app.log (the
Soulseek + AcoustID + future-dated-album combo is the discriminator);
tracked separately.
Tests (tests/imports/test_import_paths.py): unit coverage for the helper
(real dates kept, names/sentinels/short values rejected) + an integration
test reproducing #745 — a poisoned release_date yields 'Mantras (Deluxe)
[Album]' not '(Mant)' — differential-verified it produces the exact
'(Mant)' folder without the fix. Positive control keeps real (2026). 395
import + reorganize tests green.
The Duplicate Cleaner moves de-duplicated files into <transfer>/deleted/.
If a user's media server scans the transfer folder (e.g. a /music root
holding both the library and the transfer dir), those quarantined files
get real track rows in SoulSync's DB. Reorganize is purely DB-driven —
it acts on each track's stored file_path — so it would dutifully move a
quarantined file back OUT of /deleted to the template location, exactly
what Tacobell444 reported.
We can't stop the rows from existing (they come from the media server,
which the app doesn't control), so the fix is bounded to Reorganize, as
the reporter asked: skip any track whose resolved path is under
<transfer>/deleted. Surfaced as a non-matched 'In deleted/quarantine
folder — skipped' in the preview; apply mirrors it (post-process never
runs, file left in place, counted as skipped).
Detection is anchored to the <transfer>/deleted PREFIX (not a bare
substring) so a real album like 'Deleted Scenes' is kept; falls back to
an exact 'deleted' path-segment match when transfer_dir is unavailable
(mirrors the cleaner's own 'if deleted in dirs' skip). The one
unavoidable ambiguity — an artist folder named exactly 'deleted' at the
transfer root — is pinned in a test as intentional.
Guard added once where both consumers see it: preview_album_reorganize
and the apply worker (_RunContext gains transfer_dir).
Tests: tests/test_reorganize_deleted_quarantine.py (8 unit) +
test_library_reorganize_orchestrator.py (preview + apply integration,
differential-verified they fail without the fix). 128 adjacent
reorganize tests still green.
When the preflight-selected Soulseek folder produces zero usable files —
every transfer failed/aborted/stalled (the Slipknot dead-peer case: all
tracks 'Completed, Aborted' at 0 bytes) — _poll_album_bundle_downloads
returns []. download_album_to_staging used to return that with
fallback=False, so try_dispatch marked the whole batch failed and nothing
was retried elsewhere until the next wishlist run.
Flip that branch to fallback=True so the existing, proven per-track flow
takes over and re-searches every missing track across ALL sources/peers.
This reuses the per-track multi-source robustness instead of reimplementing
candidate-folder retry inside the bundle.
Tests: tests/test_soulseek_album_fallback.py drives the preflight-reuse path
with a stubbed poll — empty poll -> fallback=True (differential-verified it
fails without the fix), healthy poll -> fallback stays False. Downstream
routing (fallback=True -> per-track) already covered by
test_album_bundle_dispatch.py.
Live testing surfaced that slskd reports a peer-side abort as 'Completed,
Aborted' at 0 bytes (peer accepts then drops every transfer). That string
contains 'Completed', so the poll's completed-branch ran first and misread it as
'completed but file missing' — routing it into the #715 unresolved/download_path
grace (gives up after 45s with a misleading 'download_path mismatch' log)
instead of recognizing it as a failure.
Add 'Aborted' and 'Cancelled' to the failure-token check (which runs before the
completed branch), so these resolve immediately and correctly as failed. Test
added for the all-aborted folder.
Two related bugs from the Slipknot album never finishing.
1) _poll_album_bundle_downloads hung when the peer stalled. The finish check
needs every transfer terminal (completed/failed); the #715 grace only covers
'slskd says Completed but file not on disk'. A transfer stuck InProgress /
Queued, or dropped by slskd, is none of those — so it blocked both the finish
AND the grace exit, and the poll spun to the full ~6h timeout.
Add a bundle-level stall guard: track a progress marker (#terminal transfers,
total bytes across pending). If NOTHING advances for _stall_grace (180s) —
no terminal transition AND no pending byte movement — the peer has stalled;
mark the stuck transfers failed so the existing finish/all-failed checks
resolve the bundle with whatever completed (missing tracks then fall back to
the per-track matcher). Conservative: only trips when EVERYTHING is frozen,
so a slow-but-progressing or still-queued transfer is unaffected.
2) Failed batches lingered in the UI forever ('No tracks loaded'). The
auto-cleanup gate removed only complete/error/cancelled phases — 'failed'
(e.g. an album-bundle hard failure) was missing, so it never aged out. Add
'failed' to the terminal set so it's removed after 5 minutes like the others.
Tests (tests/test_soulseek_album_poll_stall.py): stalled peer → gives up with the
completed subset (not the deadline); progressing bundle not falsely stalled;
all-stalled → empty; dropped transfers stall out; clean finish unaffected.
124 download/soulseek tests pass; ruff clean.
Stage 2: the manual 'Download Wishlist' flow now calls the same
_run_wishlist_cycle engine the auto timer uses, so a manual scan runs the exact
same code path as an auto scan. The old bespoke manual orchestration (build
payloads + SERIAL inline dispatch) is deleted — its grouping/dispatch was a
near-duplicate of auto's that had already drifted.
Behavior changes (all intended, discussed):
- Manual now dispatches album bundles in PARALLEL (album pool) like auto, instead
of serially on one thread. A single cycle='albums' engine call covers the whole
selection (albums bundled, singles/ungroupable -> per-track residual), so no
'both cycles' pass is needed.
- The manual placeholder batch_id is reused as the engine's first sub-batch
(first_batch_id), so the modal's existing poll target stays valid.
- WishlistManualDownloadRuntime gains album_bundle_executor (wired in web_server,
falls back to the shared pool when unset).
- 'Don't start manual while auto is running' is unchanged — the existing route
guard (is_wishlist_actually_processing -> 409) already covers it; no queue added.
NOT touched: process_wishlist_automatically's behavior (proven by test_automation
staying green in Stage 1) and the per-track download mechanics.
test_manual_download.py rewritten to characterize the new behavior (engine
dispatch via the executor, parallel, placeholder reuse, album-context). Full
wishlist suite green (131); wishlist + automation = 392 passed.
The auto and manual wishlist flows each built the same ~20-field
download_batches row in separate places (auto album, auto residual, manual
placeholder, manual sub-batches) — four near-identical literals that could (and
did) drift apart, producing subtly different batch shapes between the flows.
Extract make_wishlist_batch_row() as the single source of truth: it emits the
consistent core field set, with the genuinely per-flow differences as explicit
arguments — initial phase ('queued' for auto / 'analysis' for manual), the
auto-only auto_initiated/auto_processing_timestamp/current_cycle via
extra_fields, and album-vs-residual contexts. All four sites now go through it,
so every wishlist batch has an IDENTICAL shape (this also removes the field
drift that confused the modal-hydration code).
Deliberately NOT unified — and left explicit in each caller, per the
'don't cargo-cult genuinely-different code' principle: the grouping decision
(auto groups only on the albums cycle), batch-id allocation (manual reuses the
caller's placeholder id for the first sub-batch), and dispatch (auto
parallel-submits album batches to the dedicated pool + residual to the shared
pool; manual runs them serially on one thread). Those are real behavioral
differences, not duplication.
Behavior-preserving: verified safe to normalize the row shape (grep confirmed
every reader uses .get() with defaults, no key-presence checks). The existing
auto (test_automation.py) and manual (test_manual_download.py) characterization
suites stay green = differential proof of identical behavior. Adds
test_batch_factory.py (core fields, album/residual, extra_fields, no shared
mutable state, consistent key shape). 131 wishlist tests pass.
A 2.6.3 change (c3b88e69) split the wishlist albums cycle into one batch per
album. Each album batch runs an INLINE-BLOCKING soulseek/torrent/usenet
album-bundle download (album_bundle_dispatch.try_dispatch ->
download_album_to_staging) that holds its worker thread for the whole
search+download. All of these were submitted to the shared 3-worker
missing_download_executor -- the same pool used for per-track downloads AND the
manual 'Download Wishlist' analysis.
So a large Album-Completeness 'Fix all' (-> ~819 wishlist tracks -> ~82 per-album
batches) saturated all 3 workers with blocking album downloads; the manual
wishlist analysis could never get a thread ('Library Analysis' stuck on
Pending), the other ~79 batches sat in phase='queued' forever, and auto-cleanup
(which only evicts terminal-phase batches) never cleared them -> jam until
restart. Fixing batch STATUS would not help: the threads are blocked inside the
download, not waiting on a phase flip.
Fix: add a dedicated bounded album_bundle_executor (max_workers=3) and route the
AUTO per-album bundle batches to it, keeping the shared pool free for analysis /
per-track / the manual wishlist (which always starts now). Hung/slow album
downloads can only delay other album downloads, never the user-facing path.
Additive and decoupled; the submit site falls back to the shared pool when the
album pool isn't wired (older callers / tests) so behavior is unchanged there.
The manual path is untouched (it already runs album bundles serially on a single
thread, by design).
Tests (tests/wishlist/test_automation.py): album sub-batches route to the
dedicated pool while the residual per-track batch stays on the shared pool;
fallback-to-shared-pool when no album pool is wired. Existing auto-processing
tests still green (fallback preserves prior behavior). 707 passed across
wishlist + downloads suites.
The same provider ID is stored under inconsistent column names across tables
(deezer_id vs deezer_artist_id vs album_deezer_id vs similar_artist_deezer_id;
spotify/itunes keep an entity qualifier, others don't; musicbrainz uses three
nouns), so code checks 2-5 name variants everywhere.
Add core/source_ids.py as the single source of truth for (provider, entity) ->
column, with accessors that read an ID from a dict/sqlite3.Row robustly
(canonical column first, then known aliases). NO database columns are renamed —
these are the real names today; the registry just centralizes the knowledge.
Targeted adoption (behavior-identical, verified):
- core/artist_source_lookup.SOURCE_ID_FIELD now derives from the registry
instead of duplicating the mapping (values unchanged).
- web_server.py artist-detail builds artist_source_ids via source_id_map(...)
instead of a hand-rolled per-source .get() dict.
Broader call-site adoption deferred as clearly-scoped follow-up.
Tests: tests/test_source_ids_registry.py (canonical columns, alias fallback,
canonical-preferred, sqlite3.Row, source_id_map, SOURCE_ID_FIELD unchanged).
Existing artist_source_lookup + artist_full_detail suites still green.
Migration state was scattered across PRAGMA-table_info guards, sentinel marker
tables (_genius_search_fix_applied, ...) and metadata-flag rows
(id_columns_migrated, ...), with no single source of truth and no schema
version — so a half-migrated DB was undetectable.
Add a non-gating backstop: a schema_migrations(name, applied_at) ledger plus a
_sync_migration_ledger pass (runs last in init) that back-fills the ledger from
the existing signals and stamps PRAGMA user_version. ADDITIVE only — existing
migrations keep their own idempotency gates; nothing decides whether a
migration runs based on the ledger or the version. New one-time migrations call
_record_migration (the genres migration already does).
Tests: tests/test_db_migration_ledger.py — table exists, user_version stamped,
record idempotent, genres recorded on fresh init, backfill from flag + marker,
absent signals not recorded.
artists.genres / albums.genres stored EITHER a JSON array (new writes) OR a
legacy comma-separated string (old writes), forcing every reader to
try-JSON-then-split. Add a marker-gated one-time migration
(_normalize_genres_to_json) that rewrites legacy rows to JSON in place,
mirroring the readers' exact parse (JSON list, else comma-split/strip/
drop-empties) so genre VALUES are unchanged — only the storage format.
Per-row diffed (already-canonical rows untouched, no churn) and non-fatal on
error, consistent with the other migrations. Readers still tolerate both
formats, so this breaks nothing; it just removes the dual-format debt.
Tests: tests/test_db_genres_json_normalization.py — CSV->JSON, JSON-unchanged,
whitespace/empties dropped, albums table, legacy-reader-equivalence,
idempotent re-run, marker set on fresh init.
amazon_artist_id is added to watchlist_artists via ALTER (music_database.py
~1732), but both table-rebuild migrations — the spotify_id-nullable fix
(_fix_watchlist_spotify_id_nullable, two CREATE variants) and the
profile-scoped UNIQUE rebuild — recreated the table from a hardcoded column
list that omitted amazon_artist_id. Because shared_cols filters new_cols
against the old table, the column and any stored Amazon artist IDs were
silently dropped on every init (fresh OR upgraded), so Amazon watchlist IDs
never persisted at all.
Fix: add amazon_artist_id to all three rebuild CREATE schemas, both rebuild
new_cols lists, and the base CREATE TABLE (so fresh installs are consistent
and don't rely on the ALTER). Purely additive, column-named inserts + Row
factory mean column position is irrelevant.
Tests (tests/test_db_watchlist_amazon_id_migration.py): drive the real
migrations via MusicDatabase() against a seeded pre-migration temp DB and
assert the column + data survive; differential-proven to FAIL pre-fix.
Reported by CubeComming: importing media keeps the track artist correct
(e.g. Billie Eilish) but changes the album-artist tag ("Albuminterpret") to
"Unknown Artist", breaking grouping in the media server.
Cause: in extract_source_metadata (core/metadata/source.py), album_artist is
seeded from the resolved track artist, then overridden by the album CONTEXT's
first artist. When the album lookup comes back unresolved, that first artist is
the literal "Unknown Artist" placeholder — which is truthy, so it clobbered the
real artist.
Fix: treat "Unknown Artist" (and empty) as a non-value — only let the album
context override the album_artist when it names a real artist. A genuine album
artist (e.g. "Various Artists") still overrides as before.
Tests: tests/metadata/test_album_artist_unknown.py — placeholder doesn't
clobber, real album artist still used, no-album-context falls back to track
artist, empty doesn't clobber. (Pre-existing test_album_mbid_cache.py failures
are an unrelated sandbox DB disk-I/O issue.)
Follow-up to the poll fix, covering the two things that blocked a
successful end-to-end album import once the poll itself stopped
freezing:
1. Staging dir permissions
The album-bundle private staging path defaults to
'storage/album_bundle_staging' -> /app/storage, but /app/storage was
never created or chowned by the image (unlike /app/Staging,
/app/Transfer, etc.), and /app is root-owned. The copy failed with
"[Errno 13] Permission denied: 'storage'" under the non-root soulsync
UID. Added /app/storage to the Dockerfile build-time mkdir+chown and
the entrypoint PUID/PGID chown, exactly like the sibling runtime dirs.
2. Client->local path resolution
Usenet/torrent clients report save paths from inside THEIR OWN
container (e.g. SAB '/data/downloads/music/<album>'); SoulSync often
mounts the same files at a different point ('/app/downloads/<album>').
Feeding the client path straight to the audio walker yields
"No audio files found" though the files are physically present.
New resolve_reported_save_path():
a. use the reported path as-is if readable (mirrored mounts),
b. apply explicit download_source.usenet_path_mappings
({from,to}, Sonarr/Radarr-style) for non-shared layouts,
c. basename fallback under SoulSync's own download roots —
zero-config for the standard shared-volume arr setup.
Wired into both call sites in usenet.py AND torrent.py
(download_album_to_staging + _finalize_download), logging any
translation and including the resolved path in the no-audio error.
Tests: resolver verbatim / explicit-mapping / basename-fallback /
priority / not-found / empty / mapping-miss-then-basename. ruff +
compileall + pytest green (645 in the download suites).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The earlier #721 fix tolerated a ~10s "completed but no save_path"
window, but the real production stall sits upstream of that: SABnzbd
removes a finished download from the queue and runs par2 verify /
repair / unpack *in History*, exposing the live stage in the slot
`status` ('Verifying' / 'Repairing' / 'Extracting' / 'Moving' / ...)
with `storage` empty until the final move. `_parse_history_slot` mapped
EVERY non-'Failed' status to 'completed', so a still-extracting 1.7 GB
FLAC album looked "completed with no save_path" the instant download hit
100%. The poll burned its completed-no-path budget mid-PP and bailed,
freezing the UI on the last download emit (the stuck-at-99%/100%
signature). SAB then finished fine — which is why the job shows
Completed in History but SoulSync never staged it.
Root fix
- `_parse_history_slot` routes `status` through `_map_state`, so PP
stages stay NON-terminal: the poll keeps waiting (as 'downloading')
for as long as post-processing takes and only a real 'Completed'
flips to terminal success. `save_path` is trusted only on true
completion (mid-PP path fields may point at the incomplete dir).
Supporting / defensive
- `UsenetStatus.incomplete_path`: surfaced separately from save_path
(SAB `incomplete_path`) and used by the poll loops as a LAST RESORT
after the completed-no-path window, to recover the case where
`storage` never lands but the files are physically on disk.
- `poll_album_download`: dedicated, configurable completed-no-path
window (~120s via `download_source.album_bundle_completed_no_path_seconds`)
decoupled from the ~10s transient-miss window; incomplete_path
fallback; a 30s heartbeat log so the previously-silent poll loop is
diagnosable.
- `usenet.py` `_download_thread`: per-track parity — it was erroring
immediately on the first completed-no-path read.
- `album_bundle_dispatch.py` / `status.py` / `monitor.py`: use the
project `get_logger` so download-flow logs land in app.log under the
`soulsync.*` namespace (they were console-only before, which hid the
`[Album Bundle] flow failed` line during triage).
Tests
- PP-history state mapping; end-to-end Hunky Dory PP regression
(download -> Verifying/Extracting in History past both budgets ->
Completed+storage -> success); completed-no-path window +
incomplete_path fallback; per-track thread parity. ruff + compileall +
pytest all green (the only local failures are environmental: missing
tzdata + local tools/ffmpeg.exe, neither present on CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final cluster: the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles) ->
core.discovery.endpoints.save_bubble_snapshot(...), wired via
_save_source_bubble_snapshot. All four validate a payload key, persist via
db.save_bubble_snapshot(kind, items, profile_id=...), and return a count +
timestamp; they differ only by:
- payload_key ('downloads' for discover, 'bubbles' for the rest) + its
no_data_error message.
- snapshot_kind, success_noun, and the info/except log subject + noun
("downloads"/"artists"/"albums/tracks"/"charts").
get_database / get_current_profile_id injected; get_json (request.json) invoked
inside the try, preserving the original 400/500 behavior incl. traceback dump.
Tests: +5 (missing key 400, None body 400, happy path with kind/profile/count/
timestamp, discover_downloads variant, exception -> 500). Full discovery suite:
210 passed.
web_server.py: -98 lines.
Ninth cluster: update_<source>_playlist_phase for the five sources sharing the
identical validation + full-message response (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube) -> core.discovery.endpoints.update_playlist_phase(...),
wired via _update_source_playlist_phase + the _PHASE_LIST/_PHASE_LIST_YT
constants.
Per-source params:
- valid_phases — YouTube additionally allows 'parsed'.
- apply_extra_fields — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body; Tidal and
YouTube do NOT, so they pass False (kept strictly 1:1 — the generic won't
apply those keys for them even if a caller sent them).
- not_found_message / error_label; get_json invoked inside the try.
NOT folded in: iTunes-Link — uses data.get('phase') (no "Phase not provided"
400) and returns a no-message payload.
Tests: +7 (404, missing-phase 400, invalid 400, happy path with extra-fields
suppressed, extra-fields applied when enabled, YouTube 'parsed' allowed,
exception -> 500). Full discovery suite: 205 passed.
web_server.py: -123 lines.
Eighth cluster, the heavyweights (~110 lines each). The fix-modal
update_<source>_discovery_match for the four sources with the identical
structure (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.update_discovery_match(...), wired via
_update_source_discovery_match. Applies the user-selected Spotify track to the
discovery result (status/artist/album/duration/spotify_data/match-count) and
writes the manual fix to the discovery cache.
Per-source pieces are params:
- source_log_label / error_label.
- original_track_key ('tidal_track' / 'deezer_track' / ...).
- original_artist_getter: Tidal handles string-or-object artists
(first_artist_str_or_obj); the rest assume strings (first_artist_plain).
- web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) injected.
- get_json passed as a callable and invoked INSIDE the try, preserving the
original's "request.get_json() inside try" behavior (malformed body -> 500).
NOT folded in (genuinely divergent): iTunes-Link (saves spotify_data directly
via a different cache signature), YouTube (multi-key original_track fallback),
ListenBrainz (entirely different unmatch-capable structure, no cache write),
Beatport.
Tests: +9 (extractors; 400/404/400 guards; full happy path with result
mutation + duration formatting + match-count + cache-save args; no-increment
when already found; cache error swallowed; get_json raise -> 500). Full
discovery suite: 198 passed.
web_server.py: -400 lines.
Seventh cluster: start_<source>_sync for the five sources with the identical
flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube) ->
core.discovery.endpoints.start_sync(...), wired via _start_source_sync.
Validates phase, converts discovery results, seeds sync state, posts a
"... Sync Started" activity item, and submits to the sync executor. Per-source
pieces are params:
- sync_id_prefix (f"{prefix}_{key}"), not_found/not_ready messages, convert_fn.
- name/image accessors: Tidal reads an object (playlist_name_obj/
playlist_image_obj), the rest a dict (playlist_name_strict/playlist_image_dict).
- activity_label vs error_label DIFFER for Spotify-Public ("Spotify Link
Sync Started" activity, "Spotify Public" logs).
- submit_sync_task glue (_submit_sync_task) closes over sync_executor /
_run_sync_task / get_current_profile_id so the helper stays global-free.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task WITHOUT a playlist_image_url arg), Beatport (extra debug logging, chart).
Tests: +6 (404, not-ready 400, no-matches 400, full happy path with
state/sync-infra/submit/activity assertions, resync phases allowed,
exception -> 500). Full discovery suite: 189 passed.
web_server.py: -172 lines.
Sixth cluster: the bulk-hydration get_<source>_playlist_states endpoints for
the five sources that build the identical per-entry dict + {"states": [...]}
shape (Tidal, Deezer, Qobuz, Spotify-Public, iTunes-Link) ->
core.discovery.endpoints.get_playlist_states(states, *, error_label,
info_log_label=None), wired via _get_source_playlist_states.
iTunes-Link is the only one of the five without the "Returning N stored ..."
info log, so info_log_label is optional (iTunes passes None to suppress it).
NOT folded in: the YouTube/ListenBrainz get_all_*_playlists endpoints. They
return {"playlists": [...]} (different key) with a different field set
(url / created_at / playlist, no discovery_results) and filter out
mirrored_/profile-scoped entries — genuinely divergent, kept as-is.
Tests: +4 (list build + last_accessed bump + exact shape, empty, optional ids
default None, missing-required-field -> 500). Full discovery suite: 183 passed.
web_server.py: -116 lines.
Fifth cluster: reset_<source>_playlist for the four sources with byte-
identical bodies (Tidal, Deezer, Qobuz, Spotify-Public) ->
core.discovery.endpoints.reset_playlist(states, key, *, label,
not_found_message), wired via _reset_source_playlist. Resets phase/status to
'fresh', clears discovery/sync fields, cancels any discovery_future, and
preserves the original playlist payload.
Left with their own bodies (genuinely divergent):
- YouTube: status -> 'parsed' (not 'fresh'), no download_process_id, logs the
playlist name, "reset to fresh state".
- ListenBrainz: status -> 'cached', logs playlist title, returns
{"success": True, "phase": "fresh"} (different payload), _lb_state_key.
- iTunes-Link: state.update(...), no info log, "iTunes Link reset to fresh
phase".
Tests: +4 (404, full clear + playlist preserved + future cancelled, no-future
path, exception -> 500). Full discovery suite: 179 passed.
web_server.py: -100 lines.
Fourth cluster: get_<source>_discovery_status (all eight sources, Beatport
included) -> core.discovery.endpoints.get_discovery_status(states, key, *,
not_found_message, error_label), wired via _get_source_discovery_status.
Unlike sync-status, the discovery-status response shape is byte-identical
across every source (phase/status/progress/spotify_matches/spotify_total/
results/complete), so Beatport folds in here too. Only the 404 string
("... discovery not found" vs "... playlist not found" vs "Beatport chart
not found") and the except-log label vary. ListenBrainz key via _lb_state_key.
NOT touched this cluster: get_*_playlist_state (the sibling endpoints).
Those genuinely diverge per source — different id-key name (playlist_id /
url_hash / playlist_mbid), presence of url / created_at / download_process_id,
Tidal's playlist.__dict__ serialization, and YouTube's strict (non-.get)
field access. Folding them would need a flag pile that wouldn't be a clean
1:1, so they keep their own bodies.
Tests: +4 (404, full response + last_accessed bump, complete=False when not
'discovered', missing-field -> 500). Full discovery suite: 175 passed.
web_server.py: -155 lines.
Third cluster: the get_<source>_sync_status routes (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz) -> core.discovery.
endpoints.get_sync_status(...), wired via _get_source_sync_status glue.
This cluster carries the real per-source quirks, all captured 1:1 as params:
- not_found_message (iTunes-Link uses "iTunes Link not found").
- error_label vs activity_subject — these DIFFER for Spotify-Public: the
activity feed says "Spotify Link playlist ..." while the except log says
"Error getting Spotify Public sync status".
- playlist-name accessor, three styles lifted verbatim as named helpers:
playlist_name_attr_or_unknown (Tidal: object .name), playlist_name_strict
(Deezer/Qobuz/Spotify-Public/iTunes: state['playlist']['name'], can raise),
playlist_name_safe (YouTube/ListenBrainz: .get default). The strict getter
preserves the original's behavior of raising -> 500 AFTER phase/sync_progress
were already mutated.
- ListenBrainz key via _lb_state_key (caller-resolved).
Beatport stays separate (different payload: status not sync_status, sync_id,
no lock, chart key).
Tests: +9 (3 name accessors incl. raise/fallback semantics; status 404s,
running-no-mutation, finished+activity, error+revert+activity, and strict-
getter-missing -> 500 after partial mutation). Full discovery suite: 171 passed.
web_server.py: -244 lines.
First cluster of the per-source playlist-discovery deduplication. The
convert_<source>_results_to_spotify_tracks functions (Tidal, Deezer, Qobuz,
Spotify-Public, YouTube, ListenBrainz) plus the already-generic
_convert_link_results_to_spotify_tracks were byte-identical apart from the
source label used in their log line.
Lift the shared body into core/discovery/endpoints.py as
convert_results_to_spotify_tracks(results, source_label); the 7 web_server
functions become 1-line delegations (names/signatures unchanged, so all
callers and behavior are identical — 1:1).
Beatport is intentionally NOT folded in: its converter coerces artist
objects to strings and emits a different track shape (source field, album
dict), so it keeps its own implementation.
Tests: tests/discovery/test_discovery_endpoints.py (12) pin both input
shapes (manual spotify_data / auto spotify_track+found), optional
track/disc numbers, falsy-0 omission, field defaults, skip-on-neither,
order preservation, if/elif precedence, empty input.
web_server.py: -209 lines. Full discovery suite: 151 passed.
Follow-up to the album-art resolution fix. That change upgraded MusicBrainz
Cover Art Archive thumbnails (/front-250) to the bare /front original — but
/front redirects to archive.org, which is unreliable: probing release-group
covers showed intermittent HTTP 500s (same URL 500s one second, serves the
next) and multi-MB originals (2.9 MB seen). The result was the user-reported
flakiness: cover art that "sometimes works, sometimes shows nothing", and a
huge image embedded into every track when it did work.
The sized thumbnails (/front-250, -500, -1200) are served by CAA's own CDN,
not the archive.org redirect — which is why /front-250 (240p) was always
reliable. Upgrade to /front-1200 instead: 1200x1200 is a massive jump from
240p, reliably CDN-served, and a sane ~40 KB instead of multi-MB.
Applied in all three CAA spots for consistency: the _upgrade_art_url helper
(embed + cover.jpg paths) and both prefer_caa ("CCA") blocks, which fetched
the bare /front directly with no fallback — so CCA-on users hit the same
flakiness. _fetch_art_bytes still falls back to the original /front-250 if
/front-1200 is ever refused.
Tests updated to assert the 1200px target, idempotency, and that the bare
/front original is intentionally left untouched.
User report: embedded album art came out ~600x600 while the cover.jpg in
the folder was high-res. The cover.jpg path upgraded the source CDN URL
to its highest resolution, but the tag-embed path fetched the raw URL —
so iTunes art embedded at its 600x600 default, Spotify at 640, Deezer at
1000. The "Write Tags to File" retag path had the same gap (Deezer-only
upgrade), and MusicBrainz art was worse still: every Cover Art Archive
URL is built as the /front-250 thumbnail, so MB-sourced downloads
embedded 250x250.
Factor the resolution upgrade + fetch into two shared helpers in
core/metadata/artwork.py and route every art path through them:
_upgrade_art_url(url) — bump to the source's highest resolution:
- Spotify (i.scdn.co) -> original master (~2000px+)
- iTunes (mzstatic.com) -> 3000x3000
- Deezer (dzcdn) -> 1900x1900
- Cover Art Archive -> /front original (was /front-250)
_fetch_art_bytes(url) — upgrade, fetch, and fall back once to the
original size if the CDN refuses the larger one (non-regressive).
Now consistent across: embed-into-tags (post-process), folder cover.jpg
(post-process), and the enhanced-library "Write Tags to File" retag flow.
The YouTube path already upgraded via Album.from_spotify_album, unchanged.
De-duplicates the per-source upgrade code that was copied across sites
and drops the now-unused urllib import from tag_writer.
Not covered (follow-up): Last.fm / Amazon / Tidal / Qobuz have no
explicit upgrade yet — some already serve full-res, others may hand over
a capped size that passes through unchanged.
Tests: new tests/metadata/test_artwork_resolution.py pins every upgrade
(Spotify 300/640->master, iTunes 100/600->3000, Deezer->1900, CAA
thumbnail->original, unrecognized/empty unchanged) and the fetch
fallback. Updated the two tag_writer fallback tests to patch the network
at its new home in artwork.
Opening a library artist from a non-library search result (e.g. a
MusicBrainz hit) leaves the artist-detail page holding the source ID —
the MBID — not the integer library PK. The standard /api/artist-detail
route resolves that via find_library_artist_for_source, but the
enhanced-view (`/api/library/artist/<id>/enhanced`) and quality-analysis
endpoints call get_artist_full_detail directly with whatever ID the page
holds. Its lookup was `WHERE id = ?` only, so it 404'd ("Artist with ID
<mbid> not found") and the enhanced view failed to load.
When the direct PK lookup misses, fall back to matching any per-service
ID column, reusing SOURCE_ID_FIELD as the single source of truth so the
resolution covers every source (MusicBrainz, Spotify, Deezer, iTunes,
Discogs, Hydrabase, Amazon), not just MusicBrainz.
Adds 4 isolated DB-method tests: direct PK still works, resolves by
MBID, resolves by Spotify ID, and unknown IDs still 404.
Artist-detail discography from MusicBrainz fetched releases via the
artist lookup (`/artist/<mbid>?inc=release-groups`), which MusicBrainz
hard-caps at 25 embedded release-groups and which ignores the `limit`
param entirely. Prolific artists had ~85% of their catalogue silently
dropped — Kendrick Lamar has 167 release-groups on the site but only the
first 25 ever reached SoulSync. Reported by Sokhi: "a lot of albums are
missing when searching vs what's showing on the site."
Switch `get_artist_albums` to walk the paginated browse endpoint
(`/release-group?artist=<mbid>`, offset loop) — the same pattern the
basic-search path already uses — fetching the full catalogue up to the
caller's limit. No type filter and no studio-only filter here: the
artist-detail page wants every primary/secondary type so its tabs mirror
musicbrainz.org. Verified live: now returns all 167 for Kendrick.
Adds 7 tests covering pagination past the cap, offset advance,
short-page stop, limit cap, cross-page dedup, type->bucket mapping, and
a regression pin asserting the capped inc=release-groups lookup is no
longer the discography source.
Two things in this commit. Functional download / matched-download
behaviour is untouched — same JS handlers, same routes for the
download actions, same album-expand interaction.
VISUAL REDESIGN
- Glass search-bar card with accent radial wash + focus ring + pill
primary search button
- Source chip row above the search bar (see below)
- Always-visible compact filter pill row (Type / Format / Sort) —
pills carry both ``bs-filter-pill`` (new visual) and ``filter-btn``
(legacy class for ``resetFilters`` + ``applyFiltersAndSort`` in
wishlist-tools.js to keep working)
- Accent-tinted status pill matching the dashboard / auto-sync look
- Album result cards: glass card with accent left-edge stripe,
52px brand-tinted cover icon, chevron expand indicator, pill
action buttons (Download / Matched Album), accent glow on hover
- Track result cards: glass row with accent stripe, 44px icon,
pill action buttons (Stream / Download / Matched Download)
- Multi-disc separators inside expanded album track lists styled
with the accent treatment
- Responsive: action button columns stack vertically below 900px
New CSS lives in a self-contained ``webui/static/basic-search-v2.css``
sheet linked from index.html. Selectors are scoped to
``#basic-search-section`` for any class that already exists in
style.css (``.album-result-card``, ``.album-icon``, ``.track-*``,
etc.); the new ``bs-*`` prefixed classes for the search bar /
filters / source row / status are unscoped because they only exist
in the new markup. ``!important`` is used on the card-level rules
to defeat the original unscoped ``.album-result-card`` etc. rules
in style.css that would otherwise leak heavyweight padding /
box-shadow / 56px icon styles into the new design.
Also removed ``overflow: hidden`` from the original
``.album-result-card`` and ``.track-result-card`` rules in style.css
— those two classes only render in ``downloads.js`` basic search
results (verified via grep, two render sites only), so the
removal can't impact any other UI.
SOURCE PICKER (hybrid mode)
- New ``GET /api/search/sources`` endpoint returns the list of
active sources from the orchestrator's chain (or the single
active source in single-source mode).
- Frontend renders a chip row above the search bar. Click a chip
to target that source for the next search; the chip's brand
accent fills.
- In single-source mode the lone chip is rendered as a dashed-
border label so the user always knows what they're searching
but can't accidentally try to switch to sources that aren't
configured.
- ``/api/search`` accepts an optional ``source`` body param. When
set, ``core/search/basic.py:run_basic_search`` resolves the
client directly via ``orchestrator.client(source)`` and calls
its ``.search()`` instead of going through the hybrid chain.
- Backwards compatible: omitting ``source`` falls through to the
original ``orchestrator.search()`` call exactly as before.
Unknown source names also fall back to the default — typo
protection.
TESTS (5 new + 6 pre-existing = 11 total in test_search_basic.py)
- source param routes to specific client, NOT orchestrator chain
- no source param preserves original orchestrator-default behaviour
- unknown source name falls back to orchestrator default
- ``run_basic_soulseek_search`` backwards-compat alias preserved
- source-targeted path serialises albums + tracks correctly
101 search-suite tests pass.
Reporter @Sokhii: downloading the Mushoku Tensei Original
Soundtrack II via Apple Music metadata + Tidal download
produced duplicate library entries — same audio file landed
under multiple track positions in the album view.
Root cause (verified by direct probe + isolated repro):
``MusicMatchingEngine.normalize_string`` correctly skipped
unidecode for CJK text (kanji→pinyin would have produced
gibberish — see the inline comment at line 74-76), but then
ran ``re.sub(r'[^a-z0-9\s$]', '', text)`` which stripped EVERY
CJK character. Every Japanese title normalised to ``''``.
``similarity_score`` has an early-out guard
if not str1 or not str2: return 0.0
so EVERY CJK-vs-CJK title comparison returned 0.000.
Downstream effect: the matcher fell back to duration+artist
alone. For an OST album with 24 tracks all by the same artist
with similar durations, multiple iTunes track queries landed
on the SAME Tidal candidate. SoulSync wrote each download to
a different output filename (per the iTunes track position),
so on disk there were N copies of the same audio under
different track numbers. The user's library showed 34 entries
for an album with 24 actual tracks.
Probed iTunes album 1753240110 directly — 24 distinct tracks,
zero (disc, track_number) collisions, both US + JP storefronts.
So the duplicate origin was definitely downstream of metadata
fetch.
Fix: when CJK is detected upstream, the alphanumeric-strip step
also preserves CJK Unified Ideographs + radicals
(⺀-鿿), Hiragana + Katakana (-ヿ), Halfwidth
/ Fullwidth forms (-), and Hangul syllables
(가-). CJK titles now produce a comparable normalised
form instead of an empty string. ``similarity_score`` works as
intended:
'命の灯火' vs '命の灯火' → 1.000 (was 0.000)
'命の灯火' vs '無職転生' → 0.000 (was 0.000, but now from
actual char comparison
not from the empty-string
guard)
Latin-only normalisation is completely unchanged. ``has_cjk``
is False for Latin input, so both the CJK-lowercase branch AND
the new CJK-preserve strip branch are skipped — Latin titles
go through the original unidecode + lowercase + strip path
verbatim. Tested via 4 regression tests that pin the Latin
baseline (simple, unidecode target, $-preservation, identical
+ different similarity scores).
16 new unit tests in ``tests/test_matching_engine_cjk.py``:
- Kanji / Hiragana / Katakana / Hangul / Chinese all survive
- CJK-only strip still removes Latin punctuation in the
CJK branch
- Mixed Latin + CJK lowercases the Latin half
- Identical CJK titles → 1.0
- Disjoint CJK titles → near 0
- Partially overlapping CJK titles → midrange
- CJK doesn't falsely match unrelated Latin
- 4 Latin-baseline regression pins
- Real-world Mushoku Tensei OST scenario
371 text + imports + new CJK tests pass after the fix.
Follow-up to the 2.6.3 queue→history handoff fix (#706). User
@IamGroot60 reported in #721 that on 2.6.3 the bundle still gets
stuck mid-flight: SoulSync UI sits on "Usenet downloading release
61%" forever, SAB History shows the job as Completed 2+ minutes
ago, files are physically present in the slskd downloads folder
but never copied into ``storage/album_bundle_staging/<batch>/``.
Root cause: a second-stage gap in the SAB pipeline. SAB flips a
job's ``status`` to ``Completed`` in History as soon as par2 +
unrar finish, but its post-processing pipeline writes the final
``storage`` field a few seconds LATER (the move-to-final step).
``poll_album_download`` saw the first ``Completed`` read with
``save_path=None`` and bailed:
if status.state in complete_states:
return last_save_path # ← None at this point
``download_album_to_staging`` got ``save_path=None``, set
``result['error']`` and returned. The bundle was marked failed but
the LAST progress emit before the failure was ``downloading
progress=0.61``, so the UI froze on "61%" — the terminal ``failed``
emit never registered on the user's screen because the renderer
holds the last-known progress.
Fix
- ``poll_album_download`` now tracks a separate transient counter
for "complete state seen, save_path not yet set." Up to
``transient_miss_threshold`` (default 5) consecutive reads in
that state are tolerated before the poll bails. SAB writes the
``storage`` field within 2-10 seconds of the History flip in
practice — the default 5 × 2s = 10s window covers it.
- When save_path eventually lands, return it normally.
- When the threshold is exhausted with save_path still empty,
emit terminal ``failed`` with an explicit message pointing at
the missing save_path field — no more 6-hour silent spin.
- Earlier ``downloading`` reads with a non-empty ``save_path``
(qBit / Transmission set this from the start of the download)
remain "sticky" — if the eventual ``completed`` read has empty
save_path, the cached one applies. So torrent flows aren't
affected by the retry path.
SAB adapter (``_parse_history_slot``)
- Widened the save_path field fallback chain:
storage → path → download_path → dirname → incomplete_path
Covers SAB version differences (older builds populated ``path``)
and forks that expose ``download_path`` or ``dirname``.
``incomplete_path`` is the last resort — SAB's in-progress dir
before the final move — so the bundle plugin at least has a
path to scan when nothing else lands.
- Whitespace-only values are skipped.
- Loud debug log when none of the known fields land — users on
SAB versions / forks with novel field names need to see this in
logs so we can grow ``_HISTORY_SAVE_PATH_KEYS``.
Tests
- ``test_album_bundle.py`` (3 new):
- tolerates_completed_with_late_save_path_arrival — the #721
scenario; first Completed read has no save_path, third has
it; poll returns the path normally
- gives_up_when_completed_with_no_save_path_persists — past
the threshold the poll fails loudly instead of silent-spinning
- uses_save_path_from_earlier_downloading_emit_if_completed_lacks_one
— sticky save_path keeps torrent flows working
- ``test_usenet_client_adapters.py`` (6 new):
- falls back to ``path`` when ``storage`` empty
- falls back to ``download_path``
- prefers ``storage`` when multiple fields present
- returns ``None`` when all fields empty (the #721 gap window)
- ignores whitespace-only values
- uses ``incomplete_path`` as last resort
132 album-bundle + usenet tests pass.
Branch is on dev parented at 2.6.3 — user @IamGroot60 offered
to test on dev, so this is a candidate cherry-pick for either
a 2.6.4 hotfix or merge straight into dev for the next release.
Two related leaks in ``storage/album_bundle_staging/<batch_id>/``:
1. **Soulseek bundle cleanup was excluded.** The per-batch cleanup
at the end of a bundle download gated on:
(album_bundle_source or '').lower() in ('torrent', 'usenet')
The comment justified it as "slskd keeps its own completed
folders" — but the Soulseek bundle path ALSO copies completed
files into the private staging dir (``soulseek_client.py:1599``,
``copy_audio_files_atomically(completed, Path(staging_dir))``)
for the per-track workers to claim. Those copies persisted
forever; long-running installs accumulated stale GB. Extended
the cleanup gate's allow-list to include ``soulseek`` so the
per-batch dir is removed on bundle completion — same code path
that already worked for torrent / usenet.
2. **No sweep for orphan dirs.** Any leftover ``<batch_id>``
subdir from a previous-session crash, an errored batch, or a
pre-fix Soulseek bundle stayed on disk forever. Added
``sweep_orphan_album_bundle_staging(staging_root, active_batch_ids)``
that runs ONCE at server startup, before any batch can register
a staging dir. Removes every ``<batch_id>``-shaped subdir
whose id isn't in the active set. Safe by construction:
- Only touches subdirs of the configured staging root.
- Name-shape check (``entry.name == _safe_batch_dirname(entry.name)``)
rejects hand-placed dirs like ``.git`` or stray docs.
- ``shutil.rmtree`` errors log + continue — sweep must not
crash app startup over a permission glitch.
- active_batch_ids normalised through ``_safe_batch_dirname``
so colon-bearing batch_ids match their on-disk form.
Wired into the web_server startup right after the stuck-flags
diagnostic so it fires before anything else touches batches.
Tests
- ``test_downloads_lifecycle.py`` gained one regression test
pinning that Soulseek bundles now have their staging dir
cleaned (sibling to the existing torrent test).
- ``test_album_bundle_staging_sweep.py`` (NEW, 11 tests)
covers: orphan removal with no actives, active dirs preserved,
special-char batch_id normalisation, no-op on missing /empty
/empty-string staging root, non-dir entries skipped, unsafe-
name dirs preserved (.git etc.), partial rmtree failure doesn't
abort the rest, listdir failure returns 0 cleanly, default
None active set, defensive against empty / None entries in
the active set.
488 downloads tests pass.
For users with an existing "clean up old files" automation pointed
at this dir: stop pointing it there if you want — the auto-cleanup
+ startup sweep cover it now. Or leave it as belt-and-suspenders
with a relaxed (1h+) mtime threshold so it can't race a mid-batch
download.
finished the release (#715)
Symptom (user @pavelcreates / @IamGroot60 on 2.6.2):
- Click Download on an album in the search modal
- slskd starts + completes every track of the release
- 22+ minutes after the last completed download, batch flips
to "failed" with no clear log line explaining why
- Per-track Soulseek downloads on the same machine were fine
Root cause: ``core/soulseek_client._resolve_downloaded_album_file``
probed three hard-coded candidate paths to locate each downloaded
file in the slskd download dir:
candidates = [
download_path / remote_filename,
download_path / basename,
download_path / *normalized_path_parts,
]
On the common slskd config ``directories.downloads.username = true``
slskd writes files at ``<download_dir>/<username>/<filename>`` —
none of the three candidates carry a username segment, so the
resolver returned None for every file even though the file was
physically present in a subdir one level deeper. ``_poll_album
_bundle_downloads`` saw 0 completed_paths, kept spinning, and
hit the master deadline (~30 min) before bailing the batch.
Why per-track worked: ``web_server._find_completed_file_robust``
already does a recursive walk-by-basename + path-confirm against
the remote directory components, so any layout slskd writes ends
up resolved. The bundle path didn't go through it.
Fix
- Lifted the robust finder into ``core/downloads/file_finder.py``
as a pure function ``find_completed_audio_file(download_dir,
api_filename, transfer_dir=None) -> (path, location)``. Zero
globals; recursive walk; handles slskd dedup suffix
``_<10+digit-timestamp>``, YouTube / Tidal ``id||title`` encoded
filenames, the AcoustID-quarantine subdir skip, basename
collisions disambiguated by remote-path components, and a
fuzzy-basename fallback above 0.85.
- ``_resolve_downloaded_album_file`` keeps the three-candidate
fast path (cheap probe for the slskd-flat default) but now
delegates to the new helper when none hit, instead of giving up.
- ``_poll_album_bundle_downloads`` tracks "slskd reports
Completed but local resolver returns None" per key. When every
remaining key has been in that state past a 45-second grace
window, the poll exits early with an explicit error pointing at
the likely ``soulseek.download_path`` mismatch instead of
silently spinning until the master deadline.
- ``web_server._find_completed_file_robust`` becomes a thin
delegate so both callers share one finder. Legacy inline impl
kept as ``_find_completed_file_robust_legacy`` for reference;
to be removed next release.
- Fixed misleading ``"(0 tracks, quality=)"`` log on the preflight-
reuse path — was reading attrs off a None ``picked`` object.
Tests (17 new in tests/downloads/test_file_finder.py)
- Flat slskd layout
- Username-prefixed (the #715 case)
- Full remote tree preserved
- Deeply nested username + tree
- File genuinely missing returns None
- Basename collision disambiguated by remote dirs
- Single basename match wins regardless of dirs
- slskd dedup suffix match
- Short ``_<digits>`` (year) not treated as dedup
- AcoustID quarantine subdir skipped
- YouTube / Tidal ``id||title`` encoded filenames
- transfer_dir fallback
- Both dirs miss → (None, None)
- Non-audio files ignored
- Empty api_filename
- Fuzzy match on punctuation variant
- Fuzzy rejects below threshold
475 downloads tests pass after the lift.
Pipeline-driven Auto-Sync runs against any ListenBrainz playlist
(Weekly Jams, Weekly Exploration, Top Discoveries, etc.) would sit
on ``Refreshing: "<name>"`` with no UI updates for 5-7 minutes
before the pipeline progressed. Two real bugs stacked:
1. **Double discovery.** The refresh handler called
``_maybe_discover`` (matching engine, per-track Spotify/iTunes/
Deezer matches) inline for any source returning
``needs_discovery=True`` tracks. Phase 2 of the pipeline then
ran the SAME matching engine via ``run_playlist_discovery_worker``
on the same tracks. The refresh-side run blocked the loop with
zero progress emission; Phase 2's already has the timed
progress-poll pattern. So LB tracks discovered twice, the first
time silently.
Pipeline now sets ``skip_discovery=True`` on its refresh config.
The handler honors the flag and lets Phase 2 handle discovery
end-to-end. Standalone callers (Sync-page tab, registration
action) leave the flag unset so they still get matched_data
on refresh.
2. **No targeted LB refresh.** The LB adapter's ``refresh_playlist``
called ``manager.update_all_playlists()`` — the only refresh
entry-point the manager exposed — which re-pulls every cached
LB playlist's details from the API (~12+ round-trips) even
when only one playlist needed refreshing. Wasteful;
tax-on-everyone for one-playlist work.
Added ``LBManager.refresh_playlist(mbid)`` — reads the cached
playlist_type, fetches just that playlist's details, runs the
normal ``_update_playlist`` upsert path. Defaults type to
``user`` for un-cached mbids so new-playlist discovery still
works. Skips ``_cleanup_old_playlists`` and
``_ensure_rolling_mirrors_from_cache`` (wasted work for a
single-playlist refresh).
Also: killed a silent ``except Exception: pass`` in the LB
adapter's old refresh wrapper that was masking every LB API
failure as a stale-cache hit. Refresh errors now log with full
traceback at warning level and propagate ``None`` so the outer
handler at ``refresh_mirrored.py:104`` counts the error and
surfaces it to the run-history error tally.
Pinned with 12 new unit tests across:
- ``tests/test_listenbrainz_manager.py`` (8): targeted refresh
happy path, unauthenticated guard, empty-mbid guard, upstream
``None`` return, default playlist_type for unknown mbid,
exception propagation, cost guard skipping cleanup, skipped-
when-unchanged signal
- ``tests/test_playlist_sources_adapters.py`` (3): adapter uses
targeted call (not legacy), adapter returns ``None`` on manager
error (not silent swallow), adapter resolves synthetic series
ids before calling the manager
- ``tests/automation/test_handlers_playlist.py`` (1):
skip_discovery flag bypasses ``_maybe_discover`` end-to-end
Residual per-track wishlist downloads (single tracks from different
albums, below the album-bundle threshold) were producing folders
without a year subfolder whenever the wishlist row carried a stale
``track_number=1`` from an older payload default.
Why: ``core/downloads/candidates.py`` had a single API-fetch branch
that served two concerns — resolving the track position AND
hydrating the lean ``spotify_album_context`` (release_date /
total_tracks / cover image) — gated entirely on track_number being
unresolved. When the wishlist row's ``track_number`` happened to
be 1 (a poisoned default rather than a real value), the gate
short-circuited and the album hydration the same call would have
done was skipped. Deezer-sourced discovery matches don't ship
release_date in their search-result album shape, so without the
backfill the folder lost its year.
The two concerns split:
- track_number resolution keeps its track_info → track object →
API precedence chain. track_info defaults still win.
- album hydration runs whenever release_date or total_tracks are
missing, independent of where (or whether) track_number was
resolved.
The single API round-trip still serves both — the cost contract
is preserved. The side-effect coupling is gone.
Lifted into ``core/downloads/track_metadata_backfill.py``
(``hydrate_download_metadata``) so the precedence chain is pinned
in isolation. 24 unit tests cover the precedence chain, the
poisoned-tn=1 regression case, defensive non-dict/None inputs,
the cost guard (API called at most once per invocation), and
disc_number resolution.
Also lands the upstream piece: ``core/wishlist/routes.py:_build_track_data``
no longer defaults ``track_number=1`` / ``disc_number=1`` /
``total_tracks=1`` / ``release_date=''`` when the library-modal add
payload omits them. Missing values now flow through as ``None`` so
the downstream pipeline can detect-and-recover instead of locking
to a fake position.
Real-world regression triggered by the album-bundle work earlier in
2.6.3. Tracks with full Spotify metadata were importing as
``01 - <title>`` under ``Artist - Album/`` (no year), even when the
source filename carried the correct track number and Spotify's
release_date was available.
Investigation via DB inspection of stored wishlist rows:
```
"Never Gonna Give You Up" → track_number=None, release_date=""
"idfc" → track_number=1, release_date=""
"No Sleep Till Brooklyn" → track_number=1, release_date=""
```
Source-of-truth Spotify metadata had release_date AND real track
positions, but the wishlist row was poisoned. Three regressions
compounded the loss:
**Fix A — ``track_object_to_dict`` (``core/wishlist/payloads.py:295``)
preserved only album.name during Track→dict conversion.**
Pre-fix:
```python
album_name = "Unknown Album"
if hasattr(track_object, "album") and track_object.album:
if hasattr(track_object.album, "name"):
album_name = track_object.album.name
else:
album_name = str(track_object.album)
result = {
...
"album": {"name": album_name}, # ← release_date / images / etc. all dropped
...
}
```
When a wishlist payload arrived as a Track dataclass instead of a
raw spotify_data dict, the Track→dict conversion stripped
release_date, images, album_type, total_tracks, id, and album-level
artists. Every wishlist row added through this path landed in the
DB with ``album={'name': X}`` only.
Post-fix: three branches handle the three album shapes
- ``album_attr`` is a dict → ``dict(album_attr)`` preserves every key
- ``album_attr`` is a sub-object → pull all common Album-dataclass
attrs (id, release_date, album_type, total_tracks, images, ...)
- ``album_attr`` is a bare string → build a dict from the track
object's adjacent attrs (release_date, album_id, album_type, ...)
and surface ``image_url`` as ``album.images``
**Fix B — ``core/discovery/playlist.py:309`` only added
``track_number`` / ``disc_number`` keys when truthy.**
Pre-fix:
```python
matched_data = { 'id': ..., 'name': ..., ... } # no track_number / disc_number
if track_number:
matched_data['track_number'] = track_number
if disc_number:
matched_data['disc_number'] = disc_number
```
Deezer-sourced matches always hit this branch with ``track_number=None``
because the cache enrichment at line 304 reads ``_raw.get('track_number')``
literally, but Deezer's raw shape uses ``track_position``. So the key
was omitted from ``matched_data``, downstream consumers couldn't
distinguish "missing key" from "value is 1", and the chain silently
filled 1.
Post-fix: keys are ALWAYS present (None when unknown). Also adds a
``best_match.track_number`` fallback so the Track-dataclass-mapped
value (which DOES include ``track_position``→``track_number``
mapping) gets used when the cache lookup misses.
**Fix C — Pipeline only consulted ``album_info.track_number`` before
falling to the filename (``core/imports/pipeline.py:645``).**
VA-collection source files like ``417 Fountains of Wayne - Stacys
Mom.flac`` have a leading playlist-position number that isn't the
album track number. The previous chain (album_info → filename →
floor-1) couldn't recover the real position because the filename
extractor either returned 417 (wrong) or None (caught by the floor).
But the wishlist payload's ``track_info.spotify_data.track_number``
HAD the right answer all along — Spotify says Stacy's Mom is track
3 on Welcome Interstate Managers.
Post-fix: resolution chain extracted into ``core/imports/track_number.py:resolve_track_number``
as a pure function:
1. ``album_info.track_number`` (album-bundle dispatch authoritative)
2. ``track_info.track_number`` (per-track flow payload)
3. ``track_info.spotify_data.track_number`` (nested fallback)
4. ``extract_explicit_track_number(file_path)`` (filename, returns
0 when no numeric prefix — vs the default helper that returns 1)
5. Caller (pipeline) applies the final >=1 floor
Each step coerces to a positive int or falls through to the next.
Pure function = unit-testable in isolation = single place to fix
the rule.
**Test coverage (37 new tests):**
- ``tests/wishlist/test_payloads.py`` (+4) — Track→dict conversion
preserves full album dict (dict / object / string album shapes) +
None-track-number stays None.
- ``tests/discovery/test_discovery_playlist.py`` (+2) — matched_data
always includes track_number/disc_number keys (None when unknown)
+ falls back to best_match attrs when cache misses.
- ``tests/imports/test_track_number_resolver.py`` (+16) — every
resolution-chain branch pinned: album_info-wins, track_info
fallback, spotify_data nested, JSON-string parsing, garbage-string
fall-through, zero / negative / non-numeric / string-numeric
coercion, filename fallback, explicit extractor vs default
extractor semantics, defensive None inputs, VA-collection
filename behaviour, all-sources-missing → None.
1571 wider-suite tests pass (wishlist + imports + discovery +
downloads + metadata). Ruff clean.
**Migration note:** existing wishlist rows that were saved under
the OLD ``track_object_to_dict`` (with stripped album metadata) still
have ``release_date=''`` in the DB blob. Those won't self-heal — the
next attempt loads from the poisoned blob. Users can remove + re-add
those tracks to refresh, or wait for the next sync run that
re-discovers them with full metadata. No automatic migration shipped
in this PR (scope creep — the forward path is fixed, backfill is a
separate concern).
PR 4 of 4 in the wishlist-album-bundle issue series. UI fix only —
zero behavior change.
User's 26-track wishlist run rendered all 26 sub-batches as
"Analyzing..." simultaneously. Pre-fix the rows were created with
``phase='analysis'`` BEFORE being submitted to ``missing_download_executor``
(max_workers=3 by default), so 23 batches sat in the executor queue
visually identical to the 3 actually running. Misled users into
thinking SoulSync was processing 26 in parallel; really only 3 ever
ran at once with the rest waiting their turn.
Fix:
- Wishlist auto-flow submission sites now create batch rows with
``phase='queued'``.
- The master worker (``core/downloads/master.py:328``) already flipped
phase to ``'analysis'`` as its first action on entry — that
transition becomes the real signal that the executor picked the
batch up.
- ``core/downloads/status.py`` surfaces ``analysis_progress`` for
the ``queued`` phase too so the UI has the track count to render
"Queued — N tracks" instead of an empty card.
- Frontend (``webui/static/pages-extra.js``, ``downloads.js``) renders
"Queued ⏳" for ``phase='queued'`` distinct from the spinner-laden
"Analyzing..." for ``phase='analysis'``.
Scope choices:
- Only the auto-wishlist submission sites flipped this PR
(``core/wishlist/processing.py:860`` album sub-batches +
``core/wishlist/processing.py:907`` residual). The manual-wishlist
sites at ``:451`` and ``:627`` use the same executor + worker, but
those create a caller-allocated batch_id that the frontend polls
immediately — wanted to verify the manual-poll path handles
``queued`` cleanly before flipping those. Trivial follow-up.
- Other submission sites in album_bundle_dispatch / web_server.py /
task_worker.py left untouched — they don't go through the
executor-queue pattern that causes this UI confusion.
Tests:
- Updated ``test_process_wishlist_automatically_creates_batch_for_matching_tracks``
to assert ``phase='queued'`` on creation (was ``'analysis'``); explanatory
comment names the executor-pool reason.
- New ``test_queued_phase_surfaces_analysis_progress_for_ui_count`` in
``tests/downloads/test_downloads_status.py`` pinning the new
``queued ⊂ analysis_progress`` rendering contract.
- 884 tests pass across wishlist + downloads + imports suites.
- Ruff clean on changed Python files; JS syntax OK on changed
webui files.
PR 3 (sibling-completion gate) was investigated and dropped — the
"1/26 finalized" symptom turns out to be downstream of the
staging-match bug (PR 2's instrumentation will catch it on the
user's next reproduction run), not an independent sibling-gate bug.
The gate logic itself is correct.
Two confirmed-from-code-reading bugs in the wishlist retry chain.
Both cause downstream post-process to render every retried file as
``01 - <title>`` without year in the folder path, even when the
source slskd file had the correct track number embedded and Spotify
had the album release date.
**Bug A — track_number defaults to 1 at every link in the chain.**
Pre-fix: ``.get('track_number', 1)`` defaulted at four sites:
- ``core/wishlist/payloads.py:121`` ``ensure_wishlist_track_format``
- ``core/wishlist/payloads.py:282`` Track-object conversion
- ``core/imports/context.py:421`` legacy album-info builder
- ``core/imports/pipeline.py:645`` final processing read
Each step "filled in" 1 when the upstream had dropped the key. The
downstream filename-extract fallback at ``pipeline.py:652`` ONLY
runs when the value is None — pre-filled 1 never matched, so the
fallback never fired, so the source filename's track number (e.g.
``08. No Sleep Till Brooklyn.flac``) was discarded in favour of the
default-1.
Fix: change every default from ``1`` to ``None`` along the chain.
The pipeline already has the right detect-and-recover logic — it
just needs the chain to stop poisoning it. Final ``< 1`` floor at
``pipeline.py:660`` still defaults to 1 as last resort, so callers
that genuinely have nothing still produce a valid number.
**Bug B — release_date dropped from cancelled-task wishlist payload.**
Pre-fix: ``build_cancelled_task_wishlist_payload`` only ``setdefault``ed
``name`` / ``album_type`` / ``images`` on the album dict. The
release_date field copy was load-bearing (when input was a dict, the
``dict(album_raw)`` copy preserved it), but when input was a bare
string the constructed dict had only name + album_type — no
release_date / total_tracks / etc.
Fix:
- Explicit comment on the dict-shape branch that release_date survives
via the unconditional ``dict(album_raw)`` copy + setdefault
semantics — so a future refactor that switches to a stricter copy
doesn't silently strip the field.
- String-shape branch now pulls release_date from
``track_info.album_release_date`` or ``track_info.release_date``
when present so the round-trip preserves the year for the path
template.
- track_data shape itself now carries ``track_number`` / ``disc_number``
at the top level (Bug A intersect — was dropping it entirely).
**Tests:** 4 new in tests/wishlist/test_payloads.py:
- ``test_ensure_wishlist_track_format_preserves_real_track_number``
- ``test_ensure_wishlist_track_format_keeps_missing_track_number_as_none``
- ``test_build_cancelled_task_wishlist_payload_preserves_track_number``
- ``test_build_cancelled_task_wishlist_payload_string_album_pulls_release_date_from_track_info``
14 payload tests pass; 879 across wishlist + imports + downloads
suites still green; 1410 wider suite all pass. Ruff clean.
Commits 2 + 3 of 3 in PR 2/4 of the wishlist-album-bundle issue fix
series. Commit 1 (94ba1d73) instrumented staging-match so the next
wishlist run produces the evidence we need to diagnose bug C
(staging-match silently drops album-bundle wishlist tracks); that
fix lands in a follow-up PR after the user's next reproduction run.
Real-world wishlist case the original c3b88e69 design missed: user with
26 missing tracks from 26 different albums. Each item used to promote
to its own album-bundle sub-batch (``min_tracks_per_album=1``), which
downloaded the ENTIRE album (5-42 files) to claim one track. Confirmed
in app.log:
- "Licensed To Ill" downloaded 3 times across cycles (3-4 files each)
- "The Understanding" 17 files for 1 wishlist track
- "Alright, Still" 42 files for 1 wishlist track
- ~85% wasted bandwidth, slskd hammered with 26 concurrent searches
PR 1 of a 4-PR fix series — see commit body footer for the other PRs.
Default ``min_tracks_per_album`` 1 → 2. Single-track wishlist items
fall to ``residual_tracks`` → classic per-track batch (already works,
already efficient). Album-bundle kept for the case it was designed
for: user has 2+ tracks missing from the same album.
Override via the new ``wishlist.album_bundle_min_tracks`` config key:
- 1 = previous behaviour (bundle every item)
- 2 = new default
- 3+ = stricter, for users who want bundle only on bigger gaps
Helper ``_resolve_album_bundle_threshold`` lives in
``core/wishlist/processing.py``. Defensive shape mirrors the existing
config-driven knobs (``get_poll_interval`` / ``get_transient_miss_threshold``):
non-numeric, non-positive, or config-manager-raise all fall back to
the safe default. Three test cases pin the fallback chain.
Both wishlist entry points wired through the same helper:
- ``process_wishlist_automatically`` (auto cycle, line 812)
- ``start_manual_wishlist_download_batch`` (manual run, line 539)
Tests:
- ``tests/wishlist/test_album_grouping.py`` — old ``test_default_threshold_promotes_solo_albums`` flipped to ``test_default_threshold_demotes_solo_albums`` with explanatory docstring naming the real-world cause. New ``test_default_threshold_promotes_multi_track_albums`` pins the 2+ promotion. New ``test_explicit_threshold_one_restores_solo_promotion`` pins that the kwarg still works for opt-back-in.
- ``tests/wishlist/test_processing.py`` — 3 new tests for ``_resolve_album_bundle_threshold``: default-when-config-missing, honors-config-override, falls-back-on-garbage.
- ``tests/wishlist/test_automation.py`` — ``test_wishlist_albums_cycle_splits_into_per_album_batches`` updated to use 2+ tracks per album (5 tracks across 2 albums instead of 3 across 2 with 1 solo). ``test_wishlist_albums_cycle_residual_for_orphan_tracks`` updated to include 2 tracks from Album One so it still promotes.
- ``tests/wishlist/test_manual_download.py`` — same shape update for the manual path test.
- ``tests/wishlist/test_album_grouping.py:test_multiple_albums_emit_separate_groups`` updated to reflect new default (alb1 with 2 tracks promotes, alb2 with 1 track goes residual).
- ``tests/wishlist/test_album_grouping.py:test_nested_track_data_payloads_normalized`` pinned with explicit ``min_tracks_per_album=1`` so the test stays focused on payload-shape parsing, not the threshold rule.
114 wishlist tests pass; 866 across wishlist + automation + downloads +
album_bundle + album_bundle_dispatch suites still green. Ruff clean.
Sibling PRs queued in TaskCreate:
- PR 2 — investigate post-process staging-match miss (the second-order
bug that causes the same album to redownload every cycle when the
staging step doesn't claim the requested track).
- PR 3 — fix sibling-completion gate that fires on first sibling
instead of last (log evidence: run a4945c88 finalized 1/26 batches).
- PR 4 — UI distinguish Queued from Analyzing for batches waiting
on the executor (23/26 batches sit at "Analyzing..." while really
queued at max_workers=3).
PR 3 of the schedule-types feature — see
``memory/project_auto_sync_schedule_types.md``. Backend
``next_run_at`` + ``weekly_time`` trigger handler landed in PRs 1-2.
This PR exposes them in the Auto-Sync manager so users can finally
schedule playlists by day-of-week + time instead of only hourly
intervals.
**UI layout:**
The Auto-Sync modal grows a ``Weekly Board`` tab between
``Hourly Board`` (renamed from ``Schedule Board``) and
``Automation Pipelines``. Same sidebar (mirrored playlists grouped
by source, with filter). Main panel is 7 day columns Mon-Sun
instead of 10 hour buckets. Drag a playlist onto a day column →
creates a single-day weekly schedule at the default time
(09:00 in the browser's IANA tz from
``Intl.DateTimeFormat().resolvedOptions().timeZone``). Click any
scheduled card → opens an editor popover for time, multi-day
toggles, tz override, and unschedule.
Multi-day schedules render under every matching column (Mon-Wed-Fri
schedule appears as three cards, one per column) — matches how
users think about "this playlist runs on Mon AND Wed AND Fri".
**Mutual exclusion:** one schedule per playlist. The save path on
either tab deletes any existing schedule of the OTHER kind before
installing the new one. Backend can technically run both as two
separate automation rows, but two cards under the same playlist
would surprise users and the engine has no merge semantic for
"daily-and-hourly".
**Pure-function helpers** (testable via node:test, matching the
existing ``tests/static/test_auto_sync.mjs`` pattern):
- ``detectBrowserTimezone()`` — Intl tz with UTC fallback for
browsers where Intl is absent.
- ``autoSyncWeeklyTrigger({time, days, tz})`` — defensive payload
builder: garbage time → 09:00, unrecognised days dropped,
missing tz → browser tz.
- ``autoSyncWeeklyFromTrigger(config)`` — inverse parser with
the same defensive shape. Empty days expands to every weekday
(matches ``next_run_at`` engine semantic). Returns null for
non-object configs so ``buildAutoSyncScheduleState`` can route
broken rows to automationPipelines instead of silently
bucketing them as every-day weekly.
- ``autoSyncWeeklyLabel(parsed)`` — sorted "Mon, Wed, Fri @
09:00" / collapses to "Daily @ HH:MM" for full-week / "Unscheduled"
for null. Canonical Mon-Sun ordering regardless of input order.
**Tests:** 26 new node:test cases across ``detectBrowserTimezone``
x1, ``autoSyncWeeklyTrigger`` x6, ``autoSyncWeeklyFromTrigger`` x6,
``autoSyncWeeklyLabel`` x5, and ``buildAutoSyncScheduleState``
weekly bucketing x5 (covering owned weekly_time → weeklySchedules,
hourly stays in playlistSchedules, non-owned falls through to
automationPipelines, legacy-named auto-sync rows still recognised,
garbage trigger_config falls through). All 62 node:test cases pass;
261 across the automation pytest suite still green (zero regression
on PRs 1-2's plumbing). Python wrapper at
``tests/test_auto_sync_js.py`` shells out cleanly.
**CSS** (themed to the existing Auto-Sync gradient + accent
variables):
- 7-column grid for the weekly board, narrower than the 10
hour-bucket layout.
- Editor popover with backdrop-blur, accent-tinted save / delete
buttons, hover states that pick up the user's accent color.
- ``scheduled-elsewhere`` state for playlists with an hourly
schedule visible on the weekly board (dashed border + opacity)
so the user knows a drop will replace, not stack.
**WHATS_NEW entry** under 2.6.3 unreleased — first user-visible
slice of the schedule-types feature.
PR 4 (Monthly UI tab) deferred until weekly proves wanted.
PR 1 (commit 6ad85e27) shipped the ``next_run_at`` pure function as
foundation plumbing. PR 2 wires the engine through it and adds
``monthly_time`` as a real registered trigger type. After this PR
``core/automation_engine.py`` no longer has its own datetime
arithmetic for daily / weekly schedules — every next-run computation
flows through one function with one set of defensive fallbacks.
Net user-visible change: zero (no UI surface for monthly_time yet —
that's PR 3). New ``monthly_time`` trigger is reachable only via
direct API for now.
**Engine refactor:**
- ``_finish_run`` — collapsed three inline branches (daily_time
arithmetic, weekly_time arithmetic, fallback schedule arithmetic)
into a single ``next_run_at(...)`` call with ``_dt_to_db_str``
normalising the aware-UTC result to the engine's naive-UTC string
convention. Retry-delay short-circuit preserved. Exception
swallowing preserved (logged at debug, writes None next_run).
- ``_setup_daily_time_trigger`` + ``_setup_weekly_time_trigger`` +
new ``_setup_monthly_time_trigger`` — three near-identical methods
collapsed into one ``_setup_timed_trigger`` skeleton. Each public
method is now a one-line dispatch passing trigger_type to the
shared helper with a human-readable label for the debug log.
- Existing ``_next_weekly_occurrence`` deleted — its logic now lives
in ``core/automation/schedule.py:_next_weekly`` (lifted in PR 1).
- New ``_dt_to_db_str(dt)`` module-level helper normalises aware-UTC
→ naive-UTC string. Centralised so a tz mistake here surfaces in
one place. Aware non-UTC datetimes converted to UTC first
(defensive against a future bug that passes the wrong tz).
- New ``_resolve_system_default_tz()`` reads the server's local IANA
tz via ``tzlocal``. Cached at module import (the host's tz doesn't
change while the process runs). Falls back to UTC when ``tzlocal``
is missing — defensive for minimal Docker images.
- New ``self._default_tz`` engine attribute reads from
``automation.default_timezone`` config first, falls back to the
system-detected IANA name. Override path lets users on weird
setups pin a specific tz without touching env vars.
**Convergence fix (intentional behaviour change):**
Old ``_setup_daily_time_trigger`` / ``_setup_weekly_time_trigger``
didn't check the DB for an existing future ``next_run`` — they'd
recompute from scratch on every engine startup, overwriting manual
edits or pending retries. The interval path (``_setup_schedule_trigger``)
already had this check. The new shared ``_setup_timed_trigger``
brings daily / weekly in line: existing-future next_run wins over
freshly-computed delay. Treat this as a correctness fix, not a
breaking change — the old behaviour was an inconsistency, not a
deliberate choice.
**Backward-compat:**
- Existing ``schedule`` / ``daily_time`` / ``weekly_time`` rows
continue to work unchanged. The ``_trigger_handlers`` registry
keeps every historic key.
- Existing rows without an explicit ``tz`` field use
``self._default_tz`` (server-local IANA via ``tzlocal``) —
preserves "every Monday 09:00 server-local" behaviour on
non-UTC servers. Pre-fix the engine used naive
``datetime.now()`` which is also server-local; net effect is
identical wall-clock time, just routed through a tz-aware
pipeline that handles DST correctly (the May 2026 "next in 8h"
bug fix class).
- Engine boots even when ``tzlocal`` is missing — the resolver
falls back to UTC silently. Existing tests would catch a hard
dependency on tzlocal here.
**``tzlocal>=5.0`` added to requirements.txt** alongside
``tzdata>=2024.1`` from PR 1. Both libraries are small and stable;
``tzlocal`` returns a clean IANA name across Windows / Linux /
Docker, sidestepping the platform-specific tz detection mess.
**Tests:** 20 new in ``tests/automation/test_engine_schedule_integration.py``:
- ``_dt_to_db_str`` x3 (aware UTC, aware non-UTC converted to UTC,
naive assumed UTC)
- ``_resolve_system_default_tz`` x2 (returns IANA string, falls back
to UTC without tzlocal)
- ``_finish_run`` dispatch through next_run_at for each trigger type
(schedule, daily_time, weekly_time, monthly_time)
- Retry-delay short-circuits next_run_at
- next_run_at returns None → DB next_run cleared
- next_run_at raises → engine swallows + writes None
- Event triggers skipped (no scheduled next-run)
- ``self._default_tz`` passed through to next_run_at
- monthly_time registered in _trigger_handlers
- All historic trigger types kept registered
- ``_setup_monthly_time_trigger`` arms timer + writes DB
- ``_setup_timed_trigger`` honours existing future DB next_run
- Skip-with-log when next_run_at returns None
- End-to-end no-mock smoke for monthly_time
260 automation suite tests pass; the 240 from PR 1's branch plus 20
new integration tests. Ruff clean.
No WHATS_NEW entry — UI doesn't expose monthly_time yet (PR 3),
and the backward-compat path preserves existing daily/weekly
schedule timing.
Self-review pass on ec4a55c1 — applying the standing kettui-grade
rule (see memory/feedback_always_build_kettui_grade.md). Three issues
that would have surfaced on review:
1. Silent tz fallback to UTC
``_resolve_tz`` returned UTC when the IANA name was unknown — no
log, no warning. User on a host without ``tzdata`` who configures
``America/Los_Angeles`` got schedules running silently at UTC
offset with no way to debug. Now logs WARNING once per unknown
name (deduped via ``_UNKNOWN_TZ_WARNED`` set so a misconfigured
row doesn't spam every poll cycle) and the log line names BOTH
real causes — typo or missing tzdata — so the user can fix from
a single grep.
2. ``weeks`` unit drift from engine
I added ``'weeks': 86400*7`` to ``_INTERVAL_MULTIPLIERS`` but the
engine's existing ``_calc_delay_seconds`` only recognises
minutes/hours/days. Until PR 2 collapses both paths through this
function, any row whose config snuck through with ``unit='weeks'``
would get scheduled by the engine as 1-hour and by this function
as 7-day — drift between two live implementations. Dropped
``weeks`` from the map to match the engine. Added a comment
pinning the map to the engine's contract and a regression test
that asserts ``unit='weeks'`` falls back to the same hours
default the engine produces.
3. DST edge cases unverified
The module docstring claims DST-aware via ``zoneinfo`` but no test
pinned the spring-forward gap (02:30 LA on DST-Sunday doesn't
exist) or fall-back ambiguity (01:30 LA on fall-Sunday happens
twice). Three new tests:
- ``test_dst_spring_forward_lands_after_the_gap`` — pins that the
function doesn't crash + lands on a real instant past ``now``.
- ``test_dst_fall_back_handles_ambiguous_local_time`` — pins
zoneinfo's default-earlier-instant resolution for ambiguous
local times (01:30 PDT vs 01:30 PST → picks PDT).
- ``test_weekly_across_dst_boundary_keeps_local_wall_clock`` —
pins that a "every Sunday at 09:00 LA" schedule keeps the
local wall clock across the boundary even though the UTC
equivalent shifts by an hour. This is the exact bug class
that caused the May 2026 "next in 8h" tz mismatch.
Also loosened ``tzdata==2026.2`` to ``tzdata>=2024.1``. IANA tz data
changes a few times a year for real-world DST policy updates; pinning
to one snapshot would freeze the app's tz knowledge to the build date
and miss future government-mandated rule changes.
41 schedule tests pass (5 new); 240 across the full automation suite.
Ruff clean.
Backend plumbing for upcoming weekly + monthly Auto-Sync schedules.
PR 1 of 4 in the schedule-types feature — see
``memory/project_auto_sync_schedule_types.md`` for the full plan.
Net behaviour change in this PR: zero. The automation engine still
computes next_run via its existing inline ``_calc_delay_seconds`` /
``_next_weekly_occurrence`` helpers; this module is unused until PR 2
wires the engine through. Lands separately so the foundation can sit
on dev for a beat before the engine change.
``core/automation/schedule.py:next_run_at(trigger_type, trigger_config,
now_utc, default_tz)``:
- Pure function. ``now_utc`` injected (tests freeze time without
monkeypatching ``datetime.now``); ``default_tz`` injected (so daily /
weekly / monthly schedules compute against the USER's timezone, not
the server's — the same class of bug that produced the May 2026
"Auto-Sync next in 8h" timezone fix).
- Returns aware-UTC ``datetime`` ready to serialise to the DB
``next_run`` column, or ``None`` for unrecognised / event-based
triggers (callers should not write a next_run for those).
- Naive ``now_utc`` inputs are assumed UTC for defensive symmetry
with the engine's DB-string parser convention.
Trigger types covered:
- ``schedule``: ``{interval: N, unit: 'minutes'|'hours'|'days'|'weeks'}``
— matches engine's existing ``_calc_delay_seconds``. Unknown unit
defaults to hours; zero/negative interval clamps to 1 (preserves
the engine's guard against scheduling for the past); non-numeric
interval falls back to 1.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — DST-aware via
``zoneinfo``; ``tz`` falls back to ``default_tz``; unknown IANA
string falls back to UTC; garbage ``time`` falls back to 00:00.
- ``weekly_time``: ``{time, days: ['mon',...], tz}`` — empty / all-
invalid ``days`` list means "every day" (matches engine fallback);
abbreviations case-insensitive; 8-day scan finds the next match.
- ``monthly_time``: ``{time, day_of_month: 1-31, tz}`` — NEW shape.
Day clamped to [1, 31]. Months too short for the target day clamp
to the LAST valid day rather than skipping a month (standard cron
convention; running a day early in February is less surprising
than missing the whole month). 12-iteration loop cap so a
pathological config can't infinite-loop.
Tests (36 cases, all passing):
- Interval: every unit, unknown-unit fallback, zero/negative/garbage
interval clamp, tz field ignored on interval (wall-clock-independent).
- Daily: today-at-future-time runs today, today-at-past-time rolls to
tomorrow, exact-match rolls to tomorrow (no schedule-now-then-schedule-
again-immediately), user-tz vs server-tz, default_tz fallback,
garbage time / unknown tz defensive returns.
- Weekly: same-day-still-future qualifies, same-day-past rolls to next
allowed day, wraps across week boundary, empty days = every day,
garbage abbreviations dropped, case-insensitive, tz across day
boundary (LA Wednesday evening is Thursday UTC).
- Monthly: target day this month, rolls to next month when passed,
Feb 31 → Feb 28 / Feb 29 leap year, day_of_month above 31 / below
1 clamp, Dec → Jan year roll, user-tz pre-midnight edge case.
- Result-shape contract: every returned datetime is aware UTC at
offset zero (engine relies on this when serialising to the
``next_run`` string column).
Added ``tzdata==2026.2`` to requirements.txt. Windows ``zoneinfo`` and
minimal Docker base images ship without the system tz database;
without ``tzdata`` ``ZoneInfo('America/Los_Angeles')`` raises
``ZoneInfoNotFoundError`` and the helper silently falls back to UTC.
No WHATS_NEW entry — no user-visible behaviour change in this PR.
PR 2 (engine wire-through) will land the user-facing changelog entry
when ``monthly_time`` becomes a real schedulable trigger.
Follow-up to f13d3395. Five gaps called out on self-review:
1. Per-track inline transient tolerance was duplicated between
usenet.py and torrent.py (~12 lines each, identical) and wasn't
directly tested. Extracted into ``TransientMissCounter`` in
``album_bundle.py`` — small class with ``record_miss()`` returning
True at threshold and ``reset()`` for successful reads. Both
per-track flows AND the lifted ``poll_album_download`` now use
the same counter, so the rule is in one place.
2. Threshold is now config-driven via
``download_source.album_bundle_transient_miss_threshold``
(default 5). Same defensive pattern as ``get_poll_interval`` /
``get_poll_timeout`` — non-positive / non-numeric falls back to
the default. Users with very slow servers (huge multi-disc box
sets, slow disks) can extend the tolerance window without
touching code.
3. SAB state map verified against the canonical Status enum in
``sabnzbd/constants.py`` (sabnzbd/constants.py:~95-118). Dropped
six entries I'd guessed at and couldn't verify in source
(``trying``, ``prop_paused``, ``prop_failed``, ``unpacking``,
``pp``, ``postprocessing``). Kept the verified ``deleted`` (lower-
cased from SAB's ``Deleted``) and added the one real state I'd
missed: ``Propagating`` (SAB's pre-download delay state — maps to
``queued`` since we're waiting on the NZB to be available, not
actively downloading).
4. SAB integration test exercising the queue→history gap end-to-end
through the real adapter HTTP layer. Mocks SAB's queue + history
endpoints with the exact response shapes SAB emits, runs three
gap polls (both endpoints empty), then a recovery poll where the
slot appears in history as Completed. Confirms the TransientMissCounter
absorbs the gap and ``poll_album_download`` returns the save_path
without emitting terminal failure. This was the path I had only
tested at the helper layer before — now pinned end-to-end through
the adapter.
5. SAB state mapping has new tests: every Status value from SAB's
canonical enum must map to a known adapter state (not the 'error'
default fallback), Propagating routes to queued, Deleted routes
to failed. Future SAB state additions that we miss will surface
as 'error' default → transient-miss tolerance → terminal failure
with a clear log line, but the explicit assertion list here means
we'll catch the omission in CI before users do.
Test count after: 537 download-suite tests pass; 21 new
(``TransientMissCounter`` ×4, ``get_transient_miss_threshold`` ×3,
SAB state-coverage ×3, SAB direct ``nzo_ids`` lookup ×5, SAB
queue→history integration ×1, plus the existing helper-layer
coverage from the parent commit). Ruff clean.
User reported usenet album downloads getting stuck on "downloading
release" while SABnzbd reported the job as complete. Container restart
did not help; reproducible on every usenet album download.
Three independent issues all causing the same symptom — the download
modal freezes mid-flow with no error surfaced to the user:
1. SAB queue → history transition window
SAB removes a slot from its queue BEFORE adding it to the history,
and on a busy server (par2 verify, unrar, multi-file move) that
window can span several poll iterations. The poll treated a single
None status as terminal failure ("disappeared from client") and
gave up. Now the poll tolerates up to ~10s of consecutive misses
(5 polls at the default 2s interval) before declaring the job gone.
2. SAB queue states like `Pp` were unmapped
`_SAB_QUEUE_STATE_MAP` didn't cover SAB's `Pp` (post-processing
summary), `Unpacking`, `Trying`, `Deleted`, or the `Prop_paused`
/ `Prop_failed` variants. Unmapped states fell through to the
default-'error' fallback, and the poll loop only treated explicit
'failed' / 'completed' as terminal — 'error' was neither, so the
loop spun until the 6-hour timeout. Map now covers every Status
value from SAB's `sabnzbd/api.py`, and the poll treats the default-
'error' fallback as a transient miss (warn-logged, retry within
the same tolerance window) so a brand-new unmapped state can't
infinite-loop the way `Pp` did here.
3. No terminal failure emit
The poll only logged on failure / timeout / disappeared — never
called the progress callback with 'failed', so the download modal
stayed at the last 'downloading' emit forever. Plumb a 'failed'
emit through every failure exit path so the UI flips out of the
downloading state when the poll gives up.
Plus:
4. SAB direct nzo_ids lookup instead of paging all-history
`_get_status_sync` was fetching the latest 50 history entries on
every poll and iterating to find the target nzo_id. On busy
servers (many recent downloads), the target job could roll past
the 50-entry window and look like a "disappeared" job. Replaced
with a targeted `mode=queue&nzo_ids=<id>` → `mode=history&nzo_ids=<id>`
chain. Falls back to the bulk path for SAB versions that pre-date
the nzo_ids filter — the transient-miss tolerance covers any
short-lived gap there too.
Implementation:
Lifted the album-bundle poll loop out of `usenet.py` and `torrent.py`
into `core/download_plugins/album_bundle.py:poll_album_download` —
near-duplicate implementations are now a single function with deps
injected so it's testable in isolation (kettui's extract-don't-AST-parse
standard; can't unit-test a `time.sleep` loop inside a plugin method).
The lifted helper takes:
- `get_status` callable bound to job_id, so the same loop works for
usenet UsenetStatus and torrent TorrentStatus shapes
- `complete_states` set so torrent's `{'seeding', 'completed'}` and
usenet's `{'completed'}` both Just Work
- `failed_states` set so torrent's `{'error'}` is terminal while
usenet's default-'error' fallback is transient
- `transient_miss_threshold` (default 5 ≈ 10s at 2s poll)
- `sleep` / `monotonic` injectables for deterministic tests
Per-track flows in both plugins gained the same transient-miss
tolerance inline — they don't use the emit pattern (update an
`active_downloads[id]` row dict via lock instead), so reusing the
helper would have required threading a no-op emit through. Inline
fix is small enough.
Tests:
- 11 new tests in `tests/test_album_bundle.py:poll_album_download`
cover the happy path, transient-miss tolerance with recovery,
hard-failure threshold, explicit-failed surface, timeout-emit,
default-'error' transient treatment, shutdown clean exit,
torrent's `seeding`-counts-as-complete, save_path captured across
iterations, and adapter-exception treated as transient miss.
- 521 download-suite tests pass (33 in test_album_bundle, others
pin existing torrent + usenet contracts).
- Ruff clean.
Closes#706.
BoulderBadgeDad