'No search results' was masking the case where slskd never accepted the search
(not configured / errored / rate-limited) — that returns instantly, which is why
some show up as no-results FAST instead of after the search window.
_search_for_retry now flags started=False (+ the slskd error) when there's no
search id; the processor logs "Search didn't run for 'X' — slskd not responding?"
(warning) and tallies it separately. so the next run tells us if the fast
no-results are really slskd refusing searches vs the source genuinely being empty.
a completed youtube download is removed from the wishlist (correct), but the
channel/playlist scans' downloaded_ids seam was stubbed to [] — so the next scan,
not seeing it wishlisted and not knowing it was downloaded, re-added it (still in
the channel's recent uploads / last-N net) → re-download loop for the last-N
videos every scan.
fix: db.downloaded_youtube_video_ids() pulls completed youtube grabs from the
permanent download history; both scans' _default_downloaded_ids now use it. so a
downloaded video stays gone. tested (db method + the scans already exclude
downloaded_ids in their pure logic).
'no acceptable release' was ambiguous — couldn't tell if slskd returned nothing
or returned hits the quality profile rejected. now each item logs 'No search
results for X' vs 'N result(s) for X, none accepted — <reason>' (reason from the
top hit), and the summary tallies '… · N had no results, M rejected on quality'.
makes the slskd-finds-nothing-for-video situation legible.
storage + wiring + API for the channel settings modal (UI next):
- db.get/set_channel_settings(channel_id): per-channel {custom_name, quality}
in the settings KV store (no schema change); blanks clear the override.
- enqueue applies them: custom_name overrides the $channel folder token; a
quality override is stashed in the download row's search_ctx. the worker reads
it back (quality_override_from_download) and uses it instead of the global
youtube quality profile. global default still applies when no override.
- API GET/POST /youtube/channel/<id>/settings (GET also returns the global
default quality for the modal's 'using default' hint).
pure helpers (enqueue_ctx, quality_override_from_download) + db + api seam-tested.
closes the real gap: a restart kills the yt-dlp worker threads but leaves their
rows at 'downloading', which the pump counts as busy -> the queue wedges and they
never finish.
track live worker dl_ids in _active_worker_ids (added at worker start, dropped in
finally). requeue_orphaned_youtube() puts any 'downloading' youtube row with no
live worker back to 'queued' so the pump re-runs it. after a restart the set is
empty, so all stuck rows recover; during normal operation active ones are
protected (no false positives, no timestamp guessing). the hourly drain calls it
before pumping + logs the count. seam-tested.
- 'Download X Wishlist' -> 'Process X Wishlist' everywhere (label + action_type):
matches the music side's 'process_wishlist', the already-process youtube action,
and reads right (movie/episode do search+pick+download, not just download).
action_types video_download_movie/episode_wishlist -> video_process_*.
- group the new automations as a two-stage pipeline in the seed list + builder
palette: Stage 1 SCANS (people/channels/playlists) fill the wishlist, Stage 2
PROCESSORS (movie/episode/youtube) drain it. icons/labels/drift test updated.
brand-new this session so no migration needed. 337 automation tests green.
the soulseek counterpart of the youtube drain — finally makes the people/airing
scans pay off. two automations (Auto-Download Movie / Episode Wishlist, hourly):
for each wished+released item, do a bounded blocking slskd search, pick the top
ACCEPTED release per the quality profile, and enqueue it exactly like a manual
grab (same add_video_download shape → the monitor finishes + organises it).
same standard as youtube: processes the WHOLE eligible wishlist (no total cap)
but searches a few at a time (max_concurrent, default 3, via a thread pool). a
busy guard skips the next hourly tick while a drain is still working. movies gate
on status='wanted' (skips monitored); episodes are all-wished. items already
downloading are skipped. quiet skip if the library folder isn't set.
reuses the real infra (build_query/slskd_search/_evaluate_hits/start_download/
download_monitor) — pure pick/select/record seam-tested; db queries added. 377
automation+video tests green.
playlists are followable but had no scan. new 'Scan Watchlist Playlists'
automation, sibling of the channel scan but a different rule: a playlist is a
curated finite set, so MIRROR it — wishlist every long-form video you don't have
plus any later additions (no forward-looking baseline, no last-N net).
playlist-as-show: videos wishlisted under the playlist's title, so the worker
files them as 'Playlist Name / Season YEAR / ... - date - title' (the ytdl-sub
tv_show_name-on-a-playlist convention). reuses all the channel plumbing — same
wishlist rows, same Download YouTube Wishlist drain, quality + org template.
seeded every 6h (Auto-Scan Watchlist Playlists); block + registration + icon +
drift test. dedup reuses wishlisted_video_ids_for_channel (parent_source_id=PL).
seam-tested; 944-test sweep green.
they were only builder blocks (not in the Active list). seed them like the
airing job so they appear + run out of the box:
- Auto-Scan Watchlist People — daily 03:00
- Auto-Scan Watchlist Channels — every 6h
- Auto-Download YouTube Wishlist — every 1h
scans no-op cleanly if you follow nothing. softened the download handler: an
unset youtube folder is now a quiet skip (status completed), not a per-run error,
so non-youtube users don't see a recurring failure.
ditch the per-run batch cap (a 200-video backlog would've taken weeks). now the
'Download YouTube Wishlist' automation queues the ENTIRE wishlist as 'queued'
rows, starts up to max_concurrent (default 3) right away, and each finished
download starts the next (one-out-one-in in the worker) so it all drains in a
controlled stream. the knob is 'max simultaneous downloads', not a total cap.
mirrors the music download worker's lesson (cap concurrency + space starts to
avoid yt-dlp 429s) but stays isolated on the video side:
- youtube_download: _pace() staggers fetch starts (3s); start_next_queued()
claims+spawns the next; run_youtube_download chains on finish.
- db.count_active_youtube_downloads() + claim_next_youtube_queued() (atomic,
race-safe). block field batch_size -> max_concurrent.
all seam-tested (pure select + pump); 680-test sweep green.
new 'Download YouTube Wishlist' automation: pushes a polite batch of wished
youtube videos into the shared video_downloads queue + spawns the yt-dlp worker
per video. skips in-flight ones (no double-grab); big backlogs drain over
several scheduled runs (batch_size, default 3). needs the youtube library folder
set.
- download_monitor: SKIP source='youtube' rows (owned by their worker thread, no
slskd transfer to match) — surgical, slskd path untouched.
- db.youtube_wishlist_to_download(): flat newest-first list of wished videos with
channel/title/date/thumb for organising.
block + registration + icon/label + drift test. all seam-tested.
two new video-side automation blocks that keep the wishlist fed:
- scan watchlist people: for each followed person, wishlist every un-owned
movie they acted in or directed (back catalog + upcoming). released ->
wanted, upcoming -> monitored (engine skips it til it's out, promotes on
release). grabs rich detail at add time (backdrop/cast/overview/etc +
provenance) into a new video_wishlist.detail_json col. fast re-runs skip
already-wishlisted + only promote.
- scan watchlist channels: for each followed youtube channel, wishlist new
long-form uploads (shorts excluded). forward-looking from follow time +
a last-N safety net (default 10). diffs against wishlisted/downloaded/
dismissed so it never dupes. pair with a 6h schedule trigger. scan-only;
fulfillment engine comes later.
both are pure handlers with injected seams + full seam tests. add_movie_to_
wishlist gains status + detail_json (promote-only upsert). no schema break,
music side untouched.
Unlike the cleanup twins, backup can't share the music handler — it's a different
DB file. Extract the music backup body into _backup_db_at(db_path, ...) (music
behaviour byte-identical, now a thin wrapper over DATABASE_PATH) and add
auto_backup_video_database pointing at VIDEO_DATABASE_PATH (video_library.db).
New video_backup_database action (scope='video' block + registry), owned_by='video'
system automation on the music cadence (every 3 days).
Tests: a REAL backup behaviour test — music backup lands next to music_library.db,
video backup next to video_library.db, no cross-contamination (this is the whole
reason it can't be shared); scope isolation; single video-owned seed; own handler.
Existing music maintenance tests (22) still green — refactor is non-regressing.
EXPECTED_ACTION_NAMES updated.
ruff S110 flagged two try/except/pass in video handlers that predate this work.
Both are deliberate (a progress-log failure must not abort pruning; a probe's
uncertainty just keeps probing) — extend the existing BLE001 noqa to S110 with
the rationale. ruff check . is clean again.
Same pattern: video_full_cleanup action (scope='video' block + registry), reuses
the shared auto_full_cleanup handler, owned_by='video' system automation on the
music cadence (every 12h). Music copy untouched. Seam tests + EXPECTED_ACTION_NAMES.
Same pattern as phase 2: video_clean_completed_downloads action (scope='video'
block + registry), reuses the shared auto_clean_completed_downloads handler, and
an owned_by='video' system automation on the music cadence (every 5 min). Music
copy untouched. Seam tests for scope isolation, single video-owned seed, and
shared-handler reuse; EXPECTED_ACTION_NAMES updated.
The music side's 'Clean Search History' automation now has a video counterpart so
it appears on the video Automations page too. Distinct action_type
video_clean_search_history (the system seeder keys on action_type, so reusing the
music key would collide), registered to the SAME shared handler so behaviour is
identical, scope='video' block (registry — users can build their own), and an
owned_by='video' system automation on the same 1h cadence. The music action/row
is untouched.
Seam tests: video-scoped only (not on music), music action still music-scoped,
exactly one video-owned system row at the 1h cadence, and it reuses the music
handler. Registration contract (EXPECTED_ACTION_NAMES) updated.
You can eye-add a show to the watchlist before we know its status, so ended/canceled
shows leak in (auto-airing LIBRARY shows already exclude ended ones; explicit follows
don't). Fix it as cleanup-on-process, per Boulder: the daily 'Wishlist Today's Airings'
automation now runs a watchlist-tidy pass first — scans every explicit show follow,
resolves its status (local for owned, TMDB for tmdb-only follows), and removes any
that have ended/been canceled/completed. Only prunes on a DEFINITIVE terminal status;
unknown/lookup-error → left alone. Toggle prune_ended (default on); returns shows_pruned.
DB: followed_shows(). Pure prune_ended_show_follows() with injected seams; seam tests.
The probe fired the instant a batch finished, but a fresh drop takes ~1-2 min to
appear even with the server's auto-scan ON — so it always missed and we crawled
anyway, defeating the optimization. Now probe_present_libraries POLLS each candidate
over a grace window (probe_grace_minutes, default 2), skipping a library's crawl as
soon as the server reports it has the item, and only crawling what's still missing
when grace expires. The probe target for a media type you DIDN'T just download is an
old item the server already has → confirmed instantly, no wait. grace=0 probes once.
Scanning is expensive and most servers auto-ingest new files, so a full crawl after
every download is usually wasted. Stage 1 now probes per library: take the newest
completed grab of that type from download history and ask the server (cheap targeted
search) whether it already has it. If yes, the server auto-picked it up (and the
earlier ones) → skip that library's crawl + poll entirely. Only libraries the server
is missing get rescanned. Always emits so stage 2 still reads the new items in.
- sources: PlexVideoSource.has_item / JellyfinVideoSource.has_item (match movie by
title+year, episode by show+SxE) + video_server_has_item() — conservative, any
uncertainty → False so we scan.
- handler: per-scope skip decision fed by latest_completed + server_has_item seams;
narrows the scan scope to only the missing libraries; toggle skip_if_present
(default on). Returns scanned/skipped for visibility.
Seam tests: skip-both, scan-only-missing, no-history, toggle-off, probe-error→scan;
Plex has_item match tests.
A fixed debounce can't fit a big library — 8500 movies + 4500 shows scan sequentially
through Plex's queue and can take 10-20 min, so the old 120s wait read the DB before
Plex finished and fresh downloads showed up late. Now Stage 1 (video_scan_server)
fires the rescan then POLLS the server until its scan queue goes idle, then emits the
done event.
- sources: PlexVideoSource.is_scanning (section.refreshing + activity feed, scoped by
media_type) and JellyfinVideoSource.is_scanning (scheduled-task state), plus
video_server_scan_in_progress() returning True/False/None.
- handler: pure wait_for_server_scan(scan_status, sleep, …) — grace, then poll every
interval until idle or a generous cap; falls back to the fixed wait only when the
server can't report status (None). debounce_seconds is now that fallback; new
max_wait_minutes caps the poll.
Seam tests for the poll logic (idle/poll/fallback/cap/lost-status), the handler wiring,
and Plex scan-status detection.
A deep scan is the equivalent of music's full refresh — it READS the server's
current state into video.db and prunes what's gone. It should NOT tell Plex to
rescan its disk. The deep-scan action types were wired to auto_video_scan_library
(nudge Plex + read); point them at the read-only auto_video_update_database in
'deep' mode instead. Update-db phase wording no longer says "new" for a full re-read;
deep-scan block descriptions clarify it's a read, not a disk-scan. Registration test
asserts the deep scans route to the read-only handler and never nudge the server.
The deep-scan action types weren't selectable builder actions, and Scan Video Server
/ Update Video Database had no movie-vs-TV dimension — inconsistent with the rest.
- video_deep_scan_tv / video_deep_scan_movies are now proper builder blocks
(Deep Scan TV/Movie Library), not just system-automation action types.
- video_scan_server + video_update_database gain a media_type ('all'|'movie'|'show')
config + selector, threaded through. The post-download chain carries the scope on
the scan-done event, so a TV-only rescan updates only TV (stage 2 inherits it).
- refresh_video_server_sections / Plex+Jellyfin refresh_sections scope the server
nudge to the chosen library; auto_video_scan_library now nudges only its library.
- shared normalize_media_type() in sources; update_database skips cleanly when the
singleton scanner is busy. Defaults stay 'all' so existing chains are unchanged.
Seam tests for refresh scoping, scan-server scope+event, update-db scope/inherit/skip.
Video twin of music's 'Auto-Deep Scan Library', split in two because Movies and TV
are separate libraries — scanning the TV library must not pull in new movies and
vice-versa.
- scanner: add a media_type param ('all'|'movie'|'show', friendly aliases) that
gates the movies vs shows passes (and their pruning), plus an in_progress busy
guard so the singleton scanner can't be stomped by an overlapping run.
- video_scan_library handler: thread media_type through, skip cleanly when the
scanner is busy, and name only the scanned library in the summary.
- two system automations (owned_by=video, weekly deep scan, staggered start delays):
'Auto-Deep Scan Movie Library' + 'Auto-Deep Scan TV Library'. Distinct action
types (video_deep_scan_movies / _tv) because the seeder keys on action_type; both
reuse the one handler, scoped via action_config.
- builder block gains a Library selector (Movies+TV / Movies / TV) so custom scans
can scope too; card label/icon maps cover the video action types.
Seam tests for scanner scope + busy guard, handler scope + skip, registration set.
Field-by-field against the working manual 'add to wishlist', the automation now
matches it on every column EXCEPT the show poster: the get-modal stores
poster_url = '/api/video/poster/show/<library_id>', the automation stored None — so
the wishlist orb fell back to the show's initials and read as 'not matched'. Carry
the same proxy path. With library_id (last commit) + poster_url (this) + the
tmdb_season stills/overviews, an auto-added row is now identical to a manual one.
The real difference from a manual add: the wishlist resolves a show's synopsis +
cast from /api/video/detail/show/<library_id>, and falls back to the TMDB endpoint
only when library_id is absent (which redirects/lacks cast for owned shows). A
manual add sends show.library_id; the automation sent none — so auto-added shows
read as 'not matched' with no synopsis/actors. The handler now carries the show's
library id (the calendar's show_id) through to the wishlist.
Root cause of the metadata loss: a MANUAL 'add to wishlist' gets its episode
data from the TMDB season fetch (engine.tmdb_season — absolute still, overview,
season poster), while the automation read the local DB episodes table, where
stills are frequently empty/Plex-relative. So auto-added episodes came in blank
even after carrying the DB values.
The handler now fetches the SAME TMDB season metadata (cached per season,
injected for tests) and prefers it, falling back to the calendar/DB values if
TMDB is unavailable. Auto-added episodes now match manual ones.
Auto-added airing episodes came in metadata-empty (no synopsis, no still) — the
handler only passed season/episode/title/air_date, dropping the overview the
calendar already returns and never fetching the still URL (calendar_upcoming
only returned a has_still flag, not the URL). Now calendar_upcoming also returns
e.still_url, and the handler carries overview + still_url through. The wishlist
renders the (Plex-relative) still via the same pimg() proxy as the show poster,
so it resolves. Idempotent upsert backfills the already-added empty rows on the
next run.
Two Sonarr-parity features.
1) Per-episode live tracking. "Grab season" was headless (only a button label
changed); episode rows had a status span that was never populated. Now every
episode ROW shows its own live state — Searching → Downloading % → Downloaded
/ Failed — via epTrack() polling /downloads/status?id, matching the inline
movie tracker. Grab season lights all target rows at once; manual + per-source
auto grabs also light their row; reopening the modal resumes tracking in-flight
episodes (resumeEpisodeTracking via /downloads/active + search_ctx match).
Season batch grabs through the same payload as a manual grab (_pickAndGrab →
sendGrab(buildGrabPayload)).
2) Auto-wishlist airing episodes. New daily automation (video_add_airing_episodes):
reads the calendar for episodes airing TODAY for followed shows, skips owned
ones, adds the rest to the wishlist (idempotent). Handler uses injected seams
(calendar read + wishlist write) so it's unit-tested without a DB/server.
Registered + action block + seeded as a daily system automation (01:00),
owned_by=video.
Mirrors the music flow: a finished download batch → refresh the media server → (after
it indexes) pull the new media into the DB — so a downloaded movie/episode shows as
owned without waiting for the 6h scheduled scan.
Two event-based system automations (owned_by='video'):
- 'Auto-Scan Video After Downloads' (video_batch_complete → video_scan_server)
- 'Auto-Update Video Database After Scan' (video_library_scan_completed → video_update_database)
Pieces:
- core/video/download_events.py: a callback registry (core/video can't import the
engine — isolation). The monitor publishes batch-complete; web_server bridges it to
automation_engine.emit('video_batch_complete', …), like music's web_scan_manager.
- download_monitor: fires the batch-complete event once, when the last in-flight
download finishes (none queued/downloading/searching left).
- video_scan_server handler (stage 1): refresh server, wait a debounce for indexing,
then emit 'video_library_scan_completed' (mirrors music's time-based completion).
- video_update_database handler (stage 2): incremental read (newest-first, stop after
25 consecutive known — same as music).
- blocks: 2 video triggers + 2 video actions (scope='video'); registration; seeds.
kettui: seam tests for both handlers (refresh/wait/emit, incremental read, error paths),
the event registry (idempotent register, isolated failures), and the monitor (fires once
on last completion, never while work remains). 298 automation tests + isolation green.
The video side gets its OWN automations at music-side parity, kept separate so
nothing on the music side breaks. First twin: Scan Video Library — tells the media
server to rescan the user's SELECTED video sections (movies/TV, never music), then
reads the result into video.db so freshly-downloaded media shows as owned.
Architecture (scope tags + video twins on the shared engine):
- Handler core/automation/handlers/video_scan_library.py — pure function with
injected I/O (server_refresh / run_video_scan); production lazily binds
refresh_video_server_sections() + the video scanner. Owns its own progress.
Lives on the SHARED automation side so it may import core.video (isolation only
forbids core/video & api/video from importing music, not the reverse).
- blocks.py gains a 'scope' tag ('both' generic / 'video' video-only / absent=music)
+ blocks_for_scope(). The music /api/automations/blocks now filters out video
blocks; new isolated /api/video/automations/blocks serves the video palette.
- automation_engine seeds 'Scan Video Library' (owned_by='video', schedule 6h) so
it appears ONLY on the video Automations page; ensure_system_automations now
honours owned_by + action_config. Music page excludes owned_by='video' rows.
kettui: seam-level tests for every handler path (happy/no-server/scan-error/never-
raises/mode), scope filtering (music excludes video, video gets generics, music
parity preserved), seeding (owned_by + mode), registration drift guard. 39 new
tests; full automation suite (288) + isolation guards green.
Phase 2 of the redesign. The tool that judged quality by extension and auto-dumped
matches into the wishlist is gone; quality scanning is now the reviewed
quality_upgrade repair job.
Removed:
- Frontend: Tools-page Quality Scanner card, its JS handlers/poller/socket listener,
help tooltip + tour entry (webui index.html, core.js, helper.js, wishlist-tools.js).
- Backend: /api/quality-scanner/{start,status,stop} endpoints, the in-memory state +
executor + 1s socket broadcast, the QualityScannerDeps/run_quality_scanner shim.
- core/discovery/quality_scanner.py: the auto-acting worker + deps class (the shared
match/normalize helpers stay — the new job imports them).
Rewired:
- Automation 'start_quality_scan' action now triggers the quality_upgrade repair job
via repair_worker.run_job_now() (AutomationDeps gains run_repair_job_now, drops the
4 scanner fields). Action block's vestigial scope field removed (scope lives in the
job's settings now). NOTE: the 'quality_scan_completed' trigger no longer fires (the
repair job doesn't emit it).
- Updated all automation test _build_deps helpers + conftest tool-progress harness;
deleted the obsolete worker test. 528 affected tests pass; 6123 collect cleanly.
QUALITY_TIERS / _get_quality_tier_from_extension kept (used elsewhere).
Symmetric to the post-download reconcile (which handles ADDITIONS): when a
playlist's membership is re-synced (the mirror step — scheduled refresh or the
manual mirror endpoint), rebuild its folder from current membership WITH prune
IF it's organize-by-playlist. So a track that just LEFT the playlist has its
symlink cleaned up the instant membership changes, not only on the next download.
Factored a shared _rebuild_one_from_db (used by the manual 'Rebuild' button and
the mirror hook) + rebuild_mirrored_playlist_if_organized. Gated to organized
playlists, non-fatal at both mirror call sites.
Now the invariant 'folder = the playlist's current owned members' holds on every
change: additions caught at download, removals caught at mirror. 2 new tests
(removed track pruned; non-organized skipped). 985 + 277 tests pass.
SoulSync standalone matches library tracks without Plex fetchItem,
reports missing counts correctly, and skips server playlist writes.
Automation re-syncs when the mirror grows; after sync finishes, starts
organize download (organize-by-playlist) or wishlist processing.
UI: Spotify URL playlist-folder controls, organize toggle layout in the
discovery modal, reload organize preference when reopening Download Missing.
Co-authored-by: Cursor <cursoragent@cursor.com>
Persist organize_by_playlist on mirrored playlists and run playlist-folder
downloads from the auto-sync pipeline instead of the global wishlist phase.
Register SoulSync library rows after playlist-folder post-processing, route
failed organize batches to the wishlist correctly, and skip sync-time
unmatched wishlist only when organize download handles retries.
Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL),
re-mirror on refetch, and improve standalone playlist modals (re-analysis,
Open in Mirrored). Add filesystem missing-track detection and tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
The DB-update + deep-scan automation monitor used a hard 2-hour TOTAL cap
(while elapsed < 7200). It tracked progress but only used it to print a stall
warning — the only thing that actually timed out was wall-clock. So a large
library that scans for >2h while progressing fine (reported: 4781 artists) trips
the cap and the automation card flips to 'error: timed out after 2 hours' even
though the scan thread is healthy and still running (the timeout never cancels
it, which is why it keeps progressing in the logs after the 'error').
Time out on STALL, not total runtime:
- 30 min with NO progress -> error ('stalled'); catches a genuinely hung scan.
- 10 min idle -> warning (repeats); unchanged heads-up.
- 24h absolute backstop, purely a runaway-loop guard.
- An actively-progressing scan keeps resetting the idle clock, so it never
times out no matter how many hours the whole library takes.
- Progress is judged on (processed, progress, current_item) so a slow stretch
where the rounded % holds steady (but the artist keeps changing) isn't a
false stall.
The decision is extracted into a pure, testable scan_wait_action(); both the
deep-scan and full-refresh handlers share the monitor loop, so both are fixed.
Tests: tests/test_scan_wait_action.py (9) — headline regression (5h/12h total
but progressing -> 'continue', not timeout), finished/stall-warn/stall-timeout/
abs-cap thresholds, and ordering. 280 automation tests still pass.
Post-incident hardening. A WAL-mode DB corrupted (most likely an interrupted
write during a hard restart), and the backup routine made it unrecoverable:
it (a) never checked integrity, so src.backup() faithfully copied the corrupt
pages into every rolling backup, and (b) pruned oldest-by-mtime, so each new
corrupt backup evicted the last good one. Result: all snapshots poisoned.
New core/db_integrity.py (pure, unit-tested):
- quick_check()/is_healthy(): fast read-only PRAGMA quick_check probe.
- safe_backup(): verifies the SOURCE is healthy BEFORE the Online-Backup copy
and the RESULT after; refuses + discards rather than save a corrupt copy.
- prune_backups(): rotation that NEVER deletes the most-recent verified-healthy
backup, even to honor max_keep — so a run of bad backups can't drop your last
good snapshot.
Wired into BOTH backup paths (the /api/database/backup endpoint and the
auto_backup_database automation handler) — they now refuse on integrity failure
(409 / error status, existing backups untouched) and prune safely.
Tests: tests/test_db_integrity.py (8) using REAL temp DBs incl. a physically
corrupted one — proves refuse-corrupt-source, discard-corrupt-result, and the
exact incident scenario (newest backups corrupt -> the older healthy one is
protected from pruning). Existing maintenance-handler backup test still green
(29 passed). compile + ruff clean.
NOTE: this prevents silent backup poisoning; it does NOT stop the underlying
corruption. Follow-ups still worth doing: WAL-checkpoint on clean shutdown +
a periodic live-DB integrity alert (so corruption is caught on day 1).
Pipeline-driven Auto-Sync runs against any ListenBrainz playlist
(Weekly Jams, Weekly Exploration, Top Discoveries, etc.) would sit
on ``Refreshing: "<name>"`` with no UI updates for 5-7 minutes
before the pipeline progressed. Two real bugs stacked:
1. **Double discovery.** The refresh handler called
``_maybe_discover`` (matching engine, per-track Spotify/iTunes/
Deezer matches) inline for any source returning
``needs_discovery=True`` tracks. Phase 2 of the pipeline then
ran the SAME matching engine via ``run_playlist_discovery_worker``
on the same tracks. The refresh-side run blocked the loop with
zero progress emission; Phase 2's already has the timed
progress-poll pattern. So LB tracks discovered twice, the first
time silently.
Pipeline now sets ``skip_discovery=True`` on its refresh config.
The handler honors the flag and lets Phase 2 handle discovery
end-to-end. Standalone callers (Sync-page tab, registration
action) leave the flag unset so they still get matched_data
on refresh.
2. **No targeted LB refresh.** The LB adapter's ``refresh_playlist``
called ``manager.update_all_playlists()`` — the only refresh
entry-point the manager exposed — which re-pulls every cached
LB playlist's details from the API (~12+ round-trips) even
when only one playlist needed refreshing. Wasteful;
tax-on-everyone for one-playlist work.
Added ``LBManager.refresh_playlist(mbid)`` — reads the cached
playlist_type, fetches just that playlist's details, runs the
normal ``_update_playlist`` upsert path. Defaults type to
``user`` for un-cached mbids so new-playlist discovery still
works. Skips ``_cleanup_old_playlists`` and
``_ensure_rolling_mirrors_from_cache`` (wasted work for a
single-playlist refresh).
Also: killed a silent ``except Exception: pass`` in the LB
adapter's old refresh wrapper that was masking every LB API
failure as a stale-cache hit. Refresh errors now log with full
traceback at warning level and propagate ``None`` so the outer
handler at ``refresh_mirrored.py:104`` counts the error and
surfaces it to the run-history error tally.
Pinned with 12 new unit tests across:
- ``tests/test_listenbrainz_manager.py`` (8): targeted refresh
happy path, unauthenticated guard, empty-mbid guard, upstream
``None`` return, default playlist_type for unknown mbid,
exception propagation, cost guard skipping cleanup, skipped-
when-unchanged signal
- ``tests/test_playlist_sources_adapters.py`` (3): adapter uses
targeted call (not legacy), adapter returns ``None`` on manager
error (not silent swallow), adapter resolves synthetic series
ids before calling the manager
- ``tests/automation/test_handlers_playlist.py`` (1):
skip_discovery flag bypasses ``_maybe_discover`` end-to-end
Self-review pass on ec4a55c1 — applying the standing kettui-grade
rule (see memory/feedback_always_build_kettui_grade.md). Three issues
that would have surfaced on review:
1. Silent tz fallback to UTC
``_resolve_tz`` returned UTC when the IANA name was unknown — no
log, no warning. User on a host without ``tzdata`` who configures
``America/Los_Angeles`` got schedules running silently at UTC
offset with no way to debug. Now logs WARNING once per unknown
name (deduped via ``_UNKNOWN_TZ_WARNED`` set so a misconfigured
row doesn't spam every poll cycle) and the log line names BOTH
real causes — typo or missing tzdata — so the user can fix from
a single grep.
2. ``weeks`` unit drift from engine
I added ``'weeks': 86400*7`` to ``_INTERVAL_MULTIPLIERS`` but the
engine's existing ``_calc_delay_seconds`` only recognises
minutes/hours/days. Until PR 2 collapses both paths through this
function, any row whose config snuck through with ``unit='weeks'``
would get scheduled by the engine as 1-hour and by this function
as 7-day — drift between two live implementations. Dropped
``weeks`` from the map to match the engine. Added a comment
pinning the map to the engine's contract and a regression test
that asserts ``unit='weeks'`` falls back to the same hours
default the engine produces.
3. DST edge cases unverified
The module docstring claims DST-aware via ``zoneinfo`` but no test
pinned the spring-forward gap (02:30 LA on DST-Sunday doesn't
exist) or fall-back ambiguity (01:30 LA on fall-Sunday happens
twice). Three new tests:
- ``test_dst_spring_forward_lands_after_the_gap`` — pins that the
function doesn't crash + lands on a real instant past ``now``.
- ``test_dst_fall_back_handles_ambiguous_local_time`` — pins
zoneinfo's default-earlier-instant resolution for ambiguous
local times (01:30 PDT vs 01:30 PST → picks PDT).
- ``test_weekly_across_dst_boundary_keeps_local_wall_clock`` —
pins that a "every Sunday at 09:00 LA" schedule keeps the
local wall clock across the boundary even though the UTC
equivalent shifts by an hour. This is the exact bug class
that caused the May 2026 "next in 8h" tz mismatch.
Also loosened ``tzdata==2026.2`` to ``tzdata>=2024.1``. IANA tz data
changes a few times a year for real-world DST policy updates; pinning
to one snapshot would freeze the app's tz knowledge to the build date
and miss future government-mandated rule changes.
41 schedule tests pass (5 new); 240 across the full automation suite.
Ruff clean.
Backend plumbing for upcoming weekly + monthly Auto-Sync schedules.
PR 1 of 4 in the schedule-types feature — see
``memory/project_auto_sync_schedule_types.md`` for the full plan.
Net behaviour change in this PR: zero. The automation engine still
computes next_run via its existing inline ``_calc_delay_seconds`` /
``_next_weekly_occurrence`` helpers; this module is unused until PR 2
wires the engine through. Lands separately so the foundation can sit
on dev for a beat before the engine change.
``core/automation/schedule.py:next_run_at(trigger_type, trigger_config,
now_utc, default_tz)``:
- Pure function. ``now_utc`` injected (tests freeze time without
monkeypatching ``datetime.now``); ``default_tz`` injected (so daily /
weekly / monthly schedules compute against the USER's timezone, not
the server's — the same class of bug that produced the May 2026
"Auto-Sync next in 8h" timezone fix).
- Returns aware-UTC ``datetime`` ready to serialise to the DB
``next_run`` column, or ``None`` for unrecognised / event-based
triggers (callers should not write a next_run for those).
- Naive ``now_utc`` inputs are assumed UTC for defensive symmetry
with the engine's DB-string parser convention.
Trigger types covered:
- ``schedule``: ``{interval: N, unit: 'minutes'|'hours'|'days'|'weeks'}``
— matches engine's existing ``_calc_delay_seconds``. Unknown unit
defaults to hours; zero/negative interval clamps to 1 (preserves
the engine's guard against scheduling for the past); non-numeric
interval falls back to 1.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — DST-aware via
``zoneinfo``; ``tz`` falls back to ``default_tz``; unknown IANA
string falls back to UTC; garbage ``time`` falls back to 00:00.
- ``weekly_time``: ``{time, days: ['mon',...], tz}`` — empty / all-
invalid ``days`` list means "every day" (matches engine fallback);
abbreviations case-insensitive; 8-day scan finds the next match.
- ``monthly_time``: ``{time, day_of_month: 1-31, tz}`` — NEW shape.
Day clamped to [1, 31]. Months too short for the target day clamp
to the LAST valid day rather than skipping a month (standard cron
convention; running a day early in February is less surprising
than missing the whole month). 12-iteration loop cap so a
pathological config can't infinite-loop.
Tests (36 cases, all passing):
- Interval: every unit, unknown-unit fallback, zero/negative/garbage
interval clamp, tz field ignored on interval (wall-clock-independent).
- Daily: today-at-future-time runs today, today-at-past-time rolls to
tomorrow, exact-match rolls to tomorrow (no schedule-now-then-schedule-
again-immediately), user-tz vs server-tz, default_tz fallback,
garbage time / unknown tz defensive returns.
- Weekly: same-day-still-future qualifies, same-day-past rolls to next
allowed day, wraps across week boundary, empty days = every day,
garbage abbreviations dropped, case-insensitive, tz across day
boundary (LA Wednesday evening is Thursday UTC).
- Monthly: target day this month, rolls to next month when passed,
Feb 31 → Feb 28 / Feb 29 leap year, day_of_month above 31 / below
1 clamp, Dec → Jan year roll, user-tz pre-midnight edge case.
- Result-shape contract: every returned datetime is aware UTC at
offset zero (engine relies on this when serialising to the
``next_run`` string column).
Added ``tzdata==2026.2`` to requirements.txt. Windows ``zoneinfo`` and
minimal Docker base images ship without the system tz database;
without ``tzdata`` ``ZoneInfo('America/Los_Angeles')`` raises
``ZoneInfoNotFoundError`` and the helper silently falls back to UTC.
No WHATS_NEW entry — no user-visible behaviour change in this PR.
PR 2 (engine wire-through) will land the user-facing changelog entry
when ``monthly_time`` becomes a real schedulable trigger.
Adds ``discover_tracks(tracks) -> List[NormalizedTrack]`` to the
PlaylistSource interface. Sources whose tracks already carry
provider IDs (Spotify, Tidal, Qobuz, YouTube, Deezer, Spotify
public, iTunes link, SoulSync Discovery) inherit a no-op default;
ListenBrainz + Last.fm override to run the matching engine.
This closes the last gap before LB / Last.fm / SoulSync Discovery
can land as Sync-page mirror sources: the refresh handler now
calls ``source.discover_tracks(...)`` whenever a source returns
tracks with ``needs_discovery=True``, so mirrored LB rows arrive
already discovered + ready for the sync pipeline. Previously, LB
playlists ran through a separate state-machine worker tied to the
Discover-page UI, with results stored in ``discovery_cache``
instead of ``mirrored_playlist_tracks.extra_data``.
Changes:
- ``core/playlists/sources/base.py`` — PlaylistSource switches from
Protocol to ABC so a concrete default for ``discover_tracks``
can live on the base class. The four real-work methods stay
``@abstractmethod``; instantiating an adapter that forgets one
fails loudly at construction.
- ``core/discovery/matching.py`` (new) — pure ``match_mb_tracks``
helper that runs Strategy-1-only matching-engine queries against
Spotify (primary) or iTunes (fallback). No state machine, no
discovery-cache writes, no wing-it stub — that richer flow stays
in ``core/discovery/listenbrainz.py`` for the Discover-page UI.
- ``ListenBrainzPlaylistSource`` + ``LastFMPlaylistSource`` take
an optional ``discover_callable`` constructor arg. Last.fm reuses
the LB implementation since the track shape is identical.
- ``bootstrap.build_playlist_source_registry`` accepts a
``discover_callable`` kwarg and wires it into LB + Last.fm
adapters.
- ``web_server.py`` boot constructs the discovery callable from the
existing matching engine + ``_discovery_score_candidates`` +
Spotify / iTunes clients, passes through to the registry.
- ``refresh_mirrored.py`` adds a small ``_maybe_discover`` helper
that calls ``source.discover_tracks(...)`` between fetch and
``to_mirror_track_dict`` projection — only fires when at least
one track has ``needs_discovery=True``, so the normal Spotify /
Tidal / etc. refresh path stays a zero-cost pass-through.
Tests:
- 5 new adapter tests: default no-op pass-through, LB discovery
with mixed matches/misses, LB no-callable fallback, Last.fm
shares the LB implementation, mirror-dict spotify_hint emit.
- 1 new automation test: end-to-end LB refresh with a stub
discover_callable proves the matched_data lands in
``mirror_playlist_tracks.extra_data`` after the registry
refresh + discover hop.
225 tests across adapter + automation suites green.
Phase 1a of the Discover-to-Sync unification. The mirrored-playlist
refresh handler used to branch per-source through a ~190-line
if/elif chain (Spotify, Spotify public, Deezer, Tidal, YouTube).
Each branch hand-built its own ``extra_data`` JSON for the matched-
data block. With every new source we considered for Sync-page mirror
support (ListenBrainz, Last.fm radio, SoulSync Discovery, iTunes
link), that chain would have grown a new elif.
This commit lifts the per-source logic into the existing adapter
layer and collapses the dispatch to a registry lookup:
- ``core/playlists/sources/deezer.py`` — new adapter so the registry
covers every source the refresh handler previously branched on.
- ``core/playlists/sources/bootstrap.py`` — single helper that builds
a populated registry from injected getter callables. Both
``web_server.py`` boot and the automation test fixtures call it,
so the two construction paths can't drift.
- ``core/playlists/sources/base.py`` — ``to_mirror_track_dict``
projection helper centralises the NormalizedTrack → DB-row
conversion (including the discovered/matched_data and
spotify_hint extra_data shapes the downstream sync + wishlist
consumers already expect).
- Spotify adapter now populates ``extra['discovered']`` + an
``extra['matched_data']`` block when fetching via the authed API,
so Spotify mirrors keep landing pre-discovered (matches the
pre-refactor contract pinned by
``test_spotify_refresh_writes_to_db``).
- Spotify-public adapter populates ``extra['spotify_hint']`` so the
discovery worker can skip its search step and jump straight to
enrichment for the known track ID.
- All artist-name fields now project to first-artist-only across
every adapter — matches the pre-refactor mirror_playlist DB shape
(``t.artists[0]``).
``refresh_mirrored.py`` shrinks ~190 → ~80 lines and keeps:
- the file/beatport unrefreshable-source filter,
- URL extraction from ``description`` via ``require_refresh_url``
for spotify_public + youtube,
- the Spotify-public → authed-Spotify fallback when the user is
signed in (handler-level branch, not in any adapter),
- the Tidal-not-authenticated soft-skip log (skip, not error),
- existing-extra_data preservation across refreshes,
- the ``playlist_changed`` automation event emit on track-set delta.
Test scaffolding:
- ``_build_deps`` in ``tests/automation/test_handlers_playlist.py``
now builds a default registry from the passed clients via
``build_playlist_source_registry``, so existing refresh tests
exercise the same path without per-test changes. New tests cover
Tidal-not-authed soft-skip, Deezer refresh writes plain tracks,
YouTube refresh reads URL from description, and Spotify-public
uses authed Spotify when signed in.
- 4 new adapter tests for Deezer projection +
``to_mirror_track_dict`` (minimal track, Spotify matched_data,
Spotify-public spotify_hint).
- ``playlist_source_registry`` field on ``AutomationDeps`` defaults
to ``None`` so the other 5 automation test files (which don't
exercise refresh_mirrored) keep working unchanged.
220 tests across automation + adapter suites green.
The Auto-Sync schedule board was detecting its own automations by
checking `group_name === 'Playlist Auto-Sync' || name.startsWith('Auto-Sync:')`.
That's fragile — renaming the row from the Automations page silently
hands ownership back to the read-only Automation Pipelines tab and the
board stops managing it.
This commit replaces the string convention with an explicit
`automations.owned_by` TEXT column:
- Migration `_add_automation_owned_by_column` adds the column and
backfills `'auto_sync'` for existing rows that match the legacy
`group_name`/`name`-prefix pattern, so users running the migration
don't lose their schedules.
- `database.create_automation` and `database.update_automation` accept
`owned_by` (the latter via its `allowed` kwarg set).
- `core/automation/api.py` forwards `owned_by` on both POST and PUT.
Missing field is left as None, preserving today's behavior for every
caller that doesn't opt in.
- The Auto-Sync schedule board posts `owned_by: 'auto_sync'` and the
detection helper now prefers that signal, falling back to the legacy
name/group convention so any hand-rolled rows still show up.
Tests: three new cases in `tests/automation/test_automation_api.py`
covering create-with-owned-by, create-without (defaults to None), and
update set/clear. The fake DB grew the matching kwarg.
Three changes folded into one perf+cleanup pass:
1. Indexed fast path for the per-artist pool fetch. The previous
`search_tracks(artist=name)` call hit
`unidecode_lower(artists.name) LIKE ?`, a function-in-WHERE that
can't use `idx_artists_name`. New `MusicDatabase.get_artist_tracks_indexed`
does a two-step lookup: exact-name match (indexed) plus a
case-insensitive fallback, then `tracks WHERE artist_id IN (...)`
via `idx_tracks_artist_id`. Drops per-artist fetch from seconds to
milliseconds for the common case. The sync helper falls back to
the old LIKE-based `search_tracks` only when the indexed lookup
finds nothing, preserving diacritic recall and `tracks.track_artist`
feature-artist matches with zero regression.
2. Public text-normalization helper. Lifted the body of
`MusicDatabase._normalize_for_comparison` into
`core/text/normalize.py:normalize_for_comparison` so callers outside
the database layer (matching engine, sync pool, future import-side
comparisons) don't reach across the module boundary into a
leading-underscore "private" method. The DB method now delegates,
so existing internal call sites stay untouched. Sync's lazy pool
now imports the public helper.
3. Artist-name walker extracted. `_artist_name` at module level in
`services/sync_service.py` replaces two near-identical inline
str-or-dict-or-fallback walkers (one in `sync_playlist`, one in
`_find_track_in_media_server`). Returns `''` for None instead of
the literal string `'None'`.
Plus three small tidies from the same review:
- `_POOL_FETCH_LIMIT = 10000` constant in place of the literal at the
pool-fetch call site.
- Trimmed the verbose docstring + comment block on the pool helper.
- Set-intersection predicate for the trigger-shape reset in
`core/automation/api.py` instead of a two-line `or` chain.
Also removed the duplicate `_get_active_media_client()` call at
sync_service.py:212/214 — pre-existing wart that was sitting in the
same block I was editing.
Tests: 21 new tests across `tests/database/`, `tests/sync/`, and
`tests/text/`, plus updates to the existing pool tests to cover the
new fast/fallback split. Full suite stays green (3953 passing).
The Playlist Auto-Sync schedule board was showing "next in 8h" on every
card regardless of the configured interval. Root cause: backend stores
next_run as a naive UTC string ("2026-05-25 05:00:00") and the new
auto-sync renderer was parsing it with plain `new Date(...)`, which
treats unmarked timestamps as local time. On Pacific time that offsets
the displayed countdown by ~8 hours. Auto-Sync now routes through the
existing `_autoParseUTC` helper that the rest of the Automations page
already uses, so countdowns line up with the wall clock.
A separate correctness fix in the automation update API: when a PUT
changes `trigger_type` or `trigger_config`, the stored `next_run` is
now blanked before the engine reschedules. Previously the scheduler's
restart-survival path would preserve a stale future timestamp from the
prior interval, so dragging a playlist from the 8h column to the 1h
column kept firing at the old 8h mark. Boot-time restart behavior is
unchanged — only user-driven schedule changes reset the clock.
Modal restyle: the Auto-Sync manager's hardcoded sky-blue palette is
replaced with `var(--accent-rgb)` everywhere so the modal honors the
user's chosen accent color. Tinted glow on the modal border, tabbed
header active state, scheduled-playlist chips, scrollbars, and a new
drag-over highlight on columns all follow the accent theme. The
column drag-over state is wired through new ondragleave handling so
the highlight clears reliably when leaving a column.
Expose playlist-native run and status endpoints that reuse the shared mirrored playlist pipeline engine while routing progress into playlist UI state.
Add a Run Pipeline action to mirrored playlist cards and modals with live status polling, and make the shared pipeline lock atomic for manual and scheduled callers.
Extract the all-in-one mirrored playlist lifecycle into core/playlists/pipeline.py so automation becomes a thin adapter. Preserve the existing automation action and behavior while making the pipeline reusable by future direct playlist UI controls.
Centralize mirrored playlist source reference normalization so edited links and IDs are stored consistently. Preserve URL-backed refresh refs, surface missing-source refresh failures, count background sync failures in pipeline summaries, and retry guarded automation skips after a short delay instead of losing a scheduled run. Add focused coverage for source refs, mirrored playlist source updates, refresh failures, and guarded retry behavior.
