new opt-in feature — default keeps everything, so nothing changes unless a channel sets it.
- history got smarter: youtube rows now carry channel_id + published_at (mined from
search_ctx) + a pruned_at marker. new queries: youtube_channels_with_downloads,
youtube_channel_episodes, mark_download_pruned.
- core/video/retention.py (pure): parse_retention + episodes_to_prune — age by UPLOAD
date (published_at, filename fallback); 'count_N' keeps newest N, 'days_N' keeps last N
days; undated episodes never pruned.
- handler auto_video_clean_youtube_episodes: for each channel with a policy, delete the
out-of-window video + its -thumb/.nfo sidecars (only the exact recorded dest_path, never
walks folders), then mark the history row pruned — KEPT so the scan never re-downloads it.
- 'Keep' dropdown in each channel's cog modal (Everything / Last 30 episodes / Last 3 / 6
months); playlists excluded. seeded daily automation + register/block/label/icon/sort.
pure math + handler (i/o injected) + the prune-keeps-dedup DB contract all tested.
dual triggers on one automation aren't a thing (one trigger per row, same as music), so
this adds a second automation sharing the handler — like Auto-Deep Scan TV/Movie already
do. 'Auto-Update Video Database (Hourly)' runs the same cheap incremental server-read on
an hourly schedule, so MANUAL library additions (which Plex auto-scans) appear within the
hour instead of waiting for the weekly deep scan; the existing after-scan one still gives
instant updates for SoulSync-initiated scans.
new action_type video_update_database_hourly (seeder keys on action_type) reusing
auto_video_update_database in incremental mode; registered, seeded (schedule 1h, 15m
initial delay), UI block, label/icon, sort order. tested.
the open button now opens the channel inside soulsync (channels-as-shows) via
video-open-detail {kind:'channel', source:'youtube', id}, matching how movie/show
cards open their in-app detail. enqueue_ctx now carries channel_id (new downloads)
and youtube_video_detail returns it too (so older downloads resolve the channel in
the drawer). drawer gets an 'Open channel' button alongside 'Open on YouTube';
falls back to the youtube video link only when the channel id is unknown.
- EPISODE downloads now show the specific episode: still + 'S02E05 · air date' + that
episode's own title + synopsis (was just the show synopsis). meta endpoint takes
?season=&episode= → engine.tmdb_season; show cast/logo stay as context.
- YOUTUBE drawer: big 16:9 thumbnail + channel · duration · views · upload date +
description. new yt-meta route → db.youtube_video_detail (cached duration/views).
- AVAILABILITY line: the chosen source's free-slot/queue/speed snapshot, stashed in
search_ctx at grab time (build_download_record) → 'Availability: ✓ free slot · queue 0
· 2.1 MB/s'.
drawer also reworked into clean youtube / episode / movie-show branches; first paint
shows 'Loading…' instead of flashing 'no synopsis'. tested.
drawer upgrades (the TMDB call we already make returns all this — just render it):
- FIX cast photos — used c.profile_url; the field is c.photo. now shows headshots.
- title LOGO header (falls back to text) + meta line (year · ⭐rating · runtime · network).
- tagline, director/creator, ▶ trailer link, and a 'Watch on' provider-logo row.
- meta endpoint widened to return logo/cast-photo/rating/runtime/crew/trailer/providers.
OMDb spam fix (the 'Request limit reached!' tracebacks):
- the new bulk schedule-refresh called refresh_show_art per show, which did an OMDb
ratings backfill each → blew the daily quota. refresh_show_art gains with_ratings;
the automation passes False (it only needs episode schedules).
- _backfill_ratings now latches _omdb_blocked on OMDbAuthError → one quiet warning +
stops calling OMDb for the rest of the process, instead of a traceback per show.
tested (cast/logo/trailer contract, no-ratings plumbing, the latch-off behavior).
the airing automation reads the LOCAL episodes table, which only refreshes on initial
enrichment / manual re-match / lazy on-view — so a newly-announced or rescheduled
episode could be missed indefinitely. (the deep TV scan is a SERVER scan, unrelated.)
new daily automation re-pulls TMDB episode schedules (air dates/stills) for still-airing
watchlist shows so the calendar is fresh when the airing run reads it:
- db.watchlist_continuing_shows(): effective-watchlist library shows that are still airing
(skips tmdb-only follows — no episodes — and ended/canceled shows; keeps unknown status).
- handler mirrors the standard: injected fetch_shows + refresh_show seams, live per-show
progress, _manages_own_progress; reuses engine.refresh_show_art (match + episode cascade).
- seeded daily at 23:00 (2h before the 01:00 airing run), registered, UI block, sorted in
the page right before the airing automation.
seam-level tested + a regression that it's seeded before the airing run.
surface why start_search failed (slskd error string) in the automation log instead
of a generic 'not responding?'. seam now returns (candidates, error); handler
tolerates bare list/None too (test fakes). so we can see the real reason.
'No search results' was masking the case where slskd never accepted the search
(not configured / errored / rate-limited) — that returns instantly, which is why
some show up as no-results FAST instead of after the search window.
_search_for_retry now flags started=False (+ the slskd error) when there's no
search id; the processor logs "Search didn't run for 'X' — slskd not responding?"
(warning) and tallies it separately. so the next run tells us if the fast
no-results are really slskd refusing searches vs the source genuinely being empty.
a completed youtube download is removed from the wishlist (correct), but the
channel/playlist scans' downloaded_ids seam was stubbed to [] — so the next scan,
not seeing it wishlisted and not knowing it was downloaded, re-added it (still in
the channel's recent uploads / last-N net) → re-download loop for the last-N
videos every scan.
fix: db.downloaded_youtube_video_ids() pulls completed youtube grabs from the
permanent download history; both scans' _default_downloaded_ids now use it. so a
downloaded video stays gone. tested (db method + the scans already exclude
downloaded_ids in their pure logic).
'no acceptable release' was ambiguous — couldn't tell if slskd returned nothing
or returned hits the quality profile rejected. now each item logs 'No search
results for X' vs 'N result(s) for X, none accepted — <reason>' (reason from the
top hit), and the summary tallies '… · N had no results, M rejected on quality'.
makes the slskd-finds-nothing-for-video situation legible.
storage + wiring + API for the channel settings modal (UI next):
- db.get/set_channel_settings(channel_id): per-channel {custom_name, quality}
in the settings KV store (no schema change); blanks clear the override.
- enqueue applies them: custom_name overrides the $channel folder token; a
quality override is stashed in the download row's search_ctx. the worker reads
it back (quality_override_from_download) and uses it instead of the global
youtube quality profile. global default still applies when no override.
- API GET/POST /youtube/channel/<id>/settings (GET also returns the global
default quality for the modal's 'using default' hint).
pure helpers (enqueue_ctx, quality_override_from_download) + db + api seam-tested.
closes the real gap: a restart kills the yt-dlp worker threads but leaves their
rows at 'downloading', which the pump counts as busy -> the queue wedges and they
never finish.
track live worker dl_ids in _active_worker_ids (added at worker start, dropped in
finally). requeue_orphaned_youtube() puts any 'downloading' youtube row with no
live worker back to 'queued' so the pump re-runs it. after a restart the set is
empty, so all stuck rows recover; during normal operation active ones are
protected (no false positives, no timestamp guessing). the hourly drain calls it
before pumping + logs the count. seam-tested.
- 'Download X Wishlist' -> 'Process X Wishlist' everywhere (label + action_type):
matches the music side's 'process_wishlist', the already-process youtube action,
and reads right (movie/episode do search+pick+download, not just download).
action_types video_download_movie/episode_wishlist -> video_process_*.
- group the new automations as a two-stage pipeline in the seed list + builder
palette: Stage 1 SCANS (people/channels/playlists) fill the wishlist, Stage 2
PROCESSORS (movie/episode/youtube) drain it. icons/labels/drift test updated.
brand-new this session so no migration needed. 337 automation tests green.
the soulseek counterpart of the youtube drain — finally makes the people/airing
scans pay off. two automations (Auto-Download Movie / Episode Wishlist, hourly):
for each wished+released item, do a bounded blocking slskd search, pick the top
ACCEPTED release per the quality profile, and enqueue it exactly like a manual
grab (same add_video_download shape → the monitor finishes + organises it).
same standard as youtube: processes the WHOLE eligible wishlist (no total cap)
but searches a few at a time (max_concurrent, default 3, via a thread pool). a
busy guard skips the next hourly tick while a drain is still working. movies gate
on status='wanted' (skips monitored); episodes are all-wished. items already
downloading are skipped. quiet skip if the library folder isn't set.
reuses the real infra (build_query/slskd_search/_evaluate_hits/start_download/
download_monitor) — pure pick/select/record seam-tested; db queries added. 377
automation+video tests green.
playlists are followable but had no scan. new 'Scan Watchlist Playlists'
automation, sibling of the channel scan but a different rule: a playlist is a
curated finite set, so MIRROR it — wishlist every long-form video you don't have
plus any later additions (no forward-looking baseline, no last-N net).
playlist-as-show: videos wishlisted under the playlist's title, so the worker
files them as 'Playlist Name / Season YEAR / ... - date - title' (the ytdl-sub
tv_show_name-on-a-playlist convention). reuses all the channel plumbing — same
wishlist rows, same Download YouTube Wishlist drain, quality + org template.
seeded every 6h (Auto-Scan Watchlist Playlists); block + registration + icon +
drift test. dedup reuses wishlisted_video_ids_for_channel (parent_source_id=PL).
seam-tested; 944-test sweep green.
they were only builder blocks (not in the Active list). seed them like the
airing job so they appear + run out of the box:
- Auto-Scan Watchlist People — daily 03:00
- Auto-Scan Watchlist Channels — every 6h
- Auto-Download YouTube Wishlist — every 1h
scans no-op cleanly if you follow nothing. softened the download handler: an
unset youtube folder is now a quiet skip (status completed), not a per-run error,
so non-youtube users don't see a recurring failure.
ditch the per-run batch cap (a 200-video backlog would've taken weeks). now the
'Download YouTube Wishlist' automation queues the ENTIRE wishlist as 'queued'
rows, starts up to max_concurrent (default 3) right away, and each finished
download starts the next (one-out-one-in in the worker) so it all drains in a
controlled stream. the knob is 'max simultaneous downloads', not a total cap.
mirrors the music download worker's lesson (cap concurrency + space starts to
avoid yt-dlp 429s) but stays isolated on the video side:
- youtube_download: _pace() staggers fetch starts (3s); start_next_queued()
claims+spawns the next; run_youtube_download chains on finish.
- db.count_active_youtube_downloads() + claim_next_youtube_queued() (atomic,
race-safe). block field batch_size -> max_concurrent.
all seam-tested (pure select + pump); 680-test sweep green.
new 'Download YouTube Wishlist' automation: pushes a polite batch of wished
youtube videos into the shared video_downloads queue + spawns the yt-dlp worker
per video. skips in-flight ones (no double-grab); big backlogs drain over
several scheduled runs (batch_size, default 3). needs the youtube library folder
set.
- download_monitor: SKIP source='youtube' rows (owned by their worker thread, no
slskd transfer to match) — surgical, slskd path untouched.
- db.youtube_wishlist_to_download(): flat newest-first list of wished videos with
channel/title/date/thumb for organising.
block + registration + icon/label + drift test. all seam-tested.
two new video-side automation blocks that keep the wishlist fed:
- scan watchlist people: for each followed person, wishlist every un-owned
movie they acted in or directed (back catalog + upcoming). released ->
wanted, upcoming -> monitored (engine skips it til it's out, promotes on
release). grabs rich detail at add time (backdrop/cast/overview/etc +
provenance) into a new video_wishlist.detail_json col. fast re-runs skip
already-wishlisted + only promote.
- scan watchlist channels: for each followed youtube channel, wishlist new
long-form uploads (shorts excluded). forward-looking from follow time +
a last-N safety net (default 10). diffs against wishlisted/downloaded/
dismissed so it never dupes. pair with a 6h schedule trigger. scan-only;
fulfillment engine comes later.
both are pure handlers with injected seams + full seam tests. add_movie_to_
wishlist gains status + detail_json (promote-only upsert). no schema break,
music side untouched.
Unlike the cleanup twins, backup can't share the music handler — it's a different
DB file. Extract the music backup body into _backup_db_at(db_path, ...) (music
behaviour byte-identical, now a thin wrapper over DATABASE_PATH) and add
auto_backup_video_database pointing at VIDEO_DATABASE_PATH (video_library.db).
New video_backup_database action (scope='video' block + registry), owned_by='video'
system automation on the music cadence (every 3 days).
Tests: a REAL backup behaviour test — music backup lands next to music_library.db,
video backup next to video_library.db, no cross-contamination (this is the whole
reason it can't be shared); scope isolation; single video-owned seed; own handler.
Existing music maintenance tests (22) still green — refactor is non-regressing.
EXPECTED_ACTION_NAMES updated.
ruff S110 flagged two try/except/pass in video handlers that predate this work.
Both are deliberate (a progress-log failure must not abort pruning; a probe's
uncertainty just keeps probing) — extend the existing BLE001 noqa to S110 with
the rationale. ruff check . is clean again.
Same pattern: video_full_cleanup action (scope='video' block + registry), reuses
the shared auto_full_cleanup handler, owned_by='video' system automation on the
music cadence (every 12h). Music copy untouched. Seam tests + EXPECTED_ACTION_NAMES.
Same pattern as phase 2: video_clean_completed_downloads action (scope='video'
block + registry), reuses the shared auto_clean_completed_downloads handler, and
an owned_by='video' system automation on the music cadence (every 5 min). Music
copy untouched. Seam tests for scope isolation, single video-owned seed, and
shared-handler reuse; EXPECTED_ACTION_NAMES updated.
The music side's 'Clean Search History' automation now has a video counterpart so
it appears on the video Automations page too. Distinct action_type
video_clean_search_history (the system seeder keys on action_type, so reusing the
music key would collide), registered to the SAME shared handler so behaviour is
identical, scope='video' block (registry — users can build their own), and an
owned_by='video' system automation on the same 1h cadence. The music action/row
is untouched.
Seam tests: video-scoped only (not on music), music action still music-scoped,
exactly one video-owned system row at the 1h cadence, and it reuses the music
handler. Registration contract (EXPECTED_ACTION_NAMES) updated.
You can eye-add a show to the watchlist before we know its status, so ended/canceled
shows leak in (auto-airing LIBRARY shows already exclude ended ones; explicit follows
don't). Fix it as cleanup-on-process, per Boulder: the daily 'Wishlist Today's Airings'
automation now runs a watchlist-tidy pass first — scans every explicit show follow,
resolves its status (local for owned, TMDB for tmdb-only follows), and removes any
that have ended/been canceled/completed. Only prunes on a DEFINITIVE terminal status;
unknown/lookup-error → left alone. Toggle prune_ended (default on); returns shows_pruned.
DB: followed_shows(). Pure prune_ended_show_follows() with injected seams; seam tests.
The probe fired the instant a batch finished, but a fresh drop takes ~1-2 min to
appear even with the server's auto-scan ON — so it always missed and we crawled
anyway, defeating the optimization. Now probe_present_libraries POLLS each candidate
over a grace window (probe_grace_minutes, default 2), skipping a library's crawl as
soon as the server reports it has the item, and only crawling what's still missing
when grace expires. The probe target for a media type you DIDN'T just download is an
old item the server already has → confirmed instantly, no wait. grace=0 probes once.
Scanning is expensive and most servers auto-ingest new files, so a full crawl after
every download is usually wasted. Stage 1 now probes per library: take the newest
completed grab of that type from download history and ask the server (cheap targeted
search) whether it already has it. If yes, the server auto-picked it up (and the
earlier ones) → skip that library's crawl + poll entirely. Only libraries the server
is missing get rescanned. Always emits so stage 2 still reads the new items in.
- sources: PlexVideoSource.has_item / JellyfinVideoSource.has_item (match movie by
title+year, episode by show+SxE) + video_server_has_item() — conservative, any
uncertainty → False so we scan.
- handler: per-scope skip decision fed by latest_completed + server_has_item seams;
narrows the scan scope to only the missing libraries; toggle skip_if_present
(default on). Returns scanned/skipped for visibility.
Seam tests: skip-both, scan-only-missing, no-history, toggle-off, probe-error→scan;
Plex has_item match tests.
A fixed debounce can't fit a big library — 8500 movies + 4500 shows scan sequentially
through Plex's queue and can take 10-20 min, so the old 120s wait read the DB before
Plex finished and fresh downloads showed up late. Now Stage 1 (video_scan_server)
fires the rescan then POLLS the server until its scan queue goes idle, then emits the
done event.
- sources: PlexVideoSource.is_scanning (section.refreshing + activity feed, scoped by
media_type) and JellyfinVideoSource.is_scanning (scheduled-task state), plus
video_server_scan_in_progress() returning True/False/None.
- handler: pure wait_for_server_scan(scan_status, sleep, …) — grace, then poll every
interval until idle or a generous cap; falls back to the fixed wait only when the
server can't report status (None). debounce_seconds is now that fallback; new
max_wait_minutes caps the poll.
Seam tests for the poll logic (idle/poll/fallback/cap/lost-status), the handler wiring,
and Plex scan-status detection.
A deep scan is the equivalent of music's full refresh — it READS the server's
current state into video.db and prunes what's gone. It should NOT tell Plex to
rescan its disk. The deep-scan action types were wired to auto_video_scan_library
(nudge Plex + read); point them at the read-only auto_video_update_database in
'deep' mode instead. Update-db phase wording no longer says "new" for a full re-read;
deep-scan block descriptions clarify it's a read, not a disk-scan. Registration test
asserts the deep scans route to the read-only handler and never nudge the server.
The deep-scan action types weren't selectable builder actions, and Scan Video Server
/ Update Video Database had no movie-vs-TV dimension — inconsistent with the rest.
- video_deep_scan_tv / video_deep_scan_movies are now proper builder blocks
(Deep Scan TV/Movie Library), not just system-automation action types.
- video_scan_server + video_update_database gain a media_type ('all'|'movie'|'show')
config + selector, threaded through. The post-download chain carries the scope on
the scan-done event, so a TV-only rescan updates only TV (stage 2 inherits it).
- refresh_video_server_sections / Plex+Jellyfin refresh_sections scope the server
nudge to the chosen library; auto_video_scan_library now nudges only its library.
- shared normalize_media_type() in sources; update_database skips cleanly when the
singleton scanner is busy. Defaults stay 'all' so existing chains are unchanged.
Seam tests for refresh scoping, scan-server scope+event, update-db scope/inherit/skip.
Video twin of music's 'Auto-Deep Scan Library', split in two because Movies and TV
are separate libraries — scanning the TV library must not pull in new movies and
vice-versa.
- scanner: add a media_type param ('all'|'movie'|'show', friendly aliases) that
gates the movies vs shows passes (and their pruning), plus an in_progress busy
guard so the singleton scanner can't be stomped by an overlapping run.
- video_scan_library handler: thread media_type through, skip cleanly when the
scanner is busy, and name only the scanned library in the summary.
- two system automations (owned_by=video, weekly deep scan, staggered start delays):
'Auto-Deep Scan Movie Library' + 'Auto-Deep Scan TV Library'. Distinct action
types (video_deep_scan_movies / _tv) because the seeder keys on action_type; both
reuse the one handler, scoped via action_config.
- builder block gains a Library selector (Movies+TV / Movies / TV) so custom scans
can scope too; card label/icon maps cover the video action types.
Seam tests for scanner scope + busy guard, handler scope + skip, registration set.
Field-by-field against the working manual 'add to wishlist', the automation now
matches it on every column EXCEPT the show poster: the get-modal stores
poster_url = '/api/video/poster/show/<library_id>', the automation stored None — so
the wishlist orb fell back to the show's initials and read as 'not matched'. Carry
the same proxy path. With library_id (last commit) + poster_url (this) + the
tmdb_season stills/overviews, an auto-added row is now identical to a manual one.
The real difference from a manual add: the wishlist resolves a show's synopsis +
cast from /api/video/detail/show/<library_id>, and falls back to the TMDB endpoint
only when library_id is absent (which redirects/lacks cast for owned shows). A
manual add sends show.library_id; the automation sent none — so auto-added shows
read as 'not matched' with no synopsis/actors. The handler now carries the show's
library id (the calendar's show_id) through to the wishlist.
Root cause of the metadata loss: a MANUAL 'add to wishlist' gets its episode
data from the TMDB season fetch (engine.tmdb_season — absolute still, overview,
season poster), while the automation read the local DB episodes table, where
stills are frequently empty/Plex-relative. So auto-added episodes came in blank
even after carrying the DB values.
The handler now fetches the SAME TMDB season metadata (cached per season,
injected for tests) and prefers it, falling back to the calendar/DB values if
TMDB is unavailable. Auto-added episodes now match manual ones.
Auto-added airing episodes came in metadata-empty (no synopsis, no still) — the
handler only passed season/episode/title/air_date, dropping the overview the
calendar already returns and never fetching the still URL (calendar_upcoming
only returned a has_still flag, not the URL). Now calendar_upcoming also returns
e.still_url, and the handler carries overview + still_url through. The wishlist
renders the (Plex-relative) still via the same pimg() proxy as the show poster,
so it resolves. Idempotent upsert backfills the already-added empty rows on the
next run.
Two Sonarr-parity features.
1) Per-episode live tracking. "Grab season" was headless (only a button label
changed); episode rows had a status span that was never populated. Now every
episode ROW shows its own live state — Searching → Downloading % → Downloaded
/ Failed — via epTrack() polling /downloads/status?id, matching the inline
movie tracker. Grab season lights all target rows at once; manual + per-source
auto grabs also light their row; reopening the modal resumes tracking in-flight
episodes (resumeEpisodeTracking via /downloads/active + search_ctx match).
Season batch grabs through the same payload as a manual grab (_pickAndGrab →
sendGrab(buildGrabPayload)).
2) Auto-wishlist airing episodes. New daily automation (video_add_airing_episodes):
reads the calendar for episodes airing TODAY for followed shows, skips owned
ones, adds the rest to the wishlist (idempotent). Handler uses injected seams
(calendar read + wishlist write) so it's unit-tested without a DB/server.
Registered + action block + seeded as a daily system automation (01:00),
owned_by=video.
Mirrors the music flow: a finished download batch → refresh the media server → (after
it indexes) pull the new media into the DB — so a downloaded movie/episode shows as
owned without waiting for the 6h scheduled scan.
Two event-based system automations (owned_by='video'):
- 'Auto-Scan Video After Downloads' (video_batch_complete → video_scan_server)
- 'Auto-Update Video Database After Scan' (video_library_scan_completed → video_update_database)
Pieces:
- core/video/download_events.py: a callback registry (core/video can't import the
engine — isolation). The monitor publishes batch-complete; web_server bridges it to
automation_engine.emit('video_batch_complete', …), like music's web_scan_manager.
- download_monitor: fires the batch-complete event once, when the last in-flight
download finishes (none queued/downloading/searching left).
- video_scan_server handler (stage 1): refresh server, wait a debounce for indexing,
then emit 'video_library_scan_completed' (mirrors music's time-based completion).
- video_update_database handler (stage 2): incremental read (newest-first, stop after
25 consecutive known — same as music).
- blocks: 2 video triggers + 2 video actions (scope='video'); registration; seeds.
kettui: seam tests for both handlers (refresh/wait/emit, incremental read, error paths),
the event registry (idempotent register, isolated failures), and the monitor (fires once
on last completion, never while work remains). 298 automation tests + isolation green.
The video side gets its OWN automations at music-side parity, kept separate so
nothing on the music side breaks. First twin: Scan Video Library — tells the media
server to rescan the user's SELECTED video sections (movies/TV, never music), then
reads the result into video.db so freshly-downloaded media shows as owned.
Architecture (scope tags + video twins on the shared engine):
- Handler core/automation/handlers/video_scan_library.py — pure function with
injected I/O (server_refresh / run_video_scan); production lazily binds
refresh_video_server_sections() + the video scanner. Owns its own progress.
Lives on the SHARED automation side so it may import core.video (isolation only
forbids core/video & api/video from importing music, not the reverse).
- blocks.py gains a 'scope' tag ('both' generic / 'video' video-only / absent=music)
+ blocks_for_scope(). The music /api/automations/blocks now filters out video
blocks; new isolated /api/video/automations/blocks serves the video palette.
- automation_engine seeds 'Scan Video Library' (owned_by='video', schedule 6h) so
it appears ONLY on the video Automations page; ensure_system_automations now
honours owned_by + action_config. Music page excludes owned_by='video' rows.
kettui: seam-level tests for every handler path (happy/no-server/scan-error/never-
raises/mode), scope filtering (music excludes video, video gets generics, music
parity preserved), seeding (owned_by + mode), registration drift guard. 39 new
tests; full automation suite (288) + isolation guards green.
Phase 2 of the redesign. The tool that judged quality by extension and auto-dumped
matches into the wishlist is gone; quality scanning is now the reviewed
quality_upgrade repair job.
Removed:
- Frontend: Tools-page Quality Scanner card, its JS handlers/poller/socket listener,
help tooltip + tour entry (webui index.html, core.js, helper.js, wishlist-tools.js).
- Backend: /api/quality-scanner/{start,status,stop} endpoints, the in-memory state +
executor + 1s socket broadcast, the QualityScannerDeps/run_quality_scanner shim.
- core/discovery/quality_scanner.py: the auto-acting worker + deps class (the shared
match/normalize helpers stay — the new job imports them).
Rewired:
- Automation 'start_quality_scan' action now triggers the quality_upgrade repair job
via repair_worker.run_job_now() (AutomationDeps gains run_repair_job_now, drops the
4 scanner fields). Action block's vestigial scope field removed (scope lives in the
job's settings now). NOTE: the 'quality_scan_completed' trigger no longer fires (the
repair job doesn't emit it).
- Updated all automation test _build_deps helpers + conftest tool-progress harness;
deleted the obsolete worker test. 528 affected tests pass; 6123 collect cleanly.
QUALITY_TIERS / _get_quality_tier_from_extension kept (used elsewhere).
Symmetric to the post-download reconcile (which handles ADDITIONS): when a
playlist's membership is re-synced (the mirror step — scheduled refresh or the
manual mirror endpoint), rebuild its folder from current membership WITH prune
IF it's organize-by-playlist. So a track that just LEFT the playlist has its
symlink cleaned up the instant membership changes, not only on the next download.
Factored a shared _rebuild_one_from_db (used by the manual 'Rebuild' button and
the mirror hook) + rebuild_mirrored_playlist_if_organized. Gated to organized
playlists, non-fatal at both mirror call sites.
Now the invariant 'folder = the playlist's current owned members' holds on every
change: additions caught at download, removals caught at mirror. 2 new tests
(removed track pruned; non-organized skipped). 985 + 277 tests pass.
SoulSync standalone matches library tracks without Plex fetchItem,
reports missing counts correctly, and skips server playlist writes.
Automation re-syncs when the mirror grows; after sync finishes, starts
organize download (organize-by-playlist) or wishlist processing.
UI: Spotify URL playlist-folder controls, organize toggle layout in the
discovery modal, reload organize preference when reopening Download Missing.
Co-authored-by: Cursor <cursoragent@cursor.com>
Persist organize_by_playlist on mirrored playlists and run playlist-folder
downloads from the auto-sync pipeline instead of the global wishlist phase.
Register SoulSync library rows after playlist-folder post-processing, route
failed organize batches to the wishlist correctly, and skip sync-time
unmatched wishlist only when organize download handles retries.
Invalidate stale playlist track caches on refresh (Spotify and Deezer ARL),
re-mirror on refetch, and improve standalone playlist modals (re-analysis,
Open in Mirrored). Add filesystem missing-track detection and tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
The DB-update + deep-scan automation monitor used a hard 2-hour TOTAL cap
(while elapsed < 7200). It tracked progress but only used it to print a stall
warning — the only thing that actually timed out was wall-clock. So a large
library that scans for >2h while progressing fine (reported: 4781 artists) trips
the cap and the automation card flips to 'error: timed out after 2 hours' even
though the scan thread is healthy and still running (the timeout never cancels
it, which is why it keeps progressing in the logs after the 'error').
Time out on STALL, not total runtime:
- 30 min with NO progress -> error ('stalled'); catches a genuinely hung scan.
- 10 min idle -> warning (repeats); unchanged heads-up.
- 24h absolute backstop, purely a runaway-loop guard.
- An actively-progressing scan keeps resetting the idle clock, so it never
times out no matter how many hours the whole library takes.
- Progress is judged on (processed, progress, current_item) so a slow stretch
where the rounded % holds steady (but the artist keeps changing) isn't a
false stall.
The decision is extracted into a pure, testable scan_wait_action(); both the
deep-scan and full-refresh handlers share the monitor loop, so both are fixed.
Tests: tests/test_scan_wait_action.py (9) — headline regression (5h/12h total
but progressing -> 'continue', not timeout), finished/stall-warn/stall-timeout/
abs-cap thresholds, and ordering. 280 automation tests still pass.
Post-incident hardening. A WAL-mode DB corrupted (most likely an interrupted
write during a hard restart), and the backup routine made it unrecoverable:
it (a) never checked integrity, so src.backup() faithfully copied the corrupt
pages into every rolling backup, and (b) pruned oldest-by-mtime, so each new
corrupt backup evicted the last good one. Result: all snapshots poisoned.
New core/db_integrity.py (pure, unit-tested):
- quick_check()/is_healthy(): fast read-only PRAGMA quick_check probe.
- safe_backup(): verifies the SOURCE is healthy BEFORE the Online-Backup copy
and the RESULT after; refuses + discards rather than save a corrupt copy.
- prune_backups(): rotation that NEVER deletes the most-recent verified-healthy
backup, even to honor max_keep — so a run of bad backups can't drop your last
good snapshot.
Wired into BOTH backup paths (the /api/database/backup endpoint and the
auto_backup_database automation handler) — they now refuse on integrity failure
(409 / error status, existing backups untouched) and prune safely.
Tests: tests/test_db_integrity.py (8) using REAL temp DBs incl. a physically
corrupted one — proves refuse-corrupt-source, discard-corrupt-result, and the
exact incident scenario (newest backups corrupt -> the older healthy one is
protected from pruning). Existing maintenance-handler backup test still green
(29 passed). compile + ruff clean.
NOTE: this prevents silent backup poisoning; it does NOT stop the underlying
corruption. Follow-ups still worth doing: WAL-checkpoint on clean shutdown +
a periodic live-DB integrity alert (so corruption is caught on day 1).
Pipeline-driven Auto-Sync runs against any ListenBrainz playlist
(Weekly Jams, Weekly Exploration, Top Discoveries, etc.) would sit
on ``Refreshing: "<name>"`` with no UI updates for 5-7 minutes
before the pipeline progressed. Two real bugs stacked:
1. **Double discovery.** The refresh handler called
``_maybe_discover`` (matching engine, per-track Spotify/iTunes/
Deezer matches) inline for any source returning
``needs_discovery=True`` tracks. Phase 2 of the pipeline then
ran the SAME matching engine via ``run_playlist_discovery_worker``
on the same tracks. The refresh-side run blocked the loop with
zero progress emission; Phase 2's already has the timed
progress-poll pattern. So LB tracks discovered twice, the first
time silently.
Pipeline now sets ``skip_discovery=True`` on its refresh config.
The handler honors the flag and lets Phase 2 handle discovery
end-to-end. Standalone callers (Sync-page tab, registration
action) leave the flag unset so they still get matched_data
on refresh.
2. **No targeted LB refresh.** The LB adapter's ``refresh_playlist``
called ``manager.update_all_playlists()`` — the only refresh
entry-point the manager exposed — which re-pulls every cached
LB playlist's details from the API (~12+ round-trips) even
when only one playlist needed refreshing. Wasteful;
tax-on-everyone for one-playlist work.
Added ``LBManager.refresh_playlist(mbid)`` — reads the cached
playlist_type, fetches just that playlist's details, runs the
normal ``_update_playlist`` upsert path. Defaults type to
``user`` for un-cached mbids so new-playlist discovery still
works. Skips ``_cleanup_old_playlists`` and
``_ensure_rolling_mirrors_from_cache`` (wasted work for a
single-playlist refresh).
Also: killed a silent ``except Exception: pass`` in the LB
adapter's old refresh wrapper that was masking every LB API
failure as a stale-cache hit. Refresh errors now log with full
traceback at warning level and propagate ``None`` so the outer
handler at ``refresh_mirrored.py:104`` counts the error and
surfaces it to the run-history error tally.
Pinned with 12 new unit tests across:
- ``tests/test_listenbrainz_manager.py`` (8): targeted refresh
happy path, unauthenticated guard, empty-mbid guard, upstream
``None`` return, default playlist_type for unknown mbid,
exception propagation, cost guard skipping cleanup, skipped-
when-unchanged signal
- ``tests/test_playlist_sources_adapters.py`` (3): adapter uses
targeted call (not legacy), adapter returns ``None`` on manager
error (not silent swallow), adapter resolves synthetic series
ids before calling the manager
- ``tests/automation/test_handlers_playlist.py`` (1):
skip_discovery flag bypasses ``_maybe_discover`` end-to-end
Self-review pass on ec4a55c1 — applying the standing kettui-grade
rule (see memory/feedback_always_build_kettui_grade.md). Three issues
that would have surfaced on review:
1. Silent tz fallback to UTC
``_resolve_tz`` returned UTC when the IANA name was unknown — no
log, no warning. User on a host without ``tzdata`` who configures
``America/Los_Angeles`` got schedules running silently at UTC
offset with no way to debug. Now logs WARNING once per unknown
name (deduped via ``_UNKNOWN_TZ_WARNED`` set so a misconfigured
row doesn't spam every poll cycle) and the log line names BOTH
real causes — typo or missing tzdata — so the user can fix from
a single grep.
2. ``weeks`` unit drift from engine
I added ``'weeks': 86400*7`` to ``_INTERVAL_MULTIPLIERS`` but the
engine's existing ``_calc_delay_seconds`` only recognises
minutes/hours/days. Until PR 2 collapses both paths through this
function, any row whose config snuck through with ``unit='weeks'``
would get scheduled by the engine as 1-hour and by this function
as 7-day — drift between two live implementations. Dropped
``weeks`` from the map to match the engine. Added a comment
pinning the map to the engine's contract and a regression test
that asserts ``unit='weeks'`` falls back to the same hours
default the engine produces.
3. DST edge cases unverified
The module docstring claims DST-aware via ``zoneinfo`` but no test
pinned the spring-forward gap (02:30 LA on DST-Sunday doesn't
exist) or fall-back ambiguity (01:30 LA on fall-Sunday happens
twice). Three new tests:
- ``test_dst_spring_forward_lands_after_the_gap`` — pins that the
function doesn't crash + lands on a real instant past ``now``.
- ``test_dst_fall_back_handles_ambiguous_local_time`` — pins
zoneinfo's default-earlier-instant resolution for ambiguous
local times (01:30 PDT vs 01:30 PST → picks PDT).
- ``test_weekly_across_dst_boundary_keeps_local_wall_clock`` —
pins that a "every Sunday at 09:00 LA" schedule keeps the
local wall clock across the boundary even though the UTC
equivalent shifts by an hour. This is the exact bug class
that caused the May 2026 "next in 8h" tz mismatch.
Also loosened ``tzdata==2026.2`` to ``tzdata>=2024.1``. IANA tz data
changes a few times a year for real-world DST policy updates; pinning
to one snapshot would freeze the app's tz knowledge to the build date
and miss future government-mandated rule changes.
41 schedule tests pass (5 new); 240 across the full automation suite.
Ruff clean.
Backend plumbing for upcoming weekly + monthly Auto-Sync schedules.
PR 1 of 4 in the schedule-types feature — see
``memory/project_auto_sync_schedule_types.md`` for the full plan.
Net behaviour change in this PR: zero. The automation engine still
computes next_run via its existing inline ``_calc_delay_seconds`` /
``_next_weekly_occurrence`` helpers; this module is unused until PR 2
wires the engine through. Lands separately so the foundation can sit
on dev for a beat before the engine change.
``core/automation/schedule.py:next_run_at(trigger_type, trigger_config,
now_utc, default_tz)``:
- Pure function. ``now_utc`` injected (tests freeze time without
monkeypatching ``datetime.now``); ``default_tz`` injected (so daily /
weekly / monthly schedules compute against the USER's timezone, not
the server's — the same class of bug that produced the May 2026
"Auto-Sync next in 8h" timezone fix).
- Returns aware-UTC ``datetime`` ready to serialise to the DB
``next_run`` column, or ``None`` for unrecognised / event-based
triggers (callers should not write a next_run for those).
- Naive ``now_utc`` inputs are assumed UTC for defensive symmetry
with the engine's DB-string parser convention.
Trigger types covered:
- ``schedule``: ``{interval: N, unit: 'minutes'|'hours'|'days'|'weeks'}``
— matches engine's existing ``_calc_delay_seconds``. Unknown unit
defaults to hours; zero/negative interval clamps to 1 (preserves
the engine's guard against scheduling for the past); non-numeric
interval falls back to 1.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — DST-aware via
``zoneinfo``; ``tz`` falls back to ``default_tz``; unknown IANA
string falls back to UTC; garbage ``time`` falls back to 00:00.
- ``weekly_time``: ``{time, days: ['mon',...], tz}`` — empty / all-
invalid ``days`` list means "every day" (matches engine fallback);
abbreviations case-insensitive; 8-day scan finds the next match.
- ``monthly_time``: ``{time, day_of_month: 1-31, tz}`` — NEW shape.
Day clamped to [1, 31]. Months too short for the target day clamp
to the LAST valid day rather than skipping a month (standard cron
convention; running a day early in February is less surprising
than missing the whole month). 12-iteration loop cap so a
pathological config can't infinite-loop.
Tests (36 cases, all passing):
- Interval: every unit, unknown-unit fallback, zero/negative/garbage
interval clamp, tz field ignored on interval (wall-clock-independent).
- Daily: today-at-future-time runs today, today-at-past-time rolls to
tomorrow, exact-match rolls to tomorrow (no schedule-now-then-schedule-
again-immediately), user-tz vs server-tz, default_tz fallback,
garbage time / unknown tz defensive returns.
- Weekly: same-day-still-future qualifies, same-day-past rolls to next
allowed day, wraps across week boundary, empty days = every day,
garbage abbreviations dropped, case-insensitive, tz across day
boundary (LA Wednesday evening is Thursday UTC).
- Monthly: target day this month, rolls to next month when passed,
Feb 31 → Feb 28 / Feb 29 leap year, day_of_month above 31 / below
1 clamp, Dec → Jan year roll, user-tz pre-midnight edge case.
- Result-shape contract: every returned datetime is aware UTC at
offset zero (engine relies on this when serialising to the
``next_run`` string column).
Added ``tzdata==2026.2`` to requirements.txt. Windows ``zoneinfo`` and
minimal Docker base images ship without the system tz database;
without ``tzdata`` ``ZoneInfo('America/Los_Angeles')`` raises
``ZoneInfoNotFoundError`` and the helper silently falls back to UTC.
No WHATS_NEW entry — no user-visible behaviour change in this PR.
PR 2 (engine wire-through) will land the user-facing changelog entry
when ``monthly_time`` becomes a real schedulable trigger.
Adds ``discover_tracks(tracks) -> List[NormalizedTrack]`` to the
PlaylistSource interface. Sources whose tracks already carry
provider IDs (Spotify, Tidal, Qobuz, YouTube, Deezer, Spotify
public, iTunes link, SoulSync Discovery) inherit a no-op default;
ListenBrainz + Last.fm override to run the matching engine.
This closes the last gap before LB / Last.fm / SoulSync Discovery
can land as Sync-page mirror sources: the refresh handler now
calls ``source.discover_tracks(...)`` whenever a source returns
tracks with ``needs_discovery=True``, so mirrored LB rows arrive
already discovered + ready for the sync pipeline. Previously, LB
playlists ran through a separate state-machine worker tied to the
Discover-page UI, with results stored in ``discovery_cache``
instead of ``mirrored_playlist_tracks.extra_data``.
Changes:
- ``core/playlists/sources/base.py`` — PlaylistSource switches from
Protocol to ABC so a concrete default for ``discover_tracks``
can live on the base class. The four real-work methods stay
``@abstractmethod``; instantiating an adapter that forgets one
fails loudly at construction.
- ``core/discovery/matching.py`` (new) — pure ``match_mb_tracks``
helper that runs Strategy-1-only matching-engine queries against
Spotify (primary) or iTunes (fallback). No state machine, no
discovery-cache writes, no wing-it stub — that richer flow stays
in ``core/discovery/listenbrainz.py`` for the Discover-page UI.
- ``ListenBrainzPlaylistSource`` + ``LastFMPlaylistSource`` take
an optional ``discover_callable`` constructor arg. Last.fm reuses
the LB implementation since the track shape is identical.
- ``bootstrap.build_playlist_source_registry`` accepts a
``discover_callable`` kwarg and wires it into LB + Last.fm
adapters.
- ``web_server.py`` boot constructs the discovery callable from the
existing matching engine + ``_discovery_score_candidates`` +
Spotify / iTunes clients, passes through to the registry.
- ``refresh_mirrored.py`` adds a small ``_maybe_discover`` helper
that calls ``source.discover_tracks(...)`` between fetch and
``to_mirror_track_dict`` projection — only fires when at least
one track has ``needs_discovery=True``, so the normal Spotify /
Tidal / etc. refresh path stays a zero-cost pass-through.
Tests:
- 5 new adapter tests: default no-op pass-through, LB discovery
with mixed matches/misses, LB no-callable fallback, Last.fm
shares the LB implementation, mirror-dict spotify_hint emit.
- 1 new automation test: end-to-end LB refresh with a stub
discover_callable proves the matched_data lands in
``mirror_playlist_tracks.extra_data`` after the registry
refresh + discover hop.
225 tests across adapter + automation suites green.
