On a deep scan the progress bar hits 100% when the last item is read, but the prune
(delete orphaned rows + cascades) runs AFTER that and — on a big cleanup — takes a
few seconds, during which the scan still reads as running (can't start a new one,
workers still paused). Looked stuck at 100%. Now the scanner sets a 'cleaning up
removed movies/shows' phase around the prune so the UI shows it's finalizing, not
frozen. Test spies the phase at prune time. 18 scanner tests pass.
1) Scan only the MAPPED libraries — never fall back to 'all'. The scan path used
_sections/_views with the selected name, but an empty name returned ALL sections
of that type — so a missing/unreadable selection silently scanned every library
(how the 4K movie + 'YouTube' TV libraries leaked in as movies/shows). New
_scan_sections / _scan_views return [] when a kind isn't mapped; available_libraries
still lists all (for the Settings dropdown). Now an unmapped kind scans NOTHING.
2) A library scan (full/incremental/deep) now pauses EVERY enricher, including the
YouTube date enricher — a separate singleton outside engine.workers that kept
running through scans. pause_for_scan/resume_after_scan pause+resume it too, only
if it wasn't already manually paused (never override the user).
kettui: source-scope tests (mapped-only / unmapped-scans-nothing / listing still
shows all) + engine pause tests (pauses+resumes YT / preserves manual pause).
123 scanner/source/enrichment tests + isolation guard green.
The details backfill queue is keyed on tmdb_id, but the gate (hasattr match) let the
TVDB worker run it too — feeding TMDB ids to TVDB's /series/{id}/extended (→ 404 on
every show) and double-processing each show (TMDB backfilled it, TVDB then 404'd but
still logged 'Backfilled'). Gate on self.service=='tmdb' so only the TMDB worker runs
it. Regression test: the TVDB worker no-ops (never calls its client, leaves the item
pending). 95 enrichment tests pass.
The media server pre-matches shows/movies (tmdb_id set), so the enrichment matcher
skips them and never fetches TMDB details — leaving details-only fields like `status`
(airing vs ended) blank on almost the whole library. That's why the watchlist's
airing-shows default only ever saw the handful of shows whose detail page had been
opened (the one path that force-fetches details). Library here: 3,371 matched shows,
only 18 with status.
Fix: a one-time-per-item details backfill that runs in the enrichment worker's idle
loop (after the episode-sync pass). New `details_synced` marker column on shows+movies;
detail_backfill_next/mark_details_synced/pending_count; worker._detail_backfill_one()
re-fetches an already-matched item's TMDB details and gap-fills (never clobbers server
data), then marks it done so it's attempted once. No re-scan needed — it heals the
existing library in place, and once status is populated the airing-watchlist reflects
real TV.
It's a background gap-fill on already-matched items (like episode coverage), so it
doesn't block the worker's 'Complete' status. kettui: DB seam tests + worker tests
(fills status / enrich-by-id / marks-done-when-absent). ruff + isolation guards green;
94 enrichment tests pass.
The video side gets its OWN automations at music-side parity, kept separate so
nothing on the music side breaks. First twin: Scan Video Library — tells the media
server to rescan the user's SELECTED video sections (movies/TV, never music), then
reads the result into video.db so freshly-downloaded media shows as owned.
Architecture (scope tags + video twins on the shared engine):
- Handler core/automation/handlers/video_scan_library.py — pure function with
injected I/O (server_refresh / run_video_scan); production lazily binds
refresh_video_server_sections() + the video scanner. Owns its own progress.
Lives on the SHARED automation side so it may import core.video (isolation only
forbids core/video & api/video from importing music, not the reverse).
- blocks.py gains a 'scope' tag ('both' generic / 'video' video-only / absent=music)
+ blocks_for_scope(). The music /api/automations/blocks now filters out video
blocks; new isolated /api/video/automations/blocks serves the video palette.
- automation_engine seeds 'Scan Video Library' (owned_by='video', schedule 6h) so
it appears ONLY on the video Automations page; ensure_system_automations now
honours owned_by + action_config. Music page excludes owned_by='video' rows.
kettui: seam-level tests for every handler path (happy/no-server/scan-error/never-
raises/mode), scope filtering (music excludes video, video gets generics, music
parity preserved), seeding (owned_by + mode), registration drift guard. 39 new
tests; full automation suite (288) + isolation guards green.
The foundation for the 'Scan Video Library' automation (the video twin of music's
Scan Library). PlexVideoSource.refresh_sections() triggers a Plex scan on the selected
movie/TV sections (section.update()); JellyfinVideoSource.refresh_sections() POSTs
/Items/{id}/Refresh per selected video view (the GET-only _make_request can't POST).
Module helper refresh_video_server_sections() gets the active source and refreshes —
scoped to the user's chosen video libraries (Settings), so it scans the CORRECT media,
not music. Video-only, additive; isolation guard + (my code) ruff clean.
When a grabbed release fails (transfer error / peer-cancel / never lands), the engine
now retries instead of giving up — the music-style depth:
- Grab stores the OTHER accepted results as a retry pool + the search context (schema
v16: candidates / search_ctx / tried_queries / tried_files / attempts).
- core/video/retry.py (pure, tested): plan_retry() → try the next-best candidate; when
the pool is dry, next_query() generates an ALTERNATE query (movie: drop the year; TV:
numbering variants) to re-search; budget MAX_ATTEMPTS=6. merge_candidates dedupes
against already-tried releases.
- Monitor: on failure, _fail_or_retry hops to the next candidate inline; if none, flips
the row to a new 'searching' state and a background requery thread re-searches the
alternate query, evaluates against the profile, and starts the best fresh hit — or
marks failed once truly exhausted. 'searching' rows are owned by their thread.
- Page: 'Searching' status (Trying another release…) + a 'Nx' attempt badge.
16 tests (retry engine + candidate-retry transition); ruff clean on touched files.
The poll capped at 32s but slskd results trickle in over ~50s (the music side waits
the whole search_timeout), so slow searches like 'Project Hail Mary' returned 'none'
before results landed. Now:
- /search/start returns poll_ms (slskd search_timeout + 8s); the UI polls that long
(capped 80s), streaming results as they arrive, stopping early only once results
clearly plateau (≥20s + stable) or hit 25.
- /search/poll returns total_files; when 0 video releases but slskd DID return files,
the panel says 'returned N files, but none are video — likely audio/other for this
title' instead of a blank 'none' (Soulseek is audio-heavy; many movie titles are
audiobooks there). slskd_search.poll_search() returns {hits, total_files}.
Tests green, ruff + balance clean.
Fixes the 'cancelled but still shows running' stuck bug and adds real depth:
- classify_state now distinguishes 'cancelled' from 'failed'.
- Monitor is robust to slskd forgetting a transfer: if it's gone, it first tries to
complete from the FILE on disk (survives the music 'Clean Completed Downloads'
auto-clear), else counts misses and fails the row after ~8 polls instead of hanging
on 'downloading' forever. Cancelled transfers → cancelled status.
- POST /downloads/cancel (slskd DELETE transfer + mark cancelled) and /downloads/retry
(re-grab the same release). get_video_download(id) added; clear includes cancelled.
process_download stays pure (fs/slskd injected); 15 tests, ruff + guard clean.
Phase B (page: tabs/queue/history/cancel+retry buttons) next.
The old search did a 4.5s slskd search + 8s wait; slskd responses trickle in over
10-30s, so the window closed before results arrived (you'd see them in slskd but the
panel said none). Now it works like the music side:
- slskd_search.start_search() (uses the shared soulseek.search_timeout) + poll_responses().
- POST /downloads/search/start (mock = immediate; soulseek = returns a search id) +
GET /downloads/search/poll. Shared _evaluate_hits ranks each poll's hits.
- UI streams: starts the search, polls every 1.3s, renders results live with a
'searching…' badge, stops when results plateau (4 stable polls) or ~32s. Live
re-renders suppress per-card entrance so it doesn't blink.
Tests green, ruff clean.
backfill_next returns rows keyed by title/kind/id/imdb_id (no 'name'), so the
'Enriched movie None via Trakt' spam was just the log line reading the wrong key.
Now logs the real title in the 'enriched' line, the fetch-failed line, and the
current-item status. Cosmetic only — the Trakt sweep itself was working correctly
(one-time backfill of the movie catalogue; each row marked once, never loops).
Phase 2 — the engine:
- core/video/download_monitor.py: a daemon thread polls slskd for active video
downloads, updates progress, and on completion MOVES the file from the shared
download folder into the per-type library folder + marks it completed. The
per-download decision (process_download) is pure (fs + slskd injected) — 6 tests.
- POST /downloads/grab: validates (Soulseek-only v1), resolves the target library by
kind, starts the slskd download, records the row, lazily starts the monitor.
- GET /downloads/active (list + ensure monitor running) · POST /downloads/clear
(drop finished).
14 tests, isolation guard + ruff clean. Grab button + Downloads page next.
The Soulseek ⌕ now actually queries your slskd instance instead of fabricating
results. core/video/slskd_search.py (isolated): build_query(scope,…) → POST
/api/v0/searches (shared soulseek.* URL+key) → poll /responses (bounded ~8s,
early-exit at 25) → keep video files → group_video_files() folds them into one hit
per release folder with a peer count + the fastest available source. Same
{title,size_bytes,…} shape, so parse→evaluate→rank/cards are unchanged.
Endpoint routes source=='soulseek' to slskd (torrent/usenet stay mocked), surfaces
'slskd not configured' / network errors, carries username/peers/slots/filename
through for the cards + a future real Grab, and caps to 40. Pure helpers tested (4);
isolation guard + ruff clean.
mock_search now takes the source and gives Soulseek / Torrent / Usenet distinct
hits — different release-group pools, seeder magnitudes, and which qualities show
up (Soulseek drops the top remux, Usenet drops the cam, etc.), the way real
indexers differ. Endpoint forwards body.source. Fixes 'click Torrent shows the
same results as Soulseek'. +1 test.
core/video/release_parse.py (pure, isolated): parse_release(title) → resolution,
source (remux/bluray/web-dl/webrip/hdtv/dvd/cam/screener/workprint), codec, HDR,
audio, group, repack/proper/3d, and crucially the SCOPE — single episode (SxxExx)
vs season pack (Sxx / Season N) vs complete-series pack (S01-S05 / COMPLETE). The
search layer uses this to validate that a hit matches what was searched. 9 tests,
isolation guard + ruff clean. First piece of the live search pipeline.
The shared judge both the Download modal and the later-phase engine will use:
core/video/quality_eval.py (pure, isolated) — resolution_rank/resolution_label,
meets_cutoff (loose resolution target), and evaluate_owned(file, profile) →
{meets, resolution_label, reasons[]}. A copy is 'below target' when its resolution
is under the loose cutoff, or its codec is on the reject list.
Exposed as POST /api/video/downloads/evaluate (loads the stored profile, judges the
posted file). 9 tests green, isolation guard green, ruff clean.
Boulder: 'youtube should have its own quality profile — there are not many options.'
Right — YouTube is grabbed with yt-dlp, not scene/p2p releases, so the Radarr-style
ladder/cutoff/rejects/HDR-audio tiers are meaningless. Added a small, separate profile:
- core/video/youtube_quality.py (pure, isolated): max_resolution ceiling
(best…360p), video_codec (any/av1/vp9/h264, soft), container (mp4/mkv/webm),
prefer_60fps, allow_hdr. normalize/load/save → video.db youtube_quality_profile.
- api/video/downloads.py: GET/POST /downloads/youtube-quality.
- UI: a 'YouTube Quality' card on the Downloads tab (data-video-only) — resolution
dropdown + codec/container segmented + 60fps/HDR checks, reusing the vq-* styles.
Wired into onPageShown + the save-all chain.
The (later-phase) yt-dlp downloader maps these to a format/format_sort selection.
9 new tests green, isolation guard green, ruff clean, JS/HTML balance clean.
Addressing UX feedback (Boulder): the cutoff and size guard were confusing for a
library that holds BOTH movies and TV.
- 'Upgrade until' is now a LOOSE resolution target (4K / 1080p / 720p / SD /
'best — never stop'), not a specific source×resolution tier. 4K is always in
the list regardless of which tiers are toggled on (the old dropdown only
listed enabled tiers, so 4K vanished when off). Stored as cutoff_resolution.
- Size guard split into Max movie size + Max episode size (runtime-aware, like
Radarr's MB/min but human-readable) — a flat GB cap was meaningless across a
2-hour movie and a 25-min episode. Dropped the confusing min slider.
Pure-logic + API + UI in lockstep; removed the now-dead per-row cutoff marker.
12 tests green, ruff clean, JS/HTML balance clean.
Replaces the simplified resolutions+4-sources+codec model with a real quality
ladder to actually stand in for Radarr/Sonarr:
- tiers: source×resolution ladder (Remux-2160p … SDTV) as one ranked, toggleable
list, best→worst, with a cutoff (stop upgrading once the library holds a tier
at/above it — no endless re-grabbing).
- rejects: hard blocks (cam/screener/workprint/3d, optional x264).
- soft preferences (score/tie-break, never reject alone): prefer_codec (any/hevc/av1),
prefer_hdr (off/prefer/require), prefer_audio (any/surround/lossless/atmos),
prefer_repack.
- size guard: min/max GB per item (0 = no limit), min pinned to a real cap.
Pure normalize/load/save (no DB/network), isolated from music. API endpoint is an
unchanged passthrough. 12 tests green, ruff clean. UI next.
Video-specific Download Source section: a mode dropdown limited to Soulseek / Torrent /
Usenet / Hybrid (no streaming sources — those are music-only). In hybrid mode, a chain
builder lists the three sources with arrow-reorder + enable toggles (best-first), and
NO album-level/track-level badges (a music-only concept, per spec).
- core/video/download_config.py: pure normalize for download_mode + hybrid_order
(validates to the 3 sources, dedupes, never empties); stored in video_settings.
- /api/video/downloads/config GET/POST now carries download_mode + hybrid_order
alongside the folders.
- video-settings.js: dropdown + hybrid rows render/reorder/toggle, show the hybrid
container only in hybrid mode, save on change. Reuses the .vq-row styling.
7 tests (6 pure + 1 API). Next: the shared slskd connection block (reads/writes the
music config_manager so both sides share one slskd) + confirming the shared Indexers tab.
AniList defaults off (anime opt-in), so it showed 'Not configured' — implying a
missing API key, when it's keyless and just toggled off in Settings > Community Data
(No Key). Workers now report needs_key in get_stats (True for the key-gated
fanart/opensubtitles/trakt + all matchers; False for the keyless toggles). The
manager rail/pill + dashboard-header tooltip show 'Disabled' / 'Off — enable in
Settings' for a disabled keyless worker, and keep 'Not configured' only for ones
that genuinely need a key.
(The AniList on/off toggle already exists in the collapsed 'Community Data (No Key)'
settings frame — this just makes the status honest about what's needed.)
Fifth/final service. Keyless, movies + shows, on by default → toggle in the
'Community Data (No Key)' frame.
- WikidataWorker: two-step lookup — find the entity by IMDb id (haswbstatement P345)
then read its official website (P856); stores wikidata_url. Registered in
build_backfill_workers.
- DB: wikidata_url/status/attempted on movies + shows + _BACKFILL/_BACKFILL_COLS;
both detail payloads return wikidata_url.
- config GET/POST wikidata_enabled; manager orb (green 🔗) + status poll; detail page
'Official Site' link badge alongside IMDb/TMDB/TVDB.
- 3 new tests (incl. the two-step fetch) + fixed-set/config assertions.
34 backfill tests green, ruff clean. All five new services (Trakt, TVmaze, AniList,
DeArrow, Wikidata) now have the full worker/DB/connections/orb/manager/detail parity.
Fourth service. Rides the YouTube-video enrich path (like RYD/SponsorBlock), keyless,
on by default → toggle in the YouTube-Extras frame.
- DeArrowWorker: fetches the branding API and stores the first non-original crowd
title for a cached YouTube video. apply_youtube_dearrow + youtube_video_dearrow_title
DB methods; dearrow_status added to the youtube_enrich next/breakdown whitelists.
- Schema: dearrow_title/status/attempted on youtube_video_stats (video_schema.sql +
_COLUMN_MIGRATIONS for existing DBs).
- config GET/POST dearrow_enabled; manager orb (blue 🏹, video-kind) + status poll;
YT video-detail panel shows a 'DeArrow' alt-title (payload via youtube.py + CSS).
- 4 new tests + fixed-set/config assertions.
30 backfill tests green. (My change is ruff-clean; the 12 pre-existing S110s in
api/video/youtube.py are unrelated tech debt on this branch, left untouched.)
Third service. Keyless GraphQL (new _http_post_json helper) → enable toggle in the
'Community Data (No Key)' frame, OFF by default (anime-niche + title-search match).
- AniListWorker: TV-only, searches AniList by title and stores the anime averageScore
(0-100), with a conservative normalized-title guard so a fuzzy anime search can't
attach a score to a non-anime show. anilist_enabled toggle (default off).
- DB: anilist_score/status/attempted on shows + _BACKFILL/_BACKFILL_COLS (keyed on
title); show_detail returns anilist_score.
- config GET/POST anilist_enabled (default 0); manager orb (blue 🎌) + status poll;
detail page 'AniList 85%' chip (+ CSS).
- 4 new tests (incl. the title-mismatch rejection) + fixed-set/config assertions.
27 backfill tests green, ruff clean. (Note: the youtube_status_route test still
flaps on the sandbox WSL WAL disk-I/O — environmental, unrelated.)
Second service. Keyless (no API key) → an enable toggle in a new 'Community Data
(No Key)' connections frame, mirroring the YouTube-Extras pattern.
- TVmazeWorker: TV-only (no movie DB), looks a show up by imdb/thetvdb id and
gap-fills the TVmaze community rating. On by default; tvmaze_enabled toggle.
- DB: tvmaze_rating/status/attempted on shows + _BACKFILL/_BACKFILL_COLS (show only);
show_detail returns tvmaze_rating.
- config GET/POST tvmaze_enabled; manager orb (teal 📺, show-kind) + status poll;
detail page TVmaze rating chip (+ CSS).
- 4 new tests + the 3 fixed-set/config-exact assertions updated.
Note: one UNRELATED test (youtube_status_route) flaps on a WSL WAL 'disk I/O error'
in this sandbox (pass/fail/fail across reruns of identical code) — environmental, not
this change; the TVmaze + config tests pass consistently.
First of the new enrichment services, wired to the SAME standard as the rest:
- TraktWorker backfill (core/video/enrichment/backfill.py): looks a title up by its
IMDb id (?extended=full) and gap-fills the community rating + vote count. Registered
in build_backfill_workers.
- DB: trakt_rating/trakt_votes/trakt_status/trakt_attempted columns + _BACKFILL /
_BACKFILL_COLS registration; detail payloads return trakt_rating/votes.
- Connections tab: Trakt service frame (Client ID field + Test button), wired into
video-settings.js load/save/bindings and the /enrichment/config GET+POST.
- Worker-manager modal + orb: WORKERS registry entry (red ★) + status-poll SERVICES
list, so it gets the same orb animation + per-service matched/pending/error card.
- Detail page: a 'Trakt 8.2' rating chip alongside IMDb/RT/Metacritic (+ CSS).
5 new tests + 2 fixed-set assertions updated; 249 video tests green, ruff clean.
Adds a VideoBackfillWorker base that enriches already-identified items BY id
(vs the matcher workers), with the exact same lifecycle + get_stats() shape so
the engine registry, /api/video/enrichment routes, and Manage-Workers modal
drive them identically.
Workers:
- fanart.tv (free key): gap-fills logo/clearart/banner/backdrop/poster art
- OpenSubtitles (free key): records subtitle-language availability per title
- Return YouTube Dislike (no key): like/dislike estimates on cached videos
- SponsorBlock (no key): crowd segments per video
DB: youtube_video_stats + youtube_video_segments tables; fanart/subs columns;
backfill_next/mark/breakdown + youtube_enrich_* helpers; likes/dislikes merged
into get_channel_videos. Seam tests in tests/test_video_backfill.py.
Correcting the cookies theory — ytdl-sub is just CLI yt-dlp (no browser). The real
differentiators are a CURRENT yt-dlp + not fighting YouTube's client identity. We
were pinning a static user_agent in the yt-dlp opts, which yt-dlp recommends against
(it trips YouTube's heuristics and truncates large-playlist pagination). Dropped it
so yt-dlp manages its own (current) client — the lever that lets a fresh ytdl-sub
page further than us. (_UA is still used for the InnerTube requests.) Softened the
partial-playlist note to just 'Showing N of TOTAL videos.'
A big playlist read '97' because yt-dlp flat caps playlist extraction at ~100 and
its playlist_count is often unset, so we showed the FETCHED count. Now the detail
endpoint overlays the InnerTube playlist browse (browseId VL<id>): curator-ordered
videos up to ~200 (vs yt-dlp's ~100) AND the real count from the header's
numVideosText (e.g. '512 videos'). The billboard shows the true total; when the
list is partial it says 'Showing the first N of TOTAL videos (YouTube limits
playlist browsing).' Live-validated (Veritasium uploads: total 512, 200 loaded).
103 tests green.
Honest cap: YouTube's browse only exposes ~200 of a playlist (page 2 returns no
continuation), so very large playlists show the right COUNT but list ~200.
A playlist is now a first-class watchlist type alongside channels:
- core: parse_playlist_id (URL/bare id; rejects mixes RD… + personal lists) +
resolve_playlist → {playlist_id, title, channel_title, video_count, thumbnail,
videos} in the CURATOR's order (partial set, not by year).
- db: add/remove/list/watch_state for kind='playlist' video_watchlist rows
(mirrors channels; same surrogate scheme, separate kind so they don't collide).
- api: /youtube/resolve now detects playlist links; /youtube/playlist/follow +
/unfollow; the existing /youtube/playlist/<id> is upgraded to also return the
playlist meta + following (serves the channel-page expansion AND the new detail
view from one route); /youtube/channels also returns followed playlists.
Validated live (Lex Fridman playlist resolves title+owner+videos). 190 tests green.
Frontend (search detect + result card + detail view + watchlist display) next.
TV episode rows show runtime; channel video cards showed nothing per video. The
InnerTube lockup already carries both — now we capture duration ('12:34') + an
approximate view count (parsed from '2.6M views') in innertube_parse_video_items,
remember them on youtube_channel_videos (new duration/view_count cols + migration),
and render a duration badge bottom-right of the thumb + 'N views' in the card meta.
The background stream backfills them onto the recent (yt-dlp) videos too, so the
whole catalog gets them. Live-validated (MrBeast: 32:08 / 50M, 30/30). 65 tests green.
Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
The channel page was capped at the recent 90 (yt-dlp flat, hard min(90,limit)),
so prolific channels (Ninja Kidz etc.) showed only ~90 of hundreds. Now it loads
everything via YouTube's InnerTube API, paginated by continuation token — each
page fetched once (O(n), light on rate limits), unlike yt-dlp offset batches
which re-scan from the start every time.
- core: innertube_parse_video_items (keeps title+thumbnail, not just dates) +
innertube_channel_videos_page(channel_id, continuation) → {videos, continuation}.
- api: GET /youtube/channel/<id>/videos?continuation=<tok> returns ~a batch of
videos (approx dates, refined from the date cache) + the next token.
- frontend: ytLoadAllVideos streams batches after the fast initial 90 render,
folding each into the year-seasons live — backfills dates on the videos already
shown AND appends older ones, re-rendering per batch (episode grid only when the
viewed season changed). Replaces the old date-only re-poll (this dates AND
expands). Safety ceiling 2000; a 'Loading full history… N so far' indicator.
Validated live (MrBeast: 30/page, clean continuation, all dated). 101 tests green.
Channels enriched before InnerTube existed (partial coverage, e.g. CaseyNeistat
36, Good Mythical MORE 25) were locked behind the 24h coverage gate and would
never reach the full ~240-date catalog on their own. Tag each enrichment row
with the source ('innertube'|'fallback'); legacy rows (method NULL after the
additive migration) bypass the gate ONCE so they re-enrich and upgrade, then
settle under the normal window. Non-destructive: existing follows, wished videos
(24), and cached dates (1176) are untouched — only coverage refreshes on next
open/sweep. Migration is additive (method TEXT); regression test added.
A Beatport-style custom parser, but pointed at YouTube's own InnerTube browse API
— no key, no Java, no third-party proxies. Reads the channel 'Videos' tab's
lockupViewModel items (contentId + relative 'N ago' text → approximate date,
which is fine for year-seasons), paginating via continuation tokens.
Validated LIVE before wiring in: GMM/MrBeast/Veritasium each return 240 dated
videos in ~7s with correct year spreads (e.g. Veritasium 2017→2026), ~8 requests
per channel (light on rate limits). End-to-end enricher run cached 240 dates.
Enricher order is now: InnerTube (primary) → configured proxy (opt-in, only if
empty) → yt-dlp per-video (the fallback / basic method, for undated gaps + exact
dates). Pure parsing is fully unit-tested (relative-date conversion, lockup
filtering, continuation-token selection, pagination, guards). 182 video tests
green; additive — if InnerTube ever breaks, it returns {} and falls back, so
nothing breaks.
Live-tested the proxy against real channels — it does not work: Invidious public
APIs are 401/403 (auth-walled/disabled), and the one reachable Piped instance
returns EMPTY video lists for every channel (MrBeast/LTT/Veritasium all 0). Also
fixed a real bug: _harvest bailed on an empty first page instead of following the
nextpage token.
So the proxy is now OPT-IN via a youtube_proxy_instances setting (empty by
default → skipped entirely, no wasted dead calls). The default path is RSS +
the parallelized yt-dlp fallback, which actually works (your log: 9/36/25 dates
cached). Power users can point the setting at a live instance if they find one.
Verified the three engine workers (tmdb/tvdb/omdb) log per-item INFO under
video_enrichment.worker. Matched the youtube enricher to that exactly:
- INFO is now purely per-item ('Dated <ch> '<video>' -> <date>' / 'No date
for …') + one terse per-channel summary ('Dated N/M videos for <ch>'),
mirroring 'Matched … -> TMDB ID' / 'Synced full episode list…'.
- The 'YouTube dates:'-prefixed phase/diagnostic lines (enriching / proxy
returned / failures) demoted to DEBUG.
- Logger renamed video.youtube_enrichment -> video_enrichment.youtube (same
namespace as video_enrichment.worker).
- Per-item lines as it dates each video — 'Dated <channel> '<video>' -> <date>'
/ 'No date for …', mirroring the workers' 'Matched <kind> '<title>' -> TMDB
ID: …'. (Phase lines kept for context: enriching / proxy returned / done.)
Your log showed 'proxy returned 0' (public proxies dead) → the slow per-video
path. Date those recent uploads in a 3-worker thread pool instead of serially,
so a channel finishes in ~30s rather than 1-2 min; cached as each completes.
Addresses 'says running but I see nothing':
- Real logs: 'enriching <ch>…', 'proxy returned N', 'done — N proxy + N
per-video (X/Y dated)' so you can see exactly what it's doing.
- Proxy timeout 8s→4s so dead instances (most public ones) fail fast instead of
hanging ~40s before the yt-dlp fallback.
- Channel page now re-fetches a few times (25/60/110s) after load while videos
are still undated, re-rendering only when NEW year-seasons appear — so dates
the background enricher fills in pop in without a manual reload.
The enrichment daemon was starting during API tests (it uses the default DB +
network), touching a stray real DB and causing sqlite disk-IO flakiness. enqueue()
now no-ops when PYTEST_CURRENT_TEST is set; _enrich() is still tested directly
with a tmp DB. 176 video tests green.
Two gaps from real use:
- When all proxy instances are down (common), the fallback only dated WISHED
videos, leaving most as 'Earlier videos'. Now on proxy failure it flat-resolves
the channel's recent uploads and dates THOSE via yt-dlp (throttled, cached) —
so years populate fully even with no working proxy.
- A channel followed before the enricher existed was never queued ('nothing to
process'). /youtube/channels now sweeps: any followed channel not enriched
recently gets enqueued — so viewing the Channels tab (or the wishlist badge
refresh) picks up existing follows.
The detail page still reads cache+RSS (fast); the enricher fills the cache. 51
tests green.
- The standalone enricher now reports orb telemetry: stats()/pause()/resume();
/enrichment/youtube/status (+pause/resume) special-cased in the route. Added
the 4th header worker button (red ▶), WORKER_DEFS entry, and SERVICES wiring —
it's not on the socket so video-enrichment.js polls it every 3s. It animates
(idle → active orb + inbound pulses) while a followed channel's dates are being
fetched, like the TMDB/TVDB/OMDb orbs.
- Channel detail now uses the SAME standard watchlist button as shows/movies
(library-artist-watchlist-btn, ✓/＋ icon, 'In Watchlist'/'Watchlist') instead
of a bespoke 'Follow' — consistency. Still wired to the channel follow action.
176 backend tests green; JS balanced; music untouched.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
Flat listing has no upload dates, so channels showed one 'All Videos' season.
Now real year-seasons, filled from cheap sources and cached so they grow:
- core: parse_rss_dates / channel_recent_dates — one public RSS GET dates the
~15 most-recent uploads (no yt-dlp, no bot risk).
- db: youtube_video_dates cache table + cache_video_dates / get_video_dates.
- /youtube/channel merges cached + RSS dates onto the flat videos (year-seasons)
and caches what it learns; /youtube/video caches each fetched date too — so
expanding/wishing videos progressively fills the catalog's years, instant on
repeat. Dateless tail groups as 'Earlier videos'. Missing-only toggle hidden
for channels. 84 db + 84 youtube/api tests green.
Full day-one coverage still needs the YouTube Data API's publishedAt (optional,
yt-dlp can't match it cheaply) — RSS + cache is the no-key best.
- The big one: an empty TMDB result no longer flashes 'No results' while the
(slower) YouTube channel search is still in flight — a 'YouTube channels ·
searching…' skeleton group (shimmer cards + spinner) shows instead, and the
empty state only appears once BOTH sources resolve with nothing. Users no
longer click away thinking nothing's happening.
- Channel avatars: fall back to the singular 'thumbnail' field when the flat
results omit the thumbnails list, so fewer cards are photoless (the rest keep
the clean initials placeholder). Reduced-motion respected.
