enrichment_retry handled ryd/sponsorblock + the keyed backfills, but DeArrow (also a
youtube_video_stats backfill, keyed on dearrow_status) fell through to nothing — so the
modal's Retry button did nothing for DeArrow. Fold dearrow into the youtube_video_stats
retry branch. Regression test re-queues failed dearrow rows, leaves matched ones.
Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
A Beatport-style custom parser, but pointed at YouTube's own InnerTube browse API
— no key, no Java, no third-party proxies. Reads the channel 'Videos' tab's
lockupViewModel items (contentId + relative 'N ago' text → approximate date,
which is fine for year-seasons), paginating via continuation tokens.
Validated LIVE before wiring in: GMM/MrBeast/Veritasium each return 240 dated
videos in ~7s with correct year spreads (e.g. Veritasium 2017→2026), ~8 requests
per channel (light on rate limits). End-to-end enricher run cached 240 dates.
Enricher order is now: InnerTube (primary) → configured proxy (opt-in, only if
empty) → yt-dlp per-video (the fallback / basic method, for undated gaps + exact
dates). Pure parsing is fully unit-tested (relative-date conversion, lockup
filtering, continuation-token selection, pagination, guards). 182 video tests
green; additive — if InnerTube ever breaks, it returns {} and falls back, so
nothing breaks.
Live-tested the proxy against real channels — it does not work: Invidious public
APIs are 401/403 (auth-walled/disabled), and the one reachable Piped instance
returns EMPTY video lists for every channel (MrBeast/LTT/Veritasium all 0). Also
fixed a real bug: _harvest bailed on an empty first page instead of following the
nextpage token.
So the proxy is now OPT-IN via a youtube_proxy_instances setting (empty by
default → skipped entirely, no wasted dead calls). The default path is RSS +
the parallelized yt-dlp fallback, which actually works (your log: 9/36/25 dates
cached). Power users can point the setting at a live instance if they find one.
Two gaps from real use:
- When all proxy instances are down (common), the fallback only dated WISHED
videos, leaving most as 'Earlier videos'. Now on proxy failure it flat-resolves
the channel's recent uploads and dates THOSE via yt-dlp (throttled, cached) —
so years populate fully even with no working proxy.
- A channel followed before the enricher existed was never queued ('nothing to
process'). /youtube/channels now sweeps: any followed channel not enriched
recently gets enqueued — so viewing the Channels tab (or the wishlist badge
refresh) picks up existing follows.
The detail page still reads cache+RSS (fast); the enricher fills the cache. 51
tests green.
- The standalone enricher now reports orb telemetry: stats()/pause()/resume();
/enrichment/youtube/status (+pause/resume) special-cased in the route. Added
the 4th header worker button (red ▶), WORKER_DEFS entry, and SERVICES wiring —
it's not on the socket so video-enrichment.js polls it every 3s. It animates
(idle → active orb + inbound pulses) while a followed channel's dates are being
fetched, like the TMDB/TVDB/OMDb orbs.
- Channel detail now uses the SAME standard watchlist button as shows/movies
(library-artist-watchlist-btn, ✓/＋ icon, 'In Watchlist'/'Watchlist') instead
of a bespoke 'Follow' — consistency. Still wired to the channel follow action.
176 backend tests green; JS balanced; music untouched.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
