Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
Channels enriched before InnerTube existed (partial coverage, e.g. CaseyNeistat
36, Good Mythical MORE 25) were locked behind the 24h coverage gate and would
never reach the full ~240-date catalog on their own. Tag each enrichment row
with the source ('innertube'|'fallback'); legacy rows (method NULL after the
additive migration) bypass the gate ONCE so they re-enrich and upgrade, then
settle under the normal window. Non-destructive: existing follows, wished videos
(24), and cached dates (1176) are untouched — only coverage refreshes on next
open/sweep. Migration is additive (method TEXT); regression test added.
A Beatport-style custom parser, but pointed at YouTube's own InnerTube browse API
— no key, no Java, no third-party proxies. Reads the channel 'Videos' tab's
lockupViewModel items (contentId + relative 'N ago' text → approximate date,
which is fine for year-seasons), paginating via continuation tokens.
Validated LIVE before wiring in: GMM/MrBeast/Veritasium each return 240 dated
videos in ~7s with correct year spreads (e.g. Veritasium 2017→2026), ~8 requests
per channel (light on rate limits). End-to-end enricher run cached 240 dates.
Enricher order is now: InnerTube (primary) → configured proxy (opt-in, only if
empty) → yt-dlp per-video (the fallback / basic method, for undated gaps + exact
dates). Pure parsing is fully unit-tested (relative-date conversion, lockup
filtering, continuation-token selection, pagination, guards). 182 video tests
green; additive — if InnerTube ever breaks, it returns {} and falls back, so
nothing breaks.
Live-tested the proxy against real channels — it does not work: Invidious public
APIs are 401/403 (auth-walled/disabled), and the one reachable Piped instance
returns EMPTY video lists for every channel (MrBeast/LTT/Veritasium all 0). Also
fixed a real bug: _harvest bailed on an empty first page instead of following the
nextpage token.
So the proxy is now OPT-IN via a youtube_proxy_instances setting (empty by
default → skipped entirely, no wasted dead calls). The default path is RSS +
the parallelized yt-dlp fallback, which actually works (your log: 9/36/25 dates
cached). Power users can point the setting at a live instance if they find one.
Verified the three engine workers (tmdb/tvdb/omdb) log per-item INFO under
video_enrichment.worker. Matched the youtube enricher to that exactly:
- INFO is now purely per-item ('Dated <ch> '<video>' -> <date>' / 'No date
for …') + one terse per-channel summary ('Dated N/M videos for <ch>'),
mirroring 'Matched … -> TMDB ID' / 'Synced full episode list…'.
- The 'YouTube dates:'-prefixed phase/diagnostic lines (enriching / proxy
returned / failures) demoted to DEBUG.
- Logger renamed video.youtube_enrichment -> video_enrichment.youtube (same
namespace as video_enrichment.worker).
- Per-item lines as it dates each video — 'Dated <channel> '<video>' -> <date>'
/ 'No date for …', mirroring the workers' 'Matched <kind> '<title>' -> TMDB
ID: …'. (Phase lines kept for context: enriching / proxy returned / done.)
Your log showed 'proxy returned 0' (public proxies dead) → the slow per-video
path. Date those recent uploads in a 3-worker thread pool instead of serially,
so a channel finishes in ~30s rather than 1-2 min; cached as each completes.
Addresses 'says running but I see nothing':
- Real logs: 'enriching <ch>…', 'proxy returned N', 'done — N proxy + N
per-video (X/Y dated)' so you can see exactly what it's doing.
- Proxy timeout 8s→4s so dead instances (most public ones) fail fast instead of
hanging ~40s before the yt-dlp fallback.
- Channel page now re-fetches a few times (25/60/110s) after load while videos
are still undated, re-rendering only when NEW year-seasons appear — so dates
the background enricher fills in pop in without a manual reload.
The enrichment daemon was starting during API tests (it uses the default DB +
network), touching a stray real DB and causing sqlite disk-IO flakiness. enqueue()
now no-ops when PYTEST_CURRENT_TEST is set; _enrich() is still tested directly
with a tmp DB. 176 video tests green.
Two gaps from real use:
- When all proxy instances are down (common), the fallback only dated WISHED
videos, leaving most as 'Earlier videos'. Now on proxy failure it flat-resolves
the channel's recent uploads and dates THOSE via yt-dlp (throttled, cached) —
so years populate fully even with no working proxy.
- A channel followed before the enricher existed was never queued ('nothing to
process'). /youtube/channels now sweeps: any followed channel not enriched
recently gets enqueued — so viewing the Channels tab (or the wishlist badge
refresh) picks up existing follows.
The detail page still reads cache+RSS (fast); the enricher fills the cache. 51
tests green.
- The standalone enricher now reports orb telemetry: stats()/pause()/resume();
/enrichment/youtube/status (+pause/resume) special-cased in the route. Added
the 4th header worker button (red ▶), WORKER_DEFS entry, and SERVICES wiring —
it's not on the socket so video-enrichment.js polls it every 3s. It animates
(idle → active orb + inbound pulses) while a followed channel's dates are being
fetched, like the TMDB/TVDB/OMDb orbs.
- Channel detail now uses the SAME standard watchlist button as shows/movies
(library-artist-watchlist-btn, ✓/＋ icon, 'In Watchlist'/'Watchlist') instead
of a bespoke 'Follow' — consistency. Still wired to the channel follow action.
176 backend tests green; JS balanced; music untouched.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
