The #902 'paste cookies.txt' feature added a 'custom' sentinel value for
youtube.cookies_browser, but that feature wasn't merged to this branch — and ~7 call sites
(core/youtube_client.py x5, core/video/youtube.py, web_server.py) pass cookies_browser raw to
yt-dlp's cookiesfrombrowser, which rejects 'custom' ('ERROR: unsupported browser: custom') and
broke YouTube download/enrichment. Sanitize 'custom' -> '' (no browser) at every site:
youtube_client reads via a walrus filter, the other two guard the condition. 'custom' now
means 'no browser cookies' here (the cookiefile feature isn't on this branch). Latent on dev
too — only _youtube_cookie_opts was fixed there.
Correcting the cookies theory — ytdl-sub is just CLI yt-dlp (no browser). The real
differentiators are a CURRENT yt-dlp + not fighting YouTube's client identity. We
were pinning a static user_agent in the yt-dlp opts, which yt-dlp recommends against
(it trips YouTube's heuristics and truncates large-playlist pagination). Dropped it
so yt-dlp manages its own (current) client — the lever that lets a fresh ytdl-sub
page further than us. (_UA is still used for the InnerTube requests.) Softened the
partial-playlist note to just 'Showing N of TOTAL videos.'
A big playlist read '97' because yt-dlp flat caps playlist extraction at ~100 and
its playlist_count is often unset, so we showed the FETCHED count. Now the detail
endpoint overlays the InnerTube playlist browse (browseId VL<id>): curator-ordered
videos up to ~200 (vs yt-dlp's ~100) AND the real count from the header's
numVideosText (e.g. '512 videos'). The billboard shows the true total; when the
list is partial it says 'Showing the first N of TOTAL videos (YouTube limits
playlist browsing).' Live-validated (Veritasium uploads: total 512, 200 loaded).
103 tests green.
Honest cap: YouTube's browse only exposes ~200 of a playlist (page 2 returns no
continuation), so very large playlists show the right COUNT but list ~200.
A playlist is now a first-class watchlist type alongside channels:
- core: parse_playlist_id (URL/bare id; rejects mixes RD… + personal lists) +
resolve_playlist → {playlist_id, title, channel_title, video_count, thumbnail,
videos} in the CURATOR's order (partial set, not by year).
- db: add/remove/list/watch_state for kind='playlist' video_watchlist rows
(mirrors channels; same surrogate scheme, separate kind so they don't collide).
- api: /youtube/resolve now detects playlist links; /youtube/playlist/follow +
/unfollow; the existing /youtube/playlist/<id> is upgraded to also return the
playlist meta + following (serves the channel-page expansion AND the new detail
view from one route); /youtube/channels also returns followed playlists.
Validated live (Lex Fridman playlist resolves title+owner+videos). 190 tests green.
Frontend (search detect + result card + detail view + watchlist display) next.
TV episode rows show runtime; channel video cards showed nothing per video. The
InnerTube lockup already carries both — now we capture duration ('12:34') + an
approximate view count (parsed from '2.6M views') in innertube_parse_video_items,
remember them on youtube_channel_videos (new duration/view_count cols + migration),
and render a duration badge bottom-right of the thumb + 'N views' in the card meta.
The background stream backfills them onto the recent (yt-dlp) videos too, so the
whole catalog gets them. Live-validated (MrBeast: 32:08 / 50M, 30/30). 65 tests green.
Channel pages re-fetched everything every open (re-stream + ~3s yt-dlp metadata).
Now we remember what we learn about a channel and serve it cache-first:
- schema: youtube_channel_videos (list) + youtube_channel_meta (avatar/subs/tags/
banner); dates stay in youtube_video_dates, merged on read. DB: cache_/get_
channel_videos + cache_/get_channel_meta (upserts COALESCE so refreshes never
drop fields).
- enricher: _enrich now REMEMBERS a channel — caches the full InnerTube catalog
(list, via new innertube_channel_catalog) + metadata + dates, not just dates.
Since it already sweeps followed channels, watchlisted channels get pre-warmed
in the background → opening them is instant.
- channel endpoint: cache-first. A remembered channel renders from cache with NO
network (returns from_cache:true); a miss resolves live (yt-dlp) and remembers
it. The /videos batch endpoint caches every page it streams.
- frontend: cache hit renders the full catalog instantly, then re-streams to
refresh QUIETLY (new uploads/date fixes); only the first (miss) load shows the
'loading full history' banner.
Live-validated (MrBeast catalog: 90/3 pages, all titled+dated). 190 tests green.
The channel page was capped at the recent 90 (yt-dlp flat, hard min(90,limit)),
so prolific channels (Ninja Kidz etc.) showed only ~90 of hundreds. Now it loads
everything via YouTube's InnerTube API, paginated by continuation token — each
page fetched once (O(n), light on rate limits), unlike yt-dlp offset batches
which re-scan from the start every time.
- core: innertube_parse_video_items (keeps title+thumbnail, not just dates) +
innertube_channel_videos_page(channel_id, continuation) → {videos, continuation}.
- api: GET /youtube/channel/<id>/videos?continuation=<tok> returns ~a batch of
videos (approx dates, refined from the date cache) + the next token.
- frontend: ytLoadAllVideos streams batches after the fast initial 90 render,
folding each into the year-seasons live — backfills dates on the videos already
shown AND appends older ones, re-rendering per batch (episode grid only when the
viewed season changed). Replaces the old date-only re-poll (this dates AND
expands). Safety ceiling 2000; a 'Loading full history… N so far' indicator.
Validated live (MrBeast: 30/page, clean continuation, all dated). 101 tests green.
A Beatport-style custom parser, but pointed at YouTube's own InnerTube browse API
— no key, no Java, no third-party proxies. Reads the channel 'Videos' tab's
lockupViewModel items (contentId + relative 'N ago' text → approximate date,
which is fine for year-seasons), paginating via continuation tokens.
Validated LIVE before wiring in: GMM/MrBeast/Veritasium each return 240 dated
videos in ~7s with correct year spreads (e.g. Veritasium 2017→2026), ~8 requests
per channel (light on rate limits). End-to-end enricher run cached 240 dates.
Enricher order is now: InnerTube (primary) → configured proxy (opt-in, only if
empty) → yt-dlp per-video (the fallback / basic method, for undated gaps + exact
dates). Pure parsing is fully unit-tested (relative-date conversion, lockup
filtering, continuation-token selection, pagination, guards). 182 video tests
green; additive — if InnerTube ever breaks, it returns {} and falls back, so
nothing breaks.
Live-tested the proxy against real channels — it does not work: Invidious public
APIs are 401/403 (auth-walled/disabled), and the one reachable Piped instance
returns EMPTY video lists for every channel (MrBeast/LTT/Veritasium all 0). Also
fixed a real bug: _harvest bailed on an empty first page instead of following the
nextpage token.
So the proxy is now OPT-IN via a youtube_proxy_instances setting (empty by
default → skipped entirely, no wasted dead calls). The default path is RSS +
the parallelized yt-dlp fallback, which actually works (your log: 9/36/25 dates
cached). Power users can point the setting at a live instance if they find one.
Addresses 'says running but I see nothing':
- Real logs: 'enriching <ch>…', 'proxy returned N', 'done — N proxy + N
per-video (X/Y dated)' so you can see exactly what it's doing.
- Proxy timeout 8s→4s so dead instances (most public ones) fail fast instead of
hanging ~40s before the yt-dlp fallback.
- Channel page now re-fetches a few times (25/60/110s) after load while videos
are still undated, re-rendering only when NEW year-seasons appear — so dates
the background enricher fills in pop in without a manual reload.
Follow a channel → a background job fetches its FULL upload-date catalog so the
channel page's year-seasons populate fully, cached permanently (one-time per
channel, instant after).
- core/video/youtube.py: proxy_channel_dates() — no-key BULK dates via Piped/
Invidious instances (tries several, paginates); parse_proxy_dates handles both
shapes. The clean path the user wanted.
- core/video/youtube_enrichment.py: YoutubeDateEnricher daemon — enqueue(channel)
→ proxy bulk → cache; per-video yt-dlp only as a throttled fallback (cap 60,
0.4s) for wished videos when every proxy is down. Skips channels enriched in
the last 24h.
- db: youtube_channel_enrichment table + mark/check + wishlisted_video_ids_for_
channel. Enqueued from /youtube/follow and on opening a followed channel.
Scoped to FOLLOWED channels only (per request). 174 tests green; enricher imports
nothing from music.
Flat listing has no upload dates, so channels showed one 'All Videos' season.
Now real year-seasons, filled from cheap sources and cached so they grow:
- core: parse_rss_dates / channel_recent_dates — one public RSS GET dates the
~15 most-recent uploads (no yt-dlp, no bot risk).
- db: youtube_video_dates cache table + cache_video_dates / get_video_dates.
- /youtube/channel merges cached + RSS dates onto the flat videos (year-seasons)
and caches what it learns; /youtube/video caches each fetched date too — so
expanding/wishing videos progressively fills the catalog's years, instant on
repeat. Dateless tail groups as 'Earlier videos'. Missing-only toggle hidden
for channels. 84 db + 84 youtube/api tests green.
Full day-one coverage still needs the YouTube Data API's publishedAt (optional,
yt-dlp can't match it cheaply) — RSS + cache is the no-key best.
- The big one: an empty TMDB result no longer flashes 'No results' while the
(slower) YouTube channel search is still in flight — a 'YouTube channels ·
searching…' skeleton group (shimmer cards + spinner) shows instead, and the
empty state only appears once BOTH sources resolve with nothing. Users no
longer click away thinking nothing's happening.
- Channel avatars: fall back to the singular 'thumbnail' field when the flat
results omit the thumbnails list, so fewer cards are photoless (the rest keep
the clean initials placeholder). Reduced-motion respected.
A text search now also surfaces matching YouTube channels (not just the paste-a-
link path). search_channels() extracts the channels-filtered YouTube results
page (flat); API GET /youtube/search hydrates a 'following' flag. The search
page fires it in parallel with the TMDB search and appends a 'YouTube channels'
group when it returns (best-effort; never blocks/breaks TMDB results). Cards open
the in-app channel page. 82 youtube+api tests green; balanced; music untouched.
Flat channel listing can't return descriptions, so selecting a video showed 'No
description'. Now we do a non-flat single-video extract on demand — the way the
TV nebula lazy-loads guest stars:
- core: video_detail(id) / shape_video() — description, views, likes, duration,
tags, channel, webpage_url (full extract, yt-dlp injectable for tests).
- API: GET /youtube/video/<id> — returns it AND backfills the description onto
the wishlist row (set_wishlist_video_overview), so re-opening is instant.
- Wishlist info bar (youtube): on select, lazy-fetch → real description +
eyebrow (date · duration · views) + a 'Watch on YouTube' link; cached on the
episode object. 'Loading details…' while in flight.
Mirrors the episode lazy-load + art-backfill patterns. 73 youtube+api tests
green; brace/CSS balanced; music untouched.
- Resolver: capture banner_url + separate avatar (by thumbnail id), keeping
per-video duration/views for the detail hero.
- query_youtube_wishlist reshaped to the EXACT TV-nebula shape: channel = show,
upload YEAR = season (newest first), video = episode (newest=ep1 within a
year), carrying source='youtube' + youtube_id + per-video source_id. Surrogate
int returned as tmdb_id so the nebula's int keying just works.
- New API: GET /youtube/channel/<id> (full channel detail — meta + deeper
uploads + following + per-video wished flags) and POST /youtube/wishlist/add
(per-video wish). DB: youtube_video_wish_state, remove_one_video_from_wishlist.
- Tests updated for the nebula shape + banner + detail endpoints. 82 DB + 69
api/youtube tests green.
First slice of 'follow a YouTube channel as a show'. Pure, testable resolver:
- parse_channel_url() normalizes any channel reference (@handle, /channel/UC..,
/c/.., /user/.., bare handle, with/without scheme or tab suffix) to a
canonical /videos uploads URL, and rejects non-channels (watch/playlist/
shorts/home/non-youtube).
- shape_channel() maps yt-dlp's flat-extract dict to our shape (channel meta +
recent uploads), picking best-res thumbs, parsing sparse publish dates
(timestamp or upload_date → ISO, None when flat mode omits them), filtering
null/id-less entries.
- resolve_channel() ties them together via an injectable YoutubeDL factory.
yt-dlp only (already a pinned dep), extract_flat for cheap listing. 28 seam
tests, no network. Schema bridge + API + UI are the next slices.
