Alongside the per-worker 'Retry all failed', the worker modal now has a topbar
'Retry all failed' that re-queues every failed/not_found item across ALL workers and
kinds in one click — one-shot recovery after an API outage left lots errored.
- db.retry_all_failed() derives the full service+kind set from the same _ENRICH /
_BACKFILL maps the workers use (tmdb/tvdb + omdb + fanart/opensubtitles/trakt/
tvmaze/anilist/wikidata + ryd/sponsorblock/dearrow), loops enrichment_retry, returns
the total re-queued. POST /api/video/enrichment/retry-all-failed.
- Topbar button (amber, text) → calls it, toasts the count, refreshes the modal.
DB test (resets across matcher + backfill + youtube service, deterministic count) +
frontend wiring test. ruff + node --check clean.
The details backfill queue is keyed on tmdb_id, but the gate (hasattr match) let the
TVDB worker run it too — feeding TMDB ids to TVDB's /series/{id}/extended (→ 404 on
every show) and double-processing each show (TMDB backfilled it, TVDB then 404'd but
still logged 'Backfilled'). Gate on self.service=='tmdb' so only the TMDB worker runs
it. Regression test: the TVDB worker no-ops (never calls its client, leaves the item
pending). 95 enrichment tests pass.
The media server pre-matches shows/movies (tmdb_id set), so the enrichment matcher
skips them and never fetches TMDB details — leaving details-only fields like `status`
(airing vs ended) blank on almost the whole library. That's why the watchlist's
airing-shows default only ever saw the handful of shows whose detail page had been
opened (the one path that force-fetches details). Library here: 3,371 matched shows,
only 18 with status.
Fix: a one-time-per-item details backfill that runs in the enrichment worker's idle
loop (after the episode-sync pass). New `details_synced` marker column on shows+movies;
detail_backfill_next/mark_details_synced/pending_count; worker._detail_backfill_one()
re-fetches an already-matched item's TMDB details and gap-fills (never clobbers server
data), then marks it done so it's attempted once. No re-scan needed — it heals the
existing library in place, and once status is populated the airing-watchlist reflects
real TV.
It's a background gap-fill on already-matched items (like episode coverage), so it
doesn't block the worker's 'Complete' status. kettui: DB seam tests + worker tests
(fills status / enrich-by-id / marks-done-when-absent). ruff + isolation guards green;
94 enrichment tests pass.
Adds a VideoBackfillWorker base that enriches already-identified items BY id
(vs the matcher workers), with the exact same lifecycle + get_stats() shape so
the engine registry, /api/video/enrichment routes, and Manage-Workers modal
drive them identically.
Workers:
- fanart.tv (free key): gap-fills logo/clearart/banner/backdrop/poster art
- OpenSubtitles (free key): records subtitle-language availability per title
- Return YouTube Dislike (no key): like/dislike estimates on cached videos
- SponsorBlock (no key): crowd segments per video
DB: youtube_video_stats + youtube_video_segments tables; fanart/subs columns;
backfill_next/mark/breakdown + youtube_enrich_* helpers; likes/dislikes merged
into get_channel_videos. Seam tests in tests/test_video_backfill.py.
Provider filter (#4):
- client.discover() + engine.discover_filter() take a TMDB provider id and pass
with_watch_providers + watch_region (engine._region) + flatrate. Browse gets a
streaming-service chip row (Netflix/Prime/Disney+/Max/Apple TV+/Hulu/Paramount+/
Peacock); the grid title reflects 'on <service>'.
Infinite scroll (#6):
- Grid paginates via an IntersectionObserver sentinel (600px lookahead) with a
bottom spinner; the Load more button stays only as a no-IO fallback.
Polish (#7):
- Hero keyboard nav (←/→ when Discover is the visible view, ignoring inputs and
while the trailer is open); focus-visible rings on chips/segments/cards/arrows.
Note: 'complete-the-franchise' rail (#5) needs a collection_id per movie, which
the schema doesn't store yet — deferred (would need an enrichment pass).
Tests: +1 (provider watch_region params); updated the discover_filter fake for
the new kwargs. Enrichment + API suites: 115 passed.
Performance:
- Batched ownership: new db.library_ids_for_tmdb() resolves a whole rail in one
query per kind. _stamp_owned (now also used by search + trending) groups by
kind, so a full Discover page drops from ~500 connections to a couple per rail.
Function/data:
- 'See all' on every rail opens it as a paged grid (Load more); the filter bar's
Browse routes through the same generic category grid with a back button + title.
- Personalized 'Because you like <Genre>' rails seeded from your most-owned
genres (new db.top_owned_genres + /discover/taste endpoint).
- 'Hide owned' toggle drops in-library titles from every rail/grid (CSS class,
instant).
Visual vibes:
- Ambient page-top color bleed that follows the current hero slide's hue.
- Rail edge-fade mask, gentle fade-in on load, per-title hue glow on card hover.
Tests: +4 (batched id map, server scoping, one-query-per-kind stamp, top genres).
Full video enrichment + database suites: 145 passed.
- Get-modal: prefetch the first real (un-owned) season on open so its first
expand is instant and its missing episodes pre-count in the footer (prefers
Season 1 over Specials).
- Tests: 10 new seam tests for the Discover data layer — TMDBClient curated/
discover/genres parsing (forced kind, decade date-range, tv first_air_date_year,
backdrop+overview), and engine discover_curated/discover_filter/genre_list
(owned annotation, caching, kind normalization, disabled-worker + error
swallowing). Full video enrichment suite: 79 passed.
A saved 'Where-to-watch region' picker in Settings → Detail Pages (19 common
regions, default US). The engine reads it for the providers in extras +
tmdb_detail (region in the cache key), and the detail page labels the section
'Where to Watch · <region>' so you know which market you're seeing.
Click any episode (owned or missing) to expand it: a larger still, full overview,
and the episode's guest stars (clickable to the person page). Lazy-loaded per
episode from TMDB by the show's tmdb_id and cached. New client.episode_detail +
engine.episode_extra + /api/video/episode/<show_tmdb>/<season>/<episode>.
The owned-item extras and preview detail payloads were already cached (30-min
TTL); person_detail, tmdb_season, and trending were not — so person pages and
lazy preview seasons re-hit TMDB each view. Now cached too (trending 1h). Library
ownership is re-annotated fresh on each call so 'In Library' badges stay current
while the expensive TMDB fetch is reused. Tests for person + season caching.
extras() now returns a featured review (author, rating, snippet, date); the detail
page shows it in a card with a clamped body + Read more/less. In-app (no external
link).
One TMDB call (append_to_response) now also returns: image gallery (backdrops +
posters, thumb+full), all YouTube videos (ordered trailer→teaser→clip→…), keyword
tags, facts (budget/revenue/language/country), and the FULL cast (tv via
aggregate_credits with per-actor episode counts). Shared by item_extras (owned)
and full_detail/tmdb_detail (preview).
Caching: the engine memoizes the live TMDB extras + preview payloads (30-min TTL)
so re-opening a title is instant instead of re-hitting TMDB. Tests added.
- Sort dropdown: Newest / Oldest / Most popular.
- Department filter (Acting / Directing / Writing / …) for multi-hyphenates —
only appears when a person has 2+ departments. Composes with the existing
kind + ownership filters; every chip shows a CONTEXTUAL count (what you'd get
if you clicked it, given the other active filters).
- Age in the hero meta ('47 years old', or lifespan + 'aged N' for the deceased).
- Backend: each person credit now carries its department (cast=Acting,
crew=its TMDB department). Seam test added. 249 video-suite tests pass.
'Request limit reached!' is the free-tier daily quota (1,000/day), not a bad key
— so the worker now idles ~30 min and auto-resumes instead of pausing for good.
A library bigger than the daily cap just spreads its ratings across days on its
own. A genuinely invalid/unactivated key still hard-pauses until fixed. Cooldown
reads as paused in the UI (+ a 'cooldown' flag and note). Item is never burned to
synced, so nothing is lost.
OMDb returns a JSON body even on 401, so surface its actual Error: 'Invalid API
key!' (nudges the user to click OMDb's activation email) vs 'Request limit
reached!' (free-tier daily quota). The worker's pause note uses the same message.
The log flood you saw was the OMDb worker hitting a 401 (invalid key) on every
owned title: it logged a full traceback per item AND marked each one
ratings_synced=1 — which would've stopped them ever retrying once the key was
fixed. Root-cause fixes:
- OMDBClient.ratings raises a distinct OMDbAuthError on 401 / 'Invalid API key!'
(vs a transient error vs a genuine no-data 200).
- Worker: on an auth error it PAUSES (transient, not persisted) with a reason
note + one warning, instead of churning the whole library; the item is NOT
marked synced. Transient errors no longer burn items either — they back off and
pause after 3 in a row. Only a genuine 'no data' marks an item synced. Warnings
are concise (no per-item tracebacks). get_stats exposes the pause 'note'.
- Fixing the key auto-recovers: saving a new/changed OMDb key re-queues every
still-unrated title (resets the wrongly-burned ones), and the engine rebuild
un-pauses the worker.
Seam tests: bad-key pause-without-burn, transient keep-item, ratings() raises on
401, key-change re-queues unrated. 227 video-suite tests pass.
The 'Where to Watch' section is now actionable:
- For an OWNED title it leads with a 'Play on Plex/Jellyfin' tile (green, play
glyph) that deep-links straight to the item on your server — Plex via the
app.plex.tv web app (machineIdentifier fetched once + cached), Jellyfin via its
web detail page. Built in engine.item_extras from the row's server_source +
server_id and the shared media-server config (same source poster.py uses).
- Streaming providers (TMDB/JustWatch) are now clickable → the where-to-watch
page, with a hover lift.
Owned-only: preview (tmdb) items have no library row so they get no server tile.
Seam tests cover the Jellyfin + Plex link building and the unowned no-link case.
240 video-suite tests pass.
Smart back (mirrors music's artist-detail): the top-left back button now
remembers where you actually came from, many layers deep. It keeps an origin
stack ({page} or {detail title}) and stamps each history entry with its layer
depth, so:
- the label is dynamic — '← Back to Search' / '← Back to The Bear' / '← Back to
<person>' — instead of a hardcoded 'Library'/'Back';
- backing out of the first layer returns to the page you started from (Search,
Watchlist, wherever), not always the Library;
- browser Back and our button both unwind the chain one layer at a time, in sync.
Fixes: search → person → back → movie used to mislabel as 'Library' and dump you
in the library.
Next level:
- Search isn't a blank box when idle — a 'Trending this week' rail (TMDB
trending, owned/preview annotated). Returns when you clear the query.
- Person page gets a 'Known For' hero rail (top titles by popularity) above a
full filmography now sorted chronologically (newest first).
Backend: TMDBClient.trending + engine.trending (+library annotation), route
/api/video/trending. Isolated; 237 video-suite tests pass.
Search any movie / show / person (TMDB multi-search) entirely in-app. Results
that you already own link straight to the library detail; the rest open a
TMDB-backed 'preview' detail that reuses the exact same Netflix billboard UI
(direct image URLs, nothing owned/enriched). Everything resolves back into
SoulSync — no external links on un-owned titles.
- Search page (video-search.js): debounced /api/video/search, grouped
movies/shows/people cards (reuses .library-artist-card) with owned/preview
ribbons. People open the person page.
- Source-agnostic detail (video-detail.js): loads from /api/video/detail
(library) or /api/video/tmdb (preview); art helpers pick proxy vs direct URLs;
tmdb shows lazy-load episodes per season; owned-via-tmdb-url auto-redirects to
the library detail.
- 'More Like This' now drills in-app (tmdb detail, redirects if owned); cast/crew
link to a new in-app person page (bio + filmography, each credit owned/preview).
Library credits now carry tmdb_id so owned-item cast is clickable too.
- Backend: TMDBClient.search/full_detail/person (+ shared _parse_extras);
engine.search/tmdb_detail/tmdb_season/person_detail; db.library_id_for_tmdb;
routes /search, /tmdb/<kind>/<id>, /tmdb/show/<id>/season/<n>, /person/<id>.
Isolated (one-way): video-only files, no music imports, music shell untouched.
Seam tests: search/full_detail parsing, tmdb_detail assemble+redirect, search +
person library annotation, library_id_for_tmdb, route registration, shell/JS
isolation. 234 video-suite tests pass.
OMDb now has the same setup as TMDB/TVDB: a yellow dashboard orb (★ glyph) that
spins/idles in the worker-orb animation, an entry in Manage Workers (Ratings
coverage cards, pause/resume, retry, search), and a BACKGROUND ratings pass.
- Worker 'ratings mode' (is_ratings): instead of a match queue it pulls
ratings_next() (library items with an imdb_id and ratings_synced=0), fetches
IMDb/RT/Metacritic, applies + marks synced. So the whole library gets ratings,
not just titles you open (schema v7: ratings_synced).
- enrichment_breakdown/unmatched/retry get an 'omdb' branch (coverage =
ratings-filled, not matched). build_clients includes omdb; the lazy on-view
backfill uses the omdb worker's client.
- Dashboard orb + Manage Workers entry (★ glyph fallback where there's no logo),
yellow accent.
Seam tests: omdb worker rates the queue (ratings mode), ratings breakdown.
Next-level: real critic/audience scores beyond the TMDB star. OMDb (free key,
keyed by the imdb_id we already capture) returns IMDb / RT / Metacritic.
- OMDBClient (ratings + test); built as a non-worker 'ratings_client' on the
engine. _backfill_ratings runs in both lazy detail refreshes (overwrites, since
ratings are dynamic). schema v6: imdb_rating / rt_rating / metacritic on
movies + shows; show/movie payloads return them.
- Billboard renders branded rating badges (IMDb yellow, RT tomato/splat by
fresh/rotten, Metacritic green/yellow/red by score). Lazy refresh also triggers
when an imdb_id exists but ratings are missing.
- OMDb API-key frame in Settings (parity with TMDB/TVDB) + config GET/POST +
/enrichment/omdb/test.
Seam tests: OMDb parse, engine ratings backfill, apply_ratings + payload, config
includes omdb.
So library cards show real owned/total (e.g. 8/10) WITHOUT opening each show. When
the TMDB worker's match queue is clear, it pulls the full season/episode list for
one already-matched-but-unsynced show per loop (episode_sync_next), inserting
missing episodes + marking it synced. Over time every library show gets its full
list; the on-view lazy refresh still makes the one you open instant. TMDB-only;
counts toward the worker's pending so it shows busy (not 'Complete') while syncing,
and never loops on a single failing show.
Seam tests: episode_sync_next selection + pending count, worker syncs a pre-feature
matched show to the full list.
Previously the episodes table held only what the server has (all 'Owned'), so the
detail page never showed what you're missing. Now the metadata provider defines
the full series structure and the server marks ownership:
- TMDB returns the full season list (poster optional) + full episode fields
(title/air date/runtime/still/rating) per season.
- backfill_episodes UPSERTs: owned episodes keep has_file=1; episodes the server
lacks are inserted as MISSING (has_file=0); fully-missing seasons get created.
The cascade now iterates every TMDB season, not just the ones on the server.
- The scan prune only removes SERVER-originated rows (server_id set) that vanished,
so enrichment-added missing episodes/seasons are never pruned on re-scan.
Season coverage (X / Y) is now meaningful, and the episode list shows Owned +
Missing together. Seam tests: missing-episode insert, fully-missing season,
prune preserves missing.
Phase 4: dynamic extras fetched LIVE per view (providers change, so not cached)
via GET /detail/<kind>/<id>/extras → engine.item_extras → TMDB
(videos + watch/providers + similar in one call).
- Trailer: a '▶ Trailer' action that opens an in-app YouTube modal embed (Esc /
click-away to close).
- Where to Watch: provider logos for the region (JustWatch via TMDB).
- More Like This: a poster row of similar titles linking out to TMDB.
Both movie + show pages; all keyless (same TMDB key).
Seam tests: extras parse (trailer priority, provider/similar shape), item_extras
gating on tmdb_id, route registered, markup hooks. (RT/Metacritic via OMDb needs
its own key — offered separately.)
Phase 3: the stylized transparent title logo replaces the text title in the
billboard (the big Plex/Netflix 'premium' jump). Sourced from TMDB images
(append_to_response=images, include_image_language=en,null) in the same detail
call — no Fanart key needed.
- schema v4: movies.logo_url / shows.logo_url (idempotent migration).
- TMDB client picks an English logo (then language-neutral, then any); enrichment
backfills logo_url gap-only; show/movie payloads return 'logo'.
- Billboard shows the logo img (with graceful fallback to the text title on error
/ when absent; title kept visually-hidden for a11y). Lazy on-view refresh now
also triggers when the logo is missing, so existing libraries fill it in.
Seam tests: English-logo pick, backfill + payload, schema.
- Movie cards in the library now drill into a movie-detail page (both kinds use
the same open-detail event / video-side navigation).
- New video-movie-detail subpage reuses the .vd-* hooks; video-detail.js is now
kind-aware (root() targets the active page by kind, billboard/links/actions
branch on movie vs show). Flat layout: billboard + a details strip (released /
runtime / studio / status / critic score / quality) + the shared Cast & Crew row.
- Lazy on-view backfill for movies too: engine.refresh_movie_art re-fetches TMDB
(cast/genres/backdrop/ratings) when missing, regardless of match status, via
POST /detail/movie/<id>/refresh-art. movie_match_info added.
Seam tests: movie refresh backfills cast/genres, movie_match_info, route
registered, movie subpage markup, cards clickable for both kinds.
Last capture piece, schema v3 (new people + credits tables; CREATE IF NOT EXISTS
migrates existing DBs on restart, no wipe):
- people deduped by tmdb_id; credits link to exactly one movie OR show (separate
nullable FKs + CHECK, no polymorphic id) with department/job/character/order.
- TMDB client appends credits to the detail call (free) and parses cast (name,
character, photo, billing order) + headline crew (directors/writers/creators).
- enrichment_apply backfills cast/crew gap-only (never clobbers); show/movie
detail return cast + crew. Populates on view via the existing lazy refresh-art.
- Cast & Crew section on the detail page: grouped crew line + a horizontal
cast row with circular TMDB headshots, names, characters (accent hover).
Seam tests: TMDB credit parse (+ job filtering, created_by), backfill + people
dedup across titles, gap-only no-clobber, payload shape.
Root cause: season posters / episode art backfill happen during a show's TMDB
*match*, but already-matched shows never re-run ('Retry all failed' only resets
not_found/error), so existing libraries never got the art.
Fix (Boulder's idea): fetch-on-view + cache. When a show detail opens and any
season lacks a poster, the page calls POST /detail/show/<id>/refresh-art →
engine.refresh_show_art re-fetches /tv/<id> via the TMDB client and backfills
season posters + episode art gap-only, regardless of match status. Cached, so
it's a one-time cost per show; runs once per view; re-renders when done.
Seam tests: refresh_show_art backfills a MATCHED show's seasons, needs TMDB
configured, show_match_info, route registered.
Brings the video modal to parity with music's:
- 'Process first everywhere' control (Movies/Shows/Auto) in the topbar — a global
setting that pins which kind every worker processes first. enrichment_next takes
a priority kind; the worker reads enrichment_priority each loop; GET/POST
/api/video/enrichment/priority persists it. Reuses music's .em-global styling.
- Needs-matching bar now has a live count, status filter (All unmatched / Not
found / Pending) and a debounced search (reusing .em-select / .em-search),
matching the music modal. Episode view stays read-only.
- Live glow (scoped to #vem-overlay): pulsing running dot, accent glow on the
selected worker row + active process-first/kind, and a pulsing 'now processing'
chip in the worker accent. Music's shared .em-* styles untouched.
Seam tests: priority pins kind in enrichment_next + worker honors the setting,
priority endpoint GET/POST + validation, modal feature markup pinned.
Episodes ride along with their show instead of being a separate (tens-of-thousands)
queue: when the TMDB worker matches a show, it now backfills every season's
episodes — still / overview / rating — via /tv/<id>/season/<n> (one call per
season, gap-only so server data is never clobbered). Also backfills season
overviews.
The worker manager 'knows about it': the TMDB breakdown gains an Episodes
coverage entry (matched = has art, rest = pending), shown as its own card; the
Episodes view lists episodes still missing art. It's coverage-only, kept out of
the worker's idle/pending calc so it never blocks 'Complete'.
Seam tests: client season parse, worker cascade fills episodes, gap-only backfill
+ season overview, breakdown coverage (tmdb only), missing-art list, idle calc
ignores episode coverage.
The media server rarely has distinct per-season art, so season cards fell back to
a gradient. TMDB's show detail carries a poster_path per season — the show worker
now returns those, and enrichment_apply backfills seasons.poster_url for seasons
the server left without art (gap-only, never clobbers server art). The image
proxy streams a stored full URL (TMDB) directly vs. proxying a server path.
Seam tests: TMDB returns season posters, backfill fills only missing seasons.
Enrichment now harvests the full detail payload (same call, no extra requests):
- TMDB: tagline, genres, rating (vote_average), runtime, status, first/last air
date (shows), release date + runtime (movies) — on top of overview/backdrop/ids.
- TVDB: switches to /series/<id>/extended for overview + genres.
enrichment_apply now uses BACKFILL semantics: metadata columns are written via
COALESCE(NULLIF(col,''), ?) so enrichment only fills fields the media server
left empty — it never clobbers server-provided data. Genres backfill to the
normalised link tables only when the item has none yet. Whitelist expanded for
the new columns.
Seam tests: backfill-only (server overview/genres kept, gaps filled), genre
backfill when empty, TMDB full-metadata extraction.
The worker only logged exceptions, so a normal run looked dead — no parity with
the music workers' 'Matched ... -> ID' lines. Now logs each match (noting
'(by server id)' when it used the server's provider id) and each not-found at
INFO. Logger is soulsync.video_enrichment.worker, so it already propagates to
app.log; it just had nothing to say. caplog seam test pins it.
A failed lookup CALL (network/429/5xx/timeout, or an expired TVDB token) was
recorded as 'not_found' — permanently logging a transient blip as 'no match'
and parking the item for retry_days. Now mirrors the music workers' proven
pattern:
- New 'error' status, distinct from 'not_found'; enrichment_next retries BOTH
after retry_days, so errors recover and the queue still advances (no poison
loop). breakdown/unmatched/retry-all and the modal account for it (shown with
the outstanding/pending bucket).
- TMDB/TVDB clients raise on non-200 (429/5xx) so the worker records 'error',
not a false not_found.
- TVDB re-authenticates once on a 401 (expired token) instead of failing every
match for the rest of the run.
Seam tests: error!=not_found, error retried after window, 429 raises, TVDB
token refresh, UI accounts for errors.
The deep scan stores tmdb_id/tvdb_id/imdb_id from Plex/Jellyfin, but the workers
only ever searched by title+year and ignored those ids — re-deriving matches the
server already had exact (wasteful, and a title search can mis-match).
enrichment_next now surfaces the row's known provider id; the worker forwards it
and the TMDB/TVDB clients fetch details BY ID (one call, no /search) when it's
present, falling back to title/year search only for items the server couldn't
identify. Still grabs the overview/backdrop the scan doesn't capture.
Every scan (incremental / full / deep, both entry points) now steps the
enrichment workers aside to cut DB lock contention — same as music. Mirrors
music's contract exactly: pause ONLY workers that were running (a user's manual
pause is left alone), track which we paused, and resume just those in a finally
so success, cancel, and error all clean up. Auto-pause is transient
(persist=False) so it never leaks into the saved <service>_paused flag.
Hooks are injected by get_video_scanner; the scanner is inert without them
(tests build it directly, no engine spun up). Isolated to the video side.
Pause/resume now write <service>_paused to video_settings, and the engine
restores each worker's saved pause when it's (re)built — mirrors music's
<service>_enrichment_paused boot flag. Isolated to video.db; music untouched.
