Brings the video modal to parity with music's:
- 'Process first everywhere' control (Movies/Shows/Auto) in the topbar — a global
setting that pins which kind every worker processes first. enrichment_next takes
a priority kind; the worker reads enrichment_priority each loop; GET/POST
/api/video/enrichment/priority persists it. Reuses music's .em-global styling.
- Needs-matching bar now has a live count, status filter (All unmatched / Not
found / Pending) and a debounced search (reusing .em-select / .em-search),
matching the music modal. Episode view stays read-only.
- Live glow (scoped to #vem-overlay): pulsing running dot, accent glow on the
selected worker row + active process-first/kind, and a pulsing 'now processing'
chip in the worker accent. Music's shared .em-* styles untouched.
Seam tests: priority pins kind in enrichment_next + worker honors the setting,
priority endpoint GET/POST + validation, modal feature markup pinned.
Episodes ride along with their show instead of being a separate (tens-of-thousands)
queue: when the TMDB worker matches a show, it now backfills every season's
episodes — still / overview / rating — via /tv/<id>/season/<n> (one call per
season, gap-only so server data is never clobbered). Also backfills season
overviews.
The worker manager 'knows about it': the TMDB breakdown gains an Episodes
coverage entry (matched = has art, rest = pending), shown as its own card; the
Episodes view lists episodes still missing art. It's coverage-only, kept out of
the worker's idle/pending calc so it never blocks 'Complete'.
Seam tests: client season parse, worker cascade fills episodes, gap-only backfill
+ season overview, breakdown coverage (tmdb only), missing-art list, idle calc
ignores episode coverage.
The media server rarely has distinct per-season art, so season cards fell back to
a gradient. TMDB's show detail carries a poster_path per season — the show worker
now returns those, and enrichment_apply backfills seasons.poster_url for seasons
the server left without art (gap-only, never clobbers server art). The image
proxy streams a stored full URL (TMDB) directly vs. proxying a server path.
Seam tests: TMDB returns season posters, backfill fills only missing seasons.
Enrichment now harvests the full detail payload (same call, no extra requests):
- TMDB: tagline, genres, rating (vote_average), runtime, status, first/last air
date (shows), release date + runtime (movies) — on top of overview/backdrop/ids.
- TVDB: switches to /series/<id>/extended for overview + genres.
enrichment_apply now uses BACKFILL semantics: metadata columns are written via
COALESCE(NULLIF(col,''), ?) so enrichment only fills fields the media server
left empty — it never clobbers server-provided data. Genres backfill to the
normalised link tables only when the item has none yet. Whitelist expanded for
the new columns.
Seam tests: backfill-only (server overview/genres kept, gaps filled), genre
backfill when empty, TMDB full-metadata extraction.
Captures the richer metadata the media server already exposes (schema v2;
idempotent migrations + CREATE IF NOT EXISTS, so an existing DB upgrades on
restart with no wipe):
- movies: tagline, rating (audience), rating_critic; shows: tagline, rating,
first/last air date; episodes: still_url + rating.
- Genres as a normalised many-to-many (genres + movie_genres/show_genres link
tables — no comma-blob), deduped, replace-on-upsert.
- Plex (.genres/.tagline/.audienceRating/.rating/.thumb) + Jellyfin (Genres/
Taglines/CommunityRating/CriticRating/Premiere+EndDate/episode Primary) both
extract them; episode stills served via /api/video/poster/episode/<id>.
- Detail payloads return genres/tagline/rating/air-dates + per-episode has_still;
the billboard shows a tagline, ★ score, genre chips, and episode rows render
REAL stills (no more orange placeholder once scanned).
Seam tests for genre dedup/replace, show+episode capture, episode still ref.
Cast/crew (people + credits) is the next phase.
Addresses the 'feels basic' feedback:
- Hero is now a contained glass card with the backdrop blurred INSIDE it +
gradient overlay (same treatment as the music artist hero) — no more bare
gaps around the top/sides. Bigger poster, accent external-link chips
(IMDb/TMDB/TVDB), refined badges + stat tiles.
- Seasons are a poster-art card grid (season = album) with coverage rings/bars
and hover-lift, selecting one renders its episodes below (episode = track) —
episode overviews now shown. Mirrors the artist album-grid -> tracklist.
- Scan now captures real per-season posters (Plex sh.seasons() thumbs / Jellyfin
/Seasons Primary), served via get_art_ref('season') + /api/video/poster/season.
Falls back to the show poster until a re-scan populates them.
Seam tests for the season art ref; shell markup tests still green.
The worker only logged exceptions, so a normal run looked dead — no parity with
the music workers' 'Matched ... -> ID' lines. Now logs each match (noting
'(by server id)' when it used the server's provider id) and each not-found at
INFO. Logger is soulsync.video_enrichment.worker, so it already propagates to
app.log; it just had nothing to say. caplog seam test pins it.
A failed lookup CALL (network/429/5xx/timeout, or an expired TVDB token) was
recorded as 'not_found' — permanently logging a transient blip as 'no match'
and parking the item for retry_days. Now mirrors the music workers' proven
pattern:
- New 'error' status, distinct from 'not_found'; enrichment_next retries BOTH
after retry_days, so errors recover and the queue still advances (no poison
loop). breakdown/unmatched/retry-all and the modal account for it (shown with
the outstanding/pending bucket).
- TMDB/TVDB clients raise on non-200 (429/5xx) so the worker records 'error',
not a false not_found.
- TVDB re-authenticates once on a 401 (expired token) instead of failing every
match for the rest of the run.
Seam tests: error!=not_found, error retried after window, 429 raises, TVDB
token refresh, UI accounts for errors.
The deep scan stores tmdb_id/tvdb_id/imdb_id from Plex/Jellyfin, but the workers
only ever searched by title+year and ignored those ids — re-deriving matches the
server already had exact (wasteful, and a title search can mis-match).
enrichment_next now surfaces the row's known provider id; the worker forwards it
and the TMDB/TVDB clients fetch details BY ID (one call, no /search) when it's
present, falling back to title/year search only for items the server couldn't
identify. Still grabs the overview/backdrop the scan doesn't capture.
Every scan (incremental / full / deep, both entry points) now steps the
enrichment workers aside to cut DB lock contention — same as music. Mirrors
music's contract exactly: pause ONLY workers that were running (a user's manual
pause is left alone), track which we paused, and resume just those in a finally
so success, cancel, and error all clean up. Auto-pause is transient
(persist=False) so it never leaks into the saved <service>_paused flag.
Hooks are injected by get_video_scanner; the scanner is inert without them
(tests build it directly, no engine spun up). Isolated to the video side.
Pause/resume now write <service>_paused to video_settings, and the engine
restores each worker's saved pause when it's (re)built — mirrors music's
<service>_enrichment_paused boot flag. Isolated to video.db; music untouched.
Each video connection item (TMDB/TVDB) now has a Test button that behaves like
music's: saves the key, hits POST /api/video/enrichment/<svc>/test, and toasts
the result via the shared showToast — isolated (own endpoint, own data-attr
handler, reuses the .test-button CSS).
- Client .test() pings TMDB /configuration and TVDB /login to verify the key.
- Endpoint returns {success,message,error}; unknown service -> 404.
94 tests green; music untouched.
- GET/POST /api/video/enrichment/config saves the keys into video_settings;
POST rebuilds the engine (stop old workers, rebuild clients with new keys) so
they pick up the change live.
- video-settings.js loads the saved keys into the TMDB/TVDB fields on the
Connections tab and saves them on change (workers enable once a key is set).
Backend is now end-to-end: key -> client.enabled -> worker matches the library
to TMDB/TVDB and fills ids + metadata. 91 tests green; real DB untouched.
The servers already matched everything to their agents — we were dropping the
IDs. Now we store them:
- Plex: parse item.guids (imdb://, tmdb://, tvdb://); Jellyfin: parse
item.ProviderIds (added ProviderIds to the requested Fields).
- Stored on movies (tmdb_id, imdb_id), shows (tvdb_id, tmdb_id, imdb_id), and
episodes (tvdb_id) via the upserts.
- Dropped the over-strict UNIQUE on movies.tmdb_id / shows.tvdb_id (same title
can legitimately live in two libraries; we dedupe on server_id). Scanner now
wraps each upsert in try/except so one bad item can't abort a scan.
Tests: guid/ProviderIds parsing + IDs persisted. 38 video-DB/scanner tests green.
- Incremental now does smart early-stopping like music: skips already-known
items and stops after 25 consecutive known (server lists recent first),
instead of a blind fixed cap. Falls back to a full pass when the library is
near-empty (<50), matching music's small-DB behavior.
- Deep scan gains music's 50% safety threshold: if removal would wipe >50% of a
>100-row library, it skips (assumes a partial server response, not a real
emptying) — prevents catastrophic deletion.
- Full Refresh already matched (re-read all, upsert, no removal).
Added DB helpers (server_ids, table_count). Tests: early-stop skips known,
small-lib fallback, 50% prune safety. 122 tests green.
The scan tool now behaves like music's, not just looks like it:
- Card matches: help '?' button, 'Last Scan' line, and the Movies/Shows/
Episodes/Size stats grid (populated from /api/video/dashboard on show + after
a scan). Same .tool-card-stats markup.
- Real progress bar: scanner fetches item totals up front (Plex section.
totalSize / Jellyfin TotalRecordCount) and reports a true percent as it
processes; the bar actually moves (movies → shows) instead of sitting at 100%.
- Cancel: the Scan button toggles to 'Cancel' mid-scan and POSTs
/api/video/scan/stop; the scanner checks a cancel flag between items and ends
in a 'cancelled' state. Mirrors music's stop affordance.
Tests: percent reported, cancel stops midway + saves only processed items, stop
route registered, tool-card structure. 117 video/integrity tests green.
- Paginate Jellyfin movie/series listings (StartIndex/Limit, 500/page) so large
libraries aren't truncated on full/deep scans; incremental stays a single
capped page.
- Per-item try/except in iter_movies/iter_shows (matches Plex) so one bad item
doesn't abort the scan.
- Skip Jellyfin episodes with no IndexNumber (consistency with Plex; avoids
ep-0 collisions). All three modes now solid for Plex + Jellyfin.
The scan no longer blindly grabs every movie/show section — it reads the
libraries you map, like music's 'pick your Music library'.
- GET /api/video/libraries: discover the active server's Movies/TV libraries
(Plex sections by type / Jellyfin views by CollectionType) + current
selection. POST: save {movies, tv} per server into video_settings.
- sources.py: _build_source(movies_lib, tv_lib) filters to the mapped library;
get_active_video_source() (used by the scanner) loads the saved selection;
list_video_libraries() lists them unfiltered for the UI. Falls back to all
libraries when nothing is mapped yet.
- VideoDatabase.get/set_library_selection (per-server). 6 tests added; 33 green.
Plex specials/unmatched episodes can have a null index -> getattr(...,0) still
returned None -> 'NOT NULL constraint failed: episodes.episode_number'.
- Plex adapter skips episodes with no index (logged), passes a real number.
- upsert_show_tree defensively skips any episode missing season/episode number,
so no source can crash a scan. Test added.
The scan inherited the shared client's 15s interactive timeout and fetched
episodes per-season (one request each), so a big library read-timed-out
mid-scan (port 32400).
- Dedicated Plex connection for scans with a 120s timeout (built from the
shared config; doesn't touch the interactive client).
- Fetch a show's episodes in ONE show.episodes() call grouped by season,
instead of seasons()+episodes() per season — far fewer round-trips.
- Per-item try/except in iter_movies/iter_shows so one slow/bad item is skipped
and logged, never aborting the whole scan.
Mirrors the music model (full_refresh vs smart incremental, plus deep_scan):
- incremental: only recently-added items from the server (Plex addedAt:desc /
Jellyfin DateCreated, capped); upsert; no prune.
- full: every item; upsert all (refresh metadata + add new); no prune.
- deep: every item; upsert; prune what the server no longer has (empty-scan
safety preserved).
scanner.request_scan/scan_sync take mode; /api/video/scan/request reads
{mode} from the body (default full); adapters take incremental=. Tests cover
deep-prunes / full-doesn't / empty-deep-safety / incremental-requests-recent.
Reads the active media server and mirrors it into video.db, adapting the music
scan pattern (ask the server, upsert, prune what's gone) — isolated from music.
- core/video/scanner.py: server-agnostic VideoLibraryScanner. Consumes a media
source (duck-typed) yielding normalized dicts; upserts movies + show trees,
prunes removed items, reports progress/state. Skips pruning when a scan
returns nothing (transient-failure safety). Background thread + scan_sync.
- core/video/sources.py: Plex + Jellyfin adapters that REUSE the shared
connected clients (MediaServerEngine) but own all video-section logic; produce
normalized dicts. (Validated against a live server by design; scanner itself
is fully unit-tested with a fake source.)
- api/video/scan.py: POST /api/video/scan/request, GET /api/video/scan/status.
- .gitignore: video_library.db + sidecars (mirrors music); tests inject a
tmp DB so none is ever created in the repo.
Tests: scan populate/prune/empty-safety/no-source-error, isolation guard
(core/video imports nothing from music), scan routes registered. 101 green.
Enrichment matched artists by NAME ONLY (0.85 gate), so for a common name
('Rone' has ~5 artists) it stored whichever the source ranked first — often the
wrong one, which then drove a wrong/sparse library 'Standard' discography while
'Enhanced' (the real owned albums) showed the full set.
Fix — use the decisive signal the library already has (the albums you OWN):
- worker_utils: pick_artist_by_catalog() + catalog_overlap_score() +
owned_album_titles()/release_titles(). When 2+ candidates clear the name gate,
fetch each one's catalog and choose the one overlapping the owned albums; falls
back to the current best-by-name pick when there's nothing to disambiguate or
no overlap (so the common single-candidate path makes no extra API calls).
- Wired into Spotify (covers Spotify-Free, same client), iTunes, Deezer (now
multi-candidate search_artists + get_artist_info store), and MusicBrainz
(match_artist gains owned_titles; release-groups as the catalog).
Re-match path (#868):
- build_reset_query now also clears the stored source-ID column for artist/album
item resets — previously a 're-match' only nulled match_status, so the worker's
existing-id short-circuit re-confirmed the WRONG id and never re-resolved. Tracks
excluded (ids live in tags, not a column).
- MusicBrainz also self-corrects its 90-day name->mbid cache: match_artist bypasses
a cached mbid whose catalog has ZERO overlap with the owned albums, so a re-match
isn't blocked by a stale wrong cache entry.
Tests: shared selector (9), per-worker disambiguation for all 4 sources + MB
backward-compat + MB cache-revalidation (8), reset-clears-id (2). 99 worker/
enrichment tests green.
Four refinements on top of the tiered matcher:
1. Direct source track-ID tier (new top tier): enrichment writes each source's own
track ID into the file tags (spotify_track_id/deezer_track_id/itunes_track_id/...).
If we have the active source's track ID, fetch that exact track by ID via
get_track_details — zero search. Tiers are now: track-ID -> ISRC -> album->track
-> artist+title. _read_file_ids reads ISRC + all per-source IDs in one tag read.
2. Skip already-proposed tracks: a re-run loads existing finding entity_ids for the
job and skips those tracks before any API call (pending stays deduped, dismissed
stays dismissed) — re-runs are cheap.
3. Wrong-version guard: the fuzzy tiers (album-search + track search) reject a
candidate whose length differs from ours by >5s (live/edit/remix with same title).
_load_tracks now selects t.duration; exact tiers (track-ID/ISRC/stored-album-ID)
skip the guard.
4. Tighter album matching: same-title cuts in an album are disambiguated by closest
duration when track_number doesn't decide it.
Findings record matched_via = track_id | isrc | album | search. 30 repair tests pass
(added track-ID tier, duration guard, dedup-skip, and unit coverage).
Replaces the blind fuzzy search with a smart hierarchy that uses the data we
already have, best identity first:
1. ISRC embedded in the file tags (enriched track) -> exact track.
2. Album -> track: use the album's stored source ID (albums.spotify_album_id /
itunes_album_id / deezer_id / musicbrainz_release_id / audiodb_id) when the
ALBUM is enriched (even if the track isn't); else find the album by searching
'artist album', then locate our track in that album's tracklist by normalized
title (track_number breaks ties). Pins the exact album context. (artist->album->track)
3. Plain artist+title search with similarity scoring. (artist->track) — loosest.
_load_tracks now returns dict rows (adds track_number + the album source-id
columns). Findings record matched_via = isrc | album | search. All clients
(spotify/deezer/itunes/discogs) expose search_albums + get_album_tracks with a
uniform {'items': [...]} shape, so the album tier is source-agnostic.
26 repair tests pass (added album-tier + _find_track_in_album coverage).
The job was doing a blind fuzzy search for every low-quality track, ignoring that
enrichment writes each track's ISRC + per-source IDs into the file tags. Now it
reads the file's embedded ISRC and resolves the EXACT track via each source's
'isrc:' search (universal cross-source key), guarded by an ISRC-equality check so
a source that ignores the syntax can't produce a false match — exact track, exact
album context, one call. Falls back to the name/artist fuzzy search only for
un-enriched tracks with no usable ISRC. Findings record matched_via=isrc|search.
4 new seam tests (guard accept/reject, ISRC-preferred-over-fuzzy, fuzzy fallback).
Phase 2 of the redesign. The tool that judged quality by extension and auto-dumped
matches into the wishlist is gone; quality scanning is now the reviewed
quality_upgrade repair job.
Removed:
- Frontend: Tools-page Quality Scanner card, its JS handlers/poller/socket listener,
help tooltip + tour entry (webui index.html, core.js, helper.js, wishlist-tools.js).
- Backend: /api/quality-scanner/{start,status,stop} endpoints, the in-memory state +
executor + 1s socket broadcast, the QualityScannerDeps/run_quality_scanner shim.
- core/discovery/quality_scanner.py: the auto-acting worker + deps class (the shared
match/normalize helpers stay — the new job imports them).
Rewired:
- Automation 'start_quality_scan' action now triggers the quality_upgrade repair job
via repair_worker.run_job_now() (AutomationDeps gains run_repair_job_now, drops the
4 scanner fields). Action block's vestigial scope field removed (scope lives in the
job's settings now). NOTE: the 'quality_scan_completed' trigger no longer fires (the
repair job doesn't emit it).
- Updated all automation test _build_deps helpers + conftest tool-progress harness;
deleted the obsolete worker test. 528 affected tests pass; 6123 collect cleanly.
QUALITY_TIERS / _get_quality_tier_from_extension kept (used elsewhere).
The old Quality Scanner tool judged quality by file EXTENSION only (a 128k and a
320k MP3 looked identical), ignored the bitrate-based quality profile, used min()
of enabled tiers so the default profile flagged the ENTIRE non-lossless library,
and auto-dumped every match into the wishlist with no review.
This new repair job does it properly:
- meets_preferred_quality(): pure, bitrate-AWARE decision honoring every enabled
quality bucket (320 MP3 passes a FLAC+320+256 profile; 128 MP3 doesn't). Floor
is the worst enabled bucket, not the best.
- scans watchlist artists or whole library, finds below-quality tracks, matches a
better version at scan time (reusing the existing tested match helpers), emits a
FINDING showing the match + confidence. Off by default; nothing auto-queued.
- _fix_quality_upgrade apply handler adds the matched track WITH album context to
the wishlist — the user-approved version of what the old tool did silently.
- Transcode/fake-lossless detection intentionally left to the existing Fake
Lossless Detector job.
12 seam tests incl. a regression pinning the default-profile flooding bug. The old
tool is still in place; removing it + rewiring its automation action is the next step.
Two issues in the same path:
1. The shared discovery modal pre-renders one row per track from a
separately-fetched frontend track list, then the poll dropped any backend
result without a pre-rendered row (if (!row) return). When the frontend's
track fetch came back rate-limited/partial (~21) while discovery's own fetch
got all 59, the surplus results vanished. Now the modal CREATES a row for any
result lacking one, so authoritative backend results drive the list (fixes
all sources sharing the modal).
2. get_playlist hydrated a whole relationships page in one _get_tracks_batch
call, but Tidal caps filter[id] at 20/request, silently truncating larger
pages. Chunk to the cap like get_album_tracks already does.
Seam + regression tests (tests/test_tidal_playlist_batch_chunking.py).
Status checks asked is_spotify_authenticated() (official OAuth only) instead of
is_spotify_metadata_available(), so a Spotify-Free primary read as disconnected.
get_primary_source_status had spotify_free awareness but it was dead code:
get_client_for_source('spotify') returns None unless officially authed, so the
free-availability probe never had a client. Fetch the client directly for that
check; add the missing free branch to the dashboard test message. Seam + regression tests.
The reconcile read each completed task's final_file_path to find paths — but not
every import path sets it (the verification worker marks the task completed
without it), so tracks that imported via that path were silently dropped (user
saw 3 of 5 symlinks). Root cause: leaning on a fragile per-task field.
Now reconcile_batch_playlists identifies the organize playlists the batch touched
(its own + any reached via a completed track's source_info provenance) and
rebuilds each from CURRENT library ownership via _rebuild_one_from_db
(check_track_exists over membership). It just asks the library what's owned, so
it's robust to HOW a track imported (modal worker / slskd monitor / verification
worker) and still prunes tracks that left. Takes a db handle; all three callers
pass MusicDatabase().
Reconcile tests rewritten for the DB-rebuild form (organize batch, wishlist
provenance, non-organize skip, plain no-op). 973 downloads/imports/playlist
tests pass.
on_download_completed and check_batch_completion_v2 are duplicate completion
paths. Monitor-detected downloads (Deezer / slskd-monitor / verification-worker
imports) finish the batch via the V2 path, but the materialize reconcile was
only added to on_download_completed — so those batches never built playlist
folders (no '[Playlist Folder] Rebuilt' line at all). Add the same non-fatal
reconcile to the V2 path. Now all three completion points (both lifecycle paths
+ the master.py all-owned path) materialize. 550 tests pass.
Symmetric to the post-download reconcile (which handles ADDITIONS): when a
playlist's membership is re-synced (the mirror step — scheduled refresh or the
manual mirror endpoint), rebuild its folder from current membership WITH prune
IF it's organize-by-playlist. So a track that just LEFT the playlist has its
symlink cleaned up the instant membership changes, not only on the next download.
Factored a shared _rebuild_one_from_db (used by the manual 'Rebuild' button and
the mirror hook) + rebuild_mirrored_playlist_if_organized. Gated to organized
playlists, non-fatal at both mirror call sites.
Now the invariant 'folder = the playlist's current owned members' holds on every
change: additions caught at download, removals caught at mirror. 2 new tests
(removed track pruned; non-organized skipped). 985 + 277 tests pass.
