The media server rarely has distinct per-season art, so season cards fell back to
a gradient. TMDB's show detail carries a poster_path per season — the show worker
now returns those, and enrichment_apply backfills seasons.poster_url for seasons
the server left without art (gap-only, never clobbers server art). The image
proxy streams a stored full URL (TMDB) directly vs. proxying a server path.
Seam tests: TMDB returns season posters, backfill fills only missing seasons.
Existing DBs created movies.tmdb_id / shows.tvdb_id as inline UNIQUE (can't be
dropped via migration). The new model allows the same title in >1 library, so a
second movie/show with the same id raised IntegrityError and the scanner SKIPPED
it — dropping the title (observed: 'UNIQUE constraint failed: movies.tmdb_id',
movie 548522 skipped).
upsert_movie/upsert_show_tree now use a shared _resilient_upsert: on
IntegrityError, retry WITHOUT the id columns so the row is stored (just without
the colliding id) — same pattern enrichment_apply already used. Regression tests
for both movies and shows under a simulated legacy unique index.
Enrichment now harvests the full detail payload (same call, no extra requests):
- TMDB: tagline, genres, rating (vote_average), runtime, status, first/last air
date (shows), release date + runtime (movies) — on top of overview/backdrop/ids.
- TVDB: switches to /series/<id>/extended for overview + genres.
enrichment_apply now uses BACKFILL semantics: metadata columns are written via
COALESCE(NULLIF(col,''), ?) so enrichment only fills fields the media server
left empty — it never clobbers server-provided data. Genres backfill to the
normalised link tables only when the item has none yet. Whitelist expanded for
the new columns.
Seam tests: backfill-only (server overview/genres kept, gaps filled), genre
backfill when empty, TMDB full-metadata extraction.
Captures the richer metadata the media server already exposes (schema v2;
idempotent migrations + CREATE IF NOT EXISTS, so an existing DB upgrades on
restart with no wipe):
- movies: tagline, rating (audience), rating_critic; shows: tagline, rating,
first/last air date; episodes: still_url + rating.
- Genres as a normalised many-to-many (genres + movie_genres/show_genres link
tables — no comma-blob), deduped, replace-on-upsert.
- Plex (.genres/.tagline/.audienceRating/.rating/.thumb) + Jellyfin (Genres/
Taglines/CommunityRating/CriticRating/Premiere+EndDate/episode Primary) both
extract them; episode stills served via /api/video/poster/episode/<id>.
- Detail payloads return genres/tagline/rating/air-dates + per-episode has_still;
the billboard shows a tagline, ★ score, genre chips, and episode rows render
REAL stills (no more orange placeholder once scanned).
Seam tests for genre dedup/replace, show+episode capture, episode still ref.
Cast/crew (people + credits) is the next phase.
Season selection is now switchable via a view toggle (persisted): poster RAIL
(scrollable season cards w/ coverage), TIMELINE band (segments sized by episode
count, filled by owned), TABS (pills), and the LIST dropdown. All drive the same
selection; episodes fade in on change.
- Watchlist button is now REAL: toggles shows.monitored via POST /api/video/monitor
(set_monitored), reflects 'In Watchlist' state. show_detail returns monitored.
- 'Get Missing' + a 'Missing only' toolbar toggle filter the episode list to
unowned episodes (actual downloading is the future acquisition subsystem).
Seam tests for the monitor endpoint + bad-input guards; shell hooks updated.
Addresses the 'feels basic' feedback:
- Hero is now a contained glass card with the backdrop blurred INSIDE it +
gradient overlay (same treatment as the music artist hero) — no more bare
gaps around the top/sides. Bigger poster, accent external-link chips
(IMDb/TMDB/TVDB), refined badges + stat tiles.
- Seasons are a poster-art card grid (season = album) with coverage rings/bars
and hover-lift, selecting one renders its episodes below (episode = track) —
episode overviews now shown. Mirrors the artist album-grid -> tracklist.
- Scan now captures real per-season posters (Plex sh.seasons() thumbs / Jellyfin
/Seasons Primary), served via get_art_ref('season') + /api/video/poster/season.
Falls back to the show poster until a re-scan populates them.
Seam tests for the season art ref; shell markup tests still green.
Backend for the upcoming TV/movie detail pages, isolated to video.db:
- show_detail(id): show + seasons->episodes tree with owned/total roll-ups
(season 0 -> 'Specials', missing-season-row episodes still grouped).
- movie_detail(id): movie + owned flag + best media-file (resolution/quality).
- get_art_ref generalizes the poster ref to poster|backdrop; new
/api/video/backdrop/<kind>/<id> streams the hero art server-side (Jellyfin
Backdrop vs Primary handled).
- /api/video/detail/{show,movie}/<id> endpoints.
Seam tests for the tree roll-ups, owned/file, art ref, and both endpoints.
A failed lookup CALL (network/429/5xx/timeout, or an expired TVDB token) was
recorded as 'not_found' — permanently logging a transient blip as 'no match'
and parking the item for retry_days. Now mirrors the music workers' proven
pattern:
- New 'error' status, distinct from 'not_found'; enrichment_next retries BOTH
after retry_days, so errors recover and the queue still advances (no poison
loop). breakdown/unmatched/retry-all and the modal account for it (shown with
the outstanding/pending bucket).
- TMDB/TVDB clients raise on non-200 (429/5xx) so the worker records 'error',
not a false not_found.
- TVDB re-authenticates once on a 401 (expired token) instead of failing every
match for the rest of the run.
Seam tests: error!=not_found, error retried after window, 429 raises, TVDB
token refresh, UI accounts for errors.
The deep scan stores tmdb_id/tvdb_id/imdb_id from Plex/Jellyfin, but the workers
only ever searched by title+year and ignored those ids — re-deriving matches the
server already had exact (wasteful, and a title search can mis-match).
enrichment_next now surfaces the row's known provider id; the worker forwards it
and the TMDB/TVDB clients fetch details BY ID (one call, no /search) when it's
present, falling back to title/year search only for items the server couldn't
identify. Still grabs the overview/backdrop the scan doesn't capture.
Existing DBs created before the schema dropped UNIQUE still have shows.tvdb_id /
movies.tmdb_id UNIQUE, so enrichment matching two items to the same id threw
'UNIQUE constraint failed' repeatedly. enrichment_apply now catches the
IntegrityError and retries without the id columns — keeps the existing
(authoritative) id and still records match_status + metadata. Non-destructive
(no table rebuild). Test simulates the legacy unique index.
Foundation for the video enrichment workers, mirroring music's per-source
columns/queries on video.db.
- Schema: tmdb_match_status/tmdb_last_attempted on movies; tmdb_+tvdb_ on shows.
Idempotent ALTER-TABLE migration adds them to existing DBs on init.
- VideoDatabase helpers (service+kind -> columns map):
enrichment_next (pending first, then not_found past retry window),
enrichment_apply (sets id/status/last_attempted + whitelisted metadata,
backfill-safe), enrichment_breakdown, enrichment_unmatched (paged), and
enrichment_retry. Same shape as music's enrichment API so the shared modal can
drive it. 30 DB tests green.
The servers already matched everything to their agents — we were dropping the
IDs. Now we store them:
- Plex: parse item.guids (imdb://, tmdb://, tvdb://); Jellyfin: parse
item.ProviderIds (added ProviderIds to the requested Fields).
- Stored on movies (tmdb_id, imdb_id), shows (tvdb_id, tmdb_id, imdb_id), and
episodes (tvdb_id) via the upserts.
- Dropped the over-strict UNIQUE on movies.tmdb_id / shows.tvdb_id (same title
can legitimately live in two libraries; we dedupe on server_id). Scanner now
wraps each upsert in try/except so one bad item can't abort a scan.
Tests: guid/ProviderIds parsing + IDs persisted. 38 video-DB/scanner tests green.
Handles big libraries (your ~8500 movies) like music does instead of rendering
everything at once.
- DB: sort_title populated article-aware on upsert ('The Matrix' files under M);
query_library(kind, search, letter, sort, status, page, limit) does all
filtering/sorting/paging in SQL and returns music's pagination shape
{page,total_pages,total_count,has_prev,has_next} + badge fields (resolution,
owned/episode counts).
- GET /api/video/library now takes those params (per kind) instead of dumping
everything.
- Library page: 75/page with ← Previous / Page X of Y / Next → (music's exact
controls/classes), Sort (Title/Year/Recently Added) + Owned/Wanted filter,
server-side search + A–Z. Cards gain a resolution chip (4K/1080p/…) and the
owned/wanted meta. Still not clickable.
124 tests green.
- Incremental now does smart early-stopping like music: skips already-known
items and stops after 25 consecutive known (server lists recent first),
instead of a blind fixed cap. Falls back to a full pass when the library is
near-empty (<50), matching music's small-DB behavior.
- Deep scan gains music's 50% safety threshold: if removal would wipe >50% of a
>100-row library, it skips (assumes a partial server response, not a real
emptying) — prevents catastrophic deletion.
- Full Refresh already matched (re-read all, upsert, no removal).
Added DB helpers (server_ids, table_count). Tests: early-stop skips known,
small-lib fallback, 50% prune safety. 122 tests green.
Rebuilt the Library page to reuse the music library's exact look — no
reinvention, just new data:
- Same classes: .library-container, .library-artist-card grid, .alphabet-
selector, .library-search-input, loading/empty states. Movies/Shows tab pill
is the only video-specific bit.
- Real posters via a server-side proxy: GET /api/video/poster/<kind>/<id>
streams the Plex/Jellyfin artwork (token stays server-side); cards fall back
to an emoji on miss. list_movies/list_shows now expose has_poster (no raw
server paths leaked).
- Client-side search + A-Z letter filter (article-aware) over the loaded set;
cards are divs (not clickable yet, per request). Scan button in the header
reuses the shared scan controller and reloads on done.
110 tests green.
The scan no longer blindly grabs every movie/show section — it reads the
libraries you map, like music's 'pick your Music library'.
- GET /api/video/libraries: discover the active server's Movies/TV libraries
(Plex sections by type / Jellyfin views by CollectionType) + current
selection. POST: save {movies, tv} per server into video_settings.
- sources.py: _build_source(movies_lib, tv_lib) filters to the mapped library;
get_active_video_source() (used by the scanner) loads the saved selection;
list_video_libraries() lists them unfiltered for the UI. Falls back to all
libraries when nothing is mapped yet.
- VideoDatabase.get/set_library_selection (per-server). 6 tests added; 33 green.
Plex specials/unmatched episodes can have a null index -> getattr(...,0) still
returned None -> 'NOT NULL constraint failed: episodes.episode_number'.
- Plex adapter skips episodes with no index (logged), passes a real number.
- upsert_show_tree defensively skips any episode missing season/episode number,
so no source can crash a scan. Test added.
- GET /api/video/library -> {movies, shows} from video.db (VideoDatabase.
list_movies/list_shows; shows carry episode_count + owned_count).
- Library page (video-library subpage, isolated video-library.js): tabbed
Movies/Shows grid of poster cards, count, empty-state. A 'Scan Library'
button POSTs /api/video/scan/request then polls /api/video/scan/status,
showing live phase/counts, and refreshes the grid when done.
- Reuses the music dashboard-header chrome (icon title, sweep hidden) + the
watchlist-button styling for the scan button; video-card grid styles added.
- All data-attr wired (no inline onclick); module is an isolated IIFE that
listens for soulsync:video-page-shown. 105 tests green.
Now: video.db -> scanner -> /api/video -> live dashboard + Library page, all
isolated from music. Scanner adapters await live Plex/Jellyfin validation.
Server (Plex/Jellyfin) is the source of truth, so every scanned row carries
(server_source, server_id) for upsert + stale-removal — mirroring how music
keys on server_source + ratingKey.
- schema: server_source/server_id columns on movies/shows/episodes (+ server_id
on seasons); unique (server_source,server_id) on movies/shows (multiple NULLs
allowed so wishlist rows never block).
- VideoDatabase.upsert_movie / upsert_show_tree: take normalized, server-
agnostic dicts (a Plex/Jellyfin adapter produces them — DB never touches a
media SDK), set has_file + media_files, and prune episodes/seasons the server
no longer reports.
- prune_missing(): removes top-level movies/shows the scan didn't see (cascades
clean children).
6 new tests (insert/update/file-replace, season/episode build+prune, top-level
prune); 18 video-DB tests green.
First wire from video.db -> UI, kettui-style.
- api/video/ : isolated Flask blueprint (registered at /api/video with one
additive line in web_server.py). Reads only video.db; imports nothing from
the music API or DB.
- GET /api/video/dashboard -> VideoDatabase.dashboard_stats(): live library/
download/watchlist/wishlist counts (real 0s on an empty DB).
- video-dashboard.js now fetches it and fills the stat cards + Watchlist/
Wishlist header badges (formatted bytes/speed); falls back to zeros on error.
uptime/memory stay at markup defaults for now (not video-domain).
- Tests: dashboard_stats counts (empty + populated), endpoint returns zeroed
JSON via a Flask test client, blueprint exposes the route, and the video API
imports nothing from music. 93 video/integrity tests green.
Separate SQLite file (database/video_library.db, env VIDEO_DATABASE_PATH),
fully disconnected from music — never imported by music, imports nothing from
music, no shared write lock.
Schema (database/video_schema.sql), designed to dodge the music DB's known
pain points:
- movies; shows->seasons->episodes; channels->channel_videos (YouTube as a
first-class peer); media_files (the library); downloads (queue+history);
activity feed; root_folders / quality_profiles / video_settings config.
- No polymorphic ids: media_files/downloads use separate nullable FKs + a CHECK
that exactly one owner is set; real cascades.
- Explicit external-id columns (tmdb/tvdb/imdb/youtube), no source-id blob.
- Watchlist/Wishlist/Calendar are DERIVED VIEWS over monitored + file state
(single source of truth, can't drift like music's wishlist table did).
VideoDatabase mirrors music's conventions (WAL, foreign_keys ON, 30s busy
timeout, Row factory, once-per-process init, user_version backstop) but is an
independent implementation. 13 seam tests: schema builds, CHECK constraints
reject bad rows, cascades fire, views return correct membership, KV roundtrips,
and a guard that the module imports nothing from music.
After saving a password or recovery question, a refresh made the section look
unset (passwords are never echoed back to the browser), so it seemed like you had
to redo it. Now the saved state is reflected:
- "✓ A login password is set" appears when the admin has a password; the field
becomes "Enter a new password to change it".
- "✓ Recovery question saved: <question>" appears, the saved question is pre-
selected (preset or custom), and the answer field becomes "Enter a new answer to
change it".
- Shown both on load (applyLoginSavedState from /api/profiles, which now includes
recovery_question — not secret, already shown on the sign-in screen) and
immediately after saving.
64 integrity tests pass.
The security section had grown into a flat pile of toggles with hidden
dependencies. Regrouped into three labelled cards so it reads top-to-bottom:
- 🔑 Lock with a PIN — set PIN (Step 1) → Require PIN
- 👤 User accounts (login) — Step 1 admin password → Step 2 recovery question →
Step 3 Require login. The Step 3 toggle is now visually LOCKED (greyed +
disabled + "set the admin password first" hint) until an admin password exists,
so the anti-lockout rule is obvious instead of surfacing as a 400 on save. It
unlocks the moment the password is saved.
- 🌐 Reverse proxy & remote access — the proxy toggle, with the auth-proxy header
nested under it (indented), plus WebSocket origins.
- get_all_profiles/get_profile now expose has_password + has_recovery so the UI
can reflect setup state; updateRequireLoginGate() drives the lock.
- New .security-subgroup/.security-subhead/.security-nested/.security-locked CSS.
All IDs + handlers preserved. Inert unless used; default install unaffected.
64 script-split integrity tests pass.
Closes the forgot-login-password gap. A per-profile recovery question + answer lets
a locked-out user reset their own password.
- DB: additive recovery_question + recovery_answer_hash columns (idempotent
migration). set/get-question/verify/has methods; answer is hashed (pbkdf2) and
matched forgivingly (trim + lowercase + collapse whitespace). No recovery set →
never verifies.
- Endpoints (allowlisted in the login gate so they work pre-auth):
GET /api/auth/recovery-question?username= (generic 404 when absent),
POST /api/auth/recovery-reset {username, answer, new_password} — brute-force
limited; a correct answer sets the new password + authenticates the session.
POST /api/profiles/<id>/set-recovery (admin or self) to configure it.
Tests: set/get/verify, forgiving match, hashed-not-plaintext, no-recovery-never-
verifies, full reset flow (wrong answer rejected + password intact; correct answer
resets), unknown-user 404. 25 tests pass. Next: the Settings + login-screen UI.
Opt-in username/password login — profiles become real accounts. This is the data
layer: a per-profile login password, kept SEPARATE from the quick-switch PIN
(different security purpose; a 4-digit PIN must not become the password guarding a
public instance).
- Additive migration: profiles.password_hash column (idempotent, metadata-flagged).
- set_profile_password / verify_profile_password / profile_has_password /
get_profile_by_name (the login username = profile name, unique + case-insensitive).
- Security default: a profile with NO password is NOT loginable (verify returns
False) — unlike the PIN where "no PIN = always valid". You can't authenticate to
an account with no credential.
Tests: migration adds the column; set/verify; no-password-never-loginable; clearing;
name lookup; and password is fully independent of the PIN. 6 tests pass. Next:
the login endpoint + require_login gate (increment 2).
Reported via Find & Add (Billie Eilish "bad guy"): the track was in the library
and on Plex, but never showed in the modal's 20 results. Root cause (proven
against the real 307k-track DB): the search did `ORDER BY tracks.title`, which is
case-SENSITIVE in SQLite (BINARY collation sorts 'B' before 'b'). Billie's title
is lowercase "bad guy"; everyone else's is "Bad Guy", so all the capitalised ones
sorted first, filled the LIMIT, and her exact match landed at ~#25 — cut off.
- search_tracks now ranks by relevance: exact title match first (case-insensitive
via unidecode_lower), then prefix, then alphabetical — so an exact match can't
be sorted below the limit by a capital letter. Helps every caller.
- Added a rank-only `rank_artist` hint (never filters): Find & Add already knows
the source track's artist, so it now passes it and the exact title+artist match
floats to #1. Filtering was deliberately avoided — if the track is tagged under
a slightly different artist on the server, a filter would re-hide it.
Verified on the real DB: title-only "bad guy" now surfaces Billie at #4 (was
>#20); with the artist hint she's #1. Seam tests: lowercase exact title isn't
buried; rank hint floats the match without filtering; exact title beats a
superstring title. 10 tests pass.
- ⚠ Unverified filter rows gain actions: inline play (range-streamed from the
history file path, server-side only), YouTube compare, Approve -> new
human_verified status (tag + history + tracks; AcoustID scanner skips these
entirely), Delete (file + entry)
- API: /api/verification/<id>/stream|approve|delete (path only from DB row)
- backfill: history rows with acoustid_result='fail' that exist at all were
imported despite the failure = force_imported (covers pre-fix fallback
imports like the user's 'My Ordinary Life')
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pipeline has three success exits (simple download, playlist folder mode,
main) but only the main one persisted the verification status — force-imported
playlist tracks got no tag, no history status, and never appeared in the
Unverified filter. Extracted _persist_verification_status() and call it at
every exit. One-time idempotent backfill derives status for existing history
rows from their recorded acoustid_result (pass->verified, skip->unverified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The persistent Completed list is built from library_history (not live tasks),
so the badge never showed after a session ended. Column added (additive),
written at import, passed through _build_history_download_item, rendered by
_adlVerifBadge next to the status label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Also: evaluate() treats an empty expected artist as title-only comparison
(old scanner behaviour — a missing DB artist is no evidence of a wrong file),
and the thresholds are now defined once in the core and re-exported.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second service. Each profile connects its own Tidal; its playlist reads use that
account, everything else stays global. The gotcha vs Spotify: TidalClient loads
AND saves tokens to one global slot (tidal_tokens), so a naive per-profile client
would clobber the admin's tokens on refresh.
- get_tidal_client_for_profile builds a dedicated TidalClient seeded with the
profile's tokens, refreshed via the shared/global app creds, and OVERRIDES its
_save_tokens to persist to the PROFILE row — never the global slot. Admin
(profile 1) + unconnected profiles use the global client unchanged. Cached per
profile + evicted on (dis)connect.
- DB: set_profile_tidal_tokens / get_profile_tidal (encrypted); the OAuth callback
now uses them + evicts the cached client.
- Wired the Tidal playlist reads (list + tracks) to the per-profile client; the
module import line left intact.
- My Accounts: Tidal row (Connect via /auth/tidal?profile_id=, status, Disconnect).
Connections API extended; disconnect made generic (/<service>/disconnect).
Admin sees "managed in Settings" for every service.
Tests: per-profile token refresh writes to the profile and leaves the global
tidal_tokens untouched (the safety guarantee); connect status + disconnect;
admin/unconnected → global client. 22 endpoint tests pass.
Groundwork for admin-created, per-profile-switchable credential sets ("pills")
across auth services (Spotify/Tidal/Deezer/Qobuz/Plex/Jellyfin/Navidrome).
Strictly additive and dormant — nothing reads it at runtime yet, so zero
behaviour change for existing installs.
- core/credentials/store.py: pure service registry + payload validation +
stale-safe active-set selection (pick_active_credential falls back to None
when a selected set was deleted, so a profile never breaks).
- migration service_credentials_v1: two new tables — service_credentials
(admin-created named sets; payload Fernet-encrypted at rest) and
profile_service_credentials (each profile's selected set per service).
- MusicDatabase CRUD: create/update/delete/list/get_service_credential
(list never returns the payload; get decrypts for the resolver), plus
set/get_profile_service_credential and resolve_profile_service_credential
(returns the profile's active payload or None → caller uses global default).
Tests: 12 — pure validation + stale-safe selection, and real-temp-DB storage
proving encryption round-trips, payload never lists, dup(service,label)
rejected, per-profile/per-service resolution, and delete clearing dangling
selections to a clean fallback. 95 migration/DB tests still pass.
Boulder: the live display was a cramped ~600px box showing a fraction of the
data the scan already tracks, with no animation and no history.
Live scan deck (replaces the three-column box, full width):
- Header: pulsing live dot, "x / y artists" progress text, and two live
counter chips (found / added) that pop when they change.
- Animated progress bar (artist index / total) with a shimmer sweep.
- Stage: artist avatar with accent glow + name + readable phase line
("Checking album 2 of 5"), album art + album + current track.
- "Added to wishlist this run" feed: taller, bigger art, slide-in animation
that plays once per new track (feed re-renders only when it changes).
- All data was already in scan_state (current_artist_index, total_artists,
tracks_found/added_this_scan, current_phase) — just never displayed. The
legacy fullscreen-modal markup shares element ids and lacks the new ones,
so it keeps working untouched.
Scan History (persistent):
- New watchlist_scan_runs table — one row per run (status, timestamps,
artists/found/added counts) + the full track ledger JSON. Saved at scan
completion AND cancellation; idempotent on run_id; pruned to the last 100
runs. Wishlist rows erode as tracks download, so this is the durable record.
- GET /api/watchlist/scan/history (runs) + /history/<run_id>/tracks (ledger).
- New History button on the Watchlist page → modal in the origins/blocklist
house style: run cards (date, cancelled chip, artists/found/added stats)
expanding into the Added / Skipped track lists with art and badges.
Tests: save+fetch with ledger, idempotent re-save, prune keeps newest,
unknown-run empty, cancelled runs recorded. 398 watchlist/wishlist/history
tests pass; JS syntax-checked; all rendered strings escaped.
carlosjfcasero round 2 (manual-add fix didn't help — different path). His log
pinned it: the mirrored sync auto-added 'Llamando a la tierra (Serenade From
the Stars)' by M-Clan every run even though his library has the song (stored
bare). Reproduced exactly: the subtitle restates no album context, so the #808
context strip keeps it, and the length-ratio penalty in
_calculate_track_confidence crushes the pair to 0.142 (needs 0.7). Sync →
"missing" → wishlist, forever; and the cleanup uses the SAME matcher, so it
deterministically never removed it. Self-reinforcing.
Fix at the matcher seam (benefits sync, cleanup, downloads, discography alike):
core/text/title_match.strip_subtitle_qualifiers(title, other) strips a
bracketed qualifier only when it (a) isn't restated in the other title, (b)
contains no version-marker token (EN + ES: live/remix/acoustic/version/
dueto/directo/vivo/...), and (c) introduces no new digit token ('(Pt. 2)',
'(2007)' stay different releases). Wired as a third comparison variant in
_calculate_track_confidence with its own length guard. Verified against his
log's other unmatched tracks: '(Live)' 0.15, '(Dueto 2007)' 0.179,
'(Versión 1988)' 0.167 all still correctly blocked — version qualifiers keep
their meaning; the M-Clan case goes 0.142 → 1.0 in both directions.
Also: sync's check_track_exists call now passes album= (cleanup already did),
enabling the album-aware fallback for multi-artist albums during sync.
Tests: tests/test_subtitle_qualifier_match.py — the reported case verbatim
(end-to-end through check_track_exists, both directions, batched candidate
path included), EN+ES version qualifiers still blocked, numeric guard,
'#769 Dani California' and '#808 OurVinyl' guards still hold. 1396
matcher/wishlist/sync tests pass.
Tacobell444: when tracks land in an album across multiple batches (a wishlist
run, the Album Completeness job, a missed track re-downloaded later), the folder
is rebuilt from API metadata each time — so when $albumtype or $year come back
blank/different on a later batch, the folder NAME changes and the album splits,
forcing a Reorganize.
Fix: build_final_path_for_track now checks whether the album already lives in a
single folder on disk and, if so, drops the new track there instead of a freshly
templated folder. Match (chosen): exact stored Spotify album id first, then a
STRICT >=0.85 name+artist match (vs the 0.7 used elsewhere) — a wrong match here
misplaces a file. New core/library/existing_album_folder.resolve_existing_album_folder
holds the logic; always-on with template fallback.
Safety rails: only returns a folder UNDER the transfer dir (never a read-only
library/NAS mount), only when the album lives in EXACTLY ONE folder (multiple =
disc subfolders, which DatabaseTrack can't disambiguate — those defer to the
template), and any failure falls through to the template path. Added
MusicDatabase.get_album_by_spotify_album_id for the id-first lookup.
Tests: single-folder reuse, no-match, below-threshold, multi-folder defer,
outside-transfer reject, id-first, missing transfer dir, no-files-on-disk.
8 tests; 1556 path/import/download tests pass (only the known soundcloud
failures remain).
Part 1 stopped existing full dates being destroyed; this adds first-class support
for full release dates so they can be set + persisted instead of truncated to a
year at the DB layer.
- Schema: new nullable `release_date TEXT` on the albums table (idempotent
ALTER-ADD-COLUMN repair on startup + the live CREATE). NULL = year-only, every
reader falls back to albums.year, so it ships safe/dormant.
- Tag writer: write_tags_to_file + build_tag_diff prefer db_data['release_date']
(the full date) over the year int; _date_to_write writes the full date. When
there's no release_date it's exactly Part-1 behavior (year, preserving an
equally-specific existing file date).
- Retag read path: SELECT al.release_date in the tag-preview/write queries and
thread it into _build_library_tag_db_data.
- Manual edit: release_date added to ALBUM_EDITABLE_FIELDS + a "Release Date"
field (YYYY-MM-DD, validated client-side) in the album editor; the artist-album
query returns it so existing values show. User-set dates are authoritative.
- Enrichment: Spotify + iTunes workers store the source's full release_date
(YYYY-MM / YYYY-MM-DD) when present, only when empty — never clobbering a
manual value.
Tests: writer uses release_date over year + overrides an existing file date;
falls back to year when absent; diff compares the full date. Migration verified
idempotent + enrichment no-clobber. 1435 tag/retag/db/library tests pass.
A Library Maintenance job that cleans up downloads tracked by Download Origins
once they pass a per-origin retention window — findings by default, opt-in
auto-delete.
A download is only ever proposed for deletion when ALL hold: older than its
origin's retention, NOT still in an actively-mirrored playlist / watched
artist, and played fewer than the keep-threshold (default 2 → "played more
than once is kept"). Only touches downloads recorded from the Download Origins
feature forward — never pre-existing or manual library.
- core/library/expired_cleanup.py: pure decision core (retention_cutoff,
is_expired, select_expired) — no DB/clock, fully tested. play_count is the
reliable listen signal (last_played is often unpopulated, so recency isn't
used).
- ExpiredDownloadCleanerJob: gathers facts (play_count via a new
get_origin_cleanup_candidates join; active-mirror via get_mirrored_playlists;
watch via get_watchlist_artists) and either creates 'expired_download'
findings or, with auto_delete on, deletes in-scan. Default OFF, both
retentions default 'off'. Settings auto-render in the Library Maintenance
panel (same as Cover Art / Lyrics / Re-tag).
- delete_origin_download(): shared delete (resolve path → remove file → drop
track row → drop history row); a file that won't delete keeps its row +
reports. Used by auto mode AND the _fix_expired_download apply handler.
- Frontend: type/action ('Delete')/result labels + finding detail render.
Tests: 9 on the pure brain (windows, off, per-origin, protected, play-count
threshold, bad age) + 7 on the job (no-op when off, findings, mirror/watch
protection, auto-delete, delete helper missing/real file). 185 repair/origin
tests pass.
Phase 1 guarded the wishlist; Phase 2a closes the other auto-acquisition path.
Playlist sync, album download, and discography backfill all flow through
run_full_missing_tracks_process, which queues missing tracks at one point —
right where the explicit-content filter already drops tracks. The blocklist
filter slots in beside it: each missing track is checked and a banned
artist/album/track is dropped before queueing (logged with a count), so a
blocked item can't slip in via these flows.
Same brain as Phase 1: the wishlist guard's matcher is generalized to
db.blocklist_reason_for_track(profile_id, track_data, source=None) — the new
`source` param lets the queue path supply the batch source, since an analysis
track dict may not carry a 'provider' field (artists still match by name
fallback regardless). One method, two callers (wishlist + queue), one cascade.
Manual single-track downloads (/api/download, candidate picker, redownload)
are deliberately NOT gated here — that's Phase 2b, pending a block-vs-warn-vs-
override policy decision.
Tests: source-fallback isolation (album id-only proves source drives the ID
match; artist name still matches sourceless), and a queue-filter simulation
mirroring master.py. 35 blocklist tests pass (the only failures in the
download family are the pre-existing soundcloud /app ones).
A proper artist/album/track blacklist (distinct from download_blacklist, which
stays untouched). ID-keyed across metadata sources so a ban survives a source
switch; profile-scoped; cascade artist→album→track.
- core/blocklist/matching.py — pure decision core (no I/O): build an index from
rows, candidate_block_reason() walks track→album→artist. Same-source ID match
is primary; artist NAME is a fallback (covers the ID-backfill window);
albums/tracks are ID-only (common titles like "Greatest Hits" must not
false-positive across artists). Source-isolated so a numeric Deezer id can't
collide with a numeric iTunes id of a different entity.
- DB: new `blocklist` table (profile_id, entity_type, name, 4 source-id cols,
match_status) + CRUD, match-row fetch, backfill-pending query, id-backfill
update (COALESCE — fills NULLs only).
- Guard: _wishlist_blocklist_reason at the top of add_to_wishlist — every
auto-acquisition path funnels through it, so one check covers watchlist,
discography backfill, repair, manual add. Fails OPEN (a guard error never
blocks a legitimate add).
- Discovery unified IN: legacy discovery_artist_blacklist is migrated into the
blocklist on upgrade (replicated to every profile so no global ban silently
stops working; idempotent; legacy table kept for rollback). Discovery reads
(hero + personalized-playlist SQL) now union the blocklist, so a new-modal
ban filters discovery too.
Tests: 13 on the pure matcher (cascade, id-vs-name rules, source isolation,
precedence) + 10 on the DB/guard (CRUD, profile isolation, dedup, backfill,
end-to-end wishlist refusal + cascade + the discovery migration upgrade path).
50 blocklist/personalized tests pass.
37725457 fixed _match_to_itunes to use the real iTunes client and flagged
the cross-source corruption as a possibility. Boulder's live DB proves it
happened: 6 of his 9 watchlist "iTunes" ids EQUAL the artist's Deezer id
(Taylor Swift's "iTunes" id was her Deezer id 12246; the real one is
159260351) — written back when the misnamed MetadataService.itunes slot
held a DeezerClient. The June-4 batch (Green Day, SOAD, Vulfpeck, ...) got
NULL instead because the slot now holds the Spotify primary.
The fix alone can't heal those rows: the backfill only fills EMPTY ids, so
a wrong non-empty id is permanent. New migration clears itunes_artist_id
where it equals deezer_artist_id (the corruption signature — distinct id
spaces, so a legitimate equal pair is effectively impossible, and the worst
case is a NULL that re-matches correctly on the next scan). Idempotent by
construction; similar_artists checked clean (its backfill always used the
registry correctly).
Tests: corrupted row cleared / legit + no-deezer rows kept / idempotent —
via a real re-init with the per-process init memo cleared (an app restart).
carlosjfcasero: 'Champagne Supernova (OurVinyl Sessions)' is in the library
but the artist page shows it unowned and wishlist cleanup never removes it.
Measured with the real catalogs: Deezer/iTunes title the TRACK with the
qualifier while the library track is bare (the qualifier lives in the album
title) — and _calculate_track_confidence crushed that pair to ~0.17: the
"clean" titles keep parenthetical words, so the length-ratio penalty treats
'Champagne Supernova' vs 'Champagne Supernova (OurVinyl Sessions)' as
different songs. (Also confirmed: the OurVinyl release is absent from
Deezer's discography for the artist, so the standard page's 25-release list
not showing it is the source catalog, not a bug.)
Fix 1 — core.text.title_match.strip_redundant_context_qualifiers: a
parenthetical qualifier whose text appears (word-bounded) in the db track's
ALBUM title — or in the other title — restates release context and is
stripped for a comparison variant scored with its own length guard. Genuine
version markers keep their penalty: '(Live)' on a studio album appears in no
context and still blocks; '(Live)' on 'Live at Wembley' correctly matches —
owning the live album IS owning the live cut. Wired into
_calculate_track_confidence, so every check_track_exists consumer (wishlist
cleanup, discography dedup, repair jobs) benefits.
Fix 2 — the artist-page ownership endpoint's album gate: when album-aware
narrowing eliminates EVERY library candidate (the source's album naming just
doesn't resemble the library's — 'Jillette Johnson | OurVinyl Sessions' vs
'Champagne Supernova (OurVinyl Sessions)' ~0.5), fall back to artist-wide
title matching instead of declaring everything unowned off a failed
album-NAME comparison.
Tests: 8 — the exact reported pair end-to-end through check_track_exists,
word-boundary containment ('live' in 'alive' doesn't count), version-marker
safety both ways, and prefix songs still blocked. 1125 matching/wishlist/
library tests pass.
User ask: "a modal that lists the tracks downloaded via watchlist" — extended,
as discussed, to playlists too. One modal, two tabs, opened from the Watchlist
page (watchlist tab preselected) and the Sync page (playlists tab) — same
shared-modal-different-entry-points UX as the rest of the app.
The data: library_history recorded which SERVICE a file came from but never
what TRIGGERED it. New origin/origin_context columns (migration + index) are
written once at the import chokepoint via core/downloads/origin.py, a pure
tested deriver that reads, in priority: an explicit _dl_origin stamp (set at
batch-task creation for direct playlist batches, where the playlist context
otherwise only survived in folder mode), the wishlist provenance already
riding in track_info.source_info (watchlist_artist_name / playlist_name —
watchlist_scanner has stamped these for ages), and the folder-mode playlist
thread. Manual downloads stay unclassified by design. History starts from
now — provenance can't be conjured retroactively.
API: GET /api/download-origins?origin=watchlist|playlist (paged) and POST
/api/download-origins/delete — deletes the file on disk (resolved through the
shared container/host path resolver), the matching library track row, and the
history entries; a file that refuses deletion keeps its row and reports the
error instead of lying.
UI: webui/static/origin-history.js — tabbed modal in the revamp design
language (accent light-edge, pill tabs, entry rows reusing the
library-history-entry components), per-row delete + select-all bulk delete
with honest result toasts, empty/loading states, per-tab totals.
Tests: 8 — deriver priority/shapes (incl. the exact watchlist_scanner
source_info shape and JSON-string survival), origin filtering + counts,
row fetch/delete isolation between origins, delete-track-by-path.
Users manually match an album to the regular edition, but enrichment/
repair keeps treating it as the deluxe (missing songs, renumbered tracks).
Root cause: an album has TWO identities — the enrichment match
(spotify_album_id, which manual-match sets and the worker already honors)
and a SEPARATE canonical version pin (canonical_album_id, added by #777).
The canonical pin is what track-number repair / reorganize / missing-track
detection actually read, and library_manual_match never wrote it — so it
was resolved independently and landed on the deluxe edition.
(So #777 did NOT solve #758: it added canonical pinning, but manual
matches didn't write the pin.)
Fix: a manual ALBUM match on a canonical-recognised source now also pins
AND locks the canonical version to the chosen release:
- new canonical_locked column (same migration pattern as the other
canonical cols).
- set_album_canonical(..., locked=False) gains an atomic WHERE-clause
guard: an auto write can't overwrite a locked pin; a manual write
(locked=True) always wins. get_album_canonical exposes `locked`.
- library_manual_match pins canonical for album matches via the pure
should_pin_manual_canonical(entity_type, source).
The auto resolve job already skips already-pinned albums, so the lock is
protected on two fronts; the new guard also covers any future
re-resolution. A new manual match still overrides.
18 tests: the pure gate (+ a sync-invariant test vs _ALBUM_ID_COLUMNS)
and the DB lock seam (auto can't clobber a manual lock; manual overrides;
auto-over-auto still works). Additive — locked defaults False, so the
auto path is unchanged unless a manual lock exists. Full suite clean.
Extends the manual "Import IDs from File Tags" backfill so newly-scanned
files get their embedded provider IDs pulled into the DB automatically —
no button press needed to keep up with new music.
How it works:
- insert_or_update_media_track now returns 'inserted' / 'updated' / False
(truthy-compatible; existing `if track_success` callers unaffected) so
the scan worker can tell a genuinely new row from an update.
- DatabaseUpdateWorker collects the ids it newly INSERTED this run
(self._new_track_ids) across all insert paths (Plex/Jellyfin/deep).
- After run()/run_deep_scan(), web_server calls _reconcile_after_scan(),
which gap-fills embedded IDs for just those new tracks. Runs as a
post-scan pass (the scan loop itself is untouched/fast — the media
server API never exposes these custom IDs, so the file must be read
once regardless; batching at the end keeps it out of the hot loop and
best-effort so it can never abort a scan). A progress phase ("Reading
file tags for N new tracks…") surfaces the full-refresh tail.
Shared engine:
- New reconcile_library() in core does the paging + lazy parent-map
loading (only loads albums/artists actually referenced — cheap when
scoped to a few new tracks) + per-page commits. BOTH the manual button
and the scan hook call it, so there's one tested orchestration, no
duplication. The backfill job was refactored onto it.
Same hardened safety: gap-fill only, atomically guarded against
overwrite, schema-introspected, idempotent. Scoped to new arrivals for
incremental/deep; full refresh re-inserts everything as new (recovering
the IDs a full-refresh wipe destroys).
+10 reconcile tests (reconcile_library scope/idempotency/progress/stop +
the engine). Full suite clean (only pre-existing soundcloud /app env
failures remain).
Ships the source-id cleanup to all users: a marker-gated one-time migration in
MusicDatabase init clears any source id (deezer/spotify/itunes/musicbrainz/
discogs/audiodb/qobuz/tidal) shared across differently-named artists — the
enrichment-corruption signature. Same-name cross-server duplicates are left
untouched (DISTINCT-name check). Cleared rows re-derive correct ids on the next
enrichment pass; the now name-guarded workers won't re-corrupt.
Runs once (CREATE TABLE _source_id_dedupe_v1 marker), idempotent, per-column
try/except so a missing column can't abort it. Test forces a re-run and asserts
corruption is cleared while a legit same-name dup survives.
