a60546929e
Reported by sassmastawillis: the Album Completeness maintenance job
scans 3127 albums in 0.1 seconds and reports 0 findings — for every
user, regardless of whether their library is actually complete.
Restoring an older DB surfaced 7 correct findings, so the code logic
works; the DB state is what's making everything look complete.
Root cause: `albums.track_count` is only ever written by server-sync
paths — Plex's `leafCount`/`childCount` and SoulSync standalone's
`len(tracks)`. It's the OBSERVED count of tracks SoulSync has indexed,
which is always exactly what `COUNT(tracks)` returns for that album.
The completeness job treated it as the EXPECTED total and compared it
against the observed count. They're equal by construction, so
`actual >= expected` is always true: skip, 0.1s scan, 0 findings.
Fix: new `api_track_count INTEGER` column on `albums`, written only by
metadata-source code paths. Populated in two places so the scan is
fast and the fallback is robust.
1. Enrichment workers — shared helper `set_album_api_track_count`
in `core/worker_utils.py`. Called by each worker's existing
`_update_album` method alongside its other album-column UPDATEs:
- spotify_worker: `album_obj.total_tracks` from the Spotify Album
dataclass (already in hand, zero new API calls)
- itunes_worker: same, from the iTunes Album dataclass
- deezer_worker: `nb_tracks` from full_data, falling back to
search_data when the full lookup didn't run
- discogs_worker: count of tracklist rows where `type_=='track'`
(Discogs tracklists interleave heading and index rows that
shouldn't count as songs)
Helper skips the write on zero/None/negative/non-numeric inputs
so a source lacking track info can't clobber a good value a
different source already wrote. Caller owns the transaction —
helper just queues an UPDATE on the caller's cursor without
committing, so it batches cleanly with each worker's existing
multi-UPDATE pattern.
Hydrabase worker deliberately not touched — it's a P2P mirror
that doesn't write album metadata to the local DB. Hydrabase-
primary users hit the fallback path below.
2. Album Completeness repair job — new `al.api_track_count` column
in the SELECT, read first in the scan loop. On miss (album never
enriched, or enrichment workers haven't run yet on a fresh
install), falls through to the existing `_get_expected_total()`
API lookup and persists the result via the same shared helper
(wrapped in connection/commit management since the repair job
runs outside a worker's batched transaction).
Also removed `al.track_count` from the scan's SELECT — now unused
since the observed count was the whole source of this bug, and
leaving a dead SELECT would invite a future engineer to re-introduce
the same comparison.
Help text on the job card was reworded so it honestly describes
current behavior ("counts cached during normal enrichment are used
when available; otherwise the job queries a metadata source
directly") rather than the old "active provider first, then others
as fallback" phrasing, which doesn't match how the cache actually
fills — any enrichment worker that runs can populate it, and the
last writer wins. Document-only follow-up if this edge case ever
bites in practice: add a `api_track_count_source` column so the
scan can prefer the configured primary source's count over others
(e.g. deluxe vs. standard edition mismatches). Not worth the
complexity today.
For existing users, the first completeness scan after upgrade is
fast to the extent their library is already enriched: the workers
already ran and populated `api_track_count` on their normal schedule.
For brand-new installs, the scan's fallback path handles the cold
start — slower, but correct, and subsequent scans are fast.
Does NOT affect:
- Download / post-processing / wishlist / sync code paths — none
of them read `track_count` for completeness semantics.
- Plex / Jellyfin / Navidrome / standalone sync — still write
`track_count` exactly as before; `api_track_count` is a separate
column they never touch.
- Other repair jobs.
- Any UI path — same finding schema, just correct counts now.
Files:
- database/music_database.py — idempotent migration adding
`api_track_count INTEGER DEFAULT NULL` to the existing album-column
check block.
- core/worker_utils.py — new `set_album_api_track_count` helper with
the documented skip-on-bad-input contract.
- core/spotify_worker.py, itunes_worker.py, deezer_worker.py,
discogs_worker.py — one-liner call from each `_update_album`.
- core/repair_jobs/album_completeness.py — scan uses the cache;
fallback path persists API-lookup results via the shared helper;
help text updated to match actual behavior.
- tests/test_worker_utils_album_track_count.py — 9 tests covering
the helper's write/skip contract + no-commit invariant.
- tests/test_album_completeness_job.py — 2 tests for the repair
job's fallback-path wrapper.
- webui/static/helper.js — WHATS_NEW entry.
Credit: sassmastawillis spotted the bug; the "restored older DB
finds 7 albums" signal pinpointed DB state over code logic and
made the diagnosis tractable.
69 lines
2.8 KiB
Python
69 lines
2.8 KiB
Python
"""Shared helpers for background workers."""
|
|
|
|
import logging
|
|
import threading
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
def interruptible_sleep(stop_event: threading.Event, seconds: float, step: float = 0.5) -> bool:
|
|
"""Sleep in chunks so shutdown can interrupt long waits."""
|
|
if seconds <= 0:
|
|
return stop_event.is_set()
|
|
|
|
remaining = float(seconds)
|
|
while remaining > 0 and not stop_event.is_set():
|
|
wait_for = min(step, remaining)
|
|
if stop_event.wait(wait_for):
|
|
break
|
|
remaining -= wait_for
|
|
return stop_event.is_set()
|
|
|
|
|
|
def set_album_api_track_count(cursor, album_id, count):
|
|
"""Cache an album's authoritative track count from a metadata source.
|
|
|
|
Called by enrichment workers (Spotify / iTunes / Deezer / Discogs) after
|
|
they fetch album metadata. The count is the EXPECTED total tracks
|
|
according to that source — distinct from `albums.track_count`, which
|
|
server syncs (Plex `leafCount`, SoulSync standalone `len(tracks)`)
|
|
populate with the OBSERVED count SoulSync already has indexed. The
|
|
Album Completeness repair job reads `albums.api_track_count` as the
|
|
expected total; populating it here during enrichment avoids a second
|
|
round of API calls during the repair scan.
|
|
|
|
Skips the write when the source didn't supply a positive numeric count
|
|
(None, 0, negative, or non-numeric) — that way a source lacking track
|
|
info doesn't overwrite a good value another source already wrote. If
|
|
multiple sources report different counts (rare, usually deluxe vs.
|
|
standard edition), last-write-wins across enrichment cycles; that's
|
|
fine since any metadata-source count is strictly better than the
|
|
observed-count fallback that the repair job used before this column
|
|
existed.
|
|
|
|
Caller owns the cursor (and its connection / transaction) — this
|
|
helper does not commit. Integrates with each worker's existing
|
|
`_update_album` method, which already batches several UPDATEs into
|
|
one transaction.
|
|
"""
|
|
try:
|
|
count = int(count or 0)
|
|
except (TypeError, ValueError):
|
|
return
|
|
if count <= 0:
|
|
return
|
|
# Swallow SQL errors — each worker batches several album UPDATEs into
|
|
# one transaction, and we don't want a failure here (e.g., the
|
|
# migration somehow hasn't run yet and the column is missing) to
|
|
# rollback the worker's other writes (spotify_album_id, thumb_url,
|
|
# etc.). The repair job's fallback path will eventually populate the
|
|
# column via its own save path once the column exists.
|
|
try:
|
|
cursor.execute(
|
|
"UPDATE albums SET api_track_count = ? WHERE id = ?",
|
|
(count, album_id),
|
|
)
|
|
except Exception as e:
|
|
logger.warning(
|
|
"Failed to cache api_track_count for album %s: %s", album_id, e
|
|
)
|