a60546929e
Reported by sassmastawillis: the Album Completeness maintenance job
scans 3127 albums in 0.1 seconds and reports 0 findings — for every
user, regardless of whether their library is actually complete.
Restoring an older DB surfaced 7 correct findings, so the code logic
works; the DB state is what's making everything look complete.
Root cause: `albums.track_count` is only ever written by server-sync
paths — Plex's `leafCount`/`childCount` and SoulSync standalone's
`len(tracks)`. It's the OBSERVED count of tracks SoulSync has indexed,
which is always exactly what `COUNT(tracks)` returns for that album.
The completeness job treated it as the EXPECTED total and compared it
against the observed count. They're equal by construction, so
`actual >= expected` is always true: skip, 0.1s scan, 0 findings.
Fix: new `api_track_count INTEGER` column on `albums`, written only by
metadata-source code paths. Populated in two places so the scan is
fast and the fallback is robust.
1. Enrichment workers — shared helper `set_album_api_track_count`
in `core/worker_utils.py`. Called by each worker's existing
`_update_album` method alongside its other album-column UPDATEs:
- spotify_worker: `album_obj.total_tracks` from the Spotify Album
dataclass (already in hand, zero new API calls)
- itunes_worker: same, from the iTunes Album dataclass
- deezer_worker: `nb_tracks` from full_data, falling back to
search_data when the full lookup didn't run
- discogs_worker: count of tracklist rows where `type_=='track'`
(Discogs tracklists interleave heading and index rows that
shouldn't count as songs)
Helper skips the write on zero/None/negative/non-numeric inputs
so a source lacking track info can't clobber a good value a
different source already wrote. Caller owns the transaction —
helper just queues an UPDATE on the caller's cursor without
committing, so it batches cleanly with each worker's existing
multi-UPDATE pattern.
Hydrabase worker deliberately not touched — it's a P2P mirror
that doesn't write album metadata to the local DB. Hydrabase-
primary users hit the fallback path below.
2. Album Completeness repair job — new `al.api_track_count` column
in the SELECT, read first in the scan loop. On miss (album never
enriched, or enrichment workers haven't run yet on a fresh
install), falls through to the existing `_get_expected_total()`
API lookup and persists the result via the same shared helper
(wrapped in connection/commit management since the repair job
runs outside a worker's batched transaction).
Also removed `al.track_count` from the scan's SELECT — now unused
since the observed count was the whole source of this bug, and
leaving a dead SELECT would invite a future engineer to re-introduce
the same comparison.
Help text on the job card was reworded so it honestly describes
current behavior ("counts cached during normal enrichment are used
when available; otherwise the job queries a metadata source
directly") rather than the old "active provider first, then others
as fallback" phrasing, which doesn't match how the cache actually
fills — any enrichment worker that runs can populate it, and the
last writer wins. Document-only follow-up if this edge case ever
bites in practice: add a `api_track_count_source` column so the
scan can prefer the configured primary source's count over others
(e.g. deluxe vs. standard edition mismatches). Not worth the
complexity today.
For existing users, the first completeness scan after upgrade is
fast to the extent their library is already enriched: the workers
already ran and populated `api_track_count` on their normal schedule.
For brand-new installs, the scan's fallback path handles the cold
start — slower, but correct, and subsequent scans are fast.
Does NOT affect:
- Download / post-processing / wishlist / sync code paths — none
of them read `track_count` for completeness semantics.
- Plex / Jellyfin / Navidrome / standalone sync — still write
`track_count` exactly as before; `api_track_count` is a separate
column they never touch.
- Other repair jobs.
- Any UI path — same finding schema, just correct counts now.
Files:
- database/music_database.py — idempotent migration adding
`api_track_count INTEGER DEFAULT NULL` to the existing album-column
check block.
- core/worker_utils.py — new `set_album_api_track_count` helper with
the documented skip-on-bad-input contract.
- core/spotify_worker.py, itunes_worker.py, deezer_worker.py,
discogs_worker.py — one-liner call from each `_update_album`.
- core/repair_jobs/album_completeness.py — scan uses the cache;
fallback path persists API-lookup results via the shared helper;
help text updated to match actual behavior.
- tests/test_worker_utils_album_track_count.py — 9 tests covering
the helper's write/skip contract + no-commit invariant.
- tests/test_album_completeness_job.py — 2 tests for the repair
job's fallback-path wrapper.
- webui/static/helper.js — WHATS_NEW entry.
Credit: sassmastawillis spotted the bug; the "restored older DB
finds 7 albums" signal pinpointed DB state over code logic and
made the diagnosis tractable.
116 lines
3.9 KiB
Python
116 lines
3.9 KiB
Python
"""Tests for `worker_utils.set_album_api_track_count` — the shared helper
|
|
enrichment workers call to cache authoritative track counts."""
|
|
|
|
from core.worker_utils import set_album_api_track_count
|
|
|
|
|
|
class _RecordingCursor:
|
|
"""Minimal cursor stand-in that captures execute() calls."""
|
|
|
|
def __init__(self):
|
|
self.calls = []
|
|
|
|
def execute(self, query, params=None):
|
|
self.calls.append((query, params))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Happy-path writes
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_writes_positive_int_count():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-1", 12)
|
|
assert len(cursor.calls) == 1
|
|
query, params = cursor.calls[0]
|
|
assert "UPDATE albums SET api_track_count = ?" in query
|
|
assert "WHERE id = ?" in query
|
|
assert params == (12, "album-1")
|
|
|
|
|
|
def test_coerces_numeric_string_to_int():
|
|
"""Deezer / raw API dicts often have track counts as strings."""
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-2", "14")
|
|
assert cursor.calls[0][1] == (14, "album-2")
|
|
|
|
|
|
def test_writes_one_for_single_track_album():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-single", 1)
|
|
assert cursor.calls[0][1] == (1, "album-single")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Skip-write cases (don't overwrite good values with bad ones)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_skips_write_when_count_is_zero():
|
|
"""A source that doesn't report track counts must not clobber a value
|
|
written by another source."""
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-x", 0)
|
|
assert cursor.calls == []
|
|
|
|
|
|
def test_skips_write_when_count_is_none():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-x", None)
|
|
assert cursor.calls == []
|
|
|
|
|
|
def test_skips_write_when_count_is_negative():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-x", -1)
|
|
assert cursor.calls == []
|
|
|
|
|
|
def test_skips_write_on_non_numeric_string():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-x", "not a number")
|
|
assert cursor.calls == []
|
|
|
|
|
|
def test_skips_write_on_non_numeric_object():
|
|
cursor = _RecordingCursor()
|
|
set_album_api_track_count(cursor, "album-x", object())
|
|
assert cursor.calls == []
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Does not commit — caller owns the transaction
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_helper_does_not_commit():
|
|
"""Workers batch multiple UPDATEs into one transaction. The helper
|
|
must not call commit() or it would break that batching."""
|
|
|
|
class _StrictCursor(_RecordingCursor):
|
|
commits = 0
|
|
|
|
def commit(self): # pragma: no cover — asserts it's never called
|
|
_StrictCursor.commits += 1
|
|
|
|
cursor = _StrictCursor()
|
|
set_album_api_track_count(cursor, "album-y", 5)
|
|
assert _StrictCursor.commits == 0
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Error isolation — a cursor.execute failure must not poison the worker's
|
|
# other UPDATEs in the same transaction
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def test_swallows_cursor_execute_errors():
|
|
"""If the column doesn't exist yet (e.g., migration hasn't run) or
|
|
the DB is otherwise unhappy, the helper must not propagate the error.
|
|
Otherwise the worker's other UPDATEs (spotify_album_id, thumb_url,
|
|
etc.) batched in the same transaction would roll back."""
|
|
|
|
class _BrokenCursor:
|
|
def execute(self, query, params=None):
|
|
raise RuntimeError("no such column: api_track_count")
|
|
|
|
cursor = _BrokenCursor()
|
|
# Should not raise.
|
|
set_album_api_track_count(cursor, "album-z", 10)
|