Boulder: "Taylor Swift shows only 8 albums, nothing before 2022, no singles,
no EPs" — for every artist (actually: every WATCHLIST artist). Traced live:
get_artist_albums caches its result under an UNQUALIFIED key (no limit/page
info), and the watchlist's new-release probe (limit=5, max_pages=1 — the
April "reduce watchlist API calls ~90%" optimization) stored its truncated
single page in that same slot. The artist detail page reads the cache first,
so a watchlisted artist's page showed only the newest handful of releases —
newest-first, hence "nothing before 2022" — re-poisoned on every scan, with a
30-day TTL. When the source-priority fetch comes back tiny, the page's
fallback path quietly serves it, so the symptom looked like a discography
filter bug. Not related to the #808 matching change (that is a pure max(),
provably additive).
Three pieces:
- get_artist_albums tracks whether the fetch stopped while more pages
existed (truncated) and only caches COMPLETE discographies. Individual
albums keep their opportunistic caching — they're complete entities
regardless of pagination. A small real discography that fits one page
stays cacheable even under max_pages=1.
- MetadataCache.purge_artist_album_lists(): delete the already-poisoned
album-list entries (TTL would have kept them for weeks); lists rebuild
lazily on the next artist-page visit.
- one-time startup purge in web_server, config-guarded
(maintenance.album_cache_purge_v1), mirroring the startup-repair pattern.
Tests: truncated probe never stores the list (but still returns its page),
complete multi-page fetch caches, and a genuinely-small one-page discography
under max_pages=1 still caches. 1087 spotify/cache/watchlist/artist tests
pass.
Investigation (not assumption): the cache's TTL eviction + junk cleanup ARE
correct and DO run automatically every 6h (CacheEvictorJob, auto_fix=True).
The real gap is there's NO SIZE CEILING — TTL-only eviction means 'how big can
it get' = 'however much you fetch within the 30-day window', so heavy
discovery/enrichment legitimately grew metadata_cache_entities to ~1.8M rows /
7.6 GB, bloating the main DB (a factor in the corruption incident).
Fix — add a bounded LRU cap:
- entities_to_evict_for_capacity(total, max_rows): pure decision fn (cap<=0
disables), unit-testable like core.db_integrity.prune_backups.
- MetadataCache.evict_over_capacity(): deletes the least-recently-ACCESSED rows
(uses the already-stored last_accessed_at; NULL = never-touched = evicted
first) down to the ceiling. Default 250k rows, tunable.
- Wired as Phase 5 of CacheEvictorJob — runs LAST, after TTL/junk/orphan/null
cleanup, so it only trims a still-oversized HEALTHY cache.
Verified safe to bound/wipe: audited every cache reader (get_entity/
get_entities_batch/get_search_results/get_entity_detail/browse) — all degrade
to None/[]/empty on miss, treated as 'go fetch'. Nothing depends on a row
existing, so eviction can't break callers.
Tests: tests/metadata/test_cache_capacity_eviction.py (8) — pure-fn coverage +
real temp-DB proof that it drops the LRU rows specifically (not arbitrary) and
NULL-access rows go first. 18 adjacent cache tests still green; ruff clean.
Follow-ups (separate phases, scoped): (2) move the cache to its own bounded
metadata_cache.sqlite3 (no JOINs to library tables — confirmed clean to split;
invalidate-and-rebuild rather than migrate the 7.6GB), (3) kill the
raw_json + 22-extracted-column double storage.
Keep full refresh moving when post-clear VACUUM hits a transient disk I/O error, and retry clear_server_data once when the clear step itself sees the same transient SQLite failure.
Retry metadata cache maintenance writes once on transient disk I/O errors so first-attempt cache jobs do not fail when an immediate retry would succeed.
Tests cover best-effort VACUUM, clear retry behavior, and cache maintenance retry behavior.
