88da265ef4
Measured during a live album download: ~4m15s per track in post-processing (normal is ~20s), with the time vanishing silently inside embed_source_ids — up to 5 MusicBrainz calls per track crawling against a degraded musicbrainz.org while the MB enrichment worker kept eating the same ~1 req/s per-IP budget. Only Spotify/Last.fm/Genius were in the yield set; MusicBrainz, Deezer, iTunes, Discogs etc. kept grinding through downloads. Policy (new core/enrichment/yield_policy, tested): - downloads active -> ALL enrichment workers yield (post-processing touches every metadata source). listening-stats (local-only) and repair (user-scheduled) intentionally keep running. - discovery active -> the API-contention five yield (spotify/itunes/deezer/ discogs/hydrabase) — discovery never paused anything before, despite the pause helper literally defaulting to label='discovery'. - user overrides and user-paused bookkeeping keep their existing semantics; the dashboard yield_reason label now says WHICH foreground work caused it. Observability (the 4-minute silence can never come back): - every source lookup is timed; >2s logs a warning NAMING the source and duration (core/metadata/source.py _call_source_lookup) - the pipeline always logs "Metadata enhancement took X.Xs" per track 7 policy tests (incl. the motivating case: MB yields to downloads, keeps running during discovery); 277 pipeline/enrichment tests pass.
57 lines
2.4 KiB
Python
57 lines
2.4 KiB
Python
"""Enrichment-worker yield policy: who pauses while the user's foreground
|
|
work is running.
|
|
|
|
Background enrichment workers share external API budgets with the foreground
|
|
pipelines — most painfully MusicBrainz (~1 req/s per IP), where a worker
|
|
grinding through the library can starve the import pipeline's per-track
|
|
lookups into multi-minute crawls (measured: ~4m15s/track vs the normal ~20s).
|
|
|
|
Policy (set with Boulder, 2026-06-06):
|
|
- downloads active -> EVERYTHING yields (post-processing touches every
|
|
metadata source: MusicBrainz, Spotify, iTunes,
|
|
Deezer, Discogs, Last.fm, Genius, ...)
|
|
- discovery active -> the API-contention five yield (discovery hammers
|
|
the track-matching sources only)
|
|
Workers the user explicitly resumed mid-yield are honored upstream (the
|
|
override set lives in web_server's loop, as does the user-paused bookkeeping).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Optional
|
|
|
|
# Everything that yields during active downloads. listening-stats (talks only
|
|
# to the local media server) and repair (user-scheduled job runner, not a
|
|
# background API drip) intentionally keep running.
|
|
ALL_YIELD_WORKERS = (
|
|
'musicbrainz', 'audiodb', 'discogs', 'deezer',
|
|
'spotify-enrichment', 'itunes-enrichment', 'lastfm-enrichment',
|
|
'genius-enrichment', 'tidal-enrichment', 'qobuz-enrichment',
|
|
'amazon-enrichment', 'similar_artists', 'hydrabase', 'soulid',
|
|
)
|
|
|
|
# The sources discovery contends with (track matching APIs).
|
|
API_CONTENTION_WORKERS = frozenset({
|
|
'spotify-enrichment', 'itunes-enrichment', 'deezer', 'discogs', 'hydrabase',
|
|
})
|
|
|
|
# Discovery state phases that mean "nothing running" (idle or terminal).
|
|
_INACTIVE_PHASES = frozenset({'', 'idle', 'discovered', 'error', 'failed', 'cancelled'})
|
|
|
|
|
|
def worker_yield_reason(name: str, downloading: bool, discovering: bool) -> Optional[str]:
|
|
"""Why ``name`` should be paused right now, or None to run.
|
|
Downloads outrank discovery so the label reflects the stronger cause."""
|
|
if name not in ALL_YIELD_WORKERS:
|
|
return None
|
|
if downloading:
|
|
return 'downloads'
|
|
if discovering and name in API_CONTENTION_WORKERS:
|
|
return 'discovery'
|
|
return None
|
|
|
|
|
|
def discovery_state_active(state: dict) -> bool:
|
|
"""True when a per-playlist discovery state dict represents live work."""
|
|
phase = str((state or {}).get('phase', '') or '').lower()
|
|
return phase not in _INACTIVE_PHASES
|