Similar Artists enrichment worker (MusicMap → match → store) for library artists
Closes the gap where similar artists only existed for WATCHLIST artists: a new
background worker populates them for the whole LIBRARY, slotting into the
existing enrichment-worker pattern (bubble + Manage Enrichment Workers modal,
status/pause/resume, matched/not_found/pending/errors).
Per source-matched library artist → get_musicmap_similar_artists(name, 25)
(the same matcher the artist-detail page uses: fetches MusicMap names, matches
each to the user's source chain — primary + active fallbacks — returns only
matched artists) → store via add_or_update_similar_artist keyed by the artist's
metadata source id, the SAME key the watchlist scanner + artist map use, so the
two cooperate (idempotent upsert + retry_days window).
- core/similar_artists_worker.py: pure seams (pick_source_artist_id,
map_payload_to_store_kwargs, process_artist) + the threaded worker; skips
artists not yet source-matched; classifies not_found vs transient error
(retry after 30d).
- DB migration: similar_artists_match_status / _last_attempted on artists
(mirrors every other source worker's tracking columns).
- Registered in EnrichmentService + instantiated in web_server, DEFAULT-PAUSED
(opt-in) like Amazon — MusicMap is scraped/outage-prone + this is library-wide.
- SERVICE_ENTITY_SUPPORT['similar_artists']=('artist',) so the modal breakdown
('artists with / without similars') + Retry work; manual-match (inapplicable
to a relationship) is gated out via relationship:true.
- 10 seam tests; existing 80 enrichment tests still pass.
Note: keys under profile 1 (single-profile setups); multi-profile is future work.
This commit is contained in:
parent
c8626b1e83
commit
89e3486e84
6 changed files with 491 additions and 2 deletions
|
|
@ -32,6 +32,12 @@ SERVICE_ENTITY_SUPPORT = {
|
||||||
'tidal': ('artist', 'album', 'track'),
|
'tidal': ('artist', 'album', 'track'),
|
||||||
'qobuz': ('artist', 'album', 'track'),
|
'qobuz': ('artist', 'album', 'track'),
|
||||||
'amazon': ('artist', 'album', 'track'),
|
'amazon': ('artist', 'album', 'track'),
|
||||||
|
# Relationship enrichment (not a metadata source): the Similar Artists worker
|
||||||
|
# only operates at the artist level, and its <service>_match_status tracks
|
||||||
|
# whether MusicMap similars were fetched (not a source-id match). So the
|
||||||
|
# breakdown / unmatched list here means "artists we have / don't have
|
||||||
|
# similars for" — informative, even though there's no manual-match action.
|
||||||
|
'similar_artists': ('artist',),
|
||||||
}
|
}
|
||||||
|
|
||||||
# entity_type -> table / display-name column / image expression / optional join
|
# entity_type -> table / display-name column / image expression / optional join
|
||||||
|
|
|
||||||
296
core/similar_artists_worker.py
Normal file
296
core/similar_artists_worker.py
Normal file
|
|
@ -0,0 +1,296 @@
|
||||||
|
"""Background worker that fills the ``similar_artists`` table for LIBRARY artists.
|
||||||
|
|
||||||
|
The watchlist scanner only populates similar artists for *watchlist* artists, so
|
||||||
|
the artist map / discover surfaces are rich for watchlisted artists and sparse
|
||||||
|
for the rest of the library. This worker closes that gap: for every
|
||||||
|
source-matched library artist it asks MusicMap for ~25 similar artists, matches
|
||||||
|
each one to the user's metadata source chain (primary + active fallbacks) via the
|
||||||
|
shared :func:`core.metadata.similar_artists.get_musicmap_similar_artists`, and
|
||||||
|
stores the matched results keyed by the library artist's **metadata source id** —
|
||||||
|
the same key the watchlist scanner and the artist map use, so the two cooperate
|
||||||
|
(idempotent upsert + a retry window keep them from double-fetching).
|
||||||
|
|
||||||
|
It plugs into the existing enrichment-worker pattern (background thread, status /
|
||||||
|
pause / resume, ``matched / not_found / pending / errors`` stats) so it shows up
|
||||||
|
as a bubble in the dashboard / Manage Enrichment Workers modal like every other
|
||||||
|
source worker.
|
||||||
|
|
||||||
|
The pure seams below (:func:`pick_source_artist_id`,
|
||||||
|
:func:`map_payload_to_store_kwargs`, :func:`process_artist`) carry the logic and
|
||||||
|
are unit-tested in isolation; the class wires them to the DB + MusicMap.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import threading
|
||||||
|
from typing import Any, Callable, Dict, Optional
|
||||||
|
|
||||||
|
from utils.logging_config import get_logger
|
||||||
|
from core.worker_utils import interruptible_sleep
|
||||||
|
|
||||||
|
logger = get_logger("similar_artists_worker")
|
||||||
|
|
||||||
|
|
||||||
|
# A matched MusicMap payload is {id, source, name, image_url, genres, popularity}.
|
||||||
|
# Map its single (id, source) onto the right add_or_update_similar_artist id
|
||||||
|
# kwarg. The table only has columns for these four providers; a match on any
|
||||||
|
# other source (e.g. discogs) is still stored, name-only.
|
||||||
|
_SOURCE_ID_FIELD = {
|
||||||
|
'spotify': 'similar_artist_spotify_id',
|
||||||
|
'itunes': 'similar_artist_itunes_id',
|
||||||
|
'deezer': 'similar_artist_deezer_id',
|
||||||
|
'musicbrainz': 'similar_artist_musicbrainz_id',
|
||||||
|
}
|
||||||
|
|
||||||
|
# Library-artist source-id columns, in the same priority the watchlist scanner
|
||||||
|
# uses to key its rows — so a library artist and (if also watchlisted) its
|
||||||
|
# watchlist row resolve to the SAME source_artist_id and don't duplicate work.
|
||||||
|
_LIBRARY_ID_COLUMNS = ('spotify_artist_id', 'itunes_artist_id', 'deezer_id', 'musicbrainz_id')
|
||||||
|
|
||||||
|
|
||||||
|
def map_payload_to_store_kwargs(payload: Dict[str, Any]) -> Dict[str, Optional[str]]:
|
||||||
|
"""Turn a matched MusicMap payload into the id kwarg for the store call."""
|
||||||
|
src = str(payload.get('source') or '').lower()
|
||||||
|
pid = str(payload.get('id') or '')
|
||||||
|
field = _SOURCE_ID_FIELD.get(src)
|
||||||
|
return {field: pid} if (field and pid) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def pick_source_artist_id(row: Dict[str, Any]) -> Optional[str]:
|
||||||
|
"""The metadata source id to key a library artist's similars by, or None if
|
||||||
|
the artist isn't matched to any metadata source yet (→ skip it)."""
|
||||||
|
for key in _LIBRARY_ID_COLUMNS:
|
||||||
|
v = row.get(key)
|
||||||
|
if v:
|
||||||
|
return str(v)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def process_artist(
|
||||||
|
source_artist_id: str,
|
||||||
|
artist_name: str,
|
||||||
|
fetch_similars: Callable[[str, int], Dict[str, Any]],
|
||||||
|
store_similar: Callable[..., bool],
|
||||||
|
limit: int = 25,
|
||||||
|
profile_id: int = 1,
|
||||||
|
) -> tuple:
|
||||||
|
"""Fetch + store similar artists for one library artist.
|
||||||
|
|
||||||
|
``fetch_similars(name, limit)`` returns the ``get_musicmap_similar_artists``
|
||||||
|
payload; ``store_similar(**kwargs)`` is ``add_or_update_similar_artist``.
|
||||||
|
Returns ``(status, stored_count)`` where status is one of:
|
||||||
|
- ``'matched'`` — stored ≥1 similar artist
|
||||||
|
- ``'not_found'`` — MusicMap had no entry / nothing matched a source
|
||||||
|
- ``'error'`` — MusicMap/source failure (transient; eligible for retry)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
result = fetch_similars(artist_name, limit) or {}
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug("MusicMap fetch raised for %s: %s", artist_name, exc)
|
||||||
|
return ('error', 0)
|
||||||
|
|
||||||
|
if not result.get('success'):
|
||||||
|
# 404/400 = genuinely no MusicMap entry → 'not_found' (don't keep retrying);
|
||||||
|
# anything else (timeout, 5xx, no providers) = transient → 'error' (retry).
|
||||||
|
code = result.get('status_code')
|
||||||
|
return ('not_found' if code in (400, 404) else 'error', 0)
|
||||||
|
|
||||||
|
sims = result.get('similar_artists') or []
|
||||||
|
if not sims:
|
||||||
|
return ('not_found', 0)
|
||||||
|
|
||||||
|
stored = 0
|
||||||
|
for rank, s in enumerate(sims, 1):
|
||||||
|
name = s.get('name')
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
kwargs = map_payload_to_store_kwargs(s)
|
||||||
|
try:
|
||||||
|
ok = store_similar(
|
||||||
|
source_artist_id=source_artist_id,
|
||||||
|
similar_artist_name=name,
|
||||||
|
similarity_rank=rank,
|
||||||
|
profile_id=profile_id,
|
||||||
|
image_url=s.get('image_url'),
|
||||||
|
genres=s.get('genres'),
|
||||||
|
popularity=s.get('popularity', 0) or 0,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
if ok:
|
||||||
|
stored += 1
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug("store similar failed for %s: %s", name, exc)
|
||||||
|
|
||||||
|
return ('matched' if stored else 'not_found', stored)
|
||||||
|
|
||||||
|
|
||||||
|
class SimilarArtistsWorker:
|
||||||
|
"""Background worker that populates similar artists for library artists."""
|
||||||
|
|
||||||
|
def __init__(self, database):
|
||||||
|
self.db = database
|
||||||
|
self.running = False
|
||||||
|
self.paused = False
|
||||||
|
self.should_stop = False
|
||||||
|
self.thread: Optional[threading.Thread] = None
|
||||||
|
self._stop_event = threading.Event()
|
||||||
|
self.current_item: Optional[str] = None
|
||||||
|
self.stats = {'matched': 0, 'not_found': 0, 'pending': 0, 'errors': 0}
|
||||||
|
self.retry_days = 30
|
||||||
|
self.limit = 25
|
||||||
|
# similar_artists rows are profile-scoped; the library is shared. v1 keys
|
||||||
|
# under the default profile (matches single-profile setups, which is the
|
||||||
|
# common case). Multi-profile per-source-chain population is future work.
|
||||||
|
self.profile_id = 1
|
||||||
|
logger.info("Similar Artists background worker initialized")
|
||||||
|
|
||||||
|
# ── lifecycle (mirrors the other enrichment workers) ──────────────────
|
||||||
|
def start(self):
|
||||||
|
if self.running:
|
||||||
|
return
|
||||||
|
self.running = True
|
||||||
|
self.should_stop = False
|
||||||
|
self._stop_event.clear()
|
||||||
|
self.thread = threading.Thread(target=self._run, daemon=True)
|
||||||
|
self.thread.start()
|
||||||
|
logger.info("Similar Artists worker started")
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
self.should_stop = True
|
||||||
|
self.running = False
|
||||||
|
self._stop_event.set()
|
||||||
|
|
||||||
|
def pause(self):
|
||||||
|
self.paused = True
|
||||||
|
|
||||||
|
def resume(self):
|
||||||
|
self.paused = False
|
||||||
|
|
||||||
|
def get_stats(self) -> Dict[str, Any]:
|
||||||
|
try:
|
||||||
|
self.stats['pending'] = self._count_pending()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
is_running = self.running and (self.thread is not None and self.thread.is_alive())
|
||||||
|
is_idle = is_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
|
||||||
|
return {
|
||||||
|
'enabled': True,
|
||||||
|
'running': is_running and not self.paused,
|
||||||
|
'paused': self.paused,
|
||||||
|
'idle': is_idle,
|
||||||
|
'current_item': self.current_item,
|
||||||
|
'stats': self.stats.copy(),
|
||||||
|
'progress': {},
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── worker loop ────────────────────────────────────────────────────────
|
||||||
|
def _run(self):
|
||||||
|
logger.info("Similar Artists worker thread started")
|
||||||
|
# Imported lazily so the worker module stays import-light for tests.
|
||||||
|
from core.metadata.similar_artists import get_musicmap_similar_artists
|
||||||
|
|
||||||
|
while not self.should_stop:
|
||||||
|
try:
|
||||||
|
if self.paused:
|
||||||
|
interruptible_sleep(self._stop_event, 1)
|
||||||
|
continue
|
||||||
|
|
||||||
|
self.current_item = None
|
||||||
|
artist = self._get_next_artist()
|
||||||
|
if not artist:
|
||||||
|
interruptible_sleep(self._stop_event, 15)
|
||||||
|
continue
|
||||||
|
|
||||||
|
sid = pick_source_artist_id(artist)
|
||||||
|
if not sid:
|
||||||
|
# Query already filters to artists with a source id; guard anyway.
|
||||||
|
self._mark(artist['id'], 'error')
|
||||||
|
continue
|
||||||
|
|
||||||
|
self.current_item = artist.get('name')
|
||||||
|
status, count = process_artist(
|
||||||
|
sid, artist['name'],
|
||||||
|
get_musicmap_similar_artists,
|
||||||
|
self.db.add_or_update_similar_artist,
|
||||||
|
limit=self.limit, profile_id=self.profile_id,
|
||||||
|
)
|
||||||
|
self._mark(artist['id'], status)
|
||||||
|
if status == 'matched':
|
||||||
|
self.stats['matched'] += 1
|
||||||
|
logger.debug("Similar artists: %s → stored %d", artist['name'], count)
|
||||||
|
elif status == 'not_found':
|
||||||
|
self.stats['not_found'] += 1
|
||||||
|
else:
|
||||||
|
self.stats['errors'] += 1
|
||||||
|
|
||||||
|
# Pace MusicMap (name search per candidate is heavy + rate-limited).
|
||||||
|
interruptible_sleep(self._stop_event, 3)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.error("Similar Artists worker loop error: %s", exc)
|
||||||
|
interruptible_sleep(self._stop_event, 5)
|
||||||
|
|
||||||
|
logger.info("Similar Artists worker thread finished")
|
||||||
|
|
||||||
|
# ── DB helpers (thin; the testable logic lives in the pure seams above) ──
|
||||||
|
def _has_source_id_clause(self) -> str:
|
||||||
|
return '(' + ' OR '.join(f"{c} IS NOT NULL AND {c} != ''" for c in _LIBRARY_ID_COLUMNS) + ')'
|
||||||
|
|
||||||
|
def _get_next_artist(self) -> Optional[Dict[str, Any]]:
|
||||||
|
try:
|
||||||
|
conn = self.db._get_connection()
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cols = "id, name, " + ", ".join(_LIBRARY_ID_COLUMNS)
|
||||||
|
have_id = self._has_source_id_clause()
|
||||||
|
# 1) Unattempted source-matched artists.
|
||||||
|
cursor.execute(f"""
|
||||||
|
SELECT {cols} FROM artists
|
||||||
|
WHERE id IS NOT NULL AND name IS NOT NULL
|
||||||
|
AND similar_artists_match_status IS NULL
|
||||||
|
AND {have_id}
|
||||||
|
ORDER BY id ASC LIMIT 1
|
||||||
|
""")
|
||||||
|
row = cursor.fetchone()
|
||||||
|
# 2) Retry transient failures (and re-check 'not_found') after retry_days.
|
||||||
|
if not row:
|
||||||
|
cursor.execute(f"""
|
||||||
|
SELECT {cols} FROM artists
|
||||||
|
WHERE id IS NOT NULL AND name IS NOT NULL
|
||||||
|
AND similar_artists_match_status IN ('error', 'not_found')
|
||||||
|
AND (similar_artists_last_attempted IS NULL
|
||||||
|
OR similar_artists_last_attempted < datetime('now', ?))
|
||||||
|
AND {have_id}
|
||||||
|
ORDER BY similar_artists_last_attempted ASC LIMIT 1
|
||||||
|
""", (f'-{self.retry_days} days',))
|
||||||
|
row = cursor.fetchone()
|
||||||
|
if not row:
|
||||||
|
return None
|
||||||
|
keys = ['id', 'name'] + list(_LIBRARY_ID_COLUMNS)
|
||||||
|
return dict(zip(keys, row))
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug("Similar Artists _get_next_artist failed: %s", exc)
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _mark(self, artist_id, status: str):
|
||||||
|
try:
|
||||||
|
conn = self.db._get_connection()
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE artists SET similar_artists_match_status = ?, "
|
||||||
|
"similar_artists_last_attempted = CURRENT_TIMESTAMP WHERE id = ?",
|
||||||
|
(status, artist_id),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug("Similar Artists _mark failed for %s: %s", artist_id, exc)
|
||||||
|
|
||||||
|
def _count_pending(self) -> int:
|
||||||
|
try:
|
||||||
|
conn = self.db._get_connection()
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute(
|
||||||
|
f"SELECT COUNT(*) FROM artists WHERE similar_artists_match_status IS NULL "
|
||||||
|
f"AND {self._has_source_id_clause()}"
|
||||||
|
)
|
||||||
|
return int(cursor.fetchone()[0] or 0)
|
||||||
|
except Exception:
|
||||||
|
return 0
|
||||||
|
|
@ -427,6 +427,9 @@ class MusicDatabase:
|
||||||
# Add Amazon artist ID column (migration)
|
# Add Amazon artist ID column (migration)
|
||||||
self._add_amazon_columns(cursor)
|
self._add_amazon_columns(cursor)
|
||||||
|
|
||||||
|
# Add Similar-Artists worker tracking columns (migration)
|
||||||
|
self._add_similar_artists_worker_columns(cursor)
|
||||||
|
|
||||||
# Backfill match_status for rows that already have an external ID but
|
# Backfill match_status for rows that already have an external ID but
|
||||||
# NULL status. Prevents enrichment workers from re-processing these
|
# NULL status. Prevents enrichment workers from re-processing these
|
||||||
# rows forever. Must run AFTER all *_match_status columns have been
|
# rows forever. Must run AFTER all *_match_status columns have been
|
||||||
|
|
@ -2417,6 +2420,28 @@ class MusicDatabase:
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error adding Discogs columns: {e}")
|
logger.error(f"Error adding Discogs columns: {e}")
|
||||||
|
|
||||||
|
def _add_similar_artists_worker_columns(self, cursor):
|
||||||
|
"""Add Similar-Artists worker tracking columns to the artists table.
|
||||||
|
|
||||||
|
Mirrors the per-source enrichment pattern: a match_status (NULL =
|
||||||
|
unattempted, then 'matched'/'not_found'/'error') + last_attempted
|
||||||
|
timestamp so the SimilarArtistsWorker can pick the next library artist to
|
||||||
|
fetch MusicMap similars for and retry transient failures after a window.
|
||||||
|
Idempotent — only adds columns that aren't already present.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
cursor.execute("PRAGMA table_info(artists)")
|
||||||
|
artists_columns = [column[1] for column in cursor.fetchall()]
|
||||||
|
|
||||||
|
if 'similar_artists_match_status' not in artists_columns:
|
||||||
|
cursor.execute("ALTER TABLE artists ADD COLUMN similar_artists_match_status TEXT")
|
||||||
|
if 'similar_artists_last_attempted' not in artists_columns:
|
||||||
|
cursor.execute("ALTER TABLE artists ADD COLUMN similar_artists_last_attempted TIMESTAMP")
|
||||||
|
|
||||||
|
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_similarartists_status ON artists (similar_artists_match_status)")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error adding similar-artists worker columns: {e}")
|
||||||
|
|
||||||
def _add_amazon_columns(self, cursor):
|
def _add_amazon_columns(self, cursor):
|
||||||
"""Add Amazon enrichment tracking columns to artists, albums, and tracks."""
|
"""Add Amazon enrichment tracking columns to artists, albums, and tracks."""
|
||||||
try:
|
try:
|
||||||
|
|
|
||||||
127
tests/test_similar_artists_worker.py
Normal file
127
tests/test_similar_artists_worker.py
Normal file
|
|
@ -0,0 +1,127 @@
|
||||||
|
"""Seam tests for the Similar-Artists enrichment worker's pure logic.
|
||||||
|
|
||||||
|
The worker fills the similar_artists table for LIBRARY artists (the watchlist
|
||||||
|
scanner only does watchlist artists). These tests exercise the import-light
|
||||||
|
seams in isolation — no DB, no MusicMap — via injected fakes:
|
||||||
|
|
||||||
|
- pick_source_artist_id → keying priority (and skip un-matched artists)
|
||||||
|
- map_payload_to_store_kwargs → MusicMap {id,source} → the right id column
|
||||||
|
- process_artist → fetch→match→store orchestration + status codes
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import core.similar_artists_worker as w
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# pick_source_artist_id — which id keys the artist's similars (must match the
|
||||||
|
# watchlist scanner's priority so both write the SAME source_artist_id).
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_pick_source_artist_id_priority():
|
||||||
|
row = {'spotify_artist_id': 'sp1', 'itunes_artist_id': 'it1', 'deezer_id': 'dz1'}
|
||||||
|
assert w.pick_source_artist_id(row) == 'sp1' # spotify wins
|
||||||
|
assert w.pick_source_artist_id({'itunes_artist_id': 'it1', 'deezer_id': 'dz1'}) == 'it1'
|
||||||
|
assert w.pick_source_artist_id({'deezer_id': 'dz1'}) == 'dz1'
|
||||||
|
assert w.pick_source_artist_id({'musicbrainz_id': 'mb1'}) == 'mb1'
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_source_artist_id_none_when_unmatched():
|
||||||
|
# Library artist not matched to any metadata source yet → skip (None).
|
||||||
|
assert w.pick_source_artist_id({'spotify_artist_id': None, 'itunes_artist_id': ''}) is None
|
||||||
|
assert w.pick_source_artist_id({}) is None
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# map_payload_to_store_kwargs — MusicMap payload {id, source} → store kwarg.
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_map_payload_each_source():
|
||||||
|
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'spotify'}) == {'similar_artist_spotify_id': 'x'}
|
||||||
|
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'itunes'}) == {'similar_artist_itunes_id': 'x'}
|
||||||
|
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'deezer'}) == {'similar_artist_deezer_id': 'x'}
|
||||||
|
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'musicbrainz'}) == {'similar_artist_musicbrainz_id': 'x'}
|
||||||
|
|
||||||
|
|
||||||
|
def test_map_payload_unknown_source_or_no_id():
|
||||||
|
# discogs has no column → name-only (empty kwargs), not a crash.
|
||||||
|
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'discogs'}) == {}
|
||||||
|
assert w.map_payload_to_store_kwargs({'source': 'spotify'}) == {} # no id
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# process_artist — fetch → store, status classification, keying.
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _capture_store():
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def store(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return True
|
||||||
|
return store, calls
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_matched_stores_with_keying():
|
||||||
|
store, calls = _capture_store()
|
||||||
|
payload = {
|
||||||
|
'success': True,
|
||||||
|
'similar_artists': [
|
||||||
|
{'name': 'B', 'id': 'sp_b', 'source': 'spotify', 'genres': ['rap'], 'popularity': 70},
|
||||||
|
{'name': 'C', 'id': 'it_c', 'source': 'itunes', 'image_url': 'http://x'},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
status, count = w.process_artist('SRC1', 'A', lambda n, l: payload, store, limit=25, profile_id=1)
|
||||||
|
assert status == 'matched' and count == 2
|
||||||
|
# All similars keyed by the SOURCE artist id we passed (not the library PK).
|
||||||
|
assert all(c['source_artist_id'] == 'SRC1' for c in calls)
|
||||||
|
assert all(c['profile_id'] == 1 for c in calls)
|
||||||
|
# Provider id mapped to the right column; rank preserved in order.
|
||||||
|
assert calls[0]['similar_artist_spotify_id'] == 'sp_b' and calls[0]['similarity_rank'] == 1
|
||||||
|
assert calls[1]['similar_artist_itunes_id'] == 'it_c' and calls[1]['similarity_rank'] == 2
|
||||||
|
assert calls[0]['genres'] == ['rap'] and calls[0]['popularity'] == 70
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_not_found_when_no_matches():
|
||||||
|
store, calls = _capture_store()
|
||||||
|
status, count = w.process_artist('S', 'A', lambda n, l: {'success': True, 'similar_artists': []}, store)
|
||||||
|
assert status == 'not_found' and count == 0 and calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_not_found_on_404():
|
||||||
|
# Genuinely no MusicMap entry — shouldn't be retried as an error.
|
||||||
|
store, _ = _capture_store()
|
||||||
|
status, _ = w.process_artist('S', 'A', lambda n, l: {'success': False, 'status_code': 404}, store)
|
||||||
|
assert status == 'not_found'
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_error_on_outage_is_retriable():
|
||||||
|
# 5xx / no providers → transient error (worker retries after retry_days).
|
||||||
|
store, _ = _capture_store()
|
||||||
|
status, _ = w.process_artist('S', 'A', lambda n, l: {'success': False, 'status_code': 502}, store)
|
||||||
|
assert status == 'error'
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_error_when_fetch_raises():
|
||||||
|
store, _ = _capture_store()
|
||||||
|
|
||||||
|
def boom(n, l):
|
||||||
|
raise RuntimeError('musicmap down')
|
||||||
|
status, count = w.process_artist('S', 'A', boom, store)
|
||||||
|
assert status == 'error' and count == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_artist_skips_unstorable_but_counts_real():
|
||||||
|
# A store() that fails for one row shouldn't abort the rest.
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def store(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return kwargs['similar_artist_name'] != 'B' # B fails to store
|
||||||
|
payload = {'success': True, 'similar_artists': [
|
||||||
|
{'name': 'B', 'id': '1', 'source': 'spotify'},
|
||||||
|
{'name': 'C', 'id': '2', 'source': 'spotify'},
|
||||||
|
]}
|
||||||
|
status, count = w.process_artist('S', 'A', lambda n, l: payload, store)
|
||||||
|
assert status == 'matched' and count == 1 and len(calls) == 2
|
||||||
|
|
@ -32889,6 +32889,27 @@ except Exception as e:
|
||||||
amazon_worker = None
|
amazon_worker = None
|
||||||
|
|
||||||
|
|
||||||
|
# --- Similar Artists Worker Initialization ---
|
||||||
|
# Fills the similar_artists table for LIBRARY artists (the watchlist scanner only
|
||||||
|
# covers watchlist artists). Opt-in by default like Amazon: it leans on MusicMap
|
||||||
|
# (a scraped, outage-prone source) and does per-candidate metadata-source
|
||||||
|
# searches across the whole library, so it stays paused until the user enables it.
|
||||||
|
similar_artists_worker = None
|
||||||
|
try:
|
||||||
|
from core.similar_artists_worker import SimilarArtistsWorker
|
||||||
|
similar_artists_db = MusicDatabase()
|
||||||
|
similar_artists_worker = SimilarArtistsWorker(database=similar_artists_db)
|
||||||
|
similar_artists_worker.start()
|
||||||
|
if config_manager.get('similar_artists_enrichment_paused', True):
|
||||||
|
similar_artists_worker.pause()
|
||||||
|
logger.info("Similar Artists worker initialized (paused — enable it to populate library similars)")
|
||||||
|
else:
|
||||||
|
logger.info("Similar Artists worker initialized and started")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Similar Artists worker initialization failed: {e}")
|
||||||
|
similar_artists_worker = None
|
||||||
|
|
||||||
|
|
||||||
# ================================================================================================
|
# ================================================================================================
|
||||||
# SPOTIFY ENRICHMENT INTEGRATION
|
# SPOTIFY ENRICHMENT INTEGRATION
|
||||||
# ================================================================================================
|
# ================================================================================================
|
||||||
|
|
@ -34653,6 +34674,11 @@ _register_enrichment_services([
|
||||||
worker_getter=lambda: amazon_worker,
|
worker_getter=lambda: amazon_worker,
|
||||||
config_paused_key='amazon_enrichment_paused',
|
config_paused_key='amazon_enrichment_paused',
|
||||||
),
|
),
|
||||||
|
_EnrichmentService(
|
||||||
|
id='similar_artists', display_name='Similar Artists',
|
||||||
|
worker_getter=lambda: similar_artists_worker,
|
||||||
|
config_paused_key='similar_artists_enrichment_paused',
|
||||||
|
),
|
||||||
])
|
])
|
||||||
|
|
||||||
_configure_enrichment_api(
|
_configure_enrichment_api(
|
||||||
|
|
|
||||||
|
|
@ -32,6 +32,12 @@ const ENRICHMENT_WORKERS = [
|
||||||
{ id: 'tidal', name: 'Tidal', color: '#00cfe6', logoSel: '.tidal-enrich-logo', imgFilter: 'invert(1) brightness(1.8)', imgRound: true },
|
{ id: 'tidal', name: 'Tidal', color: '#00cfe6', logoSel: '.tidal-enrich-logo', imgFilter: 'invert(1) brightness(1.8)', imgRound: true },
|
||||||
{ id: 'qobuz', name: 'Qobuz', color: '#0070ef', logoSel: '.qobuz-enrich-logo', imgFilter: 'invert(1)', imgRound: true },
|
{ id: 'qobuz', name: 'Qobuz', color: '#0070ef', logoSel: '.qobuz-enrich-logo', imgFilter: 'invert(1)', imgRound: true },
|
||||||
{ id: 'amazon', name: 'Amazon Music', color: '#ff9900', logoSel: '.amazon-enrich-logo', imgFilter: 'brightness(0) invert(1)' },
|
{ id: 'amazon', name: 'Amazon Music', color: '#ff9900', logoSel: '.amazon-enrich-logo', imgFilter: 'brightness(0) invert(1)' },
|
||||||
|
// Relationship worker (MusicMap → similar artists), not a metadata source.
|
||||||
|
// No dashboard logo element, so it uses a direct MusicMap logo URL.
|
||||||
|
// relationship:true tells the detail panel to drop the (inapplicable)
|
||||||
|
// manual-match affordance.
|
||||||
|
{ id: 'similar_artists', name: 'Similar Artists', color: '#a855f7', relationship: true,
|
||||||
|
logoUrl: 'https://www.music-map.com/elements/objects/og_logo.png', imgRound: true },
|
||||||
];
|
];
|
||||||
|
|
||||||
const _emWorkerById = Object.fromEntries(ENRICHMENT_WORKERS.map(w => [w.id, w]));
|
const _emWorkerById = Object.fromEntries(ENRICHMENT_WORKERS.map(w => [w.id, w]));
|
||||||
|
|
@ -48,7 +54,9 @@ function _emHexToRgb(hex) {
|
||||||
// Resolve a worker's logo URL from the live dashboard bubble (null if absent).
|
// Resolve a worker's logo URL from the live dashboard bubble (null if absent).
|
||||||
function _emLogoSrc(workerId) {
|
function _emLogoSrc(workerId) {
|
||||||
const w = _emWorkerById[workerId];
|
const w = _emWorkerById[workerId];
|
||||||
if (!w || !w.logoSel) return null;
|
if (!w) return null;
|
||||||
|
if (w.logoUrl) return w.logoUrl; // direct URL (workers w/o a dashboard logo)
|
||||||
|
if (!w.logoSel) return null;
|
||||||
const img = document.querySelector(w.logoSel);
|
const img = document.querySelector(w.logoSel);
|
||||||
return img && img.src ? img.src : null;
|
return img && img.src ? img.src : null;
|
||||||
}
|
}
|
||||||
|
|
@ -825,7 +833,8 @@ function _emRenderUnmatchedList() {
|
||||||
<div class="em-row-meta">${statusBadge} ${last}</div>
|
<div class="em-row-meta">${statusBadge} ${last}</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="em-row-actions">
|
<div class="em-row-actions">
|
||||||
<button class="em-btn em-btn--sm" onclick="openEnrichmentMatch('${id}','${entity}','${safeId}', this)">Match</button>
|
${(_emWorkerById[id] && _emWorkerById[id].relationship) ? ''
|
||||||
|
: `<button class="em-btn em-btn--sm" onclick="openEnrichmentMatch('${id}','${entity}','${safeId}', this)">Match</button>`}
|
||||||
<button class="em-btn em-btn--sm em-btn--ghost" title="Re-queue for the worker to try again"
|
<button class="em-btn em-btn--sm em-btn--ghost" title="Re-queue for the worker to try again"
|
||||||
onclick="retryEnrichmentItem('${id}','${entity}','${safeId}', this)">Retry</button>
|
onclick="retryEnrichmentItem('${id}','${entity}','${safeId}', this)">Retry</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue