diff --git a/DISCOVER_BEST_IN_CLASS_PLAN.md b/DISCOVER_BEST_IN_CLASS_PLAN.md new file mode 100644 index 00000000..b406ad52 --- /dev/null +++ b/DISCOVER_BEST_IN_CLASS_PLAN.md @@ -0,0 +1,74 @@ +# discover page — best in class plan (#913 + full generator audit) + +morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested. + +## what i shipped tonight (done, tested, safe) + +### 1. listening recommendations (#913) — went from BROKEN to best-in-class + +the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation: + +- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows. +- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker. +- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally). + +the fix (all in the pure, tested core + thin scan wiring): +- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more. +- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals. + +result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable. + +### 2. its own row on the discover page + +new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty. + +**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in. + +> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own. + +### 3. Fresh Tape "only 5-10 tracks" — fixed + +root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged. + +**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched. + +--- + +## best-in-class roadmap for listening recs (next phases — your call) + +these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own. + +| phase | what | value | risk | notes | +|---|---|---|---|---| +| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). | +| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. | +| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. | +| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. | +| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. | + +the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested. + +--- + +## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz) + +how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character). + +### curated (built during the scan, then hydrated) +- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time. +- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix. +- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug. + +### discovery-pool generators (live queries) +- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most. +- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.) +- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary. +- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing). +- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data. +- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*). + +### cross-cutting +- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win. +- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work. + +want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next. diff --git a/core/discovery/listening_recommendations.py b/core/discovery/listening_recommendations.py index 9af7b417..3bc544e6 100644 --- a/core/discovery/listening_recommendations.py +++ b/core/discovery/listening_recommendations.py @@ -46,6 +46,67 @@ def _get(row: object, attr: str): return getattr(row, attr, None) +def names_match(a: object, b: object) -> bool: + """Strict artist-name equality after stripping case + non-alphanumerics. + + Used to verify a name-search result before fetching that artist's top tracks, so the + "Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact + alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell". + Pure. + """ + def _alnum(x: object) -> str: + return "".join(ch for ch in str(x or "").lower() if ch.isalnum()) + na, nb = _alnum(a), _alnum(b) + return bool(na) and na == nb + + +def similarity_from_rank(rank: object, max_rank: int = 10) -> float: + """Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight. + + SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest + match). The ranker multiplies this into the score so a seed's *closest* matches count + for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0, + rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A + missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure. + """ + try: + r = int(rank) + except (TypeError, ValueError): + return 1.0 + floor = round(1.0 / max_rank, 4) + if r <= 1: + return 1.0 + if r >= max_rank: + return floor + return round((max_rank - r + 1) / max_rank, 4) + + +def build_recency_weighted_seeds( + top_artists: Sequence[dict], + recent_play_counts: Optional[Dict[str, float]] = None, + *, + recency_factor: float = 1.5, +) -> List[dict]: + """Blend lifetime + recent play counts into seed weights — "what you're into NOW". + + ``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a + lot *recently* outranks one you played a lot years ago, so the recommendations track + your current taste instead of your all-time history. ``recency_factor`` is the dial + (0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`. + Pure — the caller supplies both play-count maps from the listening history. + """ + recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()} + out: List[dict] = [] + for a in top_artists or (): + name = str(a.get("name") or "").strip() + if not name: + continue + lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0))) + boost = recency_factor * recent.get(_norm(name), 0.0) + out.append({"name": name, "weight": lifetime + boost}) + return out + + def group_similars_by_seed( seeds: Sequence[dict], similar_rows: Sequence, @@ -53,14 +114,21 @@ def group_similars_by_seed( *, source_id_attr: str = "source_artist_id", similar_name_attr: str = "similar_artist_name", + rank_attr: Optional[str] = None, ) -> Dict[str, List[dict]]: - """Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name': similar}]}``. + """Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``. The stored rows key the similar artist by the SEED's source id (``source_artist_id``), not its name, so :func:`rank_recommended_artists` can't consume them directly. This resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id: artist_name}`` for the library, built by the caller) and keeps only rows that resolve to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O. + + ``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that + is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id. + When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank` + and carried as ``score`` so closer matches weigh more; without it every similar comes out + score-less (the ranker then treats similarity as 1.0 — original behavior). """ seed_names = {_norm(s.get("name")) for s in seeds} seed_names.discard("") @@ -72,8 +140,12 @@ def group_similars_by_seed( if not seed_name or seed_name not in seed_names: continue sim_name = str(_get(row, similar_name_attr) or "").strip() - if sim_name: - out.setdefault(seed_name, []).append({"name": sim_name}) + if not sim_name: + continue + entry = {"name": sim_name} + if rank_attr is not None: + entry["score"] = similarity_from_rank(_get(row, rank_attr)) + out.setdefault(seed_name, []).append(entry) return out @@ -197,8 +269,56 @@ def aggregate_candidate_tracks( return out[:limit] +def to_mix_track(track: object, source: str) -> Optional[dict]: + """Shape one source "top tracks" API dict into the flat dict the Discover compact + playlist row renders + syncs (the "Listening Mix" #913 playlist). + + Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the + same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``). + This flattens that into the renderer's field names (``track_name/artist_name/album_name/ + album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and + sets the source-specific id field. Returns None for anything without a usable id/title so + the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can + dedup by title. Pure — no I/O. + """ + if not isinstance(track, dict): + return None + tid = track.get("id") + name = str(track.get("name") or "").strip() + if not tid or not name: + return None + artists = track.get("artists") or [] + artist_name = "" + if artists and isinstance(artists[0], dict): + artist_name = str(artists[0].get("name") or "").strip() + album = track.get("album") if isinstance(track.get("album"), dict) else {} + album_name = str(album.get("name") or "").strip() + images = album.get("images") or [] + cover = images[0].get("url") if images and isinstance(images[0], dict) else None + out = { + "track_id": str(tid), + "name": name, # for aggregate_candidate_tracks dedup + "track_name": name, # for the renderer + "artist_name": artist_name, + "album_name": album_name, + "album_cover_url": cover, + "duration_ms": track.get("duration_ms") or 0, + "track_data_json": track, # full payload for sync/download + "source": source, + } + id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id", + "itunes": "itunes_track_id"}.get(source) + if id_field: + out[id_field] = str(tid) + return out + + __all__ = [ "RecommendedArtist", + "names_match", + "similarity_from_rank", + "build_recency_weighted_seeds", + "to_mix_track", "group_similars_by_seed", "rank_recommended_artists", "aggregate_candidate_tracks", diff --git a/core/watchlist_scanner.py b/core/watchlist_scanner.py index 33a74cf4..488ffca4 100644 --- a/core/watchlist_scanner.py +++ b/core/watchlist_scanner.py @@ -3633,8 +3633,13 @@ class WatchlistScanner: for source in sources_to_process: logger.info(f"Curating Release Radar for {source}...") - # 1. Curate Release Radar - 50 tracks from recent albums - recent_albums = self.database.get_discovery_recent_albums(limit=50, source=source, profile_id=profile_id) + # 1. Curate Release Radar - 50 tracks from recent albums. + # Fetch a GENEROUS album budget (not 50) and exclude next-year announcements at the + # query: future-dated albums sort to the top of release_date DESC and used to eat the + # budget before the in-loop is_future_release skip ran, starving Fresh Tape to a few + # tracks. The downstream caps (6/artist, top 75, take 50) still bound the output. + recent_albums = self.database.get_discovery_recent_albums( + limit=300, source=source, profile_id=profile_id, exclude_future_years=True) release_radar_tracks = [] if not recent_albums: @@ -4004,45 +4009,129 @@ class WatchlistScanner: The ranking lives in core.discovery.listening_recommendations (pure + tested); this only gathers inputs — all already in the DB, NO new network — and stores the result under NEW metadata/curated keys. Fully self-contained and guarded: any failure logs and returns, so - it can never disturb the scan. Phase-1 candidate tracks come from the discovery pool (like - BYLT); a later phase swaps in a direct top-tracks fetch for richer coverage. + it can never disturb the scan. + + "Best in class" generation (the elevation over the first cut): + • Seeds are recency-weighted — recent plays boost lifetime favourites so the picks + track what you're into NOW, not just all-time totals. + • The ranker is fed the RAW per-seed edges (one row per seed→similar), so an artist + similar to several of your seeds accumulates real CONSENSUS — the old code fed the + name-collapsed ``get_top_similar_artists`` query, which flattened every similar to a + single seed (consensus could never fire). The raw edges also carry ``similarity_rank``, + so a seed's CLOSEST matches outweigh its long-tail ones. + • ``source_artist_id`` is a SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), so the id→name + map is built from the artists' source-id columns, NOT the internal row id (the first + cut keyed it by ``artists.id`` and resolved nothing — the feature produced 0 recs). + Phase-1 candidate tracks still come from the discovery pool (like BYLT); a later phase + swaps in a direct top-tracks fetch for richer coverage. """ try: import json as _json from core.discovery.listening_recommendations import ( aggregate_candidate_tracks, + build_recency_weighted_seeds, group_similars_by_seed, + names_match, rank_recommended_artists, + to_mix_track, ) - seeds = [{'name': s['name'], 'weight': s.get('play_count', 1)} - for s in (self.database.get_top_artists('all', 30) or []) if s.get('name')] - if not seeds: + # Recency-weighted seeds: lifetime top artists, boosted by recent (30d) plays. + lifetime = [s for s in (self.database.get_top_artists('all', 30) or []) if s.get('name')] + if not lifetime: return + recent_rows = self.database.get_top_artists('30d', 50) or [] + recent_counts = {r['name'].lower(): r.get('play_count', 0) + for r in recent_rows if r.get('name')} + seeds = build_recency_weighted_seeds(lifetime, recent_counts) + seed_names = {s['name'].lower() for s in seeds} - # id -> name + owned-artist set for the WHOLE library (similar_artists rows key the - # similar artist by the seed artist's id). - id_to_name, owned = {}, set() + # Owned-artist set (for exclusion) + the seeds' SOURCE ids (similar_artists.source_artist_id + # is a Spotify/iTunes/Deezer/MusicBrainz id, never the internal artists.id). We only need + # id→name for the SEED ids, since the edge query below is already scoped to them. + owned, seed_source_ids, seed_id_to_name = set(), [], {} with self.database._get_connection() as conn: cur = conn.cursor() - cur.execute("SELECT id, name FROM artists WHERE name IS NOT NULL AND name != ''") + cur.execute("SELECT name, spotify_artist_id, itunes_artist_id, deezer_id, " + "musicbrainz_id FROM artists WHERE name IS NOT NULL AND name != ''") for row in cur.fetchall(): - id_to_name[str(row[0])] = row[1] - owned.add((row[1] or '').lower()) + nm = row[0] + lname = (nm or '').lower() + owned.add(lname) + if lname in seed_names: + for sid in (row[1], row[2], row[3], row[4]): + if sid: + seed_source_ids.append(str(sid)) + seed_id_to_name[str(sid)] = nm + + # RAW per-seed edges (preserve consensus + similarity_rank). Scoped to the seeds. + edges, edge_cols = [], ('source_artist_id', 'similar_artist_name', 'similarity_rank', + 'spotify_id', 'itunes_id', 'deezer_id', 'image_url', 'genres') + if seed_source_ids: + placeholders = ",".join("?" * len(seed_source_ids)) + cur.execute( + f"SELECT source_artist_id, similar_artist_name, similarity_rank, " + f"similar_artist_spotify_id, similar_artist_itunes_id, " + f"similar_artist_deezer_id, image_url, genres FROM similar_artists " + f"WHERE profile_id = ? AND source_artist_id IN ({placeholders})", + [profile_id, *seed_source_ids]) + edges = [dict(zip(edge_cols, r, strict=False)) for r in cur.fetchall()] + + # Per-name enrichment (image/ids/genres) so the Discover row can render rich cards. + artist_meta_by_name = {} + for e in edges: + key = (e['similar_artist_name'] or '').lower() + if not key: + continue + m = artist_meta_by_name.setdefault(key, {}) + for src_k, dst_k in (('spotify_id', 'spotify_artist_id'), + ('itunes_id', 'itunes_artist_id'), + ('deezer_id', 'deezer_artist_id')): + if e.get(src_k) and not m.get(dst_k): + m[dst_k] = e[src_k] + if e.get('image_url') and not m.get('image_url'): + m['image_url'] = e['image_url'] + if e.get('genres') and not m.get('genres'): + m['genres'] = e['genres'] + + similars_by_seed = group_similars_by_seed( + seeds, edges, seed_id_to_name, rank_attr='similarity_rank') + + # Fallback: if the raw edges resolved nothing (e.g. source ids not yet populated), + # degrade to the aggregated query so the feature still works rather than going dark. + if not similars_by_seed: + agg = self.database.get_top_similar_artists(limit=1000, profile_id=profile_id) + similars_by_seed = group_similars_by_seed( + seeds, agg, seed_id_to_name, rank_attr='similarity_rank') - similar_rows = self.database.get_top_similar_artists(limit=1000, profile_id=profile_id) - similars_by_seed = group_similars_by_seed(seeds, similar_rows, id_to_name) recs = rank_recommended_artists(seeds, similars_by_seed, owned, limit=40) if not recs: logger.info("[Listening Recs] no recommendations yet (no similar-artist coverage)") return - self.database.set_metadata('listening_recs_artists', _json.dumps([ - {'name': r.name, 'seed_count': r.seed_count, 'seeds': r.seeds[:5], 'score': r.score} - for r in recs - ])) + def _enrich(r): + m = artist_meta_by_name.get(r.name.lower(), {}) + genres = m.get('genres') + if isinstance(genres, str): + try: + genres = _json.loads(genres) + except Exception: + genres = None + return {'name': r.name, 'seed_count': r.seed_count, 'seeds': r.seeds[:5], + 'score': r.score, 'spotify_artist_id': m.get('spotify_artist_id'), + 'itunes_artist_id': m.get('itunes_artist_id'), + 'deezer_artist_id': m.get('deezer_artist_id'), + 'image_url': m.get('image_url'), + 'genres': (genres[:3] if isinstance(genres, list) else None)} - # Candidate tracks from the discovery pool grouped by artist (phase 1: no new network). + self.database.set_metadata('listening_recs_artists', + _json.dumps([_enrich(r) for r in recs])) + + # Candidate tracks for the "Listening Mix" playlist row: each recommended artist's + # top tracks. Prefer a DIRECT top-tracks fetch (Spotify/Deezer — richest + most + # accurate), fall back to the discovery pool (covers iTunes + any artist the fetch + # missed). Stored as full render-ready dicts so the row needs NO pool re-hydration — + # robust against pool rotation (the bug that shrinks Fresh Tape/Archives at read time). pool, active_source = [], None for src in (sources_to_process or []): pool = self.database.get_discovery_pool_tracks( @@ -4050,24 +4139,85 @@ class WatchlistScanner: if pool: active_source = src break + if not active_source: + active_source = (sources_to_process or ['spotify'])[0] - track_ids = [] - if pool: - by_artist = {} - for t in pool: - an = (getattr(t, 'artist_name', '') or '').lower() - tid = (getattr(t, 'spotify_track_id', None) if active_source == 'spotify' - else getattr(t, 'itunes_track_id', None) if active_source == 'itunes' - else getattr(t, 'deezer_track_id', None)) - if an and tid: - by_artist.setdefault(an, []).append({'name': getattr(t, 'track_name', ''), 'id': tid}) - candidates = aggregate_candidate_tracks(recs, by_artist, per_artist=3, limit=50) - track_ids = [c['id'] for c in candidates if c.get('id')] - if track_ids: - self.database.save_curated_playlist('listening_recs_tracks', track_ids, profile_id=profile_id) + # Pool baseline grouped by artist (full render dicts), no network. + pool_by_artist = {} + for t in (pool or []): + an = (getattr(t, 'artist_name', '') or '').lower() + tid = (getattr(t, 'spotify_track_id', None) if active_source == 'spotify' + else getattr(t, 'itunes_track_id', None) if active_source == 'itunes' + else getattr(t, 'deezer_track_id', None)) + name = getattr(t, 'track_name', '') or '' + if not an or not tid or not name: + continue + tdj = getattr(t, 'track_data_json', None) + if isinstance(tdj, str): + try: + tdj = _json.loads(tdj) + except Exception: + tdj = None + pool_by_artist.setdefault(an, []).append({ + 'track_id': str(tid), 'name': name, 'track_name': name, + 'artist_name': getattr(t, 'artist_name', '') or '', + 'album_name': getattr(t, 'album_name', '') or '', + 'album_cover_url': getattr(t, 'album_cover_url', None), + 'duration_ms': getattr(t, 'duration_ms', 0) or 0, + 'track_data_json': tdj, 'source': active_source, + f'{active_source}_track_id': str(tid)}) - logger.info("[Listening Recs] %d recommended artists, %d candidate tracks", - len(recs), len(track_ids)) + # Direct top-tracks enrichment — guarded, bounded (top 20 recs), fail-soft per artist. + fetched_by_artist = {} + try: + if active_source in ('spotify', 'deezer'): + client = get_client_for_source(active_source) + if client and hasattr(client, 'get_artist_top_tracks'): + id_key = 'spotify_artist_id' if active_source == 'spotify' else 'deezer_artist_id' + can_search = hasattr(client, 'search_artists') + for r in recs[:20]: + aid = artist_meta_by_name.get(r.name.lower(), {}).get(id_key) + # Most similar-artist rows store a name but no source id, so resolve + # it by name-search — guarded by names_match so we never fetch the + # WRONG artist's tracks (a same-name act). + if not aid and can_search: + try: + found = client.search_artists(r.name, limit=1) or [] + except Exception as _s_err: + logger.debug("[Listening Recs] artist search failed for %s: %s", + r.name, _s_err) + found = [] + if found and names_match(r.name, getattr(found[0], 'name', '')): + aid = getattr(found[0], 'id', None) + if not aid: + continue + try: + raw = client.get_artist_top_tracks(str(aid), limit=8) or [] + except Exception as _tt_err: + logger.debug("[Listening Recs] top-tracks fetch failed for %s: %s", + r.name, _tt_err) + continue + shaped = [s for s in (to_mix_track(x, active_source) for x in raw) if s][:5] + if shaped: + fetched_by_artist[r.name.lower()] = shaped + except Exception as _enr_err: + logger.debug("[Listening Recs] top-tracks enrichment skipped: %s", _enr_err) + + # Merge: prefer fetched top tracks, fall back to the pool per artist. + top_tracks_by_artist = {} + for r in recs: + merged = fetched_by_artist.get(r.name.lower()) or pool_by_artist.get(r.name.lower()) + if merged: + top_tracks_by_artist[r.name.lower()] = merged + + mix = aggregate_candidate_tracks(recs, top_tracks_by_artist, per_artist=3, limit=50) + track_ids = [m.get('track_id') for m in mix if m.get('track_id')] + if mix: + self.database.set_metadata('listening_recs_tracks_full', _json.dumps(mix)) + self.database.save_curated_playlist('listening_recs_tracks', track_ids, profile_id=profile_id) + + logger.info("[Listening Recs] %d recommended artists, %d mix tracks (%d artists via top-tracks fetch)", + len(recs), len(mix), len(fetched_by_artist)) except Exception as e: logger.debug("[Listening Recs] generation skipped: %s", e) diff --git a/database/music_database.py b/database/music_database.py index c173015b..97925c32 100644 --- a/database/music_database.py +++ b/database/music_database.py @@ -10884,23 +10884,40 @@ class MusicDatabase: logger.error(f"Error caching discovery recent album: {e}") return False - def get_discovery_recent_albums(self, limit: int = 10, source: Optional[str] = None, profile_id: int = 1) -> List[Dict[str, Any]]: - """Get cached recent albums for discover page, optionally filtered by source""" + def get_discovery_recent_albums(self, limit: int = 10, source: Optional[str] = None, profile_id: int = 1, + exclude_future_years: bool = False) -> List[Dict[str, Any]]: + """Get cached recent albums for discover page, optionally filtered by source. + + exclude_future_years: drop announced-but-unreleased albums dated to a LATER YEAR. + Because rows are ordered ``release_date DESC``, future-dated albums otherwise sort to + the very top and consume the ``limit`` budget — which is exactly why Fresh Tape / Release + Radar starved down to a handful of tracks. Year-level so it's precision-safe across + 'YYYY' / 'YYYY-MM' / 'YYYY-MM-DD'; same-year future months are left for the caller's precise + ``is_future_release`` check. NULL/blank dates are kept (treated as released). + """ try: with self._get_connection() as conn: cursor = conn.cursor() + future_clause = "" + if exclude_future_years: + future_clause = ( + " AND (release_date IS NULL OR release_date = '' " + "OR CAST(substr(release_date, 1, 4) AS INTEGER) " + "<= CAST(strftime('%Y','now') AS INTEGER))" + ) + if source: - cursor.execute(""" + cursor.execute(f""" SELECT * FROM discovery_recent_albums - WHERE source = ? AND profile_id = ? + WHERE source = ? AND profile_id = ?{future_clause} ORDER BY release_date DESC LIMIT ? """, (source, profile_id, limit)) else: - cursor.execute(""" + cursor.execute(f""" SELECT * FROM discovery_recent_albums - WHERE profile_id = ? + WHERE profile_id = ?{future_clause} ORDER BY release_date DESC LIMIT ? """, (profile_id, limit)) diff --git a/tests/discovery/test_listening_recommendations.py b/tests/discovery/test_listening_recommendations.py index a4610c80..87ec52c1 100644 --- a/tests/discovery/test_listening_recommendations.py +++ b/tests/discovery/test_listening_recommendations.py @@ -4,14 +4,67 @@ from __future__ import annotations from core.discovery.listening_recommendations import ( aggregate_candidate_tracks, + build_recency_weighted_seeds, + names_match, rank_recommended_artists, + similarity_from_rank, + to_mix_track, ) +# ── names_match (guards the top-tracks fetch against wrong-artist results) ───── +def test_names_match_ignores_case_and_punctuation(): + assert names_match("Tyler, The Creator", "Tyler The Creator") + assert names_match("BEYONCÉ", "beyoncé") + assert names_match("AC/DC", "ac dc") + + +def test_names_match_rejects_near_misses_and_empty(): + assert not names_match("Drake", "Drake Bell") + assert not names_match("", "anything") + assert not names_match("X", None) + + def _seed(name, weight=1.0): return {"name": name, "weight": weight} +# ── similarity_from_rank (1=closest .. 10=farthest -> 1.0 .. 0.1) ───────────── +def test_similarity_from_rank_decays_over_documented_range(): + assert similarity_from_rank(1) == 1.0 + assert similarity_from_rank(5) == 0.6 + assert similarity_from_rank(10) == 0.1 + + +def test_similarity_from_rank_clamps_and_defaults(): + assert similarity_from_rank(0) == 1.0 # <=1 -> full weight + assert similarity_from_rank(50) == 0.1 # beyond range -> floor + assert similarity_from_rank(None) == 1.0 # missing -> full weight (no rank info) + assert similarity_from_rank("nan") == 1.0 + + +# ── build_recency_weighted_seeds (lifetime + factor*recent) ─────────────────── +def test_recency_boost_reorders_toward_current_taste(): + # Old-fav has more lifetime plays, but New-fav dominates recently. + lifetime = [{"name": "OldFav", "play_count": 100}, {"name": "NewFav", "play_count": 40}] + recent = {"newfav": 60} + seeds = build_recency_weighted_seeds(lifetime, recent, recency_factor=1.5) + by = {s["name"]: s["weight"] for s in seeds} + assert by["OldFav"] == 100.0 # no recent plays -> unchanged + assert by["NewFav"] == 40 + 1.5 * 60 # 130 -> now outranks OldFav + + +def test_recency_factor_zero_is_pure_lifetime(): + seeds = build_recency_weighted_seeds( + [{"name": "A", "play_count": 7}], {"a": 99}, recency_factor=0) + assert seeds == [{"name": "A", "weight": 7.0}] + + +def test_recency_seeds_skip_blank_names_and_tolerate_missing_recent(): + seeds = build_recency_weighted_seeds([{"name": ""}, {"name": "A", "play_count": 3}]) + assert seeds == [{"name": "A", "weight": 3.0}] + + # ── rank_recommended_artists ───────────────────────────────────────────────── def test_consensus_outranks_single_endorsement(): # 'Common' is similar to BOTH seeds; 'Solo' to one. Equal weights/scores. @@ -117,6 +170,46 @@ def test_aggregate_skips_artist_with_no_tracks(): assert [t["name"] for t in out] == ["only"] # sim-b had no tracks -> skipped +# ── to_mix_track (source top-track dict -> Discover compact-row dict) ───────── +def _sp_track(tid="t1", name="Song", artist="Artist", album="Album", cover="http://cdn/c.jpg"): + return {"id": tid, "name": name, "artists": [{"name": artist}], + "album": {"name": album, "images": ([{"url": cover}] if cover else [])}, + "duration_ms": 210000, "popularity": 55} + + +def test_to_mix_track_shapes_render_fields(): + out = to_mix_track(_sp_track(), "spotify") + assert out["track_name"] == "Song" and out["artist_name"] == "Artist" + assert out["album_name"] == "Album" and out["album_cover_url"] == "http://cdn/c.jpg" + assert out["duration_ms"] == 210000 + assert out["spotify_track_id"] == "t1" and out["track_id"] == "t1" + assert out["track_data_json"]["id"] == "t1" # full payload kept for sync + assert out["name"] == "Song" # kept for aggregate dedup + + +def test_to_mix_track_source_id_field_per_source(): + assert to_mix_track(_sp_track(), "deezer")["deezer_track_id"] == "t1" + assert to_mix_track(_sp_track(), "itunes")["itunes_track_id"] == "t1" + + +def test_to_mix_track_rejects_unusable_and_tolerates_missing_album(): + assert to_mix_track({"id": "x"}, "spotify") is None # no title + assert to_mix_track({"name": "y"}, "spotify") is None # no id + assert to_mix_track("garbage", "spotify") is None + bare = to_mix_track({"id": "z", "name": "Z"}, "spotify") # no artists/album + assert bare["artist_name"] == "" and bare["album_cover_url"] is None + + +def test_to_mix_track_feeds_aggregate_end_to_end(): + # The real pipeline: shape source tracks, then aggregate by recommended artist. + recs = _recs("A") # recommends 'sim-A' + shaped = [to_mix_track(_sp_track(tid="1", name="One", artist="sim-A"), "spotify"), + to_mix_track(_sp_track(tid="2", name="Two", artist="sim-A"), "spotify")] + out = aggregate_candidate_tracks(recs, {"sim-a": shaped}, per_artist=5, limit=10) + assert [t["track_name"] for t in out] == ["One", "Two"] + assert out[0]["spotify_track_id"] == "1" + + # ── group_similars_by_seed (id->name join) ─────────────────────────────────── from dataclasses import dataclass as _dc # noqa: E402 @@ -167,3 +260,37 @@ def test_group_then_rank_end_to_end(): ranked = rank_recommended_artists(seeds, grouped, owned_artist_names={"solo"}) assert ranked[0].name == "Common" and ranked[0].seed_count == 2 assert all(r.name != "Solo" for r in ranked) # owned excluded + + +# ── rank-aware grouping (similarity_rank -> score) ──────────────────────────── +@_dc +class _RankRow: + source_artist_id: str + similar_artist_name: str + similarity_rank: int + + +def test_group_with_rank_attr_carries_similarity_score(): + seeds = [_seed("A")] + rows = [_RankRow("ia", "Close", 1), _RankRow("ia", "Far", 10)] + out = group_similars_by_seed(seeds, rows, {"ia": "A"}, rank_attr="similarity_rank") + by = {e["name"]: e["score"] for e in out["a"]} + assert by == {"Close": 1.0, "Far": 0.1} + + +def test_group_without_rank_attr_is_scoreless_backcompat(): + seeds = [_seed("A")] + rows = [_RankRow("ia", "X", 3)] + out = group_similars_by_seed(seeds, rows, {"ia": "A"}) + assert out["a"] == [{"name": "X"}] # no score key -> original behavior + + +def test_rank_threading_changes_winner_within_a_seed(): + # The production fix: a CLOSER match (rank 1) on a heavy seed beats a far match (rank 9), + # even though both come from the same seed. Without rank threading they'd tie. + seeds = [_seed("Fav", weight=10)] + rows = [_RankRow("if", "Close", 1), _RankRow("if", "Far", 9)] + grouped = group_similars_by_seed(seeds, rows, {"if": "Fav"}, rank_attr="similarity_rank") + ranked = rank_recommended_artists(seeds, grouped) + assert [r.name for r in ranked] == ["Close", "Far"] + assert ranked[0].score > ranked[1].score diff --git a/tests/discovery/test_recent_albums_future_filter.py b/tests/discovery/test_recent_albums_future_filter.py new file mode 100644 index 00000000..30b382c4 --- /dev/null +++ b/tests/discovery/test_recent_albums_future_filter.py @@ -0,0 +1,57 @@ +"""Fresh Tape / Release Radar candidate fetch must not be starved by future albums. + +Regression: get_discovery_recent_albums orders release_date DESC, so announced-but- +unreleased albums dated to a LATER YEAR sort to the very top and consumed the album +budget before the scanner's in-loop is_future_release skip ran — leaving only a handful +of released albums to draw tracks from (the reported "Fresh Tape only has 5-10 tracks"). +exclude_future_years drops next-year albums at the query so released ones fill the budget. +""" + +from __future__ import annotations + +from datetime import datetime + +from database.music_database import MusicDatabase + + +def _insert(db, **kw): + with db._get_connection() as conn: + conn.execute( + """INSERT INTO discovery_recent_albums + (album_spotify_id, album_name, artist_name, artist_spotify_id, + album_cover_url, release_date, album_type, source, profile_id) + VALUES (?,?,?,?,?,?,?,?,?)""", + (kw['id'], kw['name'], kw['artist'], kw['id'] + '_a', '', + kw['release_date'], kw.get('album_type', 'album'), kw.get('source', 'spotify'), 1)) + conn.commit() + + +def test_future_year_albums_excluded_released_kept(tmp_path): + db = MusicDatabase(database_path=str(tmp_path / "t.db")) + this_year = datetime.now().year + next_year = this_year + 1 + _insert(db, id='past1', name='Released A', artist='X', release_date=f'{this_year - 1}-03-01') + _insert(db, id='past2', name='Released B', artist='Y', release_date=f'{this_year}-01-15') + _insert(db, id='fut1', name='Announced', artist='Z', release_date=f'{next_year}-02-01') + _insert(db, id='fut2', name='Year Only Future', artist='W', release_date=str(next_year)) + _insert(db, id='blank', name='Unknown Date', artist='Q', release_date='') + + names_all = {a['album_name'] for a in db.get_discovery_recent_albums(limit=50, source='spotify')} + assert 'Announced' in names_all # without the flag, futures are present (and sort first) + + filtered = db.get_discovery_recent_albums(limit=50, source='spotify', exclude_future_years=True) + names = {a['album_name'] for a in filtered} + assert 'Announced' not in names # next-year album dropped + assert 'Year Only Future' not in names # YYYY-only future dropped + assert {'Released A', 'Released B'} <= names # released kept + assert 'Unknown Date' in names # blank date kept (treated as released) + + +def test_future_filter_does_not_over_trim_when_all_released(tmp_path): + db = MusicDatabase(database_path=str(tmp_path / "t2.db")) + this_year = datetime.now().year + for i in range(8): + _insert(db, id=f'r{i}', name=f'Album {i}', artist=f'A{i}', + release_date=f'{this_year}-0{(i % 9) + 1}-01') + filtered = db.get_discovery_recent_albums(limit=300, source='spotify', exclude_future_years=True) + assert len(filtered) == 8 # every released album survives, budget honored diff --git a/tests/test_watchlist_scanner_scan.py b/tests/test_watchlist_scanner_scan.py index 9eda637f..26a1a7f8 100644 --- a/tests/test_watchlist_scanner_scan.py +++ b/tests/test_watchlist_scanner_scan.py @@ -1171,7 +1171,7 @@ def test_curate_discovery_playlists_uses_source_priority_for_recent_albums(monke "avg_daily_plays": 0.0, "artist_play_counts": {}, }) - monkeypatch.setattr(scanner.database, "get_discovery_recent_albums", lambda limit, source, profile_id: [recent_album] if source == "deezer" else [], raising=False) + monkeypatch.setattr(scanner.database, "get_discovery_recent_albums", lambda limit, source, profile_id, exclude_future_years=False: [recent_album] if source == "deezer" else [], raising=False) monkeypatch.setattr(scanner.database, "get_discovery_pool_tracks", lambda *args, **kwargs: [discovery_track] if kwargs.get("source") == "deezer" else [], raising=False) monkeypatch.setattr(scanner.database, "save_curated_playlist", lambda key, tracks, profile_id=1: saved_playlists.append((key, list(tracks))) or True, raising=False) monkeypatch.setattr(scanner.database, "get_top_artists", lambda *args, **kwargs: [], raising=False) diff --git a/web_server.py b/web_server.py index 4e330c92..0c81f2f3 100644 --- a/web_server.py +++ b/web_server.py @@ -29083,6 +29083,96 @@ def get_discover_similar_artists(): return jsonify({"success": False, "error": str(e)}), 500 +@app.route('/api/discover/listening-recommendations', methods=['GET']) +def get_discover_listening_recommendations(): + """#913: artists you'd love based on what you actually LISTEN to (play-weighted). + + Distinct from /api/discover/similar-artists (which is driven by your whole library / + watchlist): this is seeded by your most-PLAYED artists, consensus-ranked across the + similar-artist graph, and recency-boosted. The heavy lifting + storage happen during the + watchlist scan (core.watchlist_scanner._build_listening_recommendations -> the + 'listening_recs_artists' metadata key); this endpoint just reshapes the stored list to the + same card shape the recommended-artists row already renders. Read-only, fail-soft. + """ + try: + database = get_database() + active_source = _get_active_discovery_source() + raw = database.get_metadata('listening_recs_artists') + if not raw: + return jsonify({"success": True, "artists": [], "source": active_source, "count": 0}) + try: + stored = json.loads(raw) or [] + except (ValueError, TypeError): + stored = [] + + result_artists = [] + for a in stored: + name = a.get('name') + if not name: + continue + if active_source == 'spotify': + artist_id = a.get('spotify_artist_id') + elif active_source == 'deezer': + artist_id = a.get('deezer_artist_id') or a.get('itunes_artist_id') + else: + artist_id = a.get('itunes_artist_id') + entry = { + "artist_id": artist_id, + "spotify_artist_id": a.get('spotify_artist_id'), + "itunes_artist_id": a.get('itunes_artist_id'), + "deezer_artist_id": a.get('deezer_artist_id'), + "artist_name": name, + "seed_count": a.get('seed_count'), + "source": active_source, + } + img = a.get('image_url') + if img: + entry["image_url"] = fix_artist_image_url(img) + if a.get('genres'): + entry["genres"] = a['genres'][:3] + # "because you listen to X, Y, Z" — the most-played artists that point here. + if a.get('seeds'): + entry["because"] = a['seeds'] + result_artists.append(entry) + + return jsonify({ + "success": True, + "artists": result_artists, + "source": active_source, + "count": len(result_artists), + }) + except Exception as e: + logger.error(f"Error getting listening recommendations: {e}") + return jsonify({"success": False, "error": str(e)}), 500 + + +@app.route('/api/discover/personalized/listening-mix', methods=['GET']) +def get_discover_listening_mix(): + """#913: the "Listening Mix" playlist row — a playable track mix from the artists you'd + love based on what you actually listen to. + + The tracks are built during the watchlist scan (core.watchlist_scanner + ._build_listening_recommendations -> the 'listening_recs_tracks_full' metadata key) as full + render-ready dicts, so this endpoint just hands them back — no discovery-pool re-hydration, + which means it can't shrink when the pool rotates (the failure mode Fresh Tape/Archives hit). + Same {success, tracks} shape renderCompactPlaylist + the sync/download chains expect. + """ + try: + database = get_database() + active_source = _get_active_discovery_source() + raw = database.get_metadata('listening_recs_tracks_full') + tracks = [] + if raw: + try: + tracks = json.loads(raw) or [] + except (ValueError, TypeError): + tracks = [] + return jsonify({"success": True, "tracks": tracks, "source": active_source}) + except Exception as e: + logger.error(f"Error getting listening mix: {e}") + return jsonify({"success": False, "error": str(e)}), 500 + + @app.route('/api/discover/similar-artists/enrich', methods=['POST']) def enrich_similar_artists(): """Enrich a batch of artist IDs with images/genres from Spotify or iTunes. diff --git a/webui/index.html b/webui/index.html index 2d0e6fe3..1b83fae9 100644 --- a/webui/index.html +++ b/webui/index.html @@ -3133,6 +3133,53 @@ + + + + + +