Personalized playlists (1/N): unified storage + manager foundation

Begins the standardization of the personalized-playlist subsystem.
Pre-existing state was a patchwork: Group A (Fresh Tape / Archives /
Seasonal Mix) lived in `discovery_curated_playlists` and
`curated_seasonal_playlists` with inconsistent shapes; Group B
(Hidden Gems / Discovery Shuffle / Time Machine / Popular Picks /
Genre / Daily Mixes) was computed on-demand by
`PersonalizedPlaylistsService` with no persistence -- every call
reran the generator with `ORDER BY RANDOM()` so results rotated.

Post-overhaul (this PR) every personalized playlist lands in one
unified storage layer with stable identity, persistent track lists,
explicit refresh, and per-playlist user-tweakable config.

Foundation in this commit (no behavior change yet):

- `database/personalized_schema.py`: 3 tables created idempotently
  at app startup (wired into `MusicDatabase._initialize_database`).
  - `personalized_playlists`: one row per (profile, kind, variant)
    with config_json, track_count, last_generated_at,
    last_synced_at, last_generation_source, last_generation_error.
    Variant '' (empty string) for singletons; non-empty for
    time_machine / seasonal_mix / genre_playlist / daily_mix.
  - `personalized_playlist_tracks`: current snapshot per playlist.
    Atomically replaced on refresh.
  - `personalized_track_history`: append-only log powering the
    `exclude_recent_days` config knob.

- `core/personalized/types.py`: `Track`, `PlaylistConfig`,
  `PlaylistRecord` dataclasses. `PlaylistConfig.merged()` for
  partial-update PATCH semantics; `Track.from_dict()` accepts the
  legacy generator output shape unchanged.

- `core/personalized/specs.py`: `PlaylistKindSpec` (kind,
  name_template, default_config, generator, variant_resolver) and a
  module-level registry. Generators register at import time;
  manager dispatches by kind.

- `core/personalized/manager.py`: `PersonalizedPlaylistManager` --
  the only thing that touches the new tables. Owns:
  - ensure_playlist (auto-create row from kind defaults)
  - get_playlist / list_playlists
  - refresh_playlist (atomic snapshot replace; generator exception
    preserves previous good snapshot + records error on row)
  - get_playlist_tracks
  - update_config (deep-merge with stored config, including extra dict)
  - recent_track_ids (staleness lookup for generators)

35 boundary tests in `tests/test_personalized_manager.py` pin every
shape: config round-trip / merge semantics / extra deep-merge /
defaults; Track.from_dict tolerance + primary_id fallback chain;
registry dedup / display_name with+without variant; manager
ensure_playlist auto-create + idempotency, variant separation,
required-variant enforcement, unknown-kind error; refresh persists
+ replaces atomically + survives generator exception with previous
snapshot intact + records source from first track + round-trips
nested track_data_json; update_config patch semantics; list_playlists
profile scoping; staleness history scoped to (profile, kind, days).

3304 tests pass total. Generators ship in subsequent commits on this
branch -- each kind migrated one at a time with its own per-kind
boundary tests.
This commit is contained in:
Broque Thomas 2026-05-15 16:19:01 -07:00
parent 19a18ba992
commit 79224ed294
7 changed files with 1297 additions and 1 deletions

View file

@ -0,0 +1,24 @@
"""Standardized personalized-playlist subsystem.
Replaces the patchwork of `PersonalizedPlaylistsService` (computed-on-
demand views, no persistence) + `discovery_curated_playlists` (ID-only
storage) + `curated_seasonal_playlists` (full storage) with a single
unified abstraction:
- ``manager.PersonalizedPlaylistManager`` owns the storage layer +
generator dispatch + refresh lifecycle.
- ``specs.PlaylistKindSpec`` one spec per playlist KIND
(``hidden_gems``, ``time_machine``, ``seasonal_mix``, etc.) with
generator function, default config, variant resolver, and display-
name template.
- ``types.Track`` / ``types.PlaylistConfig`` shared dataclasses.
The legacy ``PersonalizedPlaylistsService`` keeps its existing
generator implementations they're called BY the manager rather than
duplicated. This means:
- The improved diversity logic / popularity thresholds / blacklist
filtering all stays.
- New behavior layered on top: persistence, refresh-on-demand,
per-playlist user-tweakable config, staleness windows, listening-
history cross-reference, seeded randomization.
"""

View file

@ -0,0 +1,409 @@
"""Storage layer + lifecycle for personalized playlists.
The manager is the ONLY place that touches the
``personalized_playlists`` / ``personalized_playlist_tracks`` /
``personalized_track_history`` tables. Generators (in
``core/personalized/generators/``) produce track lists; the manager
persists, refreshes, and serves them.
Key invariants:
- ``(profile_id, kind, variant)`` uniquely identifies a playlist.
Variant '' (empty string) means singleton used for kinds like
``hidden_gems`` that don't have multiple instances per profile.
- A playlist row is auto-created on first access (``ensure_playlist``)
using the kind's default config.
- Track lists are atomically replaced on refresh never partial-
mutated. Either the new snapshot lands fully or the old one
remains.
- Refresh failures get logged on the row
(``last_generation_error``) so the UI can surface them without
losing the previous good snapshot.
- Staleness history is append-only and queried by the
``exclude_recent_days`` config option (handled by individual
generators when they want to honor it).
"""
from __future__ import annotations
import json
import time
from dataclasses import asdict
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from utils.logging_config import get_logger
from core.personalized.specs import PlaylistKindRegistry, get_registry
from core.personalized.types import PlaylistConfig, PlaylistRecord, Track
logger = get_logger("personalized.manager")
class PersonalizedPlaylistManager:
"""Owns persistence + refresh lifecycle for personalized playlists."""
def __init__(self, database, deps: Any, registry: Optional[PlaylistKindRegistry] = None):
"""
Args:
database: MusicDatabase singleton (exposes ``_get_connection``).
deps: Opaque object passed through to each generator
callable. Whatever a generator needs (the legacy
``PersonalizedPlaylistsService`` instance, the
``config_manager``, a metadata client) goes in here.
Manager doesn't inspect it.
registry: optional PlaylistKindRegistry override (for tests).
"""
self.database = database
self.deps = deps
self.registry = registry or get_registry()
# ─── playlist row lifecycle ──────────────────────────────────────
def ensure_playlist(self, kind: str, variant: str = '', profile_id: int = 1) -> PlaylistRecord:
"""Return the playlist row for ``(profile, kind, variant)``,
creating it from the kind's default config if it doesn't exist."""
spec = self.registry.get(kind)
if spec is None:
raise ValueError(f"Unknown playlist kind: {kind!r}")
if spec.requires_variant and not variant:
raise ValueError(f"Kind {kind!r} requires a variant")
existing = self._fetch_playlist_row(kind, variant, profile_id)
if existing is not None:
return existing
# Insert new row using the kind's defaults.
config = spec.default_config
name = spec.display_name(variant)
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
INSERT INTO personalized_playlists
(profile_id, kind, variant, name, config_json,
track_count, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
""",
(profile_id, kind, variant, name, json.dumps(config.to_json_dict())),
)
conn.commit()
return self._fetch_playlist_row(kind, variant, profile_id) # type: ignore[return-value]
def get_playlist(self, kind: str, variant: str = '', profile_id: int = 1) -> Optional[PlaylistRecord]:
"""Return the playlist row if it exists. Does NOT auto-create."""
return self._fetch_playlist_row(kind, variant, profile_id)
def list_playlists(self, profile_id: int = 1) -> List[PlaylistRecord]:
"""List every persisted playlist for a profile, newest-first."""
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT id, profile_id, kind, variant, name, config_json,
track_count, last_generated_at, last_synced_at,
last_generation_source, last_generation_error
FROM personalized_playlists
WHERE profile_id = ?
ORDER BY COALESCE(last_generated_at, created_at) DESC
""",
(profile_id,),
)
rows = cursor.fetchall()
return [self._row_to_record(r) for r in rows]
def update_config(self, kind: str, variant: str, profile_id: int, overrides: Dict[str, Any]) -> PlaylistRecord:
"""Patch the per-playlist config with the provided overrides."""
record = self.ensure_playlist(kind, variant, profile_id)
merged = record.config.merged(overrides)
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
UPDATE personalized_playlists
SET config_json = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""",
(json.dumps(merged.to_json_dict()), record.id),
)
conn.commit()
return self._fetch_playlist_row(kind, variant, profile_id) # type: ignore[return-value]
# ─── refresh / generation ─────────────────────────────────────────
def refresh_playlist(
self,
kind: str,
variant: str = '',
profile_id: int = 1,
config_overrides: Optional[Dict[str, Any]] = None,
) -> PlaylistRecord:
"""Run the kind's generator and persist the result as the
playlist's current snapshot.
Atomic: track list is replaced in a single transaction. On
generator exception, the previous snapshot is preserved and
the error is recorded on the row.
Args:
kind: registered kind identifier.
variant: e.g. '1980s' for time machine, '' for singletons.
profile_id: profile to refresh under.
config_overrides: optional per-call config tweaks merged on
top of the stored config (e.g. UI lets the user "preview
with limit=100" without persisting that change).
Returns:
Updated PlaylistRecord with fresh ``track_count`` /
``last_generated_at`` (or ``last_generation_error`` on
failure).
"""
spec = self.registry.get(kind)
if spec is None:
raise ValueError(f"Unknown playlist kind: {kind!r}")
record = self.ensure_playlist(kind, variant, profile_id)
config = record.config
if config_overrides:
config = config.merged(config_overrides)
try:
tracks = spec.generator(self.deps, variant, config)
except Exception as exc: # noqa: BLE001 — record + re-raise after persisting
logger.exception("Generator failed for kind=%s variant=%s: %s", kind, variant, exc)
self._record_generation_failure(record.id, str(exc))
return self._fetch_playlist_row(kind, variant, profile_id) # type: ignore[return-value]
return self._persist_snapshot(record.id, kind, profile_id, tracks)
# ─── track read ──────────────────────────────────────────────────
def get_playlist_tracks(self, playlist_id: int) -> List[Track]:
"""Return the persisted track list for a playlist row."""
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT spotify_track_id, itunes_track_id, deezer_track_id,
track_name, artist_name, album_name, album_cover_url,
duration_ms, popularity, track_data_json
FROM personalized_playlist_tracks
WHERE playlist_id = ?
ORDER BY position ASC
""",
(playlist_id,),
)
rows = cursor.fetchall()
return [self._row_to_track(r) for r in rows]
# ─── staleness history ───────────────────────────────────────────
def recent_track_ids(self, profile_id: int, kind: str, days: int) -> List[str]:
"""Return track IDs served by ``kind`` for ``profile_id`` in
the last ``days`` days. Generators query this when honoring
the ``exclude_recent_days`` config knob."""
if days <= 0:
return []
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT DISTINCT track_id
FROM personalized_track_history
WHERE profile_id = ?
AND kind = ?
AND served_at >= datetime('now', ?)
""",
(profile_id, kind, f'-{int(days)} days'),
)
return [r[0] for r in cursor.fetchall() if r[0]]
# ─── internal helpers ────────────────────────────────────────────
def _persist_snapshot(self, playlist_id: int, kind: str, profile_id: int, tracks: List[Track]) -> PlaylistRecord:
"""Atomic replace of a playlist's track list + history append."""
now = datetime.now(timezone.utc).isoformat(timespec='seconds')
primary_source = next(
(t.source for t in tracks if t.source), None,
)
with self.database._get_connection() as conn:
cursor = conn.cursor()
try:
cursor.execute("BEGIN")
cursor.execute(
"DELETE FROM personalized_playlist_tracks WHERE playlist_id = ?",
(playlist_id,),
)
for position, track in enumerate(tracks):
td = track.track_data_json
if td is not None and not isinstance(td, str):
td = json.dumps(td)
cursor.execute(
"""
INSERT INTO personalized_playlist_tracks
(playlist_id, position,
spotify_track_id, itunes_track_id, deezer_track_id,
track_name, artist_name, album_name, album_cover_url,
duration_ms, popularity, track_data_json)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
playlist_id, position,
track.spotify_track_id, track.itunes_track_id, track.deezer_track_id,
track.track_name, track.artist_name, track.album_name, track.album_cover_url,
int(track.duration_ms or 0), int(track.popularity or 0), td,
),
)
cursor.execute(
"""
UPDATE personalized_playlists
SET track_count = ?, last_generated_at = CURRENT_TIMESTAMP,
last_generation_source = ?, last_generation_error = NULL,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""",
(len(tracks), primary_source, playlist_id),
)
# History append — only for tracks with a primary ID,
# used by exclude_recent_days filter on next refresh.
for track in tracks:
tid = track.primary_id()
if not tid:
continue
cursor.execute(
"""
INSERT INTO personalized_track_history
(profile_id, kind, track_id, served_at)
VALUES (?, ?, ?, CURRENT_TIMESTAMP)
""",
(profile_id, kind, tid),
)
conn.commit()
except Exception:
conn.rollback()
raise
# Re-read the row so the returned record carries the fresh
# last_generated_at timestamp.
record = self._fetch_playlist_row_by_id(playlist_id)
if record is None:
raise RuntimeError(f"playlist row {playlist_id} disappeared mid-refresh")
return record
def _record_generation_failure(self, playlist_id: int, error_message: str) -> None:
"""Stamp the error on the row WITHOUT touching tracks."""
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
UPDATE personalized_playlists
SET last_generation_error = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""",
(error_message[:500], playlist_id),
)
conn.commit()
def _fetch_playlist_row(self, kind: str, variant: str, profile_id: int) -> Optional[PlaylistRecord]:
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT id, profile_id, kind, variant, name, config_json,
track_count, last_generated_at, last_synced_at,
last_generation_source, last_generation_error
FROM personalized_playlists
WHERE profile_id = ? AND kind = ? AND variant = ?
""",
(profile_id, kind, variant),
)
row = cursor.fetchone()
return self._row_to_record(row) if row else None
def _fetch_playlist_row_by_id(self, playlist_id: int) -> Optional[PlaylistRecord]:
with self.database._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT id, profile_id, kind, variant, name, config_json,
track_count, last_generated_at, last_synced_at,
last_generation_source, last_generation_error
FROM personalized_playlists
WHERE id = ?
""",
(playlist_id,),
)
row = cursor.fetchone()
return self._row_to_record(row) if row else None
@staticmethod
def _row_to_record(row: Any) -> PlaylistRecord:
# Tolerate sqlite3.Row + plain tuples.
if hasattr(row, 'keys'):
row = dict(row)
return PlaylistRecord(
id=row['id'], profile_id=row['profile_id'],
kind=row['kind'], variant=row['variant'] or '',
name=row['name'],
config=PlaylistConfig.from_json_dict(_safe_json_loads(row['config_json'])),
track_count=row['track_count'] or 0,
last_generated_at=row.get('last_generated_at'),
last_synced_at=row.get('last_synced_at'),
last_generation_source=row.get('last_generation_source'),
last_generation_error=row.get('last_generation_error'),
)
# Tuple form: positional access matches SELECT order above.
return PlaylistRecord(
id=row[0], profile_id=row[1],
kind=row[2], variant=row[3] or '',
name=row[4],
config=PlaylistConfig.from_json_dict(_safe_json_loads(row[5])),
track_count=row[6] or 0,
last_generated_at=row[7],
last_synced_at=row[8],
last_generation_source=row[9],
last_generation_error=row[10],
)
@staticmethod
def _row_to_track(row: Any) -> Track:
if hasattr(row, 'keys'):
row = dict(row)
return Track(
spotify_track_id=row.get('spotify_track_id'),
itunes_track_id=row.get('itunes_track_id'),
deezer_track_id=row.get('deezer_track_id'),
track_name=row.get('track_name', ''),
artist_name=row.get('artist_name', ''),
album_name=row.get('album_name') or '',
album_cover_url=row.get('album_cover_url'),
duration_ms=int(row.get('duration_ms') or 0),
popularity=int(row.get('popularity') or 0),
track_data_json=_safe_json_loads(row.get('track_data_json')),
)
return Track(
spotify_track_id=row[0], itunes_track_id=row[1], deezer_track_id=row[2],
track_name=row[3], artist_name=row[4], album_name=row[5] or '',
album_cover_url=row[6], duration_ms=int(row[7] or 0),
popularity=int(row[8] or 0),
track_data_json=_safe_json_loads(row[9]),
)
def _safe_json_loads(value: Any) -> Any:
"""Tolerant JSON parse — returns None / dict / passes through
non-string values. Avoids exceptions on bad rows so the manager
never breaks on a single corrupt record."""
if value is None:
return None
if not isinstance(value, str):
return value
if not value.strip():
return None
try:
return json.loads(value)
except (ValueError, TypeError):
return None
__all__ = ['PersonalizedPlaylistManager']

121
core/personalized/specs.py Normal file
View file

@ -0,0 +1,121 @@
"""Per-kind specifications for the personalized-playlist subsystem.
A ``PlaylistKindSpec`` declares everything the manager needs to know
about one playlist type:
- The kind identifier (stable string used in URLs / configs / DB).
- Human-readable display name template (with optional ``{variant}``
substitution).
- Whether the kind supports / requires variants and what valid
variants look like.
- The default user-tweakable config for this kind.
- A generator callable that produces a fresh track list given
``(deps, variant, config)``.
Generators live in ``core/personalized/generators/`` (added in
later commits as each kind is migrated). For commit 1 the registry
ships empty schema + manager land first; generators arrive
incrementally with their per-kind tests.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, List, Optional
from core.personalized.types import PlaylistConfig, Track
# Generator callable signature.
# Args:
# deps: opaque object the generator may need (DB, service handle,
# config_manager). Each generator declares what it pulls.
# variant: e.g. '1980s' for time machine, 'halloween' for seasonal.
# Empty string '' for singleton kinds.
# config: PlaylistConfig with the user's per-playlist overrides
# merged onto the kind's defaults.
# Returns:
# List[Track] — the fresh snapshot to persist as the playlist's
# current track list.
GeneratorFn = Callable[[Any, str, PlaylistConfig], List[Track]]
# Variant resolver: returns the list of currently-valid variant
# identifiers for a kind that supports multiple instances. Used by
# the manager when auto-creating playlist rows for newly available
# variants (e.g. a new decade, a new season). Singletons return [''].
VariantResolver = Callable[[Any], List[str]]
@dataclass
class PlaylistKindSpec:
"""Declaration of one playlist kind.
See module docstring for the contract.
"""
kind: str
name_template: str # e.g. 'Time Machine — {variant}', 'Hidden Gems'
description: str
default_config: PlaylistConfig
generator: GeneratorFn
variant_resolver: Optional[VariantResolver] = None
requires_variant: bool = False
# Tags for UI grouping ('curated' / 'discovery' / 'time' / 'genre').
tags: List[str] = field(default_factory=list)
def display_name(self, variant: str) -> str:
"""Render the human-readable playlist name for a given variant."""
if not variant:
return self.name_template.replace('{variant}', '').strip(' —-')
return self.name_template.format(variant=variant)
class PlaylistKindRegistry:
"""Module-level registry of every kind the manager knows about.
Populated at import time as each generator module is loaded. The
manager queries the registry at runtime to dispatch refresh
requests, list available kinds for the UI, and resolve variants.
"""
def __init__(self) -> None:
self._kinds: Dict[str, PlaylistKindSpec] = {}
def register(self, spec: PlaylistKindSpec) -> None:
if spec.kind in self._kinds:
raise ValueError(f"Kind {spec.kind!r} already registered")
self._kinds[spec.kind] = spec
def get(self, kind: str) -> Optional[PlaylistKindSpec]:
return self._kinds.get(kind)
def all(self) -> List[PlaylistKindSpec]:
return list(self._kinds.values())
def kinds(self) -> List[str]:
return list(self._kinds.keys())
def reset_for_tests(self) -> None:
"""Drop every registration. Tests only — production runs
register at module import and never reset."""
self._kinds.clear()
# Module-level singleton. Generators register against this on import.
_registry = PlaylistKindRegistry()
def get_registry() -> PlaylistKindRegistry:
"""Public accessor for the module-level registry. Tests can reset
via ``get_registry().reset_for_tests()``."""
return _registry
__all__ = [
'PlaylistKindSpec',
'PlaylistKindRegistry',
'GeneratorFn',
'VariantResolver',
'get_registry',
]

174
core/personalized/types.py Normal file
View file

@ -0,0 +1,174 @@
"""Shared dataclasses for the personalized-playlist subsystem.
These are pure data containers no business logic, no IO. The
manager + specs + generators all speak in these types so the seam
between them stays mechanical.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
@dataclass
class Track:
"""A track in a personalized playlist.
Mirrors the shape returned by
``PersonalizedPlaylistsService._build_track_dict`` so the legacy
generators can be wrapped without translating fields. Always at
least one of the source IDs is populated; ``track_data_json`` is
the full enriched track object when available (used by sync /
download paths that need richer metadata than just the ID)."""
track_name: str
artist_name: str
album_name: str = ''
spotify_track_id: Optional[str] = None
itunes_track_id: Optional[str] = None
deezer_track_id: Optional[str] = None
album_cover_url: Optional[str] = None
duration_ms: int = 0
popularity: int = 0
track_data_json: Optional[Any] = None # dict OR JSON string OR None
source: Optional[str] = None
@classmethod
def from_dict(cls, d: Dict[str, Any]) -> 'Track':
"""Coerce a legacy generator's output dict into a Track.
Tolerates the ``_artist_genres_raw`` / ``_release_date`` extra-
column passthroughs by ignoring them this dataclass only
carries the storage-layer fields."""
return cls(
track_name=d.get('track_name', 'Unknown'),
artist_name=d.get('artist_name', 'Unknown'),
album_name=d.get('album_name', '') or '',
spotify_track_id=d.get('spotify_track_id'),
itunes_track_id=d.get('itunes_track_id'),
deezer_track_id=d.get('deezer_track_id'),
album_cover_url=d.get('album_cover_url'),
duration_ms=int(d.get('duration_ms') or 0),
popularity=int(d.get('popularity') or 0),
track_data_json=d.get('track_data_json'),
source=d.get('source'),
)
def primary_id(self) -> Optional[str]:
"""Return the first non-empty source ID. Used as the staleness-
history key + the dedupe key when persisting tracks."""
return (
self.spotify_track_id
or self.itunes_track_id
or self.deezer_track_id
or None
)
@dataclass
class PlaylistConfig:
"""User-tweakable knobs per playlist instance.
Stored as JSON in `personalized_playlists.config_json`. Defaults
come from the kind's spec; user overrides override per-playlist.
Fields:
- limit: target number of tracks
- max_per_album / max_per_artist: diversity caps
- popularity_min / popularity_max: filter bounds (None = ignore)
- exclude_recent_days: avoid tracks served by this kind in the
last N days (0 = no exclusion)
- recency_days: only include tracks released in the last N days
(None = all-time)
- seed: optional deterministic seed for randomization (None =
use system random; same seed + same pool = same output)
- extra: free-form per-kind extension dict (e.g. seasonal mix
stores ``selected_seasons``, time machine stores
``selected_decades``, genre stores ``selected_genres``).
"""
limit: int = 50
max_per_album: int = 2
max_per_artist: int = 3
popularity_min: Optional[int] = None
popularity_max: Optional[int] = None
exclude_recent_days: int = 0
recency_days: Optional[int] = None
seed: Optional[int] = None
extra: Dict[str, Any] = field(default_factory=dict)
def to_json_dict(self) -> Dict[str, Any]:
"""Serialise to a JSON-safe dict for storage."""
return {
'limit': self.limit,
'max_per_album': self.max_per_album,
'max_per_artist': self.max_per_artist,
'popularity_min': self.popularity_min,
'popularity_max': self.popularity_max,
'exclude_recent_days': self.exclude_recent_days,
'recency_days': self.recency_days,
'seed': self.seed,
'extra': dict(self.extra),
}
@classmethod
def from_json_dict(cls, d: Optional[Dict[str, Any]]) -> 'PlaylistConfig':
"""Reconstruct from a stored JSON dict. Missing fields fall
back to defaults so old rows + new code stay compatible."""
if not isinstance(d, dict):
return cls()
return cls(
limit=int(d.get('limit', 50)),
max_per_album=int(d.get('max_per_album', 2)),
max_per_artist=int(d.get('max_per_artist', 3)),
popularity_min=d.get('popularity_min'),
popularity_max=d.get('popularity_max'),
exclude_recent_days=int(d.get('exclude_recent_days', 0)),
recency_days=d.get('recency_days'),
seed=d.get('seed'),
extra=dict(d.get('extra') or {}),
)
def merged(self, overrides: Dict[str, Any]) -> 'PlaylistConfig':
"""Return a new PlaylistConfig with `overrides` merged in.
Used when a user PATCHes their per-playlist config apply
only the fields they sent, leave the rest at their stored
values."""
base = self.to_json_dict()
for key, value in (overrides or {}).items():
if key == 'extra' and isinstance(value, dict):
base['extra'] = {**base.get('extra', {}), **value}
elif key in base:
base[key] = value
return PlaylistConfig.from_json_dict(base)
@dataclass
class PlaylistRecord:
"""One row of `personalized_playlists` plus its track count.
The live track list is fetched separately via
``PersonalizedPlaylistManager.get_playlist_tracks(playlist_id)``
so list / detail responses can stay cheap when the caller only
needs metadata."""
id: int
profile_id: int
kind: str
variant: str
name: str
config: PlaylistConfig
track_count: int
last_generated_at: Optional[str]
last_synced_at: Optional[str]
last_generation_source: Optional[str]
last_generation_error: Optional[str]
__all__ = [
'Track',
'PlaylistConfig',
'PlaylistRecord',
]

View file

@ -752,13 +752,21 @@ class MusicDatabase:
)
""")
# Personalized-playlists subsystem schema (Group A + Group B
# unified storage). Idempotent — safe on every startup.
try:
from database.personalized_schema import ensure_personalized_schema
ensure_personalized_schema(conn)
except Exception as ps_err:
logger.error(f"Personalized-playlist schema init failed: {ps_err}")
conn.commit()
logger.info("Database initialized successfully")
except Exception as e:
logger.error(f"Error initializing database: {e}")
raise
def _add_mirrored_playlist_explored_column(self, cursor):
"""Add explored_at column to mirrored_playlists to persist explore badge."""
try:

View file

@ -0,0 +1,139 @@
"""Schema + migration helpers for the personalized-playlists subsystem.
Pre-existing state (this PR replaces over time):
- Group A (Fresh Tape / The Archives / Seasonal Mix) lives in
`discovery_curated_playlists` (track_ids only) and
`curated_seasonal_playlists` (track_ids + seasonal_tracks join).
Read paths exist; refresh paths are tied to specific workers.
- Group B (Hidden Gems / Discovery Shuffle / Time Machine / Popular
Picks / Genre / Daily Mixes) is computed on-demand by
`PersonalizedPlaylistsService` no persistence, every call reruns
the generator with `ORDER BY RANDOM()` so the result rotates.
Post-overhaul (this module's responsibility):
- ALL personalized playlists land in a unified storage layer with a
stable (profile_id, kind, variant) identity, JSON config per
playlist (limit, diversity caps, popularity / recency filters,
exclude-recent-days, randomization seed), and a persistent track
list that only mutates when the playlist is explicitly refreshed.
Tables created here:
- ``personalized_playlists`` one row per (profile, kind, variant).
Variants disambiguate kinds with multiple instances:
* ``time_machine``: variant = ``'1980s'`` / ``'1990s'`` / ...
* ``seasonal_mix``: variant = ``'halloween'`` / ``'christmas'`` / ...
* ``genre_playlist``: variant = ``'rock'`` / ``'electronic_dance'`` / ...
* ``daily_mix``: variant = ``'1'`` / ``'2'`` / ``'3'`` / ``'4'``
* Singletons (``hidden_gems``, ``discovery_shuffle``,
``popular_picks``, ``fresh_tape``, ``archives``): variant = ``''``.
Variant '' (empty) is used instead of NULL so the UNIQUE
constraint behaves predictably (NULL doesn't collide with NULL in
SQLite UNIQUE indexes would let multiple singleton rows
coexist).
- ``personalized_playlist_tracks`` current snapshot per playlist.
Cleared + repopulated on refresh; never partial-mutates.
- ``personalized_track_history`` append-only log of which tracks
were served by which (profile, kind) over time. Powers the
``exclude_recent_days`` config option so generators can avoid
recommending the same track twice in N days.
The schema is created idempotently `ensure_personalized_schema`
runs CREATE TABLE IF NOT EXISTS at startup, so existing installs
upgrade silently."""
from __future__ import annotations
from typing import Any
from utils.logging_config import get_logger
logger = get_logger("database.personalized_schema")
PERSONALIZED_PLAYLISTS_DDL = """
CREATE TABLE IF NOT EXISTS personalized_playlists (
id INTEGER PRIMARY KEY AUTOINCREMENT,
profile_id INTEGER NOT NULL DEFAULT 1,
kind TEXT NOT NULL,
variant TEXT NOT NULL DEFAULT '',
name TEXT NOT NULL,
config_json TEXT NOT NULL DEFAULT '{}',
track_count INTEGER NOT NULL DEFAULT 0,
last_generated_at TIMESTAMP,
last_synced_at TIMESTAMP,
last_generation_source TEXT,
last_generation_error TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE (profile_id, kind, variant)
)
"""
PERSONALIZED_PLAYLIST_TRACKS_DDL = """
CREATE TABLE IF NOT EXISTS personalized_playlist_tracks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
playlist_id INTEGER NOT NULL,
position INTEGER NOT NULL,
spotify_track_id TEXT,
itunes_track_id TEXT,
deezer_track_id TEXT,
track_name TEXT,
artist_name TEXT,
album_name TEXT,
album_cover_url TEXT,
duration_ms INTEGER,
popularity INTEGER,
track_data_json TEXT,
FOREIGN KEY (playlist_id) REFERENCES personalized_playlists(id) ON DELETE CASCADE,
UNIQUE (playlist_id, position)
)
"""
PERSONALIZED_PLAYLIST_TRACKS_INDEX = """
CREATE INDEX IF NOT EXISTS idx_personalized_tracks_playlist
ON personalized_playlist_tracks(playlist_id)
"""
PERSONALIZED_TRACK_HISTORY_DDL = """
CREATE TABLE IF NOT EXISTS personalized_track_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
profile_id INTEGER NOT NULL DEFAULT 1,
kind TEXT NOT NULL,
track_id TEXT NOT NULL,
served_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
"""
PERSONALIZED_TRACK_HISTORY_INDEX = """
CREATE INDEX IF NOT EXISTS idx_personalized_history_lookup
ON personalized_track_history(profile_id, kind, track_id, served_at)
"""
def ensure_personalized_schema(connection: Any) -> None:
"""Create the personalized-playlist tables + indexes if missing.
Idempotent. Safe to call on every app startup. Caller is responsible
for committing the connection (we leave that to the caller so this
composes with other schema-init steps in one transaction).
"""
cursor = connection.cursor()
cursor.execute(PERSONALIZED_PLAYLISTS_DDL)
cursor.execute(PERSONALIZED_PLAYLIST_TRACKS_DDL)
cursor.execute(PERSONALIZED_PLAYLIST_TRACKS_INDEX)
cursor.execute(PERSONALIZED_TRACK_HISTORY_DDL)
cursor.execute(PERSONALIZED_TRACK_HISTORY_INDEX)
logger.debug("Personalized-playlist schema ensured")
__all__ = [
'ensure_personalized_schema',
'PERSONALIZED_PLAYLISTS_DDL',
'PERSONALIZED_PLAYLIST_TRACKS_DDL',
'PERSONALIZED_PLAYLIST_TRACKS_INDEX',
'PERSONALIZED_TRACK_HISTORY_DDL',
'PERSONALIZED_TRACK_HISTORY_INDEX',
]

View file

@ -0,0 +1,421 @@
"""Boundary tests for the personalized-playlists foundation
(``core.personalized.types`` + ``core.personalized.specs`` +
``core.personalized.manager``).
Pin every shape the storage layer + lifecycle has to handle so the
generators that arrive in subsequent commits can rely on a stable
contract: ensure_playlist auto-creates from default config, refresh
atomically replaces the snapshot + appends history, generator
exceptions don't lose the previous good snapshot, config patches
preserve unsent fields, recent_track_ids honors the day window,
list_playlists orders newest-first."""
from __future__ import annotations
import os
import sqlite3
import tempfile
from typing import Any, List
from unittest.mock import MagicMock
import pytest
from core.personalized.manager import PersonalizedPlaylistManager
from core.personalized.specs import PlaylistKindRegistry, PlaylistKindSpec
from core.personalized.types import PlaylistConfig, Track
from database.personalized_schema import ensure_personalized_schema
# ─── shared fixtures ─────────────────────────────────────────────────
class _FakeDB:
"""Minimal MusicDatabase stand-in — gives the manager a real
sqlite connection so the manager exercises actual SQL."""
def __init__(self, path: str):
self.path = path
def _get_connection(self):
conn = sqlite3.connect(self.path)
conn.row_factory = sqlite3.Row
return conn
@pytest.fixture
def db_path(tmp_path):
p = str(tmp_path / 'test.db')
conn = sqlite3.connect(p)
ensure_personalized_schema(conn)
conn.commit()
conn.close()
return p
@pytest.fixture
def db(db_path):
return _FakeDB(db_path)
@pytest.fixture
def registry():
r = PlaylistKindRegistry()
return r
def _make_track(name='T1', artist='A1', sid='spot-1', source='spotify') -> Track:
return Track(
track_name=name, artist_name=artist, album_name='Album',
spotify_track_id=sid, source=source,
duration_ms=200000, popularity=50,
)
# ─── PlaylistConfig ──────────────────────────────────────────────────
class TestPlaylistConfig:
def test_default_values(self):
c = PlaylistConfig()
assert c.limit == 50
assert c.max_per_album == 2
assert c.max_per_artist == 3
assert c.popularity_min is None
assert c.popularity_max is None
assert c.exclude_recent_days == 0
assert c.recency_days is None
assert c.seed is None
assert c.extra == {}
def test_round_trip_through_json_dict(self):
c = PlaylistConfig(
limit=100, max_per_album=5, max_per_artist=10,
popularity_min=20, popularity_max=80,
exclude_recent_days=14, recency_days=180,
seed=42, extra={'selected_seasons': ['halloween', 'christmas']},
)
d = c.to_json_dict()
c2 = PlaylistConfig.from_json_dict(d)
assert c2 == c
def test_from_json_dict_handles_none(self):
c = PlaylistConfig.from_json_dict(None)
assert c == PlaylistConfig()
def test_from_json_dict_handles_non_dict(self):
c = PlaylistConfig.from_json_dict('garbage') # type: ignore
assert c == PlaylistConfig()
def test_from_json_dict_missing_fields_use_defaults(self):
c = PlaylistConfig.from_json_dict({'limit': 75})
assert c.limit == 75
assert c.max_per_album == 2 # default
def test_merged_overrides_only_named_fields(self):
base = PlaylistConfig(limit=50, popularity_min=20)
out = base.merged({'limit': 100})
assert out.limit == 100
assert out.popularity_min == 20 # untouched
def test_merged_extra_dict_is_deep_merged(self):
base = PlaylistConfig(extra={'a': 1, 'b': 2})
out = base.merged({'extra': {'b': 99, 'c': 3}})
assert out.extra == {'a': 1, 'b': 99, 'c': 3}
def test_merged_ignores_unknown_keys(self):
base = PlaylistConfig()
out = base.merged({'unknown_field': 'foo'})
assert out == base
# ─── Track ────────────────────────────────────────────────────────────
class TestTrack:
def test_from_dict_legacy_shape(self):
d = {
'track_name': 'Song', 'artist_name': 'Band',
'album_name': 'Album', 'spotify_track_id': 'spot-1',
'duration_ms': 200000, 'popularity': 60,
'_artist_genres_raw': '["rock"]', # ignored extra
}
t = Track.from_dict(d)
assert t.track_name == 'Song'
assert t.spotify_track_id == 'spot-1'
assert t.duration_ms == 200000
def test_primary_id_prefers_spotify(self):
t = Track(
track_name='', artist_name='',
spotify_track_id='spot', itunes_track_id='itu', deezer_track_id='dee',
)
assert t.primary_id() == 'spot'
def test_primary_id_falls_back_through_sources(self):
t = Track(track_name='', artist_name='', itunes_track_id='itu')
assert t.primary_id() == 'itu'
t2 = Track(track_name='', artist_name='', deezer_track_id='dee')
assert t2.primary_id() == 'dee'
def test_primary_id_none_when_no_sources(self):
t = Track(track_name='', artist_name='')
assert t.primary_id() is None
# ─── PlaylistKindRegistry ────────────────────────────────────────────
class TestRegistry:
def test_register_and_get(self, registry):
spec = PlaylistKindSpec(
kind='hidden_gems', name_template='Hidden Gems',
description='', default_config=PlaylistConfig(),
generator=lambda *a, **k: [],
)
registry.register(spec)
assert registry.get('hidden_gems') is spec
assert registry.get('nonexistent') is None
def test_duplicate_registration_raises(self, registry):
spec = PlaylistKindSpec(
kind='x', name_template='X', description='',
default_config=PlaylistConfig(), generator=lambda *a, **k: [],
)
registry.register(spec)
with pytest.raises(ValueError, match='already registered'):
registry.register(spec)
def test_display_name_singleton(self):
spec = PlaylistKindSpec(
kind='x', name_template='Hidden Gems', description='',
default_config=PlaylistConfig(), generator=lambda *a, **k: [],
)
assert spec.display_name('') == 'Hidden Gems'
def test_display_name_with_variant(self):
spec = PlaylistKindSpec(
kind='x', name_template='Time Machine — {variant}',
description='', default_config=PlaylistConfig(),
generator=lambda *a, **k: [],
)
assert spec.display_name('1980s') == 'Time Machine — 1980s'
def test_kinds_listing(self, registry):
for k in ('a', 'b', 'c'):
registry.register(PlaylistKindSpec(
kind=k, name_template=k, description='',
default_config=PlaylistConfig(), generator=lambda *a, **k: [],
))
assert set(registry.kinds()) == {'a', 'b', 'c'}
# ─── PersonalizedPlaylistManager ─────────────────────────────────────
def _register_simple_kind(registry, generator, kind='hidden_gems', requires_variant=False):
spec = PlaylistKindSpec(
kind=kind, name_template=kind.replace('_', ' ').title(),
description='', default_config=PlaylistConfig(limit=10),
generator=generator, requires_variant=requires_variant,
)
registry.register(spec)
return spec
class TestEnsurePlaylist:
def test_creates_row_with_default_config(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
record = mgr.ensure_playlist('hidden_gems', '', 1)
assert record.id > 0
assert record.kind == 'hidden_gems'
assert record.variant == ''
assert record.profile_id == 1
assert record.config.limit == 10 # from default
assert record.track_count == 0
assert record.last_generated_at is None
def test_returns_same_row_on_second_call(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
r1 = mgr.ensure_playlist('hidden_gems', '', 1)
r2 = mgr.ensure_playlist('hidden_gems', '', 1)
assert r1.id == r2.id
def test_variant_creates_separate_row(self, db, registry):
_register_simple_kind(
registry, lambda *a, **k: [], kind='time_machine', requires_variant=True,
)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
r1 = mgr.ensure_playlist('time_machine', '1980s', 1)
r2 = mgr.ensure_playlist('time_machine', '1990s', 1)
assert r1.id != r2.id
def test_unknown_kind_raises(self, db, registry):
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
with pytest.raises(ValueError, match='Unknown playlist kind'):
mgr.ensure_playlist('does_not_exist', '', 1)
def test_required_variant_missing_raises(self, db, registry):
_register_simple_kind(
registry, lambda *a, **k: [], kind='time_machine', requires_variant=True,
)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
with pytest.raises(ValueError, match='requires a variant'):
mgr.ensure_playlist('time_machine', '', 1)
class TestRefreshPlaylist:
def test_refresh_persists_tracks(self, db, registry):
tracks = [_make_track('S1', 'A1', 'sp1'), _make_track('S2', 'A1', 'sp2')]
_register_simple_kind(registry, lambda deps, variant, config: tracks)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
record = mgr.refresh_playlist('hidden_gems', '', 1)
assert record.track_count == 2
assert record.last_generated_at is not None
assert record.last_generation_error is None
persisted = mgr.get_playlist_tracks(record.id)
assert len(persisted) == 2
assert persisted[0].track_name == 'S1'
assert persisted[1].track_name == 'S2'
def test_refresh_replaces_previous_snapshot_atomically(self, db, registry):
run = {'tracks': [_make_track('first')]}
def gen(deps, variant, config):
return run['tracks']
_register_simple_kind(registry, gen)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
r1 = mgr.refresh_playlist('hidden_gems', '', 1)
assert r1.track_count == 1
run['tracks'] = [_make_track('A'), _make_track('B'), _make_track('C')]
r2 = mgr.refresh_playlist('hidden_gems', '', 1)
assert r2.id == r1.id
assert r2.track_count == 3
persisted = mgr.get_playlist_tracks(r2.id)
assert [t.track_name for t in persisted] == ['A', 'B', 'C']
def test_generator_exception_preserves_previous_snapshot(self, db, registry):
run = {'mode': 'success'}
def gen(deps, variant, config):
if run['mode'] == 'fail':
raise RuntimeError('generator boom')
return [_make_track('first')]
_register_simple_kind(registry, gen)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
r1 = mgr.refresh_playlist('hidden_gems', '', 1)
assert r1.track_count == 1
run['mode'] = 'fail'
r2 = mgr.refresh_playlist('hidden_gems', '', 1)
# Previous snapshot preserved.
assert r2.track_count == 1
# Error stamped on row.
assert r2.last_generation_error is not None
assert 'generator boom' in r2.last_generation_error
# Tracks still queryable.
persisted = mgr.get_playlist_tracks(r2.id)
assert len(persisted) == 1
def test_config_overrides_passed_to_generator(self, db, registry):
captured = {}
def gen(deps, variant, config):
captured['limit'] = config.limit
return []
_register_simple_kind(registry, gen)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.refresh_playlist('hidden_gems', '', 1, config_overrides={'limit': 200})
assert captured['limit'] == 200
def test_refresh_records_source_from_first_track(self, db, registry):
tracks = [_make_track(source='spotify'), _make_track(source='deezer')]
_register_simple_kind(registry, lambda *a, **k: tracks)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
record = mgr.refresh_playlist('hidden_gems', '', 1)
assert record.last_generation_source == 'spotify'
def test_track_data_json_round_trips(self, db, registry):
nested = {'id': 'spot-1', 'name': 'Foo', 'artists': [{'name': 'Bar'}]}
track = Track(
track_name='Foo', artist_name='Bar',
spotify_track_id='spot-1', track_data_json=nested,
)
_register_simple_kind(registry, lambda *a, **k: [track])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
record = mgr.refresh_playlist('hidden_gems', '', 1)
persisted = mgr.get_playlist_tracks(record.id)
assert persisted[0].track_data_json == nested
class TestUpdateConfig:
def test_patch_merges_with_stored(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.ensure_playlist('hidden_gems', '', 1)
record = mgr.update_config('hidden_gems', '', 1, {'limit': 75})
assert record.config.limit == 75
# Other fields kept.
assert record.config.max_per_album == 2
def test_patch_extra_dict_deep_merges(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.ensure_playlist('hidden_gems', '', 1)
mgr.update_config('hidden_gems', '', 1, {'extra': {'a': 1}})
record = mgr.update_config('hidden_gems', '', 1, {'extra': {'b': 2}})
assert record.config.extra == {'a': 1, 'b': 2}
class TestListPlaylists:
def test_lists_all_playlists_for_profile(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [], kind='hidden_gems')
_register_simple_kind(registry, lambda *a, **k: [], kind='popular_picks')
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.ensure_playlist('hidden_gems', '', 1)
mgr.ensure_playlist('popular_picks', '', 1)
records = mgr.list_playlists(1)
kinds = {r.kind for r in records}
assert kinds == {'hidden_gems', 'popular_picks'}
def test_does_not_list_other_profiles(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.ensure_playlist('hidden_gems', '', 1)
mgr.ensure_playlist('hidden_gems', '', 2)
assert len(mgr.list_playlists(1)) == 1
assert len(mgr.list_playlists(2)) == 1
class TestStalenessHistory:
def test_recent_track_ids_returns_zero_when_days_zero(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [_make_track(sid='spot-1')])
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.refresh_playlist('hidden_gems', '', 1)
assert mgr.recent_track_ids(1, 'hidden_gems', 0) == []
def test_recent_track_ids_after_refresh(self, db, registry):
_register_simple_kind(
registry,
lambda *a, **k: [_make_track(sid='spot-1'), _make_track(sid='spot-2')],
)
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.refresh_playlist('hidden_gems', '', 1)
recent = mgr.recent_track_ids(1, 'hidden_gems', 7)
assert set(recent) == {'spot-1', 'spot-2'}
def test_recent_track_ids_scoped_to_kind(self, db, registry):
_register_simple_kind(registry, lambda *a, **k: [_make_track(sid='gem-1')], kind='hidden_gems')
_register_simple_kind(registry, lambda *a, **k: [_make_track(sid='pop-1')], kind='popular_picks')
mgr = PersonalizedPlaylistManager(db, deps=None, registry=registry)
mgr.refresh_playlist('hidden_gems', '', 1)
mgr.refresh_playlist('popular_picks', '', 1)
assert mgr.recent_track_ids(1, 'hidden_gems', 7) == ['gem-1']
assert mgr.recent_track_ids(1, 'popular_picks', 7) == ['pop-1']