Merge pull request #784 from Nezreka/dev

Dev
This commit is contained in:
BoulderBadgeDad 2026-06-03 22:42:13 -07:00 committed by GitHub
commit 1c1d2eb884
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
118 changed files with 13577 additions and 1272 deletions

View file

@ -105,6 +105,21 @@ SoulSync bridges streaming services to your music library with automated discove
- Catches wrong versions (live, remix, cover) even from streaming API sources
- Fail-open design: verification errors never block downloads
#### AcoustID API key
AcoustID verification is opt-in. To enable it, request a free API key
at <https://acoustid.org/new-application> and paste it into
Settings → AcoustID. Without a key, downloads still complete but the
verification step is skipped silently.
If a track was previously tagged by AcoustID but the retag action in
the AcoustID Scanner no longer changes anything, see issue #704 — the
most common cause is that the file already carries a
`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
and therefore never overwrites. Removing the cached
`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
restores the retag.
### Metadata & Enrichment
**10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz

View file

@ -0,0 +1,144 @@
# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
**Status:** design only — no code yet.
**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
the user's actual files, so the Library Reorganizer, Track Number Repair, and
tagging/enrichment all agree on the same release. Today each re-resolves
independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
Believer=3; standard album mislinked to a deluxe release).
**Canonical-selection rule (decided):** *match the user's actual files.* The
canonical release is the candidate whose track count + per-track durations +
titles best fit what's on disk. Self-correcting: picks standard when you own the
standard, deluxe when you own the deluxe.
---
## Hard requirement: don't disrupt the running app
Every stage below is **additive and dormant until explicitly consumed**, and
every consumer **falls back to today's behavior when no canonical is set**. So:
- albums with no resolved canonical behave EXACTLY as they do now;
- each stage is independently shippable and reversible;
- nothing big-bangs.
---
## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
### Schema (additive, nullable → migration-safe)
Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
existing column-exists checks; see [[db-schema-review]] migration-safety notes):
- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
- `canonical_album_id TEXT`
- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
- `canonical_resolved_at TIMESTAMP`
All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
backfill in this stage. No reads in this stage.
### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
```
score_release_against_files(file_tracks, release_tracks) -> float
pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
```
- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
- `release_tracks`: a candidate release's tracklist (same shape).
- Scoring (tunable weights):
- **track-count fit** — exact match strongly preferred; |Δcount| penalized.
- **duration alignment** — greedily match each file to its closest release
track by duration (within a tolerance, e.g. ±3s); reward coverage.
- **title overlap** — token/fuzzy overlap as a tiebreaker.
- **graceful degradation** — if a source gives no per-track durations, fall
back to count + title only (never crash, never force-pick).
- Returns the best candidate + score, or (None, 0) when nothing clears a floor
(so we never pin a bad guess — leave it unresolved, consumers fall back).
### Tests (extreme, like the rest of this codebase)
- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
- same album, 17 files → picks deluxe.
- duration disambiguation when track counts tie (e.g. radio edit vs album).
- missing-duration source → count+title fallback still picks sanely.
- no candidate clears the floor → (None, 0).
- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
whichever the files actually match.
**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
them yet. Provably zero behavior change.**
---
## Stage 2 — Resolver populates canonical (writes, still no consumers)
A function `resolve_canonical_for_album(album_id, db, ...)`:
1. Gather on-disk file metadata for the album (durations/titles) via the
library's known file paths.
2. Gather candidate releases: every source the album has an ID for
(spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
case — sibling editions discoverable from those. Fetch each tracklist
(cached, rate-limited).
3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
on the album row if it clears the floor.
Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
unchanged — this stage only populates the new columns. Reversible: clearing the
columns reverts to unresolved.
Tests: resolver picks the right release for the standard/deluxe fixtures; stores
nothing when below floor; idempotent re-resolve.
Cost note: fetching multiple candidate releases = more API calls. Mitigate via
cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
progress so it's not silent.
---
## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
In `library_reorganize._resolve_source`: if the album has
`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
current `get_source_priority` walk. One-line precedence change, fully gated on
non-NULL.
Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
to today. Re-run the existing reorganize battery (148 tests) — must stay green.
**This alone fixes #767-Bug2** (a standard album whose files match the standard
release pins the standard, so reorganize stops targeting the deluxe folder).
---
## Stage 4 — Track Number Repair reads canonical (closes #765)
In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
everything) — if the album has a canonical `(source, album_id)`, use it. The
existing 6-level cascade stays as the fallback for albums with no canonical
(preserves its all-01-album rescue ability — the regression risk we refused to
take in the reactive fix).
Now both tools resolve the SAME release → same track numbers → no contradiction.
Tests: canonical present → both tools agree (shared-release test); canonical
NULL → existing cascade unchanged.
---
## Risks & mitigations
- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
only-on-(re)enrich, progress-logged.
- **Sources without per-track durations** → scorer degrades to count+title.
- **Schema migration** → additive nullable columns only; idempotent guards.
- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
stored for inspection/re-resolve; manual override possible later.
- **Backward-compat** → every consumer falls back to today's path when NULL, so
un-resolved albums (incl. all existing albums until backfilled) are unaffected.
## Out of scope (for now)
- Per-album manual version override UI (can layer on later — the columns support it).
- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
once they share the canonical.
## Suggested order to build
1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
after any stage and the app is consistent (just with fewer consumers wired).

View file

@ -174,6 +174,16 @@ class AmazonWorker:
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('amazon')
if _prio:
_pi = priority_pending_item(cursor, 'amazon', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists

View file

@ -162,6 +162,16 @@ class AudioDBWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('audiodb')
if _prio:
_pi = priority_pending_item(cursor, 'audiodb', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -28,7 +28,7 @@ from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Any, Callable, Optional
from typing import Any, Callable, Dict, Optional
@dataclass
@ -105,6 +105,8 @@ class AutomationDeps:
# --- Playlist pipeline entry points ---
run_playlist_discovery_worker: Callable[..., Any]
run_sync_task: Callable[..., Any]
run_playlist_organize_download: Callable[..., Dict[str, Any]]
missing_download_executor: Any
load_sync_status_file: Callable[[], dict]
get_deezer_client: Callable[[], Any]
parse_youtube_playlist: Callable[[str], Any]

View file

@ -148,9 +148,45 @@ def run_sync_and_wishlist(
log_type='success' if sync_errors == 0 else 'warning',
)
organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
organize_started = 0
if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
for pl in organize_playlists:
pl_id = pl.get('id')
if not pl_id:
continue
pl_name = pl.get('name', '')
try:
org_result = deps.run_playlist_organize_download(
mirrored_playlist_id=int(pl_id),
automation_id=automation_id,
)
if org_result.get('status') == 'started':
organize_started += 1
deps.update_progress(
automation_id,
log_line=f'Organize download started for "{pl_name}"',
log_type='success',
)
elif org_result.get('status') == 'skipped':
deps.update_progress(
automation_id,
log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
log_type='skip',
)
except Exception as org_err: # noqa: BLE001
deps.update_progress(
automation_id,
log_line=f'Organize download error for "{pl_name}": {org_err}',
log_type='warning',
)
all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
effective_skip_wishlist = skip_wishlist or all_organize
wishlist_queued = run_wishlist_phase(
deps, automation_id,
skip=skip_wishlist,
skip=effective_skip_wishlist,
progress_pct=progress_end + 1,
wishlist_phase_label=wishlist_phase_label,
wishlist_phase_start_log=wishlist_phase_start_log,
@ -161,6 +197,7 @@ def run_sync_and_wishlist(
'skipped': total_skipped,
'errors': sync_errors,
'wishlist_queued': wishlist_queued,
'organize_downloads_started': organize_started,
}

View file

@ -21,12 +21,49 @@ from typing import Any, Dict
from core.automation.deps import AutomationDeps
_TIMEOUT_SECONDS = 7200 # 2 hours — covers the worst large-library case
_STALL_WARNING_SECONDS = 600 # 10 minutes without progress = stall
# Time out on STALL (no progress), not total runtime: a large library can scan
# for many hours while progressing fine — a hard total cap would falsely mark a
# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
# up when progress hasn't moved for a long stretch, with a generous absolute
# backstop against a truly stuck monitor loop.
_STALL_WARNING_SECONDS = 600 # warn after 10 min with no progress (repeats)
_STALL_TIMEOUT_SECONDS = 1800 # 30 min with no progress at all = genuinely stalled
_ABSOLUTE_CAP_SECONDS = 86400 # 24h hard backstop (runaway-loop guard only)
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def scan_wait_action(
*,
status: str,
idle_seconds: float,
total_seconds: float,
stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
stall_warn_s: float = _STALL_WARNING_SECONDS,
abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
) -> str:
"""Decide what the monitor loop should do on a poll tick (pure/testable).
``idle_seconds`` is time since progress last changed; ``total_seconds`` is
time since the wait began. Returns one of:
``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
too long give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
(stalled long enough to warn but not give up), or ``'continue'``.
Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
it never hits ``stall_timeout`` no matter how long the whole scan takes.
"""
if status != 'running':
return 'finished'
if total_seconds >= abs_cap_s:
return 'abs_timeout'
if idle_seconds >= stall_timeout_s:
return 'stall_timeout'
if idle_seconds >= stall_warn_s:
return 'warn'
return 'continue'
def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a full or incremental DB update via ``run_db_update_task``."""
return _run_with_progress(
@ -36,7 +73,7 @@ def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) ->
initial_phase='Initializing...',
stall_label='Database update',
finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
timeout_label='Database update timed out after 2 hours',
timeout_label='Database update timed out after 24 hours',
)
@ -49,7 +86,7 @@ def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict
initial_phase='Deep scan: Initializing...',
stall_label='Deep scan',
finished_extras=lambda: {},
timeout_label='Deep scan timed out after 2 hours',
timeout_label='Deep scan timed out after 24 hours',
)
@ -83,30 +120,54 @@ def _run_with_progress(
deps.db_update_executor.submit(task, *task_args)
# Monitor progress (callbacks handle card updates, we just block until done).
# We time out on STALL, not total runtime: ``processed`` advances on every
# artist, so an actively-progressing scan keeps resetting the idle clock and
# is never falsely failed no matter how long the whole library takes.
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
last_progress_time = time.time()
last_progress_val = 0
while time.time() - poll_start < _TIMEOUT_SECONDS:
# Any of these advancing means the scan is alive. current_item (the artist
# being processed) changes every artist even when the rounded progress %
# holds steady, so it guards against a false stall during slow stretches.
last_progress_val = (0, 0, '')
last_warn_time = 0.0
outcome = 'finished'
while True:
time.sleep(_POLL_INTERVAL_SECONDS)
now = time.time()
with deps.db_update_lock:
current_status = state.get('status', 'idle')
current_progress = state.get('progress', 0)
if current_status != 'running':
current_val = (state.get('processed', 0), state.get('progress', 0),
state.get('current_item', ''))
if current_val != last_progress_val:
last_progress_val = current_val
last_progress_time = now
action = scan_wait_action(
status=current_status,
idle_seconds=now - last_progress_time,
total_seconds=now - poll_start,
)
if action in ('finished', 'stall_timeout', 'abs_timeout'):
outcome = action
break
# Stall detection — if no progress change in 10 minutes, warn.
if current_progress != last_progress_val:
last_progress_val = current_progress
last_progress_time = time.time()
elif time.time() - last_progress_time > _STALL_WARNING_SECONDS:
if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
idle_min = int((now - last_progress_time) / 60)
deps.update_progress(
automation_id,
log_line=f'{stall_label} appears stalled — waiting...',
log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
log_type='warning',
)
last_progress_time = time.time() # Reset so warning repeats every 10 min.
else:
# 2-hour timeout reached.
last_warn_time = now
if outcome == 'stall_timeout':
deps.update_progress(
automation_id, status='error', phase='Stalled',
log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
log_type='error',
)
return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
if outcome == 'abs_timeout':
deps.update_progress(
automation_id, status='error',
phase='Timed out', log_line=timeout_label, log_type='error',

View file

@ -145,13 +145,36 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
sync_id_key = f"auto_mirror_{playlist_id}"
# Full mirror identity (every source_track_id on the playlist). tracks_hash
# only covers tracks_json — if a new mirror row is skipped (no discovery /
# no source id), tracks_hash stays identical to the pre-add sync and we
# used to no-op with "unchanged" while the new song never hit wishlist.
mirror_ids_str = ','.join(
sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
)
mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
event_data = config.get('_event_data') or {}
try:
tracks_added = int(event_data.get('added') or 0)
except (TypeError, ValueError):
tracks_added = 0
force_sync = tracks_added > 0 or skipped_count > 0
try:
sync_statuses = deps.load_sync_status_file()
last_status = sync_statuses.get(sync_id_key, {})
last_hash = last_status.get('tracks_hash', '')
last_mirror_hash = last_status.get('mirror_tracks_hash', '')
last_matched = last_status.get('matched_tracks', -1)
if last_hash == tracks_hash and last_matched >= len(tracks_json):
mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
if (
not force_sync
and not mirror_changed
and last_hash == tracks_hash
and last_matched >= len(tracks_json)
):
# Exact same tracks, all matched last time — nothing to do.
deps.update_progress(
auto_id,
@ -162,6 +185,21 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
'status': 'skipped',
'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
}
if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
deps.update_progress(
auto_id,
log_line=(
f'Forcing sync: playlist changed ({tracks_added} added) or '
f'{skipped_count} track(s) need discovery'
),
log_type='info',
)
elif mirror_changed:
deps.update_progress(
auto_id,
log_line='Mirror track list changed — running sync',
log_type='info',
)
except Exception as e:
deps.logger.debug("mirror sync last-status read: %s", e)
@ -180,9 +218,11 @@ def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str
log_line=f'Starting sync: {len(tracks_json)} tracks',
log_type='success',
)
skip_wishlist_add = bool(pl.get('organize_by_playlist'))
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, pl['name'], tracks_json, auto_id, 1, pl.get('image_url', '')),
kwargs={'skip_wishlist_add': skip_wishlist_add},
daemon=True,
name=f'auto-sync-{playlist_id}',
).start()

View file

@ -163,6 +163,16 @@ class DeezerWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('deezer')
if _prio:
_pi = priority_pending_item(cursor, 'deezer', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -174,6 +174,16 @@ class DiscogsWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Discogs
# has no track endpoint, so only artist/album are honored.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('discogs')
if _prio in ('artist', 'album'):
_pi = priority_pending_item(cursor, 'discogs', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists

View file

@ -47,6 +47,90 @@ class SyncDeps:
update_and_save_sync_status: Callable
sync_states: dict
sync_lock: Any # threading.Lock
# Optional: post-sync download follow-up for mirrored-playlist automations.
process_wishlist_automatically: Callable[..., Any] | None = None
run_playlist_organize_download: Callable[..., Any] | None = None
is_wishlist_actually_processing: Callable[[], bool] | None = None
def _post_sync_automation_followup(
deps: SyncDeps,
*,
automation_id: str,
playlist_id: str,
skip_wishlist_add: bool,
result: Any,
) -> None:
"""Queue downloads after an automation sync finishes.
Sync Playlist runs in a background thread and returns immediately, so a
separate scheduled "Process Wishlist" action often runs on an empty wishlist.
Organize-by-playlist skips sync-time wishlist adds and expects a folder
download batch instead that only ran in Playlist Pipeline before this hook.
"""
if not automation_id or not str(playlist_id).startswith('auto_mirror_'):
return
try:
mirrored_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
except ValueError:
return
failed = int(getattr(result, 'failed_tracks', 0) or 0)
wishlist_added = int(getattr(result, 'wishlist_added_count', 0) or 0)
if skip_wishlist_add:
org_fn = deps.run_playlist_organize_download
if failed <= 0:
return
if not org_fn:
logger.warning(
"Organize-by-playlist sync left %s missing tracks but organize download is unavailable",
failed,
)
deps.update_automation_progress(
automation_id,
log_line=f'{failed} missing — enable Playlist Pipeline or disable Organize by Playlist',
log_type='warning',
)
return
org_result = org_fn(mirrored_playlist_id=mirrored_id, automation_id=automation_id)
status = org_result.get('status', 'unknown') if isinstance(org_result, dict) else 'unknown'
reason = org_result.get('reason', '') if isinstance(org_result, dict) else ''
log_type = 'success' if status == 'started' else 'warning'
detail = f' ({reason})' if reason and status != 'started' else ''
deps.update_automation_progress(
automation_id,
log_line=f'Organize download {status} for {failed} missing track(s){detail}',
log_type=log_type,
)
return
if wishlist_added <= 0:
if failed > 0:
deps.update_automation_progress(
automation_id,
log_line=f'{failed} missing but none added to wishlist — check logs',
log_type='warning',
)
return
proc_fn = deps.process_wishlist_automatically
if not proc_fn:
return
is_busy = deps.is_wishlist_actually_processing
if is_busy and is_busy():
deps.update_automation_progress(
automation_id,
log_line=f'Added {wishlist_added} to wishlist; download worker already running',
log_type='info',
)
return
proc_fn(automation_id=automation_id)
deps.update_automation_progress(
automation_id,
log_line=f'Started wishlist download for {wishlist_added} track(s)',
log_type='success',
)
async def _database_only_find_track(spotify_track, candidate_pool=None):
@ -133,7 +217,17 @@ async def _database_only_find_track(spotify_track, candidate_pool=None):
return None, 0.0
def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, profile_id=1, playlist_image_url='', deps: SyncDeps = None, sync_mode: str = 'replace'):
def run_sync_task(
playlist_id,
playlist_name,
tracks_json,
automation_id=None,
profile_id=1,
playlist_image_url='',
deps: SyncDeps = None,
sync_mode: str = 'replace',
skip_wishlist_add: bool = False,
):
"""The actual sync function that runs in the background thread."""
sync_states = deps.sync_states
sync_lock = deps.sync_lock
@ -360,7 +454,14 @@ def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, p
# Wing It mode — skip wishlist for unmatched tracks
with sync_lock:
is_wing_it = sync_states.get(playlist_id, {}).get('wing_it', False)
sync_service._skip_unmatched_wishlist = is_wing_it or skip_wishlist_add
sync_service._skip_wishlist = is_wing_it
if skip_wishlist_add:
logger.info(
"[Organize by Playlist] Skipping sync-time wishlist for '%s'"
"organize download + batch failure handling cover missing tracks",
playlist_name,
)
# Run the sync (this is a blocking call within this thread)
result = deps.run_async(sync_service.sync_playlist(playlist, download_missing=False, profile_id=profile_id, sync_mode=sync_mode))
@ -459,9 +560,21 @@ def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, p
matched = getattr(result, 'matched_tracks', 0)
total = getattr(result, 'total_tracks', 0)
failed = getattr(result, 'failed_tracks', 0)
wishlist_added = getattr(result, 'wishlist_added_count', 0) or 0
deps.update_automation_progress(automation_id, status='finished', progress=100,
phase='Sync complete',
log_line=f'Done: {matched}/{total} matched, {failed} failed', log_type='success')
log_line=(
f'Done: {matched}/{total} in library, {failed} missing'
+ (f', {wishlist_added} added to wishlist' if wishlist_added else '')
),
log_type='success')
_post_sync_automation_followup(
deps,
automation_id=automation_id,
playlist_id=playlist_id,
skip_wishlist_add=skip_wishlist_add,
result=result,
)
# Emit playlist_synced event for automation engine
try:
@ -480,12 +593,28 @@ def run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, p
import hashlib as _hl
_track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
_tracks_hash = _hl.md5(_track_ids_str.encode()).hexdigest()
_mirror_tracks_hash = None
if str(playlist_id).startswith('auto_mirror_'):
try:
_mp_id = int(str(playlist_id).replace('auto_mirror_', '', 1))
from database.music_database import MusicDatabase
_mtracks = MusicDatabase().get_mirrored_playlist_tracks(_mp_id)
_mids = ','.join(
sorted(t.get('source_track_id', '') or '' for t in _mtracks if t.get('source_track_id'))
)
_mirror_tracks_hash = _hl.md5(_mids.encode()).hexdigest() if _mids else ''
except Exception as e:
logger.debug("mirror_tracks_hash for sync status: %s", e)
snapshot_id = getattr(playlist, 'snapshot_id', None)
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id,
_status_kwargs = dict(
matched_tracks=getattr(result, 'matched_tracks', 0),
total_tracks=getattr(result, 'total_tracks', 0),
discovered_tracks=len(tracks_json),
tracks_hash=_tracks_hash)
tracks_hash=_tracks_hash,
)
if _mirror_tracks_hash is not None:
_status_kwargs['mirror_tracks_hash'] = _mirror_tracks_hash
deps.update_and_save_sync_status(playlist_id, playlist_name, playlist.owner, snapshot_id, **_status_kwargs)
except Exception as e:
logger.error(f"SYNC FAILED for {playlist_id}: {e}")

View file

@ -343,6 +343,10 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
batch_is_album = False
batch_profile_id = 1
batch_source = 'spotify'
batch_playlist_folder_mode = False
batch_playlist_name = 'Unknown Playlist'
batch_playlist_id = playlist_id
batch_source_playlist_ref = ''
with tasks_lock:
if batch_id in download_batches:
force_download_all = download_batches[batch_id].get('force_download_all', False)
@ -352,6 +356,30 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
batch_artist_context = download_batches[batch_id].get('artist_context')
batch_profile_id = download_batches[batch_id].get('profile_id', 1) or 1
batch_source = download_batches[batch_id].get('batch_source', 'spotify') or 'spotify'
batch_playlist_folder_mode = download_batches[batch_id].get('playlist_folder_mode', False)
batch_playlist_name = download_batches[batch_id].get('playlist_name', 'Unknown Playlist')
batch_playlist_id = download_batches[batch_id].get('playlist_id', playlist_id)
batch_source_playlist_ref = (
download_batches[batch_id].get('source_playlist_ref') or ''
).strip()
from core.downloads.playlist_folder import (
resolve_playlist_folder_mode_for_batch,
track_exists_in_playlist_folder_from_track_data,
)
effective_playlist_folder_mode, effective_playlist_name = resolve_playlist_folder_mode_for_batch(
db,
playlist_id=str(batch_playlist_id),
playlist_name=batch_playlist_name,
batch_playlist_folder_mode=batch_playlist_folder_mode,
profile_id=batch_profile_id,
source=batch_source,
)
if effective_playlist_folder_mode and not batch_playlist_folder_mode:
with tasks_lock:
if batch_id in download_batches:
download_batches[batch_id]['playlist_folder_mode'] = True
download_batches[batch_id]['playlist_name'] = effective_playlist_name
if force_download_all:
logger.warning(f"[Force Download] Force download mode enabled for batch {batch_id} - treating all tracks as missing")
@ -445,6 +473,27 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
})
continue
if effective_playlist_folder_mode and not force_download_all:
if track_exists_in_playlist_folder_from_track_data(
effective_playlist_name,
track_data,
):
logger.info(
f"[Playlist Folder] '{track_name}' already on disk in playlist folder — skipping download"
)
try:
deps.check_and_remove_track_from_wishlist_by_metadata(track_data)
except Exception as _wl_err:
logger.debug(f"[Playlist Folder] Wishlist removal attempt failed: {_wl_err}")
analysis_results.append({
'track_index': track_index,
'track': track_data,
'found': True,
'confidence': 1.0,
'match_reason': 'playlist_folder_file',
})
continue
# Skip database check if force download is enabled
if force_download_all:
logger.warning(f"[Force Download] Skipping database check for '{track_name}' - treating as missing")
@ -982,13 +1031,47 @@ def run_full_missing_tracks_process(batch_id, playlist_id, tracks_json, deps: Ma
logger.info(f"[Wishlist] Added album context for: '{track_info.get('name')}' -> '{album_ctx['name']}'")
# Add playlist folder mode flag for sync page playlists
if batch_playlist_folder_mode:
# Add playlist folder mode flag for sync page playlists and wishlist
# tracks tied to a mirrored playlist with organize_by_playlist enabled.
task_pl_folder_mode = batch_playlist_folder_mode
task_pl_name = batch_playlist_name
if not task_pl_folder_mode and playlist_id == 'wishlist':
wl_source = track_info.get('source_info') or {}
if isinstance(wl_source, str):
try:
wl_source = json.loads(wl_source)
except (json.JSONDecodeError, TypeError):
wl_source = {}
wl_pl_ref = wl_source.get('playlist_id')
wl_pl_name = wl_source.get('playlist_name')
wl_pl_source = wl_source.get('source') or 'spotify'
if wl_pl_ref and hasattr(db, 'resolve_mirrored_playlist'):
wl_mirrored = db.resolve_mirrored_playlist(
wl_pl_ref,
profile_id=batch_profile_id,
default_source=wl_pl_source,
)
if wl_mirrored and wl_mirrored.get('organize_by_playlist'):
task_pl_folder_mode = True
task_pl_name = wl_pl_name or wl_mirrored.get('name') or batch_playlist_name
if task_pl_folder_mode:
track_info['_playlist_folder_mode'] = True
track_info['_playlist_name'] = batch_playlist_name
logger.info(f"[Task Creation] Added playlist folder mode for: {track_info.get('name')}{batch_playlist_name}")
track_info['_playlist_name'] = task_pl_name
if batch_source_playlist_ref:
track_info['source_info'] = {
'playlist_id': batch_source_playlist_ref,
'playlist_name': task_pl_name,
'source': batch_source,
}
logger.info(
f"[Task Creation] Added playlist folder mode for: "
f"{track_info.get('name')}{task_pl_name}"
)
else:
logger.debug(f"[Debug] Task Creation - playlist folder mode NOT enabled for: {track_info.get('name')}")
logger.debug(
f"[Debug] Task Creation - playlist folder mode NOT enabled for: "
f"{track_info.get('name')}"
)
download_tasks[task_id] = {
'status': 'pending', 'track_info': track_info,

View file

@ -0,0 +1,128 @@
"""Playlist-folder layout helpers for download analysis and existence checks."""
from __future__ import annotations
import os
from typing import Any, Dict, List, Optional
from core.downloads.file_finder import AUDIO_EXTENSIONS
from core.imports.paths import (
_get_config_manager,
docker_resolve_path,
get_file_path_from_template,
sanitize_filename,
)
def _first_artist_name(artists: Any) -> str:
if not artists:
return ''
first = artists[0]
if isinstance(first, dict):
return str(first.get('name', '') or '').strip()
return str(first).strip()
def candidate_playlist_folder_paths(
playlist_name: str,
artist: str,
title: str,
) -> List[str]:
"""Return absolute candidate paths for a track in playlist-folder layout."""
if not playlist_name or not title:
return []
artist_name = (artist or 'Unknown Artist').strip()
track_name = title.strip()
transfer_dir = docker_resolve_path(
_get_config_manager().get('soulseek.transfer_path', './Transfer')
)
template_context = {
'artist': artist_name,
'albumartist': artist_name,
'album': track_name,
'title': track_name,
'playlist_name': playlist_name,
'track_number': 1,
'disc_number': 1,
'year': '',
'quality': '',
'albumtype': '',
'_artists_list': [{'name': artist_name}],
}
candidates: List[str] = []
folder_path, filename_base = get_file_path_from_template(template_context, 'playlist_path')
if folder_path and filename_base:
base = os.path.join(transfer_dir, folder_path, filename_base)
for ext in AUDIO_EXTENSIONS:
candidates.append(base + ext)
else:
playlist_name_sanitized = sanitize_filename(playlist_name)
playlist_dir = os.path.join(transfer_dir, playlist_name_sanitized)
artist_name_sanitized = sanitize_filename(artist_name)
track_name_sanitized = sanitize_filename(track_name)
stem = f'{artist_name_sanitized} - {track_name_sanitized}'
for ext in AUDIO_EXTENSIONS:
candidates.append(os.path.join(playlist_dir, stem + ext))
return candidates
def track_exists_in_playlist_folder(
playlist_name: str,
artist: str,
title: str,
) -> bool:
"""Return True if any audio file exists at the playlist-folder path for this track."""
for path in candidate_playlist_folder_paths(playlist_name, artist, title):
if os.path.isfile(path):
return True
return False
def track_exists_in_playlist_folder_from_track_data(
playlist_name: str,
track_data: Dict[str, Any],
) -> bool:
"""Check playlist-folder existence using Spotify-style track payload."""
title = track_data.get('name', '') or track_data.get('track_name', '')
artist = _first_artist_name(track_data.get('artists', []))
if not artist:
artist = str(track_data.get('artist_name', '') or '').strip()
return track_exists_in_playlist_folder(playlist_name, artist, title)
def resolve_playlist_folder_mode_for_batch(
db: Any,
*,
playlist_id: str,
playlist_name: str,
batch_playlist_folder_mode: bool,
profile_id: int = 1,
source: str = 'spotify',
) -> tuple[bool, str]:
"""Merge batch flag with persisted mirrored-playlist preference."""
if batch_playlist_folder_mode:
return True, playlist_name
if not hasattr(db, 'resolve_mirrored_playlist'):
return False, playlist_name
# Pass the batch's source so numeric upstream ids (e.g. Deezer) resolve by
# source instead of colliding with the mirrored-playlists primary key.
mirrored = db.resolve_mirrored_playlist(
playlist_id, profile_id=profile_id, default_source=source or 'spotify'
)
if mirrored and mirrored.get('organize_by_playlist'):
return True, mirrored.get('name') or playlist_name
return False, playlist_name
__all__ = [
'candidate_playlist_folder_paths',
'track_exists_in_playlist_folder',
'track_exists_in_playlist_folder_from_track_data',
'resolve_playlist_folder_mode_for_batch',
]

View file

@ -18,9 +18,14 @@ from __future__ import annotations
from typing import Any, Callable, Optional
from flask import Blueprint, jsonify
from flask import Blueprint, jsonify, request
from core.enrichment.services import EnrichmentService, get_service
from core.enrichment.unmatched import (
SERVICE_ENTITY_SUPPORT,
UnmatchedQueryError,
supported_entity_types,
)
from utils.logging_config import get_logger
@ -30,25 +35,33 @@ logger = get_logger("enrichment.api")
# Hooks the host wires up so the blueprint can persist pause state and
# clean up auto-pause / yield-override sets without circular imports.
_config_set: Optional[Callable[[str, Any], None]] = None
_config_get: Optional[Callable[[str, Any], Any]] = None
_auto_paused_discard: Optional[Callable[[str], None]] = None
_yield_override_add: Optional[Callable[[str], None]] = None
_db_getter: Optional[Callable[[], Any]] = None
def configure(
*,
config_set: Optional[Callable[[str, Any], None]] = None,
config_get: Optional[Callable[[str, Any], Any]] = None,
auto_paused_discard: Optional[Callable[[str], None]] = None,
yield_override_add: Optional[Callable[[str], None]] = None,
db_getter: Optional[Callable[[], Any]] = None,
) -> None:
"""Wire host-side mutators that the generic routes call after pause/resume.
Each is optional pass None for hosts that don't have a corresponding
mechanism (e.g. tests).
mechanism (e.g. tests). ``db_getter`` returns the live ``MusicDatabase``
for the unmatched-browser routes; ``config_get``/``config_set`` read and
write the per-worker priority override.
"""
global _config_set, _auto_paused_discard, _yield_override_add
global _config_set, _config_get, _auto_paused_discard, _yield_override_add, _db_getter
_config_set = config_set
_config_get = config_get
_auto_paused_discard = auto_paused_discard
_yield_override_add = yield_override_add
_db_getter = db_getter
def _persist_paused(service: EnrichmentService, paused: bool) -> None:
@ -153,4 +166,133 @@ def create_blueprint() -> Blueprint:
logger.error("Error resuming %s worker: %s", service.id, e)
return jsonify({'error': str(e)}), 500
@bp.route('/api/enrichment/<service_id>/breakdown', methods=['GET'])
def enrichment_breakdown(service_id: str):
"""matched / not_found / pending tallies per entity type for the modal."""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
try:
db = _db_getter()
breakdown = {
entity: db.get_enrichment_breakdown(service_id, entity)
for entity in supported_entity_types(service_id)
}
return jsonify({'service': service_id, 'breakdown': breakdown}), 200
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error building %s enrichment breakdown: %s", service_id, e)
return jsonify({'error': str(e)}), 500
@bp.route('/api/enrichment/<service_id>/unmatched', methods=['GET'])
def enrichment_unmatched(service_id: str):
"""Paginated list of items this source hasn't matched (for manual match).
Query params: ``entity_type`` (artist|album|track), ``status``
(not_found|pending|unmatched), ``q`` (name search), ``limit``, ``offset``.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
entity_type = (request.args.get('entity_type') or 'artist').strip()
status = (request.args.get('status') or 'not_found').strip()
query = (request.args.get('q') or '').strip() or None
try:
limit = int(request.args.get('limit', 50))
offset = int(request.args.get('offset', 0))
except (TypeError, ValueError):
return jsonify({'error': 'limit/offset must be integers'}), 400
try:
result = _db_getter().get_enrichment_unmatched(
service_id, entity_type, status, query, limit, offset
)
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error listing %s unmatched %ss: %s", service_id, entity_type, e)
return jsonify({'error': str(e)}), 500
result.update({
'service': service_id,
'entity_type': entity_type,
'status': status,
'limit': limit,
'offset': offset,
'entity_types': list(supported_entity_types(service_id)),
})
return jsonify(result), 200
@bp.route('/api/enrichment/<service_id>/retry', methods=['POST'])
def enrichment_retry(service_id: str):
"""Re-queue item(s) so the worker re-attempts them.
Body: ``entity_type`` (artist|album|track), ``scope`` (item|failed),
``entity_id`` (required when scope='item'). 'failed' re-queues every
not_found item of that entity type.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _db_getter is None:
return jsonify({'error': 'database unavailable'}), 503
data = request.get_json(silent=True) or {}
entity_type = (data.get('entity_type') or 'artist').strip()
scope = (data.get('scope') or 'item').strip()
entity_id = data.get('entity_id')
try:
count = _db_getter().reset_enrichment(service_id, entity_type, scope, entity_id)
except UnmatchedQueryError as e:
return jsonify({'error': str(e)}), 400
except Exception as e:
logger.error("Error re-queuing %s %s (%s): %s", service_id, entity_type, scope, e)
return jsonify({'error': str(e)}), 500
return jsonify({'success': True, 'reset': count, 'service': service_id,
'entity_type': entity_type, 'scope': scope}), 200
@bp.route('/api/enrichment/<service_id>/priority', methods=['GET'])
def enrichment_get_priority(service_id: str):
"""Return the pinned 'process this group first' entity for a worker."""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
priority = ''
if _config_get is not None:
try:
priority = (_config_get(f'{service_id}_enrichment_priority', '') or '').strip().lower()
except Exception as e:
logger.debug("reading %s priority: %s", service_id, e)
if priority not in supported_entity_types(service_id):
priority = ''
return jsonify({'service': service_id, 'priority': priority,
'entity_types': list(supported_entity_types(service_id))}), 200
@bp.route('/api/enrichment/<service_id>/priority', methods=['POST'])
def enrichment_set_priority(service_id: str):
"""Pin (or clear) the entity type the worker should process first.
Body: ``entity`` = 'artist'|'album'|'track' to pin, or '' / null / 'none'
to clear. Must be an entity type the source actually enriches.
"""
if service_id not in SERVICE_ENTITY_SUPPORT:
return jsonify({'error': f'Unknown enrichment service: {service_id}'}), 404
if _config_set is None:
return jsonify({'error': 'config unavailable'}), 503
data = request.get_json(silent=True) or {}
entity = (data.get('entity') or '').strip().lower()
if entity in ('none', 'clear'):
entity = ''
if entity and entity not in supported_entity_types(service_id):
return jsonify({'error': f'{service_id} does not enrich {entity!r}'}), 400
try:
_config_set(f'{service_id}_enrichment_priority', entity)
except Exception as e:
logger.error("setting %s priority: %s", service_id, e)
return jsonify({'error': str(e)}), 500
logger.info("%s enrichment priority set to %r via UI", service_id, entity or '(none)')
return jsonify({'success': True, 'service': service_id, 'priority': entity}), 200
return bp

View file

@ -0,0 +1,269 @@
"""Read-side helpers for browsing the items an enrichment source hasn't matched.
The dashboard "Manage Enrichment Workers" modal lists, per source, the
artists / albums / tracks whose ``<service>_match_status`` is ``'not_found'``
(or still pending = ``NULL``) so the user can manually match them. Every
enrichment source writes a uniform ``<service>_match_status`` column, so one
parametric query serves all 11 workers.
This module owns the column mapping and SQL construction. ``service`` and
``entity_type`` are whitelisted against :data:`SERVICE_ENTITY_SUPPORT` and the
entity table map before any column name is interpolated user-supplied values
(the search term, pagination) are always bound parameters, never interpolated.
"""
from __future__ import annotations
from typing import List, Optional, Tuple
# Which entity types each enrichment source covers. Mirrors the authoritative
# ``_SERVICE_ID_COLUMNS`` map in web_server.py (used by manual-match), kept here
# so the unmatched browser is self-contained and unit-testable. Singular keys
# ('artist'/'album'/'track') match the manual-match entity_type vocabulary.
SERVICE_ENTITY_SUPPORT = {
'spotify': ('artist', 'album', 'track'),
'musicbrainz': ('artist', 'album', 'track'),
'deezer': ('artist', 'album', 'track'),
'audiodb': ('artist', 'album', 'track'),
'discogs': ('artist', 'album'), # no track-level id column
'itunes': ('artist', 'album', 'track'),
'lastfm': ('artist', 'album', 'track'),
'genius': ('artist', 'track'), # no album-level id column
'tidal': ('artist', 'album', 'track'),
'qobuz': ('artist', 'album', 'track'),
'amazon': ('artist', 'album', 'track'),
# Relationship enrichment (not a metadata source): the Similar Artists worker
# only operates at the artist level, and its <service>_match_status tracks
# whether MusicMap similars were fetched (not a source-id match). So the
# breakdown / unmatched list here means "artists we have / don't have
# similars for" — informative, even though there's no manual-match action.
'similar_artists': ('artist',),
}
# entity_type -> table / display-name column / image expression / optional join
# / parent-context expression (the artist an album belongs to; the album a
# track belongs to) so the UI can disambiguate same-named items.
# tracks carry no artwork column of their own, so we borrow the parent album's.
_ENTITY_TABLE = {
'artist': {
'table': 'artists', 'name': 'name',
'image': 'artists.thumb_url', 'join': '', 'parent': None,
},
'album': {
'table': 'albums', 'name': 'title',
'image': 'albums.thumb_url',
'join': 'LEFT JOIN artists par ON albums.artist_id = par.id',
'parent': 'par.name',
},
'track': {
'table': 'tracks', 'name': 'title',
'image': 'al.thumb_url',
'join': 'LEFT JOIN albums al ON tracks.album_id = al.id',
'parent': 'al.title',
},
}
# 'unmatched' = not yet matched at all (pending OR explicitly not_found).
VALID_STATUSES = ('not_found', 'pending', 'unmatched')
# Hard cap so a malicious/buggy caller can't ask for the whole library at once.
MAX_LIMIT = 200
class UnmatchedQueryError(ValueError):
"""Raised for an unknown service / unsupported entity type / bad status."""
def supported_entity_types(service: str) -> Tuple[str, ...]:
"""Return the entity types a source enriches, or () for an unknown source."""
return SERVICE_ENTITY_SUPPORT.get(service, ())
def match_status_column(service: str) -> str:
return f"{service}_match_status"
def last_attempted_column(service: str) -> str:
return f"{service}_last_attempted"
def _validate(service: str, entity_type: str) -> None:
support = SERVICE_ENTITY_SUPPORT.get(service)
if support is None:
raise UnmatchedQueryError(f"Unknown enrichment service: {service!r}")
if entity_type not in support:
raise UnmatchedQueryError(
f"{service} does not enrich {entity_type!r} entities"
)
if entity_type not in _ENTITY_TABLE: # defensive — support map drift
raise UnmatchedQueryError(f"No table mapping for entity type {entity_type!r}")
def _status_predicate(service: str, status: str, qualifier: str) -> str:
"""SQL predicate selecting rows in the requested match state.
``qualifier`` (the table name/alias) is always prefixed so the predicate is
unambiguous even when the query joins a second table that also carries a
``<service>_match_status`` column (tracks LEFT JOIN albums).
"""
col = f"{qualifier}.{match_status_column(service)}"
if status == 'not_found':
return f"{col} = 'not_found'"
if status == 'pending':
return f"{col} IS NULL"
# 'unmatched'
return f"({col} IS NULL OR {col} = 'not_found')"
def build_unmatched_query(
service: str,
entity_type: str,
status: str = 'not_found',
query: Optional[str] = None,
limit: int = 50,
offset: int = 0,
) -> Tuple[str, List]:
"""Build the paginated SELECT for one (service, entity_type, status) view.
Returns ``(sql, params)``. Selected columns: id, name, image_url, status,
last_attempted.
"""
_validate(service, entity_type)
if status not in VALID_STATUSES:
raise UnmatchedQueryError(f"Invalid status: {status!r}")
meta = _ENTITY_TABLE[entity_type]
table, name_col, image_expr, join = (
meta['table'], meta['name'], meta['image'], meta['join'],
)
ms = match_status_column(service)
la = last_attempted_column(service)
where = [_status_predicate(service, status, table)]
params: List = []
if query:
where.append(f"{table}.{name_col} LIKE ?")
params.append(f"%{query}%")
parent_expr = meta.get('parent')
parent_select = f"{parent_expr} AS parent" if parent_expr else "NULL AS parent"
sql = (
f"SELECT {table}.id AS id, {table}.{name_col} AS name, "
f"{image_expr} AS image_url, {parent_select}, {table}.{ms} AS status, "
f"{table}.{la} AS last_attempted "
f"FROM {table} {join} "
f"WHERE {' AND '.join(where)} "
f"ORDER BY {table}.{name_col} COLLATE NOCASE "
f"LIMIT ? OFFSET ?"
).replace(' ', ' ')
params.append(_clamp_limit(limit))
params.append(max(int(offset or 0), 0))
return sql, params
def build_count_query(
service: str,
entity_type: str,
status: str = 'not_found',
query: Optional[str] = None,
) -> Tuple[str, List]:
"""Build the COUNT(*) matching :func:`build_unmatched_query`'s filters."""
_validate(service, entity_type)
if status not in VALID_STATUSES:
raise UnmatchedQueryError(f"Invalid status: {status!r}")
meta = _ENTITY_TABLE[entity_type]
table, name_col = meta['table'], meta['name']
where = [_status_predicate(service, status, table)]
params: List = []
if query:
where.append(f"{table}.{name_col} LIKE ?")
params.append(f"%{query}%")
sql = f"SELECT COUNT(*) FROM {table} WHERE {' AND '.join(where)}"
return sql, params
# Reset scopes for re-queuing items so the worker re-attempts them.
RESET_SCOPES = ('item', 'failed')
def build_reset_query(
service: str,
entity_type: str,
scope: str = 'item',
entity_id=None,
) -> Tuple[str, List]:
"""Build the UPDATE that re-queues item(s) for enrichment.
Re-queuing means clearing ``<service>_match_status`` back to NULL (and
``<service>_last_attempted`` to NULL): every worker's pending query selects
``match_status IS NULL`` first, so the item is retried on the next pass.
Nulling last_attempted alone is NOT enough the not_found retry path uses
``last_attempted < cutoff`` and ``NULL < cutoff`` is false, so the item
would never be picked up.
* scope='item' -> a single row (requires entity_id)
* scope='failed' -> every 'not_found' row for this entity type
"""
_validate(service, entity_type)
if scope not in RESET_SCOPES:
raise UnmatchedQueryError(f"Invalid reset scope: {scope!r}")
meta = _ENTITY_TABLE[entity_type]
table = meta['table']
ms = match_status_column(service)
la = last_attempted_column(service)
set_clause = f"SET {ms} = NULL, {la} = NULL"
if scope == 'item':
if not entity_id:
raise UnmatchedQueryError("entity_id is required for an item reset")
return f"UPDATE {table} {set_clause} WHERE id = ?", [entity_id]
# 'failed' — re-queue everything this source explicitly gave up on.
return f"UPDATE {table} {set_clause} WHERE {ms} = 'not_found'", []
def build_breakdown_query(service: str, entity_type: str) -> Tuple[str, List]:
"""Build the matched / not_found / pending / total tally for one entity type."""
_validate(service, entity_type)
meta = _ENTITY_TABLE[entity_type]
table = meta['table']
ms = f"{table}.{match_status_column(service)}"
sql = (
"SELECT "
f"SUM(CASE WHEN {ms} = 'matched' THEN 1 ELSE 0 END) AS matched, "
f"SUM(CASE WHEN {ms} = 'not_found' THEN 1 ELSE 0 END) AS not_found, "
f"SUM(CASE WHEN {ms} IS NULL THEN 1 ELSE 0 END) AS pending, "
f"COUNT(*) AS total "
f"FROM {table}"
)
return sql, []
def _clamp_limit(limit) -> int:
try:
n = int(limit)
except (TypeError, ValueError):
return 50
if n <= 0:
return 50
return min(n, MAX_LIMIT)
__all__ = [
'SERVICE_ENTITY_SUPPORT',
'VALID_STATUSES',
'MAX_LIMIT',
'UnmatchedQueryError',
'supported_entity_types',
'match_status_column',
'last_attempted_column',
'build_unmatched_query',
'build_count_query',
'build_breakdown_query',
'build_reset_query',
'RESET_SCOPES',
]

View file

@ -178,6 +178,16 @@ class GeniusWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Genius
# is artist/track only, so albums are not honored.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('genius')
if _prio in ('artist', 'track'):
_pi = priority_pending_item(cursor, 'genius', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -418,8 +418,20 @@ def _coerce_int(value: Any, default: int = 1) -> int:
return coerced if coerced > 0 else default
def build_final_path_for_track(context, artist_context, album_info, file_ext):
"""Shared path builder used by both post-processing and verification."""
def build_final_path_for_track(context, artist_context, album_info, file_ext, create_dirs: bool = True):
"""Shared path builder used by both post-processing and verification.
``create_dirs`` gates the directory-creation side effects. The download
import flow leaves it True (it's about to write the file there). The
library-reorganize PREVIEW passes False so a dry run can compute the exact
destination path WITHOUT physically creating the folder fixes #767 (dry
run was leaving empty destination folders behind)."""
_real_makedirs = os.makedirs
def _ensure_dir(path, **_kw):
if create_dirs:
_real_makedirs(path, exist_ok=True)
transfer_dir = docker_resolve_path(_get_config_manager().get("soulseek.transfer_path", "./Transfer"))
context = normalize_import_context(context)
track_info = get_import_track_info(context)
@ -440,7 +452,7 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
original_dir = os.path.dirname(original_path)
original_stem = os.path.splitext(os.path.basename(original_path))[0]
final_path = os.path.join(original_dir, original_stem + file_ext)
os.makedirs(original_dir, exist_ok=True)
_ensure_dir(original_dir, exist_ok=True)
logger.info("[Enhance] Using original file location: %s", final_path)
return final_path, True
@ -477,12 +489,12 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
folder_path, filename_base = get_file_path_from_template(template_context, "playlist_path")
if folder_path and filename_base:
final_path = os.path.join(transfer_dir, folder_path, filename_base + file_ext)
os.makedirs(os.path.join(transfer_dir, folder_path), exist_ok=True)
_ensure_dir(os.path.join(transfer_dir, folder_path), exist_ok=True)
return final_path, True
playlist_name_sanitized = sanitize_filename(playlist_name)
playlist_dir = os.path.join(transfer_dir, playlist_name_sanitized)
os.makedirs(playlist_dir, exist_ok=True)
_ensure_dir(playlist_dir, exist_ok=True)
artist_name_sanitized = sanitize_filename(template_context["artist"])
track_name_sanitized = sanitize_filename(track_name)
new_filename = f"{artist_name_sanitized} - {track_name_sanitized}{file_ext}"
@ -579,10 +591,10 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
if total_discs > 1 and not user_controls_disc:
disc_folder = f"{disc_label} {disc_number}"
final_path = os.path.join(transfer_dir, folder_path, disc_folder, filename_base + file_ext)
os.makedirs(os.path.join(transfer_dir, folder_path, disc_folder), exist_ok=True)
_ensure_dir(os.path.join(transfer_dir, folder_path, disc_folder), exist_ok=True)
else:
final_path = os.path.join(transfer_dir, folder_path, filename_base + file_ext)
os.makedirs(os.path.join(transfer_dir, folder_path), exist_ok=True)
_ensure_dir(os.path.join(transfer_dir, folder_path), exist_ok=True)
return final_path, True
artist_name_sanitized = sanitize_filename(template_context["albumartist"])
@ -592,7 +604,7 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
album_dir = os.path.join(artist_dir, album_folder_name)
if total_discs > 1:
album_dir = os.path.join(album_dir, f"{disc_label} {disc_number}")
os.makedirs(album_dir, exist_ok=True)
_ensure_dir(album_dir, exist_ok=True)
final_track_name_sanitized = sanitize_filename(clean_track_name)
new_filename = f"{track_number:02d} - {final_track_name_sanitized}{file_ext}"
return os.path.join(album_dir, new_filename), True
@ -629,10 +641,10 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
if filename_base:
if folder_path:
final_path = os.path.join(transfer_dir, folder_path, filename_base + file_ext)
os.makedirs(os.path.join(transfer_dir, folder_path), exist_ok=True)
_ensure_dir(os.path.join(transfer_dir, folder_path), exist_ok=True)
else:
final_path = os.path.join(transfer_dir, filename_base + file_ext)
os.makedirs(transfer_dir, exist_ok=True)
_ensure_dir(transfer_dir, exist_ok=True)
return final_path, True
artist_name_sanitized = sanitize_filename(template_context["artist"])
@ -640,6 +652,6 @@ def build_final_path_for_track(context, artist_context, album_info, file_ext):
artist_dir = os.path.join(transfer_dir, artist_name_sanitized)
single_folder_name = f"{artist_name_sanitized} - {final_track_name_sanitized}"
single_dir = os.path.join(artist_dir, single_folder_name)
os.makedirs(single_dir, exist_ok=True)
_ensure_dir(single_dir, exist_ok=True)
new_filename = f"{final_track_name_sanitized}{file_ext}"
return os.path.join(single_dir, new_filename), True

View file

@ -110,6 +110,33 @@ def _mark_task_quarantined(context: dict, quarantine_path: str | None) -> None:
download_tasks[task_id]['quarantine_entry_id'] = entry_id
def import_rejection_reason(context: dict) -> str | None:
"""Human-readable reason if post-processing terminally rejected the file
(quarantine or race-guard), else ``None`` for a clean import.
``post_process_matched_download`` signals these outcomes by setting context
flags and returning normally it only raises on unexpected errors. The
download path reads those flags in
``post_process_matched_download_with_verification`` and marks the task
failed, but the MANUAL-import routes call ``post_process_matched_download``
directly with no task_id, so without this check a quarantined file (now in
ss_quarantine, not the library) is counted as a successful import and the
UI shows a green "Done" (#764). Pure + testable: it only inspects the
context dict the inner pipeline populated."""
if context.get('_integrity_failure_msg'):
return f"integrity check failed: {context['_integrity_failure_msg']}"
if context.get('_acoustid_quarantined'):
return (
"AcoustID verification failed: "
f"{context.get('_acoustid_failure_msg', 'fingerprint mismatch')}"
)
if context.get('_bitdepth_rejected'):
return "rejected by bit-depth filter"
if context.get('_race_guard_failed'):
return "source file disappeared before import completed"
return None
def build_import_pipeline_runtime(
*,
automation_engine: Any | None = None,
@ -563,6 +590,26 @@ def post_process_matched_download(context_key, context, file_path, runtime, meta
record_library_history_download(context)
record_download_provenance(context)
try:
pf_album_info = build_import_album_info(context, force_album=False)
if not pf_album_info or not pf_album_info.get("album_name"):
pf_album_info = {
"is_album": True,
"album_name": playlist_name,
"track_number": track_info.get("track_number", 1) or 1,
"disc_number": track_info.get("disc_number", 1) or 1,
"clean_track_name": get_import_clean_title(
context,
default=get_import_original_search(context).get("title", "Unknown"),
),
"source": get_import_source(context) or "spotify",
}
elif not pf_album_info.get("is_album"):
pf_album_info["is_album"] = True
record_soulsync_library_entry(context, artist_context, pf_album_info)
except Exception as lib_err:
logger.error(f"[Playlist Folder] SoulSync library registration failed: {lib_err}")
task_id = context.get('task_id')
batch_id = context.get('batch_id')
if task_id and batch_id:

View file

@ -11,6 +11,7 @@ from typing import Any, Callable, Dict
from core.imports.album import build_album_import_context, build_album_import_match_payload, resolve_album_artist_context
from core.imports.context import get_import_context_artist, get_import_track_info, normalize_import_context
from core.imports.filename import parse_filename_metadata
from core.imports.pipeline import import_rejection_reason
from core.imports.staging import (
AUDIO_EXTENSIONS,
get_import_suggestions_cache,
@ -331,8 +332,17 @@ def album_process(runtime: ImportRouteRuntime, data: Dict[str, Any]) -> tuple[Di
try:
runtime.post_process_matched_download(context_key, context, file_path)
processed += 1
runtime.logger.info("Import processed: %s. %s from %s", track_number, track_name, album_name)
# A quarantine/race-guard rejection returns normally (no
# exception) and leaves the file in ss_quarantine, NOT the
# library — so it must be reported as an error, not counted
# as a successful import (#764).
reject_reason = import_rejection_reason(context)
if reject_reason:
errors.append(f"{track_name}: {reject_reason}")
runtime.logger.warning("Import rejected: %s%s", track_name, reject_reason)
else:
processed += 1
runtime.logger.info("Import processed: %s. %s from %s", track_number, track_name, album_name)
except Exception as proc_err:
err_msg = f"{track_name}: {str(proc_err)}"
errors.append(err_msg)
@ -422,6 +432,13 @@ def process_single_import_file(runtime: ImportRouteRuntime, file_info: Dict[str,
context_key = f"import_single_{uuid.uuid4().hex[:8]}"
runtime.post_process_matched_download(context_key, context, file_path)
# Quarantine/race-guard returns normally but the file is in
# ss_quarantine, not the library — report it as an error rather than
# "ok", else the UI shows a green "Done" for a file that vanished (#764).
reject_reason = import_rejection_reason(context)
if reject_reason:
runtime.logger.warning("Import single rejected: %s%s", final_title, reject_reason)
return ("error", f"{final_title}: {reject_reason}")
runtime.logger.info(
"Import single processed: %s by %s (source=%s)",
final_title,

View file

@ -172,6 +172,17 @@ class iTunesWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('itunes')
if _prio:
_pi = priority_pending_item(cursor, 'itunes', _prio,
{'album': 'album_individual', 'track': 'track_individual'})
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -177,6 +177,16 @@ class LastFMWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('lastfm')
if _prio:
_pi = priority_pending_item(cursor, 'lastfm', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -227,6 +227,25 @@ def _resolve_source(album_data: dict, primary_source: str, strict_source: bool =
"""
source_ids = _extract_source_ids(album_data)
# #765: if a canonical release was pinned for this album (best-fit to the
# user's actual files), prefer it — so reorganize agrees with Track Number
# Repair and stops mislabelling standard albums as deluxe (#767-Bug2). Gated
# on the album row carrying a canonical, and skipped when the user explicitly
# picked a source in the modal (strict_source) — their choice wins. Falls
# through to the priority walk if the canonical fetch fails.
if not strict_source:
c_source = album_data.get('canonical_source')
c_id = album_data.get('canonical_album_id')
if c_source and c_id:
try:
api_album = get_album_for_source(c_source, c_id)
api_tracks = get_album_tracks_for_source(c_source, c_id)
items = _normalize_album_tracks(api_tracks)
if items and api_album:
return c_source, api_album, items
except Exception as e:
logger.warning(f"[Reorganize] canonical {c_source} lookup raised: {e}")
if strict_source:
sources_to_try = [primary_source] if primary_source else []
else:
@ -989,7 +1008,12 @@ def preview_album_reorganize(
album_info = _build_album_info(context)
try:
spotify_artist = context['spotify_artist']
new_full, _ok = build_final_path_fn(context, spotify_artist, album_info, file_ext)
# Dry run: compute the destination path WITHOUT creating the folder.
# Previously this physically created the album dir during preview,
# leaving empty folders all over the library (#767).
new_full, _ok = build_final_path_fn(
context, spotify_artist, album_info, file_ext, create_dirs=False
)
item['new_path'] = (
os.path.relpath(new_full, transfer_dir)
if transfer_dir and new_full and new_full.startswith(transfer_dir)

157
core/metadata/art_apply.py Normal file
View file

@ -0,0 +1,157 @@
"""Apply album art to existing library files.
Two jobs, both reusing the post-processing standard so the user's
``album_art_order`` preference is honored and embedded art matches cover.jpg:
- Detect whether an album already has art ON DISK (embedded in the audio file
or a cover.jpg/folder.jpg sidecar) the Cover Art Filler previously only
looked at the DB ``thumb_url``, so albums whose files were artless but whose
DB row had a URL were never flagged.
- Embed found art into the album's audio files (``embed_album_art_metadata``)
and write a cover.jpg (``download_cover_art``). Only ADDS art it does not
clear or rewrite the user's existing tags.
"""
from __future__ import annotations
import contextlib
import os
from typing import Iterable
from core.metadata.artwork import download_cover_art, embed_album_art_metadata
from core.metadata.common import get_mutagen_symbols
from utils.logging_config import get_logger
logger = get_logger("metadata.art_apply")
# Folder-level cover files recognised across players (matches soulsync_client).
_COVER_SIDECARS = (
"cover.jpg", "cover.jpeg", "cover.png",
"folder.jpg", "folder.jpeg", "folder.png",
)
def folder_has_cover_sidecar(folder: str) -> bool:
"""True if the album folder already carries a cover.jpg/folder.jpg sidecar."""
if not folder:
return False
try:
for name in _COVER_SIDECARS:
if os.path.isfile(os.path.join(folder, name)):
return True
except OSError:
return False
return False
def file_has_embedded_art(file_path: str) -> bool:
"""True if the audio file already has embedded cover art (FLAC picture,
ID3 APIC, MP4 covr, or a Vorbis metadata_block_picture)."""
if not file_path or not os.path.isfile(file_path):
return False
symbols = get_mutagen_symbols()
if not symbols:
return False
try:
return _audio_has_art(symbols.File(file_path), symbols)
except Exception as exc:
logger.debug("art presence check failed for %s: %s", file_path, exc)
return False
def _audio_has_art(audio, symbols) -> bool:
"""True if an already-open mutagen object carries embedded cover art."""
if audio is None:
return False
# FLAC / Ogg expose picture blocks directly.
if getattr(audio, "pictures", None):
return True
if isinstance(audio, symbols.MP4):
return bool(audio.get("covr"))
tags = getattr(audio, "tags", None)
if tags is None:
return False
with contextlib.suppress(Exception):
if isinstance(tags, symbols.ID3):
return bool(tags.getall("APIC"))
with contextlib.suppress(Exception):
if "metadata_block_picture" in tags:
return True
return False
def album_has_art_on_disk(rep_file_path: str) -> bool:
"""Does this album have art on disk?
Checks the folder for a cover sidecar first (cheap stat) and only opens the
representative audio file when there's no sidecar. Returns True when there's
no local file to inspect (e.g. a media-server-only album) so such albums
aren't wrongly flagged as missing file art.
"""
if not rep_file_path:
return True
folder = os.path.dirname(rep_file_path)
if folder_has_cover_sidecar(folder):
return True
return file_has_embedded_art(rep_file_path)
def apply_art_to_album_files(
file_paths: Iterable[str],
metadata: dict,
album_info: dict,
folder: str = None,
context: dict = None,
) -> dict:
"""Embed art into each audio file + write cover.jpg, reusing the standard.
``metadata`` feeds ``embed_album_art_metadata`` (needs album_artist/artist/
album, optionally musicbrainz_release_id and album_art_url as the fallback
URL). ``album_info`` feeds ``download_cover_art`` (album_name/album_image_url/
musicbrainz_release_id). Existing tags are preserved only art is added.
Returns counts; never raises (unwritable/read-only files are skipped).
"""
result = {"embedded": 0, "failed": 0, "skipped": 0, "cover_written": False}
symbols = get_mutagen_symbols()
paths = [p for p in (file_paths or []) if p]
if not symbols:
return result
for fp in paths:
if not os.path.isfile(fp):
result["skipped"] += 1
continue
try:
audio = symbols.File(fp)
if audio is None:
result["skipped"] += 1
continue
# Purely additive: never touch a file that already has art. Embedding
# again would APPEND a duplicate picture on FLAC (add_picture doesn't
# replace), so leave already-arted files alone.
if _audio_has_art(audio, symbols):
result["skipped"] += 1
continue
# ID3 needs a tag container before APIC can be added.
if getattr(audio, "tags", None) is None and hasattr(audio, "add_tags"):
with contextlib.suppress(Exception):
audio.add_tags()
if embed_album_art_metadata(audio, metadata):
audio.save()
result["embedded"] += 1
else:
result["failed"] += 1
except Exception as exc:
# Read-only mounts / permission errors land here — skip, don't crash.
logger.warning("Could not embed art into %s: %s", fp, exc)
result["failed"] += 1
target_dir = folder or (os.path.dirname(paths[0]) if paths else None)
if target_dir and os.path.isdir(target_dir):
try:
download_cover_art(album_info, target_dir, context)
result["cover_written"] = folder_has_cover_sidecar(target_dir)
except Exception as exc:
logger.warning("cover.jpg write failed for %s: %s", target_dir, exc)
return result

View file

@ -0,0 +1,115 @@
"""Preserve embedded cover art across the metadata-enrichment rewrite.
Issue #764 (continuation of #755): imported files lost their album art.
``enhance_file_metadata`` rebuilds tags from scratch for FLAC it calls
``clear_pictures()`` and for MP3/MP4 it clears the whole tag block *before*
it has the replacement art in hand. It then saves the file regardless of
whether new art was actually embedded. So every failure mode downstream
destroyed the art that shipped with the download:
- source-metadata extraction returns nothing -> early save, no embed
- no album-art URL available / art download fails -> embed returns early
- art rejected by the min-resolution guard -> embed returns early
- art embedding disabled in config -> embed skipped entirely
In all of those the file was saved with the pictures already cleared and
nothing put back. This module captures the existing art up front (live
mutagen objects, so they re-apply verbatim) and restores it right before a
save *iff the file currently has none* so the happy path (new art embedded)
is byte-for-byte unchanged, and the only behaviour change is that we never
end up with less art than we started with.
Scope mirrors ``embed_album_art_metadata``: FLAC ``Picture`` blocks, ID3
``APIC`` frames, MP4 ``covr`` atoms. OggOpus/OggVorbis store art inside the
Vorbis comment (no ``clear_pictures``), so the enrichment rewrite never
strips it and it needs no preservation here.
"""
from __future__ import annotations
from typing import Any, List, Tuple
from utils.logging_config import get_logger
logger = get_logger("metadata.art_preservation")
# Each snapshot entry is (kind, payload) where payload is a list of live
# mutagen objects captured before the tag rewrite.
ArtSnapshot = List[Tuple[str, list]]
def has_embedded_art(audio_file: Any, symbols: Any) -> bool:
"""True iff ``audio_file`` currently carries embedded cover art in any
of the formats the enricher manages (FLAC pictures / ID3 APIC / MP4 covr)."""
try:
if getattr(audio_file, "pictures", None):
return True
tags = getattr(audio_file, "tags", None)
if tags is not None and isinstance(tags, symbols.ID3) and tags.getall("APIC"):
return True
if isinstance(audio_file, symbols.MP4) and tags and tags.get("covr"):
return True
except Exception as exc: # defensive: never let art-detection break a save
logger.debug("has_embedded_art check failed: %s", exc)
return False
def snapshot_embedded_art(audio_file: Any, symbols: Any) -> ArtSnapshot:
"""Capture existing embedded art so it can be restored if re-embedding
fails. Returns a list of ``(kind, [objects])`` entries, or ``[]`` when the
file has no art. Captures the live mutagen objects (Picture / APIC frame /
MP4Cover) so they re-apply exactly as they were.
Must be called BEFORE ``clear_pictures()`` / ``tags.clear()``."""
snap: ArtSnapshot = []
try:
pictures = getattr(audio_file, "pictures", None)
if pictures:
snap.append(("flac", list(pictures)))
tags = getattr(audio_file, "tags", None)
if tags is not None and isinstance(tags, symbols.ID3):
apics = tags.getall("APIC")
if apics:
snap.append(("id3", list(apics)))
if isinstance(audio_file, symbols.MP4) and tags:
covr = tags.get("covr")
if covr:
snap.append(("mp4", list(covr)))
except Exception as exc:
logger.debug("snapshot_embedded_art failed: %s", exc)
return snap
def restore_embedded_art(audio_file: Any, symbols: Any, snapshot: ArtSnapshot) -> bool:
"""Re-apply captured art IFF the file currently has none. Returns True if
it restored something.
No-op (returns False) when the snapshot is empty or the file already has
art so calling this before a save never overwrites freshly-embedded art,
it only puts back what the rewrite would otherwise have destroyed."""
if not snapshot or has_embedded_art(audio_file, symbols):
return False
restored = False
for kind, payload in snapshot:
try:
if kind == "flac" and hasattr(audio_file, "add_picture"):
for pic in payload:
audio_file.add_picture(pic)
restored = True
elif kind == "id3":
tags = getattr(audio_file, "tags", None)
if tags is not None:
for frame in payload:
tags.add(frame)
restored = True
elif kind == "mp4":
audio_file["covr"] = payload
restored = True
except Exception as exc:
logger.debug("restore_embedded_art (%s) failed: %s", kind, exc)
if restored:
logger.info("Preserved existing embedded cover art (re-embed produced none).")
return restored
__all__ = ["has_embedded_art", "snapshot_embedded_art", "restore_embedded_art", "ArtSnapshot"]

View file

@ -363,7 +363,7 @@ def embed_album_art_metadata(audio_file, metadata: dict):
cfg = get_config_manager()
symbols = get_mutagen_symbols()
if not symbols:
return
return False
try:
image_data = None
@ -410,12 +410,12 @@ def embed_album_art_metadata(audio_file, metadata: dict):
art_url = metadata.get("album_art_url")
if not art_url:
logger.warning("No album art URL available for embedding.")
return
return False
image_data, mime_type = _fetch_art_bytes(art_url)
if not image_data:
logger.error("Failed to download album art data.")
return
return False
if isinstance(audio_file.tags, symbols.ID3):
audio_file.tags.add(symbols.APIC(encoding=3, mime=mime_type, type=3, desc="Cover", data=image_data))
@ -434,8 +434,10 @@ def embed_album_art_metadata(audio_file, metadata: dict):
audio_file["covr"] = [symbols.MP4Cover(image_data, imageformat=fmt)]
logger.info("Album art successfully embedded.")
return True
except Exception as exc:
logger.error("Error embedding album art: %s", exc)
return False
def download_cover_art(album_info: dict, target_dir: str, context: dict = None):

View file

@ -0,0 +1,281 @@
"""Resolve (and persist) the canonical release for an album — Stage 2 of #765.
Stage 1 gave us the pure scorer (``core.metadata.canonical_version``). This
module turns it into an end-to-end resolver: gather the album's candidate
releases (one per metadata-source ID it has), score each against the on-disk
files, and return the best fit. Wiring (backfill job / enrichment hook) and the
DB store live alongside; the decision logic here is kept dependency-injected
(``fetch_tracklist`` is passed in) so it's fully unit-testable without live APIs
or real files.
Still NO consumer reads the result in Stage 2 populating the columns is
behavior-neutral. Stages 3-4 wire the Reorganizer and Track Number Repair to
read it.
"""
from __future__ import annotations
from typing import Any, Callable, Dict, List, Optional
from core.metadata.canonical_version import (
score_release_against_files,
score_release_detail,
)
# Source-selection modes (a per-job setting). See resolve_canonical_for_album.
MODE_ACTIVE_PREFERRED = "active_preferred" # default: use the active source if it fits, else best-fit
MODE_ACTIVE_ONLY = "active_only" # only ever the active source
MODE_BEST_FIT = "best_fit" # whichever source fits the files best
VALID_MODES = (MODE_ACTIVE_PREFERRED, MODE_ACTIVE_ONLY, MODE_BEST_FIT)
def resolve_canonical_for_album(
*,
album_source_ids: Dict[str, str],
file_tracks: List[Dict[str, Any]],
fetch_tracklist: Callable[[str, str], Optional[List[Dict[str, Any]]]],
source_priority: List[str],
min_score: float = 0.5,
mode: str = MODE_ACTIVE_PREFERRED,
primary_source: Optional[str] = None,
) -> Optional[Dict[str, Any]]:
"""Pick the canonical release for one album, honoring the source-selection mode.
``album_source_ids``: ``{source: album_id}`` the album is linked to.
``file_tracks``: on-disk track metadata (``{duration_ms, title}``).
``fetch_tracklist(source, album_id)``: returns that release's tracklist (or
None/[] on miss); injected so callers supply ``get_album_tracks_for_source``
while tests supply a fake.
``source_priority``: source order; ties break toward the earlier source.
``primary_source``: the user's active metadata source (defaults to the first
of ``source_priority``).
Modes:
- ``active_preferred`` (default): use the active source's release when the
album has an ID for it AND it clears ``min_score``; otherwise fall back
to the best-fit among the remaining sources. So it normally respects the
user's configured source but self-heals when that link is clearly wrong.
- ``active_only``: only ever the active source (pinned if it clears the
floor; never considers other sources).
- ``best_fit``: whichever source's release best matches the files.
Returns an enriched dict for the chosen release ``source``, ``album_id``,
``score``, the per-signal breakdown (``count_fit``/``duration_fit``/
``title_fit``), ``file_track_count`` vs ``release_track_count``, and a
``candidates`` list of everything it scored (so a finding can show WHY the
pick won and what it beat). ``None`` when there are no files, no resolvable
candidates, or nothing clears ``min_score``."""
if not file_tracks:
return None
primary = primary_source or (source_priority[0] if source_priority else None)
scored: List[Dict[str, Any]] = [] # every source we actually scored
def _score(source: Optional[str]) -> Optional[Dict[str, Any]]:
if not source or any(e['source'] == source for e in scored):
return next((e for e in scored if e['source'] == source), None)
album_id = album_source_ids.get(source)
if not album_id:
return None
try:
tracks = fetch_tracklist(source, str(album_id))
except Exception:
tracks = None
if not tracks:
return None
entry = {
'source': source, 'album_id': str(album_id),
'track_count': len(tracks), 'score': round(score_release_against_files(file_tracks, tracks), 4),
'_tracks': tracks,
}
scored.append(entry)
return entry
winner: Optional[Dict[str, Any]] = None
# Active-source modes: try the primary first.
if mode in (MODE_ACTIVE_ONLY, MODE_ACTIVE_PREFERRED):
p = _score(primary)
if p and p['score'] >= min_score:
winner = p
elif mode == MODE_ACTIVE_ONLY:
return None # never consider other sources
# best_fit, or active_preferred fallback: score the rest and pick the best.
if winner is None:
for source in source_priority:
_score(source)
best = None
for e in scored: # source_priority order -> strictly-greater = priority tiebreak
if best is None or e['score'] > best['score'] + 1e-9:
best = e
if best and best['score'] >= min_score:
winner = best
if winner is None:
return None
detail = score_release_detail(file_tracks, winner['_tracks'])
# Pinned-release track titles — already fetched, so free. Capped so a giant
# box set can't bloat the finding's details_json.
release_titles = [
(t.get('title') or t.get('name') or '') for t in winner['_tracks']
][:60]
return {
'source': winner['source'],
'album_id': winner['album_id'],
'score': winner['score'],
'file_track_count': detail['file_track_count'],
'release_track_count': detail['release_track_count'],
'count_fit': detail['count_fit'],
'duration_fit': detail['duration_fit'],
'title_fit': detail['title_fit'],
'release_track_titles': release_titles,
'candidates': [
{'source': e['source'], 'album_id': e['album_id'],
'track_count': e['track_count'], 'score': e['score']}
for e in scored
],
}
def _item_get(item: Any, key: str, default: Any = None) -> Any:
"""Read ``key`` from a track item that may be a dict or an object."""
return item.get(key, default) if isinstance(item, dict) else getattr(item, key, default)
def default_fetch_tracklist(source: str, album_id: str) -> Optional[List[Dict[str, Any]]]:
"""Production ``fetch_tracklist``: pull a release's tracklist from a metadata
source and normalise to ``{title, track_number, duration_ms}``. Duration is
best-effort (not every source exposes it); when absent the scorer just leans
on track-count + title. Returns None on any failure."""
try:
from core.metadata_service import get_album_tracks_for_source
data = get_album_tracks_for_source(source, album_id)
except Exception:
return None
items = data if isinstance(data, list) else (
(data.get('items') or data.get('tracks') or []) if isinstance(data, dict) else []
)
if isinstance(items, dict): # {'tracks': {'items': [...]}}
items = items.get('items') or []
out: List[Dict[str, Any]] = []
for it in items:
dur = _item_get(it, 'duration_ms')
if dur is None:
secs = _item_get(it, 'duration') # some sources give seconds
dur = int(secs * 1000) if isinstance(secs, (int, float)) and secs else None
out.append({
'title': _item_get(it, 'name') or _item_get(it, 'title') or '',
'track_number': _item_get(it, 'track_number'),
'duration_ms': dur,
})
return out or None
def _lookup_artist_thumb(db, artist_id) -> Optional[str]:
"""Best-effort artist thumb URL by id. Returns None on missing column / any
error (the artists table doesn't have thumb_url in every schema)."""
if not artist_id:
return None
conn = None
try:
conn = db._get_connection()
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(artists)")
if 'thumb_url' not in {r[1] for r in cursor.fetchall()}:
return None
cursor.execute("SELECT thumb_url FROM artists WHERE id = ?", (str(artist_id),))
row = cursor.fetchone()
return (row[0] or None) if row else None
except Exception:
return None
finally:
if conn:
conn.close()
def resolve_and_store_canonical_for_album(
db,
album_id,
*,
fetch_tracklist: Optional[Callable[[str, str], Any]] = None,
source_priority: Optional[List[str]] = None,
min_score: float = 0.5,
store: bool = True,
mode: str = MODE_ACTIVE_PREFERRED,
) -> Optional[Dict[str, Any]]:
"""Gather an album's source IDs + its tracks' (duration, title) from the DB,
resolve the best-fit canonical release, and (when ``store``) persist it.
Returns the resolved ``{source, album_id, score}`` or None when unresolved.
``store=False`` resolves without writing used by the backfill job's dry run.
Uses the SAME album/source-id loader the Reorganizer uses
(``load_album_and_tracks`` + ``_extract_source_ids``) so the canonical is
chosen over exactly the source IDs the reorganizer sees. Scores off the DB
track rows' ``duration`` (stored in ms) + ``title`` — the library's view of
the files so no per-file disk reads are needed."""
from core.library_reorganize import _extract_source_ids, load_album_and_tracks
album_data, tracks = load_album_and_tracks(db, album_id)
if not album_data or not tracks:
return None
source_ids = {s: v for s, v in _extract_source_ids(album_data).items() if v}
if not source_ids:
return None
file_tracks = [
{'duration_ms': t.get('duration') or 0, 'title': t.get('title') or ''}
for t in tracks
]
if fetch_tracklist is None:
fetch_tracklist = default_fetch_tracklist
primary_source = None
if source_priority is None:
try:
from core.metadata_service import get_primary_source, get_source_priority
primary_source = get_primary_source()
source_priority = get_source_priority(primary_source)
except Exception:
source_priority = list(source_ids.keys())
result = resolve_canonical_for_album(
album_source_ids=source_ids,
file_tracks=file_tracks,
fetch_tracklist=fetch_tracklist,
source_priority=source_priority,
min_score=min_score,
mode=mode,
primary_source=primary_source,
)
if result:
# Album/artist/art context for richer findings (read from the row we
# already loaded — no extra query). Storage only uses source/id/score.
result['album_title'] = album_data.get('title') or ''
result['artist_name'] = album_data.get('artist_name') or ''
# Free context off the album row + the data we already gathered.
if album_data.get('year'):
result['year'] = album_data['year']
result['db_track_count'] = album_data.get('track_count') or len(file_tracks)
if album_data.get('duration'):
result['db_duration_ms'] = album_data['duration']
result['linked_sources'] = source_ids # {source: album_id} the album points at now
result['file_track_titles'] = [ft.get('title') or '' for ft in file_tracks][:60]
if album_data.get('thumb_url'):
result['album_thumb_url'] = album_data['thumb_url']
# Artist thumb via a guarded lookup (not the shared album loader — some
# schemas have no artists.thumb_url column). Only runs for resolved
# albums, so no cost on the no-source-id short-circuit majority.
artist_thumb = _lookup_artist_thumb(db, album_data.get('artist_id'))
if artist_thumb:
result['artist_thumb_url'] = artist_thumb
if store:
db.set_album_canonical(album_id, result['source'], result['album_id'], result['score'])
return result
__all__ = [
"resolve_canonical_for_album",
"resolve_and_store_canonical_for_album",
"default_fetch_tracklist",
]

View file

@ -0,0 +1,210 @@
"""Pick the canonical album release by best-fit to the user's actual files.
Issue #765 / #767-Bug2: SoulSync never pins ONE canonical album version per
album, so the Library Reorganizer, Track Number Repair, and tagging each
re-resolve independently and can land on different releases (standard vs
deluxe; Spotify vs MusicBrainz track numbering) and contradict each other.
This module is the pure, testable heart of the fix: given the metadata of the
files actually on disk and a set of candidate releases, score each release by
how well it FITS those files and pick the best. "Best-fit to the files" means:
- track-count fit a 17-track deluxe is a poor fit for 11 files on disk
- duration alignment each file should line up with a release track by length
- title overlap a tiebreaker / sanity check
What this does and does NOT solve:
- It DOES pick the right EDITION (standard vs deluxe) the discriminating
signal is track count + durations.
- It does NOT (and cannot) decide which of two listings of the SAME album is
"more correct" when they differ only in track numbering (same files match
both equally). Instead ``pick_canonical_release`` is DETERMINISTIC and
breaks ties toward the earlier candidate so the caller passes candidates
in source-priority order and every tool that reads the pinned result agrees
on the same release. Agreement is what resolves #765, not picking a
"winner" of the numbering disagreement.
Pure, no I/O. Callers fetch candidate tracklists and read on-disk file metadata;
this module only scores.
"""
from __future__ import annotations
import re
from difflib import SequenceMatcher
from typing import Any, Dict, List, Optional, Tuple
# Weights for the three fit signals. Count + duration dominate because "matches
# my files" is fundamentally about having the right NUMBER of the right-LENGTH
# tracks; title is a tiebreaker. Missing signals are dropped and the present
# ones renormalized (see _combine).
_W_COUNT = 0.4
_W_DURATION = 0.4
_W_TITLE = 0.2
_DEFAULT_DURATION_TOLERANCE_MS = 3000 # ±3s — covers encode/version length jitter
_DEFAULT_MIN_SCORE = 0.5 # never pin below this — leave unresolved
_TITLE_FUZZY_THRESHOLD = 0.85
def _norm_title(text: str) -> str:
"""Lowercase, drop bracketed qualifiers ((feat. …), [Remastered]), strip
punctuation, collapse whitespace."""
if not text:
return ""
t = str(text).lower()
t = re.sub(r"[\(\[].*?[\)\]]", "", t)
t = re.sub(r"[^a-z0-9 ]", " ", t)
return " ".join(t.split())
def _count_fit(n_files: int, n_release: int) -> float:
"""1.0 when track counts match; decays with the relative difference."""
if n_files <= 0 or n_release <= 0:
return 0.0
return 1.0 - min(1.0, abs(n_files - n_release) / max(n_files, n_release))
def _duration_fit(
file_tracks: List[Dict[str, Any]],
release_tracks: List[Dict[str, Any]],
tolerance_ms: int,
) -> Optional[float]:
"""Fraction of tracks that line up by duration (greedy nearest match within
tolerance), over the larger of the two track counts so missing or extra
tracks are penalised. Returns ``None`` when neither side has durations."""
f_durs = [int(f["duration_ms"]) for f in file_tracks if f.get("duration_ms")]
r_durs = [int(r["duration_ms"]) for r in release_tracks if r.get("duration_ms")]
if not f_durs or not r_durs:
return None
used = [False] * len(r_durs)
matched = 0
for fd in f_durs:
best_j, best_diff = -1, tolerance_ms + 1
for j, rd in enumerate(r_durs):
if used[j]:
continue
diff = abs(fd - rd)
if diff <= tolerance_ms and diff < best_diff:
best_diff, best_j = diff, j
if best_j >= 0:
used[best_j] = True
matched += 1
denom = max(len(file_tracks), len(release_tracks))
return matched / denom if denom else 0.0
def _title_fit(
file_tracks: List[Dict[str, Any]],
release_tracks: List[Dict[str, Any]],
) -> Optional[float]:
"""Fraction of files whose title matches some release title (exact-normalised
or fuzzy), over the larger track count. ``None`` when titles are absent."""
f_titles = [_norm_title(f.get("title", "")) for f in file_tracks]
f_titles = [t for t in f_titles if t]
r_titles = [_norm_title(r.get("title", "")) for r in release_tracks]
r_titles = [t for t in r_titles if t]
if not f_titles or not r_titles:
return None
r_set = set(r_titles)
matched = 0
for ft in f_titles:
if ft in r_set or any(
SequenceMatcher(None, ft, rt).ratio() >= _TITLE_FUZZY_THRESHOLD
for rt in r_titles
):
matched += 1
denom = max(len(file_tracks), len(release_tracks))
return matched / denom if denom else 0.0
def _combine(parts: List[Tuple[Optional[float], float]]) -> float:
"""Weighted mean over present (non-None) components, renormalising weights."""
present = [(v, w) for v, w in parts if v is not None]
total_w = sum(w for _, w in present)
if total_w <= 0:
return 0.0
return sum(v * w for v, w in present) / total_w
def score_release_against_files(
file_tracks: List[Dict[str, Any]],
release_tracks: List[Dict[str, Any]],
*,
duration_tolerance_ms: int = _DEFAULT_DURATION_TOLERANCE_MS,
) -> float:
"""Score 0.01.0 of how well ``release_tracks`` fits the on-disk
``file_tracks``. Each track dict may carry ``duration_ms`` and ``title``;
missing signals are dropped and the rest renormalised so the function never
crashes on sparse metadata (it just leans on what's available)."""
if not file_tracks or not release_tracks:
return 0.0
count = _count_fit(len(file_tracks), len(release_tracks))
dur = _duration_fit(file_tracks, release_tracks, duration_tolerance_ms)
title = _title_fit(file_tracks, release_tracks)
return _combine([(count, _W_COUNT), (dur, _W_DURATION), (title, _W_TITLE)])
def score_release_detail(
file_tracks: List[Dict[str, Any]],
release_tracks: List[Dict[str, Any]],
*,
duration_tolerance_ms: int = _DEFAULT_DURATION_TOLERANCE_MS,
) -> Dict[str, Any]:
"""Like ``score_release_against_files`` but returns the per-signal breakdown
so a UI can show WHY a release scored the way it did. ``duration_fit`` /
``title_fit`` are ``None`` when that signal was absent."""
if not file_tracks or not release_tracks:
return {
'score': 0.0, 'count_fit': 0.0, 'duration_fit': None, 'title_fit': None,
'release_track_count': len(release_tracks), 'file_track_count': len(file_tracks),
}
count = _count_fit(len(file_tracks), len(release_tracks))
dur = _duration_fit(file_tracks, release_tracks, duration_tolerance_ms)
title = _title_fit(file_tracks, release_tracks)
score = _combine([(count, _W_COUNT), (dur, _W_DURATION), (title, _W_TITLE)])
return {
'score': round(score, 4),
'count_fit': round(count, 3),
'duration_fit': round(dur, 3) if dur is not None else None,
'title_fit': round(title, 3) if title is not None else None,
'release_track_count': len(release_tracks),
'file_track_count': len(file_tracks),
}
def pick_canonical_release(
file_tracks: List[Dict[str, Any]],
candidates: List[Dict[str, Any]],
*,
min_score: float = _DEFAULT_MIN_SCORE,
duration_tolerance_ms: int = _DEFAULT_DURATION_TOLERANCE_MS,
) -> Tuple[Optional[Dict[str, Any]], float]:
"""Choose the best-fit candidate release for the on-disk files.
``candidates`` is a list of dicts each with a ``'tracks'`` list (plus any
caller fields like ``source``/``album_id``, returned untouched). **Pass
candidates in source-priority order** ties break toward the EARLIER one,
so the choice is deterministic and priority-respecting (this is what makes
every tool agree, #765).
Returns ``(best_candidate, score)``, or ``(None, best_score)`` when nothing
clears ``min_score`` so a low-confidence guess is never pinned (the caller
leaves the album unresolved and falls back to today's behaviour)."""
best: Optional[Dict[str, Any]] = None
best_score = 0.0
for cand in candidates:
score = score_release_against_files(
file_tracks, cand.get("tracks") or [],
duration_tolerance_ms=duration_tolerance_ms,
)
# Strictly-greater so equal scores keep the earlier (higher-priority)
# candidate — deterministic tiebreak.
if score > best_score + 1e-9:
best, best_score = cand, score
if best is None or best_score < min_score:
return None, best_score
return best, best_score
__all__ = ["score_release_against_files", "pick_canonical_release"]

View file

@ -6,6 +6,10 @@ import os
from types import SimpleNamespace
from typing import Any
from core.metadata.art_preservation import (
restore_embedded_art,
snapshot_embedded_art,
)
from core.metadata.artwork import embed_album_art_metadata
from core.metadata.common import (
get_config_manager,
@ -76,6 +80,8 @@ def enhance_file_metadata(file_path: str, context: dict, artist: dict, album_inf
file_lock = get_file_lock(file_path)
with file_lock:
logger.info("Enhancing metadata for: %s", os.path.basename(file_path))
art_snapshot = [] # defined up front so the except handler can restore
audio_file = None
try:
strip_all_non_audio_tags(file_path)
audio_file = symbols.File(file_path)
@ -83,6 +89,13 @@ def enhance_file_metadata(file_path: str, context: dict, artist: dict, album_inf
logger.error("Could not load audio file with Mutagen: %s", file_path)
return False
# Capture any embedded cover art BEFORE we clear it. The rewrite
# below clears pictures/tags up front, but new art isn't fetched
# until much later (and may fail / be unavailable / be disabled).
# Without this snapshot, every such failure saves the file with
# the art destroyed and nothing put back — issue #764.
art_snapshot = snapshot_embedded_art(audio_file, symbols)
if hasattr(audio_file, "clear_pictures"):
audio_file.clear_pictures()
@ -99,6 +112,9 @@ def enhance_file_metadata(file_path: str, context: dict, artist: dict, album_inf
metadata = extract_source_metadata(context, artist, album_info)
if not metadata:
logger.error("Could not extract source metadata, saving with cleared tags.")
# Don't destroy the original art just because we couldn't
# enrich the tags — put it back before saving.
restore_embedded_art(audio_file, symbols, art_snapshot)
save_audio_file(audio_file, symbols)
return True
@ -202,6 +218,13 @@ def enhance_file_metadata(file_path: str, context: dict, artist: dict, album_inf
elif isinstance(audio_file, symbols.MP4):
audio_file["----:com.apple.iTunes:QUALITY"] = [symbols.MP4FreeForm(quality.encode("utf-8"))]
# If art embedding was skipped/disabled or produced nothing (no
# URL, download failed, rejected by the min-resolution guard),
# the file still has the pictures we cleared above. Restore the
# original so import never strips existing art (#764). No-op when
# new art was embedded — that path is unchanged.
restore_embedded_art(audio_file, symbols, art_snapshot)
save_audio_file(audio_file, symbols)
verified = verify_metadata_written(file_path)
@ -219,4 +242,18 @@ def enhance_file_metadata(file_path: str, context: dict, artist: dict, album_inf
logger.warning("[Metadata Debug] Artist: %s", artist.get("name", "MISSING") if artist else "None")
logger.warning("[Metadata Debug] Album info: %s", album_info.get("album_name", "MISSING") if album_info else "None")
logger.error("[Metadata Debug] Traceback:\n%s", traceback.format_exc())
# We cleared the file's art early; if the rewrite then crashed
# before re-embedding, the on-disk file (already saved cleared at
# the start) would be left art-less. Best-effort: put the original
# art back and persist it so a mid-enrichment crash never destroys
# the cover (#764). Guarded so a failure here can't mask the
# original error.
try:
if audio_file is not None and art_snapshot and restore_embedded_art(
audio_file, symbols, art_snapshot
):
save_audio_file(audio_file, symbols)
logger.info("Restored original cover art after enrichment error.")
except Exception as restore_exc:
logger.debug("Art restore after error failed: %s", restore_exc)
return False

View file

@ -269,10 +269,17 @@ def iter_musicmap_similar_artist_events(
except requests.exceptions.RequestException as exc:
logger.debug("Error fetching MusicMap for %s: %s", artist_name, exc)
# A 404 from MusicMap means the artist simply has no map page — that's a
# not-found, not a fetch failure. Surface the real HTTP status (when the
# exception carries a response) so callers classify it correctly instead
# of flattening every network-layer error to a 502 "error". Timeouts /
# connection errors carry no response → fall back to 502 (still an error).
resp = getattr(exc, 'response', None)
status_code = getattr(resp, 'status_code', None) or 502
yield {
'type': 'error',
'error': f'Failed to fetch from MusicMap: {exc}',
'status_code': 502,
'status_code': status_code,
}
except ValueError as exc:
status_code = 404 if 'Could not find artist map on MusicMap' in str(exc) else 400

View file

@ -166,6 +166,16 @@ class MusicBrainzWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('musicbrainz')
if _prio:
_pi = priority_pending_item(cursor, 'musicbrainz', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -3,6 +3,7 @@ import hashlib
import secrets
from typing import List, Optional, Dict, Any
from datetime import datetime
from urllib.parse import urlencode
import json
from utils.logging_config import get_logger
from config.settings import config_manager
@ -316,6 +317,42 @@ class NavidromeClient(MediaServerClient):
'f': 'json' # Response format
}
# Fixed salt for cover-art URLs ONLY. Subsonic token auth (t=md5(password
# +salt), s=salt) does not require a unique salt per request, and a stable
# one makes the cover URL deterministic — so the image cache and the
# browser cache actually HIT. The rotating salt from _generate_auth_params
# would make every request a unique URL → cache miss every time + a dead,
# never-reused cache row per fetch (#766 review). The password is never
# exposed either way (only its salted md5).
_COVER_ART_SALT = 'soulsync-cover'
def build_cover_art_url(self, cover_id, size=None) -> Optional[str]:
"""Absolute, Subsonic-authenticated getCoverArt URL for ``cover_id``.
Deterministic for a given (server, password, cover_id) so it caches.
The web layer proxies this to the browser (sync editor + modals).
Returns ``None`` when not connected or no id was supplied. #766: the
``/api/navidrome/cover/<id>`` route had no working URL behind it, so
every Navidrome cover came back blank."""
if not self.base_url or not cover_id:
return None
if not self.username or not self.password:
return None
salt = self._COVER_ART_SALT
token = hashlib.md5((self.password + salt).encode()).hexdigest()
params = {
'u': self.username,
't': token,
's': salt,
'v': '1.16.1',
'c': 'SoulSync',
'f': 'json', # harmless for getCoverArt — it returns image binary
'id': str(cover_id),
}
if size:
params['size'] = str(size)
return f"{self.base_url}/rest/getCoverArt?{urlencode(params)}"
# Subsonic endpoints that modify data — use POST to avoid URL length limits
_WRITE_ENDPOINTS = frozenset({
'createPlaylist', 'updatePlaylist', 'deletePlaylist',

View file

@ -0,0 +1,182 @@
"""Download missing tracks into playlist-folder layout for mirrored playlists."""
from __future__ import annotations
import json
import logging
import uuid
from typing import Any, Callable, Dict, List, Optional
logger = logging.getLogger(__name__)
def mirrored_tracks_to_download_json(tracks: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Convert mirrored playlist rows to the payload expected by the download master."""
out: List[Dict[str, Any]] = []
for t in tracks:
extra = {}
if t.get('extra_data'):
try:
extra = json.loads(t['extra_data']) if isinstance(t['extra_data'], str) else t['extra_data']
except (json.JSONDecodeError, TypeError):
extra = {}
if extra.get('discovered') and extra.get('matched_data'):
md = extra['matched_data']
album_raw = md.get('album', '')
album_obj = album_raw if isinstance(album_raw, dict) else {'name': album_raw or ''}
entry = {
'name': md.get('name', ''),
'artists': md.get('artists', [{'name': t.get('artist_name', '')}]),
'album': album_obj,
'duration_ms': md.get('duration_ms', 0),
'id': md.get('id', ''),
}
if md.get('track_number'):
entry['track_number'] = md['track_number']
if md.get('disc_number'):
entry['disc_number'] = md['disc_number']
out.append(entry)
continue
hint = extra.get('spotify_hint', {})
track_image = (t.get('image_url') or '').strip()
album_obj = {
'name': (t.get('album_name') or '').strip(),
'images': [{'url': track_image, 'height': 300, 'width': 300}] if track_image else [],
}
if hint.get('id') and hint.get('name'):
hint_artists = hint.get('artists', [])
if hint_artists and isinstance(hint_artists[0], str):
hint_artists = [{'name': a} for a in hint_artists]
elif not hint_artists:
hint_artists = [{'name': t.get('artist_name', '')}]
out.append({
'name': hint['name'],
'artists': hint_artists,
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': hint['id'],
})
elif t.get('source_track_id') and (t.get('track_name') or '').strip():
out.append({
'name': t['track_name'].strip(),
'artists': [{'name': (t.get('artist_name') or '').strip() or 'Unknown Artist'}],
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': t['source_track_id'],
})
return out
def run_playlist_organize_download(
deps: Any,
*,
mirrored_playlist_id: int,
profile_id: int = 1,
get_batch_max_concurrent: Callable[[bool], int],
run_full_missing_tracks_process: Callable[..., Any],
record_sync_history_start: Optional[Callable[..., Any]] = None,
detect_sync_source: Optional[Callable[[str], str]] = None,
) -> Dict[str, Any]:
"""Queue a playlist-folder missing-tracks batch for one mirrored playlist."""
db = deps.get_database()
pl = db.get_mirrored_playlist(int(mirrored_playlist_id))
if not pl:
return {'status': 'error', 'reason': 'Playlist not found'}
source_playlist_ref = (pl.get('source_playlist_id') or '').strip()
source = (pl.get('source') or 'spotify').strip() or 'spotify'
tracks = db.get_mirrored_playlist_tracks(int(mirrored_playlist_id))
tracks_json = mirrored_tracks_to_download_json(tracks)
if not tracks_json:
return {'status': 'skipped', 'reason': 'No processable tracks'}
batch_id = str(uuid.uuid4())
playlist_id = str(mirrored_playlist_id)
playlist_name = pl.get('name', 'Unknown Playlist')
download_batches = deps.get_download_batches()
tasks_lock = deps.tasks_lock
with tasks_lock:
active_analysis = sum(
1 for batch in download_batches.values() if batch.get('phase') == 'analysis'
)
if active_analysis >= 3:
return {'status': 'error', 'reason': 'Too many analysis processes running'}
download_batches[batch_id] = {
'phase': 'analysis',
'playlist_id': playlist_id,
'playlist_name': playlist_name,
'queue': [],
'active_count': 0,
'max_concurrent': get_batch_max_concurrent(False),
'permanently_failed_tracks': [],
'cancelled_tracks': set(),
'queue_index': 0,
'analysis_total': len(tracks_json),
'profile_id': profile_id,
'analysis_processed': 0,
'analysis_results': [],
'force_download_all': False,
'ignore_manual_matches': False,
'playlist_folder_mode': True,
'is_album_download': False,
'album_context': None,
'artist_context': None,
'wing_it': False,
'batch_source': source,
'auto_initiated': False,
'organize_by_playlist': True,
'source_playlist_ref': source_playlist_ref,
'mirrored_playlist_id': int(mirrored_playlist_id),
}
if record_sync_history_start:
try:
record_sync_history_start(
batch_id,
playlist_id,
playlist_name,
tracks_json,
False,
None,
None,
True,
source_page='automation',
)
except Exception as hist_err:
logger.debug("organize download sync history: %s", hist_err)
try:
deps.missing_download_executor.submit(
run_full_missing_tracks_process,
batch_id,
playlist_id,
tracks_json,
)
except Exception as submit_err:
# Don't leave the batch stranded in 'analysis' holding one of the limited
# analysis slots if the executor refuses the job.
logger.error("[Organize Download] Failed to submit batch %s: %s", batch_id, submit_err)
with tasks_lock:
download_batches.pop(batch_id, None)
return {'status': 'error', 'reason': f'submit failed: {submit_err}'}
logger.info(
"[Organize Download] Started batch %s for mirrored playlist %s (%s tracks)",
batch_id,
playlist_name,
len(tracks_json),
)
return {
'status': 'started',
'batch_id': batch_id,
'track_count': len(tracks_json),
}
__all__ = ['mirrored_tracks_to_download_json', 'run_playlist_organize_download']

View file

@ -186,6 +186,16 @@ class QobuzWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('qobuz')
if _prio:
_pi = priority_pending_item(cursor, 'qobuz', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -44,6 +44,7 @@ _JOB_MODULES = [
'core.repair_jobs.live_commentary_cleaner',
'core.repair_jobs.unknown_artist_fixer',
'core.repair_jobs.discography_backfill',
'core.repair_jobs.canonical_version_resolve',
]

View file

@ -100,6 +100,9 @@ class RepairJob(ABC):
default_enabled: bool = False
default_interval_hours: int = 24
default_settings: Dict[str, Any] = {}
# Optional {setting_key: [allowed values]} — the UI renders a dropdown for
# these instead of a free-text box. Keys not listed render by value type.
setting_options: Dict[str, list] = {}
auto_fix: bool = False
@abstractmethod

View file

@ -0,0 +1,221 @@
"""Resolve Canonical Album Versions — backfill job (#765 Stage 2 trigger).
Pins each album's canonical release (best-fit to its files) so the Library
Reorganizer (Stage 3) and Track Number Repair (Stage 4) resolve the SAME
release and stop contradicting each other. The resolution logic lives in the
tested core.metadata.canonical_resolver; this job is the opt-in, rate-limited,
progress-reported bulk runner.
Opt-in (``default_enabled = False``) because resolving compares an album's
candidate releases across sources, which costs metadata-source API calls done
once per album, then stored. Albums that already have a canonical are skipped.
"""
import os
from typing import Optional
from core.metadata.canonical_resolver import resolve_and_store_canonical_for_album
from core.repair_jobs import register_job
from core.repair_jobs.base import JobContext, JobResult, RepairJob
from utils.logging_config import get_logger
logger = get_logger("repair_job.canonical_version")
def _pct(v) -> str:
return f"{round(v * 100)}%" if isinstance(v, (int, float)) else "n/a"
def _describe_pin(resolved: dict) -> str:
"""Human-readable, judge-able explanation of WHY this release was chosen."""
artist = resolved.get('artist_name') or ''
album = resolved.get('album_title') or ''
head = f"{artist}{album}".strip("") or resolved.get('album_id', '')
year = resolved.get('year')
if year:
head += f" ({year})"
lines = [
f"{head}" if head else "",
f"Pin {resolved['source']} release {resolved['album_id']} "
f"(confidence {_pct(resolved.get('score'))}).",
f"Fit to your library: {resolved.get('file_track_count', '?')} files vs "
f"{resolved.get('release_track_count', '?')} tracks on this release — "
f"track count {_pct(resolved.get('count_fit'))}, "
f"durations {_pct(resolved.get('duration_fit'))}, "
f"titles {_pct(resolved.get('title_fit'))}.",
]
# What the album is currently linked to vs what we'd pin.
linked = resolved.get('linked_sources') or {}
if linked:
linked_str = ", ".join(f"{s}={i}" for s, i in linked.items())
lines.append(f"Currently linked: {linked_str} → pinning {resolved['source']}.")
others = [c for c in resolved.get('candidates', []) if c.get('source') != resolved.get('source')]
if others:
comp = ", ".join(
f"{c['source']} {_pct(c['score'])} ({c['track_count']} tk)" for c in others
)
lines.append(f"Beat: {comp}.")
elif len(resolved.get('candidates', [])) == 1:
lines.append("Only this source had a release linked for this album.")
# Track listing of the pinned release (so you can eyeball the actual songs).
titles = resolved.get('release_track_titles') or []
if titles:
shown = "; ".join(f"{i+1}. {t}" for i, t in enumerate(titles[:25]))
more = f" (+{len(titles) - 25} more)" if len(titles) > 25 else ""
lines.append(f"Release tracks: {shown}{more}")
return "\n".join(l for l in lines if l)
@register_job
class CanonicalVersionResolveJob(RepairJob):
job_id = 'canonical_version_resolve'
display_name = 'Resolve Canonical Album Versions'
description = (
'Pins the best-fit release per album (by track count + durations) so '
'reorganize and track-number repair agree (dry run by default)'
)
help_text = (
'For each album, compares the releases its linked metadata sources point '
'at and pins the one that best matches the files you actually have '
'(track count + durations + titles). The Library Reorganizer and Track '
'Number Repair then both use that pinned release, so they stop '
'contradicting each other (e.g. standard vs deluxe, or Spotify vs '
'MusicBrainz track numbering).\n\n'
'In dry run mode (default) it reports what it would pin without saving. '
'Disable dry run to store the pins. Albums already pinned are skipped.\n\n'
'Opt-in: resolving costs metadata-source API calls (once per album).'
)
icon = 'repair-icon-tracknumber'
default_enabled = False
default_interval_hours = 168 # weekly, but disabled by default
default_settings = {
'dry_run': True,
'min_score': 0.5,
# Which source's release to pin: 'active_preferred' (default — use your
# active metadata source when it fits, else best-fit fallback),
# 'active_only' (only ever the active source), or 'best_fit' (whichever
# source matches the files best, regardless of which it is).
'source_selection': 'active_preferred',
}
# Render source_selection as a dropdown (not a text box) in the settings UI.
setting_options = {
'source_selection': ['active_preferred', 'active_only', 'best_fit'],
}
auto_fix = True
def _get_settings(self, context: JobContext) -> dict:
merged = dict(self.default_settings)
if context.config_manager:
merged.update(context.config_manager.get(f'repair.jobs.{self.job_id}.settings', {}) or {})
return merged
def _load_album_ids(self, db, active_server: Optional[str]) -> list:
conn = None
try:
conn = db._get_connection()
cursor = conn.cursor()
if active_server:
cursor.execute(
"SELECT al.id, al.title FROM albums al WHERE al.server_source = ? ORDER BY al.id",
(active_server,),
)
else:
cursor.execute("SELECT al.id, al.title FROM albums al ORDER BY al.id")
return [(row[0], row[1]) for row in cursor.fetchall()]
except Exception as e:
logger.error("Error loading albums for canonical resolve: %s", e)
return []
finally:
if conn:
conn.close()
def scan(self, context: JobContext) -> JobResult:
result = JobResult()
settings = self._get_settings(context)
dry_run = settings.get('dry_run', True)
min_score = settings.get('min_score', 0.5)
mode = settings.get('source_selection', 'active_preferred')
active_server = None
if context.config_manager:
try:
active_server = context.config_manager.get_active_media_server()
except Exception as e:
logger.warning("Couldn't read active media server: %s", e)
albums = self._load_album_ids(context.db, active_server)
total = len(albums)
if context.report_progress:
mode = 'DRY RUN' if dry_run else 'LIVE'
context.report_progress(
phase=f'Resolving canonical versions for {total} albums ({mode})...',
total=total, scanned=0, log_type='info',
)
for i, (album_id, album_title) in enumerate(albums):
if context.check_stop():
return result
if i % 20 == 0 and context.wait_if_paused():
return result
# Skip albums already pinned — one-time cost per album.
try:
if context.db.get_album_canonical(album_id):
result.skipped += 1
result.scanned += 1
continue
except Exception: # noqa: S110 — best-effort skip check; on read error just resolve it
pass
try:
resolved = resolve_and_store_canonical_for_album(
context.db, album_id, min_score=min_score, store=not dry_run, mode=mode,
)
except Exception as e:
logger.warning("Canonical resolve failed for album %s ('%s'): %s",
album_id, album_title, e)
result.errors += 1
result.scanned += 1
continue
result.scanned += 1
if resolved:
if dry_run and context.create_finding:
artist = resolved.get('artist_name') or ''
label = f"{artist}{album_title}" if artist else (album_title or str(album_id))
inserted = context.create_finding(
job_id=self.job_id,
finding_type='canonical_version',
severity='info',
entity_type='album',
entity_id=str(album_id),
file_path=None,
title=f'Pin {resolved["source"]} as canonical: {label}',
description=_describe_pin(resolved),
details={'album_id': str(album_id), **resolved},
)
if inserted:
result.findings_created += 1
else:
result.findings_skipped_dedup += 1
elif not dry_run:
result.auto_fixed += 1
if context.report_progress and (i + 1) % 25 == 0:
context.report_progress(scanned=i + 1, total=total,
phase=f'Resolving ({i+1}/{total})...')
return result
def estimate_scope(self, context: JobContext) -> int:
active_server = None
if context.config_manager:
try:
active_server = context.config_manager.get_active_media_server()
except Exception: # noqa: S110 — best-effort; fall back to no server filter
pass
return len(self._load_album_ids(context.db, active_server))

View file

@ -1,5 +1,9 @@
"""Missing Cover Art Filler Job — finds albums without artwork and locates art from APIs."""
import os
import re
from core.metadata.art_apply import album_has_art_on_disk
from core.metadata_service import get_client_for_source, get_primary_source, get_source_priority
from core.repair_jobs import register_job
from core.repair_jobs.base import JobContext, JobResult, RepairJob
@ -7,6 +11,33 @@ from utils.logging_config import get_logger
logger = get_logger("repair_job.cover_art")
# Stopwords dropped before comparing album/artist names so trivial words
# ("the", "and") don't make two different names look like a match.
_NAME_STOPWORDS = {'the', 'a', 'an', 'and', 'of', 'feat', 'ft', 'featuring'}
def _norm_name(value) -> str:
"""Lowercase, strip bracketed qualifiers (Deluxe/Remaster/feat.) and
punctuation so names can be compared on their significant words."""
s = (value or '').lower()
s = re.sub(r'[\(\[\{].*?[\)\]\}]', ' ', s) # drop (...) [...] qualifiers
s = re.sub(r'\b(?:feat|ft|featuring)\b.*', ' ', s) # drop trailing "feat. X"
s = re.sub(r'[^a-z0-9]+', ' ', s)
return ' '.join(s.split())
def _name_tokens(value) -> set:
return set(_norm_name(value).split()) - _NAME_STOPWORDS
def _names_match(a, b) -> bool:
"""True when two names share all the significant words of the shorter one
(so "Album" matches "Album (Deluxe)", but unrelated titles don't)."""
ta, tb = _name_tokens(a), _name_tokens(b)
if not ta or not tb:
return False
return ta <= tb or tb <= ta
@register_job
class MissingCoverArtJob(RepairJob):
@ -59,6 +90,12 @@ class MissingCoverArtJob(RepairJob):
"al.spotify_album_id",
"al.thumb_url",
"ar.thumb_url",
# A representative local track path, so we can check whether the
# album actually has art ON DISK (embedded / cover.jpg) — not just
# whether the DB row has a thumb_url.
("(SELECT t.file_path FROM tracks t WHERE t.album_id = al.id "
"AND t.file_path IS NOT NULL AND t.file_path != '' "
"ORDER BY t.disc_number, t.track_number LIMIT 1) AS rep_path"),
]
column_map = [
("itunes_album_id", "al.itunes_album_id"),
@ -72,12 +109,14 @@ class MissingCoverArtJob(RepairJob):
column_index[alias] = len(select_cols)
select_cols.append(f"{column} AS {alias}")
# Scan every titled album — we decide per-album whether art is
# missing in the DB OR on disk (the file/cover.jpg). The on-disk
# check is cheap-first (a sidecar stat before opening any audio).
cursor.execute(f"""
SELECT {', '.join(select_cols)}
FROM albums al
LEFT JOIN artists ar ON ar.id = al.artist_id
WHERE (al.thumb_url IS NULL OR al.thumb_url = '')
AND al.title IS NOT NULL AND al.title != ''
WHERE al.title IS NOT NULL AND al.title != ''
""")
albums = cursor.fetchall()
except Exception as e:
@ -103,7 +142,7 @@ class MissingCoverArtJob(RepairJob):
if i % 10 == 0 and context.wait_if_paused():
return result
album_id, title, artist_name, spotify_album_id, _, artist_thumb = row[:6]
album_id, title, artist_name, spotify_album_id, album_thumb, artist_thumb, rep_path = row[:7]
source_album_ids = {
'spotify': spotify_album_id,
'itunes': row[column_index['itunes_album_id']] if 'itunes_album_id' in column_index else None,
@ -113,6 +152,14 @@ class MissingCoverArtJob(RepairJob):
}
result.scanned += 1
# Art can be missing in the DB (no thumb_url) and/or on disk (no
# embedded art and no cover.jpg). Skip albums that already have both.
db_missing = not (str(album_thumb).strip() if album_thumb else '')
disk_missing = bool(rep_path) and not album_has_art_on_disk(rep_path)
if not db_missing and not disk_missing:
result.skipped += 1
continue
if context.report_progress:
context.report_progress(
scanned=i + 1, total=total,
@ -154,6 +201,12 @@ class MissingCoverArtJob(RepairJob):
'found_artwork_url': artwork_url,
'spotify_album_id': spotify_album_id,
'artist_thumb_url': artist_thumb or None,
# Where the files live + what was missing, so the
# apply can embed into the audio + write cover.jpg.
'album_folder': os.path.dirname(rep_path) if rep_path else None,
'db_missing': db_missing,
'disk_missing': disk_missing,
'musicbrainz_release_id': None,
}
)
if inserted:
@ -192,19 +245,70 @@ class MissingCoverArtJob(RepairJob):
return artwork_url
if query and hasattr(client, 'search_albums'):
results = client.search_albums(query, limit=1)
if results:
artwork_url = self._extract_artwork_url(results[0])
# Pull a few results and only accept one whose title AND artist
# actually match this album. The old code grabbed results[0]'s
# artwork unconditionally, so a loose full-text search returning
# the wrong album gave the wrong cover.
results = client.search_albums(query, limit=5) or []
for res in results:
if not self._result_matches(res, title, artist_name):
continue
artwork_url = self._extract_artwork_url(res)
if artwork_url:
return artwork_url
candidate_id = self._extract_album_id(results[0])
candidate_id = self._extract_album_id(res)
if candidate_id:
album_data = self._get_album_for_source(source, client, candidate_id)
return self._extract_artwork_url(album_data)
artwork_url = self._extract_artwork_url(album_data)
if artwork_url:
return artwork_url
except Exception as e:
logger.debug("%s art lookup failed for '%s': %s", source.capitalize(), title, e)
return None
@staticmethod
def _result_title_artist(item):
"""Pull (title, artist) from a search result that may be a dict or an
Album-like object, across the various source clients."""
if item is None:
return '', ''
if isinstance(item, dict):
title = item.get('title') or item.get('name') or item.get('album') or ''
artist = item.get('artist') or item.get('artist_name') or ''
if not artist:
artists = item.get('artists') or []
if isinstance(artists, list) and artists:
a0 = artists[0]
artist = a0.get('name', '') if isinstance(a0, dict) else str(a0)
else:
title = getattr(item, 'title', None) or getattr(item, 'name', None) or getattr(item, 'album', None) or ''
artist = getattr(item, 'artist', None) or getattr(item, 'artist_name', None) or ''
if not artist:
arts = getattr(item, 'artists', None) or []
if isinstance(arts, list) and arts:
a0 = arts[0]
artist = a0.get('name', '') if isinstance(a0, dict) else str(a0)
return str(title or ''), str(artist or '')
@classmethod
def _result_matches(cls, result, album_title, album_artist) -> bool:
"""Reject a search result unless it confidently matches the album.
Title must match; if both the result and the album carry an artist, the
artist must match too (the strongest guard against wrong covers). When
the result has no artist to compare, require an exact title match.
"""
r_title, r_artist = cls._result_title_artist(result)
# Title may carry extra qualifiers (Deluxe/Remaster) → allow subset.
if not _names_match(r_title, album_title):
return False
# Artist is the strong guard, so require its significant words to match
# EXACTLY (not subset) — "Different Artist" must NOT match "Artist".
if r_artist and album_artist:
return _name_tokens(r_artist) == _name_tokens(album_artist)
# No artist on the result → require an exact title match instead.
return _norm_name(r_title) == _norm_name(album_title)
@staticmethod
def _get_album_for_source(source, client, album_id):
if source == 'spotify':
@ -248,10 +352,11 @@ class MissingCoverArtJob(RepairJob):
try:
conn = context.db._get_connection()
cursor = conn.cursor()
# Upper bound: every titled album is examined (the per-album DB/disk
# art check decides which actually need filling).
cursor.execute("""
SELECT COUNT(*) FROM albums
WHERE (thumb_url IS NULL OR thumb_url = '')
AND title IS NOT NULL AND title != ''
WHERE title IS NOT NULL AND title != ''
""")
row = cursor.fetchone()
return row[0] if row else 0

View file

@ -275,6 +275,22 @@ class TrackNumberRepairJob(RepairJob):
primary_source = get_primary_source()
source_priority = get_source_priority(primary_source)
# Fallback -1 (#765): a pinned canonical release wins over the whole
# cascade below — so Track Number Repair resolves the SAME release the
# Reorganizer does (Stage 3) and the two stop contradicting each other.
# Gated on the album carrying a canonical; everything below is untouched
# for albums without one (preserving the all-01-album rescue this job
# exists for — the regression we refused to take in a reactive fix).
canonical = _lookup_canonical_from_db(file_track_data, context)
if canonical:
c_source, c_id = canonical
if _is_valid_album_id(c_id):
tracks = _get_album_tracklist(c_source, c_id, cache)
if tracks:
logger.info("[Repair] %s — resolved via canonical %s album ID: %s",
folder_name, c_source, c_id)
return tracks
# Fallback 0: Check DB first. If any tracked file already has source IDs,
# prefer the configured source order and use the first available album ID.
source_album_ids = _lookup_album_ids_from_db(file_track_data, context)
@ -806,6 +822,46 @@ def _update_db_file_path(db, old_path: str, new_path: str):
conn.close()
def _lookup_canonical_from_db(file_track_data: List[Tuple[str, str, Any]],
context: JobContext) -> Optional[Tuple[str, str]]:
"""Return the album's pinned canonical ``(source, album_id)`` or None.
#765: when the album this folder's files belong to has a canonical release
pinned (best-fit to the files), Track Number Repair uses it first so it
agrees with the Reorganizer. Resolves by matching a file path to its DB
track row. None when no DB, no match, columns absent, or unresolved."""
if not context.db:
return None
conn = None
try:
conn = context.db._get_connection()
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(albums)")
cols = {row[1] for row in cursor.fetchall()}
if 'canonical_source' not in cols or 'canonical_album_id' not in cols:
return None
for fpath, _, _ in file_track_data:
cursor.execute(
"""
SELECT al.canonical_source, al.canonical_album_id
FROM tracks t
JOIN albums al ON al.id = t.album_id
WHERE t.file_path = ?
LIMIT 1
""",
(fpath,),
)
row = cursor.fetchone()
if row and row[0] and row[1]:
return (str(row[0]), str(row[1]))
except Exception as e:
logger.debug("Error looking up canonical from DB: %s", e)
finally:
if conn:
conn.close()
return None
def _lookup_album_ids_from_db(file_track_data: List[Tuple[str, str, Any]],
context: JobContext) -> Dict[str, Optional[str]]:
"""Look up album IDs from the database using file paths.

View file

@ -371,6 +371,9 @@ class RepairWorker:
'interval_hours': config['interval_hours'],
'settings': config['settings'],
'default_settings': job.default_settings.copy(),
# Per-setting choice lists so the UI can render a dropdown
# instead of a free-text box (e.g. canonical source_selection).
'setting_options': dict(getattr(job, 'setting_options', {}) or {}),
'last_run': last_run,
'next_run': next_run,
'is_running': self._current_job_id == job_id,
@ -1274,28 +1277,85 @@ class RepairWorker:
return {'success': False, 'error': str(e)}
def _fix_missing_cover_art(self, entity_type, entity_id, file_path, details):
"""Update album thumbnail URL from the found artwork."""
"""Apply found artwork: update the DB thumbnail AND embed art into the
album's audio files + write cover.jpg (using the post-processing
standard, so the user's album_art_order preference is honored)."""
artwork_url = details.get('found_artwork_url')
if not artwork_url:
return {'success': False, 'error': 'No artwork URL found in finding details'}
album_id = details.get('album_id') or entity_id
if not album_id:
return {'success': False, 'error': 'No album ID associated with this finding'}
conn = None
track_paths = []
album_title = details.get('album_title')
artist_name = details.get('artist')
mbid = details.get('musicbrainz_release_id')
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("UPDATE albums SET thumb_url = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
(artwork_url, album_id))
conn.commit()
if cursor.rowcount > 0:
return {'success': True, 'action': 'applied_cover_art',
'message': 'Applied cover art to album'}
return {'success': False, 'error': 'Album not found in database'}
if cursor.rowcount == 0:
return {'success': False, 'error': 'Album not found in database'}
# Pull album metadata + local track paths so we can write art to disk.
cursor.execute("""
SELECT al.title, ar.name, al.musicbrainz_release_id
FROM albums al LEFT JOIN artists ar ON ar.id = al.artist_id
WHERE al.id = ?
""", (album_id,))
meta_row = cursor.fetchone()
if meta_row:
album_title = album_title or meta_row[0]
artist_name = artist_name or meta_row[1]
mbid = mbid or meta_row[2]
cursor.execute("""
SELECT file_path FROM tracks
WHERE album_id = ? AND file_path IS NOT NULL AND file_path != ''
""", (album_id,))
track_paths = [r[0] for r in cursor.fetchall()]
finally:
if conn:
conn.close()
# Resolve container/host path mismatches, keep only files that exist.
download_folder = self._config_manager.get('soulseek.download_path', '') if self._config_manager else None
resolved = []
for p in track_paths:
rp = _resolve_file_path(p, self.transfer_folder, download_folder, config_manager=self._config_manager) or p
if os.path.isfile(rp):
resolved.append(rp)
if not resolved:
# Media-server-only album (no local files): DB thumbnail is all we can set.
return {'success': True, 'action': 'applied_cover_art',
'message': 'Applied cover art to album (database only — no local files found)'}
from core.metadata.art_apply import apply_art_to_album_files
metadata = {
'artist': artist_name, 'album_artist': artist_name,
'album': album_title, 'album_art_url': artwork_url,
'musicbrainz_release_id': mbid,
}
album_info = {
'album_name': album_title, 'album_image_url': artwork_url,
'musicbrainz_release_id': mbid,
}
folder = details.get('album_folder') or os.path.dirname(resolved[0])
art_result = apply_art_to_album_files(resolved, metadata, album_info, folder=folder)
embedded = art_result.get('embedded', 0)
msg = f'Applied cover art: embedded into {embedded}/{len(resolved)} file(s)'
if art_result.get('cover_written'):
msg += ' + wrote cover.jpg'
if embedded == 0 and not art_result.get('cover_written'):
# DB updated but nothing reached disk (e.g. read-only mount).
msg = 'Updated database thumbnail, but could not write art to files (read-only?)'
return {'success': True, 'action': 'applied_cover_art', 'message': msg, 'art_result': art_result}
def _fix_metadata_gap(self, entity_type, entity_id, file_path, details):
"""Apply found metadata fields to the track."""
found_fields = details.get('found_fields')

View file

@ -0,0 +1,343 @@
"""Background worker that fills the ``similar_artists`` table for LIBRARY artists.
The watchlist scanner only populates similar artists for *watchlist* artists, so
the artist map / discover surfaces are rich for watchlisted artists and sparse
for the rest of the library. This worker closes that gap: for every
source-matched library artist it asks MusicMap for ~25 similar artists, matches
each one to the user's metadata source chain (primary + active fallbacks) via the
shared :func:`core.metadata.similar_artists.get_musicmap_similar_artists`, and
stores the matched results keyed by the library artist's **metadata source id** —
the same key the watchlist scanner and the artist map use, so the two cooperate
(idempotent upsert + a retry window keep them from double-fetching).
It plugs into the existing enrichment-worker pattern (background thread, status /
pause / resume, ``matched / not_found / pending / errors`` stats) so it shows up
as a bubble in the dashboard / Manage Enrichment Workers modal like every other
source worker.
The pure seams below (:func:`pick_source_artist_id`,
:func:`map_payload_to_store_kwargs`, :func:`process_artist`) carry the logic and
are unit-tested in isolation; the class wires them to the DB + MusicMap.
"""
from __future__ import annotations
import threading
from typing import Any, Callable, Dict, Optional
from utils.logging_config import get_logger
from core.worker_utils import interruptible_sleep
logger = get_logger("similar_artists_worker")
# A matched MusicMap payload is {id, source, name, image_url, genres, popularity}.
# Map its single (id, source) onto the right add_or_update_similar_artist id
# kwarg. The table only has columns for these four providers; a match on any
# other source (e.g. discogs) is still stored, name-only.
_SOURCE_ID_FIELD = {
'spotify': 'similar_artist_spotify_id',
'itunes': 'similar_artist_itunes_id',
'deezer': 'similar_artist_deezer_id',
'musicbrainz': 'similar_artist_musicbrainz_id',
}
# Library-artist source-id columns, in the same priority the watchlist scanner
# uses to key its rows — so a library artist and (if also watchlisted) its
# watchlist row resolve to the SAME source_artist_id and don't duplicate work.
_LIBRARY_ID_COLUMNS = ('spotify_artist_id', 'itunes_artist_id', 'deezer_id', 'musicbrainz_id')
def map_payload_to_store_kwargs(payload: Dict[str, Any]) -> Dict[str, Optional[str]]:
"""Turn a matched MusicMap payload into the id kwarg for the store call."""
src = str(payload.get('source') or '').lower()
pid = str(payload.get('id') or '')
field = _SOURCE_ID_FIELD.get(src)
return {field: pid} if (field and pid) else {}
def pick_source_artist_id(row: Dict[str, Any]) -> Optional[str]:
"""The metadata source id to key a library artist's similars by, or None if
the artist isn't matched to any metadata source yet (→ skip it)."""
for key in _LIBRARY_ID_COLUMNS:
v = row.get(key)
if v:
return str(v)
return None
def process_artist(
source_artist_id: str,
artist_name: str,
fetch_similars: Callable[[str, int], Dict[str, Any]],
store_similar: Callable[..., bool],
limit: int = 25,
profile_id: int = 1,
) -> tuple:
"""Fetch + store similar artists for one library artist.
``fetch_similars(name, limit)`` returns the ``get_musicmap_similar_artists``
payload; ``store_similar(**kwargs)`` is ``add_or_update_similar_artist``.
Returns ``(status, stored_count, detail)`` where status is one of:
- ``'matched'`` stored 1 similar artist
- ``'not_found'`` MusicMap had no entry / nothing matched a source
- ``'error'`` MusicMap/source failure (transient; eligible for retry)
``detail`` is a short human-readable reason (status code + message, or
``'no matches'`` / ``''``) so the worker can surface WHY a fetch failed
instead of swallowing it needed to diagnose error rates.
"""
try:
result = fetch_similars(artist_name, limit) or {}
except Exception as exc:
return ('error', 0, f'exception: {exc}')
if not result.get('success'):
# 404/400 = genuinely no MusicMap entry → 'not_found' (don't keep retrying);
# anything else (timeout, 5xx, no providers) = transient → 'error' (retry).
code = result.get('status_code')
detail = f"{code}: {result.get('error') or 'unknown'}"
return ('not_found' if code in (400, 404) else 'error', 0, detail)
sims = result.get('similar_artists') or []
if not sims:
return ('not_found', 0, 'no matches')
stored = 0
for rank, s in enumerate(sims, 1):
name = s.get('name')
if not name:
continue
kwargs = map_payload_to_store_kwargs(s)
if not kwargs:
# The match resolved to a source with no id column in similar_artists
# (e.g. discogs). Storing it name-only would be useless — you can't
# navigate/explore/download it. Enforce the standard: every stored
# similar carries a metadata source id, or we skip it.
logger.debug("Skipping similar '%s' (matched %s — no storable source id)", name, s.get('source'))
continue
try:
ok = store_similar(
source_artist_id=source_artist_id,
similar_artist_name=name,
similarity_rank=rank,
profile_id=profile_id,
image_url=s.get('image_url'),
genres=s.get('genres'),
popularity=s.get('popularity', 0) or 0,
**kwargs,
)
if ok:
stored += 1
except Exception as exc:
logger.debug("store similar failed for %s: %s", name, exc)
return ('matched' if stored else 'not_found', stored, '')
class SimilarArtistsWorker:
"""Background worker that populates similar artists for library artists."""
def __init__(self, database):
self.db = database
self.running = False
self.paused = False
self.should_stop = False
self.thread: Optional[threading.Thread] = None
self._stop_event = threading.Event()
self.current_item: Optional[str] = None
self.stats = {'matched': 0, 'not_found': 0, 'pending': 0, 'errors': 0}
self.retry_days = 30
self.limit = 25
self._err_logged = 0 # how many fetch errors we've logged at WARNING this session
self._last_error = None # most recent fetch-error reason (for diagnosis)
# similar_artists rows are profile-scoped; the library is shared. v1 keys
# under the default profile (matches single-profile setups, which is the
# common case). Multi-profile per-source-chain population is future work.
self.profile_id = 1
logger.info("Similar Artists background worker initialized")
# ── lifecycle (mirrors the other enrichment workers) ──────────────────
def start(self):
if self.running:
return
self.running = True
self.should_stop = False
self._stop_event.clear()
self.thread = threading.Thread(target=self._run, daemon=True)
self.thread.start()
logger.info("Similar Artists worker started")
def stop(self):
self.should_stop = True
self.running = False
self._stop_event.set()
def pause(self):
self.paused = True
def resume(self):
self.paused = False
def get_stats(self) -> Dict[str, Any]:
# Report PERSISTENT counts from the DB (not the in-memory session
# counters), so the dashboard orb and the Manage modal always agree and
# survive restarts — same approach as the other enrichment workers.
c = self._db_counts()
self.stats = {
'matched': c['matched'], 'not_found': c['not_found'],
'pending': c['pending'], 'errors': c['error'],
}
total = c['total']
is_running = self.running and (self.thread is not None and self.thread.is_alive())
is_idle = is_running and not self.paused and c['pending'] == 0 and self.current_item is None
return {
'enabled': True,
'running': is_running and not self.paused,
'paused': self.paused,
'idle': is_idle,
'current_item': self.current_item,
'stats': self.stats.copy(),
# Artist-only progress (no album/track phases) so the orb tooltip can
# show "matched / total (percent%)" like every other worker.
'progress': {
'artists': {
'matched': c['matched'], 'total': total,
'percent': int(round(c['matched'] / total * 100)) if total else 0,
}
},
}
# ── worker loop ────────────────────────────────────────────────────────
def _run(self):
logger.info("Similar Artists worker thread started")
# Imported lazily so the worker module stays import-light for tests.
from core.metadata.similar_artists import get_musicmap_similar_artists
while not self.should_stop:
try:
if self.paused:
interruptible_sleep(self._stop_event, 1)
continue
self.current_item = None
artist = self._get_next_artist()
if not artist:
interruptible_sleep(self._stop_event, 15)
continue
sid = pick_source_artist_id(artist)
if not sid:
# Query already filters to artists with a source id; guard anyway.
self._mark(artist['id'], 'error')
continue
self.current_item = artist.get('name')
status, count, detail = process_artist(
sid, artist['name'],
get_musicmap_similar_artists,
self.db.add_or_update_similar_artist,
limit=self.limit, profile_id=self.profile_id,
)
self._mark(artist['id'], status)
if status == 'matched':
self.stats['matched'] += 1
logger.debug("Similar artists: %s → stored %d", artist['name'], count)
elif status == 'not_found':
self.stats['not_found'] += 1
else:
self.stats['errors'] += 1
# Surface WHY fetches error — the first handful at WARNING (so
# the cause is visible in app.log without spamming a 4000-artist
# run), the rest at DEBUG. Keep the most recent reason for stats.
self._last_error = detail
if self._err_logged < 15:
self._err_logged += 1
logger.warning("Similar artists fetch error for '%s'%s", artist['name'], detail)
else:
logger.debug("Similar artists fetch error for '%s'%s", artist['name'], detail)
# Pace MusicMap (name search per candidate is heavy + rate-limited).
interruptible_sleep(self._stop_event, 3)
except Exception as exc:
logger.error("Similar Artists worker loop error: %s", exc)
interruptible_sleep(self._stop_event, 5)
logger.info("Similar Artists worker thread finished")
# ── DB helpers (thin; the testable logic lives in the pure seams above) ──
def _has_source_id_clause(self) -> str:
return '(' + ' OR '.join(f"{c} IS NOT NULL AND {c} != ''" for c in _LIBRARY_ID_COLUMNS) + ')'
def _get_next_artist(self) -> Optional[Dict[str, Any]]:
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cols = "id, name, " + ", ".join(_LIBRARY_ID_COLUMNS)
have_id = self._has_source_id_clause()
# 1) Unattempted source-matched artists.
cursor.execute(f"""
SELECT {cols} FROM artists
WHERE id IS NOT NULL AND name IS NOT NULL
AND similar_artists_match_status IS NULL
AND {have_id}
ORDER BY id ASC LIMIT 1
""")
row = cursor.fetchone()
# 2) Retry transient failures (and re-check 'not_found') after retry_days.
if not row:
cursor.execute(f"""
SELECT {cols} FROM artists
WHERE id IS NOT NULL AND name IS NOT NULL
AND similar_artists_match_status IN ('error', 'not_found')
AND (similar_artists_last_attempted IS NULL
OR similar_artists_last_attempted < datetime('now', ?))
AND {have_id}
ORDER BY similar_artists_last_attempted ASC LIMIT 1
""", (f'-{self.retry_days} days',))
row = cursor.fetchone()
if not row:
return None
keys = ['id', 'name'] + list(_LIBRARY_ID_COLUMNS)
return dict(zip(keys, row, strict=False))
except Exception as exc:
logger.debug("Similar Artists _get_next_artist failed: %s", exc)
return None
def _mark(self, artist_id, status: str):
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(
"UPDATE artists SET similar_artists_match_status = ?, "
"similar_artists_last_attempted = CURRENT_TIMESTAMP WHERE id = ?",
(status, artist_id),
)
conn.commit()
except Exception as exc:
logger.debug("Similar Artists _mark failed for %s: %s", artist_id, exc)
def _count_pending(self) -> int:
return self._db_counts()['pending']
def _db_counts(self) -> Dict[str, int]:
"""Persistent tallies over the worker's universe (source-matched library
artists): matched / not_found / error / pending(NULL) / total."""
out = {'matched': 0, 'not_found': 0, 'error': 0, 'pending': 0, 'total': 0}
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"""
SELECT
SUM(CASE WHEN similar_artists_match_status = 'matched' THEN 1 ELSE 0 END),
SUM(CASE WHEN similar_artists_match_status = 'not_found' THEN 1 ELSE 0 END),
SUM(CASE WHEN similar_artists_match_status = 'error' THEN 1 ELSE 0 END),
SUM(CASE WHEN similar_artists_match_status IS NULL THEN 1 ELSE 0 END),
COUNT(*)
FROM artists WHERE {self._has_source_id_clause()}
""")
row = cursor.fetchone() or (0, 0, 0, 0, 0)
out.update(matched=int(row[0] or 0), not_found=int(row[1] or 0),
error=int(row[2] or 0), pending=int(row[3] or 0), total=int(row[4] or 0))
except Exception as exc:
logger.debug("Similar Artists _db_counts failed: %s", exc)
return out

118
core/spotify_public_api.py Normal file
View file

@ -0,0 +1,118 @@
"""Full public-playlist fetch for the 'Spotify link' path, via the OPTIONAL
SpotipyFree library (no Spotify credentials needed).
Why a library: the embed scraper caps at ~100 tracks, and getting the full list
with no login means talking to Spotify's private API the way the web player
does including client-auth headers Spotify rotates constantly. Rather than
chase those ourselves (we tried; Spotify 429s the bare token), we lean on
SpotipyFree the maintained no-creds ``spotipy`` drop-in that spotDL uses,
which tracks those rotating bits for us.
Licensing: SpotipyFree is GPL-3.0, so it is NOT bundled or required by SoulSync
(MIT). It's an OPTIONAL install — if the user has run ``pip install spotipyFree``
this lights up; otherwise the import fails, this raises, and the caller
(``spotify_public_scraper.fetch_spotify_public``) falls back to the embed
scraper (today's ≤100). So SoulSync ships zero GPL code and stays cleanly MIT.
``client_factory`` is injectable so the orchestration is unit-testable without
the library or the network.
"""
from __future__ import annotations
import hashlib
import logging
from typing import Any, Callable, Dict, List, Optional
logger = logging.getLogger('soulsync.spotify_public')
_MAX_TRACKS = 10000 # safety cap
def normalize_api_track(item: Any, index: int) -> Optional[Dict[str, Any]]:
"""Convert a spotipy-shape playlist item to the embed scraper's track shape.
Returns None for items without a usable track id (local files, removed
tracks, podcast episodes) so the caller can skip them.
"""
track = (item or {}).get('track') or {}
track_id = track.get('id')
if not track_id:
return None
artists = [{'name': a.get('name', '')} for a in (track.get('artists') or []) if a.get('name')]
return {
'id': track_id,
'name': track.get('name', 'Unknown Track'),
'artists': artists or [{'name': 'Unknown Artist'}],
'duration_ms': track.get('duration_ms', 0),
'is_explicit': bool(track.get('explicit', False)),
'track_number': index + 1,
}
def _default_client():
"""Create a no-credentials SpotipyFree client.
Raises ImportError when the optional GPL-3.0 library isn't installed — the
caller treats that like any other failure and falls back to the embed
scraper.
"""
from SpotipyFree import Spotify # optional, user-installed (GPL-3.0)
return Spotify()
def fetch_public_playlist_full(
spotify_id: str,
*,
client_factory: Optional[Callable[[], Any]] = None,
) -> Dict[str, Any]:
"""Pull a public playlist's FULL track list with no credentials.
Uses a SpotipyFree client (spotipy-compatible: ``playlist`` for metadata,
``playlist_items`` + ``next`` for paginated tracks). Returns the embed
scraper's shape. Raises on any failure (incl. the library not being
installed) so the caller can fall back to the embed scraper.
"""
client = (client_factory or _default_client)()
meta: Dict[str, Any] = {}
try:
# limit=1: we only want name/owner here — tracks come from the paginated
# playlist_items call below, so don't pull the whole list twice.
meta = client.playlist(spotify_id, limit=1) or {}
except Exception as e: # metadata is nice-to-have; tracks are the point
logger.debug("playlist metadata fetch failed (%s); continuing", e)
name = meta.get('name', 'Unknown')
subtitle = (meta.get('owner') or {}).get('display_name', '')
tracks: List[Dict[str, Any]] = []
results = client.playlist_items(spotify_id)
while results:
for item in results.get('items', []):
t = normalize_api_track(item, len(tracks))
if t:
tracks.append(t)
if len(tracks) >= _MAX_TRACKS:
break
if results.get('next'):
results = client.next(results)
else:
break
if not tracks:
raise RuntimeError('SpotipyFree returned no usable tracks')
logger.info("SpotipyFree full fetch: %s (%d tracks)", name, len(tracks))
source_url = f'https://open.spotify.com/playlist/{spotify_id}'
return {
'id': spotify_id,
'type': 'playlist',
'name': name,
'subtitle': subtitle,
'tracks': tracks,
'url': source_url,
'url_hash': hashlib.md5(source_url.encode()).hexdigest()[:12],
}
__all__ = ['normalize_api_track', 'fetch_public_playlist_full']

View file

@ -9,7 +9,8 @@ import logging
import hashlib
import requests
logger = logging.getLogger(__name__)
# 'soulsync.*' so these lines land in app.log (the bare module name isn't captured).
logger = logging.getLogger('soulsync.spotify_public')
def parse_spotify_url(url: str) -> dict:
@ -82,10 +83,18 @@ def scrape_spotify_embed(spotify_type: str, spotify_id: str) -> dict:
logger.error(f"Failed to fetch Spotify embed: {e}")
return {'error': f'Failed to fetch Spotify page: {str(e)}'}
return parse_embed_html(response.text, spotify_type, spotify_id)
def parse_embed_html(html: str, spotify_type: str, spotify_id: str) -> dict:
"""Parse a Spotify embed page's ``__NEXT_DATA__`` into the standard result
shape. Shared by the embed scraper and the full public fetch, which pulls
the name + first-page tracks from the same embed page it reads the token
from (so it needs no extra metadata request)."""
# Extract __NEXT_DATA__ JSON
match = re.search(
r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>',
response.text
html
)
if not match:
logger.error("No __NEXT_DATA__ found in Spotify embed response")
@ -151,3 +160,31 @@ def scrape_spotify_embed(spotify_type: str, spotify_id: str) -> dict:
logger.info(f"Scraped Spotify {spotify_type}: {result['name']} ({len(tracks)} tracks)")
return result
def fetch_spotify_public(spotify_type: str, spotify_id: str) -> dict:
"""Fetch a public Spotify link, preferring the FULL track list.
Playlists are pulled via the anonymous public-API path
(``spotify_public_api.fetch_public_playlist_full``), which paginates past
the embed widget's ~100-track cap. On ANY failure — or for albums, which
the embed already returns whole this falls back to ``scrape_spotify_embed``
(today's behaviour). Same return shape either way, so callers don't care
which path produced the data.
"""
if spotify_type == 'playlist':
try:
from core.spotify_public_api import fetch_public_playlist_full
result = fetch_public_playlist_full(spotify_id)
if result and result.get('tracks'):
logger.info(
f"Spotify public API (full): {result.get('name')} "
f"({len(result['tracks'])} tracks)"
)
return result
except Exception as e:
logger.info(
f"Spotify public full fetch failed ({e}); "
f"falling back to embed scraper (≤100 tracks)"
)
return scrape_spotify_embed(spotify_type, spotify_id)

View file

@ -260,6 +260,17 @@ class SpotifyWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('spotify')
if _prio:
_pi = priority_pending_item(cursor, 'spotify', _prio,
{'album': 'album_individual', 'track': 'track_individual'})
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -0,0 +1,74 @@
"""Pure planners for sync-editor playlist mutations (#768 Bug C).
The sync editor's "Find & add" and remove actions rewrite the whole server
playlist from a flat list of track IDs (Subsonic/Navidrome + Jellyfin have no
position-level ops). Two bugs lived in the inline endpoint logic:
* **Duplicate on manual match.** "Find & add" always *inserted* the chosen
track but when the user is matching an UNMATCHED source to a server track
that's already in the playlist (an orphan "extra"), the intent is to LINK
them, not add a second copy. Each attempt appended another duplicate
(positions 72, 73, 74). ``plan_playlist_add`` skips the insert when it's a
link to an already-present track (the caller still persists the override).
* **Delete removes ALL copies.** The inline remove filtered out *every* entry
with the target ID. With duplicates present, deleting one removed them all.
``remove_one_occurrence`` drops a single entry (duplicates are the same
track, so removing any one is correct).
Pure, no I/O the caller fetches the current track-id list and applies the
returned plan to the media-server client.
"""
from __future__ import annotations
from typing import List, Optional, Tuple
def plan_playlist_add(
current_ids: List[str],
track_id: str,
*,
is_link: bool,
position: Optional[int] = None,
) -> dict:
"""Plan a "Find & add" against a flat track-id playlist.
``is_link`` is True when the add carries a ``source_track_id`` (i.e. the
user is matching an unmatched source to this server track). In that case,
if the track is ALREADY in the playlist, return ``should_insert=False`` so
the caller only records the override and never duplicates it.
Returns ``{'should_insert': bool, 'new_ids': [...]}``. ``new_ids`` equals
the input (stringified) when no insert is needed."""
tid = str(track_id)
current = [str(t) for t in current_ids]
if is_link and tid in current:
return {"should_insert": False, "new_ids": current}
pos = len(current) if position is None else max(0, min(int(position), len(current)))
new_ids = current[:pos] + [tid] + current[pos:]
return {"should_insert": True, "new_ids": new_ids}
def remove_one_occurrence(
track_ids: List[str],
target_id: str,
position: Optional[int] = None,
) -> Tuple[List[str], bool]:
"""Remove a SINGLE occurrence of ``target_id`` from a flat id list.
If ``position`` is given and the id there matches, that exact entry is
removed (so the user removes the row they clicked); otherwise the first
matching id is removed. Returns ``(new_ids, removed)``. ``removed`` is
False when the id isn't present (caller should 404)."""
target = str(target_id)
ids = [str(t) for t in track_ids]
if position is not None and 0 <= position < len(ids) and ids[position] == target:
return ids[:position] + ids[position + 1:], True
for idx, tid in enumerate(ids):
if tid == target:
return ids[:idx] + ids[idx + 1:], True
return ids, False
__all__ = ["plan_playlist_add", "remove_one_occurrence"]

View file

@ -0,0 +1,197 @@
"""Reconcile a source playlist against a media-server playlist (pure).
Lifted verbatim from the inline three-pass matcher in
``web_server.get_server_playlist_tracks`` so it can be unit-tested and so the
two #768 fixes live in importable, covered code:
Pass 0 user-confirmed overrides (``sync_match_cache``), applied first.
Pass 1 exact normalized-title match.
Pass 2 fuzzy match on ``"artist title"`` (SequenceMatcher >= 0.75).
Extra server tracks no source claimed -> ``match_status='extra'``.
Two bug fixes over the original inline version:
* #768 Bug A — the source side is YouTube/streaming-shaped (title
``"Artist - Song"``, artist ``"Official Artist"``). The original passes
compared the raw title, so ``"Arctic Monkeys - Do I Wanna Know?"`` never
matched the library's ``"Do I Wanna Know?"`` and the track showed as
unmatched while its server copy showed as an orphan "extra". We now also try
the canonicalized source title/artist (see ``core.text.source_title``).
* #768 Bug B — the original built the per-source ``src_entry`` WITHOUT
``source_track_id``, so the editor UI never received it; "Find & add" then
posted an empty id and the manual match was never persisted (it reverted to
"extra" on reload, and re-adding duplicated the track). ``src_entry`` now
carries ``source_track_id``.
Pure, no I/O. ``override_pairs`` (``{source_idx: server_idx}``) is computed by
the caller via ``core.sync.match_overrides.resolve_match_overrides`` so the DB
lookup stays out of this module.
"""
from __future__ import annotations
import re
from difflib import SequenceMatcher
from typing import Any, Dict, List, Optional
from core.text.source_title import canonical_source_track
_FUZZY_THRESHOLD = 0.75
_FEAT_RE = re.compile(r'\s*[\(\[](?:feat|ft)\.?[^\)\]]*[\)\]]', re.IGNORECASE)
_REMASTER_RE = re.compile(
r'\s*[\(\[](?:\d{4}\s+)?remaster(?:ed)?(?:\s+version)?\s*[\)\]]', re.IGNORECASE)
_EDITION_RE = re.compile(
r'\s*[\(\[](?:deluxe|special|anniversary|legacy|expanded|limited)(?:\s+edition)?\s*[\)\]]',
re.IGNORECASE)
def norm_title(t: str) -> str:
"""Strip feat./ft., remaster, and edition qualifiers for comparison only.
Byte-faithful to web_server's inline ``_norm_title``."""
t = _FEAT_RE.sub('', t or '')
t = _REMASTER_RE.sub('', t)
t = _EDITION_RE.sub('', t)
return t.lower().strip()
def _src_entry(src: Dict[str, Any], position_fallback: int) -> Dict[str, Any]:
return {
'name': src.get('name', ''),
'artist': src.get('artist', ''),
'album': src.get('album', ''),
'image_url': src.get('image_url', ''),
'duration_ms': src.get('duration_ms', 0),
'position': src.get('position', position_fallback),
# #768 Bug B: echo the source id back so the editor can persist a
# manual "Find & add" override against it.
'source_track_id': src.get('source_track_id', '') or '',
}
def _resolved_artist(src: Dict[str, Any]) -> str:
"""Artist string, falling back to the first of an ``artists`` list."""
artist = src.get('artist', '')
if not artist and src.get('artists'):
a = src['artists'][0] if src['artists'] else ''
artist = a.get('name', a) if isinstance(a, dict) else str(a)
return artist or ''
def reconcile_playlist(
source_tracks: List[Dict[str, Any]],
server_tracks: List[Dict[str, Any]],
override_pairs: Optional[Dict[int, int]] = None,
) -> List[Dict[str, Any]]:
"""Return the combined matched/missing/extra view (list of dicts).
Each combined entry has ``source_track``, ``server_track``,
``match_status`` ('matched'|'missing'|'extra'), ``confidence``, and
``override: True`` on override hits."""
override_pairs = override_pairs or {}
combined: List[Dict[str, Any]] = []
used_server_indices: set[int] = set()
unmatched_source: List[tuple[int, Dict[str, Any], str]] = []
# Precompute normalized server titles once.
server_norm = [norm_title(svr.get('title', '')) for svr in server_tracks]
for i, src in enumerate(source_tracks):
src_artist = _resolved_artist(src)
src_name = src.get('name', '')
src_entry = _src_entry({**src, 'artist': src_artist}, i)
# Pass 0: user-confirmed override.
if i in override_pairs:
j = override_pairs[i]
used_server_indices.add(j)
combined.append({
'source_track': src_entry,
'server_track': server_tracks[j],
'match_status': 'matched',
'confidence': 1.0,
'override': True,
})
continue
# Pass 1: exact normalized-title match — try the raw source title AND
# the canonicalized one (strips "Artist - " prefix / channel artist).
canon_title, _canon_artist = canonical_source_track(src_name, src_artist)
candidates = {norm_title(src_name), norm_title(canon_title)}
best_idx = -1
for j, svr_norm in enumerate(server_norm):
if j in used_server_indices:
continue
if svr_norm in candidates:
best_idx = j
break
if best_idx >= 0:
used_server_indices.add(best_idx)
combined.append({
'source_track': src_entry,
'server_track': server_tracks[best_idx],
'match_status': 'matched',
'confidence': 1.0,
})
else:
idx = len(combined)
combined.append({
'source_track': src_entry,
'server_track': None,
'match_status': 'missing',
'confidence': 0.0,
})
# Carry the canonical artist for the fuzzy pass.
unmatched_source.append((idx, src_entry, _canon_artist or src_artist))
# Pass 2: fuzzy match on remaining unmatched source tracks. Build the key
# from the canonicalized title/artist so YouTube-shaped sources can pair.
for combo_idx, src_entry, canon_artist in unmatched_source:
canon_title, _ = canonical_source_track(src_entry['name'], src_entry['artist'])
src_key = f"{canon_artist} {norm_title(canon_title)}".strip().lower()
best_score = 0.0
best_j = -1
for j, svr in enumerate(server_tracks):
if j in used_server_indices:
continue
svr_key = f"{svr.get('artist', '')} {norm_title(svr.get('title', ''))}".strip().lower()
score = SequenceMatcher(None, src_key, svr_key).ratio()
if score > best_score and score >= _FUZZY_THRESHOLD:
best_score = score
best_j = j
if best_j >= 0:
used_server_indices.add(best_j)
combined[combo_idx] = {
'source_track': src_entry,
'server_track': server_tracks[best_j],
'match_status': 'matched',
'confidence': round(best_score, 3),
}
# Extra: server tracks no source claimed.
for j, svr in enumerate(server_tracks):
if j not in used_server_indices:
combined.append({
'source_track': None,
'server_track': svr,
'match_status': 'extra',
'confidence': 0.0,
})
# #766: a source row with no art of its own (e.g. a YouTube source, which
# provides none) borrows its MATCHED server track's cover so both sides of
# the editor show an image. Keyed off the actual pairing — works for
# "Artist - Title" rows that a fuzzy title lookup would miss. Source rows
# that already have their own art (Spotify CDN, etc.) keep it.
for entry in combined:
st = entry.get('source_track')
sv = entry.get('server_track')
if st and sv and not st.get('image_url') and sv.get('thumb'):
st['image_url'] = sv['thumb']
return combined
__all__ = ["reconcile_playlist", "norm_title"]

105
core/text/source_title.py Normal file
View file

@ -0,0 +1,105 @@
"""Normalize streaming/YouTube-style source track metadata for matching.
Issue #768: source playlists — especially ones seeded from YouTube — carry
video-style metadata: the title is ``"Artist - Song"`` and the artist is a
channel name like ``"Official Arctic Monkeys"``, ``"Arctic Monkeys - Topic"``,
or ``"ColdplayVEVO"``. The library/media-server side has the clean ``"Song"`` /
``"Arctic Monkeys"``. Both matching paths (the sync confidence scorer and the
playlist-editor reconcile) then fail to pair them the track is reported
"not matched" / shows up as an orphan "extra" even though it exists.
These helpers strip that channel/video decoration so the cleaned source can be
compared against the clean library metadata. Pure, no I/O.
Conservative by construction:
- ``strip_artist_prefix`` removes a leading ``"<artist><sep>"`` only when the
prefix EQUALS the artist we're matching against. So ``"Death - Pull the
Plug"`` by ``"Death"`` is helped, while ``"Marvin Gaye"`` by Charlie Puth
(title is not ``"Charlie Puth - ..."``) is left untouched, and a hyphenated
word like ``"Self-Titled"`` is never split (a separator needs surrounding
whitespace, or a colon).
- ``clean_source_artist`` only removes well-known channel decorations.
Both are intended to be applied as ADDITIONAL match candidates (best-of), so
an over-eager strip can only add a comparison, never remove the original.
"""
from __future__ import annotations
import re
from core.text.normalize import normalize_for_comparison
# Artist/title separator: a dash/pipe/tilde flanked by whitespace, OR a colon
# (with optional trailing space). Whitespace-flanking keeps "Self-Titled" and
# "Jay-Z" intact while still splitting "Artist - Title".
_SEP_SPLIT = re.compile(r"\s+[-–—|~]\s+|\s*:\s+")
# YouTube auto-generated artist channel: "Arctic Monkeys - Topic".
_TOPIC_SUFFIX = re.compile(r"\s*-\s*topic\s*$", re.IGNORECASE)
# "Official " / "The Official " channel prefix.
_OFFICIAL_PREFIX = re.compile(r"^\s*(?:the\s+)?official\s+", re.IGNORECASE)
# Trailing VEVO, attached ("ColdplayVEVO") or spaced ("Coldplay VEVO").
_VEVO_SUFFIX = re.compile(r"\s*vevo\s*$", re.IGNORECASE)
def clean_source_artist(artist: str) -> str:
"""Strip well-known streaming-channel decoration from an artist name.
``"Official Arctic Monkeys"`` ``"Arctic Monkeys"``;
``"Arctic Monkeys - Topic"`` ``"Arctic Monkeys"``;
``"ColdplayVEVO"`` ``"Coldplay"``. Returns the input unchanged when
nothing matches, and never returns empty for non-empty input."""
if not artist:
return artist
s = artist.strip()
topic = _TOPIC_SUFFIX.sub("", s).strip()
if topic and topic != s:
s = topic
official = _OFFICIAL_PREFIX.sub("", s).strip()
if official:
s = official
# Only strip VEVO if at least 2 chars of name remain (don't empty "VEVO").
vevo = _VEVO_SUFFIX.sub("", s).strip()
if len(vevo) >= 2 and vevo != s:
s = vevo
return s or artist
def strip_artist_prefix(title: str, artist: str) -> str:
"""Remove a leading ``"<artist><separator>"`` from ``title`` when the prefix
equals ``artist`` (accent/case-folded). Otherwise return ``title`` unchanged.
``("Arctic Monkeys - Do I Wanna Know?", "Arctic Monkeys")`` ``"Do I Wanna
Know?"``. Never returns an empty string."""
if not title or not artist:
return title
na = normalize_for_comparison(artist)
if not na:
return title
parts = _SEP_SPLIT.split(title, maxsplit=1)
if len(parts) == 2:
left, right = parts
right = right.strip()
if right and normalize_for_comparison(left) == na:
return right
return title
def canonical_source_track(title: str, artist: str) -> tuple[str, str]:
"""Best-effort clean (title, artist) for matching a streaming/YouTube source
against clean library metadata. Cleans the artist first, then strips a
leading artist prefix from the title using EITHER the cleaned or the raw
artist (YouTube titles prepend the real artist, not the channel name)."""
cleaned_artist = clean_source_artist(artist)
new_title = strip_artist_prefix(title, cleaned_artist)
if new_title == title and cleaned_artist != artist:
new_title = strip_artist_prefix(title, artist)
return new_title, cleaned_artist
__all__ = ["clean_source_artist", "strip_artist_prefix", "canonical_source_track"]

81
core/text/title_match.py Normal file
View file

@ -0,0 +1,81 @@
"""Guard against char-level title false positives in track matching.
Issue #769: playlist sync matched tracks that aren't in the library to a
DIFFERENT song by the SAME artist, with high confidence e.g. "Dani
California" -> "Californication" (Red Hot Chili Peppers), "Under The Bridge"
-> "Around the World". The confidence formula is ``0.5*title + 0.5*artist``,
and a same-artist comparison always yields ``artist = 1.0``, so the title score
is the only thing that can tell two of an artist's songs apart. But the title
score is a ``difflib.SequenceMatcher`` character ratio, which over-credits
unrelated titles that happen to share a long substring ("californi…") or only a
stopword ("the"): 0.67 and 0.62 respectively. With the flat 0.5 artist term
that lands at 0.83 / 0.81 well over the 0.7 sync threshold.
``titles_plausibly_same`` adds a cheap word-level sanity check on top of the
char ratio: accept a pair only when it's near-identical char-wise (so typos and
punctuation/casing variants "Beleive"/"Believe", "HUMBLE."/"Humble" still
match) OR the two titles share at least one significant (non-stopword) token.
Two genuinely different songs by the same artist share no content word, so they
get rejected; the real track is then correctly reported missing.
"""
from __future__ import annotations
import re
# Articles / prepositions / conjunctions only. Deliberately NOT pronouns
# ("you", "me", "i") — those carry meaning in song titles and dropping them
# could strip the only shared word from a real match. "the" MUST stay here:
# without it "Under The Bridge" and "Around the World" would falsely share it.
_TITLE_STOPWORDS = frozenset({
"the", "a", "an", "of", "and", "or", "to", "in", "on",
"for", "with", "at", "by", "from",
})
_TOKEN_RE = re.compile(r"[a-z0-9]+")
# Char ratio at/above which two titles are treated as the same regardless of
# shared words — covers typos, punctuation, casing, accents. Tuned so single-
# word typos ("Beleive"/"Believe" = 0.857) pass while the #769 false positives
# ("Dani California"/"Californication" = 0.667) do not.
_NEAR_IDENTICAL = 0.85
def _content_tokens(text: str) -> set[str]:
return {t for t in _TOKEN_RE.findall((text or "").lower()) if t not in _TITLE_STOPWORDS}
def titles_plausibly_same(
title_a: str,
title_b: str,
char_similarity: float,
*,
near_identical: float = _NEAR_IDENTICAL,
) -> bool:
"""Whether two titles could be the same track, given their char similarity.
``title_a`` / ``title_b`` should already be normalised/cleaned (lowercased,
brackets stripped) the same way the caller computed ``char_similarity``.
Returns ``True`` when the pair is near-identical char-wise OR shares at
least one significant (non-stopword) token. Returns ``False`` for two
titles that are only moderately char-similar and share no content word
i.e. different songs the char ratio over-credited (#769)."""
if char_similarity >= near_identical:
return True
ta = _content_tokens(title_a)
tb = _content_tokens(title_b)
# Word-overlap is only a reliable "different song" signal when at least one
# side has 2+ content words — that's the #769 case where the char ratio
# over-credits a shared substring ("Dani California"/"Californication") or
# a stopword ("Under The Bridge"/"Around the World"). For single-word
# titles there's no other word to share, so applying it would wrongly fail
# legitimate stylized spellings ("Grey"/"Gray", "Tonite"/"Tonight",
# "Thru"/"Through") that the char ratio rightly accepts. In that case defer
# to the caller's existing char-similarity floor instead of force-failing.
if max(len(ta), len(tb)) < 2 or not ta or not tb:
return True
return not ta.isdisjoint(tb)
__all__ = ["titles_plausibly_same"]

View file

@ -198,6 +198,16 @@ class TidalWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('tidal')
if _prio:
_pi = priority_pending_item(cursor, 'tidal', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name

View file

@ -117,7 +117,19 @@ class QBittorrentAdapter:
headers={'Referer': self._url},
timeout=self.DEFAULT_TIMEOUT,
)
if not resp.ok or resp.text.strip() != 'Ok.':
body = resp.text.strip()
has_sid = bool(sess.cookies.get('SID'))
# qBittorrent reports BAD credentials as HTTP 200 + body "Fails."
# (it does NOT use a 4xx). SUCCESS is the SID auth cookie and/or a
# success body: "Ok." on <= 5.1, or an empty HTTP 204 on 5.2.0+,
# which changed /api/v2/auth/login to return 204 No Content.
# The old check required body == "Ok." and so rejected 5.2.0+.
login_ok = (
resp.ok
and body.lower() != 'fails.'
and (has_sid or resp.status_code == 204 or body in ('', 'Ok.'))
)
if not login_ok:
logger.error("qBittorrent login failed: HTTP %s body=%r", resp.status_code, resp.text[:200])
return None
self._session = sess

View file

@ -394,12 +394,17 @@ def resolve_wishlist_source_type_for_batch(batch: Dict[str, Any]) -> str:
def build_wishlist_source_context(batch: Dict[str, Any], current_time: datetime | None = None) -> Dict[str, Any]:
"""Build the source_context payload used when adding failed tracks back to the wishlist."""
current_time = current_time or datetime.now()
playlist_id = batch.get('source_playlist_ref') or batch.get('playlist_id')
context = {
'playlist_name': batch.get('playlist_name', 'Unknown Playlist'),
'playlist_id': batch.get('playlist_id', None),
'playlist_id': playlist_id,
'added_from': 'webui_modal',
'timestamp': current_time.isoformat(),
}
if batch.get('mirrored_playlist_id') is not None:
context['mirrored_playlist_id'] = batch.get('mirrored_playlist_id')
if batch.get('organize_by_playlist'):
context['organize_by_playlist'] = True
# Preserve album-batch provenance so wishlist requeue has a real signal
# for album-vs-single routing instead of relying on per-track album dicts
# that may have been mangled by reconstruction fallbacks.

View file

@ -78,3 +78,67 @@ def set_album_api_track_count(cursor, album_id, count):
logger.warning(
"Failed to cache api_track_count for album %s: %s", album_id, e
)
# --- Enrichment "process this group first" override -----------------------
# Each enrichment worker normally processes artist -> album -> track. A user
# can pin one entity type to run first via the Manage Enrichment Workers modal;
# the choice is stored in config as "<service>_enrichment_priority" and read
# at the top of each worker's _get_next_item so it takes effect live. When the
# pinned group is exhausted (or unset), the worker falls back to its normal
# chain — so the default path is unchanged.
PRIORITY_ENTITIES = ('artist', 'album', 'track')
def read_enrichment_priority(service: str) -> str:
"""Return the pinned entity ('artist'|'album'|'track') for a worker, or ''.
Read every loop so the override applies without restarting the worker.
Any error / unset / invalid value yields '' (no override)."""
try:
from config.settings import config_manager
val = (config_manager.get(f'{service}_enrichment_priority', '') or '')
val = str(val).strip().lower()
return val if val in PRIORITY_ENTITIES else ''
except Exception:
return ''
def priority_pending_item(cursor, service, entity, type_overrides=None):
"""Return one pending (NULL match_status) item of `entity`, or None.
`service` is the column prefix (e.g. 'spotify' -> spotify_match_status) and
MUST be a trusted worker-supplied literal (it is interpolated into SQL).
`type_overrides` maps the canonical entity to the worker's dispatch 'type'
string Spotify/iTunes process individual items as 'album_individual' /
'track_individual', the other workers use 'album' / 'track'. The returned
dict matches the shape those workers already return from _get_next_item."""
if not str(service).isalpha() or entity not in PRIORITY_ENTITIES:
return None
type_overrides = type_overrides or {}
ms = f"{service}_match_status"
if entity == 'artist':
cursor.execute(
f"SELECT id, name FROM artists WHERE {ms} IS NULL AND id IS NOT NULL "
f"ORDER BY id ASC LIMIT 1"
)
r = cursor.fetchone()
return {'type': type_overrides.get('artist', 'artist'), 'id': r[0], 'name': r[1]} if r else None
if entity == 'album':
cursor.execute(
f"SELECT a.id, a.title, ar.name FROM albums a JOIN artists ar ON a.artist_id = ar.id "
f"WHERE a.{ms} IS NULL AND a.id IS NOT NULL ORDER BY a.id ASC LIMIT 1"
)
r = cursor.fetchone()
return {'type': type_overrides.get('album', 'album'), 'id': r[0], 'name': r[1], 'artist': r[2]} if r else None
# track
cursor.execute(
f"SELECT t.id, t.title, ar.name FROM tracks t JOIN artists ar ON t.artist_id = ar.id "
f"WHERE t.{ms} IS NULL AND t.id IS NOT NULL ORDER BY t.id ASC LIMIT 1"
)
r = cursor.fetchone()
return {'type': type_overrides.get('track', 'track'), 'id': r[0], 'name': r[1], 'artist': r[2]} if r else None

View file

@ -427,6 +427,9 @@ class MusicDatabase:
# Add Amazon artist ID column (migration)
self._add_amazon_columns(cursor)
# Add Similar-Artists worker tracking columns (migration)
self._add_similar_artists_worker_columns(cursor)
# Backfill match_status for rows that already have an external ID but
# NULL status. Prevents enrichment workers from re-processing these
# rows forever. Must run AFTER all *_match_status columns have been
@ -571,6 +574,7 @@ class MusicDatabase:
# Add explored_at to mirrored_playlists (migration)
self._add_mirrored_playlist_explored_column(cursor)
self._add_mirrored_playlist_organize_column(cursor)
# Add notification columns to automations (migration)
self._add_automation_notify_columns(cursor)
@ -956,9 +960,147 @@ class MusicDatabase:
if album_cols and 'api_track_count' not in album_cols:
cursor.execute("ALTER TABLE albums ADD COLUMN api_track_count INTEGER DEFAULT NULL")
logger.info("Repaired missing api_track_count column on albums table")
# Canonical album version (#765 / #767-Bug2). Additive + nullable:
# a NULL canonical means "unresolved" and every tool falls back to
# today's behavior, so this is safe to ship dormant. Columns are
# populated/consumed in later stages.
_canonical_cols = {
'canonical_source': 'TEXT DEFAULT NULL',
'canonical_album_id': 'TEXT DEFAULT NULL',
'canonical_score': 'REAL DEFAULT NULL',
'canonical_resolved_at': 'TIMESTAMP DEFAULT NULL',
}
for _col, _typedef in _canonical_cols.items():
if album_cols and _col not in album_cols:
cursor.execute(f"ALTER TABLE albums ADD COLUMN {_col} {_typedef}")
logger.info("Added %s column to albums table (canonical version)", _col)
except Exception as e:
logger.error("Error repairing core media schema columns: %s", e)
def set_album_canonical(self, album_id, source: str, canonical_album_id: str, score: float) -> bool:
"""Persist the resolved canonical (source, album_id, score) for an album
(#765 Stage 2). Returns True if a row was updated."""
conn = self._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"UPDATE albums SET canonical_source = ?, canonical_album_id = ?, "
"canonical_score = ?, canonical_resolved_at = CURRENT_TIMESTAMP "
"WHERE id = ?",
(source, str(canonical_album_id), float(score), album_id),
)
conn.commit()
return cursor.rowcount > 0
except Exception as e:
logger.error("Error setting album canonical for %s: %s", album_id, e)
return False
finally:
conn.close()
def get_album_canonical(self, album_id) -> Optional[dict]:
"""Return ``{'source','album_id','score','resolved_at'}`` for an album's
pinned canonical release, or ``None`` when unresolved (#765 Stage 2).
Consumers treat ``None`` as 'fall back to today's behavior'."""
conn = self._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"SELECT canonical_source, canonical_album_id, canonical_score, "
"canonical_resolved_at FROM albums WHERE id = ?",
(album_id,),
)
row = cursor.fetchone()
if not row or not row[0] or not row[1]:
return None
return {
'source': row[0],
'album_id': row[1],
'score': row[2],
'resolved_at': row[3],
}
except Exception as e:
logger.error("Error reading album canonical for %s: %s", album_id, e)
return None
finally:
conn.close()
def get_enrichment_unmatched(
self,
service: str,
entity_type: str,
status: str = 'not_found',
query: str = None,
limit: int = 50,
offset: int = 0,
) -> dict:
"""List items a given enrichment source hasn't matched, paginated.
Powers the "Manage Enrichment Workers" modal's unmatched browser.
Returns ``{'total': int, 'items': [{id, name, image_url, status,
last_attempted}]}``. Raises ``UnmatchedQueryError`` for an unknown
service / unsupported entity type / bad status (the caller maps that to
an HTTP 400)."""
from core.enrichment.unmatched import (
build_count_query,
build_unmatched_query,
)
sql, params = build_unmatched_query(
service, entity_type, status, query, limit, offset
)
count_sql, count_params = build_count_query(service, entity_type, status, query)
conn = self._get_connection()
try:
cursor = conn.cursor()
total = cursor.execute(count_sql, count_params).fetchone()[0]
rows = cursor.execute(sql, params).fetchall()
items = [dict(row) for row in rows]
return {'total': total or 0, 'items': items}
finally:
conn.close()
def get_enrichment_breakdown(self, service: str, entity_type: str) -> dict:
"""Return ``{matched, not_found, pending, total}`` for a source/entity.
The per-worker ``get_stats().progress`` lumps matched + not_found into a
single 'processed' count; this splits them so the modal can show the
real match rate. Raises ``UnmatchedQueryError`` on bad input."""
from core.enrichment.unmatched import build_breakdown_query
sql, params = build_breakdown_query(service, entity_type)
conn = self._get_connection()
try:
row = conn.cursor().execute(sql, params).fetchone()
if not row:
return {'matched': 0, 'not_found': 0, 'pending': 0, 'total': 0}
return {
'matched': row[0] or 0,
'not_found': row[1] or 0,
'pending': row[2] or 0,
'total': row[3] or 0,
}
finally:
conn.close()
def reset_enrichment(self, service: str, entity_type: str, scope: str = 'item', entity_id=None) -> int:
"""Re-queue item(s) for a source by clearing match_status back to NULL.
scope='item' resets one row (entity_id); scope='failed' resets every
'not_found' row for that entity type. Returns the number of rows reset.
Raises ``UnmatchedQueryError`` on bad input."""
from core.enrichment.unmatched import build_reset_query
sql, params = build_reset_query(service, entity_type, scope, entity_id)
conn = self._get_connection()
try:
cursor = conn.cursor()
cursor.execute(sql, params)
conn.commit()
return cursor.rowcount or 0
finally:
conn.close()
def _add_mirrored_playlist_explored_column(self, cursor):
"""Add explored_at column to mirrored_playlists to persist explore badge."""
try:
@ -970,6 +1112,19 @@ class MusicDatabase:
except Exception as e:
logger.error(f"Error adding explored_at column to mirrored_playlists: {e}")
def _add_mirrored_playlist_organize_column(self, cursor):
"""Add organize_by_playlist preference for playlist-folder downloads."""
try:
cursor.execute("PRAGMA table_info(mirrored_playlists)")
cols = [c[1] for c in cursor.fetchall()]
if 'organize_by_playlist' not in cols:
cursor.execute(
"ALTER TABLE mirrored_playlists ADD COLUMN organize_by_playlist INTEGER NOT NULL DEFAULT 0"
)
logger.info("Added organize_by_playlist column to mirrored_playlists table")
except Exception as e:
logger.error(f"Error adding organize_by_playlist column to mirrored_playlists: {e}")
def _add_automation_notify_columns(self, cursor):
"""Add notification and result columns to automations table."""
try:
@ -2279,6 +2434,28 @@ class MusicDatabase:
except Exception as e:
logger.error(f"Error adding Discogs columns: {e}")
def _add_similar_artists_worker_columns(self, cursor):
"""Add Similar-Artists worker tracking columns to the artists table.
Mirrors the per-source enrichment pattern: a match_status (NULL =
unattempted, then 'matched'/'not_found'/'error') + last_attempted
timestamp so the SimilarArtistsWorker can pick the next library artist to
fetch MusicMap similars for and retry transient failures after a window.
Idempotent only adds columns that aren't already present.
"""
try:
cursor.execute("PRAGMA table_info(artists)")
artists_columns = [column[1] for column in cursor.fetchall()]
if 'similar_artists_match_status' not in artists_columns:
cursor.execute("ALTER TABLE artists ADD COLUMN similar_artists_match_status TEXT")
if 'similar_artists_last_attempted' not in artists_columns:
cursor.execute("ALTER TABLE artists ADD COLUMN similar_artists_last_attempted TIMESTAMP")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_similarartists_status ON artists (similar_artists_match_status)")
except Exception as e:
logger.error(f"Error adding similar-artists worker columns: {e}")
def _add_amazon_columns(self, cursor):
"""Add Amazon enrichment tracking columns to artists, albums, and tracks."""
try:
@ -7248,6 +7425,22 @@ class MusicDatabase:
# Titles differ in length by more than 30% — penalize heavily
best_title_similarity *= len_ratio
# Word-level guard: SequenceMatcher's char ratio over-credits
# different songs that share a long substring or only a stopword
# ("Dani California" vs "Californication" = 0.67; "Under The Bridge"
# vs "Around the World" = 0.62). Since a same-artist comparison
# always scores artist = 1.0, the title is the only discriminator,
# so a bad-but-moderate title score gets carried over the threshold
# (#769). Reject pairs that aren't near-identical AND share no
# significant word — the real track is then reported missing.
from core.text.title_match import titles_plausibly_same
if not titles_plausibly_same(
clean_search_title or search_title_norm,
clean_db_title or db_title_norm,
best_title_similarity,
):
return best_title_similarity * 0.5 # below any threshold
# Require minimum title similarity to prevent a perfect artist match from
# carrying a bad title match over the threshold (e.g. "Time" vs "Time Flies")
if best_title_similarity < 0.6:
@ -9119,6 +9312,70 @@ class MusicDatabase:
logger.error(f"Error getting top similar artists: {e}")
return []
def get_recommendation_sources(
self,
similar_artist_names: List[str],
profile_id: int = 1,
max_per: int = 6,
) -> Dict[str, List[str]]:
"""The 'because you have X, Y, Z' explanation behind each recommendation.
For each name in ``similar_artist_names``, return the display names of the
user's OWN artists (library or watchlist) that list it as a similar
artist. ``similar_artists.source_artist_id`` is a polymorphic provider id
(the spotify / itunes / deezer / musicbrainz id of one of the user's
artists), so we resolve it back to a name by matching against every
provider-id column on ``artists`` and ``watchlist_artists``.
Returns ``{similar_artist_name: [source_name, ...]}`` deduped,
name-sorted, capped at ``max_per`` per recommendation. Names with no
resolvable source are omitted from the dict.
"""
names = [n for n in (similar_artist_names or []) if n]
if not names:
return {}
try:
with self._get_connection() as conn:
cursor = conn.cursor()
placeholders = ','.join('?' for _ in names)
cursor.execute(f"""
SELECT sa.similar_artist_name AS rec_name,
COALESCE(a.name, wa.artist_name) AS source_name
FROM similar_artists sa
LEFT JOIN artists a ON (
(a.spotify_artist_id IS NOT NULL AND a.spotify_artist_id = sa.source_artist_id)
OR (a.itunes_artist_id IS NOT NULL AND a.itunes_artist_id = sa.source_artist_id)
OR (a.deezer_id IS NOT NULL AND a.deezer_id = sa.source_artist_id)
OR (a.musicbrainz_id IS NOT NULL AND a.musicbrainz_id = sa.source_artist_id)
)
LEFT JOIN watchlist_artists wa ON (
wa.profile_id = ? AND (
(wa.spotify_artist_id IS NOT NULL AND wa.spotify_artist_id = sa.source_artist_id)
OR (wa.itunes_artist_id IS NOT NULL AND wa.itunes_artist_id = sa.source_artist_id)
OR (wa.deezer_artist_id IS NOT NULL AND wa.deezer_artist_id = sa.source_artist_id)
OR (wa.musicbrainz_artist_id IS NOT NULL AND wa.musicbrainz_artist_id = sa.source_artist_id)
)
)
WHERE sa.profile_id = ? AND sa.similar_artist_name IN ({placeholders})
""", (profile_id, profile_id, *names))
# Collect distinct source names per recommendation, preserving
# nothing-special order then sorting for a deterministic result.
buckets: Dict[str, set] = {}
for row in cursor.fetchall():
src = row['source_name']
if not src:
continue
buckets.setdefault(row['rec_name'], set()).add(src)
return {
rec: sorted(srcs, key=lambda s: s.lower())[:max_per]
for rec, srcs in buckets.items()
}
except Exception as e:
logger.error(f"Error resolving recommendation sources: {e}")
return {}
def mark_artists_featured(self, artist_names: List[str]):
"""Update last_featured timestamp for artists shown in the hero slider"""
if not artist_names:
@ -12171,7 +12428,11 @@ class MusicDatabase:
WHERE profile_id = ?
ORDER BY updated_at DESC
""", (profile_id,))
return [dict(row) for row in cursor.fetchall()]
return [
self._normalize_mirrored_playlist_row(row)
for row in cursor.fetchall()
if row
]
except Exception as e:
logger.error(f"Error getting mirrored playlists: {e}")
return []
@ -12198,11 +12459,93 @@ class MusicDatabase:
cursor = conn.cursor()
cursor.execute("SELECT * FROM mirrored_playlists WHERE id = ?", (playlist_id,))
row = cursor.fetchone()
return dict(row) if row else None
return self._normalize_mirrored_playlist_row(row)
except Exception as e:
logger.error(f"Error getting mirrored playlist: {e}")
return None
@staticmethod
def _normalize_mirrored_playlist_row(row) -> Optional[Dict]:
if not row:
return None
pl = dict(row)
pl['organize_by_playlist'] = bool(pl.get('organize_by_playlist', 0))
return pl
def get_mirrored_playlist_by_source(
self,
source: str,
source_playlist_id: str,
profile_id: int = 1,
) -> Optional[Dict]:
"""Return a mirrored playlist by upstream source id."""
try:
with self._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
SELECT * FROM mirrored_playlists
WHERE source = ? AND source_playlist_id = ? AND profile_id = ?
""",
(source, str(source_playlist_id), profile_id),
)
row = cursor.fetchone()
return self._normalize_mirrored_playlist_row(row)
except Exception as e:
logger.error(f"Error getting mirrored playlist by source: {e}")
return None
def resolve_mirrored_playlist(
self,
playlist_ref: Any,
profile_id: int = 1,
*,
default_source: str = 'spotify',
) -> Optional[Dict]:
"""Resolve a mirrored playlist from an upstream source id or numeric PK.
Resolves by ``(source, source_playlist_id)`` FIRST, then falls back to
treating an all-digit ref as the mirrored-playlists primary key. The
order matters: some sources (e.g. Deezer) use all-numeric upstream ids,
and the old PK-first logic mistook those for the PK so the Deezer
organize-by-playlist toggle resolved the wrong row (or nothing).
"""
if playlist_ref is None or playlist_ref == '':
return None
ref = str(playlist_ref).strip()
if not ref:
return None
if default_source:
row = self.get_mirrored_playlist_by_source(default_source, ref, profile_id)
if row:
return row
if ref.isdigit():
return self.get_mirrored_playlist(int(ref))
return None
def set_mirrored_playlist_organize_by_playlist(
self,
playlist_id: int,
enabled: bool,
) -> bool:
"""Persist whether downloads for this playlist use playlist-folder layout."""
try:
with self._get_connection() as conn:
cursor = conn.cursor()
cursor.execute(
"""
UPDATE mirrored_playlists
SET organize_by_playlist = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""",
(1 if enabled else 0, playlist_id),
)
conn.commit()
return cursor.rowcount > 0
except Exception as e:
logger.error(f"Error updating organize_by_playlist for playlist {playlist_id}: {e}")
return False
def get_mirrored_playlist_tracks(self, playlist_id: int) -> List[Dict]:
"""Return all tracks for a mirrored playlist ordered by position."""
try:

View file

@ -65,3 +65,13 @@ tidalapi==0.8.11
flask-socketio==5.6.1
gunicorn==26.0.0
simple-websocket==1.1.0
# Full public-playlist link imports (no Spotify credentials). The "Spotify link"
# tab scrapes Spotify's embed widget, which caps at ~100 tracks; SpotipyFree
# (the no-creds spotipy drop-in spotDL uses) pulls the full list. It is GPL-3.0,
# used here as a normal pip dependency — aggregation, exactly like spotDL (also
# MIT). It is NOT vendored into SoulSync's own code, so the project stays MIT.
# The code soft-imports it and falls back to the embed scraper (~100 tracks) if
# it's missing or fails, so this is never a hard runtime requirement.
spotipyFree>=1.1.2,<2
websockets>=12 # required by SpotipyFree/spotapi, not pulled in automatically

View file

@ -115,6 +115,12 @@ class PlaylistSyncService:
logger.error("Navidrome client not provided to sync service")
return None, "navidrome"
return client, "navidrome"
elif active_server == "soulsync":
client = self._media_client('soulsync')
if not client:
logger.error("SoulSync library client not provided to sync service")
return None, "soulsync"
return client, "soulsync"
else: # Default to Plex
client = self._media_client('plex')
if profile_id and client:
@ -292,63 +298,92 @@ class PlaylistSyncService:
return self._create_error_result(playlist.name, ["Sync cancelled"])
media_client, server_type = self._get_active_media_client()
self._update_progress(playlist.name, f"Creating/updating {server_type.title()} playlist", "", 80, 5, 4,
total_tracks=total_tracks,
matched_tracks=len(matched_tracks),
failed_tracks=len(unmatched_tracks))
# Get the actual media server track objects
media_tracks = [r.plex_track for r in matched_tracks if r.plex_track] # plex_track is a generic name here
logger.info(f"Creating playlist with {len(media_tracks)} matched tracks")
unmatched_count = len(unmatched_tracks)
# Validate that all tracks have proper ratingKey attributes for playlist creation
valid_tracks = []
for i, track in enumerate(media_tracks):
if track and hasattr(track, 'ratingKey'):
valid_tracks.append(track)
logger.debug(f"Track {i+1} valid for playlist: '{track.title}' (ratingKey: {track.ratingKey})")
else:
logger.warning(f"Track {i+1} invalid for playlist: {track} (type: {type(track)}, has ratingKey: {hasattr(track, 'ratingKey') if track else 'N/A'})")
logger.info(f"Playlist validation: {len(valid_tracks)}/{len(media_tracks)} tracks are valid {server_type.title()} objects with ratingKeys")
# Deduplicate by ratingKey — media servers silently drop duplicates,
# so count should reflect what actually ends up in the playlist
seen_keys = set()
deduped_tracks = []
for t in valid_tracks:
if t.ratingKey not in seen_keys:
seen_keys.add(t.ratingKey)
deduped_tracks.append(t)
if len(deduped_tracks) < len(valid_tracks):
logger.info(f"Deduplicated {len(valid_tracks) - len(deduped_tracks)} duplicate ratingKeys ({len(valid_tracks)}{len(deduped_tracks)} tracks)")
# Use the deduplicated tracks for the sync
plex_tracks = deduped_tracks
# Use active media server for playlist sync
media_client, server_type = self._get_active_media_client()
if not media_client:
logger.error("No active media client available for playlist sync")
sync_success = False
else:
logger.info(
f"Syncing playlist '{playlist.name}' to {server_type.upper()} server "
f"(mode: {sync_mode})"
# SoulSync standalone has no server playlists — only library match +
# wishlist for missing files. Previously we fell through to Plex here,
# showed "Creating/updating Plex playlist", playlist write failed, and
# failed_tracks was computed as total - 0 synced (= entire playlist).
if server_type == 'soulsync':
self._update_progress(
playlist.name,
"Standalone: library check complete",
"",
80, 5, 4,
total_tracks=total_tracks,
matched_tracks=len(matched_tracks),
failed_tracks=unmatched_count,
)
# sync_mode == 'append' preserves user-added tracks on the server
# playlist (CJFC discord report) — never deletes, only adds new
# ones via the per-server `append_to_playlist`. The sync UI
# hides the Sync button entirely on SoulSync standalone (which
# has no playlist methods), so every client that reaches this
# point implements both methods.
if sync_mode == 'append':
sync_success = media_client.append_to_playlist(playlist.name, valid_tracks)
sync_success = True
synced_tracks = len(matched_tracks)
failed_tracks = unmatched_count
logger.info(
f"Standalone playlist sync '{playlist.name}': "
f"{len(matched_tracks)} in library, {unmatched_count} missing"
)
else:
self._update_progress(
playlist.name,
f"Creating/updating {server_type.title()} playlist",
"",
80, 5, 4,
total_tracks=total_tracks,
matched_tracks=len(matched_tracks),
failed_tracks=unmatched_count,
)
# Get the actual media server track objects
media_tracks = [r.plex_track for r in matched_tracks if r.plex_track]
logger.info(f"Creating playlist with {len(media_tracks)} matched tracks")
# Validate that all tracks have proper ratingKey attributes for playlist creation
valid_tracks = []
for i, track in enumerate(media_tracks):
if track and hasattr(track, 'ratingKey'):
valid_tracks.append(track)
logger.debug(f"Track {i+1} valid for playlist: '{track.title}' (ratingKey: {track.ratingKey})")
else:
logger.warning(
f"Track {i+1} invalid for playlist: {track} "
f"(type: {type(track)}, has ratingKey: {hasattr(track, 'ratingKey') if track else 'N/A'})"
)
logger.info(
f"Playlist validation: {len(valid_tracks)}/{len(media_tracks)} tracks are valid "
f"{server_type.title()} objects with ratingKeys"
)
# Deduplicate by ratingKey — media servers silently drop duplicates
seen_keys = set()
deduped_tracks = []
for t in valid_tracks:
if t.ratingKey not in seen_keys:
seen_keys.add(t.ratingKey)
deduped_tracks.append(t)
if len(deduped_tracks) < len(valid_tracks):
logger.info(
f"Deduplicated {len(valid_tracks) - len(deduped_tracks)} duplicate ratingKeys "
f"({len(valid_tracks)}{len(deduped_tracks)} tracks)"
)
plex_tracks = deduped_tracks
if not media_client:
logger.error("No active media client available for playlist sync")
sync_success = False
else:
sync_success = media_client.update_playlist(playlist.name, valid_tracks)
synced_tracks = len(plex_tracks) if sync_success else 0
failed_tracks = len(playlist.tracks) - synced_tracks - downloaded_tracks
logger.info(
f"Syncing playlist '{playlist.name}' to {server_type.upper()} server "
f"(mode: {sync_mode})"
)
if sync_mode == 'append':
sync_success = media_client.append_to_playlist(playlist.name, valid_tracks)
else:
sync_success = media_client.update_playlist(playlist.name, valid_tracks)
synced_tracks = len(plex_tracks) if sync_success else 0
# Not in library (for wishlist), not "total minus playlist size".
failed_tracks = unmatched_count
self._update_progress(playlist.name, "Sync completed", "", 100, 5, 5,
total_tracks=total_tracks,
@ -357,8 +392,11 @@ class PlaylistSyncService:
# Auto-add unmatched tracks to wishlist (skip in Wing It mode)
wishlist_added_count = 0
if unmatched_tracks and getattr(self, '_skip_wishlist', False):
logger.info(f"[Wing It] Skipping wishlist for {len(unmatched_tracks)} unmatched tracks")
if unmatched_tracks and getattr(self, '_skip_unmatched_wishlist', False):
logger.info(
"Skipping sync-time wishlist for %s unmatched tracks (wing-it or organize-by-playlist)",
len(unmatched_tracks),
)
unmatched_tracks = [] # Clear so the loop below doesn't run
if unmatched_tracks:
try:
@ -548,20 +586,13 @@ class PlaylistSyncService:
server_track_id = cached['server_track_id']
db_track_check = cache_db.get_track_by_id(server_track_id)
if db_track_check:
if server_type == "jellyfin":
class JellyfinTrackFromCache:
if server_type in ("jellyfin", "navidrome", "soulsync"):
class DbTrackFromCache:
def __init__(self, db_t):
self.ratingKey = db_t.id
self.title = db_t.title
self.id = db_t.id
actual_track = JellyfinTrackFromCache(db_track_check)
elif server_type == "navidrome":
class NavidromeTrackFromCache:
def __init__(self, db_t):
self.ratingKey = db_t.id
self.title = db_t.title
self.id = db_t.id
actual_track = NavidromeTrackFromCache(db_track_check)
actual_track = DbTrackFromCache(db_track_check)
else:
try:
actual_track = media_client.server.fetchItem(int(server_track_id))
@ -586,17 +617,35 @@ class PlaylistSyncService:
artist_name = _artist_name(artist)
# YouTube/streaming sources arrive video-shaped: title
# "Artist - Song", artist "Official Artist"/"Artist - Topic"/
# "ArtistVEVO". Try the raw (title, artist) first, then a
# canonicalized variant so those still match the clean library
# metadata instead of being reported missing (#768).
from core.text.source_title import canonical_source_track
_attempts = [(original_title, artist_name)]
_canon_title, _canon_artist = canonical_source_track(original_title, artist_name)
if (_canon_title, _canon_artist) != (original_title, artist_name):
_attempts.append((_canon_title, _canon_artist))
# Use the improved database check_track_exists method with server awareness
try:
db = MusicDatabase()
artist_candidates = self._get_or_fetch_artist_candidates(
candidate_pool, db, artist_name, active_server,
)
db_track, confidence = db.check_track_exists(
original_title, artist_name,
confidence_threshold=0.7, server_source=active_server,
candidate_tracks=artist_candidates,
)
# Try every candidate form and keep the BEST — don't stop at
# the first that clears the threshold, or a marginal raw
# match could mask a stronger (correct) canonical one.
db_track, confidence = None, 0.0
for _try_title, _try_artist in _attempts:
artist_candidates = self._get_or_fetch_artist_candidates(
candidate_pool, db, _try_artist, active_server,
)
_cand_track, _cand_conf = db.check_track_exists(
_try_title, _try_artist,
confidence_threshold=0.7, server_source=active_server,
candidate_tracks=artist_candidates,
)
if _cand_conf > confidence:
db_track, confidence = _cand_track, _cand_conf
if db_track and confidence >= 0.7:
logger.debug(f"Database match found for '{original_title}' by '{artist_name}': '{db_track.title}' with confidence {confidence:.2f}")
@ -615,27 +664,19 @@ class PlaylistSyncService:
# Fetch the actual track object from active media server using the database track ID
try:
if server_type == "jellyfin":
# For Jellyfin, create a track object from database info (Jellyfin doesn't have fetchItem)
class JellyfinTrackFromDB:
def __init__(self, db_track):
self.ratingKey = db_track.id
self.title = db_track.title
self.id = db_track.id
actual_track = JellyfinTrackFromDB(db_track)
logger.debug(f"Created Jellyfin track object for '{db_track.title}' (ID: {actual_track.ratingKey})")
return actual_track, confidence
elif server_type == "navidrome":
# For Navidrome, create a track object from database info (similar to Jellyfin)
class NavidromeTrackFromDB:
if server_type in ("jellyfin", "navidrome", "soulsync"):
# DB-backed servers — no remote fetchItem (SoulSync standalone included).
class DbTrackFromDB:
def __init__(self, db_track):
self.ratingKey = db_track.id
self.title = db_track.title
self.id = db_track.id
actual_track = NavidromeTrackFromDB(db_track)
logger.debug(f"Created Navidrome track object for '{db_track.title}' (ID: {actual_track.ratingKey})")
actual_track = DbTrackFromDB(db_track)
logger.debug(
f"Created {server_type} track object for '{db_track.title}' "
f"(ID: {actual_track.ratingKey})"
)
return actual_track, confidence
else:
# For Plex, use the original fetchItem approach

View file

@ -126,6 +126,8 @@ def _build_deps(engine, scan_mgr=None) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,

View file

@ -73,6 +73,8 @@ def _build_deps(**overrides) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,

View file

@ -47,6 +47,8 @@ def _build_deps(**overrides) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,

View file

@ -108,6 +108,8 @@ def _build_deps(**overrides) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,
@ -737,8 +739,38 @@ class TestSyncPlaylist:
time.sleep(0.01)
assert len(sync_calls) == 1
def test_organize_by_playlist_passes_skip_wishlist_add(self):
discovered_track = {
'extra_data': json.dumps({
'discovered': True,
'matched_data': {
'id': 'spot-1', 'name': 'Track', 'artists': [{'name': 'X'}],
'album': {'name': 'Album'}, 'duration_ms': 200000,
},
}),
'artist_name': 'X',
}
db = _StubDB(
playlists=[{'id': 1, 'name': 'P', 'organize_by_playlist': True}],
playlist_tracks={1: [discovered_track]},
)
sync_calls: List[tuple] = []
deps = _build_deps(
get_database=lambda: db,
run_sync_task=lambda *a, **k: sync_calls.append((a, k)),
)
auto_sync_playlist({'playlist_id': '1'}, deps)
for _ in range(50):
if sync_calls:
break
import time
time.sleep(0.01)
assert sync_calls
assert sync_calls[0][1].get('skip_wishlist_add') is True
def test_unchanged_since_last_sync_returns_skipped(self):
discovered_track = {
'source_track_id': 'spot-1',
'extra_data': json.dumps({
'discovered': True,
'matched_data': {
@ -758,7 +790,11 @@ class TestSyncPlaylist:
import hashlib
expected_hash = hashlib.md5('spot-1'.encode()).hexdigest()
sync_statuses = {
'auto_mirror_1': {'tracks_hash': expected_hash, 'matched_tracks': 1}
'auto_mirror_1': {
'tracks_hash': expected_hash,
'mirror_tracks_hash': expected_hash,
'matched_tracks': 1,
}
}
deps = _build_deps(
@ -769,6 +805,96 @@ class TestSyncPlaylist:
assert result['status'] == 'skipped'
assert 'unchanged' in result['reason']
def test_playlist_changed_event_bypasses_unchanged_skip(self):
discovered_track = {
'source_track_id': 'spot-1',
'extra_data': json.dumps({
'discovered': True,
'matched_data': {
'id': 'spot-1', 'name': 'T', 'artists': [{'name': 'X'}],
'album': {'name': 'A'}, 'duration_ms': 0,
},
}),
'artist_name': 'X',
}
db = _StubDB(
playlists=[{'id': 1, 'name': 'P'}],
playlist_tracks={1: [discovered_track]},
)
import hashlib
expected_hash = hashlib.md5('spot-1'.encode()).hexdigest()
sync_statuses = {
'auto_mirror_1': {
'tracks_hash': expected_hash,
'mirror_tracks_hash': expected_hash,
'matched_tracks': 1,
}
}
sync_calls: List[tuple] = []
deps = _build_deps(
get_database=lambda: db,
load_sync_status_file=lambda: sync_statuses,
run_sync_task=lambda *a, **k: sync_calls.append((a, k)),
)
result = auto_sync_playlist(
{'playlist_id': '1', '_event_data': {'added': '1', 'playlist_id': '1'}},
deps,
)
assert result['status'] == 'started'
import time
for _ in range(50):
if sync_calls:
break
time.sleep(0.01)
assert len(sync_calls) == 1
def test_new_mirror_row_with_skipped_track_bypasses_unchanged_skip(self):
"""New playlist row without discovery must not reuse the old tracks_hash skip."""
discovered_track = {
'source_track_id': 'spot-1',
'extra_data': json.dumps({
'discovered': True,
'matched_data': {
'id': 'spot-1', 'name': 'T', 'artists': [{'name': 'X'}],
'album': {'name': 'A'}, 'duration_ms': 0,
},
}),
'artist_name': 'X',
}
db = _StubDB(
playlists=[{'id': 1, 'name': 'P'}],
playlist_tracks={1: [discovered_track, {}]}, # second row not syncable
)
import hashlib
old_hash = hashlib.md5('spot-1'.encode()).hexdigest()
new_mirror_hash = hashlib.md5('spot-1,spot-new'.encode()).hexdigest()
sync_statuses = {
'auto_mirror_1': {
'tracks_hash': old_hash,
'mirror_tracks_hash': old_hash,
'matched_tracks': 1,
}
}
# Give the new mirror row a source id so mirror hash changes.
db.playlist_tracks[1][1]['source_track_id'] = 'spot-new'
sync_calls: List[tuple] = []
deps = _build_deps(
get_database=lambda: db,
load_sync_status_file=lambda: sync_statuses,
run_sync_task=lambda *a, **k: sync_calls.append((a, k)),
)
result = auto_sync_playlist({'playlist_id': '1'}, deps)
assert result['status'] == 'started'
assert result['skipped_tracks'] == '1'
import time
for _ in range(50):
if sync_calls:
break
time.sleep(0.01)
assert len(sync_calls) == 1
assert new_mirror_hash != old_hash
# ─── playlist_pipeline ───────────────────────────────────────────────

View file

@ -60,6 +60,8 @@ def _build_deps(**overrides: Any) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,

View file

@ -0,0 +1,117 @@
"""Tests for organize-by-playlist integration in the playlist pipeline tail."""
from unittest.mock import MagicMock
from core.automation.deps import AutomationDeps, AutomationState
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
import threading
def _minimal_deps(**overrides):
base = dict(
engine=MagicMock(),
state=AutomationState(),
config_manager=MagicMock(),
update_progress=lambda *a, **k: None,
logger=MagicMock(),
get_database=lambda: MagicMock(),
spotify_client=None,
tidal_client=None,
web_scan_manager=None,
process_wishlist_automatically=lambda **k: None,
process_watchlist_scan_automatically=lambda **k: None,
is_wishlist_actually_processing=lambda: False,
is_watchlist_actually_scanning=lambda: False,
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'started', 'batch_id': 'b1'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,
get_sync_states=lambda: {},
set_db_update_automation_id=lambda v: None,
get_db_update_state=lambda: {},
db_update_lock=threading.Lock(),
db_update_executor=None,
run_db_update_task=lambda *a, **k: None,
run_deep_scan_task=lambda *a, **k: None,
get_duplicate_cleaner_state=lambda: {},
duplicate_cleaner_lock=threading.Lock(),
duplicate_cleaner_executor=None,
run_duplicate_cleaner=lambda: None,
get_quality_scanner_state=lambda: {},
quality_scanner_lock=threading.Lock(),
quality_scanner_executor=None,
run_quality_scanner=lambda *a, **k: None,
download_orchestrator=None,
run_async=lambda coro: None,
tasks_lock=threading.Lock(),
get_download_batches=lambda: {},
get_download_tasks=lambda: {},
sweep_empty_download_directories=lambda: 0,
get_staging_path=lambda: '/staging',
docker_resolve_path=lambda p: p,
get_current_profile_id=lambda: 1,
get_watchlist_scanner=lambda spc: None,
get_app=lambda: None,
get_beatport_data_cache=lambda: {'cache_lock': threading.Lock(), 'homepage': {}},
init_automation_progress=lambda *a, **k: None,
record_progress_history=lambda *a, **k: None,
build_personalized_manager=lambda: None,
)
base.update(overrides)
return AutomationDeps(**base) # type: ignore[arg-type]
def test_all_organize_playlists_skips_wishlist():
wishlist_calls = []
def sync_one(_pl):
return {'status': 'skipped', 'reason': 'unchanged'}
deps = _minimal_deps(
process_wishlist_automatically=lambda **k: wishlist_calls.append(k),
)
playlists = [
{'id': 1, 'name': 'A', 'organize_by_playlist': True},
]
result = run_sync_and_wishlist(
deps,
'auto-1',
playlists,
sync_one_fn=sync_one,
sync_id_for_fn=lambda pl: f'mirror_{pl["id"]}',
skip_wishlist=False,
)
assert result['wishlist_queued'] == 0
assert result['organize_downloads_started'] == 1
assert wishlist_calls == []
def test_mixed_playlists_still_runs_wishlist():
wishlist_calls = []
deps = _minimal_deps(
process_wishlist_automatically=lambda **k: wishlist_calls.append(1),
is_wishlist_actually_processing=lambda: False,
)
playlists = [
{'id': 1, 'name': 'Organized', 'organize_by_playlist': True},
{'id': 2, 'name': 'Normal', 'organize_by_playlist': False},
]
def sync_one(_pl):
return {'status': 'skipped'}
result = run_sync_and_wishlist(
deps,
None,
playlists,
sync_one_fn=sync_one,
sync_id_for_fn=lambda pl: f'mirror_{pl["id"]}',
)
assert result['organize_downloads_started'] == 1
assert result['wishlist_queued'] == 1
assert len(wishlist_calls) == 1

View file

@ -46,6 +46,8 @@ def _build_deps(**overrides) -> AutomationDeps:
get_watchlist_scan_state=lambda: {},
run_playlist_discovery_worker=lambda *a, **k: None,
run_sync_task=lambda *a, **k: None,
run_playlist_organize_download=lambda **k: {'status': 'skipped'},
missing_download_executor=None,
load_sync_status_file=lambda: {},
get_deezer_client=lambda: None,
parse_youtube_playlist=lambda url: None,

View file

@ -20,6 +20,7 @@ class _FakeSyncResult:
failed_tracks: int = 1
synced_tracks: int = 4
total_tracks: int = 6
wishlist_added_count: int = 0
match_details: list = None
def __post_init__(self):
@ -139,6 +140,9 @@ def _build_deps(
update_automation_progress=None,
update_and_save_sync_status=None,
run_async=None,
process_wishlist_automatically=None,
run_playlist_organize_download=None,
is_wishlist_actually_processing=None,
):
return ds.SyncDeps(
config_manager=config or _FakeConfig(),
@ -154,6 +158,9 @@ def _build_deps(
update_and_save_sync_status=update_and_save_sync_status or (lambda *a, **kw: None),
sync_states=sync_states if sync_states is not None else {},
sync_lock=sync_lock or threading.Lock(),
process_wishlist_automatically=process_wishlist_automatically,
run_playlist_organize_download=run_playlist_organize_download,
is_wishlist_actually_processing=is_wishlist_actually_processing,
)
@ -445,6 +452,64 @@ def test_update_and_save_sync_status_called(patched_db):
assert kwargs.get('tracks_hash') # md5 hash present
# ---------------------------------------------------------------------------
# Post-sync automation follow-up
# ---------------------------------------------------------------------------
def test_post_sync_triggers_wishlist_processor_for_mirror_automation(patched_db):
wishlist_calls = []
result = _FakeSyncResult(
matched_tracks=5,
failed_tracks=2,
wishlist_added_count=2,
total_tracks=7,
)
svc = _FakeSyncService(media_client=_FakeMediaClient(), sync_result=result)
deps = _build_deps(
sync_service=svc,
process_wishlist_automatically=lambda **kw: wishlist_calls.append(kw),
is_wishlist_actually_processing=lambda: False,
)
ds.run_sync_task(
'auto_mirror_42',
'Mirror',
[_track()],
automation_id='auto-1',
deps=deps,
)
assert len(wishlist_calls) == 1
assert wishlist_calls[0]['automation_id'] == 'auto-1'
def test_post_sync_starts_organize_download_when_skip_wishlist_add(patched_db):
org_calls = []
result = _FakeSyncResult(
matched_tracks=50,
failed_tracks=10,
total_tracks=60,
)
svc = _FakeSyncService(media_client=_FakeMediaClient(), sync_result=result)
deps = _build_deps(
sync_service=svc,
run_playlist_organize_download=lambda **kw: org_calls.append(kw) or {'status': 'started'},
)
ds.run_sync_task(
'auto_mirror_7',
'Organized',
[_track()],
automation_id='auto-2',
deps=deps,
skip_wishlist_add=True,
)
assert len(org_calls) == 1
assert org_calls[0]['mirrored_playlist_id'] == 7
assert org_calls[0]['automation_id'] == 'auto-2'
# ---------------------------------------------------------------------------
# Cleanup (finally)
# ---------------------------------------------------------------------------

View file

@ -0,0 +1,88 @@
"""Tests for playlist-folder existence detection."""
import os
from unittest.mock import patch
import pytest
from core.downloads.playlist_folder import (
candidate_playlist_folder_paths,
resolve_playlist_folder_mode_for_batch,
track_exists_in_playlist_folder,
)
class _FakeDB:
def __init__(self, mirrored=None):
self._mirrored = mirrored
def resolve_mirrored_playlist(self, playlist_ref, profile_id=1, default_source='spotify'):
return self._mirrored
def test_track_exists_in_playlist_folder_finds_file(tmp_path):
playlist_dir = tmp_path / 'My Playlist'
playlist_dir.mkdir()
track_file = playlist_dir / 'Artist A - Song One.flac'
track_file.write_bytes(b'x')
with patch('core.downloads.playlist_folder._get_config_manager') as cfg:
cfg.return_value.get.return_value = str(tmp_path)
with patch('core.downloads.playlist_folder.docker_resolve_path', side_effect=lambda p: p):
with patch(
'core.downloads.playlist_folder.get_file_path_from_template',
return_value=('', ''),
):
assert track_exists_in_playlist_folder('My Playlist', 'Artist A', 'Song One')
def test_track_exists_in_playlist_folder_missing(tmp_path):
with patch('core.downloads.playlist_folder._get_config_manager') as cfg:
cfg.return_value.get.return_value = str(tmp_path)
with patch('core.downloads.playlist_folder.docker_resolve_path', side_effect=lambda p: p):
with patch(
'core.downloads.playlist_folder.get_file_path_from_template',
return_value=('', ''),
):
assert not track_exists_in_playlist_folder('My Playlist', 'Artist A', 'Song One')
def test_candidate_paths_template_layout(tmp_path):
with patch('core.downloads.playlist_folder._get_config_manager') as cfg:
cfg.return_value.get.return_value = str(tmp_path)
with patch('core.downloads.playlist_folder.docker_resolve_path', side_effect=lambda p: p):
with patch(
'core.downloads.playlist_folder.get_file_path_from_template',
return_value=('Cool Mix', 'Artist - Title'),
):
paths = candidate_playlist_folder_paths('Cool Mix', 'Artist', 'Title')
assert any(p.endswith('.flac') for p in paths)
assert all('Cool Mix' in p for p in paths)
def test_resolve_playlist_folder_mode_from_mirrored():
db = _FakeDB(mirrored={
'id': 5,
'name': 'Rekordbox Set',
'organize_by_playlist': True,
})
enabled, name = resolve_playlist_folder_mode_for_batch(
db,
playlist_id='37i9dQZF1',
playlist_name='Other Name',
batch_playlist_folder_mode=False,
)
assert enabled is True
assert name == 'Rekordbox Set'
def test_resolve_playlist_folder_mode_batch_flag():
db = _FakeDB()
enabled, name = resolve_playlist_folder_mode_for_batch(
db,
playlist_id='1',
playlist_name='Batch Name',
batch_playlist_folder_mode=True,
)
assert enabled is True
assert name == 'Batch Name'

View file

@ -0,0 +1,263 @@
"""Unmatched-browser backend for the Manage Enrichment Workers modal.
Three seams:
* pure SQL builders + validation (core.enrichment.unmatched)
* the MusicDatabase read methods against a temp DB
* the Flask routes via a test client
"""
from __future__ import annotations
import pytest
from flask import Flask
from core.enrichment import api as enrichment_api
from core.enrichment.unmatched import (
MAX_LIMIT,
UnmatchedQueryError,
build_breakdown_query,
build_count_query,
build_reset_query,
build_unmatched_query,
supported_entity_types,
)
from database.music_database import MusicDatabase
# --------------------------------------------------------------------------
# Pure builders / validation
# --------------------------------------------------------------------------
def test_unknown_service_rejected():
with pytest.raises(UnmatchedQueryError):
build_unmatched_query('not-a-service', 'artist')
def test_unsupported_entity_type_rejected():
# Genius enriches artists + tracks but has no album-level id column.
assert 'album' not in supported_entity_types('genius')
with pytest.raises(UnmatchedQueryError):
build_unmatched_query('genius', 'album')
with pytest.raises(UnmatchedQueryError):
build_breakdown_query('discogs', 'track') # discogs has no track column
def test_bad_status_rejected():
with pytest.raises(UnmatchedQueryError):
build_unmatched_query('spotify', 'artist', status='bogus')
def test_status_predicates():
nf, _ = build_count_query('spotify', 'artist', 'not_found')
pend, _ = build_count_query('spotify', 'artist', 'pending')
un, _ = build_count_query('spotify', 'artist', 'unmatched')
assert "artists.spotify_match_status = 'not_found'" in nf
assert "artists.spotify_match_status IS NULL" in pend
assert "IS NULL OR" in un and "= 'not_found'" in un
def test_track_query_qualifies_status_to_avoid_join_ambiguity():
# tracks LEFT JOIN albums for artwork — both carry spotify_match_status,
# so the predicate must be qualified or SQLite errors "ambiguous column".
sql, _ = build_unmatched_query('spotify', 'track', 'not_found')
assert 'LEFT JOIN albums al' in sql
assert 'tracks.spotify_match_status' in sql
assert 'al.thumb_url AS image_url' in sql
def test_search_adds_like_param():
sql, params = build_unmatched_query('spotify', 'artist', 'not_found', query='dragons')
assert 'LIKE ?' in sql
assert '%dragons%' in params
def test_limit_is_clamped():
_, params = build_unmatched_query('spotify', 'artist', 'not_found', limit=99999)
assert params[-2] == MAX_LIMIT # limit
assert params[-1] == 0 # offset
_, params2 = build_unmatched_query('spotify', 'artist', 'not_found', limit=0)
assert params2[-2] == 50 # invalid -> default
# --------------------------------------------------------------------------
# MusicDatabase integration (temp DB)
# --------------------------------------------------------------------------
def _seed(db: MusicDatabase):
conn = db._get_connection()
cur = conn.cursor()
# 3 artists: matched / not_found / pending(NULL)
cur.execute("INSERT INTO artists (id, name, spotify_match_status) VALUES ('a1','Matched Artist','matched')")
cur.execute("INSERT INTO artists (id, name, spotify_match_status) VALUES ('a2','Failed Dragons','not_found')")
cur.execute("INSERT INTO artists (id, name) VALUES ('a3','Pending Person')") # NULL status
# album + track to exercise the join-for-artwork path
cur.execute("INSERT INTO albums (id, artist_id, title, thumb_url, spotify_match_status) "
"VALUES ('al1','a2','Evolve','http://img/evolve.jpg','not_found')")
cur.execute("INSERT INTO tracks (id, album_id, artist_id, title, spotify_match_status) "
"VALUES ('t1','al1','a2','Believer','not_found')")
conn.commit()
conn.close()
@pytest.fixture
def db(tmp_path):
d = MusicDatabase(str(tmp_path / 'enrich.db'))
_seed(d)
return d
def test_breakdown_splits_matched_notfound_pending(db):
bd = db.get_enrichment_breakdown('spotify', 'artist')
assert bd == {'matched': 1, 'not_found': 1, 'pending': 1, 'total': 3}
def test_unmatched_not_found_only(db):
res = db.get_enrichment_unmatched('spotify', 'artist', status='not_found')
assert res['total'] == 1
assert [i['name'] for i in res['items']] == ['Failed Dragons']
assert res['items'][0]['status'] == 'not_found'
def test_unmatched_pending_only(db):
res = db.get_enrichment_unmatched('spotify', 'artist', status='pending')
assert res['total'] == 1
assert res['items'][0]['name'] == 'Pending Person'
def test_unmatched_combined(db):
res = db.get_enrichment_unmatched('spotify', 'artist', status='unmatched')
assert res['total'] == 2
assert {i['name'] for i in res['items']} == {'Failed Dragons', 'Pending Person'}
def test_unmatched_search_filters_by_name(db):
res = db.get_enrichment_unmatched('spotify', 'artist', status='unmatched', query='dragons')
assert res['total'] == 1
assert res['items'][0]['name'] == 'Failed Dragons'
def test_unmatched_pagination(db):
page = db.get_enrichment_unmatched('spotify', 'artist', status='unmatched', limit=1, offset=0)
assert page['total'] == 2 and len(page['items']) == 1
page2 = db.get_enrichment_unmatched('spotify', 'artist', status='unmatched', limit=1, offset=1)
assert page2['items'][0]['name'] != page['items'][0]['name']
def test_track_unmatched_borrows_album_artwork(db):
res = db.get_enrichment_unmatched('spotify', 'track', status='not_found')
assert res['total'] == 1
assert res['items'][0]['name'] == 'Believer'
assert res['items'][0]['image_url'] == 'http://img/evolve.jpg'
def test_unmatched_includes_parent_context(db):
# album's parent is its artist; track's parent is its album
album = db.get_enrichment_unmatched('spotify', 'album', status='not_found')['items'][0]
assert album['parent'] == 'Failed Dragons'
track = db.get_enrichment_unmatched('spotify', 'track', status='not_found')['items'][0]
assert track['parent'] == 'Evolve'
artist = db.get_enrichment_unmatched('spotify', 'artist', status='not_found')['items'][0]
assert artist['parent'] is None
def test_db_raises_on_bad_input(db):
with pytest.raises(UnmatchedQueryError):
db.get_enrichment_unmatched('spotify', 'artist', status='bogus')
# --------------------------------------------------------------------------
# Reset / retry (re-queue) — must clear match_status to NULL so the worker
# re-attempts (nulling last_attempted alone leaves not_found in limbo).
# --------------------------------------------------------------------------
def test_reset_builder_requires_id_for_item():
with pytest.raises(UnmatchedQueryError):
build_reset_query('spotify', 'artist', 'item') # no entity_id
def test_reset_builder_bad_scope():
with pytest.raises(UnmatchedQueryError):
build_reset_query('spotify', 'artist', 'bogus', entity_id='x')
def test_reset_builder_nulls_status_not_just_attempted():
sql, _ = build_reset_query('spotify', 'artist', 'failed')
assert 'spotify_match_status = NULL' in sql
assert 'spotify_last_attempted = NULL' in sql
assert "WHERE spotify_match_status = 'not_found'" in sql
def test_reset_item_requeues_to_pending(db):
n = db.reset_enrichment('spotify', 'artist', 'item', entity_id='a2') # was not_found
assert n == 1
# not_found dropped by 1, pending gained 1
bd = db.get_enrichment_breakdown('spotify', 'artist')
assert bd == {'matched': 1, 'not_found': 0, 'pending': 2, 'total': 3}
def test_reset_failed_requeues_all(db):
n = db.reset_enrichment('spotify', 'album', 'failed') # one not_found album
assert n == 1
bd = db.get_enrichment_breakdown('spotify', 'album')
assert bd['not_found'] == 0 and bd['pending'] == 1
# --------------------------------------------------------------------------
# Flask routes
# --------------------------------------------------------------------------
@pytest.fixture
def client(db):
enrichment_api.configure(db_getter=lambda: db)
app = Flask(__name__)
app.register_blueprint(enrichment_api.create_blueprint())
with app.test_client() as c:
yield c
enrichment_api.configure(db_getter=None) # reset module global
def test_route_unknown_service_404(client):
assert client.get('/api/enrichment/bogus/unmatched').status_code == 404
def test_route_bad_entity_type_400(client):
# genius has no album column -> 400, not a 500
r = client.get('/api/enrichment/genius/unmatched?entity_type=album')
assert r.status_code == 400
def test_route_happy_path(client):
r = client.get('/api/enrichment/spotify/unmatched?entity_type=artist&status=unmatched')
assert r.status_code == 200
body = r.get_json()
assert body['total'] == 2
assert body['service'] == 'spotify'
assert body['entity_types'] == ['artist', 'album', 'track']
def test_route_breakdown(client):
r = client.get('/api/enrichment/spotify/breakdown')
assert r.status_code == 200
bd = r.get_json()['breakdown']
assert bd['artist'] == {'matched': 1, 'not_found': 1, 'pending': 1, 'total': 3}
def test_route_retry_item(client):
r = client.post('/api/enrichment/spotify/retry',
json={'entity_type': 'artist', 'scope': 'item', 'entity_id': 'a2'})
assert r.status_code == 200
body = r.get_json()
assert body['success'] is True and body['reset'] == 1
def test_route_retry_failed_bulk(client):
r = client.post('/api/enrichment/spotify/retry',
json={'entity_type': 'artist', 'scope': 'failed'})
assert r.status_code == 200
assert r.get_json()['reset'] == 1 # one not_found artist re-queued
def test_route_retry_item_missing_id_400(client):
r = client.post('/api/enrichment/spotify/retry',
json={'entity_type': 'artist', 'scope': 'item'})
assert r.status_code == 400

View file

@ -0,0 +1,138 @@
"""Priority 'process this group first' helper for enrichment workers.
The shared helper returns one pending item of a chosen entity type in the
shape the worker's dispatch already expects (with Spotify/iTunes mapped to
their album_individual / track_individual types). Default path (no override)
is exercised by the workers themselves and unchanged.
"""
from __future__ import annotations
import pytest
from core.worker_utils import (
PRIORITY_ENTITIES,
priority_pending_item,
read_enrichment_priority,
)
from database.music_database import MusicDatabase
@pytest.fixture
def db(tmp_path):
d = MusicDatabase(str(tmp_path / 'prio.db'))
conn = d._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name) VALUES ('a1', 'Pending Artist')") # NULL status
cur.execute("INSERT INTO artists (id, name, spotify_match_status) VALUES ('a2','Done','matched')")
cur.execute("INSERT INTO albums (id, artist_id, title) VALUES ('al1','a2','Pending Album')") # NULL status
cur.execute("INSERT INTO tracks (id, album_id, artist_id, title) VALUES ('t1','al1','a2','Pending Track')")
conn.commit()
conn.close()
return d
def _cur(db):
return db._get_connection().cursor()
def test_priority_artist_shape(db):
item = priority_pending_item(_cur(db), 'spotify', 'artist')
assert item == {'type': 'artist', 'id': 'a1', 'name': 'Pending Artist'}
def test_priority_album_default_type(db):
item = priority_pending_item(_cur(db), 'spotify', 'album')
assert item['id'] == 'al1' and item['name'] == 'Pending Album' and item['artist'] == 'Done'
assert item['type'] == 'album' # default type string
def test_priority_album_type_override_for_spotify_itunes(db):
item = priority_pending_item(_cur(db), 'spotify', 'album',
{'album': 'album_individual', 'track': 'track_individual'})
assert item['type'] == 'album_individual' # matches Spotify/iTunes dispatch
def test_priority_track_shape(db):
item = priority_pending_item(_cur(db), 'spotify', 'track')
assert item['id'] == 't1' and item['type'] == 'track' and item['artist'] == 'Done'
def test_priority_returns_none_when_group_exhausted(db):
# No pending artists once a1 is matched -> None, so worker resumes its chain.
conn = db._get_connection(); conn.execute("UPDATE artists SET spotify_match_status='matched' WHERE id='a1'"); conn.commit(); conn.close()
assert priority_pending_item(_cur(db), 'spotify', 'artist') is None
def test_priority_rejects_bad_entity_and_service(db):
assert priority_pending_item(_cur(db), 'spotify', 'bogus') is None
assert priority_pending_item(_cur(db), 'spot;drop', 'artist') is None # non-alpha service blocked
def test_read_priority_unset_is_empty():
# Unknown/unset key -> '' (no override). Uses the real config_manager.
assert read_enrichment_priority('definitely_not_a_service') == ''
def test_read_priority_roundtrip():
from config.settings import config_manager
key = 'spotify_enrichment_priority'
old = config_manager.get(key, '')
try:
config_manager.set(key, 'album')
assert read_enrichment_priority('spotify') == 'album'
config_manager.set(key, 'bogus')
assert read_enrichment_priority('spotify') == '' # invalid -> ignored
finally:
config_manager.set(key, old)
def test_priority_entities_constant():
assert PRIORITY_ENTITIES == ('artist', 'album', 'track')
# --- priority GET/POST routes ---------------------------------------------
@pytest.fixture
def client():
from flask import Flask
from core.enrichment import api as enrichment_api
store = {}
enrichment_api.configure(
config_get=lambda k, d=None: store.get(k, d),
config_set=lambda k, v: store.__setitem__(k, v),
db_getter=lambda: None,
)
app = Flask(__name__)
app.register_blueprint(enrichment_api.create_blueprint())
with app.test_client() as c:
c._store = store
yield c
enrichment_api.configure(config_get=None, config_set=None, db_getter=None)
def test_route_priority_get_default_empty(client):
r = client.get('/api/enrichment/spotify/priority')
assert r.status_code == 200
assert r.get_json()['priority'] == ''
def test_route_priority_set_and_get(client):
assert client.post('/api/enrichment/spotify/priority', json={'entity': 'album'}).status_code == 200
assert client._store['spotify_enrichment_priority'] == 'album'
assert client.get('/api/enrichment/spotify/priority').get_json()['priority'] == 'album'
def test_route_priority_clear(client):
client.post('/api/enrichment/spotify/priority', json={'entity': 'album'})
client.post('/api/enrichment/spotify/priority', json={'entity': 'none'})
assert client.get('/api/enrichment/spotify/priority').get_json()['priority'] == ''
def test_route_priority_rejects_unsupported_entity(client):
# Genius has no albums -> 400
assert client.post('/api/enrichment/genius/priority', json={'entity': 'album'}).status_code == 400
def test_route_priority_unknown_service_404(client):
assert client.get('/api/enrichment/bogus/priority').status_code == 404

View file

@ -17,6 +17,84 @@ def test_sanitize_filename_replaces_illegal_characters():
assert import_paths.sanitize_filename("AUX.txt").startswith("_")
# ── #767: dry-run path build must not create folders ──────────────────────
def _album_path_config(tmp_path):
return _Config({
"soulseek.transfer_path": str(tmp_path / "Transfer"),
"file_organization.enabled": True,
"file_organization.templates": {
"album_path": "$albumartist/$albumartist - $album/$track - $title",
"single_path": "$artist/$artist - $title",
},
"file_organization.collab_artist_mode": "first",
"file_organization.disc_label": "Disc",
})
def _album_context():
return {
"artist": {"name": "Lenka"},
"album": {"name": "Lenka", "id": "album-1", "release_date": "2008-01-01",
"total_tracks": 12, "album_type": "album", "artists": [{"name": "Lenka"}]},
"track_info": {"name": "The Show", "id": "t1", "track_number": 1,
"disc_number": 1, "artists": [{"name": "Lenka"}]},
"original_search_result": {"title": "The Show", "clean_title": "The Show",
"clean_album": "Lenka", "clean_artist": "Lenka",
"artists": [{"name": "Lenka"}]},
"source": "deezer", "is_album_download": False,
}
def test_create_dirs_false_does_not_create_folders(monkeypatch, tmp_path):
monkeypatch.setattr(import_paths, "_get_config_manager", lambda: _album_path_config(tmp_path))
monkeypatch.setattr(import_paths, "_get_album_tracks_for_source", lambda *a: None)
final_path, created = import_paths.build_final_path_for_track(
_album_context(), {"name": "Lenka"},
{"is_album": True, "album_name": "Lenka", "track_number": 1, "disc_number": 1},
".flac", create_dirs=False,
)
# Path is still computed correctly...
assert created is True
assert final_path == str(tmp_path / "Transfer" / "Lenka" / "Lenka - Lenka" / "01 - The Show.flac")
# ...but NOTHING was written to disk — not even the Transfer root.
assert not (tmp_path / "Transfer").exists()
def test_create_dirs_true_still_creates_folders(monkeypatch, tmp_path):
# The download/import flow must keep working (default behavior unchanged).
monkeypatch.setattr(import_paths, "_get_config_manager", lambda: _album_path_config(tmp_path))
monkeypatch.setattr(import_paths, "_get_album_tracks_for_source", lambda *a: None)
final_path, created = import_paths.build_final_path_for_track(
_album_context(), {"name": "Lenka"},
{"is_album": True, "album_name": "Lenka", "track_number": 1, "disc_number": 1},
".flac", # create_dirs defaults True
)
assert created is True
assert (tmp_path / "Transfer" / "Lenka" / "Lenka - Lenka").is_dir()
def test_create_dirs_false_and_true_yield_identical_path(monkeypatch, tmp_path):
monkeypatch.setattr(import_paths, "_get_config_manager", lambda: _album_path_config(tmp_path))
monkeypatch.setattr(import_paths, "_get_album_tracks_for_source", lambda *a: None)
dry, _ = import_paths.build_final_path_for_track(
_album_context(), {"name": "Lenka"},
{"is_album": True, "album_name": "Lenka", "track_number": 1, "disc_number": 1},
".flac", create_dirs=False,
)
live, _ = import_paths.build_final_path_for_track(
_album_context(), {"name": "Lenka"},
{"is_album": True, "album_name": "Lenka", "track_number": 1, "disc_number": 1},
".flac", create_dirs=True,
)
assert dry == live
def test_build_simple_download_destination_uses_album_folder(monkeypatch, tmp_path):
config = _Config({"soulseek.transfer_path": str(tmp_path / "Transfer")})
monkeypatch.setattr(import_paths, "_get_config_manager", lambda: config)

View file

@ -202,7 +202,12 @@ def test_post_process_matched_download_forwards_separate_metadata_runtime(tmp_pa
monkeypatch.setattr(import_pipeline, "emit_track_downloaded", lambda *args, **kwargs: None)
monkeypatch.setattr(import_pipeline, "record_library_history_download", lambda *args, **kwargs: None)
monkeypatch.setattr(import_pipeline, "record_download_provenance", lambda *args, **kwargs: None)
monkeypatch.setattr(import_pipeline, "record_soulsync_library_entry", lambda *args, **kwargs: None)
library_calls = []
def _record_library(context, artist_context, album_info):
library_calls.append((context, artist_context, album_info))
monkeypatch.setattr(import_pipeline, "record_soulsync_library_entry", _record_library)
monkeypatch.setattr(import_pipeline, "check_and_remove_from_wishlist", lambda *args, **kwargs: None)
monkeypatch.setattr(import_pipeline, "record_retag_download", lambda *args, **kwargs: None)
@ -221,6 +226,8 @@ def test_post_process_matched_download_forwards_separate_metadata_runtime(tmp_pa
)
assert seen["runtime"] is metadata_runtime
assert len(library_calls) == 1
assert library_calls[0][2]["album_name"] == "Album"
# ---------------------------------------------------------------------------

View file

@ -0,0 +1,145 @@
"""Tests for import_rejection_reason — the manual-import quarantine guard.
Regression for #764: the manual-import routes call
``post_process_matched_download`` directly. A quarantine (integrity / AcoustID
/ bit-depth) or race-guard rejection returns NORMALLY (no exception) and leaves
the file in ss_quarantine, not the library but the routes counted any
no-exception return as a successful import, so the UI showed a green "Done" for
a file that had actually vanished. ``import_rejection_reason`` reads the context
flags the inner pipeline sets so the routes can report those as errors.
"""
from __future__ import annotations
import os
import tempfile
from core.imports.pipeline import import_rejection_reason
from core.imports.routes import ImportRouteRuntime, process_single_import_file
def test_clean_import_returns_none():
assert import_rejection_reason({}) is None
assert import_rejection_reason({'is_album': True, 'track_info': {}}) is None
def test_integrity_failure_detected():
reason = import_rejection_reason({'_integrity_failure_msg': 'duration drift 12s'})
assert reason is not None
assert 'integrity' in reason.lower()
assert 'duration drift 12s' in reason
def test_acoustid_quarantine_detected():
reason = import_rejection_reason({
'_acoustid_quarantined': True,
'_acoustid_failure_msg': 'wrong artist: got Oasis expected Coldplay',
})
assert reason is not None
assert 'acoustid' in reason.lower()
assert 'Coldplay' in reason
def test_acoustid_quarantine_without_message_still_flags():
# The flag alone must trip it even if no message was stashed.
reason = import_rejection_reason({'_acoustid_quarantined': True})
assert reason is not None
assert 'acoustid' in reason.lower()
def test_bitdepth_rejection_detected():
reason = import_rejection_reason({'_bitdepth_rejected': True})
assert reason is not None
assert 'bit-depth' in reason.lower()
def test_race_guard_failure_detected():
reason = import_rejection_reason({'_race_guard_failed': True})
assert reason is not None
assert 'disappeared' in reason.lower()
def test_falsy_flags_do_not_trip():
# A flag present but falsy (e.g. integrity passed) must NOT be a rejection.
ctx = {
'_integrity_failure_msg': '',
'_acoustid_quarantined': False,
'_bitdepth_rejected': False,
'_race_guard_failed': False,
}
assert import_rejection_reason(ctx) is None
def test_integrity_takes_precedence_when_multiple_set():
# Deterministic ordering: integrity first.
reason = import_rejection_reason({
'_integrity_failure_msg': 'truncated',
'_acoustid_quarantined': True,
'_bitdepth_rejected': True,
})
assert 'integrity' in reason.lower()
# ── route-level wiring: a quarantine must NOT report as a successful import ──
def _runtime_with_post_process(post_process):
"""Build an ImportRouteRuntime wired with stub resolvers + the supplied
post_process_matched_download. Resolvers return a shared context dict so a
flag the post-processor sets is what import_rejection_reason later reads."""
ctx = {}
return ImportRouteRuntime(
get_single_track_import_context=lambda *a, **k: {"context": ctx, "source": "local"},
normalize_import_context=lambda c: c,
get_import_context_artist=lambda c: {"name": "Coldplay"},
get_import_track_info=lambda c: {"name": "Yellow"},
post_process_matched_download=post_process,
), ctx
def _tmp_audio_file():
fd, path = tempfile.mkstemp(suffix=".flac")
os.close(fd)
with open(path, "wb") as f:
f.write(b"fLaC") # only needs to exist for os.path.isfile
return path
def test_single_import_quarantine_reported_as_error():
# post-processing quarantines the file (sets the flag, returns normally).
def quarantining_post_process(context_key, context, file_path):
context['_acoustid_quarantined'] = True
context['_acoustid_failure_msg'] = 'wrong track'
runtime, _ctx = _runtime_with_post_process(quarantining_post_process)
path = _tmp_audio_file()
try:
outcome, payload = process_single_import_file(
runtime, {"full_path": path, "filename": "Coldplay - Yellow.flac",
"title": "Yellow", "artist": "Coldplay"},
)
finally:
os.remove(path)
assert outcome == "error" # NOT "ok" -> route won't count it processed
assert "AcoustID" in payload
def test_single_import_clean_reports_ok():
# post-processing succeeds (no flags) -> import counts as processed.
def clean_post_process(context_key, context, file_path):
return None
runtime, _ctx = _runtime_with_post_process(clean_post_process)
path = _tmp_audio_file()
try:
outcome, payload = process_single_import_file(
runtime, {"full_path": path, "filename": "Coldplay - Yellow.flac",
"title": "Yellow", "artist": "Coldplay"},
)
finally:
os.remove(path)
assert outcome == "ok"
assert payload == "Yellow"

147
tests/test_art_apply.py Normal file
View file

@ -0,0 +1,147 @@
"""Tests for core.metadata.art_apply — on-disk art detection + applying art."""
import sys
import types
from types import SimpleNamespace
# Stubs so core.metadata.artwork (pulled in transitively) imports without the
# real Spotify / config dependencies (mirrors test_missing_cover_art.py).
if 'spotipy' not in sys.modules:
spotipy = types.ModuleType('spotipy')
oauth2 = types.ModuleType('spotipy.oauth2')
spotipy.Spotify = type('S', (), {})
oauth2.SpotifyOAuth = oauth2.SpotifyClientCredentials = type('O', (), {})
spotipy.oauth2 = oauth2
sys.modules['spotipy'] = spotipy
sys.modules['spotipy.oauth2'] = oauth2
if 'config.settings' not in sys.modules:
config_mod = types.ModuleType('config')
settings_mod = types.ModuleType('config.settings')
class _Cfg:
def get(self, key, default=None):
return default
def get_active_media_server(self):
return 'plex'
settings_mod.config_manager = _Cfg()
config_mod.settings = settings_mod
sys.modules['config'] = config_mod
sys.modules['config.settings'] = settings_mod
from core.metadata import art_apply as aa
# ── sidecar detection ──
def test_folder_has_cover_sidecar(tmp_path):
assert aa.folder_has_cover_sidecar(str(tmp_path)) is False
(tmp_path / 'cover.jpg').write_bytes(b'x')
assert aa.folder_has_cover_sidecar(str(tmp_path)) is True
def test_album_has_art_on_disk_no_local_file_is_true():
# No representative file (e.g. media-server-only album) → not flagged.
assert aa.album_has_art_on_disk('') is True
assert aa.album_has_art_on_disk(None) is True
def test_album_has_art_on_disk_sidecar_short_circuits(tmp_path, monkeypatch):
(tmp_path / 'cover.jpg').write_bytes(b'x')
track = tmp_path / '01 song.flac'
track.write_bytes(b'')
# Sidecar present → True without ever opening the audio file.
called = {'n': 0}
monkeypatch.setattr(aa, 'file_has_embedded_art', lambda p: called.__setitem__('n', called['n'] + 1) or False)
assert aa.album_has_art_on_disk(str(track)) is True
assert called['n'] == 0
def test_album_has_art_on_disk_no_sidecar_checks_file(tmp_path, monkeypatch):
track = tmp_path / '01 song.flac'
track.write_bytes(b'')
monkeypatch.setattr(aa, 'file_has_embedded_art', lambda p: False)
assert aa.album_has_art_on_disk(str(track)) is False
monkeypatch.setattr(aa, 'file_has_embedded_art', lambda p: True)
assert aa.album_has_art_on_disk(str(track)) is True
# ── embedded-art detection ──
def _fake_symbols(audio):
return SimpleNamespace(
File=lambda path: audio,
ID3=type('ID3', (), {}),
MP4=type('MP4', (), {}),
FLAC=type('FLAC', (), {}),
)
def test_file_has_embedded_art_flac_picture(tmp_path, monkeypatch):
f = tmp_path / 'a.flac'
f.write_bytes(b'')
audio = SimpleNamespace(pictures=['pic'], tags=None)
monkeypatch.setattr(aa, 'get_mutagen_symbols', lambda: _fake_symbols(audio))
assert aa.file_has_embedded_art(str(f)) is True
def test_file_has_embedded_art_none(tmp_path, monkeypatch):
f = tmp_path / 'a.flac'
f.write_bytes(b'')
audio = SimpleNamespace(pictures=[], tags=None)
monkeypatch.setattr(aa, 'get_mutagen_symbols', lambda: _fake_symbols(audio))
assert aa.file_has_embedded_art(str(f)) is False
# ── applying art ──
def test_apply_embeds_into_each_file_and_writes_cover(tmp_path, monkeypatch):
f1 = tmp_path / '01.flac'; f1.write_bytes(b'')
f2 = tmp_path / '02.flac'; f2.write_bytes(b'')
saved = []
audio = SimpleNamespace(tags=None, add_tags=lambda: None, save=lambda: saved.append(True))
monkeypatch.setattr(aa, 'get_mutagen_symbols', lambda: _fake_symbols(audio))
embed_calls = []
monkeypatch.setattr(aa, 'embed_album_art_metadata', lambda a, m: embed_calls.append(m) or True)
# download_cover_art is the standard cover.jpg writer — stub it to drop one.
monkeypatch.setattr(aa, 'download_cover_art',
lambda album_info, folder, ctx=None: open(f"{folder}/cover.jpg", 'wb').close())
meta = {'artist': 'A', 'album': 'B', 'album_art_url': 'http://x/y.jpg'}
res = aa.apply_art_to_album_files([str(f1), str(f2)], meta, {'album_name': 'B'}, folder=str(tmp_path))
assert res['embedded'] == 2
assert len(saved) == 2 # each file saved after embed
assert res['cover_written'] is True
def test_apply_skips_files_that_already_have_art(tmp_path, monkeypatch):
"""Purely additive: a file that already has art is left untouched (no
duplicate FLAC picture, no re-embed)."""
f1 = tmp_path / '01.flac'; f1.write_bytes(b'')
audio = SimpleNamespace(pictures=['existing'], tags=None, save=lambda: None)
monkeypatch.setattr(aa, 'get_mutagen_symbols', lambda: _fake_symbols(audio))
embed_calls = []
monkeypatch.setattr(aa, 'embed_album_art_metadata', lambda a, m: embed_calls.append(m) or True)
monkeypatch.setattr(aa, 'download_cover_art', lambda *a, **k: None)
res = aa.apply_art_to_album_files([str(f1)], {'album': 'B'}, {'album_name': 'B'}, folder=str(tmp_path))
assert res['embedded'] == 0
assert res['skipped'] == 1
assert embed_calls == [] # never re-embedded → no duplicate picture
def test_apply_counts_failures_without_raising(tmp_path, monkeypatch):
f1 = tmp_path / '01.flac'; f1.write_bytes(b'')
audio = SimpleNamespace(tags=object(), save=lambda: (_ for _ in ()).throw(OSError('read-only')))
monkeypatch.setattr(aa, 'get_mutagen_symbols', lambda: _fake_symbols(audio))
monkeypatch.setattr(aa, 'embed_album_art_metadata', lambda a, m: True)
monkeypatch.setattr(aa, 'download_cover_art', lambda *a, **k: None)
res = aa.apply_art_to_album_files([str(f1)], {'album': 'B'}, {'album_name': 'B'}, folder=str(tmp_path))
assert res['embedded'] == 0
assert res['failed'] == 1 # save() raised (read-only) — counted, not crashed

View file

@ -0,0 +1,179 @@
"""Tests for embedded-cover-art preservation across the enrichment rewrite.
Regression for #764 (continuation of #755): importing a file destroyed its
embedded album art whenever the re-embed step couldn't produce new art
(no source metadata, no art URL, download failed, rejected by the min-size
guard, or art embedding disabled). ``enhance_file_metadata`` clears pictures
up front and saves regardless; these helpers snapshot the art first and put
it back iff the file would otherwise be saved with none.
Uses real mutagen objects (a minimal valid FLAC + a minimal MP3) so the
snapshot/restore round-trips through the actual Picture/APIC APIs the
enricher uses not a mock of them.
"""
from __future__ import annotations
import os
import tempfile
import pytest
mutagen = pytest.importorskip("mutagen")
from mutagen.flac import FLAC, Picture # noqa: E402
from mutagen.id3 import APIC, ID3, TIT2 # noqa: E402
from core.metadata.common import get_mutagen_symbols # noqa: E402
from core.metadata.art_preservation import ( # noqa: E402
has_embedded_art,
restore_embedded_art,
snapshot_embedded_art,
)
SYMBOLS = get_mutagen_symbols()
# 1x1 PNG — smallest valid image bytes for a real Picture/APIC payload.
_PNG = (
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00"
b"\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"
)
def _minimal_flac_bytes() -> bytes:
# 4-byte magic + last STREAMINFO block (34 bytes). Mirrors the fixture
# used by tests/test_tag_writer_multi_artist.py.
return (
b"fLaC"
+ b"\x80\x00\x00\x22"
+ b"\x00\x10\x00\x10"
+ b"\x00\x00\x00\x00\x00\x00"
+ b"\x0a\xc4\x42\xf0\x00\x00\x00\x00"
+ b"\x00" * 16
)
@pytest.fixture
def flac_with_art():
fd, path = tempfile.mkstemp(suffix=".flac")
os.close(fd)
with open(path, "wb") as f:
f.write(_minimal_flac_bytes())
audio = FLAC(path)
pic = Picture()
pic.data = _PNG
pic.type = 3
pic.mime = "image/png"
pic.width = 1
pic.height = 1
pic.depth = 24
audio.add_picture(pic)
audio.save()
yield path
try:
os.remove(path)
except OSError:
pass
class _ID3Holder:
"""Stand-in for an mutagen MP3 object: exposes a real ``ID3`` tag block
(the only thing the snapshot/restore helpers touch) without needing a
syncable MPEG frame on disk. ``__setitem__`` mirrors mutagen's mapping
sugar used by the MP4 restore branch unused here but kept faithful."""
def __init__(self):
self.tags = ID3()
self.tags.add(TIT2(encoding=3, text=["original"]))
self.tags.add(APIC(encoding=3, mime="image/png", type=3, desc="Cover", data=_PNG))
def __setitem__(self, key, value):
self.tags[key] = value
@pytest.fixture
def mp3_with_art():
return _ID3Holder()
# ── FLAC ────────────────────────────────────────────────────────────────
def test_flac_art_restored_after_clear(flac_with_art):
audio = FLAC(flac_with_art)
assert has_embedded_art(audio, SYMBOLS)
snap = snapshot_embedded_art(audio, SYMBOLS)
assert snap # something captured
# Simulate the enrichment rewrite: nuke the art, fail to re-embed.
audio.clear_pictures()
assert not has_embedded_art(audio, SYMBOLS)
restored = restore_embedded_art(audio, SYMBOLS, snap)
assert restored is True
assert has_embedded_art(audio, SYMBOLS)
assert audio.pictures[0].data == _PNG
def test_flac_restore_is_noop_when_new_art_present(flac_with_art):
# Happy path: re-embed succeeded, so restore must NOT touch the file.
audio = FLAC(flac_with_art)
snap = snapshot_embedded_art(audio, SYMBOLS)
audio.clear_pictures()
new = Picture()
new.data = _PNG + b"NEWART"
new.type = 3
new.mime = "image/png"
audio.add_picture(new)
restored = restore_embedded_art(audio, SYMBOLS, snap)
assert restored is False
assert len(audio.pictures) == 1
assert audio.pictures[0].data == _PNG + b"NEWART" # not clobbered/duplicated
def test_flac_no_art_snapshot_empty():
fd, path = tempfile.mkstemp(suffix=".flac")
os.close(fd)
try:
with open(path, "wb") as f:
f.write(_minimal_flac_bytes())
audio = FLAC(path)
assert snapshot_embedded_art(audio, SYMBOLS) == []
# Restoring an empty snapshot is a no-op.
assert restore_embedded_art(audio, SYMBOLS, []) is False
finally:
os.remove(path)
# ── MP3 / ID3 ─────────────────────────────────────────────────────────────
def test_mp3_apic_restored_after_tags_cleared(mp3_with_art):
audio = mp3_with_art
assert has_embedded_art(audio, SYMBOLS)
snap = snapshot_embedded_art(audio, SYMBOLS)
assert snap
# The enricher does audio_file.tags.clear() then rewrites tags.
audio.tags.clear()
assert not has_embedded_art(audio, SYMBOLS)
restored = restore_embedded_art(audio, SYMBOLS, snap)
assert restored is True
apics = audio.tags.getall("APIC")
assert apics and apics[0].data == _PNG
def test_mp3_restore_noop_when_new_apic_present(mp3_with_art):
audio = mp3_with_art
snap = snapshot_embedded_art(audio, SYMBOLS)
audio.tags.clear()
audio.tags.add(APIC(encoding=3, mime="image/png", type=3, desc="Cover", data=_PNG + b"NEW"))
assert restore_embedded_art(audio, SYMBOLS, snap) is False
apics = audio.tags.getall("APIC")
assert len(apics) == 1 and apics[0].data == _PNG + b"NEW"

View file

@ -0,0 +1,69 @@
"""Migration test for the canonical-album-version columns (#765 Stage 1).
Additive + nullable, so it must: appear on a fresh DB, be idempotent (re-running
the migration is a no-op, not an error), and ALTER them onto an older albums
table that lacks them. NULL = unresolved = tools fall back to today's behavior.
"""
from __future__ import annotations
import sqlite3
from database.music_database import MusicDatabase
_CANONICAL_COLS = {
'canonical_source', 'canonical_album_id', 'canonical_score', 'canonical_resolved_at',
}
def _album_cols(cur):
cur.execute("PRAGMA table_info(albums)")
return {c[1] for c in cur.fetchall()}
def test_fresh_db_has_canonical_columns(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
cur = db._get_connection().cursor()
assert _CANONICAL_COLS <= _album_cols(cur)
def test_canonical_columns_default_null(tmp_path):
# Unresolved by default -> every consumer falls back. Verify each canonical
# column declares DEFAULT NULL and is nullable (notnull flag == 0).
db = MusicDatabase(str(tmp_path / "m.db"))
cur = db._get_connection().cursor()
cur.execute("PRAGMA table_info(albums)")
info = {c[1]: c for c in cur.fetchall()} # name -> (cid, name, type, notnull, dflt, pk)
for col in _CANONICAL_COLS:
assert col in info, f"{col} missing"
assert info[col][3] == 0, f"{col} must be nullable"
dflt = info[col][4]
assert dflt is None or str(dflt).upper() == 'NULL', f"{col} default should be NULL"
def test_migration_is_idempotent(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
cur = db._get_connection().cursor()
before = _album_cols(cur)
# Re-running must not raise (the PRAGMA guard skips existing columns).
db._ensure_core_media_schema_columns(cur)
db._ensure_core_media_schema_columns(cur)
assert _album_cols(cur) == before
assert _CANONICAL_COLS <= _album_cols(cur)
def test_migration_adds_columns_to_old_albums_table(tmp_path):
# Simulate an upgraded DB whose albums table predates these columns.
path = str(tmp_path / "old.db")
conn = sqlite3.connect(path)
conn.execute("CREATE TABLE albums (id INTEGER PRIMARY KEY, title TEXT)")
conn.commit()
cur = conn.cursor()
assert not (_CANONICAL_COLS & _album_cols(cur)) # none present yet
# Run the real migration against this old cursor.
db = MusicDatabase(str(tmp_path / "scratch.db"))
db._ensure_core_media_schema_columns(cur)
conn.commit()
assert _CANONICAL_COLS <= _album_cols(cur)

View file

@ -0,0 +1,54 @@
"""DB persistence for canonical album version (#765 Stage 2)."""
from __future__ import annotations
from database.music_database import MusicDatabase
def _album(db, album_id="alb_evolve"):
# id columns are TEXT (GUID) post-migration, so insert explicit ids and a
# valid FK rather than relying on integer rowids.
conn = db._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name) VALUES ('art_id', 'Imagine Dragons')")
cur.execute(
"INSERT INTO albums (id, title, artist_id) VALUES (?, 'Evolve', 'art_id')",
(album_id,),
)
conn.commit()
conn.close()
return album_id
def test_set_then_get_roundtrip(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _album(db)
assert db.get_album_canonical(album_id) is None # unresolved by default
assert db.set_album_canonical(album_id, "spotify", "sp_evolve_123", 0.97) is True
got = db.get_album_canonical(album_id)
assert got["source"] == "spotify"
assert got["album_id"] == "sp_evolve_123"
assert abs(got["score"] - 0.97) < 1e-6
assert got["resolved_at"] # timestamp populated
def test_get_unresolved_returns_none(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _album(db)
assert db.get_album_canonical(album_id) is None
def test_set_overwrites_previous(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _album(db)
db.set_album_canonical(album_id, "spotify", "old", 0.6)
db.set_album_canonical(album_id, "musicbrainz", "new", 0.95)
got = db.get_album_canonical(album_id)
assert got["source"] == "musicbrainz" and got["album_id"] == "new"
def test_set_on_missing_album_returns_false(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
assert db.set_album_canonical(999999, "spotify", "x", 0.9) is False

View file

@ -0,0 +1,143 @@
"""End-to-end orchestration for canonical resolve+store (#765 Stage 2 trigger).
Uses a real temp DB (album + tracks + source IDs) and an INJECTED fetcher, so
the DB gathering + persistence are exercised for real without live APIs.
"""
from __future__ import annotations
from core.metadata.canonical_resolver import (
default_fetch_tracklist,
resolve_and_store_canonical_for_album,
)
from database.music_database import MusicDatabase
STD = [{"duration_ms": 180_000 + i * 10_000, "title": f"Song {i+1}", "track_number": i + 1} for i in range(11)]
DLX = STD + [{"duration_ms": 320_000 + i * 10_000, "title": f"Bonus {i+1}", "track_number": 12 + i} for i in range(6)]
def _seed(db, *, spotify=None, deezer=None, n_files=11):
"""Insert an album (with given source IDs) + n_files tracks whose
durations/titles match the STANDARD release."""
conn = db._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name) VALUES ('art1', 'Imagine Dragons')")
cur.execute(
"INSERT INTO albums (id, title, artist_id, spotify_album_id, deezer_id) "
"VALUES ('alb1', 'Evolve', 'art1', ?, ?)",
(spotify, deezer),
)
for i in range(n_files):
cur.execute(
"INSERT INTO tracks (id, album_id, artist_id, title, track_number, duration) "
"VALUES (?, 'alb1', 'art1', ?, ?, ?)",
(f"t{i}", f"Song {i+1}", i + 1, 180_000 + i * 10_000),
)
conn.commit()
conn.close()
return "alb1"
def test_resolve_and_store_picks_best_fit_and_persists(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _seed(db, spotify="sp_deluxe", deezer="dz_std") # 11 files
table = {("spotify", "sp_deluxe"): DLX, ("deezer", "dz_std"): STD}
out = resolve_and_store_canonical_for_album(
db, album_id,
fetch_tracklist=lambda s, a: table.get((s, a)),
source_priority=["spotify", "deezer"],
mode="best_fit",
)
# best_fit: Deezer's standard matches the 11 files better than Spotify's deluxe.
assert out["source"] == "deezer" and out["album_id"] == "dz_std"
# ...and it was persisted.
stored = db.get_album_canonical(album_id)
assert stored["source"] == "deezer" and stored["album_id"] == "dz_std"
def test_default_mode_prefers_active_source(tmp_path):
# Same setup, but default (active_preferred) mode: primary = spotify, whose
# deluxe still clears the floor -> pinned, even though deezer fits better.
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _seed(db, spotify="sp_deluxe", deezer="dz_std")
table = {("spotify", "sp_deluxe"): DLX, ("deezer", "dz_std"): STD}
out = resolve_and_store_canonical_for_album(
db, album_id,
fetch_tracklist=lambda s, a: table.get((s, a)),
source_priority=["spotify", "deezer"], # default mode
)
assert out["source"] == "spotify" # active source preferred
def test_result_includes_artist_and_album_context(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
conn = db._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name, thumb_url) VALUES ('art1', 'Imagine Dragons', 'http://artist.jpg')")
cur.execute(
"INSERT INTO albums (id, title, artist_id, thumb_url, spotify_album_id) "
"VALUES ('alb1', 'Evolve', 'art1', 'http://album.jpg', 'sp1')"
)
for i in range(11):
cur.execute(
"INSERT INTO tracks (id, album_id, artist_id, title, track_number, duration) "
"VALUES (?, 'alb1', 'art1', ?, ?, ?)",
(f"t{i}", f"Song {i+1}", i + 1, 180_000 + i * 10_000),
)
conn.commit()
conn.close()
out = resolve_and_store_canonical_for_album(
db, "alb1", fetch_tracklist=lambda s, a: STD, source_priority=["spotify"],
)
assert out["artist_name"] == "Imagine Dragons"
assert out["album_thumb_url"] == "http://album.jpg"
assert out["artist_thumb_url"] == "http://artist.jpg"
# free context: db track count, linked sources, and both title lists
assert out["db_track_count"] == 11
assert out["linked_sources"] == {"spotify": "sp1"}
assert out["file_track_titles"][0] == "Song 1" and len(out["file_track_titles"]) == 11
assert "Song 1" in out["release_track_titles"]
def test_resolve_returns_none_when_album_has_no_source_ids(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
album_id = _seed(db, spotify=None, deezer=None)
out = resolve_and_store_canonical_for_album(
db, album_id, fetch_tracklist=lambda s, a: STD, source_priority=["spotify"],
)
assert out is None
assert db.get_album_canonical(album_id) is None
def test_resolve_returns_none_for_missing_album(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
out = resolve_and_store_canonical_for_album(
db, "does-not-exist", fetch_tracklist=lambda s, a: STD, source_priority=["spotify"],
)
assert out is None
# ── default_fetch_tracklist normalization (no DB / no live API) ────────────
def test_default_fetcher_normalizes_dict_items(monkeypatch):
import core.metadata_service as ms
monkeypatch.setattr(
ms, "get_album_tracks_for_source",
lambda s, a: [{"name": "A", "track_number": 1, "duration_ms": 200000},
{"title": "B", "track_number": 2, "duration": 210}], # seconds
raising=False,
)
out = default_fetch_tracklist("spotify", "x")
assert out[0] == {"title": "A", "track_number": 1, "duration_ms": 200000}
assert out[1] == {"title": "B", "track_number": 2, "duration_ms": 210_000} # sec->ms
def test_default_fetcher_handles_failure(monkeypatch):
import core.metadata_service as ms
monkeypatch.setattr(
ms, "get_album_tracks_for_source",
lambda s, a: (_ for _ in ()).throw(RuntimeError("boom")), raising=False,
)
assert default_fetch_tracklist("spotify", "x") is None

View file

@ -0,0 +1,195 @@
"""Tests for resolve_canonical_for_album (#765 Stage 2 — injectable core)."""
from __future__ import annotations
from core.metadata.canonical_resolver import resolve_canonical_for_album
STD = [{"duration_ms": 180_000 + i * 10_000, "title": f"Song {i+1}"} for i in range(11)]
DLX = STD + [{"duration_ms": 300_000 + i * 10_000, "title": f"Bonus {i+1}"} for i in range(6)]
PRIORITY = ["spotify", "itunes", "deezer", "musicbrainz"]
def _fetcher(table):
"""fetch_tracklist backed by a {(source, album_id): tracks} table."""
def fetch(source, album_id):
return table.get((source, album_id))
return fetch
def test_best_fit_mode_picks_best_regardless_of_priority():
files = list(STD) # user owns the 11-track standard
table = {
("spotify", "sp_deluxe"): DLX, # spotify (primary) linked to deluxe (17)
("musicbrainz", "mb_std"): STD, # musicbrainz has standard (11)
}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp_deluxe", "musicbrainz": "mb_std"},
file_tracks=files,
fetch_tracklist=_fetcher(table),
source_priority=PRIORITY,
mode="best_fit",
)
# best_fit: standard matches the files, deluxe doesn't — fit beats priority.
assert out["source"] == "musicbrainz" and out["album_id"] == "mb_std"
assert out["score"] > 0.9
# ── source-selection modes ────────────────────────────────────────────────
def test_active_preferred_uses_primary_when_it_fits():
files = list(STD)
table = {("spotify", "sp1"): STD, ("musicbrainz", "mb1"): STD} # both fit
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp1", "musicbrainz": "mb1"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, # primary = spotify
) # default mode = active_preferred
assert out["source"] == "spotify"
def test_active_preferred_falls_back_when_primary_clearly_misfits():
files = list(STD) # 11 tracks
table = {
("spotify", "sp_bad"): [{"duration_ms": 60_000, "title": "X"}] * 3, # 3-track, <floor
("musicbrainz", "mb_std"): STD,
}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp_bad", "musicbrainz": "mb_std"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, mode="active_preferred",
)
# primary spotify scores below floor -> fall back to the fitting source.
assert out["source"] == "musicbrainz"
def test_active_preferred_keeps_primary_even_if_another_fits_better():
files = list(STD)
# primary spotify is a deluxe (decent fit, above floor); musicbrainz is exact.
table = {("spotify", "sp_dlx"): DLX, ("musicbrainz", "mb_std"): STD}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp_dlx", "musicbrainz": "mb_std"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, mode="active_preferred",
)
# active_preferred respects the active source as long as it clears the floor,
# even though musicbrainz would fit better (use best_fit for that).
assert out["source"] == "spotify"
def test_active_only_pins_primary_and_never_falls_back():
files = list(STD)
# primary spotify is below floor; a perfect musicbrainz exists but is ignored.
table = {
("spotify", "sp_bad"): [{"duration_ms": 60_000, "title": "X"}] * 3,
("musicbrainz", "mb_std"): STD,
}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp_bad", "musicbrainz": "mb_std"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, mode="active_only",
)
assert out is None # primary didn't fit, and active_only won't consider others
def test_result_includes_breakdown_and_candidate_comparison():
files = list(STD)
table = {("spotify", "sp1"): DLX, ("deezer", "dz1"): STD}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp1", "deezer": "dz1"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=["spotify", "deezer", "itunes", "musicbrainz"],
mode="best_fit",
)
assert out["source"] == "deezer"
assert out["file_track_count"] == 11
assert out["release_track_count"] == 11
assert out["count_fit"] == 1.0 and out["duration_fit"] == 1.0 and out["title_fit"] == 1.0
by_src = {c["source"]: c for c in out["candidates"]}
assert by_src["deezer"]["track_count"] == 11 and by_src["deezer"]["score"] > 0.9
assert by_src["spotify"]["track_count"] == 17 and by_src["spotify"]["score"] < 0.8
def test_active_only_pins_primary_when_it_fits():
files = list(STD)
table = {("spotify", "sp1"): STD, ("musicbrainz", "mb1"): STD}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "sp1", "musicbrainz": "mb1"},
file_tracks=files, fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, mode="active_only",
)
assert out["source"] == "spotify"
def test_priority_breaks_tie_between_equal_fits():
files = list(STD)
table = {("spotify", "a"): STD, ("itunes", "b"): STD} # identical fit
out = resolve_canonical_for_album(
album_source_ids={"itunes": "b", "spotify": "a"},
file_tracks=files,
fetch_tracklist=_fetcher(table),
source_priority=PRIORITY, # spotify before itunes
)
assert out["source"] == "spotify"
def test_skips_sources_without_ids_or_failed_fetch():
files = list(STD)
def fetch(source, album_id):
if source == "spotify":
raise RuntimeError("API down")
if source == "deezer":
return STD
return None
out = resolve_canonical_for_album(
album_source_ids={"spotify": "x", "deezer": "y"}, # no itunes id
file_tracks=files,
fetch_tracklist=fetch,
source_priority=PRIORITY,
)
assert out["source"] == "deezer"
def test_none_when_no_candidates():
out = resolve_canonical_for_album(
album_source_ids={},
file_tracks=list(STD),
fetch_tracklist=_fetcher({}),
source_priority=PRIORITY,
)
assert out is None
def test_none_when_no_files():
out = resolve_canonical_for_album(
album_source_ids={"spotify": "a"},
file_tracks=[],
fetch_tracklist=_fetcher({("spotify", "a"): STD}),
source_priority=PRIORITY,
)
assert out is None
def test_none_when_below_floor():
files = list(STD) # 11 tracks
# Only candidate is a wildly-wrong 3-track release.
table = {("spotify", "a"): [{"duration_ms": 60_000, "title": "X"}] * 3}
out = resolve_canonical_for_album(
album_source_ids={"spotify": "a"},
file_tracks=files,
fetch_tracklist=_fetcher(table),
source_priority=PRIORITY,
)
assert out is None
def test_score_is_rounded():
out = resolve_canonical_for_album(
album_source_ids={"spotify": "a"},
file_tracks=list(STD),
fetch_tracklist=_fetcher({("spotify", "a"): STD}),
source_priority=PRIORITY,
)
assert out["score"] == round(out["score"], 4)

View file

@ -0,0 +1,143 @@
"""Extreme battery for canonical-album-version scoring (#765 / #767-Bug2).
The scorer must: pick the right EDITION by best-fit to the files on disk
(standard when you have the standard, deluxe when you have the deluxe), break
ties deterministically toward the higher-priority candidate (so every tool
agrees), degrade gracefully when durations/titles are missing, and never pin a
low-confidence guess.
"""
from __future__ import annotations
from core.metadata.canonical_version import (
pick_canonical_release,
score_release_against_files,
)
# Helpers — build track lists ----------------------------------------------
def _tracks(n, base_ms=180_000, step_ms=10_000, titles=None):
"""n tracks with distinct, deterministic durations + optional titles."""
out = []
for i in range(n):
t = {"duration_ms": base_ms + i * step_ms, "track_number": i + 1}
if titles:
t["title"] = titles[i]
out.append(t)
return out
STANDARD_TITLES = [f"Song {i+1}" for i in range(11)]
DELUXE_TITLES = STANDARD_TITLES + [f"Bonus {i+1}" for i in range(6)]
# ── edition discrimination ────────────────────────────────────────────────
def test_eleven_files_prefer_standard_over_deluxe():
files = _tracks(11, titles=STANDARD_TITLES)
standard = _tracks(11, titles=STANDARD_TITLES)
deluxe = _tracks(17, titles=DELUXE_TITLES)
s_std = score_release_against_files(files, standard)
s_dlx = score_release_against_files(files, deluxe)
assert s_std > s_dlx
best, score = pick_canonical_release(
files,
[{"source": "standard", "tracks": standard}, {"source": "deluxe", "tracks": deluxe}],
)
assert best["source"] == "standard" and score > 0.9
def test_seventeen_files_prefer_deluxe():
files = _tracks(17, titles=DELUXE_TITLES)
standard = _tracks(11, titles=STANDARD_TITLES)
deluxe = _tracks(17, titles=DELUXE_TITLES)
best, _ = pick_canonical_release(
files,
# deluxe deliberately listed SECOND to prove count/fit beats order here
[{"source": "standard", "tracks": standard}, {"source": "deluxe", "tracks": deluxe}],
)
assert best["source"] == "deluxe"
def test_exact_count_and_durations_scores_near_one():
files = _tracks(11, titles=STANDARD_TITLES)
assert score_release_against_files(files, _tracks(11, titles=STANDARD_TITLES)) > 0.99
# ── deterministic tiebreak (the #765 resolution) ──────────────────────────
def test_identical_releases_break_tie_to_first_candidate():
# Same album from two sources (same files match both equally) — must pick
# the FIRST (higher-priority) deterministically so both tools agree.
files = _tracks(11, titles=STANDARD_TITLES)
a = {"source": "spotify", "tracks": _tracks(11, titles=STANDARD_TITLES)}
b = {"source": "musicbrainz", "tracks": _tracks(11, titles=STANDARD_TITLES)}
best, _ = pick_canonical_release(files, [a, b])
assert best["source"] == "spotify"
# ...and stable when the order flips (priority is the caller's order).
best2, _ = pick_canonical_release(files, [b, a])
assert best2["source"] == "musicbrainz"
# ── duration disambiguation when counts tie ───────────────────────────────
def test_duration_breaks_tie_when_counts_equal():
# Two 11-track candidates; the files' durations match candidate A's lengths,
# not B's (e.g. album cuts vs radio edits). A must win on duration fit.
files = _tracks(11, base_ms=200_000, step_ms=5_000)
cand_a = {"source": "album", "tracks": _tracks(11, base_ms=200_000, step_ms=5_000)}
cand_b = {"source": "edits", "tracks": _tracks(11, base_ms=140_000, step_ms=5_000)}
best, _ = pick_canonical_release(files, [cand_b, cand_a]) # B listed first
assert best["source"] == "album" # duration fit overrides order
# ── graceful degradation ──────────────────────────────────────────────────
def test_no_durations_falls_back_to_count_and_title():
files = [{"title": t} for t in STANDARD_TITLES] # no durations
standard = [{"title": t} for t in STANDARD_TITLES]
deluxe = [{"title": t} for t in DELUXE_TITLES]
best, score = pick_canonical_release(
files,
[{"source": "standard", "tracks": standard}, {"source": "deluxe", "tracks": deluxe}],
)
assert best["source"] == "standard" and score > 0.5
def test_only_counts_available_still_scores():
files = [{} for _ in range(11)]
assert score_release_against_files(files, [{} for _ in range(11)]) > 0.99
assert score_release_against_files(files, [{} for _ in range(17)]) < 0.8
def test_fuzzy_titles_still_match():
files = _tracks(3, titles=["Believer", "Whatever It Takes", "Thunder"])
rel = _tracks(3, titles=["Believer (Remastered)", "Whatever It Takes", "Thunder!"])
assert score_release_against_files(files, rel) > 0.9
# ── confidence floor / guards ─────────────────────────────────────────────
def test_below_floor_returns_none():
files = _tracks(11, titles=STANDARD_TITLES)
# A wildly wrong candidate (3 unrelated tracks) must not be pinned.
bad = {"source": "wrong", "tracks": _tracks(3, base_ms=60_000, titles=["X", "Y", "Z"])}
best, score = pick_canonical_release(files, [bad])
assert best is None
assert score < 0.5
def test_empty_inputs_are_safe():
assert score_release_against_files([], _tracks(11)) == 0.0
assert score_release_against_files(_tracks(11), []) == 0.0
best, score = pick_canonical_release(_tracks(11), [])
assert best is None and score == 0.0
def test_min_score_is_tunable():
files = _tracks(11, titles=STANDARD_TITLES)
near = {"source": "near", "tracks": _tracks(10, titles=STANDARD_TITLES[:10])}
# default floor accepts a 10/11 fit, a strict floor rejects it
assert pick_canonical_release(files, [near])[0] is not None
assert pick_canonical_release(files, [near], min_score=0.99)[0] is None

View file

@ -0,0 +1,160 @@
"""Backfill job: Resolve Canonical Album Versions (#765 Stage 2 trigger)."""
from __future__ import annotations
import types
import core.repair_jobs.canonical_version_resolve as cvr
from core.repair_jobs import get_all_jobs
from core.repair_jobs.canonical_version_resolve import (
CanonicalVersionResolveJob,
_describe_pin,
)
from database.music_database import MusicDatabase
def _ctx(db, findings):
return types.SimpleNamespace(
db=db,
config_manager=None, # -> active_server None -> all albums
check_stop=lambda: False,
wait_if_paused=lambda: False,
report_progress=None,
update_progress=None,
create_finding=lambda **kw: (findings.append(kw) or True),
)
def _seed_two_albums(db):
conn = db._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name) VALUES ('art1', 'A')")
cur.execute("INSERT INTO albums (id, title, artist_id) VALUES ('alb1', 'Album One', 'art1')")
cur.execute("INSERT INTO albums (id, title, artist_id) VALUES ('alb2', 'Album Two', 'art1')")
conn.commit()
conn.close()
def _fake_resolver(monkeypatch):
def fake(db, album_id, *, min_score=0.5, store=True, mode="active_preferred"):
res = {"source": "spotify", "album_id": f"sp_{album_id}", "score": 0.9}
if store:
db.set_album_canonical(album_id, res["source"], res["album_id"], res["score"])
return res
monkeypatch.setattr(cvr, "resolve_and_store_canonical_for_album", fake)
def test_job_is_registered():
jobs = get_all_jobs() # {job_id: cls}
assert "canonical_version_resolve" in jobs
assert jobs["canonical_version_resolve"] is CanonicalVersionResolveJob
def test_job_is_opt_in_and_dry_run_by_default():
assert CanonicalVersionResolveJob.default_enabled is False
assert CanonicalVersionResolveJob.default_settings["dry_run"] is True
def test_source_selection_defaults_to_active_preferred():
assert CanonicalVersionResolveJob.default_settings["source_selection"] == "active_preferred"
def test_source_selection_exposes_dropdown_options():
# The UI renders a <select> for keys listed in setting_options.
opts = CanonicalVersionResolveJob.setting_options.get("source_selection")
assert opts == ["active_preferred", "active_only", "best_fit"]
# default must be one of the offered options
assert CanonicalVersionResolveJob.default_settings["source_selection"] in opts
def test_describe_pin_is_judgeable():
desc = _describe_pin({
"source": "deezer", "album_id": "665666731", "score": 1.0,
"file_track_count": 11, "release_track_count": 11,
"count_fit": 1.0, "duration_fit": 1.0, "title_fit": 1.0,
"candidates": [
{"source": "deezer", "album_id": "665666731", "track_count": 11, "score": 1.0},
{"source": "spotify", "album_id": "sp", "track_count": 17, "score": 0.65},
],
})
assert "deezer" in desc and "665666731" in desc
assert "11 files vs 11 tracks" in desc # your library vs the release
assert "durations 100%" in desc and "titles 100%" in desc # the WHY
assert "Beat:" in desc and "spotify 65% (17 tk)" in desc # what it beat
def test_describe_pin_includes_year_linked_and_tracklist():
desc = _describe_pin({
"source": "deezer", "album_id": "dz1", "score": 1.0,
"artist_name": "Lenka", "album_title": "Souls of Serenity", "year": 2017,
"file_track_count": 3, "release_track_count": 3,
"count_fit": 1.0, "duration_fit": 1.0, "title_fit": 1.0,
"linked_sources": {"spotify": "sp1", "deezer": "dz1"},
"release_track_titles": ["The Show", "Trouble Is a Friend", "Everything at Once"],
"candidates": [{"source": "deezer", "album_id": "dz1", "track_count": 3, "score": 1.0}],
})
assert "Lenka — Souls of Serenity (2017)" in desc
assert "Currently linked: spotify=sp1, deezer=dz1 → pinning deezer" in desc
assert "Release tracks: 1. The Show; 2. Trouble Is a Friend; 3. Everything at Once" in desc
def test_describe_pin_single_source():
desc = _describe_pin({
"source": "spotify", "album_id": "x", "score": 0.9,
"file_track_count": 10, "release_track_count": 10,
"count_fit": 1.0, "duration_fit": None, "title_fit": 0.9,
"candidates": [{"source": "spotify", "album_id": "x", "track_count": 10, "score": 0.9}],
})
assert "Only this source" in desc
assert "durations n/a" in desc # missing signal shown honestly
def test_live_resolves_and_stores(tmp_path, monkeypatch):
db = MusicDatabase(str(tmp_path / "m.db"))
_seed_two_albums(db)
_fake_resolver(monkeypatch)
findings = []
ctx = _ctx(db, findings)
job = CanonicalVersionResolveJob()
# force live mode
monkeypatch.setattr(job, "_get_settings", lambda c: {"dry_run": False, "min_score": 0.5})
result = job.scan(ctx)
assert result.auto_fixed == 2
assert db.get_album_canonical("alb1")["source"] == "spotify"
assert db.get_album_canonical("alb2")["album_id"] == "sp_alb2"
assert findings == [] # live mode writes, doesn't create findings
def test_dry_run_creates_findings_without_storing(tmp_path, monkeypatch):
db = MusicDatabase(str(tmp_path / "m.db"))
_seed_two_albums(db)
_fake_resolver(monkeypatch)
findings = []
ctx = _ctx(db, findings)
job = CanonicalVersionResolveJob()
monkeypatch.setattr(job, "_get_settings", lambda c: {"dry_run": True, "min_score": 0.5})
result = job.scan(ctx)
assert result.findings_created == 2
assert len(findings) == 2
# dry run must NOT persist
assert db.get_album_canonical("alb1") is None
def test_skips_already_pinned_albums(tmp_path, monkeypatch):
db = MusicDatabase(str(tmp_path / "m.db"))
_seed_two_albums(db)
db.set_album_canonical("alb1", "deezer", "dz_pinned", 0.8) # alb1 already pinned
_fake_resolver(monkeypatch)
ctx = _ctx(db, [])
job = CanonicalVersionResolveJob()
monkeypatch.setattr(job, "_get_settings", lambda c: {"dry_run": False, "min_score": 0.5})
result = job.scan(ctx)
assert result.skipped == 1 # alb1 skipped
assert result.auto_fixed == 1 # only alb2 resolved
assert db.get_album_canonical("alb1")["album_id"] == "dz_pinned" # untouched

View file

@ -0,0 +1,153 @@
"""End-to-end proof that enhance_file_metadata never destroys embedded art.
Bug #764: the enrichment rewrite clears cover art up front and saves the file
regardless of whether new art gets re-embedded. These tests run the REAL
``enhance_file_metadata`` against a REAL FLAC that has embedded art, and assert
the art is still on disk after every failure path and that the happy path
(new art embedded) correctly REPLACES it. This exercises the actual wiring
(snapshot -> clear -> rewrite -> restore/save), not just the helper functions.
Only the external collaborators are stubbed (config, source-metadata extraction,
the art fetch, source-id embed, verification) the clear/snapshot/restore/save
sequence under test runs for real through mutagen.
"""
from __future__ import annotations
import os
import tempfile
from unittest.mock import patch
import pytest
pytest.importorskip("mutagen")
from mutagen.flac import FLAC, Picture # noqa: E402
import core.metadata.enrichment as enrichment # noqa: E402
_PNG = (
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00"
b"\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"
)
class _Cfg:
"""Config stub: returns the caller's default for every key, so
metadata_enhancement.enabled / embed_album_art resolve True."""
def get(self, key, default=None):
return default
def _make_flac_with_art(path):
minimal = (
b"fLaC"
+ b"\x80\x00\x00\x22"
+ b"\x00\x10\x00\x10"
+ b"\x00\x00\x00\x00\x00\x00"
+ b"\x0a\xc4\x42\xf0\x00\x00\x00\x00"
+ b"\x00" * 16
)
with open(path, "wb") as f:
f.write(minimal)
audio = FLAC(path)
pic = Picture()
pic.data = _PNG
pic.type = 3
pic.mime = "image/png"
pic.width = 1
pic.height = 1
pic.depth = 24
audio.add_picture(pic)
audio.save()
@pytest.fixture
def flac_path():
fd, path = tempfile.mkstemp(suffix=".flac")
os.close(fd)
_make_flac_with_art(path)
yield path
try:
os.remove(path)
except OSError:
pass
def _disk_art(path):
"""Return the embedded picture bytes on disk, or None."""
pics = FLAC(path).pictures
return pics[0].data if pics else None
def _run(flac_path, *, metadata, embed_side_effect):
"""Drive enhance_file_metadata with stubbed collaborators."""
with patch.object(enrichment, "get_config_manager", return_value=_Cfg()), \
patch.object(enrichment, "strip_all_non_audio_tags"), \
patch.object(enrichment, "extract_source_metadata", return_value=metadata), \
patch.object(enrichment, "embed_source_ids"), \
patch.object(enrichment, "verify_metadata_written", return_value=True), \
patch.object(enrichment, "embed_album_art_metadata", side_effect=embed_side_effect):
return enrichment.enhance_file_metadata(
flac_path, context={}, artist={"name": "Coldplay"}, album_info={},
)
# ── failure paths: art MUST survive ──────────────────────────────────────
def test_art_survives_when_source_metadata_missing(flac_path):
# extract_source_metadata returns None -> early return path.
assert _disk_art(flac_path) == _PNG
result = _run(flac_path, metadata=None, embed_side_effect=lambda *a, **k: False)
assert result is True
assert _disk_art(flac_path) == _PNG # art preserved on disk
def test_art_survives_when_embed_produces_nothing(flac_path):
# Metadata is fine, but the art fetch fails -> embed is a no-op (returns
# False, adds no picture). The original art must remain.
def embed_noop(audio_file, metadata):
return False # mirrors "no art URL" / "download failed"
result = _run(flac_path, metadata={"title": "Yellow", "artist": "Coldplay"},
embed_side_effect=embed_noop)
assert result is True
assert _disk_art(flac_path) == _PNG
def test_art_survives_when_embed_raises(flac_path):
# A hard crash mid-enrichment must not leave the file art-less.
def embed_boom(audio_file, metadata):
raise RuntimeError("art backend exploded")
result = _run(flac_path, metadata={"title": "Yellow", "artist": "Coldplay"},
embed_side_effect=embed_boom)
assert result is False # enrichment reported failure
assert _disk_art(flac_path) == _PNG # ...but art was restored on disk
# ── happy path: new art REPLACES old, no duplication ──────────────────────
def test_new_art_replaces_old_when_embed_succeeds(flac_path):
new_bytes = _PNG + b"BRANDNEW"
def embed_real(audio_file, metadata):
# Simulate a successful fetch+embed: add the new picture in-place.
pic = Picture()
pic.data = new_bytes
pic.type = 3
pic.mime = "image/png"
audio_file.add_picture(pic)
return True
result = _run(flac_path, metadata={"title": "Yellow", "artist": "Coldplay"},
embed_side_effect=embed_real)
assert result is True
on_disk = FLAC(flac_path).pictures
# Exactly one picture, and it's the NEW art — restore must not have
# re-added the old one on top.
assert len(on_disk) == 1
assert on_disk[0].data == new_bytes

View file

@ -1163,9 +1163,10 @@ def test_keeps_album_sidecars_when_a_track_failed_to_move(monkeypatch, tmpdirs):
# --- preview function (shared planning with the orchestrator) -----------
def _fake_path_builder(context, spotify_artist, _album_info, file_ext):
def _fake_path_builder(context, spotify_artist, _album_info, file_ext, **_kw):
"""Stand-in for `_build_final_path_for_track`. Inserts Disc N/ when
total_discs > 1 same convention the real builder uses."""
total_discs > 1 same convention the real builder uses. Accepts
**_kw so the preview's create_dirs=False kwarg (#767) passes through."""
album = context['spotify_album']['name']
artist = spotify_artist['name']
track_info = context['track_info']
@ -1180,7 +1181,7 @@ def _fake_path_builder(context, spotify_artist, _album_info, file_ext):
return '/'.join(parts), True
def _path_builder_album_vs_single(context, spotify_artist, album_info, file_ext):
def _path_builder_album_vs_single(context, spotify_artist, album_info, file_ext, **_kw):
"""Stand-in that emulates the real `_build_final_path_for_track`
branch on `album_info.get('is_album')`. ALBUM mode produces an
album folder with disc subfolder + numbered file; SINGLE mode

View file

@ -59,7 +59,10 @@ class _FakeClient:
self.search_calls.append((query, limit))
if self.search_image is None:
return []
return [SimpleNamespace(id='search-album', image_url=self.search_image)]
# Real source clients return album results carrying title + artist;
# the filler now validates those before trusting the artwork.
return [SimpleNamespace(id='search-album', image_url=self.search_image,
title='Album', artist='Artist')]
def _make_db(album_row):
@ -90,6 +93,18 @@ def _make_db(album_row):
)
"""
)
# The scan now joins a representative track path to check art on disk.
cursor.execute(
"""
CREATE TABLE tracks (
id INTEGER PRIMARY KEY,
album_id INTEGER,
file_path TEXT,
disc_number INTEGER,
track_number INTEGER
)
"""
)
cursor.execute(
"INSERT INTO artists (id, name, thumb_url) VALUES (?, ?, ?)",
(1, 'Artist', 'https://artist/thumb'),
@ -165,7 +180,68 @@ def test_missing_cover_art_uses_primary_when_prefer_unset(monkeypatch):
result = mca.MissingCoverArtJob().scan(context)
assert result.findings_created == 1
assert discogs_client.search_calls == [('Artist Album', 1)]
assert discogs_client.search_calls == [('Artist Album', 5)]
assert spotify_client.search_calls == []
assert itunes_client.search_calls == []
assert context.findings[0]['details']['found_artwork_url'] == 'https://img/discogs-search'
# ── Stricter matching (issue: new sources returning WRONG cover art) ──
class _SearchClient:
"""search_albums returns whatever results it's given (title/artist/image)."""
def __init__(self, results):
self._results = results
self.search_calls = []
def get_album(self, album_id, include_tracks=False):
return None
def search_albums(self, query, limit=1):
self.search_calls.append((query, limit))
return list(self._results)
def test_search_rejects_wrong_artist_result(monkeypatch):
"""A result with the right-ish title but a DIFFERENT artist must be rejected
(this is what produced wrong covers from the new sources)."""
conn = _make_db((1, 'Album', 1, '', None, None, None, None, None))
context = _make_context(conn)
client = _SearchClient([SimpleNamespace(id='x', image_url='https://img/wrong',
title='Album', artist='Different Artist')])
monkeypatch.setattr(mca, 'get_primary_source', lambda: 'discogs')
monkeypatch.setattr(mca, 'get_client_for_source', lambda s: client if s == 'discogs' else None)
result = mca.MissingCoverArtJob().scan(context)
assert result.findings_created == 0 # wrong-artist art not accepted
assert context.findings == []
def test_search_skips_wrong_result_and_takes_matching_one(monkeypatch):
"""Given several results, take the first that actually matches title+artist."""
conn = _make_db((1, 'Album', 1, '', None, None, None, None, None))
context = _make_context(conn)
client = _SearchClient([
SimpleNamespace(id='a', image_url='https://img/wrong', title='Other Record', artist='Someone'),
SimpleNamespace(id='b', image_url='https://img/right', title='Album', artist='Artist'),
])
monkeypatch.setattr(mca, 'get_primary_source', lambda: 'discogs')
monkeypatch.setattr(mca, 'get_client_for_source', lambda s: client if s == 'discogs' else None)
result = mca.MissingCoverArtJob().scan(context)
assert result.findings_created == 1
assert context.findings[0]['details']['found_artwork_url'] == 'https://img/right'
def test_result_matches_unit():
m = mca.MissingCoverArtJob._result_matches
# exact + deluxe variant + featuring all accepted when artist matches
assert m({'title': 'Album', 'artist': 'Artist'}, 'Album', 'Artist')
assert m({'title': 'Album (Deluxe Edition)', 'artist': 'Artist'}, 'Album', 'Artist')
assert m({'title': 'Album', 'artist': 'The Artist'}, 'Album', 'Artist') # stopword 'the'
# wrong artist / wrong title rejected
assert not m({'title': 'Album', 'artist': 'Nope'}, 'Album', 'Artist')
assert not m({'title': 'Totally Other', 'artist': 'Artist'}, 'Album', 'Artist')
# no artist on result → require exact title
assert m({'title': 'Album'}, 'Album', 'Artist')
assert not m({'title': 'Album Deluxe'}, 'Album', 'Artist')

View file

@ -0,0 +1,83 @@
"""Tests for Navidrome cover-art URL building (#766).
The sync editor + modals referenced /api/navidrome/cover/<id> but no route
served it, and the URL behind it had to be a fully-authenticated Subsonic
getCoverArt URL. build_cover_art_url is that builder these pin its shape and
the not-connected guards (the token/salt are random per call, so we assert
structure + required params rather than an exact string).
"""
from __future__ import annotations
from urllib.parse import parse_qs, urlsplit
from core.navidrome_client import NavidromeClient
def _connected_client():
c = NavidromeClient()
c.base_url = "https://nav.example.com"
c.username = "boulder"
c.password = "hunter2"
return c
def test_builds_authenticated_cover_url():
url = _connected_client().build_cover_art_url("al-123")
parts = urlsplit(url)
assert parts.scheme == "https"
assert parts.netloc == "nav.example.com"
assert parts.path == "/rest/getCoverArt"
q = parse_qs(parts.query)
assert q["id"] == ["al-123"]
assert q["u"] == ["boulder"]
# Subsonic token auth: salted md5, never the raw password.
assert q["t"] and q["t"][0] != "hunter2"
assert q["s"] # salt present
assert "hunter2" not in url
for required in ("t", "s", "v", "c"):
assert required in q
def test_cover_url_is_deterministic_so_the_cache_hits():
# #766 review: the URL must be stable for a given (server, password,
# cover_id) — otherwise the image cache keys on a rotating salt and misses
# every request, re-fetching Navidrome each time + leaking dead rows.
c = _connected_client()
assert c.build_cover_art_url("al-123") == c.build_cover_art_url("al-123")
# ...and different covers still produce different URLs.
assert c.build_cover_art_url("al-123") != c.build_cover_art_url("al-999")
def test_cover_url_changes_with_password():
# A password change must invalidate the cached URL (new token).
c1 = _connected_client()
c2 = _connected_client()
c2.password = "different"
assert c1.build_cover_art_url("al-1") != c2.build_cover_art_url("al-1")
def test_size_param_optional():
assert "size" not in (_connected_client().build_cover_art_url("x") or "")
assert "size=300" in _connected_client().build_cover_art_url("x", size=300)
def test_cover_id_is_stringified():
url = _connected_client().build_cover_art_url(12345)
assert "id=12345" in url
def test_returns_none_when_not_connected():
c = NavidromeClient() # base_url is None
assert c.build_cover_art_url("al-1") is None
def test_returns_none_for_empty_cover_id():
assert _connected_client().build_cover_art_url("") is None
assert _connected_client().build_cover_art_url(None) is None
def test_returns_none_without_credentials():
c = NavidromeClient()
c.base_url = "https://nav.example.com" # but no username/password
assert c.build_cover_art_url("al-1") is None

View file

@ -0,0 +1,80 @@
"""Extreme battery for sync-editor add/remove planners (#768 Bug C)."""
from __future__ import annotations
from core.sync.playlist_edit import plan_playlist_add, remove_one_occurrence
# ── plan_playlist_add: link must not duplicate ────────────────────────────
def test_link_to_existing_track_does_not_insert():
# The reported loop: matching an unmatched source to a track already in
# the playlist (an "extra") must NOT add a second copy.
plan = plan_playlist_add(["a", "b", "nv72"], "nv72", is_link=True)
assert plan["should_insert"] is False
assert plan["new_ids"] == ["a", "b", "nv72"] # unchanged
def test_link_to_absent_track_inserts():
plan = plan_playlist_add(["a", "b"], "nv99", is_link=True, position=1)
assert plan["should_insert"] is True
assert plan["new_ids"] == ["a", "nv99", "b"]
def test_non_link_add_always_inserts_even_if_present():
# A plain add (no source link) may legitimately duplicate.
plan = plan_playlist_add(["a", "b"], "a", is_link=False)
assert plan["should_insert"] is True
assert plan["new_ids"].count("a") == 2
def test_add_appends_when_no_position():
plan = plan_playlist_add(["a", "b"], "c", is_link=False)
assert plan["new_ids"] == ["a", "b", "c"]
def test_add_clamps_out_of_range_position():
assert plan_playlist_add(["a"], "c", is_link=False, position=99)["new_ids"] == ["a", "c"]
assert plan_playlist_add(["a"], "c", is_link=False, position=-5)["new_ids"] == ["c", "a"]
def test_add_stringifies_ids():
plan = plan_playlist_add([1, 2, 72], 72, is_link=True)
assert plan["should_insert"] is False
# ── remove_one_occurrence: remove ONE, not all ────────────────────────────
def test_removes_only_one_of_duplicates():
# The #768 delete bug: two copies (pos 72, 73) — removing must drop ONE.
new_ids, removed = remove_one_occurrence(["a", "nv72", "nv72", "b"], "nv72")
assert removed is True
assert new_ids == ["a", "nv72", "b"] # one copy survives
def test_removes_exact_position_when_given():
new_ids, removed = remove_one_occurrence(["x", "x", "x"], "x", position=1)
assert removed is True
assert new_ids == ["x", "x"]
def test_falls_back_to_first_when_position_mismatches():
new_ids, removed = remove_one_occurrence(["a", "b", "c"], "b", position=0)
assert removed is True
assert new_ids == ["a", "c"]
def test_remove_absent_id_reports_not_removed():
new_ids, removed = remove_one_occurrence(["a", "b"], "zzz")
assert removed is False
assert new_ids == ["a", "b"]
def test_remove_single_occurrence():
new_ids, removed = remove_one_occurrence(["a", "b", "c"], "b")
assert (new_ids, removed) == (["a", "c"], True)
def test_remove_stringifies():
new_ids, removed = remove_one_occurrence([1, 2, 2, 3], 2)
assert removed and new_ids == ["1", "2", "3"]

View file

@ -0,0 +1,162 @@
"""Extreme battery for the playlist sync-editor reconcile (#768).
Covers: the reported YouTube failure (Bug A "Artist - Title" source matching
its clean server copy instead of showing unmatched + orphan extra), the
source_track_id echo (Bug B), and parity with the original three-pass behavior
(override exact fuzzy extra), plus duplicate-server-track handling.
"""
from __future__ import annotations
from core.sync.playlist_reconcile import norm_title, reconcile_playlist
def _src(name, artist, sid="", **kw):
return {"name": name, "artist": artist, "source_track_id": sid, **kw}
def _svr(title, artist, tid):
return {"title": title, "artist": artist, "id": tid, "ratingKey": tid}
def _status(combined):
return [(c["match_status"],
(c["source_track"] or {}).get("name"),
(c["server_track"] or {}).get("title")) for c in combined]
# ── Bug A: the reported YouTube case ──────────────────────────────────────
def test_youtube_artist_title_source_matches_clean_server_track():
source = [_src("Arctic Monkeys - Do I Wanna Know?", "Official Arctic Monkeys", "sp1")]
server = [_svr("Do I Wanna Know?", "Arctic Monkeys", "nv72")]
combined = reconcile_playlist(source, server)
assert len(combined) == 1
assert combined[0]["match_status"] == "matched"
assert combined[0]["server_track"]["id"] == "nv72"
# ...and the server track is NOT left as an orphan extra.
assert not any(c["match_status"] == "extra" for c in combined)
def test_youtube_match_does_not_leave_unmatched_or_extra():
# Before the fix this produced one 'missing' + one 'extra'.
source = [_src("The Killers - Mr. Brightside", "The KillersVEVO", "sp2")]
server = [_svr("Mr. Brightside", "The Killers", "nv5")]
statuses = [c["match_status"] for c in reconcile_playlist(source, server)]
assert statuses == ["matched"]
# ── Bug B: source_track_id is echoed back ─────────────────────────────────
def test_source_track_id_present_on_matched_entry():
source = [_src("Do I Wanna Know?", "Arctic Monkeys", "spotify:track:abc")]
server = [_svr("Do I Wanna Know?", "Arctic Monkeys", "nv1")]
combined = reconcile_playlist(source, server)
assert combined[0]["source_track"]["source_track_id"] == "spotify:track:abc"
def test_source_track_id_present_on_missing_entry():
# A genuinely-missing source must still carry its id so it can be
# manually matched and persisted (the #768 manual-match loop).
source = [_src("Some Obscure B-Side", "Some Artist", "spotify:track:xyz")]
server = [_svr("Completely Different", "Other Artist", "nv9")]
combined = reconcile_playlist(source, server)
missing = [c for c in combined if c["match_status"] == "missing"]
assert missing and missing[0]["source_track"]["source_track_id"] == "spotify:track:xyz"
# ── parity: override / exact / fuzzy / extra ──────────────────────────────
def test_override_pair_wins_first():
source = [_src("Anything", "Whoever", "s1")]
server = [_svr("Totally Different Title", "Nobody", "nvX")]
combined = reconcile_playlist(source, server, override_pairs={0: 0})
assert combined[0]["match_status"] == "matched"
assert combined[0]["confidence"] == 1.0
assert combined[0].get("override") is True
def test_exact_normalized_match_strips_feat():
source = [_src("Stay (feat. Justin Bieber)", "The Kid LAROI", "s1")]
server = [_svr("Stay", "The Kid LAROI", "nv1")]
assert reconcile_playlist(source, server)[0]["match_status"] == "matched"
def test_fuzzy_match_above_threshold():
source = [_src("Mr Brightside", "The Killers", "s1")]
server = [_svr("Mr. Brightside", "The Killers", "nv1")]
c = reconcile_playlist(source, server)[0]
assert c["match_status"] == "matched"
assert c["confidence"] >= 0.75
def test_truly_absent_track_is_missing_and_unrelated_server_is_extra():
source = [_src("Nonexistent Song", "Ghost Artist", "s1")]
server = [_svr("Yellow", "Coldplay", "nv1")]
statuses = sorted(c["match_status"] for c in reconcile_playlist(source, server))
assert statuses == ["extra", "missing"]
def test_each_server_track_claimed_once_no_double_match():
# Two identical source rows must not both claim the single server track.
source = [_src("Yellow", "Coldplay", "s1"), _src("Yellow", "Coldplay", "s2")]
server = [_svr("Yellow", "Coldplay", "nv1")]
combined = reconcile_playlist(source, server)
matched = [c for c in combined if c["match_status"] == "matched"]
missing = [c for c in combined if c["match_status"] == "missing"]
assert len(matched) == 1 and len(missing) == 1
def test_duplicate_server_tracks_one_matched_one_extra():
# The #768 duplicate scenario: two copies of the same track on the server.
source = [_src("Do I Wanna Know?", "Arctic Monkeys", "s1")]
server = [_svr("Do I Wanna Know?", "Arctic Monkeys", "nv72"),
_svr("Do I Wanna Know?", "Arctic Monkeys", "nv73")]
combined = reconcile_playlist(source, server)
assert sorted(c["match_status"] for c in combined) == ["extra", "matched"]
# ── #766: source borrows the matched server cover ─────────────────────────
def _svr_art(title, artist, tid, thumb):
return {"title": title, "artist": artist, "id": tid, "ratingKey": tid, "thumb": thumb}
def test_artless_source_borrows_matched_server_cover():
# YouTube-style source row, no art of its own, matched to a server track
# that has a cover -> source side borrows it.
source = [_src("Arctic Monkeys - Do I Wanna Know?", "Official Arctic Monkeys", "s1")]
server = [_svr_art("Do I Wanna Know?", "Arctic Monkeys", "nv1", "/api/navidrome/cover/al42")]
combined = reconcile_playlist(source, server)
assert combined[0]["match_status"] == "matched"
assert combined[0]["source_track"]["image_url"] == "/api/navidrome/cover/al42"
def test_source_keeps_its_own_art_when_present():
# A Spotify-style source row with its own CDN art must NOT be overwritten.
source = [_src("Do I Wanna Know?", "Arctic Monkeys", "s1", image_url="https://cdn/spotify.jpg")]
server = [_svr_art("Do I Wanna Know?", "Arctic Monkeys", "nv1", "/api/navidrome/cover/al42")]
combined = reconcile_playlist(source, server)
assert combined[0]["source_track"]["image_url"] == "https://cdn/spotify.jpg"
def test_unmatched_source_has_no_cover_to_borrow():
source = [_src("Totally Absent Song", "Ghost", "s1")]
server = [_svr_art("Something Else", "Nobody", "nv1", "/api/navidrome/cover/al99")]
combined = reconcile_playlist(source, server)
missing = [c for c in combined if c["match_status"] == "missing"]
assert missing and not missing[0]["source_track"]["image_url"]
def test_borrow_skipped_when_server_track_has_no_thumb():
source = [_src("Do I Wanna Know?", "Arctic Monkeys", "s1")]
server = [_svr("Do I Wanna Know?", "Arctic Monkeys", "nv1")] # no thumb
combined = reconcile_playlist(source, server)
assert combined[0]["match_status"] == "matched"
assert not combined[0]["source_track"]["image_url"]
def test_norm_title_helper_parity():
assert norm_title("Stay (feat. X)") == "stay"
assert norm_title("Song (2019 Remaster)") == "song"
assert norm_title("Album (Deluxe Edition)") == "album"

View file

@ -0,0 +1,119 @@
"""Seam tests for recommendation explainability — get_recommendation_sources.
The similar-artists worker stores rows keyed by a polymorphic
``source_artist_id`` (one of the user's artists' spotify / itunes / deezer /
musicbrainz ids). The "because you have X, Y, Z" explanation has to resolve
that id back to a display name by matching it against every provider-id column
on BOTH the library (`artists`) and `watchlist_artists` tables.
These tests build a real schema via MusicDatabase(tmp) and insert rows directly
so the SQL join is exercised end to end.
"""
from __future__ import annotations
import sqlite3
from database.music_database import MusicDatabase
def _seed(path):
"""Insert two library artists + one watchlist artist, and similar_artists
rows that point at them via different provider-id columns."""
conn = sqlite3.connect(path)
cur = conn.cursor()
# Library artists, each matched on a DIFFERENT provider id column
cur.execute("INSERT INTO artists (name, spotify_artist_id) VALUES ('Radiohead', 'sp_radiohead')")
cur.execute("INSERT INTO artists (name, musicbrainz_id) VALUES ('Portishead', 'mb_portishead')")
# A watchlist-only artist (not in library), matched on deezer id
cur.execute(
"INSERT INTO watchlist_artists (artist_name, deezer_artist_id, profile_id) "
"VALUES ('Bjork', 'dz_bjork', 1)"
)
# 'Thom Yorke' is listed as similar by Radiohead (spotify id) AND Bjork
# (watchlist, deezer id) -> two distinct sources.
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES ('sp_radiohead', 'Thom Yorke', 1)"
)
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES ('dz_bjork', 'Thom Yorke', 1)"
)
# 'Massive Attack' listed by Portishead (musicbrainz id) only.
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES ('mb_portishead', 'Massive Attack', 1)"
)
# An orphan: source id matches nobody -> must NOT produce a phantom source.
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES ('sp_ghost', 'Nobody Knows', 1)"
)
conn.commit()
conn.close()
def test_resolves_library_and_watchlist_sources(tmp_path):
path = str(tmp_path / "m.db")
MusicDatabase(path) # build schema via migrations
_seed(path)
db = MusicDatabase(path)
out = db.get_recommendation_sources(["Thom Yorke", "Massive Attack", "Nobody Knows"])
# Thom Yorke: resolved from a library spotify id AND a watchlist deezer id
assert out["Thom Yorke"] == ["Bjork", "Radiohead"] # deduped + name-sorted
# Massive Attack: resolved via musicbrainz id on a library artist
assert out["Massive Attack"] == ["Portishead"]
# Orphan recommendation has no resolvable source -> omitted entirely
assert "Nobody Knows" not in out
def test_empty_input_returns_empty(tmp_path):
path = str(tmp_path / "m.db")
db = MusicDatabase(path)
assert db.get_recommendation_sources([]) == {}
assert db.get_recommendation_sources([None, ""]) == {}
def test_max_per_caps_sources(tmp_path):
path = str(tmp_path / "m.db")
MusicDatabase(path)
conn = sqlite3.connect(path)
cur = conn.cursor()
# Five library artists all listing the same recommendation
for i in range(5):
cur.execute("INSERT INTO artists (name, spotify_artist_id) VALUES (?, ?)",
(f"Artist{i}", f"sp_{i}"))
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES (?, 'Shared Rec', 1)", (f"sp_{i}",))
conn.commit()
conn.close()
db = MusicDatabase(path)
out = db.get_recommendation_sources(["Shared Rec"], max_per=3)
assert len(out["Shared Rec"]) == 3 # capped
assert out["Shared Rec"] == ["Artist0", "Artist1", "Artist2"] # name-sorted
def test_profile_scoping(tmp_path):
"""A source artist in another profile must not leak into the explanation."""
path = str(tmp_path / "m.db")
MusicDatabase(path)
conn = sqlite3.connect(path)
cur = conn.cursor()
cur.execute("INSERT INTO artists (name, spotify_artist_id) VALUES ('Mine', 'sp_mine')")
# similar row belongs to profile 2, not 1
cur.execute(
"INSERT INTO similar_artists (source_artist_id, similar_artist_name, profile_id) "
"VALUES ('sp_mine', 'Rec X', 2)"
)
conn.commit()
conn.close()
db = MusicDatabase(path)
assert db.get_recommendation_sources(["Rec X"], profile_id=1) == {}
assert db.get_recommendation_sources(["Rec X"], profile_id=2) == {"Rec X": ["Mine"]}

View file

@ -0,0 +1,75 @@
"""_resolve_source honors a pinned canonical release (#765 Stage 3, read side).
Gated + side-effect-free: only changes behavior for albums that already carry a
canonical_source/canonical_album_id, and an explicit user source pick
(strict_source) still wins. No canonical -> byte-identical to before.
"""
from __future__ import annotations
import core.library_reorganize as lr
def _patch_fetch(monkeypatch, tracklists):
"""tracklists: {(source, album_id): items_or_None}. Patches the album +
tracklist fetchers and the normaliser (pass-through)."""
def get_album(source, aid):
return {"name": f"{source}:{aid}"} if tracklists.get((source, aid)) else None
def get_tracks(source, aid):
return tracklists.get((source, aid))
monkeypatch.setattr(lr, "get_album_for_source", get_album)
monkeypatch.setattr(lr, "get_album_tracks_for_source", get_tracks)
monkeypatch.setattr(lr, "_normalize_album_tracks", lambda items: items or [])
monkeypatch.setattr(lr, "get_source_priority", lambda primary: ["spotify", "itunes", "deezer"])
def test_canonical_source_preferred_over_priority(monkeypatch):
# Album has spotify (priority winner) AND a pinned canonical = deezer.
_patch_fetch(monkeypatch, {
("spotify", "sp1"): [{"name": "x"}],
("deezer", "dz1"): [{"name": "y"}],
})
album_data = {
"spotify_album_id": "sp1", "deezer_id": "dz1",
"canonical_source": "deezer", "canonical_album_id": "dz1",
}
source, api_album, items = lr._resolve_source(album_data, "spotify")
assert source == "deezer" # canonical beats the priority walk
def test_canonical_fetch_failure_falls_back_to_priority(monkeypatch):
# Canonical points at musicbrainz but that fetch yields nothing -> fall back.
_patch_fetch(monkeypatch, {
("spotify", "sp1"): [{"name": "x"}],
# no entry for ('musicbrainz', 'mb1') -> get_tracks returns None
})
album_data = {
"spotify_album_id": "sp1",
"canonical_source": "musicbrainz", "canonical_album_id": "mb1",
}
source, _, _ = lr._resolve_source(album_data, "spotify")
assert source == "spotify" # fell back to priority
def test_strict_source_ignores_canonical(monkeypatch):
# User explicitly picked spotify in the modal — their choice wins over canonical.
_patch_fetch(monkeypatch, {
("spotify", "sp1"): [{"name": "x"}],
("deezer", "dz1"): [{"name": "y"}],
})
album_data = {
"spotify_album_id": "sp1", "deezer_id": "dz1",
"canonical_source": "deezer", "canonical_album_id": "dz1",
}
source, _, _ = lr._resolve_source(album_data, "spotify", strict_source=True)
assert source == "spotify"
def test_no_canonical_unchanged(monkeypatch):
# No canonical set -> identical to legacy priority resolution.
_patch_fetch(monkeypatch, {("spotify", "sp1"): [{"name": "x"}]})
album_data = {"spotify_album_id": "sp1"}
source, _, _ = lr._resolve_source(album_data, "spotify")
assert source == "spotify"

View file

@ -0,0 +1,57 @@
"""Seam tests for resolve_mirrored_playlist source-vs-PK resolution.
Regression cover for the PR #780 follow-up: numeric *upstream* ids (Deezer
playlist ids are all-digit) must resolve by (source, source_playlist_id), NOT
be mistaken for the mirrored-playlists primary key. The old PK-first logic made
the Deezer organize-by-playlist toggle resolve the wrong row (or nothing).
"""
from __future__ import annotations
from database.music_database import MusicDatabase
def test_numeric_source_id_resolves_by_source_not_pk(tmp_path):
"""A Deezer-style all-numeric upstream id resolves the right row."""
db = MusicDatabase(str(tmp_path / "m.db"))
pk = db.mirror_playlist(source='deezer', source_playlist_id='908622995',
name='My Deezer Mix', tracks=[], profile_id=1)
assert pk
row = db.resolve_mirrored_playlist('908622995', profile_id=1, default_source='deezer')
assert row is not None
assert row['id'] == pk
assert row['source'] == 'deezer'
# And it must NOT have been a PK lookup: 908622995 is not a valid PK here.
assert db.get_mirrored_playlist(908622995) is None
def test_spotify_alphanumeric_resolves_by_source(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
pk = db.mirror_playlist(source='spotify', source_playlist_id='37i9dQZF1DXcBWIGoYBM5M',
name='Top Hits', tracks=[], profile_id=1)
row = db.resolve_mirrored_playlist('37i9dQZF1DXcBWIGoYBM5M', profile_id=1, default_source='spotify')
assert row is not None and row['id'] == pk
def test_pk_fallback_when_no_source_match(tmp_path):
"""A numeric ref that isn't a known source id still resolves via PK fallback."""
db = MusicDatabase(str(tmp_path / "m.db"))
pk = db.mirror_playlist(source='spotify', source_playlist_id='abc123XYZ',
name='Sp', tracks=[], profile_id=1)
row = db.resolve_mirrored_playlist(str(pk), profile_id=1, default_source='spotify')
assert row is not None and row['id'] == pk
def test_resolution_is_profile_scoped(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
db.mirror_playlist(source='deezer', source_playlist_id='555',
name='D', tracks=[], profile_id=1)
# Another profile must not resolve profile 1's Deezer playlist by source.
assert db.resolve_mirrored_playlist('555', profile_id=2, default_source='deezer') is None
def test_empty_refs_return_none(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
assert db.resolve_mirrored_playlist(None) is None
assert db.resolve_mirrored_playlist('') is None
assert db.resolve_mirrored_playlist(' ') is None

View file

@ -0,0 +1,84 @@
"""Tests for the DB-update / deep-scan monitor decision (stall-based timeout).
Regression: a large library can deep-scan for many hours while progressing
fine. The old monitor used a hard 2-hour TOTAL cap, so it falsely marked a
healthy, still-running scan 'error' (the scan thread kept going uncancelled).
The decision now keys off STALL (no progress), so an actively-progressing scan
never times out no matter how long the whole library takes.
"""
from __future__ import annotations
from core.automation.handlers.database_update import (
scan_wait_action,
_STALL_WARNING_SECONDS,
_STALL_TIMEOUT_SECONDS,
_ABSOLUTE_CAP_SECONDS,
)
# --- the headline regression -------------------------------------------------
def test_long_but_progressing_scan_never_times_out():
# 5 hours elapsed total, but progress moved 5s ago -> keep waiting, NOT error.
assert scan_wait_action(
status='running', idle_seconds=5, total_seconds=5 * 3600,
) == 'continue'
def test_very_long_progressing_scan_still_continues():
# 12h total, just progressed — old code would have failed at 2h.
assert scan_wait_action(
status='running', idle_seconds=2, total_seconds=12 * 3600,
) == 'continue'
# --- finished / not-running --------------------------------------------------
def test_finished_when_not_running():
for st in ('completed', 'error', 'idle', 'finished'):
assert scan_wait_action(status=st, idle_seconds=0, total_seconds=0) == 'finished'
def test_finished_takes_precedence_even_if_stalled():
# Task already ended — don't report a stall.
assert scan_wait_action(
status='completed', idle_seconds=_STALL_TIMEOUT_SECONDS + 1, total_seconds=10,
) == 'finished'
# --- stall warning vs stall timeout ------------------------------------------
def test_warns_after_stall_warning_threshold():
assert scan_wait_action(
status='running', idle_seconds=_STALL_WARNING_SECONDS + 1, total_seconds=1000,
) == 'warn'
def test_stall_timeout_after_no_progress():
assert scan_wait_action(
status='running', idle_seconds=_STALL_TIMEOUT_SECONDS + 1, total_seconds=2000,
) == 'stall_timeout'
def test_just_below_warning_keeps_going():
assert scan_wait_action(
status='running', idle_seconds=_STALL_WARNING_SECONDS - 1, total_seconds=1000,
) == 'continue'
# --- absolute backstop -------------------------------------------------------
def test_absolute_cap_is_last_resort():
# Even if somehow progressing, a 24h+ wait trips the runaway-loop backstop.
assert scan_wait_action(
status='running', idle_seconds=1, total_seconds=_ABSOLUTE_CAP_SECONDS + 1,
) == 'abs_timeout'
def test_thresholds_are_ordered_sensibly():
assert _STALL_WARNING_SECONDS < _STALL_TIMEOUT_SECONDS < _ABSOLUTE_CAP_SECONDS

View file

@ -51,7 +51,8 @@ SPLIT_MODULES = [
]
# Other JS files that exist in static/ but are NOT part of the split
NON_SPLIT_JS = {"setup-wizard.js", "docs.js", "helper.js", "particles.js", "worker-orbs.js"}
NON_SPLIT_JS = {"setup-wizard.js", "docs.js", "helper.js", "particles.js", "worker-orbs.js",
"enrichment-manager.js"}
# Pre-existing duplicate helper functions that lived in the original monolith.
# In a plain <script> context the last-loaded declaration wins. These are NOT
@ -216,7 +217,7 @@ class TestOnclickCoverage:
self.all_fns.update(_all_function_decls(text))
# Also include non-split JS files that are loaded
for extra in ("setup-wizard.js", "docs.js", "helper.js"):
for extra in ("setup-wizard.js", "docs.js", "helper.js", "enrichment-manager.js"):
path = _STATIC / extra
if path.exists():
self.all_fns.update(_all_function_decls(_read(path)))

View file

@ -0,0 +1,93 @@
"""Regression seam tests for MusicMap fetch status-code classification.
Root cause (found via the worker's WARNING observability): MusicMap returns
HTTP 404 when an artist simply has no map page a *not-found*, not a failure.
`_fetch_musicmap_similar_artist_names` calls `response.raise_for_status()`,
which raises `requests.exceptions.HTTPError` carrying the real 404. The error
handler in `iter_musicmap_similar_artist_events` used to flatten EVERY network
error to `status_code: 502`, so the worker (which maps 400/404 not_found,
everything else error) miscounted these as errors.
These tests pin the fix: the real HTTP status is surfaced, so a 404 reads as
404 ( not_found downstream) while response-less failures (timeout, connection
drop) still fall back to 502 ( error, eligible for retry).
"""
from __future__ import annotations
import requests
import core.metadata.similar_artists as sa
def _force_reach_fetch(monkeypatch):
"""Get past the source-chain / provider-availability guard so the test
exercises the fetch error path, not the 'no providers' branch."""
monkeypatch.setattr(sa, "_get_source_chain_for_lookup", lambda _opts: ["spotify"])
monkeypatch.setattr(sa.metadata_registry, "get_client_for_source", lambda _src: object())
def _http_error(status_code):
"""A requests.HTTPError carrying a response with the given status — exactly
what response.raise_for_status() raises on a 4xx/5xx."""
resp = requests.Response()
resp.status_code = status_code
return requests.exceptions.HTTPError(f"{status_code} Client Error", response=resp)
def test_musicmap_404_surfaced_as_404(monkeypatch):
"""A 404 from MusicMap → status_code 404 (so the worker calls it not_found)."""
_force_reach_fetch(monkeypatch)
def _raise_404(_name):
raise _http_error(404)
monkeypatch.setattr(sa, "_fetch_musicmap_similar_artist_names", _raise_404)
result = sa.get_musicmap_similar_artists("Pharooo")
assert result["success"] is False
assert result["status_code"] == 404 # was wrongly 502 before the fix
def test_musicmap_timeout_falls_back_to_502(monkeypatch):
"""A response-less failure (timeout) has no status → 502 (stays an error)."""
_force_reach_fetch(monkeypatch)
def _raise_timeout(_name):
raise requests.exceptions.Timeout("timed out")
monkeypatch.setattr(sa, "_fetch_musicmap_similar_artist_names", _raise_timeout)
result = sa.get_musicmap_similar_artists("Some Artist")
assert result["success"] is False
assert result["status_code"] == 502
def test_musicmap_500_stays_an_error(monkeypatch):
"""A real upstream 5xx is surfaced as-is → not in (400,404) → error/retry."""
_force_reach_fetch(monkeypatch)
def _raise_500(_name):
raise _http_error(500)
monkeypatch.setattr(sa, "_fetch_musicmap_similar_artist_names", _raise_500)
result = sa.get_musicmap_similar_artists("Some Artist")
assert result["success"] is False
assert result["status_code"] == 500
def test_worker_classifies_404_as_not_found():
"""End-to-end seam: a 404 fetch result → the worker marks 'not_found',
not 'error' (closing the loop the upstream fix enables)."""
import core.similar_artists_worker as w
def fake_fetch(_name, limit=25):
return {"success": False, "status_code": 404, "error": "no map page"}
def fake_store(**_kwargs):
raise AssertionError("must not store anything on a not-found")
status, count, _detail = w.process_artist("sp1", "Pharooo", fake_fetch, fake_store)
assert status == "not_found"
assert count == 0

View file

@ -0,0 +1,146 @@
"""Seam tests for the Similar-Artists enrichment worker's pure logic.
The worker fills the similar_artists table for LIBRARY artists (the watchlist
scanner only does watchlist artists). These tests exercise the import-light
seams in isolation no DB, no MusicMap via injected fakes:
- pick_source_artist_id keying priority (and skip un-matched artists)
- map_payload_to_store_kwargs MusicMap {id,source} the right id column
- process_artist fetchmatchstore orchestration + status codes
"""
from __future__ import annotations
import core.similar_artists_worker as w
# --------------------------------------------------------------------------
# pick_source_artist_id — which id keys the artist's similars (must match the
# watchlist scanner's priority so both write the SAME source_artist_id).
# --------------------------------------------------------------------------
def test_pick_source_artist_id_priority():
row = {'spotify_artist_id': 'sp1', 'itunes_artist_id': 'it1', 'deezer_id': 'dz1'}
assert w.pick_source_artist_id(row) == 'sp1' # spotify wins
assert w.pick_source_artist_id({'itunes_artist_id': 'it1', 'deezer_id': 'dz1'}) == 'it1'
assert w.pick_source_artist_id({'deezer_id': 'dz1'}) == 'dz1'
assert w.pick_source_artist_id({'musicbrainz_id': 'mb1'}) == 'mb1'
def test_pick_source_artist_id_none_when_unmatched():
# Library artist not matched to any metadata source yet → skip (None).
assert w.pick_source_artist_id({'spotify_artist_id': None, 'itunes_artist_id': ''}) is None
assert w.pick_source_artist_id({}) is None
# --------------------------------------------------------------------------
# map_payload_to_store_kwargs — MusicMap payload {id, source} → store kwarg.
# --------------------------------------------------------------------------
def test_map_payload_each_source():
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'spotify'}) == {'similar_artist_spotify_id': 'x'}
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'itunes'}) == {'similar_artist_itunes_id': 'x'}
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'deezer'}) == {'similar_artist_deezer_id': 'x'}
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'musicbrainz'}) == {'similar_artist_musicbrainz_id': 'x'}
def test_map_payload_unknown_source_or_no_id():
# discogs has no column → name-only (empty kwargs), not a crash.
assert w.map_payload_to_store_kwargs({'id': 'x', 'source': 'discogs'}) == {}
assert w.map_payload_to_store_kwargs({'source': 'spotify'}) == {} # no id
# --------------------------------------------------------------------------
# process_artist — fetch → store, status classification, keying.
# --------------------------------------------------------------------------
def _capture_store():
calls = []
def store(**kwargs):
calls.append(kwargs)
return True
return store, calls
def test_process_artist_matched_stores_with_keying():
store, calls = _capture_store()
payload = {
'success': True,
'similar_artists': [
{'name': 'B', 'id': 'sp_b', 'source': 'spotify', 'genres': ['rap'], 'popularity': 70},
{'name': 'C', 'id': 'it_c', 'source': 'itunes', 'image_url': 'http://x'},
],
}
status, count, detail = w.process_artist('SRC1', 'A', lambda n, l: payload, store, limit=25, profile_id=1)
assert status == 'matched' and count == 2 and detail == ''
# All similars keyed by the SOURCE artist id we passed (not the library PK).
assert all(c['source_artist_id'] == 'SRC1' for c in calls)
assert all(c['profile_id'] == 1 for c in calls)
# Provider id mapped to the right column; rank preserved in order.
assert calls[0]['similar_artist_spotify_id'] == 'sp_b' and calls[0]['similarity_rank'] == 1
assert calls[1]['similar_artist_itunes_id'] == 'it_c' and calls[1]['similarity_rank'] == 2
assert calls[0]['genres'] == ['rap'] and calls[0]['popularity'] == 70
def test_process_artist_not_found_when_no_matches():
store, calls = _capture_store()
status, count, detail = w.process_artist('S', 'A', lambda n, l: {'success': True, 'similar_artists': []}, store)
assert status == 'not_found' and count == 0 and calls == [] and detail == 'no matches'
def test_process_artist_not_found_on_404():
# Genuinely no MusicMap entry — shouldn't be retried as an error.
store, _ = _capture_store()
status, _, _ = w.process_artist('S', 'A', lambda n, l: {'success': False, 'status_code': 404}, store)
assert status == 'not_found'
def test_process_artist_error_on_outage_is_retriable_and_explains_why():
# 5xx / no providers → transient error (retried after retry_days), and the
# reason is surfaced (code + message) so the cause is diagnosable, not silent.
store, _ = _capture_store()
status, _, detail = w.process_artist(
'S', 'A', lambda n, l: {'success': False, 'status_code': 502, 'error': 'Failed to fetch from MusicMap'}, store)
assert status == 'error'
assert '502' in detail and 'MusicMap' in detail
def test_process_artist_error_when_fetch_raises_carries_detail():
store, _ = _capture_store()
def boom(n, l):
raise RuntimeError('musicmap down')
status, count, detail = w.process_artist('S', 'A', boom, store)
assert status == 'error' and count == 0 and 'musicmap down' in detail
def test_process_artist_skips_similars_without_a_storable_source_id():
# Every stored similar MUST carry a metadata source id (spotify/itunes/deezer/
# musicbrainz) — otherwise it's not actionable. A match on a source with no id
# column (e.g. discogs) is skipped, never stored name-only.
store, calls = _capture_store()
payload = {'success': True, 'similar_artists': [
{'name': 'KeepMe', 'id': 'sp1', 'source': 'spotify'},
{'name': 'DropMe', 'id': 'dg1', 'source': 'discogs'}, # no id column → must skip
{'name': 'NoId', 'source': 'spotify'}, # no id at all → must skip
]}
status, count, _ = w.process_artist('S', 'A', lambda n, l: payload, store)
assert count == 1 and len(calls) == 1
assert calls[0]['similar_artist_name'] == 'KeepMe'
assert calls[0]['similar_artist_spotify_id'] == 'sp1'
def test_process_artist_skips_unstorable_but_counts_real():
# A store() that fails for one row shouldn't abort the rest.
calls = []
def store(**kwargs):
calls.append(kwargs)
return kwargs['similar_artist_name'] != 'B' # B fails to store
payload = {'success': True, 'similar_artists': [
{'name': 'B', 'id': '1', 'source': 'spotify'},
{'name': 'C', 'id': '2', 'source': 'spotify'},
]}
status, count, _ = w.process_artist('S', 'A', lambda n, l: payload, store)
assert status == 'matched' and count == 1 and len(calls) == 2

113
tests/test_source_title.py Normal file
View file

@ -0,0 +1,113 @@
"""Extreme test battery for source-track normalization (#768).
YouTube/streaming sources carry "Artist - Song" titles and "Official Artist"/
"Artist - Topic"/"ArtistVEVO" artist names; the library has clean metadata, so
matching fails and tracks are reported missing. These helpers strip the
decoration. The batteries below pin both the positives (must clean) and the
negatives (must NOT mangle real titles/artists).
"""
from __future__ import annotations
import pytest
from core.text.source_title import (
canonical_source_track,
clean_source_artist,
strip_artist_prefix,
)
# ── clean_source_artist ───────────────────────────────────────────────────
@pytest.mark.parametrize("raw,expected", [
("Official Arctic Monkeys", "Arctic Monkeys"),
("The Official Weeknd", "Weeknd"),
("Arctic Monkeys - Topic", "Arctic Monkeys"),
("Coldplay - Topic", "Coldplay"),
("ColdplayVEVO", "Coldplay"),
("Coldplay VEVO", "Coldplay"),
("EminemVEVO", "Eminem"),
(" Official Radiohead ", "Radiohead"),
])
def test_clean_source_artist_strips_decoration(raw, expected):
assert clean_source_artist(raw) == expected
@pytest.mark.parametrize("raw", [
"Arctic Monkeys", # already clean
"Coldplay",
"Twenty One Pilots",
"Death",
"U2",
"AJR", # would be emptied by a naive vevo/official strip
"",
])
def test_clean_source_artist_leaves_clean_names(raw):
assert clean_source_artist(raw) == raw
def test_clean_source_artist_never_empties():
# Pathological: artist that is ONLY decoration must not become "".
assert clean_source_artist("VEVO") == "VEVO"
assert clean_source_artist("Official ") == "Official"
# ── strip_artist_prefix ───────────────────────────────────────────────────
@pytest.mark.parametrize("title,artist,expected", [
("Arctic Monkeys - Do I Wanna Know?", "Arctic Monkeys", "Do I Wanna Know?"),
("Death - Pull the Plug", "Death", "Pull the Plug"),
("Coldplay Yellow", "Coldplay", "Yellow"), # en dash
("Coldplay — Yellow", "Coldplay", "Yellow"), # em dash
("Eminem: Lose Yourself", "Eminem", "Lose Yourself"), # colon
("Daft Punk | Get Lucky", "Daft Punk", "Get Lucky"), # pipe
("ARCTIC MONKEYS - 505", "arctic monkeys", "505"), # case-fold
])
def test_strip_artist_prefix_strips_when_prefix_is_artist(title, artist, expected):
assert strip_artist_prefix(title, artist) == expected
@pytest.mark.parametrize("title,artist", [
("Marvin Gaye", "Charlie Puth"), # title is not "artist - ..."
("Do I Wanna Know?", "Arctic Monkeys"), # already clean
("Self-Titled", "Whoever"), # hyphen w/o spaces — not a sep
("Jay-Z Anthem", "Somebody"), # hyphen inside a word
("Song - Live", "Coldplay"), # prefix "Song" != artist
("Stay With Me", "Sam Smith"),
("", "Arctic Monkeys"),
("Arctic Monkeys -", "Arctic Monkeys"), # nothing after sep -> unchanged
])
def test_strip_artist_prefix_leaves_others_untouched(title, artist):
assert strip_artist_prefix(title, artist) == title
def test_strip_only_first_separator():
# "Artist - Song - Remix" -> strip only the leading artist segment.
assert strip_artist_prefix("Gorillaz - Feel Good Inc - Remix", "Gorillaz") == "Feel Good Inc - Remix"
# ── canonical_source_track (combined) ─────────────────────────────────────
def test_canonical_handles_youtube_channel_and_prefix():
# The reported case: channel-name artist + "Artist - Title" title.
title, artist = canonical_source_track(
"Arctic Monkeys - Do I Wanna Know?", "Official Arctic Monkeys",
)
assert title == "Do I Wanna Know?"
assert artist == "Arctic Monkeys"
def test_canonical_strips_prefix_using_raw_artist_when_clean_differs():
# Title prefixed with the channel-style raw artist itself.
title, artist = canonical_source_track(
"Official Arctic Monkeys - 505", "Official Arctic Monkeys",
)
# cleaned artist is "Arctic Monkeys"; raw prefix "Official Arctic Monkeys"
# also stripped via the raw-artist fallback.
assert title == "505"
assert artist == "Arctic Monkeys"
def test_canonical_noop_on_clean_input():
assert canonical_source_track("Yellow", "Coldplay") == ("Yellow", "Coldplay")

View file

@ -0,0 +1,125 @@
"""Full public-playlist fetch via the optional SpotipyFree library.
The library is GPL-3.0 and user-installed, so it's never imported in tests —
a fake spotipy-compatible client is injected to exercise normalisation +
pagination, and the embed fallback orchestration is tested separately. So a
missing/broken library can never make the link path worse than the embed 100.
"""
from __future__ import annotations
import pytest
import core.spotify_public_api as papi
import core.spotify_public_scraper as scraper
# --------------------------------------------------------------------------
# Track normalisation
# --------------------------------------------------------------------------
def test_normalize_api_track_shape():
item = {'track': {'id': 't1', 'name': 'Song', 'artists': [{'name': 'A'}, {'name': 'B'}],
'duration_ms': 1000, 'explicit': True}}
assert papi.normalize_api_track(item, 4) == {
'id': 't1', 'name': 'Song', 'artists': [{'name': 'A'}, {'name': 'B'}],
'duration_ms': 1000, 'is_explicit': True, 'track_number': 5,
}
def test_normalize_api_track_skips_unusable():
assert papi.normalize_api_track({'track': {'id': None}}, 0) is None # local/removed
assert papi.normalize_api_track({}, 0) is None
t = papi.normalize_api_track({'track': {'id': 'x', 'name': 'N'}}, 0)
assert t['artists'] == [{'name': 'Unknown Artist'}] # fallback
# --------------------------------------------------------------------------
# Full fetch with an injected fake SpotipyFree client (spotipy-shaped)
# --------------------------------------------------------------------------
class _FakeClient:
"""Minimal spotipy-compatible client: playlist() + playlist_items() + next()."""
def __init__(self, total, *, fail_items=False):
self.total, self.fail_items = total, fail_items
def playlist(self, pid, limit=-1, offset=0, *args, **kwargs):
return {'name': 'My Playlist', 'owner': {'display_name': 'Owner'}}
def _page(self, offset):
n = min(100, max(0, self.total - offset))
items = [{'track': {'id': f't{offset + i}', 'name': f'S{offset + i}',
'artists': [{'name': 'A'}], 'duration_ms': 1000, 'explicit': False}}
for i in range(n)]
nxt = offset + 100
return {'items': items, 'next': ('u' if nxt < self.total else None), '_next': nxt}
def playlist_items(self, pid):
if self.fail_items:
raise RuntimeError('boom')
return self._page(0)
def next(self, results):
return self._page(results['_next'])
def test_full_fetch_paginates_past_100():
result = papi.fetch_public_playlist_full('pl1', client_factory=lambda: _FakeClient(250))
assert result['name'] == 'My Playlist'
assert result['subtitle'] == 'Owner'
assert len(result['tracks']) == 250 # 100+100+50, not capped at 100
assert result['tracks'][0]['track_number'] == 1
assert result['tracks'][-1]['id'] == 't249'
assert result['type'] == 'playlist' and result['id'] == 'pl1'
def test_full_fetch_single_page():
result = papi.fetch_public_playlist_full('pl1', client_factory=lambda: _FakeClient(30))
assert len(result['tracks']) == 30
def test_full_fetch_raises_when_library_missing():
# _default_client would raise ImportError; simulate via the factory.
def missing():
raise ImportError("No module named 'SpotipyFree'")
with pytest.raises(Exception):
papi.fetch_public_playlist_full('pl1', client_factory=missing)
def test_full_fetch_raises_when_no_tracks():
with pytest.raises(Exception):
papi.fetch_public_playlist_full('pl1', client_factory=lambda: _FakeClient(0))
# --------------------------------------------------------------------------
# Fallback orchestration (the safety net) — full path vs embed scraper
# --------------------------------------------------------------------------
def test_fetch_public_uses_full_when_it_succeeds(monkeypatch):
calls = {'embed': 0}
monkeypatch.setattr(papi, 'fetch_public_playlist_full',
lambda pid, **kw: {'name': 'Full', 'tracks': [{'id': 'a'}] * 200})
monkeypatch.setattr(scraper, 'scrape_spotify_embed',
lambda *a, **k: calls.__setitem__('embed', calls['embed'] + 1) or {'tracks': []})
out = scraper.fetch_spotify_public('playlist', 'pl1')
assert len(out['tracks']) == 200 and calls['embed'] == 0 # full won, embed not called
def test_fetch_public_falls_back_to_embed_on_failure(monkeypatch):
def boom(pid, **kw):
raise RuntimeError('library not installed / spotify changed')
monkeypatch.setattr(papi, 'fetch_public_playlist_full', boom)
monkeypatch.setattr(scraper, 'scrape_spotify_embed',
lambda *a, **k: {'name': 'Embed', 'tracks': [{'id': 'e'}]})
out = scraper.fetch_spotify_public('playlist', 'pl1')
assert out['name'] == 'Embed' # graceful fallback
def test_fetch_public_album_uses_embed_directly(monkeypatch):
full_called = {'n': 0}
monkeypatch.setattr(papi, 'fetch_public_playlist_full',
lambda pid, **kw: full_called.__setitem__('n', 1) or {})
monkeypatch.setattr(scraper, 'scrape_spotify_embed',
lambda *a, **k: {'name': 'Album', 'tracks': [{'id': 'x'}]})
out = scraper.fetch_spotify_public('album', 'al1')
assert out['name'] == 'Album' and full_called['n'] == 0 # albums skip full-fetch

View file

@ -0,0 +1,153 @@
"""Tests for the title word-overlap guard (#769).
Playlist sync matched tracks NOT in the library to a different song by the
SAME artist with high confidence ("Dani California" -> "Californication";
"Under The Bridge" -> "Around the World"). Root cause: confidence is
0.5*title + 0.5*artist, same-artist always gives artist=1.0, and the title
score is a SequenceMatcher char ratio that over-credits unrelated titles
sharing a substring or a stopword. titles_plausibly_same gates those out.
Two layers tested:
1. titles_plausibly_same in isolation (the pure decision).
2. the real _calculate_track_confidence end-to-end, asserting the two
reported false positives now fall below the 0.7 sync threshold while a
battery of genuine matches stays above it.
"""
from __future__ import annotations
import types
from core.text.title_match import titles_plausibly_same
# ── the pure guard ────────────────────────────────────────────────────────
def test_near_identical_passes_even_without_shared_token():
# Single-word typo: no shared token, but char-identical enough.
assert titles_plausibly_same("beleive", "believe", 0.857) is True
def test_punctuation_casing_variants_pass():
assert titles_plausibly_same("humble", "humble", 0.92) is True
def test_shared_significant_word_passes_below_near_identical():
# Moderate char score but a real shared content word.
assert titles_plausibly_same("hello world", "hello there", 0.6) is True
def test_different_songs_sharing_only_substring_rejected():
# #769: "Dani California" vs "Californication" — share the substring
# "californi" (high char ratio) but no whole word.
assert titles_plausibly_same("dani california", "californication", 0.667) is False
def test_different_songs_sharing_only_stopword_rejected():
# #769: "Under The Bridge" vs "Around the World" — share only "the".
assert titles_plausibly_same("under the bridge", "around the world", 0.625) is False
def test_multiword_stopword_only_overlap_rejected():
# Two 2+-word titles sharing only "the" — the #769 shape.
assert titles_plausibly_same("under the bridge", "around the world", 0.625) is False
def test_single_word_titles_defer_to_char_floor():
# Single content word on each side: no "other word" to share, so the gate
# must NOT force-fail — it defers (returns True) and lets the caller's char
# floor decide. This is what protects stylized spellings like "Grey"/"Gray"
# and "Tonite"/"Tonight" from becoming new false negatives.
assert titles_plausibly_same("grey", "gray", 0.75) is True
assert titles_plausibly_same("tonite", "tonight", 0.77) is True
# ...even when the char score is low — the floor, not the gate, rejects it.
assert titles_plausibly_same("numb", "creep", 0.2) is True
def test_all_stopword_side_defers():
# One side is all stopwords -> no word signal -> defer to char floor.
assert titles_plausibly_same("the the", "around the world", 0.5) is True
# ── end-to-end through the real confidence scorer ──────────────────────────
from database.music_database import MusicDatabase # noqa: E402
_THRESHOLD = 0.7 # services/sync_service.py confidence_threshold
class _FakeTrack:
def __init__(self, title, artist):
self.title = title
self.artist_name = artist
self.track_artist = None
def _scorer():
stub = type("S", (), {})()
for m in (
"_calculate_track_confidence", "_string_similarity",
"_normalize_for_comparison", "_clean_track_title_for_comparison",
):
setattr(stub, m, types.MethodType(getattr(MusicDatabase, m), stub))
return stub
# (source_title, library_title, same_artist, should_match)
_BATTERY = [
# genuine matches — must stay matched
("Mr. Brightside", "Mr Brightside", True),
("HUMBLE.", "Humble", True),
("Beleive", "Believe", True), # typo
("In the End", "In The End", True),
("thank u, next", "Thank U Next", True),
("Old Town Road", "Old Town Road (feat. Billy Ray Cyrus)", True),
("bad guy", "bad guy", True),
# different songs by the SAME artist — must be reported missing
("Dani California", "Californication", False), # the reported case
("Under The Bridge", "Around the World", False), # the reported case
("Otherside", "Californication", False),
("Numb", "In the End", False),
("Yellow", "The Scientist", False),
("Seven Nation Army", "Fell in Love with a Girl", False),
]
def test_confidence_battery_separates_real_from_false_matches():
s = _scorer()
artist = "Red Hot Chili Peppers"
misclassified = []
for src, lib, should_match in _BATTERY:
conf = s._calculate_track_confidence(src, artist, _FakeTrack(lib, artist))
matched = conf >= _THRESHOLD
if matched != should_match:
misclassified.append((src, lib, should_match, round(conf, 3)))
assert not misclassified, f"misclassified: {misclassified}"
def test_reported_false_positives_now_below_threshold():
s = _scorer()
a = "Red Hot Chili Peppers"
assert s._calculate_track_confidence("Dani California", a, _FakeTrack("Californication", a)) < _THRESHOLD
assert s._calculate_track_confidence("Under The Bridge", a, _FakeTrack("Around the World", a)) < _THRESHOLD
def test_exact_title_same_artist_still_perfect():
s = _scorer()
a = "Garbage"
conf = s._calculate_track_confidence("Only Happy When It Rains", a,
_FakeTrack("Only Happy When It Rains", a))
assert conf >= 0.99
def test_single_word_spelling_variants_not_regressed():
# The gate must not turn legitimate stylized single-word spellings into
# new "missing" reports (the regression the first cut of this fix had).
# These all matched before #769's gate and must still match.
s = _scorer()
a = "Some Artist"
for src, lib in [("Grey", "Gray"), ("Tonite", "Tonight"),
("4ever", "Forever"), ("Lovin'", "Loving"), ("Colour", "Color")]:
conf = s._calculate_track_confidence(src, a, _FakeTrack(lib, a))
assert conf >= _THRESHOLD, f"{src!r}->{lib!r} regressed to {conf:.3f}"

View file

@ -150,6 +150,51 @@ def test_qbit_login_failure_returns_none() -> None:
assert sess is None
def test_qbit_login_accepts_204_no_content() -> None:
"""qBittorrent 5.2.0+ returns HTTP 204 with an empty body on a successful
login (was HTTP 200 + 'Ok.'). The adapter must treat that as success even
when no SID cookie is visible to us."""
adapter = _qbit_with_config()
fake_session = MagicMock()
fake_session.cookies.get.return_value = None # no SID surfaced
resp = _mock_response(204, text='')
resp.text = ''
fake_session.post.return_value = resp
with patch('core.torrent_clients.qbittorrent.http_requests.Session',
return_value=fake_session):
sess = adapter._ensure_session_sync()
assert sess is not None
def test_qbit_login_accepts_sid_cookie_with_empty_body() -> None:
"""A SID auth cookie is the authoritative success signal regardless of body."""
adapter = _qbit_with_config()
fake_session = MagicMock()
fake_session.cookies.get.return_value = 'SID-abc123'
resp = _mock_response(200, text='')
resp.text = ''
fake_session.post.return_value = resp
with patch('core.torrent_clients.qbittorrent.http_requests.Session',
return_value=fake_session):
sess = adapter._ensure_session_sync()
assert sess is not None
def test_qbit_login_rejects_fails_even_with_stale_cookie() -> None:
"""Bad creds: qBittorrent returns HTTP 200 'Fails.' (not a 4xx). Must fail
even if a stale SID cookie lingers on the session."""
adapter = _qbit_with_config()
fake_session = MagicMock()
fake_session.cookies.get.return_value = 'SID-stale'
resp = _mock_response(200, text='Fails.')
resp.text = 'Fails.'
fake_session.post.return_value = resp
with patch('core.torrent_clients.qbittorrent.http_requests.Session',
return_value=fake_session):
sess = adapter._ensure_session_sync()
assert sess is None
def test_qbit_parse_status_normalises_native_fields() -> None:
adapter = _qbit_with_config()
status = adapter._parse_status({

View file

@ -0,0 +1,52 @@
"""Track Number Repair canonical lookup (#765 Stage 4, read side)."""
from __future__ import annotations
import types
from core.repair_jobs.track_number_repair import _lookup_canonical_from_db
from database.music_database import MusicDatabase
def _ctx(db):
return types.SimpleNamespace(db=db)
def _seed(db, *, with_canonical: bool, file_path: str = "/music/Evolve/01 - Believer.flac"):
conn = db._get_connection()
cur = conn.cursor()
cur.execute("INSERT INTO artists (id, name) VALUES ('art1', 'Imagine Dragons')")
cur.execute("INSERT INTO albums (id, title, artist_id) VALUES ('alb1', 'Evolve', 'art1')")
cur.execute(
"INSERT INTO tracks (id, album_id, artist_id, title, track_number, duration, file_path) "
"VALUES ('t1', 'alb1', 'art1', 'Believer', 1, 204000, ?)",
(file_path,),
)
conn.commit()
conn.close()
if with_canonical:
db.set_album_canonical("alb1", "spotify", "sp_evolve", 0.96)
def test_returns_canonical_when_pinned(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
fp = "/music/Evolve/01 - Believer.flac"
_seed(db, with_canonical=True, file_path=fp)
assert _lookup_canonical_from_db([(fp, "01 - Believer.flac", 1)], _ctx(db)) == ("spotify", "sp_evolve")
def test_none_when_unresolved(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
fp = "/music/Evolve/01 - Believer.flac"
_seed(db, with_canonical=False, file_path=fp)
assert _lookup_canonical_from_db([(fp, "01 - Believer.flac", 1)], _ctx(db)) is None
def test_none_when_file_not_tracked(tmp_path):
db = MusicDatabase(str(tmp_path / "m.db"))
_seed(db, with_canonical=True)
assert _lookup_canonical_from_db([("/some/other/path.flac", "x.flac", 1)], _ctx(db)) is None
def test_none_when_no_db():
assert _lookup_canonical_from_db([("/p.flac", "p.flac", 1)], types.SimpleNamespace(db=None)) is None

View file

@ -177,6 +177,22 @@ def test_build_wishlist_source_context_minimal_batch_skips_album_provenance():
assert "artist_context" not in context
def test_build_wishlist_source_context_uses_source_playlist_ref_for_organize_batches():
batch = {
"playlist_name": "Summer Mix",
"playlist_id": "42",
"source_playlist_ref": "spotifyPlaylistId123",
"mirrored_playlist_id": 42,
"organize_by_playlist": True,
}
context = processing.build_wishlist_source_context(batch)
assert context["playlist_id"] == "spotifyPlaylistId123"
assert context["mirrored_playlist_id"] == 42
assert context["organize_by_playlist"] is True
def test_build_wishlist_source_context_preserves_album_context_for_album_batches():
"""Album batches must carry album_context/artist_context through to the
wishlist row so a later requeue has authoritative routing data instead

View file

@ -40,7 +40,7 @@ logger = setup_logging(_log_level, _log_path)
# App version — single source of truth for backup metadata, system-info, update check, etc.
# Semver: MAJOR.MINOR.PATCH. Bump at each dev→main release.
_SOULSYNC_BASE_VERSION = "2.6.5"
_SOULSYNC_BASE_VERSION = "2.6.6"
def _build_version_string():
"""Append short commit hash to version when available (e.g. 2.35+abc1234)."""
@ -1084,6 +1084,8 @@ def _register_automation_handlers():
get_watchlist_scan_state=lambda: watchlist_scan_state,
run_playlist_discovery_worker=_run_playlist_discovery_worker,
run_sync_task=_run_sync_task,
run_playlist_organize_download=_run_playlist_organize_download,
missing_download_executor=missing_download_executor,
load_sync_status_file=_load_sync_status_file,
get_deezer_client=_get_deezer_client,
parse_youtube_playlist=parse_youtube_playlist,
@ -4133,6 +4135,33 @@ def select_jellyfin_music_library():
logger.error(f"Error setting Jellyfin music library: {e}")
return jsonify({"success": False, "error": str(e)}), 500
@app.route('/api/navidrome/cover/<cover_id>', methods=['GET'])
def navidrome_cover(cover_id):
"""Proxy a Navidrome (Subsonic) cover-art image to the browser.
The sync editor and other modals reference /api/navidrome/cover/<id>,
but no route served it so every Navidrome cover came back blank (#766).
We build the authenticated getCoverArt URL server-side (keeping Subsonic
credentials off the client) and stream it through the shared image cache.
"""
try:
client = media_server_engine.client('navidrome')
if not client:
return '', 404
url = client.build_cover_art_url(cover_id)
if not url:
return '', 404
from core.image_cache import get_image_cache
cached = get_image_cache().get_url(url)
response = send_file(cached.path, mimetype=cached.mime_type, conditional=True)
max_age = int(config_manager.get("image_cache.ttl_seconds", 2592000))
response.headers['Cache-Control'] = f'private, max-age={max_age}'
return response
except Exception as exc:
logger.debug("navidrome cover proxy failed for %s: %s", cover_id, exc)
return '', 502
@app.route('/api/navidrome/music-folders', methods=['GET'])
def get_navidrome_music_folders():
"""Get list of available music folders from Navidrome"""
@ -18185,23 +18214,11 @@ def get_server_playlist_tracks(playlist_id):
pass
break
# Build combined view with two-pass matching (exact then fuzzy)
import re as _re
from difflib import SequenceMatcher
def _norm_title(t):
"""Strip feat./ft., remaster, and edition qualifiers for comparison only."""
# feat./ft. — e.g. (feat. Artist), [ft. Artist]
t = _re.sub(r'\s*[\(\[](?:feat|ft)\.?[^\)\]]*[\)\]]', '', t, flags=_re.IGNORECASE)
# Remaster/Remastered — e.g. (2019 Remaster), (Remastered), (2019 Remastered Version)
t = _re.sub(r'\s*[\(\[](?:\d{4}\s+)?remaster(?:ed)?(?:\s+version)?\s*[\)\]]', '', t, flags=_re.IGNORECASE)
# Edition qualifiers — e.g. (Deluxe Edition), (Special Edition), [Anniversary Edition]
t = _re.sub(r'\s*[\(\[](?:deluxe|special|anniversary|legacy|expanded|limited)(?:\s+edition)?\s*[\)\]]', '', t, flags=_re.IGNORECASE)
return t.lower().strip()
combined = []
used_server_indices = set()
unmatched_source = [] # (index_in_combined, src_dict) for fuzzy second pass
# Reconcile source vs server playlist. Three-pass matcher lifted to
# core.sync.playlist_reconcile (pure + tested) — fixes #768 (YouTube
# "Artist - Title" sources now match, and source_track_id is echoed
# back so manual "Find & add" overrides persist).
from core.sync.playlist_reconcile import reconcile_playlist
# Pass 0: User-confirmed match overrides from sync_match_cache.
# When a user previously picked a local file via "Find & Add",
@ -18218,92 +18235,7 @@ def get_server_playlist_tracks(playlist_id):
lambda src_id: ((_db_for_overrides.read_sync_match_cache(src_id, active_server) or {}).get('server_track_id')),
)
# Pass 1: Exact title match (normalized — strips feat./ft. qualifiers)
for i, src in enumerate(source_tracks):
src_name = src.get('name', '')
src_artist = src.get('artist', '')
if not src_artist and src.get('artists'):
a = src['artists'][0] if src['artists'] else ''
src_artist = a.get('name', a) if isinstance(a, dict) else str(a)
src_entry = {
'name': src_name, 'artist': src_artist,
'album': src.get('album', ''), 'image_url': src.get('image_url', ''),
'duration_ms': src.get('duration_ms', 0), 'position': src.get('position', i),
}
# Override hit — paired by user, skip exact/fuzzy matching.
if i in _override_pairs:
j_override = _override_pairs[i]
used_server_indices.add(j_override)
combined.append({
'source_track': src_entry,
'server_track': server_tracks[j_override],
'match_status': 'matched',
'confidence': 1.0,
'override': True,
})
continue
src_norm = _norm_title(src_name)
best_idx = -1
for j, svr in enumerate(server_tracks):
if j in used_server_indices:
continue
if _norm_title(svr['title']) == src_norm:
best_idx = j
break
if best_idx >= 0:
used_server_indices.add(best_idx)
combined.append({
'source_track': src_entry,
'server_track': server_tracks[best_idx],
'match_status': 'matched',
'confidence': 1.0,
})
else:
idx = len(combined)
combined.append({
'source_track': src_entry,
'server_track': None,
'match_status': 'missing',
'confidence': 0.0,
})
unmatched_source.append((idx, src_entry))
# Pass 2: Fuzzy match on remaining unmatched source tracks (normalized keys)
for combo_idx, src_entry in unmatched_source:
src_key = f"{src_entry['artist']} {_norm_title(src_entry['name'])}".strip()
best_score = 0.0
best_j = -1
for j, svr in enumerate(server_tracks):
if j in used_server_indices:
continue
svr_key = f"{svr['artist']} {_norm_title(svr['title'])}".strip().lower()
score = SequenceMatcher(None, src_key.lower(), svr_key).ratio()
if score > best_score and score >= 0.75:
best_score = score
best_j = j
if best_j >= 0:
used_server_indices.add(best_j)
combined[combo_idx] = {
'source_track': src_entry,
'server_track': server_tracks[best_j],
'match_status': 'matched',
'confidence': round(best_score, 3),
}
# Add server tracks that aren't in the source (extra tracks on server)
for j, svr in enumerate(server_tracks):
if j not in used_server_indices:
combined.append({
'source_track': None,
'server_track': svr,
'match_status': 'extra',
'confidence': 0.0,
})
combined = reconcile_playlist(source_tracks, server_tracks, _override_pairs)
return jsonify({
"success": True,
@ -18490,6 +18422,18 @@ def server_playlist_add_track(playlist_id):
if not new_item:
return jsonify({"success": False, "error": "Track not found on server"}), 404
# Link, don't duplicate: matching an unmatched source to a track
# already in the playlist should only record the override, never
# append a second copy (#768).
if source_track_id:
try:
_existing = {str(it.ratingKey) for it in raw_playlist.items()}
except Exception:
_existing = set()
if str(track_id) in _existing:
_persist_find_and_add_match(source_track_id, active_server, track_id, server_track_title or new_item.title, source_title, source_artist)
return jsonify({"success": True, "message": "Track linked"})
logger.info(f"[ServerPlaylist] Adding track: '{new_item.title}' (ratingKey={new_item.ratingKey}) to playlist '{playlist_name}'")
raw_playlist.addItems([new_item])
@ -18515,24 +18459,30 @@ def server_playlist_add_track(playlist_id):
return jsonify({"success": True, "message": "Track added", "new_playlist_id": new_id})
elif active_server == 'jellyfin' and media_server_engine.client('jellyfin'):
from core.sync.playlist_edit import plan_playlist_add
current_tracks = media_server_engine.client('jellyfin').get_playlist_tracks(playlist_id) or []
track_ids = [str(t.ratingKey) for t in current_tracks]
pos = max(0, min(int(position), len(track_ids))) if position is not None else len(track_ids)
track_ids.insert(pos, track_id)
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in track_ids]
media_server_engine.client('jellyfin').update_playlist(playlist_name, new_track_objs)
# Matching an unmatched source to a track already in the playlist
# is a LINK, not a second copy — don't duplicate it (#768).
plan = plan_playlist_add(track_ids, track_id, is_link=bool(source_track_id), position=position)
if plan['should_insert']:
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in plan['new_ids']]
media_server_engine.client('jellyfin').update_playlist(playlist_name, new_track_objs)
_persist_find_and_add_match(source_track_id, active_server, track_id, server_track_title, source_title, source_artist)
return jsonify({"success": True, "message": "Track added"})
return jsonify({"success": True, "message": "Track linked" if not plan['should_insert'] else "Track added"})
elif active_server == 'navidrome' and media_server_engine.client('navidrome'):
from core.sync.playlist_edit import plan_playlist_add
current_tracks = media_server_engine.client('navidrome').get_playlist_tracks(playlist_id) or []
track_ids = [str(t.ratingKey) for t in current_tracks]
pos = max(0, min(int(position), len(track_ids))) if position is not None else len(track_ids)
track_ids.insert(pos, track_id)
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in track_ids]
media_server_engine.client('navidrome').create_playlist(playlist_name, new_track_objs, playlist_id=playlist_id)
# Matching an unmatched source to a track already in the playlist
# is a LINK, not a second copy — don't duplicate it (#768).
plan = plan_playlist_add(track_ids, track_id, is_link=bool(source_track_id), position=position)
if plan['should_insert']:
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in plan['new_ids']]
media_server_engine.client('navidrome').create_playlist(playlist_name, new_track_objs, playlist_id=playlist_id)
_persist_find_and_add_match(source_track_id, active_server, track_id, server_track_title, source_title, source_artist)
return jsonify({"success": True, "message": "Track added"})
return jsonify({"success": True, "message": "Track linked" if not plan['should_insert'] else "Track added"})
return jsonify({"success": False, "error": f"Unsupported server: {active_server}"}), 400
except Exception as e:
@ -18573,10 +18523,17 @@ def server_playlist_remove_track(playlist_id):
logger.warning(f"[ServerPlaylist] remove-track: playlist not found by id={playlist_id} or name='{playlist_name}'")
return jsonify({"success": False, "error": "Playlist not found"}), 404
# Rebuild without the target track
# Rebuild without ONE copy of the target track — deleting one
# duplicate must not wipe every copy (#768).
current_items = list(raw_playlist.items())
new_items = [item for item in current_items if str(item.ratingKey) != str(remove_track_id)]
if len(new_items) == len(current_items):
new_items = list(current_items)
_removed = False
for _i, _it in enumerate(new_items):
if str(_it.ratingKey) == str(remove_track_id):
del new_items[_i]
_removed = True
break
if not _removed:
return jsonify({"success": False, "error": "Track not found in playlist"}), 404
raw_playlist.delete()
if new_items:
@ -18586,18 +18543,25 @@ def server_playlist_remove_track(playlist_id):
return jsonify({"success": True, "message": "Track removed (playlist now empty)"})
elif active_server == 'jellyfin' and media_server_engine.client('jellyfin'):
from core.sync.playlist_edit import remove_one_occurrence
current_tracks = media_server_engine.client('jellyfin').get_playlist_tracks(playlist_id) or []
new_ids = [str(t.ratingKey) for t in current_tracks if str(t.ratingKey) != str(remove_track_id)]
if len(new_ids) == len(current_tracks):
track_ids = [str(t.ratingKey) for t in current_tracks]
# Remove ONE occurrence, not every copy — duplicates are the same
# track, so deleting one must not wipe them all (#768).
new_ids, removed = remove_one_occurrence(track_ids, remove_track_id)
if not removed:
return jsonify({"success": False, "error": "Track not found in playlist"}), 404
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in new_ids]
media_server_engine.client('jellyfin').update_playlist(playlist_name, new_track_objs)
return jsonify({"success": True, "message": "Track removed"})
elif active_server == 'navidrome' and media_server_engine.client('navidrome'):
from core.sync.playlist_edit import remove_one_occurrence
current_tracks = media_server_engine.client('navidrome').get_playlist_tracks(playlist_id) or []
new_ids = [str(t.ratingKey) for t in current_tracks if str(t.ratingKey) != str(remove_track_id)]
if len(new_ids) == len(current_tracks):
track_ids = [str(t.ratingKey) for t in current_tracks]
# Remove ONE occurrence, not every copy (#768).
new_ids, removed = remove_one_occurrence(track_ids, remove_track_id)
if not removed:
return jsonify({"success": False, "error": "Track not found in playlist"}), 404
new_track_objs = [type('T', (), {'ratingKey': tid, 'title': ''})() for tid in new_ids]
media_server_engine.client('navidrome').create_playlist(playlist_name, new_track_objs, playlist_id=playlist_id)
@ -18788,6 +18752,23 @@ def start_missing_tracks_process(playlist_id):
if playlist_folder_mode:
logger.info(f"[Playlist Folder] Enabled for playlist: '{playlist_name}'")
# Persist organize-by-playlist preference on the mirrored playlist row
try:
profile_id = get_current_profile_id()
db_pref = get_database()
mirrored_pl = db_pref.resolve_mirrored_playlist(
playlist_id,
profile_id=profile_id,
default_source='spotify',
)
if mirrored_pl and mirrored_pl.get('id'):
db_pref.set_mirrored_playlist_organize_by_playlist(
int(mirrored_pl['id']),
bool(playlist_folder_mode),
)
except Exception as pref_err:
logger.debug(f"[Playlist Folder] Could not persist mirrored preference: {pref_err}")
# Limit concurrent analysis processes to prevent resource exhaustion
with tasks_lock:
active_analysis_count = sum(1 for batch in download_batches.values()
@ -18998,7 +18979,7 @@ def _update_and_save_sync_status(playlist_id, playlist_name, playlist_owner, sna
'last_synced': now.isoformat()
}
# Store match counts and track hash for smart-skip on scheduled syncs
for key in ('matched_tracks', 'total_tracks', 'discovered_tracks', 'tracks_hash'):
for key in ('matched_tracks', 'total_tracks', 'discovered_tracks', 'tracks_hash', 'mirror_tracks_hash'):
if key in kwargs:
status[key] = kwargs[key]
sync_statuses[playlist_id] = status
@ -19189,11 +19170,12 @@ def get_playlist_tracks(playlist_id):
time.sleep(0.5)
if results is None:
# Both attempts failed (often a 403 on followed playlists) — fall back
# to the public embed scraper as a last resort (capped at ~100 tracks).
logger.warning(f"Playlist items unavailable after retry ({items_err}), trying public embed scraper")
# to the no-auth public path (full track list via anonymous token,
# embed scraper if that fails).
logger.warning(f"Playlist items unavailable after retry ({items_err}), trying public fetch")
try:
from core.spotify_public_scraper import scrape_spotify_embed
embed_data = scrape_spotify_embed('playlist', playlist_id)
from core.spotify_public_scraper import fetch_spotify_public
embed_data = fetch_spotify_public('playlist', playlist_id)
if embed_data and not embed_data.get('error') and embed_data.get('tracks'):
for t in embed_data['tracks']:
artists = t.get('artists', [])
@ -22308,15 +22290,15 @@ def parse_spotify_public_endpoint():
if not url:
return jsonify({"error": "Spotify URL is required"}), 400
from core.spotify_public_scraper import parse_spotify_url, scrape_spotify_embed
from core.spotify_public_scraper import parse_spotify_url, fetch_spotify_public
parsed = parse_spotify_url(url)
if not parsed:
return jsonify({"error": "Invalid Spotify URL. Please use a playlist or album link from open.spotify.com"}), 400
logger.info(f"Scraping public Spotify {parsed['type']}: {parsed['id']}")
logger.info(f"Fetching public Spotify {parsed['type']}: {parsed['id']}")
result = scrape_spotify_embed(parsed['type'], parsed['id'])
result = fetch_spotify_public(parsed['type'], parsed['id'])
if 'error' in result:
return jsonify(result), 400
@ -23612,14 +23594,44 @@ def _build_sync_deps():
update_and_save_sync_status=_update_and_save_sync_status,
sync_states=sync_states,
sync_lock=sync_lock,
process_wishlist_automatically=_process_wishlist_automatically,
run_playlist_organize_download=_run_playlist_organize_download,
is_wishlist_actually_processing=is_wishlist_actually_processing,
)
def _run_sync_task(playlist_id, playlist_name, tracks_json, automation_id=None, profile_id=1, playlist_image_url='', sync_mode='replace'):
def _run_sync_task(
playlist_id,
playlist_name,
tracks_json,
automation_id=None,
profile_id=1,
playlist_image_url='',
sync_mode='replace',
skip_wishlist_add=False,
):
return _discovery_sync.run_sync_task(
playlist_id, playlist_name, tracks_json, automation_id, profile_id, playlist_image_url,
_build_sync_deps(),
sync_mode=sync_mode,
skip_wishlist_add=skip_wishlist_add,
)
def _run_playlist_organize_download(mirrored_playlist_id, automation_id=None, profile_id=None):
"""Start a playlist-folder missing-tracks batch for automation / pipeline."""
from core.playlists.organize_download import run_playlist_organize_download
if profile_id is None:
profile_id = get_current_profile_id()
return run_playlist_organize_download(
_automation_deps,
mirrored_playlist_id=int(mirrored_playlist_id),
profile_id=profile_id,
get_batch_max_concurrent=_get_batch_max_concurrent,
run_full_missing_tracks_process=_run_full_missing_tracks_process,
record_sync_history_start=_record_sync_history_start,
detect_sync_source=_downloads_history.detect_sync_source,
)
@ -26275,6 +26287,17 @@ def get_discover_similar_artists():
if not similar_artists:
return jsonify({"success": True, "artists": [], "source": active_source, "count": 0})
# Explainability: resolve which of the user's OWN artists point to each
# recommendation, so the UI can show "because you have X, Y, Z".
try:
sources_by_name = database.get_recommendation_sources(
[a.similar_artist_name for a in similar_artists],
profile_id=get_current_profile_id(),
)
except Exception as e:
logger.debug("recommendation-sources lookup failed: %s", e)
sources_by_name = {}
# Artists already filtered by source in SQL
result_artists = []
for artist in similar_artists:
@ -26305,6 +26328,10 @@ def get_discover_similar_artists():
artist_data["genres"] = artist.genres[:3]
if artist.popularity:
artist_data["popularity"] = artist.popularity
# "because you have X, Y, Z" — the artists of yours that point here
because = sources_by_name.get(artist.similar_artist_name)
if because:
artist_data["because"] = because
result_artists.append(artist_data)
logger.info(
@ -28321,6 +28348,19 @@ def get_artist_map_explore():
return _artists_map_get_artist_map_explore()
@app.route('/api/discover/artist-map/perf', methods=['POST'])
def log_artist_map_perf():
"""Debug sink: the artist-map frontend POSTs its render timings here (toggled
with 'd' on the map) so they land in app.log the on-canvas overlay text
can't be copied. Used to find the real drag/zoom bottleneck."""
try:
data = request.get_json(silent=True) or {}
logger.info("[ARTMAP-PERF] %s", json.dumps(data, ensure_ascii=False))
except Exception as e:
logger.debug("artist-map perf log failed: %s", e)
return ('', 204)
@app.route('/api/discover/build-playlist/search-artists', methods=['GET'])
def search_artists_for_playlist():
"""Search for artists to use as seeds for custom playlist building"""
@ -31664,6 +31704,55 @@ def update_mirrored_playlist_source_ref_endpoint(playlist_id):
return jsonify({"error": str(e)}), 500
@app.route('/api/mirrored-playlists/<int:playlist_id>/preferences', methods=['PATCH'])
def update_mirrored_playlist_preferences_endpoint(playlist_id):
"""Update per-playlist download preferences (e.g. organize by playlist folder)."""
try:
data = request.get_json() or {}
if 'organize_by_playlist' not in data:
return jsonify({"error": "organize_by_playlist is required"}), 400
database = get_database()
playlist = database.get_mirrored_playlist(playlist_id)
if not playlist:
return jsonify({"error": "Playlist not found"}), 404
enabled = bool(data.get('organize_by_playlist'))
ok = database.set_mirrored_playlist_organize_by_playlist(playlist_id, enabled)
if not ok:
return jsonify({"error": "Failed to update preferences"}), 500
updated = database.get_mirrored_playlist(playlist_id) or {}
return jsonify({"success": True, "playlist": updated})
except Exception as e:
logger.error(f"Error updating mirrored playlist preferences: {e}")
return jsonify({"error": str(e)}), 500
@app.route('/api/mirrored-playlists/resolve', methods=['GET'])
def resolve_mirrored_playlist_endpoint():
"""Resolve mirrored playlist by numeric id or upstream source id (e.g. Spotify playlist id)."""
try:
playlist_ref = request.args.get('ref') or request.args.get('playlist_id')
source = request.args.get('source', 'spotify')
profile_id = get_current_profile_id()
if not playlist_ref:
return jsonify({"error": "ref or playlist_id query param required"}), 400
database = get_database()
playlist = database.resolve_mirrored_playlist(
playlist_ref,
profile_id=profile_id,
default_source=source,
)
if not playlist:
return jsonify({"found": False, "playlist": None})
return jsonify({"found": True, "playlist": playlist})
except Exception as e:
logger.error(f"Error resolving mirrored playlist: {e}")
return jsonify({"error": str(e)}), 500
def _playlist_pipeline_state_key(playlist_id):
return f"mirrored_{int(playlist_id)}"
@ -32913,6 +33002,27 @@ except Exception as e:
amazon_worker = None
# --- Similar Artists Worker Initialization ---
# Fills the similar_artists table for LIBRARY artists (the watchlist scanner only
# covers watchlist artists). Runs by default (like the metadata workers); it
# self-paces (~3s/artist) and backs off on MusicMap outages. Respects a saved
# pause choice across restarts.
similar_artists_worker = None
try:
from core.similar_artists_worker import SimilarArtistsWorker
similar_artists_db = MusicDatabase()
similar_artists_worker = SimilarArtistsWorker(database=similar_artists_db)
similar_artists_worker.start()
if config_manager.get('similar_artists_enrichment_paused', False):
similar_artists_worker.pause()
logger.info("Similar Artists worker initialized (paused — restored from config)")
else:
logger.info("Similar Artists worker initialized and started")
except Exception as e:
logger.error(f"Similar Artists worker initialization failed: {e}")
similar_artists_worker = None
# ================================================================================================
# SPOTIFY ENRICHMENT INTEGRATION
# ================================================================================================
@ -34677,12 +34787,19 @@ _register_enrichment_services([
worker_getter=lambda: amazon_worker,
config_paused_key='amazon_enrichment_paused',
),
_EnrichmentService(
id='similar_artists', display_name='Similar Artists',
worker_getter=lambda: similar_artists_worker,
config_paused_key='similar_artists_enrichment_paused',
),
])
_configure_enrichment_api(
config_set=lambda key, value: config_manager.set(key, value),
config_get=lambda key, default=None: config_manager.get(key, default),
auto_paused_discard=lambda token: _download_auto_paused.discard(token),
yield_override_add=lambda token: _download_yield_override.add(token),
db_getter=get_database,
)
app.register_blueprint(_create_enrichment_blueprint())
@ -34757,6 +34874,7 @@ def _emit_enrichment_status_loop():
'tidal-enrichment': lambda: tidal_enrichment_worker,
'qobuz-enrichment': lambda: qobuz_enrichment_worker,
'amazon-enrichment': lambda: amazon_worker,
'similar_artists': lambda: similar_artists_worker,
'hydrabase': lambda: hydrabase_worker,
'soulid': lambda: soulid_worker,
'listening-stats': lambda: listening_stats_worker,

View file

@ -565,6 +565,28 @@
</div>
</div>
</div>
<!-- Similar Artists (MusicMap) Enrichment Status Icon -->
<div class="similar-artists-enrich-button-container">
<button class="similar-artists-enrich-button" id="similar-artists-enrich-button" title="Similar Artists (MusicMap) Enrichment">
<img src="https://www.music-map.com/elements/objects/og_logo.png"
alt="Similar Artists" class="similar-artists-enrich-logo">
<div class="similar-artists-enrich-spinner"></div>
</button>
<div class="similar-artists-enrich-tooltip" id="similar-artists-enrich-tooltip">
<div class="similar-artists-enrich-tooltip-content">
<div class="similar-artists-enrich-tooltip-header">Similar Artists Enrichment</div>
<div class="similar-artists-enrich-tooltip-body" id="similar-artists-enrich-tooltip-body">
<div class="tooltip-status">Status: <span
id="similar-artists-enrich-tooltip-status">Idle</span>
</div>
<div class="tooltip-current" id="similar-artists-enrich-tooltip-current">No active matches
</div>
<div class="tooltip-progress" id="similar-artists-enrich-tooltip-progress">Progress: 0 / 0
</div>
</div>
</div>
</div>
</div>
<!-- Hydrabase P2P Mirror Status Icon -->
<div class="hydrabase-button-container" id="hydrabase-button-container" style="display: none;">
<button class="hydrabase-button" id="hydrabase-button" title="Hydrabase P2P Mirror">
@ -625,6 +647,13 @@
</div>
</div>
</div>
<!-- Manage Enrichment Workers — opens the full management modal -->
<button class="em-manage-btn" id="manage-enrichment-btn"
title="Manage enrichment workers — stats, unmatched items, manual matching"
onclick="openEnrichmentManager()">
<span class="em-manage-btn-icon"><img src="/static/trans2.png" alt="SoulSync" class="em-manage-btn-logo"></span>
<span class="em-manage-btn-label">Manage Workers</span>
</button>
</div>
<!-- Watchlist / Wishlist quick-nav (top-right corner) -->
<div class="header-quick-nav">
@ -3038,6 +3067,25 @@
<div class="artist-map-search-results" id="artist-map-search-results"></div>
</div>
<!-- Recommended For You Section (similar-artists graph) -->
<div class="discover-section" id="recommended-artists-section" style="display: none;">
<div class="discover-section-header">
<div>
<h2 class="discover-section-title">Recommended For You</h2>
<p class="discover-section-subtitle">Artists similar to ones across your library — not yet on your watchlist</p>
</div>
<div class="discover-section-actions">
<button class="btn btn--sm btn--secondary ya-header-btn ya-viewall-btn" onclick="openRecommendedArtistsModal()">
<span>View All</span>
<svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5"><path d="M5 12h14"/><path d="M12 5l7 7-7 7"/></svg>
</button>
</div>
</div>
<div class="discover-carousel" id="recommended-artists-carousel">
<!-- Populated by JS -->
</div>
</div>
<!-- Your Artists Section -->
<div class="discover-section" id="your-artists-section" style="display: none;">
<div class="discover-section-header">
@ -8032,6 +8080,7 @@
<script src="{{ url_for('static', filename='discover-section-controller.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='discover.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='enrichment.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='enrichment-manager.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='stats-automations.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='auto-sync.js', v=static_v) }}"></script>
<script src="{{ url_for('static', filename='pages-extra.js', v=static_v) }}"></script>

View file

@ -1,8 +1,14 @@
import { describe, expect, it } from 'vitest';
import { describe, expect, it, vi } from 'vitest';
import { apiClient } from '@/app/api-client';
import { HttpResponse, http, server } from '@/test/msw';
import { approveAutoImportResult, rejectAutoImportResult } from './-import.api';
import {
approveAutoImportResult,
processImportAlbumTrack,
processImportSingleFile,
rejectAutoImportResult,
} from './-import.api';
const softFailureMessage = 'Item not found or not pending review';
@ -26,4 +32,21 @@ describe('import api', () => {
await expect(approveAutoImportResult(17)).rejects.toThrow(softFailureMessage);
await expect(rejectAutoImportResult(18)).rejects.toThrow(softFailureMessage);
});
it('#772: import-process calls use a long timeout, not ky default 10s', async () => {
// Per-track import does heavy server-side enrichment (60-90s+); the default
// 10s timeout aborted it client-side -> progress bar stuck + "Failed" while
// files imported. These calls must pass an explicit long timeout.
const ok = { json: async () => ({ success: true, processed: 1, total: 1, errors: [] }) };
const spy = vi.spyOn(apiClient, 'post').mockReturnValue(ok as never);
await processImportAlbumTrack({ album: {} as never, match: {} as never });
await processImportSingleFile({});
expect(spy).toHaveBeenCalledTimes(2);
for (const call of spy.mock.calls) {
expect(call[1]).toMatchObject({ timeout: 300_000 });
}
spy.mockRestore();
});
});

View file

@ -18,6 +18,15 @@ import type {
export const IMPORT_QUERY_KEY = ['import'] as const;
// Per-track import does heavy synchronous enrichment server-side (metadata
// lookups, art, lyrics) and can legitimately take 60-90s/track — much longer
// when external sources are degraded. ky's default 10s timeout aborts those
// requests client-side even though the server completes the import (200),
// which left the progress bar stuck at 0 and showing "Failed" while files
// imported fine (#772). Give the import-process calls a generous bound so the
// responses actually arrive and the bar advances. Scoped to import only.
const IMPORT_REQUEST_TIMEOUT_MS = 300_000; // 5 min/track
export async function fetchImportStagingFiles(): Promise<ImportStagingFilesPayload> {
return readJson<ImportStagingFilesPayload>(apiClient.get('import/staging/files'));
}
@ -71,6 +80,7 @@ export async function processImportAlbumTrack(input: {
album: input.album,
matches: [input.match],
},
timeout: IMPORT_REQUEST_TIMEOUT_MS,
}),
);
}
@ -92,6 +102,7 @@ export async function processImportSingleFile(file: unknown): Promise<ImportProc
json: {
files: [file],
},
timeout: IMPORT_REQUEST_TIMEOUT_MS,
}),
);
}

View file

@ -675,6 +675,7 @@ function autoSyncWeeklyCardHtml(playlist, schedule) {
${_esc(playlist.name)}
</div>
<div class="auto-sync-scheduled-meta">${_esc(autoSyncSourceLabel(playlist.source))} &middot; ${playlist.track_count || 0} tracks</div>
${autoSyncOrganizeToggleHtml(playlist)}
<div class="auto-sync-scheduled-timing">
<span>${_esc(label)}</span>
<small>${_esc(tz)}</small>
@ -1576,6 +1577,34 @@ function autoSyncAutomationCardHtml(auto, playlists) {
`;
}
function autoSyncOrganizeToggleHtml(playlist) {
const checked = playlist.organize_by_playlist ? 'checked' : '';
return `
<label class="auto-sync-organize-toggle" onclick="event.stopPropagation();" title="Download missing tracks into a playlist-named folder (artist - track)">
<input type="checkbox" ${checked} onchange="setAutoSyncOrganizeByPlaylist(${playlist.id}, this.checked)">
<span>Organize by playlist</span>
</label>
`;
}
async function setAutoSyncOrganizeByPlaylist(playlistId, enabled) {
try {
const res = await fetch(`/api/mirrored-playlists/${playlistId}/preferences`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ organize_by_playlist: !!enabled }),
});
const data = await res.json();
if (!res.ok || data.error) throw new Error(data.error || 'Failed to update preference');
const pl = _autoSyncScheduleState.playlists.find(p => parseInt(p.id, 10) === parseInt(playlistId, 10));
if (pl) pl.organize_by_playlist = !!enabled;
showToast(enabled ? 'Auto-Sync will use playlist folders' : 'Auto-Sync will use standard download layout', 'success');
} catch (err) {
showToast(`Error: ${err.message}`, 'error');
await refreshAutoSyncScheduleModal();
}
}
function autoSyncScheduledCardHtml(playlist, schedule) {
const enabled = schedule?.enabled !== false;
const nextLabel = schedule?.next_run ? autoSyncNextRunLabel(schedule.next_run) : '';
@ -1592,6 +1621,7 @@ function autoSyncScheduledCardHtml(playlist, schedule) {
${_esc(playlist.name)}
</div>
<div class="auto-sync-scheduled-meta">${_esc(autoSyncSourceLabel(playlist.source))} &middot; ${playlist.track_count || 0} tracks</div>
${autoSyncOrganizeToggleHtml(playlist)}
<div class="auto-sync-scheduled-timing">
<span>${_esc(autoSyncIntervalLabel(schedule?.hours || 24))}</span>
${nextLabel ? `<small>${_esc(nextLabel)}</small>` : ''}

Some files were not shown because too many files have changed in this diff Show more