danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Lift _process_watchlist_scan_automatically to core/watchlist/auto_scan.py

Browse source 
Pulls the 390-line watchlist auto-scan orchestrator out of `web_server.py`
into a new `core/watchlist/` package. Watchlist (followed-artists scanner
that finds new releases) is a separate domain from kettui's wishlist
(failed-download retry queue), so this lift does not overlap with the
ongoing PR400-style extractions.

What `process_watchlist_scan_automatically` does:

1. Smart stuck-detection guard before acquiring the timer lock —
   prevents deadlock when a previous scan flag is dangling past the
   2-hour timeout.
2. Inside the timer lock: re-check + set the active scan flag with the
   current timestamp.
3. Per-profile expansion (or single-profile when manually triggered):
   - Watchlist count check + Spotify auth gate.
   - Backfill missing artist images.
4. Initialize a fresh `watchlist_scan_state` dict (the deps property
   setter rebinds the web_server.py module-level name so external
   sentinel checks via id() comparison still detect the swap).
5. Pause enrichment workers, then call
   `WatchlistScanner.scan_watchlist_artists` with a per-event progress
   callback that translates scanner events into automation log lines.
6. Post-scan steps (skipped if the scan was cancelled mid-flight):
   - Populate discovery pool from similar artists (per-profile).
   - Refresh ListenBrainz playlists.
   - Update current seasonal playlist (weekly cadence).
   - Generate Last.fm radio playlists.
   - Sync Spotify library cache.
   - Activity feed entry + automation_engine.emit('watchlist_scan_completed').
7. On exception: mark state['status']='error', re-raise so the
   automation wrapper records the failure.
8. Finally: resume enrichment workers, clear the scanner's rescan
   cutoff, reset the auto-scanning flag.

Strict 1:1 byte parity:
The original mutated `watchlist_auto_scanning`,
`watchlist_auto_scanning_timestamp`, and `watchlist_scan_state` as
module globals (with a leading `global` decl). Here those names are
exposed through the `WatchlistAutoScanDeps` proxy as Python properties
so the lifted body keeps the same `name = value` / `name[key] = value`
shape. Property setters fan writes back to web_server.py via callback
pairs.

Diff vs original after `deps.X` → global X normalization is **zero
differences** apart from the dropped `global` declaration line — Python
doesn't need it once the names are property accesses on the deps object.
390 lines orig = 390 lines lifted, byte-identical body otherwise.

Dependencies injected via `WatchlistAutoScanDeps` (15 fields total) —
Flask app, spotify_client, automation_engine, watchlist_timer_lock, plus
5 callable helpers and 6 property delegate callbacks (paired
get/set for each of the three globals).

Tests: 11 new under tests/watchlist/test_auto_scan.py covering
stuck-detection guard, race-check inside lock, zero-watchlist short-
circuit, unauthenticated Spotify gate, successful scan with all post-
scan steps, automation event emission, activity feed logging,
cancellation mid-scan skipping post-steps, profile-scoped trigger,
flag reset in finally, rescan cutoff clear in finally.

Full suite: 1319 passing (was 1308). Ruff clean.
This commit is contained in:
Broque Thomas 2026-04-29 07:57:01 -07:00
parent 962dbf4558
commit 2b2003ba4c
5 changed files with 842 additions and 379 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
core/watchlist/__init__.py Normal file
View file

@ -0,0 +1,6 @@
"""Watchlist automation package.
Houses the auto-scan orchestrator that runs on a timer / automation trigger.
The underlying `WatchlistScanner` class still lives at `core/watchlist_scanner.py`
(separate file) and is imported by the auto_scan module.
"""

471
core/watchlist/auto_scan.py Normal file
View file

@ -0,0 +1,471 @@
"""Background worker for the automatic watchlist artist scan.
`process_watchlist_scan_automatically(automation_id, profile_id, deps)` is
the orchestrator the automation engine schedules (or the user manually
triggers) to scan watchlisted artists for new releases. Strict 1:1 lift
of the original web_server.py helper.
Parity note:
The original mutated `watchlist_auto_scanning`,
`watchlist_auto_scanning_timestamp`, and `watchlist_scan_state` as
module globals (with a leading `global` decl). Here those names are
exposed through the `WatchlistAutoScanDeps` proxy as Python properties,
so the lifted body keeps the same `name = value` / `name[key] = value`
shape. The property setters fan writes back to web_server.py via
callback pairs so external sentinel checks (id() comparison in the
automation handler) still detect a state-dict swap.
The only line that drops out of byte parity is the original `global`
declaration itself — Python doesn't need it here since the names are
now `deps.X` attribute accesses.
"""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Callable
logger = logging.getLogger(__name__)
@dataclass
class WatchlistAutoScanDeps:
"""Bundle of cross-cutting deps the watchlist auto-scan needs.
The three watchlist globals (auto_scanning flag, timestamp, and
scan_state dict) are exposed as Python properties so the lifted
function body can write to them with `name = value` syntax —
the property setters fan the writes back to web_server.py.
"""
app: Any # Flask app for app_context()
spotify_client: Any
automation_engine: Any
watchlist_timer_lock: Any # threading.Lock
is_watchlist_actually_scanning: Callable[[], bool]
pause_enrichment_workers: Callable[[str], dict]
resume_enrichment_workers: Callable[[dict, str], None]
update_automation_progress: Callable
add_activity_item: Callable
_get_auto_scanning: Callable[[], bool]
_set_auto_scanning: Callable[[bool], None]
_get_auto_scanning_timestamp: Callable[[], float]
_set_auto_scanning_timestamp: Callable[[float], None]
_get_watchlist_scan_state: Callable[[], dict]
_set_watchlist_scan_state: Callable[[dict], None]
@property
def watchlist_auto_scanning(self) -> bool:
return self._get_auto_scanning()
@watchlist_auto_scanning.setter
def watchlist_auto_scanning(self, value: bool) -> None:
self._set_auto_scanning(value)
@property
def watchlist_auto_scanning_timestamp(self) -> float:
return self._get_auto_scanning_timestamp()
@watchlist_auto_scanning_timestamp.setter
def watchlist_auto_scanning_timestamp(self, value: float) -> None:
self._set_auto_scanning_timestamp(value)
@property
def watchlist_scan_state(self) -> dict:
return self._get_watchlist_scan_state()
@watchlist_scan_state.setter
def watchlist_scan_state(self, value: dict) -> None:
self._set_watchlist_scan_state(value)
def process_watchlist_scan_automatically(automation_id=None, profile_id=None, deps: WatchlistAutoScanDeps = None):
"""Main automatic scanning logic that runs in background thread.
Args:
automation_id: ID of the automation triggering this scan
profile_id: If provided, only scan this profile's watchlist (manual trigger).
If None, scan all profiles (scheduled automation).
"""
scope_label = f"profile {profile_id}" if profile_id else "all profiles"
logger.info(f"[Auto-Watchlist] Timer triggered - starting automatic watchlist scan ({scope_label})...")
_ew_state = {}
try:
# CRITICAL FIX: Use smart stuck detection BEFORE acquiring lock
# This prevents deadlock and handles stuck flags (2-hour timeout)
if deps.is_watchlist_actually_scanning():
logger.info("[Auto-Watchlist] Already scanning (verified with stuck detection), skipping.")
return
with deps.watchlist_timer_lock:
# Re-check inside lock to handle race conditions
if deps.watchlist_auto_scanning:
logger.info("[Auto-Watchlist] Already scanning (race condition check), skipping.")
return
# Set flag and timestamp
import time
deps.watchlist_auto_scanning = True
deps.watchlist_auto_scanning_timestamp = time.time()
logger.info(f"[Auto-Watchlist] Flag set at timestamp {deps.watchlist_auto_scanning_timestamp}")
# Use app context for database operations
with deps.app.app_context():
from core.watchlist_scanner import get_watchlist_scanner
from database.music_database import get_database
database = get_database()
# Determine which profiles to scan
if profile_id:
# Manual trigger — scan only the triggering profile
scan_profiles = [{'id': profile_id}]
else:
# Scheduled automation — scan all profiles
scan_profiles = database.get_all_profiles()
watchlist_count = sum(database.get_watchlist_count(profile_id=p['id']) for p in scan_profiles)
profile_label = f"profile {profile_id}" if profile_id else f"{len(scan_profiles)} profiles"
logger.info(f"[Auto-Watchlist] Watchlist count check: {watchlist_count} artists found ({profile_label})")
if watchlist_count == 0:
logger.warning("ℹ️ [Auto-Watchlist] No artists in watchlist for auto-scanning.")
with deps.watchlist_timer_lock:
deps.watchlist_auto_scanning = False
deps.watchlist_auto_scanning_timestamp = 0
return
if not deps.spotify_client or not deps.spotify_client.is_authenticated():
logger.info("ℹ️ [Auto-Watchlist] Spotify client not available or not authenticated.")
with deps.watchlist_timer_lock:
deps.watchlist_auto_scanning = False
deps.watchlist_auto_scanning_timestamp = 0
return
logger.info(f"[Auto-Watchlist] Found {watchlist_count} artists in watchlist, starting automatic scan...")
deps.update_automation_progress(automation_id, progress=5, phase='Loading watchlist',
log_line=f'{watchlist_count} artists ({profile_label})', log_type='info')
# Get list of artists to scan
watchlist_artists = []
for p in scan_profiles:
watchlist_artists.extend(database.get_watchlist_artists(profile_id=p['id']))
scanner = get_watchlist_scanner(deps.spotify_client)
all_profiles = scan_profiles # Used later for discovery pool population
for p in scan_profiles:
try:
filled = scanner.backfill_watchlist_artist_images(p['id'])
if filled:
logger.info(f"Backfilled {filled} watchlist artist images for profile {p['id']}")
except Exception as img_err:
logger.error(f"Image backfill error for profile {p['id']}: {img_err}")
# Initialize detailed progress tracking (same as manual scan)
deps.watchlist_scan_state = {
'status': 'scanning',
'started_at': datetime.now(),
'total_artists': len(watchlist_artists),
'current_artist_index': 0,
'current_artist_name': '',
'current_artist_image_url': '',
'current_phase': 'starting',
'albums_to_check': 0,
'albums_checked': 0,
'current_album': '',
'current_album_image_url': '',
'current_track_name': '',
'tracks_found_this_scan': 0,
'tracks_added_this_scan': 0,
'recent_wishlist_additions': [],
'results': [],
'summary': {},
'error': None,
'cancel_requested': False
}
scan_results = []
# Pause enrichment workers during scan to reduce API contention
_ew_state = deps.pause_enrichment_workers('auto-watchlist scan')
def _scan_progress(event_type, payload):
if event_type == 'scan_started':
deps.update_automation_progress(
automation_id,
progress=5,
phase='Loading watchlist',
log_line=f"{len(watchlist_artists)} artists ({profile_label})",
log_type='info',
)
elif event_type == 'artist_started':
total = max(1, payload.get('total_artists', len(watchlist_artists)))
idx = payload.get('artist_index', 1)
artist_name = payload.get('artist_name', '')
pct = 5 + ((idx - 1) / total) * 90
deps.update_automation_progress(
automation_id,
progress=pct,
phase=f'Scanning: {artist_name} ({idx}/{total})',
current_item=artist_name,
processed=idx - 1,
total=total,
)
elif event_type == 'artist_completed':
artist_name = payload.get('artist_name', '')
new_tracks = payload.get('new_tracks_found', 0)
added = payload.get('tracks_added_to_wishlist', 0)
if new_tracks > 0:
deps.update_automation_progress(
automation_id,
log_line=f'{artist_name} — {new_tracks} new, {added} added',
log_type='success',
)
else:
deps.update_automation_progress(
automation_id,
log_line=f'{artist_name} — no new tracks',
log_type='skip',
)
elif event_type == 'artist_error':
artist_name = payload.get('artist_name', '')
error_message = payload.get('error_message', 'error')
deps.update_automation_progress(
automation_id,
log_line=f'{artist_name} — error: {error_message[:60]}',
log_type='error',
)
elif event_type == 'cancelled':
deps.update_automation_progress(
automation_id,
progress=100,
phase='Cancelled by user',
log_line='Scan cancelled by user',
log_type='warning',
)
elif event_type == 'scan_completed':
deps.update_automation_progress(
automation_id,
progress=95,
phase='Scan complete',
log_line=(
f"Scanned {payload.get('successful_scans', 0)} artists — "
f"{payload.get('new_tracks_found', 0)} new tracks, "
f"{payload.get('tracks_added_to_wishlist', 0)} added to wishlist"
),
log_type='success' if payload.get('new_tracks_found', 0) > 0 else 'info',
)
scan_results = scanner.scan_watchlist_artists(
watchlist_artists,
scan_state=deps.watchlist_scan_state,
progress_callback=_scan_progress,
cancel_check=lambda: deps.watchlist_scan_state.get('cancel_requested'),
)
# Update state with results (skip if cancelled — already set by cancel handler)
was_cancelled = deps.watchlist_scan_state.get('cancel_requested', False)
if not was_cancelled:
successful_scans = [r for r in scan_results if r.success]
total_new_tracks = sum(r.new_tracks_found for r in successful_scans)
total_added_to_wishlist = sum(r.tracks_added_to_wishlist for r in successful_scans)
deps.watchlist_scan_state['status'] = 'completed'
deps.watchlist_scan_state['results'] = scan_results
deps.watchlist_scan_state['completed_at'] = datetime.now()
deps.watchlist_scan_state['summary'] = {
'total_artists': len(scan_results),
'successful_scans': len(successful_scans),
'new_tracks_found': total_new_tracks,
'tracks_added_to_wishlist': total_added_to_wishlist
}
logger.info(f"Automatic watchlist scan completed: {len(successful_scans)}/{len(scan_results)} artists scanned successfully")
logger.info(f"Found {total_new_tracks} new tracks, added {total_added_to_wishlist} to wishlist")
deps.update_automation_progress(automation_id, progress=95, phase='Scan complete',
log_line=f'Scanned {len(successful_scans)} artists — {total_new_tracks} new tracks, {total_added_to_wishlist} added to wishlist',
log_type='success' if total_new_tracks > 0 else 'info')
else:
total_new_tracks = deps.watchlist_scan_state.get('summary', {}).get('new_tracks_found', 0)
total_added_to_wishlist = deps.watchlist_scan_state.get('summary', {}).get('tracks_added_to_wishlist', 0)
logger.warning("Automatic watchlist scan cancelled — skipping post-scan steps")
# Post-scan steps — skip if cancelled
if not was_cancelled:
# Populate discovery pool from similar artists (per-profile)
logger.info("Starting discovery pool population...")
deps.watchlist_scan_state['current_phase'] = 'populating_discovery_pool'
deps.update_automation_progress(automation_id, progress=96, phase='Populating discovery pool',
log_line='Building discovery pool from similar artists...', log_type='info')
try:
def _discovery_progress(event_type, message):
if event_type == 'artist':
deps.update_automation_progress(automation_id, phase=f'Discovery pool: {message}',
log_line=message, log_type='info',
current_item=message)
elif event_type == 'phase':
deps.update_automation_progress(automation_id, phase=message,
log_line=message, log_type='info')
elif event_type == 'success':
deps.update_automation_progress(automation_id,
log_line=message, log_type='success')
elif event_type == 'skip':
deps.update_automation_progress(automation_id,
log_line=message, log_type='info')
for p in all_profiles:
scanner.populate_discovery_pool(profile_id=p['id'], progress_callback=_discovery_progress)
logger.info("Discovery pool population complete")
except Exception as discovery_error:
logger.error(f"Error populating discovery pool: {discovery_error}")
import traceback
traceback.print_exc()
deps.update_automation_progress(automation_id,
log_line=f'Discovery pool error: {discovery_error}', log_type='error')
# Update ListenBrainz playlists cache
logger.info("Starting ListenBrainz playlists update...")
deps.watchlist_scan_state['current_phase'] = 'updating_listenbrainz'
deps.update_automation_progress(automation_id, progress=97, phase='Updating ListenBrainz',
log_line='Fetching ListenBrainz playlists...', log_type='info')
try:
from core.listenbrainz_manager import ListenBrainzManager
db = get_database()
db_path = str(db.database_path)
lb_profiles = db.get_profiles_with_listenbrainz()
if lb_profiles:
for lb_prof in lb_profiles:
lb_manager = ListenBrainzManager(db_path, profile_id=lb_prof['id'], token=lb_prof['token'], base_url=lb_prof['base_url'])
lb_result = lb_manager.update_all_playlists()
if lb_result.get('success'):
summary = lb_result.get('summary', {})
logger.info(f"ListenBrainz update complete for profile {lb_prof['id']}: {summary}")
deps.update_automation_progress(automation_id,
log_line=f'ListenBrainz (profile {lb_prof["id"]}): playlists updated', log_type='success')
else:
lb_manager = ListenBrainzManager(db_path)
lb_result = lb_manager.update_all_playlists()
if lb_result.get('success'):
summary = lb_result.get('summary', {})
logger.info(f"ListenBrainz update complete (global): {summary}")
deps.update_automation_progress(automation_id,
log_line='ListenBrainz: playlists updated', log_type='success')
else:
logger.error(f"ListenBrainz update had issues: {lb_result.get('error', 'Unknown error')}")
deps.update_automation_progress(automation_id,
log_line=f'ListenBrainz: {lb_result.get("error", "Unknown error")}', log_type='error')
except Exception as lb_error:
logger.error(f"Error updating ListenBrainz: {lb_error}")
import traceback
traceback.print_exc()
deps.update_automation_progress(automation_id,
log_line=f'ListenBrainz error: {lb_error}', log_type='error')
# Update current seasonal playlist (weekly refresh)
logger.info("Starting seasonal content update...")
deps.watchlist_scan_state['current_phase'] = 'updating_seasonal'
deps.update_automation_progress(automation_id, progress=98, phase='Updating seasonal content',
log_line='Checking seasonal playlists...', log_type='info')
try:
from core.seasonal_discovery import get_seasonal_discovery_service
seasonal_service = get_seasonal_discovery_service(deps.spotify_client, database)
# Only update the current active season
current_season = seasonal_service.get_current_season()
if current_season:
if seasonal_service.should_populate_seasonal_content(current_season, days_threshold=7):
logger.info(f"Updating {current_season} seasonal content...")
deps.update_automation_progress(automation_id,
log_line=f'Updating {current_season} seasonal content...', log_type='info')
seasonal_service.populate_seasonal_content(current_season)
seasonal_service.curate_seasonal_playlist(current_season)
logger.info(f"{current_season.capitalize()} seasonal content updated")
deps.update_automation_progress(automation_id,
log_line=f'{current_season.capitalize()} seasonal content updated', log_type='success')
else:
logger.info(f"{current_season.capitalize()} seasonal content recently updated, skipping")
deps.update_automation_progress(automation_id,
log_line=f'{current_season.capitalize()} seasonal content up to date', log_type='info')
else:
logger.warning("ℹ️ No active season at this time")
deps.update_automation_progress(automation_id,
log_line='No active season', log_type='info')
except Exception as seasonal_error:
logger.error(f"Error updating seasonal content: {seasonal_error}")
import traceback
traceback.print_exc()
deps.update_automation_progress(automation_id,
log_line=f'Seasonal error: {seasonal_error}', log_type='error')
# Generate Last.fm radio playlists (weekly refresh)
logger.info("Starting Last.fm radio generation...")
deps.watchlist_scan_state['current_phase'] = 'generating_lastfm_radio'
deps.update_automation_progress(automation_id, progress=99, phase='Generating Last.fm radio',
log_line='Building Last.fm radio playlists...', log_type='info')
try:
scanner._generate_lastfm_radio_playlists()
logger.info("Last.fm radio generation complete")
deps.update_automation_progress(automation_id,
log_line='Last.fm radio playlists updated', log_type='success')
except Exception as lastfm_error:
logger.error(f"Error generating Last.fm radio playlists: {lastfm_error}")
deps.update_automation_progress(automation_id,
log_line=f'Last.fm radio error: {lastfm_error}', log_type='error')
# Sync Spotify library cache
logger.info("Syncing Spotify library cache...")
try:
for p in all_profiles:
scanner.sync_spotify_library_cache(profile_id=p['id'])
logger.info("Spotify library cache sync complete")
deps.update_automation_progress(automation_id,
log_line='Spotify library cache synced', log_type='info')
except Exception as lib_error:
logger.error(f"Error syncing Spotify library: {lib_error}")
deps.update_automation_progress(automation_id,
log_line=f'Library cache error: {lib_error}', log_type='error')
# Add activity for watchlist scan completion
if total_added_to_wishlist > 0:
deps.add_activity_item("", "Watchlist Scan Complete", f"{total_added_to_wishlist} new tracks added to wishlist", "Now")
try:
if deps.automation_engine:
deps.automation_engine.emit('watchlist_scan_completed', {
'artists_scanned': str(len(scan_results)),
'new_tracks_found': str(total_new_tracks),
'tracks_added': str(total_added_to_wishlist),
})
except Exception:
pass
except Exception as e:
logger.error(f"Error in automatic watchlist scan: {e}")
import traceback
traceback.print_exc()
deps.update_automation_progress(automation_id, log_line=f'Error: {str(e)}', log_type='error')
deps.watchlist_scan_state['status'] = 'error'
deps.watchlist_scan_state['error'] = str(e)
raise # re-raise so automation wrapper returns error status
finally:
# Resume enrichment workers if we paused them
deps.resume_enrichment_workers(_ew_state, 'auto-watchlist scan')
# Clear one-time rescan cutoff after full scan cycle
try:
scanner._clear_rescan_cutoff()
except Exception:
pass
# Always reset flag
with deps.watchlist_timer_lock:
deps.watchlist_auto_scanning = False
deps.watchlist_auto_scanning_timestamp = 0
logger.info("Automatic watchlist scanning complete")

0
tests/watchlist/__init__.py Normal file
View file

320
tests/watchlist/test_auto_scan.py Normal file
View file

@ -0,0 +1,320 @@
"""Tests for core/watchlist/auto_scan.py — auto-scan orchestrator."""
from __future__ import annotations
import threading
from contextlib import contextmanager
from dataclasses import dataclass
import pytest
from core.watchlist import auto_scan as autosc
# ---------------------------------------------------------------------------
# Fakes
# ---------------------------------------------------------------------------
class _FakeApp:
@contextmanager
def app_context(self):
yield
class _FakeSpotify:
def __init__(self, authenticated=True):
self._authenticated = authenticated
def is_authenticated(self):
return self._authenticated
class _FakeAutomationEngine:
def __init__(self):
self.events = []
def emit(self, event_type, data):
self.events.append((event_type, data))
@dataclass
class _ScanResult:
success: bool = True
new_tracks_found: int = 0
tracks_added_to_wishlist: int = 0
class _FakeScanner:
def __init__(self, results=None):
self._results = results or []
self.scan_calls = []
self.discovery_calls = []
self.lastfm_called = False
self.cache_calls = []
self.cutoff_cleared = False
def backfill_watchlist_artist_images(self, profile_id):
return 0
def scan_watchlist_artists(self, artists, *, scan_state, progress_callback, cancel_check):
self.scan_calls.append((artists, scan_state))
return self._results
def populate_discovery_pool(self, profile_id, progress_callback=None):
self.discovery_calls.append(profile_id)
def _generate_lastfm_radio_playlists(self):
self.lastfm_called = True
def sync_spotify_library_cache(self, profile_id):
self.cache_calls.append(profile_id)
def _clear_rescan_cutoff(self):
self.cutoff_cleared = True
class _FakeDB:
def __init__(self, profiles=None, watchlist_count=0, watchlist_artists=None,
lb_profiles=None):
self._profiles = profiles or [{'id': 1}]
self._watchlist_count = watchlist_count
self._watchlist_artists = watchlist_artists or []
self._lb_profiles = lb_profiles or []
self.database_path = '/tmp/test.db'
def get_all_profiles(self):
return self._profiles
def get_watchlist_count(self, profile_id=1):
return self._watchlist_count
def get_watchlist_artists(self, profile_id=1):
return self._watchlist_artists
def get_profiles_with_listenbrainz(self):
return self._lb_profiles
def _build_deps(
*,
actually_scanning=False,
flag_set=False,
spotify_auth=True,
progress_log=None,
activity_log=None,
):
progress_log = progress_log if progress_log is not None else []
activity_log = activity_log if activity_log is not None else []
auto_flag = [flag_set]
auto_ts = [0.0]
state_ref = [{}]
deps = autosc.WatchlistAutoScanDeps(
app=_FakeApp(),
spotify_client=_FakeSpotify(authenticated=spotify_auth),
automation_engine=_FakeAutomationEngine(),
watchlist_timer_lock=threading.Lock(),
is_watchlist_actually_scanning=lambda: actually_scanning,
pause_enrichment_workers=lambda label: {'paused': True},
resume_enrichment_workers=lambda state, label: None,
update_automation_progress=lambda *a, **kw: progress_log.append((a, kw)),
add_activity_item=lambda *a, **kw: activity_log.append((a, kw)),
_get_auto_scanning=lambda: auto_flag[0],
_set_auto_scanning=lambda v: auto_flag.__setitem__(0, v),
_get_auto_scanning_timestamp=lambda: auto_ts[0],
_set_auto_scanning_timestamp=lambda v: auto_ts.__setitem__(0, v),
_get_watchlist_scan_state=lambda: state_ref[0],
_set_watchlist_scan_state=lambda v: state_ref.__setitem__(0, v),
)
deps._auto_flag = auto_flag
deps._state_ref = state_ref
deps._progress_log = progress_log
deps._activity_log = activity_log
return deps
@pytest.fixture
def patched_modules(monkeypatch):
"""Stub out core.watchlist_scanner.get_watchlist_scanner + database access."""
scanner = _FakeScanner(results=[_ScanResult(success=True, new_tracks_found=2,
tracks_added_to_wishlist=1)])
db = _FakeDB(watchlist_count=3,
watchlist_artists=[{'id': 1, 'name': 'A1'}, {'id': 2, 'name': 'A2'}])
import core.watchlist_scanner as ws_mod
monkeypatch.setattr(ws_mod, 'get_watchlist_scanner', lambda spotify: scanner)
monkeypatch.setattr('database.music_database.get_database', lambda: db)
# Stub seasonal + listenbrainz so post-scan steps don't crash trying to import real impl
import core.seasonal_discovery as seasonal_mod
seasonal_service = type('S', (), {
'get_current_season': lambda self: None,
'should_populate_seasonal_content': lambda self, s, days_threshold: False,
'populate_seasonal_content': lambda self, s: None,
'curate_seasonal_playlist': lambda self, s: None,
})()
monkeypatch.setattr(seasonal_mod, 'get_seasonal_discovery_service',
lambda spotify, db: seasonal_service)
return scanner, db
# ---------------------------------------------------------------------------
# Stuck-detection guard
# ---------------------------------------------------------------------------
def test_already_scanning_returns_immediately(patched_modules):
"""is_watchlist_actually_scanning() True → bail before doing anything."""
deps = _build_deps(actually_scanning=True)
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
# No state initialized, no scanner called, no flag set
assert deps._state_ref[0] == {}
assert deps._auto_flag[0] is False
def test_race_check_inside_lock(patched_modules):
"""If get_auto_scanning_flag returns True after the smart-detect, bail."""
deps = _build_deps(actually_scanning=False, flag_set=True)
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
# Should have bailed at the lock-internal check; scan didn't run.
scanner, _ = patched_modules
assert scanner.scan_calls == []
# ---------------------------------------------------------------------------
# No watchlist artists
# ---------------------------------------------------------------------------
def test_zero_watchlist_count_clears_flag_and_returns(patched_modules, monkeypatch):
"""When watchlist count is 0, function clears flag and returns."""
scanner, _ = patched_modules
monkeypatch.setattr('database.music_database.get_database',
lambda: _FakeDB(watchlist_count=0))
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
# Flag was set then cleared
assert deps._auto_flag[0] is False
assert scanner.scan_calls == []
# ---------------------------------------------------------------------------
# Spotify auth gate
# ---------------------------------------------------------------------------
def test_unauthenticated_spotify_clears_flag(patched_modules):
"""Spotify not authenticated → clear flag, return without scanning."""
scanner, _ = patched_modules
deps = _build_deps(spotify_auth=False)
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
assert deps._auto_flag[0] is False
assert scanner.scan_calls == []
# ---------------------------------------------------------------------------
# Successful scan
# ---------------------------------------------------------------------------
def test_successful_scan_runs_post_steps(patched_modules):
"""Scan completes → discovery pool + lastfm + library sync all run."""
scanner, db = patched_modules
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
# Scanner was called with the watchlist
assert len(scanner.scan_calls) == 1
# Post-scan steps fired
assert scanner.discovery_calls == [1]
assert scanner.lastfm_called is True
assert scanner.cache_calls == [1]
# State has summary
assert deps._state_ref[0]['status'] == 'completed'
assert deps._state_ref[0]['summary']['new_tracks_found'] == 2
def test_completion_emits_automation_event(patched_modules):
"""Successful scan emits 'watchlist_scan_completed' on automation_engine."""
scanner, _ = patched_modules
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
assert any(name == 'watchlist_scan_completed' for name, _ in deps.automation_engine.events)
def test_activity_feed_logged_when_tracks_added(patched_modules):
"""Successful scan adding > 0 tracks logs an activity feed entry."""
scanner, _ = patched_modules
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
assert deps._activity_log # at least one activity fired
# ---------------------------------------------------------------------------
# Cancellation mid-scan
# ---------------------------------------------------------------------------
def test_cancelled_scan_skips_post_steps(patched_modules, monkeypatch):
"""If scanner sets cancel_requested mid-flight, post-scan steps skipped."""
scanner, _ = patched_modules
def cancel_during_scan(artists, *, scan_state, progress_callback, cancel_check):
scan_state['cancel_requested'] = True
return []
scanner.scan_watchlist_artists = cancel_during_scan
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
# No post-scan steps ran
assert scanner.discovery_calls == []
assert scanner.lastfm_called is False
# ---------------------------------------------------------------------------
# Profile-scoped trigger
# ---------------------------------------------------------------------------
def test_profile_scoped_trigger_only_scans_that_profile(patched_modules, monkeypatch):
"""When profile_id is provided, only that profile's watchlist is scanned."""
db = _FakeDB(profiles=[{'id': 1}, {'id': 2}],
watchlist_count=3,
watchlist_artists=[{'id': 99, 'name': 'X'}])
monkeypatch.setattr('database.music_database.get_database', lambda: db)
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', profile_id=2, deps=deps)
scanner, _ = patched_modules
assert len(scanner.scan_calls) == 1
# ---------------------------------------------------------------------------
# Cleanup runs in finally
# ---------------------------------------------------------------------------
def test_finally_resets_auto_scanning_flag(patched_modules):
"""Even after a successful scan, the auto_scanning flag is reset."""
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
assert deps._auto_flag[0] is False
def test_finally_clears_rescan_cutoff(patched_modules):
"""scanner._clear_rescan_cutoff() called via finally."""
scanner, _ = patched_modules
deps = _build_deps()
autosc.process_watchlist_scan_automatically(automation_id='a1', deps=deps)
assert scanner.cutoff_cleared is True

424
web_server.py
View file

@ -29247,396 +29247,62 @@ watchlist_scan_state = {
'error': None
}
def _process_watchlist_scan_automatically(automation_id=None, profile_id=None):
"""Main automatic scanning logic that runs in background thread.
# Watchlist auto-scan logic lives in core/watchlist/auto_scan.py.
from core.watchlist import auto_scan as _watchlist_auto_scan
Args:
automation_id: ID of the automation triggering this scan
profile_id: If provided, only scan this profile's watchlist (manual trigger).
If None, scan all profiles (scheduled automation).
def _build_watchlist_auto_scan_deps():
"""Build the WatchlistAutoScanDeps bundle from web_server.py globals on each call.
The three watchlist globals are exposed via property setters on the deps
proxy so the lifted body keeps `name = value` assignment syntax. The
callback pairs below rebind the module-level names when those setters fire.
"""
global watchlist_auto_scanning, watchlist_auto_scanning_timestamp, watchlist_scan_state
def _get_flag():
return watchlist_auto_scanning
scope_label = f"profile {profile_id}" if profile_id else "all profiles"
logger.info(f"[Auto-Watchlist] Timer triggered - starting automatic watchlist scan ({scope_label})...")
def _set_flag(value):
global watchlist_auto_scanning
watchlist_auto_scanning = value
_ew_state = {}
def _get_ts():
return watchlist_auto_scanning_timestamp
try:
# CRITICAL FIX: Use smart stuck detection BEFORE acquiring lock
# This prevents deadlock and handles stuck flags (2-hour timeout)
if is_watchlist_actually_scanning():
logger.info("[Auto-Watchlist] Already scanning (verified with stuck detection), skipping.")
return
def _set_ts(value):
global watchlist_auto_scanning_timestamp
watchlist_auto_scanning_timestamp = value
with watchlist_timer_lock:
# Re-check inside lock to handle race conditions
if watchlist_auto_scanning:
logger.info("[Auto-Watchlist] Already scanning (race condition check), skipping.")
return
def _get_state():
return watchlist_scan_state
# Set flag and timestamp
import time
watchlist_auto_scanning = True
watchlist_auto_scanning_timestamp = time.time()
logger.info(f"[Auto-Watchlist] Flag set at timestamp {watchlist_auto_scanning_timestamp}")
def _set_state(value):
global watchlist_scan_state
watchlist_scan_state = value
# Use app context for database operations
with app.app_context():
from core.watchlist_scanner import get_watchlist_scanner
from database.music_database import get_database
return _watchlist_auto_scan.WatchlistAutoScanDeps(
app=app,
spotify_client=spotify_client,
automation_engine=automation_engine,
watchlist_timer_lock=watchlist_timer_lock,
is_watchlist_actually_scanning=is_watchlist_actually_scanning,
pause_enrichment_workers=_pause_enrichment_workers,
resume_enrichment_workers=_resume_enrichment_workers,
update_automation_progress=_update_automation_progress,
add_activity_item=add_activity_item,
_get_auto_scanning=_get_flag,
_set_auto_scanning=_set_flag,
_get_auto_scanning_timestamp=_get_ts,
_set_auto_scanning_timestamp=_set_ts,
_get_watchlist_scan_state=_get_state,
_set_watchlist_scan_state=_set_state,
)
database = get_database()
# Determine which profiles to scan
if profile_id:
# Manual trigger — scan only the triggering profile
scan_profiles = [{'id': profile_id}]
else:
# Scheduled automation — scan all profiles
scan_profiles = database.get_all_profiles()
def _process_watchlist_scan_automatically(automation_id=None, profile_id=None):
return _watchlist_auto_scan.process_watchlist_scan_automatically(
automation_id, profile_id, _build_watchlist_auto_scan_deps()
)
watchlist_count = sum(database.get_watchlist_count(profile_id=p['id']) for p in scan_profiles)
profile_label = f"profile {profile_id}" if profile_id else f"{len(scan_profiles)} profiles"
logger.info(f"[Auto-Watchlist] Watchlist count check: {watchlist_count} artists found ({profile_label})")
if watchlist_count == 0:
logger.warning("ℹ️ [Auto-Watchlist] No artists in watchlist for auto-scanning.")
with watchlist_timer_lock:
watchlist_auto_scanning = False
watchlist_auto_scanning_timestamp = 0
return
if not spotify_client or not spotify_client.is_authenticated():
logger.info("ℹ️ [Auto-Watchlist] Spotify client not available or not authenticated.")
with watchlist_timer_lock:
watchlist_auto_scanning = False
watchlist_auto_scanning_timestamp = 0
return
logger.info(f"[Auto-Watchlist] Found {watchlist_count} artists in watchlist, starting automatic scan...")
_update_automation_progress(automation_id, progress=5, phase='Loading watchlist',
log_line=f'{watchlist_count} artists ({profile_label})', log_type='info')
# Get list of artists to scan
watchlist_artists = []
for p in scan_profiles:
watchlist_artists.extend(database.get_watchlist_artists(profile_id=p['id']))
scanner = get_watchlist_scanner(spotify_client)
all_profiles = scan_profiles # Used later for discovery pool population
for p in scan_profiles:
try:
filled = scanner.backfill_watchlist_artist_images(p['id'])
if filled:
logger.info(f"Backfilled {filled} watchlist artist images for profile {p['id']}")
except Exception as img_err:
logger.error(f"Image backfill error for profile {p['id']}: {img_err}")
# Initialize detailed progress tracking (same as manual scan)
watchlist_scan_state = {
'status': 'scanning',
'started_at': datetime.now(),
'total_artists': len(watchlist_artists),
'current_artist_index': 0,
'current_artist_name': '',
'current_artist_image_url': '',
'current_phase': 'starting',
'albums_to_check': 0,
'albums_checked': 0,
'current_album': '',
'current_album_image_url': '',
'current_track_name': '',
'tracks_found_this_scan': 0,
'tracks_added_this_scan': 0,
'recent_wishlist_additions': [],
'results': [],
'summary': {},
'error': None,
'cancel_requested': False
}
scan_results = []
# Pause enrichment workers during scan to reduce API contention
_ew_state = _pause_enrichment_workers('auto-watchlist scan')
def _scan_progress(event_type, payload):
if event_type == 'scan_started':
_update_automation_progress(
automation_id,
progress=5,
phase='Loading watchlist',
log_line=f"{len(watchlist_artists)} artists ({profile_label})",
log_type='info',
)
elif event_type == 'artist_started':
total = max(1, payload.get('total_artists', len(watchlist_artists)))
idx = payload.get('artist_index', 1)
artist_name = payload.get('artist_name', '')
pct = 5 + ((idx - 1) / total) * 90
_update_automation_progress(
automation_id,
progress=pct,
phase=f'Scanning: {artist_name} ({idx}/{total})',
current_item=artist_name,
processed=idx - 1,
total=total,
)
elif event_type == 'artist_completed':
artist_name = payload.get('artist_name', '')
new_tracks = payload.get('new_tracks_found', 0)
added = payload.get('tracks_added_to_wishlist', 0)
if new_tracks > 0:
_update_automation_progress(
automation_id,
log_line=f'{artist_name} — {new_tracks} new, {added} added',
log_type='success',
)
else:
_update_automation_progress(
automation_id,
log_line=f'{artist_name} — no new tracks',
log_type='skip',
)
elif event_type == 'artist_error':
artist_name = payload.get('artist_name', '')
error_message = payload.get('error_message', 'error')
_update_automation_progress(
automation_id,
log_line=f'{artist_name} — error: {error_message[:60]}',
log_type='error',
)
elif event_type == 'cancelled':
_update_automation_progress(
automation_id,
progress=100,
phase='Cancelled by user',
log_line='Scan cancelled by user',
log_type='warning',
)
elif event_type == 'scan_completed':
_update_automation_progress(
automation_id,
progress=95,
phase='Scan complete',
log_line=(
f"Scanned {payload.get('successful_scans', 0)} artists — "
f"{payload.get('new_tracks_found', 0)} new tracks, "
f"{payload.get('tracks_added_to_wishlist', 0)} added to wishlist"
),
log_type='success' if payload.get('new_tracks_found', 0) > 0 else 'info',
)
scan_results = scanner.scan_watchlist_artists(
watchlist_artists,
scan_state=watchlist_scan_state,
progress_callback=_scan_progress,
cancel_check=lambda: watchlist_scan_state.get('cancel_requested'),
)
# Update state with results (skip if cancelled — already set by cancel handler)
was_cancelled = watchlist_scan_state.get('cancel_requested', False)
if not was_cancelled:
successful_scans = [r for r in scan_results if r.success]
total_new_tracks = sum(r.new_tracks_found for r in successful_scans)
total_added_to_wishlist = sum(r.tracks_added_to_wishlist for r in successful_scans)
watchlist_scan_state['status'] = 'completed'
watchlist_scan_state['results'] = scan_results
watchlist_scan_state['completed_at'] = datetime.now()
watchlist_scan_state['summary'] = {
'total_artists': len(scan_results),
'successful_scans': len(successful_scans),
'new_tracks_found': total_new_tracks,
'tracks_added_to_wishlist': total_added_to_wishlist
}
logger.info(f"Automatic watchlist scan completed: {len(successful_scans)}/{len(scan_results)} artists scanned successfully")
logger.info(f"Found {total_new_tracks} new tracks, added {total_added_to_wishlist} to wishlist")
_update_automation_progress(automation_id, progress=95, phase='Scan complete',
log_line=f'Scanned {len(successful_scans)} artists — {total_new_tracks} new tracks, {total_added_to_wishlist} added to wishlist',
log_type='success' if total_new_tracks > 0 else 'info')
else:
total_new_tracks = watchlist_scan_state.get('summary', {}).get('new_tracks_found', 0)
total_added_to_wishlist = watchlist_scan_state.get('summary', {}).get('tracks_added_to_wishlist', 0)
logger.warning("Automatic watchlist scan cancelled — skipping post-scan steps")
# Post-scan steps — skip if cancelled
if not was_cancelled:
# Populate discovery pool from similar artists (per-profile)
logger.info("Starting discovery pool population...")
watchlist_scan_state['current_phase'] = 'populating_discovery_pool'
_update_automation_progress(automation_id, progress=96, phase='Populating discovery pool',
log_line='Building discovery pool from similar artists...', log_type='info')
try:
def _discovery_progress(event_type, message):
if event_type == 'artist':
_update_automation_progress(automation_id, phase=f'Discovery pool: {message}',
log_line=message, log_type='info',
current_item=message)
elif event_type == 'phase':
_update_automation_progress(automation_id, phase=message,
log_line=message, log_type='info')
elif event_type == 'success':
_update_automation_progress(automation_id,
log_line=message, log_type='success')
elif event_type == 'skip':
_update_automation_progress(automation_id,
log_line=message, log_type='info')
for p in all_profiles:
scanner.populate_discovery_pool(profile_id=p['id'], progress_callback=_discovery_progress)
logger.info("Discovery pool population complete")
except Exception as discovery_error:
logger.error(f"Error populating discovery pool: {discovery_error}")
import traceback
traceback.print_exc()
_update_automation_progress(automation_id,
log_line=f'Discovery pool error: {discovery_error}', log_type='error')
# Update ListenBrainz playlists cache
logger.info("Starting ListenBrainz playlists update...")
watchlist_scan_state['current_phase'] = 'updating_listenbrainz'
_update_automation_progress(automation_id, progress=97, phase='Updating ListenBrainz',
log_line='Fetching ListenBrainz playlists...', log_type='info')
try:
from core.listenbrainz_manager import ListenBrainzManager
db = get_database()
db_path = str(db.database_path)
lb_profiles = db.get_profiles_with_listenbrainz()
if lb_profiles:
for lb_prof in lb_profiles:
lb_manager = ListenBrainzManager(db_path, profile_id=lb_prof['id'], token=lb_prof['token'], base_url=lb_prof['base_url'])
lb_result = lb_manager.update_all_playlists()
if lb_result.get('success'):
summary = lb_result.get('summary', {})
logger.info(f"ListenBrainz update complete for profile {lb_prof['id']}: {summary}")
_update_automation_progress(automation_id,
log_line=f'ListenBrainz (profile {lb_prof["id"]}): playlists updated', log_type='success')
else:
lb_manager = ListenBrainzManager(db_path)
lb_result = lb_manager.update_all_playlists()
if lb_result.get('success'):
summary = lb_result.get('summary', {})
logger.info(f"ListenBrainz update complete (global): {summary}")
_update_automation_progress(automation_id,
log_line='ListenBrainz: playlists updated', log_type='success')
else:
logger.error(f"ListenBrainz update had issues: {lb_result.get('error', 'Unknown error')}")
_update_automation_progress(automation_id,
log_line=f'ListenBrainz: {lb_result.get("error", "Unknown error")}', log_type='error')
except Exception as lb_error:
logger.error(f"Error updating ListenBrainz: {lb_error}")
import traceback
traceback.print_exc()
_update_automation_progress(automation_id,
log_line=f'ListenBrainz error: {lb_error}', log_type='error')
# Update current seasonal playlist (weekly refresh)
logger.info("Starting seasonal content update...")
watchlist_scan_state['current_phase'] = 'updating_seasonal'
_update_automation_progress(automation_id, progress=98, phase='Updating seasonal content',
log_line='Checking seasonal playlists...', log_type='info')
try:
from core.seasonal_discovery import get_seasonal_discovery_service
seasonal_service = get_seasonal_discovery_service(spotify_client, database)
# Only update the current active season
current_season = seasonal_service.get_current_season()
if current_season:
if seasonal_service.should_populate_seasonal_content(current_season, days_threshold=7):
logger.info(f"Updating {current_season} seasonal content...")
_update_automation_progress(automation_id,
log_line=f'Updating {current_season} seasonal content...', log_type='info')
seasonal_service.populate_seasonal_content(current_season)
seasonal_service.curate_seasonal_playlist(current_season)
logger.info(f"{current_season.capitalize()} seasonal content updated")
_update_automation_progress(automation_id,
log_line=f'{current_season.capitalize()} seasonal content updated', log_type='success')
else:
logger.info(f"{current_season.capitalize()} seasonal content recently updated, skipping")
_update_automation_progress(automation_id,
log_line=f'{current_season.capitalize()} seasonal content up to date', log_type='info')
else:
logger.warning("ℹ️ No active season at this time")
_update_automation_progress(automation_id,
log_line='No active season', log_type='info')
except Exception as seasonal_error:
logger.error(f"Error updating seasonal content: {seasonal_error}")
import traceback
traceback.print_exc()
_update_automation_progress(automation_id,
log_line=f'Seasonal error: {seasonal_error}', log_type='error')
# Generate Last.fm radio playlists (weekly refresh)
logger.info("Starting Last.fm radio generation...")
watchlist_scan_state['current_phase'] = 'generating_lastfm_radio'
_update_automation_progress(automation_id, progress=99, phase='Generating Last.fm radio',
log_line='Building Last.fm radio playlists...', log_type='info')
try:
scanner._generate_lastfm_radio_playlists()
logger.info("Last.fm radio generation complete")
_update_automation_progress(automation_id,
log_line='Last.fm radio playlists updated', log_type='success')
except Exception as lastfm_error:
logger.error(f"Error generating Last.fm radio playlists: {lastfm_error}")
_update_automation_progress(automation_id,
log_line=f'Last.fm radio error: {lastfm_error}', log_type='error')
# Sync Spotify library cache
logger.info("Syncing Spotify library cache...")
try:
for p in all_profiles:
scanner.sync_spotify_library_cache(profile_id=p['id'])
logger.info("Spotify library cache sync complete")
_update_automation_progress(automation_id,
log_line='Spotify library cache synced', log_type='info')
except Exception as lib_error:
logger.error(f"Error syncing Spotify library: {lib_error}")
_update_automation_progress(automation_id,
log_line=f'Library cache error: {lib_error}', log_type='error')
# Add activity for watchlist scan completion
if total_added_to_wishlist > 0:
add_activity_item("", "Watchlist Scan Complete", f"{total_added_to_wishlist} new tracks added to wishlist", "Now")
try:
if automation_engine:
automation_engine.emit('watchlist_scan_completed', {
'artists_scanned': str(len(scan_results)),
'new_tracks_found': str(total_new_tracks),
'tracks_added': str(total_added_to_wishlist),
})
except Exception:
pass
except Exception as e:
logger.error(f"Error in automatic watchlist scan: {e}")
import traceback
traceback.print_exc()
_update_automation_progress(automation_id, log_line=f'Error: {str(e)}', log_type='error')
watchlist_scan_state['status'] = 'error'
watchlist_scan_state['error'] = str(e)
raise # re-raise so automation wrapper returns error status
finally:
# Resume enrichment workers if we paused them
_resume_enrichment_workers(_ew_state, 'auto-watchlist scan')
# Clear one-time rescan cutoff after full scan cycle
try:
scanner._clear_rescan_cutoff()
except Exception:
pass
# Always reset flag
with watchlist_timer_lock:
watchlist_auto_scanning = False
watchlist_auto_scanning_timestamp = 0
logger.info("Automatic watchlist scanning complete")
# --- Metadata Updater System ---