Make discovery pool population respect provider priority while keeping Spotify strict, and reduce unnecessary request volume in the hot discovery paths.
- keep discovery fan-out source-priority aware
- preserve cache use where freshness is not required
- cap Spotify artist-album pagination in discovery and cache refresh paths
- keep incremental release checks to a single page, since they only need the newest releases
- add regression coverage for provider order, strict Spotify handling, and pagination caps
Resolve Spotify artist matching through the exact Spotify client only, so watchlist ID backfill cannot drift to fallback-provider results. Remove the remaining preemptive provider availability check from the backfill loop.
Drop the old active-provider artist lookup helpers from watchlist_scanner now that the web scan flow resolves sources through the shared metadata priority.
Keep the Spotify-specific feature toggles in place for discovery and sync paths that still use them.
Move the web watchlist scan core onto the shared metadata source priority so primary provider settings are respected during artist, album, and image resolution.
Add coverage for primary-source-first discography lookup and fallback to later providers when the primary source has no albums.
Bring placeholder tracklist skipping back into the shared watchlist scan path, and centralize the DB-only artist image backfill helper so both web scan entrypoints reuse the same logic.
Drop the legacy watchlist scan entrypoints that are no longer used by the web scan flow, and keep the live refresh path pointed at the shared scanner helper.
Move the shared watchlist scan loop into core/watchlist_scanner.py so web_server.py only handles triggers, locks, progress, and post-scan orchestration.
Manual and scheduled watchlist scans now share the same scanner-side core, while the web entrypoints keep profile selection and automation progress updates.
Spotify was being called for album/artist data fetching across multiple
background workers and the Artists page search even when the user had
Deezer or iTunes set as their primary metadata source. Being authenticated
for playlist sync was treated as permission to use Spotify for everything.
- watchlist_scanner: add _spotify_is_primary_source() that checks both
auth and primary source config; use it for all album/artist data fetching
(discovery pool, recent album caching, playlist curation, similar artist
ID matching, proactive ID backfill). _spotify_available_for_run() is kept
for sync_spotify_library_cache which must run regardless of primary source
- repair_jobs/metadata_gap_filler: gate Spotify ISRC lookup on primary
source being 'spotify'; MusicBrainz lookup unaffected
- repair_jobs/unknown_artist_fixer: replace hardcoded spotify_client with
source-aware client selection — primary source ID tried first, each ID
matched to its correct client (fixes latent bug passing Deezer IDs to
Spotify)
- web_server.py /api/match/search: Artists page search was hardcoded to
spotify_client.search_artists(); now uses _get_metadata_fallback_client()
so results come from the configured primary source
Adds a new Last.fm Radio section to the Discover page that lets users
search a track on Last.fm, generate a similar-tracks playlist, and run
it through the existing discovery/download/sync pipeline. Also generates
playlists automatically from top listening history during watchlist scans
(max once per week).
- core/lastfm_client.py: Add get_similar_tracks() using track.getsimilar
- core/listenbrainz_manager.py: Add save_lastfm_radio_playlist() with
deterministic MBID (MD5 seed), cleanup limit of 5 for lastfm_radio type
- web_server.py: Add /api/lastfm/configured, /api/lastfm/search/tracks,
/api/lastfm/radio/generate, /api/discover/listenbrainz/lastfm-radio;
fix playlist['name'] KeyError in discovery worker that was resetting
phase back to 'fresh' after completion
- core/watchlist_scanner.py: Add _generate_lastfm_radio_playlists() with
weekly throttle, called at end of scan_all_watchlist_artists()
- webui/index.html: Add #lastfm-radio-section above ListenBrainz section,
hidden unless Last.fm API key is configured
- webui/static/script.js: Search/generation/card-load functions; fix
discovery modal labels (Last.fm Radio vs ListenBrainz), description
update on completion, belt-and-suspenders completion handling inside
updateYouTubeDiscoveryModal; fix album/duration display for tracks
without metadata; music note SVG placeholder for missing art
- webui/static/style.css: Styles for search bar, dropdown, result rows
Deezer and iTunes defaulted to 50 albums max, silently truncating large
discographies. Deezer now paginates (100 per page) up to 200. iTunes
raised to 200 (single call). All callers in web_server.py updated to
use the new defaults instead of hardcoding limit=50.
Also adds diagnostic logging for allow_duplicates album comparison
to help debug inconsistent singles behavior.
Singles like "idol" weren't added when the same song existed on a
different album because check_track_exists used album-aware matching
that found the track via album name. Now skips the album hint when
allow_duplicates is on so matching is title+artist only, then compares
album names ourselves with a strict 0.85 threshold. Only affects users
with allow_duplicates enabled.
The setting only affected wishlist dedup but the watchlist scanner's
library check still skipped tracks by title+artist regardless. Now
when allow_duplicates is enabled, the scanner compares album names
and only skips if the same album matches. Same song on a different
album is allowed through to the wishlist.
Spotify lists unreleased albums with placeholder names like "Track 1",
"Track 2" before the real tracklist is revealed. The scanner was trying
to download these, searching Soulseek for "Track 1" by artist which
matches random files. Now skips any album where more than half the
tracks match the placeholder pattern. Covers both the watchlist scan
and discovery pool paths.
Stripped 4,200+ emoji characters from print(), logger calls across
39 Python files. Logs are now clean text — easier to grep, more
professional, no encoding issues on terminals without Unicode support.
Seasonal config icons preserved for UI display.
Clients are for the most part being initialized per-request, which leads to a lot of redundant client initialization, as well as noise on the logs, since each client initialization emits a row on the logs, eg. 'Deezer client initialized'
All metadata source decisions now flow through get_primary_source() and
get_primary_client() in core/metadata_service.py. Previously 6 different
files reimplemented this logic with inconsistent defaults ('itunes' vs
'deezer') and auth checks, causing bugs when any one was missed.
Changes:
- metadata_service.py: Added canonical get_primary_source/get_primary_client
- web_server.py: _get_metadata_fallback_source() and _get_active_discovery_source()
are now thin wrappers delegating to metadata_service
- seasonal_discovery.py: _get_source() delegates to metadata_service
- personalized_playlists.py: _get_active_source() delegates to metadata_service
- spotify_client.py: Fixed _fallback_source default from 'itunes' to 'deezer'
- watchlist_scanner.py: _get_fallback_metadata_client() delegates to metadata_service
Future changes to source selection only need to update one file.
Albums announced but not yet released have no real audio available,
causing Soulseek to match random tracks with similar names. Both
discography methods (Spotify and generic client) now filter out
albums with release dates in the future. Skipped albums are not
marked as processed — they will be picked up on the first scan
after their release date passes.
Addresses all three points from community rate-limiting report:
1. Watchlist scans fetched ALL albums then filtered — 262 albums = 27
API calls per artist. Now determines upfront if full discography is
needed: subsequent scans and time-bounded lookbacks use max_pages=1
(1 API call). Only "full discography" global setting fetches all.
2. MIN_API_INTERVAL (350ms) now configurable via spotify.min_api_interval
setting. Users who get rate-limited frequently can increase the delay.
Floor at 100ms to prevent abuse.
3. Retry-After header extraction improved: added diagnostic logging when
headers exist but lack Retry-After key, plus regex fallback to parse
the value from the error message string.
- Add discogs_artist_id column to watchlist_artists table (migration)
- Add discogs_artist_id to WatchlistArtist dataclass
- Add to get_watchlist_artists optional_columns and constructor
- Add update_watchlist_discogs_id DB method
- Backfill loop includes Discogs when token is configured
- Add _match_to_discogs for cross-provider artist matching
- Backfill maps updated: id_attr, match_fn, update_fn all include discogs
- Was only backfilling the active provider — artists added via Deezer
never got Spotify/iTunes IDs, and vice versa
- Now backfills iTunes (always), Deezer (always), and Spotify (if
authenticated) at the start of every scan
- Added _match_to_deezer() and update_watchlist_deezer_id() for
Deezer cross-provider matching
- Generalized backfill with provider→attribute/function maps
- Add rate limiting to all 4 Spotify pagination loops (get_artist_albums,
get_user_playlists, get_playlist_tracks, get_album_tracks) — these
called sp.next() bypassing the rate_limited decorator entirely, causing
unthrottled API calls that triggered 429 bans
- Track pagination calls in API rate monitor (separate endpoint names)
- Increase DELAY_BETWEEN_ARTISTS from 2s to 4s in watchlist scanner
- Abort watchlist scan immediately if Spotify rate limit detected mid-scan
instead of continuing to hammer the API
The previous commit only added skip_cache=True to one call site in
web_server.py. The watchlist scanner in core/watchlist_scanner.py has
6 Spotify get_artist_albums calls that also need fresh data to detect
new releases. All now bypass cache. iTunes/Deezer calls are unaffected
(they don't have the skip_cache param, detected via hasattr check).
The image update code only checked Spotify's images array format,
missing iTunes direct image_url. Also used spotify_artist_id for
DB lookup which missed Deezer-only artists. Now handles both image
formats and tries all provider IDs for the DB update.
Old Deezer watchlist artists missing images will get backfilled
on the next watchlist scan automatically.
When artist-specific track search fails, falls back to album-aware
matching: finds the album by title (any artist), then checks if the
track exists on it. Fixes daily re-downloads of collaborative albums
filed under a different artist (e.g., "Spiral Staircases" tagged
under "The Alchemist" but scanned from "Larry June's" watchlist).
- check_track_exists: new album parameter, album-aware fallback with
0.8 album title threshold + 0.7 track title threshold
- Watchlist scanner: passes album_data.get('name') to track checks
- Download modal: passes batch_album_context to fallback track search
- Wishlist callers (4 spots): extract and pass track album name
- Backwards compatible: album=None default, no change for callers
without album context (singles, playlists)
New "Scan Lookback" dropdown in the watchlist artist config modal.
Each artist can override the global lookback period (7d to entire
discography). Default is "Use Global Setting" (NULL in DB).
- Database: lookback_days INTEGER DEFAULT NULL on watchlist_artists,
auto-migrated on startup
- Scanner: checks per-artist lookback_days first, falls back to
global discovery_lookback_period if NULL
- Backend: GET/POST /api/watchlist/artist/<id>/config includes
lookback_days. Changing lookback clears last_scan_timestamp to
force a rescan with the new window
- Frontend: dropdown with 8 options in artist config modal
- Fully backwards compatible — existing artists unchanged
Integrates play history data into the discovery algorithm:
- Listening profile: _get_listening_profile() builds user's top artists,
genres, play counts, and listening velocity from the last 30 days
- Artist genre cache: pre-built from local DB for O(1) genre lookups
- Release Radar: +10 genre affinity, +15 artist familiarity, -10 overplay
penalty. Weights rebalanced to 45% recency + 25% popularity + bonuses
- Discovery Weekly: serendipity scoring within tiers — boosts unheard
artists in preferred genres, penalizes overplayed artists
- Recent Albums: adaptive time window (21-60 days) based on listening
velocity — heavy listeners get fresher content, casual listeners more
- New "Because You Listen To" sections: personalized carousels based on
user's top 3 played artists via similar artists + genre fallback
- New endpoint: /api/discover/because-you-listen-to with artist images
- Frontend: BYLT sections with artist photo headers on discover page
- All changes gracefully fall back when no listening data exists
Users can now choose between iTunes/Apple Music and Deezer as their free
metadata source in Settings. Spotify always takes priority when authenticated;
the fallback handles all lookups when it's not.
Core changes:
- DeezerClient: full metadata interface (search, albums, artists, tracks)
matching iTunesClient's API surface with identical dataclass return types
- SpotifyClient: configurable _fallback property switches between iTunes/Deezer
based on live config reads (no restart needed)
- MetadataService, web_server, watchlist_scanner, api/search, repair_worker,
seasonal_discovery, personalized_playlists: all direct iTunesClient imports
replaced with fallback-aware helpers
Database:
- deezer_artist_id on watchlist_artists and similar_artists tables
- deezer_track_id/album_id/artist_id on discovery_pool and discovery_cache
- Full CRUD for Deezer IDs: add, read, update, backfill, metadata enrichment
- Watchlist duplicate detection by artist name prevents re-adding across sources
- SimilarArtist dataclass and all query/insert methods handle Deezer columns
Bug fixes found during review:
- Similar artist backfill was writing Deezer IDs into iTunes columns
- Discover hero was storing resolved Deezer IDs in wrong column
- Status cache not invalidating on settings save (source name lag)
- Watchlist add allowing duplicates when switching metadata sources
Allow multiple users to share a single SoulSync instance with isolated personal data. Each profile gets its own watchlist, wishlist, discovery pool, similar artists, and bubble snapshots — while sharing the same music library, database, and service credentials.
- Netflix-style profile picker on startup when multiple profiles exist
- Optional PIN protection per profile; admin PIN required when >1 profiles
- Admin-only profile management (create, edit, rename, delete)
- Profile avatar images via URL with colored-initial fallback
- Zero-downtime SQLite migration — all existing data maps to auto-created
admin profile
- Single-user installs see no changes — profile system is invisible until
a second profile is created
- WebSocket count emitters scoped to profile rooms (watchlist/wishlist)
- Background scanners (watchlist, wishlist, discovery) iterate all profiles
Store album artists in the Spotify track data structure (handles both dict and object album forms) so downstream processing has access to album-level artist metadata. Also update missing-track processing to fall back to source_info['watchlist_artist_name'] when artist_name is absent, ensuring artist context is preserved for legacy/watchlist-sourced entries.
When clearing the wishlist or changing the discovery lookback period, reset watchlist_artists.last_scan_timestamp to NULL so subsequent scans can re-discover older releases that were previously filtered by an earlier scan timestamp. clear_wishlist now deletes wishlist_tracks, updates last_scan_timestamp for all watchlist artists, and logs the number of tracks cleared and artists reset. set_discovery_lookback_period also resets last_scan_timestamp and reports how many artists were reset. Minor whitespace cleanups in watchlist_scanner and web_server included.
Multi-disc albums (e.g., deluxe editions, double albums) now automatically organize
tracks into Disc 1/, Disc 2/ subfolders within the album folder. Detection uses the
disc_number field from Spotify's API — when an album has total_discs > 1, subfolders
are created. Single-disc albums are completely unaffected.
- Plumb disc_number through all download paths (enhanced, non-enhanced, download
missing modal, wishlist)
- Compute total_discs from album tracklist and store on album context
- Modify path builder to insert Disc N/ subfolder for multi-disc albums
- Preserve disc_number when tracks fail and get re-added to wishlist
- Preserve disc_number when adding tracks to wishlist from library page
- Add visual disc separators in Soulseek search result track lists
When Spotify is enabled after populating similar artists with only iTunes IDs, the freshness check now detects missing Spotify IDs and triggers a refetch. This fixes the Discover page not showing data when switching from iTunes-only mode.
- Add similar artists fetching to web UI scan loop
- Add database migration for UNIQUE constraint on similar_artists table - Add source-agnostic /api/discover/album endpoint for iTunes support
- Fix NOT NULL constraint on discovery_recent_albums blocking iTunes albums
- Add fallback to watchlist artists when no similar artists exist
- Add /api/discover/refresh and /api/discover/diagnose endpoints
- Add retry logic with exponential backoff for iTunes API calls
- Ensure cache_discovery_recent_albums runs even when pool population skips
Implement dual-source architecture where iTunes serves as always-available
primary source and Spotify as preferred source when authenticated.
- Make watchlist scans provider-aware (manual and auto paths)
- Update discovery pool population to process both sources
- Update recent albums caching for both sources
- Create source-specific curated playlists (Fresh Tape, Archives)
- Add on-the-fly iTunes ID resolution for similar artists
- Add iTunes ID check to similar artists freshness validation
- Fix sqlite3.Row compatibility in personalized playlists
- Fix iTunes ISO 8601 date format parsing
- Update API endpoints to serve source-appropriate data
This ensures the app remains fully functional if Spotify becomes
unavailable (rate limits, auth issues, bans) by seamlessly falling
back to iTunes data that has been building in parallel.
