- move Spotify status publishing onto auth, disconnect, and rate-limit transitions
- keep dashboard and debug consumers on the shared cached snapshot
- leave only the initial snapshot seed as a fallback probe
Reported on Discord by winecountrygames — Spotify auth granted, then
re-banned for 4 hours within ~30 seconds, repeatedly. Trace from his
captured log:
< 12:05 [pre-log] Spotify ban active when log starts
15:21:27 First ban EXPIRED → 5-minute post-ban cooldown begins
15:26:27 Cooldown ends, spotify_client.is_authenticated() probe
allowed again → client initialized
15:26:59 First Spotify API call after cooldown — get_artist_albums
for an artist whose discography a background worker was
enriching — gets 429 immediately with no Retry-After
header → new ban activated for 14400s (4 hours)
Root cause: `_POST_BAN_COOLDOWN = 300` (5 minutes) is shorter than
Spotify's actual server-side memory of the previous offense. The
cooldown exists specifically to prevent the "ban expires → we probe →
re-ban" cycle (`spotify_client.py:65-68` documents that intent
explicitly), but the value was wrong: Spotify's server still
considered this user banned 5 minutes after our local ban window
ended, so the very first call after cooldown got slapped.
The 4-hour re-ban itself is correct behavior — `_BASE_MAX_RETRIES_BAN`
fires when spotipy reports "max retries", which means the client
exhausted its internal retry budget on 429s before raising. That's a
severe-ban signal and a long default is the right response.
Fix: bump `_POST_BAN_COOLDOWN` to 1800 seconds (30 min). This is the
smallest change that addresses the immediate "re-probe → re-ban" loop
in the report. 30 minutes is an empirical floor — long enough for
Spotify to actually clear its server-side memory in the cases we've
observed, short enough not to keep functional users locked out beyond
necessary. Can be revisited if reports persist.
What this PR does NOT fix (important context for the same user):
This bump only helps the "ban expires → we re-probe → re-ban" loop.
It does NOT help winecountrygames's other symptom — Spotify being
banned within 30 seconds of his FIRST EVER authorization (no prior
ban). That's a separate failure mode: on first auth, enrichment
workers immediately fan out across the user's library (250 artists
in his case), hammering Spotify endpoints with bulk get_artist_albums
calls before any rate-limit feedback can land. Spotify's hidden
per-endpoint daily quotas — which BoulderBadgeDad has empirically
documented but the global rate limiter doesn't see — flag the burst
and impose a multi-hour cooldown that LOOKS like a bot-detection ban
to us. A proper fix needs a fresh-auth ramp-up: start with very low
Spotify QPS for the first N minutes, scale up only if no rate-limit
feedback arrives. That's a separate PR.
Documented as additional follow-ups (NOT in this change):
- Adaptive cooldown that scales with the size of the previous ban —
a 4-hour MAX_RETRIES ban probably warrants a 1-hour cooldown,
while a 60-second Retry-After-honored ban can resume in 5 minutes.
The system already distinguishes these in `_set_global_rate_limit`,
it just doesn't propagate the distinction to cooldown duration.
- Probe-with-light-call pattern — make the first post-cooldown call
a single inexpensive endpoint (`current_user`) rather than
allowing a background worker's heavy `get_artist_albums` to be
the canary. Failed probe extends cooldown silently instead of
triggering a fresh 4-hour ban.
- Fresh-auth ramp-up (per the limitation above).
Files:
- core/spotify_client.py — `_POST_BAN_COOLDOWN` 300 → 1800. Comment
expanded to cite the report so the value isn't bumped back without
context.
- webui/static/helper.js — WHATS_NEW entry under 2.40 explaining
the change for affected users.
No tests added — the cooldown logic itself is unchanged, only the
constant. Tests asserting on a constant value are theater.
Reported on Discord by winecountrygames — his captured log made the
"ban-expires-to-re-ban" timing chain unambiguous.
PR #340 added ruff to the build-and-test.yml CI gate, which surfaced
286 pre-existing lint errors. Left unfixed, every feature branch push
fails CI. This commit resolves all of them so CI goes green and
contributors can actually land work.
Auto-fixes (248 of 286): removed unused f-string prefixes (F541),
renamed unused loop control variables with underscore prefix (B007),
removed duplicate imports (F811).
Manually fixed 10 latent bugs ruff caught (all wrapped in try/except
today, silently failing):
- music_database.py: _add_discovery_tables() called undefined
conn.commit() — would have crashed the iTunes-support migration
for existing databases. Now uses cursor.connection.commit().
- web_server.py settings GET: referenced undefined download_orchestrator
when it should be soulseek_client. Feature (_source_status on the
settings payload) was silently missing for UI auto-disable logic.
- web_server.py _process_wishlist_automatically: active_server
undefined in track-ownership check. Auto-wishlist was falling
through to the error handler and re-downloading owned tracks.
- web_server.py start_wishlist_missing_downloads: same active_server
bug in the manual wishlist path.
- web_server.py _process_failed_tracks_to_wishlist_exact: emitted
wishlist_item_added automation event with undefined artist_name
and track. Automation event silently never fired correctly.
- web_server.py discovery metadata enrichment: referenced cache
without calling get_metadata_cache() first. Track enrichment from
cached API responses was silently skipped.
- web_server.py Beatport discovery worker: wing-it fallback branch
used undefined successful_discoveries variable. Wing-it counter
never incremented correctly. Now uses state['spotify_matches']
consistently with the rest of the function.
- web_server.py _run_full_missing_tracks_process: stale import json
mid-function shadowed the module-level import, making an earlier
json.dumps() call reference an unbound local (F823).
- web_server.py discovery loop: platform loop variable shadowed
the module-level platform import (F402).
- core/watchlist_scanner.py: 7 lambda captures of loop variables
(B023 classic Python closure-in-loop bug) now bind at creation.
No existing tests had to change. Full suite stays at 263 passed.
When no cached token exists, spotipy's auth probe starts an interactive
OAuth flow that binds 127.0.0.1:<redirect_port> inside the container.
This either steals Flask's port 8008 (crash loop) or binds loopback-only
on 8888 (unreachable from Docker host — 'connection reset by peer').
Now checks for a cached token before probing. If none exists, returns
False immediately so users authenticate via the SoulSync web UI instead.
No behavior change for already-authenticated users.
Fixes#269
Allow cached Spotify search results to return even when Spotify is rate-limited or temporarily unavailable, and remove redundant rate-limit gating after auth checks.
Spotify album art: replace the 4-char size segment after '0000ab67616d'
with '82c1' to request the original uploaded master (up to 2000px+).
Applied via _upgrade_spotify_image_url() in Track, Artist, and Album
dataclass constructors and as a catch-all in _download_cover_art.
Scoped to the ab67616d album art prefix only — artist images use a
different prefix (ab676161) where the trick does not apply.
iTunes/Apple Music: replace '100x100bb' with '3000x3000bb' in all
artworkUrl100 replacements across Track, Artist, Album, and the
get_album images arrays. Also applied as a catch-all in _download_cover_art.
Deezer already uses cover_xl at its maximum — no changes needed there.
Stripped 4,200+ emoji characters from print(), logger calls across
39 Python files. Logs are now clean text — easier to grep, more
professional, no encoding issues on terminals without Unicode support.
Seasonal config icons preserved for UI display.
All metadata source decisions now flow through get_primary_source() and
get_primary_client() in core/metadata_service.py. Previously 6 different
files reimplemented this logic with inconsistent defaults ('itunes' vs
'deezer') and auth checks, causing bugs when any one was missed.
Changes:
- metadata_service.py: Added canonical get_primary_source/get_primary_client
- web_server.py: _get_metadata_fallback_source() and _get_active_discovery_source()
are now thin wrappers delegating to metadata_service
- seasonal_discovery.py: _get_source() delegates to metadata_service
- personalized_playlists.py: _get_active_source() delegates to metadata_service
- spotify_client.py: Fixed _fallback_source default from 'itunes' to 'deezer'
- watchlist_scanner.py: _get_fallback_metadata_client() delegates to metadata_service
Future changes to source selection only need to update one file.
API Call Tracker:
- Save/load 24h minute-bucketed history + events to database/api_call_history.json
- Persists across server restarts via atexit + signal handler hooks
- New record_event() for rate limit bans (called from _set_global_rate_limit)
- New get_debug_summary() for Copy Debug Info — 24h totals, peak cpm with
timestamp, per-endpoint breakdown, and last 20 rate limit events
- Fixed race condition: events iteration now inside lock during save
Spotify Rate Limit Mitigation:
- Enrichment worker: max_pages=5 on get_artist_albums (was unlimited — artist
with 217 albums caused 22 paginated API calls, now capped at 5)
- Enrichment worker: inter_item_sleep raised from 0.5s to 1.5s
Spotify Re-Auth Fix:
- Both OAuth callbacks (port 8008 + 8888) now clear rate limit ban AND
post-ban cooldown after successful re-auth — Spotify usable immediately
instead of stuck on Deezer fallback for 5 minutes
- Auth cache invalidated on both global client and enrichment worker client
YOUR ARTISTS (major feature):
- Aggregates liked/followed artists from Spotify, Tidal, Last.fm, Deezer
- Matches to ALL metadata sources (Spotify, iTunes, Deezer, Discogs)
- DB-first matching: library → watchlist → cache → API search (capped)
- Image backfill from Spotify API for artists missing artwork
- Carousel on Discover page with 20 random matched artists
- View All modal with search, source filters, sort, pagination
- Artist info modal: hero image, matched source badges, genres, bio,
listeners/plays from Last.fm, watchlist toggle, view discography
- Auto-refresh with loading state on first load, polls until ready
- Deduplication by normalized name across all services
DEEZER OAUTH:
- Full OAuth flow: /auth/deezer + /deezer/callback
- Settings UI on Connections tab (App ID, Secret, Redirect URI)
- Token stored encrypted, auto-included in API calls
- get_user_favorite_artists() for liked artists pool
SERVICE CLIENTS:
- Spotify: added user-follow-read scope + get_followed_artists()
- Tidal: get_favorite_artists() with V2/V1 fallback
- Last.fm: get_authenticated_username() + get_user_top_artists()
FAILED MB LOOKUPS MANAGER:
- Manage button on Cache Health modal
- Browse/filter/search all failed MusicBrainz lookups
- Search MusicBrainz directly and manually match entries
- Optimized cache health queries (11 → 4 consolidated)
- Dashboard cache stats now poll every 15s
EXPLORER IMPROVEMENTS:
- Discover button on undiscovered playlist cards
- Status badges: explored/wishlisted/downloaded/ready
- Auto-refresh during discovery via polling
- Redesigned controls: prominent Explore button, icons
BUG FIXES:
- Fix album artist splitting on collab albums (collab mode fed
album-level artists instead of per-track)
- Fix cover.jpg not moving during library reorganize (post-pass sweep)
- Fix cover.jpg missing when album detection takes fallback path
- Fix wishlist auto-processing toast spam (was firing every 2s)
- Fix media player collapsing on short viewports
- Fix watchlist rate limiting (~90% fewer API calls)
- Configurable spotify.min_api_interval setting
- Better Retry-After header extraction
- Encrypt Last.fm and Discogs credentials at rest
- Add $discnum template variable (unpadded disc number)
Addresses all three points from community rate-limiting report:
1. Watchlist scans fetched ALL albums then filtered — 262 albums = 27
API calls per artist. Now determines upfront if full discography is
needed: subsequent scans and time-bounded lookbacks use max_pages=1
(1 API call). Only "full discography" global setting fetches all.
2. MIN_API_INTERVAL (350ms) now configurable via spotify.min_api_interval
setting. Users who get rate-limited frequently can increase the delay.
Floor at 100ms to prevent abuse.
3. Retry-After header extraction improved: added diagnostic logging when
headers exist but lack Retry-After key, plus regex fallback to parse
the value from the error message string.
- SpotifyClient: add _discogs lazy-load property, route _fallback to
DiscogsClient when configured (requires token, falls back to iTunes)
- web_server: _get_metadata_fallback_client returns DiscogsClient when
selected and token present
- Enhanced search: Discogs added as source tab with NDJSON streaming,
only available when token configured
- Alternate sources list includes Discogs when token is set
- Frontend: source labels, tab styling, fetch list all include Discogs
- Consistent with iTunes/Deezer pattern — same interfaces, same routing
- Add rate limiting to all 4 Spotify pagination loops (get_artist_albums,
get_user_playlists, get_playlist_tracks, get_album_tracks) — these
called sp.next() bypassing the rate_limited decorator entirely, causing
unthrottled API calls that triggered 429 bans
- Track pagination calls in API rate monitor (separate endpoint names)
- Increase DELAY_BETWEEN_ARTISTS from 2s to 4s in watchlist scanner
- Abort watchlist scan immediately if Spotify rate limit detected mid-scan
instead of continuing to hammer the API
- New core/api_call_tracker.py — centralized tracker with rolling 60s
timestamps (speedometer) and 24h minute-bucketed history (charts)
- Instrument all 9 service client rate_limited decorators to record
actual API calls with per-endpoint tracking for Spotify
- 1-second WebSocket push loop for real-time gauge updates
- Modern radial arc gauges with service brand colors, glowing active
arc, endpoint dot, 0/max scale labels, smooth CSS transitions
- Click any gauge to open detail modal with 24h call history chart
(Canvas 2D, HiDPI, gradient fill, grid lines, danger zone band)
- Spotify modal shows per-endpoint history lines with color legend
and live per-endpoint breakdown bars
- Rate limited state indicator — blinking red badge with countdown
timer appears on gauge card when Spotify ban is active
- REST endpoint GET /api/rate-monitor/history/<service> for chart data
- Responsive grid layout (5 cols desktop, 3 tablet, 2 phone)
Five places in web_server.py called spotify_client.sp.search() directly,
bypassing the cached search_tracks()/search_artists() methods. Each
discovery worker (Tidal, YouTube, ListenBrainz, Beatport) was also
doubling API calls — sp.search() for raw data then search_tracks() for
Track objects.
Now all use cached methods only. Raw track data for album art is
retrieved from the metadata cache by track ID after matching. Also fixed
a pre-existing bug where Tidal discovery could pair stale Spotify raw
data with a newer iTunes match.
Bumped is_spotify_authenticated() probe cache TTL from 5 to 15 minutes
to reduce /v1/me calls (~288/day → ~96/day). Manual disconnect still
takes effect immediately via _invalidate_auth_cache().
The cache stores raw_data through _extract_fields which expects a dict
with a 'name' field. Storing a raw list caused silent AttributeError,
and storing a dict without 'name' triggered junk entity rejection
(empty string is in _JUNK_NAMES). Now wraps the albums list in a dict
with a valid name field so it passes validation and persists correctly.
The watchlist auto-scan needs fresh data from Spotify to detect new
releases, so it bypasses the cache added in the previous commit.
All other callers (UI browsing, completion badges, discography views)
continue to benefit from cached results.
get_artist_albums was making fresh API calls on every invocation with
no cache check, despite being one of the most called methods (discography
views, completion badges, watchlist scans). The method already cached
individual albums opportunistically but never checked for a cached result
before hitting the API.
Now follows the same check-then-fetch-then-cache pattern used by
get_album_tracks and get_artist. Cache key includes album_type param
so different queries (album,single vs compilation) are cached separately.
When spotipy exhausts all retries on 429 errors, the actual
Retry-After value (often 10+ hours) is consumed internally by
spotipy and not passed in the exception. The default ban was
only 1 hour, causing an endless retry cycle. Increased to 4
hours to match the escalation max and give Spotify's ban time
to expire.
The background enrichment worker now caps itself at 3,000 processed items
per calendar day. Counter resets at midnight automatically. When exhausted,
the worker sleeps and checks every 5 minutes for a new day.
This is scoped entirely to the enrichment worker — user-initiated Spotify
API calls (searches, playlist ops, album lookups, etc.) are completely
unaffected. Budget status is exposed in the worker's get_stats() response
for the dashboard widget.
- Frontend was concatenating Track and Artist inputs into a single
query string, causing Spotify to return mixed results matching
either word in any field. Now sends track and artist as separate
params; backend builds field-filtered query (track:X artist:Y).
- Result limit was silently capped at 10 in spotify_client.search_tracks
via min(limit, 10). Raised to respect requested limit up to
Spotify's API max of 50.
- iTunes fallback endpoint updated with same field-specific params.
- Legacy ?query= param still supported for backward compatibility.
Fixes#194
Both clients have their own Track class separate from iTunes. The
album preference commit added album_type/total_tracks to from_*_track()
constructors but not to the class definitions, crashing all Deezer and
Spotify discovery searches.
Add album_type and total_tracks fields to Track dataclass, populate from
Spotify/iTunes/Deezer API responses, and apply a small tiebreaker bonus
(+0.02 for albums, +0.01 for EPs) in all matching loops so album versions
win when confidence scores are otherwise equal.
Users can now choose between iTunes/Apple Music and Deezer as their free
metadata source in Settings. Spotify always takes priority when authenticated;
the fallback handles all lookups when it's not.
Core changes:
- DeezerClient: full metadata interface (search, albums, artists, tracks)
matching iTunesClient's API surface with identical dataclass return types
- SpotifyClient: configurable _fallback property switches between iTunes/Deezer
based on live config reads (no restart needed)
- MetadataService, web_server, watchlist_scanner, api/search, repair_worker,
seasonal_discovery, personalized_playlists: all direct iTunesClient imports
replaced with fallback-aware helpers
Database:
- deezer_artist_id on watchlist_artists and similar_artists tables
- deezer_track_id/album_id/artist_id on discovery_pool and discovery_cache
- Full CRUD for Deezer IDs: add, read, update, backfill, metadata enrichment
- Watchlist duplicate detection by artist name prevents re-adding across sources
- SimilarArtist dataclass and all query/insert methods handle Deezer columns
Bug fixes found during review:
- Similar artist backfill was writing Deezer IDs into iTunes columns
- Discover hero was storing resolved Deezer IDs in wrong column
- Status cache not invalidating on settings save (source name lag)
- Watchlist add allowing duplicates when switching metadata sources
Fix short artist names (e.g. "ano") getting buried by wildcard matches. Re-ranks Spotify and iTunes results so exact name matches appear first, without changing what's returned.
Increase Spotify client rate limit and reduce API contention during watchlist scans. Changes:
- core/spotify_client.py: Bumped MIN_API_INTERVAL from 0.2s to 0.35s (~171 calls/min) to stay safely under Spotify's ~180/min limit.
- web_server.py: In start_watchlist_scan and automatic scan flow, pause spotify_enrichment_worker and itunes_enrichment_worker before scanning (tracking with _enrichment_was_running/_itunes_enrichment_was_running) and resume them in finally blocks; added console prints for pause/resume. This prevents enrichment workers from contending for API quota during long scans.
- webui/static/script.js: Improved enrichment status tooltip logic to prioritize explicit currentType and then fall back to completion-based inference with explicit branches for artists, albums, and tracks for clearer progress text.
These changes aim to avoid API rate violations and make scan progress display more predictable.
Add a release_date field to the Track dataclass for both iTunes and Spotify clients (iTunes: parsed from releaseDate, Spotify: from album.release_date). Propagate release_date into enhanced search results in web_server and into the client-side script so album objects include release_date when available. Also broaden playlistId matching in the missing-tracks process to include 'enhanced_search_track_'. Removed SQLite SHM/WAL files from the repo (cleanup of DB temporary files). These changes enable showing and using track release dates across the app.
Add _get_playlist_items_page to call the new playlists/{id}/items endpoint (Feb 2026 API migration) and fall back to spotipy.playlist_items (old /tracks) on 403/404. Update _get_playlist_tracks and web_server.get_playlist_tracks to use the new helper to avoid 403 errors for Development Mode apps while preserving compatibility with Extended Quota Mode.
Add a UI button to disconnect Spotify and fall back to iTunes/Apple Music without restarting. Cache is_spotify_authenticated() with a 60s TTL to reduce redundant API calls (~46 call sites were each
triggering a live sp.current_user() call). Fix status endpoint calling the auth check twice per poll,
and ensure both OAuth callback handlers (port 8008 Flask route and port 8888 dedicated server)
invalidate the status cache so the UI updates immediately after authentication.
