danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
soulsync/core
History
BoulderBadgeDad cea0e365c2 Fix #759: Amazon enrichment floods when its public proxy is down 
After an update, installs became unusable: the Amazon enrichment worker runs by
default, the default public T2Tunes proxy (t2tunes.site) was returning
503 'Amazon Music API is not initialized', and the worker treated every album
as an individual error -- logging an ERROR per item, churning network + DB
continuously across the whole library, and marking every row 'error' (a state
the retry tiers never re-attempt, so even after the proxy recovered nothing
re-enriched). The reporter couldn't reach the UI to turn it off.

Two-part fix:

1. Source-outage circuit breaker (core/amazon_outage.py, pure + tested):
   - is_source_outage(exc) distinguishes a whole-source outage (HTTP 5xx,
     'not initialized', connection failure, non-JSON error page) from a real
     per-item miss (404, transient 400, etc.).
   - On an outage the worker now leaves the item UNTOUCHED (so it's retried once
     the proxy recovers instead of being permanently burned to 'error'), logs
     ONCE per streak, and backs off with next_poll_delay_seconds() -- escalating
     30s -> 60s -> ... capped at 30 min -- instead of grinding every 2s. It
     auto-resumes the normal cadence the moment the source answers (success OR a
     non-outage error both clear the streak).
   - AmazonClientError now carries status_code so detection doesn't rely on
     message parsing.

2. Opt-in by default (web_server.py): amazon_enrichment_paused now defaults to
   True. Because enrichment depends on an external public proxy that can be
   down, it stays paused unless the user explicitly enables it -- a proxy outage
   can no longer take down installs that never opted in. (Behaviour change:
   anyone on the old auto-on default is now paused; re-enable in Settings.)

Together: on update the worker is paused -> no flood -> UI accessible; opted-in
users are protected from future outages by the breaker.

Tests: tests/test_amazon_outage.py (12) pin the classifier across every error
surface (incl. the exact 503 'not initialized' case) and the back-off schedule
(monotonic, capped). 157 Amazon tests pass; lint clean.

Note: could not reproduce the exact 'UI fully unreachable' mechanism remotely
(WAL + 8 gthreads shouldn't hard-lock); the fix removes the flood/churn that is
the practical cause and defaults the feature off.
2026-06-01 13:10:51 -07:00
..
artists Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
automation DB backups: verify integrity + never evict the last good backup 2026-05-30 21:13:04 -07:00
discovery Fix: Spotify sync crash 'unexpected keyword argument candidate_pool' 2026-05-31 23:00:24 -07:00
download_engine Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
download_plugins Fix: torrent/usenet album bundle hard-fails on 'no results' instead of falling back 2026-06-01 09:55:59 -07:00
downloads Downloads: add per-track detail endpoint for the track-detail modal 2026-05-31 20:18:35 -07:00
enrichment Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
imports Quarantine: manage a quarantined file from the download modal (Listen / Accept / Search) 2026-05-31 15:41:04 -07:00
library Fix: duplicate detector kept lossy over lossless (rank format first) 2026-06-01 12:49:34 -07:00
matching AcoustID + quarantine modal: three bug fixes (closes #607, closes #608) 2026-05-15 08:55:06 -07:00
media_server Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
metadata Cover-art sources: skip low-res art (min-resolution guard) + max-res iTunes 2026-06-01 12:24:51 -07:00
personalized Personalized pipeline: auto-refresh stale snapshots after watchlist scan 2026-05-15 20:53:03 -07:00
playback Player: log SoulSync web-player plays (recently-played + smart-radio recency) 2026-05-30 15:11:46 -07:00
playlists Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
radio Player revamp Phase 2: smart radio ranking (play-count + popularity) 2026-05-30 08:47:18 -07:00
repair_jobs Metadata cache: hard LRU row cap to stop unbounded growth (7.6GB incident) 2026-05-30 23:22:05 -07:00
search Basic search: visual overhaul + per-source picker in hybrid mode 2026-05-28 10:22:07 -07:00
stats Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
streaming Player revamp Phase 3a: extract stream state into testable per-session store 2026-05-30 08:59:15 -07:00
sync Persist Find & Add selections as permanent server-playlist match overrides 2026-05-14 09:39:24 -07:00
text Address Cin review: extract helpers, indexed pool fetch, tidy nits 2026-05-24 23:33:55 -07:00
torrent_clients fix(torrent): use before/after diff to recover qBit info-hash 2026-05-20 18:11:30 -07:00
usenet_clients Fix: Usenet bundle stuck at 99%/100% — SAB reports post-processing in History as non-terminal (#721) 2026-05-29 00:49:02 -04:00
watchlist Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
webui Extract WebUI asset helpers 2026-05-13 22:26:25 +03:00
wishlist Wishlist: import Optional (fix ruff F821 in processing.py) 2026-05-29 18:31:45 -07:00
workers Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
acoustid_client.py AcoustID: report errors honestly instead of masking them as 'Skipped' 2026-05-31 20:10:31 -07:00
acoustid_verification.py AcoustID: report errors honestly instead of masking them as 'Skipped' 2026-05-31 20:10:31 -07:00
album_consistency.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
amazon_client.py Fix #759: Amazon enrichment floods when its public proxy is down 2026-06-01 13:10:51 -07:00
amazon_download_client.py Amazon download client: write final size==transferred before returning file path 2026-05-16 11:09:04 -07:00
amazon_outage.py Fix #759: Amazon enrichment floods when its public proxy is down 2026-06-01 13:10:51 -07:00
amazon_worker.py Fix #759: Amazon enrichment floods when its public proxy is down 2026-06-01 13:10:51 -07:00
api_call_tracker.py Wire Amazon Music as a first-class download source 2026-05-16 09:40:50 -07:00
archive_pipeline.py fix(downloads): harden album bundle staging 2026-05-20 21:39:06 -07:00
artist_source_detail.py Fix MusicBrainz artist detail showing MBID as name 2026-05-19 12:34:35 -07:00
artist_source_lookup.py Source IDs: add canonical registry, adopt at the highest-value sites 2026-05-29 12:19:59 -07:00
audiodb_client.py Add API Rate Monitor dashboard with real-time speedometer gauges 2026-04-02 10:46:43 -07:00
audiodb_worker.py AudioDB worker: stop infinite loop on direct-ID lookup failure (#553) 2026-05-11 14:18:55 -07:00
auto_import_worker.py Auto-import: fall through to other metadata sources when primary returns no match 2026-05-12 12:32:18 -07:00
automation_engine.py Wire automation engine through next_run_at + register monthly_time (PR 2/4) 2026-05-27 12:03:41 -07:00
connection_detect.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
connection_test.py Add torrent and usenet release staging support 2026-05-21 14:22:21 -07:00
database_update_worker.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
db_integrity.py DB backups: verify integrity + never evict the last good backup 2026-05-30 21:13:04 -07:00
debug_info.py Fix Copy Debug Info music_source + surface missing services 2026-05-12 16:47:55 -07:00
deezer_client.py Fix Deezer contributors tagging silently dropping for cache-polluted tracks 2026-05-14 11:10:51 -07:00
deezer_download_client.py Surface silent exceptions in metadata clients — 37 sites 2026-05-07 09:33:38 -07:00
deezer_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
discogs_client.py Discogs: strip artist disambiguation suffixes at every name surface (#634) 2026-05-27 09:05:47 -07:00
discogs_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
download_orchestrator.py Fix Usenet settings reload without restart 2026-05-22 08:28:56 -07:00
genius_client.py Add API Rate Monitor dashboard with real-time speedometer gauges 2026-04-02 10:46:43 -07:00
genius_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
genre_filter.py Expand default genre whitelist from 223 to 272 genres 2026-04-18 20:32:49 -07:00
hifi_client.py Log exception when inferring HiFi manifest ext 2026-05-21 18:24:56 -07:00
hydrabase_client.py feat: add explicit badges to discography modal and artist-detail cards 2026-05-12 23:35:08 -04:00
hydrabase_worker.py Improve graceful shutdown and rollback safety 2026-04-12 15:17:18 +03:00
image_cache.py Image cache: reject truncated downloads instead of caching broken covers (#750) 2026-05-30 17:45:01 -07:00
itunes_client.py feat: add explicit badges to discography modal and artist-detail cards 2026-05-12 23:35:08 -04:00
itunes_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
jellyfin_client.py Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
lastfm_client.py Add Last.fm Track Radio to Discover page 2026-04-14 23:41:36 -07:00
lastfm_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
library_reorganize.py Reorganize: skip files in the duplicate-cleaner /deleted quarantine (#746) 2026-05-30 00:15:06 -07:00
lidarr_download_client.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
listenbrainz_client.py Remove emojis from all Python log and print statements 2026-04-11 21:11:02 -07:00
listenbrainz_manager.py Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
listening_stats_worker.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
lyrics_client.py Retag now re-embeds LYRICS tag instead of leaving it empty 2026-05-14 15:52:05 -07:00
matching_engine.py Fix: duplicate tracks in albums with Japanese / CJK titles (#722) 2026-05-28 08:53:43 -07:00
metadata_service.py Add MusicBrainz as a metadata source 2026-05-18 18:47:13 -07:00
musicbrainz_client.py MusicBrainz manual search: field-scope the artist in non-strict mode (#754) 2026-05-31 10:05:24 -07:00
musicbrainz_search.py Fix: MusicBrainz artist discography capped at 25 releases 2026-05-28 12:00:05 -07:00
musicbrainz_service.py Cross-script artist aliases: include canonical name + non-strict fallback 2026-05-14 13:07:15 -07:00
musicbrainz_worker.py Alias resolution polish: lazy-fire on direct-match failure + worker backfill 2026-05-10 17:02:02 -07:00
navidrome_client.py Preserve Navidrome album cover art 2026-05-25 19:51:38 -07:00
personalized_playlists.py Add MusicBrainz as a metadata source 2026-05-18 18:47:13 -07:00
plex_client.py Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
prowlarr_client.py feat(settings): add Prowlarr integration as indexer aggregator 2026-05-20 14:41:54 -07:00
qobuz_client.py Fetch all Qobuz favorite tracks for discovery 2026-05-24 01:17:38 -07:00
qobuz_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
reorganize_queue.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
reorganize_runner.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
repair_worker.py Fix: duplicate detector kept lossy over lossless (rank format first) 2026-06-01 12:49:34 -07:00
replaygain.py Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
runtime_state.py Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
seasonal_discovery.py Fix source artist detail navigation from discover modals 2026-05-18 13:50:10 -07:00
socketio_cors.py Socket.IO CORS: handle self-review nits 2026-04-26 19:24:43 -07:00
soulid_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
soulseek_client.py Album bundle: fall back to per-track when the chosen folder yields nothing 2026-05-29 19:46:52 -07:00
soulsync_client.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
soundcloud_client.py Surface engine-not-wired errors + exclude soulseek from monitor aggregation 2026-05-05 12:20:51 -07:00
source_ids.py Source IDs: add canonical registry, adopt at the highest-value sites 2026-05-29 12:19:59 -07:00
spotify_client.py Add download buttons + bulk action to artist top-tracks sidebar 2026-05-07 15:44:47 -07:00
spotify_public_scraper.py Add Spotify Link tab for public playlist/album scraping without API credentials 2026-03-15 21:25:05 -07:00
spotify_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
tag_writer.py Embed highest-resolution album art across all art paths 2026-05-28 13:21:53 -07:00
tidal_client.py Your Albums: selectable wishlist modal + Tidal album resolution 2026-05-11 12:36:16 -07:00
tidal_download_client.py Handle non-JSON Tidal auth responses 2026-05-20 14:04:45 -07:00
tidal_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
watchlist_scanner.py Fix missing album art for non-Spotify sources + animate Downloads nav icon 2026-05-18 20:24:13 -07:00
web_scan_manager.py MS Cin-5: Drop per-server globals — engine owns the clients 2026-05-05 18:05:45 -07:00
wishlist_service.py Extract wishlist logic into dedicated package 2026-04-28 21:17:24 +03:00
worker_utils.py Repair stale media schema during refresh 2026-05-21 17:41:54 -07:00
youtube_client.py Escape dash-leading YouTube search queries 2026-05-24 08:54:47 -07:00