danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
soulsync/core
History
BoulderBadgeDad 79648a4f5f import: async staging scan so a large library doesn't time out the page (#947, phase 1 backend) 
A whole-library migration (ramonskie copied his Lidarr library into /staging) makes the synchronous
staging scan walk + tag-read tens of thousands of files INSIDE the GET request, blowing past
gunicorn's 120s timeout — and because the killed request never warms the cache, every reload
re-times-out. Moves the SAME scan off the request thread; the page reports progress instead of
hanging.

- _scan_staging_records gains an optional `progress` param (additive; default None = unchanged).
  Refactored to two passes: a fast walk to collect the audio-file list (total), then the slow
  tag-read loop updating scanned. A generation guard stops a scan that finishes AFTER an import
  from committing stale records.
- ensure_background_staging_scan(path): idempotent background runner filling the existing cache.
- get_staging_records_or_status(): warm cache or a scan that finishes within a ~3s grace → records
  (so small/normal folders still answer in one request, no UX change); else ("scanning", progress).
  A scan error is re-raised so the endpoints log + return it exactly as before.
- /staging/files|groups|hints return {scanning, progress} when the scan is still running instead of
  blocking; new lightweight /staging/scan-status for cheap progress polling.

Single source preserved (same scan + cache, just off the request thread). 13 new tests (progress,
idempotent ensure, grace ready-vs-scanning, generation guard discards stale, endpoint scanning
shape, error contract, status ready/cold); full import suite 626 green; ruff clean.

Next: phase 2 — the React import page polls scan-status + shows a progress bar, then renders.
2026-06-29 09:24:41 -07:00
..
artists Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
automation Remove old auto-acting Quality Scanner tool (replaced by Quality Upgrade Finder job) 2026-06-13 12:14:45 -07:00
blocklist Blocklist Phase 1 (backfill + API + modal): the Blocklist button on the watchlist page 2026-06-07 15:25:52 -07:00
credentials Profiles: review fixes — close two gating gaps + reject whitespace secrets 2026-06-10 00:48:03 -07:00
diagnostics #802: on-demand memory-growth diagnostic (tracemalloc, browser-drivable) 2026-06-06 18:31:14 -07:00
discovery Listening Mix: source-independent track fetch (Deezer public fallback) 2026-06-25 10:37:16 -07:00
download_engine lint: add strict=True to engine search zip() (B905) 2026-06-25 16:41:00 -07:00
download_plugins feat(quality): source mappers + populate real quality on streaming results 2026-06-14 12:27:03 +02:00
downloads downloads: make the "file not found" failure actionable instead of opaque 2026-06-28 15:32:47 -07:00
enrichment #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
exports playlist export: opt-in confident-search backfill for the unmatched tail (#945, increment 7) 2026-06-28 23:02:08 -07:00
imports import: async staging scan so a large library doesn't time out the page (#947, phase 1 backend) 2026-06-29 09:24:41 -07:00
library jellyfin scan: page the bulk fetch so the no-progress watchdog can't false-stall 2026-06-28 13:41:38 -07:00
maintenance Add one-off repair for source ids shared across multiple artists 2026-06-05 08:38:55 -07:00
matching #845 tests: lift history-path resolver to core/ + seam-test the delete-safety 2026-06-11 11:07:25 -07:00
media_server Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
metadata spotify: playlist-write client + single-source OAuth scope (#945, increment 2) 2026-06-28 21:12:53 -07:00
personalized SoulSync Discovery sync tab: list all kinds + Listening Mix generator 2026-06-25 11:38:41 -07:00
playback Player: log SoulSync web-player plays (recently-played + smart-radio recency) 2026-05-30 15:11:46 -07:00
playlists #901: give file-import playlist tracks a stable id so manual matches stick 2026-06-22 08:53:24 -07:00
quality lossy copy: support all lossless formats, not just FLAC (#941) 2026-06-28 12:13:05 -07:00
radio Player revamp Phase 2: smart radio ranking (play-count + popularity) 2026-05-30 08:47:18 -07:00
repair_jobs lossy copy: support all lossless formats, not just FLAC (#941) 2026-06-28 12:13:05 -07:00
search Multi-artist tags: Search → Download Now finally knows its metadata source (Netti93) 2026-06-09 17:20:16 -07:00
security Login mode: enforce "every profile has a password" at every write-point (no gaps) 2026-06-11 19:48:50 -07:00
stats Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
streaming #809 Navidrome playback: stream via the server's API when the library isn't mounted on disk 2026-06-07 13:52:14 -07:00
sync Sync wishlist re-add: skip wing-it stubs (match the sync), no clickable button 2026-06-24 16:51:43 -07:00
text Cover art: a sequel digit glued to a CJK title ('…サウンドトラック2') now blocks the wrong-album match 2026-06-18 08:24:54 -07:00
torrent_clients Add Aria2 to the torrent client list (Shdjfgatdif's request) 2026-06-11 22:29:40 -07:00
usenet_clients NZBGet: import from the final location, not the incomplete '….#NZBID' dir 2026-06-18 12:47:19 -07:00
watchlist watchlist history: record automatic scans too (#933) 2026-06-27 00:42:58 -07:00
webui Extract WebUI asset helpers 2026-05-13 22:26:25 +03:00
wishlist wishlist: render library-sourced art (album + artist) — fix blank wishlist images 2026-06-27 17:35:28 -07:00
workers Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
acoustid_client.py AcoustID: report errors honestly instead of masking them as 'Skipped' 2026-05-31 20:10:31 -07:00
acoustid_verification.py feat(verification): status vocabulary, DB column, SOULSYNC_VERIFICATION tag 2026-06-11 01:27:29 +02:00
album_consistency.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
amazon_client.py Fix #759: Amazon enrichment floods when its public proxy is down 2026-06-01 13:10:51 -07:00
amazon_download_client.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
amazon_outage.py Fix #759: Amazon enrichment floods when its public proxy is down 2026-06-01 13:10:51 -07:00
amazon_worker.py Enrichment manager: 'process this group first' + refined hero header 2026-06-02 19:45:04 -07:00
api_call_tracker.py Wire Amazon Music as a first-class download source 2026-05-16 09:40:50 -07:00
archive_pipeline.py fix(downloads): harden album bundle staging 2026-05-20 21:39:06 -07:00
artist_source_detail.py Fix MusicBrainz artist detail showing MBID as name 2026-05-19 12:34:35 -07:00
artist_source_lookup.py Fix artist-detail showing wrong artist when a source id is duplicated 2026-06-05 07:24:51 -07:00
audiodb_client.py Add API Rate Monitor dashboard with real-time speedometer gauges 2026-04-02 10:46:43 -07:00
audiodb_worker.py Tighten artist matching: 0.85 gate + shared uniqueness guard 2026-06-05 10:01:17 -07:00
auto_import_worker.py #889: FIX data loss — re-identify to the same release no longer deletes the file 2026-06-18 17:13:46 -07:00
automation_engine.py Automations: run each as its OWNER profile in the background (part 1 of per-profile sync) 2026-06-10 15:37:56 -07:00
connection_detect.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
connection_test.py Spotify (no-auth): report connected + 'Spotify (no-auth)' test result instead of a Deezer fallback 2026-06-13 10:16:23 -07:00
database_update_health.py #859: DB-update stall watchdog + UI self-heal (no more wedged 'Starting...' / frozen bar) 2026-06-12 19:38:30 -07:00
database_update_worker.py Run auto-reconcile as a scan phase inside the running window 2026-06-05 19:10:31 -07:00
db_integrity.py DB backups: verify integrity + never evict the last good backup 2026-05-30 21:13:04 -07:00
debug_info.py Fix Copy Debug Info music_source + surface missing services 2026-05-12 16:47:55 -07:00
deezer_client.py Lint: log instead of bare except-pass in two best-effort paths (ruff S110) 2026-06-21 22:47:11 -07:00
deezer_download_client.py deezer: playlist-write via the ARL gw-light gateway (#945, increment 3) 2026-06-28 21:15:04 -07:00
deezer_worker.py #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
discogs_client.py Cache artist album lists across metadata sources 2026-06-11 22:33:04 +02:00
discogs_worker.py Discogs: fix master/release ID collision fetching the wrong album 2026-06-10 15:01:31 +02:00
download_orchestrator.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
genius_client.py Genius 429 backoff: fail-fast gate instead of napping the import pipeline 2026-06-06 20:51:33 -07:00
genius_worker.py Enrichment manager: 'process this group first' + refined hero header 2026-06-02 19:45:04 -07:00
genre_filter.py Expand default genre whitelist from 223 to 272 genres 2026-04-18 20:32:49 -07:00
hifi_client.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
hydrabase_client.py feat: add explicit badges to discography modal and artist-detail cards 2026-05-12 23:35:08 -04:00
hydrabase_worker.py Improve graceful shutdown and rollback safety 2026-04-12 15:17:18 +03:00
image_cache.py Image cache: reject truncated downloads instead of caching broken covers (#750) 2026-05-30 17:45:01 -07:00
itunes_client.py #918 follow-up: self-heal stale truncated iTunes album-tracks cache 2026-06-24 09:37:14 -07:00
itunes_worker.py #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
jellyfin_client.py jellyfin scan: page the bulk fetch so the no-progress watchdog can't false-stall 2026-06-28 13:41:38 -07:00
lastfm_client.py Add Last.fm Track Radio to Discover page 2026-04-14 23:41:36 -07:00
lastfm_worker.py Enrichment manager: 'process this group first' + refined hero header 2026-06-02 19:45:04 -07:00
library_reorganize.py library reorganize: add a rename-only executor (#875) — core + tests 2026-06-28 18:52:42 -07:00
lidarr_download_client.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
listenbrainz_client.py #903: re-export updates the same ListenBrainz playlist in place (no duplicates) 2026-06-22 22:36:29 -07:00
listenbrainz_manager.py Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
listening_stats_worker.py stats: normalize image URLs at cache-build time, not per /api/stats/cached read (#935) 2026-06-28 09:32:48 -07:00
lyrics_client.py Lyrics: add a "Lyrics Filler" maintenance job + lyrics option in the Re-tag tool (Sokhi) 2026-06-07 17:27:52 -07:00
matching_engine.py fix: treat colon as separator in normalize_string so T:T matches T_T 2026-06-12 12:54:03 +02:00
metadata_service.py #922: import search labelled Spotify Free users' primary source as 'Deezer' 2026-06-24 08:43:52 -07:00
musicbrainz_client.py MusicBrainz manual search: field-scope the artist in non-strict mode (#754) 2026-05-31 10:05:24 -07:00
musicbrainz_search.py Fix: MusicBrainz artist discography capped at 25 releases 2026-05-28 12:00:05 -07:00
musicbrainz_service.py #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
musicbrainz_worker.py #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
navidrome_client.py Multi-disc (#927): capture real disc number at media-server scan time 2026-06-25 15:40:17 -07:00
personalized_playlists.py Popular Picks: fix empty result for deezer (popularity threshold scale mismatch) 2026-06-25 16:30:52 -07:00
plex_client.py Align playlists: add Plex support + cover art + modal redesign 2026-06-24 13:12:09 -07:00
profile_context.py Automations: run each as its OWNER profile in the background (part 1 of per-profile sync) 2026-06-10 15:37:56 -07:00
prowlarr_client.py feat(settings): add Prowlarr integration as indexer aggregator 2026-05-20 14:41:54 -07:00
qobuz_client.py manual search: INJECT the exact pasted-link track, don't rely on text search surfacing it (#932) 2026-06-28 12:42:20 -07:00
qobuz_worker.py Tighten artist matching: 0.85 gate + shared uniqueness guard 2026-06-05 10:01:17 -07:00
reorganize_queue.py library reorganize: wire rename-only mode through the queue (#875) 2026-06-28 19:08:42 -07:00
reorganize_runner.py library reorganize: wire rename-only mode through the queue (#875) 2026-06-28 19:08:42 -07:00
repair_worker.py lossy copy: support all lossless formats, not just FLAC (#941) 2026-06-28 12:13:05 -07:00
replaygain.py Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
runtime_state.py fix(post-processing): prevent double-claim race that fails imported tracks 2026-06-25 21:03:33 +02:00
seasonal_discovery.py Fix source artist detail navigation from discover modals 2026-05-18 13:50:10 -07:00
similar_artists_worker.py Lint: fix ruff B905 (zip strict=) + S110 (try/except/pass) in similar-artists worker + perf endpoint 2026-06-03 16:35:19 -07:00
socketio_cors.py Socket.IO CORS: handle self-review nits 2026-04-26 19:24:43 -07:00
soulid_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
soulseek_client.py feat(quality): make every audio format controllable via ranked targets 2026-06-24 23:01:20 +02:00
soulsync_client.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
soundcloud_client.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
source_ids.py Source IDs: add canonical registry, adopt at the highest-value sites 2026-05-29 12:19:59 -07:00
spotify_client.py spotify export: on-demand write-auth (restores Sync to Spotify safely, #945) 2026-06-29 08:32:19 -07:00
spotify_free_metadata.py Spotify Free: close the album-search gap via the artist's discography 2026-06-09 12:05:40 -07:00
spotify_public_api.py SpotipyFree integration confirmed working (236 tracks live); deps + meta tweak 2026-06-02 22:50:04 -07:00
spotify_public_scraper.py Diagnostics: route public-fetch logs to soulsync namespace + log HTTP status 2026-06-02 22:09:51 -07:00
spotify_token_cache.py Spotify tokens move into the database — daily Docker deauth fixed (wolf39us) 2026-06-07 12:01:33 -07:00
spotify_worker.py #887: Spotify enrichment shows 'Running (Spotify Free)' for no-auth users, not 'Not Authenticated' 2026-06-18 13:00:33 -07:00
tag_writer.py feat(verification): status vocabulary, DB column, SOULSYNC_VERIFICATION tag 2026-06-11 01:27:29 +02:00
tidal_client.py #880: retry 429 mid-walk when paginating Tidal Favorite Tracks (don't truncate) 2026-06-15 19:56:53 -07:00
tidal_download_client.py Download clients: don't crash init when the download path can't be created 2026-06-24 21:00:32 -07:00
tidal_worker.py Tighten artist matching: 0.85 gate + shared uniqueness guard 2026-06-05 10:01:17 -07:00
ui_appearance.py ui appearance: default worker orbs OFF on Firefox for first-time users 2026-06-28 16:50:22 -07:00
watchlist_scanner.py watchlist: don't fuse different editions as the same album (Sokhi: Expedition 33) 2026-06-26 20:38:10 -07:00
web_scan_manager.py MS Cin-5: Drop per-server globals — engine owns the clients 2026-05-05 18:05:45 -07:00
wishlist_service.py Extract wishlist logic into dedicated package 2026-04-28 21:17:24 +03:00
worker_utils.py #868: disambiguate same-name artists by owned-catalog overlap during enrichment 2026-06-13 14:57:17 -07:00
youtube_client.py youtube: use pasted cookies.txt instead of passing 'custom' as a browser to yt-dlp 2026-06-27 23:47:38 -07:00
youtube_cookies.py #902: YouTube Liked Music sync — paste a cookies.txt (server/Docker auth) 2026-06-22 14:41:04 -07:00
youtube_track_meta.py #863: derive YouTube track artist from music fields / -Topic channel / 'Artist - Title' instead of the playlist owner 2026-06-12 20:14:55 -07:00