danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
soulsync/core
History
Broque Thomas 5771c5ba77 Album-bundle staging: clean Soulseek copies + sweep orphans at startup 
Two related leaks in ``storage/album_bundle_staging/<batch_id>/``:

1. **Soulseek bundle cleanup was excluded.** The per-batch cleanup
   at the end of a bundle download gated on:
       (album_bundle_source or '').lower() in ('torrent', 'usenet')
   The comment justified it as "slskd keeps its own completed
   folders" — but the Soulseek bundle path ALSO copies completed
   files into the private staging dir (``soulseek_client.py:1599``,
   ``copy_audio_files_atomically(completed, Path(staging_dir))``)
   for the per-track workers to claim. Those copies persisted
   forever; long-running installs accumulated stale GB. Extended
   the cleanup gate's allow-list to include ``soulseek`` so the
   per-batch dir is removed on bundle completion — same code path
   that already worked for torrent / usenet.

2. **No sweep for orphan dirs.** Any leftover ``<batch_id>``
   subdir from a previous-session crash, an errored batch, or a
   pre-fix Soulseek bundle stayed on disk forever. Added
   ``sweep_orphan_album_bundle_staging(staging_root, active_batch_ids)``
   that runs ONCE at server startup, before any batch can register
   a staging dir. Removes every ``<batch_id>``-shaped subdir
   whose id isn't in the active set. Safe by construction:
     - Only touches subdirs of the configured staging root.
     - Name-shape check (``entry.name == _safe_batch_dirname(entry.name)``)
       rejects hand-placed dirs like ``.git`` or stray docs.
     - ``shutil.rmtree`` errors log + continue — sweep must not
       crash app startup over a permission glitch.
     - active_batch_ids normalised through ``_safe_batch_dirname``
       so colon-bearing batch_ids match their on-disk form.
   Wired into the web_server startup right after the stuck-flags
   diagnostic so it fires before anything else touches batches.

Tests
- ``test_downloads_lifecycle.py`` gained one regression test
  pinning that Soulseek bundles now have their staging dir
  cleaned (sibling to the existing torrent test).
- ``test_album_bundle_staging_sweep.py`` (NEW, 11 tests)
  covers: orphan removal with no actives, active dirs preserved,
  special-char batch_id normalisation, no-op on missing /empty
  /empty-string staging root, non-dir entries skipped, unsafe-
  name dirs preserved (.git etc.), partial rmtree failure doesn't
  abort the rest, listdir failure returns 0 cleanly, default
  None active set, defensive against empty / None entries in
  the active set.

488 downloads tests pass.

For users with an existing "clean up old files" automation pointed
at this dir: stop pointing it there if you want — the auto-cleanup
+ startup sweep cover it now. Or leave it as belt-and-suspenders
with a relaxed (1h+) mtime threshold so it can't race a mid-batch
download.
2026-05-27 22:18:42 -07:00
..
artists Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
automation Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
discovery Wishlist: fix three regressions causing all imports to land as track 01 with no year 2026-05-27 15:39:22 -07:00
download_engine Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
download_plugins Address kettui-flagged items on usenet poll fix (#706) 2026-05-27 10:05:37 -07:00
downloads Album-bundle staging: clean Soulseek copies + sweep orphans at startup 2026-05-27 22:18:42 -07:00
enrichment Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
imports Wishlist: fix three regressions causing all imports to land as track 01 with no year 2026-05-27 15:39:22 -07:00
library fix(musicbrainz): user-facing search recall + album-detail 404 2026-05-19 15:38:24 -07:00
matching AcoustID + quarantine modal: three bug fixes (closes #607, closes #608) 2026-05-15 08:55:06 -07:00
media_server Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
metadata MB rerank: prefer_known_duration is now a score boost, not a tiebreaker 2026-05-27 08:13:28 -07:00
personalized Personalized pipeline: auto-refresh stale snapshots after watchlist scan 2026-05-15 20:53:03 -07:00
playlists Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
repair_jobs Fix Picard albumartist orphan false positives 2026-05-22 08:43:57 -07:00
search feat(import): show MusicBrainz variants 2026-05-24 21:29:22 +03:00
stats Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
streaming Cin-6: Rename soulseek_client global → download_orchestrator 2026-05-04 23:23:32 -07:00
sync Persist Find & Add selections as permanent server-playlist match overrides 2026-05-14 09:39:24 -07:00
text Address Cin review: extract helpers, indexed pool fetch, tidy nits 2026-05-24 23:33:55 -07:00
torrent_clients fix(torrent): use before/after diff to recover qBit info-hash 2026-05-20 18:11:30 -07:00
usenet_clients Address kettui-flagged items on usenet poll fix (#706) 2026-05-27 10:05:37 -07:00
watchlist Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
webui Extract WebUI asset helpers 2026-05-13 22:26:25 +03:00
wishlist Wishlist: harden Spotify backfill — poisoned tn=1 can't mask lean album 2026-05-27 16:47:26 -07:00
workers Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
acoustid_client.py Clean up 286 ruff lint errors to unblock CI and fix 10 latent bugs 2026-04-21 13:30:52 -07:00
acoustid_verification.py AcoustID + quarantine modal: three bug fixes (closes #607, closes #608) 2026-05-15 08:55:06 -07:00
album_consistency.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
amazon_client.py fix(amazon): search albums/artists and track numbers for t2tunes 2026-05-19 14:04:26 -07:00
amazon_download_client.py Amazon download client: write final size==transferred before returning file path 2026-05-16 11:09:04 -07:00
amazon_worker.py Harden Amazon worker schema migration 2026-05-18 15:47:04 -07:00
api_call_tracker.py Wire Amazon Music as a first-class download source 2026-05-16 09:40:50 -07:00
archive_pipeline.py fix(downloads): harden album bundle staging 2026-05-20 21:39:06 -07:00
artist_source_detail.py Fix MusicBrainz artist detail showing MBID as name 2026-05-19 12:34:35 -07:00
artist_source_lookup.py Add amazon_id column to artists table for full source parity 2026-05-16 17:06:54 -07:00
audiodb_client.py Add API Rate Monitor dashboard with real-time speedometer gauges 2026-04-02 10:46:43 -07:00
audiodb_worker.py AudioDB worker: stop infinite loop on direct-ID lookup failure (#553) 2026-05-11 14:18:55 -07:00
auto_import_worker.py Auto-import: fall through to other metadata sources when primary returns no match 2026-05-12 12:32:18 -07:00
automation_engine.py Wire automation engine through next_run_at + register monthly_time (PR 2/4) 2026-05-27 12:03:41 -07:00
connection_detect.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
connection_test.py Add torrent and usenet release staging support 2026-05-21 14:22:21 -07:00
database_update_worker.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
debug_info.py Fix Copy Debug Info music_source + surface missing services 2026-05-12 16:47:55 -07:00
deezer_client.py Fix Deezer contributors tagging silently dropping for cache-polluted tracks 2026-05-14 11:10:51 -07:00
deezer_download_client.py Surface silent exceptions in metadata clients — 37 sites 2026-05-07 09:33:38 -07:00
deezer_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
discogs_client.py Discogs: strip artist disambiguation suffixes at every name surface (#634) 2026-05-27 09:05:47 -07:00
discogs_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
download_orchestrator.py Fix Usenet settings reload without restart 2026-05-22 08:28:56 -07:00
genius_client.py Add API Rate Monitor dashboard with real-time speedometer gauges 2026-04-02 10:46:43 -07:00
genius_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
genre_filter.py Expand default genre whitelist from 223 to 272 genres 2026-04-18 20:32:49 -07:00
hifi_client.py Log exception when inferring HiFi manifest ext 2026-05-21 18:24:56 -07:00
hydrabase_client.py feat: add explicit badges to discography modal and artist-detail cards 2026-05-12 23:35:08 -04:00
hydrabase_worker.py Improve graceful shutdown and rollback safety 2026-04-12 15:17:18 +03:00
image_cache.py fix(lint): silence ruff B007 + S110 2026-05-20 16:18:55 -07:00
itunes_client.py feat: add explicit badges to discography modal and artist-detail cards 2026-05-12 23:35:08 -04:00
itunes_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
jellyfin_client.py Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
lastfm_client.py Add Last.fm Track Radio to Discover page 2026-04-14 23:41:36 -07:00
lastfm_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
library_reorganize.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
lidarr_download_client.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
listenbrainz_client.py Remove emojis from all Python log and print statements 2026-04-11 21:11:02 -07:00
listenbrainz_manager.py Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
listening_stats_worker.py Final silent-exception sweep + ruff S110 lint guardrail — ~45 sites 2026-05-07 11:16:06 -07:00
lyrics_client.py Retag now re-embeds LYRICS tag instead of leaving it empty 2026-05-14 15:52:05 -07:00
matching_engine.py Merge remote-tracking branch 'origin/dev' into refactor/media-server-engine 2026-05-05 20:36:31 -07:00
metadata_service.py Add MusicBrainz as a metadata source 2026-05-18 18:47:13 -07:00
musicbrainz_client.py Show MusicBrainz release variants in import 2026-05-24 09:33:19 -07:00
musicbrainz_search.py Branch cleanup: lift manual-match helpers, fix length-pref ordering, profile-scope view toggle 2026-05-27 07:43:21 -07:00
musicbrainz_service.py Cross-script artist aliases: include canonical name + non-strict fallback 2026-05-14 13:07:15 -07:00
musicbrainz_worker.py Alias resolution polish: lazy-fire on direct-match failure + worker backfill 2026-05-10 17:02:02 -07:00
navidrome_client.py Preserve Navidrome album cover art 2026-05-25 19:51:38 -07:00
personalized_playlists.py Add MusicBrainz as a metadata source 2026-05-18 18:47:13 -07:00
plex_client.py Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
prowlarr_client.py feat(settings): add Prowlarr integration as indexer aggregator 2026-05-20 14:41:54 -07:00
qobuz_client.py Fetch all Qobuz favorite tracks for discovery 2026-05-24 01:17:38 -07:00
qobuz_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
reorganize_queue.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
reorganize_runner.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
repair_worker.py fix(album-completeness): block cross-artist auto-fill 2026-05-21 16:49:12 -07:00
replaygain.py Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
runtime_state.py Add module logger + surface silent exceptions in 7 logger-less files — 12 sites 2026-05-07 10:27:04 -07:00
seasonal_discovery.py Fix source artist detail navigation from discover modals 2026-05-18 13:50:10 -07:00
socketio_cors.py Socket.IO CORS: handle self-review nits 2026-04-26 19:24:43 -07:00
soulid_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
soulseek_client.py Fix: Soulseek album-bundle downloads stuck on "failed" after slskd 2026-05-27 21:20:37 -07:00
soulsync_client.py Surface silent exceptions across remaining modules — ~70 sites 2026-05-07 10:28:58 -07:00
soundcloud_client.py Surface engine-not-wired errors + exclude soulseek from monitor aggregation 2026-05-05 12:20:51 -07:00
spotify_client.py Add download buttons + bulk action to artist top-tracks sidebar 2026-05-07 15:44:47 -07:00
spotify_public_scraper.py Add Spotify Link tab for public playlist/album scraping without API credentials 2026-03-15 21:25:05 -07:00
spotify_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
tag_writer.py AcoustID scanner: multi-candidate match + duration guard + multi-value retag 2026-05-14 14:09:38 -07:00
tidal_client.py Your Albums: selectable wishlist modal + Tidal album resolution 2026-05-11 12:36:16 -07:00
tidal_download_client.py Handle non-JSON Tidal auth responses 2026-05-20 14:04:45 -07:00
tidal_worker.py Surface silent exceptions in workers + repair jobs — ~30 sites 2026-05-07 10:27:24 -07:00
watchlist_scanner.py Fix missing album art for non-Spotify sources + animate Downloads nav icon 2026-05-18 20:24:13 -07:00
web_scan_manager.py MS Cin-5: Drop per-server globals — engine owns the clients 2026-05-05 18:05:45 -07:00
wishlist_service.py Extract wishlist logic into dedicated package 2026-04-28 21:17:24 +03:00
worker_utils.py Repair stale media schema during refresh 2026-05-21 17:41:54 -07:00
youtube_client.py Escape dash-leading YouTube search queries 2026-05-24 08:54:47 -07:00