A single enriched against the deluxe gets every source ID pointing at the deluxe,
so the organizer filed it as e.g. track 2 of a 10-track album. Root cause: the
canonical resolver only ever scored the editions already linked — the correct
single was never even a candidate, and the misfit deluxe scored so low (0.1,
below the 0.5 floor) that nothing got pinned and the priority-walk grabbed the
deluxe anyway.
Fix, in three tested layers:
- resolve_canonical_for_album gains a fetch_alternates seam: when no linked
edition clears the floor, it scores the source's OTHER editions of the same
release and re-picks by best fit (dedup, injected, pure).
- default_fetch_alternates lists the artist's editions and keeps the same-release
ones (edition-blind name match: Deluxe / - Single / [Remastered] all collapse),
returning their tracklists. Favors recall; the scorer is the precision gate.
- _resolve_source does the misfit check inline: it fit-scores the walked edition
and only on a clear misfit searches for a better edition, then persists the pin
on apply (Track Number Repair + future runs agree). Cost-neutral and behavior-
identical for well-fitting albums (no extra API calls); strict_source and the
#758 manual lock are never overridden.
Tests: +4 resolver (expand/no-expand/dedupe/back-compat), +7 alternates (name
matcher + fetcher over fake APIs + cap), +3 organizer end-to-end (misfit->single
+pin, well-fit->no-expand, strict->no-expand). 300 passed across the reorganize
+ canonical family, lint clean.
_resolve_source now prefers the album's pinned canonical (source, album_id) when
set, before the source-priority walk. So once an album's canonical is resolved,
reorganize agrees with Track Number Repair (Stage 4) and stops mislabelling a
standard album as deluxe (#767-Bug2).
Gated + side-effect-free: only changes behavior for albums that already carry a
canonical (none do until the populate step runs), an explicit user source pick
(strict_source) still wins over the canonical, and a failed canonical fetch
falls through to today's priority walk. So this stage is behavior-neutral until
canonical is populated.
Tests: tests/test_reorganize_canonical_source.py (4) — canonical preferred over
priority, fetch-failure falls back, strict_source ignores canonical, no-canonical
unchanged. 113 reorganize-orchestrator/tag-source/unknown-artist tests still pass
(no regression).
The reorganize preview (dry run) was physically creating destination album
folders, littering the library with empty dirs and making "changes" before the
user ever hit Apply.
Cause: preview_album_reorganize calls build_final_path_for_track purely to
COMPUTE the destination path string — but that shared helper has 9 os.makedirs
side effects (it's also the live download/import path builder, where creating
the dir is correct). So computing the preview path created "Lenka (Expanded
Edition)/" on disk.
Fix: build_final_path_for_track gains create_dirs=True; all 9 makedirs now route
through a gated helper. The reorganize PREVIEW passes create_dirs=False, so a
dry run computes the exact destination path with zero filesystem side effects.
Everything else keeps the default True:
- the download/import post-process flow (still writes files into the dir),
- retag,
- the reorganize APPLY path — verified it goes through post_process_fn (the real
pipeline → build_final_path_for_track with create_dirs=True), so live moves
still create their destination dirs. The gate only silences the dry run.
Tests: tests/imports/test_import_paths.py — create_dirs=False computes the
correct path (matching the reported "01 - The Show.flac") but writes NOTHING to
disk (not even the Transfer root); create_dirs=True still creates folders; both
yield an identical path. Updated two reorganize-orchestrator test doubles to
accept the new kwarg. 148 reorganize/paths/retag/pipeline tests pass.
Does NOT fix the second half of #767 (Expanded Edition picked over the standard
album). That is NOT a reorganizer bug: the library album row was linked to the
deluxe release at enrichment time (its stored spotify_album_id/itunes_album_id/
deezer_id points at "Lenka (Expanded Edition)"), and the reorganizer faithfully
reorganizes to whatever the album is linked to. The real fix is in album
enrichment's edition preference — tracked separately.
The Duplicate Cleaner moves de-duplicated files into <transfer>/deleted/.
If a user's media server scans the transfer folder (e.g. a /music root
holding both the library and the transfer dir), those quarantined files
get real track rows in SoulSync's DB. Reorganize is purely DB-driven —
it acts on each track's stored file_path — so it would dutifully move a
quarantined file back OUT of /deleted to the template location, exactly
what Tacobell444 reported.
We can't stop the rows from existing (they come from the media server,
which the app doesn't control), so the fix is bounded to Reorganize, as
the reporter asked: skip any track whose resolved path is under
<transfer>/deleted. Surfaced as a non-matched 'In deleted/quarantine
folder — skipped' in the preview; apply mirrors it (post-process never
runs, file left in place, counted as skipped).
Detection is anchored to the <transfer>/deleted PREFIX (not a bare
substring) so a real album like 'Deleted Scenes' is kept; falls back to
an exact 'deleted' path-segment match when transfer_dir is unavailable
(mirrors the cleaner's own 'if deleted in dirs' skip). The one
unavoidable ambiguity — an artist folder named exactly 'deleted' at the
transfer root — is pinned in a test as intentional.
Guard added once where both consumers see it: preview_album_reorganize
and the apply worker (_RunContext gains transfer_dir).
Tests: tests/test_reorganize_deleted_quarantine.py (8 unit) +
test_library_reorganize_orchestrator.py (preview + apply integration,
differential-verified they fail without the fix). 128 adjacent
reorganize tests still green.
Adds an opt-in alternative metadata source for reorganize. The
existing API path (query Spotify / iTunes / Deezer / Discogs /
Hydrabase for the canonical tracklist) stays the default and is
unchanged. The new tag mode reads each file's embedded tags as the
source of truth instead -- useful for well-enriched libraries where
API drift can produce inconsistent renames, and avoids API calls
entirely.
- New pure helper `core/library/reorganize_tag_source.py` adapts the
output of `read_embedded_tags` (the same mutagen path the audit-
trail modal uses) to the `api_album` / `api_track` shapes that
`_build_post_process_context` already consumes. Handles ID3-style
"5/12" track + disc shapes, multi-value Artists tags, year
normalization across 5 date formats, releasetype canonical tokens,
multi-artist string splits across 9 separators.
- `plan_album_reorganize` accepts `metadata_source: 'api' | 'tags'`
(default 'api') and `resolve_file_path_fn`. Tag mode branches into
a new `_plan_from_tags` that reads each track's file and produces
per-item `api_album` + `api_track` instead of a shared one.
- `_run_post_process_for_track` accepts a per-item `api_album`
override so each file's own album metadata flows through post-
process (not a single shared dict).
- `total_discs` in tag mode honors the `totaldiscs` tag and the
trailing `/N` of an ID3 `discnumber = "1/2"`. Partial-album
reorganize still routes into the correct `Disc N/` subfolder when
the tag knows the total even if not all discs are present locally.
- Bare `discnumber = "1"` no longer poisons `total_discs` -- it
carries no total signal.
- `reorganize_album` surfaces a tag-mode-specific error when no
files are readable, instead of the API-mode "run enrichment first"
message which would mislead in tag mode.
- `QueueItem.metadata_source` field, `enqueue` / `enqueue_many`
pass-through, runner injects `item.metadata_source` into
`reorganize_album`.
- `web_server.py` endpoints accept `mode` body param. Falls back to
the `library.reorganize_metadata_source` config setting, then to
'api'. Strict allowlist (api / tags) -- anything else falls back.
- Frontend: per-album modal + reorganize-all modal both grow a new
"Metadata Mode" dropdown above the source picker. Tag mode hides
the source picker (irrelevant). Choice persisted in localStorage.
Both preview + execute fetches send `mode` in body.
Tests:
- 49 boundary tests on the pure helper pin every shape: ID3 "5/12",
multi-artist split, year normalization, releasetype validation,
total_discs precedence, defensive paths.
- 6 planner-level integration tests pin the wiring: tag-mode with
good tags, partial-disc with totaldiscs tag, file missing,
some-match-some-fail, defensive resolve_file_path_fn=None,
API-mode regression guard.
- All 3171 tests pass; 52 existing reorganize tests unchanged.
Phase B of foxxify discord report. Pre-#524 manual-import bug left
some albums in the library with `artist=Unknown Artist` and `album.title
= <numeric album_id>`. Reorganize couldn't place them (no usable
metadata source ID) and emitted a generic "run enrichment first" hint
that doesn't apply — enrichment can't fix these rows. The right tool
is the existing `Fix Unknown Artists` repair job (reads file tags,
re-resolves metadata, re-tags + moves files).
Discoverability gap, not a logic gap. Reorganize now detects the bad-
metadata shape (Unknown Artist OR album.title that's a 6+ digit
numeric id) and emits a clear "run the Fix Unknown Artists repair
job" hint at both reason-emit sites (planner + executor). No
duplication of fixer logic.
WHATS_NEW entry covers both Phase A (orphan-format sibling handling,
already committed in d944a16) and Phase B since they ship in the same
PR for the same reporter.
20 new tests pin helpers + reason routing.
Discord report (Foxxify): users with the lossy-copy feature enabled
have `track.flac` AND `track.opus` side-by-side in their library.
Reorganize is DB-driven and only knows about ONE file per track
(the lossy copy). The other format used to get left behind in the
old location while the canonical moved to its new destination.
Empty-folder cleanup never fired because the source dir still had
audio.
# What was happening
1. User downloads album → SoulSync transcodes `.flac` → `.opus`,
embeds `.lrc` lyrics
2. DB row points at `.opus` (the lossy library copy)
3. User runs Library Reorganize
4. Reorganize moves `.opus` to new template path → `Artist/Album/01 Track.opus`
5. `.flac` orphan stays at old location, `.lrc` follows `.opus`
6. Source dir still has the `.flac` → cleanup skips → empty folders pile up
# Fix
`_finalize_track` now finds sibling-stem audio files at the source
BEFORE removing the canonical and moves them to the same destination
dir, preserving both formats with the canonical's renamed stem.
Two new helpers in `core/library_reorganize.py`:
- `_find_sibling_audio_files(audio_path) -> list[str]` — returns
paths to other audio files at the same directory that share the
canonical's filename stem. Excludes the canonical itself, non-
audio extensions (sidecars handled separately by
`_delete_track_sidecars`), and different-stem tracks (different
songs in the same dir).
- `_move_sibling_to_destination(sibling_src, canonical_dst) -> str`
— moves a sibling-format file to the canonical's destination dir
with the canonical's renamed stem + the sibling's original
extension. Defensive — OS errors logged at warning, return None,
doesn't raise (caller treats as best-effort).
After the fix:
1. `.opus` → moved to new dir
2. `.flac` sibling detected → moved to same new dir with same stem
3. Source `.opus` removed, `.lrc` sidecar deleted from source
4. Source dir empty → cleanup proceeds normally
5. Both formats end up paired at the new location
# Tests added (11)
`tests/test_reorganize_orphan_format_handling.py`:
- Sibling detection: finds `.flac` when `.opus` is canonical (and
symmetric direction), excludes canonical itself, excludes
different-stem tracks, excludes non-audio (`.lrc`/`.nfo`),
finds multiple siblings (3+ formats), returns empty when source
dir missing
- Sibling move: renames to canonical stem + preserves sibling
extension, creates destination dir if missing, no-op when source
already at destination, returns None on OS failure (caller
treats as best-effort)
# Verification
- 11/11 new tests pass
- 97/97 reorganize-related tests pass total (no regression in
existing helpers)
- Ruff clean
# Follow-up in same PR
Next commit: cleanup repair job for legacy "Unknown Artist /
album_id" rows from the pre-#524 manual-import bug. Reorganize
correctly leaves those alone (they're DB-broken, not file-broken),
but a separate maintenance job to find + re-enrich them is needed.
Catches the silent excepts the awk-based earlier sweeps missed:
- Bare `except:` followed by `pass` (also swallows KeyboardInterrupt
and SystemExit — actively wrong). Upgraded to `except Exception as
e: logger.debug("...: %s", e)`. ~14 sites across connection_detect,
soulseek_client, listenbrainz_manager, watchlist_scanner,
youtube_client, navidrome_client, jellyfin_client, web_server.
- `except Exception:` + pass that the awk pattern missed (e.g.
multi-line or unusual whitespace). ~31 sites across automation_engine,
database_update_worker, music_database, spotify_client, web_server,
others.
- 14 legitimate cleanup sites left silent with explicit `# noqa: S110`
+ comment explaining why (atexit handlers, finally-block conn.close
calls). Logging during shutdown can itself crash because file handles
get torn down before the handler fires.
Also enables `S110` rule in pyproject.toml so this pattern fails CI
going forward — drift fails at PR review instead of at runtime against
a wedged worker thread. Tests path keeps S110 ignored (test fixtures
legitimately use try-except-pass for cleanup).
Adds a WHATS_NEW entry to helper.js summarizing the full #369 sweep.
Verified: `python -m ruff check .` → All checks passed.
Verified: `python -m pytest tests/` → 2188 passed.
Closes#369
Four changes addressing kettui's PR #377 review comments:
1. **`_finalize_track` no longer over-counts on DB failure (🔴 bug).**
The function previously bailed on DB-update failure but
`_process_one_track` still incremented `summary['moved']`
unconditionally — overstating how many tracks the UI knows are
at their new locations. Fixed by:
- `_finalize_track` now returns ``bool`` (True only when DB row
was updated AND original was dealt with)
- Caller checks the return; on False, records as a failed track
with a clear message ("Track landed at new location but DB
update failed — file is at both old and new paths until library
scan re-indexes")
- Existing `test_db_update_failure_leaves_original_in_place` now
also asserts `moved == 0`, `failed == 1`, and that the error
message names the cause
2. **`executeReorganize` toast no longer says "undefined tracks" (🐛
bug).** `/reorganize` doesn't return `result.total` anymore (the
track count is determined server-side after planning), so the
"Reorganizing undefined tracks..." string was meaningless. Now uses
`result.message` from the backend instead.
3. **`_pollReorganizeStatus` distinguishes completed from skipped
(🟡 risk).** Backend now propagates the orchestrator's status
(`completed` / `no_source_id` / `no_album` / `no_tracks` /
`setup_failed` / `error`) into `_reorganize_state['result_status']`
so the frontend can warn appropriately. Two new helpers:
- `_classifyReorganizeOutcome(state)` — returns 'success' only
when `result_status === 'completed'` AND `failed === 0`;
'warning' otherwise
- `_formatReorganizeResultMessage(state)` — returns a message
specific to the outcome ("Reorganize skipped — album has no
metadata source ID. Run enrichment first." for `no_source_id`,
etc.)
Zero-failure non-completed runs now show as warnings instead of
green checkmarks.
4. **Bulk mode no longer counts skipped albums as succeeded (🟡
risk).** `_executeReorganizeAll`'s loop was treating any HTTP
200 response as success, ignoring the orchestrator's actual
outcome for that album. Fixed by:
- `_waitForReorganizeComplete()` now resolves with the final
state object (was: void)
- Loop checks `finalState.result_status === 'completed'` AND
`finalState.failed === 0` before counting `succeeded++`;
otherwise increments `skipped` (with a per-album warning
toast) or `failed` accordingly
- Final summary toast now reads
"Reorganized N of M albums, K skipped, J failed" and only
shows green when nothing was skipped or failed
All four addressed in a single commit because they form one
coherent UX-correctness fix — the bug bug (#1) and the count-
overstatement bug (#4) both made the user see "everything succeeded"
when reality was different. Together they make the UI honestly
reflect what actually happened.
Files:
- core/library_reorganize.py — `_finalize_track` returns bool,
`_process_one_track` reads it
- web_server.py — `_reorganize_state['result_status']` populated
from orchestrator's summary on success and on exception
- webui/static/library.js — `_classifyReorganizeOutcome` /
`_formatReorganizeResultMessage` helpers, single-album +
bulk-mode flows both consume them
- tests/test_library_reorganize_orchestrator.py — strengthened
the existing DB-failure test to assert moved/failed counts
Credit: kettui — four PR #377 review comments named all of these
precisely with line numbers and severity.
Reported on Discord by winecountrygames. The library "Reorganize" tool
had several layered bugs that all traced to the same root cause: the
endpoint reinvented every wheel post-processing already turns — its own
template engine, its own disc-number resolution from file tags, its own
sidecar sweep, its own collision detection — and each had drifted from
the canonical path used by fresh downloads. Reported symptoms:
- 3-disc Aerosmith deluxe collapsed to a flat single-disc layout
- Half the tracks on other albums silently skipped, no error / no count
- Re-runs left empty leftover album folders cluttering the artist dir
Architecture: stop reinventing wheels. Route reorganize through exactly
the same pipeline downloads use. Per-album:
1. Fetch the canonical tracklist from a metadata source (Spotify /
iTunes / Deezer / Discogs / Hydrabase) using the album's stored
source IDs. New `core/library_reorganize.py::plan_album_reorganize`
does this — primary-source-first, fall through priority chain
unless the user picked a specific source in the modal (strict mode).
2. For each local track, find the matching API entry via a scored
candidate matcher. Score components: exact-title (100),
substring-with-length-ratio (40-90), track-number agreement (20).
Hard reject when the two titles have different version
differentiators (Remix vs no-remix means different recordings,
not annotation drift). Below threshold = unmatched, surfaced as
"not in source's tracklist, left in place" rather than silently
mis-routing.
3. Copy the file to a per-album staging directory, build the same
context dict the import flow builds (`spotify_album` /
`track_info` / etc. with `is_album_download=True` so the path
builder enters ALBUM mode, not SINGLE mode), call
`_post_process_matched_download(...)` — same function fresh
downloads use. Post-process handles tagging, multi-disc subfolder
decisions, sidecar regeneration, AcoustID verification.
4. Read `context['_final_processed_path']` to learn where it landed.
Update `tracks.file_path` in the DB BEFORE removing the original
(DB-update failure leaves the file at both locations, recoverable
via library scan; the reverse would orphan the row). Delete
per-track sidecars (post-process recreates them at the new
destination).
3 concurrent workers per album via ThreadPoolExecutor, matching the
download path's per-batch worker count. State mutations all guarded by
a single lock; staging filenames carry a UUID prefix so concurrent
copies of identically-named source files don't overwrite each other.
Source picker in the modal lets the user choose which source to read
the tracklist from. Two endpoints feed it:
- `/api/library/album/<id>/reorganize/sources` — sources for THIS
album that are both authed AND have a stored ID. For the per-
album modal.
- `/api/library/reorganize/sources` — all authed sources globally.
For the bulk "Reorganize All" modal where per-album ID coverage
varies.
When the user picks a specific source, the orchestrator runs in
`strict_source=True` mode (no fallback chain) — picking Spotify means
"use Spotify or fail", not "use Spotify and silently fall back."
Preview endpoint shares the same planning logic as apply via
`preview_album_reorganize` — the destination path comes from the same
`_build_final_path_for_track` post-process uses, so what you see in
the preview is exactly what you get on apply.
Empty destination folders (from earlier failed runs OR from the
current run when post-process creates a dir then fails AcoustID)
get cleaned up after each successful run: walk up to the artist
folder from any successful destination, prune empty album-sibling
folders one level deep. Bounded scope = won't touch unrelated user
dirs.
Web_server.py shrinks by ~450 net lines. The endpoint handler is now
a thin wrapper that builds injected callables (path resolver, post-
process function, DB updater, empty-dir cleaner), spawns a thread
that calls `reorganize_album()`, and returns. All actual logic lives
in `core/library_reorganize.py` where it's unit-testable without
spinning up Flask.
Frontend cleanup: the per-call template input in both reorganize
modals (per-album and bulk) was redundant — the backend always uses
the configured global download template. Removed the input and the
variables-grid reference UI it was for.
39 new unit tests pin every contract:
- source resolution (no_source_id when album has none, fallthrough
chain when primary returns nothing, strict mode bypasses fallback)
- matcher scoring (exact / substring / multi-disc disambiguation /
smart-quote tolerance / dash-vs-parens / bonus-track substring /
Remix-vs-original differentiator rejection / "Real" doesn't false-
match "Real Real Real" / track-number-only no longer fires)
- file safety (DB-update failure leaves original in place, post-
process failure leaves original in place, post-process exception
caught and original preserved, success removes original AND
updates DB in the right order)
- sidecar handling (per-track .lrc/.nfo deleted on success, kept on
failure; album-level cover.jpg/folder.jpg cleaned only when
directory has no remaining audio)
- staging cleanup (recreated between tracks because post-process
nukes it, dir cleaned up on success AND on failure)
- destination-dir prune (empty siblings removed, real album with
files preserved, no recursive sweep)
- source picker (only authed-with-stored-ID sources for per-album,
all authed sources for bulk; strict mode doesn't fall back)
- concurrency (3 workers in flight, state stays consistent under
races, stop_check cuts off pending tasks)
- preview parity (preview produces same destination as apply for
multi-disc; ALBUM mode not SINGLE mode; unmatched/no-path tracks
surfaced with reasons)
Limitations (deliberate punts, NOT in this PR):
- Renamed local titles on multi-disc albums where track_number
also disagrees: matcher returns nothing (track is "not in
source"). Fixable by using duration_ms as a tertiary signal.
- Per-track in-modal source switching with per-album track-count
hints (would need a second API call before opening the modal).
- UI status panel on the artist page during a run — currently
just toasts. Documented as a follow-up PR.
Files:
- core/library_reorganize.py — new module: plan_album_reorganize,
preview_album_reorganize, reorganize_album, available_sources_for_album,
authed_sources, _score_candidate, helpers for staging/post-
processing/finalizing, sidecar + dest-dir cleanup
- core/metadata_service.py — no changes; reused get_album_for_source,
get_album_tracks_for_source, get_source_priority,
get_client_for_source
- web_server.py — three endpoints (preview / apply / sources GETs)
are thin wrappers; -450 net lines
- tests/test_library_reorganize_orchestrator.py — 39 tests covering
every contract above
- webui/static/library.js — source picker UI in both modals; dead
template input + variables-grid removed
- webui/static/style.css — dropdown option styling fix (white-on-
white was unreadable)
Reported on Discord by winecountrygames — his bug report named the
trigger button (Enhanced view → Reorganize All) and both symptoms
(multi-disc collapse, half-album skip), which let the diagnosis go
straight to the architectural problem.
BoulderBadgeDad