Discord report: 16 Bandcamp indie albums sat in staging because
auto-import couldn't identify them, but the manual search bar at the
bottom of the Import Music tab found the same albums fine. Trace:
`_search_metadata_source` only queried `get_primary_source()` — single
source, no fallback. Meanwhile `search_import_albums` (manual search bar)
already iterated `get_source_priority(get_primary_source())` and broke
on the first source with results. Asymmetric behavior, same album: manual
worked, auto-import didn't.
Fix: lift `_search_metadata_source` to use the same source-chain pattern.
Try primary first; if it returns nothing OR scores below the 0.4
threshold, fall through to the next source in priority order. First
source producing a strong-enough match wins. Result dict carries the
`source` that actually matched (not the primary name) so downstream
`_match_tracks` calls the right client. Defensive per-source try/except
so a rate-limited or auth-failed source doesn't abort the chain.
Unconfigured sources (client=None) silently skipped.
Cin-shape lift: scoring math extracted to pure `_score_album_search_result`
helper so the weight tweaks (album 50% / artist 20% / track-count 30%)
are pinned at the function boundary, independent of the orchestrator
(per-source iteration, exception containment, threshold check). Weight
constants exposed at module level (`_ALBUM_NAME_WEIGHT`,
`_ARTIST_NAME_WEIGHT`, `_TRACK_COUNT_WEIGHT`) — greppable, bumpable in
one place. Pre-extraction these were magic numbers inline.
27 new tests:
- 9 integration tests in `test_auto_import_multi_source_fallback.py`:
primary-success path unchanged (no fallback fires, only primary client
called), primary-empty falls through, primary-weak-score falls through,
first fallback success stops the chain (no wasted API calls on
remaining sources), all-sources-fail returns None, per-source
exception contained, unconfigured-source skipped, result `source`
field reflects winning source, `identification_confidence` from
winning source.
- 18 helper tests in `test_album_search_scoring.py`: weights sum to
1.0, album weight dominant (invariant pin), perfect-match returns 1.0,
per-component contribution (album / artist / track-count), Bandcamp
vs streaming track-count mismatch (7-files vs 4-tracks case still
scores ~0.87 above threshold), zero-track-count and zero-file
guards, huge-mismatch non-negative guard, list-of-strings artist
shape, missing `.name` / `.artists` / `None` total_tracks edge cases.
Backwards compatible: single-source users see no change — chain just
has one entry. Existing test `test_search_metadata_source_extracts_artist_id_from_dict_artist`
needed one extra patch line for `get_source_priority`.
Full pytest sweep: 2754 passed.
Two pre-existing parity gaps in `record_soulsync_library_entry` that
the prior parity commits left untouched. Both close real holes
between auto-import writes and what the soulsync_client deep scan
would have produced.
# Gap 1: Album duration was the first-imported track's duration
`record_soulsync_library_entry` is called once per track. The album
INSERT only fires for the FIRST track of a new album (subsequent
tracks find the album row already exists). The INSERT was passing
`duration_ms` — `track_info["duration_ms"]` — as the album's
`duration` column. That's the duration of one track, not the album
total. Compare to `SoulSyncAlbum.duration` in soulsync_client which
is `sum(t.duration for t in self._tracks)`.
Fix:
- Worker computes `album_total_duration_ms = sum(...)` across every
matched track and threads it onto context as
`album.duration_ms`.
- side_effects reads that value (or falls back to the per-track
duration for legacy non-auto-import callers) and writes it as the
album row's `duration`.
# Gap 2: Re-imports of the same artist/album were insert-only
When the SELECT-by-id or SELECT-by-name found an existing soulsync
artist or album row, the function skipped completely — no UPDATE
path. Meant: artist genres / thumb / source-id reflected ONLY
whatever the FIRST imported album supplied, never refreshing as
more albums by that artist landed. Ten more imports later, the
artist row still held whatever the first random import wrote.
Conservative fix: when an existing row matches, run an UPDATE that
fills only the columns whose current value is NULL or empty. Never
overwrites populated values — protects manual edits +
enrichment-worker writes the same way the scanner UPDATE path
preserves enrichment columns.
Implementation note: the empty-check happens in Python, NOT SQL.
Initial pass tried `COALESCE(NULLIF(col, ''), NULLIF(col, 0), ?)`
but SQLite's `NULLIF(text_col, 0)` returns the original text value
instead of NULL — different types, no coercion. So the SQL-only
conditional was unreliable on text columns. New helper does
`SELECT cols FROM table WHERE id`, compares each column in Python,
and emits UPDATE clauses only for the ones that need filling.
Allowlist defense: f-string column names go through
`_SOULSYNC_FILLABLE_COLUMNS` validation before interpolation.
Misuse adding new columns without an allowlist update fails closed
(logger.debug + skip).
# Tests added (4)
- `test_album_duration_uses_album_total_not_single_track` —
album with single-track context carrying explicit
`album.duration_ms = 2_500_000` writes 2_500_000 to the album row,
not the per-track 200_000 fallback.
- `test_re_import_fills_empty_artist_fields` — first import lands
artist with empty thumb + empty genres; second import for same
artist with thumb + genres present updates the existing row.
- `test_re_import_does_not_clobber_populated_artist_fields` —
first import writes rich genres + thumb; second import with
worse / different metadata leaves the existing row untouched.
- `test_re_import_fills_empty_source_id_when_missing` — first
import had no source artist ID; second import does — fills the
empty `spotify_artist_id` column on the existing row.
# Verification
- 10/10 side-effects tests pass (including 4 new + 4 from prior
parity commit + 2 history/provenance)
- 217 imports tests pass (no regression)
- 2369 full suite passes (+4 from prior, +22 PR-total from baseline 2347)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated)
- Ruff clean
Cin pre-review followup. Two small parity gaps the prior commits left
open:
# 1. Genre tags land on the standalone artists row
`soulsync_client._scan_transfer` aggregates the GENRE tag across every
track in an album and surfaces it on `SoulSyncAlbum.genres` (which the
DatabaseUpdateWorker writes to the artists+albums row). Auto-import
was hardcoding `'spotify_artist': {'genres': []}` so the imported
artists row landed with empty genres — felt hollow compared to a
Plex/Jellyfin scan, which both pull genres from their respective APIs.
Fix:
- `_read_file_tags` now reads the GENRE tag (mutagen easy mode handles
MP3/FLAC/M4A consistently; some files carry multiple genres so it's
always returned as a list).
- `_process_matches` aggregates genres from each matched file's tags
into a deduped insertion-order list. Dedup is case-insensitive but
preserves original casing — so "Hip-Hop, Rap, Trap" reads naturally
in the JSON column instead of "hip-hop, rap, trap".
- Worker context's `spotify_artist['genres']` carries the aggregated
list, which `record_soulsync_library_entry` already filters via
`core.genre_filter.filter_genres` and writes to the artists row.
# 2. Defensive str() cast for ISRC + MBID
`_build_album_track_entry` already coerces ISRC + MBID to string today
(via `str(isrc) if isrc else ''`). But if a future metadata-source
client returns int / None for either ID, the worker would propagate
the wrong type and side_effects.py's `.strip()` would AttributeError.
Cheap insurance: explicit `str()` cast in the worker before assignment
to track_info. Future-proofs against client drift.
# Tests added (3, in test_auto_import_context_shape.py):
- `test_context_aggregates_genres_from_track_tags` — multi-file
album with overlapping genre lists produces deduped, insertion-
ordered, original-case-preserved result. Stubs `_read_file_tags`
with monkeypatch so we don't need real audio.
- `test_context_genres_empty_when_no_tags` — files without GENRE
tag → empty list. Standalone library write handles gracefully
(genres column stays empty / NULL).
- `test_context_isrc_mbid_coerced_to_string` — hostile types
(int 12345678, None, int 999) coerced to safe strings before
reaching track_info.
# Verification
- 14/14 context-shape tests pass (11 prior + 3 new)
- 213 imports tests pass (no regression)
- 2365 full suite passes (+3 from prior, +18 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
Cin pre-review followup to the standalone library parity commit. The
prior commit fixed `spotify_artist['id']` from the wrong copy-paste
value (`identification['album_id']`) to read from
`identification['artist_id']`, but the identification dict produced
by `_search_metadata_source` and `_search_single_track` never set
`artist_id` — both extracted artist NAME from the search response
and discarded the source ID sitting right next to it. Net effect of
the prior commit: artists row source-id stayed NULL, just for a more
honest reason than before.
Now properly extracted:
- `_search_metadata_source` reads `best_result.artists[0]['id']`
alongside the artist name and returns it on the identification dict
as `artist_id`.
- `_search_single_track` does the same for single-track identification.
- `_identify_single`'s tag-based-confidence path forwards
`result.get('artist_id')` so the artist source-id propagates even
when high-confidence local tags override the search result's name.
Result: identification dict now carries `artist_id` whenever the
metadata source returned an artist with an ID. The worker context
already plumbs it onto `spotify_artist['id']` and
`spotify_album['artists'][0]['id']`, so the standalone library write
finally populates `<source>_artist_id` on the artists row.
Tests added (3, in `test_auto_import_context_shape.py`):
- `test_context_artist_id_uses_identification_artist_id` — when the
identification dict carries `artist_id`, context propagates it
onto `spotify_artist['id']` AND
`spotify_album['artists'][0]['id']`. Pins that the prior copy-
paste bug (artist['id'] = album_id) doesn't return.
- `test_context_artist_id_is_empty_when_identification_missing_it` —
fallback case (filename-only identification): context gets empty
string, NOT album_id. Honest failure mode.
- `test_search_metadata_source_extracts_artist_id_from_dict_artist`
— black-box test of `_search_metadata_source`: feed it a
spotify-shaped result with `artists[0]['id']` and verify
identification dict carries it forward.
Verification:
- 11/11 context-shape tests pass (8 prior + 3 new)
- 210 imports tests pass (no regression)
- 2362 full suite passes (+3 from prior commit, +15 PR-total)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation)
- Ruff clean
# Background
SoulSync standalone is meant to be a full replacement for Plex /
Jellyfin / Navidrome — files imported via auto-import (or any other
import path) should land in the database with the same field richness
a media-server scan would write. They weren't.
# Gaps fixed
The auto-import worker built a context dict for each track and handed
it to `_post_process_matched_download` (the same callback the regular
download flow uses). That dict was missing three things downstream
needed:
1. **No `source` field anywhere.** `record_soulsync_library_entry`
reads `get_import_source(context)` to pick the source-aware ID
columns (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
etc.) on the artists / albums / tracks rows. With no source, the
resolver returned an empty string → `get_library_source_id_columns("")`
returned an empty dict → the `UPDATE tracks SET <source>_id = ?`
blocks were silently skipped. Result: every auto-imported track
landed with NULL on every source-id column. Watchlist scans
(which match by stable source IDs to detect "this track is already
in library") couldn't recognise these rows and would re-download
them on the next pass.
2. **No `_download_username='auto_import'`.** Both
`record_library_history_download` and `record_download_provenance`
default to "Soulseek" when no `username` is in the context. Every
staging-folder import was being labelled as a Soulseek download
in library history + provenance — false signal in the UI.
3. **No per-recording IDs (`isrc`, `musicbrainz_recording_id`) on
track_info.** The Navidrome scanner already writes
`musicbrainz_recording_id` directly to the tracks row when present.
Picard-tagged libraries always carry MBID; metadata sources
(Spotify via MusicBrainz enrichment, Deezer, etc.) carry ISRC.
Auto-import had access to both via the metadata-source response
but didn't propagate them — so the soulsync row went in with
NULL on both columns.
# Changes
**`core/auto_import_worker.py` — `_process_matches`:**
- Top-level `'source': source` (from `identification['source']`)
- `'_download_username': 'auto_import'`
- `track_info['isrc']`, `track_info['musicbrainz_recording_id']` —
pulled from the per-track payload returned by the metadata source
- `track_info['album_id']` — back-reference so source-aware ID
resolution works on sources whose API nests album under
`track.album.id` rather than `track.album_id`
- `spotify_artist['id']` now correctly carries the artist's source ID
(was `identification['album_id']`, a copy-paste bug from the
original implementation that made artist-id resolution fall back
to fuzzy matching)
- `spotify_album['artists'][0]['id']` carries artist source ID for
the same resolution path
**`core/imports/side_effects.py`:**
- `record_library_history_download` source_map: add
`"auto_import": "Auto-Import"` — tags imported tracks correctly
- `record_download_provenance` source_service: add
`"auto_import": "auto_import"` — provenance shows real source
- `record_soulsync_library_entry` track INSERT: now includes
`musicbrainz_recording_id` + `isrc` columns (matches
`insert_or_update_media_track`'s shape for Navidrome /
Plex / Jellyfin scans). Both default to NULL when not present.
# Behavior preserved
- Files still land in the same library template path (no path-build
change)
- Other media-server flows (Plex / Jellyfin / Navidrome users)
unaffected — `record_soulsync_library_entry` still gates on
`get_active_media_server() == "soulsync"`. Auto-import on those
servers continues to drop the file in the library folder + emits
`batch_complete` for the scan-trigger automation, same as before.
- Direct downloads (search → Download button) unaffected — they
already passed `source` + `username` correctly.
# Tests added
`tests/imports/test_auto_import_context_shape.py` (8 tests, new file):
- Worker context carries `source` for every metadata source
(parametrised across spotify / deezer / itunes / discogs)
- `_download_username='auto_import'` set unconditionally
- ISRC + MBID propagate from track payload to track_info when present
- ISRC + MBID default to empty string when absent (downstream
normalises to NULL at write time)
- track_info includes album-id back-reference
`tests/imports/test_import_side_effects.py` (4 new tests + 2 schema
column adds):
- `record_soulsync_library_entry` writes mbid + isrc columns when
present in track_info
- Deezer source maps to deezer_id column (regression case for
source-aware column resolver)
- `record_library_history_download` labels `_download_username=
'auto_import'` as "Auto-Import" not "Soulseek"
- `record_download_provenance` registers source_service as
"auto_import" not "soulseek"
# Verification
- 8/8 new context-shape tests pass
- 6/6 side-effects tests pass (4 new + 2 existing)
- 207 imports tests pass
- 2359 full suite passes (+12 from baseline 2347, no regressions)
- 1 pre-existing flake (`test_watchdog_warns_about_stuck_workers`,
passes in isolation, unrelated to this change)
- Ruff clean
# Concurrency model
Pre-refactor concurrency was emergent + unbounded:
- The worker's `_run` thread called `_scan_cycle` every 60s,
processing candidates synchronously in a for-loop.
- The `/api/auto-import/scan-now` endpoint spawned a fresh
`threading.Thread(target=_scan_cycle)` per click — extra parallel
scan cycles on top of the timer.
- Multiple "Scan Now" clicks during in-flight processing → multiple
threads racing on `_processing_paths` / `_folder_snapshots` state,
no upper bound on concurrent scanners.
- `stop()` didn't wait for in-flight processing — could leave file
moves / tag writes / DB inserts mid-flight.
Refactor to the pattern Cin uses elsewhere (`missing_download_executor`,
`sync_executor`, `import_singles_executor` all use
`ThreadPoolExecutor(max_workers=3, thread_name_prefix=...)`):
- **One scan thread** — both timer + manual triggers go through
`trigger_scan()`, gated by a non-blocking `_scan_lock`. Duplicate
triggers no-op instead of stacking parallel scanners.
- **Bounded executor** — `ThreadPoolExecutor` (default 3 workers,
configurable via `auto_import.max_workers`) runs per-candidate
work. Each candidate runs to completion in its own pool thread;
up to N candidates run in parallel.
- `_scan_and_submit()` is fast — just enumeration + executor submit,
returns immediately, doesn't block on per-candidate work.
- `_process_one_candidate(candidate)` holds the per-candidate logic
identical to the old for-loop body, lifted into a method so the
pool can run multiple instances concurrently.
- `_submitted_hashes` set + lock dedupes candidates across the
timer + manual triggers so a candidate already queued / running
doesn't get re-submitted.
- `stop()` calls `executor.shutdown(wait=True)` — clean shutdown,
no orphaned file ops.
# Per-candidate UI state isolation
The executor refactor opened two concurrency holes that the old
sequential model masked. Both fixed in this commit:
1. **Scalar UI fields stomped across pool workers.** Pre-refactor
`_current_folder` / `_current_status` / `_current_track_*` were
safe under the sequential model — only one candidate processed
at a time, so the fields tracked the in-flight one. With three
pool workers writing the same fields, the polling UI saw garbage
like "Processing AlbumA, track 7/14: SongFromAlbumB".
Replaced with `_active_imports: Dict[hash, _ActiveImport]` keyed
on folder_hash, gated by `_active_lock`. Each pool worker owns
its own entry. Helpers `_register_active` / `_update_active` /
`_unregister_active` / `_snapshot_active` are the only API.
2. **Stats counters not thread-safe.** `self._stats[k] += 1` is
read-modify-write — under load, parallel pool workers drop
increments. New `_stats_lock` + `_bump_stat()` helper wraps every
mutation. `get_status()` reads under the same lock and returns
a copy.
# Endpoint change
`/api/auto-import/scan-now` no longer spawns its own scan thread —
calls `auto_import_worker.trigger_scan()` (which routes through the
shared lock + executor). Multiple clicks while a scan is in flight
no-op deterministically. Endpoint still wraps the call in a daemon
thread so the HTTP response returns immediately even if the staging
walk is slow.
# Backward compat
The scalar `_current_folder` / `_current_status` / `_current_track_*`
fields are preserved as **read-only properties** that resolve to the
FIRST active import. The existing `get_status()` payload still
includes those fields populated from the first entry — single-import
UIs (and the test fixture) keep working unchanged. New
`active_imports` array exposes the full multi-candidate state for
parallel-aware UIs.
# Behavior preserved
- Per-candidate identify / match / process logic byte-identical
- Live-progress state preserved (per candidate now)
- Stability gate / already-processed dedup preserved
- `_record_in_progress` / `_finalize_result` UI rows preserved
- Tag-based loose-file grouping unchanged
# Behavior changes
- Multiple albums process IN PARALLEL up to `max_workers`
- "Scan Now" while scan in progress no-ops (was: spawned another)
- `stop()` waits for in-flight pool work via `shutdown(wait=True)`
- Auto-import card now lists each in-flight album (one line per
active import) instead of a single shared progress line
# UI
`webui/static/stats-automations.js`:
- Progress widget reads `active_imports` array, renders one line
per in-flight album with per-candidate status / track index
- Falls back to the legacy summary line when payload doesn't
carry `active_imports` (older backend)
- Per-row "live processing" lookup now matches by `folder_hash`
through the array instead of by `folder_name` against scalars
# Tests added (`tests/imports/test_auto_import_executor.py`)
- Pool config: default max_workers=3, configurable via constructor
+ via `auto_import.max_workers` config, floors at 1
- Scan lock: 5 concurrent `trigger_scan()` calls run only 1 scan
while lock held; releases properly so subsequent triggers run
- Executor dispatch: 5 candidates → 5 process calls via the pool
- Bounded parallelism: max_workers=3 caps at 3 concurrent;
max_workers=2 caps at 2
- Cross-trigger dedup: candidate submitted in scan A doesn't get
re-submitted by scan B while still in-flight
- Graceful shutdown: `stop()` blocks until in-flight pool work
finishes
- Per-candidate state isolation: 2 parallel workers updating their
own candidate state don't interfere — each candidate's
track_index / track_name / folder_name reads back exactly as
written for that hash
- `get_status()` returns coherent `active_imports` array with
one entry per in-flight candidate; aggregate top-level
`current_status` is 'processing' when any entry is processing
- Unregister removes only that candidate, others stay visible
- Stats counter thread-safety: 1000 parallel bumps land at 1000
(the read-modify-write race regresses without the lock)
- `get_status()` stats snapshot is a copy, not a live reference
# Verification
- 17 new tests pass (executor + state isolation)
- 2347 full suite passes (1 pre-existing flaky test —
`test_watchdog_warns_about_stuck_workers` — passes in isolation,
unrelated)
- Ruff clean
User report: all 6 staging candidates failing with "Could not match
tracks to album tracklist" despite identification correctly resolving
each album. 18 properly-tagged Chris Brown F.A.M.E. tracks, 21
properly-tagged Mr. Morale tracks, etc. — every match attempt
rejected by the duration sanity gate.
Root cause: I had Deezer in `_SECONDS_DURATION_SOURCES`, assuming
Deezer's `duration` field was raw seconds (which the API returns).
But `DeezerClient.get_album_tracks` already converts seconds → ms
INTERNALLY (`'duration_ms': item.get('duration', 0) * 1000`) before
the value reaches the matcher. My helper saw `source='deezer'` →
multiplied by 1000 again → 255000 ms became 255,000,000 ms (70 hours).
Every track-file pair failed the gate by a factor of 1000×.
Diagnostic chain that got me there:
1. Added `[Album Matching] No matches: X files, Y tracks, Z
duration-rejected, W below threshold` summary log so future "0
matches" reports surface the rejection reason.
2. Fixed the helper's logger from `logging.getLogger(__name__)` (which
resolves outside the soulsync handler tree → invisible in app.log)
to `get_logger("imports.album_matching")` (under the namespace the
file handler watches).
3. Added per-rejection-type diagnostic showing actual file vs track
duration values + raw track keys + source.
That third diagnostic surfaced `track 'United In Grief' resolved=255000000
(raw duration_ms=255000, raw duration=None, source='deezer')` —
making the bug obvious.
Fixes:
- Moved Deezer from `_SECONDS_DURATION_SOURCES` to
`_MS_DURATION_SOURCES`. Comment documents WHY (the client converts
before returning) so a future reader doesn't "fix" the
classification back the wrong way.
- Bumped `DURATION_TOLERANCE_MS` from 3000 → 10000 (3s → 10s) to
match Picard ~7s / Beets ~10-15s / Plex ~10s industry baselines.
3s was a defensive copy of the post-download integrity check
threshold but that's a different problem (catching truncated
downloads, not identifying recordings across remasters/encodings).
- `_track_duration_ms` magnitude heuristic kept as fallback for
unknown / missing source (mocked test data without `source` field).
- Added `Match aborted` warnings at the three earlier silent return
points in `_match_tracks` (no client, no album_data, no tracks)
so future "Could not match" reports show WHICH step bailed.
- Added per-run diagnostic in `match_files_to_tracks` that logs the
first duration rejection's actual values — surfaces unit mismatches
+ drift problems without spamming N×M lines per run.
Test changes:
- `test_deezer_seconds_duration_converted_to_ms` renamed +
rewritten as `test_deezer_already_normalised_to_ms_by_client`
to pin the actual contract (matcher receives ms from the Deezer
client, takes as-is).
- `test_track_duration_source_aware_dispatch` updated — Deezer test
case now uses ms input + expects ms output.
- New `test_raw_deezer_seconds_falls_back_to_magnitude_heuristic`
pins the rare edge case where raw Deezer items WITHOUT `source`
reach the matcher (no client conversion path) — heuristic catches
it.
Verification:
- 179 import tests pass after changes
- Live test: all 6 user staging candidates now matching at 95-100%
confidence
- Multi-disc Mr. Morale lands with proper Disc 1 / Disc 2 / Disc 3
folder structure
- Picard-tagged libraries hit MBID fast paths (verified earlier)
- Tracks process in parallel via the existing scan-now thread spawn
(next commit refactors this to a proper bounded executor)
User reported nothing happening on a chaotic staging root despite
6 candidates being detected. Logs showed "Processing folder" for 3
of 6 — the other 3 were silently skipped.
Root cause:
The previous commit (`a9a6168`) introduced loose-file grouping —
multiple `FolderCandidate` objects can now share a `path` (each
album group at the staging root has the same parent directory but
its own audio_files + folder_hash). But two pieces of dedup
machinery still keyed on `path`:
- `_processing_hashes` (was `_processing_paths`) — runtime set of
in-flight candidates. Path-keyed → first sibling marks the path,
second + third siblings hit "already in flight" and skip.
- `_folder_snapshots` — mtime cache for stability check. Path-keyed
→ siblings overwrite each other's mtimes, stability check returns
unreliable results for whichever sibling lost the write race.
Both kept track of an attribute that was previously unique-per-path
(one candidate per directory) but my refactor broke that
invariant without updating the dedup keys. Net effect: only the
first candidate per directory ever got processed in a chaotic-root
scenario.
Fix:
- Renamed `_processing_paths` → `_processing_hashes` set, keyed on
`candidate.folder_hash`. Hash is unique per candidate by
construction (different audio_files lists hash differently).
- `_folder_snapshots` retyped + rekeyed to `folder_hash`. Siblings
no longer overwrite each other's mtime tracking.
- Both touched in lockstep — comments document why path-keyed
dedup breaks for sibling candidates.
Test added (`test_sibling_candidates_have_unique_folder_hashes`):
verifies 3-album loose root produces 3 candidates with distinct
folder_hashes. If a future change breaks the invariant, the test
fails before the silent-skip regression ships.
Verification:
- 178 imports tests pass (8 new this commit + 170 pre-existing
this branch)
- Ruff clean
- Still scoped to import flow
Two related bugs in `AutoImportWorker._scan_directory` surfaced
during real-world testing of the chaotic-staging case (user dropped
loose tracks from multiple albums at staging root, alongside
intact album subfolders):
Bug 1 — Loose files bundled into one fake "album"
When loose audio files existed at a level, the scanner built ONE
FolderCandidate from all of them regardless of their album tags.
On a chaotic staging root with tracks from 3+ different albums,
the identifier picked the most-common album tag and the matcher
left every other album's tracks unmatched (or mis-attributed via
filename + position guessing).
Bug 2 — Subfolders silently ignored when root has loose files
The scanner only recursed into non-disc subfolders when there were
NO loose files at the parent level. So a layout like:
Staging/
loose1.flac (processed via the loose-files path)
Other Album Folder/ (silently ignored — never scanned)
would skip the album subfolders entirely. Common pattern when a
user moves a few tracks out of an album folder while leaving the
rest of the parent album folder intact, OR when other album
folders sit alongside a partially-extracted album.
Fix:
`_build_loose_file_candidates` (new method) reads each loose file's
`album` tag and groups by normalised album name. Each group becomes
its own FolderCandidate so a chaotic staging root produces one
candidate per album — identifier + matcher run cleanly per album.
Untagged loose files become individual single candidates. Disc
folders at the same level attach to whichever loose-file group's
album tag matches the disc-folder tracks; standalone disc folders
(no matching loose group) get their own multi-disc candidate.
The scanner now ALSO always recurses into non-disc subdirectories,
even when the current level has loose files. So album subfolders
sitting beside loose tracks get processed independently in their
own recursive scan.
Behavior preservation:
- Single-album loose-files staging (every file shares one album tag,
no parallel disc folders) → one FolderCandidate, identical to
pre-fix behavior. Pinned by `test_single_album_loose_files_still_one_candidate`.
- Disc-only directory (no loose files, only Disc 1/Disc 2 subdirs)
→ one multi-disc FolderCandidate, identical to pre-fix. Pinned
by `test_disc_only_directory_still_works`.
7 new tests in `tests/imports/test_auto_import_scanner_grouping.py`:
- Multiple-album loose root → multiple candidates
- Untagged loose files → individual singles
- Single-album loose-files regression guard
- Subfolders recursed even when root has loose files
- Disc folder attaches to matching loose group by album tag
- Disc folder with no matching loose group → standalone candidate
- Disc-only directory regression guard
All write real FLACs via mutagen + exercise `_scan_directory`
end-to-end (no mocking the tag reader — proves the production
read path works).
Verification:
- 7 new tests pass
- 2328 full suite passes (+7 new), 1 pre-existing flaky timing test
unrelated to this PR
- Ruff clean
- All changes still scoped to import flow — download flow byte-
identical
Brings the auto-import matcher to picard / beets / roon parity by
reaching for the existing AcoustID-grade infrastructure (typed Album
foundation, integrity check thresholds) and layering id-based exact
matches on top of the fuzzy scorer. Picard-tagged libraries now land
every track with full confidence on the first pass.
Three layered phases in `core/imports/album_matching.match_files_to_tracks`:
1. **MBID exact match** — file has `musicbrainz_trackid` tag, source
returns the same id → instant pair, full confidence, no fuzzy
scoring. Picard's primary identifier; per-recording.
2. **ISRC exact match** — file has `isrc` tag, source returns the same
id → same fast-path, slightly lower priority than mbid (isrc can
be shared across remasters). Both ids normalised before compare
(uppercase + strip dashes/spaces for isrc, lowercase for mbid).
3. **Duration sanity gate** — files in the fuzzy phase whose audio
length differs from the candidate track's duration by more than
`DURATION_TOLERANCE_MS` (3s, matching the post-download integrity
check) are rejected before scoring runs. Defends against the
cross-disc / cross-release / wrong-edit problem the integrity
check used to catch only AFTER the file had already been moved +
tagged + db-inserted.
Tag reader (`_read_file_tags`) extended:
- Reads `isrc` (uppercased, strip / / spaces normalisation deferred
to matcher)
- Reads `musicbrainz_trackid` as `mbid` (lowercased)
- Reads `audio.info.length` and converts to `duration_ms` to match
the metadata-source convention
Metadata-source layer (`_build_album_track_entry`) extended:
- Propagates `isrc` from top-level OR `external_ids.isrc` (spotify
shape — would otherwise be stripped before reaching the matcher)
- Propagates `musicbrainz_id` from top-level OR `external_ids.mbid`
/ `external_ids.musicbrainz`
- Without this layer, fast paths would silently never fire in
production even though unit tests pass — pinned by
`test_album_track_entry_propagates_isrc_and_mbid_from_source`
18 new tests in `tests/imports/test_album_matching_exact_id.py`:
- Direct: `find_exact_id_matches` with mbid, isrc, isrc normalisation,
mbid > isrc priority, spotify-shape `external_ids.isrc`, no-id
empty result, file-used-at-most-once
- Direct: `duration_sanity_ok` within / outside tolerance, missing
durations defer
- End-to-end via `match_files_to_tracks`: mbid match short-circuits
fuzzy scoring, id-matched files excluded from fuzzy phase, duration
gate rejects wrong-disc collisions in fuzzy phase, normal matches
pass through the gate, missing durations fall through, deezer
seconds-vs-ms conversion, full picard-tagged 10-track album via
mbid only
- Production-shape: `_build_album_track_entry` propagates isrc + mbid
from spotify-shape (`external_ids.isrc`) AND itunes-shape (top-
level `isrc`)
Verification:
- 35 album-matching tests pass total (17 helper + 18 fast-path)
- 23 multi-disc tests still pass after the extension (additive)
- Full suite: 2311 passed (+18 new), 1 pre-existing flaky timing test
failure (`test_watchdog_warns_about_stuck_workers` — passes in
isolation, fails only in full-suite runs, unrelated to this PR)
- Ruff clean
For users:
- Picard / Beets / Mp3Tag-tagged libraries (anyone who's organised
their music) get instant perfect-confidence matches every time.
- Soulseek-tagged downloads (which usually carry isrc when sourced
via metadata-aware soulseekers) get the fast path too.
- Naively-named files with no useful tags fall through to the
improved fuzzy + duration-gated path — same correctness as before
for the common case, much harder for the matcher to confidently
pair the wrong file.
- One step closer to standalone-DB feature parity with plex /
jellyfin / navidrome scanners. Acoustid fingerprint fallback
(for files with NO useful tags AND no MBID/ISRC) is the next
followup PR.
Cin-pass on the #524 + multi-disc fixes. Pre-merge polish.
Lifts: `core/imports/album_matching.py`
`AutoImportWorker._match_tracks` was a 100+-line method buried in a
1400-line class. Testing it required monkey-patching `_read_file_tags`
+ mocking the metadata client just to exercise the matching algorithm.
Per Cin's "lift logic out of monolithic classes" pattern (same shape
as the album-info builders / discography / quality scanner lifts),
moved the dedup + scoring into `core/imports/album_matching.py` as
pure functions over already-fetched data.
Helper exposes:
- Constants for every match weight (TITLE_WEIGHT, ARTIST_WEIGHT,
POSITION_WEIGHT, NEAR_POSITION_WEIGHT, CROSS_DISC_POSITION_WEIGHT,
ALBUM_WEIGHT, MATCH_THRESHOLD). Magic numbers killed.
- `dedupe_files_by_position(audio_files, file_tags, *, quality_rank)` —
position-keyed quality dedup.
- `score_file_against_track(file_path, file_tags, track, *,
target_album, similarity)` — pure per-(file, track) scorer.
- `match_files_to_tracks(audio_files, file_tags, tracks, *,
target_album, similarity, quality_rank)` — full matching with
greedy best-per-track + first-come-first-serve over deduped files.
Worker shrinks from 100 lines of inline algorithm to 8 lines that
fetch tags + delegate to the helper.
Tests added (26 new across 3 files):
`tests/imports/test_album_matching_helper.py` (19 tests):
- Constants pin: weights sum to 1.0, threshold above position-only
- `dedupe_files_by_position`: quality wins, cross-disc preserved,
tag-less files passed through, first-wins on equal quality
- `score_file_against_track`: perfect-agreement = 1.0, position
needs both disc+track, near-position only same-disc, missing
artist tags handled, disc field aliases (Spotify/Deezer/iTunes),
filename fallback when title tag missing
- `match_files_to_tracks`: happy path, file used at-most-once,
below-threshold left unmatched
- Edge case Cin would flag: tag-less file with strong filename title
matches multi-disc album track via title alone (perfect-name
scenario works); tag-less file with weak filename title against
multi-disc API correctly stays unmatched (the behavior delta from
the disc-aware fix — pinned so future readers see it's intentional)
`tests/test_import_album_match_endpoint.py` (3 tests):
- Backend warning fires when source missing from match POST
- No warning fires on the legit path (catches noisy-warning regression)
- Endpoint actually forwards source/name/artist to the payload
builder (catches "logging the right warning but doing the wrong
lookup" regression)
`tests/test_import_page_album_lookup_pattern.py` (4 tests):
- Source-text guard for the import-page #524 fix in stats-automations.js.
Until the file is modularized enough for a behavioral JS test (under
the existing tests/static/*.mjs pattern), regex-based assertions pin:
the `_albumLookup` field exists, the click handler reads from it,
both card renderers populate it before emitting onclick, and the
cache stores `source` per entry. Caveat documented in the test
module docstring.
Verification:
- All 26 new tests pass.
- Existing multi-disc tests (test_auto_import_multi_disc_matching.py)
still pass after the lift — proves the helper is behavior-equivalent
to the inline implementation it replaced.
- Full suite: 2293 passed, 1 flaky-timing failure
(test_library_reorganize_orchestrator.py::test_watchdog_warns_about_stuck_workers
— passes in isolation, fails only in full-suite runs, pre-existing,
unrelated to this PR).
- Ruff clean.
Notes for the reviewer:
- The frontend stats-automations.js JS test is structural-only.
Behavioral JS testing for that file requires modularizing the
~7k-line monolith first — out of scope for this fix.
- The cross-disc 5% consolation bonus is a small behavior change for
users with weak/missing tag info on multi-disc albums. Pinned
explicitly in `test_tagless_file_with_weak_title_unmatched_in_multidisc`
so the trade-off is visible: correct multi-disc matching wins over
optimistic position-only matching that produced wrong-disc files.
Caught while live-testing the #524 fix with kendrick lamar
mr morale & the big steppers (3 discs). User dropped discs 1+2
loose in staging root + disc 3 in its own folder, every file
perfectly tagged with disc_number/track_number/title — only 9
tracks ended up in the library, the rest got integrity-rejected
and quarantined.
Two related bugs in `AutoImportWorker._match_tracks`:
1. **Quality dedup keyed on track_number alone.** The dedup loop
kept `seen_track_nums[track_number] = file` and dropped any later
file with the same number, treating it as a quality duplicate.
On a multi-disc release where every disc has tracks 1..N, that
collapses the album to one disc's worth of files BEFORE the
matcher runs. User's 18 loose disc-1+disc-2 files reduced to 9
before any title/disc info was even consulted.
2. **Match scoring ignored disc_number.** The 30% track-number bonus
fired whenever `ft[track_number] == track_num` regardless of disc.
File with tag (disc=2, track=6, "Auntie Diaries", 281s) got the
full bonus matching API track (disc=1, track=6, "Rich Interlude",
103s) — wrong file → wrong destination → integrity check correctly
rejected and quarantined the file. Same for tracks 7, 8, 9.
Fix:
- Dedup keys on `(disc_number, track_number)` tuples — multi-disc
files with parallel numbering all survive.
- Match scoring's 30% bonus only when BOTH disc AND track agree.
Cross-disc same-track-number collisions get a small 5% consolation
bonus so title similarity has to carry the match (covers cases
where tag disc info is missing or wrong).
- API track disc_number read from `disc_number` (Spotify) /
`disk_number` (Deezer) / `discNumber` (iTunes) defaulting to 1.
4 new pinning tests in `tests/imports/test_auto_import_multi_disc_matching.py`:
- 18-file 2-disc regression case (dedup preserves all)
- (disc=2, track=6) file matches API (disc=2, track=6) track, not
the disc-1 same-numbered track
- Single-disc albums still match normally (no regression)
- Quality dedup within a single (disc, track) position still picks
higher-quality format (.flac over .mp3)
Verification:
- 2268 full pytest suite passes (+4 new), 1 skipped, 0 failed
- Ruff clean
Same branch as the #524 fix because both surfaced from the same
import session — easier reviewer context if they ship together.
The 'Live Per-Track Progress' work shipped a backend in-progress row + top-of-tab
progress text but the history cards themselves stayed visually stale during
processing — lowercase "processing" badge, neutral styling, no per-track hint.
Smoke-testing also surfaced two latent identification bugs that prevented
multi-disc rips with features (Kendrick GKMC Deluxe) from importing at all.
Card-level live progress (`webui/static/stats-automations.js`):
- Cache `/api/auto-import/status` response in `_autoImportLastStatus`; poller
awaits status before re-rendering results so the card has the live data.
- Add 'processing' entries to statusLabels / statusIcons / statusClass.
- When card folder_name matches `current_folder`, swap the meta line to
`track N/M: <track name>` and tag the matching row in the expanded list
as `auto-import-track-row-active`; prior rows tag as `-row-done`.
Card styling (`webui/static/style.css`):
- `.auto-import-processing` blue left border, `.auto-import-badge-processing`
pulse animation, active/done track-row classes.
Multi-disc enumeration (`core/auto_import_worker.py:_scan_directory`):
- Old code skipped disc folders during recursion AND only attached them to a
parent that had its own loose audio. A folder containing only `Disc 1/`,
`Disc 2/` was invisible. Now: when a directory has only disc subdirs and no
loose audio, treat that directory itself as the album candidate. Disc folders
still skipped when standing alone.
- Add `FolderCandidate.is_staging_root` flag (set when the staging dir itself
becomes the candidate via this path) so identification can refuse to use the
meaningless folder name.
Tag identification (`core/auto_import_worker.py:_identify_from_tags`):
- Per-track `artist` tag fragmented consensus on albums with features
("Kendrick Lamar" / "Kendrick Lamar, Drake" / "Kendrick Lamar, Dr. Dre"
produced 3 separate `(album, artist)` keys for one album). Now group by
album first, then pick the most-common artist within that album group.
- `_read_file_tags` now prefers `albumartist` over `artist` for album-level
identity; falls back to `artist` for files without albumartist.
- Add INFO-level log when tag identification rejects, showing top albums and
their counts so the user can diagnose multi-disc / tagging issues.
Folder-name false-match guard (`core/auto_import_worker.py:_identify_folder`):
- When `is_staging_root` is set, skip the folder-name strategy entirely. Logs
the skip and falls through to AcoustID. Without this, dropping disc folders
directly into staging caused the scanner to search the metadata source for
the literal name "Staging", which false-matched against random albums (e.g.
"Stamina, Dinos" — a French rap album — at 13% confidence).
What's New entries added under 2.4.2 dev cycle.
User reported (Mushy / generally) that dropping an album into the
staging folder left the auto-import history blank for the entire
processing window — sometimes 5+ minutes for a full album. Pre-
existing UX gap, not caused by the recent context-builder refactor.
Two root causes:
1. ``_record_result`` only fired AFTER ``_process_matches`` returned.
For a 14-track album with ~30s/track post-processing, that meant
~7 minutes of zero rows in auto_import_history → nothing for
``/api/auto-import/results`` to return → empty UI.
2. ``_current_status`` only ever transitioned between 'idle' and
'scanning' — never 'processing'. ``get_status()`` had no per-
track index/name fields, so the UI had no way to render
"Processing track 3/14: Mine" even if it wanted to.
Fix:
- New ``_record_in_progress`` inserts a status='processing' row
up-front (before the per-track loop starts) so the UI sees the
import the moment it begins. Returns the row id.
- New ``_finalize_result`` updates that same row with the final
outcome (completed/failed) when processing finishes. One row per
album, not per track — keeps the history list clean.
- Both share ``_serialize_match_data`` (extracted from the original
``_record_result``) so the in-progress row carries the same match
payload shape the existing review UI already understands.
- ``_process_matches`` updates ``_current_track_index``,
``_current_track_total``, and ``_current_track_name`` BEFORE each
per-track callback fires, so a polling UI sees consistent
"processing N/M: <name>" snapshots.
- ``_scan_cycle`` flips ``_current_status`` to 'processing' before
the per-album loop, resets it + the per-track fields after.
Defensive ``finally`` clears progress even if the inner code path
raised.
- ``get_status()`` exposes the new fields so the UI's existing
/api/auto-import/status polling picks them up.
- Frontend (stats-automations.js): renders the new
``current_status='processing'`` state with track index/total/name
in the existing progress bar element. New 'processing' status
class for styling parity with 'scanning'.
8 regression tests in tests/imports/test_auto_import_live_progress.py:
- get_status surfaces the new fields with sane defaults
- track_index advances 1, 2, 3 during a 3-track loop
- track_total set BEFORE the first callback fires (no '1/0' flicker)
- _record_in_progress writes status='processing' with no
processed_at
- _finalize_result updates the same row to completed +
processed_at, no second insert
- _finalize_result with failed status leaves processed_at NULL
- _finalize_result with row_id=None is a safe no-op
- Per-track fields cleared by _scan_cycle's finally block
Full pytest 1643 passed; ruff clean.
When staging files are organized as Artist/Albums/AlbumFolder or
Artist/AlbumFolder, the auto-import now uses the parent folder name
as the artist instead of trusting embedded file tags.
Uses relative path from staging root to determine folder depth, so
albums directly in staging root don't accidentally pick up container
paths as artist names. Common category subfolder names (Albums,
Singles, EPs, Mixtapes, etc.) are recognized and skipped.
Fixes mixtapes and compilations where file tags have DJ names or
incorrect artists (e.g. files tagged as "Slim" in a 2Pac folder).
Files with embedded tags (artist+title from post-processing) were
failing import because the metadata search scored low (66%) and the
AcoustID result returned before the tag-preference code could run.
- Tag-based identification now returns 85% confidence when embedded
tags have an artist field, borrowing album art from weak metadata
- AcoustID search result only accepted at 80%+ confidence, otherwise
kept as fallback (doesn't short-circuit past tag preference)
- AcoustID None artist/title falls back to tag data via 'or' operator
- Stop retrying failed/unidentified items every scan cycle
Items with status needs_identification, failed, or rejected were not
in the skip list, causing them to be re-scanned and re-logged every
60 seconds indefinitely. Now skips all terminal statuses.
Race condition: scanner re-scanned folders while post-processing was
still moving files, causing partial matches and ghost failures. Now
tracks in-progress paths and skips them on subsequent scans.
Coverage penalty fix: individual tracks that match at 80%+ confidence
now auto-import even when overall album coverage is low (e.g. 2 of 18
tracks present). Previously low coverage killed the entire import.
Import page: stats bar, filter pills, Scan Now, Approve All, Clear
History (clears imported + failed), live scan progress.
- Track numbers defaulted to 1 instead of using metadata source values
- Release dates not captured, causing missing year in path templates
- Cover art missing for Deezer (direct image_url not checked)
- Track names in expanded view showed Unknown (wrong JSON field name)
- Read year/date from embedded file tags as fallback
- Add Deezer get_album_metadata/get_album_tracks fallbacks
- Handle Deezer tracks.data response format
Loose audio files in the staging root are now picked up alongside album
folders. Singles are identified via embedded tags, filename parsing
(Artist - Title.ext), or AcoustID fingerprinting, then matched against
the configured metadata source. Confidence-gated processing applies
the same way as album folders (90%+ auto, 70-90% review, <70% manual).
Full auto-import pipeline: background worker watches the staging folder,
identifies music using embedded tags → folder name parsing → AcoustID
fingerprinting, matches files to metadata source tracklists, and
processes high-confidence matches through the existing post-processing
pipeline automatically.
Worker: AutoImportWorker with start/stop/pause/resume, configurable
scan interval (default 60s), confidence threshold (default 90%), and
auto-process toggle. Processes one folder per cycle, alphabetical
order. Disc folder detection, stability checking, content hash dedup.
Confidence gate: 90%+ auto-processes silently, 70-90% queued as
pending review with approve/dismiss actions, <70% flagged for manual
identification. Track matching uses weighted algorithm (title 45%,
artist 15%, track number 30%, album tag 10%).
Database: auto_import_history table tracks every scan result with
folder hash, match data JSON, confidence, status, timestamps.
API: 7 endpoints — status, toggle, settings (GET/POST), results
(filtered/paginated), approve, reject.
UI: Auto tab on Import page with enable toggle, confidence slider,
scan interval selector. Live result cards with album art, confidence
bar (green/yellow/red), status badges, match stats. 5-second polling.
dev