danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Reorganize: route library files through the post-processing pipeline

Browse source 
Reported on Discord by winecountrygames. The library "Reorganize" tool
had several layered bugs that all traced to the same root cause: the
endpoint reinvented every wheel post-processing already turns — its own
template engine, its own disc-number resolution from file tags, its own
sidecar sweep, its own collision detection — and each had drifted from
the canonical path used by fresh downloads. Reported symptoms:

  - 3-disc Aerosmith deluxe collapsed to a flat single-disc layout
  - Half the tracks on other albums silently skipped, no error / no count
  - Re-runs left empty leftover album folders cluttering the artist dir

Architecture: stop reinventing wheels. Route reorganize through exactly
the same pipeline downloads use. Per-album:

  1. Fetch the canonical tracklist from a metadata source (Spotify /
     iTunes / Deezer / Discogs / Hydrabase) using the album's stored
     source IDs. New `core/library_reorganize.py::plan_album_reorganize`
     does this — primary-source-first, fall through priority chain
     unless the user picked a specific source in the modal (strict mode).
  2. For each local track, find the matching API entry via a scored
     candidate matcher. Score components: exact-title (100),
     substring-with-length-ratio (40-90), track-number agreement (20).
     Hard reject when the two titles have different version
     differentiators (Remix vs no-remix means different recordings,
     not annotation drift). Below threshold = unmatched, surfaced as
     "not in source's tracklist, left in place" rather than silently
     mis-routing.
  3. Copy the file to a per-album staging directory, build the same
     context dict the import flow builds (`spotify_album` /
     `track_info` / etc. with `is_album_download=True` so the path
     builder enters ALBUM mode, not SINGLE mode), call
     `_post_process_matched_download(...)` — same function fresh
     downloads use. Post-process handles tagging, multi-disc subfolder
     decisions, sidecar regeneration, AcoustID verification.
  4. Read `context['_final_processed_path']` to learn where it landed.
     Update `tracks.file_path` in the DB BEFORE removing the original
     (DB-update failure leaves the file at both locations, recoverable
     via library scan; the reverse would orphan the row). Delete
     per-track sidecars (post-process recreates them at the new
     destination).

3 concurrent workers per album via ThreadPoolExecutor, matching the
download path's per-batch worker count. State mutations all guarded by
a single lock; staging filenames carry a UUID prefix so concurrent
copies of identically-named source files don't overwrite each other.

Source picker in the modal lets the user choose which source to read
the tracklist from. Two endpoints feed it:
  - `/api/library/album/<id>/reorganize/sources` — sources for THIS
    album that are both authed AND have a stored ID. For the per-
    album modal.
  - `/api/library/reorganize/sources` — all authed sources globally.
    For the bulk "Reorganize All" modal where per-album ID coverage
    varies.
When the user picks a specific source, the orchestrator runs in
`strict_source=True` mode (no fallback chain) — picking Spotify means
"use Spotify or fail", not "use Spotify and silently fall back."

Preview endpoint shares the same planning logic as apply via
`preview_album_reorganize` — the destination path comes from the same
`_build_final_path_for_track` post-process uses, so what you see in
the preview is exactly what you get on apply.

Empty destination folders (from earlier failed runs OR from the
current run when post-process creates a dir then fails AcoustID)
get cleaned up after each successful run: walk up to the artist
folder from any successful destination, prune empty album-sibling
folders one level deep. Bounded scope = won't touch unrelated user
dirs.

Web_server.py shrinks by ~450 net lines. The endpoint handler is now
a thin wrapper that builds injected callables (path resolver, post-
process function, DB updater, empty-dir cleaner), spawns a thread
that calls `reorganize_album()`, and returns. All actual logic lives
in `core/library_reorganize.py` where it's unit-testable without
spinning up Flask.

Frontend cleanup: the per-call template input in both reorganize
modals (per-album and bulk) was redundant — the backend always uses
the configured global download template. Removed the input and the
variables-grid reference UI it was for.

39 new unit tests pin every contract:
  - source resolution (no_source_id when album has none, fallthrough
    chain when primary returns nothing, strict mode bypasses fallback)
  - matcher scoring (exact / substring / multi-disc disambiguation /
    smart-quote tolerance / dash-vs-parens / bonus-track substring /
    Remix-vs-original differentiator rejection / "Real" doesn't false-
    match "Real Real Real" / track-number-only no longer fires)
  - file safety (DB-update failure leaves original in place, post-
    process failure leaves original in place, post-process exception
    caught and original preserved, success removes original AND
    updates DB in the right order)
  - sidecar handling (per-track .lrc/.nfo deleted on success, kept on
    failure; album-level cover.jpg/folder.jpg cleaned only when
    directory has no remaining audio)
  - staging cleanup (recreated between tracks because post-process
    nukes it, dir cleaned up on success AND on failure)
  - destination-dir prune (empty siblings removed, real album with
    files preserved, no recursive sweep)
  - source picker (only authed-with-stored-ID sources for per-album,
    all authed sources for bulk; strict mode doesn't fall back)
  - concurrency (3 workers in flight, state stays consistent under
    races, stop_check cuts off pending tasks)
  - preview parity (preview produces same destination as apply for
    multi-disc; ALBUM mode not SINGLE mode; unmatched/no-path tracks
    surfaced with reasons)

Limitations (deliberate punts, NOT in this PR):
  - Renamed local titles on multi-disc albums where track_number
    also disagrees: matcher returns nothing (track is "not in
    source"). Fixable by using duration_ms as a tertiary signal.
  - Per-track in-modal source switching with per-album track-count
    hints (would need a second API call before opening the modal).
  - UI status panel on the artist page during a run — currently
    just toasts. Documented as a follow-up PR.

Files:
  - core/library_reorganize.py — new module: plan_album_reorganize,
    preview_album_reorganize, reorganize_album, available_sources_for_album,
    authed_sources, _score_candidate, helpers for staging/post-
    processing/finalizing, sidecar + dest-dir cleanup
  - core/metadata_service.py — no changes; reused get_album_for_source,
    get_album_tracks_for_source, get_source_priority,
    get_client_for_source
  - web_server.py — three endpoints (preview / apply / sources GETs)
    are thin wrappers; -450 net lines
  - tests/test_library_reorganize_orchestrator.py — 39 tests covering
    every contract above
  - webui/static/library.js — source picker UI in both modals; dead
    template input + variables-grid removed
  - webui/static/style.css — dropdown option styling fix (white-on-
    white was unreadable)

Reported on Discord by winecountrygames — his bug report named the
trigger button (Enhanced view → Reorganize All) and both symptoms
(multi-disc collapse, half-album skip), which let the diagnosis go
straight to the architectural problem.
This commit is contained in:
Broque Thomas 2026-04-24 18:16:11 -07:00
parent 27a5b6aef1
commit 2b15260b88
6 changed files with 3643 additions and 487 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1477
core/library_reorganize.py Normal file
View file

File diff suppressed because it is too large Load diff

1930
tests/test_library_reorganize_orchestrator.py Normal file
View file

File diff suppressed because it is too large Load diff

501
web_server.py
View file

@ -13607,143 +13607,64 @@ _reorganize_state = {
_reorganize_lock = threading.Lock()
@app.route('/api/library/reorganize/sources', methods=['GET'])
def reorganize_sources_global():
"""List metadata sources the user has authed on this instance.
Used by the bulk "Reorganize All" modal where per-album ID coverage
varies. No network calls."""
try:
from core.library_reorganize import authed_sources
return jsonify({"success": True, "sources": authed_sources()})
except Exception as e:
logger.error(f"Reorganize sources (global) error: {e}")
return jsonify({"success": False, "error": str(e)}), 500
@app.route('/api/library/album/<album_id>/reorganize/sources', methods=['GET'])
def reorganize_album_sources(album_id):
"""List metadata sources the user can pick for this album's
reorganize — every entry has both a stored album ID on the local
row AND an authenticated client. No network calls."""
try:
from core.library_reorganize import available_sources_for_album, load_album_and_tracks
album_data, _tracks = load_album_and_tracks(get_database(), album_id)
if album_data is None:
return jsonify({"success": False, "error": "Album not found"}), 404
return jsonify({"success": True, "sources": available_sources_for_album(album_data)})
except Exception as e:
logger.error(f"Reorganize sources error: {e}")
return jsonify({"success": False, "error": str(e)}), 500
@app.route('/api/library/album/<album_id>/reorganize/preview', methods=['POST'])
def reorganize_album_preview(album_id):
"""Preview file reorganization for an album — returns current vs proposed paths without moving anything."""
"""Preview file reorganization for an album — returns current vs
proposed paths without moving anything. Implementation lives in
:mod:`core.library_reorganize` and shares the planning logic with
the apply endpoint, so the preview is guaranteed to match what
apply would actually produce.
Optional body param ``source``: when provided, only that metadata
source is queried (no fallback chain)."""
try:
database = get_database()
from core.library_reorganize import preview_album_reorganize
data = request.get_json() or {}
template = data.get('template', '').strip()
if not template:
return jsonify({"success": False, "error": "Template is required"}), 400
conn = database._get_connection()
cursor = conn.cursor()
# Get album + artist info
cursor.execute("""
SELECT al.*, a.name as artist_name
FROM albums al
JOIN artists a ON al.artist_id = a.id
WHERE al.id = ?
""", (str(album_id),))
album_row = cursor.fetchone()
if not album_row:
return jsonify({"success": False, "error": "Album not found"}), 404
album_data = dict(album_row)
# Get all tracks for this album
cursor.execute("""
SELECT t.*, a.name as artist_name
FROM tracks t
JOIN artists a ON t.artist_id = a.id
WHERE t.album_id = ?
ORDER BY t.track_number
""", (str(album_id),))
tracks = [dict(r) for r in cursor.fetchall()]
if not tracks:
return jsonify({"success": False, "error": "No tracks found for this album"}), 404
chosen_source = data.get('source') or None
transfer_dir = docker_resolve_path(config_manager.get('soulseek.transfer_path', './Transfer'))
# Pre-scan disc numbers so every track's template context carries the
# same total_discs. Needed by $cdnum (smart CD label) so the template
# can decide whether to emit "CDxx" or stay empty for single-disc.
track_disc_numbers = {}
for _t in tracks:
_rp = _resolve_library_file_path(_t.get('file_path')) if _t.get('file_path') else None
_dn = 1
if _rp:
try:
from core.tag_writer import read_file_tags
_dn = read_file_tags(_rp).get('disc_number') or 1
except Exception:
_dn = 1
track_disc_numbers[_t.get('id')] = int(_dn)
total_discs = max(track_disc_numbers.values(), default=1) if track_disc_numbers else 1
preview_items = []
for track in tracks:
file_path = track.get('file_path')
resolved = _resolve_library_file_path(file_path) if file_path else None
# Reuse the disc number captured in the pre-scan (avoids re-reading tags)
disc_number = track_disc_numbers.get(track.get('id'), 1)
# Get file extension from current path
file_ext = os.path.splitext(resolved or file_path or '.mp3')[1]
# Detect quality using the same format as the download pipeline
quality = _get_audio_quality_string(resolved) if resolved else ''
# Build context for template
year_val = album_data.get('year') or ''
context = {
'artist': track.get('artist_name') or 'Unknown Artist',
'albumartist': album_data.get('artist_name') or track.get('artist_name') or 'Unknown Artist',
'album': album_data.get('title') or 'Unknown Album',
'title': track.get('title') or 'Unknown Track',
'track_number': track.get('track_number') or 1,
'disc_number': disc_number,
'total_discs': total_discs,
'year': year_val,
'quality': quality,
'albumtype': _get_album_type_display(
album_data.get('record_type'),
album_data.get('track_count') or len(tracks)
),
}
# Build new path using the template
folder_path, filename = _get_file_path_from_template_raw(template, context)
new_relative = os.path.join(folder_path, f"{filename}{file_ext}") if folder_path else f"{filename}{file_ext}"
new_full = os.path.join(transfer_dir, new_relative)
# Current path relative to transfer dir for display
current_display = file_path or 'No file'
if resolved and transfer_dir and resolved.startswith(transfer_dir):
current_display = resolved[len(transfer_dir):].lstrip(os.sep).lstrip('/')
same = resolved and os.path.normpath(resolved) == os.path.normpath(new_full)
preview_items.append({
'track_id': track['id'],
'title': track.get('title', ''),
'track_number': track.get('track_number', 0),
'current_path': current_display,
'new_path': new_relative,
'new_full_normalized': os.path.normpath(new_full) if resolved else None,
'file_exists': resolved is not None,
'unchanged': same,
'collision': False,
})
# Detect collisions: multiple tracks mapping to the same destination
seen_paths = {}
for item in preview_items:
norm = item.get('new_full_normalized')
if not norm or not item['file_exists'] or item['unchanged']:
continue
if norm in seen_paths:
item['collision'] = True
# Also mark the first one that claimed this path
seen_paths[norm]['collision'] = True
else:
seen_paths[norm] = item
# Remove internal field from response
for item in preview_items:
item.pop('new_full_normalized', None)
return jsonify({
"success": True,
"album": album_data.get('title', ''),
"artist": album_data.get('artist_name', ''),
"tracks": preview_items,
"transfer_dir": transfer_dir,
})
result = preview_album_reorganize(
album_id=album_id,
db=get_database(),
transfer_dir=transfer_dir,
resolve_file_path_fn=_resolve_library_file_path,
build_final_path_fn=_build_final_path_for_track,
primary_source=chosen_source,
strict_source=bool(chosen_source),
)
if result.get('status') == 'no_album':
return jsonify({"success": False, "error": "Album not found"}), 404
if result.get('status') == 'no_tracks':
return jsonify({"success": False, "error": "No tracks found for this album"}), 404
return jsonify(result)
except Exception as e:
logger.error(f"Reorganize preview error: {e}")
return jsonify({"success": False, "error": str(e)}), 500
@ -13751,274 +13672,88 @@ def reorganize_album_preview(album_id):
@app.route('/api/library/album/<album_id>/reorganize', methods=['POST'])
def reorganize_album_files(album_id):
"""Move album files to new paths based on the provided template."""
"""Re-route an album's existing files through the same post-processing
pipeline downloads use. Implementation lives in
:mod:`core.library_reorganize` to keep this monolith from growing.
The request body's ``template`` (if any) is ignored — post-processing
always uses the configured template, matching the download path.
Optional body param ``source``: when provided, only that metadata
source is used (no fallback chain). Matches the per-album source
picker in the reorganize modal."""
try:
data = request.get_json() or {}
template = data.get('template', '').strip()
if not template:
return jsonify({"success": False, "error": "Template is required"}), 400
# Atomic check-and-set to prevent concurrent reorganizations
chosen_source = data.get('source') or None
with _reorganize_lock:
if _reorganize_state['status'] == 'running':
return jsonify({"success": False, "error": "A reorganization is already in progress"}), 409
_reorganize_state['status'] = 'running'
database = get_database()
conn = database._get_connection()
cursor = conn.cursor()
# Get album + artist info
cursor.execute("""
SELECT al.*, a.name as artist_name
FROM albums al
JOIN artists a ON al.artist_id = a.id
WHERE al.id = ?
""", (str(album_id),))
album_row = cursor.fetchone()
if not album_row:
with _reorganize_lock:
_reorganize_state['status'] = 'idle'
return jsonify({"success": False, "error": "Album not found"}), 404
album_data = dict(album_row)
# Get all tracks
cursor.execute("""
SELECT t.*, a.name as artist_name
FROM tracks t
JOIN artists a ON t.artist_id = a.id
WHERE t.album_id = ?
ORDER BY t.track_number
""", (str(album_id),))
tracks = [dict(r) for r in cursor.fetchall()]
if not tracks:
with _reorganize_lock:
_reorganize_state['status'] = 'idle'
return jsonify({"success": False, "error": "No tracks found"}), 404
transfer_dir = docker_resolve_path(config_manager.get('soulseek.transfer_path', './Transfer'))
# Initialize state (already set to 'running' above)
with _reorganize_lock:
_reorganize_state.update({
'total': len(tracks),
'processed': 0,
'moved': 0,
'skipped': 0,
'failed': 0,
'current_track': '',
'errors': [],
'total': 0, 'processed': 0, 'moved': 0, 'skipped': 0,
'failed': 0, 'current_track': '', 'errors': [],
})
def _run_reorganize():
bg_conn = None
download_dir = docker_resolve_path(config_manager.get('soulseek.download_path', './downloads'))
staging_root = os.path.join(download_dir, 'ssync_staging')
try:
os.makedirs(staging_root, exist_ok=True)
except OSError:
pass
transfer_dir = docker_resolve_path(config_manager.get('soulseek.transfer_path', './Transfer'))
def _on_progress(updates):
with _reorganize_lock:
_reorganize_state.update(updates)
def _update_track_path(track_id, new_path):
try:
# Single DB connection for the background thread
bg_db = get_database()
bg_conn = bg_db._get_connection()
_db = get_database()
with _db._get_connection() as _conn:
_conn.execute(
"UPDATE tracks SET file_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
(new_path, str(track_id)),
)
_conn.commit()
except Exception as _db_err:
logger.warning(f"[Reorganize] DB path update failed for {track_id}: {_db_err}")
# Pre-scan disc numbers for every track so total_discs is the
# same for all template contexts in this album. Needed by the
# $cdnum template variable to decide multi-disc vs single-disc.
track_disc_numbers = {}
for _t in tracks:
_rp = _resolve_library_file_path(_t.get('file_path')) if _t.get('file_path') else None
_dn = 1
if _rp:
try:
from core.tag_writer import read_file_tags
_dn = read_file_tags(_rp).get('disc_number') or 1
except Exception:
_dn = 1
track_disc_numbers[_t.get('id')] = int(_dn)
total_discs = max(track_disc_numbers.values(), default=1) if track_disc_numbers else 1
def _cleanup_empty(src_dir):
try:
_cleanup_empty_directories(transfer_dir, os.path.join(src_dir, '_'))
except Exception:
pass
# Pre-compute all destination paths to detect collisions
dest_paths = {} # normalized_new_path -> track_id
for track in tracks:
file_path = track.get('file_path')
resolved = _resolve_library_file_path(file_path) if file_path else None
if not resolved:
continue
# Reuse the disc number from the pre-scan pass above
disc_number = track_disc_numbers.get(track.get('id'), 1)
file_ext = os.path.splitext(resolved)[1]
quality = _get_audio_quality_string(resolved)
year_val = album_data.get('year') or ''
context = {
'artist': track.get('artist_name') or 'Unknown Artist',
'albumartist': album_data.get('artist_name') or track.get('artist_name') or 'Unknown Artist',
'album': album_data.get('title') or 'Unknown Album',
'title': track.get('title') or 'Unknown Track',
'track_number': track.get('track_number') or 1,
'disc_number': disc_number,
'total_discs': total_discs,
'year': year_val,
'quality': quality,
'albumtype': _get_album_type_display(
album_data.get('record_type'),
album_data.get('track_count') or len(tracks)
),
}
folder_path, filename = _get_file_path_from_template_raw(template, context)
new_relative = os.path.join(folder_path, f"{filename}{file_ext}") if folder_path else f"{filename}{file_ext}"
new_full = os.path.join(transfer_dir, new_relative)
norm_new = os.path.normpath(new_full)
# Check for collision: two tracks mapping to same destination
if norm_new in dest_paths and dest_paths[norm_new] != str(track['id']):
# Mark as collision so the move pass skips it
track['_collision'] = True
with _reorganize_lock:
_reorganize_state['failed'] += 1
_reorganize_state['processed'] += 1
_reorganize_state['errors'].append({
'track_id': track['id'],
'title': track.get('title', 'Unknown'),
'error': "Path collision with another track — add $track or $disc to template"
})
continue
dest_paths[norm_new] = str(track['id'])
# Store computed info on the track dict for the move pass
track['_resolved'] = resolved
track['_new_full'] = new_full
track['_disc_number'] = disc_number
# Now do the actual moves
moved_dirs = {} # src_dir → dest_dir for post-pass sidecar sweep
for track in tracks:
resolved = track.get('_resolved')
new_full = track.get('_new_full')
track_title = track.get('title', 'Unknown')
with _reorganize_lock:
_reorganize_state['current_track'] = track_title
# Skip tracks already handled (collision or file not found)
if track.get('_collision'):
continue
if not resolved or not new_full:
# File not found — only count if not already handled in pre-computation
if '_resolved' not in track:
with _reorganize_lock:
_reorganize_state['skipped'] += 1
_reorganize_state['processed'] += 1
_reorganize_state['errors'].append({
'track_id': track['id'],
'title': track_title,
'error': 'File not found on disk'
})
continue
# Skip if already at target
if os.path.normpath(resolved) == os.path.normpath(new_full):
with _reorganize_lock:
_reorganize_state['skipped'] += 1
_reorganize_state['processed'] += 1
continue
try:
# Move file
_safe_move_file(resolved, new_full)
# Track source→dest directory mapping for post-pass sidecar sweep
src_dir = os.path.dirname(resolved)
dest_dir = os.path.dirname(new_full)
if src_dir not in moved_dirs:
moved_dirs[src_dir] = dest_dir
# Move track-level sidecars (same filename stem as audio)
src_stem = os.path.splitext(os.path.basename(resolved))[0]
new_stem = os.path.splitext(os.path.basename(new_full))[0]
for sidecar_ext in ('.lrc', '.nfo', '.txt', '.cue'):
sidecar_src = os.path.join(src_dir, src_stem + sidecar_ext)
if os.path.isfile(sidecar_src):
sidecar_dst = os.path.join(dest_dir, new_stem + sidecar_ext)
try:
shutil.move(sidecar_src, sidecar_dst)
except Exception:
pass
# Update DB file_path
bg_cursor = bg_conn.cursor()
bg_cursor.execute(
"UPDATE tracks SET file_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
(new_full, str(track['id']))
)
bg_conn.commit()
with _reorganize_lock:
_reorganize_state['moved'] += 1
_reorganize_state['processed'] += 1
except Exception as move_err:
logger.error(f"Reorganize move error for {track_title}: {move_err}")
with _reorganize_lock:
_reorganize_state['failed'] += 1
_reorganize_state['processed'] += 1
_reorganize_state['errors'].append({
'track_id': track['id'],
'title': track_title,
'error': str(move_err)
})
# Post-pass: sweep source directories for leftover album-level sidecars.
# The per-track loop can't reliably move cover.jpg because multiple tracks
# share the same source dir — the first track's move may fail silently,
# or the file may be in a parent directory. This single pass catches them all.
_album_sidecars = ('cover.jpg', 'cover.jpeg', 'cover.png', 'folder.jpg',
'folder.png', 'front.jpg', 'front.png', 'album.jpg', 'album.png')
for src_dir, dest_dir in moved_dirs.items():
if not os.path.isdir(src_dir):
continue
# Check if any audio files remain (don't steal sidecars from a dir that still has tracks)
audio_exts = {'.flac', '.mp3', '.m4a', '.ogg', '.opus', '.wav', '.aac', '.wma'}
has_audio = any(os.path.splitext(f)[1].lower() in audio_exts
for f in os.listdir(src_dir) if os.path.isfile(os.path.join(src_dir, f)))
if has_audio:
continue
for sidecar in _album_sidecars:
sidecar_src = os.path.join(src_dir, sidecar)
if os.path.isfile(sidecar_src):
sidecar_dst = os.path.join(dest_dir, sidecar)
if not os.path.exists(sidecar_dst):
try:
shutil.move(sidecar_src, sidecar_dst)
logger.info(f"[Reorganize] Moved {sidecar} to {dest_dir}")
except Exception as sc_err:
logger.error(f"[Reorganize] Failed to move {sidecar}: {sc_err}")
# Clean up empty directories left behind (after sidecars moved)
for src_dir in moved_dirs:
try:
_cleanup_empty_directories(transfer_dir, os.path.join(src_dir, '_'))
except Exception:
pass
except Exception as e:
logger.error(f"Reorganize background error: {e}")
def _run():
from core.library_reorganize import reorganize_album
try:
summary = reorganize_album(
album_id=album_id,
db=get_database(),
staging_root=staging_root,
resolve_file_path_fn=_resolve_library_file_path,
post_process_fn=_post_process_matched_download,
update_track_path_fn=_update_track_path,
cleanup_empty_dir_fn=_cleanup_empty,
transfer_dir=transfer_dir,
on_progress=_on_progress,
primary_source=chosen_source,
strict_source=bool(chosen_source),
stop_check=lambda: bool(IS_SHUTTING_DOWN),
)
logger.info(
f"[Reorganize] Album {album_id} {summary['status']} "
f"(source={summary.get('source')}, moved={summary['moved']}, "
f"skipped={summary['skipped']}, failed={summary['failed']})"
)
except Exception as run_err:
logger.error(f"[Reorganize] Background error: {run_err}", exc_info=True)
finally:
if bg_conn:
try:
bg_conn.close()
except Exception:
pass
with _reorganize_lock:
_reorganize_state['status'] = 'done'
_reorganize_state['current_track'] = ''
thread = threading.Thread(target=_run_reorganize, daemon=True, name="ReorganizeAlbum")
thread.start()
return jsonify({"success": True, "message": "Reorganization started", "total": len(tracks)})
threading.Thread(target=_run, daemon=True, name="ReorganizeAlbum").start()
return jsonify({"success": True, "message": "Reorganization started"})
except Exception as e:
logger.error(f"Reorganize error: {e}")

1
webui/static/helper.js
View file

@ -3443,6 +3443,7 @@ const WHATS_NEW = {
'2.40': [
// --- Search & Artists unification (in progress, not yet released) ---
{ date: 'Unreleased — Search & Artists unification', unreleased: true },
{ title: 'Library Reorganize: Reroute Through the Download Pipeline', desc: 'Reported by winecountrygames — using "Reorganize All" on a 3-disc Aerosmith deluxe collapsed it to a flat 1-disc layout, and on other albums it left half the tracks in their original location with no error or count of what was skipped. Root cause: the reorganize endpoint reinvented several wheels (its own template engine, its own disc-number resolution from file tags, its own sidecar sweep, its own collision detection) and each had drifted from the canonical post-processing path used by downloads. The reorganize-only logic read disc_number from file tags and silently defaulted to 1 on any failure, so a single tag-less file collapsed the whole album to single-disc. Tracks whose file paths didn\'t resolve on disk were silently skipped. Rewrote it to follow the import page\'s pattern: copy each file to a per-album staging folder under your download path, look up the canonical tracklist from your configured metadata source (Deezer / Spotify / iTunes / Discogs / Hydrabase) using the album\'s stored source IDs, then route each file through the same `_post_process_matched_download` function fresh downloads use — same template, same tagging, same multi-disc subfolder logic, same sidecar handling, same AcoustID verification. Albums with no stored source ID are reported back and skipped entirely (degrading silently to file tags is what caused the original bug). Tracks not in the source\'s catalog version (bonus tracks on a deluxe edition) are reported as skipped and left in place rather than force-fed wrong context. Files that don\'t resolve on disk are surfaced with the offending DB path so the UI can show them. The 230-line inline reorganize logic in web_server.py was extracted into core/library_reorganize.py — net -195 lines from the monolith, +13 unit tests for the new orchestrator. Frontend behavior change: the per-call template parameter in the reorganize modal is now ignored — reorganize uses your configured download template, matching the pipeline downloads use', page: 'library', unreleased: true },
{ title: 'Spotify: Longer Post-Ban Cooldown (30 min)', desc: 'A user reported their Spotify rate-limit ban expired after 4 hours, the system ran its 5-minute post-ban cooldown, and then 32 seconds after the cooldown ended a single get_artist_albums call from a background worker was hit with another 4-hour ban. Diagnosis: Spotify\'s server-side memory of the previous offense outlasted our 5-minute cooldown, so the very first call after cooldown got slapped immediately. The cooldown exists specifically to prevent the "ban expires → we probe → re-ban" cycle, but the value was too short. Bumped from 5 minutes to 30 minutes — same mechanism, just enough room for Spotify to actually forget. A more principled follow-up (adaptive cooldown that scales with the previous ban size, plus making the first post-cooldown call a single light probe rather than allowing background workers through) is documented as a future PR if reports persist after this bump', page: 'dashboard', unreleased: true },
{ title: 'Tidal: Reject Silent Quality Downgrades', desc: 'Netti93 reported that with Tidal set to "HiRes only" and quality fallback disabled, tracks were still downloading successfully — as m4a 320kbps files. Root cause: Tidal\'s API silently serves whatever tier your account + the track + your region permits. Ask for HI_RES_LOSSLESS on a track that\'s only in LOW_320K and Tidal returns the AAC stream without raising. The downloader wrote the m4a to disk, the filesize cleared the 100KB stub threshold, and the download reported success. The worker-level fallback chain (hires → lossless → high → low) also never got a chance to advance, because every tier "succeeded" at the first one that returned anything. Fix: after getting the stream, compare stream.audio_quality against what we requested using a rank-based tier comparison (LOW < HIGH < LOSSLESS < HI_RES < HI_RES_LOSSLESS). Same tier or better = accept (so occasional Tidal upgrades don\'t get thrown away). Lower tier = treat this tier as failed, which lets the fallback chain advance when fallback is enabled or fails the whole download honestly when the user has "HiRes only, no fallback" configured. Unrecognized audioQuality values (a new Tidal tier we haven\'t mapped yet) are rejected conservatively so the final diagnostic log can name the unknown value. Older tidalapi builds without the audio_quality attribute fall through to the pre-existing codec / file-size guards so nothing regresses', page: 'downloads', unreleased: true },
{ title: 'Search Source Picker Icon Row', desc: 'The Search page now has a row of source icons above the search bar — one per source (Spotify, Apple Music, Deezer, Discogs, Hydrabase, MusicBrainz, Music Videos, Soulseek). Typing searches only the currently-selected source instead of fanning out to every one by default. Click a different icon to switch; results come back on demand. The default icon on page load is your configured primary metadata source. Replaces the short-lived "Search from" dropdown that preceded this', page: 'search', unreleased: true },

208
webui/static/library.js
View file

@ -2805,7 +2805,7 @@ function renderArtistMetaPanel(artist) {
const reorgAllBtn = document.createElement('button');
reorgAllBtn.className = 'enhanced-sync-btn';
reorgAllBtn.innerHTML = '&#128193; Reorganize All';
reorgAllBtn.title = 'Reorganize all albums for this artist using path template';
reorgAllBtn.title = 'Reorganize all albums for this artist using your configured download template';
reorgAllBtn.onclick = () => _showReorganizeAllModal();
headerRight.appendChild(reorgAllBtn);
@ -3271,7 +3271,7 @@ function renderExpandedAlbumHeader(album) {
const reorganizeBtn = document.createElement('button');
reorganizeBtn.className = 'enhanced-reorganize-album-btn';
reorganizeBtn.innerHTML = '&#128193; Reorganize';
reorganizeBtn.title = 'Reorganize album files using a custom path template';
reorganizeBtn.title = 'Reorganize album files using your configured download template';
reorganizeBtn.onclick = (e) => { e.stopPropagation(); showReorganizeModal(album.id); };
enrichRow.appendChild(reorganizeBtn);
@ -6132,52 +6132,19 @@ async function showReorganizeModal(albumId) {
applyBtn.onclick = () => executeReorganize();
}
// Build modal content
const variables = [
{ var: '$artist', desc: 'Track artist', example: artistName || 'Artist' },
{ var: '$albumartist', desc: 'Album artist', example: artistName || 'Album Artist' },
{ var: '$artistletter', desc: 'First letter of artist', example: (artistName || 'A')[0].toUpperCase() },
{ var: '$album', desc: 'Album title', example: albumData ? albumData.title : 'Album' },
{ var: '$albumtype', desc: 'Album/EP/Single', example: 'Album' },
{ var: '$title', desc: 'Track title', example: 'Track Name' },
{ var: '$track', desc: 'Track number (zero-padded)', example: '01' },
{ var: '$disc', desc: 'Disc number (filename only)', example: '01' },
{ var: '$cdnum', desc: 'CD label — "CD01" on multi-disc, empty otherwise', example: 'CD01' },
{ var: '$year', desc: 'Release year', example: albumData && albumData.year ? String(albumData.year) : '2024' },
{ var: '$quality', desc: 'Audio quality (filename only)', example: 'FLAC 16bit/44kHz' },
];
let html = '<div class="reorganize-content">';
// Template input
html += '<div class="reorganize-template-section">';
html += '<label class="reorganize-label">Path Template</label>';
html += '<div class="reorganize-template-hint">Use <code>/</code> to separate folders. The last segment becomes the filename.</div>';
// Load saved template from settings, fall back to default
let savedTemplate = '$albumartist/$albumartist - $album/$track - $title';
try {
const settingsResp = await fetch('/api/settings');
if (settingsResp.ok) {
const settings = await settingsResp.json();
savedTemplate = settings.file_organization?.templates?.album_path || savedTemplate;
}
} catch (_) { }
html += '<input type="text" id="reorganize-template-input" class="reorganize-template-input" ';
html += `value="${savedTemplate.replace(/"/g, '&quot;')}" `;
html += 'placeholder="$albumartist/$album/$track - $title" spellcheck="false">';
// Metadata source picker — populated from /reorganize/sources.
// Empty value = use configured primary (with fallback chain).
// Specific source = strict mode, that source only.
html += '<div class="reorganize-source-section">';
html += '<label class="reorganize-label">Metadata Source</label>';
html += '<div class="reorganize-template-hint">Pick which source to read the album\'s tracklist from. Defaults to your configured primary. Reorganize uses your global download template, same as fresh downloads.</div>';
html += '<select id="reorganize-source-select" class="reorganize-template-input">';
html += '<option value="">Use configured primary (auto)</option>';
html += '</select>';
html += '</div>';
// Variables reference
html += '<div class="reorganize-variables">';
html += '<label class="reorganize-label">Available Variables</label>';
html += '<div class="reorganize-var-grid">';
variables.forEach(v => {
html += `<div class="reorganize-var-chip" onclick="insertReorganizeVar('${v.var}')" title="${escapeHtml(v.desc)} — e.g. ${escapeHtml(v.example)}">`;
html += `<code>${v.var}</code><span class="reorganize-var-desc">${v.desc}</span>`;
html += '</div>';
});
html += '</div></div>';
// Preview area
html += '<div class="reorganize-preview-section">';
html += '<div class="reorganize-preview-header">';
@ -6192,31 +6159,34 @@ async function showReorganizeModal(albumId) {
body.innerHTML = html;
overlay.classList.remove('hidden');
// Wire up live preview on enter key
setTimeout(() => {
const input = document.getElementById('reorganize-template-input');
if (input) {
input.addEventListener('keydown', (e) => {
if (e.key === 'Enter') {
e.preventDefault();
loadReorganizePreview();
}
});
input.focus();
}
}, 50);
// Populate source picker after the modal mounts
setTimeout(() => _populateReorganizeSources(_reorganizeAlbumId), 50);
}
function insertReorganizeVar(varName) {
const input = document.getElementById('reorganize-template-input');
if (!input) return;
const start = input.selectionStart;
const end = input.selectionEnd;
const val = input.value;
input.value = val.substring(0, start) + varName + val.substring(end);
input.focus();
const newPos = start + varName.length;
input.setSelectionRange(newPos, newPos);
async function _populateReorganizeSources(albumId) {
const select = document.getElementById('reorganize-source-select');
if (!select || !albumId) return;
try {
const resp = await fetch(`/api/library/album/${albumId}/reorganize/sources`);
if (!resp.ok) return;
const data = await resp.json();
const sources = data.sources || [];
// Keep the "auto" default option, append concrete sources beneath it.
sources.forEach(s => {
const opt = document.createElement('option');
opt.value = s.source;
opt.textContent = s.label || s.source;
select.appendChild(opt);
});
if (sources.length === 0) {
const opt = document.createElement('option');
opt.disabled = true;
opt.textContent = 'No sources available — run enrichment first';
select.appendChild(opt);
}
} catch (err) {
console.error('Failed to load reorganize sources:', err);
}
}
function closeReorganizeModal() {
@ -6226,19 +6196,19 @@ function closeReorganizeModal() {
}
async function loadReorganizePreview() {
const template = document.getElementById('reorganize-template-input')?.value?.trim();
const previewBody = document.getElementById('reorganize-preview-body');
const applyBtn = document.getElementById('reorganize-apply-btn');
if (!template || !previewBody || !_reorganizeAlbumId) return;
if (!previewBody || !_reorganizeAlbumId) return;
if (applyBtn) applyBtn.disabled = true;
previewBody.innerHTML = '<div class="reorganize-preview-loading">Loading preview...</div>';
try {
const chosenSource = document.getElementById('reorganize-source-select')?.value || '';
const response = await fetch(`/api/library/album/${_reorganizeAlbumId}/reorganize/preview`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ template })
body: JSON.stringify({ source: chosenSource })
});
const result = await response.json();
if (!result.success) {
@ -6262,31 +6232,52 @@ async function loadReorganizePreview() {
const unchanged = t.unchanged;
const noFile = !t.file_exists;
const collision = t.collision;
if (!unchanged && t.file_exists) hasChanges = true;
const unmatched = (t.matched === false);
const missingPath = !unmatched && !noFile && !t.new_path; // matched but path-build failed
if (!unchanged && t.file_exists && !unmatched && !missingPath) hasChanges = true;
if (collision) hasCollisions = true;
const rowClass = collision ? 'reorganize-row-collision' : noFile ? 'reorganize-row-missing' : unchanged ? 'reorganize-row-unchanged' : 'reorganize-row-changed';
let rowClass;
if (collision) rowClass = 'reorganize-row-collision';
else if (noFile || unmatched || missingPath) rowClass = 'reorganize-row-missing';
else if (unchanged) rowClass = 'reorganize-row-unchanged';
else rowClass = 'reorganize-row-changed';
const arrow = collision ? '!!'
: unchanged ? '='
: (noFile || unmatched || missingPath) ? '⊘'
: '→';
const newCell = noFile ? ''
: unmatched ? `<em>${escapeHtml(t.reason || 'Not in selected source\'s tracklist')}</em>`
: missingPath ? `<em>${escapeHtml(t.reason || 'Couldn\'t compute destination path')}</em>`
: (escapeHtml(t.new_path) + (collision ? ' <em>(collision)</em>' : ''));
html += `<tr class="${rowClass}">`;
html += `<td>${t.track_number || ''}</td>`;
html += `<td>${escapeHtml(t.title)}</td>`;
html += `<td class="reorganize-path">${noFile ? '<em>File not found</em>' : escapeHtml(t.current_path)}</td>`;
html += `<td class="reorganize-arrow">${collision ? '!!' : unchanged ? '=' : noFile ? '' : '→'}</td>`;
html += `<td class="reorganize-path">${noFile ? '' : escapeHtml(t.new_path)}${collision ? ' <em>(collision)</em>' : ''}</td>`;
html += `<td class="reorganize-arrow">${arrow}</td>`;
html += `<td class="reorganize-path">${newCell}</td>`;
html += '</tr>';
});
html += '</tbody></table>';
const changedCount = tracks.filter(t => !t.unchanged && t.file_exists && !t.collision).length;
const changedCount = tracks.filter(t => !t.unchanged && t.file_exists && !t.collision && t.matched !== false && t.new_path).length;
const skippedCount = tracks.filter(t => t.unchanged).length;
const missingCount = tracks.filter(t => !t.file_exists).length;
const collisionCount = tracks.filter(t => t.collision).length;
const unmatchedCount = tracks.filter(t => t.file_exists && t.matched === false).length;
const noPathCount = tracks.filter(t => t.file_exists && t.matched !== false && !t.new_path && !t.collision).length;
let summary = `<div class="reorganize-preview-summary">`;
if (changedCount > 0) summary += `<span class="reorganize-stat changed">${changedCount} will move</span>`;
if (skippedCount > 0) summary += `<span class="reorganize-stat unchanged">${skippedCount} unchanged</span>`;
if (missingCount > 0) summary += `<span class="reorganize-stat missing">${missingCount} missing</span>`;
if (collisionCount > 0) summary += `<span class="reorganize-stat collision">${collisionCount} collision${collisionCount !== 1 ? 's' : ''} — add $track or $disc to fix</span>`;
if (unmatchedCount > 0) summary += `<span class="reorganize-stat missing">${unmatchedCount} not in source — try a different source</span>`;
if (noPathCount > 0) summary += `<span class="reorganize-stat missing">${noPathCount} couldn't compute destination</span>`;
if (missingCount > 0) summary += `<span class="reorganize-stat missing">${missingCount} missing on disk</span>`;
if (collisionCount > 0) summary += `<span class="reorganize-stat collision">${collisionCount} collision${collisionCount !== 1 ? 's' : ''} — likely a source data issue</span>`;
summary += '</div>';
previewBody.innerHTML = summary + html;
@ -6300,8 +6291,7 @@ async function loadReorganizePreview() {
}
async function executeReorganize() {
const template = document.getElementById('reorganize-template-input')?.value?.trim();
if (!template || !_reorganizeAlbumId) return;
if (!_reorganizeAlbumId) return;
const applyBtn = document.getElementById('reorganize-apply-btn');
if (applyBtn) {
@ -6310,10 +6300,11 @@ async function executeReorganize() {
}
try {
const chosenSource = document.getElementById('reorganize-source-select')?.value || '';
const response = await fetch(`/api/library/album/${_reorganizeAlbumId}/reorganize`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ template })
body: JSON.stringify({ source: chosenSource })
});
const result = await response.json();
if (!result.success) throw new Error(result.error);
@ -6392,23 +6383,17 @@ async function _showReorganizeAllModal() {
title.textContent = `Reorganize All Albums — ${artistName}`;
// Load saved template
let savedTemplate = '$albumartist/$albumartist - $album/$track - $title';
try {
const settingsResp = await fetch('/api/settings');
if (settingsResp.ok) {
const settings = await settingsResp.json();
savedTemplate = settings.file_organization?.templates?.album_path || savedTemplate;
}
} catch (_) { }
let html = '<div class="reorganize-content">';
// Template input
html += '<div class="reorganize-template-section">';
html += '<label class="reorganize-label">Path Template</label>';
html += '<div class="reorganize-template-hint">This template will be applied to all albums below. Use <code>/</code> to separate folders.</div>';
html += `<input type="text" id="reorganize-template-input" class="reorganize-template-input" value="${savedTemplate.replace(/"/g, '&quot;')}" placeholder="$albumartist/$album/$track - $title" spellcheck="false">`;
// Source picker — applies to ALL albums in this run. Albums without
// an ID for the chosen source will be skipped at the backend with
// a clear status. Auto = use configured primary with fallback chain.
html += '<div class="reorganize-source-section">';
html += '<label class="reorganize-label">Metadata Source (applies to all albums)</label>';
html += '<div class="reorganize-template-hint">Pick which source to read tracklists from. Albums without an ID for that source will be skipped. Reorganize uses your global download template, same as fresh downloads.</div>';
html += '<select id="reorganize-source-select" class="reorganize-template-input">';
html += '<option value="">Use configured primary (auto)</option>';
html += '</select>';
html += '</div>';
// Album list
@ -6434,25 +6419,37 @@ async function _showReorganizeAllModal() {
}
overlay.classList.remove('hidden');
// Populate the source dropdown from the global authed-sources endpoint
setTimeout(async () => {
const select = document.getElementById('reorganize-source-select');
if (!select) return;
try {
const resp = await fetch('/api/library/reorganize/sources');
if (!resp.ok) return;
const data = await resp.json();
(data.sources || []).forEach(s => {
const opt = document.createElement('option');
opt.value = s.source;
opt.textContent = s.label || s.source;
select.appendChild(opt);
});
} catch (err) {
console.error('Failed to load reorganize sources:', err);
}
}, 50);
}
async function _executeReorganizeAll() {
if (_reorganizeAllRunning) return;
const templateInput = document.getElementById('reorganize-template-input');
const template = templateInput ? templateInput.value.trim() : '';
if (!template) {
showToast('Template cannot be empty', 'error');
return;
}
const albums = artistDetailPageState.enhancedData.albums || [];
const total = albums.length;
const artistName = artistDetailPageState.enhancedData.artist?.name || 'this artist';
const confirmed = await showConfirmDialog({
title: 'Reorganize All Albums',
message: `This will reorganize ${total} album${total !== 1 ? 's' : ''} for ${artistName} using the template:\n\n${template}\n\nFiles will be moved and renamed. This cannot be undone.`,
message: `This will reorganize ${total} album${total !== 1 ? 's' : ''} for ${artistName} using your configured download template. Files will be moved and renamed. This cannot be undone.`,
confirmText: 'Reorganize All',
destructive: false,
});
@ -6466,6 +6463,9 @@ async function _executeReorganizeAll() {
const overlay = document.getElementById('reorganize-overlay');
if (overlay) overlay.classList.add('hidden');
// Source picker is captured ONCE before the loop — same source for every album
const chosenSource = document.getElementById('reorganize-source-select')?.value || '';
let succeeded = 0, failed = 0;
for (let i = 0; i < total; i++) {
@ -6476,7 +6476,7 @@ async function _executeReorganizeAll() {
const resp = await fetch(`/api/library/album/${album.id}/reorganize`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ template }),
body: JSON.stringify({ source: chosenSource }),
});
const result = await resp.json();
if (!result.success) {

13
webui/static/style.css
View file

@ -46230,6 +46230,19 @@ textarea.enhanced-meta-field-input {
.reorganize-template-input:focus {
border-color: rgba(100, 149, 237, 0.5);
}
/* `<select>` shares the .reorganize-template-input class but its dropdown
options are rendered by the OS, so they don't inherit page colors —
explicitly style options dark so they're visible against the OS's
default white dropdown background.*/
select.reorganize-template-input,
.reorganize-template-input option {
background: #1f1f1f;
color: #e0e0e0;
}
.reorganize-template-input:disabled {
opacity: 0.55;
cursor: not-allowed;
}
.reorganize-variables {
padding-bottom: 12px;
border-bottom: 1px solid rgba(255,255,255,0.06);