Commit graph

3116 commits

Author SHA1 Message Date
Broque Thomas
4fce832ae1 Add Amazon Music enrichment worker
Background worker matching library artists/albums/tracks to Amazon ASINs
via T2Tunes search. Follows same 6-tier priority queue as Deezer/iTunes/
Spotify/Qobuz/Tidal workers. Backfills artist thumbnails from album cover
stand-ins (T2Tunes exposes no direct artist images).

- core/amazon_worker.py: new AmazonWorker class with full parity
- database/music_database.py: expand _add_amazon_columns to cover
  amazon_id/amazon_match_status/amazon_last_attempted on artists,
  albums, and tracks (was artists-only)
- web_server.py: import, init, register in enrichment panel, add to
  scan pause/resume dicts and rate monitor key map
- helper.js: WHATS_NEW 2.5.3 entry for enrichment worker
2026-05-16 17:30:48 -07:00
Broque Thomas
121651da2c Add amazon_id column to artists table for full source parity
Schema: ALTER TABLE artists ADD COLUMN amazon_id TEXT with index, added via
_add_amazon_columns migration called after Discogs in _run_migrations.

SOURCE_ID_FIELD: add "amazon" -> "amazon_id" entry. find_library_artist_for_
source now looks up Amazon artists by slug before falling back to name match,
same as every other source. artist_source_detail already stamps artist_info
[source_id_field] = artist_id so the amazon_id is set on source-only payloads.

Tests: add "amazon": "amazon_id" to EXPECTED_SOURCE_ID_FIELD; revert test
assertion back to strict equality (SOURCE_ONLY_ARTIST_SOURCES == SOURCE_ID_
FIELD.keys() holds again now that amazon has a column).
2026-05-16 17:06:54 -07:00
Broque Thomas
265fe5233e Fix Amazon artist detail: library upgrade lookup and artist images
Library upgrade: find_library_artist_for_source returned None immediately for
Amazon because SOURCE_ID_FIELD has no 'amazon' entry (no DB column for Amazon
artist IDs). The name-based fallback was unreachable. Fix: only skip the column
query when column is None, not the whole function — name lookup now runs for
any source when artist_name + active_server are provided.

Artist images: add AmazonClient._get_artist_image_from_albums so the standard
_get_artist_image_from_source path in metadata/artist_image.py can call it as
a fallback (same hook iTunes/Deezer/Discogs expose). Searches by unslugified
artist name, matches primary artist, fetches album cover from album_metadata.

Test: updated test_source_only_set_matches_mapping_keys → _contains_all_mapped_
sources to assert subset (not equality) — SOURCE_ONLY_ARTIST_SOURCES intentionally
includes sources without a DB column that rely on name-only lookup.
2026-05-16 17:01:37 -07:00
Broque Thomas
d944884ab4 Backfill album release_date from stream tags when T2Tunes metadata omits it
T2Tunes albumList entries may not include a release_date field, leaving the
$year path template empty. get_album() now falls back to the first track's
release_date (populated from the FLAC date tag via get_album_tracks) when
album metadata has none. Also try camelCase releaseDate key at all albumList
read sites (Album.from_metadata, get_album, _fetch_album_metas consumers).

1 new test: release_date backfilled from stream date tag when absent from
album metadata. date tag "2024-11-22" added to MEDIA_RESPONSE_FLAC fixture.
2026-05-16 16:32:54 -07:00
Broque Thomas
96a1c8b7b8 Enrich Amazon album track durations via search results
media_from_asin returns no duration data. get_album_tracks now does one
search_raw call using the album name + primary artist from stream tags,
filters hits by albumAsin == requested asin, and builds a duration_map
(track asin → duration_ms). Search failures are swallowed — duration_ms
falls back to 0 so the existing behaviour is preserved on error.

2 new tests: duration populated when search returns matching hit; duration
stays 0 when search endpoint returns an error.
2026-05-16 16:21:12 -07:00
Broque Thomas
c2fcef45c2 Fix $year missing and disc_number crash on Amazon album downloads
release_date: T2Tunes album metadata may use camelCase releaseDate — try both
keys at all read sites (get_album, get_track_details, Album.from_metadata,
_fetch_album_metas consumers). Final fallback: s.date from stream tags, which
T2Tunes always populates from embedded FLAC/MP4 date tag. Wire s.date into
get_album_tracks items and get_track_details album.release_date so the $year
path template resolves correctly.

disc_number crash: .get('disc_number', 1) returns None when key is present but
value is None (Amazon stream info has Optional[int] for disc_number). Switch all
max() call sites and disc_num assignments to `or 1` guard:
- master.py: run_full_missing_tracks_process max() and disc_num read
- candidates.py: track_info and detailed_track disc_number reads
- web_server.py: enhanced and standard album download max() calls
2026-05-16 16:11:57 -07:00
Broque Thomas
8a3bb88678 Fix AcoustID quarantine and disc_number crash on Amazon album downloads
AcoustID verification was quarantining every Amazon track because T2Tunes
embeds [Explicit] and [feat. X] in stream tag titles/artists, but AcoustID
returns bare titles — triggering version-mismatch rejection on every track.

- get_track_details: apply _strip_edition to name/album, _primary_artist to
  artist; wire s.track_number / s.disc_number instead of hardcoded None
- get_album_tracks: apply _strip_edition to name, _primary_artist to artist

Also fix TypeError crash in album download paths when disc_number is None
(present in dict but explicitly None, so .get('disc_number', 1) returns None):
- master.py run_full_missing_tracks_process: or 1 guard on both max() and disc_num
- candidates.py track_info extraction: or 1 guard on both disc_number reads
- web_server.py enhanced + standard album download max() calls: or 1 guard
2026-05-16 16:01:11 -07:00
Broque Thomas
51e00d4ebf Fix Amazon Music search quality: images, dedup, explicit stripping, album/artist clicks
- All search_raw calls switched from single-type to types="track,album" — T2Tunes only
  returns results when both types are requested together
- _fetch_album_metas: parallel fetch (up to 5 workers) of album cover art via
  album_metadata(asin) — T2Tunes search results carry no image URLs
- search_tracks: populates image_url, release_date, total_tracks from album meta
- search_artists: strips feat. credits via _primary_artist() so "Artist feat. X" and
  "Artist ft. Y" collapse to one "Artist" entry; uses album cover as artist image
  stand-in (same approach as iTunes — T2Tunes has no artist images)
- search_albums: name-based dedup (display_name + artist key) instead of ASIN-based;
  populates image_url, release_date, total_tracks from album meta (cap 10 ASIN fetches)
- _strip_edition(): strips [Explicit]/(Explicit) from track/album names — explicit is
  the default version; Clean/Edited/Censored labels kept as-is so they stay distinct
- get_album(): applies _strip_edition to name and _primary_artist to artist so
  MusicBrainz preflight matching doesn't fail on "[Explicit]" album names
- get_album_tracks(): populates track_number and disc_number from T2TunesStreamInfo
  instead of hardcoding None — fixes track ordering in multi-track album downloads
- get_artist() / get_artist_albums(): _unslugify() converts slug artist IDs back to
  search names; _primary_artist() in comparison handles feat-annotated results
- SOURCE_ONLY_ARTIST_SOURCES: added "amazon" so artist detail page doesn't 404
- build_source_only_artist_detail: added amazon_client param + dispatch branch
- web_server.py: resolve amazon_client in _build_source_only_artist_detail wrapper;
  add source_override=="amazon" branch in get_spotify_album_tracks endpoint
- 77 tests covering all above paths; all pass
2026-05-16 15:55:15 -07:00
Broque Thomas
d39679951b Wire Amazon Music into enhanced search and global search source picker
- Add 'amazon' to VALID_SOURCES (and transitively VALID_STREAM_SOURCES)
  in core/search/orchestrator.py so the backend accepts it as a
  requested source without returning 400
- Add resolve_client('amazon') case — mirrors musicbrainz pattern,
  gets the cached AmazonClient from the metadata registry
- Add 'amazon' to _alternate_sources() so it appears as a tab when
  another source is primary (always available, no credentials)
- Add SERVICE_CONFIG_REGISTRY entry 'amazon': {'always': True} so
  /api/settings/config-status reports it as configured
- Add SOURCE_LABELS['amazon'] and SOURCE_ORDER entry in
  shared-helpers.js so both enhanced search and global search show
  the Amazon Music tab
- Add 'amazon' to _ALWAYS_CONFIGURED_SOURCES so the picker never
  dims the tab (no credentials required)
- Add .enh-tab-amazon.active CSS (Amazon orange #FF9900)
- 3530 tests pass
2026-05-16 14:18:18 -07:00
Broque Thomas
1f579cede8 Add Amazon Music as a primary metadata source
Wires AmazonClient into the metadata source registry following the
exact same pattern as DeezerClient. No existing source paths touched.

- Add get_album_metadata / get_artist_info / get_artist_albums_list
  aliases to AmazonClient (mirrors DeezerClient interface aliases)
- Register amazon in METADATA_SOURCE_PRIORITY and METADATA_SOURCE_LABELS
- Add _get_amazon_factory() + get_amazon_client() to registry.py
- Add amazon branch to get_client_for_source(); thread amazon_client_factory
  kwarg through get_primary_client() and get_primary_source_status()
- Re-export get_amazon_client from the core.metadata_service shim
- Add Amazon Music option to Settings metadata source dropdown
- 3530 tests pass
2026-05-16 13:52:26 -07:00
BoulderBadgeDad
e8b9f80597
Merge pull request #615 from Nezreka/feature/amazon-music-download
Feature/amazon music download
2026-05-16 12:56:09 -07:00
Broque Thomas
14a99f47ab fix(tests): use asyncio.run() instead of get_event_loop() in amazon test helper
get_event_loop() raises RuntimeError on Python 3.11+ Linux when no loop
exists. asyncio.run() creates its own loop per call — no deprecation warning,
works across all supported Python versions.
2026-05-16 12:48:36 -07:00
Broque Thomas
5d8ca70fe5 Add T2Tunes probe unit tests
Tests for tools/t2tunes_probe.py: status endpoint, nested search
response flattening, streamable typo tolerance + decryption key flag,
non-JSON error handling, and HEAD→range-GET fallback for stream probing.
2026-05-16 11:49:03 -07:00
Broque Thomas
ff27effdae Amazon download client: write final size==transferred before returning file path
The download monitor blocks post-processing with a bytes-incomplete guard:
if size > 0 and transferred < size: continue

_stream_to_file throttles engine updates to every 0.5s. The last tick before
the file finishes typically leaves transferred slightly below the Content-Length
size in the engine record. Other streaming clients (YouTube, Tidal, HiFi, etc.)
use their own download threads and don't track bytes at all, so size stays 0
and the guard is always skipped. Amazon was the only client hitting it.

Fix: just before returning the file path from _download_sync, write a final
engine record update setting size == transferred == out_path.stat().st_size
(the decrypted output size). The bytes-incomplete guard then sees
transferred == size and falls through to trigger post-processing normally.
2026-05-16 11:09:04 -07:00
Broque Thomas
b4403ed393 Amazon download client: fix engine API calls in status methods
`get_all_downloads` was calling `engine.get_all_records()` — a method that
doesn't exist on DownloadEngine. Same story for `cancel_record` and
`clear_completed`. The engine exposes `iter_records_for_source`, `get_record`,
`update_record`, and `remove_record` — matching what every other streaming
client (Deezer, HiFi, Qobuz, SoundCloud, Tidal, YouTube) already uses.

With `get_all_downloads` silently returning `[]` on every call (the missing
method raised, `except Exception: return []` swallowed it), the download monitor
never saw Amazon records as complete — tasks stayed stuck at 0% even after the
file had fully downloaded.

Changes:
- `get_all_downloads` → `iter_records_for_source('amazon')`
- `get_download_status` → `get_record('amazon', id)`, no try/except
- `cancel_download` → `get_record` check + `update_record` (Cancelled) +
  optional `remove_record` — same pattern as deezer/hifi/etc
- `clear_all_completed_downloads` → iterate + `remove_record` for terminal
  states; returns True on no-engine (nothing to clear = success)
- `_record_to_status` drops the `download_id` argument; reads `rec['id']`
  instead (worker stores `'id'` in every record — `iter_records_for_source`
  returns the full record dict)

Tests updated to match: `iter_records_for_source` mock replaces
`get_all_records`, cancel test verifies `update_record`+`remove_record`,
clear test verifies only terminal-state records are removed, graceful-error
test replaced with no-records boundary test (exception propagation is handled
at the engine aggregator layer, not per-plugin).
2026-05-16 10:49:11 -07:00
Broque Thomas
ebda0b8613 fix(amazon): _record_to_status read 'filename' not 'original_filename'
The engine worker stores the encoded filename under the key 'filename'
(see worker.py dispatch). _record_to_status was reading 'original_filename',
which always returns "" — so every DownloadStatus emitted by
get_all_downloads/get_download_status had an empty filename string.

The download monitor builds lookup keys as
_make_context_key(download.username, download.filename). With filename=""
the key was always "amazon::" which never matched the task's
"amazon::B0B1234||Artist - Title" key. Monitor never detected Amazon
download completions, so tasks sat stuck at Downloading 0% forever even
though the files had actually downloaded.

Also fixes tests that had the same wrong key.
2026-05-16 10:37:29 -07:00
Broque Thomas
9fb63ff86d fix(amazon): add set_engine/set_shutdown_check so _engine gets wired
AmazonDownloadClient was missing set_engine() and set_shutdown_check().
The download engine auto-wires plugins by calling set_engine(self) at
registration time if the method exists (engine.py:136). Without it,
_engine stayed None forever, causing every download() call to raise
RuntimeError("_engine is not set") — silently failing and marking all
tracks not found.

All other streaming clients (Deezer, Qobuz, Tidal, HiFi, SoundCloud)
expose set_engine(); Amazon now matches the pattern.

Tests added: set_engine wires _engine, set_shutdown_check wires callback,
set_engine unblocks download dispatch (the exact live failure mode).
2026-05-16 10:31:17 -07:00
Broque Thomas
791e3630ff fix(amazon): wire amazon into all streaming-source guards
`validation.py` had amazon absent from `_streaming_sources`, causing
Amazon TrackResult objects (bitrate=None, size=0) to fall through to
the Soulseek P2P code path and get rejected by
`filter_results_by_quality_preference`. Every album track was marked
not found.

Fix: add 'amazon' to every streaming-source guard tuple/set that was
previously missing it:
- core/downloads/validation.py — primary bug fix (quality-filter bypass)
- core/downloads/status.py — _STREAMING_SOURCE_NAMES frozenset
- core/downloads/task_worker.py — hybrid fallback client map
- core/imports/side_effects.py — || filename→stream-id extraction
- web_server.py — is_streaming_source, transfer list display,
  candidate source label, _try_source_reuse, _store_batch_source
- tests/test_download_plugin_conformance.py — registry count + parametrize

Also updates the 2.5.3 What's New entry to drop the stale
"not yet wired" disclaimer.
2026-05-16 10:24:13 -07:00
Broque Thomas
dcbe09c7aa Add Amazon Music to download source mode dropdown 2026-05-16 10:04:14 -07:00
Broque Thomas
fa73c41ef6 Wire Amazon Music as a first-class download source
Follows the exact same standard as Tidal, Qobuz, HiFi, and Deezer.

registry.py — import + register AmazonDownloadClient as 'amazon'.

amazon_download_client.py — read amazon_download.quality / allow_fallback
from config on init; pass quality as preferred_codec to AmazonClient;
_download_sync codec waterfall respects allow_fallback flag.

download_orchestrator.py — reload_settings() updates preferred_codec +
allow_fallback on the live client after a settings save. 'amazon' added
to _streaming_sources so search_and_download_best routes it correctly.

api_call_tracker.py — 'amazon' registered in RATE_LIMITS (120/min),
SERVICE_LABELS, and SERVICE_ORDER so API call monitoring shows Amazon.

web_server.py — 'amazon_download' added to the settings service loop.
'amazon' added to serverless_sources (no slskd probe needed). Streaming
file-finder extended to handle amazon username + ||asin||title encoding
(extension-less fuzzy match, same as Tidal/Qobuz/HiFi). New endpoint:
GET /api/amazon/test-connection → checks T2Tunes proxy status.

webui/index.html — amazon-download-settings-container: quality dropdown
(flac/opus/eac3), allow-fallback checkbox, test-connection button.

webui/static/settings.js — 'Amazon Music' added to HYBRID_SOURCES,
_hybridSourceEnabled, allSources mode list, loadSettings(), saveSettings()
payload, updateDownloadSourceUI() show/hide + auto-test. New
testAmazonConnection() function.
2026-05-16 09:40:50 -07:00
Broque Thomas
85984d4174 Amazon Music provider: metadata client + download source (T2Tunes)
core/amazon_client.py — T2Tunes-backed metadata client following the
DeezerClient/iTunesClient contract. Exposes search_tracks, search_artists,
search_albums, get_track_details, get_album, get_album_tracks, get_artist,
get_artist_albums, get_track_features. T2TunesStreamInfo dataclass captures
the hex decryption key returned by the proxy (CENC/AES-128). Handles the
"stremeable" API typo. 0.5 s rate-limit guard + api_call_tracker.

core/amazon_download_client.py — DownloadSourcePlugin backed by the above
client. Codec waterfall: FLAC → Opus → EAC3. Downloads the encrypted MP4
container, decrypts with ffmpeg -decryption_key, yields the native audio
file (.flac / .opus / .eac3). Not yet wired into the app source registry —
validated in isolation only; see tests/tools/.

tools/t2tunes_probe.py + tools/t2tunes_media_plan.py — standalone CLI tools
used for live API exploration during development.

tests/tools/test_amazon_client.py — 72 unit tests (all mocked).
tests/tools/test_amazon_download_client.py — 52 unit tests (all mocked).
124 tests pass.
2026-05-16 07:46:41 -07:00
BoulderBadgeDad
d94a6e4f7a
Merge pull request #614 from Nezreka/feature/personalized-playlist-pipeline
Feature/personalized playlist pipeline
2026-05-15 22:09:38 -07:00
Broque Thomas
115d7ed9c5 Preserve personalized playlist metadata for wishlist 2026-05-15 21:50:32 -07:00
Broque Thomas
d861a40277 Personalized pipeline: refresh snapshot on first-run too
Reproduced: selecting Fresh Tape (or any kind never generated before)
and running the pipeline silently skipped — UI showed
"No tracks in Fresh Tape — skipping sync" with no clue why.

Root cause: ensure_playlist auto-creates the playlist row on first
access with `track_count=0` and `last_generated_at=NULL`, but
`is_stale=0` by default (the column default — fresh rows aren't
"stale", they're "never generated"). Pipeline only refreshed when
`is_stale=True` OR `refresh_first=True`, so first-run rows fell
through both branches → read the empty snapshot → skip.

Fix: pipeline now also refreshes when `existing.last_generated_at is
None`. Same control flow, one extra condition:

    if refresh_first OR is_stale OR last_generated_at is None:
        refresh
    else:
        read existing snapshot

This is the right signal: "has the generator ever run for this row"
is exactly what `last_generated_at` tracks (the column is set in
`_persist_snapshot` after every successful refresh).

Stubs in test_handlers_personalized_pipeline.py updated to expose
`last_generated_at` on their SimpleNamespace returns so the new
attribute read doesn't AttributeError. Fresh stubs get a non-None
timestamp so they're treated as already-generated; the new test
`test_never_generated_snapshot_triggers_first_refresh` pins the
first-run-forces-refresh behavior with `last_generated_at=None`.
2026-05-15 21:13:16 -07:00
Broque Thomas
08725094db get_current_profile_id: catch RuntimeError so background callers don't crash
Reproduced on the personalized playlist pipeline: selecting Fresh Tape
(or any kind) and running the automation surfaced
"Working outside of application context" in the UI.

Root cause: `get_current_profile_id` reads Flask's `g.profile_id` and
only catches `AttributeError`. Outside a request — automation engine,
sync threads, watchlist scanner — `g` raises `RuntimeError` instead,
so the except misses and the handler dies.

Mirrored playlist pipeline never hit this because it hardcodes
profile_id=1 in its sync call. The personalized pipeline calls
`deps.get_current_profile_id()` from a background thread, which is
what tripped the bug. Fresh Tape's generator also resolves the
profile via the same function — same path, same crash.

Fix: broaden the except to `(AttributeError, RuntimeError)` in all
three copies of the helper (`web_server.py`, `core/artists/map.py`,
`core/discovery/hero.py`). All three now safely degrade to profile_id=1
(admin profile) when called outside a request context — matches the
existing intent that single-admin installs Just Work.

No test changes — the existing pipeline tests stub the helper, so
they never exercised the bug. The fix is in the layer above the
stubs.
2026-05-15 21:06:17 -07:00
Broque Thomas
877d0e7d81 Personalized pipeline: auto-refresh stale snapshots after watchlist scan
Snapshots now track when their source data changes. Watchlist scan
emits stale flags on the playlists whose underlying pool just got
refreshed; the next pipeline run sees the flag and regenerates the
snapshot before syncing, so the server playlist never lags the source.

Schema:
- new `is_stale INTEGER NOT NULL DEFAULT 0` column on
  `personalized_playlists`, plus an idempotent ADD COLUMN migration
  in `ensure_personalized_schema` for installs created before this PR.
- `PlaylistRecord.is_stale: bool = False` exposed on the dataclass so
  callers can branch on freshness without re-querying.

Manager:
- new `mark_kinds_stale(kinds, profile_id=None)` flips the flag in
  bulk for a list of kinds (used by upstream data refreshers).
- `_persist_snapshot` clears `is_stale = 0` on successful refresh.
- SELECT statements + `_row_to_record` updated to read the column
  (with tuple-form length guard for safety).

Pipeline:
- `_build_payloads_for_kinds` now branches: refresh_first=True OR
  `existing.is_stale` -> refresh_playlist, else read existing
  snapshot. So the auto-refresh kicks in without needing the user to
  toggle the refresh-each-run option.

Watchlist scanner emits stale flags at three sites:
- after `update_discovery_pool_timestamp` -> marks pool-fed kinds
  stale: hidden_gems, discovery_shuffle, popular_picks, time_machine,
  genre_playlist, daily_mix.
- after release_radar `save_curated_playlist` -> marks `fresh_tape`.
- after discovery_weekly `save_curated_playlist` -> marks `archives`.

All three calls go through a module-level `_mark_personalized_kinds_stale`
helper that builds a PersonalizedPlaylistManager with `deps=None` (only
DB access is needed for the flag update — no generator dispatch). Each
call is wrapped in try/except so a flag failure can never abort the
scan itself.

Tests:
- new `TestStaleFlag` class in `test_personalized_manager.py` (6
  tests): default-false, single-kind flip, multi-kind, profile
  scoping, refresh-clears, empty-list noop.
- two new pipeline tests pin the auto-refresh dispatch:
  `test_stale_snapshot_auto_refreshes_even_without_refresh_first`
  and `test_non_stale_snapshot_skips_refresh`.
- existing stub-manager `SimpleNamespace` returns gained
  `is_stale=False` so the new attribute read doesn't AttributeError.

Full suite: 3391 pass.

User-facing WHATS_NEW entry added under 2.5.2 (above the prior
pipeline auto-sync entry) describing the auto-refresh behavior.
2026-05-15 20:53:03 -07:00
Broque Thomas
66390e685a Personalized pipeline picker: full-width column layout + label overrides
The picker was rendering as a narrow centered column overlapping the
description text because:
1. The outer `.config-row` defaults to flex-direction:row with the
   label on the left and the input on the right at fixed width — works
   for a select / textbox, breaks for a tall scrolling multi-select.
2. Inner `<label>` rows in the picker were inheriting
   `.placed-block-config label` (uppercase / 50px min-width /
   letter-spacing 0.5px) so each row turned into a 50-pixel-wide
   uppercase chip.

Fixes:
- Outer wrapper switched to `flex-direction:column;align-items:stretch`
  + `width:100%;box-sizing:border-box` on the picker div.
- Inner row + section-header inline styles override font-size,
  text-transform, letter-spacing, and min-width so the picker rows
  render at normal text size with proper full-width alignment.

Variant rows indent under their kind header at 20px so the visual
grouping is obvious.
2026-05-15 19:48:06 -07:00
Broque Thomas
e1f0810df5 Personalized pipeline: UI multi-select picker for kinds + variants
The action was registered + the block declared, but the automation
builder's per-action config renderer didn't have a case for
`personalized_pipeline` so users only saw the bare card with the
generic delay-minutes input — no way to select which playlists to
sync. This commit adds the multi-select picker.

Backend:
- `core/personalized/api.list_kinds(manager=...)` now optionally
  takes a manager and includes the resolved variant list per kind
  (calls each spec's variant_resolver(deps) when present). Singleton
  kinds get an empty `variants` list. Variant-bearing kinds
  (time_machine / genre_playlist / daily_mix / seasonal_mix) get
  their full enumerated set.
- `web_server.py` `/api/personalized/kinds` route now passes a built
  manager so the variants list lands in the response.

Frontend:
- `webui/static/stats-automations.js` `_renderBlockConfigFields`
  gains a `personalized_pipeline` branch that renders a scrollable
  multi-select picker:
  - Singletons (Hidden Gems, Discovery Shuffle, Popular Picks,
    Fresh Tape, The Archives) = one checkbox row per kind
  - Variant kinds = a section header + one checkbox row per variant
    (e.g. Time Machine: 1960s/1970s/.../2020s; Seasonal: halloween/
    christmas/valentines/summer/spring/autumn)
  - Pre-checks rows that match the existing `kinds` config on edit
- New `_autoLoadPersonalizedKinds(slotKey)` fetches `/api/personalized/kinds`
  (cached after first load), renders the picker DOM, and pre-checks
  saved selections via `data-kind` / `data-variant` attributes on
  the checkboxes.
- `_renderBuilderCanvas` calls the loader for any `cfg-*-kinds-picker`
  it finds in the freshly-rendered slots.
- The save-time `_collectActionConfig` walks the picker's checked
  inputs (matched by `data-kind` attribute) and emits
  `{kinds: [{kind, variant?}, ...], refresh_first, skip_wishlist}`
  in the same shape the handler expects.

Tests:
- `tests/automation/test_automation_blocks.py::_FIELD_TYPES` adds
  'personalized_playlist_select' so the block-shape regression test
  accepts the new field type. (Test was failing because it whitelists
  every field type used across all blocks.)
- 189 automation + personalized API tests pass; full suite intact.
2026-05-15 19:33:34 -07:00
Broque Thomas
cc44254bf9 Personalized playlist pipeline: auto-sync discover-page playlists
Follow-up to the personalized-playlists standardization PR. New
`personalized_pipeline` automation action syncs selected discover-
page playlists (Hidden Gems / Discovery Shuffle / Time Machine /
Genre / Daily Mix / Fresh Tape / The Archives / Seasonal Mix) to
the active media server + queues missing tracks for download.

Same pattern as the existing mirrored `playlist_pipeline` but two
phases instead of four — no REFRESH (no external source to re-pull)
and no DISCOVER (manager-backed snapshots are already metadata-
matched). Pipeline shape:

    SNAPSHOT → SYNC → WISHLIST

Where SNAPSHOT either reads the persisted track list from
`PersonalizedPlaylistManager` (default) or refreshes it first when
`refresh_first=true` (cron use case: regenerate Hidden Gems nightly
and sync the fresh set).

Shared helper extraction:

PHASE 3 (SYNC loop) + PHASE 4 (WISHLIST tail) lifted out of mirrored
`playlist_pipeline` into `core/automation/handlers/_pipeline_shared.py`
as `run_sync_and_wishlist(deps, automation_id, playlists, sync_one_fn,
sync_id_for_fn, ...)`. Both pipelines call it. Mirrored injects
`auto_sync_playlist` as the per-playlist sync function; personalized
injects a thin wrapper that launches `_run_sync_task` directly with
a pre-built tracks_json. Same sync-state polling / progress emission
/ status counting / wishlist trigger logic — 0 duplication.

Files added:
- core/automation/handlers/_pipeline_shared.py
- core/automation/handlers/personalized_pipeline.py
- tests/automation/test_handlers_personalized_pipeline.py

Files changed:
- core/automation/handlers/playlist_pipeline.py: PHASE 3+4 replaced
  with shared helper call (~100 lines deleted, 1 helper invocation
  added; behavior identical).
- core/automation/deps.py: new `build_personalized_manager` field
  (lazy builder so the pipeline gets a fresh PersonalizedPlaylistManager
  per run).
- core/automation/handlers/__init__.py + registration.py: register
  `personalized_pipeline` action with the shared `pipeline_running`
  guard so it can't overlap mirrored.
- core/automation/blocks.py: new `personalized_pipeline` block
  declaration with config_fields (kinds multi-select, refresh_first,
  skip_wishlist).
- web_server.py: thread `_build_personalized_manager` into
  AutomationDeps construction.
- All 5 automation test fixtures: `_build_deps` adds
  `build_personalized_manager=lambda: None` stub.
- tests/automation/test_handler_registration.py:
  EXPECTED_ACTION_NAMES + EXPECTED_GUARDED_ACTIONS gain
  `personalized_pipeline`.

Trigger schema:

    {
      "_automation_id": "...",
      "kinds": [
        {"kind": "hidden_gems"},
        {"kind": "time_machine", "variant": "1980s"},
        {"kind": "seasonal_mix", "variant": "halloween"}
      ],
      "refresh_first": false,
      "skip_wishlist": false
    }

Tests (14 new, 178 automation total):
- _track_to_sync_shape: basic shape, source ID fallback chain,
  no-id returns empty string
- empty config / non-list kinds / empty kinds list all return
  error + clear pipeline_running flag
- _build_payloads_for_kinds: skips invalid entries, skips kinds
  with no tracks, refresh_first vs ensure dispatch, payload shape
  + sync_id format, manager exception swallowed continues
- _sync_personalized_playlist: launches background thread + returns
  status='started'
- happy path: stubbed sync_states drives helper to completion, flag
  cleaned up

Full suite: 3383 passed.

Note: the trigger UI block declares config_fields but the frontend
doesn't yet render the `personalized_playlist_select` multi-select
type — usable today via API; polished UI ships in a follow-up
frontend PR.
2026-05-15 18:41:08 -07:00
BoulderBadgeDad
fe6f196cac
Merge pull request #613 from Nezreka/feature/personalized-playlists-overhaul
Feature/personalized playlists overhaul
2026-05-15 18:03:01 -07:00
Broque Thomas
3f965f48cd Personalized playlists: ruff B905 — explicit strict= on zip()
CI ruff check failed on the seasonal_mix tuple-row coercion path
where a `zip(columns, row)` call lacked an explicit `strict=`.

Set `strict=False` to preserve the original intent (tolerant if
the row shape ever drifts from the column tuple). The SELECT
always returns 8 columns so the lengths match in practice; using
strict=False just avoids a future raise if a generator drift
changes that.

Live happy path stays unchanged: rows from sqlite3.Row hit the
`hasattr(r, 'keys')` branch above and never reach the zip line.
The zip branch only runs for plain-tuple rows in tests.
2026-05-15 17:57:44 -07:00
Broque Thomas
9cf1fe492b Personalized playlists (5/5): WHATS_NEW entry
User-facing summary of the standardization work — all 8 personalized
discover-page playlists unified behind one storage layer, manager,
and REST surface. Prerequisite for the playlist pipeline integration
landing in the next PR.
2026-05-15 17:26:04 -07:00
Broque Thomas
cc0828e9ff Personalized playlists (4/N): staleness post-filter (exclude_recent_days)
Adds the first quality feature on top of the manager: when
`config.exclude_recent_days > 0`, the manager drops any track from
the generator's output whose primary id was served by this kind
for this profile in the last N days.

Lives at the manager layer, not in each generator, so:
- generators stay focused on selection logic
- staleness behavior stays uniform across every kind
- enabling/disabling per playlist is just a config patch

Implementation:
- New `PersonalizedPlaylistManager._apply_quality_filters` runs after
  generator returns, before `_persist_snapshot`.
- Reads recent ids via existing `recent_track_ids` accessor.
- Tracks without a primary id pass through unchanged (nothing to
  dedupe on -- happens for sourceless tracks during edge cases).
- Returns a new list (never mutates input).

Default `exclude_recent_days = 0` preserves pre-overhaul behavior.
Per-playlist override via `PUT /api/personalized/playlist/<kind>/config`
with `{"exclude_recent_days": N}`. Recommended values:
- Discovery Shuffle: 1-3 days (high churn desired)
- Hidden Gems: 7-14 days (avoid same gems weekly)
- Time Machine / Genre: 30+ days (slow rotation OK, stable view preferred)

4 new boundary tests:
- Zero days = no filter (default behavior preserved)
- Positive days drops tracks served in window
- Filter preserves new tracks alongside dropped ones
- Tracks without primary id pass through unchanged

3369 tests pass total.

Note: listening-history cross-ref + seeded shuffle are deferred to
a future PR. Each requires deeper integration -- listening history
needs a play-events table the discovery pool can query against;
seeded shuffle needs the legacy generators to accept a seed param
without breaking their existing diversity / popularity logic.
2026-05-15 17:18:32 -07:00
Broque Thomas
9f383acbfb Personalized playlists (3/N): standardized API endpoints
Wraps the manager + generator dispatch behind one HTTP surface so
the UI can drop the patchwork `/api/discover/personalized/*` calls
in favor of a single REST shape. Legacy endpoints stay alive for
backward compat during the UI migration window.

New endpoints:
- GET    /api/personalized/kinds                                — list every registered kind + metadata
- GET    /api/personalized/playlists                            — list every persisted playlist for the active profile
- GET    /api/personalized/playlist/<kind>                       — fetch singleton + tracks
- GET    /api/personalized/playlist/<kind>/<variant>             — fetch variant + tracks
- POST   /api/personalized/playlist/<kind>/refresh               — regenerate singleton
- POST   /api/personalized/playlist/<kind>/<variant>/refresh     — regenerate variant
- PUT    /api/personalized/playlist/<kind>/config                — patch singleton config
- PUT    /api/personalized/playlist/<kind>/<variant>/config      — patch variant config

Per-call manager construction wires the deps each generator needs:
- database (MusicDatabase singleton)
- service (PersonalizedPlaylistsService for legacy generator calls)
- seasonal_service (SeasonalDiscoveryService for seasonal_mix)
- get_current_profile_id (active profile accessor)
- get_active_discovery_source (source dispatcher)

API handlers themselves live as pure functions in
`core/personalized/api.py` so they're testable without Flask. The
Flask layer in `web_server.py` is a thin parse-body / call-handler /
jsonify wrapper.

11 new boundary tests (122 personalized total):
- list_kinds enumerates registry, exposes default config + tags
- list_playlists returns empty list when none exist, serializes
  PlaylistRecord shape correctly
- get_playlist_with_tracks auto-creates on first access, returns
  persisted tracks, raises ValueError on unknown kind
- refresh_playlist runs generator and returns track snapshot,
  forwards config_overrides to the generator
- update_config patches stored config

3365 tests pass total. Manager construction triggers generator
registration via `from core.personalized import generators` import
side-effect.
2026-05-15 17:15:25 -07:00
Broque Thomas
53284ee7c8 Personalized playlists (2/N): all 8 generators wired through manager
Adds the per-kind generator modules and registers them with the
PlaylistKindRegistry so the manager's `refresh_playlist` can dispatch
to any of them.

Generators (each in its own module under
`core/personalized/generators/`):

Singletons (variant=''):
- hidden_gems       -> wraps service.get_hidden_gems
- discovery_shuffle -> wraps service.get_discovery_shuffle
- popular_picks    -> wraps service.get_popular_picks

Variant-bearing kinds:
- time_machine      -> variant = decade label ('1980s', '1990s', ...).
                       Variant resolver returns 7 standard decades.
                       Generator parses '1980s' -> 1980 + delegates
                       to service.get_decade_playlist.
- genre_playlist    -> variant = URL-safe genre key
                       ('electronic_dance', 'hip_hop_rap', ...).
                       Resolver normalizes parent-genre keys from
                       service.GENRE_MAPPING; free-form keywords
                       pass through to service.get_genre_playlist.
- daily_mix         -> variant = top-genre rank ('1' / '2' / '3' / '4').
                       Generator looks up user's Nth-ranked library
                       genre and returns discovery picks within it.
                       Library half (was a stub returning []) is
                       intentionally dropped: tracks table has no
                       source IDs, so library rows can't sync. Fixed
                       the stub to return [] cleanly without the
                       misleading log warning.
- fresh_tape        -> Spotify Release Radar. Reads curated track
                       IDs from discovery_curated_playlists (tries
                       'release_radar_<source>' first, falls back to
                       'release_radar') and hydrates against the
                       discovery pool.
- archives          -> Spotify Discover Weekly. Same hydration path
                       as fresh_tape but uses 'discovery_weekly'.
- seasonal_mix      -> variant = season key ('halloween' / 'christmas'
                       / 'valentines' / 'summer' / 'spring' / 'autumn').
                       Reads curated IDs via SeasonalDiscoveryService
                       then hydrates from seasonal_tracks (which
                       carries full track_data_json).

Each module:
- Defines `generate(deps, variant, config) -> List[Track]`.
- Defines `SPEC = PlaylistKindSpec(...)` and registers it on import
  (idempotent — re-import safe via `if registry.get(...) is None`).
- For variant-bearing kinds, also defines `variant_resolver(deps)`.

Shared helpers in `_common.py`:
- `get_service(deps)` pulls the legacy
  `PersonalizedPlaylistsService` instance (deps.service or
  deps['service']).
- `coerce_tracks(rows)` runs each dict through `Track.from_dict`,
  tolerates None / non-list inputs.

Tests (50 new, total 85 across personalized subsystem):
- Singletons: registration + display name + dispatch + limit
  forwarding + empty/None tolerance + missing-deps error +
  dict-form deps acceptance (16 tests).
- Variants: variant_resolver listing + label parsing + invalid
  variant errors + parent-key normalization + free-form passthrough
  (13 tests).
- Curated/hybrid: daily_mix rank-to-genre resolution + rank-out-of-
  range empty + invalid-variant error; fresh_tape & archives
  hydration order + missing-id skip + source-specific-then-fallback
  key dispatch + limit + missing-database-dep error; seasonal_mix
  curated-id hydration order + missing-id skip + JSON round-trip +
  empty-curated empty + limit + missing-service error (21 tests).

3304+ tests pass. No regression on existing 62 personalized tests.
2026-05-15 17:02:08 -07:00
Broque Thomas
79224ed294 Personalized playlists (1/N): unified storage + manager foundation
Begins the standardization of the personalized-playlist subsystem.
Pre-existing state was a patchwork: Group A (Fresh Tape / Archives /
Seasonal Mix) lived in `discovery_curated_playlists` and
`curated_seasonal_playlists` with inconsistent shapes; Group B
(Hidden Gems / Discovery Shuffle / Time Machine / Popular Picks /
Genre / Daily Mixes) was computed on-demand by
`PersonalizedPlaylistsService` with no persistence -- every call
reran the generator with `ORDER BY RANDOM()` so results rotated.

Post-overhaul (this PR) every personalized playlist lands in one
unified storage layer with stable identity, persistent track lists,
explicit refresh, and per-playlist user-tweakable config.

Foundation in this commit (no behavior change yet):

- `database/personalized_schema.py`: 3 tables created idempotently
  at app startup (wired into `MusicDatabase._initialize_database`).
  - `personalized_playlists`: one row per (profile, kind, variant)
    with config_json, track_count, last_generated_at,
    last_synced_at, last_generation_source, last_generation_error.
    Variant '' (empty string) for singletons; non-empty for
    time_machine / seasonal_mix / genre_playlist / daily_mix.
  - `personalized_playlist_tracks`: current snapshot per playlist.
    Atomically replaced on refresh.
  - `personalized_track_history`: append-only log powering the
    `exclude_recent_days` config knob.

- `core/personalized/types.py`: `Track`, `PlaylistConfig`,
  `PlaylistRecord` dataclasses. `PlaylistConfig.merged()` for
  partial-update PATCH semantics; `Track.from_dict()` accepts the
  legacy generator output shape unchanged.

- `core/personalized/specs.py`: `PlaylistKindSpec` (kind,
  name_template, default_config, generator, variant_resolver) and a
  module-level registry. Generators register at import time;
  manager dispatches by kind.

- `core/personalized/manager.py`: `PersonalizedPlaylistManager` --
  the only thing that touches the new tables. Owns:
  - ensure_playlist (auto-create row from kind defaults)
  - get_playlist / list_playlists
  - refresh_playlist (atomic snapshot replace; generator exception
    preserves previous good snapshot + records error on row)
  - get_playlist_tracks
  - update_config (deep-merge with stored config, including extra dict)
  - recent_track_ids (staleness lookup for generators)

35 boundary tests in `tests/test_personalized_manager.py` pin every
shape: config round-trip / merge semantics / extra deep-merge /
defaults; Track.from_dict tolerance + primary_id fallback chain;
registry dedup / display_name with+without variant; manager
ensure_playlist auto-create + idempotency, variant separation,
required-variant enforcement, unknown-kind error; refresh persists
+ replaces atomically + survives generator exception with previous
snapshot intact + records source from first track + round-trips
nested track_data_json; update_config patch semantics; list_playlists
profile scoping; staleness history scoped to (profile, kind, days).

3304 tests pass total. Generators ship in subsequent commits on this
branch -- each kind migrated one at a time with its own per-kind
boundary tests.
2026-05-15 16:19:01 -07:00
Broque Thomas
19a18ba992 Dashboard activity feed: stop showing 'NaNmo ago'
Recent activity items on the dashboard all rendered 'NaNmo ago'
because the formatter parsed `activity.time` (a human label like
'Now' / 'Just now') with `new Date(...)` -> Invalid Date -> NaN
arithmetic -> 'NaNmo ago'.

Backend (`core/runtime_state.add_activity_item`) has always emitted
`activity.timestamp` (Unix epoch seconds) alongside the label.
Frontend now uses the epoch for relative-time formatting via a new
local `_activityTimeAgo` helper:
- typeof timestamp === 'number' -> diff against Date.now() in ms
- < 60s -> 'Just now'
- < 60m -> 'Nm ago'
- < 24h -> 'Nh ago'
- < 30d -> 'Nd ago'
- otherwise 'Nmo ago'
- falls back to the literal `activity.time` label only when no
  timestamp is present (legacy items / future shapes)

Both call sites in api-monitor.js (initial render + timestamp-only
refresh path) updated to the new helper.
2026-05-15 14:02:00 -07:00
Broque Thomas
81af852f61 reenable beatport 2026-05-15 13:48:24 -07:00
BoulderBadgeDad
6a8870f070
Merge pull request #611 from Nezreka/refactor/extract-automation-handlers
Refactor/extract automation handlers
2026-05-15 13:31:12 -07:00
Broque Thomas
d3768610d7 Extract automation handlers (kettui-bar): engine-boundary tests
Per-handler boundary tests pin each handler's body in isolation.
Adding engine-boundary tests that pin the REGISTRATION layer:
- every expected action name registered, no drops, no extras
- guarded actions register a guard, unguarded ones don't
- every registered handler is callable
- every guard returns a bool
- all four progress callbacks registered in the right slots
- progress_init / progress_finish / record_history / on_library_scan_completed
  are invocable through the engine's stored callable shape (not just
  the bare extracted function)
- finish callback respects _manages_own_progress flag at the engine
  boundary too
- library_scan_completed wiring registers a callback on the scan
  manager and that callback fires engine.emit when invoked
- every handler returns a `{'status': ...}` dict on a minimal config
  trigger -- proves no handler raises into the engine, even when its
  guard / short-circuit / error path is the one taken

Uses a minimal _RecordingEngine that captures registrations + a
_RecordingScanMgr that captures completion callbacks. No real
AutomationEngine, no real Flask app, no real DB. The kettui standard
for refactor PRs: don't ship "behavior preserved" claim that's only
validated at the function boundary -- exercise the engine seam too.

EXPECTED_ACTION_NAMES + EXPECTED_GUARDED_ACTIONS frozen sets at the
top: any future drift (rename / drop / add a handler / change which
ones are guarded) fails this test immediately so refactor PRs can't
quietly mutate the registration shape.

13 new tests, 164 automation tests pass total.
2026-05-15 12:35:45 -07:00
Broque Thomas
e140da117a Extract automation handlers (4/3 — finish): progress callbacks + scan-completion emitter
Cleans up the four remaining inline callbacks at the bottom of
`web_server._register_automation_handlers` so the function is now
purely deps-construction + register_all + a logger.info line.

Lifted:
- `_progress_init`, `_progress_finish`, `_record_automation_history`,
  and `_on_library_scan_completed` -> core/automation/handlers/progress_callbacks.py

Each is a top-level function that takes deps as a parameter; the
engine sees thin lambdas through `register_progress_callbacks` /
`register_library_scan_completed_emitter` (called from `register_all`).

Two new deps fields:
- `init_automation_progress` (delegates into the live progress tracker)
- `record_progress_history` (delegates into _auto_progress.record_history)

12 new boundary tests in tests/automation/test_progress_callbacks.py
pin every shape:
- progress_init forwards to init_automation_progress
- progress_finish skips when handler manages its own progress
  (prevents double-emit of finished status)
- progress_finish: completed -> finished/Complete/success;
  error -> error/Error/error; msg falls through error -> reason ->
  status -> 'done'
- record_history threads the live db into the recorder
- on_library_scan_completed: no engine = noop, server type taken
  from web_scan_manager._current_server_type, defaults to 'unknown'
- register_library_scan_completed_emitter: no scan manager = noop,
  registered callback emits the right event when invoked

3256 tests pass, no regression.

Final state of `_register_automation_handlers`:
- Was: 1530 lines, 21 nested closures + 4 progress callbacks
- Now: ~50 lines, builds AutomationDeps and calls register_all

web_server.py: 34,220 -> 34,187 lines (-33 net, -1,406 across the
whole branch).
2026-05-15 11:59:32 -07:00
Broque Thomas
017553193f Extract automation handlers (3/3): maintenance + misc, finishing the lift
Final commit of the automation-handler refactor. With this commit
every closure that used to live in
`web_server._register_automation_handlers` is now a top-level
function in `core/automation/handlers/`.

Handlers extracted in this commit:

- start_database_update + deep_scan_library
    -> core/automation/handlers/database_update.py
    Both share the db_update_state monitoring pattern (poll until
    status flips, stall detection emits warning at 10 min, 2-hour
    outer timeout). Lifted into a shared `_run_with_progress` helper
    inside the module so the per-handler bodies stay tiny.

- run_duplicate_cleaner -> core/automation/handlers/duplicate_cleaner.py
- start_quality_scan    -> core/automation/handlers/quality_scanner.py

- clear_quarantine, cleanup_wishlist, update_discovery_pool,
  backup_database, refresh_beatport_cache
    -> core/automation/handlers/maintenance.py
    Grouped because each body is short (~20-50 lines) and they share
    no state — splitting into per-handler files would just add import
    noise.

- clean_search_history, clean_completed_downloads, full_cleanup
    -> core/automation/handlers/download_cleanup.py
    Grouped because all three reach the download orchestrator,
    tasks_lock, and download_batches/download_tasks accessors. The
    full_cleanup multi-step orchestration shares phase-detection
    logic with clean_completed_downloads.

- run_script         -> core/automation/handlers/run_script.py
- search_and_download -> core/automation/handlers/search_and_download.py

`AutomationDeps` grew with the new dependency surface:
- get_db_update_state + db_update_lock + db_update_executor +
  run_db_update_task + run_deep_scan_task
- get_duplicate_cleaner_state + duplicate_cleaner_lock +
  duplicate_cleaner_executor + run_duplicate_cleaner
- get_quality_scanner_state + quality_scanner_lock +
  quality_scanner_executor + run_quality_scanner
- download_orchestrator + run_async + tasks_lock +
  get_download_batches + get_download_tasks +
  sweep_empty_download_directories + get_staging_path
- docker_resolve_path + get_current_profile_id +
  get_watchlist_scanner + get_app + get_beatport_data_cache
- set_db_update_automation_id (writes the legacy global so the live
  DB-update progress callbacks still living in web_server.py keep
  emitting against the active automation card)

`web_server._register_automation_handlers` is now ~50 lines: build
deps once, call register_all. The 667-line block of remaining
closure definitions and engine register calls is gone.

The final orphan was the `_db_update_automation_id` module global —
the DB-update progress callbacks at line ~14080 still read it
directly, so the extracted database_update handler propagates the
automation id through `deps.set_db_update_automation_id` (a closure
in web_server that writes the global). When the legacy callbacks
get extracted in a future PR the setter goes away.

Tests:
- tests/automation/test_handlers_maintenance.py adds 21 boundary
  tests covering every newly-extracted handler shape: guard
  short-circuits (already-running returns skipped), deps wiring
  (set_db_update_automation_id called with the right id),
  exception swallow contract, status returns, path-traversal
  blocked in run_script, source-mode skip in clean_search_history,
  active-batch skip in clean_completed_downloads, etc.
- 3244 tests pass (was 3223 — 21 new), no regression.

web_server.py: 35,593 -> 34,220 lines (-1,373 net across 3 commits).
Issue #1 from the extraction punch list is now COMPLETE.
2026-05-15 11:24:35 -07:00
Broque Thomas
cde237c7e7 Extract automation handlers (2/N): playlist lifecycle group
Continues the lift from `web_server._register_automation_handlers`.
This commit extracts the four playlist-lifecycle closures:

- `refresh_mirrored`   -> core/automation/handlers/refresh_mirrored.py
- `sync_playlist`      -> core/automation/handlers/sync_playlist.py
- `discover_playlist`  -> core/automation/handlers/discover_playlist.py
- `playlist_pipeline`  -> core/automation/handlers/playlist_pipeline.py

The pipeline composes refresh + sync + discover, so all four ship
together. The pipeline imports the other three handler modules
directly (cross-handler call) instead of going through the engine,
preserving the "single trigger from the user's perspective" UX.

`AutomationDeps` grew to cover the new dependency surface:
- run_playlist_discovery_worker, run_sync_task, load_sync_status_file
  (pre-existing background-task entry points)
- get_deezer_client, parse_youtube_playlist (per-source clients)
- get_sync_states (live mutable accessor for the sync UI's state dict)

`web_server._register_automation_handlers` now wires those plus the
existing infrastructure into a single `AutomationDeps` and calls
`register_all`. The 669-line block of closure definitions and engine
register calls (lines 959-1627 pre-edit) is gone -- the file shed
743 lines net on this commit.

`tests/automation/test_handlers_playlist.py` adds 17 new boundary
tests:
- discover_playlist: no_id error, specific_id starts worker, all=True
  enumerates, no playlists in db
- refresh_mirrored: error path, source filter (file/beatport excluded),
  Spotify happy path with auto-discovered marker, per-playlist
  exception captured into errors counter
- sync_playlist: no_id, not_found, no_tracks, no-discovered-tracks
  skip, discovered-track happy path, unchanged-since-last-sync skip
- playlist_pipeline: no_playlist clears running flag, no-refreshable
  clears running flag, exception clears running flag

3223 tests pass. web_server.py: 35,593 -> 34,850 lines (743 removed).
2026-05-15 10:47:46 -07:00
Broque Thomas
ea7d5c65bb Extract automation handlers (1/N): infrastructure + 3 simple handlers
Begins the lift of `web_server._register_automation_handlers` (1530
lines, 20 nested closures) into `core/automation/handlers/`. Each
extracted handler is a top-level function that accepts
`(config, deps)` instead of reaching for module-level globals --
makes them unit-testable in isolation.

Infrastructure:
- `core/automation/deps.py`: `AutomationDeps` (dependency-injection
  bundle of clients + callables) and `AutomationState` (mutable flags
  shared across handler invocations, with thread-safe accessors).
- `core/automation/handlers/__init__.py` + `registration.py`: one-stop
  `register_all(deps)` that wires every extracted handler to the
  engine.

First batch of handlers extracted:
- `process_wishlist` -> `core/automation/handlers/process_wishlist.py`
- `scan_watchlist`   -> `core/automation/handlers/scan_watchlist.py`
- `scan_library`     -> `core/automation/handlers/scan_library.py`

`web_server._register_automation_handlers` now builds the deps once
and calls `register_all(deps)` for the extracted batch. Remaining
17 closures still live below; subsequent commits in this branch
finish the lift.

14 boundary tests in `tests/automation/test_handlers_simple.py` pin
every shape: success path, exception swallow contract, fresh-vs-stale
state detection (scan_watchlist's id() trick), guard short-circuits,
state cleanup on exceptions, AutomationState concurrent-safe accessors.
All 101 automation tests pass; no regression.
2026-05-15 10:25:41 -07:00
Broque Thomas
d9529fc801 Token leak round 2: artist endpoint + playlist sync + URL-encoded redaction
The first token-leak fix scrubbed the artwork URL fixer's own log
calls. This catches three more sites that ALSO leaked tokens, plus
one upstream gap that let URL-encoded tokens slip through the
redactor.

Three sites in `web_server.py` (artist endpoint at line 8765-8773):

- "Artist image before fix: '...'" -- logged the raw image_url with
  the auth token in plain form.
- "Artist image after fix: '...'" -- logged the URL-encoded form
  after it had been wrapped in the image proxy
  (`/api/image-proxy?url=<percent-encoded-token>`).
- "Final artist data being sent: {...}" -- dumped the entire
  artist_info dict on every render, including the image_url field.

All three were dev-time debug noise. Removed entirely. The "No
artist image URL found" warning at line 8770 stays (no URL, just
the artist name).

One site in `core/discovery/sync.py:402`:

- "[PLAYLIST IMAGE] image_url=..." -- logged the playlist poster URL
  during sync. Same auth-token leak risk for Plex / Jellyfin
  playlists. Changed to log only `has_image=True/False`.

Upstream gap in `_redact_url_secrets`:

- The original regex only matched plain query params (`?key=value`).
  When an auth-bearing URL gets wrapped inside another URL's query
  string (our `/api/image-proxy?url=<encoded>` flow) the auth params
  end up percent-encoded -- `%3FX-Plex-Token%3D...` -- and slipped
  through.
- New second pattern catches the URL-encoded form. Both passes run
  on every redact call; idempotent.

Verified manually:
  /api/image-proxy?url=...%3FX-Plex-Token%3DABC...
  -> /api/image-proxy?url=...%3FX-Plex-Token%3D***REDACTED***

6 artwork tests pass.
2026-05-15 09:33:15 -07:00
Broque Thomas
2fe1926074 Stop leaking Plex / Jellyfin / Navidrome tokens into app.log
The artwork URL normalizer was logging the full constructed media-
server URL on every cover-art lookup at INFO level, including the
auth query params (X-Plex-Token / X-Emby-Token / Subsonic t+s+p).
Those lines pile up in app.log on disk -- anyone with read access to
the log file gains full read access to the user's media server.

Also dropped the noisy per-call "Plex/Jellyfin/Navidrome config -
base_url: ..., token: ..." INFO lines that fired on every thumbnail.
Even the truncated `token[:10]` form is enough partial-known-plaintext
to be uncomfortable to leak.

- New `_redact_url_secrets` helper masks the values of X-Plex-Token,
  X-Emby-Token, api_key, apikey, Subsonic t / s / p, generic token /
  password query params. Regex anchored on `?` or `&` boundary so
  short keys like `t` don't false-match inside `format=Jpg`.
- "Fixed URL: ..." log calls moved from INFO to DEBUG so they don't
  persist by default, and the URL passed in is run through the
  redactor first.
- Per-call "Plex config - ..." / "Jellyfin config - ..." /
  "Navidrome config - ..." INFO lines removed entirely. Config
  inspection has dedicated UI; per-thumbnail spam belongs to no one.
- Error-path logging (line 149) also routed through the redactor in
  case the failing URL had auth params attached.

Users with existing app.log files containing the leaked tokens
should rotate / wipe the log. Plex tokens can be regenerated by
signing out of all devices in Plex settings; Jellyfin api_keys can
be revoked from the dashboard; Navidrome users should rotate the
account password.
2026-05-15 09:23:53 -07:00
BoulderBadgeDad
15c41fcc64
Merge pull request #610 from Nezreka/fix/acoustid-quarantine-issues
AcoustID + quarantine modal: three bug fixes (closes #607, closes #608)
2026-05-15 09:07:08 -07:00
Broque Thomas
b42cafa150 AcoustID + quarantine modal: three bug fixes (closes #607, closes #608)
Issue #607 (AfonsoG6) -- two AcoustID problems:

1. Live recordings false-quarantining as "Version mismatch: expected
   '... (Live at Venue)' (live) but file is '...' (original)" because
   MusicBrainz often stores the recording entity with a bare title --
   the venue / live annotation lives on the release entity, not the
   recording. The audio fingerprint correctly identifies the live
   recording, but the title-text comparison flagged it as wrong.

   New pure helper `core/matching/version_mismatch.py:is_acceptable_version_mismatch`
   accepts the mismatch only when:
     - One-sided AND involves 'live': exactly one side is 'live' and
       the other is bare 'original'. Two-sided mismatches stay strict.
     - Fingerprint score >= 0.85 (stricter than the existing 0.80
       minimum -- escape valve only fires when AcoustID is more
       confident than its own threshold).
     - Bare title similarity >= 0.70.
     - Artist similarity >= 0.60.

   Other version markers (instrumental, remix, acoustic, demo, etc)
   stay strict -- those have distinct fingerprints AND MB always
   annotates them in the recording title. The existing
   test_acoustid_version_mismatch.py suite passes unchanged.

2. Audio-mismatch failure message reported "identified as '' by ''
   (artist=100%)" when AcoustID returned multiple recordings -- prior
   code mixed `recordings[0]`'s strings (which can be empty) with
   `best_rec`'s scores. Now uses `matched_title` / `matched_artist`
   consistently in both the high-confidence-skip path and the final
   fail message.

Issue #608 (AfonsoG6) -- quarantine modal:

3. Approve / Delete buttons silently no-op'd when the filename
   contained an apostrophe -- the unescaped quote broke the inline JS
   in the onclick handler. Now wraps the id via
   `escapeHtml(JSON.stringify(id))`, which round-trips quotes /
   backslashes / unicode / newlines safely through the HTML attribute
   to JS string boundary.

4. Bonus UX: quarantine entry expanded view now shows source uploader
   (username) and original soulseek filename when the sidecar carries
   that context -- helps trace which uploader the bad file came from.
   Backend exposes `source_username` + `source_filename` fields from
   `sidecar.context.original_search_result`. Degrades to '' on legacy
   thin sidecars.

Tests:
- 23 new boundary tests in tests/matching/test_version_mismatch.py
  pin every shape: equal versions trivial, one-sided live both
  directions, threshold floors (each just below default -> reject),
  two-sided strict, non-live one-sided strict (covers exact
  test_instrumental_returned_for_vocal_request_fails scenario),
  custom-threshold overrides.
- 4 existing test_acoustid_version_mismatch.py tests pass unchanged.
- 507 AcoustID / matching / imports tests pass.
2026-05-15 08:55:06 -07:00
BoulderBadgeDad
4d682f42b8
Merge pull request #609 from Nezreka/feature/reorganize-tag-source
Reorganize: optional embedded-tag mode (closes #592)
2026-05-15 08:03:35 -07:00
Broque Thomas
b05ba5d498 Reorganize: optional embedded-tag mode (closes #592)
Adds an opt-in alternative metadata source for reorganize. The
existing API path (query Spotify / iTunes / Deezer / Discogs /
Hydrabase for the canonical tracklist) stays the default and is
unchanged. The new tag mode reads each file's embedded tags as the
source of truth instead -- useful for well-enriched libraries where
API drift can produce inconsistent renames, and avoids API calls
entirely.

- New pure helper `core/library/reorganize_tag_source.py` adapts the
  output of `read_embedded_tags` (the same mutagen path the audit-
  trail modal uses) to the `api_album` / `api_track` shapes that
  `_build_post_process_context` already consumes. Handles ID3-style
  "5/12" track + disc shapes, multi-value Artists tags, year
  normalization across 5 date formats, releasetype canonical tokens,
  multi-artist string splits across 9 separators.
- `plan_album_reorganize` accepts `metadata_source: 'api' | 'tags'`
  (default 'api') and `resolve_file_path_fn`. Tag mode branches into
  a new `_plan_from_tags` that reads each track's file and produces
  per-item `api_album` + `api_track` instead of a shared one.
- `_run_post_process_for_track` accepts a per-item `api_album`
  override so each file's own album metadata flows through post-
  process (not a single shared dict).
- `total_discs` in tag mode honors the `totaldiscs` tag and the
  trailing `/N` of an ID3 `discnumber = "1/2"`. Partial-album
  reorganize still routes into the correct `Disc N/` subfolder when
  the tag knows the total even if not all discs are present locally.
- Bare `discnumber = "1"` no longer poisons `total_discs` -- it
  carries no total signal.
- `reorganize_album` surfaces a tag-mode-specific error when no
  files are readable, instead of the API-mode "run enrichment first"
  message which would mislead in tag mode.
- `QueueItem.metadata_source` field, `enqueue` / `enqueue_many`
  pass-through, runner injects `item.metadata_source` into
  `reorganize_album`.
- `web_server.py` endpoints accept `mode` body param. Falls back to
  the `library.reorganize_metadata_source` config setting, then to
  'api'. Strict allowlist (api / tags) -- anything else falls back.
- Frontend: per-album modal + reorganize-all modal both grow a new
  "Metadata Mode" dropdown above the source picker. Tag mode hides
  the source picker (irrelevant). Choice persisted in localStorage.
  Both preview + execute fetches send `mode` in body.

Tests:
- 49 boundary tests on the pure helper pin every shape: ID3 "5/12",
  multi-artist split, year normalization, releasetype validation,
  total_discs precedence, defensive paths.
- 6 planner-level integration tests pin the wiring: tag-mode with
  good tags, partial-disc with totaldiscs tag, file missing,
  some-match-some-fail, defensive resolve_file_path_fn=None,
  API-mode regression guard.
- All 3171 tests pass; 52 existing reorganize tests unchanged.
2026-05-15 07:56:18 -07:00