Commit graph

605 commits

Author SHA1 Message Date
BoulderBadgeDad
2539617853
Merge pull request #306 from kettui/fix/update_discovery_pool_incremental-fixes
Refactor similar artist flows and incremental discovery pool updates in watchlist scanner
2026-04-17 07:25:36 -07:00
Broque Thomas
308773ea7c Add Auto-Import — background staging folder watcher with smart matching
Full auto-import pipeline: background worker watches the staging folder,
identifies music using embedded tags → folder name parsing → AcoustID
fingerprinting, matches files to metadata source tracklists, and
processes high-confidence matches through the existing post-processing
pipeline automatically.

Worker: AutoImportWorker with start/stop/pause/resume, configurable
scan interval (default 60s), confidence threshold (default 90%), and
auto-process toggle. Processes one folder per cycle, alphabetical
order. Disc folder detection, stability checking, content hash dedup.

Confidence gate: 90%+ auto-processes silently, 70-90% queued as
pending review with approve/dismiss actions, <70% flagged for manual
identification. Track matching uses weighted algorithm (title 45%,
artist 15%, track number 30%, album tag 10%).

Database: auto_import_history table tracks every scan result with
folder hash, match data JSON, confidence, status, timestamps.

API: 7 endpoints — status, toggle, settings (GET/POST), results
(filtered/paginated), approve, reject.

UI: Auto tab on Import page with enable toggle, confidence slider,
scan interval selector. Live result cards with album art, confidence
bar (green/yellow/red), status badges, match stats. 5-second polling.
2026-04-17 06:51:08 -07:00
Antti Kettunen
7d18d4ecb2 Clarify comments 2026-04-17 10:02:00 +03:00
Antti Kettunen
eead0c3dac Clarify similar-artist freshness and backfill
Freshness is now age-only, and scan-time backfill runs separately without Spotify-auth gating or retired iTunes compatibility flags.
2026-04-17 09:53:03 +03:00
Antti Kettunen
8382b8e247 Refactor similar artist backfill
Switch similar-artist backfill to the shared provider-priority flow instead of assuming iTunes as the fallback.
Reuse the generic metadata search helpers, keep a compatibility alias for the old helper name, and update the scanner tests to cover the new path.

Add a regression test that verifies backfill walks each available fallback provider and persists the resolved IDs per source.
2026-04-17 09:33:01 +03:00
Antti Kettunen
47a6c257ad Refactor MusicMap similar artist matching
Shift similar-artist lookup to the shared metadata provider priority flow.
Use generic provider clients for search and metadata extraction instead of
branching on Spotify/iTunes-specific paths.

Add a regression test that verifies MusicMap matching queries the provider
priority list and preserves canonical metadata from the best match.
2026-04-17 09:24:05 +03:00
Antti Kettunen
7e1fc13e52 Make watchlist update_discovery_pool_incremental use provider priority
Continuation on recent changes
2026-04-17 09:08:36 +03:00
Antti Kettunen
bc83874c6f Discovery fan-out and playlists follow source priority
Make discovery pool population and curated playlists follow the configured metadata source order. Keep Spotify strict where fallback would corrupt source-specific IDs, and trim fan-out with smaller similar-artist samples and page caps. Leave the remaining incremental path for follow-up.
2026-04-17 08:49:19 +03:00
Antti Kettunen
030374c5b0 Tune discovery fan-out and caching
Reduce request volume in the discovery helpers while keeping the source-priority model intact.

- make cache_discovery_recent_albums source-priority aware
- cap Spotify artist-album pagination in the discovery and incremental paths
- reduce the similar-artist sample size for the cache-refresh helper
- keep Spotify strict where fallback would contaminate source-specific IDs
- add regression coverage for source order, strict Spotify lookups, and pagination caps
2026-04-17 08:27:36 +03:00
Antti Kettunen
6f9ea2de56 Remove redundant spotify auth check again 2026-04-17 07:30:04 +03:00
Broque Thomas
09d358ef69 Fix watchlist scan false failures, Spotify backfill, and wishlist remove
Watchlist scanner: empty discography (no new releases in lookback) was
treated as API failure, causing "Failed to get artist discography" for
artists like Kendrick Lamar who simply had no recent releases. Now
distinguishes None (API failure → try next source) from [] (success,
no new tracks). Spotify backfill now uses the authenticated client
instance instead of creating a fresh unauthenticated one.

Wishlist nebula: album remove now sends album_name (API updated to
accept album_name as fallback alongside album_id). Track remove
re-renders the nebula after deletion. Toned down processing pulse
animation.

Updated test to verify fallback triggers on API failure (None), not
on empty results.
2026-04-16 18:06:45 -07:00
Broque Thomas
258fc39364 Picard-style release preference scoring for MusicBrainz matching
Replaced track-count-only release selection with deterministic scoring
across 6 factors: track count match (40pts), release status (10pts),
country preference with US/worldwide bias (10pts), format preference
favoring Digital/CD over Vinyl/Cassette (10pts), barcode presence (3pts),
and date completeness (2pts). Same inputs always produce the same release.

Also fixed critical bug: _embed_source_ids was missing the context
parameter, silently skipping ALL source ID tag embedding since the
MusicBrainz consistency commit. Now passes context from the caller.
2026-04-16 13:29:04 -07:00
BoulderBadgeDad
61c7848121
Merge pull request #304 from kettui/fix/populate-discovery-pool-fixes
Make discovery pool population respect provider priority, reduce request volume
2026-04-16 11:57:38 -07:00
Antti Kettunen
e447cf6ab0 Reduce discovery fan-out and pagination
Make discovery pool population respect provider priority while keeping Spotify strict, and reduce unnecessary request volume in the hot discovery paths.

- keep discovery fan-out source-priority aware
- preserve cache use where freshness is not required
- cap Spotify artist-album pagination in discovery and cache refresh paths
- keep incremental release checks to a single page, since they only need the newest releases
- add regression coverage for provider order, strict Spotify handling, and pagination caps
2026-04-16 20:59:26 +03:00
Broque Thomas
223522ce99 Upgrade AcoustID scanner to scan full library with actionable fixes
Rewrote the AcoustID scanner job to scan all library tracks (via DB file
paths resolved to disk) instead of only the Transfer folder. Checkpoints
by track ID for robust resume across restarts. Defaults changed to
enabled, 24h interval, batch size 200.

Added _fix_acoustid_mismatch handler with three actions:
- retag: update DB title/artist to match actual audio content
- redownload: add expected track to wishlist and delete wrong file
- delete: remove wrong file and DB record

This catches cases like a file tagged as "Dinosaur Bones" that is
actually "Helicopters" — the scanner fingerprints the audio, detects
the mismatch, and the user can fix it from Library Maintenance findings.
2026-04-16 10:26:39 -07:00
Antti Kettunen
08ac39bc13 Fix watchlist discography lookback handling
Route get_artist_discography through the shared client helper so it uses the existing lookback logic instead of referencing an out-of-scope variable.
2026-04-16 10:09:42 +03:00
Antti Kettunen
e657a1d432 Make watchlist Spotify matching strict
Resolve Spotify artist matching through the exact Spotify client only, so watchlist ID backfill cannot drift to fallback-provider results. Remove the remaining preemptive provider availability check from the backfill loop.
2026-04-16 09:59:49 +03:00
Antti Kettunen
992be1a056 Use cache before Spotify auth checks
Allow cached Spotify search results to return even when Spotify is rate-limited or temporarily unavailable, and remove redundant rate-limit gating after auth checks.
2026-04-16 09:54:00 +03:00
Antti Kettunen
09a7465b22 Add strict fallback opt-out to Spotify client
Expose an allow_fallback flag on Spotify search and metadata methods so strict callers can avoid silently resolving through fallback providers.
2026-04-16 09:47:36 +03:00
Antti Kettunen
7b3a32ccc5 Remove dead watchlist source helpers
Drop the old active-provider artist lookup helpers from watchlist_scanner now that the web scan flow resolves sources through the shared metadata priority.

Keep the Spotify-specific feature toggles in place for discovery and sync paths that still use them.
2026-04-16 09:14:07 +03:00
Antti Kettunen
38b907097d Make watchlist scanning source-aware
Move the web watchlist scan core onto the shared metadata source priority so primary provider settings are respected during artist, album, and image resolution.

Add coverage for primary-source-first discography lookup and fallback to later providers when the primary source has no albums.
2026-04-16 09:13:15 +03:00
Antti Kettunen
9d73b8b561 Restore placeholder filtering and shared image backfill
Bring placeholder tracklist skipping back into the shared watchlist scan path, and centralize the DB-only artist image backfill helper so both web scan entrypoints reuse the same logic.
2026-04-16 08:31:04 +03:00
Antti Kettunen
40fa139804 Remove dead watchlist scan paths
Drop the legacy watchlist scan entrypoints that are no longer used by the web scan flow, and keep the live refresh path pointed at the shared scanner helper.
2026-04-16 08:27:41 +03:00
Antti Kettunen
657d86cace Consolidate web watchlist scanning
Move the shared watchlist scan loop into core/watchlist_scanner.py so web_server.py only handles triggers, locks, progress, and post-scan orchestration.

Manual and scheduled watchlist scans now share the same scanner-side core, while the web entrypoints keep profile selection and automation progress updates.
2026-04-16 08:20:48 +03:00
Broque Thomas
bf123fed63 Reject Qobuz 30-second sample/preview downloads
Two-layer detection: (1) check the Qobuz API response for sample=True
before downloading, and (2) validate actual file duration with mutagen
after download — if under 35 seconds, delete and return None. Qobuz
returns valid audio files for previews (~2-5MB FLAC) that pass the
existing 100KB size check, so duration is the reliable signal.
2026-04-15 20:41:48 -07:00
Broque Thomas
9d77c403cc Fix Spotify enrichment worker infinite loop on pre-matched artists
Artists with an existing spotify_artist_id but NULL spotify_match_status
were fetched by the priority queue every ~3 seconds. _process_artist
returned early (preserving the ID) without marking the status, so the
same artist was re-queued indefinitely — burning CPU and inflating API
call counters. Now marks the artist as 'matched' on the early-return
path.
2026-04-15 20:37:20 -07:00
BoulderBadgeDad
19a5256feb
Merge pull request #301 from kettui/fix/respect-metadata-provider-in-more-jobs
Respect the primary metadata provider in more jobs
2026-04-15 13:04:05 -07:00
Antti Kettunen
dab58766b7 Make library reorganize source-aware
Respect the configured metadata source order when looking up album years, and re-check provider availability during the scan so Spotify can drop out cleanly if it becomes rate-limited.
2026-04-15 21:35:58 +03:00
Antti Kettunen
03711c10df Make metadata gap filler source-aware
Only fetch source track details when ISRC enrichment is enabled, and show resolved source/track provenance in the UI.
2026-04-15 21:35:57 +03:00
Antti Kettunen
6df6ecb560 Respect source preference in cover art repair job
Cover art lookup now honors an explicit prefer_source first,
falls back to the runtime primary metadata source when unset,
and uses the shared source priority for the remaining fallbacks.
2026-04-15 21:28:12 +03:00
Antti Kettunen
e7faa9f02f Use metadata source priority in track number repair job
Use the shared metadata source priority when resolving album IDs,
album searches, and tracklists in track number repair.

Keeps Deezer and iTunes ahead of Spotify where configured, while
still allowing the job to fall back through other supported sources.
2026-04-15 21:28:12 +03:00
Antti Kettunen
6dca19ca1e Use metadata source priority in unknown artist fixer job
Unknown artist resolution now uses the shared metadata source priority and only filters to the sources that can actually participate in this job. Deezer and iTunes remain direct lookup sources, while Hydrabase can now join the title-search path when it is the configured priority source.
2026-04-15 21:28:10 +03:00
Broque Thomas
71a56bf9b7 Fix playlist cover art not syncing to Plex/Jellyfin from Deezer and other sources
- Pass playlist image_url to _run_sync_task from all source-specific sync
  start handlers (Deezer, Tidal, Spotify public, YouTube, automation mirror)
  — previously only the /api/sync/start endpoint passed it
- Fix plex_client.set_playlist_image: use uploadPoster(url=) instead of
  uploadPoster(data=) which is not a valid PlexAPI argument
- deezer_client: use picture_xl > picture_big > picture_medium fallback
  for better cover art resolution
- tidal_client: extract image_url in get_playlist() from JSON:API
  relationships (was only extracted in metadata-only listing)
- parse_youtube_playlist: capture playlist thumbnail from yt-dlp result
- Add visible logging for image upload attempts and outcomes
2026-04-15 10:25:38 -07:00
Broque Thomas
fe399636b2 Fix Spotify API calls leaking when Deezer/iTunes is primary source
Spotify was being called for album/artist data fetching across multiple
background workers and the Artists page search even when the user had
Deezer or iTunes set as their primary metadata source. Being authenticated
for playlist sync was treated as permission to use Spotify for everything.

- watchlist_scanner: add _spotify_is_primary_source() that checks both
  auth and primary source config; use it for all album/artist data fetching
  (discovery pool, recent album caching, playlist curation, similar artist
  ID matching, proactive ID backfill). _spotify_available_for_run() is kept
  for sync_spotify_library_cache which must run regardless of primary source
- repair_jobs/metadata_gap_filler: gate Spotify ISRC lookup on primary
  source being 'spotify'; MusicBrainz lookup unaffected
- repair_jobs/unknown_artist_fixer: replace hardcoded spotify_client with
  source-aware client selection — primary source ID tried first, each ID
  matched to its correct client (fixes latent bug passing Deezer IDs to
  Spotify)
- web_server.py /api/match/search: Artists page search was hardcoded to
  spotify_client.search_artists(); now uses _get_metadata_fallback_client()
  so results come from the configured primary source
2026-04-15 09:47:43 -07:00
BoulderBadgeDad
3618f3fa7f
Merge pull request #298 from kettui/fix/respect-metadata-provider-in-album-completeness-job
Honor primary metadata source in album_completeness job and associated repair flow
2026-04-15 07:15:15 -07:00
Broque Thomas
251c27e006 Add Last.fm Track Radio to Discover page
Adds a new Last.fm Radio section to the Discover page that lets users
search a track on Last.fm, generate a similar-tracks playlist, and run
it through the existing discovery/download/sync pipeline. Also generates
playlists automatically from top listening history during watchlist scans
(max once per week).

- core/lastfm_client.py: Add get_similar_tracks() using track.getsimilar
- core/listenbrainz_manager.py: Add save_lastfm_radio_playlist() with
  deterministic MBID (MD5 seed), cleanup limit of 5 for lastfm_radio type
- web_server.py: Add /api/lastfm/configured, /api/lastfm/search/tracks,
  /api/lastfm/radio/generate, /api/discover/listenbrainz/lastfm-radio;
  fix playlist['name'] KeyError in discovery worker that was resetting
  phase back to 'fresh' after completion
- core/watchlist_scanner.py: Add _generate_lastfm_radio_playlists() with
  weekly throttle, called at end of scan_all_watchlist_artists()
- webui/index.html: Add #lastfm-radio-section above ListenBrainz section,
  hidden unless Last.fm API key is configured
- webui/static/script.js: Search/generation/card-load functions; fix
  discovery modal labels (Last.fm Radio vs ListenBrainz), description
  update on completion, belt-and-suspenders completion handling inside
  updateYouTubeDiscoveryModal; fix album/duration display for tracks
  without metadata; music note SVG placeholder for missing art
- webui/static/style.css: Styles for search bar, dropdown, result rows
2026-04-14 23:41:36 -07:00
Antti Kettunen
106d202ccc Share metadata source priority
Centralize the ordered metadata source list and source-priority helper so album completeness and the repair worker follow the same Deezer/iTunes-first fallback order. This also removes the last duplicate priority logic from the touched repair paths.
2026-04-15 07:48:25 +03:00
Antti Kettunen
8a1ae00946 Utilize global Spotify client in repair jobs
Album completeness and any other repair job now uses the centralized source/client helpers instead of a worker-local Spotify client or override plumbing
  - This keeps source selection aligned with the configured primary provider and removes the last Spotify-only special case from the job path.

This change ultimately is a step towards further centralizing the Spotify client access and the associated `is_spotify_authenticated` check.
  - Currently these look-ups are done all over the place in different feature implementations directly, but moving forward, any feature that uses `get_primary_client` or `get_client_for_source` to access the Spotify client, won't have to duplicate any rate-limiting or auth checks as long as these getters are used
2026-04-15 07:18:34 +03:00
Broque Thomas
ce129010e1 Add ReplayGain analysis and tagging support
New core/replaygain.py module uses FFmpeg's ebur128 filter (already a
project dependency) to analyze integrated loudness and true peak, then
writes ReplayGain 2.0 tags (-18 LUFS reference) to MP3 (TXXX frames),
FLAC/OGG/Opus (Vorbis comments), and M4A/MP4 (freeform atoms).

Three analysis modes in the enhanced library view:
- Per-track RG button: synchronous single-track analysis (~1-3 s)
- Album "ReplayGain" button: background job writing both track gain
  and album gain (mean LUFS across all album tracks) to every file
- Bulk bar "ReplayGain" button: batch track-gain for selected tracks

read_file_tags() in tag_writer.py extended with four new optional keys
(replaygain_track_gain/_peak, replaygain_album_gain/_peak) so existing
RG values surface in the tag-preview diff view. Purely additive — no
existing endpoints or DB schema changed.
2026-04-14 19:09:25 -07:00
Broque Thomas
1071b2ebe5 Make OAuth callback ports configurable via environment variables
Hardcoded ports 8888/8889 conflict when SoulSync runs behind Gluetun or
other containers that claim those ports. Introduce SOULSYNC_SPOTIFY_CALLBACK_PORT
and SOULSYNC_TIDAL_CALLBACK_PORT env vars (defaulting to 8888/8889) so
users can remap without rebuilding the image.

docker-compose.yml exposes the vars with comments explaining how to keep
the port mappings in sync with the redirect URI in Settings → Connections.
2026-04-14 14:08:31 -07:00
Broque Thomas
6e405143a7 Improve Tidal download failure diagnostics and error messaging
Track per-quality-tier failure reasons across all failure paths (stream
error, empty manifest, download exception, stub file, MP4 extraction
failure) and include them in the exhausted-tiers log message so failures
are diagnosable from logs.

When HiRes is configured with no fallback and all tiers are exhausted,
log an actionable hint directing the user to enable Quality Fallback.

Surface Tidal-specific error messages in the UI task on retry
exhaustion: distinguishes HiRes-unavailable (with actionable guidance)
from general Tidal auth/quality failures, rather than showing the
generic Soulseek error string.
2026-04-14 14:07:50 -07:00
Broque Thomas
0edb8e93bb Fix MBID Mismatch Detector comparing MB title against DB title (filename) instead of embedded tag
The scan was reading `t.title` from the database for comparison, which is
populated from the filename rather than the file's embedded TITLE tag. Any
track whose filename differs from the clean MB title (e.g. includes artist,
album, bitrate) was incorrectly flagged as a mismatch.

Fix: extend `_read_mbid_from_file` → `_read_file_tags` to also read the
embedded TITLE tag (ID3 TIT2, Vorbis `title`, MP4 `©nam`). The comparison
now uses the embedded title, falling back to the DB title only when no TITLE
tag is present in the file.

Fixes #296
2026-04-14 12:19:41 -07:00
Antti Kettunen
1a459412a3 Honor primary metadata source in album_completeness job
Album completeness and downstream repair flow now follow the configured
primary provider first, with Discogs and Hydrabase support added alongside
existing Spotify, iTunes, and Deezer paths.

Keep spotify_track_id for compatibility while preserving source-aware track
IDs for provider-neutral handling.
2026-04-14 21:14:46 +03:00
Broque Thomas
3b8b369492 Add Your Albums — multi-source liked albums pool (Spotify, Tidal, Deezer)
Builds a new Your Albums section on the Discover page that aggregates
saved/liked albums from all connected services, mirroring the Your Artists
pattern. Deezer works via both OAuth and ARL.

- tidal_client: add get_favorite_albums() with V2/V1 API fallback
- deezer_client: add get_user_favorite_albums() via OAuth (user/me/albums)
- deezer_download_client: add get_user_favorite_albums() via ARL session
- music_database: add liked_albums_pool table (deduped by artist::album
  normalized key), upsert_liked_album, get_liked_albums,
  get_liked_albums_last_fetch, clear_liked_albums
- web_server: GET /api/discover/your-albums (ownership-checked, paginated),
  GET /api/discover/your-albums/sources, POST /api/discover/your-albums/refresh,
  _fetch_liked_albums background worker (Spotify + Tidal + Deezer OAuth/ARL)
- frontend: Your Albums section with source selector cog, album grid reusing
  spotify-library-card styles, search/filter/sort/pagination, download missing
  button, auto-refresh poll on first load

Also fix: Deezer greyed out in Your Artists sources when using ARL — connection
check now accepts ARL auth (deezer_dl.is_authenticated()) in addition to OAuth,
and _fetch_and_match_liked_artists falls back to ARL client for artist fetching.
2026-04-13 23:02:37 -07:00
Broque Thomas
453eb90f19 Add Deezer ARL favorite artists support
Add get_user_favorite_artists(limit=200) to DeezerDownloadClient to fetch a user's favorite artists via the public API using an ARL-authenticated session (paginated, error-handled, returns deezer_id, name, image_url).

Update web_server to treat Deezer as connected if either OAuth or ARL is authenticated, and to fetch favorite artists from OAuth client when available or from soulseek_client.deezer_dl (ARL) otherwise. Fetched artists are upserted into the database and appropriate log/console messages and counters are updated.
2026-04-13 19:46:39 -07:00
Broque Thomas
5bb36412be Fix Jellyfin playlist track fetch truncating at default API limit
get_playlist_tracks() had no Limit or StartIndex params, so Jellyfin
defaulted to ~100 items per response. This caused the Server Playlists
comparison to show most tracks as missing even though they were present.

Now paginates in batches of 1000 until a partial batch signals the last page.
2026-04-13 15:17:52 -07:00
Broque Thomas
20c8bff85c Upgrade artwork to highest available resolution for Spotify and iTunes
Spotify album art: replace the 4-char size segment after '0000ab67616d'
with '82c1' to request the original uploaded master (up to 2000px+).
Applied via _upgrade_spotify_image_url() in Track, Artist, and Album
dataclass constructors and as a catch-all in _download_cover_art.
Scoped to the ab67616d album art prefix only — artist images use a
different prefix (ab676161) where the trick does not apply.

iTunes/Apple Music: replace '100x100bb' with '3000x3000bb' in all
artworkUrl100 replacements across Track, Artist, Album, and the
get_album images arrays. Also applied as a catch-all in _download_cover_art.

Deezer already uses cover_xl at its maximum — no changes needed there.
2026-04-13 15:04:37 -07:00
Broque Thomas
352ad5ff68 Fix AcoustID test connection falsely failing for valid API keys
The fallback test (used when no audio files exist in the library) sends
a dummy fingerprint to the AcoustID API. The API correctly rejects the
dummy fingerprint but this is not error code 4 (invalid key), so the
test was returning False instead of True. Any non-code-4 error from the
fallback means the API key was accepted — only code 4 means a bad key.
2026-04-13 09:37:50 -07:00
Broque Thomas
0edd8f5c81 Raise artist discography limit from 50 to 200 with Deezer pagination
Deezer and iTunes defaulted to 50 albums max, silently truncating large
discographies. Deezer now paginates (100 per page) up to 200. iTunes
raised to 200 (single call). All callers in web_server.py updated to
use the new defaults instead of hardcoding limit=50.

Also adds diagnostic logging for allow_duplicates album comparison
to help debug inconsistent singles behavior.
2026-04-12 21:46:04 -07:00
Broque Thomas
3a7c25f20f Fix allow_duplicates not working for singles in watchlist scanner
Singles like "idol" weren't added when the same song existed on a
different album because check_track_exists used album-aware matching
that found the track via album name. Now skips the album hint when
allow_duplicates is on so matching is title+artist only, then compares
album names ourselves with a strict 0.85 threshold. Only affects users
with allow_duplicates enabled.
2026-04-12 17:41:55 -07:00