Compare commits
No commits in common. "main" and "2.7.8" have entirely different histories.
194 changed files with 3195 additions and 18295 deletions
4
.github/workflows/docker-publish.yml
vendored
4
.github/workflows/docker-publish.yml
vendored
|
|
@ -9,9 +9,9 @@ on:
|
|||
workflow_dispatch:
|
||||
inputs:
|
||||
version_tag:
|
||||
description: 'Version tag (e.g. 2.8.2)'
|
||||
description: 'Version tag (e.g. 2.7.8)'
|
||||
required: true
|
||||
default: '2.8.2'
|
||||
default: '2.7.8'
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
|
|
|
|||
|
|
@ -1,74 +0,0 @@
|
|||
# discover page — best in class plan (#913 + full generator audit)
|
||||
|
||||
morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
|
||||
|
||||
## what i shipped tonight (done, tested, safe)
|
||||
|
||||
### 1. listening recommendations (#913) — went from BROKEN to best-in-class
|
||||
|
||||
the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
|
||||
|
||||
- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
|
||||
- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
|
||||
- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
|
||||
|
||||
the fix (all in the pure, tested core + thin scan wiring):
|
||||
- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
|
||||
- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
|
||||
|
||||
result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
|
||||
|
||||
### 2. its own row on the discover page
|
||||
|
||||
new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
|
||||
|
||||
**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
|
||||
|
||||
> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
|
||||
|
||||
### 3. Fresh Tape "only 5-10 tracks" — fixed
|
||||
|
||||
root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
|
||||
|
||||
**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
|
||||
|
||||
---
|
||||
|
||||
## best-in-class roadmap for listening recs (next phases — your call)
|
||||
|
||||
these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
|
||||
|
||||
| phase | what | value | risk | notes |
|
||||
|---|---|---|---|---|
|
||||
| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
|
||||
| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
|
||||
| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
|
||||
| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
|
||||
| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
|
||||
|
||||
the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
|
||||
|
||||
---
|
||||
|
||||
## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
|
||||
|
||||
how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
|
||||
|
||||
### curated (built during the scan, then hydrated)
|
||||
- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
|
||||
- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
|
||||
- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
|
||||
|
||||
### discovery-pool generators (live queries)
|
||||
- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
|
||||
- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
|
||||
- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
|
||||
- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
|
||||
- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
|
||||
- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
|
||||
|
||||
### cross-cutting
|
||||
- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
|
||||
- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
|
||||
|
||||
want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
**SoulSync 2.7.9** is out 🎉 a big one.
|
||||
|
||||
🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
|
||||
|
||||
🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
|
||||
|
||||
⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
|
||||
|
||||
🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
|
||||
|
||||
🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
|
||||
|
||||
⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
|
||||
|
||||
🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
|
||||
|
||||
✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
|
||||
|
||||
💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
|
||||
|
||||
🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
|
||||
|
||||
⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
|
||||
|
||||
🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
|
||||
|
||||
🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
|
||||
|
||||
💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
|
||||
|
||||
🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
|
||||
|
||||
⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
|
||||
|
||||
🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
**SoulSync 2.8.2** is out 🎉 a stability + performance release.
|
||||
|
||||
🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
|
||||
|
||||
⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
|
||||
|
||||
📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
|
||||
|
||||
enjoy! 🎶
|
||||
|
|
@ -705,12 +705,6 @@ class ConfigManager:
|
|||
},
|
||||
"import": {
|
||||
"staging_path": "./Staging",
|
||||
# Master toggle for quality-filtering on import. On by default:
|
||||
# downloaded files that don't meet the quality profile are
|
||||
# quarantined instead of imported (same gate the download
|
||||
# pipeline uses). Off → import everything regardless of quality;
|
||||
# the library Quality Upgrade Scanner still flags them.
|
||||
"quality_filter_enabled": True,
|
||||
"replace_lower_quality": False,
|
||||
# Use the top Staging folder as the artist (Artist/Album layouts,
|
||||
# mixtapes). On by default to preserve the long-standing import
|
||||
|
|
|
|||
|
|
@ -32,8 +32,6 @@ from config.settings import config_manager
|
|||
from core.amazon_client import AmazonClient, AmazonClientError
|
||||
from core.download_plugins.base import DownloadSourcePlugin
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.quality.model import AudioQuality
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("amazon_download_client")
|
||||
|
|
@ -78,12 +76,9 @@ class AmazonDownloadClient(DownloadSourcePlugin):
|
|||
if download_path is None:
|
||||
download_path = config_manager.get("soulseek.download_path", "./downloads")
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
self._quality = quality_tier_for_source("amazon", default="flac")
|
||||
self._quality = config_manager.get("amazon_download.quality", "flac")
|
||||
self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
|
||||
|
||||
self._client = AmazonClient(preferred_codec=self._quality)
|
||||
|
|
@ -138,17 +133,11 @@ class AmazonDownloadClient(DownloadSourcePlugin):
|
|||
album_map: Dict[str, AlbumResult] = {}
|
||||
album_order: List[str] = []
|
||||
preferred = self._client.preferred_codec
|
||||
# Search results only carry the codec (real sample_rate arrives at
|
||||
# stream time). Claim the format honestly — FLAC for the lossless
|
||||
# codec, lossy otherwise — so audio_quality derives a real format
|
||||
# instead of the display label ("Lossless"), and the post-download
|
||||
# probe pins the actual sample_rate/bit_depth.
|
||||
amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
|
||||
|
||||
for item in items:
|
||||
quality = _quality_label(preferred)
|
||||
if item.is_track:
|
||||
tr = TrackResult(
|
||||
track_results.append(TrackResult(
|
||||
username="amazon",
|
||||
filename=f"{item.asin}||{item.artist_name} - {item.title}",
|
||||
size=0,
|
||||
|
|
@ -166,9 +155,7 @@ class AmazonDownloadClient(DownloadSourcePlugin):
|
|||
"album_asin": item.album_asin,
|
||||
"isrc": item.isrc,
|
||||
},
|
||||
)
|
||||
tr.set_quality(amazon_q)
|
||||
track_results.append(tr)
|
||||
))
|
||||
elif item.is_album:
|
||||
album_asin = item.album_asin or item.asin
|
||||
if album_asin not in album_map:
|
||||
|
|
@ -186,7 +173,6 @@ class AmazonDownloadClient(DownloadSourcePlugin):
|
|||
title=item.title,
|
||||
album=item.album_name,
|
||||
)
|
||||
placeholder.set_quality(amazon_q)
|
||||
album_map[album_asin] = AlbumResult(
|
||||
username="amazon",
|
||||
album_path=album_asin,
|
||||
|
|
|
|||
|
|
@ -1,27 +0,0 @@
|
|||
"""Boot-phase guard for non-blocking container startup.
|
||||
|
||||
While the gunicorn worker is importing ``web_server`` (module-level client and
|
||||
worker initialization), external provider API probes must not block startup.
|
||||
Network validation is deferred until ``mark_boot_complete()`` runs at the end
|
||||
of that import pass.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
|
||||
_boot_lock = threading.Lock()
|
||||
_boot_active = True
|
||||
|
||||
|
||||
def is_boot_phase() -> bool:
|
||||
"""Return True while module import must avoid blocking provider API calls."""
|
||||
with _boot_lock:
|
||||
return _boot_active
|
||||
|
||||
|
||||
def mark_boot_complete() -> None:
|
||||
"""End the boot phase — provider clients may perform network probes again."""
|
||||
global _boot_active
|
||||
with _boot_lock:
|
||||
_boot_active = False
|
||||
|
|
@ -21,7 +21,6 @@ from typing import Any, Dict, List, Optional, Tuple
|
|||
import requests
|
||||
|
||||
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
|
||||
from core.quality.source_map import quality_from_deezer, quality_tier_for_source
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("deezer_download")
|
||||
|
|
@ -93,10 +92,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
if download_path is None:
|
||||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Engine reference is populated by set_engine() at registration
|
||||
# time. None until orchestrator wires the registry.
|
||||
|
|
@ -121,20 +117,14 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
self._license_token = None
|
||||
self._user_data = None
|
||||
self._authenticated = False
|
||||
self._pending_arl: Optional[str] = None
|
||||
|
||||
# Quality preference
|
||||
self._quality = quality_tier_for_source('deezer', default='flac')
|
||||
self._quality = config_manager.get('deezer_download.quality', 'flac')
|
||||
|
||||
# Try to authenticate on init if ARL is configured
|
||||
arl = config_manager.get('deezer_download.arl', '')
|
||||
if arl:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if is_boot_phase():
|
||||
self._pending_arl = arl
|
||||
logger.debug("Deezer ARL present — authentication deferred until after boot")
|
||||
else:
|
||||
self._authenticate(arl)
|
||||
self._authenticate(arl)
|
||||
|
||||
logger.info(f"Deezer download client initialized (download path: {self.download_path})")
|
||||
|
||||
|
|
@ -233,66 +223,12 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
return self._authenticated
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
if self._pending_arl and not self._authenticated:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if not is_boot_phase():
|
||||
self._authenticate(self._pending_arl)
|
||||
self._pending_arl = None
|
||||
return self._authenticated
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.is_available)
|
||||
|
||||
# ─── Playlist export (#945) ──────────────────────────────────
|
||||
#
|
||||
# UNOFFICIAL: rides the private gw-light gateway with the ARL session already used
|
||||
# for downloads. Deezer shut their public developer API, so this is the only write
|
||||
# path — and it's fragile by nature (breaks when Deezer changes internals).
|
||||
|
||||
def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
|
||||
public=False, description=""):
|
||||
"""Create a Deezer playlist (or append to an existing one) from a mirrored
|
||||
playlist's tracks. ``track_ids`` are stored ``deezer_id`` values per library track.
|
||||
``existing_id`` set → add to that playlist (idempotent re-export reuses the stored
|
||||
target); unset → create a new one. Returns
|
||||
``{success, playlist_id, url, added, error}``."""
|
||||
if not self._authenticated:
|
||||
return {"success": False, "error": "Deezer is not connected (ARL)"}
|
||||
song_ids = [str(t) for t in (track_ids or []) if t]
|
||||
if not song_ids:
|
||||
return {"success": False, "error": "No matching Deezer tracks to export"}
|
||||
try:
|
||||
songs = [[sid, i] for i, sid in enumerate(song_ids)]
|
||||
if existing_id:
|
||||
res = self._gw_call("playlist.addSongs",
|
||||
{"playlist_id": int(existing_id), "songs": songs})
|
||||
if res is None:
|
||||
return {"success": False, "error": "Deezer rejected the playlist update"}
|
||||
playlist_id = existing_id
|
||||
else:
|
||||
res = self._gw_call("playlist.create", {
|
||||
"title": name, "description": description,
|
||||
"is_public": bool(public), "songs": songs,
|
||||
})
|
||||
if res is None:
|
||||
return {"success": False, "error": "Deezer rejected the playlist create"}
|
||||
# gw 'playlist.create' returns the new playlist id (int) as `results`.
|
||||
if isinstance(res, dict):
|
||||
playlist_id = res.get("PLAYLIST_ID") or res.get("id")
|
||||
else:
|
||||
playlist_id = res
|
||||
if not playlist_id:
|
||||
return {"success": False, "error": "Deezer did not return a playlist id"}
|
||||
return {
|
||||
"success": True,
|
||||
"playlist_id": str(playlist_id),
|
||||
"url": f"https://www.deezer.com/playlist/{playlist_id}",
|
||||
"added": len(song_ids),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e)}
|
||||
|
||||
def reconnect(self, arl: str = None) -> bool:
|
||||
"""Re-authenticate with a new or existing ARL."""
|
||||
if arl is None:
|
||||
|
|
@ -661,7 +597,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
bitrate = 128
|
||||
quality = 'mp3'
|
||||
|
||||
tr = TrackResult(
|
||||
results.append(TrackResult(
|
||||
username='deezer_dl',
|
||||
filename=f"{track_id}||{artist} - {title}",
|
||||
size=est_size,
|
||||
|
|
@ -675,10 +611,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
|
|||
title=title,
|
||||
album=album,
|
||||
track_number=item.get('track_position'),
|
||||
)
|
||||
# Stamp CD-quality FLAC (16/44.1) so lossless ranks correctly.
|
||||
tr.set_quality(quality_from_deezer(self._quality))
|
||||
results.append(tr)
|
||||
))
|
||||
|
||||
logger.info(f"Deezer search for '{query}' returned {len(results)} results")
|
||||
return results, []
|
||||
|
|
|
|||
|
|
@ -46,81 +46,6 @@ def _get(row: object, attr: str):
|
|||
return getattr(row, attr, None)
|
||||
|
||||
|
||||
def choose_mix_fetch_source(active_source: object, active_can_fetch: bool) -> str:
|
||||
"""Pick which source to fetch the "Listening Mix" top tracks from.
|
||||
|
||||
The mix is a list of (artist, title) pairs acquired via Soulseek, so the fetch source need
|
||||
NOT match the user's active metadata source. Use the active source when it can fetch top
|
||||
tracks itself (Spotify/Deezer); otherwise fall back to Deezer, whose public ``artist/{id}/top``
|
||||
needs no auth and is available to every user — so iTunes / Discogs / MusicBrainz users still
|
||||
get a full mix without switching sources. Pure.
|
||||
"""
|
||||
if str(active_source or "").lower() in ("spotify", "deezer") and active_can_fetch:
|
||||
return str(active_source).lower()
|
||||
return "deezer"
|
||||
|
||||
|
||||
def names_match(a: object, b: object) -> bool:
|
||||
"""Strict artist-name equality after stripping case + non-alphanumerics.
|
||||
|
||||
Used to verify a name-search result before fetching that artist's top tracks, so the
|
||||
"Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact
|
||||
alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell".
|
||||
Pure.
|
||||
"""
|
||||
def _alnum(x: object) -> str:
|
||||
return "".join(ch for ch in str(x or "").lower() if ch.isalnum())
|
||||
na, nb = _alnum(a), _alnum(b)
|
||||
return bool(na) and na == nb
|
||||
|
||||
|
||||
def similarity_from_rank(rank: object, max_rank: int = 10) -> float:
|
||||
"""Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight.
|
||||
|
||||
SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest
|
||||
match). The ranker multiplies this into the score so a seed's *closest* matches count
|
||||
for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0,
|
||||
rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A
|
||||
missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure.
|
||||
"""
|
||||
try:
|
||||
r = int(rank)
|
||||
except (TypeError, ValueError):
|
||||
return 1.0
|
||||
floor = round(1.0 / max_rank, 4)
|
||||
if r <= 1:
|
||||
return 1.0
|
||||
if r >= max_rank:
|
||||
return floor
|
||||
return round((max_rank - r + 1) / max_rank, 4)
|
||||
|
||||
|
||||
def build_recency_weighted_seeds(
|
||||
top_artists: Sequence[dict],
|
||||
recent_play_counts: Optional[Dict[str, float]] = None,
|
||||
*,
|
||||
recency_factor: float = 1.5,
|
||||
) -> List[dict]:
|
||||
"""Blend lifetime + recent play counts into seed weights — "what you're into NOW".
|
||||
|
||||
``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a
|
||||
lot *recently* outranks one you played a lot years ago, so the recommendations track
|
||||
your current taste instead of your all-time history. ``recency_factor`` is the dial
|
||||
(0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`.
|
||||
Pure — the caller supplies both play-count maps from the listening history.
|
||||
"""
|
||||
recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()}
|
||||
out: List[dict] = []
|
||||
for a in top_artists or ():
|
||||
name = str(a.get("name") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0)))
|
||||
boost = recency_factor * recent.get(_norm(name), 0.0)
|
||||
out.append({"name": name, "weight": lifetime + boost})
|
||||
return out
|
||||
|
||||
|
||||
def group_similars_by_seed(
|
||||
seeds: Sequence[dict],
|
||||
similar_rows: Sequence,
|
||||
|
|
@ -128,21 +53,14 @@ def group_similars_by_seed(
|
|||
*,
|
||||
source_id_attr: str = "source_artist_id",
|
||||
similar_name_attr: str = "similar_artist_name",
|
||||
rank_attr: Optional[str] = None,
|
||||
) -> Dict[str, List[dict]]:
|
||||
"""Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``.
|
||||
"""Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name': similar}]}``.
|
||||
|
||||
The stored rows key the similar artist by the SEED's source id (``source_artist_id``),
|
||||
not its name, so :func:`rank_recommended_artists` can't consume them directly. This
|
||||
resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id:
|
||||
artist_name}`` for the library, built by the caller) and keeps only rows that resolve
|
||||
to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O.
|
||||
|
||||
``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that
|
||||
is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id.
|
||||
When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank`
|
||||
and carried as ``score`` so closer matches weigh more; without it every similar comes out
|
||||
score-less (the ranker then treats similarity as 1.0 — original behavior).
|
||||
"""
|
||||
seed_names = {_norm(s.get("name")) for s in seeds}
|
||||
seed_names.discard("")
|
||||
|
|
@ -154,12 +72,8 @@ def group_similars_by_seed(
|
|||
if not seed_name or seed_name not in seed_names:
|
||||
continue
|
||||
sim_name = str(_get(row, similar_name_attr) or "").strip()
|
||||
if not sim_name:
|
||||
continue
|
||||
entry = {"name": sim_name}
|
||||
if rank_attr is not None:
|
||||
entry["score"] = similarity_from_rank(_get(row, rank_attr))
|
||||
out.setdefault(seed_name, []).append(entry)
|
||||
if sim_name:
|
||||
out.setdefault(seed_name, []).append({"name": sim_name})
|
||||
return out
|
||||
|
||||
|
||||
|
|
@ -283,57 +197,8 @@ def aggregate_candidate_tracks(
|
|||
return out[:limit]
|
||||
|
||||
|
||||
def to_mix_track(track: object, source: str) -> Optional[dict]:
|
||||
"""Shape one source "top tracks" API dict into the flat dict the Discover compact
|
||||
playlist row renders + syncs (the "Listening Mix" #913 playlist).
|
||||
|
||||
Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the
|
||||
same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``).
|
||||
This flattens that into the renderer's field names (``track_name/artist_name/album_name/
|
||||
album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and
|
||||
sets the source-specific id field. Returns None for anything without a usable id/title so
|
||||
the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can
|
||||
dedup by title. Pure — no I/O.
|
||||
"""
|
||||
if not isinstance(track, dict):
|
||||
return None
|
||||
tid = track.get("id")
|
||||
name = str(track.get("name") or "").strip()
|
||||
if not tid or not name:
|
||||
return None
|
||||
artists = track.get("artists") or []
|
||||
artist_name = ""
|
||||
if artists and isinstance(artists[0], dict):
|
||||
artist_name = str(artists[0].get("name") or "").strip()
|
||||
album = track.get("album") if isinstance(track.get("album"), dict) else {}
|
||||
album_name = str(album.get("name") or "").strip()
|
||||
images = album.get("images") or []
|
||||
cover = images[0].get("url") if images and isinstance(images[0], dict) else None
|
||||
out = {
|
||||
"track_id": str(tid),
|
||||
"name": name, # for aggregate_candidate_tracks dedup
|
||||
"track_name": name, # for the renderer
|
||||
"artist_name": artist_name,
|
||||
"album_name": album_name,
|
||||
"album_cover_url": cover,
|
||||
"duration_ms": track.get("duration_ms") or 0,
|
||||
"track_data_json": track, # full payload for sync/download
|
||||
"source": source,
|
||||
}
|
||||
id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id",
|
||||
"itunes": "itunes_track_id"}.get(source)
|
||||
if id_field:
|
||||
out[id_field] = str(tid)
|
||||
return out
|
||||
|
||||
|
||||
__all__ = [
|
||||
"RecommendedArtist",
|
||||
"choose_mix_fetch_source",
|
||||
"names_match",
|
||||
"similarity_from_rank",
|
||||
"build_recency_weighted_seeds",
|
||||
"to_mix_track",
|
||||
"group_similars_by_seed",
|
||||
"rank_recommended_artists",
|
||||
"aggregate_candidate_tracks",
|
||||
|
|
|
|||
|
|
@ -21,11 +21,7 @@ from core.metadata.registry import get_client_for_source
|
|||
from core.metadata.types import Album
|
||||
from core.wishlist.payloads import ensure_wishlist_track_format
|
||||
|
||||
# Use the project logger namespace ("soulsync.*") so the scanner's progress and
|
||||
# diagnostics actually surface in the app log — plain getLogger(__name__) lands
|
||||
# under "core.discovery.quality_scanner", which the app log view doesn't show.
|
||||
from utils.logging_config import get_logger
|
||||
logger = get_logger("discovery.quality_scanner")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Per-source typed converter dispatch — same registry pattern as
|
||||
|
|
|
|||
|
|
@ -25,7 +25,6 @@ big-bang switchover.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
from typing import Any, Dict, Iterator, List, Optional, Tuple
|
||||
|
||||
|
|
@ -392,16 +391,6 @@ class DownloadEngine:
|
|||
(tracks, albums) tuple, or ``([], [])`` when every source
|
||||
in the chain is exhausted.
|
||||
|
||||
Priority mode is deliberately quality-AGNOSTIC at search time — source
|
||||
order is king and the first source that returns any tracks wins, exactly
|
||||
matching pre-quality-system behaviour byte-for-byte (#896 review #3).
|
||||
Quality-gating the priority path would deprioritise e.g. a soulseek
|
||||
mp3 whose bitrate slskd omitted (``bitrate=None`` → "unsatisfied"),
|
||||
changing which source wins and adding latency for users who never opted
|
||||
in. Cross-source quality pooling is the job of best_quality mode
|
||||
(``search_all_sources``); final per-result ranking still happens in the
|
||||
orchestrator's match/quality filter. RAW tracks are returned.
|
||||
|
||||
Replaces orchestrator's hand-rolled hybrid search loop. The
|
||||
chain is ordered (most-preferred first).
|
||||
"""
|
||||
|
|
@ -417,10 +406,9 @@ class DownloadEngine:
|
|||
try:
|
||||
logger.info(f"Trying {source_name} (priority {i+1}): {query}")
|
||||
tracks, albums = await plugin.search(query, timeout, progress_callback)
|
||||
if not tracks:
|
||||
continue
|
||||
logger.info(f"{source_name} found {len(tracks)} tracks")
|
||||
return (tracks, albums)
|
||||
if tracks:
|
||||
logger.info(f"{source_name} found {len(tracks)} tracks")
|
||||
return (tracks, albums)
|
||||
except Exception as e:
|
||||
logger.warning(f"{source_name} search failed: {e}")
|
||||
|
||||
|
|
@ -430,75 +418,6 @@ class DownloadEngine:
|
|||
)
|
||||
return ([], [])
|
||||
|
||||
async def search_all_sources(self, query: str, source_chain,
|
||||
timeout=None, progress_callback=None,
|
||||
exclude_sources=None):
|
||||
"""Best-quality mode: pool RAW tracks from EVERY configured source in
|
||||
``source_chain`` instead of stopping at the first satisfying one.
|
||||
|
||||
Unlike :meth:`search_with_fallback`, no source short-circuits the
|
||||
search — the caller (orchestrator/worker) ranks the combined pool
|
||||
best→worst by actual audio quality. ``exclude_sources`` drops sources
|
||||
whose per-source retry budget is already spent (so their candidates
|
||||
never re-enter the pool). Unconfigured / unregistered / raising sources
|
||||
are skipped exactly like the fallback path. Returns
|
||||
``(combined_tracks, combined_albums)``.
|
||||
"""
|
||||
excluded = {s.lower() for s in (exclude_sources or []) if s}
|
||||
pooled_tracks = []
|
||||
pooled_albums = []
|
||||
# Per-source contribution for an honest pool log — e.g. a release-level
|
||||
# source like usenet/torrent that returns nothing for a track-title
|
||||
# query should read "usenet=0", not silently hide behind the chain name.
|
||||
contributions = []
|
||||
|
||||
# Decide which sources to actually query, recording why the rest were
|
||||
# skipped. Searches then run CONCURRENTLY so the pool waits only for the
|
||||
# slowest source (e.g. usenet/Prowlarr, which can be slow) rather than
|
||||
# the sum of every source's latency.
|
||||
to_search = [] # (source_name, plugin)
|
||||
for source_name in source_chain:
|
||||
if source_name.lower() in excluded:
|
||||
contributions.append(f"{source_name}=excluded")
|
||||
continue
|
||||
plugin = self._plugins.get(source_name)
|
||||
if plugin is None:
|
||||
logger.info(f"Skipping {source_name} (not available)")
|
||||
contributions.append(f"{source_name}=unavailable")
|
||||
continue
|
||||
if hasattr(plugin, 'is_configured') and not plugin.is_configured():
|
||||
logger.info(f"Skipping {source_name} (not configured)")
|
||||
contributions.append(f"{source_name}=unconfigured")
|
||||
continue
|
||||
to_search.append((source_name, plugin))
|
||||
|
||||
async def _one(plugin):
|
||||
return await plugin.search(query, timeout, progress_callback)
|
||||
|
||||
results = await asyncio.gather(
|
||||
*[_one(plugin) for _, plugin in to_search],
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
for (source_name, _), result in zip(to_search, results, strict=True):
|
||||
if isinstance(result, Exception):
|
||||
logger.warning(f"{source_name} search failed: {result}")
|
||||
contributions.append(f"{source_name}=error")
|
||||
continue
|
||||
tracks, albums = result
|
||||
n = len(tracks) if tracks else 0
|
||||
if tracks:
|
||||
pooled_tracks.extend(tracks)
|
||||
if albums:
|
||||
pooled_albums.extend(albums)
|
||||
contributions.append(f"{source_name}={n}")
|
||||
|
||||
logger.info(
|
||||
"Best-quality pool: %d candidates [%s] for: %s",
|
||||
len(pooled_tracks), ', '.join(contributions), query,
|
||||
)
|
||||
return (pooled_tracks, pooled_albums)
|
||||
|
||||
async def download_with_fallback(self, username: str, filename: str,
|
||||
file_size: int, source_chain) -> Optional[str]:
|
||||
"""Try each source in ``source_chain`` until one accepts the
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ from config.settings import config_manager
|
|||
from core.download_engine import DownloadEngine
|
||||
from core.download_plugins.registry import DownloadPluginRegistry, build_default_registry
|
||||
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
|
||||
from core.quality.selection import load_search_mode
|
||||
|
||||
logger = get_logger("download_orchestrator")
|
||||
|
||||
|
|
@ -103,14 +102,12 @@ class DownloadOrchestrator:
|
|||
deezer_dl = self.client('deezer_dl')
|
||||
if deezer_arl and deezer_dl:
|
||||
deezer_dl.reconnect(deezer_arl)
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
deezer_dl._quality = quality_tier_for_source('deezer', default='flac')
|
||||
deezer_dl._quality = config_manager.get('deezer_download.quality', 'flac')
|
||||
|
||||
# Reload Amazon quality preference (T2Tunes needs no reconnect — public proxy)
|
||||
amazon = self.client('amazon')
|
||||
if amazon:
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
quality = quality_tier_for_source('amazon', default='flac')
|
||||
quality = config_manager.get('amazon_download.quality', 'flac')
|
||||
amazon._quality = quality
|
||||
amazon._allow_fallback = config_manager.get('amazon_download.allow_fallback', True)
|
||||
if hasattr(amazon, '_client') and amazon._client:
|
||||
|
|
@ -143,10 +140,7 @@ class DownloadOrchestrator:
|
|||
continue
|
||||
if hasattr(client, 'download_path') and client.download_path != new_path:
|
||||
client.download_path = new_path
|
||||
try:
|
||||
client.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {new_path}: {e}")
|
||||
client.download_path.mkdir(parents=True, exist_ok=True)
|
||||
# YouTube also caches path in yt-dlp opts
|
||||
if hasattr(client, 'download_opts') and 'outtmpl' in client.download_opts:
|
||||
client.download_opts['outtmpl'] = str(new_path / '%(title)s.%(ext)s')
|
||||
|
|
@ -348,11 +342,6 @@ class DownloadOrchestrator:
|
|||
if not chain:
|
||||
logger.warning("Hybrid search exhausted: no eligible sources after exclusion filter")
|
||||
return [], []
|
||||
if load_search_mode() == 'best_quality':
|
||||
logger.info(f"Best-quality search ({' → '.join(chain)}): {query}")
|
||||
return await self.engine.search_all_sources(
|
||||
query, chain, timeout, progress_callback,
|
||||
)
|
||||
logger.info(f"Hybrid search ({' → '.join(chain)}): {query}")
|
||||
return await self.engine.search_with_fallback(query, chain, timeout, progress_callback)
|
||||
|
||||
|
|
@ -423,17 +412,9 @@ class DownloadOrchestrator:
|
|||
|
||||
if scored:
|
||||
scored.sort(key=lambda x: x._match_confidence, reverse=True)
|
||||
# Match filter done (right track); now prefer the best quality
|
||||
# among the confidence-passing survivors so streaming isn't
|
||||
# quality-blind like Soulseek already isn't. Stable ranking
|
||||
# keeps confidence order within an equal quality tier; the
|
||||
# `or scored` fail-safe never leaves us with nothing to try.
|
||||
from core.quality.selection import rank_for_profile
|
||||
ranked, _ = rank_for_profile(scored)
|
||||
filtered_results = ranked or scored
|
||||
filtered_results = scored
|
||||
logger.info(f"Streaming validation: {len(scored)}/{len(tracks)} passed "
|
||||
f"(best: {scored[0]._match_confidence:.2f}, "
|
||||
f"quality pick: {filtered_results[0].audio_quality.label()})")
|
||||
f"(best: {scored[0]._match_confidence:.2f})")
|
||||
else:
|
||||
logger.warning(f"No streaming results passed validation for: {query}")
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -13,11 +13,10 @@ import from a neutral package per Cin's contract-first standard.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from core.imports.filename import parse_filename_metadata
|
||||
from core.quality.model import AudioQuality
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -33,36 +32,6 @@ class SearchResult:
|
|||
upload_speed: int
|
||||
queue_length: int
|
||||
result_type: str = "track" # "track" or "album"
|
||||
# Rich quality metadata — populated by sources that provide it.
|
||||
# None means "unknown", not "absent".
|
||||
sample_rate: Optional[int] = None # Hz (e.g. 44100, 96000, 192000)
|
||||
bit_depth: Optional[int] = None # bits per sample (16, 24)
|
||||
|
||||
@property
|
||||
def audio_quality(self) -> AudioQuality:
|
||||
"""Unified quality descriptor derived from this result's fields."""
|
||||
return AudioQuality(
|
||||
format=self.quality.lower() if self.quality else 'unknown',
|
||||
bitrate=self.bitrate,
|
||||
sample_rate=self.sample_rate,
|
||||
bit_depth=self.bit_depth,
|
||||
)
|
||||
|
||||
def set_quality(self, aq: AudioQuality) -> None:
|
||||
"""Merge a mapped :class:`AudioQuality` onto this result's fields.
|
||||
|
||||
Used by streaming sources to stamp their claimed tier (Tidal/HiFi
|
||||
tier strings, Qobuz API values, …) so ``audio_quality`` ranks
|
||||
correctly. Mapper-provided fields win; a ``None`` from the mapper
|
||||
leaves any already-reported value (e.g. a probed bitrate) intact.
|
||||
"""
|
||||
self.quality = aq.format
|
||||
if aq.bitrate is not None:
|
||||
self.bitrate = aq.bitrate
|
||||
if aq.sample_rate is not None:
|
||||
self.sample_rate = aq.sample_rate
|
||||
if aq.bit_depth is not None:
|
||||
self.bit_depth = aq.bit_depth
|
||||
|
||||
@property
|
||||
def quality_score(self) -> float:
|
||||
|
|
@ -158,19 +127,6 @@ class AlbumResult:
|
|||
queue_length: int = 0
|
||||
result_type: str = "album"
|
||||
|
||||
@property
|
||||
def audio_quality(self) -> AudioQuality:
|
||||
"""Unified quality descriptor derived from dominant track quality."""
|
||||
sample_rates = [t.sample_rate for t in self.tracks if t.sample_rate]
|
||||
bit_depths = [t.bit_depth for t in self.tracks if t.bit_depth]
|
||||
bitrates = [t.bitrate for t in self.tracks if t.bitrate]
|
||||
return AudioQuality(
|
||||
format=self.dominant_quality.lower() if self.dominant_quality else 'unknown',
|
||||
bitrate=max(bitrates) if bitrates else None,
|
||||
sample_rate=max(sample_rates) if sample_rates else None,
|
||||
bit_depth=max(bit_depths) if bit_depths else None,
|
||||
)
|
||||
|
||||
@property
|
||||
def quality_score(self) -> float:
|
||||
"""Calculate album quality score based on dominant quality and track count"""
|
||||
|
|
|
|||
|
|
@ -83,26 +83,9 @@ def clear_completed_local() -> int:
|
|||
"""
|
||||
cleared = 0
|
||||
with tasks_lock:
|
||||
# Protect tasks belonging to a still-active batch. A batch is "active"
|
||||
# while any of its queued tasks is non-terminal (still searching /
|
||||
# downloading / queued / post-processing). Pruning a batch's completed
|
||||
# or failed tasks mid-run would yank them out of the Downloads page —
|
||||
# and failed/cancelled rows aren't recoverable from library_history —
|
||||
# so the user would never see them until the batch ended. Keep the whole
|
||||
# active batch intact; it gets cleaned by a later run once it finishes.
|
||||
protected_task_ids: set = set()
|
||||
for batch in download_batches.values():
|
||||
queue = batch.get('queue', []) if isinstance(batch, dict) else []
|
||||
batch_active = any(
|
||||
download_tasks.get(tid, {}).get('status') not in _TERMINAL_STATUSES
|
||||
for tid in queue if tid in download_tasks
|
||||
)
|
||||
if batch_active:
|
||||
protected_task_ids.update(queue)
|
||||
|
||||
task_ids_to_remove = [
|
||||
tid for tid, task in download_tasks.items()
|
||||
if task.get('status') in _TERMINAL_STATUSES and tid not in protected_task_ids
|
||||
if task.get('status') in _TERMINAL_STATUSES
|
||||
]
|
||||
for tid in task_ids_to_remove:
|
||||
del download_tasks[tid]
|
||||
|
|
|
|||
|
|
@ -47,55 +47,6 @@ from core.runtime_state import (
|
|||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _priority_sort_key(r):
|
||||
"""Today's confidence-first key: never download a high-quality WRONG file."""
|
||||
return (
|
||||
getattr(r, 'confidence', 0) or 0,
|
||||
getattr(r, 'quality_score', 0) or 0,
|
||||
getattr(r, 'upload_speed', 0) or 0,
|
||||
-(getattr(r, 'queue_length', 0) or 0),
|
||||
getattr(r, 'free_upload_slots', 0) or 0,
|
||||
getattr(r, 'size', 0) or 0,
|
||||
)
|
||||
|
||||
|
||||
def _quality_first_sort_key(r, targets):
|
||||
"""Best-quality key: the user's profile quality rank dominates; all the
|
||||
priority-mode signals (confidence, speed, …) become tiebreakers.
|
||||
|
||||
Every candidate reaching this point already passed match filtering, so it
|
||||
is "correct enough" — ordering by quality among correct candidates is safe.
|
||||
Candidates with no usable quality info, or that match no target, sort last
|
||||
(never dropped). Lower target index = better target, so it's negated to fit
|
||||
the descending (reverse=True) sort.
|
||||
"""
|
||||
from core.quality.model import rank_candidate
|
||||
|
||||
aq = getattr(r, 'audio_quality', None)
|
||||
if aq is None or not targets:
|
||||
target_idx, tier = (len(targets) if targets else 0), 0.0
|
||||
else:
|
||||
try:
|
||||
target_idx, tier = rank_candidate(aq, targets)
|
||||
except Exception:
|
||||
target_idx, tier = len(targets), 0.0
|
||||
return (-target_idx, tier) + _priority_sort_key(r)
|
||||
|
||||
|
||||
def order_candidates(candidates, *, quality_first=False, targets=None):
|
||||
"""Return *candidates* ordered best-first for the download walk.
|
||||
|
||||
``quality_first=False`` (priority mode) → confidence-first, byte-for-byte
|
||||
today's behaviour. ``quality_first=True`` (best-quality mode) → the user's
|
||||
profile quality rank dominates, confidence/peer signals break ties.
|
||||
"""
|
||||
if quality_first:
|
||||
key = lambda r: _quality_first_sort_key(r, targets or [])
|
||||
else:
|
||||
key = _priority_sort_key
|
||||
return sorted(candidates, key=key, reverse=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CandidatesDeps:
|
||||
"""Bundle of cross-cutting deps the candidate-fallback logic needs."""
|
||||
|
|
@ -108,25 +59,25 @@ class CandidatesDeps:
|
|||
on_download_completed: Callable
|
||||
|
||||
|
||||
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None,
|
||||
deps: CandidatesDeps = None, *,
|
||||
quality_first=False, quality_targets=None):
|
||||
def attempt_download_with_candidates(task_id, candidates, track, batch_id=None, deps: CandidatesDeps = None):
|
||||
"""
|
||||
Attempts to download with fallback candidate logic (matches GUI's retry_parallel_download_with_fallback).
|
||||
Returns True if successful, False if all candidates fail.
|
||||
|
||||
``quality_first`` (best-quality search mode) orders the walk by the user's
|
||||
profile quality rank instead of confidence-first; ``quality_targets`` is the
|
||||
profile target list used for that ranking. Defaults preserve priority-mode
|
||||
behaviour exactly.
|
||||
"""
|
||||
# Sort candidates. Priority mode: confidence-first, then peer quality —
|
||||
# upstream Soulseek validation already considers peer speed/slots/queue when
|
||||
# scores are close; preserve that signal instead of flattening ties back to
|
||||
# arbitrary slskd response order. Best-quality mode: profile quality rank
|
||||
# dominates (all candidates here already passed match filtering).
|
||||
candidates = order_candidates(
|
||||
candidates, quality_first=quality_first, targets=quality_targets,
|
||||
# Sort candidates by match confidence first, then peer quality. Upstream
|
||||
# Soulseek validation already considers peer speed/slots/queue when scores
|
||||
# are close; preserve that signal here instead of flattening ties back to
|
||||
# arbitrary slskd response order.
|
||||
candidates.sort(
|
||||
key=lambda r: (
|
||||
getattr(r, 'confidence', 0) or 0,
|
||||
getattr(r, 'quality_score', 0) or 0,
|
||||
getattr(r, 'upload_speed', 0) or 0,
|
||||
-(getattr(r, 'queue_length', 0) or 0),
|
||||
getattr(r, 'free_upload_slots', 0) or 0,
|
||||
getattr(r, 'size', 0) or 0,
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
with tasks_lock:
|
||||
|
|
|
|||
|
|
@ -77,16 +77,7 @@ def _normalize_for_finding(text: str) -> str:
|
|||
return ""
|
||||
text = unidecode(text).lower()
|
||||
text = re.sub(r'[._/]', ' ', text)
|
||||
# Strip ONLY balanced bracket pairs (tags like "[FLAC]", "(Remastered 2016)").
|
||||
# The old combined pattern r'[\[\(].*?[\]\)]' allowed MISMATCHED delimiters, so a
|
||||
# lone unbalanced '[' — slskd reports "[34 - You & Me (Flume Remix)" but saves the
|
||||
# file as "34 - You & Me (Flume Remix)" — matched from that '[' all the way to the
|
||||
# next ')', eating the entire title and collapsing the search target to "flac". The
|
||||
# file then scored 0.40 against the real on-disk name and was reported "not found"
|
||||
# despite sitting right there. Per-delimiter pairs can't over-consume; a stray
|
||||
# unbalanced bracket simply survives to the alphanumeric strip below.
|
||||
text = re.sub(r'\[[^\]]*\]', '', text)
|
||||
text = re.sub(r'\([^)]*\)', '', text)
|
||||
text = re.sub(r'[\[\(].*?[\]\)]', '', text)
|
||||
text = re.sub(r'[^a-z0-9\s-]', '', text)
|
||||
return ' '.join(text.split()).strip()
|
||||
|
||||
|
|
|
|||
|
|
@ -1,62 +0,0 @@
|
|||
"""Match a file back to its download-history row when its path has drifted (#934).
|
||||
|
||||
``library_history.file_path`` is frozen at import time, but the file moves afterward
|
||||
(media-server import, library reorganize) and ``tracks.file_path`` — what the AcoustID
|
||||
scanner reads — no longer equals it. Matching on the exact path alone then fails twice:
|
||||
the verification status never reaches the history row (verified tracks read "unverified"),
|
||||
and a fresh ``acoustid_scan`` row gets inserted every run (thousands of duplicates).
|
||||
|
||||
This module picks the canonical history row by exact path first, then by FILENAME guarded
|
||||
by a title check — so a shared filename ("01 - Intro.flac") can never heal the wrong song.
|
||||
Pure (no DB) so the matching rules are unit-testable; the caller does the SQL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Iterable, Optional, Sequence, Tuple
|
||||
|
||||
|
||||
def _norm_title(value) -> str:
|
||||
"""Alphanumeric-only lowercase form, so "Song (Remaster)" vs "song remaster"
|
||||
style drift between the download tag and the media-server tag still agrees."""
|
||||
return ''.join(ch for ch in str(value or '').lower() if ch.isalnum())
|
||||
|
||||
|
||||
def like_filename_filter(basename: str) -> str:
|
||||
"""A ``LIKE ... ESCAPE '\\'`` pattern that coarsely matches rows whose path ends
|
||||
in ``basename``. Escapes the LIKE metacharacters (``%`` ``_`` ``\\``) — filenames
|
||||
routinely contain underscores. Callers MUST still confirm with an exact basename
|
||||
compare (``pick_history_row`` does), since ``'%name'`` also matches ``'xname'``."""
|
||||
esc = basename.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_')
|
||||
return '%' + esc
|
||||
|
||||
|
||||
def pick_history_row(candidates: Sequence[Tuple], *, current_paths: Iterable[str],
|
||||
basename: str, title: str) -> Optional[int]:
|
||||
"""Return the id of the history row to update for this file, or None.
|
||||
|
||||
``candidates``: ``(id, file_path, title, download_source)`` rows the DB pre-filtered
|
||||
(exact path or filename LIKE). A row matches when its path equals the current path OR
|
||||
its filename matches AND its title agrees — the title guard prevents a shared filename
|
||||
("01 - Intro.flac") from healing a different song's row. Among matches a REAL download
|
||||
row is preferred over a synthetic ``acoustid_scan`` row, so the scanner heals the
|
||||
genuine record and the caller can delete the synthetic duplicate. None when nothing
|
||||
matches safely (caller then inserts a fresh row — the "file SoulSync never downloaded"
|
||||
intent)."""
|
||||
paths = {p for p in current_paths if p}
|
||||
want = _norm_title(title)
|
||||
matches: list = [] # (id, is_exact, is_real)
|
||||
for cid, cpath, ctitle, csource in candidates:
|
||||
is_real = csource != 'acoustid_scan'
|
||||
if cpath and cpath in paths:
|
||||
matches.append((cid, True, is_real))
|
||||
elif (basename and cpath and os.path.basename(cpath) == basename
|
||||
and (not want or not _norm_title(ctitle) or _norm_title(ctitle) == want)):
|
||||
matches.append((cid, False, is_real))
|
||||
if not matches:
|
||||
return None
|
||||
# Prefer a REAL download row over a synthetic acoustid_scan row; within that, prefer an
|
||||
# exact-path match over a filename match. Stable, so ties keep DB order (first/oldest id).
|
||||
matches.sort(key=lambda m: (m[2], m[1]), reverse=True)
|
||||
return matches[0][0]
|
||||
|
|
@ -26,7 +26,6 @@ Lifted verbatim from web_server.py. Dependencies injected via
|
|||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
import traceback
|
||||
|
|
@ -45,27 +44,6 @@ from core.runtime_state import (
|
|||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# A task that has been in 'post_processing' longer than this is treated as stuck.
|
||||
# Post-processing (AcoustID + quality + import) is serialized, so a large batch
|
||||
# legitimately backs up — keep this generous so genuinely-slow imports aren't
|
||||
# cut off mid-flight (the old 5-min cutoff falsely "completed" queued tasks).
|
||||
_POST_PROCESSING_STUCK_TIMEOUT = 1800 # 30 minutes
|
||||
|
||||
|
||||
def _resolve_stuck_post_processing_status(task: dict) -> str:
|
||||
"""Decide the terminal status for a task stuck in post_processing.
|
||||
|
||||
Only call it 'completed' if the import actually produced a file on disk
|
||||
(``final_file_path`` is set at the end of successful post-processing). Without
|
||||
a real file, force-completing is a lie — the task shows as a downloaded track
|
||||
that isn't anywhere. Mark those 'failed' so they're retryable and honest.
|
||||
"""
|
||||
final_path = task.get('final_file_path')
|
||||
if final_path and os.path.exists(final_path):
|
||||
return 'completed'
|
||||
return 'failed'
|
||||
|
||||
|
||||
def _safe_batch_dirname(batch_id: str) -> str:
|
||||
safe = ''.join(ch if ch.isalnum() or ch in ('-', '_') else '_' for ch in str(batch_id or 'batch'))
|
||||
return safe or 'batch'
|
||||
|
|
@ -457,15 +435,9 @@ def on_download_completed(batch_id: str, task_id: str, success: bool, deps: Life
|
|||
retrying_count += 1
|
||||
elif task_status == 'post_processing':
|
||||
task_age = current_time - task.get('status_change_time', current_time)
|
||||
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
|
||||
new_status = _resolve_stuck_post_processing_status(task)
|
||||
if new_status == 'completed':
|
||||
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
|
||||
task['status'] = 'completed'
|
||||
else:
|
||||
logger.warning(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
|
||||
task['status'] = 'failed'
|
||||
task['error_message'] = 'Post-processing timed out without producing a file'
|
||||
if task_age > 300: # 5 minutes (post-processing should be fast)
|
||||
logger.info(f"⏰ [Stuck Detection] Task {queue_task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
|
||||
task['status'] = 'completed' # Assume it worked if file verification is taking too long
|
||||
finished_count += 1
|
||||
else:
|
||||
retrying_count += 1
|
||||
|
|
@ -695,15 +667,9 @@ def check_batch_completion_v2(batch_id: str, deps: LifecycleDeps) -> Optional[bo
|
|||
retrying_count += 1
|
||||
elif task_status == 'post_processing':
|
||||
task_age = current_time - task.get('status_change_time', current_time)
|
||||
if task_age > _POST_PROCESSING_STUCK_TIMEOUT:
|
||||
new_status = _resolve_stuck_post_processing_status(task)
|
||||
if new_status == 'completed':
|
||||
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s but file exists — completing")
|
||||
task['status'] = 'completed'
|
||||
else:
|
||||
logger.warning(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s with no output file — marking failed")
|
||||
task['status'] = 'failed'
|
||||
task['error_message'] = 'Post-processing timed out without producing a file'
|
||||
if task_age > 300: # 5 minutes (post-processing should be fast)
|
||||
logger.info(f"⏰ [Stuck Detection V2] Task {task_id} stuck in post_processing for {task_age:.0f}s - forcing completion")
|
||||
task['status'] = 'completed' # Assume it worked if file verification is taking too long
|
||||
finished_count += 1
|
||||
else:
|
||||
retrying_count += 1
|
||||
|
|
|
|||
|
|
@ -760,8 +760,7 @@ class WebUIDownloadMonitor:
|
|||
# used_sources keys are formatted as "{username}_{filename}", so startswith is exact.
|
||||
is_tidal = any(s.startswith('tidal_') for s in tried_sources)
|
||||
if is_tidal:
|
||||
from core.quality.source_map import quality_tier_for_source
|
||||
tidal_quality = quality_tier_for_source('tidal', default='lossless')
|
||||
tidal_quality = config_manager.get('tidal_download.quality', 'lossless')
|
||||
allow_fb = config_manager.get('tidal_download.allow_fallback', True)
|
||||
if tidal_quality == 'hires' and not allow_fb:
|
||||
task['error_message'] = (
|
||||
|
|
|
|||
|
|
@ -1,55 +0,0 @@
|
|||
"""Identify dead review-queue history rows whose file is gone (#934 follow-up).
|
||||
|
||||
The Unverified/Quarantine review queue is fed from ``library_history`` — an
|
||||
append-only log that is never pruned. When a file is deleted, replaced, or
|
||||
re-downloaded elsewhere, its old ``unverified`` row lingers forever and can
|
||||
never be healed (there's no file left to confirm). Those are *orphans*.
|
||||
|
||||
This decides which rows are orphans, given a ``resolve(row) -> path | None``
|
||||
the caller wires to the real filesystem lookup. Pure (no DB, no filesystem) so
|
||||
the rules — including the safety gate — are unit-testable.
|
||||
|
||||
Safety gate: a filesystem check mass-false-positives when the library mount is
|
||||
down (every file looks missing). So if EVERY reviewed file is unreachable and
|
||||
there are enough rows to judge, we flag it ``suspicious`` and the caller refuses
|
||||
to delete — better to clean nothing than to wipe a healthy log during an outage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable, Sequence
|
||||
|
||||
|
||||
def find_orphan_history_ids(
|
||||
rows: Sequence[dict],
|
||||
resolve: Callable[[dict], Any],
|
||||
*,
|
||||
min_for_safety: int = 5,
|
||||
deletable: Callable[[dict], bool] | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{'orphan_ids', 'checked', 'suspicious'}``.
|
||||
|
||||
A row is an orphan when it has a non-empty ``file_path`` but ``resolve`` can
|
||||
find no file for it. ``suspicious`` is True when every checked row is
|
||||
missing and there are at least ``min_for_safety`` of them — the mount-down
|
||||
signature; the caller should refuse to delete in that case.
|
||||
|
||||
``deletable`` (optional) protects rows from removal WITHOUT weakening the
|
||||
safety gate: a protected row still counts toward ``checked`` and the
|
||||
all-missing signal (so e.g. a few unverified orphans can't be swept during a
|
||||
mount outage just because protected rows were filtered out first), but it
|
||||
never appears in ``orphan_ids``. Default: every missing row is deletable.
|
||||
"""
|
||||
orphan_ids = []
|
||||
checked = 0
|
||||
missing = 0
|
||||
for row in rows:
|
||||
if not str((row.get('file_path') or '')).strip():
|
||||
continue
|
||||
checked += 1
|
||||
if resolve(row) is None:
|
||||
missing += 1
|
||||
if deletable is None or deletable(row):
|
||||
orphan_ids.append(row.get('id'))
|
||||
suspicious = checked >= min_for_safety and missing == checked
|
||||
return {'orphan_ids': orphan_ids, 'checked': checked, 'suspicious': suspicious}
|
||||
|
|
@ -171,21 +171,6 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
|
|||
logger.info(f"[Post-Processing] Task {task_id} already completed by stream processor, skipping verification")
|
||||
return
|
||||
|
||||
# RACE GUARD: the monitor sets status -> 'post_processing' immediately
|
||||
# before submitting this worker. If the status is now anything else, the
|
||||
# browser-poll post-processor already took ownership of this task — e.g.
|
||||
# it quarantined the file and requeued the next-best candidate (status
|
||||
# -> 'searching', source identity cleared). Bail WITHOUT marking failed
|
||||
# or notifying batch completion: otherwise we clobber that in-flight
|
||||
# retry with a bogus "missing file or source information" failure while a
|
||||
# parallel attempt is importing the song.
|
||||
if task['status'] != 'post_processing':
|
||||
logger.info(
|
||||
f"[Post-Processing] Task {task_id} no longer in 'post_processing' "
|
||||
f"(now '{task['status']}') — another path took over, skipping"
|
||||
)
|
||||
return
|
||||
|
||||
# Extract file information for verification
|
||||
track_info = task.get('track_info', {})
|
||||
task_filename = task.get('filename') or track_info.get('filename')
|
||||
|
|
@ -453,19 +438,7 @@ def run_post_processing_worker(task_id: str, batch_id: str, deps: PostProcessDep
|
|||
logger.error(f"[Post-Processing] Task {task_id} was completed by stream processor - not marking as failed")
|
||||
return
|
||||
download_tasks[task_id]['status'] = 'failed'
|
||||
# slskd reported the transfer complete, but the finder never located
|
||||
# the file under the configured download folder. Name the folder we
|
||||
# searched and the two real causes — "still being written" (timing)
|
||||
# or "SoulSync's download path doesn't match slskd's" (the classic
|
||||
# standalone config mismatch) — so the user can self-diagnose instead
|
||||
# of getting an opaque "not found". (Discord: Shdjfgatdif.)
|
||||
_searched_name = os.path.basename((task_filename or '').replace('\\', '/')) or task_filename
|
||||
download_tasks[task_id]['error_message'] = (
|
||||
f"slskd reported '{_searched_name}' downloaded, but it never appeared "
|
||||
f"under the download folder ({download_dir}) after {_file_search_max_retries} "
|
||||
f"checks. Either it's still being written, or SoulSync's download path "
|
||||
f"doesn't match slskd's download directory — they must point at the same folder."
|
||||
)
|
||||
download_tasks[task_id]['error_message'] = f'File not found on disk after {_file_search_max_retries} search attempts. Expected: {os.path.basename(task_filename)}'
|
||||
deps.on_download_completed(batch_id, task_id, False)
|
||||
return
|
||||
|
||||
|
|
|
|||
|
|
@ -94,10 +94,6 @@ class StatusDeps:
|
|||
run_async: Optional[Callable] = None
|
||||
on_download_completed: Optional[Callable[[str, str, bool], None]] = None
|
||||
get_persistent_download_history: Optional[Callable[[int], list[dict]]] = None
|
||||
# Returns ALL library_history rows with verification_status in
|
||||
# ('unverified', 'force_imported') — no recency limit, so historical
|
||||
# entries are never buried by the general history tail cap.
|
||||
get_unverified_download_history: Optional[Callable[[], list[dict]]] = None
|
||||
|
||||
|
||||
# Streaming sources the engine fallback applies to. Soulseek goes through
|
||||
|
|
@ -800,13 +796,6 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
|
|||
'progress': progress,
|
||||
'error': task.get('error_message'),
|
||||
'verification_status': task.get('verification_status'),
|
||||
# library_history row id (set at import) so the Unverified review
|
||||
# queue can act on a still-live completed task before it becomes
|
||||
# a persistent-history row.
|
||||
'history_id': task.get('history_id'),
|
||||
# Real probed audio quality (mutagen-read from the actual file),
|
||||
# surfaced so the Downloads page can show what was downloaded.
|
||||
'quality': task.get('quality') or '',
|
||||
'retry_info': task.get('retry_info'),
|
||||
'retry_trigger': task.get('retry_trigger'),
|
||||
'batch_id': batch_id,
|
||||
|
|
@ -823,32 +812,6 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
|
|||
'is_persistent_history': False,
|
||||
})
|
||||
|
||||
# --- Unverified history (unconditional, no limit) ---
|
||||
# Always load every library_history row that still needs human confirmation
|
||||
# (verification_status IN ('unverified', 'force_imported')). This is NOT
|
||||
# gated on len(items) < limit so that historical entries from past batches
|
||||
# are visible even during a large active batch that would otherwise exhaust
|
||||
# the limit before the history tail is read. Dedup against live tasks by
|
||||
# identity so a track currently in post-processing isn't shown twice.
|
||||
if deps.get_unverified_download_history is not None:
|
||||
try:
|
||||
unverified_entries = deps.get_unverified_download_history() or []
|
||||
except Exception as exc:
|
||||
logger.debug("[Downloads] unverified history lookup failed: %s", exc)
|
||||
unverified_entries = []
|
||||
|
||||
for entry in unverified_entries:
|
||||
item = _build_history_download_item(entry)
|
||||
identity = _download_identity(item.get('title'), item.get('artist'), item.get('album'))
|
||||
if identity in live_identities:
|
||||
continue
|
||||
items.append(item)
|
||||
live_identities.add(identity)
|
||||
|
||||
# --- General recent-history tail (capped, recency-ordered) ---
|
||||
# Fills in the completed/verified tail so the full Downloads list looks
|
||||
# populated. Gated on len(items) < limit so a busy batch doesn't trigger
|
||||
# an extra DB round-trip when we're already at capacity.
|
||||
if deps.get_persistent_download_history is not None and len(items) < limit:
|
||||
history_limit = min(limit - len(items), _PERSISTENT_HISTORY_TAIL_LIMIT)
|
||||
try:
|
||||
|
|
@ -869,14 +832,7 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
|
|||
live_identities.add(identity)
|
||||
appended_history += 1
|
||||
|
||||
# Sort: active first (by priority), then by timestamp desc within each group.
|
||||
# NOTE: the array order is presentation-only — the Downloads page filters
|
||||
# client-side per tab. What matters is that EVERY live task is present: an
|
||||
# earlier `items[:limit]` truncation (active-first) starved completed/failed/
|
||||
# unverified rows off the end during a busy batch, so those tabs stayed empty
|
||||
# until the batch drained. `limit` now bounds only the persistent-history
|
||||
# tail (handled above); live in-memory tasks are always returned in full
|
||||
# (they're already bounded by the 5-min cleanup automation).
|
||||
# Sort: active first (by priority), then by timestamp desc within each group
|
||||
items.sort(key=lambda x: (x['priority'], -x['timestamp']))
|
||||
|
||||
# Build batch summaries for the batch context panel
|
||||
|
|
@ -904,7 +860,7 @@ def build_unified_downloads_response(limit: int, deps: StatusDeps) -> dict:
|
|||
|
||||
return {
|
||||
'success': True,
|
||||
'downloads': items,
|
||||
'downloads': items[:limit],
|
||||
'total': len(items),
|
||||
'batches': batch_summaries,
|
||||
'timestamp': time.time(),
|
||||
|
|
|
|||
|
|
@ -54,37 +54,6 @@ def _cand_user_file(candidate):
|
|||
return getattr(candidate, 'username', None), getattr(candidate, 'filename', None)
|
||||
|
||||
|
||||
def _candidate_ordering():
|
||||
"""Return ``(quality_first, targets)`` for the active search mode + toggle.
|
||||
|
||||
The candidate walk is ordered by the user's profile quality rank
|
||||
(best→worst) instead of confidence-first when EITHER:
|
||||
- best-quality search mode is active (always quality-first), OR
|
||||
- priority mode and the ``rank_candidates_by_quality`` toggle is on
|
||||
(opt-in; default off keeps the byte-for-byte confidence-first walk).
|
||||
|
||||
Quality-first ordering also makes the version-mismatch force-import pick
|
||||
the highest-quality candidate, because that fallback accepts the
|
||||
first-tried (= best-ordered) quarantined entry.
|
||||
|
||||
Fails closed to confidence-first ordering on any error so a profile/DB
|
||||
hiccup never blocks a download. See
|
||||
docs/superpowers/specs/2026-06-14-best-quality-search-mode-design.md.
|
||||
"""
|
||||
try:
|
||||
from core.quality.selection import (
|
||||
load_search_mode,
|
||||
load_profile_targets,
|
||||
load_rank_candidates_by_quality,
|
||||
)
|
||||
if load_search_mode() == 'best_quality' or load_rank_candidates_by_quality():
|
||||
targets, _ = load_profile_targets()
|
||||
return True, targets
|
||||
except Exception as exc:
|
||||
logger.debug("[Modal Worker] quality ordering unavailable: %s", exc)
|
||||
return False, None
|
||||
|
||||
|
||||
def _try_cached_candidates(task_id, batch_id, track, deps):
|
||||
"""Quarantine-retry fast path: attempt the already-found candidates before
|
||||
re-searching anything.
|
||||
|
|
@ -122,10 +91,7 @@ def _try_cached_candidates(task_id, batch_id, track, deps):
|
|||
f"[Modal Worker] Quarantine retry: trying {len(remaining)} cached "
|
||||
f"candidate(s) before re-searching (task {task_id})"
|
||||
)
|
||||
_qf, _qt = _candidate_ordering()
|
||||
return deps.attempt_download_with_candidates(
|
||||
task_id, remaining, track, batch_id, quality_first=_qf, quality_targets=_qt,
|
||||
)
|
||||
return deps.attempt_download_with_candidates(task_id, remaining, track, batch_id)
|
||||
|
||||
|
||||
def _private_album_bundle_staging_miss_reason(batch_id: Optional[str], deps: Any) -> Optional[str]:
|
||||
|
|
@ -403,11 +369,6 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
|
|||
logger.info(f"[Modal Worker] Generated {len(search_queries)} smart search queries for '{track.name}': {search_queries}")
|
||||
logger.info(f"[Modal Worker] About to start search loop for task {task_id} (track: '{track.name}')")
|
||||
|
||||
# Best-quality search mode: the orchestrator already pooled candidates
|
||||
# across every source for each query, so order the candidate walk by the
|
||||
# user's profile quality rank (best→worst). Computed once per task.
|
||||
_best_quality, _quality_targets = _candidate_ordering()
|
||||
|
||||
# 2. Sequential Query Search (matches GUI's start_search_worker_parallel logic)
|
||||
search_diagnostics = [] # Track what happened per query for detailed error messages
|
||||
all_raw_results = [] # Collect raw results across queries for candidate review modal
|
||||
|
|
@ -534,10 +495,7 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
|
|||
download_tasks[task_id]['cached_candidates'] = candidates
|
||||
|
||||
# Try to download with these candidates
|
||||
success = deps.attempt_download_with_candidates(
|
||||
task_id, candidates, track, batch_id,
|
||||
quality_first=_best_quality, quality_targets=_quality_targets,
|
||||
)
|
||||
success = deps.attempt_download_with_candidates(task_id, candidates, track, batch_id)
|
||||
if success:
|
||||
# Download initiated successfully - let the download monitoring system handle completion
|
||||
if batch_id:
|
||||
|
|
@ -571,10 +529,7 @@ def download_track_worker(task_id: str, batch_id: Optional[str], deps: TaskWorke
|
|||
# === HYBRID FALLBACK: If primary source failed, try remaining sources directly ===
|
||||
# The orchestrator's hybrid search stops at the first source with results, even if
|
||||
# those results all fail quality filtering. Try remaining sources individually.
|
||||
#
|
||||
# Best-quality mode already searched EVERY source per query (the pool), so this
|
||||
# block would only re-search the same sources — skip it there.
|
||||
if not _best_quality and getattr(deps.download_orchestrator, 'mode', '') == 'hybrid':
|
||||
if getattr(deps.download_orchestrator, 'mode', '') == 'hybrid':
|
||||
try:
|
||||
orch = deps.download_orchestrator
|
||||
hybrid_order = getattr(orch, 'hybrid_order', None) or []
|
||||
|
|
|
|||
|
|
@ -13,48 +13,9 @@ Pure + import-safe: parsing only, no network.
|
|||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, List, Optional, Tuple
|
||||
from typing import Any, Optional, Tuple
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
def linked_track_id(track: Any) -> str:
|
||||
"""The source track id stamped on a search result, read from
|
||||
``_source_metadata['track_id']`` — the field every ID-downloadable source
|
||||
(Tidal, Qobuz) records. Empty string when absent. ``TrackResult`` has no
|
||||
top-level ``id``, so callers must NOT use ``getattr(t, 'id')`` (that always
|
||||
missed and left the pasted-link bubble a silent no-op — #932)."""
|
||||
meta = getattr(track, '_source_metadata', None)
|
||||
if not isinstance(meta, dict):
|
||||
return ''
|
||||
return str(meta.get('track_id') or '')
|
||||
|
||||
|
||||
def bubble_linked_track_first(tracks: List[Any], link_track_id: str) -> List[Any]:
|
||||
"""Float the result whose source id matches a pasted link to the top so the
|
||||
user sees the EXACT track they linked, not a fuzzy text-search lookalike
|
||||
(#813/#932). Stable + a graceful no-op when no result carries the id."""
|
||||
if not link_track_id or not tracks:
|
||||
return tracks
|
||||
target = str(link_track_id)
|
||||
return sorted(tracks, key=lambda t: linked_track_id(t) != target)
|
||||
|
||||
|
||||
def inject_linked_track_first(
|
||||
tracks: List[Any], linked_result: Any, link_track_id: str
|
||||
) -> List[Any]:
|
||||
"""Put the EXACT linked track first.
|
||||
|
||||
When ``linked_result`` is the track fetched directly by id, prepend it and
|
||||
drop any search duplicate of it — so an obscure track a text search never
|
||||
surfaced is still present and downloadable (#932). When it's None (the source
|
||||
can't fetch one), fall back to bubbling a matching search result. Pure."""
|
||||
if not link_track_id:
|
||||
return tracks
|
||||
target = str(link_track_id)
|
||||
if linked_result is not None:
|
||||
return [linked_result] + [t for t in tracks if linked_track_id(t) != target]
|
||||
return bubble_linked_track_first(tracks, target)
|
||||
|
||||
# host substring → download source id. Only ID-downloadable streaming sources.
|
||||
_HOSTS = (
|
||||
('tidal.com', 'tidal'),
|
||||
|
|
|
|||
|
|
@ -16,9 +16,8 @@ writes a fresh non-cache hit back to the cache so the next export of the same so
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||||
from typing import Callable, Optional, Tuple
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
|
|
@ -73,185 +72,6 @@ def db_recording_mbid(artist: str, title: str) -> Optional[str]:
|
|||
return _db_match(artist, title)[0]
|
||||
|
||||
|
||||
# Service → the tracks-table column carrying that service's track ID (set by enrichment).
|
||||
# Trusted constants — never user input — so safe to interpolate into the SELECT below.
|
||||
_SERVICE_ID_COLUMNS = {"spotify": "spotify_track_id", "deezer": "deezer_id"}
|
||||
|
||||
|
||||
def db_service_track_id(artist: str, title: str, service: str) -> Optional[str]:
|
||||
"""The service track ID (``spotify_track_id`` / ``deezer_id``) stored on a matched
|
||||
library track — what lets a mirrored playlist be exported BACK to Spotify/Deezer
|
||||
without re-searching, since enrichment already pinned it (#945). Text-matches by
|
||||
(artist, title), same as the MBID resolver. Fail-safe: any miss/error returns None."""
|
||||
column = _SERVICE_ID_COLUMNS.get((service or "").lower())
|
||||
if not column or not title:
|
||||
return None
|
||||
try:
|
||||
from database.music_database import get_database
|
||||
db = get_database()
|
||||
conn = db._get_connection()
|
||||
try:
|
||||
cur = conn.cursor()
|
||||
cur.execute(
|
||||
f"SELECT t.{column} FROM tracks t JOIN artists a ON t.artist_id = a.id "
|
||||
"WHERE LOWER(t.title) = LOWER(?) AND LOWER(a.name) = LOWER(?) LIMIT 1",
|
||||
(title, artist),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
return None
|
||||
val = row[0] if not hasattr(row, "keys") else row[column]
|
||||
return val or None
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception: # noqa: S110
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.debug(f"export service-id lookup failed for '{artist} - {title}' ({service}): {exc}")
|
||||
return None
|
||||
|
||||
|
||||
def build_service_resolve_fn(service: str) -> Callable[[str, str], Tuple[Optional[str], Optional[str]]]:
|
||||
"""resolve_fn for service-playlist export: ``(artist, title) -> (service_track_id, 'library')``.
|
||||
Plugs into ``resolve_playlist_tracks(..., id_key='service_track_id')`` exactly like the
|
||||
MBID resolver plugs in for ListenBrainz."""
|
||||
def resolve_fn(artist: str, title: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
tid = db_service_track_id(artist, title, service)
|
||||
return (tid, "library" if tid else None)
|
||||
return resolve_fn
|
||||
|
||||
|
||||
def service_id_from_extra_data(track: Any, service: str) -> Optional[str]:
|
||||
"""The target-service track ID the DISCOVERY step already resolved for this mirrored
|
||||
track, read from its ``extra_data`` blob (#945 — Boulder: "all 50 are discovered to
|
||||
Deezer already, it's not using any of that"). This is free (no API call) and reliable
|
||||
(it's the same id used to mirror the track).
|
||||
|
||||
Only trusted when the track was discovered ON the export's target service — a
|
||||
Deezer-discovered track carries a Deezer id under ``matched_data['id']``, and its
|
||||
``provider`` is the service name. A ``wing_it_fallback`` provider (the low-confidence
|
||||
guess path) deliberately does NOT match here, so those fall through to the library/
|
||||
none path rather than risk a wrong track in the exported playlist."""
|
||||
raw = track.get("extra_data") if isinstance(track, dict) else None
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(raw) if isinstance(raw, str) else raw
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(data, dict) or not data.get("discovered"):
|
||||
return None
|
||||
if str(data.get("provider") or "").lower() != str(service or "").lower():
|
||||
return None
|
||||
matched = data.get("matched_data")
|
||||
tid = matched.get("id") if isinstance(matched, dict) else None
|
||||
return str(tid) if tid else None
|
||||
|
||||
|
||||
def _track_field(track: Dict[str, Any], *names: str) -> str:
|
||||
for n in names:
|
||||
v = track.get(n)
|
||||
if v:
|
||||
return str(v)
|
||||
return ""
|
||||
|
||||
|
||||
def resolve_service_track_ids(
|
||||
tracks: List[Dict[str, Any]],
|
||||
service: str,
|
||||
*,
|
||||
db_fn: Optional[Callable[[str, str, str], Optional[str]]] = None,
|
||||
search_id_fn: Optional[Callable[[str, str], Optional[str]]] = None,
|
||||
on_progress: Optional[Callable[[int, int, Dict[str, Any]], None]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Resolve a mirrored playlist's tracks to target-service track IDs for export.
|
||||
|
||||
Waterfall per track: the discovery cache (``extra_data`` — free + already confidently
|
||||
matched) → the library track's stored service id → (only when ``search_id_fn`` is
|
||||
given, i.e. the opt-in backfill toggle) a confident live-search match. A track that
|
||||
clears none of these is reported unmatched (caller skips it — never a guessed/wrong
|
||||
id). Returns ``{"resolved": [{artist, title, album, service_track_id}], "stats":
|
||||
{...}}`` with ``from_cache`` / ``from_library`` / ``from_search`` / ``unmatched``
|
||||
tallies for the status display.
|
||||
"""
|
||||
db_fn = db_fn or db_service_track_id
|
||||
total = len(tracks or [])
|
||||
resolved: List[Dict[str, Any]] = []
|
||||
stats: Dict[str, Any] = {
|
||||
"total": total, "resolved": 0, "unmatched": 0,
|
||||
"from_cache": 0, "from_library": 0, "from_search": 0,
|
||||
}
|
||||
for i, t in enumerate(tracks or []):
|
||||
if not isinstance(t, dict):
|
||||
t = {}
|
||||
artist = _track_field(t, "artist", "artist_name", "creator")
|
||||
title = _track_field(t, "title", "track_name", "name")
|
||||
album = _track_field(t, "album", "album_name", "release_name")
|
||||
|
||||
tid = service_id_from_extra_data(t, service)
|
||||
if tid:
|
||||
stats["from_cache"] += 1
|
||||
else:
|
||||
tid = db_fn(artist, title, service)
|
||||
if tid:
|
||||
stats["from_library"] += 1
|
||||
elif search_id_fn is not None:
|
||||
tid = search_id_fn(artist, title)
|
||||
if tid:
|
||||
stats["from_search"] += 1
|
||||
|
||||
resolved.append({"artist": artist, "title": title, "album": album,
|
||||
"service_track_id": tid or None})
|
||||
stats["resolved" if tid else "unmatched"] += 1
|
||||
if on_progress is not None:
|
||||
try:
|
||||
on_progress(i + 1, total, stats)
|
||||
except Exception: # noqa: S110 — a progress error must never fail the export
|
||||
pass
|
||||
|
||||
return {"resolved": resolved, "stats": stats}
|
||||
|
||||
|
||||
# Confidence floor for backfill, on the score_track scale (~1.5 = exact title + exact
|
||||
# artist, thanks to the 1.5x artist boost). A cover/karaoke (x0.05) or a wrong-artist hit
|
||||
# (no boost, caps ~1.0) can't clear this — so backfill never adds a guessed/wrong version.
|
||||
BACKFILL_MIN_SCORE = 1.2
|
||||
|
||||
|
||||
def search_service_track_id(
|
||||
artist: str,
|
||||
title: str,
|
||||
*,
|
||||
search_fn: Callable[[str], List[Any]],
|
||||
min_score: float = BACKFILL_MIN_SCORE,
|
||||
) -> Optional[str]:
|
||||
"""Confident live-search match for export backfill (#945): search the target service
|
||||
for (artist, title), rerank by relevance, and return the top match's id ONLY if it
|
||||
clears the confidence floor. Below the floor → None: the track is left out of the
|
||||
export rather than risk a wrong/cover/karaoke version (the whole point of backfill is
|
||||
coverage WITHOUT the wrong-track risk). ``search_fn(query) -> List[Track]`` is injected
|
||||
so this is unit-testable without a live service."""
|
||||
if not title:
|
||||
return None
|
||||
from core.metadata.relevance import build_combined_search_query, filter_and_rerank
|
||||
query = build_combined_search_query(title, artist)
|
||||
try:
|
||||
candidates = list(search_fn(query) or [])
|
||||
except Exception as exc:
|
||||
logger.debug(f"export backfill search failed for '{artist} - {title}': {exc}")
|
||||
return None
|
||||
if not candidates:
|
||||
return None
|
||||
ranked = filter_and_rerank(
|
||||
candidates, expected_title=title, expected_artist=artist, min_score=min_score,
|
||||
)
|
||||
if not ranked:
|
||||
return None
|
||||
tid = getattr(ranked[0], "id", None)
|
||||
return str(tid) if tid else None
|
||||
|
||||
|
||||
def file_recording_mbid(artist: str, title: str) -> Optional[str]:
|
||||
"""Recording MBID read from the matched track's file tag (set on import post-processing)."""
|
||||
_mbid, fpath = _db_match(artist, title)
|
||||
|
|
|
|||
|
|
@ -36,22 +36,16 @@ def resolve_playlist_tracks(
|
|||
resolve_fn: ResolveFn,
|
||||
*,
|
||||
on_progress: Optional[ProgressFn] = None,
|
||||
id_key: str = "recording_mbid",
|
||||
) -> Dict[str, Any]:
|
||||
"""Resolve every track to an ID and build the export pseudo-playlist.
|
||||
|
||||
``resolve_fn(artist, title) -> (id, source)`` returns whatever ID the target needs —
|
||||
a MusicBrainz recording MBID for ListenBrainz/JSPF (the default), or a Spotify/Deezer
|
||||
track ID for service export. ``id_key`` names the field that ID lands under in each
|
||||
resolved entry (defaults to ``recording_mbid`` so existing LB/JSPF callers are
|
||||
untouched). The dedup + stats + ordering logic is identical regardless of ID type.
|
||||
"""Resolve every track to a recording MBID and build the export pseudo-playlist.
|
||||
|
||||
``tracks`` items may use ``artist``/``artist_name`` and ``title``/``track_name`` and
|
||||
``album``/``album_name`` (both the mirrored-playlist and LB-cache shapes are accepted).
|
||||
|
||||
Returns ``{"resolved": [...], "stats": {...}}`` where each resolved entry is
|
||||
``{artist, title, album, <id_key>}`` (the ID is None when unmatched), in original
|
||||
order, and stats carries ``total, resolved, unmatched, deduped, by_source``.
|
||||
``{artist, title, album, recording_mbid}`` (recording_mbid is None when unmatched),
|
||||
in original order, and stats carries ``total, resolved, unmatched, deduped,
|
||||
by_source`` for the live display.
|
||||
"""
|
||||
total = len(tracks or [])
|
||||
memo: Dict[str, Tuple[Optional[str], Optional[str]]] = {}
|
||||
|
|
@ -77,7 +71,7 @@ def resolve_playlist_tracks(
|
|||
memo[key] = (mbid, source)
|
||||
fresh = True
|
||||
|
||||
resolved.append({"artist": artist, "title": title, "album": album, id_key: mbid})
|
||||
resolved.append({"artist": artist, "title": title, "album": album, "recording_mbid": mbid})
|
||||
|
||||
if mbid:
|
||||
stats["resolved"] += 1
|
||||
|
|
|
|||
|
|
@ -36,34 +36,9 @@ import requests as http_requests
|
|||
from utils.logging_config import get_logger
|
||||
from config.settings import config_manager
|
||||
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
|
||||
from core.quality.source_map import quality_from_tidal_tier, quality_tier_for_source
|
||||
|
||||
logger = get_logger("hifi_client")
|
||||
|
||||
# A media playlist whose total runtime is below this fraction of the track's
|
||||
# real duration is a preview (some Monochrome instances only have 30s Tidal
|
||||
# DOWNLOAD access — a 220s track comes back as ~30s of segments + ENDLIST).
|
||||
_PREVIEW_DURATION_RATIO = 0.85
|
||||
_EXTINF_RE = re.compile(r'#EXTINF:\s*([0-9.]+)')
|
||||
|
||||
|
||||
def hls_total_seconds(playlist_text: str) -> float:
|
||||
"""Sum the ``#EXTINF`` segment durations in an HLS media playlist."""
|
||||
return sum(float(x) for x in _EXTINF_RE.findall(playlist_text or ''))
|
||||
|
||||
|
||||
def is_preview_playlist(playlist_s: float, track_s: float,
|
||||
ratio: float = _PREVIEW_DURATION_RATIO) -> bool:
|
||||
"""True when the playlist runtime is far shorter than the track's real
|
||||
duration (a preview). Returns False when either duration is unknown, so a
|
||||
missing reference never false-positives — the post-download audio guard is
|
||||
the safety net.
|
||||
"""
|
||||
if not playlist_s or not track_s or track_s <= 0:
|
||||
return False
|
||||
return playlist_s < track_s * ratio
|
||||
|
||||
|
||||
# HLS quality presets mapping to /trackManifests/ format parameters
|
||||
HLS_QUALITY_MAP = {
|
||||
'hires': {
|
||||
|
|
@ -257,10 +232,7 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
if download_path is None:
|
||||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
self._instances = []
|
||||
self._instance_lock = threading.Lock()
|
||||
|
|
@ -725,8 +697,7 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
|
||||
return init_uri, segment_uris
|
||||
|
||||
def _get_hls_manifest(self, track_id: int, quality: str = 'lossless',
|
||||
expected_duration_s: float = 0) -> Optional[Dict]:
|
||||
def _get_hls_manifest(self, track_id: int, quality: str = 'lossless') -> Optional[Dict]:
|
||||
q_info = HLS_QUALITY_MAP.get(quality, HLS_QUALITY_MAP['lossless'])
|
||||
formats = q_info['formats']
|
||||
|
||||
|
|
@ -780,20 +751,6 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
logger.warning(f"Failed to fetch variant playlist for track {track_id}: {e}")
|
||||
return None
|
||||
|
||||
# Preview detection — some instances only have 30s Tidal DOWNLOAD
|
||||
# access, returning a playlist far shorter than the real track. Decline
|
||||
# it (and rotate the instance) so the orchestrator falls through to a
|
||||
# real source instead of fetching a 30s file that gets quarantined.
|
||||
playlist_s = hls_total_seconds(media_text)
|
||||
if is_preview_playlist(playlist_s, expected_duration_s):
|
||||
logger.warning(
|
||||
f"HiFi manifest for track {track_id} ({quality}) is a "
|
||||
f"{playlist_s:.0f}s preview of a {expected_duration_s:.0f}s track — "
|
||||
f"declining this instance"
|
||||
)
|
||||
self._rotate_instance(self._current_instance)
|
||||
return None
|
||||
|
||||
if init_uri:
|
||||
logger.info(f"HiFi HLS manifest for track {track_id}: "
|
||||
f"init segment + {len(segment_uris)} segments ({quality})")
|
||||
|
|
@ -916,18 +873,13 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
loop = asyncio.get_event_loop()
|
||||
tracks = await loop.run_in_executor(None, lambda: self.search_raw(query))
|
||||
|
||||
quality_key = quality_tier_for_source('hifi', default='lossless')
|
||||
quality_key = config_manager.get('hifi_download.quality', 'lossless')
|
||||
q_info = HLS_QUALITY_MAP.get(quality_key, HLS_QUALITY_MAP['lossless'])
|
||||
|
||||
# HiFi is Tidal-backed; stamp the configured tier so the global
|
||||
# ranker sees real sample_rate/bit_depth, not just 'flac'.
|
||||
tier_quality = quality_from_tidal_tier(quality_key)
|
||||
|
||||
results = []
|
||||
for t in tracks:
|
||||
try:
|
||||
tr = self._to_track_result(t, q_info)
|
||||
tr.set_quality(tier_quality)
|
||||
results.append(tr)
|
||||
except Exception as e:
|
||||
logger.debug(f"Skipping track result conversion: {e}")
|
||||
|
|
@ -998,7 +950,7 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
)
|
||||
|
||||
def _download_sync(self, download_id: str, track_id: int, display_name: str) -> Optional[str]:
|
||||
quality_key = quality_tier_for_source('hifi', default='lossless')
|
||||
quality_key = config_manager.get('hifi_download.quality', 'lossless')
|
||||
chain = ['hires', 'lossless', 'high', 'low']
|
||||
start = chain.index(quality_key) if quality_key in chain else 1
|
||||
allow_fallback = config_manager.get('hifi_download.allow_fallback', True)
|
||||
|
|
@ -1006,26 +958,21 @@ class HiFiClient(DownloadSourcePlugin):
|
|||
|
||||
MIN_AUDIO_SIZE = 100 * 1024
|
||||
|
||||
# Expected track length, drives every preview/truncation guard here:
|
||||
# * _get_hls_manifest's pre-download is_preview_playlist check
|
||||
# * the pre-download is_short_audio manifest check
|
||||
# * the post-download is_preview_download faked-header decode check
|
||||
# Best-effort: a 0 here just disables the duration checks, never rejects.
|
||||
# Expected track length (for the preview/truncation guards). Best-effort: a 0
|
||||
# here just disables the duration checks for this track, never rejects.
|
||||
expected_s = 0.0
|
||||
try:
|
||||
info = self.get_track_info(track_id) or {}
|
||||
expected_s = float(info.get('duration_s') or 0)
|
||||
except Exception:
|
||||
expected_s = 0.0
|
||||
expected_duration_s = expected_s # alias for _get_hls_manifest's param name
|
||||
|
||||
for q_key in chain:
|
||||
if self.shutdown_check and self.shutdown_check():
|
||||
logger.info("Shutdown detected, aborting HiFi download")
|
||||
return None
|
||||
|
||||
manifest_info = self._get_hls_manifest(track_id, quality=q_key,
|
||||
expected_duration_s=expected_duration_s)
|
||||
manifest_info = self._get_hls_manifest(track_id, quality=q_key)
|
||||
if (
|
||||
not manifest_info
|
||||
or (
|
||||
|
|
|
|||
|
|
@ -52,14 +52,6 @@ _DEFAULT_LENGTH_TOLERANCE_S = 3.0
|
|||
_LENGTH_TOLERANCE_LONG_TRACK_S = 5.0
|
||||
_LONG_TRACK_THRESHOLD_S = 600.0 # 10 minutes
|
||||
|
||||
# A file that runs LONGER than the expected metadata is the opposite of a truncated
|
||||
# download — it's almost always a different master/version (a remaster with a longer
|
||||
# outro, an extended fade, an album cut vs the radio edit). The duration check exists to
|
||||
# catch TRUNCATION (short files) and wildly-wrong matches, so on the auto default we allow
|
||||
# more drift in the longer direction and keep the tight bound for short files. A wrong-song
|
||||
# match still trips this — it's usually off by far more than 15s. (#937)
|
||||
_LONGER_VERSION_TOLERANCE_S = 15.0
|
||||
|
||||
# Upper bound for the user-configurable override. Anything past 60s
|
||||
# means the check is effectively off — cap defends against accidental
|
||||
# nonsense like 9999 making logs misleading. Users who genuinely want
|
||||
|
|
@ -250,32 +242,18 @@ def check_audio_integrity(
|
|||
if expected_length_s > _LONG_TRACK_THRESHOLD_S
|
||||
else _DEFAULT_LENGTH_TOLERANCE_S
|
||||
)
|
||||
user_pinned_tolerance = False
|
||||
else:
|
||||
user_pinned_tolerance = True
|
||||
checks["length_tolerance_s"] = length_tolerance_s
|
||||
|
||||
# Positive drift = the file runs LONGER than expected (not truncation). On the auto
|
||||
# default, give the longer direction more room so legit longer masters/versions aren't
|
||||
# quarantined (#937); a user-pinned tolerance is honoured symmetrically.
|
||||
signed_drift_s = actual_length_s - expected_length_s
|
||||
drift_s = abs(signed_drift_s)
|
||||
drift_s = abs(actual_length_s - expected_length_s)
|
||||
checks["length_drift_s"] = drift_s
|
||||
effective_tolerance_s = length_tolerance_s
|
||||
if signed_drift_s > 0 and not user_pinned_tolerance:
|
||||
effective_tolerance_s = max(length_tolerance_s, _LONGER_VERSION_TOLERANCE_S)
|
||||
checks["effective_tolerance_s"] = effective_tolerance_s
|
||||
|
||||
if drift_s > effective_tolerance_s:
|
||||
runs_long = signed_drift_s > 0
|
||||
if drift_s > length_tolerance_s:
|
||||
return IntegrityResult(
|
||||
ok=False,
|
||||
reason=f"Duration mismatch: file is {actual_length_s:.1f}s, "
|
||||
f"expected {expected_length_s:.1f}s "
|
||||
f"(drift {drift_s:.1f}s > tolerance {effective_tolerance_s:.1f}s) — "
|
||||
+ ("runs longer than expected — likely a different version/master or wrong file"
|
||||
if runs_long
|
||||
else "likely truncated download or wrong file matched"),
|
||||
f"(drift {drift_s:.1f}s > tolerance {length_tolerance_s:.1f}s) — "
|
||||
"likely truncated download or wrong file matched",
|
||||
checks=checks,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -230,9 +230,6 @@ def get_audio_quality_string(file_path):
|
|||
if ext == ".flac":
|
||||
from mutagen.flac import FLAC
|
||||
audio = FLAC(file_path)
|
||||
sr = getattr(audio.info, "sample_rate", 0) or 0
|
||||
if sr:
|
||||
return f"FLAC {audio.info.bits_per_sample}bit/{sr / 1000:g}kHz"
|
||||
return f"FLAC {audio.info.bits_per_sample}bit"
|
||||
|
||||
if ext == ".mp3":
|
||||
|
|
@ -266,122 +263,6 @@ def get_audio_quality_string(file_path):
|
|||
return ""
|
||||
|
||||
|
||||
def probe_audio_quality(file_path: str):
|
||||
"""Read the actual file and return an AudioQuality with real measured values.
|
||||
|
||||
Uses mutagen to extract sample_rate, bit_depth, and bitrate from the
|
||||
downloaded file — these are ground-truth values, not estimates.
|
||||
Returns None when the file cannot be read.
|
||||
"""
|
||||
from core.quality.model import AudioQuality
|
||||
try:
|
||||
ext = os.path.splitext(file_path)[1].lower().lstrip('.')
|
||||
|
||||
if ext == 'flac':
|
||||
from mutagen.flac import FLAC
|
||||
audio = FLAC(file_path)
|
||||
return AudioQuality(
|
||||
format='flac',
|
||||
bitrate=audio.info.bitrate // 1000 if audio.info.bitrate else None,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
bit_depth=audio.info.bits_per_sample,
|
||||
)
|
||||
|
||||
if ext == 'mp3':
|
||||
from mutagen.mp3 import MP3
|
||||
audio = MP3(file_path)
|
||||
return AudioQuality(
|
||||
format='mp3',
|
||||
bitrate=audio.info.bitrate // 1000,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
)
|
||||
|
||||
if ext in ('m4a', 'aac', 'mp4'):
|
||||
from mutagen.mp4 import MP4
|
||||
audio = MP4(file_path)
|
||||
# .m4a can carry AAC (lossy) OR ALAC (lossless) — only the real
|
||||
# codec tells them apart, which is why extension-based classification
|
||||
# defaults to 'aac' and we correct it here from the probed file.
|
||||
codec = (getattr(audio.info, 'codec', '') or '').lower()
|
||||
if 'alac' in codec:
|
||||
return AudioQuality(
|
||||
format='alac',
|
||||
bitrate=audio.info.bitrate // 1000 if audio.info.bitrate else None,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
bit_depth=getattr(audio.info, 'bits_per_sample', None) or None,
|
||||
)
|
||||
return AudioQuality(
|
||||
format='aac',
|
||||
bitrate=audio.info.bitrate // 1000,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
)
|
||||
|
||||
if ext == 'ogg':
|
||||
from mutagen.oggvorbis import OggVorbis
|
||||
audio = OggVorbis(file_path)
|
||||
return AudioQuality(
|
||||
format='ogg',
|
||||
bitrate=audio.info.bitrate // 1000,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
)
|
||||
|
||||
if ext == 'opus':
|
||||
from mutagen.oggopus import OggOpus
|
||||
audio = OggOpus(file_path)
|
||||
return AudioQuality(
|
||||
format='opus',
|
||||
bitrate=audio.info.bitrate // 1000,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
)
|
||||
|
||||
if ext in ('wav', 'aiff', 'aif'):
|
||||
# AIFF must use mutagen.aiff.AIFF — WAVE() can't parse it and would
|
||||
# raise, making the file fail open and silently bypass the quality
|
||||
# filter. Both are uncompressed PCM, so they share the 'wav' tier.
|
||||
if ext == 'wav':
|
||||
from mutagen.wave import WAVE
|
||||
audio = WAVE(file_path)
|
||||
else:
|
||||
from mutagen.aiff import AIFF
|
||||
audio = AIFF(file_path)
|
||||
return AudioQuality(
|
||||
format='wav',
|
||||
bitrate=audio.info.bitrate // 1000 if audio.info.bitrate else None,
|
||||
sample_rate=audio.info.sample_rate,
|
||||
bit_depth=getattr(audio.info, 'bits_per_sample', None),
|
||||
)
|
||||
|
||||
if ext == 'wma':
|
||||
from mutagen.asf import ASF
|
||||
audio = ASF(file_path)
|
||||
return AudioQuality(
|
||||
format='wma',
|
||||
bitrate=audio.info.bitrate // 1000 if audio.info.bitrate else None,
|
||||
sample_rate=getattr(audio.info, 'sample_rate', None),
|
||||
)
|
||||
|
||||
if ext in ('dsf', 'dff'):
|
||||
# DSD (DSD Stream File / DSDIFF) — 1-bit hi-res lossless (#939). mutagen
|
||||
# reads .dsf (rate/bitrate/bit_depth); .dff has no mutagen reader, so it
|
||||
# still classifies as the lossless 'dsf' tier just without measured detail.
|
||||
sr = bd = br = None
|
||||
if ext == 'dsf':
|
||||
try:
|
||||
from mutagen.dsf import DSF
|
||||
info = DSF(file_path).info
|
||||
sr = getattr(info, 'sample_rate', None)
|
||||
bd = getattr(info, 'bits_per_sample', None)
|
||||
br = info.bitrate // 1000 if getattr(info, 'bitrate', None) else None
|
||||
except Exception: # noqa: S110 — unreadable DSF still classifies lossless, just without measured detail
|
||||
pass
|
||||
return AudioQuality(format='dsf', bitrate=br, sample_rate=sr, bit_depth=bd)
|
||||
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug("probe_audio_quality failed for %s: %s", file_path, e)
|
||||
return None
|
||||
|
||||
|
||||
def get_quality_tier_from_extension(file_path):
|
||||
"""Classify a file extension into a quality tier."""
|
||||
if not file_path:
|
||||
|
|
@ -524,29 +405,14 @@ def downsample_hires_flac(final_path, context):
|
|||
return None
|
||||
|
||||
|
||||
def m4a_codec(path):
|
||||
"""Codec of an .m4a/.mp4 container ('alac', 'aac', …) or None — lets the
|
||||
lossless check tell ALAC (lossless) from AAC (lossy), since both are .m4a."""
|
||||
try:
|
||||
from mutagen.mp4 import MP4
|
||||
return (getattr(MP4(path).info, 'codec', '') or '').lower() or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def create_lossy_copy(final_path):
|
||||
"""Convert a lossless file (FLAC / ALAC / WAV / AIFF / DSD) to a lossy copy
|
||||
using the configured codec. Non-lossless inputs are skipped (#941)."""
|
||||
from core.quality.lossless import (
|
||||
is_lossless_audio_path,
|
||||
lossy_output_would_overwrite_source,
|
||||
)
|
||||
"""Convert a FLAC file to a lossy copy using the configured codec."""
|
||||
from mutagen.flac import FLAC
|
||||
|
||||
if not config_manager.get("lossy_copy.enabled", False):
|
||||
return None
|
||||
|
||||
# Was FLAC-only; now any lossless source. .m4a is probed (ALAC vs AAC).
|
||||
if not is_lossless_audio_path(final_path, probe_codec=m4a_codec):
|
||||
if os.path.splitext(final_path)[1].lower() != ".flac":
|
||||
return None
|
||||
|
||||
codec = config_manager.get("lossy_copy.codec", "mp3").lower()
|
||||
|
|
@ -575,16 +441,6 @@ def create_lossy_copy(final_path):
|
|||
out_basename = out_basename.replace(original_quality, quality_label)
|
||||
out_path = os.path.join(os.path.dirname(out_path), out_basename)
|
||||
|
||||
# Safety invariant: never write the lossy copy over its own source (an .m4a
|
||||
# ALAC source + AAC target lands on the same .m4a path). ffmpeg runs with -y,
|
||||
# so this guard MUST precede it — the later delete-original guard is too late.
|
||||
if lossy_output_would_overwrite_source(final_path, out_path):
|
||||
logger.info(
|
||||
f"[Lossy Copy] Skipping — {codec.upper()} output would overwrite the "
|
||||
f"source: {os.path.basename(final_path)}"
|
||||
)
|
||||
return None
|
||||
|
||||
ffmpeg_bin = shutil.which("ffmpeg")
|
||||
if not ffmpeg_bin:
|
||||
local = os.path.join(os.path.dirname(__file__), "tools", "ffmpeg")
|
||||
|
|
|
|||
|
|
@ -105,72 +105,30 @@ def move_to_quarantine(file_path: str, context: dict, reason: str, automation_en
|
|||
|
||||
|
||||
def check_flac_bit_depth(file_path: str, context: dict) -> Optional[str]:
|
||||
"""Legacy wrapper — delegates to check_quality_target.
|
||||
|
||||
Kept for callers that still pass trigger='bit_depth'; the new guard
|
||||
covers bit_depth as part of the full quality target check.
|
||||
"""
|
||||
return check_quality_target(file_path, context)
|
||||
|
||||
|
||||
def check_quality_target(file_path: str, context: dict) -> Optional[str]:
|
||||
"""Return a rejection message when the downloaded file does not satisfy
|
||||
the user's quality priority list.
|
||||
|
||||
Probes the actual file with mutagen (ground-truth sample_rate,
|
||||
bit_depth, bitrate) and checks it against the profile's
|
||||
``ranked_targets``. Falls back gracefully when fallback_enabled=True.
|
||||
|
||||
Works for all formats and all download sources — no Soulseek-specific
|
||||
logic here.
|
||||
"""
|
||||
from core.imports.file_ops import probe_audio_quality
|
||||
from core.quality.selection import targets_from_profile, quality_meets_profile
|
||||
|
||||
# Master toggle (Settings → Import). When OFF, the quality check is skipped
|
||||
# entirely and files import regardless of quality — the user opted out of
|
||||
# quality-filtering on import. Default ON preserves existing behaviour. The
|
||||
# library Quality Upgrade scanner still flags below-profile files either way.
|
||||
if _get_config_manager().get("import.quality_filter_enabled", True) is False:
|
||||
logger.debug(
|
||||
"[QualityGuard] import.quality_filter_enabled=False — skipping quality "
|
||||
"filter for %s", os.path.basename(file_path),
|
||||
)
|
||||
"""Return a rejection message if a FLAC file violates the configured bit depth."""
|
||||
if not context.get("_audio_quality", "").startswith("FLAC"):
|
||||
return None
|
||||
|
||||
aq = probe_audio_quality(file_path)
|
||||
if aq is None:
|
||||
logger.debug("[QualityGuard] Could not probe %s — skipping check", os.path.basename(file_path))
|
||||
quality_profile = MusicDatabase().get_quality_profile()
|
||||
flac_config = quality_profile.get("qualities", {}).get("flac", {})
|
||||
flac_pref = flac_config.get("bit_depth", "any")
|
||||
if flac_pref == "any":
|
||||
return None
|
||||
|
||||
profile = MusicDatabase().get_quality_profile()
|
||||
targets, fallback_enabled = targets_from_profile(profile)
|
||||
|
||||
if not targets:
|
||||
actual_bits = context["_audio_quality"].replace("FLAC ", "").replace("bit", "")
|
||||
if actual_bits == flac_pref:
|
||||
return None
|
||||
|
||||
flac_fallback = flac_config.get("bit_depth_fallback", True)
|
||||
downsample_enabled = _get_config_manager().get("lossy_copy.downsample_hires", False)
|
||||
|
||||
matched = quality_meets_profile(aq, targets)
|
||||
|
||||
track_info = context.get("track_info", {})
|
||||
track_name = track_info.get("name", os.path.basename(file_path))
|
||||
actual_label = aq.label()
|
||||
|
||||
if matched:
|
||||
logger.info("[QualityGuard] %s meets profile: %s", track_name, actual_label)
|
||||
if flac_fallback or downsample_enabled:
|
||||
if downsample_enabled:
|
||||
logger.info("[FLAC Downsample] Accepted %s-bit FLAC (will be downsampled to %s-bit): %s", actual_bits, flac_pref, track_name)
|
||||
else:
|
||||
logger.warning("[FLAC Fallback] Accepted %s-bit FLAC (preferred %s-bit): %s", actual_bits, flac_pref, track_name)
|
||||
return None
|
||||
|
||||
# No target matched
|
||||
best_label = targets[0].label if targets else "?"
|
||||
if fallback_enabled or downsample_enabled:
|
||||
logger.warning(
|
||||
"[QualityGuard] %s did not match any target (got %s, wanted %s) — accepting via fallback",
|
||||
track_name, actual_label, best_label,
|
||||
)
|
||||
return None
|
||||
|
||||
return (
|
||||
f"Quality mismatch: file is {actual_label}, "
|
||||
f"does not satisfy any configured target (best wanted: {best_label})"
|
||||
)
|
||||
return f"FLAC bit depth mismatch: file is {actual_bits}-bit, preference is {flac_pref}-bit"
|
||||
|
|
|
|||
|
|
@ -34,11 +34,9 @@ from core.imports.context import (
|
|||
)
|
||||
from core.imports.file_integrity import check_audio_integrity, expected_duration_for_check, resolve_duration_tolerance
|
||||
from core.imports.filename import extract_track_number_from_filename
|
||||
from core.imports.guards import check_flac_bit_depth, check_quality_target, move_to_quarantine
|
||||
from core.imports.silence import detect_broken_audio
|
||||
from core.imports.guards import check_flac_bit_depth, move_to_quarantine
|
||||
from core.imports.quarantine import (
|
||||
approve_quarantine_entry,
|
||||
delete_quarantine_entry,
|
||||
entry_id_from_quarantined_filename,
|
||||
list_quarantine_entries,
|
||||
)
|
||||
|
|
@ -162,9 +160,7 @@ def import_rejection_reason(context: dict) -> str | None:
|
|||
f"{context.get('_acoustid_failure_msg', 'fingerprint mismatch')}"
|
||||
)
|
||||
if context.get('_bitdepth_rejected'):
|
||||
return "rejected by quality filter"
|
||||
if context.get('_silence_rejected'):
|
||||
return "rejected by audio guard (incomplete or silent audio)"
|
||||
return "rejected by bit-depth filter"
|
||||
if context.get('_race_guard_failed'):
|
||||
return "source file disappeared before import completed"
|
||||
return None
|
||||
|
|
@ -325,14 +321,11 @@ def post_process_matched_download(context_key, context, file_path, runtime, meta
|
|||
_mark_task_quarantined(context, quarantine_path)
|
||||
logger.error(f"File quarantined due to integrity failure: {quarantine_path}")
|
||||
except Exception as quarantine_error:
|
||||
# Quarantine MOVE failed (e.g. cross-device / permission on a NAS).
|
||||
# Do NOT delete — destroying a download we couldn't even quarantine is
|
||||
# data loss and forces a re-download. Leave it in place so it can be
|
||||
# retried; the task is still marked failed below either way (#kettui).
|
||||
logger.error(
|
||||
f"Quarantine failed ({quarantine_error}) — leaving file in place "
|
||||
f"for retry (not deleting): {file_path}"
|
||||
)
|
||||
logger.error(f"Quarantine failed ({quarantine_error}), deleting broken file: {file_path}")
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except Exception as del_error:
|
||||
logger.error(f"Could not delete broken file either: {del_error}")
|
||||
|
||||
with matched_context_lock:
|
||||
if context_key in matched_downloads_context:
|
||||
|
|
@ -360,118 +353,6 @@ def post_process_matched_download(context_key, context, file_path, runtime, meta
|
|||
f"drift={integrity.checks.get('length_drift_s', 'n/a')})"
|
||||
)
|
||||
|
||||
# Audio-completeness guard — runs right where the length is verified,
|
||||
# BEFORE the AcoustID/quality gates, so a truncated file (container
|
||||
# claims the full length but only ~30s actually decodes, e.g. HiFi/
|
||||
# Monochrome HLS assembly) or a mostly-silent file is caught regardless
|
||||
# of its quality verdict and gets the right reason. Same quarantine +
|
||||
# next-candidate retry pattern as the integrity check.
|
||||
#
|
||||
# Opt-in (default OFF): this is the one check that fully DECODES the file
|
||||
# with ffmpeg, so it is the most CPU-expensive step in post-processing.
|
||||
# Most preview/truncation cases are already caught at the source
|
||||
# (HiFi/Qobuz have their own guards), so it stays off unless the user
|
||||
# turns it on under Settings → Post-processing.
|
||||
_audio_guard_enabled = config_manager.get('post_processing.audio_completeness_check', False)
|
||||
_skip_audio = (not _audio_guard_enabled) or _should_skip_quarantine_check(context, 'silence')
|
||||
audio_reason = None if _skip_audio else detect_broken_audio(file_path)
|
||||
if audio_reason:
|
||||
logger.error(f"[AudioGuard] Rejected {_basename}: {audio_reason}")
|
||||
context['_silence_rejected'] = True
|
||||
try:
|
||||
quarantine_path = move_to_quarantine(
|
||||
file_path, context, audio_reason, automation_engine,
|
||||
trigger='silence',
|
||||
)
|
||||
_mark_task_quarantined(context, quarantine_path)
|
||||
logger.warning("File quarantined — incomplete/silent audio: %s", quarantine_path)
|
||||
except Exception as quarantine_error:
|
||||
# Don't delete a file we couldn't quarantine — leave it for retry
|
||||
# instead of forcing a re-download (data loss). See integrity block.
|
||||
logger.error(
|
||||
f"Quarantine failed ({quarantine_error}) — leaving file in place "
|
||||
f"for retry (not deleting): {file_path}"
|
||||
)
|
||||
|
||||
with matched_context_lock:
|
||||
matched_downloads_context.pop(context_key, None)
|
||||
|
||||
task_id = context.get('task_id')
|
||||
batch_id = context.get('batch_id')
|
||||
# Try the next-best candidate before giving up — same pattern as
|
||||
# the integrity / AcoustID / quality failures.
|
||||
if _requeue_quarantined_task_for_retry(task_id, batch_id, 'silence'):
|
||||
logger.info(
|
||||
"Incomplete/silent audio for task %s — retrying next-best candidate: %s",
|
||||
task_id, audio_reason,
|
||||
)
|
||||
return
|
||||
if task_id:
|
||||
with tasks_lock:
|
||||
if task_id in download_tasks:
|
||||
download_tasks[task_id]['status'] = 'failed'
|
||||
download_tasks[task_id]['error_message'] = f"Audio guard: {audio_reason}"
|
||||
if task_id and batch_id:
|
||||
_notify_download_completed(batch_id, task_id, success=False)
|
||||
return
|
||||
|
||||
# QUALITY GATE — runs BEFORE AcoustID so (1) a wrong-quality file is
|
||||
# rejected without paying for an AcoustID fingerprint, and (2) the real
|
||||
# audio quality is recorded on the context (→ quarantine sidecar) for
|
||||
# EVERY trigger, so it's known when reviewing/approving any quarantined
|
||||
# file. force_import still never fires on a quality mismatch.
|
||||
context['_audio_quality'] = get_audio_quality_string(file_path)
|
||||
if context['_audio_quality']:
|
||||
logger.info(f"Audio quality detected: {context['_audio_quality']}")
|
||||
|
||||
_skip_quality = _should_skip_quarantine_check(context, 'bit_depth') or \
|
||||
_should_skip_quarantine_check(context, 'quality')
|
||||
rejection_reason = None if _skip_quality else check_quality_target(file_path, context)
|
||||
if _skip_quality:
|
||||
logger.info(f"[QualityGuard] Skipped (user approval) for {_basename}")
|
||||
if rejection_reason:
|
||||
try:
|
||||
quarantine_path = move_to_quarantine(
|
||||
file_path,
|
||||
context,
|
||||
rejection_reason,
|
||||
automation_engine,
|
||||
trigger='quality',
|
||||
)
|
||||
_mark_task_quarantined(context, quarantine_path)
|
||||
logger.info(f"File quarantined due to quality mismatch: {quarantine_path}")
|
||||
except Exception as quarantine_error:
|
||||
# Don't delete a file we couldn't quarantine — leave it for retry
|
||||
# instead of forcing a re-download (data loss). See integrity block.
|
||||
logger.error(
|
||||
f"Quarantine failed ({quarantine_error}) — leaving file in place "
|
||||
f"for retry (not deleting): {file_path}"
|
||||
)
|
||||
|
||||
context['_bitdepth_rejected'] = True
|
||||
task_id = context.get('task_id')
|
||||
batch_id = context.get('batch_id')
|
||||
with matched_context_lock:
|
||||
matched_downloads_context.pop(context_key, None)
|
||||
|
||||
# Try the next-best candidate before giving up — same pattern
|
||||
# as AcoustID and integrity failures.
|
||||
if _requeue_quarantined_task_for_retry(task_id, batch_id, 'quality'):
|
||||
logger.info(
|
||||
"Quality mismatch for task %s — retrying next-best candidate: %s",
|
||||
task_id, rejection_reason,
|
||||
)
|
||||
return
|
||||
|
||||
if task_id:
|
||||
with tasks_lock:
|
||||
if task_id in download_tasks:
|
||||
download_tasks[task_id]['status'] = 'failed'
|
||||
download_tasks[task_id]['error_message'] = f"Quality filter: {rejection_reason}"
|
||||
if task_id and batch_id:
|
||||
_notify_download_completed(batch_id, task_id, success=False)
|
||||
return
|
||||
|
||||
_skip_acoustid = _should_skip_quarantine_check(context, 'acoustid')
|
||||
if _skip_acoustid:
|
||||
logger.info(f"[AcoustID] Skipped (user approval) for {_basename}")
|
||||
|
|
@ -509,44 +390,24 @@ def post_process_matched_download(context_key, context, file_path, runtime, meta
|
|||
logger.info(f"AcoustID verification result: {verification_result.value} - {verification_msg}")
|
||||
context['_acoustid_result'] = verification_result.value
|
||||
|
||||
# Fail-closed mode: when the user requires a hard AcoustID
|
||||
# PASS, a SKIP (ran but couldn't confirm — no fingerprint
|
||||
# match / cross-script metadata) is treated like a FAIL:
|
||||
# quarantine + try the next candidate, instead of importing
|
||||
# an unverified file. ERROR (rate-limit / infra) is NOT
|
||||
# blocked — that would stall the whole pipeline during an
|
||||
# outage; those still import with their existing flag.
|
||||
require_verified = config_manager.get('acoustid.require_verified', False)
|
||||
_skip_as_fail = (
|
||||
require_verified
|
||||
and verification_result == VerificationResult.SKIP
|
||||
)
|
||||
if _skip_as_fail:
|
||||
verification_msg = (
|
||||
f"AcoustID could not confirm the track and 'require verified' "
|
||||
f"is on — rejecting unverified file ({verification_msg})"
|
||||
)
|
||||
logger.warning("[AcoustID] Require-verified: SKIP treated as FAIL — %s", verification_msg)
|
||||
|
||||
if verification_result == VerificationResult.FAIL or _skip_as_fail:
|
||||
if verification_result == VerificationResult.FAIL:
|
||||
try:
|
||||
quarantine_path = move_to_quarantine(
|
||||
file_path,
|
||||
context,
|
||||
verification_msg,
|
||||
automation_engine,
|
||||
trigger='acoustid_unverified' if _skip_as_fail else 'acoustid',
|
||||
trigger='acoustid',
|
||||
)
|
||||
_mark_task_quarantined(context, quarantine_path)
|
||||
logger.error(f"File quarantined due to verification failure: {quarantine_path}")
|
||||
except Exception as quarantine_error:
|
||||
# Don't delete a file we couldn't quarantine — leave it for
|
||||
# retry instead of forcing a re-download (data loss). The
|
||||
# task is still marked failed / requeued below. See integrity.
|
||||
logger.error(
|
||||
f"Quarantine failed ({quarantine_error}) — leaving file "
|
||||
f"in place for retry (not deleting): {file_path}"
|
||||
)
|
||||
logger.error(f"Quarantine failed ({quarantine_error}), deleting wrong file: {file_path}")
|
||||
logger.error(f"Quarantine failed, deleting wrong file: {file_path}")
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except Exception as del_error:
|
||||
logger.error(f"Could not delete wrong file either: {del_error}")
|
||||
|
||||
context['_acoustid_quarantined'] = True
|
||||
context['_acoustid_failure_msg'] = verification_msg
|
||||
|
|
@ -716,6 +577,48 @@ def post_process_matched_download(context_key, context, file_path, runtime, meta
|
|||
album_info.get('album_name', 'None'),
|
||||
)
|
||||
|
||||
context['_audio_quality'] = get_audio_quality_string(file_path)
|
||||
if context['_audio_quality']:
|
||||
logger.info(f"Audio quality detected: {context['_audio_quality']}")
|
||||
|
||||
_skip_bit_depth = _should_skip_quarantine_check(context, 'bit_depth')
|
||||
rejection_reason = None if _skip_bit_depth else check_flac_bit_depth(file_path, context)
|
||||
if _skip_bit_depth:
|
||||
logger.info(f"[BitDepth] Skipped (user approval) for {_basename}")
|
||||
if rejection_reason:
|
||||
try:
|
||||
quarantine_path = move_to_quarantine(
|
||||
file_path,
|
||||
context,
|
||||
rejection_reason,
|
||||
automation_engine,
|
||||
trigger='bit_depth',
|
||||
)
|
||||
_mark_task_quarantined(context, quarantine_path)
|
||||
logger.info(f"File quarantined due to bit depth filter: {quarantine_path}")
|
||||
except Exception as quarantine_error:
|
||||
logger.error(f"Quarantine failed ({quarantine_error}), deleting file: {file_path}")
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except Exception as e:
|
||||
logger.debug("delete quarantine fallback: %s", e)
|
||||
|
||||
context['_bitdepth_rejected'] = True
|
||||
with matched_context_lock:
|
||||
if context_key in matched_downloads_context:
|
||||
del matched_downloads_context[context_key]
|
||||
|
||||
task_id = context.get('task_id')
|
||||
batch_id = context.get('batch_id')
|
||||
if task_id:
|
||||
with tasks_lock:
|
||||
if task_id in download_tasks:
|
||||
download_tasks[task_id]['status'] = 'failed'
|
||||
download_tasks[task_id]['error_message'] = f"Bit depth filter: {rejection_reason}"
|
||||
if task_id and batch_id:
|
||||
_notify_download_completed(batch_id, task_id, success=False)
|
||||
return
|
||||
|
||||
file_ext = os.path.splitext(file_path)[1]
|
||||
clean_track_name = get_import_clean_title(
|
||||
context,
|
||||
|
|
@ -1084,22 +987,6 @@ def post_process_matched_download_with_verification(context_key, context, file_p
|
|||
if original_batch_id:
|
||||
context['batch_id'] = original_batch_id
|
||||
|
||||
# Quality / audio-guard quarantine. Unlike acoustid/integrity (whose
|
||||
# retry+fail is driven by THIS wrapper below), the inner pipeline fully
|
||||
# owns the quality and audio-guard outcome: it quarantines the file and
|
||||
# then either re-queues the next-best candidate or marks the task failed
|
||||
# and notifies. The wrapper must NOT continue to the "assume success"
|
||||
# fall-through — that marked the quarantined file Completed, so the same
|
||||
# track showed in BOTH Completed and Quarantine (e.g. an MP3-VBR rejected
|
||||
# by a FLAC-only profile). It must also NOT mark it failed here, which
|
||||
# would clobber a successful next-candidate retry. Just return.
|
||||
if context.get('_bitdepth_rejected') or context.get('_silence_rejected'):
|
||||
logger.info(
|
||||
f"Task {task_id} quarantined by the quality/audio guard — inner "
|
||||
f"pipeline already handled retry/fail; wrapper not marking completed"
|
||||
)
|
||||
return
|
||||
|
||||
if context.get('_race_guard_failed'):
|
||||
logger.info(f"Race guard: source file gone for task {task_id} — marking as failed")
|
||||
with tasks_lock:
|
||||
|
|
@ -1113,33 +1000,6 @@ def post_process_matched_download_with_verification(context_key, context, file_p
|
|||
return
|
||||
|
||||
if context.get('_acoustid_quarantined'):
|
||||
# Race-condition guard: if the user approved an alternative quarantine
|
||||
# entry for this track while this download was already in flight, the
|
||||
# pipeline will have created a new quarantine entry for the just-finished
|
||||
# file. Delete that entry and exit — the user's choice already won.
|
||||
_approved_alt = False
|
||||
if task_id:
|
||||
with tasks_lock:
|
||||
_ct = download_tasks.get(task_id)
|
||||
if isinstance(_ct, dict) and _ct.get('_quarantine_approved_alternative'):
|
||||
_approved_alt = True
|
||||
if _approved_alt:
|
||||
_eid = context.get('_quarantine_entry_id')
|
||||
if _eid:
|
||||
try:
|
||||
from core.imports.guards import _get_config_manager
|
||||
_dl_dir = _get_config_manager().get('soulseek.download_path', './downloads')
|
||||
_qdir = os.path.join(docker_resolve_path(_dl_dir), 'ss_quarantine')
|
||||
delete_quarantine_entry(_qdir, _eid)
|
||||
logger.info(
|
||||
f"[Quarantine] Discarded late entry {_eid} — task {task_id} "
|
||||
f"was cancelled by prior quarantine approval"
|
||||
)
|
||||
except Exception as _dqe:
|
||||
logger.debug(f"[Quarantine] Could not clean up late entry {_eid}: {_dqe}")
|
||||
_notify_download_completed(batch_id, task_id, success=False)
|
||||
return
|
||||
|
||||
failure_msg = context.get('_acoustid_failure_msg', 'AcoustID verification failed')
|
||||
# Before failing outright, try the next-best candidate. The wrong
|
||||
# file was just quarantined; re-running the worker (with the bad
|
||||
|
|
@ -1256,10 +1116,6 @@ def post_process_matched_download_with_verification(context_key, context, file_p
|
|||
_mark_task_completed(task_id, context.get('track_info'))
|
||||
if context.get('_verification_status'):
|
||||
download_tasks[task_id]['verification_status'] = context['_verification_status']
|
||||
if context.get('_history_id'):
|
||||
download_tasks[task_id]['history_id'] = context['_history_id']
|
||||
if context.get('_audio_quality'):
|
||||
download_tasks[task_id]['quality'] = context['_audio_quality']
|
||||
with matched_context_lock:
|
||||
if context_key in matched_downloads_context:
|
||||
del matched_downloads_context[context_key]
|
||||
|
|
@ -1274,10 +1130,6 @@ def post_process_matched_download_with_verification(context_key, context, file_p
|
|||
download_tasks[task_id]['metadata_enhanced'] = True
|
||||
if context.get('_verification_status'):
|
||||
download_tasks[task_id]['verification_status'] = context['_verification_status']
|
||||
if context.get('_history_id'):
|
||||
download_tasks[task_id]['history_id'] = context['_history_id']
|
||||
if context.get('_audio_quality'):
|
||||
download_tasks[task_id]['quality'] = context['_audio_quality']
|
||||
redownload_ctx = download_tasks[task_id].get('_redownload_context')
|
||||
|
||||
with matched_context_lock:
|
||||
|
|
|
|||
|
|
@ -168,15 +168,16 @@ def quarantine_group_key(
|
|||
fetch — NOT the downloaded file's own tags. That matters: the file's
|
||||
metadata is frequently *wrong* (that's why it failed acoustid /
|
||||
integrity), whereas the target is fixed and identical across every
|
||||
alternative for one song.
|
||||
alternative for one song (they're all Soulseek uploads of the *same*
|
||||
source track), so grouping by it is an exact relationship, not a fuzzy
|
||||
metadata guess.
|
||||
|
||||
Uses ISRC when available (truly universal across sources and batches).
|
||||
Falls back to normalized artist|track name, which is stable across
|
||||
different batches and sources.
|
||||
|
||||
Source-specific IDs (Spotify track id, Qobuz id, uri) are intentionally
|
||||
NOT used: the same song imported from different playlists or sources gets
|
||||
different source IDs, so id-based keys break cross-batch sibling matching.
|
||||
Prefers a stable target-track id from the sidecar `context.track_info`
|
||||
when present — isrc, then source id, then uri — since those are exact
|
||||
and constant across siblings. Falls back to the normalized
|
||||
artist|track name only for legacy/thin sidecars that carry no context.
|
||||
Keys are kind-prefixed so an id-based key never collides with a
|
||||
name-based one.
|
||||
|
||||
Returns ``None`` when nothing identifies the target (no usable id and
|
||||
both name fields empty). Callers treat a ``None`` key as "its own
|
||||
|
|
@ -190,6 +191,12 @@ def quarantine_group_key(
|
|||
isrc = str(ti.get("isrc") or "").strip().lower()
|
||||
if isrc:
|
||||
return f"isrc:{isrc}"
|
||||
tid = str(ti.get("id") or "").strip()
|
||||
if tid:
|
||||
return f"id:{tid}"
|
||||
uri = str(ti.get("uri") or "").strip()
|
||||
if uri:
|
||||
return f"uri:{uri}"
|
||||
artist = " ".join(str(expected_artist or "").split()).lower()
|
||||
track = " ".join(str(expected_track or "").split()).lower()
|
||||
if not artist and not track:
|
||||
|
|
@ -295,10 +302,6 @@ def list_quarantine_entries(quarantine_dir: str) -> List[Dict[str, Any]]:
|
|||
"source_username": source_username,
|
||||
"source_filename": source_filename,
|
||||
"thumb_url": _extract_context_thumb(ctx),
|
||||
# Real probed audio quality (recorded on the context before the
|
||||
# quality/AcoustID gates) so the review UI shows what the file
|
||||
# actually is when deciding to approve/delete.
|
||||
"quality": ctx.get("_audio_quality", "") if isinstance(ctx, dict) else "",
|
||||
}
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -3,12 +3,10 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from concurrent.futures import as_completed
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
from typing import Any, Callable, Dict
|
||||
|
||||
from core.imports.album import build_album_import_context, build_album_import_match_payload, resolve_album_artist_context
|
||||
from core.imports.context import get_import_context_artist, get_import_track_info, normalize_import_context
|
||||
|
|
@ -73,219 +71,36 @@ class ImportRouteRuntime:
|
|||
logger: Any = module_logger
|
||||
|
||||
|
||||
# ── Shared staging scan ──────────────────────────────────────────────────────
|
||||
# Opening the Import page fires staging files/groups/hints together; each used to
|
||||
# os.walk the whole staging folder AND mutagen-read every file independently — 3×
|
||||
# the directory walk + 3× the tag I/O on every page open (the import-page scan
|
||||
# storm + memory spike, issue #935). They all need the same per-file tag data, so
|
||||
# scan ONCE and let all three derive their views in-memory. A short TTL + a lock
|
||||
# means the three near-simultaneous page-open requests (and any concurrent caller)
|
||||
# share a single scan instead of each kicking off a full re-read.
|
||||
_STAGING_SCAN_LOCK = threading.Lock()
|
||||
_STAGING_SCAN_TTL = 6.0 # seconds — covers the page-open burst; re-scans after
|
||||
_staging_scan_cache: Dict[str, Any] = {"path": None, "ts": 0.0, "records": None}
|
||||
# Bumped by invalidate_staging_scan_cache() so a background scan that finishes after an
|
||||
# import doesn't re-commit stale (pre-import) records (see the generation guard above).
|
||||
_staging_scan_generation: Dict[str, int] = {"value": 0}
|
||||
|
||||
# Background-scan plumbing: a large staging folder (whole-library migration, #947) makes
|
||||
# the synchronous scan exceed gunicorn's 120s request timeout. The runner moves the SAME
|
||||
# scan off the request thread; the endpoints report progress instead of blocking.
|
||||
_staging_scan_status: Dict[str, Any] = {
|
||||
"status": "idle", "scanned": 0, "total": 0, "path": None, "error": None,
|
||||
}
|
||||
_staging_scan_status_lock = threading.Lock()
|
||||
|
||||
|
||||
def _staging_cache_hit(staging_path: str) -> Optional[list]:
|
||||
"""The cached records for ``staging_path`` if still fresh, else None (no scan triggered)."""
|
||||
c = _staging_scan_cache
|
||||
if (c["records"] is not None and c["path"] == staging_path
|
||||
and (time.time() - c["ts"]) < _STAGING_SCAN_TTL):
|
||||
return c["records"]
|
||||
return None
|
||||
|
||||
|
||||
def ensure_background_staging_scan(runtime: ImportRouteRuntime, staging_path: str) -> None:
|
||||
"""Start a background scan for ``staging_path`` unless the cache is warm or a scan for
|
||||
this path is already running. Idempotent — safe to call on every request."""
|
||||
if _staging_cache_hit(staging_path) is not None:
|
||||
return
|
||||
with _staging_scan_status_lock:
|
||||
if (_staging_scan_status["status"] == "scanning"
|
||||
and _staging_scan_status["path"] == staging_path):
|
||||
return
|
||||
_staging_scan_status.update({"status": "scanning", "scanned": 0, "total": 0,
|
||||
"path": staging_path, "error": None})
|
||||
|
||||
def _run() -> None:
|
||||
try:
|
||||
_scan_staging_records(runtime, staging_path, progress=_staging_scan_status)
|
||||
with _staging_scan_status_lock:
|
||||
if _staging_scan_status["path"] == staging_path:
|
||||
_staging_scan_status["status"] = "done"
|
||||
except Exception as exc: # noqa: BLE001 — surface any scan error to the poller
|
||||
with _staging_scan_status_lock:
|
||||
_staging_scan_status.update({"status": "error", "error": str(exc)})
|
||||
|
||||
threading.Thread(target=_run, name="staging-scan", daemon=True).start()
|
||||
|
||||
|
||||
def get_staging_records_or_status(runtime: ImportRouteRuntime, staging_path: str,
|
||||
*, grace_seconds: float = 3.0) -> tuple[str, Any]:
|
||||
"""Non-blocking staging access for the page endpoints. Returns ``("ready", records)``
|
||||
when the cache is warm or the scan completes within ``grace_seconds`` (so small/normal
|
||||
folders still answer in a single request), otherwise ``("scanning", status_dict)`` after
|
||||
making sure a background scan is running."""
|
||||
records = _staging_cache_hit(staging_path)
|
||||
if records is not None:
|
||||
return ("ready", records)
|
||||
ensure_background_staging_scan(runtime, staging_path)
|
||||
deadline = time.time() + max(0.0, grace_seconds)
|
||||
while True:
|
||||
records = _staging_cache_hit(staging_path)
|
||||
if records is not None:
|
||||
return ("ready", records)
|
||||
with _staging_scan_status_lock:
|
||||
status = dict(_staging_scan_status)
|
||||
if status.get("status") == "error":
|
||||
return ("error", status)
|
||||
if time.time() >= deadline:
|
||||
return ("scanning", status)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _records_or_scanning_payload(runtime: ImportRouteRuntime, staging_path: str):
|
||||
"""Shared helper for the page endpoints: returns ``(records, None)`` when the scan is
|
||||
ready, or ``(None, payload)`` when a background scan is still running — the caller
|
||||
returns that payload so the page polls + shows progress instead of blocking/timing out.
|
||||
|
||||
A scan error is re-raised so the endpoint's own try/except logs + returns it exactly as
|
||||
when the scan ran inline (preserves the existing error contract)."""
|
||||
state, val = get_staging_records_or_status(runtime, staging_path)
|
||||
if state == "error":
|
||||
raise RuntimeError(val.get("error") or "staging scan failed")
|
||||
if state == "scanning":
|
||||
return None, {"success": True, "scanning": True,
|
||||
"progress": {"scanned": val.get("scanned", 0),
|
||||
"total": val.get("total", 0)}}
|
||||
return val, None
|
||||
|
||||
|
||||
def staging_scan_status(runtime: ImportRouteRuntime) -> tuple[Dict[str, Any], int]:
|
||||
"""Lightweight, instant scan-progress poll for the page (no grace-wait, no file I/O) —
|
||||
``ready`` true once the cache is warm and the files/groups/hints calls will answer fast."""
|
||||
try:
|
||||
staging_path = runtime.get_staging_path()
|
||||
except Exception as exc:
|
||||
return {"success": False, "error": str(exc)}, 500
|
||||
with _staging_scan_status_lock:
|
||||
st = dict(_staging_scan_status)
|
||||
return {
|
||||
"success": True,
|
||||
"ready": _staging_cache_hit(staging_path) is not None,
|
||||
"status": st.get("status", "idle"),
|
||||
"scanned": st.get("scanned", 0),
|
||||
"total": st.get("total", 0),
|
||||
"error": st.get("error"),
|
||||
}, 200
|
||||
|
||||
|
||||
def _scan_staging_records(runtime: ImportRouteRuntime, staging_path: str,
|
||||
*, progress: Optional[Dict[str, Any]] = None) -> list[Dict[str, Any]]:
|
||||
"""Walk staging + read each audio file's tags ONCE, returning per-file records
|
||||
that staging files/groups/hints all derive from. Briefly cached + locked so the
|
||||
page-open trio shares a single scan rather than each re-walking and re-reading.
|
||||
|
||||
``progress`` (optional, default None = unchanged behaviour) is a dict the scan
|
||||
updates live with ``total`` (audio-file count, from a fast first pass) and ``scanned``
|
||||
(tag-reads done so far) so a background runner can report progress. A generation guard
|
||||
keeps a scan that finishes AFTER an import (which bumped ``_staging_scan_generation``)
|
||||
from committing stale records to the cache."""
|
||||
now = time.time()
|
||||
cached = _staging_scan_cache
|
||||
if (cached["records"] is not None and cached["path"] == staging_path
|
||||
and (now - cached["ts"]) < _STAGING_SCAN_TTL):
|
||||
return cached["records"]
|
||||
|
||||
with _STAGING_SCAN_LOCK:
|
||||
# Double-check: another request may have filled the cache while we waited.
|
||||
now = time.time()
|
||||
if (cached["records"] is not None and cached["path"] == staging_path
|
||||
and (now - cached["ts"]) < _STAGING_SCAN_TTL):
|
||||
return cached["records"]
|
||||
|
||||
start_generation = _staging_scan_generation["value"]
|
||||
|
||||
# Pass 1 (fast): collect the audio-file list — no tag I/O — so we know the total.
|
||||
audio_files: list[tuple[str, str, Optional[str]]] = []
|
||||
if os.path.isdir(staging_path):
|
||||
for root, _dirs, filenames in os.walk(staging_path):
|
||||
rel_dir = os.path.relpath(root, staging_path)
|
||||
top_folder = rel_dir.split(os.sep)[0] if rel_dir != "." else None
|
||||
for fname in filenames:
|
||||
if os.path.splitext(fname)[1].lower() in AUDIO_EXTENSIONS:
|
||||
audio_files.append((root, fname, top_folder))
|
||||
if progress is not None:
|
||||
progress["total"] = len(audio_files)
|
||||
progress["scanned"] = 0
|
||||
|
||||
# Pass 2 (slow): read each file's tags, updating progress as we go.
|
||||
records: list[Dict[str, Any]] = []
|
||||
for root, fname, top_folder in audio_files:
|
||||
full_path = os.path.join(root, fname)
|
||||
rel_path = os.path.relpath(full_path, staging_path)
|
||||
meta = runtime.read_staging_file_metadata(full_path, rel_path)
|
||||
records.append({
|
||||
"filename": fname, "rel_path": rel_path, "full_path": full_path,
|
||||
"extension": os.path.splitext(fname)[1].lower(),
|
||||
"title": meta["title"], "album": meta["album"],
|
||||
"artist": meta["artist"], "albumartist": meta["albumartist"],
|
||||
"track_number": meta["track_number"], "disc_number": meta["disc_number"],
|
||||
"top_folder": top_folder,
|
||||
})
|
||||
if progress is not None:
|
||||
progress["scanned"] += 1
|
||||
|
||||
# Generation guard: if an import invalidated the cache mid-scan, these records are
|
||||
# stale — return them to this caller but do NOT commit them as the shared cache.
|
||||
if _staging_scan_generation["value"] == start_generation:
|
||||
_staging_scan_cache.update({"path": staging_path, "ts": time.time(), "records": records})
|
||||
return records
|
||||
|
||||
|
||||
def invalidate_staging_scan_cache() -> None:
|
||||
"""Drop the cached staging scan (call after an import moves/removes files so the
|
||||
next files/groups/hints request reflects the new state immediately). Also bumps the
|
||||
scan generation so an in-flight background scan won't re-commit pre-import records."""
|
||||
_staging_scan_generation["value"] += 1
|
||||
_staging_scan_cache.update({"path": None, "ts": 0.0, "records": None})
|
||||
|
||||
|
||||
def staging_files(runtime: ImportRouteRuntime) -> tuple[Dict[str, Any], int]:
|
||||
"""Scan the staging folder and return audio files with tag metadata."""
|
||||
try:
|
||||
staging_path = runtime.get_staging_path()
|
||||
os.makedirs(staging_path, exist_ok=True)
|
||||
|
||||
records, scanning = _records_or_scanning_payload(runtime, staging_path)
|
||||
if scanning is not None:
|
||||
return scanning, 200
|
||||
files = []
|
||||
for root, _dirs, filenames in os.walk(staging_path):
|
||||
for fname in filenames:
|
||||
ext = os.path.splitext(fname)[1].lower()
|
||||
if ext not in AUDIO_EXTENSIONS:
|
||||
continue
|
||||
full_path = os.path.join(root, fname)
|
||||
rel_path = os.path.relpath(full_path, staging_path)
|
||||
|
||||
files = [
|
||||
{
|
||||
"filename": r["filename"],
|
||||
"rel_path": r["rel_path"],
|
||||
"full_path": r["full_path"],
|
||||
"title": r["title"],
|
||||
"artist": r["albumartist"] or r["artist"] or "Unknown Artist",
|
||||
"album": r["album"],
|
||||
"track_number": r["track_number"],
|
||||
"disc_number": r["disc_number"],
|
||||
"extension": r["extension"],
|
||||
}
|
||||
for r in records
|
||||
]
|
||||
meta = runtime.read_staging_file_metadata(full_path, rel_path)
|
||||
|
||||
files.append(
|
||||
{
|
||||
"filename": fname,
|
||||
"rel_path": rel_path,
|
||||
"full_path": full_path,
|
||||
"title": meta["title"],
|
||||
"artist": meta["albumartist"] or meta["artist"] or "Unknown Artist",
|
||||
"album": meta["album"],
|
||||
"track_number": meta["track_number"],
|
||||
"disc_number": meta["disc_number"],
|
||||
"extension": ext,
|
||||
}
|
||||
)
|
||||
|
||||
files.sort(key=lambda f: f["filename"].lower())
|
||||
return {"success": True, "files": files, "staging_path": staging_path}, 200
|
||||
|
|
@ -301,28 +116,32 @@ def staging_groups(runtime: ImportRouteRuntime) -> tuple[Dict[str, Any], int]:
|
|||
if not os.path.isdir(staging_path):
|
||||
return {"success": True, "groups": []}, 200
|
||||
|
||||
records, scanning = _records_or_scanning_payload(runtime, staging_path)
|
||||
if scanning is not None:
|
||||
return scanning, 200
|
||||
|
||||
album_groups = {}
|
||||
for r in records:
|
||||
album = r["album"]
|
||||
artist = r["albumartist"] or r["artist"]
|
||||
if not album or not artist:
|
||||
continue
|
||||
for root, _dirs, filenames in os.walk(staging_path):
|
||||
for fname in filenames:
|
||||
ext = os.path.splitext(fname)[1].lower()
|
||||
if ext not in AUDIO_EXTENSIONS:
|
||||
continue
|
||||
full_path = os.path.join(root, fname)
|
||||
rel_path = os.path.relpath(full_path, staging_path)
|
||||
|
||||
key = (album.lower().strip(), artist.lower().strip())
|
||||
if key not in album_groups:
|
||||
album_groups[key] = {"album": album.strip(), "artist": artist.strip(), "files": []}
|
||||
album_groups[key]["files"].append(
|
||||
{
|
||||
"filename": r["filename"],
|
||||
"full_path": r["full_path"],
|
||||
"title": r["title"],
|
||||
"track_number": r["track_number"],
|
||||
}
|
||||
)
|
||||
meta = runtime.read_staging_file_metadata(full_path, rel_path)
|
||||
album = meta["album"]
|
||||
artist = meta["albumartist"] or meta["artist"]
|
||||
if not album or not artist:
|
||||
continue
|
||||
|
||||
key = (album.lower().strip(), artist.lower().strip())
|
||||
if key not in album_groups:
|
||||
album_groups[key] = {"album": album.strip(), "artist": artist.strip(), "files": []}
|
||||
album_groups[key]["files"].append(
|
||||
{
|
||||
"filename": fname,
|
||||
"full_path": full_path,
|
||||
"title": meta["title"],
|
||||
"track_number": meta["track_number"],
|
||||
}
|
||||
)
|
||||
|
||||
groups = []
|
||||
for group in album_groups.values():
|
||||
|
|
@ -352,21 +171,30 @@ def staging_hints(runtime: ImportRouteRuntime) -> tuple[Dict[str, Any], int]:
|
|||
if not os.path.isdir(staging_path):
|
||||
return {"success": True, "hints": []}, 200
|
||||
|
||||
records, scanning = _records_or_scanning_payload(runtime, staging_path)
|
||||
if scanning is not None:
|
||||
return scanning, 200
|
||||
|
||||
tag_albums = {}
|
||||
folder_hints = {}
|
||||
for r in records:
|
||||
if r["top_folder"]:
|
||||
folder_hints[r["top_folder"]] = folder_hints.get(r["top_folder"], 0) + 1
|
||||
for root, _dirs, filenames in os.walk(staging_path):
|
||||
audio_files = [f for f in filenames if os.path.splitext(f)[1].lower() in AUDIO_EXTENSIONS]
|
||||
if not audio_files:
|
||||
continue
|
||||
|
||||
album = r["album"]
|
||||
artist = r["artist"] or r["albumartist"]
|
||||
if album:
|
||||
key = (album.strip(), (artist or "").strip())
|
||||
tag_albums[key] = tag_albums.get(key, 0) + 1
|
||||
rel_dir = os.path.relpath(root, staging_path)
|
||||
if rel_dir != ".":
|
||||
top_folder = rel_dir.split(os.sep)[0]
|
||||
folder_hints[top_folder] = folder_hints.get(top_folder, 0) + len(audio_files)
|
||||
|
||||
for fname in audio_files:
|
||||
full_path = os.path.join(root, fname)
|
||||
try:
|
||||
tags = runtime.read_tags(full_path)
|
||||
if tags:
|
||||
album = (tags.get("album") or [None])[0]
|
||||
artist = (tags.get("artist") or (tags.get("albumartist") or [None]))[0]
|
||||
if album:
|
||||
key = (album.strip(), (artist or "").strip())
|
||||
tag_albums[key] = tag_albums.get(key, 0) + 1
|
||||
except Exception as exc:
|
||||
runtime.logger.debug("tag read failed: %s", exc)
|
||||
|
||||
queries = []
|
||||
seen_queries_lower = set()
|
||||
|
|
@ -543,11 +371,6 @@ def album_process(runtime: ImportRouteRuntime, data: Dict[str, Any]) -> tuple[Di
|
|||
)
|
||||
runtime.refresh_import_suggestions_cache()
|
||||
|
||||
# Files just left staging — drop the shared scan so the next files/groups/hints
|
||||
# reflects reality immediately instead of waiting out the cache TTL.
|
||||
if processed > 0:
|
||||
invalidate_staging_scan_cache()
|
||||
|
||||
return {"success": True, "processed": processed, "total": len(matches), "errors": errors}, 200
|
||||
except Exception as exc:
|
||||
runtime.logger.error("Error processing album import: %s", exc)
|
||||
|
|
@ -683,10 +506,6 @@ def singles_process(runtime: ImportRouteRuntime, files: list[Dict[str, Any]]) ->
|
|||
)
|
||||
runtime.refresh_import_suggestions_cache()
|
||||
|
||||
# Files just left staging — drop the shared scan so the list updates immediately.
|
||||
if processed > 0:
|
||||
invalidate_staging_scan_cache()
|
||||
|
||||
return {"success": True, "processed": processed, "total": len(files), "errors": errors}, 200
|
||||
except Exception as exc:
|
||||
runtime.logger.error("Error processing singles import: %s", exc)
|
||||
|
|
|
|||
|
|
@ -252,7 +252,7 @@ def record_library_history_download(context: Dict[str, Any]) -> None:
|
|||
origin, origin_context = derive_download_origin(context)
|
||||
|
||||
db = get_database()
|
||||
_history_id = db.add_library_history_entry(
|
||||
db.add_library_history_entry(
|
||||
event_type="download",
|
||||
title=title,
|
||||
artist_name=artist_name,
|
||||
|
|
@ -270,10 +270,6 @@ def record_library_history_download(context: Dict[str, Any]) -> None:
|
|||
origin_context=origin_context,
|
||||
verification_status=context.get("_verification_status"),
|
||||
)
|
||||
# Stash the row id so the live download task can link to its
|
||||
# library_history row (the Unverified review queue needs it).
|
||||
if isinstance(_history_id, int) and _history_id > 0:
|
||||
context["_history_id"] = _history_id
|
||||
except Exception as e:
|
||||
logger.debug("library history record failed: %s", e)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,289 +0,0 @@
|
|||
"""Audio-completeness guard — detect files whose container duration looks
|
||||
right but whose REAL audio is far shorter, or mostly silence.
|
||||
|
||||
Motivating bug: HiFi/Monochrome HLS assembly can yield a file whose container
|
||||
claims the full track length (e.g. 3:08) while only ~30s of audio actually
|
||||
decodes — the rest is missing. The duration-agreement and quality guards both
|
||||
pass (mutagen reads the container's 3:08 and the format/bitrate are fine), so
|
||||
nothing catches it until you listen. ffmpeg's ``time=`` even reports 0 with no
|
||||
error on such a file, so the robust signal is to DECODE the audio and compare
|
||||
the real duration (sample count / sample rate, via ``astats``) against the
|
||||
container duration. A separate ``silencedetect`` pass also flags genuine
|
||||
silence-padding.
|
||||
|
||||
The parsers here are pure and unit-tested; the ffmpeg invocations are
|
||||
integration glue that fails open (returns None) when ffmpeg or mutagen can't
|
||||
run, so a tooling problem never blocks a legitimate import.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from typing import Optional
|
||||
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("imports.silence")
|
||||
|
||||
# Real decoded audio must cover at least this fraction of the container
|
||||
# duration. A legit file decodes to ~100% (encoder padding aside); a truncated
|
||||
# file decodes to a small fraction (the Blossom file: 30s of a 188s container
|
||||
# = 16%). 0.85 leaves generous headroom against false positives.
|
||||
DEFAULT_MIN_DURATION_RATIO = 0.85
|
||||
|
||||
_SAMPLES_RE = re.compile(r"Number of samples:\s*([0-9]+)")
|
||||
|
||||
# Defaults: treat audio below -50 dB lasting >= 2s as silence, and reject when
|
||||
# more than half the track is silent. A normal song — even with quiet intros/
|
||||
# outros — sits far below 0.5; a 30s-real + padded-silence file sits near 0.85.
|
||||
DEFAULT_NOISE_DB = -50
|
||||
DEFAULT_MIN_SILENCE_S = 2.0
|
||||
DEFAULT_THRESHOLD = 0.5
|
||||
|
||||
_SILENCE_DURATION_RE = re.compile(r"silence_duration:\s*([0-9]+(?:\.[0-9]+)?)")
|
||||
|
||||
|
||||
def silence_ratio_from_output(ffmpeg_stderr: str, total_duration_s: float) -> float:
|
||||
"""Fraction of *total_duration_s* covered by detected silence.
|
||||
|
||||
Sums every ``silence_duration: X`` reported by ffmpeg ``silencedetect``
|
||||
and divides by the track length. Capped at 1.0; returns 0.0 when the
|
||||
duration is unknown/zero or no silence was reported.
|
||||
"""
|
||||
if not total_duration_s or total_duration_s <= 0:
|
||||
return 0.0
|
||||
total_silence = sum(float(m) for m in _SILENCE_DURATION_RE.findall(ffmpeg_stderr or ""))
|
||||
if total_silence <= 0:
|
||||
return 0.0
|
||||
return min(total_silence / total_duration_s, 1.0)
|
||||
|
||||
|
||||
def is_mostly_silent_reason(
|
||||
ffmpeg_stderr: str,
|
||||
total_duration_s: float,
|
||||
*,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
) -> Optional[str]:
|
||||
"""Return a rejection reason when the silent fraction meets *threshold*."""
|
||||
ratio = silence_ratio_from_output(ffmpeg_stderr, total_duration_s)
|
||||
if ratio >= threshold:
|
||||
pct = round(ratio * 100)
|
||||
audible_s = round(total_duration_s * (1 - ratio))
|
||||
return (
|
||||
f"Audio is mostly silent: {pct}% silence (only ~{audible_s}s audible of "
|
||||
f"{round(total_duration_s)}s) — likely a truncated/preview file padded "
|
||||
f"to full length"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _ffmpeg_available() -> bool:
|
||||
try:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
timeout=10, check=True,
|
||||
)
|
||||
return True
|
||||
except (FileNotFoundError, subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def _probe_duration_s(file_path: str) -> Optional[float]:
|
||||
try:
|
||||
from mutagen import File as MutagenFile
|
||||
audio = MutagenFile(file_path)
|
||||
if audio and audio.info and getattr(audio.info, "length", None):
|
||||
return float(audio.info.length)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.debug("silence guard duration probe failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
|
||||
|
||||
def detect_mostly_silent(
|
||||
file_path: str,
|
||||
*,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
noise_db: int = DEFAULT_NOISE_DB,
|
||||
min_silence_s: float = DEFAULT_MIN_SILENCE_S,
|
||||
) -> Optional[str]:
|
||||
"""Run ffmpeg ``silencedetect`` over *file_path* and return a rejection
|
||||
reason when the file is mostly silence, else None.
|
||||
|
||||
Fails open: returns None when ffmpeg/mutagen are unavailable or error, so
|
||||
a tooling problem never quarantines a legitimate file.
|
||||
"""
|
||||
if not _ffmpeg_available():
|
||||
logger.debug("silence guard skipped — ffmpeg not available")
|
||||
return None
|
||||
|
||||
total_duration_s = _probe_duration_s(file_path)
|
||||
if not total_duration_s:
|
||||
return None
|
||||
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-hide_banner", "-nostats", "-i", file_path,
|
||||
"-af", f"silencedetect=noise={noise_db}dB:d={min_silence_s}",
|
||||
"-f", "null", "-",
|
||||
],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE,
|
||||
timeout=120,
|
||||
)
|
||||
except (subprocess.SubprocessError, OSError) as exc:
|
||||
logger.debug("silence guard ffmpeg run failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
|
||||
stderr = proc.stderr.decode("utf-8", errors="replace") if proc.stderr else ""
|
||||
return is_mostly_silent_reason(stderr, total_duration_s, threshold=threshold)
|
||||
|
||||
|
||||
# ── Truncation: real decoded duration vs container duration ────────────────
|
||||
|
||||
def measured_duration_from_astats(astats_stderr: str, sample_rate: int) -> Optional[float]:
|
||||
"""Real decoded audio duration in seconds from ffmpeg ``astats`` output.
|
||||
|
||||
``astats`` reports the per-channel ``Number of samples``; dividing by the
|
||||
sample rate gives the true decoded length. Returns None when the sample
|
||||
count or sample rate is unavailable.
|
||||
"""
|
||||
if not sample_rate or sample_rate <= 0:
|
||||
return None
|
||||
m = _SAMPLES_RE.search(astats_stderr or "")
|
||||
if not m:
|
||||
return None
|
||||
return int(m.group(1)) / float(sample_rate)
|
||||
|
||||
|
||||
def is_dsd_path(file_path: str) -> bool:
|
||||
"""True for DSD audio (.dsf / .dff). The decoded-samples truncation check is
|
||||
invalid for DSD: ffmpeg decodes DSD to PCM at a different rate than the DSD
|
||||
container's 2.8 MHz, so samples ÷ container-sample-rate massively under-counts
|
||||
and would falsely report the file as truncated (#939)."""
|
||||
return os.path.splitext(str(file_path or ''))[1].lower() in ('.dsf', '.dff')
|
||||
|
||||
|
||||
def incomplete_audio_reason(
|
||||
measured_s: Optional[float],
|
||||
container_s: Optional[float],
|
||||
*,
|
||||
min_ratio: float = DEFAULT_MIN_DURATION_RATIO,
|
||||
) -> Optional[str]:
|
||||
"""Return a rejection reason when the real decoded duration falls short of
|
||||
the container duration (a truncated file whose metadata over-states length).
|
||||
"""
|
||||
if not measured_s or not container_s or container_s <= 0:
|
||||
return None
|
||||
if measured_s >= container_s * min_ratio:
|
||||
return None
|
||||
pct = round(measured_s / container_s * 100)
|
||||
return (
|
||||
f"Incomplete audio: only ~{round(measured_s)}s actually decodes of a "
|
||||
f"{round(container_s)}s file ({pct}%) — truncated/broken download "
|
||||
f"(container duration over-states the real audio)"
|
||||
)
|
||||
|
||||
|
||||
def _measured_audio_duration_s(file_path: str, sample_rate: int) -> Optional[float]:
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-nostats", "-i", file_path,
|
||||
"-af", "astats=metadata=1", "-f", "null", "-"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=120,
|
||||
)
|
||||
except (subprocess.SubprocessError, OSError) as exc:
|
||||
logger.debug("astats run failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
stderr = proc.stderr.decode("utf-8", errors="replace") if proc.stderr else ""
|
||||
return measured_duration_from_astats(stderr, sample_rate)
|
||||
|
||||
|
||||
def detect_incomplete_audio(
|
||||
file_path: str,
|
||||
*,
|
||||
min_ratio: float = DEFAULT_MIN_DURATION_RATIO,
|
||||
) -> Optional[str]:
|
||||
"""Decode the file and reject when the real audio is far shorter than the
|
||||
container claims. Fails open when ffmpeg/mutagen are unavailable.
|
||||
"""
|
||||
if not _ffmpeg_available():
|
||||
return None
|
||||
try:
|
||||
from mutagen import File as MutagenFile
|
||||
audio = MutagenFile(file_path)
|
||||
if not (audio and audio.info):
|
||||
return None
|
||||
container_s = float(getattr(audio.info, "length", 0) or 0)
|
||||
sample_rate = int(getattr(audio.info, "sample_rate", 0) or 0)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.debug("container probe failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
|
||||
measured_s = _measured_audio_duration_s(file_path, sample_rate)
|
||||
return incomplete_audio_reason(measured_s, container_s, min_ratio=min_ratio)
|
||||
|
||||
|
||||
def detect_broken_audio(
|
||||
file_path: str,
|
||||
*,
|
||||
min_ratio: float = DEFAULT_MIN_DURATION_RATIO,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
noise_db: int = DEFAULT_NOISE_DB,
|
||||
min_silence_s: float = DEFAULT_MIN_SILENCE_S,
|
||||
) -> Optional[str]:
|
||||
"""Combined post-download audio guard: reject a file that is truncated
|
||||
(real audio far shorter than the container) or mostly silence. Returns the
|
||||
first failure reason, or None when the audio looks complete.
|
||||
|
||||
Runs a SINGLE ffmpeg decode pass with both the ``astats`` (truncation) and
|
||||
``silencedetect`` (silence) filters chained — one decode of the file feeds
|
||||
both checks instead of two full decodes. Halves the CPU cost versus running
|
||||
``detect_incomplete_audio`` and ``detect_mostly_silent`` back to back.
|
||||
|
||||
Fails open: returns None when ffmpeg/mutagen are unavailable or error, so a
|
||||
tooling problem never quarantines a legitimate file.
|
||||
"""
|
||||
if not _ffmpeg_available():
|
||||
logger.debug("audio guard skipped — ffmpeg not available")
|
||||
return None
|
||||
|
||||
try:
|
||||
from mutagen import File as MutagenFile
|
||||
audio = MutagenFile(file_path)
|
||||
if not (audio and audio.info):
|
||||
return None
|
||||
container_s = float(getattr(audio.info, "length", 0) or 0)
|
||||
sample_rate = int(getattr(audio.info, "sample_rate", 0) or 0)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.debug("container probe failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-hide_banner", "-nostats", "-i", file_path,
|
||||
"-af", f"astats=metadata=1,silencedetect=noise={noise_db}dB:d={min_silence_s}",
|
||||
"-f", "null", "-",
|
||||
],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=120,
|
||||
)
|
||||
except (subprocess.SubprocessError, OSError) as exc:
|
||||
logger.debug("audio guard ffmpeg run failed for %s: %s", file_path, exc)
|
||||
return None
|
||||
|
||||
stderr = proc.stderr.decode("utf-8", errors="replace") if proc.stderr else ""
|
||||
|
||||
# Truncation check first (real audio far shorter than the container) — but
|
||||
# NOT for DSD: the astats sample-count ÷ DSD-rate math is invalid there and
|
||||
# would always false-positive (#939). Silence detection below still applies.
|
||||
if not is_dsd_path(file_path):
|
||||
measured_s = measured_duration_from_astats(stderr, sample_rate)
|
||||
reason = incomplete_audio_reason(measured_s, container_s, min_ratio=min_ratio)
|
||||
if reason:
|
||||
return reason
|
||||
|
||||
# Then silence-padding (mostly-silent file).
|
||||
return is_mostly_silent_reason(stderr, container_s, threshold=threshold)
|
||||
|
|
@ -103,12 +103,9 @@ def select_version_mismatch_fallback(
|
|||
# don't guess which the user wants.
|
||||
return None
|
||||
|
||||
# First tried = oldest = best (the retry walks candidates best-first; what
|
||||
# "best" means follows the active ordering — confidence-first by default, or
|
||||
# ranked-target quality when best_quality mode / the rank_candidates_by_quality
|
||||
# toggle is on, so this naturally accepts the highest-quality candidate then).
|
||||
# The id is a "<date>_<time>_<name>" timestamp prefix, so the lexicographically
|
||||
# smallest id is the earliest attempt.
|
||||
# First tried = oldest = highest-confidence (the retry walks candidates
|
||||
# best-first). The id is a "<date>_<time>_<name>" timestamp prefix, so the
|
||||
# lexicographically smallest id is the earliest attempt.
|
||||
return min((e for _, e in candidates), key=lambda e: e["id"])
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@ from datetime import datetime
|
|||
import json
|
||||
from utils.logging_config import get_logger
|
||||
from config.settings import config_manager
|
||||
from core.library.bulk_paginate import paginate_all_items
|
||||
|
||||
# Shared dataclasses live in the neutral media_server package — every
|
||||
# server client used to define a near-identical XTrackInfo /
|
||||
|
|
@ -106,7 +105,6 @@ class JellyfinTrack:
|
|||
self.title = jellyfin_data.get('Name', 'Unknown Track')
|
||||
self.duration = jellyfin_data.get('RunTimeTicks', 0) // 10000 # Convert from ticks to milliseconds
|
||||
self.trackNumber = jellyfin_data.get('IndexNumber')
|
||||
self.discNumber = jellyfin_data.get('ParentIndexNumber') # multi-disc: disc number
|
||||
self.year = jellyfin_data.get('ProductionYear')
|
||||
self.userRating = jellyfin_data.get('UserData', {}).get('Rating')
|
||||
self.addedAt = self._parse_date(jellyfin_data.get('DateCreated'))
|
||||
|
|
@ -513,8 +511,12 @@ class JellyfinClient(MediaServerClient):
|
|||
try:
|
||||
# SIMPLIFIED APPROACH: Fetch all tracks, then all albums separately (robust and fast)
|
||||
logger.info("Fetching all tracks in bulk...")
|
||||
|
||||
def _fetch_tracks_page(start_index, limit):
|
||||
all_tracks = []
|
||||
start_index = 0
|
||||
limit = 10000
|
||||
consecutive_failures = 0
|
||||
|
||||
while True:
|
||||
params = {
|
||||
'ParentId': self.music_library_id,
|
||||
'IncludeItemTypes': 'Audio',
|
||||
|
|
@ -525,19 +527,41 @@ class JellyfinClient(MediaServerClient):
|
|||
'StartIndex': start_index,
|
||||
'Limit': limit
|
||||
}
|
||||
|
||||
response = self._make_request(f'/Users/{self.user_id}/Items', params)
|
||||
return response.get('Items', []) if response else None # None = failed page
|
||||
|
||||
# Page in modest chunks so progress is reported every page — a single
|
||||
# huge silent request used to trip the 300s no-progress watchdog on
|
||||
# slow servers even though it was alive (see bulk_paginate docstring).
|
||||
all_tracks = paginate_all_items(
|
||||
_fetch_tracks_page,
|
||||
report_progress=self._progress_callback,
|
||||
label="tracks",
|
||||
on_retry_wait=lambda: time.sleep(5),
|
||||
)
|
||||
|
||||
if not response:
|
||||
consecutive_failures += 1
|
||||
# Wait before retrying — the server may still be processing the timed-out request
|
||||
time.sleep(5)
|
||||
if limit > 1000:
|
||||
limit = limit // 2
|
||||
consecutive_failures = 0 # Reset — give the smaller batch a fair chance
|
||||
logger.warning(f"Track fetch failed - reducing batch size to {limit}")
|
||||
continue
|
||||
elif consecutive_failures >= 2:
|
||||
logger.warning("Multiple track fetch failures at minimum batch size - stopping")
|
||||
break
|
||||
else:
|
||||
logger.warning("Track fetch failed at minimum batch size - retrying once")
|
||||
continue
|
||||
|
||||
consecutive_failures = 0
|
||||
batch_tracks = response.get('Items', [])
|
||||
if not batch_tracks:
|
||||
break
|
||||
|
||||
all_tracks.extend(batch_tracks)
|
||||
|
||||
if len(batch_tracks) < limit:
|
||||
break
|
||||
|
||||
start_index += limit
|
||||
progress_msg = f"Fetched {len(all_tracks)} tracks so far..."
|
||||
logger.info(f" {progress_msg} (batch size: {limit})")
|
||||
if self._progress_callback:
|
||||
self._progress_callback(progress_msg)
|
||||
|
||||
# Group tracks by album ID for instant lookup
|
||||
self._track_cache = {}
|
||||
for track_data in all_tracks:
|
||||
|
|
@ -553,8 +577,12 @@ class JellyfinClient(MediaServerClient):
|
|||
|
||||
# STEP 2: Fetch all albums in bulk (same proven pattern)
|
||||
logger.info("Fetching all albums in bulk...")
|
||||
|
||||
def _fetch_albums_page(start_index, limit):
|
||||
all_albums = []
|
||||
start_index = 0
|
||||
limit = 10000
|
||||
consecutive_failures = 0
|
||||
|
||||
while True:
|
||||
params = {
|
||||
'ParentId': self.music_library_id,
|
||||
'IncludeItemTypes': 'MusicAlbum',
|
||||
|
|
@ -565,16 +593,41 @@ class JellyfinClient(MediaServerClient):
|
|||
'StartIndex': start_index,
|
||||
'Limit': limit
|
||||
}
|
||||
|
||||
response = self._make_request(f'/Users/{self.user_id}/Items', params)
|
||||
return response.get('Items', []) if response else None # None = failed page
|
||||
|
||||
all_albums = paginate_all_items(
|
||||
_fetch_albums_page,
|
||||
report_progress=self._progress_callback,
|
||||
label="albums",
|
||||
on_retry_wait=lambda: time.sleep(5),
|
||||
)
|
||||
|
||||
if not response:
|
||||
consecutive_failures += 1
|
||||
# Wait before retrying — the server may still be processing the timed-out request
|
||||
time.sleep(5)
|
||||
if limit > 1000:
|
||||
limit = limit // 2
|
||||
consecutive_failures = 0 # Reset — give the smaller batch a fair chance
|
||||
logger.warning(f"Album fetch failed - reducing batch size to {limit}")
|
||||
continue
|
||||
elif consecutive_failures >= 2:
|
||||
logger.warning("Multiple album fetch failures at minimum batch size - stopping")
|
||||
break
|
||||
else:
|
||||
logger.warning("Album fetch failed at minimum batch size - retrying once")
|
||||
continue
|
||||
|
||||
consecutive_failures = 0
|
||||
batch_albums = response.get('Items', [])
|
||||
if not batch_albums:
|
||||
break
|
||||
|
||||
all_albums.extend(batch_albums)
|
||||
|
||||
if len(batch_albums) < limit:
|
||||
break
|
||||
|
||||
start_index += limit
|
||||
progress_msg = f"Fetched {len(all_albums)} albums so far..."
|
||||
logger.info(f" {progress_msg} (batch size: {limit})")
|
||||
if self._progress_callback:
|
||||
self._progress_callback(progress_msg)
|
||||
|
||||
# Group albums by artist ID for instant lookup
|
||||
self._album_cache = {}
|
||||
for album_data in all_albums:
|
||||
|
|
|
|||
|
|
@ -1,93 +0,0 @@
|
|||
"""Paginate a bulk media-server fetch while feeding the no-progress watchdog.
|
||||
|
||||
The library scan fetches every track/album by paging a server API. The DB-update
|
||||
watchdog (``core/database_update_health.py``) kills a job that reports no progress
|
||||
for 300s. The old Jellyfin fetch used a single 10 000-item page, so a whole
|
||||
library came back in ONE request that emitted NO progress while it was in flight —
|
||||
on a slow server that single request exceeded 300s and the watchdog declared the
|
||||
job "stuck" even though it was alive, not hung (Discord: DXP4800 NAS, 7148 tracks,
|
||||
"Fetching all tracks in bulk…").
|
||||
|
||||
``paginate_all_items`` pages at a size chosen so progress is emitted on a cadence
|
||||
set by the PAGE SIZE, not the library size — the watchdog is fed every page, so it
|
||||
can never starve mid-fetch regardless of how big the library is. It is pure: all
|
||||
I/O lives in the injected ``fetch_page``, so the pagination + progress + failure-
|
||||
shrink logic is unit-testable without a server.
|
||||
|
||||
This does NOT change WHAT is fetched (same query, same fields, same items) — only
|
||||
how it's paged and that every page reports progress (the old loop skipped progress
|
||||
on the final/only page, which is the entire bug for a sub-page-size library).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable, List, Optional
|
||||
|
||||
# Page size for bulk library fetches. Small enough that a single request stays
|
||||
# well under the 300s no-progress watchdog even on a slow NAS, and that progress
|
||||
# is reported every page. NOT a performance knob — a resilience/observability one.
|
||||
DEFAULT_PAGE_SIZE = 1000
|
||||
|
||||
# Floor the failure-shrink can reach before giving up — a server that can't return
|
||||
# even this many items in one request is genuinely struggling.
|
||||
DEFAULT_MIN_PAGE_SIZE = 250
|
||||
|
||||
|
||||
def paginate_all_items(
|
||||
fetch_page: Callable[[int, int], Optional[List[Any]]],
|
||||
*,
|
||||
report_progress: Optional[Callable[[str], None]] = None,
|
||||
label: str = "items",
|
||||
page_size: int = DEFAULT_PAGE_SIZE,
|
||||
min_page_size: int = DEFAULT_MIN_PAGE_SIZE,
|
||||
on_retry_wait: Optional[Callable[[], None]] = None,
|
||||
) -> List[Any]:
|
||||
"""Page through ``fetch_page(start_index, limit)`` until the server is drained.
|
||||
|
||||
``fetch_page`` returns the page's items (a list, possibly empty = end), or
|
||||
``None`` to signal a FAILED request (timeout/error) — on failure the page size
|
||||
is halved down to ``min_page_size`` and retried, then abandoned after two
|
||||
consecutive failures at the floor.
|
||||
|
||||
Progress is reported after EVERY non-empty page (including the final/only one),
|
||||
so a no-progress watchdog is fed on a cadence set by ``page_size`` — never by
|
||||
the total library size. Returns every item gathered.
|
||||
"""
|
||||
items: List[Any] = []
|
||||
start_index = 0
|
||||
limit = page_size
|
||||
consecutive_failures = 0
|
||||
|
||||
while True:
|
||||
batch = fetch_page(start_index, limit)
|
||||
|
||||
if batch is None: # failed request
|
||||
consecutive_failures += 1
|
||||
if on_retry_wait is not None:
|
||||
on_retry_wait()
|
||||
if limit > min_page_size:
|
||||
limit = max(min_page_size, limit // 2)
|
||||
consecutive_failures = 0 # give the smaller batch a fair chance
|
||||
continue
|
||||
if consecutive_failures >= 2:
|
||||
break # struggling at the floor — stop with what we have
|
||||
continue
|
||||
|
||||
consecutive_failures = 0
|
||||
if not batch:
|
||||
break # drained
|
||||
|
||||
items.extend(batch)
|
||||
# Feed the watchdog on EVERY page — this is the line the old loop only ran
|
||||
# when there was a *next* page, so a sub-page-size library reported nothing.
|
||||
if report_progress is not None:
|
||||
report_progress(f"Fetched {len(items)} {label} so far...")
|
||||
|
||||
if len(batch) < limit:
|
||||
break # last (partial) page
|
||||
start_index += limit
|
||||
|
||||
return items
|
||||
|
||||
|
||||
__all__ = ["paginate_all_items", "DEFAULT_PAGE_SIZE", "DEFAULT_MIN_PAGE_SIZE"]
|
||||
|
|
@ -138,25 +138,10 @@ def _collect_base_dirs(
|
|||
except Exception as e:
|
||||
logger.debug("music paths read failed: %s", e)
|
||||
|
||||
# Normalize to absolute forms so resolution does NOT depend on the calling
|
||||
# thread's CWD. A relative config like "./Transfer" otherwise only resolves
|
||||
# when os.path.isdir("./Transfer") happens to be true from the current CWD —
|
||||
# which fails in background workers whose CWD isn't the app root, leaving
|
||||
# base_dirs empty and every track "unresolved". For each candidate we try
|
||||
# the raw form first (cheap, preserves an already-absolute path), then its
|
||||
# os.path.abspath() form so "./Transfer" → "/app/Transfer".
|
||||
expanded: list[str] = []
|
||||
for c in candidates:
|
||||
if not c:
|
||||
continue
|
||||
expanded.append(c)
|
||||
if not os.path.isabs(c):
|
||||
expanded.append(os.path.abspath(c))
|
||||
|
||||
# De-duplicate while preserving order, drop empties / non-existent dirs.
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for c in expanded:
|
||||
for c in candidates:
|
||||
if not c or c in seen:
|
||||
continue
|
||||
seen.add(c)
|
||||
|
|
@ -239,23 +224,10 @@ def resolve_library_file_path_with_diagnostic(
|
|||
if not base_dirs:
|
||||
return None, attempt
|
||||
|
||||
# Try progressively shorter path suffixes against each base dir.
|
||||
#
|
||||
# Start at index 0 so a clean RELATIVE library path is tried in FULL first.
|
||||
# SoulSync's own library scanner stores paths like
|
||||
# "Asketa/Another Side/01 - Track.flac" (no leading slash) — index 0 is the
|
||||
# artist folder and dropping it (the old range(1, ...)) meant the artist
|
||||
# segment was never joined, so nothing under transfer/ ever resolved and
|
||||
# every track looked unreadable to the quality scanner.
|
||||
#
|
||||
# For ABSOLUTE media-server paths ("/music/Artist/Album/track.flac") index 0
|
||||
# is the empty leading segment and i=0 yields os.path.join(base, "", ...) ==
|
||||
# base/Artist/... which simply won't exist and harmlessly falls through to
|
||||
# i=1 ("music/...") etc. A Windows drive part ("E:") at i=0 likewise just
|
||||
# fails on POSIX and falls through. So starting at 0 is safe for every form
|
||||
# and only ADDS the relative-full-path match that was missing.
|
||||
# Skip index 0 to avoid drive-letter / leading-slash artifacts
|
||||
# (e.g. "E:" or "" from a leading "/").
|
||||
for base in base_dirs:
|
||||
for i in range(0, len(path_parts)):
|
||||
for i in range(1, len(path_parts)):
|
||||
candidate = os.path.join(base, *path_parts[i:])
|
||||
if os.path.exists(candidate):
|
||||
return candidate, attempt
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@ without a source ID are reported back to the caller and skipped
|
|||
entirely.
|
||||
"""
|
||||
|
||||
import errno
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
|
|
@ -35,7 +34,7 @@ import time
|
|||
import uuid
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Dict, List, Optional, Set, Tuple
|
||||
from typing import Any, Callable, Dict, List, Optional, Set
|
||||
|
||||
# Per-album track concurrency. Matches the download workers' per-batch
|
||||
# concurrency (3) so reorganize feels comparable to a fresh download.
|
||||
|
|
@ -101,7 +100,6 @@ _ALBUM_ID_COLUMNS = {
|
|||
'deezer': 'deezer_id',
|
||||
'discogs': 'discogs_id',
|
||||
'hydrabase': 'soul_id',
|
||||
'musicbrainz': 'musicbrainz_release_id',
|
||||
}
|
||||
|
||||
# Human-facing label for each source.
|
||||
|
|
@ -1102,12 +1100,6 @@ def preview_album_reorganize(
|
|||
'track_number': track.get('track_number', 0),
|
||||
'current_path': _trim_to_transfer(db_path, resolved, transfer_dir),
|
||||
'new_path': '',
|
||||
# Absolute on-disk paths (additive). `current_path`/`new_path` above are
|
||||
# display-trimmed; these carry the real paths so the rename-only executor
|
||||
# acts on EXACTLY what the preview computed — no separate path logic that
|
||||
# could drift from what the user saw (#875).
|
||||
'current_path_abs': resolved or '',
|
||||
'new_path_abs': '',
|
||||
'file_exists': resolved is not None,
|
||||
'unchanged': False,
|
||||
'collision': False,
|
||||
|
|
@ -1155,7 +1147,6 @@ def preview_album_reorganize(
|
|||
new_full, _ok = build_final_path_fn(
|
||||
context, spotify_artist, album_info, file_ext, create_dirs=False
|
||||
)
|
||||
item['new_path_abs'] = new_full or ''
|
||||
item['new_path'] = (
|
||||
os.path.relpath(new_full, transfer_dir)
|
||||
if transfer_dir and new_full and new_full.startswith(transfer_dir)
|
||||
|
|
@ -1815,150 +1806,6 @@ def reorganize_album(
|
|||
return summary
|
||||
|
||||
|
||||
def _rename_track_in_place(current_abs: str, new_abs: str) -> Tuple[bool, Optional[str]]:
|
||||
"""Move ONE file from ``current_abs`` to ``new_abs`` in place — no copy, no re-tag,
|
||||
no post-processing. Creates the destination folder, carries sibling-format files
|
||||
(e.g. a lossy ``.opus`` alongside the ``.flac``) along with the renamed stem, and
|
||||
falls back to a cross-device move when the rename crosses a filesystem boundary.
|
||||
|
||||
Refuses to overwrite a DIFFERENT existing file at the destination (returns an error
|
||||
instead) — never silent data loss. Returns ``(ok, error_message)``.
|
||||
"""
|
||||
try:
|
||||
if current_abs and not os.path.exists(current_abs):
|
||||
return False, 'source file no longer on disk'
|
||||
same = os.path.normpath(current_abs) == os.path.normpath(new_abs)
|
||||
if os.path.exists(new_abs) and not same:
|
||||
return False, 'destination already exists'
|
||||
os.makedirs(os.path.dirname(new_abs), exist_ok=True)
|
||||
# Carry sibling-format audio to the same destination with the renamed stem —
|
||||
# mirrors _finalize_track so lossy-copy pairs don't get orphaned.
|
||||
for sibling_src in _find_sibling_audio_files(current_abs):
|
||||
_move_sibling_to_destination(sibling_src, new_abs)
|
||||
try:
|
||||
os.rename(current_abs, new_abs)
|
||||
except OSError as e:
|
||||
if getattr(e, 'errno', None) == errno.EXDEV:
|
||||
shutil.move(current_abs, new_abs) # crosses a filesystem boundary
|
||||
else:
|
||||
raise
|
||||
return True, None
|
||||
except Exception as e:
|
||||
return False, str(e)
|
||||
|
||||
|
||||
def reorganize_album_rename_only(
|
||||
*,
|
||||
album_id: str,
|
||||
db,
|
||||
transfer_dir: str,
|
||||
resolve_file_path_fn: Callable[[Optional[str]], Optional[str]],
|
||||
build_final_path_fn: Callable,
|
||||
update_track_path_fn: Optional[Callable[[object, str], None]] = None,
|
||||
cleanup_empty_dir_fn: Optional[Callable[[str], None]] = None,
|
||||
on_progress: Optional[Callable[[dict], None]] = None,
|
||||
primary_source: Optional[str] = None,
|
||||
strict_source: bool = False,
|
||||
metadata_source: str = 'api',
|
||||
stop_check: Optional[Callable[[], bool]] = None,
|
||||
preview_fn: Optional[Callable] = None,
|
||||
) -> dict:
|
||||
"""RENAME-ONLY reorganize (#875): move each track's file to the path the current
|
||||
naming scheme dictates, and nothing else — no copy-to-staging, no re-tag, no
|
||||
quality/AcoustID checks.
|
||||
|
||||
It acts on EXACTLY what :func:`preview_album_reorganize` computed (injected via
|
||||
``preview_fn`` for testability), so the apply can never disagree with what the user
|
||||
saw, and ONLY files whose path actually changes are touched — files marked
|
||||
``unchanged`` are skipped, which is what keeps a rename from rewriting the whole
|
||||
album (the #875 complaint). Tags and audio are left byte-for-byte alone.
|
||||
|
||||
Returns the same summary shape as :func:`reorganize_album`.
|
||||
"""
|
||||
preview_fn = preview_fn or preview_album_reorganize
|
||||
summary = {
|
||||
'status': 'completed', 'source': None, 'total': 0,
|
||||
'moved': 0, 'skipped': 0, 'failed': 0, 'errors': [],
|
||||
}
|
||||
|
||||
def _emit(**updates):
|
||||
if on_progress is None:
|
||||
return
|
||||
try:
|
||||
on_progress(updates)
|
||||
except Exception as e:
|
||||
logger.debug("[Reorganize/rename] progress emit failed: %s", e)
|
||||
|
||||
preview = preview_fn(
|
||||
album_id=album_id, db=db, transfer_dir=transfer_dir,
|
||||
resolve_file_path_fn=resolve_file_path_fn,
|
||||
build_final_path_fn=build_final_path_fn,
|
||||
primary_source=primary_source, strict_source=strict_source,
|
||||
metadata_source=metadata_source,
|
||||
)
|
||||
summary['source'] = preview.get('source')
|
||||
if not preview.get('success'):
|
||||
summary['status'] = preview.get('status', 'error')
|
||||
return summary
|
||||
|
||||
tracks = preview.get('tracks', [])
|
||||
summary['total'] = len(tracks)
|
||||
src_dirs_touched: Set[str] = set()
|
||||
|
||||
for t in tracks:
|
||||
if stop_check and stop_check():
|
||||
break
|
||||
title = t.get('title', 'Unknown')
|
||||
_emit(current_track=title)
|
||||
|
||||
# Skip anything that isn't a real, changing move. `unchanged` is the key one —
|
||||
# it's why a rename no longer rewrites files whose name didn't change.
|
||||
if (not t.get('matched') or t.get('unchanged')
|
||||
or t.get('collision') or not t.get('new_path_abs')):
|
||||
summary['skipped'] += 1
|
||||
_emit(skipped=summary['skipped'])
|
||||
continue
|
||||
|
||||
current_abs = t.get('current_path_abs')
|
||||
new_abs = t.get('new_path_abs')
|
||||
ok, err = _rename_track_in_place(current_abs, new_abs)
|
||||
if not ok:
|
||||
summary['failed'] += 1
|
||||
summary['errors'].append({
|
||||
'track_id': t.get('track_id'), 'title': title,
|
||||
'error': err or 'rename failed',
|
||||
})
|
||||
_emit(failed=summary['failed'], errors=list(summary['errors']))
|
||||
continue
|
||||
|
||||
# File is at its new home — update the DB directly (authoritative; no need to
|
||||
# round-trip through a server scan to learn what we just did). On DB failure the
|
||||
# file still moved; a library scan reconciles it, so we don't fail the track.
|
||||
if update_track_path_fn:
|
||||
try:
|
||||
update_track_path_fn(t.get('track_id'), new_abs)
|
||||
except Exception as db_err:
|
||||
logger.warning(
|
||||
"[Reorganize/rename] DB path update failed for %s: %s "
|
||||
"(file moved to %s; a scan will reconcile)",
|
||||
t.get('track_id'), db_err, new_abs,
|
||||
)
|
||||
if current_abs:
|
||||
src_dirs_touched.add(os.path.dirname(current_abs))
|
||||
summary['moved'] += 1
|
||||
_emit(moved=summary['moved'],
|
||||
processed=summary['moved'] + summary['skipped'] + summary['failed'])
|
||||
|
||||
if cleanup_empty_dir_fn:
|
||||
for src_dir in src_dirs_touched:
|
||||
try:
|
||||
cleanup_empty_dir_fn(src_dir)
|
||||
except Exception as e:
|
||||
logger.debug("[Reorganize/rename] cleanup of %s failed: %s", src_dir, e)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
def _find_artist_dir(dest_path: str, transfer_dir: str) -> Optional[str]:
|
||||
"""Walk up from ``dest_path`` until the parent equals ``transfer_dir``;
|
||||
the directory at that point is the artist folder. Returns None if
|
||||
|
|
|
|||
|
|
@ -47,10 +47,7 @@ class LidarrDownloadClient(DownloadSourcePlugin):
|
|||
if download_path is None:
|
||||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
self.active_downloads: Dict[str, Dict[str, Any]] = {}
|
||||
self._download_lock = threading.Lock()
|
||||
|
|
|
|||
|
|
@ -291,13 +291,6 @@ class ListeningStatsWorker:
|
|||
if not (top_artists or top_albums or top_tracks):
|
||||
return
|
||||
|
||||
# Normalize image URLs HERE, at cache-build time, not on every /api/stats/cached
|
||||
# read. normalize_image_url registers each URL in the image cache (a SQLite write
|
||||
# under a lock) — doing that per-request made the "instant" stats endpoint take ~20s
|
||||
# on HDD-backed installs (#935). Done once per background rebuild it's off the hot path,
|
||||
# and the read just returns the already-browser-safe URLs.
|
||||
from core.metadata import normalize_image_url as _fix_image
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
|
|
@ -331,7 +324,7 @@ class ListeningStatsWorker:
|
|||
key = (artist.get('name') or '').lower()
|
||||
r = artist_rows.get(key)
|
||||
if r:
|
||||
artist['image_url'] = _fix_image(r[1]) or None
|
||||
artist['image_url'] = r[1] or None
|
||||
artist['id'] = r[2]
|
||||
artist['global_listeners'] = r[3]
|
||||
artist['global_playcount'] = r[4]
|
||||
|
|
@ -363,7 +356,7 @@ class ListeningStatsWorker:
|
|||
key = (album.get('name') or '').lower()
|
||||
r = album_rows.get(key)
|
||||
if r:
|
||||
album['image_url'] = _fix_image(r[1]) or None
|
||||
album['image_url'] = r[1] or None
|
||||
album['id'] = r[2]
|
||||
album['artist_id'] = r[3]
|
||||
|
||||
|
|
@ -402,7 +395,7 @@ class ListeningStatsWorker:
|
|||
(track.get('artist') or '').lower())
|
||||
r = track_rows.get(key)
|
||||
if r:
|
||||
track['image_url'] = _fix_image(r[2]) or None
|
||||
track['image_url'] = r[2] or None
|
||||
track['id'] = r[3]
|
||||
track['artist_id'] = r[4]
|
||||
except Exception as e:
|
||||
|
|
|
|||
|
|
@ -211,7 +211,7 @@ def pick_canonical_release(
|
|||
# core.library_reorganize._ALBUM_ID_COLUMNS — a test pins them in sync). A manual
|
||||
# match on any of these should pin/lock the canonical version (#758); a match on
|
||||
# a source the canonical tools don't read (e.g. lastfm) has no version to pin.
|
||||
CANONICAL_ALBUM_SOURCES = frozenset({'spotify', 'itunes', 'deezer', 'discogs', 'hydrabase', 'musicbrainz'})
|
||||
CANONICAL_ALBUM_SOURCES = frozenset({'spotify', 'itunes', 'deezer', 'discogs', 'hydrabase'})
|
||||
|
||||
|
||||
def should_pin_manual_canonical(entity_type: str, source: str) -> bool:
|
||||
|
|
|
|||
|
|
@ -167,7 +167,6 @@ def track_already_owned(
|
|||
album_name: str,
|
||||
server_source: Optional[str],
|
||||
confidence_threshold: float = 0.7,
|
||||
candidate_tracks: Optional[List[Any]] = None,
|
||||
) -> bool:
|
||||
"""Return True if the track is already in the user's library.
|
||||
|
||||
|
|
@ -187,15 +186,6 @@ def track_already_owned(
|
|||
original deleted) matches just fine — track_name + artist + album
|
||||
don't change with format.
|
||||
|
||||
``candidate_tracks`` (when not None) is the artist's library tracks,
|
||||
pre-fetched ONCE by the caller, so the check scores in-memory instead
|
||||
of firing per-track fuzzy SQL scans against the whole library. Pass an
|
||||
empty list for an artist the user owns nothing of — it still routes
|
||||
through the fast in-memory path (scores against zero candidates →
|
||||
instant "not owned") rather than the slow per-track search. None
|
||||
preserves the original per-track-SQL behaviour for callers that don't
|
||||
pre-fetch.
|
||||
|
||||
Returns False on any exception so a transient DB hiccup doesn't
|
||||
silently nuke a discography fetch — a redundant wishlist add is
|
||||
much cheaper to recover from than a missed track.
|
||||
|
|
@ -208,7 +198,6 @@ def track_already_owned(
|
|||
confidence_threshold=confidence_threshold,
|
||||
server_source=server_source,
|
||||
album=album_name or None,
|
||||
candidate_tracks=candidate_tracks,
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -230,20 +230,15 @@ def get_spotify_client_for_profile(profile_id: Optional[int] = None):
|
|||
return client
|
||||
|
||||
try:
|
||||
from core.spotify_client import SpotifyClient, normalize_spotify_oauth_config, SPOTIFY_OAUTH_SCOPE
|
||||
from core.spotify_client import SpotifyClient
|
||||
from spotipy.oauth2 import SpotifyOAuth
|
||||
import spotipy
|
||||
|
||||
normalized_creds = normalize_spotify_oauth_config({
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"redirect_uri": redirect_uri,
|
||||
})
|
||||
auth_manager = SpotifyOAuth(
|
||||
client_id=normalized_creds.get("client_id", client_id),
|
||||
client_secret=normalized_creds.get("client_secret", client_secret),
|
||||
redirect_uri=normalized_creds.get("redirect_uri", redirect_uri),
|
||||
scope=SPOTIFY_OAUTH_SCOPE,
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
redirect_uri=redirect_uri,
|
||||
scope="user-library-read user-read-private playlist-read-private playlist-read-collaborative user-read-email user-follow-read",
|
||||
cache_path=cache_path,
|
||||
state=f"profile_{profile_id}",
|
||||
)
|
||||
|
|
@ -357,26 +352,10 @@ def get_hydrabase_client(allow_fallback: bool = True, require_enabled: bool = Tr
|
|||
return None
|
||||
|
||||
|
||||
def get_configured_primary_source() -> str:
|
||||
"""Return metadata.fallback_source as stored in config, without runtime downgrade.
|
||||
|
||||
Unlike get_primary_source(), this never probes Spotify auth. Use at boot and
|
||||
anywhere blocking network I/O must be avoided (gunicorn worker import, Docker
|
||||
cold start when Spotify is unreachable).
|
||||
"""
|
||||
_default = METADATA_SOURCE_PRIORITY[0]
|
||||
return _get_config_value("metadata.fallback_source", _default) or _default
|
||||
|
||||
|
||||
def get_primary_source(spotify_client_factory: Optional[MetadataClientFactory] = None) -> str:
|
||||
"""Return configured primary metadata source."""
|
||||
from core.boot_phase import is_boot_phase
|
||||
|
||||
if is_boot_phase():
|
||||
return get_configured_primary_source()
|
||||
|
||||
_default = METADATA_SOURCE_PRIORITY[0]
|
||||
source = get_configured_primary_source()
|
||||
source = _get_config_value("metadata.fallback_source", _default) or _default
|
||||
|
||||
if source == "spotify":
|
||||
try:
|
||||
|
|
@ -463,19 +442,7 @@ def get_primary_source_status(
|
|||
musicbrainz_client_factory: Optional[MetadataClientFactory] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Return a generic status snapshot for the active primary metadata source."""
|
||||
from core.boot_phase import is_boot_phase
|
||||
|
||||
source = _get_config_value("metadata.fallback_source", "deezer") or "deezer"
|
||||
if is_boot_phase():
|
||||
display_source = source
|
||||
if source == "spotify" and _get_config_value("metadata.spotify_free", False):
|
||||
display_source = "spotify_free"
|
||||
return {
|
||||
"source": display_source,
|
||||
"connected": False,
|
||||
"response_time": 0,
|
||||
}
|
||||
|
||||
started = time.time()
|
||||
connected = False
|
||||
|
||||
|
|
@ -543,13 +510,9 @@ def get_client_for_source(
|
|||
musicbrainz_client_factory: Optional[MetadataClientFactory] = None,
|
||||
):
|
||||
"""Return exact client for a source, or None if unavailable."""
|
||||
from core.boot_phase import is_boot_phase
|
||||
|
||||
if source == "spotify":
|
||||
try:
|
||||
client = get_spotify_client(client_factory=spotify_client_factory)
|
||||
if is_boot_phase():
|
||||
return client if client and getattr(client, "sp", None) else None
|
||||
if client and client.is_spotify_authenticated():
|
||||
return client
|
||||
except Exception as e:
|
||||
|
|
|
|||
|
|
@ -52,27 +52,6 @@ from typing import List, Optional, Sequence
|
|||
from core.metadata.types import Track
|
||||
|
||||
|
||||
def build_combined_search_query(track: str = '', artist: str = '',
|
||||
legacy: str = '') -> str:
|
||||
"""Combine a track + artist into a PLAIN, source-agnostic search query.
|
||||
|
||||
Deliberately NOT field-scoped (``track:"X" artist:"Y"``). That Spotify/Lucene
|
||||
filter syntax leaks to non-Spotify sources when a search falls back: Deezer
|
||||
closed the connection on it outright (``RemoteDisconnected``), and even sources
|
||||
that parse it over-constrain and miss the canonical cut (the iTunes/Deezer search
|
||||
endpoints already dropped it for that reason). The matching ``/search_tracks``
|
||||
endpoints rerank by expected title/artist afterward, so precision is recovered
|
||||
without the brittle field syntax.
|
||||
|
||||
Falls back to ``legacy`` when neither track nor artist is given. Returns ``''``
|
||||
when there's nothing to search (caller decides how to handle empty).
|
||||
"""
|
||||
parts = [p.strip() for p in (track, artist) if p and p.strip()]
|
||||
if parts:
|
||||
return ' '.join(parts)
|
||||
return (legacy or '').strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pattern tables — public so tests can introspect, callers can extend
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -49,7 +49,6 @@ from core.metadata.registry import (
|
|||
get_discogs_client,
|
||||
get_hydrabase_client,
|
||||
get_itunes_client,
|
||||
get_configured_primary_source,
|
||||
get_primary_client,
|
||||
get_primary_source,
|
||||
get_primary_source_label,
|
||||
|
|
@ -118,7 +117,6 @@ __all__ = [
|
|||
"get_metadata_service",
|
||||
"get_musicmap_similar_artists",
|
||||
"get_primary_client",
|
||||
"get_configured_primary_source",
|
||||
"get_primary_source",
|
||||
"get_primary_source_label",
|
||||
"get_spotify_client_for_profile",
|
||||
|
|
|
|||
|
|
@ -110,7 +110,6 @@ class NavidromeTrack:
|
|||
self.title = navidrome_data.get('title', 'Unknown Track')
|
||||
self.duration = navidrome_data.get('duration', 0) * 1000 # Convert to milliseconds
|
||||
self.trackNumber = navidrome_data.get('track')
|
||||
self.discNumber = navidrome_data.get('discNumber') # multi-disc: disc number
|
||||
self.year = navidrome_data.get('year')
|
||||
self.userRating = navidrome_data.get('userRating')
|
||||
self.addedAt = self._parse_date(navidrome_data.get('created'))
|
||||
|
|
|
|||
|
|
@ -25,4 +25,3 @@ from core.personalized.generators import daily_mix # noqa: F401
|
|||
from core.personalized.generators import fresh_tape # noqa: F401
|
||||
from core.personalized.generators import archives # noqa: F401
|
||||
from core.personalized.generators import seasonal_mix # noqa: F401
|
||||
from core.personalized.generators import listening_mix # noqa: F401
|
||||
|
|
|
|||
|
|
@ -1,68 +0,0 @@
|
|||
"""Listening Mix (#913) generator.
|
||||
|
||||
Reads the full, render-ready track dicts the watchlist scan stored under the
|
||||
``listening_recs_tracks_full`` metadata key — built from the recommended artists'
|
||||
top tracks (see ``core.watchlist_scanner._build_listening_recommendations``) — and
|
||||
coerces them into ``Track`` records.
|
||||
|
||||
Unlike Fresh Tape / Archives this needs NO discovery-pool hydration: the stored
|
||||
dicts are already complete, so the snapshot can't shrink when the pool rotates. It
|
||||
also means the generator is a pure read — generation/network all happen during the
|
||||
scan; this just hands the stored tracks to the personalized manager so the mix can
|
||||
participate in the Sync-page mirror + Auto-Sync pipeline like every other kind."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, List
|
||||
|
||||
from core.personalized.specs import PlaylistKindSpec, get_registry
|
||||
from core.personalized.types import PlaylistConfig, Track
|
||||
|
||||
|
||||
KIND = 'listening_mix'
|
||||
METADATA_KEY = 'listening_recs_tracks_full'
|
||||
|
||||
|
||||
def generate(deps: Any, variant: str, config: PlaylistConfig) -> List[Track]:
|
||||
"""Return the stored Listening Mix tracks, trimmed to ``config.limit``.
|
||||
|
||||
Empty (not an error) when the scan hasn't produced a mix yet — the manager
|
||||
preserves any prior snapshot rather than dropping it. Tolerates a missing/garbage
|
||||
metadata blob the same way.
|
||||
"""
|
||||
db = getattr(deps, 'database', None) or (deps.get('database') if isinstance(deps, dict) else None)
|
||||
if db is None:
|
||||
raise RuntimeError("Listening Mix generator deps missing `database`")
|
||||
|
||||
raw = db.get_metadata(METADATA_KEY)
|
||||
if not raw:
|
||||
return []
|
||||
try:
|
||||
rows = json.loads(raw) or []
|
||||
except (ValueError, TypeError):
|
||||
return []
|
||||
|
||||
tracks: List[Track] = []
|
||||
for d in rows:
|
||||
if not isinstance(d, dict):
|
||||
continue
|
||||
tracks.append(Track.from_dict(d))
|
||||
if len(tracks) >= config.limit:
|
||||
break
|
||||
return tracks
|
||||
|
||||
|
||||
SPEC = PlaylistKindSpec(
|
||||
kind=KIND,
|
||||
name_template='Your Listening Mix',
|
||||
description='Tracks from artists matched to what you actually listen to.',
|
||||
default_config=PlaylistConfig(limit=50, max_per_album=5, max_per_artist=3),
|
||||
generator=generate,
|
||||
requires_variant=False,
|
||||
tags=['curated', 'listening'],
|
||||
)
|
||||
|
||||
|
||||
if get_registry().get(KIND) is None:
|
||||
get_registry().register(SPEC)
|
||||
|
|
@ -111,10 +111,7 @@ class PersonalizedPlaylistsService:
|
|||
Either value can be None to skip that filter for that source.
|
||||
|
||||
- Spotify: 60 / 40 (the existing 0-100 popularity scale)
|
||||
- Deezer: 60 / 50 (ALSO 0-100 — the discovery pool SYNTHESIZES deezer popularity to a
|
||||
0-100 score at scan time, capped at 100; it is NOT Deezer's raw rank. The old
|
||||
500000/100000 rank thresholds matched nothing, so Popular Picks came back empty for
|
||||
deezer users while Hidden Gems' < 100000 caught the whole pool.)
|
||||
- Deezer: 500000 / 100000 (rank values — ballpark from real data)
|
||||
- iTunes / others: None / None (no popularity data; fall back to no
|
||||
threshold filter, just diversity-on-random)
|
||||
|
||||
|
|
@ -124,7 +121,7 @@ class PersonalizedPlaylistsService:
|
|||
if normalized == 'spotify':
|
||||
return 60, 40
|
||||
if normalized == 'deezer':
|
||||
return 60, 50
|
||||
return 500_000, 100_000
|
||||
# iTunes, hydrabase, anything else — no usable popularity data
|
||||
return None, None
|
||||
|
||||
|
|
|
|||
|
|
@ -29,7 +29,6 @@ from config.settings import config_manager
|
|||
|
||||
# Import Soulseek data structures for drop-in replacement compatibility
|
||||
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
|
||||
from core.quality.source_map import quality_from_qobuz, quality_tier_for_source
|
||||
|
||||
logger = get_logger("qobuz_client")
|
||||
|
||||
|
|
@ -120,10 +119,7 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"Qobuz client using download path: {self.download_path}")
|
||||
|
||||
|
|
@ -164,8 +160,6 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
|
||||
def _restore_session(self):
|
||||
"""Try to restore saved session from config."""
|
||||
from core.boot_phase import is_boot_phase
|
||||
|
||||
saved = config_manager.get('qobuz.session', {})
|
||||
app_id = saved.get('app_id', '')
|
||||
app_secret = saved.get('app_secret', '')
|
||||
|
|
@ -180,10 +174,6 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
'X-User-Auth-Token': self.user_auth_token,
|
||||
})
|
||||
|
||||
if is_boot_phase():
|
||||
logger.info("Loaded Qobuz session from config (verification deferred until after boot)")
|
||||
return
|
||||
|
||||
# Verify the token is still valid
|
||||
try:
|
||||
resp = self.session.get(
|
||||
|
|
@ -721,26 +711,6 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
logger.error(f"Error getting Qobuz track {track_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_track_result(self, track_id):
|
||||
"""Fetch ONE track by ID and convert it to a downloadable TrackResult.
|
||||
|
||||
Used when a pasted Qobuz link must inject the EXACT track: a text search
|
||||
for an obscure track's name often doesn't surface it at all, so bubbling
|
||||
the matching id is a no-op (#932). Returns None on any failure so the
|
||||
caller falls back to the text-search path."""
|
||||
try:
|
||||
track = self.get_track(track_id)
|
||||
if not track:
|
||||
return None
|
||||
quality_key = quality_tier_for_source('qobuz', default='lossless')
|
||||
quality_info = QOBUZ_QUALITY_MAP.get(quality_key, QOBUZ_QUALITY_MAP['lossless'])
|
||||
# require_streamable=False: this is the exact track the user linked — don't
|
||||
# let a missing/absent 'streamable' flag in track/get drop it (#932).
|
||||
return self._qobuz_to_track_result(track, quality_info, require_streamable=False)
|
||||
except Exception as e:
|
||||
logger.debug(f"get_track_result failed for Qobuz {track_id}: {e}")
|
||||
return None
|
||||
|
||||
# ===================== Playlists & Favorites =====================
|
||||
#
|
||||
# Qobuz playlist sync surface — mirrors the Tidal client contract
|
||||
|
|
@ -1017,7 +987,7 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
return ([], [])
|
||||
|
||||
# Get configured quality for display
|
||||
quality_key = quality_tier_for_source('qobuz', default='lossless')
|
||||
quality_key = config_manager.get('qobuz.quality', 'lossless')
|
||||
quality_info = QOBUZ_QUALITY_MAP.get(quality_key, QOBUZ_QUALITY_MAP['lossless'])
|
||||
|
||||
track_results = []
|
||||
|
|
@ -1038,20 +1008,14 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
traceback.print_exc()
|
||||
return ([], [])
|
||||
|
||||
def _qobuz_to_track_result(self, track: Dict, quality_info: dict,
|
||||
require_streamable: bool = True) -> Optional[TrackResult]:
|
||||
"""Convert Qobuz track dict to TrackResult (Soulseek-compatible format).
|
||||
|
||||
``require_streamable`` gates bulk SEARCH results on the ``streamable`` flag
|
||||
(filters noise). For a track the user explicitly pasted a link to, pass
|
||||
False: ``track/get`` may omit the flag (which would default-False and wrongly
|
||||
drop the exact track they asked for), and they should get to try it (#932)."""
|
||||
def _qobuz_to_track_result(self, track: Dict, quality_info: dict) -> Optional[TrackResult]:
|
||||
"""Convert Qobuz track dict to TrackResult (Soulseek-compatible format)."""
|
||||
track_id = track.get('id')
|
||||
if not track_id:
|
||||
return None
|
||||
|
||||
# Check if track is streamable (skipped for explicit by-id link fetches).
|
||||
if require_streamable and not track.get('streamable', False):
|
||||
# Check if track is streamable
|
||||
if not track.get('streamable', False):
|
||||
return None
|
||||
|
||||
performer = track.get('performer', {})
|
||||
|
|
@ -1106,20 +1070,8 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
title=title,
|
||||
album=album_name,
|
||||
track_number=track.get('track_number'),
|
||||
# Stamp the Qobuz track id so a pasted-link manual search can float the
|
||||
# exact track to the top (#932). Without this the bubble never matched
|
||||
# and the linked track stayed buried among fuzzy lookalikes.
|
||||
_source_metadata={'source': 'qobuz', 'track_id': str(track.get('id') or '')},
|
||||
)
|
||||
|
||||
# Stamp real API quality so the global ranker sees actual
|
||||
# sample_rate/bit_depth. Hi-res only when the track itself is
|
||||
# hires-streamable; otherwise it streams as CD-quality FLAC.
|
||||
if hires_streamable and max_bit_depth >= 24:
|
||||
track_result.set_quality(quality_from_qobuz(max_sample_rate, max_bit_depth))
|
||||
else:
|
||||
track_result.set_quality(quality_from_qobuz(44.1, 16))
|
||||
|
||||
return track_result
|
||||
|
||||
# ===================== Download =====================
|
||||
|
|
@ -1229,7 +1181,7 @@ class QobuzClient(DownloadSourcePlugin):
|
|||
|
||||
try:
|
||||
# Determine quality
|
||||
quality_key = quality_tier_for_source('qobuz', default='lossless')
|
||||
quality_key = config_manager.get('qobuz.quality', 'lossless')
|
||||
quality_info = QOBUZ_QUALITY_MAP.get(quality_key, QOBUZ_QUALITY_MAP['lossless'])
|
||||
|
||||
# Quality fallback chain: hires_max → hires → lossless → mp3
|
||||
|
|
|
|||
|
|
@ -1,103 +0,0 @@
|
|||
"""Single source of truth for "is this audio lossless?" + the lossy-copy
|
||||
overwrite invariant.
|
||||
|
||||
The "create a lossy copy of lossless tracks" feature lives in two places (the
|
||||
import post-processing path and the Lossy Converter repair job). Both used to
|
||||
hardcode ``.flac``, which is exactly how ALAC/WAV/DSD ended up being quality-
|
||||
profile options but NOT lossy-copy sources (#941). The knowledge of which
|
||||
formats are lossless now lives HERE, derived from the same format names the
|
||||
quality model ranks, so adding a format lights it up in both sites at once and
|
||||
they can never drift.
|
||||
|
||||
Two seams, both pure (no I/O) so they're unit-testable without real files:
|
||||
|
||||
* :func:`is_lossless_format` / :func:`is_lossless_audio_path` — eligibility.
|
||||
``.m4a``/``.mp4`` are ambiguous (ALAC=lossless, AAC=lossy) and can only be told
|
||||
apart by codec, so the path check delegates them to an *injected* ``probe_codec``
|
||||
callable — the file I/O stays at the edge, the decision stays pure.
|
||||
* :func:`lossy_output_would_overwrite_source` — the safety invariant: a lossy copy
|
||||
must NEVER be written over its own source (which becomes possible once ``.m4a``
|
||||
is eligible and the target codec is AAC → same ``.m4a`` path).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from core.quality.source_map import AUDIO_EXTENSIONS, format_from_extension
|
||||
|
||||
# Canonical lossless format set. MUST stay in sync with the lossless tiers in
|
||||
# core.quality.model.tier_score and the frontend RT_LOSSLESS_FORMATS — the
|
||||
# consistency test in tests/quality/test_lossless.py pins the tier agreement.
|
||||
LOSSLESS_FORMATS = frozenset({'flac', 'alac', 'wav', 'dsf'})
|
||||
|
||||
# Container extensions that may hold EITHER lossless (ALAC) or lossy (AAC) audio —
|
||||
# only a codec probe can decide, so they're never lossless on extension alone.
|
||||
_AMBIGUOUS_EXTS = frozenset({'m4a', 'mp4'})
|
||||
|
||||
|
||||
def is_lossless_format(fmt: Any) -> bool:
|
||||
"""True when a unified format name (as returned by ``format_from_extension``)
|
||||
is lossless. ``'aac'`` is False here — an ALAC-in-m4a file reports format
|
||||
``'aac'`` by extension and must be resolved by codec, not by name."""
|
||||
return str(fmt or '').lower() in LOSSLESS_FORMATS
|
||||
|
||||
|
||||
# Extensions worth checking as possibly-lossless (a caller can pre-filter SQL by
|
||||
# these, then confirm each via is_lossless_audio_path). Derived from the format
|
||||
# map so it can't drift: every extension whose format is lossless, plus the
|
||||
# ambiguous ALAC containers. Leading dot included (e.g. '.flac', '.m4a').
|
||||
LOSSLESS_CANDIDATE_EXTENSIONS = frozenset(
|
||||
e for e in AUDIO_EXTENSIONS
|
||||
if is_lossless_format(format_from_extension(e)) or e.lstrip('.') in _AMBIGUOUS_EXTS
|
||||
)
|
||||
|
||||
|
||||
def _ext(path: Any) -> str:
|
||||
return os.path.splitext(str(path or ''))[1].lower().lstrip('.')
|
||||
|
||||
|
||||
def is_lossless_audio_path(
|
||||
path: Any,
|
||||
*,
|
||||
probe_codec: Optional[Callable[[str], Optional[str]]] = None,
|
||||
) -> bool:
|
||||
"""True when the file at ``path`` is lossless.
|
||||
|
||||
Unambiguous extensions (flac/wav/aiff/dsf/dff/alac) are decided by extension.
|
||||
``.m4a``/``.mp4`` are decided by ``probe_codec(path)`` (returns the codec, e.g.
|
||||
``'alac'`` or ``'aac'``) — without a probe they're treated as NOT lossless, so
|
||||
a missing probe can never misclassify an AAC file as lossless.
|
||||
"""
|
||||
ext = _ext(path)
|
||||
if is_lossless_format(format_from_extension(ext)):
|
||||
return True
|
||||
if ext in _AMBIGUOUS_EXTS and probe_codec is not None:
|
||||
try:
|
||||
codec = (probe_codec(str(path)) or '').lower()
|
||||
except Exception:
|
||||
return False
|
||||
return 'alac' in codec
|
||||
return False
|
||||
|
||||
|
||||
def lossy_output_would_overwrite_source(source_path: Any, output_path: Any) -> bool:
|
||||
"""True when the computed lossy-copy output path is the source file itself.
|
||||
|
||||
Safety invariant: the converter must skip (never run ffmpeg with ``-y``) when
|
||||
this is True, or it would destroy the original. Happens when an ``.m4a`` ALAC
|
||||
source is converted with the AAC codec (output is also ``.m4a``)."""
|
||||
if not source_path or not output_path:
|
||||
return False
|
||||
a = os.path.normcase(os.path.normpath(str(source_path)))
|
||||
b = os.path.normcase(os.path.normpath(str(output_path)))
|
||||
return a == b
|
||||
|
||||
|
||||
__all__ = [
|
||||
'LOSSLESS_FORMATS',
|
||||
'is_lossless_format',
|
||||
'is_lossless_audio_path',
|
||||
'lossy_output_would_overwrite_source',
|
||||
]
|
||||
|
|
@ -1,285 +0,0 @@
|
|||
"""Source-agnostic audio quality model.
|
||||
|
||||
Every download source maps its result into ``AudioQuality``.
|
||||
The ``QualityTarget`` list in the user's profile defines the
|
||||
priority order (1st choice, 2nd choice, …). ``rank_candidate``
|
||||
scores any ``AudioQuality`` against that list so the same
|
||||
logic drives Soulseek, Tidal, Deezer, torrent — no per-source
|
||||
quality pipelines needed.
|
||||
|
||||
Soulseek attribute type codes (Soulseek protocol spec):
|
||||
0 = bitrate (kbps)
|
||||
1 = duration (seconds)
|
||||
2 = VBR flag
|
||||
4 = sample rate (Hz) — FLAC / WAV only
|
||||
5 = bit depth — FLAC / WAV only
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioQuality:
|
||||
"""Unified audio quality descriptor — source-agnostic."""
|
||||
|
||||
format: str # 'flac', 'mp3', 'aac', 'ogg', 'wav', 'unknown'
|
||||
bitrate: Optional[int] = None # kbps
|
||||
sample_rate: Optional[int] = None # Hz (e.g. 44100, 96000, 192000)
|
||||
bit_depth: Optional[int] = None # bits per sample (16, 24, 32)
|
||||
|
||||
def tier_score(self) -> float:
|
||||
"""Continuous score for ranking within a matched target bucket.
|
||||
Higher = better. Used as a tiebreaker after target-list matching.
|
||||
"""
|
||||
# NOTE: this only orders the *fallback* path (nothing matched a ranked
|
||||
# target) and tie-breaks candidates of the SAME format within one
|
||||
# matched target. Cross-format PRIORITY is decided solely by the user's
|
||||
# ranked-target list (target index), never by these numbers.
|
||||
format_base: dict[str, float] = {
|
||||
'dsf': 102.0, # DSD — 1-bit hi-res lossless, ranks at/above FLAC (#939)
|
||||
'flac': 100.0,
|
||||
'alac': 98.0, # lossless (Apple)
|
||||
'wav': 95.0,
|
||||
'ogg': 70.0,
|
||||
'opus': 65.0,
|
||||
'aac': 60.0,
|
||||
'mp3': 50.0,
|
||||
'wma': 30.0,
|
||||
}
|
||||
base = format_base.get(self.format.lower(), 10.0)
|
||||
|
||||
if self.format.lower() in ('flac', 'wav'):
|
||||
sr = self.sample_rate or 44100
|
||||
bd = self.bit_depth or 16
|
||||
# sample-rate contribution: 44.1 kHz = 0, 192 kHz = +20
|
||||
sr_score = min(sr / 192_000, 1.0) * 20
|
||||
# bit-depth contribution: 16-bit = 0, 24-bit = +10
|
||||
bd_score = max(bd - 16, 0) / 8 * 10
|
||||
return base + sr_score + bd_score
|
||||
else:
|
||||
br = self.bitrate or 0
|
||||
return base + min(br / 320, 1.0) * 10
|
||||
|
||||
def matches_target(self, target: QualityTarget) -> bool:
|
||||
"""True when this quality satisfies every constraint in *target*."""
|
||||
if target.format and target.format.lower() != self.format.lower():
|
||||
return False
|
||||
if target.min_bitrate and (self.bitrate or 0) < target.min_bitrate:
|
||||
return False
|
||||
if target.min_sample_rate:
|
||||
if self.sample_rate is not None:
|
||||
if self.sample_rate < target.min_sample_rate:
|
||||
return False
|
||||
else:
|
||||
# No sample-rate metadata (common on slskd FLAC). Use the kbps
|
||||
# heuristic when a bitrate is present; otherwise we CANNOT
|
||||
# confirm the spec, so fail the strict target rather than
|
||||
# over-claim it — an unknown-spec FLAC must not outrank a known
|
||||
# 16/44 FLAC under a hi-res target (#896 review #4). It falls to
|
||||
# the plain-flac bucket instead.
|
||||
# 16-bit/44.1 kHz ≈ 1411 kbps; 24-bit/96 kHz ≈ 4608 kbps.
|
||||
if self.format.lower() == 'flac' and self.bitrate:
|
||||
required_kbps = _sample_rate_to_min_kbps(target.min_sample_rate, target.bit_depth or 24)
|
||||
if self.bitrate < required_kbps:
|
||||
return False
|
||||
else:
|
||||
return False
|
||||
if target.bit_depth:
|
||||
if self.bit_depth is not None:
|
||||
if self.bit_depth < target.bit_depth:
|
||||
return False
|
||||
else:
|
||||
# No bit-depth metadata. A hi-res (>=24-bit) target needs proof:
|
||||
# use the kbps heuristic if a bitrate is present, else fail
|
||||
# rather than over-claim. The 16-bit baseline still matches an
|
||||
# unknown-spec FLAC (any FLAC is at least CD quality). #896 review #4.
|
||||
if self.format.lower() == 'flac' and target.bit_depth >= 24:
|
||||
if self.bitrate:
|
||||
if self.bitrate < 1450:
|
||||
return False
|
||||
else:
|
||||
return False
|
||||
return True
|
||||
|
||||
def label(self) -> str:
|
||||
"""Human-readable label, e.g. 'FLAC 24-bit/192kHz' or 'MP3 320kbps'."""
|
||||
fmt = self.format.upper()
|
||||
if self.format.lower() in ('flac', 'wav'):
|
||||
bd = f"{self.bit_depth}-bit/" if self.bit_depth else ""
|
||||
sr = f"{self.sample_rate // 1000}kHz" if self.sample_rate else ""
|
||||
detail = f" {bd}{sr}".rstrip()
|
||||
return f"{fmt}{detail}" if detail.strip() else fmt
|
||||
else:
|
||||
br = f" {self.bitrate}kbps" if self.bitrate else ""
|
||||
return f"{fmt}{br}"
|
||||
|
||||
@classmethod
|
||||
def from_slskd_file(cls, file_data: dict, extension: str) -> 'AudioQuality':
|
||||
"""Build from a raw slskd API file entry.
|
||||
|
||||
slskd exposes Soulseek protocol file attributes as:
|
||||
``{"attributes": [{"type": 4, "value": 96000}, {"type": 5, "value": 24}, ...]}``
|
||||
"""
|
||||
attrs = {a['type']: a['value'] for a in file_data.get('attributes', [])}
|
||||
return cls(
|
||||
format=extension.lower().lstrip('.'),
|
||||
bitrate=file_data.get('bitRate') or attrs.get(0),
|
||||
sample_rate=attrs.get(4),
|
||||
bit_depth=attrs.get(5),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_tier(cls, fmt: str, bitrate: int, sample_rate: Optional[int] = None, bit_depth: Optional[int] = None) -> 'AudioQuality':
|
||||
"""Build from a hardcoded quality tier (Tidal, Deezer, Qobuz)."""
|
||||
return cls(format=fmt, bitrate=bitrate, sample_rate=sample_rate, bit_depth=bit_depth)
|
||||
|
||||
@classmethod
|
||||
def from_extension_and_bitrate(cls, extension: str, bitrate: Optional[int]) -> 'AudioQuality':
|
||||
"""Minimal constructor when only format + bitrate are known (torrent, YouTube)."""
|
||||
return cls(format=extension.lower().lstrip('.'), bitrate=bitrate)
|
||||
|
||||
|
||||
@dataclass
|
||||
class QualityTarget:
|
||||
"""One ranked entry in the user's quality priority list."""
|
||||
|
||||
label: str = ""
|
||||
format: Optional[str] = None # 'flac', 'mp3', 'aac', …
|
||||
bit_depth: Optional[int] = None # 16, 24
|
||||
min_sample_rate: Optional[int] = None # Hz
|
||||
min_bitrate: Optional[int] = None # kbps (lossy)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {k: v for k, v in self.__dict__.items() if v not in (None, "")}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> 'QualityTarget':
|
||||
known = {f.name for f in cls.__dataclass_fields__.values()} # type: ignore[attr-defined]
|
||||
return cls(**{k: v for k, v in d.items() if k in known})
|
||||
|
||||
|
||||
# ── Default priority list ──────────────────────────────────────────────────────
|
||||
|
||||
DEFAULT_RANKED_TARGETS: List[QualityTarget] = [
|
||||
QualityTarget(label='FLAC 24-bit/192kHz', format='flac', bit_depth=24, min_sample_rate=192_000),
|
||||
QualityTarget(label='FLAC 24-bit/96kHz', format='flac', bit_depth=24, min_sample_rate=96_000),
|
||||
QualityTarget(label='FLAC 24-bit/48kHz', format='flac', bit_depth=24, min_sample_rate=48_000),
|
||||
QualityTarget(label='FLAC 24-bit/44.1kHz',format='flac', bit_depth=24, min_sample_rate=44_100),
|
||||
QualityTarget(label='FLAC 16-bit', format='flac', bit_depth=16),
|
||||
QualityTarget(label='MP3 320kbps', format='mp3', min_bitrate=320),
|
||||
QualityTarget(label='MP3 256kbps', format='mp3', min_bitrate=256),
|
||||
QualityTarget(label='MP3 192kbps', format='mp3', min_bitrate=192),
|
||||
]
|
||||
|
||||
|
||||
# ── Ranking helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
def rank_candidate(aq: AudioQuality, targets: List[QualityTarget]) -> Tuple[int, float]:
|
||||
"""Return *(target_index, tier_score)* for sorting.
|
||||
|
||||
Lower ``target_index`` → higher priority match.
|
||||
Candidates that satisfy no target get ``index = len(targets)``
|
||||
(they sort last but are not discarded — the caller decides that).
|
||||
"""
|
||||
for i, target in enumerate(targets):
|
||||
if aq.matches_target(target):
|
||||
return (i, aq.tier_score())
|
||||
return (len(targets), aq.tier_score())
|
||||
|
||||
|
||||
def filter_and_rank(
|
||||
candidates: list,
|
||||
targets: List[QualityTarget],
|
||||
*,
|
||||
fallback_enabled: bool = True,
|
||||
) -> list:
|
||||
"""Sort *candidates* (any objects with an ``audio_quality`` attribute)
|
||||
by quality priority.
|
||||
|
||||
Returns the subset that matched the *highest-priority* satisfied target,
|
||||
sorted by ``tier_score`` descending within that group.
|
||||
Falls back to all candidates sorted by score when ``fallback_enabled``
|
||||
and nothing matches, or when targets list is empty.
|
||||
"""
|
||||
if not targets:
|
||||
candidates_copy = list(candidates)
|
||||
candidates_copy.sort(key=lambda c: c.audio_quality.tier_score(), reverse=True)
|
||||
return candidates_copy
|
||||
|
||||
scored = [(rank_candidate(c.audio_quality, targets), c) for c in candidates]
|
||||
|
||||
# Best target index that any candidate reached
|
||||
best_idx = min((s[0][0] for s in scored), default=len(targets))
|
||||
|
||||
if best_idx < len(targets):
|
||||
winners = [c for (idx, _), c in scored if idx == best_idx]
|
||||
winners.sort(key=lambda c: c.audio_quality.tier_score(), reverse=True)
|
||||
return winners
|
||||
|
||||
if fallback_enabled:
|
||||
all_sorted = list(candidates)
|
||||
all_sorted.sort(key=lambda c: c.audio_quality.tier_score(), reverse=True)
|
||||
return all_sorted
|
||||
|
||||
return []
|
||||
|
||||
|
||||
# ── Migration helper ───────────────────────────────────────────────────────────
|
||||
|
||||
def v2_qualities_to_ranked_targets(qualities: dict) -> List[dict]:
|
||||
"""Convert old v2 ``qualities`` dict to a ranked-targets list.
|
||||
|
||||
Preserves the user's existing priority order while upgrading to the
|
||||
richer target format.
|
||||
"""
|
||||
_FORMAT_MAP = {
|
||||
'flac': {'format': 'flac', 'bit_depth': None},
|
||||
'mp3_320': {'format': 'mp3', 'min_bitrate': 320},
|
||||
'mp3_256': {'format': 'mp3', 'min_bitrate': 256},
|
||||
'mp3_192': {'format': 'mp3', 'min_bitrate': 192},
|
||||
# AAC (#886): opt-in tier. Match on format alone — Soulseek AAC/.m4a
|
||||
# rarely carries a bitrate attribute, so a min_bitrate gate would
|
||||
# reject every bitrate-less AAC. Priority order (above MP3, below FLAC)
|
||||
# is preserved by the caller's priority sort, not by min_bitrate.
|
||||
'aac': {'format': 'aac'},
|
||||
}
|
||||
enabled = [
|
||||
(cfg.get('priority', 999), name, cfg)
|
||||
for name, cfg in qualities.items()
|
||||
if cfg.get('enabled', False)
|
||||
]
|
||||
enabled.sort()
|
||||
targets = []
|
||||
for _, name, cfg in enabled:
|
||||
base = _FORMAT_MAP.get(name, {}).copy()
|
||||
if not base:
|
||||
continue
|
||||
if name == 'flac':
|
||||
bd = cfg.get('bit_depth', 'any')
|
||||
if bd == '24':
|
||||
base['bit_depth'] = 24
|
||||
base['label'] = 'FLAC 24-bit'
|
||||
elif bd == '16':
|
||||
base['bit_depth'] = 16
|
||||
base['label'] = 'FLAC 16-bit'
|
||||
else:
|
||||
base['label'] = 'FLAC (any)'
|
||||
else:
|
||||
base['label'] = name.upper().replace('_', ' ')
|
||||
targets.append(base)
|
||||
return targets
|
||||
|
||||
|
||||
# ── Internal helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
def _sample_rate_to_min_kbps(sample_rate: int, bit_depth: int) -> int:
|
||||
"""Approximate minimum kbps for a lossless file at the given spec.
|
||||
Used as heuristic when actual sample-rate metadata is absent.
|
||||
"""
|
||||
# kbps = sample_rate * channels * bit_depth / 1000 * compression_ratio
|
||||
# Assume stereo (2 ch) and ~0.6 FLAC compression ratio
|
||||
raw_kbps = sample_rate * 2 * bit_depth / 1000
|
||||
return int(raw_kbps * 0.55) # conservative compressed estimate
|
||||
|
|
@ -1,148 +0,0 @@
|
|||
"""Quality-aware candidate selection shared by the search engine and the
|
||||
download orchestrator.
|
||||
|
||||
``rank_with_targets`` is the pure core: it ranks candidates against a target
|
||||
list and reports whether any candidate met a *real* target (strict, fallback
|
||||
off). The engine uses that ``satisfied`` flag to decide whether the current
|
||||
source is good enough or it should fall through to the next source in the
|
||||
hybrid chain.
|
||||
|
||||
``rank_for_profile`` is the thin DB-backed wrapper that loads the user's
|
||||
quality profile (with v2->v3 migration) and delegates.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import List, Tuple
|
||||
|
||||
from core.quality.model import (
|
||||
QualityTarget,
|
||||
filter_and_rank,
|
||||
v2_qualities_to_ranked_targets,
|
||||
)
|
||||
|
||||
|
||||
def rank_with_targets(
|
||||
candidates: list,
|
||||
targets: List[QualityTarget],
|
||||
*,
|
||||
fallback_enabled: bool = True,
|
||||
) -> Tuple[list, bool]:
|
||||
"""Rank *candidates* against *targets*.
|
||||
|
||||
Returns ``(ranked, satisfied)`` where ``satisfied`` is True when at least
|
||||
one candidate meets a real target. When no targets are configured the
|
||||
profile imposes no constraint, so any non-empty result counts as
|
||||
satisfied (the first source wins, quality-sorted).
|
||||
"""
|
||||
if not candidates:
|
||||
return [], False
|
||||
|
||||
if not targets:
|
||||
ranked = filter_and_rank(candidates, targets, fallback_enabled=True)
|
||||
return ranked, bool(ranked)
|
||||
|
||||
strict = filter_and_rank(candidates, targets, fallback_enabled=False)
|
||||
if strict:
|
||||
return strict, True
|
||||
|
||||
if fallback_enabled:
|
||||
return filter_and_rank(candidates, targets, fallback_enabled=True), False
|
||||
return [], False
|
||||
|
||||
|
||||
def targets_from_profile(profile: dict) -> Tuple[List[QualityTarget], bool]:
|
||||
"""Convert a quality-profile dict into ``(targets, fallback_enabled)`` with
|
||||
v2->v3 migration applied. The single conversion path shared by the import
|
||||
guard, the download ranker and the library quality scanner."""
|
||||
raw_targets = profile.get('ranked_targets')
|
||||
if not raw_targets and 'qualities' in profile:
|
||||
raw_targets = v2_qualities_to_ranked_targets(profile['qualities'])
|
||||
|
||||
targets = [QualityTarget.from_dict(t) for t in (raw_targets or [])]
|
||||
fallback_enabled = profile.get('fallback_enabled', True)
|
||||
return targets, fallback_enabled
|
||||
|
||||
|
||||
def load_profile_targets() -> Tuple[List[QualityTarget], bool]:
|
||||
"""Load the user's quality profile from the DB and return
|
||||
``(targets, fallback_enabled)`` with v2->v3 migration applied.
|
||||
|
||||
Callers that rank across many sources should load once and reuse via
|
||||
:func:`rank_with_targets` rather than calling :func:`rank_for_profile`
|
||||
per source.
|
||||
"""
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
return targets_from_profile(MusicDatabase().get_quality_profile())
|
||||
|
||||
|
||||
def quality_meets_profile(aq, targets: List[QualityTarget]) -> bool:
|
||||
"""Strict: True iff *aq* satisfies at least one ranked *target*.
|
||||
|
||||
The shared definition of "good enough" for both the import guard and the
|
||||
library scanner — bit depth + sample rate are minimums (see
|
||||
:meth:`AudioQuality.matches_target`). Fallback is NOT consulted here; it's a
|
||||
download-time last-resort concession, not part of what counts as meeting the
|
||||
profile. ``targets`` empty → no constraint (True). ``aq`` None (probe
|
||||
failed) → True, so an unreadable file is never falsely flagged.
|
||||
"""
|
||||
if not targets:
|
||||
return True
|
||||
if aq is None:
|
||||
return True
|
||||
from core.quality.model import rank_candidate
|
||||
|
||||
idx, _ = rank_candidate(aq, targets)
|
||||
return idx < len(targets)
|
||||
|
||||
|
||||
_VALID_SEARCH_MODES = ("priority", "best_quality")
|
||||
|
||||
|
||||
def load_search_mode() -> str:
|
||||
"""Return the download search strategy from the user's quality profile.
|
||||
|
||||
``'priority'`` (default) keeps today's behaviour — the first source in the
|
||||
hybrid chain that meets a quality target wins. ``'best_quality'`` pools
|
||||
candidates across all sources and works them best→worst by actual audio
|
||||
quality. Any missing/unknown value resolves to ``'priority'`` so existing
|
||||
installs are unaffected.
|
||||
"""
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
try:
|
||||
profile = MusicDatabase().get_quality_profile()
|
||||
mode = profile.get("search_mode", "priority")
|
||||
except Exception:
|
||||
return "priority"
|
||||
return mode if mode in _VALID_SEARCH_MODES else "priority"
|
||||
|
||||
|
||||
def load_rank_candidates_by_quality() -> bool:
|
||||
"""Opt-in: order the priority-mode download walk (and thus the
|
||||
version-mismatch force-import pick, which takes the first-tried = best)
|
||||
by ranked-target quality instead of confidence-first.
|
||||
|
||||
Best-quality search mode is always quality-first regardless of this flag;
|
||||
this toggle only affects *priority* mode. Default ``False`` keeps the
|
||||
byte-for-byte old behaviour (confidence/peer-speed first), so existing
|
||||
installs are unaffected unless they opt in. Any missing value or DB error
|
||||
resolves to ``False``.
|
||||
"""
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
try:
|
||||
profile = MusicDatabase().get_quality_profile()
|
||||
return bool(profile.get("rank_candidates_by_quality", False))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def rank_for_profile(candidates: list) -> Tuple[list, bool]:
|
||||
"""Load the user's quality profile and rank *candidates* against it.
|
||||
|
||||
Returns ``(ranked, satisfied)`` — see :func:`rank_with_targets`.
|
||||
"""
|
||||
targets, fallback_enabled = load_profile_targets()
|
||||
return rank_with_targets(candidates, targets, fallback_enabled=fallback_enabled)
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
"""Per-source quality mappers — turn each download source's tier string or
|
||||
API values into a unified :class:`~core.quality.model.AudioQuality`.
|
||||
|
||||
Every streaming source describes quality differently (Tidal/HiFi use tier
|
||||
strings, Qobuz reports real kHz + bit depth, Deezer uses config codes,
|
||||
Amazon mixes real values with HD/UHD tiers). Centralising the knowledge
|
||||
here keeps the per-client code to a single call and keeps the tier tables
|
||||
in one auditable place.
|
||||
|
||||
Each value is a *claim*: the download client populates its ``TrackResult``
|
||||
from it so the global ranker can choose a source, and the post-download
|
||||
quality guard later verifies the real file. Over-claiming is the danger —
|
||||
an unknown tier maps to ``format='unknown'`` rather than pretending to be
|
||||
lossless.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from core.quality.model import AudioQuality
|
||||
from core.quality.selection import load_profile_targets
|
||||
|
||||
|
||||
# ── Extension → format string (source-agnostic) ────────────────────────────
|
||||
#
|
||||
# The single source of truth for mapping a file extension to the unified
|
||||
# AudioQuality ``format``. Every extension-based download source (Soulseek,
|
||||
# torrent/usenet file lists, …) classifies through this, so the ranked-target
|
||||
# system behaves identically across sources and adding a format here lights it
|
||||
# up everywhere at once. Unknown extensions → 'unknown' (never matches a
|
||||
# target, so it only ever comes through via the fallback toggle).
|
||||
#
|
||||
# AIFF/AIF are uncompressed PCM like WAV → the same 'wav' tier. ``.m4a``
|
||||
# defaults to 'aac'; an ALAC-in-m4a file can't be told apart by extension
|
||||
# alone, so probe_audio_quality corrects it from the real codec post-download.
|
||||
_EXTENSION_FORMAT_MAP = {
|
||||
'flac': 'flac',
|
||||
'alac': 'alac',
|
||||
'wav': 'wav', 'wave': 'wav',
|
||||
'aiff': 'wav', 'aif': 'wav', 'aifc': 'wav',
|
||||
'mp3': 'mp3',
|
||||
'm4a': 'aac', 'mp4': 'aac', 'aac': 'aac',
|
||||
'ogg': 'ogg', 'oga': 'ogg',
|
||||
'opus': 'opus',
|
||||
'wma': 'wma',
|
||||
# DSD (DSD Stream File / DSDIFF) — 1-bit hi-res lossless (e.g. DSD64 ≈ 11 Mbps).
|
||||
# Both container types map to the single 'dsf' tier (#939).
|
||||
'dsf': 'dsf', 'dff': 'dsf',
|
||||
}
|
||||
|
||||
# Audio extensions worth probing/classifying at all — derived from the map so
|
||||
# the allow-list and the classifier never drift apart.
|
||||
AUDIO_EXTENSIONS = {f'.{e}' for e in _EXTENSION_FORMAT_MAP}
|
||||
|
||||
|
||||
def format_from_extension(ext: str) -> str:
|
||||
"""Map a file extension (with or without leading dot) to the unified
|
||||
AudioQuality format string. Unknown → 'unknown'."""
|
||||
return _EXTENSION_FORMAT_MAP.get(str(ext or '').lower().lstrip('.'), 'unknown')
|
||||
|
||||
|
||||
# ── Tidal / HiFi (Monochrome is Tidal-backed) ──────────────────────────────
|
||||
#
|
||||
# Tidal exposes UPPER_SNAKE tier strings (``HI_RES_LOSSLESS``); HiFi's config
|
||||
# uses lowercase keys (``hires``/``lossless``). We normalise both into the
|
||||
# same lookup so one mapper serves both sources.
|
||||
|
||||
_TIDAL_HIRES = AudioQuality(format='flac', sample_rate=96000, bit_depth=24)
|
||||
_TIDAL_LOSSLESS = AudioQuality(format='flac', sample_rate=44100, bit_depth=16)
|
||||
_TIDAL_HIGH = AudioQuality(format='aac', bitrate=320)
|
||||
_TIDAL_LOW = AudioQuality(format='aac', bitrate=96)
|
||||
|
||||
TIDAL_TIER_MAP = {
|
||||
'HI_RES_LOSSLESS': _TIDAL_HIRES,
|
||||
'HI_RES': _TIDAL_HIRES,
|
||||
'HIRES': _TIDAL_HIRES,
|
||||
'LOSSLESS': _TIDAL_LOSSLESS,
|
||||
'HIGH': _TIDAL_HIGH,
|
||||
'LOW': _TIDAL_LOW,
|
||||
}
|
||||
|
||||
|
||||
def quality_from_tidal_tier(tier: str) -> AudioQuality:
|
||||
"""Map a Tidal/HiFi quality tier string to an AudioQuality.
|
||||
|
||||
Case-insensitive; accepts both ``HI_RES`` and ``hires`` spellings.
|
||||
Unrecognised tiers map to ``format='unknown'`` so they never
|
||||
over-claim lossless quality.
|
||||
"""
|
||||
key = (tier or '').strip().upper()
|
||||
return TIDAL_TIER_MAP.get(key, AudioQuality(format='unknown'))
|
||||
|
||||
|
||||
# ── Qobuz (real API values) ────────────────────────────────────────────────
|
||||
|
||||
def quality_from_qobuz(sampling_rate_khz: float, bit_depth: int) -> AudioQuality:
|
||||
"""Qobuz reports ``maximum_sampling_rate`` in kHz (e.g. 44.1, 96, 192)
|
||||
and ``maximum_bit_depth``. These are real values from the API.
|
||||
"""
|
||||
sample_rate = int(round(sampling_rate_khz * 1000)) if sampling_rate_khz else None
|
||||
return AudioQuality(format='flac', sample_rate=sample_rate, bit_depth=bit_depth)
|
||||
|
||||
|
||||
# ── Deezer (config code) ───────────────────────────────────────────────────
|
||||
|
||||
DEEZER_CODE_MAP = {
|
||||
'flac': AudioQuality(format='flac', sample_rate=44100, bit_depth=16),
|
||||
'mp3_320': AudioQuality(format='mp3', bitrate=320),
|
||||
'mp3_128': AudioQuality(format='mp3', bitrate=128),
|
||||
}
|
||||
|
||||
|
||||
def quality_from_deezer(code: str) -> AudioQuality:
|
||||
"""Map a Deezer download quality code to AudioQuality.
|
||||
|
||||
Deezer FLAC is always CD-quality (16-bit/44.1 kHz).
|
||||
"""
|
||||
return DEEZER_CODE_MAP.get((code or '').lower(), AudioQuality(format='unknown'))
|
||||
|
||||
|
||||
# ── Amazon Music (real sampleRate preferred, HD/UHD tier fallback) ─────────
|
||||
|
||||
_AMAZON_TIER_MAP = {
|
||||
'UHD': AudioQuality(format='flac', sample_rate=96000, bit_depth=24),
|
||||
'HD': AudioQuality(format='flac', sample_rate=44100, bit_depth=16),
|
||||
}
|
||||
|
||||
|
||||
def quality_from_amazon(
|
||||
tier: str,
|
||||
sample_rate: Optional[int] = None,
|
||||
bit_depth: Optional[int] = None,
|
||||
) -> AudioQuality:
|
||||
"""Amazon Music is FLAC; prefer the real ``sampleRate``/``bitDepth`` from
|
||||
the stream info when present, otherwise fall back to the HD/UHD tier.
|
||||
"""
|
||||
base = _AMAZON_TIER_MAP.get((tier or '').strip().upper(), AudioQuality(format='flac'))
|
||||
return AudioQuality(
|
||||
format='flac',
|
||||
sample_rate=sample_rate if sample_rate is not None else base.sample_rate,
|
||||
bit_depth=bit_depth if bit_depth is not None else base.bit_depth,
|
||||
)
|
||||
|
||||
|
||||
# ── Profile-driven download tier (replaces per-source quality settings) ─────
|
||||
#
|
||||
# Each source's selectable download tiers, ordered best → worst, with the
|
||||
# AudioQuality the tier delivers. ``quality_tier_for_source`` walks these to
|
||||
# request the LOWEST tier that satisfies the user's top global target — so the
|
||||
# global quality profile, not a per-source dropdown, decides what each source
|
||||
# fetches.
|
||||
|
||||
_SOURCE_TIER_LADDERS: dict[str, list[tuple[str, AudioQuality]]] = {
|
||||
'tidal': [
|
||||
('hires', AudioQuality('flac', sample_rate=96000, bit_depth=24)),
|
||||
('lossless', AudioQuality('flac', sample_rate=44100, bit_depth=16)),
|
||||
('high', AudioQuality('aac', bitrate=320)),
|
||||
('low', AudioQuality('aac', bitrate=96)),
|
||||
],
|
||||
'hifi': [
|
||||
('hires', AudioQuality('flac', sample_rate=96000, bit_depth=24)),
|
||||
('lossless', AudioQuality('flac', sample_rate=44100, bit_depth=16)),
|
||||
('high', AudioQuality('aac', bitrate=320)),
|
||||
('low', AudioQuality('aac', bitrate=96)),
|
||||
],
|
||||
'qobuz': [
|
||||
('hires_max', AudioQuality('flac', sample_rate=192000, bit_depth=24)),
|
||||
('hires', AudioQuality('flac', sample_rate=96000, bit_depth=24)),
|
||||
('lossless', AudioQuality('flac', sample_rate=44100, bit_depth=16)),
|
||||
('mp3', AudioQuality('mp3', bitrate=320)),
|
||||
],
|
||||
'deezer': [
|
||||
('flac', AudioQuality('flac', sample_rate=44100, bit_depth=16)),
|
||||
('mp3_320', AudioQuality('mp3', bitrate=320)),
|
||||
('mp3_128', AudioQuality('mp3', bitrate=128)),
|
||||
],
|
||||
'amazon': [
|
||||
('flac', AudioQuality('flac', sample_rate=48000, bit_depth=24)),
|
||||
('opus', AudioQuality('aac', bitrate=320)),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def quality_tier_for_source(source_name: str, *, default: Optional[str] = None) -> Optional[str]:
|
||||
"""Return the source tier key to request, derived from the global profile.
|
||||
|
||||
Picks the lowest tier in the source's ladder that satisfies the user's
|
||||
top (most-preferred) target — respecting the quality ceiling and saving
|
||||
bandwidth. Falls back to the source's max tier when none can satisfy it
|
||||
(best effort), or to the source's max when no targets are configured.
|
||||
Returns *default* for an unknown source.
|
||||
"""
|
||||
ladder = _SOURCE_TIER_LADDERS.get(source_name)
|
||||
if not ladder:
|
||||
return default
|
||||
|
||||
targets, _ = load_profile_targets()
|
||||
if not targets:
|
||||
return ladder[0][0]
|
||||
|
||||
top = targets[0]
|
||||
for key, aq in reversed(ladder): # low → high
|
||||
if aq.matches_target(top):
|
||||
return key
|
||||
return ladder[0][0] # best effort: max tier
|
||||
|
|
@ -70,9 +70,6 @@ class QueueItem:
|
|||
# 'tags' = read each file's embedded tags as the source
|
||||
# of truth (issue #592). Zero API calls.
|
||||
metadata_source: str = 'api'
|
||||
# Rename-only mode (#875): move files to the current naming scheme WITHOUT the
|
||||
# copy + post-processing (re-tag / quality / AcoustID) the full flow runs.
|
||||
rename_only: bool = False
|
||||
status: str = 'queued' # queued | running | done | failed | cancelled
|
||||
started_at: Optional[float] = None
|
||||
finished_at: Optional[float] = None
|
||||
|
|
@ -99,7 +96,6 @@ class QueueItem:
|
|||
'artist_name': self.artist_name,
|
||||
'source': self.source,
|
||||
'metadata_source': self.metadata_source,
|
||||
'rename_only': self.rename_only,
|
||||
'enqueued_at': self.enqueued_at,
|
||||
'started_at': self.started_at,
|
||||
'finished_at': self.finished_at,
|
||||
|
|
@ -165,7 +161,6 @@ class ReorganizeQueue:
|
|||
artist_name: str,
|
||||
source: Optional[str] = None,
|
||||
metadata_source: str = 'api',
|
||||
rename_only: bool = False,
|
||||
) -> dict:
|
||||
"""Add an album to the queue. Returns a result dict:
|
||||
|
||||
|
|
@ -195,7 +190,6 @@ class ReorganizeQueue:
|
|||
source=source,
|
||||
enqueued_at=time.time(),
|
||||
metadata_source=metadata_source or 'api',
|
||||
rename_only=bool(rename_only),
|
||||
)
|
||||
self._items.append(item)
|
||||
position = sum(1 for i in self._items if i.status == 'queued')
|
||||
|
|
|
|||
|
|
@ -37,7 +37,6 @@ def build_runner(
|
|||
is_shutting_down_fn: Callable[[], bool],
|
||||
get_download_path: Callable[[], str],
|
||||
get_transfer_path: Callable[[], str],
|
||||
build_final_path_fn: Optional[Callable] = None,
|
||||
) -> Callable[[object], dict]:
|
||||
"""Return the closure the queue worker invokes per item.
|
||||
|
||||
|
|
@ -60,7 +59,7 @@ def build_runner(
|
|||
A callable ``runner(item)`` suitable for
|
||||
:meth:`core.reorganize_queue.ReorganizeQueue.set_runner`.
|
||||
"""
|
||||
from core.library_reorganize import reorganize_album, reorganize_album_rename_only
|
||||
from core.library_reorganize import reorganize_album
|
||||
from core.reorganize_queue import get_queue
|
||||
|
||||
def _update_track_path(track_id, new_path):
|
||||
|
|
@ -81,6 +80,17 @@ def build_runner(
|
|||
# server restart.
|
||||
download_dir = get_download_path()
|
||||
transfer_dir = get_transfer_path()
|
||||
staging_root = os.path.join(download_dir, 'ssync_staging')
|
||||
try:
|
||||
os.makedirs(staging_root, exist_ok=True)
|
||||
except OSError as mk_err:
|
||||
logger.error(f"[Reorganize] Cannot create staging dir {staging_root}: {mk_err}")
|
||||
return {
|
||||
'status': 'setup_failed',
|
||||
'source': None,
|
||||
'total': 0, 'moved': 0, 'skipped': 0, 'failed': 0,
|
||||
'errors': [{'error': f'Could not create staging dir: {mk_err}'}],
|
||||
}
|
||||
|
||||
def _cleanup_empty(src_dir):
|
||||
try:
|
||||
|
|
@ -95,42 +105,6 @@ def build_runner(
|
|||
# Progress fan-out failures must never break a run.
|
||||
logger.debug("reorganize progress fan-out: %s", e)
|
||||
|
||||
# Rename-only mode (#875): just move files to the current scheme — no staging,
|
||||
# no copy, no post-processing. Falls through to the full pipeline otherwise.
|
||||
if getattr(item, 'rename_only', False):
|
||||
if build_final_path_fn is None:
|
||||
return {
|
||||
'status': 'setup_failed', 'source': None,
|
||||
'total': 0, 'moved': 0, 'skipped': 0, 'failed': 0,
|
||||
'errors': [{'error': 'Rename-only mode unavailable (no path builder)'}],
|
||||
}
|
||||
return reorganize_album_rename_only(
|
||||
album_id=item.album_id,
|
||||
db=get_database(),
|
||||
transfer_dir=transfer_dir,
|
||||
resolve_file_path_fn=resolve_file_path_fn,
|
||||
build_final_path_fn=build_final_path_fn,
|
||||
update_track_path_fn=_update_track_path,
|
||||
cleanup_empty_dir_fn=_cleanup_empty,
|
||||
on_progress=_on_progress,
|
||||
primary_source=item.source,
|
||||
strict_source=bool(item.source),
|
||||
metadata_source=getattr(item, 'metadata_source', 'api') or 'api',
|
||||
stop_check=is_shutting_down_fn,
|
||||
)
|
||||
|
||||
staging_root = os.path.join(download_dir, 'ssync_staging')
|
||||
try:
|
||||
os.makedirs(staging_root, exist_ok=True)
|
||||
except OSError as mk_err:
|
||||
logger.error(f"[Reorganize] Cannot create staging dir {staging_root}: {mk_err}")
|
||||
return {
|
||||
'status': 'setup_failed',
|
||||
'source': None,
|
||||
'total': 0, 'moved': 0, 'skipped': 0, 'failed': 0,
|
||||
'errors': [{'error': f'Could not create staging dir: {mk_err}'}],
|
||||
}
|
||||
|
||||
return reorganize_album(
|
||||
album_id=item.album_id,
|
||||
db=get_database(),
|
||||
|
|
|
|||
|
|
@ -40,7 +40,6 @@ _JOB_MODULES = [
|
|||
'core.repair_jobs.metadata_gap_filler',
|
||||
'core.repair_jobs.album_completeness',
|
||||
'core.repair_jobs.fake_lossless_detector',
|
||||
'core.repair_jobs.quality_upgrade_scanner',
|
||||
'core.repair_jobs.library_reorganize',
|
||||
'core.repair_jobs.mbid_mismatch_detector',
|
||||
'core.repair_jobs.single_album_dedup',
|
||||
|
|
@ -52,7 +51,6 @@ _JOB_MODULES = [
|
|||
'core.repair_jobs.canonical_version_resolve',
|
||||
'core.repair_jobs.library_retag',
|
||||
'core.repair_jobs.quality_upgrade',
|
||||
'core.repair_jobs.short_preview_track',
|
||||
]
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -389,43 +389,14 @@ class AcoustIDScannerJob(RepairJob):
|
|||
cur.execute(
|
||||
"UPDATE tracks SET verification_status = ? WHERE id = ?",
|
||||
(status, track_id))
|
||||
exp = expected or {}
|
||||
# Find the canonical history row for this file. The stored path is frozen at
|
||||
# import time while the file has since moved (media-server import / reorganize),
|
||||
# so an exact-path match alone misses it — then the status never lands and a
|
||||
# duplicate row gets inserted every scan (#934). Match exact path first, then
|
||||
# filename guarded by title, and HEAL the row's path so future scans match cleanly.
|
||||
from core.downloads.history_match import pick_history_row, like_filename_filter
|
||||
current = fpath or db_path
|
||||
basename = os.path.basename(current) if current else ''
|
||||
clauses, params = [], []
|
||||
matched = 0
|
||||
for p in {p for p in (fpath, db_path) if p}:
|
||||
clauses.append("file_path = ?")
|
||||
params.append(p)
|
||||
if basename:
|
||||
clauses.append("file_path LIKE ? ESCAPE '\\'")
|
||||
params.append(like_filename_filter(basename))
|
||||
row_id = None
|
||||
if clauses:
|
||||
cur.execute(
|
||||
"SELECT id, file_path, title, download_source FROM library_history WHERE "
|
||||
+ " OR ".join(clauses),
|
||||
params)
|
||||
row_id = pick_history_row(
|
||||
cur.fetchall(),
|
||||
current_paths=(fpath, db_path),
|
||||
basename=basename, title=exp.get('title') or '')
|
||||
if row_id is not None:
|
||||
cur.execute(
|
||||
"UPDATE library_history SET verification_status = ?, file_path = ? WHERE id = ?",
|
||||
(status, current, row_id))
|
||||
# Drop synthetic scan-created duplicates for this exact file (the #934
|
||||
# leftovers). Exact path → collision-free; never touches a real download row.
|
||||
cur.execute(
|
||||
"DELETE FROM library_history WHERE id != ? AND download_source = 'acoustid_scan' "
|
||||
"AND file_path = ?",
|
||||
(row_id, current))
|
||||
elif status == 'unverified':
|
||||
"UPDATE library_history SET verification_status = ? WHERE file_path = ?",
|
||||
(status, p))
|
||||
matched += max(getattr(cur, 'rowcount', 0) or 0, 0)
|
||||
if status == 'unverified' and matched == 0:
|
||||
exp = expected or {}
|
||||
cur.execute(
|
||||
"""INSERT INTO library_history
|
||||
(event_type, title, artist_name, album_name, file_path,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,19 +1,13 @@
|
|||
"""Lossy Converter Job — finds lossless files that don't have a lossy copy.
|
||||
"""Lossy Converter Job — finds FLAC files that don't have a lossy copy.
|
||||
|
||||
Scans the library for lossless files without a corresponding lossy copy alongside
|
||||
Scans the library for FLAC files without a corresponding lossy copy alongside
|
||||
them, and creates a finding for each. The fix action converts the file using
|
||||
ffmpeg with the user's configured codec/bitrate settings.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from core.imports.file_ops import m4a_codec
|
||||
from core.library.path_resolver import resolve_library_file_path
|
||||
from core.quality.lossless import (
|
||||
LOSSLESS_CANDIDATE_EXTENSIONS,
|
||||
is_lossless_audio_path,
|
||||
lossy_output_would_overwrite_source,
|
||||
)
|
||||
from core.repair_jobs import register_job
|
||||
from core.repair_jobs.base import JobContext, JobResult, RepairJob
|
||||
from utils.logging_config import get_logger
|
||||
|
|
@ -27,16 +21,6 @@ CODEC_MAP = {
|
|||
}
|
||||
|
||||
|
||||
def _lossless_ext_where(col: str) -> str:
|
||||
"""SQL pre-filter matching files whose extension *might* be lossless. The
|
||||
final decision (including ALAC-in-.m4a, which needs a codec probe) is made
|
||||
per-file by is_lossless_audio_path. Extensions are trusted constants from the
|
||||
quality model, never user input — safe to interpolate."""
|
||||
return '(' + ' OR '.join(
|
||||
f"LOWER({col}) LIKE '%{ext}'" for ext in sorted(LOSSLESS_CANDIDATE_EXTENSIONS)
|
||||
) + ')'
|
||||
|
||||
|
||||
def _resolve_file_path(file_path, transfer_folder, download_folder=None, config_manager=None):
|
||||
"""Backwards-compat wrapper. Use ``resolve_library_file_path`` directly."""
|
||||
return resolve_library_file_path(
|
||||
|
|
@ -51,15 +35,15 @@ def _resolve_file_path(file_path, transfer_folder, download_folder=None, config_
|
|||
class LossyConverterJob(RepairJob):
|
||||
job_id = 'lossy_converter'
|
||||
display_name = 'Lossy Converter'
|
||||
description = 'Finds lossless files without a lossy copy'
|
||||
description = 'Finds FLAC files without a lossy copy'
|
||||
help_text = (
|
||||
'Scans your library for lossless files (FLAC/ALAC/WAV/AIFF/DSD) that don\'t already have a lossy copy '
|
||||
'Scans your library for FLAC files that don\'t already have a lossy copy '
|
||||
'(MP3, Opus, or AAC) alongside them.\n\n'
|
||||
'Uses the codec setting from your Lossy Copy configuration on the Settings '
|
||||
'page. Enable Lossy Copy in Settings first, then run this job to find FLAC '
|
||||
'files missing a lossy copy.\n\n'
|
||||
'Each finding can be fixed individually or in bulk — the fix action converts '
|
||||
'the lossless file using ffmpeg at your configured bitrate.\n\n'
|
||||
'the FLAC file using ffmpeg at your configured bitrate.\n\n'
|
||||
'Requires ffmpeg to be installed.'
|
||||
)
|
||||
icon = 'repair-icon-lossy'
|
||||
|
|
@ -97,14 +81,14 @@ class LossyConverterJob(RepairJob):
|
|||
try:
|
||||
conn = context.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(f"""
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title, ar.name, al.title, t.file_path,
|
||||
al.thumb_url, ar.thumb_url
|
||||
FROM tracks t
|
||||
LEFT JOIN artists ar ON ar.id = t.artist_id
|
||||
LEFT JOIN albums al ON al.id = t.album_id
|
||||
WHERE t.file_path IS NOT NULL AND t.file_path != ''
|
||||
AND {_lossless_ext_where('t.file_path')}
|
||||
AND LOWER(t.file_path) LIKE '%.flac'
|
||||
""")
|
||||
tracks = cursor.fetchall()
|
||||
except Exception as e:
|
||||
|
|
@ -120,7 +104,7 @@ class LossyConverterJob(RepairJob):
|
|||
context.update_progress(0, total)
|
||||
if context.report_progress:
|
||||
context.report_progress(
|
||||
phase=f'Scanning {total} lossless files for missing {quality_label} copies...',
|
||||
phase=f'Scanning {total} FLAC files for missing {quality_label} copies...',
|
||||
total=total
|
||||
)
|
||||
|
||||
|
|
@ -151,17 +135,8 @@ class LossyConverterJob(RepairJob):
|
|||
if not resolved or not os.path.exists(resolved):
|
||||
continue
|
||||
|
||||
# Confirm it's actually lossless — the SQL pre-filter lets .m4a through,
|
||||
# which is ALAC (lossless) OR AAC (lossy); only a codec probe decides.
|
||||
if not is_lossless_audio_path(resolved, probe_codec=m4a_codec):
|
||||
continue
|
||||
|
||||
# Check if lossy copy already exists
|
||||
out_path = os.path.splitext(resolved)[0] + out_ext
|
||||
# Never offer to convert a file onto itself (e.g. .m4a ALAC + AAC target
|
||||
# lands on the same path) — that conversion would destroy the original.
|
||||
if lossy_output_would_overwrite_source(resolved, out_path):
|
||||
continue
|
||||
if os.path.exists(out_path):
|
||||
continue
|
||||
|
||||
|
|
@ -184,7 +159,7 @@ class LossyConverterJob(RepairJob):
|
|||
file_path=file_path,
|
||||
title=f'No {quality_label} copy: {title or "Unknown"}',
|
||||
description=(
|
||||
f'Lossless file "{title}" by {artist_name or "Unknown"} does not have '
|
||||
f'FLAC file "{title}" by {artist_name or "Unknown"} does not have '
|
||||
f'a {quality_label} copy alongside it'
|
||||
),
|
||||
details={
|
||||
|
|
@ -220,7 +195,7 @@ class LossyConverterJob(RepairJob):
|
|||
context.report_progress(
|
||||
scanned=total, total=total,
|
||||
phase='Complete',
|
||||
log_line=f'Found {result.findings_created} lossless files without {quality_label} copies',
|
||||
log_line=f'Found {result.findings_created} FLAC files without {quality_label} copies',
|
||||
log_type='success' if result.findings_created == 0 else 'info'
|
||||
)
|
||||
|
||||
|
|
@ -233,10 +208,10 @@ class LossyConverterJob(RepairJob):
|
|||
try:
|
||||
conn = context.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(f"""
|
||||
cursor.execute("""
|
||||
SELECT COUNT(*) FROM tracks
|
||||
WHERE file_path IS NOT NULL AND file_path != ''
|
||||
AND {_lossless_ext_where('file_path')}
|
||||
AND LOWER(file_path) LIKE '%.flac'
|
||||
""")
|
||||
row = cursor.fetchone()
|
||||
return row[0] if row else 0
|
||||
|
|
|
|||
|
|
@ -14,10 +14,9 @@ is queued until you review and Apply the finding — at which point the matched
|
|||
track (carrying its album context) is added to the wishlist, exactly like every
|
||||
other acquisition path.
|
||||
|
||||
Quality is judged using the real file (mutagen-measured bit depth / sample rate /
|
||||
bitrate) checked against the user's v3 ranked profile targets — fully profile-driven,
|
||||
no hardcoded thresholds. Transcode/"fake lossless" detection is the separate Fake
|
||||
Lossless Detector job.
|
||||
The quality decision (``meets_preferred_quality``) is a pure function so it can be
|
||||
unit-tested without a database or network. Transcode/"fake lossless" detection is
|
||||
intentionally NOT done here — that's the separate Fake Lossless Detector job.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -40,25 +39,30 @@ from core.discovery.quality_scanner import (
|
|||
)
|
||||
from core.library.file_tags import read_embedded_tags
|
||||
from core.library.path_resolver import resolve_library_file_path
|
||||
# v3 quality: probe the real file + check it against the ranked profile targets,
|
||||
# the SAME definition the download import guard uses. Module-level so they're
|
||||
# monkeypatchable in tests.
|
||||
from core.imports.file_ops import probe_audio_quality
|
||||
from core.quality.model import rank_candidate
|
||||
from core.quality.selection import targets_from_profile, quality_meets_profile
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("repair_jobs.quality_upgrade")
|
||||
|
||||
|
||||
def _to_bool(val) -> bool:
|
||||
"""Coerce a setting value to bool. Handles Python bool, string 'true'/'false', and int."""
|
||||
if isinstance(val, bool):
|
||||
return val
|
||||
if isinstance(val, str):
|
||||
return val.lower() == 'true'
|
||||
return bool(val) if val is not None else False
|
||||
# Quality ranks — higher is better. Lossless tops everything; lossy tiers fall out
|
||||
# of bitrate. 0 means "below the lowest tracked tier / unknown".
|
||||
RANK_LOSSLESS = 4
|
||||
RANK_320 = 3
|
||||
RANK_256 = 2
|
||||
RANK_192 = 1
|
||||
RANK_BELOW = 0
|
||||
|
||||
LOSSLESS_EXTENSIONS = {'.flac', '.alac', '.ape', '.wav', '.aiff', '.aif', '.dsf', '.dff', '.m4a'}
|
||||
# NB: .m4a is ambiguous (ALAC vs AAC); we treat the *format* as lossy-capable and
|
||||
# rely on bitrate below — a true ALAC .m4a reports a lossless-scale bitrate.
|
||||
|
||||
# Quality-profile bucket key -> rank.
|
||||
_PROFILE_KEY_RANK = {
|
||||
'flac': RANK_LOSSLESS,
|
||||
'mp3_320': RANK_320,
|
||||
'mp3_256': RANK_256,
|
||||
'mp3_192': RANK_192,
|
||||
}
|
||||
|
||||
# Per-source file-tag key holding that source's own track ID (written by enrichment).
|
||||
_SOURCE_TRACK_ID_TAG = {
|
||||
|
|
@ -75,6 +79,93 @@ _SOURCE_TRACK_ID_TAG = {
|
|||
_DURATION_TOLERANCE_MS = 5000
|
||||
|
||||
|
||||
def _normalize_kbps(bitrate: Optional[int]) -> Optional[int]:
|
||||
"""Library bitrate may be stored in bps (e.g. 320000) or kbps (320).
|
||||
Normalize to kbps. Returns None when unknown/zero."""
|
||||
if not bitrate:
|
||||
return None
|
||||
try:
|
||||
b = int(bitrate)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if b <= 0:
|
||||
return None
|
||||
return b // 1000 if b > 4000 else b
|
||||
|
||||
|
||||
def classify_track_quality(file_path: str, bitrate: Optional[int]) -> Optional[int]:
|
||||
"""Rank a file by format + bitrate. Returns a RANK_* value, or None when it
|
||||
can't be judged (a lossy file with no known bitrate)."""
|
||||
ext = os.path.splitext(file_path or '')[1].lower()
|
||||
kbps = _normalize_kbps(bitrate)
|
||||
|
||||
# Lossless containers: a real lossless file has a high bitrate; a low one is a
|
||||
# lossy stream in a lossless container — but flagging that is the Fake Lossless
|
||||
# Detector's job, so here we treat the lossless *format* as top rank.
|
||||
if ext in {'.flac', '.alac', '.ape', '.wav', '.aiff', '.aif', '.dsf', '.dff'}:
|
||||
return RANK_LOSSLESS
|
||||
# .m4a / lossy: judge purely by bitrate. A lossless-scale bitrate (ALAC in m4a,
|
||||
# or a mislabeled lossless) ranks as lossless.
|
||||
if kbps is None:
|
||||
return None
|
||||
if kbps >= 800:
|
||||
return RANK_LOSSLESS
|
||||
if kbps >= 280:
|
||||
return RANK_320
|
||||
if kbps >= 200:
|
||||
return RANK_256
|
||||
if kbps >= 150:
|
||||
return RANK_192
|
||||
return RANK_BELOW
|
||||
|
||||
|
||||
def preferred_quality_floor(quality_profile: Dict[str, Any]) -> Optional[int]:
|
||||
"""The lowest acceptable quality rank from the profile's ENABLED buckets — the
|
||||
floor a track must meet. Returns None when nothing is enabled (caller should
|
||||
then flag nothing, rather than flagging everything)."""
|
||||
qualities = (quality_profile or {}).get('qualities', {}) or {}
|
||||
enabled_ranks = [
|
||||
_PROFILE_KEY_RANK[key]
|
||||
for key, cfg in qualities.items()
|
||||
if isinstance(cfg, dict) and cfg.get('enabled') and key in _PROFILE_KEY_RANK
|
||||
]
|
||||
if not enabled_ranks:
|
||||
return None
|
||||
return min(enabled_ranks)
|
||||
|
||||
|
||||
def meets_preferred_quality(file_path: str, bitrate: Optional[int],
|
||||
quality_profile: Dict[str, Any]) -> bool:
|
||||
"""Pure decision: does this track already meet the user's preferred quality?
|
||||
|
||||
A track meets quality when its format+bitrate rank is at least the profile's
|
||||
floor (the worst quality the user still accepts). This honors a profile that
|
||||
enables, say, FLAC *and* MP3-320: a 320 kbps MP3 passes, a 128 kbps MP3 does
|
||||
not. With nothing enabled, everything passes (we never flag the whole library
|
||||
on an empty profile)."""
|
||||
floor = preferred_quality_floor(quality_profile)
|
||||
if floor is None:
|
||||
return True
|
||||
|
||||
file_rank = classify_track_quality(file_path, bitrate)
|
||||
if file_rank is None:
|
||||
# Lossy file with unknown bitrate: only judgeable when the floor is
|
||||
# lossless (then any lossy file is below it). Otherwise don't flag.
|
||||
ext = os.path.splitext(file_path or '')[1].lower()
|
||||
if floor == RANK_LOSSLESS and ext not in LOSSLESS_EXTENSIONS:
|
||||
return False
|
||||
return True
|
||||
|
||||
return file_rank >= floor
|
||||
|
||||
|
||||
def _rank_label(rank: Optional[int]) -> str:
|
||||
return {
|
||||
RANK_LOSSLESS: 'Lossless', RANK_320: 'MP3 320', RANK_256: 'MP3 256',
|
||||
RANK_192: 'MP3 192', RANK_BELOW: 'low bitrate',
|
||||
}.get(rank, 'unknown')
|
||||
|
||||
|
||||
def _norm_isrc(value: Any) -> str:
|
||||
"""Canonicalize an ISRC for comparison: uppercase, strip dashes/spaces."""
|
||||
if not value:
|
||||
|
|
@ -82,14 +173,14 @@ def _norm_isrc(value: Any) -> str:
|
|||
return str(value).upper().replace('-', '').replace(' ', '').strip()
|
||||
|
||||
|
||||
def _read_file_ids(file_path: str, resolved_path: Optional[str] = None) -> Dict[str, str]:
|
||||
def _read_file_ids(file_path: str) -> Dict[str, str]:
|
||||
"""Read the identifiers enrichment embedded in the file's tags.
|
||||
|
||||
Enrichment matches every track to the metadata sources and writes the IDs
|
||||
(ISRC + per-source track IDs) into the file — so an already-enriched track
|
||||
carries its exact identity. Returns a dict with a normalized ``isrc`` plus any
|
||||
``<source>_track_id`` tags present; empty dict when unreadable / not enriched."""
|
||||
resolved = resolved_path or (resolve_library_file_path(file_path) if file_path else None)
|
||||
resolved = resolve_library_file_path(file_path) if file_path else None
|
||||
if not resolved and file_path and os.path.isfile(file_path):
|
||||
resolved = file_path
|
||||
if not resolved:
|
||||
|
|
@ -345,61 +436,45 @@ def _find_best_match(engine: Any, source_priority: List[str], title: str, artist
|
|||
@register_job
|
||||
class QualityUpgradeJob(RepairJob):
|
||||
job_id = 'quality_upgrade'
|
||||
display_name = 'Quality Upgrade Finder (active — proposes a replacement)'
|
||||
description = 'Finds library tracks below your quality profile and actively searches a better version to add to the wishlist'
|
||||
display_name = 'Quality Upgrade Finder'
|
||||
description = 'Finds library tracks below your preferred quality and proposes a better version'
|
||||
help_text = (
|
||||
'ACTIVE quality job. For every library track below your quality profile it '
|
||||
'goes one step further than the Quality Check (flag-only) scanner: it '
|
||||
'actively SEARCHES your metadata source for the exact better version and '
|
||||
'attaches it to the finding, so Apply adds that track straight to your '
|
||||
'wishlist.\n\n'
|
||||
'Quality is judged the SAME way as the download/import pipeline — it reads '
|
||||
'the REAL file with mutagen (measured bit depth / sample rate / bitrate) and '
|
||||
'checks it against your v3 quality profile targets (strict: fallback is '
|
||||
'ignored, that\'s a download-time concession, not "good enough" for an '
|
||||
'upgrade). So a 128 kbps MP3, a 16-bit FLAC where you want 24-bit, etc. are '
|
||||
'all caught accurately.\n\n'
|
||||
'For every below-profile track it resolves the better version by the most '
|
||||
'precise identity available, in order: the source track ID enrichment wrote '
|
||||
"into the file → the file's ISRC → the album's tracklist (by stored album ID "
|
||||
'or album search) → a name/artist search (with a duration guard against wrong '
|
||||
'live/edit cuts). It skips tracks it already proposed, so re-runs are cheap. '
|
||||
'Nothing is queued automatically — applying a finding adds the matched track '
|
||||
'(with album context) to the wishlist, like any other download.\n\n'
|
||||
'Scans your library (or just your watchlist artists) and compares each '
|
||||
"track against your Quality Profile using BOTH the file format and its "
|
||||
'bitrate — so a 128 kbps MP3 is no longer treated the same as a 320 kbps '
|
||||
'one, and enabling MP3-320/256 in your profile actually counts.\n\n'
|
||||
'For every track below your preferred quality it resolves the exact better '
|
||||
'version using the most precise identity available, in order: the source '
|
||||
"track ID enrichment wrote into the file → the file's ISRC → the album's "
|
||||
'tracklist (by stored album ID or album search) → a name/artist search. The '
|
||||
'fuzzy steps also reject candidates whose length is off (wrong live/edit cut). '
|
||||
'It skips tracks it already proposed, so re-runs are cheap. Nothing is queued '
|
||||
'automatically: applying a finding adds that matched track — with its album '
|
||||
'context — to the wishlist, the same as any other download.\n\n'
|
||||
'Settings:\n'
|
||||
'- Scope: "watchlist" (watchlisted artists only) or "all" (whole library)\n'
|
||||
'- Min confidence: minimum match confidence (0-1) to surface a finding\n'
|
||||
'- Deep audio verify (default OFF): also run the ffmpeg decode guard '
|
||||
'(truncation + silence) per track — catches broken/incomplete files the '
|
||||
'header hides, but decodes every file (seconds per track, CPU-heavy).\n\n'
|
||||
'Sibling job: "Quality Check (flag only)" finds the same below-profile tracks '
|
||||
'but only flags them for you to decide per finding (re-download / delete / '
|
||||
'ignore) instead of searching a replacement. Fake/transcoded lossless '
|
||||
'detection is the separate Fake Lossless Detector job.'
|
||||
'- Min confidence: minimum match confidence (0-1) to surface a finding\n\n'
|
||||
'Note: detecting fake/transcoded lossless files is handled by the separate '
|
||||
'Fake Lossless Detector job.'
|
||||
)
|
||||
icon = 'repair-icon-lossy'
|
||||
default_enabled = False
|
||||
default_interval_hours = 168
|
||||
default_settings = {'scope': 'all', 'min_confidence': 0.7, 'deep_audio_verify': False, 'require_top_target': False}
|
||||
setting_options = {'scope': ['all', 'watchlist'], 'deep_audio_verify': [True, False], 'require_top_target': [True, False]}
|
||||
default_settings = {'scope': 'watchlist', 'min_confidence': 0.7}
|
||||
setting_options = {'scope': ['watchlist', 'all']}
|
||||
auto_fix = False
|
||||
|
||||
def _get_settings(self, context: JobContext) -> Dict[str, Any]:
|
||||
merged = dict(self.default_settings)
|
||||
if context.config_manager:
|
||||
cfg = context.config_manager
|
||||
scope = 'watchlist'
|
||||
min_conf = 0.7
|
||||
if cfg:
|
||||
scope = cfg.get(self.get_config_key('settings.scope'), 'watchlist') or 'watchlist'
|
||||
try:
|
||||
cfg = context.config_manager.get(f'repair.jobs.{self.job_id}.settings', {})
|
||||
if isinstance(cfg, dict):
|
||||
merged.update(cfg)
|
||||
except Exception as e:
|
||||
logger.debug("settings read failed: %s", e)
|
||||
try:
|
||||
merged['min_confidence'] = float(merged.get('min_confidence', 0.7))
|
||||
except (TypeError, ValueError):
|
||||
merged['min_confidence'] = 0.7
|
||||
merged['deep_audio_verify'] = _to_bool(merged.get('deep_audio_verify'))
|
||||
merged['require_top_target'] = _to_bool(merged.get('require_top_target'))
|
||||
return merged
|
||||
min_conf = float(cfg.get(self.get_config_key('settings.min_confidence'), 0.7))
|
||||
except (TypeError, ValueError):
|
||||
min_conf = 0.7
|
||||
return {'scope': scope, 'min_confidence': min_conf}
|
||||
|
||||
def _load_tracks(self, db: Any, scope: str) -> List[dict]:
|
||||
conn = db._get_connection()
|
||||
|
|
@ -455,27 +530,13 @@ class QualityUpgradeJob(RepairJob):
|
|||
settings = self._get_settings(context)
|
||||
scope = settings['scope']
|
||||
min_conf = settings['min_confidence']
|
||||
deep_verify = settings['deep_audio_verify']
|
||||
require_top = settings['require_top_target']
|
||||
|
||||
# v3 quality: judge the REAL file (mutagen-measured bit depth / sample rate
|
||||
# / bitrate) against the profile's ranked targets — the SAME definition the
|
||||
# download import guard uses. Strict: fallback is ignored (a download-time
|
||||
# concession, not "good enough" for an upgrade). targets_from_profile /
|
||||
# quality_meets_profile / probe_audio_quality are imported at module level.
|
||||
db = context.db
|
||||
quality_profile = db.get_quality_profile()
|
||||
targets, _fallback = targets_from_profile(quality_profile)
|
||||
if not targets:
|
||||
logger.info("[Quality Upgrade] No quality targets in profile — nothing to flag")
|
||||
if preferred_quality_floor(quality_profile) is None:
|
||||
logger.info("[Quality Upgrade] No quality buckets enabled in profile — nothing to flag")
|
||||
return result
|
||||
|
||||
logger.info(
|
||||
"[Quality Upgrade] scope=%s require_top=%s · all targets: %s",
|
||||
scope, require_top,
|
||||
[t.label for t in targets] if targets else '(none — all pass)',
|
||||
)
|
||||
|
||||
try:
|
||||
tracks = self._load_tracks(db, scope)
|
||||
except Exception as e:
|
||||
|
|
@ -522,53 +583,13 @@ class QualityUpgradeJob(RepairJob):
|
|||
result.findings_skipped_dedup += 1
|
||||
continue
|
||||
|
||||
# v3 quality decision — probe the REAL file. Resolve the library path
|
||||
# first (the DB stores a possibly-relative path). Pass config_manager so
|
||||
# the resolver can find the transfer/music folders and expand relative paths.
|
||||
resolved_path = resolve_library_file_path(
|
||||
file_path,
|
||||
transfer_folder=context.transfer_folder,
|
||||
config_manager=context.config_manager,
|
||||
) if file_path else None
|
||||
if not resolved_path and file_path and os.path.isfile(file_path):
|
||||
resolved_path = file_path
|
||||
|
||||
measured_aq = probe_audio_quality(resolved_path) if resolved_path else None
|
||||
|
||||
# Optional ffmpeg deep verify (default off): a truncated/silent file is
|
||||
# treated as "needs a replacement" just like a below-profile one.
|
||||
broken_reason = None
|
||||
if deep_verify and resolved_path:
|
||||
try:
|
||||
from core.imports.silence import detect_broken_audio
|
||||
broken_reason = detect_broken_audio(resolved_path)
|
||||
except Exception as e:
|
||||
logger.debug("[Quality Upgrade] deep verify failed for %s: %s", file_path, e)
|
||||
|
||||
if measured_aq is None and not broken_reason:
|
||||
# Can't read the file → can't judge it; leave it alone.
|
||||
if meets_preferred_quality(file_path, bitrate, quality_profile):
|
||||
result.skipped += 1
|
||||
if context.update_progress and (i + 1) % 25 == 0:
|
||||
context.update_progress(i + 1, total)
|
||||
continue
|
||||
|
||||
if not broken_reason and measured_aq is not None:
|
||||
if require_top:
|
||||
# ranking-based: skip only if the file already sits at rank 0
|
||||
# (the top configured target). Any lower rank → flag for upgrade.
|
||||
idx, _ = rank_candidate(measured_aq, targets)
|
||||
already_best = (idx == 0)
|
||||
else:
|
||||
# default: skip if the file meets ANY configured target (i.e.
|
||||
# it's not below the acceptable floor).
|
||||
already_best = quality_meets_profile(measured_aq, targets)
|
||||
if already_best:
|
||||
result.skipped += 1
|
||||
if context.update_progress and (i + 1) % 25 == 0:
|
||||
context.update_progress(i + 1, total)
|
||||
continue
|
||||
|
||||
# Below profile (or broken) — find a better version to propose.
|
||||
# Below preferred quality — find a better version to propose.
|
||||
if engine is None:
|
||||
from core.matching_engine import MusicMatchingEngine
|
||||
engine = MusicMatchingEngine()
|
||||
|
|
@ -581,20 +602,17 @@ class QualityUpgradeJob(RepairJob):
|
|||
logger.info("[Quality Upgrade] Spotify rate-limited — stopping scan early")
|
||||
return result
|
||||
|
||||
current_label = measured_aq.label() if measured_aq is not None else 'broken/unreadable'
|
||||
if broken_reason:
|
||||
current_label = f'{current_label} (broken: {broken_reason})' if measured_aq is not None else f'broken ({broken_reason})'
|
||||
current_rank = classify_track_quality(file_path, bitrate)
|
||||
current_label = _rank_label(current_rank)
|
||||
if context.report_progress:
|
||||
_why = 'broken audio' if broken_reason else 'low quality'
|
||||
context.report_progress(
|
||||
scanned=i + 1, total=total,
|
||||
log_line=f'{_why} ({current_label}): {artist_name} - {title}',
|
||||
log_line=f'Low quality ({current_label}): {artist_name} - {title}',
|
||||
log_type='info')
|
||||
|
||||
# Read the identifiers enrichment embedded in the file once (ISRC +
|
||||
# per-source track IDs), used by the two most-exact tiers below.
|
||||
# Pass resolved_path so the inner resolver doesn't redo the lookup.
|
||||
file_ids = _read_file_ids(file_path, resolved_path=resolved_path)
|
||||
file_ids = _read_file_ids(file_path)
|
||||
|
||||
# Tiered match, best identity first, loosest last:
|
||||
# 0. The active source's OWN track ID, embedded in the file by
|
||||
|
|
@ -667,9 +685,8 @@ class QualityUpgradeJob(RepairJob):
|
|||
file_path=file_path,
|
||||
title=f'Upgrade: {artist_name} - {title} ({current_label})',
|
||||
description=(
|
||||
f'"{title}" by {artist_name} is {current_label}'
|
||||
+ (f', below your preferred quality ({targets[0].label})' if require_top and targets else ', below your preferred quality')
|
||||
+ f'. Best match: "{_track_name(best)}" via {source} '
|
||||
f'"{title}" by {artist_name} is {current_label}, below your preferred '
|
||||
f'quality. Best match: "{_track_name(best)}" via {source} '
|
||||
f'({_MATCH_NOTE.get(matched_via, "matched") if matched_via != "search" else f"confidence {conf:.0%}"}). '
|
||||
'Apply to add it to the wishlist.'),
|
||||
details={
|
||||
|
|
@ -698,13 +715,6 @@ class QualityUpgradeJob(RepairJob):
|
|||
|
||||
if context.update_progress:
|
||||
context.update_progress(total, total)
|
||||
logger.info("[Quality Upgrade] %d scanned, %d upgrades found, %d already met profile / skipped",
|
||||
logger.info("[Quality Upgrade] %d scanned, %d upgrades found, %d met/skip",
|
||||
result.scanned, result.findings_created, result.skipped)
|
||||
if result.scanned > 0 and result.findings_created == 0 and result.errors == 0:
|
||||
logger.info(
|
||||
"[Quality Upgrade] All tracks already satisfy the configured targets. "
|
||||
"If you expected upgrades, check your quality profile — the current "
|
||||
"top target is: %s",
|
||||
targets[0].label if targets else '(none)',
|
||||
)
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -1,416 +0,0 @@
|
|||
"""Quality Upgrade Scanner Job — flags library tracks below the user's profile.
|
||||
|
||||
Walks the music folders ON DISK (transfer + download + every configured library
|
||||
path) exactly like the Orphan / Fake-Lossless detectors — those reliably "see"
|
||||
files because they os.walk real directories instead of trying to resolve the
|
||||
DB's stored (often relative) paths. For each audio file it probes the ACTUAL
|
||||
measured audio quality (bit depth / sample rate / bitrate via the same
|
||||
`probe_audio_quality` the download import guard uses) and checks it against the
|
||||
user's v3 ranked targets with `quality_meets_profile` (strict — fallback
|
||||
ignored, that's a download-time concession, not a definition of "good enough").
|
||||
|
||||
Every file that satisfies none of the targets becomes a finding the user can:
|
||||
- 'redownload': add the track to the wishlist and delete the low-quality file
|
||||
- 'delete': remove the low-quality file (+ DB row when known)
|
||||
- 'ignore': dismiss the finding (handled in the UI via the dismiss endpoint)
|
||||
|
||||
Each walked file is matched back to its DB track (by path suffix) so the finding
|
||||
carries the real title/artist/album + track id; when no DB row matches, the
|
||||
file's own tags are used and the finding is filed as a loose 'file'.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from core.repair_jobs import register_job
|
||||
from core.repair_jobs.base import JobContext, JobResult, RepairJob
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("repair_job.quality_upgrade")
|
||||
|
||||
AUDIO_EXTENSIONS = {'.mp3', '.flac', '.ogg', '.opus', '.m4a', '.aac', '.wav', '.wma', '.aiff', '.aif'}
|
||||
|
||||
|
||||
@register_job
|
||||
class QualityUpgradeScannerJob(RepairJob):
|
||||
job_id = 'quality_upgrade_scanner'
|
||||
display_name = 'Quality Check (flag only — you decide per finding)'
|
||||
description = 'Flags library tracks below your quality profile; you choose re-download / delete / ignore per finding'
|
||||
help_text = (
|
||||
'FLAG-ONLY quality job. Walks your music library folder on disk (so it also '
|
||||
'catches loose files not in the DB) and checks every track against your v3 '
|
||||
'quality profile — then just FLAGS what is below profile. Unlike the active '
|
||||
'"Quality Upgrade Finder", it does NOT search a replacement; you decide what '
|
||||
'to do per finding: Re-download / Delete / Ignore.\n\n'
|
||||
'Two-stage check (same as the download/import pipeline):\n'
|
||||
'1. Real-audio guard (optional, ffmpeg) — decodes the file (truncation + '
|
||||
'silence detection) to catch broken/incomplete audio the header hides.\n'
|
||||
'2. Quality gate — measured bit depth / sample rate / bitrate vs your '
|
||||
'profile targets.\n\n'
|
||||
'Settings:\n'
|
||||
'- Deep audio verify (default OFF): run the ffmpeg decode guard. Off = fast '
|
||||
'header-only quality pass (milliseconds/track). On = full decode '
|
||||
'(seconds/track, CPU-heavy) but catches broken/silent audio.\n'
|
||||
'- library_tracks_only (default off): only check files matched to a '
|
||||
'library DB track (skip loose/orphan files).\n\n'
|
||||
'The scan only reports — it never deletes or re-downloads on its own. '
|
||||
'Use the sibling "Quality Upgrade Finder" instead if you want it to actively '
|
||||
'find and queue a better version for you.'
|
||||
)
|
||||
icon = 'repair-icon-lossless'
|
||||
default_enabled = False
|
||||
default_interval_hours = 168
|
||||
# library_tracks_only: when ON, only check files that match a library DB
|
||||
# track (skips loose/orphan files). Default OFF — the scan checks EVERY
|
||||
# audio file in the Music Library output folder, which is what users expect
|
||||
# ("check my library folder"). DB matching after a reset is unreliable and
|
||||
# would wrongly skip everything. Turn ON to ignore non-DB files.
|
||||
#
|
||||
# deep_audio_verify default OFF: the ffmpeg decode is the CPU-heavy step. Most
|
||||
# users want the fast header-only quality pass; turn it on for a deep scan that
|
||||
# also catches broken/silent audio. (Matches the download pipeline's default.)
|
||||
default_settings = {'library_tracks_only': False, 'deep_audio_verify': False, 'require_top_target': False}
|
||||
setting_options = {'library_tracks_only': [True, False],
|
||||
'deep_audio_verify': [True, False],
|
||||
'require_top_target': [True, False]}
|
||||
auto_fix = False # User chooses fix action per finding
|
||||
|
||||
def scan(self, context: JobContext) -> JobResult:
|
||||
result = JobResult()
|
||||
|
||||
# Load the user's v3 ranked targets — the SAME definition the download
|
||||
# import guard uses. Strict: a track is below-profile when its measured
|
||||
# quality satisfies NONE of the targets (fallback is not consulted).
|
||||
from core.quality.selection import targets_from_profile, quality_meets_profile
|
||||
try:
|
||||
profile = context.db.get_quality_profile()
|
||||
except Exception as e:
|
||||
logger.warning("Could not load quality profile: %s", e)
|
||||
return result
|
||||
targets, _fallback = targets_from_profile(profile)
|
||||
if not targets:
|
||||
logger.info("Quality profile has no targets — nothing to check against")
|
||||
return result
|
||||
|
||||
logger.info("Quality upgrade scan — profile targets (strict): %s",
|
||||
[t.label for t in targets])
|
||||
|
||||
from core.imports.file_ops import probe_audio_quality
|
||||
# Same real-file AudioGuard the download/import pipeline runs: ffmpeg
|
||||
# DECODES the file (astats + silencedetect) to catch truncated or
|
||||
# mostly-silent audio the header can't reveal.
|
||||
from core.imports.silence import detect_broken_audio
|
||||
|
||||
# --- Collect the music folders to walk (real dirs, abspath'd) ---
|
||||
base_dirs = self._collect_music_dirs(context)
|
||||
if not base_dirs:
|
||||
logger.warning(
|
||||
"[QualityScan] No existing music folder to walk (transfer=%r, cwd=%r). "
|
||||
"Set soulseek.transfer_path to the real mount or add your library under "
|
||||
"Settings → Library → Music Paths.",
|
||||
context.transfer_folder, os.getcwd())
|
||||
return result
|
||||
logger.info("[QualityScan] Walking %d folder(s): %r", len(base_dirs), base_dirs)
|
||||
|
||||
# --- Gather audio files (dedup by real path) ---
|
||||
audio_files = []
|
||||
seen = set()
|
||||
for base in base_dirs:
|
||||
for root, _dirs, files in os.walk(base):
|
||||
if context.check_stop():
|
||||
return result
|
||||
for fname in files:
|
||||
if os.path.splitext(fname)[1].lower() in AUDIO_EXTENSIONS:
|
||||
fpath = os.path.join(root, fname)
|
||||
rp = os.path.realpath(fpath)
|
||||
if rp in seen:
|
||||
continue
|
||||
seen.add(rp)
|
||||
audio_files.append(fpath)
|
||||
|
||||
total = len(audio_files)
|
||||
logger.info("[QualityScan] Found %d audio file(s) to check", total)
|
||||
if context.report_progress:
|
||||
context.report_progress(phase=f'Checking {total} files...', total=total)
|
||||
if context.update_progress:
|
||||
context.update_progress(0, total)
|
||||
|
||||
# --- DB suffix index so a walked file maps back to its track row ---
|
||||
db_index = self._build_db_suffix_index(context)
|
||||
# Only check files that are part of the LIBRARY (have a DB track row).
|
||||
# The transfer/download folders also hold pre-import leftovers (e.g.
|
||||
# residue after a DB reset) — those are orphans, not library tracks, and
|
||||
# belong to the Orphan File Detector, not a quality upgrade scan. Default
|
||||
# ON so the scan reflects the user's actual library, not download junk.
|
||||
_settings = self._get_settings(context)
|
||||
library_only = _settings.get('library_tracks_only', False)
|
||||
# Deep verify = run the ffmpeg AudioGuard (real decode) per file, exactly
|
||||
# like the download pipeline. Slower than a header read (seconds vs ms) but
|
||||
# it verifies the REAL audio, not just the metadata. OFF by default (the
|
||||
# decode is the CPU-heavy step); turn on for a deep scan.
|
||||
deep_verify = _settings.get('deep_audio_verify', False)
|
||||
# require_top_target: flag files that meet a lower target but not the
|
||||
# highest-priority one (e.g. 16-bit FLAC when 24-bit is preferred).
|
||||
require_top = _settings.get('require_top_target', False)
|
||||
check_targets = targets[:1] if require_top and len(targets) > 1 else targets
|
||||
|
||||
probe_failed = 0
|
||||
not_in_library = 0
|
||||
for i, fpath in enumerate(audio_files):
|
||||
if context.check_stop():
|
||||
return result
|
||||
if i % 20 == 0 and context.wait_if_paused():
|
||||
return result
|
||||
|
||||
fname = os.path.basename(fpath)
|
||||
|
||||
# Map to a DB track up front (cheap suffix lookup). When scoping to
|
||||
# the library, skip anything with no DB row BEFORE probing — no point
|
||||
# reading hundreds of orphan files.
|
||||
meta = self._match_db(fpath, db_index)
|
||||
if library_only and meta is None:
|
||||
not_in_library += 1
|
||||
result.skipped += 1
|
||||
continue
|
||||
if meta is None:
|
||||
meta = self._read_file_tags(fpath)
|
||||
|
||||
result.scanned += 1
|
||||
if context.report_progress and i % 25 == 0:
|
||||
context.report_progress(
|
||||
scanned=i + 1, total=total,
|
||||
phase=f'Checking {i + 1} / {total}',
|
||||
log_line=f'Checking: {fname}',
|
||||
log_type='info',
|
||||
)
|
||||
|
||||
# === Real-file verification — the SAME two stages the download /
|
||||
# import pipeline runs on every file ===
|
||||
# 1) AudioGuard: ffmpeg DECODES the audio (astats / silencedetect)
|
||||
# to catch truncated or mostly-silent files the header hides.
|
||||
# 2) Quality gate: measured quality (mutagen) vs the ranked profile.
|
||||
try:
|
||||
broken_reason = detect_broken_audio(fpath) if deep_verify else None
|
||||
except Exception as e:
|
||||
logger.debug("AudioGuard failed for %s: %s", fname, e)
|
||||
broken_reason = None
|
||||
|
||||
try:
|
||||
aq = probe_audio_quality(fpath)
|
||||
except Exception as e:
|
||||
logger.debug("Probe failed for %s: %s", fname, e)
|
||||
aq = None
|
||||
|
||||
if broken_reason:
|
||||
issue = 'broken_audio'
|
||||
current_label = aq.label() if aq is not None else 'unknown'
|
||||
elif aq is None:
|
||||
# Header unreadable → can't judge quality; leave it unflagged.
|
||||
probe_failed += 1
|
||||
result.skipped += 1
|
||||
continue
|
||||
elif not quality_meets_profile(aq, check_targets):
|
||||
issue = 'below_profile'
|
||||
current_label = aq.label()
|
||||
else:
|
||||
# Decodes fully AND meets the profile → genuinely good.
|
||||
if context.update_progress and (i + 1) % 25 == 0:
|
||||
context.update_progress(i + 1, total)
|
||||
continue
|
||||
|
||||
# Build the finding (broken audio OR below profile).
|
||||
target_labels = [t.label for t in targets]
|
||||
disp_title = meta.get('title') or os.path.splitext(fname)[0]
|
||||
disp_artist = meta.get('artist') or 'Unknown'
|
||||
if issue == 'broken_audio':
|
||||
_title = f'Broken/incomplete audio: {disp_title}'
|
||||
_desc = (f'"{disp_title}" by {disp_artist} failed real-audio '
|
||||
f'verification (ffmpeg): {broken_reason}')
|
||||
_severity = 'warning'
|
||||
else:
|
||||
_pref = targets[0].label if require_top and len(targets) > 1 else None
|
||||
_title = f'{"Upgradeable" if _pref else "Below quality"}: {disp_title} ({current_label})'
|
||||
_desc = (f'"{disp_title}" by {disp_artist} is {current_label}'
|
||||
+ (f', below your preferred quality ({_pref}).' if _pref else
|
||||
f', which does not meet your quality profile '
|
||||
f'({", ".join(target_labels[:3])}'
|
||||
f'{"…" if len(target_labels) > 3 else ""}).'))
|
||||
_severity = 'info'
|
||||
|
||||
if context.report_progress:
|
||||
context.report_progress(log_line=_title, log_type='error')
|
||||
if context.create_finding:
|
||||
inserted = context.create_finding(
|
||||
job_id=self.job_id,
|
||||
finding_type='quality_upgrade',
|
||||
severity=_severity,
|
||||
entity_type='track' if meta.get('track_id') else 'file',
|
||||
entity_id=str(meta['track_id']) if meta.get('track_id') else None,
|
||||
file_path=fpath,
|
||||
title=_title,
|
||||
description=_desc,
|
||||
details={
|
||||
'quality_issue': issue,
|
||||
'broken_audio_reason': broken_reason or '',
|
||||
'current_quality': current_label,
|
||||
'current_format': aq.format if aq is not None else '',
|
||||
'current_bitrate': aq.bitrate if aq is not None else None,
|
||||
'current_sample_rate': aq.sample_rate if aq is not None else None,
|
||||
'current_bit_depth': aq.bit_depth if aq is not None else None,
|
||||
'target_qualities': target_labels,
|
||||
'expected_title': disp_title,
|
||||
'expected_artist': disp_artist,
|
||||
'album_title': meta.get('album', ''),
|
||||
'track_number': meta.get('track_number'),
|
||||
'album_thumb_url': meta.get('album_thumb_url'),
|
||||
'artist_thumb_url': meta.get('artist_thumb_url'),
|
||||
},
|
||||
)
|
||||
if inserted:
|
||||
result.findings_created += 1
|
||||
else:
|
||||
result.findings_skipped_dedup += 1
|
||||
|
||||
if context.update_progress and (i + 1) % 25 == 0:
|
||||
context.update_progress(i + 1, total)
|
||||
|
||||
if context.update_progress:
|
||||
context.update_progress(total, total)
|
||||
|
||||
if probe_failed:
|
||||
logger.warning("[QualityScan] %d/%d files could not be probed (unreadable)",
|
||||
probe_failed, total)
|
||||
if not_in_library:
|
||||
logger.info(
|
||||
"[QualityScan] %d/%d files skipped — not in the library DB (orphan "
|
||||
"leftovers in transfer/downloads; disable 'library_tracks_only' to "
|
||||
"include them)", not_in_library, total)
|
||||
logger.info("Quality upgrade scan: %d checked, %d below profile, %d skipped",
|
||||
result.scanned, result.findings_created, result.skipped)
|
||||
return result
|
||||
|
||||
def _get_settings(self, context: JobContext) -> dict:
|
||||
merged = dict(self.default_settings)
|
||||
if context.config_manager:
|
||||
try:
|
||||
cfg = context.config_manager.get(f'repair.jobs.{self.job_id}.settings', {})
|
||||
if isinstance(cfg, dict):
|
||||
merged.update(cfg)
|
||||
except Exception as e:
|
||||
logger.debug("settings read failed: %s", e)
|
||||
for key in ('library_tracks_only', 'deep_audio_verify', 'require_top_target'):
|
||||
val = merged.get(key)
|
||||
if not isinstance(val, bool):
|
||||
merged[key] = str(val).lower() == 'true' if val is not None else False
|
||||
return merged
|
||||
|
||||
def _collect_music_dirs(self, context: JobContext) -> list:
|
||||
"""The music-library directories to walk, as absolute paths (dedup).
|
||||
|
||||
Only the user's MUSIC LIBRARY is scanned — that's the "Output Folder
|
||||
(Music Library)" setting (soulseek.transfer_path) plus any custom
|
||||
library paths (library.music_paths, for media-server setups). The
|
||||
download/staging folders are deliberately NOT walked: they hold raw,
|
||||
pre-import downloads and leftovers, not the finished library, and the
|
||||
user expects quality checks to run on their library only. Whatever
|
||||
custom path the user configured for the output folder is respected,
|
||||
because it's read live from config here.
|
||||
"""
|
||||
cm = context.config_manager
|
||||
raw = [context.transfer_folder]
|
||||
if cm:
|
||||
try:
|
||||
raw.append(cm.get('soulseek.transfer_path', './Transfer'))
|
||||
mp = cm.get('library.music_paths', []) or []
|
||||
if isinstance(mp, list):
|
||||
raw.extend([p for p in mp if isinstance(p, str) and p.strip()])
|
||||
except Exception as e:
|
||||
logger.debug("music dir config read failed: %s", e)
|
||||
out, seen = [], set()
|
||||
for d in raw:
|
||||
if not d:
|
||||
continue
|
||||
ad = os.path.abspath(d)
|
||||
if ad in seen:
|
||||
continue
|
||||
seen.add(ad)
|
||||
if os.path.isdir(ad):
|
||||
out.append(ad)
|
||||
return out
|
||||
|
||||
def _build_db_suffix_index(self, context: JobContext) -> dict:
|
||||
"""Map normalized path suffixes (last 1-3 components, lowercased) →
|
||||
track metadata, so a walked absolute file can be matched to its DB row
|
||||
even when the DB stores a different (relative) path prefix."""
|
||||
index = {}
|
||||
conn = None
|
||||
try:
|
||||
conn = context.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title,
|
||||
COALESCE(NULLIF(t.track_artist, ''), ar.name) AS artist,
|
||||
t.file_path, t.track_number,
|
||||
al.title AS album_title, al.thumb_url, ar.thumb_url
|
||||
FROM tracks t
|
||||
LEFT JOIN artists ar ON ar.id = t.artist_id
|
||||
LEFT JOIN albums al ON al.id = t.album_id
|
||||
WHERE t.file_path IS NOT NULL AND t.file_path != ''
|
||||
""")
|
||||
for row in cursor.fetchall():
|
||||
fp = (row[3] or '').replace('\\', '/')
|
||||
if not fp:
|
||||
continue
|
||||
parts = fp.split('/')
|
||||
meta = {
|
||||
'track_id': row[0],
|
||||
'title': row[1] or '',
|
||||
'artist': row[2] or '',
|
||||
'track_number': row[4],
|
||||
'album': row[5] or '',
|
||||
'album_thumb_url': row[6] or None,
|
||||
'artist_thumb_url': row[7] or None,
|
||||
}
|
||||
for depth in range(1, min(4, len(parts) + 1)):
|
||||
suffix = '/'.join(parts[-depth:]).lower()
|
||||
index.setdefault(suffix, meta)
|
||||
except Exception as e:
|
||||
logger.error("Error building DB suffix index: %s", e)
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
return index
|
||||
|
||||
def _match_db(self, fpath: str, db_index: dict):
|
||||
"""Match a walked file to a DB track via path suffix. Returns the track
|
||||
meta dict, or None when the file isn't part of the library."""
|
||||
parts = fpath.replace('\\', '/').split('/')
|
||||
for depth in range(min(3, len(parts)), 0, -1):
|
||||
suffix = '/'.join(parts[-depth:]).lower()
|
||||
hit = db_index.get(suffix)
|
||||
if hit:
|
||||
return hit
|
||||
return None
|
||||
|
||||
def _read_file_tags(self, fpath: str) -> dict:
|
||||
"""Read title/artist/album from the file's own tags (for loose files
|
||||
when library_tracks_only is off)."""
|
||||
meta = {'track_id': None}
|
||||
try:
|
||||
from mutagen import File as MutagenFile
|
||||
audio = MutagenFile(fpath, easy=True)
|
||||
if audio:
|
||||
meta['title'] = (audio.get('title') or [None])[0] or ''
|
||||
meta['artist'] = (audio.get('artist') or audio.get('albumartist') or [None])[0] or ''
|
||||
meta['album'] = (audio.get('album') or [None])[0] or ''
|
||||
except Exception as e:
|
||||
logger.debug("tag read failed for %s: %s", os.path.basename(fpath), e)
|
||||
return meta
|
||||
|
||||
def estimate_scope(self, context: JobContext) -> int:
|
||||
count = 0
|
||||
for base in self._collect_music_dirs(context):
|
||||
for _root, _dirs, files in os.walk(base):
|
||||
for fname in files:
|
||||
if os.path.splitext(fname)[1].lower() in AUDIO_EXTENSIONS:
|
||||
count += 1
|
||||
return count
|
||||
|
|
@ -1,267 +0,0 @@
|
|||
"""Repair job: detect ~30s PREVIEW clips and re-fetch the full track.
|
||||
|
||||
The HiFi endpoint (and occasionally others) sometimes deliver a ~30-second preview/sample
|
||||
instead of the full song. Those land in the library looking like real tracks. This job
|
||||
finds short tracks, looks up the EXPECTED length from the track's metadata source, and —
|
||||
when the source says the real track is meaningfully longer than the file — flags it as a
|
||||
preview clip.
|
||||
|
||||
Approving the finding (in repair_worker._fix_short_preview_track) deletes the preview file,
|
||||
drops the DB row so the track goes missing again, and re-adds it to the wishlist with a full
|
||||
payload so the real version gets downloaded. The scan itself ONLY creates findings — nothing
|
||||
is deleted, removed, or wishlisted without the user approving, exactly like the other tools.
|
||||
|
||||
Conservative by design (it deletes a file): a track is only flagged when the source confirms
|
||||
it should be much longer. Genuine short tracks (intros, skits, interludes — where the source
|
||||
agrees the track is short) are left alone, and tracks whose length can't be verified from a
|
||||
source are skipped, never flagged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from core.repair_jobs import register_job
|
||||
from core.repair_jobs.base import JobContext, JobResult, RepairJob
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("repair_jobs.short_preview")
|
||||
|
||||
|
||||
def _art_from_details(details: Dict[str, Any]) -> Optional[str]:
|
||||
"""Pull a renderable album-art URL out of a get_track_details() response. The cleaned dict
|
||||
doesn't carry images, but raw_data does: Spotify → raw_data.album.images[0].url, iTunes →
|
||||
raw_data.artworkUrl100 (upscaled). Returns None if neither is present."""
|
||||
raw = (details or {}).get("raw_data") or {}
|
||||
album = raw.get("album")
|
||||
if isinstance(album, dict):
|
||||
images = album.get("images")
|
||||
if isinstance(images, list) and images and isinstance(images[0], dict) and images[0].get("url"):
|
||||
return images[0]["url"]
|
||||
art = raw.get("artworkUrl100") or raw.get("artworkUrl60") or raw.get("artworkUrl30")
|
||||
if art:
|
||||
# iTunes serves tiny thumbnails by default; bump to a usable size.
|
||||
for small in ("100x100bb", "60x60bb", "30x30bb"):
|
||||
art = art.replace(small, "600x600bb")
|
||||
return art
|
||||
return None
|
||||
|
||||
|
||||
@register_job
|
||||
class ShortPreviewTrackJob(RepairJob):
|
||||
job_id = "short_preview_track"
|
||||
display_name = "Preview Clip Cleanup"
|
||||
description = (
|
||||
"Finds ~30s preview clips that slipped in instead of the full song (common from the "
|
||||
"HiFi endpoint) and re-fetches the real track."
|
||||
)
|
||||
help_text = (
|
||||
"Some downloads — especially via the HiFi source — deliver a ~30-second preview clip "
|
||||
"instead of the full song. They look like normal tracks in your library. This job scans "
|
||||
"for short tracks, checks how long the track ACTUALLY is from its metadata source "
|
||||
"(Spotify / iTunes / MusicBrainz), and flags any whose real length is much greater than "
|
||||
"the file — i.e. a preview.\n\n"
|
||||
"Approving a finding deletes the preview file, removes the track from the database (so it "
|
||||
"shows as missing), and re-adds it to your Wishlist so the full version downloads.\n\n"
|
||||
"It's conservative: genuine short tracks (intros, skits) where the source agrees the track "
|
||||
"is short are left alone, and tracks whose length can't be verified are skipped.\n\n"
|
||||
"Settings:\n"
|
||||
" - max_duration_seconds: only tracks at or below this length are considered (default 30).\n"
|
||||
" - min_expected_drift_seconds: the source must say the real track is at least this many "
|
||||
"seconds longer than the file before it's flagged (default 30)."
|
||||
)
|
||||
icon = "scissors"
|
||||
default_enabled = False
|
||||
default_interval_hours = 168 # weekly
|
||||
default_settings = {
|
||||
"max_duration_seconds": 30,
|
||||
"min_expected_drift_seconds": 30,
|
||||
}
|
||||
setting_options: Dict[str, list] = {}
|
||||
auto_fix = False
|
||||
|
||||
def _setting_int(self, context: JobContext, key: str, default: int) -> int:
|
||||
cm = getattr(context, "config_manager", None)
|
||||
if cm is None:
|
||||
return default
|
||||
try:
|
||||
return int(cm.get(self.get_config_key(key), default) or default)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
def scan(self, context: JobContext) -> JobResult:
|
||||
result = JobResult()
|
||||
max_dur_s = self._setting_int(context, "max_duration_seconds", 30)
|
||||
min_drift_s = self._setting_int(context, "min_expected_drift_seconds", 30)
|
||||
max_dur_ms = max_dur_s * 1000
|
||||
|
||||
conn = context.db._get_connection()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT t.id, t.title, t.duration, t.file_path,
|
||||
t.spotify_track_id, t.itunes_track_id, t.musicbrainz_recording_id,
|
||||
ar.name AS artist_name, ar.thumb_url AS artist_thumb,
|
||||
al.title AS album_title, al.thumb_url AS album_thumb
|
||||
FROM tracks t
|
||||
LEFT JOIN artists ar ON ar.id = t.artist_id
|
||||
LEFT JOIN albums al ON al.id = t.album_id
|
||||
WHERE t.duration IS NOT NULL AND t.duration > 0 AND t.duration <= ?
|
||||
AND t.file_path IS NOT NULL AND t.file_path != ''
|
||||
""",
|
||||
(max_dur_ms,),
|
||||
)
|
||||
rows = [dict(r) for r in cursor.fetchall()]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
total = len(rows)
|
||||
if context.report_progress:
|
||||
try:
|
||||
context.report_progress(phase=f"Checking {total} short tracks for previews…", total=total)
|
||||
except Exception: # noqa: S110 — progress is best-effort
|
||||
pass
|
||||
|
||||
for i, row in enumerate(rows):
|
||||
if context.check_stop() or context.wait_if_paused():
|
||||
break
|
||||
result.scanned += 1
|
||||
|
||||
title = row["title"] or "Unknown"
|
||||
artist = row["artist_name"] or "Unknown"
|
||||
# Live progress EVERY track — the source lookup below is a network call, so without
|
||||
# per-track reporting the UI looks frozen at "Starting…" (the #937-follow-up report).
|
||||
if context.update_progress:
|
||||
try:
|
||||
context.update_progress(i + 1, total)
|
||||
except Exception: # noqa: S110 — best-effort
|
||||
pass
|
||||
if context.report_progress:
|
||||
try:
|
||||
context.report_progress(
|
||||
phase=f"Checking {i + 1}/{total} short tracks for previews…",
|
||||
log_line=f"{artist} — {title}", scanned=i + 1, total=total,
|
||||
)
|
||||
except Exception: # noqa: S110 — best-effort
|
||||
pass
|
||||
|
||||
file_dur_s = (row["duration"] or 0) / 1000.0
|
||||
source = self._lookup_source(context, row)
|
||||
|
||||
# Can't verify the real length → never flag (a delete must be backed by evidence).
|
||||
if source is None:
|
||||
result.skipped += 1
|
||||
continue
|
||||
expected_dur_s = source["duration_s"]
|
||||
# Prefer the source's album art (a renderable CDN url) over the library thumb, which
|
||||
# is often empty/non-renderable for un-enriched HiFi previews → art-less wishlist orb.
|
||||
album_image = source.get("album_image") or row["album_thumb"]
|
||||
|
||||
# Source agrees the track is short (genuine intro/skit) → leave it alone. Only a
|
||||
# source that says the real track is MUCH longer than the file marks a preview.
|
||||
if (expected_dur_s - file_dur_s) < min_drift_s:
|
||||
result.skipped += 1
|
||||
continue
|
||||
|
||||
if context.create_finding:
|
||||
title = row["title"] or "Unknown"
|
||||
artist = row["artist_name"] or "Unknown"
|
||||
try:
|
||||
inserted = context.create_finding(
|
||||
job_id=self.job_id,
|
||||
finding_type="short_preview_track",
|
||||
severity="warning",
|
||||
entity_type="track",
|
||||
entity_id=str(row["id"]),
|
||||
file_path=row["file_path"],
|
||||
title=f"Preview clip: {artist} - {title}",
|
||||
description=(
|
||||
f'File is {file_dur_s:.0f}s but "{title}" by {artist} is '
|
||||
f"{expected_dur_s:.0f}s at the source — looks like a preview clip. "
|
||||
"Approve to delete it and re-download the full version."
|
||||
),
|
||||
details={
|
||||
"track_id": row["id"],
|
||||
"title": row["title"],
|
||||
"artist": row["artist_name"],
|
||||
"album": row["album_title"],
|
||||
"album_thumb_url": album_image,
|
||||
"artist_thumb_url": row["artist_thumb"],
|
||||
"file_duration_s": round(file_dur_s, 1),
|
||||
"expected_duration_s": round(expected_dur_s, 1),
|
||||
"original_path": row["file_path"],
|
||||
},
|
||||
)
|
||||
if inserted:
|
||||
result.findings_created += 1
|
||||
else:
|
||||
result.findings_skipped_dedup += 1
|
||||
except Exception as exc:
|
||||
logger.debug("create_finding failed for track %s: %s", row["id"], exc)
|
||||
result.errors += 1
|
||||
|
||||
return result
|
||||
|
||||
def _lookup_source(self, context: JobContext, row: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""Look the track up at its metadata source: returns {'duration_s', 'album_image'} or
|
||||
None when no source id is usable / the lookup fails. The SAME lookup that confirms the
|
||||
real length also carries the album art (in raw_data), which we capture so the re-wishlist
|
||||
isn't art-less when the library album thumb is missing (the #937-follow-up: HiFi previews
|
||||
on un-enriched albums). Every metadata client exposes get_track_details(id)."""
|
||||
|
||||
def _build(details) -> Optional[Dict[str, Any]]:
|
||||
ms = (details or {}).get("duration_ms")
|
||||
if not ms or ms <= 0:
|
||||
return None
|
||||
return {"duration_s": ms / 1000.0, "album_image": _art_from_details(details)}
|
||||
|
||||
# Spotify — pass allow_fallback=False. The default fallback scrapes the configured
|
||||
# metadata source, which is slow and can BLOCK a scan loop indefinitely when the
|
||||
# official API isn't authed (the #937-follow-up hang). Official-only is fast and
|
||||
# returns None cleanly when unavailable, so we just move to the next source.
|
||||
sp_id = row.get("spotify_track_id")
|
||||
if sp_id and context.spotify_client and not context.is_spotify_rate_limited():
|
||||
try:
|
||||
r = _build(context.spotify_client.get_track_details(str(sp_id), allow_fallback=False))
|
||||
if r:
|
||||
return r
|
||||
except TypeError:
|
||||
pass # older client without the flag — skip, don't risk the slow path
|
||||
except Exception as exc:
|
||||
logger.debug("spotify lookup failed for %s: %s", sp_id, exc)
|
||||
|
||||
# iTunes (public API, no auth, fast) then MusicBrainz.
|
||||
for source_id, client in (
|
||||
(row.get("itunes_track_id"), context.itunes_client),
|
||||
(row.get("musicbrainz_recording_id"), context.mb_client),
|
||||
):
|
||||
if not source_id or client is None:
|
||||
continue
|
||||
getter = getattr(client, "get_track_details", None)
|
||||
if getter is None:
|
||||
continue
|
||||
try:
|
||||
r = _build(getter(str(source_id)))
|
||||
if r:
|
||||
return r
|
||||
except Exception as exc:
|
||||
logger.debug("lookup failed for %s: %s", source_id, exc)
|
||||
return None
|
||||
|
||||
def estimate_scope(self, context: JobContext) -> int:
|
||||
try:
|
||||
max_dur_ms = self._setting_int(context, "max_duration_seconds", 30) * 1000
|
||||
conn = context.db._get_connection()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"SELECT COUNT(*) FROM tracks WHERE duration > 0 AND duration <= ? "
|
||||
"AND file_path IS NOT NULL AND file_path != ''",
|
||||
(max_dur_ms,),
|
||||
)
|
||||
return (cursor.fetchone() or [0])[0]
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception:
|
||||
return 0
|
||||
|
|
@ -994,10 +994,9 @@ class RepairWorker:
|
|||
'unwanted_content': self._fix_unwanted_content,
|
||||
'unknown_artist': self._fix_unknown_artist,
|
||||
'acoustid_mismatch': self._fix_acoustid_mismatch,
|
||||
'quality_upgrade': self._fix_quality_upgrade,
|
||||
'missing_discography_track': self._fix_discography_backfill,
|
||||
'library_retag': self._fix_library_retag,
|
||||
'short_preview_track': self._fix_short_preview_track,
|
||||
'quality_upgrade': self._fix_quality_upgrade,
|
||||
}
|
||||
handler = handlers.get(finding_type)
|
||||
if not handler:
|
||||
|
|
@ -1025,43 +1024,9 @@ class RepairWorker:
|
|||
return {'success': False, 'error': str(e)}
|
||||
|
||||
def _fix_quality_upgrade(self, entity_type, entity_id, file_path, details):
|
||||
"""Apply a Quality Upgrade finding (user-approved; the old Quality
|
||||
Scanner did this without review). Action via ``details['_fix_action']``:
|
||||
|
||||
'redownload' (default): add the matched higher-quality version to the
|
||||
wishlist (with album context) for a profile-gated re-download.
|
||||
The low-quality file stays in place — it's replaced only after the
|
||||
better version actually imports (safe pattern; auto-delete-on-
|
||||
import is handled separately).
|
||||
'delete': remove the low-quality file + its DB row outright.
|
||||
'ignore' is handled in the UI by dismissing the finding — never here.
|
||||
"""
|
||||
fix_action = details.get('_fix_action', 'redownload')
|
||||
|
||||
if fix_action == 'delete':
|
||||
if file_path:
|
||||
resolved = _resolve_file_path(
|
||||
file_path, self.transfer_folder,
|
||||
config_manager=self._config_manager)
|
||||
if resolved and os.path.exists(resolved):
|
||||
try:
|
||||
os.remove(resolved)
|
||||
self._cleanup_empty_parents(resolved)
|
||||
except Exception as e:
|
||||
logger.warning("Could not delete low-quality file %s: %s",
|
||||
resolved, e)
|
||||
if entity_id:
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
conn.cursor().execute("DELETE FROM tracks WHERE id = ?", (entity_id,))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
return {'success': False, 'error': f'DB delete failed: {e}'}
|
||||
return {'success': True, 'action': 'deleted_file',
|
||||
'message': f'Deleted low-quality file: '
|
||||
f'{os.path.basename(file_path or "")}'}
|
||||
|
||||
"""Add the matched higher-quality version to the wishlist (with album
|
||||
context). Applying a Quality Upgrade finding is the user-approved step
|
||||
that the old auto-acting Quality Scanner did without review."""
|
||||
track_data = details.get('matched_track_data')
|
||||
if not track_data:
|
||||
return {'success': False, 'error': 'No matched track in finding'}
|
||||
|
|
@ -1212,118 +1177,6 @@ class RepairWorker:
|
|||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _fix_short_preview_track(self, entity_type, entity_id, file_path, details):
|
||||
"""Approve a preview-clip finding: delete the ~30s preview file, drop its DB row, and
|
||||
re-add the track to the wishlist (full payload) so the real version downloads. Mirrors
|
||||
the dead-file 'redownload' payload + the acoustid-mismatch file delete. (Tools #937-adj)
|
||||
"""
|
||||
if not entity_id:
|
||||
return {'success': False, 'error': 'No track ID associated with this finding'}
|
||||
conn = None
|
||||
try:
|
||||
conn = self.db._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT t.id, t.title, t.track_number, t.duration, t.bitrate,
|
||||
t.spotify_track_id, t.itunes_track_id, t.deezer_id, t.isrc,
|
||||
ar.name AS artist_name, ar.spotify_artist_id,
|
||||
al.title AS album_title, al.spotify_album_id,
|
||||
al.record_type, al.track_count, al.year, al.thumb_url AS album_thumb
|
||||
FROM tracks t
|
||||
LEFT JOIN artists ar ON ar.id = t.artist_id
|
||||
LEFT JOIN albums al ON al.id = t.album_id
|
||||
WHERE t.id = ?
|
||||
""", (entity_id,))
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
return {'success': False, 'error': 'Track not found in database'}
|
||||
|
||||
track_name = row['title'] or details.get('title', 'Unknown')
|
||||
artist_name = row['artist_name'] or details.get('artist', 'Unknown Artist')
|
||||
album_title = row['album_title'] or details.get('album', '')
|
||||
|
||||
wishlist_id = (row['spotify_track_id']
|
||||
or row['itunes_track_id']
|
||||
or row['deezer_id']
|
||||
or f"preview_redl_{entity_id}")
|
||||
|
||||
# Prefer the finding's stored art (the scan captures the metadata source's CDN image)
|
||||
# over the library album thumb, which is often empty for un-enriched HiFi previews.
|
||||
album_images = []
|
||||
album_thumb = details.get('album_thumb_url') or row['album_thumb']
|
||||
if album_thumb:
|
||||
album_images = [{'url': album_thumb}]
|
||||
|
||||
spotify_track_data = {
|
||||
'id': wishlist_id,
|
||||
'name': track_name,
|
||||
'artists': [{'name': artist_name}],
|
||||
'album': {
|
||||
'name': album_title or track_name,
|
||||
'id': row['spotify_album_id'] or '',
|
||||
'release_date': str(row['year']) if row['year'] else '',
|
||||
'images': album_images,
|
||||
'album_type': row['record_type'] or 'album',
|
||||
'total_tracks': row['track_count'] or 0,
|
||||
'artists': [{'name': artist_name}],
|
||||
},
|
||||
'duration_ms': int((details.get('expected_duration_s') or 0) * 1000) or (row['duration'] or 0),
|
||||
'track_number': row['track_number'] or 1,
|
||||
'disc_number': 1,
|
||||
'explicit': False,
|
||||
'external_urls': {},
|
||||
'popularity': 0,
|
||||
'preview_url': None,
|
||||
'uri': f"spotify:track:{row['spotify_track_id']}" if row['spotify_track_id'] else '',
|
||||
'is_local': False,
|
||||
}
|
||||
|
||||
source_info = {
|
||||
'original_path': file_path or details.get('original_path', ''),
|
||||
'album_title': album_title,
|
||||
'artist': artist_name,
|
||||
'reason': 'preview_clip_redownload',
|
||||
}
|
||||
|
||||
added = self.db.add_to_wishlist(
|
||||
spotify_track_data,
|
||||
failure_reason='Preview clip — re-downloading full track',
|
||||
source_type='redownload',
|
||||
source_info=source_info,
|
||||
)
|
||||
if not added:
|
||||
return {'success': False, 'error': 'Failed to add to wishlist (may already exist or be blocklisted)'}
|
||||
|
||||
# Delete the preview file (path resolved like the other delete tools).
|
||||
deleted_file = False
|
||||
target_path = file_path or details.get('original_path')
|
||||
if target_path:
|
||||
download_folder = self._config_manager.get('soulseek.download_path', '') if self._config_manager else None
|
||||
resolved = _resolve_file_path(target_path, self.transfer_folder,
|
||||
download_folder=download_folder,
|
||||
config_manager=self._config_manager)
|
||||
if resolved and os.path.exists(resolved):
|
||||
try:
|
||||
os.remove(resolved)
|
||||
deleted_file = True
|
||||
except Exception as e:
|
||||
logger.warning("Could not delete preview file %s: %s", resolved, e)
|
||||
|
||||
# Drop the DB row so the track shows as missing.
|
||||
cursor.execute("DELETE FROM tracks WHERE id = ?", (entity_id,))
|
||||
conn.commit()
|
||||
|
||||
return {'success': True, 'action': 'added_to_wishlist',
|
||||
'message': (f'Deleted preview clip and re-wishlisted "{track_name}" for full download'
|
||||
if deleted_file else
|
||||
f'Re-wishlisted "{track_name}" (preview file already gone)')}
|
||||
except Exception as e:
|
||||
logger.error("Preview-clip fix failed for track %s: %s", entity_id, e)
|
||||
return {'success': False, 'error': str(e)}
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def _fix_orphan_file(self, entity_type, entity_id, file_path, details):
|
||||
"""Handle an orphan file — move to staging or delete based on user choice.
|
||||
|
||||
|
|
@ -3373,14 +3226,6 @@ class RepairWorker:
|
|||
return {'success': False, 'error': f'Source file not found: {file_path}'}
|
||||
|
||||
out_path = os.path.splitext(resolved)[0] + out_ext
|
||||
# Safety invariant: ffmpeg runs with -y, so refuse to convert a file onto
|
||||
# itself (an .m4a ALAC source + AAC target shares the .m4a path) — that
|
||||
# would destroy the original lossless file (#941).
|
||||
from core.quality.lossless import lossy_output_would_overwrite_source
|
||||
if lossy_output_would_overwrite_source(resolved, out_path):
|
||||
return {'success': False,
|
||||
'error': f'{codec.upper()} output would overwrite the source file; '
|
||||
f'choose a different lossy codec'}
|
||||
if os.path.exists(out_path):
|
||||
return {'success': True, 'action': 'already_exists',
|
||||
'message': f'{quality_label} copy already exists'}
|
||||
|
|
@ -3539,8 +3384,7 @@ class RepairWorker:
|
|||
'album_tag_inconsistency',
|
||||
'incomplete_album', 'path_mismatch',
|
||||
'missing_lossy_copy', 'missing_replaygain', 'empty_folder',
|
||||
'missing_discography_track', 'acoustid_mismatch',
|
||||
'quality_upgrade', 'short_preview_track')
|
||||
'missing_discography_track', 'acoustid_mismatch')
|
||||
placeholders = ','.join(['?'] * len(fixable_types))
|
||||
where_parts = [f"finding_type IN ({placeholders})", "status = 'pending'"]
|
||||
params = list(fixable_types)
|
||||
|
|
|
|||
|
|
@ -66,39 +66,6 @@ def add_activity_item(icon, title, subtitle, time_ago="Now", show_toast=True):
|
|||
return activity_item
|
||||
|
||||
|
||||
def claim_for_post_processing(task_id: str) -> bool:
|
||||
"""Atomically claim a download task for post-processing.
|
||||
|
||||
The browser-poll status endpoint AND the background download monitor both
|
||||
watch the same slskd/streaming transfers and each tries to post-process a
|
||||
completed file. Without a single claim, BOTH run the verification pipeline
|
||||
on the same download — double imports, and a nasty race where one path
|
||||
quarantines + requeues the next-best candidate (clearing the source identity
|
||||
and resetting status to ``searching``) while the other, mid-flight, then
|
||||
reports a bogus "missing file or source information" failure that clobbers
|
||||
the in-flight retry.
|
||||
|
||||
The claim is the ``downloading``/``queued`` -> ``post_processing`` status
|
||||
transition, done under ``tasks_lock``. Exactly one caller wins:
|
||||
|
||||
- Returns ``True`` (and flips the status) for the caller that claimed it.
|
||||
- Returns ``False`` if the task is gone, already ``post_processing`` (owned
|
||||
by the other path), requeued (``searching``), or terminal — the caller
|
||||
must then NOT process the file.
|
||||
|
||||
Acquires ``tasks_lock`` itself, so callers must NOT already hold it.
|
||||
"""
|
||||
with tasks_lock:
|
||||
task = download_tasks.get(task_id)
|
||||
if not task:
|
||||
return False
|
||||
if task.get("status") in ("downloading", "queued"):
|
||||
task["status"] = "post_processing"
|
||||
task["status_change_time"] = time.time()
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
@caller_must_hold_tasks_lock
|
||||
def mark_task_completed(task_id: str, track_info: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""Mark a download task as completed.
|
||||
|
|
|
|||
|
|
@ -24,8 +24,6 @@ from core.download_plugins.album_bundle import (
|
|||
get_poll_timeout,
|
||||
)
|
||||
from core.download_plugins.base import DownloadSourcePlugin
|
||||
from core.quality.model import QualityTarget, filter_and_rank, v2_qualities_to_ranked_targets
|
||||
from core.quality.source_map import AUDIO_EXTENSIONS, format_from_extension
|
||||
from utils.async_helpers import run_async
|
||||
|
||||
logger = get_logger("soulseek_client")
|
||||
|
|
@ -411,7 +409,7 @@ class SoulseekClient(DownloadSourcePlugin):
|
|||
|
||||
|
||||
# Audio file extensions to filter for
|
||||
audio_extensions = AUDIO_EXTENSIONS
|
||||
audio_extensions = {'.mp3', '.flac', '.ogg', '.aac', '.wma', '.wav', '.m4a'}
|
||||
|
||||
for response_data in responses_data:
|
||||
username = response_data.get('username', '')
|
||||
|
|
@ -428,28 +426,26 @@ class SoulseekClient(DownloadSourcePlugin):
|
|||
if f'.{file_ext}' not in audio_extensions:
|
||||
continue
|
||||
|
||||
# Source-agnostic extension → format (shared with every other
|
||||
# extension-based source). Ranked targets do the rest.
|
||||
quality = format_from_extension(file_ext)
|
||||
# .m4a is the usual AAC container — bucket it as 'aac' (the
|
||||
# quality filter treats AAC as an opt-in tier; off by default).
|
||||
quality = 'aac' if file_ext == 'm4a' else (
|
||||
file_ext if file_ext in ['flac', 'mp3', 'ogg', 'aac', 'wma'] else 'unknown')
|
||||
|
||||
# Create TrackResult
|
||||
# Convert duration from seconds to milliseconds (slskd returns seconds, Spotify uses ms)
|
||||
raw_duration = file_data.get('length')
|
||||
duration_ms = raw_duration * 1000 if raw_duration else None
|
||||
|
||||
slskd_attrs = {a['type']: a['value'] for a in file_data.get('attributes', [])}
|
||||
track = TrackResult(
|
||||
username=username,
|
||||
filename=filename,
|
||||
size=size,
|
||||
bitrate=file_data.get('bitRate') or slskd_attrs.get(0),
|
||||
bitrate=file_data.get('bitRate'),
|
||||
duration=duration_ms,
|
||||
quality=quality,
|
||||
free_upload_slots=response_data.get('freeUploadSlots', 0),
|
||||
upload_speed=response_data.get('uploadSpeed', 0),
|
||||
queue_length=response_data.get('queueLength', 0),
|
||||
sample_rate=slskd_attrs.get(4),
|
||||
bit_depth=slskd_attrs.get(5),
|
||||
queue_length=response_data.get('queueLength', 0)
|
||||
)
|
||||
|
||||
all_tracks.append(track)
|
||||
|
|
@ -1140,7 +1136,7 @@ class SoulseekClient(DownloadSourcePlugin):
|
|||
Returns:
|
||||
List of TrackResult objects for audio files
|
||||
"""
|
||||
audio_extensions = AUDIO_EXTENSIONS
|
||||
audio_extensions = {'.mp3', '.flac', '.ogg', '.aac', '.wma', '.wav', '.m4a'}
|
||||
results = []
|
||||
if files:
|
||||
logger.debug(f"Browse raw file sample: {files[0]}")
|
||||
|
|
@ -1154,17 +1150,15 @@ class SoulseekClient(DownloadSourcePlugin):
|
|||
ext = Path(filename).suffix.lower()
|
||||
if ext not in audio_extensions:
|
||||
continue
|
||||
quality = format_from_extension(ext)
|
||||
_qext = ext.lstrip('.')
|
||||
quality = 'aac' if _qext == 'm4a' else (
|
||||
_qext if _qext in ['flac', 'mp3', 'ogg', 'aac', 'wma'] else 'unknown')
|
||||
raw_duration = file_data.get('length')
|
||||
duration_ms = raw_duration * 1000 if raw_duration else None
|
||||
slskd_attrs = {a['type']: a['value'] for a in file_data.get('attributes', [])}
|
||||
results.append(TrackResult(
|
||||
username=username, filename=filename, size=file_data.get('size', 0),
|
||||
bitrate=file_data.get('bitRate') or slskd_attrs.get(0),
|
||||
duration=duration_ms, quality=quality,
|
||||
free_upload_slots=free_slots, upload_speed=upload_speed, queue_length=queue_length,
|
||||
sample_rate=slskd_attrs.get(4),
|
||||
bit_depth=slskd_attrs.get(5),
|
||||
bitrate=file_data.get('bitRate'), duration=duration_ms, quality=quality,
|
||||
free_upload_slots=free_slots, upload_speed=upload_speed, queue_length=queue_length
|
||||
))
|
||||
return results
|
||||
|
||||
|
|
@ -2014,57 +2008,189 @@ class SoulseekClient(DownloadSourcePlugin):
|
|||
return kept
|
||||
|
||||
def filter_results_by_quality_preference(self, results: List[TrackResult]) -> List[TrackResult]:
|
||||
"""Filter and rank candidates using the global quality target list.
|
||||
"""
|
||||
Filter candidates based on user's quality profile with bitrate density constraints.
|
||||
Uses priority waterfall logic: tries highest priority quality first, falls back to lower priorities.
|
||||
Returns candidates matching quality profile constraints, sorted by confidence and effective bitrate.
|
||||
|
||||
Replaces the old bucket+heuristic approach with ``core.quality.model``
|
||||
so every download source shares the same ranking logic.
|
||||
|
||||
Issue #652: also drops candidates whose ``(username, filename)``
|
||||
matches a previously-quarantined download to break infinite retry loops.
|
||||
Issue #652: also drops candidates whose `(username, filename)`
|
||||
matches a previously-quarantined download. Without this pre-filter
|
||||
the auto-wishlist processor's ranking is deterministic — the same
|
||||
`(uploader, file)` keeps winning the quality picker, downloading,
|
||||
failing AcoustID, quarantining, and re-queueing in an infinite
|
||||
loop. Users wake up to hundreds of duplicate `.quarantined` files
|
||||
for the same source URL.
|
||||
"""
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
if not results:
|
||||
return []
|
||||
|
||||
# Issue #652: drop candidates on the quarantine record BEFORE ranking,
|
||||
# so a previously-quarantined source can't win the quality picker by
|
||||
# superior bitrate and re-trigger the same failed download in a loop.
|
||||
# Drop sources already quarantined — bypass the quality picker
|
||||
# entirely so the same bad upload doesn't get re-selected on the
|
||||
# next wishlist cycle. Filesystem read is bounded (~few hundred
|
||||
# sidecars in practical use × <1ms each).
|
||||
results = self._drop_quarantined_sources(results)
|
||||
if not results:
|
||||
return []
|
||||
|
||||
# Get quality profile from database
|
||||
db = MusicDatabase()
|
||||
profile = db.get_quality_profile()
|
||||
|
||||
# Build ranked target list — v3 profiles carry it directly;
|
||||
# v2 profiles are converted on the fly (no DB write needed here).
|
||||
raw_targets = profile.get('ranked_targets')
|
||||
if not raw_targets and 'qualities' in profile:
|
||||
raw_targets = v2_qualities_to_ranked_targets(profile['qualities'])
|
||||
logger.debug(f"Quality Filter: Using profile preset '{profile.get('preset', 'custom')}', filtering {len(results)} candidates")
|
||||
|
||||
targets = [QualityTarget.from_dict(t) for t in (raw_targets or [])]
|
||||
fallback_enabled = profile.get('fallback_enabled', True)
|
||||
# Categorize candidates by quality with bitrate density constraints
|
||||
quality_buckets = {
|
||||
'flac': [],
|
||||
'mp3_320': [],
|
||||
'mp3_256': [],
|
||||
'mp3_192': [],
|
||||
'aac': [],
|
||||
'other': []
|
||||
}
|
||||
|
||||
# Every format (AAC included) follows the SAME universal rule: a
|
||||
# candidate passes only if it matches a ranked target; if nothing
|
||||
# matches, the fallback toggle decides. No per-format special-casing.
|
||||
# Track all candidates that pass checks (for fallback)
|
||||
density_filtered_all = []
|
||||
|
||||
logger.debug(
|
||||
"Quality Filter: profile='%s', %d targets, %d candidates",
|
||||
profile.get('preset', 'custom'), len(targets), len(results),
|
||||
)
|
||||
for candidate in results:
|
||||
if not candidate.quality:
|
||||
quality_buckets['other'].append(candidate)
|
||||
continue
|
||||
|
||||
ranked = filter_and_rank(results, targets, fallback_enabled=fallback_enabled)
|
||||
track_format = candidate.quality.lower()
|
||||
track_bitrate = candidate.bitrate or 0
|
||||
|
||||
if ranked:
|
||||
best_label = ranked[0].audio_quality.label()
|
||||
logger.info("Quality Filter: returning %d candidate(s), best=%s", len(ranked), best_label)
|
||||
# Determine quality key
|
||||
if track_format == 'flac':
|
||||
quality_key = 'flac'
|
||||
elif track_format == 'mp3':
|
||||
if track_bitrate >= 320:
|
||||
quality_key = 'mp3_320'
|
||||
elif track_bitrate >= 256:
|
||||
quality_key = 'mp3_256'
|
||||
elif track_bitrate >= 192:
|
||||
quality_key = 'mp3_192'
|
||||
else:
|
||||
quality_buckets['other'].append(candidate)
|
||||
continue
|
||||
elif track_format in ('aac', 'm4a'):
|
||||
# Opt-in AAC tier. ADDITIVE: when AAC isn't enabled in the
|
||||
# profile (the default, and every profile that predates this),
|
||||
# route exactly where AAC went before — the 'other' bucket — so
|
||||
# behaviour is byte-identical. Only a user who turns AAC on lets
|
||||
# it become a first-class, selectable tier.
|
||||
aac_cfg = profile['qualities'].get('aac')
|
||||
if not (aac_cfg and aac_cfg.get('enabled')):
|
||||
quality_buckets['other'].append(candidate)
|
||||
continue
|
||||
quality_key = 'aac'
|
||||
else:
|
||||
quality_buckets['other'].append(candidate)
|
||||
continue
|
||||
|
||||
quality_config = profile['qualities'].get(quality_key, {})
|
||||
min_kbps = quality_config.get('min_kbps', 0)
|
||||
max_kbps = quality_config.get('max_kbps', 99999)
|
||||
|
||||
effective_kbps = self._calculate_effective_kbps(candidate.size, candidate.duration)
|
||||
|
||||
if effective_kbps is not None:
|
||||
# Primary: bitrate density check
|
||||
if min_kbps <= effective_kbps <= max_kbps:
|
||||
if quality_config.get('enabled', False):
|
||||
quality_buckets[quality_key].append(candidate)
|
||||
density_filtered_all.append(candidate)
|
||||
else:
|
||||
logger.debug(f"Quality Filter: {quality_key} rejected - {effective_kbps:.0f} kbps outside {min_kbps}-{max_kbps} kbps range")
|
||||
else:
|
||||
# Fallback: duration unavailable, use generous raw-size sanity check
|
||||
file_size_mb = candidate.size / (1024 * 1024)
|
||||
size_min, size_max = self._FALLBACK_SIZE_LIMITS.get(quality_key, (0, 500))
|
||||
if size_min <= file_size_mb <= size_max:
|
||||
if quality_config.get('enabled', False):
|
||||
quality_buckets[quality_key].append(candidate)
|
||||
density_filtered_all.append(candidate)
|
||||
logger.debug(f"Quality Filter: {quality_key} accepted via size fallback ({file_size_mb:.1f} MB, no duration available)")
|
||||
else:
|
||||
logger.debug(f"Quality Filter: {quality_key} rejected via size fallback - {file_size_mb:.1f} MB outside {size_min}-{size_max} MB safety limits")
|
||||
|
||||
# Sort each bucket: effective bitrate first (prefer highest audio quality),
|
||||
# then peer quality score as tiebreaker (prefer fastest peer at same quality)
|
||||
for bucket in quality_buckets.values():
|
||||
bucket.sort(key=lambda x: (self._calculate_effective_kbps(x.size, x.duration) or 0, x.quality_score), reverse=True)
|
||||
|
||||
# Enforce FLAC bit depth preference from quality profile
|
||||
flac_config = profile['qualities'].get('flac', {})
|
||||
bit_depth_pref = flac_config.get('bit_depth', 'any')
|
||||
bit_depth_fallback = flac_config.get('bit_depth_fallback', True)
|
||||
|
||||
if bit_depth_pref != 'any' and quality_buckets['flac']:
|
||||
# 16-bit/44.1kHz FLAC theoretical max is 1411 kbps; 24-bit starts at ~2116 kbps
|
||||
# Real-world compressed: 16-bit = 800-1400 kbps, 24-bit = 1500+ kbps
|
||||
DEPTH_THRESHOLD = 1450
|
||||
|
||||
if bit_depth_pref == '24':
|
||||
hi_res = [c for c in quality_buckets['flac']
|
||||
if (self._calculate_effective_kbps(c.size, c.duration) or 0) > DEPTH_THRESHOLD]
|
||||
if hi_res:
|
||||
logger.info(f"Quality Filter: Bit depth 24-bit preference — {len(hi_res)}/{len(quality_buckets['flac'])} FLAC candidates are hi-res")
|
||||
quality_buckets['flac'] = hi_res
|
||||
elif not bit_depth_fallback:
|
||||
logger.info("Quality Filter: No 24-bit FLAC found and fallback disabled — rejecting all FLAC")
|
||||
quality_buckets['flac'] = []
|
||||
else:
|
||||
logger.info("Quality Filter: No 24-bit FLAC found — falling back to 16-bit")
|
||||
|
||||
elif bit_depth_pref == '16':
|
||||
lo_res = [c for c in quality_buckets['flac']
|
||||
if (self._calculate_effective_kbps(c.size, c.duration) or 0) <= DEPTH_THRESHOLD]
|
||||
if lo_res:
|
||||
logger.info(f"Quality Filter: Bit depth 16-bit preference — {len(lo_res)}/{len(quality_buckets['flac'])} FLAC candidates are standard")
|
||||
quality_buckets['flac'] = lo_res
|
||||
elif not bit_depth_fallback:
|
||||
logger.info("Quality Filter: No 16-bit FLAC found and fallback disabled — rejecting all FLAC")
|
||||
quality_buckets['flac'] = []
|
||||
else:
|
||||
logger.info("Quality Filter: No 16-bit FLAC found — falling back to 24-bit")
|
||||
|
||||
# Debug logging
|
||||
for quality, bucket in quality_buckets.items():
|
||||
if bucket:
|
||||
logger.debug(f"Quality Filter: Found {len(bucket)} '{quality}' candidates (after bitrate + bit depth filtering)")
|
||||
|
||||
# Waterfall priority logic: try qualities in priority order
|
||||
# Build priority list from enabled qualities
|
||||
quality_priorities = []
|
||||
for quality_name, quality_config in profile['qualities'].items():
|
||||
if quality_config.get('enabled', False):
|
||||
priority = quality_config.get('priority', 999)
|
||||
quality_priorities.append((priority, quality_name))
|
||||
|
||||
# Sort by priority (lower number = higher priority)
|
||||
quality_priorities.sort()
|
||||
|
||||
# Try each quality in priority order
|
||||
for priority, quality_name in quality_priorities:
|
||||
candidates_for_quality = quality_buckets.get(quality_name, [])
|
||||
if candidates_for_quality:
|
||||
logger.info(f"Quality Filter: Returning {len(candidates_for_quality)} '{quality_name}' candidates (priority {priority})")
|
||||
return candidates_for_quality
|
||||
|
||||
# If no enabled qualities matched, check if fallback is enabled
|
||||
if profile.get('fallback_enabled', True):
|
||||
logger.warning("Quality Filter: No enabled qualities matched, falling back to density-filtered candidates")
|
||||
if density_filtered_all:
|
||||
density_filtered_all.sort(key=lambda x: (x.quality_score, self._calculate_effective_kbps(x.size, x.duration) or 0), reverse=True)
|
||||
logger.info(f"Quality Filter: Returning {len(density_filtered_all)} fallback candidates (bitrate-filtered, any quality)")
|
||||
return density_filtered_all
|
||||
else:
|
||||
logger.warning("Quality Filter: All candidates failed bitrate checks, returning empty (respecting constraints)")
|
||||
return []
|
||||
else:
|
||||
logger.warning("Quality Filter: no candidates passed quality constraints")
|
||||
|
||||
return ranked
|
||||
|
||||
logger.warning("Quality Filter: No enabled qualities matched and fallback is disabled, returning empty")
|
||||
return []
|
||||
|
||||
async def get_session_info(self) -> Optional[Dict[str, Any]]:
|
||||
"""Get slskd session information including version"""
|
||||
if not self.base_url:
|
||||
|
|
|
|||
|
|
@ -128,14 +128,7 @@ class SoundcloudClient(DownloadSourcePlugin):
|
|||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
|
||||
self.download_path = Path(download_path)
|
||||
# Don't crash construction if the path isn't creatable yet (e.g. an
|
||||
# unmounted/misconfigured volume) — the registry would null the whole
|
||||
# client and the source vanishes. Warn and continue, same as
|
||||
# SoulseekClient; the dir is (re)created lazily at download time.
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify SoundCloud download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"SoundCloud client using download path: {self.download_path}")
|
||||
|
||||
|
|
|
|||
|
|
@ -13,61 +13,6 @@ from core.metadata.cache import get_metadata_cache
|
|||
|
||||
logger = get_logger("spotify_client")
|
||||
|
||||
|
||||
# Single source of truth for the Spotify OAuth scope. Used by EVERY SpotifyOAuth
|
||||
# construction (the client, the per-profile registry, and all web_server callbacks) so
|
||||
# the authorize URL and token exchange can never request different scopes — a mismatch
|
||||
# silently re-prompts or denies.
|
||||
#
|
||||
# IMPORTANT — do NOT add scopes here lightly. Spotipy's validate_token treats a cached
|
||||
# token as invalid the moment the requested scope is no longer a subset of the token's
|
||||
# granted scope, so GROWING this string invalidates EVERY existing user's token and forces
|
||||
# a re-auth on upgrade. `playlist-modify-*` (for exporting a playlist back to Spotify, #945)
|
||||
# was pulled back out for exactly that reason — it broke all Spotify users on upgrade. The
|
||||
# Spotify export must request write access on-demand (incremental auth) instead.
|
||||
SPOTIFY_OAUTH_SCOPE = (
|
||||
"user-library-read user-read-private playlist-read-private "
|
||||
"playlist-read-collaborative user-read-email user-follow-read"
|
||||
)
|
||||
|
||||
# The export scope = the normal login scope PLUS playlist write. Requested ONLY by the
|
||||
# on-demand export-auth route (/auth/spotify/export) when a user chooses to export a playlist
|
||||
# to Spotify — NEVER by the normal login. That's the whole safety property: the global scope
|
||||
# above is unchanged, so no existing token is invalidated. The token Spotify returns from the
|
||||
# export flow is a SUPERSET of the read scope, so it still passes the normal auth check
|
||||
# (read ⊆ read+write) — one account, one token, just with write added for the opt-in user.
|
||||
SPOTIFY_EXPORT_SCOPE = (
|
||||
SPOTIFY_OAUTH_SCOPE + " playlist-modify-public playlist-modify-private"
|
||||
)
|
||||
|
||||
|
||||
def normalize_spotify_oauth_config(config: Optional[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Normalize Spotify OAuth config before building an auth manager.
|
||||
|
||||
Spotify rejects values that include surrounding whitespace or quotes, and the
|
||||
settings UI can paste values carrying such formatting, so we trim those — they
|
||||
can never be part of a real credential.
|
||||
|
||||
We deliberately do NOT strip a trailing slash from ``redirect_uri``: Spotify
|
||||
matches the redirect URI EXACTLY against the app's dashboard registration, and
|
||||
a trailing slash is a legitimate part of a URI. Stripping it would silently
|
||||
break anyone who registered ``…/callback/`` (we'd send ``…/callback`` →
|
||||
"INVALID_CLIENT: Invalid redirect URI"). So the value is preserved verbatim
|
||||
apart from the unambiguous whitespace/quote garbage (#942 follow-up).
|
||||
"""
|
||||
if not isinstance(config, dict):
|
||||
return {}
|
||||
|
||||
normalized = {}
|
||||
for key in ("client_id", "client_secret", "redirect_uri"):
|
||||
value = config.get(key, "")
|
||||
if isinstance(value, str):
|
||||
normalized[key] = value.strip().strip('"').strip("'")
|
||||
else:
|
||||
normalized[key] = value
|
||||
return normalized
|
||||
|
||||
|
||||
def _upgrade_spotify_image_url(url: str) -> str:
|
||||
"""Upgrade a Spotify CDN image URL to the highest available resolution.
|
||||
|
||||
|
|
@ -752,7 +697,7 @@ class SpotifyClient:
|
|||
self._setup_client()
|
||||
|
||||
def _setup_client(self):
|
||||
config = normalize_spotify_oauth_config(config_manager.get_spotify_config())
|
||||
config = config_manager.get_spotify_config()
|
||||
|
||||
if not config.get('client_id') or not config.get('client_secret'):
|
||||
logger.warning("Spotify credentials not configured")
|
||||
|
|
@ -771,7 +716,7 @@ class SpotifyClient:
|
|||
client_id=config['client_id'],
|
||||
client_secret=config['client_secret'],
|
||||
redirect_uri=config.get('redirect_uri', "http://127.0.0.1:8888/callback"),
|
||||
scope=SPOTIFY_OAUTH_SCOPE,
|
||||
scope="user-library-read user-read-private playlist-read-private playlist-read-collaborative user-read-email user-follow-read",
|
||||
cache_handler=DatabaseTokenCache(config_manager)
|
||||
)
|
||||
|
||||
|
|
@ -806,16 +751,6 @@ class SpotifyClient:
|
|||
self._auth_cached_result = None
|
||||
self._auth_cache_time = 0
|
||||
|
||||
def _has_cached_oauth_token(self) -> bool:
|
||||
"""Return True when a persisted OAuth token exists (no network I/O)."""
|
||||
if self.sp is None:
|
||||
return False
|
||||
try:
|
||||
cache_handler = getattr(self.sp.auth_manager, 'cache_handler', None)
|
||||
return bool(cache_handler and cache_handler.get_cached_token() is not None)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def is_spotify_authenticated(self) -> bool:
|
||||
"""Check if Spotify client is specifically authenticated (not just iTunes fallback).
|
||||
Results are cached for 60 seconds to avoid excessive API calls.
|
||||
|
|
@ -891,26 +826,6 @@ class SpotifyClient:
|
|||
logger.debug("publish_spotify_status cache hit: %s", e)
|
||||
return self._auth_cached_result
|
||||
|
||||
from core.boot_phase import is_boot_phase
|
||||
if is_boot_phase():
|
||||
result = self._has_cached_oauth_token()
|
||||
with self._auth_cache_lock:
|
||||
self._auth_cached_result = result
|
||||
self._auth_cache_time = time.time()
|
||||
try:
|
||||
from core.metadata.status import publish_spotify_status
|
||||
|
||||
publish_spotify_status(
|
||||
connected=result,
|
||||
authenticated=result,
|
||||
rate_limited=False,
|
||||
rate_limit=None,
|
||||
post_ban_cooldown=None,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("publish_spotify_status boot-phase: %s", e)
|
||||
return result
|
||||
|
||||
# Cache miss — make API call outside the lock.
|
||||
# Safety: if there's no cached token, return False immediately.
|
||||
# Without this guard, spotipy's auth_manager will try to start an interactive
|
||||
|
|
@ -942,8 +857,7 @@ class SpotifyClient:
|
|||
# Use a dedicated probe client (retries=0) so a 429 here propagates
|
||||
# immediately and we can detect long Retry-After bans.
|
||||
try:
|
||||
probe = spotipy.Spotify(
|
||||
auth_manager=self.sp.auth_manager, retries=0, requests_timeout=15)
|
||||
probe = spotipy.Spotify(auth_manager=self.sp.auth_manager, retries=0)
|
||||
probe.current_user()
|
||||
result = True
|
||||
except Exception as e:
|
||||
|
|
@ -1194,69 +1108,6 @@ class SpotifyClient:
|
|||
return playlists
|
||||
return []
|
||||
|
||||
def has_write_scope(self) -> bool:
|
||||
"""True when the cached Spotify token carries playlist-modify (the export write scope).
|
||||
The export endpoint uses this to decide whether to run, or to first send the user
|
||||
through the on-demand export-auth flow. Fail-safe: any error → False (not authorized)."""
|
||||
if self.sp is None:
|
||||
return False
|
||||
try:
|
||||
cache_handler = getattr(self.sp.auth_manager, "cache_handler", None)
|
||||
token = cache_handler.get_cached_token() if cache_handler else None
|
||||
return "playlist-modify" in ((token or {}).get("scope") or "")
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
|
||||
public=False, description=""):
|
||||
"""Create a Spotify playlist owned by the authed user (or replace an existing
|
||||
one's tracks in place), for exporting a mirrored playlist back to Spotify (#945).
|
||||
|
||||
``track_ids`` are Spotify track IDs (the stored ``spotify_track_id`` per library
|
||||
track). ``existing_id`` set → replace that playlist's contents (idempotent
|
||||
re-export); unset → create a new playlist. Requires the ``playlist-modify-*``
|
||||
scope — a token issued before that scope was added gets a clear "reconnect"
|
||||
error rather than a raw 403. Returns
|
||||
``{success, playlist_id, url, added, error}``.
|
||||
"""
|
||||
if not self.is_spotify_authenticated():
|
||||
return {"success": False, "error": "Spotify is not connected"}
|
||||
uris = [f"spotify:track:{t}" for t in (track_ids or []) if t]
|
||||
if not uris:
|
||||
return {"success": False, "error": "No matching Spotify tracks to export"}
|
||||
try:
|
||||
playlist_id = existing_id
|
||||
if playlist_id:
|
||||
# Replace contents (re-export). replace_items caps at 100 — seed with the
|
||||
# first 100, then append the rest in 100-track chunks.
|
||||
self.sp.playlist_replace_items(playlist_id, uris[:100])
|
||||
for i in range(100, len(uris), 100):
|
||||
self.sp.playlist_add_items(playlist_id, uris[i:i + 100])
|
||||
else:
|
||||
user_id = (self.sp.current_user() or {}).get("id")
|
||||
created = self.sp.user_playlist_create(
|
||||
user_id, name, public=public, description=description,
|
||||
)
|
||||
playlist_id = (created or {}).get("id")
|
||||
if not playlist_id:
|
||||
return {"success": False, "error": "Spotify did not return a playlist id"}
|
||||
for i in range(0, len(uris), 100):
|
||||
self.sp.playlist_add_items(playlist_id, uris[i:i + 100])
|
||||
return {
|
||||
"success": True,
|
||||
"playlist_id": playlist_id,
|
||||
"url": f"https://open.spotify.com/playlist/{playlist_id}",
|
||||
"added": len(uris),
|
||||
}
|
||||
except Exception as e:
|
||||
msg = str(e)
|
||||
if "403" in msg or "scope" in msg.lower() or "insufficient" in msg.lower():
|
||||
return {"success": False,
|
||||
"error": "Reconnect Spotify to grant playlist write access "
|
||||
"(Settings → reconnect Spotify)."}
|
||||
_detect_and_set_rate_limit(e, "create_or_update_playlist")
|
||||
return {"success": False, "error": msg}
|
||||
|
||||
@rate_limited
|
||||
def get_saved_tracks_count(self) -> int:
|
||||
"""Get the total count of user's saved/liked songs without fetching all tracks"""
|
||||
|
|
|
|||
|
|
@ -520,18 +520,8 @@ class TidalClient:
|
|||
|
||||
return True
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
def is_authenticated(self):
|
||||
"""Check if client is authenticated, refreshing expired tokens if possible"""
|
||||
from core.boot_phase import is_boot_phase
|
||||
|
||||
if is_boot_phase():
|
||||
if self.access_token and time.time() < self.token_expires_at:
|
||||
return True
|
||||
return bool(self.access_token and self.refresh_token)
|
||||
|
||||
# Token still valid — no refresh needed. (Restored: #949 moved this short-circuit
|
||||
# into the boot-phase branch only, so every post-boot call fell through to the
|
||||
# refresh below — a constant silent-refresh loop on a perfectly valid token.)
|
||||
if self.access_token and time.time() < self.token_expires_at:
|
||||
return True
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,6 @@ from config.settings import config_manager
|
|||
|
||||
# Import Soulseek data structures for drop-in replacement compatibility
|
||||
from core.download_plugins.types import TrackResult, AlbumResult, DownloadStatus
|
||||
from core.quality.source_map import quality_from_tidal_tier, quality_tier_for_source
|
||||
|
||||
logger = get_logger("tidal_download_client")
|
||||
|
||||
|
|
@ -120,10 +119,7 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
download_path = config_manager.get('soulseek.download_path', './downloads')
|
||||
|
||||
self.download_path = Path(download_path)
|
||||
try:
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f"Could not verify download path {self.download_path}: {e}")
|
||||
self.download_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"Tidal download client using download path: {self.download_path}")
|
||||
|
||||
|
|
@ -134,7 +130,6 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
|
||||
self._device_auth_future = None
|
||||
self._device_auth_link = None
|
||||
self._boot_session_tokens: Optional[dict] = None
|
||||
|
||||
# Engine reference is populated by set_engine() at registration
|
||||
# time. Until then dispatch returns None — orchestrator wires
|
||||
|
|
@ -164,14 +159,6 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
expiry_time = saved.get('expiry_time', 0)
|
||||
|
||||
if token_type and access_token:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if is_boot_phase():
|
||||
self._boot_session_tokens = saved
|
||||
logger.info(
|
||||
"Loaded Tidal download session from config (verification deferred until after boot)"
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
expiry_dt = datetime.fromtimestamp(expiry_time, tz=timezone.utc) if expiry_time else None
|
||||
|
||||
|
|
@ -190,39 +177,6 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
except Exception as e:
|
||||
logger.warning(f"Could not restore Tidal session: {e}")
|
||||
|
||||
def _complete_deferred_session(self) -> bool:
|
||||
"""Finish restoring a session that was deferred during boot."""
|
||||
pending = getattr(self, '_boot_session_tokens', None)
|
||||
if not pending or tidalapi is None:
|
||||
self._boot_session_tokens = None
|
||||
return False
|
||||
|
||||
if not self.session:
|
||||
self.session = tidalapi.Session()
|
||||
|
||||
token_type = pending.get('token_type', '')
|
||||
access_token = pending.get('access_token', '')
|
||||
refresh_token = pending.get('refresh_token', '')
|
||||
expiry_time = pending.get('expiry_time', 0)
|
||||
self._boot_session_tokens = None
|
||||
|
||||
try:
|
||||
expiry_dt = datetime.fromtimestamp(expiry_time, tz=timezone.utc) if expiry_time else None
|
||||
restored = self.session.load_oauth_session(
|
||||
token_type=token_type,
|
||||
access_token=access_token,
|
||||
refresh_token=refresh_token,
|
||||
expiry_time=expiry_dt,
|
||||
)
|
||||
if restored and self.session.check_login():
|
||||
logger.info("Restored Tidal download session from saved tokens")
|
||||
self._save_session()
|
||||
return True
|
||||
logger.warning("Saved Tidal session tokens are invalid/expired")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not restore Tidal session: {e}")
|
||||
return False
|
||||
|
||||
def _save_session(self):
|
||||
if not self.session:
|
||||
return
|
||||
|
|
@ -234,14 +188,6 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
})
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
from core.boot_phase import is_boot_phase
|
||||
if is_boot_phase():
|
||||
pending = getattr(self, '_boot_session_tokens', None)
|
||||
return bool(pending and pending.get('access_token'))
|
||||
|
||||
if getattr(self, '_boot_session_tokens', None):
|
||||
return self._complete_deferred_session()
|
||||
|
||||
if not self.session:
|
||||
return False
|
||||
try:
|
||||
|
|
@ -509,17 +455,13 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
if successful_query and successful_query != query:
|
||||
logger.info(f"Tidal fallback query succeeded: '{successful_query}' (original: '{query}')")
|
||||
|
||||
quality_key = quality_tier_for_source('tidal', default='lossless')
|
||||
quality_key = config_manager.get('tidal_download.quality', 'lossless')
|
||||
quality_info = QUALITY_MAP.get(quality_key, QUALITY_MAP['lossless'])
|
||||
# Stamp the configured tier (what will actually be downloaded) so
|
||||
# the global ranker sees real sample_rate/bit_depth.
|
||||
tier_quality = quality_from_tidal_tier(quality_key)
|
||||
|
||||
track_results = []
|
||||
for track in tidal_tracks:
|
||||
try:
|
||||
track_result = self._tidal_to_track_result(track, quality_info)
|
||||
track_result.set_quality(tier_quality)
|
||||
track_results.append(track_result)
|
||||
except Exception as e:
|
||||
logger.debug(f"Skipping track conversion error: {e}")
|
||||
|
|
@ -832,7 +774,7 @@ class TidalDownloadClient(DownloadSourcePlugin):
|
|||
logger.error("Tidal session not authenticated")
|
||||
return None
|
||||
|
||||
quality_key = quality_tier_for_source('tidal', default='lossless')
|
||||
quality_key = config_manager.get('tidal_download.quality', 'lossless')
|
||||
chain = ['hires', 'lossless', 'high', 'low']
|
||||
start = chain.index(quality_key) if quality_key in chain else 1
|
||||
allow_fallback = config_manager.get('tidal_download.allow_fallback', True)
|
||||
|
|
|
|||
|
|
@ -1,40 +0,0 @@
|
|||
"""Pure decisions for UI-appearance defaults.
|
||||
|
||||
Kept here (importable, no Flask/config coupling) so the rules are unit-testable in
|
||||
isolation; web_server does only the request/config plumbing around them.
|
||||
|
||||
Worker orbs are a blurred 60fps canvas — the main remaining Firefox lag source after
|
||||
the #935 sweep. So for a FIRST-TIME user (no saved preference) we default them OFF on
|
||||
Firefox and ON everywhere else: a smooth first impression where it's needed, full
|
||||
polish where the browser handles it. An explicit saved choice ALWAYS wins — this only
|
||||
picks the default when the user hasn't chosen.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def is_firefox_user_agent(user_agent: Optional[str]) -> bool:
|
||||
"""True when a User-Agent string is Firefox.
|
||||
|
||||
Used ONLY to pick a performance-friendly default — never to gate functionality —
|
||||
so a spoofed or unusual UA simply gets the default and the user can toggle. Chrome,
|
||||
Edge, Safari, Opera, Brave do not carry 'firefox' in their UA; Firefox does
|
||||
('… Gecko/… Firefox/<ver>')."""
|
||||
return 'firefox' in str(user_agent or '').lower()
|
||||
|
||||
|
||||
def resolve_worker_orbs_default(explicit: object, is_firefox: bool) -> bool:
|
||||
"""Whether worker orbs should be on.
|
||||
|
||||
``explicit`` is the saved config value: ``True``/``False`` when the user has chosen,
|
||||
``None`` when unset. An explicit choice always wins; when unset, default OFF on
|
||||
Firefox (perf) and ON elsewhere.
|
||||
"""
|
||||
if explicit is None:
|
||||
return not is_firefox
|
||||
return explicit is not False
|
||||
|
||||
|
||||
__all__ = ['is_firefox_user_agent', 'resolve_worker_orbs_default']
|
||||
|
|
@ -185,11 +185,7 @@ def process_watchlist_scan_automatically(automation_id=None, profile_id=None, de
|
|||
'results': [],
|
||||
'summary': {},
|
||||
'error': None,
|
||||
'cancel_requested': False,
|
||||
# #933: stamp these so this scan lands in the History modal too —
|
||||
# the scanner fills scan_track_events; persist_scan_run reads both.
|
||||
'scan_run_id': datetime.now().strftime('%Y%m%d-%H%M%S'),
|
||||
'scan_track_events': [],
|
||||
'cancel_requested': False
|
||||
}
|
||||
|
||||
scan_results = []
|
||||
|
|
@ -298,17 +294,6 @@ def process_watchlist_scan_automatically(automation_id=None, profile_id=None, de
|
|||
total_added_to_wishlist = deps.watchlist_scan_state.get('summary', {}).get('tracks_added_to_wishlist', 0)
|
||||
logger.warning("Automatic watchlist scan cancelled — skipping post-scan steps")
|
||||
|
||||
# #933: record this run in the History ledger — same helper the manual
|
||||
# scan uses, so scheduled scans show up alongside manual ones.
|
||||
try:
|
||||
from core.watchlist.scan_history import persist_scan_run
|
||||
persist_scan_run(
|
||||
database, deps.watchlist_scan_state,
|
||||
profile_id=profile_id, was_cancelled=was_cancelled,
|
||||
)
|
||||
except Exception as _hist_err:
|
||||
logger.error(f"Failed to persist watchlist scan run: {_hist_err}")
|
||||
|
||||
# Post-scan steps — skip if cancelled
|
||||
if not was_cancelled:
|
||||
# Populate discovery pool from similar artists (per-profile)
|
||||
|
|
|
|||
|
|
@ -1,51 +0,0 @@
|
|||
"""Persist a finished watchlist scan to the History ledger (#831 / #933).
|
||||
|
||||
Both the manual scan (``web_server.start_watchlist_scan``) and the automatic
|
||||
scan (``core.watchlist.auto_scan.process_watchlist_scan_automatically``) finish
|
||||
with the same ``watchlist_scan_state`` shape, but only the manual path used to
|
||||
record a history row — so scheduled/nightly scans never showed up in the
|
||||
History modal (#933). This single helper is the shared seam: both paths call it,
|
||||
so they can't drift apart again.
|
||||
|
||||
Pure except for the one ``database.save_watchlist_scan_run`` call — the field
|
||||
extraction is unit-testable with a fake database.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
def _iso(value: Any) -> Optional[str]:
|
||||
"""ISO-format a datetime; pass through an already-stringified timestamp."""
|
||||
if value is None:
|
||||
return None
|
||||
return value.isoformat() if hasattr(value, 'isoformat') else str(value)
|
||||
|
||||
|
||||
def persist_scan_run(database: Any, state: Dict[str, Any], *,
|
||||
profile_id: Any, was_cancelled: bool) -> bool:
|
||||
"""Record one watchlist scan run + its track ledger from ``state``.
|
||||
|
||||
Reads the counts/timestamps/ledger off the live ``watchlist_scan_state`` the
|
||||
scanner just finished writing, and writes a single history row. ``run_id``
|
||||
comes from ``state['scan_run_id']`` (both paths stamp it); a timestamp
|
||||
fallback keeps it from ever colliding if that's somehow missing. Returns the
|
||||
DB call's truthiness; callers wrap in their own try/except so a history-write
|
||||
failure never breaks the scan.
|
||||
"""
|
||||
summary = state.get('summary') or {}
|
||||
run_id = state.get('scan_run_id') or datetime.now().strftime('%Y%m%d-%H%M%S')
|
||||
return database.save_watchlist_scan_run(
|
||||
run_id=run_id,
|
||||
profile_id=profile_id if profile_id else 1,
|
||||
status='cancelled' if was_cancelled else 'completed',
|
||||
started_at=_iso(state.get('started_at')),
|
||||
completed_at=_iso(state.get('completed_at')) or datetime.now().isoformat(),
|
||||
total_artists=summary.get('total_artists', state.get('total_artists', 0)),
|
||||
artists_scanned=summary.get('successful_scans', 0),
|
||||
tracks_found=state.get('tracks_found_this_scan', 0),
|
||||
tracks_added=state.get('tracks_added_this_scan', 0),
|
||||
track_events=state.get('scan_track_events') or [],
|
||||
)
|
||||
|
|
@ -328,22 +328,6 @@ _ALBUM_QUALIFIER_RE = re.compile(
|
|||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# A trailing "- ..." clause is stripped ONLY when EVERY token in it is an edition/format
|
||||
# qualifier (+ connectors / a year-ordinal). So "- Single", "- Acoustic Version", "- 2011
|
||||
# Remaster" collapse to the base name, but a real distinguishing subtitle ("- Nos vies en
|
||||
# Lumière", "- Live in Berlin") is kept — the bug was a blanket "- anything$" strip that
|
||||
# erased subtitles and fused different editions (Sokhi: Expedition 33 OST vs Bonus Edition).
|
||||
_DASH_QUALIFIER_WORD = (
|
||||
r'live|acoustic|electric|instrumental|unplugged|mono|stereo|demos?|reissue|'
|
||||
r'remix(?:es)?|edit(?:ed)?|radio|single|ep|lp|version|mix(?:es)?|sessions?|bootleg|'
|
||||
r'covers?|original|redux|deluxe|expanded|remaster(?:ed)?|anniversary|special|'
|
||||
r'edition|bonus|extended|explicit|clean|soundtrack|ost|score'
|
||||
)
|
||||
_TRAILING_DASH_QUALIFIER_RE = re.compile(
|
||||
r'\s*-\s*(?:(?:' + _DASH_QUALIFIER_WORD + r'|the|a|and|&|\+|\d+(?:st|nd|rd|th)?)\b[\s\-]*)+$',
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_album_for_match(name: str) -> str:
|
||||
"""Return a canonical form of an album name suitable for fuzzy comparison.
|
||||
|
|
@ -363,9 +347,8 @@ def _normalize_album_for_match(name: str) -> str:
|
|||
# they're almost always edition or commentary noise, not part of the
|
||||
# album's identifying name.
|
||||
cleaned = re.sub(r'\s*[\(\[][^\)\]]*[\)\]]\s*', ' ', cleaned)
|
||||
# Trailing dash-clause, but ONLY when it's entirely edition/format qualifiers — a real
|
||||
# subtitle is preserved (see _TRAILING_DASH_QUALIFIER_RE).
|
||||
cleaned = _TRAILING_DASH_QUALIFIER_RE.sub(' ', cleaned)
|
||||
# Trailing dash-clauses ("Album - Remastered", "Album - Live")
|
||||
cleaned = re.sub(r'\s*-\s*[^-]+$', '', cleaned)
|
||||
cleaned = re.sub(r'[^a-z0-9 ]+', ' ', cleaned.lower())
|
||||
cleaned = re.sub(r'\s+', ' ', cleaned).strip()
|
||||
return cleaned
|
||||
|
|
@ -399,7 +382,7 @@ def _extract_volume_marker(normalized_name: str):
|
|||
return last.group(1) or last.group(2)
|
||||
|
||||
|
||||
def _albums_likely_match(spotify_album: str, lib_album: str, threshold: float = 0.85) -> bool:
|
||||
def _albums_likely_match(spotify_album: str, lib_album: str, threshold: float = 0.6) -> bool:
|
||||
"""Return True when two album names plausibly identify the same release.
|
||||
|
||||
Designed to swallow naming drift between metadata sources and the
|
||||
|
|
@ -423,11 +406,11 @@ def _albums_likely_match(spotify_album: str, lib_album: str, threshold: float =
|
|||
return False
|
||||
if norm_a == norm_b:
|
||||
return True
|
||||
# No loose substring shortcut: after qualifier-stripping, a short name being a
|
||||
# prefix of a longer one is usually a DIFFERENT edition carrying a real subtitle
|
||||
# ("clair obscur expedition 33" ⊂ "clair obscur expedition 33 nos vies en lumiere"),
|
||||
# not naming drift. Genuine drift collapses to an EXACT match above; everything else
|
||||
# must clear a high overall-similarity bar.
|
||||
# After normalization the shorter name often becomes a prefix /
|
||||
# substring of the longer one ("napoleon dynamite" ⊂ "napoleon
|
||||
# dynamite music from the motion picture" before stripping).
|
||||
if norm_a in norm_b or norm_b in norm_a:
|
||||
return True
|
||||
return SequenceMatcher(None, norm_a, norm_b).ratio() >= threshold
|
||||
|
||||
|
||||
|
|
@ -3650,13 +3633,8 @@ class WatchlistScanner:
|
|||
for source in sources_to_process:
|
||||
logger.info(f"Curating Release Radar for {source}...")
|
||||
|
||||
# 1. Curate Release Radar - 50 tracks from recent albums.
|
||||
# Fetch a GENEROUS album budget (not 50) and exclude next-year announcements at the
|
||||
# query: future-dated albums sort to the top of release_date DESC and used to eat the
|
||||
# budget before the in-loop is_future_release skip ran, starving Fresh Tape to a few
|
||||
# tracks. The downstream caps (6/artist, top 75, take 50) still bound the output.
|
||||
recent_albums = self.database.get_discovery_recent_albums(
|
||||
limit=300, source=source, profile_id=profile_id, exclude_future_years=True)
|
||||
# 1. Curate Release Radar - 50 tracks from recent albums
|
||||
recent_albums = self.database.get_discovery_recent_albums(limit=50, source=source, profile_id=profile_id)
|
||||
release_radar_tracks = []
|
||||
|
||||
if not recent_albums:
|
||||
|
|
@ -4026,130 +4004,45 @@ class WatchlistScanner:
|
|||
The ranking lives in core.discovery.listening_recommendations (pure + tested); this only
|
||||
gathers inputs — all already in the DB, NO new network — and stores the result under NEW
|
||||
metadata/curated keys. Fully self-contained and guarded: any failure logs and returns, so
|
||||
it can never disturb the scan.
|
||||
|
||||
"Best in class" generation (the elevation over the first cut):
|
||||
• Seeds are recency-weighted — recent plays boost lifetime favourites so the picks
|
||||
track what you're into NOW, not just all-time totals.
|
||||
• The ranker is fed the RAW per-seed edges (one row per seed→similar), so an artist
|
||||
similar to several of your seeds accumulates real CONSENSUS — the old code fed the
|
||||
name-collapsed ``get_top_similar_artists`` query, which flattened every similar to a
|
||||
single seed (consensus could never fire). The raw edges also carry ``similarity_rank``,
|
||||
so a seed's CLOSEST matches outweigh its long-tail ones.
|
||||
• ``source_artist_id`` is a SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), so the id→name
|
||||
map is built from the artists' source-id columns, NOT the internal row id (the first
|
||||
cut keyed it by ``artists.id`` and resolved nothing — the feature produced 0 recs).
|
||||
Phase-1 candidate tracks still come from the discovery pool (like BYLT); a later phase
|
||||
swaps in a direct top-tracks fetch for richer coverage.
|
||||
it can never disturb the scan. Phase-1 candidate tracks come from the discovery pool (like
|
||||
BYLT); a later phase swaps in a direct top-tracks fetch for richer coverage.
|
||||
"""
|
||||
try:
|
||||
import json as _json
|
||||
from core.discovery.listening_recommendations import (
|
||||
aggregate_candidate_tracks,
|
||||
build_recency_weighted_seeds,
|
||||
choose_mix_fetch_source,
|
||||
group_similars_by_seed,
|
||||
names_match,
|
||||
rank_recommended_artists,
|
||||
to_mix_track,
|
||||
)
|
||||
|
||||
# Recency-weighted seeds: lifetime top artists, boosted by recent (30d) plays.
|
||||
lifetime = [s for s in (self.database.get_top_artists('all', 30) or []) if s.get('name')]
|
||||
if not lifetime:
|
||||
seeds = [{'name': s['name'], 'weight': s.get('play_count', 1)}
|
||||
for s in (self.database.get_top_artists('all', 30) or []) if s.get('name')]
|
||||
if not seeds:
|
||||
return
|
||||
recent_rows = self.database.get_top_artists('30d', 50) or []
|
||||
recent_counts = {r['name'].lower(): r.get('play_count', 0)
|
||||
for r in recent_rows if r.get('name')}
|
||||
seeds = build_recency_weighted_seeds(lifetime, recent_counts)
|
||||
seed_names = {s['name'].lower() for s in seeds}
|
||||
|
||||
# Owned-artist set (for exclusion) + the seeds' SOURCE ids (similar_artists.source_artist_id
|
||||
# is a Spotify/iTunes/Deezer/MusicBrainz id, never the internal artists.id). We only need
|
||||
# id→name for the SEED ids, since the edge query below is already scoped to them.
|
||||
owned, seed_source_ids, seed_id_to_name = set(), [], {}
|
||||
# id -> name + owned-artist set for the WHOLE library (similar_artists rows key the
|
||||
# similar artist by the seed artist's id).
|
||||
id_to_name, owned = {}, set()
|
||||
with self.database._get_connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute("SELECT name, spotify_artist_id, itunes_artist_id, deezer_id, "
|
||||
"musicbrainz_id FROM artists WHERE name IS NOT NULL AND name != ''")
|
||||
cur.execute("SELECT id, name FROM artists WHERE name IS NOT NULL AND name != ''")
|
||||
for row in cur.fetchall():
|
||||
nm = row[0]
|
||||
lname = (nm or '').lower()
|
||||
owned.add(lname)
|
||||
if lname in seed_names:
|
||||
for sid in (row[1], row[2], row[3], row[4]):
|
||||
if sid:
|
||||
seed_source_ids.append(str(sid))
|
||||
seed_id_to_name[str(sid)] = nm
|
||||
|
||||
# RAW per-seed edges (preserve consensus + similarity_rank). Scoped to the seeds.
|
||||
edges, edge_cols = [], ('source_artist_id', 'similar_artist_name', 'similarity_rank',
|
||||
'spotify_id', 'itunes_id', 'deezer_id', 'image_url', 'genres')
|
||||
if seed_source_ids:
|
||||
placeholders = ",".join("?" * len(seed_source_ids))
|
||||
cur.execute(
|
||||
f"SELECT source_artist_id, similar_artist_name, similarity_rank, "
|
||||
f"similar_artist_spotify_id, similar_artist_itunes_id, "
|
||||
f"similar_artist_deezer_id, image_url, genres FROM similar_artists "
|
||||
f"WHERE profile_id = ? AND source_artist_id IN ({placeholders})",
|
||||
[profile_id, *seed_source_ids])
|
||||
edges = [dict(zip(edge_cols, r, strict=False)) for r in cur.fetchall()]
|
||||
|
||||
# Per-name enrichment (image/ids/genres) so the Discover row can render rich cards.
|
||||
artist_meta_by_name = {}
|
||||
for e in edges:
|
||||
key = (e['similar_artist_name'] or '').lower()
|
||||
if not key:
|
||||
continue
|
||||
m = artist_meta_by_name.setdefault(key, {})
|
||||
for src_k, dst_k in (('spotify_id', 'spotify_artist_id'),
|
||||
('itunes_id', 'itunes_artist_id'),
|
||||
('deezer_id', 'deezer_artist_id')):
|
||||
if e.get(src_k) and not m.get(dst_k):
|
||||
m[dst_k] = e[src_k]
|
||||
if e.get('image_url') and not m.get('image_url'):
|
||||
m['image_url'] = e['image_url']
|
||||
if e.get('genres') and not m.get('genres'):
|
||||
m['genres'] = e['genres']
|
||||
|
||||
similars_by_seed = group_similars_by_seed(
|
||||
seeds, edges, seed_id_to_name, rank_attr='similarity_rank')
|
||||
|
||||
# Fallback: if the raw edges resolved nothing (e.g. source ids not yet populated),
|
||||
# degrade to the aggregated query so the feature still works rather than going dark.
|
||||
if not similars_by_seed:
|
||||
agg = self.database.get_top_similar_artists(limit=1000, profile_id=profile_id)
|
||||
similars_by_seed = group_similars_by_seed(
|
||||
seeds, agg, seed_id_to_name, rank_attr='similarity_rank')
|
||||
id_to_name[str(row[0])] = row[1]
|
||||
owned.add((row[1] or '').lower())
|
||||
|
||||
similar_rows = self.database.get_top_similar_artists(limit=1000, profile_id=profile_id)
|
||||
similars_by_seed = group_similars_by_seed(seeds, similar_rows, id_to_name)
|
||||
recs = rank_recommended_artists(seeds, similars_by_seed, owned, limit=40)
|
||||
if not recs:
|
||||
logger.info("[Listening Recs] no recommendations yet (no similar-artist coverage)")
|
||||
return
|
||||
|
||||
def _enrich(r):
|
||||
m = artist_meta_by_name.get(r.name.lower(), {})
|
||||
genres = m.get('genres')
|
||||
if isinstance(genres, str):
|
||||
try:
|
||||
genres = _json.loads(genres)
|
||||
except Exception:
|
||||
genres = None
|
||||
return {'name': r.name, 'seed_count': r.seed_count, 'seeds': r.seeds[:5],
|
||||
'score': r.score, 'spotify_artist_id': m.get('spotify_artist_id'),
|
||||
'itunes_artist_id': m.get('itunes_artist_id'),
|
||||
'deezer_artist_id': m.get('deezer_artist_id'),
|
||||
'image_url': m.get('image_url'),
|
||||
'genres': (genres[:3] if isinstance(genres, list) else None)}
|
||||
self.database.set_metadata('listening_recs_artists', _json.dumps([
|
||||
{'name': r.name, 'seed_count': r.seed_count, 'seeds': r.seeds[:5], 'score': r.score}
|
||||
for r in recs
|
||||
]))
|
||||
|
||||
self.database.set_metadata('listening_recs_artists',
|
||||
_json.dumps([_enrich(r) for r in recs]))
|
||||
|
||||
# Candidate tracks for the "Listening Mix" playlist row: each recommended artist's
|
||||
# top tracks. Prefer a DIRECT top-tracks fetch (Spotify/Deezer — richest + most
|
||||
# accurate), fall back to the discovery pool (covers iTunes + any artist the fetch
|
||||
# missed). Stored as full render-ready dicts so the row needs NO pool re-hydration —
|
||||
# robust against pool rotation (the bug that shrinks Fresh Tape/Archives at read time).
|
||||
# Candidate tracks from the discovery pool grouped by artist (phase 1: no new network).
|
||||
pool, active_source = [], None
|
||||
for src in (sources_to_process or []):
|
||||
pool = self.database.get_discovery_pool_tracks(
|
||||
|
|
@ -4157,94 +4050,24 @@ class WatchlistScanner:
|
|||
if pool:
|
||||
active_source = src
|
||||
break
|
||||
if not active_source:
|
||||
active_source = (sources_to_process or ['spotify'])[0]
|
||||
|
||||
# Pool baseline grouped by artist (full render dicts), no network.
|
||||
pool_by_artist = {}
|
||||
for t in (pool or []):
|
||||
an = (getattr(t, 'artist_name', '') or '').lower()
|
||||
tid = (getattr(t, 'spotify_track_id', None) if active_source == 'spotify'
|
||||
else getattr(t, 'itunes_track_id', None) if active_source == 'itunes'
|
||||
else getattr(t, 'deezer_track_id', None))
|
||||
name = getattr(t, 'track_name', '') or ''
|
||||
if not an or not tid or not name:
|
||||
continue
|
||||
tdj = getattr(t, 'track_data_json', None)
|
||||
if isinstance(tdj, str):
|
||||
try:
|
||||
tdj = _json.loads(tdj)
|
||||
except Exception:
|
||||
tdj = None
|
||||
pool_by_artist.setdefault(an, []).append({
|
||||
'track_id': str(tid), 'name': name, 'track_name': name,
|
||||
'artist_name': getattr(t, 'artist_name', '') or '',
|
||||
'album_name': getattr(t, 'album_name', '') or '',
|
||||
'album_cover_url': getattr(t, 'album_cover_url', None),
|
||||
'duration_ms': getattr(t, 'duration_ms', 0) or 0,
|
||||
'track_data_json': tdj, 'source': active_source,
|
||||
f'{active_source}_track_id': str(tid)})
|
||||
track_ids = []
|
||||
if pool:
|
||||
by_artist = {}
|
||||
for t in pool:
|
||||
an = (getattr(t, 'artist_name', '') or '').lower()
|
||||
tid = (getattr(t, 'spotify_track_id', None) if active_source == 'spotify'
|
||||
else getattr(t, 'itunes_track_id', None) if active_source == 'itunes'
|
||||
else getattr(t, 'deezer_track_id', None))
|
||||
if an and tid:
|
||||
by_artist.setdefault(an, []).append({'name': getattr(t, 'track_name', ''), 'id': tid})
|
||||
candidates = aggregate_candidate_tracks(recs, by_artist, per_artist=3, limit=50)
|
||||
track_ids = [c['id'] for c in candidates if c.get('id')]
|
||||
if track_ids:
|
||||
self.database.save_curated_playlist('listening_recs_tracks', track_ids, profile_id=profile_id)
|
||||
|
||||
# Direct top-tracks enrichment — guarded, bounded (top 20 recs), fail-soft per artist.
|
||||
# Source-independent: the active source is used when it can fetch top tracks
|
||||
# (Spotify/Deezer); otherwise we fall back to Deezer's public top-tracks API (no auth,
|
||||
# available to every user) so iTunes / Discogs / MusicBrainz users still get a full mix
|
||||
# without switching sources — the tracks are acquired via Soulseek by artist+title, so
|
||||
# the fetch source need not match the user's active source.
|
||||
fetched_by_artist = {}
|
||||
try:
|
||||
active_client = get_client_for_source(active_source) if active_source in ('spotify', 'deezer') else None
|
||||
active_can_fetch = bool(active_client and hasattr(active_client, 'get_artist_top_tracks'))
|
||||
fetch_source = choose_mix_fetch_source(active_source, active_can_fetch)
|
||||
client = active_client if (active_can_fetch and fetch_source == active_source) \
|
||||
else get_client_for_source(fetch_source)
|
||||
if client and hasattr(client, 'get_artist_top_tracks'):
|
||||
id_key = 'spotify_artist_id' if fetch_source == 'spotify' else 'deezer_artist_id'
|
||||
can_search = hasattr(client, 'search_artists')
|
||||
for r in recs[:20]:
|
||||
aid = artist_meta_by_name.get(r.name.lower(), {}).get(id_key)
|
||||
# Most similar-artist rows store a name but no source id (and a fallback
|
||||
# fetch source won't have the active source's ids at all), so resolve it
|
||||
# by name-search — guarded by names_match so we never fetch the WRONG
|
||||
# artist's tracks (a same-name act).
|
||||
if not aid and can_search:
|
||||
try:
|
||||
found = client.search_artists(r.name, limit=1) or []
|
||||
except Exception as _s_err:
|
||||
logger.debug("[Listening Recs] artist search failed for %s: %s",
|
||||
r.name, _s_err)
|
||||
found = []
|
||||
if found and names_match(r.name, getattr(found[0], 'name', '')):
|
||||
aid = getattr(found[0], 'id', None)
|
||||
if not aid:
|
||||
continue
|
||||
try:
|
||||
raw = client.get_artist_top_tracks(str(aid), limit=8) or []
|
||||
except Exception as _tt_err:
|
||||
logger.debug("[Listening Recs] top-tracks fetch failed for %s: %s",
|
||||
r.name, _tt_err)
|
||||
continue
|
||||
shaped = [s for s in (to_mix_track(x, fetch_source) for x in raw) if s][:5]
|
||||
if shaped:
|
||||
fetched_by_artist[r.name.lower()] = shaped
|
||||
except Exception as _enr_err:
|
||||
logger.debug("[Listening Recs] top-tracks enrichment skipped: %s", _enr_err)
|
||||
|
||||
# Merge: prefer fetched top tracks, fall back to the pool per artist.
|
||||
top_tracks_by_artist = {}
|
||||
for r in recs:
|
||||
merged = fetched_by_artist.get(r.name.lower()) or pool_by_artist.get(r.name.lower())
|
||||
if merged:
|
||||
top_tracks_by_artist[r.name.lower()] = merged
|
||||
|
||||
mix = aggregate_candidate_tracks(recs, top_tracks_by_artist, per_artist=3, limit=50)
|
||||
track_ids = [m.get('track_id') for m in mix if m.get('track_id')]
|
||||
if mix:
|
||||
self.database.set_metadata('listening_recs_tracks_full', _json.dumps(mix))
|
||||
self.database.save_curated_playlist('listening_recs_tracks', track_ids, profile_id=profile_id)
|
||||
|
||||
logger.info("[Listening Recs] %d recommended artists, %d mix tracks (%d artists via top-tracks fetch)",
|
||||
len(recs), len(mix), len(fetched_by_artist))
|
||||
logger.info("[Listening Recs] %d recommended artists, %d candidate tracks",
|
||||
len(recs), len(track_ids))
|
||||
except Exception as e:
|
||||
logger.debug("[Listening Recs] generation skipped: %s", e)
|
||||
|
||||
|
|
|
|||
|
|
@ -8,8 +8,6 @@ import threading
|
|||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, Dict
|
||||
|
||||
from core.metadata import normalize_image_url
|
||||
from core.metadata.artwork import is_internal_image_host
|
||||
from core.wishlist.reporting import build_wishlist_stats_payload
|
||||
from core.wishlist.selection import prepare_wishlist_tracks_for_display
|
||||
from core.wishlist.service import get_wishlist_service
|
||||
|
|
@ -212,74 +210,6 @@ def set_wishlist_cycle(runtime: WishlistRouteRuntime, cycle: str) -> tuple[Dict[
|
|||
return {"error": str(exc)}, 500
|
||||
|
||||
|
||||
def _needs_image_fix(url: str | None) -> bool:
|
||||
"""True when an image URL won't render in the browser as-is — a media-server RELATIVE
|
||||
path (/library/.., /Items/.., /rest/..) or an internal/localhost host. Spotify/iTunes CDN
|
||||
URLs render directly and are left untouched, so already-working items never change."""
|
||||
if not url or not isinstance(url, str):
|
||||
return False
|
||||
if url.startswith('/') and not url.startswith('//'):
|
||||
return True
|
||||
if url.startswith('http://') or url.startswith('https://'):
|
||||
return is_internal_image_host(url)
|
||||
return False
|
||||
|
||||
|
||||
def _enrich_wishlist_images(tracks: list[dict[str, Any]], db: Any) -> dict[str, str]:
|
||||
"""Make wishlist art browser-renderable using the library data we already have.
|
||||
|
||||
The library stores album/artist art as media-server RELATIVE paths (e.g. Plex
|
||||
/library/metadata/..) which don't render in a browser <img>. Normal wishlist items carry
|
||||
Spotify CDN URLs (fine), but library-sourced items — re-downloads and preview-clip
|
||||
re-fetches — carry the relative path, so their art comes up blank. We fix two things here,
|
||||
on read, so it also repairs items already sitting in the wishlist:
|
||||
|
||||
1. Normalize each track's album.images[*].url that needs it (relative/internal only —
|
||||
CDN URLs are left as-is to avoid regressing items that already render).
|
||||
2. Build an artist-name -> normalized library photo map so the nebula can show artist
|
||||
photos for non-watchlist artists (it otherwise only has watchlisted-artist photos).
|
||||
"""
|
||||
artist_names: set[str] = set()
|
||||
for track in tracks:
|
||||
sd = track.get('spotify_data')
|
||||
if isinstance(sd, dict):
|
||||
album = sd.get('album')
|
||||
if isinstance(album, dict):
|
||||
images = album.get('images')
|
||||
if isinstance(images, list):
|
||||
for img in images:
|
||||
if isinstance(img, dict) and _needs_image_fix(img.get('url')):
|
||||
fixed = normalize_image_url(img['url'])
|
||||
if fixed:
|
||||
img['url'] = fixed
|
||||
name = track.get('artist_name')
|
||||
if name and name != 'Unknown Artist':
|
||||
artist_names.add(name)
|
||||
|
||||
artist_images: dict[str, str] = {}
|
||||
if not artist_names:
|
||||
return artist_images
|
||||
try:
|
||||
conn = db._get_connection()
|
||||
try:
|
||||
placeholders = ','.join('?' * len(artist_names))
|
||||
rows = conn.execute(
|
||||
f"SELECT name, thumb_url FROM artists "
|
||||
f"WHERE name IN ({placeholders}) AND thumb_url IS NOT NULL AND thumb_url != ''",
|
||||
list(artist_names),
|
||||
).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
for row in rows:
|
||||
name, thumb = row[0], row[1]
|
||||
fixed = normalize_image_url(thumb) if _needs_image_fix(thumb) else thumb
|
||||
if name and fixed:
|
||||
artist_images[name.lower()] = fixed
|
||||
except Exception as exc: # noqa: BLE001 — art is cosmetic, never fail the tracks endpoint
|
||||
logger.debug("Could not build wishlist artist-image map: %s", exc)
|
||||
return artist_images
|
||||
|
||||
|
||||
def get_wishlist_tracks(
|
||||
runtime: WishlistRouteRuntime,
|
||||
*,
|
||||
|
|
@ -312,9 +242,6 @@ def get_wishlist_tracks(
|
|||
prepared["duplicates_found"],
|
||||
)
|
||||
|
||||
# Make library-sourced art renderable + supply artist photos (see _enrich_wishlist_images).
|
||||
artist_images = _enrich_wishlist_images(prepared["tracks"], db)
|
||||
|
||||
if category:
|
||||
runtime.logger.info(
|
||||
"Wishlist filter: %s/%s tracks in '%s' category (limit: %s)",
|
||||
|
|
@ -323,18 +250,9 @@ def get_wishlist_tracks(
|
|||
category,
|
||||
limit or "none",
|
||||
)
|
||||
return {
|
||||
"tracks": prepared["tracks"],
|
||||
"category": category,
|
||||
"total": prepared["total"],
|
||||
"artist_images": artist_images,
|
||||
}, 200
|
||||
return {"tracks": prepared["tracks"], "category": category, "total": prepared["total"]}, 200
|
||||
|
||||
return {
|
||||
"tracks": prepared["tracks"],
|
||||
"total": prepared["total"],
|
||||
"artist_images": artist_images,
|
||||
}, 200
|
||||
return {"tracks": prepared["tracks"], "total": prepared["total"]}, 200
|
||||
except Exception as exc:
|
||||
runtime.logger.error("Error getting wishlist tracks: %s", exc)
|
||||
return {"error": str(exc)}, 500
|
||||
|
|
|
|||
|
|
@ -38,20 +38,6 @@ from core.download_plugins.types import SearchResult, TrackResult, AlbumResult,
|
|||
logger = get_logger("youtube_client")
|
||||
|
||||
|
||||
def _resolve_cookie_opts() -> dict:
|
||||
"""yt-dlp cookie options from Settings → YouTube: either a browser store OR a
|
||||
pasted cookies.txt. The 'Paste cookies.txt' dropdown value is the sentinel
|
||||
'custom' — which must become a yt-dlp ``cookiefile`` pointing at the saved file,
|
||||
NOT be passed through as a browser name (yt-dlp rejects: 'unsupported browser:
|
||||
custom'). Delegates to the shared, tested precedence in core.youtube_cookies."""
|
||||
from config.settings import config_manager
|
||||
from core.youtube_cookies import build_youtube_cookie_opts
|
||||
mode = config_manager.get('youtube.cookies_browser', '')
|
||||
cookiefile = config_manager.get('youtube.cookies_file', '')
|
||||
exists = bool(cookiefile) and os.path.exists(cookiefile)
|
||||
return build_youtube_cookie_opts(mode, cookiefile, cookiefile_exists=exists)
|
||||
|
||||
|
||||
@dataclass
|
||||
class YouTubeSearchResult:
|
||||
"""YouTube search result with metadata parsing"""
|
||||
|
|
@ -234,9 +220,11 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
'age_limit': None, # Don't skip age-restricted
|
||||
}
|
||||
|
||||
# Cookie support — a logged-in browser store OR a pasted cookies.txt
|
||||
# (the 'custom' paste mode resolves to a cookiefile, not a browser name).
|
||||
self.download_opts.update(_resolve_cookie_opts())
|
||||
# Cookie support — use browser cookies for YouTube auth
|
||||
from config.settings import config_manager
|
||||
cookies_browser = config_manager.get('youtube.cookies_browser', '')
|
||||
if cookies_browser:
|
||||
self.download_opts['cookiesfrombrowser'] = (cookies_browser,)
|
||||
|
||||
# Track current download progress (mirrors Soulseek transfer tracking)
|
||||
self.current_download_id: Optional[str] = None
|
||||
|
|
@ -321,12 +309,11 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
"""Reload YouTube settings from config (called when settings are saved)."""
|
||||
from config.settings import config_manager
|
||||
self._download_delay = config_manager.get('youtube.download_delay', 3)
|
||||
# Clear both cookie sources, then re-apply from current settings (browser
|
||||
# store or pasted cookies.txt) so a mode switch doesn't leave a stale arg.
|
||||
self.download_opts.pop('cookiesfrombrowser', None)
|
||||
self.download_opts.pop('cookiefile', None)
|
||||
_cookie_opts = _resolve_cookie_opts()
|
||||
self.download_opts.update(_cookie_opts)
|
||||
cookies_browser = config_manager.get('youtube.cookies_browser', '')
|
||||
if cookies_browser:
|
||||
self.download_opts['cookiesfrombrowser'] = (cookies_browser,)
|
||||
elif 'cookiesfrombrowser' in self.download_opts:
|
||||
del self.download_opts['cookiesfrombrowser']
|
||||
|
||||
# Reload download path
|
||||
new_path = Path(config_manager.get('soulseek.download_path', './downloads'))
|
||||
|
|
@ -336,7 +323,7 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
self.download_opts['outtmpl'] = str(self.download_path / '%(title)s.%(ext)s')
|
||||
logger.info(f"YouTube download path updated to: {self.download_path}")
|
||||
|
||||
logger.info(f"YouTube settings reloaded (delay={self._download_delay}s, cookies={'enabled' if _cookie_opts else 'disabled'})")
|
||||
logger.info(f"YouTube settings reloaded (delay={self._download_delay}s, cookies={'enabled' if cookies_browser else 'disabled'})")
|
||||
|
||||
async def check_connection(self) -> bool:
|
||||
"""
|
||||
|
|
@ -741,6 +728,7 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _search():
|
||||
from config.settings import config_manager
|
||||
ydl_opts = {
|
||||
'quiet': True,
|
||||
'no_warnings': True,
|
||||
|
|
@ -748,7 +736,9 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
'default_search': 'ytsearch',
|
||||
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
|
||||
}
|
||||
ydl_opts.update(_resolve_cookie_opts())
|
||||
cookies_browser = config_manager.get('youtube.cookies_browser', '')
|
||||
if cookies_browser:
|
||||
ydl_opts['cookiesfrombrowser'] = (cookies_browser,)
|
||||
|
||||
search_query = self._escape_ytsearch_query(query)
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
|
|
@ -816,6 +806,7 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _search():
|
||||
from config.settings import config_manager
|
||||
ydl_opts = {
|
||||
'quiet': True,
|
||||
'no_warnings': True,
|
||||
|
|
@ -825,7 +816,9 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
}
|
||||
|
||||
# Add cookie support for search (avoids bot detection)
|
||||
ydl_opts.update(_resolve_cookie_opts())
|
||||
cookies_browser = config_manager.get('youtube.cookies_browser', '')
|
||||
if cookies_browser:
|
||||
ydl_opts['cookiesfrombrowser'] = (cookies_browser,)
|
||||
|
||||
search_query = self._escape_ytsearch_query(query)
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
|
|
@ -1094,12 +1087,10 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
|
||||
# On retry, try different strategies
|
||||
if attempt == 1:
|
||||
# Drop cookies — authenticated sessions (browser store OR a
|
||||
# pasted cookies.txt) sometimes get restricted formats.
|
||||
if 'cookiesfrombrowser' in download_opts or 'cookiefile' in download_opts:
|
||||
logger.info(f"Retry {attempt + 1}/{max_retries} without cookies")
|
||||
# Drop browser cookies — authenticated sessions sometimes get restricted formats
|
||||
if 'cookiesfrombrowser' in download_opts:
|
||||
logger.info(f"Retry {attempt + 1}/{max_retries} without browser cookies")
|
||||
download_opts.pop('cookiesfrombrowser', None)
|
||||
download_opts.pop('cookiefile', None)
|
||||
else:
|
||||
logger.info(f"Retry {attempt + 1}/{max_retries} with web_creator client")
|
||||
download_opts['extractor_args'] = {
|
||||
|
|
@ -1109,7 +1100,6 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
logger.info(f"Retry {attempt + 1}/{max_retries} with 'best' format (video fallback)")
|
||||
download_opts['format'] = 'best'
|
||||
download_opts.pop('cookiesfrombrowser', None)
|
||||
download_opts.pop('cookiefile', None)
|
||||
download_opts.pop('extractor_args', None)
|
||||
|
||||
|
||||
|
|
@ -1170,6 +1160,8 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
Final file path if successful, None otherwise
|
||||
"""
|
||||
try:
|
||||
from config.settings import config_manager
|
||||
|
||||
def _progress_hook(d):
|
||||
if progress_callback and d.get('status') == 'downloading':
|
||||
total = d.get('total_bytes') or d.get('total_bytes_estimate') or 0
|
||||
|
|
@ -1188,7 +1180,9 @@ class YouTubeClient(DownloadSourcePlugin):
|
|||
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
|
||||
}
|
||||
|
||||
download_opts.update(_resolve_cookie_opts())
|
||||
cookies_browser = config_manager.get('youtube.cookies_browser', '')
|
||||
if cookies_browser:
|
||||
download_opts['cookiesfrombrowser'] = (cookies_browser,)
|
||||
|
||||
with yt_dlp.YoutubeDL(download_opts) as ydl:
|
||||
info = ydl.extract_info(video_url, download=True)
|
||||
|
|
|
|||
|
|
@ -677,12 +677,6 @@ class MusicDatabase:
|
|||
cursor.execute(f"ALTER TABLE library_history ADD COLUMN {_col} TEXT")
|
||||
logger.info(f"Added {_col} column to library_history")
|
||||
|
||||
# Index on verification_status — MUST come after the ALTER above:
|
||||
# on a fresh DB the base CREATE TABLE has no verification_status
|
||||
# column, so indexing it before the migration adds it raises
|
||||
# "no such column: verification_status" and aborts DB init.
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_lh_verification_status ON library_history (verification_status)")
|
||||
|
||||
# One-time backfill: derive verification_status for history rows
|
||||
# written before the column existed (or by pipeline exits that
|
||||
# missed it) from the acoustid_result those imports already
|
||||
|
|
@ -1000,10 +994,6 @@ class MusicDatabase:
|
|||
|
||||
self._init_manual_library_match_table()
|
||||
self._backfill_mirrored_track_source_ids()
|
||||
# Self-heal the Unverified review queue: lift history rows stuck at
|
||||
# 'unverified' whose file has since been verified (issue #934). Cheap,
|
||||
# idempotent (only touches rows that need it), so it's safe every boot.
|
||||
self.reconcile_unverified_history_from_tracks()
|
||||
|
||||
def _backfill_mirrored_track_source_ids(self) -> int:
|
||||
"""One-time, idempotent: assign a stable source_track_id to mirrored tracks
|
||||
|
|
@ -1175,13 +1165,6 @@ class MusicDatabase:
|
|||
if track_cols and 'year' not in track_cols:
|
||||
cursor.execute("ALTER TABLE tracks ADD COLUMN year INTEGER")
|
||||
logger.info("Repaired missing year column on tracks table (#910)")
|
||||
# #927 — multi-disc fix: the scan now writes a real disc_number, but the column
|
||||
# was only ever added by a separate migration that doesn't run on fresh installs,
|
||||
# so the new INSERT/UPDATE would hard-fail with "no column named disc_number".
|
||||
# Same shape as the year repair above: additive, defaults to 1, ensured on every DB.
|
||||
if track_cols and 'disc_number' not in track_cols:
|
||||
cursor.execute("ALTER TABLE tracks ADD COLUMN disc_number INTEGER DEFAULT 1")
|
||||
logger.info("Repaired missing disc_number column on tracks table (#927)")
|
||||
|
||||
cursor.execute("PRAGMA table_info(albums)")
|
||||
album_cols = {c[1] for c in cursor.fetchall()}
|
||||
|
|
@ -6646,19 +6629,6 @@ class MusicDatabase:
|
|||
track_id = str(track_obj.ratingKey)
|
||||
title = track_obj.title
|
||||
track_number = getattr(track_obj, 'trackNumber', None)
|
||||
# Multi-disc: capture the disc number so multi-disc albums don't all
|
||||
# collapse onto disc 1 (which mis-files disc-2+ tracks and flags them
|
||||
# "missing"). Jellyfin/Navidrome wrappers set .discNumber; plexapi's Track
|
||||
# exposes .parentIndex. Floor to >=1 — a missing/0 disc is disc 1.
|
||||
_raw_disc = getattr(track_obj, 'discNumber', None)
|
||||
if _raw_disc is None:
|
||||
_raw_disc = getattr(track_obj, 'parentIndex', None)
|
||||
try:
|
||||
disc_number = int(_raw_disc)
|
||||
if disc_number < 1:
|
||||
disc_number = 1
|
||||
except (TypeError, ValueError):
|
||||
disc_number = 1
|
||||
duration = getattr(track_obj, 'duration', None)
|
||||
|
||||
# Get file path and media info (Plex-specific, Jellyfin may not have these)
|
||||
|
|
@ -6750,9 +6720,9 @@ class MusicDatabase:
|
|||
if is_new_track:
|
||||
cursor.execute("""
|
||||
INSERT INTO tracks
|
||||
(id, album_id, artist_id, title, track_number, disc_number, duration, file_path, bitrate, file_size, server_source, track_artist, musicbrainz_recording_id, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
|
||||
""", (track_id, album_id, artist_id, title, track_number, disc_number, duration, file_path, bitrate, file_size, server_source, track_artist, mbid))
|
||||
(id, album_id, artist_id, title, track_number, duration, file_path, bitrate, file_size, server_source, track_artist, musicbrainz_recording_id, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
|
||||
""", (track_id, album_id, artist_id, title, track_number, duration, file_path, bitrate, file_size, server_source, track_artist, mbid))
|
||||
else:
|
||||
# Update server-provided fields only — preserves spotify_track_id, deezer_id,
|
||||
# isrc, bpm, and all other enrichment data. file_size uses
|
||||
|
|
@ -6761,7 +6731,7 @@ class MusicDatabase:
|
|||
# an existing value.
|
||||
cursor.execute("""
|
||||
UPDATE tracks
|
||||
SET album_id = ?, artist_id = ?, title = ?, track_number = ?, disc_number = ?,
|
||||
SET album_id = ?, artist_id = ?, title = ?, track_number = ?,
|
||||
duration = ?, file_path = ?, bitrate = ?,
|
||||
file_size = COALESCE(?, file_size),
|
||||
server_source = ?,
|
||||
|
|
@ -6769,7 +6739,7 @@ class MusicDatabase:
|
|||
musicbrainz_recording_id = COALESCE(?, musicbrainz_recording_id),
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (album_id, artist_id, title, track_number, disc_number, duration, file_path, bitrate, file_size, server_source, track_artist, mbid, track_id))
|
||||
""", (album_id, artist_id, title, track_number, duration, file_path, bitrate, file_size, server_source, track_artist, mbid, track_id))
|
||||
|
||||
conn.commit()
|
||||
|
||||
|
|
@ -8772,7 +8742,7 @@ class MusicDatabase:
|
|||
# Quality profile management methods
|
||||
|
||||
def get_quality_profile(self) -> dict:
|
||||
"""Get the quality profile configuration, returns default if not set."""
|
||||
"""Get the quality profile configuration, returns default if not set"""
|
||||
import json
|
||||
|
||||
profile_json = self.get_preference('quality_profile')
|
||||
|
|
@ -8780,86 +8750,21 @@ class MusicDatabase:
|
|||
if profile_json:
|
||||
try:
|
||||
profile = json.loads(profile_json)
|
||||
version = profile.get('version', 1)
|
||||
if version < 2:
|
||||
logger.info("Migrating quality profile v1 → v3")
|
||||
# Migrate v1 profiles (min_mb/max_mb) to v2 (min_kbps/max_kbps)
|
||||
if profile.get('version', 1) < 2:
|
||||
logger.info("Migrating quality profile from v1 (file size) to v2 (bitrate density)")
|
||||
return self._get_default_quality_profile()
|
||||
if version == 2:
|
||||
logger.info("Migrating quality profile v2 → v3 (adding ranked_targets)")
|
||||
return self._migrate_v2_to_v3(profile)
|
||||
return profile
|
||||
except json.JSONDecodeError:
|
||||
logger.error("Failed to parse quality profile JSON, returning default")
|
||||
|
||||
return self._get_default_quality_profile()
|
||||
|
||||
# 24-bit FLAC ladder seeded on migration for users who had a streaming
|
||||
# source on Hi-Res under the old (now removed) per-source quality dropdowns.
|
||||
_HIRES_24BIT_TARGETS = [
|
||||
{"label": "FLAC 24-bit/192kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 192000},
|
||||
{"label": "FLAC 24-bit/96kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 96000},
|
||||
{"label": "FLAC 24-bit/48kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 48000},
|
||||
{"label": "FLAC 24-bit/44.1kHz","format": "flac", "bit_depth": 24, "min_sample_rate": 44100},
|
||||
]
|
||||
|
||||
def _had_hires_source_preference(self) -> bool:
|
||||
"""True if the user had any streaming source set to a Hi-Res tier under
|
||||
the old per-source quality dropdowns (tidal_download/qobuz/hifi_download
|
||||
.quality = 'hires'|'hires_max'), which #896 removed in favour of the
|
||||
global profile. Used to preserve their intent on migration."""
|
||||
try:
|
||||
from config.settings import config_manager
|
||||
except Exception:
|
||||
return False
|
||||
hires = {'hires', 'hires_max'}
|
||||
for key in ('tidal_download.quality', 'qobuz.quality', 'hifi_download.quality'):
|
||||
try:
|
||||
if str(config_manager.get(key) or '').strip().lower() in hires:
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
def _migrate_v2_to_v3(self, profile: dict) -> dict:
|
||||
"""Add ranked_targets to a v2 profile without losing its qualities dict."""
|
||||
from core.quality.model import v2_qualities_to_ranked_targets
|
||||
profile = dict(profile)
|
||||
profile['version'] = 3
|
||||
if 'ranked_targets' not in profile:
|
||||
ranked = v2_qualities_to_ranked_targets(profile.get('qualities', {}))
|
||||
# #896 review #5: the per-source quality dropdowns are gone — sources
|
||||
# now derive their tier from this profile. If the user had a source on
|
||||
# Hi-Res, seed 24-bit FLAC targets at the top so they keep Hi-Res
|
||||
# instead of silently dropping to lossless. Skip when the profile
|
||||
# already expresses 24-bit (don't duplicate the ladder).
|
||||
already_24bit = any(
|
||||
t.get('format') == 'flac' and (t.get('bit_depth') or 0) >= 24
|
||||
for t in ranked
|
||||
)
|
||||
if not already_24bit and self._had_hires_source_preference():
|
||||
ranked = [dict(t) for t in self._HIRES_24BIT_TARGETS] + ranked
|
||||
profile['ranked_targets'] = ranked
|
||||
return profile
|
||||
|
||||
def _get_default_quality_profile(self) -> dict:
|
||||
"""Return the default v3 quality profile (balanced preset)."""
|
||||
"""Return the default v2 quality profile (balanced preset)"""
|
||||
return {
|
||||
"version": 3,
|
||||
"version": 2,
|
||||
"preset": "balanced",
|
||||
"fallback_enabled": True,
|
||||
"search_mode": "priority",
|
||||
"rank_candidates_by_quality": False,
|
||||
"ranked_targets": [
|
||||
{"label": "FLAC 24-bit/192kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 192000},
|
||||
{"label": "FLAC 24-bit/96kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 96000},
|
||||
{"label": "FLAC 24-bit/48kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 48000},
|
||||
{"label": "FLAC 24-bit/44.1kHz","format": "flac", "bit_depth": 24, "min_sample_rate": 44100},
|
||||
{"label": "FLAC 16-bit", "format": "flac", "bit_depth": 16},
|
||||
{"label": "MP3 320kbps", "format": "mp3", "min_bitrate": 320},
|
||||
{"label": "MP3 256kbps", "format": "mp3", "min_bitrate": 256},
|
||||
{"label": "MP3 192kbps", "format": "mp3", "min_bitrate": 192},
|
||||
],
|
||||
# Keep qualities dict for backwards compat with any old code paths still reading it
|
||||
"qualities": {
|
||||
"flac": {
|
||||
"enabled": True,
|
||||
|
|
@ -8896,119 +8801,141 @@ class MusicDatabase:
|
|||
"priority": 1.5
|
||||
}
|
||||
},
|
||||
"fallback_enabled": True
|
||||
}
|
||||
|
||||
# Presets whose per-preset customizations we remember across switches.
|
||||
_KNOWN_PRESETS = ('audiophile', 'balanced', 'space_saver')
|
||||
|
||||
def set_quality_profile(self, profile: dict) -> bool:
|
||||
"""Save quality profile configuration.
|
||||
|
||||
Besides the single active profile (read by the download pipeline), we also
|
||||
stash the profile under its preset name so switching presets and coming
|
||||
back restores the user's edits instead of the factory defaults. 'custom'
|
||||
and unknown preset names are not stashed."""
|
||||
"""Save quality profile configuration"""
|
||||
import json
|
||||
|
||||
try:
|
||||
profile_json = json.dumps(profile)
|
||||
self.set_preference('quality_profile', profile_json)
|
||||
|
||||
preset_name = profile.get('preset')
|
||||
if preset_name in self._KNOWN_PRESETS:
|
||||
store = self._load_preset_store()
|
||||
store[preset_name] = profile
|
||||
self.set_preference('quality_profile_presets', json.dumps(store))
|
||||
|
||||
logger.info(f"Quality profile saved: preset={profile.get('preset', 'custom')}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save quality profile: {e}")
|
||||
return False
|
||||
|
||||
def _load_preset_store(self) -> dict:
|
||||
"""Per-preset customizations, keyed by preset name. {} if none saved."""
|
||||
import json
|
||||
raw = self.get_preference('quality_profile_presets')
|
||||
if raw:
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
except json.JSONDecodeError:
|
||||
logger.error("Failed to parse quality_profile_presets, ignoring")
|
||||
return {}
|
||||
|
||||
def reset_quality_preset(self, preset_name: str) -> dict:
|
||||
"""Forget a preset's saved customizations and return its factory defaults."""
|
||||
import json
|
||||
store = self._load_preset_store()
|
||||
if preset_name in store:
|
||||
del store[preset_name]
|
||||
self.set_preference('quality_profile_presets', json.dumps(store))
|
||||
return self.get_quality_preset(preset_name, customized=False)
|
||||
|
||||
def get_quality_preset(self, preset_name: str, *, customized: bool = True) -> dict:
|
||||
"""Get a quality preset (v3 format with ranked_targets).
|
||||
|
||||
With ``customized`` (default), a preset the user has edited is returned in
|
||||
its saved form; otherwise the hard-coded factory defaults are returned."""
|
||||
if customized:
|
||||
saved = self._load_preset_store().get(preset_name)
|
||||
if saved:
|
||||
return saved
|
||||
return self._factory_quality_preset(preset_name)
|
||||
|
||||
def _factory_quality_preset(self, preset_name: str) -> dict:
|
||||
"""The hard-coded factory defaults for a preset (ignores customizations)."""
|
||||
# Strict 24-bit FLAC ladder — no 16-bit, no lossy. This is what
|
||||
# "audiophile" means: only true hi-res passes.
|
||||
_FLAC_24BIT_TARGETS = [
|
||||
{"label": "FLAC 24-bit/192kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 192000},
|
||||
{"label": "FLAC 24-bit/96kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 96000},
|
||||
{"label": "FLAC 24-bit/48kHz", "format": "flac", "bit_depth": 24, "min_sample_rate": 48000},
|
||||
{"label": "FLAC 24-bit/44.1kHz","format": "flac", "bit_depth": 24, "min_sample_rate": 44100},
|
||||
]
|
||||
# Lossless ladder used by "balanced" — hi-res first, then CD-quality 16-bit.
|
||||
_FLAC_HI_RES_TARGETS = _FLAC_24BIT_TARGETS + [
|
||||
{"label": "FLAC 16-bit", "format": "flac", "bit_depth": 16},
|
||||
]
|
||||
_MP3_TARGETS = [
|
||||
{"label": "MP3 320kbps", "format": "mp3", "min_bitrate": 320},
|
||||
{"label": "MP3 256kbps", "format": "mp3", "min_bitrate": 256},
|
||||
{"label": "MP3 192kbps", "format": "mp3", "min_bitrate": 192},
|
||||
]
|
||||
# Legacy v2 ``qualities`` dict carried alongside ranked_targets for
|
||||
# backwards compat — read by the settings UI and the #886 AAC opt-in
|
||||
# toggle. AAC ships OFF in every preset; its priority sits it above MP3
|
||||
# but below FLAC (space_saver puts it at 0.5, still above its MP3 tiers).
|
||||
def _quals(*, flac_en, flac_prio, mp3_320_en, mp3_256_en, mp3_192_en,
|
||||
mp3_320_prio=2, mp3_256_prio=3, mp3_192_prio=4, aac_prio=1.5):
|
||||
return {
|
||||
"flac": {"enabled": flac_en, "min_kbps": 500, "max_kbps": 10000, "priority": flac_prio, "bit_depth": "any"},
|
||||
"mp3_320": {"enabled": mp3_320_en, "min_kbps": 280, "max_kbps": 500, "priority": mp3_320_prio},
|
||||
"mp3_256": {"enabled": mp3_256_en, "min_kbps": 200, "max_kbps": 400, "priority": mp3_256_prio},
|
||||
"mp3_192": {"enabled": mp3_192_en, "min_kbps": 150, "max_kbps": 300, "priority": mp3_192_prio},
|
||||
"aac": {"enabled": False, "min_kbps": 128, "max_kbps": 400, "priority": aac_prio},
|
||||
}
|
||||
|
||||
def get_quality_preset(self, preset_name: str) -> dict:
|
||||
"""Get a predefined quality preset"""
|
||||
presets = {
|
||||
"audiophile": {
|
||||
"version": 3, "preset": "audiophile", "fallback_enabled": False,
|
||||
"ranked_targets": _FLAC_24BIT_TARGETS,
|
||||
"qualities": _quals(flac_en=True, flac_prio=1, mp3_320_en=False, mp3_256_en=False, mp3_192_en=False),
|
||||
"version": 2,
|
||||
"preset": "audiophile",
|
||||
"qualities": {
|
||||
"flac": {
|
||||
"enabled": True,
|
||||
"min_kbps": 500,
|
||||
"max_kbps": 10000,
|
||||
"priority": 1,
|
||||
"bit_depth": "any"
|
||||
},
|
||||
"mp3_320": {
|
||||
"enabled": False,
|
||||
"min_kbps": 280,
|
||||
"max_kbps": 500,
|
||||
"priority": 2
|
||||
},
|
||||
"mp3_256": {
|
||||
"enabled": False,
|
||||
"min_kbps": 200,
|
||||
"max_kbps": 400,
|
||||
"priority": 3
|
||||
},
|
||||
"mp3_192": {
|
||||
"enabled": False,
|
||||
"min_kbps": 150,
|
||||
"max_kbps": 300,
|
||||
"priority": 4
|
||||
},
|
||||
"aac": {
|
||||
"enabled": False,
|
||||
"min_kbps": 128,
|
||||
"max_kbps": 400,
|
||||
"priority": 1.5
|
||||
}
|
||||
},
|
||||
"fallback_enabled": False
|
||||
},
|
||||
"balanced": {
|
||||
"version": 3, "preset": "balanced", "fallback_enabled": True,
|
||||
"ranked_targets": _FLAC_HI_RES_TARGETS + _MP3_TARGETS,
|
||||
"qualities": _quals(flac_en=True, flac_prio=1, mp3_320_en=True, mp3_256_en=True, mp3_192_en=False),
|
||||
"version": 2,
|
||||
"preset": "balanced",
|
||||
"qualities": {
|
||||
"flac": {
|
||||
"enabled": True,
|
||||
"min_kbps": 500,
|
||||
"max_kbps": 10000,
|
||||
"priority": 1,
|
||||
"bit_depth": "any"
|
||||
},
|
||||
"mp3_320": {
|
||||
"enabled": True,
|
||||
"min_kbps": 280,
|
||||
"max_kbps": 500,
|
||||
"priority": 2
|
||||
},
|
||||
"mp3_256": {
|
||||
"enabled": True,
|
||||
"min_kbps": 200,
|
||||
"max_kbps": 400,
|
||||
"priority": 3
|
||||
},
|
||||
"mp3_192": {
|
||||
"enabled": False,
|
||||
"min_kbps": 150,
|
||||
"max_kbps": 300,
|
||||
"priority": 4
|
||||
},
|
||||
"aac": {
|
||||
"enabled": False,
|
||||
"min_kbps": 128,
|
||||
"max_kbps": 400,
|
||||
"priority": 1.5
|
||||
}
|
||||
},
|
||||
"fallback_enabled": True
|
||||
},
|
||||
"space_saver": {
|
||||
"version": 3, "preset": "space_saver", "fallback_enabled": True,
|
||||
"ranked_targets": _MP3_TARGETS,
|
||||
"qualities": _quals(flac_en=False, flac_prio=4, mp3_320_en=True, mp3_256_en=True, mp3_192_en=True,
|
||||
mp3_320_prio=1, mp3_256_prio=2, mp3_192_prio=3, aac_prio=0.5),
|
||||
},
|
||||
"version": 2,
|
||||
"preset": "space_saver",
|
||||
"qualities": {
|
||||
"flac": {
|
||||
"enabled": False,
|
||||
"min_kbps": 500,
|
||||
"max_kbps": 10000,
|
||||
"priority": 4,
|
||||
"bit_depth": "any"
|
||||
},
|
||||
"mp3_320": {
|
||||
"enabled": True,
|
||||
"min_kbps": 280,
|
||||
"max_kbps": 500,
|
||||
"priority": 1
|
||||
},
|
||||
"mp3_256": {
|
||||
"enabled": True,
|
||||
"min_kbps": 200,
|
||||
"max_kbps": 400,
|
||||
"priority": 2
|
||||
},
|
||||
"mp3_192": {
|
||||
"enabled": True,
|
||||
"min_kbps": 150,
|
||||
"max_kbps": 300,
|
||||
"priority": 3
|
||||
},
|
||||
# Space-saver favours small files, where AAC shines — but it
|
||||
# still ships OFF (opt-in). Priority 0.5 puts it above MP3.
|
||||
"aac": {
|
||||
"enabled": False,
|
||||
"min_kbps": 128,
|
||||
"max_kbps": 400,
|
||||
"priority": 0.5
|
||||
}
|
||||
},
|
||||
"fallback_enabled": True
|
||||
}
|
||||
}
|
||||
|
||||
return presets.get(preset_name, presets["balanced"])
|
||||
|
|
@ -10909,40 +10836,23 @@ class MusicDatabase:
|
|||
logger.error(f"Error caching discovery recent album: {e}")
|
||||
return False
|
||||
|
||||
def get_discovery_recent_albums(self, limit: int = 10, source: Optional[str] = None, profile_id: int = 1,
|
||||
exclude_future_years: bool = False) -> List[Dict[str, Any]]:
|
||||
"""Get cached recent albums for discover page, optionally filtered by source.
|
||||
|
||||
exclude_future_years: drop announced-but-unreleased albums dated to a LATER YEAR.
|
||||
Because rows are ordered ``release_date DESC``, future-dated albums otherwise sort to
|
||||
the very top and consume the ``limit`` budget — which is exactly why Fresh Tape / Release
|
||||
Radar starved down to a handful of tracks. Year-level so it's precision-safe across
|
||||
'YYYY' / 'YYYY-MM' / 'YYYY-MM-DD'; same-year future months are left for the caller's precise
|
||||
``is_future_release`` check. NULL/blank dates are kept (treated as released).
|
||||
"""
|
||||
def get_discovery_recent_albums(self, limit: int = 10, source: Optional[str] = None, profile_id: int = 1) -> List[Dict[str, Any]]:
|
||||
"""Get cached recent albums for discover page, optionally filtered by source"""
|
||||
try:
|
||||
with self._get_connection() as conn:
|
||||
cursor = conn.cursor()
|
||||
|
||||
future_clause = ""
|
||||
if exclude_future_years:
|
||||
future_clause = (
|
||||
" AND (release_date IS NULL OR release_date = '' "
|
||||
"OR CAST(substr(release_date, 1, 4) AS INTEGER) "
|
||||
"<= CAST(strftime('%Y','now') AS INTEGER))"
|
||||
)
|
||||
|
||||
if source:
|
||||
cursor.execute(f"""
|
||||
cursor.execute("""
|
||||
SELECT * FROM discovery_recent_albums
|
||||
WHERE source = ? AND profile_id = ?{future_clause}
|
||||
WHERE source = ? AND profile_id = ?
|
||||
ORDER BY release_date DESC
|
||||
LIMIT ?
|
||||
""", (source, profile_id, limit))
|
||||
else:
|
||||
cursor.execute(f"""
|
||||
cursor.execute("""
|
||||
SELECT * FROM discovery_recent_albums
|
||||
WHERE profile_id = ?{future_clause}
|
||||
WHERE profile_id = ?
|
||||
ORDER BY release_date DESC
|
||||
LIMIT ?
|
||||
""", (profile_id, limit))
|
||||
|
|
@ -13165,73 +13075,6 @@ class MusicDatabase:
|
|||
logger.error(f"Error getting discovery pool stats: {e}")
|
||||
return {'matched': 0, 'failed': 0}
|
||||
|
||||
# Wing It Pool: two states on a mirrored track's extra_data. Both key off wing_it_fallback,
|
||||
# which is set by the wing-it stub and SURVIVES a manual fix (update_mirrored_track_extra_data
|
||||
# merges rather than replaces), so the only difference is the manual_match flag:
|
||||
# needs attention : wing_it_fallback=true AND NOT manual_match (unverified guess)
|
||||
# resolved : wing_it_fallback=true AND manual_match=true (user fixed it — incl. fixes
|
||||
# made before this feature existed, since the flag was never wiped)
|
||||
_WING_IT_ATTENTION = ("mpt.extra_data LIKE '%\"wing_it_fallback\": true%' "
|
||||
"AND mpt.extra_data NOT LIKE '%\"manual_match\": true%'")
|
||||
_WING_IT_RESOLVED = ("mpt.extra_data LIKE '%\"wing_it_fallback\": true%' "
|
||||
"AND mpt.extra_data LIKE '%\"manual_match\": true%'")
|
||||
|
||||
def get_wing_it_pool(self, profile_id: int = None, playlist_id: int = None,
|
||||
resolved: bool = False) -> list:
|
||||
"""Get Wing It tracks — the unverified guesses (default) or the ones you've resolved.
|
||||
|
||||
Wing-it tracks are persisted on extra_data with ``wing_it_fallback: true`` (a best-effort
|
||||
stub when a track couldn't match a metadata source). They count as 'discovered', so the
|
||||
Discovery Pool hides them — this is the only surface that lists them. ``resolved=True``
|
||||
returns the ones a manual match has since fixed (carrying the ``was_wing_it`` marker).
|
||||
"""
|
||||
try:
|
||||
conn = self._get_connection()
|
||||
cursor = conn.cursor()
|
||||
where = self._WING_IT_RESOLVED if resolved else self._WING_IT_ATTENTION
|
||||
query = f"""
|
||||
SELECT mpt.id, mpt.track_name, mpt.artist_name, mpt.album_name,
|
||||
mpt.playlist_id, mp.name as playlist_name, mpt.extra_data
|
||||
FROM mirrored_playlist_tracks mpt
|
||||
JOIN mirrored_playlists mp ON mpt.playlist_id = mp.id
|
||||
WHERE {where}
|
||||
"""
|
||||
params = []
|
||||
if playlist_id:
|
||||
query += " AND mpt.playlist_id = ?"
|
||||
params.append(playlist_id)
|
||||
elif profile_id:
|
||||
query += " AND mp.profile_id = ?"
|
||||
params.append(profile_id)
|
||||
query += " ORDER BY mp.name, mpt.track_name"
|
||||
cursor.execute(query, params)
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting wing it pool: {e}")
|
||||
return []
|
||||
|
||||
def get_wing_it_pool_stats(self, profile_id: int = None) -> dict:
|
||||
"""Counts for both Wing It states: unverified (``wing_it``) + resolved (``matched``)."""
|
||||
try:
|
||||
conn = self._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
def _count(where):
|
||||
q = (f"SELECT COUNT(*) as cnt FROM mirrored_playlist_tracks mpt "
|
||||
f"JOIN mirrored_playlists mp ON mpt.playlist_id = mp.id WHERE {where}")
|
||||
params = []
|
||||
if profile_id:
|
||||
q += " AND mp.profile_id = ?"
|
||||
params.append(profile_id)
|
||||
cursor.execute(q, params)
|
||||
return cursor.fetchone()['cnt']
|
||||
|
||||
return {'wing_it': _count(self._WING_IT_ATTENTION),
|
||||
'matched': _count(self._WING_IT_RESOLVED)}
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting wing it pool stats: {e}")
|
||||
return {'wing_it': 0, 'matched': 0}
|
||||
|
||||
# ==================== Retag Tool Methods ====================
|
||||
|
||||
def add_retag_group(self, group_type: str, artist_name: str, album_name: str,
|
||||
|
|
@ -13640,10 +13483,7 @@ class MusicDatabase:
|
|||
download_source, source_track_id, source_track_title, source_filename,
|
||||
acoustid_result, source_artist, origin, origin_context, verification_status))
|
||||
conn.commit()
|
||||
# Return the new row id (truthy on success) so callers can link the
|
||||
# live download task to its library_history row — e.g. the Unverified
|
||||
# review queue needs the id for its play/approve/delete actions.
|
||||
return cursor.lastrowid
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.debug(f"Error adding library history entry: {e}")
|
||||
return False
|
||||
|
|
@ -13799,28 +13639,6 @@ class MusicDatabase:
|
|||
logger.debug(f"Error deleting history rows: {e}")
|
||||
return 0
|
||||
|
||||
def clear_completed_download_history(self) -> int:
|
||||
"""Delete the persisted completed-download history shown on the Downloads
|
||||
page (every event_type='download' row). This also clears the verification
|
||||
review queue, since those unverified/force_imported rows ARE download-history
|
||||
rows — that's intended: 'Clear Completed' empties the list. It only removes
|
||||
HISTORY rows; the actual files and their `tracks` entries are untouched, so
|
||||
nothing in the library is lost — only the 'needs verification' review flags.
|
||||
Returns the number of rows removed."""
|
||||
conn = None
|
||||
try:
|
||||
conn = self._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM library_history WHERE event_type = 'download'")
|
||||
conn.commit()
|
||||
return cursor.rowcount
|
||||
except Exception as e:
|
||||
logger.error("Error clearing completed download history: %s", e)
|
||||
return 0
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
def delete_track_by_file_path(self, file_path):
|
||||
"""Delete a library track row whose stored path matches. Returns count."""
|
||||
if not file_path:
|
||||
|
|
@ -13863,127 +13681,6 @@ class MusicDatabase:
|
|||
logger.error(f"Error querying library history: {e}")
|
||||
return [], 0
|
||||
|
||||
def get_library_history_unverified(self) -> list[dict]:
|
||||
"""Return every library_history row that still needs human confirmation.
|
||||
|
||||
Fetches all rows where verification_status is 'unverified' or
|
||||
'force_imported', ordered newest-first. No row limit — the full
|
||||
set must always be visible on the Downloads → Unverified tab.
|
||||
"""
|
||||
try:
|
||||
conn = self._get_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT * FROM library_history
|
||||
WHERE verification_status IN ('unverified', 'force_imported')
|
||||
ORDER BY created_at DESC
|
||||
""")
|
||||
return [dict(row) for row in cursor.fetchall()]
|
||||
except Exception as e:
|
||||
logger.error("Error querying unverified library history: %s", e)
|
||||
return []
|
||||
|
||||
def reconcile_unverified_history_from_tracks(self) -> int:
|
||||
"""Heal library_history rows stuck at 'unverified' whose underlying file
|
||||
has since been confirmed in the tracks table (AcoustID scan PASS or a
|
||||
human decision). Matches by exact path AND basename — the same physical
|
||||
file keeps its filename across path-form differences (relative vs
|
||||
absolute, library moved/reorganized, different mount), which is why an
|
||||
exact-path-only heal left thousands of already-verified files showing as
|
||||
Unverified (issue #934).
|
||||
|
||||
A basename match is title-guarded: a shared track-number filename
|
||||
("01 - Intro.flac") must NOT heal a different song. When both the history
|
||||
row and the candidate track carry a title they have to agree
|
||||
(alphanumeric-lowercase) — the same guard the AcoustID matcher uses. When
|
||||
a title is missing on either side we can't tell which file the basename
|
||||
refers to, so we only heal if that basename is unambiguous (a single
|
||||
verified candidate). An exact-path match needs no guard.
|
||||
|
||||
Upgrade-only and non-destructive: it only lifts 'unverified' rows to the
|
||||
confirmed status, never downgrades and never deletes. Returns the number
|
||||
of rows healed. Genuinely-unverified rows and orphans (no matching
|
||||
track) are left untouched.
|
||||
"""
|
||||
healed = 0
|
||||
conn = None
|
||||
try:
|
||||
conn = self._get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
def _norm(value):
|
||||
return ''.join(ch for ch in str(value or '').lower() if ch.isalnum())
|
||||
|
||||
# Load the stuck rows first. Cheap early-out when nothing is stuck —
|
||||
# and their paths/basenames scope the tracks scan below, so the
|
||||
# lookup dicts stay proportional to the (small) review queue instead
|
||||
# of the whole library.
|
||||
cursor.execute(
|
||||
"SELECT id, file_path, title FROM library_history "
|
||||
"WHERE verification_status = 'unverified' "
|
||||
"AND file_path IS NOT NULL AND file_path != ''")
|
||||
stuck_rows = cursor.fetchall()
|
||||
if not stuck_rows:
|
||||
return 0
|
||||
needed_paths = {fp for _, fp, _ in stuck_rows if fp}
|
||||
needed_bases = {os.path.basename(fp) for _, fp, _ in stuck_rows if fp}
|
||||
|
||||
rank = {'verified': 1, 'human_verified': 2}
|
||||
by_path = {} # exact path -> status (unambiguous; no title guard)
|
||||
by_base = {} # basename -> list of (norm_title, status)
|
||||
cursor.execute(
|
||||
"SELECT file_path, verification_status, title FROM tracks "
|
||||
"WHERE verification_status IN ('verified', 'human_verified') "
|
||||
"AND file_path IS NOT NULL AND file_path != ''")
|
||||
for fp, st, ttitle in cursor.fetchall():
|
||||
if not fp:
|
||||
continue
|
||||
base = os.path.basename(fp)
|
||||
# Skip verified tracks that can't possibly match a queued row.
|
||||
if fp not in needed_paths and base not in needed_bases:
|
||||
continue
|
||||
if rank.get(st, 0) >= rank.get(by_path.get(fp), 0):
|
||||
by_path[fp] = st
|
||||
if base:
|
||||
by_base.setdefault(base, []).append((_norm(ttitle), st))
|
||||
|
||||
updates = []
|
||||
for rid, fp, rtitle in stuck_rows:
|
||||
target = by_path.get(fp)
|
||||
if not target:
|
||||
want = _norm(rtitle)
|
||||
candidates = by_base.get(os.path.basename(fp or ''), ())
|
||||
best = 0
|
||||
for ttitle, st in candidates:
|
||||
if want and ttitle:
|
||||
# Both titled: must agree.
|
||||
if want != ttitle:
|
||||
continue
|
||||
elif len(candidates) > 1:
|
||||
# Title missing on a side AND the basename collides
|
||||
# across verified files — can't tell which one this
|
||||
# row is, so don't risk healing the wrong song.
|
||||
continue
|
||||
if rank.get(st, 0) >= best:
|
||||
best = rank.get(st, 0)
|
||||
target = st
|
||||
if target:
|
||||
updates.append((target, rid))
|
||||
for status, rid in updates:
|
||||
cursor.execute(
|
||||
"UPDATE library_history SET verification_status = ? WHERE id = ?",
|
||||
(status, rid))
|
||||
healed += 1
|
||||
if healed:
|
||||
conn.commit()
|
||||
logger.info("Reconciled %d unverified history rows from tracks truth", healed)
|
||||
except Exception as e:
|
||||
logger.error("Error reconciling unverified history: %s", e)
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
return healed
|
||||
|
||||
def get_library_history_stats(self):
|
||||
"""Return counts per event_type and per download_source."""
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -1,21 +1,38 @@
|
|||
# soulsync 2.8.2 — `dev` → `main`
|
||||
# soulsync 2.7.8 — `dev` → `main`
|
||||
|
||||
a stability + performance release. the headline is **Spotify reliability** (the Docker boot hang and the "logged out / re-auth won't stick" issues are fixed), a big **performance** win that explains the "slow after update" reports, and **large-library imports** that no longer time out the import page.
|
||||
a feature patch on top of 2.7.7 — playlists can now be put back in order on the server, you can re-wishlist a missed track straight from sync history, plus a couple of reported fixes.
|
||||
|
||||
---
|
||||
|
||||
## what's new
|
||||
|
||||
### 🎧 Spotify reliability
|
||||
- **Docker boot hang fixed (#949 — thanks HellRa1SeR)** — with Spotify set as your primary metadata source, an unreachable Spotify API could block the gunicorn worker during startup, so the container bound port 8008 but never actually served the Web UI. provider auth probes are now deferred during boot (and capped with a timeout), so startup can't hang on a slow Spotify. same guard added for Qobuz / Deezer / Tidal.
|
||||
- **"re-auth didn't stick" fixed** — the OAuth callback wrote your token to one cache while the app read another, so re-authenticating could silently fail validation (and trip an `Address already in use` on the callback port). unified on one token store — re-auth takes effect now.
|
||||
- **Sync to Spotify works** — exporting a mirrored playlist to Spotify now asks for playlist-write permission **once**, on-demand, the first time you use it. your normal Spotify login is untouched, so upgrading never forces a re-auth.
|
||||
### align playlists — server order, not just contents
|
||||
the server-playlist editor only ever cared about *which* tracks were on the server, never their order — and it rendered the server column in the source's order, so a playlist with the right tracks in the wrong sequence read as "in sync" when it wasn't. now it tells the truth:
|
||||
- an **"out of order"** badge appears when the tracks match but the sequence differs (relative order, so missing/extra tracks don't false-flag it), and a **read-only view** shows the server's *actual* order with cover art.
|
||||
- a new **"Align playlists"** action reorders the server playlist to match the source — **Plex** (in-place via moveItem), **Navidrome** (ordered rewrite), and **Jellyfin** (Move endpoint), all of which preserve the playlist's identity/poster. two choices for server-only extras: **mirror source** (drop them) or **keep extras** (park them at the end). it's order-only — it never adds the missing tracks (that's a normal sync's job) and never touches metadata, just reshuffles ids already on the server.
|
||||
|
||||
### ⚡ Performance — the "slow after update" fix
|
||||
- **Password-manager autofill storm fixed (#948 — thanks @nick2000713)** — the real cause of the post-update lag wasn't SoulSync rendering, it was browser password managers (Bitwarden / 1Password / etc.) rebuilding their autofill overlay on **every** DOM change — and SoulSync mutates the DOM constantly (live status, progress bars, countdowns). non-credential fields are now marked so managers skip them (your login fields are left alone). the reporter measured **~110× less main-thread blocking** and ~20 → ~96 FPS.
|
||||
- **Max Performance mode (new)** — Settings → Appearance. one switch kills the worker orbs, particles, all blur/shadows, and every animation/transition, and greys out the individual effect toggles so it's clearly in charge. for software-rendered / no-GPU setups (Docker, remote desktop) where even simple animations cost real CPU.
|
||||
### re-add to wishlist from sync history
|
||||
in the dashboard's **Recent Syncs → details**, the "→ Wishlist" status on an unmatched track is now a button — click it to re-add that exact track to the wishlist with the **same context the sync used** (source playlist, cover art, everything), so it's indistinguishable from the original auto-add. the re-add and the live sync now build the *identical* payload from one shared path, so the cover and album/single classification carry through. wing-it fallback stubs (tracks that couldn't be resolved to real metadata) are correctly shown as **"Unmatched"** and aren't re-addable — matching what the sync itself does.
|
||||
|
||||
### 📥 Large-library imports no longer time out (#947 — thanks @ramonskie)
|
||||
- dropping a whole library into your staging folder used to make the import page scan every file synchronously and blow past the request timeout — so the page never loaded, and every reload re-timed-out. the scan now runs in the **background** with a live **"Scanning N of M…"** progress, and the page fills in automatically when it's done. (auto-import remains the hands-off path for the actual matching.)
|
||||
### fixes
|
||||
- **import search said "Deezer" for Spotify Free users (#922)** — manual album-import told no-auth Spotify users that Deezer was their primary source. the functional source legitimately downgrades to a working fallback (the free path has no album-name search), but the *label* should name what you configured. now it reads "Spotify."
|
||||
- **iTunes albums >50 tracks could still truncate (#918 follow-up)** — the limit=200 fix only helped fresh fetches; albums cached at 50 before the fix kept serving 50 from the persistent cache. now a cached tracklist shorter than the album's known track count self-heals on next load.
|
||||
|
||||
enjoy 🎶
|
||||
### under the hood
|
||||
- `.gitignore` now covers **all** `database/*.db` (+ wal/shm/backup), not just `music_library` — so the video db and any future db can't be committed by accident.
|
||||
|
||||
---
|
||||
|
||||
## a brief recap of what came before
|
||||
2.7.7 was a fix-heavy patch — the metadata-parity fix so downloads tag + path right without a manual reorganize (#915), the listening-recs foundation (#913), jellyfin atomic writes, and a big reported-issue sweep (#905/#908–#912/#914/#916–#918). 2.7.6 exported playlists TO listenbrainz + youtube liked-music sync; 2.7.5 matching & artwork accuracy; 2.7.4 re-identify; 2.7.3 the Quality Upgrade Finder; 2.7.2 playlist-folder mirroring; 2.7.1 download verification; 2.7.0 made multi-user real.
|
||||
|
||||
---
|
||||
|
||||
## tests
|
||||
additive + scoped — the new write paths are their own routes that don't touch the normal sync. new seam/regression suites for the order-status detection (incl. the reported "moved to #2" case + missing/extra false-flag guards), the pure align-rewrite planner (mirror vs keep-extras, never-injects-a-foreign-track, stale-data rejection), the sync re-add payload (a direct parity assertion that the re-add == the live-sync payload, plus the wing-it skip), and the `get_primary_source_label` fix (#922). iTunes self-heal proven against the real persistent-cache shape. relevant suites green; `ruff check` clean.
|
||||
|
||||
## post-merge
|
||||
- [ ] tag `v2.7.8` on `main`
|
||||
- [ ] docker-publish with `version_tag: 2.7.8`
|
||||
- [ ] discord announce (auto-fired by the workflow)
|
||||
- [ ] reply on #922 and the #918 follow-up
|
||||
|
|
|
|||
|
|
@ -1,79 +0,0 @@
|
|||
"""Listening Mix v2 generator (#913) — reads the scan's stored mix into Track records.
|
||||
|
||||
The generator is what lets the Listening Mix appear on the Sync page's SoulSync
|
||||
Discovery tab + flow through the mirror/Auto-Sync pipeline like the other kinds. It
|
||||
must hand back exactly what the scan stored under 'listening_recs_tracks_full', with
|
||||
NO pool hydration, and degrade to empty (never raise) when there's no mix yet.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
from core.personalized.generators.listening_mix import KIND, generate
|
||||
from core.personalized.specs import get_registry
|
||||
from core.personalized.types import PlaylistConfig
|
||||
|
||||
|
||||
class _FakeDB:
|
||||
def __init__(self, meta):
|
||||
self._meta = meta
|
||||
|
||||
def get_metadata(self, key):
|
||||
return self._meta.get(key)
|
||||
|
||||
|
||||
def _full_track(tid, name, artist, source='spotify'):
|
||||
d = {'track_id': tid, 'name': name, 'track_name': name, 'artist_name': artist,
|
||||
'album_name': f'{name} - Album', 'album_cover_url': f'http://cdn/{tid}.jpg',
|
||||
'duration_ms': 200000, 'track_data_json': {'id': tid}, 'source': source,
|
||||
f'{source}_track_id': tid, '_seed_artist': artist}
|
||||
return d
|
||||
|
||||
|
||||
def _deps(meta):
|
||||
return SimpleNamespace(database=_FakeDB(meta))
|
||||
|
||||
|
||||
def test_generate_reads_stored_full_tracks():
|
||||
rows = [_full_track('s1', 'One', 'Arcangel'), _full_track('s2', 'Two', 'Maluma')]
|
||||
out = generate(_deps({'listening_recs_tracks_full': json.dumps(rows)}), '', PlaylistConfig(limit=50))
|
||||
assert [t.track_name for t in out] == ['One', 'Two']
|
||||
assert [t.artist_name for t in out] == ['Arcangel', 'Maluma']
|
||||
assert out[0].spotify_track_id == 's1' # source id carried -> mirror can sync
|
||||
assert out[0].album_cover_url == 'http://cdn/s1.jpg'
|
||||
assert out[0].track_data_json == {'id': 's1'} # full payload preserved for download
|
||||
|
||||
|
||||
def test_generate_respects_limit():
|
||||
rows = [_full_track(f's{i}', f'T{i}', f'A{i}') for i in range(10)]
|
||||
out = generate(_deps({'listening_recs_tracks_full': json.dumps(rows)}), '', PlaylistConfig(limit=3))
|
||||
assert len(out) == 3
|
||||
|
||||
|
||||
def test_generate_empty_when_no_mix_yet():
|
||||
assert generate(_deps({}), '', PlaylistConfig(limit=50)) == []
|
||||
assert generate(_deps({'listening_recs_tracks_full': ''}), '', PlaylistConfig(limit=50)) == []
|
||||
|
||||
|
||||
def test_generate_tolerates_bad_json_and_non_dict_rows():
|
||||
assert generate(_deps({'listening_recs_tracks_full': 'not json'}), '', PlaylistConfig(limit=50)) == []
|
||||
rows = json.dumps([_full_track('s1', 'Good', 'A'), 'garbage', None])
|
||||
out = generate(_deps({'listening_recs_tracks_full': rows}), '', PlaylistConfig(limit=50))
|
||||
assert [t.track_name for t in out] == ['Good'] # junk rows skipped, valid kept
|
||||
|
||||
|
||||
def test_generator_is_registered():
|
||||
# importing the package must register the kind so the manager + Sync tab discover it.
|
||||
import core.personalized.generators # noqa: F401
|
||||
spec = get_registry().get(KIND)
|
||||
assert spec is not None and spec.requires_variant is False
|
||||
assert spec.display_name('') == 'Your Listening Mix'
|
||||
|
||||
|
||||
def test_generate_supports_deezer_id_tracks():
|
||||
# iTunes/MusicBrainz users get Deezer-sourced tracks -> deezer_track_id must carry.
|
||||
rows = [_full_track('d9', 'Song', 'Artist', source='deezer')]
|
||||
out = generate(_deps({'listening_recs_tracks_full': json.dumps(rows)}), '', PlaylistConfig(limit=50))
|
||||
assert out[0].deezer_track_id == 'd9' and out[0].spotify_track_id is None
|
||||
|
|
@ -4,87 +4,14 @@ from __future__ import annotations
|
|||
|
||||
from core.discovery.listening_recommendations import (
|
||||
aggregate_candidate_tracks,
|
||||
build_recency_weighted_seeds,
|
||||
choose_mix_fetch_source,
|
||||
names_match,
|
||||
rank_recommended_artists,
|
||||
similarity_from_rank,
|
||||
to_mix_track,
|
||||
)
|
||||
|
||||
|
||||
# ── choose_mix_fetch_source (universal Deezer fallback) ───────────────────────
|
||||
def test_fetch_source_uses_active_when_it_can_fetch():
|
||||
assert choose_mix_fetch_source("spotify", True) == "spotify"
|
||||
assert choose_mix_fetch_source("deezer", True) == "deezer"
|
||||
|
||||
|
||||
def test_fetch_source_falls_back_to_deezer_for_other_sources():
|
||||
# iTunes / Discogs / MusicBrainz can't fetch top tracks -> Deezer public.
|
||||
assert choose_mix_fetch_source("itunes", False) == "deezer"
|
||||
assert choose_mix_fetch_source("musicbrainz", False) == "deezer"
|
||||
assert choose_mix_fetch_source("discogs", False) == "deezer"
|
||||
|
||||
|
||||
def test_fetch_source_falls_back_when_active_client_unavailable():
|
||||
# Active source is Spotify but its client isn't usable (not authed) -> Deezer.
|
||||
assert choose_mix_fetch_source("spotify", False) == "deezer"
|
||||
assert choose_mix_fetch_source(None, False) == "deezer"
|
||||
|
||||
|
||||
# ── names_match (guards the top-tracks fetch against wrong-artist results) ─────
|
||||
def test_names_match_ignores_case_and_punctuation():
|
||||
assert names_match("Tyler, The Creator", "Tyler The Creator")
|
||||
assert names_match("BEYONCÉ", "beyoncé")
|
||||
assert names_match("AC/DC", "ac dc")
|
||||
|
||||
|
||||
def test_names_match_rejects_near_misses_and_empty():
|
||||
assert not names_match("Drake", "Drake Bell")
|
||||
assert not names_match("", "anything")
|
||||
assert not names_match("X", None)
|
||||
|
||||
|
||||
def _seed(name, weight=1.0):
|
||||
return {"name": name, "weight": weight}
|
||||
|
||||
|
||||
# ── similarity_from_rank (1=closest .. 10=farthest -> 1.0 .. 0.1) ─────────────
|
||||
def test_similarity_from_rank_decays_over_documented_range():
|
||||
assert similarity_from_rank(1) == 1.0
|
||||
assert similarity_from_rank(5) == 0.6
|
||||
assert similarity_from_rank(10) == 0.1
|
||||
|
||||
|
||||
def test_similarity_from_rank_clamps_and_defaults():
|
||||
assert similarity_from_rank(0) == 1.0 # <=1 -> full weight
|
||||
assert similarity_from_rank(50) == 0.1 # beyond range -> floor
|
||||
assert similarity_from_rank(None) == 1.0 # missing -> full weight (no rank info)
|
||||
assert similarity_from_rank("nan") == 1.0
|
||||
|
||||
|
||||
# ── build_recency_weighted_seeds (lifetime + factor*recent) ───────────────────
|
||||
def test_recency_boost_reorders_toward_current_taste():
|
||||
# Old-fav has more lifetime plays, but New-fav dominates recently.
|
||||
lifetime = [{"name": "OldFav", "play_count": 100}, {"name": "NewFav", "play_count": 40}]
|
||||
recent = {"newfav": 60}
|
||||
seeds = build_recency_weighted_seeds(lifetime, recent, recency_factor=1.5)
|
||||
by = {s["name"]: s["weight"] for s in seeds}
|
||||
assert by["OldFav"] == 100.0 # no recent plays -> unchanged
|
||||
assert by["NewFav"] == 40 + 1.5 * 60 # 130 -> now outranks OldFav
|
||||
|
||||
|
||||
def test_recency_factor_zero_is_pure_lifetime():
|
||||
seeds = build_recency_weighted_seeds(
|
||||
[{"name": "A", "play_count": 7}], {"a": 99}, recency_factor=0)
|
||||
assert seeds == [{"name": "A", "weight": 7.0}]
|
||||
|
||||
|
||||
def test_recency_seeds_skip_blank_names_and_tolerate_missing_recent():
|
||||
seeds = build_recency_weighted_seeds([{"name": ""}, {"name": "A", "play_count": 3}])
|
||||
assert seeds == [{"name": "A", "weight": 3.0}]
|
||||
|
||||
|
||||
# ── rank_recommended_artists ─────────────────────────────────────────────────
|
||||
def test_consensus_outranks_single_endorsement():
|
||||
# 'Common' is similar to BOTH seeds; 'Solo' to one. Equal weights/scores.
|
||||
|
|
@ -190,46 +117,6 @@ def test_aggregate_skips_artist_with_no_tracks():
|
|||
assert [t["name"] for t in out] == ["only"] # sim-b had no tracks -> skipped
|
||||
|
||||
|
||||
# ── to_mix_track (source top-track dict -> Discover compact-row dict) ─────────
|
||||
def _sp_track(tid="t1", name="Song", artist="Artist", album="Album", cover="http://cdn/c.jpg"):
|
||||
return {"id": tid, "name": name, "artists": [{"name": artist}],
|
||||
"album": {"name": album, "images": ([{"url": cover}] if cover else [])},
|
||||
"duration_ms": 210000, "popularity": 55}
|
||||
|
||||
|
||||
def test_to_mix_track_shapes_render_fields():
|
||||
out = to_mix_track(_sp_track(), "spotify")
|
||||
assert out["track_name"] == "Song" and out["artist_name"] == "Artist"
|
||||
assert out["album_name"] == "Album" and out["album_cover_url"] == "http://cdn/c.jpg"
|
||||
assert out["duration_ms"] == 210000
|
||||
assert out["spotify_track_id"] == "t1" and out["track_id"] == "t1"
|
||||
assert out["track_data_json"]["id"] == "t1" # full payload kept for sync
|
||||
assert out["name"] == "Song" # kept for aggregate dedup
|
||||
|
||||
|
||||
def test_to_mix_track_source_id_field_per_source():
|
||||
assert to_mix_track(_sp_track(), "deezer")["deezer_track_id"] == "t1"
|
||||
assert to_mix_track(_sp_track(), "itunes")["itunes_track_id"] == "t1"
|
||||
|
||||
|
||||
def test_to_mix_track_rejects_unusable_and_tolerates_missing_album():
|
||||
assert to_mix_track({"id": "x"}, "spotify") is None # no title
|
||||
assert to_mix_track({"name": "y"}, "spotify") is None # no id
|
||||
assert to_mix_track("garbage", "spotify") is None
|
||||
bare = to_mix_track({"id": "z", "name": "Z"}, "spotify") # no artists/album
|
||||
assert bare["artist_name"] == "" and bare["album_cover_url"] is None
|
||||
|
||||
|
||||
def test_to_mix_track_feeds_aggregate_end_to_end():
|
||||
# The real pipeline: shape source tracks, then aggregate by recommended artist.
|
||||
recs = _recs("A") # recommends 'sim-A'
|
||||
shaped = [to_mix_track(_sp_track(tid="1", name="One", artist="sim-A"), "spotify"),
|
||||
to_mix_track(_sp_track(tid="2", name="Two", artist="sim-A"), "spotify")]
|
||||
out = aggregate_candidate_tracks(recs, {"sim-a": shaped}, per_artist=5, limit=10)
|
||||
assert [t["track_name"] for t in out] == ["One", "Two"]
|
||||
assert out[0]["spotify_track_id"] == "1"
|
||||
|
||||
|
||||
# ── group_similars_by_seed (id->name join) ───────────────────────────────────
|
||||
from dataclasses import dataclass as _dc # noqa: E402
|
||||
|
||||
|
|
@ -280,37 +167,3 @@ def test_group_then_rank_end_to_end():
|
|||
ranked = rank_recommended_artists(seeds, grouped, owned_artist_names={"solo"})
|
||||
assert ranked[0].name == "Common" and ranked[0].seed_count == 2
|
||||
assert all(r.name != "Solo" for r in ranked) # owned excluded
|
||||
|
||||
|
||||
# ── rank-aware grouping (similarity_rank -> score) ────────────────────────────
|
||||
@_dc
|
||||
class _RankRow:
|
||||
source_artist_id: str
|
||||
similar_artist_name: str
|
||||
similarity_rank: int
|
||||
|
||||
|
||||
def test_group_with_rank_attr_carries_similarity_score():
|
||||
seeds = [_seed("A")]
|
||||
rows = [_RankRow("ia", "Close", 1), _RankRow("ia", "Far", 10)]
|
||||
out = group_similars_by_seed(seeds, rows, {"ia": "A"}, rank_attr="similarity_rank")
|
||||
by = {e["name"]: e["score"] for e in out["a"]}
|
||||
assert by == {"Close": 1.0, "Far": 0.1}
|
||||
|
||||
|
||||
def test_group_without_rank_attr_is_scoreless_backcompat():
|
||||
seeds = [_seed("A")]
|
||||
rows = [_RankRow("ia", "X", 3)]
|
||||
out = group_similars_by_seed(seeds, rows, {"ia": "A"})
|
||||
assert out["a"] == [{"name": "X"}] # no score key -> original behavior
|
||||
|
||||
|
||||
def test_rank_threading_changes_winner_within_a_seed():
|
||||
# The production fix: a CLOSER match (rank 1) on a heavy seed beats a far match (rank 9),
|
||||
# even though both come from the same seed. Without rank threading they'd tie.
|
||||
seeds = [_seed("Fav", weight=10)]
|
||||
rows = [_RankRow("if", "Close", 1), _RankRow("if", "Far", 9)]
|
||||
grouped = group_similars_by_seed(seeds, rows, {"if": "Fav"}, rank_attr="similarity_rank")
|
||||
ranked = rank_recommended_artists(seeds, grouped)
|
||||
assert [r.name for r in ranked] == ["Close", "Far"]
|
||||
assert ranked[0].score > ranked[1].score
|
||||
|
|
|
|||
|
|
@ -1,57 +0,0 @@
|
|||
"""Fresh Tape / Release Radar candidate fetch must not be starved by future albums.
|
||||
|
||||
Regression: get_discovery_recent_albums orders release_date DESC, so announced-but-
|
||||
unreleased albums dated to a LATER YEAR sort to the very top and consumed the album
|
||||
budget before the scanner's in-loop is_future_release skip ran — leaving only a handful
|
||||
of released albums to draw tracks from (the reported "Fresh Tape only has 5-10 tracks").
|
||||
exclude_future_years drops next-year albums at the query so released ones fill the budget.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from database.music_database import MusicDatabase
|
||||
|
||||
|
||||
def _insert(db, **kw):
|
||||
with db._get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO discovery_recent_albums
|
||||
(album_spotify_id, album_name, artist_name, artist_spotify_id,
|
||||
album_cover_url, release_date, album_type, source, profile_id)
|
||||
VALUES (?,?,?,?,?,?,?,?,?)""",
|
||||
(kw['id'], kw['name'], kw['artist'], kw['id'] + '_a', '',
|
||||
kw['release_date'], kw.get('album_type', 'album'), kw.get('source', 'spotify'), 1))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def test_future_year_albums_excluded_released_kept(tmp_path):
|
||||
db = MusicDatabase(database_path=str(tmp_path / "t.db"))
|
||||
this_year = datetime.now().year
|
||||
next_year = this_year + 1
|
||||
_insert(db, id='past1', name='Released A', artist='X', release_date=f'{this_year - 1}-03-01')
|
||||
_insert(db, id='past2', name='Released B', artist='Y', release_date=f'{this_year}-01-15')
|
||||
_insert(db, id='fut1', name='Announced', artist='Z', release_date=f'{next_year}-02-01')
|
||||
_insert(db, id='fut2', name='Year Only Future', artist='W', release_date=str(next_year))
|
||||
_insert(db, id='blank', name='Unknown Date', artist='Q', release_date='')
|
||||
|
||||
names_all = {a['album_name'] for a in db.get_discovery_recent_albums(limit=50, source='spotify')}
|
||||
assert 'Announced' in names_all # without the flag, futures are present (and sort first)
|
||||
|
||||
filtered = db.get_discovery_recent_albums(limit=50, source='spotify', exclude_future_years=True)
|
||||
names = {a['album_name'] for a in filtered}
|
||||
assert 'Announced' not in names # next-year album dropped
|
||||
assert 'Year Only Future' not in names # YYYY-only future dropped
|
||||
assert {'Released A', 'Released B'} <= names # released kept
|
||||
assert 'Unknown Date' in names # blank date kept (treated as released)
|
||||
|
||||
|
||||
def test_future_filter_does_not_over_trim_when_all_released(tmp_path):
|
||||
db = MusicDatabase(database_path=str(tmp_path / "t2.db"))
|
||||
this_year = datetime.now().year
|
||||
for i in range(8):
|
||||
_insert(db, id=f'r{i}', name=f'Album {i}', artist=f'A{i}',
|
||||
release_date=f'{this_year}-0{(i % 9) + 1}-01')
|
||||
filtered = db.get_discovery_recent_albums(limit=300, source='spotify', exclude_future_years=True)
|
||||
assert len(filtered) == 8 # every released album survives, budget honored
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
"""order_candidates — the candidate sort used by attempt_download_with_candidates.
|
||||
|
||||
Default (priority mode) sorts confidence-first, then peer quality — today's
|
||||
behaviour, locked here as a regression guard. quality_first=True (best-quality
|
||||
mode) makes the user's profile quality rank dominate, with confidence as the
|
||||
tiebreaker. Both keep correctly-matched candidates; ordering only changes which
|
||||
is tried first.
|
||||
"""
|
||||
|
||||
from core.downloads.candidates import order_candidates
|
||||
from core.quality.model import AudioQuality, QualityTarget
|
||||
|
||||
|
||||
class _Cand:
|
||||
def __init__(self, name, aq, confidence, quality_score=0,
|
||||
upload_speed=0, queue_length=0, free_upload_slots=0, size=0):
|
||||
self.name = name
|
||||
self.audio_quality = aq
|
||||
self.confidence = confidence
|
||||
self.quality_score = quality_score
|
||||
self.upload_speed = upload_speed
|
||||
self.queue_length = queue_length
|
||||
self.free_upload_slots = free_upload_slots
|
||||
self.size = size
|
||||
|
||||
|
||||
FLAC_HI = AudioQuality('flac', sample_rate=96000, bit_depth=24)
|
||||
FLAC_CD = AudioQuality('flac', sample_rate=44100, bit_depth=16)
|
||||
|
||||
TARGETS = [
|
||||
QualityTarget(label='FLAC 24', format='flac', bit_depth=24, min_sample_rate=96000),
|
||||
QualityTarget(label='FLAC 16', format='flac', bit_depth=16),
|
||||
]
|
||||
|
||||
|
||||
def test_priority_mode_is_confidence_first():
|
||||
hi = _Cand('hi-flac', FLAC_HI, confidence=0.80)
|
||||
lo = _Cand('cd-flac', FLAC_CD, confidence=0.95)
|
||||
|
||||
ordered = order_candidates([hi, lo], quality_first=False, targets=TARGETS)
|
||||
|
||||
assert [c.name for c in ordered] == ['cd-flac', 'hi-flac'] # higher confidence wins
|
||||
|
||||
|
||||
def test_quality_first_lets_better_quality_win_over_confidence():
|
||||
hi = _Cand('hi-flac', FLAC_HI, confidence=0.80)
|
||||
lo = _Cand('cd-flac', FLAC_CD, confidence=0.95)
|
||||
|
||||
ordered = order_candidates([hi, lo], quality_first=True, targets=TARGETS)
|
||||
|
||||
assert [c.name for c in ordered] == ['hi-flac', 'cd-flac'] # 24-bit beats higher-confidence 16-bit
|
||||
|
||||
|
||||
def test_quality_first_uses_confidence_as_tiebreak_within_same_quality():
|
||||
a = _Cand('a', FLAC_HI, confidence=0.70)
|
||||
b = _Cand('b', FLAC_HI, confidence=0.90)
|
||||
|
||||
ordered = order_candidates([a, b], quality_first=True, targets=TARGETS)
|
||||
|
||||
assert [c.name for c in ordered] == ['b', 'a'] # same quality → confidence breaks tie
|
||||
|
||||
|
||||
def test_quality_first_ranks_unmatched_quality_last():
|
||||
matched = _Cand('matched', FLAC_CD, confidence=0.50)
|
||||
off_list = _Cand('off', AudioQuality('mp3', bitrate=320), confidence=0.99)
|
||||
|
||||
ordered = order_candidates([off_list, matched], quality_first=True, targets=TARGETS)
|
||||
|
||||
assert [c.name for c in ordered] == ['matched', 'off'] # off-list sorts last despite high confidence
|
||||
|
|
@ -176,30 +176,15 @@ def test_clear_completed_drops_empty_batches():
|
|||
assert download_batches['b3']['queue'] == ['t3']
|
||||
|
||||
|
||||
def test_clear_completed_keeps_terminal_tasks_in_active_batch():
|
||||
"""A completed/failed task in a batch that still has active work is KEPT, so
|
||||
the Downloads page shows it for the whole run (the 5-min Clean Completed
|
||||
automation must not yank terminal tasks out from under a live batch)."""
|
||||
def test_clear_completed_prunes_terminal_task_ids_from_batch_queues():
|
||||
"""Batch with mix of terminal + active tasks gets queue trimmed, not deleted."""
|
||||
download_tasks['t1'] = {'status': 'completed'}
|
||||
download_tasks['t2'] = {'status': 'downloading'} # batch still active
|
||||
download_tasks['t2'] = {'status': 'downloading'}
|
||||
download_batches['b1'] = {'queue': ['t1', 't2']}
|
||||
|
||||
cleared = cancel.clear_completed_local()
|
||||
assert cleared == 0 # nothing removed — batch is live
|
||||
cancel.clear_completed_local()
|
||||
assert 'b1' in download_batches
|
||||
assert download_batches['b1']['queue'] == ['t1', 't2'] # queue intact
|
||||
|
||||
|
||||
def test_clear_completed_clears_terminal_tasks_once_batch_is_finished():
|
||||
"""When every task in a batch is terminal, the batch is done — its tasks are
|
||||
cleared and the empty batch dropped."""
|
||||
download_tasks['t1'] = {'status': 'completed'}
|
||||
download_tasks['t2'] = {'status': 'failed'}
|
||||
download_batches['b1'] = {'queue': ['t1', 't2']}
|
||||
|
||||
cleared = cancel.clear_completed_local()
|
||||
assert cleared == 2
|
||||
assert 'b1' not in download_batches
|
||||
assert download_batches['b1']['queue'] == ['t2']
|
||||
|
||||
|
||||
def test_clear_completed_drops_batch_locks_for_deleted_batches():
|
||||
|
|
|
|||
|
|
@ -480,33 +480,11 @@ def test_stuck_searching_task_forced_to_not_found():
|
|||
assert download_tasks['t1']['status'] == 'not_found'
|
||||
|
||||
|
||||
def test_stuck_post_processing_without_file_forced_to_failed():
|
||||
"""Task stuck in post_processing past the timeout with NO output file must
|
||||
be marked FAILED, not falsely completed — otherwise it shows as a phantom
|
||||
download with nothing on disk (big batches back up post-processing)."""
|
||||
download_tasks['t1'] = {
|
||||
'status': 'post_processing', 'track_info': {'name': 'X'},
|
||||
'status_change_time': 0, # ancient → past any timeout
|
||||
}
|
||||
download_batches['b1'] = {
|
||||
'queue': ['t1'], 'queue_index': 1, 'active_count': 1,
|
||||
'max_concurrent': 1, 'permanently_failed_tracks': [],
|
||||
'cancelled_tracks': set(),
|
||||
}
|
||||
deps, _ = _build_deps()
|
||||
lc.on_download_completed('b1', 't1', True, deps)
|
||||
assert download_tasks['t1']['status'] == 'failed'
|
||||
|
||||
|
||||
def test_stuck_post_processing_with_existing_file_completed(tmp_path):
|
||||
"""If the import really finished (final_file_path exists on disk), a stuck
|
||||
post_processing task is legitimately completed."""
|
||||
real_file = tmp_path / 'track.flac'
|
||||
real_file.write_bytes(b'x')
|
||||
def test_stuck_post_processing_task_forced_to_completed():
|
||||
"""Task post_processing > 5min gets forced to completed."""
|
||||
download_tasks['t1'] = {
|
||||
'status': 'post_processing', 'track_info': {'name': 'X'},
|
||||
'status_change_time': 0,
|
||||
'final_file_path': str(real_file),
|
||||
}
|
||||
download_batches['b1'] = {
|
||||
'queue': ['t1'], 'queue_index': 1, 'active_count': 1,
|
||||
|
|
|
|||
|
|
@ -117,31 +117,6 @@ def test_stream_processed_task_returns_early():
|
|||
assert rec.calls == []
|
||||
|
||||
|
||||
def test_requeued_task_bails_without_marking_failed():
|
||||
"""RACE GUARD: the monitor sets status -> 'post_processing' and submits this
|
||||
worker. If, before the worker runs, the browser-poll post-processor
|
||||
quarantines the file and requeues the next-best candidate (status ->
|
||||
'searching', username/filename cleared), this worker must bail WITHOUT
|
||||
marking failed or notifying batch completion. Otherwise it clobbers the
|
||||
in-flight retry with a false 'missing file or source information' failure
|
||||
while a parallel attempt imports the song."""
|
||||
download_tasks['t1'] = {'status': 'searching', 'track_info': {}}
|
||||
deps, rec = _build_deps()
|
||||
pp.run_post_processing_worker('t1', 'b1', deps)
|
||||
assert download_tasks['t1']['status'] == 'searching' # untouched
|
||||
assert 'error_message' not in download_tasks['t1']
|
||||
assert not any(c[0] == 'on_complete' for c in rec.calls)
|
||||
|
||||
|
||||
def test_queued_task_bails_without_marking_failed():
|
||||
"""Same race guard for a task another path reset to 'queued'."""
|
||||
download_tasks['t1'] = {'status': 'queued', 'track_info': {}}
|
||||
deps, rec = _build_deps()
|
||||
pp.run_post_processing_worker('t1', 'b1', deps)
|
||||
assert download_tasks['t1']['status'] == 'queued'
|
||||
assert not any(c[0] == 'on_complete' for c in rec.calls)
|
||||
|
||||
|
||||
def test_missing_filename_marks_failed_and_calls_on_complete():
|
||||
download_tasks['t1'] = {'status': 'post_processing', 'username': 'u1', 'track_info': {}}
|
||||
deps, rec = _build_deps()
|
||||
|
|
@ -170,12 +145,7 @@ def test_file_not_found_after_retries_marks_failed(monkeypatch):
|
|||
deps, rec = _build_deps()
|
||||
pp.run_post_processing_worker('t1', 'b1', deps)
|
||||
assert download_tasks['t1']['status'] == 'failed'
|
||||
# Actionable failure: names the folder searched + the two real causes, so a
|
||||
# standalone user with a path mismatch can self-diagnose (Discord: Shdjfgatdif).
|
||||
msg = download_tasks['t1']['error_message']
|
||||
assert './downloads' in msg # the folder we actually searched
|
||||
assert "download path doesn't match slskd" in msg # the config-mismatch hint
|
||||
assert 'song.flac' in msg # the file slskd reported
|
||||
assert 'File not found on disk' in download_tasks['t1']['error_message']
|
||||
assert ('on_complete', ('b1', 't1', False), {}) in rec.calls
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -667,44 +667,15 @@ def test_unified_response_includes_batch_summaries():
|
|||
assert bs['queued'] == 1
|
||||
|
||||
|
||||
def test_unified_response_returns_all_live_tasks_even_past_limit():
|
||||
"""`limit` bounds the persistent-history tail, NOT live in-memory tasks.
|
||||
|
||||
Live tasks are already bounded by the 5-min cleanup, and the Downloads page
|
||||
filters them client-side per tab — truncating them (active-first) starved
|
||||
completed/failed/unverified rows out of the response during a busy batch so
|
||||
they never showed until the batch drained.
|
||||
"""
|
||||
def test_unified_response_respects_limit():
|
||||
deps, _ = _build_deps()
|
||||
for i in range(20):
|
||||
download_tasks[f't{i}'] = {
|
||||
'track_index': i, 'status': 'completed', 'track_info': {},
|
||||
}
|
||||
out = st.build_unified_downloads_response(5, deps)
|
||||
assert len(out['downloads']) == 20 # all live tasks returned
|
||||
assert out['total'] == 20
|
||||
|
||||
|
||||
def test_unified_response_does_not_truncate_terminal_tasks_behind_active():
|
||||
"""A busy batch (many queued/active tasks) must not push completed/failed
|
||||
rows off the end of the response — they're what the Completed/Failed tabs
|
||||
show during the run."""
|
||||
deps, _ = _build_deps()
|
||||
for i in range(120):
|
||||
download_tasks[f'q{i}'] = {
|
||||
'track_index': i, 'status': 'queued',
|
||||
'track_info': {'name': f'Q{i}'}, 'status_change_time': i,
|
||||
}
|
||||
download_tasks['done'] = {
|
||||
'status': 'completed', 'track_info': {'name': 'DONE'}, 'status_change_time': 999,
|
||||
}
|
||||
download_tasks['fail'] = {
|
||||
'status': 'failed', 'track_info': {'name': 'FAIL'}, 'status_change_time': 999,
|
||||
}
|
||||
out = st.build_unified_downloads_response(100, deps)
|
||||
titles = {d['title'] for d in out['downloads']}
|
||||
assert 'DONE' in titles
|
||||
assert 'FAIL' in titles
|
||||
assert len(out['downloads']) == 5
|
||||
assert out['total'] == 20 # total still reflects all
|
||||
|
||||
|
||||
def test_unified_response_includes_persistent_download_history():
|
||||
|
|
@ -792,68 +763,6 @@ def test_unified_response_caps_persistent_history_tail():
|
|||
assert out['total'] == 50
|
||||
|
||||
|
||||
def test_unverified_history_always_loaded_even_when_at_limit():
|
||||
"""Unverified entries must appear even during a large batch that fills the limit."""
|
||||
for i in range(200):
|
||||
download_tasks[f'live-{i}'] = {
|
||||
'track_index': i,
|
||||
'status': 'completed',
|
||||
'track_info': {'name': f'Track {i}', 'artist': f'Artist {i}', 'album': 'Album'},
|
||||
'verification_status': 'verified',
|
||||
}
|
||||
|
||||
deps, _ = _build_deps(
|
||||
persistent_history=lambda limit: [],
|
||||
)
|
||||
deps.get_unverified_download_history = lambda: [
|
||||
{
|
||||
'id': 999,
|
||||
'title': 'Old Unverified',
|
||||
'artist_name': 'Forgotten Artist',
|
||||
'album_name': 'Old Album',
|
||||
'created_at': '2026-01-01 00:00:00',
|
||||
'verification_status': 'unverified',
|
||||
}
|
||||
]
|
||||
|
||||
out = st.build_unified_downloads_response(200, deps)
|
||||
|
||||
titles = {d['title'] for d in out['downloads']}
|
||||
assert 'Old Unverified' in titles, "historical unverified entry must appear regardless of limit"
|
||||
|
||||
unverified = [d for d in out['downloads'] if d.get('verification_status') == 'unverified']
|
||||
assert len(unverified) == 1
|
||||
assert unverified[0]['task_id'] == 'history-999'
|
||||
assert unverified[0]['is_persistent_history'] is True
|
||||
|
||||
|
||||
def test_unverified_history_deduped_against_live_task():
|
||||
"""A live task that's still unverified must not appear twice."""
|
||||
download_tasks['live-unv'] = {
|
||||
'track_index': 0,
|
||||
'status': 'completed',
|
||||
'track_info': {'name': 'Live Unverified', 'artist': 'Artsy', 'album': 'Alb'},
|
||||
'verification_status': 'unverified',
|
||||
}
|
||||
|
||||
deps, _ = _build_deps()
|
||||
deps.get_unverified_download_history = lambda: [
|
||||
{
|
||||
'id': 77,
|
||||
'title': 'Live Unverified',
|
||||
'artist_name': 'Artsy',
|
||||
'album_name': 'Alb',
|
||||
'created_at': '2026-06-15 10:00:00',
|
||||
'verification_status': 'unverified',
|
||||
}
|
||||
]
|
||||
|
||||
out = st.build_unified_downloads_response(200, deps)
|
||||
matches = [d for d in out['downloads'] if d['title'] == 'Live Unverified']
|
||||
assert len(matches) == 1, "same track must not be duplicated"
|
||||
assert matches[0]['is_persistent_history'] is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# #836 — a rejected slskd transfer must not hang the task at 'downloading'
|
||||
# forever. The monitor normally retries; when it can't make progress, a backstop
|
||||
|
|
|
|||
|
|
@ -352,7 +352,7 @@ def test_first_query_success_returns_after_storing_source():
|
|||
_seed_task()
|
||||
rec = _Recorder()
|
||||
|
||||
def _attempt_success(task_id, candidates, track, batch_id, **kwargs):
|
||||
def _attempt_success(task_id, candidates, track, batch_id):
|
||||
download_tasks[task_id]['filename'] = 'song.flac'
|
||||
download_tasks[task_id]['username'] = 'u1'
|
||||
return True
|
||||
|
|
@ -478,7 +478,7 @@ def test_hybrid_fallback_tries_secondary_sources():
|
|||
},
|
||||
)
|
||||
|
||||
def _attempt_yt_success(task_id, candidates, track, batch_id, **kwargs):
|
||||
def _attempt_yt_success(task_id, candidates, track, batch_id):
|
||||
return True
|
||||
|
||||
deps, _ = _build_deps(
|
||||
|
|
@ -528,7 +528,7 @@ def test_quarantine_retry_tries_cached_candidates_without_searching():
|
|||
)
|
||||
attempted = []
|
||||
|
||||
def _attempt(task_id, candidates, track, batch_id, **kwargs):
|
||||
def _attempt(task_id, candidates, track, batch_id):
|
||||
attempted.append([getattr(c, 'filename', None) for c in candidates])
|
||||
return True
|
||||
|
||||
|
|
@ -555,7 +555,7 @@ def test_quarantine_retry_skips_cached_from_exhausted_source():
|
|||
)
|
||||
attempted = []
|
||||
|
||||
def _attempt(task_id, candidates, track, batch_id, **kwargs):
|
||||
def _attempt(task_id, candidates, track, batch_id):
|
||||
attempted.append([getattr(c, 'username', None) for c in candidates])
|
||||
return True
|
||||
|
||||
|
|
|
|||
|
|
@ -396,56 +396,3 @@ def test_real_soulseek_path_still_basenamed(tmp_path):
|
|||
str(downloads), r'shared\Billy Ocean\Very Best Of\01 - Suddenly.flac',
|
||||
)
|
||||
assert found == str(target)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unbalanced bracket — slskd REPORTS "[34 - Title.flac" but SAVES the file as
|
||||
# "34 - Title.flac" (it sanitises the leading '['). The normaliser's old combined
|
||||
# bracket-strip r'[\[\(].*?[\]\)]' matched from that lone '[' all the way to the
|
||||
# next ')', eating the whole title and collapsing the search target to just "flac"
|
||||
# → 0.40 fuzzy score → "File not found on disk" despite the file sitting right
|
||||
# there. (Discord: Shdjfgatdif — "You & Me (Flume Remix)".)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_finds_file_when_slskd_strips_a_leading_bracket(tmp_path):
|
||||
downloads = tmp_path / 'downloads'
|
||||
# On disk: no leading '['. API filename (slskd-reported): has the '['.
|
||||
target = downloads / 'Disclosure' / '34 - You & Me (Flume Remix).flac'
|
||||
_touch(target)
|
||||
|
||||
found, location = find_completed_audio_file(
|
||||
str(downloads), r'Music\Disclosure\[34 - You & Me (Flume Remix).flac',
|
||||
)
|
||||
|
||||
assert found == str(target), \
|
||||
'the lone "[" used to collapse the target to "flac" and miss the file'
|
||||
assert location == 'downloads'
|
||||
|
||||
|
||||
def test_balanced_bracket_tags_still_stripped(tmp_path):
|
||||
"""No regression: balanced "[FLAC]" / "(Remastered 2016)" tags in a Soulseek
|
||||
filename must still be stripped so it matches the clean saved file."""
|
||||
downloads = tmp_path / 'downloads'
|
||||
target = downloads / 'Song.mp3'
|
||||
_touch(target)
|
||||
|
||||
found, _ = find_completed_audio_file(
|
||||
str(downloads), r'shared\Artist\Album\Song [FLAC] (Remastered 2016).mp3',
|
||||
)
|
||||
|
||||
assert found == str(target)
|
||||
|
||||
|
||||
def test_stray_closing_bracket_does_not_break_match(tmp_path):
|
||||
"""The other shape in the wild (Discord: "Abort, Retry, Fail_]1-01 …") — a
|
||||
stray ']' must not wreck the match either."""
|
||||
downloads = tmp_path / 'downloads'
|
||||
target = downloads / 'White Town' / "Fail_]1-01 Your Woman.flac"
|
||||
_touch(target)
|
||||
|
||||
found, _ = find_completed_audio_file(
|
||||
str(downloads), r"@@digadom\Music\White Town\Fail_]1-01 Your Woman.flac",
|
||||
)
|
||||
|
||||
assert found == str(target)
|
||||
|
|
|
|||
|
|
@ -1,66 +0,0 @@
|
|||
"""orchestrator.search routes to the right engine method per search_mode:
|
||||
|
||||
- priority (default) → engine.search_with_fallback (first source wins)
|
||||
- best_quality + hybrid → engine.search_all_sources (pool every source)
|
||||
- single-source mode → the single client's search (toggle is a no-op)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
import core.download_orchestrator as orch_mod
|
||||
from core.download_orchestrator import DownloadOrchestrator
|
||||
|
||||
|
||||
def _run(coro):
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
return loop.run_until_complete(coro)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
|
||||
class _SpyEngine:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
async def search_with_fallback(self, query, chain, timeout=None, progress_callback=None):
|
||||
self.calls.append(('fallback', tuple(chain)))
|
||||
return (['fb'], [])
|
||||
|
||||
async def search_all_sources(self, query, chain, timeout=None,
|
||||
progress_callback=None, exclude_sources=None):
|
||||
self.calls.append(('all', tuple(chain)))
|
||||
return (['pool'], [])
|
||||
|
||||
|
||||
def _hybrid_orch():
|
||||
orch = DownloadOrchestrator.__new__(DownloadOrchestrator)
|
||||
orch.mode = 'hybrid'
|
||||
orch.hybrid_order = ['soulseek', 'hifi']
|
||||
orch.hybrid_primary = 'soulseek'
|
||||
orch.hybrid_secondary = 'hifi'
|
||||
orch.engine = _SpyEngine()
|
||||
# _resolve_source_chain normalizes names through the registry; stub it so the
|
||||
# test doesn't need a full registry.
|
||||
orch._resolve_source_chain = lambda: ['soulseek', 'hifi']
|
||||
return orch
|
||||
|
||||
|
||||
def test_priority_mode_uses_search_with_fallback(monkeypatch):
|
||||
monkeypatch.setattr(orch_mod, 'load_search_mode', lambda: 'priority')
|
||||
orch = _hybrid_orch()
|
||||
|
||||
tracks, _ = _run(orch.search('q'))
|
||||
|
||||
assert orch.engine.calls == [('fallback', ('soulseek', 'hifi'))]
|
||||
assert tracks == ['fb']
|
||||
|
||||
|
||||
def test_best_quality_mode_uses_search_all_sources(monkeypatch):
|
||||
monkeypatch.setattr(orch_mod, 'load_search_mode', lambda: 'best_quality')
|
||||
orch = _hybrid_orch()
|
||||
|
||||
tracks, _ = _run(orch.search('q'))
|
||||
|
||||
assert orch.engine.calls == [('all', ('soulseek', 'hifi'))]
|
||||
assert tracks == ['pool']
|
||||
|
|
@ -38,14 +38,14 @@ def _cand(quality, size_mb, bitrate=None):
|
|||
queue_length=0, artist='A', title='Song', album='B', track_number=1)
|
||||
|
||||
|
||||
def _q(enabled_flac=True, enabled_mp3=True, aac=None, fallback=True):
|
||||
def _q(enabled_flac=True, enabled_mp3=True, aac=None):
|
||||
qualities = {
|
||||
'flac': {'enabled': enabled_flac, 'min_kbps': 500, 'max_kbps': 10000, 'priority': 1, 'bit_depth': 'any'},
|
||||
'mp3_320': {'enabled': enabled_mp3, 'min_kbps': 280, 'max_kbps': 500, 'priority': 2},
|
||||
}
|
||||
if aac is not None: # None => omit the tier entirely
|
||||
if aac is not None: # None => omit the tier entirely (pre-existing profile)
|
||||
qualities['aac'] = {'enabled': aac, 'min_kbps': 128, 'max_kbps': 400, 'priority': 1.5}
|
||||
return {'preset': 'custom', 'qualities': qualities, 'fallback_enabled': fallback}
|
||||
return {'preset': 'custom', 'qualities': qualities, 'fallback_enabled': True}
|
||||
|
||||
|
||||
def _filter(candidates, profile):
|
||||
|
|
@ -56,44 +56,51 @@ def _filter(candidates, profile):
|
|||
return c.filter_results_by_quality_preference(candidates)
|
||||
|
||||
|
||||
# ── AAC follows the UNIVERSAL rule (no per-format special-casing) ──────────────
|
||||
# AAC is just another format: it passes only if it matches a ranked target;
|
||||
# when nothing matches, the fallback toggle decides — exactly like every format.
|
||||
|
||||
def test_aac_not_targeted_fallback_off_is_dropped():
|
||||
out = _filter([_cand('aac', 5)], _q(aac=None, fallback=False))
|
||||
assert out == [] # no AAC target + fallback off → nothing comes through
|
||||
# ── additive proof: AAC off == today (dropped) ────────────────────────────────
|
||||
def test_aac_dropped_when_tier_absent_pre_existing_profile():
|
||||
# A profile saved before this feature has no 'aac' key at all.
|
||||
out = _filter([_cand('aac', 5)], _q(aac=None))
|
||||
assert out == [] # AAC went to 'other' -> never returned, exactly as before
|
||||
|
||||
|
||||
def test_aac_not_targeted_fallback_on_comes_through():
|
||||
out = _filter([_cand('aac', 5)], _q(aac=None, fallback=True))
|
||||
assert len(out) == 1 and out[0].quality == 'aac' # fallback grabs it
|
||||
|
||||
|
||||
def test_aac_disabled_tier_behaves_like_not_targeted():
|
||||
# An explicitly-disabled aac tier is not a target → same as absent.
|
||||
assert _filter([_cand('aac', 5)], _q(aac=False, fallback=False)) == []
|
||||
def test_aac_dropped_when_tier_present_but_disabled():
|
||||
out = _filter([_cand('aac', 5)], _q(aac=False))
|
||||
assert out == []
|
||||
|
||||
|
||||
def test_flac_mp3_selection_unchanged_when_aac_absent():
|
||||
# The headline no-regression guard: a normal FLAC/MP3 mix is unaffected.
|
||||
flac, mp3 = _cand('flac', 30), _cand('mp3', 5, bitrate=320)
|
||||
out = _filter([mp3, flac], _q(aac=None))
|
||||
assert out and out[0].quality == 'flac' # FLAC still wins
|
||||
assert out and out[0].quality == 'flac' # FLAC still wins, as before
|
||||
|
||||
|
||||
# ── AAC as a real ranked target ───────────────────────────────────────────────
|
||||
def test_aac_selected_when_targeted():
|
||||
# ── opt-in: AAC on makes it a real tier ───────────────────────────────────────
|
||||
def test_aac_selected_when_enabled():
|
||||
out = _filter([_cand('aac', 5)], _q(aac=True))
|
||||
assert len(out) == 1 and out[0].quality == 'aac'
|
||||
|
||||
|
||||
def test_flac_beats_aac_when_listed_higher():
|
||||
def test_flac_beats_aac_when_both_present():
|
||||
flac, aac = _cand('flac', 30), _cand('aac', 5)
|
||||
out = _filter([aac, flac], _q(aac=True))
|
||||
assert out[0].quality == 'flac' # flac target ranked above aac (priority 1 < 1.5)
|
||||
assert out[0].quality == 'flac' # priority 1 < 1.5
|
||||
|
||||
|
||||
def test_aac_beats_mp3_when_listed_higher():
|
||||
def test_aac_beats_mp3_when_both_present():
|
||||
mp3, aac = _cand('mp3', 5, bitrate=320), _cand('aac', 5)
|
||||
out = _filter([mp3, aac], _q(aac=True))
|
||||
assert out[0].quality == 'aac' # aac target (1.5) ranked above mp3 (2)
|
||||
assert out[0].quality == 'aac' # priority 1.5 < 2
|
||||
|
||||
|
||||
def test_default_and_presets_ship_aac_disabled_above_mp3():
|
||||
from database.music_database import MusicDatabase
|
||||
db = MusicDatabase.__new__(MusicDatabase) # no DB init
|
||||
profiles = [db._get_default_quality_profile()]
|
||||
profiles += [db.get_quality_preset(p) for p in ('audiophile', 'balanced', 'space_saver')]
|
||||
for prof in profiles:
|
||||
aac = prof['qualities']['aac']
|
||||
assert aac['enabled'] is False # opt-in everywhere
|
||||
# above MP3: lower priority number than the best MP3 tier present
|
||||
mp3_prios = [v['priority'] for k, v in prof['qualities'].items() if k.startswith('mp3')]
|
||||
assert aac['priority'] < min(mp3_prios)
|
||||
|
|
|
|||
|
|
@ -78,156 +78,3 @@ def test_payload_non_dict_or_empty():
|
|||
assert q('tidal', None) is None
|
||||
assert q('tidal', {}) is None
|
||||
assert q('qobuz', 'garbage') is None
|
||||
|
||||
|
||||
# ── bubble the pasted-link track to the top (#932) ──
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
from core.downloads.track_link import linked_track_id, bubble_linked_track_first
|
||||
|
||||
|
||||
def _result(track_id=None):
|
||||
"""Mimic a TrackResult: NO top-level `id`, the source id lives in
|
||||
_source_metadata['track_id'] (or absent entirely)."""
|
||||
meta = {'source': 'qobuz', 'track_id': track_id} if track_id is not None else None
|
||||
return SimpleNamespace(_source_metadata=meta, title='t')
|
||||
|
||||
|
||||
def test_linked_track_id_reads_source_metadata():
|
||||
assert linked_track_id(_result('296427754')) == '296427754'
|
||||
|
||||
|
||||
def test_linked_track_id_empty_when_absent():
|
||||
# the exact #932 trap: there is no top-level `id`, so getattr(t,'id') would miss.
|
||||
r = _result()
|
||||
assert not hasattr(r, 'id')
|
||||
assert linked_track_id(r) == ''
|
||||
|
||||
|
||||
def test_bubble_floats_exact_track_to_top():
|
||||
fuzzy_a, exact, fuzzy_b = _result('111'), _result('296427754'), _result('222')
|
||||
out = bubble_linked_track_first([fuzzy_a, exact, fuzzy_b], '296427754')
|
||||
assert out[0] is exact # exact track surfaced first
|
||||
assert out[1:] == [fuzzy_a, fuzzy_b] # stable order for the rest
|
||||
|
||||
|
||||
def test_bubble_handles_int_vs_str_id():
|
||||
out = bubble_linked_track_first([_result('999'), _result('296427754')], 296427754)
|
||||
assert linked_track_id(out[0]) == '296427754'
|
||||
|
||||
|
||||
def test_bubble_noop_when_nothing_matches_or_empty():
|
||||
a, b = _result('1'), _result('2')
|
||||
assert bubble_linked_track_first([a, b], '296427754') == [a, b] # unchanged
|
||||
assert bubble_linked_track_first([], '296427754') == []
|
||||
assert bubble_linked_track_first([a, b], '') == [a, b]
|
||||
|
||||
|
||||
# ── inject the EXACT fetched track first (#932 reopen: text search misses obscure tracks) ──
|
||||
|
||||
from core.downloads.track_link import inject_linked_track_first
|
||||
|
||||
|
||||
def test_inject_puts_fetched_track_first_when_search_missed_it():
|
||||
# The reported case: the linked track isn't among the fuzzy text-search
|
||||
# results at all, so the directly-fetched result is injected at the top.
|
||||
fetched = _result('296427754')
|
||||
search = [_result('111'), _result('222')] # unrelated lookalikes
|
||||
out = inject_linked_track_first(search, fetched, '296427754')
|
||||
assert out[0] is fetched
|
||||
assert len(out) == 3
|
||||
|
||||
|
||||
def test_inject_dedups_a_search_copy_of_the_linked_track():
|
||||
fetched = _result('296427754')
|
||||
dup = _result('296427754') # search also returned it
|
||||
search = [_result('111'), dup, _result('222')]
|
||||
out = inject_linked_track_first(search, fetched, '296427754')
|
||||
assert out[0] is fetched
|
||||
# exactly one element carries the linked id, and it's the injected one
|
||||
assert [linked_track_id(t) for t in out].count('296427754') == 1
|
||||
assert all(t is not dup for t in out) # the search copy (by identity) was dropped
|
||||
assert len(out) == 3
|
||||
|
||||
|
||||
def test_inject_falls_back_to_bubble_without_a_fetched_result():
|
||||
exact = _result('296427754')
|
||||
out = inject_linked_track_first([_result('111'), exact, _result('222')], None, '296427754')
|
||||
assert linked_track_id(out[0]) == '296427754' # bubbled, not injected
|
||||
assert len(out) == 3
|
||||
|
||||
|
||||
def test_inject_is_noop_without_a_link_id():
|
||||
search = [_result('111')]
|
||||
assert inject_linked_track_first(search, _result('x'), '') == search
|
||||
|
||||
|
||||
def test_inject_int_track_id_is_str_safe():
|
||||
fetched = _result('296427754')
|
||||
out = inject_linked_track_first([_result('111')], fetched, 296427754)
|
||||
assert out[0] is fetched
|
||||
|
||||
|
||||
# ── QobuzClient.get_track_result: fetch by id → downloadable TrackResult ──
|
||||
|
||||
from core.qobuz_client import QobuzClient
|
||||
|
||||
|
||||
def _bare_qobuz():
|
||||
return QobuzClient.__new__(QobuzClient) # no network / __init__
|
||||
|
||||
|
||||
def test_get_track_result_converts_fetched_track(monkeypatch):
|
||||
client = _bare_qobuz()
|
||||
sentinel = object()
|
||||
monkeypatch.setattr(client, 'get_track', lambda tid: {'id': tid, 'streamable': True})
|
||||
monkeypatch.setattr(client, '_qobuz_to_track_result',
|
||||
lambda track, qi, require_streamable=True: sentinel)
|
||||
assert client.get_track_result('296427754') is sentinel
|
||||
|
||||
|
||||
def test_get_track_result_none_when_track_unavailable(monkeypatch):
|
||||
client = _bare_qobuz()
|
||||
monkeypatch.setattr(client, 'get_track', lambda tid: None)
|
||||
assert client.get_track_result('nope') is None
|
||||
|
||||
|
||||
def test_get_track_result_none_on_exception(monkeypatch):
|
||||
client = _bare_qobuz()
|
||||
def _boom(_tid):
|
||||
raise RuntimeError('api down')
|
||||
monkeypatch.setattr(client, 'get_track', _boom)
|
||||
assert client.get_track_result('x') is None
|
||||
|
||||
|
||||
# ── #932 hardening: a pasted-link fetch must not be dropped by a missing
|
||||
# 'streamable' flag (track/get may omit it). Exercises the REAL converter. ──
|
||||
|
||||
_QOBUZ_TRACK = {'id': 296427754, 'title': 'foreign lavennew',
|
||||
'performer': {'name': 'colacola'}, 'duration': 180}
|
||||
|
||||
|
||||
def test_converter_rejects_non_streamable_on_search_path():
|
||||
# Default (bulk search) still filters on streamable — unchanged behaviour.
|
||||
client = _bare_qobuz()
|
||||
qi = {'codec': 'flac', 'bitrate': 1411}
|
||||
assert client._qobuz_to_track_result(dict(_QOBUZ_TRACK), qi) is None # no streamable flag
|
||||
|
||||
|
||||
def test_converter_builds_for_link_fetch_without_streamable():
|
||||
client = _bare_qobuz()
|
||||
qi = {'codec': 'flac', 'bitrate': 1411}
|
||||
r = client._qobuz_to_track_result(dict(_QOBUZ_TRACK), qi, require_streamable=False)
|
||||
assert r is not None
|
||||
assert linked_track_id(r) == '296427754' # carries the id → injectable + downloadable
|
||||
|
||||
|
||||
def test_get_track_result_survives_missing_streamable_flag(monkeypatch):
|
||||
# The feared track/get shape (no 'streamable') must STILL yield the track —
|
||||
# this is the end-to-end gap that couldn't be confirmed against a live API.
|
||||
client = _bare_qobuz()
|
||||
monkeypatch.setattr(client, 'get_track', lambda _tid: dict(_QOBUZ_TRACK))
|
||||
r = client.get_track_result('296427754')
|
||||
assert r is not None
|
||||
assert linked_track_id(r) == '296427754'
|
||||
|
|
|
|||
|
|
@ -1,47 +0,0 @@
|
|||
"""core.downloads.task_worker._candidate_ordering — decides whether the
|
||||
download walk is ordered quality-first or confidence-first.
|
||||
|
||||
Quality-first applies when:
|
||||
- best_quality search mode is active (always), OR
|
||||
- priority mode AND the rank_candidates_by_quality toggle is on.
|
||||
|
||||
Default (priority mode, toggle off) → confidence-first, the byte-for-byte old
|
||||
behaviour. Any error fails closed to confidence-first so a DB hiccup never
|
||||
blocks a download.
|
||||
"""
|
||||
|
||||
import core.quality.selection as selection
|
||||
import core.downloads.task_worker as task_worker
|
||||
|
||||
_TARGETS = ["t1", "t2"]
|
||||
|
||||
|
||||
def _patch(monkeypatch, *, mode, rank_toggle):
|
||||
monkeypatch.setattr(selection, "load_search_mode", lambda: mode)
|
||||
monkeypatch.setattr(
|
||||
selection, "load_rank_candidates_by_quality", lambda: rank_toggle
|
||||
)
|
||||
monkeypatch.setattr(selection, "load_profile_targets", lambda: (_TARGETS, True))
|
||||
|
||||
|
||||
def test_priority_mode_toggle_off_is_confidence_first(monkeypatch):
|
||||
_patch(monkeypatch, mode="priority", rank_toggle=False)
|
||||
assert task_worker._candidate_ordering() == (False, None)
|
||||
|
||||
|
||||
def test_priority_mode_toggle_on_is_quality_first(monkeypatch):
|
||||
_patch(monkeypatch, mode="priority", rank_toggle=True)
|
||||
assert task_worker._candidate_ordering() == (True, _TARGETS)
|
||||
|
||||
|
||||
def test_best_quality_mode_is_quality_first_regardless_of_toggle(monkeypatch):
|
||||
_patch(monkeypatch, mode="best_quality", rank_toggle=False)
|
||||
assert task_worker._candidate_ordering() == (True, _TARGETS)
|
||||
|
||||
|
||||
def test_fails_closed_to_confidence_first_on_error(monkeypatch):
|
||||
def _boom():
|
||||
raise RuntimeError("db down")
|
||||
|
||||
monkeypatch.setattr(selection, "load_search_mode", _boom)
|
||||
assert task_worker._candidate_ordering() == (False, None)
|
||||
|
|
@ -57,182 +57,3 @@ def test_all_miss_returns_none_and_no_write():
|
|||
fn, recorded = _wire()
|
||||
assert fn("A", "T") == (None, None)
|
||||
assert recorded == {}
|
||||
|
||||
|
||||
# ── service track-id resolver (#945 export to Spotify/Deezer) ──
|
||||
|
||||
from core.exports.export_sources import (
|
||||
db_service_track_id,
|
||||
build_service_resolve_fn,
|
||||
_SERVICE_ID_COLUMNS,
|
||||
)
|
||||
|
||||
|
||||
def test_service_id_column_mapping():
|
||||
assert _SERVICE_ID_COLUMNS == {'spotify': 'spotify_track_id', 'deezer': 'deezer_id'}
|
||||
|
||||
|
||||
def test_db_service_track_id_unknown_service_is_none():
|
||||
assert db_service_track_id('A', 'X', 'tidal') is None
|
||||
assert db_service_track_id('A', 'X', '') is None
|
||||
|
||||
|
||||
def test_db_service_track_id_no_title_is_none():
|
||||
assert db_service_track_id('A', '', 'spotify') is None
|
||||
|
||||
|
||||
def test_build_service_resolve_fn_returns_id_and_source(monkeypatch):
|
||||
import core.exports.export_sources as es
|
||||
monkeypatch.setattr(es, 'db_service_track_id',
|
||||
lambda a, t, s: 'spid-99' if t == 'Hit' else None)
|
||||
fn = build_service_resolve_fn('spotify')
|
||||
assert fn('Artist', 'Hit') == ('spid-99', 'library')
|
||||
assert fn('Artist', 'Miss') == (None, None)
|
||||
|
||||
|
||||
def test_db_service_track_id_real_sql_executes(tmp_path, monkeypatch):
|
||||
"""Run the ACTUAL query against a real (temp) tracks/artists schema — the broad
|
||||
except→None in db_service_track_id would otherwise mask a column/join typo as
|
||||
'no match' for every track (#945 verification)."""
|
||||
import sqlite3
|
||||
import types
|
||||
import core.exports.export_sources as es
|
||||
|
||||
dbfile = tmp_path / "lib.db"
|
||||
con = sqlite3.connect(str(dbfile))
|
||||
con.executescript(
|
||||
"CREATE TABLE artists (id TEXT PRIMARY KEY, name TEXT);"
|
||||
"CREATE TABLE tracks (id TEXT, artist_id TEXT, title TEXT, "
|
||||
"spotify_track_id TEXT, deezer_id TEXT);"
|
||||
"INSERT INTO artists VALUES ('a1','Kendrick Lamar');"
|
||||
"INSERT INTO tracks VALUES ('t1','a1','Not Like Us','spid-NLU','dz-NLU');"
|
||||
)
|
||||
con.commit()
|
||||
con.close()
|
||||
|
||||
# fresh connection per call (db_service_track_id closes it in finally)
|
||||
fake_db = types.SimpleNamespace(_get_connection=lambda: sqlite3.connect(str(dbfile)))
|
||||
monkeypatch.setattr("database.music_database.get_database", lambda: fake_db)
|
||||
|
||||
assert es.db_service_track_id("Kendrick Lamar", "Not Like Us", "spotify") == "spid-NLU"
|
||||
assert es.db_service_track_id("kendrick lamar", "not like us", "deezer") == "dz-NLU" # case-insensitive
|
||||
assert es.db_service_track_id("Kendrick Lamar", "Unknown Song", "spotify") is None
|
||||
|
||||
|
||||
# ── discovery-cache resolution (#945: use the already-discovered IDs, no API call) ──
|
||||
|
||||
import json as _json
|
||||
from core.exports.export_sources import (
|
||||
service_id_from_extra_data,
|
||||
resolve_service_track_ids,
|
||||
)
|
||||
|
||||
|
||||
def _extra(service, tid, discovered=True, provider=None):
|
||||
return {'extra_data': _json.dumps({'discovered': discovered,
|
||||
'provider': provider or service,
|
||||
'matched_data': {'id': tid}})}
|
||||
|
||||
|
||||
def test_extra_data_id_when_discovered_to_that_service():
|
||||
assert service_id_from_extra_data(_extra('deezer', 111), 'deezer') == '111'
|
||||
# dict (not str) extra_data also works
|
||||
raw = {'extra_data': {'discovered': True, 'provider': 'spotify', 'matched_data': {'id': 'spX'}}}
|
||||
assert service_id_from_extra_data(raw, 'spotify') == 'spX'
|
||||
|
||||
|
||||
def test_extra_data_provider_must_match_service():
|
||||
# discovered to Spotify, exporting to Deezer → don't reuse the (wrong-service) id
|
||||
assert service_id_from_extra_data(_extra('spotify', 111), 'deezer') is None
|
||||
|
||||
|
||||
def test_extra_data_wing_it_fallback_is_not_trusted():
|
||||
track = _extra('deezer', 111, provider='wing_it_fallback')
|
||||
assert service_id_from_extra_data(track, 'deezer') is None
|
||||
|
||||
|
||||
def test_extra_data_misc_none_cases():
|
||||
assert service_id_from_extra_data({}, 'deezer') is None # no extra_data
|
||||
assert service_id_from_extra_data({'extra_data': 'not json{'}, 'deezer') is None # bad json
|
||||
assert service_id_from_extra_data(_extra('deezer', 111, discovered=False), 'deezer') is None
|
||||
|
||||
|
||||
def test_resolve_waterfall_cache_then_library_then_unmatched():
|
||||
tracks = [
|
||||
_extra('deezer', 111) | {'artist_name': 'A', 'track_name': 'Cached'}, # cache hit
|
||||
{'artist_name': 'A', 'track_name': 'InLib'}, # library hit (db_fn)
|
||||
{'artist_name': 'A', 'track_name': 'Nowhere'}, # unmatched
|
||||
]
|
||||
db_fn = lambda a, t, s: 'lib-222' if t == 'InLib' else None
|
||||
out = resolve_service_track_ids(tracks, 'deezer', db_fn=db_fn)
|
||||
ids = [r['service_track_id'] for r in out['resolved']]
|
||||
assert ids == ['111', 'lib-222', None]
|
||||
s = out['stats']
|
||||
assert s == {'total': 3, 'resolved': 2, 'unmatched': 1, 'from_cache': 1,
|
||||
'from_library': 1, 'from_search': 0}
|
||||
|
||||
|
||||
# ── backfill: confident live-search match for the un-cached/un-enriched tail (#945) ──
|
||||
|
||||
from core.metadata.types import Track as _Track
|
||||
from core.exports.export_sources import search_service_track_id, BACKFILL_MIN_SCORE
|
||||
|
||||
|
||||
def _cand(name, artist, tid, album_type='album'):
|
||||
return _Track(id=tid, name=name, artists=[artist], album='A',
|
||||
duration_ms=200000, album_type=album_type)
|
||||
|
||||
|
||||
def test_backfill_exact_match_returned():
|
||||
search = lambda q: [_cand('Not Like Us', 'Kendrick Lamar', 'dz-NLU')]
|
||||
assert search_service_track_id('Kendrick Lamar', 'Not Like Us', search_fn=search) == 'dz-NLU'
|
||||
|
||||
|
||||
def test_backfill_wrong_artist_rejected():
|
||||
"""SAFETY: an exact-title hit by the WRONG artist scores below the floor (no 1.5x exact-
|
||||
artist boost) → None, so backfill never adds someone else's same-named track."""
|
||||
search = lambda q: [_cand('Not Like Us', 'Some Other Guy', 'wrong-id')]
|
||||
assert search_service_track_id('Kendrick Lamar', 'Not Like Us', search_fn=search) is None
|
||||
|
||||
|
||||
def test_backfill_karaoke_cover_rejected():
|
||||
"""SAFETY: a karaoke/cover version is buried (x0.05) below the floor → None."""
|
||||
search = lambda q: [_cand('Not Like Us (Karaoke Version)', 'Karaoke All Stars', 'kar-id')]
|
||||
assert search_service_track_id('Kendrick Lamar', 'Not Like Us', search_fn=search) is None
|
||||
|
||||
|
||||
def test_backfill_picks_real_over_cover():
|
||||
search = lambda q: [
|
||||
_cand('Not Like Us (Karaoke Version)', 'Karaoke All Stars', 'kar-id'),
|
||||
_cand('Not Like Us', 'Kendrick Lamar', 'real-id'),
|
||||
]
|
||||
assert search_service_track_id('Kendrick Lamar', 'Not Like Us', search_fn=search) == 'real-id'
|
||||
|
||||
|
||||
def test_backfill_empty_and_error_and_no_title():
|
||||
assert search_service_track_id('A', 'X', search_fn=lambda q: []) is None
|
||||
def boom(q):
|
||||
raise RuntimeError('deezer flaked')
|
||||
assert search_service_track_id('A', 'X', search_fn=boom) is None # fail-safe
|
||||
assert search_service_track_id('A', '', search_fn=lambda q: [_cand('X', 'A', 'i')]) is None
|
||||
|
||||
|
||||
def test_resolve_waterfall_uses_search_only_when_cache_and_library_miss():
|
||||
tracks = [
|
||||
_extra('deezer', 111) | {'artist_name': 'A', 'track_name': 'Cached'},
|
||||
{'artist_name': 'A', 'track_name': 'InLib'},
|
||||
{'artist_name': 'A', 'track_name': 'OnlyOnSvc'},
|
||||
]
|
||||
db_fn = lambda a, t, s: 'lib-2' if t == 'InLib' else None
|
||||
search_id_fn = lambda a, t: 'srch-3' if t == 'OnlyOnSvc' else None
|
||||
out = resolve_service_track_ids(tracks, 'deezer', db_fn=db_fn, search_id_fn=search_id_fn)
|
||||
assert [r['service_track_id'] for r in out['resolved']] == ['111', 'lib-2', 'srch-3']
|
||||
s = out['stats']
|
||||
assert (s['from_cache'], s['from_library'], s['from_search'], s['unmatched']) == (1, 1, 1, 0)
|
||||
|
||||
|
||||
def test_resolve_no_search_fn_leaves_tail_unmatched():
|
||||
out = resolve_service_track_ids([{'artist_name': 'A', 'track_name': 'X'}], 'deezer',
|
||||
db_fn=lambda a, t, s: None) # search_id_fn omitted
|
||||
assert out['resolved'][0]['service_track_id'] is None
|
||||
assert out['stats']['unmatched'] == 1 and out['stats']['from_search'] == 0
|
||||
|
|
|
|||
|
|
@ -72,31 +72,3 @@ def test_empty_playlist():
|
|||
out = resolve_playlist_tracks([], lambda a, t: (None, None))
|
||||
assert out["resolved"] == []
|
||||
assert out["stats"]["total"] == 0
|
||||
|
||||
|
||||
# ── id_key generalization (#945 service export reuses the LB resolver) ──
|
||||
|
||||
from core.exports.playlist_export import resolve_playlist_tracks as _rpt
|
||||
|
||||
|
||||
def _const_resolver(mapping):
|
||||
return lambda artist, title: mapping.get((artist, title), (None, None))
|
||||
|
||||
|
||||
def test_default_id_key_is_recording_mbid_unchanged():
|
||||
# ListenBrainz/JSPF callers must be byte-for-byte unaffected by the generalization.
|
||||
out = _rpt([{'artist': 'A', 'title': 'X'}], _const_resolver({('A', 'X'): ('mbid-1', 'db')}))
|
||||
assert out['resolved'][0]['recording_mbid'] == 'mbid-1'
|
||||
assert 'service_track_id' not in out['resolved'][0]
|
||||
|
||||
|
||||
def test_custom_id_key_carries_service_id():
|
||||
out = _rpt(
|
||||
[{'artist': 'A', 'title': 'X'}, {'artist': 'B', 'title': 'Y'}],
|
||||
_const_resolver({('A', 'X'): ('spid-1', 'library')}), # B/Y unmatched
|
||||
id_key='service_track_id',
|
||||
)
|
||||
assert out['resolved'][0]['service_track_id'] == 'spid-1'
|
||||
assert out['resolved'][1]['service_track_id'] is None
|
||||
assert 'recording_mbid' not in out['resolved'][0]
|
||||
assert out['stats']['resolved'] == 1 and out['stats']['unmatched'] == 1
|
||||
|
|
|
|||
|
|
@ -201,60 +201,6 @@ def test_rejects_truncated_file(tmp_path: Path) -> None:
|
|||
assert result.checks["length_drift_s"] > 3.0
|
||||
|
||||
|
||||
# ── #937: a file that runs LONGER than expected is a version/master difference, not
|
||||
# truncation — it gets more leeway, while SHORTER files stay tight. ──
|
||||
|
||||
def test_accepts_longer_master_beyond_short_tolerance(tmp_path: Path) -> None:
|
||||
"""The reported case (A-Ha remaster): file runs ~3.5s LONGER than the metadata.
|
||||
Past the 3s short-tolerance but a remaster, not a bad download — must pass."""
|
||||
f = tmp_path / "remaster.wav"
|
||||
_write_minimal_wav(f, duration_s=9.0) # 9.0s file vs 5.5s expected → +3.5s longer
|
||||
|
||||
result = file_integrity.check_audio_integrity(str(f), expected_duration_ms=5500)
|
||||
|
||||
assert result.ok is True
|
||||
assert result.checks["length_check"] == "passed"
|
||||
assert result.checks["effective_tolerance_s"] == pytest.approx(15.0)
|
||||
|
||||
|
||||
def test_shorter_file_still_tight_after_longer_loosening(tmp_path: Path) -> None:
|
||||
"""Loosening the LONGER direction must not loosen truncation detection — a file
|
||||
3.5s SHORTER than expected is still rejected at the 3s tolerance."""
|
||||
f = tmp_path / "short.wav"
|
||||
_write_minimal_wav(f, duration_s=5.5) # 5.5s vs 9.0s expected → -3.5s shorter
|
||||
|
||||
result = file_integrity.check_audio_integrity(str(f), expected_duration_ms=9000)
|
||||
|
||||
assert result.ok is False
|
||||
assert "truncated" in result.reason.lower()
|
||||
assert result.checks["effective_tolerance_s"] == pytest.approx(3.0)
|
||||
|
||||
|
||||
def test_wildly_longer_file_still_rejected(tmp_path: Path) -> None:
|
||||
"""A different/wrong song that happens to run long is still caught — +25s blows
|
||||
past even the 15s longer-tolerance."""
|
||||
f = tmp_path / "wronglong.wav"
|
||||
_write_minimal_wav(f, duration_s=30.0) # 30s vs 5s expected → +25s
|
||||
|
||||
result = file_integrity.check_audio_integrity(str(f), expected_duration_ms=5000)
|
||||
|
||||
assert result.ok is False
|
||||
assert "longer than expected" in result.reason.lower()
|
||||
|
||||
|
||||
def test_user_pinned_tolerance_is_symmetric(tmp_path: Path) -> None:
|
||||
"""An explicit user tolerance is honoured in BOTH directions — the longer-direction
|
||||
loosening only applies to the auto default, not a value the user pinned."""
|
||||
f = tmp_path / "long.wav"
|
||||
_write_minimal_wav(f, duration_s=9.0) # +4s longer
|
||||
|
||||
result = file_integrity.check_audio_integrity(
|
||||
str(f), expected_duration_ms=5000, length_tolerance_s=2.0)
|
||||
|
||||
assert result.ok is False
|
||||
assert result.checks["effective_tolerance_s"] == pytest.approx(2.0)
|
||||
|
||||
|
||||
def test_rejects_wrong_file_substituted(tmp_path: Path) -> None:
|
||||
"""A 10-second clip masquerading as a 3-minute album track. slskd
|
||||
matched on a similar filename but the actual content is a snippet."""
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue