danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Compare commits

base: danvics:main
Branches Tags
danvics:main
danvics:dev
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65
..
compare: danvics:2.3
Branches Tags
danvics:dev
danvics:main
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65

No commits in common. "main" and "2.3" have entirely different histories.
main ... 2.3

1207 changed files with 162033 additions and 396349 deletions
Show stats Expand all files Collapse all files

27
.dockerignore
View file

@ -1,8 +1,5 @@
# Docker ignore file for SoulSync WebUI
# Hidden folders and files
.*
# Git
.git
.gitignore
@ -25,13 +22,6 @@ __pycache__/
dist/
build/
# Frontend build artifacts and local dependency caches
webui/.tanstack/
webui/.vite/
webui/node_modules/
webui/test-results/
webui/static/dist/
# Virtual environments
venv/
env/
@ -58,20 +48,6 @@ Incomplete/*
artist_bubble_snapshots.json
.spotify_cache
# Auto-downloaded ffmpeg binaries — the YouTube client downloads these
# into tools/ when system ffmpeg isn't on PATH. The Dockerfile installs
# system ffmpeg via apt, so the container never needs the bundled
# binaries. If a CI run leaves them in the workspace before the docker
# build (e.g. because a test imported web_server which initialized the
# YouTube client), they'd otherwise get baked into the image — adding
# ~388 MB and getting duplicated again by the chown layer.
tools/ffmpeg
tools/ffprobe
tools/ffmpeg.exe
tools/ffprobe.exe
tools/*.zip
tools/*.tar.xz
# Documentation
*.md
README.md
@ -80,9 +56,10 @@ multi-server-database-plan.md
server-source.md
plans.md
# GUI-specific files (removed by PR #311)
# GUI-specific files
main.py
ui/
requirements.txt
# Dev-specific files
requirements-dev.txt

36
.github/workflows/build-and-test.yml vendored
View file

@ -1,36 +0,0 @@
name: Compile the app and run tests
on:
push:
branches-ignore:
- main
- dev
jobs:
sanity-check:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.11"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: python -m pip install --upgrade pip && python -m pip install -r requirements-dev.txt
- name: Lint with ruff
run: python -m ruff check --output-format=github .
- name: Build app
run: python -m compileall api core database services scripts web_server.py wsgi.py beatport_unified_scraper.py
- name: Run tests
env:
PYTHONPATH: ${{ github.workspace }}
run: python -m pytest

25
.github/workflows/cleanup-dev-images.yml vendored
View file

@ -1,25 +0,0 @@
name: Cleanup old dev images
on:
schedule:
# Weekly on Sunday at 6 AM UTC
- cron: '0 6 * * 0'
workflow_dispatch:
jobs:
cleanup:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- name: Delete old dev image tags
uses: actions/delete-package-versions@v5
with:
package-name: soulsync
package-type: container
min-versions-to-keep: 10
delete-only-pre-release-versions: false
# Only prune tags matching the dev nightly pattern (keep rolling + version tags)
ignore-versions: '^(dev|nightly|latest|\\d+\\.\\d+)$'

104
.github/workflows/dev-nightly.yml vendored
View file

@ -1,104 +0,0 @@
name: Dev Nightly Build
on:
# Nightly at 4:00 AM UTC — only if dev branch has new commits
schedule:
- cron: '0 4 * * *'
# Also build on every push to dev for immediate feedback
push:
branches:
- dev
# Manual trigger for testing
workflow_dispatch:
jobs:
nightly:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
# Skip scheduled runs if dev branch has no new commits in the last 24h
# (pushes and manual triggers always run)
permissions:
contents: read
packages: write
steps:
- name: Checkout dev branch
uses: actions/checkout@v6
with:
ref: dev
- name: Set lowercase owner
run: echo "OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> "$GITHUB_ENV"
- name: Check for recent commits (scheduled only)
if: github.event_name == 'schedule'
id: recent
run: |
LAST_COMMIT=$(git log -1 --format=%ct)
NOW=$(date +%s)
DIFF=$(( NOW - LAST_COMMIT ))
if [ "$DIFF" -gt 86400 ]; then
echo "No commits in the last 24h, skipping build"
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- name: Set up Python
if: steps.recent.outputs.skip != 'true'
uses: actions/setup-python@v6
with:
python-version: "3.11"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies and run tests
if: steps.recent.outputs.skip != 'true'
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txt
python -m ruff check --output-format=github .
python -m compileall api core database services scripts web_server.py wsgi.py beatport_unified_scraper.py
python -m pytest
- name: Set up QEMU
if: steps.recent.outputs.skip != 'true'
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
if: steps.recent.outputs.skip != 'true'
uses: docker/setup-buildx-action@v4
- name: Log in to GHCR
if: steps.recent.outputs.skip != 'true'
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ env.OWNER }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Generate build tag
if: steps.recent.outputs.skip != 'true'
id: tag
run: |
echo "date=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"
echo "short_sha=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
- name: Build and push to GHCR
if: steps.recent.outputs.skip != 'true'
uses: docker/build-push-action@v7
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
pull: true
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
COMMIT_SHA=${{ github.sha }}
tags: |
ghcr.io/${{ env.OWNER }}/soulsync:dev
ghcr.io/${{ env.OWNER }}/soulsync:dev-${{ steps.tag.outputs.date }}-${{ steps.tag.outputs.short_sha }}
${{ github.event_name == 'schedule' && format('ghcr.io/{0}/soulsync:nightly', env.OWNER) || '' }}

43
.github/workflows/docker-publish.yml vendored
View file

@ -1,72 +1,49 @@
name: Build and Push Docker Image
on:
# Auto-build :latest on every push to main
push:
branches:
- main
# Manual trigger for tagged releases (e.g. 2.33)
workflow_dispatch:
inputs:
version_tag:
description: 'Version tag (e.g. 2.8.2)'
description: 'Version tag (e.g. 1.6, 1.7)'
required: true
default: '2.8.2'
default: '2.3'
jobs:
build-and-push:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set lowercase owner
run: echo "OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> "$GITHUB_ENV"
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@v3
- name: Log in to Docker Hub
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ env.OWNER }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v7
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
no-cache: true
pull: true
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
COMMIT_SHA=${{ github.sha }}
tags: |
boulderbadgedad/soulsync:latest
ghcr.io/${{ env.OWNER }}/soulsync:latest
${{ inputs.version_tag && format('boulderbadgedad/soulsync:{0}', inputs.version_tag) || '' }}
${{ inputs.version_tag && format('ghcr.io/{0}/soulsync:{1}', env.OWNER, inputs.version_tag) || '' }}
boulderbadgedad/soulsync:${{ inputs.version_tag }}
- name: Announce release to Discord
if: success() && inputs.version_tag
if: success()
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_ANNOUNCEMENTS_WEBHOOK }}
run: |

16
.gitignore vendored
View file

@ -12,21 +12,13 @@ __pycache__/
# User-specific files (auto-created by the app if missing)
config/config.json
config/youtube_cookies.txt
# All app databases are live user data — never commit (music_library, video_library, …)
database/*.db
database/*.db-shm
database/*.db-wal
database/*.db.backup_*
database/music_library.db
database/music_library.db-shm
database/music_library.db-wal
database/music_library.db.backup_*
database/api_call_history.json
storage/image_cache/
logs/*.log
logs/*.log.*
# Auto-downloaded binaries
bin/
# Development compose/config files
*.dev.yml
# Any hidden folders
**/.*/

74
DISCOVER_BEST_IN_CLASS_PLAN.md
View file

@ -1,74 +0,0 @@
# discover page — best in class plan (#913 + full generator audit)
morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
## what i shipped tonight (done, tested, safe)
### 1. listening recommendations (#913) — went from BROKEN to best-in-class
the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
the fix (all in the pure, tested core + thin scan wiring):
- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
### 2. its own row on the discover page
new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
### 3. Fresh Tape "only 5-10 tracks" — fixed
root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
---
## best-in-class roadmap for listening recs (next phases — your call)
these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
| phase | what | value | risk | notes |
|---|---|---|---|---|
| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
---
## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
### curated (built during the scan, then hydrated)
- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
### discovery-pool generators (live queries)
- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
### cross-cutting
- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.

77
Dockerfile
View file

@ -1,16 +1,6 @@
# SoulSync WebUI Dockerfile
# Multi-architecture support for AMD64 and ARM64
FROM node:24-slim AS webui-builder
WORKDIR /app/webui
COPY webui/package.json webui/package-lock.json ./
RUN npm ci
COPY webui/ ./
RUN npm run build
# Stage 1: Builder — install Python dependencies with compilation tools
FROM python:3.11-slim AS builder
@ -25,19 +15,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
COPY requirements-webui.txt .
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# yt-dlp must track YouTube faster than its stable channel ships — stable can
# lag months behind a breaking YouTube change while extraction is broken
# ("Requested format is not available"). Build images with the NIGHTLY channel.
# COMMIT_SHA is referenced in the RUN so CI's layer cache (cache-from: gha)
# busts on every new commit — otherwise this layer could pin a stale "nightly"
# for months, silently defeating its purpose.
ARG COMMIT_SHA=""
RUN echo "yt-dlp nightly for build ${COMMIT_SHA}" && \
pip install --no-cache-dir -U --pre "yt-dlp[default]"
pip install --no-cache-dir -r requirements-webui.txt
# Stage 2: Runtime — only runtime dependencies, no build tools
FROM python:3.11-slim
@ -54,61 +34,24 @@ ENV PATH="/opt/venv/bin:$PATH"
# Set working directory
WORKDIR /app
# Install runtime-only system dependencies (no gcc/build tools).
# unzip is needed by the Deno installer below.
# Install runtime-only system dependencies (no gcc/build tools)
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
gosu \
ffmpeg \
libchromaprint-tools \
unzip \
&& rm -rf /var/lib/apt/lists/*
# Deno — JavaScript runtime for yt-dlp. YouTube gates its downloadable formats
# behind JS challenges (nsig); without a JS runtime, yt-dlp's extraction is
# deprecated and streams / music-video downloads fail with "Requested format
# is not available". Deno is yt-dlp's default-enabled runtime; the official
# installer auto-detects amd64/arm64. `deno --version` fails the build early
# if the install ever breaks.
RUN curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh && \
deno --version
# Create non-root user for security
RUN useradd --create-home --shell /bin/bash --uid 1000 soulsync
# Copy application code with ownership baked in.
# Using `COPY --chown` instead of `COPY` + `chown -R /app` avoids an
# extra image layer that duplicates the entire /app tree just to flip
# ownership bits — Docker layers are immutable, so chown -R rewrites
# every file into a new layer. On a clean repo that's small; if any
# bulky workspace file slips in (e.g. auto-downloaded ffmpeg binaries
# in tools/), it gets counted twice in the image. Cin caught this on
# 2026-05-08 — see the .dockerignore comment for the same incident.
COPY --chown=soulsync:soulsync . .
COPY --chown=soulsync:soulsync --from=webui-builder /app/webui/static/dist /app/webui/static/dist
# Copy application code
COPY . .
# Create runtime mount-point directories the app expects to exist.
# Create necessary directories with proper permissions
# NOTE: /app/data is for database FILES, /app/database is the Python package
# NOTE: /app/Staging is required even though most users bind-mount it —
# the entrypoint mkdir runs early and is gated by `set -e`, so a missing
# pre-baked directory would crash the container into a restart loop on
# rootless Docker/Podman where in-container "root" can't write to /app.
# Pre-baking the dir here makes the entrypoint mkdir a guaranteed no-op.
# NOTE: /app/Stream is the transient single-file streaming cache used by
# the basic-search "Play" flow (cleared per use, never persistent). It's
# created lazily by `core/streaming/prepare.py` via `os.makedirs`, which
# fails silently on rootless Docker where the soulsync UID can't write
# to /app — playback then errors out with no obvious cause. Pre-baking
# at build time (when the layer is owned by root) avoids that path.
# NOTE: /app/storage is the PRIVATE album-bundle staging area for the
# torrent / usenet whole-release flow (download_source.album_bundle_staging_path
# defaults to 'storage/album_bundle_staging'). Like /app/Stream it's created
# lazily at runtime via mkdir(parents=True); without pre-baking it owned by
# soulsync, the album-bundle copy fails with "[Errno 13] Permission denied:
# 'storage'" because /app itself is root-owned and the soulsync UID can't
# create a top-level dir there.
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts && \
chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/MusicVideos /app/scripts && \
chown -R soulsync:soulsync /app
# Create defaults directory and copy template files
# These will be used by entrypoint.sh to initialize empty volumes
@ -138,6 +81,8 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
# Set environment variables
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
ENV FLASK_APP=web_server.py
ENV FLASK_ENV=production
ENV DATABASE_PATH=/app/data/music_library.db
ENV PUID=1000
ENV PGID=1000
@ -145,4 +90,4 @@ ENV UMASK=022
# Set entrypoint and default command
ENTRYPOINT ["/entrypoint.sh"]
CMD ["gunicorn", "-c", "gunicorn.conf.py", "wsgi:application"]
CMD ["python", "web_server.py"]

198
README.md
View file

@ -8,13 +8,13 @@
> **IMPORTANT**: Configure file sharing in slskd to avoid Soulseek bans. Set up shared folders at `http://localhost:5030/shares`.
**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
---
## What It Does
SoulSync bridges streaming services to your music library with automated discovery:
SoulSync bridges streaming services to your media server with automated discovery:
1. **Monitors artists** → Automatically detects new releases from your watchlist
2. **Generates playlists** → Release Radar, Discovery Weekly, Seasonal, Decade/Genre mixes, Cache-powered discovery
@ -23,7 +23,7 @@ SoulSync bridges streaming services to your music library with automated discove
5. **Enriches metadata** → 10 enrichment workers (Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz)
6. **Tags consistently** → Picard-style MusicBrainz release preflight ensures all album tracks get the same release ID
7. **Organizes files** → Custom templates for clean folder structures
8. **Manages library** → Plex, Jellyfin, Navidrome, or SoulSync Standalone (no media server required)
8. **Syncs media server** → Plex, Jellyfin, or Navidrome stay updated automatically
9. **Scrobbles plays** → Automatic scrobbling to Last.fm and ListenBrainz from your media server
---
@ -84,7 +84,6 @@ SoulSync bridges streaming services to your music library with automated discove
- Hi-Res FLAC downsampling to 16-bit/44.1kHz CD quality
- Blasphemy Mode — delete original FLAC after conversion
- Synchronized lyrics (LRC) via LRClib
- ReplayGain analysis — optional track-level loudness tagging via ffmpeg, runs before lossy copy so both files get tagged
- Picard-style album consistency — pre-flight MusicBrainz release lookup ensures all tracks get the same release ID
### Listening Stats & Scrobbling
@ -105,21 +104,6 @@ SoulSync bridges streaming services to your music library with automated discove
- Catches wrong versions (live, remix, cover) even from streaming API sources
- Fail-open design: verification errors never block downloads
#### AcoustID API key
AcoustID verification is opt-in. To enable it, request a free API key
at <https://acoustid.org/new-application> and paste it into
Settings → AcoustID. Without a key, downloads still complete but the
verification step is skipped silently.
If a track was previously tagged by AcoustID but the retag action in
the AcoustID Scanner no longer changes anything, see issue #704 — the
most common cause is that the file already carries a
`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
and therefore never overwrites. Removing the cached
`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
restores the retag.
### Metadata & Enrichment
**10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz
@ -133,13 +117,8 @@ restores the retag.
- Spotify auth still enables playlists, followed artists, and enrichment
- MusicBrainz enrichment with Picard-style album consistency
**Hydrabase** (optional P2P metadata network) — replaces iTunes as the metadata source when connected. Federated lookup with community-matched results, falls back automatically if disconnected. Dev-mode feature, enable in Settings → Connections.
**Genre Whitelist** — filter junk genre tags (artist names, radio show names, playlist names) from all 10 enrichment sources. 272 curated default genres, fully customizable. Off by default for backward compatibility.
**Post-Processing Tag Embedding**
- Granular per-service tag toggles (18+ MusicBrainz tags, Spotify/iTunes/Deezer IDs, AudioDB mood/style, Tidal/Qobuz ISRCs, Last.fm tags, Genius URLs)
- Multi-artist tagging options: configurable separator (comma/semicolon/slash), multi-value ARTISTS tag for Navidrome/Jellyfin multi-artist linking, optional "move featured artists to title" mode
- Album art embedding, cover.jpg download
- Spotify rate limit protection across all API calls
@ -155,13 +134,12 @@ restores the retag.
### Automation
**Automation Engine** — Visual drag-and-drop builder for custom workflows
- **Triggers**: Schedule, Daily/Weekly Time, Track Downloaded, Batch Complete, Playlist Changed, Discovery Complete, Signal Received, Library Scan Complete, Watchlist Match, Wishlist Item Added, and more
- **Actions**: Process Wishlist, Scan Watchlist, Refresh Mirrored, Discover Playlist, Sync Playlist, Scan Library, Database Update, Quality Scan, Full Cleanup, and 10+ more
- **Then Actions** (up to 3 per automation): Fire Signal (chain to other automations), Discord/Telegram/Pushbullet notifications, audible chimes
- **Signal Chains** — One automation fires `signal:foo`, another listens for it. Cycle detection + chain depth limit + cooldown prevent runaway chains.
- **Playlist Pipeline** — Single automation for full playlist lifecycle: refresh → discover → sync → download missing. No manual signal wiring.
- **Pipelines** — Pre-built one-click deployments (New Music, Nightly Operations, Full Library Maintenance, etc.) that install a linked group of automations at once
- **Automation Groups** — Drag-and-drop organization, bulk enable/disable, rename, right-click context menus
- **Triggers**: Schedule, Daily/Weekly Time, Track Downloaded, Batch Complete, Playlist Changed, Discovery Complete, Signal Received, and 10+ more
- **Actions**: Process Wishlist, Scan Watchlist, Refresh Mirrored, Discover Playlist, Sync Playlist, Scan Library, Database Update, Quality Scan, Full Cleanup, and more
- **Then Actions**: Fire Signal (chain automations), Discord, Telegram, Pushbullet notifications
- **Playlist Pipeline**: Single automation for full playlist lifecycle — refresh → discover → sync → download missing. No signal wiring needed.
- **Pipelines**: Pre-built one-click pipeline deployments (New Music, Nightly Operations, etc.)
- **Signal Chains**: playlist_id forwarded from events to action handlers for proper chain execution
**Watchlist** — Monitor unlimited artists with per-artist configuration
- Release type filters: Albums, EPs, Singles
@ -171,11 +149,8 @@ restores the retag.
**Wishlist** — Failed downloads automatically queued for retry with auto-processing
**Mirrored Playlists** — Mirror from Spotify, Tidal, YouTube, Deezer and keep synced
- Auto-refresh detects source changes via URL/ID tracking in playlist metadata
- Discovery pipeline matches source tracks to user's primary metadata source (Spotify/iTunes/Deezer/Discogs)
- Auto Wing It fallback — tracks that fail all metadata APIs get stub metadata from the raw source title and flow through the normal download pipeline anyway
- Followed Spotify playlists that hit 403 errors fall back to public embed scraper
- Unmatch button on found tracks with DB persistence for mirrored playlists
- Automatic refresh detects changes, discovery pipeline matches metadata
- Followed Spotify playlists with 403 errors fall back to public embed scraper
**Local Profiles** — Multiple configuration profiles with isolated settings, watchlists, and playlists
@ -201,17 +176,7 @@ restores the retag.
**Database Storage Visualization** — Donut chart showing per-table storage breakdown
**Live Log Viewer** — Real-time terminal-style log viewer on Settings → Logs. Color-coded levels (DEBUG/INFO/WARNING/ERROR), live filter + search, switch between log files (app, post-processing, AcoustID, source reuse). Auto-scroll, copy, clear. Updates via WebSocket every 0.5s.
**Import System** — Tag-first matching, auto-grouped album cards, staging folder workflow
- Auto-Import worker: recursive scan, single file support, AcoustID fingerprinting fallback
- Confidence-gated: 90%+ auto-imports, 70-90% queued for review
**SoulSync Standalone Mode** — Use SoulSync without Plex, Jellyfin, or Navidrome
- Downloads and imports write directly to the library database
- Filesystem scanner for incremental and deep scan of Transfer folder
- Pre-populated enrichment IDs from download context (Spotify, Deezer, MusicBrainz)
- Select in Settings → Connections → Standalone
**Template Organization** — `$albumartist/$album/$track - $title` and 10+ variables
@ -240,43 +205,6 @@ docker-compose up -d
# Access at http://localhost:8008
```
### Release Channels
SoulSync publishes two Docker image tracks so you can choose your level of stability.
**Stable — `:latest`** (recommended for most users). Hand-promoted from the `dev` branch to `main` when a batch of changes is ready for release. Published to Docker Hub. Your `docker-compose.yml` pulls this by default — no changes needed.
```bash
docker pull boulderbadgedad/soulsync:latest
```
**Nightly — `:dev`**. Rebuilt every night from the `dev` branch (and on every push to dev). Published to GitHub Container Registry. Gets new features and bug fixes before they reach `:latest`, at the cost of occasional instability as changes settle. Good for early adopters, contributors validating their own merges, and anyone helping shake out bugs on Discord before a stable release.
To switch, edit `docker-compose.yml`:
```yaml
image: ghcr.io/nezreka/soulsync:dev
```
Then run `docker-compose pull && docker-compose up -d`.
Pinned dev builds are also published as `ghcr.io/nezreka/soulsync:dev-YYYYMMDD-<sha>` if you want to stick with an exact known-good snapshot.
**Version-tagged releases** (e.g. `:2.3`, `:2.4`) are permanent tags published on both registries when a stable release is promoted:
```bash
docker pull boulderbadgedad/soulsync:2.4
# or
docker pull ghcr.io/nezreka/soulsync:2.4
```
| You are... | Use |
|---|---|
| A typical user who wants things to work | `:latest` |
| Pinning to a specific version for stability | `:2.3`, `:2.4`, etc. |
| An early adopter who wants new features early and is OK reporting bugs | `:dev` |
| A contributor testing post-merge behavior | `:dev` or a pinned dev build |
### Unraid
SoulSync is available as an Unraid template. Install from Community Applications or manually add the template from:
@ -286,71 +214,23 @@ https://raw.githubusercontent.com/Nezreka/SoulSync/main/templates/soulsync.xml
PUID/PGID are exposed in the template — set them to match your Unraid permissions (default: 99/100 for nobody/users).
The template points at `boulderbadgedad/soulsync:latest` (stable) by default. To use the nightly `:dev` channel on Unraid, edit the container's **Repository** field to `ghcr.io/nezreka/soulsync:dev` after installing from the template.
### Python (No Docker)
```bash
git clone https://github.com/Nezreka/SoulSync
cd SoulSync
python -m pip install -r requirements.txt
# Build the React WebUI bundle used by the Python server.
# Docker does this automatically; Python installs must do it manually.
cd webui
npm ci
npm run build
cd ..
gunicorn -c gunicorn.conf.py wsgi:application
pip install -r requirements-webui.txt
python web_server.py
# Open http://localhost:8008
```
When updating a Python/no-Docker install with `git pull`, rebuild the WebUI before restarting SoulSync:
For local development and tests:
```bash
cd webui
npm ci
npm run build
cd ..
pip install -r requirements-dev.txt
pytest
```
If `webui/static/dist/.vite/manifest.json` is missing or stale, React-owned routes and route handoffs may not load correctly.
**YouTube streaming / music videos** need two extra things on bare-metal installs (Docker bundles both):
- **Deno** — yt-dlp now requires a JavaScript runtime to unlock YouTube formats. Without it, streams and music-video downloads fail with `Requested format is not available`. Install: `winget install DenoLand.Deno` (Windows) or see [deno.com](https://docs.deno.com/runtime/), then restart SoulSync.
- **yt-dlp nightly** — the stable release can lag months behind YouTube changes. If YouTube breaks, update with: `python -m pip install -U --pre "yt-dlp[default]"`
### Local Development
This is only for contributors working on the WebUI with hot reload. Normal Python/no-Docker installs should build once with `npm run build` as shown above, then run only Gunicorn.
For active frontend development, use two terminals so the backend and Vite stay independent:
1. Backend
```bash
python -m pip install -r requirements-dev.txt
gunicorn -c gunicorn.dev.conf.py wsgi:application
```
The dev Gunicorn config watches backend files and restarts the Python server when they change.
2. Frontend
```bash
cd webui
npm ci
npm run dev
```
Vite hot reloads the React side when you change webui files.
Run tests separately when needed:
```bash
python -m pytest
```
If you want a convenience launcher, `python dev.py` starts both halves together
on any OS. `./dev.sh` remains available as a Unix shell wrapper.
---
## Setup Guide
@ -360,7 +240,6 @@ on any OS. `./dev.sh` remains available as a Unix shell wrapper.
- **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases)) — required for Soulseek downloads
- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended for discovery
- **Media Server** (optional): Plex, Jellyfin, or Navidrome
- **Deno** (Python/no-Docker installs only): JavaScript runtime required by yt-dlp for YouTube streaming/music videos — `winget install DenoLand.Deno` or [deno.com](https://docs.deno.com/runtime/). Docker images bundle it.
- **Deezer ARL token** (optional): For Deezer downloads — get from browser cookies after logging into deezer.com
- **Tidal account** (optional): For Tidal downloads — authenticate via device flow in Settings
- **Qobuz account** (optional): For Qobuz downloads — email/password login in Settings
@ -392,9 +271,9 @@ Open SoulSync at `http://localhost:8008` and go to Settings.
**Download Source**: Choose your preferred source (Soulseek, Deezer, Tidal, Qobuz, HiFi, YouTube, or Hybrid)
**Paths**:
- **Input Folder**: Container path to slskd's download folder (e.g., `/app/downloads`)
- **Output Folder**: Where organized music goes (e.g., `/app/Transfer`)
- **Import Folder**: Optional folder for importing existing music (e.g., `/app/Staging`)
- **Download Path**: Container path to slskd's download folder (e.g., `/app/downloads`)
- **Transfer Path**: Where organized music goes (e.g., `/app/Transfer`)
- **Staging Path**: Optional import folder (e.g., `/app/Staging`)
**Media Server** (optional): Use your machine's actual IP (not `localhost` — that means inside the container)
@ -405,9 +284,9 @@ Open SoulSync at `http://localhost:8008` and go to Settings.
| Config | `/app/config` | Your config folder |
| Logs | `/app/logs` | Your logs folder |
| Database | `/app/data` | Named volume (recommended) |
| Input | `/app/downloads` | Same folder slskd downloads to |
| Output | `/app/Transfer` | Where organized music goes |
| Import | `/app/Staging` | Optional folder for importing music |
| Downloads | `/app/downloads` | Same folder slskd downloads to |
| Transfer | `/app/Transfer` | Where organized music goes |
| Staging | `/app/Staging` | Optional import folder |
**Important:** Use a named volume for the database (`soulsync_database:/app/data`). Direct host path mounts to `/app/data` can overwrite Python module files.
@ -455,36 +334,3 @@ Open SoulSync at `http://localhost:8008` and go to Settings.
- **Album Consistency** — pre-flight MusicBrainz release lookup before album downloads
- **Automation Engine** — event-driven workflows with signal chains and pipeline deployment
- **SoulID System** — deterministic cross-instance artist/album/track identifiers via track-verified API lookup
---
## Contributing
### Branch workflow
SoulSync uses a `dev` → `main` flow:
- **`main`** — release branch. `:latest` images auto-build from this. Only receives merges from `dev`.
- **`dev`** — integration branch. Nightly `:dev` images build from here. PRs land here first for validation before being promoted to `main`.
- **Feature branches** — branched from `dev`. PRs target `dev`.
### Opening a PR
1. Fork and clone the repo
2. Branch off `dev`: `git checkout -b fix/your-change dev`
3. Make your changes and commit
4. Push and open a PR against **`dev`** (not `main`)
5. CI (`build-and-test.yml`) runs ruff lint + compile + `python -m pytest` on your branch — wait for green
6. A maintainer reviews and merges
### Running locally
Use the [Local Development](#local-development) section above for the full repo-wide setup and the portable dev launcher.
For web UI work, see [webui/README.md](webui/README.md). It keeps the React-side notes close to the app while this file stays the single place for repo-wide dev instructions.
Ruff config lives in `pyproject.toml`. The ruleset is intentionally lenient — it catches real bugs (undefined names, import shadowing, closure-in-loop) without style nits.
### Reporting bugs / requesting features
Open an issue on GitHub. For user-side support, the Discord community is the fastest place to ask.

13
RELEASE_2.7.9_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.7.9** is out 🎉 a big one.
🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶

13
RELEASE_2.8.0_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
enjoy! 🎶

15
RELEASE_2.8.1_discord.md
View file

@ -1,15 +0,0 @@
**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
enjoy! 🎶

9
RELEASE_2.8.2_discord.md
View file

@ -1,9 +0,0 @@
**SoulSync 2.8.2** is out 🎉 a stability + performance release.
🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
enjoy! 🎶

144
SPEC_canonical_album_version.md
View file

@ -1,144 +0,0 @@
# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
**Status:** design only — no code yet.
**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
the user's actual files, so the Library Reorganizer, Track Number Repair, and
tagging/enrichment all agree on the same release. Today each re-resolves
independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
Believer=3; standard album mislinked to a deluxe release).
**Canonical-selection rule (decided):** *match the user's actual files.* The
canonical release is the candidate whose track count + per-track durations +
titles best fit what's on disk. Self-correcting: picks standard when you own the
standard, deluxe when you own the deluxe.
---
## Hard requirement: don't disrupt the running app
Every stage below is **additive and dormant until explicitly consumed**, and
every consumer **falls back to today's behavior when no canonical is set**. So:
- albums with no resolved canonical behave EXACTLY as they do now;
- each stage is independently shippable and reversible;
- nothing big-bangs.
---
## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
### Schema (additive, nullable → migration-safe)
Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
existing column-exists checks; see [[db-schema-review]] migration-safety notes):
- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
- `canonical_album_id TEXT`
- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
- `canonical_resolved_at TIMESTAMP`
All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
backfill in this stage. No reads in this stage.
### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
```
score_release_against_files(file_tracks, release_tracks) -> float
pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
```
- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
- `release_tracks`: a candidate release's tracklist (same shape).
- Scoring (tunable weights):
- **track-count fit** — exact match strongly preferred; |Δcount| penalized.
- **duration alignment** — greedily match each file to its closest release
track by duration (within a tolerance, e.g. ±3s); reward coverage.
- **title overlap** — token/fuzzy overlap as a tiebreaker.
- **graceful degradation** — if a source gives no per-track durations, fall
back to count + title only (never crash, never force-pick).
- Returns the best candidate + score, or (None, 0) when nothing clears a floor
(so we never pin a bad guess — leave it unresolved, consumers fall back).
### Tests (extreme, like the rest of this codebase)
- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
- same album, 17 files → picks deluxe.
- duration disambiguation when track counts tie (e.g. radio edit vs album).
- missing-duration source → count+title fallback still picks sanely.
- no candidate clears the floor → (None, 0).
- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
whichever the files actually match.
**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
them yet. Provably zero behavior change.**
---
## Stage 2 — Resolver populates canonical (writes, still no consumers)
A function `resolve_canonical_for_album(album_id, db, ...)`:
1. Gather on-disk file metadata for the album (durations/titles) via the
library's known file paths.
2. Gather candidate releases: every source the album has an ID for
(spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
case — sibling editions discoverable from those. Fetch each tracklist
(cached, rate-limited).
3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
on the album row if it clears the floor.
Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
unchanged — this stage only populates the new columns. Reversible: clearing the
columns reverts to unresolved.
Tests: resolver picks the right release for the standard/deluxe fixtures; stores
nothing when below floor; idempotent re-resolve.
Cost note: fetching multiple candidate releases = more API calls. Mitigate via
cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
progress so it's not silent.
---
## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
In `library_reorganize._resolve_source`: if the album has
`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
current `get_source_priority` walk. One-line precedence change, fully gated on
non-NULL.
Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
to today. Re-run the existing reorganize battery (148 tests) — must stay green.
**This alone fixes #767-Bug2** (a standard album whose files match the standard
release pins the standard, so reorganize stops targeting the deluxe folder).
---
## Stage 4 — Track Number Repair reads canonical (closes #765)
In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
everything) — if the album has a canonical `(source, album_id)`, use it. The
existing 6-level cascade stays as the fallback for albums with no canonical
(preserves its all-01-album rescue ability — the regression risk we refused to
take in the reactive fix).
Now both tools resolve the SAME release → same track numbers → no contradiction.
Tests: canonical present → both tools agree (shared-release test); canonical
NULL → existing cascade unchanged.
---
## Risks & mitigations
- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
only-on-(re)enrich, progress-logged.
- **Sources without per-track durations** → scorer degrades to count+title.
- **Schema migration** → additive nullable columns only; idempotent guards.
- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
stored for inspection/re-resolve; manual override possible later.
- **Backward-compat** → every consumer falls back to today's path when NULL, so
un-resolved albums (incl. all existing albums until backfilled) are unaffected.
## Out of scope (for now)
- Per-album manual version override UI (can layer on later — the columns support it).
- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
once they share the canonical.
## Suggested order to build
1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
after any stage and the app is consistent (just with fewer consumers wired).

13
Support/README-Docker.md
View file

@ -1,15 +1,5 @@
# SoulSync WebUI - Docker Deployment Guide
## Release Channels
SoulSync publishes two Docker image tracks:
- **Stable — `boulderbadgedad/soulsync:latest`** (Docker Hub). Hand-promoted from the `dev` branch when a batch of changes is ready. Default in `docker-compose.yml`.
- **Nightly — `ghcr.io/nezreka/soulsync:dev`** (GHCR). Rebuilt every night and on every push to `dev`. Faster access to new features at the cost of occasional instability.
- **Version-tagged — `:2.3`, `:2.4`, etc.** on both registries for pinning to a specific release.
To switch a running install to the nightly channel, edit the `image:` line in `docker-compose.yml` to `ghcr.io/nezreka/soulsync:dev` and run `docker-compose pull && docker-compose up -d`. See the [main README](../README.md#release-channels) for the full channel guide.
## 🐳 Quick Start
### Prerequisites
@ -73,7 +63,6 @@ environment:
- FLASK_ENV=production # Flask environment
- PYTHONPATH=/app # Python path
- SOULSYNC_CONFIG_PATH=/app/config/config.json # Config file location
- SOULSYNC_LOG_LEVEL=INFO # Optional startup log level override, takes precedence over the UI-configured log level
- TZ=America/New_York # Timezone
```
@ -279,4 +268,4 @@ services:
- [ ] Configure firewall rules
- [ ] Set up backup strategy
- [ ] Test health checks
- [ ] Verify external service connectivity
- [ ] Verify external service connectivity

134
Support/REVERSE-PROXY.md
View file

@ -1,134 +0,0 @@
# Running SoulSync behind a reverse proxy (nginx / Caddy / Traefik)
Putting SoulSync behind a reverse proxy lets you serve it over **HTTPS** and — the
important part — put **authentication** in front of it before exposing it to the
internet. This guide covers the safe setup.
> **The golden rule:** the safest way to expose *any* self-hosted app publicly is
> to require authentication at the proxy (an auth layer), **not** to rely on the
> app's own protection. SoulSync's launch PIN is a useful fallback, but it is not
> a substitute for a real auth layer on a public instance.
---
## 1. Turn on reverse-proxy mode
By default SoulSync does **not** trust proxy headers (so a direct client can't spoof
its IP or pretend the connection is HTTPS). If you're behind a proxy that
terminates TLS, turn on **Settings → Security → "Behind a reverse proxy"** and
**restart SoulSync** (this option applies at startup).
When enabled, SoulSync:
- trusts `X-Forwarded-For/Proto/Host/Port` from **one** proxy hop (correct client
IP, HTTPS detection, redirects),
- marks its session cookie `Secure` (HTTPS-only) + `SameSite=Lax`, and
- sends conservative security headers (`X-Content-Type-Options: nosniff`,
`X-Frame-Options: SAMEORIGIN`, `Strict-Transport-Security`). No CSP is set — tune
one at your proxy if you want it.
**Leave it off if you access SoulSync directly over http:// on your LAN** — turning
it on would make the session cookie HTTPS-only and break plain-HTTP access. With it
off, none of the above applies and SoulSync behaves exactly as before.
> The launch PIN is also brute-force limited (10 wrong attempts from an IP → a
> short cooldown), regardless of this setting — a correct PIN is never affected.
Restart SoulSync after changing it.
---
## 2. nginx
SoulSync uses WebSockets (Socket.IO), so the `Upgrade`/`Connection` headers are
**required** — without them live updates silently stop working.
```nginx
server {
listen 443 ssl;
server_name soulsync.example.com;
ssl_certificate /etc/letsencrypt/live/soulsync.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/soulsync.example.com/privkey.pem;
# Large library scans / uploads
client_max_body_size 0;
location / {
proxy_pass http://127.0.0.1:8008;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
# Required for Socket.IO / live updates
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 3600s; # long-running scans
proxy_send_timeout 3600s;
}
}
```
---
## 3. Caddy
Caddy handles TLS automatically and proxies WebSockets out of the box:
```caddy
soulsync.example.com {
reverse_proxy 127.0.0.1:8008
}
```
Caddy sets `X-Forwarded-*` for you. (Add an auth provider directive if you want
auth at the proxy — see below.)
---
## 4. Traefik
Traefik proxies WebSockets automatically and forwards the headers. Point a router
at the SoulSync service on port `8008` with your TLS resolver; no extra WebSocket
config is needed.
---
## 5. Add authentication in front (recommended for public instances)
Pick one:
- **Auth proxy** — [Authelia](https://www.authelia.com/),
[Authentik](https://goauthentik.io/), or
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). These sit in front
of SoulSync and force a login (with 2FA) before any request reaches it. Best
option for internet exposure.
SoulSync can **trust the proxy's authenticated-user header** so the launch PIN is
skipped once the proxy has logged you in. Set the header name in **Settings →
Security → "Auth proxy user header"** (e.g. `Remote-User`).
> ⚠️ **Only enable this behind a proxy you control that STRIPS any client-supplied
> copy of that header.** Otherwise a direct visitor could send `Remote-User: admin`
> and walk straight in. It's **off by default** — an unset header name means
> SoulSync ignores the header entirely (a spoofed one does nothing).
- **HTTP Basic Auth** — quick and simple (nginx `auth_basic` / Caddy `basicauth`).
Better than nothing; weaker than an auth proxy.
- **SoulSync launch PIN** — set an admin PIN in Settings. Enforced server-side, so
it can't be bypassed by hitting the API directly — but it's a shared PIN, so
treat it as a fallback, not your only gate.
---
## Troubleshooting
- **Live updates / progress bars don't move** → the WebSocket `Upgrade`/`Connection`
headers are missing (nginx) or your proxy is buffering. Check section 2.
- **Login won't stick / "session expired"** → you enabled `trust_reverse_proxy` but
are reaching SoulSync over plain `http://`. The session cookie is now HTTPS-only;
use `https://`, or turn the setting off for direct HTTP access.
- **Scans time out** → raise `proxy_read_timeout` / `proxy_send_timeout`.

8
api/__init__.py
View file

@ -42,8 +42,6 @@ def create_api_blueprint():
from .retag import register_routes as reg_retag
from .listenbrainz import register_routes as reg_listenbrainz
from .cache import register_routes as reg_cache
from .request import register_routes as reg_request
from .request import start_cleanup_thread as _start_request_cleanup
# ---- rate-limit only /api/v1 routes (not the whole app) ----
limiter.limit("60 per minute")(bp)
@ -61,12 +59,6 @@ def create_api_blueprint():
reg_retag(bp)
reg_listenbrainz(bp)
reg_cache(bp)
reg_request(bp)
# Start the periodic cleanup timer for in-memory request tracking so
# idle periods don't leave stale entries in memory. Idempotent across
# calls; safe with multi-blueprint registration.
_start_request_cleanup()
# ---- error handlers (scoped to this Blueprint) ----
@bp.errorhandler(400)

33
api/auth.py
View file

@ -4,9 +4,8 @@ API key authentication for the SoulSync public API.
import hashlib
import secrets
import threading
import uuid
from datetime import datetime, timedelta, timezone
from datetime import datetime, timezone
from functools import wraps
from flask import request, current_app
@ -14,27 +13,6 @@ from flask import request, current_app
from .helpers import api_error
# Throttle persistence of `last_used_at` so every authenticated request
# does not rewrite the full app config. Maps key_hash -> last-persisted datetime.
_USAGE_WRITE_INTERVAL = timedelta(minutes=15)
_last_persisted_usage: dict[str, datetime] = {}
_usage_lock = threading.Lock()
def _should_persist_usage(key_hash: str, now: datetime) -> bool:
"""Return True if `last_used_at` for the given key should be written to disk.
Thread-safe: tracks the last write per key hash in memory and only returns
True once per `_USAGE_WRITE_INTERVAL`.
"""
with _usage_lock:
previous = _last_persisted_usage.get(key_hash)
if previous is None or (now - previous) >= _USAGE_WRITE_INTERVAL:
_last_persisted_usage[key_hash] = now
return True
return False
def generate_api_key(label=""):
"""Generate a new API key.
@ -89,12 +67,9 @@ def require_api_key(f):
if not matched:
return api_error("INVALID_KEY", "Invalid API key.", 403)
# Update last-used timestamp (best-effort, throttled to avoid rewriting
# the full app config on every authenticated request).
now = datetime.now(timezone.utc)
matched["last_used_at"] = now.isoformat()
if _should_persist_usage(key_hash, now):
config_mgr.set("api_keys", stored_keys)
# Update last-used timestamp (best-effort)
matched["last_used_at"] = datetime.now(timezone.utc).isoformat()
config_mgr.set("api_keys", stored_keys)
return f(*args, **kwargs)

70
api/downloads.py
View file

@ -5,7 +5,6 @@ Download management endpoints — list, cancel active downloads.
from flask import request, current_app
from .auth import require_api_key
from .helpers import api_success, api_error
from core.runtime_state import download_tasks, tasks_lock
def _serialize_download(task_id, task):
@ -41,66 +40,16 @@ def register_routes(bp):
@bp.route("/downloads", methods=["GET"])
@require_api_key
def list_downloads():
"""List download tasks with optional filtering and pagination.
Query params:
status: comma-separated statuses to include (e.g. "downloading,queued").
Default includes all.
limit: max tasks to return (default 100, max 500).
offset: skip the first N tasks (default 0).
Response includes `total` (post-filter count) so clients can paginate
without fetching everything. Tasks are sorted by `status_change_time`
descending so newest/in-flight tasks appear first.
"""
"""List active and recent download tasks."""
try:
# Parse pagination params
try:
limit = int(request.args.get("limit", 100))
except (TypeError, ValueError):
limit = 100
try:
offset = int(request.args.get("offset", 0))
except (TypeError, ValueError):
offset = 0
# Clamp to sensible bounds
limit = max(1, min(limit, 500))
offset = max(0, offset)
from web_server import download_tasks, tasks_lock
status_param = request.args.get("status", "").strip()
status_filter = (
{s.strip() for s in status_param.split(",") if s.strip()}
if status_param
else None
)
# Snapshot under the lock, then sort/slice outside.
tasks = []
with tasks_lock:
snapshot = list(download_tasks.items())
for task_id, task in download_tasks.items():
tasks.append(_serialize_download(task_id, task))
if status_filter:
snapshot = [
(tid, t) for tid, t in snapshot
if (t.get("status") or "") in status_filter
]
# Sort newest-first by status_change_time; fall back to string id
# so ordering is stable when timestamps are missing or tied.
snapshot.sort(
key=lambda item: (item[1].get("status_change_time") or "", item[0]),
reverse=True,
)
total = len(snapshot)
page = snapshot[offset:offset + limit]
tasks = [_serialize_download(tid, t) for tid, t in page]
return api_success({
"downloads": tasks,
"total": total,
"limit": limit,
"offset": offset,
})
return api_success({"downloads": tasks})
except ImportError:
return api_error("NOT_AVAILABLE", "Download tracking not available.", 501)
except Exception as e:
@ -121,13 +70,10 @@ def register_routes(bp):
try:
from utils.async_helpers import run_async
soulseek = current_app.soulsync.get("download_orchestrator")
soulseek = current_app.soulsync.get("soulseek_client")
if not soulseek:
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
current_app.logger.info(
f"[CancelTrigger:api.public_cancel] download_id={download_id} username={username}"
)
ok = run_async(soulseek.cancel_download(download_id, username, remove=True))
if ok:
return api_success({"message": "Download cancelled."})
@ -141,7 +87,7 @@ def register_routes(bp):
"""Cancel all active downloads and clear completed ones."""
try:
from utils.async_helpers import run_async
soulseek = current_app.soulsync.get("download_orchestrator")
soulseek = current_app.soulsync.get("soulseek_client")
if not soulseek:
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)

11
api/library.py
View file

@ -186,8 +186,15 @@ def register_routes(bp):
try:
db = get_database()
tracks = db.api_search_tracks(title=title, artist=artist, limit=limit)
return api_success({"tracks": [serialize_track(t, fields) for t in tracks]})
tracks = db.search_tracks(title=title, artist=artist, limit=limit)
if not tracks:
return api_success({"tracks": []})
# Re-query by IDs to get full row data
track_ids = [t.id for t in tracks]
full_tracks = db.api_get_tracks_by_ids(track_ids)
return api_success({"tracks": [serialize_track(t, fields) for t in full_tracks]})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)

2
api/playlists.py
View file

@ -138,7 +138,7 @@ def register_routes(bp):
try:
# Forward to the internal sync endpoint
import requests as http_requests
internal_url = "http://127.0.0.1:8008/api/sync/start"
internal_url = f"http://127.0.0.1:8008/api/sync/start"
resp = http_requests.post(internal_url, json={
"playlist_id": playlist_id,
"playlist_name": playlist_name,

237
api/request.py
View file

@ -1,237 +0,0 @@
"""
Inbound music request endpoint — accept a search query from external sources
(Discord bots, curl, etc.) and trigger the search-match-download pipeline.
"""
import threading
import uuid
from datetime import datetime, timedelta
import requests as http_requests
from flask import request, current_app
from utils.logging_config import get_logger
from .auth import require_api_key
from .helpers import api_success, api_error
logger = get_logger("api_request")
# In-memory request tracking (ephemeral — survives until restart)
_pending_requests = {}
_requests_lock = threading.Lock()
# Max age before auto-cleanup
_MAX_REQUEST_AGE = timedelta(hours=1)
# How often the background cleanup timer runs. Short enough to keep memory
# bounded during idle periods, long enough that slow-polling external clients
# still see their request for close to the TTL.
_CLEANUP_INTERVAL_SECONDS = 300 # 5 minutes
# Guards for the singleton background cleanup thread.
_cleanup_thread: "threading.Thread | None" = None
_cleanup_stop_event = threading.Event()
_cleanup_thread_lock = threading.Lock()
def _cleanup_old_requests():
"""Remove requests older than 1 hour to prevent unbounded growth."""
cutoff = datetime.now() - _MAX_REQUEST_AGE
with _requests_lock:
expired = [rid for rid, r in _pending_requests.items()
if r.get('created_at', datetime.now()) < cutoff]
for rid in expired:
del _pending_requests[rid]
return len(expired) if expired else 0
def _cleanup_loop():
"""Background thread: periodically evict expired requests."""
while not _cleanup_stop_event.is_set():
# wait() returns True if the event was set (shutdown), False on timeout
if _cleanup_stop_event.wait(timeout=_CLEANUP_INTERVAL_SECONDS):
return
try:
removed = _cleanup_old_requests()
if removed:
logger.debug(f"Request cleanup: evicted {removed} stale entries")
except Exception as e:
logger.warning(f"Request cleanup loop error: {e}")
def start_cleanup_thread() -> bool:
"""Start the background cleanup timer once per process.
Returns True if a new thread was started, False if one was already
running. Safe to call multiple times; callers in multi-worker setups
should still gate on worker identity if they want exactly one thread
across the entire deployment.
"""
global _cleanup_thread
with _cleanup_thread_lock:
if _cleanup_thread is not None and _cleanup_thread.is_alive():
return False
_cleanup_stop_event.clear()
_cleanup_thread = threading.Thread(
target=_cleanup_loop,
name="api-request-cleanup",
daemon=True,
)
_cleanup_thread.start()
logger.info("Started api/request cleanup timer (interval=%ss)" % _CLEANUP_INTERVAL_SECONDS)
return True
def stop_cleanup_thread(timeout: float = 2.0) -> None:
"""Signal the cleanup thread to exit. Used in tests and shutdown paths."""
global _cleanup_thread
with _cleanup_thread_lock:
thread = _cleanup_thread
_cleanup_stop_event.set()
if thread is not None and thread.is_alive():
thread.join(timeout=timeout)
with _cleanup_thread_lock:
_cleanup_thread = None
_cleanup_stop_event.clear()
def _run_search_and_download(request_id, query, notify_url):
"""Background worker: search, download, update status, notify."""
try:
from utils.async_helpers import run_async
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'searching'
soulseek = current_app._get_current_object().soulsync.get('download_orchestrator')
if not soulseek:
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'failed'
_pending_requests[request_id]['error'] = 'Download source not configured'
return
result = run_async(soulseek.search_and_download_best(query))
with _requests_lock:
if request_id in _pending_requests:
if result:
_pending_requests[request_id]['status'] = 'downloading'
_pending_requests[request_id]['download_id'] = result
else:
_pending_requests[request_id]['status'] = 'not_found'
_pending_requests[request_id]['error'] = 'No match found'
_pending_requests[request_id]['completed_at'] = datetime.now().isoformat()
# Send notification to callback URL if provided
if notify_url:
try:
with _requests_lock:
payload = dict(_pending_requests.get(request_id, {}))
# Remove non-serializable datetime
payload.pop('created_at', None)
http_requests.post(notify_url, json=payload, timeout=10)
except Exception as e:
logger.warning(f"Failed to POST to notify_url {notify_url}: {e}")
except Exception as e:
logger.error(f"Request {request_id} failed: {e}")
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'failed'
_pending_requests[request_id]['error'] = str(e)
_pending_requests[request_id]['completed_at'] = datetime.now().isoformat()
def register_routes(bp):
@bp.route("/request", methods=["POST"])
@require_api_key
def create_request():
"""Accept a music search query and trigger the download pipeline.
Body:
query (str, required): Search query, e.g. "Artist - Track Name"
notify_url (str, optional): URL to POST results to on completion
metadata (dict, optional): Passthrough data included in automation events
Returns 202 with request_id for async status polling.
"""
body = request.get_json(silent=True) or {}
query = (body.get("query") or "").strip()
if not query:
return api_error("BAD_REQUEST", "Missing 'query' in request body.", 400)
# Cleanup old requests on each new request
_cleanup_old_requests()
request_id = str(uuid.uuid4())
notify_url = (body.get("notify_url") or "").strip() or None
metadata = body.get("metadata") or {}
with _requests_lock:
_pending_requests[request_id] = {
'request_id': request_id,
'query': query,
'status': 'queued',
'created_at': datetime.now(),
'completed_at': None,
'download_id': None,
'error': None,
}
# Emit webhook_received event so automation engine triggers fire
engine = current_app.soulsync.get('automation_engine')
if engine:
engine.emit('webhook_received', {
'query': query,
'request_id': request_id,
'source': 'api',
'metadata': metadata,
})
# Start background search-download (Feature A: works without automations)
app = current_app._get_current_object()
thread = threading.Thread(
target=lambda: _run_with_app_context(app, request_id, query, notify_url),
daemon=True
)
thread.start()
logger.info(f"Music request queued: '{query}' (id={request_id})")
return api_success({
"request_id": request_id,
"status": "queued",
"query": query,
}), 202
@bp.route("/request/<request_id>", methods=["GET"])
@require_api_key
def get_request_status(request_id):
"""Check the status of a music request.
Returns current status: queued → searching → downloading → completed/not_found/failed
"""
with _requests_lock:
req = _pending_requests.get(request_id)
if not req:
return api_error("NOT_FOUND", "Request not found or expired.", 404)
return api_success({
"request_id": req['request_id'],
"query": req['query'],
"status": req['status'],
"download_id": req.get('download_id'),
"error": req.get('error'),
"completed_at": req.get('completed_at'),
})
def _run_with_app_context(app, request_id, query, notify_url):
"""Run the background worker within Flask app context."""
with app.app_context():
_run_search_and_download(request_id, query, notify_url)

8
api/search.py
View file

@ -2,14 +2,10 @@
Search endpoints — search external sources (Spotify, iTunes, Hydrabase).
"""
import logging
from flask import request, current_app
from .auth import require_api_key
from .helpers import api_success, api_error
logger = logging.getLogger(__name__)
def register_routes(bp):
@ -42,8 +38,8 @@ def register_routes(bp):
if hydra_results:
tracks = [_serialize_track(t) for t in hydra_results]
return api_success({"tracks": tracks, "source": "hydrabase"})
except Exception as e:
logger.debug("hydrabase search failed: %s", e)
except Exception:
pass
spotify = ctx.get("spotify_client")
from core.metadata_service import get_primary_source, get_primary_client

27
api/serializers.py
View file

@ -287,12 +287,12 @@ def serialize_watchlist_artist(obj, fields: Optional[Set[str]] = None) -> dict:
def serialize_wishlist_track(obj, fields: Optional[Set[str]] = None) -> dict:
"""Standardized wishlist track serialization."""
d = _to_dict(obj)
track_data = d.get("track_data", d.get("spotify_data", {}))
if isinstance(track_data, str):
spotify_data = d.get("spotify_data", {})
if isinstance(spotify_data, str):
try:
track_data = json.loads(track_data)
spotify_data = json.loads(spotify_data)
except (json.JSONDecodeError, TypeError):
track_data = {}
spotify_data = {}
source_info = d.get("source_info")
if isinstance(source_info, str):
@ -303,23 +303,17 @@ def serialize_wishlist_track(obj, fields: Optional[Set[str]] = None) -> dict:
result = {
"id": d.get("id"),
"track_id": d.get("track_id") or d.get("spotify_track_id") or d.get("id"),
"spotify_track_id": d.get("spotify_track_id"),
"track_name": (
track_data.get("name", "Unknown") if isinstance(track_data, dict) else d.get("track_name", "Unknown")
),
"track_name": spotify_data.get("name", "Unknown") if isinstance(spotify_data, dict) else "Unknown",
"artist_name": ", ".join(
a.get("name", "") if isinstance(a, dict) else str(a)
for a in track_data.get("artists", [])
) if isinstance(track_data, dict) and isinstance(track_data.get("artists"), list) else "",
a.get("name", "") for a in spotify_data.get("artists", [])
) if isinstance(spotify_data, dict) and isinstance(spotify_data.get("artists"), list) else "",
"album_name": (
track_data.get("album", {}).get("name")
if isinstance(track_data, dict) and isinstance(track_data.get("album"), dict)
spotify_data.get("album", {}).get("name")
if isinstance(spotify_data, dict) and isinstance(spotify_data.get("album"), dict)
else None
),
"track_data": track_data,
"spotify_data": track_data,
"provider": track_data.get("provider") if isinstance(track_data, dict) else d.get("provider"),
"spotify_data": spotify_data,
"failure_reason": d.get("failure_reason"),
"retry_count": d.get("retry_count", 0),
"last_attempted": _isoformat(d.get("last_attempted")),
@ -368,7 +362,6 @@ def serialize_similar_artist(obj, fields: Optional[Set[str]] = None) -> dict:
"source_artist_id": d.get("source_artist_id"),
"similar_artist_spotify_id": d.get("similar_artist_spotify_id"),
"similar_artist_itunes_id": d.get("similar_artist_itunes_id"),
"similar_artist_musicbrainz_id": d.get("similar_artist_musicbrainz_id"),
"similar_artist_name": d.get("similar_artist_name"),
"similarity_rank": d.get("similarity_rank"),
"occurrence_count": d.get("occurrence_count"),

13
api/system.py
View file

@ -2,15 +2,12 @@
System endpoints — status, activity feed, stats.
"""
import logging
import time
from flask import current_app
from .auth import require_api_key
from .helpers import api_success, api_error
logger = logging.getLogger(__name__)
def register_routes(bp):
@ -29,7 +26,7 @@ def register_routes(bp):
spotify = ctx.get("spotify_client")
spotify_ok = bool(spotify and spotify.is_authenticated())
soulseek = ctx.get("download_orchestrator")
soulseek = ctx.get("soulseek_client")
soulseek_ok = bool(soulseek)
hydrabase = ctx.get("hydrabase_client")
@ -38,8 +35,8 @@ def register_routes(bp):
try:
ws, _ = hydrabase.get_ws_and_lock()
hydrabase_ok = ws is not None and ws.connected
except Exception as e:
logger.debug("hydrabase status probe failed: %s", e)
except Exception:
pass
return api_success({
"uptime": f"{hours}h {minutes}m {seconds}s",
@ -58,7 +55,7 @@ def register_routes(bp):
def system_activity():
"""Recent activity feed."""
try:
from core.runtime_state import activity_feed
from web_server import activity_feed
items = list(activity_feed) if activity_feed else []
return api_success({"activities": items})
except Exception as e:
@ -77,7 +74,7 @@ def register_routes(bp):
# Active download count
download_count = 0
try:
from core.runtime_state import download_tasks, tasks_lock
from web_server import download_tasks, tasks_lock
with tasks_lock:
download_count = sum(
1 for t in download_tasks.values()

42
api/wishlist.py
View file

@ -19,19 +19,21 @@ def register_routes(bp):
fields = parse_fields(request)
profile_id = parse_profile_id(request)
category_filter = category if category in ("singles", "albums") else None
try:
from database.music_database import get_database
db = get_database()
offset = (page - 1) * limit
tracks = db.get_wishlist_tracks(
profile_id=profile_id,
category=category_filter,
limit=limit,
offset=offset,
)
total = db.get_wishlist_count(profile_id=profile_id, category=category_filter)
raw_tracks = db.get_wishlist_tracks(profile_id=profile_id)
# Category filter
if category in ("singles", "albums"):
raw_tracks = [
t for t in raw_tracks
if _track_category(t) == category
]
total = len(raw_tracks)
start = (page - 1) * limit
tracks = raw_tracks[start:start + limit]
return api_success(
{"tracks": [serialize_wishlist_track(t, fields) for t in tracks]},
@ -45,16 +47,16 @@ def register_routes(bp):
def add_to_wishlist():
"""Add a track to the wishlist.
Body: {"track_data": {...}, "failure_reason": "...", "source_type": "..."}
Body: {"spotify_track_data": {...}, "failure_reason": "...", "source_type": "..."}
"""
body = request.get_json(silent=True) or {}
track_data = body.get("track_data") or body.get("spotify_track_data")
track_data = body.get("spotify_track_data")
reason = body.get("failure_reason", "Added via API")
source_type = body.get("source_type", "api")
profile_id = parse_profile_id(request)
if not track_data:
return api_error("BAD_REQUEST", "Missing 'track_data' in body.", 400)
return api_error("BAD_REQUEST", "Missing 'spotify_track_data' in body.", 400)
try:
from database.music_database import get_database
@ -74,7 +76,7 @@ def register_routes(bp):
@bp.route("/wishlist/<track_id>", methods=["DELETE"])
@require_api_key
def remove_from_wishlist(track_id):
"""Remove a track from the wishlist by its track ID."""
"""Remove a track from the wishlist by its Spotify track ID."""
profile_id = parse_profile_id(request)
try:
from database.music_database import get_database
@ -102,3 +104,15 @@ def register_routes(bp):
return api_error("NOT_AVAILABLE", "Wishlist processing function not available.", 501)
except Exception as e:
return api_error("WISHLIST_ERROR", str(e), 500)
def _track_category(track):
"""Determine if a wishlist track is a single or album track."""
album_type = ""
if isinstance(track, dict):
sd = track.get("spotify_data", {})
if isinstance(sd, dict):
album = sd.get("album", {})
if isinstance(album, dict):
album_type = album.get("album_type", "")
return "albums" if album_type == "album" else "singles"

783
beatport_unified_scraper.py
View file

File diff suppressed because it is too large Load diff

10
config/config.example.json
View file

@ -44,8 +44,7 @@
},
"metadata_enhancement": {
"enabled": true,
"embed_album_art": true,
"single_to_album": false
"embed_album_art": true
},
"file_organization": {
"enabled": true,
@ -57,11 +56,6 @@
"playlist_path": "$playlist/$artist - $title"
}
},
"import": {
"staging_path": "./Staging",
"replace_lower_quality": false,
"folder_artist_override": true
},
"lossy_copy": {
"enabled": false,
"bitrate": "320",
@ -73,4 +67,4 @@
"listenbrainz": {
"token": "LISTENBRAINZ_TOKEN"
}
}
}

436
config/settings.py
View file

@ -2,18 +2,11 @@ import copy
import json
import os
import sqlite3
import time
from typing import Dict, Any, Optional
from cryptography.fernet import Fernet, InvalidToken
from pathlib import Path
from utils.logging_config import get_logger
logger = get_logger("config")
class ConfigManager:
_VALID_LOG_LEVELS = frozenset({"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"})
def __init__(self, config_path: str = "config/config.json"):
# Determine strict absolute path to settings.py directory to help resolve config.json
# This handles cases where CWD is different (e.g. running from /Users vs /Users/project)
@ -40,7 +33,7 @@ class ConfigManager:
# Default to project path even if it doesn't exist yet (for creation/fallback)
self.config_path = project_path
logger.info(f"ConfigManager initialized with path: {self.config_path}")
print(f"ConfigManager initialized with path: {self.config_path}")
self.config_data: Dict[str, Any] = {}
self._fernet: Optional[Fernet] = None
@ -52,7 +45,7 @@ class ConfigManager:
else:
self.database_path = self.base_dir / "database" / "music_library.db"
logger.info(f"Database path set to: {self.database_path}")
print(f"Database path set to: {self.database_path}")
self.load_config(str(self.config_path))
@ -66,12 +59,6 @@ class ConfigManager:
self._load_config()
# Placeholder shipped to the browser in place of a configured secret
# (#832 follow-up). The settings UI shows it as masked dots; if it's
# round-tripped back on save, ``set()`` treats it as "keep existing" so the
# real value is never overwritten by the mask.
REDACTED_SENTINEL = '__redacted_unchanged__'
# Dot-notation paths to sensitive config values that must be encrypted at rest.
# Paths pointing to dicts encrypt the entire dict as a JSON blob.
_SENSITIVE_PATHS = frozenset({
@ -93,10 +80,6 @@ class ConfigManager:
'soulseek.api_key',
'deezer_download.arl',
'lidarr_download.api_key',
'prowlarr.api_key',
'torrent_client.password',
'usenet_client.api_key',
'usenet_client.password',
# Enrichment services
'listenbrainz.token',
'acoustid.api_key',
@ -124,7 +107,7 @@ class ConfigManager:
try:
import shutil
shutil.move(str(old_key_file), str(key_file))
logger.info(f"Moved encryption key to {key_file}")
print(f"[MIGRATE] Moved encryption key to {key_file}")
except Exception:
key_file = old_key_file # Fall back to old location
if key_file.exists():
@ -172,10 +155,8 @@ class ConfigManager:
return decrypted
except InvalidToken:
# Key mismatch — encrypted with a different key (key file deleted/replaced)
logger.error(
"Failed to decrypt a config value — encryption key may have changed. "
"Re-enter credentials in Settings or restore the original .encryption_key file."
)
print(f"[ERROR] Failed to decrypt a config value — encryption key may have changed. "
f"Re-enter credentials in Settings or restore the original .encryption_key file.")
return value
except Exception:
return value
@ -231,7 +212,7 @@ class ConfigManager:
"""Re-save config to encrypt any plaintext sensitive values still in the DB."""
try:
# Read raw DB content to check if any sensitive value is still plaintext
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path))
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
row = cursor.fetchone()
@ -262,24 +243,11 @@ class ConfigManager:
needs_migration = True
break
if needs_migration:
logger.info("Encrypting sensitive config values at rest...")
print("[MIGRATE] Encrypting sensitive config values at rest...")
self._save_to_database(self.config_data)
logger.info("Sensitive config values encrypted successfully")
print("[OK] Sensitive config values encrypted successfully")
except Exception as e:
logger.warning(f"Could not migrate encryption: {e}")
def _connect_db(self) -> sqlite3.Connection:
"""Open a configured SQLite connection for the config DB.
Centralizes pragma setup so every connection gets WAL mode,
a 30s busy timeout, and synchronous=NORMAL (the safe pairing
with WAL that avoids unnecessary fsyncs on slow disks).
"""
conn = sqlite3.connect(str(self.database_path), timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=30000")
conn.execute("PRAGMA synchronous=NORMAL")
return conn
print(f"[WARN] Could not migrate encryption: {e}")
def _ensure_database_exists(self):
"""Ensure database file and metadata table exist"""
@ -288,7 +256,8 @@ class ConfigManager:
self.database_path.parent.mkdir(parents=True, exist_ok=True)
# Connect to database (creates file if it doesn't exist)
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path), timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL")
cursor = conn.cursor()
# Create metadata table if it doesn't exist
@ -302,7 +271,7 @@ class ConfigManager:
conn.commit()
conn.close()
except Exception as e:
logger.warning(f"Could not ensure database exists: {e}")
print(f"Warning: Could not ensure database exists: {e}")
def _load_from_database(self) -> Optional[Dict[str, Any]]:
"""Load configuration from database, decrypting sensitive values."""
@ -310,7 +279,8 @@ class ConfigManager:
try:
self._ensure_database_exists()
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path), timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL")
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
row = cursor.fetchone()
@ -319,77 +289,20 @@ class ConfigManager:
config_data = json.loads(row[0])
# Decrypt sensitive values (gracefully handles plaintext migration)
config_data = self._decrypt_sensitive(config_data)
logger.info("Configuration loaded from database")
print("[OK] Configuration loaded from database")
return config_data
else:
return None
except Exception as e:
logger.warning(f"Could not load config from database: {e}")
print(f"Warning: Could not load config from database: {e}")
return None
finally:
if conn:
conn.close()
def _load_stored_log_level(self) -> Optional[str]:
"""Load the persisted UI log level preference, if one exists."""
conn = None
try:
self._ensure_database_exists()
conn = self._connect_db()
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'log_level'")
row = cursor.fetchone()
if not row or not row[0]:
return None
level = str(row[0]).upper()
if level not in self._VALID_LOG_LEVELS:
logger.warning(f"Ignoring invalid stored log level: {row[0]}")
return None
return level
except Exception as e:
logger.warning(f"Could not load stored log level from database: {e}")
return None
finally:
if conn:
conn.close()
def _load_env_log_level(self) -> Optional[str]:
"""Load the log level override from the environment, if one exists."""
raw_level = os.environ.get("SOULSYNC_LOG_LEVEL")
if not raw_level:
return None
level = raw_level.upper()
if level not in self._VALID_LOG_LEVELS:
logger.warning(f"Ignoring invalid SOULSYNC_LOG_LEVEL value: {raw_level}")
return None
return level
def _apply_log_level_overrides(self, config_data: Dict[str, Any]) -> Dict[str, Any]:
"""Overlay env and persisted log level preferences onto the loaded config."""
env_level = self._load_env_log_level()
if env_level:
config_data.setdefault("logging", {})["level"] = env_level
logger.info(f"Using log level from SOULSYNC_LOG_LEVEL: {env_level}")
return config_data
stored_level = self._load_stored_log_level()
if stored_level:
config_data.setdefault("logging", {})["level"] = stored_level
logger.info(f"Using stored logging level from database: {stored_level}")
return config_data
def _save_to_database(self, config_data: Dict[str, Any]) -> bool:
"""Save configuration to database, encrypting sensitive values.
Returns ``True`` on success. Transient ``database is locked``
failures are logged at DEBUG so the caller's retry loop owns the
user-visible error message — otherwise every retry would spam
ERROR-level logs even when the next attempt succeeds.
"""
"""Save configuration to database, encrypting sensitive values."""
conn = None
try:
self._ensure_database_exists()
@ -397,7 +310,9 @@ class ConfigManager:
# Encrypt sensitive values before writing (original dict is untouched)
encrypted_data = self._encrypt_sensitive(config_data)
conn = self._connect_db()
# Use longer timeout (30s) to handle contention from enrichment workers
conn = sqlite3.connect(str(self.database_path), timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL")
cursor = conn.cursor()
config_json = json.dumps(encrypted_data, indent=2)
@ -409,18 +324,8 @@ class ConfigManager:
conn.commit()
return True
except sqlite3.OperationalError as e:
# SQLite raises OperationalError("database is locked") when the
# busy_timeout expires while another writer holds the lock.
# Log at DEBUG so the caller can decide whether the final
# outcome warrants an ERROR-level message.
if "locked" in str(e).lower():
logger.debug(f"Config DB locked, will retry: {e}")
else:
logger.error(f"Could not save config to database: {e}")
return False
except Exception as e:
logger.error(f"Could not save config to database: {e}")
print(f"Error: Could not save config to database: {e}")
return False
finally:
if conn:
@ -432,12 +337,12 @@ class ConfigManager:
if self.config_path.exists():
with open(self.config_path, 'r') as f:
config_data = json.load(f)
logger.info(f"Configuration loaded from {self.config_path}")
print(f"[OK] Configuration loaded from {self.config_path}")
return config_data
else:
return None
except Exception as e:
logger.warning(f"Could not load config from file: {e}")
print(f"Warning: Could not load config from file: {e}")
return None
def _get_default_config(self) -> Dict[str, Any]:
@ -476,71 +381,14 @@ class ConfigManager:
"download_path": "./downloads",
"transfer_path": "./Transfer",
"max_peer_queue": 0,
"download_timeout": 600,
# Reddit report (YeloMelo95, Bell Canada): the existing
# 35-per-220s sliding-window cap allows all 35 searches in
# rapid succession before throttling — that burst trips ISP
# anti-abuse. This knob forces a min gap between consecutive
# searches even when the window cap isn't hit. 0 = disabled
# (preserves prior behavior).
"search_min_delay_seconds": 0,
"download_timeout": 600
},
"download_source": {
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid", "torrent", "usenet"
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid"
"hybrid_primary": "soulseek", # Legacy: primary source for hybrid mode
"hybrid_secondary": "youtube", # Legacy: fallback source for hybrid mode
"hybrid_order": [], # Ordered list of sources for hybrid mode (overrides primary/secondary)
"stream_source": "youtube", # Options: "youtube" (instant, default), "active" (use download source; falls back to youtube if soulseek)
# Album-bundle (torrent / usenet single-source) poll tuning.
# Downloader is polled every N seconds until the release
# lands; whole job aborts at the timeout. Defaults match
# the previous hard-coded constants. Users on slow private
# trackers / large box sets can extend the timeout without
# editing source.
"album_bundle_poll_interval_seconds": 2.0,
"album_bundle_timeout_seconds": 6 * 60 * 60, # 6 hours
# Stalled-torrent handling (noldevin): abandon a torrent that
# makes zero download progress for this long (dead magnet
# stuck on "downloading metadata", no seeders) instead of
# holding the worker for the full album timeout. 0 disables.
"torrent_stall_timeout_seconds": 10 * 60, # 10 minutes
# What to do when a torrent stalls: "abandon" (remove it +
# its partial data, fail the download so the next source can
# try) or "pause" (pause in the client, leave for the user).
"torrent_stall_action": "abandon",
# Where THIS container can read completed torrent/usenet
# downloads (#857). The downloader (qBit/SAB) reports a save
# path from inside ITS OWN container — often a category folder
# like /data/downloads/music — which may be mounted at a
# different point here. Set these to the in-container path(s)
# where SoulSync sees those finished downloads; the resolver
# then finds the release by name under them. Empty = fall back
# to the soulseek download/transfer dirs (the shared-volume
# default). See core.download_plugins.album_bundle.resolve_reported_save_path.
"torrent_download_path": "",
"usenet_download_path": "",
# Explicit remote→local prefix mappings for non-shared / oddly
# mounted layouts (Sonarr/Radarr "Remote Path Mapping" style):
# a list of {"from": "<client path>", "to": "<soulsync path>"}.
# Tried before the basename fallback above.
"usenet_path_mappings": [],
},
"post_processing": {
# When a download is quarantined (AcoustID mismatch, integrity /
# duration failure), retry the next-best candidate instead of
# failing outright. Default ON (PR #801's documented default —
# the monitor reads this with inline default True; this template
# said False, so fresh installs silently shipped with the retry
# engine off while existing configs got it on. CI caught the
# split: its fresh default config failed all 7 requeue tests).
"retry_next_candidate_on_mismatch": True,
# Opt-in exhaustive retry: budget retries PER SOURCE so every
# source (Soulseek, then HiFi/Tidal/…) gets its own attempts
# before the track gives up. Default off (single global cap).
"retry_exhaustive": False,
# Retries per search query per source in exhaustive mode. The
# per-source budget is query_count × this value.
"retries_per_query": 5,
},
"tidal_download": {
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
@ -562,16 +410,6 @@ class ConfigManager:
"hifi_download": {
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
},
"hifi": {
"embed_tags": True,
"tags": {
"track_id": True,
"artist_id": True,
"isrc": True,
"bpm": True,
"copyright": True,
}
},
"lidarr_download": {
"url": "",
"api_key": "",
@ -579,44 +417,6 @@ class ConfigManager:
"quality_profile": "Any",
"cleanup_after_import": True,
},
# Prowlarr — indexer aggregator. Feeds the torrent / usenet
# download plugins. Not a standalone source.
"prowlarr": {
"url": "",
"api_key": "",
# Comma-separated list of indexer IDs to limit searches to.
# Empty = search all enabled indexers.
"indexer_ids": "",
},
# Torrent client — receives .torrent / magnet URIs from the
# torrent download plugin. ``type`` picks which adapter to
# instantiate (qbittorrent | transmission | deluge).
"torrent_client": {
"type": "qbittorrent",
"url": "",
"username": "",
"password": "",
"category": "soulsync",
"save_path": "",
},
# Usenet client — receives .nzb URLs / payloads. ``type``
# picks the adapter (sabnzbd | nzbget). SABnzbd uses an
# API key; NZBGet uses username + password.
"usenet_client": {
"type": "sabnzbd",
"url": "",
"api_key": "",
"username": "",
"password": "",
"category": "soulsync",
},
"soundcloud_download": {
# Anonymous-only for now — SoundCloud Go+ OAuth tier could be
# added later, with credentials living under a "session" subkey
# alongside Tidal/Qobuz. No quality knob: anonymous SoundCloud
# caps at the upload's transcoding (typically 128 kbps MP3 or
# AAC). yt-dlp resolves bestaudio at download time.
},
"listenbrainz": {
"base_url": "",
"token": "",
@ -643,43 +443,16 @@ class ConfigManager:
"path": os.environ.get('DATABASE_PATH', 'database/music_library.db'),
"max_workers": 5
},
"image_cache": {
"enabled": True,
"path": "storage/image_cache",
"ttl_seconds": 2592000,
"failed_ttl_seconds": 21600,
"max_download_mb": 15
},
"metadata_enhancement": {
"enabled": True,
"embed_album_art": True,
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"],
# Ordered preferred cover-art sources (empty = use the
# download's own art, i.e. today's behavior). Resolved + walked
# with fallback by core/metadata/art_sources.py.
"album_art_order": [],
# Minimum cover-art resolution (shortest side, px). A preferred
# source whose art is smaller is skipped so the next source is
# tried — stops a low-res Cover Art Archive upload from winning.
# 0 disables the size gate.
"min_art_size": 1000,
# When a track matches a SINGLE release, look up the parent ALBUM
# that contains it and tag it as that album, so it groups with its
# album-mates and gets the album cover (not the single's). Off by
# default — it's an extra per-import metadata lookup.
"single_to_album": False
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"]
},
"musicbrainz": {
"embed_tags": True
},
"playlist_sync": {
"create_backup": True,
# How a re-sync writes to the server playlist:
# replace — delete + recreate (default; today's behavior)
# reconcile — edit in place (add/remove delta), preserving the
# playlist's custom image, description, and identity (#792)
# append — only add new tracks, never remove
"mode": "replace"
"create_backup": True
},
"settings": {
"audio_quality": "flac"
@ -705,34 +478,10 @@ class ConfigManager:
},
"import": {
"staging_path": "./Staging",
# Master toggle for quality-filtering on import. On by default:
# downloaded files that don't meet the quality profile are
# quarantined instead of imported (same gate the download
# pipeline uses). Off → import everything regardless of quality;
# the library Quality Upgrade Scanner still flags them.
"quality_filter_enabled": True,
"replace_lower_quality": False,
# Use the top Staging folder as the artist (Artist/Album layouts,
# mixtapes). On by default to preserve the long-standing import
# behaviour for existing users. Turn OFF if you stage a mixed pile
# of songs under one container folder, otherwise that folder's name
# overrides every metadata-identified artist (the "soulsync" case).
"folder_artist_override": True
"replace_lower_quality": False
},
"m3u_export": {
"enabled": False,
"entry_base_path": ""
},
"playlists": {
# Where "Organize by playlist" materializes playlist folders.
# MUST be a separate root from the music library so the media
# server (and the maintenance jobs) never scan it — otherwise the
# same track would show up twice. Mapped separately for Docker.
"materialize_path": "./Playlists",
# "symlink" (relative links, ~zero disk) or "copy" (real
# duplicates for FAT/USB/DAPs that can't follow links). Symlink
# auto-falls back to copy when the filesystem can't link.
"materialize_mode": "symlink"
"enabled": False
},
"youtube": {
"cookies_browser": "", # "", "chrome", "firefox", "edge", "brave", "opera", "safari"
@ -756,79 +505,66 @@ class ConfigManager:
2. config.json (migration from file-based config)
3. Defaults (fresh install)
"""
logger.info("Loading configuration...")
print(f"Loading configuration...")
# Try loading from database first
config_data = self._load_from_database()
if config_data:
# Configuration exists in database
self.config_data = self._apply_log_level_overrides(config_data)
self.config_data = config_data
# Ensure sensitive values are encrypted at rest (one-time migration)
self._migrate_encrypt_if_needed()
return
# Database is empty - try migration from config.json
logger.info(f"Configuration not found in database. Attempting migration from: {self.config_path}")
print(f"Configuration not found in database. Attempting migration from: {self.config_path}")
config_data = self._load_from_config_file()
if config_data:
# Migrate from config.json to database
logger.info("Migrating configuration from config.json to database...")
print("[MIGRATE] Migrating configuration from config.json to database...")
if self._save_to_database(config_data):
logger.info("Configuration migrated successfully to database.")
self.config_data = self._apply_log_level_overrides(config_data)
print("[OK] Configuration migrated successfully to database.")
self.config_data = config_data
return
else:
logger.warning("Migration failed - using file-based config temporarily.")
self.config_data = self._apply_log_level_overrides(config_data)
print("[WARN] Migration failed - using file-based config temporarily.")
self.config_data = config_data
return
# No config.json either - use defaults
logger.info("No existing configuration found (DB or File) - using defaults")
print("[INFO] ℹ️ No existing configuration found (DB or File) - using defaults")
config_data = self._get_default_config()
# Try to save defaults to database
if self._save_to_database(config_data):
logger.info("Default configuration saved to database")
print("[OK] Default configuration saved to database")
else:
logger.warning("Could not save defaults to database - using in-memory config")
print("[WARN] Could not save defaults to database - using in-memory config")
self.config_data = self._apply_log_level_overrides(config_data)
self.config_data = config_data
def _save_config(self):
"""Save configuration to database with exponential-backoff retry on lock.
"""Save configuration to database with retry on lock."""
success = self._save_to_database(self.config_data)
Spread retries over ~7 seconds so a long-held writer (enrichment
worker batch insert, library scan commit, etc.) on a slow disk
has time to release the lock before we fall back to the JSON
file. The single 1s retry that used to live here gave up too
early on HDD-backed Docker volumes.
"""
# Cumulative delay across attempts: 0.2 + 0.5 + 1.0 + 2.0 + 4.0 = 7.7s
# plus the 30s busy_timeout that already runs inside each attempt.
retry_delays = [0.2, 0.5, 1.0, 2.0, 4.0]
if self._save_to_database(self.config_data):
return
if not success:
# Retry once after a brief wait (handles transient lock contention)
import time
time.sleep(1)
success = self._save_to_database(self.config_data)
for delay in retry_delays:
time.sleep(delay)
if self._save_to_database(self.config_data):
return
# All retries exhausted — fall back to config.json so the user
# doesn't lose their settings, then log a single error.
logger.error(
f"Config DB save failed after {len(retry_delays) + 1} attempts (database is locked) — "
"falling back to config.json"
)
try:
self.config_path.parent.mkdir(parents=True, exist_ok=True)
with open(self.config_path, 'w') as f:
json.dump(self.config_data, f, indent=2)
logger.warning("Configuration saved to config.json as fallback")
except Exception as e:
logger.error(f"Failed to save configuration: {e}")
if not success:
# Fallback: Try to save to config.json if database fails
print("[WARN] Database save failed - attempting file fallback")
try:
self.config_path.parent.mkdir(parents=True, exist_ok=True)
with open(self.config_path, 'w') as f:
json.dump(self.config_data, f, indent=2)
print("[OK] Configuration saved to config.json as fallback")
except Exception as e:
print(f"[ERROR] Failed to save configuration: {e}")
def get(self, key: str, default: Any = None) -> Any:
keys = key.split('.')
@ -842,40 +578,7 @@ class ConfigManager:
return value
def redacted_config(self) -> Dict[str, Any]:
"""Deep copy of the live config with every sensitive value masked.
Used for ``GET /api/settings`` so decrypted secrets never reach the
browser (#832 follow-up). A *set* secret becomes ``REDACTED_SENTINEL``
(the UI renders it as masked dots); an unset one stays empty so the UI
can show "not configured". Dict-valued secrets (OAuth sessions) collapse
to the sentinel too — the UI has no field for them anyway. The matching
guard in ``set()`` turns a round-tripped sentinel back into a no-op.
"""
import copy
data = copy.deepcopy(self.config_data)
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
parent = data
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf in parent and parent[leaf] not in (None, '', {}, [], 0, False):
parent[leaf] = self.REDACTED_SENTINEL
return data
def set(self, key: str, value: Any):
# The UI round-trips REDACTED_SENTINEL for any secret the user didn't
# touch — never let the mask overwrite the real value (#832 follow-up).
if value == self.REDACTED_SENTINEL and key in self._SENSITIVE_PATHS:
return
keys = key.split('.')
config = self.config_data
@ -887,20 +590,6 @@ class ConfigManager:
config[keys[-1]] = value
self._save_config()
def resolve_secret(self, key: str, posted: Any) -> str:
"""Resolve a secret value coming back from the settings UI.
The UI renders a saved-but-untouched secret as the REDACTED_SENTINEL (shown
masked); empty or that sentinel means "use the stored value", while a real
string is a genuine new secret. A connection-test endpoint should test the
EFFECTIVE secret, not the mask — otherwise testing a saved-but-untouched
token sends the sentinel and the source rejects it (#870)."""
if isinstance(posted, str):
posted = posted.strip()
if not posted or posted == self.REDACTED_SENTINEL:
return self.get(key, '') or ''
return posted
def get_spotify_config(self) -> Dict[str, str]:
return self.get('spotify', {})
@ -932,8 +621,8 @@ class ConfigManager:
return self.get('active_media_server', 'plex')
def set_active_media_server(self, server: str):
"""Set the active media server (plex, jellyfin, navidrome, or soulsync)"""
if server not in ['plex', 'jellyfin', 'navidrome', 'soulsync']:
"""Set the active media server (plex, jellyfin, or navidrome)"""
if server not in ['plex', 'jellyfin', 'navidrome']:
raise ValueError(f"Invalid media server: {server}")
self.set('active_media_server', server)
@ -946,8 +635,6 @@ class ConfigManager:
return self.get_jellyfin_config()
elif active_server == 'navidrome':
return self.get_navidrome_config()
elif active_server == 'soulsync':
return {'transfer_path': self.get('soulseek.transfer_path', './Transfer')}
else:
return {}
@ -967,8 +654,6 @@ class ConfigManager:
elif active_server == 'navidrome':
navidrome = self.get_navidrome_config()
media_server_configured = bool(navidrome.get('base_url')) and bool(navidrome.get('username')) and bool(navidrome.get('password'))
elif active_server == 'soulsync':
media_server_configured = True # SoulSync standalone is always configured
return (
bool(spotify.get('client_id')) and
@ -989,7 +674,6 @@ class ConfigManager:
validation['plex'] = bool(self.get('plex.base_url')) and bool(self.get('plex.token'))
validation['jellyfin'] = bool(self.get('jellyfin.base_url')) and bool(self.get('jellyfin.api_key'))
validation['navidrome'] = bool(self.get('navidrome.base_url')) and bool(self.get('navidrome.username')) and bool(self.get('navidrome.password'))
validation['soulsync'] = True # Standalone mode is always valid
validation['active_media_server'] = active_server
return validation

174
core/acoustid_client.py
View file

@ -20,7 +20,6 @@ from typing import Dict, List, Optional, Any, Tuple
from pathlib import Path
import os
import shutil
import logging
import logging.handlers
from utils.logging_config import get_logger
@ -30,23 +29,22 @@ from config.settings import config_manager
FPCALC_BIN_DIR = Path(__file__).parent.parent / "bin"
CHROMAPRINT_VERSION = "1.5.1"
_acoustid_logger = logging.getLogger("soulsync.acoustid")
_acoustid_logger.setLevel(logging.DEBUG)
_acoustid_log_path = Path(config_manager.get('logging.path', 'logs/app.log')).parent / "acoustid.log"
_acoustid_log_path.parent.mkdir(parents=True, exist_ok=True)
if not _acoustid_logger.handlers:
_acoustid_file_handler = logging.handlers.RotatingFileHandler(
_acoustid_log_path, encoding='utf-8', maxBytes=5*1024*1024, backupCount=2
)
_acoustid_file_handler.setLevel(logging.DEBUG)
_acoustid_file_handler.setFormatter(logging.Formatter(
fmt='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
))
_acoustid_logger.addHandler(_acoustid_file_handler)
_acoustid_logger.propagate = False
# Set up dedicated AcoustID logger with its own file
logger = get_logger("acoustid_client")
logger = get_logger("acoustid.client")
# Add dedicated file handler for AcoustID logs
_acoustid_log_path = Path(__file__).parent.parent / "logs" / "acoustid.log"
_acoustid_log_path.parent.mkdir(parents=True, exist_ok=True)
_acoustid_file_handler = logging.handlers.RotatingFileHandler(
_acoustid_log_path, encoding='utf-8', maxBytes=5*1024*1024, backupCount=2
)
_acoustid_file_handler.setLevel(logging.DEBUG)
_acoustid_file_handler.setFormatter(logging.Formatter(
fmt='%(asctime)s - %(name)s - %(levelname)s - %(funcName)s:%(lineno)d - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
))
logger.addHandler(_acoustid_file_handler)
logging.getLogger("newmusic.acoustid_verification").addHandler(_acoustid_file_handler)
# Check if pyacoustid is available
try:
@ -196,7 +194,7 @@ class AcoustIDClient:
result = client.fingerprint_and_lookup("/path/to/audio.mp3")
if result:
for mbid in result['recording_mbids']:
logger.info(f"Match: {mbid}")
print(f"Match: {mbid}")
"""
def __init__(self):
@ -274,7 +272,7 @@ class AcoustIDClient:
if not search_dir.exists():
continue
# Walk up to 2 levels deep to find an audio file quickly
for _depth, pattern in enumerate(['*', '*/*']):
for depth, pattern in enumerate(['*', '*/*']):
for f in search_dir.glob(pattern):
if f.is_file() and f.suffix.lower() in audio_extensions:
return str(f)
@ -282,9 +280,8 @@ class AcoustIDClient:
def test_api_key(self) -> Tuple[bool, str]:
"""
Validate the API key with a direct AcoustID lookup call. An invalid key
is reported as invalid (error code 4); any other error means the key was
accepted.
Validate the API key by fingerprinting a real audio file and looking it up.
Falls back to a direct API call if no audio files are available.
Returns:
Tuple of (success, message)
@ -295,12 +292,24 @@ class AcoustIDClient:
import requests
try:
# Authoritative key check: a direct API lookup with a dummy
# fingerprint. AcoustID validates the client key first, so an
# invalid key returns error code 4 regardless of the fingerprint.
# (The previous real-file path trusted "no exception = valid", but
# fingerprint_and_lookup swallows the invalid-key error and returns
# None — so it reported broken keys as valid. #756-adjacent.)
# Try to find a real audio file to fingerprint for an end-to-end test
test_file = self._find_test_audio_file()
if test_file and CHROMAPRINT_AVAILABLE:
logger.info(f"Testing API key with real audio file: {test_file}")
try:
result = self.fingerprint_and_lookup(test_file)
# If we get here without exception, the API key is valid
# (invalid keys raise or return error before results)
return True, "AcoustID API key is valid"
except Exception as e:
error_str = str(e).lower()
if 'invalid' in error_str and 'api' in error_str:
return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
# Fingerprint/lookup failed for non-key reasons, fall through to direct test
logger.warning(f"Real file test failed ({e}), trying direct API call")
# Fallback: direct API call with minimal fingerprint
url = 'https://api.acoustid.org/v2/lookup'
params = {
'client': self.api_key,
@ -315,13 +324,12 @@ class AcoustIDClient:
if data.get('status') == 'error':
error = data.get('error', {})
error_code = error.get('code', 0)
error_msg = error.get('message', 'Unknown error')
# Error code 4 is specifically "invalid API key"
if error_code == 4:
return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
# Any other error (e.g. "invalid fingerprint") means the API key
# was accepted — the dummy test fingerprint is just rejected as expected
return True, "AcoustID API key is valid"
return False, f"AcoustID API error: {error_msg}"
# Status is 'ok' - key is valid
return True, "AcoustID API key is valid"
@ -334,33 +342,33 @@ class AcoustIDClient:
logger.error(f"Error testing AcoustID API key: {e}")
return False, f"Error: {str(e)}"
def lookup_with_status(self, audio_file: str) -> Dict[str, Any]:
"""Fingerprint + AcoustID lookup returning a STRUCTURED result.
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
"""
Generate fingerprint and look up recording in AcoustID.
Unlike fingerprint_and_lookup() (which collapses every outcome into
dict-or-None), this distinguishes a genuine no-match from an actual
error — an invalid API key, rate limit, missing chromaprint, or a
fingerprint failure. That distinction is what lets the UI show "AcoustID
Error" (something is broken — fix it) instead of a benign-looking
"Skipped" that silently hides a dead key.
This is the main method - combines fingerprinting and lookup in one call.
Returns dict with:
'status': 'ok' | 'no_match' | 'error' | 'no_backend'
| 'fingerprint_error' | 'unsupported' | 'unavailable'
| 'not_found'
'recordings': list (meaningful only for 'ok')
'best_score': float
'recording_mbids': list
'error': human-readable detail for any non-'ok' status
'invalid_key': bool (True when the API specifically rejected the key)
Args:
audio_file: Path to the audio file
Returns:
Dict with:
'recordings': list of dicts with 'mbid', 'title', 'artist', 'score'
'best_score': float (highest score across all results)
'recording_mbids': list of unique MBIDs (for backward compat)
Or None on error.
"""
if not ACOUSTID_AVAILABLE:
return {'status': 'unavailable', 'recordings': [], 'error': 'pyacoustid library not installed'}
logger.debug("Cannot lookup: pyacoustid not available")
return None
if not self.api_key:
return {'status': 'unavailable', 'recordings': [], 'error': 'No AcoustID API key configured'}
logger.debug("Cannot lookup: no API key")
return None
if not os.path.isfile(audio_file):
logger.warning(f"Cannot lookup: file not found: {audio_file}")
return {'status': 'not_found', 'recordings': [], 'error': f'File not found: {audio_file}'}
return None
# Check channel count — chromaprint crashes (SIGABRT) on >2 channel files (e.g. 5.1 surround)
try:
@ -370,8 +378,7 @@ class AcoustIDClient:
channels = getattr(mf.info, 'channels', 2)
if channels and channels > 2:
logger.warning(f"Skipping AcoustID: file has {channels} channels (surround audio): {audio_file}")
return {'status': 'unsupported', 'recordings': [],
'error': f'{channels}-channel (surround) audio not supported by chromaprint'}
return None
except Exception as e:
logger.debug(f"Could not check channel count, proceeding anyway: {e}")
@ -381,12 +388,17 @@ class AcoustIDClient:
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "NOT SET"
logger.info(f"Fingerprinting and looking up: {audio_file} (API key: {api_key_preview})")
# Use match() which handles fingerprinting + lookup + parsing
logger.debug("Running acoustid.match()...")
recordings = []
seen_mbids = set()
best_score = 0.0
for result in acoustid.match(self.api_key, audio_file, parse=True):
for result in acoustid.match(
self.api_key,
audio_file,
parse=True
):
# match() with parse=True returns (score, recording_id, title, artist)
if not isinstance(result, tuple) or len(result) < 2:
logger.warning(f"Unexpected result format: {result}")
@ -404,57 +416,45 @@ class AcoustIDClient:
if recording_id and recording_id not in seen_mbids:
seen_mbids.add(recording_id)
recordings.append({'mbid': recording_id, 'title': title, 'artist': artist, 'score': score})
recordings.append({
'mbid': recording_id,
'title': title,
'artist': artist,
'score': score,
})
logger.debug(f"Found match: {title} by {artist} (MBID: {recording_id}, score: {score})")
if not recordings:
logger.info(f"No AcoustID matches found for: {audio_file}")
return {'status': 'no_match', 'recordings': [], 'best_score': best_score,
'recording_mbids': [], 'error': 'Track not found in AcoustID database'}
return None
logger.info(f"AcoustID found {len(recordings)} recording(s) (best score: {best_score:.2f})")
return {'status': 'ok', 'recordings': recordings, 'best_score': best_score,
'recording_mbids': list(seen_mbids)}
return {
'recordings': recordings,
'best_score': best_score,
'recording_mbids': list(seen_mbids),
}
except acoustid.NoBackendError:
logger.error("Chromaprint library not found and fpcalc not available")
return {'status': 'no_backend', 'recordings': [],
'error': 'Chromaprint/fpcalc not installed (install libchromaprint1)'}
return None
except acoustid.FingerprintGenerationError as e:
logger.warning(f"Failed to fingerprint {audio_file}: {e}")
return {'status': 'fingerprint_error', 'recordings': [], 'error': f'Could not fingerprint file: {e}'}
return None
except acoustid.WebServiceError as e:
# Log more details about the API error
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "???"
logger.warning(f"AcoustID API error (key: {api_key_preview}): {e}")
# Check for common errors
error_str = str(e).lower()
# Old pyacoustid reports an invalid key as the bare "status: error"
# (it drops the detail), so treat that as an invalid-key signal too.
invalid = ('invalid' in error_str or 'unknown' in error_str or 'status: error' in error_str)
if invalid:
logger.error("AcoustID API key appears to be invalid — check your AcoustID settings")
if 'invalid' in error_str or 'unknown' in error_str:
logger.error("API key appears to be invalid - check your AcoustID settings")
elif 'rate' in error_str or 'limit' in error_str:
logger.warning("Rate limited by AcoustID — will retry later")
return {'status': 'error', 'recordings': [], 'invalid_key': invalid,
'error': f'AcoustID API error: {e}'}
logger.warning("Rate limited by AcoustID - will retry later")
return None
except Exception as e:
logger.error(f"Unexpected error in AcoustID lookup: {e}", exc_info=True)
return {'status': 'error', 'recordings': [], 'error': f'Unexpected error: {e}'}
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
"""Legacy dict-or-None lookup. Returns the recordings dict on a confirmed
match, else None. Kept for callers that only need "did we identify it"
(library scanner, auto-import). Callers that must report WHY a lookup
didn't match (verification badge, key test) should use
``lookup_with_status`` so an error isn't mistaken for a no-match.
"""
res = self.lookup_with_status(audio_file)
if res.get('status') == 'ok':
return {
'recordings': res['recordings'],
'best_score': res.get('best_score', 0.0),
'recording_mbids': res.get('recording_mbids', []),
}
return None
return None
def refresh_config(self):
"""Refresh cached config values (call after settings change)."""

317
core/acoustid_verification.py
View file

@ -15,82 +15,96 @@ from typing import Optional, Dict, Any, Tuple, List
from enum import Enum
from utils.logging_config import get_logger
from core.acoustid_client import AcoustIDClient
from core.matching_engine import MusicMatchingEngine
from core.matching.version_mismatch import is_acceptable_version_mismatch
from core.matching.script_compat import is_cross_script_mismatch
from core.musicbrainz_client import MusicBrainzClient
logger = get_logger("acoustid.verification")
logger = get_logger("acoustid_verification")
# Thresholds — single definition lives in the shared core; re-exported here so
# existing importers keep working and the values can't drift between paths.
from core.matching.audio_verification import ( # noqa: E402
MIN_ACOUSTID_SCORE, TITLE_MATCH_THRESHOLD, ARTIST_MATCH_THRESHOLD,
)
# Single matching-engine instance so version detection reuses the same patterns
# used by the pre-download Soulseek matcher (remix / live / acoustic /
# instrumental / etc). detect_version_type doesn't use self state, so one
# shared instance is fine.
_match_engine_for_version = MusicMatchingEngine()
def _detect_title_version(title: str) -> str:
"""Return version label for a track title.
Returns ``'original'`` when no version marker is detected, otherwise one
of the labels produced by ``MusicMatchingEngine.detect_version_type``
(``'instrumental'``, ``'live'``, ``'acoustic'``, ``'remix'``, etc).
"""
if not title:
return 'original'
version_type, _ = _match_engine_for_version.detect_version_type(title)
return version_type
# Thresholds
MIN_ACOUSTID_SCORE = 0.80 # Minimum AcoustID fingerprint score to trust
TITLE_MATCH_THRESHOLD = 0.70 # Title similarity needed to consider a match
ARTIST_MATCH_THRESHOLD = 0.60 # Artist similarity needed to consider a match
class VerificationResult(Enum):
"""Possible outcomes of audio verification."""
PASS = "pass" # Title/artist match - file is correct
FAIL = "fail" # Title/artist mismatch - wrong file downloaded
SKIP = "skip" # Genuinely couldn't verify (no match in DB) - continue normally
SKIP = "skip" # Could not verify (error or unavailable) - continue normally
DISABLED = "disabled" # Verification not enabled
ERROR = "error" # Lookup errored (invalid key / rate limit / no backend) - continue, but flag it
# normalize() + similarity() + the alias-aware comparison now live in the shared
# decision core (core/matching/audio_verification.py) so import-time verification
# and the library scan share ONE definition — the <>-strip fix, CJK handling and
# thresholds can't drift apart again. Names kept (`_normalize` etc.) for existing
# importers/tests.
from core.matching.audio_verification import ( # noqa: E402
normalize as _normalize,
similarity as _similarity,
_alias_aware_artist_sim,
_find_best_title_artist_match as _core_find_best_title_artist_match,
evaluate as _core_evaluate,
Decision as _CoreDecision,
)
def _normalize(text: str) -> str:
"""Normalize a string for comparison: lowercase, strip parentheticals, punctuation."""
if not text:
return ""
s = text.lower().strip()
# Remove ALL parenthetical suffixes — these are metadata annotations, not core title
# Covers: (Live), (Remastered), (Parody of ...), (from "..." Soundtrack), (feat. ...), etc.
s = re.sub(r'\s*\([^)]*\)', '', s)
# Remove ALL square bracket suffixes: [Live], [Remastered], [Deluxe], etc.
s = re.sub(r'\s*\[[^\]]*\]', '', s)
# Remove trailing featuring info not in parentheses: "feat. ...", "ft. ...", "featuring ..."
s = re.sub(r'\s+(?:feat\.?|ft\.?|featuring)\s+.*$', '', s, flags=re.IGNORECASE)
# Remove dash-separated version tags: "- Vocal", "- Instrumental", "- Acoustic", etc.
s = re.sub(r'\s*-\s*(?:vocal|instrumental|acoustic|live|remix|cover|clean|explicit|radio\s*edit|original\s*mix|extended\s*mix|club\s*mix)\s*$', '', s, flags=re.IGNORECASE)
# Remove soundtrack/source subtitles: ' - From "..." Soundtrack', ' - from the film ...'
s = re.sub(r'\s*-\s*from\s+.+$', '', s, flags=re.IGNORECASE)
# Remove non-alphanumeric except spaces
s = re.sub(r'[^\w\s]', '', s)
# Collapse whitespace
s = re.sub(r'\s+', ' ', s).strip()
return s
def _find_best_title_artist_match(recordings, expected_title, expected_artist,
expected_artist_aliases=None):
"""Back-compat wrapper around the shared core matcher (keeps the
``expected_artist_aliases`` kwarg name for existing callers/tests)."""
return _core_find_best_title_artist_match(
recordings, expected_title, expected_artist, expected_artist_aliases,
)
def _similarity(a: str, b: str) -> float:
"""Calculate similarity between two strings (0.0-1.0) after normalization."""
na = _normalize(a)
nb = _normalize(b)
if not na or not nb:
return 0.0
if na == nb:
return 1.0
return SequenceMatcher(None, na, nb).ratio()
def _find_best_title_artist_match(
recordings: List[Dict[str, Any]],
expected_title: str,
expected_artist: str,
) -> Tuple[Optional[Dict], float, float]:
"""
Find the AcoustID recording that best matches expected title/artist.
Returns:
(best_recording, title_similarity, artist_similarity)
"""
best_rec = None
best_title_sim = 0.0
best_artist_sim = 0.0
best_combined = 0.0
for rec in recordings:
title = rec.get('title') or ''
artist = rec.get('artist') or ''
title_sim = _similarity(expected_title, title)
artist_sim = _similarity(expected_artist, artist)
# Weight title higher since that's the primary identifier
combined = (title_sim * 0.6) + (artist_sim * 0.4)
if combined > best_combined:
best_combined = combined
best_rec = rec
best_title_sim = title_sim
best_artist_sim = artist_sim
return best_rec, best_title_sim, best_artist_sim
# Shared MusicBrainz client for enrichment lookups
_mb_client = None
_mb_client_lock = threading.Lock()
# Shared MusicBrainzService for alias lookups (issue #442). Service
# layer wraps the raw client + adds caching + DB access — all of which
# the alias resolution chain (library DB → cache → live MB) needs.
_mb_service = None
_mb_service_lock = threading.Lock()
MAX_MB_ENRICHMENT_LOOKUPS = 3
@ -104,42 +118,6 @@ def _get_mb_client() -> MusicBrainzClient:
return _mb_client
def _get_mb_service():
"""Get or create a shared MusicBrainzService instance.
Used by the alias-resolution chain in `verify_audio_file`. Lazy
init so importing this module doesn't trigger a DB connection on
paths that never run AcoustID verification (test runs, dry runs).
"""
global _mb_service
if _mb_service is None:
with _mb_service_lock:
if _mb_service is None:
from core.musicbrainz_service import MusicBrainzService
from database.music_database import get_database
_mb_service = MusicBrainzService(get_database())
return _mb_service
def _resolve_expected_artist_aliases(expected_artist_name: str) -> List[str]:
"""Look up alternate-spelling aliases for the expected artist.
Issue #442 — bridges cross-script artist comparisons (Japanese
kanji ↔ romanized, Cyrillic ↔ Latin, etc.) without forcing the
verifier to know about the resolution chain. Best-effort: any
failure (no MB service, network down, no library DB) returns
empty list so verification falls back to the prior direct
similarity check.
"""
if not expected_artist_name:
return []
try:
return _get_mb_service().lookup_artist_aliases(expected_artist_name)
except Exception as e:
logger.debug("alias lookup failed for %r: %s", expected_artist_name, e)
return []
def _enrich_recordings_from_musicbrainz(
recordings: List[Dict[str, Any]],
) -> List[Dict[str, Any]]:
@ -265,33 +243,18 @@ class AcoustIDVerification:
logger.debug(f"AcoustID verification skipped: {reason}")
return VerificationResult.SKIP, reason
# Step 2: Fingerprint and lookup in AcoustID (structured so an
# actual error — invalid key / rate limit / no chromaprint — is
# reported distinctly from a genuine no-match, instead of both
# silently surfacing as "Skipped").
# Step 2: Fingerprint and lookup in AcoustID
logger.info(f"Fingerprinting and looking up: {audio_file_path}")
lookup = self.acoustid_client.lookup_with_status(audio_file_path) or {}
status = lookup.get('status')
# Infer status by content when absent (a caller/stub that returned
# just recordings): recordings => matched, none => no match.
if status is None:
status = 'ok' if lookup.get('recordings') else 'no_match'
acoustid_result = self.acoustid_client.fingerprint_and_lookup(audio_file_path)
if status in ('error', 'no_backend', 'fingerprint_error', 'unavailable'):
# Something is broken (not the track's fault) — never quarantine
# on this; surface it so the user can fix it.
return VerificationResult.ERROR, lookup.get('error', 'AcoustID lookup failed')
if not acoustid_result:
return VerificationResult.SKIP, "Track not found in AcoustID database"
if status != 'ok':
# no_match / unsupported / not_found — genuinely could not verify.
return VerificationResult.SKIP, lookup.get('error', 'No match in AcoustID database')
acoustid_result = lookup
recordings = acoustid_result.get('recordings', [])
best_score = acoustid_result.get('best_score', 0)
if not recordings:
return VerificationResult.SKIP, "No match in AcoustID database"
return VerificationResult.SKIP, "AcoustID returned no recordings"
logger.debug(
f"AcoustID returned {len(recordings)} recording(s) "
@ -307,55 +270,93 @@ class AcoustIDVerification:
# Enrich recordings that are missing title/artist via MusicBrainz lookup
recordings = _enrich_recordings_from_musicbrainz(recordings)
# Issue #442 — alias resolution is LAZY. We pass a memoising
# thunk to the artist-comparison sites; it only fires the
# multi-tier lookup (library DB → cache → live MB) when
# direct artist similarity falls below threshold. Verifications
# where the direct match already passes (the common case for
# same-script artist names) never trigger any lookup work,
# so the fix doesn't add a per-verification DB query for the
# happy path. When the thunk DOES fire, the result is cached
# in the closure so the 3 comparison sites within one
# verification share a single resolution pass.
_alias_cache: Dict[str, Any] = {}
def _aliases_provider() -> List[str]:
if 'value' not in _alias_cache:
resolved = _resolve_expected_artist_aliases(expected_artist_name)
_alias_cache['value'] = resolved
if resolved:
logger.debug(
"Resolved %d aliases for expected artist '%s'",
len(resolved), expected_artist_name,
)
return _alias_cache['value']
# Steps 4-5: delegate the PASS/SKIP/FAIL decision to the shared core
# (core/matching/audio_verification.evaluate) so import verification
# and the library scan apply identical logic.
outcome = _core_evaluate(
expected_track_name, expected_artist_name, recordings,
fingerprint_score=best_score,
aliases_provider=_aliases_provider,
# Step 4: Find best title/artist match among AcoustID results
best_rec, title_sim, artist_sim = _find_best_title_artist_match(
recordings, expected_track_name, expected_artist_name
)
if not best_rec:
return VerificationResult.SKIP, "No recordings with title/artist info"
matched_title = best_rec.get('title', '?')
matched_artist = best_rec.get('artist', '?')
logger.info(
"Best match: '%s' by '%s' (title_sim=%.2f, artist_sim=%.2f) -> %s",
outcome.matched_title, outcome.matched_artist,
outcome.title_sim, outcome.artist_sim, outcome.decision.value,
f"Best match: '{matched_title}' by '{matched_artist}' "
f"(title_sim={title_sim:.2f}, artist_sim={artist_sim:.2f})"
)
_decision_map = {
_CoreDecision.PASS: VerificationResult.PASS,
_CoreDecision.SKIP: VerificationResult.SKIP,
_CoreDecision.FAIL: VerificationResult.FAIL,
}
result = _decision_map[outcome.decision]
if result == VerificationResult.PASS:
logger.info("AcoustID verification PASSED - %s", outcome.reason)
elif result == VerificationResult.FAIL:
logger.warning("AcoustID verification FAILED - %s", outcome.reason)
else:
logger.info("AcoustID verification SKIPPED - %s", outcome.reason)
return result, outcome.reason
# Step 5: Decide pass/fail based on similarity
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim >= ARTIST_MATCH_THRESHOLD:
msg = (
f"Audio verified: '{matched_title}' by '{matched_artist}' "
f"matches expected '{expected_track_name}' by '{expected_artist_name}' "
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
)
logger.info(f"AcoustID verification PASSED - {msg}")
return VerificationResult.PASS, msg
# Title matches but artist doesn't — could be a cover or collab, skip
if title_sim >= TITLE_MATCH_THRESHOLD and artist_sim < ARTIST_MATCH_THRESHOLD:
# Check if the expected artist appears anywhere in the AcoustID results
for rec in recordings:
if _similarity(expected_artist_name, rec.get('artist', '')) >= ARTIST_MATCH_THRESHOLD:
msg = (
f"Audio verified: found '{expected_track_name}' by '{expected_artist_name}' "
f"in AcoustID results"
)
logger.info(f"AcoustID verification PASSED (secondary match) - {msg}")
return VerificationResult.PASS, msg
msg = (
f"Title matches but artist unclear: "
f"AcoustID='{matched_title}' by '{matched_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}'"
)
logger.info(f"AcoustID verification SKIPPED - {msg}")
return VerificationResult.SKIP, msg
# Title doesn't match — check ALL recordings for any title/artist match
# (the best combined match might not be the right one if there are many results)
for rec in recordings:
t = rec.get('title') or ''
a = rec.get('artist') or ''
if (_similarity(expected_track_name, t) >= TITLE_MATCH_THRESHOLD and
_similarity(expected_artist_name, a) >= ARTIST_MATCH_THRESHOLD):
msg = (
f"Audio verified: found '{t}' by '{a}' in AcoustID results "
f"matching expected '{expected_track_name}' by '{expected_artist_name}'"
)
logger.info(f"AcoustID verification PASSED (scan match) - {msg}")
return VerificationResult.PASS, msg
# No match found — but if fingerprint score is very high (≥0.95),
# the audio IS correct and the title mismatch is likely a language
# difference (e.g. Japanese kanji vs English title for the same song).
# Skip rather than quarantine a correct file.
if best_score >= 0.95:
top = recordings[0]
msg = (
f"Title/artist mismatch but fingerprint confidence very high ({best_score:.2f}): "
f"AcoustID='{top.get('title', '?')}' by '{top.get('artist', '?')}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' — "
f"likely same song in different language/script"
)
logger.info(f"AcoustID verification SKIPPED (high confidence) - {msg}")
return VerificationResult.SKIP, msg
# Low fingerprint score + no metadata match — file is likely wrong
top = recordings[0]
top_title = top.get('title', '?')
top_artist = top.get('artist', '?')
msg = (
f"Audio mismatch: file identified as '{top_title}' by '{top_artist}', "
f"expected '{expected_track_name}' by '{expected_artist_name}' "
f"(title={title_sim:.0%}, artist={artist_sim:.0%})"
)
logger.warning(f"AcoustID verification FAILED - {msg}")
return VerificationResult.FAIL, msg
except Exception as e:
# Any unexpected error -> SKIP (fail open)

239
core/album_consistency.py
View file

@ -57,86 +57,6 @@ _ID3_TXXX_MAP = {
# MP4 freeform keys
_MP4_KEY_PREFIX = '----:com.apple.iTunes:'
# ── Picard-style release preference scoring ──
# Preferred countries (higher = better). US/GB/XW(worldwide) are most common
# for English-language music. XE = Europe-wide.
_COUNTRY_SCORES = {
'US': 10, 'XW': 10, 'GB': 8, 'XE': 7, 'CA': 6, 'AU': 5, 'DE': 4,
'FR': 4, 'JP': 3, 'NL': 3, 'SE': 3, 'IT': 2,
}
# Preferred formats (higher = better). Digital/CD are the standard;
# vinyl and cassette are niche reissues that often differ from the
# canonical tracklist.
_FORMAT_SCORES = {
'Digital Media': 10, 'CD': 9, 'Enhanced CD': 8,
'SACD': 7, 'Hybrid SACD': 7, 'Blu-spec CD': 7,
'Vinyl': 3, '12" Vinyl': 3, '7" Vinyl': 2,
'Cassette': 1,
}
# Release status preference
_STATUS_SCORES = {
'Official': 10, 'Promotion': 5, 'Bootleg': 1, 'Pseudo-Release': 1,
}
def _score_release(release: dict, expected_track_count: int) -> float:
"""Score a MusicBrainz release for preference ranking.
Higher score = better candidate. Factors:
- Track count match (most important — wrong count is wrong release)
- Release status (Official > Promo > Bootleg)
- Country preference (US/worldwide > regional)
- Format preference (Digital/CD > Vinyl > Cassette)
- Has barcode (sign of a real commercial release)
- Penalize releases with no media info (incomplete data)
"""
score = 0.0
# Track count match (0-40 points, biggest factor)
media = release.get('media', [])
mb_track_count = sum(len(m.get('tracks') or m.get('track-list', []))
for m in media)
track_diff = abs(mb_track_count - expected_track_count)
if track_diff == 0:
score += 40
elif track_diff <= 1:
score += 30
elif track_diff <= 2:
score += 20
elif track_diff <= 5:
score += 10
# else: 0 points
# Status (0-10 points)
status = release.get('status', '')
score += _STATUS_SCORES.get(status, 2)
# Country (0-10 points)
country = release.get('country', '')
score += _COUNTRY_SCORES.get(country, 1)
# Format from first medium (0-10 points)
if media:
fmt = media[0].get('format', '')
score += _FORMAT_SCORES.get(fmt, 4)
else:
score -= 5 # No media info = suspect
# Barcode (0-3 points) — real commercial releases have barcodes
if release.get('barcode'):
score += 3
# Date completeness (0-2 points) — prefer releases with full dates
date = release.get('date', '')
if len(date) >= 10:
score += 2 # Full YYYY-MM-DD
elif len(date) >= 4:
score += 1 # Year only
return score
def _normalize_title(s):
"""Normalize a title for comparison."""
@ -151,91 +71,106 @@ def _normalize_title(s):
def _find_best_release(album_name, artist_name, track_count, mb_service):
"""Search MusicBrainz for the best release matching this album.
Uses Picard-style preference scoring: track count match, release status,
country (US/worldwide preferred), format (Digital/CD preferred), barcode
presence, and date completeness. Deterministic — same inputs always
produce the same release.
"""
Prefers releases where track count matches the download."""
try:
import re
# First try our existing match_release (uses version qualifier scoring)
match = mb_service.match_release(album_name, artist_name)
if not match or not match.get('mbid'):
# Try stripping edition qualifiers — Spotify uses "Album (Super Deluxe)"
# but MusicBrainz just calls it "Album"
import re
stripped = re.sub(
r'\s*[\(\[]'
r'[^)\]]*'
r'(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard|edition)'
r'[^)\]]*'
r'[\)\]]',
'', album_name, flags=re.IGNORECASE
).strip()
# Also strip trailing bare editions
stripped = re.sub(
r'\s+(?:-\s+)?(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard)'
r'(?:\s+(?:edition|version))?\s*$',
'', stripped, flags=re.IGNORECASE
).strip()
# Build search name variants
search_names = [album_name]
stripped = re.sub(
r'\s*[\(\[]'
r'[^)\]]*'
r'(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard|edition)'
r'[^)\]]*'
r'[\)\]]',
'', album_name, flags=re.IGNORECASE
).strip()
stripped = re.sub(
r'\s+(?:-\s+)?(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard)'
r'(?:\s+(?:edition|version))?\s*$',
'', stripped, flags=re.IGNORECASE
).strip()
if stripped and stripped.lower() != album_name.lower():
search_names.append(stripped)
if stripped and stripped.lower() != album_name.lower():
logger.info(f"Retrying MB search with stripped name: '{stripped}' (was '{album_name}')")
match = mb_service.match_release(stripped, artist_name)
# Collect candidate release MBIDs from all search variants
candidate_mbids = []
for name in search_names:
# Try cached match first
match = mb_service.match_release(name, artist_name)
if match and match.get('mbid'):
candidate_mbids.append(match['mbid'])
if not match or not match.get('mbid'):
# Final fallback: direct API search with stripped name
search_name = stripped or album_name
logger.info(f"No cached MB release — trying direct search for '{search_name}'")
search_results = mb_service.mb_client.search_release(search_name, artist_name, limit=5)
if not search_results:
logger.info(f"No MB release found for '{album_name}' by '{artist_name}'")
return None
mbid = search_results[0].get('id', '')
if not mbid:
return None
logger.info(f"Direct search found: {search_results[0].get('title', '')} ({mbid[:8]}...)")
else:
mbid = match['mbid']
else:
mbid = match['mbid']
# Also try direct search for more candidates
try:
search_results = mb_service.mb_client.search_release(name, artist_name, limit=5)
for sr in (search_results or []):
sr_id = sr.get('id', '')
if sr_id and sr_id not in candidate_mbids:
candidate_mbids.append(sr_id)
except Exception as e:
logger.debug("search_release fallback failed: %s", e)
if not candidate_mbids:
logger.info(f"No MB release found for '{album_name}' by '{artist_name}'")
# Fetch full release with tracklist
release = mb_service.mb_client.get_release(
mbid, includes=['recordings', 'release-groups', 'labels', 'media', 'artist-credits']
)
if not release:
return None
# Fetch full release data for each candidate and score them
best_release = None
best_score = -1
# Check track count match
mb_track_count = sum(len(m.get('tracks') or m.get('track-list', [])) for m in release.get('media', []))
if abs(mb_track_count - track_count) <= 2:
logger.info(f"Accepted release '{release.get('title')}' ({mbid[:8]}...) — "
f"{mb_track_count} tracks (downloaded {track_count})")
return release
for mbid in candidate_mbids[:8]: # Cap at 8 to limit API calls
# Track count mismatch — try searching for a better release
# Use stripped name for search (MB often doesn't include edition suffixes)
import re
_search_name = re.sub(
r'\s*[\(\[][^)\]]*(?:deluxe|expanded|remaster|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard|edition)[^)\]]*[\)\]]',
'', album_name, flags=re.IGNORECASE
).strip() or album_name
logger.info(f"Release '{release.get('title')}' has {mb_track_count} tracks but we have {track_count} — "
f"searching for better match with '{_search_name}'")
search_results = mb_service.mb_client.search_release(_search_name, artist_name, limit=5)
if not search_results:
# Fall back to the first match even if count doesn't match perfectly
return release
best_release = release
best_diff = abs(mb_track_count - track_count)
for sr in search_results:
sr_mbid = sr.get('id', '')
if not sr_mbid or sr_mbid == mbid:
continue
try:
release = mb_service.mb_client.get_release(
mbid, includes=['recordings', 'release-groups', 'labels',
'media', 'artist-credits']
candidate = mb_service.mb_client.get_release(
sr_mbid, includes=['recordings', 'release-groups', 'labels', 'media', 'artist-credits']
)
if not release:
if not candidate:
continue
score = _score_release(release, track_count)
if score > best_score:
best_score = score
best_release = release
cand_count = sum(len(m.get('tracks') or m.get('track-list', [])) for m in candidate.get('media', []))
cand_diff = abs(cand_count - track_count)
if cand_diff < best_diff:
best_diff = cand_diff
best_release = candidate
if cand_diff == 0:
break # Perfect match
except Exception:
continue
if best_release:
mb_count = sum(len(m.get('tracks') or m.get('track-list', []))
for m in best_release.get('media', []))
logger.info(
f"Selected release '{best_release.get('title')}' "
f"({best_release.get('id', '')[:8]}...) — "
f"score={best_score:.0f}, tracks={mb_count}, "
f"country={best_release.get('country', '?')}, "
f"format={best_release.get('media', [{}])[0].get('format', '?')}, "
f"status={best_release.get('status', '?')}"
)
logger.info(f"Best release: '{best_release.get('title')}' ({best_release.get('id', '')[:8]}...) — "
f"track count diff: {best_diff}")
return best_release
except Exception as e:

796
core/amazon_client.py
View file

@ -1,796 +0,0 @@
"""Amazon Music metadata client backed by a T2Tunes proxy instance.
T2Tunes exposes the Amazon Music catalog (search, album metadata, stream info)
through a simple REST API. This module wraps those endpoints and normalises the
responses into the same Track / Artist / Album dataclass shape used by
DeezerClient and iTunesClient.
Endpoints used:
GET /api/status
GET /api/amazon-music/search
GET /api/amazon-music/metadata
GET /api/amazon-music/media-from-asin
Config keys (all optional — fall back to public defaults):
amazon.base_url T2Tunes instance URL (default: https://t2tunes.site)
amazon.country ISO-3166 country code (default: US)
amazon.preferred_codec Preferred audio codec (default: flac)
"""
from __future__ import annotations
import re
import threading
import time
from dataclasses import dataclass
from typing import Any, Dict, Iterator, List, Optional
from urllib.parse import urljoin
import requests
from config.settings import config_manager
from core.api_call_tracker import api_call_tracker
from utils.logging_config import get_logger
logger = get_logger("amazon_client")
# Strips featuring credits like "Artist feat. X", "Artist ft. Y" so artist
# deduplication works on the primary artist name only.
_FEAT_RE = re.compile(r'\s+(?:feat(?:uring)?\.?|ft\.?)\s+.*', re.IGNORECASE)
# Strips the Explicit marker — explicit is treated as the default version.
# Clean/Edited/Censored stay in the name so users can distinguish them.
_EDITION_RE = re.compile(r'\s*[\[\(]explicit[\]\)]', re.IGNORECASE)
def _primary_artist(name: str) -> str:
return _FEAT_RE.sub('', name).strip()
def _strip_edition(name: str) -> str:
return _EDITION_RE.sub('', name).strip()
def _unslugify(name: str) -> str:
"""Convert a slug-form artist ID (e.g. 'kendrick_lamar') to a search name."""
return name.replace('_', ' ')
DEFAULT_BASE_URL = "https://t2tunes.site"
DEFAULT_COUNTRY = "US"
DEFAULT_CODEC = "flac"
MIN_API_INTERVAL = 0.5 # seconds — T2Tunes has no published rate limit
_last_api_call: float = 0.0
_api_call_lock = threading.Lock()
_META_CACHE_TTL = 300 # seconds
_meta_cache: Dict[str, tuple] = {} # asin -> (fetched_at, meta_dict)
_meta_cache_lock = threading.Lock()
class AmazonClientError(RuntimeError):
"""Raised on unrecoverable T2Tunes API errors.
Carries the HTTP ``status_code`` when the failure was an HTTP error, so
callers (the worker's outage detection) can tell a source outage (5xx) from
a per-item miss without parsing the message.
"""
def __init__(self, *args, status_code=None):
super().__init__(*args)
self.status_code = status_code
# ---------------------------------------------------------------------------
# Dataclasses — field layout matches DeezerClient / iTunesClient exactly
# ---------------------------------------------------------------------------
@dataclass
class Track:
id: str # Amazon track ASIN
name: str
artists: List[str]
album: str
duration_ms: int
popularity: int
preview_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
image_url: Optional[str] = None
release_date: Optional[str] = None
track_number: Optional[int] = None
disc_number: Optional[int] = None
album_type: Optional[str] = None
total_tracks: Optional[int] = None
isrc: Optional[str] = None
@classmethod
def from_search_hit(cls, doc: Dict[str, Any]) -> "Track":
return cls(
id=str(doc.get("asin") or ""),
name=str(doc.get("title") or ""),
artists=[str(doc.get("artistName") or "Unknown Artist")],
album=str(doc.get("albumName") or ""),
duration_ms=int(doc.get("duration") or 0) * 1000,
popularity=0,
isrc=str(doc.get("isrc") or "") or None,
)
@classmethod
def from_stream_info(
cls,
stream: "T2TunesStreamInfo",
album_meta: Optional[Dict[str, Any]] = None,
) -> "Track":
album = album_meta or {}
return cls(
id=stream.asin,
name=stream.title,
artists=[stream.artist] if stream.artist else ["Unknown Artist"],
album=stream.album,
duration_ms=0,
popularity=0,
image_url=album.get("image"),
release_date=album.get("release_date"),
total_tracks=album.get("trackCount"),
isrc=stream.isrc or None,
)
@dataclass
class Artist:
id: str # Slugified artist name — T2Tunes exposes no artist IDs
name: str
popularity: int
genres: List[str]
followers: int
image_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
@classmethod
def from_name(cls, name: str) -> "Artist":
slug = name.lower().replace(" ", "_")
return cls(id=slug, name=name, popularity=0, genres=[], followers=0)
@dataclass
class Album:
id: str # Amazon album ASIN
name: str
artists: List[str]
release_date: str
total_tracks: int
album_type: str
image_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
explicit: Optional[bool] = None
@classmethod
def from_search_hit(cls, doc: Dict[str, Any]) -> "Album":
return cls(
id=str(doc.get("albumAsin") or doc.get("asin") or ""),
name=str(doc.get("albumName") or doc.get("title") or ""),
artists=[str(doc.get("artistName") or "Unknown Artist")],
release_date="",
total_tracks=0,
album_type="album",
)
@classmethod
def from_metadata(cls, album_meta: Dict[str, Any], asin: str = "") -> "Album":
return cls(
id=str(album_meta.get("asin") or asin or ""),
name=str(album_meta.get("title") or ""),
artists=[str(album_meta.get("artistName") or "Unknown Artist")],
release_date=str(album_meta.get("release_date") or album_meta.get("releaseDate") or ""),
total_tracks=int(album_meta.get("trackCount") or 0),
album_type="album",
image_url=album_meta.get("image"),
explicit=album_meta.get("explicit"),
)
# ---------------------------------------------------------------------------
# Internal dataclasses for raw T2Tunes response parsing
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class T2TunesSearchItem:
asin: str
title: str
artist_name: str
item_type: str
album_name: str = ""
album_asin: str = ""
duration_seconds: int = 0
isrc: str = ""
@property
def is_album(self) -> bool:
return "album" in self.item_type.lower()
@property
def is_track(self) -> bool:
return "track" in self.item_type.lower()
@dataclass(frozen=True)
class T2TunesStreamInfo:
asin: str
streamable: bool
codec: str
format: str
sample_rate: Optional[int]
stream_url: str
decryption_key: Optional[str] # hex-encoded AES key; None when stream is clear
title: str = ""
artist: str = ""
album: str = ""
isrc: str = ""
cover_url: str = ""
track_number: Optional[int] = None
disc_number: Optional[int] = None
genre: str = ""
label: str = ""
date: str = ""
@property
def has_decryption_key(self) -> bool:
return bool(self.decryption_key)
# ---------------------------------------------------------------------------
# Rate-limit enforcement
# ---------------------------------------------------------------------------
def _rate_limit() -> None:
global _last_api_call
with _api_call_lock:
elapsed = time.monotonic() - _last_api_call
if elapsed < MIN_API_INTERVAL:
time.sleep(MIN_API_INTERVAL - elapsed)
_last_api_call = time.monotonic()
api_call_tracker.record_call("amazon")
# ---------------------------------------------------------------------------
# Main client
# ---------------------------------------------------------------------------
class AmazonClient:
"""T2Tunes-backed Amazon Music metadata and stream-info client."""
def __init__(
self,
base_url: Optional[str] = None,
country: Optional[str] = None,
preferred_codec: Optional[str] = None,
timeout: int = 30,
session: Optional[Any] = None,
) -> None:
self.base_url = (base_url or config_manager.get("amazon.base_url", DEFAULT_BASE_URL)).rstrip("/")
self.country = (country or config_manager.get("amazon.country", DEFAULT_COUNTRY)).upper()
self.preferred_codec = (
preferred_codec or config_manager.get("amazon.preferred_codec", DEFAULT_CODEC)
).lower()
self.timeout = timeout
self.session: Any = session or requests.Session()
if isinstance(self.session, requests.Session):
self.session.headers.update({
"Accept": "application/json",
"User-Agent": "SoulSync/1.0",
"Referer": self.base_url,
})
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def reload_config(self) -> None:
self.base_url = config_manager.get("amazon.base_url", DEFAULT_BASE_URL).rstrip("/")
self.country = config_manager.get("amazon.country", DEFAULT_COUNTRY).upper()
self.preferred_codec = config_manager.get("amazon.preferred_codec", DEFAULT_CODEC).lower()
def is_authenticated(self) -> bool:
"""Return True when the T2Tunes instance reports Amazon Music as up."""
try:
return str(self.status().get("amazonMusic", "")).lower() == "up"
except AmazonClientError:
return False
# ------------------------------------------------------------------
# Low-level API wrappers
# ------------------------------------------------------------------
def status(self) -> Dict[str, Any]:
return self._get_json("/api/status")
def search_raw(self, query: str, *, types: str = "track,album") -> List[T2TunesSearchItem]:
data = self._get_json(
"/api/amazon-music/search",
params={"query": query, "types": types, "country": self.country},
)
return list(self._iter_search_items(data))
def album_metadata(self, asin: str) -> Dict[str, Any]:
return self._get_json(
"/api/amazon-music/metadata",
params={"asin": asin, "country": self.country},
)
def media_from_asin(self, asin: str, codec: Optional[str] = None) -> List[T2TunesStreamInfo]:
effective_codec = (codec or self.preferred_codec).lower()
data = self._get_json(
"/api/amazon-music/media-from-asin",
params={"asin": asin, "country": self.country, "codec": effective_codec},
)
if isinstance(data, list):
return [self._parse_stream_info(item) for item in data if isinstance(item, dict)]
if isinstance(data, dict):
return [self._parse_stream_info(data)]
raise AmazonClientError(f"Unexpected media-from-asin response type: {type(data).__name__}")
# ------------------------------------------------------------------
# Metadata interface — mirrors DeezerClient / iTunesClient signatures
# ------------------------------------------------------------------
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
_rate_limit()
items = self.search_raw(query, types="track")
track_pairs: List[tuple] = [] # (Track, album_asin)
seen_album_asins: List[str] = []
for item in items:
if not item.is_track:
continue
track = Track.from_search_hit({
"asin": item.asin,
"title": _strip_edition(item.title),
"artistName": _primary_artist(item.artist_name),
"albumName": _strip_edition(item.album_name),
"albumAsin": item.album_asin,
"duration": item.duration_seconds,
"isrc": item.isrc,
})
track_pairs.append((track, item.album_asin))
if item.album_asin and item.album_asin not in seen_album_asins:
seen_album_asins.append(item.album_asin)
if len(track_pairs) >= limit:
break
album_metas = self._fetch_album_metas(seen_album_asins[:5])
tracks: List[Track] = []
for track, album_asin in track_pairs:
if album_asin and album_asin in album_metas:
meta = album_metas[album_asin]
track.image_url = meta.get("image")
track.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
track.total_tracks = meta.get("trackCount")
tracks.append(track)
return tracks
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
_rate_limit()
items = self.search_raw(query, types="track")
seen: Dict[str, Artist] = {}
artist_album_asin: Dict[str, str] = {} # artist name → first album ASIN seen
for item in items:
name = _primary_artist(item.artist_name)
if not name:
continue
if name not in seen:
seen[name] = Artist.from_name(name)
if name not in artist_album_asin and item.album_asin:
artist_album_asin[name] = item.album_asin
if len(seen) >= limit:
break
# T2Tunes has no artist images — use an album cover as stand-in.
unique_asins = list({v for v in artist_album_asin.values()})[:5]
album_metas = self._fetch_album_metas(unique_asins)
for name, artist in seen.items():
asin = artist_album_asin.get(name)
if asin and asin in album_metas:
artist.image_url = album_metas[asin].get("image")
return list(seen.values())
def search_albums(self, query: str, limit: int = 20) -> List[Album]:
_rate_limit()
items = self.search_raw(query, types="track")
album_candidates: List[tuple] = [] # (Album, asin)
seen_keys: set = set()
for item in items:
album_asin = item.album_asin
raw_name = item.album_name
if not raw_name:
continue
display_name = _strip_edition(raw_name)
artist = _primary_artist(item.artist_name)
# Collapse Explicit/Clean variants: same normalised name + artist = same album
dedup_key = (display_name.lower(), artist.lower())
if dedup_key in seen_keys:
continue
seen_keys.add(dedup_key)
album = Album.from_search_hit({
"albumAsin": album_asin,
"albumName": display_name,
"artistName": artist,
})
album_candidates.append((album, album_asin))
if len(album_candidates) >= limit:
break
album_metas = self._fetch_album_metas([a for _, a in album_candidates[:10]])
albums: List[Album] = []
for album, asin in album_candidates:
if asin in album_metas:
meta = album_metas[asin]
album.image_url = meta.get("image")
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
album.total_tracks = int(meta.get("trackCount") or 0)
albums.append(album)
return albums
def get_track_details(self, asin: str) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible dict for a single track ASIN."""
_rate_limit()
try:
streams = self.media_from_asin(asin)
except AmazonClientError:
return None
if not streams:
return None
s = streams[0]
album_data: Dict[str, Any] = {}
try:
meta = self.album_metadata(asin)
albums = meta.get("albumList")
if isinstance(albums, list) and albums and isinstance(albums[0], dict):
album_data = albums[0]
except AmazonClientError:
pass
return {
"id": s.asin,
"name": _strip_edition(s.title),
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
"album": {
"id": album_data.get("asin", ""),
"name": _strip_edition(s.album),
"images": [{"url": album_data["image"]}] if album_data.get("image") else [],
"release_date": album_data.get("release_date") or album_data.get("releaseDate") or s.date or "",
"total_tracks": album_data.get("trackCount", 0),
},
"duration_ms": 0,
"popularity": 0,
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
"track_number": s.track_number,
"disc_number": s.disc_number,
"isrc": s.isrc,
"is_album_track": True,
"raw_data": {
"codec": s.codec,
"format": s.format,
"sample_rate": s.sample_rate,
"streamable": s.streamable,
"has_decryption_key": s.has_decryption_key,
},
}
def get_album(self, asin: str, include_tracks: bool = True) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible album dict."""
_rate_limit()
try:
meta = self.album_metadata(asin)
except AmazonClientError:
return None
albums = meta.get("albumList")
if not isinstance(albums, list) or not albums:
return None
album = albums[0] if isinstance(albums[0], dict) else {}
result: Dict[str, Any] = {
"id": asin,
"name": _strip_edition(album.get("title", "")),
"artists": [{"name": _primary_artist(album.get("artistName", "")), "id": ""}],
"release_date": album.get("release_date") or album.get("releaseDate") or "",
"total_tracks": album.get("trackCount", 0),
"album_type": "album",
"images": [{"url": album["image"]}] if album.get("image") else [],
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
"label": album.get("label", ""),
}
if include_tracks:
tracks_data = self.get_album_tracks(asin) or {
"items": [], "total": 0, "limit": 50, "next": None,
}
result["tracks"] = tracks_data
# Backfill release_date from stream tags when album metadata lacks it.
if not result["release_date"]:
items = tracks_data.get("items") or []
for item in items:
rd = item.get("release_date") or ""
if rd and len(rd) >= 4:
result["release_date"] = rd
break
return result
def get_album_tracks(self, asin: str) -> Optional[Dict[str, Any]]:
"""Return album tracks in Spotify pagination format."""
_rate_limit()
try:
streams = self.media_from_asin(asin)
except AmazonClientError:
return None
# media_from_asin has no duration — enrich from search results which do.
duration_map: Dict[str, int] = {} # track asin → duration_ms
if streams:
album_name = _strip_edition(streams[0].album)
artist_name = _primary_artist(streams[0].artist)
try:
search_items = self.search_raw(
f"{album_name} {artist_name}", types="track"
)
for item in search_items:
if item.album_asin == asin and item.duration_seconds:
duration_map[item.asin] = item.duration_seconds * 1000
except Exception as exc:
logger.debug("Duration backfill failed for ASIN %s: %s", asin, exc)
items = [
{
"id": s.asin,
"name": _strip_edition(s.title),
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
"duration_ms": duration_map.get(s.asin, 0),
"track_number": s.track_number if s.track_number is not None else idx + 1,
"disc_number": s.disc_number,
"release_date": s.date or "",
"isrc": s.isrc,
}
for idx, s in enumerate(streams)
]
return {"items": items, "total": len(items), "limit": 50, "next": None}
def get_artist(self, artist_name: str) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible artist dict inferred from search results."""
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(search_name, types="track")
except AmazonClientError:
return None
name_lower = search_name.lower()
match = next(
(i for i in items if _primary_artist(i.artist_name).lower() == name_lower),
next((i for i in items if name_lower in _primary_artist(i.artist_name).lower()), None),
)
if not match:
return None
return {
"id": match.artist_name.lower().replace(" ", "_"),
"name": match.artist_name,
"genres": [],
"popularity": 0,
"followers": {"total": 0},
"images": [],
"external_urls": {},
}
def get_artist_albums(
self,
artist_name: str,
album_type: str = "album,single",
limit: int = 200,
) -> List[Album]:
"""Return albums for an artist inferred from search results."""
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(f"{search_name} album", types="track")
except AmazonClientError:
return []
album_candidates: List[tuple] = [] # (Album, asin)
seen_asins: set = set()
name_lower = search_name.lower()
for item in items:
if not item.is_album:
continue
if _primary_artist(item.artist_name).lower() != name_lower:
continue
album_asin = item.album_asin or item.asin
if album_asin in seen_asins:
continue
seen_asins.add(album_asin)
album = Album.from_search_hit({
"albumAsin": album_asin,
"albumName": _strip_edition(item.album_name or item.title),
"artistName": _primary_artist(item.artist_name),
})
album_candidates.append((album, album_asin))
if len(album_candidates) >= limit:
break
# Fetch metadata for art, release_date, track_count, and type inference.
# No explicit cap here — search_raw naturally bounds results to ~20 items,
# so album_candidates is already small.
asins_to_fetch = [a for _, a in album_candidates]
metas = self._fetch_album_metas(asins_to_fetch)
albums: List[Album] = []
for album, asin in album_candidates:
meta = metas.get(asin, {})
if meta:
album.image_url = meta.get("image")
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
total = int(meta.get("trackCount") or 0)
album.total_tracks = total
# T2Tunes doesn't expose release type — infer from track count.
# 1-track releases are singles; keep default "album" otherwise.
if total == 1:
album.album_type = "single"
albums.append(album)
return albums
def get_track_features(self, track_id: str) -> Optional[Dict[str, Any]]:
"""Not available from Amazon Music — returns None for compatibility."""
return None
def _get_artist_image_from_albums(self, artist_id: str) -> Optional[str]:
"""Return an album cover as artist image stand-in (T2Tunes has no artist images)."""
search_name = _unslugify(artist_id)
try:
items = self.search_raw(search_name, types="track")
except AmazonClientError:
return None
name_lower = search_name.lower()
for item in items:
if _primary_artist(item.artist_name).lower() != name_lower:
continue
asin = item.album_asin or item.asin
if not asin:
continue
metas = self._fetch_album_metas([asin])
if asin in metas and metas[asin].get("image"):
return metas[asin]["image"]
return None
# ==================== Interface Aliases (match DeezerClient method names) ====================
get_album_metadata = get_album
get_artist_info = get_artist
get_artist_albums_list = get_artist_albums
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
def _fetch_album_metas(self, asins: List[str]) -> Dict[str, Dict[str, Any]]:
"""Parallel-fetch album metadata for up to N ASINs. Returns {asin: albumList[0]}."""
if not asins:
return {}
metas: Dict[str, Dict[str, Any]] = {}
def _fetch(asin: str) -> None:
now = time.monotonic()
with _meta_cache_lock:
entry = _meta_cache.get(asin)
if entry and (now - entry[0]) < _META_CACHE_TTL:
metas[asin] = entry[1]
return
_rate_limit()
try:
raw = self.album_metadata(asin)
lst = raw.get("albumList")
if isinstance(lst, list) and lst and isinstance(lst[0], dict):
meta = lst[0]
with _meta_cache_lock:
_meta_cache[asin] = (time.monotonic(), meta)
metas[asin] = meta
except Exception as exc:
logger.debug("Album metadata fetch failed for ASIN %s: %s", asin, exc)
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=min(len(asins), 5)) as pool:
list(pool.map(_fetch, asins))
return metas
def _get_json(self, path: str, params: Optional[Dict[str, Any]] = None) -> Any:
url = urljoin(f"{self.base_url}/", path.lstrip("/"))
last_exc: Optional[Exception] = None
for attempt in range(3):
if attempt:
time.sleep(1.0 * attempt)
try:
resp = self.session.get(url, params=params, timeout=self.timeout)
resp.raise_for_status()
except requests.HTTPError as exc:
body = exc.response.text[:500].replace("\n", " ")
# T2Tunes returns 400 for transient Amazon-side failures — retry those.
if exc.response.status_code == 400 and "Failed to search" in exc.response.text:
logger.debug("T2Tunes transient 400 on attempt %d, retrying: %s", attempt + 1, url)
last_exc = AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}"
)
continue
raise AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}",
status_code=exc.response.status_code,
) from exc
except requests.RequestException as exc:
raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
try:
return resp.json()
except ValueError as exc:
preview = resp.text[:200].replace("\n", " ")
raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
raise last_exc # type: ignore[misc]
@staticmethod
def _iter_search_items(response: Any) -> Iterator[T2TunesSearchItem]:
if not isinstance(response, dict):
raise AmazonClientError(
f"Unexpected search response type: {type(response).__name__}"
)
for result in response.get("results") or []:
if not isinstance(result, dict):
continue
for hit in result.get("hits") or []:
if not isinstance(hit, dict):
continue
doc = hit.get("document")
if not isinstance(doc, dict):
continue
asin = str(doc.get("asin") or "")
if not asin:
continue
yield T2TunesSearchItem(
asin=asin,
title=str(doc.get("title") or ""),
artist_name=str(doc.get("artistName") or ""),
item_type=str(doc.get("__type") or ""),
album_name=str(doc.get("albumName") or ""),
album_asin=str(doc.get("albumAsin") or ""),
duration_seconds=int(doc.get("duration") or 0),
isrc=str(doc.get("isrc") or ""),
)
@staticmethod
def _parse_stream_info(item: Dict[str, Any]) -> T2TunesStreamInfo:
stream_info = item.get("streamInfo") if isinstance(item.get("streamInfo"), dict) else {}
tags = item.get("tags") if isinstance(item.get("tags"), dict) else {}
# T2Tunes API has a typo: "stremeable" in some responses
streamable = item.get("streamable")
if streamable is None:
streamable = item.get("stremeable")
raw_key = item.get("decryptionKey")
decryption_key = str(raw_key) if raw_key else None
def _int_tag(key: str) -> Optional[int]:
v = tags.get(key)
try:
return int(v) if v is not None else None
except (TypeError, ValueError):
return None
return T2TunesStreamInfo(
asin=str(item.get("asin") or ""),
streamable=bool(streamable),
codec=str(stream_info.get("codec") or ""),
format=str(stream_info.get("format") or ""),
sample_rate=(
stream_info.get("sampleRate")
if isinstance(stream_info.get("sampleRate"), int)
else None
),
stream_url=str(stream_info.get("streamUrl") or ""),
decryption_key=decryption_key,
title=str(tags.get("title") or ""),
artist=str(tags.get("artist") or ""),
album=str(tags.get("album") or ""),
isrc=str(tags.get("isrc") or ""),
cover_url=str(item.get("coverUrl") or ""),
track_number=_int_tag("trackNumber"),
disc_number=_int_tag("discNumber"),
genre=str(tags.get("genre") or ""),
label=str(tags.get("label") or ""),
date=str(tags.get("date") or ""),
)

466
core/amazon_download_client.py
View file

@ -1,466 +0,0 @@
"""Amazon Music download source plugin backed by a T2Tunes proxy.
NOT yet wired into the app registry — validated in isolation only.
See tests/tools/test_amazon_download_client.py.
Filename encoding (the DownloadSourcePlugin dispatch contract):
"{asin}||{display_name}"
e.g. "B09XYZ1234||Kendrick Lamar - Not Like Us"
Codec preference order: FLAC → Opus → EAC3.
Download flow (from Tubifarry reference implementation):
1. GET stream_url → encrypted bytes on disk
2. FFmpeg -decryption_key <hex> to decrypt in place
3. Embed metadata tags (handled by the app's standard post-processing)
"""
from __future__ import annotations
import os
import shutil
import subprocess
import threading
import time
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
import requests as http_requests
from config.settings import config_manager
from core.amazon_client import AmazonClient, AmazonClientError
from core.download_plugins.base import DownloadSourcePlugin
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.quality.model import AudioQuality
from core.quality.source_map import quality_tier_for_source
from utils.logging_config import get_logger
logger = get_logger("amazon_download_client")
# Quality / codec helpers
CODEC_PREFERENCE = ["flac", "opus", "eac3"]
_CODEC_EXTENSIONS: Dict[str, str] = {
"flac": "flac",
"ogg_vorbis": "ogg",
"opus": "opus",
"eac3": "eac3",
"mp4": "m4a",
"aac": "m4a",
"mp3": "mp3",
}
MIN_AUDIO_BYTES = 512 * 1024 # 512 KB — anything smaller is a broken stream
def _codec_key(codec: str) -> str:
return codec.lower().replace("-", "_").replace(" ", "_")
def _file_extension(codec: str) -> str:
return _CODEC_EXTENSIONS.get(_codec_key(codec), "bin")
def _quality_label(codec: str, sample_rate: Optional[int] = None) -> str:
ck = _codec_key(codec)
if ck == "flac":
if sample_rate and sample_rate > 48000:
return "Hi-Res"
return "Lossless"
return "Lossy"
class AmazonDownloadClient(DownloadSourcePlugin):
"""DownloadSourcePlugin — Amazon Music via T2Tunes proxy."""
def __init__(self, download_path: Optional[str] = None) -> None:
if download_path is None:
download_path = config_manager.get("soulseek.download_path", "./downloads")
self.download_path = Path(download_path)
try:
self.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {self.download_path}: {e}")
self._quality = quality_tier_for_source("amazon", default="flac")
self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
self._client = AmazonClient(preferred_codec=self._quality)
self.session = http_requests.Session()
self.session.headers.update({
"User-Agent": "SoulSync/1.0",
"Accept": "*/*",
})
self._engine: Any = None
self.shutdown_check: Any = None
# ------------------------------------------------------------------
# DownloadSourcePlugin — lifecycle
# ------------------------------------------------------------------
def set_engine(self, engine) -> None:
"""Engine callback — wires the central thread worker + state store."""
self._engine = engine
def set_shutdown_check(self, check_callable) -> None:
self.shutdown_check = check_callable
def is_configured(self) -> bool:
# T2Tunes has a public default instance; no credentials required.
# Return True unconditionally so the source shows as available.
return True
async def check_connection(self) -> bool:
try:
return self._client.is_authenticated()
except Exception:
return False
# ------------------------------------------------------------------
# DownloadSourcePlugin — search
# ------------------------------------------------------------------
async def search(
self,
query: str,
timeout: Optional[int] = None,
progress_callback: Any = None,
) -> Tuple[List[TrackResult], List[AlbumResult]]:
try:
items = self._client.search_raw(query, types="track,album")
except AmazonClientError as exc:
logger.warning(f"Amazon search failed for {query!r}: {exc}")
return [], []
track_results: List[TrackResult] = []
album_map: Dict[str, AlbumResult] = {}
album_order: List[str] = []
preferred = self._client.preferred_codec
# Search results only carry the codec (real sample_rate arrives at
# stream time). Claim the format honestly — FLAC for the lossless
# codec, lossy otherwise — so audio_quality derives a real format
# instead of the display label ("Lossless"), and the post-download
# probe pins the actual sample_rate/bit_depth.
amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
for item in items:
quality = _quality_label(preferred)
if item.is_track:
tr = TrackResult(
username="amazon",
filename=f"{item.asin}||{item.artist_name} - {item.title}",
size=0,
bitrate=None,
duration=item.duration_seconds * 1000 if item.duration_seconds else None,
quality=quality,
free_upload_slots=999,
upload_speed=999_999,
queue_length=0,
artist=item.artist_name,
title=item.title,
album=item.album_name,
_source_metadata={
"asin": item.asin,
"album_asin": item.album_asin,
"isrc": item.isrc,
},
)
tr.set_quality(amazon_q)
track_results.append(tr)
elif item.is_album:
album_asin = item.album_asin or item.asin
if album_asin not in album_map:
placeholder = TrackResult(
username="amazon",
filename=f"{item.asin}||{item.artist_name} - {item.title}",
size=0,
bitrate=None,
duration=None,
quality=quality,
free_upload_slots=999,
upload_speed=999_999,
queue_length=0,
artist=item.artist_name,
title=item.title,
album=item.album_name,
)
placeholder.set_quality(amazon_q)
album_map[album_asin] = AlbumResult(
username="amazon",
album_path=album_asin,
album_title=item.album_name or item.title,
artist=item.artist_name,
track_count=0,
total_size=0,
tracks=[placeholder],
dominant_quality=quality,
)
album_order.append(album_asin)
return track_results, [album_map[k] for k in album_order]
# ------------------------------------------------------------------
# DownloadSourcePlugin — download dispatch
# ------------------------------------------------------------------
async def download(
self,
username: str,
filename: str,
file_size: int = 0,
) -> Optional[str]:
if "||" not in filename:
logger.error(f"Invalid Amazon filename format (no '||'): {filename!r}")
return None
if self._engine is None:
raise RuntimeError(
"AmazonDownloadClient._engine is not set — "
"client not connected to download infrastructure"
)
asin, display_name = filename.split("||", 1)
asin = asin.strip()
display_name = display_name.strip()
return self._engine.worker.dispatch(
source_name="amazon",
target_id=asin,
display_name=display_name,
original_filename=filename,
impl_callable=self._download_sync,
)
def _download_sync(
self,
download_id: str,
target_id: str,
display_name: str,
) -> Optional[str]:
asin = str(target_id)
codecs = CODEC_PREFERENCE if self._allow_fallback else [self._quality]
for codec in codecs:
try:
streams = self._client.media_from_asin(asin, codec=codec)
except AmazonClientError as exc:
logger.warning(f"media_from_asin({asin!r}, {codec}) failed: {exc}")
continue
stream = next(
(s for s in streams if s.streamable and s.stream_url),
None,
)
if not stream:
logger.debug(f"No streamable result for {asin} at codec={codec}")
continue
ext = _file_extension(stream.codec or codec)
safe = "".join(
ch if ch.isalnum() or ch in " -_." else "_"
for ch in display_name
)[:80]
# T2Tunes always serves audio in an encrypted MP4 container.
# The codec extension (.flac, .opus, .eac3) is only for the
# final decrypted output.
enc_ext = "mp4" if stream.decryption_key else ext
enc_path = self._unique_path(self.download_path / f"{safe}.enc.{enc_ext}")
out_path = self._unique_path(self.download_path / f"{safe}.{ext}")
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {"state": "downloading", "progress": 0.0}
)
try:
downloaded = self._stream_to_file(stream.stream_url, enc_path, download_id)
except Exception as exc:
logger.warning(f"Stream download failed for {asin} ({codec}): {exc}")
enc_path.unlink(missing_ok=True)
continue
if downloaded < MIN_AUDIO_BYTES:
logger.warning(
f"File too small ({downloaded} B) for {asin} at {codec} — trying next"
)
enc_path.unlink(missing_ok=True)
continue
if stream.decryption_key:
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {"state": "decrypting", "progress": 1.0}
)
try:
self._decrypt_with_ffmpeg(enc_path, out_path, stream.decryption_key)
enc_path.unlink(missing_ok=True)
except Exception as exc:
logger.error(f"Decryption failed for {asin} ({codec}): {exc}")
enc_path.unlink(missing_ok=True)
out_path.unlink(missing_ok=True)
continue
else:
enc_path.rename(out_path)
final_size = out_path.stat().st_size
logger.info(
f"Amazon download complete ({codec}): {out_path} "
f"({final_size / (1024 * 1024):.1f} MB)"
)
# Sync size == transferred so the download monitor's bytes-incomplete
# guard doesn't block post-processing. The throttled updates in
# _stream_to_file leave transferred < size after the last 0.5s tick;
# other streaming clients avoid this by not tracking bytes at all
# (size stays 0, the guard is skipped). Writing the final output size
# here restores parity.
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {'size': final_size, 'transferred': final_size}
)
return str(out_path)
logger.error(f"All codec tiers exhausted for '{display_name}' ({asin})")
return None
def _decrypt_with_ffmpeg(
self, enc_path: Path, out_path: Path, hex_key: str
) -> None:
"""Decrypt a T2Tunes encrypted audio file using FFmpeg -decryption_key.
T2Tunes uses CENC (Common Encryption) for DRM-protected tracks.
FFmpeg supports decryption via the -decryption_key flag when the
key is provided in hex.
"""
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
tools_dir = Path(__file__).parent.parent / "tools"
candidate = tools_dir / ("ffmpeg.exe" if os.name == "nt" else "ffmpeg")
if candidate.exists():
ffmpeg = str(candidate)
else:
raise RuntimeError(
"ffmpeg is required for Amazon Music decryption. "
"Install ffmpeg and ensure it is on your PATH."
)
cmd = [
ffmpeg,
"-y",
"-hide_banner",
"-loglevel", "error",
"-decryption_key", hex_key,
"-i", str(enc_path),
"-map", "0:a:0", # extract first audio stream (FLAC/Opus/EAC3 inside MP4)
"-c", "copy",
str(out_path),
]
logger.debug(f"Decrypting {enc_path.name} → {out_path.name}")
result = subprocess.run(cmd, capture_output=True)
if result.returncode != 0:
stderr = result.stderr.decode("utf-8", errors="replace").strip()
raise RuntimeError(f"FFmpeg decryption failed (exit {result.returncode}): {stderr}")
def _stream_to_file(self, url: str, out_path: Path, download_id: str) -> int:
resp = self.session.get(url, stream=True, timeout=60)
resp.raise_for_status()
total = int(resp.headers.get("content-length", 0))
downloaded = 0
last_report = time.monotonic()
shutdown_triggered = False
with out_path.open("wb") as fh:
for chunk in resp.iter_content(chunk_size=64 * 1024):
if not chunk:
continue
if self.shutdown_check and self.shutdown_check():
shutdown_triggered = True
break
fh.write(chunk)
downloaded += len(chunk)
now = time.monotonic()
if self._engine and now - last_report >= 0.5:
self._engine.update_record(
"amazon",
download_id,
{
"transferred": downloaded,
"size": total,
"progress": downloaded / total if total else 0.0,
},
)
last_report = now
if shutdown_triggered:
out_path.unlink(missing_ok=True)
raise RuntimeError("Shutdown requested mid-download")
return downloaded
# ------------------------------------------------------------------
# DownloadSourcePlugin — status interface
# ------------------------------------------------------------------
async def get_all_downloads(self) -> List[DownloadStatus]:
if self._engine is None:
return []
return [
self._record_to_status(record)
for record in self._engine.iter_records_for_source('amazon')
]
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
if self._engine is None:
return None
record = self._engine.get_record('amazon', download_id)
return self._record_to_status(record) if record is not None else None
async def cancel_download(
self,
download_id: str,
username: Optional[str] = None,
remove: bool = False,
) -> bool:
if self._engine is None:
return False
if self._engine.get_record('amazon', download_id) is None:
return False
self._engine.update_record('amazon', download_id, {'state': 'Cancelled'})
if remove:
self._engine.remove_record('amazon', download_id)
return True
async def clear_all_completed_downloads(self) -> bool:
if self._engine is None:
return True
terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
for record in list(self._engine.iter_records_for_source('amazon')):
if record.get('state') in terminal:
self._engine.remove_record('amazon', record['id'])
return True
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
@staticmethod
def _unique_path(path: Path) -> Path:
if not path.exists():
return path
stem, suffix = path.stem, path.suffix
for i in range(1, 100):
candidate = path.with_name(f"{stem} ({i}){suffix}")
if not candidate.exists():
return candidate
return path.with_name(f"{stem}_{uuid.uuid4().hex[:8]}{suffix}")
@staticmethod
def _record_to_status(rec: Dict[str, Any]) -> DownloadStatus:
return DownloadStatus(
id=str(rec.get('id', '')),
filename=str(rec.get('filename', '')),
username='amazon',
state=str(rec.get('state', 'queued')),
progress=float(rec.get('progress', 0.0)),
size=int(rec.get('size', 0)),
transferred=int(rec.get('transferred', 0)),
speed=int(rec.get('speed', 0)),
time_remaining=rec.get('time_remaining'),
file_path=rec.get('file_path'),
)

61
core/amazon_outage.py
View file

@ -1,61 +0,0 @@
"""Amazon enrichment outage detection + back-off — pure, importable, testable.
The Amazon worker enriches via a public T2Tunes proxy instance. When that
instance is down (HTTP 5xx, "Amazon Music API is not initialized", or an
unreachable host), the worker must NOT treat every album as an individual
failure: doing so floods the logs with an error per item, churns network + DB
continuously, and permanently marks the whole library ``error`` (which the
retry tiers never re-attempt) for what is really a transient outage.
Instead it recognizes "the whole source is down", leaves the item untouched so
it's retried once the instance recovers, and backs off hard. These two pure
helpers carry that logic so it can be unit-tested without the worker, the DB,
or the network.
"""
from __future__ import annotations
import re
# HTTP statuses that mean "the source/proxy is unhealthy", not "no match".
_OUTAGE_STATUS = {500, 502, 503, 504}
# Substrings (lower-cased) in an error message that indicate a source outage
# rather than a per-item miss: proxy not ready, gateway errors, the host being
# unreachable, or an error page returned instead of JSON.
_OUTAGE_PHRASES = (
"not initialized", "not configured", "service unavailable",
"bad gateway", "gateway time", "request failed", "response not json",
"max retries", "connection", "timed out", "temporarily unavailable",
)
# Back-off schedule while the source is down.
_NORMAL_DELAY = 2 # seconds between items when healthy
_OUTAGE_BASE = 30 # first back-off step
_OUTAGE_CAP = 1800 # 30 minutes max
def is_source_outage(exc: Exception) -> bool:
"""True when ``exc`` indicates the Amazon source/proxy is down (transient,
whole-source), as opposed to a normal per-item error.
Robust to how the error is surfaced: an explicit ``status_code`` attribute,
an ``HTTP <code>`` prefix in the message, or an outage phrase (covers
connection failures and non-JSON error pages that carry no status code)."""
code = getattr(exc, "status_code", None)
if isinstance(code, int) and code in _OUTAGE_STATUS:
return True
msg = str(exc).lower()
m = re.search(r"http\s+(\d{3})", msg)
if m and int(m.group(1)) in _OUTAGE_STATUS:
return True
return any(p in msg for p in _OUTAGE_PHRASES)
def next_poll_delay_seconds(outage_streak: int) -> int:
"""Seconds to wait before the next item. Normal cadence when healthy;
escalating back-off (30s, 60s, 120s, … capped at 30 min) the longer the
source has been down, so a dead instance can't flood logs/CPU/DB."""
if outage_streak <= 0:
return _NORMAL_DELAY
return min(_OUTAGE_BASE * (2 ** min(outage_streak - 1, 6)), _OUTAGE_CAP)

645
core/amazon_worker.py
View file

@ -1,645 +0,0 @@
import re
import threading
import time
from difflib import SequenceMatcher
from typing import Optional, Dict, Any
from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.amazon_client import AmazonClient
from core.worker_utils import interruptible_sleep, set_album_api_track_count
from core.enrichment.manual_match_honoring import honor_stored_match
from core.amazon_outage import is_source_outage, next_poll_delay_seconds
logger = get_logger("amazon_worker")
class AmazonWorker:
"""Background worker for enriching library artists, albums, and tracks with Amazon Music metadata."""
def __init__(self, database: MusicDatabase):
self.db = database
self.client = AmazonClient()
self._amazon_schema_checked = False
self.running = False
self.paused = False
self.should_stop = False
self.thread = None
self._stop_event = threading.Event()
self.current_item = None
self.stats = {
'matched': 0,
'not_found': 0,
'pending': 0,
'errors': 0,
}
self.retry_days = 30
self.name_similarity_threshold = 0.80
# Source-outage circuit breaker: counts consecutive whole-source
# failures (proxy down / "not initialized" / unreachable) so the loop
# backs off instead of grinding the whole library item-by-item.
self._outage_streak = 0
logger.info("Amazon background worker initialized")
def _ensure_amazon_schema(self, cursor) -> None:
"""Ensure upgraded installs have the Amazon enrichment columns.
MusicDatabase normally runs this migration during startup, but the
worker should still be defensive because it is the code path that
repeatedly queries these columns in the background.
"""
if self._amazon_schema_checked:
return
table_columns = {
'artists': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'albums': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'tracks': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
}
for table, columns in table_columns.items():
cursor.execute(f"PRAGMA table_info({table})")
existing = {row[1] for row in cursor.fetchall()}
for column in columns:
if column not in existing:
column_type = 'TIMESTAMP' if column == 'amazon_last_attempted' else 'TEXT'
cursor.execute(f"ALTER TABLE {table} ADD COLUMN {column} {column_type}")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_id ON artists (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_status ON artists (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_id ON albums (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_status ON albums (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_id ON tracks (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_status ON tracks (amazon_match_status)")
cursor.connection.commit()
self._amazon_schema_checked = True
def start(self):
if self.running:
logger.warning("Worker already running")
return
self.running = True
self.should_stop = False
self._stop_event.clear()
self.thread = threading.Thread(target=self._run, daemon=True)
self.thread.start()
logger.info("Amazon background worker started")
def stop(self):
if not self.running:
return
logger.info("Stopping Amazon worker...")
self.should_stop = True
self.running = False
self._stop_event.set()
if self.thread:
self.thread.join(timeout=1)
logger.info("Amazon worker stopped")
def pause(self):
if not self.running:
logger.warning("Worker not running, cannot pause")
return
self.paused = True
logger.info("Amazon worker paused")
def resume(self):
if not self.running:
logger.warning("Worker not running, start it first")
return
self.paused = False
logger.info("Amazon worker resumed")
def get_stats(self) -> Dict[str, Any]:
self.stats['pending'] = self._count_pending_items()
progress = self._get_progress_breakdown()
is_actually_running = self.running and (self.thread is not None and self.thread.is_alive())
is_idle = is_actually_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
return {
'enabled': True,
'running': is_actually_running and not self.paused,
'paused': self.paused,
'idle': is_idle,
'current_item': self.current_item,
'stats': self.stats.copy(),
'progress': progress,
}
def _run(self):
logger.info("Amazon worker thread started")
while not self.should_stop:
try:
if self.paused:
interruptible_sleep(self._stop_event, 1)
continue
self.current_item = None
item = self._get_next_item()
if not item:
logger.debug("No pending items, sleeping...")
interruptible_sleep(self._stop_event, 10)
continue
self.current_item = item
item_id = item.get('id') or item.get('artist_id') or item.get('album_id')
if item_id is None:
logger.warning(f"Skipping {item.get('type', 'unknown')} with NULL id: {item.get('name', '?')}")
continue
self._process_item(item)
# Normal 2s cadence when healthy; escalating back-off (up to
# 30 min) while the source is in an outage streak.
interruptible_sleep(self._stop_event, next_poll_delay_seconds(self._outage_streak))
except Exception as e:
logger.error(f"Error in worker loop: {e}")
interruptible_sleep(self._stop_event, 5)
logger.info("Amazon worker thread finished")
def _get_next_item(self) -> Optional[Dict[str, Any]]:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('amazon')
if _prio:
_pi = priority_pending_item(cursor, 'amazon', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists
WHERE amazon_match_status IS NULL AND id IS NOT NULL
ORDER BY id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 2: Unattempted albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.amazon_match_status IS NULL AND a.id IS NOT NULL
ORDER BY a.id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
# Priority 3: Unattempted tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.amazon_match_status IS NULL AND t.id IS NOT NULL
ORDER BY t.id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
# Priority 4: Retry not_found artists
cursor.execute("""
SELECT id, name FROM artists
WHERE amazon_match_status = 'not_found' AND amazon_last_attempted < ?
ORDER BY amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 5: Retry not_found albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.amazon_match_status = 'not_found' AND a.amazon_last_attempted < ?
ORDER BY a.amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
# Priority 6: Retry not_found tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.amazon_match_status = 'not_found' AND t.amazon_last_attempted < ?
ORDER BY t.amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
return None
except Exception as e:
logger.error(f"Error getting next item: {e}")
return None
finally:
if conn:
conn.close()
def _normalize_name(self, name: str) -> str:
name = name.lower().strip()
name = re.sub(r'\s+[-–—]\s+.*$', '', name)
name = re.sub(r'\s*\(.*?\)\s*', ' ', name)
name = re.sub(r'[^\w\s]', '', name)
name = re.sub(r'\s+', ' ', name).strip()
return name
def _name_matches(self, query_name: str, result_name: str) -> bool:
norm_query = self._normalize_name(query_name)
norm_result = self._normalize_name(result_name)
similarity = SequenceMatcher(None, norm_query, norm_result).ratio()
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
return similarity >= self.name_similarity_threshold
def _process_item(self, item: Dict[str, Any]):
try:
item_type = item['type']
item_id = item['id']
item_name = item['name']
logger.debug(f"Processing {item_type} #{item_id}: {item_name}")
if item_type == 'artist':
self._process_artist(item_id, item_name)
elif item_type == 'album':
self._process_album(item_id, item_name, item.get('artist', ''), item)
elif item_type == 'track':
self._process_track(item_id, item_name, item.get('artist', ''), item)
# The source answered (match or not_found) — clear any outage streak.
if self._outage_streak:
logger.info("Amazon source recovered after %d outage(s), resuming",
self._outage_streak)
self._outage_streak = 0
except Exception as e:
if is_source_outage(e):
# The whole source is down (proxy 5xx / "not initialized" /
# unreachable). Do NOT mark the item 'error' — that would burn
# the entire library to a state the retry tiers never re-attempt
# for a transient outage. Leave it untouched so it's retried once
# the instance recovers, and let the loop back off. Log once per
# streak to avoid flooding.
self._outage_streak += 1
if self._outage_streak == 1:
logger.warning("Amazon source unavailable — pausing enrichment "
"until it recovers: %s", e)
else:
logger.debug("Amazon source still unavailable (streak=%d): %s",
self._outage_streak, e)
return
# A non-outage error means the source actually answered (e.g. a
# 404/parse error on a real response), so the outage is over —
# clear the streak and handle this as a normal per-item error.
self._outage_streak = 0
logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
self.stats['errors'] += 1
try:
self._mark_status(item['type'], item['id'], 'error')
except Exception as e2:
logger.error(f"Error updating item status: {e2}")
def _get_existing_id(self, entity_type: str, entity_id: int) -> Optional[str]:
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
return None
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"SELECT amazon_id FROM {table} WHERE id = ?", (entity_id,))
row = cursor.fetchone()
return row[0] if row and row[0] else None
except Exception:
return None
finally:
if conn:
conn.close()
def _process_artist(self, artist_id: int, artist_name: str):
existing_id = self._get_existing_id('artist', artist_id)
if existing_id:
logger.debug(f"Preserving existing Amazon ID for artist '{artist_name}': {existing_id}")
return
results = self.client.search_artists(artist_name, limit=5)
if results:
result = results[0]
if self._name_matches(artist_name, result.name):
self._update_artist(artist_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{artist_name}' -> Amazon ID: {result.id}")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result.name}')")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for artist '{artist_name}'")
def _refresh_album_via_stored_id(self, album_id, stored_id, api_data):
self._update_album(album_id, api_data, stored_id)
def _refresh_track_via_stored_id(self, track_id, stored_id, api_data):
self._update_track(track_id, api_data, stored_id)
def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
if honor_stored_match(
db=self.db, entity_table='albums', entity_id=album_id,
id_column='amazon_id',
client_fetch_fn=lambda asin: self.client.get_album(asin, include_tracks=False),
on_match_fn=self._refresh_album_via_stored_id,
log_prefix='Amazon',
):
self.stats['matched'] += 1
return
query = f"{artist_name} {album_name}"
results = self.client.search_albums(query, limit=10)
if results:
result = results[0]
if self._name_matches(album_name, result.name):
full_album = None
if result.id:
try:
full_album = self.client.get_album(result.id, include_tracks=False)
except Exception as e:
logger.warning(f"Failed to fetch full album '{album_name}' (ASIN: {result.id}): {e}")
if full_album is None:
self._mark_status('album', album_id, 'error')
self.stats['errors'] += 1
logger.warning(f"Album '{album_name}' matched but full details unavailable, will retry")
return
self._update_album(album_id, full_album, result.id)
self.stats['matched'] += 1
logger.info(f"Matched album '{album_name}' -> Amazon ASIN: {result.id}")
else:
self._mark_status('album', album_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for album '{album_name}' (got '{result.name}')")
else:
self._mark_status('album', album_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for album '{album_name}'")
def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
if honor_stored_match(
db=self.db, entity_table='tracks', entity_id=track_id,
id_column='amazon_id',
client_fetch_fn=self.client.get_track_details,
on_match_fn=self._refresh_track_via_stored_id,
log_prefix='Amazon',
):
self.stats['matched'] += 1
return
query = f"{artist_name} {track_name}"
results = self.client.search_tracks(query, limit=10)
if results:
result = results[0]
if self._name_matches(track_name, result.name):
full_track = None
if result.id:
try:
full_track = self.client.get_track_details(result.id)
except Exception as e:
logger.warning(f"Failed to fetch full track '{track_name}' (ASIN: {result.id}): {e}")
if full_track is None:
self._mark_status('track', track_id, 'error')
self.stats['errors'] += 1
logger.warning(f"Track '{track_name}' matched but full details unavailable, will retry")
return
self._update_track(track_id, full_track, result.id)
self.stats['matched'] += 1
logger.info(f"Matched track '{track_name}' -> Amazon ASIN: {result.id}")
else:
self._mark_status('track', track_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for track '{track_name}' (got '{result.name}')")
else:
self._mark_status('track', track_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for track '{track_name}'")
def _update_artist(self, artist_id: int, result):
"""Store Amazon metadata for an artist. ``result`` is an Artist dataclass."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE artists SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (str(result.id), artist_id))
# Backfill thumb_url from album cover stand-in when artist has no image
image_url = result.image_url
if not image_url:
try:
image_url = self.client._get_artist_image_from_albums(result.id)
except Exception as exc:
logger.debug("Artist image from albums failed for %s: %s", result.id, exc)
if image_url:
cursor.execute("""
UPDATE artists SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (image_url, artist_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating artist #{artist_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _update_album(self, album_id: int, full_data: Dict[str, Any], asin: str):
"""Store Amazon metadata for an album. ``full_data`` is a get_album() dict."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE albums SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (asin, album_id))
# Backfill label when missing
label = full_data.get('label')
if label:
cursor.execute("""
UPDATE albums SET label = ?
WHERE id = ? AND (label IS NULL OR label = '')
""", (label, album_id))
# Backfill thumb_url
images = full_data.get('images') or []
thumb_url = images[0].get('url') if images else None
if thumb_url:
cursor.execute("""
UPDATE albums SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (thumb_url, album_id))
# Cache authoritative track count for completeness repair
total_tracks = full_data.get('total_tracks') or (
full_data.get('tracks', {}).get('total') if isinstance(full_data.get('tracks'), dict) else None
)
set_album_api_track_count(cursor, album_id, total_tracks)
conn.commit()
except Exception as e:
logger.error(f"Error updating album #{album_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _update_track(self, track_id: int, full_data: Dict[str, Any], asin: str):
"""Store Amazon metadata for a track. ``full_data`` is a get_track_details() dict."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE tracks SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP
WHERE id = ?
""", (asin, track_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating track #{track_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _mark_status(self, entity_type: str, entity_id: int, status: str):
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
logger.error(f"Unknown entity type: {entity_type}")
return
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"""
UPDATE {table} SET
amazon_match_status = ?,
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (status, entity_id))
conn.commit()
except Exception as e:
logger.error(f"Error marking {entity_type} #{entity_id} status: {e}")
finally:
if conn:
conn.close()
def _count_pending_items(self) -> int:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
cursor.execute("""
SELECT
(SELECT COUNT(*) FROM artists WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM albums WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM tracks WHERE amazon_match_status IS NULL AND id IS NOT NULL)
AS pending
""")
row = cursor.fetchone()
return row[0] if row else 0
except Exception as e:
logger.error(f"Error counting pending items: {e}")
return 0
finally:
if conn:
conn.close()
def _get_progress_breakdown(self) -> Dict[str, Dict[str, int]]:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
progress = {}
for table in ('artists', 'albums', 'tracks'):
cursor.execute(f"""
SELECT
COUNT(*) AS total,
SUM(CASE WHEN amazon_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
FROM {table}
""")
row = cursor.fetchone()
if row:
total, processed = row[0], row[1] or 0
progress[table] = {
'matched': processed,
'total': total,
'percent': int((processed / total * 100) if total > 0 else 0),
}
return progress
except Exception as e:
logger.error(f"Error getting progress breakdown: {e}")
return {}
finally:
if conn:
conn.close()

43
core/api_call_tracker.py
View file

@ -12,11 +12,6 @@ import threading
import time
from collections import deque, defaultdict
from utils.logging_config import get_logger
logger = get_logger("api_call_tracker")
# Known rate limits per service (calls/minute)
RATE_LIMITS = {
@ -30,7 +25,6 @@ RATE_LIMITS = {
'tidal': 120, # MIN_API_INTERVAL=0.5s → ~120/min
'qobuz': 60, # Variable throttle, ~60/min estimate
'discogs': 60, # MIN_API_INTERVAL=1.0s with auth → ~60/min
'amazon': 120, # MIN_API_INTERVAL=0.5s → ~120/min (T2Tunes proxy)
}
# Display names for UI
@ -45,13 +39,12 @@ SERVICE_LABELS = {
'tidal': 'Tidal',
'qobuz': 'Qobuz',
'discogs': 'Discogs',
'amazon': 'Amazon Music',
}
# Display order
SERVICE_ORDER = [
'spotify', 'itunes', 'deezer', 'lastfm', 'genius',
'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs', 'amazon',
'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs',
]
@ -269,12 +262,7 @@ class ApiCallTracker:
def save(self):
"""Persist 24h minute history to disk. Call on shutdown.
Uses an atomic write (write to a sibling .tmp file, fsync, rename) so
a SIGINT/SIGTERM/crash mid-write can't leave the JSON file truncated.
Without this, an interrupted shutdown corrupts the history file and
the next startup loses 24h of metrics."""
"""Persist 24h minute history to disk. Call on shutdown."""
try:
now = time.time()
cutoff = now - 86400
@ -290,31 +278,16 @@ class ApiCallTracker:
if entries:
data[key] = entries
events = [dict(e) for e in self._events if e['ts'] >= cutoff]
payload = {'ts': now, 'history': data, 'events': events}
tmp_path = _PERSIST_PATH + '.tmp'
with open(tmp_path, 'w') as f:
json.dump(payload, f)
f.flush()
os.fsync(f.fileno())
os.replace(tmp_path, _PERSIST_PATH)
with open(_PERSIST_PATH, 'w') as f:
json.dump({'ts': now, 'history': data, 'events': events}, f)
except Exception as e:
logger.error(f"[ApiCallTracker] Failed to save history: {e}")
# Best-effort cleanup of stale tmp file from a failed write.
try:
if os.path.exists(_PERSIST_PATH + '.tmp'):
os.remove(_PERSIST_PATH + '.tmp')
except Exception as e:
logger.debug("remove stale tmp file failed: %s", e)
print(f"[ApiCallTracker] Failed to save history: {e}")
def _load(self):
"""Restore 24h minute history from disk. Called on init."""
try:
if not os.path.exists(_PERSIST_PATH):
return
if os.path.getsize(_PERSIST_PATH) == 0:
logger.info(f"[ApiCallTracker] History file is empty, starting fresh: {_PERSIST_PATH}")
return
with open(_PERSIST_PATH, 'r') as f:
raw = json.load(f)
saved_ts = raw.get('ts', 0)
@ -332,11 +305,9 @@ class ApiCallTracker:
for e in events:
if e.get('ts', 0) >= cutoff:
self._events.append(e)
logger.info(f"[ApiCallTracker] Restored history for {len(history)} services, {len(events)} events")
except json.JSONDecodeError as e:
logger.warning(f"[ApiCallTracker] History file is not valid JSON, starting fresh: {_PERSIST_PATH} ({e})")
print(f"[ApiCallTracker] Restored history for {len(history)} services, {len(events)} events")
except Exception as e:
logger.error(f"[ApiCallTracker] Failed to load history: {e}")
print(f"[ApiCallTracker] Failed to load history: {e}")
# Singleton instance

238
core/archive_pipeline.py
View file

@ -1,238 +0,0 @@
"""Archive extraction + audio-file discovery for torrent / usenet downloads.
The torrent and usenet download plugins need a uniform way to:
1. Walk the downloader's save directory and find every audio file in it.
2. If the directory contains an archive (``.zip`` / ``.rar`` / ``.tar`` /
``.7z``), extract it first so the audio files inside become walkable.
This module is intentionally narrow — no matching, no tagging, no
import. The download plugin layer composes this with the existing
post-processing / matching pipeline. Lidarr does NOT use this module:
Lidarr extracts archives in its own import step before SoulSync sees
the files at all. Usenet downloaders (SABnzbd, NZBGet) also auto-
extract by default. Torrents are the main case where SoulSync may
need to do the extract step itself — most music torrents ship loose,
but some bundle the album in a ``.rar`` archive.
``rarfile`` is an optional dependency. If it isn't installed, archives
with ``.rar`` content are skipped with a single warning rather than
crashing the download.
"""
from __future__ import annotations
import tarfile
import zipfile
from pathlib import Path
from typing import List, Optional
from utils.logging_config import get_logger
logger = get_logger("archive_pipeline")
# Same audio-extension set as ``core/imports/file_ops.py`` ``quality_tiers``.
# Keep them in sync — if a new format is added to file_ops, add it here too
# or the walker will skip it and the download plugin will mark the download
# failed even when files arrived.
AUDIO_EXTENSIONS = frozenset([
# lossless
'.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
# high lossy
'.opus', '.ogg',
# standard lossy
'.m4a', '.aac',
# low lossy
'.mp3', '.wma',
])
ARCHIVE_EXTENSIONS = frozenset(['.zip', '.rar', '.tar', '.tar.gz', '.tgz', '.7z'])
def is_archive(path: Path) -> bool:
"""True if the file extension looks like a supported archive.
Compound extensions (``.tar.gz``, ``.tar.bz2``) are detected by
checking the last two suffixes joined together — Path.suffix
only returns the final suffix.
"""
if not path.is_file():
return False
name = path.name.lower()
if name.endswith(('.tar.gz', '.tar.bz2', '.tar.xz')):
return True
return path.suffix.lower() in ARCHIVE_EXTENSIONS
def walk_audio_files(directory: Path) -> List[Path]:
"""Recursively scan ``directory`` for audio files. Returns
a sorted list of absolute paths. Empty list if the directory
doesn't exist or contains no audio.
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
out: List[Path] = []
for child in directory.rglob('*'):
if not child.is_file():
continue
if child.suffix.lower() in AUDIO_EXTENSIONS:
out.append(child.resolve())
out.sort()
return out
def find_archives_in_dir(directory: Path) -> List[Path]:
"""Find every archive file directly inside ``directory`` (one
level deep — torrents normally put the archive at the root of
their folder; we don't search nested dirs to avoid extracting
something we shouldn't).
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
return sorted(p for p in directory.iterdir() if is_archive(p))
def extract_archive(archive_path: Path, extract_to: Optional[Path] = None) -> Optional[Path]:
"""Extract a single archive in-place (or to ``extract_to`` if
given). Returns the directory the archive was extracted into,
or ``None`` on failure.
Supports ``.zip``, ``.tar``/``.tar.gz``/``.tar.bz2``/``.tar.xz``,
and ``.rar`` (only when the optional ``rarfile`` library is
installed). ``.7z`` is recognised but extraction requires
``py7zr``; without it, the call logs and returns None.
"""
if not archive_path or not archive_path.exists():
logger.warning("archive_pipeline: %s does not exist", archive_path)
return None
dest = extract_to or archive_path.parent
dest.mkdir(parents=True, exist_ok=True)
name = archive_path.name.lower()
try:
if name.endswith('.zip'):
with zipfile.ZipFile(archive_path) as zf:
_safe_extract_zip(zf, dest)
return dest
if name.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
with tarfile.open(archive_path) as tf:
_safe_extract_tar(tf, dest)
return dest
if name.endswith('.rar'):
return _extract_rar(archive_path, dest)
if name.endswith('.7z'):
return _extract_7z(archive_path, dest)
except (zipfile.BadZipFile, tarfile.TarError, OSError) as e:
logger.error("archive_pipeline: failed to extract %s: %s", archive_path, e)
return None
logger.warning("archive_pipeline: unknown archive type for %s", archive_path)
return None
def extract_all_in_dir(directory: Path) -> List[Path]:
"""Find every archive in ``directory`` and extract each in place.
Returns the list of directories archives were extracted into
(usually all the same — ``directory`` itself). Archives that
failed to extract are skipped silently after a warning.
"""
out: List[Path] = []
for archive in find_archives_in_dir(directory):
result = extract_archive(archive)
if result is not None:
out.append(result)
return out
def collect_audio_after_extraction(directory: Path) -> List[Path]:
"""One-shot helper for the download plugins: extract any archives
in the directory, then return the walked audio file list. This is
the common pattern — torrent / usenet plugin gets a save_path,
calls this, hands the resulting files to the matching pipeline.
"""
extract_all_in_dir(directory)
return walk_audio_files(directory)
# ---------------------------------------------------------------------------
# Safety helpers
# ---------------------------------------------------------------------------
def _safe_extract_zip(zf: zipfile.ZipFile, dest: Path) -> None:
"""Extract a zipfile after rejecting any member whose resolved
path escapes ``dest`` (path traversal protection).
"""
dest = dest.resolve()
for member in zf.namelist():
target = (dest / member).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member)
return
zf.extractall(dest)
def _safe_extract_tar(tf: tarfile.TarFile, dest: Path) -> None:
"""Same path-traversal protection for tarfiles."""
dest = dest.resolve()
for member in tf.getmembers():
target = (dest / member.name).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member.name)
return
# ``filter='data'`` is the Python 3.12+ safe extractor; fall back
# to the legacy call on older runtimes.
try:
tf.extractall(dest, filter='data') # type: ignore[call-arg]
except TypeError:
tf.extractall(dest)
def _extract_rar(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import rarfile # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — rarfile library not installed. "
"Install with: pip install rarfile (and ensure unrar is on PATH).",
archive_path,
)
return None
try:
with rarfile.RarFile(archive_path) as rf:
dest_resolved = dest.resolve()
for name in rf.namelist():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal rar member %r", name)
return None
rf.extractall(dest)
return dest
except Exception as e:
logger.error("archive_pipeline: rar extract failed for %s: %s", archive_path, e)
return None
def _extract_7z(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import py7zr # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — py7zr library not installed. "
"Install with: pip install py7zr.",
archive_path,
)
return None
try:
with py7zr.SevenZipFile(archive_path, 'r') as sz:
dest_resolved = dest.resolve()
for name in sz.getnames():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal 7z member %r", name)
return None
sz.extractall(path=dest)
return dest
except Exception as e:
logger.error("archive_pipeline: 7z extract failed for %s: %s", archive_path, e)
return None

212
core/artist_source_detail.py
View file

@ -1,212 +0,0 @@
"""Synthesize an artist-detail response for an artist that isn't in the library.
Extracted from ``web_server.py`` so the logic is importable at test time.
The route handler in ``web_server.py`` is now a thin wrapper that builds the
per-source clients (which live as module globals there), calls this function,
and wraps the return value in ``jsonify``.
Used by ``/api/artist-detail/<id>`` when the URL is called with a ``source``
query parameter and the library DB lookup misses. Enriches the response with
whatever metadata we can pull on demand:
* Image URL (via ``core.metadata.artist_image.get_artist_image_url``)
* Source-specific artist info — genres + follower count from the named
source's ``get_artist`` / ``get_artist_info`` helper
* Last.fm bio + listeners + playcount + URL (by artist name)
* Discography from the named source, with variant dedup disabled so every
release surfaces
All per-source clients are passed in explicitly. Callers that can't or don't
want to provide a given client pass ``None`` — the corresponding enrichment
branch becomes a no-op.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, Optional, Tuple
from core.artist_source_lookup import SOURCE_ID_FIELD
from core.metadata import artist_image as metadata_artist_image
from core.metadata import discography as metadata_discography
from core.metadata.lookup import MetadataLookupOptions
logger = logging.getLogger("artist_source_detail")
def build_source_only_artist_detail(
artist_id: str,
artist_name: str,
source: str,
*,
spotify_client: Optional[Any] = None,
deezer_client: Optional[Any] = None,
itunes_client: Optional[Any] = None,
discogs_client: Optional[Any] = None,
amazon_client: Optional[Any] = None,
lastfm_api_key: Optional[str] = None,
) -> Tuple[Dict[str, Any], int]:
"""Build the artist-detail payload for a source-only artist.
Returns ``(payload_dict, http_status)``. Callers wrap the dict in
``jsonify`` or equivalent. Status is 200 on success, 404 when the
source's discography lookup returned no releases.
"""
resolved_name = (artist_name or "").strip()
# 1. Image URL via the same helper /api/artist/<id>/image uses.
image_url: Optional[str] = None
try:
image_url = metadata_artist_image.get_artist_image_url(artist_id, source_override=source)
except Exception as e:
logger.debug(f"Artist image lookup failed for {source}:{artist_id}: {e}")
# 2. Source-side artist info (image, genres, followers depending on source).
source_genres: list = []
source_followers: Optional[int] = None
try:
if source == "spotify" and spotify_client is not None:
sp_artist = spotify_client.get_artist(artist_id, allow_fallback=False)
if sp_artist:
if not artist_name and sp_artist.get("name"):
resolved_name = sp_artist["name"]
source_genres = sp_artist.get("genres") or []
source_followers = (sp_artist.get("followers") or {}).get("total")
if not image_url and sp_artist.get("images"):
image_url = sp_artist["images"][0].get("url")
elif source == "deezer" and deezer_client is not None:
dz_artist = deezer_client.get_artist_info(artist_id)
if dz_artist:
if not artist_name and dz_artist.get("name"):
resolved_name = dz_artist["name"]
source_genres = dz_artist.get("genres") or []
source_followers = (dz_artist.get("followers") or {}).get("total")
elif source == "itunes" and itunes_client is not None:
it_artist = itunes_client.get_artist(artist_id)
if it_artist:
if not artist_name and it_artist.get("name"):
resolved_name = it_artist["name"]
source_genres = it_artist.get("genres") or []
elif source == "discogs" and discogs_client is not None:
dc_artist = discogs_client.get_artist(artist_id)
if dc_artist:
if not artist_name and dc_artist.get("name"):
resolved_name = dc_artist["name"]
source_genres = dc_artist.get("genres") or []
elif source == "amazon" and amazon_client is not None:
az_artist = amazon_client.get_artist(resolved_name or artist_id)
if az_artist:
if not artist_name and az_artist.get("name"):
resolved_name = az_artist["name"]
source_genres = az_artist.get("genres") or []
if not image_url and az_artist.get("images"):
image_url = az_artist["images"][0].get("url")
elif source == "musicbrainz":
try:
from core.musicbrainz_search import MusicBrainzSearchClient
mb = MusicBrainzSearchClient()
mb_artist = mb.get_artist(artist_id)
if mb_artist:
if not artist_name and mb_artist.get("name"):
resolved_name = mb_artist["name"]
source_genres = mb_artist.get("genres") or []
except Exception as e:
logger.debug(f"MusicBrainz artist info lookup failed for {artist_id}: {e}")
except Exception as e:
logger.debug(f"Source-side artist info lookup failed for {source}:{artist_id}: {e}")
# 3. Last.fm enrichment by artist name.
lastfm_bio: Optional[str] = None
lastfm_listeners: Optional[int] = None
lastfm_playcount: Optional[int] = None
lastfm_url: Optional[str] = None
if resolved_name and lastfm_api_key:
try:
from core.lastfm_client import LastFMClient
lastfm = LastFMClient(api_key=lastfm_api_key)
lf_info = lastfm.get_artist_info(resolved_name)
if lf_info:
bio_obj = lf_info.get("bio") or {}
lastfm_bio = bio_obj.get("content") or bio_obj.get("summary")
stats_obj = lf_info.get("stats") or {}
if stats_obj.get("listeners"):
try:
lastfm_listeners = int(stats_obj["listeners"])
except (ValueError, TypeError):
pass
if stats_obj.get("playcount"):
try:
lastfm_playcount = int(stats_obj["playcount"])
except (ValueError, TypeError):
pass
lastfm_url = lf_info.get("url")
except Exception as e:
logger.debug(f"Last.fm enrichment failed for {resolved_name}: {e}")
# 4. Discography from the specified source. Skip variant dedup so the
# page shows every release the source returns — matches the inline
# Artists-page behaviour that this view was modelled after.
discography_result = metadata_discography.get_artist_detail_discography(
artist_id,
artist_name=resolved_name or artist_id,
options=MetadataLookupOptions(
source_override=source,
allow_fallback=True,
skip_cache=False,
max_pages=0,
# Match the Download Discography endpoint cap (200).
# Spotify already paginates all; Deezer / iTunes / Discogs /
# Hydrabase clamp at the outer limit. 200 covers prolific
# catalogues without exceeding iTunes/Discogs internal caps.
limit=200,
artist_source_ids={source: artist_id},
dedup_variants=False,
),
)
if not discography_result.get("success"):
return {
"success": False,
"error": discography_result.get("error", "Could not load discography"),
"source": source,
}, 404
artist_info: Dict[str, Any] = {
"id": artist_id,
"name": resolved_name or artist_id,
"image_url": image_url,
"server_source": None, # not in library
"genres": source_genres,
}
# Stamp the source-specific ID so the correct service badge renders on the
# hero (e.g. source=deezer -> deezer_id; source=spotify -> spotify_artist_id).
source_id_field = SOURCE_ID_FIELD.get(source)
if source_id_field:
artist_info[source_id_field] = artist_id
if source_followers is not None:
artist_info["followers"] = source_followers
if lastfm_bio:
artist_info["lastfm_bio"] = lastfm_bio
if lastfm_listeners is not None:
artist_info["lastfm_listeners"] = lastfm_listeners
if lastfm_playcount is not None:
artist_info["lastfm_playcount"] = lastfm_playcount
if lastfm_url:
artist_info["lastfm_url"] = lastfm_url
logger.info(
f"Source-only artist-detail: {artist_info['name']} from {source} — "
f"albums={len(discography_result.get('albums', []))}, "
f"eps={len(discography_result.get('eps', []))}, "
f"singles={len(discography_result.get('singles', []))}, "
f"genres={len(source_genres)}, lastfm_bio={'yes' if lastfm_bio else 'no'}"
)
return {
"success": True,
"artist": artist_info,
"discography": discography_result,
"enrichment_coverage": {},
}, 200

105
core/artist_source_lookup.py
View file

@ -1,105 +0,0 @@
"""Source-artist → library lookup helpers.
Extracted from `web_server.py` so the logic can be imported and unit-tested
without booting the Flask app, Spotify client, Soulseek connection, etc.
Two concepts live here:
* ``SOURCE_ID_FIELD`` — the per-source column on the ``artists`` table that
stores the external service ID (Spotify track ID, Deezer artist ID, …).
This map is what ties a result clicked in the source-aware Search results
back to a library record so we can serve the richer library view.
* ``find_library_artist_for_source`` — given a source-aware click (e.g.
``deezer:525046``), try to locate a matching library artist. First by
direct column match against the source's ID column, then by case-
insensitive name match scoped to the active media server.
"""
from __future__ import annotations
import logging
from typing import Optional
from core.source_ids import id_column as _artist_id_column
logger = logging.getLogger("artist_source_lookup")
SOURCE_ONLY_ARTIST_SOURCES = frozenset({
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
})
# The per-source column on the ``artists`` table, derived from the canonical
# source-ID registry (the single source of truth). Values are unchanged from the
# previous hardcoded map — this just stops duplicating that knowledge here.
SOURCE_ID_FIELD = {
source: _artist_id_column(source, "artist")
for source in (
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
)
}
def find_library_artist_for_source(
database,
source: str,
source_artist_id: str,
artist_name: Optional[str] = None,
active_server: Optional[str] = None,
) -> Optional[str]:
"""Return the library PK of an artist matching the source-aware click.
Lookup order:
1. Direct match on the source-specific ID column (server-agnostic — any
library record with the right external ID is a hit). If that id is
stamped on MORE than one library artist, the mapping is corrupt /
ambiguous (e.g. an enrichment bug wrote one Deezer id onto several
artists) — we refuse to guess and fall through, so the caller can
show the source artist directly instead of an arbitrary wrong one.
2. Case-insensitive name match within ``active_server`` (defaults to the
active media server when not provided), so we don't jump the user
across server contexts on a name collision.
Returns ``None`` on miss or on any database error.
"""
column = SOURCE_ID_FIELD.get(source)
if not column:
return None
try:
with database._get_connection() as conn:
cursor = conn.cursor()
# LIMIT 2 so we can tell a unique match from an ambiguous one.
cursor.execute(
f"SELECT id FROM artists WHERE {column} = ? LIMIT 2",
(str(source_artist_id),),
)
rows = cursor.fetchall()
if len(rows) == 1:
return rows[0][0]
if len(rows) > 1:
# Same source id on multiple artists — corrupt mapping. Don't
# upgrade on the id; fall through to the name match (and, if
# that misses, let the caller render the source artist).
logger.warning(
f"Source id {source}:{source_artist_id} maps to "
f"{len(rows)}+ library artists — ambiguous, skipping "
f"id-based library upgrade"
)
if artist_name and active_server:
cursor.execute(
"SELECT id FROM artists "
"WHERE LOWER(name) = LOWER(?) AND server_source = ? LIMIT 1",
(artist_name, active_server),
)
row = cursor.fetchone()
if row:
return row[0]
except Exception as e:
logger.debug(
f"Library upgrade lookup failed for {source}:{source_artist_id}: {e}"
)
return None

324
core/artists/liked_match.py
View file

@ -1,324 +0,0 @@
"""Liked-artist multi-source matching — lifted from web_server.py.
Both function bodies are byte-identical to the originals. The
``spotify_client`` proxy + ``_get_*_client`` shims let the bodies resolve
their original names without any modification.
"""
import logging
import time
from config.settings import config_manager
from core.metadata.registry import (
get_deezer_client,
get_discogs_client,
get_itunes_client,
get_musicbrainz_client,
get_spotify_client,
)
logger = logging.getLogger(__name__)
def _get_itunes_client():
"""Mirror of web_server._get_itunes_client — delegates to registry."""
return get_itunes_client()
def _get_deezer_client():
"""Mirror of web_server._get_deezer_client — delegates to registry."""
return get_deezer_client()
def _get_discogs_client(token=None):
"""Mirror of web_server._get_discogs_client — delegates to registry."""
return get_discogs_client(token)
def _get_musicbrainz_client():
"""Mirror of web_server._get_musicbrainz_client — delegates to registry."""
return get_musicbrainz_client()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily so a Spotify re-auth that
rebinds the cached client in core.metadata.registry is visible to the
lifted bodies."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
spotify_client = _SpotifyClientProxy()
def _match_liked_artists_to_all_sources(database, profile_id: int):
"""Match pending liked artists to ALL metadata sources (Spotify, iTunes, Deezer, Discogs).
Uses the same matching pattern as the watchlist scanner: DB-first, then API search
with fuzzy name matching. Stores all resolved IDs so source switching works instantly."""
pending = database.get_liked_artists_pending_match(profile_id, limit=200)
if not pending:
return
# Source → column mapping
source_cols = {
'spotify': 'spotify_artist_id',
'itunes': 'itunes_artist_id',
'deezer': 'deezer_artist_id',
'discogs': 'discogs_artist_id',
'musicbrainz': 'musicbrainz_artist_id',
}
id_cols = list(source_cols.values())
# Reject known placeholder images and local server paths
_placeholder_hashes = {'2a96cbd8b46e442fc41c2b86b821562f'}
def _valid_image(url):
if not url or not url.strip():
return None
if any(ph in url for ph in _placeholder_hashes):
return None
# Reject local media server paths (Plex/Jellyfin) — not loadable in browser
if url.startswith('/') or url.startswith('\\'):
return None
if not url.startswith('http'):
return None
return url
# Build search clients for each source
from core.deezer_client import DeezerClient
search_clients = {}
if spotify_client and spotify_client.is_spotify_authenticated():
search_clients['spotify'] = spotify_client
try:
search_clients['itunes'] = _get_itunes_client()
except Exception as e:
logger.debug("itunes client init failed: %s", e)
try:
search_clients['deezer'] = _get_deezer_client()
except Exception as e:
logger.debug("deezer client init failed: %s", e)
try:
dc = _get_discogs_client()
# Only use Discogs if token is configured
from config.settings import config_manager as _cm
if _cm.get('discogs.token', ''):
search_clients['discogs'] = dc
except Exception as e:
logger.debug("discogs client init failed: %s", e)
try:
search_clients['musicbrainz'] = _get_musicbrainz_client()
except Exception as e:
logger.debug("musicbrainz client init failed: %s", e)
# Reuse watchlist scanner's fuzzy matching logic
from core.watchlist_scanner import WatchlistScanner
_normalize = WatchlistScanner._normalize_artist_name
def _best_match(results, artist_name):
"""Pick best match from search results using name similarity (same as watchlist scanner)."""
if not results:
return None
# Exact normalized match
for r in results:
if _normalize(r.name) == _normalize(artist_name):
return r
# Fuzzy scoring
best = None
best_sim = 0
for r in results:
# Simple normalized comparison
n1 = _normalize(artist_name)
n2 = _normalize(r.name)
if n1 == n2:
return r
# Levenshtein-style similarity
max_len = max(len(n1), len(n2))
if max_len == 0:
continue
distance = sum(1 for a, b in zip(n1, n2, strict=False) if a != b) + abs(len(n1) - len(n2))
sim = (max_len - distance) / max_len
if sim > best_sim:
best_sim = sim
best = r
if best and best_sim >= 0.85:
return best
return None
api_calls = 0
matched = 0
for entry in pending:
name = entry['artist_name']
pool_id = entry['id']
harvested_ids = {}
best_image = None
# Pre-load existing IDs from the entry itself
for col in id_cols:
if entry.get(col):
harvested_ids[col] = entry[col]
# --- DB STRATEGIES (free, no API calls) ---
# 1. Library artists table
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("SELECT * FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (name,))
row = cursor.fetchone()
if row:
r = dict(row)
for col in id_cols:
if r.get(col) and col not in harvested_ids:
harvested_ids[col] = str(r[col])
if _valid_image(r.get('thumb_url')):
best_image = r['thumb_url']
except Exception as e:
logger.debug("library artist lookup failed: %s", e)
# 2. Watchlist artists
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute(
"SELECT * FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE AND profile_id = ? LIMIT 1",
(name, profile_id)
)
row = cursor.fetchone()
if row:
wl = dict(row)
for col in id_cols:
if wl.get(col) and col not in harvested_ids:
harvested_ids[col] = str(wl[col])
if _valid_image(wl.get('image_url')) and not best_image:
best_image = wl['image_url']
except Exception as e:
logger.debug("watchlist artist lookup failed: %s", e)
# 3. Metadata cache (all sources)
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute(
"SELECT entity_id, source, image_url FROM metadata_cache_entities WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE",
(name,)
)
for row in cursor.fetchall():
col = source_cols.get(row['source'])
if col and col not in harvested_ids:
harvested_ids[col] = row['entity_id']
if _valid_image(row['image_url']) and not best_image:
best_image = row['image_url']
except Exception as e:
logger.debug("metadata cache lookup failed: %s", e)
# --- API STRATEGIES (search each missing source) ---
# Same pattern as watchlist scanner's _backfill_missing_ids
for source, col in source_cols.items():
if col in harvested_ids:
continue # Already have this source's ID
client = search_clients.get(source)
if not client:
continue
if api_calls >= 200: # Hard cap per refresh cycle
break
try:
results = client.search_artists(name, limit=5)
best = _best_match(results, name)
if best:
harvested_ids[col] = best.id
if hasattr(best, 'image_url') and _valid_image(best.image_url) and not best_image:
best_image = best.image_url
api_calls += 1
time.sleep(0.4) # Rate limit breathing room
except Exception as e:
logger.debug(f"[Your Artists] {source} search failed for '{name}': {e}")
api_calls += 1
# Save all harvested IDs
if harvested_ids:
# Determine best active source/ID — prefer Spotify, then iTunes, Deezer, Discogs
resolved_source = None
resolved_id = None
for src in ('spotify', 'itunes', 'deezer', 'discogs', 'musicbrainz'):
col = source_cols[src]
if col in harvested_ids:
resolved_source = src
resolved_id = harvested_ids[col]
break
database.update_liked_artist_match(
pool_id, active_source=resolved_source, active_source_id=resolved_id,
image_url=best_image, all_ids=harvested_ids
)
matched += 1
database.sync_liked_artists_watchlist_flags(profile_id)
logger.info(f"[Your Artists] Matched {matched}/{len(pending)} artists to {len(search_clients)} sources ({api_calls} API calls)")
# Image backfill: fetch images for matched artists that have IDs but no image
_backfill_liked_artist_images(database, profile_id, search_clients)
def _backfill_liked_artist_images(database, profile_id: int, search_clients: dict):
"""Fetch images for matched artists missing artwork using their stored source IDs."""
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id
FROM liked_artists_pool
WHERE profile_id = ? AND match_status = 'matched'
AND (image_url IS NULL OR image_url = ''
OR image_url LIKE '%2a96cbd8b46e442fc41c2b86b821562f%'
OR image_url NOT LIKE 'http%')
LIMIT 100
""", (profile_id,))
rows = cursor.fetchall()
if not rows:
return
logger.info(f"[Your Artists] Backfilling images for {len(rows)} artists...")
filled = 0
for row in rows:
r = dict(row)
image_url = None
# Try Spotify artist lookup (has best images)
if r.get('spotify_artist_id') and 'spotify' in search_clients:
try:
sp = search_clients['spotify']
if hasattr(sp, 'sp') and sp.sp:
artist_data = sp.sp.artist(r['spotify_artist_id'])
if artist_data and artist_data.get('images'):
image_url = artist_data['images'][0]['url']
except Exception as e:
logger.debug("spotify artist image fetch failed: %s", e)
# Try Deezer (direct image URL from ID)
if not image_url and r.get('deezer_artist_id'):
image_url = f"https://api.deezer.com/artist/{r['deezer_artist_id']}/image?size=big"
if image_url:
try:
cursor2 = conn.cursor()
cursor2.execute(
"UPDATE liked_artists_pool SET image_url = ? WHERE id = ?",
(image_url, r['id'])
)
filled += 1
except Exception as e:
logger.debug("liked artist image update failed: %s", e)
time.sleep(0.3)
conn.commit()
if filled:
logger.info(f"[Your Artists] Backfilled {filled}/{len(rows)} artist images")
except Exception as e:
logger.debug(f"[Your Artists] Image backfill error: {e}")

994
core/artists/map.py
View file

@ -1,994 +0,0 @@
"""Artist Map endpoints — lifted from web_server.py.
The four route bodies (``get_artist_map_data``, ``get_artist_map_genre_list``,
``get_artist_map_genres``, ``get_artist_map_explore``) plus their cache helpers
and the artist-map cache are byte-identical to the originals. Module-level
shims for ``get_current_profile_id``, ``_get_itunes_client``, and the
``spotify_client`` proxy let the bodies resolve their original names without
modification.
"""
import json
import logging
import time
from flask import g, jsonify, request
from database.music_database import get_database
from core.metadata.registry import get_itunes_client, get_spotify_client
logger = logging.getLogger(__name__)
def get_current_profile_id() -> int:
"""Mirror of web_server.get_current_profile_id — uses Flask g.
Catches RuntimeError too because reading `g` outside a request
context raises that (not AttributeError) — happens when this is
called from background threads (sync, automation, scanners)."""
try:
return g.profile_id
except (AttributeError, RuntimeError):
return 1
def _get_itunes_client():
"""Mirror of web_server._get_itunes_client — delegates to registry."""
return get_itunes_client()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily so a Spotify re-auth that
rebinds the cached client in core.metadata.registry is visible to the
lifted route bodies."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
spotify_client = _SpotifyClientProxy()
# Artist Map data cache — avoids re-querying 4+ tables on every request
# Keys: 'watchlist_{profile}', 'genres_{profile}', 'genre_list'
# Values: {'data': <json-ready dict>, 'ts': <timestamp>}
_artist_map_cache = {}
_ARTIST_MAP_CACHE_TTL = 300 # 5 minutes
def _artmap_cache_get(key):
"""Get cached artist map data if still fresh."""
entry = _artist_map_cache.get(key)
if entry and (time.time() - entry['ts']) < _ARTIST_MAP_CACHE_TTL:
return entry['data']
return None
def _artmap_cache_set(key, data):
"""Store artist map data in cache."""
_artist_map_cache[key] = {'data': data, 'ts': time.time()}
def _artmap_cache_invalidate(profile_id=None):
"""Invalidate artist map cache (call after watchlist changes, scans, etc.)."""
if profile_id:
_artist_map_cache.pop(f'watchlist_{profile_id}', None)
_artist_map_cache.pop(f'genres_{profile_id}', None)
_artist_map_cache.pop('genre_list', None)
def get_artist_map_data():
"""Get watchlist artists + their similar artists for the force-directed artist map."""
try:
database = get_database()
profile_id = get_current_profile_id()
cached = _artmap_cache_get(f'watchlist_{profile_id}')
if cached:
return jsonify(cached)
# Get all watchlist artists
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
discogs_artist_id, image_url
FROM watchlist_artists WHERE profile_id = ?
""", (profile_id,))
watchlist_rows = cursor.fetchall()
nodes = [] # {id, name, image_url, type: 'watchlist'|'similar', genres, size}
edges = [] # {source, target, weight}
seen_names = {} # normalized_name → node index
def _norm(name):
return (name or '').lower().strip()
# Add watchlist artists as anchor nodes
for wa in watchlist_rows:
w = dict(wa)
norm = _norm(w['artist_name'])
if norm in seen_names:
continue
idx = len(nodes)
seen_names[norm] = idx
# Get image — prefer HTTP URLs
img = w.get('image_url', '') or ''
if img and not img.startswith('http'):
img = ''
nodes.append({
'id': idx,
'name': w['artist_name'],
'image_url': img,
'type': 'watchlist',
'genres': [],
'spotify_id': w.get('spotify_artist_id') or '',
'itunes_id': w.get('itunes_artist_id') or '',
'deezer_id': w.get('deezer_artist_id') or '',
'discogs_id': w.get('discogs_artist_id') or '',
'source_db_id': str(w['id']),
})
# Get all similar artists for all watchlist artists
watchlist_ids = [dict(wa)['spotify_artist_id'] or dict(wa)['itunes_artist_id'] or str(dict(wa)['id']) for wa in watchlist_rows]
if watchlist_ids:
placeholders = ','.join(['?'] * len(watchlist_ids))
cursor.execute(f"""
SELECT source_artist_id, similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similarity_rank, occurrence_count, image_url, genres, popularity
FROM similar_artists
WHERE profile_id = ? AND source_artist_id IN ({placeholders})
ORDER BY similarity_rank ASC
""", [profile_id] + watchlist_ids)
for row in cursor.fetchall():
r = dict(row)
sim_norm = _norm(r['similar_artist_name'])
# Find or create similar artist node
if sim_norm not in seen_names:
idx = len(nodes)
seen_names[sim_norm] = idx
img = r.get('image_url', '') or ''
if img and not img.startswith('http'):
img = ''
genres = []
if r.get('genres'):
try:
genres = json.loads(r['genres'])
except Exception as e:
logger.debug("similar node genres parse failed: %s", e)
nodes.append({
'id': idx,
'name': r['similar_artist_name'],
'image_url': img,
'type': 'similar',
'genres': genres,
'spotify_id': r.get('similar_artist_spotify_id') or '',
'itunes_id': r.get('similar_artist_itunes_id') or '',
'deezer_id': r.get('similar_artist_deezer_id') or '',
'musicbrainz_id': r.get('similar_artist_musicbrainz_id') or '',
'rank': r.get('similarity_rank', 5),
'occurrence': r.get('occurrence_count', 1),
'popularity': r.get('popularity', 0),
})
sim_idx = seen_names[sim_norm]
# Find the watchlist node that sourced this similar artist
source_norm = None
for wa in watchlist_rows:
w = dict(wa)
sid = w.get('spotify_artist_id') or w.get('itunes_artist_id') or str(w['id'])
if sid == r['source_artist_id']:
source_norm = _norm(w['artist_name'])
break
if source_norm and source_norm in seen_names:
source_idx = seen_names[source_norm]
# Weight: inverse of rank (rank 1 = strongest connection)
weight = max(1, 11 - (r.get('similarity_rank', 5)))
edges.append({
'source': source_idx,
'target': sim_idx,
'weight': weight,
})
# Also check if any similar artists ARE watchlist artists (cross-links)
# These create extra connections between watchlist nodes
for i, node in enumerate(nodes):
if node['type'] == 'similar':
# Check if this similar artist is also a watchlist artist
for j, wnode in enumerate(nodes):
if wnode['type'] == 'watchlist' and i != j:
if _norm(node['name']) == _norm(wnode['name']):
# Merge: upgrade the similar node to watchlist
node['type'] = 'watchlist'
break
# ── Backfill from metadata cache: batch-lookup all node names across all sources ──
# Single query to get ALL cached artist entries matching ANY node name
try:
all_names = list(set(_norm(n['name']) for n in nodes if n.get('name')))
if all_names:
# Build case-insensitive IN clause via temp matching
# Lightweight query — no raw_json (can be huge)
cursor.execute("""
SELECT entity_id, source, name, image_url, genres, popularity
FROM metadata_cache_entities
WHERE entity_type = 'artist'
""")
cache_rows = cursor.fetchall()
# Index cache by normalized name → {source: {id, image_url, genres}}
cache_by_name = {}
for cr in cache_rows:
cn = _norm(cr['name'] or '')
if cn not in cache_by_name:
cache_by_name[cn] = {}
source = cr['source']
genres = []
if cr['genres']:
try:
genres = json.loads(cr['genres']) if isinstance(cr['genres'], str) else []
except Exception as e:
logger.debug("backfill cache genres parse failed: %s", e)
cache_by_name[cn][source] = {
'id': cr['entity_id'],
'image_url': cr['image_url'] or '',
'genres': genres,
}
# Apply cache data to nodes
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
for n in nodes:
nn = _norm(n['name'])
cached = cache_by_name.get(nn)
if not cached:
continue
for source, field in source_id_map.items():
if not n.get(field) and source in cached:
n[field] = cached[source]['id']
# Backfill image if missing or local path
if not n.get('image_url') or not n['image_url'].startswith('http'):
for source in ('spotify', 'deezer', 'itunes'):
if source in cached and cached[source].get('image_url', '').startswith('http'):
n['image_url'] = cached[source]['image_url']
break
# Backfill genres if missing
if not n.get('genres') or len(n.get('genres', [])) == 0:
for source in ('spotify', 'deezer', 'itunes', 'discogs', 'musicbrainz'):
if source in cached and cached[source].get('genres'):
n['genres'] = cached[source]['genres'][:5]
break
# Deezer direct URL fallback
for n in nodes:
if not n.get('image_url') or not n['image_url'].startswith('http'):
if n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
# Album art fallback (iTunes artists have no artist images)
_album_art = {}
try:
cursor.execute("""
SELECT artist_name, image_url FROM metadata_cache_entities
WHERE entity_type = 'album' AND image_url LIKE 'http%'
AND artist_name IS NOT NULL AND artist_name != ''
""")
for r in cursor.fetchall():
an = _norm(r['artist_name'])
if an and an not in _album_art:
_album_art[an] = r['image_url']
except Exception as e:
logger.debug("artist map album-art cache build failed: %s", e)
for n in nodes:
if not n.get('image_url') or not n['image_url'].startswith('http'):
nn = _norm(n['name'])
if nn in _album_art:
n['image_url'] = _album_art[nn]
except Exception as cache_err:
logger.debug(f"Artist map cache backfill error: {cache_err}")
result = {
'success': True,
'nodes': nodes,
'edges': edges,
'watchlist_count': sum(1 for n in nodes if n['type'] == 'watchlist'),
'similar_count': sum(1 for n in nodes if n['type'] == 'similar'),
}
_artmap_cache_set(f'watchlist_{profile_id}', result)
return jsonify(result)
except Exception as e:
logger.error(f"Error getting artist map data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_genre_list():
"""Lightweight endpoint — just genre names + counts for the picker. No node data."""
try:
cached = _artmap_cache_get('genre_list')
if cached:
return jsonify(cached)
database = get_database()
conn = database._get_connection()
cursor = conn.cursor()
# Fast query: just count artists per genre from cache
genre_counts = {}
cursor.execute("""
SELECT genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND genres IS NOT NULL AND genres != '' AND genres != '[]'
""")
for r in cursor.fetchall():
try:
for g in json.loads(r['genres']):
if g and isinstance(g, str):
gl = g.lower().strip()
genre_counts[gl] = genre_counts.get(gl, 0) + 1
except Exception as e:
logger.debug("genre count row parse failed: %s", e)
# Sort by count descending
sorted_genres = sorted(genre_counts.items(), key=lambda x: -x[1])
result = {
'success': True,
'genres': [{'name': g, 'count': c} for g, c in sorted_genres],
'total': len(sorted_genres)
}
_artmap_cache_set('genre_list', result)
return jsonify(result)
except Exception as e:
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_genres():
"""Get ALL artists from every data source, grouped by genre for the genre map."""
try:
database = get_database()
profile_id = get_current_profile_id()
cached = _artmap_cache_get(f'genres_{profile_id}')
if cached:
return jsonify(cached)
conn = database._get_connection()
cursor = conn.cursor()
artists_by_name = {} # normalized_name → {name, image, genres[], sources, ids}
def _norm(n):
return (n or '').lower().strip()
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, musicbrainz_id=None, source='unknown', popularity=0):
n = _norm(name)
if not n or len(n) < 2:
return
if n not in artists_by_name:
artists_by_name[n] = {
'name': name, 'image_url': '', 'genres': set(),
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': '',
'sources': set(), 'popularity': 0
}
a = artists_by_name[n]
if image_url and image_url.startswith('http') and not a['image_url']:
a['image_url'] = image_url
if genres:
for g in (genres if isinstance(genres, list) else []):
if g and isinstance(g, str):
a['genres'].add(g.lower().strip())
if spotify_id and not a['spotify_id']:
a['spotify_id'] = str(spotify_id)
if itunes_id and not a['itunes_id']:
a['itunes_id'] = str(itunes_id)
if deezer_id and not a['deezer_id']:
a['deezer_id'] = str(deezer_id)
if discogs_id and not a['discogs_id']:
a['discogs_id'] = str(discogs_id)
if musicbrainz_id and not a['musicbrainz_id']:
a['musicbrainz_id'] = str(musicbrainz_id)
if popularity > a['popularity']:
a['popularity'] = popularity
a['sources'].add(source)
# 1. Metadata cache — biggest source
cursor.execute("""
SELECT name, entity_id, source, image_url, genres, popularity
FROM metadata_cache_entities WHERE entity_type = 'artist'
""")
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("cache artist genres parse failed: %s", e)
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
kwargs = {src_map.get(r['source'], 'spotify_id'): r['entity_id']}
_add(r['name'], image_url=r['image_url'], genres=genres, source='cache', popularity=r['popularity'] or 0, **kwargs)
# 2. Similar artists
cursor.execute("""
SELECT similar_artist_name, similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id, image_url, genres, popularity
FROM similar_artists WHERE profile_id = ?
""", (profile_id,))
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("similar artist genres parse failed: %s", e)
_add(r['similar_artist_name'], image_url=r['image_url'], genres=genres,
spotify_id=r['similar_artist_spotify_id'], itunes_id=r['similar_artist_itunes_id'],
deezer_id=r['similar_artist_deezer_id'],
musicbrainz_id=r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else None,
source='similar', popularity=r['popularity'] or 0)
# 3. Watchlist artists
cursor.execute("""
SELECT artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
discogs_artist_id, image_url
FROM watchlist_artists WHERE profile_id = ?
""", (profile_id,))
for r in cursor.fetchall():
_add(r['artist_name'], image_url=r['image_url'],
spotify_id=r['spotify_artist_id'], itunes_id=r['itunes_artist_id'],
deezer_id=r['deezer_artist_id'], discogs_id=r['discogs_artist_id'], source='watchlist')
# 4. Library artists
cursor.execute("SELECT name, thumb_url, genres FROM artists")
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("library artist genres parse failed: %s", e)
img = r['thumb_url'] if r['thumb_url'] and r['thumb_url'].startswith('http') else None
_add(r['name'], image_url=img, genres=genres, source='library')
# Filter: only include artists that have at least one genre
genre_artists = {k: v for k, v in artists_by_name.items() if v['genres']}
# Build genre → artists map
genre_map = {} # genre_name → [artist_keys]
for key, a in genre_artists.items():
for g in a['genres']:
if g not in genre_map:
genre_map[g] = []
genre_map[g].append(key)
# Sort genres by artist count, take top genres
sorted_genres = sorted(genre_map.items(), key=lambda x: -len(x[1]))
# Build nodes
nodes = []
node_idx = {}
for key, a in genre_artists.items():
idx = len(nodes)
node_idx[key] = idx
nodes.append({
'id': idx,
'name': a['name'],
'image_url': a['image_url'],
'genres': list(a['genres'])[:5],
'spotify_id': a['spotify_id'],
'itunes_id': a['itunes_id'],
'deezer_id': a['deezer_id'],
'discogs_id': a['discogs_id'],
'musicbrainz_id': a['musicbrainz_id'],
'popularity': a['popularity'],
'type': 'watchlist' if 'watchlist' in a['sources'] else 'similar',
})
# Build genre clusters — allow artists in multiple genres
top_genres = sorted_genres[:40]
# Sort genres by co-occurrence so related genres are adjacent in the list.
# This makes the spiral layout place related genres near each other.
if len(top_genres) > 2:
genre_sets = {g: set(keys) for g, keys in top_genres}
ordered = [top_genres[0][0]] # Start with biggest genre
remaining = {g for g, _ in top_genres[1:]}
while remaining:
last = ordered[-1]
last_set = genre_sets.get(last, set())
# Find most similar remaining genre (highest artist overlap)
best = None
best_overlap = -1
for g in remaining:
overlap = len(last_set & genre_sets.get(g, set()))
if overlap > best_overlap:
best_overlap = overlap
best = g
ordered.append(best)
remaining.remove(best)
# Rebuild top_genres in the ordered sequence
genre_dict = dict(top_genres)
top_genres = [(g, genre_dict[g]) for g in ordered if g in genre_dict]
genres_out = []
for genre, artist_keys in top_genres:
genres_out.append({
'name': genre,
'count': len(artist_keys),
'artist_ids': [node_idx[k] for k in artist_keys if k in node_idx],
})
# Image cleanup + multi-source fallback
# Build two lookups: name→image_url AND name→deezer_entity_id
_img_cache = {}
_deezer_id_cache = {}
_album_art_cache = {} # artist_name → album image (iTunes fallback)
try:
# Artist images + Deezer IDs
cursor.execute("""
SELECT name, entity_id, source, image_url FROM metadata_cache_entities
WHERE entity_type = 'artist'
AND ((image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%')
OR source = 'deezer')
""")
for r in cursor.fetchall():
nn = (r['name'] or '').lower().strip()
if not nn:
continue
if r['image_url'] and r['image_url'].startswith('http') and nn not in _img_cache:
_img_cache[nn] = r['image_url']
if r['source'] == 'deezer' and r['entity_id'] and nn not in _deezer_id_cache:
_deezer_id_cache[nn] = r['entity_id']
# Album art by artist name (for iTunes artists with no artist image)
cursor.execute("""
SELECT artist_name, image_url FROM metadata_cache_entities
WHERE entity_type = 'album'
AND image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%'
AND artist_name IS NOT NULL AND artist_name != ''
""")
for r in cursor.fetchall():
nn = (r['artist_name'] or '').lower().strip()
if nn and nn not in _album_art_cache:
_album_art_cache[nn] = r['image_url']
except Exception as e:
logger.debug("genre map cache build failed: %s", e)
for n in nodes:
img = n.get('image_url', '')
if img in ('None', 'null', '') or (img and not img.startswith('http')):
n['image_url'] = ''
nn = n['name'].lower().strip()
if not n['image_url']:
# Try cache image by name
n['image_url'] = _img_cache.get(nn, '')
if not n['image_url'] and n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
if not n['image_url']:
# Try Deezer ID from cache by name
did = _deezer_id_cache.get(nn)
if did:
n['deezer_id'] = did
n['image_url'] = f"https://api.deezer.com/artist/{did}/image?size=big"
if not n['image_url']:
# Try album art by artist name (iTunes artists have no artist images)
n['image_url'] = _album_art_cache.get(nn, '')
_img_count = sum(1 for n in nodes if n.get('image_url'))
_deezer_count = sum(1 for n in nodes if n.get('image_url', '').startswith('https://api.deezer'))
_none_count = sum(1 for n in nodes if not n.get('image_url'))
logger.info(f"[Genre Map] {len(nodes)} artists, {len(sorted_genres)} genres")
logger.warning(f"[Genre Map] Images: {_img_count} have URLs, {_deezer_count} Deezer fallback, {_none_count} missing")
if _none_count > 0:
samples = [n['name'] for n in nodes if not n.get('image_url')][:5]
logger.warning(f"[Genre Map] Missing image samples: {samples}")
result = {
'success': True,
'nodes': nodes,
'genres': genres_out,
'total_artists': len(nodes),
'total_genres': len(sorted_genres),
}
_artmap_cache_set(f'genres_{profile_id}', result)
return jsonify(result)
except Exception as e:
logger.error(f"Error getting genre map data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_explore():
"""Build an exploration map outward from a single artist."""
try:
artist_name = request.args.get('name', '').strip()
artist_id = request.args.get('id', '').strip()
if not artist_name and not artist_id:
return jsonify({"success": False, "error": "Provide artist name or id"}), 400
database = get_database()
profile_id = get_current_profile_id()
conn = database._get_connection()
cursor = conn.cursor()
def _norm(n):
return (n or '').lower().strip()
nodes = []
edges = []
seen = {} # norm_name → node index
# Find the center artist
center_name = artist_name
center_image = ''
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': ''}
center_genres = []
# Search metadata cache for the center artist
if artist_id:
cursor.execute("""
SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND entity_id = ? LIMIT 1
""", (artist_id,))
else:
cursor.execute("""
SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE LIMIT 1
""", (artist_name,))
row = cursor.fetchone()
artist_found = False
if row:
artist_found = True
center_name = row['name']
if row['image_url'] and row['image_url'].startswith('http'):
center_image = row['image_url']
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(row['source'], 'spotify_id')
center_ids[k] = row['entity_id']
if row['genres']:
try:
center_genres = json.loads(row['genres']) if isinstance(row['genres'], str) else []
except Exception as e:
logger.debug("initial center genres parse failed: %s", e)
# Check watchlist + library if not in cache
if not artist_found and not artist_id:
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id, musicbrainz_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
wr = cursor.fetchone()
if wr:
artist_found = True
center_name = wr['artist_name']
if wr['image_url'] and str(wr['image_url']).startswith('http'):
center_image = wr['image_url']
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id'), ('musicbrainz_id', 'musicbrainz_artist_id')]:
if wr[col]:
center_ids[k] = str(wr[col])
else:
cursor.execute("SELECT name, thumb_url FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
lr = cursor.fetchone()
if lr:
artist_found = True
center_name = lr['name']
if lr['thumb_url'] and str(lr['thumb_url']).startswith('http'):
center_image = lr['thumb_url']
# If not found locally, validate via metadata API search
if not artist_found and not artist_id:
try:
api_match = None
if spotify_client and spotify_client.is_spotify_authenticated():
results = spotify_client.search_artists(artist_name, limit=1)
if results and len(results) > 0:
sa = results[0]
if sa.name.lower().strip() == artist_name.lower().strip() or \
artist_name.lower().strip() in sa.name.lower().strip():
api_match = sa
center_name = sa.name
center_ids['spotify_id'] = sa.id
center_image = sa.image_url if hasattr(sa, 'image_url') else ''
center_genres = sa.genres if hasattr(sa, 'genres') else []
artist_found = True
if not artist_found:
ic = _get_itunes_client()
results = ic.search_artists(artist_name, limit=1)
if results and len(results) > 0:
ia = results[0]
if ia.name.lower().strip() == artist_name.lower().strip() or \
artist_name.lower().strip() in ia.name.lower().strip():
center_name = ia.name
center_ids['itunes_id'] = str(ia.id)
center_image = ia.image_url if hasattr(ia, 'image_url') else ''
artist_found = True
except Exception as e:
logger.debug(f"[Artist Explorer] API validation failed for '{artist_name}': {e}")
if not artist_found:
return jsonify({"success": False, "error": f"Artist '{artist_name}' not found"}), 404
# Also check cache for other source IDs
cursor.execute("""
SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
""", (center_name,))
for r in cursor.fetchall():
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(r['source'], 'spotify_id')
if not center_ids.get(k):
center_ids[k] = r['entity_id']
if r['image_url'] and r['image_url'].startswith('http') and not center_image:
center_image = r['image_url']
if r['genres'] and not center_genres:
try:
center_genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("center genres parse failed: %s", e)
# Add center node
center_idx = 0
seen[_norm(center_name)] = center_idx
nodes.append({
'id': 0, 'name': center_name, 'image_url': center_image,
'type': 'center', 'genres': center_genres[:5],
**center_ids, 'ring': 0
})
# Ring 1: Direct similar artists from similar_artists table
# Search by all known IDs
id_values = [v for v in center_ids.values() if v]
ring1_artists = []
if id_values:
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", id_values + [profile_id])
ring1_artists = cursor.fetchall()
# Also search by name (the center artist might be a watchlist source)
cursor.execute("""
SELECT DISTINCT sa.similar_artist_name, sa.similar_artist_spotify_id,
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id, sa.similar_artist_musicbrainz_id,
sa.image_url, sa.genres, sa.popularity, sa.similarity_rank
FROM similar_artists sa
JOIN watchlist_artists wa ON sa.source_artist_id = COALESCE(wa.spotify_artist_id, wa.itunes_artist_id, CAST(wa.id AS TEXT))
WHERE wa.artist_name = ? COLLATE NOCASE AND sa.profile_id = ?
ORDER BY sa.similarity_rank ASC
""", (center_name, profile_id))
ring1_artists.extend(cursor.fetchall())
# If no similar artists in DB, fetch from MusicMap on-the-fly
if not ring1_artists:
try:
logger.debug(f"[Artist Explorer] No stored similar artists for '{center_name}', fetching from MusicMap...")
from core.watchlist_scanner import WatchlistScanner
scanner = WatchlistScanner(spotify_client=spotify_client) if spotify_client else None
if scanner:
similar = scanner._fetch_similar_artists_from_musicmap(center_name, limit=15)
if similar:
source_artist_id = center_ids.get('spotify_id') or center_ids.get('itunes_id') or center_name
# Store in DB for future use
for rank, sa in enumerate(similar, 1):
try:
database.add_or_update_similar_artist(
source_artist_id=source_artist_id,
similar_artist_name=sa['name'],
similar_artist_spotify_id=sa.get('spotify_id'),
similar_artist_itunes_id=sa.get('itunes_id'),
similarity_rank=rank,
profile_id=profile_id,
image_url=sa.get('image_url'),
genres=sa.get('genres'),
popularity=sa.get('popularity', 0),
similar_artist_deezer_id=sa.get('deezer_id'),
similar_artist_musicbrainz_id=sa.get('musicbrainz_id'),
)
except Exception as e:
logger.debug("similar artist insert failed: %s", e)
# Re-query from DB to get consistent format
if id_values:
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", id_values + [profile_id])
ring1_artists = cursor.fetchall()
if not ring1_artists:
# Fallback: query by name-based source ID
cursor.execute("""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id = ? AND profile_id = ?
ORDER BY similarity_rank ASC
""", (source_artist_id, profile_id))
ring1_artists = cursor.fetchall()
logger.debug(f"[Artist Explorer] Fetched {len(ring1_artists)} similar artists from MusicMap for '{center_name}'")
_artmap_cache_invalidate(profile_id) # New similar artists added
except Exception as e:
logger.debug(f"[Artist Explorer] MusicMap fetch failed for '{center_name}': {e}")
# Deduplicate ring 1
for r in ring1_artists:
nn = _norm(r['similar_artist_name'])
if nn in seen:
continue
idx = len(nodes)
seen[nn] = idx
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("ring1 genres parse failed: %s", e)
img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
nodes.append({
'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
'type': 'ring1', 'genres': genres[:5],
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
'ring': 1,
})
weight = max(1, 11 - (r['similarity_rank'] or 5))
edges.append({'source': center_idx, 'target': idx, 'weight': weight})
# Ring 2: Similar artists of ring 1 artists (from similar_artists table)
ring1_ids = []
for n in nodes[1:]: # skip center
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid:
ring1_ids.append(sid)
if ring1_ids:
placeholders = ','.join(['?'] * len(ring1_ids))
cursor.execute(f"""
SELECT DISTINCT source_artist_id, similar_artist_name,
similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", ring1_ids + [profile_id])
for r in cursor.fetchall():
nn = _norm(r['similar_artist_name'])
if nn in seen:
# Create edge to existing node if not center
existing_idx = seen[nn]
# Find the ring1 node that sourced this
source_norm = None
for n in nodes[1:]:
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid == r['source_artist_id']:
source_norm = _norm(n['name'])
break
if source_norm:
break
if source_norm and source_norm in seen and existing_idx != seen[source_norm]:
edges.append({'source': seen[source_norm], 'target': existing_idx, 'weight': 3})
continue
idx = len(nodes)
if idx >= 500: # Cap at 500 nodes for performance
break
seen[nn] = idx
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("ring2 genres parse failed: %s", e)
img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
nodes.append({
'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
'type': 'ring2', 'genres': genres[:5],
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
'ring': 2,
})
# Find the ring1 source
for n in nodes[1:]:
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid == r['source_artist_id']:
edges.append({'source': n['id'], 'target': idx, 'weight': max(1, 11 - (r['similarity_rank'] or 5))})
break
# Backfill images/genres from ALL cache sources + Deezer fallback
for n in nodes:
# Clean up string "None" stored as image URL
if n['image_url'] in ('None', 'null', ''):
n['image_url'] = ''
if n['image_url'] and n['genres']:
continue
# Check all cache entries for this artist (multiple sources)
cursor.execute("""
SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
""", (n['name'],))
for cr in cursor.fetchall():
if not n['image_url'] and cr['image_url'] and cr['image_url'].startswith('http'):
n['image_url'] = cr['image_url']
if not n['genres'] and cr['genres']:
try:
n['genres'] = json.loads(cr['genres'])[:5] if isinstance(cr['genres'], str) else []
except Exception as e:
logger.debug("explorer node genres parse failed: %s", e)
# Harvest missing IDs from cache
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(cr['source'])
if k and not n.get(k):
n[k] = cr['entity_id']
# Deezer image fallback — construct URL directly from ID
if not n['image_url'] and n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
# Spotify image fallback — try API if authenticated
if not n['image_url'] and n.get('spotify_id'):
try:
if spotify_client and spotify_client.is_spotify_authenticated():
from core.api_call_tracker import api_call_tracker
api_call_tracker.record_call('spotify', endpoint='artist')
artist_data = spotify_client.sp.artist(n['spotify_id'])
if artist_data and artist_data.get('images'):
n['image_url'] = artist_data['images'][0]['url']
if not n['genres'] and artist_data.get('genres'):
n['genres'] = artist_data['genres'][:5]
except Exception as e:
logger.debug("spotify artist image fallback failed: %s", e)
# Album art fallback (iTunes artists have no artist images)
if not n['image_url']:
cursor.execute("""
SELECT image_url FROM metadata_cache_entities
WHERE entity_type = 'album' AND image_url LIKE 'http%'
AND artist_name = ? COLLATE NOCASE LIMIT 1
""", (n['name'],))
alb = cursor.fetchone()
if alb:
n['image_url'] = alb['image_url']
logger.info(f"[Artist Explorer] Center: {center_name}, Ring 1: {sum(1 for n in nodes if n.get('ring')==1)}, Ring 2: {sum(1 for n in nodes if n.get('ring')==2)}, Edges: {len(edges)}")
return jsonify({
'success': True,
'nodes': nodes,
'edges': edges,
'center': center_name,
})
except Exception as e:
logger.error(f"Error getting artist explorer data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500

496
core/artists/quality.py
View file

@ -1,496 +0,0 @@
"""Artist quality enhancement helper.
`enhance_artist_quality(artist_id, track_ids, deps)` is the route-handler
body for the `/api/library/artist/<artist_id>/enhance` endpoint. It walks
the user's selected tracks, finds the best metadata match against the
configured primary source, and queues high-quality re-downloads on the
wishlist with `source_type='enhance'`.
Per-track flow:
1. Resolve the existing track via the artist's full detail map (built up
front from `database.get_artist_full_detail`).
2. Read current quality tier from the file extension.
3. Build `matched_track_data` for the wishlist entry, in priority order:
- **Direct lookup using stored source IDs** — for every source the
user has configured, if the library track has the corresponding
stored ID (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
`soul_id`), call `client.get_track_details(stored_id)` and convert
the result to the wishlist payload. First success wins; the user's
configured primary source is tried first. Mirrors what Download
Discography does — stable IDs straight to the source's API, no
fuzzy text matching.
- **Multi-source parallel text search fallback** — if no stored ID
resolved, run the shared `core.metadata.multi_source_search`
against every configured source in parallel and pick the best
cross-source match (auto-accept threshold 0.7).
4. Validate the match has non-empty title, album, and artists. Reject
matches with empty fields — those propagated as
"unknown artist - unknown album - unknown track" wishlist entries
pre-fix because the wishlist payload normalizer's truthy-check
passthrough accepted dicts with empty string fields.
5. Add to wishlist via `wishlist_service.add_spotify_track_to_wishlist`
with `source_type='enhance'` and a `source_context` carrying the
original file path, format tier, bitrate, and artist name.
6. Tally `enhanced_count` / `failed_count` / per-track failure reasons.
The flow originally had Spotify-only logic with an iTunes search-only
fallback. Two failure modes drove the rewrite:
- Users with neither Spotify nor Deezer connected got silent failures
("unknown artist - unknown album - unknown track" wishlist entries)
because iTunes's text search returned junk matches with empty fields
that cleared the 0.7 confidence threshold.
- Library tracks with messy tags ("Title (Live)", featured artists in
the artist field, etc.) failed fuzzy text search even when a perfect
stored ID was available — Download Discography had no such problem
because it resolves albums by stable ID.
Direct-lookup-via-stored-ID matches the Download Discography contract
for every source where we have an ID column. Text search is only the
fallback now.
Returns `(payload_dict, http_status_code)` so the route wrapper can
`jsonify()` and return.
"""
from __future__ import annotations
import os
from dataclasses import dataclass
from typing import Any, Callable, Optional
from utils.logging_config import get_logger
logger = get_logger('artists.quality')
@dataclass
class ArtistQualityDeps:
"""Bundle of cross-cutting deps the artist quality enhancement needs."""
matching_engine: Any
get_database: Callable[[], Any]
get_wishlist_service: Callable[[], Any]
get_current_profile_id: Callable[[], int]
get_quality_tier_from_extension: Callable
# Returns ``[(source_name, client), ...]`` for every metadata source
# the user has configured. Powers both the direct-lookup fast path
# (resolves stored source IDs straight from each source's API,
# like Download Discography) and the multi-source parallel text
# search fallback (shared with Track Redownload via
# ``core.metadata.multi_source_search``).
get_metadata_search_sources: Callable[[], list]
def _has_complete_metadata(payload: Optional[dict]) -> bool:
"""Reject matches with empty / missing core fields. Pre-fix, iTunes
returned matches that cleared the 0.7 confidence threshold while
having empty artist / album / title — those propagated as junk
wishlist entries displayed as 'unknown artist - unknown album -
unknown track'."""
if not payload:
return False
if not (payload.get('name') or '').strip():
return False
artists = payload.get('artists') or []
has_artist = any(
(a.get('name') or '').strip() if isinstance(a, dict) else (a or '').strip()
for a in artists
)
if not has_artist:
return False
album = payload.get('album') or {}
if isinstance(album, dict):
if not (album.get('name') or '').strip():
return False
elif not (album or '').strip():
return False
return True
def _build_payload_from_track(track_obj) -> dict:
"""Build a Spotify-shaped wishlist payload from any metadata source's
Track-shaped object (Spotify Track, iTunes Track, Deezer Track,
Discogs Track — they all have the same .id / .name / .artists /
.album / .duration_ms / etc shape because each client mimics
Spotify's surface).
The wishlist's downstream pipeline expects Spotify shape; this helper
is the single place that knows how to produce it. Replaces the
duplicated payload construction that used to live in the Spotify
search path AND the iTunes fallback path.
Does NOT substitute defaults for missing artists / album / title —
``_has_complete_metadata`` rejects empty matches downstream so the
user sees a clear failure instead of a junk wishlist entry with
fabricated values.
"""
image_url = getattr(track_obj, 'image_url', '') or ''
album_images = (
[{'url': image_url, 'height': 600, 'width': 600}]
if image_url else []
)
artist_names = list(getattr(track_obj, 'artists', None) or [])
return {
'id': getattr(track_obj, 'id', ''),
'name': getattr(track_obj, 'name', '') or '',
'artists': [{'name': a} for a in artist_names],
'album': {
'name': getattr(track_obj, 'album', '') or '',
'artists': [{'name': a} for a in artist_names],
'album_type': getattr(track_obj, 'album_type', None) or 'album',
'images': album_images,
'release_date': getattr(track_obj, 'release_date', '') or '',
'total_tracks': 1,
},
'duration_ms': getattr(track_obj, 'duration_ms', 0) or 0,
'track_number': getattr(track_obj, 'track_number', None) or 1,
'disc_number': getattr(track_obj, 'disc_number', None) or 1,
'popularity': getattr(track_obj, 'popularity', None) or 0,
'preview_url': getattr(track_obj, 'preview_url', None),
'external_urls': getattr(track_obj, 'external_urls', None) or {},
}
# Map metadata source name → DB column on the ``tracks`` table that
# stores that source's native track ID. Used to drive the direct-lookup
# fast path: when a library track has a stored ID for source X and the
# user has source X configured, skip fuzzy text search and resolve
# straight from X's API. Mirrors what Download Discography does — stable
# IDs all the way, no fuzzy text matching.
#
# Discogs is release-based and has no per-track ID column; not listed
# here, so direct lookup never tries Discogs (search-fallback still
# runs for Discogs as one of the parallel sources).
_STORED_ID_COLUMNS = {
'spotify': 'spotify_track_id',
'deezer': 'deezer_id',
'itunes': 'itunes_track_id',
'hydrabase': 'soul_id',
}
def _enhanced_to_wishlist_payload(enhanced: dict,
fallback_title: str,
fallback_artist: str,
fallback_album: str) -> Optional[dict]:
"""Convert a ``get_track_details`` enhanced-shape dict to the
Spotify-shape wishlist payload.
Every metadata source's ``get_track_details`` returns the same
"enhanced" intermediate shape (top-level ``id``, ``name``,
``artists`` as a list of strings, ``album.artists`` as strings),
documented and pinned across spotify_client / itunes_client /
deezer_client / hydrabase_client. The wishlist downstream expects
Spotify's native shape (``artists`` as ``[{'name': ...}]``), so
this helper does the conversion in one place.
Spotify's ``raw_data`` field is already in wishlist shape (the
raw Spotify API response), so we return it as-is when detected,
preserving full ``album.images`` and ``external_urls`` that the
enhanced top-level fields drop. Other sources' ``raw_data`` is
in source-native shape and gets ignored.
"""
if not enhanced:
return None
raw = enhanced.get('raw_data')
if isinstance(raw, dict):
raw_artists = raw.get('artists')
if (isinstance(raw_artists, list) and raw_artists
and isinstance(raw_artists[0], dict)):
return raw
artists = enhanced.get('artists') or [fallback_artist]
album_data = enhanced.get('album') or {}
album_artists = album_data.get('artists') or artists
def _to_dict_artists(seq):
return [a if isinstance(a, dict) else {'name': a} for a in seq]
image_url = enhanced.get('image_url') or ''
album_images_field = album_data.get('images')
if isinstance(album_images_field, list) and album_images_field:
album_images = album_images_field
elif image_url:
album_images = [{'url': image_url, 'height': 600, 'width': 600}]
else:
album_images = []
return {
'id': str(enhanced.get('id', '')),
'name': enhanced.get('name') or fallback_title,
'artists': _to_dict_artists(artists),
'album': {
'id': str(album_data.get('id', '')),
'name': album_data.get('name') or fallback_album,
'album_type': album_data.get('album_type', 'album'),
'release_date': album_data.get('release_date', ''),
'total_tracks': album_data.get('total_tracks', 1),
'artists': _to_dict_artists(album_artists),
'images': album_images,
},
'duration_ms': enhanced.get('duration_ms', 0),
'track_number': enhanced.get('track_number', 1),
'disc_number': enhanced.get('disc_number', 1),
'popularity': enhanced.get('popularity', 0),
'preview_url': enhanced.get('preview_url'),
'external_urls': enhanced.get('external_urls', {}),
}
def _try_direct_lookup_all_sources(track: dict,
sources: list,
preferred_source: Optional[str],
title: str,
artist_name: str,
album_title: str
) -> tuple:
"""Try direct ID-based lookup on every source where the library
track has a stored ID. Returns ``(payload, source_name)`` on first
success, or ``(None, None)`` if no source has a stored ID with a
successful lookup.
Mirrors what Download Discography does — stable IDs straight to the
source's API, no fuzzy text matching. Avoids the failure mode where
library text tags don't match the source's canonical title (the
Discord report case: track tagged "Title (Live)" and source has
"Title" → fuzzy search misses, but stored ID resolves directly).
Preferred source attempted first when present in ``sources``,
typically the user's configured primary metadata source — so a
Deezer-primary user gets Deezer art / album shape on the wishlist
entry instead of whichever source happened to have a stored ID
first in iteration order.
"""
def _priority(entry):
name = entry[0]
return 0 if name == preferred_source else 1
ordered = sorted(sources, key=_priority)
for source_name, client in ordered:
column = _STORED_ID_COLUMNS.get(source_name)
if not column:
continue
stored_id = track.get(column)
if not stored_id:
continue
if not hasattr(client, 'get_track_details'):
continue
try:
enhanced = client.get_track_details(str(stored_id))
except Exception as exc:
logger.error(
f"[Enhance] {source_name} direct lookup failed for "
f"ID {stored_id}: {exc}"
)
continue
if not enhanced:
continue
payload = _enhanced_to_wishlist_payload(
enhanced, title, artist_name, album_title,
)
if _has_complete_metadata(payload):
logger.info(
f"[Enhance] Direct lookup matched: {source_name} "
f"ID {stored_id} → '{payload.get('name')}'"
)
return payload, source_name
return None, None
# Minimum match-score threshold for accepting a search-fallback match
# without user confirmation. Mirrors the legacy threshold the enhance
# flow has always used.
_AUTO_ACCEPT_SCORE_THRESHOLD = 0.7
def enhance_artist_quality(artist_id, track_ids, deps: ArtistQualityDeps):
"""Add selected tracks to wishlist for quality enhancement re-download.
Per-track flow:
1. **Direct lookup using stored source IDs** (mirrors what Download
Discography does — stable IDs straight to the source's API, no
fuzzy text matching). For each source the user has configured,
if the library track has the corresponding stored ID
(``spotify_track_id`` / ``deezer_id`` / ``itunes_track_id`` /
``soul_id``), call ``client.get_track_details(stored_id)`` and
convert to wishlist payload. First success wins; preferred
source (user's configured primary) tried first.
2. **Multi-source parallel text search fallback** (via the shared
``core.metadata.multi_source_search`` module — same code path
Track Redownload uses) for tracks with no stored IDs / lookup
misses.
3. **Validation**: reject matches with empty title / album / artists
so the user sees a clear failure instead of an "unknown artist"
wishlist entry.
Pre-refactor: only Spotify had a direct-lookup fast path; everything
else went through fuzzy text search. Discogs / Hydrabase / Deezer-
primary users got far worse coverage than Download Discography
despite both flows asking the same question.
"""
from core.metadata.multi_source_search import TrackQuery, search_all_sources
from core.metadata.registry import get_primary_source
try:
if not track_ids:
return {"success": False, "error": "No track IDs provided"}, 400
database = deps.get_database()
wishlist_service = deps.get_wishlist_service()
profile_id = deps.get_current_profile_id()
# Get artist info
artist_result = database.get_artist_full_detail(artist_id)
if not artist_result.get('success'):
return {"success": False, "error": "Artist not found"}, 404
artist_name = artist_result.get('artist', {}).get('name', 'Unknown Artist')
# Build lookup of all tracks for this artist
track_lookup = {}
for album in artist_result.get('albums', []):
album_title = album.get('title', '')
for track in album.get('tracks', []):
tid = str(track.get('id', ''))
track['_album_title'] = album_title
track['_album_id'] = album.get('id')
track_lookup[tid] = track
# Resolve every configured metadata source up front.
search_sources = deps.get_metadata_search_sources()
# User's configured primary source — direct-lookup tries this
# first so Deezer-primary users get Deezer payloads on the
# wishlist entry (correct cover art / album shape) even when
# other sources also have stored IDs for the same track.
try:
preferred_source = get_primary_source()
except Exception:
preferred_source = None
enhanced_count = 0
failed_count = 0
failed_tracks = []
for track_id in track_ids:
track_id_str = str(track_id)
track = track_lookup.get(track_id_str)
if not track:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'reason': 'Track not found'})
continue
file_path = track.get('file_path')
if not file_path:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'reason': 'No file path'})
continue
tier_name, tier_num = deps.get_quality_tier_from_extension(file_path)
title = track.get('title', '') or ''
if not title.strip():
title = os.path.splitext(os.path.basename(file_path))[0]
album_title = track.get('_album_title', '')
matched_track_data = None
chosen_source = None
# 1. Direct lookup via every stored source ID — like Download
# Discography. Stable IDs, no fuzzy text matching.
if search_sources:
matched_track_data, chosen_source = _try_direct_lookup_all_sources(
track, search_sources, preferred_source,
title, artist_name, album_title,
)
# 2. Multi-source parallel text search fallback — for tracks
# with no stored IDs / lookup misses.
if not matched_track_data and search_sources:
try:
track_query = TrackQuery(
title=title,
artist=artist_name,
album=album_title,
duration_ms=track.get('duration', 0) or 0,
spotify_track_id=track.get('spotify_track_id'),
deezer_id=track.get('deezer_id'),
)
multi_result = search_all_sources(track_query, search_sources)
if multi_result.best_match and multi_result.best_match['score'] >= _AUTO_ACCEPT_SCORE_THRESHOLD:
chosen_source = multi_result.best_match['source']
best_track_obj = multi_result.best_track()
if best_track_obj:
matched_track_data = _build_payload_from_track(best_track_obj)
except Exception as exc:
logger.error(f"[Enhance] Multi-source search failed for {title}: {exc}")
# 3. Reject matches with empty / missing core fields.
if not _has_complete_metadata(matched_track_data):
if matched_track_data:
logger.warning(
f"[Enhance] {chosen_source} match for '{title}' rejected — "
f"empty title / album / artists (would render as 'unknown')"
)
matched_track_data = None
if not matched_track_data:
failed_count += 1
source_list = ', '.join(name for name, _ in (search_sources or []))
if not source_list:
reason = (
'No metadata source configured — connect Spotify / '
'iTunes / Deezer / Discogs / Hydrabase to enable enhance'
)
else:
reason = (
f'No usable match across {source_list} — '
f'try connecting an additional metadata source'
)
failed_tracks.append({
'track_id': track_id,
'title': title,
'reason': reason,
})
continue
# Add to wishlist with enhance source
source_context = {
'enhance': True,
'original_file_path': file_path,
'original_format': tier_name,
'original_bitrate': track.get('bitrate'),
'original_tier': tier_num,
'artist_name': artist_name,
}
success = wishlist_service.add_spotify_track_to_wishlist(
spotify_track_data=matched_track_data,
failure_reason=f"Quality enhance - upgrading from {tier_name.replace('_', ' ').title()}",
source_type='enhance',
source_context=source_context,
profile_id=profile_id
)
if success:
enhanced_count += 1
logger.info(f"[Enhance] Queued for upgrade: {artist_name} - {title} ({tier_name})")
else:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'title': title, 'reason': 'Wishlist add failed'})
return {
'success': True,
'enhanced_count': enhanced_count,
'failed_count': failed_count,
'failed_tracks': failed_tracks
}, 200
except Exception as e:
logger.error(f"[Enhance] {e}")
import traceback
traceback.print_exc()
return {"success": False, "error": str(e)}, 500

108
core/audiodb_worker.py
View file

@ -8,7 +8,7 @@ from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.audiodb_client import AudioDBClient
from core.worker_utils import accept_artist_match, interruptible_sleep
from core.worker_utils import interruptible_sleep
logger = get_logger("audiodb_worker")
@ -140,8 +140,8 @@ class AudioDBWorker:
itype = item.get('type', '')
table = 'artists' if 'artist' in itype else ('albums' if 'album' in itype else 'tracks')
# Can't mark status without an ID — just skip
except Exception as e:
logger.debug("null id table resolve failed: %s", e)
except Exception:
pass
continue
@ -162,16 +162,6 @@ class AudioDBWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('audiodb')
if _prio:
_pi = priority_pending_item(cursor, 'audiodb', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name
@ -210,45 +200,42 @@ class AudioDBWorker:
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
# Priority 4: Retry 'not_found' OR 'error' artists after retry_days.
# 'error' status covers transient AudioDB outages (timeouts, 500s)
# that the issue-#553 fix marks rather than leaving NULL — without
# this retry path those rows would stay errored forever.
retry_cutoff = datetime.now() - timedelta(days=self.retry_days)
# Priority 4: Retry 'not_found' artists after retry_days
not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
cursor.execute("""
SELECT id, name
FROM artists
WHERE audiodb_match_status IN ('not_found', 'error') AND audiodb_last_attempted < ?
WHERE audiodb_match_status = 'not_found' AND audiodb_last_attempted < ?
ORDER BY audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 5: Retry 'not_found' OR 'error' albums
# Priority 5: Retry 'not_found' albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.audiodb_match_status IN ('not_found', 'error') AND a.audiodb_last_attempted < ?
WHERE a.audiodb_match_status = 'not_found' AND a.audiodb_last_attempted < ?
ORDER BY a.audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
# Priority 6: Retry 'not_found' OR 'error' tracks
# Priority 6: Retry 'not_found' tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.audiodb_match_status IN ('not_found', 'error') AND t.audiodb_last_attempted < ?
WHERE t.audiodb_match_status = 'not_found' AND t.audiodb_last_attempted < ?
ORDER BY t.audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
@ -273,13 +260,8 @@ class AudioDBWorker:
def _verify_artist_id(self, item: Dict[str, Any], result: Dict[str, Any]) -> bool:
"""Verify that the result's artist ID matches the parent artist's stored AudioDB ID.
If mismatched, the album/track search is more specific (uses artist+title),
so we trust it and correct the parent artist's audiodb_id — BUT only when
the result's artist *name* matches our parent artist. Without that guard,
a collaboration/compilation (a track our library credits to one artist
that lives on another artist's album) would stamp the wrong AudioDB id
onto our artist. See the Deezer fix for the full write-up."""
so we trust it and correct the parent artist's audiodb_id."""
parent_audiodb_id = item.get('artist_audiodb_id')
if not parent_audiodb_id:
return True
@ -289,18 +271,6 @@ class AudioDBWorker:
return True
if str(result_artist_id) != str(parent_audiodb_id):
parent_name = item.get('artist') or ''
result_artist_name = result.get('strArtist') or ''
if (result_artist_name and parent_name
and not self._name_matches(parent_name, result_artist_name)):
logger.info(
f"Skipping artist-ID correction from {item['type']} "
f"'{item['name']}': result artist '{result_artist_name}' "
f"≠ parent '{parent_name}' (collab/compilation, not a "
f"correction)"
)
return True
logger.info(
f"Artist ID correction from {item['type']} '{item['name']}': "
f"updating parent artist AudioDB ID from {parent_audiodb_id} to {result_artist_id}"
@ -404,40 +374,22 @@ class AudioDBWorker:
return
except Exception as e:
logger.warning(f"Direct lookup failed for existing AudioDB ID {existing_id}: {e}")
# Direct lookup returned no metadata (None) or raised — don't
# fall through to the name-search path below, which could
# overwrite a manually-matched audiodb_id with a wrong guess.
# Mark status='error' so the queue's NULL-status filter stops
# re-picking this row on every tick (issue #553: AudioDB
# `track.php` timeouts caused infinite enrichment loops as
# the row was repeatedly picked + re-attempted because it
# never left the NULL state). The error-retry priority block
# in `_get_next_item` re-attempts after `retry_days` so
# transient AudioDB outages still recover automatically.
self._mark_status(item_type, item_id, 'error')
self.stats['errors'] += 1
logger.debug(
f"Preserving manual match for {item_type} '{item_name}' "
f"(AudioDB ID: {existing_id}); marked error pending retry"
)
# Direct lookup failed — don't overwrite manual match
logger.debug(f"Preserving manual match for {item_type} '{item_name}' (AudioDB ID: {existing_id})")
return
if item_type == 'artist':
result = self.client.search_artist(item_name)
if result:
result_name = result.get('strArtist', '')
ok, reason = accept_artist_match(
self.db, 'audiodb_id', result.get('idArtist'), item_id,
item_name, result_name,
)
if ok:
if self._name_matches(item_name, result_name):
self._update_artist(item_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{item_name}' -> AudioDB ID: {result.get('idArtist')}")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Artist '{item_name}' not matched: {reason}")
logger.debug(f"Name mismatch for artist '{item_name}' (got '{result_name}')")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1
@ -528,14 +480,10 @@ class AudioDBWorker:
# Backfill genres if artist has none
genre = data.get('strGenre')
if genre:
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
_filtered = filter_genres([genre], _cfg)
if _filtered:
cursor.execute("""
UPDATE artists SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(_filtered), artist_id))
cursor.execute("""
UPDATE artists SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps([genre]), artist_id))
conn.commit()
@ -580,14 +528,10 @@ class AudioDBWorker:
# Backfill genres if album has none
genre = data.get('strGenre')
if genre:
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
_filtered = filter_genres([genre], _cfg)
if _filtered:
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(_filtered), album_id))
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps([genre]), album_id))
conn.commit()

2086
core/auto_import_worker.py
View file

File diff suppressed because it is too large Load diff

11
core/automation/__init__.py
View file

@ -1,11 +0,0 @@
"""Automation API + progress + handlers package.
Lifted from web_server.py:
- `/api/automations/*` route helpers → `api.py`
- block library used by the trigger/action UI → `blocks.py`
- progress tracker (init / update / finish) → `progress.py`
- cross-handler signal bus → `signals.py`
- per-action handler functions → `handlers/` subpackage (with
`deps.py` defining the dependency-injection surface so handlers
stay testable in isolation)
"""

377
core/automation/api.py
View file

@ -1,377 +0,0 @@
"""Automation REST API helpers.
CRUD + run + progress + history logic for /api/automations/* routes.
Each function takes the deps it needs (database, automation_engine,
profile_id) so the route layer is left as pure HTTP shuffling.
Out of scope:
- /api/automations/blocks — static JSON + one call into signals.py;
stays inline in web_server.py for now.
- /api/test/automation — touches scan manager + media clients +
config_manager; stays inline.
"""
from __future__ import annotations
import json
import logging
from typing import Any, Optional
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Hydration helpers — convert raw DB rows to API-friendly dicts
# ---------------------------------------------------------------------------
_JSON_FIELDS = ('trigger_config', 'action_config', 'notify_config', 'last_result')
_JSON_DEFAULT_DICT = {'trigger_config', 'action_config', 'notify_config'}
def _hydrate_automation(auto: dict) -> dict:
"""Parse JSON columns and backfill `then_actions` from legacy notify_*."""
for field in _JSON_FIELDS:
try:
auto[field] = json.loads(auto[field]) if isinstance(auto[field], str) else auto[field]
except (json.JSONDecodeError, TypeError):
auto[field] = {} if field in _JSON_DEFAULT_DICT else None
try:
raw = auto.get('then_actions')
auto['then_actions'] = json.loads(raw or '[]') if isinstance(raw, str) else (raw or [])
except (json.JSONDecodeError, TypeError):
auto['then_actions'] = []
if not auto['then_actions'] and auto.get('notify_type'):
auto['then_actions'] = [{
'type': auto['notify_type'],
'config': auto.get('notify_config', {}),
}]
return auto
# ---------------------------------------------------------------------------
# Signal cycle detection
# ---------------------------------------------------------------------------
def _has_signal_concern(trigger_type: str, then_actions: list[dict]) -> bool:
return trigger_type == 'signal_received' or any(
t.get('type') == 'fire_signal' for t in then_actions
)
def _check_create_cycle(
automation_engine,
database,
profile_id: int,
trigger_type: str,
trigger_config_json: str,
then_actions_json: str,
then_actions: list[dict],
) -> Optional[str]:
"""Return cycle path string if creating this automation would loop, else None."""
if not automation_engine or not _has_signal_concern(trigger_type, then_actions):
return None
all_autos = database.get_automations(profile_id)
test_auto = {
'trigger_type': trigger_type,
'trigger_config': trigger_config_json,
'then_actions': then_actions_json,
'enabled': True,
}
all_autos.append(test_auto)
cycle = automation_engine.detect_signal_cycles(all_autos)
if cycle:
return ' → '.join(cycle)
return None
def _check_update_cycle(
automation_engine,
database,
automation_id: int,
data: dict,
) -> Optional[str]:
"""Return cycle path string if updating this automation would loop, else None."""
if not automation_engine:
return None
trigger_type = data.get('trigger_type', '')
then_actions = data.get('then_actions', [])
if not _has_signal_concern(trigger_type, then_actions):
return None
all_autos = database.get_automations()
test_autos = []
for a in all_autos:
if a['id'] == automation_id:
merged = dict(a)
if 'trigger_type' in data:
merged['trigger_type'] = data['trigger_type']
if 'trigger_config' in data:
merged['trigger_config'] = json.dumps(data['trigger_config'])
if 'then_actions' in data:
merged['then_actions'] = json.dumps(data['then_actions'])
merged['enabled'] = True
test_autos.append(merged)
else:
test_autos.append(a)
cycle = automation_engine.detect_signal_cycles(test_autos)
if cycle:
return ' → '.join(cycle)
return None
# ---------------------------------------------------------------------------
# CRUD helpers — return (response_dict, http_status)
# ---------------------------------------------------------------------------
def list_automations(database, profile_id: int) -> list[dict]:
"""All automations for the profile, with JSON columns parsed."""
automations = database.get_automations(profile_id)
return [_hydrate_automation(a) for a in automations]
def get_automation(database, automation_id: int) -> Optional[dict]:
"""One automation, hydrated. Returns None if not found."""
auto = database.get_automation(automation_id)
if not auto:
return None
return _hydrate_automation(auto)
def create_automation(
database,
automation_engine,
profile_id: int,
data: dict,
) -> tuple[dict, int]:
"""Create + schedule an automation. Returns (response_body, http_status)."""
name = (data.get('name') or '').strip()
if not name:
return {'error': 'Name is required'}, 400
trigger_type = data.get('trigger_type', 'schedule')
trigger_config = json.dumps(data.get('trigger_config', {}))
action_type = data.get('action_type', 'process_wishlist')
action_config = json.dumps(data.get('action_config', {}))
then_actions = data.get('then_actions', [])
then_actions_json = json.dumps(then_actions)
if then_actions:
notify_type = then_actions[0].get('type')
notify_config = json.dumps(then_actions[0].get('config', {}))
else:
notify_type = data.get('notify_type') or None
notify_config = json.dumps(data.get('notify_config', {})) if notify_type else '{}'
cycle_path = _check_create_cycle(
automation_engine, database, profile_id,
trigger_type, trigger_config, then_actions_json, then_actions,
)
if cycle_path:
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
group_name = data.get('group_name') or None
owned_by = data.get('owned_by') or None
auto_id = database.create_automation(
name, trigger_type, trigger_config, action_type, action_config,
profile_id, notify_type, notify_config, then_actions_json, group_name,
owned_by=owned_by,
)
if auto_id is None:
return {'error': 'Failed to create automation'}, 500
if automation_engine:
automation_engine.schedule_automation(auto_id)
return {'success': True, 'id': auto_id}, 200
def update_automation(
database,
automation_engine,
automation_id: int,
data: dict,
) -> tuple[dict, int]:
"""Update + reschedule an automation. Returns (response_body, http_status)."""
update_fields: dict[str, Any] = {}
if 'name' in data:
update_fields['name'] = data['name'].strip()
if 'trigger_type' in data:
update_fields['trigger_type'] = data['trigger_type']
if 'trigger_config' in data:
update_fields['trigger_config'] = json.dumps(data['trigger_config'])
if 'action_type' in data:
update_fields['action_type'] = data['action_type']
if 'action_config' in data:
update_fields['action_config'] = json.dumps(data['action_config'])
if 'then_actions' in data:
then_actions = data['then_actions']
update_fields['then_actions'] = json.dumps(then_actions)
if then_actions:
update_fields['notify_type'] = then_actions[0].get('type')
update_fields['notify_config'] = json.dumps(then_actions[0].get('config', {}))
else:
update_fields['notify_type'] = None
update_fields['notify_config'] = '{}'
elif 'notify_type' in data:
update_fields['notify_type'] = data['notify_type'] or None
if 'notify_config' in data and 'then_actions' not in data:
update_fields['notify_config'] = json.dumps(data['notify_config'])
if 'group_name' in data:
update_fields['group_name'] = data['group_name'] or None
if 'owned_by' in data:
update_fields['owned_by'] = data['owned_by'] or None
if not update_fields:
return {'error': 'No fields to update'}, 400
cycle_path = _check_update_cycle(automation_engine, database, automation_id, data)
if cycle_path:
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
# Schedule-shape changes must invalidate the stored next_run so the
# scheduler recomputes it; otherwise restart-survival logic keeps the
# leftover timestamp from the previous interval.
if {'trigger_type', 'trigger_config'} & update_fields.keys():
update_fields['next_run'] = None
success = database.update_automation(automation_id, **update_fields)
if not success:
return {'error': 'Automation not found'}, 404
if automation_engine:
auto = database.get_automation(automation_id)
if auto and auto.get('enabled'):
automation_engine.schedule_automation(automation_id)
else:
automation_engine.cancel_automation(automation_id)
return {'success': True}, 200
def batch_update_group(database, automation_ids: list, group_name: Optional[str]) -> tuple[dict, int]:
"""Move/rename a set of automations into a single group (or ungroup)."""
if not automation_ids or not isinstance(automation_ids, list):
return {'error': 'automation_ids must be a non-empty list'}, 400
try:
automation_ids = [int(aid) for aid in automation_ids]
except (ValueError, TypeError):
return {'error': 'automation_ids must contain integers'}, 400
updated = database.batch_update_group(automation_ids, group_name)
return {'success': True, 'updated': updated}, 200
def bulk_toggle(
database,
automation_engine,
automation_ids: list,
enabled: bool,
) -> tuple[dict, int]:
"""Bulk enable/disable a set of automations + reschedule each affected."""
if not automation_ids or not isinstance(automation_ids, list):
return {'error': 'automation_ids must be a non-empty list'}, 400
try:
automation_ids = [int(aid) for aid in automation_ids]
except (ValueError, TypeError):
return {'error': 'automation_ids must contain integers'}, 400
updated = database.bulk_set_enabled(automation_ids, bool(enabled))
if automation_engine and updated > 0:
for aid in automation_ids:
auto = database.get_automation(aid)
if auto:
if auto.get('enabled'):
automation_engine.schedule_automation(auto)
else:
automation_engine.cancel_automation(aid)
return {'success': True, 'updated': updated}, 200
def delete_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
"""Delete an automation. System automations are protected."""
auto = database.get_automation(automation_id)
if auto and auto.get('is_system'):
return {'error': 'System automations cannot be deleted'}, 403
if automation_engine:
automation_engine.cancel_automation(automation_id)
success = database.delete_automation(automation_id)
if not success:
return {'error': 'Automation not found'}, 404
return {'success': True}, 200
def duplicate_automation(
database,
automation_engine,
profile_id: int,
automation_id: int,
) -> tuple[dict, int]:
"""Duplicate an automation. System automations are protected."""
auto = database.get_automation(automation_id)
if not auto:
return {'error': 'Automation not found'}, 404
if auto.get('is_system'):
return {'error': 'System automations cannot be duplicated'}, 403
new_id = database.create_automation(
name=f"{auto['name']} (Copy)",
trigger_type=auto['trigger_type'],
trigger_config=auto.get('trigger_config', '{}'),
action_type=auto['action_type'],
action_config=auto.get('action_config', '{}'),
profile_id=profile_id,
notify_type=auto.get('notify_type'),
notify_config=auto.get('notify_config', '{}'),
then_actions=auto.get('then_actions', '[]'),
group_name=auto.get('group_name'),
)
if new_id is None:
return {'error': 'Failed to duplicate automation'}, 500
if automation_engine:
automation_engine.schedule_automation(new_id)
return {'success': True, 'id': new_id}, 200
def toggle_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
"""Toggle an automation's enabled state + reschedule/cancel."""
success = database.toggle_automation(automation_id)
if not success:
return {'error': 'Automation not found'}, 404
if automation_engine:
auto = database.get_automation(automation_id)
if auto and auto.get('enabled'):
automation_engine.schedule_automation(automation_id)
else:
automation_engine.cancel_automation(automation_id)
return {'success': True}, 200
def run_automation(automation_engine, automation_id: int, profile_id: int) -> tuple[dict, int]:
"""Manually trigger an automation."""
if not automation_engine:
return {'error': 'Automation engine not available'}, 500
success = automation_engine.run_now(automation_id, profile_id=profile_id)
if not success:
return {'error': 'Automation not found'}, 404
return {'success': True}, 200
def get_history(database, automation_id: int, *, limit: int, offset: int) -> dict:
"""Run-history page for an automation, with log_lines/result_json parsed."""
data = database.get_automation_run_history(automation_id, limit=limit, offset=offset)
for entry in data.get('history', []):
if entry.get('log_lines'):
try:
entry['log_lines'] = json.loads(entry['log_lines'])
except (json.JSONDecodeError, TypeError):
entry['log_lines'] = []
else:
entry['log_lines'] = []
if entry.get('result_json'):
try:
entry['result_json'] = json.loads(entry['result_json'])
except (json.JSONDecodeError, TypeError):
pass
data['automation_id'] = automation_id
return data

219
core/automation/blocks.py
View file

@ -1,219 +0,0 @@
"""Static block definitions for the automation builder UI.
Returned verbatim by `/api/automations/blocks` (with `known_signals`
injected by the route from `signals.collect_known_signals`).
Three top-level lists:
- `TRIGGERS` — WHEN blocks: schedule, daily/weekly time, app started,
event triggers (track_downloaded, batch_complete, etc.), signal_received,
webhook_received.
- `ACTIONS` — DO blocks: process_wishlist, scan_library, etc.
- `NOTIFICATIONS` — THEN blocks: discord/pushbullet/telegram/webhook,
plus fire_signal and run_script then-actions.
"""
from __future__ import annotations
TRIGGERS: list[dict] = [
{"type": "schedule", "label": "Schedule", "icon": "clock", "description": "Run on a timer interval", "available": True,
"config_fields": [
{"key": "interval", "type": "number", "label": "Every", "default": 6, "min": 1},
{"key": "unit", "type": "select", "label": "Unit",
"options": [{"value": "minutes", "label": "Minutes"}, {"value": "hours", "label": "Hours"}, {"value": "days", "label": "Days"}],
"default": "hours"}
]},
{"type": "daily_time", "label": "Daily Time", "icon": "clock", "description": "Run every day at a specific time", "available": True,
"config_fields": [
{"key": "time", "type": "time", "label": "At", "default": "03:00"}
]},
{"type": "weekly_time", "label": "Weekly Schedule", "icon": "calendar", "description": "Run on specific days of the week at a set time", "available": True,
"config_fields": [
{"key": "time", "type": "time", "label": "At", "default": "03:00"},
{"key": "days", "type": "multi_select", "label": "Days",
"options": [{"value": "mon", "label": "Mon"}, {"value": "tue", "label": "Tue"}, {"value": "wed", "label": "Wed"},
{"value": "thu", "label": "Thu"}, {"value": "fri", "label": "Fri"}, {"value": "sat", "label": "Sat"}, {"value": "sun", "label": "Sun"}]}
]},
{"type": "app_started", "label": "App Started", "icon": "power", "description": "When SoulSync starts up", "available": True},
{"type": "track_downloaded", "label": "Track Downloaded", "icon": "download", "description": "When a track finishes downloading", "available": True,
"has_conditions": True,
"condition_fields": ["artist", "title", "album", "quality"],
"variables": ["artist", "title", "album", "quality"]},
{"type": "batch_complete", "label": "Batch Complete", "icon": "check-circle", "description": "When an album/playlist download finishes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "completed_tracks", "failed_tracks"]},
{"type": "watchlist_new_release", "label": "New Release Found", "icon": "bell", "description": "When watchlist detects new music", "available": True,
"has_conditions": True,
"condition_fields": ["artist"],
"variables": ["artist", "new_tracks", "added_to_wishlist"]},
{"type": "playlist_synced", "label": "Playlist Synced", "icon": "refresh", "description": "When a playlist sync completes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "matched_tracks", "synced_tracks", "failed_tracks"]},
{"type": "playlist_changed", "label": "Playlist Changed", "icon": "edit", "description": "When a mirrored playlist detects track changes from source", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "old_count", "new_count", "added", "removed"]},
{"type": "discovery_completed", "label": "Discovery Complete", "icon": "search", "description": "When playlist track discovery finishes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "discovered_count", "failed_count", "skipped_count"]},
# Phase 3 triggers
{"type": "wishlist_processing_completed", "label": "Wishlist Processed", "icon": "check-circle",
"description": "When auto-wishlist processing finishes", "available": True,
"variables": ["tracks_processed", "tracks_found", "tracks_failed"]},
{"type": "watchlist_scan_completed", "label": "Watchlist Scan Done", "icon": "check-circle",
"description": "When watchlist scan finishes", "available": True,
"variables": ["artists_scanned", "new_tracks_found", "tracks_added"]},
{"type": "database_update_completed", "label": "Database Updated", "icon": "database",
"description": "When library database refresh finishes", "available": True,
"variables": ["total_artists", "total_albums", "total_tracks"]},
{"type": "library_scan_completed", "label": "Library Scan Done", "icon": "hard-drive",
"description": "When media library scan finishes", "available": True,
"variables": ["server_type"]},
{"type": "download_failed", "label": "Download Failed", "icon": "x-circle",
"description": "When a track permanently fails to download", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title", "reason"],
"variables": ["artist", "title", "reason"]},
{"type": "download_quarantined", "label": "File Quarantined", "icon": "alert-triangle",
"description": "When AcoustID verification fails", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title"],
"variables": ["artist", "title", "reason"]},
{"type": "wishlist_item_added", "label": "Wishlist Item Added", "icon": "plus-circle",
"description": "When a track is added to wishlist", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title"],
"variables": ["artist", "title", "reason"]},
{"type": "watchlist_artist_added", "label": "Artist Watched", "icon": "user-plus",
"description": "When an artist is added to watchlist", "available": True,
"has_conditions": True, "condition_fields": ["artist"],
"variables": ["artist", "artist_id"]},
{"type": "watchlist_artist_removed", "label": "Artist Unwatched", "icon": "user-minus",
"description": "When an artist is removed from watchlist", "available": True,
"has_conditions": True, "condition_fields": ["artist"],
"variables": ["artist", "artist_id"]},
{"type": "import_completed", "label": "Import Complete", "icon": "upload",
"description": "When album/track import finishes", "available": True,
"has_conditions": True, "condition_fields": ["artist", "album_name"],
"variables": ["track_count", "album_name", "artist"]},
{"type": "mirrored_playlist_created", "label": "Playlist Mirrored", "icon": "copy",
"description": "When a new playlist is mirrored", "available": True,
"has_conditions": True, "condition_fields": ["playlist_name", "source"],
"variables": ["playlist_name", "source", "track_count"]},
{"type": "quality_scan_completed", "label": "Quality Scan Done", "icon": "bar-chart",
"description": "When quality scan finishes", "available": True,
"variables": ["quality_met", "low_quality", "total_scanned"]},
{"type": "duplicate_scan_completed", "label": "Duplicate Scan Done", "icon": "layers",
"description": "When duplicate cleaner finishes", "available": True,
"variables": ["files_scanned", "duplicates_found", "space_freed"]},
# Signal trigger
{"type": "signal_received", "label": "Signal Received", "icon": "zap",
"description": "When another automation fires a named signal", "available": True,
"config_fields": [
{"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
],
"variables": ["signal_name"]},
# Webhook trigger
{"type": "webhook_received", "label": "Webhook Received", "icon": "globe",
"description": "When an external API request is received (POST /api/v1/request)", "available": True,
"variables": ["query", "request_id", "source"]},
]
ACTIONS: list[dict] = [
{"type": "process_wishlist", "label": "Process Wishlist", "icon": "list", "description": "Retry failed downloads from wishlist", "available": True,
"config_fields": [{"key": "category", "type": "select", "label": "Category", "options": [{"value": "all", "label": "All"}, {"value": "albums", "label": "Albums"}, {"value": "singles", "label": "Singles"}], "default": "all"}]},
{"type": "scan_watchlist", "label": "Scan Watchlist", "icon": "eye", "description": "Check watched artists for new releases", "available": True},
{"type": "scan_library", "label": "Scan Library", "icon": "refresh", "description": "Trigger media server library scan", "available": True},
{"type": "refresh_mirrored", "label": "Refresh Mirrored Playlist", "icon": "copy", "description": "Re-fetch playlist from source and update mirror", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Refresh all mirrored playlists", "default": False}
]},
{"type": "sync_playlist", "label": "Sync Playlist", "icon": "sync", "description": "Sync mirrored playlist to media server", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"}
]},
{"type": "discover_playlist", "label": "Discover Playlist", "icon": "search", "description": "Find official Spotify/iTunes metadata for mirrored playlist tracks", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Discover all mirrored playlists", "default": False}
]},
{"type": "playlist_pipeline", "label": "Playlist Pipeline", "icon": "rocket",
"description": "Full lifecycle: refresh → discover → sync → download missing. One automation for the entire flow.",
"available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Process all mirrored playlists", "default": False},
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
]},
{"type": "personalized_pipeline", "label": "Personalized Playlist Pipeline", "icon": "sparkles",
"description": "Sync personalized / discover-page playlists (Hidden Gems, Time Machine, Fresh Tape, etc.) to your media server + queue missing tracks for download.",
"available": True,
"config_fields": [
{"key": "kinds", "type": "personalized_playlist_select", "label": "Playlists to sync",
"description": "Multi-select: Hidden Gems, Discovery Shuffle, Time Machine (per decade), Genre playlists, Fresh Tape, The Archives, Seasonal Mix (per season)"},
{"key": "refresh_first", "type": "checkbox", "label": "Refresh playlists before sync (regenerate snapshots)", "default": False},
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
]},
{"type": "notify_only", "label": "Notify Only", "icon": "bell", "description": "No action — just send notification", "available": True},
# Phase 3 actions
{"type": "start_database_update", "label": "Update Database", "icon": "database",
"description": "Trigger library database refresh", "available": True,
"config_fields": [
{"key": "full_refresh", "type": "checkbox", "label": "Full refresh (slower)", "default": False}
]},
{"type": "run_duplicate_cleaner", "label": "Run Duplicate Cleaner", "icon": "layers",
"description": "Scan for and remove duplicate files", "available": True},
{"type": "clear_quarantine", "label": "Clear Quarantine", "icon": "trash",
"description": "Delete all quarantined files", "available": True},
{"type": "cleanup_wishlist", "label": "Clean Up Wishlist", "icon": "filter",
"description": "Remove duplicate/owned tracks from wishlist", "available": True},
{"type": "update_discovery_pool", "label": "Update Discovery", "icon": "compass",
"description": "Refresh discovery pool with new tracks", "available": True},
{"type": "start_quality_scan", "label": "Run Quality Scan", "icon": "bar-chart",
"description": "Run the Quality Upgrade Finder (scope is set in Library Maintenance)", "available": True},
{"type": "backup_database", "label": "Backup Database", "icon": "save",
"description": "Create timestamped database backup", "available": True},
{"type": "refresh_beatport_cache", "label": "Refresh Beatport Cache", "icon": "music",
"description": "Scrape Beatport homepage and warm the cache", "available": True},
{"type": "clean_search_history", "label": "Clean Search History", "icon": "trash-2",
"description": "Remove old searches from Soulseek", "available": True},
{"type": "clean_completed_downloads", "label": "Clean Completed Downloads", "icon": "check-square",
"description": "Clear completed downloads and empty directories", "available": True},
{"type": "full_cleanup", "label": "Full Cleanup", "icon": "trash",
"description": "Clear quarantine, download queue, import folder, and search history in one sweep", "available": True},
{"type": "deep_scan_library", "label": "Deep Scan Library", "icon": "search",
"description": "Full library comparison without losing enrichment data", "available": True},
{"type": "run_script", "label": "Run Script", "icon": "terminal",
"description": "Execute a script from the scripts folder", "available": True},
{"type": "search_and_download", "label": "Search & Download", "icon": "download",
"description": "Search for a track and download the best match", "available": True,
"config_fields": [
{"key": "query", "type": "text", "label": "Search Query",
"placeholder": "Artist - Track (leave empty to use trigger's query)"}
]},
]
NOTIFICATIONS: list[dict] = [
{"type": "discord_webhook", "label": "Discord Webhook", "icon": "message", "description": "Send a Discord notification", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "pushbullet", "label": "Pushbullet", "icon": "push", "description": "Push notification to phone/desktop", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "telegram", "label": "Telegram", "icon": "message", "description": "Send a Telegram message", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "webhook", "label": "Webhook (POST)", "icon": "globe", "description": "Send a POST request to any URL", "available": True,
"variables": ["time", "name", "run_count", "status"]},
# Signal fire action
{"type": "fire_signal", "label": "Fire Signal", "icon": "zap",
"description": "Fire a signal that other automations can listen for", "available": True,
"config_fields": [
{"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
]},
# Run script then-action
{"type": "run_script", "label": "Run Script", "icon": "terminal",
"description": "Execute a script after the action completes", "available": True,
"config_fields": [
{"key": "script_name", "type": "script_select", "label": "Script"}
]},
]

163
core/automation/deps.py
View file

@ -1,163 +0,0 @@
"""Dependency-injection surface for automation handlers.
Each handler in ``core.automation.handlers`` is a top-level pure
function that accepts ``(config: dict, deps: AutomationDeps)`` instead
of reaching for module-level globals in ``web_server``. The deps
namespace bundles every callable, client, and mutable-state container
the handlers need.
Construction happens once at app startup in ``web_server.py``:
from core.automation.deps import AutomationDeps, AutomationState
state = AutomationState()
deps = AutomationDeps(
engine=automation_engine,
state=state,
get_database=get_database,
spotify_client=spotify_client,
...
)
register_all(deps)
Tests construct a fake ``AutomationDeps`` with stub callables — every
handler is then exercisable without spinning up Flask, the DB, or
the real media-server clients.
"""
from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, Optional
@dataclass
class AutomationState:
"""Mutable flags shared across handler invocations.
Pre-refactor each was a ``global`` or ``nonlocal`` variable inside
the registration closure. Lifted here so handlers + their guards
can read/write a single object instead of importing globals.
All mutations should hold ``lock``; the helper methods below do
so for the common get/set patterns.
"""
scan_library_automation_id: Optional[str] = None
db_update_automation_id: Optional[str] = None
pipeline_running: bool = False
lock: threading.Lock = field(default_factory=threading.Lock)
def is_scan_library_active(self) -> bool:
with self.lock:
return self.scan_library_automation_id is not None
def is_pipeline_running(self) -> bool:
with self.lock:
return self.pipeline_running
def try_start_pipeline(self) -> bool:
"""Atomically mark the shared playlist pipeline as running."""
with self.lock:
if self.pipeline_running:
return False
self.pipeline_running = True
return True
def set_scan_library_id(self, automation_id: Optional[str]) -> None:
with self.lock:
self.scan_library_automation_id = automation_id
def set_pipeline_running(self, value: bool) -> None:
with self.lock:
self.pipeline_running = value
@dataclass
class AutomationDeps:
"""Bundle of every callable + client an automation handler may need.
Add fields as new handlers are extracted. Every field is required
at construction (no defaults) so a missing dep fails loudly at
startup, not silently mid-handler.
"""
# --- Engine + shared state ---
engine: Any # AutomationEngine instance
state: AutomationState
config_manager: Any # config.settings.ConfigManager singleton
update_progress: Callable[..., None] # _update_automation_progress
logger: Any # module-level logger from utils.logging_config
# --- Service clients (each may be None depending on user config) ---
get_database: Callable[[], Any] # late-binding so tests don't need DB
spotify_client: Any
tidal_client: Any
web_scan_manager: Any
# --- Background-task entry points ---
process_wishlist_automatically: Callable[..., Any]
process_watchlist_scan_automatically: Callable[..., Any]
is_wishlist_actually_processing: Callable[[], bool]
is_watchlist_actually_scanning: Callable[[], bool]
get_watchlist_scan_state: Callable[[], dict] # accessor returns the live mutable dict
# --- Playlist pipeline entry points ---
run_playlist_discovery_worker: Callable[..., Any]
run_sync_task: Callable[..., Any]
run_playlist_organize_download: Callable[..., Dict[str, Any]]
missing_download_executor: Any
load_sync_status_file: Callable[[], dict]
get_deezer_client: Callable[[], Any]
parse_youtube_playlist: Callable[[str], Any]
get_sync_states: Callable[[], dict] # accessor returns the live dict shared with the sync UI
# --- Database update + quality scanner (shared state + executors) ---
set_db_update_automation_id: Callable[[Optional[str]], None] # syncs the legacy `_db_update_automation_id` global so the live DB-update progress callbacks (which still read the global directly) keep firing for the active automation
get_db_update_state: Callable[[], dict]
db_update_lock: Any # threading.Lock
db_update_executor: Any # ThreadPoolExecutor
run_db_update_task: Callable[..., Any]
run_deep_scan_task: Callable[..., Any]
get_duplicate_cleaner_state: Callable[[], dict]
duplicate_cleaner_lock: Any
duplicate_cleaner_executor: Any
run_duplicate_cleaner: Callable[..., Any]
# Triggers a "Run Now" of a library-maintenance repair job by id (e.g.
# 'quality_upgrade'). Returns truthy if the job was queued. Replaces the old
# standalone quality-scanner executor/state (the scanner is now a repair job).
run_repair_job_now: Callable[[str], Any]
# --- Download orchestrator + queue accessors ---
download_orchestrator: Any
run_async: Callable[..., Any]
tasks_lock: Any
get_download_batches: Callable[[], dict]
get_download_tasks: Callable[[], dict]
sweep_empty_download_directories: Callable[[], int]
get_staging_path: Callable[[], str]
# --- Maintenance helpers ---
docker_resolve_path: Callable[[str], str]
get_current_profile_id: Callable[[], int]
get_watchlist_scanner: Callable[[Any], Any]
get_app: Callable[[], Any] # Flask app for test_client (beatport refresh)
get_beatport_data_cache: Callable[[], dict]
# --- Progress + history callbacks (used by register_all to wire
# the engine's progress callback hooks). ---
init_automation_progress: Callable[..., Any]
record_progress_history: Callable[..., Any]
# --- Personalized playlist pipeline ---
# Lazy builder so the pipeline handler can construct a fresh
# PersonalizedPlaylistManager per run (cheap accessors inside,
# no caching needed yet).
build_personalized_manager: Callable[[], Any]
# --- Unified PlaylistSource registry ---
# Optional so test fixtures that don't exercise refresh_mirrored
# can keep their existing scaffolding. Production wiring in
# ``web_server.py`` always populates it via
# ``core.playlists.sources.bootstrap.build_playlist_source_registry``.
playlist_source_registry: Optional[Any] = None

64
core/automation/handlers/__init__.py
View file

@ -1,64 +0,0 @@
"""Per-action automation handlers.
Each module in this subpackage exposes one top-level handler function
(or a small cluster of related handlers) of the form::
def auto_<action_name>(config: dict, deps: AutomationDeps) -> dict
The ``register_all`` helper in :mod:`registration` wires every handler
to the engine in one place. ``web_server.py`` calls
``register_all(deps)`` once at startup.
"""
from core.automation.handlers.process_wishlist import auto_process_wishlist
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
from core.automation.handlers.scan_library import auto_scan_library
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.automation.handlers.discover_playlist import auto_discover_playlist
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
from core.automation.handlers.database_update import auto_start_database_update, auto_deep_scan_library
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
from core.automation.handlers.quality_scanner import auto_start_quality_scan
from core.automation.handlers.maintenance import (
auto_clear_quarantine,
auto_cleanup_wishlist,
auto_update_discovery_pool,
auto_backup_database,
auto_refresh_beatport_cache,
)
from core.automation.handlers.download_cleanup import (
auto_clean_search_history,
auto_clean_completed_downloads,
auto_full_cleanup,
)
from core.automation.handlers.run_script import auto_run_script
from core.automation.handlers.search_and_download import auto_search_and_download
from core.automation.handlers.registration import register_all
__all__ = [
'auto_process_wishlist',
'auto_scan_watchlist',
'auto_scan_library',
'auto_refresh_mirrored',
'auto_sync_playlist',
'auto_discover_playlist',
'auto_playlist_pipeline',
'auto_personalized_pipeline',
'auto_start_database_update',
'auto_deep_scan_library',
'auto_run_duplicate_cleaner',
'auto_start_quality_scan',
'auto_clear_quarantine',
'auto_cleanup_wishlist',
'auto_update_discovery_pool',
'auto_backup_database',
'auto_refresh_beatport_cache',
'auto_clean_search_history',
'auto_clean_completed_downloads',
'auto_full_cleanup',
'auto_run_script',
'auto_search_and_download',
'register_all',
]

259
core/automation/handlers/_pipeline_shared.py
View file

@ -1,259 +0,0 @@
"""Shared helpers between mirrored + personalized playlist pipelines.
Both pipelines end in the same shape:
1. SYNC each playlist to the active media server.
2. WISHLIST: trigger the wishlist processor for missing tracks.
The differing prefix (mirrored = REFRESH external sources + DISCOVER
metadata; personalized = SNAPSHOT manager-backed playlists) is owned
by each pipeline. This module owns the SYNC + WISHLIST tail so both
pipelines stay consistent + DRY.
"""
from __future__ import annotations
import time
from typing import Any, Callable, Dict, List, Optional
from core.automation.deps import AutomationDeps
# Per-playlist sync poll cap (mirrored side already used this).
_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS = 600
# Sync-status final-state markers.
_SYNC_TERMINAL_STATUSES = ('finished', 'complete', 'error', 'failed')
def run_sync_and_wishlist(
deps: AutomationDeps,
automation_id: Optional[str],
playlists: List[Dict[str, Any]],
*,
sync_one_fn: Callable[[Dict[str, Any]], Dict[str, Any]],
sync_id_for_fn: Callable[[Dict[str, Any]], str],
skip_wishlist: bool = False,
progress_start: int = 56,
progress_end: int = 85,
sync_phase_label: str = 'Phase: Syncing to server...',
sync_phase_start_log: str = 'Sync',
wishlist_phase_label: str = 'Phase: Processing wishlist...',
wishlist_phase_start_log: str = 'Wishlist',
) -> Dict[str, int]:
"""Run the SYNC + WISHLIST tail of a playlist pipeline.
The caller supplies:
- ``playlists``: list of playlist payload dicts. Each must have at
least a ``name`` key (used in progress logs). The shape beyond
``name`` is opaque to the helper — ``sync_one_fn`` receives the
payload and returns a sync_result dict.
- ``sync_one_fn(payload) -> sync_result``: launches sync for one
playlist. Result dict must carry ``status`` ∈ ``('started',
'skipped', 'error')`` and may carry ``reason``.
- ``sync_id_for_fn(payload) -> str``: returns the sync-state key
the helper polls on (so we can wait for the background sync
thread to complete + read the matched_tracks count).
Returns ``{'synced': int, 'skipped': int, 'errors': int,
'wishlist_queued': int}`` so the caller can stitch it into its
final status.
"""
deps.update_progress(
automation_id,
progress=progress_start,
phase=sync_phase_label,
log_line=sync_phase_start_log,
log_type='info',
)
total_synced = 0
total_skipped = 0
sync_errors = 0
sync_states = deps.get_sync_states()
n_playlists = max(1, len(playlists))
progress_span = max(1, progress_end - progress_start - 1)
for pl_idx, pl in enumerate(playlists):
pl_name = pl.get('name', '')
sync_result = sync_one_fn(pl)
sync_status = sync_result.get('status', '')
if sync_status == 'started':
sync_id = sync_id_for_fn(pl)
sync_poll_start = time.time()
timed_out = True
while time.time() - sync_poll_start < _SYNC_PER_PLAYLIST_TIMEOUT_SECONDS:
if (sync_id in sync_states
and sync_states[sync_id].get('status') in _SYNC_TERMINAL_STATUSES):
timed_out = False
break
time.sleep(2)
elapsed = int(time.time() - sync_poll_start)
sub_progress = progress_start + 1 + ((pl_idx + 1) / n_playlists) * progress_span
deps.update_progress(
automation_id,
progress=min(int(sub_progress), progress_end - 1),
phase=f'{sync_phase_label.rstrip(".")} — "{pl_name}" ({elapsed}s)',
)
ss = sync_states.get(sync_id, {})
final_status = ss.get('status')
if timed_out:
sync_errors += 1
deps.update_progress(
automation_id,
log_line=f'Sync timed out "{pl_name}" after {_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS}s',
log_type='error',
)
continue
if final_status in ('error', 'failed'):
sync_errors += 1
reason = ss.get('error') or ss.get('reason') or 'background sync failed'
deps.update_progress(
automation_id,
log_line=f'Sync error "{pl_name}": {reason}',
log_type='error',
)
continue
ss_result = ss.get('result', ss.get('progress', {}))
matched = ss_result.get('matched_tracks', 0) if isinstance(ss_result, dict) else 0
total_synced += int(matched) if matched else 0
deps.update_progress(
automation_id,
log_line=f'Synced "{pl_name}": {matched} tracks matched',
log_type='success',
)
elif sync_status == 'skipped':
total_skipped += 1
reason = sync_result.get('reason', 'unchanged')
deps.update_progress(
automation_id,
log_line=f'Skipped "{pl_name}": {reason}',
log_type='skip',
)
elif sync_status == 'error':
sync_errors += 1
deps.update_progress(
automation_id,
log_line=f'Sync error "{pl_name}": {sync_result.get("reason", "unknown")}',
log_type='error',
)
deps.update_progress(
automation_id,
progress=progress_end,
phase=f'{sync_phase_label.rstrip(".")} complete',
log_line=f'Sync done: {total_synced} matched, {total_skipped} skipped, {sync_errors} errors',
log_type='success' if sync_errors == 0 else 'warning',
)
organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
organize_started = 0
if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
for pl in organize_playlists:
pl_id = pl.get('id')
if not pl_id:
continue
pl_name = pl.get('name', '')
try:
org_result = deps.run_playlist_organize_download(
mirrored_playlist_id=int(pl_id),
automation_id=automation_id,
)
if org_result.get('status') == 'started':
organize_started += 1
deps.update_progress(
automation_id,
log_line=f'Organize download started for "{pl_name}"',
log_type='success',
)
elif org_result.get('status') == 'skipped':
deps.update_progress(
automation_id,
log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
log_type='skip',
)
except Exception as org_err: # noqa: BLE001
deps.update_progress(
automation_id,
log_line=f'Organize download error for "{pl_name}": {org_err}',
log_type='warning',
)
all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
effective_skip_wishlist = skip_wishlist or all_organize
wishlist_queued = run_wishlist_phase(
deps, automation_id,
skip=effective_skip_wishlist,
progress_pct=progress_end + 1,
wishlist_phase_label=wishlist_phase_label,
wishlist_phase_start_log=wishlist_phase_start_log,
)
return {
'synced': total_synced,
'skipped': total_skipped,
'errors': sync_errors,
'wishlist_queued': wishlist_queued,
'organize_downloads_started': organize_started,
}
def run_wishlist_phase(
deps: AutomationDeps,
automation_id: Optional[str],
*,
skip: bool,
progress_pct: int,
wishlist_phase_label: str = 'Phase: Processing wishlist...',
wishlist_phase_start_log: str = 'Wishlist',
) -> int:
"""Trigger the wishlist processor unless skipped or already running.
Returns 1 when the processor was triggered, 0 otherwise. Errors are
logged but never raised — wishlist failure should not abort the
pipeline."""
if skip:
deps.update_progress(
automation_id,
progress=progress_pct,
log_line=f'{wishlist_phase_start_log}: skipped (disabled)',
log_type='skip',
)
return 0
deps.update_progress(
automation_id,
progress=progress_pct,
phase=wishlist_phase_label,
log_line=wishlist_phase_start_log,
log_type='info',
)
try:
if not deps.is_wishlist_actually_processing():
deps.process_wishlist_automatically(automation_id=None)
deps.update_progress(
automation_id,
log_line='Wishlist processing triggered',
log_type='success',
)
return 1
deps.update_progress(
automation_id,
log_line='Wishlist already running — skipped',
log_type='skip',
)
return 0
except Exception as e: # noqa: BLE001 — wishlist failure must never abort pipeline
deps.update_progress(
automation_id,
log_line=f'Wishlist error: {e}',
log_type='warning',
)
return 0
__all__ = ['run_sync_and_wishlist', 'run_wishlist_phase']

197
core/automation/handlers/database_update.py
View file

@ -1,197 +0,0 @@
"""Automation handlers: ``start_database_update`` and
``deep_scan_library`` actions.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_start_database_update`` and ``_auto_deep_scan_library``
closures). Both share the same ``db_update_state`` / executor / lock
infrastructure -- the only difference is which task they submit
(``run_db_update_task`` vs ``run_deep_scan_task``).
Pattern: pre-set state to running, submit task to executor, then
poll the state dict until it transitions away from ``running``.
Stall-detection emits a warning every 10 minutes when progress
hasn't budged. 2-hour outer timeout caps the worst case.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# Time out on STALL (no progress), not total runtime: a large library can scan
# for many hours while progressing fine — a hard total cap would falsely mark a
# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
# up when progress hasn't moved for a long stretch, with a generous absolute
# backstop against a truly stuck monitor loop.
_STALL_WARNING_SECONDS = 600 # warn after 10 min with no progress (repeats)
_STALL_TIMEOUT_SECONDS = 1800 # 30 min with no progress at all = genuinely stalled
_ABSOLUTE_CAP_SECONDS = 86400 # 24h hard backstop (runaway-loop guard only)
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def scan_wait_action(
*,
status: str,
idle_seconds: float,
total_seconds: float,
stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
stall_warn_s: float = _STALL_WARNING_SECONDS,
abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
) -> str:
"""Decide what the monitor loop should do on a poll tick (pure/testable).
``idle_seconds`` is time since progress last changed; ``total_seconds`` is
time since the wait began. Returns one of:
``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
too long → give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
(stalled long enough to warn but not give up), or ``'continue'``.
Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
it never hits ``stall_timeout`` no matter how long the whole scan takes.
"""
if status != 'running':
return 'finished'
if total_seconds >= abs_cap_s:
return 'abs_timeout'
if idle_seconds >= stall_timeout_s:
return 'stall_timeout'
if idle_seconds >= stall_warn_s:
return 'warn'
return 'continue'
def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a full or incremental DB update via ``run_db_update_task``."""
return _run_with_progress(
config, deps,
task=deps.run_db_update_task,
task_args=(config.get('full_refresh', False), deps.config_manager.get_active_media_server()),
initial_phase='Initializing...',
stall_label='Database update',
finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
timeout_label='Database update timed out after 24 hours',
)
def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a deep library scan via ``run_deep_scan_task``."""
return _run_with_progress(
config, deps,
task=deps.run_deep_scan_task,
task_args=(deps.config_manager.get_active_media_server(),),
initial_phase='Deep scan: Initializing...',
stall_label='Deep scan',
finished_extras=lambda: {},
timeout_label='Deep scan timed out after 24 hours',
)
def _run_with_progress(
config: Dict[str, Any],
deps: AutomationDeps,
*,
task,
task_args: tuple,
initial_phase: str,
stall_label: str,
finished_extras,
timeout_label: str,
) -> Dict[str, Any]:
"""Shared poll-and-wait body for both DB-update handlers."""
automation_id = config.get('_automation_id')
state = deps.get_db_update_state()
if state.get('status') == 'running':
return {'status': 'skipped', 'reason': 'Database update already running'}
deps.state.db_update_automation_id = automation_id
# Sync legacy module global so the DB-update progress callbacks
# (still living in web_server.py) emit against this automation.
deps.set_db_update_automation_id(automation_id)
with deps.db_update_lock:
state.update({
'status': 'running', 'phase': initial_phase,
'progress': 0, 'current_item': '', 'processed': 0, 'total': 0,
'error_message': '',
})
deps.db_update_executor.submit(task, *task_args)
# Monitor progress (callbacks handle card updates, we just block until done).
# We time out on STALL, not total runtime: ``processed`` advances on every
# artist, so an actively-progressing scan keeps resetting the idle clock and
# is never falsely failed no matter how long the whole library takes.
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
last_progress_time = time.time()
# Any of these advancing means the scan is alive. current_item (the artist
# being processed) changes every artist even when the rounded progress %
# holds steady, so it guards against a false stall during slow stretches.
last_progress_val = (0, 0, '')
last_warn_time = 0.0
outcome = 'finished'
while True:
time.sleep(_POLL_INTERVAL_SECONDS)
now = time.time()
with deps.db_update_lock:
current_status = state.get('status', 'idle')
current_val = (state.get('processed', 0), state.get('progress', 0),
state.get('current_item', ''))
if current_val != last_progress_val:
last_progress_val = current_val
last_progress_time = now
action = scan_wait_action(
status=current_status,
idle_seconds=now - last_progress_time,
total_seconds=now - poll_start,
)
if action in ('finished', 'stall_timeout', 'abs_timeout'):
outcome = action
break
if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
idle_min = int((now - last_progress_time) / 60)
deps.update_progress(
automation_id,
log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
log_type='warning',
)
last_warn_time = now
if outcome == 'stall_timeout':
deps.update_progress(
automation_id, status='error', phase='Stalled',
log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
log_type='error',
)
return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
if outcome == 'abs_timeout':
deps.update_progress(
automation_id, status='error',
phase='Timed out', log_line=timeout_label, log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
# Finished/error callback already updated the card — return matching status.
with deps.db_update_lock:
final_status = state.get('status', 'unknown')
if final_status == 'error':
return {
'status': 'error',
'reason': state.get('error_message', 'Unknown error'),
'_manages_own_progress': True,
}
with deps.db_update_lock:
stats = {
'status': 'completed', '_manages_own_progress': True,
'artists': state.get('total', 0),
'albums': state.get('total_albums', 0),
'tracks': state.get('total_tracks', 0),
'removed_artists': state.get('removed_artists', 0),
'removed_albums': state.get('removed_albums', 0),
'removed_tracks': state.get('removed_tracks', 0),
}
stats.update(finished_extras())
return stats

48
core/automation/handlers/discover_playlist.py
View file

@ -1,48 +0,0 @@
"""Automation handler: ``discover_playlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_discover_playlist`` closure). Kicks off background discovery
of official Spotify / iTunes metadata for mirrored playlist tracks.
The worker runs in a daemon thread and emits its own progress; this
handler returns immediately after launching it (``_manages_own_progress``).
"""
from __future__ import annotations
import threading
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_discover_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Discover official Spotify/iTunes metadata for mirrored
playlist tracks. Runs the worker in a background thread."""
db = deps.get_database()
playlist_id = config.get('playlist_id')
discover_all = config.get('all', False)
if discover_all:
playlists = db.get_mirrored_playlists()
elif playlist_id:
p = db.get_mirrored_playlist(int(playlist_id))
playlists = [p] if p else []
else:
return {'status': 'error', 'reason': 'No playlist specified'}
if not playlists:
return {'status': 'error', 'reason': 'No playlists found'}
threading.Thread(
target=deps.run_playlist_discovery_worker,
args=(playlists, config.get('_automation_id')),
daemon=True,
name='auto-discover-playlist',
).start()
names = ', '.join(p['name'] for p in playlists[:3])
return {
'status': 'started',
'playlist_count': str(len(playlists)),
'playlists': names,
'_manages_own_progress': True,
}

267
core/automation/handlers/download_cleanup.py
View file

@ -1,267 +0,0 @@
"""Automation handlers: download-queue cleanup actions.
Lifted from ``web_server._register_automation_handlers``:
- ``clean_search_history`` → :func:`auto_clean_search_history`
- ``clean_completed_downloads`` → :func:`auto_clean_completed_downloads`
- ``full_cleanup`` → :func:`auto_full_cleanup`
All three share the download-orchestrator + tasks_lock /
download_batches / download_tasks accessors. ``full_cleanup`` is a
multi-step orchestration that pulls in quarantine purge + staging
sweep on top of the queue cleanup -- kept as one big handler since
its phases share state-detection logic.
"""
from __future__ import annotations
import os
import shutil as _shutil
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# ─── clean_search_history ────────────────────────────────────────────
def auto_clean_search_history(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Remove old searches from Soulseek when configured."""
automation_id = config.get('_automation_id')
# Skip if soulseek is not the active download source or in hybrid order.
dl_mode = deps.config_manager.get('download_source.mode', 'hybrid')
hybrid_order = deps.config_manager.get(
'download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'],
)
soulseek_active = (
dl_mode == 'soulseek'
or (dl_mode == 'hybrid' and 'soulseek' in hybrid_order)
)
# Reach the underlying SoulseekClient via the orchestrator's
# generic accessor.
slskd = deps.download_orchestrator.client('soulseek') if deps.download_orchestrator else None
if not soulseek_active or not slskd or not slskd.base_url:
deps.update_progress(automation_id, log_line='Soulseek not active — skipped', log_type='skip')
return {'status': 'skipped'}
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
deps.update_progress(
automation_id, log_line='Auto-clear disabled in settings', log_type='skip',
)
return {'status': 'skipped'}
try:
success = deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
keep_searches=50, trigger_threshold=200,
))
if success:
deps.update_progress(
automation_id,
log_line='Search history maintenance completed',
log_type='success',
)
return {'status': 'completed'}
else:
deps.update_progress(automation_id, log_line='No cleanup needed', log_type='skip')
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'error': str(e)}
# ─── clean_completed_downloads ───────────────────────────────────────
def auto_clean_completed_downloads(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Clear completed downloads + sweep empty download directories.
Skips when active batches or post-processing is in flight."""
automation_id = config.get('_automation_id')
try:
has_active_batches = False
has_post_processing = False
with deps.tasks_lock:
batches = deps.get_download_batches()
for batch_data in batches.values():
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
has_active_batches = True
break
if not has_active_batches:
tasks = deps.get_download_tasks()
for task_data in tasks.values():
if task_data.get('status') == 'post_processing':
has_post_processing = True
break
if has_active_batches:
deps.update_progress(
automation_id, log_line='Skipped — downloads active', log_type='skip',
)
return {'status': 'completed'}
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
if not has_post_processing:
deps.sweep_empty_download_directories()
deps.update_progress(
automation_id, log_line='Download cleanup completed', log_type='success',
)
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'reason': str(e)}
# ─── full_cleanup ────────────────────────────────────────────────────
def auto_full_cleanup(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run all cleanup tasks: quarantine purge → download queue clear
→ empty-dir sweep → staging sweep → search history."""
automation_id = config.get('_automation_id')
steps = []
# --- 1. Clear quarantine ---
deps.update_progress(automation_id, phase='Clearing quarantine...', progress=0)
quarantine_path = os.path.join(
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
'ss_quarantine',
)
q_removed = 0
if os.path.exists(quarantine_path):
for f in os.listdir(quarantine_path):
fp = os.path.join(quarantine_path, f)
try:
if os.path.isfile(fp):
os.remove(fp)
q_removed += 1
elif os.path.isdir(fp):
_shutil.rmtree(fp)
q_removed += 1
except Exception as e: # noqa: BLE001 — best-effort purge
deps.logger.debug("quarantine entry purge failed: %s", e)
steps.append(f'Quarantine: removed {q_removed} items')
deps.update_progress(
automation_id,
log_line=f'Quarantine: removed {q_removed} items',
log_type='success' if q_removed else 'info',
)
# --- 2. Clear completed/errored/cancelled downloads from Soulseek queue ---
deps.update_progress(automation_id, phase='Clearing download queue...', progress=20)
has_active_batches = False
has_post_processing = False
with deps.tasks_lock:
batches = deps.get_download_batches()
for batch_data in batches.values():
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
has_active_batches = True
break
if not has_active_batches:
tasks = deps.get_download_tasks()
for task_data in tasks.values():
if task_data.get('status') == 'post_processing':
has_post_processing = True
break
if has_active_batches:
steps.append('Download queue: skipped (active batches)')
deps.update_progress(
automation_id,
log_line='Download queue: skipped (active batches)',
log_type='skip',
)
else:
try:
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
steps.append('Download queue: cleared')
deps.update_progress(
automation_id, log_line='Download queue: cleared', log_type='success',
)
except Exception as e: # noqa: BLE001 — per-step best-effort
steps.append(f'Download queue: error ({e})')
deps.update_progress(
automation_id,
log_line=f'Download queue: error ({e})',
log_type='error',
)
# --- 3. Sweep empty download directories ---
deps.update_progress(automation_id, phase='Sweeping empty directories...', progress=40)
if has_active_batches or has_post_processing:
reason = 'active batches' if has_active_batches else 'post-processing active'
steps.append(f'Empty directories: skipped ({reason})')
deps.update_progress(
automation_id,
log_line=f'Empty directories: skipped ({reason})',
log_type='skip',
)
else:
dirs_removed = deps.sweep_empty_download_directories()
steps.append(f'Empty directories: removed {dirs_removed}')
deps.update_progress(
automation_id,
log_line=f'Empty directories: removed {dirs_removed}',
log_type='success' if dirs_removed else 'info',
)
# --- 4. Sweep empty staging directories ---
deps.update_progress(automation_id, phase='Sweeping import folder...', progress=60)
staging_path = deps.get_staging_path()
s_removed = 0
if os.path.isdir(staging_path):
for dirpath, _dirnames, _filenames in os.walk(staging_path, topdown=False):
if os.path.normpath(dirpath) == os.path.normpath(staging_path):
continue
try:
entries = os.listdir(dirpath)
except OSError:
continue
visible = [e for e in entries if not e.startswith('.')]
if not visible:
for hidden in entries:
try:
os.remove(os.path.join(dirpath, hidden))
except Exception as e: # noqa: BLE001 — best-effort
deps.logger.debug("hidden file cleanup failed: %s", e)
try:
os.rmdir(dirpath)
s_removed += 1
except OSError:
pass
steps.append(f'Staging: removed {s_removed} empty directories')
deps.update_progress(
automation_id,
log_line=f'Staging: removed {s_removed} empty directories',
log_type='success' if s_removed else 'info',
)
# --- 5. Clean search history (if enabled) ---
deps.update_progress(automation_id, phase='Cleaning search history...', progress=80)
try:
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
steps.append('Search cleanup: disabled in settings')
deps.update_progress(
automation_id, log_line='Search cleanup: disabled in settings', log_type='skip',
)
else:
deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
keep_searches=50, trigger_threshold=200,
))
steps.append('Search history: cleaned')
deps.update_progress(
automation_id, log_line='Search history: cleaned', log_type='success',
)
except Exception as e: # noqa: BLE001 — per-step best-effort
steps.append(f'Search history: error ({e})')
deps.update_progress(
automation_id, log_line=f'Search history: error ({e})', log_type='error',
)
total_removed = q_removed + s_removed
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Full cleanup complete — {total_removed} items removed',
log_type='success',
)
return {
'status': 'completed',
'quarantine_removed': str(q_removed),
'staging_removed': str(s_removed),
'total_removed': str(total_removed),
'steps': steps,
'_manages_own_progress': True,
}

87
core/automation/handlers/duplicate_cleaner.py
View file

@ -1,87 +0,0 @@
"""Automation handler: ``run_duplicate_cleaner`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_run_duplicate_cleaner`` closure). Submits the duplicate
cleaner to its executor, then polls the shared state dict until
the worker transitions away from ``running``.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
_TIMEOUT_SECONDS = 7200 # 2 hours
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def auto_run_duplicate_cleaner(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Kick off the duplicate cleaner and report final stats."""
automation_id = config.get('_automation_id')
state = deps.get_duplicate_cleaner_state()
if state.get('status') == 'running':
return {'status': 'skipped', 'reason': 'Duplicate cleaner already running'}
# Pre-set status before submit so the polling loop doesn't see a
# stale 'finished' from a previous run.
with deps.duplicate_cleaner_lock:
state['status'] = 'running'
deps.duplicate_cleaner_executor.submit(deps.run_duplicate_cleaner)
deps.update_progress(automation_id, log_line='Duplicate cleaner started', log_type='info')
# Monitor progress (max 2 hours).
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
while time.time() - poll_start < _TIMEOUT_SECONDS:
time.sleep(_POLL_INTERVAL_SECONDS)
current_status = state.get('status', 'idle')
if current_status not in ('running',):
break
deps.update_progress(
automation_id,
phase=state.get('phase', 'Scanning...'),
progress=state.get('progress', 0),
processed=state.get('files_scanned', 0),
total=state.get('total_files', 0),
)
else:
# 2-hour timeout reached.
deps.update_progress(
automation_id, status='error',
phase='Timed out',
log_line='Duplicate cleaner timed out after 2 hours',
log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
# Check actual exit status (could be 'finished' or 'error').
final_status = state.get('status', 'idle')
if final_status == 'error':
err = state.get('error_message', 'Unknown error')
deps.update_progress(
automation_id, status='error', progress=100,
phase='Error', log_line=err, log_type='error',
)
return {'status': 'error', 'reason': err, '_manages_own_progress': True}
dupes = state.get('duplicates_found', 0)
removed = state.get('deleted', 0)
space_freed = state.get('space_freed', 0)
scanned = state.get('files_scanned', 0)
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Found {dupes} duplicates, removed {removed} files',
log_type='success',
)
return {
'status': 'completed', '_manages_own_progress': True,
'files_scanned': scanned,
'duplicates_found': dupes,
'files_deleted': removed,
'space_freed_mb': round(space_freed / (1024 * 1024), 1),
}

221
core/automation/handlers/maintenance.py
View file

@ -1,221 +0,0 @@
"""Automation handlers: short maintenance actions.
Lifted from ``web_server._register_automation_handlers``:
- ``clear_quarantine`` → :func:`auto_clear_quarantine`
- ``cleanup_wishlist`` → :func:`auto_cleanup_wishlist`
- ``update_discovery_pool`` → :func:`auto_update_discovery_pool`
- ``backup_database`` → :func:`auto_backup_database`
- ``refresh_beatport_cache`` → :func:`auto_refresh_beatport_cache`
Each is a thin wrapper around an existing service / helper. Grouped
in one module because every body is short and they share no state
between them — splitting into per-handler files would just add
import noise.
"""
from __future__ import annotations
import glob as _glob
import os
import shutil as _shutil
import sqlite3
import time
from datetime import datetime
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# ─── clear_quarantine ────────────────────────────────────────────────
def auto_clear_quarantine(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Purge every file/folder under the configured ss_quarantine path."""
automation_id = config.get('_automation_id')
quarantine_path = os.path.join(
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
'ss_quarantine',
)
if not os.path.exists(quarantine_path):
deps.update_progress(automation_id, log_line='No quarantine folder found', log_type='info')
return {'status': 'completed', 'removed': '0'}
removed = 0
for f in os.listdir(quarantine_path):
fp = os.path.join(quarantine_path, f)
try:
if os.path.isfile(fp):
os.remove(fp)
removed += 1
elif os.path.isdir(fp):
_shutil.rmtree(fp)
removed += 1
except Exception as e: # noqa: BLE001 — best-effort purge
deps.logger.debug("quarantine entry purge failed: %s", e)
deps.update_progress(
automation_id,
log_line=f'Removed {removed} quarantined items',
log_type='success' if removed > 0 else 'info',
)
return {'status': 'completed', 'removed': str(removed)}
# ─── cleanup_wishlist ────────────────────────────────────────────────
def auto_cleanup_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Drop duplicate entries from the wishlist for the active profile."""
automation_id = config.get('_automation_id')
db = deps.get_database()
removed = db.remove_wishlist_duplicates(deps.get_current_profile_id())
deps.update_progress(
automation_id,
log_line=f'Removed {removed or 0} duplicate wishlist entries',
log_type='success' if removed else 'info',
)
return {'status': 'completed', 'removed': str(removed or 0)}
# ─── update_discovery_pool ───────────────────────────────────────────
def auto_update_discovery_pool(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run an incremental refresh of the discovery pool via the
watchlist scanner."""
automation_id = config.get('_automation_id')
try:
scanner = deps.get_watchlist_scanner(deps.spotify_client)
deps.update_progress(automation_id, log_line='Updating discovery pool...', log_type='info')
scanner.update_discovery_pool_incremental(deps.get_current_profile_id())
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete', log_line='Discovery pool updated', log_type='success',
)
return {'status': 'completed', '_manages_own_progress': True}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
deps.update_progress(
automation_id, status='error',
phase='Error', log_line=str(e), log_type='error',
)
return {'status': 'error', 'reason': str(e), '_manages_own_progress': True}
# ─── backup_database ─────────────────────────────────────────────────
_MAX_BACKUPS = 5
def auto_backup_database(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Create a hot SQLite backup, then prune old backups so only the
newest ``_MAX_BACKUPS`` remain."""
automation_id = config.get('_automation_id')
db_path = os.environ.get('DATABASE_PATH', 'database/music_library.db')
if not os.path.exists(db_path):
return {'status': 'error', 'reason': 'Database file not found'}
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
backup_path = f"{db_path}.backup_{timestamp}"
# safe_backup verifies source + result integrity, so an automated backup
# can never silently snapshot a corrupt DB (the incident where every
# rolling backup faithfully copied the corruption).
from core.db_integrity import DBIntegrityError, safe_backup, prune_backups
try:
safe_backup(db_path, backup_path)
except DBIntegrityError as integ:
deps.logger.error("Auto-backup refused — DB integrity check failed: %s", integ)
deps.update_progress(
automation_id,
log_line=f'Backup SKIPPED — database failed integrity check: {integ}',
log_type='error',
)
return {'status': 'error', 'reason': f'Database integrity check failed: {integ}'}
size_mb = round(os.path.getsize(backup_path) / (1024 * 1024), 1)
# Rolling cleanup — never evict the most-recent verified-healthy backup.
existing = list(_glob.glob(f"{db_path}.backup_*"))
for removed in prune_backups(existing, _MAX_BACKUPS):
try:
os.remove(removed)
except Exception as e: # noqa: BLE001 — best-effort cleanup
deps.logger.debug("rolling backup cleanup failed: %s", e)
deps.update_progress(
automation_id,
log_line=f'Backup created: {size_mb}MB ({os.path.basename(backup_path)})',
log_type='success',
)
return {'status': 'completed', 'backup_path': backup_path, 'size_mb': str(size_mb)}
# ─── refresh_beatport_cache ──────────────────────────────────────────
_BEATPORT_SECTIONS = (
('hero_tracks', '/api/beatport/hero-tracks', 'Hero Tracks'),
('new_releases', '/api/beatport/new-releases', 'New Releases'),
('featured_charts', '/api/beatport/featured-charts', 'Featured Charts'),
('dj_charts', '/api/beatport/dj-charts', 'DJ Charts'),
('top_10_lists', '/api/beatport/homepage/top-10-lists', 'Top 10 Lists'),
('top_10_releases', '/api/beatport/homepage/top-10-releases-cards', 'Top 10 Releases'),
('hype_picks', '/api/beatport/hype-picks', 'Hype Picks'),
)
def auto_refresh_beatport_cache(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Refresh Beatport homepage cache by calling each endpoint internally
via Flask's ``test_client``. Invalidates the homepage cache first
so endpoints re-scrape rather than returning stale data."""
automation_id = config.get('_automation_id')
cache = deps.get_beatport_data_cache()
# Invalidate all homepage cache timestamps so endpoints re-scrape.
with cache['cache_lock']:
for key in cache['homepage']:
cache['homepage'][key]['timestamp'] = 0
cache['homepage'][key]['data'] = None
refreshed = 0
errors = []
app = deps.get_app()
with app.test_client() as client:
for idx, (_, endpoint, label) in enumerate(_BEATPORT_SECTIONS):
deps.update_progress(
automation_id,
progress=(idx / len(_BEATPORT_SECTIONS)) * 100,
phase=f'Scraping: {label}',
current_item=label,
)
try:
resp = client.get(endpoint)
if resp.status_code == 200:
refreshed += 1
deps.update_progress(
automation_id, log_line=f'{label}: cached', log_type='success',
)
else:
errors.append(label)
deps.update_progress(
automation_id,
log_line=f'{label}: HTTP {resp.status_code}',
log_type='error',
)
except Exception as e: # noqa: BLE001 — per-section best-effort
errors.append(label)
deps.update_progress(
automation_id,
log_line=f'{label}: {str(e)}',
log_type='error',
)
if idx < len(_BEATPORT_SECTIONS) - 1:
time.sleep(2)
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Refreshed {refreshed}/{len(_BEATPORT_SECTIONS)} sections',
log_type='success',
)
return {
'status': 'completed',
'refreshed': str(refreshed),
'errors': str(len(errors)),
'_manages_own_progress': True,
}

356
core/automation/handlers/personalized_pipeline.py
View file

@ -1,356 +0,0 @@
"""Personalized Playlist Pipeline automation handler.
Sibling to ``auto_playlist_pipeline`` (mirrored). Where the mirrored
pipeline runs REFRESH external sources → DISCOVER metadata → SYNC →
WISHLIST, the personalized pipeline is simpler:
SNAPSHOT → SYNC → WISHLIST
SNAPSHOT reads the persisted track list from
``PersonalizedPlaylistManager``. When ``refresh_first=True`` (config),
each playlist is refreshed BEFORE syncing — useful when the user
wants the cron to capture a fresh-each-run view (e.g. "give me a new
Hidden Gems set every night"). Default is to sync the existing
snapshot, on the assumption the user / a separate cron has already
refreshed when they wanted to.
Config schema:
{
'kinds': [
{'kind': 'hidden_gems'},
{'kind': 'time_machine', 'variant': '1980s'},
{'kind': 'seasonal_mix', 'variant': 'halloween'},
...
],
'refresh_first': bool, # default false
'skip_wishlist': bool, # default false
}
Each kind dict has at minimum ``kind``; ``variant`` is required for
kinds that need it (time_machine, genre_playlist, daily_mix,
seasonal_mix). Singleton kinds (hidden_gems, discovery_shuffle,
popular_picks, fresh_tape, archives) ignore variant.
Pipeline-running flag (``deps.state.pipeline_running``) is shared
with the mirrored pipeline so the two can't overlap. (One sync
queue, one wishlist worker — overlapping triggers would step on
each other.)"""
from __future__ import annotations
import json
import threading
import time
from typing import Any, Dict, List, Optional
from core.automation.deps import AutomationDeps
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
# Sync state key prefix so personalized syncs don't collide with
# mirrored ones (`auto_mirror_<id>`).
_SYNC_ID_PREFIX = 'auto_personalized'
def auto_personalized_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run SNAPSHOT → SYNC → WISHLIST for selected personalized playlists."""
deps.state.set_pipeline_running(True)
automation_id = config.get('_automation_id')
pipeline_start = time.time()
try:
kinds_config = config.get('kinds') or []
if not isinstance(kinds_config, list) or not kinds_config:
deps.state.set_pipeline_running(False)
return {
'status': 'error',
'error': 'No personalized playlist kinds selected',
}
refresh_first = bool(config.get('refresh_first', False))
skip_wishlist = bool(config.get('skip_wishlist', False))
manager = deps.build_personalized_manager()
deps.update_progress(
automation_id,
progress=2,
phase=f'Personalized pipeline: {len(kinds_config)} playlist(s)',
log_line=f'Starting pipeline for {len(kinds_config)} playlist(s)',
log_type='info',
)
# ── PHASE 1: SNAPSHOT (optionally refresh) ──────────────────
deps.update_progress(
automation_id,
progress=3,
phase='Phase 1/2: Loading snapshots...' if not refresh_first
else 'Phase 1/2: Refreshing snapshots...',
log_line='Phase 1: Snapshot' + (' (with refresh)' if refresh_first else ''),
log_type='info',
)
profile_id = deps.get_current_profile_id()
playload_payloads = _build_payloads_for_kinds(
deps, manager, kinds_config, profile_id,
automation_id=automation_id,
refresh_first=refresh_first,
)
if not playload_payloads:
deps.state.set_pipeline_running(False)
deps.update_progress(
automation_id,
status='finished', progress=100,
phase='No playlists to sync',
log_line='No personalized playlists had tracks to sync',
log_type='warning',
)
return {
'status': 'completed',
'_manages_own_progress': True,
'playlists_synced': '0',
'tracks_synced': '0',
'duration_seconds': str(int(time.time() - pipeline_start)),
}
deps.update_progress(
automation_id,
progress=50,
phase='Phase 1/2: Snapshot complete',
log_line=f'Phase 1 done: {len(playload_payloads)} playlist(s) ready to sync',
log_type='success',
)
# ── PHASE 2: SYNC + WISHLIST (shared helper) ────────────────
sync_summary = run_sync_and_wishlist(
deps,
automation_id,
playload_payloads,
sync_one_fn=lambda pl: _sync_personalized_playlist(deps, pl),
sync_id_for_fn=lambda pl: pl['sync_id'],
skip_wishlist=skip_wishlist,
progress_start=51,
progress_end=90,
sync_phase_label='Phase 2/2: Syncing to server...',
sync_phase_start_log='Phase 2: Sync',
wishlist_phase_label='Phase 2/2: Processing wishlist...',
wishlist_phase_start_log='Wishlist',
)
# ── COMPLETE ────────────────────────────────────────────────
duration = int(time.time() - pipeline_start)
deps.update_progress(
automation_id,
status='finished', progress=100,
phase='Pipeline complete',
log_line=f'Personalized pipeline finished in {duration // 60}m {duration % 60}s',
log_type='success',
)
deps.state.set_pipeline_running(False)
return {
'status': 'completed',
'_manages_own_progress': True,
'playlists_synced': str(len(playload_payloads)),
'tracks_synced': str(sync_summary['synced']),
'sync_skipped': str(sync_summary['skipped']),
'wishlist_queued': str(sync_summary['wishlist_queued']),
'duration_seconds': str(duration),
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into engine
deps.state.set_pipeline_running(False)
deps.update_progress(
automation_id,
status='error', progress=100,
phase='Pipeline error',
log_line=f'Personalized pipeline failed: {e}',
log_type='error',
)
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
def _build_payloads_for_kinds(
deps: AutomationDeps,
manager: Any,
kinds_config: List[Dict[str, Any]],
profile_id: int,
*,
automation_id: Optional[str],
refresh_first: bool,
) -> List[Dict[str, Any]]:
"""Resolve each requested kind+variant into a sync-payload dict.
Each payload has: ``{'name', 'kind', 'variant', 'tracks_json',
'image_url', 'sync_id'}``. Playlists with no tracks (e.g. a
seasonal mix that hasn't been populated yet) are omitted from
the result so the sync loop doesn't waste time on empty pushes.
"""
payloads: List[Dict[str, Any]] = []
for entry in kinds_config:
if not isinstance(entry, dict):
continue
kind = entry.get('kind')
variant = entry.get('variant') or ''
if not kind:
continue
try:
# Refresh when ANY of:
# - explicit user flag (cron use case: regenerate each run)
# - snapshot marked stale by upstream data refresher
# - playlist was never generated yet (auto-created by
# ensure_playlist; track_count=0, last_generated_at=NULL).
# Without this branch, a first-run pipeline reads the
# empty snapshot and silently skips — user picks a kind,
# hits run, gets "No tracks to sync" with no clue why.
if refresh_first:
record = manager.refresh_playlist(kind, variant, profile_id)
else:
existing = manager.ensure_playlist(kind, variant, profile_id)
needs_first_gen = existing.last_generated_at is None
if existing.is_stale or needs_first_gen:
record = manager.refresh_playlist(kind, variant, profile_id)
else:
record = existing
except Exception as exc: # noqa: BLE001 — log + continue with next kind
deps.update_progress(
automation_id,
log_line=f'Skipping {kind}{("/" + variant) if variant else ""}: {exc}',
log_type='warning',
)
continue
tracks = manager.get_playlist_tracks(record.id)
if not tracks:
deps.update_progress(
automation_id,
log_line=f'No tracks in {record.name} — skipping sync',
log_type='skip',
)
continue
tracks_json = [_track_to_sync_shape(t) for t in tracks]
payloads.append({
'name': record.name,
'kind': record.kind,
'variant': record.variant,
'tracks_json': tracks_json,
'image_url': '', # personalized playlists don't have a cover image yet
'sync_id': f'{_SYNC_ID_PREFIX}_{record.kind}_{record.variant or "_"}',
})
return payloads
def _track_to_sync_shape(track: Any) -> Dict[str, Any]:
"""Convert a personalized.types.Track into the dict shape
`_run_sync_task` expects. Mirrors what the mirrored pipeline
builds from extra_data.matched_data, preserving enriched metadata
from personalized snapshots when available."""
primary_id = track.spotify_track_id or track.itunes_track_id or track.deezer_track_id or ''
rich_data = _coerce_track_data_json(getattr(track, 'track_data_json', None))
if not rich_data:
album = {'name': track.album_name or ''}
cover_url = getattr(track, 'album_cover_url', None)
if cover_url:
album['images'] = [{'url': cover_url}]
return {
'name': track.track_name,
'artists': [{'name': track.artist_name}],
'album': album,
'duration_ms': int(track.duration_ms or 0),
'id': primary_id,
}
payload = dict(rich_data)
cover_url = (
getattr(track, 'album_cover_url', None)
or payload.get('album_cover_url')
or payload.get('image_url')
)
payload['id'] = payload.get('id') or primary_id
payload['name'] = payload.get('name') or track.track_name
payload['artists'] = _normalize_artists(payload.get('artists'), track.artist_name)
payload['album'] = _normalize_album(payload.get('album'), track, cover_url=cover_url)
payload['duration_ms'] = int(payload.get('duration_ms') or track.duration_ms or 0)
if 'popularity' not in payload and getattr(track, 'popularity', None) is not None:
payload['popularity'] = int(track.popularity or 0)
if cover_url and not payload.get('image_url'):
payload['image_url'] = cover_url
return payload
def _coerce_track_data_json(value: Any) -> Dict[str, Any]:
if isinstance(value, dict):
return value
if isinstance(value, str) and value.strip():
try:
loaded = json.loads(value)
except (TypeError, ValueError):
return {}
return loaded if isinstance(loaded, dict) else {}
return {}
def _normalize_artists(artists: Any, fallback_artist: str) -> List[Dict[str, Any]]:
if not artists:
return [{'name': fallback_artist or 'Unknown Artist'}]
if isinstance(artists, list):
normalized = []
for artist in artists:
if isinstance(artist, dict):
normalized.append(artist if artist.get('name') else {'name': str(artist)})
elif isinstance(artist, str):
normalized.append({'name': artist})
else:
normalized.append({'name': str(artist)})
return normalized or [{'name': fallback_artist or 'Unknown Artist'}]
if isinstance(artists, dict):
return [artists if artists.get('name') else {'name': str(artists)}]
return [{'name': str(artists)}]
def _normalize_album(album: Any, track: Any, cover_url: Optional[str] = None) -> Dict[str, Any]:
if isinstance(album, dict):
normalized = dict(album)
else:
normalized = {'name': str(album) if album else (track.album_name or '')}
normalized['name'] = normalized.get('name') or track.album_name or ''
images = normalized.get('images')
if not isinstance(images, list):
images = []
if cover_url and not images:
images = [{'url': cover_url}]
if images:
normalized['images'] = images
return normalized
def _sync_personalized_playlist(deps: AutomationDeps, payload: Dict[str, Any]) -> Dict[str, Any]:
"""Launch a personalized playlist sync via _run_sync_task on a
daemon thread + return immediately with status='started'.
Mirrors the mirrored ``auto_sync_playlist`` return contract so the
shared helper can poll on ``sync_states[sync_id]`` and aggregate
results identically."""
sync_id = payload['sync_id']
name = payload['name']
tracks_json = payload['tracks_json']
profile_id = deps.get_current_profile_id()
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, name, tracks_json, None, profile_id, payload.get('image_url', '')),
daemon=True,
name=f'auto-personalized-{sync_id}',
).start()
return {
'status': 'started',
'playlist_name': name,
'_manages_own_progress': True,
}

27
core/automation/handlers/playlist_pipeline.py
View file

@ -1,27 +0,0 @@
"""Automation adapter for the mirrored playlist pipeline.
The actual all-in-one playlist lifecycle lives in
``core.playlists.pipeline`` so it can be reused by non-automation UI actions.
This module only wires automation-specific dependencies and handlers.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.playlists.pipeline import run_mirrored_playlist_pipeline
def auto_playlist_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run REFRESH -> DISCOVER -> SYNC -> WISHLIST for mirrored playlists."""
return run_mirrored_playlist_pipeline(
config,
deps,
refresh_fn=auto_refresh_mirrored,
sync_one_fn=auto_sync_playlist,
sync_and_wishlist_fn=run_sync_and_wishlist,
)

27
core/automation/handlers/process_wishlist.py
View file

@ -1,27 +0,0 @@
"""Automation handler: ``process_wishlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_process_wishlist`` closure). Wishlist processing is async —
the helper submits a batch to an executor and returns immediately;
per-track stats arrive later via batch-completion callbacks.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_process_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Kick off the wishlist processor for an automation trigger.
Returns immediately after submission; the wishlist worker emits
per-batch progress via its own callbacks. We only report
``status: completed`` to mark the trigger fired successfully.
"""
try:
deps.process_wishlist_automatically(automation_id=config.get('_automation_id'))
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
return {'status': 'error', 'error': str(e)}

89
core/automation/handlers/progress_callbacks.py
View file

@ -1,89 +0,0 @@
"""Progress + history callbacks the automation engine invokes around
each handler run.
Lifted from the closures at the bottom of
``web_server._register_automation_handlers``:
- ``_progress_init`` → :func:`progress_init`
- ``_progress_finish`` → :func:`progress_finish`
- ``_record_automation_history`` → :func:`record_history`
- ``_on_library_scan_completed`` → :func:`on_library_scan_completed`
The engine accepts four callables via
``register_progress_callbacks(init, finish, update, history)``;
``registration.register_all`` wires these here. The
``library_scan_completed`` callback is registered separately on the
``web_scan_manager`` (when one is available) -- see
``register_library_scan_completed_emitter``.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def progress_init(aid: Any, name: str, action_type: str, deps: AutomationDeps) -> None:
"""Initialize per-automation progress state when the engine starts
a handler. Thin wrapper so the engine receives a closure that
delegates into the live progress tracker."""
deps.init_automation_progress(aid, name, action_type)
def progress_finish(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
"""Emit the final progress update when a handler returns.
Skipped for handlers that manage their own progress lifecycle
(they call ``update_progress(status='finished')`` themselves and
set ``_manages_own_progress: True`` in the returned dict).
Otherwise translates the handler's status into a finished/error
progress emit with a status-appropriate phase + log line.
"""
if result.get('_manages_own_progress'):
return
result_status = result.get('status', '')
status = 'error' if result_status == 'error' else 'finished'
msg = result.get('error', result.get('reason', result_status or 'done'))
deps.update_progress(
aid,
status=status,
progress=100,
phase='Error' if status == 'error' else 'Complete',
log_line=msg,
log_type='error' if status == 'error' else 'success',
)
def record_history(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
"""Capture progress state into run history before the engine's
cleanup pass clears it. Thin wrapper so the engine sees a stable
callable."""
deps.record_progress_history(aid, result, deps.get_database())
def on_library_scan_completed(deps: AutomationDeps) -> None:
"""Emit the ``library_scan_completed`` automation event with the
active media-server type. Replaces the hard-coded
``scan_completion_callback → trigger_automatic_database_update``
chain so any automation can listen for scan completion as a
trigger."""
if not deps.engine:
return
server_type = (
getattr(deps.web_scan_manager, '_current_server_type', None)
or 'unknown'
)
deps.engine.emit('library_scan_completed', {
'server_type': server_type,
})
def register_library_scan_completed_emitter(deps: AutomationDeps) -> None:
"""Wire :func:`on_library_scan_completed` to the
``web_scan_manager``'s scan-completion callback list. No-op when
no scan manager is configured (e.g. headless / test contexts)."""
if not deps.web_scan_manager:
return
deps.web_scan_manager.add_scan_completion_callback(
lambda: on_library_scan_completed(deps),
)

35
core/automation/handlers/quality_scanner.py
View file

@ -1,35 +0,0 @@
"""Automation handler: ``start_quality_scan`` action.
The quality scanner was redesigned from an auto-acting tool into the
``quality_upgrade`` library-maintenance repair job (findings-based, reviewed
before anything is wishlisted). This action now simply triggers a "Run Now" of
that job; its progress and findings surface in Library Maintenance. The action
name is kept so existing automation rules keep working.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_start_quality_scan(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
automation_id = config.get('_automation_id')
triggered = deps.run_repair_job_now('quality_upgrade')
if not triggered:
deps.update_progress(
automation_id, status='error', phase='Unavailable',
log_line='Quality Upgrade job could not be triggered (library worker unavailable)',
log_type='error',
)
return {'status': 'error', 'reason': 'library worker unavailable',
'_manages_own_progress': True}
deps.update_progress(
automation_id, status='finished', progress=100, phase='Triggered',
log_line='Quality Upgrade scan queued — findings appear in Library Maintenance',
log_type='success',
)
return {'status': 'completed', 'triggered': True, '_manages_own_progress': True}

341
core/automation/handlers/refresh_mirrored.py
View file

@ -1,341 +0,0 @@
"""Automation handler: ``refresh_mirrored`` action.
Re-pulls track lists from each mirrored playlist's source via the
unified ``PlaylistSourceRegistry`` (Phase 1 of the Discover-to-Sync
unification). The pre-extraction handler had ~190 lines of per-source
if/elif branches; this version delegates to the adapter for each
source, leaving the handler responsible only for:
- filtering sources that can't be refreshed (``file``, ``beatport``),
- extracting upstream URLs from the stored ``description`` for URL-
backed sources (``spotify_public``, ``youtube``),
- the Spotify-public → authenticated-Spotify fallback (uses the
``spotify`` adapter when the user is signed in so the mirror keeps
album art),
- the Tidal-not-authenticated skip log type (vs error),
- preserving existing per-track ``extra_data`` on tracks that survive
the refresh, and
- emitting the ``playlist_changed`` automation event when the track
set actually shifts.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
from core.automation.deps import AutomationDeps
from core.playlists.source_refs import require_refresh_url
from core.playlists.sources import PlaylistDetail, to_mirror_track_dict
from core.playlists.sources.base import (
SOURCE_SPOTIFY,
SOURCE_SPOTIFY_PUBLIC,
SOURCE_TIDAL,
SOURCE_YOUTUBE,
)
# Sources that store the upstream URL in ``description`` (because their
# ``source_playlist_id`` is a deterministic hash, not the native ID).
# The refresh path has to recover the URL before calling the adapter.
_URL_BACKED_SOURCES = {SOURCE_SPOTIFY_PUBLIC, SOURCE_YOUTUBE}
def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Refresh mirrored playlist(s) from source.
Returns ``{'status': 'completed', 'refreshed': '<int>',
'errors': '<int>'}`` on success (counts stringified to match the
automation engine's stat-rendering convention).
"""
db = deps.get_database()
playlist_id = config.get('playlist_id')
refresh_all = config.get('all', False)
auto_id = config.get('_automation_id')
# Pipeline runs (``run_mirrored_playlist_pipeline``) set this flag
# because Phase 2 of the pipeline already runs the playlist
# discovery worker — the same matching engine ``_maybe_discover``
# would call here. Running both means LB tracks discover twice
# AND the refresh-side discovery blocks 5+ minutes with no
# progress emission, leaving the UI stuck on "Refreshing:" until
# the loop returns. Standalone callers (Sync page, registration
# action) leave it False so LB tracks still get matched_data on
# refresh.
skip_discovery = bool(config.get('skip_discovery', False))
if refresh_all:
playlists = db.get_mirrored_playlists()
elif playlist_id:
p = db.get_mirrored_playlist(int(playlist_id))
playlists = [p] if p else []
else:
return {'status': 'error', 'reason': 'No playlist specified'}
# Filter out sources that can't be refreshed (no external API).
playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
refreshed = 0
errors: List[str] = []
for idx, pl in enumerate(playlists):
try:
source = pl.get('source', '')
source_id = pl.get('source_playlist_id', '')
deps.update_progress(
auto_id,
progress=(idx / max(1, len(playlists))) * 100,
phase=f'Refreshing: "{pl.get("name", "")}"',
current_item=pl.get('name', ''),
)
detail = _fetch_detail(source, source_id, pl, deps, auto_id)
if detail is None:
# _fetch_detail already logged the specific failure;
# mark the playlist as a generic refresh error so the
# automation result tally matches the legacy handler.
errors.append(f"{pl.get('name', '?')}: no tracks returned from source")
deps.update_progress(
auto_id,
log_line=f'Refresh failed: "{pl.get("name", "")}" - no tracks returned from source',
log_type='error',
)
continue
# Sources that return MB-metadata-only tracks (LB, Last.fm)
# mark them ``needs_discovery=True``. Hand them to the
# adapter's matcher so the resulting mirror rows carry
# provider IDs + matched_data, ready for the sync pipeline.
#
# Pipeline runs skip this because Phase 2's discovery
# worker handles it with proper progress emission — see
# ``skip_discovery`` resolution at the top of this fn.
detail_tracks = (
detail.tracks if skip_discovery
else _maybe_discover(detail.tracks, source, deps)
)
tracks = [to_mirror_track_dict(t) for t in detail_tracks]
refreshed += _commit_refresh(pl, source, source_id, tracks, db, deps, auto_id)
except _SkipPlaylist:
# Source-specific soft-skip (e.g. Tidal not authenticated).
# Logging was already emitted; do not count as error.
continue
except Exception as e:
errors.append(f"{pl.get('name', '?')}: {str(e)}")
deps.update_progress(
auto_id,
log_line=f'Error: {pl.get("name", "?")} — {str(e)}',
log_type='error',
)
return {'status': 'completed', 'refreshed': str(refreshed), 'errors': str(len(errors))}
def _maybe_discover(
tracks: List[Any],
source: str,
deps: AutomationDeps,
) -> List[Any]:
"""Run the adapter's ``discover_tracks`` when any track needs it.
Most sources are no-ops here (their tracks have provider IDs
already). LB / Last.fm are the ones that actually do work."""
if not tracks:
return tracks
if not any(getattr(t, "needs_discovery", False) for t in tracks):
return tracks
registry = deps.playlist_source_registry
if registry is None:
return tracks
adapter = registry.get_source(source)
if adapter is None:
return tracks
try:
return adapter.discover_tracks(tracks)
except Exception as exc:
deps.logger.warning(f"{source} discover_tracks failed: {exc}")
return tracks
class _SkipPlaylist(Exception):
"""Internal sentinel: source-specific soft-skip (e.g. not authed).
The per-playlist loop catches it specifically so the skip isn't
counted in the error tally — matches the pre-extraction behavior
where ``continue`` was used inline."""
def _fetch_detail(
source: str,
source_id: str,
pl: Dict[str, Any],
deps: AutomationDeps,
auto_id: Optional[str],
) -> Optional[PlaylistDetail]:
"""Resolve the playlist's tracks through the registry.
Handler-level branches (URL extraction, Spotify-public→authed
fallback, Tidal not-authed skip) live here; everything else
delegates to the adapter."""
registry = deps.playlist_source_registry
if registry is None:
return None
# URL-backed sources: pull the upstream URL out of `description`.
playlist_input = source_id
if source in _URL_BACKED_SOURCES:
# ``require_refresh_url`` raises ValueError on missing URL.
# The outer try/except in the loop catches it and reports as
# an error — matching the pre-extraction behavior.
playlist_input = require_refresh_url(
source, pl.get('description', ''), pl.get('name', '')
)
# Spotify-public refresh: prefer the authenticated Spotify API
# when the user is signed in. Better album art, matches the
# pre-extraction handler. Falls through to the public scraper on
# auth failure or non-playlist URL types (e.g. album URLs).
if source == SOURCE_SPOTIFY_PUBLIC:
detail = _try_spotify_authed_for_public(playlist_input, deps)
if detail is not None:
return detail
# Tidal not-authed: soft-skip with a 'skip' log line, not an error.
if source == SOURCE_TIDAL:
tidal_source = registry.get_source(SOURCE_TIDAL)
if tidal_source is None or not tidal_source.is_authenticated():
deps.logger.warning(
f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'"
)
deps.update_progress(
auto_id,
log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
log_type='skip',
)
raise _SkipPlaylist
adapter = registry.get_source(source)
if adapter is None:
return None
try:
return adapter.refresh_playlist(playlist_input)
except Exception as exc:
deps.logger.warning(
f"{source} playlist refresh failed for {playlist_input}: {exc}"
)
return None
def _try_spotify_authed_for_public(
spotify_url: str, deps: AutomationDeps
) -> Optional[PlaylistDetail]:
"""Best-effort: use the authenticated Spotify adapter on a public URL.
Returns ``None`` to signal "fall through to the public-scraper
adapter" — never raises. Only applies to ``playlist``-type URLs;
album URLs fall through unconditionally."""
if not spotify_url:
return None
spotify_client = deps.spotify_client
if spotify_client is None or not spotify_client.is_spotify_authenticated():
return None
try:
from core.spotify_public_scraper import parse_spotify_url
parsed = parse_spotify_url(spotify_url)
except Exception:
return None
if not parsed or parsed.get('type') != 'playlist':
return None
adapter = deps.playlist_source_registry.get_source(SOURCE_SPOTIFY)
if adapter is None:
return None
try:
return adapter.refresh_playlist(parsed['id'])
except Exception as exc:
deps.logger.debug(f"Spotify authed fallback for public mirror failed: {exc}")
return None
def _commit_refresh(
pl: Dict[str, Any],
source: str,
source_id: str,
tracks: List[Dict[str, Any]],
db: Any,
deps: AutomationDeps,
auto_id: Optional[str],
) -> int:
"""Persist the refreshed track list + emit playlist_changed when delta.
Returns 1 when a refresh successfully landed, 0 otherwise. The
caller is responsible for incrementing the running tally."""
old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
# Preserve existing extra_data (matched_data + discovery state)
# for tracks that still exist in the refreshed snapshot, unless
# the adapter already provided fresh extra_data for that track.
old_extra_map = (
db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
)
for t in tracks:
sid = t.get('source_track_id', '')
if sid and sid in old_extra_map and 'extra_data' not in t:
t['extra_data'] = old_extra_map[sid]
db.mirror_playlist(
source=source,
source_playlist_id=source_id,
name=pl['name'],
tracks=tracks,
profile_id=pl.get('profile_id', 1),
description=pl.get('description'),
owner=pl.get('owner'),
image_url=pl.get('image_url'),
)
# Membership just changed — if this playlist is organize-by-playlist, rebuild
# its folder (with prune) so a track that LEFT the playlist has its symlink
# cleaned up now. Gated to organized playlists, non-fatal — never disturbs
# the refresh. (Additions are handled by the post-download reconcile.)
try:
from core.playlists.materialize_service import rebuild_mirrored_playlist_if_organized
rebuild_mirrored_playlist_if_organized(
db, deps.config_manager, pl.get('id'), profile_id=pl.get('profile_id', 1)
)
except Exception as _mat_err:
deps.logger.debug(f"[Playlist Folder] mirror-refresh cleanup skipped: {_mat_err}")
if old_ids != new_ids:
added = len(new_ids - old_ids)
removed = len(old_ids - new_ids)
deps.logger.info(
f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
f"{added} added, {removed} removed (old={len(old_ids)}, new={len(new_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'"{pl.get("name", "")}" — {added} added, {removed} removed',
log_type='success',
)
try:
if deps.engine:
deps.engine.emit('playlist_changed', {
'playlist_name': pl.get('name', ''),
'playlist_id': str(pl.get('id', '')),
'old_count': str(len(old_ids)),
'new_count': str(len(new_ids)),
'added': str(added),
'removed': str(removed),
})
except Exception as e:
deps.logger.debug("playlist_synced automation emit failed: %s", e)
else:
deps.logger.warning(
f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'No changes: "{pl.get("name", "")}"',
log_type='skip',
)
return 1

184
core/automation/handlers/registration.py
View file

@ -1,184 +0,0 @@
"""One-stop registration of every extracted automation handler.
``web_server`` builds the deps once at startup and calls
:func:`register_all` here. Each new handler module gets one line in
this file when it lands.
"""
from __future__ import annotations
from core.automation.deps import AutomationDeps
from core.automation.handlers.process_wishlist import auto_process_wishlist
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
from core.automation.handlers.scan_library import auto_scan_library
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.automation.handlers.discover_playlist import auto_discover_playlist
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
from core.automation.handlers.database_update import (
auto_start_database_update, auto_deep_scan_library,
)
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
from core.automation.handlers.quality_scanner import auto_start_quality_scan
from core.automation.handlers.maintenance import (
auto_clear_quarantine,
auto_cleanup_wishlist,
auto_update_discovery_pool,
auto_backup_database,
auto_refresh_beatport_cache,
)
from core.automation.handlers.download_cleanup import (
auto_clean_search_history,
auto_clean_completed_downloads,
auto_full_cleanup,
)
from core.automation.handlers.run_script import auto_run_script
from core.automation.handlers.search_and_download import auto_search_and_download
from core.automation.handlers.progress_callbacks import (
progress_init,
progress_finish,
record_history,
register_library_scan_completed_emitter,
)
def register_all(deps: AutomationDeps) -> None:
"""Wire every extracted handler to the engine.
Each ``register_action_handler`` call binds the action name (the
string the trigger uses to look up its action) to a thin lambda
that injects ``deps`` and forwards the engine-supplied config.
Guards stay alongside their handler so duplicate-run prevention
behaves identically to the pre-extraction code.
"""
engine = deps.engine
# Self-guards prevent duplicate runs of the SAME operation, but
# different operations can run concurrently — wishlist downloads
# use bandwidth, watchlist scans use API calls, library scans use
# media-server CPU. Different resources, no contention.
engine.register_action_handler(
'process_wishlist',
lambda config: auto_process_wishlist(config, deps),
guard_fn=deps.is_wishlist_actually_processing,
)
engine.register_action_handler(
'scan_watchlist',
lambda config: auto_scan_watchlist(config, deps),
guard_fn=deps.is_watchlist_actually_scanning,
)
engine.register_action_handler(
'scan_library',
lambda config: auto_scan_library(config, deps),
deps.state.is_scan_library_active,
)
# Playlist lifecycle handlers. The pipeline composes refresh +
# sync + discover (it imports them directly), so all four ship
# together. The pipeline guard prevents an in-flight pipeline
# from being re-triggered mid-run.
engine.register_action_handler(
'refresh_mirrored',
lambda config: auto_refresh_mirrored(config, deps),
)
engine.register_action_handler(
'sync_playlist',
lambda config: auto_sync_playlist(config, deps),
)
engine.register_action_handler(
'discover_playlist',
lambda config: auto_discover_playlist(config, deps),
)
engine.register_action_handler(
'playlist_pipeline',
lambda config: auto_playlist_pipeline(config, deps),
deps.state.is_pipeline_running,
)
# Personalized pipeline shares the pipeline_running flag with the
# mirrored pipeline so the two can't overlap (single sync queue,
# single wishlist worker).
engine.register_action_handler(
'personalized_pipeline',
lambda config: auto_personalized_pipeline(config, deps),
deps.state.is_pipeline_running,
)
# Database update + deep scan share the db_update_state guard —
# only one operation can mutate that state at a time.
engine.register_action_handler(
'start_database_update',
lambda config: auto_start_database_update(config, deps),
lambda: deps.get_db_update_state().get('status') == 'running',
)
engine.register_action_handler(
'deep_scan_library',
lambda config: auto_deep_scan_library(config, deps),
lambda: deps.get_db_update_state().get('status') == 'running',
)
engine.register_action_handler(
'run_duplicate_cleaner',
lambda config: auto_run_duplicate_cleaner(config, deps),
lambda: deps.get_duplicate_cleaner_state().get('status') == 'running',
)
engine.register_action_handler(
'clear_quarantine',
lambda config: auto_clear_quarantine(config, deps),
)
engine.register_action_handler(
'cleanup_wishlist',
lambda config: auto_cleanup_wishlist(config, deps),
)
engine.register_action_handler(
'update_discovery_pool',
lambda config: auto_update_discovery_pool(config, deps),
)
engine.register_action_handler(
'start_quality_scan',
lambda config: auto_start_quality_scan(config, deps),
lambda: False, # repair worker dedupes Run-Now requests itself
)
engine.register_action_handler(
'backup_database',
lambda config: auto_backup_database(config, deps),
)
engine.register_action_handler(
'refresh_beatport_cache',
lambda config: auto_refresh_beatport_cache(config, deps),
)
engine.register_action_handler(
'clean_search_history',
lambda config: auto_clean_search_history(config, deps),
)
engine.register_action_handler(
'clean_completed_downloads',
lambda config: auto_clean_completed_downloads(config, deps),
)
engine.register_action_handler(
'full_cleanup',
lambda config: auto_full_cleanup(config, deps),
)
engine.register_action_handler(
'run_script',
lambda config: auto_run_script(config, deps),
)
engine.register_action_handler(
'search_and_download',
lambda config: auto_search_and_download(config, deps),
)
# Progress + history callbacks: the engine invokes these around
# each handler run. Lift the closures from
# `web_server._register_automation_handlers` into thin lambdas
# that delegate into the extracted top-level functions.
engine.register_progress_callbacks(
lambda aid, name, action_type: progress_init(aid, name, action_type, deps),
lambda aid, result: progress_finish(aid, result, deps),
deps.update_progress,
lambda aid, result: record_history(aid, result, deps),
)
# `library_scan_completed` event: when the media-server scan
# manager finishes a scan, emit the event so any automation can
# trigger off it. No-op when no scan manager is configured.
register_library_scan_completed_emitter(deps)

103
core/automation/handlers/run_script.py
View file

@ -1,103 +0,0 @@
"""Automation handler: ``run_script`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_run_script`` closure). Runs a user-provided shell or Python
script from the configured scripts directory with bounded timeout +
captured stdout/stderr. Path-traversal guard ensures users can't
escape the scripts directory.
Environment variables exposed to the script:
- ``SOULSYNC_EVENT``: triggering event type (when fired by an event)
- ``SOULSYNC_AUTOMATION``: automation name
- ``SOULSYNC_SCRIPTS_DIR``: absolute path to the scripts dir
"""
from __future__ import annotations
import os
import subprocess as _sp
from typing import Any, Dict
from core.automation.deps import AutomationDeps
_MAX_TIMEOUT_SECONDS = 300 # Hard cap on user-supplied timeout config.
def auto_run_script(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
script_name = config.get('script_name', '')
timeout = min(int(config.get('timeout', 60)), _MAX_TIMEOUT_SECONDS)
automation_id = config.get('_automation_id')
if not script_name:
return {'status': 'error', 'error': 'No script selected'}
scripts_dir = deps.docker_resolve_path(deps.config_manager.get('scripts.path', './scripts'))
if not scripts_dir or not os.path.isdir(scripts_dir):
os.makedirs(scripts_dir, exist_ok=True)
return {
'status': 'error',
'error': 'Scripts directory is empty. Add scripts to the scripts/ folder.',
}
script_path = os.path.join(scripts_dir, script_name)
script_path = os.path.realpath(script_path)
# Security: block path traversal — script must resolve under
# the scripts dir, no symlinks/.. tricks allowed out.
if not script_path.startswith(os.path.realpath(scripts_dir)):
return {'status': 'error', 'error': 'Script path traversal blocked'}
if not os.path.isfile(script_path):
return {'status': 'error', 'error': f'Script not found: {script_name}'}
deps.update_progress(automation_id, phase=f'Running {script_name}...', progress=10)
# Build environment with SoulSync context.
env = os.environ.copy()
event_data = config.get('_event_data') or {}
env['SOULSYNC_EVENT'] = str(event_data.get('type', ''))
env['SOULSYNC_AUTOMATION'] = config.get('_automation_name', '')
env['SOULSYNC_SCRIPTS_DIR'] = scripts_dir
try:
# Determine how to run the script.
if script_path.endswith('.py'):
cmd = ['python', script_path]
elif script_path.endswith('.sh'):
cmd = ['bash', script_path]
else:
cmd = [script_path]
result = _sp.run(
cmd,
capture_output=True, text=True, timeout=timeout,
cwd=scripts_dir, env=env,
)
deps.update_progress(automation_id, phase='Script completed', progress=100)
stdout = result.stdout[:2000] if result.stdout else ''
stderr = result.stderr[:1000] if result.stderr else ''
if result.returncode == 0:
deps.logger.info(f"Script '{script_name}' completed (exit 0)")
else:
deps.logger.warning(f"Script '{script_name}' exited with code {result.returncode}")
return {
'status': 'completed' if result.returncode == 0 else 'error',
'exit_code': str(result.returncode),
'stdout': stdout,
'stderr': stderr,
'script': script_name,
}
except _sp.TimeoutExpired:
deps.update_progress(automation_id, phase='Script timed out', progress=100)
return {
'status': 'error',
'error': f'Script timed out after {timeout}s',
'script': script_name,
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'error': str(e), 'script': script_name}

158
core/automation/handlers/scan_library.py
View file

@ -1,158 +0,0 @@
"""Automation handler: ``scan_library`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_scan_library`` closure). The handler triggers a media-server
scan via ``web_scan_manager``, then polls the manager's status until
the scan completes (or a 30-minute timeout fires). Progress phases
are emitted via :func:`AutomationDeps.update_progress` so the
trigger card stays current throughout the run.
The handler manages its own progress reporting (it sets
``_manages_own_progress: True`` in the result) so the engine doesn't
overwrite the live phase string with a generic 'completed' label.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# Outer poll cap — covers extreme worst case (long Plex scans on
# huge libraries). Past this point we surface a clear timeout error
# so users notice rather than letting the trigger hang forever.
_SCAN_TIMEOUT_SECONDS = 1800
# Per-phase poll intervals.
_POLL_SCHEDULED_SECONDS = 2
_POLL_SCANNING_SECONDS = 5
_POLL_UNKNOWN_SECONDS = 2
# Progress percentage waypoints.
_PROGRESS_SCHEDULED_MAX = 14
_PROGRESS_SCAN_START = 15
_PROGRESS_SCAN_MAX = 95
def auto_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a media-server library scan and stream progress to the
trigger card.
Returns one of:
- ``{'status': 'completed', '_manages_own_progress': True, ...}``
- ``{'status': 'skipped', 'reason': 'Scan already being tracked'}``
- ``{'status': 'error', 'reason': '...', '_manages_own_progress': True}``
"""
automation_id = config.get('_automation_id')
if not deps.web_scan_manager:
return {'status': 'error', 'reason': 'Scan manager not available'}
# If another automation is already tracking the scan, just forward
# the request — the original tracker keeps emitting progress.
if deps.state.is_scan_library_active():
deps.web_scan_manager.request_scan('Automation trigger (additional batch)')
return {'status': 'skipped', 'reason': 'Scan already being tracked'}
deps.state.set_scan_library_id(automation_id)
try:
result = deps.web_scan_manager.request_scan('Automation trigger')
scan_status_val = result.get('status', 'unknown')
if scan_status_val == 'queued':
deps.update_progress(
automation_id,
log_line='Scan already in progress — waiting for completion',
log_type='info',
)
else:
delay = result.get('delay_seconds', 60)
deps.update_progress(
automation_id,
log_line=f'Scan scheduled (debounce: {delay}s)',
log_type='info',
)
# Unified polling loop — handles debounce → scanning → idle.
poll_start = time.time()
scan_started = (scan_status_val == 'queued')
while time.time() - poll_start < _SCAN_TIMEOUT_SECONDS:
status = deps.web_scan_manager.get_scan_status()
st = status.get('status')
if st == 'idle':
break # Scan completed (or finished before we polled)
if st == 'scheduled':
elapsed = int(time.time() - poll_start)
deps.update_progress(
automation_id,
phase=f'Waiting for scan to start... ({elapsed}s)',
progress=min(int(elapsed / 60 * 10), _PROGRESS_SCHEDULED_MAX),
)
time.sleep(_POLL_SCHEDULED_SECONDS)
continue
if st == 'scanning':
if not scan_started:
scan_started = True
deps.update_progress(
automation_id,
progress=_PROGRESS_SCAN_START,
log_line='Scan triggered on media server',
log_type='success',
)
elapsed = status.get('elapsed_seconds', 0)
max_time = status.get('max_time_seconds', 300)
pct = min(_PROGRESS_SCAN_START + int(elapsed / max_time * 80), _PROGRESS_SCAN_MAX)
mins, secs = divmod(elapsed, 60)
deps.update_progress(
automation_id,
phase=f'Library scan in progress... ({mins}m {secs}s)',
progress=pct,
)
time.sleep(_POLL_SCANNING_SECONDS)
continue
time.sleep(_POLL_UNKNOWN_SECONDS)
else:
# 30-min timeout reached
deps.update_progress(
automation_id,
status='error',
phase='Timed out',
log_line='Library scan timed out after 30 minutes',
log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
elapsed = round(time.time() - poll_start, 1)
deps.update_progress(
automation_id,
status='finished',
progress=100,
phase='Complete',
log_line='Library scan completed',
log_type='success',
)
return {
'status': 'completed',
'_manages_own_progress': True,
'scan_duration_seconds': elapsed,
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
deps.update_progress(
automation_id,
status='error',
phase='Error',
log_line=str(e),
log_type='error',
)
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
finally:
deps.state.set_scan_library_id(None)

46
core/automation/handlers/scan_watchlist.py
View file

@ -1,46 +0,0 @@
"""Automation handler: ``scan_watchlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_scan_watchlist`` closure). The watchlist scanner returns
summary stats for the trigger card only when a fresh scan actually
ran — detected by snapshotting ``id(state_dict)`` before/after, since
the live processor reassigns the dict on each new scan.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_scan_watchlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a watchlist scan when the automation triggers.
Pre-scan we capture ``id(watchlist_scan_state)`` so we can tell
afterwards whether the worker ran (and reassigned the state dict)
or short-circuited (kept the same dict). Only fresh scans report
summary stats — repeat triggers without an intervening run return
a bare ``completed``.
"""
try:
pre_state = deps.get_watchlist_scan_state()
pre_state_id = id(pre_state)
deps.process_watchlist_scan_automatically(
automation_id=config.get('_automation_id'),
profile_id=config.get('_profile_id'),
)
post_state = deps.get_watchlist_scan_state()
# Fresh scan = state dict was reassigned mid-run.
if id(post_state) != pre_state_id:
summary = post_state.get('summary', {}) if isinstance(post_state, dict) else {}
return {
'status': 'completed',
'artists_scanned': summary.get('total_artists', 0),
'successful_scans': summary.get('successful_scans', 0),
'new_tracks_found': summary.get('new_tracks_found', 0),
'tracks_added_to_wishlist': summary.get('tracks_added_to_wishlist', 0),
}
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
return {'status': 'error', 'error': str(e)}

57
core/automation/handlers/search_and_download.py
View file

@ -1,57 +0,0 @@
"""Automation handler: ``search_and_download`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_search_and_download`` closure). Searches for a track by
name/artist string and dispatches the best match through the
download orchestrator. Query can come from the trigger config
(direct value) or from event data (e.g. webhook payload).
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_search_and_download(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
automation_id = config.get('_automation_id')
query = config.get('query', '').strip()
# Event-triggered: pull query from event data (e.g. webhook_received).
if not query:
event_data = config.get('_event_data', {})
query = (event_data.get('query', '') or '').strip()
if not query:
if automation_id:
deps.update_progress(
automation_id, log_line='No search query provided', log_type='error',
)
return {'status': 'error', 'error': 'No search query provided'}
try:
if automation_id:
deps.update_progress(
automation_id, phase='Searching',
log_line=f'Searching: {query}', log_type='info',
)
result = deps.run_async(deps.download_orchestrator.search_and_download_best(query))
if result:
if automation_id:
deps.update_progress(
automation_id,
log_line=f'Download started for: {query}',
log_type='success',
)
return {'status': 'completed', 'query': query, 'download_id': result}
if automation_id:
deps.update_progress(
automation_id,
log_line=f'No match found for: {query}',
log_type='warning',
)
return {'status': 'not_found', 'query': query, 'error': 'No match found'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
if automation_id:
deps.update_progress(
automation_id, log_line=f'Error: {e}', log_type='error',
)
return {'status': 'error', 'query': query, 'error': str(e)}

240
core/automation/handlers/sync_playlist.py
View file

@ -1,240 +0,0 @@
"""Automation handler: ``sync_playlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_sync_playlist`` closure). Syncs a mirrored playlist to the
configured media server, using discovered metadata when available
and skipping undiscovered tracks. When triggered on a schedule with
no track changes since the last sync, short-circuits with
``status: skipped`` (saves Plex / Jellyfin / Navidrome from
needless rewrites)."""
from __future__ import annotations
import hashlib
import json
import threading
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Sync a mirrored playlist to the active media server.
Behavior:
- Tracks with discovered metadata (extra_data.discovered + matched_data)
are routed via the official metadata.
- Tracks with a Spotify hint (real Spotify ID from the embed
scraper) are included so they can still hit Soulseek + the
wishlist.
- Tracks with neither are counted as ``skipped_tracks``.
- Empty result → ``status: skipped`` with the skipped count.
- Same track set as last sync (matched_tracks unchanged) →
``status: skipped`` (no-op).
- Otherwise spawns a daemon thread running ``run_sync_task`` and
returns ``status: started`` with ``_manages_own_progress: True``.
"""
auto_id = config.get('_automation_id')
playlist_id = config.get('playlist_id')
if not playlist_id:
return {'status': 'error', 'reason': 'No playlist specified'}
db = deps.get_database()
pl = db.get_mirrored_playlist(int(playlist_id))
if not pl:
return {'status': 'error', 'reason': 'Playlist not found'}
tracks = db.get_mirrored_playlist_tracks(int(playlist_id))
if not tracks:
return {'status': 'error', 'reason': 'No tracks in playlist'}
# Convert mirrored tracks to format expected by run_sync_task.
# Use discovered metadata when available, fall back to Spotify
# hint or raw playlist fields when not.
tracks_json = []
skipped_count = 0
for t in tracks:
# Parse extra_data for discovery info.
extra = {}
if t.get('extra_data'):
try:
extra = json.loads(t['extra_data']) if isinstance(t['extra_data'], str) else t['extra_data']
except (json.JSONDecodeError, TypeError):
pass
if extra.get('discovered') and extra.get('matched_data'):
# Use official discovered metadata.
md = extra['matched_data']
album_raw = md.get('album', '')
album_obj = album_raw if isinstance(album_raw, dict) else {'name': album_raw or ''}
_track_entry = {
'name': md.get('name', ''),
'artists': md.get('artists', [{'name': t.get('artist_name', '')}]),
'album': album_obj,
'duration_ms': md.get('duration_ms', 0),
'id': md.get('id', ''),
}
if md.get('track_number'):
_track_entry['track_number'] = md['track_number']
if md.get('disc_number'):
_track_entry['disc_number'] = md['disc_number']
tracks_json.append(_track_entry)
else:
# NOT discovered — try to include using available metadata so
# the track can still be searched on Soulseek and added to
# wishlist. Without this, failed discovery blocks the entire
# download pipeline.
#
# Priority: spotify_hint (has real Spotify ID from embed
# scraper) > raw playlist fields (only if source_track_id
# is valid).
hint = extra.get('spotify_hint', {})
# Build album object with cover art from the mirrored playlist track.
track_image = (t.get('image_url') or '').strip()
album_obj = {
'name': (t.get('album_name') or '').strip(),
'images': [{'url': track_image, 'height': 300, 'width': 300}] if track_image else [],
}
if hint.get('id') and hint.get('name'):
# spotify_hint has proper Spotify track ID + metadata from embed scraper.
hint_artists = hint.get('artists', [])
if hint_artists and isinstance(hint_artists[0], str):
hint_artists = [{'name': a} for a in hint_artists]
elif hint_artists and isinstance(hint_artists[0], dict):
pass # Already in correct format
else:
hint_artists = [{'name': t.get('artist_name', '')}]
tracks_json.append({
'name': hint['name'],
'artists': hint_artists,
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': hint['id'],
})
elif t.get('source_track_id') and (t.get('track_name') or '').strip():
# Has a valid source ID and track name — usable for wishlist.
tracks_json.append({
'name': t['track_name'].strip(),
'artists': [{'name': (t.get('artist_name') or '').strip() or 'Unknown Artist'}],
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': t['source_track_id'],
})
else:
skipped_count += 1 # No usable ID or name — truly can't process.
if not tracks_json:
deps.update_progress(
auto_id,
log_line=f'No discovered tracks — {skipped_count} need discovery first',
log_type='skip',
)
return {
'status': 'skipped',
'reason': f'No discovered tracks to sync ({skipped_count} tracks need discovery first)',
'skipped_tracks': str(skipped_count),
}
# Preflight: hash the track list and compare against last sync.
# Skip if the exact same set of tracks was already synced and
# everything matched (no-op preserves Plex / Jellyfin / Navidrome
# from needless rewrites).
track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
sync_id_key = f"auto_mirror_{playlist_id}"
# Full mirror identity (every source_track_id on the playlist). tracks_hash
# only covers tracks_json — if a new mirror row is skipped (no discovery /
# no source id), tracks_hash stays identical to the pre-add sync and we
# used to no-op with "unchanged" while the new song never hit wishlist.
mirror_ids_str = ','.join(
sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
)
mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
event_data = config.get('_event_data') or {}
try:
tracks_added = int(event_data.get('added') or 0)
except (TypeError, ValueError):
tracks_added = 0
force_sync = tracks_added > 0 or skipped_count > 0
try:
sync_statuses = deps.load_sync_status_file()
last_status = sync_statuses.get(sync_id_key, {})
last_hash = last_status.get('tracks_hash', '')
last_mirror_hash = last_status.get('mirror_tracks_hash', '')
last_matched = last_status.get('matched_tracks', -1)
mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
if (
not force_sync
and not mirror_changed
and last_hash == tracks_hash
and last_matched >= len(tracks_json)
):
# Exact same tracks, all matched last time — nothing to do.
deps.update_progress(
auto_id,
log_line=f'All {len(tracks_json)} tracks unchanged since last sync — skipping',
log_type='skip',
)
return {
'status': 'skipped',
'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
}
if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
deps.update_progress(
auto_id,
log_line=(
f'Forcing sync: playlist changed ({tracks_added} added) or '
f'{skipped_count} track(s) need discovery'
),
log_type='info',
)
elif mirror_changed:
deps.update_progress(
auto_id,
log_line='Mirror track list changed — running sync',
log_type='info',
)
except Exception as e:
deps.logger.debug("mirror sync last-status read: %s", e)
# Sync under the user's custom alias when set, else the upstream name (#865
# follow-up). The server-side playlist is named with this.
from core.playlists.naming import effective_mirrored_name
sync_name = effective_mirrored_name(pl) or pl.get('name') or 'Playlist'
deps.update_progress(
auto_id,
progress=50,
phase=f'Syncing "{sync_name}"',
log_line=f'{len(tracks_json)} discovered, {skipped_count} skipped',
log_type='info',
)
sync_id = f"auto_mirror_{playlist_id}"
deps.update_progress(
auto_id,
progress=90,
log_line=f'Starting sync: {len(tracks_json)} tracks',
log_type='success',
)
skip_wishlist_add = bool(pl.get('organize_by_playlist'))
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, sync_name, tracks_json, auto_id, 1, pl.get('image_url', '')),
kwargs={'skip_wishlist_add': skip_wishlist_add},
daemon=True,
name=f'auto-sync-{playlist_id}',
).start()
return {
'status': 'started',
'playlist_name': sync_name,
'discovered_tracks': str(len(tracks_json)),
'skipped_tracks': str(skipped_count),
'_manages_own_progress': True,
}

216
core/automation/progress.py
View file

@ -1,216 +0,0 @@
"""Automation progress tracking.
Owns the in-memory progress state dict that backs both
`/api/automations/progress` polling and the WebSocket
`automation:progress` push emitter. State is per-automation, capped at
50 log entries each, and finished/error states are reaped 60s after
they finish so the frontend has a window to show the final state.
Functions are written so the route layer / engine callbacks can pass
their own socketio emitter, db handle, and shutdown flag. The progress
state dict (`progress_states`) and its lock (`progress_lock`) are
module-level so all callers share one view — same as the original
web_server.py globals.
"""
from __future__ import annotations
import json
import logging
import threading
from datetime import datetime, timezone
from typing import Any, Callable, Optional
logger = logging.getLogger(__name__)
# Shared mutable state — module globals so every caller (routes, engine
# progress callbacks, emit loop) sees the same dict. Mirrors the original
# `automation_progress_states` / `automation_progress_lock` in web_server.
progress_states: dict[int, dict] = {}
progress_lock = threading.Lock()
def init_progress(automation_id: int, automation_name: str, action_type: str) -> None:
"""Initialize progress state when an automation starts running."""
with progress_lock:
progress_states[automation_id] = {
'status': 'running',
'action_type': action_type,
'progress': 0,
'phase': 'Starting...',
'current_item': '',
'processed': 0,
'total': 0,
'log': [{'type': 'info', 'text': f'Starting {automation_name}'}],
'started_at': datetime.now(timezone.utc).isoformat(),
'finished_at': None,
}
def update_progress(
automation_id: Optional[int],
*,
socketio_emit: Optional[Callable[[str, Any], None]] = None,
**kwargs,
) -> None:
"""Update progress state from handler threads. Thread-safe.
`socketio_emit` lets callers wire in the live socketio.emit so that
finished/error transitions push immediately without waiting for the
1s emitter loop. Falls back to no-op if not provided.
"""
if automation_id is None:
return
with progress_lock:
state = progress_states.get(automation_id)
if not state:
return
for k, v in kwargs.items():
if k == 'log_line':
state['log'].append({'type': kwargs.get('log_type', 'info'), 'text': v})
if len(state['log']) > 50:
state['log'] = state['log'][-50:]
elif k != 'log_type':
state[k] = v
if kwargs.get('status') in ('finished', 'error'):
state['finished_at'] = datetime.now(timezone.utc).isoformat()
if socketio_emit is not None:
try:
socketio_emit('automation:progress', {str(automation_id): dict(state)})
except Exception as e:
logger.debug("socketio progress emit: %s", e)
def get_running_progress() -> dict[str, dict]:
"""Snapshot of running/finished/error states for the polling endpoint."""
with progress_lock:
result: dict[str, dict] = {}
for aid, state in progress_states.items():
if state['status'] in ('running', 'finished', 'error'):
cp = dict(state)
cp['log'] = list(state['log'])
result[str(aid)] = cp
return result
def record_history(
automation_id: int,
result: dict,
database,
) -> None:
"""Capture progress state into run history before cleanup clears it.
`database` is passed in so the function works without a `get_database()`
global.
"""
try:
with progress_lock:
state = progress_states.get(automation_id)
if state:
started_at = state.get('started_at')
finished_at = state.get('finished_at') or datetime.now(timezone.utc).isoformat()
log_entries = list(state.get('log', []))
else:
started_at = datetime.now(timezone.utc).isoformat()
finished_at = datetime.now(timezone.utc).isoformat()
log_entries = []
duration = None
if started_at and finished_at:
try:
t0 = datetime.fromisoformat(started_at)
t1 = datetime.fromisoformat(finished_at)
duration = (t1 - t0).total_seconds()
except Exception as e:
logger.debug("duration parse: %s", e)
r_status = result.get('status', 'completed') if result else 'completed'
if r_status == 'error':
status = 'error'
elif r_status == 'skipped':
status = 'skipped'
elif r_status == 'timeout':
status = 'timeout'
else:
status = 'completed'
summary = None
for entry in reversed(log_entries):
if entry.get('type') in ('success', 'error'):
summary = entry.get('text', '')
break
if not summary and log_entries:
summary = log_entries[-1].get('text', '')
if not summary and result:
summary = result.get('reason') or result.get('error') or result.get('status', '')
result_json = json.dumps({k: v for k, v in result.items() if not k.startswith('_')}) if result else None
log_json = json.dumps(log_entries) if log_entries else None
database.insert_automation_run_history(
automation_id=automation_id,
started_at=started_at,
finished_at=finished_at,
duration_seconds=duration,
status=status,
summary=summary,
result_json=result_json,
log_lines=log_json,
)
except Exception as e:
logger.error(f"Error recording automation history for {automation_id}: {e}")
def emit_progress_loop(
socketio,
*,
is_shutting_down: Callable[[], bool],
poll_interval: float = 1.0,
timeout_seconds: int = 7200,
cleanup_after_seconds: int = 60,
) -> None:
"""Push `automation:progress` events for active automations.
Long-running loop — caller wires this into a socketio background task.
- Times out zombie running states after `timeout_seconds` (default 2h).
- Reaps finished/error states `cleanup_after_seconds` after finish so the
frontend has a final-state window before they disappear.
"""
while not is_shutting_down():
socketio.sleep(poll_interval)
try:
with progress_lock:
active: dict[str, dict] = {}
stale: list[int] = []
now = datetime.now()
for aid, state in progress_states.items():
if state['status'] == 'running':
try:
started = datetime.fromisoformat(state.get('started_at', ''))
if (now - started).total_seconds() > timeout_seconds:
state['status'] = 'error'
state['phase'] = 'Timed out'
state['finished_at'] = now.isoformat()
state['log'].append({'type': 'error', 'text': f'Timed out after {timeout_seconds // 3600} hours'})
cp = dict(state)
cp['log'] = list(state['log'])
active[str(aid)] = cp
continue
except (ValueError, TypeError):
pass
cp = dict(state)
cp['log'] = list(state['log'])
active[str(aid)] = cp
elif state['status'] in ('finished', 'error') and state.get('finished_at'):
try:
finished_time = datetime.fromisoformat(state['finished_at'])
if (now - finished_time).total_seconds() > cleanup_after_seconds:
stale.append(aid)
except (ValueError, TypeError):
stale.append(aid)
for aid in stale:
del progress_states[aid]
if active:
socketio.emit('automation:progress', active)
except Exception as e:
logger.debug(f"Error emitting automation progress: {e}")

321
core/automation/schedule.py
View file

@ -1,321 +0,0 @@
"""Pure functions for computing the next-run datetime of a scheduled
automation trigger.
The Auto-Sync schedule board currently exposes interval-based scheduling
(``every N hours``) backed by ``trigger_type='schedule'``. The
automation engine ALSO supports ``daily_time`` and ``weekly_time``
triggers via separate ``_setup_*_trigger`` methods inline on the engine
class. None of that logic is currently testable in isolation — the
engine's ``_finish_run`` reaches for ``datetime.now()``, threads it
through ``_next_weekly_occurrence``, and writes the result to the DB,
all on the same call.
This module lifts the "given a trigger config, what's the next run?"
question out of the engine into a pure function:
next_run_at(trigger_type, trigger_config, now_utc, default_tz)
-> Optional[datetime]
That means:
- ``now_utc`` is INJECTED, not pulled from the system clock. Tests
freeze time without monkeypatching ``datetime.now``.
- ``default_tz`` is INJECTED. Daily / weekly / monthly schedules are
inherently in the USER'S timezone (cron "every Monday at 9am" is
not UTC), and the historic engine implicitly used the server's
local tz via naive ``datetime.now()``. That broke for users on a
different tz than their server. The pure function takes the tz
explicitly so the caller controls it.
- Returns an aware UTC ``datetime`` ready to serialise to the DB's
``next_run`` string column, or ``None`` for unrecognised /
event-based triggers (engine should not store a next_run for those).
PR 1 of the schedule-types feature ships ONLY this module + tests.
The engine continues to compute next_run via its existing inline
helpers; PR 2 collapses those into a single ``next_run_at`` call.
Net behavior is identical until the engine is wired through — this
PR is pure plumbing.
Schedule types supported here:
- ``schedule`` (interval): ``{interval: N, unit: 'minutes'|'hours'|'days'}``
— adds the interval to ``now_utc``; no tz needed.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — runs every day at
the given local time in the given timezone. ``tz`` falls back to
``default_tz`` when absent.
- ``weekly_time``: ``{time: 'HH:MM', days: ['mon','wed',...], tz: '<IANA>'}``
— runs on the matching weekday(s) at the given local time. Empty
``days`` list means "every day" (matches the engine's existing
fallback in ``_next_weekly_occurrence``).
- ``monthly_time``: ``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}``
— runs on the given day each month. Days that don't exist in a
given month (Feb 30, Apr 31) clamp to the LAST valid day of that
month rather than skipping the run entirely; missing a whole
month silently because the schedule was over-eager is worse than
running a day early.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, Optional
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from utils.logging_config import get_logger
logger = get_logger("automation.schedule")
# Unknown-tz names already warned about in this process — avoids
# spamming the log on every poll cycle for the same misconfigured row.
_UNKNOWN_TZ_WARNED: set = set()
# Weekday abbreviation → ``datetime.weekday()`` index (Mon=0..Sun=6).
# Mirrors the engine's existing ``_next_weekly_occurrence`` mapping so
# schedules created against either implementation accept the same
# ``days`` strings.
_WEEKDAY_MAP = {
'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6,
}
# Interval multipliers — kept aligned with the engine's existing
# ``_calc_delay_seconds`` in ``core/automation_engine.py``. Adding
# entries here without also updating the engine would silently drift:
# this function would honour the new unit while the live engine path
# defaults it to hours. Keep the maps in sync until PR 2 collapses the
# engine through this function.
_INTERVAL_MULTIPLIERS = {
'minutes': 60,
'hours': 60 * 60,
'days': 60 * 60 * 24,
}
def next_run_at(
trigger_type: str,
trigger_config: Dict[str, Any],
now_utc: datetime,
default_tz: str = 'UTC',
) -> Optional[datetime]:
"""Compute the next-run timestamp (UTC, aware) for a scheduled
trigger. Returns ``None`` for unrecognised types or event-based
triggers — callers should not write a next_run for those.
See module docstring for supported trigger types + config shapes.
"""
if not isinstance(trigger_config, dict):
trigger_config = {}
if trigger_type == 'schedule':
return _next_interval(trigger_config, now_utc)
if trigger_type == 'daily_time':
return _next_daily(trigger_config, now_utc, default_tz)
if trigger_type == 'weekly_time':
return _next_weekly(trigger_config, now_utc, default_tz)
if trigger_type == 'monthly_time':
return _next_monthly(trigger_config, now_utc, default_tz)
return None
# ---------------------------------------------------------------------------
# Interval
# ---------------------------------------------------------------------------
def _next_interval(config: Dict[str, Any], now_utc: datetime) -> datetime:
"""``{interval: N, unit: 'hours'}`` → ``now_utc + N hours``.
Mirrors the engine's existing ``_calc_delay_seconds``. Unit defaults
to ``hours`` for backward compat with legacy DB rows that pre-date
the unit field being mandatory; interval defaults to 1 so a fully
empty config doesn't divide-by-zero or schedule for the past."""
try:
interval = max(int(config.get('interval', 1)), 1)
except (TypeError, ValueError):
interval = 1
unit = config.get('unit') or 'hours'
seconds = interval * _INTERVAL_MULTIPLIERS.get(unit, _INTERVAL_MULTIPLIERS['hours'])
return _ensure_utc(now_utc) + timedelta(seconds=seconds)
# ---------------------------------------------------------------------------
# Daily
# ---------------------------------------------------------------------------
def _next_daily(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', tz: '<IANA>'}`` → next occurrence of that
wall-clock time in the user's timezone, expressed as aware UTC.
DST-aware via ``zoneinfo``: when the local time falls during a
spring-forward gap, the ``replace`` lands on a non-existent
instant; ``zoneinfo`` resolves that to the gap's later side
(e.g. 02:30 on the DST-forward day becomes 03:30 local). Tests
pin both spring-forward and fall-back behaviour."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
now_local = _ensure_utc(now_utc).astimezone(tz)
target_local = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target_local <= now_local:
target_local = target_local + timedelta(days=1)
return target_local.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Weekly
# ---------------------------------------------------------------------------
def _next_weekly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', days: ['mon',...], tz: '<IANA>'}`` → next
occurrence of that wall-clock time on any of the listed weekdays
in the user's timezone.
Empty ``days`` list ≡ every day, matching the engine's existing
fallback. Unrecognised day abbreviations are silently dropped
(an empty result-set then triggers the every-day fallback)."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
days = _parse_weekdays(config.get('days'))
now_local = _ensure_utc(now_utc).astimezone(tz)
# Scan today + next 7 days; the matching day with a future
# local time wins. 8-day scan is enough to handle the case where
# today already passed the time AND today is the only allowed
# weekday (next occurrence is exactly one week out).
for offset in range(8):
candidate = now_local + timedelta(days=offset)
if candidate.weekday() not in days:
continue
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target > now_local:
return target.astimezone(timezone.utc)
# Shouldn't reach: 8-day scan always finds a hit when ``days``
# is non-empty. Defensive fallback: next week, same weekday as today.
fallback = (now_local + timedelta(days=7)).replace(
hour=hour, minute=minute, second=0, microsecond=0,
)
return fallback.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Monthly
# ---------------------------------------------------------------------------
def _next_monthly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}`` → next
occurrence in the user's timezone.
``day_of_month`` is clamped to ``[1, 31]``. When the target day
doesn't exist in a given month (Feb 30, Apr 31), the schedule
falls back to the LAST valid day of that month — running a day
or two early in short months is less surprising than skipping
a month entirely. This matches the convention every cron
implementation in the wild settled on."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
raw_day = config.get('day_of_month', 1)
try:
target_day = max(1, min(31, int(raw_day)))
except (TypeError, ValueError):
target_day = 1
now_local = _ensure_utc(now_utc).astimezone(tz)
# Try this month first; if the target day has already passed
# (or doesn't exist this month and the clamped day is in the
# past), advance to next month. Loop bounded to 12 iterations
# so a pathologically broken config can't infinite-loop us.
year, month = now_local.year, now_local.month
for _ in range(12):
day = min(target_day, _days_in_month(year, month))
target = now_local.replace(
year=year, month=month, day=day,
hour=hour, minute=minute, second=0, microsecond=0,
)
if target > now_local:
return target.astimezone(timezone.utc)
# Roll to next month.
if month == 12:
year, month = year + 1, 1
else:
month += 1
# Defensive — should be unreachable.
return (now_local + timedelta(days=30)).astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _ensure_utc(dt: datetime) -> datetime:
"""Coerce a possibly-naive datetime to aware UTC. Naive inputs
are assumed UTC (matches the convention the engine uses when
parsing the DB ``next_run`` column)."""
if dt.tzinfo is None:
return dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def _resolve_tz(name: Optional[str]):
"""Look up an IANA tz by name. Falls back to UTC when the name is
unknown — ``ZoneInfoNotFoundError`` is the symptom of either a
typo in the tz string or ``tzdata`` missing on the host. Logged
once per unknown name so the user can see WHY their schedule
isn't running in the timezone they configured."""
if not name:
return timezone.utc
try:
return ZoneInfo(name)
except ZoneInfoNotFoundError:
if name not in _UNKNOWN_TZ_WARNED:
_UNKNOWN_TZ_WARNED.add(name)
logger.warning(
"Unknown timezone %r — schedule will run against UTC. "
"Check the spelling (IANA format like 'America/Los_Angeles') "
"or install the `tzdata` package on minimal hosts.",
name,
)
return timezone.utc
def _parse_hhmm(time_str: Optional[str]) -> tuple:
"""Parse ``HH:MM`` → ``(hour, minute)``. Defaults to 00:00 on
garbage input — same defensive shape as the engine's existing
daily/weekly time parsing."""
if not isinstance(time_str, str):
return 0, 0
try:
h, m = time_str.split(':', 1)
return max(0, min(23, int(h))), max(0, min(59, int(m)))
except (ValueError, AttributeError):
return 0, 0
def _parse_weekdays(days) -> set:
"""``['mon', 'wed']`` → ``{0, 2}``. Empty / missing / all-invalid
list returns ``set(range(7))`` ("every day"), matching the
engine's existing ``_next_weekly_occurrence`` fallback."""
if not isinstance(days, (list, tuple)):
return set(range(7))
parsed = {_WEEKDAY_MAP[d.lower()] for d in days
if isinstance(d, str) and d.lower() in _WEEKDAY_MAP}
return parsed or set(range(7))
def _days_in_month(year: int, month: int) -> int:
"""Last calendar day of ``year-month``. Stdlib-only — no calendar
module import needed; cycle through the 12 months."""
if month == 12:
next_first = datetime(year + 1, 1, 1)
else:
next_first = datetime(year, month + 1, 1)
last_day = next_first - timedelta(days=1)
return last_day.day

46
core/automation/signals.py
View file

@ -1,46 +0,0 @@
"""Automation signal helpers — name collection for autocomplete.
Signal cycle detection itself lives in core/automation_engine.py
(`detect_signal_cycles`); this module just enumerates known signal
names from the saved automation set so the builder UI can autocomplete.
"""
from __future__ import annotations
import json
import logging
logger = logging.getLogger(__name__)
def collect_known_signals(database) -> list[str]:
"""Return sorted, deduped signal names referenced by any saved automation.
Walks every automation and pulls signal names from both the
`signal_received` trigger config and any `fire_signal` then-actions.
Errors at every layer are swallowed — the autocomplete is best-effort.
"""
signals: set[str] = set()
try:
for auto in database.get_automations():
if auto.get('trigger_type') == 'signal_received':
try:
tc = json.loads(auto.get('trigger_config') or '{}')
sig = tc.get('signal_name', '').strip()
if sig:
signals.add(sig)
except (json.JSONDecodeError, TypeError):
pass
try:
ta = json.loads(auto.get('then_actions') or '[]')
for item in ta:
if item.get('type') == 'fire_signal':
sig = item.get('config', {}).get('signal_name', '').strip()
if sig:
signals.add(sig)
except (json.JSONDecodeError, TypeError):
pass
except Exception as e:
logger.debug("collect known signals failed: %s", e)
return sorted(signals)

319
core/automation_engine.py
View file

@ -18,14 +18,10 @@ import time
import threading
import requests
from datetime import datetime, timedelta, timezone
from typing import Optional
from utils.logging_config import get_logger
from core.automation.schedule import next_run_at
logger = get_logger("automation_engine")
def _utcnow():
"""Return current UTC time as timezone-aware datetime."""
return datetime.now(timezone.utc)
@ -38,42 +34,6 @@ def _utc_after(seconds):
"""Return UTC time N seconds from now as naive string for DB storage."""
return (datetime.now(timezone.utc) + timedelta(seconds=seconds)).strftime('%Y-%m-%d %H:%M:%S')
def _dt_to_db_str(dt: datetime) -> str:
"""Convert an aware-UTC datetime to the naive-UTC string the DB
``next_run`` column stores. Centralised so a tz mistake here
surfaces in one place, not scattered through every caller of
``next_run_at``."""
if dt.tzinfo is None:
return dt.strftime('%Y-%m-%d %H:%M:%S')
return dt.astimezone(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')
def _resolve_system_default_tz() -> str:
"""Return the IANA tz name the engine uses when a schedule's
trigger config doesn't carry an explicit ``tz`` field.
Existing daily / weekly rows pre-date the ``tz`` field — the
historic engine computed delays from naive ``datetime.now()``,
which is implicitly the server's local timezone. Falling back to
that same tz here preserves "every Monday at 09:00" running at
09:00 server local for rows that already exist in the DB.
Without ``tzlocal`` installed (or a system without a discoverable
tz), falls back to UTC."""
try:
import tzlocal
return tzlocal.get_localzone_name() or 'UTC'
except Exception:
return 'UTC'
# Server-local tz cached at import time. Re-reading per-call is
# pointless: the host's timezone doesn't change while the process is
# running. Tests that need a different default tz inject it through
# the engine's ``_default_tz`` attribute or via the
# ``automation.default_timezone`` config key.
_SYSTEM_DEFAULT_TZ = _resolve_system_default_tz()
SYSTEM_AUTOMATIONS = [
{
'name': 'Auto-Process Wishlist',
@ -174,25 +134,11 @@ class AutomationEngine:
self._max_chain_depth = 5
self._signal_cooldown_seconds = 10
# Default tz used when a schedule's ``trigger_config`` doesn't
# carry an explicit ``tz`` field — preserves historic behaviour
# for daily / weekly rows created before the field existed
# (engine used naive ``datetime.now()`` = server local). Reads
# from the ``automation.default_timezone`` config key first to
# let users override without touching env vars; falls back to
# the system-detected local tz.
try:
from config.settings import config_manager
self._default_tz = (config_manager.get('automation.default_timezone', '') or _SYSTEM_DEFAULT_TZ)
except Exception:
self._default_tz = _SYSTEM_DEFAULT_TZ
# Trigger registry: type → setup function (schedule only — events use emit())
self._trigger_handlers = {
'schedule': self._setup_schedule_trigger,
'daily_time': self._setup_daily_time_trigger,
'weekly_time': self._setup_weekly_time_trigger,
'monthly_time': self._setup_monthly_time_trigger,
}
# --- Action Handler Registration ---
@ -324,7 +270,7 @@ class AutomationEngine:
"""Cancel all timers on shutdown."""
self._running = False
with self._lock:
for _aid, timer in self._timers.items():
for aid, timer in self._timers.items():
timer.cancel()
count = len(self._timers)
self._timers.clear()
@ -509,10 +455,8 @@ class AutomationEngine:
if delay_minutes and delay_minutes > 0:
# Initialize progress BEFORE delay so card glows during wait
if self._progress_init_fn:
try:
self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception as e:
logger.debug("event progress init (delay): %s", e)
try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception: pass
_delay_already_inited = True
delay_seconds = int(delay_minutes) * 60
@ -542,17 +486,13 @@ class AutomationEngine:
logger.info(f"Event automation '{auto.get('name')}' skipped — {action_type} busy")
# If progress was initialized during delay, finalize it
if _delay_already_inited and self._progress_finish_fn:
try:
self._progress_finish_fn(automation_id, result)
except Exception as e:
logger.debug("event progress finish (skipped): %s", e)
try: self._progress_finish_fn(automation_id, result)
except Exception: pass
else:
# Initialize progress tracking (skip if already done during delay)
if not _delay_already_inited and self._progress_init_fn:
try:
self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception as e:
logger.debug("event progress init: %s", e)
try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception: pass
try:
result = handler_info['handler'](action_config) or {}
logger.info(f"Event automation '{auto.get('name')}' executed: {result.get('status', 'ok')}")
@ -561,10 +501,8 @@ class AutomationEngine:
logger.error(f"Event automation '{auto.get('name')}' action failed: {e}")
# Finalize progress tracking
if self._progress_finish_fn:
try:
self._progress_finish_fn(automation_id, result)
except Exception as e:
logger.debug("event progress finish: %s", e)
try: self._progress_finish_fn(automation_id, result)
except Exception: pass
# Merge event data into result for then-action variables
merged = {**event_data, **result}
@ -577,24 +515,14 @@ class AutomationEngine:
# Update run stats (no reschedule — event triggers don't use timers)
last_result = json.dumps({k: v for k, v in merged.items() if not k.startswith('_')})
# Surface every failure mode to last_error: handlers in this codebase use
# 'error', 'reason', or 'message' interchangeably when returning gracefully.
if result.get('status') == 'error':
error = (
result.get('error')
or result.get('reason')
or result.get('message')
or 'Handler reported failure'
)
else:
error = None
error = result.get('error') if result.get('status') == 'error' else None
self.db.update_automation_run(automation_id, error=error, last_result=last_result)
if self._history_record_fn:
try:
self._history_record_fn(automation_id, result)
except Exception as e:
logger.debug("history record failed: %s", e)
except Exception:
pass
# --- Schedule Execution (timer-based) ---
@ -635,11 +563,6 @@ class AutomationEngine:
action_config['_automation_name'] = auto.get('name', '')
if profile_id is not None:
action_config['_profile_id'] = profile_id
# The profile this run acts AS: an explicit trigger profile, else the
# automation's owner, else admin. System + admin automations are
# profile 1, so this is a no-op for them — only non-admin-owned
# automations gain their correct identity in the background.
_effective_profile_id = profile_id if profile_id is not None else (auto.get('profile_id') or 1)
# Action delay (skipped for manual run_now)
delay_minutes = action_config.get('delay', 0)
@ -647,10 +570,8 @@ class AutomationEngine:
if not skip_delay and delay_minutes and delay_minutes > 0:
# Initialize progress BEFORE delay so card glows during wait
if self._progress_init_fn:
try:
self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception as e:
logger.debug("scheduled progress init (delay): %s", e)
try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception: pass
_delay_already_inited = True
delay_seconds = int(delay_minutes) * 60
@ -672,55 +593,31 @@ class AutomationEngine:
logger.info(f"Automation '{auto['name']}' skipped — {action_type} already running")
# If progress was initialized during delay, finalize it
if _delay_already_inited and self._progress_finish_fn:
try:
self._progress_finish_fn(automation_id, result)
except Exception as e:
logger.debug("scheduled progress finish (skipped): %s", e)
self._finish_run(auto, automation_id, result, error=None, retry_delay_seconds=300)
try: self._progress_finish_fn(automation_id, result)
except Exception: pass
self._finish_run(auto, automation_id, result, error=None)
return
# Initialize progress tracking (skip if already done during delay)
if not _delay_already_inited and self._progress_init_fn:
try:
self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception as e:
logger.debug("scheduled progress init: %s", e)
try: self._progress_init_fn(automation_id, auto.get('name', ''), action_type)
except Exception: pass
# Execute the action under the owner's profile so get_current_profile_id()
# (and the per-profile clients it resolves) act as the automation's owner
# in the background, not admin. Reset in finally so a pooled thread can't
# leak the override to the next job.
# Execute the action
error = None
result = {}
from core.profile_context import set_background_profile, reset_background_profile
_bg_token = set_background_profile(_effective_profile_id)
try:
result = handler_info['handler'](action_config) or {}
logger.info(f"Automation '{auto['name']}' (id={automation_id}) executed: {result.get('status', 'ok')}")
# Handlers may signal failure by RETURNING {'status': 'error', ...} instead of
# raising. Surface that to the DB so `last_error` reflects every failure mode,
# not just uncaught exceptions. Falls back through ('error', 'reason', 'message')
# because handlers in this codebase aren't consistent about which key they set.
if result.get('status') == 'error':
error = (
result.get('error')
or result.get('reason')
or result.get('message')
or 'Handler reported failure'
)
except Exception as e:
error = str(e)
result = {'status': 'error', 'error': error}
logger.error(f"Automation '{auto['name']}' (id={automation_id}) failed: {e}")
finally:
reset_background_profile(_bg_token)
# Finalize progress tracking
if self._progress_finish_fn:
try:
self._progress_finish_fn(automation_id, result)
except Exception as e:
logger.debug("scheduled progress finish: %s", e)
try: self._progress_finish_fn(automation_id, result)
except Exception: pass
# Execute then-actions (notifications + fire_signal)
try:
@ -730,7 +627,7 @@ class AutomationEngine:
self._finish_run(auto, automation_id, result, error)
def _finish_run(self, auto, automation_id, result, error, retry_delay_seconds=None):
def _finish_run(self, auto, automation_id, result, error):
"""Update DB with run stats and reschedule."""
next_run_str = None
trigger_type = auto.get('trigger_type', '')
@ -738,25 +635,25 @@ class AutomationEngine:
if trigger_type in self._trigger_handlers:
try:
trigger_config = json.loads(auto.get('trigger_config') or '{}')
if retry_delay_seconds:
next_run_str = _utc_after(retry_delay_seconds)
if trigger_type == 'daily_time':
# Next run is tomorrow at the configured time (compute delay from local time, store as UTC)
time_str = trigger_config.get('time', '00:00')
hour, minute = map(int, time_str.split(':'))
now_local = datetime.now()
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
next_run_str = _utc_after((target - now_local).total_seconds())
elif trigger_type == 'weekly_time':
time_str = trigger_config.get('time', '00:00')
hour, minute = map(int, time_str.split(':'))
now_local = datetime.now()
target = self._next_weekly_occurrence(hour, minute, trigger_config.get('days', []))
next_run_str = _utc_after((target - now_local).total_seconds())
else:
# Single integration point with ``next_run_at``. The
# helper handles every trigger type the engine
# supports (interval / daily / weekly / monthly) and
# returns aware-UTC; ``_dt_to_db_str`` normalises to
# the naive-UTC string the DB column stores. Tests
# injecting a different ``now_utc`` patch this same
# path — no scattered ``datetime.now()`` calls left.
next_run_dt = next_run_at(
trigger_type, trigger_config,
now_utc=_utcnow(),
default_tz=self._default_tz,
)
if next_run_dt is not None:
next_run_str = _dt_to_db_str(next_run_dt)
except Exception as e:
logger.debug("next run calc failed: %s", e)
delay = self._calc_delay_seconds(trigger_config)
if delay:
next_run_str = _utc_after(delay)
except Exception:
pass
last_result = json.dumps(result) if result else None
self.db.update_automation_run(automation_id, next_run=next_run_str, error=error, last_result=last_result)
@ -764,8 +661,8 @@ class AutomationEngine:
if self._history_record_fn:
try:
self._history_record_fn(automation_id, result)
except Exception as e:
logger.debug("history record failed: %s", e)
except Exception:
pass
if self._running:
self.schedule_automation(automation_id)
@ -828,72 +725,22 @@ class AutomationEngine:
logger.debug(f"Scheduled automation {automation_id} in {delay:.0f}s")
def _setup_daily_time_trigger(self, automation_id, config):
"""Config: ``{"time": "03:00", "tz": "<IANA>"}`` — runs daily
at the specified local time. Tz defaults to ``self._default_tz``
when absent."""
self._setup_timed_trigger(automation_id, 'daily_time', config,
label=f"Daily at {config.get('time', '00:00')}")
"""Config: {"time": "03:00"} — runs daily at the specified local time."""
time_str = config.get('time', '00:00')
try:
hour, minute = map(int, time_str.split(':'))
except (ValueError, AttributeError):
hour, minute = 0, 0
def _setup_weekly_time_trigger(self, automation_id, config):
"""Config: ``{"time": "03:00", "days": ["mon","wed","fri"], "tz": "<IANA>"}``."""
day_names = ', '.join(config.get('days') or []) or 'every day'
self._setup_timed_trigger(automation_id, 'weekly_time', config,
label=f"Weekly {config.get('time', '00:00')} on {day_names}")
now_local = datetime.now()
target = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target <= now_local:
target += timedelta(days=1)
def _setup_monthly_time_trigger(self, automation_id, config):
"""Config: ``{"time": "09:00", "day_of_month": 15, "tz": "<IANA>"}``.
delay = (target - now_local).total_seconds()
Day clamped to [1, 31]; months too short for the target day
clamp to the last valid day (Feb 31 → Feb 28 / Feb 29 leap
year) per standard cron convention — see
``core.automation.schedule._next_monthly`` for the rule."""
day = config.get('day_of_month', 1)
self._setup_timed_trigger(automation_id, 'monthly_time', config,
label=f"Monthly {config.get('time', '00:00')} on day {day}")
def _setup_timed_trigger(self, automation_id, trigger_type, config, *, label):
"""Shared setup for daily / weekly / monthly time triggers.
All three flow through the same skeleton: compute next-run
via ``next_run_at``, persist to DB, arm a ``threading.Timer``
that fires the automation when the delay elapses. Lifting
these out of three near-identical methods means there's one
place to fix when (e.g.) timer rearm semantics need a tweak.
Honours an existing future ``next_run`` row in the DB —
prevents losing a hand-edited next_run when the engine
reschedules at startup. Same guard as the interval path."""
target_dt = next_run_at(
trigger_type, config or {},
now_utc=_utcnow(),
default_tz=self._default_tz,
)
if target_dt is None:
logger.warning(
f"Skip scheduling automation {automation_id}: next_run_at returned "
f"None for {trigger_type!r}",
)
return
delay = max(0.0, (target_dt - _utcnow()).total_seconds())
# If the DB already carries a future next_run, prefer it —
# matches the interval-path behaviour and lets manual edits
# or pending retries survive a process restart.
auto = self.db.get_automation(automation_id)
if auto and auto.get('next_run'):
try:
existing = datetime.strptime(
auto['next_run'], '%Y-%m-%d %H:%M:%S',
).replace(tzinfo=timezone.utc)
remaining = (existing - _utcnow()).total_seconds()
if remaining > 0:
delay = remaining
target_dt = existing
except (ValueError, TypeError):
pass
self.db.update_automation(automation_id, next_run=_dt_to_db_str(target_dt))
next_run_str = _utc_after(delay)
self.db.update_automation(automation_id, next_run=next_run_str)
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
timer.daemon = True
@ -902,7 +749,48 @@ class AutomationEngine:
with self._lock:
self._timers[automation_id] = timer
logger.debug(f"{label} automation {automation_id} scheduled (in {delay:.0f}s)")
logger.debug(f"Daily automation {automation_id} scheduled for {time_str} (in {delay:.0f}s)")
def _setup_weekly_time_trigger(self, automation_id, config):
"""Config: {"time": "03:00", "days": ["mon", "wed", "fri"]}"""
time_str = config.get('time', '00:00')
try:
hour, minute = map(int, time_str.split(':'))
except (ValueError, AttributeError):
hour, minute = 0, 0
target = self._next_weekly_occurrence(hour, minute, config.get('days', []))
delay = (target - datetime.now()).total_seconds()
next_run_str = _utc_after(delay)
self.db.update_automation(automation_id, next_run=next_run_str)
timer = threading.Timer(delay, self.run_automation, args=(automation_id,))
timer.daemon = True
timer.start()
with self._lock:
self._timers[automation_id] = timer
day_names = ', '.join(config.get('days', [])) or 'every day'
logger.debug(f"Weekly automation {automation_id} scheduled for {time_str} on {day_names} (in {delay:.0f}s)")
def _next_weekly_occurrence(self, hour, minute, days):
"""Find the next datetime matching one of the given weekday abbreviations."""
day_map = {'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6}
allowed = {day_map[d] for d in days if d in day_map}
if not allowed:
allowed = set(range(7)) # no days selected = every day
now = datetime.now()
for offset in range(8): # check today + next 7 days
candidate = now + timedelta(days=offset)
if candidate.weekday() in allowed:
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target > now:
return target
# Fallback: tomorrow (shouldn't happen with 8-day scan)
return now.replace(hour=hour, minute=minute, second=0, microsecond=0) + timedelta(days=1)
# --- Then Actions (notifications + signals) ---
@ -1072,8 +960,6 @@ class AutomationEngine:
"""Send message via Telegram Bot API."""
bot_token = config.get('bot_token', '').strip()
chat_id = config.get('chat_id', '').strip()
thread_id = config.get('thread_id', '').strip()
if not bot_token or not chat_id:
raise ValueError("Bot token and chat ID are required for Telegram")
@ -1082,16 +968,9 @@ class AutomationEngine:
for key, value in variables.items():
message = message.replace('{' + key + '}', value)
payload = {"chat_id": chat_id, "text": message, "parse_mode": "HTML"}
if thread_id:
try:
payload["message_thread_id"] = int(thread_id)
except ValueError:
pass # invalid — fall back to main chat
resp = requests.post(
f'https://api.telegram.org/bot{bot_token}/sendMessage',
json=payload,
json={"chat_id": chat_id, "text": message, "parse_mode": "HTML"},
timeout=10,
)
data = resp.json() if resp.status_code == 200 else {}

37
core/blocklist/__init__.py
View file

@ -1,37 +0,0 @@
"""Artist / album / track blocklist (the "proper" blacklist).
Distinct from ``download_blacklist`` (which skips one bad source file from one
Soulseek peer — untouched here). This blocklist bans an ARTIST, ALBUM, or
TRACK from being acquired, keyed by metadata-source IDs (Spotify / iTunes /
Deezer / MusicBrainz) so a ban survives a source switch.
Phase 1 enforces at the single ``add_to_wishlist`` chokepoint: every
auto-acquisition path (watchlist, discography backfill, repair, manual
wishlist add) funnels through it, so one guard covers them all.
- ``matching`` — the pure decision core (no DB, no I/O): build an index from
blocklist rows, ask whether a candidate is blocked, with artist→album→track
cascade.
"""
from core.blocklist.matching import (
ENTITY_ALBUM,
ENTITY_ARTIST,
ENTITY_TRACK,
ENTITY_TYPES,
SOURCE_ID_FIELDS,
BlocklistIndex,
build_index,
candidate_block_reason,
)
__all__ = [
"ENTITY_ARTIST",
"ENTITY_ALBUM",
"ENTITY_TRACK",
"ENTITY_TYPES",
"SOURCE_ID_FIELDS",
"BlocklistIndex",
"build_index",
"candidate_block_reason",
]

50
core/blocklist/backfill.py
View file

@ -1,50 +0,0 @@
"""Cross-source ID backfill for blocklist entries.
When a user blocks an item, the modal gives us the ID for ONE source (the one
they searched). For the ban to survive a source switch, we resolve the OTHER
sources' IDs too — matching the blocked artist/album/track by name on each
source and taking a confident hit.
The resolution is kept pure + injected so it tests without a network: callers
pass a ``resolvers`` map ``{source: fn(entity_type, name, parent_name) -> id |
None}``. ``core/blocklist/runtime.py`` wires the real metadata clients.
Honest about fragility (acknowledged in design): artist matching is reliable,
album/track cross-source matching is best-effort (editions, common titles), so
a resolver returning None just leaves that source unmatched — the artist
name-fallback in matching.py covers artist gaps; album/track gaps mean that
ban only applies on sources where an ID resolved.
"""
from __future__ import annotations
from typing import Any, Callable, Dict, Optional
from core.blocklist.matching import SOURCE_ID_FIELDS
def resolve_missing_ids(
entry: Dict[str, Any],
resolvers: Dict[str, Callable[..., Optional[str]]],
) -> Dict[str, str]:
"""Return ``{id_column: resolved_id}`` for the sources currently missing an
ID on ``entry``. Never raises — a resolver that errors is skipped."""
out: Dict[str, str] = {}
entity_type = entry.get("entity_type")
name = entry.get("name")
parent = entry.get("parent_name")
if not entity_type or not name:
return out
for source, col in SOURCE_ID_FIELDS.items():
if entry.get(col):
continue # already known
fn = resolvers.get(source)
if not fn:
continue
try:
rid = fn(entity_type, name, parent)
except Exception:
rid = None
if rid:
out[col] = str(rid)
return out

128
core/blocklist/matching.py
View file

@ -1,128 +0,0 @@
"""Pure blocklist matching — no DB, no I/O, fully unit-testable.
The brain of the blocklist: given the stored blocklist rows and a candidate
track being considered for the wishlist, decide whether it's blocked.
Design decisions (per Boulder):
- **ID-keyed.** Each row carries the candidate's IDs in up to four metadata
sources. A candidate is matched against the SAME source it came in on
(the wishlist payload carries active-source IDs), so a Deezer-numeric id
can't collide with an iTunes-numeric id of a different entity.
- **Cascade.** Blocking an artist blocks their albums + tracks; blocking an
album blocks its tracks. The candidate carries its own artist/album/track
IDs, so the check walks track → album → artist and blocks on the first hit.
- **Name fallback for ARTISTS only.** A blocked artist also matches by
case-folded name — this covers the window before the background ID-backfill
has resolved the active source's id. Albums/tracks do NOT fall back to name
(common titles like "Greatest Hits" would false-positive across artists);
they rely on IDs, which backfill fills in.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
ENTITY_ARTIST = "artist"
ENTITY_ALBUM = "album"
ENTITY_TRACK = "track"
ENTITY_TYPES = (ENTITY_ARTIST, ENTITY_ALBUM, ENTITY_TRACK)
# Blocklist-row column → the metadata source it belongs to.
SOURCE_ID_FIELDS = {
"spotify": "spotify_id",
"itunes": "itunes_id",
"deezer": "deezer_id",
"musicbrainz": "musicbrainz_id",
}
def _norm(text: Any) -> str:
return str(text or "").strip().casefold()
@dataclass
class _TypeIndex:
# per-source set of blocked ids, plus a case-folded name set (artists only)
ids: Dict[str, Set[str]] = field(default_factory=lambda: {s: set() for s in SOURCE_ID_FIELDS})
names: Set[str] = field(default_factory=set)
def hit(self, source: Optional[str], entity_id: Any, name: Any, use_name: bool) -> bool:
if entity_id and source in self.ids and str(entity_id) in self.ids[source]:
return True
if use_name and name and _norm(name) in self.names:
return True
return False
@dataclass
class BlocklistIndex:
"""Membership index built once per scan from the blocklist rows."""
artists: _TypeIndex = field(default_factory=_TypeIndex)
albums: _TypeIndex = field(default_factory=_TypeIndex)
tracks: _TypeIndex = field(default_factory=_TypeIndex)
@property
def is_empty(self) -> bool:
for ti in (self.artists, self.albums, self.tracks):
if ti.names or any(ti.ids.values()):
return False
return True
def build_index(rows: Iterable[Dict[str, Any]]) -> BlocklistIndex:
"""Build a BlocklistIndex from blocklist DB rows.
Each row needs ``entity_type``, ``name``, and the source id columns
(``spotify_id`` / ``itunes_id`` / ``deezer_id`` / ``musicbrainz_id``).
Unknown entity types are ignored."""
idx = BlocklistIndex()
by_type = {ENTITY_ARTIST: idx.artists, ENTITY_ALBUM: idx.albums, ENTITY_TRACK: idx.tracks}
for row in rows or []:
ti = by_type.get((row.get("entity_type") or "").strip().lower())
if ti is None:
continue
for source, col in SOURCE_ID_FIELDS.items():
val = row.get(col)
if val:
ti.ids[source].add(str(val))
name = row.get("name")
if name:
ti.names.add(_norm(name))
return idx
def candidate_block_reason(
index: BlocklistIndex,
*,
source: Optional[str],
track_id: Any = None,
track_name: Any = None,
album_id: Any = None,
album_name: Any = None,
artists: Optional[List[Dict[str, Any]]] = None,
) -> Optional[Tuple[str, str]]:
"""Return ``(entity_type, label)`` for the first cascade hit, else None.
``source`` is the metadata source the candidate IDs came from (the wishlist
payload's provider). ``artists`` is a list of ``{'id', 'name'}`` dicts.
Order matters only for the returned reason — any hit blocks."""
if index.is_empty:
return None
# Track level — id only (names too ambiguous to ban across artists).
if index.tracks.hit(source, track_id, track_name, use_name=False):
return (ENTITY_TRACK, str(track_name or track_id or "track"))
# Album level — id only.
if index.albums.hit(source, album_id, album_name, use_name=False):
return (ENTITY_ALBUM, str(album_name or album_id or "album"))
# Artist level — id OR case-folded name (safe + covers the backfill window).
for artist in artists or []:
a_id = artist.get("id") if isinstance(artist, dict) else None
a_name = artist.get("name") if isinstance(artist, dict) else artist
if index.artists.hit(source, a_id, a_name, use_name=True):
return (ENTITY_ARTIST, str(a_name or a_id or "artist"))
return None

82
core/blocklist/runtime.py
View file

@ -1,82 +0,0 @@
"""Wire real metadata clients to the blocklist backfill resolvers.
Resolves a blocked item's ID on each metadata source by searching that source
for the name and taking a confidently name-matched hit. Confidence = exact
significant-token match (drops articles/punctuation) so we never hang a wrong
ID on an entry. Albums/tracks additionally require the parent artist to match
when both sides expose one.
"""
from __future__ import annotations
import re
from typing import Any, Callable, Dict, Optional
from utils.logging_config import get_logger
logger = get_logger("blocklist.runtime")
_STOP = {"the", "a", "an", "feat", "ft", "featuring", "with"}
def _tokens(text: Any) -> frozenset:
words = re.sub(r"[^a-z0-9]+", " ", str(text or "").lower()).split()
return frozenset(w for w in words if w not in _STOP)
def _name_of(obj: Any) -> str:
if isinstance(obj, dict):
return str(obj.get("name") or obj.get("title") or "")
return str(getattr(obj, "name", None) or getattr(obj, "title", None) or "")
def _id_of(obj: Any) -> Optional[str]:
val = obj.get("id") if isinstance(obj, dict) else getattr(obj, "id", None)
return str(val) if val else None
def _confident(result_name: str, want_name: str) -> bool:
rt, wt = _tokens(result_name), _tokens(want_name)
return bool(rt) and rt == wt
def _make_resolver(source: str) -> Callable[..., Optional[str]]:
def resolve(entity_type: str, name: str, parent_name: Optional[str] = None) -> Optional[str]:
from core.metadata.registry import get_client_for_source
client = get_client_for_source(source)
if not client:
return None
method = {
"artist": "search_artists",
"album": "search_albums",
"track": "search_tracks",
}.get(entity_type)
fn = getattr(client, method, None) if method else None
if not fn:
return None
try:
results = fn(name, limit=5) or []
except Exception as e:
logger.debug("%s %s search failed for %r: %s", source, entity_type, name, e)
return None
for r in results:
if not _confident(_name_of(r), name):
continue
# For album/track, also require the artist to line up when known.
if entity_type in ("album", "track") and parent_name:
artists = (r.get("artists") if isinstance(r, dict) else getattr(r, "artists", None)) or []
cand_artists = " ".join(
a.get("name", "") if isinstance(a, dict) else str(a) for a in artists)
if _tokens(parent_name) and not (_tokens(parent_name) & _tokens(cand_artists)):
continue
rid = _id_of(r)
if rid:
return rid
return None
return resolve
def build_resolvers() -> Dict[str, Callable[..., Optional[str]]]:
"""Source→resolver map for core.blocklist.backfill.resolve_missing_ids."""
return {s: _make_resolver(s) for s in ("spotify", "itunes", "deezer", "musicbrainz")}

27
core/boot_phase.py
View file

@ -1,27 +0,0 @@
"""Boot-phase guard for non-blocking container startup.
While the gunicorn worker is importing ``web_server`` (module-level client and
worker initialization), external provider API probes must not block startup.
Network validation is deferred until ``mark_boot_complete()`` runs at the end
of that import pass.
"""
from __future__ import annotations
import threading
_boot_lock = threading.Lock()
_boot_active = True
def is_boot_phase() -> bool:
"""Return True while module import must avoid blocking provider API calls."""
with _boot_lock:
return _boot_active
def mark_boot_complete() -> None:
"""End the boot phase — provider clients may perform network probes again."""
global _boot_active
with _boot_lock:
_boot_active = False

257
core/connection_detect.py
View file

@ -1,257 +0,0 @@
"""Network detection — lifted from web_server.py.
Body is byte-identical to the original. Pure stdlib + requests, no
web_server-specific globals or runtime state.
"""
import ipaddress
import logging
import platform
import socket
import subprocess
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
logger = logging.getLogger(__name__)
def run_detection(server_type):
"""
Performs comprehensive network detection for a given server type (plex, jellyfin, slskd).
This implements the same scanning logic as the GUI's detection threads.
"""
logger.info(f"Running comprehensive detection for {server_type}...")
def get_network_info():
"""Get comprehensive network information with subnet detection"""
try:
# Get local IP using socket method
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
local_ip = s.getsockname()[0]
s.close()
# Try to get actual subnet mask
try:
if platform.system() == "Windows":
# Windows: Use netsh to get subnet info
result = subprocess.run(['netsh', 'interface', 'ip', 'show', 'config'],
capture_output=True, text=True, timeout=3)
# Parse output for subnet mask (simplified)
subnet_mask = "255.255.255.0" # Default fallback
else:
# Linux/Mac: Try to parse network interfaces
result = subprocess.run(['ip', 'route', 'show'],
capture_output=True, text=True, timeout=3)
subnet_mask = "255.255.255.0" # Default fallback
except:
subnet_mask = "255.255.255.0" # Default /24
# Calculate network range
network = ipaddress.IPv4Network(f"{local_ip}/{subnet_mask}", strict=False)
return str(network.network_address), str(network.netmask), local_ip, network
except Exception as e:
# Fallback to original method
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
local_ip = s.getsockname()[0]
s.close()
# Default to /24 network
network = ipaddress.IPv4Network(f"{local_ip}/24", strict=False)
return str(network.network_address), "255.255.255.0", local_ip, network
def test_plex_server(ip, port=32400):
"""Test if a Plex server is running at the given IP and port"""
try:
url = f"http://{ip}:{port}/web/index.html"
response = requests.get(url, timeout=2, allow_redirects=True)
# Check for Plex-specific indicators
if response.status_code == 200:
# Check if it's actually Plex
if 'plex' in response.text.lower() or 'X-Plex' in str(response.headers):
return f"http://{ip}:{port}"
# Also try the API endpoint
api_url = f"http://{ip}:{port}/identity"
api_response = requests.get(api_url, timeout=1)
if api_response.status_code == 200 and 'MediaContainer' in api_response.text:
return f"http://{ip}:{port}"
except Exception as e:
logger.debug("plex probe %s: %s", ip, e)
return None
def test_jellyfin_server(ip, port=8096):
"""Test if a Jellyfin server is running at the given IP and port"""
try:
# Try the system info endpoint first
url = f"http://{ip}:{port}/System/Info"
response = requests.get(url, timeout=2, allow_redirects=True)
if response.status_code == 200:
# Check if response contains Jellyfin-specific content
if 'jellyfin' in response.text.lower() or 'ServerName' in response.text:
return f"http://{ip}:{port}"
# Also try the web interface
web_url = f"http://{ip}:{port}/web/index.html"
web_response = requests.get(web_url, timeout=1)
if web_response.status_code == 200 and 'jellyfin' in web_response.text.lower():
return f"http://{ip}:{port}"
except Exception as e:
logger.debug("jellyfin probe %s: %s", ip, e)
return None
def test_slskd_server(ip, port=5030):
"""Test if a slskd server is running at the given IP and port"""
try:
# slskd specific API endpoint
url = f"http://{ip}:{port}/api/v0/session"
response = requests.get(url, timeout=2)
# slskd returns 401 when not authenticated, which is still a valid response
if response.status_code in [200, 401]:
return f"http://{ip}:{port}"
except Exception as e:
logger.debug("slskd probe %s: %s", ip, e)
return None
def test_navidrome_server(ip, port=4533):
"""Test if a Navidrome server is running at the given IP and port"""
try:
# Try Navidrome's ping endpoint (part of Subsonic API)
url = f"http://{ip}:{port}/rest/ping"
response = requests.get(url, timeout=2, params={
'u': 'test', # Dummy username for ping test
'v': '1.16.1', # API version
'c': 'soulsync', # Client name
'f': 'json' # Response format
})
# Navidrome should respond even with invalid credentials for ping
if response.status_code in [200, 401, 403]:
try:
data = response.json()
# Check for Subsonic/Navidrome API response structure
if 'subsonic-response' in data:
return f"http://{ip}:{port}"
except Exception as e:
logger.debug("navidrome json parse: %s", e)
# Also try the web interface
web_url = f"http://{ip}:{port}/"
web_response = requests.get(web_url, timeout=2)
if web_response.status_code == 200 and 'navidrome' in web_response.text.lower():
return f"http://{ip}:{port}"
except Exception as e:
logger.debug("navidrome probe %s: %s", ip, e)
return None
try:
network_addr, netmask, local_ip, network = get_network_info()
# Select the appropriate test function
test_functions = {
'plex': test_plex_server,
'jellyfin': test_jellyfin_server,
'navidrome': test_navidrome_server,
'slskd': test_slskd_server
}
test_func = test_functions.get(server_type)
if not test_func:
return None
# Priority 1: Test localhost first
logger.debug(f"Testing localhost for {server_type}...")
localhost_result = test_func("localhost")
if localhost_result:
logger.info(f"Found {server_type} at localhost!")
return localhost_result
# Priority 1.5: In Docker, try Docker host IP
import os
if os.path.exists('/.dockerenv'):
logger.info(f"Docker detected, testing Docker host for {server_type}...")
try:
# Try host.docker.internal (Windows/Mac)
host_result = test_func("host.docker.internal")
if host_result:
logger.info(f"Found {server_type} at Docker host!")
return host_result.replace("host.docker.internal", "localhost") # Convert back to localhost for config
# Try Docker bridge gateway (Linux)
gateway_result = test_func("172.17.0.1")
if gateway_result:
logger.info(f"Found {server_type} at Docker gateway!")
return gateway_result.replace("172.17.0.1", "localhost") # Convert back to localhost for config
except Exception as e:
logger.error(f"Docker host detection failed: {e}")
# Priority 2: Test local IP
logger.debug(f"Testing local IP {local_ip} for {server_type}...")
local_result = test_func(local_ip)
if local_result:
logger.info(f"Found {server_type} at {local_ip}!")
return local_result
# Priority 3: Test common IPs (router gateway, etc.)
common_ips = [
local_ip.rsplit('.', 1)[0] + '.1', # Typical gateway
local_ip.rsplit('.', 1)[0] + '.2', # Alternative gateway
local_ip.rsplit('.', 1)[0] + '.100', # Common static IP
]
logger.debug(f"Testing common IPs for {server_type}...")
for ip in common_ips:
logger.info(f" Checking {ip}...")
result = test_func(ip)
if result:
logger.info(f"Found {server_type} at {ip}!")
return result
# Priority 4: Scan the network range (limited to reasonable size)
network_hosts = list(network.hosts())
if len(network_hosts) > 50:
# Limit scan to reasonable size for performance
step = max(1, len(network_hosts) // 50)
network_hosts = network_hosts[::step]
logger.debug(f"Scanning network range for {server_type} ({len(network_hosts)} hosts)...")
# Use ThreadPoolExecutor for concurrent scanning (limited for web context)
with ThreadPoolExecutor(max_workers=5) as executor:
# Submit all tasks
future_to_ip = {executor.submit(test_func, str(ip)): str(ip)
for ip in network_hosts}
try:
for future in as_completed(future_to_ip):
ip = future_to_ip[future]
try:
result = future.result()
if result:
logger.info(f"Found {server_type} at {ip}!")
# Cancel all pending futures before returning
for f in future_to_ip:
if not f.done():
f.cancel()
return result
except Exception as e:
logger.error(f"Error testing {ip}: {e}")
continue
except Exception as e:
logger.error(f"Error in concurrent scanning: {e}")
logger.warning(f"No {server_type} server found on network")
return None
except Exception as e:
logger.error(f"Error during {server_type} detection: {e}")
return None

502
core/connection_test.py
View file

@ -1,502 +0,0 @@
"""Service connection test — lifted from web_server.py.
The function body is byte-identical to the original. download_orchestrator,
qobuz_enrichment_worker, hydrabase_client, docker_resolve_url, and
docker_resolve_path are injected at runtime because they live in
web_server.py and are constructed there.
"""
import logging
import os
import requests
from config.settings import config_manager
from core.jellyfin_client import JellyfinClient
from core.metadata.registry import get_primary_source
from core.plex_client import PlexClient
from core.spotify_client import SpotifyClient
from core.tidal_client import TidalClient
from utils.async_helpers import run_async
logger = logging.getLogger(__name__)
def _get_metadata_fallback_source():
"""Mirror of web_server._get_metadata_fallback_source — delegates to registry."""
return get_primary_source()
# Injected at runtime via init().
download_orchestrator = None
qobuz_enrichment_worker = None
hydrabase_client = None
docker_resolve_url = None
docker_resolve_path = None
def init(
download_orchestrator_obj,
qobuz_worker,
hydrabase_client_obj,
docker_resolve_url_fn,
docker_resolve_path_fn,
):
"""Bind web_server-side helpers/globals so the lifted body can resolve them."""
global download_orchestrator, qobuz_enrichment_worker, hydrabase_client
global docker_resolve_url, docker_resolve_path
download_orchestrator = download_orchestrator_obj
qobuz_enrichment_worker = qobuz_worker
hydrabase_client = hydrabase_client_obj
docker_resolve_url = docker_resolve_url_fn
docker_resolve_path = docker_resolve_path_fn
def run_service_test(service, test_config):
"""
Performs the actual connection test for a given service.
This logic is adapted from your ServiceTestThread.
It temporarily modifies the config, runs the test, then restores the config.
"""
original_config = {}
try:
# 1. Save original config for the specific service
original_config = config_manager.get(service, {})
# 2. Temporarily set the new config for the test (with Docker URL resolution)
for key, value in test_config.items():
# Apply Docker URL resolution for URL/URI fields
if isinstance(value, str) and ('url' in key.lower() or 'uri' in key.lower()):
value = docker_resolve_url(value)
config_manager.set(f"{service}.{key}", value)
# 3. Run the test with the temporary config
if service == "spotify":
temp_client = SpotifyClient()
# Check if Spotify credentials are configured
spotify_config = config_manager.get('spotify', {})
spotify_configured = bool(spotify_config.get('client_id') and spotify_config.get('client_secret'))
if temp_client.is_authenticated():
# Determine which source is active
if temp_client.is_spotify_authenticated():
return True, "Spotify connection successful!"
else:
# Spotify-Free (no-auth) metadata path: officially unauthenticated,
# but the no-creds source is selected and available. Report it as the
# working source rather than the generic Deezer/Discogs/iTunes fallback.
try:
spotify_free_available = temp_client.is_spotify_metadata_available()
except Exception:
spotify_free_available = False
if spotify_free_available:
return True, "Spotify (no-auth) connection successful!"
# Using a different fallback metadata source
fb_src = _get_metadata_fallback_source()
fallback_name = 'Deezer' if fb_src == 'deezer' else 'Discogs' if fb_src == 'discogs' else 'iTunes'
if spotify_configured:
return True, f"{fallback_name} connection successful! (Spotify configured but not authenticated)"
else:
return True, f"{fallback_name} connection successful! (Spotify not configured)"
else:
return False, "Music service authentication failed. Check credentials and complete OAuth flow in browser if prompted."
elif service == "tidal":
temp_client = TidalClient()
if temp_client.is_authenticated():
user_info = temp_client.get_user_info()
username = user_info.get('display_name', 'Tidal User') if user_info else 'Tidal User'
return True, f"Tidal connection successful! Connected as: {username}"
else:
return False, "Tidal authentication failed. Please use the 'Authenticate' button and complete the flow in your browser."
elif service == "plex":
temp_client = PlexClient()
if temp_client.is_connected():
return True, f"Successfully connected to Plex server: {temp_client.server.friendlyName}"
else:
return False, "Could not connect to Plex. Check URL and Token."
elif service == "jellyfin":
temp_client = JellyfinClient()
if temp_client.is_connected():
# FIX: Check if server_info exists before accessing it.
server_name = "Unknown Server"
if hasattr(temp_client, 'server_info') and temp_client.server_info:
server_name = temp_client.server_info.get('ServerName', 'Unknown Server')
return True, f"Successfully connected to Jellyfin server: {server_name}"
else:
return False, "Could not connect to Jellyfin. Check URL and API Key."
elif service == "navidrome":
# Test Navidrome connection using Subsonic API
base_url = test_config.get('base_url', '')
username = test_config.get('username', '')
password = test_config.get('password', '')
if not all([base_url, username, password]):
return False, "Missing Navidrome URL, username, or password."
try:
import hashlib
import random
import string
# Generate salt and token for Subsonic API authentication
salt = ''.join(random.choices(string.ascii_letters + string.digits, k=6))
token = hashlib.md5((password + salt).encode()).hexdigest()
# Test ping endpoint
url = f"{base_url.rstrip('/')}/rest/ping"
response = requests.get(url, params={
'u': username,
't': token,
's': salt,
'v': '1.16.1',
'c': 'soulsync',
'f': 'json'
}, timeout=5)
if response.status_code == 200:
data = response.json()
if data.get('subsonic-response', {}).get('status') == 'ok':
server_version = data.get('subsonic-response', {}).get('version', 'Unknown')
return True, f"Successfully connected to Navidrome server (v{server_version})"
else:
error = data.get('subsonic-response', {}).get('error', {})
return False, f"Navidrome authentication failed: {error.get('message', 'Unknown error')}"
else:
return False, f"Could not connect to Navidrome server (HTTP {response.status_code})"
except Exception as e:
return False, f"Navidrome connection error: {str(e)}"
elif service == "soulsync":
transfer_path = docker_resolve_path(config_manager.get('soulseek.transfer_path', './Transfer'))
if os.path.isdir(transfer_path):
# Quick check — count a few audio files to confirm it's a music folder
audio_exts = {'.mp3', '.flac', '.ogg', '.opus', '.m4a', '.aac', '.wav'}
count = 0
found_enough = False
for _root, _dirs, files in os.walk(transfer_path):
for f in files:
if os.path.splitext(f)[1].lower() in audio_exts:
count += 1
if count >= 10:
found_enough = True
break
if found_enough:
break
return True, f"SoulSync standalone ready! Output folder: {transfer_path}" + (f" ({count}+ audio files)" if count > 0 else " (empty)")
else:
return False, f"Output folder not found: {transfer_path}"
elif service == "soulseek":
if download_orchestrator is None:
return False, "Download orchestrator failed to initialize. Check server logs for startup errors."
# Test the orchestrator's configured download source (not just Soulseek)
download_mode = config_manager.get('download_source.mode', 'hybrid')
if run_async(download_orchestrator.check_connection()):
# Success message based on active mode
mode_messages = {
'soulseek': "Successfully connected to Soulseek network via slskd.",
'youtube': "YouTube download source ready.",
'tidal': "Tidal download source ready.",
'qobuz': "Qobuz download source ready.",
'hifi': "HiFi download source ready.",
'deezer_dl': "Deezer download source ready.",
'amazon': "Amazon download source ready.",
'lidarr': "Lidarr download source ready.",
'soundcloud': "SoundCloud download source ready.",
'torrent': "Torrent download source ready.",
'usenet': "Usenet download source ready.",
'hybrid': "Download sources ready (Hybrid mode)."
}
message = mode_messages.get(download_mode, "Download source connected.")
return True, message
else:
# Failure message based on active mode
mode_errors = {
'soulseek': "slskd is not connected to the Soulseek network. Check slskd status and credentials.",
'youtube': "YouTube download source not available.",
'tidal': "Tidal download source not available. Check authentication.",
'qobuz': "Qobuz download source not available. Check authentication.",
'hifi': "HiFi download source not available. Public API instances may be down.",
'deezer_dl': "Deezer download source not available. Check authentication.",
'amazon': "Amazon download source not available.",
'lidarr': "Lidarr download source not available. Check Lidarr URL and API key.",
'soundcloud': "SoundCloud download source not available.",
'torrent': "Torrent download source not available. Check Prowlarr and torrent client settings.",
'usenet': "Usenet download source not available. Check Prowlarr and usenet client settings.",
'hybrid': "Could not connect to download sources. Check configuration."
}
error = mode_errors.get(download_mode, "Download source connection failed.")
return False, error
elif service == "listenbrainz":
token = test_config.get('token', '')
if not token:
return False, "Missing ListenBrainz user token."
try:
# Test ListenBrainz API by validating the token
custom_base = test_config.get('base_url', '').rstrip('/')
if custom_base:
if not custom_base.endswith('/1'):
custom_base += '/1'
lb_api_base = custom_base
else:
lb_api_base = "https://api.listenbrainz.org/1"
url = f"{lb_api_base}/validate-token"
headers = {
'Authorization': f'Token {token}'
}
response = requests.get(url, headers=headers, timeout=5)
if response.status_code == 200:
data = response.json()
if data.get('valid'):
username = data.get('user_name', 'Unknown')
return True, f"Successfully connected to ListenBrainz! Connected as: {username}"
else:
return False, "Invalid ListenBrainz token."
elif response.status_code == 401:
return False, "Invalid ListenBrainz token (unauthorized)."
else:
return False, f"Could not connect to ListenBrainz (HTTP {response.status_code})"
except Exception as e:
return False, f"ListenBrainz connection error: {str(e)}"
elif service == "acoustid":
api_key = test_config.get('api_key', '')
if not api_key:
return False, "Missing AcoustID API key."
try:
from core.acoustid_client import AcoustIDClient, CHROMAPRINT_AVAILABLE, ACOUSTID_AVAILABLE, FPCALC_PATH
if not ACOUSTID_AVAILABLE:
return False, "pyacoustid library not installed. Run: pip install pyacoustid"
client = AcoustIDClient()
# Override the cached API key with the test config key
client._api_key = api_key
# Check chromaprint/fpcalc availability
if CHROMAPRINT_AVAILABLE and FPCALC_PATH:
fingerprint_status = f"fpcalc ready: {FPCALC_PATH}"
elif CHROMAPRINT_AVAILABLE:
fingerprint_status = "Fingerprint backend available"
else:
fingerprint_status = "fpcalc not found (will auto-download on first use)"
# Validate API key with test request
success, message = client.test_api_key()
if success:
return True, f"AcoustID API key is valid! {fingerprint_status}"
else:
return False, f"{message}. {fingerprint_status}"
except Exception as e:
return False, f"AcoustID test error: {str(e)}"
elif service == "lastfm":
api_key = test_config.get('api_key', '')
if not api_key:
return False, "Missing Last.fm API key."
try:
from core.lastfm_client import LastFMClient
client = LastFMClient(api_key=api_key)
if client.validate_api_key():
return True, "Successfully connected to Last.fm!"
else:
return False, "Invalid Last.fm API key."
except Exception as e:
return False, f"Last.fm connection error: {str(e)}"
elif service == "genius":
access_token = test_config.get('access_token', '')
if not access_token:
return False, "Missing Genius access token."
try:
from core.genius_client import GeniusClient
client = GeniusClient(access_token=access_token)
if client.validate_token():
return True, "Successfully connected to Genius!"
else:
return False, "Invalid Genius access token."
except Exception as e:
return False, f"Genius connection error: {str(e)}"
elif service == "usenet_client":
client_type = (config_manager.get('usenet_client.type', '') or '').strip().lower()
url = config_manager.get('usenet_client.url', '')
if not url:
return False, "Usenet client URL is required."
if not client_type:
return False, "Pick a usenet client (SABnzbd or NZBGet)."
try:
from core.usenet_clients import adapter_for_type as _usenet_adapter_for_type
adapter = _usenet_adapter_for_type(client_type)
if adapter is None:
return False, f"Unknown usenet client type: {client_type}"
if not adapter.is_configured():
if client_type == "sabnzbd":
return False, "SABnzbd needs both URL and API key."
return False, "NZBGet needs URL, username, and password."
if run_async(adapter.check_connection()):
return True, f"Connected to {client_type}"
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
except Exception as e:
return False, f"Usenet client connection error: {str(e)}"
elif service == "torrent_client":
client_type = (config_manager.get('torrent_client.type', '') or '').strip().lower()
url = config_manager.get('torrent_client.url', '')
if not url:
return False, "Torrent client URL is required."
if not client_type:
return False, "Pick a torrent client (qBittorrent, Transmission, or Deluge)."
try:
from core.torrent_clients import adapter_for_type
adapter = adapter_for_type(client_type)
if adapter is None:
return False, f"Unknown torrent client type: {client_type}"
if not adapter.is_configured():
return False, "Torrent client missing required credentials."
if run_async(adapter.check_connection()):
return True, f"Connected to {client_type}"
return False, f"{client_type} probe failed — check URL, credentials, and that the client is running."
except Exception as e:
return False, f"Torrent client connection error: {str(e)}"
elif service == "prowlarr":
url = config_manager.get('prowlarr.url', '')
api_key = config_manager.get('prowlarr.api_key', '')
if not url or not api_key:
return False, "Prowlarr URL and API key are required."
try:
import requests as _req
resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
headers={'X-Api-Key': api_key}, timeout=10)
if resp.ok:
version = resp.json().get('version', '?')
return True, f"Connected to Prowlarr v{version}"
return False, f"Prowlarr returned HTTP {resp.status_code}"
except Exception as e:
return False, f"Prowlarr connection error: {str(e)}"
elif service == "lidarr" or service == "lidarr_download":
url = config_manager.get('lidarr_download.url', '')
api_key = config_manager.get('lidarr_download.api_key', '')
if not url or not api_key:
return False, "Lidarr URL and API key are required."
try:
import requests as _req
resp = _req.get(f"{url.rstrip('/')}/api/v1/system/status",
headers={'X-Api-Key': api_key}, timeout=10)
if resp.ok:
version = resp.json().get('version', '?')
return True, f"Connected to Lidarr v{version}"
return False, f"Lidarr returned HTTP {resp.status_code}"
except Exception as e:
return False, f"Lidarr connection error: {str(e)}"
elif service == "itunes":
# Public API — just confirm we can reach it with a cheap search
try:
storefront = config_manager.get('itunes.storefront', 'US') or 'US'
resp = requests.get(
'https://itunes.apple.com/search',
params={'term': 'beatles', 'limit': 1, 'country': storefront, 'media': 'music'},
timeout=5,
)
if resp.ok and resp.json().get('resultCount', 0) >= 0:
return True, f"iTunes Search API reachable (storefront: {storefront})"
return False, f"iTunes returned HTTP {resp.status_code}"
except Exception as e:
return False, f"iTunes connection error: {str(e)}"
elif service == "deezer":
# Public API — anon search works without credentials
try:
resp = requests.get(
'https://api.deezer.com/search/artist',
params={'q': 'beatles', 'limit': 1},
timeout=5,
)
if resp.ok and isinstance(resp.json(), dict):
return True, "Deezer Public API reachable"
return False, f"Deezer returned HTTP {resp.status_code}"
except Exception as e:
return False, f"Deezer connection error: {str(e)}"
elif service == "discogs":
token = test_config.get('token', '') or config_manager.get('discogs.token', '')
if not token:
return False, "Missing Discogs personal token."
try:
resp = requests.get(
'https://api.discogs.com/database/search',
params={'q': 'beatles', 'per_page': 1},
headers={'Authorization': f'Discogs token={token}', 'User-Agent': 'SoulSync/1.0'},
timeout=10,
)
if resp.ok:
return True, "Discogs API reachable with provided token"
if resp.status_code == 401:
return False, "Discogs token rejected (HTTP 401)"
return False, f"Discogs returned HTTP {resp.status_code}"
except Exception as e:
return False, f"Discogs connection error: {str(e)}"
elif service == "qobuz":
try:
if qobuz_enrichment_worker and qobuz_enrichment_worker.client and qobuz_enrichment_worker.client.is_authenticated():
return True, "Qobuz client authenticated"
return False, "Qobuz not authenticated. Provide email/password or user auth token."
except Exception as e:
return False, f"Qobuz connection error: {str(e)}"
elif service == "hydrabase":
try:
if hydrabase_client and hydrabase_client.is_connected():
return True, "Hydrabase connected"
return False, "Hydrabase not connected. Configure URL + API key and click Connect."
except Exception as e:
return False, f"Hydrabase connection error: {str(e)}"
elif service == "musicbrainz":
try:
from core.metadata.registry import get_musicbrainz_client
mb = get_musicbrainz_client()
results = mb.search_artists("radiohead", limit=1)
if results:
return True, "MusicBrainz reachable"
return False, "MusicBrainz returned no results — may be rate-limited or unreachable."
except Exception as e:
return False, f"MusicBrainz connection error: {str(e)}"
elif service == "soundcloud":
# Anonymous SoundCloud has no auth, so "test" really means
# "is yt-dlp installed and can it reach SoundCloud right now."
# This mirrors the /api/soundcloud/status check.
try:
from core.soundcloud_client import SoundcloudClient
sc = SoundcloudClient()
if not sc.is_available():
return False, "SoundCloud unavailable — yt-dlp not installed."
# Run a tiny live probe via asyncio so the dashboard test
# gives a meaningful pass/fail.
import asyncio
reachable = asyncio.new_event_loop().run_until_complete(sc.check_connection())
if reachable:
return True, "SoundCloud reachable (anonymous)"
return False, "SoundCloud unreachable — search probe failed. Try again."
except Exception as e:
return False, f"SoundCloud connection error: {str(e)}"
return False, "Unknown service."
except AttributeError as e:
# This specifically catches the error you reported for Jellyfin
if "'JellyfinClient' object has no attribute 'server_info'" in str(e):
return False, "Connection failed. Please check your Jellyfin URL and API Key."
else:
return False, f"An unexpected error occurred: {e}"
except Exception as e:
import traceback
traceback.print_exc()
return False, str(e)
finally:
# 4. CRITICAL: Restore the original config
if original_config:
for key, value in original_config.items():
config_manager.set(f"{service}.{key}", value)
logger.debug(f"Restored original config for '{service}' after test.")

0
core/credentials/__init__.py
View file

85
core/credentials/store.py
View file

@ -1,85 +0,0 @@
"""Named, switchable service-credential sets — pure logic (Phase 0 foundation).
Today every auth service (Spotify, Tidal, Deezer, Qobuz, Plex, Jellyfin,
Navidrome) holds ONE credential set in config, and clients are global singletons
built from that single slot. This module is the groundwork for letting an admin
save MULTIPLE named credential sets per service ("pills") that each profile can
switch between, without anyone but the admin creating them.
Kept PURE — service registry, payload validation, and active-set selection,
free of DB/Flask so it's unit-testable. Encrypted storage lives in MusicDatabase
(service_credentials / profile_service_credentials tables); runtime client
resolution + UI come in later phases. Nothing here changes existing behaviour;
it's dormant capability until wired.
"""
from __future__ import annotations
# Services that support multiple named credential sets, mapped to the payload
# keys that MUST be present for a set to be usable. Extra keys (OAuth tokens,
# redirect URIs, quality prefs) are allowed and preserved — these are only the
# minimum required to validate a set the admin is saving.
SERVICE_CREDENTIAL_SCHEMA = {
'spotify': ('client_id', 'client_secret'),
'tidal': ('access_token', 'refresh_token'),
'deezer': ('arl',),
'qobuz': ('user_auth_token',),
'plex': ('base_url', 'token'),
'jellyfin': ('base_url', 'api_key'),
'navidrome': ('base_url', 'username', 'password'),
}
SUPPORTED_SERVICES = frozenset(SERVICE_CREDENTIAL_SCHEMA)
def is_supported_service(service: str) -> bool:
"""True when the service supports named credential sets."""
return service in SERVICE_CREDENTIAL_SCHEMA
def validate_credential_payload(service: str, payload):
"""Return ``(ok, missing_keys)`` for a credential set.
Valid when every required key for the service is present and truthy. An
unknown service is invalid with no missing list (caller should reject it
as unsupported, not as "incomplete").
"""
required = SERVICE_CREDENTIAL_SCHEMA.get(service)
if required is None:
return False, []
if not isinstance(payload, dict):
return False, list(required)
def _present(v):
# Whitespace-only strings count as missing — they'd otherwise save a
# blank secret that fails confusingly at the real service later.
return bool(v.strip()) if isinstance(v, str) else bool(v)
missing = [k for k in required if not _present(payload.get(k))]
return (not missing), missing
def pick_active_credential(credentials, selected_id):
"""From ``credentials`` (a list of dicts each carrying ``id``), return the
one whose id == ``selected_id``.
Returns None when there's no selection OR the selected id isn't present —
i.e. a stale pointer whose credential set was deleted. The caller then
falls back to the global/admin default, so a deleted set never breaks a
profile. Pure + stale-safe.
"""
if not selected_id:
return None
for cred in credentials or []:
if cred.get('id') == selected_id:
return cred
return None
__all__ = [
'SERVICE_CREDENTIAL_SCHEMA',
'SUPPORTED_SERVICES',
'is_supported_service',
'validate_credential_payload',
'pick_active_credential',
]

82
core/database_update_health.py
View file

@ -1,82 +0,0 @@
"""Stall detection for the database-update job.
The DB updater keeps a single in-memory state dict whose ``status`` is set to
``running`` at start and only flipped to ``finished``/``error`` by the worker's
completion/error callbacks. If the worker thread hangs — e.g. a media-server API
call with no timeout, a DB lock — those callbacks never fire, so ``status`` stays
``running`` forever and the UI shows a frozen progress bar with no way to recover
(GitHub #859).
This module is the single, *pure* decision for "is a running job stalled?". It
takes the state dict plus the current wall-clock time and a timeout, and answers
yes/no — no DB, no globals, no clock of its own. That keeps it unit-testable and
lets the watchdog wiring in web_server.py stay a thin call. The job carries a
``last_progress_at`` epoch timestamp that the start path and every progress/phase
callback bump; staleness is simply "running, and that timestamp is older than the
timeout".
"""
from __future__ import annotations
from typing import Any, Mapping
# 5 minutes with zero forward progress = presumed hung. A healthy scan ticks
# progress (per-artist) far more often than this even for large libraries, so
# the timeout won't false-positive a slow-but-working run.
DEFAULT_STALL_TIMEOUT_SECONDS = 300
def is_db_update_stalled(
state: Mapping[str, Any],
now: float,
timeout_seconds: float = DEFAULT_STALL_TIMEOUT_SECONDS,
) -> bool:
"""Return True when the job is ``running`` but has made no progress within
``timeout_seconds``.
Conservative by design — it only ever reports a stall it can prove:
- Only a ``running`` job can stall (idle/finished/error never do).
- With no usable ``last_progress_at`` timestamp we cannot judge, so we return
False rather than risk killing a job we have no clock for.
- A non-positive timeout is treated as "disabled" (never stalls).
"""
if not isinstance(state, Mapping):
return False
if state.get("status") != "running":
return False
if timeout_seconds is None or timeout_seconds <= 0:
return False
last = state.get("last_progress_at")
if not last:
return False
try:
elapsed = float(now) - float(last)
except (TypeError, ValueError):
return False
return elapsed >= float(timeout_seconds)
def stalled_error_message(state: Mapping[str, Any], now: float) -> str:
"""Build a clear, human-facing message for a stalled job, including how long
it has been silent and the phase it died in."""
last = state.get("last_progress_at") if isinstance(state, Mapping) else None
phase = state.get("phase") if isinstance(state, Mapping) else None
try:
secs = int(float(now) - float(last)) if last else 0
except (TypeError, ValueError):
secs = 0
msg = "Update appears stuck — no progress"
if secs > 0:
msg += f" for {secs}s"
if phase:
msg += f" (last phase: {phase})"
msg += (". The worker may be hung on the media server. Start a new update "
"to try again, or restart SoulSync if it keeps stalling.")
return msg
__all__ = [
"DEFAULT_STALL_TIMEOUT_SECONDS",
"is_db_update_stalled",
"stalled_error_message",
]

265
core/database_update_worker.py
View file

@ -1,5 +1,35 @@
#!/usr/bin/env python3
# Conditional PyQt6 import for backward compatibility with GUI version
try:
from PyQt6.QtCore import QThread, pyqtSignal
QT_AVAILABLE = True
except ImportError:
QT_AVAILABLE = False
# Define dummy classes for headless operation
class QThread:
def __init__(self):
self.callbacks = {}
def start(self):
import threading
self.thread = threading.Thread(target=self.run)
self.thread.daemon = True
self.thread.start()
def wait(self):
if hasattr(self, 'thread'):
self.thread.join()
def emit_signal(self, signal_name, *args):
if signal_name in self.callbacks:
for callback in self.callbacks[signal_name]:
try:
callback(*args)
except Exception as e:
logger.error(f"Error in callback for {signal_name}: {e}")
def connect_signal(self, signal_name, callback):
if signal_name not in self.callbacks:
self.callbacks[signal_name] = []
self.callbacks[signal_name].append(callback)
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Optional, List, Callable
@ -12,19 +42,32 @@ from config.settings import config_manager
logger = get_logger("database_update_worker")
class DatabaseUpdateWorker:
"""Worker for updating SoulSync database with media server library data."""
class DatabaseUpdateWorker(QThread):
"""Worker thread for updating SoulSync database with media server library data (Plex or Jellyfin)"""
# Qt signals (only available when PyQt6 is installed)
if QT_AVAILABLE:
progress_updated = pyqtSignal(str, int, int, float) # current_item, processed, total, percentage
artist_processed = pyqtSignal(str, bool, str, int, int) # artist_name, success, details, albums_count, tracks_count
finished = pyqtSignal(int, int, int, int, int) # total_artists, total_albums, total_tracks, successful, failed
error = pyqtSignal(str) # error_message
phase_changed = pyqtSignal(str) # current_phase (artists, albums, tracks)
def __init__(self, media_client, database_path: str = "database/music_library.db", full_refresh: bool = False, server_type: str = "plex", force_sequential: bool = False):
super().__init__()
# Force sequential processing for web server mode to avoid threading issues
self.force_sequential = force_sequential
self.callbacks = {
'progress_updated': [],
'artist_processed': [],
'finished': [],
'error': [],
'phase_changed': [],
}
# Initialize signal callbacks for headless mode
if not QT_AVAILABLE:
self.callbacks = {
'progress_updated': [],
'artist_processed': [],
'finished': [],
'error': [],
'phase_changed': []
}
# Support both old plex_client parameter and new media_client parameter for backward compatibility
if hasattr(media_client, '__class__') and 'plex' in media_client.__class__.__name__.lower():
@ -41,19 +84,7 @@ class DatabaseUpdateWorker:
self.database_path = database_path
self.full_refresh = full_refresh
self.should_stop = False
# Track ids of rows newly INSERTED this run (not updates). The web
# layer reads this to gap-fill embedded provider IDs for the new files
# (auto-reconcile), so newly-added music contributes its
# Spotify/MusicBrainz/etc. ids without a manual backfill.
self._new_track_ids = set()
# Optional callback(worker) run as the FINAL scan phase, immediately
# before the 'finished' signal — so the auto-reconcile is inside the
# scan's running window (automations/UI treat it as a normal phase and
# wait for it). Injected by the web layer (which owns path resolution).
self.post_scan_hook = None
# Statistics tracking
self.processed_artists = 0
self.processed_albums = 0
@ -85,35 +116,19 @@ class DatabaseUpdateWorker:
self.database: Optional[MusicDatabase] = None
def _emit_signal(self, signal_name: str, *args):
"""Emit a signal through the callback registry."""
for callback in self.callbacks.get(signal_name, []):
try:
callback(*args)
except Exception as e:
logger.error(f"Error in callback for {signal_name}: {e}")
def _emit_finished(self, *args):
"""Run the post-scan hook (auto-reconcile) as the final phase, THEN
emit 'finished'.
Running the hook before 'finished' keeps the scan's status at
'running' through the reconcile, so every caller (automations that
poll for completion, the dashboard card, the Tools page) treats it as
a normal scan phase and waits for it — rather than seeing 'finished'
and missing the tail. Best-effort: a hook failure never blocks the
completion signal.
"""
if self.post_scan_hook:
try:
self.post_scan_hook(self)
except Exception as e:
logger.warning(f"post-scan hook failed (non-fatal): {e}")
self._emit_signal('finished', *args)
"""Emit a signal in both Qt and headless modes"""
if QT_AVAILABLE and hasattr(self, signal_name):
# Qt mode - use actual signal
getattr(self, signal_name).emit(*args)
elif not QT_AVAILABLE:
# Headless mode - use callback system
self.emit_signal(signal_name, *args)
def connect_callback(self, signal_name: str, callback: Callable):
"""Connect a callback for progress notifications."""
self.callbacks.setdefault(signal_name, []).append(callback)
"""Connect a callback for headless mode"""
if not QT_AVAILABLE:
self.connect_signal(signal_name, callback)
# In Qt mode, use the normal signal.connect() method
def stop(self):
"""Stop the database update process"""
@ -177,7 +192,7 @@ class DatabaseUpdateWorker:
logger.info(f"Merged {merged} duplicate artists")
except Exception as e:
logger.warning(f"Could not merge duplicate artists: {e}")
self._emit_finished(0, 0, 0, 0, 0)
self._emit_signal('finished', 0, 0, 0, 0, 0)
return
logger.info(f"Incremental update: Found {len(artists_to_process)} artists to process")
@ -261,7 +276,7 @@ class DatabaseUpdateWorker:
self.removed_tracks = removal.get('tracks_removed', 0) if removal else 0
# Emit final results
self._emit_finished(
self._emit_signal('finished',
self.processed_artists,
self.processed_albums,
self.processed_tracks,
@ -362,7 +377,7 @@ class DatabaseUpdateWorker:
f"{self.processed_albums} albums, {self.processed_tracks} new tracks, "
f"{stale_removed} stale tracks removed")
self._emit_finished(
self._emit_signal('finished',
self.processed_artists,
self.processed_albums,
self.processed_tracks,
@ -379,7 +394,7 @@ class DatabaseUpdateWorker:
total_artists = len(artists)
logger.info(f"Deep scan: Processing {total_artists} artists (sequential, skip-existing mode)")
for _i, artist in enumerate(artists):
for i, artist in enumerate(artists):
if self.should_stop:
break
@ -438,14 +453,9 @@ class DatabaseUpdateWorker:
logger.error(f"Could not connect to {self.server_type} server — check URL, credentials, and network (Docker users: use container name or host.docker.internal instead of host IP)")
return []
# Check for music library (Plex-specific check). Routes
# through ``is_fully_configured`` so all-libraries mode (in
# which ``music_library`` is None but ``_all_libraries_mode``
# is True) counts as configured. Pre-fix this bailed out on
# the bare music_library None check, silently aborting the
# deep scan for any all-libraries-mode user.
if self.server_type == "plex" and not self.media_client.is_fully_configured():
logger.error("No music library configured in Plex")
# Check for music library (Plex-specific check)
if self.server_type == "plex" and not self.media_client.music_library:
logger.error("No music library found in Plex")
return []
# Check if database has enough content for incremental updates (server-specific)
@ -635,7 +645,7 @@ class DatabaseUpdateWorker:
result_msg = f"Smart incremental scan result: {len(artists_to_process)} artists to process from {albums_with_new_content} albums with new content"
if stopped_early:
result_msg += " (stopped early after finding 25 consecutive complete albums)"
result_msg += f" (stopped early after finding 25 consecutive complete albums)"
else:
result_msg += f" (checked all {total_tracks_checked} tracks from {len(recent_albums)} recent albums)"
@ -911,8 +921,6 @@ class DatabaseUpdateWorker:
track_success = self.database.insert_or_update_media_track(track, album_id, artist_id, server_source=self.server_type)
if track_success:
total_processed_tracks += 1
if track_success == 'inserted':
self._new_track_ids.add(str(track.ratingKey))
logger.debug(f"Processed new track: {track.title}")
except Exception as e:
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
@ -968,16 +976,10 @@ class DatabaseUpdateWorker:
if (db_track.title != current_title or
db_track.artist_name != current_artist or
db_track.album_title != current_album):
logger.debug(
"Metadata change detected for track %s: title=%r→%r artist=%r→%r album=%r→%r",
track_id,
db_track.title,
current_title,
db_track.artist_name,
current_artist,
db_track.album_title,
current_album,
)
logger.debug(f"Metadata change detected for track ID {track_id}:")
logger.debug(f" Title: '{db_track.title}' → '{current_title}'")
logger.debug(f" Artist: '{db_track.artist_name}' → '{current_artist}'")
logger.debug(f" Album: '{db_track.album_title}' → '{current_album}'")
changes_detected += 1
except Exception as e:
@ -1084,8 +1086,8 @@ class DatabaseUpdateWorker:
batch + [self.server_type])
cascade_album_ids.update(row[0] for row in cursor.fetchall())
removed_album_ids -= cascade_album_ids
except Exception as e:
logger.debug("cascade album cleanup optimization: %s", e)
except Exception:
pass # If this optimization fails, double-delete is harmless
if not removed_artist_ids and not removed_album_ids:
logger.info("Removal detection: no stale content found")
@ -1122,31 +1124,24 @@ class DatabaseUpdateWorker:
return []
def _get_recent_albums_plex(self) -> List:
"""Get recently added and updated albums from Plex.
Routes through ``PlexClient.get_recently_added_albums`` and
``get_recently_updated_albums`` so the all-libraries mode union
works (pre-fix this reached ``self.media_client.music_library.X``
directly which crashed when music_library is None in all-
libraries mode).
"""
"""Get recently added and updated albums from Plex"""
all_recent_content = []
try:
# Get recently added albums (up to 400 to catch more recent content)
recently_added = self.media_client.get_recently_added_albums(maxresults=400, libtype='album')
if recently_added:
# Get recently added albums (up to 400 to catch more recent content)
try:
recently_added = self.media_client.music_library.recentlyAdded(libtype='album', maxresults=400)
all_recent_content.extend(recently_added)
logger.info(f"Found {len(recently_added)} recently added albums")
else:
# Fallback to mixed-type recents.
recently_added = self.media_client.get_recently_added_albums(maxresults=400, libtype=None)
all_recent_content.extend(recently_added or [])
logger.info(f"Found {len(recently_added or [])} recently added items (mixed types)")
except:
# Fallback to general recently added
recently_added = self.media_client.music_library.recentlyAdded(maxresults=400)
all_recent_content.extend(recently_added)
logger.info(f"Found {len(recently_added)} recently added items (mixed types)")
# Get recently updated albums (catches metadata corrections)
try:
recently_updated = self.media_client.get_recently_updated_albums(limit=400)
recently_updated = self.media_client.music_library.search(sort='updatedAt:desc', libtype='album', limit=400)
# Remove duplicates (items that are both recently added and updated)
added_keys = {getattr(item, 'ratingKey', None) for item in all_recent_content}
unique_updated = [item for item in recently_updated if getattr(item, 'ratingKey', None) not in added_keys]
@ -1266,10 +1261,10 @@ class DatabaseUpdateWorker:
logger.error(f"Error processing artist {getattr(artist, 'title', 'Unknown')}: {e}")
return (getattr(artist, 'title', 'Unknown'), False, f"Error: {str(e)}", 0, 0)
# Process artists sequentially when requested (the web server uses this path).
if self.force_sequential:
# Process artists - use sequential processing in web server mode to avoid threading issues
if not QT_AVAILABLE or self.force_sequential:
# Sequential processing for web server mode
for _i, artist in enumerate(artists):
for i, artist in enumerate(artists):
if self.should_stop:
break
@ -1282,7 +1277,7 @@ class DatabaseUpdateWorker:
# Emit progress signal
self._emit_signal('artist_processed', artist_name, success, details, album_count, track_count)
else:
# Parallel processing for local/manual runs
# Process artists in parallel using ThreadPoolExecutor (Qt mode only)
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
# Submit all tasks
future_to_artist = {executor.submit(process_single_artist, artist): artist
@ -1377,8 +1372,6 @@ class DatabaseUpdateWorker:
skipped_count += 1
elif track_success:
track_count += 1
if track_success == 'inserted':
self._new_track_ids.add(track_id_str)
except Exception as e:
logger.warning(f"Failed to process track '{getattr(track, 'title', 'Unknown')}': {e}")
@ -1416,3 +1409,71 @@ class DatabaseUpdateWorker:
except Exception as e:
logger.error(f"Error in run_with_callback: {e}")
class DatabaseStatsWorker(QThread):
"""Simple worker for getting database statistics without blocking UI"""
# Qt signals (only available when PyQt6 is installed)
if QT_AVAILABLE:
stats_updated = pyqtSignal(dict) # Database statistics
def __init__(self, database_path: str = "database/music_library.db"):
super().__init__()
self.database_path = database_path
self.should_stop = False
# Initialize signal callbacks for headless mode
if not QT_AVAILABLE:
self.callbacks = {
'stats_updated': []
}
def stop(self):
"""Stop the worker"""
self.should_stop = True
def _emit_signal(self, signal_name: str, *args):
"""Emit a signal in both Qt and headless modes"""
if QT_AVAILABLE and hasattr(self, signal_name):
# Qt mode - use actual signal
getattr(self, signal_name).emit(*args)
elif not QT_AVAILABLE:
# Headless mode - use callback system
self.emit_signal(signal_name, *args)
def connect_callback(self, signal_name: str, callback: Callable):
"""Connect a callback for headless mode"""
if not QT_AVAILABLE:
self.connect_signal(signal_name, callback)
# In Qt mode, use the normal signal.connect() method
def run(self):
"""Get database statistics and full info including last refresh"""
try:
if self.should_stop:
return
database = get_database(self.database_path)
if self.should_stop:
return
# Get database info for active server (server-aware statistics)
info = database.get_database_info_for_server()
if not self.should_stop:
self._emit_signal('stats_updated', info)
except Exception as e:
logger.error(f"Error getting database stats: {e}")
if not self.should_stop:
# Import here to avoid circular imports
from config.settings import config_manager
active_server = config_manager.get_active_media_server()
self._emit_signal('stats_updated', {
'artists': 0,
'albums': 0,
'tracks': 0,
'database_size_mb': 0.0,
'last_update': None,
'last_full_refresh': None,
'server_source': active_server
})

153
core/db_integrity.py
View file

@ -1,153 +0,0 @@
"""SQLite integrity + safe-backup helpers.
Born out of a real incident: a WAL-mode DB got corrupted (most likely an
interrupted write during a hard restart), and because the backup routine
(a) never checked integrity and (b) rotated the oldest backup out by mtime,
every rolling backup ended up being a faithful copy of the already-corrupt
file — so when recovery was needed, all snapshots were poisoned.
This module makes that impossible:
* ``quick_check(path)`` / ``is_healthy(path)`` — fast read-only integrity probe.
* ``safe_backup(...)`` — verifies the SOURCE is healthy before copying, uses the
SQLite Online Backup API, then verifies the RESULT. A corrupt source never
produces (or keeps) a backup.
* ``prune_backups(...)`` — rotation that NEVER deletes the most recent
*verified-healthy* backup, even to honor the max-count, so a run of bad
backups can't evict your last good one.
Pure-ish: only touches sqlite3 + the filesystem paths it's given; no Flask, no
app globals. Unit-testable with real (and deliberately-corrupted) temp DBs.
"""
from __future__ import annotations
import logging
import os
import sqlite3
from typing import Optional
logger = logging.getLogger("db_integrity")
def _close_quietly(conn) -> None:
"""Best-effort close; a failure to close during cleanup must not mask the
real error we're handling, but we log it rather than swallow silently."""
if conn is None:
return
try:
conn.close()
except Exception as e: # noqa: BLE001 — cleanup path, real error already in flight
logger.debug("db_integrity: connection close failed: %s", e)
class DBIntegrityError(Exception):
"""Raised when a database fails its integrity check."""
def quick_check(db_path: str, *, timeout: float = 30.0) -> str:
"""Run ``PRAGMA quick_check`` read-only and return its first result row.
Returns ``'ok'`` for a healthy DB, otherwise the first error line. Raises
``DBIntegrityError`` if the file can't even be opened/read (malformed
header, I/O error) — i.e. unambiguously bad.
"""
if not os.path.exists(db_path):
raise DBIntegrityError(f"Database file not found: {db_path}")
conn = None
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=timeout)
row = conn.execute("PRAGMA quick_check(1)").fetchone()
return (row[0] if row else "no result")
except sqlite3.DatabaseError as e:
# malformed header / disk image malformed / disk I/O error
raise DBIntegrityError(f"{db_path}: {e}") from e
finally:
_close_quietly(conn)
def is_healthy(db_path: str, *, timeout: float = 30.0) -> bool:
"""True iff the DB opens and ``quick_check`` reports 'ok'. Never raises."""
try:
return quick_check(db_path, timeout=timeout) == "ok"
except DBIntegrityError:
return False
def safe_backup(src_path: str, dst_path: str, *, verify_source: bool = True,
verify_result: bool = True) -> None:
"""Back up ``src_path`` to ``dst_path`` via the SQLite Online Backup API,
refusing to produce a backup from (or keep a backup of) a corrupt DB.
Raises ``DBIntegrityError`` and removes any partial ``dst_path`` when the
source is unhealthy (``verify_source``) or the produced backup fails its
own check (``verify_result``). On success ``dst_path`` is a verified-good
copy.
"""
if verify_source and not is_healthy(src_path):
# Don't immortalize corruption — surface it so the caller can alert
# and, crucially, NOT rotate out the existing good backups.
raise DBIntegrityError(
f"Refusing to back up: source database failed integrity check ({src_path})"
)
src = dst = None
try:
src = sqlite3.connect(src_path)
dst = sqlite3.connect(dst_path)
src.backup(dst)
finally:
_close_quietly(dst)
_close_quietly(src)
if verify_result and not is_healthy(dst_path):
# The copy itself came out bad — discard it rather than keep a dud.
try:
os.remove(dst_path)
except OSError:
pass
raise DBIntegrityError(
f"Backup produced a corrupt file and was discarded ({dst_path})"
)
def prune_backups(backup_paths, max_keep: int,
health_check=is_healthy) -> list:
"""Decide which backups to delete to honor ``max_keep`` WITHOUT ever
deleting the most-recent verified-healthy backup.
``backup_paths`` is an iterable of paths; order does not matter (we sort by
mtime). Returns the list of paths that SHOULD be deleted (does not delete
them — the caller does the IO, so this stays pure/testable).
Rule: oldest-first deletion until <= max_keep, but the single newest
*healthy* backup is protected and never selected for deletion. So even if
the newest few backups are corrupt, the last good snapshot survives.
"""
paths = [p for p in backup_paths]
# Newest first.
paths.sort(key=lambda p: _safe_mtime(p), reverse=True)
# Find the newest healthy backup — the one we must never drop.
protected: Optional[str] = None
for p in paths:
if health_check(p):
protected = p
break
if len(paths) <= max_keep:
return []
# Delete oldest-first beyond max_keep, but skip the protected one.
deletable = [p for p in paths if p != protected]
# oldest first among deletable
deletable.sort(key=lambda p: _safe_mtime(p))
num_to_delete = len(paths) - max_keep
return deletable[:num_to_delete]
def _safe_mtime(path: str) -> float:
try:
return os.path.getmtime(path)
except OSError:
return 0.0

413
core/debug_info.py
View file

@ -1,413 +0,0 @@
"""Debug info endpoint — lifted from web_server.py.
The function bodies are byte-identical to the originals. Module-level
shims for ``spotify_client`` and ``tidal_client`` (proxies that resolve
through the metadata registry / runtime client registry) plus injected
state dicts and helpers let the bodies resolve their original names
without modification.
"""
import logging
import os
import platform
from pathlib import Path
from flask import jsonify, request
from config.settings import config_manager
from core.metadata.registry import (
get_spotify_client,
get_primary_source,
is_hydrabase_enabled,
)
from core.metadata.status import get_spotify_status
logger = logging.getLogger(__name__)
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily through core.metadata.registry."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
class _TidalClientProxy:
"""Resolves the global Tidal client lazily via an injected getter so a
Tidal re-auth that rebinds web_server.tidal_client is visible here."""
def __getattr__(self, name):
if _get_tidal_client is None:
raise AttributeError(name)
client = _get_tidal_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
if _get_tidal_client is None:
return False
return _get_tidal_client() is not None
spotify_client = _SpotifyClientProxy()
tidal_client = _TidalClientProxy()
_get_tidal_client = None # injected via init()
# Injected at runtime via init().
SOULSYNC_VERSION = None
_DIRECT_RUN = None
_status_cache = None
qobuz_enrichment_worker = None
download_batches = None
sync_states = None
youtube_playlist_states = None
tidal_discovery_states = None
download_orchestrator = None
_log_path = None
_log_dir = None
app = None
get_database = None
def init(
soulsync_version,
direct_run,
status_cache,
qobuz_worker,
download_batches_dict,
sync_states_dict,
youtube_playlist_states_dict,
tidal_discovery_states_dict,
download_orchestrator_obj,
log_path,
log_dir,
flask_app,
get_database_fn,
tidal_client_getter,
):
"""Bind shared state/helpers from web_server."""
global SOULSYNC_VERSION, _DIRECT_RUN, _status_cache, qobuz_enrichment_worker
global download_batches, sync_states, youtube_playlist_states
global tidal_discovery_states, download_orchestrator, _log_path, _log_dir
global app, get_database, _get_tidal_client
SOULSYNC_VERSION = soulsync_version
_DIRECT_RUN = direct_run
_status_cache = status_cache
qobuz_enrichment_worker = qobuz_worker
download_batches = download_batches_dict
sync_states = sync_states_dict
youtube_playlist_states = youtube_playlist_states_dict
tidal_discovery_states = tidal_discovery_states_dict
download_orchestrator = download_orchestrator_obj
_log_path = log_path
_log_dir = log_dir
app = flask_app
get_database = get_database_fn
_get_tidal_client = tidal_client_getter
def _safe_check(fn, default=False):
"""Safely evaluate a check function, returning default on any error."""
try:
return fn()
except Exception:
return default
def get_debug_info():
"""Collect system diagnostics for troubleshooting support requests."""
import sys
import psutil
import time
from datetime import timedelta
log_lines = request.args.get('lines', 20, type=int)
log_lines = max(10, min(log_lines, 500))
log_source = request.args.get('log', 'app')
info = {}
# App info
info['version'] = SOULSYNC_VERSION
info['os'] = f"{platform.system()} {platform.release()}"
info['python'] = sys.version.split()[0]
info['docker'] = os.path.exists('/.dockerenv')
info['runner'] = 'gunicorn' if not _DIRECT_RUN else 'direct (python web_server.py)'
# ffmpeg version
try:
import subprocess
result = subprocess.run(['ffmpeg', '-version'], capture_output=True, text=True, timeout=5)
first_line = result.stdout.split('\n')[0] if result.stdout else ''
# e.g. "ffmpeg version 6.1.1 Copyright ..."
info['ffmpeg'] = first_line.split('Copyright')[0].replace('ffmpeg version', '').strip() if first_line else 'installed (version unknown)'
except FileNotFoundError:
info['ffmpeg'] = 'NOT INSTALLED'
except Exception:
info['ffmpeg'] = 'unknown'
# Uptime
start_time = getattr(app, 'start_time', time.time())
uptime_seconds = time.time() - start_time
info['uptime'] = str(timedelta(seconds=int(uptime_seconds)))
# Paths
download_path = config_manager.get('soulseek.download_path', './downloads')
transfer_folder = config_manager.get('soulseek.transfer_path', './Transfer')
staging_folder = config_manager.get('import.staging_path', '')
info['paths'] = {
'download_path': download_path,
'download_path_exists': os.path.isdir(download_path) if download_path else False,
'download_path_writable': os.access(download_path, os.W_OK) if download_path and os.path.isdir(download_path) else False,
'transfer_folder': transfer_folder,
'transfer_folder_exists': os.path.isdir(transfer_folder) if transfer_folder else False,
'transfer_folder_writable': os.access(transfer_folder, os.W_OK) if transfer_folder and os.path.isdir(transfer_folder) else False,
'staging_folder': staging_folder,
'staging_folder_exists': os.path.isdir(staging_folder) if staging_folder else False,
}
# Music library paths (Settings > Library)
music_paths = config_manager.get('library.music_paths', [])
if isinstance(music_paths, list) and music_paths:
info['paths']['music_library_paths'] = []
for p in music_paths:
if p and isinstance(p, str):
info['paths']['music_library_paths'].append({
'path': p,
'exists': os.path.isdir(p),
})
# Music videos directory
music_videos_path = config_manager.get('library.music_videos_path', '')
if music_videos_path:
info['paths']['music_videos_path'] = music_videos_path
info['paths']['music_videos_path_exists'] = os.path.isdir(music_videos_path)
# Services. `_status_cache` only carries 'media_server' and 'soulseek'
# (no 'spotify' key) so anything we used to read from `spotify_cache`
# silently defaulted to the missing-value fallback — that's the
# "music_source: unknown" bug. Spotify status now comes from the
# canonical `get_spotify_status` accessor; primary metadata source
# comes from `get_primary_source` (which already accounts for the
# auth-fallback chain — Spotify drops back to Deezer when not
# authenticated).
media_server_cache = _status_cache.get('media_server', {})
soulseek_cache = _status_cache.get('soulseek', {})
spotify_status = _safe_check(lambda: get_spotify_status(spotify_client=spotify_client), default={})
if not isinstance(spotify_status, dict):
spotify_status = {}
info['services'] = {
'music_source': _safe_check(get_primary_source, default='unknown') or 'unknown',
'spotify_connected': bool(spotify_status.get('connected', False)),
'spotify_rate_limited': bool(spotify_status.get('rate_limited', False)),
'media_server_type': media_server_cache.get('type', 'none'),
'media_server_connected': media_server_cache.get('connected', False),
'soulseek_connected': soulseek_cache.get('connected', False),
'download_source': config_manager.get('download_source.mode', 'hybrid'),
'tidal_connected': _safe_check(lambda: bool(tidal_client and tidal_client.is_authenticated())),
'qobuz_connected': _safe_check(lambda: bool(qobuz_enrichment_worker and qobuz_enrichment_worker.client and qobuz_enrichment_worker.client.is_authenticated())),
'hydrabase_connected': _safe_check(is_hydrabase_enabled),
# YouTube is URL-based via yt-dlp — no auth, always reachable as
# long as the binary is installed. Surfaced so the debug dump
# documents that YouTube is one of the available download sources
# rather than implying it doesn't exist.
'youtube_available': True,
}
# HiFi instance count — separate from connection status because each
# instance is its own independent endpoint with its own auth state.
info['services']['hifi_instance_count'] = _safe_check(
lambda: len(get_database().get_hifi_instances()), default=0
)
# Always-available public metadata sources (no auth, no per-user
# connection state). Listed so the debug dump reflects the full
# metadata surface SoulSync queries from, not just the auth-gated ones.
info['services']['always_available_metadata_sources'] = [
'deezer', 'itunes', 'musicbrainz',
]
# Enrichment workers
workers = {}
worker_names = ['musicbrainz', 'audiodb', 'deezer', 'spotify', 'itunes', 'lastfm', 'genius', 'discogs', 'tidal', 'qobuz']
for name in worker_names:
paused_key = f'{name}_enrichment_paused'
workers[name] = 'paused' if config_manager.get(paused_key, False) else 'active'
info['enrichment_workers'] = workers
# Library stats — use same method as dashboard (filters by active server)
try:
db = get_database()
lib_stats = db.get_database_info_for_server()
info['library'] = {
'artists': lib_stats.get('artists', 0),
'albums': lib_stats.get('albums', 0),
'tracks': lib_stats.get('tracks', 0),
}
except Exception:
info['library'] = {'artists': 0, 'albums': 0, 'tracks': 0}
# Watchlist count
try:
db = get_database()
info['watchlist_count'] = db.get_watchlist_count()
except Exception:
info['watchlist_count'] = 0
# Wishlist pending count
try:
db = get_database()
info['wishlist_count'] = db.get_wishlist_count()
except Exception:
info['wishlist_count'] = 0
# Automation count
try:
db = get_database()
automations = db.get_automations()
info['automations'] = {
'total': len(automations),
'enabled': len([a for a in automations if a.get('enabled', False)]),
}
except Exception:
info['automations'] = {'total': 0, 'enabled': 0}
# Active downloads & syncs (use list() snapshots to avoid RuntimeError from concurrent mutation)
try:
active_downloads = len([bid for bid, bd in list(download_batches.items()) if bd.get('phase') == 'downloading'])
except Exception:
active_downloads = 0
active_syncs = 0
try:
for _pid, ss in list(sync_states.items()):
if ss.get('status') == 'syncing':
active_syncs += 1
for _uh, st in list(youtube_playlist_states.items()):
if st.get('phase') == 'syncing':
active_syncs += 1
for _pid, st in list(tidal_discovery_states.items()):
if st.get('phase') == 'syncing':
active_syncs += 1
except Exception as e:
logger.debug("count active syncs failed: %s", e)
info['active_downloads'] = active_downloads
info['active_syncs'] = active_syncs
# Config settings relevant to troubleshooting
source_mode = config_manager.get('download_source.mode', 'hybrid')
info['config'] = {
'source_mode': source_mode,
'quality_profile': config_manager.get('download_source.quality_profile', 'default'),
'organization_template': config_manager.get('organization.folder_template', ''),
'post_processing_enabled': config_manager.get('post_processing.enabled', True),
'acoustid_enabled': bool(config_manager.get('acoustid.api_key', '')),
'auto_scan_enabled': config_manager.get('watchlist.auto_scan', False),
'm3u_export_enabled': config_manager.get('m3u.enabled', False),
'log_level': config_manager.get('logging.level', 'INFO'),
'primary_metadata_source': config_manager.get('metadata.fallback_source', 'deezer'),
'lossy_copy_enabled': config_manager.get('post_processing.lossy_copy.enabled', False),
'lossy_copy_format': config_manager.get('post_processing.lossy_copy.format', 'mp3'),
'lossy_copy_bitrate': config_manager.get('post_processing.lossy_copy.bitrate', 320),
'allow_duplicate_tracks': config_manager.get('library.allow_duplicate_tracks', False),
'replace_lower_quality': config_manager.get('import.replace_lower_quality', False),
'auto_import_enabled': config_manager.get('import.auto_import_enabled', False),
}
# Hybrid source priority order
if source_mode == 'hybrid':
info['config']['hybrid_sources'] = config_manager.get('download_source.hybrid_order', [])
# Discogs connection status
info['services']['discogs_connected'] = bool(config_manager.get('discogs.token', ''))
# Download client init failures
info['download_client_failures'] = []
if download_orchestrator and hasattr(download_orchestrator, '_init_failures'):
info['download_client_failures'] = download_orchestrator._init_failures
elif not download_orchestrator:
info['download_client_failures'] = ['ALL (orchestrator failed to initialize)']
# API rate monitor — current calls/min, 24h totals, peaks, rate limit events
try:
from core.api_call_tracker import api_call_tracker
# `get_spotify_status` is already imported at module level. A
# local re-import here would make Python treat the name as a
# function-scoped local for the WHOLE body, breaking the lambda
# at the top of get_debug_info that closes over the module-level
# binding (Python 3.12 NameError on free variables).
rates = api_call_tracker.get_all_rates()
info['api_rates'] = rates
# Rich 24h debug summary with peaks, totals, per-endpoint breakdown, events
info['api_debug_summary'] = api_call_tracker.get_debug_summary()
# Spotify rate limit details
spotify_status = get_spotify_status(spotify_client=spotify_client)
rl_info = spotify_status.get('rate_limit')
if spotify_status.get('rate_limited') and rl_info:
info['spotify_rate_limit'] = {
'active': True,
'remaining_seconds': rl_info.get('remaining_seconds', 0),
'retry_after': rl_info.get('retry_after', 0),
'endpoint': rl_info.get('endpoint', ''),
'expires_at': rl_info.get('expires_at', ''),
}
else:
info['spotify_rate_limit'] = {'active': False}
except Exception:
info['api_rates'] = {}
info['api_debug_summary'] = {}
info['spotify_rate_limit'] = {'active': False}
# Database size
db_path = os.path.join('database', 'music_library.db')
if os.path.exists(db_path):
db_size_mb = os.path.getsize(db_path) / (1024 * 1024)
info['database_size'] = f"{db_size_mb:.1f} MB"
else:
info['database_size'] = 'not found'
# Memory & CPU
process = psutil.Process(os.getpid())
mem = process.memory_info()
info['memory_usage'] = f"{mem.rss / (1024 * 1024):.0f} MB"
info['system_memory'] = f"{psutil.virtual_memory().percent}%"
try:
info['cpu_percent'] = f"{process.cpu_percent(interval=0.1):.1f}%"
except Exception:
info['cpu_percent'] = 'unknown'
info['thread_count'] = process.num_threads()
# Log lines
log_map = {
'app': Path(_log_path),
'acoustid': _log_dir / 'acoustid.log',
'post_processing': _log_dir / 'post_processing.log',
'source_reuse': _log_dir / 'source_reuse.log',
}
log_path = log_map.get(log_source, log_map['app'])
info['log_source'] = log_source
info['log_lines_requested'] = log_lines
info['recent_logs'] = []
if os.path.exists(log_path):
try:
with open(log_path, 'r', encoding='utf-8', errors='replace') as f:
lines = f.readlines()
info['recent_logs'] = [line.rstrip() for line in lines[-log_lines:]]
except Exception:
info['recent_logs'] = ['(could not read log file)']
# Available log files
info['available_logs'] = []
logs_dir = 'logs'
if os.path.isdir(logs_dir):
for fname in sorted(os.listdir(logs_dir)):
if fname.endswith('.log'):
fpath = os.path.join(logs_dir, fname)
size_kb = os.path.getsize(fpath) / 1024
info['available_logs'].append({
'name': fname.replace('.log', ''),
'file': fname,
'size': f"{size_kb:.0f} KB" if size_kb < 1024 else f"{size_kb/1024:.1f} MB",
})
return jsonify(info)

475
core/deezer_client.py
View file

@ -6,8 +6,7 @@ from typing import Dict, List, Optional, Any
from functools import wraps
from dataclasses import dataclass
from utils.logging_config import get_logger
from core.metadata.artist_album_cache import get_cached_artist_album_items, store_artist_album_items
from core.metadata.cache import get_metadata_cache
from core.metadata_cache import get_metadata_cache
logger = get_logger("deezer_client")
@ -46,119 +45,6 @@ def rate_limited(func):
return wrapper
# Pattern matches Deezer's CDN cover/picture URL: a numeric width-x-height
# segment in the path (e.g. ``/1000x1000-000000-80-0-0.jpg``). Captures
# both halves so the replacement can use a single dimension and preserve
# the rest of the path verbatim.
_DEEZER_CDN_SIZE_PATTERN = re.compile(r'/(\d+)x(\d+)-')
# Maximum size Deezer's CDN serves before returning 403. Verified
# empirically against multiple albums — 1900 works reliably, 2000+
# returns Forbidden. CDN serves the source-native size when it's
# smaller than requested, so asking for 1900 is safe even on albums
# whose source upload was lower-res (no upscaling, just same bytes).
_DEEZER_MAX_COVER_SIZE = 1900
def _upgrade_deezer_cover_url(url: str, target_size: int = _DEEZER_MAX_COVER_SIZE) -> str:
"""Rewrite a Deezer CDN cover/picture URL to request a larger size.
Deezer's API returns ``cover_xl`` / ``picture_xl`` URLs at
1000×1000, but the underlying CDN serves up to 1900×1900 by
rewriting the size segment in the URL path. This helper does the
rewrite — same idea as ``_upgrade_spotify_image_url`` in
``spotify_client`` and the ``mzstatic.com`` size-replacement in
``download_cover_art``.
Defensive on every input shape:
- Empty / None URL → returned as-is
- Non-Deezer URL (no ``dzcdn`` host, no size segment) → returned as-is
- Already at or above target size → returned as-is (no point rewriting)
The CDN returns the source-native image bytes when source < target,
so asking for 1900 on an album whose source was uploaded at 600
just returns the 600-pixel image — no upscaling, no failure.
"""
if not url or 'dzcdn' not in url:
return url
match = _DEEZER_CDN_SIZE_PATTERN.search(url)
if not match:
return url
current = int(match.group(1))
if current >= target_size:
return url
return _DEEZER_CDN_SIZE_PATTERN.sub(f'/{target_size}x{target_size}-', url, count=1)
def _is_full_track_payload(payload: Optional[Dict[str, Any]]) -> bool:
"""Distinguish a full `/track/<id>` cache hit from partial album-tracks data.
Three Deezer endpoints feed the per-track cache:
- `/track/<id>` — full record, includes both `track_position` AND
`contributors` (the multi-artist list the contributors-upgrade
path reads).
- `/album/<id>/tracks` — partial; includes `track_position` but
omits `contributors`.
- `/search/track` — minimal; lacks `track_position`.
Pre-fix `get_track_details` only checked `track_position`, so
partial album-tracks payloads were treated as full hits and the
contributors-upgrade silently fell back to single-artist tagging
whenever an album had been fetched before its individual tracks
were post-processed (issue #588).
`contributors` key presence is the load-bearing distinction —
`[]` is a valid value for genuinely single-artist tracks fetched
via the per-track endpoint, so test for key membership not
truthiness.
"""
if not isinstance(payload, dict):
return False
return 'track_position' in payload and 'contributors' in payload
def resolve_album_track_positions(session, base_url, album_ids, cache=None, sleep_s=0.2):
"""Build ``{str(track_id): track_position}`` for a set of Deezer album ids.
Deezer PLAYLIST and SEARCH track objects (and even the album object's embedded
``tracks.data``) omit ``track_position`` — only ``/album/<id>/tracks`` and
``/track/<id>`` carry it. So numbering playlist tracks by their playlist index
silently poisons the real album track number, which then rides onto the
downloaded file's tag. This resolves the authoritative position per album
(cache-first, best-effort — a failed album just isn't in the map)."""
import time as _time
positions: Dict[str, int] = {}
for aid in album_ids:
aid = str(aid)
at_list = None
if cache:
try:
ct = cache.get_entity('deezer', 'album_tracks', aid)
if ct and ct.get('data'):
at_list = ct['data']
except Exception: # noqa: BLE001 - cache is best-effort
at_list = None
if at_list is None:
try:
if sleep_s:
_time.sleep(sleep_s) # respect Deezer rate limits
r = session.get(f"{base_url}/album/{aid}/tracks", params={'limit': 500}, timeout=10)
if getattr(r, 'ok', False):
at_list = (r.json() or {}).get('data', [])
if cache and at_list is not None:
try:
cache.store_entity('deezer', 'album_tracks', aid, {'data': at_list})
except Exception as _cache_err: # noqa: BLE001
logger.debug("album_tracks cache store failed for %s: %s", aid, _cache_err)
except Exception: # noqa: BLE001 - never let metadata resolution break the fetch
at_list = None
for at in (at_list or []):
tp = at.get('track_position')
if at.get('id') and tp:
positions[str(at['id'])] = tp
return positions
# ==================== Dataclasses (match iTunesClient / SpotifyClient format) ====================
@dataclass
@ -282,7 +168,6 @@ class Album:
album_type: str
image_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
explicit: Optional[bool] = None
@classmethod
def from_deezer_album(cls, album_data: Dict[str, Any]) -> 'Album':
@ -314,8 +199,7 @@ class Album:
total_tracks=album_data.get('nb_tracks', 0),
album_type=album_type,
image_url=image_url,
external_urls=external_urls if external_urls else None,
explicit=bool(album_data.get('explicit_lyrics', False)),
external_urls=external_urls if external_urls else None
)
@ -414,136 +298,39 @@ class DeezerClient:
# can serve as a drop-in fallback metadata source in SpotifyClient.
@rate_limited
def search_tracks(
self,
query: str = '',
limit: int = 20,
*,
track: Optional[str] = None,
artist: Optional[str] = None,
album: Optional[str] = None,
) -> List[Track]:
"""Search for tracks — returns Track dataclass list (metadata source interface).
Two call modes:
1. **Free-text** (`query='Foreigner Dirty White Boy'`) — legacy
shape, passes the string straight to Deezer's `q` param.
Same behaviour as before, kept for backward compat.
2. **Field-scoped** (`track='Dirty White Boy', artist='Foreigner'`) —
builds Deezer's advanced search syntax (`track:"X" artist:"Y"`).
Massively tighter relevance than the free-text path because
the API matches each term in the right field instead of
anywhere across title / lyrics / artist / album / contributors.
Without this, the Deezer ranking buries the canonical track
under karaoke / cover / "originally performed by" variants
— see issue #534.
Field-scoped form is used whenever ``track`` or ``artist`` is
provided. ``query`` is ignored in that case (the field params
are authoritative). When both are missing, falls through to
``query``. The cache key is the constructed query string in
either case so the two paths share entries naturally.
"""
# Build the actual API query — advanced syntax when callers pass
# field hints, raw query otherwise.
used_advanced = bool(track or artist or album)
if used_advanced:
api_query = self._build_advanced_query(track=track, artist=artist, album=album)
else:
api_query = query
if not api_query:
return []
tracks = self._search_tracks_with_query(api_query, limit)
# Safety net: Deezer's advanced syntax is `artist:"X"`-style
# substring match, but in practice it's brittle on artist name
# variants ("Foreigner [US]", "The Foreigner", etc.) and on
# tracks indexed under non-canonical title spellings. When the
# advanced query returns nothing, fall back to a free-text join
# so the user sees the prior (less-relevant but non-empty) result
# set rather than "No matches" — same behaviour as pre-fix for
# this edge case. Caller-side rerank still tightens the result.
if not tracks and used_advanced:
fallback_parts = [p for p in (track, artist, album) if p]
fallback_query = ' '.join(fallback_parts)
if fallback_query and fallback_query != api_query:
logger.debug(
"[Deezer] Advanced query returned 0 results, falling back "
"to free-text: %r → %r", api_query, fallback_query,
)
tracks = self._search_tracks_with_query(fallback_query, limit)
return tracks
def _search_tracks_with_query(self, api_query: str, limit: int) -> List[Track]:
"""Cache-aware single API call. Pulled out so the
``search_tracks`` orchestration can call this twice (advanced
query → free-text fallback) without duplicating the cache +
parse + store dance."""
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
"""Search for tracks — returns Track dataclass list (metadata source interface)"""
cache = get_metadata_cache()
cached_results = cache.get_search_results('deezer', 'track', api_query, limit)
cached_results = cache.get_search_results('deezer', 'track', query, limit)
if cached_results is not None:
tracks = []
for raw in cached_results:
try:
tracks.append(Track.from_deezer_track(raw))
except Exception as e:
logger.debug("Track.from_deezer_track cache parse: %s", e)
except Exception:
pass
if tracks:
return tracks
data = self._api_get('search/track', {'q': api_query, 'limit': min(limit, 100)})
data = self._api_get('search/track', {'q': query, 'limit': min(limit, 100)})
if not data or 'data' not in data:
return []
tracks = []
raw_items = []
for track_data in data['data']:
track_obj = Track.from_deezer_track(track_data)
tracks.append(track_obj)
track = Track.from_deezer_track(track_data)
tracks.append(track)
raw_items.append(track_data)
entries = [(str(td.get('id', '')), td) for td in raw_items if td.get('id')]
if entries:
cache.store_entities_bulk('deezer', 'track', entries)
cache.store_search_results('deezer', 'track', api_query, limit,
cache.store_search_results('deezer', 'track', query, limit,
[str(td.get('id', '')) for td in raw_items if td.get('id')])
return tracks
@staticmethod
def _build_advanced_query(
*,
track: Optional[str] = None,
artist: Optional[str] = None,
album: Optional[str] = None,
) -> str:
"""Compose Deezer's advanced search syntax from field hints.
Per Deezer's docs:
https://developers.deezer.com/api/search
q=track:"X" artist:"Y" album:"Z"
Quotes around each value preserve multi-word phrases. Empty
fields are skipped. Embedded double-quotes get stripped (no
escape mechanism in Deezer's syntax) — rare in practice, but
a search for `O"Hara` would otherwise produce a malformed
query.
"""
parts = []
if track:
parts.append(f'track:"{track.replace(chr(34), "")}"')
if artist:
parts.append(f'artist:"{artist.replace(chr(34), "")}"')
if album:
parts.append(f'album:"{album.replace(chr(34), "")}"')
return ' '.join(parts)
@rate_limited
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
"""Search for artists — returns Artist dataclass list (metadata source interface)"""
@ -554,8 +341,8 @@ class DeezerClient:
for raw in cached_results:
try:
artists.append(Artist.from_deezer_artist(raw))
except Exception as e:
logger.debug("Artist.from_deezer_artist cache parse: %s", e)
except Exception:
pass
if artists:
return artists
@ -588,8 +375,8 @@ class DeezerClient:
for raw in cached_results:
try:
albums.append(Album.from_deezer_album(raw))
except Exception as e:
logger.debug("Album.from_deezer_album cache parse: %s", e)
except Exception:
pass
if albums:
return albums
@ -616,9 +403,14 @@ class DeezerClient:
"""Get detailed track info — returns Spotify-compatible dict (metadata source interface)"""
cache = get_metadata_cache()
cached = cache.get_entity('deezer', 'track', str(track_id))
if cached and cached.get('title') and _is_full_track_payload(cached):
return self._build_enhanced_track(cached)
# Otherwise fall through to fetch full data from API
if cached and cached.get('title'):
# Search results are cached with minimal data (no track_position).
# Only use cache if it has track_position — the key field from /track/{id}.
# Search results include 'isrc' and 'release_date' but NOT track_position,
# so those fields alone are not sufficient to distinguish full from partial data.
if 'track_position' in cached:
return self._build_enhanced_track(cached)
# Otherwise fall through to fetch full data from API
data = self._api_get(f'track/{track_id}')
if not data:
@ -807,74 +599,6 @@ class DeezerClient:
return result
def get_artist_top_tracks(self, artist_id: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Return the artist's top tracks in Spotify-compatible dict format.
Wraps Deezer's `/artist/{id}/top?limit=N`. Returns dicts with the same
shape Spotify's `artist_top_tracks` produces — id, name, artists, album
(with album_type / total_tracks / release_date / images), duration_ms,
track_number, disc_number — so callers don't need to branch on source.
"""
if not artist_id:
return []
try:
limit = max(1, min(int(limit or 10), 100))
except (TypeError, ValueError):
limit = 10
data = self._api_get(f'artist/{artist_id}/top', {'limit': limit})
if not data or 'data' not in data:
return []
tracks = []
for track_data in data['data']:
if not isinstance(track_data, dict):
continue
artist_data = track_data.get('artist') or {}
album_data = track_data.get('album') or {}
# Build images list from any cover sizes Deezer returned for the album
images = []
if isinstance(album_data, dict):
for size_key, dim in [('cover_xl', 1000), ('cover_big', 500),
('cover_medium', 250), ('cover_small', 56)]:
if album_data.get(size_key):
images.append({'url': album_data[size_key], 'height': dim, 'width': dim})
# Deezer `/artist/{id}/top` results don't include record_type on the
# nested album object; we don't have a track-count to infer from
# either. Default 'album' so the path-builder template variable
# always has something to substitute (existing behavior elsewhere).
album_payload = {
'id': str(album_data.get('id', '')) if isinstance(album_data, dict) else '',
'name': album_data.get('title', '') if isinstance(album_data, dict) else '',
'album_type': 'album',
'images': images,
'release_date': '',
'total_tracks': 0,
'artists': [{'name': artist_data.get('name', '')}] if isinstance(artist_data, dict) else [],
}
tracks.append({
'id': str(track_data.get('id', '')),
'name': track_data.get('title', ''),
'artists': [{
'id': str(artist_data.get('id', '')) if isinstance(artist_data, dict) else '',
'name': artist_data.get('name', '') if isinstance(artist_data, dict) else '',
}],
'album': album_payload,
'duration_ms': (track_data.get('duration') or 0) * 1000, # Deezer is seconds
'popularity': track_data.get('rank', 0),
'preview_url': track_data.get('preview'),
'external_urls': {'deezer': track_data['link']} if track_data.get('link') else {},
'track_number': track_data.get('track_position'),
'disc_number': track_data.get('disk_number', 1),
'explicit': bool(track_data.get('explicit_lyrics', False)),
'_source': 'deezer',
})
return tracks
def get_artist_info(self, artist_id: str) -> Optional[Dict[str, Any]]:
"""Get full artist details — returns Spotify-compatible dict (metadata source interface).
@ -911,80 +635,41 @@ class DeezerClient:
'_raw_data': artist_data
}
def get_artist_albums_list(self, artist_id: str, album_type: str = 'album,single', limit: int = 200) -> List[Album]:
def get_artist_albums_list(self, artist_id: str, album_type: str = 'album,single', limit: int = 50) -> List[Album]:
"""Get albums by artist ID — returns Album dataclass list (metadata source interface).
Matches iTunesClient.get_artist_albums() interface.
Paginates through all results up to the requested limit."""
cache = get_metadata_cache()
cached_items = get_cached_artist_album_items(cache, 'deezer', artist_id, album_type=album_type, limit=limit)
if cached_items:
try:
requested_types = [t.strip() for t in album_type.split(',')]
cached_albums = []
for album_data in cached_items:
album = Album.from_deezer_album(album_data)
if album_type != 'album,single':
if album.album_type not in requested_types:
if not (album.album_type == 'ep' and 'single' in requested_types):
continue
cached_albums.append(album)
return cached_albums[:limit]
except Exception as e:
logger.debug("Deezer artist albums cache reuse failed: %s", e)
Matches iTunesClient.get_artist_albums() interface."""
data = self._api_get(f'artist/{artist_id}/albums', {'limit': min(limit, 100)})
if not data or 'data' not in data:
return []
albums = []
all_raw = []
requested_types = [t.strip() for t in album_type.split(',')]
offset = 0
page_size = 100 # Deezer API max per request
complete = True # cleared if pagination breaks on a transient/malformed error
while offset < limit:
fetch_limit = min(page_size, limit - offset)
data = self._api_get(f'artist/{artist_id}/albums', {'limit': fetch_limit, 'index': offset})
if not data or 'data' not in data:
# Malformed/transient response mid-pagination — what we have is a
# PARTIAL discography. Don't cache it as the full list (mirrors the
# Spotify truncated-fetch guard). #853 follow-up.
complete = False
break
if len(data['data']) == 0:
break # No more albums — a clean end of pagination.
for album_data in data['data']:
album = Album.from_deezer_album(album_data)
for album_data in data['data']:
all_raw.append(album_data)
album = Album.from_deezer_album(album_data)
if album_type != 'album,single':
if album.album_type not in requested_types:
if not (album.album_type == 'ep' and 'single' in requested_types):
continue
if album_type != 'album,single':
if album.album_type not in requested_types:
if not (album.album_type == 'ep' and 'single' in requested_types):
continue
albums.append(album)
if len(data['data']) < fetch_limit:
break # Last page
offset += len(data['data'])
albums.append(album)
cache = get_metadata_cache()
# Deezer's /artist/{id}/albums endpoint doesn't include artist info on each album.
# Inject it so cached album entities have artist_name for discover page display.
artist_stub = None
if albums and albums[0].artists:
artist_stub = {'id': int(artist_id) if artist_id.isdigit() else 0, 'name': albums[0].artists[0]}
entries = []
for ad in all_raw:
for ad in data['data']:
if ad.get('id'):
if artist_stub and not ad.get('artist'):
ad['artist'] = artist_stub
entries.append((str(ad['id']), ad))
if entries:
cache.store_entities_bulk('deezer', 'album', entries, skip_if_exists=True)
# Only cache the artist→album-LIST when pagination finished cleanly; a
# partial list would otherwise serve an incomplete discography until TTL.
# (Individual album entities above are complete, so they cache regardless.)
if complete:
store_artist_album_items(cache, 'deezer', artist_id, all_raw, album_type=album_type, limit=limit)
logger.info(f"Retrieved {len(albums)} albums for artist {artist_id}")
return albums[:limit]
@ -1042,49 +727,6 @@ class DeezerClient:
logger.error(f"Error fetching Deezer favorite artists: {e}")
return []
@rate_limited
def get_user_favorite_albums(self, limit: int = 200) -> list:
"""Fetch user's favorite albums from Deezer. Requires OAuth access token.
Returns list of dicts with deezer_id, album_name, artist_name, image_url, release_date, total_tracks."""
if not self._access_token:
logger.debug("Deezer not user-authenticated — cannot fetch favorite albums")
return []
try:
albums = []
index = 0
while len(albums) < limit:
data = self._api_get('user/me/albums', params={
'limit': min(100, limit - len(albums)),
'index': index
})
if not data or 'data' not in data:
break
items = data['data']
if not items:
break
for a in items:
artist_name = ''
if isinstance(a.get('artist'), dict):
artist_name = a['artist'].get('name', '')
albums.append({
'deezer_id': str(a.get('id', '')),
'album_name': a.get('title', ''),
'artist_name': artist_name,
'image_url': a.get('cover_xl') or a.get('cover_big') or a.get('cover_medium', ''),
'release_date': a.get('release_date', ''),
'total_tracks': a.get('nb_tracks', 0),
})
if not data.get('next'):
break
index += len(items)
time.sleep(0.3)
logger.info(f"Retrieved {len(albums)} favorite albums from Deezer")
return albums
except Exception as e:
logger.error(f"Error fetching Deezer favorite albums: {e}")
return []
# ==================== Stub Methods (match iTunesClient interface) ====================
def get_user_playlists(self) -> List[Playlist]:
@ -1146,8 +788,8 @@ class DeezerClient:
try:
cache = get_metadata_cache()
cache.store_entity('deezer', 'artist', str(result.get('id', '')), result)
except Exception as e:
logger.debug("cache store_entity artist search: %s", e)
except Exception:
pass
logger.debug(f"Found artist for query: {artist_name}")
return result
@ -1191,8 +833,8 @@ class DeezerClient:
try:
cache = get_metadata_cache()
cache.store_entity('deezer', 'album', str(result.get('id', '')), result)
except Exception as e:
logger.debug("cache store_entity album search: %s", e)
except Exception:
pass
logger.debug(f"Found album for query: {artist_name} - {album_title}")
return result
@ -1236,8 +878,8 @@ class DeezerClient:
try:
cache = get_metadata_cache()
cache.store_entity('deezer', 'track', str(result.get('id', '')), result)
except Exception as e:
logger.debug("cache store_entity track search: %s", e)
except Exception:
pass
logger.debug(f"Found track for query: {artist_name} - {track_title}")
return result
@ -1269,8 +911,8 @@ class DeezerClient:
# Cache hit with full details (has label = was a get_album response, not just search)
logger.debug(f"Cache hit for album {album_id}")
return cached
except Exception as e:
logger.debug("cache get_entity album: %s", e)
except Exception:
pass
try:
response = self.session.get(
@ -1288,8 +930,8 @@ class DeezerClient:
try:
cache = get_metadata_cache()
cache.store_entity('deezer', 'album', str(album_id), data)
except Exception as e:
logger.debug("cache store_entity album full: %s", e)
except Exception:
pass
logger.debug(f"Got full album details for ID: {album_id}")
return data
@ -1317,8 +959,8 @@ class DeezerClient:
if cached and cached.get('bpm'):
logger.debug(f"Cache hit for track {track_id}")
return cached
except Exception as e:
logger.debug("cache get_entity track: %s", e)
except Exception:
pass
try:
response = self.session.get(
@ -1336,8 +978,8 @@ class DeezerClient:
try:
cache = get_metadata_cache()
cache.store_entity('deezer', 'track', str(track_id), data)
except Exception as e:
logger.debug("cache store_entity track full: %s", e)
except Exception:
pass
logger.debug(f"Got full track details for ID: {track_id}")
return data
@ -1399,16 +1041,6 @@ class DeezerClient:
raw_tracks.extend(page_tracks)
# Real album track positions — playlist tracks don't carry track_position,
# so numbering by playlist index would poison the downloaded file's tag.
album_ids = {str(t.get('album', {}).get('id')) for t in raw_tracks if t.get('album', {}).get('id')}
try:
from core.metadata.cache import get_metadata_cache
_cache = get_metadata_cache()
except Exception:
_cache = None
track_positions = resolve_album_track_positions(self.session, self.BASE_URL, album_ids, _cache)
# Normalize tracks
tracks: List[Dict[str, Any]] = []
for i, t in enumerate(raw_tracks, start=1):
@ -1420,8 +1052,7 @@ class DeezerClient:
'artists': [artist_name],
'album': t.get('album', {}).get('title', ''),
'duration_ms': t.get('duration', 0) * 1000,
# REAL album position; the playlist index is a last resort only.
'track_number': track_positions.get(str(t.get('id'))) or i,
'track_number': i,
})
result = {
@ -1429,7 +1060,7 @@ class DeezerClient:
'name': data.get('title', ''),
'description': data.get('description', ''),
'track_count': total_tracks,
'image_url': data.get('picture_xl') or data.get('picture_big') or data.get('picture_medium', ''),
'image_url': data.get('picture_medium', ''),
'owner': data.get('creator', {}).get('name', ''),
'tracks': tracks,
}

457
core/deezer_download_client.py
View file

@ -20,8 +20,7 @@ from typing import Any, Dict, List, Optional, Tuple
import requests
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.quality.source_map import quality_from_deezer, quality_tier_for_source
from core.soulseek_client import AlbumResult, DownloadStatus, TrackResult
from utils.logging_config import get_logger
logger = get_logger("deezer_download")
@ -73,17 +72,14 @@ def _decrypt_chunk(chunk: bytes, key: bytes) -> bytes:
cipher = Cipher(algorithms.Blowfish(key), modes.CBC(iv))
decryptor = cipher.decryptor()
return decryptor.update(chunk) + decryptor.finalize()
except ImportError as exc:
except ImportError:
raise ImportError(
"Deezer downloads require pycryptodome or cryptography package. "
"Install with: pip install pycryptodome"
) from exc
)
from core.download_plugins.base import DownloadSourcePlugin
class DeezerDownloadClient(DownloadSourcePlugin):
class DeezerDownloadClient:
"""Deezer download client using ARL token authentication."""
def __init__(self, download_path: str = None):
@ -93,14 +89,11 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if download_path is None:
download_path = config_manager.get('soulseek.download_path', './downloads')
self.download_path = Path(download_path)
try:
self.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {self.download_path}: {e}")
self.download_path.mkdir(parents=True, exist_ok=True)
# Engine reference is populated by set_engine() at registration
# time. None until orchestrator wires the registry.
self._engine = None
# Download tracking (same pattern as Tidal/Qobuz/HiFi)
self.active_downloads: Dict[str, Dict[str, Any]] = {}
self._download_lock = threading.Lock()
# Shutdown check callback (set by web_server)
self.shutdown_check = None
@ -121,27 +114,17 @@ class DeezerDownloadClient(DownloadSourcePlugin):
self._license_token = None
self._user_data = None
self._authenticated = False
self._pending_arl: Optional[str] = None
# Quality preference
self._quality = quality_tier_for_source('deezer', default='flac')
self._quality = config_manager.get('deezer_download.quality', 'flac')
# Try to authenticate on init if ARL is configured
arl = config_manager.get('deezer_download.arl', '')
if arl:
from core.boot_phase import is_boot_phase
if is_boot_phase():
self._pending_arl = arl
logger.debug("Deezer ARL present — authentication deferred until after boot")
else:
self._authenticate(arl)
self._authenticate(arl)
logger.info(f"Deezer download client initialized (download path: {self.download_path})")
def set_engine(self, engine):
"""Engine callback — wires the central thread worker + state store."""
self._engine = engine
# ─── Authentication ──────────────────────────────────────────
def _authenticate(self, arl: str) -> bool:
@ -233,66 +216,12 @@ class DeezerDownloadClient(DownloadSourcePlugin):
return self._authenticated
def is_authenticated(self) -> bool:
if self._pending_arl and not self._authenticated:
from core.boot_phase import is_boot_phase
if not is_boot_phase():
self._authenticate(self._pending_arl)
self._pending_arl = None
return self._authenticated
async def check_connection(self) -> bool:
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.is_available)
# ─── Playlist export (#945) ──────────────────────────────────
#
# UNOFFICIAL: rides the private gw-light gateway with the ARL session already used
# for downloads. Deezer shut their public developer API, so this is the only write
# path — and it's fragile by nature (breaks when Deezer changes internals).
def create_or_update_playlist(self, name, track_ids, *, existing_id=None,
public=False, description=""):
"""Create a Deezer playlist (or append to an existing one) from a mirrored
playlist's tracks. ``track_ids`` are stored ``deezer_id`` values per library track.
``existing_id`` set → add to that playlist (idempotent re-export reuses the stored
target); unset → create a new one. Returns
``{success, playlist_id, url, added, error}``."""
if not self._authenticated:
return {"success": False, "error": "Deezer is not connected (ARL)"}
song_ids = [str(t) for t in (track_ids or []) if t]
if not song_ids:
return {"success": False, "error": "No matching Deezer tracks to export"}
try:
songs = [[sid, i] for i, sid in enumerate(song_ids)]
if existing_id:
res = self._gw_call("playlist.addSongs",
{"playlist_id": int(existing_id), "songs": songs})
if res is None:
return {"success": False, "error": "Deezer rejected the playlist update"}
playlist_id = existing_id
else:
res = self._gw_call("playlist.create", {
"title": name, "description": description,
"is_public": bool(public), "songs": songs,
})
if res is None:
return {"success": False, "error": "Deezer rejected the playlist create"}
# gw 'playlist.create' returns the new playlist id (int) as `results`.
if isinstance(res, dict):
playlist_id = res.get("PLAYLIST_ID") or res.get("id")
else:
playlist_id = res
if not playlist_id:
return {"success": False, "error": "Deezer did not return a playlist id"}
return {
"success": True,
"playlist_id": str(playlist_id),
"url": f"https://www.deezer.com/playlist/{playlist_id}",
"added": len(song_ids),
}
except Exception as e:
return {"success": False, "error": str(e)}
def reconnect(self, arl: str = None) -> bool:
"""Re-authenticate with a new or existing ARL."""
if arl is None:
@ -353,94 +282,6 @@ class DeezerDownloadClient(DownloadSourcePlugin):
logger.info(f"Fetched {len(playlists)} user playlists from Deezer")
return playlists
def get_user_favorite_artists(self, limit: int = 200) -> list:
"""Fetch the authenticated user's favorite artists via public API with ARL cookies."""
if not self._authenticated or not self._user_data:
return []
user_id = self._user_data.get('USER_ID')
if not user_id:
return []
artists = []
index = 0
while len(artists) < limit:
try:
resp = self._session.get(
f'https://api.deezer.com/user/{user_id}/artists',
params={'index': index, 'limit': min(100, limit - len(artists))},
timeout=15
)
resp.raise_for_status()
data = resp.json()
if 'error' in data:
logger.warning(f"Deezer artists error: {data['error']}")
break
items = data.get('data', [])
if not items:
break
for a in items:
artists.append({
'deezer_id': str(a.get('id', '')),
'name': a.get('name', ''),
'image_url': a.get('picture_xl') or a.get('picture_big') or a.get('picture_medium', ''),
})
if not data.get('next'):
break
index += len(items)
except Exception as e:
logger.error(f"Error fetching favorite artists at index {index}: {e}")
break
logger.info(f"Fetched {len(artists)} favorite artists from Deezer (ARL)")
return artists
def get_user_favorite_albums(self, limit: int = 200) -> list:
"""Fetch the authenticated user's favorite albums via public API with ARL cookies."""
if not self._authenticated or not self._user_data:
return []
user_id = self._user_data.get('USER_ID')
if not user_id:
return []
albums = []
index = 0
while len(albums) < limit:
try:
resp = self._session.get(
f'https://api.deezer.com/user/{user_id}/albums',
params={'index': index, 'limit': min(100, limit - len(albums))},
timeout=15
)
resp.raise_for_status()
data = resp.json()
if 'error' in data:
logger.warning(f"Deezer albums error: {data['error']}")
break
items = data.get('data', [])
if not items:
break
for a in items:
artist_name = ''
if isinstance(a.get('artist'), dict):
artist_name = a['artist'].get('name', '')
albums.append({
'deezer_id': str(a.get('id', '')),
'album_name': a.get('title', ''),
'artist_name': artist_name,
'image_url': a.get('cover_xl') or a.get('cover_big') or a.get('cover_medium', ''),
'release_date': a.get('release_date', ''),
'total_tracks': a.get('nb_tracks', 0),
})
if not data.get('next'):
break
index += len(items)
except Exception as e:
logger.error(f"Error fetching favorite albums at index {index}: {e}")
break
logger.info(f"Fetched {len(albums)} favorite albums from Deezer (ARL)")
return albums
def get_playlist_tracks(self, playlist_id: str) -> Optional[dict]:
"""Fetch full playlist details with tracks via public API (ARL cookies grant private access)."""
try:
@ -481,14 +322,8 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if aid:
album_ids.add(str(aid))
album_release_dates = {}
# Deezer PLAYLIST tracks do NOT carry `track_position` (only `/track/<id>`
# and `/album/<id>/tracks` do), so numbering them by their playlist index
# poisons the real album track number — which then rides into the wishlist
# and onto the downloaded file's tag (e.g. 'Apologize' tagged track 1 instead
# of 16). Resolve the REAL position from each album's track list (cache-first).
track_positions: Dict[str, int] = {} # str(track_id) -> album track_position
try:
from core.metadata.cache import get_metadata_cache
from core.metadata_cache import get_metadata_cache
cache = get_metadata_cache()
except Exception:
cache = None
@ -499,32 +334,24 @@ class DeezerDownloadClient(DownloadSourcePlugin):
cached = cache.get_entity('deezer', 'album', aid)
if cached and cached.get('release_date'):
album_release_dates[aid] = cached['release_date']
except Exception as e:
logger.debug("cache get_entity album release_date: %s", e)
continue
except Exception:
pass
# Cache miss — fetch from API
if aid not in album_release_dates:
try:
time.sleep(0.3) # Respect rate limits
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
if a_resp.ok:
a_data = a_resp.json()
album_release_dates[aid] = a_data.get('release_date', '')
# Store in metadata cache for future use
if cache:
try:
cache.store_entity('deezer', 'album', aid, a_data)
except Exception as e:
logger.debug("cache store_entity album release_date: %s", e)
except Exception as e:
logger.debug("fetch deezer album release_date %s: %s", aid, e)
# Real album track positions (separate endpoint — playlist tracks AND the
# album object's embedded tracks both omit track_position). Cache-first.
try:
from core.deezer_client import resolve_album_track_positions
track_positions = resolve_album_track_positions(
self._session, 'https://api.deezer.com', album_ids, cache)
except Exception as e:
logger.debug("resolve deezer album track positions: %s", e)
try:
time.sleep(0.3) # Respect rate limits
a_resp = self._session.get(f'https://api.deezer.com/album/{aid}', timeout=10)
if a_resp.ok:
a_data = a_resp.json()
album_release_dates[aid] = a_data.get('release_date', '')
# Store in metadata cache for future use
if cache:
try:
cache.store_entity('deezer', 'album', aid, a_data)
except Exception:
pass
except Exception:
pass
tracks = []
for i, t in enumerate(raw_tracks, start=1):
@ -545,9 +372,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
'id': album_id,
},
'duration_ms': t.get('duration', 0) * 1000,
# REAL album position (resolved above); the playlist index is a last
# resort only when the album lookup failed, never the default.
'track_number': track_positions.get(str(t.get('id'))) or t.get('track_position') or i,
'track_number': i,
})
return {
@ -661,7 +486,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
bitrate = 128
quality = 'mp3'
tr = TrackResult(
results.append(TrackResult(
username='deezer_dl',
filename=f"{track_id}||{artist} - {title}",
size=est_size,
@ -675,10 +500,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
title=title,
album=album,
track_number=item.get('track_position'),
)
# Stamp CD-quality FLAC (16/44.1) so lossless ranks correctly.
tr.set_quality(quality_from_deezer(self._quality))
results.append(tr)
))
logger.info(f"Deezer search for '{query}' returned {len(results)} results")
return results, []
@ -695,67 +517,87 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if not self._authenticated:
logger.error("Deezer not authenticated — cannot download")
return None
if self._engine is None:
# Raise rather than return None so the orchestrator's
# download_with_fallback surfaces a real warning + tries
# the next source. Returning None silently dropped the
# download with no user feedback (per JohnBaumb).
raise RuntimeError("Deezer client has no engine reference — cannot dispatch download")
# Parse filename: "track_id||display_name"
parts = filename.split('||', 1)
track_id = parts[0]
display_name = parts[1] if len(parts) > 1 else f"Track {track_id}"
return self._engine.worker.dispatch(
source_name='deezer',
target_id=track_id,
display_name=display_name,
original_filename=filename,
impl_callable=self._download_sync,
extra_record_fields={
download_id = str(uuid.uuid4())
with self._download_lock:
self.active_downloads[download_id] = {
'id': download_id,
'track_id': track_id,
'display_name': display_name,
'filename': filename,
'username': 'deezer_dl',
'state': 'Initializing',
'progress': 0.0,
'size': file_size,
'transferred': 0,
'speed': 0,
'file_path': None,
'error': None,
},
# Legacy username slot — frontend status indicators key off
# ``deezer_dl``, not the canonical ``deezer``.
username_override='deezer_dl',
# Diagnostic thread name for multi-thread debugging.
thread_name=f'deezer-dl-{track_id}',
}
thread = threading.Thread(
target=self._download_thread_worker,
args=(download_id, track_id, display_name),
daemon=True,
name=f'deezer-dl-{track_id}'
)
thread.start()
def _set_error(self, download_id: str, message: str) -> None:
"""Helper: set the engine record's `error` slot. No-op if
engine isn't wired or record was already removed."""
if self._engine is None:
return
self._engine.update_record('deezer', download_id, {'error': message})
logger.info(f"Started Deezer download {download_id}: {display_name}")
return download_id
def _is_cancelled(self, download_id: str) -> bool:
if self._engine is None:
return False
record = self._engine.get_record('deezer', download_id)
return record is not None and record.get('state') == 'Cancelled'
def _download_thread_worker(self, download_id: str, track_id: str, display_name: str):
"""Background worker for a single download."""
try:
result_path = self._download_sync(download_id, track_id, display_name)
with self._download_lock:
if download_id in self.active_downloads:
dl = self.active_downloads[download_id]
if dl['state'] == 'Cancelled':
return
if result_path:
dl['state'] = 'Completed, Succeeded'
dl['progress'] = 100.0
dl['file_path'] = result_path
logger.info(f"Deezer download {download_id} completed: {result_path}")
else:
dl['state'] = 'Errored'
logger.error(f"Deezer download {download_id} failed: {dl.get('error', 'unknown')}")
except Exception as e:
logger.error(f"Deezer download thread error: {e}")
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['state'] = 'Errored'
self.active_downloads[download_id]['error'] = str(e)
def _download_sync(self, download_id: str, track_id: str, display_name: str) -> Optional[str]:
"""Synchronous download: get URL, download, decrypt, save."""
# Check for shutdown
if self.shutdown_check and self.shutdown_check():
if self._engine is not None:
self._engine.update_record('deezer', download_id, {'state': 'Aborted'})
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['state'] = 'Aborted'
return None
# Get track data from private API
track_data = self._get_track_data(track_id)
if not track_data:
self._set_error(download_id, 'Failed to get track data')
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['error'] = 'Failed to get track data'
return None
track_token = track_data.get('TRACK_TOKEN', '')
if not track_token:
self._set_error(download_id, 'No track token available')
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['error'] = 'No track token available'
return None
# Determine quality and get media URL with fallback
@ -765,6 +607,7 @@ class DeezerDownloadClient(DownloadSourcePlugin):
if allow_fallback:
quality_order = _QUALITY_ORDER.copy()
# Start from user's preferred quality
try:
pref_idx = quality_order.index(self._quality)
quality_order = quality_order[pref_idx:] + quality_order[:pref_idx]
@ -781,16 +624,27 @@ class DeezerDownloadClient(DownloadSourcePlugin):
break
if not media_url:
self._set_error(download_id, 'No media URL available (may require higher subscription tier)')
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['error'] = 'No media URL available (may require higher subscription tier)'
return None
if actual_quality != self._quality:
logger.info(f"Quality fallback: {self._quality} → {actual_quality} for {display_name}")
# Determine file extension
ext = '.flac' if actual_quality == 'flac' else '.mp3'
# Sanitize filename
safe_name = self._sanitize_filename(display_name)
out_path = str(self.download_path / f"{safe_name}{ext}")
# Update state
with self._download_lock:
if download_id in self.active_downloads:
dl = self.active_downloads[download_id]
dl['state'] = 'InProgress, Downloading'
# Download and decrypt
try:
bf_key = _get_blowfish_key(track_id)
@ -798,8 +652,9 @@ class DeezerDownloadClient(DownloadSourcePlugin):
resp.raise_for_status()
total_size = int(resp.headers.get('content-length', 0))
if self._engine is not None:
self._engine.update_record('deezer', download_id, {'size': total_size})
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['size'] = total_size
downloaded = 0
chunk_index = 0
@ -812,20 +667,23 @@ class DeezerDownloadClient(DownloadSourcePlugin):
# Check for cancellation/shutdown
if self.shutdown_check and self.shutdown_check():
if self._engine is not None:
self._engine.update_record('deezer', download_id, {'state': 'Aborted'})
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['state'] = 'Aborted'
try:
os.remove(out_path)
except OSError:
pass
return None
if self._is_cancelled(download_id):
try:
os.remove(out_path)
except OSError:
pass
return None
with self._download_lock:
if download_id in self.active_downloads:
if self.active_downloads[download_id]['state'] == 'Cancelled':
try:
os.remove(out_path)
except OSError:
pass
return None
# Decrypt every 3rd chunk (Deezer's encryption pattern)
if chunk_index % 3 == 0 and len(raw_chunk) == _CHUNK_SIZE:
@ -842,12 +700,12 @@ class DeezerDownloadClient(DownloadSourcePlugin):
speed = int(downloaded / elapsed) if elapsed > 0 else 0
progress = (downloaded / total_size * 100) if total_size > 0 else 0
if self._engine is not None:
self._engine.update_record('deezer', download_id, {
'transferred': downloaded,
'progress': min(progress, 99.9),
'speed': speed,
})
with self._download_lock:
if download_id in self.active_downloads:
dl = self.active_downloads[download_id]
dl['transferred'] = downloaded
dl['progress'] = min(progress, 99.9)
dl['speed'] = speed
# Validate file size
file_size = os.path.getsize(out_path)
@ -857,7 +715,9 @@ class DeezerDownloadClient(DownloadSourcePlugin):
os.remove(out_path)
except OSError:
pass
self._set_error(download_id, f'File too small ({file_size} bytes)')
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['error'] = f'File too small ({file_size} bytes)'
return None
logger.info(f"Deezer download complete: {out_path} ({file_size / 1048576:.1f} MB, {actual_quality})")
@ -869,58 +729,59 @@ class DeezerDownloadClient(DownloadSourcePlugin):
os.remove(out_path)
except OSError:
pass
self._set_error(download_id, str(e))
with self._download_lock:
if download_id in self.active_downloads:
self.active_downloads[download_id]['error'] = str(e)
return None
# ─── Download Status ─────────────────────────────────────────
def _record_to_status(self, record: dict) -> DownloadStatus:
return DownloadStatus(
id=record['id'],
filename=record['filename'],
username=record['username'],
state=record['state'],
progress=record['progress'],
size=record.get('size', 0),
transferred=record.get('transferred', 0),
speed=record.get('speed', 0),
file_path=record.get('file_path'),
)
async def get_all_downloads(self) -> List[DownloadStatus]:
if self._engine is None:
return []
return [
self._record_to_status(record)
for record in self._engine.iter_records_for_source('deezer')
]
"""Return all active downloads."""
with self._download_lock:
return [self._to_status(dl) for dl in self.active_downloads.values()]
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
if self._engine is None:
return None
record = self._engine.get_record('deezer', download_id)
return self._record_to_status(record) if record is not None else None
"""Get status of a specific download."""
with self._download_lock:
dl = self.active_downloads.get(download_id)
return self._to_status(dl) if dl else None
async def cancel_download(self, download_id: str, username: str = None,
remove: bool = False) -> bool:
if self._engine is None:
return False
if self._engine.get_record('deezer', download_id) is None:
return False
self._engine.update_record('deezer', download_id, {'state': 'Cancelled'})
if remove:
self._engine.remove_record('deezer', download_id)
"""Cancel a download."""
with self._download_lock:
dl = self.active_downloads.get(download_id)
if not dl:
return False
dl['state'] = 'Cancelled'
if remove:
del self.active_downloads[download_id]
return True
async def clear_all_completed_downloads(self) -> bool:
if self._engine is None:
return True
terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
for record in list(self._engine.iter_records_for_source('deezer')):
if record.get('state') in terminal:
self._engine.remove_record('deezer', record['id'])
"""Remove all terminal downloads."""
terminal_states = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
with self._download_lock:
to_remove = [k for k, v in self.active_downloads.items() if v['state'] in terminal_states]
for k in to_remove:
del self.active_downloads[k]
return True
def _to_status(self, dl: dict) -> DownloadStatus:
"""Convert internal dict to DownloadStatus."""
return DownloadStatus(
id=dl['id'],
filename=dl['filename'],
username=dl['username'],
state=dl['state'],
progress=dl['progress'],
size=dl['size'],
transferred=dl['transferred'],
speed=dl['speed'],
file_path=dl.get('file_path'),
)
# ─── Utilities ───────────────────────────────────────────────
@staticmethod

136
core/deezer_worker.py
View file

@ -8,16 +8,7 @@ from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.deezer_client import DeezerClient
from core.worker_utils import (
accept_artist_match,
artist_name_matches,
interruptible_sleep,
owned_album_titles,
pick_artist_by_catalog,
release_titles,
set_album_api_track_count,
)
from core.enrichment.manual_match_honoring import honor_stored_match
from core.worker_utils import interruptible_sleep
logger = get_logger("deezer_worker")
@ -149,8 +140,8 @@ class DeezerWorker:
itype = item.get('type', '')
table = 'artists' if 'artist' in itype else ('albums' if 'album' in itype else 'tracks')
# Can't mark status without an ID — just skip
except Exception as e:
logger.debug("null id table resolve failed: %s", e)
except Exception:
pass
continue
@ -171,16 +162,6 @@ class DeezerWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('deezer')
if _prio:
_pi = priority_pending_item(cursor, 'deezer', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name
@ -286,19 +267,10 @@ class DeezerWorker:
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
return similarity >= self.name_similarity_threshold
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id,
result_artist_name: Optional[str] = None) -> bool:
def _verify_artist_id(self, item: Dict[str, Any], result_artist_id) -> bool:
"""Verify that the result's artist ID matches the parent artist's stored Deezer ID.
If mismatched, the album/track search is more specific (uses artist+title),
so we trust it and correct the parent artist's deezer_id — BUT only when
the result's artist *name* actually matches our parent artist. Without
that guard, a collaboration or compilation track (e.g. a track our
library credits to Jorja Smith that lives on Kendrick Lamar's curated
"Black Panther" album) would search up to an album whose Deezer primary
artist is someone else (Kendrick), and we'd stamp that wrong Deezer ID
onto our artist — corrupting it (and causing duplicate ids shared across
unrelated artists)."""
so we trust it and correct the parent artist's deezer_id."""
parent_deezer_id = item.get('artist_deezer_id')
if not parent_deezer_id:
return True
@ -307,20 +279,6 @@ class DeezerWorker:
return True
if str(result_artist_id) != str(parent_deezer_id):
# Guard: only correct when the album/track's primary artist is the
# SAME artist by name. A mismatch means it's a collab/compilation,
# not a stale-id correction.
parent_name = item.get('artist') or ''
if (result_artist_name and parent_name
and not self._name_matches(parent_name, result_artist_name)):
logger.info(
f"Skipping artist-ID correction from {item['type']} "
f"'{item['name']}': result artist '{result_artist_name}' "
f"≠ parent '{parent_name}' (collab/compilation, not a "
f"correction)"
)
return True
logger.info(
f"Artist ID correction from {item['type']} '{item['name']}': "
f"updating parent artist Deezer ID from {parent_deezer_id} to {result_artist_id}"
@ -409,66 +367,27 @@ class DeezerWorker:
logger.debug(f"Preserving existing Deezer ID for artist '{artist_name}': {existing_id}")
return
# Multi-candidate search (was single search_artist) so same-name artists
# can be disambiguated: gate by name, then pick the one whose catalog
# overlaps the albums this library owns.
results = self.client.search_artists(artist_name, limit=5)
gated = [a for a in (results or []) if artist_name_matches(artist_name, getattr(a, 'name', ''))]
chosen, _overlap = pick_artist_by_catalog(
gated,
owned_album_titles(self.db, artist_id),
lambda a: release_titles(self.client.get_artist_albums_list(a.id)),
)
# search_artists returns lean Artist objects; fetch the full dict (same
# shape the old search_artist returned) for storage.
result = self.client.get_artist_info(chosen.id) if chosen else None
result = self.client.search_artist(artist_name)
if result:
result_name = result.get('name', '')
ok, reason = accept_artist_match(
self.db, 'deezer_id', result.get('id'), artist_id,
artist_name, result_name,
)
if ok:
if self._name_matches(artist_name, result_name):
self._update_artist(artist_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{artist_name}' -> Deezer ID: {result.get('id')}")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Artist '{artist_name}' not matched: {reason}")
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result_name}')")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for artist '{artist_name}'")
def _refresh_album_via_stored_id(self, album_id, stored_id, full_album_dict):
"""Issue #501 callback. Stored ID exists → fetched full Deezer
album payload. Use it as both args to ``_update_album`` (search-
result and full-data shapes overlap on the fields we need —
artist verification skipped since manual match presumably
already vetted)."""
self._update_album(album_id, full_album_dict, full_album_dict)
def _refresh_track_via_stored_id(self, track_id, stored_id, full_track_dict):
"""Issue #501 callback for tracks — same pattern as albums."""
self._update_track(track_id, full_track_dict, full_track_dict)
def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
"""Process an album: search Deezer, verify, fetch full details, store metadata"""
# Issue #501: honor manual matches. Pre-fix this method just
# SKIPPED when a stored ID was present (preserved the ID but
# never refreshed metadata). Now it goes through the full
# refresh path via the stored ID, picking up label / genres /
# explicit updates without ever overwriting the manual match.
if honor_stored_match(
db=self.db, entity_table='albums', entity_id=album_id,
id_column='deezer_id',
client_fetch_fn=self.client.get_album_raw,
on_match_fn=self._refresh_album_via_stored_id,
log_prefix='Deezer',
):
self.stats['matched'] += 1
existing_id = self._get_existing_id('album', album_id)
if existing_id:
logger.debug(f"Preserving existing Deezer ID for album '{album_name}': {existing_id}")
return
result = self.client.search_album(artist_name, album_name)
@ -478,8 +397,7 @@ class DeezerWorker:
# Verify artist ID
result_artist = result.get('artist', {})
result_artist_id = result_artist.get('id') if result_artist else None
result_artist_name = result_artist.get('name') if result_artist else None
self._verify_artist_id(item, result_artist_id, result_artist_name)
self._verify_artist_id(item, result_artist_id)
# Fetch full album details for label, genres, explicit
deezer_album_id = result.get('id')
@ -512,15 +430,9 @@ class DeezerWorker:
def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
"""Process a track: search Deezer, verify, fetch full details for BPM, store metadata"""
# Issue #501: honor manual matches (see _process_album).
if honor_stored_match(
db=self.db, entity_table='tracks', entity_id=track_id,
id_column='deezer_id',
client_fetch_fn=self.client.get_track_raw,
on_match_fn=self._refresh_track_via_stored_id,
log_prefix='Deezer',
):
self.stats['matched'] += 1
existing_id = self._get_existing_id('track', track_id)
if existing_id:
logger.debug(f"Preserving existing Deezer ID for track '{track_name}': {existing_id}")
return
result = self.client.search_track(artist_name, track_name)
@ -530,8 +442,7 @@ class DeezerWorker:
# Verify artist ID
result_artist = result.get('artist', {})
result_artist_id = result_artist.get('id') if result_artist else None
result_artist_name = result_artist.get('name') if result_artist else None
self._verify_artist_id(item, result_artist_id, result_artist_name)
self._verify_artist_id(item, result_artist_id)
# Fetch full track details for BPM
deezer_track_id = result.get('id')
@ -658,27 +569,12 @@ class DeezerWorker:
genres_data = full_data.get('genres', {}).get('data', [])
if genres_data:
genre_names = [g.get('name') for g in genres_data if g.get('name')]
if genre_names:
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
genre_names = filter_genres(genre_names, _cfg)
if genre_names:
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(genre_names), album_id))
# Cache the authoritative expected track count for the Album
# Completeness repair job. Deezer's field is `nb_tracks`; prefer
# full_data over search_data so we pick up the richer count when
# the full album lookup ran. Helper handles the int conversion
# and skip-on-missing semantics.
set_album_api_track_count(
cursor,
album_id,
(full_data.get('nb_tracks') if full_data else None) or search_data.get('nb_tracks'),
)
conn.commit()
except Exception as e:

0
core/diagnostics/__init__.py
View file

136
core/diagnostics/memory_tracker.py
View file

@ -1,136 +0,0 @@
"""On-demand memory-growth diagnostic (issue #802: ~0.7 MiB/s RSS growth).
Wraps ``tracemalloc`` so a user seeing runaway memory can capture WHERE the
allocations come from instead of us guessing:
1. start_tracking() — begins tracing + stores a baseline snapshot
2. ...reproduce the growth for a few minutes...
3. report() — top allocation sites, with the DELTA since baseline
(the delta is the leak; absolute sizes are mostly
startup noise)
4. stop_tracking() — ends tracing, frees trace memory
Opt-in by design: tracemalloc costs CPU and memory while active (it shadows
every allocation), so it must never run by default. The Flask endpoints that
expose this live in web_server (GET /api/debug/memory/...) so a user can drive
the whole flow from a browser.
"""
from __future__ import annotations
import os
import time
import tracemalloc
from typing import Any, Dict, List, Optional
from utils.logging_config import get_logger
logger = get_logger("diagnostics.memory")
_baseline: Optional[tracemalloc.Snapshot] = None
_started_at: Optional[float] = None
# Allocation-site traces this deep give useful "who called it" context without
# pathological overhead.
_TRACE_FRAMES = 15
def is_tracking() -> bool:
return tracemalloc.is_tracing()
def start_tracking() -> Dict[str, Any]:
"""Begin tracing and store the baseline snapshot. Idempotent."""
global _baseline, _started_at
if tracemalloc.is_tracing():
return {"tracking": True, "already_running": True, "started_at": _started_at}
tracemalloc.start(_TRACE_FRAMES)
_baseline = tracemalloc.take_snapshot()
_started_at = time.time()
logger.info("Memory tracking started (tracemalloc, %d frames)", _TRACE_FRAMES)
return {"tracking": True, "already_running": False, "started_at": _started_at}
def stop_tracking() -> Dict[str, Any]:
"""End tracing and free the trace bookkeeping."""
global _baseline, _started_at
was = tracemalloc.is_tracing()
if was:
tracemalloc.stop()
logger.info("Memory tracking stopped")
_baseline = None
_started_at = None
return {"tracking": False, "was_tracking": was}
def _rss_mb() -> Optional[float]:
"""Process RSS in MiB, best-effort (psutil, then /proc fallback)."""
try:
import psutil
return round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 1)
except Exception: # noqa: S110 — RSS is optional context; fall through to /proc
pass
try:
with open("/proc/self/status", encoding="utf-8") as fh:
for line in fh:
if line.startswith("VmRSS:"):
return round(int(line.split()[1]) / 1024, 1)
except Exception: # noqa: S110 — no /proc on this platform; RSS stays None
pass
return None
def format_stat(stat: Any) -> Dict[str, Any]:
"""Project one tracemalloc StatisticDiff/Statistic into a plain dict.
Duck-typed (reads size/count/size_diff/count_diff/traceback) so it's
unit-testable without real snapshots."""
tb = getattr(stat, "traceback", None)
frames: List[str] = []
if tb:
# Most-recent-call-last reads naturally top-down in a report.
for frame in list(tb)[-3:]:
frames.append(f"{frame.filename}:{frame.lineno}")
return {
"location": frames[-1] if frames else "?",
"trace": frames,
"size_mb": round(getattr(stat, "size", 0) / (1024 * 1024), 3),
"size_diff_mb": round(getattr(stat, "size_diff", 0) / (1024 * 1024), 3),
"count": getattr(stat, "count", 0),
"count_diff": getattr(stat, "count_diff", 0),
}
def report(top: int = 25) -> Dict[str, Any]:
"""Current snapshot vs the start_tracking() baseline: the top allocation
sites by GROWTH (size_diff). Includes traced totals + process RSS so the
user can see how much of the real growth tracemalloc accounts for."""
if not tracemalloc.is_tracing():
return {
"tracking": False,
"rss_mb": _rss_mb(),
"hint": "Start with /api/debug/memory/start, reproduce the growth "
"for a few minutes, then call this again.",
}
snapshot = tracemalloc.take_snapshot()
# Filter the tracer's own bookkeeping out of the picture.
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, tracemalloc.__file__),
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
))
current, peak = tracemalloc.get_traced_memory()
if _baseline is not None:
stats = snapshot.compare_to(_baseline, "traceback")
stats.sort(key=lambda s: s.size_diff, reverse=True)
else:
stats = snapshot.statistics("traceback")
return {
"tracking": True,
"started_at": _started_at,
"elapsed_seconds": round(time.time() - _started_at, 1) if _started_at else None,
"traced_current_mb": round(current / (1024 * 1024), 1),
"traced_peak_mb": round(peak / (1024 * 1024), 1),
"rss_mb": _rss_mb(),
"top_growth": [format_stat(s) for s in stats[:top]],
}

302
core/discogs_client.py
View file

@ -12,8 +12,7 @@ import re
import time
import threading
import requests
from core.metadata.artist_album_cache import get_cached_artist_album_payload, store_artist_album_items
from core.metadata.cache import get_metadata_cache
from core.metadata_cache import get_metadata_cache
from typing import List, Dict, Any, Optional
from dataclasses import dataclass
from functools import wraps
@ -62,69 +61,6 @@ def rate_limited(func):
return wrapper
# Discogs disambiguates duplicate artist names two ways: numeric `(N)`
# suffix on the older convention (e.g. "Bullet (2)") and a trailing `*`
# on the newer convention (e.g. "John Smith*"). Both are presentation-
# only — the underlying canonical name is what users expect to see in
# search results, on cards, and especially in import paths (otherwise
# library folders end up named `Foo*` on disk). Strip both at every
# point a Discogs payload becomes a name string.
_DISCOGS_DISAMBIG_RE = re.compile(r'(?:\s*\(\d+\))?\s*\*+\s*$|\s*\(\d+\)\s*$')
def _clean_discogs_artist_name(name: Optional[str]) -> str:
"""Strip Discogs disambiguation suffixes — both `(N)` and trailing
`*` — from an artist name. Returns '' for None / empty input."""
if not name:
return ''
return _DISCOGS_DISAMBIG_RE.sub('', name).strip()
# --- Discogs album ID typing -------------------------------------------------
# Discogs has two album object types — masters (/masters/{id}) and releases
# (/releases/{id}) — whose numeric IDs share one space, so release N and master
# N are DIFFERENT albums. A bare numeric ID is therefore ambiguous. We tag the
# type into the ID string ('m12345' / 'r12345') at the point we parse it, so the
# correct endpoint can be chosen later without guessing. (Artist IDs are a single
# namespace and stay untagged.)
def _discogs_album_kind(data: Dict[str, Any]) -> str:
"""Classify a Discogs album payload as 'master' or 'release'.
Search results and artist-discography items carry an explicit ``type``;
full detail responses don't, but only master detail has ``main_release``."""
t = (data.get('type') or '').lower()
if t in ('master', 'release'):
return t
return 'master' if 'main_release' in data else 'release'
def _tag_discogs_album_id(raw_id: Any, kind: str) -> str:
"""``'12345'`` + ``'master'`` -> ``'m12345'``; empty input -> ``''``."""
s = str(raw_id or '').strip()
if not s:
return ''
return f"{'m' if kind == 'master' else 'r'}{s}"
def _discogs_album_endpoints(album_id: Any) -> List[str]:
"""Map a (possibly tagged) album ID to the API path(s) to try, in order.
``'m12345'`` -> ``['/masters/12345']``
``'r12345'`` -> ``['/releases/12345']``
``'12345'`` (legacy untagged) -> ``['/releases/12345', '/masters/12345']``
Legacy bare IDs are tried release-first because stored IDs originate
overwhelmingly from search / manual-match / collection sync (all releases);
this also self-heals pre-fix bad matches. Returns ``[]`` for unusable input."""
s = str(album_id or '').strip()
if len(s) > 1 and s[0] in ('m', 'r') and s[1:].isdigit():
return [f"/{'masters' if s[0] == 'm' else 'releases'}/{s[1:]}"]
if s.isdigit():
return [f'/releases/{s}', f'/masters/{s}']
return []
# --- Shared dataclasses (same shape as iTunes/Deezer/Spotify) ---
@dataclass
@ -181,10 +117,9 @@ class Track:
# Artists from track-level or release-level
track_artists = []
if track_data.get('artists'):
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in track_data['artists'] if a.get('name')]
track_artists = [a.get('name', '') for a in track_data['artists'] if a.get('name')]
if not track_artists and release.get('artists'):
track_artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release['artists'] if a.get('name')]
track_artists = [a for a in track_artists if a]
track_artists = [a.get('name', '') for a in release['artists'] if a.get('name')]
if not track_artists:
track_artists = ['Unknown Artist']
@ -255,9 +190,7 @@ class Artist:
return cls(
id=str(artist_data.get('id', '')),
name=_clean_discogs_artist_name(
artist_data.get('name', artist_data.get('title', ''))
),
name=artist_data.get('name', artist_data.get('title', '')),
popularity=0,
genres=[],
followers=0,
@ -283,15 +216,14 @@ class Album:
artists = []
title = release_data.get('title', '')
if release_data.get('artists'):
artists = [_clean_discogs_artist_name(a.get('name', '')) for a in release_data['artists'] if a.get('name')]
artists = [a.get('name', '') for a in release_data['artists'] if a.get('name')]
elif release_data.get('artist'):
artists = [_clean_discogs_artist_name(release_data['artist'])]
artists = [release_data['artist']]
elif ' - ' in title:
# Search results: "Radiohead - OK Computer" → artist="Radiohead", title="OK Computer"
parts = title.split(' - ', 1)
artists = [_clean_discogs_artist_name(parts[0])]
artists = [parts[0].strip()]
title = parts[1].strip()
artists = [a for a in artists if a]
if not artists:
artists = ['Unknown Artist']
@ -350,7 +282,7 @@ class Album:
external_urls['discogs_api'] = release_data['resource_url']
return cls(
id=_tag_discogs_album_id(release_data.get('id', ''), _discogs_album_kind(release_data)),
id=str(release_data.get('id', '')),
name=title,
artists=artists,
release_date=release_date,
@ -388,8 +320,8 @@ class DiscogsClient:
try:
from config.settings import config_manager
self.token = config_manager.get('discogs.token', '')
except Exception as e:
logger.debug("load discogs.token from config: %s", e)
except Exception:
pass
if self.token:
self.session.headers['Authorization'] = f'Discogs token={self.token}'
@ -437,125 +369,6 @@ class DiscogsClient:
logger.error(f"Discogs API error ({endpoint}): {e}")
return None
# --- User Collection (powers Your Albums Discogs source) ---
def get_authenticated_username(self) -> Optional[str]:
"""Resolve the username for the configured personal token.
Discogs's `/oauth/identity` endpoint returns the user's
username when called with a valid token. Cached on the
instance so subsequent calls don't re-hit the API.
"""
if hasattr(self, '_cached_username'):
return self._cached_username
if not self.is_authenticated():
self._cached_username = None
return None
data = self._api_get('/oauth/identity')
username = data.get('username') if data else None
self._cached_username = username
return username
def get_user_collection(self, username: Optional[str] = None,
folder_id: int = 0,
per_page: int = 100,
max_pages: int = 50) -> List[Dict[str, Any]]:
"""Fetch a Discogs user's collection (folder 0 = "All").
Returns a list of normalized release dicts ready for
``database.upsert_liked_album``:
{
'album_name': str,
'artist_name': str,
'release_id': int, # Discogs release id
'image_url': str | None,
'release_date': str, # 'YYYY' (Discogs only stores year)
'total_tracks': int,
}
Pagination caps at ``max_pages`` to bound runtime — at 100/page
that's 5000 releases, more than enough for typical collections.
Authenticated calls only (Discogs collection is private).
"""
if not self.is_authenticated():
logger.warning("Discogs collection fetch attempted without token")
return []
if not username:
username = self.get_authenticated_username()
if not username:
logger.warning("Could not resolve Discogs username for token")
return []
results: List[Dict[str, Any]] = []
page = 1
while page <= max_pages:
data = self._api_get(
f'/users/{username}/collection/folders/{folder_id}/releases',
{'page': page, 'per_page': per_page, 'sort': 'added', 'sort_order': 'desc'},
)
if not data:
break
releases = data.get('releases', []) or []
if not releases:
break
for entry in releases:
info = entry.get('basic_information') or {}
release_id = entry.get('id') or info.get('id')
if not release_id:
continue
title = info.get('title') or ''
# Discogs `artists` is a list of {name, id, ...}; first is primary.
artists = info.get('artists') or []
artist_name = ''
if artists and isinstance(artists[0], dict):
artist_name = (artists[0].get('name') or '').strip()
# Strip Discogs disambiguation suffixes — both `(N)` and `*`.
artist_name = _clean_discogs_artist_name(artist_name)
if not title or not artist_name:
continue
# Image URLs: cover_image is the primary, also has thumb.
image_url = (info.get('cover_image')
or info.get('thumb')
or '')
year = info.get('year')
release_date = str(year) if year and year > 0 else ''
results.append({
'album_name': title.strip(),
'artist_name': artist_name,
'release_id': int(release_id),
'image_url': image_url or None,
'release_date': release_date,
'total_tracks': 0, # Not in basic_information; populated via get_release if needed
})
pagination = data.get('pagination') or {}
if page >= int(pagination.get('pages') or 1):
break
page += 1
logger.info(f"Discogs collection: fetched {len(results)} releases for {username}")
return results
def get_release(self, release_id: int) -> Optional[Dict[str, Any]]:
"""Fetch full Discogs release detail including tracklist.
Returns the raw API response so callers can render rich
Discogs context (year, format, label, country, tracklist).
"""
if not release_id:
return None
try:
release_id = int(release_id)
except (TypeError, ValueError):
return None
return self._api_get(f'/releases/{release_id}')
# --- Search Methods (same signatures as iTunes/Deezer) ---
def search_artists(self, query: str, limit: int = 10) -> List[Artist]:
@ -567,8 +380,8 @@ class DiscogsClient:
for raw in cached_results:
try:
artists.append(Artist.from_discogs_artist(raw))
except Exception as e:
logger.debug("Artist.from_discogs_artist cache parse: %s", e)
except Exception:
pass
if artists:
return artists
@ -604,8 +417,8 @@ class DiscogsClient:
for raw in cached_results:
try:
albums.append(Album.from_discogs_release(raw))
except Exception as e:
logger.debug("Album.from_discogs_release cache parse: %s", e)
except Exception:
pass
if albums:
return albums
@ -689,13 +502,10 @@ class DiscogsClient:
if cached and cached.get('title'):
data = cached
else:
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('title'):
break
data = None
# Try as master first (artist discography returns master IDs)
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('title'):
data = self._api_get(f'/releases/{release_id}')
if not data:
return None
cache.store_entity('discogs', 'album', release_id, data)
@ -729,45 +539,22 @@ class DiscogsClient:
def get_artist_albums(self, artist_id: str, album_type: str = 'album,single', limit: int = 50) -> List[Album]:
"""Get releases by an artist. Prefers master releases, filters features."""
cache = get_metadata_cache()
cached_payload = get_cached_artist_album_payload(cache, 'discogs', artist_id, album_type=album_type, limit=limit)
releases = cached_payload.get('_releases') if cached_payload else None
artist_name = ''
if cached_payload:
artist_name = str(cached_payload.get('artist_name') or '').lower()
# First get the artist name for feature filtering
artist_data = self._api_get(f'/artists/{artist_id}')
artist_name = artist_data.get('name', '').lower() if artist_data else ''
if not isinstance(releases, list) or not releases:
# First get the artist name for feature filtering. Strip Discogs
# disambiguation suffix so feature-vs-primary matching below
# compares against the canonical name, not "Beyoncé*".
artist_data = self._api_get(f'/artists/{artist_id}')
artist_name = _clean_discogs_artist_name(
artist_data.get('name', '') if artist_data else ''
).lower()
data = self._api_get(f'/artists/{artist_id}/releases', {
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
})
if not data or not data.get('releases'):
return []
releases = data.get('releases') or []
store_artist_album_items(
cache,
'discogs',
artist_id,
releases,
album_type=album_type,
limit=limit,
items_field='_releases',
extra_fields={'artist_name': artist_name},
)
data = self._api_get(f'/artists/{artist_id}/releases', {
'sort': 'year', 'sort_order': 'desc', 'per_page': min(limit * 3, 200),
})
if not data or not data.get('releases'):
return []
# Separate masters from individual releases — prefer masters (canonical versions)
masters = []
releases_no_master = []
master_titles = set()
for item in releases:
for item in data['releases']:
# Skip non-main roles
role = item.get('role', 'Main').lower()
if role not in ('main', ''):
@ -835,13 +622,10 @@ class DiscogsClient:
if cached:
return cached
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('tracklist'):
break
data = None
# Try as master first (master IDs are used in artist discography)
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('tracklist'):
data = self._api_get(f'/releases/{release_id}')
if not data or not data.get('tracklist'):
return None
@ -853,7 +637,7 @@ class DiscogsClient:
image_url = (primary or images[0]).get('uri')
album_info = {
'id': str(release_id),
'id': str(data.get('id', release_id)),
'name': data.get('title', ''),
'images': [{'url': image_url, 'height': 600, 'width': 600}] if image_url else [],
'release_date': str(data.get('year', '')) if data.get('year') else '',
@ -862,11 +646,7 @@ class DiscogsClient:
# Get artists
artists_list = []
if data.get('artists'):
artists_list = [
{'name': _clean_discogs_artist_name(a.get('name', ''))}
for a in data['artists']
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
]
artists_list = [{'name': a.get('name', '')} for a in data['artists'] if a.get('name')]
if not artists_list:
artists_list = [{'name': 'Unknown Artist'}]
@ -895,11 +675,7 @@ class DiscogsClient:
# Per-track artists or fall back to release artists
track_artists = artists_list
if t.get('artists'):
track_artists = [
{'name': _clean_discogs_artist_name(a.get('name', ''))}
for a in t['artists']
if a.get('name') and _clean_discogs_artist_name(a.get('name', ''))
]
track_artists = [{'name': a.get('name', '')} for a in t['artists'] if a.get('name')]
tracks.append({
'id': f"{release_id}_t{track_num}",
@ -942,13 +718,9 @@ class DiscogsClient:
cached = cache.get_entity('discogs', 'album', str(release_id))
if cached and cached.get('title'):
return cached
# Hit the endpoint that matches the ID's type (tag-driven, no guessing).
data = None
for path in _discogs_album_endpoints(release_id):
data = self._api_get(path)
if data and data.get('title'):
break
data = None
data = self._api_get(f'/masters/{release_id}')
if not data or not data.get('title'):
data = self._api_get(f'/releases/{release_id}')
if data:
cache.store_entity('discogs', 'album', str(release_id), data)
return data

75
core/discogs_worker.py
View file

@ -17,35 +17,12 @@ from typing import Optional, Dict, Any
from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.discogs_client import DiscogsClient, _discogs_album_kind, _tag_discogs_album_id
from core.worker_utils import accept_artist_match, interruptible_sleep, set_album_api_track_count
from core.discogs_client import DiscogsClient
from core.worker_utils import interruptible_sleep
logger = get_logger("discogs_worker")
def count_discogs_real_tracks(tracklist) -> int:
"""Count actual songs in a Discogs tracklist response.
Discogs tracklists interleave real tracks with section headings
(``type_=='heading'``), index markers (``type_=='index'``),
and sub-tracks (``type_=='sub_track'``) that aren't themselves
songs. We count anything that's explicitly typed as ``'track'`` OR
has an empty/missing ``type_`` field — matching exactly what
:meth:`core.discogs_client.DiscogsClient.get_album_tracks` itself
treats as a real track (`type_ in ('track', '')`). Counting any
narrower set silently disagrees with the repair job's fallback
`_get_expected_total` path, which calls `get_album_tracks_for_source`
under the hood and therefore uses the client's count.
Reported by kettui on PR #374 — original filter only counted
``type_=='track'`` and undercounted releases where the discogs
response left ``type_`` empty for some real tracks.
"""
if not tracklist:
return 0
return sum(1 for t in tracklist if (t.get('type_') or '') in ('track', ''))
class DiscogsWorker:
"""Background worker for enriching library artists and albums with Discogs metadata."""
@ -174,16 +151,6 @@ class DiscogsWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Discogs
# has no track endpoint, so only artist/album are honored.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('discogs')
if _prio in ('artist', 'album'):
_pi = priority_pending_item(cursor, 'discogs', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists
@ -308,8 +275,8 @@ class DiscogsWorker:
self.stats['errors'] += 1
try:
self._mark_status(item['type'], item['id'], 'error')
except Exception as e:
logger.debug("mark item status error failed: %s", e)
except Exception:
pass
def _get_existing_id(self, entity_type: str, entity_id) -> Optional[str]:
"""Check if entity already has a discogs_id."""
@ -332,13 +299,9 @@ class DiscogsWorker:
self.stats['not_found'] += 1
return
# Find best match by name similarity (skipping ids already claimed by
# a differently-named artist, so we don't create a shared/duplicate id).
# Find best match by name similarity
for result in results:
ok, reason = accept_artist_match(
self.db, 'discogs_id', result.id, artist_id, artist_name, result.name,
)
if ok:
if self._name_matches(artist_name, result.name):
# Fetch full artist detail (uses cache)
data = self.client._fetch_and_cache_artist(result.id)
if data:
@ -436,9 +399,7 @@ class DiscogsWorker:
conn = self.db._get_connection()
cursor = conn.cursor()
# Tag the ID with its Discogs type so later re-fetches hit the right
# endpoint (master vs release share one numeric space).
discogs_id = _tag_discogs_album_id(data.get('id', ''), _discogs_album_kind(data))
discogs_id = str(data.get('id', ''))
genres = json.dumps(data.get('genres', []))
styles = json.dumps(data.get('styles', []))
labels = data.get('labels', [])
@ -477,14 +438,10 @@ class DiscogsWorker:
# Backfill genres if empty
if data.get('genres'):
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
_filtered = filter_genres(data.get('genres', []), _cfg)
if _filtered:
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(_filtered), album_id))
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (genres, album_id))
# Backfill thumb_url if empty
if image_url:
@ -493,16 +450,6 @@ class DiscogsWorker:
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (image_url, album_id))
# Cache the authoritative expected track count for the Album
# Completeness repair job. See `count_discogs_real_tracks`
# for why we accept both `type_ == 'track'` and empty `type_`
# (kettui's PR #374 review — narrower filter undercounted).
set_album_api_track_count(
cursor,
album_id,
count_discogs_real_tracks(data.get('tracklist')),
)
conn.commit()
except Exception as e:

0
core/discovery/__init__.py
View file

388
core/discovery/beatport.py
View file

@ -1,388 +0,0 @@
"""Background worker for Beatport chart discovery.
`run_beatport_discovery_worker(url_hash, deps)` is the function the
beatport discovery start-endpoint submits to its executor to match each
Beatport chart track against Spotify (preferred) or iTunes (fallback):
1. Pause enrichment workers (release shared resources).
2. For each Beatport track:
- Cancellation gate (state['phase'] != 'discovering').
- Clean Beatport text (artist/title) of common annotations.
- Discovery cache lookup; cache hit short-circuits the search and
normalizes cached artists from ['str'] → [{'name': 'str'}].
- matching_engine search-query generation, with high min_confidence
(0.9) to avoid bad matches.
- Strategy 1: scored candidates from initial Spotify/iTunes searches.
- Strategy 4: extended search with limit=50 if no high-confidence
match found.
- On Spotify match: format artists as [{'name': str}] objects, pull
full album object from raw cache when available.
- On iTunes match: format with image_url-derived album.images entry.
- Save matched result to discovery cache when confidence >= 0.75.
- On miss: Wing It stub stored as 'wing-it' status (success ticked).
3. After all tracks: phase='discovered', activity feed entry, sync
discovery results back to mirrored playlist via
`_sync_discovery_results_to_mirrored`.
4. On error: state['phase']='fresh' + status='error'.
5. Finally: resume enrichment workers.
Lifted verbatim from web_server.py. Wide dependency surface (Spotify
and iTunes clients, matching engine, multiple discovery helpers, state
dict, mirrored sync) all injected via `BeatportDiscoveryDeps`.
"""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass
from typing import Any, Callable
logger = logging.getLogger(__name__)
@dataclass
class BeatportDiscoveryDeps:
"""Bundle of cross-cutting deps the Beatport discovery worker needs."""
beatport_chart_states: dict
spotify_client: Any
matching_engine: Any
pause_enrichment_workers: Callable[[str], dict]
resume_enrichment_workers: Callable[[dict, str], None]
get_active_discovery_source: Callable[[], str]
get_metadata_fallback_client: Callable[[], Any]
clean_beatport_text: Callable[[str], str]
get_discovery_cache_key: Callable
get_database: Callable[[], Any]
validate_discovery_cache_artist: Callable
spotify_rate_limited: Callable[[], bool]
discovery_score_candidates: Callable
get_metadata_cache: Callable[[], Any]
build_discovery_wing_it_stub: Callable
add_activity_item: Callable
sync_discovery_results_to_mirrored: Callable
def run_beatport_discovery_worker(url_hash, deps: BeatportDiscoveryDeps):
"""Background worker for Beatport discovery process (Spotify preferred, iTunes fallback)"""
_ew_state = {}
try:
_ew_state = deps.pause_enrichment_workers('Beatport discovery')
state = deps.beatport_chart_states[url_hash]
chart = state['chart']
tracks = chart['tracks']
# Determine which provider to use
discovery_source = deps.get_active_discovery_source()
use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
# Initialize fallback client if needed
itunes_client_instance = None
if not use_spotify:
itunes_client_instance = deps.get_metadata_fallback_client()
logger.info(f"Starting {discovery_source.upper()} discovery for {len(tracks)} Beatport tracks...")
# Store discovery source in state for frontend
state['discovery_source'] = discovery_source
# Process each track for discovery
for i, track in enumerate(tracks):
try:
# Check for cancellation
if state.get('phase') != 'discovering':
logger.warning(f"Beatport discovery cancelled (phase changed to '{state.get('phase')}')")
return
# Update progress
state['discovery_progress'] = int((i / len(tracks)) * 100)
# Get track info from Beatport data (frontend sends 'name' and 'artists' fields)
track_title = deps.clean_beatport_text(track.get('name', 'Unknown Title'))
track_artists = track.get('artists', ['Unknown Artist'])
# Handle artists - could be a list or string
if isinstance(track_artists, list):
if len(track_artists) > 0 and isinstance(track_artists[0], str):
# Handle case like ["CID,Taylr Renee"] - split on comma and clean
track_artist = deps.clean_beatport_text(track_artists[0].split(',')[0].strip())
else:
track_artist = deps.clean_beatport_text(track_artists[0] if track_artists else 'Unknown Artist')
else:
track_artist = deps.clean_beatport_text(str(track_artists))
logger.debug(f"Searching {discovery_source.upper()} for: '{track_artist}' - '{track_title}'")
# Check discovery cache first
cache_key = deps.get_discovery_cache_key(track_title, track_artist)
try:
cache_db = deps.get_database()
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
if cached_match and deps.validate_discovery_cache_artist(track_artist, cached_match):
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_artist} - {track_title}")
# Convert artists from ['str'] to [{'name': 'str'}] for Beatport frontend format
beatport_artists = cached_match.get('artists', [])
if beatport_artists and isinstance(beatport_artists[0], str):
cached_match['artists'] = [{'name': a} for a in beatport_artists]
result_entry = {
'index': i,
'beatport_track': {
'title': track_title,
'artist': track_artist
},
'status': 'found',
'status_class': 'found',
'discovery_source': discovery_source,
'spotify_data': cached_match
}
state['spotify_matches'] += 1
state['discovery_results'].append(result_entry)
continue
except Exception as cache_err:
logger.error(f"Cache lookup error: {cache_err}")
# Use matching engine for track matching
found_track = None
best_confidence = 0.0
best_raw_track = None
min_confidence = 0.9 # Higher threshold for Beatport to avoid bad matches
# Generate search queries using matching engine (with fallback)
try:
temp_track = type('TempTrack', (), {
'name': track_title,
'artists': [track_artist],
'album': None
})()
search_queries = deps.matching_engine.generate_download_queries(temp_track)
logger.debug(f"Generated {len(search_queries)} search queries using matching engine")
except Exception as e:
logger.error(f"Matching engine failed for Beatport, falling back to basic queries: {e}")
if use_spotify:
search_queries = [
f"{track_artist} {track_title}",
f'artist:"{track_artist}" track:"{track_title}"',
f'"{track_artist}" "{track_title}"'
]
else:
search_queries = [
f"{track_artist} {track_title}",
f"{track_title} {track_artist}",
track_title
]
for query_idx, search_query in enumerate(search_queries):
try:
logger.debug(f"Query {query_idx + 1}/{len(search_queries)}: {search_query} ({discovery_source.upper()})")
search_results = None
if use_spotify and not deps.spotify_rate_limited():
search_results = deps.spotify_client.search_tracks(search_query, limit=10)
else:
search_results = itunes_client_instance.search_tracks(search_query, limit=10)
if not search_results:
continue
# Score all results using the matching engine
match, confidence, match_idx = deps.discovery_score_candidates(
track_title, track_artist, 0, search_results
)
if match and confidence > best_confidence and confidence >= min_confidence:
best_confidence = confidence
found_track = match
if use_spotify and match.id:
_cache = deps.get_metadata_cache()
best_raw_track = _cache.get_entity('spotify', 'track', match.id)
else:
best_raw_track = None
logger.debug(f"New best Beatport match: {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
if best_confidence >= 0.9:
logger.debug(f"High confidence match found ({best_confidence:.3f}), stopping search")
break
except Exception as e:
logger.debug(f"Error in {discovery_source.upper()} search for query '{search_query}': {e}")
continue
# Strategy 4: Extended search with higher limit (last resort)
if not found_track:
logger.debug("Beatport Strategy 4: Extended search with limit=50")
query = f"{track_artist} {track_title}"
if use_spotify:
extended_results = deps.spotify_client.search_tracks(query, limit=50)
else:
extended_results = itunes_client_instance.search_tracks(query, limit=50)
if extended_results:
match, confidence, _ = deps.discovery_score_candidates(
track_title, track_artist, 0, extended_results
)
if match and confidence >= min_confidence:
found_track = match
best_confidence = confidence
logger.debug(f"Strategy 4 Beatport match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
if found_track:
logger.info(f"Final Beatport match: {found_track.artists[0]} - {found_track.name} (confidence: {best_confidence:.3f})")
else:
logger.warning(f"No suitable match found (best confidence was {best_confidence:.3f}, required {min_confidence:.3f})")
# Create result entry
result_entry = {
'index': i, # Add index for frontend table row identification
'beatport_track': {
'title': track_title,
'artist': track_artist
},
'status': 'found' if found_track else 'not_found',
'status_class': 'found' if found_track else 'not-found',
'discovery_source': discovery_source,
'confidence': best_confidence
}
if found_track:
if use_spotify:
# SPOTIFY result formatting
# Debug: show available attributes
logger.debug(f"Spotify track attributes: {dir(found_track)}")
# Format artists correctly for frontend compatibility
formatted_artists = []
if isinstance(found_track.artists, list):
# If it's already a list of strings, convert to objects with 'name' property
for artist in found_track.artists:
if isinstance(artist, str):
formatted_artists.append({'name': artist})
else:
# If it's already an object, use as-is
formatted_artists.append(artist)
else:
# Single artist case
formatted_artists = [{'name': str(found_track.artists)}]
# Use full album object from raw Spotify data if available
album_data = best_raw_track.get('album', {}) if best_raw_track else {}
if not album_data:
# Fallback to string album name
album_data = {'name': found_track.album, 'album_type': 'album', 'release_date': getattr(found_track, 'release_date', '') or '', 'images': []}
result_entry['spotify_data'] = {
'name': found_track.name,
'artists': formatted_artists, # Now formatted as list of objects with 'name' property
'album': album_data, # Full album object with images
'id': found_track.id,
'source': 'spotify'
}
else:
# ITUNES result formatting
# Note: iTunes Track dataclass has 'artists' (list) and 'image_url', not 'artist' and 'artwork_url'
result_artists = found_track.artists if hasattr(found_track, 'artists') else []
result_artist = result_artists[0] if result_artists else 'Unknown'
result_name = found_track.name if hasattr(found_track, 'name') else 'Unknown'
album_name = found_track.album if hasattr(found_track, 'album') else 'Unknown Album'
image_url = found_track.image_url if hasattr(found_track, 'image_url') else ''
track_id = found_track.id if hasattr(found_track, 'id') else ''
# Format artists as list of objects for frontend compatibility
formatted_artists = [{'name': result_artist}]
# Build album data with artwork
album_data = {
'name': album_name,
'album_type': 'album',
'release_date': getattr(found_track, 'release_date', '') or '',
'images': [{'url': image_url, 'height': 300, 'width': 300}] if image_url else []
}
result_entry['spotify_data'] = { # Use same key for frontend compatibility
'name': result_name,
'artists': formatted_artists,
'album': album_data,
'id': track_id,
'source': discovery_source
}
state['spotify_matches'] += 1
# Save to discovery cache (normalize artists from [{name:str}] to [str] for canonical format)
if best_confidence >= 0.75:
try:
cache_data = dict(result_entry['spotify_data'])
cache_artists = cache_data.get('artists', [])
if cache_artists and isinstance(cache_artists[0], dict):
cache_data['artists'] = [a.get('name', '') for a in cache_artists]
# Extract image URL for discovery pool display
if 'image_url' not in cache_data:
_bp_album = cache_data.get('album', {})
_bp_images = _bp_album.get('images', []) if isinstance(_bp_album, dict) else []
cache_data['image_url'] = _bp_images[0].get('url', '') if _bp_images else ''
cache_db = deps.get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], discovery_source, best_confidence,
cache_data, track_title, track_artist
)
logger.debug(f"CACHE SAVED: {track_artist} - {track_title} (confidence: {best_confidence:.3f})")
except Exception as cache_err:
logger.error(f"Cache save error: {cache_err}")
# Auto Wing It fallback for unmatched tracks
if result_entry.get('status_class') == 'not-found':
bp_t = result_entry.get('beatport_track', {})
stub = deps.build_discovery_wing_it_stub(
bp_t.get('title', ''),
bp_t.get('artist', ''),
)
result_entry['status'] = 'found'
result_entry['status_class'] = 'wing-it'
result_entry['spotify_data'] = stub
result_entry['match_data'] = stub
result_entry['wing_it_fallback'] = True
result_entry['confidence'] = 0
state['spotify_matches'] = state.get('spotify_matches', 0) + 1
state['wing_it_count'] = state.get('wing_it_count', 0) + 1
state['discovery_results'].append(result_entry)
# Small delay to avoid rate limiting
time.sleep(0.1)
except Exception as e:
logger.error(f"Error processing Beatport track {i}: {e}")
# Add error result
state['discovery_results'].append({
'index': i, # Add index for frontend table row identification
'beatport_track': {
'title': track.get('name', 'Unknown'), # Changed from 'title' to 'name' to match track structure
'artist': track.get('artists', ['Unknown'])[0] if isinstance(track.get('artists'), list) else 'Unknown'
},
'status': 'error',
'status_class': 'error', # Add status class for CSS styling
'error': str(e),
'discovery_source': discovery_source
})
# Mark discovery as complete
state['discovery_progress'] = 100
state['phase'] = 'discovered'
state['status'] = 'discovered'
# Add activity for completion
chart_name = chart.get('name', 'Unknown Chart')
source_label = discovery_source.upper()
deps.add_activity_item("", f"Beatport Discovery Complete ({source_label})",
f"'{chart_name}' - {state['spotify_matches']}/{len(tracks)} tracks found", "Now")
logger.info(f"Beatport discovery complete ({source_label}): {state['spotify_matches']}/{len(tracks)} tracks found")
# Sync discovery results back to mirrored playlist
deps.sync_discovery_results_to_mirrored('beatport', url_hash, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
except Exception as e:
logger.error(f"Error in Beatport discovery worker: {e}")
if url_hash in deps.beatport_chart_states:
deps.beatport_chart_states[url_hash]['status'] = 'error'
deps.beatport_chart_states[url_hash]['phase'] = 'fresh'
finally:
deps.resume_enrichment_workers(_ew_state, 'Beatport discovery')

327
core/discovery/deezer.py
View file

@ -1,327 +0,0 @@
"""Background worker for Deezer playlist discovery.
`run_deezer_discovery_worker(playlist_id, deps)` is the function the
Deezer discovery start-endpoint submits to the deezer discovery executor
to match each Deezer playlist track against Spotify (preferred) or iTunes
(fallback):
1. Pause enrichment workers (release shared resources).
2. For each Deezer track:
- Cancellation gate (state['cancelled']).
- Discovery cache lookup; cache hit short-circuits the search.
- SimpleNamespace duck-type → `_search_spotify_for_tidal_track`
(shared search helper, returns tuple for Spotify or dict for iTunes).
- On match: build `match_data` (Spotify path preserves track_number /
disc_number from raw API data, image extracted from album images).
- Save to discovery cache.
- On miss: Wing It stub created from raw Deezer track data.
3. After all tracks: phase='discovered', activity feed entry.
4. Sync discovery results back to mirrored playlist via
`_sync_discovery_results_to_mirrored`.
5. On error: state['phase']='error' + status with error string.
6. Finally: resume enrichment workers.
Lifted verbatim from web_server.py. Wide dependency surface (Spotify and
iTunes clients, multiple metadata helpers, state dict, mirrored sync,
shared tidal search helper) all injected via `DeezerDiscoveryDeps`.
"""
from __future__ import annotations
import logging
import time
import types
from dataclasses import dataclass
from typing import Any, Callable
logger = logging.getLogger(__name__)
@dataclass
class DeezerDiscoveryDeps:
"""Bundle of cross-cutting deps the Deezer discovery worker needs."""
deezer_discovery_states: dict
spotify_client: Any
pause_enrichment_workers: Callable[[str], dict]
resume_enrichment_workers: Callable[[dict, str], None]
get_active_discovery_source: Callable[[], str]
get_metadata_fallback_client: Callable[[], Any]
get_discovery_cache_key: Callable
get_database: Callable[[], Any]
validate_discovery_cache_artist: Callable
search_spotify_for_tidal_track: Callable
build_discovery_wing_it_stub: Callable
add_activity_item: Callable
sync_discovery_results_to_mirrored: Callable
def run_deezer_discovery_worker(playlist_id, deps: DeezerDiscoveryDeps):
"""Background worker for Deezer discovery process (Spotify preferred, iTunes fallback)"""
_ew_state = {}
try:
_ew_state = deps.pause_enrichment_workers('Deezer discovery')
state = deps.deezer_discovery_states[playlist_id]
playlist = state['playlist']
# Determine which provider to use
discovery_source = deps.get_active_discovery_source()
use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
# Initialize fallback client if needed
itunes_client_instance = None
if not use_spotify:
itunes_client_instance = deps.get_metadata_fallback_client()
logger.info(f"Starting Deezer discovery for: {playlist['name']} (using {discovery_source.upper()})")
# Store discovery source in state for frontend
state['discovery_source'] = discovery_source
successful_discoveries = 0
tracks = playlist['tracks']
for i, deezer_track in enumerate(tracks):
if state.get('cancelled', False):
break
try:
track_name = deezer_track['name']
track_artists = deezer_track['artists']
track_id = deezer_track['id']
track_album = deezer_track.get('album', '')
track_duration_ms = deezer_track.get('duration_ms', 0)
logger.info(f"[{i+1}/{len(tracks)}] Searching {discovery_source.upper()}: {track_name} by {', '.join(track_artists)}")
# Check discovery cache first
cache_key = deps.get_discovery_cache_key(track_name, track_artists[0] if track_artists else '')
try:
cache_db = deps.get_database()
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
if cached_match and deps.validate_discovery_cache_artist(track_artists[0] if track_artists else '', cached_match):
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {track_name} by {', '.join(track_artists)}")
# Extract display-friendly artist string from cached match
cached_artists = cached_match.get('artists', [])
if cached_artists:
cached_artist_str = ', '.join(
a if isinstance(a, str) else a.get('name', '') for a in cached_artists
)
else:
cached_artist_str = ''
cached_album = cached_match.get('album', '')
if isinstance(cached_album, dict):
cached_album = cached_album.get('name', '')
result = {
'deezer_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
'album': track_album,
'duration_ms': track_duration_ms,
},
'spotify_data': cached_match,
'match_data': cached_match,
'status': 'Found',
'status_class': 'found',
'spotify_track': cached_match.get('name', ''),
'spotify_artist': cached_artist_str,
'spotify_album': cached_album,
'spotify_id': cached_match.get('id', ''),
'discovery_source': discovery_source,
'index': i
}
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
continue
except Exception as cache_err:
logger.error(f"Cache lookup error: {cache_err}")
# Create a SimpleNamespace duck-type object for _search_spotify_for_tidal_track
track_ns = types.SimpleNamespace(
id=track_id,
name=track_name,
artists=track_artists,
album=track_album,
duration_ms=track_duration_ms
)
# Use the search function with appropriate provider
track_result = deps.search_spotify_for_tidal_track(
track_ns,
use_spotify=use_spotify,
itunes_client=itunes_client_instance
)
# Create result entry
result = {
'deezer_track': {
'id': track_id,
'name': track_name,
'artists': track_artists or [],
'album': track_album,
'duration_ms': track_duration_ms,
},
'spotify_data': None,
'match_data': None,
'status': 'Not Found',
'status_class': 'not-found',
'spotify_track': '',
'spotify_artist': '',
'spotify_album': '',
'discovery_source': discovery_source
}
match_confidence = 0.0
if use_spotify and isinstance(track_result, tuple):
# Spotify: Function returns (Track, raw_data, confidence)
track_obj, raw_track_data, match_confidence = track_result
album_obj = raw_track_data.get('album', {}) if raw_track_data else {}
# Ensure album has a name — fall back to track_obj.album if raw_data was missing
if isinstance(album_obj, dict) and not album_obj.get('name') and track_obj.album:
album_obj['name'] = track_obj.album
elif not album_obj and track_obj.album:
album_obj = {'name': track_obj.album}
# Ensure release_date is present (raw Spotify data has it, but fallback may not)
if isinstance(album_obj, dict) and not album_obj.get('release_date'):
album_obj['release_date'] = getattr(track_obj, 'release_date', '') or ''
# Extract image URL from album data or track object
_album_images = album_obj.get('images', []) if isinstance(album_obj, dict) else []
_image_url = _album_images[0].get('url', '') if _album_images else (getattr(track_obj, 'image_url', '') or '')
match_data = {
'id': track_obj.id,
'name': track_obj.name,
'artists': track_obj.artists,
'album': album_obj,
'duration_ms': track_obj.duration_ms,
'external_urls': track_obj.external_urls,
'image_url': _image_url,
'source': 'spotify'
}
# Preserve track_number/disc_number from raw Spotify API data
if raw_track_data and raw_track_data.get('track_number'):
match_data['track_number'] = raw_track_data['track_number']
if raw_track_data and raw_track_data.get('disc_number'):
match_data['disc_number'] = raw_track_data['disc_number']
result['spotify_data'] = match_data
result['match_data'] = match_data
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = track_obj.name
result['spotify_artist'] = ', '.join(track_obj.artists) if isinstance(track_obj.artists, list) else str(track_obj.artists)
result['spotify_album'] = album_obj.get('name', '') if isinstance(album_obj, dict) else str(album_obj)
result['spotify_id'] = track_obj.id
result['confidence'] = match_confidence
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
elif not use_spotify and track_result and isinstance(track_result, dict):
# Fallback: Function returns a dict with track data (includes 'confidence' key)
match_confidence = track_result.pop('confidence', 0.80)
match_data = track_result
match_data['source'] = discovery_source
# Extract image URL from album images
_fb_album = match_data.get('album', {})
_fb_images = _fb_album.get('images', []) if isinstance(_fb_album, dict) else []
if _fb_images and 'image_url' not in match_data:
match_data['image_url'] = _fb_images[0].get('url', '')
result['spotify_data'] = match_data
result['match_data'] = match_data
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = match_data.get('name', '')
itunes_artists = match_data.get('artists', [])
result['spotify_artist'] = ', '.join(a if isinstance(a, str) else a.get('name', '') for a in itunes_artists) if itunes_artists else ''
result['spotify_album'] = match_data.get('album', {}).get('name', '') if isinstance(match_data.get('album'), dict) else match_data.get('album', '')
result['spotify_id'] = match_data.get('id', '')
result['confidence'] = match_confidence
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
# Save to discovery cache if match found
if result['status_class'] == 'found' and result.get('match_data'):
try:
cache_db = deps.get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], discovery_source, match_confidence,
result['match_data'], track_name,
track_artists[0] if track_artists else ''
)
logger.info(f"CACHE SAVED: {track_name} (confidence: {match_confidence:.3f})")
except Exception as cache_err:
logger.error(f"Cache save error: {cache_err}")
# Auto Wing It fallback for unmatched tracks
if result['status_class'] == 'not-found':
deezer_t = result.get('deezer_track', {})
stub = deps.build_discovery_wing_it_stub(
deezer_t.get('name', ''),
', '.join(deezer_t.get('artists', [])),
deezer_t.get('duration_ms', 0)
)
result['status'] = 'Wing It'
result['status_class'] = 'wing-it'
result['spotify_data'] = stub
result['match_data'] = stub
result['spotify_track'] = deezer_t.get('name', '')
result['spotify_artist'] = ', '.join(deezer_t.get('artists', []))
result['wing_it_fallback'] = True
result['confidence'] = 0
successful_discoveries += 1
state['spotify_matches'] = successful_discoveries
state['wing_it_count'] = state.get('wing_it_count', 0) + 1
result['index'] = i
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
# Add delay between requests
time.sleep(0.1)
except Exception as e:
logger.error(f"Error processing track {i+1}: {e}")
# Add error result
result = {
'deezer_track': {
'name': deezer_track.get('name', 'Unknown'),
'artists': deezer_track.get('artists', []),
},
'spotify_data': None,
'match_data': None,
'status': 'Error',
'status_class': 'error',
'spotify_track': '',
'spotify_artist': '',
'spotify_album': '',
'error': str(e),
'discovery_source': discovery_source,
'index': i
}
state['discovery_results'].append(result)
state['discovery_progress'] = int(((i + 1) / len(tracks)) * 100)
# Mark as complete
state['phase'] = 'discovered'
state['status'] = 'discovered'
state['discovery_progress'] = 100
# Add activity for discovery completion
source_label = discovery_source.upper()
deps.add_activity_item("", f"Deezer Discovery Complete ({source_label})", f"'{playlist['name']}' - {successful_discoveries}/{len(tracks)} tracks found", "Now")
logger.info(f"Deezer discovery complete ({source_label}): {successful_discoveries}/{len(tracks)} tracks found")
# Sync discovery results back to mirrored playlist
deps.sync_discovery_results_to_mirrored('deezer', playlist_id, state.get('discovery_results', []), discovery_source, profile_id=state.get('_profile_id', 1))
except Exception as e:
logger.error(f"Error in Deezer discovery worker: {e}")
if playlist_id in deps.deezer_discovery_states:
deps.deezer_discovery_states[playlist_id]['phase'] = 'error'
deps.deezer_discovery_states[playlist_id]['status'] = f'error: {str(e)}'
finally:
deps.resume_enrichment_workers(_ew_state, 'Deezer discovery')

767
core/discovery/endpoints.py
View file

@ -1,767 +0,0 @@
"""Generic, source-agnostic helpers for the playlist-discovery route layer.
The discovery/sync endpoints in ``web_server.py`` were copy-pasted once per
source (Tidal, Deezer, Qobuz, Spotify-public, iTunes-link, YouTube,
ListenBrainz, Beatport). The per-source copies differ only by a source label
string and which ``<source>_discovery_states`` global they read. This module
lifts the source-agnostic pieces into importable, unit-testable helpers so the
route functions become thin wrappers — exactly preserving behavior (1:1).
Each helper is lifted verbatim from its web_server.py counterpart; any
per-source quirk that genuinely differs (e.g. Beatport's distinct result
shape) is intentionally NOT routed through here and stays in its own function.
"""
from __future__ import annotations
import time
from typing import Any, Dict, List, Tuple
from utils.logging_config import get_logger
logger = get_logger("discovery.endpoints")
def convert_results_to_spotify_tracks(
discovery_results: List[Dict[str, Any]],
source_label: str,
) -> List[Dict[str, Any]]:
"""Convert a source's discovery results into the Spotify-track dicts the
sync pipeline expects.
Lifted verbatim from the per-source ``convert_<source>_results_to_spotify_tracks``
functions (and the already-generic ``_convert_link_results_to_spotify_tracks``),
which were byte-identical apart from the ``source_label`` used in the log
line. Two input shapes are supported, matching the originals exactly:
- ``spotify_data`` (manual-fix shape): copied through, preserving optional
``track_number`` / ``disc_number``.
- ``spotify_track`` + ``status_class == 'found'`` (auto-discovery shape):
rebuilt from the flat ``spotify_*`` fields.
Any result matching neither shape is skipped, identical to the originals.
NOTE: Beatport deliberately does NOT use this — its converter coerces
artist objects to strings and emits a different track shape (``source``
field, album dict), so it keeps its own implementation.
"""
spotify_tracks: List[Dict[str, Any]] = []
for result in discovery_results:
# Support both data formats: spotify_data (manual fixes) and individual
# fields (automatic discovery).
if result.get('spotify_data'):
spotify_data = result['spotify_data']
track = {
'id': spotify_data['id'],
'name': spotify_data['name'],
'artists': spotify_data['artists'],
'album': spotify_data['album'],
'duration_ms': spotify_data.get('duration_ms', 0),
}
if spotify_data.get('track_number'):
track['track_number'] = spotify_data['track_number']
if spotify_data.get('disc_number'):
track['disc_number'] = spotify_data['disc_number']
spotify_tracks.append(track)
elif result.get('spotify_track') and result.get('status_class') == 'found':
spotify_tracks.append({
'id': result.get('spotify_id', 'unknown'),
'name': result.get('spotify_track', 'Unknown Track'),
'artists': [result.get('spotify_artist', 'Unknown Artist')] if result.get('spotify_artist') else ['Unknown Artist'],
'album': result.get('spotify_album', 'Unknown Album'),
'duration_ms': 0,
})
logger.info(f"Converted {len(spotify_tracks)} {source_label} matches to Spotify tracks for sync")
return spotify_tracks
def cancel_sync(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
sync_lock: Any,
sync_states: Dict[str, Any],
active_sync_workers: Dict[str, Any],
) -> Tuple[Dict[str, Any], int]:
"""Cancel an in-progress sync for one discovery playlist.
1:1 lift of the byte-identical ``cancel_<source>_sync`` bodies (Tidal,
Deezer, Qobuz, Spotify-Public, iTunes-Link, YouTube, ListenBrainz). The
caller passes the already-resolved state key (ListenBrainz transforms it
via ``_lb_state_key`` first), the source ``label``, the exact 404 message
(iTunes-Link uses "iTunes Link not found", not "... playlist not found"),
and the shared sync infrastructure (so this stays free of web_server
globals / Flask).
Returns ``(payload_dict, status_code)``; the caller wraps in ``jsonify``.
Beatport is NOT routed here — it cancels a stored ``sync_future`` and
returns a different payload.
"""
try:
if key not in states:
# Idempotent: the live discovery state is gone (a restart wiped the
# in-memory state, or it was already cancelled). Cancelling a sync
# that isn't running is a no-op SUCCESS, not a 404 — otherwise a
# mirrored playlist (e.g. a ListenBrainz weekly) whose state vanished
# is permanently wedged with "playlist not found" and can never be
# re-synced or dismissed (#702).
return {"success": True, "message": f"No active {label} sync to cancel"}, 200
state = states[key]
state['last_accessed'] = time.time()
sync_playlist_id = state.get('sync_playlist_id')
if sync_playlist_id:
with sync_lock:
sync_states[sync_playlist_id] = {"status": "cancelled"}
if sync_playlist_id in active_sync_workers:
del active_sync_workers[sync_playlist_id]
state['phase'] = 'discovered'
state['sync_playlist_id'] = None
state['sync_progress'] = {}
return {"success": True, "message": f"{label} sync cancelled"}, 200
except Exception as e:
logger.error(f"Error cancelling {label} sync: {e}")
return {"error": str(e)}, 500
def delete_playlist_state(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
) -> Tuple[Dict[str, Any], int]:
"""Delete a discovery playlist's state entry, cancelling any active
discovery first.
1:1 lift of the byte-identical ``delete_<source>_playlist`` bodies
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
The iTunes-Link / YouTube / ListenBrainz / Beatport deletes intentionally
keep their own bodies — they differ in success message, info-log wording,
name extraction, and/or key transform.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
if 'discovery_future' in state and state['discovery_future']:
state['discovery_future'].cancel()
del states[key]
logger.info(f"Deleted {label} playlist state: {key}")
return {"success": True, "message": "Playlist deleted"}, 200
except Exception as e:
logger.error(f"Error deleting {label} playlist: {e}")
return {"error": str(e)}, 500
# --- playlist-name accessors -------------------------------------------------
# The per-source sync-status handlers read the display name three different
# ways. Each is reproduced verbatim so the 1:1 behavior (including which ones
# raise vs. fall back to 'Unknown Playlist') is preserved.
def playlist_name_attr_or_unknown(state: Dict[str, Any]) -> str:
"""Tidal: playlist is an object — use ``.name`` or 'Unknown Playlist'."""
pl = state.get('playlist')
return pl.name if pl and hasattr(pl, 'name') else 'Unknown Playlist'
def playlist_name_strict(state: Dict[str, Any]) -> str:
"""Deezer / Qobuz / Spotify-Public / iTunes-Link: strict dict access —
raises (→ 500) if 'playlist' is missing, exactly like the originals."""
return state['playlist']['name']
def playlist_name_safe(state: Dict[str, Any]) -> str:
"""YouTube / ListenBrainz: safe dict access, defaulting to 'Unknown
Playlist'."""
return state.get('playlist', {}).get('name', 'Unknown Playlist')
def playlist_name_obj(state: Dict[str, Any]) -> str:
"""Tidal start-sync: playlist is an object — strict ``.name`` (raises if
absent, exactly like the original)."""
return state['playlist'].name
def playlist_image_obj(state: Dict[str, Any]) -> str:
"""Tidal: ``getattr(playlist, 'image_url', '')`` (object attribute)."""
return getattr(state['playlist'], 'image_url', '')
def playlist_image_dict(state: Dict[str, Any]) -> str:
"""Deezer/Qobuz/Spotify-Public/YouTube: ``playlist.get('image_url', '')``
(dict access)."""
return state['playlist'].get('image_url', '')
def get_sync_status(
states: Dict[str, Any],
key: str,
*,
not_found_message: str,
error_label: str,
activity_subject: str,
playlist_name_getter,
sync_lock: Any,
sync_states: Dict[str, Any],
add_activity_item,
) -> Tuple[Dict[str, Any], int]:
"""Report sync status for one discovery playlist, posting an activity-feed
item when the sync finishes or errors.
1:1 lift of the ``get_<source>_sync_status`` bodies (Tidal, Deezer, Qobuz,
Spotify-Public, iTunes-Link, YouTube, ListenBrainz). Per-source variation
is captured by the parameters:
- ``not_found_message`` — the 404 string (iTunes-Link drops "playlist").
- ``error_label`` — used in the except log ("Error getting <X> sync status").
- ``activity_subject`` — the activity-feed prefix; note Spotify-Public uses
"Spotify Link playlist" while its error_label is "Spotify Public".
- ``playlist_name_getter`` — one of the accessors above (attr/strict/safe);
the strict one can raise, matching the originals (→ 500). The state's
phase/sync_progress are mutated BEFORE the name is read, so a raising
getter leaves the same partial mutation the original did.
Beatport is NOT routed here — it returns a different payload (``status``
not ``sync_status``, includes ``sync_id``, no lock, ``chart`` key).
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
sync_playlist_id = state.get('sync_playlist_id')
if not sync_playlist_id:
return {"error": "No sync in progress"}, 404
with sync_lock:
sync_state = sync_states.get(sync_playlist_id, {})
response = {
'phase': state['phase'],
'sync_status': sync_state.get('status', 'unknown'),
'progress': sync_state.get('progress', {}),
'complete': sync_state.get('status') == 'finished',
'error': sync_state.get('error'),
}
if sync_state.get('status') == 'finished':
state['phase'] = 'sync_complete'
state['sync_progress'] = sync_state.get('progress', {})
playlist_name = playlist_name_getter(state)
add_activity_item("", "Sync Complete", f"{activity_subject} '{playlist_name}' synced successfully", "Now")
elif sync_state.get('status') == 'error':
state['phase'] = 'discovered'
playlist_name = playlist_name_getter(state)
add_activity_item("", "Sync Failed", f"{activity_subject} '{playlist_name}' sync failed", "Now")
return response, 200
except Exception as e:
logger.error(f"Error getting {error_label} sync status: {e}")
return {"error": str(e)}, 500
def get_discovery_status(
states: Dict[str, Any],
key: str,
*,
not_found_message: str,
error_label: str,
) -> Tuple[Dict[str, Any], int]:
"""Report real-time discovery progress/results for one playlist.
1:1 lift of the byte-identical ``get_<source>_discovery_status`` bodies.
Unlike sync-status, this shape is identical for ALL eight sources —
Beatport included — so it folds in too. Only the 404 message
(".../discovery not found" vs ".../playlist not found" vs "Beatport chart
not found") and the except-log label vary, both passed in. The caller
resolves the key (ListenBrainz via ``_lb_state_key``).
Returns ``(payload, status_code)``.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
return {
'phase': state['phase'],
'status': state['status'],
'progress': state['discovery_progress'],
'spotify_matches': state['spotify_matches'],
'spotify_total': state['spotify_total'],
'results': state['discovery_results'],
'complete': state['phase'] == 'discovered',
}, 200
except Exception as e:
logger.error(f"Error getting {error_label} discovery status: {e}")
return {"error": str(e)}, 500
def reset_playlist(
states: Dict[str, Any],
key: str,
*,
label: str,
not_found_message: str,
) -> Tuple[Dict[str, Any], int]:
"""Reset a discovery playlist back to the 'fresh' phase, clearing all
discovery/sync data while preserving the original playlist payload.
1:1 lift of the byte-identical ``reset_<source>_playlist`` bodies
(Tidal, Deezer, Qobuz, Spotify-Public). Returns ``(payload, status_code)``.
NOT folded in (genuinely divergent): YouTube (status -> 'parsed', no
download_process_id, logs the playlist name, "reset to fresh state"),
ListenBrainz (status -> 'cached', logs playlist title, returns
{"phase": "fresh"}), iTunes-Link (uses state.update, no info log, distinct
message). Those keep their own bodies.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
if 'discovery_future' in state and state['discovery_future']:
state['discovery_future'].cancel()
state['phase'] = 'fresh'
state['status'] = 'fresh'
state['discovery_results'] = []
state['discovery_progress'] = 0
state['spotify_matches'] = 0
state['sync_playlist_id'] = None
state['converted_spotify_playlist_id'] = None
state['download_process_id'] = None
state['sync_progress'] = {}
state['discovery_future'] = None
state['last_accessed'] = time.time()
logger.info(f"Reset {label} playlist to fresh: {key}")
return {"success": True, "message": "Playlist reset to fresh phase"}, 200
except Exception as e:
logger.error(f"Error resetting {label} playlist: {e}")
return {"error": str(e)}, 500
def get_playlist_states(
states: Dict[str, Any],
*,
error_label: str,
info_log_label: str = None,
) -> Tuple[Dict[str, Any], int]:
"""Return all stored discovery states for a source as a list for frontend
card hydration (``{"states": [...]}``).
1:1 lift of the ``get_<source>_playlist_states`` bodies (Tidal, Deezer,
Qobuz, Spotify-Public, iTunes-Link), which build the same per-entry dict.
iTunes-Link is the only one without the "Returning N ..." info log, so
``info_log_label`` is optional (pass None to suppress it, as iTunes did).
NOT folded in: the YouTube/ListenBrainz ``get_all_*_playlists`` endpoints —
they return ``{"playlists": [...]}`` (different key + fields: url/created_at,
no discovery_results) and filter mirrored/profile-scoped entries.
"""
try:
result = []
current_time = time.time()
for key, state in states.items():
state['last_accessed'] = current_time
result.append({
'playlist_id': key,
'phase': state['phase'],
'status': state['status'],
'discovery_progress': state['discovery_progress'],
'spotify_matches': state['spotify_matches'],
'spotify_total': state['spotify_total'],
'discovery_results': state['discovery_results'],
'converted_spotify_playlist_id': state.get('converted_spotify_playlist_id'),
'download_process_id': state.get('download_process_id'),
'last_accessed': state['last_accessed'],
})
if info_log_label:
logger.info(f"Returning {len(result)} stored {info_log_label} playlist states for hydration")
return {"states": result}, 200
except Exception as e:
logger.error(f"Error getting {error_label} playlist states: {e}")
return {"error": str(e)}, 500
def save_bubble_snapshot(
get_json,
*,
payload_key: str,
no_data_error: str,
snapshot_kind: str,
success_noun: str,
log_subject: str,
log_noun: str,
get_database,
get_current_profile_id,
) -> Tuple[Dict[str, Any], int]:
"""Persist a bubble/download snapshot for cross-refresh hydration.
1:1 lift of the four structurally-identical snapshot endpoints
(discover_downloads, artist_bubbles, search_bubbles, beatport_bubbles),
which differ only by:
- ``payload_key`` ('downloads' for discover, 'bubbles' for the rest) and
its ``no_data_error`` message.
- ``snapshot_kind`` — the db.save_bubble_snapshot category.
- ``success_noun`` — fills "Snapshot saved with N <noun>".
- ``log_subject`` / ``log_noun`` — the info ("Saved <subject>: N <noun>")
and except ("Error saving <subject>") log lines.
Returns ``(payload, status_code)``. ``get_json`` is invoked inside the try
like the original ``request.json``.
"""
try:
from datetime import datetime
data = get_json()
if not data or payload_key not in data:
return {'success': False, 'error': no_data_error}, 400
items = data[payload_key]
db = get_database()
db.save_bubble_snapshot(snapshot_kind, items, profile_id=get_current_profile_id())
count = len(items)
logger.info(f"Saved {log_subject}: {count} {log_noun}")
return {
'success': True,
'message': f'Snapshot saved with {count} {success_noun}',
'timestamp': datetime.now().isoformat(),
}, 200
except Exception as e:
logger.error(f"Error saving {log_subject}: {e}")
import traceback
traceback.print_exc()
return {'success': False, 'error': str(e)}, 500
def update_playlist_phase(
states: Dict[str, Any],
key: str,
get_json,
*,
not_found_message: str,
error_label: str,
valid_phases: List[str],
apply_extra_fields: bool,
) -> Tuple[Dict[str, Any], int]:
"""Update a discovery playlist's phase (used when the modal closes, e.g. to
reset download_complete -> discovered).
1:1 lift of the ``update_<source>_playlist_phase`` bodies for the five
sources with the identical validation + full-message response (Tidal,
Deezer, Qobuz, Spotify-Public, YouTube). Per-source params:
- ``valid_phases`` — YouTube's list additionally includes 'parsed'.
- ``apply_extra_fields`` — Deezer/Qobuz/Spotify-Public also persist
download_process_id / converted_spotify_playlist_id from the body;
Tidal/YouTube do NOT (so pass False to keep them 1:1).
- ``not_found_message`` / ``error_label``; ``get_json`` invoked inside the
try like the original ``request.get_json()``.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link — it uses ``data.get('phase')`` (no separate
"Phase not provided" 400) and returns a no-message payload.
"""
try:
if key not in states:
return {"error": not_found_message}, 404
data = get_json()
if not data or 'phase' not in data:
return {"error": "Phase not provided"}, 400
new_phase = data['phase']
if new_phase not in valid_phases:
return {"error": f"Invalid phase. Must be one of: {', '.join(valid_phases)}"}, 400
state = states[key]
old_phase = state.get('phase', 'unknown')
state['phase'] = new_phase
state['last_accessed'] = time.time()
if apply_extra_fields:
if 'download_process_id' in data:
state['download_process_id'] = data['download_process_id']
if 'converted_spotify_playlist_id' in data:
state['converted_spotify_playlist_id'] = data['converted_spotify_playlist_id']
logger.info(f"Updated {error_label} playlist {key} phase: {old_phase} → {new_phase}")
return {"success": True, "message": f"Phase updated to {new_phase}", "old_phase": old_phase, "new_phase": new_phase}, 200
except Exception as e:
logger.error(f"Error updating {error_label} playlist phase: {e}")
return {"error": str(e)}, 500
def first_artist_str_or_obj(original_track: Dict[str, Any]) -> str:
"""Tidal: first artist from an artists list that may hold strings OR
objects ({'name': ...}); '' when empty."""
artists = original_track.get('artists', [])
if artists:
return artists[0] if isinstance(artists[0], str) else artists[0].get('name', '')
return ''
def first_artist_plain(original_track: Dict[str, Any]) -> str:
"""Deezer/Qobuz/Spotify-Public: first artist assuming a list of strings;
'' when empty."""
artists = original_track.get('artists', [])
return artists[0] if artists else ''
def update_discovery_match(
states: Dict[str, Any],
get_json,
*,
source_log_label: str,
error_label: str,
original_track_key: str,
original_artist_getter,
join_artist_names,
extract_artist_name,
build_fix_modal_spotify_data,
get_discovery_cache_key,
get_database,
get_active_discovery_source,
) -> Tuple[Dict[str, Any], int]:
"""Apply a manually-selected Spotify track to a discovery result (the
fix-modal flow) and persist it to the discovery cache.
1:1 lift of the ``update_<source>_discovery_match`` bodies for the four
sources with the identical structure (Tidal, Deezer, Qobuz, Spotify-Public).
Per-source pieces are params:
- ``source_log_label`` (lowercase, e.g. "tidal") for the "Manual match
updated: ..." line; ``error_label`` for the except log.
- ``original_track_key`` — the raw-source track key on the result
('tidal_track', 'deezer_track', ...).
- ``original_artist_getter`` — Tidal handles string-or-object artists
(``first_artist_str_or_obj``); the rest assume strings
(``first_artist_plain``).
- the web_server helpers (join/extract artist, build_fix_modal_spotify_data,
cache-key, get_database, active-discovery-source) are injected so this
stays free of those globals.
- ``get_json`` is called INSIDE the try (like the original's
``request.get_json()``) so a malformed body yields the same 500.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link (saves spotify_data directly via a different
cache signature), YouTube (multi-key original_track fallback), ListenBrainz
(entirely different unmatch-capable structure, no cache write), Beatport.
"""
try:
data = get_json()
identifier = data.get('identifier')
track_index = data.get('track_index')
spotify_track = data.get('spotify_track')
if not identifier or track_index is None or not spotify_track:
return {'error': 'Missing required fields'}, 400
state = states.get(identifier)
result = None
if state:
if track_index >= len(state['discovery_results']):
return {'error': 'Invalid track index'}, 400
result = state['discovery_results'][track_index]
old_status = result.get('status')
result['status'] = 'Found'
result['status_class'] = 'found'
result['spotify_track'] = spotify_track['name']
result['spotify_artist'] = join_artist_names(spotify_track['artists']) if isinstance(spotify_track['artists'], list) else extract_artist_name(spotify_track['artists'])
result['spotify_album'] = spotify_track['album']
result['spotify_id'] = spotify_track['id']
duration_ms = spotify_track.get('duration_ms', 0)
if duration_ms:
minutes = duration_ms // 60000
seconds = (duration_ms % 60000) // 1000
result['duration'] = f"{minutes}:{seconds:02d}"
else:
result['duration'] = '0:00'
result['spotify_data'] = build_fix_modal_spotify_data(spotify_track)
result['wing_it_fallback'] = False
result['manual_match'] = True
if old_status != 'found' and old_status != 'Found':
state['spotify_matches'] = state.get('spotify_matches', 0) + 1
logger.info(f"Manual match updated: {source_log_label} - {identifier} - track {track_index}")
logger.info(f" → {result['spotify_artist']} - {result['spotify_track']}")
original_track = result.get(original_track_key, {})
original_name = original_track.get('name', spotify_track['name'])
original_artist = original_artist_getter(original_track)
else:
# #843: the in-memory discovery state can be gone — a server restart,
# or an imported playlist that wasn't discovered in THIS process —
# while the card is still shown from persisted data. The DURABLE part
# of a manual fix (writing the match to the discovery cache so future
# syncs resolve it) doesn't need the in-memory state, only the original
# track's name + artist, which the client now sends. Fall back to those
# instead of 404ing the fix into uselessness.
original_name = (data.get('original_name') or '').strip()
original_artist = (data.get('original_artist') or '').strip()
if not original_name and not original_artist:
return {'error': 'Discovery state not found'}, 404
if not original_name:
original_name = spotify_track['name']
# Key the cache by the FIRST artist — every in-memory + sync path uses
# artists[0], but the client may send a joined "A, B, C" string. Without
# this, a multi-artist track would save under a key the sync never looks
# up (full string ≠ first artist), so the fix would silently never apply.
if original_artist:
original_artist = original_artist.split(',')[0].strip()
logger.info(
f"Manual match (no in-memory state) → discovery cache: "
f"{source_log_label} - {identifier} - '{original_name}' by '{original_artist}'"
)
try:
cache_key = get_discovery_cache_key(original_name, original_artist)
artists_list = spotify_track['artists']
if isinstance(artists_list, list):
artists_list = [a if isinstance(a, str) else a.get('name', '') for a in artists_list]
image_url = spotify_track.get('image_url') or ''
album_raw = spotify_track.get('album', '')
if isinstance(album_raw, dict):
album_obj = dict(album_raw)
if image_url and not album_obj.get('image_url'):
album_obj['image_url'] = image_url
if image_url and not album_obj.get('images'):
album_obj['images'] = [{'url': image_url}]
else:
album_obj = {'name': album_raw or ''}
if image_url:
album_obj['image_url'] = image_url
album_obj['images'] = [{'url': image_url}]
matched_data = {
'id': spotify_track['id'],
'name': spotify_track['name'],
'artists': artists_list,
'album': album_obj,
'duration_ms': spotify_track.get('duration_ms', 0),
'image_url': image_url,
'source': 'spotify',
}
cache_db = get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], get_active_discovery_source(), 1.0, matched_data,
original_name, original_artist
)
logger.info(f"Manual fix saved to discovery cache: {original_name} by {original_artist}")
except Exception as cache_err:
logger.error(f"Error saving manual fix to discovery cache: {cache_err}")
return {'success': True, 'result': result}, 200
except Exception as e:
logger.error(f"Error updating {error_label} discovery match: {e}")
return {'error': str(e)}, 500
def start_sync(
states: Dict[str, Any],
key: str,
*,
sync_id_prefix: str,
not_found_message: str,
not_ready_message: str,
convert_fn,
playlist_name_getter,
playlist_image_getter,
activity_label: str,
error_label: str,
sync_lock: Any,
sync_states: Dict[str, Any],
active_sync_workers: Dict[str, Any],
submit_sync_task,
add_activity_item,
) -> Tuple[Dict[str, Any], int]:
"""Kick off a playlist sync from a source's discovered Spotify matches.
1:1 lift of the ``start_<source>_sync`` bodies for the five sources with
the identical flow (Tidal, Deezer, Qobuz, Spotify-Public, YouTube). The
per-source pieces are parameters:
- ``sync_id_prefix`` — the ``f"{prefix}_{key}"`` sync id.
- ``convert_fn`` — the source's discovery->spotify-tracks converter.
- ``playlist_name_getter`` / ``playlist_image_getter`` — Tidal reads an
object (``.name`` / ``getattr``), the rest read a dict; lifted as the
``playlist_name_obj``/``playlist_image_obj`` vs ``playlist_name_strict``/
``playlist_image_dict`` accessors.
- ``activity_label`` vs ``error_label`` — these DIFFER for Spotify-Public:
activity says "Spotify Link Sync Started" while logs say "Spotify Public".
- ``submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks,
playlist_image_url) -> Future`` — wraps sync_executor/_run_sync_task/
get_current_profile_id so this stays free of those globals.
Returns ``(payload, status_code)``.
NOT folded in: iTunes-Link (no final info log), ListenBrainz (submits the
task without an image arg), Beatport (extra debug logging, 'chart' key).
"""
try:
if key not in states:
return {"error": not_found_message}, 404
state = states[key]
state['last_accessed'] = time.time()
if state['phase'] not in ['discovered', 'sync_complete', 'download_complete']:
return {"error": not_ready_message}, 400
spotify_tracks = convert_fn(state['discovery_results'])
if not spotify_tracks:
return {"error": "No Spotify matches found for sync"}, 400
sync_playlist_id = f"{sync_id_prefix}_{key}"
playlist_name = playlist_name_getter(state)
add_activity_item("", f"{activity_label} Sync Started", f"'{playlist_name}' - {len(spotify_tracks)} tracks", "Now")
state['phase'] = 'syncing'
state['sync_playlist_id'] = sync_playlist_id
state['sync_progress'] = {}
with sync_lock:
sync_states[sync_playlist_id] = {"status": "starting", "progress": {}}
playlist_image_url = playlist_image_getter(state)
future = submit_sync_task(sync_playlist_id, playlist_name, spotify_tracks, playlist_image_url)
active_sync_workers[sync_playlist_id] = future
logger.info(f"Started {error_label} sync for: {playlist_name} ({len(spotify_tracks)} tracks)")
return {"success": True, "sync_playlist_id": sync_playlist_id}, 200
except Exception as e:
logger.error(f"Error starting {error_label} sync: {e}")
return {"error": str(e)}, 500

258
core/discovery/hero.py
View file

@ -1,258 +0,0 @@
"""Discover Hero endpoint — lifted from web_server.py.
The function body is byte-identical to the original. The
``spotify_client`` proxy + helper shims let the body resolve its
original names; the more complex ``_get_metadata_fallback_client``
is injected via init() because it composes multiple registry helpers
that web_server.py wires together.
"""
import logging
from flask import g, jsonify
from database.music_database import get_database
from core.metadata.registry import get_primary_source, get_spotify_client
logger = logging.getLogger(__name__)
def get_current_profile_id() -> int:
"""Mirror of web_server.get_current_profile_id — uses Flask g.
Catches RuntimeError too because reading `g` outside a request
context raises that (not AttributeError) — happens when this is
called from background threads (sync, automation, scanners)."""
try:
return g.profile_id
except (AttributeError, RuntimeError):
return 1
def _get_active_discovery_source():
"""Mirror of web_server._get_active_discovery_source — delegates to registry."""
return get_primary_source()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily through core.metadata.registry."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
spotify_client = _SpotifyClientProxy()
# Injected at runtime via init().
_get_metadata_fallback_client = None
def init(get_metadata_fallback_client_fn):
"""Bind web_server's _get_metadata_fallback_client helper."""
global _get_metadata_fallback_client
_get_metadata_fallback_client = get_metadata_fallback_client_fn
def get_discover_hero():
"""Get featured similar artists for hero slideshow"""
try:
database = get_database()
# Determine active source
active_source = _get_active_discovery_source()
logger.info(f"Discover hero using source: {active_source}")
# Import fallback client for non-Spotify lookups
itunes_client = _get_metadata_fallback_client()
# Get top similar artists (excluding watchlist, cycled by last_featured)
# Fetch more than needed since strict source filtering may drop many
pid = get_current_profile_id()
logger.info(f"[Discover Hero] Profile ID: {pid}, Active source: {active_source}")
similar_artists = database.get_top_similar_artists(limit=200, profile_id=pid, require_source=active_source)
# FALLBACK: If no similar artists exist, use watchlist artists for Hero section
if not similar_artists:
logger.warning("[Discover Hero] No similar artists found, falling back to watchlist artists")
watchlist_artists = database.get_watchlist_artists(profile_id=pid)
if not watchlist_artists:
return jsonify({"success": True, "artists": [], "source": active_source})
# Convert watchlist artists to hero format
import random
shuffled_watchlist = list(watchlist_artists)
random.shuffle(shuffled_watchlist)
hero_artists = []
for artist in shuffled_watchlist[:10]:
if active_source == 'spotify':
artist_id = artist.spotify_artist_id
elif active_source == 'deezer':
artist_id = getattr(artist, 'deezer_artist_id', None) or artist.itunes_artist_id
elif active_source == 'musicbrainz':
artist_id = getattr(artist, 'musicbrainz_artist_id', None) or artist.itunes_artist_id
else:
artist_id = artist.itunes_artist_id
if not artist_id:
continue
artist_data = {
"spotify_artist_id": artist.spotify_artist_id,
"itunes_artist_id": artist.itunes_artist_id,
"artist_id": artist_id,
"artist_name": artist.artist_name,
"occurrence_count": 1,
"similarity_rank": 1,
"source": active_source,
"is_watchlist": True
}
# Use cached image from watchlist — no API call needed
if hasattr(artist, 'image_url') and artist.image_url:
artist_data['image_url'] = artist.image_url
hero_artists.append(artist_data)
logger.warning(f"[Discover Hero] Returning {len(hero_artists)} watchlist artists as fallback")
return jsonify({"success": True, "artists": hero_artists, "source": active_source, "fallback": "watchlist"})
# Artists are already filtered by source in SQL — no post-filter needed
valid_artists = list(similar_artists)
# FALLBACK: If no valid artists for fallback source, try to resolve IDs on-the-fly
if active_source in ('itunes', 'deezer', 'musicbrainz') and not valid_artists:
logger.warning(f"[{active_source} Fallback] No artists with {active_source} IDs found, attempting on-the-fly resolution for {len(similar_artists)} artists")
resolved_count = 0
for artist in similar_artists:
existing_id = getattr(artist, f'similar_artist_{active_source}_id', None) or (artist.similar_artist_itunes_id if active_source == 'itunes' else None)
if existing_id:
valid_artists.append(artist)
continue
# Try to resolve ID by name
try:
resolve_client = itunes_client
if active_source == 'musicbrainz':
from core.metadata.registry import get_musicbrainz_client
resolve_client = get_musicbrainz_client()
search_results = resolve_client.search_artists(artist.similar_artist_name, limit=1)
if search_results and len(search_results) > 0:
resolved_id = search_results[0].id
# Cache the resolved ID for future use
if active_source == 'deezer':
database.update_similar_artist_deezer_id(artist.id, resolved_id)
artist.similar_artist_deezer_id = resolved_id
elif active_source == 'musicbrainz':
database.update_similar_artist_musicbrainz_id(artist.id, resolved_id)
artist.similar_artist_musicbrainz_id = resolved_id
else:
database.update_similar_artist_itunes_id(artist.id, resolved_id)
artist.similar_artist_itunes_id = resolved_id
valid_artists.append(artist)
resolved_count += 1
logger.info(f" [Resolved] {artist.similar_artist_name} -> {active_source} ID: {resolved_id}")
except Exception as resolve_err:
logger.error(f" [Failed] Could not resolve {active_source} ID for {artist.similar_artist_name}: {resolve_err}")
# Stop after 10 successful resolutions to avoid rate limiting
if len(valid_artists) >= 10:
break
logger.warning(f"[{active_source} Fallback] Resolved {resolved_count} artists with IDs")
logger.info(f"[Discover Hero] Found {len(valid_artists)} valid artists for source: {active_source}")
# Filter out blacklisted artists
blacklisted = database.get_discovery_blacklist_names()
if blacklisted:
valid_artists = [a for a in valid_artists if a.similar_artist_name.lower() not in blacklisted]
# Take top 10 (already ordered by least-recently-featured, then quality)
similar_artists = valid_artists[:10]
# Convert to JSON format — use cached metadata, only fetch from API if missing
hero_artists = []
for artist in similar_artists:
# Use the ID for the active source, falling back to the other if needed
if active_source == 'spotify':
artist_id = artist.similar_artist_spotify_id or artist.similar_artist_itunes_id
elif active_source == 'deezer':
artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
elif active_source == 'musicbrainz':
artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None) or artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
else:
artist_id = artist.similar_artist_itunes_id or artist.similar_artist_spotify_id
artist_data = {
"spotify_artist_id": artist.similar_artist_spotify_id,
"itunes_artist_id": artist.similar_artist_itunes_id,
"musicbrainz_artist_id": getattr(artist, 'similar_artist_musicbrainz_id', None),
"artist_id": artist_id,
"artist_name": artist.similar_artist_name,
"occurrence_count": artist.occurrence_count,
"similarity_rank": artist.similarity_rank,
"source": active_source
}
# Use cached metadata if available
if artist.image_url:
artist_data['image_url'] = artist.image_url
artist_data['genres'] = artist.genres or []
artist_data['popularity'] = artist.popularity or 0
else:
# No cached metadata — fetch from API and cache for next time
try:
if active_source == 'spotify' and artist.similar_artist_spotify_id:
if spotify_client and spotify_client.is_authenticated():
sp_artist = spotify_client.get_artist(artist.similar_artist_spotify_id)
if sp_artist and sp_artist.get('images'):
artist_data['artist_name'] = sp_artist.get('name', artist.similar_artist_name)
artist_data['image_url'] = sp_artist['images'][0]['url'] if sp_artist['images'] else None
artist_data['genres'] = sp_artist.get('genres', [])
artist_data['popularity'] = sp_artist.get('popularity', 0)
# Cache it
database.update_similar_artist_metadata(
artist.id, artist_data.get('image_url'),
artist_data.get('genres'), artist_data.get('popularity')
)
elif active_source in ('itunes', 'deezer', 'musicbrainz'):
if active_source == 'deezer':
fb_artist_id = getattr(artist, 'similar_artist_deezer_id', None) or artist.similar_artist_itunes_id
fetch_client = itunes_client
elif active_source == 'musicbrainz':
fb_artist_id = getattr(artist, 'similar_artist_musicbrainz_id', None)
from core.metadata.registry import get_musicbrainz_client
fetch_client = get_musicbrainz_client()
else:
fb_artist_id = artist.similar_artist_itunes_id
fetch_client = itunes_client
if fb_artist_id:
fb_artist_data = fetch_client.get_artist(fb_artist_id)
if fb_artist_data:
artist_data['artist_name'] = fb_artist_data.get('name', artist.similar_artist_name)
artist_data['image_url'] = fb_artist_data.get('images', [{}])[0].get('url') if fb_artist_data.get('images') else None
artist_data['genres'] = fb_artist_data.get('genres', [])
artist_data['popularity'] = fb_artist_data.get('popularity', 0)
# Cache it
database.update_similar_artist_metadata(
artist.id, artist_data.get('image_url'),
artist_data.get('genres'), artist_data.get('popularity')
)
except Exception as img_err:
logger.error(f"Could not fetch artist image: {img_err}")
hero_artists.append(artist_data)
# Mark these artists as featured so they cycle to the back of the queue
featured_names = [a["artist_name"] for a in hero_artists]
database.mark_artists_featured(featured_names)
return jsonify({"success": True, "artists": hero_artists, "source": active_source})
except Exception as e:
logger.error(f"Error getting discover hero: {e}")
return jsonify({"success": False, "error": str(e)}), 500

342
core/discovery/listenbrainz.py
View file

@ -1,342 +0,0 @@
"""Background worker for ListenBrainz playlist discovery.
`run_listenbrainz_discovery_worker(state_key, deps)` is the function
the listenbrainz discovery start-endpoint submits to its executor to
match each ListenBrainz playlist track against Spotify (preferred) or
iTunes (fallback):
1. Pause enrichment workers (release shared resources).
2. For each ListenBrainz track:
- Cancellation gate (state['phase'] != 'discovering').
- Discovery cache lookup; cache hit short-circuits the search.
- Strategy 1: matching_engine search queries with confidence scoring.
- Strategy 2: swapped artist/title query.
- Strategy 3: album-based query (if album_name available).
- Strategy 4: extended search with limit=50.
- On match → save to discovery cache.
- On miss → build Wing It stub from raw source data.
3. After loop: phase='discovered', activity feed entry.
4. On error → state['status']='error', phase='fresh'.
5. Finally: resume enrichment workers.
Lifted verbatim from web_server.py. Wide dependency surface (Spotify
and iTunes clients, matching engine, multiple metadata helpers, state
dict, database access) all injected via `ListenbrainzDiscoveryDeps`.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any, Callable
logger = logging.getLogger(__name__)
@dataclass
class ListenbrainzDiscoveryDeps:
"""Bundle of cross-cutting deps the ListenBrainz discovery worker needs."""
listenbrainz_playlist_states: dict
spotify_client: Any
matching_engine: Any
pause_enrichment_workers: Callable[[str], dict]
resume_enrichment_workers: Callable[[dict, str], None]
get_active_discovery_source: Callable[[], str]
get_metadata_fallback_client: Callable[[], Any]
get_discovery_cache_key: Callable
get_database: Callable[[], Any]
validate_discovery_cache_artist: Callable
extract_artist_name: Callable
spotify_rate_limited: Callable[[], bool]
discovery_score_candidates: Callable
get_metadata_cache: Callable[[], Any]
build_discovery_wing_it_stub: Callable
add_activity_item: Callable
def run_listenbrainz_discovery_worker(state_key, deps: ListenbrainzDiscoveryDeps):
"""Background worker for ListenBrainz music discovery process (Spotify preferred, iTunes fallback)"""
playlist_mbid = state_key.split(':', 1)[1] if ':' in state_key else state_key
_ew_state = {}
try:
_ew_state = deps.pause_enrichment_workers('ListenBrainz discovery')
state = deps.listenbrainz_playlist_states[state_key]
playlist = state['playlist']
tracks = playlist['tracks']
# Determine which provider to use (Spotify preferred, iTunes fallback)
discovery_source = deps.get_active_discovery_source()
use_spotify = (discovery_source == 'spotify') and deps.spotify_client and deps.spotify_client.is_spotify_authenticated()
# Get fallback client
itunes_client = deps.get_metadata_fallback_client()
logger.info(f"Starting {discovery_source} discovery for {len(tracks)} ListenBrainz tracks...")
# Store the discovery source in state
state['discovery_source'] = discovery_source
# Process each track for discovery
for i, track in enumerate(tracks):
try:
# Check for cancellation
if state.get('phase') != 'discovering':
logger.warning(f"ListenBrainz discovery cancelled (phase changed to '{state.get('phase')}')")
return
# Update progress
state['discovery_progress'] = int((i / len(tracks)) * 100)
# Get cleaned track data from ListenBrainz
cleaned_title = track['track_name']
cleaned_artist = track['artist_name']
album_name = track.get('album_name', '')
duration_ms = track.get('duration_ms', 0)
logger.info(f"Searching {discovery_source} for: '{cleaned_artist}' - '{cleaned_title}'")
# Check discovery cache first
cache_key = deps.get_discovery_cache_key(cleaned_title, cleaned_artist)
try:
cache_db = deps.get_database()
cached_match = cache_db.get_discovery_cache_match(cache_key[0], cache_key[1], discovery_source)
if cached_match and deps.validate_discovery_cache_artist(cleaned_artist, cached_match):
logger.debug(f"CACHE HIT [{i+1}/{len(tracks)}]: {cleaned_artist} - {cleaned_title}")
result = {
'index': i,
'lb_track': cleaned_title,
'lb_artist': cleaned_artist,
'status': 'Found',
'status_class': 'found',
'spotify_track': cached_match.get('name', ''),
'spotify_artist': deps.extract_artist_name(cached_match.get('artists', [''])[0]) if cached_match.get('artists') else '',
'spotify_album': cached_match.get('album', {}).get('name', '') if isinstance(cached_match.get('album'), dict) else cached_match.get('album', ''),
'duration': f"{int(duration_ms) // 60000}:{(int(duration_ms) % 60000) // 1000:02d}" if duration_ms else '0:00',
'discovery_source': discovery_source,
'matched_data': cached_match,
'spotify_data': cached_match
}
state['spotify_matches'] += 1
state['discovery_results'].append(result)
continue
except Exception as cache_err:
logger.error(f"Cache lookup error: {cache_err}")
# Try multiple search strategies using matching engine
matched_track = None
best_confidence = 0.0
best_raw_track = None
min_confidence = 0.9
source_duration = duration_ms or 0
# Strategy 1: Use matching_engine search queries
try:
temp_track = type('TempTrack', (), {
'name': cleaned_title,
'artists': [cleaned_artist],
'album': album_name if album_name else None
})()
search_queries = deps.matching_engine.generate_download_queries(temp_track)
logger.info(f"Generated {len(search_queries)} search queries for ListenBrainz track")
except Exception as e:
logger.error(f"Matching engine failed for ListenBrainz, falling back to basic query: {e}")
search_queries = [f"{cleaned_artist} {cleaned_title}", cleaned_title]
for query_idx, search_query in enumerate(search_queries):
try:
logger.debug(f"ListenBrainz query {query_idx + 1}/{len(search_queries)}: {search_query}")
search_results = None
if use_spotify and not deps.spotify_rate_limited():
search_results = deps.spotify_client.search_tracks(search_query, limit=10)
else:
search_results = itunes_client.search_tracks(search_query, limit=10)
if not search_results:
continue
# Score all results using the matching engine
match, confidence, match_idx = deps.discovery_score_candidates(
cleaned_title, cleaned_artist, source_duration, search_results
)
if match and confidence > best_confidence and confidence >= min_confidence:
best_confidence = confidence
matched_track = match
if use_spotify and match.id:
_cache = deps.get_metadata_cache()
best_raw_track = _cache.get_entity('spotify', 'track', match.id)
else:
best_raw_track = None
logger.info(f"New best ListenBrainz match: {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
if best_confidence >= 0.9:
logger.info(f"High confidence ListenBrainz match found ({best_confidence:.3f}), stopping search")
break
except Exception as e:
logger.debug(f"Error in ListenBrainz search for query '{search_query}': {e}")
continue
if matched_track:
logger.info(f"Strategy 1 ListenBrainz match: {matched_track.artists[0]} - {matched_track.name} (confidence: {best_confidence:.3f})")
# Strategy 2: Swapped search (if first failed) - score results properly
if not matched_track:
logger.info("ListenBrainz Strategy 2: Trying swapped search (artist/title reversed)")
if use_spotify:
query = f"artist:{cleaned_title} track:{cleaned_artist}"
fallback_results = deps.spotify_client.search_tracks(query, limit=5)
else:
query = f"{cleaned_title} {cleaned_artist}"
fallback_results = itunes_client.search_tracks(query, limit=5)
if fallback_results:
match, confidence, _ = deps.discovery_score_candidates(
cleaned_title, cleaned_artist, source_duration, fallback_results
)
if match and confidence >= min_confidence:
matched_track = match
best_confidence = confidence
logger.info(f"Strategy 2 ListenBrainz match (swapped): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
# Strategy 3: Album-based search (if still failed and we have album name) - score results properly
if not matched_track and album_name:
logger.info(f"ListenBrainz Strategy 3: Trying album-based search: '{cleaned_artist} {album_name} {cleaned_title}'")
if use_spotify:
query = f"artist:{cleaned_artist} album:{album_name} track:{cleaned_title}"
fallback_results = deps.spotify_client.search_tracks(query, limit=5)
else:
query = f"{cleaned_artist} {album_name} {cleaned_title}"
fallback_results = itunes_client.search_tracks(query, limit=5)
if fallback_results:
match, confidence, _ = deps.discovery_score_candidates(
cleaned_title, cleaned_artist, source_duration, fallback_results
)
if match and confidence >= min_confidence:
matched_track = match
best_confidence = confidence
logger.info(f"Strategy 3 ListenBrainz match (album): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
# Strategy 4: Extended search with higher limit (last resort)
if not matched_track:
logger.info("ListenBrainz Strategy 4: Extended search with limit=50")
query = f"{cleaned_artist} {cleaned_title}"
if use_spotify:
extended_results = deps.spotify_client.search_tracks(query, limit=50)
else:
extended_results = itunes_client.search_tracks(query, limit=50)
if extended_results:
match, confidence, _ = deps.discovery_score_candidates(
cleaned_title, cleaned_artist, source_duration, extended_results
)
if match and confidence >= min_confidence:
matched_track = match
best_confidence = confidence
logger.info(f"Strategy 4 ListenBrainz match (extended): {match.artists[0]} - {match.name} (confidence: {confidence:.3f})")
# Create result entry
result = {
'index': i,
'lb_track': cleaned_title,
'lb_artist': cleaned_artist,
'status': 'Found' if matched_track else 'Not Found',
'status_class': 'found' if matched_track else 'not-found',
'spotify_track': matched_track.name if matched_track else '',
'spotify_artist': deps.extract_artist_name(matched_track.artists[0]) if matched_track else '',
'spotify_album': matched_track.album if matched_track else '',
'duration': f"{int(duration_ms) // 60000}:{(int(duration_ms) % 60000) // 1000:02d}" if duration_ms else '0:00',
'discovery_source': discovery_source,
'confidence': best_confidence
}
if matched_track:
state['spotify_matches'] += 1
# Build album data based on provider
if use_spotify and best_raw_track:
album_data = best_raw_track.get('album', {})
else:
album_data = {
'name': matched_track.album,
'album_type': 'album',
'release_date': getattr(matched_track, 'release_date', '') or '',
'images': [{'url': matched_track.image_url}] if hasattr(matched_track, 'image_url') and matched_track.image_url else []
}
# Extract image URL for discovery pool display
_yt_album_images = album_data.get('images', [])
_yt_image_url = _yt_album_images[0].get('url', '') if _yt_album_images else (getattr(matched_track, 'image_url', '') or '')
result['matched_data'] = {
'id': matched_track.id,
'name': matched_track.name,
'artists': matched_track.artists,
'album': album_data,
'duration_ms': matched_track.duration_ms,
'image_url': _yt_image_url,
'source': discovery_source
}
result['spotify_data'] = result['matched_data']
# Save to discovery cache (only high-confidence matches)
if best_confidence >= 0.7:
try:
cache_db = deps.get_database()
cache_db.save_discovery_cache_match(
cache_key[0], cache_key[1], discovery_source, best_confidence,
result['matched_data'], cleaned_title, cleaned_artist
)
logger.info(f"CACHE SAVED: {cleaned_artist} - {cleaned_title} (confidence: {best_confidence:.3f})")
except Exception as cache_err:
logger.error(f"Cache save error: {cache_err}")
else:
# Auto Wing It fallback — build stub from raw source data
stub = deps.build_discovery_wing_it_stub(cleaned_title, cleaned_artist, duration_ms)
result['status'] = 'Wing It'
result['status_class'] = 'wing-it'
result['spotify_track'] = cleaned_title
result['spotify_artist'] = cleaned_artist
result['spotify_album'] = ''
result['matched_data'] = stub
result['spotify_data'] = stub
result['wing_it_fallback'] = True
state['wing_it_count'] = state.get('wing_it_count', 0) + 1
state['discovery_results'].append(result)
logger.info(f" {'' if matched_track else ''} Track {i+1}/{len(tracks)}: {result['status']}")
except Exception as e:
logger.error(f"Error processing track {i}: {e}")
result = {
'index': i,
'lb_track': track['track_name'],
'lb_artist': track['artist_name'],
'status': 'Error',
'status_class': 'error',
'spotify_track': '',
'spotify_artist': '',
'spotify_album': '',
'duration': '0:00'
}
state['discovery_results'].append(result)
# Complete discovery
state['phase'] = 'discovered'
state['status'] = 'complete'
state['discovery_progress'] = 100
playlist_name = playlist.get('name') or playlist.get('title') or 'Unknown Playlist'
source_label = discovery_source.upper()
deps.add_activity_item("", f"ListenBrainz Discovery Complete ({source_label})", f"'{playlist_name}' - {state['spotify_matches']}/{len(tracks)} tracks found", "Now")
logger.info(f"ListenBrainz discovery complete ({discovery_source}): {state['spotify_matches']}/{len(tracks)} tracks matched")
except Exception as e:
logger.error(f"Error in ListenBrainz discovery worker: {e}")
state['status'] = 'error'
state['phase'] = 'fresh'
finally:
deps.resume_enrichment_workers(_ew_state, 'ListenBrainz discovery')

340
core/discovery/listening_recommendations.py
View file

@ -1,340 +0,0 @@
"""Listening-driven recommendation core (#913).
PURE, side-effect-free ranking that turns "the artists you listen to most" plus
"who's similar to each" into:
1. a consensus-ranked list of artists you'd probably love but don't own, and
2. an aggregated candidate-track list for a generated playlist.
No DB / network / config here. The caller (the watchlist scanner) supplies the
seeds (top-played artists), the ``similar_artists`` rows per seed, and the
owned-artist set, then fetches top tracks for the winners. Keeping the decision
logic in one pure place makes it fully unit-testable without the live stack and
keeps the scan wiring thin — and additive, so it can't disturb existing flows.
Scoring rationale (the "best in class" bit): a recommended artist's score is
``Σ over the seeds that recommend it of (seed_weight × similarity)``. That single
sum rewards all three signals at once — **consensus** (an artist endorsed by many
of your seeds accumulates more terms), your **play weight** (heavier seeds push
harder), and **similarity strength** — instead of a flat "appears in N lists".
``seed_count`` is exposed separately for display ("because you like A, B, C") and
as the adventurousness dial's lever (``min_seed_count``).
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Sequence, Set
def _norm(name: object) -> str:
return str(name or "").strip().lower()
def _positive_float(value: object, default: float = 1.0) -> float:
try:
f = float(value) # type: ignore[arg-type]
except (TypeError, ValueError):
return default
return f if f > 0 else default
def _get(row: object, attr: str):
"""Read a field from a dataclass row or a dict row."""
if isinstance(row, dict):
return row.get(attr)
return getattr(row, attr, None)
def choose_mix_fetch_source(active_source: object, active_can_fetch: bool) -> str:
"""Pick which source to fetch the "Listening Mix" top tracks from.
The mix is a list of (artist, title) pairs acquired via Soulseek, so the fetch source need
NOT match the user's active metadata source. Use the active source when it can fetch top
tracks itself (Spotify/Deezer); otherwise fall back to Deezer, whose public ``artist/{id}/top``
needs no auth and is available to every user — so iTunes / Discogs / MusicBrainz users still
get a full mix without switching sources. Pure.
"""
if str(active_source or "").lower() in ("spotify", "deezer") and active_can_fetch:
return str(active_source).lower()
return "deezer"
def names_match(a: object, b: object) -> bool:
"""Strict artist-name equality after stripping case + non-alphanumerics.
Used to verify a name-search result before fetching that artist's top tracks, so the
"Listening Mix" can never pull the WRONG artist's songs (e.g. a same-name act). Exact
alphanumeric match: "Tyler, The Creator" == "Tyler The Creator", but "Drake" != "Drake Bell".
Pure.
"""
def _alnum(x: object) -> str:
return "".join(ch for ch in str(x or "").lower() if ch.isalnum())
na, nb = _alnum(a), _alnum(b)
return bool(na) and na == nb
def similarity_from_rank(rank: object, max_rank: int = 10) -> float:
"""Turn a stored ``similarity_rank`` (1 = most similar … 10 = least) into a 0–1 weight.
SoulSync stores each ``(seed → similar)`` edge with a 1–10 rank (``1`` is the closest
match). The ranker multiplies this into the score so a seed's *closest* matches count
for more than its long-tail ones. Linear decay over the documented range: rank 1 → 1.0,
rank 5 → 0.6, rank 10 → 0.1, with a 0.1 floor so a far match still contributes. A
missing/garbage rank falls back to 1.0 (treat as "no rank info, full weight"). Pure.
"""
try:
r = int(rank)
except (TypeError, ValueError):
return 1.0
floor = round(1.0 / max_rank, 4)
if r <= 1:
return 1.0
if r >= max_rank:
return floor
return round((max_rank - r + 1) / max_rank, 4)
def build_recency_weighted_seeds(
top_artists: Sequence[dict],
recent_play_counts: Optional[Dict[str, float]] = None,
*,
recency_factor: float = 1.5,
) -> List[dict]:
"""Blend lifetime + recent play counts into seed weights — "what you're into NOW".
``weight = lifetime_plays + recency_factor × recent_plays``. An artist you've played a
lot *recently* outranks one you played a lot years ago, so the recommendations track
your current taste instead of your all-time history. ``recency_factor`` is the dial
(0 = pure lifetime). Returns ``[{'name', 'weight'}]`` for :func:`rank_recommended_artists`.
Pure — the caller supplies both play-count maps from the listening history.
"""
recent = {_norm(k): _positive_float(v, 0.0) for k, v in (recent_play_counts or {}).items()}
out: List[dict] = []
for a in top_artists or ():
name = str(a.get("name") or "").strip()
if not name:
continue
lifetime = _positive_float(a.get("play_count", a.get("weight", 1.0)))
boost = recency_factor * recent.get(_norm(name), 0.0)
out.append({"name": name, "weight": lifetime + boost})
return out
def group_similars_by_seed(
seeds: Sequence[dict],
similar_rows: Sequence,
id_to_name: Dict[str, str],
*,
source_id_attr: str = "source_artist_id",
similar_name_attr: str = "similar_artist_name",
rank_attr: Optional[str] = None,
) -> Dict[str, List[dict]]:
"""Reshape flat ``similar_artists`` rows into ``{seed_name_lower: [{'name', 'score'?}]}``.
The stored rows key the similar artist by the SEED's source id (``source_artist_id``),
not its name, so :func:`rank_recommended_artists` can't consume them directly. This
resolves each row's source id to a name via ``id_to_name`` (``{source_artist_id:
artist_name}`` for the library, built by the caller) and keeps only rows that resolve
to one of the ``seeds``. Rows may be dataclass objects or dicts. Pure — no I/O.
``id_to_name`` MUST be keyed by whatever id the edges actually store — for SoulSync that
is the artist's SOURCE id (Spotify/iTunes/Deezer/MusicBrainz), NOT the internal row id.
When ``rank_attr`` is given, each row's rank is converted via :func:`similarity_from_rank`
and carried as ``score`` so closer matches weigh more; without it every similar comes out
score-less (the ranker then treats similarity as 1.0 — original behavior).
"""
seed_names = {_norm(s.get("name")) for s in seeds}
seed_names.discard("")
id_to_norm = {str(k): _norm(v) for k, v in (id_to_name or {}).items()}
out: Dict[str, List[dict]] = {}
for row in similar_rows or ():
seed_name = id_to_norm.get(str(_get(row, source_id_attr) or ""), "")
if not seed_name or seed_name not in seed_names:
continue
sim_name = str(_get(row, similar_name_attr) or "").strip()
if not sim_name:
continue
entry = {"name": sim_name}
if rank_attr is not None:
entry["score"] = similarity_from_rank(_get(row, rank_attr))
out.setdefault(seed_name, []).append(entry)
return out
@dataclass
class RecommendedArtist:
"""One artist recommended from your listening, with the why."""
name: str # display name (first-seen casing)
score: float # Σ seed_weight × similarity
seed_count: int # distinct seeds endorsing it (consensus)
seeds: List[str] = field(default_factory=list) # display names of those seeds
def rank_recommended_artists(
seeds: Sequence[dict],
similars_by_seed: Dict[str, Sequence[dict]],
owned_artist_names: Optional[Set[str]] = None,
*,
limit: int = 30,
min_seed_count: int = 1,
) -> List[RecommendedArtist]:
"""Rank artists similar to your most-played by consensus + play weight + similarity.
Args:
seeds: ``[{'name': str, 'weight': float}]`` — your top-played artists.
``weight`` (play count or any positive number) defaults to 1.0.
similars_by_seed: ``{seed_name_lower: [{'name': str, 'score': float}]}`` — the
similar-artist rows for each seed. ``score`` is optional (defaults 1.0).
owned_artist_names: lowercased names already in the library — excluded so the
result is artists you DON'T have. The seeds themselves are always excluded.
limit: max results.
min_seed_count: drop recommendations endorsed by fewer than N seeds — the
adventurousness dial's "Safer" end raises this for higher-confidence picks.
Returns up to ``limit`` :class:`RecommendedArtist`, highest score first.
"""
owned = {_norm(a) for a in (owned_artist_names or set())}
seed_norms = {_norm(s.get("name")) for s in seeds}
seed_norms.discard("")
exclude = owned | seed_norms
acc: Dict[str, dict] = {}
for seed in seeds:
s_name = _norm(seed.get("name"))
if not s_name:
continue
s_display = str(seed.get("name") or "").strip()
weight = _positive_float(seed.get("weight", 1.0))
for sim in similars_by_seed.get(s_name, ()) or ():
a_norm = _norm(sim.get("name"))
if not a_norm or a_norm in exclude:
continue
sim_score = _positive_float(sim.get("score", 1.0))
row = acc.setdefault(
a_norm, {"name": str(sim.get("name") or "").strip(), "score": 0.0, "seeds": {}}
)
row["score"] += weight * sim_score
row["seeds"].setdefault(s_name, s_display) # one seed counts once
out: List[RecommendedArtist] = []
floor = max(1, int(min_seed_count))
for row in acc.values():
seed_count = len(row["seeds"])
if seed_count < floor:
continue
out.append(RecommendedArtist(
name=row["name"],
score=round(row["score"], 6),
seed_count=seed_count,
seeds=list(row["seeds"].values()),
))
out.sort(key=lambda r: (-r.score, -r.seed_count, r.name.lower()))
return out[:limit]
def aggregate_candidate_tracks(
recommended_artists: Sequence[RecommendedArtist],
top_tracks_by_artist: Dict[str, Sequence[dict]],
owned_track_keys: Optional[Set] = None,
*,
per_artist: int = 3,
limit: int = 50,
exclude_owned: bool = True,
) -> List[dict]:
"""Build the candidate track list for the generated playlist.
Takes the top ``per_artist`` tracks from each recommended artist **in artist-rank
order**, dedups by ``(artist, title)``, optionally drops owned tracks (the
"discovery" flavor) and caps at ``limit``. Each returned track dict is the source
track plus ``_seed_artist`` (which recommended artist it came from).
Args:
recommended_artists: ranked output of :func:`rank_recommended_artists`.
top_tracks_by_artist: ``{artist_name_lower: [track_dict, ...]}`` — fetched by
the caller (Last.fm / source top tracks), NOT limited to a curated pool.
owned_track_keys: set of ``(artist_lower, title_lower)`` already in the library.
exclude_owned: drop tracks in ``owned_track_keys`` (discovery flavor). Set False
for a "replay" playlist of tracks you already own.
"""
owned = owned_track_keys or set()
seen: Set = set()
out: List[dict] = []
for art in recommended_artists:
tracks = top_tracks_by_artist.get(_norm(art.name), ()) or ()
taken = 0
for t in tracks:
if taken >= per_artist:
break
title = str(t.get("name") or t.get("title") or "").strip()
if not title:
continue
key = (_norm(art.name), _norm(title))
if key in seen:
continue
if exclude_owned and key in owned:
continue
seen.add(key)
out.append({**t, "_seed_artist": art.name})
taken += 1
if len(out) >= limit:
break
return out[:limit]
def to_mix_track(track: object, source: str) -> Optional[dict]:
"""Shape one source "top tracks" API dict into the flat dict the Discover compact
playlist row renders + syncs (the "Listening Mix" #913 playlist).
Spotify's ``artist_top_tracks`` and Deezer's ``get_artist_top_tracks`` both return the
same Spotify-shape object (``id, name, artists[], album{name,images[]}, duration_ms``).
This flattens that into the renderer's field names (``track_name/artist_name/album_name/
album_cover_url/duration_ms``), keeps the original under ``track_data_json`` for sync, and
sets the source-specific id field. Returns None for anything without a usable id/title so
the caller can filter. A ``name`` key is kept so :func:`aggregate_candidate_tracks` can
dedup by title. Pure — no I/O.
"""
if not isinstance(track, dict):
return None
tid = track.get("id")
name = str(track.get("name") or "").strip()
if not tid or not name:
return None
artists = track.get("artists") or []
artist_name = ""
if artists and isinstance(artists[0], dict):
artist_name = str(artists[0].get("name") or "").strip()
album = track.get("album") if isinstance(track.get("album"), dict) else {}
album_name = str(album.get("name") or "").strip()
images = album.get("images") or []
cover = images[0].get("url") if images and isinstance(images[0], dict) else None
out = {
"track_id": str(tid),
"name": name, # for aggregate_candidate_tracks dedup
"track_name": name, # for the renderer
"artist_name": artist_name,
"album_name": album_name,
"album_cover_url": cover,
"duration_ms": track.get("duration_ms") or 0,
"track_data_json": track, # full payload for sync/download
"source": source,
}
id_field = {"spotify": "spotify_track_id", "deezer": "deezer_track_id",
"itunes": "itunes_track_id"}.get(source)
if id_field:
out[id_field] = str(tid)
return out
__all__ = [
"RecommendedArtist",
"choose_mix_fetch_source",
"names_match",
"similarity_from_rank",
"build_recency_weighted_seeds",
"to_mix_track",
"group_similars_by_seed",
"rank_recommended_artists",
"aggregate_candidate_tracks",
]

120
core/discovery/manual_match.py
View file

@ -1,120 +0,0 @@
"""Helpers for Fix-popup manual match persistence.
When the user manually fixes a mirrored-playlist discovery via the Fix
popup, two questions land at the web_server route layer that are easier
to test in isolation:
1. *Which metadata source did the manual match come from?* — the popup
cascade queries the user's primary source first, then Spotify /
Deezer / iTunes / MusicBrainz as fallbacks; each search endpoint
stamps `source` on its rows but the MBID-paste lookup uses a lean
flat shape that doesn't carry it. `derive_manual_match_provider`
collapses the fallback chain into a single string.
2. *Should the discovery layer re-run for this track when the current
active provider differs from the cached one?* — re-running silently
overwrites the user's deliberate pick with whatever the auto-search
ranks first, so manual matches are exempt regardless of provider
drift. `is_drifted_for_redo` encapsulates the decision.
3. *Should the Playlist Pipeline pre-scan (re)discover this track at all?*
— `should_rediscover` encapsulates that gate, with the manual match
checked FIRST so a leftover Wing It flag can't override the user's pick.
"""
from __future__ import annotations
from typing import Any, Dict, Optional
def derive_manual_match_provider(
payload_track: Dict[str, Any],
active_provider: Optional[str],
) -> str:
"""Return the provider string to stamp on a manually-fixed match.
Resolution order:
1. ``payload_track['source']`` — every *_search_tracks endpoint
sets this; the MBID-paste path doesn't.
2. ``active_provider`` — what the user has configured as their
primary discovery source.
3. ``'spotify'`` — last-ditch default matching the historic
hardcode (so behaviour is identical when both upstream
signals are absent).
"""
if not isinstance(payload_track, dict):
payload_track = {}
source = payload_track.get('source')
if source:
return source
if active_provider:
return active_provider
return 'spotify'
def is_drifted_for_redo(
extra_data: Optional[Dict[str, Any]],
active_provider: Optional[str],
) -> bool:
"""Return True when a cached discovery entry should be treated as
stale because the user's active provider has changed since it was
cached AND the entry isn't a manual match.
Manual matches are *always* considered fresh: re-running discovery
against the current source would overwrite the user's deliberate
pick with whatever auto-search ranks first. The first Playlist
Pipeline run after a manual fix used to clobber it for exactly
this reason — the check lives here now so it's pinned by tests.
"""
if not isinstance(extra_data, dict):
return False
if extra_data.get('manual_match'):
return False
cached_provider = extra_data.get('provider', 'spotify')
return cached_provider != active_provider
def should_rediscover(extra_data: Optional[Dict[str, Any]]) -> bool:
"""Return True when a mirrored track needs (re)discovery, False to skip it.
This is the gate the Playlist Pipeline pre-scan runs over every mirrored
track before discovering. The **ordering is the fix**: a manual match is
authoritative and is checked FIRST.
``extra_data`` is *merged* on save (see ``update_mirrored_track_extra_data``),
so a track that was a Wing It stub and is then manually fixed still carries
``wing_it_fallback: True`` alongside the new ``manual_match: True``. The old
pre-scan tested ``wing_it_fallback`` before ``manual_match``, so the stale
flag won and the pipeline re-discovered the track — silently reverting the
user's pick to Wing It. Checking ``manual_match`` first makes the fix stick.
Decision order:
* manual_match -> skip (authoritative; never re-discover)
* wing_it_fallback -> redo (stub — keep trying for a real match)
* discovered + complete -> skip (full metadata already stored)
* discovered + incomplete -> redo (backfill track_number / album fields)
* unmatched_by_user -> skip (user deliberately removed the match)
* never discovered -> redo (first-time discovery)
"""
extra = extra_data if isinstance(extra_data, dict) else {}
if extra.get('discovered'):
if extra.get('manual_match'):
return False
if extra.get('wing_it_fallback'):
return True
# Otherwise re-discover only when the stored match is missing the
# enriched fields (track_number + release_date/album.id) that older
# discoveries dropped via the Track dataclass.
matched = extra.get('matched_data')
matched = matched if isinstance(matched, dict) else {}
album = matched.get('album')
album = album if isinstance(album, dict) else {}
has_track_num = matched.get('track_number')
has_release = album.get('release_date')
has_album_id = album.get('id')
return not (has_track_num and (has_release or has_album_id))
if extra.get('unmatched_by_user'):
return False
return True

Some files were not shown because too many files have changed in this diff Show more