danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Compare commits

base: danvics:main
Branches Tags
danvics:main
danvics:dev
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65
..
compare: danvics:1.2
Branches Tags
danvics:dev
danvics:main
danvics:video
danvics:fix/disable-beatport-features
danvics:johnbaumb-discover-redesign
danvics:2.8.2
danvics:2.8.1
danvics:2.8.0
danvics:2.7.9
danvics:2.7.8
danvics:2.7.7
danvics:2.7.6
danvics:2.7.5
danvics:2.7.4
danvics:2.7.3
danvics:2.7.2
danvics:2.7.1
danvics:2.7.0
danvics:2.6.9
danvics:2.6.8
danvics:2.6.7
danvics:2.6.6
danvics:2.6.5
danvics:2.6.4
danvics:2.6.3
danvics:2.6.2
danvics:2.6.1
danvics:2.6.0
danvics:2.5.9
danvics:2.5.7
danvics:2.5.6
danvics:2.5.5
danvics:2.5.4
danvics:2.5.3
danvics:2.5.2
danvics:2.5.1
danvics:2.5.0
danvics:2.4.2
danvics:2.4.1
danvics:2.4.0
danvics:2.3
danvics:2.2
danvics:2.1
danvics:2.0
danvics:1.9
danvics:1.8
danvics:1.7
danvics:1.6
danvics:1.5
danvics:1.4
danvics:1.3
danvics:1.2
danvics:1.1
danvics:1.0
danvics:v0.65

No commits in common. "main" and "1.2" have entirely different histories.
main ... 1.2

1306 changed files with 95176 additions and 517256 deletions
Show stats Expand all files Collapse all files

16
.claude/settings.local.json Normal file
View file

@ -0,0 +1,16 @@
{
"permissions": {
"allow": [
"Bash(mkdir:*)",
"Bash(rm:*)",
"Bash(rg:*)",
"Bash(grep:*)",
"WebFetch(domain:python-plexapi.readthedocs.io)",
"Bash(git restore:*)",
"Bash(python3:*)",
"Bash(awk:*)",
"Bash(cat:*)"
],
"deny": []
}
}

32
.dockerignore
View file

@ -1,8 +1,5 @@
# Docker ignore file for SoulSync WebUI
# Hidden folders and files
.*
# Git
.git
.gitignore
@ -25,13 +22,6 @@ __pycache__/
dist/
build/
# Frontend build artifacts and local dependency caches
webui/.tanstack/
webui/.vite/
webui/node_modules/
webui/test-results/
webui/static/dist/
# Virtual environments
venv/
env/
@ -58,20 +48,6 @@ Incomplete/*
artist_bubble_snapshots.json
.spotify_cache
# Auto-downloaded ffmpeg binaries — the YouTube client downloads these
# into tools/ when system ffmpeg isn't on PATH. The Dockerfile installs
# system ffmpeg via apt, so the container never needs the bundled
# binaries. If a CI run leaves them in the workspace before the docker
# build (e.g. because a test imported web_server which initialized the
# YouTube client), they'd otherwise get baked into the image — adding
# ~388 MB and getting duplicated again by the chown layer.
tools/ffmpeg
tools/ffprobe
tools/ffmpeg.exe
tools/ffprobe.exe
tools/*.zip
tools/*.tar.xz
# Documentation
*.md
README.md
@ -80,12 +56,10 @@ multi-server-database-plan.md
server-source.md
plans.md
# GUI-specific files (removed by PR #311)
# GUI-specific files
main.py
ui/
# Dev-specific files
requirements-dev.txt
requirements.txt
# OS generated files
.DS_Store
@ -94,4 +68,4 @@ requirements-dev.txt
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
Thumbs.db

36
.github/workflows/build-and-test.yml vendored
View file

@ -1,36 +0,0 @@
name: Compile the app and run tests
on:
push:
branches-ignore:
- main
- dev
jobs:
sanity-check:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.11"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: python -m pip install --upgrade pip && python -m pip install -r requirements-dev.txt
- name: Lint with ruff
run: python -m ruff check --output-format=github .
- name: Build app
run: python -m compileall api core database services scripts web_server.py wsgi.py beatport_unified_scraper.py
- name: Run tests
env:
PYTHONPATH: ${{ github.workspace }}
run: python -m pytest

25
.github/workflows/cleanup-dev-images.yml vendored
View file

@ -1,25 +0,0 @@
name: Cleanup old dev images
on:
schedule:
# Weekly on Sunday at 6 AM UTC
- cron: '0 6 * * 0'
workflow_dispatch:
jobs:
cleanup:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- name: Delete old dev image tags
uses: actions/delete-package-versions@v5
with:
package-name: soulsync
package-type: container
min-versions-to-keep: 10
delete-only-pre-release-versions: false
# Only prune tags matching the dev nightly pattern (keep rolling + version tags)
ignore-versions: '^(dev|nightly|latest|\\d+\\.\\d+)$'

104
.github/workflows/dev-nightly.yml vendored
View file

@ -1,104 +0,0 @@
name: Dev Nightly Build
on:
# Nightly at 4:00 AM UTC — only if dev branch has new commits
schedule:
- cron: '0 4 * * *'
# Also build on every push to dev for immediate feedback
push:
branches:
- dev
# Manual trigger for testing
workflow_dispatch:
jobs:
nightly:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
# Skip scheduled runs if dev branch has no new commits in the last 24h
# (pushes and manual triggers always run)
permissions:
contents: read
packages: write
steps:
- name: Checkout dev branch
uses: actions/checkout@v6
with:
ref: dev
- name: Set lowercase owner
run: echo "OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> "$GITHUB_ENV"
- name: Check for recent commits (scheduled only)
if: github.event_name == 'schedule'
id: recent
run: |
LAST_COMMIT=$(git log -1 --format=%ct)
NOW=$(date +%s)
DIFF=$(( NOW - LAST_COMMIT ))
if [ "$DIFF" -gt 86400 ]; then
echo "No commits in the last 24h, skipping build"
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- name: Set up Python
if: steps.recent.outputs.skip != 'true'
uses: actions/setup-python@v6
with:
python-version: "3.11"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies and run tests
if: steps.recent.outputs.skip != 'true'
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txt
python -m ruff check --output-format=github .
python -m compileall api core database services scripts web_server.py wsgi.py beatport_unified_scraper.py
python -m pytest
- name: Set up QEMU
if: steps.recent.outputs.skip != 'true'
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
if: steps.recent.outputs.skip != 'true'
uses: docker/setup-buildx-action@v4
- name: Log in to GHCR
if: steps.recent.outputs.skip != 'true'
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ env.OWNER }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Generate build tag
if: steps.recent.outputs.skip != 'true'
id: tag
run: |
echo "date=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"
echo "short_sha=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
- name: Build and push to GHCR
if: steps.recent.outputs.skip != 'true'
uses: docker/build-push-action@v7
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
pull: true
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
COMMIT_SHA=${{ github.sha }}
tags: |
ghcr.io/${{ env.OWNER }}/soulsync:dev
ghcr.io/${{ env.OWNER }}/soulsync:dev-${{ steps.tag.outputs.date }}-${{ steps.tag.outputs.short_sha }}
${{ github.event_name == 'schedule' && format('ghcr.io/{0}/soulsync:nightly', env.OWNER) || '' }}

76
.github/workflows/docker-publish.yml vendored
View file

@ -1,76 +0,0 @@
name: Build and Push Docker Image
on:
# Auto-build :latest on every push to main
push:
branches:
- main
# Manual trigger for tagged releases (e.g. 2.33)
workflow_dispatch:
inputs:
version_tag:
description: 'Version tag (e.g. 2.8.2)'
required: true
default: '2.8.2'
jobs:
build-and-push:
if: github.repository == 'Nezreka/SoulSync'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set lowercase owner
run: echo "OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> "$GITHUB_ENV"
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Log in to Docker Hub
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ env.OWNER }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v7
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
pull: true
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
COMMIT_SHA=${{ github.sha }}
tags: |
boulderbadgedad/soulsync:latest
ghcr.io/${{ env.OWNER }}/soulsync:latest
${{ inputs.version_tag && format('boulderbadgedad/soulsync:{0}', inputs.version_tag) || '' }}
${{ inputs.version_tag && format('ghcr.io/{0}/soulsync:{1}', env.OWNER, inputs.version_tag) || '' }}
- name: Announce release to Discord
if: success() && inputs.version_tag
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_ANNOUNCEMENTS_WEBHOOK }}
run: |
if [ -z "$DISCORD_WEBHOOK" ]; then echo "No webhook configured, skipping"; exit 0; fi
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"SoulSync v${{ inputs.version_tag }} Released\", \"description\": \"A new version of SoulSync is available! Pull the latest Docker image to update.\n\n\`\`\`\ndocker pull boulderbadgedad/soulsync:${{ inputs.version_tag }}\n\`\`\`\", \"color\": 5025616, \"footer\": {\"text\": \"SoulSync Auto-Release\"}, \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}]}" \
"$DISCORD_WEBHOOK"

24
.gitignore vendored
View file

@ -6,27 +6,3 @@ __pycache__/
**/__pycache__/
*.pyc
*.pyo
# Encryption key (generated per-instance, lives next to database)
.encryption_key
# User-specific files (auto-created by the app if missing)
config/config.json
config/youtube_cookies.txt
# All app databases are live user data — never commit (music_library, video_library, …)
database/*.db
database/*.db-shm
database/*.db-wal
database/*.db.backup_*
database/api_call_history.json
storage/image_cache/
logs/*.log
logs/*.log.*
# Auto-downloaded binaries
bin/
# Development compose/config files
*.dev.yml
# Any hidden folders
**/.*/

74
DISCOVER_BEST_IN_CLASS_PLAN.md
View file

@ -1,74 +0,0 @@
# discover page — best in class plan (#913 + full generator audit)
morning notes. did the work overnight. tl;dr at top, details below, all of it `break nothing` + tested.
## what i shipped tonight (done, tested, safe)
### 1. listening recommendations (#913) — went from BROKEN to best-in-class
the feature was silently producing **zero** recs on real data. dug in and found three stacked bugs in the generation:
- **wrong id key (the killer).** `similar_artists.source_artist_id` is a *source* id (spotify/itunes/deezer), but the scanner built its id→name map from `artists.id` (the internal row id). so every edge resolved to nothing → 0 recs. proved it on your live db: internal-id join = 0 rows, spotify-id join = 71,636 rows.
- **consensus could never fire.** it fed the ranker `get_top_similar_artists`, which does `GROUP BY similar_artist_name` + `MAX(source_artist_id)` — collapsing every similar artist down to a *single* seed. the whole point of the ranker is "artist X is similar to 3 of your seeds = strong signal," and that signal was being flattened away before it ever reached the ranker.
- **similarity strength thrown away.** each edge stores a 1-10 closeness rank; it was ignored (everything weighted equally).
the fix (all in the pure, tested core + thin scan wiring):
- build id→name from the **source-id columns**, query the **raw per-seed edges** (consensus preserved), and thread **similarity_rank** into the score so a seed's closest matches count for more.
- **recency-weighted seeds**: `weight = lifetime_plays + 1.5 × recent_30d_plays`. picks now track what you're into *now*, not just all-time totals.
result on your actual library (simulated through the real code path): **40 recommendations, 13 with multi-seed consensus, all 40 with cached art.** top picks: Arcangel (Bad Bunny + Ozuna + J Balvin), Melanie Martinez (Ariana + Billie), Maluma, De La Ghetto — all coherent, all explainable.
### 2. its own row on the discover page
new row **"Based On Your Listening"** — play-weighted, consensus-ranked artist cards with a **"Because you listen to X, Y"** line. sits right above the library-driven "Recommended For You" row. purely additive: new endpoint `/api/discover/listening-recommendations`, new loader, hides itself when empty.
**you need to run one watchlist scan** for the row to populate (the data regenerates during the scan — i did NOT touch your live db). before that scan the row just stays hidden; after it, it fills in.
> note: this is deliberately different from the existing "Recommended For You" row. that one is driven by your *whole library / watchlist*. this one is driven by your *actual listening intensity* — the ~30 artists you really play, not the thousands you happen to own.
### 3. Fresh Tape "only 5-10 tracks" — fixed
root cause: `get_discovery_recent_albums` orders `release_date DESC`, so announced-but-unreleased albums sort to the *top* and ate the 50-album budget. the scanner skipped them *after* the budget was already spent → only a handful of released albums left → 5-10 tracks. fixed by fetching a generous budget (300) **and** excluding next-year albums at the query, so released albums fill the budget. the precise same-year `is_future_release` skip stays as a second guard. downstream caps (6/artist, top 75, take 50) unchanged.
**tests:** 25 pure-core cases (consensus/similarity/recency) + 2 Fresh Tape regression tests, all green. full discovery suite (255) green. nothing else touched.
---
## best-in-class roadmap for listening recs (next phases — your call)
these are the levers to take it further. ordered by value-to-risk. none are required; tonight's work stands on its own.
| phase | what | value | risk | notes |
|---|---|---|---|---|
| **3** | **playable track row** ✅ DONE | high | low-med | shipped: "🎧 Your Listening Mix" row — a track playlist (play/queue/download/sync) right under the artist row. stored as full render-ready dicts (not pool-hydrated, so it can't shrink on pool rotation like Fresh Tape does). |
| **4** | **direct top-tracks fetch** ✅ DONE | high | med | shipped: scan fetches each recommended artist's top tracks (Spotify/Deezer), resolving the artist id by name-search when the similar-artist row lacks one — guarded by a strict name-match so it never pulls the wrong artist. bounded (top 20 recs), per-call guarded, fail-soft to the pool. iTunes has no top-tracks API → pool-only there. needs a live scan to populate. |
| **5** | **genre-affinity boost** | med | low | we already compute your genre breakdown. boost recs whose genres match your top genres → tighter taste alignment. pure scoring add. |
| **6** | **adventurousness dial** | med | low | the ranker already supports `min_seed_count` (consensus floor). expose it as a "Safe ↔ Adventurous" slider on the row. |
| **7** | **diversity pass** | low-med | low | avoid 40 recs all orbiting your single heaviest seed — cap picks-per-seed so the row spans your taste. |
the core is built to absorb all of these without re-plumbing — `similarity_from_rank`, `build_recency_weighted_seeds`, and the scoring formula are all pure + tested.
---
## full discover-page generator audit (every soulsync-built row, excluding last.fm + listenbrainz)
how each one is generated today, and whether it can be elevated. "clear win" = safe + additive. "product call" = needs your decision (changes the row's character).
### curated (built during the scan, then hydrated)
- **Fresh Tape / Release Radar** — new releases from watchlist+similar artists. **FIXED tonight** (see above). one more *clear win* available: hydration silently drops any curated id no longer in the discovery pool — could fall back to the stored `track_data_json` blob so the row can't shrink at read time.
- **The Archives / Discovery Weekly** — strong already. nice 3-tier popularity split + serendipity scoring (boost never-played artists, penalize overplayed). same hydration-drop caveat as Fresh Tape; same cheap fallback fix.
- **Seasonal Mix** — cleanest of the bunch. hydrates from a dedicated `seasonal_tracks` table (carries its own data), so it doesn't suffer the pool-drop problem. no bug.
### discovery-pool generators (live queries)
- **Popular Picks** — ranks by popularity DESC. solid. only nit: on iTunes (no popularity scale) it silently degrades to random — indistinguishable from Shuffle there. UI-label thing at most.
- **Hidden Gems** — *clear win*. currently `ORDER BY RANDOM()` over low-popularity tracks — so it's "random obscure," not "*best* obscure." a light ranking (popularity just under the threshold, or genre-affinity to you) would make it feel curated instead of arbitrary. (a deeper *product call*: add personalization like Archives has — bigger lift, changes its "pure underground" character.)
- **Genre Playlists** — good. pushes the genre match into SQL. `RANDOM()` ordering is fine for a browse; a popularity/affinity tiebreak (*clear win*) would make thin genres feel less arbitrary.
- **Discovery Shuffle** — random by design, correct. only possible add: exclude tracks already shown in other rows this refresh (needs a cross-section seen-set — medium plumbing).
- **Time Machine (by decade)** — *clear win, low risk*: decades are hardcoded, so a modern-only library shows 7 decade tabs, 5 empty. filter the tabs to decades that actually have pool data.
- **Daily Mix** — the weakest row. the "50% your library" half permanently returns nothing (library tracks have no source ids to play), so each Daily Mix is really just a relabeled Genre Playlist. real fix = backfill source ids into library rows (*schema-level, higher risk*) — worth a dedicated pass, not a quick tweak. also silently falls back to "top artists as pseudo-genres" when genre data is missing → "Daily Mix 1" becomes mislabeled artist-radio. gate/label that (*clear win*).
### cross-cutting
- **hydration fragility** (Fresh Tape + Archives): both depend on curated ids still living in the pool at read time; misses are dropped silently. Seasonal already solved this with a dedicated table. giving the two spotify-style rows the same data-blob fallback is the single most robust cross-cutting fix. low risk, clear win.
- **RANDOM-ordering pattern** (Hidden Gems, Shuffle, Genre, Decade): intentional for variety, but leaves quality signal on the table for the non-shuffle rows. adding a light ranking pass to Hidden Gems + Genre is the biggest "best-in-class" lever after tonight's work.
want me to take any of these? the Hidden Gems ranking + Time Machine empty-decade filter + the Fresh Tape/Archives hydration fallback are all safe, additive, same-shape-as-tonight wins i can knock out next.

64
DOCKER-OAUTH-FIX.md Normal file
View file

@ -0,0 +1,64 @@
# 🔐 Docker OAuth Authentication Fix
## Problem: "Insecure redirect URI" Error
When accessing SoulSync from a **different device** than the Docker host, you may encounter:
- `INVALID_CLIENT: Insecure redirect URI`
- `Spotify authentication failed: error: invalid_client`
**Why this happens:** Spotify requires HTTPS for OAuth callbacks when not using localhost.
## ✅ Simple Solution: SSH Port Forwarding
### Step 1: Set up SSH tunnel from your device to Docker host
**On the device you're browsing from** (laptop/phone/etc):
```bash
# Replace 'user' and 'docker-host-ip' with your actual values
ssh -L 8888:localhost:8888 -L 8889:localhost:8889 user@docker-host-ip
# Example:
ssh -L 8888:localhost:8888 -L 8889:localhost:8889 john@192.168.1.100
```
**Keep this SSH connection open** while using SoulSync.
### Step 2: Configure OAuth redirect URIs
**In your Spotify Developer App:**
- Set redirect URI to: `http://127.0.0.1:8888/callback`
**In your Tidal Developer App:**
- Set redirect URI to: `http://127.0.0.1:8889/tidal/callback`
**In SoulSync Settings:**
- Set Spotify redirect URI to: `http://127.0.0.1:8888/callback`
- Set Tidal redirect URI to: `http://127.0.0.1:8889/tidal/callback`
### Step 3: Use SoulSync normally
- Access SoulSync: `http://docker-host-ip:8008` (normal HTTP)
- OAuth callbacks will tunnel through SSH to localhost
- Authentication will work without HTTPS requirements
## 🖥️ Alternative: Direct Access from Docker Host
If you can access SoulSync directly from the Docker host machine:
- Use: `http://127.0.0.1:8008`
- Set OAuth redirect URIs to localhost (as above)
- No SSH tunnel needed
## 🔧 For Advanced Users: Reverse Proxy
Set up nginx/traefik with proper SSL certificates for true HTTPS support. See community guides for Docker reverse proxy setups.
## 📝 Summary
The core issue is that **Spotify requires HTTPS for non-localhost** OAuth redirects. The SSH tunnel makes remote devices appear as localhost to bypass this requirement.
**Key points:**
- ✅ Always use `127.0.0.1` in OAuth redirect URIs
- ✅ Use SSH tunnel when accessing from different device
- ✅ Keep tunnel open during authentication
- ✅ Works with existing Docker setup - no changes needed

0
Support/DOCKER-TRANSFER-GUIDE.md → DOCKER-TRANSFER-GUIDE.md
View file

0
Support/DOCKER.md → DOCKER.md
View file

0
Support/DOCKER_PERMISSIONS.md → DOCKER_PERMISSIONS.md
View file

120
Dockerfile
View file

@ -1,114 +1,35 @@
# SoulSync WebUI Dockerfile
# Multi-architecture support for AMD64 and ARM64
FROM node:24-slim AS webui-builder
WORKDIR /app/webui
COPY webui/package.json webui/package-lock.json ./
RUN npm ci
COPY webui/ ./
RUN npm run build
# Stage 1: Builder — install Python dependencies with compilation tools
FROM python:3.11-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
libc6-dev \
libffi-dev \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*
# Create virtualenv and install dependencies
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# yt-dlp must track YouTube faster than its stable channel ships — stable can
# lag months behind a breaking YouTube change while extraction is broken
# ("Requested format is not available"). Build images with the NIGHTLY channel.
# COMMIT_SHA is referenced in the RUN so CI's layer cache (cache-from: gha)
# busts on every new commit — otherwise this layer could pin a stale "nightly"
# for months, silently defeating its purpose.
ARG COMMIT_SHA=""
RUN echo "yt-dlp nightly for build ${COMMIT_SHA}" && \
pip install --no-cache-dir -U --pre "yt-dlp[default]"
# Stage 2: Runtime — only runtime dependencies, no build tools
FROM python:3.11-slim
# Build-time commit SHA for update detection
ARG COMMIT_SHA=""
ENV SOULSYNC_COMMIT_SHA=${COMMIT_SHA}
# Copy pre-built virtualenv from builder
COPY --from=builder /opt/venv /opt/venv
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="/opt/venv/bin:$PATH"
# Set working directory
WORKDIR /app
# Install runtime-only system dependencies (no gcc/build tools).
# unzip is needed by the Deno installer below.
RUN apt-get update && apt-get install -y --no-install-recommends \
# Install system dependencies
RUN apt-get update && apt-get install -y \
gcc \
libc6-dev \
libffi-dev \
libssl-dev \
curl \
gosu \
ffmpeg \
libchromaprint-tools \
unzip \
&& rm -rf /var/lib/apt/lists/*
# Deno — JavaScript runtime for yt-dlp. YouTube gates its downloadable formats
# behind JS challenges (nsig); without a JS runtime, yt-dlp's extraction is
# deprecated and streams / music-video downloads fail with "Requested format
# is not available". Deno is yt-dlp's default-enabled runtime; the official
# installer auto-detects amd64/arm64. `deno --version` fails the build early
# if the install ever breaks.
RUN curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh && \
deno --version
# Create non-root user for security
RUN useradd --create-home --shell /bin/bash --uid 1000 soulsync
# Copy application code with ownership baked in.
# Using `COPY --chown` instead of `COPY` + `chown -R /app` avoids an
# extra image layer that duplicates the entire /app tree just to flip
# ownership bits — Docker layers are immutable, so chown -R rewrites
# every file into a new layer. On a clean repo that's small; if any
# bulky workspace file slips in (e.g. auto-downloaded ffmpeg binaries
# in tools/), it gets counted twice in the image. Cin caught this on
# 2026-05-08 — see the .dockerignore comment for the same incident.
COPY --chown=soulsync:soulsync . .
COPY --chown=soulsync:soulsync --from=webui-builder /app/webui/static/dist /app/webui/static/dist
# Copy requirements and install Python dependencies
COPY requirements-webui.txt .
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements-webui.txt
# Create runtime mount-point directories the app expects to exist.
# NOTE: /app/data is for database FILES, /app/database is the Python package
# NOTE: /app/Staging is required even though most users bind-mount it —
# the entrypoint mkdir runs early and is gated by `set -e`, so a missing
# pre-baked directory would crash the container into a restart loop on
# rootless Docker/Podman where in-container "root" can't write to /app.
# Pre-baking the dir here makes the entrypoint mkdir a guaranteed no-op.
# NOTE: /app/Stream is the transient single-file streaming cache used by
# the basic-search "Play" flow (cleared per use, never persistent). It's
# created lazily by `core/streaming/prepare.py` via `os.makedirs`, which
# fails silently on rootless Docker where the soulsync UID can't write
# to /app — playback then errors out with no obvious cause. Pre-baking
# at build time (when the layer is owned by root) avoids that path.
# NOTE: /app/storage is the PRIVATE album-bundle staging area for the
# torrent / usenet whole-release flow (download_source.album_bundle_staging_path
# defaults to 'storage/album_bundle_staging'). Like /app/Stream it's created
# lazily at runtime via mkdir(parents=True); without pre-baking it owned by
# soulsync, the album-bundle copy fails with "[Errno 13] Permission denied:
# 'storage'" because /app itself is root-owned and the soulsync UID can't
# create a top-level dir there.
RUN mkdir -p /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts && \
chown soulsync:soulsync /app/config /app/data /app/logs /app/downloads /app/Transfer /app/Staging /app/Stream /app/storage /app/MusicVideos /app/scripts
# Copy application code
COPY . .
# Create necessary directories with proper permissions
RUN mkdir -p /app/config /app/database /app/logs /app/downloads /app/Transfer && \
chown -R soulsync:soulsync /app
# Create defaults directory and copy template files
# These will be used by entrypoint.sh to initialize empty volumes
@ -118,8 +39,7 @@ RUN mkdir -p /defaults && \
chmod 644 /defaults/config.json /defaults/settings.py
# Create volume mount points
# NOTE: Changed /app/database to /app/data to avoid overwriting Python package
VOLUME ["/app/config", "/app/data", "/app/logs", "/app/downloads", "/app/Transfer", "/app/MusicVideos", "/app/scripts"]
VOLUME ["/app/config", "/app/database", "/app/logs", "/app/downloads", "/app/Transfer"]
# Copy and set up entrypoint script
COPY entrypoint.sh /entrypoint.sh
@ -137,12 +57,12 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
# Set environment variables
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
ENV DATABASE_PATH=/app/data/music_library.db
ENV FLASK_APP=web_server.py
ENV FLASK_ENV=production
ENV PUID=1000
ENV PGID=1000
ENV UMASK=022
# Set entrypoint and default command
ENTRYPOINT ["/entrypoint.sh"]
CMD ["gunicorn", "-c", "gunicorn.conf.py", "wsgi:application"]
CMD ["python", "web_server.py"]

17
Support/README-Docker.md → README-Docker.md
View file

@ -1,15 +1,5 @@
# SoulSync WebUI - Docker Deployment Guide
## Release Channels
SoulSync publishes two Docker image tracks:
- **Stable — `boulderbadgedad/soulsync:latest`** (Docker Hub). Hand-promoted from the `dev` branch when a batch of changes is ready. Default in `docker-compose.yml`.
- **Nightly — `ghcr.io/nezreka/soulsync:dev`** (GHCR). Rebuilt every night and on every push to `dev`. Faster access to new features at the cost of occasional instability.
- **Version-tagged — `:2.3`, `:2.4`, etc.** on both registries for pinning to a specific release.
To switch a running install to the nightly channel, edit the `image:` line in `docker-compose.yml` to `ghcr.io/nezreka/soulsync:dev` and run `docker-compose pull && docker-compose up -d`. See the [main README](../README.md#release-channels) for the full channel guide.
## 🐳 Quick Start
### Prerequisites
@ -60,7 +50,7 @@ open http://localhost:8008
SoulSync requires persistent storage for:
- **`./config`** → `/app/config` - Configuration files
- **`./data`** → `/app/data` - SQLite database files
- **`./database`** → `/app/database` - SQLite database files
- **`./logs`** → `/app/logs` - Application logs
- **`./downloads`** → `/app/downloads` - Downloaded music files
- **`./Transfer`** → `/app/Transfer` - Processed/matched music files
@ -73,7 +63,6 @@ environment:
- FLASK_ENV=production # Flask environment
- PYTHONPATH=/app # Python path
- SOULSYNC_CONFIG_PATH=/app/config/config.json # Config file location
- SOULSYNC_LOG_LEVEL=INFO # Optional startup log level override, takes precedence over the UI-configured log level
- TZ=America/New_York # Timezone
```
@ -240,7 +229,7 @@ services:
- "8888:8888"
volumes:
- ./config:/app/config
- ./data:/app/data
- ./database:/app/database
- ./logs:/app/logs
- ./downloads:/app/downloads
- ./Transfer:/app/Transfer
@ -279,4 +268,4 @@ services:
- [ ] Configure firewall rules
- [ ] Set up backup strategy
- [ ] Test health checks
- [ ] Verify external service connectivity
- [ ] Verify external service connectivity

625
README.md
View file

@ -2,489 +2,244 @@
<img src="./assets/trans.png" alt="SoulSync Logo">
</p>
# SoulSync - Intelligent Music Discovery & Automation Platform
# 🎵 SoulSync - Automated Music Discovery & Collection Manager
**Spotify-quality music discovery for self-hosted libraries.** Automates downloads, curates playlists, monitors artists, and organizes your collection with zero manual effort.
**Bridge streaming services to your local music library.** Automatically sync Spotify/Tidal/YouTube playlists to Plex/Jellyfin/Navidrome via Soulseek with intelligent matching, metadata enhancement, and automated discovery.
> **IMPORTANT**: Configure file sharing in slskd to avoid Soulseek bans. Set up shared folders at `http://localhost:5030/shares`.
> ⚠️ **CRITICAL**: Configure file sharing in slskd before use. Users who only download without sharing get banned by the Soulseek community. Set up shared folders at `http://localhost:5030/shares`.
**Community**: [Discord](https://discord.gg/wGvKqVQwmy) | **Website**: [ssync.net](https://www.ssync.net/) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
> 📢 **Development Focus**: New features are developed for the **Web UI** version. The Desktop GUI receives maintenance and bug fixes only.
---
## 💬 Community
## What It Does
Join the Discord server for support, feature requests, and discussions:
- **Discord**: [https://discord.gg/ePx7xYuV](https://discord.gg/ePx7xYuV)
SoulSync bridges streaming services to your music library with automated discovery:
## ✨ Core Features
1. **Monitors artists** → Automatically detects new releases from your watchlist
2. **Generates playlists** → Release Radar, Discovery Weekly, Seasonal, Decade/Genre mixes, Cache-powered discovery
3. **Downloads missing tracks** → From Soulseek, Deezer, Tidal, Qobuz, HiFi, YouTube, or any combination via Hybrid mode
4. **Verifies downloads** → AcoustID fingerprinting for all download sources
5. **Enriches metadata** → 10 enrichment workers (Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz)
6. **Tags consistently** → Picard-style MusicBrainz release preflight ensures all album tracks get the same release ID
7. **Organizes files** → Custom templates for clean folder structures
8. **Manages library** → Plex, Jellyfin, Navidrome, or SoulSync Standalone (no media server required)
9. **Scrobbles plays** → Automatic scrobbling to Last.fm and ListenBrainz from your media server
**Search & Download**
- **Enhanced Search**: Unified search across Spotify, your library, and Soulseek with categorized results (artists, albums, tracks)
- **Basic Search**: Direct Soulseek search with instant streaming and download
- Auto-sync playlists from Spotify/Tidal/YouTube to your media server
- Smart matching against your existing library
- FLAC-priority downloads from Soulseek with automatic fallback
- Customizable file organization with template-based path structures
- Synchronized lyrics (LRC) for every track via LRClib.net
---
**Metadata & Organization**
- Enhanced metadata with album art and proper tags
- Flexible folder templates: `$albumartist/$album/$track - $title`
- Automatic library scanning and database updates
- Clean, organized music collection
## Key Features
**Discovery & Automation**
- Browse complete artist discographies with similar artist recommendations
- Intelligent music discovery using your watchlist ([music-map.com](https://music-map.com) integration)
- Curated playlists: Release Radar, Discovery Weekly, Seasonal Mixes
- Beatport chart integration for electronic music
- Artist watchlist monitors new releases automatically
<p align="center">
<img src="./assets/pages.gif" alt="SoulSync Interface">
</p>
**Management**
- Comprehensive library browser with search and completion tracking
- Wishlist system with automatic retry (30-minute intervals)
- Granular wishlist management (remove individual tracks or entire albums)
- Dynamic log level control (DEBUG/INFO/WARNING/ERROR)
- Background automation handles retries and database updates
### Discovery Engine
**Release Radar** — New tracks from watchlist artists, personalized by listening history
**Discovery Weekly** — 50 tracks from similar artists with serendipity weighting
**Seasonal Playlists** — Halloween, Christmas, Valentine's, Summer, Spring, Autumn (hemisphere-aware)
**Personalized Playlists** (12+ types)
- Recently Added, Top Tracks, Forgotten Favorites
- Decade Playlists (1960s-2020s), Genre Playlists (15+ categories)
- Because You Listen To, Daily Mixes, Hidden Gems, Popular Picks, Discovery Shuffle, Familiar Favorites
- Custom Playlist Builder (1-5 seed artists → similar artists → random albums → shuffled tracks)
**Cache-Powered Discovery** (zero API calls)
- Undiscovered Albums — albums by your most-played artists that aren't in your library
- New In Your Genres — recently released albums matching your top genres
- From Your Labels — popular albums on labels already in your library
- Deep Cuts — low-popularity tracks from artists you listen to
- Genre Explorer — genre landscape pills with artist counts, tap for Genre Deep Dive modal
**ListenBrainz** — Import recommendation and community playlists
**Beatport** — Full electronic music integration with genre browser (39+ genres)
### Multi-Source Downloads
**6 Download Sources**: Soulseek, Deezer, Tidal, Qobuz, HiFi, YouTube — use any single source or Hybrid mode with drag-to-reorder priority
**Deezer Downloads** — ARL token authentication, FLAC lossless / MP3 320 / MP3 128 with automatic quality fallback and Blowfish decryption
**Tidal Downloads** — Device-flow OAuth, quality tiers from AAC 96kbps to FLAC 24-bit/96kHz Hi-Res
**Qobuz Downloads** — Email/password auth, quality up to Hi-Res Max (FLAC 24-bit/192kHz)
**HiFi Downloads** — Free lossless via public API instances, no account required
**Soulseek** — FLAC priority with quality profiles, peer quality scoring, source reuse for album consistency
**YouTube** — Audio extraction with cookie-based bot detection bypass
**Hybrid Mode** — Enable any combination of sources, drag to set priority order, automatic fallback chain
**Playlist Sources**: Spotify, Tidal, YouTube, Deezer, Beatport charts, ListenBrainz, Spotify Link (no API needed)
**Post-Download**
- Lossy copy creation: MP3, Opus, AAC with configurable bitrate (Opus capped at 256kbps)
- Hi-Res FLAC downsampling to 16-bit/44.1kHz CD quality
- Blasphemy Mode — delete original FLAC after conversion
- Synchronized lyrics (LRC) via LRClib
- ReplayGain analysis — optional track-level loudness tagging via ffmpeg, runs before lossy copy so both files get tagged
- Picard-style album consistency — pre-flight MusicBrainz release lookup ensures all tracks get the same release ID
### Listening Stats & Scrobbling
**Listening Stats Page** — Full dashboard with Chart.js visualizations
- Overview cards: total plays, listening time, unique artists/albums/tracks
- Timeline bar chart, genre breakdown donut with legend
- Top artists visual bubbles, top albums and tracks with play buttons and cover art
- Library health: format breakdown bar, enrichment coverage rings, database storage chart
- Time range filters: 7 days, 30 days, 12 months, all time
**Scrobbling** — Automatic Last.fm and ListenBrainz scrobbling from Plex, Jellyfin, or Navidrome
### Audio Verification
**AcoustID Fingerprinting** (optional) — Verifies downloaded files match expected tracks
- Runs for all download sources (Soulseek, Tidal, Qobuz, HiFi, Deezer, YouTube)
- Catches wrong versions (live, remix, cover) even from streaming API sources
- Fail-open design: verification errors never block downloads
#### AcoustID API key
AcoustID verification is opt-in. To enable it, request a free API key
at <https://acoustid.org/new-application> and paste it into
Settings → AcoustID. Without a key, downloads still complete but the
verification step is skipped silently.
If a track was previously tagged by AcoustID but the retag action in
the AcoustID Scanner no longer changes anything, see issue #704 — the
most common cause is that the file already carries a
`MUSICBRAINZ_TRACKID` tag, which the retag step uses as a short-circuit
and therefore never overwrites. Removing the cached
`MUSICBRAINZ_TRACKID` (and the `ACOUSTID_ID` if present) from the file
restores the retag.
### Metadata & Enrichment
**10 Background Enrichment Workers**: Spotify, MusicBrainz, iTunes, Deezer, Discogs, AudioDB, Last.fm, Genius, Tidal, Qobuz
- Each worker independently processes artists, albums, and tracks
- Pause/resume controls on dashboard, auto-pause during database scans
- Error items don't auto-retry in infinite loops (fixed in v2.1)
**Multi-Source Metadata**
- Primary source selectable: Spotify, iTunes/Apple Music, Deezer, or Discogs
- Spotify no longer auto-overrides — user chooses their preferred source in Settings
- Spotify auth still enables playlists, followed artists, and enrichment
- MusicBrainz enrichment with Picard-style album consistency
**Hydrabase** (optional P2P metadata network) — replaces iTunes as the metadata source when connected. Federated lookup with community-matched results, falls back automatically if disconnected. Dev-mode feature, enable in Settings → Connections.
**Genre Whitelist** — filter junk genre tags (artist names, radio show names, playlist names) from all 10 enrichment sources. 272 curated default genres, fully customizable. Off by default for backward compatibility.
**Post-Processing Tag Embedding**
- Granular per-service tag toggles (18+ MusicBrainz tags, Spotify/iTunes/Deezer IDs, AudioDB mood/style, Tidal/Qobuz ISRCs, Last.fm tags, Genius URLs)
- Multi-artist tagging options: configurable separator (comma/semicolon/slash), multi-value ARTISTS tag for Navidrome/Jellyfin multi-artist linking, optional "move featured artists to title" mode
- Album art embedding, cover.jpg download
- Spotify rate limit protection across all API calls
### Advanced Matching Engine
- Version-aware matching: strictly rejects remixes when you want the original (and vice versa)
- Unicode and accent handling (KoЯn, Bjork, A$AP Rocky)
- Fuzzy matching with weighted confidence scoring (title, artist, duration)
- Album variation detection (Deluxe, Remastered, Taylor's Version, etc.)
- Streaming source match validation: same confidence scoring applied to Tidal/Qobuz/HiFi/Deezer results as Soulseek
- Short title protection: prevents "Love" from matching "Loveless"
### Automation
**Automation Engine** — Visual drag-and-drop builder for custom workflows
- **Triggers**: Schedule, Daily/Weekly Time, Track Downloaded, Batch Complete, Playlist Changed, Discovery Complete, Signal Received, Library Scan Complete, Watchlist Match, Wishlist Item Added, and more
- **Actions**: Process Wishlist, Scan Watchlist, Refresh Mirrored, Discover Playlist, Sync Playlist, Scan Library, Database Update, Quality Scan, Full Cleanup, and 10+ more
- **Then Actions** (up to 3 per automation): Fire Signal (chain to other automations), Discord/Telegram/Pushbullet notifications, audible chimes
- **Signal Chains** — One automation fires `signal:foo`, another listens for it. Cycle detection + chain depth limit + cooldown prevent runaway chains.
- **Playlist Pipeline** — Single automation for full playlist lifecycle: refresh → discover → sync → download missing. No manual signal wiring.
- **Pipelines** — Pre-built one-click deployments (New Music, Nightly Operations, Full Library Maintenance, etc.) that install a linked group of automations at once
- **Automation Groups** — Drag-and-drop organization, bulk enable/disable, rename, right-click context menus
**Watchlist** — Monitor unlimited artists with per-artist configuration
- Release type filters: Albums, EPs, Singles
- Content filters: Live, Remixes, Acoustic, Compilations
- Auto-discover similar artists, periodic scanning
**Wishlist** — Failed downloads automatically queued for retry with auto-processing
**Mirrored Playlists** — Mirror from Spotify, Tidal, YouTube, Deezer and keep synced
- Auto-refresh detects source changes via URL/ID tracking in playlist metadata
- Discovery pipeline matches source tracks to user's primary metadata source (Spotify/iTunes/Deezer/Discogs)
- Auto Wing It fallback — tracks that fail all metadata APIs get stub metadata from the raw source title and flow through the normal download pipeline anyway
- Followed Spotify playlists that hit 403 errors fall back to public embed scraper
- Unmatch button on found tracks with DB persistence for mirrored playlists
**Local Profiles** — Multiple configuration profiles with isolated settings, watchlists, and playlists
### Library Management
**Dashboard** — Service status, system stats, activity feed, enrichment worker controls
- Unified glass UI design across all tool cards, service cards, and stat cards
**Library Page** — Artist grid with staggered card animations, per-artist enrichment coverage rings
- Artist Radio button — play random track with auto-queue radio mode
- Play buttons on Last.fm top tracks sidebar
**Enhanced Library Manager** — Toggle between Standard and Enhanced views
- Inline metadata editing, per-service manual matching
- Write Tags to File (MP3/FLAC/OGG/M4A), tag preview with diff
- Server sync after tag writes (Plex, Jellyfin, Navidrome)
- Bulk operations, sortable columns, multi-disc support
**Library Maintenance** — 10+ automated repair jobs
- Track Number, Dead Files, Duplicates, Metadata Gaps, Album Completeness, Missing Cover Art, AcoustID Scanner, Orphan Files, Fake Lossless, Library Reorganize, Lossy Converter, MBID Mismatch, Album Tag Consistency, Live/Commentary Cleaner
- Enrichment workers auto-pause during database scans
- One-click Fix All with findings dashboard
**Database Storage Visualization** — Donut chart showing per-table storage breakdown
**Live Log Viewer** — Real-time terminal-style log viewer on Settings → Logs. Color-coded levels (DEBUG/INFO/WARNING/ERROR), live filter + search, switch between log files (app, post-processing, AcoustID, source reuse). Auto-scroll, copy, clear. Updates via WebSocket every 0.5s.
**Import System** — Tag-first matching, auto-grouped album cards, staging folder workflow
- Auto-Import worker: recursive scan, single file support, AcoustID fingerprinting fallback
- Confidence-gated: 90%+ auto-imports, 70-90% queued for review
**SoulSync Standalone Mode** — Use SoulSync without Plex, Jellyfin, or Navidrome
- Downloads and imports write directly to the library database
- Filesystem scanner for incremental and deep scan of Transfer folder
- Pre-populated enrichment IDs from download context (Spotify, Deezer, MusicBrainz)
- Select in Settings → Connections → Standalone
**Template Organization** — `$albumartist/$album/$track - $title` and 10+ variables
### Built-in Media Player
- Stream tracks from your library with queue system
- Now Playing modal with album art ambient glow and Web Audio visualizer
- Smart Radio mode — auto-queue similar tracks by genre, mood, and style
- Repeat modes, shuffle, keyboard shortcuts, Media Session API
### Mobile Responsive
- Comprehensive mobile layouts for Stats, Automations, Hydrabase, Issues, Help pages
- Artist hero section, enhanced library track table with bottom sheet action popover
- Enrichment rings, filter bars, and discover cards all adapt to narrow screens
---
## Installation
## 🚀 Installation
### Docker (Recommended)
```bash
# Using docker-compose
curl -O https://raw.githubusercontent.com/Nezreka/SoulSync/main/docker-compose.yml
docker-compose up -d
# Or run directly
docker run -d -p 8008:8008 boulderbadgedad/soulsync:latest
# Access at http://localhost:8008
```
### Release Channels
SoulSync publishes two Docker image tracks so you can choose your level of stability.
**Stable — `:latest`** (recommended for most users). Hand-promoted from the `dev` branch to `main` when a batch of changes is ready for release. Published to Docker Hub. Your `docker-compose.yml` pulls this by default — no changes needed.
```bash
docker pull boulderbadgedad/soulsync:latest
```
**Nightly — `:dev`**. Rebuilt every night from the `dev` branch (and on every push to dev). Published to GitHub Container Registry. Gets new features and bug fixes before they reach `:latest`, at the cost of occasional instability as changes settle. Good for early adopters, contributors validating their own merges, and anyone helping shake out bugs on Discord before a stable release.
To switch, edit `docker-compose.yml`:
```yaml
image: ghcr.io/nezreka/soulsync:dev
```
Then run `docker-compose pull && docker-compose up -d`.
Pinned dev builds are also published as `ghcr.io/nezreka/soulsync:dev-YYYYMMDD-<sha>` if you want to stick with an exact known-good snapshot.
**Version-tagged releases** (e.g. `:2.3`, `:2.4`) are permanent tags published on both registries when a stable release is promoted:
```bash
docker pull boulderbadgedad/soulsync:2.4
# or
docker pull ghcr.io/nezreka/soulsync:2.4
```
| You are... | Use |
|---|---|
| A typical user who wants things to work | `:latest` |
| Pinning to a specific version for stability | `:2.3`, `:2.4`, etc. |
| An early adopter who wants new features early and is OK reporting bugs | `:dev` |
| A contributor testing post-merge behavior | `:dev` or a pinned dev build |
### Unraid
SoulSync is available as an Unraid template. Install from Community Applications or manually add the template from:
```
https://raw.githubusercontent.com/Nezreka/SoulSync/main/templates/soulsync.xml
```
PUID/PGID are exposed in the template — set them to match your Unraid permissions (default: 99/100 for nobody/users).
The template points at `boulderbadgedad/soulsync:latest` (stable) by default. To use the nightly `:dev` channel on Unraid, edit the container's **Repository** field to `ghcr.io/nezreka/soulsync:dev` after installing from the template.
### Python (No Docker)
### Web UI (Python)
```bash
git clone https://github.com/Nezreka/SoulSync
cd SoulSync
python -m pip install -r requirements.txt
# Build the React WebUI bundle used by the Python server.
# Docker does this automatically; Python installs must do it manually.
cd webui
npm ci
npm run build
cd ..
gunicorn -c gunicorn.conf.py wsgi:application
pip install -r requirements.txt
python web_server.py
# Open http://localhost:8008
```
When updating a Python/no-Docker install with `git pull`, rebuild the WebUI before restarting SoulSync:
### Desktop GUI
```bash
cd webui
npm ci
npm run build
cd ..
git clone https://github.com/Nezreka/SoulSync
cd SoulSync
pip install -r requirements.txt
python main.py
```
If `webui/static/dist/.vite/manifest.json` is missing or stale, React-owned routes and route handoffs may not load correctly.
**YouTube streaming / music videos** need two extra things on bare-metal installs (Docker bundles both):
- **Deno** — yt-dlp now requires a JavaScript runtime to unlock YouTube formats. Without it, streams and music-video downloads fail with `Requested format is not available`. Install: `winget install DenoLand.Deno` (Windows) or see [deno.com](https://docs.deno.com/runtime/), then restart SoulSync.
- **yt-dlp nightly** — the stable release can lag months behind YouTube changes. If YouTube breaks, update with: `python -m pip install -U --pre "yt-dlp[default]"`
### Local Development
This is only for contributors working on the WebUI with hot reload. Normal Python/no-Docker installs should build once with `npm run build` as shown above, then run only Gunicorn.
For active frontend development, use two terminals so the backend and Vite stay independent:
1. Backend
```bash
python -m pip install -r requirements-dev.txt
gunicorn -c gunicorn.dev.conf.py wsgi:application
```
The dev Gunicorn config watches backend files and restarts the Python server when they change.
2. Frontend
```bash
cd webui
npm ci
npm run dev
```
Vite hot reloads the React side when you change webui files.
Run tests separately when needed:
```bash
python -m pytest
```
If you want a convenience launcher, `python dev.py` starts both halves together
on any OS. `./dev.sh` remains available as a Unix shell wrapper.
---
## Setup Guide
## ⚡ Quick Setup
### Prerequisites
- **slskd** running and accessible ([Download](https://github.com/slskd/slskd/releases)) — required for Soulseek downloads
- **Spotify API** credentials ([Dashboard](https://developer.spotify.com/dashboard)) — optional but recommended for discovery
- **slskd**: [Download](https://github.com/slskd/slskd/releases), run on port 5030
- **Spotify API**: Client ID/Secret from [Developer Dashboard](https://developer.spotify.com/dashboard)
- **Tidal API** (optional): Client ID/Secret from [Developer Dashboard](https://developer.tidal.com/dashboard)
- **Media Server** (optional): Plex, Jellyfin, or Navidrome
- **Deno** (Python/no-Docker installs only): JavaScript runtime required by yt-dlp for YouTube streaming/music videos — `winget install DenoLand.Deno` or [deno.com](https://docs.deno.com/runtime/). Docker images bundle it.
- **Deezer ARL token** (optional): For Deezer downloads — get from browser cookies after logging into deezer.com
- **Tidal account** (optional): For Tidal downloads — authenticate via device flow in Settings
- **Qobuz account** (optional): For Qobuz downloads — email/password login in Settings
### Step 1: Set Up slskd
### API Credentials
SoulSync talks to slskd through its API. See the [slskd setup guide](https://github.com/slskd/slskd) for API key configuration.
**Spotify**
1. [Create app](https://developer.spotify.com/dashboard) → Settings
2. Add redirect URI: `http://127.0.0.1:8888/callback`
3. Copy Client ID and Secret
1. Add an API key in slskd's `settings.yml` under `web > authentication > api_keys`
2. Restart slskd
3. Paste the key into SoulSync's Settings → Downloads → Soulseek section
**Tidal**
1. [Create app](https://developer.tidal.com/dashboard)
2. Add redirect URI: `http://127.0.0.1:8889/callback`
3. Add scopes: `user.read`, `playlists.read`
4. Copy Client ID and Secret
**Configure file sharing in slskd to avoid Soulseek bans.** Set up shared folders at `http://localhost:5030/shares`.
**Plex**
- Get token from any media item URL: `?X-Plex-Token=YOUR_TOKEN`
- Server URL: `http://YOUR_IP:32400`
### Step 2: Set Up Spotify API (Optional)
**Jellyfin**
- Settings → API Keys → Generate new key
- Server URL: `http://YOUR_IP:8096`
Spotify gives you the best discovery features. Without it, SoulSync falls back to iTunes/Deezer for metadata.
**Navidrome**
- Settings → Users → Generate API Token
- Or use username/password
- Server URL: `http://YOUR_IP:4533`
1. Create an app at [developer.spotify.com/dashboard](https://developer.spotify.com/dashboard)
2. Add Redirect URI: `http://127.0.0.1:8888/callback`
3. Copy Client ID and Client Secret into SoulSync Settings
### Configuration
More detail in [Support/DOCKER-OAUTH-FIX.md](Support/DOCKER-OAUTH-FIX.md).
1. Launch SoulSync and go to Settings
2. Enter API credentials for streaming services and media server
3. Configure slskd URL (`http://localhost:5030`) and API key
4. Set download and transfer paths
5. **Customize file organization** (optional):
- Enable custom templates in Settings → File Organization
- Default: `$albumartist/$albumartist - $album/$track - $title`
- Variables: `$artist`, `$albumartist`, `$album`, `$title`, `$track`, `$playlist`
- Example: `Music/$artist/$year - $album/$track - $title`
6. **Share files in slskd** to avoid bans
### Step 3: Configure SoulSync
## 📁 File Organization
Open SoulSync at `http://localhost:8008` and go to Settings.
SoulSync supports customizable path templates with validation and fallback protection.
**Download Source**: Choose your preferred source (Soulseek, Deezer, Tidal, Qobuz, HiFi, YouTube, or Hybrid)
**Default Structure**
```
Transfer/
Artist/
Artist - Album/
01 - Track.flac
01 - Track.lrc
```
**Paths**:
- **Input Folder**: Container path to slskd's download folder (e.g., `/app/downloads`)
- **Output Folder**: Where organized music goes (e.g., `/app/Transfer`)
- **Import Folder**: Optional folder for importing existing music (e.g., `/app/Staging`)
**Template System**
- **Albums**: `$albumartist/$albumartist - $album/$track - $title`
- **Singles**: `$artist/$artist - $title/$title`
- **Playlists**: `$playlist/$artist - $title`
**Media Server** (optional): Use your machine's actual IP (not `localhost` — that means inside the container)
**Available Variables**
- `$artist`, `$albumartist`, `$album`, `$title`
- `$track` (zero-padded: 01, 02...)
- `$playlist` (playlist name)
### Step 4: Docker Path Mapping
**Features**
- Client-side validation prevents invalid templates
- Reset to defaults button in settings
- Automatic fallback if template fails
- Changes apply immediately to new downloads
| What | Container Path | Host Path |
|------|---------------|-----------|
| Config | `/app/config` | Your config folder |
| Logs | `/app/logs` | Your logs folder |
| Database | `/app/data` | Named volume (recommended) |
| Input | `/app/downloads` | Same folder slskd downloads to |
| Output | `/app/Transfer` | Where organized music goes |
| Import | `/app/Staging` | Optional folder for importing music |
## 🐳 Docker Notes
**Important:** Use a named volume for the database (`soulsync_database:/app/data`). Direct host path mounts to `/app/data` can overwrite Python module files.
**Path Configuration**
```yaml
volumes:
- ./config:/app/config # Settings persist
- ./logs:/app/logs # Log files
- /mnt/c:/host/mnt/c:rw # Mount Windows drives
- /mnt/d:/host/mnt/d:rw
```
Use `/host/mnt/X/path` in settings where X is your drive letter.
**OAuth from Remote Devices**
When accessing from a different machine, Spotify redirects may fail:
1. Complete OAuth flow - get redirected to `http://127.0.0.1:8888/callback?code=...`
2. Edit URL to use your server IP: `http://192.168.1.5:8888/callback?code=...`
3. Press Enter to complete authentication
See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) for details.
## 📊 Workflow
1. **Sync**: Select Spotify/Tidal/YouTube playlist
2. **Match**: SoulSync compares against your library
3. **Download**: Missing tracks queued from Soulseek
4. **Process**: Files enhanced with metadata, lyrics, and album art
5. **Organize**: Moved to transfer folder with template-based structure
6. **Scan**: Media server automatically rescans library
7. **Update**: SoulSync database syncs with your collection
## 🐛 Troubleshooting
**Enable Debug Logging**
- Settings → Log Level → DEBUG
- Check `logs/app.log` for detailed information
- Change takes effect immediately
**Common Issues**
*Files not organizing properly*
- Verify transfer path points to your music library
- Check template syntax in Settings → File Organization
- Use "Reset to Defaults" if templates are broken
- Review logs for path-related errors
*Docker drive access*
- Ensure drives are mounted in docker-compose.yml
- Restart Docker Desktop if mounts fail
- Verify paths use `/host/mnt/X/` prefix
*Wishlist tracks stuck*
- Remove items using delete buttons on wishlist page
- Auto-retry runs every 30 minutes
- Check logs for download failures
*Multi-library setups*
- Select correct library from dropdown in settings (Plex/Jellyfin)
- Test connection to verify credentials
## 🏗️ Architecture
- **Services**: Spotify, Tidal, Plex, Jellyfin, Navidrome, Soulseek clients
- **Database**: SQLite with automatic library caching and updates
- **UI**: PyQt6 Desktop + Flask Web Interface
- **Matching**: Advanced text normalization and fuzzy scoring
- **Metadata**: Mutagen + LRClib.net for tags and lyrics
- **Automation**: Multi-threaded with retry logic and background tasks
## 📝 Recent Updates
- **Customizable file organization** with template-based paths and validation
- **Log level control** without restart
- **Jellyfin library selector** for multi-library setups
- **Enhanced wishlist management** with track/album removal
- **Docker config persistence** between container restarts
---
## Comparison
<p align="center">
<a href="https://ko-fi.com/boulderbadgedad">
<img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support on Ko-fi">
</a>
</p>
| Feature | SoulSync | Lidarr | Headphones | Beets |
|---------|----------|--------|------------|-------|
| Custom Discovery Playlists (15+) | ✓ | ✗ | ✗ | ✗ |
| Cache-Powered Discovery (zero API) | ✓ | ✗ | ✗ | ✗ |
| Listening Stats Dashboard | ✓ | ✗ | ✗ | ✗ |
| Last.fm/ListenBrainz Scrobbling | ✓ | ✗ | ✗ | ✗ |
| 6 Download Sources | ✓ | ✗ | ✗ | ✗ |
| Deezer Downloads (FLAC) | ✓ | ✗ | ✗ | ✗ |
| Tidal Downloads (Hi-Res) | ✓ | ✗ | ✗ | ✗ |
| Qobuz Downloads (Hi-Res Max) | ✓ | ✗ | ✗ | ✗ |
| Soulseek Downloads | ✓ | ✗ | ✗ | ✗ |
| Beatport Integration | ✓ | ✗ | ✗ | ✗ |
| Audio Fingerprint Verification | ✓ | ✗ | ✗ | ✓ |
| 9 Enrichment Workers | ✓ | ✗ | ✗ | Plugin |
| Picard-Style Album Tagging | ✓ | ✗ | ✗ | ✗ |
| Visual Automation Builder | ✓ | ✗ | ✗ | ✗ |
| Enhanced Library Manager | ✓ | ✗ | ✗ | ✗ |
| Library Maintenance Suite (10+ jobs) | ✓ | ✗ | ✗ | ✓ |
| Multi-Profile Support | ✓ | ✗ | ✗ | ✗ |
| Mobile Responsive | ✓ | ✓ | ✗ | ✗ |
| Built-in Media Player + Radio | ✓ | ✗ | ✗ | ✗ |
---
## Architecture
**Scale**: ~120,000 lines across Python backend and JavaScript frontend, 80+ API endpoints, handles 10,000+ album libraries
**Integrations**: Spotify, iTunes/Apple Music, Deezer, Tidal, Qobuz, YouTube, Soulseek (slskd), HiFi, Beatport, ListenBrainz, MusicBrainz, AcoustID, AudioDB, Last.fm, Genius, LRClib, music-map.com, Plex, Jellyfin, Navidrome
**Stack**: Python 3.11, Flask, SQLite (WAL mode), vanilla JavaScript SPA, Chart.js
**Core Components**:
- **Matching Engine** — version-aware fuzzy matching with streaming source bypass
- **Download Orchestrator** — routes between 6 sources with hybrid fallback and batch processing
- **Discovery System** — personalized playlists, cache-powered sections, seasonal content
- **Metadata Pipeline** — 9 enrichment workers, Picard-style album consistency, dual-source fallback
- **Album Consistency** — pre-flight MusicBrainz release lookup before album downloads
- **Automation Engine** — event-driven workflows with signal chains and pipeline deployment
- **SoulID System** — deterministic cross-instance artist/album/track identifiers via track-verified API lookup
---
## Contributing
### Branch workflow
SoulSync uses a `dev` → `main` flow:
- **`main`** — release branch. `:latest` images auto-build from this. Only receives merges from `dev`.
- **`dev`** — integration branch. Nightly `:dev` images build from here. PRs land here first for validation before being promoted to `main`.
- **Feature branches** — branched from `dev`. PRs target `dev`.
### Opening a PR
1. Fork and clone the repo
2. Branch off `dev`: `git checkout -b fix/your-change dev`
3. Make your changes and commit
4. Push and open a PR against **`dev`** (not `main`)
5. CI (`build-and-test.yml`) runs ruff lint + compile + `python -m pytest` on your branch — wait for green
6. A maintainer reviews and merges
### Running locally
Use the [Local Development](#local-development) section above for the full repo-wide setup and the portable dev launcher.
For web UI work, see [webui/README.md](webui/README.md). It keeps the React-side notes close to the app while this file stays the single place for repo-wide dev instructions.
Ruff config lives in `pyproject.toml`. The ruleset is intentionally lenient — it catches real bugs (undefined names, import shadowing, closure-in-loop) without style nits.
### Reporting bugs / requesting features
Open an issue on GitHub. For user-side support, the Discord community is the fastest place to ask.
<p align="center">
<a href="https://star-history.com/#Nezreka/SoulSync&type=date&legend=top-left">
<img src="https://api.star-history.com/svg?repos=Nezreka/SoulSync&type=date&legend=top-left" alt="Star History">
</a>
</p>

13
RELEASE_2.7.9_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.7.9** is out 🎉 a big one.
🎚️ **Best-quality downloads** — downloads now follow a ranked quality profile you drag to order (FLAC 24/192 → mp3). best-quality mode grabs the highest-quality copy across *every* source; priority mode gets an opt-in rank-based order toggle. quarantine is folded into the Downloads page + safer imports (AcoustID fail-closed, silence/truncation guards).
🎧 **Discover got smart** — "Based On Your Listening" ranks artists from who you *actually play*, and "Your Listening Mix" is a playable track playlist of their top tracks (works on any source). Fresh Tape fills properly now.
⚡ **Wing It Pool** — a new spot next to Discovery Pool to review + re-match the tracks Wing It guessed at (they used to be invisible).
🔁 **Auto-Sync redesign** — the scheduling board is now clean horizontal lanes instead of a side-scrolling column wall.
🐛 **Fixes** — multi-disc albums no longer show disc-2 as "missing" (#927), playlists no longer stuck on "Never Synced" (#925), and tracks can't import while quarantined (#928). thanks @ramonskie + @nick2000713 🙏
⚠️ **re-scan your library once** so the multi-disc fix can backfill disc numbers on existing tracks. enjoy! 🎶

13
RELEASE_2.8.0_discord.md
View file

@ -1,13 +0,0 @@
**SoulSync 2.8.0** is out 🎉 a quality + reliability release.
🧹 **The Unverified queue, finally under control** — if you saw thousands of "unverified" rows piling up, this is for you. the AcoustID scan stops duplicating history rows, a one-time reconcile on startup clears the existing backlog from your library (no re-scan), and a new 🧹 *Clean orphaned* button sweeps dead rows whose file is gone. (#934 — thanks @nick2000713 for #938)
✂️ **Preview Clip Cleanup** — a new Tools job that finds the ~30s preview clips the HiFi source sometimes hands back instead of the full song, then deletes them and re-wishlists the real version. each finding has a ▶ Play button so you can confirm before approving.
💿 **Album Completeness handles split albums** — an album split across multiple library rows no longer shows every fragment as falsely "incomplete"; it groups the validated fragments into one correct finding. (#936 — thanks @ragnarlotus)
🐛 **Fixes** — pasted YouTube cookies no longer throw `unsupported browser: "custom"` on Docker (thanks HellRa1SeR); longer remasters aren't quarantined as "truncated" anymore (#937, thanks @diegocade1); "Add to Wishlist" from a discography went from ~15–30s *per track* to instant; wishlist art renders for re-downloads; and **Clear Completed** is back on the Downloads page.
⚡ **Performance** — trimmed the dashboard GPU usage that was hammering Firefox/Zen (and Background Particles are OFF by default now), plus bounded the runaway memory growth that could lock the app up on big libraries. (#935 / #802)
enjoy! 🎶

15
RELEASE_2.8.1_discord.md
View file

@ -1,15 +0,0 @@
**SoulSync 2.8.1** is out 🎉 a feature + reliability release.
🎧 **Export playlists to Spotify & Deezer** — the mirrored-playlist export now has **Sync to Spotify** and **Sync to Deezer** next to the ListenBrainz / JSPF options. it builds a playlist in your account from the IDs soulsync already has (the discovery cache first, then your library), so an already-discovered playlist exports **instantly with zero API calls**. re-exporting updates the same playlist instead of duplicating it, and an optional *"match missing tracks"* toggle confidently searches for the stragglers — a wrong-artist or karaoke version is left out, never guessed. the first Spotify export asks permission once. (#945)
🏷️ **Library Reorganize — Rename only** — a lighter action that just **renames your files** to your naming scheme: no re-tag, no quality/AcoustID re-check, no copy-to-staging. much faster on a NAS, and only touches files whose path actually changes. pick it from the new Action dropdown. (#875 — thanks @tsoulard / @Tacobell444)
💿 **Broader lossless handling** — lossy-copy now covers **all lossless formats**, not just FLAC (#941); and **DSD** (`.dsf`/`.dff`) is recognized as lossless instead of false-flagged "truncated" (#939).
🐛 **Download + search fixes** — an unbalanced bracket no longer false-fails as "file not found"; a file we couldn't quarantine is left for retry instead of deleted; "file not found" errors are actionable now; pasted Qobuz/Tidal links inject the exact track into manual search (#932); the Wing It pool "Fix Match" works again.
⚡ **Reduce visual effects, refined** — it no longer freezes functional motion (spinners, progress), only the expensive GPU stuff (blur, shadows, glow). worker orbs default OFF on Firefox and run at ~30fps under reduce-effects. plus a jellyfin scan watchdog fix for big libraries.
🔧 **Under the hood** — settings cleanup (#943, @nick2000713), spotify oauth hardening (#942) + npm security fixes (#944, HellRa1SeR).
enjoy! 🎶

9
RELEASE_2.8.2_discord.md
View file

@ -1,9 +0,0 @@
**SoulSync 2.8.2** is out 🎉 a stability + performance release.
🎧 **Spotify, reliably** — the Docker boot hang is fixed: with Spotify as your primary source, an unreachable Spotify API could block startup so the container bound `:8008` but never served the UI. auth probes are now deferred during boot + capped with a timeout. the "re-auth didn't stick" bug is fixed too (the OAuth callback and the app were reading different token caches), and **Sync to Spotify** now works — it asks for playlist-write permission once, on-demand, leaving your normal login untouched. (#949 — thanks HellRa1SeR)
⚡ **The "slow after update" fix** — the post-update lag wasn't SoulSync, it was browser password managers (Bitwarden/1Password/etc.) rebuilding their autofill overlay on *every* DOM change. non-credential fields are now marked so they skip them — **~110× less main-thread blocking** in the reporter's benchmark. plus a new **Max Performance** mode (Settings → Appearance) that kills every effect for no-GPU / Docker setups. (#948 — thanks @nick2000713)
📥 **Large-library imports no longer time out** — dropping a whole library into staging used to make the import page scan every file synchronously and never load. the scan runs in the background now with a live "Scanning N of M…" progress, and fills in when done. (#947 — thanks @ramonskie)
enjoy! 🎶

144
SPEC_canonical_album_version.md
View file

@ -1,144 +0,0 @@
# Spec: Canonical Album Version (fixes #765 + #767-Bug2)
**Status:** design only — no code yet.
**Goal:** Pin ONE canonical `(source, album_id)` per album, chosen by best-fit to
the user's actual files, so the Library Reorganizer, Track Number Repair, and
tagging/enrichment all agree on the same release. Today each re-resolves
independently and they contradict each other (Spotify Believer=4 vs MusicBrainz
Believer=3; standard album mislinked to a deluxe release).
**Canonical-selection rule (decided):** *match the user's actual files.* The
canonical release is the candidate whose track count + per-track durations +
titles best fit what's on disk. Self-correcting: picks standard when you own the
standard, deluxe when you own the deluxe.
---
## Hard requirement: don't disrupt the running app
Every stage below is **additive and dormant until explicitly consumed**, and
every consumer **falls back to today's behavior when no canonical is set**. So:
- albums with no resolved canonical behave EXACTLY as they do now;
- each stage is independently shippable and reversible;
- nothing big-bangs.
---
## Stage 1 — Schema + pure scorer (ships dormant, zero behavior change)
### Schema (additive, nullable → migration-safe)
Add to `albums` (guarded `ALTER TABLE ... ADD COLUMN`, idempotent — mirror the
existing column-exists checks; see [[db-schema-review]] migration-safety notes):
- `canonical_source TEXT` — e.g. 'spotify' / 'itunes' / 'musicbrainz'
- `canonical_album_id TEXT`
- `canonical_score REAL` — best-fit score (for transparency / re-resolve gating)
- `canonical_resolved_at TIMESTAMP`
All nullable. Existing rows = NULL → "unresolved" → consumers fall back. No
backfill in this stage. No reads in this stage.
### Pure core helper (the testable heart) — `core/metadata/canonical_version.py`
```
score_release_against_files(file_tracks, release_tracks) -> float
pick_canonical_release(file_tracks, candidates) -> (best, score) | (None, 0)
```
- `file_tracks`: list of {duration_ms, title, track_number?} read from disk.
- `release_tracks`: a candidate release's tracklist (same shape).
- Scoring (tunable weights):
- **track-count fit** — exact match strongly preferred; |Δcount| penalized.
- **duration alignment** — greedily match each file to its closest release
track by duration (within a tolerance, e.g. ±3s); reward coverage.
- **title overlap** — token/fuzzy overlap as a tiebreaker.
- **graceful degradation** — if a source gives no per-track durations, fall
back to count + title only (never crash, never force-pick).
- Returns the best candidate + score, or (None, 0) when nothing clears a floor
(so we never pin a bad guess — leave it unresolved, consumers fall back).
### Tests (extreme, like the rest of this codebase)
- standard (11) vs deluxe (17) with 11 files on disk → picks standard.
- same album, 17 files → picks deluxe.
- duration disambiguation when track counts tie (e.g. radio edit vs album).
- missing-duration source → count+title fallback still picks sanely.
- no candidate clears the floor → (None, 0).
- "Believer" standard(=track 3 listing) vs Spotify(=4) with the user's files →
whichever the files actually match.
**End of Stage 1: scorer exists + tested, columns exist, NOTHING reads/writes
them yet. Provably zero behavior change.**
---
## Stage 2 — Resolver populates canonical (writes, still no consumers)
A function `resolve_canonical_for_album(album_id, db, ...)`:
1. Gather on-disk file metadata for the album (durations/titles) via the
library's known file paths.
2. Gather candidate releases: every source the album has an ID for
(spotify/itunes/deezer/discogs/soul/musicbrainz) AND — for the deluxe/standard
case — sibling editions discoverable from those. Fetch each tracklist
(cached, rate-limited).
3. `pick_canonical_release(files, candidates)` → store `(source, album_id, score)`
on the album row if it clears the floor.
Wiring: a small **backfill repair job** (dry-run-capable) + a hook in enrichment
when an album is (re)enriched. Still **no tool READS canonical**, so behavior is
unchanged — this stage only populates the new columns. Reversible: clearing the
columns reverts to unresolved.
Tests: resolver picks the right release for the standard/deluxe fixtures; stores
nothing when below floor; idempotent re-resolve.
Cost note: fetching multiple candidate releases = more API calls. Mitigate via
cache + only-on-(re)enrich + the existing rate trackers. Surface in the job's
progress so it's not silent.
---
## Stage 3 — Reorganizer reads canonical (first real behavior change, gated)
In `library_reorganize._resolve_source`: if the album has
`canonical_source`/`canonical_album_id`, use THAT first; else fall back to the
current `get_source_priority` walk. One-line precedence change, fully gated on
non-NULL.
Tests: with canonical set → resolves to it; with canonical NULL → byte-identical
to today. Re-run the existing reorganize battery (148 tests) — must stay green.
**This alone fixes #767-Bug2** (a standard album whose files match the standard
release pins the standard, so reorganize stops targeting the deluxe folder).
---
## Stage 4 — Track Number Repair reads canonical (closes #765)
In `track_number_repair._resolve_album_tracklist`: add **Fallback -1** (before
everything) — if the album has a canonical `(source, album_id)`, use it. The
existing 6-level cascade stays as the fallback for albums with no canonical
(preserves its all-01-album rescue ability — the regression risk we refused to
take in the reactive fix).
Now both tools resolve the SAME release → same track numbers → no contradiction.
Tests: canonical present → both tools agree (shared-release test); canonical
NULL → existing cascade unchanged.
---
## Risks & mitigations
- **Extra API calls** (Stage 2 fetches multiple releases) → cache, rate-limit,
only-on-(re)enrich, progress-logged.
- **Sources without per-track durations** → scorer degrades to count+title.
- **Schema migration** → additive nullable columns only; idempotent guards.
- **Wrong pick** → floor gate (never pin a low-confidence guess); `canonical_score`
stored for inspection/re-resolve; manual override possible later.
- **Backward-compat** → every consumer falls back to today's path when NULL, so
un-resolved albums (incl. all existing albums until backfilled) are unaffected.
## Out of scope (for now)
- Per-album manual version override UI (can layer on later — the columns support it).
- Merging the two tools into one (the reporter's alt suggestion) — unnecessary
once they share the canonical.
## Suggested order to build
1, then 2, then 3, then 4 — each shippable and verifiable on its own. We can stop
after any stage and the app is consistent (just with fewer consumers wired).

2436
Support/API.md
View file

File diff suppressed because it is too large Load diff

237
Support/AUTOMATIONS.md
View file

@ -1,237 +0,0 @@
# Automations Guide
## Overview
The Automations page lets you build custom workflows that run automatically. Each automation connects a **trigger** (when to run) to an **action** (what to do), with optional **conditions** (filters) and **notifications** (alerts when it runs).
Navigate to the Automations page from the sidebar. You'll see your automation cards and a builder panel for creating new ones.
---
## Building an Automation
### The Builder
The builder has three slots:
- **WHEN** — drag a trigger here (required)
- **DO** — drag an action here (required)
- **NOTIFY** — drag a notification method here (optional)
Drag blocks from the sidebar into the slots. Each block expands to show its configuration fields. Give your automation a name and click **Save**.
### Conditions
Event-based triggers support conditions to filter when they fire. For example, a "Track Downloaded" trigger can have a condition like `artist contains "Taylor"` so it only fires for specific artists.
- **Match mode**: "All" (every condition must pass) or "Any" (at least one must pass)
- **Operators**: contains, equals, starts_with, not_contains
### Delay
Action blocks have an optional **Delay** field (in minutes). The action waits that long after the trigger fires before executing. Useful for letting other processes finish first.
---
## Triggers
### Timer-Based
| Trigger | Description | Configuration |
|---------|-------------|---------------|
| **Schedule** | Run on a repeating interval | Interval + unit (minutes/hours/days) |
| **Daily Time** | Run every day at a specific time | Time picker (e.g., 03:00) |
| **Weekly Schedule** | Run on specific days at a set time | Day selector + time picker |
### Event-Based
| Trigger | Fires When | Condition Fields | Variables |
|---------|-----------|-----------------|-----------|
| **App Started** | SoulSync starts up | — | — |
| **Track Downloaded** | A track finishes downloading | artist, title, album, quality | artist, title, album, quality |
| **Batch Complete** | An album/playlist download finishes | playlist_name | playlist_name, total_tracks, completed_tracks, failed_tracks |
| **New Release Found** | Watchlist detects new music | artist | artist, new_tracks, added_to_wishlist |
| **Playlist Synced** | A playlist sync completes | playlist_name | playlist_name, total_tracks, matched_tracks, synced_tracks, failed_tracks |
| **Playlist Changed** | A mirrored playlist detects track changes | playlist_name | playlist_name, old_count, new_count, added, removed |
| **Discovery Complete** | Playlist track discovery finishes | playlist_name | playlist_name, total_tracks, discovered_count, failed_count, skipped_count |
| **Wishlist Processed** | Auto-wishlist processing finishes | — | tracks_processed, tracks_found, tracks_failed |
| **Watchlist Scan Done** | Watchlist scan finishes | — | artists_scanned, new_tracks_found, tracks_added |
| **Database Updated** | Library database refresh finishes | — | total_artists, total_albums, total_tracks |
| **Download Failed** | A track permanently fails to download | artist, title, reason | artist, title, reason |
| **File Quarantined** | AcoustID verification fails | artist, title | artist, title, reason |
| **Wishlist Item Added** | A track is added to wishlist | artist, title | artist, title, reason |
| **Artist Watched** | An artist is added to watchlist | artist | artist, artist_id |
| **Artist Unwatched** | An artist is removed from watchlist | artist | artist, artist_id |
| **Import Complete** | Album/track import finishes | artist, album_name | track_count, album_name, artist |
| **Playlist Mirrored** | A new playlist is mirrored | playlist_name, source | playlist_name, source, track_count |
| **Quality Scan Done** | Quality scan finishes | — | quality_met, low_quality, total_scanned |
| **Duplicate Scan Done** | Duplicate cleaner finishes | — | files_scanned, duplicates_found, space_freed |
---
## Actions
| Action | Description | Configuration |
|--------|-------------|---------------|
| **Process Wishlist** | Retry failed downloads from wishlist | Category: All, Albums, or Singles |
| **Scan Watchlist** | Check watched artists for new releases | — |
| **Scan Library** | Trigger media server library scan | — |
| **Refresh Mirrored Playlist** | Re-fetch playlist from source (Spotify/Tidal/YouTube) and update the mirror | Select playlist or "Refresh all" |
| **Discover Playlist** | Find official Spotify/iTunes metadata for mirrored playlist tracks | Select playlist or "Discover all" |
| **Sync Playlist** | Sync mirrored playlist to media server (only discovered tracks are included) | Select playlist |
| **Notify Only** | No action — just send the notification | — |
| **Update Database** | Trigger library database refresh | Full refresh checkbox |
| **Run Duplicate Cleaner** | Scan for and remove duplicate files | — |
| **Clear Quarantine** | Delete all quarantined files | — |
| **Clean Up Wishlist** | Remove duplicate/already-owned tracks from wishlist | — |
| **Update Discovery** | Refresh discovery pool with new tracks | — |
| **Run Quality Scan** | Scan for low-quality audio files | Scope: Watchlist Artists or Full Library |
| **Backup Database** | Create timestamped database backup | — |
---
## Notifications
Add a notification block to get alerted when an automation runs.
| Method | Configuration | Notes |
|--------|---------------|-------|
| **Discord Webhook** | Webhook URL + message template | Posts to a Discord channel |
| **Pushbullet** | Access token + title + message | Push to phone/desktop |
| **Telegram** | Bot token + chat ID + message | Sends via Telegram Bot API |
### Variable Substitution
Notification messages support `{variable}` placeholders that get replaced with actual values when the automation runs.
**Always available**: `{time}`, `{name}` (automation name), `{run_count}`, `{status}`
**Event-specific**: Each trigger provides additional variables (see the Variables column in the triggers table above). For example, a "Track Downloaded" trigger provides `{artist}`, `{title}`, `{album}`, `{quality}`.
**Example message**:
```
Downloaded {title} by {artist} from {album} — quality: {quality}
```
---
## System Automations
SoulSync includes two built-in system automations that cannot be deleted:
| Automation | Schedule | Initial Delay |
|-----------|----------|---------------|
| **Auto-Process Wishlist** | Every 30 minutes | 1 minute after startup |
| **Auto-Scan Watchlist** | Every 24 hours | 5 minutes after startup |
These appear with a "System" badge on their cards. You can:
- Change the interval
- Enable or disable them
- Add notifications
You cannot:
- Delete them
- Change the trigger or action type
---
## Mirrored Playlist Sync Pipeline
For mirrored playlists (especially from YouTube and Tidal), a multi-step automation chain ensures tracks are synced with proper metadata:
### The Problem
YouTube and Tidal playlists have raw metadata — cleaned video titles, uploader names. If you sync these directly, unmatched tracks hit the wishlist with garbage data (no Spotify ID, wrong album, no cover art). Downloads would fail or get the wrong track.
### The Solution
Three automations chained via events:
**Step 1: Refresh** — Re-fetch the playlist from its source
```
WHEN: Schedule (every 6 hours)
DO: Refresh Mirrored Playlist (all)
```
This detects added/removed tracks by comparing source track IDs. If changes are found, it emits a "Playlist Changed" event.
**Step 2: Discover** — Match raw tracks to official Spotify/iTunes metadata
```
WHEN: Playlist Changed
DO: Discover Playlist (all)
```
For each undiscovered track, the discovery pipeline:
1. Checks the discovery cache (instant if previously matched)
2. Searches Spotify (preferred) or iTunes (fallback) using the matching engine
3. Scores candidates with title/artist fuzzy matching
4. Stores the official match (Spotify ID, proper title, artist, album) on the track
When done, emits a "Discovery Complete" event.
**Step 3: Sync** — Push to media server with verified metadata
```
WHEN: Discovery Complete
DO: Sync Playlist (select playlist)
```
Only discovered tracks are included in the sync. Undiscovered tracks are skipped entirely — they never reach the wishlist with bad data. Unmatched discovered tracks go to the wishlist with proper Spotify/iTunes IDs and album context.
### Spotify Playlists
Spotify-sourced mirrored playlists skip Step 2 automatically. Their data is already official, so tracks are marked as discovered during refresh with confidence 1.0. You can go directly from "Playlist Changed" to "Sync Playlist".
### Discovery Caching
Discovery results are cached globally. If the same track appears in multiple playlists, or was discovered previously, the cache provides instant results without hitting the Spotify/iTunes API again. The cache persists across restarts.
---
## Examples
### Get notified when a watched artist drops new music
```
WHEN: New Release Found (artist contains "Kendrick")
DO: Notify Only
NOTIFY: Discord Webhook — "{artist} dropped {new_tracks} new tracks!"
```
### Nightly library maintenance
```
WHEN: Daily Time (03:00)
DO: Update Database (full refresh)
```
### Auto-download wishlist failures every hour
```
WHEN: Schedule (every 1 hour)
DO: Process Wishlist (all)
NOTIFY: Telegram — "Wishlist processed: {tracks_found} found, {tracks_failed} failed"
```
### Quality upgrade pipeline
```
WHEN: Database Updated
DO: Run Quality Scan (watchlist artists)
```
### Discord alert on download failures
```
WHEN: Download Failed
DO: Notify Only
NOTIFY: Discord Webhook — "Failed to download {title} by {artist}: {reason}"
```
### Weekly database backup
```
WHEN: Weekly Schedule (Sun at 02:00)
DO: Backup Database
```
---
## Tips
- **Test with "Run Now"**: Every automation card has a play button that triggers it immediately, regardless of its schedule. Use this to verify your setup before waiting for the timer.
- **Check the activity feed**: The Dashboard activity feed shows when automations run and their results.
- **Conditions narrow, not widen**: Without conditions, an event trigger fires for every event of that type. Conditions filter it down to specific cases.
- **Delay is per-execution**: If you set a 5-minute delay, the action waits 5 minutes after each trigger fire, not 5 minutes after the last execution.
- **Cross-guards**: The system automations (wishlist/watchlist) have mutual exclusion — if one is running, the other waits until the next scheduled time rather than queueing up.
- **Discovery is incremental**: Running "Discover Playlist" only processes tracks that haven't been discovered yet. Already-discovered tracks are skipped. Failed tracks are re-attempted on subsequent runs.

114
Support/DOCKER-OAUTH-FIX.md
View file

@ -1,114 +0,0 @@
# 🔐 Docker OAuth Authentication Fix
## Problem: "Insecure redirect URI" Error
When accessing SoulSync from a **different device** than the Docker host, you may encounter:
- `INVALID_CLIENT: Insecure redirect URI`
- `Spotify authentication failed: error: invalid_client`
**Why this happens:** Spotify requires HTTPS for OAuth callbacks when not using localhost.
## ✅ Simple Solution: SSH Port Forwarding
### Step 1: Set up SSH tunnel from your device to Docker host
**On the device you're browsing from** (laptop/phone/etc):
```bash
# Replace 'user' and 'docker-host-ip' with your actual values
ssh -L 8888:localhost:8888 -L 8889:localhost:8889 user@docker-host-ip
# Example:
ssh -L 8888:localhost:8888 -L 8889:localhost:8889 john@192.168.1.100
```
**Keep this SSH connection open** while using SoulSync.
### Step 2: Configure OAuth redirect URIs
**In your Spotify Developer App:**
- Set redirect URI to: `http://127.0.0.1:8888/callback`
**In your Tidal Developer App:**
- Set redirect URI to: `http://127.0.0.1:8889/tidal/callback`
**In SoulSync Settings:**
- Set Spotify redirect URI to: `http://127.0.0.1:8888/callback`
- Set Tidal redirect URI to: `http://127.0.0.1:8889/tidal/callback`
### Step 3: Use SoulSync normally
- Access SoulSync: `http://docker-host-ip:8008` (normal HTTP)
- OAuth callbacks will tunnel through SSH to localhost
- Authentication will work without HTTPS requirements
## 🖥️ Alternative: Direct Access from Docker Host
If you can access SoulSync directly from the Docker host machine:
- Use: `http://127.0.0.1:8008`
- Set OAuth redirect URIs to localhost (as above)
- No SSH tunnel needed
## 🔧 Reverse Proxy Setup (Caddy, Nginx, Traefik)
If you're running SoulSync behind a reverse proxy with HTTPS, you can use the **main app port (8008)** for OAuth callbacks instead of the standalone port 8888. This is the recommended approach for reverse proxy setups.
### Step 1: Set your redirect URI to your proxy URL
**In SoulSync Settings:**
- Set Spotify redirect URI to: `https://yourdomain.com/callback`
**In your Spotify Developer Dashboard:**
- Add the same redirect URI: `https://yourdomain.com/callback`
### Step 2: Ensure your reverse proxy forwards to port 8008
Your reverse proxy should forward traffic to SoulSync's main port (8008). The `/callback` path is handled by the main Flask app — no need to expose port 8888.
**Example Caddy config:**
```
soulsync.yourdomain.com {
reverse_proxy localhost:8008
}
```
**Example Nginx config:**
```nginx
server {
listen 443 ssl;
server_name soulsync.yourdomain.com;
location / {
proxy_pass http://localhost:8008;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
### Step 3: Authenticate normally
Click "Connect Spotify" in SoulSync settings. After authorizing on Spotify, you'll be redirected back through your reverse proxy automatically.
### Important notes for reverse proxy users
- The redirect URI **must use HTTPS** for non-localhost domains (Spotify requirement)
- The redirect URI in SoulSync settings **must exactly match** the one in your Spotify Dashboard
- Port 8888 is only needed for direct/local access — you do **not** need to expose it through your proxy
- Make sure your proxy passes query parameters through unmodified (most do by default)
## 📝 Summary
The core issue is that **Spotify requires HTTPS for non-localhost** OAuth redirects.
**Choose your approach:**
- **Reverse proxy with HTTPS**: Set redirect URI to `https://yourdomain.com/callback` (recommended for production)
- **SSH tunnel**: Makes remote devices appear as localhost — set redirect URI to `http://127.0.0.1:8888/callback`
- **Local access**: No special config needed — default `http://127.0.0.1:8888/callback` works
**Key points:**
- ✅ Reverse proxy users: use `https://yourdomain.com/callback` on port 8008
- ✅ SSH tunnel users: use `http://127.0.0.1:8888/callback` on port 8888
- ✅ Redirect URI must match exactly in SoulSync settings AND Spotify Dashboard
- ✅ Query parameters must be preserved through the redirect chain

190
Support/IMPORT-STAGING-GUIDE.md
View file

@ -1,190 +0,0 @@
# Import & Staging Folder Guide
## Overview
Got a mess of audio files — ripped CDs, old downloads, files from other apps — that you want in your library with proper metadata? That's what Import is for.
Drop your unorganized files into the staging folder, open the Import page, and match them to albums or tracks. SoulSync takes care of the rest: full metadata tagging (artist, album, track number, genres), album art embedding, lyrics fetching, renaming to your path template, and moving everything into your organized library. Files go from a chaotic pile to properly tagged, organized tracks in your transfer folder.
---
## Setup
### 1. Configure the Staging Path
In **Settings**, find the **"Import Staging Dir"** field. The default is `./Staging`.
For Docker, map a host folder to the container:
```yaml
volumes:
- /path/to/your/staging:/app/Staging
```
Then set the staging path in SoulSync settings to `/app/Staging`.
### 2. Add Files to the Staging Folder
Drop audio files into your staging folder. Supported formats:
| Format | Extension |
|--------|-----------|
| FLAC | `.flac` |
| MP3 | `.mp3` |
| AAC | `.aac`, `.m4a` |
| OGG Vorbis | `.ogg` |
| Opus | `.opus` |
| WAV | `.wav` |
| WMA | `.wma` |
| AIFF | `.aiff`, `.aif` |
| Monkey's Audio | `.ape` |
You can organize files in subfolders — SoulSync reads folder names as hints for album suggestions (e.g., a folder named `Artist - Album` improves matching).
---
## Using the Import Page
Click **Import** in the sidebar to open the Import page. The top bar shows your staging folder path, file count, and total size. Click **Refresh** to re-scan if you've added new files.
There are two modes: **Albums** and **Singles**.
---
### Album Mode
Use this when your staging files belong to a complete album (or part of one).
#### Step 1: Find the Album
SoulSync automatically suggests albums based on your files' metadata tags and folder structure. These appear as album cards with cover art.
If the right album isn't suggested, use the **search bar** to find it by name.
#### Step 2: Select an Album
Click an album card to select it. You'll see:
- Album cover art, title, artist, track count, and release year
- The full tracklist with track numbers and names
- Automatic file-to-track matching with confidence percentages
#### Step 3: Review Matches
SoulSync matches your staging files to album tracks using title similarity and track numbering. Each match shows a confidence percentage:
- **70%+** — High confidence, likely correct
- **Below 70%** — Worth double-checking
- **100%** — You assigned it manually
Unmatched files appear in an **"Unmatched Files"** pool at the bottom.
#### Step 4: Fix Mismatches (Drag-and-Drop)
If a file was matched to the wrong track:
1. **Drag** a file from the unmatched pool
2. **Drop** it onto the correct track row
3. The previous match (if any) returns to the unmatched pool
To remove a match, click the **X** button next to the matched file.
You can also click **"Re-match Automatically"** to reset all manual overrides and let SoulSync re-run its matching.
**On mobile:** Tap a file chip to select it, then tap the track row to assign it.
#### Step 5: Process
The bottom of the page shows how many tracks are matched (e.g., "8 of 12 tracks matched"). Click **"Process X Tracks"** to start importing.
---
### Singles Mode
Use this for individual tracks that aren't part of an album.
#### Step 1: Browse Files
All staging files appear as a list showing filename and any metadata tags found.
#### Step 2: Identify Tracks
Click **"Identify"** next to a file to search for the matching Spotify track. Select the correct result.
#### Step 3: Select and Process
Check the boxes next to files you want to import. Use **"Select All"** to toggle everything. Click **"Process Selected (N)"** to queue them.
---
## Processing Queue
When you start an import, a processing queue appears showing real-time progress:
- **Progress bar** fills as tracks complete
- **Status** shows counts like "3/10" (processed/total)
- **Errors** are shown inline (e.g., "8/10 (2 err)")
- **"Clear finished"** button removes completed/errored jobs
Processing continues in the background even if you navigate away from the Import page. After all jobs finish, the staging folder is automatically re-scanned and suggestions refresh.
### What Processing Does
For each matched track, SoulSync:
1. Enriches the file with full Spotify metadata (artist, album, track number, disc number, genres)
2. Embeds album artwork
3. Fetches synchronized lyrics (LRC) when available
4. Renames and moves the file to your transfer folder using your configured path template
5. Triggers a media server library scan (if connected)
---
## Tips
- **Organize by album** — Putting files in an `Artist - Album` subfolder significantly improves automatic suggestions and matching
- **Tag your files first** — Files with proper ID3/metadata tags get better automatic matches
- **Start with Album mode** — It's faster for grouped files since you match a whole album at once
- **Use Singles mode for loose tracks** — Mixtapes, random downloads, one-offs
- **Check confidence scores** — Low percentages mean the match might be wrong
- **Drag-and-drop is your friend** — Faster than re-searching when a match is close but wrong
---
## Troubleshooting
| Problem | Solution |
|---------|----------|
| No files showing up | Check that your staging path is correct in Settings and the folder isn't empty. Click Refresh. |
| No album suggestions | Files may lack metadata tags. Try searching manually by album name. |
| Wrong track matched | Drag the correct file from the unmatched pool onto the track, or click X to unmatch and try again. |
| Processing fails | Check that your transfer path is writable. Enable DEBUG logging in Settings and check `logs/app.log`. |
| Files not disappearing after import | Successfully processed files are moved to your transfer folder. Check there. Failed files remain in staging. |
| Docker: staging folder empty | Verify your volume mapping points to the right host folder and the container path matches your Settings value. |
---
## Docker Example
```yaml
volumes:
# Your staging folder for imports
- /mnt/user/Music/Staging:/app/Staging
# Where processed files end up
- /mnt/user/Music/Library:/app/Transfer:rw
```
**SoulSync Settings:**
- Import Staging Dir: `/app/Staging`
- Transfer Path: `/app/Transfer`
**Workflow:**
```
1. You drop files into /mnt/user/Music/Staging (host)
→ /app/Staging (container)
2. Open Import page → Match to albums/tracks
3. Process → Files move to /app/Transfer (container)
→ /mnt/user/Music/Library (host)
→ Media server picks them up
```

1219
Support/METADATA-FALLBACK-IMPLEMENTATION.md
View file

File diff suppressed because it is too large Load diff

134
Support/REVERSE-PROXY.md
View file

@ -1,134 +0,0 @@
# Running SoulSync behind a reverse proxy (nginx / Caddy / Traefik)
Putting SoulSync behind a reverse proxy lets you serve it over **HTTPS** and — the
important part — put **authentication** in front of it before exposing it to the
internet. This guide covers the safe setup.
> **The golden rule:** the safest way to expose *any* self-hosted app publicly is
> to require authentication at the proxy (an auth layer), **not** to rely on the
> app's own protection. SoulSync's launch PIN is a useful fallback, but it is not
> a substitute for a real auth layer on a public instance.
---
## 1. Turn on reverse-proxy mode
By default SoulSync does **not** trust proxy headers (so a direct client can't spoof
its IP or pretend the connection is HTTPS). If you're behind a proxy that
terminates TLS, turn on **Settings → Security → "Behind a reverse proxy"** and
**restart SoulSync** (this option applies at startup).
When enabled, SoulSync:
- trusts `X-Forwarded-For/Proto/Host/Port` from **one** proxy hop (correct client
IP, HTTPS detection, redirects),
- marks its session cookie `Secure` (HTTPS-only) + `SameSite=Lax`, and
- sends conservative security headers (`X-Content-Type-Options: nosniff`,
`X-Frame-Options: SAMEORIGIN`, `Strict-Transport-Security`). No CSP is set — tune
one at your proxy if you want it.
**Leave it off if you access SoulSync directly over http:// on your LAN** — turning
it on would make the session cookie HTTPS-only and break plain-HTTP access. With it
off, none of the above applies and SoulSync behaves exactly as before.
> The launch PIN is also brute-force limited (10 wrong attempts from an IP → a
> short cooldown), regardless of this setting — a correct PIN is never affected.
Restart SoulSync after changing it.
---
## 2. nginx
SoulSync uses WebSockets (Socket.IO), so the `Upgrade`/`Connection` headers are
**required** — without them live updates silently stop working.
```nginx
server {
listen 443 ssl;
server_name soulsync.example.com;
ssl_certificate /etc/letsencrypt/live/soulsync.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/soulsync.example.com/privkey.pem;
# Large library scans / uploads
client_max_body_size 0;
location / {
proxy_pass http://127.0.0.1:8008;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
# Required for Socket.IO / live updates
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 3600s; # long-running scans
proxy_send_timeout 3600s;
}
}
```
---
## 3. Caddy
Caddy handles TLS automatically and proxies WebSockets out of the box:
```caddy
soulsync.example.com {
reverse_proxy 127.0.0.1:8008
}
```
Caddy sets `X-Forwarded-*` for you. (Add an auth provider directive if you want
auth at the proxy — see below.)
---
## 4. Traefik
Traefik proxies WebSockets automatically and forwards the headers. Point a router
at the SoulSync service on port `8008` with your TLS resolver; no extra WebSocket
config is needed.
---
## 5. Add authentication in front (recommended for public instances)
Pick one:
- **Auth proxy** — [Authelia](https://www.authelia.com/),
[Authentik](https://goauthentik.io/), or
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). These sit in front
of SoulSync and force a login (with 2FA) before any request reaches it. Best
option for internet exposure.
SoulSync can **trust the proxy's authenticated-user header** so the launch PIN is
skipped once the proxy has logged you in. Set the header name in **Settings →
Security → "Auth proxy user header"** (e.g. `Remote-User`).
> ⚠️ **Only enable this behind a proxy you control that STRIPS any client-supplied
> copy of that header.** Otherwise a direct visitor could send `Remote-User: admin`
> and walk straight in. It's **off by default** — an unset header name means
> SoulSync ignores the header entirely (a spoofed one does nothing).
- **HTTP Basic Auth** — quick and simple (nginx `auth_basic` / Caddy `basicauth`).
Better than nothing; weaker than an auth proxy.
- **SoulSync launch PIN** — set an admin PIN in Settings. Enforced server-side, so
it can't be bypassed by hitting the API directly — but it's a shared PIN, so
treat it as a fallback, not your only gate.
---
## Troubleshooting
- **Live updates / progress bars don't move** → the WebSocket `Upgrade`/`Connection`
headers are missing (nginx) or your proxy is buffering. Check section 2.
- **Login won't stick / "session expired"** → you enabled `trust_reverse_proxy` but
are reaching SoulSync over plain `http://`. The session cookie is now HTTPS-only;
use `https://`, or turn the setting off for direct HTTP access.
- **Scans time out** → raise `proxy_read_timeout` / `proxy_send_timeout`.

0
Support/UNRAID.md → UNRAID.md
View file

BIN
advanced_settings_screenshot.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 59 KiB

93
api/__init__.py
View file

@ -1,93 +0,0 @@
"""
SoulSync Public REST API (v1)
Blueprint factory + rate-limiter initialisation.
"""
from flask import Blueprint
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
from utils.logging_config import get_logger
from .helpers import api_error
logger = get_logger("api_v1")
# ---------------------------------------------------------------------------
# Rate limiter (initialised with the app in web_server.py via limiter.init_app)
# ---------------------------------------------------------------------------
limiter = Limiter(
key_func=get_remote_address,
default_limits=[], # No global default — limits are applied per-blueprint
storage_uri="memory://",
)
def create_api_blueprint():
"""Build and return the /api/v1 Blueprint with all sub-modules registered."""
bp = Blueprint("api_v1", __name__)
# ---- import & register sub-module routes ----
from .library import register_routes as reg_library
from .system import register_routes as reg_system
from .search import register_routes as reg_search
from .wishlist import register_routes as reg_wishlist
from .watchlist import register_routes as reg_watchlist
from .downloads import register_routes as reg_downloads
from .playlists import register_routes as reg_playlists
from .settings import register_routes as reg_settings
from .discover import register_routes as reg_discover
from .profiles import register_routes as reg_profiles
from .retag import register_routes as reg_retag
from .listenbrainz import register_routes as reg_listenbrainz
from .cache import register_routes as reg_cache
from .request import register_routes as reg_request
from .request import start_cleanup_thread as _start_request_cleanup
# ---- rate-limit only /api/v1 routes (not the whole app) ----
limiter.limit("60 per minute")(bp)
reg_library(bp)
reg_system(bp)
reg_search(bp)
reg_wishlist(bp)
reg_watchlist(bp)
reg_downloads(bp)
reg_playlists(bp)
reg_settings(bp)
reg_discover(bp)
reg_profiles(bp)
reg_retag(bp)
reg_listenbrainz(bp)
reg_cache(bp)
reg_request(bp)
# Start the periodic cleanup timer for in-memory request tracking so
# idle periods don't leave stale entries in memory. Idempotent across
# calls; safe with multi-blueprint registration.
_start_request_cleanup()
# ---- error handlers (scoped to this Blueprint) ----
@bp.errorhandler(400)
def _bad_request(e):
return api_error("BAD_REQUEST", str(e), 400)
@bp.errorhandler(404)
def _not_found(e):
return api_error("NOT_FOUND", "Resource not found.", 404)
@bp.errorhandler(429)
def _rate_limited(e):
return api_error("RATE_LIMITED", "Too many requests. Please slow down.", 429)
@bp.errorhandler(500)
def _internal(e):
return api_error("INTERNAL_ERROR", "An internal server error occurred.", 500)
@bp.errorhandler(Exception)
def _unhandled(e):
logger.error(f"Unhandled API error: {e}", exc_info=True)
return api_error("INTERNAL_ERROR", "An unexpected error occurred.", 500)
return bp

101
api/auth.py
View file

@ -1,101 +0,0 @@
"""
API key authentication for the SoulSync public API.
"""
import hashlib
import secrets
import threading
import uuid
from datetime import datetime, timedelta, timezone
from functools import wraps
from flask import request, current_app
from .helpers import api_error
# Throttle persistence of `last_used_at` so every authenticated request
# does not rewrite the full app config. Maps key_hash -> last-persisted datetime.
_USAGE_WRITE_INTERVAL = timedelta(minutes=15)
_last_persisted_usage: dict[str, datetime] = {}
_usage_lock = threading.Lock()
def _should_persist_usage(key_hash: str, now: datetime) -> bool:
"""Return True if `last_used_at` for the given key should be written to disk.
Thread-safe: tracks the last write per key hash in memory and only returns
True once per `_USAGE_WRITE_INTERVAL`.
"""
with _usage_lock:
previous = _last_persisted_usage.get(key_hash)
if previous is None or (now - previous) >= _USAGE_WRITE_INTERVAL:
_last_persisted_usage[key_hash] = now
return True
return False
def generate_api_key(label=""):
"""Generate a new API key.
Returns (raw_key, key_record). The raw key is shown to the user
exactly once; only the SHA-256 hash is persisted.
"""
raw_key = f"sk_{secrets.token_urlsafe(32)}"
key_hash = hashlib.sha256(raw_key.encode()).hexdigest()
record = {
"id": str(uuid.uuid4()),
"label": label,
"key_hash": key_hash,
"key_prefix": raw_key[:11], # "sk_" + first 8 chars
"created_at": datetime.now(timezone.utc).isoformat(),
"last_used_at": None,
}
return raw_key, record
def _hash_key(raw_key):
return hashlib.sha256(raw_key.encode()).hexdigest()
def require_api_key(f):
"""Decorator that enforces API key authentication."""
@wraps(f)
def decorated(*args, **kwargs):
# Extract key from header or query param
api_key = None
auth_header = request.headers.get("Authorization", "")
if auth_header.startswith("Bearer "):
api_key = auth_header[7:]
if not api_key:
api_key = request.args.get("api_key")
if not api_key:
return api_error("AUTH_REQUIRED", "API key is required. "
"Pass via Authorization: Bearer <key> header "
"or ?api_key= query parameter.", 401)
config_mgr = current_app.soulsync["config_manager"]
stored_keys = config_mgr.get("api_keys", [])
key_hash = _hash_key(api_key)
matched = None
for stored in stored_keys:
if stored.get("key_hash") == key_hash:
matched = stored
break
if not matched:
return api_error("INVALID_KEY", "Invalid API key.", 403)
# Update last-used timestamp (best-effort, throttled to avoid rewriting
# the full app config on every authenticated request).
now = datetime.now(timezone.utc)
matched["last_used_at"] = now.isoformat()
if _should_persist_usage(key_hash, now):
config_mgr.set("api_keys", stored_keys)
return f(*args, **kwargs)
return decorated

206
api/cache.py
View file

@ -1,206 +0,0 @@
"""
Cache endpoints — browse MusicBrainz and discovery match caches.
"""
import json
from flask import request
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error, parse_pagination, build_pagination
def register_routes(bp):
# ── MusicBrainz Cache ──────────────────────────────────────
@bp.route("/cache/musicbrainz", methods=["GET"])
@require_api_key
def list_musicbrainz_cache():
"""List cached MusicBrainz lookups.
Query params:
entity_type: Filter by type ('artist', 'album', 'track')
search: Filter by entity_name
page: Page number
limit: Items per page
"""
page, limit = parse_pagination(request)
entity_type = request.args.get("entity_type")
search = request.args.get("search", "").strip()
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
where_parts = []
params = []
if entity_type:
where_parts.append("entity_type = ?")
params.append(entity_type)
if search:
where_parts.append("LOWER(entity_name) LIKE LOWER(?)")
params.append(f"%{search}%")
where_clause = f"WHERE {' AND '.join(where_parts)}" if where_parts else ""
cursor.execute(f"SELECT COUNT(*) as cnt FROM musicbrainz_cache {where_clause}", params)
total = cursor.fetchone()["cnt"]
offset = (page - 1) * limit
cursor.execute(f"""
SELECT * FROM musicbrainz_cache
{where_clause}
ORDER BY last_updated DESC
LIMIT ? OFFSET ?
""", params + [limit, offset])
entries = []
for row in cursor.fetchall():
entry = dict(row)
if entry.get("metadata_json") and isinstance(entry["metadata_json"], str):
try:
entry["metadata_json"] = json.loads(entry["metadata_json"])
except (json.JSONDecodeError, TypeError):
pass
entries.append(entry)
return api_success(
{"entries": entries},
pagination=build_pagination(page, limit, total),
)
except Exception as e:
return api_error("CACHE_ERROR", str(e), 500)
@bp.route("/cache/musicbrainz/stats", methods=["GET"])
@require_api_key
def musicbrainz_cache_stats():
"""Get MusicBrainz cache statistics."""
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
cursor.execute("SELECT COUNT(*) as total FROM musicbrainz_cache")
total = cursor.fetchone()["total"]
cursor.execute("""
SELECT entity_type, COUNT(*) as count
FROM musicbrainz_cache
GROUP BY entity_type
ORDER BY count DESC
""")
by_type = {row["entity_type"]: row["count"] for row in cursor.fetchall()}
cursor.execute("""
SELECT COUNT(*) as matched FROM musicbrainz_cache
WHERE musicbrainz_id IS NOT NULL
""")
matched = cursor.fetchone()["matched"]
return api_success({
"total": total,
"matched": matched,
"unmatched": total - matched,
"by_type": by_type,
})
except Exception as e:
return api_error("CACHE_ERROR", str(e), 500)
# ── Discovery Match Cache ──────────────────────────────────
@bp.route("/cache/discovery-matches", methods=["GET"])
@require_api_key
def list_discovery_match_cache():
"""List cached discovery provider matches.
Query params:
provider: Filter by provider ('spotify', 'itunes', etc.)
search: Filter by title or artist
page: Page number
limit: Items per page
"""
page, limit = parse_pagination(request)
provider = request.args.get("provider")
search = request.args.get("search", "").strip()
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
where_parts = []
params = []
if provider:
where_parts.append("provider = ?")
params.append(provider)
if search:
where_parts.append("(LOWER(original_title) LIKE LOWER(?) OR LOWER(original_artist) LIKE LOWER(?))")
params.extend([f"%{search}%", f"%{search}%"])
where_clause = f"WHERE {' AND '.join(where_parts)}" if where_parts else ""
cursor.execute(f"SELECT COUNT(*) as cnt FROM discovery_match_cache {where_clause}", params)
total = cursor.fetchone()["cnt"]
offset = (page - 1) * limit
cursor.execute(f"""
SELECT * FROM discovery_match_cache
{where_clause}
ORDER BY last_used_at DESC
LIMIT ? OFFSET ?
""", params + [limit, offset])
entries = []
for row in cursor.fetchall():
entry = dict(row)
if entry.get("matched_data_json") and isinstance(entry["matched_data_json"], str):
try:
entry["matched_data_json"] = json.loads(entry["matched_data_json"])
except (json.JSONDecodeError, TypeError):
pass
entries.append(entry)
return api_success(
{"entries": entries},
pagination=build_pagination(page, limit, total),
)
except Exception as e:
return api_error("CACHE_ERROR", str(e), 500)
@bp.route("/cache/discovery-matches/stats", methods=["GET"])
@require_api_key
def discovery_match_cache_stats():
"""Get discovery match cache statistics."""
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
cursor.execute("SELECT COUNT(*) as total FROM discovery_match_cache")
total = cursor.fetchone()["total"]
cursor.execute("""
SELECT provider, COUNT(*) as count
FROM discovery_match_cache
GROUP BY provider
ORDER BY count DESC
""")
by_provider = {row["provider"]: row["count"] for row in cursor.fetchall()}
cursor.execute("SELECT SUM(use_count) as total_uses FROM discovery_match_cache")
total_uses = cursor.fetchone()["total_uses"] or 0
cursor.execute("SELECT AVG(match_confidence) as avg_confidence FROM discovery_match_cache")
avg_confidence = cursor.fetchone()["avg_confidence"]
return api_success({
"total": total,
"total_uses": total_uses,
"avg_confidence": round(avg_confidence, 3) if avg_confidence else None,
"by_provider": by_provider,
})
except Exception as e:
return api_error("CACHE_ERROR", str(e), 500)

198
api/discover.py
View file

@ -1,198 +0,0 @@
"""
Discovery endpoints — browse discovery pool, similar artists, and recent releases.
"""
from flask import request
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error, parse_pagination, build_pagination, parse_fields, parse_profile_id
from .serializers import serialize_discovery_track, serialize_similar_artist, serialize_recent_release
def register_routes(bp):
@bp.route("/discover/pool", methods=["GET"])
@require_api_key
def list_discovery_pool():
"""List discovery pool tracks with optional filters.
Query params:
new_releases_only: 'true' to filter to new releases (default: false)
source: 'spotify' or 'itunes' (default: all)
limit: max tracks (default: 100, max: 500)
page: page number for pagination
"""
page, limit = parse_pagination(request, default_limit=100, max_limit=500)
new_releases_only = request.args.get("new_releases_only", "").lower() == "true"
source = request.args.get("source")
fields = parse_fields(request)
profile_id = parse_profile_id(request)
if source and source not in ("spotify", "itunes"):
return api_error("BAD_REQUEST", "source must be 'spotify' or 'itunes'.", 400)
try:
db = get_database()
# Get total count for accurate pagination
conn = db._get_connection()
cursor = conn.cursor()
count_wheres = ["profile_id = ?"]
count_params = [profile_id]
if new_releases_only:
count_wheres.append("is_new_release = 1")
if source:
count_wheres.append("source = ?")
count_params.append(source)
cursor.execute(
f"SELECT COUNT(*) as cnt FROM discovery_pool WHERE {' AND '.join(count_wheres)}",
count_params,
)
total = cursor.fetchone()["cnt"]
# Fetch page using offset/limit
offset = (page - 1) * limit
where_clauses = list(count_wheres)
params = list(count_params)
params.extend([limit, offset])
cursor.execute(f"""
SELECT * FROM discovery_pool
WHERE {' AND '.join(where_clauses)}
ORDER BY added_date DESC
LIMIT ? OFFSET ?
""", params)
rows = cursor.fetchall()
page_tracks = [dict(row) for row in rows]
return api_success(
{"tracks": [serialize_discovery_track(t, fields) for t in page_tracks]},
pagination=build_pagination(page, limit, total),
)
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)
@bp.route("/discover/similar-artists", methods=["GET"])
@require_api_key
def list_similar_artists():
"""List top similar artists discovered from the watchlist.
Query params:
limit: max artists (default: 50, max: 200)
"""
try:
limit = min(200, max(1, int(request.args.get("limit", 50))))
except (ValueError, TypeError):
limit = 50
fields = parse_fields(request)
profile_id = parse_profile_id(request)
try:
db = get_database()
artists = db.get_top_similar_artists(limit=limit, profile_id=profile_id)
return api_success({
"artists": [serialize_similar_artist(a, fields) for a in artists]
})
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)
@bp.route("/discover/recent-releases", methods=["GET"])
@require_api_key
def list_recent_releases():
"""List recent releases from watched artists.
Query params:
limit: max releases (default: 50, max: 200)
"""
try:
limit = min(200, max(1, int(request.args.get("limit", 50))))
except (ValueError, TypeError):
limit = 50
fields = parse_fields(request)
profile_id = parse_profile_id(request)
try:
db = get_database()
releases = db.get_recent_releases(limit=limit, profile_id=profile_id)
return api_success({
"releases": [serialize_recent_release(r, fields) for r in releases]
})
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)
@bp.route("/discover/pool/metadata", methods=["GET"])
@require_api_key
def discovery_pool_metadata():
"""Get discovery pool metadata (last populated timestamp, track count)."""
profile_id = parse_profile_id(request)
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT last_populated_timestamp, track_count, updated_at
FROM discovery_pool_metadata
WHERE profile_id = ?
""", (profile_id,))
row = cursor.fetchone()
if not row:
return api_success({
"last_populated": None,
"track_count": 0,
"updated_at": None,
})
return api_success({
"last_populated": row["last_populated_timestamp"],
"track_count": row["track_count"],
"updated_at": row["updated_at"],
})
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)
# ── Bubble Snapshots ───────────────────────────────────────
@bp.route("/discover/bubbles", methods=["GET"])
@require_api_key
def list_bubble_snapshots():
"""List all bubble snapshots for the current profile.
Returns snapshots for all types: artist_bubbles, search_bubbles, discover_downloads.
"""
profile_id = parse_profile_id(request)
try:
db = get_database()
result = {}
for snap_type in ("artist_bubbles", "search_bubbles", "discover_downloads"):
snapshot = db.get_bubble_snapshot(snap_type, profile_id=profile_id)
result[snap_type] = snapshot # None if not found
return api_success({"snapshots": result})
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)
@bp.route("/discover/bubbles/<snapshot_type>", methods=["GET"])
@require_api_key
def get_bubble_snapshot(snapshot_type):
"""Get a specific bubble snapshot by type.
Types: artist_bubbles, search_bubbles, discover_downloads
"""
valid_types = ("artist_bubbles", "search_bubbles", "discover_downloads")
if snapshot_type not in valid_types:
return api_error("BAD_REQUEST", f"type must be one of: {', '.join(valid_types)}", 400)
profile_id = parse_profile_id(request)
try:
db = get_database()
snapshot = db.get_bubble_snapshot(snapshot_type, profile_id=profile_id)
if not snapshot:
return api_error("NOT_FOUND", f"No '{snapshot_type}' snapshot found.", 404)
return api_success({"snapshot": snapshot})
except Exception as e:
return api_error("DISCOVER_ERROR", str(e), 500)

152
api/downloads.py
View file

@ -1,152 +0,0 @@
"""
Download management endpoints — list, cancel active downloads.
"""
from flask import request, current_app
from .auth import require_api_key
from .helpers import api_success, api_error
from core.runtime_state import download_tasks, tasks_lock
def _serialize_download(task_id, task):
"""Serialize a download task with all available fields."""
track_info = task.get("track_info") or {}
# Track names can be top-level or inside track_info
track_name = task.get("track_name") or track_info.get("title") or track_info.get("track_name")
artist_name = task.get("artist_name") or track_info.get("artist") or track_info.get("artist_name")
album_name = task.get("album_name") or track_info.get("album") or track_info.get("album_name")
return {
"id": task_id,
"status": task.get("status"),
"track_name": track_name,
"artist_name": artist_name,
"album_name": album_name,
"username": task.get("username"),
"filename": task.get("filename"),
"progress": task.get("progress", 0),
"size": task.get("size"),
"error": task.get("error") or task.get("error_message"),
"batch_id": task.get("batch_id"),
"track_index": task.get("track_index"),
"retry_count": task.get("retry_count", 0),
"metadata_enhanced": task.get("metadata_enhanced", False),
"status_change_time": task.get("status_change_time"),
}
def register_routes(bp):
@bp.route("/downloads", methods=["GET"])
@require_api_key
def list_downloads():
"""List download tasks with optional filtering and pagination.
Query params:
status: comma-separated statuses to include (e.g. "downloading,queued").
Default includes all.
limit: max tasks to return (default 100, max 500).
offset: skip the first N tasks (default 0).
Response includes `total` (post-filter count) so clients can paginate
without fetching everything. Tasks are sorted by `status_change_time`
descending so newest/in-flight tasks appear first.
"""
try:
# Parse pagination params
try:
limit = int(request.args.get("limit", 100))
except (TypeError, ValueError):
limit = 100
try:
offset = int(request.args.get("offset", 0))
except (TypeError, ValueError):
offset = 0
# Clamp to sensible bounds
limit = max(1, min(limit, 500))
offset = max(0, offset)
status_param = request.args.get("status", "").strip()
status_filter = (
{s.strip() for s in status_param.split(",") if s.strip()}
if status_param
else None
)
# Snapshot under the lock, then sort/slice outside.
with tasks_lock:
snapshot = list(download_tasks.items())
if status_filter:
snapshot = [
(tid, t) for tid, t in snapshot
if (t.get("status") or "") in status_filter
]
# Sort newest-first by status_change_time; fall back to string id
# so ordering is stable when timestamps are missing or tied.
snapshot.sort(
key=lambda item: (item[1].get("status_change_time") or "", item[0]),
reverse=True,
)
total = len(snapshot)
page = snapshot[offset:offset + limit]
tasks = [_serialize_download(tid, t) for tid, t in page]
return api_success({
"downloads": tasks,
"total": total,
"limit": limit,
"offset": offset,
})
except ImportError:
return api_error("NOT_AVAILABLE", "Download tracking not available.", 501)
except Exception as e:
return api_error("DOWNLOAD_ERROR", str(e), 500)
@bp.route("/downloads/<download_id>/cancel", methods=["POST"])
@require_api_key
def cancel_download(download_id):
"""Cancel a specific download.
Body: {"username": "..."}
"""
body = request.get_json(silent=True) or {}
username = body.get("username")
if not username:
return api_error("BAD_REQUEST", "Missing 'username' in body.", 400)
try:
from utils.async_helpers import run_async
soulseek = current_app.soulsync.get("download_orchestrator")
if not soulseek:
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
current_app.logger.info(
f"[CancelTrigger:api.public_cancel] download_id={download_id} username={username}"
)
ok = run_async(soulseek.cancel_download(download_id, username, remove=True))
if ok:
return api_success({"message": "Download cancelled."})
return api_error("CANCEL_FAILED", "Failed to cancel download.", 500)
except Exception as e:
return api_error("DOWNLOAD_ERROR", str(e), 500)
@bp.route("/downloads/cancel-all", methods=["POST"])
@require_api_key
def cancel_all_downloads():
"""Cancel all active downloads and clear completed ones."""
try:
from utils.async_helpers import run_async
soulseek = current_app.soulsync.get("download_orchestrator")
if not soulseek:
return api_error("NOT_AVAILABLE", "Soulseek client not configured.", 503)
run_async(soulseek.cancel_all_downloads())
run_async(soulseek.clear_all_completed_downloads())
return api_success({"message": "All downloads cancelled and cleared."})
except Exception as e:
return api_error("DOWNLOAD_ERROR", str(e), 500)

74
api/helpers.py
View file

@ -1,74 +0,0 @@
"""
Shared response helpers for the SoulSync public API.
"""
from typing import Optional, Set
from flask import jsonify
def api_success(data, pagination=None, status=200):
"""Wrap a successful response in the standard envelope."""
return jsonify({
"success": True,
"data": data,
"error": None,
"pagination": pagination,
}), status
def api_error(code, message, status=400):
"""Wrap an error response in the standard envelope."""
return jsonify({
"success": False,
"data": None,
"error": {"code": code, "message": message},
"pagination": None,
}), status
def build_pagination(page, limit, total):
"""Build a pagination dict from page/limit/total."""
total_pages = max(1, (total + limit - 1) // limit)
return {
"page": page,
"limit": limit,
"total": total,
"total_pages": total_pages,
"has_next": page < total_pages,
"has_prev": page > 1,
}
def parse_pagination(request, default_limit=50, max_limit=200):
"""Extract and validate page/limit from a Flask request."""
try:
page = max(1, int(request.args.get("page", 1)))
except (ValueError, TypeError):
page = 1
try:
limit = min(max_limit, max(1, int(request.args.get("limit", default_limit))))
except (ValueError, TypeError):
limit = default_limit
return page, limit
def parse_fields(request) -> Optional[Set[str]]:
"""Parse ?fields=id,name,thumb_url into a set. Returns None if not specified."""
raw = request.args.get("fields", "").strip()
if not raw:
return None
return {f.strip() for f in raw.split(",") if f.strip()}
def parse_profile_id(request, default: int = 1) -> int:
"""Extract profile_id from X-Profile-Id header or ?profile_id query param."""
try:
header = request.headers.get("X-Profile-Id")
if header:
return max(1, int(header))
param = request.args.get("profile_id")
if param:
return max(1, int(param))
except (ValueError, TypeError):
pass
return default

311
api/library.py
View file

@ -1,311 +0,0 @@
"""
Library endpoints — browse artists, albums, tracks, genres, and stats.
"""
from flask import request, current_app
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error, build_pagination, parse_pagination, parse_fields, parse_profile_id
from .serializers import serialize_artist, serialize_album, serialize_track
def register_routes(bp):
@bp.route("/library/artists", methods=["GET"])
@require_api_key
def list_artists():
"""List library artists with optional search, letter filter, and pagination."""
page, limit = parse_pagination(request)
search = request.args.get("search", "")
letter = request.args.get("letter", "all")
watchlist = request.args.get("watchlist", "all")
fields = parse_fields(request)
profile_id = parse_profile_id(request)
try:
db = get_database()
result = db.get_library_artists(
search_query=search,
letter=letter,
page=page,
limit=limit,
watchlist_filter=watchlist,
profile_id=profile_id,
)
artists = result.get("artists", [])
pag = result.get("pagination", {})
pagination = build_pagination(
page, limit, pag.get("total_count", len(artists))
)
# Artists from get_library_artists are already dicts with external IDs
serialized = [serialize_artist(a, fields) for a in artists]
return api_success({"artists": serialized}, pagination=pagination)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/artists/<artist_id>", methods=["GET"])
@require_api_key
def get_artist(artist_id):
"""Get a single artist by ID with all metadata and album list."""
fields = parse_fields(request)
try:
db = get_database()
artist = db.api_get_artist(int(artist_id))
if not artist:
return api_error("NOT_FOUND", f"Artist {artist_id} not found.", 404)
albums = db.api_get_albums_by_artist(int(artist_id))
return api_success({
"artist": serialize_artist(artist, fields),
"albums": [serialize_album(a, fields) for a in albums],
})
except ValueError:
return api_error("BAD_REQUEST", "artist_id must be an integer.", 400)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/artists/<artist_id>/albums", methods=["GET"])
@require_api_key
def get_artist_albums(artist_id):
"""List albums for an artist with full metadata."""
fields = parse_fields(request)
try:
db = get_database()
albums = db.api_get_albums_by_artist(int(artist_id))
return api_success({"albums": [serialize_album(a, fields) for a in albums]})
except ValueError:
return api_error("BAD_REQUEST", "artist_id must be an integer.", 400)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/albums", methods=["GET"])
@require_api_key
def list_albums():
"""List/search albums with pagination and optional filters."""
page, limit = parse_pagination(request)
search = request.args.get("search", "")
fields = parse_fields(request)
artist_id = request.args.get("artist_id")
year = request.args.get("year")
try:
artist_id_int = int(artist_id) if artist_id else None
except ValueError:
return api_error("BAD_REQUEST", "artist_id must be an integer.", 400)
try:
year_int = int(year) if year else None
except ValueError:
return api_error("BAD_REQUEST", "year must be an integer.", 400)
try:
db = get_database()
result = db.api_list_albums(
search=search,
artist_id=artist_id_int,
year=year_int,
page=page,
limit=limit,
)
albums = result.get("albums", [])
total = result.get("total", 0)
pagination = build_pagination(page, limit, total)
return api_success(
{"albums": [serialize_album(a, fields) for a in albums]},
pagination=pagination,
)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/albums/<album_id>", methods=["GET"])
@require_api_key
def get_album(album_id):
"""Get a single album by ID with all metadata and embedded tracks."""
fields = parse_fields(request)
try:
db = get_database()
album = db.api_get_album(int(album_id))
if not album:
return api_error("NOT_FOUND", f"Album {album_id} not found.", 404)
tracks = db.api_get_tracks_by_album(int(album_id))
return api_success({
"album": serialize_album(album, fields),
"tracks": [serialize_track(t, fields) for t in tracks],
})
except ValueError:
return api_error("BAD_REQUEST", "album_id must be an integer.", 400)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/albums/<album_id>/tracks", methods=["GET"])
@require_api_key
def get_album_tracks(album_id):
"""List tracks in an album with full metadata."""
fields = parse_fields(request)
try:
db = get_database()
tracks = db.api_get_tracks_by_album(int(album_id))
return api_success({"tracks": [serialize_track(t, fields) for t in tracks]})
except ValueError:
return api_error("BAD_REQUEST", "album_id must be an integer.", 400)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/tracks/<track_id>", methods=["GET"])
@require_api_key
def get_track(track_id):
"""Get a single track by ID with all metadata."""
fields = parse_fields(request)
try:
db = get_database()
track = db.api_get_track(int(track_id))
if not track:
return api_error("NOT_FOUND", f"Track {track_id} not found.", 404)
return api_success({"track": serialize_track(track, fields)})
except ValueError:
return api_error("BAD_REQUEST", "track_id must be an integer.", 400)
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/tracks", methods=["GET"])
@require_api_key
def library_search_tracks():
"""Search tracks by title and/or artist."""
title = request.args.get("title", "")
artist = request.args.get("artist", "")
try:
limit = min(200, max(1, int(request.args.get("limit", 50))))
except (ValueError, TypeError):
limit = 50
fields = parse_fields(request)
if not title and not artist:
return api_error("BAD_REQUEST", "Provide at least 'title' or 'artist' query param.", 400)
try:
db = get_database()
tracks = db.api_search_tracks(title=title, artist=artist, limit=limit)
return api_success({"tracks": [serialize_track(t, fields) for t in tracks]})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/genres", methods=["GET"])
@require_api_key
def list_genres():
"""List all genres with occurrence counts.
Query params:
source: 'artists' or 'albums' (default: 'artists')
"""
source = request.args.get("source", "artists")
if source not in ("artists", "albums"):
return api_error("BAD_REQUEST", "source must be 'artists' or 'albums'.", 400)
try:
db = get_database()
genres = db.api_get_genres(table=source)
return api_success({"genres": genres, "source": source})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/recently-added", methods=["GET"])
@require_api_key
def recently_added():
"""Get recently added content ordered by created_at.
Query params:
type: 'albums', 'artists', or 'tracks' (default: 'albums')
limit: max items to return (default: 50, max: 200)
"""
entity_type = request.args.get("type", "albums")
if entity_type not in ("albums", "artists", "tracks"):
return api_error("BAD_REQUEST", "type must be 'albums', 'artists', or 'tracks'.", 400)
try:
limit = min(200, max(1, int(request.args.get("limit", 50))))
except (ValueError, TypeError):
limit = 50
fields = parse_fields(request)
try:
db = get_database()
items = db.api_get_recently_added(entity_type=entity_type, limit=limit)
serializer = {
"artists": serialize_artist,
"albums": serialize_album,
"tracks": serialize_track,
}[entity_type]
return api_success({
"items": [serializer(item, fields) for item in items],
"type": entity_type,
})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/lookup", methods=["GET"])
@require_api_key
def lookup_by_external_id():
"""Look up a library entity by external provider ID.
Query params:
type: 'artist', 'album', or 'track' (required)
provider: 'spotify', 'musicbrainz', 'itunes', 'deezer', 'audiodb', 'tidal', 'qobuz', 'genius' (required)
id: the external ID value (required)
"""
entity_type = request.args.get("type")
provider = request.args.get("provider")
external_id = request.args.get("id")
fields = parse_fields(request)
if not entity_type or not provider or not external_id:
return api_error("BAD_REQUEST", "Required params: type, provider, id.", 400)
table_map = {"artist": "artists", "album": "albums", "track": "tracks"}
table = table_map.get(entity_type)
if not table:
return api_error("BAD_REQUEST", "type must be 'artist', 'album', or 'track'.", 400)
# genius only exists on artists and tracks, not albums
valid_providers = ("spotify", "musicbrainz", "itunes", "deezer", "audiodb", "tidal", "qobuz", "genius")
if provider not in valid_providers:
return api_error("BAD_REQUEST", f"provider must be one of: {', '.join(valid_providers)}.", 400)
if provider == "genius" and entity_type == "album":
return api_error("BAD_REQUEST", "Genius IDs are not available for albums. Use artist or track.", 400)
try:
db = get_database()
result = db.api_lookup_by_external_id(table, provider, external_id)
if not result:
return api_error("NOT_FOUND", f"No {entity_type} found for {provider} ID: {external_id}", 404)
serializer = {
"artists": serialize_artist,
"albums": serialize_album,
"tracks": serialize_track,
}[table]
return api_success({entity_type: serializer(result, fields)})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)
@bp.route("/library/stats", methods=["GET"])
@require_api_key
def library_stats():
"""Get library statistics (artist/album/track counts, DB info)."""
try:
db = get_database()
info = db.get_database_info_for_server()
stats = db.get_statistics_for_server()
return api_success({
"artists": stats.get("artists", 0),
"albums": stats.get("albums", 0),
"tracks": stats.get("tracks", 0),
"database_size_mb": info.get("database_size_mb"),
"last_update": info.get("last_update"),
})
except Exception as e:
return api_error("LIBRARY_ERROR", str(e), 500)

124
api/listenbrainz.py
View file

@ -1,124 +0,0 @@
"""
ListenBrainz endpoints — browse cached ListenBrainz playlists and tracks.
"""
import json
from flask import request
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error, parse_pagination, build_pagination
def register_routes(bp):
@bp.route("/listenbrainz/playlists", methods=["GET"])
@require_api_key
def list_listenbrainz_playlists():
"""List cached ListenBrainz playlists.
Query params:
type: Filter by playlist_type (e.g. 'weekly-jams', 'weekly-exploration')
page: Page number
limit: Items per page
"""
page, limit = parse_pagination(request)
playlist_type = request.args.get("type")
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
where_parts = []
params = []
if playlist_type:
where_parts.append("playlist_type = ?")
params.append(playlist_type)
where_clause = f"WHERE {' AND '.join(where_parts)}" if where_parts else ""
# Count
cursor.execute(f"SELECT COUNT(*) as cnt FROM listenbrainz_playlists {where_clause}", params)
total = cursor.fetchone()["cnt"]
# Fetch page
offset = (page - 1) * limit
cursor.execute(f"""
SELECT * FROM listenbrainz_playlists
{where_clause}
ORDER BY last_updated DESC
LIMIT ? OFFSET ?
""", params + [limit, offset])
playlists = []
for row in cursor.fetchall():
p = dict(row)
# Parse annotation_data if it's JSON
if p.get("annotation_data") and isinstance(p["annotation_data"], str):
try:
p["annotation_data"] = json.loads(p["annotation_data"])
except (json.JSONDecodeError, TypeError):
pass
playlists.append(p)
return api_success(
{"playlists": playlists},
pagination=build_pagination(page, limit, total),
)
except Exception as e:
return api_error("LISTENBRAINZ_ERROR", str(e), 500)
@bp.route("/listenbrainz/playlists/<playlist_id>", methods=["GET"])
@require_api_key
def get_listenbrainz_playlist(playlist_id):
"""Get a ListenBrainz playlist with its tracks.
playlist_id can be the internal ID or the MusicBrainz playlist MBID.
"""
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
# Try by internal ID first, then by MBID
try:
int_id = int(playlist_id)
cursor.execute("SELECT * FROM listenbrainz_playlists WHERE id = ?", (int_id,))
except ValueError:
cursor.execute("SELECT * FROM listenbrainz_playlists WHERE playlist_mbid = ?", (playlist_id,))
row = cursor.fetchone()
if not row:
return api_error("NOT_FOUND", f"ListenBrainz playlist '{playlist_id}' not found.", 404)
playlist = dict(row)
if playlist.get("annotation_data") and isinstance(playlist["annotation_data"], str):
try:
playlist["annotation_data"] = json.loads(playlist["annotation_data"])
except (json.JSONDecodeError, TypeError):
pass
# Get tracks
cursor.execute("""
SELECT * FROM listenbrainz_tracks
WHERE playlist_id = ?
ORDER BY position ASC
""", (playlist["id"],))
tracks = []
for t_row in cursor.fetchall():
track = dict(t_row)
if track.get("additional_metadata") and isinstance(track["additional_metadata"], str):
try:
track["additional_metadata"] = json.loads(track["additional_metadata"])
except (json.JSONDecodeError, TypeError):
pass
tracks.append(track)
return api_success({
"playlist": playlist,
"tracks": tracks,
})
except Exception as e:
return api_error("LISTENBRAINZ_ERROR", str(e), 500)

152
api/playlists.py
View file

@ -1,152 +0,0 @@
"""
Playlist endpoints — list and inspect playlists from Spotify/Tidal.
"""
from flask import request, current_app
from .auth import require_api_key
from .helpers import api_success, api_error
def register_routes(bp):
@bp.route("/playlists", methods=["GET"])
@require_api_key
def list_playlists():
"""List user playlists from Spotify or Tidal.
Query: ?source=spotify|tidal (default: spotify)
"""
source = request.args.get("source", "spotify")
ctx = current_app.soulsync
try:
if source == "spotify":
spotify = ctx.get("spotify_client")
if not spotify or not spotify.is_authenticated():
return api_error("NOT_AUTHENTICATED", "Spotify not authenticated.", 401)
playlists = spotify.get_user_playlists_metadata_only()
return api_success({
"playlists": [
{
"id": p.id,
"name": p.name,
"owner": p.owner,
"track_count": p.total_tracks,
"image_url": getattr(p, "image_url", None),
}
for p in playlists
],
"source": "spotify",
})
elif source == "tidal":
tidal = ctx.get("tidal_client")
if not tidal:
return api_error("NOT_AVAILABLE", "Tidal client not configured.", 503)
playlists = tidal.get_user_playlists_metadata_only()
return api_success({
"playlists": [
{
"id": p.get("id") or p.get("uuid"),
"name": p.get("title") or p.get("name"),
"track_count": p.get("numberOfTracks", 0),
"image_url": p.get("image"),
}
for p in (playlists or [])
],
"source": "tidal",
})
return api_error("BAD_REQUEST", "source must be 'spotify' or 'tidal'.", 400)
except Exception as e:
return api_error("PLAYLIST_ERROR", str(e), 500)
@bp.route("/playlists/<playlist_id>", methods=["GET"])
@require_api_key
def get_playlist(playlist_id):
"""Get playlist details with tracks.
Query: ?source=spotify (default: spotify)
"""
source = request.args.get("source", "spotify")
ctx = current_app.soulsync
try:
if source == "spotify":
spotify = ctx.get("spotify_client")
if not spotify or not spotify.is_authenticated():
return api_error("NOT_AUTHENTICATED", "Spotify not authenticated.", 401)
playlist = spotify.get_playlist_by_id(playlist_id)
if not playlist:
return api_error("NOT_FOUND", "Playlist not found.", 404)
tracks = []
for item in playlist.get("tracks", {}).get("items", []):
t = item.get("track")
if not t:
continue
tracks.append({
"id": t.get("id"),
"name": t.get("name"),
"artists": [a.get("name") for a in t.get("artists", [])],
"album": t.get("album", {}).get("name"),
"duration_ms": t.get("duration_ms"),
"image_url": (t.get("album", {}).get("images", [{}])[0].get("url")
if t.get("album", {}).get("images") else None),
})
return api_success({
"playlist": {
"id": playlist.get("id"),
"name": playlist.get("name"),
"owner": playlist.get("owner", {}).get("display_name"),
"total_tracks": playlist.get("tracks", {}).get("total", len(tracks)),
"tracks": tracks,
},
"source": "spotify",
})
return api_error("BAD_REQUEST", "source must be 'spotify'.", 400)
except Exception as e:
return api_error("PLAYLIST_ERROR", str(e), 500)
@bp.route("/playlists/<playlist_id>/sync", methods=["POST"])
@require_api_key
def sync_playlist(playlist_id):
"""Trigger playlist sync/download.
This delegates to the internal sync endpoint by forwarding the request.
Body: {"playlist_name": "...", "tracks": [...]}
"""
body = request.get_json(silent=True) or {}
playlist_name = body.get("playlist_name")
tracks = body.get("tracks")
if not playlist_name or not tracks:
return api_error("BAD_REQUEST", "Missing 'playlist_name' or 'tracks' in body.", 400)
try:
from web_server import sync_states
if playlist_id in sync_states and sync_states[playlist_id].get("phase") not in ("complete", "error", None):
return api_error("CONFLICT", "Sync already in progress for this playlist.", 409)
except ImportError:
pass
try:
# Forward to the internal sync endpoint
import requests as http_requests
internal_url = "http://127.0.0.1:8008/api/sync/start"
resp = http_requests.post(internal_url, json={
"playlist_id": playlist_id,
"playlist_name": playlist_name,
"tracks": tracks,
}, timeout=10)
data = resp.json()
if data.get("success"):
return api_success({"message": "Playlist sync started.", "playlist_id": playlist_id})
return api_error("SYNC_FAILED", data.get("error", "Sync failed to start."), 500)
except Exception as e:
return api_error("PLAYLIST_ERROR", str(e), 500)

130
api/profiles.py
View file

@ -1,130 +0,0 @@
"""
Profile management endpoints — list, create, update, delete profiles.
"""
from flask import request
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error
def register_routes(bp):
@bp.route("/profiles", methods=["GET"])
@require_api_key
def list_profiles():
"""List all profiles."""
try:
db = get_database()
profiles = db.get_all_profiles()
return api_success({"profiles": profiles})
except Exception as e:
return api_error("PROFILE_ERROR", str(e), 500)
@bp.route("/profiles/<int:profile_id>", methods=["GET"])
@require_api_key
def get_profile(profile_id):
"""Get a single profile by ID."""
try:
db = get_database()
profile = db.get_profile(profile_id)
if not profile:
return api_error("NOT_FOUND", f"Profile {profile_id} not found.", 404)
return api_success({"profile": profile})
except Exception as e:
return api_error("PROFILE_ERROR", str(e), 500)
@bp.route("/profiles", methods=["POST"])
@require_api_key
def create_profile():
"""Create a new profile.
Body: {"name": "...", "avatar_color": "#hex", "avatar_url": "...", "is_admin": false}
"""
body = request.get_json(silent=True) or {}
name = body.get("name", "").strip()
if not name:
return api_error("BAD_REQUEST", "Missing 'name' in body.", 400)
avatar_color = body.get("avatar_color", "#6366f1")
avatar_url = body.get("avatar_url")
is_admin = bool(body.get("is_admin", False))
# Handle optional PIN
pin_hash = None
pin = body.get("pin")
if pin:
from werkzeug.security import generate_password_hash
pin_hash = generate_password_hash(pin, method="pbkdf2:sha256")
try:
db = get_database()
profile_id = db.create_profile(
name=name,
avatar_color=avatar_color,
pin_hash=pin_hash,
is_admin=is_admin,
avatar_url=avatar_url,
)
if profile_id:
profile = db.get_profile(profile_id)
return api_success({"profile": profile}, status=201)
return api_error("CONFLICT", "Profile name already exists.", 409)
except Exception as e:
return api_error("PROFILE_ERROR", str(e), 500)
@bp.route("/profiles/<int:profile_id>", methods=["PUT"])
@require_api_key
def update_profile(profile_id):
"""Update a profile.
Body: {"name": "...", "avatar_color": "#hex", "avatar_url": "...", "is_admin": false}
"""
body = request.get_json(silent=True) or {}
kwargs = {}
if "name" in body:
kwargs["name"] = body["name"].strip()
if "avatar_color" in body:
kwargs["avatar_color"] = body["avatar_color"]
if "avatar_url" in body:
kwargs["avatar_url"] = body["avatar_url"]
if "is_admin" in body:
kwargs["is_admin"] = int(bool(body["is_admin"]))
if "pin" in body:
pin = body["pin"]
if pin:
from werkzeug.security import generate_password_hash
kwargs["pin_hash"] = generate_password_hash(pin, method="pbkdf2:sha256")
else:
kwargs["pin_hash"] = None # Clear PIN
if not kwargs:
return api_error("BAD_REQUEST", "No valid fields to update.", 400)
try:
db = get_database()
ok = db.update_profile(profile_id, **kwargs)
if ok:
profile = db.get_profile(profile_id)
return api_success({"profile": profile})
return api_error("NOT_FOUND", f"Profile {profile_id} not found.", 404)
except Exception as e:
return api_error("PROFILE_ERROR", str(e), 500)
@bp.route("/profiles/<int:profile_id>", methods=["DELETE"])
@require_api_key
def delete_profile(profile_id):
"""Delete a profile and all its data. Cannot delete profile 1 (admin)."""
if profile_id == 1:
return api_error("FORBIDDEN", "Cannot delete the default admin profile.", 403)
try:
db = get_database()
ok = db.delete_profile(profile_id)
if ok:
return api_success({"message": f"Profile {profile_id} deleted."})
return api_error("NOT_FOUND", f"Profile {profile_id} not found.", 404)
except Exception as e:
return api_error("PROFILE_ERROR", str(e), 500)

237
api/request.py
View file

@ -1,237 +0,0 @@
"""
Inbound music request endpoint — accept a search query from external sources
(Discord bots, curl, etc.) and trigger the search-match-download pipeline.
"""
import threading
import uuid
from datetime import datetime, timedelta
import requests as http_requests
from flask import request, current_app
from utils.logging_config import get_logger
from .auth import require_api_key
from .helpers import api_success, api_error
logger = get_logger("api_request")
# In-memory request tracking (ephemeral — survives until restart)
_pending_requests = {}
_requests_lock = threading.Lock()
# Max age before auto-cleanup
_MAX_REQUEST_AGE = timedelta(hours=1)
# How often the background cleanup timer runs. Short enough to keep memory
# bounded during idle periods, long enough that slow-polling external clients
# still see their request for close to the TTL.
_CLEANUP_INTERVAL_SECONDS = 300 # 5 minutes
# Guards for the singleton background cleanup thread.
_cleanup_thread: "threading.Thread | None" = None
_cleanup_stop_event = threading.Event()
_cleanup_thread_lock = threading.Lock()
def _cleanup_old_requests():
"""Remove requests older than 1 hour to prevent unbounded growth."""
cutoff = datetime.now() - _MAX_REQUEST_AGE
with _requests_lock:
expired = [rid for rid, r in _pending_requests.items()
if r.get('created_at', datetime.now()) < cutoff]
for rid in expired:
del _pending_requests[rid]
return len(expired) if expired else 0
def _cleanup_loop():
"""Background thread: periodically evict expired requests."""
while not _cleanup_stop_event.is_set():
# wait() returns True if the event was set (shutdown), False on timeout
if _cleanup_stop_event.wait(timeout=_CLEANUP_INTERVAL_SECONDS):
return
try:
removed = _cleanup_old_requests()
if removed:
logger.debug(f"Request cleanup: evicted {removed} stale entries")
except Exception as e:
logger.warning(f"Request cleanup loop error: {e}")
def start_cleanup_thread() -> bool:
"""Start the background cleanup timer once per process.
Returns True if a new thread was started, False if one was already
running. Safe to call multiple times; callers in multi-worker setups
should still gate on worker identity if they want exactly one thread
across the entire deployment.
"""
global _cleanup_thread
with _cleanup_thread_lock:
if _cleanup_thread is not None and _cleanup_thread.is_alive():
return False
_cleanup_stop_event.clear()
_cleanup_thread = threading.Thread(
target=_cleanup_loop,
name="api-request-cleanup",
daemon=True,
)
_cleanup_thread.start()
logger.info("Started api/request cleanup timer (interval=%ss)" % _CLEANUP_INTERVAL_SECONDS)
return True
def stop_cleanup_thread(timeout: float = 2.0) -> None:
"""Signal the cleanup thread to exit. Used in tests and shutdown paths."""
global _cleanup_thread
with _cleanup_thread_lock:
thread = _cleanup_thread
_cleanup_stop_event.set()
if thread is not None and thread.is_alive():
thread.join(timeout=timeout)
with _cleanup_thread_lock:
_cleanup_thread = None
_cleanup_stop_event.clear()
def _run_search_and_download(request_id, query, notify_url):
"""Background worker: search, download, update status, notify."""
try:
from utils.async_helpers import run_async
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'searching'
soulseek = current_app._get_current_object().soulsync.get('download_orchestrator')
if not soulseek:
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'failed'
_pending_requests[request_id]['error'] = 'Download source not configured'
return
result = run_async(soulseek.search_and_download_best(query))
with _requests_lock:
if request_id in _pending_requests:
if result:
_pending_requests[request_id]['status'] = 'downloading'
_pending_requests[request_id]['download_id'] = result
else:
_pending_requests[request_id]['status'] = 'not_found'
_pending_requests[request_id]['error'] = 'No match found'
_pending_requests[request_id]['completed_at'] = datetime.now().isoformat()
# Send notification to callback URL if provided
if notify_url:
try:
with _requests_lock:
payload = dict(_pending_requests.get(request_id, {}))
# Remove non-serializable datetime
payload.pop('created_at', None)
http_requests.post(notify_url, json=payload, timeout=10)
except Exception as e:
logger.warning(f"Failed to POST to notify_url {notify_url}: {e}")
except Exception as e:
logger.error(f"Request {request_id} failed: {e}")
with _requests_lock:
if request_id in _pending_requests:
_pending_requests[request_id]['status'] = 'failed'
_pending_requests[request_id]['error'] = str(e)
_pending_requests[request_id]['completed_at'] = datetime.now().isoformat()
def register_routes(bp):
@bp.route("/request", methods=["POST"])
@require_api_key
def create_request():
"""Accept a music search query and trigger the download pipeline.
Body:
query (str, required): Search query, e.g. "Artist - Track Name"
notify_url (str, optional): URL to POST results to on completion
metadata (dict, optional): Passthrough data included in automation events
Returns 202 with request_id for async status polling.
"""
body = request.get_json(silent=True) or {}
query = (body.get("query") or "").strip()
if not query:
return api_error("BAD_REQUEST", "Missing 'query' in request body.", 400)
# Cleanup old requests on each new request
_cleanup_old_requests()
request_id = str(uuid.uuid4())
notify_url = (body.get("notify_url") or "").strip() or None
metadata = body.get("metadata") or {}
with _requests_lock:
_pending_requests[request_id] = {
'request_id': request_id,
'query': query,
'status': 'queued',
'created_at': datetime.now(),
'completed_at': None,
'download_id': None,
'error': None,
}
# Emit webhook_received event so automation engine triggers fire
engine = current_app.soulsync.get('automation_engine')
if engine:
engine.emit('webhook_received', {
'query': query,
'request_id': request_id,
'source': 'api',
'metadata': metadata,
})
# Start background search-download (Feature A: works without automations)
app = current_app._get_current_object()
thread = threading.Thread(
target=lambda: _run_with_app_context(app, request_id, query, notify_url),
daemon=True
)
thread.start()
logger.info(f"Music request queued: '{query}' (id={request_id})")
return api_success({
"request_id": request_id,
"status": "queued",
"query": query,
}), 202
@bp.route("/request/<request_id>", methods=["GET"])
@require_api_key
def get_request_status(request_id):
"""Check the status of a music request.
Returns current status: queued → searching → downloading → completed/not_found/failed
"""
with _requests_lock:
req = _pending_requests.get(request_id)
if not req:
return api_error("NOT_FOUND", "Request not found or expired.", 404)
return api_success({
"request_id": req['request_id'],
"query": req['query'],
"status": req['status'],
"download_id": req.get('download_id'),
"error": req.get('error'),
"completed_at": req.get('completed_at'),
})
def _run_with_app_context(app, request_id, query, notify_url):
"""Run the background worker within Flask app context."""
with app.app_context():
_run_search_and_download(request_id, query, notify_url)

77
api/retag.py
View file

@ -1,77 +0,0 @@
"""
Retag queue endpoints — view and manage pending metadata corrections.
"""
from flask import request
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error
def register_routes(bp):
@bp.route("/retag/groups", methods=["GET"])
@require_api_key
def list_retag_groups():
"""List all retag groups with track counts."""
try:
db = get_database()
groups = db.get_retag_groups()
return api_success({"groups": groups})
except Exception as e:
return api_error("RETAG_ERROR", str(e), 500)
@bp.route("/retag/groups/<int:group_id>", methods=["GET"])
@require_api_key
def get_retag_group(group_id):
"""Get a retag group with its tracks."""
try:
db = get_database()
# Get group info
groups = db.get_retag_groups()
group = next((g for g in groups if g["id"] == group_id), None)
if not group:
return api_error("NOT_FOUND", f"Retag group {group_id} not found.", 404)
tracks = db.get_retag_tracks(group_id)
return api_success({
"group": group,
"tracks": tracks,
})
except Exception as e:
return api_error("RETAG_ERROR", str(e), 500)
@bp.route("/retag/groups/<int:group_id>", methods=["DELETE"])
@require_api_key
def delete_retag_group(group_id):
"""Delete a retag group and its tracks."""
try:
db = get_database()
ok = db.delete_retag_group(group_id)
if ok:
return api_success({"message": f"Retag group {group_id} deleted."})
return api_error("NOT_FOUND", f"Retag group {group_id} not found.", 404)
except Exception as e:
return api_error("RETAG_ERROR", str(e), 500)
@bp.route("/retag/groups", methods=["DELETE"])
@require_api_key
def clear_retag_groups():
"""Delete all retag groups and tracks."""
try:
db = get_database()
count = db.clear_all_retag_groups()
return api_success({"message": f"Cleared {count} retag groups."})
except Exception as e:
return api_error("RETAG_ERROR", str(e), 500)
@bp.route("/retag/stats", methods=["GET"])
@require_api_key
def retag_stats():
"""Get retag queue statistics."""
try:
db = get_database()
stats = db.get_retag_stats()
return api_success(stats)
except Exception as e:
return api_error("RETAG_ERROR", str(e), 500)

180
api/search.py
View file

@ -1,180 +0,0 @@
"""
Search endpoints — search external sources (Spotify, iTunes, Hydrabase).
"""
import logging
from flask import request, current_app
from .auth import require_api_key
from .helpers import api_success, api_error
logger = logging.getLogger(__name__)
def register_routes(bp):
@bp.route("/search/tracks", methods=["POST"])
@require_api_key
def search_tracks():
"""Search for tracks across music sources.
Body: {"query": "...", "source": "spotify"|"itunes"|"auto", "limit": 20}
"""
body = request.get_json(silent=True) or {}
query = body.get("query", "").strip()
source = body.get("source", "auto")
limit = min(50, max(1, int(body.get("limit", 20))))
if not query:
return api_error("BAD_REQUEST", "Missing 'query' in request body.", 400)
try:
ctx = current_app.soulsync
tracks = []
# Hydrabase first when active
hydrabase = ctx.get("hydrabase_client")
if source == "auto" and hydrabase:
try:
from web_server import _is_hydrabase_active
if _is_hydrabase_active():
hydra_results = hydrabase.search_tracks(query, limit=limit)
if hydra_results:
tracks = [_serialize_track(t) for t in hydra_results]
return api_success({"tracks": tracks, "source": "hydrabase"})
except Exception as e:
logger.debug("hydrabase search failed: %s", e)
spotify = ctx.get("spotify_client")
from core.metadata_service import get_primary_source, get_primary_client
primary = get_primary_source()
if source in ("spotify", "auto") and primary == 'spotify' and spotify and spotify.is_spotify_authenticated():
results = spotify.search_tracks(query, limit=limit)
if results:
tracks = [_serialize_track(t) for t in results]
return api_success({"tracks": tracks, "source": "spotify"})
if source in ("itunes", "deezer", "auto"):
fallback = get_primary_client()
fallback_source = get_primary_source()
results = fallback.search_tracks(query, limit=limit)
if results:
tracks = [_serialize_track(t) for t in results]
return api_success({"tracks": tracks, "source": fallback_source})
return api_success({"tracks": [], "source": source})
except Exception as e:
return api_error("SEARCH_ERROR", str(e), 500)
@bp.route("/search/albums", methods=["POST"])
@require_api_key
def search_albums():
"""Search for albums.
Body: {"query": "...", "limit": 20}
"""
body = request.get_json(silent=True) or {}
query = body.get("query", "").strip()
limit = min(50, max(1, int(body.get("limit", 20))))
if not query:
return api_error("BAD_REQUEST", "Missing 'query' in request body.", 400)
try:
ctx = current_app.soulsync
spotify = ctx.get("spotify_client")
from core.metadata_service import get_primary_source, get_primary_client
primary = get_primary_source()
if primary == 'spotify' and spotify and spotify.is_spotify_authenticated():
results = spotify.search_albums(query, limit=limit)
if results:
return api_success({
"albums": [_serialize_album(a) for a in results],
"source": "spotify",
})
fallback = get_primary_client()
fallback_source = get_primary_source()
results = fallback.search_albums(query, limit=limit)
return api_success({
"albums": [_serialize_album(a) for a in results] if results else [],
"source": fallback_source,
})
except Exception as e:
return api_error("SEARCH_ERROR", str(e), 500)
@bp.route("/search/artists", methods=["POST"])
@require_api_key
def search_artists():
"""Search for artists.
Body: {"query": "...", "limit": 20}
"""
body = request.get_json(silent=True) or {}
query = body.get("query", "").strip()
limit = min(50, max(1, int(body.get("limit", 20))))
if not query:
return api_error("BAD_REQUEST", "Missing 'query' in request body.", 400)
try:
ctx = current_app.soulsync
spotify = ctx.get("spotify_client")
from core.metadata_service import get_primary_source, get_primary_client
primary = get_primary_source()
if primary == 'spotify' and spotify and spotify.is_spotify_authenticated():
results = spotify.search_artists(query, limit=limit)
if results:
return api_success({
"artists": [_serialize_artist(a) for a in results],
"source": "spotify",
})
fallback = get_primary_client()
fallback_source = get_primary_source()
results = fallback.search_artists(query, limit=limit)
return api_success({
"artists": [_serialize_artist(a) for a in results] if results else [],
"source": fallback_source,
})
except Exception as e:
return api_error("SEARCH_ERROR", str(e), 500)
# ---- serialization (from core dataclasses) ----
def _serialize_track(t):
return {
"id": t.id,
"name": t.name,
"artists": t.artists,
"album": t.album,
"duration_ms": t.duration_ms,
"popularity": t.popularity,
"preview_url": t.preview_url,
"image_url": t.image_url,
"release_date": t.release_date,
}
def _serialize_album(a):
return {
"id": a.id,
"name": a.name,
"artists": a.artists,
"release_date": a.release_date,
"total_tracks": a.total_tracks,
"album_type": a.album_type,
"image_url": a.image_url,
}
def _serialize_artist(a):
return {
"id": a.id,
"name": a.name,
"popularity": a.popularity,
"genres": a.genres,
"followers": a.followers,
"image_url": a.image_url,
}

396
api/serializers.py
View file

@ -1,396 +0,0 @@
"""
Centralized serializers for the SoulSync API v1.
All serializers accept a sqlite3.Row, a dict, or a dataclass instance
and normalize the output to a plain dict. This allows the same serializer
to be used whether the data comes from raw queries or existing methods.
"""
import json
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
def _to_dict(obj) -> dict:
"""Convert a sqlite3.Row, dataclass, or dict to a plain dict."""
if isinstance(obj, dict):
return obj
if hasattr(obj, "keys"): # sqlite3.Row
return {k: obj[k] for k in obj.keys()}
if hasattr(obj, "__dataclass_fields__"):
from dataclasses import asdict
return asdict(obj)
raise TypeError(f"Cannot serialize {type(obj)}")
def _parse_genres(raw) -> list:
"""Parse genres from JSON string, list, or comma-separated string."""
if isinstance(raw, list):
return raw
if isinstance(raw, str):
try:
parsed = json.loads(raw)
return parsed if isinstance(parsed, list) else []
except (json.JSONDecodeError, TypeError):
return [g.strip() for g in raw.split(",") if g.strip()]
return []
def _isoformat(val) -> Optional[str]:
"""Safely convert datetime or string to ISO format string."""
if val is None:
return None
if isinstance(val, datetime):
return val.isoformat()
if isinstance(val, str):
return val
return str(val)
def _bool_or_none(val):
"""Convert to bool, returning None if val is None."""
if val is None:
return None
return bool(val)
def filter_fields(data: dict, fields: Optional[Set[str]]) -> dict:
"""If fields set is provided, return only those keys."""
if not fields:
return data
return {k: v for k, v in data.items() if k in fields}
# ── Library Entity Serializers ────────────────────────────────
def serialize_artist(obj, fields: Optional[Set[str]] = None) -> dict:
"""Full artist serialization — all columns."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"name": d.get("name"),
"thumb_url": d.get("thumb_url"),
"banner_url": d.get("banner_url"),
"genres": _parse_genres(d.get("genres")),
"summary": d.get("summary"),
"style": d.get("style"),
"mood": d.get("mood"),
"label": d.get("label"),
"server_source": d.get("server_source"),
"created_at": _isoformat(d.get("created_at")),
"updated_at": _isoformat(d.get("updated_at")),
# External IDs
"musicbrainz_id": d.get("musicbrainz_id"),
"spotify_artist_id": d.get("spotify_artist_id"),
"itunes_artist_id": d.get("itunes_artist_id"),
"audiodb_id": d.get("audiodb_id"),
"deezer_id": d.get("deezer_id"),
"tidal_id": d.get("tidal_id"),
"qobuz_id": d.get("qobuz_id"),
"genius_id": d.get("genius_id"),
# Match statuses
"musicbrainz_match_status": d.get("musicbrainz_match_status"),
"spotify_match_status": d.get("spotify_match_status"),
"itunes_match_status": d.get("itunes_match_status"),
"audiodb_match_status": d.get("audiodb_match_status"),
"deezer_match_status": d.get("deezer_match_status"),
"lastfm_match_status": d.get("lastfm_match_status"),
"genius_match_status": d.get("genius_match_status"),
"tidal_match_status": d.get("tidal_match_status"),
"qobuz_match_status": d.get("qobuz_match_status"),
# Last attempted timestamps
"musicbrainz_last_attempted": _isoformat(d.get("musicbrainz_last_attempted")),
"spotify_last_attempted": _isoformat(d.get("spotify_last_attempted")),
"itunes_last_attempted": _isoformat(d.get("itunes_last_attempted")),
"audiodb_last_attempted": _isoformat(d.get("audiodb_last_attempted")),
"deezer_last_attempted": _isoformat(d.get("deezer_last_attempted")),
"lastfm_last_attempted": _isoformat(d.get("lastfm_last_attempted")),
"genius_last_attempted": _isoformat(d.get("genius_last_attempted")),
"tidal_last_attempted": _isoformat(d.get("tidal_last_attempted")),
"qobuz_last_attempted": _isoformat(d.get("qobuz_last_attempted")),
# Last.fm metadata
"lastfm_listeners": d.get("lastfm_listeners"),
"lastfm_playcount": d.get("lastfm_playcount"),
"lastfm_tags": d.get("lastfm_tags"),
"lastfm_similar": d.get("lastfm_similar"),
"lastfm_bio": d.get("lastfm_bio"),
"lastfm_url": d.get("lastfm_url"),
# Genius metadata
"genius_description": d.get("genius_description"),
"genius_alt_names": d.get("genius_alt_names"),
"genius_url": d.get("genius_url"),
}
# Preserve extra keys from enriched queries (album_count, track_count, is_watched)
for extra_key in ("album_count", "track_count", "is_watched", "image_url"):
if extra_key in d:
result[extra_key] = d[extra_key]
return filter_fields(result, fields)
def serialize_album(obj, fields: Optional[Set[str]] = None) -> dict:
"""Full album serialization — all columns."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"artist_id": d.get("artist_id"),
"title": d.get("title"),
"year": d.get("year"),
"thumb_url": d.get("thumb_url"),
"genres": _parse_genres(d.get("genres")),
"track_count": d.get("track_count"),
"duration": d.get("duration"),
"style": d.get("style"),
"mood": d.get("mood"),
"label": d.get("label"),
"explicit": _bool_or_none(d.get("explicit")),
"record_type": d.get("record_type"),
"server_source": d.get("server_source"),
"created_at": _isoformat(d.get("created_at")),
"updated_at": _isoformat(d.get("updated_at")),
"upc": d.get("upc"),
"copyright": d.get("copyright"),
# External IDs
"musicbrainz_release_id": d.get("musicbrainz_release_id"),
"spotify_album_id": d.get("spotify_album_id"),
"itunes_album_id": d.get("itunes_album_id"),
"audiodb_id": d.get("audiodb_id"),
"deezer_id": d.get("deezer_id"),
"tidal_id": d.get("tidal_id"),
"qobuz_id": d.get("qobuz_id"),
# Match statuses
"musicbrainz_match_status": d.get("musicbrainz_match_status"),
"spotify_match_status": d.get("spotify_match_status"),
"itunes_match_status": d.get("itunes_match_status"),
"audiodb_match_status": d.get("audiodb_match_status"),
"deezer_match_status": d.get("deezer_match_status"),
"lastfm_match_status": d.get("lastfm_match_status"),
"tidal_match_status": d.get("tidal_match_status"),
"qobuz_match_status": d.get("qobuz_match_status"),
# Last attempted timestamps
"musicbrainz_last_attempted": _isoformat(d.get("musicbrainz_last_attempted")),
"spotify_last_attempted": _isoformat(d.get("spotify_last_attempted")),
"itunes_last_attempted": _isoformat(d.get("itunes_last_attempted")),
"audiodb_last_attempted": _isoformat(d.get("audiodb_last_attempted")),
"deezer_last_attempted": _isoformat(d.get("deezer_last_attempted")),
"lastfm_last_attempted": _isoformat(d.get("lastfm_last_attempted")),
"tidal_last_attempted": _isoformat(d.get("tidal_last_attempted")),
"qobuz_last_attempted": _isoformat(d.get("qobuz_last_attempted")),
# Last.fm metadata
"lastfm_listeners": d.get("lastfm_listeners"),
"lastfm_playcount": d.get("lastfm_playcount"),
"lastfm_tags": d.get("lastfm_tags"),
"lastfm_wiki": d.get("lastfm_wiki"),
"lastfm_url": d.get("lastfm_url"),
}
return filter_fields(result, fields)
def serialize_track(obj, fields: Optional[Set[str]] = None) -> dict:
"""Full track serialization — all columns."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"album_id": d.get("album_id"),
"artist_id": d.get("artist_id"),
"title": d.get("title"),
"track_number": d.get("track_number"),
"duration": d.get("duration"),
"file_path": d.get("file_path"),
"bitrate": d.get("bitrate"),
"bpm": d.get("bpm"),
"explicit": _bool_or_none(d.get("explicit")),
"style": d.get("style"),
"mood": d.get("mood"),
"repair_status": d.get("repair_status"),
"repair_last_checked": _isoformat(d.get("repair_last_checked")),
"server_source": d.get("server_source"),
"created_at": _isoformat(d.get("created_at")),
"updated_at": _isoformat(d.get("updated_at")),
"isrc": d.get("isrc"),
"copyright": d.get("copyright"),
# External IDs
"musicbrainz_recording_id": d.get("musicbrainz_recording_id"),
"spotify_track_id": d.get("spotify_track_id"),
"itunes_track_id": d.get("itunes_track_id"),
"audiodb_id": d.get("audiodb_id"),
"deezer_id": d.get("deezer_id"),
"tidal_id": d.get("tidal_id"),
"qobuz_id": d.get("qobuz_id"),
"genius_id": d.get("genius_id"),
# Match statuses
"musicbrainz_match_status": d.get("musicbrainz_match_status"),
"spotify_match_status": d.get("spotify_match_status"),
"itunes_match_status": d.get("itunes_match_status"),
"audiodb_match_status": d.get("audiodb_match_status"),
"deezer_match_status": d.get("deezer_match_status"),
"lastfm_match_status": d.get("lastfm_match_status"),
"genius_match_status": d.get("genius_match_status"),
"tidal_match_status": d.get("tidal_match_status"),
"qobuz_match_status": d.get("qobuz_match_status"),
# Last attempted timestamps
"musicbrainz_last_attempted": _isoformat(d.get("musicbrainz_last_attempted")),
"spotify_last_attempted": _isoformat(d.get("spotify_last_attempted")),
"itunes_last_attempted": _isoformat(d.get("itunes_last_attempted")),
"audiodb_last_attempted": _isoformat(d.get("audiodb_last_attempted")),
"deezer_last_attempted": _isoformat(d.get("deezer_last_attempted")),
"lastfm_last_attempted": _isoformat(d.get("lastfm_last_attempted")),
"genius_last_attempted": _isoformat(d.get("genius_last_attempted")),
"tidal_last_attempted": _isoformat(d.get("tidal_last_attempted")),
"qobuz_last_attempted": _isoformat(d.get("qobuz_last_attempted")),
# Last.fm metadata
"lastfm_listeners": d.get("lastfm_listeners"),
"lastfm_playcount": d.get("lastfm_playcount"),
"lastfm_tags": d.get("lastfm_tags"),
"lastfm_url": d.get("lastfm_url"),
# Genius metadata
"genius_lyrics": d.get("genius_lyrics"),
"genius_description": d.get("genius_description"),
"genius_url": d.get("genius_url"),
}
# Preserve extra keys from joined queries (artist_name, album_title)
for extra_key in ("artist_name", "album_title"):
if extra_key in d:
result[extra_key] = d[extra_key]
return filter_fields(result, fields)
# ── Watchlist / Wishlist Serializers ──────────────────────────
def serialize_watchlist_artist(obj, fields: Optional[Set[str]] = None) -> dict:
"""Full watchlist artist serialization — all columns including all content filters."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"spotify_artist_id": d.get("spotify_artist_id"),
"itunes_artist_id": d.get("itunes_artist_id"),
"artist_name": d.get("artist_name"),
"image_url": d.get("image_url"),
"date_added": _isoformat(d.get("date_added")),
"last_scan_timestamp": _isoformat(d.get("last_scan_timestamp")),
"created_at": _isoformat(d.get("created_at")),
"updated_at": _isoformat(d.get("updated_at")),
"profile_id": d.get("profile_id"),
# Content type filters — ALL of them
"include_albums": bool(d.get("include_albums", True)),
"include_eps": bool(d.get("include_eps", True)),
"include_singles": bool(d.get("include_singles", True)),
"include_live": bool(d.get("include_live", False)),
"include_remixes": bool(d.get("include_remixes", False)),
"include_acoustic": bool(d.get("include_acoustic", False)),
"include_compilations": bool(d.get("include_compilations", False)),
}
return filter_fields(result, fields)
def serialize_wishlist_track(obj, fields: Optional[Set[str]] = None) -> dict:
"""Standardized wishlist track serialization."""
d = _to_dict(obj)
track_data = d.get("track_data", d.get("spotify_data", {}))
if isinstance(track_data, str):
try:
track_data = json.loads(track_data)
except (json.JSONDecodeError, TypeError):
track_data = {}
source_info = d.get("source_info")
if isinstance(source_info, str):
try:
source_info = json.loads(source_info)
except (json.JSONDecodeError, TypeError):
source_info = None
result = {
"id": d.get("id"),
"track_id": d.get("track_id") or d.get("spotify_track_id") or d.get("id"),
"spotify_track_id": d.get("spotify_track_id"),
"track_name": (
track_data.get("name", "Unknown") if isinstance(track_data, dict) else d.get("track_name", "Unknown")
),
"artist_name": ", ".join(
a.get("name", "") if isinstance(a, dict) else str(a)
for a in track_data.get("artists", [])
) if isinstance(track_data, dict) and isinstance(track_data.get("artists"), list) else "",
"album_name": (
track_data.get("album", {}).get("name")
if isinstance(track_data, dict) and isinstance(track_data.get("album"), dict)
else None
),
"track_data": track_data,
"spotify_data": track_data,
"provider": track_data.get("provider") if isinstance(track_data, dict) else d.get("provider"),
"failure_reason": d.get("failure_reason"),
"retry_count": d.get("retry_count", 0),
"last_attempted": _isoformat(d.get("last_attempted")),
"date_added": _isoformat(d.get("date_added")),
"source_type": d.get("source_type"),
"source_info": source_info,
"profile_id": d.get("profile_id"),
}
return filter_fields(result, fields)
# ── Discovery Serializers ─────────────────────────────────────
def serialize_discovery_track(obj, fields: Optional[Set[str]] = None) -> dict:
"""Discovery pool track serialization."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"spotify_track_id": d.get("spotify_track_id"),
"spotify_album_id": d.get("spotify_album_id"),
"spotify_artist_id": d.get("spotify_artist_id"),
"itunes_track_id": d.get("itunes_track_id"),
"itunes_album_id": d.get("itunes_album_id"),
"itunes_artist_id": d.get("itunes_artist_id"),
"source": d.get("source"),
"track_name": d.get("track_name"),
"artist_name": d.get("artist_name"),
"album_name": d.get("album_name"),
"album_cover_url": d.get("album_cover_url"),
"duration_ms": d.get("duration_ms"),
"popularity": d.get("popularity"),
"release_date": d.get("release_date"),
"is_new_release": bool(d.get("is_new_release", False)),
"artist_genres": _parse_genres(d.get("artist_genres")),
"added_date": _isoformat(d.get("added_date")),
}
return filter_fields(result, fields)
def serialize_similar_artist(obj, fields: Optional[Set[str]] = None) -> dict:
"""Similar artist serialization."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"source_artist_id": d.get("source_artist_id"),
"similar_artist_spotify_id": d.get("similar_artist_spotify_id"),
"similar_artist_itunes_id": d.get("similar_artist_itunes_id"),
"similar_artist_musicbrainz_id": d.get("similar_artist_musicbrainz_id"),
"similar_artist_name": d.get("similar_artist_name"),
"similarity_rank": d.get("similarity_rank"),
"occurrence_count": d.get("occurrence_count"),
"last_updated": _isoformat(d.get("last_updated")),
"last_featured": _isoformat(d.get("last_featured")),
}
return filter_fields(result, fields)
def serialize_recent_release(obj, fields: Optional[Set[str]] = None) -> dict:
"""Recent release serialization."""
d = _to_dict(obj)
result = {
"id": d.get("id"),
"watchlist_artist_id": d.get("watchlist_artist_id"),
"album_spotify_id": d.get("album_spotify_id"),
"album_itunes_id": d.get("album_itunes_id"),
"source": d.get("source"),
"album_name": d.get("album_name"),
"release_date": d.get("release_date"),
"album_cover_url": d.get("album_cover_url"),
"track_count": d.get("track_count"),
"added_date": _isoformat(d.get("added_date")),
}
return filter_fields(result, fields)

189
api/settings.py
View file

@ -1,189 +0,0 @@
"""
Settings and API key management endpoints.
"""
from flask import request, current_app
from .auth import require_api_key, generate_api_key, _hash_key
from .helpers import api_success, api_error
# Keys that must NEVER be exposed via the API
_SENSITIVE_KEYS = {
"spotify.client_id",
"spotify.client_secret",
"tidal.client_id",
"tidal.client_secret",
"tidal_tokens",
"tidal_download.session",
"qobuz.session",
"plex.token",
"jellyfin.api_key",
"navidrome.password",
"soulseek.api_key",
"listenbrainz.token",
"acoustid.api_key",
"lastfm.api_key",
"genius.access_token",
"hydrabase.api_key",
}
def register_routes(bp):
# ---- Settings ----
@bp.route("/settings", methods=["GET"])
@require_api_key
def get_settings():
"""Get current settings (sensitive values redacted)."""
try:
cfg = current_app.soulsync["config_manager"]
raw = dict(cfg.config_data) if hasattr(cfg, "config_data") else {}
sanitized = _redact_sensitive(raw)
return api_success({"settings": sanitized})
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
@bp.route("/settings", methods=["PATCH"])
@require_api_key
def update_settings():
"""Update settings (partial).
Body: {"key": "value", ...} — dot-notation keys accepted.
"""
body = request.get_json(silent=True) or {}
if not body:
return api_error("BAD_REQUEST", "Empty body.", 400)
try:
cfg = current_app.soulsync["config_manager"]
updated = []
for key, value in body.items():
# Block writing API keys through settings endpoint
if key == "api_keys":
continue
cfg.set(key, value)
updated.append(key)
return api_success({"message": "Settings updated.", "updated_keys": updated})
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
# ---- API Key Management ----
@bp.route("/api-keys", methods=["GET"])
@require_api_key
def list_api_keys():
"""List all API keys (prefix + label only, never the full key)."""
try:
cfg = current_app.soulsync["config_manager"]
keys = cfg.get("api_keys", [])
return api_success({
"keys": [
{
"id": k.get("id"),
"label": k.get("label", ""),
"key_prefix": k.get("key_prefix", ""),
"created_at": k.get("created_at"),
"last_used_at": k.get("last_used_at"),
}
for k in keys
]
})
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
@bp.route("/api-keys", methods=["POST"])
@require_api_key
def create_api_key():
"""Generate a new API key.
Body: {"label": "My Bot"}
The raw key is returned ONCE in the response.
"""
body = request.get_json(silent=True) or {}
label = body.get("label", "")
try:
cfg = current_app.soulsync["config_manager"]
raw_key, record = generate_api_key(label)
keys = cfg.get("api_keys", [])
keys.append(record)
cfg.set("api_keys", keys)
return api_success({
"key": raw_key,
"id": record["id"],
"label": record["label"],
"key_prefix": record["key_prefix"],
"created_at": record["created_at"],
}, status=201)
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
@bp.route("/api-keys/<key_id>", methods=["DELETE"])
@require_api_key
def revoke_api_key(key_id):
"""Revoke (delete) an API key by its ID."""
try:
cfg = current_app.soulsync["config_manager"]
keys = cfg.get("api_keys", [])
original_len = len(keys)
keys = [k for k in keys if k.get("id") != key_id]
if len(keys) == original_len:
return api_error("NOT_FOUND", "API key not found.", 404)
cfg.set("api_keys", keys)
return api_success({"message": "API key revoked."})
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
# ---- Bootstrap endpoint (no auth required) ----
@bp.route("/api-keys/bootstrap", methods=["POST"])
def bootstrap_api_key():
"""Generate the first API key when none exist (no auth required).
This endpoint only works when zero API keys are configured.
Body: {"label": "My First Key"}
"""
try:
cfg = current_app.soulsync["config_manager"]
existing = cfg.get("api_keys", [])
if existing:
return api_error("FORBIDDEN",
"API keys already exist. Use an authenticated request to create more.", 403)
body = request.get_json(silent=True) or {}
label = body.get("label", "Default")
raw_key, record = generate_api_key(label)
cfg.set("api_keys", [record])
return api_success({
"key": raw_key,
"id": record["id"],
"label": record["label"],
"key_prefix": record["key_prefix"],
"created_at": record["created_at"],
}, status=201)
except Exception as e:
return api_error("SETTINGS_ERROR", str(e), 500)
def _redact_sensitive(config, prefix=""):
"""Recursively redact sensitive values from a config dict."""
if not isinstance(config, dict):
return config
result = {}
for key, value in config.items():
full_key = f"{prefix}.{key}" if prefix else key
if any(full_key.startswith(s) for s in _SENSITIVE_KEYS):
result[key] = "***REDACTED***"
elif isinstance(value, dict):
result[key] = _redact_sensitive(value, full_key)
else:
result[key] = value
return result

104
api/system.py
View file

@ -1,104 +0,0 @@
"""
System endpoints — status, activity feed, stats.
"""
import logging
import time
from flask import current_app
from .auth import require_api_key
from .helpers import api_success, api_error
logger = logging.getLogger(__name__)
def register_routes(bp):
@bp.route("/system/status", methods=["GET"])
@require_api_key
def system_status():
"""Server status including uptime and service connectivity."""
try:
app = current_app._get_current_object()
ctx = app.soulsync
uptime_seconds = time.time() - getattr(app, "start_time", time.time())
hours, remainder = divmod(int(uptime_seconds), 3600)
minutes, seconds = divmod(remainder, 60)
spotify = ctx.get("spotify_client")
spotify_ok = bool(spotify and spotify.is_authenticated())
soulseek = ctx.get("download_orchestrator")
soulseek_ok = bool(soulseek)
hydrabase = ctx.get("hydrabase_client")
hydrabase_ok = False
if hydrabase:
try:
ws, _ = hydrabase.get_ws_and_lock()
hydrabase_ok = ws is not None and ws.connected
except Exception as e:
logger.debug("hydrabase status probe failed: %s", e)
return api_success({
"uptime": f"{hours}h {minutes}m {seconds}s",
"uptime_seconds": int(uptime_seconds),
"services": {
"spotify": spotify_ok,
"soulseek": soulseek_ok,
"hydrabase": hydrabase_ok,
},
})
except Exception as e:
return api_error("SYSTEM_ERROR", str(e), 500)
@bp.route("/system/activity", methods=["GET"])
@require_api_key
def system_activity():
"""Recent activity feed."""
try:
from core.runtime_state import activity_feed
items = list(activity_feed) if activity_feed else []
return api_success({"activities": items})
except Exception as e:
return api_error("SYSTEM_ERROR", str(e), 500)
@bp.route("/system/stats", methods=["GET"])
@require_api_key
def system_stats():
"""Combined library + download statistics."""
try:
from database.music_database import get_database
db = get_database()
lib_stats = db.get_statistics_for_server()
db_info = db.get_database_info_for_server()
# Active download count
download_count = 0
try:
from core.runtime_state import download_tasks, tasks_lock
with tasks_lock:
download_count = sum(
1 for t in download_tasks.values()
if t.get("status") in ("downloading", "queued", "searching")
)
except ImportError:
pass
return api_success({
"library": {
"artists": lib_stats.get("artists", 0),
"albums": lib_stats.get("albums", 0),
"tracks": lib_stats.get("tracks", 0),
},
"database": {
"size_mb": db_info.get("database_size_mb"),
"last_update": db_info.get("last_update"),
},
"downloads": {
"active": download_count,
},
})
except Exception as e:
return api_error("SYSTEM_ERROR", str(e), 500)

125
api/watchlist.py
View file

@ -1,125 +0,0 @@
"""
Watchlist endpoints — view, add, remove, update watched artists, trigger scans.
"""
from flask import request, current_app
from database.music_database import get_database
from .auth import require_api_key
from .helpers import api_success, api_error, parse_fields, parse_profile_id
from .serializers import serialize_watchlist_artist
def register_routes(bp):
@bp.route("/watchlist", methods=["GET"])
@require_api_key
def list_watchlist():
"""List all watchlist artists for the current profile."""
fields = parse_fields(request)
profile_id = parse_profile_id(request)
try:
db = get_database()
artists = db.get_watchlist_artists(profile_id=profile_id)
return api_success({
"artists": [serialize_watchlist_artist(a, fields) for a in artists]
})
except Exception as e:
return api_error("WATCHLIST_ERROR", str(e), 500)
@bp.route("/watchlist", methods=["POST"])
@require_api_key
def add_to_watchlist():
"""Add an artist to the watchlist.
Body: {"artist_id": "...", "artist_name": "..."}
"""
body = request.get_json(silent=True) or {}
artist_id = body.get("artist_id")
artist_name = body.get("artist_name")
profile_id = parse_profile_id(request)
if not artist_id or not artist_name:
return api_error("BAD_REQUEST", "Missing 'artist_id' or 'artist_name'.", 400)
try:
db = get_database()
ok = db.add_artist_to_watchlist(artist_id, artist_name, profile_id=profile_id)
if ok:
return api_success({"message": f"Added {artist_name} to watchlist."}, status=201)
return api_error("INTERNAL_ERROR", "Failed to add artist to watchlist.", 500)
except Exception as e:
return api_error("WATCHLIST_ERROR", str(e), 500)
@bp.route("/watchlist/<artist_id>", methods=["DELETE"])
@require_api_key
def remove_from_watchlist(artist_id):
"""Remove an artist from the watchlist."""
profile_id = parse_profile_id(request)
try:
db = get_database()
ok = db.remove_artist_from_watchlist(artist_id, profile_id=profile_id)
if ok:
return api_success({"message": "Artist removed from watchlist."})
return api_error("NOT_FOUND", "Artist not found in watchlist.", 404)
except Exception as e:
return api_error("WATCHLIST_ERROR", str(e), 500)
@bp.route("/watchlist/<artist_id>", methods=["PATCH"])
@require_api_key
def update_watchlist_filters(artist_id):
"""Update content type filters for a watchlist artist.
Body: {"include_albums": true, "include_live": false, ...}
Accepts any combination of: include_albums, include_eps, include_singles,
include_live, include_remixes, include_acoustic, include_compilations
"""
body = request.get_json(silent=True) or {}
profile_id = parse_profile_id(request)
allowed_fields = {
"include_albums", "include_eps", "include_singles",
"include_live", "include_remixes", "include_acoustic", "include_compilations",
}
updates = {k: v for k, v in body.items() if k in allowed_fields}
if not updates:
return api_error("BAD_REQUEST", f"No valid filter fields provided. Allowed: {', '.join(sorted(allowed_fields))}", 400)
try:
db = get_database()
conn = db._get_connection()
cursor = conn.cursor()
# Build SET clause
set_parts = [f"{k} = ?" for k in updates]
values = [int(bool(v)) for v in updates.values()]
cursor.execute(f"""
UPDATE watchlist_artists
SET {', '.join(set_parts)}, updated_at = CURRENT_TIMESTAMP
WHERE (spotify_artist_id = ? OR itunes_artist_id = ?) AND profile_id = ?
""", values + [artist_id, artist_id, profile_id])
if cursor.rowcount > 0:
conn.commit()
return api_success({"message": "Watchlist filters updated.", "updated": updates})
return api_error("NOT_FOUND", "Artist not found in watchlist.", 404)
except Exception as e:
return api_error("WATCHLIST_ERROR", str(e), 500)
@bp.route("/watchlist/scan", methods=["POST"])
@require_api_key
def trigger_scan():
"""Trigger a watchlist scan for new releases."""
try:
from web_server import is_watchlist_actually_scanning
if is_watchlist_actually_scanning():
return api_error("CONFLICT", "Watchlist scan is already running.", 409)
from web_server import start_watchlist_scan
start_watchlist_scan()
return api_success({"message": "Watchlist scan started."})
except ImportError:
return api_error("NOT_AVAILABLE", "Watchlist scan function not available.", 501)
except Exception as e:
return api_error("WATCHLIST_ERROR", str(e), 500)

104
api/wishlist.py
View file

@ -1,104 +0,0 @@
"""
Wishlist endpoints — view, add, remove, and trigger processing.
"""
from flask import request
from .auth import require_api_key
from .helpers import api_success, api_error, parse_pagination, build_pagination, parse_fields, parse_profile_id
from .serializers import serialize_wishlist_track
def register_routes(bp):
@bp.route("/wishlist", methods=["GET"])
@require_api_key
def list_wishlist():
"""List wishlist tracks with optional category filter and standardized format."""
category = request.args.get("category") # "singles" or "albums"
page, limit = parse_pagination(request)
fields = parse_fields(request)
profile_id = parse_profile_id(request)
category_filter = category if category in ("singles", "albums") else None
try:
from database.music_database import get_database
db = get_database()
offset = (page - 1) * limit
tracks = db.get_wishlist_tracks(
profile_id=profile_id,
category=category_filter,
limit=limit,
offset=offset,
)
total = db.get_wishlist_count(profile_id=profile_id, category=category_filter)
return api_success(
{"tracks": [serialize_wishlist_track(t, fields) for t in tracks]},
pagination=build_pagination(page, limit, total),
)
except Exception as e:
return api_error("WISHLIST_ERROR", str(e), 500)
@bp.route("/wishlist", methods=["POST"])
@require_api_key
def add_to_wishlist():
"""Add a track to the wishlist.
Body: {"track_data": {...}, "failure_reason": "...", "source_type": "..."}
"""
body = request.get_json(silent=True) or {}
track_data = body.get("track_data") or body.get("spotify_track_data")
reason = body.get("failure_reason", "Added via API")
source_type = body.get("source_type", "api")
profile_id = parse_profile_id(request)
if not track_data:
return api_error("BAD_REQUEST", "Missing 'track_data' in body.", 400)
try:
from database.music_database import get_database
db = get_database()
ok = db.add_to_wishlist(
track_data,
failure_reason=reason,
source_type=source_type,
profile_id=profile_id,
)
if ok:
return api_success({"message": "Track added to wishlist."}, status=201)
return api_error("CONFLICT", "Track may already be in wishlist.", 409)
except Exception as e:
return api_error("WISHLIST_ERROR", str(e), 500)
@bp.route("/wishlist/<track_id>", methods=["DELETE"])
@require_api_key
def remove_from_wishlist(track_id):
"""Remove a track from the wishlist by its track ID."""
profile_id = parse_profile_id(request)
try:
from database.music_database import get_database
db = get_database()
ok = db.remove_from_wishlist(track_id, profile_id=profile_id)
if ok:
return api_success({"message": "Track removed from wishlist."})
return api_error("NOT_FOUND", "Track not found in wishlist.", 404)
except Exception as e:
return api_error("WISHLIST_ERROR", str(e), 500)
@bp.route("/wishlist/process", methods=["POST"])
@require_api_key
def process_wishlist():
"""Trigger wishlist download processing."""
try:
from web_server import is_wishlist_actually_processing
if is_wishlist_actually_processing():
return api_error("CONFLICT", "Wishlist processing is already running.", 409)
from web_server import start_wishlist_missing_downloads
start_wishlist_missing_downloads()
return api_success({"message": "Wishlist processing started."})
except ImportError:
return api_error("NOT_AVAILABLE", "Wishlist processing function not available.", 501)
except Exception as e:
return api_error("WISHLIST_ERROR", str(e), 500)

BIN
assets/pages.gif
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 3.1 MiB

1220
beatport_unified_scraper.py
View file

File diff suppressed because it is too large Load diff

16
config/config.example.json
View file

@ -44,12 +44,10 @@
},
"metadata_enhancement": {
"enabled": true,
"embed_album_art": true,
"single_to_album": false
"embed_album_art": true
},
"file_organization": {
"enabled": true,
"_template_variables": "Available: $artist, $albumartist, $artistletter, $album, $title, $track, $disc, $year, $playlist, $quality (filename only)",
"templates": {
"album_path": "$albumartist/$albumartist - $album/$track - $title",
"single_path": "$artist/$artist - $title/$title",
@ -57,20 +55,10 @@
"playlist_path": "$playlist/$artist - $title"
}
},
"import": {
"staging_path": "./Staging",
"replace_lower_quality": false,
"folder_artist_override": true
},
"lossy_copy": {
"enabled": false,
"bitrate": "320",
"downsample_hires": false
},
"playlist_sync": {
"create_backup": true
},
"listenbrainz": {
"token": "LISTENBRAINZ_TOKEN"
}
}
}

53
config/config.json Normal file
View file

@ -0,0 +1,53 @@
{
"active_media_server": "plex",
"spotify": {
"client_id": "SpotifyClientID",
"client_secret": "SpotifyClientSecret",
"redirect_uri": "http://127.0.0.1:8888/callback"
},
"tidal": {
"client_id": "TidalClientID",
"client_secret": "TidalClientSecret",
"redirect_uri": "http://127.0.0.1:8889/tidal/callback"
},
"plex": {
"base_url": "http://192.168.86.36:32400",
"token": "PLEX_API_TOKEN",
"auto_detect": true
},
"jellyfin": {
"base_url": "http://localhost:8096",
"api_key": "JELLYFIN_API_KEY",
"auto_detect": true
},
"navidrome": {
"base_url": "http://localhost:4533",
"username": "NAVIDROME_USERNAME",
"password": "NAVIDROME_PASSWORD",
"auto_detect": true
},
"soulseek": {
"slskd_url": "http://host.docker.internal:5030",
"api_key": "SoulseekAPIKey",
"download_path": "/app/downloads",
"transfer_path": "/app/Transfer"
},
"logging": {
"path": "logs/app.log",
"level": "INFO"
},
"database": {
"path": "database/music_library.db",
"max_workers": 5
},
"metadata_enhancement": {
"enabled": true,
"embed_album_art": true
},
"playlist_sync": {
"create_backup": true
},
"listenbrainz": {
"token": "LISTENBRAINZ_TOKEN"
}
}

767
config/settings.py
View file

@ -1,285 +1,29 @@
import copy
import json
import os
import sqlite3
import time
from typing import Dict, Any, Optional
from cryptography.fernet import Fernet, InvalidToken
from cryptography.fernet import Fernet
from pathlib import Path
from utils.logging_config import get_logger
logger = get_logger("config")
class ConfigManager:
_VALID_LOG_LEVELS = frozenset({"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"})
def __init__(self, config_path: str = "config/config.json"):
# Determine strict absolute path to settings.py directory to help resolve config.json
# This handles cases where CWD is different (e.g. running from /Users vs /Users/project)
self.base_dir = Path(__file__).parent.parent.absolute()
# Check for environment variable override first (Unified logic with web_server.py)
env_config_path = os.environ.get('SOULSYNC_CONFIG_PATH')
if env_config_path:
config_path = env_config_path
# Resolve config path
if os.path.isabs(config_path):
self.config_path = Path(config_path)
else:
# Try to resolve relative to CWD first (legacy behavior), then relative to project root
cwd_path = Path(config_path)
project_path = self.base_dir / config_path
if cwd_path.exists():
self.config_path = cwd_path.absolute()
elif project_path.exists():
self.config_path = project_path
else:
# Default to project path even if it doesn't exist yet (for creation/fallback)
self.config_path = project_path
logger.info(f"ConfigManager initialized with path: {self.config_path}")
self.config_path = Path(config_path)
self.config_data: Dict[str, Any] = {}
self._fernet: Optional[Fernet] = None
# Use DATABASE_PATH env var, fallback to database/music_library.db
db_path_env = os.environ.get('DATABASE_PATH')
if db_path_env:
self.database_path = Path(db_path_env)
else:
self.database_path = self.base_dir / "database" / "music_library.db"
logger.info(f"Database path set to: {self.database_path}")
self.load_config(str(self.config_path))
def load_config(self, config_path: str = None):
"""
Load configuration from database or file.
Can be called to reload settings into the existing instance.
"""
if config_path:
self.config_path = Path(config_path)
self.encryption_key: Optional[bytes] = None
self.database_path = Path("database/music_library.db") # Hardcoded - same as MusicDatabase
self._load_config()
# Placeholder shipped to the browser in place of a configured secret
# (#832 follow-up). The settings UI shows it as masked dots; if it's
# round-tripped back on save, ``set()`` treats it as "keep existing" so the
# real value is never overwritten by the mask.
REDACTED_SENTINEL = '__redacted_unchanged__'
# Dot-notation paths to sensitive config values that must be encrypted at rest.
# Paths pointing to dicts encrypt the entire dict as a JSON blob.
_SENSITIVE_PATHS = frozenset({
# Spotify
'spotify.client_id',
'spotify.client_secret',
# Tidal
'tidal.client_id',
'tidal.client_secret',
'tidal_tokens', # full dict (access/refresh tokens)
'tidal_download.session', # full dict (access/refresh/expiry)
# Qobuz
'qobuz.session', # full dict (app_id, app_secret, user_auth_token)
# Media servers
'plex.token',
'jellyfin.api_key',
'navidrome.password',
# Download sources
'soulseek.api_key',
'deezer_download.arl',
'lidarr_download.api_key',
'prowlarr.api_key',
'torrent_client.password',
'usenet_client.api_key',
'usenet_client.password',
# Enrichment services
'listenbrainz.token',
'acoustid.api_key',
'lastfm.api_key',
'lastfm.api_secret',
'lastfm.session_key',
'genius.access_token',
# Deezer OAuth
'deezer.app_id',
'deezer.app_secret',
'deezer.access_token',
# Other
'hydrabase.api_key',
'discogs.token',
})
def _get_fernet(self) -> Fernet:
"""Return a cached Fernet instance, creating the key file if needed."""
if self._fernet is not None:
return self._fernet
key_file = self.database_path.parent / ".encryption_key"
# Migrate key from old location (config/) to new location (database/)
old_key_file = self.config_path.parent / ".encryption_key"
if not key_file.exists() and old_key_file.exists():
try:
import shutil
shutil.move(str(old_key_file), str(key_file))
logger.info(f"Moved encryption key to {key_file}")
except Exception:
key_file = old_key_file # Fall back to old location
def _get_encryption_key(self) -> bytes:
key_file = self.config_path.parent / ".encryption_key"
if key_file.exists():
with open(key_file, 'rb') as f:
key = f.read()
return f.read()
else:
key = Fernet.generate_key()
key_file.parent.mkdir(parents=True, exist_ok=True)
with open(key_file, 'wb') as f:
f.write(key)
try:
key_file.chmod(0o600)
except OSError:
pass # Windows may not support Unix permissions
self._fernet = Fernet(key)
return self._fernet
def _encrypt_value(self, value) -> str:
"""Encrypt a config value (string or dict/list) into a Fernet token string."""
f = self._get_fernet()
if isinstance(value, (dict, list)):
plaintext = json.dumps(value)
else:
plaintext = str(value)
return f.encrypt(plaintext.encode('utf-8')).decode('ascii')
def _decrypt_value(self, value):
"""Decrypt a Fernet token string back to the original value.
If value is not encrypted (migration), returns it unchanged."""
if not isinstance(value, str):
return value
# Fernet tokens always start with 'gAAAAA'
if not value.startswith('gAAAAA'):
return value
try:
f = self._get_fernet()
decrypted = f.decrypt(value.encode('ascii')).decode('utf-8')
# Only parse JSON for dicts/lists (starts with { or [).
# Plain strings (including numeric ones like API keys) stay as strings.
if decrypted and decrypted[0] in ('{', '['):
try:
return json.loads(decrypted)
except (json.JSONDecodeError, ValueError):
pass
return decrypted
except InvalidToken:
# Key mismatch — encrypted with a different key (key file deleted/replaced)
logger.error(
"Failed to decrypt a config value — encryption key may have changed. "
"Re-enter credentials in Settings or restore the original .encryption_key file."
)
return value
except Exception:
return value
def _encrypt_sensitive(self, config_data: Dict[str, Any]) -> Dict[str, Any]:
"""Return a deep copy of config_data with sensitive values encrypted."""
encrypted = copy.deepcopy(config_data)
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
# Navigate to the parent
parent = encrypted
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if parent is None or not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf not in parent:
continue
value = parent[leaf]
# Skip empty values (no point encrypting empty strings/dicts)
if not value and value != 0:
continue
# Skip already-encrypted values (idempotent)
if isinstance(value, str) and value.startswith('gAAAAA'):
continue
parent[leaf] = self._encrypt_value(value)
return encrypted
def _decrypt_sensitive(self, config_data: Dict[str, Any]) -> Dict[str, Any]:
"""Decrypt sensitive values in-place and return the config dict."""
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
parent = config_data
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if parent is None or not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf not in parent:
continue
parent[leaf] = self._decrypt_value(parent[leaf])
return config_data
def _migrate_encrypt_if_needed(self):
"""Re-save config to encrypt any plaintext sensitive values still in the DB."""
try:
# Read raw DB content to check if any sensitive value is still plaintext
conn = self._connect_db()
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
row = cursor.fetchone()
conn.close()
if not row or not row[0]:
return
raw = json.loads(row[0])
needs_migration = False
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
parent = raw
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if parent is None or not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf not in parent:
continue
value = parent[leaf]
if not value and value != 0:
continue
# If the value is NOT a Fernet token, it's still plaintext
if not (isinstance(value, str) and value.startswith('gAAAAA')):
needs_migration = True
break
if needs_migration:
logger.info("Encrypting sensitive config values at rest...")
self._save_to_database(self.config_data)
logger.info("Sensitive config values encrypted successfully")
except Exception as e:
logger.warning(f"Could not migrate encryption: {e}")
def _connect_db(self) -> sqlite3.Connection:
"""Open a configured SQLite connection for the config DB.
Centralizes pragma setup so every connection gets WAL mode,
a 30s busy timeout, and synchronous=NORMAL (the safe pairing
with WAL that avoids unnecessary fsyncs on slow disks).
"""
conn = sqlite3.connect(str(self.database_path), timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=30000")
conn.execute("PRAGMA synchronous=NORMAL")
return conn
key_file.chmod(0o600)
return key
def _ensure_database_exists(self):
"""Ensure database file and metadata table exist"""
@ -288,7 +32,7 @@ class ConfigManager:
self.database_path.parent.mkdir(parents=True, exist_ok=True)
# Connect to database (creates file if it doesn't exist)
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path))
cursor = conn.cursor()
# Create metadata table if it doesn't exist
@ -302,129 +46,51 @@ class ConfigManager:
conn.commit()
conn.close()
except Exception as e:
logger.warning(f"Could not ensure database exists: {e}")
print(f"Warning: Could not ensure database exists: {e}")
def _load_from_database(self) -> Optional[Dict[str, Any]]:
"""Load configuration from database, decrypting sensitive values."""
conn = None
"""Load configuration from database"""
try:
self._ensure_database_exists()
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path))
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'app_config'")
row = cursor.fetchone()
conn.close()
if row and row[0]:
config_data = json.loads(row[0])
# Decrypt sensitive values (gracefully handles plaintext migration)
config_data = self._decrypt_sensitive(config_data)
logger.info("Configuration loaded from database")
print("[OK] Configuration loaded from database")
return config_data
else:
return None
except Exception as e:
logger.warning(f"Could not load config from database: {e}")
print(f"Warning: Could not load config from database: {e}")
return None
finally:
if conn:
conn.close()
def _load_stored_log_level(self) -> Optional[str]:
"""Load the persisted UI log level preference, if one exists."""
conn = None
try:
self._ensure_database_exists()
conn = self._connect_db()
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = 'log_level'")
row = cursor.fetchone()
if not row or not row[0]:
return None
level = str(row[0]).upper()
if level not in self._VALID_LOG_LEVELS:
logger.warning(f"Ignoring invalid stored log level: {row[0]}")
return None
return level
except Exception as e:
logger.warning(f"Could not load stored log level from database: {e}")
return None
finally:
if conn:
conn.close()
def _load_env_log_level(self) -> Optional[str]:
"""Load the log level override from the environment, if one exists."""
raw_level = os.environ.get("SOULSYNC_LOG_LEVEL")
if not raw_level:
return None
level = raw_level.upper()
if level not in self._VALID_LOG_LEVELS:
logger.warning(f"Ignoring invalid SOULSYNC_LOG_LEVEL value: {raw_level}")
return None
return level
def _apply_log_level_overrides(self, config_data: Dict[str, Any]) -> Dict[str, Any]:
"""Overlay env and persisted log level preferences onto the loaded config."""
env_level = self._load_env_log_level()
if env_level:
config_data.setdefault("logging", {})["level"] = env_level
logger.info(f"Using log level from SOULSYNC_LOG_LEVEL: {env_level}")
return config_data
stored_level = self._load_stored_log_level()
if stored_level:
config_data.setdefault("logging", {})["level"] = stored_level
logger.info(f"Using stored logging level from database: {stored_level}")
return config_data
def _save_to_database(self, config_data: Dict[str, Any]) -> bool:
"""Save configuration to database, encrypting sensitive values.
Returns ``True`` on success. Transient ``database is locked``
failures are logged at DEBUG so the caller's retry loop owns the
user-visible error message — otherwise every retry would spam
ERROR-level logs even when the next attempt succeeds.
"""
conn = None
"""Save configuration to database"""
try:
self._ensure_database_exists()
# Encrypt sensitive values before writing (original dict is untouched)
encrypted_data = self._encrypt_sensitive(config_data)
conn = self._connect_db()
conn = sqlite3.connect(str(self.database_path))
cursor = conn.cursor()
config_json = json.dumps(encrypted_data, indent=2)
config_json = json.dumps(config_data, indent=2)
cursor.execute("""
INSERT OR REPLACE INTO metadata (key, value, updated_at)
VALUES ('app_config', ?, CURRENT_TIMESTAMP)
""", (config_json,))
conn.commit()
conn.close()
return True
except sqlite3.OperationalError as e:
# SQLite raises OperationalError("database is locked") when the
# busy_timeout expires while another writer holds the lock.
# Log at DEBUG so the caller can decide whether the final
# outcome warrants an ERROR-level message.
if "locked" in str(e).lower():
logger.debug(f"Config DB locked, will retry: {e}")
else:
logger.error(f"Could not save config to database: {e}")
return False
except Exception as e:
logger.error(f"Could not save config to database: {e}")
print(f"Error: Could not save config to database: {e}")
return False
finally:
if conn:
conn.close()
def _load_from_config_file(self) -> Optional[Dict[str, Any]]:
"""Load configuration from config.json file (for migration)"""
@ -432,12 +98,12 @@ class ConfigManager:
if self.config_path.exists():
with open(self.config_path, 'r') as f:
config_data = json.load(f)
logger.info(f"Configuration loaded from {self.config_path}")
print(f"[OK] Configuration loaded from {self.config_path}")
return config_data
else:
return None
except Exception as e:
logger.warning(f"Could not load config from file: {e}")
print(f"Warning: Could not load config from file: {e}")
return None
def _get_default_config(self) -> Dict[str, Any]:
@ -474,278 +140,28 @@ class ConfigManager:
"slskd_url": "",
"api_key": "",
"download_path": "./downloads",
"transfer_path": "./Transfer",
"max_peer_queue": 0,
"download_timeout": 600,
# Reddit report (YeloMelo95, Bell Canada): the existing
# 35-per-220s sliding-window cap allows all 35 searches in
# rapid succession before throttling — that burst trips ISP
# anti-abuse. This knob forces a min gap between consecutive
# searches even when the window cap isn't hit. 0 = disabled
# (preserves prior behavior).
"search_min_delay_seconds": 0,
},
"download_source": {
"mode": "soulseek", # Options: "soulseek", "youtube", "tidal", "qobuz", "hifi", "hybrid", "torrent", "usenet"
"hybrid_primary": "soulseek", # Legacy: primary source for hybrid mode
"hybrid_secondary": "youtube", # Legacy: fallback source for hybrid mode
"hybrid_order": [], # Ordered list of sources for hybrid mode (overrides primary/secondary)
"stream_source": "youtube", # Options: "youtube" (instant, default), "active" (use download source; falls back to youtube if soulseek)
# Album-bundle (torrent / usenet single-source) poll tuning.
# Downloader is polled every N seconds until the release
# lands; whole job aborts at the timeout. Defaults match
# the previous hard-coded constants. Users on slow private
# trackers / large box sets can extend the timeout without
# editing source.
"album_bundle_poll_interval_seconds": 2.0,
"album_bundle_timeout_seconds": 6 * 60 * 60, # 6 hours
# Stalled-torrent handling (noldevin): abandon a torrent that
# makes zero download progress for this long (dead magnet
# stuck on "downloading metadata", no seeders) instead of
# holding the worker for the full album timeout. 0 disables.
"torrent_stall_timeout_seconds": 10 * 60, # 10 minutes
# What to do when a torrent stalls: "abandon" (remove it +
# its partial data, fail the download so the next source can
# try) or "pause" (pause in the client, leave for the user).
"torrent_stall_action": "abandon",
# Where THIS container can read completed torrent/usenet
# downloads (#857). The downloader (qBit/SAB) reports a save
# path from inside ITS OWN container — often a category folder
# like /data/downloads/music — which may be mounted at a
# different point here. Set these to the in-container path(s)
# where SoulSync sees those finished downloads; the resolver
# then finds the release by name under them. Empty = fall back
# to the soulseek download/transfer dirs (the shared-volume
# default). See core.download_plugins.album_bundle.resolve_reported_save_path.
"torrent_download_path": "",
"usenet_download_path": "",
# Explicit remote→local prefix mappings for non-shared / oddly
# mounted layouts (Sonarr/Radarr "Remote Path Mapping" style):
# a list of {"from": "<client path>", "to": "<soulsync path>"}.
# Tried before the basename fallback above.
"usenet_path_mappings": [],
},
"post_processing": {
# When a download is quarantined (AcoustID mismatch, integrity /
# duration failure), retry the next-best candidate instead of
# failing outright. Default ON (PR #801's documented default —
# the monitor reads this with inline default True; this template
# said False, so fresh installs silently shipped with the retry
# engine off while existing configs got it on. CI caught the
# split: its fresh default config failed all 7 requeue tests).
"retry_next_candidate_on_mismatch": True,
# Opt-in exhaustive retry: budget retries PER SOURCE so every
# source (Soulseek, then HiFi/Tidal/…) gets its own attempts
# before the track gives up. Default off (single global cap).
"retry_exhaustive": False,
# Retries per search query per source in exhaustive mode. The
# per-source budget is query_count × this value.
"retries_per_query": 5,
},
"tidal_download": {
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
"session": {
"token_type": "",
"access_token": "",
"refresh_token": "",
"expiry_time": 0
}
},
"qobuz": {
"quality": "lossless", # Options: "mp3", "lossless", "hires", "hires_max"
"session": {
"app_id": "",
"app_secret": "",
"user_auth_token": ""
}
},
"hifi_download": {
"quality": "lossless", # Options: "low", "high", "lossless", "hires"
},
"hifi": {
"embed_tags": True,
"tags": {
"track_id": True,
"artist_id": True,
"isrc": True,
"bpm": True,
"copyright": True,
}
},
"lidarr_download": {
"url": "",
"api_key": "",
"root_folder": "",
"quality_profile": "Any",
"cleanup_after_import": True,
},
# Prowlarr — indexer aggregator. Feeds the torrent / usenet
# download plugins. Not a standalone source.
"prowlarr": {
"url": "",
"api_key": "",
# Comma-separated list of indexer IDs to limit searches to.
# Empty = search all enabled indexers.
"indexer_ids": "",
},
# Torrent client — receives .torrent / magnet URIs from the
# torrent download plugin. ``type`` picks which adapter to
# instantiate (qbittorrent | transmission | deluge).
"torrent_client": {
"type": "qbittorrent",
"url": "",
"username": "",
"password": "",
"category": "soulsync",
"save_path": "",
},
# Usenet client — receives .nzb URLs / payloads. ``type``
# picks the adapter (sabnzbd | nzbget). SABnzbd uses an
# API key; NZBGet uses username + password.
"usenet_client": {
"type": "sabnzbd",
"url": "",
"api_key": "",
"username": "",
"password": "",
"category": "soulsync",
},
"soundcloud_download": {
# Anonymous-only for now — SoundCloud Go+ OAuth tier could be
# added later, with credentials living under a "session" subkey
# alongside Tidal/Qobuz. No quality knob: anonymous SoundCloud
# caps at the upload's transcoding (typically 128 kbps MP3 or
# AAC). yt-dlp resolves bestaudio at download time.
"transfer_path": "./Transfer"
},
"listenbrainz": {
"base_url": "",
"token": "",
"scrobble_enabled": False
},
"acoustid": {
"api_key": "",
"enabled": False # Disabled by default - requires API key and fpcalc
},
"lastfm": {
"api_key": "",
"api_secret": "",
"session_key": "",
"scrobble_enabled": False
},
"genius": {
"access_token": ""
"token": ""
},
"logging": {
"path": "logs/app.log",
"level": "INFO"
},
"database": {
"path": os.environ.get('DATABASE_PATH', 'database/music_library.db'),
"path": "database/music_library.db",
"max_workers": 5
},
"image_cache": {
"enabled": True,
"path": "storage/image_cache",
"ttl_seconds": 2592000,
"failed_ttl_seconds": 21600,
"max_download_mb": 15
},
"metadata_enhancement": {
"enabled": True,
"embed_album_art": True,
"post_process_order": ["musicbrainz", "deezer", "audiodb", "tidal", "qobuz", "lastfm", "genius"],
# Ordered preferred cover-art sources (empty = use the
# download's own art, i.e. today's behavior). Resolved + walked
# with fallback by core/metadata/art_sources.py.
"album_art_order": [],
# Minimum cover-art resolution (shortest side, px). A preferred
# source whose art is smaller is skipped so the next source is
# tried — stops a low-res Cover Art Archive upload from winning.
# 0 disables the size gate.
"min_art_size": 1000,
# When a track matches a SINGLE release, look up the parent ALBUM
# that contains it and tag it as that album, so it groups with its
# album-mates and gets the album cover (not the single's). Off by
# default — it's an extra per-import metadata lookup.
"single_to_album": False
},
"musicbrainz": {
"embed_tags": True
"embed_album_art": True
},
"playlist_sync": {
"create_backup": True,
# How a re-sync writes to the server playlist:
# replace — delete + recreate (default; today's behavior)
# reconcile — edit in place (add/remove delta), preserving the
# playlist's custom image, description, and identity (#792)
# append — only add new tracks, never remove
"mode": "replace"
"create_backup": True
},
"settings": {
"audio_quality": "flac"
},
"lossy_copy": {
"enabled": False,
"codec": "mp3",
"bitrate": "320",
"delete_original": False,
"downsample_hires": False
},
"listening_stats": {
"enabled": True,
"poll_interval": 30
},
"library": {
"music_paths": [],
"music_videos_path": ""
},
"scripts": {
"path": "./scripts",
"timeout": 60
},
"import": {
"staging_path": "./Staging",
# Master toggle for quality-filtering on import. On by default:
# downloaded files that don't meet the quality profile are
# quarantined instead of imported (same gate the download
# pipeline uses). Off → import everything regardless of quality;
# the library Quality Upgrade Scanner still flags them.
"quality_filter_enabled": True,
"replace_lower_quality": False,
# Use the top Staging folder as the artist (Artist/Album layouts,
# mixtapes). On by default to preserve the long-standing import
# behaviour for existing users. Turn OFF if you stage a mixed pile
# of songs under one container folder, otherwise that folder's name
# overrides every metadata-identified artist (the "soulsync" case).
"folder_artist_override": True
},
"m3u_export": {
"enabled": False,
"entry_base_path": ""
},
"playlists": {
# Where "Organize by playlist" materializes playlist folders.
# MUST be a separate root from the music library so the media
# server (and the maintenance jobs) never scan it — otherwise the
# same track would show up twice. Mapped separately for Docker.
"materialize_path": "./Playlists",
# "symlink" (relative links, ~zero disk) or "copy" (real
# duplicates for FAT/USB/DAPs that can't follow links). Symlink
# auto-falls back to copy when the filesystem can't link.
"materialize_mode": "symlink"
},
"youtube": {
"cookies_browser": "", # "", "chrome", "firefox", "edge", "brave", "opera", "safari"
"download_delay": 3, # seconds between sequential downloads
},
"hydrabase": {
"url": "",
"api_key": "",
"auto_connect": False,
"enabled": False
},
"content_filter": {
"allow_explicit": True
}
}
@ -756,79 +172,55 @@ class ConfigManager:
2. config.json (migration from file-based config)
3. Defaults (fresh install)
"""
logger.info("Loading configuration...")
# Try loading from database first
config_data = self._load_from_database()
if config_data:
# Configuration exists in database
self.config_data = self._apply_log_level_overrides(config_data)
# Ensure sensitive values are encrypted at rest (one-time migration)
self._migrate_encrypt_if_needed()
self.config_data = config_data
return
# Database is empty - try migration from config.json
logger.info(f"Configuration not found in database. Attempting migration from: {self.config_path}")
config_data = self._load_from_config_file()
if config_data:
# Migrate from config.json to database
logger.info("Migrating configuration from config.json to database...")
print("[MIGRATE] Migrating configuration from config.json to database...")
if self._save_to_database(config_data):
logger.info("Configuration migrated successfully to database.")
self.config_data = self._apply_log_level_overrides(config_data)
print("[OK] Configuration migrated successfully")
self.config_data = config_data
return
else:
logger.warning("Migration failed - using file-based config temporarily.")
self.config_data = self._apply_log_level_overrides(config_data)
print("[WARN] Migration failed - using file-based config")
self.config_data = config_data
return
# No config.json either - use defaults
logger.info("No existing configuration found (DB or File) - using defaults")
print("[INFO] No existing configuration found - using defaults")
config_data = self._get_default_config()
# Try to save defaults to database
if self._save_to_database(config_data):
logger.info("Default configuration saved to database")
print("[OK] Default configuration saved to database")
else:
logger.warning("Could not save defaults to database - using in-memory config")
print("[WARN] Could not save defaults to database - using in-memory config")
self.config_data = self._apply_log_level_overrides(config_data)
self.config_data = config_data
def _save_config(self):
"""Save configuration to database with exponential-backoff retry on lock.
"""Save configuration to database"""
success = self._save_to_database(self.config_data)
Spread retries over ~7 seconds so a long-held writer (enrichment
worker batch insert, library scan commit, etc.) on a slow disk
has time to release the lock before we fall back to the JSON
file. The single 1s retry that used to live here gave up too
early on HDD-backed Docker volumes.
"""
# Cumulative delay across attempts: 0.2 + 0.5 + 1.0 + 2.0 + 4.0 = 7.7s
# plus the 30s busy_timeout that already runs inside each attempt.
retry_delays = [0.2, 0.5, 1.0, 2.0, 4.0]
if self._save_to_database(self.config_data):
return
for delay in retry_delays:
time.sleep(delay)
if self._save_to_database(self.config_data):
return
# All retries exhausted — fall back to config.json so the user
# doesn't lose their settings, then log a single error.
logger.error(
f"Config DB save failed after {len(retry_delays) + 1} attempts (database is locked) — "
"falling back to config.json"
)
try:
self.config_path.parent.mkdir(parents=True, exist_ok=True)
with open(self.config_path, 'w') as f:
json.dump(self.config_data, f, indent=2)
logger.warning("Configuration saved to config.json as fallback")
except Exception as e:
logger.error(f"Failed to save configuration: {e}")
if not success:
# Fallback: Try to save to config.json if database fails
print("[WARN] Database save failed - attempting file fallback")
try:
self.config_path.parent.mkdir(parents=True, exist_ok=True)
with open(self.config_path, 'w') as f:
json.dump(self.config_data, f, indent=2)
print("[OK] Configuration saved to config.json as fallback")
except Exception as e:
print(f"[ERROR] Failed to save configuration: {e}")
def get(self, key: str, default: Any = None) -> Any:
keys = key.split('.')
@ -842,40 +234,7 @@ class ConfigManager:
return value
def redacted_config(self) -> Dict[str, Any]:
"""Deep copy of the live config with every sensitive value masked.
Used for ``GET /api/settings`` so decrypted secrets never reach the
browser (#832 follow-up). A *set* secret becomes ``REDACTED_SENTINEL``
(the UI renders it as masked dots); an unset one stays empty so the UI
can show "not configured". Dict-valued secrets (OAuth sessions) collapse
to the sentinel too — the UI has no field for them anyway. The matching
guard in ``set()`` turns a round-tripped sentinel back into a no-op.
"""
import copy
data = copy.deepcopy(self.config_data)
for path in self._SENSITIVE_PATHS:
keys = path.split('.')
parent = data
for k in keys[:-1]:
if isinstance(parent, dict) and k in parent:
parent = parent[k]
else:
parent = None
break
if not isinstance(parent, dict):
continue
leaf = keys[-1]
if leaf in parent and parent[leaf] not in (None, '', {}, [], 0, False):
parent[leaf] = self.REDACTED_SENTINEL
return data
def set(self, key: str, value: Any):
# The UI round-trips REDACTED_SENTINEL for any secret the user didn't
# touch — never let the mask overwrite the real value (#832 follow-up).
if value == self.REDACTED_SENTINEL and key in self._SENSITIVE_PATHS:
return
keys = key.split('.')
config = self.config_data
@ -887,20 +246,6 @@ class ConfigManager:
config[keys[-1]] = value
self._save_config()
def resolve_secret(self, key: str, posted: Any) -> str:
"""Resolve a secret value coming back from the settings UI.
The UI renders a saved-but-untouched secret as the REDACTED_SENTINEL (shown
masked); empty or that sentinel means "use the stored value", while a real
string is a genuine new secret. A connection-test endpoint should test the
EFFECTIVE secret, not the mask — otherwise testing a saved-but-untouched
token sends the sentinel and the source rejects it (#870)."""
if isinstance(posted, str):
posted = posted.strip()
if not posted or posted == self.REDACTED_SENTINEL:
return self.get(key, '') or ''
return posted
def get_spotify_config(self) -> Dict[str, str]:
return self.get('spotify', {})
@ -916,9 +261,6 @@ class ConfigManager:
def get_soulseek_config(self) -> Dict[str, str]:
return self.get('soulseek', {})
def get_hydrabase_config(self) -> Dict[str, str]:
return self.get('hydrabase', {})
def get_settings(self) -> Dict[str, Any]:
return self.get('settings', {})
@ -932,8 +274,8 @@ class ConfigManager:
return self.get('active_media_server', 'plex')
def set_active_media_server(self, server: str):
"""Set the active media server (plex, jellyfin, navidrome, or soulsync)"""
if server not in ['plex', 'jellyfin', 'navidrome', 'soulsync']:
"""Set the active media server (plex, jellyfin, or navidrome)"""
if server not in ['plex', 'jellyfin', 'navidrome']:
raise ValueError(f"Invalid media server: {server}")
self.set('active_media_server', server)
@ -946,8 +288,6 @@ class ConfigManager:
return self.get_jellyfin_config()
elif active_server == 'navidrome':
return self.get_navidrome_config()
elif active_server == 'soulsync':
return {'transfer_path': self.get('soulseek.transfer_path', './Transfer')}
else:
return {}
@ -967,8 +307,6 @@ class ConfigManager:
elif active_server == 'navidrome':
navidrome = self.get_navidrome_config()
media_server_configured = bool(navidrome.get('base_url')) and bool(navidrome.get('username')) and bool(navidrome.get('password'))
elif active_server == 'soulsync':
media_server_configured = True # SoulSync standalone is always configured
return (
bool(spotify.get('client_id')) and
@ -989,7 +327,6 @@ class ConfigManager:
validation['plex'] = bool(self.get('plex.base_url')) and bool(self.get('plex.token'))
validation['jellyfin'] = bool(self.get('jellyfin.base_url')) and bool(self.get('jellyfin.api_key'))
validation['navidrome'] = bool(self.get('navidrome.base_url')) and bool(self.get('navidrome.username')) and bool(self.get('navidrome.password'))
validation['soulsync'] = True # Standalone mode is always valid
validation['active_media_server'] = active_server
return validation

462
core/acoustid_client.py
View file

@ -1,462 +0,0 @@
"""
AcoustID Client for audio fingerprinting and lookup.
Uses the pyacoustid library which handles:
- Fingerprint generation via chromaprint library
- AcoustID API lookups
- Rate limiting
The fpcalc binary is auto-downloaded if not found (Windows, macOS, Linux x86_64).
"""
import threading
import sys
import platform
import zipfile
import tarfile
import tempfile
import urllib.request
from typing import Dict, List, Optional, Any, Tuple
from pathlib import Path
import os
import shutil
import logging
import logging.handlers
from utils.logging_config import get_logger
from config.settings import config_manager
# fpcalc binary location (downloaded automatically if needed)
FPCALC_BIN_DIR = Path(__file__).parent.parent / "bin"
CHROMAPRINT_VERSION = "1.5.1"
_acoustid_logger = logging.getLogger("soulsync.acoustid")
_acoustid_logger.setLevel(logging.DEBUG)
_acoustid_log_path = Path(config_manager.get('logging.path', 'logs/app.log')).parent / "acoustid.log"
_acoustid_log_path.parent.mkdir(parents=True, exist_ok=True)
if not _acoustid_logger.handlers:
_acoustid_file_handler = logging.handlers.RotatingFileHandler(
_acoustid_log_path, encoding='utf-8', maxBytes=5*1024*1024, backupCount=2
)
_acoustid_file_handler.setLevel(logging.DEBUG)
_acoustid_file_handler.setFormatter(logging.Formatter(
fmt='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
))
_acoustid_logger.addHandler(_acoustid_file_handler)
_acoustid_logger.propagate = False
logger = get_logger("acoustid.client")
# Check if pyacoustid is available
try:
import acoustid
ACOUSTID_AVAILABLE = True
logger.info("pyacoustid library loaded successfully")
except ImportError:
ACOUSTID_AVAILABLE = False
logger.warning("pyacoustid library not installed - run: pip install pyacoustid")
def _get_fpcalc_download_url() -> Optional[str]:
"""Get the download URL for fpcalc based on current platform."""
system = platform.system().lower()
machine = platform.machine().lower()
# Map architecture names
if machine in ('x86_64', 'amd64'):
arch = 'x86_64'
elif machine in ('i386', 'i686', 'x86'):
arch = 'i686'
elif machine in ('arm64', 'aarch64'):
arch = 'aarch64'
else:
logger.warning(f"Unknown architecture: {machine}")
return None
base_url = f"https://github.com/acoustid/chromaprint/releases/download/v{CHROMAPRINT_VERSION}"
if system == 'windows':
if arch == 'x86_64':
return f"{base_url}/chromaprint-fpcalc-{CHROMAPRINT_VERSION}-windows-x86_64.zip"
elif system == 'darwin':
# Universal build supports both Intel and Apple Silicon natively
return f"{base_url}/chromaprint-fpcalc-{CHROMAPRINT_VERSION}-macos-universal.tar.gz"
elif system == 'linux':
if arch == 'x86_64':
return f"{base_url}/chromaprint-fpcalc-{CHROMAPRINT_VERSION}-linux-x86_64.tar.gz"
logger.warning(f"No fpcalc download available for {system}-{arch}")
return None
def _download_fpcalc() -> Optional[str]:
"""
Download and extract fpcalc binary for the current platform.
Returns:
Path to fpcalc binary if successful, None otherwise.
"""
url = _get_fpcalc_download_url()
if not url:
return None
try:
logger.info(f"Downloading fpcalc from: {url}")
# Create bin directory
FPCALC_BIN_DIR.mkdir(parents=True, exist_ok=True)
# Download to temp file
with tempfile.NamedTemporaryFile(delete=False, suffix=Path(url).suffix) as tmp:
tmp_path = tmp.name
urllib.request.urlretrieve(url, tmp_path)
# Extract based on file type
fpcalc_name = "fpcalc.exe" if platform.system().lower() == 'windows' else "fpcalc"
fpcalc_dest = FPCALC_BIN_DIR / fpcalc_name
if url.endswith('.zip'):
with zipfile.ZipFile(tmp_path, 'r') as zf:
# Find fpcalc in the archive
for name in zf.namelist():
if name.endswith(fpcalc_name):
# Extract to bin directory
with zf.open(name) as src, open(fpcalc_dest, 'wb') as dst:
dst.write(src.read())
break
elif url.endswith('.tar.gz'):
with tarfile.open(tmp_path, 'r:gz') as tf:
for member in tf.getmembers():
if member.name.endswith('fpcalc'):
# Extract to bin directory
member.name = fpcalc_name
tf.extract(member, FPCALC_BIN_DIR)
break
# Clean up temp file
os.unlink(tmp_path)
# Make executable on Unix
if platform.system().lower() != 'windows':
os.chmod(fpcalc_dest, 0o755)
if fpcalc_dest.exists():
logger.info(f"fpcalc downloaded successfully: {fpcalc_dest}")
return str(fpcalc_dest)
else:
logger.error("fpcalc not found in downloaded archive")
return None
except Exception as e:
logger.error(f"Failed to download fpcalc: {e}")
return None
def _find_fpcalc() -> Optional[str]:
"""Find fpcalc binary, downloading if necessary."""
# Check PATH first
fpcalc = shutil.which("fpcalc") or shutil.which("fpcalc.exe")
if fpcalc:
return fpcalc
# Check our bin directory
fpcalc_name = "fpcalc.exe" if platform.system().lower() == 'windows' else "fpcalc"
local_fpcalc = FPCALC_BIN_DIR / fpcalc_name
if local_fpcalc.exists():
return str(local_fpcalc)
# Try to download
return _download_fpcalc()
# Check if chromaprint/fpcalc is available for fingerprinting
CHROMAPRINT_AVAILABLE = False
FPCALC_PATH = None
if ACOUSTID_AVAILABLE:
# Try to find or download fpcalc
FPCALC_PATH = _find_fpcalc()
if FPCALC_PATH:
CHROMAPRINT_AVAILABLE = True
logger.info(f"fpcalc binary ready: {FPCALC_PATH}")
# Set environment variable so pyacoustid can find it
os.environ['FPCALC'] = FPCALC_PATH
else:
logger.warning("fpcalc not available - fingerprinting will not work")
class AcoustIDClient:
"""
Client for audio fingerprinting via pyacoustid.
Usage:
client = AcoustIDClient()
available, reason = client.is_available()
if available:
result = client.fingerprint_and_lookup("/path/to/audio.mp3")
if result:
for mbid in result['recording_mbids']:
logger.info(f"Match: {mbid}")
"""
def __init__(self):
"""Initialize AcoustID client with settings from config."""
self._api_key = None
self._enabled = None
@property
def api_key(self) -> str:
"""Get API key from config (cached)."""
if self._api_key is None:
self._api_key = config_manager.get('acoustid.api_key', '')
return self._api_key
@property
def enabled(self) -> bool:
"""Check if AcoustID verification is enabled in config."""
if self._enabled is None:
self._enabled = config_manager.get('acoustid.enabled', False)
return self._enabled
def is_available(self) -> Tuple[bool, str]:
"""
Check if AcoustID verification is available and ready.
Returns:
Tuple of (is_available, reason_message)
"""
if not ACOUSTID_AVAILABLE:
return False, "pyacoustid library not installed"
if not self.api_key:
return False, "No AcoustID API key configured"
if not self.enabled:
return False, "AcoustID verification is disabled"
# Check if chromaprint or fpcalc is available
if not self._check_fingerprint_available():
return False, "Chromaprint library not installed (install libchromaprint1)"
return True, "AcoustID verification ready"
def _check_fingerprint_available(self) -> bool:
"""Check if we can generate fingerprints (chromaprint lib or fpcalc)."""
global CHROMAPRINT_AVAILABLE, FPCALC_PATH
if CHROMAPRINT_AVAILABLE:
return True
# Try to find/download fpcalc if not already available
FPCALC_PATH = _find_fpcalc()
if FPCALC_PATH:
CHROMAPRINT_AVAILABLE = True
os.environ['FPCALC'] = FPCALC_PATH
logger.info(f"fpcalc now available: {FPCALC_PATH}")
return True
return False
def _find_test_audio_file(self) -> Optional[str]:
"""Find an audio file to use for testing the AcoustID API key."""
audio_extensions = {'.mp3', '.flac', '.ogg', '.m4a', '.wav', '.wma', '.aac'}
search_dirs = []
# Check transfer and download paths from config
transfer_path = config_manager.get('soulseek.transfer_path', '')
download_path = config_manager.get('soulseek.download_path', '')
if transfer_path:
search_dirs.append(Path(transfer_path))
if download_path:
search_dirs.append(Path(download_path))
for search_dir in search_dirs:
if not search_dir.exists():
continue
# Walk up to 2 levels deep to find an audio file quickly
for _depth, pattern in enumerate(['*', '*/*']):
for f in search_dir.glob(pattern):
if f.is_file() and f.suffix.lower() in audio_extensions:
return str(f)
return None
def test_api_key(self) -> Tuple[bool, str]:
"""
Validate the API key with a direct AcoustID lookup call. An invalid key
is reported as invalid (error code 4); any other error means the key was
accepted.
Returns:
Tuple of (success, message)
"""
if not self.api_key:
return False, "No API key configured"
import requests
try:
# Authoritative key check: a direct API lookup with a dummy
# fingerprint. AcoustID validates the client key first, so an
# invalid key returns error code 4 regardless of the fingerprint.
# (The previous real-file path trusted "no exception = valid", but
# fingerprint_and_lookup swallows the invalid-key error and returns
# None — so it reported broken keys as valid. #756-adjacent.)
url = 'https://api.acoustid.org/v2/lookup'
params = {
'client': self.api_key,
'duration': 187,
'fingerprint': 'AQADtMkWaYkSZRGO',
'meta': 'recordings'
}
response = requests.get(url, params=params, timeout=10)
data = response.json()
if data.get('status') == 'error':
error = data.get('error', {})
error_code = error.get('code', 0)
# Error code 4 is specifically "invalid API key"
if error_code == 4:
return False, "Invalid AcoustID API key - get one from https://acoustid.org/new-application"
# Any other error (e.g. "invalid fingerprint") means the API key
# was accepted — the dummy test fingerprint is just rejected as expected
return True, "AcoustID API key is valid"
# Status is 'ok' - key is valid
return True, "AcoustID API key is valid"
except requests.exceptions.Timeout:
return False, "AcoustID API timeout - try again later"
except requests.exceptions.RequestException as e:
return False, f"Network error: {str(e)}"
except Exception as e:
logger.error(f"Error testing AcoustID API key: {e}")
return False, f"Error: {str(e)}"
def lookup_with_status(self, audio_file: str) -> Dict[str, Any]:
"""Fingerprint + AcoustID lookup returning a STRUCTURED result.
Unlike fingerprint_and_lookup() (which collapses every outcome into
dict-or-None), this distinguishes a genuine no-match from an actual
error — an invalid API key, rate limit, missing chromaprint, or a
fingerprint failure. That distinction is what lets the UI show "AcoustID
Error" (something is broken — fix it) instead of a benign-looking
"Skipped" that silently hides a dead key.
Returns dict with:
'status': 'ok' | 'no_match' | 'error' | 'no_backend'
| 'fingerprint_error' | 'unsupported' | 'unavailable'
| 'not_found'
'recordings': list (meaningful only for 'ok')
'best_score': float
'recording_mbids': list
'error': human-readable detail for any non-'ok' status
'invalid_key': bool (True when the API specifically rejected the key)
"""
if not ACOUSTID_AVAILABLE:
return {'status': 'unavailable', 'recordings': [], 'error': 'pyacoustid library not installed'}
if not self.api_key:
return {'status': 'unavailable', 'recordings': [], 'error': 'No AcoustID API key configured'}
if not os.path.isfile(audio_file):
logger.warning(f"Cannot lookup: file not found: {audio_file}")
return {'status': 'not_found', 'recordings': [], 'error': f'File not found: {audio_file}'}
# Check channel count — chromaprint crashes (SIGABRT) on >2 channel files (e.g. 5.1 surround)
try:
from mutagen import File as MutagenFile
mf = MutagenFile(audio_file)
if mf and mf.info:
channels = getattr(mf.info, 'channels', 2)
if channels and channels > 2:
logger.warning(f"Skipping AcoustID: file has {channels} channels (surround audio): {audio_file}")
return {'status': 'unsupported', 'recordings': [],
'error': f'{channels}-channel (surround) audio not supported by chromaprint'}
except Exception as e:
logger.debug(f"Could not check channel count, proceeding anyway: {e}")
try:
import acoustid
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "NOT SET"
logger.info(f"Fingerprinting and looking up: {audio_file} (API key: {api_key_preview})")
logger.debug("Running acoustid.match()...")
recordings = []
seen_mbids = set()
best_score = 0.0
for result in acoustid.match(self.api_key, audio_file, parse=True):
# match() with parse=True returns (score, recording_id, title, artist)
if not isinstance(result, tuple) or len(result) < 2:
logger.warning(f"Unexpected result format: {result}")
continue
score = result[0]
recording_id = result[1]
title = result[2] if len(result) > 2 else None
artist = result[3] if len(result) > 3 else None
logger.debug(f"Got result: score={score}, id={recording_id}, title={title}, artist={artist}")
if score > best_score:
best_score = score
if recording_id and recording_id not in seen_mbids:
seen_mbids.add(recording_id)
recordings.append({'mbid': recording_id, 'title': title, 'artist': artist, 'score': score})
logger.debug(f"Found match: {title} by {artist} (MBID: {recording_id}, score: {score})")
if not recordings:
logger.info(f"No AcoustID matches found for: {audio_file}")
return {'status': 'no_match', 'recordings': [], 'best_score': best_score,
'recording_mbids': [], 'error': 'Track not found in AcoustID database'}
logger.info(f"AcoustID found {len(recordings)} recording(s) (best score: {best_score:.2f})")
return {'status': 'ok', 'recordings': recordings, 'best_score': best_score,
'recording_mbids': list(seen_mbids)}
except acoustid.NoBackendError:
logger.error("Chromaprint library not found and fpcalc not available")
return {'status': 'no_backend', 'recordings': [],
'error': 'Chromaprint/fpcalc not installed (install libchromaprint1)'}
except acoustid.FingerprintGenerationError as e:
logger.warning(f"Failed to fingerprint {audio_file}: {e}")
return {'status': 'fingerprint_error', 'recordings': [], 'error': f'Could not fingerprint file: {e}'}
except acoustid.WebServiceError as e:
api_key_preview = f"{self.api_key[:8]}..." if self.api_key and len(self.api_key) > 8 else "???"
logger.warning(f"AcoustID API error (key: {api_key_preview}): {e}")
error_str = str(e).lower()
# Old pyacoustid reports an invalid key as the bare "status: error"
# (it drops the detail), so treat that as an invalid-key signal too.
invalid = ('invalid' in error_str or 'unknown' in error_str or 'status: error' in error_str)
if invalid:
logger.error("AcoustID API key appears to be invalid — check your AcoustID settings")
elif 'rate' in error_str or 'limit' in error_str:
logger.warning("Rate limited by AcoustID — will retry later")
return {'status': 'error', 'recordings': [], 'invalid_key': invalid,
'error': f'AcoustID API error: {e}'}
except Exception as e:
logger.error(f"Unexpected error in AcoustID lookup: {e}", exc_info=True)
return {'status': 'error', 'recordings': [], 'error': f'Unexpected error: {e}'}
def fingerprint_and_lookup(self, audio_file: str) -> Optional[Dict[str, Any]]:
"""Legacy dict-or-None lookup. Returns the recordings dict on a confirmed
match, else None. Kept for callers that only need "did we identify it"
(library scanner, auto-import). Callers that must report WHY a lookup
didn't match (verification badge, key test) should use
``lookup_with_status`` so an error isn't mistaken for a no-match.
"""
res = self.lookup_with_status(audio_file)
if res.get('status') == 'ok':
return {
'recordings': res['recordings'],
'best_score': res.get('best_score', 0.0),
'recording_mbids': res.get('recording_mbids', []),
}
return None
def refresh_config(self):
"""Refresh cached config values (call after settings change)."""
self._api_key = None
self._enabled = None

372
core/acoustid_verification.py
View file

@ -1,372 +0,0 @@
"""
AcoustID Verification Service
Verifies downloaded audio files match expected track metadata by comparing
title/artist from AcoustID fingerprint results against the expected track info.
If the audio fingerprint confidently identifies a DIFFERENT song than expected,
the file is flagged as incorrect.
"""
import re
import threading
from difflib import SequenceMatcher
from typing import Optional, Dict, Any, Tuple, List
from enum import Enum
from utils.logging_config import get_logger
from core.acoustid_client import AcoustIDClient
from core.matching_engine import MusicMatchingEngine
from core.matching.version_mismatch import is_acceptable_version_mismatch
from core.matching.script_compat import is_cross_script_mismatch
from core.musicbrainz_client import MusicBrainzClient
logger = get_logger("acoustid.verification")
# Thresholds — single definition lives in the shared core; re-exported here so
# existing importers keep working and the values can't drift between paths.
from core.matching.audio_verification import ( # noqa: E402
MIN_ACOUSTID_SCORE, TITLE_MATCH_THRESHOLD, ARTIST_MATCH_THRESHOLD,
)
# Single matching-engine instance so version detection reuses the same patterns
# used by the pre-download Soulseek matcher (remix / live / acoustic /
# instrumental / etc). detect_version_type doesn't use self state, so one
# shared instance is fine.
_match_engine_for_version = MusicMatchingEngine()
def _detect_title_version(title: str) -> str:
"""Return version label for a track title.
Returns ``'original'`` when no version marker is detected, otherwise one
of the labels produced by ``MusicMatchingEngine.detect_version_type``
(``'instrumental'``, ``'live'``, ``'acoustic'``, ``'remix'``, etc).
"""
if not title:
return 'original'
version_type, _ = _match_engine_for_version.detect_version_type(title)
return version_type
class VerificationResult(Enum):
"""Possible outcomes of audio verification."""
PASS = "pass" # Title/artist match - file is correct
FAIL = "fail" # Title/artist mismatch - wrong file downloaded
SKIP = "skip" # Genuinely couldn't verify (no match in DB) - continue normally
DISABLED = "disabled" # Verification not enabled
ERROR = "error" # Lookup errored (invalid key / rate limit / no backend) - continue, but flag it
# normalize() + similarity() + the alias-aware comparison now live in the shared
# decision core (core/matching/audio_verification.py) so import-time verification
# and the library scan share ONE definition — the <>-strip fix, CJK handling and
# thresholds can't drift apart again. Names kept (`_normalize` etc.) for existing
# importers/tests.
from core.matching.audio_verification import ( # noqa: E402
normalize as _normalize,
similarity as _similarity,
_alias_aware_artist_sim,
_find_best_title_artist_match as _core_find_best_title_artist_match,
evaluate as _core_evaluate,
Decision as _CoreDecision,
)
def _find_best_title_artist_match(recordings, expected_title, expected_artist,
expected_artist_aliases=None):
"""Back-compat wrapper around the shared core matcher (keeps the
``expected_artist_aliases`` kwarg name for existing callers/tests)."""
return _core_find_best_title_artist_match(
recordings, expected_title, expected_artist, expected_artist_aliases,
)
# Shared MusicBrainz client for enrichment lookups
_mb_client = None
_mb_client_lock = threading.Lock()
# Shared MusicBrainzService for alias lookups (issue #442). Service
# layer wraps the raw client + adds caching + DB access — all of which
# the alias resolution chain (library DB → cache → live MB) needs.
_mb_service = None
_mb_service_lock = threading.Lock()
MAX_MB_ENRICHMENT_LOOKUPS = 3
def _get_mb_client() -> MusicBrainzClient:
"""Get or create a shared MusicBrainz client instance."""
global _mb_client
if _mb_client is None:
with _mb_client_lock:
if _mb_client is None:
_mb_client = MusicBrainzClient()
return _mb_client
def _get_mb_service():
"""Get or create a shared MusicBrainzService instance.
Used by the alias-resolution chain in `verify_audio_file`. Lazy
init so importing this module doesn't trigger a DB connection on
paths that never run AcoustID verification (test runs, dry runs).
"""
global _mb_service
if _mb_service is None:
with _mb_service_lock:
if _mb_service is None:
from core.musicbrainz_service import MusicBrainzService
from database.music_database import get_database
_mb_service = MusicBrainzService(get_database())
return _mb_service
def _resolve_expected_artist_aliases(expected_artist_name: str) -> List[str]:
"""Look up alternate-spelling aliases for the expected artist.
Issue #442 — bridges cross-script artist comparisons (Japanese
kanji ↔ romanized, Cyrillic ↔ Latin, etc.) without forcing the
verifier to know about the resolution chain. Best-effort: any
failure (no MB service, network down, no library DB) returns
empty list so verification falls back to the prior direct
similarity check.
"""
if not expected_artist_name:
return []
try:
return _get_mb_service().lookup_artist_aliases(expected_artist_name)
except Exception as e:
logger.debug("alias lookup failed for %r: %s", expected_artist_name, e)
return []
def _enrich_recordings_from_musicbrainz(
recordings: List[Dict[str, Any]],
) -> List[Dict[str, Any]]:
"""
Enrich recordings that are missing title/artist by looking up their
MBIDs via MusicBrainz.
AcoustID often returns recordings with title=None, artist=None even though
the MBIDs are valid. This resolves the metadata so verification can compare
title/artist instead of skipping.
Args:
recordings: List of recording dicts from fingerprint_and_lookup()
Returns:
The same list, with title/artist filled in where possible.
"""
# Fast path: if any recording already has title AND artist, no enrichment needed
if any(rec.get('title') and rec.get('artist') for rec in recordings):
return recordings
logger.info(f"Enriching {len(recordings)} recordings via MusicBrainz (all missing title/artist)...")
mb = _get_mb_client()
enriched_count = 0
for rec in recordings[:MAX_MB_ENRICHMENT_LOOKUPS]:
mbid = rec.get('mbid')
if not mbid:
continue
try:
data = mb.get_recording(mbid, includes=['artist-credits'])
if not data:
logger.debug(f"MusicBrainz returned no data for recording {mbid}")
continue
title = data.get('title')
artist_credit = data.get('artist-credit', [])
# Build artist string from artist-credit array
# Each entry has {"artist": {"name": "..."}, "joinphrase": "..."}
artist_parts = []
for credit in artist_credit:
name = credit.get('artist', {}).get('name', '')
joinphrase = credit.get('joinphrase', '')
if name:
artist_parts.append(name + joinphrase)
artist = ''.join(artist_parts).strip() if artist_parts else None
if title:
rec['title'] = title
logger.debug(f"Enriched {mbid}: title='{title}'")
if artist:
rec['artist'] = artist
logger.debug(f"Enriched {mbid}: artist='{artist}'")
if title or artist:
enriched_count += 1
except Exception as e:
logger.debug(f"Failed to enrich recording {mbid}: {e}")
continue
logger.info(f"Enriched {enriched_count}/{min(len(recordings), MAX_MB_ENRICHMENT_LOOKUPS)} recordings from MusicBrainz")
return recordings
class AcoustIDVerification:
"""
Verification service that compares audio fingerprint identity
against expected track metadata using title/artist matching.
Design Principle: FAIL OPEN
- Only returns FAIL when we are CONFIDENT the file is wrong
- Any error or uncertainty results in SKIP (continue normally)
- Never blocks downloads due to verification infrastructure issues
Usage:
verifier = AcoustIDVerification()
result, message = verifier.verify_audio_file(
"/path/to/downloaded.mp3",
"Expected Song Title",
"Expected Artist"
)
if result == VerificationResult.FAIL:
# Move to quarantine
else:
# Continue with normal processing (PASS, SKIP, or DISABLED)
"""
def __init__(self):
"""Initialize verification service."""
self.acoustid_client = AcoustIDClient()
def verify_audio_file(
self,
audio_file_path: str,
expected_track_name: str,
expected_artist_name: str,
context: Optional[Dict[str, Any]] = None
) -> Tuple[VerificationResult, str]:
"""
Verify that an audio file matches expected track metadata.
Compares title/artist from AcoustID fingerprint results against
the expected track info. No MusicBrainz lookup needed.
Args:
audio_file_path: Path to the downloaded audio file
expected_track_name: Track name we expected to download
expected_artist_name: Artist name we expected
context: Optional download context for logging/debugging
Returns:
Tuple of (VerificationResult, reason_message)
"""
try:
# Step 1: Check availability
available, reason = self.acoustid_client.is_available()
if not available:
logger.debug(f"AcoustID verification skipped: {reason}")
return VerificationResult.SKIP, reason
# Step 2: Fingerprint and lookup in AcoustID (structured so an
# actual error — invalid key / rate limit / no chromaprint — is
# reported distinctly from a genuine no-match, instead of both
# silently surfacing as "Skipped").
logger.info(f"Fingerprinting and looking up: {audio_file_path}")
lookup = self.acoustid_client.lookup_with_status(audio_file_path) or {}
status = lookup.get('status')
# Infer status by content when absent (a caller/stub that returned
# just recordings): recordings => matched, none => no match.
if status is None:
status = 'ok' if lookup.get('recordings') else 'no_match'
if status in ('error', 'no_backend', 'fingerprint_error', 'unavailable'):
# Something is broken (not the track's fault) — never quarantine
# on this; surface it so the user can fix it.
return VerificationResult.ERROR, lookup.get('error', 'AcoustID lookup failed')
if status != 'ok':
# no_match / unsupported / not_found — genuinely could not verify.
return VerificationResult.SKIP, lookup.get('error', 'No match in AcoustID database')
acoustid_result = lookup
recordings = acoustid_result.get('recordings', [])
best_score = acoustid_result.get('best_score', 0)
if not recordings:
return VerificationResult.SKIP, "No match in AcoustID database"
logger.debug(
f"AcoustID returned {len(recordings)} recording(s) "
f"(best fingerprint score: {best_score:.2f})"
)
# Step 3: Check fingerprint confidence
if best_score < MIN_ACOUSTID_SCORE:
msg = f"AcoustID fingerprint score too low ({best_score:.2f}) to verify"
logger.info(msg)
return VerificationResult.SKIP, msg
# Enrich recordings that are missing title/artist via MusicBrainz lookup
recordings = _enrich_recordings_from_musicbrainz(recordings)
# Issue #442 — alias resolution is LAZY. We pass a memoising
# thunk to the artist-comparison sites; it only fires the
# multi-tier lookup (library DB → cache → live MB) when
# direct artist similarity falls below threshold. Verifications
# where the direct match already passes (the common case for
# same-script artist names) never trigger any lookup work,
# so the fix doesn't add a per-verification DB query for the
# happy path. When the thunk DOES fire, the result is cached
# in the closure so the 3 comparison sites within one
# verification share a single resolution pass.
_alias_cache: Dict[str, Any] = {}
def _aliases_provider() -> List[str]:
if 'value' not in _alias_cache:
resolved = _resolve_expected_artist_aliases(expected_artist_name)
_alias_cache['value'] = resolved
if resolved:
logger.debug(
"Resolved %d aliases for expected artist '%s'",
len(resolved), expected_artist_name,
)
return _alias_cache['value']
# Steps 4-5: delegate the PASS/SKIP/FAIL decision to the shared core
# (core/matching/audio_verification.evaluate) so import verification
# and the library scan apply identical logic.
outcome = _core_evaluate(
expected_track_name, expected_artist_name, recordings,
fingerprint_score=best_score,
aliases_provider=_aliases_provider,
)
logger.info(
"Best match: '%s' by '%s' (title_sim=%.2f, artist_sim=%.2f) -> %s",
outcome.matched_title, outcome.matched_artist,
outcome.title_sim, outcome.artist_sim, outcome.decision.value,
)
_decision_map = {
_CoreDecision.PASS: VerificationResult.PASS,
_CoreDecision.SKIP: VerificationResult.SKIP,
_CoreDecision.FAIL: VerificationResult.FAIL,
}
result = _decision_map[outcome.decision]
if result == VerificationResult.PASS:
logger.info("AcoustID verification PASSED - %s", outcome.reason)
elif result == VerificationResult.FAIL:
logger.warning("AcoustID verification FAILED - %s", outcome.reason)
else:
logger.info("AcoustID verification SKIPPED - %s", outcome.reason)
return result, outcome.reason
except Exception as e:
# Any unexpected error -> SKIP (fail open)
logger.error(f"Unexpected error during AcoustID verification: {e}")
return VerificationResult.SKIP, f"Verification error: {str(e)}"
def quick_check_available(self) -> Tuple[bool, str]:
"""
Quick check if verification is available without doing a full verification.
Returns:
Tuple of (is_available, reason)
"""
return self.acoustid_client.is_available()

515
core/album_consistency.py
View file

@ -1,515 +0,0 @@
"""Picard-style Album Consistency — after all tracks in an album batch finish
post-processing, pick ONE MusicBrainz release and overwrite album-level tags
on every file so they're consistent. Prevents media server album splits.
"""
import os
import threading
from difflib import SequenceMatcher
from typing import Any, Dict, List, Optional
from mutagen import File as MutagenFile
from mutagen.flac import FLAC
from mutagen.id3 import ID3, TALB, TPE2, TXXX
from mutagen.mp4 import MP4, MP4FreeForm
from mutagen.oggvorbis import OggVorbis
from utils.logging_config import get_logger
logger = get_logger("album_consistency")
# Tags written to EVERY file (album-level, same value)
_ALBUM_LEVEL_TAGS = [
'MUSICBRAINZ_RELEASE_ID',
'MUSICBRAINZ_RELEASEGROUPID',
'MUSICBRAINZ_ALBUMARTISTID',
'RELEASETYPE',
'RELEASESTATUS',
'RELEASECOUNTRY',
'ORIGINALDATE',
'BARCODE',
'MEDIA',
'TOTALDISCS',
'CATALOGNUMBER',
'SCRIPT',
'ASIN',
]
# Vorbis comment keys (FLAC/OGG) — same as _ALBUM_LEVEL_TAGS (uppercase)
# ID3 TXXX desc mapping
_ID3_TXXX_MAP = {
'MUSICBRAINZ_RELEASE_ID': 'MusicBrainz Album Id',
'MUSICBRAINZ_RELEASEGROUPID': 'MusicBrainz Release Group Id',
'MUSICBRAINZ_ALBUMARTISTID': 'MusicBrainz Album Artist Id',
'MUSICBRAINZ_RELEASETRACKID': 'MusicBrainz Release Track Id',
'RELEASETYPE': 'MusicBrainz Album Type',
'RELEASESTATUS': 'MusicBrainz Album Status',
'RELEASECOUNTRY': 'MusicBrainz Album Release Country',
'ORIGINALDATE': 'ORIGINALDATE',
'BARCODE': 'BARCODE',
'MEDIA': 'MEDIA',
'TOTALDISCS': 'TOTALDISCS',
'CATALOGNUMBER': 'CATALOGNUMBER',
'SCRIPT': 'SCRIPT',
'ASIN': 'ASIN',
}
# MP4 freeform keys
_MP4_KEY_PREFIX = '----:com.apple.iTunes:'
# ── Picard-style release preference scoring ──
# Preferred countries (higher = better). US/GB/XW(worldwide) are most common
# for English-language music. XE = Europe-wide.
_COUNTRY_SCORES = {
'US': 10, 'XW': 10, 'GB': 8, 'XE': 7, 'CA': 6, 'AU': 5, 'DE': 4,
'FR': 4, 'JP': 3, 'NL': 3, 'SE': 3, 'IT': 2,
}
# Preferred formats (higher = better). Digital/CD are the standard;
# vinyl and cassette are niche reissues that often differ from the
# canonical tracklist.
_FORMAT_SCORES = {
'Digital Media': 10, 'CD': 9, 'Enhanced CD': 8,
'SACD': 7, 'Hybrid SACD': 7, 'Blu-spec CD': 7,
'Vinyl': 3, '12" Vinyl': 3, '7" Vinyl': 2,
'Cassette': 1,
}
# Release status preference
_STATUS_SCORES = {
'Official': 10, 'Promotion': 5, 'Bootleg': 1, 'Pseudo-Release': 1,
}
def _score_release(release: dict, expected_track_count: int) -> float:
"""Score a MusicBrainz release for preference ranking.
Higher score = better candidate. Factors:
- Track count match (most important — wrong count is wrong release)
- Release status (Official > Promo > Bootleg)
- Country preference (US/worldwide > regional)
- Format preference (Digital/CD > Vinyl > Cassette)
- Has barcode (sign of a real commercial release)
- Penalize releases with no media info (incomplete data)
"""
score = 0.0
# Track count match (0-40 points, biggest factor)
media = release.get('media', [])
mb_track_count = sum(len(m.get('tracks') or m.get('track-list', []))
for m in media)
track_diff = abs(mb_track_count - expected_track_count)
if track_diff == 0:
score += 40
elif track_diff <= 1:
score += 30
elif track_diff <= 2:
score += 20
elif track_diff <= 5:
score += 10
# else: 0 points
# Status (0-10 points)
status = release.get('status', '')
score += _STATUS_SCORES.get(status, 2)
# Country (0-10 points)
country = release.get('country', '')
score += _COUNTRY_SCORES.get(country, 1)
# Format from first medium (0-10 points)
if media:
fmt = media[0].get('format', '')
score += _FORMAT_SCORES.get(fmt, 4)
else:
score -= 5 # No media info = suspect
# Barcode (0-3 points) — real commercial releases have barcodes
if release.get('barcode'):
score += 3
# Date completeness (0-2 points) — prefer releases with full dates
date = release.get('date', '')
if len(date) >= 10:
score += 2 # Full YYYY-MM-DD
elif len(date) >= 4:
score += 1 # Year only
return score
def _normalize_title(s):
"""Normalize a title for comparison."""
import re
if not s:
return ''
s = s.lower().strip()
s = re.sub(r'\s*[\(\[].*?[\)\]]\s*', ' ', s) # Strip parentheticals/brackets
s = re.sub(r'[^\w\s]', '', s) # Strip punctuation
return ' '.join(s.split())
def _find_best_release(album_name, artist_name, track_count, mb_service):
"""Search MusicBrainz for the best release matching this album.
Uses Picard-style preference scoring: track count match, release status,
country (US/worldwide preferred), format (Digital/CD preferred), barcode
presence, and date completeness. Deterministic — same inputs always
produce the same release.
"""
try:
import re
# Build search name variants
search_names = [album_name]
stripped = re.sub(
r'\s*[\(\[]'
r'[^)\]]*'
r'(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard|edition)'
r'[^)\]]*'
r'[\)\]]',
'', album_name, flags=re.IGNORECASE
).strip()
stripped = re.sub(
r'\s+(?:-\s+)?(?:deluxe|expanded|remaster(?:ed)?|anniversary|special|collector|'
r'limited|bonus|platinum|gold|super\s*deluxe|standard)'
r'(?:\s+(?:edition|version))?\s*$',
'', stripped, flags=re.IGNORECASE
).strip()
if stripped and stripped.lower() != album_name.lower():
search_names.append(stripped)
# Collect candidate release MBIDs from all search variants
candidate_mbids = []
for name in search_names:
# Try cached match first
match = mb_service.match_release(name, artist_name)
if match and match.get('mbid'):
candidate_mbids.append(match['mbid'])
# Also try direct search for more candidates
try:
search_results = mb_service.mb_client.search_release(name, artist_name, limit=5)
for sr in (search_results or []):
sr_id = sr.get('id', '')
if sr_id and sr_id not in candidate_mbids:
candidate_mbids.append(sr_id)
except Exception as e:
logger.debug("search_release fallback failed: %s", e)
if not candidate_mbids:
logger.info(f"No MB release found for '{album_name}' by '{artist_name}'")
return None
# Fetch full release data for each candidate and score them
best_release = None
best_score = -1
for mbid in candidate_mbids[:8]: # Cap at 8 to limit API calls
try:
release = mb_service.mb_client.get_release(
mbid, includes=['recordings', 'release-groups', 'labels',
'media', 'artist-credits']
)
if not release:
continue
score = _score_release(release, track_count)
if score > best_score:
best_score = score
best_release = release
except Exception:
continue
if best_release:
mb_count = sum(len(m.get('tracks') or m.get('track-list', []))
for m in best_release.get('media', []))
logger.info(
f"Selected release '{best_release.get('title')}' "
f"({best_release.get('id', '')[:8]}...) — "
f"score={best_score:.0f}, tracks={mb_count}, "
f"country={best_release.get('country', '?')}, "
f"format={best_release.get('media', [{}])[0].get('format', '?')}, "
f"status={best_release.get('status', '?')}"
)
return best_release
except Exception as e:
logger.error(f"Error finding best release for '{album_name}': {e}")
return None
def _match_files_to_tracklist(file_infos, release):
"""Match downloaded files to MB release tracklist entries.
Returns {file_path: mb_track_entry} for matched files."""
# Build MB tracklist lookup: (disc, track) -> track entry
mb_lookup = {}
for medium in release.get('media', []):
disc_num = medium.get('position', 1)
for track in (medium.get('tracks') or medium.get('track-list', [])):
pos = track.get('position', track.get('number', 0))
try:
pos = int(pos)
except (ValueError, TypeError):
continue
mb_lookup[(disc_num, pos)] = track
matched = {}
unmatched = []
# Pass 1: exact disc+track number match
for fi in file_infos:
key = (fi.get('disc_number', 1), fi.get('track_number', 1))
if key in mb_lookup:
matched[fi['path']] = mb_lookup[key]
else:
unmatched.append(fi)
# Pass 2: title similarity for unmatched
remaining_mb = {k: v for k, v in mb_lookup.items() if v not in matched.values()}
for fi in unmatched:
norm_title = _normalize_title(fi.get('title', ''))
best_score = 0
best_entry = None
for _key, mb_track in remaining_mb.items():
recording = mb_track.get('recording', {})
mb_title = _normalize_title(recording.get('title', ''))
if not mb_title:
continue
score = SequenceMatcher(None, norm_title, mb_title).ratio()
if score > best_score:
best_score = score
best_entry = mb_track
if best_entry and best_score >= 0.70:
matched[fi['path']] = best_entry
# Remove from remaining so it's not double-matched
remaining_mb = {k: v for k, v in remaining_mb.items() if v is not best_entry}
return matched
def _write_tag_to_file(audio, tag_key, value):
"""Write a single custom tag to an audio file (Mutagen object)."""
if value is None:
return
value = str(value)
try:
if isinstance(audio.tags, ID3):
desc = _ID3_TXXX_MAP.get(tag_key, tag_key)
# Remove existing TXXX with this desc
to_remove = [k for k in audio.tags if k.startswith('TXXX:') and desc in k]
for k in to_remove:
del audio.tags[k]
audio.tags.add(TXXX(encoding=3, desc=desc, text=[value]))
elif isinstance(audio, (FLAC, OggVorbis)):
audio[tag_key] = [value]
elif isinstance(audio, MP4):
key = _MP4_KEY_PREFIX + _ID3_TXXX_MAP.get(tag_key, tag_key)
audio[key] = [MP4FreeForm(value.encode('utf-8'))]
except Exception as e:
logger.debug(f"Failed to write {tag_key}: {e}")
def _write_standard_tag(audio, tag_name, value):
"""Write album/albumartist standard tags."""
if value is None:
return
try:
if isinstance(audio.tags, ID3):
if tag_name == 'album':
audio.tags.delall('TALB')
audio.tags.add(TALB(encoding=3, text=[value]))
elif tag_name == 'albumartist':
audio.tags.delall('TPE2')
audio.tags.add(TPE2(encoding=3, text=[value]))
elif isinstance(audio, (FLAC, OggVorbis)):
audio[tag_name.upper()] = [value]
elif isinstance(audio, MP4):
tag_map = {'album': '\xa9alb', 'albumartist': 'aART'}
key = tag_map.get(tag_name)
if key:
audio[key] = [value]
except Exception as e:
logger.debug(f"Failed to write standard tag {tag_name}: {e}")
def run_album_consistency(
file_infos: List[Dict[str, Any]],
album_name: str,
artist_name: str,
mb_service: Any,
total_discs: int = 1,
file_lock_fn=None,
) -> Dict[str, Any]:
"""
Picard-style album consistency: pick ONE MusicBrainz release for the album,
then overwrite album-level tags on all files to match.
Args:
file_infos: List of {path, track_number, disc_number, title}
album_name: Album name from download context
artist_name: Artist name from download context
mb_service: MusicBrainzService instance
total_discs: Number of discs in the album
file_lock_fn: Optional function(path) -> context manager for thread-safe writes
Returns:
{success, release_mbid, matched_tracks, total_files, tags_written, error}
"""
result = {
'success': False,
'release_mbid': None,
'matched_tracks': 0,
'total_files': len(file_infos),
'tags_written': 0,
'error': None,
}
if not file_infos:
result['error'] = 'No files provided'
return result
if not mb_service:
result['error'] = 'MusicBrainz service not available'
return result
# Step 1: Find the best release
release = _find_best_release(album_name, artist_name, len(file_infos), mb_service)
if not release:
result['error'] = f'No MusicBrainz release found for "{album_name}"'
return result
release_mbid = release.get('id', '')
result['release_mbid'] = release_mbid
# Step 2: Match files to tracklist
matched = _match_files_to_tracklist(file_infos, release)
result['matched_tracks'] = len(matched)
if len(matched) < len(file_infos) * 0.5:
result['error'] = (f'Only {len(matched)}/{len(file_infos)} tracks matched the release — '
f'aborting to avoid incorrect tagging')
return result
# Step 3: Build album-level tags (same for all files)
album_tags = {}
album_tags['MUSICBRAINZ_RELEASE_ID'] = release_mbid
rg = release.get('release-group', {})
if rg.get('id'):
album_tags['MUSICBRAINZ_RELEASEGROUPID'] = rg['id']
if rg.get('primary-type'):
album_tags['RELEASETYPE'] = rg['primary-type']
if rg.get('first-release-date'):
album_tags['ORIGINALDATE'] = rg['first-release-date']
ac = release.get('artist-credit', [])
if ac and isinstance(ac[0], dict):
aa = ac[0].get('artist', {})
if aa.get('id'):
album_tags['MUSICBRAINZ_ALBUMARTISTID'] = aa['id']
if release.get('status'):
album_tags['RELEASESTATUS'] = release['status']
if release.get('country'):
album_tags['RELEASECOUNTRY'] = release['country']
if release.get('barcode'):
album_tags['BARCODE'] = release['barcode']
media_list = release.get('media', [])
if media_list:
fmt = media_list[0].get('format', '')
if fmt:
album_tags['MEDIA'] = fmt
album_tags['TOTALDISCS'] = str(len(media_list))
label_info = release.get('label-info', [])
if label_info and isinstance(label_info[0], dict):
cat = label_info[0].get('catalog-number', '')
if cat:
album_tags['CATALOGNUMBER'] = cat
text_rep = release.get('text-representation', {})
if isinstance(text_rep, dict) and text_rep.get('script'):
album_tags['SCRIPT'] = text_rep['script']
if release.get('asin'):
album_tags['ASIN'] = release['asin']
# Album name and artist from the release (canonical MB values)
release_album_name = release.get('title', album_name)
release_artist_name = artist_name
if ac:
# Build full artist credit string
parts = []
for credit in ac:
if isinstance(credit, dict):
parts.append(credit.get('artist', {}).get('name', ''))
parts.append(credit.get('joinphrase', ''))
elif isinstance(credit, str):
parts.append(credit)
full_credit = ''.join(parts).strip()
if full_credit:
release_artist_name = full_credit
# Step 4: Write tags to matched files only (unmatched files keep their existing tags)
tags_written = 0
for fi in file_infos:
file_path = fi['path']
mb_track = matched.get(file_path)
# Only write to files that matched the tracklist — avoids corrupting
# bonus tracks or files from a different edition
if not mb_track:
continue
if not os.path.exists(file_path):
continue
try:
if file_lock_fn:
lock = file_lock_fn(file_path)
else:
lock = _DummyLock()
with lock:
audio = MutagenFile(file_path, easy=False)
if audio is None:
continue
# Write album-level tags
for tag_key, value in album_tags.items():
_write_tag_to_file(audio, tag_key, value)
# Write standard album/albumartist tags
_write_standard_tag(audio, 'album', release_album_name)
_write_standard_tag(audio, 'albumartist', release_artist_name)
# Write per-track tag (release track ID) if matched
if mb_track and mb_track.get('id'):
_write_tag_to_file(audio, 'MUSICBRAINZ_RELEASETRACKID', mb_track['id'])
audio.save()
tags_written += 1
except Exception as e:
logger.error(f"Error writing consistency tags to {file_path}: {e}")
result['tags_written'] = tags_written
result['success'] = tags_written > 0
logger.info(f"Album consistency complete: {tags_written}/{len(file_infos)} files tagged "
f"with release '{release_album_name}' ({release_mbid[:8]}...)")
return result
class _DummyLock:
"""No-op context manager when no file lock is provided."""
def __enter__(self):
return self
def __exit__(self, *args):
pass

796
core/amazon_client.py
View file

@ -1,796 +0,0 @@
"""Amazon Music metadata client backed by a T2Tunes proxy instance.
T2Tunes exposes the Amazon Music catalog (search, album metadata, stream info)
through a simple REST API. This module wraps those endpoints and normalises the
responses into the same Track / Artist / Album dataclass shape used by
DeezerClient and iTunesClient.
Endpoints used:
GET /api/status
GET /api/amazon-music/search
GET /api/amazon-music/metadata
GET /api/amazon-music/media-from-asin
Config keys (all optional — fall back to public defaults):
amazon.base_url T2Tunes instance URL (default: https://t2tunes.site)
amazon.country ISO-3166 country code (default: US)
amazon.preferred_codec Preferred audio codec (default: flac)
"""
from __future__ import annotations
import re
import threading
import time
from dataclasses import dataclass
from typing import Any, Dict, Iterator, List, Optional
from urllib.parse import urljoin
import requests
from config.settings import config_manager
from core.api_call_tracker import api_call_tracker
from utils.logging_config import get_logger
logger = get_logger("amazon_client")
# Strips featuring credits like "Artist feat. X", "Artist ft. Y" so artist
# deduplication works on the primary artist name only.
_FEAT_RE = re.compile(r'\s+(?:feat(?:uring)?\.?|ft\.?)\s+.*', re.IGNORECASE)
# Strips the Explicit marker — explicit is treated as the default version.
# Clean/Edited/Censored stay in the name so users can distinguish them.
_EDITION_RE = re.compile(r'\s*[\[\(]explicit[\]\)]', re.IGNORECASE)
def _primary_artist(name: str) -> str:
return _FEAT_RE.sub('', name).strip()
def _strip_edition(name: str) -> str:
return _EDITION_RE.sub('', name).strip()
def _unslugify(name: str) -> str:
"""Convert a slug-form artist ID (e.g. 'kendrick_lamar') to a search name."""
return name.replace('_', ' ')
DEFAULT_BASE_URL = "https://t2tunes.site"
DEFAULT_COUNTRY = "US"
DEFAULT_CODEC = "flac"
MIN_API_INTERVAL = 0.5 # seconds — T2Tunes has no published rate limit
_last_api_call: float = 0.0
_api_call_lock = threading.Lock()
_META_CACHE_TTL = 300 # seconds
_meta_cache: Dict[str, tuple] = {} # asin -> (fetched_at, meta_dict)
_meta_cache_lock = threading.Lock()
class AmazonClientError(RuntimeError):
"""Raised on unrecoverable T2Tunes API errors.
Carries the HTTP ``status_code`` when the failure was an HTTP error, so
callers (the worker's outage detection) can tell a source outage (5xx) from
a per-item miss without parsing the message.
"""
def __init__(self, *args, status_code=None):
super().__init__(*args)
self.status_code = status_code
# ---------------------------------------------------------------------------
# Dataclasses — field layout matches DeezerClient / iTunesClient exactly
# ---------------------------------------------------------------------------
@dataclass
class Track:
id: str # Amazon track ASIN
name: str
artists: List[str]
album: str
duration_ms: int
popularity: int
preview_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
image_url: Optional[str] = None
release_date: Optional[str] = None
track_number: Optional[int] = None
disc_number: Optional[int] = None
album_type: Optional[str] = None
total_tracks: Optional[int] = None
isrc: Optional[str] = None
@classmethod
def from_search_hit(cls, doc: Dict[str, Any]) -> "Track":
return cls(
id=str(doc.get("asin") or ""),
name=str(doc.get("title") or ""),
artists=[str(doc.get("artistName") or "Unknown Artist")],
album=str(doc.get("albumName") or ""),
duration_ms=int(doc.get("duration") or 0) * 1000,
popularity=0,
isrc=str(doc.get("isrc") or "") or None,
)
@classmethod
def from_stream_info(
cls,
stream: "T2TunesStreamInfo",
album_meta: Optional[Dict[str, Any]] = None,
) -> "Track":
album = album_meta or {}
return cls(
id=stream.asin,
name=stream.title,
artists=[stream.artist] if stream.artist else ["Unknown Artist"],
album=stream.album,
duration_ms=0,
popularity=0,
image_url=album.get("image"),
release_date=album.get("release_date"),
total_tracks=album.get("trackCount"),
isrc=stream.isrc or None,
)
@dataclass
class Artist:
id: str # Slugified artist name — T2Tunes exposes no artist IDs
name: str
popularity: int
genres: List[str]
followers: int
image_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
@classmethod
def from_name(cls, name: str) -> "Artist":
slug = name.lower().replace(" ", "_")
return cls(id=slug, name=name, popularity=0, genres=[], followers=0)
@dataclass
class Album:
id: str # Amazon album ASIN
name: str
artists: List[str]
release_date: str
total_tracks: int
album_type: str
image_url: Optional[str] = None
external_urls: Optional[Dict[str, str]] = None
explicit: Optional[bool] = None
@classmethod
def from_search_hit(cls, doc: Dict[str, Any]) -> "Album":
return cls(
id=str(doc.get("albumAsin") or doc.get("asin") or ""),
name=str(doc.get("albumName") or doc.get("title") or ""),
artists=[str(doc.get("artistName") or "Unknown Artist")],
release_date="",
total_tracks=0,
album_type="album",
)
@classmethod
def from_metadata(cls, album_meta: Dict[str, Any], asin: str = "") -> "Album":
return cls(
id=str(album_meta.get("asin") or asin or ""),
name=str(album_meta.get("title") or ""),
artists=[str(album_meta.get("artistName") or "Unknown Artist")],
release_date=str(album_meta.get("release_date") or album_meta.get("releaseDate") or ""),
total_tracks=int(album_meta.get("trackCount") or 0),
album_type="album",
image_url=album_meta.get("image"),
explicit=album_meta.get("explicit"),
)
# ---------------------------------------------------------------------------
# Internal dataclasses for raw T2Tunes response parsing
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class T2TunesSearchItem:
asin: str
title: str
artist_name: str
item_type: str
album_name: str = ""
album_asin: str = ""
duration_seconds: int = 0
isrc: str = ""
@property
def is_album(self) -> bool:
return "album" in self.item_type.lower()
@property
def is_track(self) -> bool:
return "track" in self.item_type.lower()
@dataclass(frozen=True)
class T2TunesStreamInfo:
asin: str
streamable: bool
codec: str
format: str
sample_rate: Optional[int]
stream_url: str
decryption_key: Optional[str] # hex-encoded AES key; None when stream is clear
title: str = ""
artist: str = ""
album: str = ""
isrc: str = ""
cover_url: str = ""
track_number: Optional[int] = None
disc_number: Optional[int] = None
genre: str = ""
label: str = ""
date: str = ""
@property
def has_decryption_key(self) -> bool:
return bool(self.decryption_key)
# ---------------------------------------------------------------------------
# Rate-limit enforcement
# ---------------------------------------------------------------------------
def _rate_limit() -> None:
global _last_api_call
with _api_call_lock:
elapsed = time.monotonic() - _last_api_call
if elapsed < MIN_API_INTERVAL:
time.sleep(MIN_API_INTERVAL - elapsed)
_last_api_call = time.monotonic()
api_call_tracker.record_call("amazon")
# ---------------------------------------------------------------------------
# Main client
# ---------------------------------------------------------------------------
class AmazonClient:
"""T2Tunes-backed Amazon Music metadata and stream-info client."""
def __init__(
self,
base_url: Optional[str] = None,
country: Optional[str] = None,
preferred_codec: Optional[str] = None,
timeout: int = 30,
session: Optional[Any] = None,
) -> None:
self.base_url = (base_url or config_manager.get("amazon.base_url", DEFAULT_BASE_URL)).rstrip("/")
self.country = (country or config_manager.get("amazon.country", DEFAULT_COUNTRY)).upper()
self.preferred_codec = (
preferred_codec or config_manager.get("amazon.preferred_codec", DEFAULT_CODEC)
).lower()
self.timeout = timeout
self.session: Any = session or requests.Session()
if isinstance(self.session, requests.Session):
self.session.headers.update({
"Accept": "application/json",
"User-Agent": "SoulSync/1.0",
"Referer": self.base_url,
})
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def reload_config(self) -> None:
self.base_url = config_manager.get("amazon.base_url", DEFAULT_BASE_URL).rstrip("/")
self.country = config_manager.get("amazon.country", DEFAULT_COUNTRY).upper()
self.preferred_codec = config_manager.get("amazon.preferred_codec", DEFAULT_CODEC).lower()
def is_authenticated(self) -> bool:
"""Return True when the T2Tunes instance reports Amazon Music as up."""
try:
return str(self.status().get("amazonMusic", "")).lower() == "up"
except AmazonClientError:
return False
# ------------------------------------------------------------------
# Low-level API wrappers
# ------------------------------------------------------------------
def status(self) -> Dict[str, Any]:
return self._get_json("/api/status")
def search_raw(self, query: str, *, types: str = "track,album") -> List[T2TunesSearchItem]:
data = self._get_json(
"/api/amazon-music/search",
params={"query": query, "types": types, "country": self.country},
)
return list(self._iter_search_items(data))
def album_metadata(self, asin: str) -> Dict[str, Any]:
return self._get_json(
"/api/amazon-music/metadata",
params={"asin": asin, "country": self.country},
)
def media_from_asin(self, asin: str, codec: Optional[str] = None) -> List[T2TunesStreamInfo]:
effective_codec = (codec or self.preferred_codec).lower()
data = self._get_json(
"/api/amazon-music/media-from-asin",
params={"asin": asin, "country": self.country, "codec": effective_codec},
)
if isinstance(data, list):
return [self._parse_stream_info(item) for item in data if isinstance(item, dict)]
if isinstance(data, dict):
return [self._parse_stream_info(data)]
raise AmazonClientError(f"Unexpected media-from-asin response type: {type(data).__name__}")
# ------------------------------------------------------------------
# Metadata interface — mirrors DeezerClient / iTunesClient signatures
# ------------------------------------------------------------------
def search_tracks(self, query: str, limit: int = 20) -> List[Track]:
_rate_limit()
items = self.search_raw(query, types="track")
track_pairs: List[tuple] = [] # (Track, album_asin)
seen_album_asins: List[str] = []
for item in items:
if not item.is_track:
continue
track = Track.from_search_hit({
"asin": item.asin,
"title": _strip_edition(item.title),
"artistName": _primary_artist(item.artist_name),
"albumName": _strip_edition(item.album_name),
"albumAsin": item.album_asin,
"duration": item.duration_seconds,
"isrc": item.isrc,
})
track_pairs.append((track, item.album_asin))
if item.album_asin and item.album_asin not in seen_album_asins:
seen_album_asins.append(item.album_asin)
if len(track_pairs) >= limit:
break
album_metas = self._fetch_album_metas(seen_album_asins[:5])
tracks: List[Track] = []
for track, album_asin in track_pairs:
if album_asin and album_asin in album_metas:
meta = album_metas[album_asin]
track.image_url = meta.get("image")
track.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
track.total_tracks = meta.get("trackCount")
tracks.append(track)
return tracks
def search_artists(self, query: str, limit: int = 20) -> List[Artist]:
_rate_limit()
items = self.search_raw(query, types="track")
seen: Dict[str, Artist] = {}
artist_album_asin: Dict[str, str] = {} # artist name → first album ASIN seen
for item in items:
name = _primary_artist(item.artist_name)
if not name:
continue
if name not in seen:
seen[name] = Artist.from_name(name)
if name not in artist_album_asin and item.album_asin:
artist_album_asin[name] = item.album_asin
if len(seen) >= limit:
break
# T2Tunes has no artist images — use an album cover as stand-in.
unique_asins = list({v for v in artist_album_asin.values()})[:5]
album_metas = self._fetch_album_metas(unique_asins)
for name, artist in seen.items():
asin = artist_album_asin.get(name)
if asin and asin in album_metas:
artist.image_url = album_metas[asin].get("image")
return list(seen.values())
def search_albums(self, query: str, limit: int = 20) -> List[Album]:
_rate_limit()
items = self.search_raw(query, types="track")
album_candidates: List[tuple] = [] # (Album, asin)
seen_keys: set = set()
for item in items:
album_asin = item.album_asin
raw_name = item.album_name
if not raw_name:
continue
display_name = _strip_edition(raw_name)
artist = _primary_artist(item.artist_name)
# Collapse Explicit/Clean variants: same normalised name + artist = same album
dedup_key = (display_name.lower(), artist.lower())
if dedup_key in seen_keys:
continue
seen_keys.add(dedup_key)
album = Album.from_search_hit({
"albumAsin": album_asin,
"albumName": display_name,
"artistName": artist,
})
album_candidates.append((album, album_asin))
if len(album_candidates) >= limit:
break
album_metas = self._fetch_album_metas([a for _, a in album_candidates[:10]])
albums: List[Album] = []
for album, asin in album_candidates:
if asin in album_metas:
meta = album_metas[asin]
album.image_url = meta.get("image")
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
album.total_tracks = int(meta.get("trackCount") or 0)
albums.append(album)
return albums
def get_track_details(self, asin: str) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible dict for a single track ASIN."""
_rate_limit()
try:
streams = self.media_from_asin(asin)
except AmazonClientError:
return None
if not streams:
return None
s = streams[0]
album_data: Dict[str, Any] = {}
try:
meta = self.album_metadata(asin)
albums = meta.get("albumList")
if isinstance(albums, list) and albums and isinstance(albums[0], dict):
album_data = albums[0]
except AmazonClientError:
pass
return {
"id": s.asin,
"name": _strip_edition(s.title),
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
"album": {
"id": album_data.get("asin", ""),
"name": _strip_edition(s.album),
"images": [{"url": album_data["image"]}] if album_data.get("image") else [],
"release_date": album_data.get("release_date") or album_data.get("releaseDate") or s.date or "",
"total_tracks": album_data.get("trackCount", 0),
},
"duration_ms": 0,
"popularity": 0,
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
"track_number": s.track_number,
"disc_number": s.disc_number,
"isrc": s.isrc,
"is_album_track": True,
"raw_data": {
"codec": s.codec,
"format": s.format,
"sample_rate": s.sample_rate,
"streamable": s.streamable,
"has_decryption_key": s.has_decryption_key,
},
}
def get_album(self, asin: str, include_tracks: bool = True) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible album dict."""
_rate_limit()
try:
meta = self.album_metadata(asin)
except AmazonClientError:
return None
albums = meta.get("albumList")
if not isinstance(albums, list) or not albums:
return None
album = albums[0] if isinstance(albums[0], dict) else {}
result: Dict[str, Any] = {
"id": asin,
"name": _strip_edition(album.get("title", "")),
"artists": [{"name": _primary_artist(album.get("artistName", "")), "id": ""}],
"release_date": album.get("release_date") or album.get("releaseDate") or "",
"total_tracks": album.get("trackCount", 0),
"album_type": "album",
"images": [{"url": album["image"]}] if album.get("image") else [],
"external_urls": {"amazon": f"https://music.amazon.com/albums/{asin}"},
"label": album.get("label", ""),
}
if include_tracks:
tracks_data = self.get_album_tracks(asin) or {
"items": [], "total": 0, "limit": 50, "next": None,
}
result["tracks"] = tracks_data
# Backfill release_date from stream tags when album metadata lacks it.
if not result["release_date"]:
items = tracks_data.get("items") or []
for item in items:
rd = item.get("release_date") or ""
if rd and len(rd) >= 4:
result["release_date"] = rd
break
return result
def get_album_tracks(self, asin: str) -> Optional[Dict[str, Any]]:
"""Return album tracks in Spotify pagination format."""
_rate_limit()
try:
streams = self.media_from_asin(asin)
except AmazonClientError:
return None
# media_from_asin has no duration — enrich from search results which do.
duration_map: Dict[str, int] = {} # track asin → duration_ms
if streams:
album_name = _strip_edition(streams[0].album)
artist_name = _primary_artist(streams[0].artist)
try:
search_items = self.search_raw(
f"{album_name} {artist_name}", types="track"
)
for item in search_items:
if item.album_asin == asin and item.duration_seconds:
duration_map[item.asin] = item.duration_seconds * 1000
except Exception as exc:
logger.debug("Duration backfill failed for ASIN %s: %s", asin, exc)
items = [
{
"id": s.asin,
"name": _strip_edition(s.title),
"artists": [{"name": _primary_artist(s.artist), "id": ""}],
"duration_ms": duration_map.get(s.asin, 0),
"track_number": s.track_number if s.track_number is not None else idx + 1,
"disc_number": s.disc_number,
"release_date": s.date or "",
"isrc": s.isrc,
}
for idx, s in enumerate(streams)
]
return {"items": items, "total": len(items), "limit": 50, "next": None}
def get_artist(self, artist_name: str) -> Optional[Dict[str, Any]]:
"""Return a Spotify-compatible artist dict inferred from search results."""
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(search_name, types="track")
except AmazonClientError:
return None
name_lower = search_name.lower()
match = next(
(i for i in items if _primary_artist(i.artist_name).lower() == name_lower),
next((i for i in items if name_lower in _primary_artist(i.artist_name).lower()), None),
)
if not match:
return None
return {
"id": match.artist_name.lower().replace(" ", "_"),
"name": match.artist_name,
"genres": [],
"popularity": 0,
"followers": {"total": 0},
"images": [],
"external_urls": {},
}
def get_artist_albums(
self,
artist_name: str,
album_type: str = "album,single",
limit: int = 200,
) -> List[Album]:
"""Return albums for an artist inferred from search results."""
_rate_limit()
search_name = _unslugify(artist_name)
try:
items = self.search_raw(f"{search_name} album", types="track")
except AmazonClientError:
return []
album_candidates: List[tuple] = [] # (Album, asin)
seen_asins: set = set()
name_lower = search_name.lower()
for item in items:
if not item.is_album:
continue
if _primary_artist(item.artist_name).lower() != name_lower:
continue
album_asin = item.album_asin or item.asin
if album_asin in seen_asins:
continue
seen_asins.add(album_asin)
album = Album.from_search_hit({
"albumAsin": album_asin,
"albumName": _strip_edition(item.album_name or item.title),
"artistName": _primary_artist(item.artist_name),
})
album_candidates.append((album, album_asin))
if len(album_candidates) >= limit:
break
# Fetch metadata for art, release_date, track_count, and type inference.
# No explicit cap here — search_raw naturally bounds results to ~20 items,
# so album_candidates is already small.
asins_to_fetch = [a for _, a in album_candidates]
metas = self._fetch_album_metas(asins_to_fetch)
albums: List[Album] = []
for album, asin in album_candidates:
meta = metas.get(asin, {})
if meta:
album.image_url = meta.get("image")
album.release_date = str(meta.get("release_date") or meta.get("releaseDate") or "")
total = int(meta.get("trackCount") or 0)
album.total_tracks = total
# T2Tunes doesn't expose release type — infer from track count.
# 1-track releases are singles; keep default "album" otherwise.
if total == 1:
album.album_type = "single"
albums.append(album)
return albums
def get_track_features(self, track_id: str) -> Optional[Dict[str, Any]]:
"""Not available from Amazon Music — returns None for compatibility."""
return None
def _get_artist_image_from_albums(self, artist_id: str) -> Optional[str]:
"""Return an album cover as artist image stand-in (T2Tunes has no artist images)."""
search_name = _unslugify(artist_id)
try:
items = self.search_raw(search_name, types="track")
except AmazonClientError:
return None
name_lower = search_name.lower()
for item in items:
if _primary_artist(item.artist_name).lower() != name_lower:
continue
asin = item.album_asin or item.asin
if not asin:
continue
metas = self._fetch_album_metas([asin])
if asin in metas and metas[asin].get("image"):
return metas[asin]["image"]
return None
# ==================== Interface Aliases (match DeezerClient method names) ====================
get_album_metadata = get_album
get_artist_info = get_artist
get_artist_albums_list = get_artist_albums
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
def _fetch_album_metas(self, asins: List[str]) -> Dict[str, Dict[str, Any]]:
"""Parallel-fetch album metadata for up to N ASINs. Returns {asin: albumList[0]}."""
if not asins:
return {}
metas: Dict[str, Dict[str, Any]] = {}
def _fetch(asin: str) -> None:
now = time.monotonic()
with _meta_cache_lock:
entry = _meta_cache.get(asin)
if entry and (now - entry[0]) < _META_CACHE_TTL:
metas[asin] = entry[1]
return
_rate_limit()
try:
raw = self.album_metadata(asin)
lst = raw.get("albumList")
if isinstance(lst, list) and lst and isinstance(lst[0], dict):
meta = lst[0]
with _meta_cache_lock:
_meta_cache[asin] = (time.monotonic(), meta)
metas[asin] = meta
except Exception as exc:
logger.debug("Album metadata fetch failed for ASIN %s: %s", asin, exc)
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=min(len(asins), 5)) as pool:
list(pool.map(_fetch, asins))
return metas
def _get_json(self, path: str, params: Optional[Dict[str, Any]] = None) -> Any:
url = urljoin(f"{self.base_url}/", path.lstrip("/"))
last_exc: Optional[Exception] = None
for attempt in range(3):
if attempt:
time.sleep(1.0 * attempt)
try:
resp = self.session.get(url, params=params, timeout=self.timeout)
resp.raise_for_status()
except requests.HTTPError as exc:
body = exc.response.text[:500].replace("\n", " ")
# T2Tunes returns 400 for transient Amazon-side failures — retry those.
if exc.response.status_code == 400 and "Failed to search" in exc.response.text:
logger.debug("T2Tunes transient 400 on attempt %d, retrying: %s", attempt + 1, url)
last_exc = AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}"
)
continue
raise AmazonClientError(
f"HTTP {exc.response.status_code} for {url} — body: {body!r}",
status_code=exc.response.status_code,
) from exc
except requests.RequestException as exc:
raise AmazonClientError(f"Request failed for {url}: {exc}") from exc
try:
return resp.json()
except ValueError as exc:
preview = resp.text[:200].replace("\n", " ")
raise AmazonClientError(f"Response not JSON for {url}: {preview!r}") from exc
raise last_exc # type: ignore[misc]
@staticmethod
def _iter_search_items(response: Any) -> Iterator[T2TunesSearchItem]:
if not isinstance(response, dict):
raise AmazonClientError(
f"Unexpected search response type: {type(response).__name__}"
)
for result in response.get("results") or []:
if not isinstance(result, dict):
continue
for hit in result.get("hits") or []:
if not isinstance(hit, dict):
continue
doc = hit.get("document")
if not isinstance(doc, dict):
continue
asin = str(doc.get("asin") or "")
if not asin:
continue
yield T2TunesSearchItem(
asin=asin,
title=str(doc.get("title") or ""),
artist_name=str(doc.get("artistName") or ""),
item_type=str(doc.get("__type") or ""),
album_name=str(doc.get("albumName") or ""),
album_asin=str(doc.get("albumAsin") or ""),
duration_seconds=int(doc.get("duration") or 0),
isrc=str(doc.get("isrc") or ""),
)
@staticmethod
def _parse_stream_info(item: Dict[str, Any]) -> T2TunesStreamInfo:
stream_info = item.get("streamInfo") if isinstance(item.get("streamInfo"), dict) else {}
tags = item.get("tags") if isinstance(item.get("tags"), dict) else {}
# T2Tunes API has a typo: "stremeable" in some responses
streamable = item.get("streamable")
if streamable is None:
streamable = item.get("stremeable")
raw_key = item.get("decryptionKey")
decryption_key = str(raw_key) if raw_key else None
def _int_tag(key: str) -> Optional[int]:
v = tags.get(key)
try:
return int(v) if v is not None else None
except (TypeError, ValueError):
return None
return T2TunesStreamInfo(
asin=str(item.get("asin") or ""),
streamable=bool(streamable),
codec=str(stream_info.get("codec") or ""),
format=str(stream_info.get("format") or ""),
sample_rate=(
stream_info.get("sampleRate")
if isinstance(stream_info.get("sampleRate"), int)
else None
),
stream_url=str(stream_info.get("streamUrl") or ""),
decryption_key=decryption_key,
title=str(tags.get("title") or ""),
artist=str(tags.get("artist") or ""),
album=str(tags.get("album") or ""),
isrc=str(tags.get("isrc") or ""),
cover_url=str(item.get("coverUrl") or ""),
track_number=_int_tag("trackNumber"),
disc_number=_int_tag("discNumber"),
genre=str(tags.get("genre") or ""),
label=str(tags.get("label") or ""),
date=str(tags.get("date") or ""),
)

466
core/amazon_download_client.py
View file

@ -1,466 +0,0 @@
"""Amazon Music download source plugin backed by a T2Tunes proxy.
NOT yet wired into the app registry — validated in isolation only.
See tests/tools/test_amazon_download_client.py.
Filename encoding (the DownloadSourcePlugin dispatch contract):
"{asin}||{display_name}"
e.g. "B09XYZ1234||Kendrick Lamar - Not Like Us"
Codec preference order: FLAC → Opus → EAC3.
Download flow (from Tubifarry reference implementation):
1. GET stream_url → encrypted bytes on disk
2. FFmpeg -decryption_key <hex> to decrypt in place
3. Embed metadata tags (handled by the app's standard post-processing)
"""
from __future__ import annotations
import os
import shutil
import subprocess
import threading
import time
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
import requests as http_requests
from config.settings import config_manager
from core.amazon_client import AmazonClient, AmazonClientError
from core.download_plugins.base import DownloadSourcePlugin
from core.download_plugins.types import AlbumResult, DownloadStatus, TrackResult
from core.quality.model import AudioQuality
from core.quality.source_map import quality_tier_for_source
from utils.logging_config import get_logger
logger = get_logger("amazon_download_client")
# Quality / codec helpers
CODEC_PREFERENCE = ["flac", "opus", "eac3"]
_CODEC_EXTENSIONS: Dict[str, str] = {
"flac": "flac",
"ogg_vorbis": "ogg",
"opus": "opus",
"eac3": "eac3",
"mp4": "m4a",
"aac": "m4a",
"mp3": "mp3",
}
MIN_AUDIO_BYTES = 512 * 1024 # 512 KB — anything smaller is a broken stream
def _codec_key(codec: str) -> str:
return codec.lower().replace("-", "_").replace(" ", "_")
def _file_extension(codec: str) -> str:
return _CODEC_EXTENSIONS.get(_codec_key(codec), "bin")
def _quality_label(codec: str, sample_rate: Optional[int] = None) -> str:
ck = _codec_key(codec)
if ck == "flac":
if sample_rate and sample_rate > 48000:
return "Hi-Res"
return "Lossless"
return "Lossy"
class AmazonDownloadClient(DownloadSourcePlugin):
"""DownloadSourcePlugin — Amazon Music via T2Tunes proxy."""
def __init__(self, download_path: Optional[str] = None) -> None:
if download_path is None:
download_path = config_manager.get("soulseek.download_path", "./downloads")
self.download_path = Path(download_path)
try:
self.download_path.mkdir(parents=True, exist_ok=True)
except OSError as e:
logger.warning(f"Could not verify download path {self.download_path}: {e}")
self._quality = quality_tier_for_source("amazon", default="flac")
self._allow_fallback = config_manager.get("amazon_download.allow_fallback", True)
self._client = AmazonClient(preferred_codec=self._quality)
self.session = http_requests.Session()
self.session.headers.update({
"User-Agent": "SoulSync/1.0",
"Accept": "*/*",
})
self._engine: Any = None
self.shutdown_check: Any = None
# ------------------------------------------------------------------
# DownloadSourcePlugin — lifecycle
# ------------------------------------------------------------------
def set_engine(self, engine) -> None:
"""Engine callback — wires the central thread worker + state store."""
self._engine = engine
def set_shutdown_check(self, check_callable) -> None:
self.shutdown_check = check_callable
def is_configured(self) -> bool:
# T2Tunes has a public default instance; no credentials required.
# Return True unconditionally so the source shows as available.
return True
async def check_connection(self) -> bool:
try:
return self._client.is_authenticated()
except Exception:
return False
# ------------------------------------------------------------------
# DownloadSourcePlugin — search
# ------------------------------------------------------------------
async def search(
self,
query: str,
timeout: Optional[int] = None,
progress_callback: Any = None,
) -> Tuple[List[TrackResult], List[AlbumResult]]:
try:
items = self._client.search_raw(query, types="track,album")
except AmazonClientError as exc:
logger.warning(f"Amazon search failed for {query!r}: {exc}")
return [], []
track_results: List[TrackResult] = []
album_map: Dict[str, AlbumResult] = {}
album_order: List[str] = []
preferred = self._client.preferred_codec
# Search results only carry the codec (real sample_rate arrives at
# stream time). Claim the format honestly — FLAC for the lossless
# codec, lossy otherwise — so audio_quality derives a real format
# instead of the display label ("Lossless"), and the post-download
# probe pins the actual sample_rate/bit_depth.
amazon_q = AudioQuality(format='flac' if _codec_key(preferred) == 'flac' else 'aac')
for item in items:
quality = _quality_label(preferred)
if item.is_track:
tr = TrackResult(
username="amazon",
filename=f"{item.asin}||{item.artist_name} - {item.title}",
size=0,
bitrate=None,
duration=item.duration_seconds * 1000 if item.duration_seconds else None,
quality=quality,
free_upload_slots=999,
upload_speed=999_999,
queue_length=0,
artist=item.artist_name,
title=item.title,
album=item.album_name,
_source_metadata={
"asin": item.asin,
"album_asin": item.album_asin,
"isrc": item.isrc,
},
)
tr.set_quality(amazon_q)
track_results.append(tr)
elif item.is_album:
album_asin = item.album_asin or item.asin
if album_asin not in album_map:
placeholder = TrackResult(
username="amazon",
filename=f"{item.asin}||{item.artist_name} - {item.title}",
size=0,
bitrate=None,
duration=None,
quality=quality,
free_upload_slots=999,
upload_speed=999_999,
queue_length=0,
artist=item.artist_name,
title=item.title,
album=item.album_name,
)
placeholder.set_quality(amazon_q)
album_map[album_asin] = AlbumResult(
username="amazon",
album_path=album_asin,
album_title=item.album_name or item.title,
artist=item.artist_name,
track_count=0,
total_size=0,
tracks=[placeholder],
dominant_quality=quality,
)
album_order.append(album_asin)
return track_results, [album_map[k] for k in album_order]
# ------------------------------------------------------------------
# DownloadSourcePlugin — download dispatch
# ------------------------------------------------------------------
async def download(
self,
username: str,
filename: str,
file_size: int = 0,
) -> Optional[str]:
if "||" not in filename:
logger.error(f"Invalid Amazon filename format (no '||'): {filename!r}")
return None
if self._engine is None:
raise RuntimeError(
"AmazonDownloadClient._engine is not set — "
"client not connected to download infrastructure"
)
asin, display_name = filename.split("||", 1)
asin = asin.strip()
display_name = display_name.strip()
return self._engine.worker.dispatch(
source_name="amazon",
target_id=asin,
display_name=display_name,
original_filename=filename,
impl_callable=self._download_sync,
)
def _download_sync(
self,
download_id: str,
target_id: str,
display_name: str,
) -> Optional[str]:
asin = str(target_id)
codecs = CODEC_PREFERENCE if self._allow_fallback else [self._quality]
for codec in codecs:
try:
streams = self._client.media_from_asin(asin, codec=codec)
except AmazonClientError as exc:
logger.warning(f"media_from_asin({asin!r}, {codec}) failed: {exc}")
continue
stream = next(
(s for s in streams if s.streamable and s.stream_url),
None,
)
if not stream:
logger.debug(f"No streamable result for {asin} at codec={codec}")
continue
ext = _file_extension(stream.codec or codec)
safe = "".join(
ch if ch.isalnum() or ch in " -_." else "_"
for ch in display_name
)[:80]
# T2Tunes always serves audio in an encrypted MP4 container.
# The codec extension (.flac, .opus, .eac3) is only for the
# final decrypted output.
enc_ext = "mp4" if stream.decryption_key else ext
enc_path = self._unique_path(self.download_path / f"{safe}.enc.{enc_ext}")
out_path = self._unique_path(self.download_path / f"{safe}.{ext}")
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {"state": "downloading", "progress": 0.0}
)
try:
downloaded = self._stream_to_file(stream.stream_url, enc_path, download_id)
except Exception as exc:
logger.warning(f"Stream download failed for {asin} ({codec}): {exc}")
enc_path.unlink(missing_ok=True)
continue
if downloaded < MIN_AUDIO_BYTES:
logger.warning(
f"File too small ({downloaded} B) for {asin} at {codec} — trying next"
)
enc_path.unlink(missing_ok=True)
continue
if stream.decryption_key:
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {"state": "decrypting", "progress": 1.0}
)
try:
self._decrypt_with_ffmpeg(enc_path, out_path, stream.decryption_key)
enc_path.unlink(missing_ok=True)
except Exception as exc:
logger.error(f"Decryption failed for {asin} ({codec}): {exc}")
enc_path.unlink(missing_ok=True)
out_path.unlink(missing_ok=True)
continue
else:
enc_path.rename(out_path)
final_size = out_path.stat().st_size
logger.info(
f"Amazon download complete ({codec}): {out_path} "
f"({final_size / (1024 * 1024):.1f} MB)"
)
# Sync size == transferred so the download monitor's bytes-incomplete
# guard doesn't block post-processing. The throttled updates in
# _stream_to_file leave transferred < size after the last 0.5s tick;
# other streaming clients avoid this by not tracking bytes at all
# (size stays 0, the guard is skipped). Writing the final output size
# here restores parity.
if self._engine is not None:
self._engine.update_record(
"amazon", download_id, {'size': final_size, 'transferred': final_size}
)
return str(out_path)
logger.error(f"All codec tiers exhausted for '{display_name}' ({asin})")
return None
def _decrypt_with_ffmpeg(
self, enc_path: Path, out_path: Path, hex_key: str
) -> None:
"""Decrypt a T2Tunes encrypted audio file using FFmpeg -decryption_key.
T2Tunes uses CENC (Common Encryption) for DRM-protected tracks.
FFmpeg supports decryption via the -decryption_key flag when the
key is provided in hex.
"""
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
tools_dir = Path(__file__).parent.parent / "tools"
candidate = tools_dir / ("ffmpeg.exe" if os.name == "nt" else "ffmpeg")
if candidate.exists():
ffmpeg = str(candidate)
else:
raise RuntimeError(
"ffmpeg is required for Amazon Music decryption. "
"Install ffmpeg and ensure it is on your PATH."
)
cmd = [
ffmpeg,
"-y",
"-hide_banner",
"-loglevel", "error",
"-decryption_key", hex_key,
"-i", str(enc_path),
"-map", "0:a:0", # extract first audio stream (FLAC/Opus/EAC3 inside MP4)
"-c", "copy",
str(out_path),
]
logger.debug(f"Decrypting {enc_path.name} → {out_path.name}")
result = subprocess.run(cmd, capture_output=True)
if result.returncode != 0:
stderr = result.stderr.decode("utf-8", errors="replace").strip()
raise RuntimeError(f"FFmpeg decryption failed (exit {result.returncode}): {stderr}")
def _stream_to_file(self, url: str, out_path: Path, download_id: str) -> int:
resp = self.session.get(url, stream=True, timeout=60)
resp.raise_for_status()
total = int(resp.headers.get("content-length", 0))
downloaded = 0
last_report = time.monotonic()
shutdown_triggered = False
with out_path.open("wb") as fh:
for chunk in resp.iter_content(chunk_size=64 * 1024):
if not chunk:
continue
if self.shutdown_check and self.shutdown_check():
shutdown_triggered = True
break
fh.write(chunk)
downloaded += len(chunk)
now = time.monotonic()
if self._engine and now - last_report >= 0.5:
self._engine.update_record(
"amazon",
download_id,
{
"transferred": downloaded,
"size": total,
"progress": downloaded / total if total else 0.0,
},
)
last_report = now
if shutdown_triggered:
out_path.unlink(missing_ok=True)
raise RuntimeError("Shutdown requested mid-download")
return downloaded
# ------------------------------------------------------------------
# DownloadSourcePlugin — status interface
# ------------------------------------------------------------------
async def get_all_downloads(self) -> List[DownloadStatus]:
if self._engine is None:
return []
return [
self._record_to_status(record)
for record in self._engine.iter_records_for_source('amazon')
]
async def get_download_status(self, download_id: str) -> Optional[DownloadStatus]:
if self._engine is None:
return None
record = self._engine.get_record('amazon', download_id)
return self._record_to_status(record) if record is not None else None
async def cancel_download(
self,
download_id: str,
username: Optional[str] = None,
remove: bool = False,
) -> bool:
if self._engine is None:
return False
if self._engine.get_record('amazon', download_id) is None:
return False
self._engine.update_record('amazon', download_id, {'state': 'Cancelled'})
if remove:
self._engine.remove_record('amazon', download_id)
return True
async def clear_all_completed_downloads(self) -> bool:
if self._engine is None:
return True
terminal = {'Completed, Succeeded', 'Cancelled', 'Errored', 'Aborted'}
for record in list(self._engine.iter_records_for_source('amazon')):
if record.get('state') in terminal:
self._engine.remove_record('amazon', record['id'])
return True
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
@staticmethod
def _unique_path(path: Path) -> Path:
if not path.exists():
return path
stem, suffix = path.stem, path.suffix
for i in range(1, 100):
candidate = path.with_name(f"{stem} ({i}){suffix}")
if not candidate.exists():
return candidate
return path.with_name(f"{stem}_{uuid.uuid4().hex[:8]}{suffix}")
@staticmethod
def _record_to_status(rec: Dict[str, Any]) -> DownloadStatus:
return DownloadStatus(
id=str(rec.get('id', '')),
filename=str(rec.get('filename', '')),
username='amazon',
state=str(rec.get('state', 'queued')),
progress=float(rec.get('progress', 0.0)),
size=int(rec.get('size', 0)),
transferred=int(rec.get('transferred', 0)),
speed=int(rec.get('speed', 0)),
time_remaining=rec.get('time_remaining'),
file_path=rec.get('file_path'),
)

61
core/amazon_outage.py
View file

@ -1,61 +0,0 @@
"""Amazon enrichment outage detection + back-off — pure, importable, testable.
The Amazon worker enriches via a public T2Tunes proxy instance. When that
instance is down (HTTP 5xx, "Amazon Music API is not initialized", or an
unreachable host), the worker must NOT treat every album as an individual
failure: doing so floods the logs with an error per item, churns network + DB
continuously, and permanently marks the whole library ``error`` (which the
retry tiers never re-attempt) for what is really a transient outage.
Instead it recognizes "the whole source is down", leaves the item untouched so
it's retried once the instance recovers, and backs off hard. These two pure
helpers carry that logic so it can be unit-tested without the worker, the DB,
or the network.
"""
from __future__ import annotations
import re
# HTTP statuses that mean "the source/proxy is unhealthy", not "no match".
_OUTAGE_STATUS = {500, 502, 503, 504}
# Substrings (lower-cased) in an error message that indicate a source outage
# rather than a per-item miss: proxy not ready, gateway errors, the host being
# unreachable, or an error page returned instead of JSON.
_OUTAGE_PHRASES = (
"not initialized", "not configured", "service unavailable",
"bad gateway", "gateway time", "request failed", "response not json",
"max retries", "connection", "timed out", "temporarily unavailable",
)
# Back-off schedule while the source is down.
_NORMAL_DELAY = 2 # seconds between items when healthy
_OUTAGE_BASE = 30 # first back-off step
_OUTAGE_CAP = 1800 # 30 minutes max
def is_source_outage(exc: Exception) -> bool:
"""True when ``exc`` indicates the Amazon source/proxy is down (transient,
whole-source), as opposed to a normal per-item error.
Robust to how the error is surfaced: an explicit ``status_code`` attribute,
an ``HTTP <code>`` prefix in the message, or an outage phrase (covers
connection failures and non-JSON error pages that carry no status code)."""
code = getattr(exc, "status_code", None)
if isinstance(code, int) and code in _OUTAGE_STATUS:
return True
msg = str(exc).lower()
m = re.search(r"http\s+(\d{3})", msg)
if m and int(m.group(1)) in _OUTAGE_STATUS:
return True
return any(p in msg for p in _OUTAGE_PHRASES)
def next_poll_delay_seconds(outage_streak: int) -> int:
"""Seconds to wait before the next item. Normal cadence when healthy;
escalating back-off (30s, 60s, 120s, … capped at 30 min) the longer the
source has been down, so a dead instance can't flood logs/CPU/DB."""
if outage_streak <= 0:
return _NORMAL_DELAY
return min(_OUTAGE_BASE * (2 ** min(outage_streak - 1, 6)), _OUTAGE_CAP)

645
core/amazon_worker.py
View file

@ -1,645 +0,0 @@
import re
import threading
import time
from difflib import SequenceMatcher
from typing import Optional, Dict, Any
from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.amazon_client import AmazonClient
from core.worker_utils import interruptible_sleep, set_album_api_track_count
from core.enrichment.manual_match_honoring import honor_stored_match
from core.amazon_outage import is_source_outage, next_poll_delay_seconds
logger = get_logger("amazon_worker")
class AmazonWorker:
"""Background worker for enriching library artists, albums, and tracks with Amazon Music metadata."""
def __init__(self, database: MusicDatabase):
self.db = database
self.client = AmazonClient()
self._amazon_schema_checked = False
self.running = False
self.paused = False
self.should_stop = False
self.thread = None
self._stop_event = threading.Event()
self.current_item = None
self.stats = {
'matched': 0,
'not_found': 0,
'pending': 0,
'errors': 0,
}
self.retry_days = 30
self.name_similarity_threshold = 0.80
# Source-outage circuit breaker: counts consecutive whole-source
# failures (proxy down / "not initialized" / unreachable) so the loop
# backs off instead of grinding the whole library item-by-item.
self._outage_streak = 0
logger.info("Amazon background worker initialized")
def _ensure_amazon_schema(self, cursor) -> None:
"""Ensure upgraded installs have the Amazon enrichment columns.
MusicDatabase normally runs this migration during startup, but the
worker should still be defensive because it is the code path that
repeatedly queries these columns in the background.
"""
if self._amazon_schema_checked:
return
table_columns = {
'artists': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'albums': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
'tracks': ('amazon_id', 'amazon_match_status', 'amazon_last_attempted'),
}
for table, columns in table_columns.items():
cursor.execute(f"PRAGMA table_info({table})")
existing = {row[1] for row in cursor.fetchall()}
for column in columns:
if column not in existing:
column_type = 'TIMESTAMP' if column == 'amazon_last_attempted' else 'TEXT'
cursor.execute(f"ALTER TABLE {table} ADD COLUMN {column} {column_type}")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_id ON artists (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_artists_amazon_status ON artists (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_id ON albums (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_albums_amazon_status ON albums (amazon_match_status)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_id ON tracks (amazon_id)")
cursor.execute("CREATE INDEX IF NOT EXISTS idx_tracks_amazon_status ON tracks (amazon_match_status)")
cursor.connection.commit()
self._amazon_schema_checked = True
def start(self):
if self.running:
logger.warning("Worker already running")
return
self.running = True
self.should_stop = False
self._stop_event.clear()
self.thread = threading.Thread(target=self._run, daemon=True)
self.thread.start()
logger.info("Amazon background worker started")
def stop(self):
if not self.running:
return
logger.info("Stopping Amazon worker...")
self.should_stop = True
self.running = False
self._stop_event.set()
if self.thread:
self.thread.join(timeout=1)
logger.info("Amazon worker stopped")
def pause(self):
if not self.running:
logger.warning("Worker not running, cannot pause")
return
self.paused = True
logger.info("Amazon worker paused")
def resume(self):
if not self.running:
logger.warning("Worker not running, start it first")
return
self.paused = False
logger.info("Amazon worker resumed")
def get_stats(self) -> Dict[str, Any]:
self.stats['pending'] = self._count_pending_items()
progress = self._get_progress_breakdown()
is_actually_running = self.running and (self.thread is not None and self.thread.is_alive())
is_idle = is_actually_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
return {
'enabled': True,
'running': is_actually_running and not self.paused,
'paused': self.paused,
'idle': is_idle,
'current_item': self.current_item,
'stats': self.stats.copy(),
'progress': progress,
}
def _run(self):
logger.info("Amazon worker thread started")
while not self.should_stop:
try:
if self.paused:
interruptible_sleep(self._stop_event, 1)
continue
self.current_item = None
item = self._get_next_item()
if not item:
logger.debug("No pending items, sleeping...")
interruptible_sleep(self._stop_event, 10)
continue
self.current_item = item
item_id = item.get('id') or item.get('artist_id') or item.get('album_id')
if item_id is None:
logger.warning(f"Skipping {item.get('type', 'unknown')} with NULL id: {item.get('name', '?')}")
continue
self._process_item(item)
# Normal 2s cadence when healthy; escalating back-off (up to
# 30 min) while the source is in an outage streak.
interruptible_sleep(self._stop_event, next_poll_delay_seconds(self._outage_streak))
except Exception as e:
logger.error(f"Error in worker loop: {e}")
interruptible_sleep(self._stop_event, 5)
logger.info("Amazon worker thread finished")
def _get_next_item(self) -> Optional[Dict[str, Any]]:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('amazon')
if _prio:
_pi = priority_pending_item(cursor, 'amazon', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name FROM artists
WHERE amazon_match_status IS NULL AND id IS NOT NULL
ORDER BY id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 2: Unattempted albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.amazon_match_status IS NULL AND a.id IS NOT NULL
ORDER BY a.id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
# Priority 3: Unattempted tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.amazon_match_status IS NULL AND t.id IS NOT NULL
ORDER BY t.id ASC LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
not_found_cutoff = datetime.now() - timedelta(days=self.retry_days)
# Priority 4: Retry not_found artists
cursor.execute("""
SELECT id, name FROM artists
WHERE amazon_match_status = 'not_found' AND amazon_last_attempted < ?
ORDER BY amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 5: Retry not_found albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.amazon_match_status = 'not_found' AND a.amazon_last_attempted < ?
ORDER BY a.amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
# Priority 6: Retry not_found tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.amazon_id AS artist_amazon_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.amazon_match_status = 'not_found' AND t.amazon_last_attempted < ?
ORDER BY t.amazon_last_attempted ASC LIMIT 1
""", (not_found_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_amazon_id': row[3]}
return None
except Exception as e:
logger.error(f"Error getting next item: {e}")
return None
finally:
if conn:
conn.close()
def _normalize_name(self, name: str) -> str:
name = name.lower().strip()
name = re.sub(r'\s+[-–—]\s+.*$', '', name)
name = re.sub(r'\s*\(.*?\)\s*', ' ', name)
name = re.sub(r'[^\w\s]', '', name)
name = re.sub(r'\s+', ' ', name).strip()
return name
def _name_matches(self, query_name: str, result_name: str) -> bool:
norm_query = self._normalize_name(query_name)
norm_result = self._normalize_name(result_name)
similarity = SequenceMatcher(None, norm_query, norm_result).ratio()
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
return similarity >= self.name_similarity_threshold
def _process_item(self, item: Dict[str, Any]):
try:
item_type = item['type']
item_id = item['id']
item_name = item['name']
logger.debug(f"Processing {item_type} #{item_id}: {item_name}")
if item_type == 'artist':
self._process_artist(item_id, item_name)
elif item_type == 'album':
self._process_album(item_id, item_name, item.get('artist', ''), item)
elif item_type == 'track':
self._process_track(item_id, item_name, item.get('artist', ''), item)
# The source answered (match or not_found) — clear any outage streak.
if self._outage_streak:
logger.info("Amazon source recovered after %d outage(s), resuming",
self._outage_streak)
self._outage_streak = 0
except Exception as e:
if is_source_outage(e):
# The whole source is down (proxy 5xx / "not initialized" /
# unreachable). Do NOT mark the item 'error' — that would burn
# the entire library to a state the retry tiers never re-attempt
# for a transient outage. Leave it untouched so it's retried once
# the instance recovers, and let the loop back off. Log once per
# streak to avoid flooding.
self._outage_streak += 1
if self._outage_streak == 1:
logger.warning("Amazon source unavailable — pausing enrichment "
"until it recovers: %s", e)
else:
logger.debug("Amazon source still unavailable (streak=%d): %s",
self._outage_streak, e)
return
# A non-outage error means the source actually answered (e.g. a
# 404/parse error on a real response), so the outage is over —
# clear the streak and handle this as a normal per-item error.
self._outage_streak = 0
logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
self.stats['errors'] += 1
try:
self._mark_status(item['type'], item['id'], 'error')
except Exception as e2:
logger.error(f"Error updating item status: {e2}")
def _get_existing_id(self, entity_type: str, entity_id: int) -> Optional[str]:
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
return None
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"SELECT amazon_id FROM {table} WHERE id = ?", (entity_id,))
row = cursor.fetchone()
return row[0] if row and row[0] else None
except Exception:
return None
finally:
if conn:
conn.close()
def _process_artist(self, artist_id: int, artist_name: str):
existing_id = self._get_existing_id('artist', artist_id)
if existing_id:
logger.debug(f"Preserving existing Amazon ID for artist '{artist_name}': {existing_id}")
return
results = self.client.search_artists(artist_name, limit=5)
if results:
result = results[0]
if self._name_matches(artist_name, result.name):
self._update_artist(artist_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{artist_name}' -> Amazon ID: {result.id}")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for artist '{artist_name}' (got '{result.name}')")
else:
self._mark_status('artist', artist_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for artist '{artist_name}'")
def _refresh_album_via_stored_id(self, album_id, stored_id, api_data):
self._update_album(album_id, api_data, stored_id)
def _refresh_track_via_stored_id(self, track_id, stored_id, api_data):
self._update_track(track_id, api_data, stored_id)
def _process_album(self, album_id: int, album_name: str, artist_name: str, item: Dict[str, Any]):
if honor_stored_match(
db=self.db, entity_table='albums', entity_id=album_id,
id_column='amazon_id',
client_fetch_fn=lambda asin: self.client.get_album(asin, include_tracks=False),
on_match_fn=self._refresh_album_via_stored_id,
log_prefix='Amazon',
):
self.stats['matched'] += 1
return
query = f"{artist_name} {album_name}"
results = self.client.search_albums(query, limit=10)
if results:
result = results[0]
if self._name_matches(album_name, result.name):
full_album = None
if result.id:
try:
full_album = self.client.get_album(result.id, include_tracks=False)
except Exception as e:
logger.warning(f"Failed to fetch full album '{album_name}' (ASIN: {result.id}): {e}")
if full_album is None:
self._mark_status('album', album_id, 'error')
self.stats['errors'] += 1
logger.warning(f"Album '{album_name}' matched but full details unavailable, will retry")
return
self._update_album(album_id, full_album, result.id)
self.stats['matched'] += 1
logger.info(f"Matched album '{album_name}' -> Amazon ASIN: {result.id}")
else:
self._mark_status('album', album_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for album '{album_name}' (got '{result.name}')")
else:
self._mark_status('album', album_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for album '{album_name}'")
def _process_track(self, track_id: int, track_name: str, artist_name: str, item: Dict[str, Any]):
if honor_stored_match(
db=self.db, entity_table='tracks', entity_id=track_id,
id_column='amazon_id',
client_fetch_fn=self.client.get_track_details,
on_match_fn=self._refresh_track_via_stored_id,
log_prefix='Amazon',
):
self.stats['matched'] += 1
return
query = f"{artist_name} {track_name}"
results = self.client.search_tracks(query, limit=10)
if results:
result = results[0]
if self._name_matches(track_name, result.name):
full_track = None
if result.id:
try:
full_track = self.client.get_track_details(result.id)
except Exception as e:
logger.warning(f"Failed to fetch full track '{track_name}' (ASIN: {result.id}): {e}")
if full_track is None:
self._mark_status('track', track_id, 'error')
self.stats['errors'] += 1
logger.warning(f"Track '{track_name}' matched but full details unavailable, will retry")
return
self._update_track(track_id, full_track, result.id)
self.stats['matched'] += 1
logger.info(f"Matched track '{track_name}' -> Amazon ASIN: {result.id}")
else:
self._mark_status('track', track_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for track '{track_name}' (got '{result.name}')")
else:
self._mark_status('track', track_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for track '{track_name}'")
def _update_artist(self, artist_id: int, result):
"""Store Amazon metadata for an artist. ``result`` is an Artist dataclass."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE artists SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (str(result.id), artist_id))
# Backfill thumb_url from album cover stand-in when artist has no image
image_url = result.image_url
if not image_url:
try:
image_url = self.client._get_artist_image_from_albums(result.id)
except Exception as exc:
logger.debug("Artist image from albums failed for %s: %s", result.id, exc)
if image_url:
cursor.execute("""
UPDATE artists SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (image_url, artist_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating artist #{artist_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _update_album(self, album_id: int, full_data: Dict[str, Any], asin: str):
"""Store Amazon metadata for an album. ``full_data`` is a get_album() dict."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE albums SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (asin, album_id))
# Backfill label when missing
label = full_data.get('label')
if label:
cursor.execute("""
UPDATE albums SET label = ?
WHERE id = ? AND (label IS NULL OR label = '')
""", (label, album_id))
# Backfill thumb_url
images = full_data.get('images') or []
thumb_url = images[0].get('url') if images else None
if thumb_url:
cursor.execute("""
UPDATE albums SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (thumb_url, album_id))
# Cache authoritative track count for completeness repair
total_tracks = full_data.get('total_tracks') or (
full_data.get('tracks', {}).get('total') if isinstance(full_data.get('tracks'), dict) else None
)
set_album_api_track_count(cursor, album_id, total_tracks)
conn.commit()
except Exception as e:
logger.error(f"Error updating album #{album_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _update_track(self, track_id: int, full_data: Dict[str, Any], asin: str):
"""Store Amazon metadata for a track. ``full_data`` is a get_track_details() dict."""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE tracks SET
amazon_id = ?,
amazon_match_status = 'matched',
amazon_last_attempted = CURRENT_TIMESTAMP
WHERE id = ?
""", (asin, track_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating track #{track_id} with Amazon data: {e}")
raise
finally:
if conn:
conn.close()
def _mark_status(self, entity_type: str, entity_id: int, status: str):
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
logger.error(f"Unknown entity type: {entity_type}")
return
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"""
UPDATE {table} SET
amazon_match_status = ?,
amazon_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (status, entity_id))
conn.commit()
except Exception as e:
logger.error(f"Error marking {entity_type} #{entity_id} status: {e}")
finally:
if conn:
conn.close()
def _count_pending_items(self) -> int:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
cursor.execute("""
SELECT
(SELECT COUNT(*) FROM artists WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM albums WHERE amazon_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM tracks WHERE amazon_match_status IS NULL AND id IS NOT NULL)
AS pending
""")
row = cursor.fetchone()
return row[0] if row else 0
except Exception as e:
logger.error(f"Error counting pending items: {e}")
return 0
finally:
if conn:
conn.close()
def _get_progress_breakdown(self) -> Dict[str, Dict[str, int]]:
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
self._ensure_amazon_schema(cursor)
progress = {}
for table in ('artists', 'albums', 'tracks'):
cursor.execute(f"""
SELECT
COUNT(*) AS total,
SUM(CASE WHEN amazon_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
FROM {table}
""")
row = cursor.fetchone()
if row:
total, processed = row[0], row[1] or 0
progress[table] = {
'matched': processed,
'total': total,
'percent': int((processed / total * 100) if total > 0 else 0),
}
return progress
except Exception as e:
logger.error(f"Error getting progress breakdown: {e}")
return {}
finally:
if conn:
conn.close()

343
core/api_call_tracker.py
View file

@ -1,343 +0,0 @@
"""
Centralized API call tracker for all enrichment services.
Tracks actual API calls (not items processed) with rolling timestamps
for real-time rate monitoring and minute-bucketed history for 24-hour graphs.
Thread-safe, persists 24h history to disk on shutdown and restores on startup.
"""
import json
import os
import threading
import time
from collections import deque, defaultdict
from utils.logging_config import get_logger
logger = get_logger("api_call_tracker")
# Known rate limits per service (calls/minute)
RATE_LIMITS = {
'spotify': 171, # MIN_API_INTERVAL=0.35s → ~171/min
'itunes': 20, # MIN_API_INTERVAL=3.0s → ~20/min
'deezer': 60, # MIN_API_INTERVAL=1.0s → ~60/min
'lastfm': 300, # MIN_API_INTERVAL=0.2s → ~300/min
'genius': 30, # MIN_API_INTERVAL=2.0s → ~30/min
'musicbrainz': 60, # MIN_API_INTERVAL=1.0s → ~60/min
'audiodb': 30, # MIN_API_INTERVAL=2.0s → ~30/min
'tidal': 120, # MIN_API_INTERVAL=0.5s → ~120/min
'qobuz': 60, # Variable throttle, ~60/min estimate
'discogs': 60, # MIN_API_INTERVAL=1.0s with auth → ~60/min
'amazon': 120, # MIN_API_INTERVAL=0.5s → ~120/min (T2Tunes proxy)
}
# Display names for UI
SERVICE_LABELS = {
'spotify': 'Spotify',
'itunes': 'Apple Music',
'deezer': 'Deezer',
'lastfm': 'Last.fm',
'genius': 'Genius',
'musicbrainz': 'MusicBrainz',
'audiodb': 'AudioDB',
'tidal': 'Tidal',
'qobuz': 'Qobuz',
'discogs': 'Discogs',
'amazon': 'Amazon Music',
}
# Display order
SERVICE_ORDER = [
'spotify', 'itunes', 'deezer', 'lastfm', 'genius',
'musicbrainz', 'audiodb', 'tidal', 'qobuz', 'discogs', 'amazon',
]
_PERSIST_PATH = os.path.join('database', 'api_call_history.json')
class ApiCallTracker:
"""Centralized tracker for actual API calls across all enrichment services."""
def __init__(self):
self._lock = threading.Lock()
# Recent call timestamps per service (last 60s window for speedometer)
# maxlen=600 covers 10 calls/sec for 60s — more than any service does
self._recent_calls = defaultdict(lambda: deque(maxlen=600))
# 24-hour minute-bucketed history per service
# Each entry: (minute_floor_timestamp, call_count)
self._minute_history = defaultdict(lambda: deque(maxlen=1440))
self._current_minute_counts = defaultdict(int)
self._current_minute_ts = {}
# Rate limit event log — records bans, peaks, escalations
# Each entry: {ts, event, service, endpoint, duration, detail}
self._events = deque(maxlen=200)
# Restore persisted history from disk
self._load()
def record_call(self, service_key, endpoint=None):
"""Record an API call. Called from rate_limited decorators.
Args:
service_key: Service identifier ('spotify', 'itunes', etc.)
endpoint: Optional endpoint name for per-endpoint tracking (Spotify only)
"""
now = time.time()
minute_floor = int(now // 60) * 60
with self._lock:
# Record in recent timestamps
self._recent_calls[service_key].append(now)
# Roll minute bucket
self._roll_minute(service_key, minute_floor)
# Spotify per-endpoint tracking
if endpoint and service_key == 'spotify':
ep_key = f"spotify:{endpoint}"
self._recent_calls[ep_key].append(now)
self._roll_minute(ep_key, minute_floor)
def _roll_minute(self, key, minute_floor):
"""Roll the minute bucket forward if we've moved to a new minute.
Must be called while holding self._lock."""
prev_ts = self._current_minute_ts.get(key)
if prev_ts is None or minute_floor > prev_ts:
# Flush previous minute's count to history
if prev_ts is not None and self._current_minute_counts[key] > 0:
self._minute_history[key].append((prev_ts, self._current_minute_counts[key]))
# Fill gaps with zeros (if minutes were skipped)
if prev_ts is not None:
gap_start = prev_ts + 60
while gap_start < minute_floor:
self._minute_history[key].append((gap_start, 0))
gap_start += 60
# Start new minute
self._current_minute_ts[key] = minute_floor
self._current_minute_counts[key] = 1
else:
self._current_minute_counts[key] += 1
def record_event(self, service_key, event_type, detail='', endpoint='', duration=0):
"""Record a rate limit event (ban, escalation, cooldown, etc.).
Called from spotify_client.py when rate limits are detected."""
with self._lock:
self._events.append({
'ts': time.time(),
'event': event_type,
'service': service_key,
'endpoint': endpoint,
'duration': duration,
'detail': detail,
})
def get_events(self, since=None):
"""Get rate limit events, optionally filtered by timestamp."""
cutoff = since or (time.time() - 86400)
with self._lock:
return [e for e in self._events if e['ts'] >= cutoff]
def get_debug_summary(self):
"""Build a comprehensive debug summary for Copy Debug Info.
Includes 24h totals, peaks, rate limit events, and per-endpoint breakdown."""
now = time.time()
cutoff_24h = now - 86400
summary = {}
with self._lock:
for svc in SERVICE_ORDER:
# 24h total calls
total = 0
peak_cpm = 0
peak_ts = 0
for ts, count in self._minute_history.get(svc, []):
if ts >= cutoff_24h:
total += count
if count > peak_cpm:
peak_cpm = count
peak_ts = ts
# Include current minute
cur_ts = self._current_minute_ts.get(svc)
cur_count = self._current_minute_counts.get(svc, 0)
if cur_ts and cur_ts >= cutoff_24h:
total += cur_count
if cur_count > peak_cpm:
peak_cpm = cur_count
peak_ts = cur_ts
if total == 0:
continue
entry = {
'total_24h': total,
'peak_cpm': peak_cpm,
'limit_cpm': RATE_LIMITS.get(svc, 60),
}
if peak_ts:
entry['peak_at'] = time.strftime('%Y-%m-%d %H:%M', time.localtime(peak_ts))
summary[svc] = entry
# Spotify per-endpoint breakdown
if svc == 'spotify':
ep_totals = {}
for key in list(self._minute_history.keys()):
if key.startswith('spotify:'):
ep_name = key[8:]
ep_total = sum(c for ts, c in self._minute_history[key] if ts >= cutoff_24h)
cur = self._current_minute_counts.get(key, 0)
ep_total += cur
if ep_total > 0:
ep_totals[ep_name] = ep_total
if ep_totals:
summary[svc]['endpoints'] = ep_totals
# Rate limit events
events = [e for e in self._events if e['ts'] >= cutoff_24h]
if events:
summary['_rate_limit_events'] = [{
'time': time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(e['ts'])),
'event': e['event'],
'service': e['service'],
'endpoint': e.get('endpoint', ''),
'duration': e.get('duration', 0),
'detail': e.get('detail', ''),
} for e in events[-20:]] # Last 20 events
return summary
def get_calls_per_minute(self, service_key):
"""Get current calls/minute rate from last 60 seconds."""
now = time.time()
cutoff = now - 60.0
with self._lock:
dq = self._recent_calls.get(service_key)
if not dq:
return 0.0
count = sum(1 for ts in dq if ts > cutoff)
return float(count)
def get_24h_history(self, service_key):
"""Return list of [minute_timestamp, count] for last 24 hours.
Includes the current in-progress minute."""
now = time.time()
cutoff = now - 86400
with self._lock:
history = []
for ts, count in self._minute_history.get(service_key, []):
if ts >= cutoff:
history.append([ts, count])
# Include current minute in progress
cur_ts = self._current_minute_ts.get(service_key)
cur_count = self._current_minute_counts.get(service_key, 0)
if cur_ts is not None and cur_count > 0 and cur_ts >= cutoff:
history.append([cur_ts, cur_count])
return history
def get_all_rates(self):
"""Get current rates for all services. Used by WebSocket emission."""
result = {}
for svc in SERVICE_ORDER:
cpm = self.get_calls_per_minute(svc)
entry = {
'cpm': round(cpm, 1),
'limit': RATE_LIMITS.get(svc, 60),
}
# Spotify per-endpoint breakdown
if svc == 'spotify':
endpoints = {}
for key in list(self._recent_calls.keys()):
if key.startswith('spotify:'):
ep_name = key[8:] # strip 'spotify:'
ep_cpm = self.get_calls_per_minute(key)
if ep_cpm > 0:
endpoints[ep_name] = round(ep_cpm, 1)
entry['endpoints'] = endpoints
result[svc] = entry
return result
def save(self):
"""Persist 24h minute history to disk. Call on shutdown.
Uses an atomic write (write to a sibling .tmp file, fsync, rename) so
a SIGINT/SIGTERM/crash mid-write can't leave the JSON file truncated.
Without this, an interrupted shutdown corrupts the history file and
the next startup loses 24h of metrics."""
try:
now = time.time()
cutoff = now - 86400
data = {}
with self._lock:
for key, hist in self._minute_history.items():
entries = [[ts, count] for ts, count in hist if ts >= cutoff]
# Include current in-progress minute
cur_ts = self._current_minute_ts.get(key)
cur_count = self._current_minute_counts.get(key, 0)
if cur_ts is not None and cur_count > 0 and cur_ts >= cutoff:
entries.append([cur_ts, cur_count])
if entries:
data[key] = entries
events = [dict(e) for e in self._events if e['ts'] >= cutoff]
payload = {'ts': now, 'history': data, 'events': events}
tmp_path = _PERSIST_PATH + '.tmp'
with open(tmp_path, 'w') as f:
json.dump(payload, f)
f.flush()
os.fsync(f.fileno())
os.replace(tmp_path, _PERSIST_PATH)
except Exception as e:
logger.error(f"[ApiCallTracker] Failed to save history: {e}")
# Best-effort cleanup of stale tmp file from a failed write.
try:
if os.path.exists(_PERSIST_PATH + '.tmp'):
os.remove(_PERSIST_PATH + '.tmp')
except Exception as e:
logger.debug("remove stale tmp file failed: %s", e)
def _load(self):
"""Restore 24h minute history from disk. Called on init."""
try:
if not os.path.exists(_PERSIST_PATH):
return
if os.path.getsize(_PERSIST_PATH) == 0:
logger.info(f"[ApiCallTracker] History file is empty, starting fresh: {_PERSIST_PATH}")
return
with open(_PERSIST_PATH, 'r') as f:
raw = json.load(f)
saved_ts = raw.get('ts', 0)
# Only restore if saved within last 24h
if time.time() - saved_ts > 86400:
return
history = raw.get('history', {})
events = raw.get('events', [])
cutoff = time.time() - 86400
with self._lock:
for key, entries in history.items():
for ts, count in entries:
if ts >= cutoff:
self._minute_history[key].append((ts, count))
for e in events:
if e.get('ts', 0) >= cutoff:
self._events.append(e)
logger.info(f"[ApiCallTracker] Restored history for {len(history)} services, {len(events)} events")
except json.JSONDecodeError as e:
logger.warning(f"[ApiCallTracker] History file is not valid JSON, starting fresh: {_PERSIST_PATH} ({e})")
except Exception as e:
logger.error(f"[ApiCallTracker] Failed to load history: {e}")
# Singleton instance
api_call_tracker = ApiCallTracker()

238
core/archive_pipeline.py
View file

@ -1,238 +0,0 @@
"""Archive extraction + audio-file discovery for torrent / usenet downloads.
The torrent and usenet download plugins need a uniform way to:
1. Walk the downloader's save directory and find every audio file in it.
2. If the directory contains an archive (``.zip`` / ``.rar`` / ``.tar`` /
``.7z``), extract it first so the audio files inside become walkable.
This module is intentionally narrow — no matching, no tagging, no
import. The download plugin layer composes this with the existing
post-processing / matching pipeline. Lidarr does NOT use this module:
Lidarr extracts archives in its own import step before SoulSync sees
the files at all. Usenet downloaders (SABnzbd, NZBGet) also auto-
extract by default. Torrents are the main case where SoulSync may
need to do the extract step itself — most music torrents ship loose,
but some bundle the album in a ``.rar`` archive.
``rarfile`` is an optional dependency. If it isn't installed, archives
with ``.rar`` content are skipped with a single warning rather than
crashing the download.
"""
from __future__ import annotations
import tarfile
import zipfile
from pathlib import Path
from typing import List, Optional
from utils.logging_config import get_logger
logger = get_logger("archive_pipeline")
# Same audio-extension set as ``core/imports/file_ops.py`` ``quality_tiers``.
# Keep them in sync — if a new format is added to file_ops, add it here too
# or the walker will skip it and the download plugin will mark the download
# failed even when files arrived.
AUDIO_EXTENSIONS = frozenset([
# lossless
'.flac', '.ape', '.wav', '.alac', '.dsf', '.dff', '.aiff', '.aif',
# high lossy
'.opus', '.ogg',
# standard lossy
'.m4a', '.aac',
# low lossy
'.mp3', '.wma',
])
ARCHIVE_EXTENSIONS = frozenset(['.zip', '.rar', '.tar', '.tar.gz', '.tgz', '.7z'])
def is_archive(path: Path) -> bool:
"""True if the file extension looks like a supported archive.
Compound extensions (``.tar.gz``, ``.tar.bz2``) are detected by
checking the last two suffixes joined together — Path.suffix
only returns the final suffix.
"""
if not path.is_file():
return False
name = path.name.lower()
if name.endswith(('.tar.gz', '.tar.bz2', '.tar.xz')):
return True
return path.suffix.lower() in ARCHIVE_EXTENSIONS
def walk_audio_files(directory: Path) -> List[Path]:
"""Recursively scan ``directory`` for audio files. Returns
a sorted list of absolute paths. Empty list if the directory
doesn't exist or contains no audio.
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
out: List[Path] = []
for child in directory.rglob('*'):
if not child.is_file():
continue
if child.suffix.lower() in AUDIO_EXTENSIONS:
out.append(child.resolve())
out.sort()
return out
def find_archives_in_dir(directory: Path) -> List[Path]:
"""Find every archive file directly inside ``directory`` (one
level deep — torrents normally put the archive at the root of
their folder; we don't search nested dirs to avoid extracting
something we shouldn't).
"""
if not directory or not directory.exists() or not directory.is_dir():
return []
return sorted(p for p in directory.iterdir() if is_archive(p))
def extract_archive(archive_path: Path, extract_to: Optional[Path] = None) -> Optional[Path]:
"""Extract a single archive in-place (or to ``extract_to`` if
given). Returns the directory the archive was extracted into,
or ``None`` on failure.
Supports ``.zip``, ``.tar``/``.tar.gz``/``.tar.bz2``/``.tar.xz``,
and ``.rar`` (only when the optional ``rarfile`` library is
installed). ``.7z`` is recognised but extraction requires
``py7zr``; without it, the call logs and returns None.
"""
if not archive_path or not archive_path.exists():
logger.warning("archive_pipeline: %s does not exist", archive_path)
return None
dest = extract_to or archive_path.parent
dest.mkdir(parents=True, exist_ok=True)
name = archive_path.name.lower()
try:
if name.endswith('.zip'):
with zipfile.ZipFile(archive_path) as zf:
_safe_extract_zip(zf, dest)
return dest
if name.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
with tarfile.open(archive_path) as tf:
_safe_extract_tar(tf, dest)
return dest
if name.endswith('.rar'):
return _extract_rar(archive_path, dest)
if name.endswith('.7z'):
return _extract_7z(archive_path, dest)
except (zipfile.BadZipFile, tarfile.TarError, OSError) as e:
logger.error("archive_pipeline: failed to extract %s: %s", archive_path, e)
return None
logger.warning("archive_pipeline: unknown archive type for %s", archive_path)
return None
def extract_all_in_dir(directory: Path) -> List[Path]:
"""Find every archive in ``directory`` and extract each in place.
Returns the list of directories archives were extracted into
(usually all the same — ``directory`` itself). Archives that
failed to extract are skipped silently after a warning.
"""
out: List[Path] = []
for archive in find_archives_in_dir(directory):
result = extract_archive(archive)
if result is not None:
out.append(result)
return out
def collect_audio_after_extraction(directory: Path) -> List[Path]:
"""One-shot helper for the download plugins: extract any archives
in the directory, then return the walked audio file list. This is
the common pattern — torrent / usenet plugin gets a save_path,
calls this, hands the resulting files to the matching pipeline.
"""
extract_all_in_dir(directory)
return walk_audio_files(directory)
# ---------------------------------------------------------------------------
# Safety helpers
# ---------------------------------------------------------------------------
def _safe_extract_zip(zf: zipfile.ZipFile, dest: Path) -> None:
"""Extract a zipfile after rejecting any member whose resolved
path escapes ``dest`` (path traversal protection).
"""
dest = dest.resolve()
for member in zf.namelist():
target = (dest / member).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member)
return
zf.extractall(dest)
def _safe_extract_tar(tf: tarfile.TarFile, dest: Path) -> None:
"""Same path-traversal protection for tarfiles."""
dest = dest.resolve()
for member in tf.getmembers():
target = (dest / member.name).resolve()
if dest not in target.parents and target != dest:
logger.error("archive_pipeline: refusing path-traversal member %r", member.name)
return
# ``filter='data'`` is the Python 3.12+ safe extractor; fall back
# to the legacy call on older runtimes.
try:
tf.extractall(dest, filter='data') # type: ignore[call-arg]
except TypeError:
tf.extractall(dest)
def _extract_rar(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import rarfile # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — rarfile library not installed. "
"Install with: pip install rarfile (and ensure unrar is on PATH).",
archive_path,
)
return None
try:
with rarfile.RarFile(archive_path) as rf:
dest_resolved = dest.resolve()
for name in rf.namelist():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal rar member %r", name)
return None
rf.extractall(dest)
return dest
except Exception as e:
logger.error("archive_pipeline: rar extract failed for %s: %s", archive_path, e)
return None
def _extract_7z(archive_path: Path, dest: Path) -> Optional[Path]:
try:
import py7zr # type: ignore[import-untyped]
except ImportError:
logger.warning(
"archive_pipeline: cannot extract %s — py7zr library not installed. "
"Install with: pip install py7zr.",
archive_path,
)
return None
try:
with py7zr.SevenZipFile(archive_path, 'r') as sz:
dest_resolved = dest.resolve()
for name in sz.getnames():
target = (dest_resolved / name).resolve()
if dest_resolved not in target.parents and target != dest_resolved:
logger.error("archive_pipeline: refusing path-traversal 7z member %r", name)
return None
sz.extractall(path=dest)
return dest
except Exception as e:
logger.error("archive_pipeline: 7z extract failed for %s: %s", archive_path, e)
return None

212
core/artist_source_detail.py
View file

@ -1,212 +0,0 @@
"""Synthesize an artist-detail response for an artist that isn't in the library.
Extracted from ``web_server.py`` so the logic is importable at test time.
The route handler in ``web_server.py`` is now a thin wrapper that builds the
per-source clients (which live as module globals there), calls this function,
and wraps the return value in ``jsonify``.
Used by ``/api/artist-detail/<id>`` when the URL is called with a ``source``
query parameter and the library DB lookup misses. Enriches the response with
whatever metadata we can pull on demand:
* Image URL (via ``core.metadata.artist_image.get_artist_image_url``)
* Source-specific artist info — genres + follower count from the named
source's ``get_artist`` / ``get_artist_info`` helper
* Last.fm bio + listeners + playcount + URL (by artist name)
* Discography from the named source, with variant dedup disabled so every
release surfaces
All per-source clients are passed in explicitly. Callers that can't or don't
want to provide a given client pass ``None`` — the corresponding enrichment
branch becomes a no-op.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, Optional, Tuple
from core.artist_source_lookup import SOURCE_ID_FIELD
from core.metadata import artist_image as metadata_artist_image
from core.metadata import discography as metadata_discography
from core.metadata.lookup import MetadataLookupOptions
logger = logging.getLogger("artist_source_detail")
def build_source_only_artist_detail(
artist_id: str,
artist_name: str,
source: str,
*,
spotify_client: Optional[Any] = None,
deezer_client: Optional[Any] = None,
itunes_client: Optional[Any] = None,
discogs_client: Optional[Any] = None,
amazon_client: Optional[Any] = None,
lastfm_api_key: Optional[str] = None,
) -> Tuple[Dict[str, Any], int]:
"""Build the artist-detail payload for a source-only artist.
Returns ``(payload_dict, http_status)``. Callers wrap the dict in
``jsonify`` or equivalent. Status is 200 on success, 404 when the
source's discography lookup returned no releases.
"""
resolved_name = (artist_name or "").strip()
# 1. Image URL via the same helper /api/artist/<id>/image uses.
image_url: Optional[str] = None
try:
image_url = metadata_artist_image.get_artist_image_url(artist_id, source_override=source)
except Exception as e:
logger.debug(f"Artist image lookup failed for {source}:{artist_id}: {e}")
# 2. Source-side artist info (image, genres, followers depending on source).
source_genres: list = []
source_followers: Optional[int] = None
try:
if source == "spotify" and spotify_client is not None:
sp_artist = spotify_client.get_artist(artist_id, allow_fallback=False)
if sp_artist:
if not artist_name and sp_artist.get("name"):
resolved_name = sp_artist["name"]
source_genres = sp_artist.get("genres") or []
source_followers = (sp_artist.get("followers") or {}).get("total")
if not image_url and sp_artist.get("images"):
image_url = sp_artist["images"][0].get("url")
elif source == "deezer" and deezer_client is not None:
dz_artist = deezer_client.get_artist_info(artist_id)
if dz_artist:
if not artist_name and dz_artist.get("name"):
resolved_name = dz_artist["name"]
source_genres = dz_artist.get("genres") or []
source_followers = (dz_artist.get("followers") or {}).get("total")
elif source == "itunes" and itunes_client is not None:
it_artist = itunes_client.get_artist(artist_id)
if it_artist:
if not artist_name and it_artist.get("name"):
resolved_name = it_artist["name"]
source_genres = it_artist.get("genres") or []
elif source == "discogs" and discogs_client is not None:
dc_artist = discogs_client.get_artist(artist_id)
if dc_artist:
if not artist_name and dc_artist.get("name"):
resolved_name = dc_artist["name"]
source_genres = dc_artist.get("genres") or []
elif source == "amazon" and amazon_client is not None:
az_artist = amazon_client.get_artist(resolved_name or artist_id)
if az_artist:
if not artist_name and az_artist.get("name"):
resolved_name = az_artist["name"]
source_genres = az_artist.get("genres") or []
if not image_url and az_artist.get("images"):
image_url = az_artist["images"][0].get("url")
elif source == "musicbrainz":
try:
from core.musicbrainz_search import MusicBrainzSearchClient
mb = MusicBrainzSearchClient()
mb_artist = mb.get_artist(artist_id)
if mb_artist:
if not artist_name and mb_artist.get("name"):
resolved_name = mb_artist["name"]
source_genres = mb_artist.get("genres") or []
except Exception as e:
logger.debug(f"MusicBrainz artist info lookup failed for {artist_id}: {e}")
except Exception as e:
logger.debug(f"Source-side artist info lookup failed for {source}:{artist_id}: {e}")
# 3. Last.fm enrichment by artist name.
lastfm_bio: Optional[str] = None
lastfm_listeners: Optional[int] = None
lastfm_playcount: Optional[int] = None
lastfm_url: Optional[str] = None
if resolved_name and lastfm_api_key:
try:
from core.lastfm_client import LastFMClient
lastfm = LastFMClient(api_key=lastfm_api_key)
lf_info = lastfm.get_artist_info(resolved_name)
if lf_info:
bio_obj = lf_info.get("bio") or {}
lastfm_bio = bio_obj.get("content") or bio_obj.get("summary")
stats_obj = lf_info.get("stats") or {}
if stats_obj.get("listeners"):
try:
lastfm_listeners = int(stats_obj["listeners"])
except (ValueError, TypeError):
pass
if stats_obj.get("playcount"):
try:
lastfm_playcount = int(stats_obj["playcount"])
except (ValueError, TypeError):
pass
lastfm_url = lf_info.get("url")
except Exception as e:
logger.debug(f"Last.fm enrichment failed for {resolved_name}: {e}")
# 4. Discography from the specified source. Skip variant dedup so the
# page shows every release the source returns — matches the inline
# Artists-page behaviour that this view was modelled after.
discography_result = metadata_discography.get_artist_detail_discography(
artist_id,
artist_name=resolved_name or artist_id,
options=MetadataLookupOptions(
source_override=source,
allow_fallback=True,
skip_cache=False,
max_pages=0,
# Match the Download Discography endpoint cap (200).
# Spotify already paginates all; Deezer / iTunes / Discogs /
# Hydrabase clamp at the outer limit. 200 covers prolific
# catalogues without exceeding iTunes/Discogs internal caps.
limit=200,
artist_source_ids={source: artist_id},
dedup_variants=False,
),
)
if not discography_result.get("success"):
return {
"success": False,
"error": discography_result.get("error", "Could not load discography"),
"source": source,
}, 404
artist_info: Dict[str, Any] = {
"id": artist_id,
"name": resolved_name or artist_id,
"image_url": image_url,
"server_source": None, # not in library
"genres": source_genres,
}
# Stamp the source-specific ID so the correct service badge renders on the
# hero (e.g. source=deezer -> deezer_id; source=spotify -> spotify_artist_id).
source_id_field = SOURCE_ID_FIELD.get(source)
if source_id_field:
artist_info[source_id_field] = artist_id
if source_followers is not None:
artist_info["followers"] = source_followers
if lastfm_bio:
artist_info["lastfm_bio"] = lastfm_bio
if lastfm_listeners is not None:
artist_info["lastfm_listeners"] = lastfm_listeners
if lastfm_playcount is not None:
artist_info["lastfm_playcount"] = lastfm_playcount
if lastfm_url:
artist_info["lastfm_url"] = lastfm_url
logger.info(
f"Source-only artist-detail: {artist_info['name']} from {source} — "
f"albums={len(discography_result.get('albums', []))}, "
f"eps={len(discography_result.get('eps', []))}, "
f"singles={len(discography_result.get('singles', []))}, "
f"genres={len(source_genres)}, lastfm_bio={'yes' if lastfm_bio else 'no'}"
)
return {
"success": True,
"artist": artist_info,
"discography": discography_result,
"enrichment_coverage": {},
}, 200

105
core/artist_source_lookup.py
View file

@ -1,105 +0,0 @@
"""Source-artist → library lookup helpers.
Extracted from `web_server.py` so the logic can be imported and unit-tested
without booting the Flask app, Spotify client, Soulseek connection, etc.
Two concepts live here:
* ``SOURCE_ID_FIELD`` — the per-source column on the ``artists`` table that
stores the external service ID (Spotify track ID, Deezer artist ID, …).
This map is what ties a result clicked in the source-aware Search results
back to a library record so we can serve the richer library view.
* ``find_library_artist_for_source`` — given a source-aware click (e.g.
``deezer:525046``), try to locate a matching library artist. First by
direct column match against the source's ID column, then by case-
insensitive name match scoped to the active media server.
"""
from __future__ import annotations
import logging
from typing import Optional
from core.source_ids import id_column as _artist_id_column
logger = logging.getLogger("artist_source_lookup")
SOURCE_ONLY_ARTIST_SOURCES = frozenset({
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
})
# The per-source column on the ``artists`` table, derived from the canonical
# source-ID registry (the single source of truth). Values are unchanged from the
# previous hardcoded map — this just stops duplicating that knowledge here.
SOURCE_ID_FIELD = {
source: _artist_id_column(source, "artist")
for source in (
"spotify", "itunes", "deezer", "discogs", "hydrabase", "musicbrainz", "amazon",
)
}
def find_library_artist_for_source(
database,
source: str,
source_artist_id: str,
artist_name: Optional[str] = None,
active_server: Optional[str] = None,
) -> Optional[str]:
"""Return the library PK of an artist matching the source-aware click.
Lookup order:
1. Direct match on the source-specific ID column (server-agnostic — any
library record with the right external ID is a hit). If that id is
stamped on MORE than one library artist, the mapping is corrupt /
ambiguous (e.g. an enrichment bug wrote one Deezer id onto several
artists) — we refuse to guess and fall through, so the caller can
show the source artist directly instead of an arbitrary wrong one.
2. Case-insensitive name match within ``active_server`` (defaults to the
active media server when not provided), so we don't jump the user
across server contexts on a name collision.
Returns ``None`` on miss or on any database error.
"""
column = SOURCE_ID_FIELD.get(source)
if not column:
return None
try:
with database._get_connection() as conn:
cursor = conn.cursor()
# LIMIT 2 so we can tell a unique match from an ambiguous one.
cursor.execute(
f"SELECT id FROM artists WHERE {column} = ? LIMIT 2",
(str(source_artist_id),),
)
rows = cursor.fetchall()
if len(rows) == 1:
return rows[0][0]
if len(rows) > 1:
# Same source id on multiple artists — corrupt mapping. Don't
# upgrade on the id; fall through to the name match (and, if
# that misses, let the caller render the source artist).
logger.warning(
f"Source id {source}:{source_artist_id} maps to "
f"{len(rows)}+ library artists — ambiguous, skipping "
f"id-based library upgrade"
)
if artist_name and active_server:
cursor.execute(
"SELECT id FROM artists "
"WHERE LOWER(name) = LOWER(?) AND server_source = ? LIMIT 1",
(artist_name, active_server),
)
row = cursor.fetchone()
if row:
return row[0]
except Exception as e:
logger.debug(
f"Library upgrade lookup failed for {source}:{source_artist_id}: {e}"
)
return None

324
core/artists/liked_match.py
View file

@ -1,324 +0,0 @@
"""Liked-artist multi-source matching — lifted from web_server.py.
Both function bodies are byte-identical to the originals. The
``spotify_client`` proxy + ``_get_*_client`` shims let the bodies resolve
their original names without any modification.
"""
import logging
import time
from config.settings import config_manager
from core.metadata.registry import (
get_deezer_client,
get_discogs_client,
get_itunes_client,
get_musicbrainz_client,
get_spotify_client,
)
logger = logging.getLogger(__name__)
def _get_itunes_client():
"""Mirror of web_server._get_itunes_client — delegates to registry."""
return get_itunes_client()
def _get_deezer_client():
"""Mirror of web_server._get_deezer_client — delegates to registry."""
return get_deezer_client()
def _get_discogs_client(token=None):
"""Mirror of web_server._get_discogs_client — delegates to registry."""
return get_discogs_client(token)
def _get_musicbrainz_client():
"""Mirror of web_server._get_musicbrainz_client — delegates to registry."""
return get_musicbrainz_client()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily so a Spotify re-auth that
rebinds the cached client in core.metadata.registry is visible to the
lifted bodies."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
spotify_client = _SpotifyClientProxy()
def _match_liked_artists_to_all_sources(database, profile_id: int):
"""Match pending liked artists to ALL metadata sources (Spotify, iTunes, Deezer, Discogs).
Uses the same matching pattern as the watchlist scanner: DB-first, then API search
with fuzzy name matching. Stores all resolved IDs so source switching works instantly."""
pending = database.get_liked_artists_pending_match(profile_id, limit=200)
if not pending:
return
# Source → column mapping
source_cols = {
'spotify': 'spotify_artist_id',
'itunes': 'itunes_artist_id',
'deezer': 'deezer_artist_id',
'discogs': 'discogs_artist_id',
'musicbrainz': 'musicbrainz_artist_id',
}
id_cols = list(source_cols.values())
# Reject known placeholder images and local server paths
_placeholder_hashes = {'2a96cbd8b46e442fc41c2b86b821562f'}
def _valid_image(url):
if not url or not url.strip():
return None
if any(ph in url for ph in _placeholder_hashes):
return None
# Reject local media server paths (Plex/Jellyfin) — not loadable in browser
if url.startswith('/') or url.startswith('\\'):
return None
if not url.startswith('http'):
return None
return url
# Build search clients for each source
from core.deezer_client import DeezerClient
search_clients = {}
if spotify_client and spotify_client.is_spotify_authenticated():
search_clients['spotify'] = spotify_client
try:
search_clients['itunes'] = _get_itunes_client()
except Exception as e:
logger.debug("itunes client init failed: %s", e)
try:
search_clients['deezer'] = _get_deezer_client()
except Exception as e:
logger.debug("deezer client init failed: %s", e)
try:
dc = _get_discogs_client()
# Only use Discogs if token is configured
from config.settings import config_manager as _cm
if _cm.get('discogs.token', ''):
search_clients['discogs'] = dc
except Exception as e:
logger.debug("discogs client init failed: %s", e)
try:
search_clients['musicbrainz'] = _get_musicbrainz_client()
except Exception as e:
logger.debug("musicbrainz client init failed: %s", e)
# Reuse watchlist scanner's fuzzy matching logic
from core.watchlist_scanner import WatchlistScanner
_normalize = WatchlistScanner._normalize_artist_name
def _best_match(results, artist_name):
"""Pick best match from search results using name similarity (same as watchlist scanner)."""
if not results:
return None
# Exact normalized match
for r in results:
if _normalize(r.name) == _normalize(artist_name):
return r
# Fuzzy scoring
best = None
best_sim = 0
for r in results:
# Simple normalized comparison
n1 = _normalize(artist_name)
n2 = _normalize(r.name)
if n1 == n2:
return r
# Levenshtein-style similarity
max_len = max(len(n1), len(n2))
if max_len == 0:
continue
distance = sum(1 for a, b in zip(n1, n2, strict=False) if a != b) + abs(len(n1) - len(n2))
sim = (max_len - distance) / max_len
if sim > best_sim:
best_sim = sim
best = r
if best and best_sim >= 0.85:
return best
return None
api_calls = 0
matched = 0
for entry in pending:
name = entry['artist_name']
pool_id = entry['id']
harvested_ids = {}
best_image = None
# Pre-load existing IDs from the entry itself
for col in id_cols:
if entry.get(col):
harvested_ids[col] = entry[col]
# --- DB STRATEGIES (free, no API calls) ---
# 1. Library artists table
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("SELECT * FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (name,))
row = cursor.fetchone()
if row:
r = dict(row)
for col in id_cols:
if r.get(col) and col not in harvested_ids:
harvested_ids[col] = str(r[col])
if _valid_image(r.get('thumb_url')):
best_image = r['thumb_url']
except Exception as e:
logger.debug("library artist lookup failed: %s", e)
# 2. Watchlist artists
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute(
"SELECT * FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE AND profile_id = ? LIMIT 1",
(name, profile_id)
)
row = cursor.fetchone()
if row:
wl = dict(row)
for col in id_cols:
if wl.get(col) and col not in harvested_ids:
harvested_ids[col] = str(wl[col])
if _valid_image(wl.get('image_url')) and not best_image:
best_image = wl['image_url']
except Exception as e:
logger.debug("watchlist artist lookup failed: %s", e)
# 3. Metadata cache (all sources)
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute(
"SELECT entity_id, source, image_url FROM metadata_cache_entities WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE",
(name,)
)
for row in cursor.fetchall():
col = source_cols.get(row['source'])
if col and col not in harvested_ids:
harvested_ids[col] = row['entity_id']
if _valid_image(row['image_url']) and not best_image:
best_image = row['image_url']
except Exception as e:
logger.debug("metadata cache lookup failed: %s", e)
# --- API STRATEGIES (search each missing source) ---
# Same pattern as watchlist scanner's _backfill_missing_ids
for source, col in source_cols.items():
if col in harvested_ids:
continue # Already have this source's ID
client = search_clients.get(source)
if not client:
continue
if api_calls >= 200: # Hard cap per refresh cycle
break
try:
results = client.search_artists(name, limit=5)
best = _best_match(results, name)
if best:
harvested_ids[col] = best.id
if hasattr(best, 'image_url') and _valid_image(best.image_url) and not best_image:
best_image = best.image_url
api_calls += 1
time.sleep(0.4) # Rate limit breathing room
except Exception as e:
logger.debug(f"[Your Artists] {source} search failed for '{name}': {e}")
api_calls += 1
# Save all harvested IDs
if harvested_ids:
# Determine best active source/ID — prefer Spotify, then iTunes, Deezer, Discogs
resolved_source = None
resolved_id = None
for src in ('spotify', 'itunes', 'deezer', 'discogs', 'musicbrainz'):
col = source_cols[src]
if col in harvested_ids:
resolved_source = src
resolved_id = harvested_ids[col]
break
database.update_liked_artist_match(
pool_id, active_source=resolved_source, active_source_id=resolved_id,
image_url=best_image, all_ids=harvested_ids
)
matched += 1
database.sync_liked_artists_watchlist_flags(profile_id)
logger.info(f"[Your Artists] Matched {matched}/{len(pending)} artists to {len(search_clients)} sources ({api_calls} API calls)")
# Image backfill: fetch images for matched artists that have IDs but no image
_backfill_liked_artist_images(database, profile_id, search_clients)
def _backfill_liked_artist_images(database, profile_id: int, search_clients: dict):
"""Fetch images for matched artists missing artwork using their stored source IDs."""
try:
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id
FROM liked_artists_pool
WHERE profile_id = ? AND match_status = 'matched'
AND (image_url IS NULL OR image_url = ''
OR image_url LIKE '%2a96cbd8b46e442fc41c2b86b821562f%'
OR image_url NOT LIKE 'http%')
LIMIT 100
""", (profile_id,))
rows = cursor.fetchall()
if not rows:
return
logger.info(f"[Your Artists] Backfilling images for {len(rows)} artists...")
filled = 0
for row in rows:
r = dict(row)
image_url = None
# Try Spotify artist lookup (has best images)
if r.get('spotify_artist_id') and 'spotify' in search_clients:
try:
sp = search_clients['spotify']
if hasattr(sp, 'sp') and sp.sp:
artist_data = sp.sp.artist(r['spotify_artist_id'])
if artist_data and artist_data.get('images'):
image_url = artist_data['images'][0]['url']
except Exception as e:
logger.debug("spotify artist image fetch failed: %s", e)
# Try Deezer (direct image URL from ID)
if not image_url and r.get('deezer_artist_id'):
image_url = f"https://api.deezer.com/artist/{r['deezer_artist_id']}/image?size=big"
if image_url:
try:
cursor2 = conn.cursor()
cursor2.execute(
"UPDATE liked_artists_pool SET image_url = ? WHERE id = ?",
(image_url, r['id'])
)
filled += 1
except Exception as e:
logger.debug("liked artist image update failed: %s", e)
time.sleep(0.3)
conn.commit()
if filled:
logger.info(f"[Your Artists] Backfilled {filled}/{len(rows)} artist images")
except Exception as e:
logger.debug(f"[Your Artists] Image backfill error: {e}")

994
core/artists/map.py
View file

@ -1,994 +0,0 @@
"""Artist Map endpoints — lifted from web_server.py.
The four route bodies (``get_artist_map_data``, ``get_artist_map_genre_list``,
``get_artist_map_genres``, ``get_artist_map_explore``) plus their cache helpers
and the artist-map cache are byte-identical to the originals. Module-level
shims for ``get_current_profile_id``, ``_get_itunes_client``, and the
``spotify_client`` proxy let the bodies resolve their original names without
modification.
"""
import json
import logging
import time
from flask import g, jsonify, request
from database.music_database import get_database
from core.metadata.registry import get_itunes_client, get_spotify_client
logger = logging.getLogger(__name__)
def get_current_profile_id() -> int:
"""Mirror of web_server.get_current_profile_id — uses Flask g.
Catches RuntimeError too because reading `g` outside a request
context raises that (not AttributeError) — happens when this is
called from background threads (sync, automation, scanners)."""
try:
return g.profile_id
except (AttributeError, RuntimeError):
return 1
def _get_itunes_client():
"""Mirror of web_server._get_itunes_client — delegates to registry."""
return get_itunes_client()
class _SpotifyClientProxy:
"""Resolves the global Spotify client lazily so a Spotify re-auth that
rebinds the cached client in core.metadata.registry is visible to the
lifted route bodies."""
def __getattr__(self, name):
client = get_spotify_client()
if client is None:
raise AttributeError(name)
return getattr(client, name)
def __bool__(self):
return get_spotify_client() is not None
spotify_client = _SpotifyClientProxy()
# Artist Map data cache — avoids re-querying 4+ tables on every request
# Keys: 'watchlist_{profile}', 'genres_{profile}', 'genre_list'
# Values: {'data': <json-ready dict>, 'ts': <timestamp>}
_artist_map_cache = {}
_ARTIST_MAP_CACHE_TTL = 300 # 5 minutes
def _artmap_cache_get(key):
"""Get cached artist map data if still fresh."""
entry = _artist_map_cache.get(key)
if entry and (time.time() - entry['ts']) < _ARTIST_MAP_CACHE_TTL:
return entry['data']
return None
def _artmap_cache_set(key, data):
"""Store artist map data in cache."""
_artist_map_cache[key] = {'data': data, 'ts': time.time()}
def _artmap_cache_invalidate(profile_id=None):
"""Invalidate artist map cache (call after watchlist changes, scans, etc.)."""
if profile_id:
_artist_map_cache.pop(f'watchlist_{profile_id}', None)
_artist_map_cache.pop(f'genres_{profile_id}', None)
_artist_map_cache.pop('genre_list', None)
def get_artist_map_data():
"""Get watchlist artists + their similar artists for the force-directed artist map."""
try:
database = get_database()
profile_id = get_current_profile_id()
cached = _artmap_cache_get(f'watchlist_{profile_id}')
if cached:
return jsonify(cached)
# Get all watchlist artists
conn = database._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT id, artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
discogs_artist_id, image_url
FROM watchlist_artists WHERE profile_id = ?
""", (profile_id,))
watchlist_rows = cursor.fetchall()
nodes = [] # {id, name, image_url, type: 'watchlist'|'similar', genres, size}
edges = [] # {source, target, weight}
seen_names = {} # normalized_name → node index
def _norm(name):
return (name or '').lower().strip()
# Add watchlist artists as anchor nodes
for wa in watchlist_rows:
w = dict(wa)
norm = _norm(w['artist_name'])
if norm in seen_names:
continue
idx = len(nodes)
seen_names[norm] = idx
# Get image — prefer HTTP URLs
img = w.get('image_url', '') or ''
if img and not img.startswith('http'):
img = ''
nodes.append({
'id': idx,
'name': w['artist_name'],
'image_url': img,
'type': 'watchlist',
'genres': [],
'spotify_id': w.get('spotify_artist_id') or '',
'itunes_id': w.get('itunes_artist_id') or '',
'deezer_id': w.get('deezer_artist_id') or '',
'discogs_id': w.get('discogs_artist_id') or '',
'source_db_id': str(w['id']),
})
# Get all similar artists for all watchlist artists
watchlist_ids = [dict(wa)['spotify_artist_id'] or dict(wa)['itunes_artist_id'] or str(dict(wa)['id']) for wa in watchlist_rows]
if watchlist_ids:
placeholders = ','.join(['?'] * len(watchlist_ids))
cursor.execute(f"""
SELECT source_artist_id, similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
similarity_rank, occurrence_count, image_url, genres, popularity
FROM similar_artists
WHERE profile_id = ? AND source_artist_id IN ({placeholders})
ORDER BY similarity_rank ASC
""", [profile_id] + watchlist_ids)
for row in cursor.fetchall():
r = dict(row)
sim_norm = _norm(r['similar_artist_name'])
# Find or create similar artist node
if sim_norm not in seen_names:
idx = len(nodes)
seen_names[sim_norm] = idx
img = r.get('image_url', '') or ''
if img and not img.startswith('http'):
img = ''
genres = []
if r.get('genres'):
try:
genres = json.loads(r['genres'])
except Exception as e:
logger.debug("similar node genres parse failed: %s", e)
nodes.append({
'id': idx,
'name': r['similar_artist_name'],
'image_url': img,
'type': 'similar',
'genres': genres,
'spotify_id': r.get('similar_artist_spotify_id') or '',
'itunes_id': r.get('similar_artist_itunes_id') or '',
'deezer_id': r.get('similar_artist_deezer_id') or '',
'musicbrainz_id': r.get('similar_artist_musicbrainz_id') or '',
'rank': r.get('similarity_rank', 5),
'occurrence': r.get('occurrence_count', 1),
'popularity': r.get('popularity', 0),
})
sim_idx = seen_names[sim_norm]
# Find the watchlist node that sourced this similar artist
source_norm = None
for wa in watchlist_rows:
w = dict(wa)
sid = w.get('spotify_artist_id') or w.get('itunes_artist_id') or str(w['id'])
if sid == r['source_artist_id']:
source_norm = _norm(w['artist_name'])
break
if source_norm and source_norm in seen_names:
source_idx = seen_names[source_norm]
# Weight: inverse of rank (rank 1 = strongest connection)
weight = max(1, 11 - (r.get('similarity_rank', 5)))
edges.append({
'source': source_idx,
'target': sim_idx,
'weight': weight,
})
# Also check if any similar artists ARE watchlist artists (cross-links)
# These create extra connections between watchlist nodes
for i, node in enumerate(nodes):
if node['type'] == 'similar':
# Check if this similar artist is also a watchlist artist
for j, wnode in enumerate(nodes):
if wnode['type'] == 'watchlist' and i != j:
if _norm(node['name']) == _norm(wnode['name']):
# Merge: upgrade the similar node to watchlist
node['type'] = 'watchlist'
break
# ── Backfill from metadata cache: batch-lookup all node names across all sources ──
# Single query to get ALL cached artist entries matching ANY node name
try:
all_names = list(set(_norm(n['name']) for n in nodes if n.get('name')))
if all_names:
# Build case-insensitive IN clause via temp matching
# Lightweight query — no raw_json (can be huge)
cursor.execute("""
SELECT entity_id, source, name, image_url, genres, popularity
FROM metadata_cache_entities
WHERE entity_type = 'artist'
""")
cache_rows = cursor.fetchall()
# Index cache by normalized name → {source: {id, image_url, genres}}
cache_by_name = {}
for cr in cache_rows:
cn = _norm(cr['name'] or '')
if cn not in cache_by_name:
cache_by_name[cn] = {}
source = cr['source']
genres = []
if cr['genres']:
try:
genres = json.loads(cr['genres']) if isinstance(cr['genres'], str) else []
except Exception as e:
logger.debug("backfill cache genres parse failed: %s", e)
cache_by_name[cn][source] = {
'id': cr['entity_id'],
'image_url': cr['image_url'] or '',
'genres': genres,
}
# Apply cache data to nodes
source_id_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
for n in nodes:
nn = _norm(n['name'])
cached = cache_by_name.get(nn)
if not cached:
continue
for source, field in source_id_map.items():
if not n.get(field) and source in cached:
n[field] = cached[source]['id']
# Backfill image if missing or local path
if not n.get('image_url') or not n['image_url'].startswith('http'):
for source in ('spotify', 'deezer', 'itunes'):
if source in cached and cached[source].get('image_url', '').startswith('http'):
n['image_url'] = cached[source]['image_url']
break
# Backfill genres if missing
if not n.get('genres') or len(n.get('genres', [])) == 0:
for source in ('spotify', 'deezer', 'itunes', 'discogs', 'musicbrainz'):
if source in cached and cached[source].get('genres'):
n['genres'] = cached[source]['genres'][:5]
break
# Deezer direct URL fallback
for n in nodes:
if not n.get('image_url') or not n['image_url'].startswith('http'):
if n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
# Album art fallback (iTunes artists have no artist images)
_album_art = {}
try:
cursor.execute("""
SELECT artist_name, image_url FROM metadata_cache_entities
WHERE entity_type = 'album' AND image_url LIKE 'http%'
AND artist_name IS NOT NULL AND artist_name != ''
""")
for r in cursor.fetchall():
an = _norm(r['artist_name'])
if an and an not in _album_art:
_album_art[an] = r['image_url']
except Exception as e:
logger.debug("artist map album-art cache build failed: %s", e)
for n in nodes:
if not n.get('image_url') or not n['image_url'].startswith('http'):
nn = _norm(n['name'])
if nn in _album_art:
n['image_url'] = _album_art[nn]
except Exception as cache_err:
logger.debug(f"Artist map cache backfill error: {cache_err}")
result = {
'success': True,
'nodes': nodes,
'edges': edges,
'watchlist_count': sum(1 for n in nodes if n['type'] == 'watchlist'),
'similar_count': sum(1 for n in nodes if n['type'] == 'similar'),
}
_artmap_cache_set(f'watchlist_{profile_id}', result)
return jsonify(result)
except Exception as e:
logger.error(f"Error getting artist map data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_genre_list():
"""Lightweight endpoint — just genre names + counts for the picker. No node data."""
try:
cached = _artmap_cache_get('genre_list')
if cached:
return jsonify(cached)
database = get_database()
conn = database._get_connection()
cursor = conn.cursor()
# Fast query: just count artists per genre from cache
genre_counts = {}
cursor.execute("""
SELECT genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND genres IS NOT NULL AND genres != '' AND genres != '[]'
""")
for r in cursor.fetchall():
try:
for g in json.loads(r['genres']):
if g and isinstance(g, str):
gl = g.lower().strip()
genre_counts[gl] = genre_counts.get(gl, 0) + 1
except Exception as e:
logger.debug("genre count row parse failed: %s", e)
# Sort by count descending
sorted_genres = sorted(genre_counts.items(), key=lambda x: -x[1])
result = {
'success': True,
'genres': [{'name': g, 'count': c} for g, c in sorted_genres],
'total': len(sorted_genres)
}
_artmap_cache_set('genre_list', result)
return jsonify(result)
except Exception as e:
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_genres():
"""Get ALL artists from every data source, grouped by genre for the genre map."""
try:
database = get_database()
profile_id = get_current_profile_id()
cached = _artmap_cache_get(f'genres_{profile_id}')
if cached:
return jsonify(cached)
conn = database._get_connection()
cursor = conn.cursor()
artists_by_name = {} # normalized_name → {name, image, genres[], sources, ids}
def _norm(n):
return (n or '').lower().strip()
def _add(name, image_url=None, genres=None, spotify_id=None, itunes_id=None, deezer_id=None, discogs_id=None, musicbrainz_id=None, source='unknown', popularity=0):
n = _norm(name)
if not n or len(n) < 2:
return
if n not in artists_by_name:
artists_by_name[n] = {
'name': name, 'image_url': '', 'genres': set(),
'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': '',
'sources': set(), 'popularity': 0
}
a = artists_by_name[n]
if image_url and image_url.startswith('http') and not a['image_url']:
a['image_url'] = image_url
if genres:
for g in (genres if isinstance(genres, list) else []):
if g and isinstance(g, str):
a['genres'].add(g.lower().strip())
if spotify_id and not a['spotify_id']:
a['spotify_id'] = str(spotify_id)
if itunes_id and not a['itunes_id']:
a['itunes_id'] = str(itunes_id)
if deezer_id and not a['deezer_id']:
a['deezer_id'] = str(deezer_id)
if discogs_id and not a['discogs_id']:
a['discogs_id'] = str(discogs_id)
if musicbrainz_id and not a['musicbrainz_id']:
a['musicbrainz_id'] = str(musicbrainz_id)
if popularity > a['popularity']:
a['popularity'] = popularity
a['sources'].add(source)
# 1. Metadata cache — biggest source
cursor.execute("""
SELECT name, entity_id, source, image_url, genres, popularity
FROM metadata_cache_entities WHERE entity_type = 'artist'
""")
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("cache artist genres parse failed: %s", e)
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
kwargs = {src_map.get(r['source'], 'spotify_id'): r['entity_id']}
_add(r['name'], image_url=r['image_url'], genres=genres, source='cache', popularity=r['popularity'] or 0, **kwargs)
# 2. Similar artists
cursor.execute("""
SELECT similar_artist_name, similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id, image_url, genres, popularity
FROM similar_artists WHERE profile_id = ?
""", (profile_id,))
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("similar artist genres parse failed: %s", e)
_add(r['similar_artist_name'], image_url=r['image_url'], genres=genres,
spotify_id=r['similar_artist_spotify_id'], itunes_id=r['similar_artist_itunes_id'],
deezer_id=r['similar_artist_deezer_id'],
musicbrainz_id=r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else None,
source='similar', popularity=r['popularity'] or 0)
# 3. Watchlist artists
cursor.execute("""
SELECT artist_name, spotify_artist_id, itunes_artist_id, deezer_artist_id,
discogs_artist_id, image_url
FROM watchlist_artists WHERE profile_id = ?
""", (profile_id,))
for r in cursor.fetchall():
_add(r['artist_name'], image_url=r['image_url'],
spotify_id=r['spotify_artist_id'], itunes_id=r['itunes_artist_id'],
deezer_id=r['deezer_artist_id'], discogs_id=r['discogs_artist_id'], source='watchlist')
# 4. Library artists
cursor.execute("SELECT name, thumb_url, genres FROM artists")
for r in cursor.fetchall():
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("library artist genres parse failed: %s", e)
img = r['thumb_url'] if r['thumb_url'] and r['thumb_url'].startswith('http') else None
_add(r['name'], image_url=img, genres=genres, source='library')
# Filter: only include artists that have at least one genre
genre_artists = {k: v for k, v in artists_by_name.items() if v['genres']}
# Build genre → artists map
genre_map = {} # genre_name → [artist_keys]
for key, a in genre_artists.items():
for g in a['genres']:
if g not in genre_map:
genre_map[g] = []
genre_map[g].append(key)
# Sort genres by artist count, take top genres
sorted_genres = sorted(genre_map.items(), key=lambda x: -len(x[1]))
# Build nodes
nodes = []
node_idx = {}
for key, a in genre_artists.items():
idx = len(nodes)
node_idx[key] = idx
nodes.append({
'id': idx,
'name': a['name'],
'image_url': a['image_url'],
'genres': list(a['genres'])[:5],
'spotify_id': a['spotify_id'],
'itunes_id': a['itunes_id'],
'deezer_id': a['deezer_id'],
'discogs_id': a['discogs_id'],
'musicbrainz_id': a['musicbrainz_id'],
'popularity': a['popularity'],
'type': 'watchlist' if 'watchlist' in a['sources'] else 'similar',
})
# Build genre clusters — allow artists in multiple genres
top_genres = sorted_genres[:40]
# Sort genres by co-occurrence so related genres are adjacent in the list.
# This makes the spiral layout place related genres near each other.
if len(top_genres) > 2:
genre_sets = {g: set(keys) for g, keys in top_genres}
ordered = [top_genres[0][0]] # Start with biggest genre
remaining = {g for g, _ in top_genres[1:]}
while remaining:
last = ordered[-1]
last_set = genre_sets.get(last, set())
# Find most similar remaining genre (highest artist overlap)
best = None
best_overlap = -1
for g in remaining:
overlap = len(last_set & genre_sets.get(g, set()))
if overlap > best_overlap:
best_overlap = overlap
best = g
ordered.append(best)
remaining.remove(best)
# Rebuild top_genres in the ordered sequence
genre_dict = dict(top_genres)
top_genres = [(g, genre_dict[g]) for g in ordered if g in genre_dict]
genres_out = []
for genre, artist_keys in top_genres:
genres_out.append({
'name': genre,
'count': len(artist_keys),
'artist_ids': [node_idx[k] for k in artist_keys if k in node_idx],
})
# Image cleanup + multi-source fallback
# Build two lookups: name→image_url AND name→deezer_entity_id
_img_cache = {}
_deezer_id_cache = {}
_album_art_cache = {} # artist_name → album image (iTunes fallback)
try:
# Artist images + Deezer IDs
cursor.execute("""
SELECT name, entity_id, source, image_url FROM metadata_cache_entities
WHERE entity_type = 'artist'
AND ((image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%')
OR source = 'deezer')
""")
for r in cursor.fetchall():
nn = (r['name'] or '').lower().strip()
if not nn:
continue
if r['image_url'] and r['image_url'].startswith('http') and nn not in _img_cache:
_img_cache[nn] = r['image_url']
if r['source'] == 'deezer' and r['entity_id'] and nn not in _deezer_id_cache:
_deezer_id_cache[nn] = r['entity_id']
# Album art by artist name (for iTunes artists with no artist image)
cursor.execute("""
SELECT artist_name, image_url FROM metadata_cache_entities
WHERE entity_type = 'album'
AND image_url IS NOT NULL AND image_url != '' AND image_url LIKE 'http%'
AND artist_name IS NOT NULL AND artist_name != ''
""")
for r in cursor.fetchall():
nn = (r['artist_name'] or '').lower().strip()
if nn and nn not in _album_art_cache:
_album_art_cache[nn] = r['image_url']
except Exception as e:
logger.debug("genre map cache build failed: %s", e)
for n in nodes:
img = n.get('image_url', '')
if img in ('None', 'null', '') or (img and not img.startswith('http')):
n['image_url'] = ''
nn = n['name'].lower().strip()
if not n['image_url']:
# Try cache image by name
n['image_url'] = _img_cache.get(nn, '')
if not n['image_url'] and n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
if not n['image_url']:
# Try Deezer ID from cache by name
did = _deezer_id_cache.get(nn)
if did:
n['deezer_id'] = did
n['image_url'] = f"https://api.deezer.com/artist/{did}/image?size=big"
if not n['image_url']:
# Try album art by artist name (iTunes artists have no artist images)
n['image_url'] = _album_art_cache.get(nn, '')
_img_count = sum(1 for n in nodes if n.get('image_url'))
_deezer_count = sum(1 for n in nodes if n.get('image_url', '').startswith('https://api.deezer'))
_none_count = sum(1 for n in nodes if not n.get('image_url'))
logger.info(f"[Genre Map] {len(nodes)} artists, {len(sorted_genres)} genres")
logger.warning(f"[Genre Map] Images: {_img_count} have URLs, {_deezer_count} Deezer fallback, {_none_count} missing")
if _none_count > 0:
samples = [n['name'] for n in nodes if not n.get('image_url')][:5]
logger.warning(f"[Genre Map] Missing image samples: {samples}")
result = {
'success': True,
'nodes': nodes,
'genres': genres_out,
'total_artists': len(nodes),
'total_genres': len(sorted_genres),
}
_artmap_cache_set(f'genres_{profile_id}', result)
return jsonify(result)
except Exception as e:
logger.error(f"Error getting genre map data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500
def get_artist_map_explore():
"""Build an exploration map outward from a single artist."""
try:
artist_name = request.args.get('name', '').strip()
artist_id = request.args.get('id', '').strip()
if not artist_name and not artist_id:
return jsonify({"success": False, "error": "Provide artist name or id"}), 400
database = get_database()
profile_id = get_current_profile_id()
conn = database._get_connection()
cursor = conn.cursor()
def _norm(n):
return (n or '').lower().strip()
nodes = []
edges = []
seen = {} # norm_name → node index
# Find the center artist
center_name = artist_name
center_image = ''
center_ids = {'spotify_id': '', 'itunes_id': '', 'deezer_id': '', 'discogs_id': '', 'musicbrainz_id': ''}
center_genres = []
# Search metadata cache for the center artist
if artist_id:
cursor.execute("""
SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND entity_id = ? LIMIT 1
""", (artist_id,))
else:
cursor.execute("""
SELECT name, entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE LIMIT 1
""", (artist_name,))
row = cursor.fetchone()
artist_found = False
if row:
artist_found = True
center_name = row['name']
if row['image_url'] and row['image_url'].startswith('http'):
center_image = row['image_url']
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(row['source'], 'spotify_id')
center_ids[k] = row['entity_id']
if row['genres']:
try:
center_genres = json.loads(row['genres']) if isinstance(row['genres'], str) else []
except Exception as e:
logger.debug("initial center genres parse failed: %s", e)
# Check watchlist + library if not in cache
if not artist_found and not artist_id:
cursor.execute("SELECT artist_name, image_url, spotify_artist_id, itunes_artist_id, deezer_artist_id, discogs_artist_id, musicbrainz_artist_id FROM watchlist_artists WHERE artist_name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
wr = cursor.fetchone()
if wr:
artist_found = True
center_name = wr['artist_name']
if wr['image_url'] and str(wr['image_url']).startswith('http'):
center_image = wr['image_url']
for k, col in [('spotify_id', 'spotify_artist_id'), ('itunes_id', 'itunes_artist_id'), ('deezer_id', 'deezer_artist_id'), ('discogs_id', 'discogs_artist_id'), ('musicbrainz_id', 'musicbrainz_artist_id')]:
if wr[col]:
center_ids[k] = str(wr[col])
else:
cursor.execute("SELECT name, thumb_url FROM artists WHERE name = ? COLLATE NOCASE LIMIT 1", (artist_name,))
lr = cursor.fetchone()
if lr:
artist_found = True
center_name = lr['name']
if lr['thumb_url'] and str(lr['thumb_url']).startswith('http'):
center_image = lr['thumb_url']
# If not found locally, validate via metadata API search
if not artist_found and not artist_id:
try:
api_match = None
if spotify_client and spotify_client.is_spotify_authenticated():
results = spotify_client.search_artists(artist_name, limit=1)
if results and len(results) > 0:
sa = results[0]
if sa.name.lower().strip() == artist_name.lower().strip() or \
artist_name.lower().strip() in sa.name.lower().strip():
api_match = sa
center_name = sa.name
center_ids['spotify_id'] = sa.id
center_image = sa.image_url if hasattr(sa, 'image_url') else ''
center_genres = sa.genres if hasattr(sa, 'genres') else []
artist_found = True
if not artist_found:
ic = _get_itunes_client()
results = ic.search_artists(artist_name, limit=1)
if results and len(results) > 0:
ia = results[0]
if ia.name.lower().strip() == artist_name.lower().strip() or \
artist_name.lower().strip() in ia.name.lower().strip():
center_name = ia.name
center_ids['itunes_id'] = str(ia.id)
center_image = ia.image_url if hasattr(ia, 'image_url') else ''
artist_found = True
except Exception as e:
logger.debug(f"[Artist Explorer] API validation failed for '{artist_name}': {e}")
if not artist_found:
return jsonify({"success": False, "error": f"Artist '{artist_name}' not found"}), 404
# Also check cache for other source IDs
cursor.execute("""
SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
""", (center_name,))
for r in cursor.fetchall():
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(r['source'], 'spotify_id')
if not center_ids.get(k):
center_ids[k] = r['entity_id']
if r['image_url'] and r['image_url'].startswith('http') and not center_image:
center_image = r['image_url']
if r['genres'] and not center_genres:
try:
center_genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("center genres parse failed: %s", e)
# Add center node
center_idx = 0
seen[_norm(center_name)] = center_idx
nodes.append({
'id': 0, 'name': center_name, 'image_url': center_image,
'type': 'center', 'genres': center_genres[:5],
**center_ids, 'ring': 0
})
# Ring 1: Direct similar artists from similar_artists table
# Search by all known IDs
id_values = [v for v in center_ids.values() if v]
ring1_artists = []
if id_values:
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", id_values + [profile_id])
ring1_artists = cursor.fetchall()
# Also search by name (the center artist might be a watchlist source)
cursor.execute("""
SELECT DISTINCT sa.similar_artist_name, sa.similar_artist_spotify_id,
sa.similar_artist_itunes_id, sa.similar_artist_deezer_id, sa.similar_artist_musicbrainz_id,
sa.image_url, sa.genres, sa.popularity, sa.similarity_rank
FROM similar_artists sa
JOIN watchlist_artists wa ON sa.source_artist_id = COALESCE(wa.spotify_artist_id, wa.itunes_artist_id, CAST(wa.id AS TEXT))
WHERE wa.artist_name = ? COLLATE NOCASE AND sa.profile_id = ?
ORDER BY sa.similarity_rank ASC
""", (center_name, profile_id))
ring1_artists.extend(cursor.fetchall())
# If no similar artists in DB, fetch from MusicMap on-the-fly
if not ring1_artists:
try:
logger.debug(f"[Artist Explorer] No stored similar artists for '{center_name}', fetching from MusicMap...")
from core.watchlist_scanner import WatchlistScanner
scanner = WatchlistScanner(spotify_client=spotify_client) if spotify_client else None
if scanner:
similar = scanner._fetch_similar_artists_from_musicmap(center_name, limit=15)
if similar:
source_artist_id = center_ids.get('spotify_id') or center_ids.get('itunes_id') or center_name
# Store in DB for future use
for rank, sa in enumerate(similar, 1):
try:
database.add_or_update_similar_artist(
source_artist_id=source_artist_id,
similar_artist_name=sa['name'],
similar_artist_spotify_id=sa.get('spotify_id'),
similar_artist_itunes_id=sa.get('itunes_id'),
similarity_rank=rank,
profile_id=profile_id,
image_url=sa.get('image_url'),
genres=sa.get('genres'),
popularity=sa.get('popularity', 0),
similar_artist_deezer_id=sa.get('deezer_id'),
similar_artist_musicbrainz_id=sa.get('musicbrainz_id'),
)
except Exception as e:
logger.debug("similar artist insert failed: %s", e)
# Re-query from DB to get consistent format
if id_values:
placeholders = ','.join(['?'] * len(id_values))
cursor.execute(f"""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", id_values + [profile_id])
ring1_artists = cursor.fetchall()
if not ring1_artists:
# Fallback: query by name-based source ID
cursor.execute("""
SELECT DISTINCT similar_artist_name, similar_artist_spotify_id,
similar_artist_itunes_id, similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id = ? AND profile_id = ?
ORDER BY similarity_rank ASC
""", (source_artist_id, profile_id))
ring1_artists = cursor.fetchall()
logger.debug(f"[Artist Explorer] Fetched {len(ring1_artists)} similar artists from MusicMap for '{center_name}'")
_artmap_cache_invalidate(profile_id) # New similar artists added
except Exception as e:
logger.debug(f"[Artist Explorer] MusicMap fetch failed for '{center_name}': {e}")
# Deduplicate ring 1
for r in ring1_artists:
nn = _norm(r['similar_artist_name'])
if nn in seen:
continue
idx = len(nodes)
seen[nn] = idx
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("ring1 genres parse failed: %s", e)
img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
nodes.append({
'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
'type': 'ring1', 'genres': genres[:5],
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
'ring': 1,
})
weight = max(1, 11 - (r['similarity_rank'] or 5))
edges.append({'source': center_idx, 'target': idx, 'weight': weight})
# Ring 2: Similar artists of ring 1 artists (from similar_artists table)
ring1_ids = []
for n in nodes[1:]: # skip center
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid:
ring1_ids.append(sid)
if ring1_ids:
placeholders = ','.join(['?'] * len(ring1_ids))
cursor.execute(f"""
SELECT DISTINCT source_artist_id, similar_artist_name,
similar_artist_spotify_id, similar_artist_itunes_id,
similar_artist_deezer_id, similar_artist_musicbrainz_id,
image_url, genres, popularity, similarity_rank
FROM similar_artists
WHERE source_artist_id IN ({placeholders}) AND profile_id = ?
ORDER BY similarity_rank ASC
""", ring1_ids + [profile_id])
for r in cursor.fetchall():
nn = _norm(r['similar_artist_name'])
if nn in seen:
# Create edge to existing node if not center
existing_idx = seen[nn]
# Find the ring1 node that sourced this
source_norm = None
for n in nodes[1:]:
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid == r['source_artist_id']:
source_norm = _norm(n['name'])
break
if source_norm:
break
if source_norm and source_norm in seen and existing_idx != seen[source_norm]:
edges.append({'source': seen[source_norm], 'target': existing_idx, 'weight': 3})
continue
idx = len(nodes)
if idx >= 500: # Cap at 500 nodes for performance
break
seen[nn] = idx
genres = []
if r['genres']:
try:
genres = json.loads(r['genres']) if isinstance(r['genres'], str) else []
except Exception as e:
logger.debug("ring2 genres parse failed: %s", e)
img = r['image_url'] if r['image_url'] and r['image_url'].startswith('http') else ''
nodes.append({
'id': idx, 'name': r['similar_artist_name'], 'image_url': img,
'type': 'ring2', 'genres': genres[:5],
'spotify_id': r['similar_artist_spotify_id'] or '',
'itunes_id': r['similar_artist_itunes_id'] or '',
'deezer_id': r['similar_artist_deezer_id'] or '',
'musicbrainz_id': r['similar_artist_musicbrainz_id'] if 'similar_artist_musicbrainz_id' in r.keys() else '',
'discogs_id': '',
'popularity': r['popularity'] or 0,
'rank': r['similarity_rank'] or 5,
'ring': 2,
})
# Find the ring1 source
for n in nodes[1:]:
for sid in [n.get('spotify_id'), n.get('itunes_id')]:
if sid == r['source_artist_id']:
edges.append({'source': n['id'], 'target': idx, 'weight': max(1, 11 - (r['similarity_rank'] or 5))})
break
# Backfill images/genres from ALL cache sources + Deezer fallback
for n in nodes:
# Clean up string "None" stored as image URL
if n['image_url'] in ('None', 'null', ''):
n['image_url'] = ''
if n['image_url'] and n['genres']:
continue
# Check all cache entries for this artist (multiple sources)
cursor.execute("""
SELECT entity_id, source, image_url, genres FROM metadata_cache_entities
WHERE entity_type = 'artist' AND name = ? COLLATE NOCASE
""", (n['name'],))
for cr in cursor.fetchall():
if not n['image_url'] and cr['image_url'] and cr['image_url'].startswith('http'):
n['image_url'] = cr['image_url']
if not n['genres'] and cr['genres']:
try:
n['genres'] = json.loads(cr['genres'])[:5] if isinstance(cr['genres'], str) else []
except Exception as e:
logger.debug("explorer node genres parse failed: %s", e)
# Harvest missing IDs from cache
src_map = {'spotify': 'spotify_id', 'itunes': 'itunes_id', 'deezer': 'deezer_id', 'discogs': 'discogs_id', 'musicbrainz': 'musicbrainz_id'}
k = src_map.get(cr['source'])
if k and not n.get(k):
n[k] = cr['entity_id']
# Deezer image fallback — construct URL directly from ID
if not n['image_url'] and n.get('deezer_id'):
n['image_url'] = f"https://api.deezer.com/artist/{n['deezer_id']}/image?size=big"
# Spotify image fallback — try API if authenticated
if not n['image_url'] and n.get('spotify_id'):
try:
if spotify_client and spotify_client.is_spotify_authenticated():
from core.api_call_tracker import api_call_tracker
api_call_tracker.record_call('spotify', endpoint='artist')
artist_data = spotify_client.sp.artist(n['spotify_id'])
if artist_data and artist_data.get('images'):
n['image_url'] = artist_data['images'][0]['url']
if not n['genres'] and artist_data.get('genres'):
n['genres'] = artist_data['genres'][:5]
except Exception as e:
logger.debug("spotify artist image fallback failed: %s", e)
# Album art fallback (iTunes artists have no artist images)
if not n['image_url']:
cursor.execute("""
SELECT image_url FROM metadata_cache_entities
WHERE entity_type = 'album' AND image_url LIKE 'http%'
AND artist_name = ? COLLATE NOCASE LIMIT 1
""", (n['name'],))
alb = cursor.fetchone()
if alb:
n['image_url'] = alb['image_url']
logger.info(f"[Artist Explorer] Center: {center_name}, Ring 1: {sum(1 for n in nodes if n.get('ring')==1)}, Ring 2: {sum(1 for n in nodes if n.get('ring')==2)}, Edges: {len(edges)}")
return jsonify({
'success': True,
'nodes': nodes,
'edges': edges,
'center': center_name,
})
except Exception as e:
logger.error(f"Error getting artist explorer data: {e}")
import traceback
traceback.print_exc()
return jsonify({"success": False, "error": str(e)}), 500

496
core/artists/quality.py
View file

@ -1,496 +0,0 @@
"""Artist quality enhancement helper.
`enhance_artist_quality(artist_id, track_ids, deps)` is the route-handler
body for the `/api/library/artist/<artist_id>/enhance` endpoint. It walks
the user's selected tracks, finds the best metadata match against the
configured primary source, and queues high-quality re-downloads on the
wishlist with `source_type='enhance'`.
Per-track flow:
1. Resolve the existing track via the artist's full detail map (built up
front from `database.get_artist_full_detail`).
2. Read current quality tier from the file extension.
3. Build `matched_track_data` for the wishlist entry, in priority order:
- **Direct lookup using stored source IDs** — for every source the
user has configured, if the library track has the corresponding
stored ID (`spotify_track_id` / `deezer_id` / `itunes_track_id` /
`soul_id`), call `client.get_track_details(stored_id)` and convert
the result to the wishlist payload. First success wins; the user's
configured primary source is tried first. Mirrors what Download
Discography does — stable IDs straight to the source's API, no
fuzzy text matching.
- **Multi-source parallel text search fallback** — if no stored ID
resolved, run the shared `core.metadata.multi_source_search`
against every configured source in parallel and pick the best
cross-source match (auto-accept threshold 0.7).
4. Validate the match has non-empty title, album, and artists. Reject
matches with empty fields — those propagated as
"unknown artist - unknown album - unknown track" wishlist entries
pre-fix because the wishlist payload normalizer's truthy-check
passthrough accepted dicts with empty string fields.
5. Add to wishlist via `wishlist_service.add_spotify_track_to_wishlist`
with `source_type='enhance'` and a `source_context` carrying the
original file path, format tier, bitrate, and artist name.
6. Tally `enhanced_count` / `failed_count` / per-track failure reasons.
The flow originally had Spotify-only logic with an iTunes search-only
fallback. Two failure modes drove the rewrite:
- Users with neither Spotify nor Deezer connected got silent failures
("unknown artist - unknown album - unknown track" wishlist entries)
because iTunes's text search returned junk matches with empty fields
that cleared the 0.7 confidence threshold.
- Library tracks with messy tags ("Title (Live)", featured artists in
the artist field, etc.) failed fuzzy text search even when a perfect
stored ID was available — Download Discography had no such problem
because it resolves albums by stable ID.
Direct-lookup-via-stored-ID matches the Download Discography contract
for every source where we have an ID column. Text search is only the
fallback now.
Returns `(payload_dict, http_status_code)` so the route wrapper can
`jsonify()` and return.
"""
from __future__ import annotations
import os
from dataclasses import dataclass
from typing import Any, Callable, Optional
from utils.logging_config import get_logger
logger = get_logger('artists.quality')
@dataclass
class ArtistQualityDeps:
"""Bundle of cross-cutting deps the artist quality enhancement needs."""
matching_engine: Any
get_database: Callable[[], Any]
get_wishlist_service: Callable[[], Any]
get_current_profile_id: Callable[[], int]
get_quality_tier_from_extension: Callable
# Returns ``[(source_name, client), ...]`` for every metadata source
# the user has configured. Powers both the direct-lookup fast path
# (resolves stored source IDs straight from each source's API,
# like Download Discography) and the multi-source parallel text
# search fallback (shared with Track Redownload via
# ``core.metadata.multi_source_search``).
get_metadata_search_sources: Callable[[], list]
def _has_complete_metadata(payload: Optional[dict]) -> bool:
"""Reject matches with empty / missing core fields. Pre-fix, iTunes
returned matches that cleared the 0.7 confidence threshold while
having empty artist / album / title — those propagated as junk
wishlist entries displayed as 'unknown artist - unknown album -
unknown track'."""
if not payload:
return False
if not (payload.get('name') or '').strip():
return False
artists = payload.get('artists') or []
has_artist = any(
(a.get('name') or '').strip() if isinstance(a, dict) else (a or '').strip()
for a in artists
)
if not has_artist:
return False
album = payload.get('album') or {}
if isinstance(album, dict):
if not (album.get('name') or '').strip():
return False
elif not (album or '').strip():
return False
return True
def _build_payload_from_track(track_obj) -> dict:
"""Build a Spotify-shaped wishlist payload from any metadata source's
Track-shaped object (Spotify Track, iTunes Track, Deezer Track,
Discogs Track — they all have the same .id / .name / .artists /
.album / .duration_ms / etc shape because each client mimics
Spotify's surface).
The wishlist's downstream pipeline expects Spotify shape; this helper
is the single place that knows how to produce it. Replaces the
duplicated payload construction that used to live in the Spotify
search path AND the iTunes fallback path.
Does NOT substitute defaults for missing artists / album / title —
``_has_complete_metadata`` rejects empty matches downstream so the
user sees a clear failure instead of a junk wishlist entry with
fabricated values.
"""
image_url = getattr(track_obj, 'image_url', '') or ''
album_images = (
[{'url': image_url, 'height': 600, 'width': 600}]
if image_url else []
)
artist_names = list(getattr(track_obj, 'artists', None) or [])
return {
'id': getattr(track_obj, 'id', ''),
'name': getattr(track_obj, 'name', '') or '',
'artists': [{'name': a} for a in artist_names],
'album': {
'name': getattr(track_obj, 'album', '') or '',
'artists': [{'name': a} for a in artist_names],
'album_type': getattr(track_obj, 'album_type', None) or 'album',
'images': album_images,
'release_date': getattr(track_obj, 'release_date', '') or '',
'total_tracks': 1,
},
'duration_ms': getattr(track_obj, 'duration_ms', 0) or 0,
'track_number': getattr(track_obj, 'track_number', None) or 1,
'disc_number': getattr(track_obj, 'disc_number', None) or 1,
'popularity': getattr(track_obj, 'popularity', None) or 0,
'preview_url': getattr(track_obj, 'preview_url', None),
'external_urls': getattr(track_obj, 'external_urls', None) or {},
}
# Map metadata source name → DB column on the ``tracks`` table that
# stores that source's native track ID. Used to drive the direct-lookup
# fast path: when a library track has a stored ID for source X and the
# user has source X configured, skip fuzzy text search and resolve
# straight from X's API. Mirrors what Download Discography does — stable
# IDs all the way, no fuzzy text matching.
#
# Discogs is release-based and has no per-track ID column; not listed
# here, so direct lookup never tries Discogs (search-fallback still
# runs for Discogs as one of the parallel sources).
_STORED_ID_COLUMNS = {
'spotify': 'spotify_track_id',
'deezer': 'deezer_id',
'itunes': 'itunes_track_id',
'hydrabase': 'soul_id',
}
def _enhanced_to_wishlist_payload(enhanced: dict,
fallback_title: str,
fallback_artist: str,
fallback_album: str) -> Optional[dict]:
"""Convert a ``get_track_details`` enhanced-shape dict to the
Spotify-shape wishlist payload.
Every metadata source's ``get_track_details`` returns the same
"enhanced" intermediate shape (top-level ``id``, ``name``,
``artists`` as a list of strings, ``album.artists`` as strings),
documented and pinned across spotify_client / itunes_client /
deezer_client / hydrabase_client. The wishlist downstream expects
Spotify's native shape (``artists`` as ``[{'name': ...}]``), so
this helper does the conversion in one place.
Spotify's ``raw_data`` field is already in wishlist shape (the
raw Spotify API response), so we return it as-is when detected,
preserving full ``album.images`` and ``external_urls`` that the
enhanced top-level fields drop. Other sources' ``raw_data`` is
in source-native shape and gets ignored.
"""
if not enhanced:
return None
raw = enhanced.get('raw_data')
if isinstance(raw, dict):
raw_artists = raw.get('artists')
if (isinstance(raw_artists, list) and raw_artists
and isinstance(raw_artists[0], dict)):
return raw
artists = enhanced.get('artists') or [fallback_artist]
album_data = enhanced.get('album') or {}
album_artists = album_data.get('artists') or artists
def _to_dict_artists(seq):
return [a if isinstance(a, dict) else {'name': a} for a in seq]
image_url = enhanced.get('image_url') or ''
album_images_field = album_data.get('images')
if isinstance(album_images_field, list) and album_images_field:
album_images = album_images_field
elif image_url:
album_images = [{'url': image_url, 'height': 600, 'width': 600}]
else:
album_images = []
return {
'id': str(enhanced.get('id', '')),
'name': enhanced.get('name') or fallback_title,
'artists': _to_dict_artists(artists),
'album': {
'id': str(album_data.get('id', '')),
'name': album_data.get('name') or fallback_album,
'album_type': album_data.get('album_type', 'album'),
'release_date': album_data.get('release_date', ''),
'total_tracks': album_data.get('total_tracks', 1),
'artists': _to_dict_artists(album_artists),
'images': album_images,
},
'duration_ms': enhanced.get('duration_ms', 0),
'track_number': enhanced.get('track_number', 1),
'disc_number': enhanced.get('disc_number', 1),
'popularity': enhanced.get('popularity', 0),
'preview_url': enhanced.get('preview_url'),
'external_urls': enhanced.get('external_urls', {}),
}
def _try_direct_lookup_all_sources(track: dict,
sources: list,
preferred_source: Optional[str],
title: str,
artist_name: str,
album_title: str
) -> tuple:
"""Try direct ID-based lookup on every source where the library
track has a stored ID. Returns ``(payload, source_name)`` on first
success, or ``(None, None)`` if no source has a stored ID with a
successful lookup.
Mirrors what Download Discography does — stable IDs straight to the
source's API, no fuzzy text matching. Avoids the failure mode where
library text tags don't match the source's canonical title (the
Discord report case: track tagged "Title (Live)" and source has
"Title" → fuzzy search misses, but stored ID resolves directly).
Preferred source attempted first when present in ``sources``,
typically the user's configured primary metadata source — so a
Deezer-primary user gets Deezer art / album shape on the wishlist
entry instead of whichever source happened to have a stored ID
first in iteration order.
"""
def _priority(entry):
name = entry[0]
return 0 if name == preferred_source else 1
ordered = sorted(sources, key=_priority)
for source_name, client in ordered:
column = _STORED_ID_COLUMNS.get(source_name)
if not column:
continue
stored_id = track.get(column)
if not stored_id:
continue
if not hasattr(client, 'get_track_details'):
continue
try:
enhanced = client.get_track_details(str(stored_id))
except Exception as exc:
logger.error(
f"[Enhance] {source_name} direct lookup failed for "
f"ID {stored_id}: {exc}"
)
continue
if not enhanced:
continue
payload = _enhanced_to_wishlist_payload(
enhanced, title, artist_name, album_title,
)
if _has_complete_metadata(payload):
logger.info(
f"[Enhance] Direct lookup matched: {source_name} "
f"ID {stored_id} → '{payload.get('name')}'"
)
return payload, source_name
return None, None
# Minimum match-score threshold for accepting a search-fallback match
# without user confirmation. Mirrors the legacy threshold the enhance
# flow has always used.
_AUTO_ACCEPT_SCORE_THRESHOLD = 0.7
def enhance_artist_quality(artist_id, track_ids, deps: ArtistQualityDeps):
"""Add selected tracks to wishlist for quality enhancement re-download.
Per-track flow:
1. **Direct lookup using stored source IDs** (mirrors what Download
Discography does — stable IDs straight to the source's API, no
fuzzy text matching). For each source the user has configured,
if the library track has the corresponding stored ID
(``spotify_track_id`` / ``deezer_id`` / ``itunes_track_id`` /
``soul_id``), call ``client.get_track_details(stored_id)`` and
convert to wishlist payload. First success wins; preferred
source (user's configured primary) tried first.
2. **Multi-source parallel text search fallback** (via the shared
``core.metadata.multi_source_search`` module — same code path
Track Redownload uses) for tracks with no stored IDs / lookup
misses.
3. **Validation**: reject matches with empty title / album / artists
so the user sees a clear failure instead of an "unknown artist"
wishlist entry.
Pre-refactor: only Spotify had a direct-lookup fast path; everything
else went through fuzzy text search. Discogs / Hydrabase / Deezer-
primary users got far worse coverage than Download Discography
despite both flows asking the same question.
"""
from core.metadata.multi_source_search import TrackQuery, search_all_sources
from core.metadata.registry import get_primary_source
try:
if not track_ids:
return {"success": False, "error": "No track IDs provided"}, 400
database = deps.get_database()
wishlist_service = deps.get_wishlist_service()
profile_id = deps.get_current_profile_id()
# Get artist info
artist_result = database.get_artist_full_detail(artist_id)
if not artist_result.get('success'):
return {"success": False, "error": "Artist not found"}, 404
artist_name = artist_result.get('artist', {}).get('name', 'Unknown Artist')
# Build lookup of all tracks for this artist
track_lookup = {}
for album in artist_result.get('albums', []):
album_title = album.get('title', '')
for track in album.get('tracks', []):
tid = str(track.get('id', ''))
track['_album_title'] = album_title
track['_album_id'] = album.get('id')
track_lookup[tid] = track
# Resolve every configured metadata source up front.
search_sources = deps.get_metadata_search_sources()
# User's configured primary source — direct-lookup tries this
# first so Deezer-primary users get Deezer payloads on the
# wishlist entry (correct cover art / album shape) even when
# other sources also have stored IDs for the same track.
try:
preferred_source = get_primary_source()
except Exception:
preferred_source = None
enhanced_count = 0
failed_count = 0
failed_tracks = []
for track_id in track_ids:
track_id_str = str(track_id)
track = track_lookup.get(track_id_str)
if not track:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'reason': 'Track not found'})
continue
file_path = track.get('file_path')
if not file_path:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'reason': 'No file path'})
continue
tier_name, tier_num = deps.get_quality_tier_from_extension(file_path)
title = track.get('title', '') or ''
if not title.strip():
title = os.path.splitext(os.path.basename(file_path))[0]
album_title = track.get('_album_title', '')
matched_track_data = None
chosen_source = None
# 1. Direct lookup via every stored source ID — like Download
# Discography. Stable IDs, no fuzzy text matching.
if search_sources:
matched_track_data, chosen_source = _try_direct_lookup_all_sources(
track, search_sources, preferred_source,
title, artist_name, album_title,
)
# 2. Multi-source parallel text search fallback — for tracks
# with no stored IDs / lookup misses.
if not matched_track_data and search_sources:
try:
track_query = TrackQuery(
title=title,
artist=artist_name,
album=album_title,
duration_ms=track.get('duration', 0) or 0,
spotify_track_id=track.get('spotify_track_id'),
deezer_id=track.get('deezer_id'),
)
multi_result = search_all_sources(track_query, search_sources)
if multi_result.best_match and multi_result.best_match['score'] >= _AUTO_ACCEPT_SCORE_THRESHOLD:
chosen_source = multi_result.best_match['source']
best_track_obj = multi_result.best_track()
if best_track_obj:
matched_track_data = _build_payload_from_track(best_track_obj)
except Exception as exc:
logger.error(f"[Enhance] Multi-source search failed for {title}: {exc}")
# 3. Reject matches with empty / missing core fields.
if not _has_complete_metadata(matched_track_data):
if matched_track_data:
logger.warning(
f"[Enhance] {chosen_source} match for '{title}' rejected — "
f"empty title / album / artists (would render as 'unknown')"
)
matched_track_data = None
if not matched_track_data:
failed_count += 1
source_list = ', '.join(name for name, _ in (search_sources or []))
if not source_list:
reason = (
'No metadata source configured — connect Spotify / '
'iTunes / Deezer / Discogs / Hydrabase to enable enhance'
)
else:
reason = (
f'No usable match across {source_list} — '
f'try connecting an additional metadata source'
)
failed_tracks.append({
'track_id': track_id,
'title': title,
'reason': reason,
})
continue
# Add to wishlist with enhance source
source_context = {
'enhance': True,
'original_file_path': file_path,
'original_format': tier_name,
'original_bitrate': track.get('bitrate'),
'original_tier': tier_num,
'artist_name': artist_name,
}
success = wishlist_service.add_spotify_track_to_wishlist(
spotify_track_data=matched_track_data,
failure_reason=f"Quality enhance - upgrading from {tier_name.replace('_', ' ').title()}",
source_type='enhance',
source_context=source_context,
profile_id=profile_id
)
if success:
enhanced_count += 1
logger.info(f"[Enhance] Queued for upgrade: {artist_name} - {title} ({tier_name})")
else:
failed_count += 1
failed_tracks.append({'track_id': track_id, 'title': title, 'reason': 'Wishlist add failed'})
return {
'success': True,
'enhanced_count': enhanced_count,
'failed_count': failed_count,
'failed_tracks': failed_tracks
}, 200
except Exception as e:
logger.error(f"[Enhance] {e}")
import traceback
traceback.print_exc()
return {"success": False, "error": str(e)}, 500

214
core/audiodb_client.py
View file

@ -1,214 +0,0 @@
import requests
import time
import threading
from typing import Dict, Optional, Any
from functools import wraps
from utils.logging_config import get_logger
logger = get_logger("audiodb_client")
# Global rate limiting variables
_last_api_call_time = 0
_api_call_lock = threading.Lock()
MIN_API_INTERVAL = 2.0 # 2 seconds between API calls (30 req/min free tier)
def rate_limited(func):
"""Decorator to enforce rate limiting on AudioDB API calls"""
@wraps(func)
def wrapper(*args, **kwargs):
global _last_api_call_time
with _api_call_lock:
current_time = time.time()
time_since_last_call = current_time - _last_api_call_time
if time_since_last_call < MIN_API_INTERVAL:
sleep_time = MIN_API_INTERVAL - time_since_last_call
time.sleep(sleep_time)
_last_api_call_time = time.time()
from core.api_call_tracker import api_call_tracker
api_call_tracker.record_call('audiodb')
try:
result = func(*args, **kwargs)
return result
except Exception as e:
if "rate limit" in str(e).lower() or "429" in str(e):
logger.warning(f"AudioDB rate limit hit, implementing backoff: {e}")
time.sleep(4.0)
raise e
return wrapper
class AudioDBClient:
"""Client for interacting with TheAudioDB API"""
BASE_URL = "https://www.theaudiodb.com/api/v1/json/2"
def __init__(self):
self.session = requests.Session()
self.session.headers.update({
'User-Agent': 'SoulSync/1.0',
'Accept': 'application/json'
})
logger.info("AudioDB client initialized")
@rate_limited
def search_artist(self, artist_name: str) -> Optional[Dict[str, Any]]:
"""
Search for an artist by name.
Args:
artist_name: Name of the artist to search for
Returns:
Artist dict from AudioDB or None if not found
"""
try:
response = self.session.get(
f"{self.BASE_URL}/search.php",
params={'s': artist_name},
timeout=10
)
response.raise_for_status()
data = response.json()
artists = data.get('artists')
if artists and len(artists) > 0:
logger.debug(f"Found artist for query: {artist_name}")
return artists[0]
logger.debug(f"No artist found for query: {artist_name}")
return None
except Exception as e:
logger.error(f"Error searching for artist '{artist_name}': {e}")
return None
@rate_limited
def search_album(self, artist_name: str, album_title: str) -> Optional[Dict[str, Any]]:
"""
Search for an album by artist name and album title.
Args:
artist_name: Name of the artist
album_title: Title of the album
Returns:
Album dict from AudioDB or None if not found
"""
try:
response = self.session.get(
f"{self.BASE_URL}/searchalbum.php",
params={'s': artist_name, 'a': album_title},
timeout=10
)
response.raise_for_status()
data = response.json()
albums = data.get('album')
if albums and len(albums) > 0:
logger.debug(f"Found album for query: {artist_name} - {album_title}")
return albums[0]
logger.debug(f"No album found for query: {artist_name} - {album_title}")
return None
except Exception as e:
logger.error(f"Error searching for album '{artist_name} - {album_title}': {e}")
return None
@rate_limited
def search_track(self, artist_name: str, track_title: str) -> Optional[Dict[str, Any]]:
"""
Search for a track by artist name and track title.
Args:
artist_name: Name of the artist
track_title: Title of the track
Returns:
Track dict from AudioDB or None if not found
"""
try:
response = self.session.get(
f"{self.BASE_URL}/searchtrack.php",
params={'s': artist_name, 't': track_title},
timeout=10
)
response.raise_for_status()
data = response.json()
tracks = data.get('track')
if tracks and len(tracks) > 0:
logger.debug(f"Found track for query: {artist_name} - {track_title}")
return tracks[0]
logger.debug(f"No track found for query: {artist_name} - {track_title}")
return None
except Exception as e:
logger.error(f"Error searching for track '{artist_name} - {track_title}': {e}")
return None
@rate_limited
def lookup_artist_by_id(self, artist_id: str) -> Optional[Dict[str, Any]]:
"""Lookup an artist directly by AudioDB ID."""
try:
response = self.session.get(
f"{self.BASE_URL}/artist.php",
params={'i': artist_id},
timeout=10
)
response.raise_for_status()
data = response.json()
artists = data.get('artists')
if artists and len(artists) > 0:
return artists[0]
return None
except Exception as e:
logger.error(f"Error looking up artist by ID {artist_id}: {e}")
return None
@rate_limited
def lookup_album_by_id(self, album_id: str) -> Optional[Dict[str, Any]]:
"""Lookup an album directly by AudioDB ID."""
try:
response = self.session.get(
f"{self.BASE_URL}/album.php",
params={'m': album_id},
timeout=10
)
response.raise_for_status()
data = response.json()
albums = data.get('album')
if albums and len(albums) > 0:
return albums[0]
return None
except Exception as e:
logger.error(f"Error looking up album by ID {album_id}: {e}")
return None
@rate_limited
def lookup_track_by_id(self, track_id: str) -> Optional[Dict[str, Any]]:
"""Lookup a track directly by AudioDB ID."""
try:
response = self.session.get(
f"{self.BASE_URL}/track.php",
params={'m': track_id},
timeout=10
)
response.raise_for_status()
data = response.json()
tracks = data.get('track')
if tracks and len(tracks) > 0:
return tracks[0]
return None
except Exception as e:
logger.error(f"Error looking up track by ID {track_id}: {e}")
return None

747
core/audiodb_worker.py
View file

@ -1,747 +0,0 @@
import json
import re
import threading
import time
from difflib import SequenceMatcher
from typing import Optional, Dict, Any
from datetime import datetime, timedelta
from utils.logging_config import get_logger
from database.music_database import MusicDatabase
from core.audiodb_client import AudioDBClient
from core.worker_utils import accept_artist_match, interruptible_sleep
logger = get_logger("audiodb_worker")
class AudioDBWorker:
"""Background worker for enriching library artists, albums, and tracks with AudioDB metadata"""
def __init__(self, database: MusicDatabase):
self.db = database
self.client = AudioDBClient()
# Worker state
self.running = False
self.paused = False
self.should_stop = False
self.thread = None
self._stop_event = threading.Event()
# Current item being processed (for UI tooltip)
self.current_item = None
# Statistics
self.stats = {
'matched': 0,
'not_found': 0,
'pending': 0,
'errors': 0
}
# Retry configuration
self.retry_days = 30
# Name matching threshold
self.name_similarity_threshold = 0.80
logger.info("AudioDB background worker initialized")
def start(self):
"""Start the background worker"""
if self.running:
logger.warning("Worker already running")
return
self.running = True
self.should_stop = False
self._stop_event.clear()
self.thread = threading.Thread(target=self._run, daemon=True)
self.thread.start()
logger.info("AudioDB background worker started")
def stop(self):
"""Stop the background worker"""
if not self.running:
return
logger.info("Stopping AudioDB worker...")
self.should_stop = True
self.running = False
self._stop_event.set()
if self.thread:
self.thread.join(timeout=1)
logger.info("AudioDB worker stopped")
def pause(self):
"""Pause the worker"""
if not self.running:
logger.warning("Worker not running, cannot pause")
return
self.paused = True
logger.info("AudioDB worker paused")
def resume(self):
"""Resume the worker"""
if not self.running:
logger.warning("Worker not running, start it first")
return
self.paused = False
logger.info("AudioDB worker resumed")
def get_stats(self) -> Dict[str, Any]:
"""Get current statistics"""
self.stats['pending'] = self._count_pending_items()
progress = self._get_progress_breakdown()
is_actually_running = self.running and (self.thread is not None and self.thread.is_alive())
is_idle = is_actually_running and not self.paused and self.stats['pending'] == 0 and self.current_item is None
return {
'enabled': True,
'running': is_actually_running and not self.paused,
'paused': self.paused,
'idle': is_idle,
'current_item': self.current_item,
'stats': self.stats.copy(),
'progress': progress
}
def _run(self):
"""Main worker loop"""
logger.info("AudioDB worker thread started")
while not self.should_stop:
try:
if self.paused:
interruptible_sleep(self._stop_event, 1)
continue
self.current_item = None
item = self._get_next_item()
if not item:
logger.debug("No pending items, sleeping...")
interruptible_sleep(self._stop_event, 10)
continue
self.current_item = item
# Guard: skip items with None/NULL IDs to prevent infinite enrichment loops
item_id = item.get('id') or item.get('artist_id') or item.get('album_id')
if item_id is None:
logger.warning(f"Skipping {item.get('type', 'unknown')} with NULL id: {item.get('name', '?')} — marking as error")
try:
itype = item.get('type', '')
table = 'artists' if 'artist' in itype else ('albums' if 'album' in itype else 'tracks')
# Can't mark status without an ID — just skip
except Exception as e:
logger.debug("null id table resolve failed: %s", e)
continue
self._process_item(item)
interruptible_sleep(self._stop_event, 2)
except Exception as e:
logger.error(f"Error in worker loop: {e}")
interruptible_sleep(self._stop_event, 5)
logger.info("AudioDB worker thread finished")
def _get_next_item(self) -> Optional[Dict[str, Any]]:
"""Get next item to process from priority queue (artists → albums → tracks)"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
# Pinned-group override (Manage Enrichment Workers): process one
# entity type first, then fall through to the normal chain. Unset or
# exhausted ⇒ default artist→album→track order, unchanged.
from core.worker_utils import read_enrichment_priority, priority_pending_item
_prio = read_enrichment_priority('audiodb')
if _prio:
_pi = priority_pending_item(cursor, 'audiodb', _prio)
if _pi:
return _pi
# Priority 1: Unattempted artists
cursor.execute("""
SELECT id, name
FROM artists
WHERE audiodb_match_status IS NULL AND id IS NOT NULL
ORDER BY id ASC
LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 2: Unattempted albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.audiodb_match_status IS NULL AND a.id IS NOT NULL
ORDER BY a.id ASC
LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
# Priority 3: Unattempted tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.audiodb_match_status IS NULL AND t.id IS NOT NULL
ORDER BY t.id ASC
LIMIT 1
""")
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
# Priority 4: Retry 'not_found' OR 'error' artists after retry_days.
# 'error' status covers transient AudioDB outages (timeouts, 500s)
# that the issue-#553 fix marks rather than leaving NULL — without
# this retry path those rows would stay errored forever.
retry_cutoff = datetime.now() - timedelta(days=self.retry_days)
cursor.execute("""
SELECT id, name
FROM artists
WHERE audiodb_match_status IN ('not_found', 'error') AND audiodb_last_attempted < ?
ORDER BY audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
row = cursor.fetchone()
if row:
logger.info(f"Retrying artist '{row[1]}' (last attempted before cutoff)")
return {'type': 'artist', 'id': row[0], 'name': row[1]}
# Priority 5: Retry 'not_found' OR 'error' albums
cursor.execute("""
SELECT a.id, a.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM albums a
JOIN artists ar ON a.artist_id = ar.id
WHERE a.audiodb_match_status IN ('not_found', 'error') AND a.audiodb_last_attempted < ?
ORDER BY a.audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'album', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
# Priority 6: Retry 'not_found' OR 'error' tracks
cursor.execute("""
SELECT t.id, t.title, ar.name AS artist_name, ar.audiodb_id AS artist_audiodb_id
FROM tracks t
JOIN artists ar ON t.artist_id = ar.id
WHERE t.audiodb_match_status IN ('not_found', 'error') AND t.audiodb_last_attempted < ?
ORDER BY t.audiodb_last_attempted ASC
LIMIT 1
""", (retry_cutoff,))
row = cursor.fetchone()
if row:
return {'type': 'track', 'id': row[0], 'name': row[1], 'artist': row[2], 'artist_audiodb_id': row[3]}
return None
except Exception as e:
logger.error(f"Error getting next item: {e}")
return None
finally:
if conn:
conn.close()
def _normalize_name(self, name: str) -> str:
"""Normalize artist name for comparison"""
name = name.lower().strip()
name = re.sub(r'\s+[-–—]\s+.*$', '', name)
name = re.sub(r'\s*\(.*?\)\s*', ' ', name)
name = re.sub(r'[^\w\s]', '', name)
name = re.sub(r'\s+', ' ', name).strip()
return name
def _verify_artist_id(self, item: Dict[str, Any], result: Dict[str, Any]) -> bool:
"""Verify that the result's artist ID matches the parent artist's stored AudioDB ID.
If mismatched, the album/track search is more specific (uses artist+title),
so we trust it and correct the parent artist's audiodb_id — BUT only when
the result's artist *name* matches our parent artist. Without that guard,
a collaboration/compilation (a track our library credits to one artist
that lives on another artist's album) would stamp the wrong AudioDB id
onto our artist. See the Deezer fix for the full write-up."""
parent_audiodb_id = item.get('artist_audiodb_id')
if not parent_audiodb_id:
return True
result_artist_id = result.get('idArtist')
if not result_artist_id:
return True
if str(result_artist_id) != str(parent_audiodb_id):
parent_name = item.get('artist') or ''
result_artist_name = result.get('strArtist') or ''
if (result_artist_name and parent_name
and not self._name_matches(parent_name, result_artist_name)):
logger.info(
f"Skipping artist-ID correction from {item['type']} "
f"'{item['name']}': result artist '{result_artist_name}' "
f"≠ parent '{parent_name}' (collab/compilation, not a "
f"correction)"
)
return True
logger.info(
f"Artist ID correction from {item['type']} '{item['name']}': "
f"updating parent artist AudioDB ID from {parent_audiodb_id} to {result_artist_id}"
)
self._correct_artist_audiodb_id(item, str(result_artist_id))
return True
def _correct_artist_audiodb_id(self, item: Dict[str, Any], correct_audiodb_id: str):
"""Correct the parent artist's audiodb_id based on a more specific album/track match"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
# Find the artist_id from the album/track
table = 'albums' if item['type'] == 'album' else 'tracks'
cursor.execute(f"SELECT artist_id FROM {table} WHERE id = ?", (item['id'],))
row = cursor.fetchone()
if not row:
return
artist_id = row[0]
cursor.execute("""
UPDATE artists SET
audiodb_id = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (correct_audiodb_id, artist_id))
conn.commit()
logger.info(f"Corrected artist #{artist_id} AudioDB ID to {correct_audiodb_id}")
except Exception as e:
logger.error(f"Error correcting artist AudioDB ID: {e}")
finally:
if conn:
conn.close()
def _name_matches(self, query_name: str, result_name: str) -> bool:
"""Check if AudioDB result name matches our query with fuzzy matching"""
norm_query = self._normalize_name(query_name)
norm_result = self._normalize_name(result_name)
similarity = SequenceMatcher(None, norm_query, norm_result).ratio()
logger.debug(f"Name similarity: '{query_name}' vs '{result_name}' = {similarity:.2f}")
return similarity >= self.name_similarity_threshold
def _get_existing_id(self, entity_type: str, entity_id: int) -> Optional[str]:
"""Check if an entity already has an audiodb_id (e.g. from manual match)."""
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
return None
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"SELECT audiodb_id FROM {table} WHERE id = ?", (entity_id,))
row = cursor.fetchone()
return row[0] if row and row[0] else None
except Exception:
return None
finally:
if conn:
conn.close()
def _process_item(self, item: Dict[str, Any]):
"""Process a single item (artist, album, or track).
If the entity already has an audiodb_id (e.g. from manual match),
uses it for direct lookup instead of searching by name."""
try:
item_type = item['type']
item_id = item['id']
item_name = item['name']
logger.debug(f"Processing {item_type} #{item_id}: {item_name}")
# Check for existing ID (manual match) — use direct lookup instead of name search
existing_id = self._get_existing_id(item_type, item_id)
if existing_id:
lookup_methods = {
'artist': self.client.lookup_artist_by_id,
'album': self.client.lookup_album_by_id,
'track': self.client.lookup_track_by_id,
}
update_methods = {
'artist': lambda r: self._update_artist(item_id, r),
'album': lambda r: (self._verify_artist_id(item, r), self._update_album(item_id, r)),
'track': lambda r: (self._verify_artist_id(item, r), self._update_track(item_id, r)),
}
lookup = lookup_methods.get(item_type)
update = update_methods.get(item_type)
if lookup and update:
try:
result = lookup(existing_id)
if result:
update(result)
self.stats['matched'] += 1
logger.info(f"Enriched {item_type} '{item_name}' from existing AudioDB ID: {existing_id}")
return
except Exception as e:
logger.warning(f"Direct lookup failed for existing AudioDB ID {existing_id}: {e}")
# Direct lookup returned no metadata (None) or raised — don't
# fall through to the name-search path below, which could
# overwrite a manually-matched audiodb_id with a wrong guess.
# Mark status='error' so the queue's NULL-status filter stops
# re-picking this row on every tick (issue #553: AudioDB
# `track.php` timeouts caused infinite enrichment loops as
# the row was repeatedly picked + re-attempted because it
# never left the NULL state). The error-retry priority block
# in `_get_next_item` re-attempts after `retry_days` so
# transient AudioDB outages still recover automatically.
self._mark_status(item_type, item_id, 'error')
self.stats['errors'] += 1
logger.debug(
f"Preserving manual match for {item_type} '{item_name}' "
f"(AudioDB ID: {existing_id}); marked error pending retry"
)
return
if item_type == 'artist':
result = self.client.search_artist(item_name)
if result:
result_name = result.get('strArtist', '')
ok, reason = accept_artist_match(
self.db, 'audiodb_id', result.get('idArtist'), item_id,
item_name, result_name,
)
if ok:
self._update_artist(item_id, result)
self.stats['matched'] += 1
logger.info(f"Matched artist '{item_name}' -> AudioDB ID: {result.get('idArtist')}")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Artist '{item_name}' not matched: {reason}")
else:
self._mark_status('artist', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for artist '{item_name}'")
elif item_type == 'album':
artist_name = item.get('artist', '')
result = self.client.search_album(artist_name, item_name)
if result:
result_name = result.get('strAlbum', '')
if self._name_matches(item_name, result_name):
self._verify_artist_id(item, result)
self._update_album(item_id, result)
self.stats['matched'] += 1
logger.info(f"Matched album '{item_name}' -> AudioDB ID: {result.get('idAlbum')}")
else:
self._mark_status('album', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for album '{item_name}' (got '{result_name}')")
else:
self._mark_status('album', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for album '{item_name}'")
elif item_type == 'track':
artist_name = item.get('artist', '')
result = self.client.search_track(artist_name, item_name)
if result:
result_name = result.get('strTrack', '')
if self._name_matches(item_name, result_name):
self._verify_artist_id(item, result)
self._update_track(item_id, result)
self.stats['matched'] += 1
logger.info(f"Matched track '{item_name}' -> AudioDB ID: {result.get('idTrack')}")
else:
self._mark_status('track', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"Name mismatch for track '{item_name}' (got '{result_name}')")
else:
self._mark_status('track', item_id, 'not_found')
self.stats['not_found'] += 1
logger.debug(f"No match for track '{item_name}'")
except Exception as e:
logger.error(f"Error processing {item['type']} #{item['id']}: {e}")
self.stats['errors'] += 1
try:
self._mark_status(item['type'], item['id'], 'error')
except Exception as e2:
logger.error(f"Error updating item status: {e2}")
def _update_artist(self, artist_id: int, data: Dict[str, Any]):
"""Store AudioDB metadata for an artist using generic column names"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
# Update AudioDB tracking + generic metadata columns
cursor.execute("""
UPDATE artists SET
audiodb_id = ?,
audiodb_match_status = 'matched',
audiodb_last_attempted = CURRENT_TIMESTAMP,
style = ?,
mood = ?,
label = ?,
banner_url = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (
data.get('idArtist'),
data.get('strStyle'),
data.get('strMood'),
data.get('strLabel'),
data.get('strArtistBanner'),
artist_id
))
# Backfill thumb_url if artist has no image
thumb_url = data.get('strArtistThumb')
if thumb_url:
cursor.execute("""
UPDATE artists SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (thumb_url, artist_id))
# Backfill genres if artist has none
genre = data.get('strGenre')
if genre:
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
_filtered = filter_genres([genre], _cfg)
if _filtered:
cursor.execute("""
UPDATE artists SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(_filtered), artist_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating artist #{artist_id} with AudioDB data: {e}")
raise
finally:
if conn:
conn.close()
def _update_album(self, album_id: int, data: Dict[str, Any]):
"""Store AudioDB metadata for an album using generic column names"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE albums SET
audiodb_id = ?,
audiodb_match_status = 'matched',
audiodb_last_attempted = CURRENT_TIMESTAMP,
style = ?,
mood = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (
data.get('idAlbum'),
data.get('strStyle'),
data.get('strMood'),
album_id
))
# Backfill thumb_url if album has no image
thumb_url = data.get('strAlbumThumb')
if thumb_url:
cursor.execute("""
UPDATE albums SET thumb_url = ?
WHERE id = ? AND (thumb_url IS NULL OR thumb_url = '')
""", (thumb_url, album_id))
# Backfill genres if album has none
genre = data.get('strGenre')
if genre:
from core.genre_filter import filter_genres
from config.settings import config_manager as _cfg
_filtered = filter_genres([genre], _cfg)
if _filtered:
cursor.execute("""
UPDATE albums SET genres = ?
WHERE id = ? AND (genres IS NULL OR genres = '' OR genres = '[]')
""", (json.dumps(_filtered), album_id))
conn.commit()
except Exception as e:
logger.error(f"Error updating album #{album_id} with AudioDB data: {e}")
raise
finally:
if conn:
conn.close()
def _update_track(self, track_id: int, data: Dict[str, Any]):
"""Store AudioDB metadata for a track using generic column names"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
UPDATE tracks SET
audiodb_id = ?,
audiodb_match_status = 'matched',
audiodb_last_attempted = CURRENT_TIMESTAMP,
style = ?,
mood = ?
WHERE id = ?
""", (
data.get('idTrack'),
data.get('strStyle'),
data.get('strMood'),
track_id
))
conn.commit()
except Exception as e:
logger.error(f"Error updating track #{track_id} with AudioDB data: {e}")
raise
finally:
if conn:
conn.close()
def _mark_status(self, entity_type: str, entity_id: int, status: str):
"""Mark an entity (artist, album, or track) with a match status"""
table_map = {'artist': 'artists', 'album': 'albums', 'track': 'tracks'}
table = table_map.get(entity_type)
if not table:
logger.error(f"Unknown entity type: {entity_type}")
return
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute(f"""
UPDATE {table} SET
audiodb_match_status = ?,
audiodb_last_attempted = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (status, entity_id))
conn.commit()
except Exception as e:
logger.error(f"Error marking {entity_type} #{entity_id} status: {e}")
finally:
if conn:
conn.close()
def _count_pending_items(self) -> int:
"""Count how many items still need processing across all entity types"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
cursor.execute("""
SELECT
(SELECT COUNT(*) FROM artists WHERE audiodb_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM albums WHERE audiodb_match_status IS NULL AND id IS NOT NULL) +
(SELECT COUNT(*) FROM tracks WHERE audiodb_match_status IS NULL AND id IS NOT NULL)
AS pending
""")
row = cursor.fetchone()
return row[0] if row else 0
except Exception as e:
logger.error(f"Error counting pending items: {e}")
return 0
finally:
if conn:
conn.close()
def _get_progress_breakdown(self) -> Dict[str, Dict[str, int]]:
"""Get progress breakdown by entity type"""
conn = None
try:
conn = self.db._get_connection()
cursor = conn.cursor()
progress = {}
# Artists progress
cursor.execute("""
SELECT
COUNT(*) AS total,
SUM(CASE WHEN audiodb_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
FROM artists
""")
row = cursor.fetchone()
if row:
total, processed = row[0], row[1] or 0
progress['artists'] = {
'matched': processed,
'total': total,
'percent': int((processed / total * 100) if total > 0 else 0)
}
# Albums progress
cursor.execute("""
SELECT
COUNT(*) AS total,
SUM(CASE WHEN audiodb_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
FROM albums
""")
row = cursor.fetchone()
if row:
total, processed = row[0], row[1] or 0
progress['albums'] = {
'matched': processed,
'total': total,
'percent': int((processed / total * 100) if total > 0 else 0)
}
# Tracks progress
cursor.execute("""
SELECT
COUNT(*) AS total,
SUM(CASE WHEN audiodb_match_status IS NOT NULL THEN 1 ELSE 0 END) AS processed
FROM tracks
""")
row = cursor.fetchone()
if row:
total, processed = row[0], row[1] or 0
progress['tracks'] = {
'matched': processed,
'total': total,
'percent': int((processed / total * 100) if total > 0 else 0)
}
return progress
except Exception as e:
logger.error(f"Error getting progress breakdown: {e}")
return {}
finally:
if conn:
conn.close()

2086
core/auto_import_worker.py
View file

File diff suppressed because it is too large Load diff

11
core/automation/__init__.py
View file

@ -1,11 +0,0 @@
"""Automation API + progress + handlers package.
Lifted from web_server.py:
- `/api/automations/*` route helpers → `api.py`
- block library used by the trigger/action UI → `blocks.py`
- progress tracker (init / update / finish) → `progress.py`
- cross-handler signal bus → `signals.py`
- per-action handler functions → `handlers/` subpackage (with
`deps.py` defining the dependency-injection surface so handlers
stay testable in isolation)
"""

377
core/automation/api.py
View file

@ -1,377 +0,0 @@
"""Automation REST API helpers.
CRUD + run + progress + history logic for /api/automations/* routes.
Each function takes the deps it needs (database, automation_engine,
profile_id) so the route layer is left as pure HTTP shuffling.
Out of scope:
- /api/automations/blocks — static JSON + one call into signals.py;
stays inline in web_server.py for now.
- /api/test/automation — touches scan manager + media clients +
config_manager; stays inline.
"""
from __future__ import annotations
import json
import logging
from typing import Any, Optional
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Hydration helpers — convert raw DB rows to API-friendly dicts
# ---------------------------------------------------------------------------
_JSON_FIELDS = ('trigger_config', 'action_config', 'notify_config', 'last_result')
_JSON_DEFAULT_DICT = {'trigger_config', 'action_config', 'notify_config'}
def _hydrate_automation(auto: dict) -> dict:
"""Parse JSON columns and backfill `then_actions` from legacy notify_*."""
for field in _JSON_FIELDS:
try:
auto[field] = json.loads(auto[field]) if isinstance(auto[field], str) else auto[field]
except (json.JSONDecodeError, TypeError):
auto[field] = {} if field in _JSON_DEFAULT_DICT else None
try:
raw = auto.get('then_actions')
auto['then_actions'] = json.loads(raw or '[]') if isinstance(raw, str) else (raw or [])
except (json.JSONDecodeError, TypeError):
auto['then_actions'] = []
if not auto['then_actions'] and auto.get('notify_type'):
auto['then_actions'] = [{
'type': auto['notify_type'],
'config': auto.get('notify_config', {}),
}]
return auto
# ---------------------------------------------------------------------------
# Signal cycle detection
# ---------------------------------------------------------------------------
def _has_signal_concern(trigger_type: str, then_actions: list[dict]) -> bool:
return trigger_type == 'signal_received' or any(
t.get('type') == 'fire_signal' for t in then_actions
)
def _check_create_cycle(
automation_engine,
database,
profile_id: int,
trigger_type: str,
trigger_config_json: str,
then_actions_json: str,
then_actions: list[dict],
) -> Optional[str]:
"""Return cycle path string if creating this automation would loop, else None."""
if not automation_engine or not _has_signal_concern(trigger_type, then_actions):
return None
all_autos = database.get_automations(profile_id)
test_auto = {
'trigger_type': trigger_type,
'trigger_config': trigger_config_json,
'then_actions': then_actions_json,
'enabled': True,
}
all_autos.append(test_auto)
cycle = automation_engine.detect_signal_cycles(all_autos)
if cycle:
return ' → '.join(cycle)
return None
def _check_update_cycle(
automation_engine,
database,
automation_id: int,
data: dict,
) -> Optional[str]:
"""Return cycle path string if updating this automation would loop, else None."""
if not automation_engine:
return None
trigger_type = data.get('trigger_type', '')
then_actions = data.get('then_actions', [])
if not _has_signal_concern(trigger_type, then_actions):
return None
all_autos = database.get_automations()
test_autos = []
for a in all_autos:
if a['id'] == automation_id:
merged = dict(a)
if 'trigger_type' in data:
merged['trigger_type'] = data['trigger_type']
if 'trigger_config' in data:
merged['trigger_config'] = json.dumps(data['trigger_config'])
if 'then_actions' in data:
merged['then_actions'] = json.dumps(data['then_actions'])
merged['enabled'] = True
test_autos.append(merged)
else:
test_autos.append(a)
cycle = automation_engine.detect_signal_cycles(test_autos)
if cycle:
return ' → '.join(cycle)
return None
# ---------------------------------------------------------------------------
# CRUD helpers — return (response_dict, http_status)
# ---------------------------------------------------------------------------
def list_automations(database, profile_id: int) -> list[dict]:
"""All automations for the profile, with JSON columns parsed."""
automations = database.get_automations(profile_id)
return [_hydrate_automation(a) for a in automations]
def get_automation(database, automation_id: int) -> Optional[dict]:
"""One automation, hydrated. Returns None if not found."""
auto = database.get_automation(automation_id)
if not auto:
return None
return _hydrate_automation(auto)
def create_automation(
database,
automation_engine,
profile_id: int,
data: dict,
) -> tuple[dict, int]:
"""Create + schedule an automation. Returns (response_body, http_status)."""
name = (data.get('name') or '').strip()
if not name:
return {'error': 'Name is required'}, 400
trigger_type = data.get('trigger_type', 'schedule')
trigger_config = json.dumps(data.get('trigger_config', {}))
action_type = data.get('action_type', 'process_wishlist')
action_config = json.dumps(data.get('action_config', {}))
then_actions = data.get('then_actions', [])
then_actions_json = json.dumps(then_actions)
if then_actions:
notify_type = then_actions[0].get('type')
notify_config = json.dumps(then_actions[0].get('config', {}))
else:
notify_type = data.get('notify_type') or None
notify_config = json.dumps(data.get('notify_config', {})) if notify_type else '{}'
cycle_path = _check_create_cycle(
automation_engine, database, profile_id,
trigger_type, trigger_config, then_actions_json, then_actions,
)
if cycle_path:
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
group_name = data.get('group_name') or None
owned_by = data.get('owned_by') or None
auto_id = database.create_automation(
name, trigger_type, trigger_config, action_type, action_config,
profile_id, notify_type, notify_config, then_actions_json, group_name,
owned_by=owned_by,
)
if auto_id is None:
return {'error': 'Failed to create automation'}, 500
if automation_engine:
automation_engine.schedule_automation(auto_id)
return {'success': True, 'id': auto_id}, 200
def update_automation(
database,
automation_engine,
automation_id: int,
data: dict,
) -> tuple[dict, int]:
"""Update + reschedule an automation. Returns (response_body, http_status)."""
update_fields: dict[str, Any] = {}
if 'name' in data:
update_fields['name'] = data['name'].strip()
if 'trigger_type' in data:
update_fields['trigger_type'] = data['trigger_type']
if 'trigger_config' in data:
update_fields['trigger_config'] = json.dumps(data['trigger_config'])
if 'action_type' in data:
update_fields['action_type'] = data['action_type']
if 'action_config' in data:
update_fields['action_config'] = json.dumps(data['action_config'])
if 'then_actions' in data:
then_actions = data['then_actions']
update_fields['then_actions'] = json.dumps(then_actions)
if then_actions:
update_fields['notify_type'] = then_actions[0].get('type')
update_fields['notify_config'] = json.dumps(then_actions[0].get('config', {}))
else:
update_fields['notify_type'] = None
update_fields['notify_config'] = '{}'
elif 'notify_type' in data:
update_fields['notify_type'] = data['notify_type'] or None
if 'notify_config' in data and 'then_actions' not in data:
update_fields['notify_config'] = json.dumps(data['notify_config'])
if 'group_name' in data:
update_fields['group_name'] = data['group_name'] or None
if 'owned_by' in data:
update_fields['owned_by'] = data['owned_by'] or None
if not update_fields:
return {'error': 'No fields to update'}, 400
cycle_path = _check_update_cycle(automation_engine, database, automation_id, data)
if cycle_path:
return {'error': f'Signal cycle detected: {cycle_path}. This would cause an infinite loop.'}, 400
# Schedule-shape changes must invalidate the stored next_run so the
# scheduler recomputes it; otherwise restart-survival logic keeps the
# leftover timestamp from the previous interval.
if {'trigger_type', 'trigger_config'} & update_fields.keys():
update_fields['next_run'] = None
success = database.update_automation(automation_id, **update_fields)
if not success:
return {'error': 'Automation not found'}, 404
if automation_engine:
auto = database.get_automation(automation_id)
if auto and auto.get('enabled'):
automation_engine.schedule_automation(automation_id)
else:
automation_engine.cancel_automation(automation_id)
return {'success': True}, 200
def batch_update_group(database, automation_ids: list, group_name: Optional[str]) -> tuple[dict, int]:
"""Move/rename a set of automations into a single group (or ungroup)."""
if not automation_ids or not isinstance(automation_ids, list):
return {'error': 'automation_ids must be a non-empty list'}, 400
try:
automation_ids = [int(aid) for aid in automation_ids]
except (ValueError, TypeError):
return {'error': 'automation_ids must contain integers'}, 400
updated = database.batch_update_group(automation_ids, group_name)
return {'success': True, 'updated': updated}, 200
def bulk_toggle(
database,
automation_engine,
automation_ids: list,
enabled: bool,
) -> tuple[dict, int]:
"""Bulk enable/disable a set of automations + reschedule each affected."""
if not automation_ids or not isinstance(automation_ids, list):
return {'error': 'automation_ids must be a non-empty list'}, 400
try:
automation_ids = [int(aid) for aid in automation_ids]
except (ValueError, TypeError):
return {'error': 'automation_ids must contain integers'}, 400
updated = database.bulk_set_enabled(automation_ids, bool(enabled))
if automation_engine and updated > 0:
for aid in automation_ids:
auto = database.get_automation(aid)
if auto:
if auto.get('enabled'):
automation_engine.schedule_automation(auto)
else:
automation_engine.cancel_automation(aid)
return {'success': True, 'updated': updated}, 200
def delete_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
"""Delete an automation. System automations are protected."""
auto = database.get_automation(automation_id)
if auto and auto.get('is_system'):
return {'error': 'System automations cannot be deleted'}, 403
if automation_engine:
automation_engine.cancel_automation(automation_id)
success = database.delete_automation(automation_id)
if not success:
return {'error': 'Automation not found'}, 404
return {'success': True}, 200
def duplicate_automation(
database,
automation_engine,
profile_id: int,
automation_id: int,
) -> tuple[dict, int]:
"""Duplicate an automation. System automations are protected."""
auto = database.get_automation(automation_id)
if not auto:
return {'error': 'Automation not found'}, 404
if auto.get('is_system'):
return {'error': 'System automations cannot be duplicated'}, 403
new_id = database.create_automation(
name=f"{auto['name']} (Copy)",
trigger_type=auto['trigger_type'],
trigger_config=auto.get('trigger_config', '{}'),
action_type=auto['action_type'],
action_config=auto.get('action_config', '{}'),
profile_id=profile_id,
notify_type=auto.get('notify_type'),
notify_config=auto.get('notify_config', '{}'),
then_actions=auto.get('then_actions', '[]'),
group_name=auto.get('group_name'),
)
if new_id is None:
return {'error': 'Failed to duplicate automation'}, 500
if automation_engine:
automation_engine.schedule_automation(new_id)
return {'success': True, 'id': new_id}, 200
def toggle_automation(database, automation_engine, automation_id: int) -> tuple[dict, int]:
"""Toggle an automation's enabled state + reschedule/cancel."""
success = database.toggle_automation(automation_id)
if not success:
return {'error': 'Automation not found'}, 404
if automation_engine:
auto = database.get_automation(automation_id)
if auto and auto.get('enabled'):
automation_engine.schedule_automation(automation_id)
else:
automation_engine.cancel_automation(automation_id)
return {'success': True}, 200
def run_automation(automation_engine, automation_id: int, profile_id: int) -> tuple[dict, int]:
"""Manually trigger an automation."""
if not automation_engine:
return {'error': 'Automation engine not available'}, 500
success = automation_engine.run_now(automation_id, profile_id=profile_id)
if not success:
return {'error': 'Automation not found'}, 404
return {'success': True}, 200
def get_history(database, automation_id: int, *, limit: int, offset: int) -> dict:
"""Run-history page for an automation, with log_lines/result_json parsed."""
data = database.get_automation_run_history(automation_id, limit=limit, offset=offset)
for entry in data.get('history', []):
if entry.get('log_lines'):
try:
entry['log_lines'] = json.loads(entry['log_lines'])
except (json.JSONDecodeError, TypeError):
entry['log_lines'] = []
else:
entry['log_lines'] = []
if entry.get('result_json'):
try:
entry['result_json'] = json.loads(entry['result_json'])
except (json.JSONDecodeError, TypeError):
pass
data['automation_id'] = automation_id
return data

219
core/automation/blocks.py
View file

@ -1,219 +0,0 @@
"""Static block definitions for the automation builder UI.
Returned verbatim by `/api/automations/blocks` (with `known_signals`
injected by the route from `signals.collect_known_signals`).
Three top-level lists:
- `TRIGGERS` — WHEN blocks: schedule, daily/weekly time, app started,
event triggers (track_downloaded, batch_complete, etc.), signal_received,
webhook_received.
- `ACTIONS` — DO blocks: process_wishlist, scan_library, etc.
- `NOTIFICATIONS` — THEN blocks: discord/pushbullet/telegram/webhook,
plus fire_signal and run_script then-actions.
"""
from __future__ import annotations
TRIGGERS: list[dict] = [
{"type": "schedule", "label": "Schedule", "icon": "clock", "description": "Run on a timer interval", "available": True,
"config_fields": [
{"key": "interval", "type": "number", "label": "Every", "default": 6, "min": 1},
{"key": "unit", "type": "select", "label": "Unit",
"options": [{"value": "minutes", "label": "Minutes"}, {"value": "hours", "label": "Hours"}, {"value": "days", "label": "Days"}],
"default": "hours"}
]},
{"type": "daily_time", "label": "Daily Time", "icon": "clock", "description": "Run every day at a specific time", "available": True,
"config_fields": [
{"key": "time", "type": "time", "label": "At", "default": "03:00"}
]},
{"type": "weekly_time", "label": "Weekly Schedule", "icon": "calendar", "description": "Run on specific days of the week at a set time", "available": True,
"config_fields": [
{"key": "time", "type": "time", "label": "At", "default": "03:00"},
{"key": "days", "type": "multi_select", "label": "Days",
"options": [{"value": "mon", "label": "Mon"}, {"value": "tue", "label": "Tue"}, {"value": "wed", "label": "Wed"},
{"value": "thu", "label": "Thu"}, {"value": "fri", "label": "Fri"}, {"value": "sat", "label": "Sat"}, {"value": "sun", "label": "Sun"}]}
]},
{"type": "app_started", "label": "App Started", "icon": "power", "description": "When SoulSync starts up", "available": True},
{"type": "track_downloaded", "label": "Track Downloaded", "icon": "download", "description": "When a track finishes downloading", "available": True,
"has_conditions": True,
"condition_fields": ["artist", "title", "album", "quality"],
"variables": ["artist", "title", "album", "quality"]},
{"type": "batch_complete", "label": "Batch Complete", "icon": "check-circle", "description": "When an album/playlist download finishes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "completed_tracks", "failed_tracks"]},
{"type": "watchlist_new_release", "label": "New Release Found", "icon": "bell", "description": "When watchlist detects new music", "available": True,
"has_conditions": True,
"condition_fields": ["artist"],
"variables": ["artist", "new_tracks", "added_to_wishlist"]},
{"type": "playlist_synced", "label": "Playlist Synced", "icon": "refresh", "description": "When a playlist sync completes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "matched_tracks", "synced_tracks", "failed_tracks"]},
{"type": "playlist_changed", "label": "Playlist Changed", "icon": "edit", "description": "When a mirrored playlist detects track changes from source", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "old_count", "new_count", "added", "removed"]},
{"type": "discovery_completed", "label": "Discovery Complete", "icon": "search", "description": "When playlist track discovery finishes", "available": True,
"has_conditions": True,
"condition_fields": ["playlist_name"],
"variables": ["playlist_name", "total_tracks", "discovered_count", "failed_count", "skipped_count"]},
# Phase 3 triggers
{"type": "wishlist_processing_completed", "label": "Wishlist Processed", "icon": "check-circle",
"description": "When auto-wishlist processing finishes", "available": True,
"variables": ["tracks_processed", "tracks_found", "tracks_failed"]},
{"type": "watchlist_scan_completed", "label": "Watchlist Scan Done", "icon": "check-circle",
"description": "When watchlist scan finishes", "available": True,
"variables": ["artists_scanned", "new_tracks_found", "tracks_added"]},
{"type": "database_update_completed", "label": "Database Updated", "icon": "database",
"description": "When library database refresh finishes", "available": True,
"variables": ["total_artists", "total_albums", "total_tracks"]},
{"type": "library_scan_completed", "label": "Library Scan Done", "icon": "hard-drive",
"description": "When media library scan finishes", "available": True,
"variables": ["server_type"]},
{"type": "download_failed", "label": "Download Failed", "icon": "x-circle",
"description": "When a track permanently fails to download", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title", "reason"],
"variables": ["artist", "title", "reason"]},
{"type": "download_quarantined", "label": "File Quarantined", "icon": "alert-triangle",
"description": "When AcoustID verification fails", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title"],
"variables": ["artist", "title", "reason"]},
{"type": "wishlist_item_added", "label": "Wishlist Item Added", "icon": "plus-circle",
"description": "When a track is added to wishlist", "available": True,
"has_conditions": True, "condition_fields": ["artist", "title"],
"variables": ["artist", "title", "reason"]},
{"type": "watchlist_artist_added", "label": "Artist Watched", "icon": "user-plus",
"description": "When an artist is added to watchlist", "available": True,
"has_conditions": True, "condition_fields": ["artist"],
"variables": ["artist", "artist_id"]},
{"type": "watchlist_artist_removed", "label": "Artist Unwatched", "icon": "user-minus",
"description": "When an artist is removed from watchlist", "available": True,
"has_conditions": True, "condition_fields": ["artist"],
"variables": ["artist", "artist_id"]},
{"type": "import_completed", "label": "Import Complete", "icon": "upload",
"description": "When album/track import finishes", "available": True,
"has_conditions": True, "condition_fields": ["artist", "album_name"],
"variables": ["track_count", "album_name", "artist"]},
{"type": "mirrored_playlist_created", "label": "Playlist Mirrored", "icon": "copy",
"description": "When a new playlist is mirrored", "available": True,
"has_conditions": True, "condition_fields": ["playlist_name", "source"],
"variables": ["playlist_name", "source", "track_count"]},
{"type": "quality_scan_completed", "label": "Quality Scan Done", "icon": "bar-chart",
"description": "When quality scan finishes", "available": True,
"variables": ["quality_met", "low_quality", "total_scanned"]},
{"type": "duplicate_scan_completed", "label": "Duplicate Scan Done", "icon": "layers",
"description": "When duplicate cleaner finishes", "available": True,
"variables": ["files_scanned", "duplicates_found", "space_freed"]},
# Signal trigger
{"type": "signal_received", "label": "Signal Received", "icon": "zap",
"description": "When another automation fires a named signal", "available": True,
"config_fields": [
{"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
],
"variables": ["signal_name"]},
# Webhook trigger
{"type": "webhook_received", "label": "Webhook Received", "icon": "globe",
"description": "When an external API request is received (POST /api/v1/request)", "available": True,
"variables": ["query", "request_id", "source"]},
]
ACTIONS: list[dict] = [
{"type": "process_wishlist", "label": "Process Wishlist", "icon": "list", "description": "Retry failed downloads from wishlist", "available": True,
"config_fields": [{"key": "category", "type": "select", "label": "Category", "options": [{"value": "all", "label": "All"}, {"value": "albums", "label": "Albums"}, {"value": "singles", "label": "Singles"}], "default": "all"}]},
{"type": "scan_watchlist", "label": "Scan Watchlist", "icon": "eye", "description": "Check watched artists for new releases", "available": True},
{"type": "scan_library", "label": "Scan Library", "icon": "refresh", "description": "Trigger media server library scan", "available": True},
{"type": "refresh_mirrored", "label": "Refresh Mirrored Playlist", "icon": "copy", "description": "Re-fetch playlist from source and update mirror", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Refresh all mirrored playlists", "default": False}
]},
{"type": "sync_playlist", "label": "Sync Playlist", "icon": "sync", "description": "Sync mirrored playlist to media server", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"}
]},
{"type": "discover_playlist", "label": "Discover Playlist", "icon": "search", "description": "Find official Spotify/iTunes metadata for mirrored playlist tracks", "available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Discover all mirrored playlists", "default": False}
]},
{"type": "playlist_pipeline", "label": "Playlist Pipeline", "icon": "rocket",
"description": "Full lifecycle: refresh → discover → sync → download missing. One automation for the entire flow.",
"available": True,
"config_fields": [
{"key": "playlist_id", "type": "mirrored_playlist_select", "label": "Playlist"},
{"key": "all", "type": "checkbox", "label": "Process all mirrored playlists", "default": False},
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
]},
{"type": "personalized_pipeline", "label": "Personalized Playlist Pipeline", "icon": "sparkles",
"description": "Sync personalized / discover-page playlists (Hidden Gems, Time Machine, Fresh Tape, etc.) to your media server + queue missing tracks for download.",
"available": True,
"config_fields": [
{"key": "kinds", "type": "personalized_playlist_select", "label": "Playlists to sync",
"description": "Multi-select: Hidden Gems, Discovery Shuffle, Time Machine (per decade), Genre playlists, Fresh Tape, The Archives, Seasonal Mix (per season)"},
{"key": "refresh_first", "type": "checkbox", "label": "Refresh playlists before sync (regenerate snapshots)", "default": False},
{"key": "skip_wishlist", "type": "checkbox", "label": "Skip wishlist processing", "default": False},
]},
{"type": "notify_only", "label": "Notify Only", "icon": "bell", "description": "No action — just send notification", "available": True},
# Phase 3 actions
{"type": "start_database_update", "label": "Update Database", "icon": "database",
"description": "Trigger library database refresh", "available": True,
"config_fields": [
{"key": "full_refresh", "type": "checkbox", "label": "Full refresh (slower)", "default": False}
]},
{"type": "run_duplicate_cleaner", "label": "Run Duplicate Cleaner", "icon": "layers",
"description": "Scan for and remove duplicate files", "available": True},
{"type": "clear_quarantine", "label": "Clear Quarantine", "icon": "trash",
"description": "Delete all quarantined files", "available": True},
{"type": "cleanup_wishlist", "label": "Clean Up Wishlist", "icon": "filter",
"description": "Remove duplicate/owned tracks from wishlist", "available": True},
{"type": "update_discovery_pool", "label": "Update Discovery", "icon": "compass",
"description": "Refresh discovery pool with new tracks", "available": True},
{"type": "start_quality_scan", "label": "Run Quality Scan", "icon": "bar-chart",
"description": "Run the Quality Upgrade Finder (scope is set in Library Maintenance)", "available": True},
{"type": "backup_database", "label": "Backup Database", "icon": "save",
"description": "Create timestamped database backup", "available": True},
{"type": "refresh_beatport_cache", "label": "Refresh Beatport Cache", "icon": "music",
"description": "Scrape Beatport homepage and warm the cache", "available": True},
{"type": "clean_search_history", "label": "Clean Search History", "icon": "trash-2",
"description": "Remove old searches from Soulseek", "available": True},
{"type": "clean_completed_downloads", "label": "Clean Completed Downloads", "icon": "check-square",
"description": "Clear completed downloads and empty directories", "available": True},
{"type": "full_cleanup", "label": "Full Cleanup", "icon": "trash",
"description": "Clear quarantine, download queue, import folder, and search history in one sweep", "available": True},
{"type": "deep_scan_library", "label": "Deep Scan Library", "icon": "search",
"description": "Full library comparison without losing enrichment data", "available": True},
{"type": "run_script", "label": "Run Script", "icon": "terminal",
"description": "Execute a script from the scripts folder", "available": True},
{"type": "search_and_download", "label": "Search & Download", "icon": "download",
"description": "Search for a track and download the best match", "available": True,
"config_fields": [
{"key": "query", "type": "text", "label": "Search Query",
"placeholder": "Artist - Track (leave empty to use trigger's query)"}
]},
]
NOTIFICATIONS: list[dict] = [
{"type": "discord_webhook", "label": "Discord Webhook", "icon": "message", "description": "Send a Discord notification", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "pushbullet", "label": "Pushbullet", "icon": "push", "description": "Push notification to phone/desktop", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "telegram", "label": "Telegram", "icon": "message", "description": "Send a Telegram message", "available": True,
"variables": ["time", "name", "run_count", "status"]},
{"type": "webhook", "label": "Webhook (POST)", "icon": "globe", "description": "Send a POST request to any URL", "available": True,
"variables": ["time", "name", "run_count", "status"]},
# Signal fire action
{"type": "fire_signal", "label": "Fire Signal", "icon": "zap",
"description": "Fire a signal that other automations can listen for", "available": True,
"config_fields": [
{"key": "signal_name", "type": "signal_input", "label": "Signal Name"}
]},
# Run script then-action
{"type": "run_script", "label": "Run Script", "icon": "terminal",
"description": "Execute a script after the action completes", "available": True,
"config_fields": [
{"key": "script_name", "type": "script_select", "label": "Script"}
]},
]

163
core/automation/deps.py
View file

@ -1,163 +0,0 @@
"""Dependency-injection surface for automation handlers.
Each handler in ``core.automation.handlers`` is a top-level pure
function that accepts ``(config: dict, deps: AutomationDeps)`` instead
of reaching for module-level globals in ``web_server``. The deps
namespace bundles every callable, client, and mutable-state container
the handlers need.
Construction happens once at app startup in ``web_server.py``:
from core.automation.deps import AutomationDeps, AutomationState
state = AutomationState()
deps = AutomationDeps(
engine=automation_engine,
state=state,
get_database=get_database,
spotify_client=spotify_client,
...
)
register_all(deps)
Tests construct a fake ``AutomationDeps`` with stub callables — every
handler is then exercisable without spinning up Flask, the DB, or
the real media-server clients.
"""
from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, Optional
@dataclass
class AutomationState:
"""Mutable flags shared across handler invocations.
Pre-refactor each was a ``global`` or ``nonlocal`` variable inside
the registration closure. Lifted here so handlers + their guards
can read/write a single object instead of importing globals.
All mutations should hold ``lock``; the helper methods below do
so for the common get/set patterns.
"""
scan_library_automation_id: Optional[str] = None
db_update_automation_id: Optional[str] = None
pipeline_running: bool = False
lock: threading.Lock = field(default_factory=threading.Lock)
def is_scan_library_active(self) -> bool:
with self.lock:
return self.scan_library_automation_id is not None
def is_pipeline_running(self) -> bool:
with self.lock:
return self.pipeline_running
def try_start_pipeline(self) -> bool:
"""Atomically mark the shared playlist pipeline as running."""
with self.lock:
if self.pipeline_running:
return False
self.pipeline_running = True
return True
def set_scan_library_id(self, automation_id: Optional[str]) -> None:
with self.lock:
self.scan_library_automation_id = automation_id
def set_pipeline_running(self, value: bool) -> None:
with self.lock:
self.pipeline_running = value
@dataclass
class AutomationDeps:
"""Bundle of every callable + client an automation handler may need.
Add fields as new handlers are extracted. Every field is required
at construction (no defaults) so a missing dep fails loudly at
startup, not silently mid-handler.
"""
# --- Engine + shared state ---
engine: Any # AutomationEngine instance
state: AutomationState
config_manager: Any # config.settings.ConfigManager singleton
update_progress: Callable[..., None] # _update_automation_progress
logger: Any # module-level logger from utils.logging_config
# --- Service clients (each may be None depending on user config) ---
get_database: Callable[[], Any] # late-binding so tests don't need DB
spotify_client: Any
tidal_client: Any
web_scan_manager: Any
# --- Background-task entry points ---
process_wishlist_automatically: Callable[..., Any]
process_watchlist_scan_automatically: Callable[..., Any]
is_wishlist_actually_processing: Callable[[], bool]
is_watchlist_actually_scanning: Callable[[], bool]
get_watchlist_scan_state: Callable[[], dict] # accessor returns the live mutable dict
# --- Playlist pipeline entry points ---
run_playlist_discovery_worker: Callable[..., Any]
run_sync_task: Callable[..., Any]
run_playlist_organize_download: Callable[..., Dict[str, Any]]
missing_download_executor: Any
load_sync_status_file: Callable[[], dict]
get_deezer_client: Callable[[], Any]
parse_youtube_playlist: Callable[[str], Any]
get_sync_states: Callable[[], dict] # accessor returns the live dict shared with the sync UI
# --- Database update + quality scanner (shared state + executors) ---
set_db_update_automation_id: Callable[[Optional[str]], None] # syncs the legacy `_db_update_automation_id` global so the live DB-update progress callbacks (which still read the global directly) keep firing for the active automation
get_db_update_state: Callable[[], dict]
db_update_lock: Any # threading.Lock
db_update_executor: Any # ThreadPoolExecutor
run_db_update_task: Callable[..., Any]
run_deep_scan_task: Callable[..., Any]
get_duplicate_cleaner_state: Callable[[], dict]
duplicate_cleaner_lock: Any
duplicate_cleaner_executor: Any
run_duplicate_cleaner: Callable[..., Any]
# Triggers a "Run Now" of a library-maintenance repair job by id (e.g.
# 'quality_upgrade'). Returns truthy if the job was queued. Replaces the old
# standalone quality-scanner executor/state (the scanner is now a repair job).
run_repair_job_now: Callable[[str], Any]
# --- Download orchestrator + queue accessors ---
download_orchestrator: Any
run_async: Callable[..., Any]
tasks_lock: Any
get_download_batches: Callable[[], dict]
get_download_tasks: Callable[[], dict]
sweep_empty_download_directories: Callable[[], int]
get_staging_path: Callable[[], str]
# --- Maintenance helpers ---
docker_resolve_path: Callable[[str], str]
get_current_profile_id: Callable[[], int]
get_watchlist_scanner: Callable[[Any], Any]
get_app: Callable[[], Any] # Flask app for test_client (beatport refresh)
get_beatport_data_cache: Callable[[], dict]
# --- Progress + history callbacks (used by register_all to wire
# the engine's progress callback hooks). ---
init_automation_progress: Callable[..., Any]
record_progress_history: Callable[..., Any]
# --- Personalized playlist pipeline ---
# Lazy builder so the pipeline handler can construct a fresh
# PersonalizedPlaylistManager per run (cheap accessors inside,
# no caching needed yet).
build_personalized_manager: Callable[[], Any]
# --- Unified PlaylistSource registry ---
# Optional so test fixtures that don't exercise refresh_mirrored
# can keep their existing scaffolding. Production wiring in
# ``web_server.py`` always populates it via
# ``core.playlists.sources.bootstrap.build_playlist_source_registry``.
playlist_source_registry: Optional[Any] = None

64
core/automation/handlers/__init__.py
View file

@ -1,64 +0,0 @@
"""Per-action automation handlers.
Each module in this subpackage exposes one top-level handler function
(or a small cluster of related handlers) of the form::
def auto_<action_name>(config: dict, deps: AutomationDeps) -> dict
The ``register_all`` helper in :mod:`registration` wires every handler
to the engine in one place. ``web_server.py`` calls
``register_all(deps)`` once at startup.
"""
from core.automation.handlers.process_wishlist import auto_process_wishlist
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
from core.automation.handlers.scan_library import auto_scan_library
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.automation.handlers.discover_playlist import auto_discover_playlist
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
from core.automation.handlers.database_update import auto_start_database_update, auto_deep_scan_library
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
from core.automation.handlers.quality_scanner import auto_start_quality_scan
from core.automation.handlers.maintenance import (
auto_clear_quarantine,
auto_cleanup_wishlist,
auto_update_discovery_pool,
auto_backup_database,
auto_refresh_beatport_cache,
)
from core.automation.handlers.download_cleanup import (
auto_clean_search_history,
auto_clean_completed_downloads,
auto_full_cleanup,
)
from core.automation.handlers.run_script import auto_run_script
from core.automation.handlers.search_and_download import auto_search_and_download
from core.automation.handlers.registration import register_all
__all__ = [
'auto_process_wishlist',
'auto_scan_watchlist',
'auto_scan_library',
'auto_refresh_mirrored',
'auto_sync_playlist',
'auto_discover_playlist',
'auto_playlist_pipeline',
'auto_personalized_pipeline',
'auto_start_database_update',
'auto_deep_scan_library',
'auto_run_duplicate_cleaner',
'auto_start_quality_scan',
'auto_clear_quarantine',
'auto_cleanup_wishlist',
'auto_update_discovery_pool',
'auto_backup_database',
'auto_refresh_beatport_cache',
'auto_clean_search_history',
'auto_clean_completed_downloads',
'auto_full_cleanup',
'auto_run_script',
'auto_search_and_download',
'register_all',
]

259
core/automation/handlers/_pipeline_shared.py
View file

@ -1,259 +0,0 @@
"""Shared helpers between mirrored + personalized playlist pipelines.
Both pipelines end in the same shape:
1. SYNC each playlist to the active media server.
2. WISHLIST: trigger the wishlist processor for missing tracks.
The differing prefix (mirrored = REFRESH external sources + DISCOVER
metadata; personalized = SNAPSHOT manager-backed playlists) is owned
by each pipeline. This module owns the SYNC + WISHLIST tail so both
pipelines stay consistent + DRY.
"""
from __future__ import annotations
import time
from typing import Any, Callable, Dict, List, Optional
from core.automation.deps import AutomationDeps
# Per-playlist sync poll cap (mirrored side already used this).
_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS = 600
# Sync-status final-state markers.
_SYNC_TERMINAL_STATUSES = ('finished', 'complete', 'error', 'failed')
def run_sync_and_wishlist(
deps: AutomationDeps,
automation_id: Optional[str],
playlists: List[Dict[str, Any]],
*,
sync_one_fn: Callable[[Dict[str, Any]], Dict[str, Any]],
sync_id_for_fn: Callable[[Dict[str, Any]], str],
skip_wishlist: bool = False,
progress_start: int = 56,
progress_end: int = 85,
sync_phase_label: str = 'Phase: Syncing to server...',
sync_phase_start_log: str = 'Sync',
wishlist_phase_label: str = 'Phase: Processing wishlist...',
wishlist_phase_start_log: str = 'Wishlist',
) -> Dict[str, int]:
"""Run the SYNC + WISHLIST tail of a playlist pipeline.
The caller supplies:
- ``playlists``: list of playlist payload dicts. Each must have at
least a ``name`` key (used in progress logs). The shape beyond
``name`` is opaque to the helper — ``sync_one_fn`` receives the
payload and returns a sync_result dict.
- ``sync_one_fn(payload) -> sync_result``: launches sync for one
playlist. Result dict must carry ``status`` ∈ ``('started',
'skipped', 'error')`` and may carry ``reason``.
- ``sync_id_for_fn(payload) -> str``: returns the sync-state key
the helper polls on (so we can wait for the background sync
thread to complete + read the matched_tracks count).
Returns ``{'synced': int, 'skipped': int, 'errors': int,
'wishlist_queued': int}`` so the caller can stitch it into its
final status.
"""
deps.update_progress(
automation_id,
progress=progress_start,
phase=sync_phase_label,
log_line=sync_phase_start_log,
log_type='info',
)
total_synced = 0
total_skipped = 0
sync_errors = 0
sync_states = deps.get_sync_states()
n_playlists = max(1, len(playlists))
progress_span = max(1, progress_end - progress_start - 1)
for pl_idx, pl in enumerate(playlists):
pl_name = pl.get('name', '')
sync_result = sync_one_fn(pl)
sync_status = sync_result.get('status', '')
if sync_status == 'started':
sync_id = sync_id_for_fn(pl)
sync_poll_start = time.time()
timed_out = True
while time.time() - sync_poll_start < _SYNC_PER_PLAYLIST_TIMEOUT_SECONDS:
if (sync_id in sync_states
and sync_states[sync_id].get('status') in _SYNC_TERMINAL_STATUSES):
timed_out = False
break
time.sleep(2)
elapsed = int(time.time() - sync_poll_start)
sub_progress = progress_start + 1 + ((pl_idx + 1) / n_playlists) * progress_span
deps.update_progress(
automation_id,
progress=min(int(sub_progress), progress_end - 1),
phase=f'{sync_phase_label.rstrip(".")} — "{pl_name}" ({elapsed}s)',
)
ss = sync_states.get(sync_id, {})
final_status = ss.get('status')
if timed_out:
sync_errors += 1
deps.update_progress(
automation_id,
log_line=f'Sync timed out "{pl_name}" after {_SYNC_PER_PLAYLIST_TIMEOUT_SECONDS}s',
log_type='error',
)
continue
if final_status in ('error', 'failed'):
sync_errors += 1
reason = ss.get('error') or ss.get('reason') or 'background sync failed'
deps.update_progress(
automation_id,
log_line=f'Sync error "{pl_name}": {reason}',
log_type='error',
)
continue
ss_result = ss.get('result', ss.get('progress', {}))
matched = ss_result.get('matched_tracks', 0) if isinstance(ss_result, dict) else 0
total_synced += int(matched) if matched else 0
deps.update_progress(
automation_id,
log_line=f'Synced "{pl_name}": {matched} tracks matched',
log_type='success',
)
elif sync_status == 'skipped':
total_skipped += 1
reason = sync_result.get('reason', 'unchanged')
deps.update_progress(
automation_id,
log_line=f'Skipped "{pl_name}": {reason}',
log_type='skip',
)
elif sync_status == 'error':
sync_errors += 1
deps.update_progress(
automation_id,
log_line=f'Sync error "{pl_name}": {sync_result.get("reason", "unknown")}',
log_type='error',
)
deps.update_progress(
automation_id,
progress=progress_end,
phase=f'{sync_phase_label.rstrip(".")} complete',
log_line=f'Sync done: {total_synced} matched, {total_skipped} skipped, {sync_errors} errors',
log_type='success' if sync_errors == 0 else 'warning',
)
organize_playlists = [pl for pl in playlists if pl.get('organize_by_playlist')]
organize_started = 0
if organize_playlists and hasattr(deps, 'run_playlist_organize_download'):
for pl in organize_playlists:
pl_id = pl.get('id')
if not pl_id:
continue
pl_name = pl.get('name', '')
try:
org_result = deps.run_playlist_organize_download(
mirrored_playlist_id=int(pl_id),
automation_id=automation_id,
)
if org_result.get('status') == 'started':
organize_started += 1
deps.update_progress(
automation_id,
log_line=f'Organize download started for "{pl_name}"',
log_type='success',
)
elif org_result.get('status') == 'skipped':
deps.update_progress(
automation_id,
log_line=f'Organize download skipped for "{pl_name}": {org_result.get("reason", "")}',
log_type='skip',
)
except Exception as org_err: # noqa: BLE001
deps.update_progress(
automation_id,
log_line=f'Organize download error for "{pl_name}": {org_err}',
log_type='warning',
)
all_organize = bool(playlists) and len(organize_playlists) == len(playlists)
effective_skip_wishlist = skip_wishlist or all_organize
wishlist_queued = run_wishlist_phase(
deps, automation_id,
skip=effective_skip_wishlist,
progress_pct=progress_end + 1,
wishlist_phase_label=wishlist_phase_label,
wishlist_phase_start_log=wishlist_phase_start_log,
)
return {
'synced': total_synced,
'skipped': total_skipped,
'errors': sync_errors,
'wishlist_queued': wishlist_queued,
'organize_downloads_started': organize_started,
}
def run_wishlist_phase(
deps: AutomationDeps,
automation_id: Optional[str],
*,
skip: bool,
progress_pct: int,
wishlist_phase_label: str = 'Phase: Processing wishlist...',
wishlist_phase_start_log: str = 'Wishlist',
) -> int:
"""Trigger the wishlist processor unless skipped or already running.
Returns 1 when the processor was triggered, 0 otherwise. Errors are
logged but never raised — wishlist failure should not abort the
pipeline."""
if skip:
deps.update_progress(
automation_id,
progress=progress_pct,
log_line=f'{wishlist_phase_start_log}: skipped (disabled)',
log_type='skip',
)
return 0
deps.update_progress(
automation_id,
progress=progress_pct,
phase=wishlist_phase_label,
log_line=wishlist_phase_start_log,
log_type='info',
)
try:
if not deps.is_wishlist_actually_processing():
deps.process_wishlist_automatically(automation_id=None)
deps.update_progress(
automation_id,
log_line='Wishlist processing triggered',
log_type='success',
)
return 1
deps.update_progress(
automation_id,
log_line='Wishlist already running — skipped',
log_type='skip',
)
return 0
except Exception as e: # noqa: BLE001 — wishlist failure must never abort pipeline
deps.update_progress(
automation_id,
log_line=f'Wishlist error: {e}',
log_type='warning',
)
return 0
__all__ = ['run_sync_and_wishlist', 'run_wishlist_phase']

197
core/automation/handlers/database_update.py
View file

@ -1,197 +0,0 @@
"""Automation handlers: ``start_database_update`` and
``deep_scan_library`` actions.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_start_database_update`` and ``_auto_deep_scan_library``
closures). Both share the same ``db_update_state`` / executor / lock
infrastructure -- the only difference is which task they submit
(``run_db_update_task`` vs ``run_deep_scan_task``).
Pattern: pre-set state to running, submit task to executor, then
poll the state dict until it transitions away from ``running``.
Stall-detection emits a warning every 10 minutes when progress
hasn't budged. 2-hour outer timeout caps the worst case.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# Time out on STALL (no progress), not total runtime: a large library can scan
# for many hours while progressing fine — a hard total cap would falsely mark a
# healthy scan 'error' (the scan thread keeps running uncancelled). We only give
# up when progress hasn't moved for a long stretch, with a generous absolute
# backstop against a truly stuck monitor loop.
_STALL_WARNING_SECONDS = 600 # warn after 10 min with no progress (repeats)
_STALL_TIMEOUT_SECONDS = 1800 # 30 min with no progress at all = genuinely stalled
_ABSOLUTE_CAP_SECONDS = 86400 # 24h hard backstop (runaway-loop guard only)
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def scan_wait_action(
*,
status: str,
idle_seconds: float,
total_seconds: float,
stall_timeout_s: float = _STALL_TIMEOUT_SECONDS,
stall_warn_s: float = _STALL_WARNING_SECONDS,
abs_cap_s: float = _ABSOLUTE_CAP_SECONDS,
) -> str:
"""Decide what the monitor loop should do on a poll tick (pure/testable).
``idle_seconds`` is time since progress last changed; ``total_seconds`` is
time since the wait began. Returns one of:
``'finished'`` (task no longer running), ``'stall_timeout'`` (no progress for
too long → give up), ``'abs_timeout'`` (absolute backstop), ``'warn'``
(stalled long enough to warn but not give up), or ``'continue'``.
Crucially, an actively-progressing scan keeps resetting ``idle_seconds``, so
it never hits ``stall_timeout`` no matter how long the whole scan takes.
"""
if status != 'running':
return 'finished'
if total_seconds >= abs_cap_s:
return 'abs_timeout'
if idle_seconds >= stall_timeout_s:
return 'stall_timeout'
if idle_seconds >= stall_warn_s:
return 'warn'
return 'continue'
def auto_start_database_update(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a full or incremental DB update via ``run_db_update_task``."""
return _run_with_progress(
config, deps,
task=deps.run_db_update_task,
task_args=(config.get('full_refresh', False), deps.config_manager.get_active_media_server()),
initial_phase='Initializing...',
stall_label='Database update',
finished_extras=lambda: {'full_refresh': str(config.get('full_refresh', False))},
timeout_label='Database update timed out after 24 hours',
)
def auto_deep_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a deep library scan via ``run_deep_scan_task``."""
return _run_with_progress(
config, deps,
task=deps.run_deep_scan_task,
task_args=(deps.config_manager.get_active_media_server(),),
initial_phase='Deep scan: Initializing...',
stall_label='Deep scan',
finished_extras=lambda: {},
timeout_label='Deep scan timed out after 24 hours',
)
def _run_with_progress(
config: Dict[str, Any],
deps: AutomationDeps,
*,
task,
task_args: tuple,
initial_phase: str,
stall_label: str,
finished_extras,
timeout_label: str,
) -> Dict[str, Any]:
"""Shared poll-and-wait body for both DB-update handlers."""
automation_id = config.get('_automation_id')
state = deps.get_db_update_state()
if state.get('status') == 'running':
return {'status': 'skipped', 'reason': 'Database update already running'}
deps.state.db_update_automation_id = automation_id
# Sync legacy module global so the DB-update progress callbacks
# (still living in web_server.py) emit against this automation.
deps.set_db_update_automation_id(automation_id)
with deps.db_update_lock:
state.update({
'status': 'running', 'phase': initial_phase,
'progress': 0, 'current_item': '', 'processed': 0, 'total': 0,
'error_message': '',
})
deps.db_update_executor.submit(task, *task_args)
# Monitor progress (callbacks handle card updates, we just block until done).
# We time out on STALL, not total runtime: ``processed`` advances on every
# artist, so an actively-progressing scan keeps resetting the idle clock and
# is never falsely failed no matter how long the whole library takes.
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
last_progress_time = time.time()
# Any of these advancing means the scan is alive. current_item (the artist
# being processed) changes every artist even when the rounded progress %
# holds steady, so it guards against a false stall during slow stretches.
last_progress_val = (0, 0, '')
last_warn_time = 0.0
outcome = 'finished'
while True:
time.sleep(_POLL_INTERVAL_SECONDS)
now = time.time()
with deps.db_update_lock:
current_status = state.get('status', 'idle')
current_val = (state.get('processed', 0), state.get('progress', 0),
state.get('current_item', ''))
if current_val != last_progress_val:
last_progress_val = current_val
last_progress_time = now
action = scan_wait_action(
status=current_status,
idle_seconds=now - last_progress_time,
total_seconds=now - poll_start,
)
if action in ('finished', 'stall_timeout', 'abs_timeout'):
outcome = action
break
if action == 'warn' and (now - last_warn_time) > _STALL_WARNING_SECONDS:
idle_min = int((now - last_progress_time) / 60)
deps.update_progress(
automation_id,
log_line=f'{stall_label} — no progress for {idle_min} min, still waiting...',
log_type='warning',
)
last_warn_time = now
if outcome == 'stall_timeout':
deps.update_progress(
automation_id, status='error', phase='Stalled',
log_line=f'{stall_label} made no progress for {_STALL_TIMEOUT_SECONDS // 60} minutes — giving up',
log_type='error',
)
return {'status': 'error', 'reason': 'Stalled (no progress)', '_manages_own_progress': True}
if outcome == 'abs_timeout':
deps.update_progress(
automation_id, status='error',
phase='Timed out', log_line=timeout_label, log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
# Finished/error callback already updated the card — return matching status.
with deps.db_update_lock:
final_status = state.get('status', 'unknown')
if final_status == 'error':
return {
'status': 'error',
'reason': state.get('error_message', 'Unknown error'),
'_manages_own_progress': True,
}
with deps.db_update_lock:
stats = {
'status': 'completed', '_manages_own_progress': True,
'artists': state.get('total', 0),
'albums': state.get('total_albums', 0),
'tracks': state.get('total_tracks', 0),
'removed_artists': state.get('removed_artists', 0),
'removed_albums': state.get('removed_albums', 0),
'removed_tracks': state.get('removed_tracks', 0),
}
stats.update(finished_extras())
return stats

48
core/automation/handlers/discover_playlist.py
View file

@ -1,48 +0,0 @@
"""Automation handler: ``discover_playlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_discover_playlist`` closure). Kicks off background discovery
of official Spotify / iTunes metadata for mirrored playlist tracks.
The worker runs in a daemon thread and emits its own progress; this
handler returns immediately after launching it (``_manages_own_progress``).
"""
from __future__ import annotations
import threading
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_discover_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Discover official Spotify/iTunes metadata for mirrored
playlist tracks. Runs the worker in a background thread."""
db = deps.get_database()
playlist_id = config.get('playlist_id')
discover_all = config.get('all', False)
if discover_all:
playlists = db.get_mirrored_playlists()
elif playlist_id:
p = db.get_mirrored_playlist(int(playlist_id))
playlists = [p] if p else []
else:
return {'status': 'error', 'reason': 'No playlist specified'}
if not playlists:
return {'status': 'error', 'reason': 'No playlists found'}
threading.Thread(
target=deps.run_playlist_discovery_worker,
args=(playlists, config.get('_automation_id')),
daemon=True,
name='auto-discover-playlist',
).start()
names = ', '.join(p['name'] for p in playlists[:3])
return {
'status': 'started',
'playlist_count': str(len(playlists)),
'playlists': names,
'_manages_own_progress': True,
}

267
core/automation/handlers/download_cleanup.py
View file

@ -1,267 +0,0 @@
"""Automation handlers: download-queue cleanup actions.
Lifted from ``web_server._register_automation_handlers``:
- ``clean_search_history`` → :func:`auto_clean_search_history`
- ``clean_completed_downloads`` → :func:`auto_clean_completed_downloads`
- ``full_cleanup`` → :func:`auto_full_cleanup`
All three share the download-orchestrator + tasks_lock /
download_batches / download_tasks accessors. ``full_cleanup`` is a
multi-step orchestration that pulls in quarantine purge + staging
sweep on top of the queue cleanup -- kept as one big handler since
its phases share state-detection logic.
"""
from __future__ import annotations
import os
import shutil as _shutil
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# ─── clean_search_history ────────────────────────────────────────────
def auto_clean_search_history(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Remove old searches from Soulseek when configured."""
automation_id = config.get('_automation_id')
# Skip if soulseek is not the active download source or in hybrid order.
dl_mode = deps.config_manager.get('download_source.mode', 'hybrid')
hybrid_order = deps.config_manager.get(
'download_source.hybrid_order', ['hifi', 'youtube', 'soulseek'],
)
soulseek_active = (
dl_mode == 'soulseek'
or (dl_mode == 'hybrid' and 'soulseek' in hybrid_order)
)
# Reach the underlying SoulseekClient via the orchestrator's
# generic accessor.
slskd = deps.download_orchestrator.client('soulseek') if deps.download_orchestrator else None
if not soulseek_active or not slskd or not slskd.base_url:
deps.update_progress(automation_id, log_line='Soulseek not active — skipped', log_type='skip')
return {'status': 'skipped'}
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
deps.update_progress(
automation_id, log_line='Auto-clear disabled in settings', log_type='skip',
)
return {'status': 'skipped'}
try:
success = deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
keep_searches=50, trigger_threshold=200,
))
if success:
deps.update_progress(
automation_id,
log_line='Search history maintenance completed',
log_type='success',
)
return {'status': 'completed'}
else:
deps.update_progress(automation_id, log_line='No cleanup needed', log_type='skip')
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'error': str(e)}
# ─── clean_completed_downloads ───────────────────────────────────────
def auto_clean_completed_downloads(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Clear completed downloads + sweep empty download directories.
Skips when active batches or post-processing is in flight."""
automation_id = config.get('_automation_id')
try:
has_active_batches = False
has_post_processing = False
with deps.tasks_lock:
batches = deps.get_download_batches()
for batch_data in batches.values():
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
has_active_batches = True
break
if not has_active_batches:
tasks = deps.get_download_tasks()
for task_data in tasks.values():
if task_data.get('status') == 'post_processing':
has_post_processing = True
break
if has_active_batches:
deps.update_progress(
automation_id, log_line='Skipped — downloads active', log_type='skip',
)
return {'status': 'completed'}
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
if not has_post_processing:
deps.sweep_empty_download_directories()
deps.update_progress(
automation_id, log_line='Download cleanup completed', log_type='success',
)
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'reason': str(e)}
# ─── full_cleanup ────────────────────────────────────────────────────
def auto_full_cleanup(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run all cleanup tasks: quarantine purge → download queue clear
→ empty-dir sweep → staging sweep → search history."""
automation_id = config.get('_automation_id')
steps = []
# --- 1. Clear quarantine ---
deps.update_progress(automation_id, phase='Clearing quarantine...', progress=0)
quarantine_path = os.path.join(
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
'ss_quarantine',
)
q_removed = 0
if os.path.exists(quarantine_path):
for f in os.listdir(quarantine_path):
fp = os.path.join(quarantine_path, f)
try:
if os.path.isfile(fp):
os.remove(fp)
q_removed += 1
elif os.path.isdir(fp):
_shutil.rmtree(fp)
q_removed += 1
except Exception as e: # noqa: BLE001 — best-effort purge
deps.logger.debug("quarantine entry purge failed: %s", e)
steps.append(f'Quarantine: removed {q_removed} items')
deps.update_progress(
automation_id,
log_line=f'Quarantine: removed {q_removed} items',
log_type='success' if q_removed else 'info',
)
# --- 2. Clear completed/errored/cancelled downloads from Soulseek queue ---
deps.update_progress(automation_id, phase='Clearing download queue...', progress=20)
has_active_batches = False
has_post_processing = False
with deps.tasks_lock:
batches = deps.get_download_batches()
for batch_data in batches.values():
if batch_data.get('phase') not in ['complete', 'error', 'cancelled', None]:
has_active_batches = True
break
if not has_active_batches:
tasks = deps.get_download_tasks()
for task_data in tasks.values():
if task_data.get('status') == 'post_processing':
has_post_processing = True
break
if has_active_batches:
steps.append('Download queue: skipped (active batches)')
deps.update_progress(
automation_id,
log_line='Download queue: skipped (active batches)',
log_type='skip',
)
else:
try:
deps.run_async(deps.download_orchestrator.clear_all_completed_downloads())
steps.append('Download queue: cleared')
deps.update_progress(
automation_id, log_line='Download queue: cleared', log_type='success',
)
except Exception as e: # noqa: BLE001 — per-step best-effort
steps.append(f'Download queue: error ({e})')
deps.update_progress(
automation_id,
log_line=f'Download queue: error ({e})',
log_type='error',
)
# --- 3. Sweep empty download directories ---
deps.update_progress(automation_id, phase='Sweeping empty directories...', progress=40)
if has_active_batches or has_post_processing:
reason = 'active batches' if has_active_batches else 'post-processing active'
steps.append(f'Empty directories: skipped ({reason})')
deps.update_progress(
automation_id,
log_line=f'Empty directories: skipped ({reason})',
log_type='skip',
)
else:
dirs_removed = deps.sweep_empty_download_directories()
steps.append(f'Empty directories: removed {dirs_removed}')
deps.update_progress(
automation_id,
log_line=f'Empty directories: removed {dirs_removed}',
log_type='success' if dirs_removed else 'info',
)
# --- 4. Sweep empty staging directories ---
deps.update_progress(automation_id, phase='Sweeping import folder...', progress=60)
staging_path = deps.get_staging_path()
s_removed = 0
if os.path.isdir(staging_path):
for dirpath, _dirnames, _filenames in os.walk(staging_path, topdown=False):
if os.path.normpath(dirpath) == os.path.normpath(staging_path):
continue
try:
entries = os.listdir(dirpath)
except OSError:
continue
visible = [e for e in entries if not e.startswith('.')]
if not visible:
for hidden in entries:
try:
os.remove(os.path.join(dirpath, hidden))
except Exception as e: # noqa: BLE001 — best-effort
deps.logger.debug("hidden file cleanup failed: %s", e)
try:
os.rmdir(dirpath)
s_removed += 1
except OSError:
pass
steps.append(f'Staging: removed {s_removed} empty directories')
deps.update_progress(
automation_id,
log_line=f'Staging: removed {s_removed} empty directories',
log_type='success' if s_removed else 'info',
)
# --- 5. Clean search history (if enabled) ---
deps.update_progress(automation_id, phase='Cleaning search history...', progress=80)
try:
if not deps.config_manager.get('soulseek.auto_clear_searches', True):
steps.append('Search cleanup: disabled in settings')
deps.update_progress(
automation_id, log_line='Search cleanup: disabled in settings', log_type='skip',
)
else:
deps.run_async(deps.download_orchestrator.maintain_search_history_with_buffer(
keep_searches=50, trigger_threshold=200,
))
steps.append('Search history: cleaned')
deps.update_progress(
automation_id, log_line='Search history: cleaned', log_type='success',
)
except Exception as e: # noqa: BLE001 — per-step best-effort
steps.append(f'Search history: error ({e})')
deps.update_progress(
automation_id, log_line=f'Search history: error ({e})', log_type='error',
)
total_removed = q_removed + s_removed
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Full cleanup complete — {total_removed} items removed',
log_type='success',
)
return {
'status': 'completed',
'quarantine_removed': str(q_removed),
'staging_removed': str(s_removed),
'total_removed': str(total_removed),
'steps': steps,
'_manages_own_progress': True,
}

87
core/automation/handlers/duplicate_cleaner.py
View file

@ -1,87 +0,0 @@
"""Automation handler: ``run_duplicate_cleaner`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_run_duplicate_cleaner`` closure). Submits the duplicate
cleaner to its executor, then polls the shared state dict until
the worker transitions away from ``running``.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
_TIMEOUT_SECONDS = 7200 # 2 hours
_POLL_INTERVAL_SECONDS = 3
_INITIAL_DELAY_SECONDS = 1
def auto_run_duplicate_cleaner(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Kick off the duplicate cleaner and report final stats."""
automation_id = config.get('_automation_id')
state = deps.get_duplicate_cleaner_state()
if state.get('status') == 'running':
return {'status': 'skipped', 'reason': 'Duplicate cleaner already running'}
# Pre-set status before submit so the polling loop doesn't see a
# stale 'finished' from a previous run.
with deps.duplicate_cleaner_lock:
state['status'] = 'running'
deps.duplicate_cleaner_executor.submit(deps.run_duplicate_cleaner)
deps.update_progress(automation_id, log_line='Duplicate cleaner started', log_type='info')
# Monitor progress (max 2 hours).
time.sleep(_INITIAL_DELAY_SECONDS)
poll_start = time.time()
while time.time() - poll_start < _TIMEOUT_SECONDS:
time.sleep(_POLL_INTERVAL_SECONDS)
current_status = state.get('status', 'idle')
if current_status not in ('running',):
break
deps.update_progress(
automation_id,
phase=state.get('phase', 'Scanning...'),
progress=state.get('progress', 0),
processed=state.get('files_scanned', 0),
total=state.get('total_files', 0),
)
else:
# 2-hour timeout reached.
deps.update_progress(
automation_id, status='error',
phase='Timed out',
log_line='Duplicate cleaner timed out after 2 hours',
log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
# Check actual exit status (could be 'finished' or 'error').
final_status = state.get('status', 'idle')
if final_status == 'error':
err = state.get('error_message', 'Unknown error')
deps.update_progress(
automation_id, status='error', progress=100,
phase='Error', log_line=err, log_type='error',
)
return {'status': 'error', 'reason': err, '_manages_own_progress': True}
dupes = state.get('duplicates_found', 0)
removed = state.get('deleted', 0)
space_freed = state.get('space_freed', 0)
scanned = state.get('files_scanned', 0)
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Found {dupes} duplicates, removed {removed} files',
log_type='success',
)
return {
'status': 'completed', '_manages_own_progress': True,
'files_scanned': scanned,
'duplicates_found': dupes,
'files_deleted': removed,
'space_freed_mb': round(space_freed / (1024 * 1024), 1),
}

221
core/automation/handlers/maintenance.py
View file

@ -1,221 +0,0 @@
"""Automation handlers: short maintenance actions.
Lifted from ``web_server._register_automation_handlers``:
- ``clear_quarantine`` → :func:`auto_clear_quarantine`
- ``cleanup_wishlist`` → :func:`auto_cleanup_wishlist`
- ``update_discovery_pool`` → :func:`auto_update_discovery_pool`
- ``backup_database`` → :func:`auto_backup_database`
- ``refresh_beatport_cache`` → :func:`auto_refresh_beatport_cache`
Each is a thin wrapper around an existing service / helper. Grouped
in one module because every body is short and they share no state
between them — splitting into per-handler files would just add
import noise.
"""
from __future__ import annotations
import glob as _glob
import os
import shutil as _shutil
import sqlite3
import time
from datetime import datetime
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# ─── clear_quarantine ────────────────────────────────────────────────
def auto_clear_quarantine(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Purge every file/folder under the configured ss_quarantine path."""
automation_id = config.get('_automation_id')
quarantine_path = os.path.join(
deps.docker_resolve_path(deps.config_manager.get('soulseek.download_path', './downloads')),
'ss_quarantine',
)
if not os.path.exists(quarantine_path):
deps.update_progress(automation_id, log_line='No quarantine folder found', log_type='info')
return {'status': 'completed', 'removed': '0'}
removed = 0
for f in os.listdir(quarantine_path):
fp = os.path.join(quarantine_path, f)
try:
if os.path.isfile(fp):
os.remove(fp)
removed += 1
elif os.path.isdir(fp):
_shutil.rmtree(fp)
removed += 1
except Exception as e: # noqa: BLE001 — best-effort purge
deps.logger.debug("quarantine entry purge failed: %s", e)
deps.update_progress(
automation_id,
log_line=f'Removed {removed} quarantined items',
log_type='success' if removed > 0 else 'info',
)
return {'status': 'completed', 'removed': str(removed)}
# ─── cleanup_wishlist ────────────────────────────────────────────────
def auto_cleanup_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Drop duplicate entries from the wishlist for the active profile."""
automation_id = config.get('_automation_id')
db = deps.get_database()
removed = db.remove_wishlist_duplicates(deps.get_current_profile_id())
deps.update_progress(
automation_id,
log_line=f'Removed {removed or 0} duplicate wishlist entries',
log_type='success' if removed else 'info',
)
return {'status': 'completed', 'removed': str(removed or 0)}
# ─── update_discovery_pool ───────────────────────────────────────────
def auto_update_discovery_pool(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run an incremental refresh of the discovery pool via the
watchlist scanner."""
automation_id = config.get('_automation_id')
try:
scanner = deps.get_watchlist_scanner(deps.spotify_client)
deps.update_progress(automation_id, log_line='Updating discovery pool...', log_type='info')
scanner.update_discovery_pool_incremental(deps.get_current_profile_id())
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete', log_line='Discovery pool updated', log_type='success',
)
return {'status': 'completed', '_manages_own_progress': True}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
deps.update_progress(
automation_id, status='error',
phase='Error', log_line=str(e), log_type='error',
)
return {'status': 'error', 'reason': str(e), '_manages_own_progress': True}
# ─── backup_database ─────────────────────────────────────────────────
_MAX_BACKUPS = 5
def auto_backup_database(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Create a hot SQLite backup, then prune old backups so only the
newest ``_MAX_BACKUPS`` remain."""
automation_id = config.get('_automation_id')
db_path = os.environ.get('DATABASE_PATH', 'database/music_library.db')
if not os.path.exists(db_path):
return {'status': 'error', 'reason': 'Database file not found'}
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
backup_path = f"{db_path}.backup_{timestamp}"
# safe_backup verifies source + result integrity, so an automated backup
# can never silently snapshot a corrupt DB (the incident where every
# rolling backup faithfully copied the corruption).
from core.db_integrity import DBIntegrityError, safe_backup, prune_backups
try:
safe_backup(db_path, backup_path)
except DBIntegrityError as integ:
deps.logger.error("Auto-backup refused — DB integrity check failed: %s", integ)
deps.update_progress(
automation_id,
log_line=f'Backup SKIPPED — database failed integrity check: {integ}',
log_type='error',
)
return {'status': 'error', 'reason': f'Database integrity check failed: {integ}'}
size_mb = round(os.path.getsize(backup_path) / (1024 * 1024), 1)
# Rolling cleanup — never evict the most-recent verified-healthy backup.
existing = list(_glob.glob(f"{db_path}.backup_*"))
for removed in prune_backups(existing, _MAX_BACKUPS):
try:
os.remove(removed)
except Exception as e: # noqa: BLE001 — best-effort cleanup
deps.logger.debug("rolling backup cleanup failed: %s", e)
deps.update_progress(
automation_id,
log_line=f'Backup created: {size_mb}MB ({os.path.basename(backup_path)})',
log_type='success',
)
return {'status': 'completed', 'backup_path': backup_path, 'size_mb': str(size_mb)}
# ─── refresh_beatport_cache ──────────────────────────────────────────
_BEATPORT_SECTIONS = (
('hero_tracks', '/api/beatport/hero-tracks', 'Hero Tracks'),
('new_releases', '/api/beatport/new-releases', 'New Releases'),
('featured_charts', '/api/beatport/featured-charts', 'Featured Charts'),
('dj_charts', '/api/beatport/dj-charts', 'DJ Charts'),
('top_10_lists', '/api/beatport/homepage/top-10-lists', 'Top 10 Lists'),
('top_10_releases', '/api/beatport/homepage/top-10-releases-cards', 'Top 10 Releases'),
('hype_picks', '/api/beatport/hype-picks', 'Hype Picks'),
)
def auto_refresh_beatport_cache(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Refresh Beatport homepage cache by calling each endpoint internally
via Flask's ``test_client``. Invalidates the homepage cache first
so endpoints re-scrape rather than returning stale data."""
automation_id = config.get('_automation_id')
cache = deps.get_beatport_data_cache()
# Invalidate all homepage cache timestamps so endpoints re-scrape.
with cache['cache_lock']:
for key in cache['homepage']:
cache['homepage'][key]['timestamp'] = 0
cache['homepage'][key]['data'] = None
refreshed = 0
errors = []
app = deps.get_app()
with app.test_client() as client:
for idx, (_, endpoint, label) in enumerate(_BEATPORT_SECTIONS):
deps.update_progress(
automation_id,
progress=(idx / len(_BEATPORT_SECTIONS)) * 100,
phase=f'Scraping: {label}',
current_item=label,
)
try:
resp = client.get(endpoint)
if resp.status_code == 200:
refreshed += 1
deps.update_progress(
automation_id, log_line=f'{label}: cached', log_type='success',
)
else:
errors.append(label)
deps.update_progress(
automation_id,
log_line=f'{label}: HTTP {resp.status_code}',
log_type='error',
)
except Exception as e: # noqa: BLE001 — per-section best-effort
errors.append(label)
deps.update_progress(
automation_id,
log_line=f'{label}: {str(e)}',
log_type='error',
)
if idx < len(_BEATPORT_SECTIONS) - 1:
time.sleep(2)
deps.update_progress(
automation_id, status='finished', progress=100,
phase='Complete',
log_line=f'Refreshed {refreshed}/{len(_BEATPORT_SECTIONS)} sections',
log_type='success',
)
return {
'status': 'completed',
'refreshed': str(refreshed),
'errors': str(len(errors)),
'_manages_own_progress': True,
}

356
core/automation/handlers/personalized_pipeline.py
View file

@ -1,356 +0,0 @@
"""Personalized Playlist Pipeline automation handler.
Sibling to ``auto_playlist_pipeline`` (mirrored). Where the mirrored
pipeline runs REFRESH external sources → DISCOVER metadata → SYNC →
WISHLIST, the personalized pipeline is simpler:
SNAPSHOT → SYNC → WISHLIST
SNAPSHOT reads the persisted track list from
``PersonalizedPlaylistManager``. When ``refresh_first=True`` (config),
each playlist is refreshed BEFORE syncing — useful when the user
wants the cron to capture a fresh-each-run view (e.g. "give me a new
Hidden Gems set every night"). Default is to sync the existing
snapshot, on the assumption the user / a separate cron has already
refreshed when they wanted to.
Config schema:
{
'kinds': [
{'kind': 'hidden_gems'},
{'kind': 'time_machine', 'variant': '1980s'},
{'kind': 'seasonal_mix', 'variant': 'halloween'},
...
],
'refresh_first': bool, # default false
'skip_wishlist': bool, # default false
}
Each kind dict has at minimum ``kind``; ``variant`` is required for
kinds that need it (time_machine, genre_playlist, daily_mix,
seasonal_mix). Singleton kinds (hidden_gems, discovery_shuffle,
popular_picks, fresh_tape, archives) ignore variant.
Pipeline-running flag (``deps.state.pipeline_running``) is shared
with the mirrored pipeline so the two can't overlap. (One sync
queue, one wishlist worker — overlapping triggers would step on
each other.)"""
from __future__ import annotations
import json
import threading
import time
from typing import Any, Dict, List, Optional
from core.automation.deps import AutomationDeps
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
# Sync state key prefix so personalized syncs don't collide with
# mirrored ones (`auto_mirror_<id>`).
_SYNC_ID_PREFIX = 'auto_personalized'
def auto_personalized_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run SNAPSHOT → SYNC → WISHLIST for selected personalized playlists."""
deps.state.set_pipeline_running(True)
automation_id = config.get('_automation_id')
pipeline_start = time.time()
try:
kinds_config = config.get('kinds') or []
if not isinstance(kinds_config, list) or not kinds_config:
deps.state.set_pipeline_running(False)
return {
'status': 'error',
'error': 'No personalized playlist kinds selected',
}
refresh_first = bool(config.get('refresh_first', False))
skip_wishlist = bool(config.get('skip_wishlist', False))
manager = deps.build_personalized_manager()
deps.update_progress(
automation_id,
progress=2,
phase=f'Personalized pipeline: {len(kinds_config)} playlist(s)',
log_line=f'Starting pipeline for {len(kinds_config)} playlist(s)',
log_type='info',
)
# ── PHASE 1: SNAPSHOT (optionally refresh) ──────────────────
deps.update_progress(
automation_id,
progress=3,
phase='Phase 1/2: Loading snapshots...' if not refresh_first
else 'Phase 1/2: Refreshing snapshots...',
log_line='Phase 1: Snapshot' + (' (with refresh)' if refresh_first else ''),
log_type='info',
)
profile_id = deps.get_current_profile_id()
playload_payloads = _build_payloads_for_kinds(
deps, manager, kinds_config, profile_id,
automation_id=automation_id,
refresh_first=refresh_first,
)
if not playload_payloads:
deps.state.set_pipeline_running(False)
deps.update_progress(
automation_id,
status='finished', progress=100,
phase='No playlists to sync',
log_line='No personalized playlists had tracks to sync',
log_type='warning',
)
return {
'status': 'completed',
'_manages_own_progress': True,
'playlists_synced': '0',
'tracks_synced': '0',
'duration_seconds': str(int(time.time() - pipeline_start)),
}
deps.update_progress(
automation_id,
progress=50,
phase='Phase 1/2: Snapshot complete',
log_line=f'Phase 1 done: {len(playload_payloads)} playlist(s) ready to sync',
log_type='success',
)
# ── PHASE 2: SYNC + WISHLIST (shared helper) ────────────────
sync_summary = run_sync_and_wishlist(
deps,
automation_id,
playload_payloads,
sync_one_fn=lambda pl: _sync_personalized_playlist(deps, pl),
sync_id_for_fn=lambda pl: pl['sync_id'],
skip_wishlist=skip_wishlist,
progress_start=51,
progress_end=90,
sync_phase_label='Phase 2/2: Syncing to server...',
sync_phase_start_log='Phase 2: Sync',
wishlist_phase_label='Phase 2/2: Processing wishlist...',
wishlist_phase_start_log='Wishlist',
)
# ── COMPLETE ────────────────────────────────────────────────
duration = int(time.time() - pipeline_start)
deps.update_progress(
automation_id,
status='finished', progress=100,
phase='Pipeline complete',
log_line=f'Personalized pipeline finished in {duration // 60}m {duration % 60}s',
log_type='success',
)
deps.state.set_pipeline_running(False)
return {
'status': 'completed',
'_manages_own_progress': True,
'playlists_synced': str(len(playload_payloads)),
'tracks_synced': str(sync_summary['synced']),
'sync_skipped': str(sync_summary['skipped']),
'wishlist_queued': str(sync_summary['wishlist_queued']),
'duration_seconds': str(duration),
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into engine
deps.state.set_pipeline_running(False)
deps.update_progress(
automation_id,
status='error', progress=100,
phase='Pipeline error',
log_line=f'Personalized pipeline failed: {e}',
log_type='error',
)
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
def _build_payloads_for_kinds(
deps: AutomationDeps,
manager: Any,
kinds_config: List[Dict[str, Any]],
profile_id: int,
*,
automation_id: Optional[str],
refresh_first: bool,
) -> List[Dict[str, Any]]:
"""Resolve each requested kind+variant into a sync-payload dict.
Each payload has: ``{'name', 'kind', 'variant', 'tracks_json',
'image_url', 'sync_id'}``. Playlists with no tracks (e.g. a
seasonal mix that hasn't been populated yet) are omitted from
the result so the sync loop doesn't waste time on empty pushes.
"""
payloads: List[Dict[str, Any]] = []
for entry in kinds_config:
if not isinstance(entry, dict):
continue
kind = entry.get('kind')
variant = entry.get('variant') or ''
if not kind:
continue
try:
# Refresh when ANY of:
# - explicit user flag (cron use case: regenerate each run)
# - snapshot marked stale by upstream data refresher
# - playlist was never generated yet (auto-created by
# ensure_playlist; track_count=0, last_generated_at=NULL).
# Without this branch, a first-run pipeline reads the
# empty snapshot and silently skips — user picks a kind,
# hits run, gets "No tracks to sync" with no clue why.
if refresh_first:
record = manager.refresh_playlist(kind, variant, profile_id)
else:
existing = manager.ensure_playlist(kind, variant, profile_id)
needs_first_gen = existing.last_generated_at is None
if existing.is_stale or needs_first_gen:
record = manager.refresh_playlist(kind, variant, profile_id)
else:
record = existing
except Exception as exc: # noqa: BLE001 — log + continue with next kind
deps.update_progress(
automation_id,
log_line=f'Skipping {kind}{("/" + variant) if variant else ""}: {exc}',
log_type='warning',
)
continue
tracks = manager.get_playlist_tracks(record.id)
if not tracks:
deps.update_progress(
automation_id,
log_line=f'No tracks in {record.name} — skipping sync',
log_type='skip',
)
continue
tracks_json = [_track_to_sync_shape(t) for t in tracks]
payloads.append({
'name': record.name,
'kind': record.kind,
'variant': record.variant,
'tracks_json': tracks_json,
'image_url': '', # personalized playlists don't have a cover image yet
'sync_id': f'{_SYNC_ID_PREFIX}_{record.kind}_{record.variant or "_"}',
})
return payloads
def _track_to_sync_shape(track: Any) -> Dict[str, Any]:
"""Convert a personalized.types.Track into the dict shape
`_run_sync_task` expects. Mirrors what the mirrored pipeline
builds from extra_data.matched_data, preserving enriched metadata
from personalized snapshots when available."""
primary_id = track.spotify_track_id or track.itunes_track_id or track.deezer_track_id or ''
rich_data = _coerce_track_data_json(getattr(track, 'track_data_json', None))
if not rich_data:
album = {'name': track.album_name or ''}
cover_url = getattr(track, 'album_cover_url', None)
if cover_url:
album['images'] = [{'url': cover_url}]
return {
'name': track.track_name,
'artists': [{'name': track.artist_name}],
'album': album,
'duration_ms': int(track.duration_ms or 0),
'id': primary_id,
}
payload = dict(rich_data)
cover_url = (
getattr(track, 'album_cover_url', None)
or payload.get('album_cover_url')
or payload.get('image_url')
)
payload['id'] = payload.get('id') or primary_id
payload['name'] = payload.get('name') or track.track_name
payload['artists'] = _normalize_artists(payload.get('artists'), track.artist_name)
payload['album'] = _normalize_album(payload.get('album'), track, cover_url=cover_url)
payload['duration_ms'] = int(payload.get('duration_ms') or track.duration_ms or 0)
if 'popularity' not in payload and getattr(track, 'popularity', None) is not None:
payload['popularity'] = int(track.popularity or 0)
if cover_url and not payload.get('image_url'):
payload['image_url'] = cover_url
return payload
def _coerce_track_data_json(value: Any) -> Dict[str, Any]:
if isinstance(value, dict):
return value
if isinstance(value, str) and value.strip():
try:
loaded = json.loads(value)
except (TypeError, ValueError):
return {}
return loaded if isinstance(loaded, dict) else {}
return {}
def _normalize_artists(artists: Any, fallback_artist: str) -> List[Dict[str, Any]]:
if not artists:
return [{'name': fallback_artist or 'Unknown Artist'}]
if isinstance(artists, list):
normalized = []
for artist in artists:
if isinstance(artist, dict):
normalized.append(artist if artist.get('name') else {'name': str(artist)})
elif isinstance(artist, str):
normalized.append({'name': artist})
else:
normalized.append({'name': str(artist)})
return normalized or [{'name': fallback_artist or 'Unknown Artist'}]
if isinstance(artists, dict):
return [artists if artists.get('name') else {'name': str(artists)}]
return [{'name': str(artists)}]
def _normalize_album(album: Any, track: Any, cover_url: Optional[str] = None) -> Dict[str, Any]:
if isinstance(album, dict):
normalized = dict(album)
else:
normalized = {'name': str(album) if album else (track.album_name or '')}
normalized['name'] = normalized.get('name') or track.album_name or ''
images = normalized.get('images')
if not isinstance(images, list):
images = []
if cover_url and not images:
images = [{'url': cover_url}]
if images:
normalized['images'] = images
return normalized
def _sync_personalized_playlist(deps: AutomationDeps, payload: Dict[str, Any]) -> Dict[str, Any]:
"""Launch a personalized playlist sync via _run_sync_task on a
daemon thread + return immediately with status='started'.
Mirrors the mirrored ``auto_sync_playlist`` return contract so the
shared helper can poll on ``sync_states[sync_id]`` and aggregate
results identically."""
sync_id = payload['sync_id']
name = payload['name']
tracks_json = payload['tracks_json']
profile_id = deps.get_current_profile_id()
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, name, tracks_json, None, profile_id, payload.get('image_url', '')),
daemon=True,
name=f'auto-personalized-{sync_id}',
).start()
return {
'status': 'started',
'playlist_name': name,
'_manages_own_progress': True,
}

27
core/automation/handlers/playlist_pipeline.py
View file

@ -1,27 +0,0 @@
"""Automation adapter for the mirrored playlist pipeline.
The actual all-in-one playlist lifecycle lives in
``core.playlists.pipeline`` so it can be reused by non-automation UI actions.
This module only wires automation-specific dependencies and handlers.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
from core.automation.handlers._pipeline_shared import run_sync_and_wishlist
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.playlists.pipeline import run_mirrored_playlist_pipeline
def auto_playlist_pipeline(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run REFRESH -> DISCOVER -> SYNC -> WISHLIST for mirrored playlists."""
return run_mirrored_playlist_pipeline(
config,
deps,
refresh_fn=auto_refresh_mirrored,
sync_one_fn=auto_sync_playlist,
sync_and_wishlist_fn=run_sync_and_wishlist,
)

27
core/automation/handlers/process_wishlist.py
View file

@ -1,27 +0,0 @@
"""Automation handler: ``process_wishlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_process_wishlist`` closure). Wishlist processing is async —
the helper submits a batch to an executor and returns immediately;
per-track stats arrive later via batch-completion callbacks.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_process_wishlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Kick off the wishlist processor for an automation trigger.
Returns immediately after submission; the wishlist worker emits
per-batch progress via its own callbacks. We only report
``status: completed`` to mark the trigger fired successfully.
"""
try:
deps.process_wishlist_automatically(automation_id=config.get('_automation_id'))
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
return {'status': 'error', 'error': str(e)}

89
core/automation/handlers/progress_callbacks.py
View file

@ -1,89 +0,0 @@
"""Progress + history callbacks the automation engine invokes around
each handler run.
Lifted from the closures at the bottom of
``web_server._register_automation_handlers``:
- ``_progress_init`` → :func:`progress_init`
- ``_progress_finish`` → :func:`progress_finish`
- ``_record_automation_history`` → :func:`record_history`
- ``_on_library_scan_completed`` → :func:`on_library_scan_completed`
The engine accepts four callables via
``register_progress_callbacks(init, finish, update, history)``;
``registration.register_all`` wires these here. The
``library_scan_completed`` callback is registered separately on the
``web_scan_manager`` (when one is available) -- see
``register_library_scan_completed_emitter``.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def progress_init(aid: Any, name: str, action_type: str, deps: AutomationDeps) -> None:
"""Initialize per-automation progress state when the engine starts
a handler. Thin wrapper so the engine receives a closure that
delegates into the live progress tracker."""
deps.init_automation_progress(aid, name, action_type)
def progress_finish(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
"""Emit the final progress update when a handler returns.
Skipped for handlers that manage their own progress lifecycle
(they call ``update_progress(status='finished')`` themselves and
set ``_manages_own_progress: True`` in the returned dict).
Otherwise translates the handler's status into a finished/error
progress emit with a status-appropriate phase + log line.
"""
if result.get('_manages_own_progress'):
return
result_status = result.get('status', '')
status = 'error' if result_status == 'error' else 'finished'
msg = result.get('error', result.get('reason', result_status or 'done'))
deps.update_progress(
aid,
status=status,
progress=100,
phase='Error' if status == 'error' else 'Complete',
log_line=msg,
log_type='error' if status == 'error' else 'success',
)
def record_history(aid: Any, result: Dict[str, Any], deps: AutomationDeps) -> None:
"""Capture progress state into run history before the engine's
cleanup pass clears it. Thin wrapper so the engine sees a stable
callable."""
deps.record_progress_history(aid, result, deps.get_database())
def on_library_scan_completed(deps: AutomationDeps) -> None:
"""Emit the ``library_scan_completed`` automation event with the
active media-server type. Replaces the hard-coded
``scan_completion_callback → trigger_automatic_database_update``
chain so any automation can listen for scan completion as a
trigger."""
if not deps.engine:
return
server_type = (
getattr(deps.web_scan_manager, '_current_server_type', None)
or 'unknown'
)
deps.engine.emit('library_scan_completed', {
'server_type': server_type,
})
def register_library_scan_completed_emitter(deps: AutomationDeps) -> None:
"""Wire :func:`on_library_scan_completed` to the
``web_scan_manager``'s scan-completion callback list. No-op when
no scan manager is configured (e.g. headless / test contexts)."""
if not deps.web_scan_manager:
return
deps.web_scan_manager.add_scan_completion_callback(
lambda: on_library_scan_completed(deps),
)

35
core/automation/handlers/quality_scanner.py
View file

@ -1,35 +0,0 @@
"""Automation handler: ``start_quality_scan`` action.
The quality scanner was redesigned from an auto-acting tool into the
``quality_upgrade`` library-maintenance repair job (findings-based, reviewed
before anything is wishlisted). This action now simply triggers a "Run Now" of
that job; its progress and findings surface in Library Maintenance. The action
name is kept so existing automation rules keep working.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_start_quality_scan(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
automation_id = config.get('_automation_id')
triggered = deps.run_repair_job_now('quality_upgrade')
if not triggered:
deps.update_progress(
automation_id, status='error', phase='Unavailable',
log_line='Quality Upgrade job could not be triggered (library worker unavailable)',
log_type='error',
)
return {'status': 'error', 'reason': 'library worker unavailable',
'_manages_own_progress': True}
deps.update_progress(
automation_id, status='finished', progress=100, phase='Triggered',
log_line='Quality Upgrade scan queued — findings appear in Library Maintenance',
log_type='success',
)
return {'status': 'completed', 'triggered': True, '_manages_own_progress': True}

341
core/automation/handlers/refresh_mirrored.py
View file

@ -1,341 +0,0 @@
"""Automation handler: ``refresh_mirrored`` action.
Re-pulls track lists from each mirrored playlist's source via the
unified ``PlaylistSourceRegistry`` (Phase 1 of the Discover-to-Sync
unification). The pre-extraction handler had ~190 lines of per-source
if/elif branches; this version delegates to the adapter for each
source, leaving the handler responsible only for:
- filtering sources that can't be refreshed (``file``, ``beatport``),
- extracting upstream URLs from the stored ``description`` for URL-
backed sources (``spotify_public``, ``youtube``),
- the Spotify-public → authenticated-Spotify fallback (uses the
``spotify`` adapter when the user is signed in so the mirror keeps
album art),
- the Tidal-not-authenticated skip log type (vs error),
- preserving existing per-track ``extra_data`` on tracks that survive
the refresh, and
- emitting the ``playlist_changed`` automation event when the track
set actually shifts.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
from core.automation.deps import AutomationDeps
from core.playlists.source_refs import require_refresh_url
from core.playlists.sources import PlaylistDetail, to_mirror_track_dict
from core.playlists.sources.base import (
SOURCE_SPOTIFY,
SOURCE_SPOTIFY_PUBLIC,
SOURCE_TIDAL,
SOURCE_YOUTUBE,
)
# Sources that store the upstream URL in ``description`` (because their
# ``source_playlist_id`` is a deterministic hash, not the native ID).
# The refresh path has to recover the URL before calling the adapter.
_URL_BACKED_SOURCES = {SOURCE_SPOTIFY_PUBLIC, SOURCE_YOUTUBE}
def auto_refresh_mirrored(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Refresh mirrored playlist(s) from source.
Returns ``{'status': 'completed', 'refreshed': '<int>',
'errors': '<int>'}`` on success (counts stringified to match the
automation engine's stat-rendering convention).
"""
db = deps.get_database()
playlist_id = config.get('playlist_id')
refresh_all = config.get('all', False)
auto_id = config.get('_automation_id')
# Pipeline runs (``run_mirrored_playlist_pipeline``) set this flag
# because Phase 2 of the pipeline already runs the playlist
# discovery worker — the same matching engine ``_maybe_discover``
# would call here. Running both means LB tracks discover twice
# AND the refresh-side discovery blocks 5+ minutes with no
# progress emission, leaving the UI stuck on "Refreshing:" until
# the loop returns. Standalone callers (Sync page, registration
# action) leave it False so LB tracks still get matched_data on
# refresh.
skip_discovery = bool(config.get('skip_discovery', False))
if refresh_all:
playlists = db.get_mirrored_playlists()
elif playlist_id:
p = db.get_mirrored_playlist(int(playlist_id))
playlists = [p] if p else []
else:
return {'status': 'error', 'reason': 'No playlist specified'}
# Filter out sources that can't be refreshed (no external API).
playlists = [pl for pl in playlists if pl.get('source', '') not in ('file', 'beatport')]
refreshed = 0
errors: List[str] = []
for idx, pl in enumerate(playlists):
try:
source = pl.get('source', '')
source_id = pl.get('source_playlist_id', '')
deps.update_progress(
auto_id,
progress=(idx / max(1, len(playlists))) * 100,
phase=f'Refreshing: "{pl.get("name", "")}"',
current_item=pl.get('name', ''),
)
detail = _fetch_detail(source, source_id, pl, deps, auto_id)
if detail is None:
# _fetch_detail already logged the specific failure;
# mark the playlist as a generic refresh error so the
# automation result tally matches the legacy handler.
errors.append(f"{pl.get('name', '?')}: no tracks returned from source")
deps.update_progress(
auto_id,
log_line=f'Refresh failed: "{pl.get("name", "")}" - no tracks returned from source',
log_type='error',
)
continue
# Sources that return MB-metadata-only tracks (LB, Last.fm)
# mark them ``needs_discovery=True``. Hand them to the
# adapter's matcher so the resulting mirror rows carry
# provider IDs + matched_data, ready for the sync pipeline.
#
# Pipeline runs skip this because Phase 2's discovery
# worker handles it with proper progress emission — see
# ``skip_discovery`` resolution at the top of this fn.
detail_tracks = (
detail.tracks if skip_discovery
else _maybe_discover(detail.tracks, source, deps)
)
tracks = [to_mirror_track_dict(t) for t in detail_tracks]
refreshed += _commit_refresh(pl, source, source_id, tracks, db, deps, auto_id)
except _SkipPlaylist:
# Source-specific soft-skip (e.g. Tidal not authenticated).
# Logging was already emitted; do not count as error.
continue
except Exception as e:
errors.append(f"{pl.get('name', '?')}: {str(e)}")
deps.update_progress(
auto_id,
log_line=f'Error: {pl.get("name", "?")} — {str(e)}',
log_type='error',
)
return {'status': 'completed', 'refreshed': str(refreshed), 'errors': str(len(errors))}
def _maybe_discover(
tracks: List[Any],
source: str,
deps: AutomationDeps,
) -> List[Any]:
"""Run the adapter's ``discover_tracks`` when any track needs it.
Most sources are no-ops here (their tracks have provider IDs
already). LB / Last.fm are the ones that actually do work."""
if not tracks:
return tracks
if not any(getattr(t, "needs_discovery", False) for t in tracks):
return tracks
registry = deps.playlist_source_registry
if registry is None:
return tracks
adapter = registry.get_source(source)
if adapter is None:
return tracks
try:
return adapter.discover_tracks(tracks)
except Exception as exc:
deps.logger.warning(f"{source} discover_tracks failed: {exc}")
return tracks
class _SkipPlaylist(Exception):
"""Internal sentinel: source-specific soft-skip (e.g. not authed).
The per-playlist loop catches it specifically so the skip isn't
counted in the error tally — matches the pre-extraction behavior
where ``continue`` was used inline."""
def _fetch_detail(
source: str,
source_id: str,
pl: Dict[str, Any],
deps: AutomationDeps,
auto_id: Optional[str],
) -> Optional[PlaylistDetail]:
"""Resolve the playlist's tracks through the registry.
Handler-level branches (URL extraction, Spotify-public→authed
fallback, Tidal not-authed skip) live here; everything else
delegates to the adapter."""
registry = deps.playlist_source_registry
if registry is None:
return None
# URL-backed sources: pull the upstream URL out of `description`.
playlist_input = source_id
if source in _URL_BACKED_SOURCES:
# ``require_refresh_url`` raises ValueError on missing URL.
# The outer try/except in the loop catches it and reports as
# an error — matching the pre-extraction behavior.
playlist_input = require_refresh_url(
source, pl.get('description', ''), pl.get('name', '')
)
# Spotify-public refresh: prefer the authenticated Spotify API
# when the user is signed in. Better album art, matches the
# pre-extraction handler. Falls through to the public scraper on
# auth failure or non-playlist URL types (e.g. album URLs).
if source == SOURCE_SPOTIFY_PUBLIC:
detail = _try_spotify_authed_for_public(playlist_input, deps)
if detail is not None:
return detail
# Tidal not-authed: soft-skip with a 'skip' log line, not an error.
if source == SOURCE_TIDAL:
tidal_source = registry.get_source(SOURCE_TIDAL)
if tidal_source is None or not tidal_source.is_authenticated():
deps.logger.warning(
f"Tidal not authenticated — skipping refresh for '{pl.get('name', '')}'"
)
deps.update_progress(
auto_id,
log_line=f'Skipped "{pl.get("name", "")}" — Tidal not authenticated',
log_type='skip',
)
raise _SkipPlaylist
adapter = registry.get_source(source)
if adapter is None:
return None
try:
return adapter.refresh_playlist(playlist_input)
except Exception as exc:
deps.logger.warning(
f"{source} playlist refresh failed for {playlist_input}: {exc}"
)
return None
def _try_spotify_authed_for_public(
spotify_url: str, deps: AutomationDeps
) -> Optional[PlaylistDetail]:
"""Best-effort: use the authenticated Spotify adapter on a public URL.
Returns ``None`` to signal "fall through to the public-scraper
adapter" — never raises. Only applies to ``playlist``-type URLs;
album URLs fall through unconditionally."""
if not spotify_url:
return None
spotify_client = deps.spotify_client
if spotify_client is None or not spotify_client.is_spotify_authenticated():
return None
try:
from core.spotify_public_scraper import parse_spotify_url
parsed = parse_spotify_url(spotify_url)
except Exception:
return None
if not parsed or parsed.get('type') != 'playlist':
return None
adapter = deps.playlist_source_registry.get_source(SOURCE_SPOTIFY)
if adapter is None:
return None
try:
return adapter.refresh_playlist(parsed['id'])
except Exception as exc:
deps.logger.debug(f"Spotify authed fallback for public mirror failed: {exc}")
return None
def _commit_refresh(
pl: Dict[str, Any],
source: str,
source_id: str,
tracks: List[Dict[str, Any]],
db: Any,
deps: AutomationDeps,
auto_id: Optional[str],
) -> int:
"""Persist the refreshed track list + emit playlist_changed when delta.
Returns 1 when a refresh successfully landed, 0 otherwise. The
caller is responsible for incrementing the running tally."""
old_tracks = db.get_mirrored_playlist_tracks(pl['id']) if pl.get('id') else []
old_ids = {t.get('source_track_id') for t in old_tracks if t.get('source_track_id')}
new_ids = {t.get('source_track_id') for t in tracks if t.get('source_track_id')}
# Preserve existing extra_data (matched_data + discovery state)
# for tracks that still exist in the refreshed snapshot, unless
# the adapter already provided fresh extra_data for that track.
old_extra_map = (
db.get_mirrored_tracks_extra_data_map(pl['id']) if pl.get('id') else {}
)
for t in tracks:
sid = t.get('source_track_id', '')
if sid and sid in old_extra_map and 'extra_data' not in t:
t['extra_data'] = old_extra_map[sid]
db.mirror_playlist(
source=source,
source_playlist_id=source_id,
name=pl['name'],
tracks=tracks,
profile_id=pl.get('profile_id', 1),
description=pl.get('description'),
owner=pl.get('owner'),
image_url=pl.get('image_url'),
)
# Membership just changed — if this playlist is organize-by-playlist, rebuild
# its folder (with prune) so a track that LEFT the playlist has its symlink
# cleaned up now. Gated to organized playlists, non-fatal — never disturbs
# the refresh. (Additions are handled by the post-download reconcile.)
try:
from core.playlists.materialize_service import rebuild_mirrored_playlist_if_organized
rebuild_mirrored_playlist_if_organized(
db, deps.config_manager, pl.get('id'), profile_id=pl.get('profile_id', 1)
)
except Exception as _mat_err:
deps.logger.debug(f"[Playlist Folder] mirror-refresh cleanup skipped: {_mat_err}")
if old_ids != new_ids:
added = len(new_ids - old_ids)
removed = len(old_ids - new_ids)
deps.logger.info(
f"[AUTOMATION] Playlist changed: '{pl.get('name', '')}' — "
f"{added} added, {removed} removed (old={len(old_ids)}, new={len(new_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'"{pl.get("name", "")}" — {added} added, {removed} removed',
log_type='success',
)
try:
if deps.engine:
deps.engine.emit('playlist_changed', {
'playlist_name': pl.get('name', ''),
'playlist_id': str(pl.get('id', '')),
'old_count': str(len(old_ids)),
'new_count': str(len(new_ids)),
'added': str(added),
'removed': str(removed),
})
except Exception as e:
deps.logger.debug("playlist_synced automation emit failed: %s", e)
else:
deps.logger.warning(
f"[AUTOMATION] No changes: '{pl.get('name', '')}' (tracks={len(old_ids)})"
)
deps.update_progress(
auto_id,
log_line=f'No changes: "{pl.get("name", "")}"',
log_type='skip',
)
return 1

184
core/automation/handlers/registration.py
View file

@ -1,184 +0,0 @@
"""One-stop registration of every extracted automation handler.
``web_server`` builds the deps once at startup and calls
:func:`register_all` here. Each new handler module gets one line in
this file when it lands.
"""
from __future__ import annotations
from core.automation.deps import AutomationDeps
from core.automation.handlers.process_wishlist import auto_process_wishlist
from core.automation.handlers.scan_watchlist import auto_scan_watchlist
from core.automation.handlers.scan_library import auto_scan_library
from core.automation.handlers.refresh_mirrored import auto_refresh_mirrored
from core.automation.handlers.sync_playlist import auto_sync_playlist
from core.automation.handlers.discover_playlist import auto_discover_playlist
from core.automation.handlers.playlist_pipeline import auto_playlist_pipeline
from core.automation.handlers.personalized_pipeline import auto_personalized_pipeline
from core.automation.handlers.database_update import (
auto_start_database_update, auto_deep_scan_library,
)
from core.automation.handlers.duplicate_cleaner import auto_run_duplicate_cleaner
from core.automation.handlers.quality_scanner import auto_start_quality_scan
from core.automation.handlers.maintenance import (
auto_clear_quarantine,
auto_cleanup_wishlist,
auto_update_discovery_pool,
auto_backup_database,
auto_refresh_beatport_cache,
)
from core.automation.handlers.download_cleanup import (
auto_clean_search_history,
auto_clean_completed_downloads,
auto_full_cleanup,
)
from core.automation.handlers.run_script import auto_run_script
from core.automation.handlers.search_and_download import auto_search_and_download
from core.automation.handlers.progress_callbacks import (
progress_init,
progress_finish,
record_history,
register_library_scan_completed_emitter,
)
def register_all(deps: AutomationDeps) -> None:
"""Wire every extracted handler to the engine.
Each ``register_action_handler`` call binds the action name (the
string the trigger uses to look up its action) to a thin lambda
that injects ``deps`` and forwards the engine-supplied config.
Guards stay alongside their handler so duplicate-run prevention
behaves identically to the pre-extraction code.
"""
engine = deps.engine
# Self-guards prevent duplicate runs of the SAME operation, but
# different operations can run concurrently — wishlist downloads
# use bandwidth, watchlist scans use API calls, library scans use
# media-server CPU. Different resources, no contention.
engine.register_action_handler(
'process_wishlist',
lambda config: auto_process_wishlist(config, deps),
guard_fn=deps.is_wishlist_actually_processing,
)
engine.register_action_handler(
'scan_watchlist',
lambda config: auto_scan_watchlist(config, deps),
guard_fn=deps.is_watchlist_actually_scanning,
)
engine.register_action_handler(
'scan_library',
lambda config: auto_scan_library(config, deps),
deps.state.is_scan_library_active,
)
# Playlist lifecycle handlers. The pipeline composes refresh +
# sync + discover (it imports them directly), so all four ship
# together. The pipeline guard prevents an in-flight pipeline
# from being re-triggered mid-run.
engine.register_action_handler(
'refresh_mirrored',
lambda config: auto_refresh_mirrored(config, deps),
)
engine.register_action_handler(
'sync_playlist',
lambda config: auto_sync_playlist(config, deps),
)
engine.register_action_handler(
'discover_playlist',
lambda config: auto_discover_playlist(config, deps),
)
engine.register_action_handler(
'playlist_pipeline',
lambda config: auto_playlist_pipeline(config, deps),
deps.state.is_pipeline_running,
)
# Personalized pipeline shares the pipeline_running flag with the
# mirrored pipeline so the two can't overlap (single sync queue,
# single wishlist worker).
engine.register_action_handler(
'personalized_pipeline',
lambda config: auto_personalized_pipeline(config, deps),
deps.state.is_pipeline_running,
)
# Database update + deep scan share the db_update_state guard —
# only one operation can mutate that state at a time.
engine.register_action_handler(
'start_database_update',
lambda config: auto_start_database_update(config, deps),
lambda: deps.get_db_update_state().get('status') == 'running',
)
engine.register_action_handler(
'deep_scan_library',
lambda config: auto_deep_scan_library(config, deps),
lambda: deps.get_db_update_state().get('status') == 'running',
)
engine.register_action_handler(
'run_duplicate_cleaner',
lambda config: auto_run_duplicate_cleaner(config, deps),
lambda: deps.get_duplicate_cleaner_state().get('status') == 'running',
)
engine.register_action_handler(
'clear_quarantine',
lambda config: auto_clear_quarantine(config, deps),
)
engine.register_action_handler(
'cleanup_wishlist',
lambda config: auto_cleanup_wishlist(config, deps),
)
engine.register_action_handler(
'update_discovery_pool',
lambda config: auto_update_discovery_pool(config, deps),
)
engine.register_action_handler(
'start_quality_scan',
lambda config: auto_start_quality_scan(config, deps),
lambda: False, # repair worker dedupes Run-Now requests itself
)
engine.register_action_handler(
'backup_database',
lambda config: auto_backup_database(config, deps),
)
engine.register_action_handler(
'refresh_beatport_cache',
lambda config: auto_refresh_beatport_cache(config, deps),
)
engine.register_action_handler(
'clean_search_history',
lambda config: auto_clean_search_history(config, deps),
)
engine.register_action_handler(
'clean_completed_downloads',
lambda config: auto_clean_completed_downloads(config, deps),
)
engine.register_action_handler(
'full_cleanup',
lambda config: auto_full_cleanup(config, deps),
)
engine.register_action_handler(
'run_script',
lambda config: auto_run_script(config, deps),
)
engine.register_action_handler(
'search_and_download',
lambda config: auto_search_and_download(config, deps),
)
# Progress + history callbacks: the engine invokes these around
# each handler run. Lift the closures from
# `web_server._register_automation_handlers` into thin lambdas
# that delegate into the extracted top-level functions.
engine.register_progress_callbacks(
lambda aid, name, action_type: progress_init(aid, name, action_type, deps),
lambda aid, result: progress_finish(aid, result, deps),
deps.update_progress,
lambda aid, result: record_history(aid, result, deps),
)
# `library_scan_completed` event: when the media-server scan
# manager finishes a scan, emit the event so any automation can
# trigger off it. No-op when no scan manager is configured.
register_library_scan_completed_emitter(deps)

103
core/automation/handlers/run_script.py
View file

@ -1,103 +0,0 @@
"""Automation handler: ``run_script`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_run_script`` closure). Runs a user-provided shell or Python
script from the configured scripts directory with bounded timeout +
captured stdout/stderr. Path-traversal guard ensures users can't
escape the scripts directory.
Environment variables exposed to the script:
- ``SOULSYNC_EVENT``: triggering event type (when fired by an event)
- ``SOULSYNC_AUTOMATION``: automation name
- ``SOULSYNC_SCRIPTS_DIR``: absolute path to the scripts dir
"""
from __future__ import annotations
import os
import subprocess as _sp
from typing import Any, Dict
from core.automation.deps import AutomationDeps
_MAX_TIMEOUT_SECONDS = 300 # Hard cap on user-supplied timeout config.
def auto_run_script(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
script_name = config.get('script_name', '')
timeout = min(int(config.get('timeout', 60)), _MAX_TIMEOUT_SECONDS)
automation_id = config.get('_automation_id')
if not script_name:
return {'status': 'error', 'error': 'No script selected'}
scripts_dir = deps.docker_resolve_path(deps.config_manager.get('scripts.path', './scripts'))
if not scripts_dir or not os.path.isdir(scripts_dir):
os.makedirs(scripts_dir, exist_ok=True)
return {
'status': 'error',
'error': 'Scripts directory is empty. Add scripts to the scripts/ folder.',
}
script_path = os.path.join(scripts_dir, script_name)
script_path = os.path.realpath(script_path)
# Security: block path traversal — script must resolve under
# the scripts dir, no symlinks/.. tricks allowed out.
if not script_path.startswith(os.path.realpath(scripts_dir)):
return {'status': 'error', 'error': 'Script path traversal blocked'}
if not os.path.isfile(script_path):
return {'status': 'error', 'error': f'Script not found: {script_name}'}
deps.update_progress(automation_id, phase=f'Running {script_name}...', progress=10)
# Build environment with SoulSync context.
env = os.environ.copy()
event_data = config.get('_event_data') or {}
env['SOULSYNC_EVENT'] = str(event_data.get('type', ''))
env['SOULSYNC_AUTOMATION'] = config.get('_automation_name', '')
env['SOULSYNC_SCRIPTS_DIR'] = scripts_dir
try:
# Determine how to run the script.
if script_path.endswith('.py'):
cmd = ['python', script_path]
elif script_path.endswith('.sh'):
cmd = ['bash', script_path]
else:
cmd = [script_path]
result = _sp.run(
cmd,
capture_output=True, text=True, timeout=timeout,
cwd=scripts_dir, env=env,
)
deps.update_progress(automation_id, phase='Script completed', progress=100)
stdout = result.stdout[:2000] if result.stdout else ''
stderr = result.stderr[:1000] if result.stderr else ''
if result.returncode == 0:
deps.logger.info(f"Script '{script_name}' completed (exit 0)")
else:
deps.logger.warning(f"Script '{script_name}' exited with code {result.returncode}")
return {
'status': 'completed' if result.returncode == 0 else 'error',
'exit_code': str(result.returncode),
'stdout': stdout,
'stderr': stderr,
'script': script_name,
}
except _sp.TimeoutExpired:
deps.update_progress(automation_id, phase='Script timed out', progress=100)
return {
'status': 'error',
'error': f'Script timed out after {timeout}s',
'script': script_name,
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
return {'status': 'error', 'error': str(e), 'script': script_name}

158
core/automation/handlers/scan_library.py
View file

@ -1,158 +0,0 @@
"""Automation handler: ``scan_library`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_scan_library`` closure). The handler triggers a media-server
scan via ``web_scan_manager``, then polls the manager's status until
the scan completes (or a 30-minute timeout fires). Progress phases
are emitted via :func:`AutomationDeps.update_progress` so the
trigger card stays current throughout the run.
The handler manages its own progress reporting (it sets
``_manages_own_progress: True`` in the result) so the engine doesn't
overwrite the live phase string with a generic 'completed' label.
"""
from __future__ import annotations
import time
from typing import Any, Dict
from core.automation.deps import AutomationDeps
# Outer poll cap — covers extreme worst case (long Plex scans on
# huge libraries). Past this point we surface a clear timeout error
# so users notice rather than letting the trigger hang forever.
_SCAN_TIMEOUT_SECONDS = 1800
# Per-phase poll intervals.
_POLL_SCHEDULED_SECONDS = 2
_POLL_SCANNING_SECONDS = 5
_POLL_UNKNOWN_SECONDS = 2
# Progress percentage waypoints.
_PROGRESS_SCHEDULED_MAX = 14
_PROGRESS_SCAN_START = 15
_PROGRESS_SCAN_MAX = 95
def auto_scan_library(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a media-server library scan and stream progress to the
trigger card.
Returns one of:
- ``{'status': 'completed', '_manages_own_progress': True, ...}``
- ``{'status': 'skipped', 'reason': 'Scan already being tracked'}``
- ``{'status': 'error', 'reason': '...', '_manages_own_progress': True}``
"""
automation_id = config.get('_automation_id')
if not deps.web_scan_manager:
return {'status': 'error', 'reason': 'Scan manager not available'}
# If another automation is already tracking the scan, just forward
# the request — the original tracker keeps emitting progress.
if deps.state.is_scan_library_active():
deps.web_scan_manager.request_scan('Automation trigger (additional batch)')
return {'status': 'skipped', 'reason': 'Scan already being tracked'}
deps.state.set_scan_library_id(automation_id)
try:
result = deps.web_scan_manager.request_scan('Automation trigger')
scan_status_val = result.get('status', 'unknown')
if scan_status_val == 'queued':
deps.update_progress(
automation_id,
log_line='Scan already in progress — waiting for completion',
log_type='info',
)
else:
delay = result.get('delay_seconds', 60)
deps.update_progress(
automation_id,
log_line=f'Scan scheduled (debounce: {delay}s)',
log_type='info',
)
# Unified polling loop — handles debounce → scanning → idle.
poll_start = time.time()
scan_started = (scan_status_val == 'queued')
while time.time() - poll_start < _SCAN_TIMEOUT_SECONDS:
status = deps.web_scan_manager.get_scan_status()
st = status.get('status')
if st == 'idle':
break # Scan completed (or finished before we polled)
if st == 'scheduled':
elapsed = int(time.time() - poll_start)
deps.update_progress(
automation_id,
phase=f'Waiting for scan to start... ({elapsed}s)',
progress=min(int(elapsed / 60 * 10), _PROGRESS_SCHEDULED_MAX),
)
time.sleep(_POLL_SCHEDULED_SECONDS)
continue
if st == 'scanning':
if not scan_started:
scan_started = True
deps.update_progress(
automation_id,
progress=_PROGRESS_SCAN_START,
log_line='Scan triggered on media server',
log_type='success',
)
elapsed = status.get('elapsed_seconds', 0)
max_time = status.get('max_time_seconds', 300)
pct = min(_PROGRESS_SCAN_START + int(elapsed / max_time * 80), _PROGRESS_SCAN_MAX)
mins, secs = divmod(elapsed, 60)
deps.update_progress(
automation_id,
phase=f'Library scan in progress... ({mins}m {secs}s)',
progress=pct,
)
time.sleep(_POLL_SCANNING_SECONDS)
continue
time.sleep(_POLL_UNKNOWN_SECONDS)
else:
# 30-min timeout reached
deps.update_progress(
automation_id,
status='error',
phase='Timed out',
log_line='Library scan timed out after 30 minutes',
log_type='error',
)
return {'status': 'error', 'reason': 'Timed out', '_manages_own_progress': True}
elapsed = round(time.time() - poll_start, 1)
deps.update_progress(
automation_id,
status='finished',
progress=100,
phase='Complete',
log_line='Library scan completed',
log_type='success',
)
return {
'status': 'completed',
'_manages_own_progress': True,
'scan_duration_seconds': elapsed,
}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
deps.update_progress(
automation_id,
status='error',
phase='Error',
log_line=str(e),
log_type='error',
)
return {'status': 'error', 'error': str(e), '_manages_own_progress': True}
finally:
deps.state.set_scan_library_id(None)

46
core/automation/handlers/scan_watchlist.py
View file

@ -1,46 +0,0 @@
"""Automation handler: ``scan_watchlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_scan_watchlist`` closure). The watchlist scanner returns
summary stats for the trigger card only when a fresh scan actually
ran — detected by snapshotting ``id(state_dict)`` before/after, since
the live processor reassigns the dict on each new scan.
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_scan_watchlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Run a watchlist scan when the automation triggers.
Pre-scan we capture ``id(watchlist_scan_state)`` so we can tell
afterwards whether the worker ran (and reassigned the state dict)
or short-circuited (kept the same dict). Only fresh scans report
summary stats — repeat triggers without an intervening run return
a bare ``completed``.
"""
try:
pre_state = deps.get_watchlist_scan_state()
pre_state_id = id(pre_state)
deps.process_watchlist_scan_automatically(
automation_id=config.get('_automation_id'),
profile_id=config.get('_profile_id'),
)
post_state = deps.get_watchlist_scan_state()
# Fresh scan = state dict was reassigned mid-run.
if id(post_state) != pre_state_id:
summary = post_state.get('summary', {}) if isinstance(post_state, dict) else {}
return {
'status': 'completed',
'artists_scanned': summary.get('total_artists', 0),
'successful_scans': summary.get('successful_scans', 0),
'new_tracks_found': summary.get('new_tracks_found', 0),
'tracks_added_to_wishlist': summary.get('tracks_added_to_wishlist', 0),
}
return {'status': 'completed'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise into the engine
return {'status': 'error', 'error': str(e)}

57
core/automation/handlers/search_and_download.py
View file

@ -1,57 +0,0 @@
"""Automation handler: ``search_and_download`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_search_and_download`` closure). Searches for a track by
name/artist string and dispatches the best match through the
download orchestrator. Query can come from the trigger config
(direct value) or from event data (e.g. webhook payload).
"""
from __future__ import annotations
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_search_and_download(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
automation_id = config.get('_automation_id')
query = config.get('query', '').strip()
# Event-triggered: pull query from event data (e.g. webhook_received).
if not query:
event_data = config.get('_event_data', {})
query = (event_data.get('query', '') or '').strip()
if not query:
if automation_id:
deps.update_progress(
automation_id, log_line='No search query provided', log_type='error',
)
return {'status': 'error', 'error': 'No search query provided'}
try:
if automation_id:
deps.update_progress(
automation_id, phase='Searching',
log_line=f'Searching: {query}', log_type='info',
)
result = deps.run_async(deps.download_orchestrator.search_and_download_best(query))
if result:
if automation_id:
deps.update_progress(
automation_id,
log_line=f'Download started for: {query}',
log_type='success',
)
return {'status': 'completed', 'query': query, 'download_id': result}
if automation_id:
deps.update_progress(
automation_id,
log_line=f'No match found for: {query}',
log_type='warning',
)
return {'status': 'not_found', 'query': query, 'error': 'No match found'}
except Exception as e: # noqa: BLE001 — automation handlers must never raise
if automation_id:
deps.update_progress(
automation_id, log_line=f'Error: {e}', log_type='error',
)
return {'status': 'error', 'query': query, 'error': str(e)}

240
core/automation/handlers/sync_playlist.py
View file

@ -1,240 +0,0 @@
"""Automation handler: ``sync_playlist`` action.
Lifted from ``web_server._register_automation_handlers`` (the
``_auto_sync_playlist`` closure). Syncs a mirrored playlist to the
configured media server, using discovered metadata when available
and skipping undiscovered tracks. When triggered on a schedule with
no track changes since the last sync, short-circuits with
``status: skipped`` (saves Plex / Jellyfin / Navidrome from
needless rewrites)."""
from __future__ import annotations
import hashlib
import json
import threading
from typing import Any, Dict
from core.automation.deps import AutomationDeps
def auto_sync_playlist(config: Dict[str, Any], deps: AutomationDeps) -> Dict[str, Any]:
"""Sync a mirrored playlist to the active media server.
Behavior:
- Tracks with discovered metadata (extra_data.discovered + matched_data)
are routed via the official metadata.
- Tracks with a Spotify hint (real Spotify ID from the embed
scraper) are included so they can still hit Soulseek + the
wishlist.
- Tracks with neither are counted as ``skipped_tracks``.
- Empty result → ``status: skipped`` with the skipped count.
- Same track set as last sync (matched_tracks unchanged) →
``status: skipped`` (no-op).
- Otherwise spawns a daemon thread running ``run_sync_task`` and
returns ``status: started`` with ``_manages_own_progress: True``.
"""
auto_id = config.get('_automation_id')
playlist_id = config.get('playlist_id')
if not playlist_id:
return {'status': 'error', 'reason': 'No playlist specified'}
db = deps.get_database()
pl = db.get_mirrored_playlist(int(playlist_id))
if not pl:
return {'status': 'error', 'reason': 'Playlist not found'}
tracks = db.get_mirrored_playlist_tracks(int(playlist_id))
if not tracks:
return {'status': 'error', 'reason': 'No tracks in playlist'}
# Convert mirrored tracks to format expected by run_sync_task.
# Use discovered metadata when available, fall back to Spotify
# hint or raw playlist fields when not.
tracks_json = []
skipped_count = 0
for t in tracks:
# Parse extra_data for discovery info.
extra = {}
if t.get('extra_data'):
try:
extra = json.loads(t['extra_data']) if isinstance(t['extra_data'], str) else t['extra_data']
except (json.JSONDecodeError, TypeError):
pass
if extra.get('discovered') and extra.get('matched_data'):
# Use official discovered metadata.
md = extra['matched_data']
album_raw = md.get('album', '')
album_obj = album_raw if isinstance(album_raw, dict) else {'name': album_raw or ''}
_track_entry = {
'name': md.get('name', ''),
'artists': md.get('artists', [{'name': t.get('artist_name', '')}]),
'album': album_obj,
'duration_ms': md.get('duration_ms', 0),
'id': md.get('id', ''),
}
if md.get('track_number'):
_track_entry['track_number'] = md['track_number']
if md.get('disc_number'):
_track_entry['disc_number'] = md['disc_number']
tracks_json.append(_track_entry)
else:
# NOT discovered — try to include using available metadata so
# the track can still be searched on Soulseek and added to
# wishlist. Without this, failed discovery blocks the entire
# download pipeline.
#
# Priority: spotify_hint (has real Spotify ID from embed
# scraper) > raw playlist fields (only if source_track_id
# is valid).
hint = extra.get('spotify_hint', {})
# Build album object with cover art from the mirrored playlist track.
track_image = (t.get('image_url') or '').strip()
album_obj = {
'name': (t.get('album_name') or '').strip(),
'images': [{'url': track_image, 'height': 300, 'width': 300}] if track_image else [],
}
if hint.get('id') and hint.get('name'):
# spotify_hint has proper Spotify track ID + metadata from embed scraper.
hint_artists = hint.get('artists', [])
if hint_artists and isinstance(hint_artists[0], str):
hint_artists = [{'name': a} for a in hint_artists]
elif hint_artists and isinstance(hint_artists[0], dict):
pass # Already in correct format
else:
hint_artists = [{'name': t.get('artist_name', '')}]
tracks_json.append({
'name': hint['name'],
'artists': hint_artists,
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': hint['id'],
})
elif t.get('source_track_id') and (t.get('track_name') or '').strip():
# Has a valid source ID and track name — usable for wishlist.
tracks_json.append({
'name': t['track_name'].strip(),
'artists': [{'name': (t.get('artist_name') or '').strip() or 'Unknown Artist'}],
'album': album_obj,
'duration_ms': t.get('duration_ms', 0),
'id': t['source_track_id'],
})
else:
skipped_count += 1 # No usable ID or name — truly can't process.
if not tracks_json:
deps.update_progress(
auto_id,
log_line=f'No discovered tracks — {skipped_count} need discovery first',
log_type='skip',
)
return {
'status': 'skipped',
'reason': f'No discovered tracks to sync ({skipped_count} tracks need discovery first)',
'skipped_tracks': str(skipped_count),
}
# Preflight: hash the track list and compare against last sync.
# Skip if the exact same set of tracks was already synced and
# everything matched (no-op preserves Plex / Jellyfin / Navidrome
# from needless rewrites).
track_ids_str = ','.join(sorted(t.get('id', '') for t in tracks_json))
tracks_hash = hashlib.md5(track_ids_str.encode()).hexdigest()
sync_id_key = f"auto_mirror_{playlist_id}"
# Full mirror identity (every source_track_id on the playlist). tracks_hash
# only covers tracks_json — if a new mirror row is skipped (no discovery /
# no source id), tracks_hash stays identical to the pre-add sync and we
# used to no-op with "unchanged" while the new song never hit wishlist.
mirror_ids_str = ','.join(
sorted(t.get('source_track_id', '') or '' for t in tracks if t.get('source_track_id'))
)
mirror_tracks_hash = hashlib.md5(mirror_ids_str.encode()).hexdigest() if mirror_ids_str else ''
event_data = config.get('_event_data') or {}
try:
tracks_added = int(event_data.get('added') or 0)
except (TypeError, ValueError):
tracks_added = 0
force_sync = tracks_added > 0 or skipped_count > 0
try:
sync_statuses = deps.load_sync_status_file()
last_status = sync_statuses.get(sync_id_key, {})
last_hash = last_status.get('tracks_hash', '')
last_mirror_hash = last_status.get('mirror_tracks_hash', '')
last_matched = last_status.get('matched_tracks', -1)
mirror_changed = bool(mirror_tracks_hash) and mirror_tracks_hash != last_mirror_hash
if (
not force_sync
and not mirror_changed
and last_hash == tracks_hash
and last_matched >= len(tracks_json)
):
# Exact same tracks, all matched last time — nothing to do.
deps.update_progress(
auto_id,
log_line=f'All {len(tracks_json)} tracks unchanged since last sync — skipping',
log_type='skip',
)
return {
'status': 'skipped',
'reason': f'All {len(tracks_json)} tracks unchanged since last sync',
}
if force_sync and last_hash == tracks_hash and last_matched >= len(tracks_json):
deps.update_progress(
auto_id,
log_line=(
f'Forcing sync: playlist changed ({tracks_added} added) or '
f'{skipped_count} track(s) need discovery'
),
log_type='info',
)
elif mirror_changed:
deps.update_progress(
auto_id,
log_line='Mirror track list changed — running sync',
log_type='info',
)
except Exception as e:
deps.logger.debug("mirror sync last-status read: %s", e)
# Sync under the user's custom alias when set, else the upstream name (#865
# follow-up). The server-side playlist is named with this.
from core.playlists.naming import effective_mirrored_name
sync_name = effective_mirrored_name(pl) or pl.get('name') or 'Playlist'
deps.update_progress(
auto_id,
progress=50,
phase=f'Syncing "{sync_name}"',
log_line=f'{len(tracks_json)} discovered, {skipped_count} skipped',
log_type='info',
)
sync_id = f"auto_mirror_{playlist_id}"
deps.update_progress(
auto_id,
progress=90,
log_line=f'Starting sync: {len(tracks_json)} tracks',
log_type='success',
)
skip_wishlist_add = bool(pl.get('organize_by_playlist'))
threading.Thread(
target=deps.run_sync_task,
args=(sync_id, sync_name, tracks_json, auto_id, 1, pl.get('image_url', '')),
kwargs={'skip_wishlist_add': skip_wishlist_add},
daemon=True,
name=f'auto-sync-{playlist_id}',
).start()
return {
'status': 'started',
'playlist_name': sync_name,
'discovered_tracks': str(len(tracks_json)),
'skipped_tracks': str(skipped_count),
'_manages_own_progress': True,
}

216
core/automation/progress.py
View file

@ -1,216 +0,0 @@
"""Automation progress tracking.
Owns the in-memory progress state dict that backs both
`/api/automations/progress` polling and the WebSocket
`automation:progress` push emitter. State is per-automation, capped at
50 log entries each, and finished/error states are reaped 60s after
they finish so the frontend has a window to show the final state.
Functions are written so the route layer / engine callbacks can pass
their own socketio emitter, db handle, and shutdown flag. The progress
state dict (`progress_states`) and its lock (`progress_lock`) are
module-level so all callers share one view — same as the original
web_server.py globals.
"""
from __future__ import annotations
import json
import logging
import threading
from datetime import datetime, timezone
from typing import Any, Callable, Optional
logger = logging.getLogger(__name__)
# Shared mutable state — module globals so every caller (routes, engine
# progress callbacks, emit loop) sees the same dict. Mirrors the original
# `automation_progress_states` / `automation_progress_lock` in web_server.
progress_states: dict[int, dict] = {}
progress_lock = threading.Lock()
def init_progress(automation_id: int, automation_name: str, action_type: str) -> None:
"""Initialize progress state when an automation starts running."""
with progress_lock:
progress_states[automation_id] = {
'status': 'running',
'action_type': action_type,
'progress': 0,
'phase': 'Starting...',
'current_item': '',
'processed': 0,
'total': 0,
'log': [{'type': 'info', 'text': f'Starting {automation_name}'}],
'started_at': datetime.now(timezone.utc).isoformat(),
'finished_at': None,
}
def update_progress(
automation_id: Optional[int],
*,
socketio_emit: Optional[Callable[[str, Any], None]] = None,
**kwargs,
) -> None:
"""Update progress state from handler threads. Thread-safe.
`socketio_emit` lets callers wire in the live socketio.emit so that
finished/error transitions push immediately without waiting for the
1s emitter loop. Falls back to no-op if not provided.
"""
if automation_id is None:
return
with progress_lock:
state = progress_states.get(automation_id)
if not state:
return
for k, v in kwargs.items():
if k == 'log_line':
state['log'].append({'type': kwargs.get('log_type', 'info'), 'text': v})
if len(state['log']) > 50:
state['log'] = state['log'][-50:]
elif k != 'log_type':
state[k] = v
if kwargs.get('status') in ('finished', 'error'):
state['finished_at'] = datetime.now(timezone.utc).isoformat()
if socketio_emit is not None:
try:
socketio_emit('automation:progress', {str(automation_id): dict(state)})
except Exception as e:
logger.debug("socketio progress emit: %s", e)
def get_running_progress() -> dict[str, dict]:
"""Snapshot of running/finished/error states for the polling endpoint."""
with progress_lock:
result: dict[str, dict] = {}
for aid, state in progress_states.items():
if state['status'] in ('running', 'finished', 'error'):
cp = dict(state)
cp['log'] = list(state['log'])
result[str(aid)] = cp
return result
def record_history(
automation_id: int,
result: dict,
database,
) -> None:
"""Capture progress state into run history before cleanup clears it.
`database` is passed in so the function works without a `get_database()`
global.
"""
try:
with progress_lock:
state = progress_states.get(automation_id)
if state:
started_at = state.get('started_at')
finished_at = state.get('finished_at') or datetime.now(timezone.utc).isoformat()
log_entries = list(state.get('log', []))
else:
started_at = datetime.now(timezone.utc).isoformat()
finished_at = datetime.now(timezone.utc).isoformat()
log_entries = []
duration = None
if started_at and finished_at:
try:
t0 = datetime.fromisoformat(started_at)
t1 = datetime.fromisoformat(finished_at)
duration = (t1 - t0).total_seconds()
except Exception as e:
logger.debug("duration parse: %s", e)
r_status = result.get('status', 'completed') if result else 'completed'
if r_status == 'error':
status = 'error'
elif r_status == 'skipped':
status = 'skipped'
elif r_status == 'timeout':
status = 'timeout'
else:
status = 'completed'
summary = None
for entry in reversed(log_entries):
if entry.get('type') in ('success', 'error'):
summary = entry.get('text', '')
break
if not summary and log_entries:
summary = log_entries[-1].get('text', '')
if not summary and result:
summary = result.get('reason') or result.get('error') or result.get('status', '')
result_json = json.dumps({k: v for k, v in result.items() if not k.startswith('_')}) if result else None
log_json = json.dumps(log_entries) if log_entries else None
database.insert_automation_run_history(
automation_id=automation_id,
started_at=started_at,
finished_at=finished_at,
duration_seconds=duration,
status=status,
summary=summary,
result_json=result_json,
log_lines=log_json,
)
except Exception as e:
logger.error(f"Error recording automation history for {automation_id}: {e}")
def emit_progress_loop(
socketio,
*,
is_shutting_down: Callable[[], bool],
poll_interval: float = 1.0,
timeout_seconds: int = 7200,
cleanup_after_seconds: int = 60,
) -> None:
"""Push `automation:progress` events for active automations.
Long-running loop — caller wires this into a socketio background task.
- Times out zombie running states after `timeout_seconds` (default 2h).
- Reaps finished/error states `cleanup_after_seconds` after finish so the
frontend has a final-state window before they disappear.
"""
while not is_shutting_down():
socketio.sleep(poll_interval)
try:
with progress_lock:
active: dict[str, dict] = {}
stale: list[int] = []
now = datetime.now()
for aid, state in progress_states.items():
if state['status'] == 'running':
try:
started = datetime.fromisoformat(state.get('started_at', ''))
if (now - started).total_seconds() > timeout_seconds:
state['status'] = 'error'
state['phase'] = 'Timed out'
state['finished_at'] = now.isoformat()
state['log'].append({'type': 'error', 'text': f'Timed out after {timeout_seconds // 3600} hours'})
cp = dict(state)
cp['log'] = list(state['log'])
active[str(aid)] = cp
continue
except (ValueError, TypeError):
pass
cp = dict(state)
cp['log'] = list(state['log'])
active[str(aid)] = cp
elif state['status'] in ('finished', 'error') and state.get('finished_at'):
try:
finished_time = datetime.fromisoformat(state['finished_at'])
if (now - finished_time).total_seconds() > cleanup_after_seconds:
stale.append(aid)
except (ValueError, TypeError):
stale.append(aid)
for aid in stale:
del progress_states[aid]
if active:
socketio.emit('automation:progress', active)
except Exception as e:
logger.debug(f"Error emitting automation progress: {e}")

321
core/automation/schedule.py
View file

@ -1,321 +0,0 @@
"""Pure functions for computing the next-run datetime of a scheduled
automation trigger.
The Auto-Sync schedule board currently exposes interval-based scheduling
(``every N hours``) backed by ``trigger_type='schedule'``. The
automation engine ALSO supports ``daily_time`` and ``weekly_time``
triggers via separate ``_setup_*_trigger`` methods inline on the engine
class. None of that logic is currently testable in isolation — the
engine's ``_finish_run`` reaches for ``datetime.now()``, threads it
through ``_next_weekly_occurrence``, and writes the result to the DB,
all on the same call.
This module lifts the "given a trigger config, what's the next run?"
question out of the engine into a pure function:
next_run_at(trigger_type, trigger_config, now_utc, default_tz)
-> Optional[datetime]
That means:
- ``now_utc`` is INJECTED, not pulled from the system clock. Tests
freeze time without monkeypatching ``datetime.now``.
- ``default_tz`` is INJECTED. Daily / weekly / monthly schedules are
inherently in the USER'S timezone (cron "every Monday at 9am" is
not UTC), and the historic engine implicitly used the server's
local tz via naive ``datetime.now()``. That broke for users on a
different tz than their server. The pure function takes the tz
explicitly so the caller controls it.
- Returns an aware UTC ``datetime`` ready to serialise to the DB's
``next_run`` string column, or ``None`` for unrecognised /
event-based triggers (engine should not store a next_run for those).
PR 1 of the schedule-types feature ships ONLY this module + tests.
The engine continues to compute next_run via its existing inline
helpers; PR 2 collapses those into a single ``next_run_at`` call.
Net behavior is identical until the engine is wired through — this
PR is pure plumbing.
Schedule types supported here:
- ``schedule`` (interval): ``{interval: N, unit: 'minutes'|'hours'|'days'}``
— adds the interval to ``now_utc``; no tz needed.
- ``daily_time``: ``{time: 'HH:MM', tz: '<IANA>'}`` — runs every day at
the given local time in the given timezone. ``tz`` falls back to
``default_tz`` when absent.
- ``weekly_time``: ``{time: 'HH:MM', days: ['mon','wed',...], tz: '<IANA>'}``
— runs on the matching weekday(s) at the given local time. Empty
``days`` list means "every day" (matches the engine's existing
fallback in ``_next_weekly_occurrence``).
- ``monthly_time``: ``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}``
— runs on the given day each month. Days that don't exist in a
given month (Feb 30, Apr 31) clamp to the LAST valid day of that
month rather than skipping the run entirely; missing a whole
month silently because the schedule was over-eager is worse than
running a day early.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, Optional
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from utils.logging_config import get_logger
logger = get_logger("automation.schedule")
# Unknown-tz names already warned about in this process — avoids
# spamming the log on every poll cycle for the same misconfigured row.
_UNKNOWN_TZ_WARNED: set = set()
# Weekday abbreviation → ``datetime.weekday()`` index (Mon=0..Sun=6).
# Mirrors the engine's existing ``_next_weekly_occurrence`` mapping so
# schedules created against either implementation accept the same
# ``days`` strings.
_WEEKDAY_MAP = {
'mon': 0, 'tue': 1, 'wed': 2, 'thu': 3, 'fri': 4, 'sat': 5, 'sun': 6,
}
# Interval multipliers — kept aligned with the engine's existing
# ``_calc_delay_seconds`` in ``core/automation_engine.py``. Adding
# entries here without also updating the engine would silently drift:
# this function would honour the new unit while the live engine path
# defaults it to hours. Keep the maps in sync until PR 2 collapses the
# engine through this function.
_INTERVAL_MULTIPLIERS = {
'minutes': 60,
'hours': 60 * 60,
'days': 60 * 60 * 24,
}
def next_run_at(
trigger_type: str,
trigger_config: Dict[str, Any],
now_utc: datetime,
default_tz: str = 'UTC',
) -> Optional[datetime]:
"""Compute the next-run timestamp (UTC, aware) for a scheduled
trigger. Returns ``None`` for unrecognised types or event-based
triggers — callers should not write a next_run for those.
See module docstring for supported trigger types + config shapes.
"""
if not isinstance(trigger_config, dict):
trigger_config = {}
if trigger_type == 'schedule':
return _next_interval(trigger_config, now_utc)
if trigger_type == 'daily_time':
return _next_daily(trigger_config, now_utc, default_tz)
if trigger_type == 'weekly_time':
return _next_weekly(trigger_config, now_utc, default_tz)
if trigger_type == 'monthly_time':
return _next_monthly(trigger_config, now_utc, default_tz)
return None
# ---------------------------------------------------------------------------
# Interval
# ---------------------------------------------------------------------------
def _next_interval(config: Dict[str, Any], now_utc: datetime) -> datetime:
"""``{interval: N, unit: 'hours'}`` → ``now_utc + N hours``.
Mirrors the engine's existing ``_calc_delay_seconds``. Unit defaults
to ``hours`` for backward compat with legacy DB rows that pre-date
the unit field being mandatory; interval defaults to 1 so a fully
empty config doesn't divide-by-zero or schedule for the past."""
try:
interval = max(int(config.get('interval', 1)), 1)
except (TypeError, ValueError):
interval = 1
unit = config.get('unit') or 'hours'
seconds = interval * _INTERVAL_MULTIPLIERS.get(unit, _INTERVAL_MULTIPLIERS['hours'])
return _ensure_utc(now_utc) + timedelta(seconds=seconds)
# ---------------------------------------------------------------------------
# Daily
# ---------------------------------------------------------------------------
def _next_daily(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', tz: '<IANA>'}`` → next occurrence of that
wall-clock time in the user's timezone, expressed as aware UTC.
DST-aware via ``zoneinfo``: when the local time falls during a
spring-forward gap, the ``replace`` lands on a non-existent
instant; ``zoneinfo`` resolves that to the gap's later side
(e.g. 02:30 on the DST-forward day becomes 03:30 local). Tests
pin both spring-forward and fall-back behaviour."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
now_local = _ensure_utc(now_utc).astimezone(tz)
target_local = now_local.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target_local <= now_local:
target_local = target_local + timedelta(days=1)
return target_local.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Weekly
# ---------------------------------------------------------------------------
def _next_weekly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', days: ['mon',...], tz: '<IANA>'}`` → next
occurrence of that wall-clock time on any of the listed weekdays
in the user's timezone.
Empty ``days`` list ≡ every day, matching the engine's existing
fallback. Unrecognised day abbreviations are silently dropped
(an empty result-set then triggers the every-day fallback)."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
days = _parse_weekdays(config.get('days'))
now_local = _ensure_utc(now_utc).astimezone(tz)
# Scan today + next 7 days; the matching day with a future
# local time wins. 8-day scan is enough to handle the case where
# today already passed the time AND today is the only allowed
# weekday (next occurrence is exactly one week out).
for offset in range(8):
candidate = now_local + timedelta(days=offset)
if candidate.weekday() not in days:
continue
target = candidate.replace(hour=hour, minute=minute, second=0, microsecond=0)
if target > now_local:
return target.astimezone(timezone.utc)
# Shouldn't reach: 8-day scan always finds a hit when ``days``
# is non-empty. Defensive fallback: next week, same weekday as today.
fallback = (now_local + timedelta(days=7)).replace(
hour=hour, minute=minute, second=0, microsecond=0,
)
return fallback.astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Monthly
# ---------------------------------------------------------------------------
def _next_monthly(
config: Dict[str, Any], now_utc: datetime, default_tz: str,
) -> datetime:
"""``{time: 'HH:MM', day_of_month: 1-31, tz: '<IANA>'}`` → next
occurrence in the user's timezone.
``day_of_month`` is clamped to ``[1, 31]``. When the target day
doesn't exist in a given month (Feb 30, Apr 31), the schedule
falls back to the LAST valid day of that month — running a day
or two early in short months is less surprising than skipping
a month entirely. This matches the convention every cron
implementation in the wild settled on."""
tz = _resolve_tz(config.get('tz') or default_tz)
hour, minute = _parse_hhmm(config.get('time'))
raw_day = config.get('day_of_month', 1)
try:
target_day = max(1, min(31, int(raw_day)))
except (TypeError, ValueError):
target_day = 1
now_local = _ensure_utc(now_utc).astimezone(tz)
# Try this month first; if the target day has already passed
# (or doesn't exist this month and the clamped day is in the
# past), advance to next month. Loop bounded to 12 iterations
# so a pathologically broken config can't infinite-loop us.
year, month = now_local.year, now_local.month
for _ in range(12):
day = min(target_day, _days_in_month(year, month))
target = now_local.replace(
year=year, month=month, day=day,
hour=hour, minute=minute, second=0, microsecond=0,
)
if target > now_local:
return target.astimezone(timezone.utc)
# Roll to next month.
if month == 12:
year, month = year + 1, 1
else:
month += 1
# Defensive — should be unreachable.
return (now_local + timedelta(days=30)).astimezone(timezone.utc)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _ensure_utc(dt: datetime) -> datetime:
"""Coerce a possibly-naive datetime to aware UTC. Naive inputs
are assumed UTC (matches the convention the engine uses when
parsing the DB ``next_run`` column)."""
if dt.tzinfo is None:
return dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def _resolve_tz(name: Optional[str]):
"""Look up an IANA tz by name. Falls back to UTC when the name is
unknown — ``ZoneInfoNotFoundError`` is the symptom of either a
typo in the tz string or ``tzdata`` missing on the host. Logged
once per unknown name so the user can see WHY their schedule
isn't running in the timezone they configured."""
if not name:
return timezone.utc
try:
return ZoneInfo(name)
except ZoneInfoNotFoundError:
if name not in _UNKNOWN_TZ_WARNED:
_UNKNOWN_TZ_WARNED.add(name)
logger.warning(
"Unknown timezone %r — schedule will run against UTC. "
"Check the spelling (IANA format like 'America/Los_Angeles') "
"or install the `tzdata` package on minimal hosts.",
name,
)
return timezone.utc
def _parse_hhmm(time_str: Optional[str]) -> tuple:
"""Parse ``HH:MM`` → ``(hour, minute)``. Defaults to 00:00 on
garbage input — same defensive shape as the engine's existing
daily/weekly time parsing."""
if not isinstance(time_str, str):
return 0, 0
try:
h, m = time_str.split(':', 1)
return max(0, min(23, int(h))), max(0, min(59, int(m)))
except (ValueError, AttributeError):
return 0, 0
def _parse_weekdays(days) -> set:
"""``['mon', 'wed']`` → ``{0, 2}``. Empty / missing / all-invalid
list returns ``set(range(7))`` ("every day"), matching the
engine's existing ``_next_weekly_occurrence`` fallback."""
if not isinstance(days, (list, tuple)):
return set(range(7))
parsed = {_WEEKDAY_MAP[d.lower()] for d in days
if isinstance(d, str) and d.lower() in _WEEKDAY_MAP}
return parsed or set(range(7))
def _days_in_month(year: int, month: int) -> int:
"""Last calendar day of ``year-month``. Stdlib-only — no calendar
module import needed; cycle through the 12 months."""
if month == 12:
next_first = datetime(year + 1, 1, 1)
else:
next_first = datetime(year, month + 1, 1)
last_day = next_first - timedelta(days=1)
return last_day.day

46
core/automation/signals.py
View file

@ -1,46 +0,0 @@
"""Automation signal helpers — name collection for autocomplete.
Signal cycle detection itself lives in core/automation_engine.py
(`detect_signal_cycles`); this module just enumerates known signal
names from the saved automation set so the builder UI can autocomplete.
"""
from __future__ import annotations
import json
import logging
logger = logging.getLogger(__name__)
def collect_known_signals(database) -> list[str]:
"""Return sorted, deduped signal names referenced by any saved automation.
Walks every automation and pulls signal names from both the
`signal_received` trigger config and any `fire_signal` then-actions.
Errors at every layer are swallowed — the autocomplete is best-effort.
"""
signals: set[str] = set()
try:
for auto in database.get_automations():
if auto.get('trigger_type') == 'signal_received':
try:
tc = json.loads(auto.get('trigger_config') or '{}')
sig = tc.get('signal_name', '').strip()
if sig:
signals.add(sig)
except (json.JSONDecodeError, TypeError):
pass
try:
ta = json.loads(auto.get('then_actions') or '[]')
for item in ta:
if item.get('type') == 'fire_signal':
sig = item.get('config', {}).get('signal_name', '').strip()
if sig:
signals.add(sig)
except (json.JSONDecodeError, TypeError):
pass
except Exception as e:
logger.debug("collect known signals failed: %s", e)
return sorted(signals)

1132
core/automation_engine.py
View file

File diff suppressed because it is too large Load diff

37
core/blocklist/__init__.py
View file

@ -1,37 +0,0 @@
"""Artist / album / track blocklist (the "proper" blacklist).
Distinct from ``download_blacklist`` (which skips one bad source file from one
Soulseek peer — untouched here). This blocklist bans an ARTIST, ALBUM, or
TRACK from being acquired, keyed by metadata-source IDs (Spotify / iTunes /
Deezer / MusicBrainz) so a ban survives a source switch.
Phase 1 enforces at the single ``add_to_wishlist`` chokepoint: every
auto-acquisition path (watchlist, discography backfill, repair, manual
wishlist add) funnels through it, so one guard covers them all.
- ``matching`` — the pure decision core (no DB, no I/O): build an index from
blocklist rows, ask whether a candidate is blocked, with artist→album→track
cascade.
"""
from core.blocklist.matching import (
ENTITY_ALBUM,
ENTITY_ARTIST,
ENTITY_TRACK,
ENTITY_TYPES,
SOURCE_ID_FIELDS,
BlocklistIndex,
build_index,
candidate_block_reason,
)
__all__ = [
"ENTITY_ARTIST",
"ENTITY_ALBUM",
"ENTITY_TRACK",
"ENTITY_TYPES",
"SOURCE_ID_FIELDS",
"BlocklistIndex",
"build_index",
"candidate_block_reason",
]

50
core/blocklist/backfill.py
View file

@ -1,50 +0,0 @@
"""Cross-source ID backfill for blocklist entries.
When a user blocks an item, the modal gives us the ID for ONE source (the one
they searched). For the ban to survive a source switch, we resolve the OTHER
sources' IDs too — matching the blocked artist/album/track by name on each
source and taking a confident hit.
The resolution is kept pure + injected so it tests without a network: callers
pass a ``resolvers`` map ``{source: fn(entity_type, name, parent_name) -> id |
None}``. ``core/blocklist/runtime.py`` wires the real metadata clients.
Honest about fragility (acknowledged in design): artist matching is reliable,
album/track cross-source matching is best-effort (editions, common titles), so
a resolver returning None just leaves that source unmatched — the artist
name-fallback in matching.py covers artist gaps; album/track gaps mean that
ban only applies on sources where an ID resolved.
"""
from __future__ import annotations
from typing import Any, Callable, Dict, Optional
from core.blocklist.matching import SOURCE_ID_FIELDS
def resolve_missing_ids(
entry: Dict[str, Any],
resolvers: Dict[str, Callable[..., Optional[str]]],
) -> Dict[str, str]:
"""Return ``{id_column: resolved_id}`` for the sources currently missing an
ID on ``entry``. Never raises — a resolver that errors is skipped."""
out: Dict[str, str] = {}
entity_type = entry.get("entity_type")
name = entry.get("name")
parent = entry.get("parent_name")
if not entity_type or not name:
return out
for source, col in SOURCE_ID_FIELDS.items():
if entry.get(col):
continue # already known
fn = resolvers.get(source)
if not fn:
continue
try:
rid = fn(entity_type, name, parent)
except Exception:
rid = None
if rid:
out[col] = str(rid)
return out

128
core/blocklist/matching.py
View file

@ -1,128 +0,0 @@
"""Pure blocklist matching — no DB, no I/O, fully unit-testable.
The brain of the blocklist: given the stored blocklist rows and a candidate
track being considered for the wishlist, decide whether it's blocked.
Design decisions (per Boulder):
- **ID-keyed.** Each row carries the candidate's IDs in up to four metadata
sources. A candidate is matched against the SAME source it came in on
(the wishlist payload carries active-source IDs), so a Deezer-numeric id
can't collide with an iTunes-numeric id of a different entity.
- **Cascade.** Blocking an artist blocks their albums + tracks; blocking an
album blocks its tracks. The candidate carries its own artist/album/track
IDs, so the check walks track → album → artist and blocks on the first hit.
- **Name fallback for ARTISTS only.** A blocked artist also matches by
case-folded name — this covers the window before the background ID-backfill
has resolved the active source's id. Albums/tracks do NOT fall back to name
(common titles like "Greatest Hits" would false-positive across artists);
they rely on IDs, which backfill fills in.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
ENTITY_ARTIST = "artist"
ENTITY_ALBUM = "album"
ENTITY_TRACK = "track"
ENTITY_TYPES = (ENTITY_ARTIST, ENTITY_ALBUM, ENTITY_TRACK)
# Blocklist-row column → the metadata source it belongs to.
SOURCE_ID_FIELDS = {
"spotify": "spotify_id",
"itunes": "itunes_id",
"deezer": "deezer_id",
"musicbrainz": "musicbrainz_id",
}
def _norm(text: Any) -> str:
return str(text or "").strip().casefold()
@dataclass
class _TypeIndex:
# per-source set of blocked ids, plus a case-folded name set (artists only)
ids: Dict[str, Set[str]] = field(default_factory=lambda: {s: set() for s in SOURCE_ID_FIELDS})
names: Set[str] = field(default_factory=set)
def hit(self, source: Optional[str], entity_id: Any, name: Any, use_name: bool) -> bool:
if entity_id and source in self.ids and str(entity_id) in self.ids[source]:
return True
if use_name and name and _norm(name) in self.names:
return True
return False
@dataclass
class BlocklistIndex:
"""Membership index built once per scan from the blocklist rows."""
artists: _TypeIndex = field(default_factory=_TypeIndex)
albums: _TypeIndex = field(default_factory=_TypeIndex)
tracks: _TypeIndex = field(default_factory=_TypeIndex)
@property
def is_empty(self) -> bool:
for ti in (self.artists, self.albums, self.tracks):
if ti.names or any(ti.ids.values()):
return False
return True
def build_index(rows: Iterable[Dict[str, Any]]) -> BlocklistIndex:
"""Build a BlocklistIndex from blocklist DB rows.
Each row needs ``entity_type``, ``name``, and the source id columns
(``spotify_id`` / ``itunes_id`` / ``deezer_id`` / ``musicbrainz_id``).
Unknown entity types are ignored."""
idx = BlocklistIndex()
by_type = {ENTITY_ARTIST: idx.artists, ENTITY_ALBUM: idx.albums, ENTITY_TRACK: idx.tracks}
for row in rows or []:
ti = by_type.get((row.get("entity_type") or "").strip().lower())
if ti is None:
continue
for source, col in SOURCE_ID_FIELDS.items():
val = row.get(col)
if val:
ti.ids[source].add(str(val))
name = row.get("name")
if name:
ti.names.add(_norm(name))
return idx
def candidate_block_reason(
index: BlocklistIndex,
*,
source: Optional[str],
track_id: Any = None,
track_name: Any = None,
album_id: Any = None,
album_name: Any = None,
artists: Optional[List[Dict[str, Any]]] = None,
) -> Optional[Tuple[str, str]]:
"""Return ``(entity_type, label)`` for the first cascade hit, else None.
``source`` is the metadata source the candidate IDs came from (the wishlist
payload's provider). ``artists`` is a list of ``{'id', 'name'}`` dicts.
Order matters only for the returned reason — any hit blocks."""
if index.is_empty:
return None
# Track level — id only (names too ambiguous to ban across artists).
if index.tracks.hit(source, track_id, track_name, use_name=False):
return (ENTITY_TRACK, str(track_name or track_id or "track"))
# Album level — id only.
if index.albums.hit(source, album_id, album_name, use_name=False):
return (ENTITY_ALBUM, str(album_name or album_id or "album"))
# Artist level — id OR case-folded name (safe + covers the backfill window).
for artist in artists or []:
a_id = artist.get("id") if isinstance(artist, dict) else None
a_name = artist.get("name") if isinstance(artist, dict) else artist
if index.artists.hit(source, a_id, a_name, use_name=True):
return (ENTITY_ARTIST, str(a_name or a_id or "artist"))
return None

82
core/blocklist/runtime.py
View file

@ -1,82 +0,0 @@
"""Wire real metadata clients to the blocklist backfill resolvers.
Resolves a blocked item's ID on each metadata source by searching that source
for the name and taking a confidently name-matched hit. Confidence = exact
significant-token match (drops articles/punctuation) so we never hang a wrong
ID on an entry. Albums/tracks additionally require the parent artist to match
when both sides expose one.
"""
from __future__ import annotations
import re
from typing import Any, Callable, Dict, Optional
from utils.logging_config import get_logger
logger = get_logger("blocklist.runtime")
_STOP = {"the", "a", "an", "feat", "ft", "featuring", "with"}
def _tokens(text: Any) -> frozenset:
words = re.sub(r"[^a-z0-9]+", " ", str(text or "").lower()).split()
return frozenset(w for w in words if w not in _STOP)
def _name_of(obj: Any) -> str:
if isinstance(obj, dict):
return str(obj.get("name") or obj.get("title") or "")
return str(getattr(obj, "name", None) or getattr(obj, "title", None) or "")
def _id_of(obj: Any) -> Optional[str]:
val = obj.get("id") if isinstance(obj, dict) else getattr(obj, "id", None)
return str(val) if val else None
def _confident(result_name: str, want_name: str) -> bool:
rt, wt = _tokens(result_name), _tokens(want_name)
return bool(rt) and rt == wt
def _make_resolver(source: str) -> Callable[..., Optional[str]]:
def resolve(entity_type: str, name: str, parent_name: Optional[str] = None) -> Optional[str]:
from core.metadata.registry import get_client_for_source
client = get_client_for_source(source)
if not client:
return None
method = {
"artist": "search_artists",
"album": "search_albums",
"track": "search_tracks",
}.get(entity_type)
fn = getattr(client, method, None) if method else None
if not fn:
return None
try:
results = fn(name, limit=5) or []
except Exception as e:
logger.debug("%s %s search failed for %r: %s", source, entity_type, name, e)
return None
for r in results:
if not _confident(_name_of(r), name):
continue
# For album/track, also require the artist to line up when known.
if entity_type in ("album", "track") and parent_name:
artists = (r.get("artists") if isinstance(r, dict) else getattr(r, "artists", None)) or []
cand_artists = " ".join(
a.get("name", "") if isinstance(a, dict) else str(a) for a in artists)
if _tokens(parent_name) and not (_tokens(parent_name) & _tokens(cand_artists)):
continue
rid = _id_of(r)
if rid:
return rid
return None
return resolve
def build_resolvers() -> Dict[str, Callable[..., Optional[str]]]:
"""Source→resolver map for core.blocklist.backfill.resolve_missing_ids."""
return {s: _make_resolver(s) for s in ("spotify", "itunes", "deezer", "musicbrainz")}

27
core/boot_phase.py
View file

@ -1,27 +0,0 @@
"""Boot-phase guard for non-blocking container startup.
While the gunicorn worker is importing ``web_server`` (module-level client and
worker initialization), external provider API probes must not block startup.
Network validation is deferred until ``mark_boot_complete()`` runs at the end
of that import pass.
"""
from __future__ import annotations
import threading
_boot_lock = threading.Lock()
_boot_active = True
def is_boot_phase() -> bool:
"""Return True while module import must avoid blocking provider API calls."""
with _boot_lock:
return _boot_active
def mark_boot_complete() -> None:
"""End the boot phase — provider clients may perform network probes again."""
global _boot_active
with _boot_lock:
_boot_active = False

Some files were not shown because too many files have changed in this diff Show more