Update README.md

This commit is contained in:
Broque Thomas 2025-12-31 09:39:17 -08:00
parent b426d9d1e4
commit 8eb41a545c

575
README.md
View file

@ -2,53 +2,161 @@
<img src="./assets/trans.png" alt="SoulSync Logo"> <img src="./assets/trans.png" alt="SoulSync Logo">
</p> </p>
# 🎵 SoulSync - Automated Music Discovery & Collection Manager # SoulSync - Intelligent Music Discovery & Automation Platform
**Bridge streaming services to your local music library.** Automatically sync Spotify/Tidal/YouTube playlists to Plex/Jellyfin/Navidrome via Soulseek with intelligent matching, metadata enhancement, and automated discovery. **Bring Spotify-quality music discovery to your self-hosted library.** SoulSync automates music collection with intelligent discovery algorithms, multi-source downloads, and zero manual intervention.
> ⚠️ **CRITICAL**: Configure file sharing in slskd before use. Users who only download without sharing get banned by the Soulseek community. Set up shared folders at `http://localhost:5030/shares`. > **IMPORTANT**: Configure file sharing in slskd before use. The Soulseek community bans users who only download without sharing. Set up shared folders at `http://localhost:5030/shares`.
> 📢 **Development Focus**: New features are developed for the **Web UI** version. The Desktop GUI receives maintenance and bug fixes only. > **Development Status**: New features are developed for the **Web UI**. The Desktop GUI receives maintenance and bug fixes only.
## 💬 Community **Community**: [Discord Server](https://discord.gg/ePx7xYuV) | **Support**: [GitHub Issues](https://github.com/Nezreka/SoulSync/issues) | **Donate**: [Ko-fi](https://ko-fi.com/boulderbadgedad)
Join the Discord server for support, feature requests, and discussions: ---
- **Discord**: [https://discord.gg/ePx7xYuV](https://discord.gg/ePx7xYuV)
## ✨ Core Features ## The Problem
**Search & Download** You want Spotify's discovery features (Release Radar, Discovery Weekly, personalized playlists) but for your local music library. Existing tools require manual playlist management or lack intelligent discovery.
- **Enhanced Search**: Unified search across Spotify, your library, and Soulseek with categorized results (artists, albums, tracks)
- **Basic Search**: Direct Soulseek search with instant streaming and download
- Auto-sync playlists from Spotify/Tidal/YouTube to your media server
- Smart matching against your existing library
- FLAC-priority downloads from Soulseek with automatic fallback
- Customizable file organization with template-based path structures
- Synchronized lyrics (LRC) for every track via LRClib.net
**Metadata & Organization** ## The Solution
- Enhanced metadata with album art and proper tags
- Flexible folder templates: `$albumartist/$album/$track - $title`
- Automatic library scanning and database updates
- Clean, organized music collection
**Discovery & Automation** SoulSync bridges streaming services to your self-hosted media server with **automated discovery and collection**:
- Browse complete artist discographies with similar artist recommendations
- Intelligent music discovery using your watchlist ([music-map.com](https://music-map.com) integration)
- Curated playlists: Release Radar, Discovery Weekly, Seasonal Mixes
- Beatport chart integration for electronic music
- Artist watchlist monitors new releases automatically
**Management** 1. **Monitors artists** for new releases automatically
- Comprehensive library browser with search and completion tracking 2. **Generates personalized playlists** using custom recommendation algorithms
- Wishlist system with automatic retry (30-minute intervals) 3. **Downloads missing tracks** from Soulseek with FLAC priority
- Granular wishlist management (remove individual tracks or entire albums) 4. **Enriches metadata** with lyrics, album art, and proper tags
- Dynamic log level control (DEBUG/INFO/WARNING/ERROR) 5. **Organizes files** using customizable templates
- Background automation handles retries and database updates 6. **Syncs with Plex/Jellyfin/Navidrome** automatically
## 🚀 Installation ---
## Key Features
### Intelligent Discovery System
**Release Radar** - 30 new tracks from your watchlist, updated daily
- Monitors 100+ artists automatically
- Only includes releases from the last 30 days
- Curated using popularity scoring
**Discovery Weekly** - 50 tracks from similar artists you don't own
- Custom algorithm: 20 popular + 20 mid-tier + 10 deep cuts
- Built from 1000+ track discovery pool
- Updated every 24 hours
**Seasonal Playlists** - Auto-generated themed collections
- Halloween, Christmas, Valentine's Day, Summer, Spring, Autumn
- Smart keyword matching and genre analysis
- Active during appropriate months
**Personalized Playlists** (12+ types)
- Recently Added, Top Tracks, Forgotten Favorites
- Decade Playlists (1960s-2020s)
- Genre Playlists (15 parent categories, 50+ sub-genres)
- Daily Mixes, Hidden Gems, Popular Picks
- Custom Playlist Builder (seed with artists)
**ListenBrainz Integration**
- Import recommendation playlists
- Sync user and collaborative playlists
- Access community-curated content
**Beatport Integration** - Electronic music discovery
- Browse by genre (House, Techno, Trance, Drum & Bass, etc.)
- Top 100, Hype Charts, DJ Charts
- Staff Picks, New Releases, Latest Tracks
- Per-genre hero sections and featured content
### Multi-Source Downloads
**Primary Sources**
- **Soulseek**: FLAC-priority with automatic quality selection
- **Beatport Charts**: Electronic music with Spotify matching
- **Spotify Playlists**: Public and private playlist sync
- **Tidal Playlists**: Alternative streaming source
- **YouTube Playlists**: Fallback option
**Smart Download Pipeline**
- Quality profiles: Audiophile, Balanced, Mobile
- Automatic format fallback (FLAC → MP3 → other)
- Duplicate prevention against existing library
- Batch processing with concurrent downloads
- Automatic retry on failure (30-minute intervals)
### Advanced Matching Engine
**Text Normalization**
- Unicode handling (handles KoЯn, Björk, etc.)
- Special character preservation (A$AP Rocky)
- Accent normalization (Beyoncé → Beyonce)
- Abbreviation expansion (feat. → featured, pt. → part)
**Fuzzy Matching**
- Multi-strategy: exact → normalized → Unicode fallback
- Album variation handling (Deluxe, Remastered, Platinum Edition)
- Artist name preservation (doesn't break "Daryl Hall & John Oates")
- Confidence scoring with configurable thresholds
### Automation & Monitoring
**Watchlist System**
- Monitor unlimited artists for new releases
- Automatic similar artist discovery via music-map.com
- Occurrence-based ranking (tracks artist overlap)
- Configurable scan intervals
**Wishlist System**
- Tracks failed downloads automatically
- Auto-retry every 30 minutes
- Granular management (remove tracks or entire albums)
- Source tracking (playlist, album, manual)
**Background Tasks**
- Database synchronization with media server
- Discovery pool population (50 artists × 10 releases)
- Seasonal content updates
- Library completion tracking
### Library Management
**Tools**
- **Quality Scanner**: Find low-bitrate files to replace
- **Duplicate Cleaner**: Identify redundant tracks
- **Completion Tracking**: See album progress percentages
- **Enhanced Search**: Unified search across Spotify, library, and Soulseek
**Metadata Enhancement**
- Synchronized lyrics (LRC format) via LRClib.net
- Album art embedding
- Proper ID3/Vorbis tags
- Custom file organization templates
**File Organization**
- Template-based paths: `$albumartist/$album/$track - $title`
- Separate templates for albums, singles, playlists
- Client-side validation
- Automatic fallback on errors
### Media Server Integration
**Supported Servers**
- Plex (with library selection)
- Jellyfin (with multi-library support)
- Navidrome
**Features**
- Automatic library scanning after downloads
- Database caching for fast access
- Incremental updates
- Connection testing and validation
---
## Installation
### Docker (Recommended) ### Docker (Recommended)
```bash ```bash
# Using docker-compose # Using docker-compose
curl -O https://raw.githubusercontent.com/Nezreka/SoulSync/main/docker-compose.yml curl -O https://raw.githubusercontent.com/Nezreka/SoulSync/main/docker-compose.yml
@ -60,7 +168,8 @@ docker run -d -p 8008:8008 boulderbadgedad/soulsync:latest
# Access at http://localhost:8008 # Access at http://localhost:8008
``` ```
### Web UI (Python) ### Python (Web UI)
```bash ```bash
git clone https://github.com/Nezreka/SoulSync git clone https://github.com/Nezreka/SoulSync
cd SoulSync cd SoulSync
@ -70,6 +179,7 @@ python web_server.py
``` ```
### Desktop GUI ### Desktop GUI
```bash ```bash
git clone https://github.com/Nezreka/SoulSync git clone https://github.com/Nezreka/SoulSync
cd SoulSync cd SoulSync
@ -77,85 +187,55 @@ pip install -r requirements.txt
python main.py python main.py
``` ```
## ⚡ Quick Setup ---
## Quick Setup
### Prerequisites ### Prerequisites
- **slskd**: [Download](https://github.com/slskd/slskd/releases), run on port 5030
- **Spotify API**: Client ID/Secret from [Developer Dashboard](https://developer.spotify.com/dashboard) - **slskd** running on port 5030 ([Download](https://github.com/slskd/slskd/releases))
- **Tidal API** (optional): Client ID/Secret from [Developer Dashboard](https://developer.tidal.com/dashboard) - **Spotify API** credentials ([Developer Dashboard](https://developer.spotify.com/dashboard))
- **Tidal API** (optional) ([Developer Dashboard](https://developer.tidal.com/dashboard))
- **Media Server** (optional): Plex, Jellyfin, or Navidrome - **Media Server** (optional): Plex, Jellyfin, or Navidrome
### API Credentials ### API Configuration
**Spotify** **Spotify**
1. [Create app](https://developer.spotify.com/dashboard) → Settings 1. Create app at [Developer Dashboard](https://developer.spotify.com/dashboard)
2. Add redirect URI: `http://127.0.0.1:8888/callback` 2. Add redirect URI: `http://127.0.0.1:8888/callback`
3. Copy Client ID and Secret 3. Copy Client ID and Secret
**Tidal** **Tidal** (optional)
1. [Create app](https://developer.tidal.com/dashboard) 1. Create app at [Developer Dashboard](https://developer.tidal.com/dashboard)
2. Add redirect URI: `http://127.0.0.1:8889/callback` 2. Add redirect URI: `http://127.0.0.1:8889/callback`
3. Add scopes: `user.read`, `playlists.read` 3. Add scopes: `user.read`, `playlists.read`
4. Copy Client ID and Secret 4. Copy Client ID and Secret
**Plex** **Plex** (optional)
- Get token from any media item URL: `?X-Plex-Token=YOUR_TOKEN` - Get token from media item URL: `?X-Plex-Token=YOUR_TOKEN`
- Server URL: `http://YOUR_IP:32400` - Server URL: `http://YOUR_IP:32400`
**Jellyfin** **Jellyfin** (optional)
- Settings → API Keys → Generate new key - Settings → API Keys → Generate new key
- Server URL: `http://YOUR_IP:8096` - Server URL: `http://YOUR_IP:8096`
**Navidrome** **Navidrome** (optional)
- Settings → Users → Generate API Token - Settings → Users → Generate API Token
- Or use username/password
- Server URL: `http://YOUR_IP:4533` - Server URL: `http://YOUR_IP:4533`
### Configuration ### Initial Configuration
1. Launch SoulSync and go to Settings 1. Launch SoulSync and navigate to Settings
2. Enter API credentials for streaming services and media server 2. Enter API credentials for streaming services
3. Configure slskd URL (`http://localhost:5030`) and API key 3. Configure media server connection (if using)
4. Set download and transfer paths 4. Set slskd URL (`http://localhost:5030`) and API key
5. **Customize file organization** (optional): 5. Configure download path and transfer path
- Enable custom templates in Settings → File Organization 6. Customize file organization templates (optional)
- Default: `$albumartist/$albumartist - $album/$track - $title` 7. **Configure file sharing in slskd to avoid bans**
- Variables: `$artist`, `$albumartist`, `$album`, `$title`, `$track`, `$playlist`
- Example: `Music/$artist/$year - $album/$track - $title`
6. **Share files in slskd** to avoid bans
## 📁 File Organization ### Docker-Specific Setup
SoulSync supports customizable path templates with validation and fallback protection. **Path Mapping**
**Default Structure**
```
Transfer/
Artist/
Artist - Album/
01 - Track.flac
01 - Track.lrc
```
**Template System**
- **Albums**: `$albumartist/$albumartist - $album/$track - $title`
- **Singles**: `$artist/$artist - $title/$title`
- **Playlists**: `$playlist/$artist - $title`
**Available Variables**
- `$artist`, `$albumartist`, `$album`, `$title`
- `$track` (zero-padded: 01, 02...)
- `$playlist` (playlist name)
**Features**
- Client-side validation prevents invalid templates
- Reset to defaults button in settings
- Automatic fallback if template fails
- Changes apply immediately to new downloads
## 🐳 Docker Notes
**Path Configuration**
```yaml ```yaml
volumes: volumes:
- ./config:/app/config # Settings persist - ./config:/app/config # Settings persist
@ -166,69 +246,308 @@ volumes:
Use `/host/mnt/X/path` in settings where X is your drive letter. Use `/host/mnt/X/path` in settings where X is your drive letter.
**OAuth from Remote Devices** **OAuth Authentication from Remote Devices**
When accessing from a different machine, Spotify redirects may fail:
1. Complete OAuth flow - get redirected to `http://127.0.0.1:8888/callback?code=...` Due to Spotify API requirements (127.0.0.1 mandatory, localhost banned), remote OAuth needs a workaround:
2. Edit URL to use your server IP: `http://192.168.1.5:8888/callback?code=...`
1. Complete OAuth flow - redirected to `http://127.0.0.1:8888/callback?code=...`
2. Manually edit URL to your server IP: `http://192.168.1.5:8888/callback?code=...`
3. Press Enter to complete authentication 3. Press Enter to complete authentication
See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) for details. See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md) for details.
## 📊 Workflow ---
1. **Sync**: Select Spotify/Tidal/YouTube playlist ## How It Works
2. **Match**: SoulSync compares against your library
3. **Download**: Missing tracks queued from Soulseek
4. **Process**: Files enhanced with metadata, lyrics, and album art
5. **Organize**: Moved to transfer folder with template-based structure
6. **Scan**: Media server automatically rescans library
7. **Update**: SoulSync database syncs with your collection
## 🐛 Troubleshooting ### Discovery Pipeline
**Enable Debug Logging** 1. **Add artists to watchlist** → SoulSync monitors for new releases
- Settings → Log Level → DEBUG 2. **Fetch similar artists** → music-map.com provides 10 similar artists per watchlist artist
- Check `logs/app.log` for detailed information 3. **Aggregate by occurrence** → Ranks similar artists by how many watchlist artists recommend them
- Change takes effect immediately 4. **Build discovery pool** → Top 50 similar artists × 10 recent releases = ~500 albums
5. **Extract tracks** → Pool contains 1000-2000 tracks, rolling 1-year window
6. **Curate playlists** → Algorithms generate Release Radar, Discovery Weekly, Seasonal
**Common Issues** ### Download Workflow
*Files not organizing properly* 1. **Source Selection** → User picks playlist, album, or uses discovery features
2. **Library Matching** → SoulSync checks existing library to avoid duplicates
3. **Quality Filtering** → Applies user-defined quality profile (FLAC priority)
4. **Download Queue** → Batches requests with configurable concurrency (default: 3)
5. **Metadata Enhancement** → Adds lyrics (LRC), album art, proper tags
6. **File Organization** → Moves to transfer folder using custom templates
7. **Media Server Sync** → Triggers library rescan, updates internal database
### Automation Loop
- **Every 30 minutes**: Wishlist retry for failed downloads
- **Every 24 hours**: Discovery pool refresh, playlist curation
- **On media server scan**: Database incremental update
- **On new release**: Watchlist triggers download if configured
---
## Who Should Use SoulSync
### Perfect For
- **Self-hosters** with Plex, Jellyfin, or Navidrome libraries
- **Music enthusiasts** with 500+ album collections who want automated discovery
- **Electronic music fans** who follow Beatport charts
- **Former Spotify users** who want discovery features for local files
- **Power users** comfortable with API configuration and Docker
### Not Ideal For
- Casual users wanting simple one-click playlist sync
- Users on slow/metered internet (download-heavy workflow)
- People uncomfortable with terminal commands or API keys
- Those seeking streaming-only solutions (not a media server replacement)
---
## Comparison with Alternatives
| Feature | SoulSync | Lidarr | Headphones | Beets |
|---------|----------|--------|------------|-------|
| **Custom Discovery Algorithm** | ✓ | ✗ | ✗ | ✗ |
| **Personalized Playlists** | 12+ types | Manual lists | ✗ | ✗ |
| **Beatport Integration** | ✓ (charts, genres) | ✗ | ✗ | ✗ |
| **ListenBrainz Playlists** | ✓ | ✗ | ✗ | ✗ |
| **Multi-Source Downloads** | Spotify/Tidal/YouTube | MusicBrainz | ✗ | ✗ |
| **Watchlist Monitoring** | ✓ (100+ artists) | ✓ (manual add) | ✓ | ✗ |
| **LRC Lyrics** | ✓ (auto) | ✗ | ✗ | Plugin |
| **Advanced Matching** | Unicode, fuzzy, confidence | Basic | Basic | ✓ |
| **Quality Scanner** | ✓ | ✗ | ✗ | ✓ |
| **Duplicate Cleaner** | ✓ | ✗ | ✗ | ✓ |
| **Web UI** | Modern Flask | ✓ | Basic | CLI only |
| **Template-Based Organization** | ✓ | ✗ | ✗ | ✓ |
| **Seasonal Playlists** | ✓ (auto) | ✗ | ✗ | ✗ |
**SoulSync's Unique Position**: Only tool combining intelligent discovery (Release Radar, Discovery Weekly) with multi-source automation (Beatport charts, ListenBrainz) and self-hosted library management.
---
## Architecture
### Technical Stack
- **Language**: Python 3.8+
- **Web Framework**: Flask with 120+ API endpoints
- **Database**: SQLite with connection pooling and indexing
- **UI**: Modern JavaScript with real-time updates
- **Desktop**: PyQt6 (maintenance mode)
### Service Integrations
- **Spotify API**: Artist monitoring, playlist sync, metadata
- **Tidal API**: Alternative playlist source
- **Plex API**: Library scanning, metadata sync
- **Jellyfin API**: Multi-library support
- **Navidrome API**: Subsonic-compatible server
- **Slskd API**: Download management, search
- **ListenBrainz API**: Community playlists, recommendations
- **LRClib.net**: Synchronized lyrics
- **music-map.com**: Similar artist discovery
### Core Components
**Matching Engine** (`core/matching_engine.py`)
- Text normalization with Unicode support
- Fuzzy string matching with confidence scoring
- Album variation handling
- Special character preservation
**Discovery System** (`core/watchlist_scanner.py`, `core/personalized_playlists.py`)
- Watchlist monitoring
- Discovery pool population
- Playlist curation algorithms
- Seasonal content generation
**Download Pipeline** (`core/soulseek_client.py`, `services/sync_service.py`)
- Quality profile filtering
- Concurrent download management
- Automatic retry logic
- Batch processing
**Metadata Enhancement** (`core/lyrics_client.py`)
- LRC lyrics fetching
- Album art embedding
- Tag normalization
- File organization
### Database Schema
- **Tracks**: Full track metadata with file paths
- **Albums**: Album info with completion tracking
- **Artists**: Artist profiles with watchlist status
- **Discovery Pool**: 1000-2000 track rotating pool
- **Seasonal Content**: Cached seasonal albums/tracks
- **Wishlist**: Failed downloads with retry tracking
- **Similar Artists**: Occurrence-ranked recommendations
### Scale
- **83,000+ lines** of Python code
- **120+ API endpoints**
- **15+ service clients**
- **20+ database tables**
- **Handles libraries of 10,000+ albums**
---
## File Organization
SoulSync uses customizable path templates with validation.
### Default Structure
```
Transfer/
Artist/
Artist - Album/
01 - Track.flac
01 - Track.lrc
```
### Template System
**Available Variables**
- `$artist` - Track artist
- `$albumartist` - Album artist
- `$album` - Album name
- `$title` - Track title
- `$track` - Track number (zero-padded: 01, 02...)
- `$playlist` - Playlist name
**Default Templates**
- **Albums**: `$albumartist/$albumartist - $album/$track - $title`
- **Singles**: `$artist/$artist - $title/$title`
- **Playlists**: `$playlist/$artist - $title`
**Features**
- Client-side validation prevents invalid templates
- Automatic fallback on errors
- Reset to defaults button
- Changes apply immediately
---
## Troubleshooting
### Enable Debug Logging
Settings → Log Level → DEBUG (takes effect immediately)
Check `logs/app.log` for detailed information
### Common Issues
**Files not organizing properly**
- Verify transfer path points to your music library - Verify transfer path points to your music library
- Check template syntax in Settings → File Organization - Check template syntax in Settings → File Organization
- Use "Reset to Defaults" if templates are broken - Use "Reset to Defaults" if templates are broken
- Review logs for path-related errors - Review logs for path-related errors
*Docker drive access* **Docker drive access issues**
- Ensure drives are mounted in docker-compose.yml - Ensure drives are mounted in docker-compose.yml
- Restart Docker Desktop if mounts fail - Restart Docker Desktop if mounts fail
- Verify paths use `/host/mnt/X/` prefix - Verify paths use `/host/mnt/X/` prefix in settings
*Wishlist tracks stuck* **OAuth failing from remote devices**
- Spotify requires 127.0.0.1, not server IP
- Manually edit callback URL to use server IP
- See [DOCKER-OAUTH-FIX.md](DOCKER-OAUTH-FIX.md)
**Wishlist tracks stuck**
- Remove items using delete buttons on wishlist page - Remove items using delete buttons on wishlist page
- Auto-retry runs every 30 minutes - Auto-retry runs every 30 minutes
- Check logs for download failures - Check logs for persistent download failures
- Verify slskd is running and accessible
*Multi-library setups* **Multi-library Plex/Jellyfin setups**
- Select correct library from dropdown in settings (Plex/Jellyfin) - Select correct library from dropdown in settings
- Test connection to verify credentials - Test connection to verify credentials
- Check library permissions
## 🏗️ Architecture **Quality scanner finding false positives**
- Adjust quality profile thresholds
- Review format priorities (FLAC vs. MP3)
- Check logs for matching errors
- **Services**: Spotify, Tidal, Plex, Jellyfin, Navidrome, Soulseek clients ---
- **Database**: SQLite with automatic library caching and updates
- **UI**: PyQt6 Desktop + Flask Web Interface
- **Matching**: Advanced text normalization and fuzzy scoring
- **Metadata**: Mutagen + LRClib.net for tags and lyrics
- **Automation**: Multi-threaded with retry logic and background tasks
## 📝 Recent Updates ## Development
- **Customizable file organization** with template-based paths and validation ### Project Structure
- **Log level control** without restart
- **Jellyfin library selector** for multi-library setups ```
- **Enhanced wishlist management** with track/album removal SoulSync/
- **Docker config persistence** between container restarts ├── core/ # Core service clients
│ ├── spotify_client.py
│ ├── soulseek_client.py
│ ├── matching_engine.py
│ ├── watchlist_scanner.py
│ └── personalized_playlists.py
├── database/ # Database layer
│ └── music_database.py
├── services/ # Business logic
│ └── sync_service.py
├── webui/ # Web interface
│ ├── static/
│ └── index.html
├── ui/ # Desktop GUI (PyQt6)
├── config/ # Configuration management
├── utils/ # Utilities and logging
└── web_server.py # Flask application (22k lines)
```
### Contributing
Contributions welcome! Please:
1. Check existing issues before creating new ones
2. Follow existing code style
3. Add tests for new features
4. Update documentation
---
## Roadmap
### Planned Features
- WebSocket support (replace polling architecture)
- Batch wishlist operations (select 20 tracks → remove)
- Download history browser UI
- Source reliability tracking (learn which Slskd users are best)
- Notification center (persistent toast history)
- Mobile-responsive UI improvements
- Playlist collaboration between SoulSync instances
- Smart bandwidth management (time-based rules)
### Under Consideration
- MusicBrainz ID integration
- Additional streaming sources (Deezer, Apple Music)
- Advanced playlist scheduling
- Export to external playlists (Spotify, Tidal)
- Machine learning for improved matching
---
## License
[Include your license here]
---
## Acknowledgments
- **slskd** - Soulseek daemon
- **music-map.com** - Similar artist data
- **LRClib.net** - Synchronized lyrics
- **Spotify, Tidal, Plex, Jellyfin, Navidrome** - API providers
- **Community contributors** - Feature requests and bug reports
--- ---