soulsync/core/stats/queries.py
Broque Thomas 2ab460f5c4 Add Library Disk Usage card to System Statistics
Discord request (Samuel [KC]): show how much disk space the library
takes on the Stats page. Implementation piggybacks on the existing
deep scan — Plex/Jellyfin/Navidrome all return file size in their
track API responses, so we read it during the deep scan and store
it on the tracks row. Aggregation is then a single SQL query — no
filesystem walk, no extra I/O during the scan, no separate stat
job. SoulSync standalone gets size from os.path.getsize at insert
time (different code path; the file is local when we write the row).

Schema (`database/music_database.py`):
- New `file_size INTEGER` column on `tracks`. Migration uses the
  established `try SELECT, except ALTER TABLE ADD COLUMN` pattern.
  Idempotent; safe on existing installs. NULL on legacy rows so
  they don't contribute to totals until next deep scan refreshes.
- Added the column to the canonical CREATE TABLE so fresh installs
  get it without going through the migration path.

Track-object plumbing:
- `core/jellyfin_client.py` — JellyfinTrack reads MediaSources[0].Size
  alongside existing Bitrate read. None when 0 / missing.
- `core/navidrome_client.py` — NavidromeTrack reads `size` from
  the Subsonic song object (int coercion + None on parse fail).
- `core/soulsync_client.py` — SoulSyncTrack does os.path.getsize
  (only "server" where size has to come from disk).
- Plex needs no client-side change: track.media[0].parts[0].size
  is read directly inside insert_or_update_media_track.

Persistence — TWO separate insert paths:

(a) `database/music_database.py:insert_or_update_media_track` —
    Plex/Jellyfin/Navidrome flows. Reads file_size from Plex's
    MediaPart OR `track_obj.file_size` wrapper attribute (defensive
    Plex-attr-not-present check + > 0 type guard).
    INSERT writes the new column.
    UPDATE uses COALESCE(?, file_size) so a None from the server
    on a re-sync (rare Jellyfin Size omission) doesn't blank an
    existing value. Pinned via test.

(b) `core/imports/side_effects.py:record_soulsync_library_entry` —
    SoulSync standalone flow. Completely separate code path: the
    standalone deep scan moves files to staging for auto-import
    rather than calling insert_or_update_media_track. After the
    auto-import processes them, side_effects writes the tracks row
    directly. Reads file_size via os.path.getsize(final_path) at
    insert time (file is local) and includes it in the INSERT
    column list. SoulSync only does INSERT-if-not-exists (no
    UPDATE path), so no COALESCE concern.

Aggregator (`database/music_database.py:get_library_disk_usage`):
- SELECT COALESCE(SUM(file_size), 0), COUNT(file_size),
  COUNT(*) - COUNT(file_size) for the totals.
- Per-format breakdown done in Python via os.path.splitext over
  (file_path, file_size) rows — sidesteps SQLite's first-vs-last-dot
  ambiguity for paths like /music/Kendrick/M.A.A.D City/01.flac.
- Defensive: skips empty paths, paths without extension, and
  implausibly long extensions (>6 chars). Returns the full
  empty-shape dict (NOT a partial / undefined) when the column
  doesn't exist or queries fail, so the UI's `if (!data.has_data)`
  branch handles fresh installs cleanly.

API + UI:
- `core/stats/queries.py` — thin pass-through get_library_disk_usage
  matching the existing query-helper convention.
- `web_server.py` — new /api/stats/library-disk-usage endpoint
  mirroring the /api/stats/db-storage pattern.
- `webui/index.html` — new card in System Statistics above the
  Database Storage card.
- `webui/static/stats-automations.js` — _loadLibraryDiskUsage +
  _renderLibraryDiskUsage. Empty state: "Run a Deep Scan to
  populate (X tracks pending)". Partial: "X measured (+Y pending)".
  Full: total + format bars proportional to the largest format.
- `webui/static/style.css` — .stats-disk-* styled to match the
  Database Storage card.

Backward compatibility:
- Migration is additive; existing rows get NULL file_size; the
  empty-shape return from the aggregator means the UI renders
  cleanly without errors before any deep scan runs.
- Old installs upgrading will see "Run a Deep Scan to populate
  (N tracks pending)". Running their next deep scan fills sizes —
  the existing scan flow doesn't need any changes, just consumes
  the new track-wrapper attribute.

Tests:
- `tests/test_library_disk_usage.py` — 13 cases covering schema
  migration, NULL defaults on legacy inserts, fresh-install empty
  shape, summing with mixed NULL/known sizes, per-format breakdown,
  mixed-case extensions, paths with album-name dots, missing
  extensions, empty file_path, implausibly long extensions,
  JellyfinTrack.file_size persistence via insert_or_update_media_track,
  COALESCE preservation on null re-sync.
- `tests/imports/test_import_side_effects.py` — extended the
  existing record_soulsync_library_entry test to assert
  track_row['file_size'] == os.path.getsize(final_path), pinning
  the SoulSync-standalone path. Test fixture's tracks schema also
  updated to include the file_size column.

Verified: full suite 1813 pass (13 new, 1 existing-test extension),
ruff clean, smoke test populating + reading the column round-trips
correctly.

WHATS_NEW entry under '2.4.2' dev cycle.
2026-05-03 20:17:06 -07:00

287 lines
9.2 KiB
Python

"""Stats API query helpers.
Lifted from web_server.py /api/stats/* and /api/listening-stats/* routes.
Pure-ish functions: take dependencies as args, return data dicts/lists. Route
handlers stay in web_server.py and are responsible for request parsing,
jsonify, and error responses.
"""
from __future__ import annotations
import json
import logging
import threading
import time
import traceback
from typing import Any, Callable, Optional
logger = logging.getLogger(__name__)
ImageUrlFixer = Callable[[Optional[str]], Optional[str]]
def get_cached_stats(database, image_url_fixer: ImageUrlFixer, time_range: str) -> dict:
"""Read pre-computed stats cache for a time range. Instant response."""
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute("SELECT value FROM metadata WHERE key = ?", (f'stats_cache_{time_range}',))
row = cursor.fetchone()
data = json.loads(row[0]) if row and row[0] else {}
cursor.execute("SELECT value FROM metadata WHERE key = 'stats_cache_recent'")
row = cursor.fetchone()
recent = json.loads(row[0]) if row and row[0] else []
cursor.execute("SELECT value FROM metadata WHERE key = 'stats_cache_health'")
row = cursor.fetchone()
health = json.loads(row[0]) if row and row[0] else {}
finally:
conn.close()
for item in (data.get('top_artists') or []) + (data.get('top_albums') or []) + (data.get('top_tracks') or []):
if item.get('image_url'):
item['image_url'] = image_url_fixer(item['image_url'])
return {
'cached': True,
**data,
'recent': recent,
'health': health,
}
def get_overview(database, time_range: str) -> dict:
"""Aggregate listening stats for a time range."""
return database.get_listening_stats(time_range)
def get_top_artists(database, image_url_fixer: ImageUrlFixer, time_range: str, limit: int) -> list[dict]:
"""Top artists by play count, enriched with image / Last.fm stats / soul_id."""
artists = database.get_top_artists(time_range, limit)
for artist in artists:
try:
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"""
SELECT thumb_url, id, lastfm_listeners, lastfm_playcount, soul_id
FROM artists
WHERE LOWER(name) = LOWER(?)
LIMIT 1
""",
(artist['name'],),
)
row = cursor.fetchone()
if row:
artist['image_url'] = image_url_fixer(row[0]) if row[0] else None
artist['id'] = row[1]
artist['global_listeners'] = row[2]
artist['global_playcount'] = row[3]
artist['soul_id'] = row[4]
finally:
conn.close()
except Exception:
pass
return artists
def get_top_albums(database, image_url_fixer: ImageUrlFixer, time_range: str, limit: int) -> list[dict]:
"""Top albums by play count, enriched with album thumb."""
albums = database.get_top_albums(time_range, limit)
for album in albums:
try:
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"""
SELECT al.thumb_url, al.id, al.artist_id FROM albums al
WHERE LOWER(al.title) = LOWER(?) AND al.thumb_url IS NOT NULL AND al.thumb_url != ''
LIMIT 1
""",
(album['name'],),
)
row = cursor.fetchone()
if row:
album['image_url'] = image_url_fixer(row[0]) if row[0] else None
album['id'] = row[1]
album['artist_id'] = row[2]
finally:
conn.close()
except Exception:
pass
return albums
def get_top_tracks(database, image_url_fixer: ImageUrlFixer, time_range: str, limit: int) -> list[dict]:
"""Top tracks by play count, enriched with album thumb."""
tracks = database.get_top_tracks(time_range, limit)
for track in tracks:
try:
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"""
SELECT al.thumb_url, t.id, t.artist_id FROM tracks t
JOIN albums al ON al.id = t.album_id
JOIN artists ar ON ar.id = t.artist_id
WHERE LOWER(t.title) = LOWER(?) AND LOWER(ar.name) = LOWER(?)
LIMIT 1
""",
(track['name'], track['artist']),
)
row = cursor.fetchone()
if row:
track['image_url'] = image_url_fixer(row[0]) if row[0] else None
track['id'] = row[1]
track['artist_id'] = row[2]
finally:
conn.close()
except Exception:
pass
return tracks
def get_timeline(database, time_range: str, granularity: str) -> Any:
"""Play count per time period for chart rendering."""
return database.get_listening_timeline(time_range, granularity)
def get_genres(database, time_range: str) -> Any:
"""Genre distribution by play count."""
return database.get_genre_breakdown(time_range)
def get_library_health(database) -> dict:
"""Library health metrics."""
return database.get_library_health()
def get_db_storage(database) -> dict:
"""Database storage breakdown by table."""
return database.get_db_storage_stats()
def get_library_disk_usage(database) -> dict:
"""On-disk size of the library, with per-format breakdown.
Backed by `tracks.file_size` populated during the deep scan from
media-server-reported sizes (Plex MediaPart.size, Jellyfin
MediaSources[].Size, Navidrome <song size="...">,
SoulSync standalone os.path.getsize).
"""
return database.get_library_disk_usage()
def get_recent_tracks(database, limit: int) -> list[dict]:
"""Recently played tracks from listening_history."""
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"""
SELECT title, artist, album, played_at, duration_ms
FROM listening_history
ORDER BY played_at DESC
LIMIT ?
""",
(limit,),
)
rows = cursor.fetchall()
finally:
conn.close()
return [
{
'title': row[0],
'artist': row[1],
'album': row[2],
'played_at': row[3],
'duration_ms': row[4],
}
for row in rows
]
def resolve_track(database, image_url_fixer: ImageUrlFixer, title: str, artist: str) -> Optional[dict]:
"""Resolve a track by title+artist to its file_path / metadata. Returns None if not found."""
conn = database._get_connection()
try:
cursor = conn.cursor()
cursor.execute(
"""
SELECT t.id, t.title, t.file_path, t.bitrate, t.duration,
ar.name as artist_name, al.title as album_title,
al.thumb_url, t.artist_id, t.album_id
FROM tracks t
JOIN artists ar ON ar.id = t.artist_id
LEFT JOIN albums al ON al.id = t.album_id
WHERE LOWER(t.title) = LOWER(?) AND LOWER(ar.name) = LOWER(?)
AND t.file_path IS NOT NULL AND t.file_path != ''
LIMIT 1
""",
(title.strip(), artist.strip()),
)
row = cursor.fetchone()
finally:
conn.close()
if not row:
return None
return {
'id': row[0],
'title': row[1],
'file_path': row[2],
'bitrate': row[3],
'duration': row[4],
'artist_name': row[5],
'album_title': row[6],
'image_url': image_url_fixer(row[7]) if row[7] else None,
'artist_id': row[8],
'album_id': row[9],
}
def trigger_listening_sync(worker) -> None:
"""Spawn a daemon thread that runs the worker's poll loop once.
Caller is responsible for verifying worker is not None before calling.
"""
def _do_sync():
try:
logger.info("[Stats Sync] Starting manual poll...")
worker._poll()
worker.stats['polls_completed'] += 1
worker.stats['last_poll'] = time.strftime('%Y-%m-%d %H:%M:%S')
logger.info("[Stats Sync] Manual poll completed")
except Exception as e:
logger.error(f"[Stats Sync] Manual poll failed: {e}")
traceback.print_exc()
logger.error(f"Manual stats sync failed: {e}")
threading.Thread(target=_do_sync, daemon=True).start()
def get_listening_status(worker) -> dict:
"""Worker status dict. Returns disabled-state shape if worker is None."""
if worker is None:
return {
'enabled': False,
'running': False,
'paused': False,
'idle': False,
'current_item': None,
'stats': {},
}
return worker.get_stats()