danvics/soulsync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
soulsync/tests
History
Broque Thomas 258905ff5c Fix: duplicate tracks in albums with Japanese / CJK titles (#722) 
Reporter @Sokhii: downloading the Mushoku Tensei Original
Soundtrack II via Apple Music metadata + Tidal download
produced duplicate library entries — same audio file landed
under multiple track positions in the album view.

Root cause (verified by direct probe + isolated repro):
``MusicMatchingEngine.normalize_string`` correctly skipped
unidecode for CJK text (kanji→pinyin would have produced
gibberish — see the inline comment at line 74-76), but then
ran ``re.sub(r'[^a-z0-9\s$]', '', text)`` which stripped EVERY
CJK character. Every Japanese title normalised to ``''``.
``similarity_score`` has an early-out guard
    if not str1 or not str2: return 0.0
so EVERY CJK-vs-CJK title comparison returned 0.000.

Downstream effect: the matcher fell back to duration+artist
alone. For an OST album with 24 tracks all by the same artist
with similar durations, multiple iTunes track queries landed
on the SAME Tidal candidate. SoulSync wrote each download to
a different output filename (per the iTunes track position),
so on disk there were N copies of the same audio under
different track numbers. The user's library showed 34 entries
for an album with 24 actual tracks.

Probed iTunes album 1753240110 directly — 24 distinct tracks,
zero (disc, track_number) collisions, both US + JP storefronts.
So the duplicate origin was definitely downstream of metadata
fetch.

Fix: when CJK is detected upstream, the alphanumeric-strip step
also preserves CJK Unified Ideographs + radicals
(⺀-鿿), Hiragana + Katakana (぀-ヿ), Halfwidth
/ Fullwidth forms (＀-￯), and Hangul syllables
(가-힯). CJK titles now produce a comparable normalised
form instead of an empty string. ``similarity_score`` works as
intended:

  '命の灯火' vs '命の灯火' → 1.000  (was 0.000)
  '命の灯火' vs '無職転生' → 0.000  (was 0.000, but now from
                                       actual char comparison
                                       not from the empty-string
                                       guard)

Latin-only normalisation is completely unchanged. ``has_cjk``
is False for Latin input, so both the CJK-lowercase branch AND
the new CJK-preserve strip branch are skipped — Latin titles
go through the original unidecode + lowercase + strip path
verbatim. Tested via 4 regression tests that pin the Latin
baseline (simple, unidecode target, $-preservation, identical
+ different similarity scores).

16 new unit tests in ``tests/test_matching_engine_cjk.py``:
- Kanji / Hiragana / Katakana / Hangul / Chinese all survive
- CJK-only strip still removes Latin punctuation in the
  CJK branch
- Mixed Latin + CJK lowercases the Latin half
- Identical CJK titles → 1.0
- Disjoint CJK titles → near 0
- Partially overlapping CJK titles → midrange
- CJK doesn't falsely match unrelated Latin
- 4 Latin-baseline regression pins
- Real-world Mushoku Tensei OST scenario

371 text + imports + new CJK tests pass after the fix.
2026-05-28 08:53:43 -07:00
..
artists Direct ID lookup in Enhance Quality, like Download Discography 2026-05-06 12:05:41 -07:00
automation Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
database Preserve Navidrome album cover art 2026-05-25 19:51:38 -07:00
discovery Wishlist: fix three regressions causing all imports to land as track 01 with no year 2026-05-27 15:39:22 -07:00
downloads Album-bundle staging: clean Soulseek copies + sweep orphans at startup 2026-05-27 22:18:42 -07:00
enrichment Honor manually-matched source IDs in per-source enrichment workers 2026-05-06 19:00:53 -07:00
imports Wishlist: fix three regressions causing all imports to land as track 01 with no year 2026-05-27 15:39:22 -07:00
library Test missing-track existing file imports 2026-05-17 14:18:17 -07:00
matching AcoustID + quarantine modal: three bug fixes (closes #607, closes #608) 2026-05-15 08:55:06 -07:00
media_server Preserve Navidrome album cover art 2026-05-25 19:51:38 -07:00
metadata Branch cleanup: lift manual-match helpers, fix length-pref ordering, profile-scope view toggle 2026-05-27 07:43:21 -07:00
playlists Expose mirrored playlist source-ref health 2026-05-24 19:32:39 -07:00
search feat(import): show MusicBrainz variants 2026-05-24 21:29:22 +03:00
static Auto-Sync Weekly Board: weekday schedules in the UI (PR 3/4) 2026-05-27 12:39:56 -07:00
stats Lift /api/stats/* and /api/listening-stats/* into core/stats/ 2026-04-27 14:27:03 -07:00
streaming Cin-6: Rename soulseek_client global → download_orchestrator 2026-05-04 23:23:32 -07:00
sync Address Cin review: extract helpers, indexed pool fetch, tidy nits 2026-05-24 23:33:55 -07:00
text Address Cin review: extract helpers, indexed pool fetch, tidy nits 2026-05-24 23:33:55 -07:00
tools test(amazon): update search_albums test for derived-from-tracks behavior 2026-05-19 18:24:27 -07:00
watchlist Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
webui Fix missing album placeholder asset path 2026-05-22 08:34:42 -07:00
wishlist Wishlist: fix three regressions causing all imports to land as track 01 with no year 2026-05-27 15:39:22 -07:00
conftest.py Split metadata source and Spotify status 2026-05-02 22:02:00 +03:00
test_acoustid_scanner.py AcoustID scanner: multi-candidate match + duration guard + multi-value retag 2026-05-14 14:09:38 -07:00
test_acoustid_skip_logic.py Fix: tasks showed Completed when file was quarantined 2026-05-03 18:28:32 -07:00
test_acoustid_version_mismatch.py Reject AcoustID matches whose version disagrees with the expected track 2026-05-07 13:25:30 -07:00
test_album_bundle.py Fix: Usenet bundle stuck on "downloading release" when SAB History flips before storage lands (#721) 2026-05-28 08:01:52 -07:00
test_album_bundle_dispatch.py Add Soulseek album bundle downloads 2026-05-23 15:08:21 -07:00
test_album_completeness_job.py Fix: maintenance findings badge inflated when scan dedup-skipped 2026-05-04 08:55:13 -07:00
test_album_mbid_consistency.py Update mbid consistency test mock to match new create_finding bool contract 2026-05-04 09:11:01 -07:00
test_archive_pipeline.py feat(downloads): add archive_pipeline module for torrent/usenet downloads 2026-05-20 17:05:00 -07:00
test_artist_top_tracks_clients.py Add download buttons + bulk action to artist top-tracks sidebar 2026-05-07 15:44:47 -07:00
test_audiodb_worker_stuck_track.py AudioDB worker: stop infinite loop on direct-ID lookup failure (#553) 2026-05-11 14:18:55 -07:00
test_auth_usage_throttle.py test: auth last_used_at write throttle 2026-04-19 15:22:24 -07:00
test_auto_sync_js.py Add node:test contract for auto-sync.js helpers 2026-05-25 00:01:34 -07:00
test_config_save_retry.py Stop config retry tests from writing to the real DB 2026-04-30 14:07:00 -07:00
test_content_filter_regex.py Fix watchlist content filters: live false positives + auto-scan bypass 2026-04-21 19:16:25 -07:00
test_create_finding_dedup_counter.py Repair job card badge — show pending count, not last-scan count 2026-05-07 08:28:17 -07:00
test_dashboard_events.py Realign test file names 2026-04-17 21:25:01 +03:00
test_database_io_resilience.py Handle transient SQLite IO during maintenance 2026-05-21 17:50:30 -07:00
test_debug_info_services.py Fix Copy Debug Info music_source + surface missing services 2026-05-12 16:47:55 -07:00
test_discogs_collection_source.py Discogs: strip artist disambiguation suffixes at every name surface (#634) 2026-05-27 09:05:47 -07:00
test_discogs_track_count.py Discogs: count rows with empty type_ as real tracks too 2026-04-25 08:27:44 -07:00
test_discover_section_controller_js.py Discover controller — Cin pre-review polish 2026-05-07 20:35:10 -07:00
test_download_orchestrator_soundcloud.py Cin-5: Drop per-source attrs from orchestrator 2026-05-04 23:14:05 -07:00
test_download_plugin_conformance.py Add torrent usenet PR notes and test updates 2026-05-21 15:28:48 -07:00
test_downloads_pagination.py Lift shared runtime state into core 2026-04-27 19:54:44 +03:00
test_duplicate_detector_mount_paths.py Filter same-physical-file duplicates from duplicate detector 2026-04-30 14:06:10 -07:00
test_duplicate_detector_slskd_dedup.py Fix: maintenance findings badge inflated when scan dedup-skipped 2026-05-04 08:55:13 -07:00
test_enrichment_services.py Consolidate enrichment bubble routes behind a service registry 2026-05-01 20:14:30 -07:00
test_featured_artist_completion.py Match featured-artist tracks across discography completion 2026-04-30 19:31:11 -07:00
test_hifi_instance_methods.py Fix "no such table: hifi_instances" via defensive lazy-create 2026-05-06 17:49:44 -07:00
test_hls_parsing.py fix: pick OS-specific ffmpeg binary in hls demux fallback test 2026-04-28 21:13:54 -07:00
test_image_cache.py feat(webui): cache artwork images on disk 2026-05-20 10:43:47 -07:00
test_import_album_match_endpoint.py Lift auto-import matching to testable helper + pin contracts 2026-05-09 09:13:23 -07:00
test_import_singles_parallel.py Restore soulsync logger state between parallel-imports tests 2026-05-01 08:45:17 -07:00
test_integrity_failure_marks_task_failed.py Fix: tasks showed Completed when file was quarantined 2026-05-03 18:28:32 -07:00
test_lb_series_detect.py Roll LB Weekly / Top series into single rolling mirrors (Phase 1c.2.1) 2026-05-26 15:49:49 -07:00
test_library_disk_usage.py Repair stale media schema during refresh 2026-05-21 17:41:54 -07:00
test_library_reorganize.py Rewrite Library Reorganize job to delegate to per-album planner 2026-05-06 21:18:20 -07:00
test_library_reorganize_orchestrator.py Drop flaky log-assertion in watchdog test, keep behavioural assertion 2026-05-10 00:01:03 -07:00
test_library_tag_payload.py Add _build_library_tag_db_data helper 2026-05-25 18:48:45 -07:00
test_library_track_identity.py Make extract_external_ids recognize all source-tagging conventions 2026-05-02 18:26:12 -07:00
test_lidarr_download_client.py Fix three Lidarr bugs that prevented it from being a real download source 2026-05-03 15:49:38 -07:00
test_listenbrainz_manager.py Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
test_listening_stats_batch_queries.py test: listening stats worker batched query paths 2026-04-19 15:22:25 -07:00
test_lyrics_reembed_from_sidecar.py Retag now re-embeds LYRICS tag instead of leaving it empty 2026-05-14 15:52:05 -07:00
test_manual_library_match.py Honor manual library matches across source labels 2026-05-18 16:32:38 -07:00
test_manual_pick_no_auto_retry.py Add torrent and usenet release staging support 2026-05-21 14:22:21 -07:00
test_manual_search_endpoint.py Manual picks: stream results, don't auto-retry, fix stuck-at-0% 2026-05-08 15:12:58 -07:00
test_matching_engine_cjk.py Fix: duplicate tracks in albums with Japanese / CJK titles (#722) 2026-05-28 08:53:43 -07:00
test_missing_cover_art.py Fix: maintenance findings badge inflated when scan dedup-skipped 2026-05-04 08:55:13 -07:00
test_orphan_file_detector.py Fix Picard albumartist orphan false positives 2026-05-22 08:43:57 -07:00
test_personalized_api.py Personalized playlists (3/N): standardized API endpoints 2026-05-15 17:15:25 -07:00
test_personalized_generators_curated.py Personalized playlists (2/N): all 8 generators wired through manager 2026-05-15 17:02:08 -07:00
test_personalized_generators_singletons.py Personalized playlists (2/N): all 8 generators wired through manager 2026-05-15 17:02:08 -07:00
test_personalized_generators_variants.py Personalized playlists (2/N): all 8 generators wired through manager 2026-05-15 17:02:08 -07:00
test_personalized_manager.py Personalized pipeline: auto-refresh stale snapshots after watchlist scan 2026-05-15 20:53:03 -07:00
test_personalized_playlists_id_gate.py Discover: sharpen track selection (diversity, source-aware popularity, library dedup, SQL genre) 2026-05-08 08:49:22 -07:00
test_playlist_sources_adapters.py Auto-Sync: fix LB pipelines stuck on "Refreshing:" for 5+ minutes 2026-05-27 18:04:55 -07:00
test_provenance_id_persistence.py Persist source IDs at download time + backfill onto tracks on sync 2026-05-02 17:44:10 -07:00
test_prowlarr_client.py test: cover Prowlarr + torrent + usenet adapters 2026-05-20 15:43:18 -07:00
test_qobuz_credential_sync.py Address Copilot review nits 2026-05-05 11:22:01 -07:00
test_qobuz_playlists.py Fetch all Qobuz favorite tracks for discovery 2026-05-24 01:17:38 -07:00
test_reorganize_db_methods.py Reorganize queue: race + dedupe fixes from kettui review 2026-04-26 08:40:24 -07:00
test_reorganize_orphan_format_handling.py Reorganize: move orphan-format siblings alongside the canonical 2026-05-10 19:58:31 -07:00
test_reorganize_queue.py Reorganize queue: race + dedupe fixes from kettui review 2026-04-26 08:40:24 -07:00
test_reorganize_runner.py Library reorganize: FIFO queue with live status panel 2026-04-25 18:01:32 -07:00
test_reorganize_tag_source.py Reorganize: optional embedded-tag mode (closes #592) 2026-05-15 07:56:18 -07:00
test_reorganize_unknown_artist_hint.py Reorganize: hint at Unknown Artist Fixer for placeholder-metadata rows 2026-05-10 20:16:28 -07:00
test_repair_worker_album_fill.py fix(album-completeness): block cross-artist auto-fill 2026-05-21 16:49:12 -07:00
test_repair_worker_unresolvable_folder_error.py Album Completeness: surface diagnostic when resolver can't find album folder 2026-05-12 14:04:15 -07:00
test_replaygain_summary_parse.py Fix: ReplayGain wrote same +52 dB gain to every track 2026-05-03 19:08:35 -07:00
test_request_cleanup_timer.py test: api/request periodic cleanup timer 2026-04-19 15:22:25 -07:00
test_runtime_state.py Tighten metadata and import safety 2026-04-27 20:28:05 +03:00
test_script_split_integrity.py Add node:test contract for auto-sync.js helpers 2026-05-25 00:01:34 -07:00
test_server_playlist_append_mode.py Server playlist sync: append mode (preserve user-added tracks) 2026-05-10 22:52:11 -07:00
test_socketio_cors.py Socket.IO CORS: handle self-review nits 2026-04-26 19:24:43 -07:00
test_soulseek_search_throttle.py Add min delay between slskd searches (Bell Canada anti-abuse fix) 2026-05-12 19:11:12 -07:00
test_soundcloud_client.py Lift shared download dataclasses + boot via singleton factory 2026-05-05 09:08:39 -07:00
test_spa_deep_linking.py feat: artist-detail deep linking — /artist-detail/:source/:id 2026-05-18 13:07:54 -07:00
test_staging_album_provenance.py refactor(staging): inject batch-field accessor instead of importing runtime_state 2026-05-20 20:43:35 -07:00
test_status_engine_fallback.py Manual picks: stream results, don't auto-retry, fix stuck-at-0% 2026-05-08 15:12:58 -07:00
test_sync_discovery_scan_events.py Realign test file names 2026-04-17 21:25:01 +03:00
test_tag_writer_multi_artist.py AcoustID scanner: multi-candidate match + duration guard + multi-value retag 2026-05-14 14:09:38 -07:00
test_tidal_album_tracks.py Your Albums: selectable wishlist modal + Tidal album resolution 2026-05-11 12:36:16 -07:00
test_tidal_auth_instructions.py Show Tidal callback port (not Spotify's) in auth instructions 2026-04-30 20:54:06 -07:00
test_tidal_auth_redirect_uri.py Honor configured Tidal redirect_uri, drop request-host fallback 2026-05-02 19:20:18 -07:00
test_tidal_collection_tracks.py Tidal: surface Favorite Tracks as virtual playlist (issue #502) 2026-05-10 21:36:22 -07:00
test_tidal_favorite_albums_artists.py Tidal: rewire favorite albums + artists to V2 user-collection endpoints 2026-05-11 10:03:27 -07:00
test_tidal_qualifier_filter.py Fix MTV Unplugged & live-album false-quarantine pipeline 2026-05-14 12:14:31 -07:00
test_tidal_search_shortening.py Reject Tidal streams that silently downgrade from the requested quality 2026-04-24 13:12:30 -07:00
test_tool_progress_events.py Realign test file names 2026-04-17 21:25:01 +03:00
test_torrent_client_adapters.py test: cover Prowlarr + torrent + usenet adapters 2026-05-20 15:43:18 -07:00
test_torrent_usenet_plugins.py Fix Usenet settings reload without restart 2026-05-22 08:28:56 -07:00
test_track_artist_matching.py Add fallback negative-case test for track-artist matching 2026-04-30 17:13:15 -07:00
test_track_number_repair.py Fix CI test failures from incomplete dummy config and encoding 2026-04-19 13:04:55 -07:00
test_track_search_single_query.py test: api_search_tracks single-query track search 2026-04-19 15:22:24 -07:00
test_unknown_artist_fixer.py fix(repair): rewire Unknown Artist Fixer deferred imports (#646) 2026-05-19 22:09:15 -07:00
test_usenet_client_adapters.py Fix: Usenet bundle stuck on "downloading release" when SAB History flips before storage lands (#721) 2026-05-28 08:01:52 -07:00
test_watchlist_album_match.py Stop watchlist re-downloading compilation tracks; catch slskd dedup orphans 2026-05-01 12:57:50 -07:00
test_watchlist_bulk_add.py Complete MusicBrainz watchlist source parity 2026-05-18 19:19:25 -07:00
test_watchlist_scanner_scan.py Move the import flows and related post-processing pipelines into separate modules 2026-04-27 19:54:43 +03:00
test_websocket_client_transport.py Repoint websocket transport test to core.js after split 2026-04-22 11:30:39 -07:00
test_websocket_infrastructure.py Split metadata source and Spotify status 2026-05-02 22:02:00 +03:00
test_wishlist_pagination.py test: wishlist SQL pagination and category filter 2026-04-19 15:22:24 -07:00
test_worker_existing_id_marks_matched.py test: enrichment worker re-processing fix and migration backfill 2026-04-19 15:22:24 -07:00
test_worker_utils_album_track_count.py Repair stale media schema during refresh 2026-05-21 17:41:54 -07:00
test_youtube_ffmpeg_no_eager_download.py ffmpeg gate also covers is_available — fixes the actual leak path 2026-05-08 15:46:31 -07:00
test_youtube_search_dash_query.py Escape dash-leading YouTube search queries 2026-05-24 08:54:47 -07:00