haiku.rag/evaluations/tests
Yiorgis Gozadinos daa6629879
Add the federated ClapNQ retrieval dataset
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.

The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.

The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.

build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.

Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
2026-08-31 19:08:58 +03:00
..
__init__.py Tests for gepa 2026-03-12 12:05:25 +02:00
test_benchmark.py State what the code does, not what it replaced 2026-08-28 15:13:52 +03:00
test_capability_runner.py Finish the comment pass, and escape document fields everywhere Rich renders 2026-08-28 15:34:47 +03:00
test_citation_evaluators.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_config.py Finish the comment pass, and escape document fields everywhere Rich renders 2026-08-28 15:34:47 +03:00
test_conversation_evaluator.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_datasets.py Add stable ids to FRAMES question rows 2026-08-24 09:03:44 +03:00
test_evaluators.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_mtrag.py Simplify the eval harness and share the embed-fill path. 2026-08-17 10:56:26 +03:00
test_mtrag_federated.py Add the federated ClapNQ retrieval dataset 2026-08-31 19:08:58 +03:00
test_numbers.py Normalize unicode signs 2026-06-06 14:52:04 +03:00
test_reference_configs.py Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
test_submission.py Add T²-RAGBench leaderboard submission exporter 2026-06-08 11:51:02 +03:00