haiku.rag/evaluations/configs
Yiorgis Gozadinos daa6629879
Add the federated ClapNQ retrieval dataset
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.

The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.

The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.

build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.

Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
2026-08-31 19:08:58 +03:00
..
frames.yaml Point FRAMES at the live reranker and give the judge room to think 2026-08-24 09:03:44 +03:00
hotpotqa.yaml Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
mtrag_clapnq.yaml Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
mtrag_federated.yaml Add the federated ClapNQ retrieval dataset 2026-08-31 19:08:58 +03:00
orb_multimodal.yaml Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
orb_multimodal_nemotron.yaml Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
orb_text.yaml Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
t2_finqa.yaml Pin the eval judge sampling and standardise on Qwen3-Reranker 2026-08-06 13:17:58 +03:00