haiku.rag/evaluations/tests
Yiorgis Gozadinos a60d0f45d3
Strip the speaker tag from MTRAG retrieval queries
The query files encode the speaker into the text, so every retrieval query
arrived as "|user|: How many teams are in the NFL?". That reaches the
embedder, the BM25 query and the reranker's query.

Measured paired over 777 queries on four domains: stripping is worth +3.60pp
recall@5 with a reranker (94 queries better, 33 worse, 650 tied) and nothing
without one (40 better, 40 worse). A cross-encoder scores query against
document directly, so junk tokens on the query side hurt it where a
bag-of-words branch and a pooled embedding absorb them.

Confined to the retrieval query files: 208 of 208 in both lastturn and
rewrite carry it, while QA turn texts, answers and live questions carry none.

Changes retrieval scores for mtrag_clapnq, mtrag_clapnq_rewrite,
mtrag_federated and mtrag_pooled. The single-database direction is small and
signed: hybrid -0.36pp, vector -1.83pp, FTS +1.25pp, the branches moving
oppositely and nearly cancelling.

Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
2026-09-01 08:17:35 +03:00
..
__init__.py Tests for gepa 2026-03-12 12:05:25 +02:00
test_benchmark.py Convert batched ingest content as text, not as a locator 2026-08-31 19:08:58 +03:00
test_capability_runner.py Finish the comment pass, and escape document fields everywhere Rich renders 2026-08-28 15:34:47 +03:00
test_citation_evaluators.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_config.py Finish the comment pass, and escape document fields everywhere Rich renders 2026-08-28 15:34:47 +03:00
test_conversation_evaluator.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_datasets.py Add stable ids to FRAMES question rows 2026-08-24 09:03:44 +03:00
test_evaluators.py Add MTRAG ClapNQ multi-turn evaluation 2026-08-17 10:53:16 +03:00
test_mtrag.py Strip the speaker tag from MTRAG retrieval queries 2026-09-01 08:17:35 +03:00
test_mtrag_federated.py Sample and partition the pooled corpus at passage level 2026-08-31 19:08:58 +03:00
test_numbers.py Normalize unicode signs 2026-06-06 14:52:04 +03:00
test_reference_configs.py Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
test_submission.py Add T²-RAGBench leaderboard submission exporter 2026-06-08 11:51:02 +03:00