cc344fb205
`title` is the empty string for every cloud and fiqa passage, so two of the four domains have exactly one title covering 72,442 and 61,022 passages, and govt's titles are web-scrape artifacts with one 10,192-passage bucket. Only clapnq has titles that identify a document. Keeping whole titles therefore put the pooled gold floor at 146,543 passages: a budget of 120,000 yielded zero distractors, and 58 gold titles alone accounted for 135,479 passages. Passage level costs nothing the heterogeneous comparison needs. At alpha=0 the domain places a collection, so a query's gold is concentrated by construction rather than by the atom, and a titleless domain now spreads across its own collections instead of collapsing into one. Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_benchmark.py | ||
| test_capability_runner.py | ||
| test_citation_evaluators.py | ||
| test_config.py | ||
| test_conversation_evaluator.py | ||
| test_datasets.py | ||
| test_evaluators.py | ||
| test_mtrag.py | ||
| test_mtrag_federated.py | ||
| test_numbers.py | ||
| test_reference_configs.py | ||
| test_submission.py | ||