Document hotpotqa benchmark results and finalize the reference config

CHANGELOG.md

@@ -1,6 +1,10 @@
 # Changelog
 ## [Unreleased]
 
+### Added
+
+- `hotpotqa` evaluation dataset.
+
 ## [0.67.0] - 2026-07-16
 
 ### Added

docs/benchmarks.md

@@ -1,6 +1,6 @@
 # Benchmarks
 
-We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, and Wix are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the rag and rag-analysis skills.
+We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, HotpotQA, and Wix are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the rag and rag-analysis skills.
 
 ## Running Evaluations
 
@@ -37,6 +37,7 @@ Active datasets:
 | `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
 | `orb_multimodal_nemotron` — OpenRAG Bench, multimodal embedder (`nvidia/llama-nemotron-embed-vl-1b-v2`), the embedder behind the published headline results | ~16 GB |
 | `t2_finqa` — T²-RAGBench (FinQA) financial QA, text embedder (`qwen3-embedding:4b`); scored by exact numeric match, run with `--target analysis-skill` | ~2 GB |
+| `hotpotqa` — HotpotQA multi-hop QA over Wikipedia paragraphs, text embedder (`qwen3-embedding:4b`) | ~1.5 GB |
 
 After downloading, run benchmarks with `--skip-db`. Each database is built with a specific embedder, so pass its reference config from `evaluations/configs/` (a database only opens against a config whose embedder matches):
 
@@ -167,6 +168,27 @@ Two approaches are benchmarked separately:
 
 *Measured on haiku.rag v0.55.0, deterministic Number-Match scoring (ε=0.01), 2560-dim `qwen3-embedding:4b` (vLLM) with `mxbai-rerank-base-v2`. 341 / 8281 cases excluded as nulls (analysis spirals from the request limit and in-generation loops). Accuracy and `cited_map` are over the 7939 scored cases. Mean 16.0s/case.*
 
+### HotpotQA
+
+[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is multi-hop question answering over Wikipedia: each question requires combining facts from two supporting paragraphs, with distractor paragraphs in the corpus. We use the distractor validation split: 7,405 questions over ~66k unique paragraphs, each question mapping to two gold documents.
+
+##### Retrieval (MAP)
+
+| Embedding Model      | Reranker               | Cases | MAP    |
+|----------------------|------------------------|------:|-------:|
+| `qwen3-embedding:4b` | `mxbai-rerank-base-v2` |  7405 | 0.8235 |
+| `qwen3-embedding:4b` | none                   |  7405 | 0.6995 |
+
+The reranker's contribution is larger here than on the single-doc datasets: hybrid search usually surfaces the first-hop document at rank 1, while the second-hop document often needs the reranker to climb into the result window.
+
+##### QA accuracy + citation retrieval
+
+| Skill model                  | Reranker | QA accuracy | Mean `cited_map` |
+|------------------------------|----------|-------------|------------------|
+| `vllm:Gemma-4-26B-A4B-NVFP4` | none     | 0.83        | 0.75             |
+
+*Measured on haiku.rag v0.66.0 with `qwen3-embedding:4b` (vLLM, dim 2560), judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. 7,405 cases (8 errored). Mean `cited_map` (0.75) exceeds the no-reranker retrieval MAP (0.70): the skill reformulates queries across search calls, partially recovering second-hop documents that a single query misses.*
+
 ### Wix
 
 [WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases.

evaluations/README.md

@@ -9,6 +9,7 @@ This package is not published to PyPI and is only used for development and testi
 Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets:
 
 - WiX (`wix`)
+- HotpotQA (`hotpotqa`) — multi-hop QA over Wikipedia paragraphs (distractor validation split, 7,405 questions, two gold documents per question)
 - OpenRAG Bench, two variants:
   - `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.
   - `orb_multimodal` — multimodal embedder (`qwen3-vl-embedding-8b`, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer.

evaluations/configs/hotpotqa.yaml

@@ -15,13 +15,17 @@ embeddings:
     vector_dim: 2560
     base_url: http://vllm:11431/v1
 
-reranking:
-  model:
-    provider: cross-encoder
-    name: mixedbread-ai/mxbai-rerank-base-v2
-
 qa:
   model:
     provider: openai
     name: gemma4-26b
     base_url: http://vllm:11432/v1
+    max_tokens: 49152
+
+evaluations:
+  judge:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.0
+    max_tokens: 32768