Merge pull request #528 from ggozad/chore/pin-eval-judge

Pin the eval judge sampling and standardise on Qwen3-Reranker

CHANGELOG.md

@@ -1,9 +1,15 @@
 # Changelog
 ## [Unreleased]
 
+### Added
+
+- `evaluations run` records `judge_extra_body`, `qa_extra_body` and `capability_extra_body` in experiment metadata.
+
 ### Changed
 
 - `get_document_by_id` / `get_document_by_uri` no longer load the docling structure and page-image blobs. Load them with `DocumentRepository.get_docling_data` / `get_pages_data`, or `get_by_id(..., include_blobs=True)`.
+- Pinned judge sampling in every reference config under `evaluations/configs/` whose dataset is judged: `temperature` 0.6, `max_tokens` 16384, `extra_body` `top_p` 0.95 / `top_k` 20 / `min_p` 0 / `chat_template_kwargs.enable_thinking` true. `DEFAULT_JUDGE_MODEL` takes `temperature` 0.6, `max_tokens` 16384 and `top_p` 0.95, the subset ollama honours.
+- Reranker in the `orb_text`, `wix` and `t2_finqa` reference configs and in the reranking provider docs: `mixedbread-ai/mxbai-rerank-base-v2` → `Qwen/Qwen3-Reranker-4B` on the `vllm` provider.
 
 ### Fixed
 

docs/benchmarks.md

@@ -69,14 +69,22 @@ evaluations run wix --config /path/to/haiku.rag.yaml --db /path/to/custom.lanced
 
 If no config file is specified, the script searches standard locations: `./haiku.rag.yaml`, user config directory, then falls back to defaults.
 
-To pin the LLM judge in YAML (rather than the default `ollama:qwen3.6`):
+To pin the LLM judge in YAML (rather than the default `ollama:qwen3.6`). These are the recommended settings:
 
 ```yaml
 evaluations:
   judge:
     provider: openai
-    name: gpt-4o-mini
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
     base_url: http://localhost:8000/v1   # optional, for OpenAI-compatible servers (vLLM, LM Studio, etc.)
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true
 ```
 
 ## Methodology
@@ -95,6 +103,8 @@ evaluations:
 
 `pydantic-evals` coordinates an LLM judge to determine whether the capability's answer is correct. The default judge is `ollama:qwen3.6`, pinned so changes to the capability model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions.
 
+A dataset that brings its own deterministic evaluator is scored by that evaluator instead, and no judge runs. T²-RAGBench is the only such dataset today, scored by `NumberMatchEvaluator`.
+
 We picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs.
 
 ### Citation Retrieval

docs/configuration/providers.md

@@ -447,7 +447,7 @@ For high-performance local reranking using dedicated reranking models:
 reranking:
   model:
     provider: vllm
-    name: mixedbread-ai/mxbai-rerank-base-v2
+    name: Qwen/Qwen3-Reranker-4B
     base_url: http://localhost:8001
 ```
 
@@ -524,7 +524,7 @@ Then configure with any HuggingFace model id:
 reranking:
   model:
     provider: cross-encoder
-    name: mixedbread-ai/mxbai-rerank-base-v2
+    name: Qwen/Qwen3-Reranker-0.6B
 ```
 
-Other tested models: `BAAI/bge-reranker-v2-m3`, `Qwen/Qwen3-Reranker-0.6B`, `cross-encoder/ms-marco-MiniLM-L-6-v2`. Any model exposed as a `sentence_transformers.CrossEncoder` works.
+Other tested models: `BAAI/bge-reranker-v2-m3`, `cross-encoder/ms-marco-MiniLM-L-6-v2`. Any model exposed as a `sentence_transformers.CrossEncoder` works.

evaluations/configs/hotpotqa.yaml

@@ -27,5 +27,11 @@ evaluations:
     provider: openai
     name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
     base_url: http://vllm:11430/v1
-    temperature: 0.0
-    max_tokens: 32768
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true

evaluations/configs/orb_multimodal.yaml

@@ -25,3 +25,17 @@ qa:
     name: gemma4-26b
     base_url: http://vllm:11432/v1
     vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true

evaluations/configs/orb_multimodal_nemotron.yaml

@@ -26,3 +26,17 @@ qa:
     name: gemma4-26b
     base_url: http://vllm:11432/v1
     vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true

evaluations/configs/orb_text.yaml

@@ -17,8 +17,9 @@ embeddings:
 
 reranking:
   model:
-    provider: cross-encoder
-    name: mixedbread-ai/mxbai-rerank-base-v2
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
 
 qa:
   model:
@@ -26,3 +27,17 @@ qa:
     name: gemma4-26b
     base_url: http://vllm:11432/v1
     vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true

evaluations/configs/t2_finqa.yaml

@@ -1,5 +1,7 @@
 # Reference config for the `t2_finqa` pre-built evaluation database.
 # T²-RAGBench (FinQA) financial QA, scored by exact numeric match.
+# No `evaluations.judge` block: the spec sets `NumberMatchEvaluator`, which
+# replaces the evaluator list, so no LLM judge is constructed for this dataset.
 # Run: evaluations run t2_finqa --skip-db --target analysis-capability --config configs/t2_finqa.yaml
 # base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
 
@@ -22,8 +24,9 @@ processing:
 
 reranking:
   model:
-    provider: cross-encoder
-    name: mixedbread-ai/mxbai-rerank-base-v2
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
 
 qa:
   model:

evaluations/configs/wix.yaml

@@ -16,8 +16,9 @@ embeddings:
 
 reranking:
   model:
-    provider: cross-encoder
-    name: mixedbread-ai/mxbai-rerank-base-v2
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
 
 qa:
   model:
@@ -25,3 +26,17 @@ qa:
     name: gemma4-26b
     base_url: http://vllm:11432/v1
     vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        enable_thinking: true

evaluations/evaluations/benchmark.py

@@ -34,7 +34,18 @@ TARGETS: tuple[Target, ...] = ("rag-capability", "analysis-capability")
 # Pinned judge model. Decoupled from `config.qa.model` so a user changing
 # their QA model does not inadvertently change the judge — keeps cross-run
 # comparisons stable. Override per-run with `--judge-model provider:name`.
-DEFAULT_JUDGE_MODEL = ModelConfig(provider="ollama", name="qwen3.6")
+#
+# Sampling follows Qwen's recommendation for thinking mode; its model cards
+# forbid greedy decoding. Only the keys ollama honours are set: it silently
+# ignores `top_k`, `min_p` and `chat_template_kwargs`. The vLLM reference
+# configs under `evaluations/configs/` carry those too.
+DEFAULT_JUDGE_MODEL = ModelConfig(
+    provider="ollama",
+    name="qwen3.6",
+    temperature=0.6,
+    max_tokens=16384,
+    extra_body={"top_p": 0.95},
+)
 
 load_dotenv(find_dotenv(usecwd=True))
 
@@ -75,6 +86,7 @@ def build_experiment_metadata(
         "qa_temperature": config.qa.model.temperature,
         "qa_max_tokens": config.qa.model.max_tokens,
         "qa_enable_thinking": config.qa.model.enable_thinking,
+        "qa_extra_body": config.qa.model.extra_body,
         "qa_max_searches": config.qa.max_searches,
     }
     if judge_config is not None:
@@ -85,6 +97,9 @@ def build_experiment_metadata(
                 "judge_temperature": judge_config.temperature,
                 "judge_max_tokens": judge_config.max_tokens,
                 "judge_enable_thinking": judge_config.enable_thinking,
+                # Sampling and thinking reach vLLM through extra_body, so
+                # without it a trace cannot tell which judge settings ran.
+                "judge_extra_body": judge_config.extra_body,
             }
         )
     if capability_config is not None:
@@ -95,6 +110,7 @@ def build_experiment_metadata(
                 "capability_temperature": capability_config.temperature,
                 "capability_max_tokens": capability_config.max_tokens,
                 "capability_enable_thinking": capability_config.enable_thinking,
+                "capability_extra_body": capability_config.extra_body,
             }
         )
     return metadata

evaluations/tests/test_benchmark.py

@@ -66,6 +66,31 @@ class TestBuildExperimentMetadata:
         assert result["judge_temperature"] == 0.0
         assert result["judge_enable_thinking"] is False
 
+    def test_records_extra_body(self) -> None:
+        config = AppConfig()
+        config.qa.model.extra_body = {"top_k": 5}
+        judge = ModelConfig(
+            provider="openai",
+            name="qwen",
+            extra_body={"chat_template_kwargs": {"enable_thinking": True}},
+        )
+        capability = ModelConfig(
+            provider="openai", name="gemma", extra_body={"min_p": 0}
+        )
+        result = build_experiment_metadata(
+            dataset_key="test",
+            test_cases=1,
+            config=config,
+            judge_config=judge,
+            capability_config=capability,
+        )
+
+        assert result["qa_extra_body"] == {"top_k": 5}
+        assert result["judge_extra_body"] == {
+            "chat_template_kwargs": {"enable_thinking": True}
+        }
+        assert result["capability_extra_body"] == {"min_p": 0}
+
     def test_no_reranker(self) -> None:
         config = AppConfig()
         result = build_experiment_metadata(
@@ -77,13 +102,13 @@ class TestBuildExperimentMetadata:
     def test_with_reranker(self) -> None:
         config = AppConfig()
         config.reranking.model = ModelConfig(
-            provider="cross-encoder", name="mixedbread-ai/mxbai-rerank-base-v2"
+            provider="vllm", name="Qwen/Qwen3-Reranker-4B"
         )
         result = build_experiment_metadata(
             dataset_key="test", test_cases=1, config=config
         )
-        assert result["rerank_provider"] == "cross-encoder"
-        assert result["rerank_model"] == "mixedbread-ai/mxbai-rerank-base-v2"
+        assert result["rerank_provider"] == "vllm"
+        assert result["rerank_model"] == "Qwen/Qwen3-Reranker-4B"
 
 
 class TestResolveDataset:
@@ -178,6 +203,13 @@ class TestRunQaBenchmarkJudgeModel:
 
         mock_get_model.assert_any_call(DEFAULT_JUDGE_MODEL, AppConfig())
 
+    def test_pinned_judge_avoids_greedy_decoding(self) -> None:
+        from evaluations.benchmark import DEFAULT_JUDGE_MODEL
+
+        assert DEFAULT_JUDGE_MODEL.temperature == 0.6
+        assert DEFAULT_JUDGE_MODEL.max_tokens == 16384
+        assert DEFAULT_JUDGE_MODEL.extra_body == {"top_p": 0.95}
+
 
 class TestEvaluateDatasetJudgeModel:
     @pytest.mark.asyncio

evaluations/tests/test_reference_configs.py

@@ -0,0 +1,61 @@
+from pathlib import Path
+
+import pytest
+
+from evaluations.datasets import DATASETS
+from haiku.rag.config import load_yaml_config
+from haiku.rag.config.models import AppConfig
+
+CONFIG_DIR = Path(__file__).parent.parent / "configs"
+
+PINNED_JUDGE_SAMPLING = {
+    "temperature": 0.6,
+    "max_tokens": 16384,
+    "extra_body": {
+        "top_p": 0.95,
+        "top_k": 20,
+        "min_p": 0,
+        "chat_template_kwargs": {"enable_thinking": True},
+    },
+}
+
+
+def _config_paths() -> list[Path]:
+    return sorted(CONFIG_DIR.glob("*.yaml"))
+
+
+def _load(path: Path) -> AppConfig:
+    return AppConfig.model_validate(load_yaml_config(path))
+
+
+def test_configs_present() -> None:
+    assert _config_paths(), f"no reference configs found in {CONFIG_DIR}"
+
+
+@pytest.mark.parametrize("path", _config_paths(), ids=lambda p: p.stem)
+def test_config_validates(path: Path) -> None:
+    _load(path)
+
+
+@pytest.mark.parametrize("path", _config_paths(), ids=lambda p: p.stem)
+def test_filename_names_a_dataset(path: Path) -> None:
+    assert path.stem in DATASETS
+
+
+@pytest.mark.parametrize("path", _config_paths(), ids=lambda p: p.stem)
+def test_judge_pinned_where_the_judge_runs(path: Path) -> None:
+    """Datasets without their own qa_evaluator are scored by the LLM judge.
+
+    Those configs must carry the frozen judge settings, so accuracy stays
+    comparable across runs. Datasets that bring a deterministic evaluator
+    never construct a judge, so a judge block there would be dead config.
+    """
+    judge = _load(path).evaluations.judge
+    if DATASETS[path.stem].qa_evaluator is not None:
+        assert judge is None
+        return
+
+    assert judge is not None
+    assert judge.temperature == PINNED_JUDGE_SAMPLING["temperature"]
+    assert judge.max_tokens == PINNED_JUDGE_SAMPLING["max_tokens"]
+    assert judge.extra_body == PINNED_JUDGE_SAMPLING["extra_body"]