pin judge model to ollama:qwen3.6

CHANGELOG.md

@@ -9,6 +9,8 @@
 
 ### Changed
 
+- **Pinned eval judge defaults to `ollama:gpt-oss`.** Previously `--judge-model` defaulted to `config.qa.model`, so changing the QA or skill model also changed the judge — destabilizing cross-run comparisons and re-introducing self-judging whenever the answerer was already gpt-oss. The default is now a fixed `ollama:gpt-oss`; pass `--judge-model provider:name` to override.
+- **Tightened `cite` framing in the RAG skill's `SKILL.md`.** `cite` is now a precondition for the final answer: the model identifies supporting chunk IDs and calls `cite` *before* writing the response. The "MUST cite before answering" requirement carries an explicit refusal carve-out so the model does not cite irrelevant chunks when knowledge is missing. On the wix benchmark this lifted cite rate from 32% → 96%, mean `cited_map` from 0.15 → 0.48, and cut the "correct answer with no citation" pattern from 52% of cases to 1%, with QA accuracy holding at ~78%.
 - **Removed dataset-specific eval system prompts.** `WIX_SUPPORT_PROMPT` and `ORB_SYSTEM_PROMPT` duplicated guidance already in the shipped `QA_SYSTEM_PROMPT` and `SKILL.md`, and ORB's referenced the obsolete `search_documents` tool name. The eval-side machinery for injecting them (`DatasetSpec.system_prompt`, `resolve_system_prompt()`) is removed. `config.prompts.qa` remains as the user-facing override knob.
 
 ## [0.43.1] - 2026-04-25

docs/benchmarks.md

@@ -60,7 +60,7 @@ evaluations run repliqa --config /path/to/haiku.rag.yaml --db /path/to/custom.la
 - `--skip-qa` - Skip QA benchmark
 - `--limit N` - Limit number of test cases
 - `--name NAME` - Override the evaluation name
-- `--judge-model PROVIDER:NAME` - Override the LLM judge model (default: `config.qa.model`)
+- `--judge-model PROVIDER:NAME` - Override the LLM judge model. Defaults to `ollama:qwen3.6` so the judge stays stable when the QA / skill model changes.
 - `--target {qa,rag-skill,analysis-skill}` - Choose what to benchmark (default: `qa`). `rag-skill` and `analysis-skill` run the corresponding [skill](skills/index.md) end-to-end against the same datasets and judge as the QA agent.
 - `--skill-model PROVIDER:NAME` - Override the skill model independently from the judge (default: `config.qa.model`). Only valid with skill targets.
 
@@ -87,7 +87,9 @@ If no config file is specified, the script searches standard locations: `./haiku
 
 ### QA Accuracy
 
-For question-answering evaluation, `pydantic-evals` coordinates an LLM judge to determine whether answers are correct. By default the judge uses the same model as QA (`config.qa.model`); override with `--judge-model provider:name`. Accuracy is the fraction of correctly answered questions.
+For question-answering evaluation, `pydantic-evals` coordinates an LLM judge to determine whether answers are correct. The default judge is `ollama:qwen3.6` — pinned so changes to the QA or skill model don't change the judge underneath. Override per run with `--judge-model provider:name`. Accuracy is the fraction of correctly answered questions.
+
+We picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs.
 
 ### Citation Retrieval
 
@@ -144,6 +146,16 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML
 | `qwen3-embedding:4b` | 256        | `gpt-oss:20b` - no thinking | 0.80     | html, `chunk-radius=2`       |
 | `qwen3-embedding:4b` | 256        | `gpt-oss:20b` - no thinking | 0.83     | html, `chunk-radius=2`, `jinaai/jina-reranker-v3` |
 
+### Skill QA + citation retrieval
+
+`evaluations run wix --target rag-skill` benchmarks the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`.
+
+| Skill model      | QA accuracy | Cite rate | Mean `cited_map` |
+|------------------|-------------|-----------|------------------|
+| `ollama:gpt-oss` | 0.78        | 0.96      | 0.48             |
+
+35 % of cases produce a perfect citation (`cited_map` = 1.0). 1 % of correct answers come back without a citation — the rest are grounded.
+
 ## HotpotQA
 
 [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents.

evaluations/evaluations/benchmark.py

@@ -39,6 +39,11 @@ _CITATION_EVALUATORS: dict[type[Evaluator], type[Evaluator]] = {
 Target = Literal["qa", "rag-skill", "analysis-skill"]
 TARGETS: tuple[Target, ...] = ("qa", "rag-skill", "analysis-skill")
 
+# Pinned judge model. Decoupled from `config.qa.model` so a user changing
+# their QA model does not inadvertently change the judge — keeps cross-run
+# comparisons stable. Override per-run with `--judge-model provider:name`.
+DEFAULT_JUDGE_MODEL = ModelConfig(provider="ollama", name="qwen3.6")
+
 load_dotenv(find_dotenv(usecwd=True))
 
 HF_REPO_ID = "ggozad/haiku-rag-eval-dbs"
@@ -357,7 +362,7 @@ async def run_qa_benchmark(
         for index, doc in enumerate(corpus, start=1)
     ]
 
-    judge_config = judge_model or config.qa.model
+    judge_config = judge_model or DEFAULT_JUDGE_MODEL
     skill_config = (skill_model or config.qa.model) if target != "qa" else None
     db = spec.db_path(db_path)
 
@@ -598,7 +603,7 @@ def run(
     judge_model: str | None = typer.Option(
         None,
         "--judge-model",
-        help="Judge model as 'provider:name' (e.g. 'ollama:gpt-oss').",
+        help="Judge model as 'provider:name'. Defaults to ollama:qwen3.6.",
     ),
     target: str = typer.Option(
         "qa",
@@ -666,7 +671,7 @@ def optimize(
     judge_model: str | None = typer.Option(
         None,
         "--judge-model",
-        help="Judge model as 'provider:name' (e.g. 'ollama:gpt-oss').",
+        help="Judge model as 'provider:name'. Defaults to ollama:qwen3.6.",
     ),
     reflect_model: str | None = typer.Option(
         None,

evaluations/tests/test_benchmark.py

@@ -145,7 +145,9 @@ class TestRunQaBenchmarkJudgeModel:
         mock_get_model.assert_called_once_with(custom_judge, AppConfig())
 
     @pytest.mark.asyncio
-    async def test_defaults_to_judge_model_config(self, tmp_path: Path) -> None:
+    async def test_defaults_to_pinned_judge_model(self, tmp_path: Path) -> None:
+        from evaluations.benchmark import DEFAULT_JUDGE_MODEL
+
         with (
             patch("evaluations.benchmark.get_model") as mock_get_model,
             patch("evaluations.benchmark.HaikuRAG"),
@@ -158,7 +160,7 @@ class TestRunQaBenchmarkJudgeModel:
                 db_path=tmp_path / "test.lancedb",
             )
 
-        mock_get_model.assert_called_once_with(AppConfig().qa.model, AppConfig())
+        mock_get_model.assert_called_once_with(DEFAULT_JUDGE_MODEL, AppConfig())
 
 
 class TestEvaluateDatasetJudgeModel: