From 5f08a2a8ae6435ffb1304269127c9cca5621967c Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Tue, 28 Apr 2026 14:51:28 +0300 Subject: [PATCH 1/3] Adapt rag SKILL.md in order to improve citation rate --- haiku_rag_slim/haiku/rag/skills/rag/SKILL.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/haiku_rag_slim/haiku/rag/skills/rag/SKILL.md b/haiku_rag_slim/haiku/rag/skills/rag/SKILL.md index 77cb2939..fafa8c39 100644 --- a/haiku_rag_slim/haiku/rag/skills/rag/SKILL.md +++ b/haiku_rag_slim/haiku/rag/skills/rag/SKILL.md @@ -26,15 +26,17 @@ List available documents in the knowledge base. Use when the user wants to brows Retrieve a document by ID, title, or URI. Partial matches work. Use when the user wants the full content of a specific document. ### cite -Register chunk IDs as citations for your answer. Call this AFTER formulating your answer, with the `chunk_id` values from search results that support it. +Register the chunk IDs that ground your answer. Call this BEFORE writing your final answer, with the `chunk_id` values from search results that support each claim. Every answer that uses search results must be backed by `cite`. ## How to answer questions 1. Call `search` with relevant keywords from the question 2. Review the results — they are ordered by relevance (rank 1 = best match) 3. If needed, search again with different keywords (you have a limited number of searches) -4. Synthesize a concise answer based strictly on the retrieved content -5. Call `cite` with the chunk IDs of search results that informed your answer +4. Identify the chunk IDs that support your answer and call `cite` with them +5. Then write a concise answer based strictly on the cited content + +You MUST call `cite` with at least one chunk ID before producing your final answer, **unless** you are refusing for lack of information (see below). Answers without citations are considered ungrounded. ## Guidelines @@ -43,7 +45,7 @@ Register chunk IDs as citations for your answer. Call this AFTER formulating you - If multiple results are relevant, synthesize them coherently - Be concise and direct — avoid elaboration unless asked - If the search tool tells you the search limit is reached, stop searching and answer with what you have -- If the retrieved documents do not directly address the question, say: "I cannot find enough information in the knowledge base to answer this question." Do not guess or infer from tangentially related content. +- If the retrieved documents do not directly address the question, say: "I cannot find enough information in the knowledge base to answer this question." Do not guess or infer from tangentially related content. In this refusal case do **not** call `cite` — there is nothing to cite. - Do NOT include chunk IDs or UUIDs in your answer text — your answer should read naturally. Use the `cite` tool separately to register citations. ## When the user mentions a specific document From 8962987ff95faaf5f8fa1e25083dfc1043c334e5 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Tue, 28 Apr 2026 16:20:49 +0300 Subject: [PATCH 2/3] pin judge model to ollama:qwen3.6 --- CHANGELOG.md | 2 ++ docs/benchmarks.md | 16 ++++++++++++++-- evaluations/evaluations/benchmark.py | 11 ++++++++--- evaluations/tests/test_benchmark.py | 6 ++++-- 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ec86a6b..2526d5b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ ### Changed +- **Pinned eval judge defaults to `ollama:gpt-oss`.** Previously `--judge-model` defaulted to `config.qa.model`, so changing the QA or skill model also changed the judge — destabilizing cross-run comparisons and re-introducing self-judging whenever the answerer was already gpt-oss. The default is now a fixed `ollama:gpt-oss`; pass `--judge-model provider:name` to override. +- **Tightened `cite` framing in the RAG skill's `SKILL.md`.** `cite` is now a precondition for the final answer: the model identifies supporting chunk IDs and calls `cite` *before* writing the response. The "MUST cite before answering" requirement carries an explicit refusal carve-out so the model does not cite irrelevant chunks when knowledge is missing. On the wix benchmark this lifted cite rate from 32% → 96%, mean `cited_map` from 0.15 → 0.48, and cut the "correct answer with no citation" pattern from 52% of cases to 1%, with QA accuracy holding at ~78%. - **Removed dataset-specific eval system prompts.** `WIX_SUPPORT_PROMPT` and `ORB_SYSTEM_PROMPT` duplicated guidance already in the shipped `QA_SYSTEM_PROMPT` and `SKILL.md`, and ORB's referenced the obsolete `search_documents` tool name. The eval-side machinery for injecting them (`DatasetSpec.system_prompt`, `resolve_system_prompt()`) is removed. `config.prompts.qa` remains as the user-facing override knob. ## [0.43.1] - 2026-04-25 diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 56ce43da..9c24c0f9 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -60,7 +60,7 @@ evaluations run repliqa --config /path/to/haiku.rag.yaml --db /path/to/custom.la - `--skip-qa` - Skip QA benchmark - `--limit N` - Limit number of test cases - `--name NAME` - Override the evaluation name -- `--judge-model PROVIDER:NAME` - Override the LLM judge model (default: `config.qa.model`) +- `--judge-model PROVIDER:NAME` - Override the LLM judge model. Defaults to `ollama:qwen3.6` so the judge stays stable when the QA / skill model changes. - `--target {qa,rag-skill,analysis-skill}` - Choose what to benchmark (default: `qa`). `rag-skill` and `analysis-skill` run the corresponding [skill](skills/index.md) end-to-end against the same datasets and judge as the QA agent. - `--skill-model PROVIDER:NAME` - Override the skill model independently from the judge (default: `config.qa.model`). Only valid with skill targets. @@ -87,7 +87,9 @@ If no config file is specified, the script searches standard locations: `./haiku ### QA Accuracy -For question-answering evaluation, `pydantic-evals` coordinates an LLM judge to determine whether answers are correct. By default the judge uses the same model as QA (`config.qa.model`); override with `--judge-model provider:name`. Accuracy is the fraction of correctly answered questions. +For question-answering evaluation, `pydantic-evals` coordinates an LLM judge to determine whether answers are correct. The default judge is `ollama:qwen3.6` — pinned so changes to the QA or skill model don't change the judge underneath. Override per run with `--judge-model provider:name`. Accuracy is the fraction of correctly answered questions. + +We picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs. ### Citation Retrieval @@ -144,6 +146,16 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML | `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.80 | html, `chunk-radius=2` | | `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.83 | html, `chunk-radius=2`, `jinaai/jina-reranker-v3` | +### Skill QA + citation retrieval + +`evaluations run wix --target rag-skill` benchmarks the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. + +| Skill model | QA accuracy | Cite rate | Mean `cited_map` | +|------------------|-------------|-----------|------------------| +| `ollama:gpt-oss` | 0.78 | 0.96 | 0.48 | + +35 % of cases produce a perfect citation (`cited_map` = 1.0). 1 % of correct answers come back without a citation — the rest are grounded. + ## HotpotQA [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents. diff --git a/evaluations/evaluations/benchmark.py b/evaluations/evaluations/benchmark.py index d963da4a..5fcb8388 100644 --- a/evaluations/evaluations/benchmark.py +++ b/evaluations/evaluations/benchmark.py @@ -39,6 +39,11 @@ _CITATION_EVALUATORS: dict[type[Evaluator], type[Evaluator]] = { Target = Literal["qa", "rag-skill", "analysis-skill"] TARGETS: tuple[Target, ...] = ("qa", "rag-skill", "analysis-skill") +# Pinned judge model. Decoupled from `config.qa.model` so a user changing +# their QA model does not inadvertently change the judge — keeps cross-run +# comparisons stable. Override per-run with `--judge-model provider:name`. +DEFAULT_JUDGE_MODEL = ModelConfig(provider="ollama", name="qwen3.6") + load_dotenv(find_dotenv(usecwd=True)) HF_REPO_ID = "ggozad/haiku-rag-eval-dbs" @@ -357,7 +362,7 @@ async def run_qa_benchmark( for index, doc in enumerate(corpus, start=1) ] - judge_config = judge_model or config.qa.model + judge_config = judge_model or DEFAULT_JUDGE_MODEL skill_config = (skill_model or config.qa.model) if target != "qa" else None db = spec.db_path(db_path) @@ -598,7 +603,7 @@ def run( judge_model: str | None = typer.Option( None, "--judge-model", - help="Judge model as 'provider:name' (e.g. 'ollama:gpt-oss').", + help="Judge model as 'provider:name'. Defaults to ollama:qwen3.6.", ), target: str = typer.Option( "qa", @@ -666,7 +671,7 @@ def optimize( judge_model: str | None = typer.Option( None, "--judge-model", - help="Judge model as 'provider:name' (e.g. 'ollama:gpt-oss').", + help="Judge model as 'provider:name'. Defaults to ollama:qwen3.6.", ), reflect_model: str | None = typer.Option( None, diff --git a/evaluations/tests/test_benchmark.py b/evaluations/tests/test_benchmark.py index a80cc7fc..d5d07843 100644 --- a/evaluations/tests/test_benchmark.py +++ b/evaluations/tests/test_benchmark.py @@ -145,7 +145,9 @@ class TestRunQaBenchmarkJudgeModel: mock_get_model.assert_called_once_with(custom_judge, AppConfig()) @pytest.mark.asyncio - async def test_defaults_to_judge_model_config(self, tmp_path: Path) -> None: + async def test_defaults_to_pinned_judge_model(self, tmp_path: Path) -> None: + from evaluations.benchmark import DEFAULT_JUDGE_MODEL + with ( patch("evaluations.benchmark.get_model") as mock_get_model, patch("evaluations.benchmark.HaikuRAG"), @@ -158,7 +160,7 @@ class TestRunQaBenchmarkJudgeModel: db_path=tmp_path / "test.lancedb", ) - mock_get_model.assert_called_once_with(AppConfig().qa.model, AppConfig()) + mock_get_model.assert_called_once_with(DEFAULT_JUDGE_MODEL, AppConfig()) class TestEvaluateDatasetJudgeModel: From 318266e681da4bf9d84e6c858cf0fcba080ae453 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Wed, 29 Apr 2026 12:40:55 +0300 Subject: [PATCH 3/3] Update benchmark --- CHANGELOG.md | 2 +- docs/benchmarks.md | 84 ++++++++++++++++++++++++++++------------------ 2 files changed, 53 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2526d5b4..1901100d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ ### Changed -- **Pinned eval judge defaults to `ollama:gpt-oss`.** Previously `--judge-model` defaulted to `config.qa.model`, so changing the QA or skill model also changed the judge — destabilizing cross-run comparisons and re-introducing self-judging whenever the answerer was already gpt-oss. The default is now a fixed `ollama:gpt-oss`; pass `--judge-model provider:name` to override. +- **Pinned eval judge defaults to `ollama:qwen3.6`.** Previously `--judge-model` defaulted to `config.qa.model`, so changing the QA or skill model also changed the judge — destabilizing cross-run comparisons and re-introducing self-judging whenever the answerer matched. A 2×2 calibration vs Claude Opus 4.7 (gpt-oss / qwen3.6 as both answerer and judge) showed `qwen3.6` had κ ≥ 0.66 on both same- and cross-family answerers (vs 0.39–0.55 for `gpt-oss`) with no detectable self-preference bias. Pass `--judge-model provider:name` to override. - **Tightened `cite` framing in the RAG skill's `SKILL.md`.** `cite` is now a precondition for the final answer: the model identifies supporting chunk IDs and calls `cite` *before* writing the response. The "MUST cite before answering" requirement carries an explicit refusal carve-out so the model does not cite irrelevant chunks when knowledge is missing. On the wix benchmark this lifted cite rate from 32% → 96%, mean `cited_map` from 0.15 → 0.48, and cut the "correct answer with no citation" pattern from 52% of cases to 1%, with QA accuracy holding at ~78%. - **Removed dataset-specific eval system prompts.** `WIX_SUPPORT_PROMPT` and `ORB_SYSTEM_PROMPT` duplicated guidance already in the shipped `QA_SYSTEM_PROMPT` and `SKILL.md`, and ORB's referenced the obsolete `search_documents` tool name. The eval-side machinery for injecting them (`DatasetSpec.system_prompt`, `resolve_system_prompt()`) is removed. `config.prompts.qa` remains as the user-facing override knob. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 9c24c0f9..fc072de2 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -97,19 +97,43 @@ When benchmarking a skill (`--target rag-skill` or `--target analysis-skill`), a This is computed alongside QA accuracy from the same skill run — no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the skill grounded its answer on it. -## RepliQA +## Current results + +Numbers measured under the current pinned judge (`ollama:qwen3.6`) on a recent `haiku.rag` version. + +### Wix + +[WixQA](https://huggingface.co/datasets/Wix/WixQA) — real customer support questions paired with curated answers. 200 cases. + +#### Skill QA + citation retrieval + +`evaluations run wix --target rag-skill` benchmarks the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. + +| Skill model | QA accuracy | Mean `cited_map` | +|------------------|-------------|------------------| +| `ollama:gpt-oss` | 0.85 | 0.40 | + +*Measured on haiku.rag v0.43.1, judged by `ollama:qwen3.6` (current default), on 199 of 200 completed cases.* 28 % of cases produce a perfect citation (`cited_map` = 1.0). + +## Past results + +These were measured under the prior pinned judge (`ollama:gpt-oss`). The pinned default has since switched to `ollama:qwen3.6` (see [Methodology — QA Accuracy](#qa-accuracy)) — under the new judge the QA accuracy numbers below typically shift up by ~5–10 pp. + +Retrieval tables don't depend on the judge but are kept here because they were measured on the same older `haiku.rag` versions as their accompanying QA tables. + +### RepliQA [RepliQA](https://huggingface.co/datasets/ServiceNow/repliqa) contains synthetic news stories with question-answer pairs. We use `News Stories` from `repliqa_3` (1035 documents). Each question has exactly one relevant document, so we use MRR for retrieval evaluation. -*Results from v0.19.6* - -### Retrieval (MRR) +#### Retrieval (MRR) | Embedding Model | MRR | Reranker | |-------------------------------|------|----------| | Ollama / `qwen3-embedding:8b` | 0.91 | - | -### QA Accuracy +*Measured on haiku.rag v0.19.6.* + +#### QA Accuracy | Embedding Model | QA Model | Accuracy | Reranker | |------------------------------|----------------------------------|----------|------------------------| @@ -119,17 +143,15 @@ This is computed alongside QA accuracy from the same skill run — no extra invo | Ollama / `mxbai-embed-large` | Ollama / `qwen3` - thinking | 0.87 | `mxbai-rerank-base-v2` | | Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None | +*Measured on haiku.rag v0.19.6, judged by `ollama:gpt-oss`.* + Note the significant degradation when very small models are used such as `qwen3:0.6b`. -## Wix +### Wix -[WixQA](https://huggingface.co/datasets/Wix/WixQA) contains real customer support questions paired with curated answers from Wix. The benchmark follows the evaluation protocol from the [WixQA paper](https://arxiv.org/abs/2505.08643). Each query can have multiple relevant passages, so we use MAP for retrieval evaluation. +[WixQA](https://huggingface.co/datasets/Wix/WixQA) — see description above. We benchmark both the plain text version (HTML stripped, no structure) and HTML version. Since HTML chunks are small (typically a phrase), we use `chunk_radius=2` to expand context. -We benchmark both the plain text version (HTML stripped, no structure) and HTML version. Since HTML chunks are small (typically a phrase), we use `chunk_radius=2` to expand context. - -*Results from v0.27.2* - -### Retrieval (MAP) +#### Retrieval (MAP) | Embedding Model | Chunk size | MAP | Reranker | Notes | |------------------------|------------|------|------------------------|------------------------------| @@ -138,7 +160,9 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML | `qwen3-embedding:4b` | 256 | 0.43 | None | plain text, `chunk-radius=0` | | `qwen3-embedding:4b` | 512 | 0.45 | None | plain text, `chunk-radius=0` | -### QA Accuracy +*Measured on haiku.rag v0.27.2.* + +#### QA Accuracy | Embedding Model | Chunk size | QA Model | Accuracy | Notes | |----------------------|------------|-----------------------------|----------|------------------------------| @@ -146,50 +170,46 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML | `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.80 | html, `chunk-radius=2` | | `qwen3-embedding:4b` | 256 | `gpt-oss:20b` - no thinking | 0.83 | html, `chunk-radius=2`, `jinaai/jina-reranker-v3` | -### Skill QA + citation retrieval +*Measured on haiku.rag v0.27.2, judged by `ollama:gpt-oss`.* -`evaluations run wix --target rag-skill` benchmarks the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. - -| Skill model | QA accuracy | Cite rate | Mean `cited_map` | -|------------------|-------------|-----------|------------------| -| `ollama:gpt-oss` | 0.78 | 0.96 | 0.48 | - -35 % of cases produce a perfect citation (`cited_map` = 1.0). 1 % of correct answers come back without a citation — the rest are grounded. - -## HotpotQA +### HotpotQA [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents. -*Results from v0.20.2* - -### Retrieval (MAP) +#### Retrieval (MAP) | Embedding Model | MAP | Reranker | |----------------------|------|----------| | `qwen3-embedding:4b` | 0.69 | none | -### QA Accuracy +*Measured on haiku.rag v0.20.2.* + +#### QA Accuracy | Embedding Model | QA Model | Accuracy | |----------------------|--------------------------|----------| | `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 | -## OpenRAG Bench (ORB) +*Measured on haiku.rag v0.20.2, judged by `ollama:gpt-oss`.* + +### OpenRAG Bench (ORB) [OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval over visual content like figures, charts, and diagrams. We use MAP for retrieval evaluation since each query maps to one relevant document. **Multimodal processing**: Picture descriptions are generated using a Vision Language Model (VLM) during document conversion, making embedded images searchable via text queries. See [Picture Description configuration](configuration/processing.md#picture-description-vlm). -*Results from v0.26.8* - -### Retrieval (MAP) +#### Retrieval (MAP) | Embedding Model | MAP | VLM | |----------------------|--------|----------------------| | `qwen3-embedding:4b` | 0.9626 | Ollama / ministral-3 | -### QA Accuracy +*Measured on haiku.rag v0.26.8.* + +#### QA Accuracy | Embedding Model | QA Model | Accuracy | VLM | |----------------------|-----------------------------|----------|----------------------| | `qwen3-embedding:4b` | `gpt-oss:20b` - no thinking | 0.912 | Ollama / ministral-3 | + +*Measured on haiku.rag v0.26.8, judged by `ollama:gpt-oss`.*