From 5e31c159078ecd91551771d3a95fd9170a99e669 Mon Sep 17 00:00:00 2001
From: Yiorgis Gozadinos <ggozadinos@gmail.com>
Date: Tue, 28 Apr 2026 14:44:27 +0300
Subject: [PATCH] docs

---
 CHANGELOG.md          | 10 ++++++++++
 docs/benchmarks.md    |  8 ++++++++
 evaluations/README.md | 15 +++++++++++++++
 3 files changed, 33 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6229d49e..4ec86a6b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,16 @@
 # Changelog
 ## [Unreleased]
 
+### Added
+
+- **Skill-based QA evaluation via `evaluations run --target {qa,rag-skill,analysis-skill}`.** Benchmark the RAG and analysis skills end-to-end alongside the existing QA agent path, against the same datasets and judge. `--skill-model "provider:name"` overrides the skill model independently from the judge.
+- **Citation retrieval as a second eval metric.** `CitationMRREvaluator` and `CitationMAPEvaluator` score the URIs the skill registered via the `cite` tool against each dataset's gold `expected_uris`, alongside the existing LLMJudge. Console output gains a "Citation Retrieval" summary (mean score, cite rate, mean citations per case). Zero extra skill runs — cited URIs are surfaced via `pydantic_evals.set_eval_attribute`.
+- Bumps `haiku.skills` to `>=0.16.0` for the public `run_skill` API and `Skill.request_limit`.
+
+### Changed
+
+- **Removed dataset-specific eval system prompts.** `WIX_SUPPORT_PROMPT` and `ORB_SYSTEM_PROMPT` duplicated guidance already in the shipped `QA_SYSTEM_PROMPT` and `SKILL.md`, and ORB's referenced the obsolete `search_documents` tool name. The eval-side machinery for injecting them (`DatasetSpec.system_prompt`, `resolve_system_prompt()`) is removed. `config.prompts.qa` remains as the user-facing override knob.
+
 ## [0.43.1] - 2026-04-25
 
 ### Fixed
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index 9add1242..56ce43da 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -61,6 +61,8 @@ evaluations run repliqa --config /path/to/haiku.rag.yaml --db /path/to/custom.la
 - `--limit N` - Limit number of test cases
 - `--name NAME` - Override the evaluation name
 - `--judge-model PROVIDER:NAME` - Override the LLM judge model (default: `config.qa.model`)
+- `--target {qa,rag-skill,analysis-skill}` - Choose what to benchmark (default: `qa`). `rag-skill` and `analysis-skill` run the corresponding [skill](skills/index.md) end-to-end against the same datasets and judge as the QA agent.
+- `--skill-model PROVIDER:NAME` - Override the skill model independently from the judge (default: `config.qa.model`). Only valid with skill targets.
 
 If no config file is specified, the script searches standard locations: `./haiku.rag.yaml`, user config directory, then falls back to defaults.
 
@@ -87,6 +89,12 @@ If no config file is specified, the script searches standard locations: `./haiku
 
 For question-answering evaluation, `pydantic-evals` coordinates an LLM judge to determine whether answers are correct. By default the judge uses the same model as QA (`config.qa.model`); override with `--judge-model provider:name`. Accuracy is the fraction of correctly answered questions.
 
+### Citation Retrieval
+
+When benchmarking a skill (`--target rag-skill` or `--target analysis-skill`), a second metric scores the URIs the skill registered via the `cite` tool against each dataset's gold `expected_uris`, using the same MRR / MAP math as raw retrieval. The score key is `cited_mrr` for single-doc datasets and `cited_map` for multi-doc. Console output also includes the cite rate (% of cases with at least one citation) and the mean number of citations per case.
+
+This is computed alongside QA accuracy from the same skill run — no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the skill grounded its answer on it.
+
 ## RepliQA
 
 [RepliQA](https://huggingface.co/datasets/ServiceNow/repliqa) contains synthetic news stories with question-answer pairs. We use `News Stories` from `repliqa_3` (1035 documents). Each question has exactly one relevant document, so we use MRR for retrieval evaluation.
diff --git a/evaluations/README.md b/evaluations/README.md
index 8781a1be..8777c3c3 100644
--- a/evaluations/README.md
+++ b/evaluations/README.md
@@ -39,6 +39,21 @@ evaluations run repliqa --skip-qa
 evaluations run repliqa --limit 100
 ```
 
+### Benchmarking the skills
+
+By default `evaluations run` benchmarks the QA agent. Pass `--target` to
+benchmark the RAG or analysis skill instead, against the same datasets and judge:
+
+```bash
+evaluations run wix --target rag-skill
+evaluations run wix --target analysis-skill --skill-model ollama:gpt-oss
+```
+
+`--skill-model "provider:name"` overrides the skill model independently from
+the judge (defaults to `qa.model`). For skill targets, a citation retrieval
+metric (`cited_mrr` / `cited_map`) is computed alongside QA accuracy from the
+URIs the skill registered via the `cite` tool.
+
 ### Pre-built Databases
 
 Download pre-built evaluation databases from HuggingFace: