27ed0b3bb3
The notice told the model to answer from what it had the moment qa.max_searches ran out, while up to 15 code executions remained and in-code search() does not count against that budget. It now names the spent tool and points at whichever evidence tool still has budget, falling back to answer-and-cite only when none do. Also count RetryPromptPart in n_failed_tools: _cite rejects with ModelRetry, so a run whose every cite attempt was refused reported zero failures. And note that n_requests is the run's request count, which tracks a capability's own budget only while it stays loaded. |
||
|---|---|---|
| .. | ||
| configs | ||
| evaluations | ||
| scripts | ||
| tests | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
Haiku RAG - Evaluations
Internal benchmarking and evaluation scripts for haiku.rag.
This package is not published to PyPI and is only used for development and testing purposes.
Overview
Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets:
- WiX (
wix) - HotpotQA (
hotpotqa) — multi-hop QA over Wikipedia paragraphs (distractor validation split, 7,405 questions, two gold documents per question) - OpenRAG Bench, two variants:
orb_text— text embedder (qwen3-embedding:4b, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.orb_multimodal— multimodal embedder (qwen3-vl-embedding-8b, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer.
Usage
After installing the package, you can run evaluations using the evaluations command:
# Run retrieval + QA benchmarks
evaluations run wix
evaluations run orb_text
# Use a custom config file
evaluations run wix --config /path/to/haiku.rag.yaml
# Override the database path
evaluations run wix --db /path/to/custom.lancedb
# Skip database population and run only benchmarks
evaluations run wix --skip-db
# Skip specific benchmarks
evaluations run wix --skip-retrieval
evaluations run wix --skip-qa
# Limit the number of test cases
evaluations run wix --limit 100
Choosing the target
evaluations run benchmarks --target rag-capability by default. Use
--target analysis-capability to benchmark the analysis capability against the same
datasets and judge:
evaluations run wix --target rag-capability
evaluations run wix --target analysis-capability --capability-model ollama:gpt-oss
--capability-model "provider:name" overrides the capability model independently from
the judge (defaults to qa.model, or analysis.model when set for the
analysis-capability target). A citation retrieval metric (cited_map) is computed
alongside QA accuracy from the URIs the capability registered via the cite tool.
Debugging runs in Logfire
With LOGFIRE_TOKEN set, runs ship spans under service_name = 'evals'. The
debug-evals skill in .claude/skills/ turns these into ready-made Logfire
queries (recent runs, per-case pass rate and cited_map, failing and slowest
cases) for use from Claude Code.
Pre-built Databases
Download pre-built evaluation databases from HuggingFace:
evaluations download wix
evaluations download all
evaluations download wix --force
Upload databases (maintainer only):
evaluations upload wix
evaluations upload all
Database Storage
By default, evaluation databases are stored in the haiku.rag data directory:
- Linux:
~/.local/share/haiku.rag/evaluations/dbs/ - macOS:
~/Library/Application Support/haiku.rag/evaluations/dbs/ - Windows:
C:/Users/<USER>/AppData/Roaming/haiku.rag/evaluations/dbs/
You can override this with the --db option.