haiku.rag/evaluations/evaluations
Yiorgis Gozadinos 044da7ae99
Open the eval database read-only outside population
Retrieval and QA only read from the database, but the benchmark opened it
writable, where an embedder identity differing from the stored one aborts
instead of warning. Running a pre-built database against a different
serving stack then needed a `rebuild --set-embedder` first.

Correct the debug-evals skill alongside it: the pydantic-ai span names are
`execute_tool {tool_name}` and `invoke_agent agent`, targets are
`{rag,analysis}-capability`, and no `skill_model` metadata key exists.
2026-07-28 11:36:27 +03:00
..
datasets Restore hotpotqa evaluation dataset 2026-07-17 16:25:27 +03:00
evaluators replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
__init__.py Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00
benchmark.py Open the eval database read-only outside population 2026-07-28 11:36:27 +03:00
capability_runner.py fix per-question capability limits and tool isolation 2026-07-24 15:26:18 +03:00
config.py Add deterministic Number-Match QA scoring for T²-RAGBench 2026-06-06 14:52:04 +03:00
numbers.py Normalize unicode signs 2026-06-06 14:52:04 +03:00
submission.py replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00