Commit graph

21 commits

Author SHA1 Message Date
Yiorgis Gozadinos
8ed24d0e24
Tighten the multi-database documentation
Restructure the storage page into search and provenance, duplicate ids,
ranking, Python operations and CLI commands, and state each in reference
voice. `Several Databases` becomes `Multiple Databases`, with the anchor
carried through every referrer, and the instruction files follow the same
name. Example databases are `papers`, `wiki` and `notes`.

`AmbiguousCitationError` is raised for a cited chunk id held by more than one
selected database, not for any shared id.
2026-08-26 17:23:02 +03:00
Yiorgis Gozadinos
503db3271c
Name the evaluations set check for what it answers
`covers_a_set` is true for a mapping of one, which is a configured database
like any other; `uses_configured_databases` says that. Its population guard
said "several" for the same reason. Document evaluating the configured set
with `--skip-db`, against population, which writes one database and needs a
path.
2026-08-26 13:04:06 +03:00
Yiorgis Gozadinos
aa49ec6cb3
Exclude FRAMES questions whose articles were deleted from Wikipedia 2026-08-24 08:44:01 +03:00
Yiorgis Gozadinos
152d57d6e2
Add frames evaluation dataset 2026-08-24 08:44:01 +03:00
Yiorgis Gozadinos
4bcfb9cc3b
Lead the README with what haiku.rag does 2026-08-18 14:37:05 +03:00
Yiorgis Gozadinos
73d9d93db9
Add MTRAG ClapNQ multi-turn evaluation
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.

Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
2026-08-17 10:53:16 +03:00
Yiorgis Gozadinos
e522d8cfa8
Remove the wix evaluation dataset 2026-08-14 14:46:51 +03:00
Yiorgis Gozadinos
43c17a6777
fix capability execution limits and chat loading 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
11303aa714
Document hotpotqa benchmark results and finalize the reference config 2026-07-17 16:28:16 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event 2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
00f2a40a60
Refresh benchmarks doc and remove unused eval datasets 2026-06-01 10:40:51 +03:00
Yiorgis Gozadinos
ceb645564c
Drop GEPA prompt optimization and --target qa from evaluations 2026-05-19 11:01:07 +03:00
Yiorgis Gozadinos
ce7201271f
split open_rag_bench dataset into orb_text and orb_multimodal variants 2026-05-06 12:49:03 +03:00
Yiorgis Gozadinos
5e31c15907
docs 2026-04-28 14:44:27 +03:00
Yiorgis Gozadinos
6ac5f277d8
Rename iterations to num_candidates 2026-03-12 13:00:51 +02:00
Yiorgis Gozadinos
5b3ad9bae7
Refactor QAPromptAdapter to accept QA agent directly, rewrite mock-heavy tests 2026-03-12 12:35:39 +02:00
Yiorgis Gozadinos
4e1752a740
Default eval db location, evaluation script 2025-11-25 11:07:20 +02:00
Yiorgis Gozadinos
f06dac7102
Update READMEs 2025-11-05 10:47:00 +02:00
Yiorgis Gozadinos
2f9c907031
Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00