Commit graph

22 commits

Author SHA1 Message Date
Yiorgis Gozadinos
afdef92b5b
Finish the comment pass, and escape document fields everywhere Rich renders
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
2026-08-28 15:34:47 +03:00
Yiorgis Gozadinos
8ed24d0e24
Tighten the multi-database documentation
Restructure the storage page into search and provenance, duplicate ids,
ranking, Python operations and CLI commands, and state each in reference
voice. `Several Databases` becomes `Multiple Databases`, with the anchor
carried through every referrer, and the instruction files follow the same
name. Example databases are `papers`, `wiki` and `notes`.

`AmbiguousCitationError` is raised for a cited chunk id held by more than one
selected database, not for any shared id.
2026-08-26 17:23:02 +03:00
Yiorgis Gozadinos
503db3271c
Name the evaluations set check for what it answers
`covers_a_set` is true for a mapping of one, which is a configured database
like any other; `uses_configured_databases` says that. Its population guard
said "several" for the same reason. Document evaluating the configured set
with `--skip-db`, against population, which writes one database and needs a
path.
2026-08-26 13:04:06 +03:00
Yiorgis Gozadinos
aa49ec6cb3
Exclude FRAMES questions whose articles were deleted from Wikipedia 2026-08-24 08:44:01 +03:00
Yiorgis Gozadinos
152d57d6e2
Add frames evaluation dataset 2026-08-24 08:44:01 +03:00
Yiorgis Gozadinos
4bcfb9cc3b
Lead the README with what haiku.rag does 2026-08-18 14:37:05 +03:00
Yiorgis Gozadinos
73d9d93db9
Add MTRAG ClapNQ multi-turn evaluation
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.

Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
2026-08-17 10:53:16 +03:00
Yiorgis Gozadinos
e522d8cfa8
Remove the wix evaluation dataset 2026-08-14 14:46:51 +03:00
Yiorgis Gozadinos
43c17a6777
fix capability execution limits and chat loading 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
11303aa714
Document hotpotqa benchmark results and finalize the reference config 2026-07-17 16:28:16 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event 2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
00f2a40a60
Refresh benchmarks doc and remove unused eval datasets 2026-06-01 10:40:51 +03:00
Yiorgis Gozadinos
ceb645564c
Drop GEPA prompt optimization and --target qa from evaluations 2026-05-19 11:01:07 +03:00
Yiorgis Gozadinos
ce7201271f
split open_rag_bench dataset into orb_text and orb_multimodal variants 2026-05-06 12:49:03 +03:00
Yiorgis Gozadinos
5e31c15907
docs 2026-04-28 14:44:27 +03:00
Yiorgis Gozadinos
6ac5f277d8
Rename iterations to num_candidates 2026-03-12 13:00:51 +02:00
Yiorgis Gozadinos
5b3ad9bae7
Refactor QAPromptAdapter to accept QA agent directly, rewrite mock-heavy tests 2026-03-12 12:35:39 +02:00
Yiorgis Gozadinos
4e1752a740
Default eval db location, evaluation script 2025-11-25 11:07:20 +02:00
Yiorgis Gozadinos
f06dac7102
Update READMEs 2025-11-05 10:47:00 +02:00
Yiorgis Gozadinos
2f9c907031
Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00