Commit graph

13 commits

Author SHA1 Message Date
Yiorgis Gozadinos
a04a16c717
Remove the environment overrides and the last unnamed-database wording
HAIKU_RAG_DB and DB_PATH are gone: a capability covers what the
configuration places or the db_path it is given, and the app backend and
the AG-UI example load their configuration as the CLI does. The compose
files point HAIKU_RAG_CONFIG_PATH at the mounted haiku.rag.yaml, which
places the database at /data where DB_VOLUME is mounted; the backend
refuses a configured set since it serves one database. The chat scopes a
selection by source only over a set and names databases on filter rows
only across several. Docstrings, docs and test fixtures stop describing an
unnamed database; every database a search, listing or citation reports
carries a name.
2026-09-03 15:12:09 +03:00
Yiorgis Gozadinos
afdef92b5b
Finish the comment pass, and escape document fields everywhere Rich renders
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
2026-08-28 15:34:47 +03:00
Yiorgis Gozadinos
09a7076b7e
State what the code does, not what it replaced
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
2026-08-28 15:13:52 +03:00
Yiorgis Gozadinos
2bbd949a29
Record the database each citation came from
A run over several databases could report which documents were cited but
not which database grounded the answer: `_result_from_run` walked the
citation index for `document_uri` and dropped `Citation.source`. The
distribution is not recoverable from the report afterwards, so a sharded
run would have measured everything except attribution.

`cited_sources` is one entry per cited chunk, in citation order, empty
where the database is unnamed.
2026-08-24 10:03:47 +03:00
Yiorgis Gozadinos
587ba75a62
Thread the document filter through the live QA runner 2026-08-17 11:03:52 +03:00
Yiorgis Gozadinos
db2b8fb883
Add compaction arms and grounding status to MTRAG live runs 2026-08-17 10:54:20 +03:00
Yiorgis Gozadinos
73d9d93db9
Add MTRAG ClapNQ multi-turn evaluation
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.

Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
2026-08-17 10:53:16 +03:00
Yiorgis Gozadinos
c137305468
Scope the limit notice and spend the cite window on own turns only
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.

The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.

Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
2026-07-30 19:14:14 +03:00
Yiorgis Gozadinos
27ed0b3bb3
Send analysis to the sandbox when only its search budget is spent
The notice told the model to answer from what it had the moment
qa.max_searches ran out, while up to 15 code executions remained and
in-code search() does not count against that budget. It now names the
spent tool and points at whichever evidence tool still has budget,
falling back to answer-and-cite only when none do.

Also count RetryPromptPart in n_failed_tools: _cite rejects with
ModelRetry, so a run whose every cite attempt was refused reported zero
failures. And note that n_requests is the run's request count, which
tracks a capability's own budget only while it stays loaded.
2026-07-30 18:17:26 +03:00
Yiorgis Gozadinos
721acbcf38
Address review on PR #524
- _budget_notice no longer names the cite tool after prepare_tools has
  withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
  raises ToolFailed for every error in model-written Python, so
  budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
2026-07-30 15:50:06 +03:00
Yiorgis Gozadinos
a528ab912f
Record per-case retrieval diagnostics in evaluations
Count tool traffic from the message history, where refused and repeated
calls stay visible, unlike state.searches which is keyed by query.
2026-07-30 14:56:17 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation 2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00