IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
_store_documents_with_chunks embedded each document's chunks in its own
embed_chunks call; chunks missing embeddings are now flattened across the
whole batch, embedded in one pass honoring embeddings.batch_size, and
assigned back positionally.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
The guard accepted one carried record as covering the whole request, so a host
retaining only the RAG namespace had its earlier analysis evidence replaced by
receipts and lost, with the RAG record making the loss look accounted for.
Each capability is now judged on its own, and only when its own evidence is at
stake: another capability's record says nothing about this one's, and requiring
every record would stop a host that registers both capabilities and only ever
uses one.
Both optional capabilities read what earlier questions retrieved and cited from
the capability's state, so a host that carries only the message history hands
every run an empty record. Compaction then replaced the earlier evidence with
receipts and retained nothing, and the loss was invisible: the citations the host
already displayed were still there. It now refuses when it finds evidence from an
earlier question and no record of what that question cited.
`state_carried` reaches the optional capabilities through discovery, so the
refusal distinguishes a host that never carries state from a question that simply
cited nothing.
The documentation taught the pattern that breaks: the compose example is now
stateful and the requirement is stated where each capability is introduced.
The app's browser storage was doing exactly this, keeping only the fields the UI
reads. It now persists the whole namespace map, so the citation policy's
violations survive a reload as well as the evidence record.
The session store rebuilt the rag namespace from four known keys, so the
evidence record never survived a turn, let alone a reload from localStorage.
Compaction then ran with an empty ledger: earlier evidence was replaced by
receipts retaining nothing, while the citations already in citation_index kept
the UI looking correct.
The UI still names the fields it reads, but everything else in the namespace
passes through untouched, and seeding the namespace no longer replaces sibling
namespaces.
The README feature list and the overview stopped at the analysis capability.
Both examples and the app backend composed agents without the capabilities the
documentation recommends alongside an evidence capability.
custom_agent.py ran each input as an independent agent run, so it needed a state
dict and a carried history before compaction could mean anything there: without
state the evidence record is empty, and earlier evidence would reduce to
receipts retaining nothing.
A cite call naming one good id and one mangled one registered the good one and
then asked the model to cite again. A model that mangles ids obeys, mangles
again, and the run dies on output retries with the answer lost: observed at 22
consecutive cite calls against gemma4-26b, with the citation policy registered
to press for a declaration. The branch is only reachable once something has
registered, so the answer already has grounding and there is nothing to ask for.
Resumption was inferred from the transcript, and no shape says it. A missing
prompt is how UI adapters ask their first question as much as how pydantic-ai
resumes one, so every AG-UI host failed on its first message. Reading the tail
instead moved the error rather than fixing it: a settled structured answer ends
with an output tool's return, indistinguishable from results delivered to a
question still in progress, so following questions inherited the first one's
identity.
`CapabilityEvidenceRecord.in_progress` is now the authority. `begin_question`
sets it, `after_run` clears it unless the run is only pausing for deferred work,
and a run that raised never reaches `after_run`, which is what leaves an
interrupted question claimable. The history is consulted only to catch a host
that dropped the state of a question the model is unmistakably still owed.
The documentation claimed a conversation that has never cited anything is not enforced,
which reads as though a first question could answer from fresh evidence without
declaring it. Enforcement needs neither condition to hold: no evidence outcome in this
question, and nothing cited earlier.
The structured-output test accepted a redirect or a recorded violation, so a break in
ending detection would still have passed on the backstop alone. The cite tool is
available in that scenario, so it asserts the redirect.
An output tool call is a `ToolCallPart` like any other, and treating every tool call as
intermediate meant a model could search, skip citing, emit its structured answer and
finish with neither a redirect nor a record. A response ends the question when it
carries no tool calls, or when one of its calls names an output tool.
Some endings are not visible from a single response — a host running
`end_strategy="early"` can finish on text beside a function call — so `after_run` is
the backstop: it cannot ask the model for anything by then, but it records a question
that reached the end of its run undeclared. That also covers a question that was asked
once, ignored, and finished anyway, which previously returned early on the redirect
marker and went unrecorded.
The capability documentation and the changelog said a question that gathered no
evidence is left alone. That describes neither the code nor the intent: enforcement
applies wherever there is something to declare, which includes a follow-up that reuses
evidence cited earlier without searching again.
`_already_asked` matched the redirect's phrasing, so a question that merely contained
it read as a redirect we had sent and switched enforcement off for that question. The
redirect carries `[haiku.rag/citation-redirect]` and detection matches that, the same
way retrieved pictures are identified by a tag rather than by the prose beside them.
Requiring an evidence outcome from the current question exempted the case enforcement
exists for: a follow-up about evidence already cited needs no new search, since that
evidence is still on the wire — in a capsule when a compactor is registered, in full
when not. The condition is now that the conversation has something to declare, either
an outcome in this question or evidence it has already cited, which is independent of
whether anything compacts. A conversation that has neither is still left alone.
Citing again cannot narrow a question at any epoch. Declarations merged only within one
epoch, so an empty second thought a request later replaced the refs with nothing and
reported a grounded question ungrounded. They merge while no evidence outcome has
followed the standing declaration, and only genuinely newer evidence starts one afresh.
Whether a question has already been asked to declare is read from the message history
rather than remembered on the run instance, which a resumption's `for_run` discarded —
the same question was asked twice. Reading the history also makes the right call when a
redirect was enqueued but the run ended before it reached the model: nothing is in the
history, so it is asked again. Violations are recorded once per question for the same
reason.
`CitationPolicyCapability` makes the single enforcement decision, whatever mix of
evidence capabilities is registered: two of them must not each demand a citation for
one answer. It decides in `after_model_request`, when a response carries no tool calls
and the question can still be redirected. An explicitly ungrounded answer is a
declaration and is left alone; a question that gathered no evidence is left alone too,
read from the ledger rather than from a searches dict that a new question clears. When
the cite tool is already withdrawn the question is recorded in
`CitationPolicyState.violations` instead of pointing the model at a tool that is gone.
Registering it is the only switch. `DiscoveredEvidence` and discovery move to
`capabilities.evidence` so both optional capabilities share them, and `cite_available`
joins `evidence_tool_names` as public for the same reason.
Measured on Qwen3.6-35B over two arms of 37 questions, 29 of them unanswerable from
the corpus: explicit ungrounded declarations rose from 23 to 26, grounded answers to
unanswerable questions fell from 4 to 2, answerable questions stayed at 8 of 8, and
one redirect fired in the whole arm. Reading every answer found no invented grounding.
`rag_cite` and `analysis_cite` accepted only a non-empty `chunk_ids`, so a model with
nothing to cite could comply only by staying silent — indistinguishable from
forgetting. An empty list is now a valid answer to "what grounds this?", recorded as a
declaration with no refs, which derives `ungrounded` rather than leaving the question
undeclared. Citing again cannot narrow it: an empty call after a grounded one leaves
it grounded.
The instructions lose their carve-outs. Refusing for lack of information no longer
exempts the call, and a corpus-level computation cites an empty list instead of
skipping.
The nemotron rows for retrieval and for both capabilities are re-measured on
this release with no reranker, judged by Qwen3.6-35B with thinking on. Rows for
other embedders and older versions keep their own attribution.
A question's evidence epoch and declaration describe that question and end with
it, so `begin_question` clears them. Held past it, they let the last question's
citations ground the next one and hold a horizon its own declarations cannot
pass, and they force every question's message count to be comparable to counts
taken over a history that a host may since have rebuilt: a thread reconstructed
without a run that never finished shifts every later position, and the next
question was refused where answering it is correct.
Identities still separate one question from the next. Occurrences outlive the
question that recorded them and carry the identities that cited them, which the
capsule is grouped and ordered by, so reuse merges two questions into one group
and a lower identity renders later evidence as earlier.
A request can carry several of this capability's returns — a model can call search
twice in one response — and the carrier was identified by message alone, so each of
them received the whole capsule. It is selected by message and part now, so exactly
one carries it however many share the request.
The chat TUI passed its persisted state into the run, so tool synchronisation mutated
it in place while the message history was promoted only on success. A cancelled or
failed run therefore kept the evidence the tools had recorded and discarded the
messages that justified it, and the next question derived its identity from the
shorter history: behind the recorded epoch, refused as non-append-only, the
conversation unusable until cleared. The run gets a copy, promoted with the messages
or not at all.
Five decorators had been left attached to a helper by an earlier extraction, which
pytest does not collect, so the resume case they carried was silently untested. The
wire test covers both resume shapes again, no prompt and deferred results.
The capability pages said tool results from earlier turns are replaced before every
model request. That is now the compaction capability's job, and only when a host
registers it, so both pages point at it instead of describing it as automatic.
`RAGState` gains its `evidence` field, and the note about per-run resets now says
what a resumption keeps.
`_compact_old_tool_returns`, `PRIOR_TURN_NOTICE` and `turn_start` leave
`RAGCapabilityBase`, along with its `wrap_model_request` hook. The evidence
capabilities now retrieve and validate, and nothing else. Registering the compaction
capability is what rewrites a request; leaving it out sends the transcript untouched,
which was never a choice a host could make before.
The boundary is the recorded question identity rather than message shape, so a
resumption compacts what lies below the question in progress instead of switching
compaction off for the whole run. The newest earlier evidence return carries the
capsule and every other becomes a receipt, so one capsule exists by construction and
every return stays paired with its call.
Pictures of cited evidence are fetched through the capability that retrieved them and
re-attached beside the capsule with fresh labels. Ownership of a picture on the wire
requires the machine tag we write and an image directly after it, since neither
position nor prose is proof: several tools' results can arrive in one request, and a
user quoting our wording above their own picture had it removed. A picture that cannot
be fetched or decoded is emitted with neither its image nor its label.
The chat TUI and the example backend register the compactor, being multi-turn.
`client.ask`, `client.analyze` and the MCP tools do not: a single-shot question has
nothing earlier to compact.
`EvidenceCompactionCapability` reads what the evidence capabilities recorded out of
the run registry, and `build_capsule` renders it: every cited item, grouped by the
question that last cited it, newest group first, each rendered once, with the
pictures of cited evidence and the labels that must accompany them. Discovery runs
one way and reads only, so no capability holds a reference to another and a host
running one, both or neither needs no wiring change.
Everything cited is kept whole and everything else is dropped. There is no character
budget, no picture cap and nothing to configure: a cap would only half-rescue models
that fail on long conversations regardless, and a host that needs earlier evidence
pruned can compact its own requests further.
A capability reports which of its tools produce evidence, so a cite acknowledgement
is never mistaken for one. Pictures are identified by owner, document and reference,
so one figure cited through overlapping chunks is attached once while the same
reference in another document stays a different picture.
The builder does no I/O and never sees the message history, so a picture travels with
its label and the caller fetches the bytes. Nothing reaches the wire yet.
A question identity is unset until a run establishes it. Testing whether the
record exists cannot stand in for that: a host with no state to send seeds a
default record, and a default record is truthy, so a resumption missing its real
state would have proceeded as question zero.
Every way of recording a message count now refuses one behind what is already
stored, through one shared check: a new identity, an evidence outcome and a
declaration, each against the newest identity, evidence epoch and declaration
epoch. Identities and epochs are only comparable while the conversation grows,
and `before_model_request` results are assigned back onto history, so that is a
constraint on the host rather than a guarantee of the framework. Left unchecked,
an evidence outcome moving backwards freezes every later declaration as stale,
and a declaration moving backwards replaces a newer one with an older one and
revives the answer it grounded.
The searches, citations and executions of a question in progress survive a
resumption. Clearing them cost the results the model was still answering from: a
citation afterwards recorded no provenance and could not resolve against the
expanded result it had seen, falling through to a database lookup.
A code execution counts as evidence when it succeeded or printed something. A
raised error with an empty stdout grounds nothing.
