IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
Both optional capabilities read what earlier questions retrieved and cited from
the capability's state, so a host that carries only the message history hands
every run an empty record. Compaction then replaced the earlier evidence with
receipts and retained nothing, and the loss was invisible: the citations the host
already displayed were still there. It now refuses when it finds evidence from an
earlier question and no record of what that question cited.
`state_carried` reaches the optional capabilities through discovery, so the
refusal distinguishes a host that never carries state from a question that simply
cited nothing.
The documentation taught the pattern that breaks: the compose example is now
stateful and the requirement is stated where each capability is introduced.
The app's browser storage was doing exactly this, keeping only the fields the UI
reads. It now persists the whole namespace map, so the citation policy's
violations survive a reload as well as the evidence record.
The README feature list and the overview stopped at the analysis capability.
Both examples and the app backend composed agents without the capabilities the
documentation recommends alongside an evidence capability.
custom_agent.py ran each input as an independent agent run, so it needed a state
dict and a carried history before compaction could mean anything there: without
state the evidence record is empty, and earlier evidence would reduce to
receipts retaining nothing.
The documentation claimed a conversation that has never cited anything is not enforced,
which reads as though a first question could answer from fresh evidence without
declaring it. Enforcement needs neither condition to hold: no evidence outcome in this
question, and nothing cited earlier.
The structured-output test accepted a redirect or a recorded violation, so a break in
ending detection would still have passed on the backstop alone. The cite tool is
available in that scenario, so it asserts the redirect.
An output tool call is a `ToolCallPart` like any other, and treating every tool call as
intermediate meant a model could search, skip citing, emit its structured answer and
finish with neither a redirect nor a record. A response ends the question when it
carries no tool calls, or when one of its calls names an output tool.
Some endings are not visible from a single response — a host running
`end_strategy="early"` can finish on text beside a function call — so `after_run` is
the backstop: it cannot ask the model for anything by then, but it records a question
that reached the end of its run undeclared. That also covers a question that was asked
once, ignored, and finished anyway, which previously returned early on the redirect
marker and went unrecorded.
The capability documentation and the changelog said a question that gathered no
evidence is left alone. That describes neither the code nor the intent: enforcement
applies wherever there is something to declare, which includes a follow-up that reuses
evidence cited earlier without searching again.
`CitationPolicyCapability` makes the single enforcement decision, whatever mix of
evidence capabilities is registered: two of them must not each demand a citation for
one answer. It decides in `after_model_request`, when a response carries no tool calls
and the question can still be redirected. An explicitly ungrounded answer is a
declaration and is left alone; a question that gathered no evidence is left alone too,
read from the ledger rather than from a searches dict that a new question clears. When
the cite tool is already withdrawn the question is recorded in
`CitationPolicyState.violations` instead of pointing the model at a tool that is gone.
Registering it is the only switch. `DiscoveredEvidence` and discovery move to
`capabilities.evidence` so both optional capabilities share them, and `cite_available`
joins `evidence_tool_names` as public for the same reason.
Measured on Qwen3.6-35B over two arms of 37 questions, 29 of them unanswerable from
the corpus: explicit ungrounded declarations rose from 23 to 26, grounded answers to
unanswerable questions fell from 4 to 2, answerable questions stayed at 8 of 8, and
one redirect fired in the whole arm. Reading every answer found no invented grounding.
The nemotron rows for retrieval and for both capabilities are re-measured on
this release with no reranker, judged by Qwen3.6-35B with thinking on. Rows for
other embedders and older versions keep their own attribution.
The capability pages said tool results from earlier turns are replaced before every
model request. That is now the compaction capability's job, and only when a host
registers it, so both pages point at it instead of describing it as automatic.
`RAGState` gains its `evidence` field, and the note about per-run resets now says
what a resumption keeps.
`_compact_old_tool_returns`, `PRIOR_TURN_NOTICE` and `turn_start` leave
`RAGCapabilityBase`, along with its `wrap_model_request` hook. The evidence
capabilities now retrieve and validate, and nothing else. Registering the compaction
capability is what rewrites a request; leaving it out sends the transcript untouched,
which was never a choice a host could make before.
The boundary is the recorded question identity rather than message shape, so a
resumption compacts what lies below the question in progress instead of switching
compaction off for the whole run. The newest earlier evidence return carries the
capsule and every other becomes a receipt, so one capsule exists by construction and
every return stays paired with its call.
Pictures of cited evidence are fetched through the capability that retrieved them and
re-attached beside the capsule with fresh labels. Ownership of a picture on the wire
requires the machine tag we write and an image directly after it, since neither
position nor prose is proof: several tools' results can arrive in one request, and a
user quoting our wording above their own picture had it removed. A picture that cannot
be fetched or decoded is emitted with neither its image nor its label.
The chat TUI and the example backend register the compactor, being multi-turn.
`client.ask`, `client.analyze` and the MCP tools do not: a single-shot question has
nothing earlier to compact.
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.
The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.
Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
- _budget_notice no longer names the cite tool after prepare_tools has
withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
raises ToolFailed for every error in model-written Python, so
budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
The pool watchdog counts only the time a worker spends running code, so a read
that blocks the worker never trips it. The margin above code_timeout therefore
guarded a race that cannot happen, and only bought a runaway call 90s where 60s
was configured. Pass code_timeout straight through. The two limits are disjoint:
the watchdog bounds a call that computes, and the read deadline bounds a call
that reads.
Drop the ordering test, which was true for any positive margin. The containment
test no longer sets code_timeout to zero, because that value now also disables
the watchdog and kills the worker before the read guard can refuse. It patches
the guard instead.
Monty 0.0.19 runs a plain class and a class with __enter__ and __exit__. Only
inheritance and metaclasses raise. Say that in the instructions.
MemoryFile is no longer constructed, so drop the import, the annotation, and the
stale references in the sandbox docstring and CLAUDE.md. That CLAUDE.md line
also still claimed a ThreadPoolExecutor, _run_async, and a fresh interpreter per
call.
The VFS bridge suspends the Monty worker for the length of a read. Monty checks
its duration budget between interpreter steps, so it cannot check while a read
is in flight. Code that reads in a loop overran a 60s budget by minutes. A read
takes about 20ms on a 2789-document corpus, so a full scan spends about 55s in
reads alone.
Check the deadline before each read. Raising from inside the callback answers the
worker's suspension, which keeps the session usable.
Monty also spends max_duration_secs across the session rather than per call, and
the sandbox reuses the session so that variables persist. Budget it for
code_timeout * max_executions. At the old per-call value the first slow call
starved every later one.
Do not wrap feed_run in asyncio.wait_for. Cancelling during pure compute is
clean, but cancelling while a read waits for an answer wedges the session with a
protocol RuntimeError that escapes execute(). A call that computes without
reading stays bounded by the session budget alone.
`docs/installation.md` claimed the full package contained every extra. It
pulls `docling`, `voyageai`, `cohere`, `zeroentropy`, `cross-encoder` and
`tui`, so `jina`, `s3` and `ingester` need installing separately. Without
`jina`, `provider: jina-local` falls back to no reranking. Anthropic is not
built in, and only the slim Docker image is published.
`docs/tuning.md` pointed at `claim_timeout_s`, which `WorkerConfig` now
rejects.
Drive analysis-only chat with analysis.model (falling back to qa.model) so
the running model matches the one the analysis capability configures,
including its vision flag; RAG-bearing chats still run on qa.model.
Give the AG-UI example state-bearing deps and a final STATE_SNAPSHOT so
registered citations reach the client, mirroring the app backend.
Update chat docs: haiku-rag chat uses --capability/-c, and drop the removed
"View state" command-palette entry.
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
