Benchmarks state that every published number is exact brute-force kNN;
no benchmark database carries a vector index. The Vector Indexing
section gains the measured with/without IVF_PQ comparison (hotpotqa,
orb_multimodal_nemotron, frames: free to ~121k chunks, 0.0044 MAP at
426k, ~30 s / 4 GB build) and corrects the re-indexing story: optimize()
folds new chunks into the index as delta parts via auto_vacuum, so a
rebuild is about retraining centroids, not covering new rows.
Current results moved above Methodology and Running Evaluations. A reader arriving from an external link met four screens of CLI flags and download instructions before any number.
Benchmarks is promoted to a top-level nav entry, out of Reference, which holds Development and Changelog.
The page move itself changes no wording or figures.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
The nemotron rows for retrieval and for both capabilities are re-measured on
this release with no reranker, judged by Qwen3.6-35B with thinking on. Rows for
other embedders and older versions keep their own attribution.
