`_ingest_batched` asserts its payload is inline content and then passed it to
`HaikuRAG.convert`, which disambiguates a str by parsing it: anything whose
scheme reads as http or https is fetched over the network instead of stored.
`urlparse` strips leading whitespace, so a passage beginning with a newline
and a URL qualifies.
187 passages across MTRAG's cloud and fiqa corpora start that way, which
crashed the pooled build. No clapnq passage does, so mtrag_clapnq and every
other existing dataset is unaffected.
Now converts through the configured converter's text path, which is what
create_document already does.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`title` is the empty string for every cloud and fiqa passage, so two of the
four domains have exactly one title covering 72,442 and 61,022 passages, and
govt's titles are web-scrape artifacts with one 10,192-passage bucket. Only
clapnq has titles that identify a document.
Keeping whole titles therefore put the pooled gold floor at 146,543
passages: a budget of 120,000 yielded zero distractors, and 58 gold titles
alone accounted for 135,479 passages.
Passage level costs nothing the heterogeneous comparison needs. At alpha=0
the domain places a collection, so a query's gold is concentrated by
construction rather than by the atom, and a titleless domain now spreads
across its own collections instead of collapsing into one.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`mtrag_federated` partitions one domain by article title, which is
round-robin fusion's friendliest case: no collection is ever off-topic for a
query, so the guaranteed-slot waste that hurts a real deployment is never
exercised. Every fusion conclusion measured on it is therefore provisional.
`mtrag_pooled` pools all four MTRAG domains, so a query belongs to one and
the rest are genuinely off-topic. `collection_of` gains `alpha`, which now
means something: 0 keeps a collection to one domain, 1 ignores the domain
and shards titles uniformly. Domains map onto collections proportionally,
subdividing by title where there are more collections than domains and
grouping where there are fewer.
Passage ids are checked for collisions across domains, since gold is
uri-keyed and a shared id would make it ambiguous.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
The chunks FTS index is created once when the table is created, over zero
rows, and nothing folds later rows into it but an optimize. build_databases
bypasses populate_db, and with it the closing vacuum, so every database it
built had an index covering nothing.
Full-text search then returns near-arbitrary rows while still returning
results, so nothing looks wrong: measured FTS recall@5 of 0.0000 across 208
queries on a 40k pool, with hybrid losing 9.9pp to pure vector because the
dead branch still consumed half the fused slots.
Reachable outside the eval too, on any storage.auto_vacuum: false, which
every reference config sets. Reproduced through create_document alone:
five documents, five chunks, zero indexed rows.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Hybrid search inside one database fuses its own vector and FTS rankings
with lancedb's RRFReranker over exactly the requested limit, and both
branch queries derive from the same inner query, so there is no
branch-depth knob. Below roughly 50 candidates the two rankings stop
overlapping, nothing sums, and the fusion degenerates: measured recall@5
on a single database was 0.000 at fetch 5, 10 and 20, then 0.267 at 50 and
0.350 at 100.
A dataset's retrieval_limit therefore fixes which regime it measures, and
comparing regimes would otherwise need one dataset per depth.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.
The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.
The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.
build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
`covers_a_set` is true for a mapping of one, which is a configured database
like any other; `uses_configured_databases` says that. Its population guard
said "several" for the same reason. Document evaluating the configured set
with `--skip-db`, against population, which writes one database and needs a
path.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
A run over several databases could report which documents were cited but
not which database grounded the answer: `_result_from_run` walked the
citation index for `document_uri` and dropped `Citation.source`. The
distribution is not recoverable from the report afterwards, so a sharded
run would have measured everything except attribution.
`cited_sources` is one entry per cited chunk, in citation order, empty
where the database is unnamed.
The runner passed the dataset's path to every arm, which opens one database
and is what makes `--db` meaningful. A run over `lancedb.databases` has to
pass none instead, so the client resolves the set, and `DatasetSpec.covers_a_set`
is the one place that decides which of the two a run is.
benchmark.py was 1220 lines holding six unrelated jobs: populating a
database, running retrieval, running QA, resolving datasets, moving
databases to and from HuggingFace, and wiring the Typer CLI.
qa.py takes both QA runners with their live summary, refusal metrics and
target resolution. population.py takes populate_db and the batched ingest.
retrieval.py takes run_retrieval_benchmark. artifacts.py takes HF_REPO_ID and
the download/upload bodies. experiment.py takes DEFAULT_JUDGE_MODEL and
build_experiment_metadata, which retrieval and QA both record.
benchmark.py keeps the CLI at 258 lines: the Typer app, config and case-id
loading, dataset resolution, evaluate_dataset, and three commands whose
bodies are now a loop over specs. The module-level side effects stay with it
— load_dotenv before configure_telemetry, so credentials and LOGFIRE_TOKEN
are in the environment before telemetry and model setup read them — so no
importable module carries one.
Test patch targets follow the code. get_model, run_capability_question,
run_capability_conversation, set_eval_attribute and HaikuRAG are patched
inside moved code, so they move with it; run_qa_benchmark,
run_retrieval_benchmark and find_config_file stay patchable on
evaluations.benchmark because evaluate_dataset and _load_config still look
them up there.
One assertion got stronger: a QA test patched benchmark.HaikuRAG to prove the
QA path does not open its own client. qa.py has no HaikuRAG reference at all
now, so the test asserts that instead.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.
The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.
Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
The notice told the model to answer from what it had the moment
qa.max_searches ran out, while up to 15 code executions remained and
in-code search() does not count against that budget. It now names the
spent tool and points at whichever evidence tool still has budget,
falling back to answer-and-cite only when none do.
Also count RetryPromptPart in n_failed_tools: _cite rejects with
ModelRetry, so a run whose every cite attempt was refused reported zero
failures. And note that n_requests is the run's request count, which
tracks a capability's own budget only while it stays loaded.
- _budget_notice no longer names the cite tool after prepare_tools has
withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
raises ToolFailed for every error in model-written Python, so
budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
