The pooled builder printed gold-bearing titles via pool_composition, which
counts nothing meaningful when cloud and fiqa have one empty title each: it
reported 17,086 gold-bearing of 40,000 where the passage-level truth is
1,800 gold and 38,200 distractors. The build itself was correct; only the
line was wrong.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`title` is the empty string for every cloud and fiqa passage, so two of the
four domains have exactly one title covering 72,442 and 61,022 passages, and
govt's titles are web-scrape artifacts with one 10,192-passage bucket. Only
clapnq has titles that identify a document.
Keeping whole titles therefore put the pooled gold floor at 146,543
passages: a budget of 120,000 yielded zero distractors, and 58 gold titles
alone accounted for 135,479 passages.
Passage level costs nothing the heterogeneous comparison needs. At alpha=0
the domain places a collection, so a query's gold is concentrated by
construction rather than by the atom, and a titleless domain now spreads
across its own collections instead of collapsing into one.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`mtrag_federated` partitions one domain by article title, which is
round-robin fusion's friendliest case: no collection is ever off-topic for a
query, so the guaranteed-slot waste that hurts a real deployment is never
exercised. Every fusion conclusion measured on it is therefore provisional.
`mtrag_pooled` pools all four MTRAG domains, so a query belongs to one and
the rest are genuinely off-topic. `collection_of` gains `alpha`, which now
means something: 0 keeps a collection to one domain, 1 ignores the domain
and shards titles uniformly. Domains map onto collections proportionally,
subdividing by title where there are more collections than domains and
grouping where there are fewer.
Passage ids are checked for collisions across domains, since gold is
uri-keyed and a shared id would make it ambiguous.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
The chunks FTS index is created once when the table is created, over zero
rows, and nothing folds later rows into it but an optimize. build_databases
bypasses populate_db, and with it the closing vacuum, so every database it
built had an index covering nothing.
Full-text search then returns near-arbitrary rows while still returning
results, so nothing looks wrong: measured FTS recall@5 of 0.0000 across 208
queries on a 40k pool, with hybrid losing 9.9pp to pure vector because the
dead branch still consumed half the fused slots.
Reachable outside the eval too, on any storage.auto_vacuum: false, which
every reference config sets. Reproduced through create_document alone:
five documents, five chunks, zero indexed rows.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Hybrid search inside one database fuses its own vector and FTS rankings
with lancedb's RRFReranker over exactly the requested limit, and both
branch queries derive from the same inner query, so there is no
branch-depth knob. Below roughly 50 candidates the two rankings stop
overlapping, nothing sums, and the fusion degenerates: measured recall@5
on a single database was 0.000 at fetch 5, 10 and 20, then 0.267 at 50 and
0.350 at 100.
A dataset's retrieval_limit therefore fixes which regime it measures, and
comparing regimes would otherwise need one dataset per depth.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.
The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.
The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.
build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
Restructure the storage page into search and provenance, duplicate ids,
ranking, Python operations and CLI commands, and state each in reference
voice. `Several Databases` becomes `Multiple Databases`, with the anchor
carried through every referrer, and the instruction files follow the same
name. Example databases are `papers`, `wiki` and `notes`.
`AmbiguousCitationError` is raised for a cited chunk id held by more than one
selected database, not for any shared id.
`covers_a_set` is true for a mapping of one, which is a configured database
like any other; `uses_configured_databases` says that. Its population guard
said "several" for the same reason. Document evaluating the configured set
with `--skip-db`, against population, which writes one database and needs a
path.
0.78.0 released `api_key` and two authorization fixes out of the section
this branch was still adding to, so the automatic merge filed every
multi-database entry under it and dropped the `0.77.0` heading, which both
sides had de-duplicated. Everything from `0.78.0` down is main's record
verbatim; the multi-database entries stay under Unreleased.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
A run over several databases could report which documents were cited but
not which database grounded the answer: `_result_from_run` walked the
citation index for `document_uri` and dropped `Citation.source`. The
distribution is not recoverable from the report afterwards, so a sharded
run would have measured everything except attribution.
`cited_sources` is one entry per cited chunk, in citation order, empty
where the database is unnamed.
The runner passed the dataset's path to every arm, which opens one database
and is what makes `--db` meaningful. A run over `lancedb.databases` has to
pass none instead, so the client resolves the set, and `DatasetSpec.covers_a_set`
is the one place that decides which of the two a run is.
Port 11433 serves nothing; Qwen3-Reranker-4B is on 11455. Qwen3.6 spends
its budget reasoning before it answers, and its 131072 window leaves ample
input space at 32768.
benchmark.py was 1220 lines holding six unrelated jobs: populating a
database, running retrieval, running QA, resolving datasets, moving
databases to and from HuggingFace, and wiring the Typer CLI.
qa.py takes both QA runners with their live summary, refusal metrics and
target resolution. population.py takes populate_db and the batched ingest.
retrieval.py takes run_retrieval_benchmark. artifacts.py takes HF_REPO_ID and
the download/upload bodies. experiment.py takes DEFAULT_JUDGE_MODEL and
build_experiment_metadata, which retrieval and QA both record.
benchmark.py keeps the CLI at 258 lines: the Typer app, config and case-id
loading, dataset resolution, evaluate_dataset, and three commands whose
bodies are now a loop over specs. The module-level side effects stay with it
— load_dotenv before configure_telemetry, so credentials and LOGFIRE_TOKEN
are in the environment before telemetry and model setup read them — so no
importable module carries one.
Test patch targets follow the code. get_model, run_capability_question,
run_capability_conversation, set_eval_attribute and HaikuRAG are patched
inside moved code, so they move with it; run_qa_benchmark,
run_retrieval_benchmark and find_config_file stay patchable on
evaluations.benchmark because evaluate_dataset and _load_config still look
them up there.
One assertion got stronger: a QA test patched benchmark.HaikuRAG to prove the
QA path does not open its own client. qa.py has no HaikuRAG reference at all
now, so the test asserts that instead.
RAGState and AnalysisState each declared the same five fields, so the generic
base could not name them: StateT was bound to BaseModel, and every access
went through cast(Any, state), a getattr by string, or a loop clearing fields
by name so it could skip the one only AnalysisState has.
EvidenceState declares them once. RAGState adds nothing, AnalysisState adds
executions and overrides begin_invocation to clear them. StateT binds to
EvidenceState, which removes all ten casts and both state-shape getattrs; the
three getattr(ctx.deps, "state") probes stay, since those check a
host-supplied object rather than our own state.
discover_evidence reached into capability.state for two fields. It now asks
through evidence_record() and citation_index(), alongside the
evidence_tool_names() and cite_available accessors it already used. The eval
runner's _RagLikeState protocol and the chat app's getattr reads described
this shape from outside and are gone.
Compatibility is semantic JSON-object equivalence, not bytes: field names and
nesting are unchanged, so a dict stored by 0.75.0 loads and re-dumps equal,
but deriving from a shared base reorders AnalysisState's keys. Nothing
serializes, hashes or string-compares this state — every carry point
re-validates by key.
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.
`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:
- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
be deliberate, serving different query shapes, so an index this function did
not declare survives even on a column it does. The one thing it overwrites is
an index at LanceDB's default name, `{column}_idx`, which is the name it
creates itself.
- A column already carrying the declared type is skipped, so a database with the
full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.
It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.
The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.
`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
