Collapsing the caption text into `get_pictures_grouped` served the enrichment
path, which uses it, but the multimodal reranker discards the second return value
while still paying to read the column. That is the widest fan-out in the codebase,
`limit * 10` candidates, and it previously projected self_ref and picture_data
alone.
`with_text` is opt-in and off by default, so the cheap projection is what a caller
gets unless it asks for more. The reranker test asserts the projection as well as
the query count, since a count alone would not notice the column coming back.
Splitting the assembly into a passthrough pass and an expandable pass reordered
equal-scored results: the score sort that follows is stable, so the order results
arrive in is the tiebreak. Results are assembled in document_groups order again,
after the batched fetch rather than around it.
Also ports the caption negative cases the removed single-document test carried: a
table's caption and an ordinary text reference map to no picture.
`resolve_refs`, `get_items_in_range`, `get_caption_picture_refs` and
`get_all_items_grouped` have no callers left: the grouped equivalents serve every
path that used them. `get_all_items_grouped` had none even before this branch.
Tests whose subject was a removed method go with it. Tests that only used one to
fetch a fixture now use the grouped call, so what they assert is unchanged.
`_attach_picture_data` fetched picture bytes once per document, over the
`limit * 10` candidates reranking asks for, so it was the per-document fetch with
the most candidates behind it. It now issues one query however many documents the
candidates span: one for ten documents, as for one.
Removes `get_text_for_refs`, whose only caller now gets the text back with the
bytes from `get_pictures_grouped`.
`test_client_search_include_images_false_skips_lookup` returned no search
results, so asserting the picture accessor went uncalled held whatever the code
did. It now returns a picture-carrying result, making "did not fetch" the
assertion rather than "had nothing to fetch".
`expand_with_items` fetched its own inputs per document: one query to resolve
refs to positions, one for the window of items around them. A result set spanning
N documents cost 2N queries, which was 10 of the 18 measured for a limit=5 search
on a remote object-store corpus.
`expand_context` now does both fetches once for every document it is expanding,
and `expand_with_items` takes the positions and items it needs. Two queries for
one document, and two for five.
Each document keeps its own inclusive window in `get_items_in_ranges`. Positions
repeat across documents, so a shared range would splice one document's items into
another's context.
`_populate_image_data` ran its stages once per result document, so a result set
spanning N documents cost 4N `document_items` queries. Measured on a remote
object-store corpus, a limit=5 search with expansion was 18 queries, 16 of them
against `document_items`.
The stages now run once each across every document, and flat in document count:
two queries for the dependent caption-to-picture mapping when results ranked on a
caption, one for the picture bytes. Two queries for a picture-ref result set,
three at most.
Picture text comes back with the bytes rather than from a second query, since it
is on the same rows.
Predicates are per document, `(document_id = 'a' AND self_ref IN (…)) OR (…)`,
rather than `self_ref IN (union)`. self_ref and position values repeat across
documents, so a union predicate would return other documents' rows: for
picture_data that fetches blobs nobody asked for, and it can hand one document
another document's picture.
`client.ask` and `client.analyze` built their capability from a db_path, so the
capability opened a second connection to the database the client already had
open, once per call.
Ownership is now explicit rather than inferred. `rag` stays the connection the
capability opened and must close; `borrowed_rag` is a caller's, which
`_ensure_rag` prefers and `_close` never touches. Two fields rather than a flag,
so closing a borrowed connection is not expressible.
`for_run` still clears `rag` per run, since a run owns what it opens. It leaves
`borrowed_rag` alone: that connection belongs to the caller and outlives the run.
All ten tool bodies opened their own `HaikuRAG`, so every tool call paid a
connection open and, on object storage, refetched the index the previous call had
just cached. The client is now opened once, lazily so that calling a tool
function directly still works, and eagerly from the lifespan so an unopenable
database fails startup instead of every call. Teardown clears the cached client
in a finally, since `_lifespan_manager` can be re-entered and would otherwise
hand out a closed connection, including when the close itself fails.
`delete_document` no longer opens its own connection with `skip_validation=True`.
Keeping it separate broke consistency once connections became long-lived: the
delete committed on one connection while reads served from another, which with a
30s consistency interval showed the deleted document as still present. A
connection always sees its own writes, so sharing one is what makes delete
visible to the next read.
So the server no longer opts out of embedding-config validation. Drift that
validation rejects now fails MCP startup, where before the server started and
only `delete_document` worked while every read returned empty. Same-dimension
identity drift still starts a read-only server, matching every other read verb.
Delete under drift is now a CLI operation; CLAUDE.md and the CHANGELOG record it.
Opening a database ran `list_tables` three times, opened the settings table
twice, and read and parsed the same settings row three times: once for the stored
vector dimension, once for the version behind the migration check, and once for
config validation. On object storage each of those is a round trip.
`_initialize` now reads both once and threads them down. `_init_tables` and
`_check_migrations` take what it read instead of fetching their own copy, and
`validate_config_compatibility` accepts the settings it should compare against,
still reading for itself when called directly.
Passing the pre-init read to validation is equivalent: nothing between the read
and the validation rewrites `embeddings`, which is all it compares.
The settings read no longer swallows every exception. It did before, when the
only consequence was falling back to the configured vector dimension; now the
same empty result feeds the migration check, where it would read as version
0.0.0 and declare every migration pending. Only decode failures are tolerated,
and a decoded non-object normalizes to {} rather than reaching callers that
expect a mapping.
Every `Store` built its own connection with its own caches and discarded them on
close, so the index a vector query loads was refetched by the next connection.
On object storage that first fetch dominates: measured on a ~500k-chunk 2560-dim
corpus over a ~200ms link, the first query cost ~41s and the second ~3s, and a
new connection reusing the session cost ~7s instead of ~47s.
`connect_lancedb` now passes a process-wide session, keyed on the configured
cache sizes so a caller asking for different sizes gets its own.
Also sets `read_consistency_interval`, defaulting to 30s. It was None, meaning a
connection never re-checked for other processes' writes. Per-call connections hid
that; a shared session makes connections long-lived enough for a reader to go
stale against the ingester.
All three settings reject negatives at the config boundary. A negative cache size
raises OverflowError and a negative interval panics inside Lance, so neither is
catchable further in. Zero stays valid for both: no cache, and check on every
read.
The routing tests now assert the kwargs they care about rather than the full call
signature, since every connection carries the two new kwargs.
Current results moved above Methodology and Running Evaluations. A reader arriving from an external link met four screens of CLI flags and download instructions before any number.
Benchmarks is promoted to a top-level nav entry, out of Reference, which holds Development and Changelog.
The page move itself changes no wording or figures.
Also drops two things that were stale rather than merely verbose: the CLI docs
note for 0.75.0, which was the only release-tagged note in the docs tree while
the CHANGELOG already records that existing databases need `haiku-rag migrate`;
and "(created or corrected)" from the migration log line, left over from the
earlier behaviour that replaced wrong-typed indexes.
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.
`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:
- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
be deliberate, serving different query shapes, so an index this function did
not declare survives even on a column it does. The one thing it overwrites is
an index at LanceDB's default name, `{column}_idx`, which is the name it
creates itself.
- A column already carrying the declared type is skipped, so a database with the
full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.
It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.
The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.
`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
`_init_tables` left `chunks.id`, `chunks.document_id` and `documents.id`
unindexed, so those lookups scanned the column. On object storage that is
network I/O per query, on paths that run per document: citation lookup,
delete-by-document, the re-ingest merge, and every dedup probe.
Declare the index set per table in `index_specs()` and apply it through
`ensure_indexes()`, which skips a column only when it is already indexed with
the declared type. Both halves of that are load-bearing. Skipping is required
because `create_index(replace=True)` rebuilds an identical index, writing a new
index and a new table version and orphaning the old files until the next vacuum.
Comparing the type is required because column coverage alone would let a
wrong-typed index stand, and a BTree on `label` silently loses the
low-cardinality equality lookup the Bitmap is there for.
Columns not declared for a table are left alone, so an externally created index
such as a vector index on `chunks` survives.
`_init_tables`, `recreate_embeddings_table`, `ChunkRepository.delete_all` and
`DocumentRepository.delete_all` now all route through it instead of repeating
their own subsets.
Also recreate `document_items` from `get_document_items_arrow_schema()` in
`DocumentRepository.delete_all`, which was using the LanceModel and so returned
`picture_data` as 32-bit `binary`.
Existing databases are unchanged; the migration follows separately.
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.
Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
_store_documents_with_chunks embedded each document's chunks in its own
embed_chunks call; chunks missing embeddings are now flattened across the
whole batch, embedded in one pass honoring embeddings.batch_size, and
assigned back positionally.
No `DatasetSpec` declared `search_filter`, so `resolve_search_filter`
and the `--filter ""` clearing rule reconciled the flag against a
default that never existed. The flag alone covers the case. An empty
clause reaches `ChunkRepository.search`, which already treats it as
unfiltered.
Rename to `document_filter` throughout, matching
`run_capability_question`'s parameter and the metadata key that lands
in Logfire.
`_stub_spec` merges its overrides, so a test can override a loader
instead of rebuilding the whole spec.
The `--filter` examples used `uri LIKE '%arxiv%'`, which matches no
`orb_text` document: its URIs are bare arXiv ids such as `2407.01528v3`.
A clause that matches nothing scores MAP 0 rather than erroring, so the
example failed silently.
`await_args` is typed `_Call | None`, so subscripting it fails
`ty check`; `call_args` carries the same call for an AsyncMock.
