The chunks FTS index is created once when the table is created, over zero
rows, and nothing folds later rows into it but an optimize. build_databases
bypasses populate_db, and with it the closing vacuum, so every database it
built had an index covering nothing.
Full-text search then returns near-arbitrary rows while still returning
results, so nothing looks wrong: measured FTS recall@5 of 0.0000 across 208
queries on a 40k pool, with hybrid losing 9.9pp to pure vector because the
dead branch still consumed half the fused slots.
Reachable outside the eval too, on any storage.auto_vacuum: false, which
every reference config sets. Reproduced through create_document alone:
five documents, five chunks, zero indexed rows.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Hybrid search inside one database fuses its own vector and FTS rankings
with lancedb's RRFReranker over exactly the requested limit, and both
branch queries derive from the same inner query, so there is no
branch-depth knob. Below roughly 50 candidates the two rankings stop
overlapping, nothing sums, and the fusion degenerates: measured recall@5
on a single database was 0.000 at fetch 5, 10 and 20, then 0.267 at 50 and
0.350 at 100.
A dataset's retrieval_limit therefore fixes which regime it measures, and
comparing regimes would otherwise need one dataset per depth.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.
The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.
The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.
build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
The newest release with wheels for every supported platform: 0.38.0
publishes none for x86_64 Linux or Windows. The zero-coverage FTS scan
path is still broken at 0.37.1, so the guard, prevention and repair
carry unchanged. ensure_indexes names the index it declares, keeping
replace deterministic on 0.38+, where an unnamed create_index builds a
suffixed sibling instead of replacing a different-typed index. An
invalid search filter raises ValueError.
Deleting or replacing every indexed row while unindexed rows remain
returns lance to the zero-coverage scan path. ensure_indexes rebuilds
a declared FTS index that covers none of a populated table's rows,
and delete_by_document_id runs index maintenance like the other chunk
writes. A legacy database in that state is repaired by its first write.
An FTS index built over an empty table indexes nothing and lance never
catches it up on add. ensure_indexes skips FTS while the table is
empty; create, replace_for_document and embed-only rebuild ensure
indexes after writing, so the index always covers at least its first
rows. The first write into a fresh table writes one extra chunks table
version for the index build; a failed build fails the write.
lance serves unsorted results with matching rows dropped when a
declared FTS index has indexed nothing, or when a populated table has
no FTS index at all. doctor fails on both: vacuum remediation for an
existing index, rebuild --embed-only for a missing one, since optimize
never creates an index. The chunk repository warns once per repository
on the first FTS or hybrid search against either state; a failing
coverage check is logged and never fails the search. Removes
_ensure_fts_index, which had no callers.
Both were exempt from coverage, which is how the 0.38.0 blob-encoding bug
reached a release: nothing exercised the chain that produced it.
Move the historical `documents` shapes into tests/store/legacy_documents.py
so each migration's tests can seed the table as its predecessor left it.
v0.25.0 compresses through compress_json, which switched from gzip to
zstd in 0.38.0 — the same release that added this migration. v0.38.0
decompressed with gzip and fell back to a raw UTF-8 decode, and both
fail on a zstd frame, so upgrading any database older than 0.25.0 has
died with UnicodeDecodeError since the migration shipped.
Reported with a fix by @omaer0 in #586.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
Sandbox.close released the pool and the held federated client only when
the monty session's __aexit__ returned. Each release now runs under
suppress(Exception), matching _discard_session and aclose_quietly, so a
raising step neither masks an unwinding error nor leaks the databases a
federated `_opened` holds.
The chat filter modal keys a selection by (database, id): copies of a
database share document ids, and checking one copy left the other
reading as selected. Applying the filter narrows the question's
`sources` to the databases the selection names. A twin id inside
another selected database still matches there: a serialized id filter
cannot carry a source.
A database name, document title, uri or heading containing Rich markup
crashed search output, chat citations and the chat document filter with
MarkupError. Every interpolation into markup-parsed text now escapes.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
Comments that narrated a failure mode across three or four lines say the
invariant they protect instead: the repository's late embedding, one database
keeping its hybrid scores, one over-fetch decision for a selection, and a cite
fallback that covers exactly what the question covers.
`tag_restore` reached into `HaikuRAGApp._display_path` and `_path`, and seven
places spelled out `self._is_local and not self._path.exists()`. `display_path`
and `database_missing` say both, and `database_missing` is False for a database
behind a URI, which has no path to check.
`init` keeps its own check: it asks the opposite question.
A database name, a location and an exception message all reach the info modal
as configuration-derived text, and Rich reads `[...]` in any of them as markup:
a name like `beta [prod]` disappeared, and a message carrying `[Errno 2]` or a
stray closing tag could break the line it sat in.
Two collections can hold documents with the same title, and the collapsed
citation carried only that title, so the two read as one source repeated. The
name is appended when the client covers a set, the decision already made for
search results and retrieved images.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
`_resolve_scope` returned a scope without keeping it, so a client re-entered
after its configuration was edited covered whatever the configuration then said.
Resolving once is what the rest of the design rests on: the scope is what names
results, citations and errors.
A configuration is still free to change before first entry.
Search results and capsule entries name the collection they came from; the
images attached beside them carried only the chunk id and reference. Two
collections can return the same picture of the same document, so the two
labels were identical and the model could place neither.
The decision is the one already made for the text: `covers_multiple` at the
generic search tool, and the flag `search_corpus` computed for the capability
tools, which it now returns.
`asyncio.gather` propagates the first failure while its siblings run on, and the
caller unwinding from that closes the set through `async with` — so a sibling
still reading reads through a closed session. Seven fan-outs were affected:
lookup, search, image enrichment, multimodal picture loading, context expansion,
document listing and counting, and the sandbox's document load.
`gather_all` cancels and drains the rest, then re-raises the original exception.
A `TaskGroup` would drain them too but raise an `ExceptionGroup`, which every
caller and both CLIs' exception handlers would have to unwrap. The two
`return_exceptions=True` gathers in session opening and teardown already drain
their children and are left alone.
Three places identified a picture by document and reference alone, and one
identified a citation's images by chunk id alone. Both repeat between copies of
a database, so a search returning a figure from two collections sent one, a
capsule retained one, and a citation rendered the other collection's figures.
Keyed on the source as well: `(source, document_id, self_ref)` for search
pictures, the capability and source for retained ones, and `qualified_id` for
the chat's citation images.
A capsule replaces earlier questions' evidence with what they cited, so the
searches that carried a `Collection:` line are gone by the time the model reads
it. Cited content survived; which collection it came from did not, and a
follow-up question attributed it to whichever the model guessed.
Named the same way a search result names it: the capsule decides, and only when
it spans more than one.
`ConfigMismatchError` and `SourceUnavailableError` escaped the entry point as
tracebacks rather than a message and exit code 1. The entry-point test is
parametrized over the translated types, so the two CLIs' lists cannot drift
apart silently again.
Typing narrowed the mounted checkboxes without touching the search the listing
was built from, so a term matching more than one page hid rows from whichever
page the user happened to be on, left the rest of the matches a page away, and
paged on the previous search while the term still sat in the box. The term now
applies on enter, and the footer says so until it does.
The set shares the reranker its facades borrow, not an embedder: each covered
database builds and closes its own. Sharing one needs the lender pattern inside
`Store`, since a store's embedder also serves writes and is closed with the
session, so it is left as follow-up rather than done here.
`reported_location` returned `client.location` and nothing else; its test
asserted that a mock returns what it was given. `list_documents` names its
results through `name_all`.
Four passages said every missing database raises `FileNotFoundError` and that
errors show locations. A database named in `lancedb.databases` raises
`SourceUnavailableError` instead, naming the database and not its location,
which is the point of naming them. A path you gave keeps `FileNotFoundError` and
still shows the path.
The lock was applied to an owner as well, so every owner-backed file read queued
behind the capability's tool calls to guard state it does not touch. An owner is
a session of its own and is yielded straight through, which is what the
docstring already claimed.
Chat asked the covering client for a picture with the citation's source, which
raises when there is none, losing the answer to one figure. It resolves the
reader first and omits the bytes when the database cannot be placed, as the rich
formatter already did.
Reads tagged the document they returned and writes did not, so creating in
`alpha` came back with `source=None` while reading the same document came back
with `"alpha"`. Every outward write result goes through the owning session.
`settings` and `download-models` read the configuration and open nothing, but
resolved a scope to be constructed, so `--db-name nope` failed them over a
selection they never use. `HaikuRAGApp` takes no scope for those, and asking it
for one is an error rather than a silent default.
`--db-name` selects one database, not a subset: it is not repeatable.
The reranker recommendation was one-sided. It reports both measured effects now:
stronger aggregate retrieval across shards, and weaker attribution between
near-identical documents, where fusion keeps twins apart because each database
contributes its own top-ranked result.
Image queries are vector-only and skip the reranker.
The changelog described components of the new feature as fixes to the last
release. They are one Added entry, and the changes a user upgrading does see —
`settings` printing YAML, the document filter paging and searching, `list`
printing only the fields a document has — are listed.
`test_an_image_query_builds_no_reranker` never submitted an image query: it
opened clients and found nothing built, which is lazy construction. It sends one
now, so consulting the reranker before the query type fails it.
`TestComparingEmbedders` called `_require_one_embedder` and asserted nothing.
It rejects a disagreement and accepts an absent record, and searches through.
Three `ModelRetry` tests took any reason where the reason is the subject.
`haiku-ingester` reached the client's refusal, which names `sources=[name]`, a
Python argument no CLI user can pass. It refuses a configured set itself, with
`--db PATH` and the one-ingester-per-database layout.
`haiku-rag`'s own refusal now says where each option goes: `--db-name` is global
and precedes the command, `--db` follows it.
Creating re-read the settings blob and left `stored_embedding` at None, so a
client that created a database compared as though it recorded no embedder.
`_remember_settings` takes both, and the comment no longer says one follows the
other.
The lock claimed to serialize a whole read, while `_documents` resolves the
owners under it and then reads through them outside. That is right: resolving
the owners is an operation on the shared connection, reading through them is
not, and each owner is a session of its own. Holding the lock across a read per
database would serialize them against the capability's searches to guard state
none of them touch.
