Replaces the rank-interleaved fusion with a straight merge of the union
sorted by each candidate's raw retrieval score, ties still falling to
declaration order, cut at limit. This is the depth-quota candidate: within
one database rank order and raw-score order coincide, so the whole
difference from the tie-break fix is the cross-database interleaving.
This branch exists to be measured and discarded. The rank-dominance tests
in tests/multi_db/test_search.py pin the behaviour this deliberately
inverts, so the root suite FAILS here by design. If the arm wins, the
shipping implementation belongs to the session that owns client/search.py,
with its own tests.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Deleting or replacing every indexed row while unindexed rows remain
returns lance to the zero-coverage scan path. ensure_indexes rebuilds
a declared FTS index that covers none of a populated table's rows,
and delete_by_document_id runs index maintenance like the other chunk
writes. A legacy database in that state is repaired by its first write.
An FTS index built over an empty table indexes nothing and lance never
catches it up on add. ensure_indexes skips FTS while the table is
empty; create, replace_for_document and embed-only rebuild ensure
indexes after writing, so the index always covers at least its first
rows. The first write into a fresh table writes one extra chunks table
version for the index build; a failed build fails the write.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
Comments that narrated a failure mode across three or four lines say the
invariant they protect instead: the repository's late embedding, one database
keeping its hybrid scores, one over-fetch decision for a selection, and a cite
fallback that covers exactly what the question covers.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
`_resolve_scope` returned a scope without keeping it, so a client re-entered
after its configuration was edited covered whatever the configuration then said.
Resolving once is what the rest of the design rests on: the scope is what names
results, citations and errors.
A configuration is still free to change before first entry.
`asyncio.gather` propagates the first failure while its siblings run on, and the
caller unwinding from that closes the set through `async with` — so a sibling
still reading reads through a closed session. Seven fan-outs were affected:
lookup, search, image enrichment, multimodal picture loading, context expansion,
document listing and counting, and the sandbox's document load.
`gather_all` cancels and drains the rest, then re-raises the original exception.
A `TaskGroup` would drain them too but raise an `ExceptionGroup`, which every
caller and both CLIs' exception handlers would have to unwrap. The two
`return_exceptions=True` gathers in session opening and teardown already drain
their children and are left alone.
The set shares the reranker its facades borrow, not an embedder: each covered
database builds and closes its own. Sharing one needs the lender pattern inside
`Store`, since a store's embedder also serves writes and is closed with the
session, so it is left as follow-up rather than done here.
`reported_location` returned `client.location` and nothing else; its test
asserted that a mock returns what it was given. `list_documents` names its
results through `name_all`.
Reads tagged the document they returned and writes did not, so creating in
`alpha` came back with `source=None` while reading the same document came back
with `"alpha"`. Every outward write result goes through the owning session.
A name nothing covers raised `KeyError` in four places and
`AmbiguousDatabaseError` in a fifth, so a caller had to catch both and neither
name said what happened. `UnknownDatabaseError` is all of them, exported from
`haiku.rag.store` beside the other errors.
It subclasses `KeyError`, since selecting by name is a lookup, and prints its
message plainly rather than quoted as a missing key. Both CLIs turn it into the
same clean exit they already gave the others.
`close()` refused a client covering a set, which left one with no method to
call: `async with` was the only lifecycle it had. `aclose()` runs that teardown
for a caller that owns the client some other way, whatever it covers, and
nothing to release is not an error, so it is safe before entering and after
closing.
`close()` stays what it is, one connection and nothing else, and says so:
draining the background vacuum and releasing the embedder and reranker are
awaitable.
`ask(sources=["typo"])` reached the model, which discovered the name only if it
searched: requests spent on a selection that could never answer, and a run that
never searched answered anyway.
Checked by name, not by opening: a client covering a set opens a database when a
query reaches it, and validating by opening would open every one of them before
any search, letting a database nobody asked about fail the run.
Every client covering a database built its own, so a set of five loaded the same
local model weights five times over, beside the federator's. A client covering a
database for another borrows that one's, and only its owner closes it.
`sources` was ignored beside a database path, so `sources=["nope"]` opened the
path and read as though the selection had been honoured. A path and a name
already conflict inside `DatabaseScope.resolve`; this is the same rule where a
caller can reach it.
`reader_for` returned itself before looking at the name, so an "alpha" client
answered `reader_for("beta")` with alpha's reader. A citation naming another
database would have been read from the wrong one.
Both paths decide through `clients_covering` now, so one database refuses a
wrong name the way a set already did.
The chat citation test said mounting lends the client without showing it; a
chat test shows it and the docstring points there.
`reader_for` documented None as covering any database it could not place, but a
name outside the set raises `KeyError` like `clients_for` does: provenance
naming a database this client does not cover is wrong rather than absent. None
means one thing, a federated client given no name.
`sources=[]` means two things. On a search it selects nothing to search; on the
constructor it raises, since a client over no database can do nothing. Both are
written down now.
A capability reads through a lent client, so what a citation records is that
client's database and not the scope the capability was built with. Chat lends
one, and had no test saying so.
`sources=["alpha"]` on a client covering a set went through fusion, which
scores position: a result the database ranked at 0.6549 was reported as
1/(60+rank). Embedding also moved ahead of the repository, so a filter matching
no document embedded the query anyway.
A selection of one now runs the single-database search. Fusion reconciles
rankings from separate indexes, and one ranking has nothing to reconcile.
A reranker that returns chunks it built rather than the ones it was given loses
which database each came from, since ownership is by identity. That is named
now instead of surfacing as a KeyError, and stated on `RerankerBase._rerank`.
A URI-backed database is constructed with a local `db_path` nothing connects
to, and the info modal reported that, so `--db-name` on an S3 database named a
local directory. `location` reads the configuration instead, and the modal
labels it as one.
`InfoModal`'s `db_path` argument is gone: both callers passed None, so only a
test reached the branch that used it.
`sessions_for` recorded what opened from `gather`'s results, which arrive only
when it runs to completion. Cancelling it discarded them, so a database that
opened while a sibling was still pending was never recorded and `aclose` never
closed it. `return_exceptions=True` covers a failing child, not a cancelled
parent.
`_open` now registers its own session, so what opened is reachable however the
fan-out ends.
`HaikuRAG.__init__` said an omitted `db_path` uses `storage.data_dir`, which
is the last of three; and that `sources` is ignored for a single `uri`, where
it raises, since only `lancedb.databases` names databases.
The name is not the only identity leaving the configuration: results,
citations, model input and errors opening a named database carry it, while
`info`, `init` and `tag` print the location. `sources=[]` returns no search
results, but `ask` and `analyze` still answer, without evidence.
Trim the comments that still narrated a failure or an alternative to the
invariant they were there for.
`covers_several_databases` had one line of body and two call sites.
`_db_path_given` guarded a default path manufactured in `__init__` and
overwritten in `__aenter__`; `_requested_db_path` is what the caller asked
for, and the effective path falls out of the resolved scope.
One sentence for the contract, one or two more only where an invariant is not
obvious. The reasons that stay are about correctness and ownership: which
database a result belongs to, who closes what, why assembly order is the
tiebreak. The ones that go narrated how the code got here.
`clients_for` hands back a client over a database the covering one owns, so
`async with` on it opened a second session and assigned it, and teardown
declined to close what this client did not open. Entry returns the client as
it stands.
Closes#582. `lancedb.databases` entries already classify a location by
whether it carries a scheme; `lancedb.uri` was taken as a URI whatever it
said, so a local path was opened as object storage and a mistyped one
became a new empty database instead of failing. Both settings now place a
database the same way, and `--db PATH` overrides either.
A chunk id is unique within a database and says nothing across them, so a
database copied from another holds the same ids. `qualified_id` keys the
two in-memory identity sites on the database and the id together:
`merge_results` was dropping the second database's result when a query
repeated, and the arrival map that breaks fused score ties was ranking one
of the pair as the other.
Everything serialized records the id alone, so there ambiguity is refused
rather than qualified. `resolve_citations` raises `AmbiguousCitationError`
for a cited id held by two of the databases searched, where it used to
resolve to whichever result came last; `_register_citations` raises for one
already cited from another database in an earlier question. `_cite` turns
both into a `ModelRetry` asking for other evidence. The direct-id fallback
asks every database the question covers instead of taking the first that
answers, so an id no search returned is refused on the same terms.
`all_found` collects them and `first_found` reads its first, which document
reads keep doing on purpose.
Also drop a duplicated 0.77.0 heading from the changelog.
Expansion branched on whether the client covered a set, so the
single-database half reached for repositories through a facade that may
have none. `expand_sources` groups results by database and hands each
group the session that owns it; `expand_context` and `visualize_chunk`
take that session, so `visualize_chunk` stops narrowing to one database
and discarding the result.
Inline `_fetch`, a pass-through to the chunk repository.
The CLI decides only what it knows — that --db and --database are the same
thing said twice, and whether a command reads more than one — and hands the
resolved scope down. Nothing rewrites the configuration, so a named database
keeps the name results and citations carry, and a remote one opens the URI
it was configured with rather than the local path standing in for it.
HaikuRAGApp, ChatApp and InspectorApp take that scope and nothing else.
Selection reaches the client through a private constructor, so the public
signature still takes a path or names.
covers_multiple, source_names, source and reader_for replace the private
state seven modules were reading to work out how many databases they had.
The configured selection is kept intact, so entering a client twice derives
the same database rather than the last derivation.
Every implementation in documents.py and rebuild.py takes the session it
writes to, so a set cannot reach one: the facade narrows once and passes the
database on, rather than checking and carrying a union. Tests calling an
implementation directly go through `writing()`.
One session field, whichever kind it is, with the covered databases derived
from it rather than kept beside it. `_single_session` returns the database a
write works on in place of ten guards that only asserted one existed, and the
single-database operations move to the session that owns them.
FederatedSession opens the databases a query covers and owns their
teardown; the client keeps the wrappers it hands out. A wrapper releases
what it built and never closes the database it borrowed.
SingleDatabaseSession owns the store, the repositories and the vacuum
machinery, so nothing above has to ask whether it has a store. The client
keeps every name callers already use.
DatabaseScope.resolve reads configuration and at most one selector; a
DatabaseRef carries the configured name and a location already resolved to
a URI or a path, so a path a caller names is never reinterpreted. Nothing
consumes it yet.
An embedder is a function of configuration, not of a database, and the
databases in a selection share one, so a client covering a set builds it
on first use and closes it on teardown. Operations that need one database
say so instead of surfacing a missing store.
An image query has no text to match, so it is vector-only whatever the
caller asked for, and full-text search embeds nothing, so it needs no
agreement on embedders.
`first_found` replaces the client's private version and the capability's
sequential one. The info modal reports through the connection the client
already holds.
Each database owns an embedder, so embedding per database cost a round
trip each. One database still embeds inside the repository, which returns
early for a filter that matches nothing.
A blank name is falsy, so source routing reads it as absent, and a blank
location resolves to the working directory. `is_read_only` reports the
mode the client was opened with, which a client covering a set can answer
without a store.
Document.source names the configured database, as SearchResult and
Citation already do. A listing spanning databases is unreadable
without it, and `--database NAME list` could not name the one it
opened.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
The modal listed every document and mounted a checkbox per document, so a
corpus of tens of thousands never finished rendering: 67k sequential
mounts across two databases, and the same for one database that size.
Titles repeat at that scale too, and the ids were derived from the title,
so they collided.
It shows a page of 200 now, mounted in one call and identified by
position, and the search box asks the database for the rest on enter,
matching titles and URIs. Typing still narrows the page on screen, for
feedback while typing. `search_filter` escapes the term, which is
whatever was typed.
A federated listing takes its window across the databases rather than
filling it from the first one: concatenating hid every database after
whichever was listed first, which for a set of a thousand papers and
sixty thousand articles meant a page of papers alone. Sorting would not
have helped, since document ids and article titles sort into separate
runs, so the page is picked by interleaving and the modal sorts it for
display.
Chat answers with the same capabilities `ask` does, so it federates as
naturally as `ask` and `analyze` — but it went through the one-database
guard and refused a configured set outright, which left no way to chat
across several databases.
The guard was the visible half. `run_chat` also defaulted `db_path` to the
single default path whenever it was None, so lifting the refusal alone
would still have opened one database. It now leaves the path unresolved
when `lancedb.databases` names the set, and the client resolves it.
Listing and counting documents fan out over the set, which is what the
document filter reads, and visual grounding resolves the database holding
the cited chunk through the citation's source: chunks, pages and bounding
boxes all come from that one database. A limit on a listing means that
many documents in total, not that many per database.
The info modal reports every database it covers, each under its
configured name and without its location, since names are the only
identity that leaves the configuration. `database_lines` is what one
database reports about itself, shared by both paths, and it reports a
failure as a line so one unreachable database does not cost the report on
the others.
`inspect` stays a one-database command. It browses one database's
documents and chunks, so a set has nothing to show it.
The CLI reports this error, and importing it from the settings repository
pulled lancedb onto the CLI's import path. Deferring the import into cli()
bought nothing, since cli() runs on every invocation: it cost about 1.9
seconds on a cold start, `--help` included.
It now sits beside the other store exceptions, in a module that imports
nothing, and every importer points there.
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.
The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.
The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.
On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
`ask(sources=[…])` scopes a question to some of the configured databases, carried
on the capability state so its search tool searches those. `Citation.source` names
the database a cited chunk came from, resolved from the search results the model
saw, which already carry it.
Context expansion routes each result through the database it came from: a
federating client has no repositories of its own.
The cite fallback, which looks up an id absent from this run's results, searches
only the selected databases. A chunk id says nothing about which database holds
it, so placing one means asking, and asking outside the selection would let a
question scoped to some databases cite another.
The loosely-specced client mocks in the capability tests now say they stand in for
a single-database client. A bare AsyncMock answers any attribute with a truthy
Mock, so `_federated` sent the fallback down the multi-database branch, and
`_source` reached a validated field.
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.
Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.
A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.
Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.
The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.
A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.
With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
