Three groups, not two: `search`, `ask`, `analyze` and `chat` cover the set,
`settings`, `init-config` and `download-models` open no database, and every
other command works on one — or on a configured set of one, which is
unambiguous and keeps its name. A database named in `lancedb.databases` keeps
that name whether or not it is the only one covered; only `lancedb.uri` places
one without naming it.
Document ids repeat between copies of a database, where the sandbox refuses a
duplicate but the chat filter's `id IN (...)` matches the document in every
copy. `build_document_id_filter` claimed ids never widen a selection.
Document the facade: `covers_multiple`, `source_names`, `source`,
`reader_for`, `clients_for`, the lifetime of a borrowed client, and
`sources=None` against `sources=[]`.
Drop the vision callout from the README, which the features list already
covers.
`clients_for` hands back a client over a database the covering one owns, so
`async with` on it opened a second session and assigned it, and teardown
declined to close what this client did not open. Entry returns the client as
it stands.
They opened a client on `tmp_path / "unused"` with the bucket in the
configuration. An explicit path now selects the database, so the client
tests ran against the local disk and the two app tests failed. Nothing names
a path any more, and both helpers assert the connection is remote before
yielding, so a later precedence change cannot quietly localize them again.
0.78.0 released `api_key` and two authorization fixes out of the section
this branch was still adding to, so the automatic merge filed every
multi-database entry under it and dropped the `0.77.0` heading, which both
sides had de-duplicated. Everything from `0.78.0` down is main's record
verbatim; the multi-database entries stay under Unreleased.
`resolve_db_path` manufactured the default path whenever `lancedb.databases`
was empty, and `covers_several_databases` read coverage back out of the
configuration, so a capability built without a client opened
`storage.data_dir/haiku.rag.lancedb` instead of what `lancedb.uri` placed.
The entry point resolves a `DatabaseScope` instead: instructions ask it what
it covers and `_ensure_rag` opens it through `HaikuRAG._covering`, so
coverage is decided once rather than encoded in a path and re-derived.
`Sandbox._covering` takes the scope the capability already resolved, beside
the public constructor that takes a path. The factory signatures are
unchanged.
Closes#582. `lancedb.databases` entries already classify a location by
whether it carries a scheme; `lancedb.uri` was taken as a URI whatever it
said, so a local path was opened as object storage and a mistyped one
became a new empty database instead of failing. Both settings now place a
database the same way, and `--db PATH` overrides either.
`list_documents()` reached into the ownership map for a name the document
already carries. The map is still built, so a document id two databases
claim is still refused before anything mounts. A listing over one named
database now reports that name, as in-code `search()` already does.
`Store` already parses the settings blob on open, so keep it: the
inspector's info modal was opening the settings table, querying it and
parsing the JSON a second time, and `doctor` was asking
`SettingsRepository` for it on a store it already held. Creating a
database refreshes it, so a new one reports the version init wrote.
`gather_database_info` keeps its own parse: it goes around `Store` so a
pre-migration database still reports what it can.
A chunk id is unique within a database and says nothing across them, so a
database copied from another holds the same ids. `qualified_id` keys the
two in-memory identity sites on the database and the id together:
`merge_results` was dropping the second database's result when a query
repeated, and the arrival map that breaks fused score ties was ranking one
of the pair as the other.
Everything serialized records the id alone, so there ambiguity is refused
rather than qualified. `resolve_citations` raises `AmbiguousCitationError`
for a cited id held by two of the databases searched, where it used to
resolve to whichever result came last; `_register_citations` raises for one
already cited from another database in an earlier question. `_cite` turns
both into a `ModelRetry` asking for other evidence. The direct-id fallback
asks every database the question covers instead of taking the first that
answers, so an id no search returned is refused on the same terms.
`all_found` collects them and `first_found` reads its first, which document
reads keep doing on purpose.
Also drop a duplicated 0.77.0 heading from the changelog.
Expansion branched on whether the client covered a set, so the
single-database half reached for repositories through a facade that may
have none. `expand_sources` groups results by database and hands each
group the session that owns it; `expand_context` and `visualize_chunk`
take that session, so `visualize_chunk` stops narrowing to one database
and discarding the result.
Inline `_fetch`, a pass-through to the chunk repository.
The CLI decides only what it knows — that --db and --database are the same
thing said twice, and whether a command reads more than one — and hands the
resolved scope down. Nothing rewrites the configuration, so a named database
keeps the name results and citations carry, and a remote one opens the URI
it was configured with rather than the local path standing in for it.
HaikuRAGApp, ChatApp and InspectorApp take that scope and nothing else.
Selection reaches the client through a private constructor, so the public
signature still takes a path or names.
covers_multiple, source_names, source and reader_for replace the private
state seven modules were reading to work out how many databases they had.
The configured selection is kept intact, so entering a client twice derives
the same database rather than the last derivation.
Every implementation in documents.py and rebuild.py takes the session it
writes to, so a set cannot reach one: the facade narrows once and passes the
database on, rather than checking and carrying a union. Tests calling an
implementation directly go through `writing()`.
One session field, whichever kind it is, with the covered databases derived
from it rather than kept beside it. `_single_session` returns the database a
write works on in place of ten guards that only asserted one existed, and the
single-database operations move to the session that owns them.
FederatedSession opens the databases a query covers and owns their
teardown; the client keeps the wrappers it hands out. A wrapper releases
what it built and never closes the database it borrowed.
SingleDatabaseSession owns the store, the repositories and the vacuum
machinery, so nothing above has to ask whether it has a store. The client
keeps every name callers already use.
DatabaseScope.resolve reads configuration and at most one selector; a
DatabaseRef carries the configured name and a location already resolved to
a URI or a path, so a path a caller names is never reinterpreted. Nothing
consumes it yet.
An embedder is a function of configuration, not of a database, and the
databases in a selection share one, so a client covering a set builds it
on first use and closes it on teardown. Operations that need one database
say so instead of surfacing a missing store.
An image query has no text to match, so it is vector-only whatever the
caller asked for, and full-text search embeds nothing, so it needs no
agreement on embedders.
`first_found` replaces the client's private version and the capability's
sequential one. The info modal reports through the connection the client
already holds.
A title repeats within a corpus and across databases, so a substring
match on the displayed name widened the filter to documents the user did
not pick. The label names the database.
Each database owns an embedder, so embedding per database cost a round
trip each. One database still embeds inside the repository, which returns
early for a filter that matches nothing.
A blank name is falsy, so source routing reads it as absent, and a blank
location resolves to the working directory. `is_read_only` reports the
mode the client was opened with, which a client covering a set can answer
without a store.
Document.source names the configured database, as SearchResult and
Citation already do. A listing spanning databases is unreadable
without it, and `--database NAME list` could not name the one it
opened.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
Reciprocal rank fusion compares ranks, so every database contributes its
own best matches whether or not they answer the question, and results from
databases holding nothing relevant displace better ones. Measured on one
corpus split three ways over 3,045 queries: retrieval MAP 0.6044 without a
reranker against 0.9798 for the same corpus in a single database, and
0.9918 against 0.9914 with one.
A run over several databases could report which documents were cited but
not which database grounded the answer: `_result_from_run` walked the
citation index for `document_uri` and dropped `Citation.source`. The
distribution is not recoverable from the report afterwards, so a sharded
run would have measured everything except attribution.
`cited_sources` is one entry per cited chunk, in citation order, empty
where the database is unnamed.
The runner passed the dataset's path to every arm, which opens one database
and is what makes `--db` meaningful. A run over `lancedb.databases` has to
pass none instead, so the client resolves the set, and `DatasetSpec.covers_a_set`
is the one place that decides which of the two a run is.
A capability covering several databases received `source` on every document
and search result and never used it: asked how many documents were in each
database, the model read the titles and answered that there was one corpus
of 67,581. The instruction files enumerate what a result carries, and both
enumerations had gone stale.
The note follows what the capability opens rather than what the
configuration names, through `covers_several_databases`: an explicit
`db_path` or a lent client covering one database is instructed as before,
as is every `uri` or path deployment and every eval dataset. The analysis
note separates the three interfaces, since they differ: an
`analysis_search` result carries a `Database:` line, in-code `search` and
`list_documents` return `source`, and the mounted files carry neither.
The modal listed every document and mounted a checkbox per document, so a
corpus of tens of thousands never finished rendering: 67k sequential
mounts across two databases, and the same for one database that size.
Titles repeat at that scale too, and the ids were derived from the title,
so they collided.
It shows a page of 200 now, mounted in one call and identified by
position, and the search box asks the database for the rest on enter,
matching titles and URIs. Typing still narrows the page on screen, for
feedback while typing. `search_filter` escapes the term, which is
whatever was typed.
A federated listing takes its window across the databases rather than
filling it from the first one: concatenating hid every database after
whichever was listed first, which for a set of a thousand papers and
sixty thousand articles meant a page of papers alone. Sorting would not
have helped, since document ids and article titles sort into separate
runs, so the page is picked by interleaving and the modal sorts it for
display.
Chat answers with the same capabilities `ask` does, so it federates as
naturally as `ask` and `analyze` — but it went through the one-database
guard and refused a configured set outright, which left no way to chat
across several databases.
The guard was the visible half. `run_chat` also defaulted `db_path` to the
single default path whenever it was None, so lifting the refusal alone
would still have opened one database. It now leaves the path unresolved
when `lancedb.databases` names the set, and the client resolves it.
Listing and counting documents fan out over the set, which is what the
document filter reads, and visual grounding resolves the database holding
the cited chunk through the citation's source: chunks, pages and bounding
boxes all come from that one database. A limit on a listing means that
many documents in total, not that many per database.
The info modal reports every database it covers, each under its
configured name and without its location, since names are the only
identity that leaves the configuration. `database_lines` is what one
database reports about itself, shared by both paths, and it reports a
failure as a line so one unreachable database does not cost the report on
the others.
`inspect` stays a one-database command. It browses one database's
documents and chunks, so a set has nothing to show it.
The CLI reports this error, and importing it from the settings repository
pulled lancedb onto the CLI's import path. Deferring the import into cli()
bought nothing, since cli() runs on every invocation: it cost about 1.9
seconds on a cold start, `--help` included.
It now sits beside the other store exceptions, in a module that imports
nothing, and every importer points there.
A database whose stored embedder disagrees with the configuration raised
ConfigMismatchError through Typer, so the operator got a traceback wrapped
around the one message that says what to run, while every sibling failure
exits with its message. It joins the errors the CLI reports.
Imported inside cli() rather than at module scope: the settings module
pulls in lancedb, and importing the CLI must not pay for it.
lancedb.databases, the sources argument and the --database selector had no
documentation. Adds a Several Databases section to the storage
configuration page covering the name-to-location map, its mutual
exclusion with uri, the shared embedding configuration the set requires,
and which commands cover the set against which work on one. Adds a
Searching Several Databases section to the Python API page, --database to
the CLI's global options, the key to the sample configuration, and one
README feature line.
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.
The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.
The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.
On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
`ask(sources=[…])` scopes a question to some of the configured databases, carried
on the capability state so its search tool searches those. `Citation.source` names
the database a cited chunk came from, resolved from the search results the model
saw, which already carry it.
Context expansion routes each result through the database it came from: a
federating client has no repositories of its own.
The cite fallback, which looks up an id absent from this run's results, searches
only the selected databases. A chunk id says nothing about which database holds
it, so placing one means asking, and asking outside the selection would let a
question scoped to some databases cite another.
The loosely-specced client mocks in the capability tests now say they stand in for
a single-database client. A bare AsyncMock answers any attribute with a truthy
Mock, so `_federated` sent the fallback down the multi-database branch, and
`_source` reached a validated field.
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.
Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.
A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.
Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.
The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.
A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.
With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
`search` fetched, reranked and truncated in one pass, with the reranker's
over-fetch and the reranking itself interleaved in the same branch. Searching
several databases needs to fuse their candidates before anything is ranked, so
the phases have to be separable.
`_fetch` returns one database's candidates, over-fetching only when a reranker
will re-order them. `_rank` orders and cuts them, leaving an image query's vector
ranking alone since there is no text for a reranker to score against. The
over-fetch multiplier is named rather than a literal 10 at the point of use.
Both check the query type before reading `client.reranker`, which is a
cached_property that builds the reranker on first access and loads model weights
for a local one. An image query never used it and must not start.
No behaviour change: the same suite passes, and the search outputs digest
identically to before.
Port 11433 serves nothing; Qwen3-Reranker-4B is on 11455. Qwen3.6 spends
its budget reasoning before it answers, and its 131072 window leaves ample
input space at 32768.
