Collision detection replaced the lookup's dict comprehension with
`setdefault`, which also flipped a chunk found by several searches from its
last occurrence to its first. The copies differ in everything the expansion
window decides, figures included, so that silently changed what a citation
renders.
The rules are separate now: a repeated (source, chunk_id) takes the later
result, a chunk_id under two sources is still refused.
The collection block was chosen when the capability was built, so a run narrowed
through `state.sources` to one collection was still told how to attribute across
collections it could not reach, while its results correctly carried no
`Collection:` line.
`get_instructions` composes it per run instead, from `state.sources` where the
question narrowed the conversation and from the lent client or the scope
otherwise. Order is preamble, base instructions, collection block.
`format_for_agent` named the database whenever one was named, so a search over
a single named database carried a line with nothing to distinguish. It now takes
`include_collection` from the caller, which decides from the search selection
rather than from the hits: a search that could have drawn on two collections
names them even when everything came back from one.
`Collection:` at the model boundary, database in configuration and
administration. `source` on results, documents, citations and analysis
dictionaries is unchanged.
`_documents` took its owner clients inside an ephemeral connection and used
them after that connection closed, then stored them for the file reads that
follow, so every standalone multi-database read went through a closed database.
A sandbox covering a set now retains the client it opened until `close()`. One
database has no owners and keeps its connection no longer than the read that
opened it, so a write from elsewhere is still visible to the next read.
A URI-backed database is constructed with a local `db_path` nothing connects
to, and the info modal reported that, so `--db-name` on an S3 database named a
local directory. `location` reads the configuration instead, and the modal
labels it as one.
`InfoModal`'s `db_path` argument is gone: both callers passed None, so only a
test reached the branch that used it.
`sessions_for` recorded what opened from `gather`'s results, which arrive only
when it runs to completion. Cancelling it discarded them, so a database that
opened while a sibling was still pending was never recorded and `aclose` never
closed it. `return_exceptions=True` covers a failing child, not a cancelled
parent.
`_open` now registers its own session, so what opened is reachable however the
fan-out ends.
`database_lines` reports through the connection the client already holds and
never calls `connect_lancedb`, so patching it detects nothing; `asked ==
[connection]` is what proves the reuse.
It was not merely dead. The patch was installed by name, and the next
`monkeypatch.setattr` triggered the first import of `haiku.rag.store.info`,
whose module body binds `connect_lancedb` — capturing the fake into a
namespace teardown does not restore, failing four `test_info` tests under
`-n0`.
Every load looked up the loading indicator, which only the first page has,
so pressing enter in the search box raised `NoMatches`. The indicator is a
child of the list `remove_children()` already clears.
A document's id and search text travel on a `DocumentCheckbox` instead of
being assigned onto a `Checkbox` behind type suppressions. The positional
widget ids are gone; nothing queried them.
`_rich_print_search_result` asked how many databases were configured, so
`--db-name alpha` labelled every line with the database the caller had just
named. Citations already ask the client what it covers, which is the same
question the scope answers.
`settings` printed each top-level block as one Python dict repr, so nesting
was invisible, long paths wrapped mid-token and `PosixPath(...)` leaked. It
renders as YAML now, which is what `haiku.rag.yaml` is written in.
`list` does not load content, which is where the docling blobs live, so the
`content:` header announced a field the command declined to fetch. A document
without a uri or metadata printed `uri: None` and `meta: {}`, where `title`
was already omitted when absent.
`--database` and `--db` read as one word abbreviated but take different
things, a configured name against a filesystem path, and they sit in
different positions: `haiku-rag --db /path info` fails with `No such option`.
The CLI already says "database" to the user everywhere it prints one, so the
name it selects by should say the same. Unreleased, so no deprecation.
Also: without a reranker, raise `search.limit` with the number of databases
searched, since each contributes its best matches to a list truncated back
to the limit.
Two files of 1,405 and 814 lines become seven: scope resolution, lifecycle,
search, documents, expansion, citations and capabilities. `_config`, `_seed`
and the rest move to `helpers.py`, importable by the sandbox tests that share
them, and the package points VCR back at `tests/cassettes/multi_db/`.
`covers_several_databases` had one line of body and two call sites.
`_db_path_given` guarded a default path manufactured in `__init__` and
overwritten in `__aenter__`; `_requested_db_path` is what the caller asked
for, and the effective path falls out of the resolved scope.
`clients_for` hands back a client over a database the covering one owns, so
`async with` on it opened a second session and assigned it, and teardown
declined to close what this client did not open. Entry returns the client as
it stands.
They opened a client on `tmp_path / "unused"` with the bucket in the
configuration. An explicit path now selects the database, so the client
tests ran against the local disk and the two app tests failed. Nothing names
a path any more, and both helpers assert the connection is remote before
yielding, so a later precedence change cannot quietly localize them again.
`resolve_db_path` manufactured the default path whenever `lancedb.databases`
was empty, and `covers_several_databases` read coverage back out of the
configuration, so a capability built without a client opened
`storage.data_dir/haiku.rag.lancedb` instead of what `lancedb.uri` placed.
The entry point resolves a `DatabaseScope` instead: instructions ask it what
it covers and `_ensure_rag` opens it through `HaikuRAG._covering`, so
coverage is decided once rather than encoded in a path and re-derived.
`Sandbox._covering` takes the scope the capability already resolved, beside
the public constructor that takes a path. The factory signatures are
unchanged.
Closes#582. `lancedb.databases` entries already classify a location by
whether it carries a scheme; `lancedb.uri` was taken as a URI whatever it
said, so a local path was opened as object storage and a mistyped one
became a new empty database instead of failing. Both settings now place a
database the same way, and `--db PATH` overrides either.
`list_documents()` reached into the ownership map for a name the document
already carries. The map is still built, so a document id two databases
claim is still refused before anything mounts. A listing over one named
database now reports that name, as in-code `search()` already does.
`Store` already parses the settings blob on open, so keep it: the
inspector's info modal was opening the settings table, querying it and
parsing the JSON a second time, and `doctor` was asking
`SettingsRepository` for it on a store it already held. Creating a
database refreshes it, so a new one reports the version init wrote.
`gather_database_info` keeps its own parse: it goes around `Store` so a
pre-migration database still reports what it can.
A chunk id is unique within a database and says nothing across them, so a
database copied from another holds the same ids. `qualified_id` keys the
two in-memory identity sites on the database and the id together:
`merge_results` was dropping the second database's result when a query
repeated, and the arrival map that breaks fused score ties was ranking one
of the pair as the other.
Everything serialized records the id alone, so there ambiguity is refused
rather than qualified. `resolve_citations` raises `AmbiguousCitationError`
for a cited id held by two of the databases searched, where it used to
resolve to whichever result came last; `_register_citations` raises for one
already cited from another database in an earlier question. `_cite` turns
both into a `ModelRetry` asking for other evidence. The direct-id fallback
asks every database the question covers instead of taking the first that
answers, so an id no search returned is refused on the same terms.
`all_found` collects them and `first_found` reads its first, which document
reads keep doing on purpose.
Also drop a duplicated 0.77.0 heading from the changelog.
Expansion branched on whether the client covered a set, so the
single-database half reached for repositories through a facade that may
have none. `expand_sources` groups results by database and hands each
group the session that owns it; `expand_context` and `visualize_chunk`
take that session, so `visualize_chunk` stops narrowing to one database
and discarding the result.
Inline `_fetch`, a pass-through to the chunk repository.
The CLI decides only what it knows — that --db and --database are the same
thing said twice, and whether a command reads more than one — and hands the
resolved scope down. Nothing rewrites the configuration, so a named database
keeps the name results and citations carry, and a remote one opens the URI
it was configured with rather than the local path standing in for it.
HaikuRAGApp, ChatApp and InspectorApp take that scope and nothing else.
Selection reaches the client through a private constructor, so the public
signature still takes a path or names.
covers_multiple, source_names, source and reader_for replace the private
state seven modules were reading to work out how many databases they had.
The configured selection is kept intact, so entering a client twice derives
the same database rather than the last derivation.
Every implementation in documents.py and rebuild.py takes the session it
writes to, so a set cannot reach one: the facade narrows once and passes the
database on, rather than checking and carrying a union. Tests calling an
implementation directly go through `writing()`.
One session field, whichever kind it is, with the covered databases derived
from it rather than kept beside it. `_single_session` returns the database a
write works on in place of ten guards that only asserted one existed, and the
single-database operations move to the session that owns them.
FederatedSession opens the databases a query covers and owns their
teardown; the client keeps the wrappers it hands out. A wrapper releases
what it built and never closes the database it borrowed.
SingleDatabaseSession owns the store, the repositories and the vacuum
machinery, so nothing above has to ask whether it has a store. The client
keeps every name callers already use.
DatabaseScope.resolve reads configuration and at most one selector; a
DatabaseRef carries the configured name and a location already resolved to
a URI or a path, so a path a caller names is never reinterpreted. Nothing
consumes it yet.
An embedder is a function of configuration, not of a database, and the
databases in a selection share one, so a client covering a set builds it
on first use and closes it on teardown. Operations that need one database
say so instead of surfacing a missing store.
An image query has no text to match, so it is vector-only whatever the
caller asked for, and full-text search embeds nothing, so it needs no
agreement on embedders.
`first_found` replaces the client's private version and the capability's
sequential one. The info modal reports through the connection the client
already holds.
A title repeats within a corpus and across databases, so a substring
match on the displayed name widened the filter to documents the user did
not pick. The label names the database.
Each database owns an embedder, so embedding per database cost a round
trip each. One database still embeds inside the repository, which returns
early for a filter that matches nothing.
A blank name is falsy, so source routing reads it as absent, and a blank
location resolves to the working directory. `is_read_only` reports the
mode the client was opened with, which a client covering a set can answer
without a store.
Document.source names the configured database, as SearchResult and
Citation already do. A listing spanning databases is unreadable
without it, and `--database NAME list` could not name the one it
opened.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
A capability covering several databases received `source` on every document
and search result and never used it: asked how many documents were in each
database, the model read the titles and answered that there was one corpus
of 67,581. The instruction files enumerate what a result carries, and both
enumerations had gone stale.
The note follows what the capability opens rather than what the
configuration names, through `covers_several_databases`: an explicit
`db_path` or a lent client covering one database is instructed as before,
as is every `uri` or path deployment and every eval dataset. The analysis
note separates the three interfaces, since they differ: an
`analysis_search` result carries a `Database:` line, in-code `search` and
`list_documents` return `source`, and the mounted files carry neither.
The modal listed every document and mounted a checkbox per document, so a
corpus of tens of thousands never finished rendering: 67k sequential
mounts across two databases, and the same for one database that size.
Titles repeat at that scale too, and the ids were derived from the title,
so they collided.
It shows a page of 200 now, mounted in one call and identified by
position, and the search box asks the database for the rest on enter,
matching titles and URIs. Typing still narrows the page on screen, for
feedback while typing. `search_filter` escapes the term, which is
whatever was typed.
A federated listing takes its window across the databases rather than
filling it from the first one: concatenating hid every database after
whichever was listed first, which for a set of a thousand papers and
sixty thousand articles meant a page of papers alone. Sorting would not
have helped, since document ids and article titles sort into separate
runs, so the page is picked by interleaving and the modal sorts it for
display.
Chat answers with the same capabilities `ask` does, so it federates as
naturally as `ask` and `analyze` — but it went through the one-database
guard and refused a configured set outright, which left no way to chat
across several databases.
The guard was the visible half. `run_chat` also defaulted `db_path` to the
single default path whenever it was None, so lifting the refusal alone
would still have opened one database. It now leaves the path unresolved
when `lancedb.databases` names the set, and the client resolves it.
Listing and counting documents fan out over the set, which is what the
document filter reads, and visual grounding resolves the database holding
the cited chunk through the citation's source: chunks, pages and bounding
boxes all come from that one database. A limit on a listing means that
many documents in total, not that many per database.
The info modal reports every database it covers, each under its
configured name and without its location, since names are the only
identity that leaves the configuration. `database_lines` is what one
database reports about itself, shared by both paths, and it reports a
failure as a line so one unreachable database does not cost the report on
the others.
`inspect` stays a one-database command. It browses one database's
documents and chunks, so a set has nothing to show it.
The CLI reports this error, and importing it from the settings repository
pulled lancedb onto the CLI's import path. Deferring the import into cli()
bought nothing, since cli() runs on every invocation: it cost about 1.9
seconds on a cold start, `--help` included.
It now sits beside the other store exceptions, in a module that imports
nothing, and every importer points there.
A database whose stored embedder disagrees with the configuration raised
ConfigMismatchError through Typer, so the operator got a traceback wrapped
around the one message that says what to run, while every sibling failure
exits with its message. It joins the errors the CLI reports.
Imported inside cli() rather than at module scope: the settings module
pulls in lancedb, and importing the CLI must not pay for it.
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.
The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.
The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.
On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
`ask(sources=[…])` scopes a question to some of the configured databases, carried
on the capability state so its search tool searches those. `Citation.source` names
the database a cited chunk came from, resolved from the search results the model
saw, which already carry it.
Context expansion routes each result through the database it came from: a
federating client has no repositories of its own.
The cite fallback, which looks up an id absent from this run's results, searches
only the selected databases. A chunk id says nothing about which database holds
it, so placing one means asking, and asking outside the selection would let a
question scoped to some databases cite another.
The loosely-specced client mocks in the capability tests now say they stand in for
a single-database client. A bare AsyncMock answers any attribute with a truthy
Mock, so `_federated` sent the fallback down the multi-database branch, and
`_source` reached a validated field.
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.
Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.
A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.
Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.
The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.
A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.
With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
