A configured or default database that does not exist raises
SourceUnavailableError with the way to create it and without its location.
An unknown --db-name lists the databases there are, configured or default.
The CHANGELOG records Store.db_path, SingleDatabaseSession(ref, config),
the empty lancedb.uri migration and the error-type change.
HAIKU_RAG_DB and DB_PATH are gone: a capability covers what the
configuration places or the db_path it is given, and the app backend and
the AG-UI example load their configuration as the CLI does. The compose
files point HAIKU_RAG_CONFIG_PATH at the mounted haiku.rag.yaml, which
places the database at /data where DB_VOLUME is mounted; the backend
refuses a configured set since it serves one database. The chat scopes a
selection by source only over a set and names databases on filter rows
only across several. Docstrings, docs and test fixtures stop describing an
unnamed database; every database a search, listing or citation reports
carries a name.
DatabaseRef is a name and a location. The configuration places databases
through lancedb.databases alone; with none configured the default is the
entry haiku.rag under storage.data_dir, selectable like any other.
lancedb.uri is removed, and a config carrying it fails to load with the
replacement spelled out. A path passed from Python is valid where the
configuration places nothing and raises AmbiguousDatabaseError beside
lancedb.databases; haiku-rag --db and haiku-ingester --db construct the
scope directly, so a human's override keeps working. Every database
answers to a name, and a database given as a path keeps its own errors.
A picture deduplicated within a burst must still reach the model once in
its own question and, when cited, again via the capsule re-fetch after
compaction; uncited it is dropped. Two wire tests pin both paths. The
harness builds the rag capability with defer_loading=False: a deferred
evidence capability the model has not loaded presents an empty record,
so compaction computes boundary 0 and rewrites nothing.
qa.md describes max_searches in units.
The IVF probe count was the one vector search parameter with no
setting, and it is what bounds recall on a large indexed corpus:
partitions grow with the corpus, so a fixed probe count covers less of
it, and vector_refine_factor can only re-score what the probes
returned. Defaults to 20, matching lance, so search is unchanged.
Also corrects the create-index docs, which predate the measurement:
optimize() covers new chunks, so a rebuild is about retraining
centroids rather than reaching unindexed rows.
Retrieval scores are each database's own rank arithmetic; the databases
in a selection share an embedder, so similarity in that one space is the
signal comparable across databases by construction. Measured product to
product against score ordering: +8.3 to +16.6pp recall@5 across five
cells on two corpora, flat in collection count and corpus shape where
score ordering dips with both, closing roughly 60% of the gap to a
reranker; order-sensitivity residual 0.00pp in every cell. Exact ties
collapse from 51-81% of candidates to under 1%. Full-text-only searches
keep retrieval-score order, having no query vector. The vector column
already travels with every search result, so the similarity costs no
additional transfer; per-chunk embeddings are materialized only for the
federated path that reads them.
Rank interleaving guarantees every database slots regardless of content;
on domain-split collections it allocates no better than chance and costs
4.7pp recall@5 at four collections against score ordering (7.1pp at
eight). Hybrid scores are each database's own vector/FTS rank agreement,
which carries across databases; equal scores resolve by within-database
rank, and only a tie on both falls to configured order, leaving
permutation sensitivity at 0.02-0.26pp. Fused results carry the
candidate's own retrieval score, so the context-expansion re-sort
preserves fused order.
Benchmarks state that every published number is exact brute-force kNN;
no benchmark database carries a vector index. The Vector Indexing
section gains the measured with/without IVF_PQ comparison (hotpotqa,
orb_multimodal_nemotron, frames: free to ~121k chunks, 0.0044 MAP at
426k, ~30 s / 4 GB build) and corrects the re-indexing story: optimize()
folds new chunks into the index as delta parts via auto_vacuum, so a
rebuild is about retraining centroids, not covering new rows.
An FTS index built over an empty table indexes nothing and lance never
catches it up on add. ensure_indexes skips FTS while the table is
empty; create, replace_for_document and embed-only rebuild ensure
indexes after writing, so the index always covers at least its first
rows. The first write into a fresh table writes one extra chunks table
version for the index build; a failed build fails the write.
lance serves unsorted results with matching rows dropped when a
declared FTS index has indexed nothing, or when a populated table has
no FTS index at all. doctor fails on both: vacuum remediation for an
existing index, rebuild --embed-only for a missing one, since optimize
never creates an index. The chunk repository warns once per repository
on the first FTS or hybrid search against either state; a failing
coverage check is logged and never fails the search. Removes
_ensure_fts_index, which had no callers.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
The chat filter modal keys a selection by (database, id): copies of a
database share document ids, and checking one copy left the other
reading as selected. Applying the filter narrows the question's
`sources` to the databases the selection names. A twin id inside
another selected database still matches there: a serialized id filter
cannot carry a source.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
A capsule replaces earlier questions' evidence with what they cited, so the
searches that carried a `Collection:` line are gone by the time the model reads
it. Cited content survived; which collection it came from did not, and a
follow-up question attributed it to whichever the model guessed.
Named the same way a search result names it: the capsule decides, and only when
it spans more than one.
Four passages said every missing database raises `FileNotFoundError` and that
errors show locations. A database named in `lancedb.databases` raises
`SourceUnavailableError` instead, naming the database and not its location,
which is the point of naming them. A path you gave keeps `FileNotFoundError` and
still shows the path.
`--db-name` selects one database, not a subset: it is not repeatable.
The reranker recommendation was one-sided. It reports both measured effects now:
stronger aggregate retrieval across shards, and weaker attribution between
near-identical documents, where fusion keeps twins apart because each database
contributes its own top-ranked result.
Image queries are vector-only and skip the reranker.
The changelog described components of the new feature as fixes to the last
release. They are one Added entry, and the changes a user upgrading does see —
`settings` printing YAML, the document filter paging and searching, `list`
printing only the fields a document has — are listed.
A name nothing covers raised `KeyError` in four places and
`AmbiguousDatabaseError` in a fifth, so a caller had to catch both and neither
name said what happened. `UnknownDatabaseError` is all of them, exported from
`haiku.rag.store` beside the other errors.
It subclasses `KeyError`, since selecting by name is a lookup, and prints its
message plainly rather than quoted as a missing key. Both CLIs turn it into the
same clean exit they already gave the others.
`close()` refused a client covering a set, which left one with no method to
call: `async with` was the only lifecycle it had. `aclose()` runs that teardown
for a caller that owns the client some other way, whatever it covers, and
nothing to release is not an error, so it is safe before entering and after
closing.
`close()` stays what it is, one connection and nothing else, and says so:
draining the background vacuum and releasing the embedder and reranker are
awaitable.
`create_mcp_server` promised one database and accepted a scope covering a set,
where the write tools exist and fail on use. It refuses that now.
Resolving is the public factory's job, as it is `HaikuRAG`'s: `_covering` takes
a scope someone already resolved, so the configured name survives without a
`DatabaseScope` reaching the public signature.
The test that a scope decides the database asserted `all(...)` over a search
that could return nothing, which held whatever the server read. It reads the
listing instead, so alpha's documents being present and beta's absent both have
to be true.
`sources` was ignored beside a database path, so `sources=["nope"]` opened the
path and read as though the selection had been honoured. A path and a name
already conflict inside `DatabaseScope.resolve`; this is the same rule where a
caller can reach it.
`reader_for` documented None as covering any database it could not place, but a
name outside the set raises `KeyError` like `clients_for` does: provenance
naming a database this client does not cover is wrong rather than absent. None
means one thing, a federated client given no name.
`sources=[]` means two things. On a search it selects nothing to search; on the
constructor it raises, since a client over no database can do nothing. Both are
written down now.
A capability reads through a lent client, so what a citation records is that
client's database and not the scope the capability was built with. Chat lends
one, and had no test saying so.
The documented rule was existence in more than one selected database. The rule
is narrower and does not need to be wider: a copy the search never returned
grounded nothing, so the retrieved result is the citation and its database is
recorded rather than guessed.
The page now states the whole rule, including the fallback's own check for an
id no search returned. Two retrieved copies are refused, and so is an id
already cited from another database in the conversation. The case between them
had no test.
The ingester manufactured `data_dir / haiku.rag.lancedb` whenever `--db` was
absent and passed it to HaikuRAG. A path is now an explicit override that
clears the configured URI, so a `lancedb.uri` deployment wrote to local disk
while the control plane reported on the remote.
`IngesterApp` resolves the databases it works on once, in its constructor, and
both the client it opens and the control plane read that scope. `--db` names
one directly and nothing stands in for it, so a configured set is refused
rather than guessed, and `AmbiguousDatabaseError` joins the errors `cli()`
turns into a clean exit.
`format_for_agent` named the database whenever one was named, so a search over
a single named database carried a line with nothing to distinguish. It now takes
`include_collection` from the caller, which decides from the search selection
rather than from the hits: a search that could have drawn on two collections
names them even when everything came back from one.
`Collection:` at the model boundary, database in configuration and
administration. `source` on results, documents, citations and analysis
dictionaries is unchanged.
`resolve_citations` documented the behaviour it now rejects: a shared chunk id
resolving to whichever result came last.
The storage page ran two embedding checks together as if the second explained
the first. They are separate: each database against the configuration on open,
and the databases in a selection against each other, which raises in read-only
mode too and does not apply to full-text search.
`several` becomes `multiple` where it names the feature, matching the docs and
`covers_multiple`.
Restructure the storage page into search and provenance, duplicate ids,
ranking, Python operations and CLI commands, and state each in reference
voice. `Several Databases` becomes `Multiple Databases`, with the anchor
carried through every referrer, and the instruction files follow the same
name. Example databases are `papers`, `wiki` and `notes`.
`AmbiguousCitationError` is raised for a cited chunk id held by more than one
selected database, not for any shared id.
`--database` and `--db` read as one word abbreviated but take different
things, a configured name against a filesystem path, and they sit in
different positions: `haiku-rag --db /path info` fails with `No such option`.
The CLI already says "database" to the user everywhere it prints one, so the
name it selects by should say the same. Unreleased, so no deprecation.
Also: without a reranker, raise `search.limit` with the number of databases
searched, since each contributes its best matches to a list truncated back
to the limit.
`HaikuRAG.__init__` said an omitted `db_path` uses `storage.data_dir`, which
is the last of three; and that `sources` is ignored for a single `uri`, where
it raises, since only `lancedb.databases` names databases.
The name is not the only identity leaving the configuration: results,
citations, model input and errors opening a named database carry it, while
`info`, `init` and `tag` print the location. `sources=[]` returns no search
results, but `ask` and `analyze` still answer, without evidence.
Trim the comments that still narrated a failure or an alternative to the
invariant they were there for.
Three groups, not two: `search`, `ask`, `analyze` and `chat` cover the set,
`settings`, `init-config` and `download-models` open no database, and every
other command works on one — or on a configured set of one, which is
unambiguous and keeps its name. A database named in `lancedb.databases` keeps
that name whether or not it is the only one covered; only `lancedb.uri` places
one without naming it.
Document ids repeat between copies of a database, where the sandbox refuses a
duplicate but the chat filter's `id IN (...)` matches the document in every
copy. `build_document_id_filter` claimed ids never widen a selection.
Document the facade: `covers_multiple`, `source_names`, `source`,
`reader_for`, `clients_for`, the lifetime of a borrowed client, and
`sources=None` against `sources=[]`.
Drop the vision callout from the README, which the features list already
covers.
`resolve_db_path` manufactured the default path whenever `lancedb.databases`
was empty, and `covers_several_databases` read coverage back out of the
configuration, so a capability built without a client opened
`storage.data_dir/haiku.rag.lancedb` instead of what `lancedb.uri` placed.
The entry point resolves a `DatabaseScope` instead: instructions ask it what
it covers and `_ensure_rag` opens it through `HaikuRAG._covering`, so
coverage is decided once rather than encoded in a path and re-derived.
`Sandbox._covering` takes the scope the capability already resolved, beside
the public constructor that takes a path. The factory signatures are
unchanged.
Closes#582. `lancedb.databases` entries already classify a location by
whether it carries a scheme; `lancedb.uri` was taken as a URI whatever it
said, so a local path was opened as object storage and a mistyped one
became a new empty database instead of failing. Both settings now place a
database the same way, and `--db PATH` overrides either.
A chunk id is unique within a database and says nothing across them, so a
database copied from another holds the same ids. `qualified_id` keys the
two in-memory identity sites on the database and the id together:
`merge_results` was dropping the second database's result when a query
repeated, and the arrival map that breaks fused score ties was ranking one
of the pair as the other.
Everything serialized records the id alone, so there ambiguity is refused
rather than qualified. `resolve_citations` raises `AmbiguousCitationError`
for a cited id held by two of the databases searched, where it used to
resolve to whichever result came last; `_register_citations` raises for one
already cited from another database in an earlier question. `_cite` turns
both into a `ModelRetry` asking for other evidence. The direct-id fallback
asks every database the question covers instead of taking the first that
answers, so an id no search returned is refused on the same terms.
`all_found` collects them and `first_found` reads its first, which document
reads keep doing on purpose.
Also drop a duplicated 0.77.0 heading from the changelog.
An embedder is a function of configuration, not of a database, and the
databases in a selection share one, so a client covering a set builds it
on first use and closes it on teardown. Operations that need one database
say so instead of surfacing a missing store.
Document.source names the configured database, as SearchResult and
Citation already do. A listing spanning databases is unreadable
without it, and `--database NAME list` could not name the one it
opened.
Reciprocal rank fusion compares ranks, so every database contributes its
own best matches whether or not they answer the question, and results from
databases holding nothing relevant displace better ones. Measured on one
corpus split three ways over 3,045 queries: retrieval MAP 0.6044 without a
reranker against 0.9798 for the same corpus in a single database, and
0.9918 against 0.9914 with one.
Chat answers with the same capabilities `ask` does, so it federates as
naturally as `ask` and `analyze` — but it went through the one-database
guard and refused a configured set outright, which left no way to chat
across several databases.
The guard was the visible half. `run_chat` also defaulted `db_path` to the
single default path whenever it was None, so lifting the refusal alone
would still have opened one database. It now leaves the path unresolved
when `lancedb.databases` names the set, and the client resolves it.
Listing and counting documents fan out over the set, which is what the
document filter reads, and visual grounding resolves the database holding
the cited chunk through the citation's source: chunks, pages and bounding
boxes all come from that one database. A limit on a listing means that
many documents in total, not that many per database.
The info modal reports every database it covers, each under its
configured name and without its location, since names are the only
identity that leaves the configuration. `database_lines` is what one
database reports about itself, shared by both paths, and it reports a
failure as a line so one unreachable database does not cost the report on
the others.
`inspect` stays a one-database command. It browses one database's
documents and chunks, so a set has nothing to show it.
lancedb.databases, the sources argument and the --database selector had no
documentation. Adds a Several Databases section to the storage
configuration page covering the name-to-location map, its mutual
exclusion with uri, the shared embedding configuration the set requires,
and which commands cover the set against which work on one. Adds a
Searching Several Databases section to the Python API page, --database to
the CLI's global options, the key to the sample configuration, and one
README feature line.
`Agent.from_spec` raised `TypeError` on `RAGCapability` and `AnalysisCapability`,
whose constructors take a state class, packaged instruction text and a tool-name
set, and silently omitted both from the generated spec schema. The two
zero-configuration capabilities constructed but with `id=None`, so pydantic-ai's
duplicate-id rejection no longer held and a spec could register two citation
policies, defeating the single-decision-maker invariant.
Override `from_spec` on all four, delegating to `create_capability()` so ids and
instructions come from one place. The spec surface is `db_path`, `config`,
`defer_loading`, `request_limit` and `vision`; a live `HaikuRAG` client stays out
of it, and a `config` mapping is validated through `AppConfig`.
Sixty-three comments said what the next statement already said: # Connect to
LanceDB above connect_lancedb, # Path object above isinstance(source, Path),
# Get page numbers from provenance above the prov loop, # Clear and populate
results above list_view.clear(). They cost a read and carry nothing.
The line is whether a comment restates one statement or labels a phase. Phase
labels stay: the migrations keep # Create staging table with new schema and
# Copy from staging to final table in batches, each heading ten lines of a
long procedure. So do comments carrying a fact the code cannot: the
merge_insert update-only note on document_meta, why the poller builds sources
eagerly, why create_document_from_source returns a list for directories, that
indexes need training data, the field-group markers in the config models, and
the file:// URL-encoding note in create_document_from_source.
capabilities/ is untouched. Its docstrings sit next to prompt surface, and
changing them needs an eval to back it.
The cassette-recording docs were wrong three ways. They named
tests/test_qa.py::test_qa_anthropic, which no longer exists; they targeted
whole modules, so a rewrite would re-record cassettes for services the
recorder is not running; and they used COHERE_API_KEY where the SDK reads
CO_API_KEY. docs/development.md now names exact tests with -n0, and the keyed
example is test_cohere_reranker, which owns the one cassette recording
api.cohere.com.
