`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
`tag_restore` reached into `HaikuRAGApp._display_path` and `_path`, and seven
places spelled out `self._is_local and not self._path.exists()`. `display_path`
and `database_missing` say both, and `database_missing` is False for a database
behind a URI, which has no path to check.
`init` keeps its own check: it asks the opposite question.
A database name, a location and an exception message all reach the info modal
as configuration-derived text, and Rich reads `[...]` in any of them as markup:
a name like `beta [prod]` disappeared, and a message carrying `[Errno 2]` or a
stray closing tag could break the line it sat in.
Two collections can hold documents with the same title, and the collapsed
citation carried only that title, so the two read as one source repeated. The
name is appended when the client covers a set, the decision already made for
search results and retrieved images.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
`_resolve_scope` returned a scope without keeping it, so a client re-entered
after its configuration was edited covered whatever the configuration then said.
Resolving once is what the rest of the design rests on: the scope is what names
results, citations and errors.
A configuration is still free to change before first entry.
Search results and capsule entries name the collection they came from; the
images attached beside them carried only the chunk id and reference. Two
collections can return the same picture of the same document, so the two
labels were identical and the model could place neither.
The decision is the one already made for the text: `covers_multiple` at the
generic search tool, and the flag `search_corpus` computed for the capability
tools, which it now returns.
`asyncio.gather` propagates the first failure while its siblings run on, and the
caller unwinding from that closes the set through `async with` — so a sibling
still reading reads through a closed session. Seven fan-outs were affected:
lookup, search, image enrichment, multimodal picture loading, context expansion,
document listing and counting, and the sandbox's document load.
`gather_all` cancels and drains the rest, then re-raises the original exception.
A `TaskGroup` would drain them too but raise an `ExceptionGroup`, which every
caller and both CLIs' exception handlers would have to unwrap. The two
`return_exceptions=True` gathers in session opening and teardown already drain
their children and are left alone.
Three places identified a picture by document and reference alone, and one
identified a citation's images by chunk id alone. Both repeat between copies of
a database, so a search returning a figure from two collections sent one, a
capsule retained one, and a citation rendered the other collection's figures.
Keyed on the source as well: `(source, document_id, self_ref)` for search
pictures, the capability and source for retained ones, and `qualified_id` for
the chat's citation images.
A capsule replaces earlier questions' evidence with what they cited, so the
searches that carried a `Collection:` line are gone by the time the model reads
it. Cited content survived; which collection it came from did not, and a
follow-up question attributed it to whichever the model guessed.
Named the same way a search result names it: the capsule decides, and only when
it spans more than one.
`ConfigMismatchError` and `SourceUnavailableError` escaped the entry point as
tracebacks rather than a message and exit code 1. The entry-point test is
parametrized over the translated types, so the two CLIs' lists cannot drift
apart silently again.
Typing narrowed the mounted checkboxes without touching the search the listing
was built from, so a term matching more than one page hid rows from whichever
page the user happened to be on, left the rest of the matches a page away, and
paged on the previous search while the term still sat in the box. The term now
applies on enter, and the footer says so until it does.
The set shares the reranker its facades borrow, not an embedder: each covered
database builds and closes its own. Sharing one needs the lender pattern inside
`Store`, since a store's embedder also serves writes and is closed with the
session, so it is left as follow-up rather than done here.
`reported_location` returned `client.location` and nothing else; its test
asserted that a mock returns what it was given. `list_documents` names its
results through `name_all`.
The lock was applied to an owner as well, so every owner-backed file read queued
behind the capability's tool calls to guard state it does not touch. An owner is
a session of its own and is yielded straight through, which is what the
docstring already claimed.
Chat asked the covering client for a picture with the citation's source, which
raises when there is none, losing the answer to one figure. It resolves the
reader first and omits the bytes when the database cannot be placed, as the rich
formatter already did.
Reads tagged the document they returned and writes did not, so creating in
`alpha` came back with `source=None` while reading the same document came back
with `"alpha"`. Every outward write result goes through the owning session.
`settings` and `download-models` read the configuration and open nothing, but
resolved a scope to be constructed, so `--db-name nope` failed them over a
selection they never use. `HaikuRAGApp` takes no scope for those, and asking it
for one is an error rather than a silent default.
`test_an_image_query_builds_no_reranker` never submitted an image query: it
opened clients and found nothing built, which is lazy construction. It sends one
now, so consulting the reranker before the query type fails it.
`TestComparingEmbedders` called `_require_one_embedder` and asserted nothing.
It rejects a disagreement and accepts an absent record, and searches through.
Three `ModelRetry` tests took any reason where the reason is the subject.
`haiku-ingester` reached the client's refusal, which names `sources=[name]`, a
Python argument no CLI user can pass. It refuses a configured set itself, with
`--db PATH` and the one-ingester-per-database layout.
`haiku-rag`'s own refusal now says where each option goes: `--db-name` is global
and precedes the command, `--db` follows it.
Creating re-read the settings blob and left `stored_embedding` at None, so a
client that created a database compared as though it recorded no embedder.
`_remember_settings` takes both, and the comment no longer says one follows the
other.
A name nothing covers raised `KeyError` in four places and
`AmbiguousDatabaseError` in a fifth, so a caller had to catch both and neither
name said what happened. `UnknownDatabaseError` is all of them, exported from
`haiku.rag.store` beside the other errors.
It subclasses `KeyError`, since selecting by name is a lookup, and prints its
message plainly rather than quoted as a missing key. Both CLIs turn it into the
same clean exit they already gave the others.
`close()` refused a client covering a set, which left one with no method to
call: `async with` was the only lifecycle it had. `aclose()` runs that teardown
for a caller that owns the client some other way, whatever it covers, and
nothing to release is not an error, so it is safe before entering and after
closing.
`close()` stays what it is, one connection and nothing else, and says so:
draining the background vacuum and releasing the embedder and reranker are
awaitable.
A selection outside the page stayed applied while its checkbox was gone, so
there was no way to remove it. Appending those documents to the page instead
loses the bound the page exists to keep, since selections accumulate across
searches.
Both listings page at `DOCUMENT_PAGE`, and `Selected` switches between them, so
the mounted widgets stay bounded whichever is showing and every selection is a
page away rather than unreachable.
The count reads the checkboxes on screen, so narrowing as the user types reports
what is visible. A listing with nothing in it says so, instead of leaving an
empty box that reads as still loading.
`ask(sources=["typo"])` reached the model, which discovered the name only if it
searched: requests spent on a selection that could never answer, and a run that
never searched answered anyway.
Checked by name, not by opening: a client covering a set opens a database when a
query reaches it, and validating by opening would open every one of them before
any search, letting a database nobody asked about fail the run.
Every client covering a database built its own, so a set of five loaded the same
local model weights five times over, beside the federator's. A client covering a
database for another borrows that one's, and only its owner closes it.
`format_citations_rich` asked a client covering a set for a picture whose
citation named no database, which raises, losing the whole answer to one figure.
Citations recorded before databases could be named carry no source, so the
figure marker the caller already renders stands in.
`_fuse` ran in tests without anything asserting its output, so reversing the
sort or dropping the truncation changed nothing. Databases interleave by rank,
the score is the reciprocal of it, ties keep configured order, and the limit
cuts the fused list.
Every native score in one database beats every one in the other, so ranking by
score rather than position fails all four.
`create_mcp_server` promised one database and accepted a scope covering a set,
where the write tools exist and fail on use. It refuses that now.
Resolving is the public factory's job, as it is `HaikuRAG`'s: `_covering` takes
a scope someone already resolved, so the configured name survives without a
`DatabaseScope` reaching the public signature.
The test that a scope decides the database asserted `all(...)` over a search
that could return nothing, which held whatever the server read. It reads the
listing instead, so alpha's documents being present and beta's absent both have
to be true.
The test searched through the capability, which embeds the query, so it passed
against a local Ollama and failed CI where there is none. What it exists to show
is that a lent client is what the capability reads through: `_ensure_rag`
returns it, and a full-text search through it names its results.
`sources` was ignored beside a database path, so `sources=["nope"]` opened the
path and read as though the selection had been honoured. A path and a name
already conflict inside `DatabaseScope.resolve`; this is the same rule where a
caller can reach it.
`run_mcp` derived a path and a configuration for the server, and deriving drops
the configured name: MCP results and citations carried `source=None` where the
same database named through any other path carried "alpha".
`create_mcp_server` takes the resolved scope and opens through it, so the name
survives. Passing a path and configuration still works and resolves to the same
place.
`reader_for` returned itself before looking at the name, so an "alpha" client
answered `reader_for("beta")` with alpha's reader. A citation naming another
database would have been read from the wrong one.
Both paths decide through `clients_covering` now, so one database refuses a
wrong name the way a set already did.
The chat citation test said mounting lends the client without showing it; a
chat test shows it and the docstring points there.
`reader_for` documented None as covering any database it could not place, but a
name outside the set raises `KeyError` like `clients_for` does: provenance
naming a database this client does not cover is wrong rather than absent. None
means one thing, a federated client given no name.
`sources=[]` means two things. On a search it selects nothing to search; on the
constructor it raises, since a client over no database can do nothing. Both are
written down now.
A capability reads through a lent client, so what a citation records is that
client's database and not the scope the capability was built with. Chat lends
one, and had no test saying so.
`run_mcp` passed `_path`, the local stand-in a URI-backed ref resolves to for
display. A path overrides `lancedb.uri`, so `--db-name` on an S3 database
served the local default. It passes the ref's own path now, None where a URI
placed the database, and `create_mcp_server` accepts that.
The client opened around the server is gone. It never served a request, and it
opened the scope rather than the derived path, so startup validated the remote
database while the server read the local one.
`sources=["alpha"]` on a client covering a set went through fusion, which
scores position: a result the database ranked at 0.6549 was reported as
1/(60+rank). Embedding also moved ahead of the repository, so a filter matching
no document embedded the query anyway.
A selection of one now runs the single-database search. Fusion reconciles
rankings from separate indexes, and one ranking has nothing to reconcile.
A reranker that returns chunks it built rather than the ones it was given loses
which database each came from, since ownership is by identity. That is named
now instead of surfacing as a KeyError, and stated on `RerankerBase._rerank`.
The documented rule was existence in more than one selected database. The rule
is narrower and does not need to be wider: a copy the search never returned
grounded nothing, so the retrieved result is the citation and its database is
recorded rather than guessed.
The page now states the whole rule, including the fallback's own check for an
id no search returned. Two retrieved copies are refused, and so is an id
already cited from another database in the conversation. The case between them
had no test.
The ingester manufactured `data_dir / haiku.rag.lancedb` whenever `--db` was
absent and passed it to HaikuRAG. A path is now an explicit override that
clears the configured URI, so a `lancedb.uri` deployment wrote to local disk
while the control plane reported on the remote.
`IngesterApp` resolves the databases it works on once, in its constructor, and
both the client it opens and the control plane read that scope. `--db` names
one directly and nothing stands in for it, so a configured set is refused
rather than guessed, and `AmbiguousDatabaseError` joins the errors `cli()`
turns into a clean exit.
Collision detection replaced the lookup's dict comprehension with
`setdefault`, which also flipped a chunk found by several searches from its
last occurrence to its first. The copies differ in everything the expansion
window decides, figures included, so that silently changed what a citation
renders.
The rules are separate now: a repeated (source, chunk_id) takes the later
result, a chunk_id under two sources is still refused.
The collection block was chosen when the capability was built, so a run narrowed
through `state.sources` to one collection was still told how to attribute across
collections it could not reach, while its results correctly carried no
`Collection:` line.
`get_instructions` composes it per run instead, from `state.sources` where the
question narrowed the conversation and from the lent client or the scope
otherwise. Order is preamble, base instructions, collection block.
`format_for_agent` named the database whenever one was named, so a search over
a single named database carried a line with nothing to distinguish. It now takes
`include_collection` from the caller, which decides from the search selection
rather than from the hits: a search that could have drawn on two collections
names them even when everything came back from one.
`Collection:` at the model boundary, database in configuration and
administration. `source` on results, documents, citations and analysis
dictionaries is unchanged.
`_documents` took its owner clients inside an ephemeral connection and used
them after that connection closed, then stored them for the file reads that
follow, so every standalone multi-database read went through a closed database.
A sandbox covering a set now retains the client it opened until `close()`. One
database has no owners and keeps its connection no longer than the read that
opened it, so a write from elsewhere is still visible to the next read.
A URI-backed database is constructed with a local `db_path` nothing connects
to, and the info modal reported that, so `--db-name` on an S3 database named a
local directory. `location` reads the configuration instead, and the modal
labels it as one.
`InfoModal`'s `db_path` argument is gone: both callers passed None, so only a
test reached the branch that used it.
`sessions_for` recorded what opened from `gather`'s results, which arrive only
when it runs to completion. Cancelling it discarded them, so a database that
opened while a sibling was still pending was never recorded and `aclose` never
closed it. `return_exceptions=True` covers a failing child, not a cancelled
parent.
`_open` now registers its own session, so what opened is reachable however the
fan-out ends.
`database_lines` reports through the connection the client already holds and
never calls `connect_lancedb`, so patching it detects nothing; `asked ==
[connection]` is what proves the reuse.
It was not merely dead. The patch was installed by name, and the next
`monkeypatch.setattr` triggered the first import of `haiku.rag.store.info`,
whose module body binds `connect_lancedb` — capturing the fake into a
namespace teardown does not restore, failing four `test_info` tests under
`-n0`.
Every load looked up the loading indicator, which only the first page has,
so pressing enter in the search box raised `NoMatches`. The indicator is a
child of the list `remove_children()` already clears.
A document's id and search text travel on a `DocumentCheckbox` instead of
being assigned onto a `Checkbox` behind type suppressions. The positional
widget ids are gone; nothing queried them.
`_rich_print_search_result` asked how many databases were configured, so
`--db-name alpha` labelled every line with the database the caller had just
named. Citations already ask the client what it covers, which is the same
question the scope answers.
`settings` printed each top-level block as one Python dict repr, so nesting
was invisible, long paths wrapped mid-token and `PosixPath(...)` leaked. It
renders as YAML now, which is what `haiku.rag.yaml` is written in.
`list` does not load content, which is where the docling blobs live, so the
`content:` header announced a field the command declined to fetch. A document
without a uri or metadata printed `uri: None` and `meta: {}`, where `title`
was already omitted when absent.
`--database` and `--db` read as one word abbreviated but take different
things, a configured name against a filesystem path, and they sit in
different positions: `haiku-rag --db /path info` fails with `No such option`.
The CLI already says "database" to the user everywhere it prints one, so the
name it selects by should say the same. Unreleased, so no deprecation.
Also: without a reranker, raise `search.limit` with the number of databases
searched, since each contributes its best matches to a list truncated back
to the limit.
Two files of 1,405 and 814 lines become seven: scope resolution, lifecycle,
search, documents, expansion, citations and capabilities. `_config`, `_seed`
and the rest move to `helpers.py`, importable by the sandbox tests that share
them, and the package points VCR back at `tests/cassettes/multi_db/`.
`covers_several_databases` had one line of body and two call sites.
`_db_path_given` guarded a default path manufactured in `__init__` and
overwritten in `__aenter__`; `_requested_db_path` is what the caller asked
for, and the effective path falls out of the resolved scope.
`clients_for` hands back a client over a database the covering one owns, so
`async with` on it opened a second session and assigned it, and teardown
declined to close what this client did not open. Entry returns the client as
it stands.
They opened a client on `tmp_path / "unused"` with the bucket in the
configuration. An explicit path now selects the database, so the client
tests ran against the local disk and the two app tests failed. Nothing names
a path any more, and both helpers assert the connection is remote before
yielding, so a later precedence change cannot quietly localize them again.
`resolve_db_path` manufactured the default path whenever `lancedb.databases`
was empty, and `covers_several_databases` read coverage back out of the
configuration, so a capability built without a client opened
`storage.data_dir/haiku.rag.lancedb` instead of what `lancedb.uri` placed.
The entry point resolves a `DatabaseScope` instead: instructions ask it what
it covers and `_ensure_rag` opens it through `HaikuRAG._covering`, so
coverage is decided once rather than encoded in a path and re-derived.
`Sandbox._covering` takes the scope the capability already resolved, beside
the public constructor that takes a path. The factory signatures are
unchanged.
Closes#582. `lancedb.databases` entries already classify a location by
whether it carries a scheme; `lancedb.uri` was taken as a URI whatever it
said, so a local path was opened as object storage and a mistyped one
became a new empty database instead of failing. Both settings now place a
database the same way, and `--db PATH` overrides either.
`list_documents()` reached into the ownership map for a name the document
already carries. The map is still built, so a document id two databases
claim is still refused before anything mounts. A listing over one named
database now reports that name, as in-code `search()` already does.
`Store` already parses the settings blob on open, so keep it: the
inspector's info modal was opening the settings table, querying it and
parsing the JSON a second time, and `doctor` was asking
`SettingsRepository` for it on a store it already held. Creating a
database refreshes it, so a new one reports the version init wrote.
`gather_database_info` keeps its own parse: it goes around `Store` so a
pre-migration database still reports what it can.
A chunk id is unique within a database and says nothing across them, so a
database copied from another holds the same ids. `qualified_id` keys the
two in-memory identity sites on the database and the id together:
`merge_results` was dropping the second database's result when a query
repeated, and the arrival map that breaks fused score ties was ranking one
of the pair as the other.
Everything serialized records the id alone, so there ambiguity is refused
rather than qualified. `resolve_citations` raises `AmbiguousCitationError`
for a cited id held by two of the databases searched, where it used to
resolve to whichever result came last; `_register_citations` raises for one
already cited from another database in an earlier question. `_cite` turns
both into a `ModelRetry` asking for other evidence. The direct-id fallback
asks every database the question covers instead of taking the first that
answers, so an id no search returned is refused on the same terms.
`all_found` collects them and `first_found` reads its first, which document
reads keep doing on purpose.
Also drop a duplicated 0.77.0 heading from the changelog.
Expansion branched on whether the client covered a set, so the
single-database half reached for repositories through a facade that may
have none. `expand_sources` groups results by database and hands each
group the session that owns it; `expand_context` and `visualize_chunk`
take that session, so `visualize_chunk` stops narrowing to one database
and discarding the result.
Inline `_fetch`, a pass-through to the chunk repository.
The CLI decides only what it knows — that --db and --database are the same
thing said twice, and whether a command reads more than one — and hands the
resolved scope down. Nothing rewrites the configuration, so a named database
keeps the name results and citations carry, and a remote one opens the URI
it was configured with rather than the local path standing in for it.
HaikuRAGApp, ChatApp and InspectorApp take that scope and nothing else.
Selection reaches the client through a private constructor, so the public
signature still takes a path or names.
covers_multiple, source_names, source and reader_for replace the private
state seven modules were reading to work out how many databases they had.
The configured selection is kept intact, so entering a client twice derives
the same database rather than the last derivation.
Every implementation in documents.py and rebuild.py takes the session it
writes to, so a set cannot reach one: the facade narrows once and passes the
database on, rather than checking and carrying a union. Tests calling an
implementation directly go through `writing()`.
One session field, whichever kind it is, with the covered databases derived
from it rather than kept beside it. `_single_session` returns the database a
write works on in place of ten guards that only asserted one existed, and the
single-database operations move to the session that owns them.
FederatedSession opens the databases a query covers and owns their
teardown; the client keeps the wrappers it hands out. A wrapper releases
what it built and never closes the database it borrowed.
SingleDatabaseSession owns the store, the repositories and the vacuum
machinery, so nothing above has to ask whether it has a store. The client
keeps every name callers already use.
DatabaseScope.resolve reads configuration and at most one selector; a
DatabaseRef carries the configured name and a location already resolved to
a URI or a path, so a path a caller names is never reinterpreted. Nothing
consumes it yet.
An embedder is a function of configuration, not of a database, and the
databases in a selection share one, so a client covering a set builds it
on first use and closes it on teardown. Operations that need one database
say so instead of surfacing a missing store.
An image query has no text to match, so it is vector-only whatever the
caller asked for, and full-text search embeds nothing, so it needs no
agreement on embedders.
`first_found` replaces the client's private version and the capability's
sequential one. The info modal reports through the connection the client
already holds.
A title repeats within a corpus and across databases, so a substring
match on the displayed name widened the filter to documents the user did
not pick. The label names the database.
Each database owns an embedder, so embedding per database cost a round
trip each. One database still embeds inside the repository, which returns
early for a filter that matches nothing.
A blank name is falsy, so source routing reads it as absent, and a blank
location resolves to the working directory. `is_read_only` reports the
mode the client was opened with, which a client covering a set can answer
without a store.
Document.source names the configured database, as SearchResult and
Citation already do. A listing spanning databases is unreadable
without it, and `--database NAME list` could not name the one it
opened.
`get_document_by_id`, `get_document_by_uri` and `get_chunk_by_id` read
through repositories a client covering a set does not have, so a lookup by
identifier raised AttributeError and `resolve_document` with it. An
identifier says nothing about which database holds it, so every database is
asked at once and the first that has it, in configured order, answers.
On the evaluation side, `--db` overrides the configured set as the CLI
documents, and population refuses a set rather than ingesting into a
database the run would not read. A case filter matching nothing raises
instead of reporting 0.0000 as though it were a score.
A capability covering several databases received `source` on every document
and search result and never used it: asked how many documents were in each
database, the model read the titles and answered that there was one corpus
of 67,581. The instruction files enumerate what a result carries, and both
enumerations had gone stale.
The note follows what the capability opens rather than what the
configuration names, through `covers_several_databases`: an explicit
`db_path` or a lent client covering one database is instructed as before,
as is every `uri` or path deployment and every eval dataset. The analysis
note separates the three interfaces, since they differ: an
`analysis_search` result carries a `Database:` line, in-code `search` and
`list_documents` return `source`, and the mounted files carry neither.
The modal listed every document and mounted a checkbox per document, so a
corpus of tens of thousands never finished rendering: 67k sequential
mounts across two databases, and the same for one database that size.
Titles repeat at that scale too, and the ids were derived from the title,
so they collided.
It shows a page of 200 now, mounted in one call and identified by
position, and the search box asks the database for the rest on enter,
matching titles and URIs. Typing still narrows the page on screen, for
feedback while typing. `search_filter` escapes the term, which is
whatever was typed.
A federated listing takes its window across the databases rather than
filling it from the first one: concatenating hid every database after
whichever was listed first, which for a set of a thousand papers and
sixty thousand articles meant a page of papers alone. Sorting would not
have helped, since document ids and article titles sort into separate
runs, so the page is picked by interleaving and the modal sorts it for
display.
Chat answers with the same capabilities `ask` does, so it federates as
naturally as `ask` and `analyze` — but it went through the one-database
guard and refused a configured set outright, which left no way to chat
across several databases.
The guard was the visible half. `run_chat` also defaulted `db_path` to the
single default path whenever it was None, so lifting the refusal alone
would still have opened one database. It now leaves the path unresolved
when `lancedb.databases` names the set, and the client resolves it.
Listing and counting documents fan out over the set, which is what the
document filter reads, and visual grounding resolves the database holding
the cited chunk through the citation's source: chunks, pages and bounding
boxes all come from that one database. A limit on a listing means that
many documents in total, not that many per database.
The info modal reports every database it covers, each under its
configured name and without its location, since names are the only
identity that leaves the configuration. `database_lines` is what one
database reports about itself, shared by both paths, and it reports a
failure as a line so one unreachable database does not cost the report on
the others.
`inspect` stays a one-database command. It browses one database's
documents and chunks, so a set has nothing to show it.
The CLI reports this error, and importing it from the settings repository
pulled lancedb onto the CLI's import path. Deferring the import into cli()
bought nothing, since cli() runs on every invocation: it cost about 1.9
seconds on a cold start, `--help` included.
It now sits beside the other store exceptions, in a module that imports
nothing, and every importer points there.
A database whose stored embedder disagrees with the configuration raised
ConfigMismatchError through Typer, so the operator got a traceback wrapped
around the one message that says what to run, while every sibling failure
exits with its message. It joins the errors the CLI reports.
Imported inside cli() rather than at module scope: the settings module
pulls in lancedb, and importing the CLI must not pay for it.
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.
The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.
The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.
On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
`ask(sources=[…])` scopes a question to some of the configured databases, carried
on the capability state so its search tool searches those. `Citation.source` names
the database a cited chunk came from, resolved from the search results the model
saw, which already carry it.
Context expansion routes each result through the database it came from: a
federating client has no repositories of its own.
The cite fallback, which looks up an id absent from this run's results, searches
only the selected databases. A chunk id says nothing about which database holds
it, so placing one means asking, and asking outside the selection would let a
question scoped to some databases cite another.
The loosely-specced client mocks in the capability tests now say they stand in for
a single-database client. A bare AsyncMock answers any attribute with a truthy
Mock, so `_federated` sent the fallback down the multi-database branch, and
`_source` reached a validated field.
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.
Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.
A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.
Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.
The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.
A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.
With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
`search` fetched, reranked and truncated in one pass, with the reranker's
over-fetch and the reranking itself interleaved in the same branch. Searching
several databases needs to fuse their candidates before anything is ranked, so
the phases have to be separable.
`_fetch` returns one database's candidates, over-fetching only when a reranker
will re-order them. `_rank` orders and cuts them, leaving an image query's vector
ranking alone since there is no text for a reranker to score against. The
over-fetch multiplier is named rather than a literal 10 at the point of use.
Both check the query type before reading `client.reranker`, which is a
cached_property that builds the reranker on first access and loads model weights
for a local one. An image query never used it and must not start.
No behaviour change: the same suite passes, and the search outputs digest
identically to before.
`Agent.from_spec` raised `TypeError` on `RAGCapability` and `AnalysisCapability`,
whose constructors take a state class, packaged instruction text and a tool-name
set, and silently omitted both from the generated spec schema. The two
zero-configuration capabilities constructed but with `id=None`, so pydantic-ai's
duplicate-id rejection no longer held and a spec could register two citation
policies, defeating the single-decision-maker invariant.
Override `from_spec` on all four, delegating to `create_capability()` so ids and
instructions come from one place. The spec surface is `db_path`, `config`,
`defer_loading`, `request_limit` and `vision`; a live `HaikuRAG` client stays out
of it, and a `config` mapping is validated through `AppConfig`.
The documented `HaikuRAG("knowledge.lancedb")` and `rag(db_path="my.lancedb")`
forms both raised `AttributeError: 'str' object has no attribute 'exists'`.
`Store.__init__` assigned its argument to a `Path`-annotated attribute without
coercing, and `resolve_db_path` returned a non-None argument unchanged. Every
runnable example wraps the path in `Path(...)`, which is why it survived.
Coerce in `Store.__init__` and `resolve_db_path`; widen the annotations on
`Store`, `HaikuRAG` and both `create_capability` factories to accept `str`.
urlparse().path keeps the leading slash in front of a Windows drive, so
file:///C:/docs/a.pdf read as \C:\docs\a.pdf and the ingester reported
"File does not exist" for every file it discovered. url2pathname is the
stdlib conversion that strips it, per platform.
Four sites each decided both "is this local" and "what path is this":
FSSource._uri_to_path and supports, resolve_adhoc_fetcher,
create_document_from_source and check_source_accessible, and convert.
is_local_uri and uri_to_path in haiku.rag.uri own those two decisions now,
which closes two more cases of the same root cause. A bare C:\docs\a.pdf
parses with scheme "c", so add-src raised "No source adapter for URI scheme
'c'" and convert silently treated the path as raw text. And convert and
check_source_accessible never percent-decoded at all, so a file named
a[b] c.md read as missing on Linux and macOS too.
A file URI's host is reattached after conversion rather than passed to
url2pathname, which as of 3.14 rejects a non-local authority off Windows.
file:////server/share is the empty-authority spelling of a UNC path, its
host being the first path segment, so that host is normalised into the
authority before conversion. Output is identical on 3.12, 3.13 and 3.14.
The ad-hoc FS fetcher roots at the path's own anchor rather than "/", which
on Windows is only the current drive.
test_uri.py runs on ubuntu, macos and windows across 3.13 and 3.14 without
the project installed: --noconftest because the repo conftest imports
dependencies that job does not need, and -o addopts= to drop the
repository's -n auto. The Windows legs are what cover the drive conversion.
Fixes#574.
cli.py carried 40 pragmas over whole command bodies and app.py a
class-level one over all 412 statements, while tests/test_cli.py already
drove 29 commands through CliRunner. The pragmas hid lines the suite
executed, so the 100% gate understated real coverage and gave new CLI code
no scrutiny.
Both are measured now. 38 CLI tests stub HaikuRAGApp and assert the parsed
arguments reach the right method; 60 app tests stub the client and record
the console, pinning what each command asks for and what it prints. The only
pragma left in either file is cli() under __main__. The omit list is back to
the two Textual TUIs.
Three defects the coverage surfaced:
haiku-rag settings masked only top-level secret-named fields, so nested ones
printed in full — lancedb.api_key, providers.docling_serve.api_key, WebDAV
source passwords. It uses redact_secrets, which walks the dump.
chat guarded the wrong thing: haiku.rag.chat imports without Textual, and
run_chat raises when it imports ChatApp, so the missing extra escaped as an
ImportError. The guard is on the call. inspector raises at module import
instead, so inspect keeps its guard on the import; each has a test that
fails the way the real installation fails.
search --limit/--search-type and history --limit default to None so the
config resolves the default. Now pinned.
CI passed --cov=haiku while pyproject declares source = ["haiku_rag_slim"];
pass --cov and let the config decide. build-docs.yml only ran on push to
main, so a broken docs build merged and failed at deploy: build on pull
requests, with configure-pages, upload-pages-artifact and deploy gated to
push, and a per-ref concurrency group.
engine.py held four unrelated things: what the tables are, how to open a
connection, how to read a database's state, and the Store that coordinates
writes. At 1240 lines the Store's own concerns — locks, migrations, vacuum,
tags — were hard to find among them.
Table records, Arrow schemas, index_specs, ensure_indexes, REQUIRED_TABLES
and query_to_pydantic move to store/schema.py, which imports nothing from
haiku.rag: it describes the tables and never opens or mutates one.
gather_database_info, get_database_stats, DatabaseInfo and its result models
move to store/info.py. Nothing in Store calls them — they are read paths for
the CLI, doctor, inspector and ingester API — so info depends on engine and
not the reverse.
engine.py keeps the Store, ConnectionMode, connect_lancedb, the tag helpers
and the restore-order and retention constants. No re-exports: importers
point at the new modules.
test_app_info_uses_connect_lancedb_for_remote patched
haiku.rag.store.engine.connect_lancedb; gather_database_info now binds that
name in info.py, so the patch targets where the call is looked up.
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.
Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.
The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.
Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
Five call sites repeated the same post-conversion preparation: store the
Docling representation and resolve a title when none was supplied.
_prepare_and_title now owns that sequence. update_document continues to
call _prepare_document_from_docling directly because an explicit update
must preserve an existing empty title.
create_document, both content-replacement branches of update_document,
and source ingestion embedded eagerly before passing chunks to a
persistence funnel that checked them again. The funnels now own
embedding, including the checks required by import_document and
import_documents for caller-supplied chunks.
Move the document.embed span into ensure_chunks_embedded after its early
return. Every path that performs embedding is now instrumented, while
operations whose chunks are already embedded emit no span.
convert() previously used its own HTTP client and temporary-file path.
Route URL conversion through HTTPSource, matching source ingestion, and
move _write_fetch_body to processing.py so both paths share temporary
file handling without an import cycle.
Add walk_files for filesystem enumeration and use it from both
FSSource.discover and one-shot directory ingestion. Symlink escape
filtering now has one implementation.
RAGState and AnalysisState each declared the same five fields, so the generic
base could not name them: StateT was bound to BaseModel, and every access
went through cast(Any, state), a getattr by string, or a loop clearing fields
by name so it could skip the one only AnalysisState has.
EvidenceState declares them once. RAGState adds nothing, AnalysisState adds
executions and overrides begin_invocation to clear them. StateT binds to
EvidenceState, which removes all ten casts and both state-shape getattrs; the
three getattr(ctx.deps, "state") probes stay, since those check a
host-supplied object rather than our own state.
discover_evidence reached into capability.state for two fields. It now asks
through evidence_record() and citation_index(), alongside the
evidence_tool_names() and cite_available accessors it already used. The eval
runner's _RagLikeState protocol and the chat app's getattr reads described
this shape from outside and are gone.
Compatibility is semantic JSON-object equivalence, not bytes: field names and
nesting are unchanged, so a dict stored by 0.75.0 loads and re-dumps equal,
but deriving from a shared base reorders AnalysisState's keys. Nothing
serializes, hashes or string-compares this state — every carry point
re-validates by key.
search.limit was documented as 10 in three places while the default is 5.
The documented way to disable reranking, provider: "", is a valid
ModelConfig, so it raised "Unknown reranking provider" — disabling means
omitting reranking.model or setting it to null. The inline provider list
named four of the six rerankers. prompts.picture_description: null fails
validation, since the field is a non-optional str.
storage.data_dir: "" coerced to Path("") — the working directory — while two
doc pages promise the platform default and soliplex's example config relies
on it. Empty or whitespace now resolves to the platform directory; an
explicit "." is still honoured, so a config that wants the working directory
says so.
Three tests keep this from drifting again: every fenced yaml block in the
docs validates against AppConfig, every value in the complete example either
equals its default or is listed as a deliberate deviation, and empty
data_dir resolves to the platform default.
init-config's test reimplemented the command body instead of invoking it,
which is why the command carried a coverage pragma. It now goes through
CliRunner, with the refuse-to-overwrite guard covered too.
Every section inherited plain BaseModel, so unknown keys were dropped
silently: providers.docling_serve.timeout was documented for months while
being ignored, and a typo in any setting took the default. Sections now
derive from ConfigModel, which forbids extras, so a stale or misspelled key
fails with its path. This already found search.context_radius in a live app
config and providers.vllm in soliplex's example.
converter, chunker and chunker_type are Literals. Sizes, limits,
dimensions, token budgets, attempt counts and breaker thresholds must be
positive; retention, delays, intervals and cooldowns non-negative;
similarity_threshold within 0-1; port within 0-65535. port 0 keeps its
OS-assigned meaning and worker_count allows 0 for an API-and-reaper-only
process.
get_reranker caught ImportError and returned None, so a configured reranker
whose extra was missing silently disappeared. It now propagates.
raise_missing_extra names the install command and re-raises when the failure
came from inside an installed package, so a broken transitive import is not
reported as a missing one. zeroentropy imported bare and now guards like the
others.
The haiku.rag package declares the jina extra. jina-local already worked
there through cross-encoder's transitive transformers and torch; the
resolved package set is unchanged, but the support is now promised rather
than inherited.
Provider fields stay unconstrained: get_model ends in a pass-through to
pydantic-ai for any provider it supports, so a Literal there would reject
valid configurations.
