Vacuum compacts documents and document_items through lance directly
with a fragment target derived from storage.compaction_target_bytes
(default 2 GiB), instead of AsyncTable.optimize, whose default
1,048,576-row fragment target re-merges the whole table on every pass
and peaks at 5-6x the table's bytes. The target is sized from the
widest fragment's bytes per row, so peak memory is bounded by the
target instead of the corpus. lancedb's async API exposes no
compaction options (lancedb/lancedb#2325), hence the pylance
dependency.
A configured or default database that does not exist raises
SourceUnavailableError with the way to create it and without its location.
An unknown --db-name lists the databases there are, configured or default.
The CHANGELOG records Store.db_path, SingleDatabaseSession(ref, config),
the empty lancedb.uri migration and the error-type change.
An empty lancedb.uri, which init-config used to emit, is told to remove the
key; only a non-empty value is told the databases spelling. haiku-rag --db
and haiku-ingester --db turn a path with no stem into typer.BadParameter.
HAIKU_RAG_DB and DB_PATH are gone: a capability covers what the
configuration places or the db_path it is given, and the app backend and
the AG-UI example load their configuration as the CLI does. The compose
files point HAIKU_RAG_CONFIG_PATH at the mounted haiku.rag.yaml, which
places the database at /data where DB_VOLUME is mounted; the backend
refuses a configured set since it serves one database. The chat scopes a
selection by source only over a set and names databases on filter rows
only across several. Docstrings, docs and test fixtures stop describing an
unnamed database; every database a search, listing or citation reports
carries a name.
DatabaseRef is a name and a location. The configuration places databases
through lancedb.databases alone; with none configured the default is the
entry haiku.rag under storage.data_dir, selectable like any other.
lancedb.uri is removed, and a config carrying it fails to load with the
replacement spelled out. A path passed from Python is valid where the
configuration places nothing and raises AmbiguousDatabaseError beside
lancedb.databases; haiku-rag --db and haiku-ingester --db construct the
scope directly, so a human's override keeps working. Every database
answers to a name, and a database given as a path keeps its own errors.
Store, connect_lancedb, gather_database_info and run_doctor take a
location, a path or a URI, and classify it with ConnectionMode.of.
SingleDatabaseSession owns the resolved DatabaseRef and passes its
location down. This removes DatabaseRef.connection(), default_db_path,
the placeholder path for URI-backed databases and the per-database
config copies, so the configuration a client holds is the one the caller
gave it. The chat hands its capabilities the scope it opened along with
the client it lends, and the v0.58.0 migration no longer checks local
free disk for a database behind a URI.
A picture deduplicated within a burst must still reach the model once in
its own question and, when cited, again via the capsule re-fetch after
compaction; uncited it is dropped. Two wire tests pin both paths. The
harness builds the rag capability with defer_loading=False: a deferred
evidence capability the model has not loaded presents an empty record,
so compaction computes boundary 0 and rewrites nothing.
qa.md describes max_searches in units.
Sibling searches emitted in one response overlap heavily (40.6% of
returned chunk slots on Glimmer ORB fan-out cases). A result whose
rendered evidence a sibling already showed keeps its rank slot but
collapses to a reference line, and a picture attaches once per response
keyed on (source, document_id, self_ref). Equivalence is the
format_for_agent rendering at neutral rank/total plus picture keys,
bucketed under the qualified chunk id, so another database's copy or a
different expansion of the same anchor formats in full.
Search state now commits only after formatting and image construction
succeed: a raising image build no longer leaves results citable that
the model never saw, notes evidence for them, or suppresses a later
sibling.
Searches a model emits in one response share a budget unit, up to
FREE_SIBLINGS_PER_ROUND (3) per unit; sequential searches pay one unit
each, as before. Grouping keys on RunContext.run_step, which pydantic-ai
increments once per model request. A budget-rejected round fails all its
remaining siblings, and tracking resets per run.
Glimmer opens most questions with a burst of ~3 rephrasings in a single
response (95.8% of its three-search ORB cases are one-response bursts),
spending 3 of 5 searches before reading anything. Pass rate at 3 calls
equals 1 call, so a burst is priced as one probe.
The IVF probe count was the one vector search parameter with no
setting, and it is what bounds recall on a large indexed corpus:
partitions grow with the corpus, so a fixed probe count covers less of
it, and vector_refine_factor can only re-score what the probes
returned. Defaults to 20, matching lance, so search is unchanged.
Also corrects the create-index docs, which predate the measurement:
optimize() covers new chunks, so a rebuild is about retraining
centroids rather than reaching unindexed rows.
Retrieval scores are each database's own rank arithmetic; the databases
in a selection share an embedder, so similarity in that one space is the
signal comparable across databases by construction. Measured product to
product against score ordering: +8.3 to +16.6pp recall@5 across five
cells on two corpora, flat in collection count and corpus shape where
score ordering dips with both, closing roughly 60% of the gap to a
reranker; order-sensitivity residual 0.00pp in every cell. Exact ties
collapse from 51-81% of candidates to under 1%. Full-text-only searches
keep retrieval-score order, having no query vector. The vector column
already travels with every search result, so the similarity costs no
additional transfer; per-chunk embeddings are materialized only for the
federated path that reads them.
Rank interleaving guarantees every database slots regardless of content;
on domain-split collections it allocates no better than chance and costs
4.7pp recall@5 at four collections against score ordering (7.1pp at
eight). Hybrid scores are each database's own vector/FTS rank agreement,
which carries across databases; equal scores resolve by within-database
rank, and only a tie on both falls to configured order, leaving
permutation sensitivity at 0.02-0.26pp. Fused results carry the
candidate's own retrieval score, so the context-expansion re-sort
preserves fused order.
Disjoint corpora give every database's rank-r candidate the same RRF
score, and the stable sort resolved those ties to lancedb.databases
declaration order, discarding the retrieval scores entirely. Ties now
break on the raw retrieval score, which is uncalibrated across indexes
but only ever orders candidates within one rank tier: the databases in
a fusion share an embedder and ran the same search type, and it can
never lift a candidate above another rank. Hybrid per-database scores
are themselves rank-derived, so exact agreement still ties and keeps
configured order, deterministically. The n > limit depth quota is
unchanged, pending the retrieval eval.
The newest release with wheels for every supported platform: 0.38.0
publishes none for x86_64 Linux or Windows. The zero-coverage FTS scan
path is still broken at 0.37.1, so the guard, prevention and repair
carry unchanged. ensure_indexes names the index it declares, keeping
replace deterministic on 0.38+, where an unnamed create_index builds a
suffixed sibling instead of replacing a different-typed index. An
invalid search filter raises ValueError.
Deleting or replacing every indexed row while unindexed rows remain
returns lance to the zero-coverage scan path. ensure_indexes rebuilds
a declared FTS index that covers none of a populated table's rows,
and delete_by_document_id runs index maintenance like the other chunk
writes. A legacy database in that state is repaired by its first write.
An FTS index built over an empty table indexes nothing and lance never
catches it up on add. ensure_indexes skips FTS while the table is
empty; create, replace_for_document and embed-only rebuild ensure
indexes after writing, so the index always covers at least its first
rows. The first write into a fresh table writes one extra chunks table
version for the index build; a failed build fails the write.
lance serves unsorted results with matching rows dropped when a
declared FTS index has indexed nothing, or when a populated table has
no FTS index at all. doctor fails on both: vacuum remediation for an
existing index, rebuild --embed-only for a missing one, since optimize
never creates an index. The chunk repository warns once per repository
on the first FTS or hybrid search against either state; a failing
coverage check is logged and never fails the search. Removes
_ensure_fts_index, which had no callers.
Both were exempt from coverage, which is how the 0.38.0 blob-encoding bug
reached a release: nothing exercised the chain that produced it.
Move the historical `documents` shapes into tests/store/legacy_documents.py
so each migration's tests can seed the table as its predecessor left it.
v0.25.0 compresses through compress_json, which switched from gzip to
zstd in 0.38.0 — the same release that added this migration. v0.38.0
decompressed with gzip and fell back to a raw UTF-8 decode, and both
fail on a zstd frame, so upgrading any database older than 0.25.0 has
died with UnicodeDecodeError since the migration shipped.
Reported with a fix by @omaer0 in #586.
`_rich_print_document` escapes uri, title and metadata, the sibling of
the escaped search-result renderer. The remaining comments and
docstrings that narrated rejected alternatives, consequences or history
now state the current invariant. The Sandbox class docstring names the
held connection close() releases, and wrapped docs paragraphs join to
one line.
Comments and docstrings across the branch narrated rejected
alternatives, consequences and history; each now states the current
contract. Renames test_a_legacy_uri_client_keeps_its_error to
test_an_unnamed_database_keeps_its_error. Documents the Sandbox
connection paths, the citation header's database segment, both
AmbiguousDatabaseError conditions on create_app, and run_inspector's
scope parameter. Doc paragraphs added by the branch in python.md,
storage.md and cli.md are one physical line each.
Sandbox.close released the pool and the held federated client only when
the monty session's __aexit__ returned. Each release now runs under
suppress(Exception), matching _discard_session and aclose_quietly, so a
raising step neither masks an unwinding error nor leaks the databases a
federated `_opened` holds.
The chat filter modal keys a selection by (database, id): copies of a
database share document ids, and checking one copy left the other
reading as selected. Applying the filter narrows the question's
`sources` to the databases the selection names. A twin id inside
another selected database still matches there: a serialized id filter
cannot carry a source.
A database name, document title, uri or heading containing Rich markup
crashed search output, chat citations and the chat document filter with
MarkupError. Every interpolation into markup-parsed text now escapes.
`_fetch_limit` had no test: a text query over-fetches `limit * 10` only with a
reranker, and an image query keeps its vector ranking either way. The sandbox
test asserted `KeyError`, which `UnknownDatabaseError` subclasses, so it could
not tell the contract from a bare one.
`uses_configured_databases` documents a mapping of one as covered; the test
named for it passed no mapping at all. Its `config` parameter is an `AppConfig`.
`test_an_analysis_capability_mounts_the_configured_set` carried a VCR marker and
no cassette, making no HTTP calls.
`tag_restore` reached into `HaikuRAGApp._display_path` and `_path`, and seven
places spelled out `self._is_local and not self._path.exists()`. `display_path`
and `database_missing` say both, and `database_missing` is False for a database
behind a URI, which has no path to check.
`init` keeps its own check: it asks the opposite question.
A database name, a location and an exception message all reach the info modal
as configuration-derived text, and Rich reads `[...]` in any of them as markup:
a name like `beta [prod]` disappeared, and a message carrying `[Errno 2]` or a
stray closing tag could break the line it sat in.
Two collections can hold documents with the same title, and the collapsed
citation carried only that title, so the two read as one source repeated. The
name is appended when the client covers a set, the decision already made for
search results and retrieved images.
`resolve_document` and `find_document` selected a document through a listing,
then dropped its source and looked the id up across the set. Ids repeat between
copies of a database, so a title that matched in one could be answered by
another's document.
`get_document_by_id` and `get_chunk_by_id` join `get_picture_bytes` in taking an
optional `source`, and all three route it through `clients_covering`, so a name
the client does not cover raises `UnknownDatabaseError` rather than being
answered by the database it does cover. Without a source the reads are as they
were, answering from the first database in configured order that holds the id.
`_resolve_scope` returned a scope without keeping it, so a client re-entered
after its configuration was edited covered whatever the configuration then said.
Resolving once is what the rest of the design rests on: the scope is what names
results, citations and errors.
A configuration is still free to change before first entry.
Search results and capsule entries name the collection they came from; the
images attached beside them carried only the chunk id and reference. Two
collections can return the same picture of the same document, so the two
labels were identical and the model could place neither.
The decision is the one already made for the text: `covers_multiple` at the
generic search tool, and the flag `search_corpus` computed for the capability
tools, which it now returns.
`asyncio.gather` propagates the first failure while its siblings run on, and the
caller unwinding from that closes the set through `async with` — so a sibling
still reading reads through a closed session. Seven fan-outs were affected:
lookup, search, image enrichment, multimodal picture loading, context expansion,
document listing and counting, and the sandbox's document load.
`gather_all` cancels and drains the rest, then re-raises the original exception.
A `TaskGroup` would drain them too but raise an `ExceptionGroup`, which every
caller and both CLIs' exception handlers would have to unwrap. The two
`return_exceptions=True` gathers in session opening and teardown already drain
their children and are left alone.
Three places identified a picture by document and reference alone, and one
identified a citation's images by chunk id alone. Both repeat between copies of
a database, so a search returning a figure from two collections sent one, a
capsule retained one, and a citation rendered the other collection's figures.
Keyed on the source as well: `(source, document_id, self_ref)` for search
pictures, the capability and source for retained ones, and `qualified_id` for
the chat's citation images.
A capsule replaces earlier questions' evidence with what they cited, so the
searches that carried a `Collection:` line are gone by the time the model reads
it. Cited content survived; which collection it came from did not, and a
follow-up question attributed it to whichever the model guessed.
Named the same way a search result names it: the capsule decides, and only when
it spans more than one.
`ConfigMismatchError` and `SourceUnavailableError` escaped the entry point as
tracebacks rather than a message and exit code 1. The entry-point test is
parametrized over the translated types, so the two CLIs' lists cannot drift
apart silently again.
Typing narrowed the mounted checkboxes without touching the search the listing
was built from, so a term matching more than one page hid rows from whichever
page the user happened to be on, left the rest of the matches a page away, and
paged on the previous search while the term still sat in the box. The term now
applies on enter, and the footer says so until it does.
The set shares the reranker its facades borrow, not an embedder: each covered
database builds and closes its own. Sharing one needs the lender pattern inside
`Store`, since a store's embedder also serves writes and is closed with the
session, so it is left as follow-up rather than done here.
`reported_location` returned `client.location` and nothing else; its test
asserted that a mock returns what it was given. `list_documents` names its
results through `name_all`.
The lock was applied to an owner as well, so every owner-backed file read queued
behind the capability's tool calls to guard state it does not touch. An owner is
a session of its own and is yielded straight through, which is what the
docstring already claimed.
Chat asked the covering client for a picture with the citation's source, which
raises when there is none, losing the answer to one figure. It resolves the
reader first and omits the bytes when the database cannot be placed, as the rich
formatter already did.
Reads tagged the document they returned and writes did not, so creating in
`alpha` came back with `source=None` while reading the same document came back
with `"alpha"`. Every outward write result goes through the owning session.
`settings` and `download-models` read the configuration and open nothing, but
resolved a scope to be constructed, so `--db-name nope` failed them over a
selection they never use. `HaikuRAGApp` takes no scope for those, and asking it
for one is an error rather than a silent default.
`test_an_image_query_builds_no_reranker` never submitted an image query: it
opened clients and found nothing built, which is lazy construction. It sends one
now, so consulting the reranker before the query type fails it.
`TestComparingEmbedders` called `_require_one_embedder` and asserted nothing.
It rejects a disagreement and accepts an absent record, and searches through.
Three `ModelRetry` tests took any reason where the reason is the subject.
`haiku-ingester` reached the client's refusal, which names `sources=[name]`, a
Python argument no CLI user can pass. It refuses a configured set itself, with
`--db PATH` and the one-ingester-per-database layout.
`haiku-rag`'s own refusal now says where each option goes: `--db-name` is global
and precedes the command, `--db` follows it.
Creating re-read the settings blob and left `stored_embedding` at None, so a
client that created a database compared as though it recorded no embedder.
`_remember_settings` takes both, and the comment no longer says one follows the
other.
