`format_for_agent` named the database whenever one was named, so a search over
a single named database carried a line with nothing to distinguish. It now takes
`include_collection` from the caller, which decides from the search selection
rather than from the hits: a search that could have drawn on two collections
names them even when everything came back from one.
`Collection:` at the model boundary, database in configuration and
administration. `source` on results, documents, citations and analysis
dictionaries is unchanged.
Two files of 1,405 and 814 lines become seven: scope resolution, lifecycle,
search, documents, expansion, citations and capabilities. `_config`, `_seed`
and the rest move to `helpers.py`, importable by the sandbox tests that share
them, and the package points VCR back at `tests/cassettes/multi_db/`.
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.
The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.
The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.
On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
Four call sites repeated lock, snapshot, try/except, restore. Each used
restore_table_versions, which restores in _tables() order — documents
first, contradicting RESTORE_TABLE_ORDER — and each caught Exception, so
a cancellation mid-write skipped rollback and left the earlier table
writes committed.
Store.write_transaction() holds the lock, snapshots under it, and rolls
back through _rollback_to_snapshot: RESTORE_TABLE_ORDER, shielded from
cancellation, absorbed cancellation re-delivered, rollback failure raised
with the original as cause. The two single-table update_meta sites keep
the bare lock.
The batch documents write moves inside the guarded body; it was outside
the try, so a failure there was never rolled back. Auto-vacuum is
scheduled after the transaction rather than inside it.
restore_table_versions is removed; those four sites were its only callers.
create_document_from_source resolves a fetcher per call and never closed
it, so every one-shot URL or WebDAV ingest leaked an httpx connection
pool. Close it, but only when we built it: resolve_adhoc_fetcher returns
a caller-supplied source when one matches the URI, and the ingester keeps
those open across jobs.
Directory ingestion also yielded symlinked files resolving outside the
directory it was given. rglob does not recurse into symlinked
directories, so a symlinked file was the only way out of the tree; skip
those, as FSSource.discover already does.
The chunk repository docstring named client._ensure_chunks_embedded,
which does not exist.
FULL rebuild deleted a document before re-ingesting it from its URI, and
the handler around that logged and continued. A 404, a timeout or any
conversion error therefore removed the document permanently.
Deleting after a successful create is not an alternative:
create_document_from_source resolves the same URI to the existing
document and updates it in place, so a trailing delete would remove the
freshly rebuilt row.
Refresh in place instead. create_document_from_source takes an internal
force flag that skips the revision and MD5 short-circuits, so an
unchanged source is still re-converted, re-chunked and re-embedded into
the existing document, and the document id survives a rebuild.
A failed refresh now falls through to the stored-content path rather
than skipping the document: FULL recreates the chunks table before the
loop, so skipping left the document present but unsearchable until the
next rebuild.
The pending-batch flush moves out of the try. A failed flush is a lost
write and should abort the rebuild, not be logged and skipped.
A crashed worker used to poison the rest of the run. execute() reported the
crash as a failed result, but kept the dead session, and every later call then
raised RuntimeError out of the tool. Clear the session so the next call checks
out a replacement. Keep the session for a syntax or runtime error, which leaves
the worker healthy, and say in the failure text that a restart loses the
variables.
A MemoryFile accepts writes, so metadata.json took them while the other three
document files refused. Mount it through the same read and deny pair. The
write-denial test now covers all four files.
Document files can be read with open() and with-blocks (.read(),
.readline(), .readlines()); writes raise PermissionError. File objects
remain non-iterable and the collections module is still unavailable.
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.
Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.
Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.
Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.
Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.
Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
Add vector-index creation tests including the warned failure, chunk
repository get_by_id, list_all pagination, blank-query and precomputed-vector
search, the unknown-score-column guard, settings row recreation, and
replace_for_document with no items.
Add tests for the vacuum-failure warning, documents deleted mid-rebuild,
chunkless documents, batch flushes in the embed-only, descriptions and full
paths, a missing docling blob under rechunk, missing and recoverable picture
bytes, and the source-missing fallback. Direct-call tests cover the staging
helpers and the idempotent phase-1 marker.
Parametrize sibling tests that differed only in a literal value, and fold
two strict-subset tests into the survivors that already covered their
scenario. Every case that ran before still runs; the union of assertions
is applied to each case, strengthening list_all, get_pages_data and
resolve_doc_items.
Replace four hand-rolled log-capture handlers with a shared
capture_logs() contextmanager in conftest.
13 fewer test functions, 348 fewer lines.
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
