Vacuum compacts documents and document_items through lance directly
with a fragment target derived from storage.compaction_target_bytes
(default 2 GiB), instead of AsyncTable.optimize, whose default
1,048,576-row fragment target re-merges the whole table on every pass
and peaks at 5-6x the table's bytes. The target is sized from the
widest fragment's bytes per row, so peak memory is bounded by the
target instead of the corpus. lancedb's async API exposes no
compaction options (lancedb/lancedb#2325), hence the pylance
dependency.
An empty lancedb.uri, which init-config used to emit, is told to remove the
key; only a non-empty value is told the databases spelling. haiku-rag --db
and haiku-ingester --db turn a path with no stem into typer.BadParameter.
DatabaseRef is a name and a location. The configuration places databases
through lancedb.databases alone; with none configured the default is the
entry haiku.rag under storage.data_dir, selectable like any other.
lancedb.uri is removed, and a config carrying it fails to load with the
replacement spelled out. A path passed from Python is valid where the
configuration places nothing and raises AmbiguousDatabaseError beside
lancedb.databases; haiku-rag --db and haiku-ingester --db construct the
scope directly, so a human's override keeps working. Every database
answers to a name, and a database given as a path keeps its own errors.
The IVF probe count was the one vector search parameter with no
setting, and it is what bounds recall on a large indexed corpus:
partitions grow with the corpus, so a fixed probe count covers less of
it, and vector_refine_factor can only re-score what the probes
returned. Defaults to 20, matching lance, so search is unchanged.
Also corrects the create-index docs, which predate the measurement:
optimize() covers new chunks, so a rebuild is about retraining
centroids rather than reaching unindexed rows.
`resolve_citations` documented the behaviour it now rejects: a shared chunk id
resolving to whichever result came last.
The storage page ran two embedding checks together as if the second explained
the first. They are separate: each database against the configuration on open,
and the databases in a selection against each other, which raises in read-only
mode too and does not apply to full-text search.
`several` becomes `multiple` where it names the feature, matching the docs and
`covers_multiple`.
Three groups, not two: `search`, `ask`, `analyze` and `chat` cover the set,
`settings`, `init-config` and `download-models` open no database, and every
other command works on one — or on a configured set of one, which is
unambiguous and keeps its name. A database named in `lancedb.databases` keeps
that name whether or not it is the only one covered; only `lancedb.uri` places
one without naming it.
Document ids repeat between copies of a database, where the sandbox refuses a
duplicate but the chat filter's `id IN (...)` matches the document in every
copy. `build_document_id_filter` claimed ids never widen a selection.
Document the facade: `covers_multiple`, `source_names`, `source`,
`reader_for`, `clients_for`, the lifetime of a borrowed client, and
`sources=None` against `sources=[]`.
Drop the vision callout from the README, which the features list already
covers.
A blank name is falsy, so source routing reads it as absent, and a blank
location resolves to the working directory. `is_read_only` reports the
mode the client was opened with, which a client covering a set can answer
without a store.
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.
Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.
A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.
Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.
The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.
A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.
With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
search.limit was documented as 10 in three places while the default is 5.
The documented way to disable reranking, provider: "", is a valid
ModelConfig, so it raised "Unknown reranking provider" — disabling means
omitting reranking.model or setting it to null. The inline provider list
named four of the six rerankers. prompts.picture_description: null fails
validation, since the field is a non-optional str.
storage.data_dir: "" coerced to Path("") — the working directory — while two
doc pages promise the platform default and soliplex's example config relies
on it. Empty or whitespace now resolves to the platform directory; an
explicit "." is still honoured, so a config that wants the working directory
says so.
Three tests keep this from drifting again: every fenced yaml block in the
docs validates against AppConfig, every value in the complete example either
equals its default or is listed as a deliberate deviation, and empty
data_dir resolves to the platform default.
init-config's test reimplemented the command body instead of invoking it,
which is why the command carried a coverage pragma. It now goes through
CliRunner, with the refuse-to-overwrite guard covered too.
Every section inherited plain BaseModel, so unknown keys were dropped
silently: providers.docling_serve.timeout was documented for months while
being ignored, and a typo in any setting took the default. Sections now
derive from ConfigModel, which forbids extras, so a stale or misspelled key
fails with its path. This already found search.context_radius in a live app
config and providers.vllm in soliplex's example.
converter, chunker and chunker_type are Literals. Sizes, limits,
dimensions, token budgets, attempt counts and breaker thresholds must be
positive; retention, delays, intervals and cooldowns non-negative;
similarity_threshold within 0-1; port within 0-65535. port 0 keeps its
OS-assigned meaning and worker_count allows 0 for an API-and-reaper-only
process.
get_reranker caught ImportError and returned None, so a configured reranker
whose extra was missing silently disappeared. It now propagates.
raise_missing_extra names the install command and re-raises when the failure
came from inside an installed package, so a broken transitive import is not
reported as a missing one. zeroentropy imported bare and now guards like the
others.
The haiku.rag package declares the jina extra. jina-local already worked
there through cross-encoder's transitive transformers and torch; the
resolved package set is unchanged, but the support is now promised rather
than inherited.
Provider fields stay unconstrained: get_model ends in a pass-through to
pydantic-ai for any provider it supports, so a Literal there would reject
valid configurations.
The setting was documented but did not exist on DoclingServeConfig, so it
was silently dropped, and DoclingServeClient.from_config never forwarded
the timeout parameter it already accepted. The per-request timeout was
therefore pinned at the constructor default of 300s with no way to change
it. Add the field, forward it, and reject a non-positive value.
Also parametrize over the checked-in *.yaml.example files and validate
each through AppConfig, so an example that no longer loads fails a test
rather than a user's first run.
Every `Store` built its own connection with its own caches and discarded them on
close, so the index a vector query loads was refetched by the next connection.
On object storage that first fetch dominates: measured on a ~500k-chunk 2560-dim
corpus over a ~200ms link, the first query cost ~41s and the second ~3s, and a
new connection reusing the session cost ~7s instead of ~47s.
`connect_lancedb` now passes a process-wide session, keyed on the configured
cache sizes so a caller asking for different sizes gets its own.
Also sets `read_consistency_interval`, defaulting to 30s. It was None, meaning a
connection never re-checked for other processes' writes. Per-call connections hid
that; a shared session makes connections long-lived enough for a reader to go
stale against the ingester.
All three settings reject negatives at the config boundary. A negative cache size
raises OverflowError and a negative interval panics inside Lance, so neither is
catchable further in. Zero stays valid for both: no cache, and check on every
read.
The routing tests now assert the kwargs they care about rather than the full call
signature, since every connection carries the two new kwargs.
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.
Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `ingester.api.root_path` so the HTTP control plane (dashboard + API)
can be reverse-proxied behind a sub-path (e.g. /ingester/) on a shared
origin, instead of needing nginx sub_filter URL-rewriting.
- APIConfig.root_path: normalized ('', or single leading slash, no trailing
slash) via a field_validator; validate_assignment so CLI overrides
normalize the same way as config-file values.
- Forwarded to FastAPI(root_path=) and uvicorn.Config(root_path=) so
OpenAPI/docs links are prefix-aware.
- Dashboard route injects a <base href> matching root_path; all dashboard
fetches are now base-relative, so they resolve under the prefix while
staying identical at the root.
- `serve --root-path` CLI flag.
- Docs: "Behind a reverse proxy" section with an nginx example.
Closes#431
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the ingester queue storage from raw aiosqlite to SQLAlchemy Core
async. The backend is chosen by ingester.queue.dburi: a SQLAlchemy async
URL points the queue at a database server, and SQLite remains the default
when unset. The Postgres path claims jobs with FOR UPDATE SKIP LOCKED so
multiple ingester processes can share one queue; SQLite caps the pool to a
single connection to keep the select-then-update claim atomic.
Large files buffered entirely in RAM can OOM workers. Add
max_file_size to source config (default None = no limit).
FS checks stat().st_size before read_bytes(). HTTP and WebDAV issue
a HEAD request before GET when a limit is configured. S3 checks the
size from the existing head_async() call before get_async().
FileTooLargeError is classified as PermanentError so oversized files
go straight to the DLQ instead of retrying.
