Document searching several databases

lancedb.databases, the sources argument and the --database selector had no
documentation. Adds a Several Databases section to the storage
configuration page covering the name-to-location map, its mutual
exclusion with uri, the shared embedding configuration the set requires,
and which commands cover the set against which work on one. Adds a
Searching Several Databases section to the Python API page, --database to
the CLI's global options, the key to the sample configuration, and one
README feature line.
This commit is contained in:
Yiorgis Gozadinos 2026-08-20 16:17:32 +03:00
parent fdb5710491
commit e15f73a387
No known key found for this signature in database
5 changed files with 103 additions and 0 deletions

View file

@ -26,6 +26,7 @@ Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
- **Several databases** — Name databases in `lancedb.databases` and search, ask or analyze across them at once, with each result and citation carrying the database it came from
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
- **CLI & Python API** — Full functionality from command line or code
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)

View file

@ -7,6 +7,7 @@ The `haiku-rag` CLI provides complete document management functionality.
- `--config` - Specify custom configuration file
- `--read-only` - Open database in read-only mode (blocks writes, skips upgrades)
- `--database` - Name of a database from `lancedb.databases` to work on
- `--version` / `-v` - Show version and exit
Per-command options:
@ -19,9 +20,15 @@ The `haiku-rag` CLI provides complete document management functionality.
haiku-rag --config /path/to/config.yaml list
haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db
haiku-rag --read-only search "query"
haiku-rag --database medic list
haiku-rag add -h
```
With `lancedb.databases` configured, `search`, `ask` and `analyze` cover
every database in it. Every other command works on one, named with
`--database` or `--db`. See
[Several Databases](configuration/storage.md#several-databases).
## Document Management
### Add Documents

View file

@ -88,6 +88,7 @@ lancedb:
uri: "" # Empty for local, or db://, s3://, az://, gs://
api_key: ""
region: ""
databases: {} # Name-to-location map to search several at once, instead of uri
embeddings:
model:

View file

@ -174,6 +174,72 @@ The recommended layout for production is "different buckets, same account, separ
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
## Several Databases
`lancedb.databases` maps a name to a location, for searching several databases at
once:
```yaml
lancedb:
databases:
medic: s3://my-bucket/medic.lancedb
st: s3://my-bucket/st.lancedb
local: /data/notes.lancedb
```
A location is a URI or a local path. `databases` and `uri` are mutually
exclusive, and setting both fails validation.
The name is the only identity that leaves the configuration. Results, citations
and error messages carry it, so a path or a bucket never reaches a log, a trace
or a model.
Every database in the set is opened with the same embedding configuration, so
they have to agree on it. One whose stored settings differ raises
`ConfigMismatchError` when it is opened.
### Searching a set
`search`, `ask` and `analyze` cover the whole set, or the subset named by
`sources`:
```python
results = await client.search("query") # every database
results = await client.search("query", sources=["medic"]) # one of them
```
Candidates from each database are fused into one ranked list, by the configured
reranker where there is one and by reciprocal rank fusion otherwise. Each result
carries `source`, the name of the database it came from, and so does each
citation.
A database that cannot be opened fails the whole query and is named in the error.
A result set silently missing one of the databases asked for cannot be told apart
from a complete one.
### Commands that work on one database
`haiku-rag search`, `ask` and `analyze` cover the configured set. Every other
command works on a single database, named with the global `--database` option:
```bash
haiku-rag search "query" # every configured database
haiku-rag --database medic list # one of them
haiku-rag --database medic migrate
```
`--database` takes a name from `lancedb.databases`, which is how a database
behind a URI is reached. `--db` takes a path, and overrides the configured set
with that one database. A command that works on one database and is given
neither fails rather than choosing for you.
Each database is created, migrated and vacuumed on its own:
```bash
haiku-rag --database medic init
haiku-rag --database st init
```
## Vector Indexing
Configure vector search settings:

View file

@ -230,6 +230,34 @@ for result in results:
print(f"Document Title: {result.document_title}") # when available
```
### Searching Several Databases
With [`lancedb.databases`](configuration/storage.md#several-databases)
configured, a client covers every database in it. `sources` narrows a call to
some of them, and each result names the database it came from:
```python
results = await client.search("machine learning") # all of them
results = await client.search("machine learning", sources=["medic"]) # one of them
for result in results:
print(f"{result.source}: {result.content}")
```
`ask` and `analyze` take `sources` too, and every citation carries the database
it was drawn from:
```python
answer, citations = await client.ask("What changed?", sources=["medic", "st"])
for cite in citations:
print(f"[{cite.source}] {cite.document_title or cite.document_uri}")
result = await client.analyze("How many documents mention it?", sources=["medic"])
```
A question scoped to some databases can only cite those, and the analysis
sandbox mounts only their documents.
### Filtering Search Results
Filter search results to only include chunks from documents matching specific criteria: