Document searching several databases
lancedb.databases, the sources argument and the --database selector had no documentation. Adds a Several Databases section to the storage configuration page covering the name-to-location map, its mutual exclusion with uri, the shared embedding configuration the set requires, and which commands cover the set against which work on one. Adds a Searching Several Databases section to the Python API page, --database to the CLI's global options, the key to the sample configuration, and one README feature line.
This commit is contained in:
parent
fdb5710491
commit
e15f73a387
5 changed files with 103 additions and 0 deletions
|
|
@ -26,6 +26,7 @@ Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/
|
|||
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
|
||||
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
|
||||
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
|
||||
- **Several databases** — Name databases in `lancedb.databases` and search, ask or analyze across them at once, with each result and citation carrying the database it came from
|
||||
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
|
||||
- **CLI & Python API** — Full functionality from command line or code
|
||||
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ The `haiku-rag` CLI provides complete document management functionality.
|
|||
|
||||
- `--config` - Specify custom configuration file
|
||||
- `--read-only` - Open database in read-only mode (blocks writes, skips upgrades)
|
||||
- `--database` - Name of a database from `lancedb.databases` to work on
|
||||
- `--version` / `-v` - Show version and exit
|
||||
|
||||
Per-command options:
|
||||
|
|
@ -19,9 +20,15 @@ The `haiku-rag` CLI provides complete document management functionality.
|
|||
haiku-rag --config /path/to/config.yaml list
|
||||
haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db
|
||||
haiku-rag --read-only search "query"
|
||||
haiku-rag --database medic list
|
||||
haiku-rag add -h
|
||||
```
|
||||
|
||||
With `lancedb.databases` configured, `search`, `ask` and `analyze` cover
|
||||
every database in it. Every other command works on one, named with
|
||||
`--database` or `--db`. See
|
||||
[Several Databases](configuration/storage.md#several-databases).
|
||||
|
||||
## Document Management
|
||||
|
||||
### Add Documents
|
||||
|
|
|
|||
|
|
@ -88,6 +88,7 @@ lancedb:
|
|||
uri: "" # Empty for local, or db://, s3://, az://, gs://
|
||||
api_key: ""
|
||||
region: ""
|
||||
databases: {} # Name-to-location map to search several at once, instead of uri
|
||||
|
||||
embeddings:
|
||||
model:
|
||||
|
|
|
|||
|
|
@ -174,6 +174,72 @@ The recommended layout for production is "different buckets, same account, separ
|
|||
|
||||
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
|
||||
|
||||
## Several Databases
|
||||
|
||||
`lancedb.databases` maps a name to a location, for searching several databases at
|
||||
once:
|
||||
|
||||
```yaml
|
||||
lancedb:
|
||||
databases:
|
||||
medic: s3://my-bucket/medic.lancedb
|
||||
st: s3://my-bucket/st.lancedb
|
||||
local: /data/notes.lancedb
|
||||
```
|
||||
|
||||
A location is a URI or a local path. `databases` and `uri` are mutually
|
||||
exclusive, and setting both fails validation.
|
||||
|
||||
The name is the only identity that leaves the configuration. Results, citations
|
||||
and error messages carry it, so a path or a bucket never reaches a log, a trace
|
||||
or a model.
|
||||
|
||||
Every database in the set is opened with the same embedding configuration, so
|
||||
they have to agree on it. One whose stored settings differ raises
|
||||
`ConfigMismatchError` when it is opened.
|
||||
|
||||
### Searching a set
|
||||
|
||||
`search`, `ask` and `analyze` cover the whole set, or the subset named by
|
||||
`sources`:
|
||||
|
||||
```python
|
||||
results = await client.search("query") # every database
|
||||
results = await client.search("query", sources=["medic"]) # one of them
|
||||
```
|
||||
|
||||
Candidates from each database are fused into one ranked list, by the configured
|
||||
reranker where there is one and by reciprocal rank fusion otherwise. Each result
|
||||
carries `source`, the name of the database it came from, and so does each
|
||||
citation.
|
||||
|
||||
A database that cannot be opened fails the whole query and is named in the error.
|
||||
A result set silently missing one of the databases asked for cannot be told apart
|
||||
from a complete one.
|
||||
|
||||
### Commands that work on one database
|
||||
|
||||
`haiku-rag search`, `ask` and `analyze` cover the configured set. Every other
|
||||
command works on a single database, named with the global `--database` option:
|
||||
|
||||
```bash
|
||||
haiku-rag search "query" # every configured database
|
||||
haiku-rag --database medic list # one of them
|
||||
haiku-rag --database medic migrate
|
||||
```
|
||||
|
||||
`--database` takes a name from `lancedb.databases`, which is how a database
|
||||
behind a URI is reached. `--db` takes a path, and overrides the configured set
|
||||
with that one database. A command that works on one database and is given
|
||||
neither fails rather than choosing for you.
|
||||
|
||||
Each database is created, migrated and vacuumed on its own:
|
||||
|
||||
```bash
|
||||
haiku-rag --database medic init
|
||||
haiku-rag --database st init
|
||||
```
|
||||
|
||||
## Vector Indexing
|
||||
|
||||
Configure vector search settings:
|
||||
|
|
|
|||
|
|
@ -230,6 +230,34 @@ for result in results:
|
|||
print(f"Document Title: {result.document_title}") # when available
|
||||
```
|
||||
|
||||
### Searching Several Databases
|
||||
|
||||
With [`lancedb.databases`](configuration/storage.md#several-databases)
|
||||
configured, a client covers every database in it. `sources` narrows a call to
|
||||
some of them, and each result names the database it came from:
|
||||
|
||||
```python
|
||||
results = await client.search("machine learning") # all of them
|
||||
results = await client.search("machine learning", sources=["medic"]) # one of them
|
||||
|
||||
for result in results:
|
||||
print(f"{result.source}: {result.content}")
|
||||
```
|
||||
|
||||
`ask` and `analyze` take `sources` too, and every citation carries the database
|
||||
it was drawn from:
|
||||
|
||||
```python
|
||||
answer, citations = await client.ask("What changed?", sources=["medic", "st"])
|
||||
for cite in citations:
|
||||
print(f"[{cite.source}] {cite.document_title or cite.document_uri}")
|
||||
|
||||
result = await client.analyze("How many documents mention it?", sources=["medic"])
|
||||
```
|
||||
|
||||
A question scoped to some databases can only cite those, and the analysis
|
||||
sandbox mounts only their documents.
|
||||
|
||||
### Filtering Search Results
|
||||
|
||||
Filter search results to only include chunks from documents matching specific criteria:
|
||||
|
|
|
|||
Loading…
Reference in a new issue