Document searching several databases
lancedb.databases, the sources argument and the --database selector had no documentation. Adds a Several Databases section to the storage configuration page covering the name-to-location map, its mutual exclusion with uri, the shared embedding configuration the set requires, and which commands cover the set against which work on one. Adds a Searching Several Databases section to the Python API page, --database to the CLI's global options, the key to the sample configuration, and one README feature line.
This commit is contained in:
parent
fdb5710491
commit
e15f73a387
5 changed files with 103 additions and 0 deletions
|
|
@ -26,6 +26,7 @@ Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/
|
||||||
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
|
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
|
||||||
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
|
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
|
||||||
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
|
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
|
||||||
|
- **Several databases** — Name databases in `lancedb.databases` and search, ask or analyze across them at once, with each result and citation carrying the database it came from
|
||||||
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
|
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
|
||||||
- **CLI & Python API** — Full functionality from command line or code
|
- **CLI & Python API** — Full functionality from command line or code
|
||||||
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
|
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,7 @@ The `haiku-rag` CLI provides complete document management functionality.
|
||||||
|
|
||||||
- `--config` - Specify custom configuration file
|
- `--config` - Specify custom configuration file
|
||||||
- `--read-only` - Open database in read-only mode (blocks writes, skips upgrades)
|
- `--read-only` - Open database in read-only mode (blocks writes, skips upgrades)
|
||||||
|
- `--database` - Name of a database from `lancedb.databases` to work on
|
||||||
- `--version` / `-v` - Show version and exit
|
- `--version` / `-v` - Show version and exit
|
||||||
|
|
||||||
Per-command options:
|
Per-command options:
|
||||||
|
|
@ -19,9 +20,15 @@ The `haiku-rag` CLI provides complete document management functionality.
|
||||||
haiku-rag --config /path/to/config.yaml list
|
haiku-rag --config /path/to/config.yaml list
|
||||||
haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db
|
haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db
|
||||||
haiku-rag --read-only search "query"
|
haiku-rag --read-only search "query"
|
||||||
|
haiku-rag --database medic list
|
||||||
haiku-rag add -h
|
haiku-rag add -h
|
||||||
```
|
```
|
||||||
|
|
||||||
|
With `lancedb.databases` configured, `search`, `ask` and `analyze` cover
|
||||||
|
every database in it. Every other command works on one, named with
|
||||||
|
`--database` or `--db`. See
|
||||||
|
[Several Databases](configuration/storage.md#several-databases).
|
||||||
|
|
||||||
## Document Management
|
## Document Management
|
||||||
|
|
||||||
### Add Documents
|
### Add Documents
|
||||||
|
|
|
||||||
|
|
@ -88,6 +88,7 @@ lancedb:
|
||||||
uri: "" # Empty for local, or db://, s3://, az://, gs://
|
uri: "" # Empty for local, or db://, s3://, az://, gs://
|
||||||
api_key: ""
|
api_key: ""
|
||||||
region: ""
|
region: ""
|
||||||
|
databases: {} # Name-to-location map to search several at once, instead of uri
|
||||||
|
|
||||||
embeddings:
|
embeddings:
|
||||||
model:
|
model:
|
||||||
|
|
|
||||||
|
|
@ -174,6 +174,72 @@ The recommended layout for production is "different buckets, same account, separ
|
||||||
|
|
||||||
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
|
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
|
||||||
|
|
||||||
|
## Several Databases
|
||||||
|
|
||||||
|
`lancedb.databases` maps a name to a location, for searching several databases at
|
||||||
|
once:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
lancedb:
|
||||||
|
databases:
|
||||||
|
medic: s3://my-bucket/medic.lancedb
|
||||||
|
st: s3://my-bucket/st.lancedb
|
||||||
|
local: /data/notes.lancedb
|
||||||
|
```
|
||||||
|
|
||||||
|
A location is a URI or a local path. `databases` and `uri` are mutually
|
||||||
|
exclusive, and setting both fails validation.
|
||||||
|
|
||||||
|
The name is the only identity that leaves the configuration. Results, citations
|
||||||
|
and error messages carry it, so a path or a bucket never reaches a log, a trace
|
||||||
|
or a model.
|
||||||
|
|
||||||
|
Every database in the set is opened with the same embedding configuration, so
|
||||||
|
they have to agree on it. One whose stored settings differ raises
|
||||||
|
`ConfigMismatchError` when it is opened.
|
||||||
|
|
||||||
|
### Searching a set
|
||||||
|
|
||||||
|
`search`, `ask` and `analyze` cover the whole set, or the subset named by
|
||||||
|
`sources`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
results = await client.search("query") # every database
|
||||||
|
results = await client.search("query", sources=["medic"]) # one of them
|
||||||
|
```
|
||||||
|
|
||||||
|
Candidates from each database are fused into one ranked list, by the configured
|
||||||
|
reranker where there is one and by reciprocal rank fusion otherwise. Each result
|
||||||
|
carries `source`, the name of the database it came from, and so does each
|
||||||
|
citation.
|
||||||
|
|
||||||
|
A database that cannot be opened fails the whole query and is named in the error.
|
||||||
|
A result set silently missing one of the databases asked for cannot be told apart
|
||||||
|
from a complete one.
|
||||||
|
|
||||||
|
### Commands that work on one database
|
||||||
|
|
||||||
|
`haiku-rag search`, `ask` and `analyze` cover the configured set. Every other
|
||||||
|
command works on a single database, named with the global `--database` option:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag search "query" # every configured database
|
||||||
|
haiku-rag --database medic list # one of them
|
||||||
|
haiku-rag --database medic migrate
|
||||||
|
```
|
||||||
|
|
||||||
|
`--database` takes a name from `lancedb.databases`, which is how a database
|
||||||
|
behind a URI is reached. `--db` takes a path, and overrides the configured set
|
||||||
|
with that one database. A command that works on one database and is given
|
||||||
|
neither fails rather than choosing for you.
|
||||||
|
|
||||||
|
Each database is created, migrated and vacuumed on its own:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag --database medic init
|
||||||
|
haiku-rag --database st init
|
||||||
|
```
|
||||||
|
|
||||||
## Vector Indexing
|
## Vector Indexing
|
||||||
|
|
||||||
Configure vector search settings:
|
Configure vector search settings:
|
||||||
|
|
|
||||||
|
|
@ -230,6 +230,34 @@ for result in results:
|
||||||
print(f"Document Title: {result.document_title}") # when available
|
print(f"Document Title: {result.document_title}") # when available
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Searching Several Databases
|
||||||
|
|
||||||
|
With [`lancedb.databases`](configuration/storage.md#several-databases)
|
||||||
|
configured, a client covers every database in it. `sources` narrows a call to
|
||||||
|
some of them, and each result names the database it came from:
|
||||||
|
|
||||||
|
```python
|
||||||
|
results = await client.search("machine learning") # all of them
|
||||||
|
results = await client.search("machine learning", sources=["medic"]) # one of them
|
||||||
|
|
||||||
|
for result in results:
|
||||||
|
print(f"{result.source}: {result.content}")
|
||||||
|
```
|
||||||
|
|
||||||
|
`ask` and `analyze` take `sources` too, and every citation carries the database
|
||||||
|
it was drawn from:
|
||||||
|
|
||||||
|
```python
|
||||||
|
answer, citations = await client.ask("What changed?", sources=["medic", "st"])
|
||||||
|
for cite in citations:
|
||||||
|
print(f"[{cite.source}] {cite.document_title or cite.document_uri}")
|
||||||
|
|
||||||
|
result = await client.analyze("How many documents mention it?", sources=["medic"])
|
||||||
|
```
|
||||||
|
|
||||||
|
A question scoped to some databases can only cite those, and the analysis
|
||||||
|
sandbox mounts only their documents.
|
||||||
|
|
||||||
### Filtering Search Results
|
### Filtering Search Results
|
||||||
|
|
||||||
Filter search results to only include chunks from documents matching specific criteria:
|
Filter search results to only include chunks from documents matching specific criteria:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue