diff --git a/README.md b/README.md index 1cc489b0..7b0d20af 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/ - **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory - **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion - **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI +- **Several databases** — Name databases in `lancedb.databases` and search, ask or analyze across them at once, with each result and citation carrying the database it came from - **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud - **CLI & Python API** — Full functionality from command line or code - **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.) diff --git a/docs/cli.md b/docs/cli.md index 06c2b726..808e1d9c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -7,6 +7,7 @@ The `haiku-rag` CLI provides complete document management functionality. - `--config` - Specify custom configuration file - `--read-only` - Open database in read-only mode (blocks writes, skips upgrades) + - `--database` - Name of a database from `lancedb.databases` to work on - `--version` / `-v` - Show version and exit Per-command options: @@ -19,9 +20,15 @@ The `haiku-rag` CLI provides complete document management functionality. haiku-rag --config /path/to/config.yaml list haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db haiku-rag --read-only search "query" + haiku-rag --database medic list haiku-rag add -h ``` + With `lancedb.databases` configured, `search`, `ask` and `analyze` cover + every database in it. Every other command works on one, named with + `--database` or `--db`. See + [Several Databases](configuration/storage.md#several-databases). + ## Document Management ### Add Documents diff --git a/docs/configuration/index.md b/docs/configuration/index.md index d2434af7..8fd23184 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -88,6 +88,7 @@ lancedb: uri: "" # Empty for local, or db://, s3://, az://, gs:// api_key: "" region: "" + databases: {} # Name-to-location map to search several at once, instead of uri embeddings: model: diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index 91e3193a..dc149e5d 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -174,6 +174,72 @@ The recommended layout for production is "different buckets, same account, separ Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files. +## Several Databases + +`lancedb.databases` maps a name to a location, for searching several databases at +once: + +```yaml +lancedb: + databases: + medic: s3://my-bucket/medic.lancedb + st: s3://my-bucket/st.lancedb + local: /data/notes.lancedb +``` + +A location is a URI or a local path. `databases` and `uri` are mutually +exclusive, and setting both fails validation. + +The name is the only identity that leaves the configuration. Results, citations +and error messages carry it, so a path or a bucket never reaches a log, a trace +or a model. + +Every database in the set is opened with the same embedding configuration, so +they have to agree on it. One whose stored settings differ raises +`ConfigMismatchError` when it is opened. + +### Searching a set + +`search`, `ask` and `analyze` cover the whole set, or the subset named by +`sources`: + +```python +results = await client.search("query") # every database +results = await client.search("query", sources=["medic"]) # one of them +``` + +Candidates from each database are fused into one ranked list, by the configured +reranker where there is one and by reciprocal rank fusion otherwise. Each result +carries `source`, the name of the database it came from, and so does each +citation. + +A database that cannot be opened fails the whole query and is named in the error. +A result set silently missing one of the databases asked for cannot be told apart +from a complete one. + +### Commands that work on one database + +`haiku-rag search`, `ask` and `analyze` cover the configured set. Every other +command works on a single database, named with the global `--database` option: + +```bash +haiku-rag search "query" # every configured database +haiku-rag --database medic list # one of them +haiku-rag --database medic migrate +``` + +`--database` takes a name from `lancedb.databases`, which is how a database +behind a URI is reached. `--db` takes a path, and overrides the configured set +with that one database. A command that works on one database and is given +neither fails rather than choosing for you. + +Each database is created, migrated and vacuumed on its own: + +```bash +haiku-rag --database medic init +haiku-rag --database st init +``` + ## Vector Indexing Configure vector search settings: diff --git a/docs/python.md b/docs/python.md index f7ed4854..fff2eea5 100644 --- a/docs/python.md +++ b/docs/python.md @@ -230,6 +230,34 @@ for result in results: print(f"Document Title: {result.document_title}") # when available ``` +### Searching Several Databases + +With [`lancedb.databases`](configuration/storage.md#several-databases) +configured, a client covers every database in it. `sources` narrows a call to +some of them, and each result names the database it came from: + +```python +results = await client.search("machine learning") # all of them +results = await client.search("machine learning", sources=["medic"]) # one of them + +for result in results: + print(f"{result.source}: {result.content}") +``` + +`ask` and `analyze` take `sources` too, and every citation carries the database +it was drawn from: + +```python +answer, citations = await client.ask("What changed?", sources=["medic", "st"]) +for cite in citations: + print(f"[{cite.source}] {cite.document_title or cite.document_uri}") + +result = await client.analyze("How many documents mention it?", sources=["medic"]) +``` + +A question scoped to some databases can only cite those, and the analysis +sandbox mounts only their documents. + ### Filtering Search Results Filter search results to only include chunks from documents matching specific criteria: