Say which commands cover a set, and what a shared id does
Three groups, not two: `search`, `ask`, `analyze` and `chat` cover the set, `settings`, `init-config` and `download-models` open no database, and every other command works on one — or on a configured set of one, which is unambiguous and keeps its name. A database named in `lancedb.databases` keeps that name whether or not it is the only one covered; only `lancedb.uri` places one without naming it. Document ids repeat between copies of a database, where the sandbox refuses a duplicate but the chat filter's `id IN (...)` matches the document in every copy. `build_document_id_filter` claimed ids never widen a selection. Document the facade: `covers_multiple`, `source_names`, `source`, `reader_for`, `clients_for`, the lifetime of a borrowed client, and `sources=None` against `sources=[]`. Drop the vision callout from the README, which the features list already covers.
This commit is contained in:
parent
2b2de5a61f
commit
2000098e16
7 changed files with 57 additions and 17 deletions
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
### Added
|
||||
|
||||
- `lancedb.databases`: a name-to-location mapping for searching several databases at once, mutually exclusive with `lancedb.uri`. `client.search(..., sources=[...])` selects which to search, `sources=None` searches all of them, and `SearchResult.source` carries the configured name a result came from. `Document.source` names it on a document from a listing or a lookup, so a listing that spans databases says which one each came from. Candidates are fused by the configured reranker over the union, or by reciprocal rank fusion when none is configured. Databases searched together must have been written with the same embedder; two that disagree raise `ConfigMismatchError`. The query is embedded once for the whole selection. `SearchResult.format_for_agent` names the database, so the model can attribute evidence to one while it answers. `haiku-rag search`, `ask`, `analyze` and `chat` cover the configured set and label each result and citation with its database; every other command works on one, named with `--database NAME` or `--db PATH`.
|
||||
- `lancedb.databases`: a name-to-location mapping for searching several databases at once, mutually exclusive with `lancedb.uri`. `client.search(..., sources=[...])` selects which to search, `sources=None` searches all of them, and `SearchResult.source` carries the configured name a result came from. `Document.source` names it on a document from a listing or a lookup, so a listing that spans databases says which one each came from. Candidates are fused by the configured reranker over the union, or by reciprocal rank fusion when none is configured. Databases searched together must have been written with the same embedder; two that disagree raise `ConfigMismatchError`. The query is embedded once for the whole selection. `SearchResult.format_for_agent` names the database, so the model can attribute evidence to one while it answers. `haiku-rag search`, `ask`, `analyze` and `chat` cover the configured set and label each result and citation with its database. `settings`, `init-config` and `download-models` open no database; every other command works on one, named with `--database NAME` or `--db PATH`, or resolved from a configured set of one.
|
||||
- `client.ask(..., sources=[...])` asks across the selected databases, and `Citation.source` names the one a cited chunk came from. The cite fallback for an id absent from the run's results looks only in the selected databases, so a question scoped to some cannot cite another. A chunk id held by two of the databases searched raises `AmbiguousCitationError`, which reaches the model as a retry; the fallback refuses it too, rather than answering from the first database that holds it.
|
||||
- `client.analyze(..., sources=[...])` analyzes across the selected databases: the sandbox mounts their documents under one flat `/documents/{id}/` namespace, resolving each id to the database holding it, and in-code `search()` covers the same selection.
|
||||
|
||||
|
|
|
|||
|
|
@ -11,8 +11,6 @@ Agentic RAG that answers questions about your own documents with citations to pa
|
|||
|
||||
Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/). Full documentation at [ggozad.github.io/haiku.rag](https://ggozad.github.io/haiku.rag/).
|
||||
|
||||
> **New: vision and multimodal search.** Picture-aware ingestion captures embedded figure bytes; vision-capable QA models receive them alongside text. Multimodal embedders put picture vectors in the same space as text, enabling text-as-query → figure hits and image-as-query retrieval.
|
||||
|
||||
## Features
|
||||
|
||||
- **Hybrid search** — Vector + full-text with Reciprocal Rank Fusion
|
||||
|
|
@ -26,7 +24,7 @@ Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/
|
|||
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
|
||||
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
|
||||
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
|
||||
- **Several databases** — Name databases in `lancedb.databases` and search, ask or analyze across them at once, with each result and citation carrying the database it came from
|
||||
- **Several databases** — Name databases in `lancedb.databases` and search, ask, analyze or chat across them at once, with each result and citation carrying the database it came from
|
||||
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
|
||||
- **CLI & Python API** — Full functionality from command line or code
|
||||
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
|
||||
|
|
|
|||
|
|
@ -25,7 +25,8 @@ The `haiku-rag` CLI provides complete document management functionality.
|
|||
```
|
||||
|
||||
With `lancedb.databases` configured, `search`, `ask`, `analyze` and `chat`
|
||||
cover every database in it. Every other command works on one, named with
|
||||
cover every database in it. `settings`, `init-config` and `download-models`
|
||||
open no database at all. Every other command works on one, named with
|
||||
`--database` or `--db`. See
|
||||
[Several Databases](configuration/storage.md#several-databases).
|
||||
|
||||
|
|
|
|||
|
|
@ -243,9 +243,11 @@ Candidates from each database are fused into one ranked list, by the configured
|
|||
reranker where there is one and by reciprocal rank fusion otherwise. Each result
|
||||
carries `source`, the name of the database it came from, and so does each
|
||||
citation. A document from `list_documents`, `get_document_by_id`,
|
||||
`get_document_by_uri` or `resolve_document` carries it too. Naming one database
|
||||
on the command line points the configuration at it, so commands that work on one
|
||||
database report no name.
|
||||
`get_document_by_uri` or `resolve_document` carries it too, and a database
|
||||
named in `lancedb.databases` keeps that name even when it is the only one a
|
||||
client covers. Only a database placed by `lancedb.uri`, which names none, has no
|
||||
name to carry. The CLI does not print the name of a single database it was told
|
||||
to use, since the caller just named it.
|
||||
|
||||
Chunk ids are unique within a database and say nothing across them, so a database
|
||||
copied from another holds the same ids. Results are told apart by the database
|
||||
|
|
@ -253,6 +255,14 @@ and the id together. A chunk id held by two of the databases searched cannot be
|
|||
cited: `resolve_citations` raises `AmbiguousCitationError`, and the capability
|
||||
asks the model for other evidence instead.
|
||||
|
||||
Document ids behave the same way, and two places treat a collision differently.
|
||||
The analysis sandbox mounts one document per id and refuses a set where two
|
||||
databases claim one, since the mount has a single path per id. The chat document
|
||||
filter does not: it selects by document id and applies `id IN (...)` to every
|
||||
covered database, so selecting an id that exists in copies of a database matches
|
||||
the document in each of them. Independently built databases use UUID document
|
||||
ids and do not collide.
|
||||
|
||||
**Configure a reranker when searching several databases.** Reciprocal rank fusion
|
||||
compares ranks, not scores, so every database contributes its own best matches
|
||||
whether or not they are relevant to the question, and results from databases
|
||||
|
|
@ -276,11 +286,18 @@ A database that cannot be opened fails the whole query and is named in the error
|
|||
A result set silently missing one of the databases asked for cannot be told apart
|
||||
from a complete one.
|
||||
|
||||
### Commands that work on one database
|
||||
### How commands treat the set
|
||||
|
||||
`haiku-rag search`, `ask`, `analyze` and `chat` cover the configured set. Every
|
||||
other command works on a single database, named with the global `--database`
|
||||
option:
|
||||
Commands fall into three groups:
|
||||
|
||||
- **Set-capable**: `search`, `ask`, `analyze` and `chat` cover the whole
|
||||
configured set, or the subset named by `--database`.
|
||||
- **Config-only**: `settings`, `init-config` and `download-models` open no
|
||||
database, so the set is irrelevant to them.
|
||||
- **Single-database**: everything else — document writes, `rebuild`, `vacuum`,
|
||||
`migrate`, `init`, `info`, `history`, `tag`, `doctor`, `list`, `inspect`,
|
||||
`visualize` and `mcp` — works on one database, named with the global
|
||||
`--database` option.
|
||||
|
||||
```bash
|
||||
haiku-rag search "query" # every configured database
|
||||
|
|
@ -289,9 +306,10 @@ haiku-rag --database medic migrate
|
|||
```
|
||||
|
||||
`--database` takes a name from `lancedb.databases`, which is how a database
|
||||
behind a URI is reached. `--db` takes a path, and overrides the configured set
|
||||
with that one database. A command that works on one database and is given
|
||||
neither fails rather than choosing for you.
|
||||
behind a URI is reached. `--db` takes a path, and overrides the configured
|
||||
location with that one database. A single-database command given neither fails
|
||||
rather than choosing for you, unless `lancedb.databases` names exactly one: a
|
||||
set of one is unambiguous and is used, keeping its configured name.
|
||||
|
||||
Each database is created, migrated and vacuumed on its own:
|
||||
|
||||
|
|
|
|||
|
|
@ -258,6 +258,28 @@ result = await client.analyze("How many documents mention it?", sources=["medic"
|
|||
A question scoped to some databases can only cite those, and the analysis
|
||||
sandbox mounts only their documents.
|
||||
|
||||
`sources=None` covers every database the client covers; `sources=[]` covers none
|
||||
and returns nothing, which is not the same thing.
|
||||
|
||||
#### Asking a client what it covers
|
||||
|
||||
```python
|
||||
client.covers_multiple # True while reading more than one database
|
||||
client.source_names # the configured names covered, in configured order
|
||||
client.source # the one name, or None while covering a set
|
||||
|
||||
owner = await client.reader_for("medic") # the client reading that database
|
||||
medic, st = await client.clients_for(["medic", "st"])
|
||||
```
|
||||
|
||||
`clients_for` opens the databases it names, on first use rather than at entry,
|
||||
and returns a client for each. Those clients borrow their databases from the
|
||||
covering one: they are valid only while it is open, and closing or entering one
|
||||
leaves its database alone. The covering client closes them all on teardown.
|
||||
|
||||
A borrowed client reads one database, so it has the `store` and repositories a
|
||||
covering client cannot have, and it is writable when the covering client is.
|
||||
|
||||
### Filtering Search Results
|
||||
|
||||
Filter search results to only include chunks from documents matching specific criteria:
|
||||
|
|
|
|||
|
|
@ -121,7 +121,7 @@ class LanceDBConfig(ConfigModel):
|
|||
if self.uri and self.databases:
|
||||
raise ValueError(
|
||||
"lancedb.uri and lancedb.databases are mutually exclusive: "
|
||||
"use uri for one database, databases for several"
|
||||
"use uri for one unnamed location, or databases for named ones"
|
||||
)
|
||||
for name, location in self.databases.items():
|
||||
# A blank name is falsy, so source routing reads it as absent; a
|
||||
|
|
|
|||
|
|
@ -20,7 +20,8 @@ def build_document_id_filter(document_ids: list[str]) -> str | None:
|
|||
|
||||
Unlike name matching, an id identifies one document: names repeat within a
|
||||
corpus and across databases, so a name filter can widen to documents the
|
||||
caller did not pick.
|
||||
caller did not pick. Ids repeat only between copies of a database, where the
|
||||
same id names the same document in each.
|
||||
"""
|
||||
if not document_ids:
|
||||
return None
|
||||
|
|
|
|||
Loading…
Reference in a new issue