danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Give the docs an architecture page and one extras list

Browse source 
overview.md repeated the landing page: the same install-and-ask block and
five of six identical links. It was positioning prose, where the docs had no
page describing how the system works.

Rewrite it as Architecture, following the data through: source adapter,
converter, chunker, embedder, transaction; then storage and its versioning;
then retrieval, with the 10x rerank fetch and section-bounded expansion; then
the two capabilities; then laptop versus ingester. Retitled in the nav and on
the landing page, filename kept so existing links resolve.

Extras were listed in three places and none was complete.
docs/installation.md now carries a table of all fifteen slim extras, what each
provides, and which the full package already includes.
haiku_rag_slim/README.md names them and links there. The claim that other
providers need their own pydantic-ai extra was wrong: haiku.rag-slim defines
anthropic, google, groq, mistral, bedrock and vertexai itself.

configuration/storage.md opens with the four operational constraints, which
were either buried in an S3 section or undocumented: one writer per URI,
reader lag by read_consistency_interval_seconds, migrate after a
schema-changing upgrade, and the fixed embedding dimension with what
ConfigMismatchError means and which rebuild mode resolves it.

The one-writer rule is stated as a haiku.rag constraint, which is what it is:
the multi-table lock, version snapshot and rollback are process-local, so a
second writer can commit inside another's transaction and be reverted by its
rollback. storage.md and ingester.md both claimed it was a LanceDB property
that corrupts manifests. The S3 deployment section now links to the
constraint instead of restating it.

Get started reads index, Quickstart, Installation, Architecture. The landing
page's list was missing Installation.
This commit is contained in:
Yiorgis Gozadinos 2026-08-20 15:07:06 +03:00
parent d6cfda22b5
commit 483c0ec354
No known key found for this signature in database
7 changed files with 146 additions and 81 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

31
docs/configuration/storage.md
View file

@ -1,5 +1,33 @@
# Database and Storage
## Operational constraints
Four things to know before deploying.
**Run one writer per database.** This is a haiku.rag constraint, not a LanceDB
one. A write that spans several tables is serialized by an in-process lock and
rolled back by restoring each table to the version it had when the write started.
Both are process-local: a second writing process can commit between that snapshot
and the mutation, and a rollback would then revert its work along with ours. Run
a single writer, either the [`haiku-ingester`](../ingester.md) service or your own
application. Read-only consumers are unrestricted.
**Readers lag by an interval.** A connection always sees its own writes. It sees
another process's writes after `lancedb.read_consistency_interval_seconds`
(default 30).
**Migrate after an upgrade that changes the schema.** `haiku-rag migrate` applies
pending migrations in place, and `haiku-rag info` lists what is pending. A
release that needs it says so in the [changelog](../changelog.md).
**The embedding dimension is fixed per database.** Every chunk vector has the
dimension the database was created with. Changing `embeddings.model.vector_dim`
raises `ConfigMismatchError` on open, because stored vectors cannot be compared
against new ones. Changing the provider or model name while keeping the dimension
warns on a read-only open and raises on a writable one. `haiku-rag rebuild
--set-embedder` adopts the new identity without re-embedding, and `haiku-rag
rebuild --embed-only` re-embeds against the new model.
## Local Storage
By default, `haiku.rag` uses a local LanceDB database:
@ -136,7 +164,8 @@ lancedb:
### Deployment Pattern: One Writer, Many Readers
LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state. This is a LanceDB property, not something `haiku.rag` enforces.
The [one-writer constraint](#operational-constraints) shapes the deployment: one
writing process per database URI, any number of read-only consumers.
The recommended layout for production is "different buckets, same account, separate IAM roles per process":

3
docs/index.md
View file

@ -28,7 +28,8 @@ haiku-rag ask "what does it conclude?"
## Start here
- [Quickstart](tutorial.md): install, index, chat.
- [Overview](overview.md): what haiku.rag does, end to end.
- [Installation](installation.md): packages and extras.
- [Architecture](overview.md): how a document becomes a cited answer.
- [Capabilities](capabilities/index.md): native RAG and analysis capabilities for Pydantic AI agents.
- [Python API](python.md): use haiku.rag from code.
- [MCP server](mcp.md): expose haiku.rag to Claude Desktop or other AI assistants.

4
docs/ingester.md
View file

@ -437,7 +437,9 @@ server, then pollers, then in-flight workers.
### Single-writer constraint
LanceDB supports exactly one writer + N readers per database URI. Run
haiku.rag serializes multi-table writes with a process-local lock and rolls
them back by restoring table versions, so a second writing process can
commit inside another's transaction and be reverted by its rollback. Run
exactly one `haiku-ingester serve` against a given LanceDB. Multiple
MCP servers or read-only consumers against the same DB are fine. Sharing
the Postgres queue across processes is safe (the claim/lease lifecycle is

52
docs/installation.md
View file

@ -10,12 +10,8 @@
uv pip install haiku.rag
```
The full package pulls the `docling`, `voyageai`, `cohere`, `zeroentropy`, `cross-encoder`, `jina` and `tui` extras:
- **Document processing** (Docling) - PDF, DOCX, PPTX, images, and 40+ file formats
- **Embedding providers** - VoyageAI and Cohere
- **Rerankers** - local cross-encoders, local Jina, Cohere, Zero Entropy
It does not include the `s3` or `ingester` extras:
The full package pulls the `docling`, `voyageai`, `cohere`, `zeroentropy`,
`cross-encoder`, `jina` and `tui` extras. It does not include `s3` or `ingester`:
```bash
uv pip install 'haiku.rag[ingester]' # the haiku-ingester service
@ -25,27 +21,35 @@ uv pip install 'haiku.rag[s3]' # S3 and object storage
### Slim Package (Minimal Dependencies)
```bash
# Minimal installation (no document processing)
uv pip install haiku.rag-slim
# With document processing
uv pip install haiku.rag-slim[docling]
# With specific providers
uv pip install haiku.rag-slim[docling,voyageai,cross-encoder]
uv pip install 'haiku.rag-slim[docling]'
uv pip install 'haiku.rag-slim[docling,voyageai,cross-encoder]'
```
The slim package has minimal dependencies and lets you install only what you need:
### Extras
- `docling` - PDF, DOCX, PPTX, images, and other document formats
- `voyageai` - VoyageAI embeddings
- `cross-encoder` - Local reranking via sentence-transformers
- `jina` - Local Jina reranking (`provider: jina-local`). Needs transformers and torch, which `cross-encoder` also pulls
- `cohere` - Cohere embeddings and reranking
- `zeroentropy` - Zero Entropy reranking
- `s3` - S3 and object-storage access
- `ingester` - The `haiku-ingester` service (also pulls `s3`)
- `tui` - Terminal UI for `chat` and `inspect` commands
Every extra `haiku.rag-slim` defines. The right-hand column marks the ones the
full `haiku.rag` package already includes.
| Extra | Provides | In `haiku.rag` |
|---|---|---|
| `docling` | PDF, DOCX, PPTX, images and 40+ formats, converted locally | yes |
| `tui` | Terminal UI for `chat` and `inspect` | yes |
| `voyageai` | VoyageAI embeddings | yes |
| `cohere` | Cohere embeddings and reranking | yes |
| `zeroentropy` | Zero Entropy reranking | yes |
| `cross-encoder` | Local reranking via sentence-transformers | yes |
| `jina` | Local Jina reranking (`provider: jina-local`) | yes |
| `s3` | S3 and object-storage access | no |
| `ingester` | The `haiku-ingester` service (also pulls `s3`) | no |
| `anthropic` | Anthropic Claude models | no |
| `google` | Google Gemini models | no |
| `groq` | Groq models | no |
| `mistral` | Mistral models | no |
| `bedrock` | AWS Bedrock models | no |
| `vertexai` | Google Vertex AI models | no |
Ollama and any OpenAI-compatible endpoint work with no extra at all.
**Built-in providers** (no extras needed):
- **Ollama** (default embedding provider)
@ -53,7 +57,7 @@ The slim package has minimal dependencies and lets you install only what you nee
- **vLLM** and other OpenAI-compatible endpoints (embeddings, QA, reranking)
- **Jina** reranking via `provider: jina`, which calls the Jina HTTP API
Other Pydantic AI providers need their own Pydantic AI extra. For Claude models, install `pydantic-ai-slim[anthropic]`.
Other providers come from the extras above, which pull the matching Pydantic AI extra. For Claude models, `uv pip install 'haiku.rag-slim[anthropic]'`.
See [Configuration](configuration/index.md) for configuring providers including advanced options like vLLM.

106
docs/overview.md
View file

@ -1,47 +1,89 @@
# Overview
# Architecture
haiku.rag is an agentic RAG that runs locally and scales to production. Index PDFs, web pages, or whole directories. Ask questions and get cited answers. Build agents, capabilities, and MCP integrations on top.
haiku.rag ingests documents, retrieves from them with hybrid search, and answers
with citations. This page follows the data through the system. For a working
setup, start with the [Quickstart](tutorial.md).
haiku.rag is open-source first. The defaults run open models through [Ollama](https://ollama.com/) so the full pipeline works without external API keys. Any provider Pydantic AI supports works in its place.
## Ingestion
Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/). Embedded database, no servers required.
## See it work
```bash
uv pip install haiku.rag
ollama pull qwen3-embedding:4b
ollama pull gpt-oss
haiku-rag init
haiku-rag add-src ~/Documents/some-paper.pdf
haiku-rag chat
```text
source adapter -> converter -> chunker -> embedder -> LanceDB
```
The chat TUI is one way to interact with the database. `haiku-rag ask` and `haiku-rag search` cover one-shot CLI usage. Python integrations, capabilities, and the MCP server work against the same database.
A **source adapter** owns the I/O and the identity of a document: it fetches
bytes, reports the backend's revision (mtime for a file, ETag for S3 or HTTP),
and computes the content hash. The same adapters serve one-shot ingestion
(`haiku-rag add-src`, `HaikuRAG.create_document_from_source`) and the continuous
[`haiku-ingester`](ingester.md) service, so both agree on what a document is and
when it has changed.
## What it does
The **converter** turns those bytes into a `DoclingDocument`, the structured form
that carries headings, tables, pictures and page provenance. It runs in-process
with the `docling` extra, or against a [docling-serve](remote-processing.md)
fleet.
**Ingest.** PDFs, DOCX, HTML, images, and 40+ formats via Docling. Add files, URLs, or whole directories with `haiku-rag add-src`, or run the [`haiku-ingester`](ingester.md) service for continuous, queue-backed ingestion from filesystem, HTTP, S3, or WebDAV sources.
The **chunker** splits that structure into chunks, each keeping the headings it
sits under, the page numbers it came from, and references to the document items
it covers. With a multimodal embedder, pictures become chunks of their own.
**Search.** Hybrid retrieval (vector + full-text with reciprocal rank fusion), optional cross-encoder reranking, structure-aware context expansion. Image-as-query and cross-modal retrieval when configured with a multimodal embedder.
The **embedder** vectorizes them in batches. The document, its mutable metadata,
its chunks and its structural items are written under one process-local
transaction: it takes a version snapshot, and on failure restores each table to
it. A rollback that cannot complete raises rather than reporting success, and the
snapshot is only meaningful while this process is the only writer.
**Answer.** RAG capability with citations including page numbers, section headings, and visual grounding. Vision-capable models receive figure bytes alongside chunk text. Analysis capability with a sandboxed Python interpreter for aggregation and computation across documents. Optional capabilities compact a long conversation down to the evidence it cited, and require every answer to declare its grounding.
## Storage
**Integrate.** Use it from Python, the CLI, the [MCP server](mcp.md), or through composable native Pydantic AI [capabilities](capabilities/index.md).
LanceDB is embedded, so there is no server. The same code runs against a local
directory, S3, GCS, Azure or LanceDB Cloud by changing `lancedb.uri`.
**Operate.** Embedded LanceDB by default. Also runs on S3, GCS, Azure, or LanceDB Cloud. Time-travel queries via LanceDB versioning. The [`haiku-ingester`](ingester.md) service runs continuously for production deployments.
Tables are versioned. Vacuum collapses old versions on a retention window, and
[tags](cli.md) name a state across all tables so a database can be restored to
it later.
## Where to go next
One process writes at a time. Reads are unrestricted, and a reader sees another
process's writes after `lancedb.read_consistency_interval_seconds`.
- [Quickstart](tutorial.md): install, index, chat.
- [Capabilities](capabilities/index.md): native RAG and analysis capabilities for Pydantic AI agents.
- [Python API](python.md): use haiku.rag from code.
- [MCP server](mcp.md): expose haiku.rag to Claude Desktop or other AI assistants.
- [Tuning](tuning.md): improve retrieval quality.
- [Configuration](configuration/index.md): every setting.
## Retrieval
## License
```text
query -> vector + full-text search -> fusion -> rerank -> context expansion
```
MIT. Source on [GitHub](https://github.com/ggozad/haiku.rag).
Search runs a vector query and a full-text query and fuses the rankings. With a
reranker configured, it retrieves ten times the requested limit and reranks down
to it, so quality improves without changing the caller's limit.
Results then expand: a chunk is returned with the section it belongs to, bounded
by `search.max_context_chars`. Sections that fit come back whole, larger ones
grow outward from the match, and small ones grow across boundaries. Every result
carries its page numbers and headings, which is what makes a citation checkable.
## Answering
Two [capabilities](capabilities/index.md) sit on top, both native Pydantic AI
capabilities you can attach to your own agent:
- The **RAG capability** searches and cites. Its citations carry page numbers and
headings, and `haiku-rag visualize` draws the cited chunk on the page image.
- The **analysis capability** adds a sandboxed Python interpreter with the
documents mounted as a filesystem, for questions that need computation across
documents rather than retrieval.
Two optional capabilities compose with them: evidence compaction replaces older
turns' evidence with what was actually cited, and citation policy requires every
answer to declare what grounds it.
The same database is reachable from [Python](python.md), the [CLI](cli.md), and
the [MCP server](mcp.md).
## Running it
A laptop needs nothing but the package and Ollama. Production adds the
[`haiku-ingester`](ingester.md) service, which polls its sources, queues work in
SQLite or Postgres, and retries with a circuit breaker per source.
Before deploying, read the operational constraints in
[Storage](configuration/storage.md): one writer per database, `haiku-rag migrate`
after an upgrade that changes the schema, and a fixed embedding dimension per
database.

29
haiku_rag_slim/README.md
View file

@ -28,31 +28,18 @@ Adds support for 40+ file formats including PDF, DOCX, HTML, and more.
### Available Extras
**Document Processing:**
- `docling` - PDF, DOCX, HTML, and 40+ file formats
**Embedding Providers:**
- `voyageai` - VoyageAI embeddings
**Rerankers:**
- `cross-encoder` - Local reranking via sentence-transformers
- `cohere` - Cohere
- `zeroentropy` - Zero Entropy
**Model Providers:**
- OpenAI/Ollama - included in core (OpenAI-compatible APIs)
- `anthropic` - Anthropic Claude
- `groq` - Groq
- `google` - Google Gemini
- `mistral` - Mistral AI
- `bedrock` - AWS Bedrock
- `vertexai` - Google Vertex AI
`docling`, `tui`, `voyageai`, `cohere`, `zeroentropy`, `cross-encoder`, `jina`,
`s3`, `ingester`, and one per model provider: `anthropic`, `google`, `groq`,
`mistral`, `bedrock`, `vertexai`. Ollama and any OpenAI-compatible endpoint need
no extra.
What each provides, and which ones the full `haiku.rag` package already
includes: [Installation](https://ggozad.github.io/haiku.rag/installation/).
```bash
# Common combinations
uv pip install haiku.rag-slim[docling,anthropic,cross-encoder]
uv pip install haiku.rag-slim[docling,groq]
uv pip install 'haiku.rag-slim[docling,anthropic,cross-encoder]'
uv pip install 'haiku.rag-slim[docling,groq]'
```
## Usage

2
zensical.toml
View file

@ -11,9 +11,9 @@ extra_css = ["stylesheets/extra.css"]
nav = [
{ "Get started" = [
"index.md",
{ Overview = "overview.md" },
{ Quickstart = "tutorial.md" },
{ Installation = "installation.md" },
{ Architecture = "overview.md" },
] },
{ "Use it" = [
{ CLI = "cli.md" },