danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Docs & cl

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-05-08 11:04:32 +03:00
parent 6af463699d
commit ab5cfdd04a
No known key found for this signature in database
4 changed files with 63 additions and 140 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

36
CHANGELOG.md
View file

@ -3,30 +3,24 @@
### Added
- **`rebuild --descriptions` mode.** Adds VLM picture descriptions to an existing database without re-converting from source. Loads each document's stored docling blob, identifies pictures lacking `meta.description.text`, drives the configured VLM (via pydantic-ai `BinaryContent`) over the picture bytes already stored in `document_items.picture_data`, patches descriptions into the blob, then re-chunks + re-embeds so chunk text reflects them. Skips the docling parse entirely; only the VLM time is paid. Idempotent — pictures that already carry a description are not re-described, so the operation is safe to re-run after a partial failure. Errors clearly when `picture_description.enabled` is false. Exposed as `haiku-rag rebuild --descriptions` and `RebuildMode.DESCRIPTIONS`.
- **Silent-failure guard for picture descriptions.** When `picture_description.enabled=True` and a converted document has at least one picture but zero of them came back with a description, `client.processing.convert()` now logs a clear warning naming the source path, picture count, configured VLM model, and base URL. docling-serve swallows VLM errors (network failures, missing models, unreachable hosts) and returns a "successful" conversion with empty descriptions; this guard surfaces the failure before a long ingest produces a corpus with no descriptions in it.
- **MCP image-query tool + CLI `--image PATH`.** New MCP tool `search_documents_by_image(image_base64, limit, include_images)` routes a base64-encoded image through `client.search()`. Registered only when the configured embedder supports images, so non-multimodal MCP servers don't expose a tool that would always fail. The `haiku-rag search` CLI gains an `--image PATH` flag that reads the file and runs the same image-as-query path.
- **Image-as-query search.** `client.search()` now accepts `str | bytes | PIL.Image.Image`. Bytes/PIL queries embed via the multimodal embedder's `embed_image_query` and dispatch to vector-only chunk search (FTS doesn't apply to non-text queries; reranking is also skipped). Raises a clear error if the configured embedder is text-only. `ChunkRepository.search()` gains an optional `query_vector` parameter that bypasses `embed_query` and forces the vector-only path.
- **`vision: bool` flag on `ModelConfig`.** Tracks whether a configured language model can interpret images. Default `False`. The agent's `search` tool only attaches picture bytes (as `BinaryContent`) to the `ToolReturn` when `qa.model.vision = True`. Without the gate, sending image content to a text-only model behaves inconsistently across providers — Ollama silently accepts and the model hallucinates a confident wrong answer; OpenAI returns 400; others vary. Capability detection from a probe or a model-name whitelist is unreliable, so `vision` is an explicit user-set capability declaration. Set it to `True` for vision-capable QA models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, ...).
- **Synthetic picture chunks at ingest under multimodal embedders.** `build_picture_chunks` (in `client/processing.py`) walks a `DoclingDocument`'s `pictures` and emits one synthetic `Chunk` per `PictureItem` with available bytes. Bytes ride on a `Chunk._picture_data` PrivateAttr (not serialized) so `embed_chunks` can route them through `embed_images` while text chunks keep going through `embed_documents`. Wired into the three ingest paths (`create_document`, `_create_document_from_file`, `_create_or_update_document_from_url`, `_update_document_with_chunks`, and `_rebuild_rechunk`) — guarded by `embedder.supports_images` so text-only configurations are unaffected. Snapshot/merge with `existing_picture_data` keeps picture chunks alive across rebuild round-trips. Picture chunks land in the same `chunks` table with the same vector dim as text chunks, so cross-modal search reuses the existing hybrid+RRF pipeline.
- **Multimodal embedder support (`provider="vllm"`).** `EmbedderWrapper` gains `supports_images: bool` and `embed_image_query`. The `vllm` provider talks HTTP to a vLLM server's OpenAI-compatible `/v1/embeddings` endpoint — text inputs use the standard `input` field for true server-side batching; image inputs use vLLM's `messages` superset with `image_url` content parts carrying base64 data URIs. Works with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No Python ML deps added — uses `httpx`. Text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) report `supports_images=False` and raise a clear error if image methods are called.
- **Picture bytes are always stored.** Every ingest captures embedded picture bytes into `document_items.picture_data`; rebuilds and updates always preserve them. The single ingest-time decision is whether to run a VLM at ingest, exposed via `processing.conversion_options.picture_description.enabled: bool` (default `False`). When enabled, descriptions are woven into chunk text; when disabled, chunks contain only their natural text. No reingest is required to flip the VLM on or off — `haiku-rag rebuild --rechunk` recomputes chunk text against the new setting. The earlier `processing.conversion_options.generate_picture_images` flag is dropped from loaded YAMLs and a warning prints once telling the user to remove it from their config.
- **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`.
- New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`.
- Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory.
- **Embedded picture bytes captured at ingestion.** `extract_items` decodes `PictureItem.image.uri` data URIs into raw bytes on `document_items.picture_data`, and surfaces VLM picture descriptions (`meta.description.text`) into `DocumentItem.text`.
- `compress_docling_split` strips picture URIs from the stored docling blob so they live in one place.
- The 0.45.0 migration adds the column on existing DBs and backfills it from the legacy blob.
- Rebuild and update snapshot existing bytes via `get_all_picture_data` so re-chunk preserves them.
- **Picture image bytes in search results and vision-capable QA.** `SearchResult.image_data: dict[str, str] | None` carries base64 picture bytes keyed by `self_ref`.
- `client.search()` and the MCP `search_documents` tool gain `include_images: bool = True`.
- `expand_context` preserves picture self_refs with empty text and re-attaches bytes to the rebuilt `SearchResult`s.
- The agent's search tool returns `pydantic_ai.messages.ToolReturn(return_value=text, content=[BinaryContent(...)])` when picture data is present, otherwise a plain string.
- **Vision capabilities.** Picture-aware ingestion, vision QA, multimodal embeddings, and image-as-query search.
- **Picture bytes always stored** at ingest in a new `document_items.picture_data` column (`large_binary`), addressable by `(document_id, self_ref)`. Bulk read paths project metadata-only so bytes never leak into context expansion or analysis-sandbox builds. The 0.45.0 migration adds the column on existing DBs and backfills it from each doc's docling blob; URIs are then stripped from the blob so bytes live in one place.
- **VLM picture descriptions** at ingest via `processing.conversion_options.picture_description.enabled` (default `false`). When enabled, descriptions are woven into chunk text. The earlier `generate_picture_images` flag is dropped with a one-time warning. `haiku-rag rebuild --descriptions` runs the VLM over stored bytes after the fact, idempotently — skipping the docling parse entirely.
- **Multimodal embedder (`provider="vllm"`)** for cross-modal retrieval. Talks HTTP to a vLLM `/v1/embeddings` endpoint (`input` array for text, `messages` superset with `image_url` for images). Tested with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No new Python ML dependencies. Under multimodal embedders, ingest emits one synthetic picture chunk per `PictureItem`, sharing the chunks table with text.
- **Image-as-query search.** `client.search()` accepts `str | bytes | PIL.Image.Image`. Image queries embed once and run vector-only against the chunks table. New CLI flag `haiku-rag search --image PATH` and new MCP tool `search_documents_by_image(image_base64, ...)` (registered only when the embedder supports images).
- **Vision QA via `qa.model.vision: bool` flag** on `ModelConfig` (default `false`). When `true`, the agent's `search` tool attaches picture bytes as `BinaryContent` parts on its `ToolReturn`. Default is `false` because providers behave inconsistently when an image is sent to a text-only model (Ollama silently accepts and confabulates; OpenAI returns 400). `SearchResult.image_data: dict[str, str] | None` carries base64 picture bytes keyed by `self_ref`; `client.search()` and MCP `search_documents` gain `include_images: bool = True`.
- **Silent-failure guard for picture descriptions.** When `picture_description.enabled=true` and a conversion returns at least one picture but zero descriptions, log a warning naming the source, picture count, VLM model, and base URL. Surfaces docling-serve's swallowed VLM errors (unreachable host, missing model) before they pollute a long ingest.
- **Inspector renders attached pictures** under `qa.model.vision=true` in the context modal (`c` key) so the inspector reflects what the LLM actually receives.
### Fixed
- **docling-serve picture-image extraction.** docling-serve only emits picture bytes under `image_export_mode="referenced"` (upstream [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576)). The converter now switches to `referenced` + `target_type="zip"` when picture images are requested, parses the returned zip, and rehydrates `artifacts/<filename>` URIs back into `data:<mime>;base64,...` URIs. The previously-`xfail`ed `test_convert_pdf_with_picture_images` test now passes.
- **`rebuild --rechunk` now reuses the stored docling blob** instead of re-converting from the markdown export, which dropped every `PictureItem` (and its bytes) on the floor. Documents without a stored docling blob now raise instead of silently falling back to markdown.
- **`rebuild --descriptions` no longer destroys `docling_pages`.** The previous implementation called `set_docling()` after a structure-only docling load, which writes `docling_pages=None` and clobbered page rasters for every doc with at least one undescribed picture (silently breaking `visualize_chunk` for the affected docs).
- **docling-serve picture-image extraction.** docling-serve only emits picture bytes under `image_export_mode="referenced"` (upstream [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576)). The converter switches to `referenced` + `target_type="zip"` when picture images are requested and rehydrates `artifacts/<filename>` URIs back into `data:` URIs.
- **`rebuild --rechunk` reuses the stored docling blob** instead of re-converting from the markdown export, which dropped every `PictureItem` on the floor. Documents without a stored docling blob now raise instead of silently falling back.
### Changed
- **Lazy document hydration during rebuild.** Each mode loop now fetches one full record at a time instead of eagerly loading all docs with their multi-MB blobs. Drops startup memory from ~15 GB to ~one document on a 1000-doc database.
## [0.44.0] - 2026-04-29

144
docs/configuration/processing.md
View file

@ -106,159 +106,67 @@ conversion_options:
#### Picture Handling
Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest time to generate textual descriptions, set via `picture_description.enabled`:
Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest to generate textual descriptions:
```yaml
processing:
conversion_options:
picture_description:
enabled: true # default false; runs the VLM at ingest
enabled: true # default false
model:
provider: ollama # ollama, openai, or custom
name: ministral-3 # VLM model name
temperature: 0.0
timeout: 90 # Request timeout in seconds
max_tokens: 200 # Maximum tokens in response
provider: ollama # any OpenAI-compatible /v1/chat/completions provider
name: ministral-3
timeout: 90
max_tokens: 200
```
When `enabled: false` (default), the VLM doesn't run; chunks contain only their natural text (captions, surrounding paragraphs). When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS.
When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS. The prompt is configurable under `prompts.picture_description` — see [Prompts](prompts.md).
**Switching the VLM on or off on an existing database.** Picture bytes are already stored, so no reingest is required.
**Switching the VLM on or off on an existing database** doesn't require reingesting (the bytes are already there):
- To turn the VLM **off** (descriptions already exist, you want to drop them): flip `enabled: false` and run `haiku-rag rebuild --rechunk`. Chunk text recomposes from the stripped docling blob.
- To turn the VLM **on** (descriptions don't exist yet, you want them now): flip `enabled: true` and run `haiku-rag rebuild --descriptions`. The VLM is driven over the picture bytes already in `document_items.picture_data`, descriptions are patched into the docling blob, and chunks are recomposed. The docling parse is skipped entirely. See [Rebuild Database](../cli.md#rebuild-database) for full details.
- Off → on: `haiku-rag rebuild --descriptions` runs the VLM over stored bytes and re-chunks. Skips the docling parse entirely.
- On → off: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions.
#### Picture descriptions × embedder × QA model: how the pieces compose
When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag — see [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve).
Three settings drive what gets stored, what gets retrieved, and what reaches the QA model:
#### Pictures × embedder × QA model: how the pieces compose
Three independent settings drive ingest, retrieval, and QA:
| Setting | Question it answers | Values |
|---|---|---|
| `picture_description.enabled` | Should a VLM weave descriptions into chunk text at ingest? | `false` (default) / `true` |
| `embeddings.model.provider` | Can the embedder index image content? | text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
| `embeddings.model.provider` | Can the embedder index image content? | text-only (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
| `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` |
Picture bytes are always stored, regardless of these settings.
**What gets stored** for each `picture_description.enabled` × embedder combination:
**What gets stored** by `enabled` × embedder:
| `enabled` | Embedder | Text chunks contain… | Synthetic picture chunks |
| `enabled` | Embedder | Text chunks | Synthetic picture chunks |
|---|---|---|---|
| `false` | text-only | text only (caption/surrounding) | none |
| `false` | multimodal | text only | one per picture, content = caption/empty, vector = image embedding |
| `true` | text-only | text + VLM descriptions | none |
| `true` | multimodal | text + VLM descriptions | one per picture, content = description, vector = image embedding |
| `false` | multimodal | text only | one per picture, vector = image embedding |
| `true` | text-only | text + descriptions | none |
| `true` | multimodal | text + descriptions | one per picture, vector = image embedding |
**What QA receives** at search time, given stored state and `qa.model.vision`:
**What QA receives** at search time:
| `qa.model.vision` | QA receives |
|---|---|
| `false` | text chunks only (descriptions in chunk text answer figure questions in prose when `picture_description.enabled` was true) |
| `true` | text chunks + raw picture bytes; vision model reads the figures directly |
- `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose).
- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`; the model reads figures directly.
A few invariants worth knowing:
`qa.model.vision` is independent of ingestion — flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI; default `false` is the safe choice.
- **`qa.model.vision` is independent of ingestion.** It only controls whether the agent's `search` tool attaches picture bytes to its `ToolReturn`. A text-only QA model with `vision: true` won't suddenly understand images — it will silently accept the bytes and confabulate. Default `false` is the safe choice.
- **The bytes are always there**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself.
- **Cross-modal search** (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks.
**Recommended combinations** by use case:
**Recommended combinations:**
| Use case | `picture_description.enabled` | Embedder | `qa.model.vision` |
|---|---|---|---|
| Pure text RAG, no figures | `false` | text-only | `false` |
| Text RAG, figures answered through descriptions | `true` | text-only | `false` |
| Vision QA on figure-rich docs (no cross-modal search) | `true` or `false` | text-only | `true` |
| Cross-modal search + vision QA (the full multimodal stack) | `true` or `false` | multimodal | `true` |
| Cross-modal search + vision QA | `true` or `false` | multimodal | `true` |
| Cross-modal search, text QA only | `true` | multimodal | `false` |
**`picture_description.model` configuration** (used only when `picture_description.enabled: true`):
- **model**: Standard model configuration
- `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
- `name`: Model name (e.g., `ministral-3`, `granite3.2-vision`, `gpt-4-vision`)
- `base_url`: Optional custom API endpoint for vLLM, LM Studio, etc.
- **timeout**: Request timeout in seconds
- **max_tokens**: Maximum tokens in the VLM response
**Note:** Requires an OpenAI-compatible `/v1/chat/completions` endpoint. Providers with different API formats (e.g., Anthropic Claude) are not supported.
**Default prompt** (configured in `prompts.picture_description`):
```
Describe this image for a blind user. State the image type
(screenshot, chart, photo, etc.), what it depicts, any visible text,
and key visual details. Be concise and accurate.
```
To customize the prompt globally:
```yaml
prompts:
picture_description: "Your custom prompt here..."
```
**Using with Ollama:**
```yaml
processing:
conversion_options:
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
```
Requires Ollama running with a vision-capable model:
```bash
ollama pull ministral-3
ollama serve
```
**Using with vLLM or custom endpoints:**
```yaml
processing:
conversion_options:
picture_description:
enabled: true
model:
provider: openai # Use OpenAI-compatible API format
name: granite-vision
base_url: http://my-vllm-server:8000
```
**How it works:**
1. During PDF conversion, docling extracts embedded images
2. Each image is sent to the configured VLM for description
3. Descriptions are added as annotations on the image
4. When exported to markdown, descriptions appear as searchable text
**Using with docling-serve:**
When using `converter: docling-serve`, the VLM calls are made by the docling-serve instance, not by haiku.rag. You must:
1. Set `DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` when running docling-serve
2. Ensure the VLM endpoint is accessible from where docling-serve is running
**Docker networking:** If docling-serve runs in Docker and your VLM runs on the host, use `host.docker.internal` instead of `localhost`:
```yaml
processing:
conversion_options:
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
base_url: http://host.docker.internal:11434 # NOT localhost!
```
See [VLM Picture Description with docling-serve](../remote-processing.md#vlm-picture-description-with-docling-serve) for a complete example.
### Automatic Title Generation
Enable automatic title generation during document ingestion:

17
docs/configuration/providers.md
View file

@ -190,6 +190,23 @@ embeddings:
**Note:** The `base_url` must include the `/v1` path for OpenAI-compatible endpoints.
### vLLM (multimodal)
For cross-modal retrieval (text and pictures share a single vector space), use the dedicated `vllm` provider against a vLLM server hosting a multimodal embedding model:
```yaml
embeddings:
model:
provider: vllm
name: Qwen/Qwen3-VL-Embedding-8B
vector_dim: 4096
base_url: http://localhost:8000/v1
```
Tested with `Qwen/Qwen3-VL-Embedding-8B` (4096-dim) and `jinaai/jina-embeddings-v4` (2048-dim). Run vLLM separately; haiku.rag adds no Python ML dependencies for this path. Text inputs use the standard OpenAI `input` field; image inputs use vLLM's `messages`-with-`image_url` superset, transparently to the caller.
Picture chunks for retrieval are emitted at ingest under any embedder reporting `supports_images=True`. See [Picture Handling](processing.md#picture-handling).
## Question Answering Providers
Configure which LLM provider to use for question answering. Any provider and model supported by [Pydantic AI](https://ai.pydantic.dev/models/) can be used.

6
docs/index.md
View file

@ -2,16 +2,20 @@
Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/).
> **New: vision and multimodal search.** Picture-aware ingestion captures embedded figure bytes; vision-capable QA models receive them alongside text. Multimodal embedders (vLLM with `Qwen3-VL-Embedding-8B` or `jinaai/jina-embeddings-v4`) put picture vectors in the same space as text, enabling text-as-query → figure hits and image-as-query retrieval.
## Features
- **Hybrid search** — Vector + full-text with Reciprocal Rank Fusion
- **Multimodal & cross-modal search** — Multimodal embedders (vLLM) put picture vectors in the same space as text; supports text-as-query → figure hits and image-as-query
- **Question answering** — QA agents with citations (page numbers, section headings)
- **Vision QA** — Vision-capable models receive figure bytes alongside chunk text via pydantic-ai `BinaryContent` when `qa.model.vision = true`
- **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM
- **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize
- **Analysis agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis)
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM (multimodal). QA/Research: any model supported by Pydantic AI
- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
- **CLI & Python API** — Full functionality from command line or code
- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)