From ab5cfdd04a7798c39153bf11930ad6c25cc68360 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Fri, 8 May 2026 11:04:32 +0300 Subject: [PATCH] Docs & cl --- CHANGELOG.md | 36 ++++---- docs/configuration/processing.md | 144 ++++++------------------------- docs/configuration/providers.md | 17 ++++ docs/index.md | 6 +- 4 files changed, 63 insertions(+), 140 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a1868cfb..1c08b7ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,30 +3,24 @@ ### Added -- **`rebuild --descriptions` mode.** Adds VLM picture descriptions to an existing database without re-converting from source. Loads each document's stored docling blob, identifies pictures lacking `meta.description.text`, drives the configured VLM (via pydantic-ai `BinaryContent`) over the picture bytes already stored in `document_items.picture_data`, patches descriptions into the blob, then re-chunks + re-embeds so chunk text reflects them. Skips the docling parse entirely; only the VLM time is paid. Idempotent — pictures that already carry a description are not re-described, so the operation is safe to re-run after a partial failure. Errors clearly when `picture_description.enabled` is false. Exposed as `haiku-rag rebuild --descriptions` and `RebuildMode.DESCRIPTIONS`. -- **Silent-failure guard for picture descriptions.** When `picture_description.enabled=True` and a converted document has at least one picture but zero of them came back with a description, `client.processing.convert()` now logs a clear warning naming the source path, picture count, configured VLM model, and base URL. docling-serve swallows VLM errors (network failures, missing models, unreachable hosts) and returns a "successful" conversion with empty descriptions; this guard surfaces the failure before a long ingest produces a corpus with no descriptions in it. -- **MCP image-query tool + CLI `--image PATH`.** New MCP tool `search_documents_by_image(image_base64, limit, include_images)` routes a base64-encoded image through `client.search()`. Registered only when the configured embedder supports images, so non-multimodal MCP servers don't expose a tool that would always fail. The `haiku-rag search` CLI gains an `--image PATH` flag that reads the file and runs the same image-as-query path. -- **Image-as-query search.** `client.search()` now accepts `str | bytes | PIL.Image.Image`. Bytes/PIL queries embed via the multimodal embedder's `embed_image_query` and dispatch to vector-only chunk search (FTS doesn't apply to non-text queries; reranking is also skipped). Raises a clear error if the configured embedder is text-only. `ChunkRepository.search()` gains an optional `query_vector` parameter that bypasses `embed_query` and forces the vector-only path. -- **`vision: bool` flag on `ModelConfig`.** Tracks whether a configured language model can interpret images. Default `False`. The agent's `search` tool only attaches picture bytes (as `BinaryContent`) to the `ToolReturn` when `qa.model.vision = True`. Without the gate, sending image content to a text-only model behaves inconsistently across providers — Ollama silently accepts and the model hallucinates a confident wrong answer; OpenAI returns 400; others vary. Capability detection from a probe or a model-name whitelist is unreliable, so `vision` is an explicit user-set capability declaration. Set it to `True` for vision-capable QA models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, ...). -- **Synthetic picture chunks at ingest under multimodal embedders.** `build_picture_chunks` (in `client/processing.py`) walks a `DoclingDocument`'s `pictures` and emits one synthetic `Chunk` per `PictureItem` with available bytes. Bytes ride on a `Chunk._picture_data` PrivateAttr (not serialized) so `embed_chunks` can route them through `embed_images` while text chunks keep going through `embed_documents`. Wired into the three ingest paths (`create_document`, `_create_document_from_file`, `_create_or_update_document_from_url`, `_update_document_with_chunks`, and `_rebuild_rechunk`) — guarded by `embedder.supports_images` so text-only configurations are unaffected. Snapshot/merge with `existing_picture_data` keeps picture chunks alive across rebuild round-trips. Picture chunks land in the same `chunks` table with the same vector dim as text chunks, so cross-modal search reuses the existing hybrid+RRF pipeline. -- **Multimodal embedder support (`provider="vllm"`).** `EmbedderWrapper` gains `supports_images: bool` and `embed_image_query`. The `vllm` provider talks HTTP to a vLLM server's OpenAI-compatible `/v1/embeddings` endpoint — text inputs use the standard `input` field for true server-side batching; image inputs use vLLM's `messages` superset with `image_url` content parts carrying base64 data URIs. Works with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No Python ML deps added — uses `httpx`. Text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) report `supports_images=False` and raise a clear error if image methods are called. -- **Picture bytes are always stored.** Every ingest captures embedded picture bytes into `document_items.picture_data`; rebuilds and updates always preserve them. The single ingest-time decision is whether to run a VLM at ingest, exposed via `processing.conversion_options.picture_description.enabled: bool` (default `False`). When enabled, descriptions are woven into chunk text; when disabled, chunks contain only their natural text. No reingest is required to flip the VLM on or off — `haiku-rag rebuild --rechunk` recomputes chunk text against the new setting. The earlier `processing.conversion_options.generate_picture_images` flag is dropped from loaded YAMLs and a warning prints once telling the user to remove it from their config. -- **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`. - - New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`. - - Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory. -- **Embedded picture bytes captured at ingestion.** `extract_items` decodes `PictureItem.image.uri` data URIs into raw bytes on `document_items.picture_data`, and surfaces VLM picture descriptions (`meta.description.text`) into `DocumentItem.text`. - - `compress_docling_split` strips picture URIs from the stored docling blob so they live in one place. - - The 0.45.0 migration adds the column on existing DBs and backfills it from the legacy blob. - - Rebuild and update snapshot existing bytes via `get_all_picture_data` so re-chunk preserves them. -- **Picture image bytes in search results and vision-capable QA.** `SearchResult.image_data: dict[str, str] | None` carries base64 picture bytes keyed by `self_ref`. - - `client.search()` and the MCP `search_documents` tool gain `include_images: bool = True`. - - `expand_context` preserves picture self_refs with empty text and re-attaches bytes to the rebuilt `SearchResult`s. - - The agent's search tool returns `pydantic_ai.messages.ToolReturn(return_value=text, content=[BinaryContent(...)])` when picture data is present, otherwise a plain string. +- **Vision capabilities.** Picture-aware ingestion, vision QA, multimodal embeddings, and image-as-query search. +- **Picture bytes always stored** at ingest in a new `document_items.picture_data` column (`large_binary`), addressable by `(document_id, self_ref)`. Bulk read paths project metadata-only so bytes never leak into context expansion or analysis-sandbox builds. The 0.45.0 migration adds the column on existing DBs and backfills it from each doc's docling blob; URIs are then stripped from the blob so bytes live in one place. +- **VLM picture descriptions** at ingest via `processing.conversion_options.picture_description.enabled` (default `false`). When enabled, descriptions are woven into chunk text. The earlier `generate_picture_images` flag is dropped with a one-time warning. `haiku-rag rebuild --descriptions` runs the VLM over stored bytes after the fact, idempotently — skipping the docling parse entirely. +- **Multimodal embedder (`provider="vllm"`)** for cross-modal retrieval. Talks HTTP to a vLLM `/v1/embeddings` endpoint (`input` array for text, `messages` superset with `image_url` for images). Tested with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No new Python ML dependencies. Under multimodal embedders, ingest emits one synthetic picture chunk per `PictureItem`, sharing the chunks table with text. +- **Image-as-query search.** `client.search()` accepts `str | bytes | PIL.Image.Image`. Image queries embed once and run vector-only against the chunks table. New CLI flag `haiku-rag search --image PATH` and new MCP tool `search_documents_by_image(image_base64, ...)` (registered only when the embedder supports images). +- **Vision QA via `qa.model.vision: bool` flag** on `ModelConfig` (default `false`). When `true`, the agent's `search` tool attaches picture bytes as `BinaryContent` parts on its `ToolReturn`. Default is `false` because providers behave inconsistently when an image is sent to a text-only model (Ollama silently accepts and confabulates; OpenAI returns 400). `SearchResult.image_data: dict[str, str] | None` carries base64 picture bytes keyed by `self_ref`; `client.search()` and MCP `search_documents` gain `include_images: bool = True`. +- **Silent-failure guard for picture descriptions.** When `picture_description.enabled=true` and a conversion returns at least one picture but zero descriptions, log a warning naming the source, picture count, VLM model, and base URL. Surfaces docling-serve's swallowed VLM errors (unreachable host, missing model) before they pollute a long ingest. +- **Inspector renders attached pictures** under `qa.model.vision=true` in the context modal (`c` key) so the inspector reflects what the LLM actually receives. ### Fixed -- **docling-serve picture-image extraction.** docling-serve only emits picture bytes under `image_export_mode="referenced"` (upstream [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576)). The converter now switches to `referenced` + `target_type="zip"` when picture images are requested, parses the returned zip, and rehydrates `artifacts/` URIs back into `data:;base64,...` URIs. The previously-`xfail`ed `test_convert_pdf_with_picture_images` test now passes. -- **`rebuild --rechunk` now reuses the stored docling blob** instead of re-converting from the markdown export, which dropped every `PictureItem` (and its bytes) on the floor. Documents without a stored docling blob now raise instead of silently falling back to markdown. +- **`rebuild --descriptions` no longer destroys `docling_pages`.** The previous implementation called `set_docling()` after a structure-only docling load, which writes `docling_pages=None` and clobbered page rasters for every doc with at least one undescribed picture (silently breaking `visualize_chunk` for the affected docs). +- **docling-serve picture-image extraction.** docling-serve only emits picture bytes under `image_export_mode="referenced"` (upstream [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576)). The converter switches to `referenced` + `target_type="zip"` when picture images are requested and rehydrates `artifacts/` URIs back into `data:` URIs. +- **`rebuild --rechunk` reuses the stored docling blob** instead of re-converting from the markdown export, which dropped every `PictureItem` on the floor. Documents without a stored docling blob now raise instead of silently falling back. + +### Changed + +- **Lazy document hydration during rebuild.** Each mode loop now fetches one full record at a time instead of eagerly loading all docs with their multi-MB blobs. Drops startup memory from ~15 GB to ~one document on a 1000-doc database. ## [0.44.0] - 2026-04-29 diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index 941573a2..c96cf48a 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -106,159 +106,67 @@ conversion_options: #### Picture Handling -Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest time to generate textual descriptions, set via `picture_description.enabled`: +Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest to generate textual descriptions: ```yaml processing: conversion_options: picture_description: - enabled: true # default false; runs the VLM at ingest + enabled: true # default false model: - provider: ollama # ollama, openai, or custom - name: ministral-3 # VLM model name - temperature: 0.0 - timeout: 90 # Request timeout in seconds - max_tokens: 200 # Maximum tokens in response + provider: ollama # any OpenAI-compatible /v1/chat/completions provider + name: ministral-3 + timeout: 90 + max_tokens: 200 ``` -When `enabled: false` (default), the VLM doesn't run; chunks contain only their natural text (captions, surrounding paragraphs). When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS. +When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS. The prompt is configurable under `prompts.picture_description` — see [Prompts](prompts.md). -**Switching the VLM on or off on an existing database.** Picture bytes are already stored, so no reingest is required. +**Switching the VLM on or off on an existing database** doesn't require reingesting (the bytes are already there): -- To turn the VLM **off** (descriptions already exist, you want to drop them): flip `enabled: false` and run `haiku-rag rebuild --rechunk`. Chunk text recomposes from the stripped docling blob. -- To turn the VLM **on** (descriptions don't exist yet, you want them now): flip `enabled: true` and run `haiku-rag rebuild --descriptions`. The VLM is driven over the picture bytes already in `document_items.picture_data`, descriptions are patched into the docling blob, and chunks are recomposed. The docling parse is skipped entirely. See [Rebuild Database](../cli.md#rebuild-database) for full details. +- Off → on: `haiku-rag rebuild --descriptions` runs the VLM over stored bytes and re-chunks. Skips the docling parse entirely. +- On → off: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions. -#### Picture descriptions × embedder × QA model: how the pieces compose +When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag — see [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve). -Three settings drive what gets stored, what gets retrieved, and what reaches the QA model: +#### Pictures × embedder × QA model: how the pieces compose + +Three independent settings drive ingest, retrieval, and QA: | Setting | Question it answers | Values | |---|---|---| | `picture_description.enabled` | Should a VLM weave descriptions into chunk text at ingest? | `false` (default) / `true` | -| `embeddings.model.provider` | Can the embedder index image content? | text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) | +| `embeddings.model.provider` | Can the embedder index image content? | text-only (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) | | `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` | Picture bytes are always stored, regardless of these settings. -**What gets stored** for each `picture_description.enabled` × embedder combination: +**What gets stored** by `enabled` × embedder: -| `enabled` | Embedder | Text chunks contain… | Synthetic picture chunks | +| `enabled` | Embedder | Text chunks | Synthetic picture chunks | |---|---|---|---| | `false` | text-only | text only (caption/surrounding) | none | -| `false` | multimodal | text only | one per picture, content = caption/empty, vector = image embedding | -| `true` | text-only | text + VLM descriptions | none | -| `true` | multimodal | text + VLM descriptions | one per picture, content = description, vector = image embedding | +| `false` | multimodal | text only | one per picture, vector = image embedding | +| `true` | text-only | text + descriptions | none | +| `true` | multimodal | text + descriptions | one per picture, vector = image embedding | -**What QA receives** at search time, given stored state and `qa.model.vision`: +**What QA receives** at search time: -| `qa.model.vision` | QA receives | -|---|---| -| `false` | text chunks only (descriptions in chunk text answer figure questions in prose when `picture_description.enabled` was true) | -| `true` | text chunks + raw picture bytes; vision model reads the figures directly | +- `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose). +- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`; the model reads figures directly. -A few invariants worth knowing: +`qa.model.vision` is independent of ingestion — flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI; default `false` is the safe choice. -- **`qa.model.vision` is independent of ingestion.** It only controls whether the agent's `search` tool attaches picture bytes to its `ToolReturn`. A text-only QA model with `vision: true` won't suddenly understand images — it will silently accept the bytes and confabulate. Default `false` is the safe choice. -- **The bytes are always there**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself. -- **Cross-modal search** (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks. - -**Recommended combinations** by use case: +**Recommended combinations:** | Use case | `picture_description.enabled` | Embedder | `qa.model.vision` | |---|---|---|---| | Pure text RAG, no figures | `false` | text-only | `false` | | Text RAG, figures answered through descriptions | `true` | text-only | `false` | | Vision QA on figure-rich docs (no cross-modal search) | `true` or `false` | text-only | `true` | -| Cross-modal search + vision QA (the full multimodal stack) | `true` or `false` | multimodal | `true` | +| Cross-modal search + vision QA | `true` or `false` | multimodal | `true` | | Cross-modal search, text QA only | `true` | multimodal | `false` | -**`picture_description.model` configuration** (used only when `picture_description.enabled: true`): - -- **model**: Standard model configuration - - `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints - - `name`: Model name (e.g., `ministral-3`, `granite3.2-vision`, `gpt-4-vision`) - - `base_url`: Optional custom API endpoint for vLLM, LM Studio, etc. -- **timeout**: Request timeout in seconds -- **max_tokens**: Maximum tokens in the VLM response - -**Note:** Requires an OpenAI-compatible `/v1/chat/completions` endpoint. Providers with different API formats (e.g., Anthropic Claude) are not supported. - -**Default prompt** (configured in `prompts.picture_description`): - -``` -Describe this image for a blind user. State the image type -(screenshot, chart, photo, etc.), what it depicts, any visible text, -and key visual details. Be concise and accurate. -``` - -To customize the prompt globally: - -```yaml -prompts: - picture_description: "Your custom prompt here..." -``` - -**Using with Ollama:** - -```yaml -processing: - conversion_options: - picture_description: - enabled: true - model: - provider: ollama - name: ministral-3 -``` - -Requires Ollama running with a vision-capable model: - -```bash -ollama pull ministral-3 -ollama serve -``` - -**Using with vLLM or custom endpoints:** - -```yaml -processing: - conversion_options: - picture_description: - enabled: true - model: - provider: openai # Use OpenAI-compatible API format - name: granite-vision - base_url: http://my-vllm-server:8000 -``` - -**How it works:** - -1. During PDF conversion, docling extracts embedded images -2. Each image is sent to the configured VLM for description -3. Descriptions are added as annotations on the image -4. When exported to markdown, descriptions appear as searchable text - -**Using with docling-serve:** - -When using `converter: docling-serve`, the VLM calls are made by the docling-serve instance, not by haiku.rag. You must: - -1. Set `DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` when running docling-serve -2. Ensure the VLM endpoint is accessible from where docling-serve is running - -**Docker networking:** If docling-serve runs in Docker and your VLM runs on the host, use `host.docker.internal` instead of `localhost`: - -```yaml -processing: - conversion_options: - picture_description: - enabled: true - model: - provider: ollama - name: ministral-3 - base_url: http://host.docker.internal:11434 # NOT localhost! -``` - -See [VLM Picture Description with docling-serve](../remote-processing.md#vlm-picture-description-with-docling-serve) for a complete example. - ### Automatic Title Generation Enable automatic title generation during document ingestion: diff --git a/docs/configuration/providers.md b/docs/configuration/providers.md index 7ab86f6e..15169d44 100644 --- a/docs/configuration/providers.md +++ b/docs/configuration/providers.md @@ -190,6 +190,23 @@ embeddings: **Note:** The `base_url` must include the `/v1` path for OpenAI-compatible endpoints. +### vLLM (multimodal) + +For cross-modal retrieval (text and pictures share a single vector space), use the dedicated `vllm` provider against a vLLM server hosting a multimodal embedding model: + +```yaml +embeddings: + model: + provider: vllm + name: Qwen/Qwen3-VL-Embedding-8B + vector_dim: 4096 + base_url: http://localhost:8000/v1 +``` + +Tested with `Qwen/Qwen3-VL-Embedding-8B` (4096-dim) and `jinaai/jina-embeddings-v4` (2048-dim). Run vLLM separately; haiku.rag adds no Python ML dependencies for this path. Text inputs use the standard OpenAI `input` field; image inputs use vLLM's `messages`-with-`image_url` superset, transparently to the caller. + +Picture chunks for retrieval are emitted at ingest under any embedder reporting `supports_images=True`. See [Picture Handling](processing.md#picture-handling). + ## Question Answering Providers Configure which LLM provider to use for question answering. Any provider and model supported by [Pydantic AI](https://ai.pydantic.dev/models/) can be used. diff --git a/docs/index.md b/docs/index.md index 73f6a846..9ad17be2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,16 +2,20 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/). +> **New: vision and multimodal search.** Picture-aware ingestion captures embedded figure bytes; vision-capable QA models receive them alongside text. Multimodal embedders (vLLM with `Qwen3-VL-Embedding-8B` or `jinaai/jina-embeddings-v4`) put picture vectors in the same space as text, enabling text-as-query → figure hits and image-as-query retrieval. + ## Features - **Hybrid search** — Vector + full-text with Reciprocal Rank Fusion +- **Multimodal & cross-modal search** — Multimodal embedders (vLLM) put picture vectors in the same space as text; supports text-as-query → figure hits and image-as-query - **Question answering** — QA agents with citations (page numbers, section headings) +- **Vision QA** — Vision-capable models receive figure bytes alongside chunk text via pydantic-ai `BinaryContent` when `qa.model.vision = true` - **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM - **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize - **Analysis agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis) - **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory - **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion -- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI +- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM (multimodal). QA/Research: any model supported by Pydantic AI - **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud - **CLI & Python API** — Full functionality from command line or code - **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)