collapse pictures enum to picture_description.enabled boolean

This commit is contained in:
Yiorgis Gozadinos 2026-05-05 11:16:08 +03:00
parent e88321767d
commit 2038d43435
No known key found for this signature in database
13 changed files with 157 additions and 192 deletions

View file

@ -8,12 +8,7 @@
- **`vision: bool` flag on `ModelConfig`.** Tracks whether a configured language model can interpret images. Default `False`. The agent's `search` tool only attaches picture bytes (as `BinaryContent`) to the `ToolReturn` when `qa.model.vision = True`. Without the gate, sending image content to a text-only model behaves inconsistently across providers — Ollama silently accepts and the model hallucinates a confident wrong answer; OpenAI returns 400; others vary. Capability detection from a probe or a model-name whitelist is unreliable, so `vision` is an explicit user-set capability declaration. Set it to `True` for vision-capable QA models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, ...). - **`vision: bool` flag on `ModelConfig`.** Tracks whether a configured language model can interpret images. Default `False`. The agent's `search` tool only attaches picture bytes (as `BinaryContent`) to the `ToolReturn` when `qa.model.vision = True`. Without the gate, sending image content to a text-only model behaves inconsistently across providers — Ollama silently accepts and the model hallucinates a confident wrong answer; OpenAI returns 400; others vary. Capability detection from a probe or a model-name whitelist is unreliable, so `vision` is an explicit user-set capability declaration. Set it to `True` for vision-capable QA models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, ...).
- **Synthetic picture chunks at ingest under multimodal embedders.** `build_picture_chunks` (in `client/processing.py`) walks a `DoclingDocument`'s `pictures` and emits one synthetic `Chunk` per `PictureItem` with available bytes. Bytes ride on a `Chunk._picture_data` PrivateAttr (not serialized) so `embed_chunks` can route them through `embed_images` while text chunks keep going through `embed_documents`. Wired into the three ingest paths (`create_document`, `_create_document_from_file`, `_create_or_update_document_from_url`, `_update_document_with_chunks`, and `_rebuild_rechunk`) — guarded by `embedder.supports_images` so text-only configurations are unaffected. Snapshot/merge with `existing_picture_data` keeps picture chunks alive across rebuild round-trips. Picture chunks land in the same `chunks` table with the same vector dim as text chunks, so cross-modal search reuses the existing hybrid+RRF pipeline. - **Synthetic picture chunks at ingest under multimodal embedders.** `build_picture_chunks` (in `client/processing.py`) walks a `DoclingDocument`'s `pictures` and emits one synthetic `Chunk` per `PictureItem` with available bytes. Bytes ride on a `Chunk._picture_data` PrivateAttr (not serialized) so `embed_chunks` can route them through `embed_images` while text chunks keep going through `embed_documents`. Wired into the three ingest paths (`create_document`, `_create_document_from_file`, `_create_or_update_document_from_url`, `_update_document_with_chunks`, and `_rebuild_rechunk`) — guarded by `embedder.supports_images` so text-only configurations are unaffected. Snapshot/merge with `existing_picture_data` keeps picture chunks alive across rebuild round-trips. Picture chunks land in the same `chunks` table with the same vector dim as text chunks, so cross-modal search reuses the existing hybrid+RRF pipeline.
- **Multimodal embedder support (`provider="vllm"`).** `EmbedderWrapper` gains `supports_images: bool` and `embed_image_query`. The `vllm` provider talks HTTP to a vLLM server's OpenAI-compatible `/v1/embeddings` endpoint — text inputs use the standard `input` field for true server-side batching; image inputs use vLLM's `messages` superset with `image_url` content parts carrying base64 data URIs. Works with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No Python ML deps added — uses `httpx`. Text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) report `supports_images=False` and raise a clear error if image methods are called. - **Multimodal embedder support (`provider="vllm"`).** `EmbedderWrapper` gains `supports_images: bool` and `embed_image_query`. The `vllm` provider talks HTTP to a vLLM server's OpenAI-compatible `/v1/embeddings` endpoint — text inputs use the standard `input` field for true server-side batching; image inputs use vLLM's `messages` superset with `image_url` content parts carrying base64 data URIs. Works with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No Python ML deps added — uses `httpx`. Text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) report `supports_images=False` and raise a clear error if image methods are called.
- **Picture-handling mode enum.** `processing.pictures: "none" | "description" | "image"` (default `"none"`) replaces the legacy `generate_picture_images` + `picture_description.enabled` pair. - **Picture bytes are always stored.** Every ingest captures embedded picture bytes into `document_items.picture_data`; rebuilds and updates always preserve them. The single ingest-time decision is whether to run a VLM at ingest, exposed via `processing.conversion_options.picture_description.enabled: bool` (default `False`). When enabled, descriptions are woven into chunk text; when disabled, chunks contain only their natural text. No reingest is required to flip the VLM on or off — `haiku-rag rebuild --rechunk` recomputes chunk text against the new setting. The legacy `processing.pictures: "none" | "description" | "image"` enum and `processing.conversion_options.generate_picture_images` flag are auto-translated by the YAML loader with a one-time deprecation warning per field.
- `none` — no picture bytes, no VLM, just structural picture rows.
- `description` — VLM runs at ingest, descriptions woven into chunk text, bytes also retained on `document_items.picture_data` so vision QA can be turned on later without reingesting.
- `image` — bytes retained, no VLM.
- YAML loader translates legacy configs and emits a deprecation warning.
- Rebuild and update under `pictures="none"` skip the picture-bytes snapshot/merge so storage is reclaimed.
- **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`. - **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`.
- New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`. - New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`.
- Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory. - Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory.

View file

@ -28,9 +28,6 @@ processing:
name: gpt-oss name: gpt-oss
enable_thinking: false enable_thinking: false
# Picture handling (none / description / image)
pictures: none # See "Picture Handling" below
# Conversion options (works with both local and remote converters) # Conversion options (works with both local and remote converters)
conversion_options: conversion_options:
# OCR settings # OCR settings
@ -48,8 +45,9 @@ processing:
images_scale: 2.0 # Image scale factor images_scale: 2.0 # Image scale factor
generate_page_images: true # Include rendered page images (for visualize_chunk) generate_page_images: true # Include rendered page images (for visualize_chunk)
# VLM picture description settings (only effective when pictures: description) # VLM picture description (off by default; see "Picture Handling" below)
picture_description: picture_description:
enabled: false
model: model:
provider: ollama provider: ollama
name: ministral-3 name: ministral-3
@ -106,27 +104,15 @@ conversion_options:
- **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0. - **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
- **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size. - **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size.
Embedded picture extraction is controlled by `processing.pictures` (see [Picture Handling](#picture-handling) below), not by an image-settings flag.
#### Picture Handling #### Picture Handling
`processing.pictures` is a single enum that decides how embedded picture images (figures, diagrams) are handled at ingest. Three modes: Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest time to generate textual descriptions, set via `picture_description.enabled`:
| Mode | VLM at ingest | `picture_data` populated | Chunk text contains description |
|---|---|---|---|
| `none` (default) | no | no | no |
| `description` | yes | yes | yes |
| `image` | no | yes | no |
- **`none`**: docling skips picture-image generation. `label="picture"` rows still appear in the items table for structural metadata, but they carry no bytes and no description. Cheapest mode; non-vision QA is unaffected.
- **`description`**: docling generates picture images, the configured VLM produces a description woven into chunk text, and the bytes are also retained in `document_items.picture_data`. Picture-text is searchable via FTS, vision-capable QA models also receive the bytes via the agent's search tool. Bytes are kept (not just thrown away after the VLM runs) so a vision-only QA strategy can be turned on later without reingesting.
- **`image`**: docling generates picture images and stores them in `document_items.picture_data` without running the VLM. Vision-capable QA models reason directly about the figures; non-vision QA only sees the picture's caption/surrounding text.
```yaml ```yaml
processing: processing:
pictures: description # none / description / image
conversion_options: conversion_options:
picture_description: # only effective when pictures: description picture_description:
enabled: true # default false; runs the VLM at ingest
model: model:
provider: ollama # ollama, openai, or custom provider: ollama # ollama, openai, or custom
name: ministral-3 # VLM model name name: ministral-3 # VLM model name
@ -135,56 +121,55 @@ processing:
max_tokens: 200 # Maximum tokens in response max_tokens: 200 # Maximum tokens in response
``` ```
**Switching modes on an existing database.** No reingest is required if you only need to change between `description` and `image` — the bytes are already there. Run `haiku-rag rebuild --rechunk` after the config change so the chunk-text composition reflects the new mode. Switching *down* to `none` clears `picture_data` on rebuild, reclaiming storage. When `enabled: false` (default), the VLM doesn't run; chunks contain only their natural text (captions, surrounding paragraphs). When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS.
#### Pictures × embedder × QA model: how the pieces compose **Switching the VLM on or off on an existing database.** Picture bytes are already stored, so no reingest is required. Run `haiku-rag rebuild --rechunk` after flipping `enabled` so the chunk-text composition reflects the new setting.
Three orthogonal settings drive what gets stored, what gets retrieved, and what reaches the QA model. Each setting answers one question: #### Picture descriptions × embedder × QA model: how the pieces compose
Three settings drive what gets stored, what gets retrieved, and what reaches the QA model:
| Setting | Question it answers | Values | | Setting | Question it answers | Values |
|---|---|---| |---|---|---|
| `processing.pictures` | What gets captured at ingest? | `none` / `description` / `image` | | `picture_description.enabled` | Should a VLM weave descriptions into chunk text at ingest? | `false` (default) / `true` |
| `embeddings.model.provider` | Can the embedder index image content? | text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) | | `embeddings.model.provider` | Can the embedder index image content? | text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
| `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` | | `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` |
**What gets stored** for each `pictures` × embedder combination: Picture bytes are always stored, regardless of these settings.
| `pictures` | Embedder | Text chunks contain… | `document_items.picture_data` | Synthetic picture chunks | **What gets stored** for each `picture_description.enabled` × embedder combination:
|---|---|---|---|---|
| `none` | text-only or multimodal | regular text only | empty | none | | `enabled` | Embedder | Text chunks contain… | Synthetic picture chunks |
| `description` | text-only | text + VLM descriptions | populated (kept for later use) | none | |---|---|---|---|
| `description` | multimodal | text + VLM descriptions | populated | one per picture, content = description, vector = image embedding | | `false` | text-only | text only (caption/surrounding) | none |
| `image` | text-only | text only (caption/surrounding) | populated (kept for later use) | none | | `false` | multimodal | text only | one per picture, content = caption/empty, vector = image embedding |
| `image` | multimodal | text only | populated | one per picture, content = caption/empty, vector = image embedding | | `true` | text-only | text + VLM descriptions | none |
| `true` | multimodal | text + VLM descriptions | one per picture, content = description, vector = image embedding |
**What QA receives** at search time, given stored state and `qa.model.vision`: **What QA receives** at search time, given stored state and `qa.model.vision`:
| `pictures` at ingest | `qa.model.vision` | QA receives | | `qa.model.vision` | QA receives |
|---|---|---| |---|---|
| `none` | either | text chunks only | | `false` | text chunks only (descriptions in chunk text answer figure questions in prose when `picture_description.enabled` was true) |
| `description` | `false` | text chunks (descriptions in chunk text answer figure questions in prose) | | `true` | text chunks + raw picture bytes; vision model reads the figures directly |
| `description` | `true` | text chunks + raw picture bytes; vision model uses both signals |
| `image` | `false` | text chunks only (caption + surrounding text); model has no figure content to draw on |
| `image` | `true` | text chunks + raw picture bytes; vision model reads the figures directly |
A few invariants worth knowing: A few invariants worth knowing:
- **`qa.model.vision` is independent of ingestion.** It only controls whether the agent's `search` tool attaches picture bytes to its `ToolReturn`. A text-only QA model with `vision: true` won't suddenly understand images — it will silently accept the bytes and confabulate. Default `false` is the safe choice. - **`qa.model.vision` is independent of ingestion.** It only controls whether the agent's `search` tool attaches picture bytes to its `ToolReturn`. A text-only QA model with `vision: true` won't suddenly understand images — it will silently accept the bytes and confabulate. Default `false` is the safe choice.
- **`description` mode preserves the bytes**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself. - **The bytes are always there**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself.
- **Cross-modal search** (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks. - **Cross-modal search** (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks.
- **`image` mode + text-only embedder** is rarely the right choice — pictures are stored but neither searchable as image vectors nor described in text. Picture bytes are then only reachable via expand-context's section bounds when an adjacent text chunk matches.
**Recommended combinations** by use case: **Recommended combinations** by use case:
| Use case | `pictures` | Embedder | `qa.model.vision` | | Use case | `picture_description.enabled` | Embedder | `qa.model.vision` |
|---|---|---|---| |---|---|---|---|
| Pure text RAG, no figures | `none` | text-only | `false` | | Pure text RAG, no figures | `false` | text-only | `false` |
| Text RAG, figures answered through descriptions | `description` | text-only | `false` | | Text RAG, figures answered through descriptions | `true` | text-only | `false` |
| Vision QA on figure-rich docs (no cross-modal search) | `description` | text-only | `true` | | Vision QA on figure-rich docs (no cross-modal search) | `true` or `false` | text-only | `true` |
| Cross-modal search + vision QA (the full multimodal stack) | `description` or `image` | multimodal | `true` | | Cross-modal search + vision QA (the full multimodal stack) | `true` or `false` | multimodal | `true` |
| Cross-modal search, text QA only | `description` | multimodal | `false` | | Cross-modal search, text QA only | `true` | multimodal | `false` |
**`picture_description.model` configuration** (used only under `pictures: description`): **`picture_description.model` configuration** (used only when `picture_description.enabled: true`):
- **model**: Standard model configuration - **model**: Standard model configuration
- `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints - `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
@ -214,9 +199,9 @@ prompts:
```yaml ```yaml
processing: processing:
pictures: description
conversion_options: conversion_options:
picture_description: picture_description:
enabled: true
model: model:
provider: ollama provider: ollama
name: ministral-3 name: ministral-3
@ -233,9 +218,9 @@ ollama serve
```yaml ```yaml
processing: processing:
pictures: description
conversion_options: conversion_options:
picture_description: picture_description:
enabled: true
model: model:
provider: openai # Use OpenAI-compatible API format provider: openai # Use OpenAI-compatible API format
name: granite-vision name: granite-vision
@ -260,9 +245,9 @@ When using `converter: docling-serve`, the VLM calls are made by the docling-ser
```yaml ```yaml
processing: processing:
pictures: description
conversion_options: conversion_options:
picture_description: picture_description:
enabled: true
model: model:
provider: ollama provider: ollama
name: ministral-3 name: ministral-3

View file

@ -119,7 +119,7 @@ prompts:
Be concise and factual. Be concise and factual.
``` ```
The prompt is used when `processing.pictures` is set to `description`. See [Picture Handling](processing.md#picture-handling) for full configuration. The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Handling](processing.md#picture-handling) for full configuration.
## Programmatic Configuration ## Programmatic Configuration

View file

@ -105,10 +105,7 @@ async def download_models(
if config.reranking.model and config.reranking.model.provider == "ollama": if config.reranking.model and config.reranking.model.provider == "ollama":
required_models.add(config.reranking.model.name) required_models.add(config.reranking.model.name)
pic_desc = config.processing.conversion_options.picture_description pic_desc = config.processing.conversion_options.picture_description
if ( if pic_desc.enabled and pic_desc.model.provider == "ollama":
config.processing.pictures == "description"
and pic_desc.model.provider == "ollama"
):
required_models.add(pic_desc.model.name) required_models.add(pic_desc.model.name)
if ( if (
config.processing.auto_title config.processing.auto_title

View file

@ -53,56 +53,54 @@ def load_yaml_config(path: Path) -> dict:
def _translate_legacy_picture_fields(data: dict) -> None: def _translate_legacy_picture_fields(data: dict) -> None:
"""Map legacy picture knobs onto ``processing.pictures``. """Map legacy picture-handling knobs onto
``processing.conversion_options.picture_description.enabled``.
Older configs expressed the same intent through two booleans on Two earlier shapes need translating:
``conversion_options``: ``generate_picture_images`` and
``picture_description.enabled``. Translation, in priority order:
- ``picture_description.enabled = true`` (regardless of the image flag) - ``processing.pictures: "description"``
``pictures = "description"``. ``picture_description.enabled = true``. The other values
- ``generate_picture_images = true`` (and no description) ``"image"``. (``"none"``, ``"image"``) collapse to ``false`` since the only
- both false / missing no translation; default ``"none"`` applies. remaining decision is whether the VLM runs; picture bytes are
always stored.
- ``processing.conversion_options.generate_picture_images: <any>`` is a
no-op now (docling always extracts picture bytes) and is dropped
with a one-time warning.
If ``pictures`` is already set on the loaded YAML it wins. Mutates If ``picture_description.enabled`` is already explicitly set on the
``data`` in-place and emits one warning per legacy field encountered. loaded YAML, it wins. Mutates ``data`` in-place and emits one warning
per legacy field encountered.
""" """
processing = data.get("processing") processing = data.get("processing")
if not isinstance(processing, dict): if not isinstance(processing, dict):
return return
if "pictures" in processing: legacy_pictures = processing.pop("pictures", None)
# User has migrated; legacy fields may still be present but should not if legacy_pictures is not None:
# override the explicit choice. Drop them silently to avoid confusion. opts = processing.setdefault("conversion_options", {})
opts = processing.get("conversion_options") if not isinstance(opts, dict):
if isinstance(opts, dict): opts = {}
opts.pop("generate_picture_images", None) processing["conversion_options"] = opts
pic = opts.get("picture_description") pic = opts.setdefault("picture_description", {})
if isinstance(pic, dict): if not isinstance(pic, dict):
pic.pop("enabled", None) pic = {}
return opts["picture_description"] = pic
if "enabled" not in pic:
pic["enabled"] = legacy_pictures == "description"
logger.warning(
"Config: 'processing.pictures' is deprecated; use "
"'processing.conversion_options.picture_description.enabled' "
"instead. Picture bytes are now always stored. Please update "
"your haiku.rag.yaml."
)
opts = processing.get("conversion_options") opts = processing.get("conversion_options")
if not isinstance(opts, dict): if isinstance(opts, dict) and "generate_picture_images" in opts:
return opts.pop("generate_picture_images", None)
pic = opts.get("picture_description") if isinstance(opts, dict) else None
legacy_describe = pic.pop("enabled", None) if isinstance(pic, dict) else None
legacy_image = opts.pop("generate_picture_images", None)
if legacy_describe:
processing["pictures"] = "description"
logger.warning( logger.warning(
"Config: 'processing.conversion_options.picture_description.enabled=true' is " "Config: 'processing.conversion_options.generate_picture_images' "
"deprecated; mapped to 'processing.pictures: description'. Please update your " "is deprecated and ignored; picture bytes are always extracted. "
"haiku.rag.yaml." "Please update your haiku.rag.yaml."
)
elif legacy_image:
processing["pictures"] = "image"
logger.warning(
"Config: 'processing.conversion_options.generate_picture_images=true' is "
"deprecated; mapped to 'processing.pictures: image'. Please update your "
"haiku.rag.yaml."
) )

View file

@ -112,12 +112,14 @@ class AnalysisConfig(BaseModel):
class PictureDescriptionConfig(BaseModel): class PictureDescriptionConfig(BaseModel):
"""Settings for the VLM that runs at ingest under ``pictures="description"``. """Whether (and how) to run a VLM over each picture at ingest.
Whether the VLM runs at all is decided by ``ProcessingConfig.pictures``; When ``enabled`` is True, picture descriptions are generated by the
these fields only describe *how* it runs once enabled. configured VLM and woven into chunk text. Picture bytes are stored
regardless this flag only controls the description pass.
""" """
enabled: bool = False
model: ModelConfig = Field( model: ModelConfig = Field(
default_factory=lambda: ModelConfig( default_factory=lambda: ModelConfig(
provider="ollama", provider="ollama",
@ -149,15 +151,11 @@ class ConversionOptions(BaseModel):
images_scale: float = 2.0 images_scale: float = 2.0
generate_page_images: bool = True generate_page_images: bool = True
# VLM picture description (only effective when ProcessingConfig.pictures == "description")
picture_description: PictureDescriptionConfig = Field( picture_description: PictureDescriptionConfig = Field(
default_factory=PictureDescriptionConfig default_factory=PictureDescriptionConfig
) )
PicturesMode = Literal["none", "description", "image"]
class ProcessingConfig(BaseModel): class ProcessingConfig(BaseModel):
chunk_size: int = 256 chunk_size: int = 256
converter: str = "docling-local" converter: str = "docling-local"
@ -167,18 +165,6 @@ class ProcessingConfig(BaseModel):
chunking_merge_peers: bool = True chunking_merge_peers: bool = True
chunking_use_markdown_tables: bool = False chunking_use_markdown_tables: bool = False
conversion_options: ConversionOptions = Field(default_factory=ConversionOptions) conversion_options: ConversionOptions = Field(default_factory=ConversionOptions)
pictures: PicturesMode = "none"
"""How embedded pictures are handled at ingest.
- ``"none"``: docling skips picture-image generation; structural
``label="picture"`` rows still exist but carry no bytes or description.
- ``"description"``: docling generates picture images, the VLM produces
text descriptions woven into chunk text, and the bytes are also
retained in ``document_items.picture_data`` so a vision-capable QA
model can be turned on later without reingesting.
- ``"image"``: docling generates picture images and stores them in
``document_items.picture_data``; no VLM runs at ingest.
"""
auto_title: bool = False auto_title: bool = False
title_model: ModelConfig = Field( title_model: ModelConfig = Field(
default_factory=lambda: ModelConfig( default_factory=lambda: ModelConfig(

View file

@ -123,7 +123,7 @@ class DoclingLocalConverter(DocumentConverter):
opts = self.config.processing.conversion_options opts = self.config.processing.conversion_options
pic_desc = opts.picture_description pic_desc = opts.picture_description
runs_vlm = self.config.processing.pictures == "description" runs_vlm = pic_desc.enabled
pipeline_options = PdfPipelineOptions( pipeline_options = PdfPipelineOptions(
do_ocr=opts.do_ocr, do_ocr=opts.do_ocr,

View file

@ -99,7 +99,7 @@ class DoclingServeConverter(DocumentConverter):
""" """
opts = self.config.processing.conversion_options opts = self.config.processing.conversion_options
pic_desc = opts.picture_description pic_desc = opts.picture_description
runs_vlm = self.config.processing.pictures == "description" runs_vlm = pic_desc.enabled
data: dict[str, str | list[str]] = { data: dict[str, str | list[str]] = {
"to_formats": "json", "to_formats": "json",

View file

@ -5,7 +5,6 @@ from haiku.rag.client.documents import (
_store_document_with_chunks, _store_document_with_chunks,
_update_document_with_chunks, _update_document_with_chunks,
) )
from haiku.rag.config import AppConfig
from haiku.rag.store.engine import Store from haiku.rag.store.engine import Store
from haiku.rag.store.models.document_item import ( from haiku.rag.store.models.document_item import (
DocumentItem, DocumentItem,
@ -647,11 +646,7 @@ class TestPictureDataPreservedThroughRoundTrip:
docling_doc = _docling_doc_with_picture() docling_doc = _docling_doc_with_picture()
config = AppConfig() async with HaikuRAG(temp_db_path, create=True) as rag:
# Preservation only kicks in under modes that retain picture bytes.
config.processing.pictures = "image"
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
document = Document(content="Hello world", uri="test://doc") document = Document(content="Hello world", uri="test://doc")
document.set_docling(docling_doc) document.set_docling(docling_doc)
created = await _store_document_with_chunks(rag, document, [], docling_doc) created = await _store_document_with_chunks(rag, document, [], docling_doc)

View file

@ -676,7 +676,6 @@ async def test_serve_chunker_accepts_picture_laden_docling():
from haiku.rag.converters.docling_serve import DoclingServeConverter from haiku.rag.converters.docling_serve import DoclingServeConverter
config = AppConfig() config = AppConfig()
config.processing.pictures = "image"
config.processing.conversion_options.do_ocr = False config.processing.conversion_options.do_ocr = False
config.processing.chunk_size = 256 config.processing.chunk_size = 256
config.processing.chunker_type = "hybrid" config.processing.chunker_type = "hybrid"

View file

@ -239,8 +239,10 @@ def test_init_config_creates_valid_yaml(tmp_path):
assert config.environment == "production" assert config.environment == "production"
# Legacy picture-handling field translation (`generate_picture_images` and # Legacy picture-handling field translation:
# `picture_description.enabled` → `processing.pictures` enum) # processing.pictures: "description" -> picture_description.enabled = true
# processing.pictures: "none" | "image" -> picture_description.enabled = false
# processing.conversion_options.generate_picture_images: <any> -> ignored
def _write(tmp_path, body: str): def _write(tmp_path, body: str):
@ -249,17 +251,16 @@ def _write(tmp_path, body: str):
return p return p
def test_load_yaml_legacy_picture_description_maps_to_description(tmp_path): def test_load_yaml_legacy_pictures_description_maps_to_enabled(tmp_path):
"""`picture_description.enabled=true` (with or without the image flag) """`processing.pictures: description` maps to
maps to `processing.pictures: description`.""" `picture_description.enabled = true` and warns."""
config_file = _write( config_file = _write(
tmp_path, tmp_path,
""" """
processing: processing:
pictures: description
conversion_options: conversion_options:
generate_picture_images: false
picture_description: picture_description:
enabled: true
timeout: 120 timeout: 120
""", """,
) )
@ -270,15 +271,49 @@ processing:
finally: finally:
loader_logger.removeHandler(handler) loader_logger.removeHandler(handler)
cfg = AppConfig.model_validate(data) cfg = AppConfig.model_validate(data)
assert cfg.processing.pictures == "description" assert cfg.processing.conversion_options.picture_description.enabled is True
assert cfg.processing.conversion_options.picture_description.timeout == 120 assert cfg.processing.conversion_options.picture_description.timeout == 120
assert any( assert any("processing.pictures" in r.getMessage() for r in handler.records)
"picture_description.enabled=true" in r.getMessage() for r in handler.records
def test_load_yaml_legacy_pictures_image_maps_to_disabled(tmp_path):
"""`processing.pictures: image` is the bytes-only mode in the old
enum; under always-store semantics it collapses to
`picture_description.enabled = false`."""
config_file = _write(
tmp_path,
"""
processing:
pictures: image
""",
) )
handler = _ListHandler()
loader_logger.addHandler(handler)
try:
data = load_yaml_config(config_file)
finally:
loader_logger.removeHandler(handler)
cfg = AppConfig.model_validate(data)
assert cfg.processing.conversion_options.picture_description.enabled is False
def test_load_yaml_legacy_generate_picture_images_maps_to_image(tmp_path): def test_load_yaml_legacy_pictures_none_maps_to_disabled(tmp_path):
"""`generate_picture_images=true` alone maps to `pictures: image`.""" """`processing.pictures: none` maps to `picture_description.enabled = false`."""
config_file = _write(
tmp_path,
"""
processing:
pictures: none
""",
)
data = load_yaml_config(config_file)
cfg = AppConfig.model_validate(data)
assert cfg.processing.conversion_options.picture_description.enabled is False
def test_load_yaml_legacy_generate_picture_images_warns_and_drops(tmp_path):
"""`generate_picture_images` is now a no-op (bytes always extracted);
it gets dropped with a deprecation warning."""
config_file = _write( config_file = _write(
tmp_path, tmp_path,
""" """
@ -294,15 +329,14 @@ processing:
finally: finally:
loader_logger.removeHandler(handler) loader_logger.removeHandler(handler)
cfg = AppConfig.model_validate(data) cfg = AppConfig.model_validate(data)
assert cfg.processing.pictures == "image" assert cfg.processing.conversion_options.picture_description.enabled is False
assert any( assert "generate_picture_images" not in data["processing"]["conversion_options"]
"generate_picture_images=true" in r.getMessage() for r in handler.records assert any("generate_picture_images" in r.getMessage() for r in handler.records)
)
def test_load_yaml_no_legacy_fields_keeps_default_none(tmp_path): def test_load_yaml_no_legacy_fields_keeps_default_disabled(tmp_path):
"""Empty processing block leaves the default `none` mode untouched and """Empty processing block leaves picture_description.enabled at the
does not warn.""" default (False) and does not warn."""
config_file = _write( config_file = _write(
tmp_path, tmp_path,
""" """
@ -317,29 +351,23 @@ processing:
finally: finally:
loader_logger.removeHandler(handler) loader_logger.removeHandler(handler)
cfg = AppConfig.model_validate(data) cfg = AppConfig.model_validate(data)
assert cfg.processing.pictures == "none" assert cfg.processing.conversion_options.picture_description.enabled is False
assert not handler.records assert not handler.records
def test_load_yaml_explicit_pictures_wins_over_legacy(tmp_path): def test_load_yaml_explicit_enabled_wins_over_legacy_pictures(tmp_path):
"""When the user has migrated to `pictures: ...` we keep their choice """If the user already set `picture_description.enabled` explicitly,
and silently drop legacy fields if both are present (e.g. from a a stale `processing.pictures` value does not override it."""
half-migrated config)."""
config_file = _write( config_file = _write(
tmp_path, tmp_path,
""" """
processing: processing:
pictures: image pictures: none
conversion_options: conversion_options:
generate_picture_images: false
picture_description: picture_description:
enabled: true enabled: true
""", """,
) )
data = load_yaml_config(config_file) data = load_yaml_config(config_file)
cfg = AppConfig.model_validate(data) cfg = AppConfig.model_validate(data)
assert cfg.processing.pictures == "image" assert cfg.processing.conversion_options.picture_description.enabled is True
# Legacy fields scrubbed so Pydantic validation doesn't trip on extras.
opts = data["processing"]["conversion_options"]
assert "generate_picture_images" not in opts
assert "enabled" not in opts.get("picture_description", {})

View file

@ -507,30 +507,23 @@ class TestDoclingLocalConverter:
def test_picture_description_config_defaults(self, config): def test_picture_description_config_defaults(self, config):
"""Test that picture description config has correct defaults.""" """Test that picture description config has correct defaults."""
assert config.processing.pictures == "none" pic_desc = config.processing.conversion_options.picture_description
assert ( assert pic_desc.enabled is False
config.processing.conversion_options.picture_description.model.provider assert pic_desc.model.provider == "ollama"
== "ollama" assert pic_desc.model.name == "ministral-3"
) assert pic_desc.timeout == 90
assert ( assert pic_desc.max_tokens == 200
config.processing.conversion_options.picture_description.model.name
== "ministral-3"
)
assert config.processing.conversion_options.picture_description.timeout == 90
assert (
config.processing.conversion_options.picture_description.max_tokens == 200
)
# Default prompt is in PromptsConfig # Default prompt is in PromptsConfig
assert "blind user" in config.prompts.picture_description assert "blind user" in config.prompts.picture_description
def test_picture_description_config_applied(self, config): def test_picture_description_config_applied(self, config):
"""Test that picture description config is applied to converter.""" """Test that picture description config is applied to converter."""
config.processing.pictures = "description" config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.timeout = 120 config.processing.conversion_options.picture_description.timeout = 120
converter = DoclingLocalConverter(config) converter = DoclingLocalConverter(config)
assert converter.config.processing.pictures == "description"
pic_desc = converter.config.processing.conversion_options.picture_description pic_desc = converter.config.processing.conversion_options.picture_description
assert pic_desc.enabled is True
assert pic_desc.timeout == 120 assert pic_desc.timeout == 120
@pytest.mark.asyncio @pytest.mark.asyncio
@ -542,7 +535,7 @@ class TestDoclingLocalConverter:
# Disable OCR (not needed for native PDF, avoids model downloads) # Disable OCR (not needed for native PDF, avoids model downloads)
config.processing.conversion_options.do_ocr = False config.processing.conversion_options.do_ocr = False
# Enable picture description with Ollama # Enable picture description with Ollama
config.processing.pictures = "description" config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = ( config.processing.conversion_options.picture_description.model.provider = (
"ollama" "ollama"
) )
@ -947,15 +940,11 @@ class TestDoclingServeConverterPictureDescription:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_picture_description_options_passed_to_api(self, config): async def test_picture_description_options_passed_to_api(self, config):
"""Test that picture description options are passed to docling-serve API. """Picture-description options reach the docling-serve API when the
VLM is enabled."""
``pictures="description"`` requires picture images for the VLM, which
routes the request through the ``target_type=zip`` path. The test
mocks the zip workflow.
"""
import json import json
config.processing.pictures = "description" config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = ( config.processing.conversion_options.picture_description.model.provider = (
"ollama" "ollama"
) )
@ -1066,7 +1055,7 @@ class TestDoclingServeConverterIntegration:
Note: Not using VCR because this test involves polling with changing task IDs. Note: Not using VCR because this test involves polling with changing task IDs.
""" """
pdf_path = Path("tests/data/doclaynet.pdf") pdf_path = Path("tests/data/doclaynet.pdf")
config.processing.pictures = "description" config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = ( config.processing.conversion_options.picture_description.model.provider = (
"ollama" "ollama"
) )

View file

@ -178,10 +178,7 @@ async def test_rechunk_preserves_picture_data(temp_db_path):
docling_doc = _docling_doc_with_picture() docling_doc = _docling_doc_with_picture()
config = AppConfig() async with HaikuRAG(temp_db_path, create=True) as rag:
config.processing.pictures = "image"
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
document = Document(content="x", uri="test://doc") document = Document(content="x", uri="test://doc")
document.set_docling(docling_doc) document.set_docling(docling_doc)
created = await _store_document_with_chunks(rag, document, [], docling_doc) created = await _store_document_with_chunks(rag, document, [], docling_doc)
@ -253,7 +250,6 @@ class _Deps:
async def test_search_tool_returns_multimodal_when_picture_present(): async def test_search_tool_returns_multimodal_when_picture_present():
"""The agent-facing search tool must wrap text + BinaryContent in ToolReturn """The agent-facing search tool must wrap text + BinaryContent in ToolReturn
whenever a result carries picture image_data AND the QA model is vision-capable.""" whenever a result carries picture image_data AND the QA model is vision-capable."""
from haiku.rag.config import AppConfig
picture_result = SearchResult( picture_result = SearchResult(
content="A diagram of the layout", content="A diagram of the layout",
@ -352,7 +348,6 @@ async def test_chunk_interleaves_picture_in_structural_order(monkeypatch):
sit where they appear in the document, not appended at the end. sit where they appear in the document, not appended at the end.
""" """
from haiku.rag.client.processing import chunk from haiku.rag.client.processing import chunk
from haiku.rag.config import AppConfig
from haiku.rag.embeddings import EmbedderWrapper from haiku.rag.embeddings import EmbedderWrapper
from haiku.rag.store.models.chunk import Chunk from haiku.rag.store.models.chunk import Chunk
@ -513,14 +508,13 @@ async def test_ingest_emits_picture_chunks_with_multimodal_embedder(
docling_doc = _docling_doc_with_picture() docling_doc = _docling_doc_with_picture()
from haiku.rag.config import AppConfig, EmbeddingModelConfig, EmbeddingsConfig from haiku.rag.config import EmbeddingModelConfig, EmbeddingsConfig
config = AppConfig( config = AppConfig(
embeddings=EmbeddingsConfig( embeddings=EmbeddingsConfig(
model=EmbeddingModelConfig(provider="ollama", name="stub", vector_dim=4) model=EmbeddingModelConfig(provider="ollama", name="stub", vector_dim=4)
) )
) )
config.processing.pictures = "image"
async with HaikuRAG(temp_db_path, config=config, create=True) as rag: async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
chunks = await rag.chunk(docling_doc) chunks = await rag.chunk(docling_doc)
@ -546,7 +540,6 @@ async def test_search_tool_skips_binary_content_when_qa_model_is_text_only():
is text-only (``qa.model.vision = False``, the default). Sending image is text-only (``qa.model.vision = False``, the default). Sending image
parts to a text-only model would cause it to hallucinate confidently parts to a text-only model would cause it to hallucinate confidently
Ollama silently accepts the bytes and the model guesses.""" Ollama silently accepts the bytes and the model guesses."""
from haiku.rag.config import AppConfig
picture_result = SearchResult( picture_result = SearchResult(
content="A diagram of the layout", content="A diagram of the layout",