collapse pictures enum to picture_description.enabled boolean
This commit is contained in:
Yiorgis Gozadinos
parent
e88321767d
commit
2038d43435
13 changed files with 157 additions and 192 deletions
|
|
@ -8,12 +8,7 @@
|
|||
- **`vision: bool` flag on `ModelConfig`.** Tracks whether a configured language model can interpret images. Default `False`. The agent's `search` tool only attaches picture bytes (as `BinaryContent`) to the `ToolReturn` when `qa.model.vision = True`. Without the gate, sending image content to a text-only model behaves inconsistently across providers — Ollama silently accepts and the model hallucinates a confident wrong answer; OpenAI returns 400; others vary. Capability detection from a probe or a model-name whitelist is unreliable, so `vision` is an explicit user-set capability declaration. Set it to `True` for vision-capable QA models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, ...).
|
||||
- **Synthetic picture chunks at ingest under multimodal embedders.** `build_picture_chunks` (in `client/processing.py`) walks a `DoclingDocument`'s `pictures` and emits one synthetic `Chunk` per `PictureItem` with available bytes. Bytes ride on a `Chunk._picture_data` PrivateAttr (not serialized) so `embed_chunks` can route them through `embed_images` while text chunks keep going through `embed_documents`. Wired into the three ingest paths (`create_document`, `_create_document_from_file`, `_create_or_update_document_from_url`, `_update_document_with_chunks`, and `_rebuild_rechunk`) — guarded by `embedder.supports_images` so text-only configurations are unaffected. Snapshot/merge with `existing_picture_data` keeps picture chunks alive across rebuild round-trips. Picture chunks land in the same `chunks` table with the same vector dim as text chunks, so cross-modal search reuses the existing hybrid+RRF pipeline.
|
||||
- **Multimodal embedder support (`provider="vllm"`).** `EmbedderWrapper` gains `supports_images: bool` and `embed_image_query`. The `vllm` provider talks HTTP to a vLLM server's OpenAI-compatible `/v1/embeddings` endpoint — text inputs use the standard `input` field for true server-side batching; image inputs use vLLM's `messages` superset with `image_url` content parts carrying base64 data URIs. Works with `Qwen/Qwen3-VL-Embedding-8B` and `jinaai/jina-embeddings-v4`. No Python ML deps added — uses `httpx`. Text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) report `supports_images=False` and raise a clear error if image methods are called.
|
||||
- **Picture-handling mode enum.** `processing.pictures: "none" | "description" | "image"` (default `"none"`) replaces the legacy `generate_picture_images` + `picture_description.enabled` pair.
|
||||
- `none` — no picture bytes, no VLM, just structural picture rows.
|
||||
- `description` — VLM runs at ingest, descriptions woven into chunk text, bytes also retained on `document_items.picture_data` so vision QA can be turned on later without reingesting.
|
||||
- `image` — bytes retained, no VLM.
|
||||
- YAML loader translates legacy configs and emits a deprecation warning.
|
||||
- Rebuild and update under `pictures="none"` skip the picture-bytes snapshot/merge so storage is reclaimed.
|
||||
- **Picture bytes are always stored.** Every ingest captures embedded picture bytes into `document_items.picture_data`; rebuilds and updates always preserve them. The single ingest-time decision is whether to run a VLM at ingest, exposed via `processing.conversion_options.picture_description.enabled: bool` (default `False`). When enabled, descriptions are woven into chunk text; when disabled, chunks contain only their natural text. No reingest is required to flip the VLM on or off — `haiku-rag rebuild --rechunk` recomputes chunk text against the new setting. The legacy `processing.pictures: "none" | "description" | "image"` enum and `processing.conversion_options.generate_picture_images` flag are auto-translated by the YAML loader with a one-time deprecation warning per field.
|
||||
- **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`.
|
||||
- New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`.
|
||||
- Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory.
|
||||
|
|
|
|||
|
|
@ -28,9 +28,6 @@ processing:
|
|||
name: gpt-oss
|
||||
enable_thinking: false
|
||||
|
||||
# Picture handling (none / description / image)
|
||||
pictures: none # See "Picture Handling" below
|
||||
|
||||
# Conversion options (works with both local and remote converters)
|
||||
conversion_options:
|
||||
# OCR settings
|
||||
|
|
@ -48,8 +45,9 @@ processing:
|
|||
images_scale: 2.0 # Image scale factor
|
||||
generate_page_images: true # Include rendered page images (for visualize_chunk)
|
||||
|
||||
# VLM picture description settings (only effective when pictures: description)
|
||||
# VLM picture description (off by default; see "Picture Handling" below)
|
||||
picture_description:
|
||||
enabled: false
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
|
|
@ -106,27 +104,15 @@ conversion_options:
|
|||
- **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
|
||||
- **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size.
|
||||
|
||||
Embedded picture extraction is controlled by `processing.pictures` (see [Picture Handling](#picture-handling) below), not by an image-settings flag.
|
||||
|
||||
#### Picture Handling
|
||||
|
||||
`processing.pictures` is a single enum that decides how embedded picture images (figures, diagrams) are handled at ingest. Three modes:
|
||||
|
||||
| Mode | VLM at ingest | `picture_data` populated | Chunk text contains description |
|
||||
|---|---|---|---|
|
||||
| `none` (default) | no | no | no |
|
||||
| `description` | yes | yes | yes |
|
||||
| `image` | no | yes | no |
|
||||
|
||||
- **`none`**: docling skips picture-image generation. `label="picture"` rows still appear in the items table for structural metadata, but they carry no bytes and no description. Cheapest mode; non-vision QA is unaffected.
|
||||
- **`description`**: docling generates picture images, the configured VLM produces a description woven into chunk text, and the bytes are also retained in `document_items.picture_data`. Picture-text is searchable via FTS, vision-capable QA models also receive the bytes via the agent's search tool. Bytes are kept (not just thrown away after the VLM runs) so a vision-only QA strategy can be turned on later without reingesting.
|
||||
- **`image`**: docling generates picture images and stores them in `document_items.picture_data` without running the VLM. Vision-capable QA models reason directly about the figures; non-vision QA only sees the picture's caption/surrounding text.
|
||||
Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest time to generate textual descriptions, set via `picture_description.enabled`:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
pictures: description # none / description / image
|
||||
conversion_options:
|
||||
picture_description: # only effective when pictures: description
|
||||
picture_description:
|
||||
enabled: true # default false; runs the VLM at ingest
|
||||
model:
|
||||
provider: ollama # ollama, openai, or custom
|
||||
name: ministral-3 # VLM model name
|
||||
|
|
@ -135,56 +121,55 @@ processing:
|
|||
max_tokens: 200 # Maximum tokens in response
|
||||
```
|
||||
|
||||
**Switching modes on an existing database.** No reingest is required if you only need to change between `description` and `image` — the bytes are already there. Run `haiku-rag rebuild --rechunk` after the config change so the chunk-text composition reflects the new mode. Switching *down* to `none` clears `picture_data` on rebuild, reclaiming storage.
|
||||
When `enabled: false` (default), the VLM doesn't run; chunks contain only their natural text (captions, surrounding paragraphs). When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS.
|
||||
|
||||
#### Pictures × embedder × QA model: how the pieces compose
|
||||
**Switching the VLM on or off on an existing database.** Picture bytes are already stored, so no reingest is required. Run `haiku-rag rebuild --rechunk` after flipping `enabled` so the chunk-text composition reflects the new setting.
|
||||
|
||||
Three orthogonal settings drive what gets stored, what gets retrieved, and what reaches the QA model. Each setting answers one question:
|
||||
#### Picture descriptions × embedder × QA model: how the pieces compose
|
||||
|
||||
Three settings drive what gets stored, what gets retrieved, and what reaches the QA model:
|
||||
|
||||
| Setting | Question it answers | Values |
|
||||
|---|---|---|
|
||||
| `processing.pictures` | What gets captured at ingest? | `none` / `description` / `image` |
|
||||
| `picture_description.enabled` | Should a VLM weave descriptions into chunk text at ingest? | `false` (default) / `true` |
|
||||
| `embeddings.model.provider` | Can the embedder index image content? | text-only providers (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
|
||||
| `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` |
|
||||
|
||||
**What gets stored** for each `pictures` × embedder combination:
|
||||
Picture bytes are always stored, regardless of these settings.
|
||||
|
||||
| `pictures` | Embedder | Text chunks contain… | `document_items.picture_data` | Synthetic picture chunks |
|
||||
|---|---|---|---|---|
|
||||
| `none` | text-only or multimodal | regular text only | empty | none |
|
||||
| `description` | text-only | text + VLM descriptions | populated (kept for later use) | none |
|
||||
| `description` | multimodal | text + VLM descriptions | populated | one per picture, content = description, vector = image embedding |
|
||||
| `image` | text-only | text only (caption/surrounding) | populated (kept for later use) | none |
|
||||
| `image` | multimodal | text only | populated | one per picture, content = caption/empty, vector = image embedding |
|
||||
**What gets stored** for each `picture_description.enabled` × embedder combination:
|
||||
|
||||
| `enabled` | Embedder | Text chunks contain… | Synthetic picture chunks |
|
||||
|---|---|---|---|
|
||||
| `false` | text-only | text only (caption/surrounding) | none |
|
||||
| `false` | multimodal | text only | one per picture, content = caption/empty, vector = image embedding |
|
||||
| `true` | text-only | text + VLM descriptions | none |
|
||||
| `true` | multimodal | text + VLM descriptions | one per picture, content = description, vector = image embedding |
|
||||
|
||||
**What QA receives** at search time, given stored state and `qa.model.vision`:
|
||||
|
||||
| `pictures` at ingest | `qa.model.vision` | QA receives |
|
||||
|---|---|---|
|
||||
| `none` | either | text chunks only |
|
||||
| `description` | `false` | text chunks (descriptions in chunk text answer figure questions in prose) |
|
||||
| `description` | `true` | text chunks + raw picture bytes; vision model uses both signals |
|
||||
| `image` | `false` | text chunks only (caption + surrounding text); model has no figure content to draw on |
|
||||
| `image` | `true` | text chunks + raw picture bytes; vision model reads the figures directly |
|
||||
| `qa.model.vision` | QA receives |
|
||||
|---|---|
|
||||
| `false` | text chunks only (descriptions in chunk text answer figure questions in prose when `picture_description.enabled` was true) |
|
||||
| `true` | text chunks + raw picture bytes; vision model reads the figures directly |
|
||||
|
||||
A few invariants worth knowing:
|
||||
|
||||
- **`qa.model.vision` is independent of ingestion.** It only controls whether the agent's `search` tool attaches picture bytes to its `ToolReturn`. A text-only QA model with `vision: true` won't suddenly understand images — it will silently accept the bytes and confabulate. Default `false` is the safe choice.
|
||||
- **`description` mode preserves the bytes**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself.
|
||||
- **The bytes are always there**, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change `qa.model.vision` and optionally `qa.model` itself.
|
||||
- **Cross-modal search** (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks.
|
||||
- **`image` mode + text-only embedder** is rarely the right choice — pictures are stored but neither searchable as image vectors nor described in text. Picture bytes are then only reachable via expand-context's section bounds when an adjacent text chunk matches.
|
||||
|
||||
**Recommended combinations** by use case:
|
||||
|
||||
| Use case | `pictures` | Embedder | `qa.model.vision` |
|
||||
| Use case | `picture_description.enabled` | Embedder | `qa.model.vision` |
|
||||
|---|---|---|---|
|
||||
| Pure text RAG, no figures | `none` | text-only | `false` |
|
||||
| Text RAG, figures answered through descriptions | `description` | text-only | `false` |
|
||||
| Vision QA on figure-rich docs (no cross-modal search) | `description` | text-only | `true` |
|
||||
| Cross-modal search + vision QA (the full multimodal stack) | `description` or `image` | multimodal | `true` |
|
||||
| Cross-modal search, text QA only | `description` | multimodal | `false` |
|
||||
| Pure text RAG, no figures | `false` | text-only | `false` |
|
||||
| Text RAG, figures answered through descriptions | `true` | text-only | `false` |
|
||||
| Vision QA on figure-rich docs (no cross-modal search) | `true` or `false` | text-only | `true` |
|
||||
| Cross-modal search + vision QA (the full multimodal stack) | `true` or `false` | multimodal | `true` |
|
||||
| Cross-modal search, text QA only | `true` | multimodal | `false` |
|
||||
|
||||
**`picture_description.model` configuration** (used only under `pictures: description`):
|
||||
**`picture_description.model` configuration** (used only when `picture_description.enabled: true`):
|
||||
|
||||
- **model**: Standard model configuration
|
||||
- `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
|
||||
|
|
@ -214,9 +199,9 @@ prompts:
|
|||
|
||||
```yaml
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
|
|
@ -233,9 +218,9 @@ ollama serve
|
|||
|
||||
```yaml
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: openai # Use OpenAI-compatible API format
|
||||
name: granite-vision
|
||||
|
|
@ -260,9 +245,9 @@ When using `converter: docling-serve`, the VLM calls are made by the docling-ser
|
|||
|
||||
```yaml
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ prompts:
|
|||
Be concise and factual.
|
||||
```
|
||||
|
||||
The prompt is used when `processing.pictures` is set to `description`. See [Picture Handling](processing.md#picture-handling) for full configuration.
|
||||
The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Handling](processing.md#picture-handling) for full configuration.
|
||||
|
||||
## Programmatic Configuration
|
||||
|
||||
|
|
|
|||
|
|
@ -105,10 +105,7 @@ async def download_models(
|
|||
if config.reranking.model and config.reranking.model.provider == "ollama":
|
||||
required_models.add(config.reranking.model.name)
|
||||
pic_desc = config.processing.conversion_options.picture_description
|
||||
if (
|
||||
config.processing.pictures == "description"
|
||||
and pic_desc.model.provider == "ollama"
|
||||
):
|
||||
if pic_desc.enabled and pic_desc.model.provider == "ollama":
|
||||
required_models.add(pic_desc.model.name)
|
||||
if (
|
||||
config.processing.auto_title
|
||||
|
|
|
|||
|
|
@ -53,56 +53,54 @@ def load_yaml_config(path: Path) -> dict:
|
|||
|
||||
|
||||
def _translate_legacy_picture_fields(data: dict) -> None:
|
||||
"""Map legacy picture knobs onto ``processing.pictures``.
|
||||
"""Map legacy picture-handling knobs onto
|
||||
``processing.conversion_options.picture_description.enabled``.
|
||||
|
||||
Older configs expressed the same intent through two booleans on
|
||||
``conversion_options``: ``generate_picture_images`` and
|
||||
``picture_description.enabled``. Translation, in priority order:
|
||||
Two earlier shapes need translating:
|
||||
|
||||
- ``picture_description.enabled = true`` (regardless of the image flag)
|
||||
→ ``pictures = "description"``.
|
||||
- ``generate_picture_images = true`` (and no description) → ``"image"``.
|
||||
- both false / missing → no translation; default ``"none"`` applies.
|
||||
- ``processing.pictures: "description"`` →
|
||||
``picture_description.enabled = true``. The other values
|
||||
(``"none"``, ``"image"``) collapse to ``false`` since the only
|
||||
remaining decision is whether the VLM runs; picture bytes are
|
||||
always stored.
|
||||
- ``processing.conversion_options.generate_picture_images: <any>`` is a
|
||||
no-op now (docling always extracts picture bytes) and is dropped
|
||||
with a one-time warning.
|
||||
|
||||
If ``pictures`` is already set on the loaded YAML it wins. Mutates
|
||||
``data`` in-place and emits one warning per legacy field encountered.
|
||||
If ``picture_description.enabled`` is already explicitly set on the
|
||||
loaded YAML, it wins. Mutates ``data`` in-place and emits one warning
|
||||
per legacy field encountered.
|
||||
"""
|
||||
processing = data.get("processing")
|
||||
if not isinstance(processing, dict):
|
||||
return
|
||||
|
||||
if "pictures" in processing:
|
||||
# User has migrated; legacy fields may still be present but should not
|
||||
# override the explicit choice. Drop them silently to avoid confusion.
|
||||
opts = processing.get("conversion_options")
|
||||
if isinstance(opts, dict):
|
||||
opts.pop("generate_picture_images", None)
|
||||
pic = opts.get("picture_description")
|
||||
if isinstance(pic, dict):
|
||||
pic.pop("enabled", None)
|
||||
return
|
||||
legacy_pictures = processing.pop("pictures", None)
|
||||
if legacy_pictures is not None:
|
||||
opts = processing.setdefault("conversion_options", {})
|
||||
if not isinstance(opts, dict):
|
||||
opts = {}
|
||||
processing["conversion_options"] = opts
|
||||
pic = opts.setdefault("picture_description", {})
|
||||
if not isinstance(pic, dict):
|
||||
pic = {}
|
||||
opts["picture_description"] = pic
|
||||
if "enabled" not in pic:
|
||||
pic["enabled"] = legacy_pictures == "description"
|
||||
logger.warning(
|
||||
"Config: 'processing.pictures' is deprecated; use "
|
||||
"'processing.conversion_options.picture_description.enabled' "
|
||||
"instead. Picture bytes are now always stored. Please update "
|
||||
"your haiku.rag.yaml."
|
||||
)
|
||||
|
||||
opts = processing.get("conversion_options")
|
||||
if not isinstance(opts, dict):
|
||||
return
|
||||
|
||||
pic = opts.get("picture_description") if isinstance(opts, dict) else None
|
||||
legacy_describe = pic.pop("enabled", None) if isinstance(pic, dict) else None
|
||||
legacy_image = opts.pop("generate_picture_images", None)
|
||||
|
||||
if legacy_describe:
|
||||
processing["pictures"] = "description"
|
||||
if isinstance(opts, dict) and "generate_picture_images" in opts:
|
||||
opts.pop("generate_picture_images", None)
|
||||
logger.warning(
|
||||
"Config: 'processing.conversion_options.picture_description.enabled=true' is "
|
||||
"deprecated; mapped to 'processing.pictures: description'. Please update your "
|
||||
"haiku.rag.yaml."
|
||||
)
|
||||
elif legacy_image:
|
||||
processing["pictures"] = "image"
|
||||
logger.warning(
|
||||
"Config: 'processing.conversion_options.generate_picture_images=true' is "
|
||||
"deprecated; mapped to 'processing.pictures: image'. Please update your "
|
||||
"haiku.rag.yaml."
|
||||
"Config: 'processing.conversion_options.generate_picture_images' "
|
||||
"is deprecated and ignored; picture bytes are always extracted. "
|
||||
"Please update your haiku.rag.yaml."
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -112,12 +112,14 @@ class AnalysisConfig(BaseModel):
|
|||
|
||||
|
||||
class PictureDescriptionConfig(BaseModel):
|
||||
"""Settings for the VLM that runs at ingest under ``pictures="description"``.
|
||||
"""Whether (and how) to run a VLM over each picture at ingest.
|
||||
|
||||
Whether the VLM runs at all is decided by ``ProcessingConfig.pictures``;
|
||||
these fields only describe *how* it runs once enabled.
|
||||
When ``enabled`` is True, picture descriptions are generated by the
|
||||
configured VLM and woven into chunk text. Picture bytes are stored
|
||||
regardless — this flag only controls the description pass.
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
model: ModelConfig = Field(
|
||||
default_factory=lambda: ModelConfig(
|
||||
provider="ollama",
|
||||
|
|
@ -149,15 +151,11 @@ class ConversionOptions(BaseModel):
|
|||
images_scale: float = 2.0
|
||||
generate_page_images: bool = True
|
||||
|
||||
# VLM picture description (only effective when ProcessingConfig.pictures == "description")
|
||||
picture_description: PictureDescriptionConfig = Field(
|
||||
default_factory=PictureDescriptionConfig
|
||||
)
|
||||
|
||||
|
||||
PicturesMode = Literal["none", "description", "image"]
|
||||
|
||||
|
||||
class ProcessingConfig(BaseModel):
|
||||
chunk_size: int = 256
|
||||
converter: str = "docling-local"
|
||||
|
|
@ -167,18 +165,6 @@ class ProcessingConfig(BaseModel):
|
|||
chunking_merge_peers: bool = True
|
||||
chunking_use_markdown_tables: bool = False
|
||||
conversion_options: ConversionOptions = Field(default_factory=ConversionOptions)
|
||||
pictures: PicturesMode = "none"
|
||||
"""How embedded pictures are handled at ingest.
|
||||
|
||||
- ``"none"``: docling skips picture-image generation; structural
|
||||
``label="picture"`` rows still exist but carry no bytes or description.
|
||||
- ``"description"``: docling generates picture images, the VLM produces
|
||||
text descriptions woven into chunk text, and the bytes are also
|
||||
retained in ``document_items.picture_data`` so a vision-capable QA
|
||||
model can be turned on later without reingesting.
|
||||
- ``"image"``: docling generates picture images and stores them in
|
||||
``document_items.picture_data``; no VLM runs at ingest.
|
||||
"""
|
||||
auto_title: bool = False
|
||||
title_model: ModelConfig = Field(
|
||||
default_factory=lambda: ModelConfig(
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ class DoclingLocalConverter(DocumentConverter):
|
|||
|
||||
opts = self.config.processing.conversion_options
|
||||
pic_desc = opts.picture_description
|
||||
runs_vlm = self.config.processing.pictures == "description"
|
||||
runs_vlm = pic_desc.enabled
|
||||
|
||||
pipeline_options = PdfPipelineOptions(
|
||||
do_ocr=opts.do_ocr,
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@ class DoclingServeConverter(DocumentConverter):
|
|||
"""
|
||||
opts = self.config.processing.conversion_options
|
||||
pic_desc = opts.picture_description
|
||||
runs_vlm = self.config.processing.pictures == "description"
|
||||
runs_vlm = pic_desc.enabled
|
||||
|
||||
data: dict[str, str | list[str]] = {
|
||||
"to_formats": "json",
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@ from haiku.rag.client.documents import (
|
|||
_store_document_with_chunks,
|
||||
_update_document_with_chunks,
|
||||
)
|
||||
from haiku.rag.config import AppConfig
|
||||
from haiku.rag.store.engine import Store
|
||||
from haiku.rag.store.models.document_item import (
|
||||
DocumentItem,
|
||||
|
|
@ -647,11 +646,7 @@ class TestPictureDataPreservedThroughRoundTrip:
|
|||
|
||||
docling_doc = _docling_doc_with_picture()
|
||||
|
||||
config = AppConfig()
|
||||
# Preservation only kicks in under modes that retain picture bytes.
|
||||
config.processing.pictures = "image"
|
||||
|
||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||
async with HaikuRAG(temp_db_path, create=True) as rag:
|
||||
document = Document(content="Hello world", uri="test://doc")
|
||||
document.set_docling(docling_doc)
|
||||
created = await _store_document_with_chunks(rag, document, [], docling_doc)
|
||||
|
|
|
|||
|
|
@ -676,7 +676,6 @@ async def test_serve_chunker_accepts_picture_laden_docling():
|
|||
from haiku.rag.converters.docling_serve import DoclingServeConverter
|
||||
|
||||
config = AppConfig()
|
||||
config.processing.pictures = "image"
|
||||
config.processing.conversion_options.do_ocr = False
|
||||
config.processing.chunk_size = 256
|
||||
config.processing.chunker_type = "hybrid"
|
||||
|
|
|
|||
|
|
@ -239,8 +239,10 @@ def test_init_config_creates_valid_yaml(tmp_path):
|
|||
assert config.environment == "production"
|
||||
|
||||
|
||||
# Legacy picture-handling field translation (`generate_picture_images` and
|
||||
# `picture_description.enabled` → `processing.pictures` enum)
|
||||
# Legacy picture-handling field translation:
|
||||
# processing.pictures: "description" -> picture_description.enabled = true
|
||||
# processing.pictures: "none" | "image" -> picture_description.enabled = false
|
||||
# processing.conversion_options.generate_picture_images: <any> -> ignored
|
||||
|
||||
|
||||
def _write(tmp_path, body: str):
|
||||
|
|
@ -249,17 +251,16 @@ def _write(tmp_path, body: str):
|
|||
return p
|
||||
|
||||
|
||||
def test_load_yaml_legacy_picture_description_maps_to_description(tmp_path):
|
||||
"""`picture_description.enabled=true` (with or without the image flag)
|
||||
maps to `processing.pictures: description`."""
|
||||
def test_load_yaml_legacy_pictures_description_maps_to_enabled(tmp_path):
|
||||
"""`processing.pictures: description` maps to
|
||||
`picture_description.enabled = true` and warns."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
generate_picture_images: false
|
||||
picture_description:
|
||||
enabled: true
|
||||
timeout: 120
|
||||
""",
|
||||
)
|
||||
|
|
@ -270,15 +271,49 @@ processing:
|
|||
finally:
|
||||
loader_logger.removeHandler(handler)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "description"
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is True
|
||||
assert cfg.processing.conversion_options.picture_description.timeout == 120
|
||||
assert any(
|
||||
"picture_description.enabled=true" in r.getMessage() for r in handler.records
|
||||
assert any("processing.pictures" in r.getMessage() for r in handler.records)
|
||||
|
||||
|
||||
def test_load_yaml_legacy_pictures_image_maps_to_disabled(tmp_path):
|
||||
"""`processing.pictures: image` is the bytes-only mode in the old
|
||||
enum; under always-store semantics it collapses to
|
||||
`picture_description.enabled = false`."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
pictures: image
|
||||
""",
|
||||
)
|
||||
handler = _ListHandler()
|
||||
loader_logger.addHandler(handler)
|
||||
try:
|
||||
data = load_yaml_config(config_file)
|
||||
finally:
|
||||
loader_logger.removeHandler(handler)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is False
|
||||
|
||||
|
||||
def test_load_yaml_legacy_generate_picture_images_maps_to_image(tmp_path):
|
||||
"""`generate_picture_images=true` alone maps to `pictures: image`."""
|
||||
def test_load_yaml_legacy_pictures_none_maps_to_disabled(tmp_path):
|
||||
"""`processing.pictures: none` maps to `picture_description.enabled = false`."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
pictures: none
|
||||
""",
|
||||
)
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is False
|
||||
|
||||
|
||||
def test_load_yaml_legacy_generate_picture_images_warns_and_drops(tmp_path):
|
||||
"""`generate_picture_images` is now a no-op (bytes always extracted);
|
||||
it gets dropped with a deprecation warning."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
|
|
@ -294,15 +329,14 @@ processing:
|
|||
finally:
|
||||
loader_logger.removeHandler(handler)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "image"
|
||||
assert any(
|
||||
"generate_picture_images=true" in r.getMessage() for r in handler.records
|
||||
)
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is False
|
||||
assert "generate_picture_images" not in data["processing"]["conversion_options"]
|
||||
assert any("generate_picture_images" in r.getMessage() for r in handler.records)
|
||||
|
||||
|
||||
def test_load_yaml_no_legacy_fields_keeps_default_none(tmp_path):
|
||||
"""Empty processing block leaves the default `none` mode untouched and
|
||||
does not warn."""
|
||||
def test_load_yaml_no_legacy_fields_keeps_default_disabled(tmp_path):
|
||||
"""Empty processing block leaves picture_description.enabled at the
|
||||
default (False) and does not warn."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
|
|
@ -317,29 +351,23 @@ processing:
|
|||
finally:
|
||||
loader_logger.removeHandler(handler)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "none"
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is False
|
||||
assert not handler.records
|
||||
|
||||
|
||||
def test_load_yaml_explicit_pictures_wins_over_legacy(tmp_path):
|
||||
"""When the user has migrated to `pictures: ...` we keep their choice
|
||||
and silently drop legacy fields if both are present (e.g. from a
|
||||
half-migrated config)."""
|
||||
def test_load_yaml_explicit_enabled_wins_over_legacy_pictures(tmp_path):
|
||||
"""If the user already set `picture_description.enabled` explicitly,
|
||||
a stale `processing.pictures` value does not override it."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
pictures: image
|
||||
pictures: none
|
||||
conversion_options:
|
||||
generate_picture_images: false
|
||||
picture_description:
|
||||
enabled: true
|
||||
""",
|
||||
)
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "image"
|
||||
# Legacy fields scrubbed so Pydantic validation doesn't trip on extras.
|
||||
opts = data["processing"]["conversion_options"]
|
||||
assert "generate_picture_images" not in opts
|
||||
assert "enabled" not in opts.get("picture_description", {})
|
||||
assert cfg.processing.conversion_options.picture_description.enabled is True
|
||||
|
|
|
|||
|
|
@ -507,30 +507,23 @@ class TestDoclingLocalConverter:
|
|||
|
||||
def test_picture_description_config_defaults(self, config):
|
||||
"""Test that picture description config has correct defaults."""
|
||||
assert config.processing.pictures == "none"
|
||||
assert (
|
||||
config.processing.conversion_options.picture_description.model.provider
|
||||
== "ollama"
|
||||
)
|
||||
assert (
|
||||
config.processing.conversion_options.picture_description.model.name
|
||||
== "ministral-3"
|
||||
)
|
||||
assert config.processing.conversion_options.picture_description.timeout == 90
|
||||
assert (
|
||||
config.processing.conversion_options.picture_description.max_tokens == 200
|
||||
)
|
||||
pic_desc = config.processing.conversion_options.picture_description
|
||||
assert pic_desc.enabled is False
|
||||
assert pic_desc.model.provider == "ollama"
|
||||
assert pic_desc.model.name == "ministral-3"
|
||||
assert pic_desc.timeout == 90
|
||||
assert pic_desc.max_tokens == 200
|
||||
# Default prompt is in PromptsConfig
|
||||
assert "blind user" in config.prompts.picture_description
|
||||
|
||||
def test_picture_description_config_applied(self, config):
|
||||
"""Test that picture description config is applied to converter."""
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.conversion_options.picture_description.timeout = 120
|
||||
converter = DoclingLocalConverter(config)
|
||||
|
||||
assert converter.config.processing.pictures == "description"
|
||||
pic_desc = converter.config.processing.conversion_options.picture_description
|
||||
assert pic_desc.enabled is True
|
||||
assert pic_desc.timeout == 120
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -542,7 +535,7 @@ class TestDoclingLocalConverter:
|
|||
# Disable OCR (not needed for native PDF, avoids model downloads)
|
||||
config.processing.conversion_options.do_ocr = False
|
||||
# Enable picture description with Ollama
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
@ -947,15 +940,11 @@ class TestDoclingServeConverterPictureDescription:
|
|||
|
||||
@pytest.mark.asyncio
|
||||
async def test_picture_description_options_passed_to_api(self, config):
|
||||
"""Test that picture description options are passed to docling-serve API.
|
||||
|
||||
``pictures="description"`` requires picture images for the VLM, which
|
||||
routes the request through the ``target_type=zip`` path. The test
|
||||
mocks the zip workflow.
|
||||
"""
|
||||
"""Picture-description options reach the docling-serve API when the
|
||||
VLM is enabled."""
|
||||
import json
|
||||
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
@ -1066,7 +1055,7 @@ class TestDoclingServeConverterIntegration:
|
|||
Note: Not using VCR because this test involves polling with changing task IDs.
|
||||
"""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -178,10 +178,7 @@ async def test_rechunk_preserves_picture_data(temp_db_path):
|
|||
|
||||
docling_doc = _docling_doc_with_picture()
|
||||
|
||||
config = AppConfig()
|
||||
config.processing.pictures = "image"
|
||||
|
||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||
async with HaikuRAG(temp_db_path, create=True) as rag:
|
||||
document = Document(content="x", uri="test://doc")
|
||||
document.set_docling(docling_doc)
|
||||
created = await _store_document_with_chunks(rag, document, [], docling_doc)
|
||||
|
|
@ -253,7 +250,6 @@ class _Deps:
|
|||
async def test_search_tool_returns_multimodal_when_picture_present():
|
||||
"""The agent-facing search tool must wrap text + BinaryContent in ToolReturn
|
||||
whenever a result carries picture image_data AND the QA model is vision-capable."""
|
||||
from haiku.rag.config import AppConfig
|
||||
|
||||
picture_result = SearchResult(
|
||||
content="A diagram of the layout",
|
||||
|
|
@ -352,7 +348,6 @@ async def test_chunk_interleaves_picture_in_structural_order(monkeypatch):
|
|||
sit where they appear in the document, not appended at the end.
|
||||
"""
|
||||
from haiku.rag.client.processing import chunk
|
||||
from haiku.rag.config import AppConfig
|
||||
from haiku.rag.embeddings import EmbedderWrapper
|
||||
from haiku.rag.store.models.chunk import Chunk
|
||||
|
||||
|
|
@ -513,14 +508,13 @@ async def test_ingest_emits_picture_chunks_with_multimodal_embedder(
|
|||
|
||||
docling_doc = _docling_doc_with_picture()
|
||||
|
||||
from haiku.rag.config import AppConfig, EmbeddingModelConfig, EmbeddingsConfig
|
||||
from haiku.rag.config import EmbeddingModelConfig, EmbeddingsConfig
|
||||
|
||||
config = AppConfig(
|
||||
embeddings=EmbeddingsConfig(
|
||||
model=EmbeddingModelConfig(provider="ollama", name="stub", vector_dim=4)
|
||||
)
|
||||
)
|
||||
config.processing.pictures = "image"
|
||||
|
||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||
chunks = await rag.chunk(docling_doc)
|
||||
|
|
@ -546,7 +540,6 @@ async def test_search_tool_skips_binary_content_when_qa_model_is_text_only():
|
|||
is text-only (``qa.model.vision = False``, the default). Sending image
|
||||
parts to a text-only model would cause it to hallucinate confidently —
|
||||
Ollama silently accepts the bytes and the model guesses."""
|
||||
from haiku.rag.config import AppConfig
|
||||
|
||||
picture_result = SearchResult(
|
||||
content="A diagram of the layout",
|
||||
|
|
|
|||
Loading…
Reference in a new issue