add processing.pictures enum (none|description|image), replacing the implicit pair of generate_picture_images+picture_description.enabled flags
This commit is contained in:
Yiorgis Gozadinos
parent
b01c649684
commit
7831057339
14 changed files with 316 additions and 81 deletions
|
|
@ -3,6 +3,7 @@
|
|||
|
||||
### Added
|
||||
|
||||
- **Picture-handling mode enum.** `processing.pictures: "none" | "description" | "image"` (default `"none"`) replaces the previous two-flag dance (`generate_picture_images` + `picture_description.enabled`) where flipping picture descriptions on silently forced docling to also generate picture images, leaving raw bytes inside the compressed `docling_document` blob with no user-facing knob to control it. The new enum makes intent explicit: `description` runs the VLM at ingest *and* preserves bytes (so a vision-capable QA model can be turned on later without reingesting), `image` preserves bytes without running the VLM, `none` is bytes-free. The legacy fields are removed from the schema (`ConversionOptions.generate_picture_images` and `PictureDescriptionConfig.enabled`); the YAML loader silently translates pre-existing configs to the new enum and emits a one-line deprecation warning so existing `haiku.rag.yaml` files keep working until the user updates them. No data migration is required — the 0.45.0 backfill already populates `picture_data` for every doc that had bytes inline. Rebuild and update flows now skip the picture-bytes snapshot/merge under `pictures="none"`, so users who downgrade to `none` and rebuild reclaim storage.
|
||||
- **Storage column for embedded picture bytes.** `DocumentItemRecord` gains a `picture_data: bytes | None` column (Arrow `large_binary`) to hold per-`PictureItem` image bytes addressable by `(document_id, self_ref)`. New repository accessors `get_picture_bytes` and `get_pictures_for_chunk` expose them; the existing items read paths (`get_all_items`, `get_all_items_grouped`, `get_items_in_range`, `_record_to_item`) now project an explicit lightweight column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory. Existing databases pick up the column via the `0.45.0` migration alongside the picture-byte backfill (see below). Foundation for upcoming vision-in-context retrieval; not yet wired into ingestion or search.
|
||||
- **Embedded picture bytes captured at ingestion.** `extract_items` now decodes each `PictureItem.image.uri` data URI into raw bytes and writes them to `document_items.picture_data` so per-figure lookups don't require decompressing the full docling blob. The same path also surfaces VLM-generated picture descriptions (`meta.description.text`) into `DocumentItem.text` so picture-only chunks survive `expand_with_items`' text filter. The `0.45.0` migration adds the `picture_data` column to existing databases and backfills it by extracting bytes out of `docling_document`, stripping picture URIs from the structure blob in the process; `compress_docling_split` does the same for new ingests so the structure stays lean. Rebuild and update flows snapshot picture bytes via the new `DocumentItemRepository.get_all_picture_data` accessor before re-extraction so a re-chunk doesn't drop them.
|
||||
- **Picture image bytes in search results and vision-capable QA.** `SearchResult` gains an `image_data: dict[str, str] | None` field carrying base64-encoded picture bytes keyed by `self_ref` for picture-labeled chunks. `client.search()` and the MCP `search_documents` tool gain an `include_images: bool = True` flag; set False to omit the bytes for plain-text consumers. `expand_context` now preserves picture self_refs with empty text so they aren't filtered out before reaching the image-data lookup. The agent-facing search tool (`tools/search.py`) returns `pydantic_ai.messages.ToolReturn(return_value=text, content=[BinaryContent(...), ...])` when picture data is present so a vision-capable QA model sees the figures alongside the text; otherwise it returns a plain string and non-vision flows are unchanged.
|
||||
|
|
|
|||
|
|
@ -28,6 +28,9 @@ processing:
|
|||
name: gpt-oss
|
||||
enable_thinking: false
|
||||
|
||||
# Picture handling (none / description / image)
|
||||
pictures: none # See "Picture Handling" below
|
||||
|
||||
# Conversion options (works with both local and remote converters)
|
||||
conversion_options:
|
||||
# OCR settings
|
||||
|
|
@ -44,11 +47,9 @@ processing:
|
|||
# Image settings
|
||||
images_scale: 2.0 # Image scale factor
|
||||
generate_page_images: true # Include rendered page images (for visualize_chunk)
|
||||
generate_picture_images: false # Include embedded figure/diagram images
|
||||
|
||||
# VLM picture description (optional)
|
||||
# VLM picture description settings (only effective when pictures: description)
|
||||
picture_description:
|
||||
enabled: false # Enable VLM image descriptions
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
|
|
@ -100,34 +101,44 @@ conversion_options:
|
|||
conversion_options:
|
||||
images_scale: 2.0 # Image resolution scale factor
|
||||
generate_page_images: true # Include rendered page images
|
||||
generate_picture_images: false # Include embedded figure/diagram images
|
||||
```
|
||||
|
||||
- **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
|
||||
- **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size.
|
||||
- **generate_picture_images**: When `true`, embedded images (figures, diagrams) are included as base64-encoded data in the document. When `false` (default), images are excluded to reduce chunk size and avoid context bloat.
|
||||
|
||||
**Note:** With `docling-serve`, `generate_picture_images` has limited support - picture image data may not be returned in the JSON response. Page images work correctly with both local and remote converters.
|
||||
Embedded picture extraction is controlled by `processing.pictures` (see [Picture Handling](#picture-handling) below), not by an image-settings flag.
|
||||
|
||||
#### Picture Description (VLM)
|
||||
#### Picture Handling
|
||||
|
||||
Use a Vision Language Model (VLM) to automatically describe images in documents. Descriptions become searchable text, improving RAG retrieval for visual content.
|
||||
`processing.pictures` is a single enum that decides how embedded picture images (figures, diagrams) are handled at ingest. Three modes:
|
||||
|
||||
| Mode | VLM at ingest | `picture_data` populated | Chunk text contains description |
|
||||
|---|---|---|---|
|
||||
| `none` (default) | no | no | no |
|
||||
| `description` | yes | yes | yes |
|
||||
| `image` | no | yes | no |
|
||||
|
||||
- **`none`**: docling skips picture-image generation. `label="picture"` rows still appear in the items table for structural metadata, but they carry no bytes and no description. Cheapest mode; non-vision QA is unaffected.
|
||||
- **`description`**: docling generates picture images, the configured VLM produces a description woven into chunk text, and the bytes are also retained in `document_items.picture_data`. Picture-text is searchable via FTS, vision-capable QA models also receive the bytes via the agent's search tool. Bytes are kept (not just thrown away after the VLM runs) so a vision-only QA strategy can be turned on later without reingesting.
|
||||
- **`image`**: docling generates picture images and stores them in `document_items.picture_data` without running the VLM. Vision-capable QA models reason directly about the figures; non-vision QA only sees the picture's caption/surrounding text.
|
||||
|
||||
```yaml
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true # Enable VLM picture description
|
||||
model:
|
||||
provider: ollama # ollama, openai, or custom
|
||||
name: ministral-3 # VLM model name
|
||||
temperature: 0.0 # Default: 0.0 (factual descriptions)
|
||||
timeout: 90 # Request timeout in seconds
|
||||
max_tokens: 200 # Maximum tokens in response
|
||||
processing:
|
||||
pictures: description # none / description / image
|
||||
conversion_options:
|
||||
picture_description: # only effective when pictures: description
|
||||
model:
|
||||
provider: ollama # ollama, openai, or custom
|
||||
name: ministral-3 # VLM model name
|
||||
temperature: 0.0
|
||||
timeout: 90 # Request timeout in seconds
|
||||
max_tokens: 200 # Maximum tokens in response
|
||||
```
|
||||
|
||||
**Configuration options:**
|
||||
**Switching modes on an existing database.** No reingest is required if you only need to change between `description` and `image` — the bytes are already there. Run `haiku-rag rebuild --rechunk` after the config change so the chunk-text composition reflects the new mode. Switching *down* to `none` clears `picture_data` on rebuild, reclaiming storage.
|
||||
|
||||
**`picture_description.model` configuration** (used only under `pictures: description`):
|
||||
|
||||
- **enabled**: When `true`, each embedded image is sent to a VLM for description. Requires `generate_picture_images` to be `true` (automatically enabled).
|
||||
- **model**: Standard model configuration
|
||||
- `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
|
||||
- `name`: Model name (e.g., `ministral-3`, `granite3.2-vision`, `gpt-4-vision`)
|
||||
|
|
@ -155,12 +166,13 @@ prompts:
|
|||
**Using with Ollama:**
|
||||
|
||||
```yaml
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
```
|
||||
|
||||
Requires Ollama running with a vision-capable model:
|
||||
|
|
@ -173,13 +185,14 @@ ollama serve
|
|||
**Using with vLLM or custom endpoints:**
|
||||
|
||||
```yaml
|
||||
conversion_options:
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: openai # Use OpenAI-compatible API format
|
||||
name: granite-vision
|
||||
base_url: http://my-vllm-server:8000
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
model:
|
||||
provider: openai # Use OpenAI-compatible API format
|
||||
name: granite-vision
|
||||
base_url: http://my-vllm-server:8000
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
|
|
@ -199,12 +212,14 @@ When using `converter: docling-serve`, the VLM calls are made by the docling-ser
|
|||
**Docker networking:** If docling-serve runs in Docker and your VLM runs on the host, use `host.docker.internal` instead of `localhost`:
|
||||
|
||||
```yaml
|
||||
picture_description:
|
||||
enabled: true
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
base_url: http://host.docker.internal:11434 # NOT localhost!
|
||||
processing:
|
||||
pictures: description
|
||||
conversion_options:
|
||||
picture_description:
|
||||
model:
|
||||
provider: ollama
|
||||
name: ministral-3
|
||||
base_url: http://host.docker.internal:11434 # NOT localhost!
|
||||
```
|
||||
|
||||
See [VLM Picture Description with docling-serve](../remote-processing.md#vlm-picture-description-with-docling-serve) for a complete example.
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ prompts:
|
|||
Be concise and factual.
|
||||
```
|
||||
|
||||
The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Description (VLM)](processing.md#picture-description-vlm) for full configuration.
|
||||
The prompt is used when `processing.pictures` is set to `description`. See [Picture Handling](processing.md#picture-handling) for full configuration.
|
||||
|
||||
## Programmatic Configuration
|
||||
|
||||
|
|
|
|||
|
|
@ -99,12 +99,16 @@ async def _update_document_with_chunks(
|
|||
# Replace document items when a new DoclingDocument is provided.
|
||||
# Snapshot existing picture bytes first so they survive the
|
||||
# delete-and-re-extract cycle when the live docling has already had
|
||||
# its picture URIs stripped (rebuild / round-trip scenarios).
|
||||
# its picture URIs stripped (rebuild / round-trip scenarios). Under
|
||||
# `pictures="none"` we skip the snapshot so updates reclaim storage.
|
||||
if docling_document is not None:
|
||||
keep_picture_data = client._config.processing.pictures != "none"
|
||||
existing_picture_data = (
|
||||
await client.document_item_repository.get_all_picture_data(
|
||||
updated_doc.id
|
||||
)
|
||||
if keep_picture_data
|
||||
else None
|
||||
)
|
||||
await client.document_item_repository.delete_by_document_id(updated_doc.id)
|
||||
items = extract_items(
|
||||
|
|
|
|||
|
|
@ -105,7 +105,10 @@ async def download_models(
|
|||
if config.reranking.model and config.reranking.model.provider == "ollama":
|
||||
required_models.add(config.reranking.model.name)
|
||||
pic_desc = config.processing.conversion_options.picture_description
|
||||
if pic_desc.enabled and pic_desc.model.provider == "ollama":
|
||||
if (
|
||||
config.processing.pictures == "description"
|
||||
and pic_desc.model.provider == "ollama"
|
||||
):
|
||||
required_models.add(pic_desc.model.name)
|
||||
if (
|
||||
config.processing.auto_title
|
||||
|
|
|
|||
|
|
@ -197,14 +197,19 @@ async def _flush_rebuild_batch(
|
|||
|
||||
# Repopulate document items from stored docling data. The stored docling
|
||||
# blob has had its picture URIs stripped (compress_docling_split), so
|
||||
# re-extracting from it would lose picture_data; snapshot the existing
|
||||
# bytes per document and merge them back during extraction.
|
||||
# re-extracting from it would lose picture_data; under modes that retain
|
||||
# bytes (`description`/`image`) we snapshot the existing bytes per
|
||||
# document and merge them back. Under `none`, we deliberately skip the
|
||||
# snapshot so the rebuild reclaims storage.
|
||||
keep_picture_data = client._config.processing.pictures != "none"
|
||||
for doc in documents:
|
||||
assert doc.id is not None
|
||||
docling_doc = doc.get_docling_document()
|
||||
if docling_doc is not None:
|
||||
existing_picture_data = (
|
||||
await client.document_item_repository.get_all_picture_data(doc.id)
|
||||
if keep_picture_data
|
||||
else None
|
||||
)
|
||||
await client.document_item_repository.delete_by_document_id(doc.id)
|
||||
items = extract_items(
|
||||
|
|
|
|||
|
|
@ -1,8 +1,11 @@
|
|||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def find_config_file(cli_path: Path | None = None) -> Path | None:
|
||||
"""Find the YAML config file using the search path.
|
||||
|
|
@ -44,7 +47,65 @@ def load_yaml_config(path: Path) -> dict:
|
|||
"""Load and parse a YAML config file."""
|
||||
with open(path) as f:
|
||||
data = yaml.safe_load(f)
|
||||
return data or {}
|
||||
data = data or {}
|
||||
_translate_legacy_picture_fields(data)
|
||||
return data
|
||||
|
||||
|
||||
def _translate_legacy_picture_fields(data: dict) -> None:
|
||||
"""Map pre-A4 picture knobs onto ``processing.pictures``.
|
||||
|
||||
Pre-A4 the same intent was expressed by two booleans on
|
||||
``conversion_options``: ``generate_picture_images`` and
|
||||
``picture_description.enabled``. Translation, in priority order:
|
||||
|
||||
- ``picture_description.enabled = true`` (regardless of the image flag)
|
||||
→ ``pictures = "description"``. Mirrors the original behavior where
|
||||
enabling the VLM implicitly forced docling to produce picture bytes.
|
||||
- ``generate_picture_images = true`` (and no description) → ``"image"``.
|
||||
- both false / missing → no translation; default ``"none"`` applies.
|
||||
|
||||
If ``pictures`` is already set on the loaded YAML it wins — users who
|
||||
have migrated keep their explicit choice. Mutates ``data`` in-place
|
||||
and emits one warning per legacy field encountered.
|
||||
"""
|
||||
processing = data.get("processing")
|
||||
if not isinstance(processing, dict):
|
||||
return
|
||||
|
||||
if "pictures" in processing:
|
||||
# User has migrated; legacy fields may still be present but should not
|
||||
# override the explicit choice. Drop them silently to avoid confusion.
|
||||
opts = processing.get("conversion_options")
|
||||
if isinstance(opts, dict):
|
||||
opts.pop("generate_picture_images", None)
|
||||
pic = opts.get("picture_description")
|
||||
if isinstance(pic, dict):
|
||||
pic.pop("enabled", None)
|
||||
return
|
||||
|
||||
opts = processing.get("conversion_options")
|
||||
if not isinstance(opts, dict):
|
||||
return
|
||||
|
||||
pic = opts.get("picture_description") if isinstance(opts, dict) else None
|
||||
legacy_describe = pic.pop("enabled", None) if isinstance(pic, dict) else None
|
||||
legacy_image = opts.pop("generate_picture_images", None)
|
||||
|
||||
if legacy_describe:
|
||||
processing["pictures"] = "description"
|
||||
logger.warning(
|
||||
"Config: 'processing.conversion_options.picture_description.enabled=true' is "
|
||||
"deprecated; mapped to 'processing.pictures: description'. Please update your "
|
||||
"haiku.rag.yaml."
|
||||
)
|
||||
elif legacy_image:
|
||||
processing["pictures"] = "image"
|
||||
logger.warning(
|
||||
"Config: 'processing.conversion_options.generate_picture_images=true' is "
|
||||
"deprecated; mapped to 'processing.pictures: image'. Please update your "
|
||||
"haiku.rag.yaml."
|
||||
)
|
||||
|
||||
|
||||
def generate_default_config() -> dict:
|
||||
|
|
|
|||
|
|
@ -110,9 +110,12 @@ class AnalysisConfig(BaseModel):
|
|||
|
||||
|
||||
class PictureDescriptionConfig(BaseModel):
|
||||
"""Configuration for VLM-based picture description."""
|
||||
"""Settings for the VLM that runs at ingest under ``pictures="description"``.
|
||||
|
||||
Whether the VLM runs at all is decided by ``ProcessingConfig.pictures``;
|
||||
these fields only describe *how* it runs once enabled.
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
model: ModelConfig = Field(
|
||||
default_factory=lambda: ModelConfig(
|
||||
provider="ollama",
|
||||
|
|
@ -143,14 +146,16 @@ class ConversionOptions(BaseModel):
|
|||
# Image options
|
||||
images_scale: float = 2.0
|
||||
generate_page_images: bool = True
|
||||
generate_picture_images: bool = False
|
||||
|
||||
# VLM picture description
|
||||
# VLM picture description (only effective when ProcessingConfig.pictures == "description")
|
||||
picture_description: PictureDescriptionConfig = Field(
|
||||
default_factory=PictureDescriptionConfig
|
||||
)
|
||||
|
||||
|
||||
PicturesMode = Literal["none", "description", "image"]
|
||||
|
||||
|
||||
class ProcessingConfig(BaseModel):
|
||||
chunk_size: int = 256
|
||||
converter: str = "docling-local"
|
||||
|
|
@ -160,6 +165,18 @@ class ProcessingConfig(BaseModel):
|
|||
chunking_merge_peers: bool = True
|
||||
chunking_use_markdown_tables: bool = False
|
||||
conversion_options: ConversionOptions = Field(default_factory=ConversionOptions)
|
||||
pictures: PicturesMode = "none"
|
||||
"""How embedded pictures are handled at ingest.
|
||||
|
||||
- ``"none"``: docling skips picture-image generation; structural
|
||||
``label="picture"`` rows still exist but carry no bytes or description.
|
||||
- ``"description"``: docling generates picture images, the VLM produces
|
||||
text descriptions woven into chunk text, and the bytes are also
|
||||
retained in ``document_items.picture_data`` so a vision-capable QA
|
||||
model can be turned on later without reingesting.
|
||||
- ``"image"``: docling generates picture images and stores them in
|
||||
``document_items.picture_data``; no VLM runs at ingest.
|
||||
"""
|
||||
auto_title: bool = False
|
||||
title_model: ModelConfig = Field(
|
||||
default_factory=lambda: ModelConfig(
|
||||
|
|
|
|||
|
|
@ -123,13 +123,16 @@ class DoclingLocalConverter(DocumentConverter):
|
|||
|
||||
opts = self.config.processing.conversion_options
|
||||
pic_desc = opts.picture_description
|
||||
pictures_mode = self.config.processing.pictures
|
||||
wants_picture_images = pictures_mode != "none"
|
||||
runs_vlm = pictures_mode == "description"
|
||||
|
||||
pipeline_options = PdfPipelineOptions(
|
||||
do_ocr=opts.do_ocr,
|
||||
do_table_structure=opts.do_table_structure,
|
||||
images_scale=opts.images_scale,
|
||||
generate_page_images=opts.generate_page_images,
|
||||
generate_picture_images=opts.generate_picture_images or pic_desc.enabled,
|
||||
generate_picture_images=wants_picture_images,
|
||||
table_structure_options=TableStructureOptions(
|
||||
do_cell_matching=opts.table_cell_matching,
|
||||
mode=(
|
||||
|
|
@ -139,10 +142,10 @@ class DoclingLocalConverter(DocumentConverter):
|
|||
),
|
||||
),
|
||||
ocr_options=self._get_ocr_options(opts),
|
||||
do_picture_description=pic_desc.enabled,
|
||||
do_picture_description=runs_vlm,
|
||||
)
|
||||
|
||||
if pic_desc.enabled:
|
||||
if runs_vlm:
|
||||
from pydantic import AnyUrl
|
||||
|
||||
prompt = self.config.prompts.picture_description
|
||||
|
|
|
|||
|
|
@ -85,13 +85,8 @@ class DoclingServeConverter(DocumentConverter):
|
|||
raise ValueError(f"Unsupported VLM provider: {model.provider}")
|
||||
|
||||
def _picture_images_enabled(self) -> bool:
|
||||
"""Whether the conversion should produce embedded picture images.
|
||||
|
||||
True if the user explicitly enabled ``generate_picture_images`` or if
|
||||
``picture_description.enabled`` is on (the VLM needs the picture bytes).
|
||||
"""
|
||||
opts = self.config.processing.conversion_options
|
||||
return opts.generate_picture_images or opts.picture_description.enabled
|
||||
"""Whether the conversion should produce embedded picture images."""
|
||||
return self.config.processing.pictures != "none"
|
||||
|
||||
def _build_conversion_data(self) -> dict[str, str | list[str]]:
|
||||
"""Build form data for conversion request.
|
||||
|
|
@ -108,7 +103,9 @@ class DoclingServeConverter(DocumentConverter):
|
|||
"""
|
||||
opts = self.config.processing.conversion_options
|
||||
pic_desc = opts.picture_description
|
||||
picture_images_enabled = self._picture_images_enabled()
|
||||
pictures_mode = self.config.processing.pictures
|
||||
picture_images_enabled = pictures_mode != "none"
|
||||
runs_vlm = pictures_mode == "description"
|
||||
|
||||
if picture_images_enabled:
|
||||
image_export_mode = "referenced"
|
||||
|
|
@ -128,7 +125,7 @@ class DoclingServeConverter(DocumentConverter):
|
|||
"images_scale": str(opts.images_scale),
|
||||
"image_export_mode": image_export_mode,
|
||||
"include_images": str(picture_images_enabled).lower(),
|
||||
"do_picture_description": str(pic_desc.enabled).lower(),
|
||||
"do_picture_description": str(runs_vlm).lower(),
|
||||
}
|
||||
|
||||
if picture_images_enabled:
|
||||
|
|
@ -137,7 +134,7 @@ class DoclingServeConverter(DocumentConverter):
|
|||
if opts.ocr_lang:
|
||||
data["ocr_lang"] = opts.ocr_lang
|
||||
|
||||
if pic_desc.enabled:
|
||||
if runs_vlm:
|
||||
prompt = self.config.prompts.picture_description
|
||||
picture_description_api = {
|
||||
"url": self._get_vlm_api_url(pic_desc.model),
|
||||
|
|
|
|||
|
|
@ -431,6 +431,8 @@ class TestPictureDataStorage:
|
|||
async with HaikuRAG(temp_db_path, create=True) as rag:
|
||||
schema = await rag.store.document_items_table.schema()
|
||||
assert "picture_data" in {f.name for f in schema}
|
||||
|
||||
|
||||
def _docling_doc_with_picture():
|
||||
"""Build a tiny DoclingDocument with one PictureItem carrying real PNG bytes
|
||||
via ImageRef.from_pil. Used by the picture-extraction tests."""
|
||||
|
|
@ -634,6 +636,8 @@ class TestPictureDataPreservedThroughRoundTrip:
|
|||
docling_doc = _docling_doc_with_picture()
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as rag:
|
||||
# Preservation only kicks in under modes that retain picture bytes.
|
||||
rag._config.processing.pictures = "image"
|
||||
document = Document(content="Hello world", uri="test://doc")
|
||||
document.set_docling(docling_doc)
|
||||
created = await _store_document_with_chunks(rag, document, [], docling_doc)
|
||||
|
|
|
|||
|
|
@ -225,3 +225,92 @@ def test_init_config_creates_valid_yaml(tmp_path):
|
|||
# Validate it
|
||||
config = AppConfig.model_validate(loaded_data)
|
||||
assert config.environment == "production"
|
||||
|
||||
|
||||
# A4: legacy `generate_picture_images` + `picture_description.enabled` translation
|
||||
|
||||
|
||||
def _write(tmp_path, body: str):
|
||||
p = tmp_path / "haiku.rag.yaml"
|
||||
p.write_text(body)
|
||||
return p
|
||||
|
||||
|
||||
def test_load_yaml_legacy_picture_description_maps_to_description(tmp_path, caplog):
|
||||
"""`picture_description.enabled=true` (with or without the image flag)
|
||||
maps to `processing.pictures: description`."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
conversion_options:
|
||||
generate_picture_images: false
|
||||
picture_description:
|
||||
enabled: true
|
||||
timeout: 120
|
||||
""",
|
||||
)
|
||||
with caplog.at_level("WARNING", logger="haiku.rag.config.loader"):
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "description"
|
||||
assert cfg.processing.conversion_options.picture_description.timeout == 120
|
||||
assert any("picture_description.enabled=true" in m.message for m in caplog.records)
|
||||
|
||||
|
||||
def test_load_yaml_legacy_generate_picture_images_maps_to_image(tmp_path, caplog):
|
||||
"""`generate_picture_images=true` alone maps to `pictures: image`."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
conversion_options:
|
||||
generate_picture_images: true
|
||||
""",
|
||||
)
|
||||
with caplog.at_level("WARNING", logger="haiku.rag.config.loader"):
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "image"
|
||||
assert any("generate_picture_images=true" in m.message for m in caplog.records)
|
||||
|
||||
|
||||
def test_load_yaml_no_legacy_fields_keeps_default_none(tmp_path, caplog):
|
||||
"""Empty processing block leaves the default `none` mode untouched and
|
||||
does not warn."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
chunk_size: 256
|
||||
""",
|
||||
)
|
||||
with caplog.at_level("WARNING", logger="haiku.rag.config.loader"):
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "none"
|
||||
assert not caplog.records
|
||||
|
||||
|
||||
def test_load_yaml_explicit_pictures_wins_over_legacy(tmp_path):
|
||||
"""When the user has migrated to `pictures: ...` we keep their choice
|
||||
and silently drop legacy fields if both are present (e.g. from a
|
||||
half-migrated config)."""
|
||||
config_file = _write(
|
||||
tmp_path,
|
||||
"""
|
||||
processing:
|
||||
pictures: image
|
||||
conversion_options:
|
||||
generate_picture_images: false
|
||||
picture_description:
|
||||
enabled: true
|
||||
""",
|
||||
)
|
||||
data = load_yaml_config(config_file)
|
||||
cfg = AppConfig.model_validate(data)
|
||||
assert cfg.processing.pictures == "image"
|
||||
# Legacy fields scrubbed so Pydantic validation doesn't trip on extras.
|
||||
opts = data["processing"]["conversion_options"]
|
||||
assert "generate_picture_images" not in opts
|
||||
assert "enabled" not in opts.get("picture_description", {})
|
||||
|
|
|
|||
|
|
@ -378,7 +378,7 @@ class TestDoclingLocalConverter:
|
|||
async def test_convert_pdf_without_picture_images(self, config):
|
||||
"""Test PDF conversion excludes embedded images by default."""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.conversion_options.generate_picture_images = False
|
||||
config.processing.pictures = "none"
|
||||
converter = DoclingLocalConverter(config)
|
||||
|
||||
doc = await converter.convert_file(pdf_path)
|
||||
|
|
@ -387,14 +387,14 @@ class TestDoclingLocalConverter:
|
|||
# Check that pictures don't have image data
|
||||
for picture in doc.pictures:
|
||||
assert picture.image is None, (
|
||||
"Pictures should not have image data when generate_picture_images=False"
|
||||
'Pictures should not have image data when pictures="none"'
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_convert_pdf_with_picture_images(self, config):
|
||||
"""Test PDF conversion includes embedded images when enabled."""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.conversion_options.generate_picture_images = True
|
||||
config.processing.pictures = "image"
|
||||
converter = DoclingLocalConverter(config)
|
||||
|
||||
doc = await converter.convert_file(pdf_path)
|
||||
|
|
@ -404,7 +404,7 @@ class TestDoclingLocalConverter:
|
|||
pictures_with_images = [p for p in doc.pictures if p.image is not None]
|
||||
if doc.pictures:
|
||||
assert len(pictures_with_images) > 0, (
|
||||
"Pictures should have image data when generate_picture_images=True"
|
||||
'Pictures should have image data when pictures="image"'
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -549,7 +549,7 @@ class TestDoclingLocalConverter:
|
|||
|
||||
def test_picture_description_config_defaults(self, config):
|
||||
"""Test that picture description config has correct defaults."""
|
||||
assert config.processing.conversion_options.picture_description.enabled is False
|
||||
assert config.processing.pictures == "none"
|
||||
assert (
|
||||
config.processing.conversion_options.picture_description.model.provider
|
||||
== "ollama"
|
||||
|
|
@ -567,12 +567,12 @@ class TestDoclingLocalConverter:
|
|||
|
||||
def test_picture_description_config_applied(self, config):
|
||||
"""Test that picture description config is applied to converter."""
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.timeout = 120
|
||||
converter = DoclingLocalConverter(config)
|
||||
|
||||
assert converter.config.processing.pictures == "description"
|
||||
pic_desc = converter.config.processing.conversion_options.picture_description
|
||||
assert pic_desc.enabled is True
|
||||
assert pic_desc.timeout == 120
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -584,7 +584,7 @@ class TestDoclingLocalConverter:
|
|||
# Disable OCR (not needed for native PDF, avoids model downloads)
|
||||
config.processing.conversion_options.do_ocr = False
|
||||
# Enable picture description with Ollama
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
@ -700,7 +700,7 @@ class TestDoclingServeConverter:
|
|||
config.processing.conversion_options.table_cell_matching = False
|
||||
config.processing.conversion_options.do_table_structure = False
|
||||
config.processing.conversion_options.images_scale = 3.0
|
||||
config.processing.conversion_options.generate_picture_images = False
|
||||
config.processing.pictures = "none"
|
||||
converter = DoclingServeConverter(config)
|
||||
|
||||
doc_json = create_mock_docling_document("test")
|
||||
|
|
@ -754,11 +754,11 @@ class TestDoclingServeConverter:
|
|||
|
||||
@pytest.mark.asyncio
|
||||
async def test_picture_images_request_uses_referenced_zip(self, config):
|
||||
"""When generate_picture_images is on, the request flips to
|
||||
"""When pictures="image" the request flips to
|
||||
image_export_mode=referenced + target_type=zip and consumes a zip
|
||||
response — mirrors the upstream docling-serve#576 workaround.
|
||||
"""
|
||||
config.processing.conversion_options.generate_picture_images = True
|
||||
config.processing.pictures = "image"
|
||||
converter = DoclingServeConverter(config)
|
||||
|
||||
doc_json = create_mock_docling_document("test")
|
||||
|
|
@ -1038,13 +1038,13 @@ class TestDoclingServeConverterPictureDescription:
|
|||
async def test_picture_description_options_passed_to_api(self, config):
|
||||
"""Test that picture description options are passed to docling-serve API.
|
||||
|
||||
Picture descriptions force ``generate_picture_images=True`` upstream,
|
||||
which routes the request through the ``target_type=zip`` path so the
|
||||
VLM can see the actual figures. The test mocks the zip workflow.
|
||||
``pictures="description"`` requires picture images for the VLM, which
|
||||
routes the request through the ``target_type=zip`` path. The test
|
||||
mocks the zip workflow.
|
||||
"""
|
||||
import json
|
||||
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
@ -1155,7 +1155,7 @@ class TestDoclingServeConverterIntegration:
|
|||
Note: Not using VCR because this test involves polling with changing task IDs.
|
||||
"""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.conversion_options.picture_description.enabled = True
|
||||
config.processing.pictures = "description"
|
||||
config.processing.conversion_options.picture_description.model.provider = (
|
||||
"ollama"
|
||||
)
|
||||
|
|
@ -1228,7 +1228,7 @@ class TestDoclingServeConverterIntegration:
|
|||
async def test_convert_pdf_without_picture_images(self, config):
|
||||
"""Test PDF conversion excludes picture images when disabled."""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.conversion_options.generate_picture_images = False
|
||||
config.processing.pictures = "none"
|
||||
converter = DoclingServeConverter(config)
|
||||
|
||||
doc = await converter.convert_file(pdf_path)
|
||||
|
|
@ -1237,7 +1237,7 @@ class TestDoclingServeConverterIntegration:
|
|||
# Check that pictures don't have image data
|
||||
for picture in doc.pictures:
|
||||
assert picture.image is None, (
|
||||
"Pictures should not have image data when generate_picture_images=False"
|
||||
'Pictures should not have image data when pictures="none"'
|
||||
)
|
||||
|
||||
@pytest.mark.vcr()
|
||||
|
|
@ -1266,7 +1266,7 @@ class TestDoclingServeConverterIntegration:
|
|||
URIs so the result is shape-equivalent to the local converter.
|
||||
"""
|
||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||
config.processing.conversion_options.generate_picture_images = True
|
||||
config.processing.pictures = "image"
|
||||
converter = DoclingServeConverter(config)
|
||||
|
||||
doc = await converter.convert_file(pdf_path)
|
||||
|
|
@ -1275,7 +1275,7 @@ class TestDoclingServeConverterIntegration:
|
|||
pictures_with_images = [p for p in doc.pictures if p.image is not None]
|
||||
assert doc.pictures, "doclaynet.pdf is expected to contain at least one picture"
|
||||
assert len(pictures_with_images) > 0, (
|
||||
"Pictures should have image data when generate_picture_images=True"
|
||||
'Pictures should have image data when pictures="image"'
|
||||
)
|
||||
sample = pictures_with_images[0]
|
||||
assert sample.image is not None
|
||||
|
|
|
|||
|
|
@ -166,6 +166,42 @@ async def test_expand_context_preserves_picture_refs_with_empty_text(temp_db_pat
|
|||
assert "picture" in out.labels
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_clears_picture_data_when_mode_none(temp_db_path):
|
||||
"""Switching to ``pictures="none"`` and re-running update_document
|
||||
drops picture_data — the snapshot/merge gate is what gives users a
|
||||
"rebuild reclaims storage" path when they downgrade modes."""
|
||||
from haiku.rag.client.documents import (
|
||||
_store_document_with_chunks,
|
||||
_update_document_with_chunks,
|
||||
)
|
||||
from haiku.rag.store.models.document import Document
|
||||
from tests.store.test_document_items import _docling_doc_with_picture
|
||||
|
||||
docling_doc = _docling_doc_with_picture()
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as rag:
|
||||
# Ingest under "image" so picture bytes land in document_items.
|
||||
rag._config.processing.pictures = "image"
|
||||
document = Document(content="x", uri="test://doc")
|
||||
document.set_docling(docling_doc)
|
||||
created = await _store_document_with_chunks(rag, document, [], docling_doc)
|
||||
assert created.id is not None
|
||||
before = await rag.document_item_repository.get_all_picture_data(created.id)
|
||||
assert before.get("#/pictures/0") is not None
|
||||
|
||||
# Downgrade to "none" and re-run update with the (already stripped)
|
||||
# docling pulled from storage. The snapshot/merge must be skipped so
|
||||
# picture_data is cleared on the new items rows.
|
||||
rag._config.processing.pictures = "none"
|
||||
from_blob = created.get_docling_document()
|
||||
assert from_blob is not None
|
||||
await _update_document_with_chunks(rag, created, [], from_blob)
|
||||
|
||||
after = await rag.document_item_repository.get_all_picture_data(created.id)
|
||||
assert after.get("#/pictures/0") is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expand_context_repopulates_image_data(temp_db_path):
|
||||
"""expand_context rebuilds SearchResult objects via expand_with_items, so
|
||||
|
|
|
|||
Loading…
Reference in a new issue