diff --git a/CHANGELOG.md b/CHANGELOG.md index e15d7e6b..0ab06cc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,15 +3,28 @@ ### Added -- **Picture-handling mode enum.** `processing.pictures: "none" | "description" | "image"` (default `"none"`) replaces the previous two-flag dance (`generate_picture_images` + `picture_description.enabled`) where flipping picture descriptions on silently forced docling to also generate picture images, leaving raw bytes inside the compressed `docling_document` blob with no user-facing knob to control it. The new enum makes intent explicit: `description` runs the VLM at ingest *and* preserves bytes (so a vision-capable QA model can be turned on later without reingesting), `image` preserves bytes without running the VLM, `none` is bytes-free. The legacy fields are removed from the schema (`ConversionOptions.generate_picture_images` and `PictureDescriptionConfig.enabled`); the YAML loader silently translates pre-existing configs to the new enum and emits a one-line deprecation warning so existing `haiku.rag.yaml` files keep working until the user updates them. No data migration is required — the 0.45.0 backfill already populates `picture_data` for every doc that had bytes inline. Rebuild and update flows now skip the picture-bytes snapshot/merge under `pictures="none"`, so users who downgrade to `none` and rebuild reclaim storage. -- **Storage column for embedded picture bytes.** `DocumentItemRecord` gains a `picture_data: bytes | None` column (Arrow `large_binary`) to hold per-`PictureItem` image bytes addressable by `(document_id, self_ref)`. New repository accessors `get_picture_bytes` and `get_pictures_for_chunk` expose them; the existing items read paths (`get_all_items`, `get_all_items_grouped`, `get_items_in_range`, `_record_to_item`) now project an explicit lightweight column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory. Existing databases pick up the column via the `0.45.0` migration alongside the picture-byte backfill (see below). Foundation for upcoming vision-in-context retrieval; not yet wired into ingestion or search. -- **Embedded picture bytes captured at ingestion.** `extract_items` now decodes each `PictureItem.image.uri` data URI into raw bytes and writes them to `document_items.picture_data` so per-figure lookups don't require decompressing the full docling blob. The same path also surfaces VLM-generated picture descriptions (`meta.description.text`) into `DocumentItem.text` so picture-only chunks survive `expand_with_items`' text filter. The `0.45.0` migration adds the `picture_data` column to existing databases and backfills it by extracting bytes out of `docling_document`, stripping picture URIs from the structure blob in the process; `compress_docling_split` does the same for new ingests so the structure stays lean. Rebuild and update flows snapshot picture bytes via the new `DocumentItemRepository.get_all_picture_data` accessor before re-extraction so a re-chunk doesn't drop them. -- **Picture image bytes in search results and vision-capable QA.** `SearchResult` gains an `image_data: dict[str, str] | None` field carrying base64-encoded picture bytes keyed by `self_ref` for picture-labeled chunks. `client.search()` and the MCP `search_documents` tool gain an `include_images: bool = True` flag; set False to omit the bytes for plain-text consumers. `expand_context` now preserves picture self_refs with empty text so they aren't filtered out before reaching the image-data lookup. The agent-facing search tool (`tools/search.py`) returns `pydantic_ai.messages.ToolReturn(return_value=text, content=[BinaryContent(...), ...])` when picture data is present so a vision-capable QA model sees the figures alongside the text; otherwise it returns a plain string and non-vision flows are unchanged. +- **Picture-handling mode enum.** `processing.pictures: "none" | "description" | "image"` (default `"none"`) replaces the legacy `generate_picture_images` + `picture_description.enabled` pair. + - `none` — no picture bytes, no VLM, just structural picture rows. + - `description` — VLM runs at ingest, descriptions woven into chunk text, bytes also retained on `document_items.picture_data` so vision QA can be turned on later without reingesting. + - `image` — bytes retained, no VLM. + - YAML loader translates legacy configs and emits a deprecation warning. + - Rebuild and update under `pictures="none"` skip the picture-bytes snapshot/merge so storage is reclaimed. +- **Storage column for embedded picture bytes.** `DocumentItemRecord.picture_data: bytes | None` (Arrow `large_binary`), addressable by `(document_id, self_ref)`. + - New accessors `get_picture_bytes`, `get_pictures_for_chunk`, `get_all_picture_data` on `DocumentItemRepository`. + - Bulk read paths project a metadata-only column set so context expansion and the analysis-sandbox `items.jsonl` build never pull picture bytes into memory. +- **Embedded picture bytes captured at ingestion.** `extract_items` decodes `PictureItem.image.uri` data URIs into raw bytes on `document_items.picture_data`, and surfaces VLM picture descriptions (`meta.description.text`) into `DocumentItem.text`. + - `compress_docling_split` strips picture URIs from the stored docling blob so they live in one place. + - The 0.45.0 migration adds the column on existing DBs and backfills it from the legacy blob. + - Rebuild and update snapshot existing bytes via `get_all_picture_data` so re-chunk preserves them. +- **Picture image bytes in search results and vision-capable QA.** `SearchResult.image_data: dict[str, str] | None` carries base64 picture bytes keyed by `self_ref`. + - `client.search()` and the MCP `search_documents` tool gain `include_images: bool = True`. + - `expand_context` preserves picture self_refs with empty text and re-attaches bytes to the rebuilt `SearchResult`s. + - The agent's search tool returns `pydantic_ai.messages.ToolReturn(return_value=text, content=[BinaryContent(...)])` when picture data is present, otherwise a plain string. ### Fixed -- **docling-serve picture-image extraction.** docling-serve never populated `PictureItem.image` when called with `image_export_mode="embedded"` (per upstream issue [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576) — picture-image generation is only triggered when the server's `image_export_mode == "referenced"`). The converter now switches to `image_export_mode="referenced"` + `target_type="zip"` whenever picture images are requested, parses the returned zip, and rehydrates `artifacts/` URIs back into `data:;base64,...` URIs so downstream code sees the same shape as docling-local. The previously-`xfail`ed `test_convert_pdf_with_picture_images` integration test now passes. -- **`expand_context` now repopulates `image_data` after rebuilding `SearchResult` objects.** `expand_with_items` constructs fresh `SearchResult`s from the items table, so `image_data` attached upstream by `client.search()` was being dropped before reaching the agent's search tool — vision-capable QA models received the picture text/captions but never the picture bytes. The expansion path now re-runs the picture-bytes lookup against the post-expansion `doc_item_refs` (which can grow to include sibling pictures pulled in via section bounds). +- **docling-serve picture-image extraction.** docling-serve only emits picture bytes under `image_export_mode="referenced"` (upstream [docling-project/docling-serve#576](https://github.com/docling-project/docling-serve/issues/576)). The converter now switches to `referenced` + `target_type="zip"` when picture images are requested, parses the returned zip, and rehydrates `artifacts/` URIs back into `data:;base64,...` URIs. The previously-`xfail`ed `test_convert_pdf_with_picture_images` test now passes. +- **`rebuild --rechunk` now reuses the stored docling blob** instead of re-converting from the markdown export, which dropped every `PictureItem` (and its bytes) on the floor. Documents without a stored docling blob now raise instead of silently falling back to markdown. ## [0.44.0] - 2026-04-29