Merge pull request #218 from ggozad/feat/vlm-image-handling

Support for VLM-based picture description for embedded images

CHANGELOG.md

@@ -1,6 +1,15 @@
 # Changelog
 ## [Unreleased]
 
+### Added
+
+- **VLM Picture Description**: Describe embedded images using Vision Language Models during document conversion
+  - Images are sent to a VLM for automatic description via OpenAI-compatible API
+  - Descriptions become searchable text, improving RAG retrieval for visual content
+  - Configure via `processing.conversion_options.picture_description` with `enabled`, `model`, `timeout`, `max_tokens`
+  - Default prompt customizable via `prompts.picture_description`
+  - Requires OpenAI-compatible `/v1/chat/completions` endpoint (Ollama, OpenAI, vLLM, LM Studio)
+
 ## [0.23.2] - 2026-01-05
 
 ### Fixed

docs/configuration/processing.md

@@ -36,6 +36,13 @@ processing:
     # Image settings
     images_scale: 2.0                        # Image scale factor
     generate_picture_images: false           # Include embedded images in output
+
+    # VLM picture description (optional)
+    picture_description:
+      enabled: false                         # Enable VLM image descriptions
+      model:
+        provider: ollama
+        name: ministral-3
 ```
 
 ### Conversion Options
@@ -81,6 +88,92 @@ conversion_options:
 - **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
 - **generate_picture_images**: When `true`, embedded images (figures, diagrams) are included as base64-encoded data in the document. When `false` (default), images are excluded to reduce chunk size and avoid context bloat.
 
+#### Picture Description (VLM)
+
+Use a Vision Language Model (VLM) to automatically describe images in documents. Descriptions become searchable text, improving RAG retrieval for visual content.
+
+```yaml
+conversion_options:
+  picture_description:
+    enabled: true                  # Enable VLM picture description
+    model:
+      provider: ollama             # ollama, openai, or custom
+      name: ministral-3            # VLM model name
+    timeout: 90                    # Request timeout in seconds
+    max_tokens: 200                # Maximum tokens in response
+```
+
+**Configuration options:**
+
+- **enabled**: When `true`, each embedded image is sent to a VLM for description. Requires `generate_picture_images` to be `true` (automatically enabled).
+- **model**: Standard model configuration
+  - `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
+  - `name`: Model name (e.g., `ministral-3`, `granite3.2-vision`, `gpt-4-vision`)
+  - `base_url`: Optional custom API endpoint for vLLM, LM Studio, etc.
+- **timeout**: Request timeout in seconds
+- **max_tokens**: Maximum tokens in the VLM response
+
+**Note:** Requires an OpenAI-compatible `/v1/chat/completions` endpoint. Providers with different API formats (e.g., Anthropic Claude) are not supported.
+
+**Default prompt** (configured in `prompts.picture_description`):
+
+```
+Describe this image for a blind user. State the image type
+(screenshot, chart, photo, etc.), what it depicts, any visible text,
+and key visual details. Be concise and accurate.
+```
+
+To customize the prompt globally:
+
+```yaml
+prompts:
+  picture_description: "Your custom prompt here..."
+```
+
+**Using with Ollama:**
+
+```yaml
+conversion_options:
+  picture_description:
+    enabled: true
+    model:
+      provider: ollama
+      name: ministral-3
+```
+
+Requires Ollama running with a vision-capable model:
+
+```bash
+ollama pull ministral-3
+ollama serve
+```
+
+**Using with vLLM or custom endpoints:**
+
+```yaml
+conversion_options:
+  picture_description:
+    enabled: true
+    model:
+      provider: openai           # Use OpenAI-compatible API format
+      name: granite-vision
+      base_url: http://my-vllm-server:8000
+```
+
+**How it works:**
+
+1. During PDF conversion, docling extracts embedded images
+2. Each image is sent to the configured VLM for description
+3. Descriptions are added as annotations on the image
+4. When exported to markdown, descriptions appear as searchable text
+
+**Using with docling-serve:**
+
+When using `converter: docling-serve`, the VLM calls are made by the docling-serve instance, not by haiku.rag. You must:
+
+1. Set `DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` when running docling-serve
+2. Ensure the VLM endpoint is accessible from where docling-serve is running
+
 ### Local vs Remote Processing
 
 **Local processing** (default):
@@ -106,7 +199,6 @@ providers:
   docling_serve:
     base_url: http://localhost:5001
     api_key: "your-api-key"  # Optional
-    timeout: 300              # Request timeout in seconds
 ```
 
 Conversion options work identically for both local and remote processing.

docs/configuration/prompts.md

@@ -16,6 +16,9 @@ prompts:
 
   # Full replacement for research synthesis prompt (optional)
   synthesis: null
+
+  # VLM prompt for image description during conversion (optional)
+  picture_description: null  # Uses default prompt
 ```
 
 ## Domain Preamble
@@ -92,6 +95,29 @@ prompts:
     - Avoid meta-commentary like "This report covers..."
 ```
 
+## Picture Description Prompt
+
+Customize the prompt used when generating VLM descriptions for embedded images during document conversion. This prompt is sent to the configured Vision Language Model for each image.
+
+**Default prompt:**
+
+```
+Describe this image for a blind user. State the image type (screenshot, chart, photo, etc.),
+what it depicts, any visible text, and key visual details. Be concise and accurate.
+```
+
+**Custom example:**
+
+```yaml
+prompts:
+  picture_description: |
+    Describe this image for a document search system.
+    Focus on: image type, main content, any text, key visual elements.
+    Be concise and factual.
+```
+
+The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Description (VLM)](processing.md#picture-description-vlm) for full configuration.
+
 ## Programmatic Configuration
 
 ```python
@@ -103,6 +129,7 @@ config = AppConfig(
         domain_preamble="You are answering questions about our product documentation.",
         qa=None,  # Use default QA prompt
         synthesis=None,  # Use default synthesis prompt
+        picture_description="Describe this image for search indexing.",
     )
 )
 ```

docs/remote-processing.md

@@ -55,8 +55,7 @@ processing:
 providers:
   docling_serve:
     base_url: http://localhost:5001
-    api_key: ""     # Optional API key for authentication
+    api_key: ""  # Optional API key for authentication
-    timeout: 300    # Request timeout in seconds
 ```
 
 ## Features

examples/ag-ui-research/haiku.rag.yaml.example

@@ -12,7 +12,6 @@ providers:
   docling_serve:
     base_url: http://docling-serve:5001
     api_key: ""
-    timeout: 300
   ollama:
     base_url: http://host.docker.internal:11434
 

examples/docker/haiku.rag.yaml.example

@@ -24,7 +24,6 @@ providers:
   docling_serve:
     base_url: http://docling-serve:5001
     api_key: ""
-    timeout: 300
   ollama:
     base_url: http://host.docker.internal:11434
 

haiku_rag_slim/haiku/rag/chunkers/docling_serve.py

@@ -2,10 +2,9 @@ import re
 from io import BytesIO
 from typing import TYPE_CHECKING
 
-import httpx
-
 from haiku.rag.chunkers.base import DocumentChunker
 from haiku.rag.config import AppConfig, Config
+from haiku.rag.providers.docling_serve import DoclingServeClient
 from haiku.rag.store.models.chunk import Chunk, ChunkMetadata
 
 if TYPE_CHECKING:
@@ -56,11 +55,25 @@ class DoclingServeChunker(DocumentChunker):
 
     def __init__(self, config: AppConfig = Config):
         self.config = config
-        self.base_url = config.providers.docling_serve.base_url.rstrip("/")
+        self.client = DoclingServeClient(
-        self.api_key = config.providers.docling_serve.api_key
+            base_url=config.providers.docling_serve.base_url,
-        self.timeout = config.providers.docling_serve.timeout
+            api_key=config.providers.docling_serve.api_key,
+        )
         self.chunker_type = config.processing.chunker_type
 
+    def _build_chunking_data(self) -> dict[str, str]:
+        """Build form data for chunking request."""
+        return {
+            "chunking_max_tokens": str(self.config.processing.chunk_size),
+            "chunking_tokenizer": self.config.processing.chunking_tokenizer,
+            "chunking_merge_peers": str(
+                self.config.processing.chunking_merge_peers
+            ).lower(),
+            "chunking_use_markdown_tables": str(
+                self.config.processing.chunking_use_markdown_tables
+            ).lower(),
+        }
+
     async def _call_chunk_api(self, document: "DoclingDocument") -> list[dict]:
         """Call docling-serve chunking API and return raw chunk data.
 
@@ -75,9 +88,9 @@ class DoclingServeChunker(DocumentChunker):
         """
         # Determine endpoint based on chunker_type
         if self.chunker_type == "hierarchical":
-            url = f"{self.base_url}/v1/chunk/hierarchical/file"
+            endpoint = "/v1/chunk/hierarchical/file/async"
         else:
-            url = f"{self.base_url}/v1/chunk/hybrid/file"
+            endpoint = "/v1/chunk/hybrid/file/async"
 
         # Export document to JSON
         doc_json = document.model_dump_json()
@@ -85,53 +98,16 @@ class DoclingServeChunker(DocumentChunker):
 
         # Prepare multipart request with DoclingDocument JSON
         files = {"files": ("document.json", BytesIO(doc_bytes), "application/json")}
+        data = self._build_chunking_data()
 
-        # Build form data with chunking parameters
+        result = await self.client.submit_and_poll(
-        data = {
+            endpoint=endpoint,
-            "chunking_max_tokens": str(self.config.processing.chunk_size),
+            files=files,
-            "chunking_tokenizer": self.config.processing.chunking_tokenizer,
+            data=data,
-            "chunking_merge_peers": str(
+            name="document",
-                self.config.processing.chunking_merge_peers
+        )
-            ).lower(),
-            "chunking_use_markdown_tables": str(
-                self.config.processing.chunking_use_markdown_tables
-            ).lower(),
-        }
 
-        headers = {}
+        return result.get("chunks", [])
-        if self.api_key:
-            headers["X-Api-Key"] = self.api_key
-
-        try:
-            async with httpx.AsyncClient(timeout=self.timeout) as client:
-                response = await client.post(
-                    url,
-                    files=files,
-                    data=data,
-                    headers=headers,
-                )
-                response.raise_for_status()
-                result = response.json()
-                return result.get("chunks", [])
-
-        except httpx.ConnectError as e:
-            raise ValueError(
-                f"Could not connect to docling-serve at {self.base_url}. "
-                f"Ensure the service is running and accessible. Error: {e}"
-            )
-        except httpx.TimeoutException as e:
-            raise ValueError(
-                f"Request to docling-serve timed out after {self.timeout}s. "
-                f"Consider increasing the timeout in configuration. Error: {e}"
-            )
-        except httpx.HTTPStatusError as e:
-            if e.response.status_code == 401:
-                raise ValueError(
-                    "Authentication failed. Check your API key configuration."
-                )
-            raise ValueError(f"HTTP error from docling-serve: {e}")
-        except Exception as e:
-            raise ValueError(f"Failed to chunk via docling-serve: {e}")
 
     async def chunk(self, document: "DoclingDocument") -> list[Chunk]:
         """Split the document into chunks with metadata via docling-serve.

haiku_rag_slim/haiku/rag/client.py

@@ -1672,6 +1672,9 @@ class HaikuRAG:
             and self._config.reranking.model.provider == "ollama"
         ):
             required_models.add(self._config.reranking.model.name)
+        pic_desc = self._config.processing.conversion_options.picture_description
+        if pic_desc.enabled and pic_desc.model.provider == "ollama":
+            required_models.add(pic_desc.model.name)
 
         if not required_models:
             return

haiku_rag_slim/haiku/rag/config/models.py

@@ -96,6 +96,20 @@ class ResearchConfig(BaseModel):
     max_concurrency: int = 1
 
 
+class PictureDescriptionConfig(BaseModel):
+    """Configuration for VLM-based picture description."""
+
+    enabled: bool = False
+    model: ModelConfig = Field(
+        default_factory=lambda: ModelConfig(
+            provider="ollama",
+            name="ministral-3",
+        )
+    )
+    timeout: int = 90
+    max_tokens: int = 200
+
+
 class ConversionOptions(BaseModel):
     """Options for document conversion."""
 
@@ -113,6 +127,11 @@ class ConversionOptions(BaseModel):
     images_scale: float = 2.0
     generate_picture_images: bool = False
 
+    # VLM picture description
+    picture_description: PictureDescriptionConfig = Field(
+        default_factory=PictureDescriptionConfig
+    )
+
 
 class ProcessingConfig(BaseModel):
     chunk_size: int = 256
@@ -145,7 +164,6 @@ class OllamaConfig(BaseModel):
 class DoclingServeConfig(BaseModel):
     base_url: str = "http://localhost:5001"
     api_key: str = ""
-    timeout: int = 300
 
 
 class ProvidersConfig(BaseModel):
@@ -166,6 +184,12 @@ class PromptsConfig(BaseModel):
     domain_preamble: str = ""
     qa: str | None = None
     synthesis: str | None = None
+    picture_description: str = (
+        "Describe this image for a blind user. "
+        "State the image type (screenshot, chart, photo, etc.), "
+        "what it depicts, any visible text, and key visual details. "
+        "Be concise and accurate."
+    )
 
 
 class AppConfig(BaseModel):

haiku_rag_slim/haiku/rag/converters/docling_local.py

@@ -11,6 +11,8 @@ from haiku.rag.converters.text_utils import TextFileHandler
 if TYPE_CHECKING:
     from docling_core.types.doc.document import DoclingDocument
 
+    from haiku.rag.config.models import ModelConfig
+
 
 class DoclingLocalConverter(DocumentConverter):
     """Converter that uses local docling for document conversion.
@@ -54,6 +56,21 @@ class DoclingLocalConverter(DocumentConverter):
         """Return list of file extensions supported by this converter."""
         return self.docling_extensions + TextFileHandler.text_extensions
 
+    def _get_vlm_api_url(self, model: "ModelConfig") -> str:
+        """Construct VLM API URL from model config."""
+        if model.base_url:
+            base = model.base_url.rstrip("/")
+            return f"{base}/v1/chat/completions"
+
+        if model.provider == "ollama":
+            base = self.config.providers.ollama.base_url.rstrip("/")
+            return f"{base}/v1/chat/completions"
+
+        if model.provider == "openai":
+            return "https://api.openai.com/v1/chat/completions"
+
+        raise ValueError(f"Unsupported VLM provider: {model.provider}")
+
     def _sync_convert_docling_file(self, path: Path) -> "DoclingDocument":
         """Synchronous conversion of docling-supported files."""
         from docling.backend.docling_parse_backend import DoclingParseDocumentBackend
@@ -61,6 +78,7 @@ class DoclingLocalConverter(DocumentConverter):
         from docling.datamodel.pipeline_options import (
             OcrAutoOptions,
             PdfPipelineOptions,
+            PictureDescriptionApiOptions,
             TableFormerMode,
             TableStructureOptions,
         )
@@ -73,13 +91,14 @@ class DoclingLocalConverter(DocumentConverter):
         )
 
         opts = self.config.processing.conversion_options
+        pic_desc = opts.picture_description
 
         pipeline_options = PdfPipelineOptions(
             do_ocr=opts.do_ocr,
             do_table_structure=opts.do_table_structure,
             images_scale=opts.images_scale,
             generate_page_images=True,
-            generate_picture_images=opts.generate_picture_images,
+            generate_picture_images=opts.generate_picture_images or pic_desc.enabled,
             table_structure_options=TableStructureOptions(
                 do_cell_matching=opts.table_cell_matching,
                 mode=(
@@ -92,8 +111,25 @@ class DoclingLocalConverter(DocumentConverter):
                 force_full_page_ocr=opts.force_ocr,
                 lang=opts.ocr_lang if opts.ocr_lang else [],
             ),
+            do_picture_description=pic_desc.enabled,
         )
 
+        if pic_desc.enabled:
+            from pydantic import AnyUrl
+
+            prompt = self.config.prompts.picture_description
+
+            pipeline_options.enable_remote_services = True
+            pipeline_options.picture_description_options = PictureDescriptionApiOptions(
+                url=AnyUrl(self._get_vlm_api_url(pic_desc.model)),
+                params=dict(
+                    model=pic_desc.model.name,
+                    max_completion_tokens=pic_desc.max_tokens,
+                ),
+                prompt=prompt,
+                timeout=pic_desc.timeout,
+            )
+
         format_options = cast(
             dict[InputFormat, FormatOption],
             {

haiku_rag_slim/haiku/rag/converters/docling_serve.py

@@ -1,18 +1,20 @@
 """docling-serve remote converter implementation."""
 
 import asyncio
+import json
 from pathlib import Path
 from typing import TYPE_CHECKING, ClassVar
 
-import httpx
-
 from haiku.rag.config import AppConfig
 from haiku.rag.converters.base import DocumentConverter
 from haiku.rag.converters.text_utils import TextFileHandler
+from haiku.rag.providers.docling_serve import DoclingServeClient
 
 if TYPE_CHECKING:
     from docling_core.types.doc.document import DoclingDocument
 
+    from haiku.rag.config.models import ModelConfig
+
 
 class DoclingServeConverter(DocumentConverter):
     """Converter that uses docling-serve for document conversion.
@@ -53,17 +55,70 @@ class DoclingServeConverter(DocumentConverter):
             config: Application configuration containing docling-serve settings.
         """
         self.config = config
-        self.base_url = config.providers.docling_serve.base_url.rstrip("/")
+        self.client = DoclingServeClient(
-        self.api_key = config.providers.docling_serve.api_key
+            base_url=config.providers.docling_serve.base_url,
-        self.timeout = config.providers.docling_serve.timeout
+            api_key=config.providers.docling_serve.api_key,
+        )
 
     @property
     def supported_extensions(self) -> list[str]:
         """Return list of file extensions supported by this converter."""
         return self.docling_serve_extensions + TextFileHandler.text_extensions
 
+    def _get_vlm_api_url(self, model: "ModelConfig") -> str:
+        """Construct VLM API URL from model config."""
+        if model.base_url:
+            base = model.base_url.rstrip("/")
+            return f"{base}/v1/chat/completions"
+
+        if model.provider == "ollama":
+            base = self.config.providers.ollama.base_url.rstrip("/")
+            return f"{base}/v1/chat/completions"
+
+        if model.provider == "openai":
+            return "https://api.openai.com/v1/chat/completions"
+
+        raise ValueError(f"Unsupported VLM provider: {model.provider}")
+
+    def _build_conversion_data(self) -> dict[str, str | list[str]]:
+        """Build form data for conversion request."""
+        opts = self.config.processing.conversion_options
+        pic_desc = opts.picture_description
+
+        data: dict[str, str | list[str]] = {
+            "to_formats": "json",
+            "do_ocr": str(opts.do_ocr).lower(),
+            "force_ocr": str(opts.force_ocr).lower(),
+            "do_table_structure": str(opts.do_table_structure).lower(),
+            "table_mode": opts.table_mode,
+            "table_cell_matching": str(opts.table_cell_matching).lower(),
+            "images_scale": str(opts.images_scale),
+            "generate_picture_images": str(
+                opts.generate_picture_images or pic_desc.enabled
+            ).lower(),
+            "do_picture_description": str(pic_desc.enabled).lower(),
+        }
+
+        if opts.ocr_lang:
+            data["ocr_lang"] = opts.ocr_lang
+
+        if pic_desc.enabled:
+            prompt = self.config.prompts.picture_description
+            picture_description_api = {
+                "url": self._get_vlm_api_url(pic_desc.model),
+                "params": {
+                    "model": pic_desc.model.name,
+                    "max_completion_tokens": pic_desc.max_tokens,
+                },
+                "prompt": prompt,
+                "timeout": pic_desc.timeout,
+            }
+            data["picture_description_api"] = json.dumps(picture_description_api)
+
+        return data
+
     async def _make_request(self, files: dict, name: str) -> "DoclingDocument":
-        """Make a request to docling-serve and return the DoclingDocument.
+        """Make an async request to docling-serve and poll for results.
 
         Args:
             files: Dictionary with files parameter for httpx
@@ -77,73 +132,27 @@ class DoclingServeConverter(DocumentConverter):
         """
         from docling_core.types.doc.document import DoclingDocument
 
-        try:
+        data = self._build_conversion_data()
-            opts = self.config.processing.conversion_options
+        result = await self.client.submit_and_poll(
+            endpoint="/v1/convert/file/async",
+            files=files,
+            data=data,
+            name=name,
+        )
 
-            data: dict[str, str | list[str]] = {
+        if result.get("status") not in ("success", "partial_success", None):
-                "to_formats": "json",
+            errors = result.get("errors", [])
-                "do_ocr": str(opts.do_ocr).lower(),
+            raise ValueError(f"Conversion failed: {errors}")
-                "force_ocr": str(opts.force_ocr).lower(),
-                "do_table_structure": str(opts.do_table_structure).lower(),
-                "table_mode": opts.table_mode,
-                "table_cell_matching": str(opts.table_cell_matching).lower(),
-                "images_scale": str(opts.images_scale),
-                "generate_picture_images": str(opts.generate_picture_images).lower(),
-            }
 
-            if opts.ocr_lang:
+        json_content = result.get("document", {}).get("json_content")
-                data["ocr_lang"] = opts.ocr_lang
 
-            headers = {}
+        if json_content is None:
-            if self.api_key:
-                headers["X-Api-Key"] = self.api_key
-
-            url = f"{self.base_url}/v1/convert/file"
-
-            async with httpx.AsyncClient(timeout=self.timeout) as client:
-                response = await client.post(
-                    url,
-                    files=files,
-                    data=data,
-                    headers=headers,
-                )
-                response.raise_for_status()
-                result = response.json()
-
-            if result["status"] not in ("success", "partial_success"):
-                errors = result.get("errors", [])
-                raise ValueError(f"Conversion failed: {errors}")
-
-            json_content = result["document"]["json_content"]
-
-            if json_content is None:
-                raise ValueError(
-                    f"docling-serve did not return JSON content for {name}. "
-                    "This may indicate an unsupported file format."
-                )
-
-            return DoclingDocument.model_validate(json_content)
-
-        except httpx.ConnectError as e:
             raise ValueError(
-                f"Could not connect to docling-serve at {self.base_url}. "
+                f"docling-serve did not return JSON content for {name}. "
-                f"Ensure the service is running and accessible. Error: {e}"
+                "This may indicate an unsupported file format."
             )
-        except httpx.TimeoutException as e:
+
-            raise ValueError(
+        return DoclingDocument.model_validate(json_content)
-                f"Request to docling-serve timed out after {self.timeout}s. "
-                f"Consider increasing the timeout in configuration. Error: {e}"
-            )
-        except httpx.HTTPStatusError as e:
-            if e.response.status_code == 401:
-                raise ValueError(
-                    "Authentication failed. Check your API key configuration."
-                )
-            raise ValueError(f"HTTP error from docling-serve: {e}")
-        except ValueError:
-            raise
-        except Exception as e:
-            raise ValueError(f"Failed to convert via docling-serve: {e}")
 
     async def convert_file(self, path: Path) -> "DoclingDocument":
         """Convert a file to DoclingDocument using docling-serve.

haiku_rag_slim/haiku/rag/providers/__init__.py

@@ -0,0 +1,5 @@
+"""Provider clients for external services."""
+
+from haiku.rag.providers.docling_serve import DoclingServeClient
+
+__all__ = ["DoclingServeClient"]

haiku_rag_slim/haiku/rag/providers/docling_serve.py

@@ -0,0 +1,108 @@
+"""Shared client for docling-serve async API."""
+
+import asyncio
+from typing import Any
+
+import httpx
+
+
+class DoclingServeClient:
+    """Client for docling-serve async workflow.
+
+    Handles the submit → poll → fetch pattern used by both conversion and chunking.
+    """
+
+    def __init__(self, base_url: str, api_key: str | None = None, timeout: float = 300):
+        self.base_url = base_url.rstrip("/")
+        self.api_key = api_key
+        self.timeout = timeout
+
+    def _get_headers(self) -> dict[str, str]:
+        """Get headers for API requests."""
+        headers: dict[str, str] = {}
+        if self.api_key:
+            headers["X-Api-Key"] = self.api_key
+        return headers
+
+    async def submit_and_poll(
+        self,
+        endpoint: str,
+        files: dict[str, Any],
+        data: dict[str, Any],
+        name: str = "document",
+    ) -> dict[str, Any]:
+        """Submit a task and poll until completion.
+
+        Args:
+            endpoint: The async endpoint path (e.g., "/v1/convert/file/async")
+            files: Files to upload
+            data: Form data parameters
+            name: Name for error messages
+
+        Returns:
+            The result dictionary from the completed task
+
+        Raises:
+            ValueError: If the task fails or service is unavailable
+        """
+        headers = self._get_headers()
+
+        try:
+            async with httpx.AsyncClient(timeout=self.timeout) as client:
+                # Submit async task
+                submit_url = f"{self.base_url}{endpoint}"
+                response = await client.post(
+                    submit_url,
+                    files=files,
+                    data=data,
+                    headers=headers,
+                )
+                response.raise_for_status()
+                submit_result = response.json()
+                task_id = submit_result.get("task_id")
+
+                if not task_id:
+                    raise ValueError("docling-serve did not return a task_id")
+
+                # Poll for completion
+                poll_url = f"{self.base_url}/v1/status/poll/{task_id}"
+                while True:
+                    poll_response = await client.get(poll_url, headers=headers)
+                    poll_response.raise_for_status()
+                    poll_result = poll_response.json()
+                    status = poll_result.get("task_status")
+
+                    if status == "success":
+                        break
+                    elif status in ("failure", "error"):
+                        raise ValueError(
+                            f"docling-serve task failed for {name}: {poll_result}"
+                        )
+
+                    await asyncio.sleep(1)
+
+                # Fetch result
+                result_url = f"{self.base_url}/v1/result/{task_id}"
+                result_response = await client.get(result_url, headers=headers)
+                result_response.raise_for_status()
+                return result_response.json()
+
+        except httpx.ConnectError as e:
+            raise ValueError(
+                f"Could not connect to docling-serve at {self.base_url}. "
+                f"Ensure the service is running and accessible. Error: {e}"
+            )
+        except httpx.TimeoutException as e:
+            raise ValueError(
+                f"Request to docling-serve timed out after {self.timeout}s. Error: {e}"
+            )
+        except httpx.HTTPStatusError as e:
+            if e.response.status_code == 401:
+                raise ValueError(
+                    "Authentication failed. Check your API key configuration."
+                )
+            raise ValueError(f"HTTP error from docling-serve: {e}")
+        except ValueError:
+            raise
+        except Exception as e:
+            raise ValueError(f"Failed to process via docling-serve: {e}")

tests/cassettes/test_converters/TestDoclingServeConverterIntegration.test_convert_code_file_real_service.yaml

@@ -1,13 +1,14 @@
 interactions:
 - request:
-    body: "--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
+    body: "--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
-      form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
+      form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
-      form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
+      form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
-      name=\"table_mode\"\r\n\r\naccurate\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
+      name=\"table_mode\"\r\n\r\naccurate\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
-      form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
+      form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
-      name=\"generate_picture_images\"\r\n\r\nfalse\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
+      name=\"generate_picture_images\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
-      name=\"files\"; filename=\"tmpgc9za8mg.md\"\r\nContent-Type: text/markdown\r\n\r\n```python\ndef test():\n    return
+      name=\"do_picture_description\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
-      42\n```\r\n--313fb28cff30a333a81f610f9c6bdaf3--\r\n"
+      name=\"files\"; filename=\"tmp0aa4633w.md\"\r\nContent-Type: text/markdown\r\n\r\n```python\ndef test():\n    return
+      42\n```\r\n--26e45846cd2f5c82db7b770428d74912--\r\n"
     headers:
       accept:
       - '*/*'
@@ -16,23 +17,107 @@ interactions:
       connection:
       - keep-alive
       content-length:
-      - '1011'
+      - '1119'
       content-type:
-      - multipart/form-data; boundary=313fb28cff30a333a81f610f9c6bdaf3
+      - multipart/form-data; boundary=26e45846cd2f5c82db7b770428d74912
       host:
       - localhost:5001
     method: POST
-    uri: http://localhost:5001/v1/convert/file
+    uri: http://localhost:5001/v1/convert/file/async
   response:
     headers:
       content-length:
-      - '1113'
+      - '131'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 7436648c-490d-4938-b42b-9755823f8d78
+      task_meta: null
+      task_position: 1
+      task_status: pending
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/status/poll/7436648c-490d-4938-b42b-9755823f8d78
+  response:
+    headers:
+      content-length:
+      - '134'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 7436648c-490d-4938-b42b-9755823f8d78
+      task_meta: null
+      task_position: null
+      task_status: started
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/status/poll/7436648c-490d-4938-b42b-9755823f8d78
+  response:
+    headers:
+      content-length:
+      - '134'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 7436648c-490d-4938-b42b-9755823f8d78
+      task_meta: null
+      task_position: null
+      task_status: success
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/result/7436648c-490d-4938-b42b-9755823f8d78
+  response:
+    headers:
+      content-length:
+      - '1114'
       content-type:
       - application/json
     parsed_body:
       document:
         doctags_content: null
-        filename: tmpgc9za8mg.md
+        filename: tmp0aa4633w.md
         html_content: null
         json_content:
           body:
@@ -55,10 +140,10 @@ interactions:
             self_ref: '#/furniture'
           groups: []
           key_value_items: []
-          name: tmpgc9za8mg
+          name: tmp0aa4633w
           origin:
             binary_hash: 9008975733065065710
-            filename: tmpgc9za8mg.md
+            filename: tmp0aa4633w.md
             mimetype: text/markdown
             uri: null
           pages: {}
@@ -91,7 +176,7 @@ interactions:
         md_content: null
         text_content: null
       errors: []
-      processing_time: 0.002258999999980915
+      processing_time: 0.0027105000044684857
       status: success
       timings: {}
     status:

tests/cassettes/test_converters/TestDoclingServeConverterIntegration.test_convert_text_real_service.yaml

@@ -1,12 +1,13 @@
 interactions:
 - request:
-    body: "--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
+    body: "--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
-      form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
+      form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
-      form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
+      form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
-      name=\"table_mode\"\r\n\r\naccurate\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
+      name=\"table_mode\"\r\n\r\naccurate\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
-      form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
+      form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
-      name=\"generate_picture_images\"\r\n\r\nfalse\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
+      name=\"generate_picture_images\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
-      name=\"files\"; filename=\"content.md\"\r\nContent-Type: text/markdown\r\n\r\n# Test Document\n\nThis is a test.\r\n--a906caa5bbe5d30e17b44fb63c456240--\r\n"
+      name=\"do_picture_description\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
+      name=\"files\"; filename=\"content.md\"\r\nContent-Type: text/markdown\r\n\r\n# Test Document\n\nThis is a test.\r\n--9df242c136eddc0594e7eb126b1f39cd--\r\n"
     headers:
       accept:
       - '*/*'
@@ -15,13 +16,97 @@ interactions:
       connection:
       - keep-alive
       content-length:
-      - '1000'
+      - '1108'
       content-type:
-      - multipart/form-data; boundary=a906caa5bbe5d30e17b44fb63c456240
+      - multipart/form-data; boundary=9df242c136eddc0594e7eb126b1f39cd
       host:
       - localhost:5001
     method: POST
-    uri: http://localhost:5001/v1/convert/file
+    uri: http://localhost:5001/v1/convert/file/async
+  response:
+    headers:
+      content-length:
+      - '131'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
+      task_meta: null
+      task_position: 1
+      task_status: pending
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/status/poll/9f7f1455-1ab6-4830-b5a5-3e63419379ae
+  response:
+    headers:
+      content-length:
+      - '134'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
+      task_meta: null
+      task_position: null
+      task_status: started
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/status/poll/9f7f1455-1ab6-4830-b5a5-3e63419379ae
+  response:
+    headers:
+      content-length:
+      - '134'
+      content-type:
+      - application/json
+    parsed_body:
+      task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
+      task_meta: null
+      task_position: null
+      task_status: success
+      task_type: convert
+    status:
+      code: 200
+      message: OK
+- request:
+    body: ''
+    headers:
+      accept:
+      - '*/*'
+      accept-encoding:
+      - gzip, deflate, zstd
+      connection:
+      - keep-alive
+      host:
+      - localhost:5001
+    method: GET
+    uri: http://localhost:5001/v1/result/9f7f1455-1ab6-4830-b5a5-3e63419379ae
   response:
     headers:
       content-length:
@@ -94,7 +179,7 @@ interactions:
         md_content: null
         text_content: null
       errors: []
-      processing_time: 0.004331874999934371
+      processing_time: 0.004173958994215354
       status: success
       timings: {}
     status:

tests/test_chunker.py

@@ -232,6 +232,28 @@ def test_get_chunker_docling_serve():
     assert isinstance(chunker, DoclingServeChunker)
 
 
+def create_async_workflow_mocks(
+    result_data: dict, task_id: str = "test-task-123"
+) -> tuple[Mock, Mock, Mock]:
+    """Create mock responses for docling-serve async workflow."""
+    submit_response = Mock()
+    submit_response.status_code = 200
+    submit_response.json.return_value = {"task_id": task_id, "task_status": "pending"}
+    submit_response.raise_for_status = Mock()
+
+    poll_response = Mock()
+    poll_response.status_code = 200
+    poll_response.json.return_value = {"task_id": task_id, "task_status": "success"}
+    poll_response.raise_for_status = Mock()
+
+    result_response = Mock()
+    result_response.status_code = 200
+    result_response.json.return_value = result_data
+    result_response.raise_for_status = Mock()
+
+    return submit_response, poll_response, result_response
+
+
 class TestDoclingServeChunker:
     """Tests for DoclingServeChunker (mocked)."""
 
@@ -241,7 +263,6 @@ class TestDoclingServeChunker:
         config = AppConfig()
         config.providers.docling_serve.base_url = "http://localhost:5001"
         config.providers.docling_serve.api_key = ""
-        config.providers.docling_serve.timeout = 300
         config.processing.chunk_size = 256
         config.processing.chunking_tokenizer = "Qwen/Qwen3-Embedding-0.6B"
         return config
@@ -252,21 +273,20 @@ class TestDoclingServeChunker:
         return DoclingServeChunker(config)
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_success(self, mock_client_class, chunker):
-        """Test successful chunking via docling-serve."""
+        """Test successful chunking via docling-serve async workflow."""
-        mock_response = Mock()
+        result_data = {
-        mock_response.status_code = 200
-        mock_response.json.return_value = {
             "chunks": [
                 {"text": "Chunk 1", "chunk_index": 0},
                 {"text": "Chunk 2", "chunk_index": 1},
             ]
         }
-        mock_response.raise_for_status = Mock()
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
 
         mock_client = AsyncMock()
-        mock_client.post.return_value = mock_response
+        mock_client.post = AsyncMock(return_value=submit_resp)
+        mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
         mock_client_class.return_value.__aenter__.return_value = mock_client
 
         # Create a simple document
@@ -280,21 +300,18 @@ class TestDoclingServeChunker:
         mock_client.post.assert_called_once()
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_with_api_key(self, mock_client_class, config):
         """Test that API key is included in request headers."""
         config.providers.docling_serve.api_key = "test-key"
         chunker = DoclingServeChunker(config)
 
-        mock_response = Mock()
+        result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
-        mock_response.json.return_value = {
-            "chunks": [{"text": "Chunk 1", "chunk_index": 0}]
-        }
-        mock_response.raise_for_status = Mock()
 
         mock_client = AsyncMock()
-        mock_client.post.return_value = mock_response
+        mock_client.post = AsyncMock(return_value=submit_resp)
+        mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
         mock_client_class.return_value.__aenter__.return_value = mock_client
 
         converter = get_converter(Config)
@@ -306,21 +323,18 @@ class TestDoclingServeChunker:
         assert call_kwargs["headers"]["X-Api-Key"] == "test-key"
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_hierarchical_endpoint(self, mock_client_class, config):
         """Test that hierarchical chunker uses correct endpoint."""
         config.processing.chunker_type = "hierarchical"
         chunker = DoclingServeChunker(config)
 
-        mock_response = Mock()
+        result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
-        mock_response.json.return_value = {
-            "chunks": [{"text": "Chunk 1", "chunk_index": 0}]
-        }
-        mock_response.raise_for_status = Mock()
 
         mock_client = AsyncMock()
-        mock_client.post.return_value = mock_response
+        mock_client.post = AsyncMock(return_value=submit_resp)
+        mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
         mock_client_class.return_value.__aenter__.return_value = mock_client
 
         converter = get_converter(Config)
@@ -328,10 +342,10 @@ class TestDoclingServeChunker:
         await chunker.chunk(doc)
 
         call_args = mock_client.post.call_args
-        assert "/v1/chunk/hierarchical/file" in call_args[0][0]
+        assert "/v1/chunk/hierarchical/file/async" in call_args[0][0]
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_passes_config_parameters(self, mock_client_class, config):
         """Test that all config parameters are passed to API."""
         config.processing.chunk_size = 512
@@ -339,15 +353,12 @@ class TestDoclingServeChunker:
         config.processing.chunking_use_markdown_tables = True
         chunker = DoclingServeChunker(config)
 
-        mock_response = Mock()
+        result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
-        mock_response.json.return_value = {
-            "chunks": [{"text": "Chunk 1", "chunk_index": 0}]
-        }
-        mock_response.raise_for_status = Mock()
 
         mock_client = AsyncMock()
-        mock_client.post.return_value = mock_response
+        mock_client.post = AsyncMock(return_value=submit_resp)
+        mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
         mock_client_class.return_value.__aenter__.return_value = mock_client
 
         converter = get_converter(Config)
@@ -361,7 +372,7 @@ class TestDoclingServeChunker:
         assert data["chunking_use_markdown_tables"] == "true"
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_connection_error(self, mock_client_class, chunker):
         """Test handling of connection errors."""
         import httpx
@@ -377,7 +388,7 @@ class TestDoclingServeChunker:
             await chunker.chunk(doc)
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_timeout_error(self, mock_client_class, chunker):
         """Test handling of timeout errors."""
         import httpx
@@ -393,7 +404,7 @@ class TestDoclingServeChunker:
             await chunker.chunk(doc)
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_auth_error(self, mock_client_class, chunker):
         """Test handling of authentication errors."""
         import httpx
@@ -416,18 +427,14 @@ class TestDoclingServeChunker:
             await chunker.chunk(doc)
 
     @pytest.mark.asyncio
-    @patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
+    @patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
     async def test_chunk_metadata_extraction(self, mock_client_class, chunker):
         """Test that metadata is correctly extracted from API response.
 
         Labels are resolved from the DoclingDocument using the refs, so we need
         to create a document with matching structure for the mocked API response.
         """
-        mock_response = Mock()
+        result_data = {
-        mock_response.status_code = 200
-        # docling-serve returns doc_items as list of ref strings
-        # We'll reference texts[0], texts[1], and tables[0]
-        mock_response.json.return_value = {
             "chunks": [
                 {
                     "text": "Chapter 1\nThis is content.",
@@ -443,10 +450,11 @@ class TestDoclingServeChunker:
                 },
             ]
         }
-        mock_response.raise_for_status = Mock()
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
 
         mock_client = AsyncMock()
-        mock_client.post.return_value = mock_response
+        mock_client.post = AsyncMock(return_value=submit_resp)
+        mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
         mock_client_class.return_value.__aenter__.return_value = mock_client
 
         # Create a document with texts and tables that match the mocked refs

tests/test_converters.py

@@ -54,6 +54,31 @@ def create_mock_docling_document_json(name: str = "test") -> dict:
     }
 
 
+def create_async_workflow_mocks(
+    doc_json: dict, task_id: str = "test-task-123"
+) -> tuple[Mock, Mock, Mock]:
+    """Create mock responses for docling-serve async workflow.
+
+    Returns tuple of (submit_response, poll_response, result_response).
+    """
+    submit_response = Mock()
+    submit_response.status_code = 200
+    submit_response.json.return_value = {"task_id": task_id, "task_status": "pending"}
+    submit_response.raise_for_status = Mock()
+
+    poll_response = Mock()
+    poll_response.status_code = 200
+    poll_response.json.return_value = {"task_id": task_id, "task_status": "success"}
+    poll_response.raise_for_status = Mock()
+
+    result_response = Mock()
+    result_response.status_code = 200
+    result_response.json.return_value = {"document": {"json_content": doc_json}}
+    result_response.raise_for_status = Mock()
+
+    return submit_response, poll_response, result_response
+
+
 class TestTextFileHandler:
     """Tests for TextFileHandler utility class."""
 
@@ -308,6 +333,117 @@ class TestDoclingLocalConverter:
                 "Pictures should have image data when generate_picture_images=True"
             )
 
+    def test_get_vlm_api_url_with_ollama(self, config):
+        """Test VLM API URL construction for Ollama provider."""
+        converter = DoclingLocalConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="ollama", name="ministral-3")
+        url = converter._get_vlm_api_url(model)
+        assert url == "http://localhost:11434/v1/chat/completions"
+
+    def test_get_vlm_api_url_with_custom_base_url(self, config):
+        """Test VLM API URL construction with custom base_url."""
+        converter = DoclingLocalConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(
+            provider="openai", name="gpt-4-vision", base_url="http://my-vllm:8000"
+        )
+        url = converter._get_vlm_api_url(model)
+        assert url == "http://my-vllm:8000/v1/chat/completions"
+
+    def test_get_vlm_api_url_with_openai(self, config):
+        """Test VLM API URL construction for OpenAI provider."""
+        converter = DoclingLocalConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="openai", name="gpt-4-vision")
+        url = converter._get_vlm_api_url(model)
+        assert url == "https://api.openai.com/v1/chat/completions"
+
+    def test_get_vlm_api_url_unsupported_provider(self, config):
+        """Test VLM API URL construction raises error for unsupported provider."""
+        converter = DoclingLocalConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="unsupported", name="test")
+        with pytest.raises(ValueError, match="Unsupported VLM provider"):
+            converter._get_vlm_api_url(model)
+
+    def test_picture_description_config_defaults(self, config):
+        """Test that picture description config has correct defaults."""
+        assert config.processing.conversion_options.picture_description.enabled is False
+        assert (
+            config.processing.conversion_options.picture_description.model.provider
+            == "ollama"
+        )
+        assert (
+            config.processing.conversion_options.picture_description.model.name
+            == "ministral-3"
+        )
+        assert config.processing.conversion_options.picture_description.timeout == 90
+        assert (
+            config.processing.conversion_options.picture_description.max_tokens == 200
+        )
+        # Default prompt is in PromptsConfig
+        assert "blind user" in config.prompts.picture_description
+
+    def test_picture_description_config_applied(self, config):
+        """Test that picture description config is applied to converter."""
+        config.processing.conversion_options.picture_description.enabled = True
+        config.processing.conversion_options.picture_description.timeout = 120
+        converter = DoclingLocalConverter(config)
+
+        pic_desc = converter.config.processing.conversion_options.picture_description
+        assert pic_desc.enabled is True
+        assert pic_desc.timeout == 120
+
+    @pytest.mark.asyncio
+    @pytest.mark.integration
+    @pytest.mark.vcr()
+    async def test_picture_description_end_to_end(self, config):
+        """End-to-end test: convert PDF with VLM picture descriptions."""
+        pdf_path = Path("tests/data/doclaynet.pdf")
+        if not pdf_path.exists():
+            pytest.skip("doclaynet.pdf not found")
+
+        # Enable picture description with Ollama
+        config.processing.conversion_options.picture_description.enabled = True
+        config.processing.conversion_options.picture_description.model.provider = (
+            "ollama"
+        )
+        config.processing.conversion_options.picture_description.model.name = (
+            "ministral-3"
+        )
+
+        converter = DoclingLocalConverter(config)
+        doc = await converter.convert_file(pdf_path)
+
+        # Export to markdown and check for picture descriptions
+        markdown = doc.export_to_markdown()
+
+        # The document should have pictures with descriptions
+        assert doc.pictures, "Document should have pictures"
+
+        # Check that at least one picture has a description annotation
+        from docling_core.types.doc.document import PictureDescriptionData
+
+        pictures_with_descriptions = []
+        for pic in doc.pictures:
+            for ann in pic.annotations:
+                if isinstance(ann, PictureDescriptionData):
+                    pictures_with_descriptions.append(pic)
+                    # Description should appear in markdown output
+                    assert ann.text in markdown, (
+                        f"Picture description '{ann.text[:50]}...' should be in markdown"
+                    )
+                    break
+
+        assert pictures_with_descriptions, (
+            "At least one picture should have a VLM description"
+        )
+
 
 class TestDoclingServeConverter:
     """Tests for DoclingServeConverter (mocked)."""
@@ -318,7 +454,6 @@ class TestDoclingServeConverter:
         config = AppConfig()
         config.providers.docling_serve.base_url = "http://localhost:5001"
         config.providers.docling_serve.api_key = ""
-        config.providers.docling_serve.timeout = 300
         return config
 
     @pytest.fixture
@@ -328,8 +463,7 @@ class TestDoclingServeConverter:
 
     def test_initialization(self, converter):
         """Test converter initialization."""
-        assert converter.base_url == "http://localhost:5001"
+        assert converter.client.base_url == "http://localhost:5001"
-        assert converter.timeout == 300
 
     def test_supported_extensions(self, converter):
         """Test that converter reports correct supported extensions."""
@@ -341,18 +475,14 @@ class TestDoclingServeConverter:
 
     @pytest.mark.asyncio
     async def test_convert_text_success(self, converter):
-        """Test successful text conversion via docling-serve."""
+        """Test successful text conversion via docling-serve async workflow."""
-        mock_response = Mock()
+        doc_json = create_mock_docling_document_json("test")
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
-        mock_response.json.return_value = {
-            "status": "success",
-            "document": {"json_content": create_mock_docling_document_json("test")},
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -368,17 +498,13 @@ class TestDoclingServeConverter:
         config.providers.docling_serve.api_key = "test-key"
         converter = DoclingServeConverter(config)
 
-        mock_response = Mock()
+        doc_json = create_mock_docling_document_json("test")
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
-        mock_response.json.return_value = {
-            "status": "success",
-            "document": {"json_content": create_mock_docling_document_json("test")},
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -401,17 +527,13 @@ class TestDoclingServeConverter:
         config.processing.conversion_options.images_scale = 3.0
         converter = DoclingServeConverter(config)
 
-        mock_response = Mock()
+        doc_json = create_mock_docling_document_json("test")
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
-        mock_response.json.return_value = {
-            "status": "success",
-            "document": {"json_content": create_mock_docling_document_json("test")},
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -480,17 +602,16 @@ class TestDoclingServeConverter:
     @pytest.mark.asyncio
     async def test_convert_text_no_json_content(self, converter):
         """Test handling when docling-serve returns no JSON content."""
-        mock_response = Mock()
+        submit_resp, poll_resp, _ = create_async_workflow_mocks({})
-        mock_response.status_code = 200
+        result_resp = Mock()
-        mock_response.json.return_value = {
+        result_resp.status_code = 200
-            "status": "success",
+        result_resp.json.return_value = {"document": {"json_content": None}}
-            "document": {"json_content": None},
+        result_resp.raise_for_status = Mock()
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -500,18 +621,14 @@ class TestDoclingServeConverter:
 
     @pytest.mark.asyncio
     async def test_convert_file_pdf(self, converter):
-        """Test converting PDF file via docling-serve."""
+        """Test converting PDF file via docling-serve async workflow."""
-        mock_response = Mock()
+        doc_json = create_mock_docling_document_json("test")
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
-        mock_response.json.return_value = {
-            "status": "success",
-            "document": {"json_content": create_mock_docling_document_json("test")},
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -528,17 +645,13 @@ class TestDoclingServeConverter:
     @pytest.mark.asyncio
     async def test_convert_file_text(self, converter):
         """Test converting text file (reads locally, sends to docling-serve)."""
-        mock_response = Mock()
+        doc_json = create_mock_docling_document_json("test")
-        mock_response.status_code = 200
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
-        mock_response.json.return_value = {
-            "status": "success",
-            "document": {"json_content": create_mock_docling_document_json("test")},
-        }
-        mock_response.raise_for_status = Mock()
 
         with patch("httpx.AsyncClient") as mock_client_class:
             mock_client = AsyncMock()
-            mock_client.post = AsyncMock(return_value=mock_response)
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
             mock_client.__aenter__ = AsyncMock(return_value=mock_client)
             mock_client.__aexit__ = AsyncMock(return_value=None)
             mock_client_class.return_value = mock_client
@@ -555,6 +668,124 @@ class TestDoclingServeConverter:
             assert "files" in call_kwargs
 
 
+class TestDoclingServeConverterPictureDescription:
+    """Tests for DoclingServeConverter picture description support."""
+
+    @pytest.fixture
+    def config(self):
+        """Create test configuration."""
+        config = AppConfig()
+        config.providers.docling_serve.base_url = "http://localhost:5001"
+        config.providers.docling_serve.api_key = ""
+        return config
+
+    def test_get_vlm_api_url_with_ollama(self, config):
+        """Test VLM API URL construction for Ollama provider."""
+        converter = DoclingServeConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="ollama", name="ministral-3")
+        url = converter._get_vlm_api_url(model)
+        assert url == "http://localhost:11434/v1/chat/completions"
+
+    def test_get_vlm_api_url_with_custom_base_url(self, config):
+        """Test VLM API URL construction with custom base_url."""
+        converter = DoclingServeConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(
+            provider="openai", name="gpt-4-vision", base_url="http://my-vllm:8000"
+        )
+        url = converter._get_vlm_api_url(model)
+        assert url == "http://my-vllm:8000/v1/chat/completions"
+
+    def test_get_vlm_api_url_with_openai(self, config):
+        """Test VLM API URL construction for OpenAI provider."""
+        converter = DoclingServeConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="openai", name="gpt-4-vision")
+        url = converter._get_vlm_api_url(model)
+        assert url == "https://api.openai.com/v1/chat/completions"
+
+    def test_get_vlm_api_url_unsupported_provider(self, config):
+        """Test VLM API URL construction raises error for unsupported provider."""
+        converter = DoclingServeConverter(config)
+        from haiku.rag.config.models import ModelConfig
+
+        model = ModelConfig(provider="unsupported", name="test")
+        with pytest.raises(ValueError, match="Unsupported VLM provider"):
+            converter._get_vlm_api_url(model)
+
+    @pytest.mark.asyncio
+    async def test_picture_description_options_passed_to_api(self, config):
+        """Test that picture description options are passed to docling-serve API."""
+        import json
+
+        config.processing.conversion_options.picture_description.enabled = True
+        config.processing.conversion_options.picture_description.model.provider = (
+            "ollama"
+        )
+        config.processing.conversion_options.picture_description.model.name = (
+            "ministral-3"
+        )
+        config.processing.conversion_options.picture_description.timeout = 120
+        config.processing.conversion_options.picture_description.max_tokens = 300
+        config.prompts.picture_description = "Test prompt for picture description"
+        converter = DoclingServeConverter(config)
+
+        doc_json = create_mock_docling_document_json("test")
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
+
+        with patch("httpx.AsyncClient") as mock_client_class:
+            mock_client = AsyncMock()
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
+            mock_client.__aenter__ = AsyncMock(return_value=mock_client)
+            mock_client.__aexit__ = AsyncMock(return_value=None)
+            mock_client_class.return_value = mock_client
+
+            await converter.convert_text("# Test")
+
+            call_kwargs = mock_client.post.call_args.kwargs
+            assert "data" in call_kwargs
+            data = call_kwargs["data"]
+
+            assert data["do_picture_description"] == "true"
+            assert data["generate_picture_images"] == "true"
+            assert "picture_description_api" in data
+
+            api_config = json.loads(data["picture_description_api"])
+            assert api_config["url"] == "http://localhost:11434/v1/chat/completions"
+            assert api_config["params"]["model"] == "ministral-3"
+            assert api_config["params"]["max_completion_tokens"] == 300
+            assert api_config["prompt"] == "Test prompt for picture description"
+            assert api_config["timeout"] == 120
+
+    @pytest.mark.asyncio
+    async def test_picture_description_disabled_by_default(self, config):
+        """Test that picture description is disabled by default."""
+        converter = DoclingServeConverter(config)
+
+        doc_json = create_mock_docling_document_json("test")
+        submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
+
+        with patch("httpx.AsyncClient") as mock_client_class:
+            mock_client = AsyncMock()
+            mock_client.post = AsyncMock(return_value=submit_resp)
+            mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
+            mock_client.__aenter__ = AsyncMock(return_value=mock_client)
+            mock_client.__aexit__ = AsyncMock(return_value=None)
+            mock_client_class.return_value = mock_client
+
+            await converter.convert_text("# Test")
+
+            call_kwargs = mock_client.post.call_args.kwargs
+            data = call_kwargs["data"]
+            assert data["do_picture_description"] == "false"
+            assert "picture_description_api" not in data
+
+
 class TestDoclingServeConverterIntegration:
     """Integration tests with real docling-serve recorded via VCR."""
 
@@ -591,3 +822,48 @@ class TestDoclingServeConverterIntegration:
         assert isinstance(doc, DoclingDocument)
         result = doc.export_to_markdown()
         assert "def test():" in result
+
+    @pytest.mark.asyncio
+    @pytest.mark.integration
+    async def test_picture_description_end_to_end(self, config):
+        """End-to-end test: convert PDF with VLM picture descriptions via docling-serve.
+
+        Note: Not using VCR because this test involves polling with changing task IDs.
+        """
+        pdf_path = Path("tests/data/doclaynet.pdf")
+        if not pdf_path.exists():
+            pytest.skip("doclaynet.pdf not found")
+
+        config.processing.conversion_options.picture_description.enabled = True
+        config.processing.conversion_options.picture_description.model.provider = (
+            "ollama"
+        )
+        config.processing.conversion_options.picture_description.model.name = (
+            "ministral-3"
+        )
+        # Use host.docker.internal so docling-serve in Docker can reach host's Ollama
+        config.processing.conversion_options.picture_description.model.base_url = (
+            "http://host.docker.internal:11434"
+        )
+        converter = DoclingServeConverter(config)
+
+        doc = await converter.convert_file(pdf_path)
+
+        assert doc.pictures, "Document should have pictures"
+
+        from docling_core.types.doc.document import PictureDescriptionData
+
+        pictures_with_descriptions = []
+        markdown = doc.export_to_markdown()
+        for pic in doc.pictures:
+            for ann in pic.annotations:
+                if isinstance(ann, PictureDescriptionData):
+                    pictures_with_descriptions.append(pic)
+                    assert ann.text in markdown, (
+                        f"Picture description '{ann.text[:50]}...' should be in markdown"
+                    )
+                    break
+
+        assert pictures_with_descriptions, (
+            "At least one picture should have a VLM description"
+        )