danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #218 from ggozad/feat/vlm-image-handling

Browse source 
Support for VLM-based picture description for embedded images
This commit is contained in:
Yiorgis Gozadinos 2026-01-07 11:18:23 +02:00 committed by GitHub
commit 8939b8a313
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
20 changed files with 31387 additions and 14958 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
CHANGELOG.md
View file

@ -1,6 +1,15 @@
# Changelog
## [Unreleased]
### Added
- **VLM Picture Description**: Describe embedded images using Vision Language Models during document conversion
- Images are sent to a VLM for automatic description via OpenAI-compatible API
- Descriptions become searchable text, improving RAG retrieval for visual content
- Configure via `processing.conversion_options.picture_description` with `enabled`, `model`, `timeout`, `max_tokens`
- Default prompt customizable via `prompts.picture_description`
- Requires OpenAI-compatible `/v1/chat/completions` endpoint (Ollama, OpenAI, vLLM, LM Studio)
## [0.23.2] - 2026-01-05
### Fixed

94
docs/configuration/processing.md
View file

@ -36,6 +36,13 @@ processing:
# Image settings
images_scale: 2.0 # Image scale factor
generate_picture_images: false # Include embedded images in output
# VLM picture description (optional)
picture_description:
enabled: false # Enable VLM image descriptions
model:
provider: ollama
name: ministral-3
```
### Conversion Options
@ -81,6 +88,92 @@ conversion_options:
- **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
- **generate_picture_images**: When `true`, embedded images (figures, diagrams) are included as base64-encoded data in the document. When `false` (default), images are excluded to reduce chunk size and avoid context bloat.
#### Picture Description (VLM)
Use a Vision Language Model (VLM) to automatically describe images in documents. Descriptions become searchable text, improving RAG retrieval for visual content.
```yaml
conversion_options:
picture_description:
enabled: true # Enable VLM picture description
model:
provider: ollama # ollama, openai, or custom
name: ministral-3 # VLM model name
timeout: 90 # Request timeout in seconds
max_tokens: 200 # Maximum tokens in response
```
**Configuration options:**
- **enabled**: When `true`, each embedded image is sent to a VLM for description. Requires `generate_picture_images` to be `true` (automatically enabled).
- **model**: Standard model configuration
- `provider`: `ollama` (default), `openai`, or use `base_url` for custom endpoints
- `name`: Model name (e.g., `ministral-3`, `granite3.2-vision`, `gpt-4-vision`)
- `base_url`: Optional custom API endpoint for vLLM, LM Studio, etc.
- **timeout**: Request timeout in seconds
- **max_tokens**: Maximum tokens in the VLM response
**Note:** Requires an OpenAI-compatible `/v1/chat/completions` endpoint. Providers with different API formats (e.g., Anthropic Claude) are not supported.
**Default prompt** (configured in `prompts.picture_description`):
```
Describe this image for a blind user. State the image type
(screenshot, chart, photo, etc.), what it depicts, any visible text,
and key visual details. Be concise and accurate.
```
To customize the prompt globally:
```yaml
prompts:
picture_description: "Your custom prompt here..."
```
**Using with Ollama:**
```yaml
conversion_options:
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
```
Requires Ollama running with a vision-capable model:
```bash
ollama pull ministral-3
ollama serve
```
**Using with vLLM or custom endpoints:**
```yaml
conversion_options:
picture_description:
enabled: true
model:
provider: openai # Use OpenAI-compatible API format
name: granite-vision
base_url: http://my-vllm-server:8000
```
**How it works:**
1. During PDF conversion, docling extracts embedded images
2. Each image is sent to the configured VLM for description
3. Descriptions are added as annotations on the image
4. When exported to markdown, descriptions appear as searchable text
**Using with docling-serve:**
When using `converter: docling-serve`, the VLM calls are made by the docling-serve instance, not by haiku.rag. You must:
1. Set `DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` when running docling-serve
2. Ensure the VLM endpoint is accessible from where docling-serve is running
### Local vs Remote Processing
**Local processing** (default):
@ -106,7 +199,6 @@ providers:
docling_serve:
base_url: http://localhost:5001
api_key: "your-api-key" # Optional
timeout: 300 # Request timeout in seconds
```
Conversion options work identically for both local and remote processing.

27
docs/configuration/prompts.md
View file

@ -16,6 +16,9 @@ prompts:
# Full replacement for research synthesis prompt (optional)
synthesis: null
# VLM prompt for image description during conversion (optional)
picture_description: null # Uses default prompt
```
## Domain Preamble
@ -92,6 +95,29 @@ prompts:
- Avoid meta-commentary like "This report covers..."
```
## Picture Description Prompt
Customize the prompt used when generating VLM descriptions for embedded images during document conversion. This prompt is sent to the configured Vision Language Model for each image.
**Default prompt:**
```
Describe this image for a blind user. State the image type (screenshot, chart, photo, etc.),
what it depicts, any visible text, and key visual details. Be concise and accurate.
```
**Custom example:**
```yaml
prompts:
picture_description: |
Describe this image for a document search system.
Focus on: image type, main content, any text, key visual elements.
Be concise and factual.
```
The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Description (VLM)](processing.md#picture-description-vlm) for full configuration.
## Programmatic Configuration
```python
@ -103,6 +129,7 @@ config = AppConfig(
domain_preamble="You are answering questions about our product documentation.",
qa=None, # Use default QA prompt
synthesis=None, # Use default synthesis prompt
picture_description="Describe this image for search indexing.",
)
)
```

3
docs/remote-processing.md
View file

@ -55,8 +55,7 @@ processing:
providers:
docling_serve:
base_url: http://localhost:5001
api_key: "" # Optional API key for authentication
timeout: 300 # Request timeout in seconds
api_key: "" # Optional API key for authentication
```
## Features

1
examples/ag-ui-research/haiku.rag.yaml.example
View file

@ -12,7 +12,6 @@ providers:
docling_serve:
base_url: http://docling-serve:5001
api_key: ""
timeout: 300
ollama:
base_url: http://host.docker.internal:11434

1
examples/docker/haiku.rag.yaml.example
View file

@ -24,7 +24,6 @@ providers:
docling_serve:
base_url: http://docling-serve:5001
api_key: ""
timeout: 300
ollama:
base_url: http://host.docker.internal:11434

80
haiku_rag_slim/haiku/rag/chunkers/docling_serve.py
View file

@ -2,10 +2,9 @@ import re
from io import BytesIO
from typing import TYPE_CHECKING
import httpx
from haiku.rag.chunkers.base import DocumentChunker
from haiku.rag.config import AppConfig, Config
from haiku.rag.providers.docling_serve import DoclingServeClient
from haiku.rag.store.models.chunk import Chunk, ChunkMetadata
if TYPE_CHECKING:
@ -56,11 +55,25 @@ class DoclingServeChunker(DocumentChunker):
def __init__(self, config: AppConfig = Config):
self.config = config
self.base_url = config.providers.docling_serve.base_url.rstrip("/")
self.api_key = config.providers.docling_serve.api_key
self.timeout = config.providers.docling_serve.timeout
self.client = DoclingServeClient(
base_url=config.providers.docling_serve.base_url,
api_key=config.providers.docling_serve.api_key,
)
self.chunker_type = config.processing.chunker_type
def _build_chunking_data(self) -> dict[str, str]:
"""Build form data for chunking request."""
return {
"chunking_max_tokens": str(self.config.processing.chunk_size),
"chunking_tokenizer": self.config.processing.chunking_tokenizer,
"chunking_merge_peers": str(
self.config.processing.chunking_merge_peers
).lower(),
"chunking_use_markdown_tables": str(
self.config.processing.chunking_use_markdown_tables
).lower(),
}
async def _call_chunk_api(self, document: "DoclingDocument") -> list[dict]:
"""Call docling-serve chunking API and return raw chunk data.
@ -75,9 +88,9 @@ class DoclingServeChunker(DocumentChunker):
"""
# Determine endpoint based on chunker_type
if self.chunker_type == "hierarchical":
url = f"{self.base_url}/v1/chunk/hierarchical/file"
endpoint = "/v1/chunk/hierarchical/file/async"
else:
url = f"{self.base_url}/v1/chunk/hybrid/file"
endpoint = "/v1/chunk/hybrid/file/async"
# Export document to JSON
doc_json = document.model_dump_json()
@ -85,53 +98,16 @@ class DoclingServeChunker(DocumentChunker):
# Prepare multipart request with DoclingDocument JSON
files = {"files": ("document.json", BytesIO(doc_bytes), "application/json")}
data = self._build_chunking_data()
# Build form data with chunking parameters
data = {
"chunking_max_tokens": str(self.config.processing.chunk_size),
"chunking_tokenizer": self.config.processing.chunking_tokenizer,
"chunking_merge_peers": str(
self.config.processing.chunking_merge_peers
).lower(),
"chunking_use_markdown_tables": str(
self.config.processing.chunking_use_markdown_tables
).lower(),
}
result = await self.client.submit_and_poll(
endpoint=endpoint,
files=files,
data=data,
name="document",
)
headers = {}
if self.api_key:
headers["X-Api-Key"] = self.api_key
try:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
url,
files=files,
data=data,
headers=headers,
)
response.raise_for_status()
result = response.json()
return result.get("chunks", [])
except httpx.ConnectError as e:
raise ValueError(
f"Could not connect to docling-serve at {self.base_url}. "
f"Ensure the service is running and accessible. Error: {e}"
)
except httpx.TimeoutException as e:
raise ValueError(
f"Request to docling-serve timed out after {self.timeout}s. "
f"Consider increasing the timeout in configuration. Error: {e}"
)
except httpx.HTTPStatusError as e:
if e.response.status_code == 401:
raise ValueError(
"Authentication failed. Check your API key configuration."
)
raise ValueError(f"HTTP error from docling-serve: {e}")
except Exception as e:
raise ValueError(f"Failed to chunk via docling-serve: {e}")
return result.get("chunks", [])
async def chunk(self, document: "DoclingDocument") -> list[Chunk]:
"""Split the document into chunks with metadata via docling-serve.

3
haiku_rag_slim/haiku/rag/client.py
View file

@ -1672,6 +1672,9 @@ class HaikuRAG:
and self._config.reranking.model.provider == "ollama"
):
required_models.add(self._config.reranking.model.name)
pic_desc = self._config.processing.conversion_options.picture_description
if pic_desc.enabled and pic_desc.model.provider == "ollama":
required_models.add(pic_desc.model.name)
if not required_models:
return

26
haiku_rag_slim/haiku/rag/config/models.py
View file

@ -96,6 +96,20 @@ class ResearchConfig(BaseModel):
max_concurrency: int = 1
class PictureDescriptionConfig(BaseModel):
"""Configuration for VLM-based picture description."""
enabled: bool = False
model: ModelConfig = Field(
default_factory=lambda: ModelConfig(
provider="ollama",
name="ministral-3",
)
)
timeout: int = 90
max_tokens: int = 200
class ConversionOptions(BaseModel):
"""Options for document conversion."""
@ -113,6 +127,11 @@ class ConversionOptions(BaseModel):
images_scale: float = 2.0
generate_picture_images: bool = False
# VLM picture description
picture_description: PictureDescriptionConfig = Field(
default_factory=PictureDescriptionConfig
)
class ProcessingConfig(BaseModel):
chunk_size: int = 256
@ -145,7 +164,6 @@ class OllamaConfig(BaseModel):
class DoclingServeConfig(BaseModel):
base_url: str = "http://localhost:5001"
api_key: str = ""
timeout: int = 300
class ProvidersConfig(BaseModel):
@ -166,6 +184,12 @@ class PromptsConfig(BaseModel):
domain_preamble: str = ""
qa: str | None = None
synthesis: str | None = None
picture_description: str = (
"Describe this image for a blind user. "
"State the image type (screenshot, chart, photo, etc.), "
"what it depicts, any visible text, and key visual details. "
"Be concise and accurate."
)
class AppConfig(BaseModel):

38
haiku_rag_slim/haiku/rag/converters/docling_local.py
View file

@ -11,6 +11,8 @@ from haiku.rag.converters.text_utils import TextFileHandler
if TYPE_CHECKING:
from docling_core.types.doc.document import DoclingDocument
from haiku.rag.config.models import ModelConfig
class DoclingLocalConverter(DocumentConverter):
"""Converter that uses local docling for document conversion.
@ -54,6 +56,21 @@ class DoclingLocalConverter(DocumentConverter):
"""Return list of file extensions supported by this converter."""
return self.docling_extensions + TextFileHandler.text_extensions
def _get_vlm_api_url(self, model: "ModelConfig") -> str:
"""Construct VLM API URL from model config."""
if model.base_url:
base = model.base_url.rstrip("/")
return f"{base}/v1/chat/completions"
if model.provider == "ollama":
base = self.config.providers.ollama.base_url.rstrip("/")
return f"{base}/v1/chat/completions"
if model.provider == "openai":
return "https://api.openai.com/v1/chat/completions"
raise ValueError(f"Unsupported VLM provider: {model.provider}")
def _sync_convert_docling_file(self, path: Path) -> "DoclingDocument":
"""Synchronous conversion of docling-supported files."""
from docling.backend.docling_parse_backend import DoclingParseDocumentBackend
@ -61,6 +78,7 @@ class DoclingLocalConverter(DocumentConverter):
from docling.datamodel.pipeline_options import (
OcrAutoOptions,
PdfPipelineOptions,
PictureDescriptionApiOptions,
TableFormerMode,
TableStructureOptions,
)
@ -73,13 +91,14 @@ class DoclingLocalConverter(DocumentConverter):
)
opts = self.config.processing.conversion_options
pic_desc = opts.picture_description
pipeline_options = PdfPipelineOptions(
do_ocr=opts.do_ocr,
do_table_structure=opts.do_table_structure,
images_scale=opts.images_scale,
generate_page_images=True,
generate_picture_images=opts.generate_picture_images,
generate_picture_images=opts.generate_picture_images or pic_desc.enabled,
table_structure_options=TableStructureOptions(
do_cell_matching=opts.table_cell_matching,
mode=(
@ -92,8 +111,25 @@ class DoclingLocalConverter(DocumentConverter):
force_full_page_ocr=opts.force_ocr,
lang=opts.ocr_lang if opts.ocr_lang else [],
),
do_picture_description=pic_desc.enabled,
)
if pic_desc.enabled:
from pydantic import AnyUrl
prompt = self.config.prompts.picture_description
pipeline_options.enable_remote_services = True
pipeline_options.picture_description_options = PictureDescriptionApiOptions(
url=AnyUrl(self._get_vlm_api_url(pic_desc.model)),
params=dict(
model=pic_desc.model.name,
max_completion_tokens=pic_desc.max_tokens,
),
prompt=prompt,
timeout=pic_desc.timeout,
)
format_options = cast(
dict[InputFormat, FormatOption],
{

145
haiku_rag_slim/haiku/rag/converters/docling_serve.py
View file

@ -1,18 +1,20 @@
"""docling-serve remote converter implementation."""
import asyncio
import json
from pathlib import Path
from typing import TYPE_CHECKING, ClassVar
import httpx
from haiku.rag.config import AppConfig
from haiku.rag.converters.base import DocumentConverter
from haiku.rag.converters.text_utils import TextFileHandler
from haiku.rag.providers.docling_serve import DoclingServeClient
if TYPE_CHECKING:
from docling_core.types.doc.document import DoclingDocument
from haiku.rag.config.models import ModelConfig
class DoclingServeConverter(DocumentConverter):
"""Converter that uses docling-serve for document conversion.
@ -53,17 +55,70 @@ class DoclingServeConverter(DocumentConverter):
config: Application configuration containing docling-serve settings.
"""
self.config = config
self.base_url = config.providers.docling_serve.base_url.rstrip("/")
self.api_key = config.providers.docling_serve.api_key
self.timeout = config.providers.docling_serve.timeout
self.client = DoclingServeClient(
base_url=config.providers.docling_serve.base_url,
api_key=config.providers.docling_serve.api_key,
)
@property
def supported_extensions(self) -> list[str]:
"""Return list of file extensions supported by this converter."""
return self.docling_serve_extensions + TextFileHandler.text_extensions
def _get_vlm_api_url(self, model: "ModelConfig") -> str:
"""Construct VLM API URL from model config."""
if model.base_url:
base = model.base_url.rstrip("/")
return f"{base}/v1/chat/completions"
if model.provider == "ollama":
base = self.config.providers.ollama.base_url.rstrip("/")
return f"{base}/v1/chat/completions"
if model.provider == "openai":
return "https://api.openai.com/v1/chat/completions"
raise ValueError(f"Unsupported VLM provider: {model.provider}")
def _build_conversion_data(self) -> dict[str, str | list[str]]:
"""Build form data for conversion request."""
opts = self.config.processing.conversion_options
pic_desc = opts.picture_description
data: dict[str, str | list[str]] = {
"to_formats": "json",
"do_ocr": str(opts.do_ocr).lower(),
"force_ocr": str(opts.force_ocr).lower(),
"do_table_structure": str(opts.do_table_structure).lower(),
"table_mode": opts.table_mode,
"table_cell_matching": str(opts.table_cell_matching).lower(),
"images_scale": str(opts.images_scale),
"generate_picture_images": str(
opts.generate_picture_images or pic_desc.enabled
).lower(),
"do_picture_description": str(pic_desc.enabled).lower(),
}
if opts.ocr_lang:
data["ocr_lang"] = opts.ocr_lang
if pic_desc.enabled:
prompt = self.config.prompts.picture_description
picture_description_api = {
"url": self._get_vlm_api_url(pic_desc.model),
"params": {
"model": pic_desc.model.name,
"max_completion_tokens": pic_desc.max_tokens,
},
"prompt": prompt,
"timeout": pic_desc.timeout,
}
data["picture_description_api"] = json.dumps(picture_description_api)
return data
async def _make_request(self, files: dict, name: str) -> "DoclingDocument":
"""Make a request to docling-serve and return the DoclingDocument.
"""Make an async request to docling-serve and poll for results.
Args:
files: Dictionary with files parameter for httpx
@ -77,73 +132,27 @@ class DoclingServeConverter(DocumentConverter):
"""
from docling_core.types.doc.document import DoclingDocument
try:
opts = self.config.processing.conversion_options
data = self._build_conversion_data()
result = await self.client.submit_and_poll(
endpoint="/v1/convert/file/async",
files=files,
data=data,
name=name,
)
data: dict[str, str | list[str]] = {
"to_formats": "json",
"do_ocr": str(opts.do_ocr).lower(),
"force_ocr": str(opts.force_ocr).lower(),
"do_table_structure": str(opts.do_table_structure).lower(),
"table_mode": opts.table_mode,
"table_cell_matching": str(opts.table_cell_matching).lower(),
"images_scale": str(opts.images_scale),
"generate_picture_images": str(opts.generate_picture_images).lower(),
}
if result.get("status") not in ("success", "partial_success", None):
errors = result.get("errors", [])
raise ValueError(f"Conversion failed: {errors}")
if opts.ocr_lang:
data["ocr_lang"] = opts.ocr_lang
json_content = result.get("document", {}).get("json_content")
headers = {}
if self.api_key:
headers["X-Api-Key"] = self.api_key
url = f"{self.base_url}/v1/convert/file"
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.post(
url,
files=files,
data=data,
headers=headers,
)
response.raise_for_status()
result = response.json()
if result["status"] not in ("success", "partial_success"):
errors = result.get("errors", [])
raise ValueError(f"Conversion failed: {errors}")
json_content = result["document"]["json_content"]
if json_content is None:
raise ValueError(
f"docling-serve did not return JSON content for {name}. "
"This may indicate an unsupported file format."
)
return DoclingDocument.model_validate(json_content)
except httpx.ConnectError as e:
if json_content is None:
raise ValueError(
f"Could not connect to docling-serve at {self.base_url}. "
f"Ensure the service is running and accessible. Error: {e}"
f"docling-serve did not return JSON content for {name}. "
"This may indicate an unsupported file format."
)
except httpx.TimeoutException as e:
raise ValueError(
f"Request to docling-serve timed out after {self.timeout}s. "
f"Consider increasing the timeout in configuration. Error: {e}"
)
except httpx.HTTPStatusError as e:
if e.response.status_code == 401:
raise ValueError(
"Authentication failed. Check your API key configuration."
)
raise ValueError(f"HTTP error from docling-serve: {e}")
except ValueError:
raise
except Exception as e:
raise ValueError(f"Failed to convert via docling-serve: {e}")
return DoclingDocument.model_validate(json_content)
async def convert_file(self, path: Path) -> "DoclingDocument":
"""Convert a file to DoclingDocument using docling-serve.

5
haiku_rag_slim/haiku/rag/providers/__init__.py Normal file
View file

@ -0,0 +1,5 @@
"""Provider clients for external services."""
from haiku.rag.providers.docling_serve import DoclingServeClient
__all__ = ["DoclingServeClient"]

108
haiku_rag_slim/haiku/rag/providers/docling_serve.py Normal file
View file

@ -0,0 +1,108 @@
"""Shared client for docling-serve async API."""
import asyncio
from typing import Any
import httpx
class DoclingServeClient:
"""Client for docling-serve async workflow.
Handles the submit → poll → fetch pattern used by both conversion and chunking.
"""
def __init__(self, base_url: str, api_key: str | None = None, timeout: float = 300):
self.base_url = base_url.rstrip("/")
self.api_key = api_key
self.timeout = timeout
def _get_headers(self) -> dict[str, str]:
"""Get headers for API requests."""
headers: dict[str, str] = {}
if self.api_key:
headers["X-Api-Key"] = self.api_key
return headers
async def submit_and_poll(
self,
endpoint: str,
files: dict[str, Any],
data: dict[str, Any],
name: str = "document",
) -> dict[str, Any]:
"""Submit a task and poll until completion.
Args:
endpoint: The async endpoint path (e.g., "/v1/convert/file/async")
files: Files to upload
data: Form data parameters
name: Name for error messages
Returns:
The result dictionary from the completed task
Raises:
ValueError: If the task fails or service is unavailable
"""
headers = self._get_headers()
try:
async with httpx.AsyncClient(timeout=self.timeout) as client:
# Submit async task
submit_url = f"{self.base_url}{endpoint}"
response = await client.post(
submit_url,
files=files,
data=data,
headers=headers,
)
response.raise_for_status()
submit_result = response.json()
task_id = submit_result.get("task_id")
if not task_id:
raise ValueError("docling-serve did not return a task_id")
# Poll for completion
poll_url = f"{self.base_url}/v1/status/poll/{task_id}"
while True:
poll_response = await client.get(poll_url, headers=headers)
poll_response.raise_for_status()
poll_result = poll_response.json()
status = poll_result.get("task_status")
if status == "success":
break
elif status in ("failure", "error"):
raise ValueError(
f"docling-serve task failed for {name}: {poll_result}"
)
await asyncio.sleep(1)
# Fetch result
result_url = f"{self.base_url}/v1/result/{task_id}"
result_response = await client.get(result_url, headers=headers)
result_response.raise_for_status()
return result_response.json()
except httpx.ConnectError as e:
raise ValueError(
f"Could not connect to docling-serve at {self.base_url}. "
f"Ensure the service is running and accessible. Error: {e}"
)
except httpx.TimeoutException as e:
raise ValueError(
f"Request to docling-serve timed out after {self.timeout}s. Error: {e}"
)
except httpx.HTTPStatusError as e:
if e.response.status_code == 401:
raise ValueError(
"Authentication failed. Check your API key configuration."
)
raise ValueError(f"HTTP error from docling-serve: {e}")
except ValueError:
raise
except Exception as e:
raise ValueError(f"Failed to process via docling-serve: {e}")

30060
tests/cassettes/test_chunker/test_local_and_serve_chunkers_produce_same_output.yaml
View file

File diff suppressed because one or more lines are too long

85
tests/cassettes/test_converters/TestDoclingLocalConverter.test_picture_description_end_to_end.yaml Normal file
View file

File diff suppressed because one or more lines are too long

117
tests/cassettes/test_converters/TestDoclingServeConverterIntegration.test_convert_code_file_real_service.yaml
View file

@ -1,13 +1,14 @@
interactions:
- request:
body: "--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
name=\"table_mode\"\r\n\r\naccurate\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition:
form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
name=\"generate_picture_images\"\r\n\r\nfalse\r\n--313fb28cff30a333a81f610f9c6bdaf3\r\nContent-Disposition: form-data;
name=\"files\"; filename=\"tmpgc9za8mg.md\"\r\nContent-Type: text/markdown\r\n\r\n```python\ndef test():\n return
42\n```\r\n--313fb28cff30a333a81f610f9c6bdaf3--\r\n"
body: "--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
name=\"table_mode\"\r\n\r\naccurate\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition:
form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
name=\"generate_picture_images\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
name=\"do_picture_description\"\r\n\r\nfalse\r\n--26e45846cd2f5c82db7b770428d74912\r\nContent-Disposition: form-data;
name=\"files\"; filename=\"tmp0aa4633w.md\"\r\nContent-Type: text/markdown\r\n\r\n```python\ndef test():\n return
42\n```\r\n--26e45846cd2f5c82db7b770428d74912--\r\n"
headers:
accept:
- '*/*'
@ -16,23 +17,107 @@ interactions:
connection:
- keep-alive
content-length:
- '1011'
- '1119'
content-type:
- multipart/form-data; boundary=313fb28cff30a333a81f610f9c6bdaf3
- multipart/form-data; boundary=26e45846cd2f5c82db7b770428d74912
host:
- localhost:5001
method: POST
uri: http://localhost:5001/v1/convert/file
uri: http://localhost:5001/v1/convert/file/async
response:
headers:
content-length:
- '1113'
- '131'
content-type:
- application/json
parsed_body:
task_id: 7436648c-490d-4938-b42b-9755823f8d78
task_meta: null
task_position: 1
task_status: pending
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/status/poll/7436648c-490d-4938-b42b-9755823f8d78
response:
headers:
content-length:
- '134'
content-type:
- application/json
parsed_body:
task_id: 7436648c-490d-4938-b42b-9755823f8d78
task_meta: null
task_position: null
task_status: started
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/status/poll/7436648c-490d-4938-b42b-9755823f8d78
response:
headers:
content-length:
- '134'
content-type:
- application/json
parsed_body:
task_id: 7436648c-490d-4938-b42b-9755823f8d78
task_meta: null
task_position: null
task_status: success
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/result/7436648c-490d-4938-b42b-9755823f8d78
response:
headers:
content-length:
- '1114'
content-type:
- application/json
parsed_body:
document:
doctags_content: null
filename: tmpgc9za8mg.md
filename: tmp0aa4633w.md
html_content: null
json_content:
body:
@ -55,10 +140,10 @@ interactions:
self_ref: '#/furniture'
groups: []
key_value_items: []
name: tmpgc9za8mg
name: tmp0aa4633w
origin:
binary_hash: 9008975733065065710
filename: tmpgc9za8mg.md
filename: tmp0aa4633w.md
mimetype: text/markdown
uri: null
pages: {}
@ -91,7 +176,7 @@ interactions:
md_content: null
text_content: null
errors: []
processing_time: 0.002258999999980915
processing_time: 0.0027105000044684857
status: success
timings: {}
status:

107
tests/cassettes/test_converters/TestDoclingServeConverterIntegration.test_convert_text_real_service.yaml
View file

@ -1,12 +1,13 @@
interactions:
- request:
body: "--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
name=\"table_mode\"\r\n\r\naccurate\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition:
form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
name=\"generate_picture_images\"\r\n\r\nfalse\r\n--a906caa5bbe5d30e17b44fb63c456240\r\nContent-Disposition: form-data;
name=\"files\"; filename=\"content.md\"\r\nContent-Type: text/markdown\r\n\r\n# Test Document\n\nThis is a test.\r\n--a906caa5bbe5d30e17b44fb63c456240--\r\n"
body: "--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"to_formats\"\r\n\r\njson\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
form-data; name=\"do_ocr\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"force_ocr\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
form-data; name=\"do_table_structure\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
name=\"table_mode\"\r\n\r\naccurate\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data; name=\"table_cell_matching\"\r\n\r\ntrue\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition:
form-data; name=\"images_scale\"\r\n\r\n2.0\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
name=\"generate_picture_images\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
name=\"do_picture_description\"\r\n\r\nfalse\r\n--9df242c136eddc0594e7eb126b1f39cd\r\nContent-Disposition: form-data;
name=\"files\"; filename=\"content.md\"\r\nContent-Type: text/markdown\r\n\r\n# Test Document\n\nThis is a test.\r\n--9df242c136eddc0594e7eb126b1f39cd--\r\n"
headers:
accept:
- '*/*'
@ -15,13 +16,97 @@ interactions:
connection:
- keep-alive
content-length:
- '1000'
- '1108'
content-type:
- multipart/form-data; boundary=a906caa5bbe5d30e17b44fb63c456240
- multipart/form-data; boundary=9df242c136eddc0594e7eb126b1f39cd
host:
- localhost:5001
method: POST
uri: http://localhost:5001/v1/convert/file
uri: http://localhost:5001/v1/convert/file/async
response:
headers:
content-length:
- '131'
content-type:
- application/json
parsed_body:
task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
task_meta: null
task_position: 1
task_status: pending
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/status/poll/9f7f1455-1ab6-4830-b5a5-3e63419379ae
response:
headers:
content-length:
- '134'
content-type:
- application/json
parsed_body:
task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
task_meta: null
task_position: null
task_status: started
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/status/poll/9f7f1455-1ab6-4830-b5a5-3e63419379ae
response:
headers:
content-length:
- '134'
content-type:
- application/json
parsed_body:
task_id: 9f7f1455-1ab6-4830-b5a5-3e63419379ae
task_meta: null
task_position: null
task_status: success
task_type: convert
status:
code: 200
message: OK
- request:
body: ''
headers:
accept:
- '*/*'
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
host:
- localhost:5001
method: GET
uri: http://localhost:5001/v1/result/9f7f1455-1ab6-4830-b5a5-3e63419379ae
response:
headers:
content-length:
@ -94,7 +179,7 @@ interactions:
md_content: null
text_content: null
errors: []
processing_time: 0.004331874999934371
processing_time: 0.004173958994215354
status: success
timings: {}
status:

14958
tests/cassettes/test_converters/TestDoclingServeConverterIntegration.test_picture_description_end_to_end.yaml Normal file
View file

File diff suppressed because it is too large Load diff

96
tests/test_chunker.py
View file

@ -232,6 +232,28 @@ def test_get_chunker_docling_serve():
assert isinstance(chunker, DoclingServeChunker)
def create_async_workflow_mocks(
result_data: dict, task_id: str = "test-task-123"
) -> tuple[Mock, Mock, Mock]:
"""Create mock responses for docling-serve async workflow."""
submit_response = Mock()
submit_response.status_code = 200
submit_response.json.return_value = {"task_id": task_id, "task_status": "pending"}
submit_response.raise_for_status = Mock()
poll_response = Mock()
poll_response.status_code = 200
poll_response.json.return_value = {"task_id": task_id, "task_status": "success"}
poll_response.raise_for_status = Mock()
result_response = Mock()
result_response.status_code = 200
result_response.json.return_value = result_data
result_response.raise_for_status = Mock()
return submit_response, poll_response, result_response
class TestDoclingServeChunker:
"""Tests for DoclingServeChunker (mocked)."""
@ -241,7 +263,6 @@ class TestDoclingServeChunker:
config = AppConfig()
config.providers.docling_serve.base_url = "http://localhost:5001"
config.providers.docling_serve.api_key = ""
config.providers.docling_serve.timeout = 300
config.processing.chunk_size = 256
config.processing.chunking_tokenizer = "Qwen/Qwen3-Embedding-0.6B"
return config
@ -252,21 +273,20 @@ class TestDoclingServeChunker:
return DoclingServeChunker(config)
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_success(self, mock_client_class, chunker):
"""Test successful chunking via docling-serve."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"""Test successful chunking via docling-serve async workflow."""
result_data = {
"chunks": [
{"text": "Chunk 1", "chunk_index": 0},
{"text": "Chunk 2", "chunk_index": 1},
]
}
mock_response.raise_for_status = Mock()
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client_class.return_value.__aenter__.return_value = mock_client
# Create a simple document
@ -280,21 +300,18 @@ class TestDoclingServeChunker:
mock_client.post.assert_called_once()
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_with_api_key(self, mock_client_class, config):
"""Test that API key is included in request headers."""
config.providers.docling_serve.api_key = "test-key"
chunker = DoclingServeChunker(config)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"chunks": [{"text": "Chunk 1", "chunk_index": 0}]
}
mock_response.raise_for_status = Mock()
result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client_class.return_value.__aenter__.return_value = mock_client
converter = get_converter(Config)
@ -306,21 +323,18 @@ class TestDoclingServeChunker:
assert call_kwargs["headers"]["X-Api-Key"] == "test-key"
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_hierarchical_endpoint(self, mock_client_class, config):
"""Test that hierarchical chunker uses correct endpoint."""
config.processing.chunker_type = "hierarchical"
chunker = DoclingServeChunker(config)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"chunks": [{"text": "Chunk 1", "chunk_index": 0}]
}
mock_response.raise_for_status = Mock()
result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client_class.return_value.__aenter__.return_value = mock_client
converter = get_converter(Config)
@ -328,10 +342,10 @@ class TestDoclingServeChunker:
await chunker.chunk(doc)
call_args = mock_client.post.call_args
assert "/v1/chunk/hierarchical/file" in call_args[0][0]
assert "/v1/chunk/hierarchical/file/async" in call_args[0][0]
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_passes_config_parameters(self, mock_client_class, config):
"""Test that all config parameters are passed to API."""
config.processing.chunk_size = 512
@ -339,15 +353,12 @@ class TestDoclingServeChunker:
config.processing.chunking_use_markdown_tables = True
chunker = DoclingServeChunker(config)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"chunks": [{"text": "Chunk 1", "chunk_index": 0}]
}
mock_response.raise_for_status = Mock()
result_data = {"chunks": [{"text": "Chunk 1", "chunk_index": 0}]}
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client_class.return_value.__aenter__.return_value = mock_client
converter = get_converter(Config)
@ -361,7 +372,7 @@ class TestDoclingServeChunker:
assert data["chunking_use_markdown_tables"] == "true"
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_connection_error(self, mock_client_class, chunker):
"""Test handling of connection errors."""
import httpx
@ -377,7 +388,7 @@ class TestDoclingServeChunker:
await chunker.chunk(doc)
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_timeout_error(self, mock_client_class, chunker):
"""Test handling of timeout errors."""
import httpx
@ -393,7 +404,7 @@ class TestDoclingServeChunker:
await chunker.chunk(doc)
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_auth_error(self, mock_client_class, chunker):
"""Test handling of authentication errors."""
import httpx
@ -416,18 +427,14 @@ class TestDoclingServeChunker:
await chunker.chunk(doc)
@pytest.mark.asyncio
@patch("haiku.rag.chunkers.docling_serve.httpx.AsyncClient")
@patch("haiku.rag.providers.docling_serve.httpx.AsyncClient")
async def test_chunk_metadata_extraction(self, mock_client_class, chunker):
"""Test that metadata is correctly extracted from API response.
Labels are resolved from the DoclingDocument using the refs, so we need
to create a document with matching structure for the mocked API response.
"""
mock_response = Mock()
mock_response.status_code = 200
# docling-serve returns doc_items as list of ref strings
# We'll reference texts[0], texts[1], and tables[0]
mock_response.json.return_value = {
result_data = {
"chunks": [
{
"text": "Chapter 1\nThis is content.",
@ -443,10 +450,11 @@ class TestDoclingServeChunker:
},
]
}
mock_response.raise_for_status = Mock()
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(result_data)
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client_class.return_value.__aenter__.return_value = mock_client
# Create a document with texts and tables that match the mocked refs

382
tests/test_converters.py
View file

@ -54,6 +54,31 @@ def create_mock_docling_document_json(name: str = "test") -> dict:
}
def create_async_workflow_mocks(
doc_json: dict, task_id: str = "test-task-123"
) -> tuple[Mock, Mock, Mock]:
"""Create mock responses for docling-serve async workflow.
Returns tuple of (submit_response, poll_response, result_response).
"""
submit_response = Mock()
submit_response.status_code = 200
submit_response.json.return_value = {"task_id": task_id, "task_status": "pending"}
submit_response.raise_for_status = Mock()
poll_response = Mock()
poll_response.status_code = 200
poll_response.json.return_value = {"task_id": task_id, "task_status": "success"}
poll_response.raise_for_status = Mock()
result_response = Mock()
result_response.status_code = 200
result_response.json.return_value = {"document": {"json_content": doc_json}}
result_response.raise_for_status = Mock()
return submit_response, poll_response, result_response
class TestTextFileHandler:
"""Tests for TextFileHandler utility class."""
@ -308,6 +333,117 @@ class TestDoclingLocalConverter:
"Pictures should have image data when generate_picture_images=True"
)
def test_get_vlm_api_url_with_ollama(self, config):
"""Test VLM API URL construction for Ollama provider."""
converter = DoclingLocalConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="ollama", name="ministral-3")
url = converter._get_vlm_api_url(model)
assert url == "http://localhost:11434/v1/chat/completions"
def test_get_vlm_api_url_with_custom_base_url(self, config):
"""Test VLM API URL construction with custom base_url."""
converter = DoclingLocalConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(
provider="openai", name="gpt-4-vision", base_url="http://my-vllm:8000"
)
url = converter._get_vlm_api_url(model)
assert url == "http://my-vllm:8000/v1/chat/completions"
def test_get_vlm_api_url_with_openai(self, config):
"""Test VLM API URL construction for OpenAI provider."""
converter = DoclingLocalConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="openai", name="gpt-4-vision")
url = converter._get_vlm_api_url(model)
assert url == "https://api.openai.com/v1/chat/completions"
def test_get_vlm_api_url_unsupported_provider(self, config):
"""Test VLM API URL construction raises error for unsupported provider."""
converter = DoclingLocalConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="unsupported", name="test")
with pytest.raises(ValueError, match="Unsupported VLM provider"):
converter._get_vlm_api_url(model)
def test_picture_description_config_defaults(self, config):
"""Test that picture description config has correct defaults."""
assert config.processing.conversion_options.picture_description.enabled is False
assert (
config.processing.conversion_options.picture_description.model.provider
== "ollama"
)
assert (
config.processing.conversion_options.picture_description.model.name
== "ministral-3"
)
assert config.processing.conversion_options.picture_description.timeout == 90
assert (
config.processing.conversion_options.picture_description.max_tokens == 200
)
# Default prompt is in PromptsConfig
assert "blind user" in config.prompts.picture_description
def test_picture_description_config_applied(self, config):
"""Test that picture description config is applied to converter."""
config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.timeout = 120
converter = DoclingLocalConverter(config)
pic_desc = converter.config.processing.conversion_options.picture_description
assert pic_desc.enabled is True
assert pic_desc.timeout == 120
@pytest.mark.asyncio
@pytest.mark.integration
@pytest.mark.vcr()
async def test_picture_description_end_to_end(self, config):
"""End-to-end test: convert PDF with VLM picture descriptions."""
pdf_path = Path("tests/data/doclaynet.pdf")
if not pdf_path.exists():
pytest.skip("doclaynet.pdf not found")
# Enable picture description with Ollama
config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = (
"ollama"
)
config.processing.conversion_options.picture_description.model.name = (
"ministral-3"
)
converter = DoclingLocalConverter(config)
doc = await converter.convert_file(pdf_path)
# Export to markdown and check for picture descriptions
markdown = doc.export_to_markdown()
# The document should have pictures with descriptions
assert doc.pictures, "Document should have pictures"
# Check that at least one picture has a description annotation
from docling_core.types.doc.document import PictureDescriptionData
pictures_with_descriptions = []
for pic in doc.pictures:
for ann in pic.annotations:
if isinstance(ann, PictureDescriptionData):
pictures_with_descriptions.append(pic)
# Description should appear in markdown output
assert ann.text in markdown, (
f"Picture description '{ann.text[:50]}...' should be in markdown"
)
break
assert pictures_with_descriptions, (
"At least one picture should have a VLM description"
)
class TestDoclingServeConverter:
"""Tests for DoclingServeConverter (mocked)."""
@ -318,7 +454,6 @@ class TestDoclingServeConverter:
config = AppConfig()
config.providers.docling_serve.base_url = "http://localhost:5001"
config.providers.docling_serve.api_key = ""
config.providers.docling_serve.timeout = 300
return config
@pytest.fixture
@ -328,8 +463,7 @@ class TestDoclingServeConverter:
def test_initialization(self, converter):
"""Test converter initialization."""
assert converter.base_url == "http://localhost:5001"
assert converter.timeout == 300
assert converter.client.base_url == "http://localhost:5001"
def test_supported_extensions(self, converter):
"""Test that converter reports correct supported extensions."""
@ -341,18 +475,14 @@ class TestDoclingServeConverter:
@pytest.mark.asyncio
async def test_convert_text_success(self, converter):
"""Test successful text conversion via docling-serve."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": create_mock_docling_document_json("test")},
}
mock_response.raise_for_status = Mock()
"""Test successful text conversion via docling-serve async workflow."""
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -368,17 +498,13 @@ class TestDoclingServeConverter:
config.providers.docling_serve.api_key = "test-key"
converter = DoclingServeConverter(config)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": create_mock_docling_document_json("test")},
}
mock_response.raise_for_status = Mock()
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -401,17 +527,13 @@ class TestDoclingServeConverter:
config.processing.conversion_options.images_scale = 3.0
converter = DoclingServeConverter(config)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": create_mock_docling_document_json("test")},
}
mock_response.raise_for_status = Mock()
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -480,17 +602,16 @@ class TestDoclingServeConverter:
@pytest.mark.asyncio
async def test_convert_text_no_json_content(self, converter):
"""Test handling when docling-serve returns no JSON content."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": None},
}
mock_response.raise_for_status = Mock()
submit_resp, poll_resp, _ = create_async_workflow_mocks({})
result_resp = Mock()
result_resp.status_code = 200
result_resp.json.return_value = {"document": {"json_content": None}}
result_resp.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -500,18 +621,14 @@ class TestDoclingServeConverter:
@pytest.mark.asyncio
async def test_convert_file_pdf(self, converter):
"""Test converting PDF file via docling-serve."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": create_mock_docling_document_json("test")},
}
mock_response.raise_for_status = Mock()
"""Test converting PDF file via docling-serve async workflow."""
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -528,17 +645,13 @@ class TestDoclingServeConverter:
@pytest.mark.asyncio
async def test_convert_file_text(self, converter):
"""Test converting text file (reads locally, sends to docling-serve)."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"document": {"json_content": create_mock_docling_document_json("test")},
}
mock_response.raise_for_status = Mock()
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
@ -555,6 +668,124 @@ class TestDoclingServeConverter:
assert "files" in call_kwargs
class TestDoclingServeConverterPictureDescription:
"""Tests for DoclingServeConverter picture description support."""
@pytest.fixture
def config(self):
"""Create test configuration."""
config = AppConfig()
config.providers.docling_serve.base_url = "http://localhost:5001"
config.providers.docling_serve.api_key = ""
return config
def test_get_vlm_api_url_with_ollama(self, config):
"""Test VLM API URL construction for Ollama provider."""
converter = DoclingServeConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="ollama", name="ministral-3")
url = converter._get_vlm_api_url(model)
assert url == "http://localhost:11434/v1/chat/completions"
def test_get_vlm_api_url_with_custom_base_url(self, config):
"""Test VLM API URL construction with custom base_url."""
converter = DoclingServeConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(
provider="openai", name="gpt-4-vision", base_url="http://my-vllm:8000"
)
url = converter._get_vlm_api_url(model)
assert url == "http://my-vllm:8000/v1/chat/completions"
def test_get_vlm_api_url_with_openai(self, config):
"""Test VLM API URL construction for OpenAI provider."""
converter = DoclingServeConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="openai", name="gpt-4-vision")
url = converter._get_vlm_api_url(model)
assert url == "https://api.openai.com/v1/chat/completions"
def test_get_vlm_api_url_unsupported_provider(self, config):
"""Test VLM API URL construction raises error for unsupported provider."""
converter = DoclingServeConverter(config)
from haiku.rag.config.models import ModelConfig
model = ModelConfig(provider="unsupported", name="test")
with pytest.raises(ValueError, match="Unsupported VLM provider"):
converter._get_vlm_api_url(model)
@pytest.mark.asyncio
async def test_picture_description_options_passed_to_api(self, config):
"""Test that picture description options are passed to docling-serve API."""
import json
config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = (
"ollama"
)
config.processing.conversion_options.picture_description.model.name = (
"ministral-3"
)
config.processing.conversion_options.picture_description.timeout = 120
config.processing.conversion_options.picture_description.max_tokens = 300
config.prompts.picture_description = "Test prompt for picture description"
converter = DoclingServeConverter(config)
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await converter.convert_text("# Test")
call_kwargs = mock_client.post.call_args.kwargs
assert "data" in call_kwargs
data = call_kwargs["data"]
assert data["do_picture_description"] == "true"
assert data["generate_picture_images"] == "true"
assert "picture_description_api" in data
api_config = json.loads(data["picture_description_api"])
assert api_config["url"] == "http://localhost:11434/v1/chat/completions"
assert api_config["params"]["model"] == "ministral-3"
assert api_config["params"]["max_completion_tokens"] == 300
assert api_config["prompt"] == "Test prompt for picture description"
assert api_config["timeout"] == 120
@pytest.mark.asyncio
async def test_picture_description_disabled_by_default(self, config):
"""Test that picture description is disabled by default."""
converter = DoclingServeConverter(config)
doc_json = create_mock_docling_document_json("test")
submit_resp, poll_resp, result_resp = create_async_workflow_mocks(doc_json)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=submit_resp)
mock_client.get = AsyncMock(side_effect=[poll_resp, result_resp])
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await converter.convert_text("# Test")
call_kwargs = mock_client.post.call_args.kwargs
data = call_kwargs["data"]
assert data["do_picture_description"] == "false"
assert "picture_description_api" not in data
class TestDoclingServeConverterIntegration:
"""Integration tests with real docling-serve recorded via VCR."""
@ -591,3 +822,48 @@ class TestDoclingServeConverterIntegration:
assert isinstance(doc, DoclingDocument)
result = doc.export_to_markdown()
assert "def test():" in result
@pytest.mark.asyncio
@pytest.mark.integration
async def test_picture_description_end_to_end(self, config):
"""End-to-end test: convert PDF with VLM picture descriptions via docling-serve.
Note: Not using VCR because this test involves polling with changing task IDs.
"""
pdf_path = Path("tests/data/doclaynet.pdf")
if not pdf_path.exists():
pytest.skip("doclaynet.pdf not found")
config.processing.conversion_options.picture_description.enabled = True
config.processing.conversion_options.picture_description.model.provider = (
"ollama"
)
config.processing.conversion_options.picture_description.model.name = (
"ministral-3"
)
# Use host.docker.internal so docling-serve in Docker can reach host's Ollama
config.processing.conversion_options.picture_description.model.base_url = (
"http://host.docker.internal:11434"
)
converter = DoclingServeConverter(config)
doc = await converter.convert_file(pdf_path)
assert doc.pictures, "Document should have pictures"
from docling_core.types.doc.document import PictureDescriptionData
pictures_with_descriptions = []
markdown = doc.export_to_markdown()
for pic in doc.pictures:
for ann in pic.annotations:
if isinstance(ann, PictureDescriptionData):
pictures_with_descriptions.append(pic)
assert ann.text in markdown, (
f"Picture description '{ann.text[:50]}...' should be in markdown"
)
break
assert pictures_with_descriptions, (
"At least one picture should have a VLM description"
)