Clarify docs about using a VLM with docling-serve

This commit is contained in:
Yiorgis Gozadinos 2026-01-08 10:57:52 +02:00
parent 92f0fd956f
commit 4cae2cf26a
No known key found for this signature in database
2 changed files with 93 additions and 1 deletions

View file

@ -174,6 +174,19 @@ When using `converter: docling-serve`, the VLM calls are made by the docling-ser
1. Set `DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` when running docling-serve
2. Ensure the VLM endpoint is accessible from where docling-serve is running
**Docker networking:** If docling-serve runs in Docker and your VLM runs on the host, use `host.docker.internal` instead of `localhost`:
```yaml
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
base_url: http://host.docker.internal:11434 # NOT localhost!
```
See [VLM Picture Description with docling-serve](../remote-processing.md#vlm-picture-description-with-docling-serve) for a complete example.
### Local vs Remote Processing
**Local processing** (default):

View file

@ -39,7 +39,13 @@ The easiest way to use haiku.rag with docling-serve is using the slim Docker ima
See the [official docling-serve repository](https://github.com/docling-project/docling-serve) for installation options. The quickest way is using Docker:
```bash
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=1 quay.io/docling-project/docling-serve
docker run -p 5001:5001 quay.io/docling-project/docling-serve
```
To enable the web UI for debugging:
```bash
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=true quay.io/docling-project/docling-serve
```
### Configuration
@ -128,6 +134,79 @@ processing:
- `false` (default): Tables as narrative text
- `true`: Tables as markdown format
## VLM Picture Description with docling-serve
When using VLM picture description with docling-serve, the VLM API calls are made by the docling-serve container, not by haiku.rag. This requires additional configuration.
### Enable Remote Services
docling-serve blocks external API calls by default. To enable VLM picture description, start docling-serve with:
```bash
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve
```
### Docker Networking
When docling-serve runs in Docker and your VLM (e.g., Ollama) runs on the host, `localhost` inside the container refers to the container itself, not your host machine.
Use `host.docker.internal` to reach host services from within Docker:
```yaml
# haiku.rag.yaml
processing:
converter: docling-serve
chunker: docling-serve
conversion_options:
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
base_url: http://host.docker.internal:11434 # NOT localhost!
```
### Complete Example
1. Start Ollama with a vision model on your host:
```bash
ollama pull ministral-3
ollama serve
```
2. Start docling-serve with remote services enabled:
```bash
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve
```
3. Configure haiku.rag:
```yaml
# haiku.rag.yaml
processing:
converter: docling-serve
chunker: docling-serve
conversion_options:
picture_description:
enabled: true
model:
provider: ollama
name: ministral-3
base_url: http://host.docker.internal:11434
providers:
docling_serve:
base_url: http://localhost:5001
```
4. Add a document:
```bash
haiku-rag add-src document.pdf
```
## Resources
- [docling-serve GitHub](https://github.com/docling-project/docling-serve)