Replace picture_description.enabled with processing.pictures enum
This commit is contained in:
parent
bdab201c42
commit
a98ddc14b8
17 changed files with 118 additions and 136 deletions
|
|
@ -15,6 +15,7 @@
|
||||||
|
|
||||||
- **Chat TUI streams markdown incrementally.** Assistant messages now use Textual's `MarkdownStream` (`Markdown.get_stream`) and write per-token deltas instead of re-parsing the entire accumulated message on every token. Removes the O(n²) re-parse that visibly stuttered long responses. Bumps `textual` floor to `>=8.2.4` so `Markdown.get_stream` is reachable via the public API.
|
- **Chat TUI streams markdown incrementally.** Assistant messages now use Textual's `MarkdownStream` (`Markdown.get_stream`) and write per-token deltas instead of re-parsing the entire accumulated message on every token. Removes the O(n²) re-parse that visibly stuttered long responses. Bumps `textual` floor to `>=8.2.4` so `Markdown.get_stream` is reachable via the public API.
|
||||||
- **Embedding compatibility check only raises on `vector_dim` mismatch.** `provider` and `name` drift (legitimate when the same model is served by a different stack, e.g. Ollama → vLLM-via-openai) now logs a one-time warning and updates the stored settings to match the current config. Subsequent opens are silent. Run `rebuild --embed-only` if you also want to re-embed under the new stack.
|
- **Embedding compatibility check only raises on `vector_dim` mismatch.** `provider` and `name` drift (legitimate when the same model is served by a different stack, e.g. Ollama → vLLM-via-openai) now logs a one-time warning and updates the stored settings to match the current config. Subsequent opens are silent. Run `rebuild --embed-only` if you also want to re-embed under the new stack.
|
||||||
|
- **`processing.pictures` enum replaces `picture_description.enabled`.** Three modes: `none` (skip picture generation entirely — lower RAM, smaller DBs), `description` (generate images, run VLM, store bytes), `image` (default — generate images, store bytes, no VLM). Closes #366. Breaking change: rename `picture_description.enabled: true` → `pictures: description`, `picture_description.enabled: false` → `pictures: image`. The pre-April-30 `generate_picture_images` flag is also gone; use `pictures: none` for that opt-out.
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -497,7 +497,7 @@ haiku-rag rebuild --descriptions
|
||||||
| Title only | `--title-only` | Generate titles for documents without one |
|
| Title only | `--title-only` | Generate titles for documents without one |
|
||||||
| Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database |
|
| Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database |
|
||||||
|
|
||||||
**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `picture_description.enabled: true` in the config. Idempotent — pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely; only the VLM time is paid.
|
**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent — pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely; only the VLM time is paid.
|
||||||
|
|
||||||
### Download Models
|
### Download Models
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,12 +45,12 @@ processing:
|
||||||
images_scale: 2.0 # Image scale factor
|
images_scale: 2.0 # Image scale factor
|
||||||
generate_page_images: true # Include rendered page images (for visualize_chunk)
|
generate_page_images: true # Include rendered page images (for visualize_chunk)
|
||||||
|
|
||||||
# VLM picture description (off by default; see "Picture Handling" below)
|
# VLM settings used when processing.pictures == "description" (see "Picture Handling" below)
|
||||||
picture_description:
|
picture_description:
|
||||||
enabled: false
|
|
||||||
model:
|
model:
|
||||||
provider: ollama
|
provider: ollama
|
||||||
name: ministral-3
|
name: ministral-3
|
||||||
|
pictures: image # none | description | image
|
||||||
```
|
```
|
||||||
|
|
||||||
### Conversion Options
|
### Conversion Options
|
||||||
|
|
@ -121,7 +121,7 @@ Per-image failures (404, timeout, oversized, unreadable) leave that picture as a
|
||||||
|
|
||||||
**Scope of conversion options across formats:**
|
**Scope of conversion options across formats:**
|
||||||
|
|
||||||
| Input | OCR / table options | `images_scale` / `generate_page_images` | `picture_description` | `fetch_remote_images` |
|
| Input | OCR / table options | `images_scale` / `generate_page_images` | `pictures` | `fetch_remote_images` |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| `.pdf` | ✅ | ✅ | ✅ | n/a |
|
| `.pdf` | ✅ | ✅ | ✅ | n/a |
|
||||||
| `.png` / `.jpg` / `.jpeg` / `.bmp` / `.tiff` / `.webp` | ✅ | ✅ | ✅ | n/a |
|
| `.png` / `.jpg` / `.jpeg` / `.bmp` / `.tiff` / `.webp` | ✅ | ✅ | ✅ | n/a |
|
||||||
|
|
@ -132,26 +132,36 @@ Per-image failures (404, timeout, oversized, unreadable) leave that picture as a
|
||||||
|
|
||||||
#### Picture Handling
|
#### Picture Handling
|
||||||
|
|
||||||
Picture bytes (figures, diagrams) are always extracted and stored in `document_items.picture_data` for every ingested document. The single configurable knob is whether a Vision Language Model (VLM) runs at ingest to generate textual descriptions:
|
`processing.pictures` picks one of three modes:
|
||||||
|
|
||||||
|
| Mode | Picture-image generation in docling | Bytes stored in `document_items.picture_data` | VLM runs at ingest |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `none` | off | no | no |
|
||||||
|
| `description` | on | yes | yes |
|
||||||
|
| `image` (default) | on | yes | no |
|
||||||
|
|
||||||
|
Use `none` when you don't need picture content (e.g. very large reference manuals where RAM is tight); use `description` to weave VLM-generated text into chunk content and keep bytes for later; use `image` (default) to keep bytes without paying the VLM cost. The prompt is configurable under `prompts.picture_description` — see [Prompts](prompts.md).
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
processing:
|
processing:
|
||||||
|
pictures: description # none | description | image
|
||||||
conversion_options:
|
conversion_options:
|
||||||
picture_description:
|
picture_description: # only consulted when pictures == "description"
|
||||||
enabled: true # default false
|
|
||||||
model:
|
model:
|
||||||
provider: ollama # any OpenAI-compatible /v1/chat/completions provider
|
provider: ollama # any OpenAI-compatible /v1/chat/completions provider
|
||||||
name: ministral-3
|
name: ministral-3
|
||||||
timeout: 90
|
timeout: 90
|
||||||
max_tokens: 200
|
max_tokens: 200
|
||||||
```
|
```
|
||||||
|
|
||||||
When `enabled: true`, each picture's description is woven into the chunk text and is searchable via FTS. The prompt is configurable under `prompts.picture_description` — see [Prompts](prompts.md).
|
!!! warning "Breaking change"
|
||||||
|
`processing.conversion_options.picture_description.enabled` is replaced by `processing.pictures`. Map `enabled: true` → `pictures: description`, `enabled: false` → `pictures: image`. The pre-April-30 `generate_picture_images` flag also no longer exists; use `pictures: none` for the old opt-out.
|
||||||
|
|
||||||
**Switching the VLM on or off on an existing database** doesn't require reingesting (the bytes are already there):
|
**Switching modes on an existing database** doesn't require reingesting when the bytes are already stored:
|
||||||
|
|
||||||
- Off → on: `haiku-rag rebuild --descriptions` runs the VLM over stored bytes and re-chunks. Skips the docling parse entirely.
|
- `image` → `description`: `haiku-rag rebuild --descriptions` runs the VLM over stored bytes and re-chunks. Skips the docling parse entirely.
|
||||||
- On → off: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions.
|
- `description` → `image`: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions.
|
||||||
|
- Switching to/from `none`: a full reingest is needed since the bytes either weren't stored or need to be discarded.
|
||||||
|
|
||||||
When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag — see [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve).
|
When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag — see [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve).
|
||||||
|
|
||||||
|
|
@ -161,37 +171,37 @@ Three independent settings drive ingest, retrieval, and QA:
|
||||||
|
|
||||||
| Setting | Question it answers | Values |
|
| Setting | Question it answers | Values |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `picture_description.enabled` | Should a VLM weave descriptions into chunk text at ingest? | `false` (default) / `true` |
|
| `processing.pictures` | Generate and/or describe pictures at ingest? | `none` / `description` / `image` (default) |
|
||||||
| `embeddings.model.provider` | Can the embedder index image content? | text-only (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
|
| `embeddings.model.provider` | Can the embedder index image content? | text-only (`ollama`, `openai`, `cohere`, `sentence-transformers`) vs `vllm` (multimodal) |
|
||||||
| `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` |
|
| `qa.model.vision` | Can the QA model interpret images? | `false` (default) / `true` |
|
||||||
|
|
||||||
Picture bytes are always stored, regardless of these settings.
|
**What gets stored** by `pictures` × embedder:
|
||||||
|
|
||||||
**What gets stored** by `enabled` × embedder:
|
| `pictures` | Embedder | Text chunks | Synthetic picture chunks |
|
||||||
|
|
||||||
| `enabled` | Embedder | Text chunks | Synthetic picture chunks |
|
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `false` | text-only | text only (caption/surrounding) | none |
|
| `none` | any | text only (caption/surrounding) | none |
|
||||||
| `false` | multimodal | text only | one per picture, vector = image embedding |
|
| `image` | text-only | text only (caption/surrounding) | none |
|
||||||
| `true` | text-only | text + descriptions | none |
|
| `image` | multimodal | text only | one per picture, vector = image embedding |
|
||||||
| `true` | multimodal | text + descriptions | one per picture, vector = image embedding |
|
| `description` | text-only | text + descriptions | none |
|
||||||
|
| `description` | multimodal | text + descriptions | one per picture, vector = image embedding |
|
||||||
|
|
||||||
**What QA receives** at search time:
|
**What QA receives** at search time:
|
||||||
|
|
||||||
- `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose).
|
- `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose).
|
||||||
- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`; the model reads figures directly.
|
- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`; the model reads figures directly. Requires `pictures != none` so the bytes exist.
|
||||||
|
|
||||||
`qa.model.vision` is independent of ingestion — flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI; default `false` is the safe choice.
|
`qa.model.vision` is independent of ingestion — flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI; default `false` is the safe choice.
|
||||||
|
|
||||||
**Recommended combinations:**
|
**Recommended combinations:**
|
||||||
|
|
||||||
| Use case | `picture_description.enabled` | Embedder | `qa.model.vision` |
|
| Use case | `processing.pictures` | Embedder | `qa.model.vision` |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Pure text RAG, no figures | `false` | text-only | `false` |
|
| Pure text RAG, no figures, lowest RAM | `none` | text-only | `false` |
|
||||||
| Text RAG, figures answered through descriptions | `true` | text-only | `false` |
|
| Text RAG, store figure bytes for later | `image` | text-only | `false` |
|
||||||
| Vision QA on figure-rich docs (no cross-modal search) | `true` or `false` | text-only | `true` |
|
| Text RAG, figures answered through descriptions | `description` | text-only | `false` |
|
||||||
| Cross-modal search + vision QA | `true` or `false` | multimodal | `true` |
|
| Vision QA on figure-rich docs (no cross-modal search) | `image` or `description` | text-only | `true` |
|
||||||
| Cross-modal search, text QA only | `true` | multimodal | `false` |
|
| Cross-modal search + vision QA | `image` or `description` | multimodal | `true` |
|
||||||
|
| Cross-modal search, text QA only | `description` | multimodal | `false` |
|
||||||
|
|
||||||
### Automatic Title Generation
|
### Automatic Title Generation
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -119,7 +119,7 @@ prompts:
|
||||||
Be concise and factual.
|
Be concise and factual.
|
||||||
```
|
```
|
||||||
|
|
||||||
The prompt is used when `processing.conversion_options.picture_description.enabled` is `true`. See [Picture Handling](processing.md#picture-handling) for full configuration.
|
The prompt is used when `processing.pictures` is `"description"`. See [Picture Handling](processing.md#picture-handling) for full configuration.
|
||||||
|
|
||||||
## Programmatic Configuration
|
## Programmatic Configuration
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -226,7 +226,7 @@ async for doc_id in client.rebuild_database(mode=RebuildMode.EMBED_ONLY):
|
||||||
# Add VLM picture descriptions to an existing database — runs the VLM
|
# Add VLM picture descriptions to an existing database — runs the VLM
|
||||||
# over already-stored picture bytes, patches descriptions into the
|
# over already-stored picture bytes, patches descriptions into the
|
||||||
# docling blob, then re-chunks + re-embeds. Requires
|
# docling blob, then re-chunks + re-embeds. Requires
|
||||||
# picture_description.enabled=true in the config.
|
# processing.pictures='description' in the config.
|
||||||
async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
||||||
print(f"Described pictures in {doc_id}")
|
print(f"Described pictures in {doc_id}")
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -498,7 +498,7 @@ def rebuild(
|
||||||
help=(
|
help=(
|
||||||
"Run the VLM over already-stored picture bytes, patch descriptions "
|
"Run the VLM over already-stored picture bytes, patch descriptions "
|
||||||
"into the docling blob, then re-chunk + re-embed. Skips the docling "
|
"into the docling blob, then re-chunk + re-embed. Skips the docling "
|
||||||
"parse entirely. Requires picture_description.enabled=true."
|
"parse entirely. Requires processing.pictures='description'."
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
):
|
):
|
||||||
|
|
|
||||||
|
|
@ -105,7 +105,10 @@ async def download_models(
|
||||||
if config.reranking.model and config.reranking.model.provider == "ollama":
|
if config.reranking.model and config.reranking.model.provider == "ollama":
|
||||||
required_models.add(config.reranking.model.name)
|
required_models.add(config.reranking.model.name)
|
||||||
pic_desc = config.processing.conversion_options.picture_description
|
pic_desc = config.processing.conversion_options.picture_description
|
||||||
if pic_desc.enabled and pic_desc.model.provider == "ollama":
|
if (
|
||||||
|
config.processing.pictures == "description"
|
||||||
|
and pic_desc.model.provider == "ollama"
|
||||||
|
):
|
||||||
required_models.add(pic_desc.model.name)
|
required_models.add(pic_desc.model.name)
|
||||||
if (
|
if (
|
||||||
config.processing.auto_title
|
config.processing.auto_title
|
||||||
|
|
|
||||||
|
|
@ -26,12 +26,12 @@ def _warn_if_descriptions_missing(
|
||||||
docling-serve swallows VLM errors (network failures, missing models,
|
docling-serve swallows VLM errors (network failures, missing models,
|
||||||
etc.) and returns a successful conversion with empty descriptions.
|
etc.) and returns a successful conversion with empty descriptions.
|
||||||
docling-local can do the same when the VLM endpoint is unreachable.
|
docling-local can do the same when the VLM endpoint is unreachable.
|
||||||
Surface the silent failure: when ``picture_description.enabled=True``
|
Surface the silent failure: when ``processing.pictures="description"``
|
||||||
AND the document has at least one picture AND zero descriptions came
|
AND the document has at least one picture AND zero descriptions came
|
||||||
back, log a clear warning so the user can fix their VLM config before
|
back, log a clear warning so the user can fix their VLM config before
|
||||||
a thousand-document ingest produces an empty corpus.
|
a thousand-document ingest produces an empty corpus.
|
||||||
"""
|
"""
|
||||||
if not config.processing.conversion_options.picture_description.enabled:
|
if config.processing.pictures != "description":
|
||||||
return
|
return
|
||||||
if not doc.pictures:
|
if not doc.pictures:
|
||||||
return
|
return
|
||||||
|
|
@ -39,7 +39,7 @@ def _warn_if_descriptions_missing(
|
||||||
if described == 0:
|
if described == 0:
|
||||||
model = config.processing.conversion_options.picture_description.model
|
model = config.processing.conversion_options.picture_description.model
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"picture_description.enabled is True but no descriptions came back "
|
"processing.pictures='description' but no descriptions came back "
|
||||||
"for %s (%d pictures, 0 described). The VLM call likely failed "
|
"for %s (%d pictures, 0 described). The VLM call likely failed "
|
||||||
"silently inside the converter. Check that the VLM at %s is "
|
"silently inside the converter. Check that the VLM at %s is "
|
||||||
"reachable from the converter and that the model name '%s' "
|
"reachable from the converter and that the model name '%s' "
|
||||||
|
|
|
||||||
|
|
@ -383,10 +383,9 @@ async def _rebuild_descriptions(
|
||||||
"""
|
"""
|
||||||
from haiku.rag.embeddings import embed_chunks, get_embedder
|
from haiku.rag.embeddings import embed_chunks, get_embedder
|
||||||
|
|
||||||
if not client._config.processing.conversion_options.picture_description.enabled:
|
if client._config.processing.pictures != "description":
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"rebuild --descriptions requires "
|
"rebuild --descriptions requires processing.pictures = 'description' "
|
||||||
"processing.conversion_options.picture_description.enabled = true "
|
|
||||||
"in your config."
|
"in your config."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -47,30 +47,7 @@ def load_yaml_config(path: Path) -> dict:
|
||||||
"""Load and parse a YAML config file."""
|
"""Load and parse a YAML config file."""
|
||||||
with open(path) as f:
|
with open(path) as f:
|
||||||
data = yaml.safe_load(f)
|
data = yaml.safe_load(f)
|
||||||
data = data or {}
|
return data or {}
|
||||||
_drop_legacy_generate_picture_images(data)
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _drop_legacy_generate_picture_images(data: dict) -> None:
|
|
||||||
"""Drop ``processing.conversion_options.generate_picture_images`` from
|
|
||||||
loaded YAML and log that it is no longer needed.
|
|
||||||
|
|
||||||
Picture bytes are always extracted now, so the flag has no effect.
|
|
||||||
Old YAMLs keep working; users see one log line telling them they can
|
|
||||||
remove the entry.
|
|
||||||
"""
|
|
||||||
processing = data.get("processing")
|
|
||||||
if not isinstance(processing, dict):
|
|
||||||
return
|
|
||||||
opts = processing.get("conversion_options")
|
|
||||||
if isinstance(opts, dict) and "generate_picture_images" in opts:
|
|
||||||
opts.pop("generate_picture_images", None)
|
|
||||||
logger.warning(
|
|
||||||
"Config: 'processing.conversion_options.generate_picture_images' "
|
|
||||||
"no longer has any effect (picture bytes are always extracted). "
|
|
||||||
"Remove it from your haiku.rag.yaml."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def generate_default_config() -> dict:
|
def generate_default_config() -> dict:
|
||||||
|
|
|
||||||
|
|
@ -129,14 +129,12 @@ class AnalysisConfig(BaseModel):
|
||||||
|
|
||||||
|
|
||||||
class PictureDescriptionConfig(BaseModel):
|
class PictureDescriptionConfig(BaseModel):
|
||||||
"""Whether (and how) to run a VLM over each picture at ingest.
|
"""How the VLM runs over each picture when it runs at all.
|
||||||
|
|
||||||
When ``enabled`` is True, picture descriptions are generated by the
|
Activation lives on ``ProcessingConfig.pictures`` — these fields only
|
||||||
configured VLM and woven into chunk text. Picture bytes are stored
|
describe *how* the VLM runs once ``pictures == "description"``.
|
||||||
regardless — this flag only controls the description pass.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
enabled: bool = False
|
|
||||||
model: ModelConfig = Field(
|
model: ModelConfig = Field(
|
||||||
default_factory=lambda: ModelConfig(
|
default_factory=lambda: ModelConfig(
|
||||||
provider="ollama",
|
provider="ollama",
|
||||||
|
|
@ -177,6 +175,9 @@ class ConversionOptions(BaseModel):
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
PicturesMode = Literal["none", "description", "image"]
|
||||||
|
|
||||||
|
|
||||||
class ProcessingConfig(BaseModel):
|
class ProcessingConfig(BaseModel):
|
||||||
chunk_size: int = 256
|
chunk_size: int = 256
|
||||||
converter: str = "docling-local"
|
converter: str = "docling-local"
|
||||||
|
|
@ -186,6 +187,21 @@ class ProcessingConfig(BaseModel):
|
||||||
chunking_merge_peers: bool = True
|
chunking_merge_peers: bool = True
|
||||||
chunking_use_markdown_tables: bool = False
|
chunking_use_markdown_tables: bool = False
|
||||||
conversion_options: ConversionOptions = Field(default_factory=ConversionOptions)
|
conversion_options: ConversionOptions = Field(default_factory=ConversionOptions)
|
||||||
|
pictures: PicturesMode = "image"
|
||||||
|
"""How embedded pictures are handled at ingest.
|
||||||
|
|
||||||
|
- ``"none"``: docling skips picture-image generation; ``label="picture"``
|
||||||
|
rows still exist as structure but carry no bytes or description. Use
|
||||||
|
this when you don't need picture content and want to keep RAM and DB
|
||||||
|
size low on large documents.
|
||||||
|
- ``"description"``: docling generates picture images, the configured
|
||||||
|
VLM produces text descriptions woven into chunk text, AND the bytes
|
||||||
|
are retained in ``document_items.picture_data`` so a vision-capable
|
||||||
|
QA model or multimodal embedder can be enabled later without
|
||||||
|
reingesting.
|
||||||
|
- ``"image"``: docling generates picture images and stores them in
|
||||||
|
``document_items.picture_data``; no VLM runs at ingest.
|
||||||
|
"""
|
||||||
auto_title: bool = False
|
auto_title: bool = False
|
||||||
title_model: ModelConfig = Field(
|
title_model: ModelConfig = Field(
|
||||||
default_factory=lambda: ModelConfig(
|
default_factory=lambda: ModelConfig(
|
||||||
|
|
|
||||||
|
|
@ -119,14 +119,15 @@ class DoclingLocalConverter(DocumentConverter):
|
||||||
|
|
||||||
opts = self.config.processing.conversion_options
|
opts = self.config.processing.conversion_options
|
||||||
pic_desc = opts.picture_description
|
pic_desc = opts.picture_description
|
||||||
runs_vlm = pic_desc.enabled
|
pictures = self.config.processing.pictures
|
||||||
|
runs_vlm = pictures == "description"
|
||||||
|
|
||||||
pipeline_options = PdfPipelineOptions(
|
pipeline_options = PdfPipelineOptions(
|
||||||
do_ocr=opts.do_ocr,
|
do_ocr=opts.do_ocr,
|
||||||
do_table_structure=opts.do_table_structure,
|
do_table_structure=opts.do_table_structure,
|
||||||
images_scale=opts.images_scale,
|
images_scale=opts.images_scale,
|
||||||
generate_page_images=opts.generate_page_images,
|
generate_page_images=opts.generate_page_images,
|
||||||
generate_picture_images=True,
|
generate_picture_images=pictures != "none",
|
||||||
table_structure_options=TableStructureOptions(
|
table_structure_options=TableStructureOptions(
|
||||||
do_cell_matching=opts.table_cell_matching,
|
do_cell_matching=opts.table_cell_matching,
|
||||||
mode=(
|
mode=(
|
||||||
|
|
|
||||||
|
|
@ -99,7 +99,8 @@ class DoclingServeConverter(DocumentConverter):
|
||||||
"""
|
"""
|
||||||
opts = self.config.processing.conversion_options
|
opts = self.config.processing.conversion_options
|
||||||
pic_desc = opts.picture_description
|
pic_desc = opts.picture_description
|
||||||
runs_vlm = pic_desc.enabled
|
pictures = self.config.processing.pictures
|
||||||
|
runs_vlm = pictures == "description"
|
||||||
|
|
||||||
data: dict[str, str | list[str]] = {
|
data: dict[str, str | list[str]] = {
|
||||||
"to_formats": "json",
|
"to_formats": "json",
|
||||||
|
|
@ -111,7 +112,7 @@ class DoclingServeConverter(DocumentConverter):
|
||||||
"table_cell_matching": str(opts.table_cell_matching).lower(),
|
"table_cell_matching": str(opts.table_cell_matching).lower(),
|
||||||
"images_scale": str(opts.images_scale),
|
"images_scale": str(opts.images_scale),
|
||||||
"image_export_mode": "referenced",
|
"image_export_mode": "referenced",
|
||||||
"include_images": "true",
|
"include_images": str(pictures != "none").lower(),
|
||||||
"do_picture_description": str(runs_vlm).lower(),
|
"do_picture_description": str(runs_vlm).lower(),
|
||||||
"target_type": "zip",
|
"target_type": "zip",
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,3 @@
|
||||||
import logging
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
|
|
@ -9,16 +7,6 @@ from haiku.rag.config.loader import (
|
||||||
generate_default_config,
|
generate_default_config,
|
||||||
load_yaml_config,
|
load_yaml_config,
|
||||||
)
|
)
|
||||||
from haiku.rag.config.loader import logger as loader_logger
|
|
||||||
|
|
||||||
|
|
||||||
class _ListHandler(logging.Handler):
|
|
||||||
def __init__(self) -> None:
|
|
||||||
super().__init__(level=logging.WARNING)
|
|
||||||
self.records: list[logging.LogRecord] = []
|
|
||||||
|
|
||||||
def emit(self, record: logging.LogRecord) -> None:
|
|
||||||
self.records.append(record)
|
|
||||||
|
|
||||||
|
|
||||||
def test_load_yaml_config(tmp_path):
|
def test_load_yaml_config(tmp_path):
|
||||||
|
|
@ -239,39 +227,25 @@ def test_init_config_creates_valid_yaml(tmp_path):
|
||||||
assert config.environment == "production"
|
assert config.environment == "production"
|
||||||
|
|
||||||
|
|
||||||
# Removed-field handling: processing.conversion_options.generate_picture_images
|
|
||||||
# was honored by earlier releases. It's dropped from loaded YAML now and a
|
|
||||||
# warning is logged so the user knows they can remove it from their config.
|
|
||||||
|
|
||||||
|
|
||||||
def _write(tmp_path, body: str):
|
def _write(tmp_path, body: str):
|
||||||
p = tmp_path / "haiku.rag.yaml"
|
p = tmp_path / "haiku.rag.yaml"
|
||||||
p.write_text(body)
|
p.write_text(body)
|
||||||
return p
|
return p
|
||||||
|
|
||||||
|
|
||||||
def test_load_yaml_drops_generate_picture_images(tmp_path):
|
def test_pictures_field_defaults_to_image():
|
||||||
"""``generate_picture_images`` from earlier releases is dropped from
|
"""`processing.pictures` defaults to `'image'` — current ingest behavior
|
||||||
the parsed dict and a warning tells the user to remove it from the
|
(store picture bytes, no VLM) preserved on fresh configs."""
|
||||||
YAML."""
|
cfg = AppConfig()
|
||||||
config_file = _write(
|
assert cfg.processing.pictures == "image"
|
||||||
tmp_path,
|
|
||||||
"""
|
|
||||||
processing:
|
def test_picture_description_has_no_enabled_field():
|
||||||
conversion_options:
|
"""`enabled` is gone from PictureDescriptionConfig — activation is now
|
||||||
generate_picture_images: true
|
controlled by `processing.pictures == 'description'`."""
|
||||||
""",
|
from haiku.rag.config.models import PictureDescriptionConfig
|
||||||
)
|
|
||||||
handler = _ListHandler()
|
assert "enabled" not in PictureDescriptionConfig.model_fields
|
||||||
loader_logger.addHandler(handler)
|
|
||||||
try:
|
|
||||||
data = load_yaml_config(config_file)
|
|
||||||
finally:
|
|
||||||
loader_logger.removeHandler(handler)
|
|
||||||
cfg = AppConfig.model_validate(data)
|
|
||||||
assert cfg.processing.conversion_options.picture_description.enabled is False
|
|
||||||
assert "generate_picture_images" not in data["processing"]["conversion_options"]
|
|
||||||
assert any("generate_picture_images" in r.getMessage() for r in handler.records)
|
|
||||||
|
|
||||||
|
|
||||||
def test_fetch_remote_images_default_true():
|
def test_fetch_remote_images_default_true():
|
||||||
|
|
|
||||||
|
|
@ -574,7 +574,7 @@ class TestDoclingLocalConverter:
|
||||||
|
|
||||||
def test_image_format_shares_pdf_pipeline_options(self, config):
|
def test_image_format_shares_pdf_pipeline_options(self, config):
|
||||||
"""IMAGE FormatOption shares the same PdfPipelineOptions instance as
|
"""IMAGE FormatOption shares the same PdfPipelineOptions instance as
|
||||||
PDF. Without this, `do_ocr` / `picture_description.enabled` / etc.
|
PDF. Without this, `do_ocr` / `processing.pictures='description'` / etc.
|
||||||
silently no-op when ingesting raw `.png` / `.jpg` files (which run
|
silently no-op when ingesting raw `.png` / `.jpg` files (which run
|
||||||
through StandardPdfPipeline). End-to-end image conversion is covered
|
through StandardPdfPipeline). End-to-end image conversion is covered
|
||||||
by the PDF picture test — both paths feed the same pipeline class."""
|
by the PDF picture test — both paths feed the same pipeline class."""
|
||||||
|
|
@ -785,7 +785,7 @@ class TestDoclingLocalConverter:
|
||||||
def test_picture_description_config_defaults(self, config):
|
def test_picture_description_config_defaults(self, config):
|
||||||
"""Test that picture description config has correct defaults."""
|
"""Test that picture description config has correct defaults."""
|
||||||
pic_desc = config.processing.conversion_options.picture_description
|
pic_desc = config.processing.conversion_options.picture_description
|
||||||
assert pic_desc.enabled is False
|
assert config.processing.pictures == "image"
|
||||||
assert pic_desc.model.provider == "ollama"
|
assert pic_desc.model.provider == "ollama"
|
||||||
assert pic_desc.model.name == "ministral-3"
|
assert pic_desc.model.name == "ministral-3"
|
||||||
assert pic_desc.timeout == 90
|
assert pic_desc.timeout == 90
|
||||||
|
|
@ -795,12 +795,12 @@ class TestDoclingLocalConverter:
|
||||||
|
|
||||||
def test_picture_description_config_applied(self, config):
|
def test_picture_description_config_applied(self, config):
|
||||||
"""Test that picture description config is applied to converter."""
|
"""Test that picture description config is applied to converter."""
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
config.processing.conversion_options.picture_description.timeout = 120
|
config.processing.conversion_options.picture_description.timeout = 120
|
||||||
converter = DoclingLocalConverter(config)
|
converter = DoclingLocalConverter(config)
|
||||||
|
|
||||||
|
assert converter.config.processing.pictures == "description"
|
||||||
pic_desc = converter.config.processing.conversion_options.picture_description
|
pic_desc = converter.config.processing.conversion_options.picture_description
|
||||||
assert pic_desc.enabled is True
|
|
||||||
assert pic_desc.timeout == 120
|
assert pic_desc.timeout == 120
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|
@ -812,7 +812,7 @@ class TestDoclingLocalConverter:
|
||||||
# Disable OCR (not needed for native PDF, avoids model downloads)
|
# Disable OCR (not needed for native PDF, avoids model downloads)
|
||||||
config.processing.conversion_options.do_ocr = False
|
config.processing.conversion_options.do_ocr = False
|
||||||
# Enable picture description with Ollama
|
# Enable picture description with Ollama
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
config.processing.conversion_options.picture_description.model.provider = (
|
config.processing.conversion_options.picture_description.model.provider = (
|
||||||
"ollama"
|
"ollama"
|
||||||
)
|
)
|
||||||
|
|
@ -1221,7 +1221,7 @@ class TestDoclingServeConverterPictureDescription:
|
||||||
VLM is enabled."""
|
VLM is enabled."""
|
||||||
import json
|
import json
|
||||||
|
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
config.processing.conversion_options.picture_description.model.provider = (
|
config.processing.conversion_options.picture_description.model.provider = (
|
||||||
"ollama"
|
"ollama"
|
||||||
)
|
)
|
||||||
|
|
@ -1332,7 +1332,7 @@ class TestDoclingServeConverterIntegration:
|
||||||
Note: Not using VCR because this test involves polling with changing task IDs.
|
Note: Not using VCR because this test involves polling with changing task IDs.
|
||||||
"""
|
"""
|
||||||
pdf_path = Path("tests/data/doclaynet.pdf")
|
pdf_path = Path("tests/data/doclaynet.pdf")
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
config.processing.conversion_options.picture_description.model.provider = (
|
config.processing.conversion_options.picture_description.model.provider = (
|
||||||
"ollama"
|
"ollama"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -65,10 +65,10 @@ def caplog_warnings(caplog):
|
||||||
|
|
||||||
|
|
||||||
def test_no_warning_when_picture_description_disabled(caplog_warnings):
|
def test_no_warning_when_picture_description_disabled(caplog_warnings):
|
||||||
"""Default config has picture_description.enabled=False — even a
|
"""Default config has pictures='image' — even a document full of
|
||||||
document full of pictures with no descriptions should not warn."""
|
pictures with no descriptions should not warn."""
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
assert config.processing.conversion_options.picture_description.enabled is False
|
assert config.processing.pictures == "image"
|
||||||
doc = _doc_with_pictures(with_descriptions=False)
|
doc = _doc_with_pictures(with_descriptions=False)
|
||||||
|
|
||||||
_warn_if_descriptions_missing(config, doc, "fake.pdf")
|
_warn_if_descriptions_missing(config, doc, "fake.pdf")
|
||||||
|
|
@ -80,7 +80,7 @@ def test_no_warning_when_doc_has_no_pictures(caplog_warnings):
|
||||||
"""A picture-less document under enabled=True shouldn't warn — there
|
"""A picture-less document under enabled=True shouldn't warn — there
|
||||||
was simply nothing to describe."""
|
was simply nothing to describe."""
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
doc = _doc_without_pictures()
|
doc = _doc_without_pictures()
|
||||||
|
|
||||||
_warn_if_descriptions_missing(config, doc, "no-pictures.txt")
|
_warn_if_descriptions_missing(config, doc, "no-pictures.txt")
|
||||||
|
|
@ -89,12 +89,12 @@ def test_no_warning_when_doc_has_no_pictures(caplog_warnings):
|
||||||
|
|
||||||
|
|
||||||
def test_warns_when_pictures_present_but_no_descriptions(caplog_warnings):
|
def test_warns_when_pictures_present_but_no_descriptions(caplog_warnings):
|
||||||
"""VLM was requested via ``picture_description.enabled = True``, the
|
"""VLM was requested via ``processing.pictures='description'``, the
|
||||||
doc has pictures, but the converter returned zero descriptions
|
doc has pictures, but the converter returned zero descriptions
|
||||||
(docling-serve swallows VLM errors). Warn loudly so the user can fix
|
(docling-serve swallows VLM errors). Warn loudly so the user can fix
|
||||||
their VLM config before a long ingest."""
|
their VLM config before a long ingest."""
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
config.processing.conversion_options.picture_description.model.name = "qwen3.6"
|
config.processing.conversion_options.picture_description.model.name = "qwen3.6"
|
||||||
config.processing.conversion_options.picture_description.model.base_url = (
|
config.processing.conversion_options.picture_description.model.base_url = (
|
||||||
"http://host.docker.internal:11434"
|
"http://host.docker.internal:11434"
|
||||||
|
|
@ -117,7 +117,7 @@ def test_no_warning_when_at_least_one_description_came_back(caplog_warnings):
|
||||||
and does not warn — the user might have area-threshold filtering or
|
and does not warn — the user might have area-threshold filtering or
|
||||||
classification gating."""
|
classification gating."""
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
doc = _doc_with_pictures(with_descriptions=True)
|
doc = _doc_with_pictures(with_descriptions=True)
|
||||||
|
|
||||||
_warn_if_descriptions_missing(config, doc, "fine.pdf")
|
_warn_if_descriptions_missing(config, doc, "fine.pdf")
|
||||||
|
|
@ -160,7 +160,7 @@ async def test_convert_emits_warning_via_chokepoint(
|
||||||
)
|
)
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
await convert(config, pdf)
|
await convert(config, pdf)
|
||||||
|
|
||||||
|
|
@ -201,7 +201,7 @@ async def test_convert_text_path_also_warns(monkeypatch, caplog_warnings):
|
||||||
)
|
)
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
# No URL scheme, no Path → drops into the convert_text branch.
|
# No URL scheme, no Path → drops into the convert_text branch.
|
||||||
await convert(config, "<html><img src='...'/></html>")
|
await convert(config, "<html><img src='...'/></html>")
|
||||||
|
|
|
||||||
|
|
@ -427,12 +427,12 @@ async def test_rebuild_batch_size_flush(temp_db_path, monkeypatch):
|
||||||
assert len(chunks) > 0
|
assert len(chunks) > 0
|
||||||
|
|
||||||
|
|
||||||
async def test_rebuild_descriptions_requires_enabled(temp_db_path):
|
async def test_rebuild_descriptions_requires_description_mode(temp_db_path):
|
||||||
"""Calling rebuild --descriptions without picture_description.enabled is
|
"""Calling rebuild --descriptions without `processing.pictures='description'`
|
||||||
a config error: the user has nothing to gain and the resulting state is
|
is a config error: the user has nothing to gain and the resulting state is
|
||||||
indistinguishable from a plain --rechunk."""
|
indistinguishable from a plain --rechunk."""
|
||||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||||
with pytest.raises(ValueError, match="picture_description.enabled"):
|
with pytest.raises(ValueError, match="processing.pictures"):
|
||||||
async for _ in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
async for _ in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
@ -459,7 +459,7 @@ async def test_rebuild_descriptions_patches_blob_and_chunks(temp_db_path, monkey
|
||||||
docling_doc = _docling_doc_with_picture()
|
docling_doc = _docling_doc_with_picture()
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
document = Document(content="x", uri="test://doc")
|
document = Document(content="x", uri="test://doc")
|
||||||
|
|
@ -543,7 +543,7 @@ async def test_rebuild_descriptions_skips_already_described(temp_db_path, monkey
|
||||||
)
|
)
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
document = Document(content="x", uri="test://doc")
|
document = Document(content="x", uri="test://doc")
|
||||||
|
|
@ -589,7 +589,7 @@ async def test_patch_picture_descriptions_returns_zero_for_doc_without_pictures(
|
||||||
from haiku.rag.config import AppConfig
|
from haiku.rag.config import AppConfig
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
doc = await rag.create_document(content="Just text, no pictures.")
|
doc = await rag.create_document(content="Just text, no pictures.")
|
||||||
|
|
@ -613,7 +613,7 @@ async def test_patch_picture_descriptions_warns_on_missing_bytes(temp_db_path, c
|
||||||
|
|
||||||
docling_doc = _docling_doc_with_picture()
|
docling_doc = _docling_doc_with_picture()
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
document = Document(content="x", uri="test://doc")
|
document = Document(content="x", uri="test://doc")
|
||||||
|
|
@ -670,7 +670,7 @@ async def test_patch_picture_descriptions_skips_when_all_already_described(
|
||||||
)
|
)
|
||||||
|
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
document = Document(content="x", uri="test://doc")
|
document = Document(content="x", uri="test://doc")
|
||||||
|
|
@ -708,7 +708,7 @@ async def test_rebuild_descriptions_raises_when_blob_is_missing(
|
||||||
|
|
||||||
docling_doc = _docling_doc_with_picture()
|
docling_doc = _docling_doc_with_picture()
|
||||||
config = AppConfig()
|
config = AppConfig()
|
||||||
config.processing.conversion_options.picture_description.enabled = True
|
config.processing.pictures = "description"
|
||||||
|
|
||||||
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
async with HaikuRAG(temp_db_path, config=config, create=True) as rag:
|
||||||
document = Document(content="x", uri="test://doc")
|
document = Document(content="x", uri="test://doc")
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue