danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
haiku.rag/docs/configuration/processing.md

17 KiB
Raw Blame History

Document Processing & Monitoring

This guide covers how haiku.rag converts, chunks, and monitors documents.

Document Processing

Configure how documents are converted and chunked:

processing:
  # Chunking configuration
  chunk_size: 256                            # Maximum tokens per chunk

  # Converter selection
  converter: docling-local                   # docling-local or docling-serve

  # Chunker selection and configuration
  chunker: docling-local                     # docling-local or docling-serve
  chunker_type: hybrid                       # hybrid or hierarchical
  chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B"  # HuggingFace model for tokenization
  chunking_merge_peers: true                 # Merge undersized successive chunks
  chunking_use_markdown_tables: false        # Use markdown tables vs narrative format

  # Automatic title generation
  auto_title: false                          # Auto-generate titles on ingestion
  title_model:                               # LLM for title generation (fallback)
    provider: ollama
    name: gpt-oss
    enable_thinking: false

  # Picture handling (none / description / image)
  pictures: none                             # See "Picture Handling" below

  # Conversion options (works with both local and remote converters)
  conversion_options:
    # OCR settings
    do_ocr: true                             # Enable OCR for bitmap content
    force_ocr: false                         # Replace existing text with OCR
    ocr_engine: auto                         # OCR engine: auto, easyocr, rapidocr, tesseract, tesserocr, ocrmac
    ocr_lang: []                             # OCR languages (e.g., ["en", "fr", "de"])

    # Table extraction
    do_table_structure: true                 # Extract table structure
    table_mode: accurate                     # fast or accurate
    table_cell_matching: true                # Match table cells back to PDF cells

    # Image settings
    images_scale: 2.0                        # Image scale factor
    generate_page_images: true               # Include rendered page images (for visualize_chunk)

    # VLM picture description settings (only effective when pictures: description)
    picture_description:
      model:
        provider: ollama
        name: ministral-3

Conversion Options

The conversion_options section allows fine-grained control over document conversion. These options work with both docling-local and docling-serve converters.

OCR Settings

conversion_options:
  do_ocr: true          # Enable OCR for bitmap/scanned content
  force_ocr: false      # Replace all text with OCR output
  ocr_engine: auto      # OCR engine selection
  ocr_lang: []          # List of OCR languages, e.g., ["en", "fr", "de"]
  • do_ocr: When true, applies OCR to images and scanned pages. Disable for faster processing if documents contain only native text.
  • force_ocr: When true, replaces existing text layers with OCR output. Useful for documents with poor text extraction.
  • ocr_engine: Select the OCR engine to use. Options:
    • auto (default): Automatically select the best available engine
    • easyocr: EasyOCR - supports many languages, good accuracy
    • rapidocr: RapidOCR - fast processing
    • tesseract: Tesseract OCR
    • tesserocr: Tesseract via tesserocr Python binding
    • ocrmac: macOS native OCR (macOS only)
  • ocr_lang: List of language codes for OCR. Empty list uses default language detection. Examples: ["en"], ["en", "fr", "de"].

Table Extraction

conversion_options:
  do_table_structure: true    # Extract structured table data
  table_mode: accurate        # fast or accurate
  table_cell_matching: true   # Match cells back to PDF
  • do_table_structure: When true, extracts table structure. Disable for faster processing if tables aren't important.
  • table_mode:
    • accurate: Better table structure recognition (slower)
    • fast: Faster processing with simpler table detection
  • table_cell_matching: When true, matches detected table cells back to PDF cells. Disable if tables have merged cells across columns.

Image Settings

conversion_options:
  images_scale: 2.0               # Image resolution scale factor
  generate_page_images: true      # Include rendered page images
  • images_scale: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
  • generate_page_images: When true (default), rendered images of each PDF page are included in the document. Required for visualize_chunk() to show visual grounding. When false, page images are excluded to reduce document size.

Embedded picture extraction is controlled by processing.pictures (see Picture Handling below), not by an image-settings flag.

Picture Handling

processing.pictures is a single enum that decides how embedded picture images (figures, diagrams) are handled at ingest. Three modes:

Mode VLM at ingest picture_data populated Chunk text contains description
none (default) no no no
description yes yes yes
image no yes no
  • none: docling skips picture-image generation. label="picture" rows still appear in the items table for structural metadata, but they carry no bytes and no description. Cheapest mode; non-vision QA is unaffected.
  • description: docling generates picture images, the configured VLM produces a description woven into chunk text, and the bytes are also retained in document_items.picture_data. Picture-text is searchable via FTS, vision-capable QA models also receive the bytes via the agent's search tool. Bytes are kept (not just thrown away after the VLM runs) so a vision-only QA strategy can be turned on later without reingesting.
  • image: docling generates picture images and stores them in document_items.picture_data without running the VLM. Vision-capable QA models reason directly about the figures; non-vision QA only sees the picture's caption/surrounding text.
processing:
  pictures: description          # none / description / image
  conversion_options:
    picture_description:         # only effective when pictures: description
      model:
        provider: ollama         # ollama, openai, or custom
        name: ministral-3        # VLM model name
        temperature: 0.0
      timeout: 90                # Request timeout in seconds
      max_tokens: 200            # Maximum tokens in response

Switching modes on an existing database. No reingest is required if you only need to change between description and image — the bytes are already there. Run haiku-rag rebuild --rechunk after the config change so the chunk-text composition reflects the new mode. Switching down to none clears picture_data on rebuild, reclaiming storage.

Pictures × embedder × QA model: how the pieces compose

Three orthogonal settings drive what gets stored, what gets retrieved, and what reaches the QA model. Each setting answers one question:

Setting Question it answers Values
processing.pictures What gets captured at ingest? none / description / image
embeddings.model.provider Can the embedder index image content? text-only providers (ollama, openai, cohere, sentence-transformers) vs vllm (multimodal)
qa.model.vision Can the QA model interpret images? false (default) / true

What gets stored for each pictures × embedder combination:

pictures Embedder Text chunks contain… document_items.picture_data Synthetic picture chunks
none text-only or multimodal regular text only empty none
description text-only text + VLM descriptions populated (kept for later use) none
description multimodal text + VLM descriptions populated one per picture, content = description, vector = image embedding
image text-only text only (caption/surrounding) populated (kept for later use) none
image multimodal text only populated one per picture, content = caption/empty, vector = image embedding

What QA receives at search time, given stored state and qa.model.vision:

pictures at ingest qa.model.vision QA receives
none either text chunks only
description false text chunks (descriptions in chunk text answer figure questions in prose)
description true text chunks + raw picture bytes; vision model uses both signals
image false text chunks only (caption + surrounding text); model has no figure content to draw on
image true text chunks + raw picture bytes; vision model reads the figures directly

A few invariants worth knowing:

  • qa.model.vision is independent of ingestion. It only controls whether the agent's search tool attaches picture bytes to its ToolReturn. A text-only QA model with vision: true won't suddenly understand images — it will silently accept the bytes and confabulate. Default false is the safe choice.
  • description mode preserves the bytes, so flipping the QA strategy later (text-only → vision, or vice versa) doesn't require reingesting. Just change qa.model.vision and optionally qa.model itself.
  • Cross-modal search (text query → picture-chunk hits) requires a multimodal embedder. With a text-only embedder, picture-chunk vectors aren't generated; figures only surface via section-bounded expansion off matching text chunks.
  • image mode + text-only embedder is rarely the right choice — pictures are stored but neither searchable as image vectors nor described in text. Picture bytes are then only reachable via expand-context's section bounds when an adjacent text chunk matches.

Recommended combinations by use case:

Use case pictures Embedder qa.model.vision
Pure text RAG, no figures none text-only false
Text RAG, figures answered through descriptions description text-only false
Vision QA on figure-rich docs (no cross-modal search) description text-only true
Cross-modal search + vision QA (the full multimodal stack) description or image multimodal true
Cross-modal search, text QA only description multimodal false

picture_description.model configuration (used only under pictures: description):

  • model: Standard model configuration
    • provider: ollama (default), openai, or use base_url for custom endpoints
    • name: Model name (e.g., ministral-3, granite3.2-vision, gpt-4-vision)
    • base_url: Optional custom API endpoint for vLLM, LM Studio, etc.
  • timeout: Request timeout in seconds
  • max_tokens: Maximum tokens in the VLM response

Note: Requires an OpenAI-compatible /v1/chat/completions endpoint. Providers with different API formats (e.g., Anthropic Claude) are not supported.

Default prompt (configured in prompts.picture_description):

Describe this image for a blind user. State the image type
(screenshot, chart, photo, etc.), what it depicts, any visible text,
and key visual details. Be concise and accurate.

To customize the prompt globally:

prompts:
  picture_description: "Your custom prompt here..."

Using with Ollama:

processing:
  pictures: description
  conversion_options:
    picture_description:
      model:
        provider: ollama
        name: ministral-3

Requires Ollama running with a vision-capable model:

ollama pull ministral-3
ollama serve

Using with vLLM or custom endpoints:

processing:
  pictures: description
  conversion_options:
    picture_description:
      model:
        provider: openai           # Use OpenAI-compatible API format
        name: granite-vision
        base_url: http://my-vllm-server:8000

How it works:

  1. During PDF conversion, docling extracts embedded images
  2. Each image is sent to the configured VLM for description
  3. Descriptions are added as annotations on the image
  4. When exported to markdown, descriptions appear as searchable text

Using with docling-serve:

When using converter: docling-serve, the VLM calls are made by the docling-serve instance, not by haiku.rag. You must:

  1. Set DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true when running docling-serve
  2. Ensure the VLM endpoint is accessible from where docling-serve is running

Docker networking: If docling-serve runs in Docker and your VLM runs on the host, use host.docker.internal instead of localhost:

processing:
  pictures: description
  conversion_options:
    picture_description:
      model:
        provider: ollama
        name: ministral-3
        base_url: http://host.docker.internal:11434  # NOT localhost!

See VLM Picture Description with docling-serve for a complete example.

Automatic Title Generation

Enable automatic title generation during document ingestion:

processing:
  auto_title: true
  title_model:
    provider: ollama
    name: gpt-oss
    enable_thinking: false

When auto_title is enabled, haiku.rag attempts to extract a title for each document during ingestion using a two-tier approach:

  1. Structural extraction (free, no model calls): Scans the DoclingDocument for semantic labels — HTML <title> tags, <h1> headings, PDF title blocks, and section headers
  2. LLM fallback: When no structural title is found (e.g., plain text), generates a title using the configured title_model

Priority order: HTML <title> (furniture layer) → h1/PDF title (body layer) → first section header → LLM generation.

Explicit titles passed via title= parameter always take precedence and are never overridden. When updating documents, existing titles are preserved — auto-generation only applies to untitled documents.

To generate titles for existing untitled documents, use rebuild --title-only.

Local vs Remote Processing

Local processing (default):

  • Uses docling library locally
  • No external dependencies
  • Good for development and small workloads

Remote processing (docling-serve):

  • Offloads processing to docling-serve API
  • Better for heavy workloads and production
  • Requires docling-serve instance (see Remote processing setup)

To use remote processing:

processing:
  converter: docling-serve
  chunker: docling-serve

providers:
  docling_serve:
    base_url: http://localhost:5001
    api_key: "your-api-key"  # Optional

Conversion options work identically for both local and remote processing.

Note: When using chunker: docling-serve, OCR options (do_ocr, force_ocr, ocr_engine, ocr_lang) from conversion_options are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail—set do_ocr: false to disable OCR entirely.

Chunking Strategies

Hybrid chunking (default):

  • Structure-aware chunking
  • Respects document boundaries
  • Best for most use cases

Hierarchical chunking:

  • Creates hierarchical chunk structure
  • Preserves document hierarchy
  • Useful for complex documents

Table Serialization

Control how tables are represented in chunks:

processing:
  chunking_use_markdown_tables: false  # Default: narrative format
  • false: Tables as narrative text ("Value A, Column 2 = Value B")
  • true: Tables as markdown (preserves table structure)

Chunk Size

processing:
  chunk_size: 256  # Maximum tokens per chunk

Context expansion settings (for enriching search results with surrounding content) are configured in the search section. See Search Settings.

File Monitoring

Set directories to monitor for automatic indexing:

monitor:
  directories:
    - /path/to/documents
    - /another_path/to/documents

Filtering Monitored Files

Use gitignore-style patterns to control which files are monitored:

monitor:
  directories:
    - /path/to/documents

  # Exclude specific files or directories
  ignore_patterns:
    - "*draft*"         # Ignore files with "draft" in the name
    - "temp/"           # Ignore temp directory
    - "**/archive/**"   # Ignore all archive directories
    - "*.backup"        # Ignore backup files

  # Only include specific files (whitelist mode)
  include_patterns:
    - "*.md"            # Only markdown files
    - "*.pdf"           # Only PDF files
    - "**/docs/**"      # Only files in docs directories

How patterns work:

  1. Extension filtering - Only supported file types are considered
  2. Include patterns - If specified, only matching files are included (whitelist)
  3. Ignore patterns - Matching files are excluded (blacklist)
  4. Combining both - Include patterns are applied first, then ignore patterns

Common patterns:

# Only monitor markdown documentation, but ignore drafts
monitor:
  include_patterns:
    - "*.md"
  ignore_patterns:
    - "*draft*"
    - "*WIP*"

# Monitor all supported files except in specific directories
monitor:
  ignore_patterns:
    - "node_modules/"
    - ".git/"
    - "**/test/**"
    - "**/temp/**"

Patterns follow gitignore syntax:

  • * matches anything except /
  • ** matches zero or more directories
  • ? matches any single character
  • [abc] matches any character in the set