diff --git a/README.md b/README.md index ec7a5f5f..f7d3aff3 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p - **CLI & Python API** — Full functionality from command line or code - **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.) - **Visual grounding** — View chunks highlighted on original page images -- **Production ingester** — Long-lived `haiku-ingester` service with persistent SQLite queue, async worker pool with retries and a dead-letter queue, FS / HTTP / S3 / WebDAV source adapters, and a FastAPI control plane. See [docs/ingester.md](docs/ingester.md). +- **Production ingester** — Long-lived `haiku-ingester` service with persistent SQLite queue, async worker pool with retries and a dead-letter queue, FS / HTTP / S3 / WebDAV source adapters, FastAPI control plane, and a browser dashboard for operators. See [docs/ingester.md](docs/ingester.md). - **Time travel** — Query the database at any historical point with `--before` - **Inspector** — TUI for browsing documents, chunks, and search results @@ -60,7 +60,7 @@ haiku-rag add-src paper.pdf haiku-rag search "attention mechanism" # Ask questions with citations -haiku-rag ask "What datasets were used for evaluation?" --cite +haiku-rag ask "What datasets were used for evaluation?" # Analyze — complex analytical tasks via code execution haiku-rag analyze "How many documents mention transformers?" @@ -132,15 +132,17 @@ See the [examples directory](examples/) for working examples: Full documentation at: https://ggozad.github.io/haiku.rag/ -- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup -- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML configuration +- [Quickstart](https://ggozad.github.io/haiku.rag/tutorial/) - Provider setup and first ingestion +- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Packages and extras +- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML reference - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs - [Skills](https://ggozad.github.io/haiku.rag/skills/) - The RAG and analysis skills the client wraps -- [Analysis skill](https://ggozad.github.io/haiku.rag/skills/analysis/) - Complex analytical tasks via code execution -- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector -- [Ingester](https://ggozad.github.io/haiku.rag/ingester/) - Production ingester service for continuous indexing from FS, HTTP, S3, and WebDAV +- [Tuning](https://ggozad.github.io/haiku.rag/tuning/) - Retrieval and answer-quality tuning +- [Ingester](https://ggozad.github.io/haiku.rag/ingester/) - Production ingester for continuous indexing from FS, HTTP, S3, and WebDAV - [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration +- [Remote processing](https://ggozad.github.io/haiku.rag/remote-processing/) - Offload conversion to docling-serve +- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector - [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks - [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index f496059f..8508016e 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -212,7 +212,7 @@ conversion_options: - **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0. - **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size. -- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to docling-local. See [Remote processing](../remote-processing.md#html-image-fetching) for the docling-serve limitation. +- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to `docling-local`. **docling-serve doesn't fetch external `` URLs** (the `ConvertDocumentsOptions` API exposes no equivalent flag, and HTML falls through to docling's `fetch_images=False` default); HTML ingested via docling-serve produces picture items with `picture_data=NULL`. Use `converter: docling-local` if you need image bytes from HTML/Markdown. #### External image fetching diff --git a/docs/ingester.md b/docs/ingester.md index 4deaf1ba..79d991d5 100644 --- a/docs/ingester.md +++ b/docs/ingester.md @@ -14,6 +14,19 @@ Use the ingester when: For one-off ingestion, the `haiku-rag add-src` CLI is enough — see [CLI → Add Documents](cli.md). +**On this page:** + +- [Install](#install) +- [Configure sources](#configure-sources) (FS, S3, HTTP, WebDAV) +- [Workers and retry](#workers-and-retry) +- [Circuit breaker](#circuit-breaker) +- [Run it](#run-it) +- [HTTP control plane](#http-control-plane) +- [Operating](#operating) (smoke test, queue inspection, logs, API) + +Single-writer constraint: only one ingester per LanceDB. See +[Storage → Deployment Pattern](configuration/storage.md#deployment-pattern-one-writer-many-readers). + ## Install The ingester ships behind an optional extra: diff --git a/docs/remote-processing.md b/docs/remote-processing.md index 1e84414d..2ba5d9eb 100644 --- a/docs/remote-processing.md +++ b/docs/remote-processing.md @@ -64,156 +64,38 @@ providers: api_key: "" # Optional API key for authentication ``` -## Features +For converter / chunker config options (chunking strategy, tokenizer, +OCR, table handling, picture description), see +[Document Processing](configuration/processing.md). The configuration is +identical between `docling-local` and `docling-serve` modes — this page +covers only what's specific to running docling-serve as a separate +service. -### Remote Document Conversion +## VLM picture description with docling-serve -When `converter: docling-serve` is configured, documents are sent to the docling-serve API for conversion: +When `processing.pictures = "description"` and `converter: docling-serve`, +the VLM API calls are made by the docling-serve container, not by +haiku.rag. Two deployment caveats: -```python -from haiku.rag.client import HaikuRAG +### Enable remote services -async with HaikuRAG() as client: - # PDF is processed by docling-serve - doc = await client.create_document_from_source("complex.pdf") -``` - -### Remote Chunking - -When `chunker: docling-serve` is configured, chunking is performed remotely: - -```yaml -processing: - chunker: docling-serve - chunker_type: hybrid # or hierarchical - chunk_size: 256 - chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" - chunking_merge_peers: true - chunking_use_markdown_tables: false -``` - -## Advanced Configuration - -### Custom Tokenizers - -You can use any HuggingFace tokenizer model: - -```yaml -processing: - chunking_tokenizer: "bert-base-uncased" # Or any HF model -``` - -### Chunking Strategies - -**Hybrid Chunking** (default): - -- Best for most documents -- Preserves semantic boundaries -- Structure-aware splitting - -**Hierarchical Chunking**: - -- Maintains document hierarchy -- Better for deeply nested documents -- Preserves parent-child relationships - -```yaml -processing: - chunker_type: hierarchical -``` - -### Table Handling - -Control how tables are represented: - -```yaml -processing: - chunking_use_markdown_tables: true # Preserve table structure -``` - -- `false` (default): Tables as narrative text -- `true`: Tables as markdown format - -## HTML Image Fetching - -docling-serve does **not** fetch external `` URLs in HTML inputs. The `ConvertDocumentsOptions` API exposes no equivalent of docling-local's `HTMLBackendOptions.fetch_images` / `enable_remote_fetch`, and the server-side `DoclingConverterManager` registers `format_options` only for PDF and IMAGE. HTML falls through to docling's defaults (`fetch_images=False`). - -Consequence: ingesting HTML with external image references through docling-serve produces picture items with `picture_data=NULL`. The same input through docling-local fetches the bytes (subject to the SSRF / size / timeout guards documented in [Configuration → External image fetching](configuration/processing.md#external-image-fetching)). - -To preserve image bytes when ingesting HTML or Markdown that references remote images, use `converter: docling-local`. The `processing.conversion_options.fetch_remote_images` flag has no effect on docling-serve and the `source_uri` kwarg on `HaikuRAG.convert()` is accepted but ignored on this path. - -## VLM Picture Description with docling-serve - -When using VLM picture description with docling-serve, the VLM API calls are made by the docling-serve container, not by haiku.rag. This requires additional configuration. - -### Enable Remote Services - -docling-serve blocks external API calls by default. To enable VLM picture description, start docling-serve with: +docling-serve blocks outbound calls by default. Enable them by setting +`DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` on the container: ```bash -docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve +docker run -p 5001:5001 \ + -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \ + quay.io/docling-project/docling-serve ``` -### Docker Networking +### Reach host services from inside the container -When docling-serve runs in Docker and your VLM (e.g., Ollama) runs on the host, `localhost` inside the container refers to the container itself, not your host machine. - -Use `host.docker.internal` to reach host services from within Docker: - -```yaml -# haiku.rag.yaml -processing: - converter: docling-serve - chunker: docling-serve - conversion_options: - picture_description: - enabled: true - model: - provider: ollama - name: ministral-3 - base_url: http://host.docker.internal:11434 # NOT localhost! -``` - -### Complete Example - -1. Start Ollama with a vision model on your host: - -```bash -ollama pull ministral-3 -ollama serve -``` - -2. Start docling-serve with remote services enabled: - -```bash -docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve -``` - -3. Configure haiku.rag: - -```yaml -# haiku.rag.yaml -processing: - converter: docling-serve - chunker: docling-serve - conversion_options: - picture_description: - enabled: true - model: - provider: ollama - name: ministral-3 - base_url: http://host.docker.internal:11434 - -providers: - docling_serve: - base_url: http://localhost:5001 -``` - -4. Add a document: - -```bash -haiku-rag add-src document.pdf -``` +If your VLM (e.g. Ollama) runs on the host while docling-serve runs in +Docker, set the VLM's `base_url` in +`processing.conversion_options.picture_description.model` to +`http://host.docker.internal:11434` rather than `localhost`. See +[Document Processing → Picture Handling](configuration/processing.md#picture-handling) +for the full config snippet. ## Operational notes diff --git a/docs/tuning.md b/docs/tuning.md index 904ded76..79ffc657 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -2,6 +2,8 @@ How to adjust haiku.rag's pipeline for better retrieval and answer quality. For individual setting definitions and defaults, see [Configuration](configuration/index.md). +For ingester-side tuning (worker count, claim timeout, retry policy, backpressure, circuit breakers), see [Ingester → Workers and retry](ingester.md#workers-and-retry). + ## Pipeline Overview Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is the highest-leverage stage. If the LLM never sees the right chunks, no prompt or model change will help. diff --git a/zensical.toml b/zensical.toml index 16071d4a..1bcfc242 100644 --- a/zensical.toml +++ b/zensical.toml @@ -35,8 +35,8 @@ nav = [ { Tuning = "tuning.md" }, ] }, { Production = [ - { MCP = "mcp.md" }, { Ingester = "ingester.md" }, + { MCP = "mcp.md" }, { "Remote processing" = "remote-processing.md" }, ] }, { Develop = [