Docs update

README.md

@@ -22,7 +22,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p
 - **CLI & Python API** — Full functionality from command line or code
 - **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
 - **Visual grounding** — View chunks highlighted on original page images
-- **Production ingester** — Long-lived `haiku-ingester` service with persistent SQLite queue, async worker pool with retries and a dead-letter queue, FS / HTTP / S3 / WebDAV source adapters, and a FastAPI control plane. See [docs/ingester.md](docs/ingester.md).
+- **Production ingester** — Long-lived `haiku-ingester` service with persistent SQLite queue, async worker pool with retries and a dead-letter queue, FS / HTTP / S3 / WebDAV source adapters, FastAPI control plane, and a browser dashboard for operators. See [docs/ingester.md](docs/ingester.md).
 - **Time travel** — Query the database at any historical point with `--before`
 - **Inspector** — TUI for browsing documents, chunks, and search results
 
@@ -60,7 +60,7 @@ haiku-rag add-src paper.pdf
 haiku-rag search "attention mechanism"
 
 # Ask questions with citations
-haiku-rag ask "What datasets were used for evaluation?" --cite
+haiku-rag ask "What datasets were used for evaluation?"
 
 # Analyze — complex analytical tasks via code execution
 haiku-rag analyze "How many documents mention transformers?"
@@ -132,15 +132,17 @@ See the [examples directory](examples/) for working examples:
 
 Full documentation at: https://ggozad.github.io/haiku.rag/
 
-- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
+- [Quickstart](https://ggozad.github.io/haiku.rag/tutorial/) - Provider setup and first ingestion
-- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML configuration
+- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Packages and extras
+- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML reference
 - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
 - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
 - [Skills](https://ggozad.github.io/haiku.rag/skills/) - The RAG and analysis skills the client wraps
-- [Analysis skill](https://ggozad.github.io/haiku.rag/skills/analysis/) - Complex analytical tasks via code execution
+- [Tuning](https://ggozad.github.io/haiku.rag/tuning/) - Retrieval and answer-quality tuning
-- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector
+- [Ingester](https://ggozad.github.io/haiku.rag/ingester/) - Production ingester for continuous indexing from FS, HTTP, S3, and WebDAV
-- [Ingester](https://ggozad.github.io/haiku.rag/ingester/) - Production ingester service for continuous indexing from FS, HTTP, S3, and WebDAV
 - [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
+- [Remote processing](https://ggozad.github.io/haiku.rag/remote-processing/) - Offload conversion to docling-serve
+- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector
 - [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks
 - [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history
 

docs/configuration/processing.md

@@ -212,7 +212,7 @@ conversion_options:
 
 - **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
 - **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size.
-- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `<img src="https://...">` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to docling-local. See [Remote processing](../remote-processing.md#html-image-fetching) for the docling-serve limitation.
+- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `<img src="https://...">` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to `docling-local`. **docling-serve doesn't fetch external `<img>` URLs** (the `ConvertDocumentsOptions` API exposes no equivalent flag, and HTML falls through to docling's `fetch_images=False` default); HTML ingested via docling-serve produces picture items with `picture_data=NULL`. Use `converter: docling-local` if you need image bytes from HTML/Markdown.
 
 #### External image fetching
 

docs/ingester.md

@@ -14,6 +14,19 @@ Use the ingester when:
 For one-off ingestion, the `haiku-rag add-src` CLI is enough — see
 [CLI → Add Documents](cli.md).
 
+**On this page:**
+
+- [Install](#install)
+- [Configure sources](#configure-sources) (FS, S3, HTTP, WebDAV)
+- [Workers and retry](#workers-and-retry)
+- [Circuit breaker](#circuit-breaker)
+- [Run it](#run-it)
+- [HTTP control plane](#http-control-plane)
+- [Operating](#operating) (smoke test, queue inspection, logs, API)
+
+Single-writer constraint: only one ingester per LanceDB. See
+[Storage → Deployment Pattern](configuration/storage.md#deployment-pattern-one-writer-many-readers).
+
 ## Install
 
 The ingester ships behind an optional extra:

docs/remote-processing.md

@@ -64,156 +64,38 @@ providers:
     api_key: ""  # Optional API key for authentication
 ```
 
-## Features
+For converter / chunker config options (chunking strategy, tokenizer,
+OCR, table handling, picture description), see
+[Document Processing](configuration/processing.md). The configuration is
+identical between `docling-local` and `docling-serve` modes — this page
+covers only what's specific to running docling-serve as a separate
+service.
 
-### Remote Document Conversion
+## VLM picture description with docling-serve
 
-When `converter: docling-serve` is configured, documents are sent to the docling-serve API for conversion:
+When `processing.pictures = "description"` and `converter: docling-serve`,
+the VLM API calls are made by the docling-serve container, not by
+haiku.rag. Two deployment caveats:
 
-```python
+### Enable remote services
-from haiku.rag.client import HaikuRAG
 
-async with HaikuRAG() as client:
+docling-serve blocks outbound calls by default. Enable them by setting
-    # PDF is processed by docling-serve
+`DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` on the container:
-    doc = await client.create_document_from_source("complex.pdf")
-```
-
-### Remote Chunking
-
-When `chunker: docling-serve` is configured, chunking is performed remotely:
-
-```yaml
-processing:
-  chunker: docling-serve
-  chunker_type: hybrid              # or hierarchical
-  chunk_size: 256
-  chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B"
-  chunking_merge_peers: true
-  chunking_use_markdown_tables: false
-```
-
-## Advanced Configuration
-
-### Custom Tokenizers
-
-You can use any HuggingFace tokenizer model:
-
-```yaml
-processing:
-  chunking_tokenizer: "bert-base-uncased"  # Or any HF model
-```
-
-### Chunking Strategies
-
-**Hybrid Chunking** (default):
-
-- Best for most documents
-- Preserves semantic boundaries
-- Structure-aware splitting
-
-**Hierarchical Chunking**:
-
-- Maintains document hierarchy
-- Better for deeply nested documents
-- Preserves parent-child relationships
-
-```yaml
-processing:
-  chunker_type: hierarchical
-```
-
-### Table Handling
-
-Control how tables are represented:
-
-```yaml
-processing:
-  chunking_use_markdown_tables: true  # Preserve table structure
-```
-
-- `false` (default): Tables as narrative text
-- `true`: Tables as markdown format
-
-## HTML Image Fetching
-
-docling-serve does **not** fetch external `<img src="https://...">` URLs in HTML inputs. The `ConvertDocumentsOptions` API exposes no equivalent of docling-local's `HTMLBackendOptions.fetch_images` / `enable_remote_fetch`, and the server-side `DoclingConverterManager` registers `format_options` only for PDF and IMAGE. HTML falls through to docling's defaults (`fetch_images=False`).
-
-Consequence: ingesting HTML with external image references through docling-serve produces picture items with `picture_data=NULL`. The same input through docling-local fetches the bytes (subject to the SSRF / size / timeout guards documented in [Configuration → External image fetching](configuration/processing.md#external-image-fetching)).
-
-To preserve image bytes when ingesting HTML or Markdown that references remote images, use `converter: docling-local`. The `processing.conversion_options.fetch_remote_images` flag has no effect on docling-serve and the `source_uri` kwarg on `HaikuRAG.convert()` is accepted but ignored on this path.
-
-## VLM Picture Description with docling-serve
-
-When using VLM picture description with docling-serve, the VLM API calls are made by the docling-serve container, not by haiku.rag. This requires additional configuration.
-
-### Enable Remote Services
-
-docling-serve blocks external API calls by default. To enable VLM picture description, start docling-serve with:
 
 ```bash
-docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve
+docker run -p 5001:5001 \
+  -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \
+  quay.io/docling-project/docling-serve
 ```
 
-### Docker Networking
+### Reach host services from inside the container
 
-When docling-serve runs in Docker and your VLM (e.g., Ollama) runs on the host, `localhost` inside the container refers to the container itself, not your host machine.
+If your VLM (e.g. Ollama) runs on the host while docling-serve runs in
-
+Docker, set the VLM's `base_url` in
-Use `host.docker.internal` to reach host services from within Docker:
+`processing.conversion_options.picture_description.model` to
-
+`http://host.docker.internal:11434` rather than `localhost`. See
-```yaml
+[Document Processing → Picture Handling](configuration/processing.md#picture-handling)
-# haiku.rag.yaml
+for the full config snippet.
-processing:
-  converter: docling-serve
-  chunker: docling-serve
-  conversion_options:
-    picture_description:
-      enabled: true
-      model:
-        provider: ollama
-        name: ministral-3
-        base_url: http://host.docker.internal:11434  # NOT localhost!
-```
-
-### Complete Example
-
-1. Start Ollama with a vision model on your host:
-
-```bash
-ollama pull ministral-3
-ollama serve
-```
-
-2. Start docling-serve with remote services enabled:
-
-```bash
-docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true quay.io/docling-project/docling-serve
-```
-
-3. Configure haiku.rag:
-
-```yaml
-# haiku.rag.yaml
-processing:
-  converter: docling-serve
-  chunker: docling-serve
-  conversion_options:
-    picture_description:
-      enabled: true
-      model:
-        provider: ollama
-        name: ministral-3
-        base_url: http://host.docker.internal:11434
-
-providers:
-  docling_serve:
-    base_url: http://localhost:5001
-```
-
-4. Add a document:
-
-```bash
-haiku-rag add-src document.pdf
-```
 
 ## Operational notes
 

docs/tuning.md

@@ -2,6 +2,8 @@
 
 How to adjust haiku.rag's pipeline for better retrieval and answer quality. For individual setting definitions and defaults, see [Configuration](configuration/index.md).
 
+For ingester-side tuning (worker count, claim timeout, retry policy, backpressure, circuit breakers), see [Ingester → Workers and retry](ingester.md#workers-and-retry).
+
 ## Pipeline Overview
 
 Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is the highest-leverage stage. If the LLM never sees the right chunks, no prompt or model change will help.

zensical.toml

@@ -35,8 +35,8 @@ nav = [
         { Tuning = "tuning.md" },
     ] },
     { Production = [
-        { MCP = "mcp.md" },
         { Ingester = "ingester.md" },
+        { MCP = "mcp.md" },
         { "Remote processing" = "remote-processing.md" },
     ] },
     { Develop = [