Wait for drained jobs instead of sleeping in the breaker isolation test

test_breaker_isolates_sources snapshotted job status after a fixed 0.2s
sleep and failed on a loaded runner while two good jobs were still in
flight. Poll for the drained jobs under a 5s deadline.

Assert the paused source's jobs are unattempted: the queued-uri assertion
alone passes with source isolation disabled, since a retried job returns
to QUEUED.

.agents/plugins/marketplace.json

@@ -0,0 +1,20 @@
+{
+  "name": "haiku-rag",
+  "interface": {
+    "displayName": "haiku.rag"
+  },
+  "plugins": [
+    {
+      "name": "haiku-rag",
+      "source": {
+        "source": "local",
+        "path": "./plugins/haiku-rag"
+      },
+      "policy": {
+        "installation": "AVAILABLE",
+        "authentication": "ON_INSTALL"
+      },
+      "category": "Productivity"
+    }
+  ]
+}

.claude-plugin/marketplace.json

@@ -0,0 +1,14 @@
+{
+  "name": "haiku-rag",
+  "description": "The haiku.rag knowledge base as Claude Code tools and a skill.",
+  "owner": {
+    "name": "Yiorgis Gozadinos"
+  },
+  "plugins": [
+    {
+      "name": "haiku-rag",
+      "source": "./plugins/haiku-rag",
+      "description": "Search, read and analyze your haiku.rag knowledge base from Claude Code."
+    }
+  ]
+}

.claude/skills/debug-evals/SKILL.md

@@ -0,0 +1,295 @@
+---
+name: debug-evals
+description: Debug haiku.rag evaluation runs in Logfire. Use when asked to look at Logfire for an eval run, find failing or low-scoring eval cases, compare runs, check citation quality (cited_map) or judge pass rate (answer_equivalent), or explain why an eval case failed. Drives the Logfire MCP against the `evals` service, or the Logfire HTTP query API when the MCP is not loaded. Also covers monitoring a run that is still in flight.
+---
+
+# Debug eval runs in Logfire
+
+Eval runs (`evaluations/`) ship spans to Logfire under `service_name = 'evals'`.
+This skill finds a run, surfaces its metrics and failures, and drills into a
+single case. Read-only.
+
+## How to query
+
+1. Confirm the current schema with `mcp__logfire__query_schema_reference` (spans
+   and logs share the `records` table).
+2. Run SQL with `mcp__logfire__query_run` (`query` + `project: "haiku"` +
+   `start_timestamp`/`end_timestamp`, max 14 days). The remote MCP is
+   org-scoped, so `project` is required; eval runs land in project `haiku`.
+   The same SQL works pasted into Logfire's Explore UI.
+3. Read span attributes as JSON: `attributes->>'key'`, nested as
+   `attributes->'a'->'b'->>'c'`. Cast when needed: `(...)::float`, `(...)::int`.
+4. Hand back a clickable trace with
+   `mcp__logfire__project_logfire_link(trace_id, project="haiku")`.
+
+## When the Logfire MCP is not available
+
+The `mcp__logfire__*` tools are not loaded in every session. The HTTP query API is
+the fallback and needs no MCP:
+
+```
+POST https://logfire-eu.pydantic.dev/v2/query      # EU projects
+POST https://logfire-us.pydantic.dev/v2/query      # US projects
+Authorization: Bearer <api-key>
+Content-Type: application/json
+{"sql": "...", "min_timestamp": "2026-08-23T00:00:00Z"}
+```
+
+- **`min_timestamp` is mandatory** and silently bounds every result. Too recent a
+  value is indistinguishable from "no data".
+- Read tokens are being replaced by **API keys** (`pylf_v2_<region>_...`). Same
+  `Authorization: Bearer` header; the region is in the prefix.
+- Keep the key in `~/.logfire-read-key` (mode 600) and read it from there so it never
+  lands in a transcript. Helper next to this skill: `.claude/skills/debug-evals/lf-query.sh "<SQL>" [min_ts]`.
+- The API is **project-scoped**. A key for the wrong project authenticates fine and
+  returns zero rows — it does not error. Diagnose in this order:
+  1. wrong region → `HTTP 401 Invalid read token` on the other host;
+  2. wrong project → auth succeeds, `count(*)` over months is 0;
+  3. right project → `SELECT service_name, count(*) ... GROUP BY service_name` shows
+     `evals`, `haiku-rag`, `haiku-ingester`.
+  Eval runs live in project **`haiku`**. There is an empty project named `evals`,
+  which is the natural wrong guess.
+- **The API caps returned rows and does not say so.** Aggregate server-side
+  (`count(*)`, `avg(...)`, `sum(CASE WHEN ...)`) rather than pulling rows and counting
+  them locally. A pass rate computed from a clipped page is wrong and looks fine.
+
+### JSON access via the HTTP API
+
+`assertions`, `scores`, `metrics` and `case_name` are **keys inside the `attributes`
+column, not columns** — `SELECT assertions` fails with `column not found`. Both of
+these work:
+
+```sql
+attributes->'assertions'->'answer_equivalent'          -- returns JSON
+json_get_bool(attributes,'assertions','answer_equivalent','value')
+json_get_float(attributes,'scores','cited_map','value')
+json_get_int(attributes,'attributes','n_searches')
+json_get_str(attributes,'attributes','citation_status')
+```
+
+Prefer the `json_get_*` form inside aggregates — it yields a typed value, so no cast
+is needed and `sum(CASE WHEN ...)` behaves.
+
+## Counting cases, not spans
+
+**One exception appears once per span level.** A single failing case emits the same
+`exception_type` on `case: {case_name}`, `execute {task}` and `invoke_agent agent`
+(and often `chat {model}`), so a raw count over-reports by 3-4x. Always add
+`AND span_name='case: {case_name}'` when counting failures. Cross-check that the
+number equals the count of unjudged cases.
+
+## Always report floor as well as judged
+
+`assertion_pass_rate` **excludes unjudged cases from the denominator**, so cases that
+died produce no verdict and silently inflate the headline. Report both:
+
+- judged rate = passed / (cases - unjudged)
+- floor = passed / cases
+
+A run with 4.5% deaths reads ~3pp better than it is. Quote them together, always.
+
+## Vocabulary
+
+A run is one experiment span; its cases are direct children sharing its
+`trace_id`.
+
+- Experiment span: `span_name = 'evaluate {name}'` (scope `pydantic-evals`).
+  - `attributes->>'name'` — run label (the `--name` arg, or `{dataset}_qa_evaluation` / `{dataset}_retrieval_evaluation`).
+  - `attributes->>'dataset_name'` — dataset.
+  - `(attributes->>'assertion_pass_rate')::float` — overall judge pass rate (QA runs).
+  - `attributes->'logfire.experiment.metadata'->'metadata'` — run config: `target` (`rag-capability`|`analysis-capability`), `qa_model`, `embedder_model`, `chunk_size`, `search_limit`, `rerank_model`, `judge_model`, `qa_max_searches`, etc.
+  - `trace_id` — scopes the whole run.
+- Case span: `span_name = 'case: {case_name}'` (scope `pydantic-evals`).
+  - `message` — `case: <id>`.
+  - `attributes->'assertions'->'answer_equivalent'->>'value'` — `'true'`/`'false'` (LLM judge verdict). `->>'reason'` — why.
+  - `attributes->'scores'->'cited_map'->>'value'` — citation average precision (0..1).
+  - `attributes->'scores'->'number_match'->>'value'` — numeric-answer match (datasets that use it).
+  - `duration` — task time in seconds.
+- Inside each case the capability under test emits agent spans (scope `pydantic-ai`):
+  `execute {task}`, `invoke_agent agent`, `execute_tool {tool_name}`, `chat {model}`.
+
+The service is `evals` regardless of model, so filter on `service_name = 'evals'`
+first. `otel_scope_name` separates the layers (`pydantic-evals` for run/case,
+`pydantic-ai` for the agent).
+
+## Canned queries
+
+Recent runs (pick a `trace_id` to drill in):
+
+```sql
+SELECT attributes->>'name' AS run,
+       attributes->>'dataset_name' AS dataset,
+       service_version,
+       (attributes->>'assertion_pass_rate')::float AS pass_rate,
+       start_timestamp, trace_id
+FROM records
+WHERE service_name='evals' AND span_name='evaluate {name}'
+ORDER BY start_timestamp DESC
+LIMIT 20;
+```
+
+Run summary (pass rate, mean citation score, mean task time):
+
+```sql
+SELECT count(*) AS cases,
+       avg(CASE WHEN attributes->'assertions'->'answer_equivalent'->>'value'='true'
+                THEN 1.0 ELSE 0.0 END) AS pass_rate,
+       avg((attributes->'scores'->'cited_map'->>'value')::float) AS mean_cited_map,
+       avg(duration) AS mean_task_seconds
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND trace_id='<TRACE_ID>';
+```
+
+Failing cases (judge said not equivalent), newest first, with the reason:
+
+```sql
+SELECT message AS case_name, duration,
+       attributes->'assertions'->'answer_equivalent'->>'reason' AS reason
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND trace_id='<TRACE_ID>'
+  AND attributes->'assertions'->'answer_equivalent'->>'value'='false'
+ORDER BY start_timestamp;
+```
+
+Low-citation cases (answer may be right but grounding is weak):
+
+```sql
+SELECT message AS case_name,
+       (attributes->'scores'->'cited_map'->>'value')::float AS cited_map
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND trace_id='<TRACE_ID>'
+  AND (attributes->'scores'->'cited_map'->>'value')::float < 0.5
+ORDER BY cited_map;
+```
+
+Error / null cases in a run (an exception aborted the case):
+
+```sql
+SELECT message, span_name, exception_type, exception_message
+FROM records
+WHERE service_name='evals' AND trace_id='<TRACE_ID>' AND is_exception=true
+ORDER BY start_timestamp
+LIMIT 50;
+```
+
+Slowest cases (task time drives run cost):
+
+```sql
+SELECT message AS case_name, duration
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND trace_id='<TRACE_ID>'
+ORDER BY duration DESC
+LIMIT 20;
+```
+
+Drill into one case's agent activity (all cases share the run `trace_id`, so
+bound by the case's own time window):
+
+```sql
+SELECT span_name, message, duration, is_exception
+FROM records
+WHERE service_name='evals' AND trace_id='<TRACE_ID>'
+  AND otel_scope_name='pydantic-ai'
+  AND start_timestamp BETWEEN '<CASE_START>' AND '<CASE_END>'
+ORDER BY start_timestamp;
+```
+
+## Workflow
+
+1. List recent runs, pick the one to inspect by `name` + `start_timestamp`, note
+   its `trace_id`.
+2. Run the summary query for the headline numbers (pass rate, mean_cited_map,
+   mean_task_seconds — always report task time).
+3. Pull failing and low-citation cases, read the judge `reason`.
+4. To understand one case, take its start/end from the case query and run the
+   agent-activity query, then `project_logfire_link(trace_id)` so the user can
+   expand that case in the UI.
+
+## When a query returns nothing
+
+Span names or attributes may have changed. Probe:
+
+```sql
+SELECT DISTINCT otel_scope_name, span_name
+FROM records WHERE service_name='evals'
+ORDER BY 1,2;
+```
+
+## Monitoring a run that is still in flight
+
+An eval prints nothing until it finishes, so a live run's only progress signal is its
+case spans. Everything below works mid-run.
+
+Progress and ETA:
+
+```sql
+SELECT count(*) AS cases_done,
+       min(start_timestamp) AS first_case,
+       max(start_timestamp) AS latest_case,
+       avg(duration) AS avg_case_s
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND start_timestamp > '<RUN_LAUNCH_TS>';
+```
+
+**Cases run serially**, so `ETA_total = total_cases * avg_case_s`. Verify rather than
+assume: wall-clock per case (`latest_case - first_case` over `cases_done`) should equal
+`avg_case_s`. If it does, concurrency is 1 and the multiplication is valid. Concurrent
+requests seen on the model endpoint (`vllm:num_requests_running` > 1) are parallel
+searches *within* one case, not parallel cases.
+
+**Never estimate a run's length from a `--limit N` smoke.** Its "avg task time per
+case" is a per-case duration on the easiest N cases of a deterministic prefix; the full
+set ran 32% slower (72.7s vs 55.2s) on FRAMES. Smokes validate wiring, not wall-clock.
+
+Live headline, behaviour and failure composition in one pass:
+
+```sql
+SELECT count(*) AS cases,
+       sum(CASE WHEN json_get_bool(attributes,'assertions','answer_equivalent','value')
+                THEN 1 ELSE 0 END) AS passed,
+       sum(CASE WHEN json_get(attributes,'assertions','answer_equivalent') IS NULL
+                THEN 1 ELSE 0 END) AS unjudged,
+       avg(json_get_float(attributes,'scores','cited_map','value')) AS cited_map,
+       avg(json_get_int(attributes,'attributes','n_requests')) AS req_per_case,
+       avg(json_get_int(attributes,'attributes','n_searches')) AS searches,
+       avg(json_get_int(attributes,'attributes','n_executions')) AS execs,
+       sum(json_get_int(attributes,'attributes','n_rejected_searches')) AS rejected,
+       sum(CASE WHEN json_get_str(attributes,'attributes','citation_status')='grounded'
+                THEN 1 ELSE 0 END) AS grounded
+FROM records
+WHERE service_name='evals' AND span_name='case: {case_name}'
+  AND start_timestamp > '<RUN_LAUNCH_TS>';
+```
+
+Why a case died, counted correctly:
+
+```sql
+SELECT sum(CASE WHEN exception_message LIKE '%token limit%' THEN 1 ELSE 0 END) AS token_limit,
+       sum(CASE WHEN exception_message NOT LIKE '%token limit%' THEN 1 ELSE 0 END) AS other,
+       count(*) AS dead_cases
+FROM records
+WHERE service_name='evals' AND is_exception
+  AND span_name='case: {case_name}'
+  AND start_timestamp > '<RUN_LAUNCH_TS>';
+```
+
+`ToolFailedError` is mostly **not** a defect — an exhausted search or code budget
+reports failure to the model on purpose. `UnexpectedModelBehavior` is the one that
+kills a case.
+
+### The per-case diagnostic attributes
+
+`attributes->'attributes'` on a case span carries what the capability actually did:
+`n_requests`, `n_searches`, `n_search_calls`, `n_rejected_searches`, `n_failed_tools`,
+`n_executions`, `cited_uris`, `cited_chunk_ids`, `searched_uris`, `citation_status`
+(`grounded` | `missing` | `ungrounded`). `attributes->'metrics'` carries `requests`,
+`input_tokens`, `output_tokens` for the case.
+
+Cite rate comes from `citation_status`, not from `cited_map` — they answer different
+questions, and conflating them has produced wrong claims before. And never steer on
+raw cite rate: it is confounded by task success, so measure it among *correct* answers.

.claude/skills/debug-evals/lf-query.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Query the Logfire API. Key is read from ~/.logfire-read-key and never echoed.
+# usage: lf-query.sh "<SQL>" [min_timestamp]
+set -uo pipefail
+KEY_FILE="$HOME/.logfire-read-key"
+[ -r "$KEY_FILE" ] || { echo "missing $KEY_FILE"; exit 2; }
+SQL="${1:?need SQL}"
+MIN="${2:-$(date -u -v-2d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '2 days ago' +%Y-%m-%dT%H:%M:%SZ)}"
+python3 - "$SQL" "$MIN" <<'PY'
+import json, os, sys, urllib.request, urllib.error
+sql, min_ts = sys.argv[1], sys.argv[2]
+key = open(os.path.expanduser("~/.logfire-read-key")).read().strip()
+# region comes from the key prefix: pylf_v2_<region>_...
+parts = key.split("_")
+region = parts[2] if len(parts) > 3 and parts[0] == "pylf" else "eu"
+req = urllib.request.Request(
+    f"https://logfire-{region}.pydantic.dev/v2/query",
+    data=json.dumps({"sql": sql, "min_timestamp": min_ts}).encode(),
+    headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"},
+)
+try:
+    print(json.dumps(json.load(urllib.request.urlopen(req, timeout=120)), indent=2)[:6000])
+except urllib.error.HTTPError as e:
+    print(f"HTTP {e.code}: {e.read().decode()[:600]}")
+PY

.claude/skills/debug-ingestion/SKILL.md

@@ -0,0 +1,203 @@
+---
+name: debug-ingestion
+description: Debug haiku.rag ingestion in Logfire. Use when asked to look at Logfire for ingestion, find failed or dead ingestion jobs, investigate retries or circuit-breaker events, trace a document through convert/chunk/embed/store, find which docling-serve instance served a request, spot slow conversions, or tell concurrent ingesters apart. Drives the Logfire MCP against the `haiku-ingester` service.
+---
+
+# Debug ingestion in Logfire
+
+The ingester ships spans to Logfire under `service_name = 'haiku-ingester'` (or a
+custom `OTEL_SERVICE_NAME` if set per process). This skill finds failing jobs,
+traces a document through the pipeline, and pinpoints the docling-serve instance
+that served a request. Read-only.
+
+## How to query
+
+1. Confirm the current schema with `mcp__logfire__query_schema_reference` (spans
+   and logs share the `records` table).
+2. Run SQL with `mcp__logfire__query_run` (`query` + `project: "haiku"` +
+   `start_timestamp`/`end_timestamp`, max 14 days). The remote MCP is
+   org-scoped, so `project` is required; the ingester ships to project `haiku`.
+   The same SQL works pasted into Logfire's Explore UI.
+3. Read span attributes as JSON: `attributes->>'key'`, cast when needed
+   (`(attributes->>'attempt')::int`).
+4. Hand back a clickable trace with
+   `mcp__logfire__project_logfire_link(trace_id, project="haiku")`.
+5. For recent exceptions tied to a file,
+   `mcp__logfire__query_find_exceptions_in_file` accepts
+   `client/documents.py`, `ingester/workers/pool.py`, or
+   `ingester/pollers/base.py`.
+
+Adjust `service_name` if the operator set `OTEL_SERVICE_NAME` (e.g. per tenant).
+Interactive `haiku-rag` ingests emit the same `document.*` spans under the CLI's
+service name (or `unknown_service` for older runs), not `haiku-ingester`.
+
+## Vocabulary
+
+Span tree (all scope `haiku.rag`), each level nests under the one above and
+shares its `trace_id`:
+
+- Poller: `ingester.poller.sweep` | `ingester.poller.dry_run` |
+  `ingester.poller.watch_event`.
+  - `attributes->>'source_id'`; watch adds `change`, `uri`.
+  - sweep sets `skipped`, `skip_reason` (`pending_work` / `circuit_open`),
+    `upsert`, `delete`, `unchanged`, `consecutive_failures`. A failed sweep
+    records the exception on the span (`exception_type` / `exception_message`).
+- Job: `ingester.job` — `attributes->>'source_id'`, `->>'uri'`, `->>'op'`
+  (`UPSERT`/`DELETE`), `(->>'attempt')::int`. `is_exception=true` marks a job
+  that raised.
+- Document pipeline: `document.fetch` (`bytes`, `content_hash`),
+  `document.convert`, `document.chunk` (`chunks_created`), `document.embed`,
+  `document.store` (`op` `create`/`update`, `document_id`).
+- docling-serve: `docling_serve.request` — `attributes->>'name'` (operation),
+  `->>'url'` (instance), `(->>'attempt')::int`. A retry emits a new span with a
+  different `url`, so failover shows as sibling spans.
+
+Failures in Logfire are span-level: sweep and job exceptions sit on the span
+(`is_exception`, `exception_type`, `exception_message`, `level >= 17`). A job that
+raised `PermanentError` was dead-lettered on that attempt; `attempt >= 2` on an
+`ingester.job` span is a retry. The worker circuit breaker opening emits a
+dedicated event `span_name = 'ingester.worker breaker opened'` (attributes
+`source_id`, `threshold`, `cooldown_s`), fired once per closed->open transition.
+The ingester's per-job dead/reschedule narration stays on stderr and the queue
+(`GET /jobs?status=dead` on the control-plane API), not Logfire.
+
+## Canned queries
+
+Job outcomes by source:
+
+```sql
+SELECT attributes->>'source_id' AS source_id,
+       attributes->>'op' AS op,
+       is_exception,
+       count(*) AS n
+FROM records
+WHERE service_name='haiku-ingester' AND span_name='ingester.job'
+GROUP BY 1,2,3
+ORDER BY n DESC;
+```
+
+Failed jobs with the exception and trace to drill in:
+
+```sql
+SELECT attributes->>'uri' AS uri,
+       (attributes->>'attempt')::int AS attempt,
+       exception_type, exception_message, trace_id
+FROM records
+WHERE service_name='haiku-ingester' AND span_name='ingester.job'
+  AND is_exception=true
+ORDER BY start_timestamp DESC
+LIMIT 50;
+```
+
+Retried and dead-lettered jobs (a `PermanentError` was dead-lettered on that
+attempt; `TransientError` at a high `attempt` was retried):
+
+```sql
+SELECT attributes->>'uri' AS uri,
+       (attributes->>'attempt')::int AS attempt,
+       attributes->>'op' AS op,
+       exception_type, trace_id
+FROM records
+WHERE service_name='haiku-ingester' AND span_name='ingester.job'
+  AND (is_exception=true OR (attributes->>'attempt')::int > 1)
+ORDER BY start_timestamp DESC
+LIMIT 50;
+```
+
+Trace one document end-to-end (take `trace_id` from a job above):
+
+```sql
+SELECT span_name, duration, is_exception,
+       attributes->>'url' AS docling_url,
+       attributes->>'op' AS op
+FROM records
+WHERE trace_id='<TRACE_ID>'
+ORDER BY start_timestamp;
+```
+
+Worker circuit-breaker trips (a source paused after consecutive transient
+failures):
+
+```sql
+SELECT start_timestamp,
+       attributes->>'source_id' AS source_id,
+       (attributes->>'cooldown_s')::float AS cooldown_s
+FROM records
+WHERE service_name='haiku-ingester'
+  AND span_name='ingester.worker breaker opened'
+ORDER BY start_timestamp DESC
+LIMIT 50;
+```
+
+docling-serve instance health and failover:
+
+```sql
+SELECT attributes->>'url' AS instance,
+       (attributes->>'attempt')::int AS attempt,
+       is_exception,
+       count(*) AS n
+FROM records
+WHERE service_name='haiku-ingester' AND span_name='docling_serve.request'
+GROUP BY 1,2,3
+ORDER BY n DESC;
+```
+
+Slowest pipeline stages:
+
+```sql
+SELECT span_name,
+       attributes->>'uri' AS uri,
+       duration, trace_id
+FROM records
+WHERE service_name='haiku-ingester'
+  AND span_name IN ('document.convert','docling_serve.request','document.embed','document.chunk')
+ORDER BY duration DESC
+LIMIT 20;
+```
+
+Per-source sweep summary:
+
+```sql
+SELECT attributes->>'source_id' AS source_id,
+       sum((attributes->>'upsert')::int) AS upserts,
+       sum((attributes->>'delete')::int) AS deletes,
+       sum((attributes->>'unchanged')::int) AS unchanged,
+       max(attributes->>'skip_reason') AS last_skip_reason
+FROM records
+WHERE service_name='haiku-ingester' AND span_name='ingester.poller.sweep'
+GROUP BY 1;
+```
+
+Tell concurrent ingesters apart (set distinct `OTEL_SERVICE_NAME` per process; a
+single host still separates by `process_pid` / `service_instance_id`):
+
+```sql
+SELECT service_name, service_instance_id, process_pid, count(*) AS n
+FROM records
+WHERE span_name='ingester.job'
+GROUP BY 1,2,3
+ORDER BY n DESC;
+```
+
+## Workflow
+
+1. Job outcomes by source shows where failures cluster.
+2. Failed jobs surfaces the exception and each failing `trace_id`.
+3. Trace one document end-to-end to see which stage failed and, for a docling
+   source, which `docling_url` served it (and whether it failed over).
+4. Retried/dead-lettered jobs show what the queue kept struggling with; a
+   `ingester.worker breaker opened` event flags a source the pool paused. The
+   per-job dead/reschedule narration stays in the ingester console and the queue
+   API, not Logfire.
+5. `project_logfire_link(trace_id)` for a document lets the user expand the
+   full tree.
+
+## When a query returns nothing
+
+Span names or attributes may have changed. Probe:
+
+```sql
+SELECT DISTINCT span_name
+FROM records WHERE service_name='haiku-ingester'
+ORDER BY 1;
+```

.github/workflows/build-docs.yml

@@ -3,26 +3,37 @@ on:
   push:
     branches:
       - main
+  pull_request:
+    branches:
+      - main
 permissions:
-  contents: write
+  contents: read
+  pages: write
+  id-token: write
+concurrency:
+  group: pages-${{ github.ref }}
+  cancel-in-progress: false
 jobs:
-  deploy:
+  build:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - name: Configure Git Credentials
-        run: |
-          git config user.name github-actions[bot]
-          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
-      - uses: actions/setup-python@v5
+      - uses: astral-sh/setup-uv@v9.0.0
+      - run: uv sync --group dev
+      - run: uv run zensical build
+      - uses: actions/configure-pages@v5
+        if: github.event_name == 'push'
+      - uses: actions/upload-pages-artifact@v3
+        if: github.event_name == 'push'
         with:
-          python-version: 3.x
-      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
-      - uses: actions/cache@v4
-        with:
-          key: mkdocs-material-${{ env.cache_id }}
-          path: .cache
-          restore-keys: |
-            mkdocs-material-
-      - run: pip install mkdocs-material
-      - run: mkdocs gh-deploy --force
+          path: ./site
+  deploy:
+    needs: build
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

.github/workflows/build-publish-slim.yml

@@ -9,7 +9,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: astral-sh/setup-uv@v4
+      - uses: astral-sh/setup-uv@v9.0.0
         with:
           enable-cache: true
       - name: Set up Python

.github/workflows/build-publish.yml

@@ -12,7 +12,7 @@ jobs:
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
     steps:
       - uses: actions/checkout@v4
-      - uses: astral-sh/setup-uv@v4
+      - uses: astral-sh/setup-uv@v9.0.0
         with:
           enable-cache: true
       - name: Set up Python

.github/workflows/docker-publish-slim.yml

@@ -1,7 +1,7 @@
-name: Build & publish Docker image
+name: Build & publish Docker slim image
 on:
   workflow_run:
-    workflows: ["Build & publish haiku.rag to pypi"]
+    workflows: ["Build & publish haiku.rag-slim to pypi"]
     types:
       - completed
   workflow_dispatch:
@@ -43,15 +43,13 @@ jobs:
           VERSION=$(grep -oP '^version = "\K[^"]+' haiku_rag_slim/pyproject.toml)
           echo "version=$VERSION" >> $GITHUB_OUTPUT
 
-      - name: Build and push Docker image
+      - name: Build and push Docker slim image
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: docker/Dockerfile
+          file: docker/Dockerfile.slim
           platforms: linux/amd64,linux/arm64
           push: true
           tags: |
-            ghcr.io/${{ github.repository }}:${{ steps.version.outputs.version }}
-            ghcr.io/${{ github.repository }}:latest
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+            ghcr.io/ggozad/haiku.rag-slim:${{ steps.version.outputs.version }}
+            ghcr.io/ggozad/haiku.rag-slim:latest

.github/workflows/publish-mcp.yml

@@ -14,9 +14,9 @@ jobs:
       contents: read
       id-token: write
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: '3.12'
       - name: Generate server.json from template
@@ -26,9 +26,11 @@ jobs:
           mv server.json.tmp server.json
           echo "Generated server.json with version: $VERSION"
       - name: Install MCP Publisher
+        env:
+          GH_TOKEN: ${{ github.token }}
         run: |
-          LATEST_RELEASE=$(curl -s https://api.github.com/repos/modelcontextprotocol/registry/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
-          curl -L https://github.com/modelcontextprotocol/registry/releases/download/${LATEST_RELEASE}/mcp-publisher_linux_amd64.tar.gz -o mcp-publisher.tar.gz
+          gh release download --repo modelcontextprotocol/registry \
+            --pattern 'mcp-publisher_linux_amd64.tar.gz' -O mcp-publisher.tar.gz
           tar -xzf mcp-publisher.tar.gz
           chmod +x mcp-publisher
           sudo mv mcp-publisher /usr/local/bin/

.github/workflows/test.yml

@@ -0,0 +1,105 @@
+name: Tests
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v9.0.0
+        with:
+          enable-cache: true
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: ".python-version"
+      - name: Install dependencies
+        run: uv sync --all-extras
+      - name: Lint
+        run: uv run ruff check
+      - name: Type check
+        run: uv run ty check
+
+  lint-frontend:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: pnpm/action-setup@v4
+        with:
+          version: 10
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+          cache: "pnpm"
+          cache-dependency-path: app/frontend/pnpm-lock.yaml
+      - name: Install dependencies
+        working-directory: app/frontend
+        run: pnpm install --frozen-lockfile
+      - name: Lint and format check
+        working-directory: app/frontend
+        run: pnpm run check
+
+  test-uri-platforms:
+    name: URI paths (${{ matrix.os }}, py${{ matrix.python }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python: ["3.13", "3.14"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v9.0.0
+      # --noconftest: tests/conftest.py imports the project's dependencies,
+      # which this job deliberately does not install. test_uri.py is stdlib-only.
+      - name: Test platform URI paths
+        env:
+          PYTHONPATH: haiku_rag_slim
+        run: >
+          uv run --no-project --python ${{ matrix.python }} --with pytest
+          pytest tests/test_uri.py -q --noconftest -o addopts=
+
+  test:
+    needs: [lint, lint-frontend]
+    runs-on: ubuntu-latest
+    env:
+      HF_TOKEN: ${{ secrets.HF_TOKEN }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v9.0.0
+        with:
+          enable-cache: true
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: ".python-version"
+      - name: Install dependencies
+        run: uv sync --all-extras
+      - name: Cache HuggingFace models
+        id: hf-cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: huggingface-${{ runner.os }}-test-models-v2
+      - name: Pre-download tokenizer
+        if: steps.hf-cache.outputs.cache-hit != 'true'
+        run: uv run python -c "from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('Qwen/Qwen3-Embedding-0.6B')"
+      - name: Pre-download cross-encoder test model
+        if: steps.hf-cache.outputs.cache-hit != 'true'
+        run: uv run python -c "from sentence_transformers import CrossEncoder; CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')"
+      - name: Run tests with coverage
+        env:
+          HF_HUB_OFFLINE: ${{ steps.hf-cache.outputs.cache-hit == 'true' && '1' || '0' }}
+          TRANSFORMERS_OFFLINE: ${{ steps.hf-cache.outputs.cache-hit == 'true' && '1' || '0' }}
+        run: uv run pytest -m "not integration" --cov --cov-report=xml --cov-report=term-missing:skip-covered
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          files: ./coverage.xml
+          fail_ci_if_error: false

.gitignore

@@ -5,6 +5,7 @@ build/
 dist/
 wheels/
 *.egg-info
+**/.DS_Store
 
 # Virtual environments
 .venv
@@ -12,6 +13,8 @@ wheels/
 # tests
 .coverage*
 evaluations/evaluations/data/
+evaluations/scripts/*
+!evaluations/scripts/build_t2_submission.py
 tests/data/
 .pytest_cache/
 .ruff_cache/
@@ -27,5 +30,14 @@ DEVNOTES.md
 .mcpregistry_github_token
 .mcpregistry_registry_token
 
-# MkDocs site directory when doing local docs builds
+# zensical site directory when doing local docs builds
 site/
+
+# Claude Code personal notes
+.claude.local.md
+
+# Publish shared Claude skills, keep local Claude settings private
+!.claude/
+.claude/*
+!.claude/skills/
+!.claude/skills/**

.pre-commit-config.yaml

@@ -1,22 +1,36 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v5.0.0
+    rev: v6.0.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-merge-conflict
       - id: check-toml
       - id: debug-statements
-  - repo: https://github.com/astral-sh/ruff-pre-commit
-    # Ruff version.
-    rev: v0.14.3
+  - repo: local
     hooks:
-      # Run the linter.
       - id: ruff
-      # Run the formatter.
+        name: ruff check
+        entry: uv run ruff check --force-exclude
+        language: system
+        types: [python]
       - id: ruff-format
+        name: ruff format
+        entry: uv run ruff format --force-exclude --check
+        language: system
+        types: [python]
+      - id: ty
+        name: ty check
+        entry: uv run ty check
+        language: system
+        types: [python]
+        pass_filenames: false
 
-  - repo: https://github.com/RobertCraigie/pyright-python
-    rev: v1.1.407
+  - repo: local
     hooks:
-      - id: pyright
+      - id: biome
+        name: biome check
+        entry: bash -c 'cd app/frontend && npm run check'
+        language: system
+        files: ^app/frontend/
+        types_or: [javascript, jsx, ts, tsx, json]

README.md

@@ -1,22 +1,37 @@
-# Haiku RAG
+# haiku.rag
 
-Retrieval-Augmented Generation (RAG) library built on LanceDB.
+[![PyPI](https://img.shields.io/pypi/v/haiku.rag)](https://pypi.org/project/haiku.rag/)
+[![Python](https://img.shields.io/pypi/pyversions/haiku.rag)](https://pypi.org/project/haiku.rag/)
+[![Downloads](https://static.pepy.tech/badge/haiku-rag-slim/month)](https://pepy.tech/projects/haiku-rag-slim)
+[![Docs](https://img.shields.io/badge/docs-ggozad.github.io-blue)](https://ggozad.github.io/haiku.rag/)
+[![Tests](https://github.com/ggozad/haiku.rag/actions/workflows/test.yml/badge.svg)](https://github.com/ggozad/haiku.rag/actions/workflows/test.yml)
+[![codecov](https://codecov.io/gh/ggozad/haiku.rag/graph/badge.svg)](https://codecov.io/gh/ggozad/haiku.rag)
 
-`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
+Agentic RAG that answers questions about your own documents with citations to page numbers and section headings. Runs locally on an embedded database, no server required.
+
+Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/). Full documentation at [ggozad.github.io/haiku.rag](https://ggozad.github.io/haiku.rag/).
 
 ## Features
 
-- **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
-- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI, vLLM
-- **Multiple QA providers**: Any provider/model supported by Pydantic AI
-- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI
-- **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking
-- **Reranking**: Default search result reranking with MixedBread AI, Cohere, Zero Entropy, or vLLM
-- **Question answering**: Built-in QA agents on your documents
-- **File monitoring**: Auto-index files when run as server
-- **40+ file formats**: PDF, DOCX, HTML, Markdown, code files, URLs
-- **MCP server**: Expose as tools for AI assistants
-- **CLI & Python API**: Use from command line or Python
+- **Hybrid search** — Vector + full-text with Reciprocal Rank Fusion
+- **Multimodal & cross-modal search** — Multimodal embedders (vLLM, VoyageAI, Cohere) put picture vectors in the same space as text; supports text-as-query → figure hits and image-as-query
+- **Question answering** — RAG capability with citations (page numbers, section headings)
+- **Vision QA** — Vision-capable models receive figure bytes alongside chunk text; attach your own images to questions in `ask`, `analyze` and the chat TUI
+- **Reranking** — local cross-encoders, Cohere, Zero Entropy, or vLLM
+- **Analysis capability** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis)
+- **Evidence compaction** — Optional capability that replaces earlier questions' search results on the request with the evidence they cited, so long conversations stop resending everything they retrieved
+- **Citation policy** — Optional capability that requires every answer to declare what grounds it, including declaring that nothing does
+- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
+- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
+- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, Cohere, LM Studio, vLLM (multimodal via `multimodal: true` on vLLM/VoyageAI/Cohere). QA: any model supported by Pydantic AI
+- **Multi-database search** — Search, ask, analyze, or chat across named databases with source attribution on results and citations
+- **Local-first** — Embedded LanceDB, no servers required. Also supports S3, GCS, Azure, and LanceDB Cloud
+- **CLI & Python API** — Full functionality from command line or code
+- **MCP server** — Expose as tools for AI assistants (Claude Desktop, etc.)
+- **Visual grounding** — View chunks highlighted on original page images
+- **Production ingester** — Long-lived `haiku-ingester` service with persistent SQLite queue, async worker pool with retries and a dead-letter queue, FS / HTTP / S3 / WebDAV source adapters, FastAPI control plane, and a browser dashboard for operators. See [docs/ingester.md](docs/ingester.md).
+- **Tags** — Name database states with `haiku-rag tag` and roll back to them
+- **Inspector** — TUI for browsing documents, chunks, and search results
 
 ## Installation
 
@@ -25,146 +40,139 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB.
 ### Full Package (Recommended)
 
 ```bash
-uv pip install haiku.rag
+pip install haiku.rag
 ```
 
 Includes all features: document processing, all embedding providers, and rerankers.
 
+Using [uv](https://docs.astral.sh/uv/)? `uv pip install haiku.rag`
+
 ### Slim Package (Minimal Dependencies)
 
 ```bash
-uv pip install haiku.rag-slim
+pip install haiku.rag-slim
 ```
 
-Install only the extras you need. See the [Installation](https://ggozad.github.io/haiku.rag/installation/) documentation for available options
+Install only the extras you need. See the [Installation](https://ggozad.github.io/haiku.rag/installation/) documentation for available options.
 
 ## Quick Start
 
+> **Note**: Requires an embedding provider (Ollama, OpenAI, etc.). See the [Tutorial](https://ggozad.github.io/haiku.rag/tutorial/) for setup instructions.
+
 ```bash
-# Add documents
-haiku-rag add "Your content here"
-haiku-rag add "Your content here" --meta author=alice --meta topic=notes
-haiku-rag add-src document.pdf --meta source=manual
+# Index a PDF
+haiku-rag add-src paper.pdf
 
 # Search
-haiku-rag search "query"
-
-# Search with filters
-haiku-rag search "query" --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
-
-# Ask questions
-haiku-rag ask "Who is the author of haiku.rag?"
+haiku-rag search "attention mechanism"
 
 # Ask questions with citations
-haiku-rag ask "Who is the author of haiku.rag?" --cite
+haiku-rag ask "What datasets were used for evaluation?"
 
-# Deep QA (multi-agent question decomposition)
-haiku-rag ask "Who is the author of haiku.rag?" --deep --cite
+# Ask about an image (vision-capable model)
+haiku-rag ask "Does this figure match the spec in the design doc?" --image figure.png
 
-# Deep QA with verbose output
-haiku-rag ask "Who is the author of haiku.rag?" --deep --verbose
+# Analyze — complex analytical tasks via code execution
+haiku-rag analyze "How many documents mention transformers?"
 
-# Multi‑agent research (iterative plan/search/evaluate)
-haiku-rag research \
-  "What are the main drivers and trends of global temperature anomalies since 1990?" \
-  --max-iterations 2 \
-  --confidence-threshold 0.8 \
-  --max-concurrency 3 \
-  --verbose
+# Interactive chat — multi-turn conversations with memory
+haiku-rag chat
 
-# Rebuild database (re-chunk and re-embed all documents)
-haiku-rag rebuild
-
-# Start server with file monitoring
-haiku-rag serve --monitor
+# Continuously ingest from configured sources (FS, HTTP, S3, WebDAV)
+haiku-ingester serve
 ```
 
-To customize settings, create a `haiku.rag.yaml` config file (see [Configuration](https://ggozad.github.io/haiku.rag/configuration/)).
+See [Configuration](https://ggozad.github.io/haiku.rag/configuration/) for customization options.
 
-## Python Usage
+## Python API
 
 ```python
 from haiku.rag.client import HaikuRAG
-from haiku.rag.config import Config
-from haiku.rag.research import (
-    ResearchContext,
-    ResearchDeps,
-    ResearchState,
-    build_research_graph,
-    stream_research_graph,
-)
 
-async with HaikuRAG("database.lancedb") as client:
-    # Add document
-    doc = await client.create_document("Your content")
+async with HaikuRAG("knowledge.lancedb", create=True) as rag:
+    # Index documents
+    await rag.create_document_from_source("paper.pdf")
+    await rag.create_document_from_source("https://arxiv.org/pdf/1706.03762")
 
-    # Search (reranking enabled by default)
-    results = await client.search("query")
-    for chunk, score in results:
-        print(f"{score:.3f}: {chunk.content}")
+    # Search — returns chunks with provenance
+    results = await rag.search("self-attention")
+    for result in results:
+        print(f"{result.score:.2f} | p.{result.page_numbers} | {result.content[:100]}")
 
-    # Ask questions
-    answer = await client.ask("Who is the author of haiku.rag?")
+    # QA with citations
+    answer, citations = await rag.ask("What is the complexity of self-attention?")
     print(answer)
-
-    # Ask questions with citations
-    answer = await client.ask("Who is the author of haiku.rag?", cite=True)
-    print(answer)
-
-    # Multi‑agent research pipeline (Plan → Search → Evaluate → Synthesize)
-    # Graph settings (provider, model, max_iterations, etc.) come from config
-    graph = build_research_graph(config=Config)
-    question = (
-        "What are the main drivers and trends of global temperature "
-        "anomalies since 1990?"
-    )
-    context = ResearchContext(original_question=question)
-    state = ResearchState.from_config(context=context, config=Config)
-    deps = ResearchDeps(client=client)
-
-    # Blocking run (final result only)
-    report = await graph.run(state=state, deps=deps)
-    print(report.title)
-
-    # Streaming progress (log/report/error events)
-    async for event in stream_research_graph(graph, state, deps):
-        if event.type == "log":
-            iteration = event.state.iterations if event.state else state.iterations
-            print(f"[{iteration}] {event.message}")
-        elif event.type == "report":
-            print("\nResearch complete!\n")
-            print(event.report.title)
-            print(event.report.executive_summary)
+    for cite in citations:
+        print(f"  [{cite.chunk_id}] p.{cite.page_numbers}: {cite.content[:80]}")
 ```
 
+For direct agent composition, see the [capabilities documentation](https://ggozad.github.io/haiku.rag/capabilities/).
+
 ## MCP Server
 
-Use with AI assistants like Claude Desktop:
+Use with AI assistants like Claude Code, Codex, and Claude Desktop:
 
 ```bash
-haiku-rag serve --stdio
+haiku-rag mcp --stdio
 ```
 
-Provides tools for document management and search directly in your AI assistant.
+In Claude Code, install the plugin, which registers the server and a skill:
+
+```bash
+claude plugin marketplace add ggozad/haiku.rag
+claude plugin install haiku-rag
+```
+
+In Codex, install the same plugin from its marketplace:
+
+```bash
+codex plugin marketplace add ggozad/haiku.rag
+codex plugin add haiku-rag@haiku-rag
+```
+
+Add to your Claude Desktop configuration:
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["mcp", "--stdio"]
+    }
+  }
+}
+```
+
+Provides search, document reading, and analysis tools directly in your AI assistant.
 
 ## Examples
 
 See the [examples directory](examples/) for working examples:
 
-- **[Interactive Research Assistant](examples/ag-ui-research/)** - Full-stack research assistant with Pydantic AI and AG-UI featuring human-in-the-loop approval and real-time state synchronization
-- **[Docker Setup](examples/docker/)** - Complete Docker deployment with file monitoring and MCP server
-- **[A2A Server](examples/a2a-server/)** - Self-contained A2A protocol server package with conversational agent interface
+- **[Docker Setup](examples/docker/)** - Complete Docker deployment with continuous ingestion (`haiku-ingester`) and MCP server
+- **[Web Application](app/)** - Full-stack conversational RAG with CopilotKit frontend
 
 ## Documentation
 
 Full documentation at: https://ggozad.github.io/haiku.rag/
 
-- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
-- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML configuration
+- [Quickstart](https://ggozad.github.io/haiku.rag/tutorial/) - Provider setup and first ingestion
+- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Packages and extras
+- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML reference
 - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
 - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
-- [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research
-- [MCP Server](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
-- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance Benchmarks
+- [Capabilities](https://ggozad.github.io/haiku.rag/capabilities/) - Native Pydantic AI RAG and analysis capabilities
+- [Tuning](https://ggozad.github.io/haiku.rag/tuning/) - Retrieval and answer-quality tuning
+- [Ingester](https://ggozad.github.io/haiku.rag/ingester/) - Production ingester for continuous indexing from FS, HTTP, S3, and WebDAV
+- [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
+- [Remote processing](https://ggozad.github.io/haiku.rag/remote-processing/) - Offload conversion to docling-serve
+- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector
+- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks
+- [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history
 
+## License
+
+This project is licensed under the [MIT License](LICENSE).
+
+<!-- mcp-name is used by the MCP registry to identify this server -->
 mcp-name: io.github.ggozad/haiku-rag

app/.env.example

@@ -0,0 +1,11 @@
+# API Keys (at least one required for LLM)
+ANTHROPIC_API_KEY=your-anthropic-key
+OPENAI_API_KEY=your-openai-key
+
+# Host path of the LanceDB database, mounted at /data where haiku.rag.yaml
+# places it
+DB_VOLUME=./data/haiku.rag.lancedb
+
+# Optional: Ollama base URL (if using local models)
+# Use host.docker.internal to reach Ollama running on the host machine
+OLLAMA_BASE_URL=http://host.docker.internal:11434

app/README.md

@@ -0,0 +1,110 @@
+# haiku.rag Chat App
+
+A conversational RAG interface built with [CopilotKit](https://copilotkit.ai/) and [pydantic-ai](https://github.com/pydantic/pydantic-ai)'s AG-UI protocol.
+
+> **Note:** An illustrative example meant as a starting point, with no authentication. The compose files bind the backend to `127.0.0.1`; don't expose it to an untrusted network.
+
+## Prerequisites
+
+- Docker and Docker Compose
+- A haiku.rag database (created via the `haiku-rag` CLI)
+- An LLM API key (Anthropic, OpenAI, or local Ollama)
+
+## Quick Start
+
+1. **Set up environment variables:**
+
+   ```bash
+   cp .env.example .env
+   # Edit .env with your API keys and database path
+   ```
+
+2. **Configure the LLM and embedding models:**
+
+   ```bash
+   cp haiku.rag.yaml.example haiku.rag.yaml
+   # Edit haiku.rag.yaml to configure your models
+   ```
+
+3. **Start the app:**
+
+   ```bash
+   docker compose up -d
+   ```
+
+4. **Open the chat interface:** http://localhost:3000
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `DB_VOLUME` | Host path of the LanceDB database the compose files mount at `/data`, where `haiku.rag.yaml` places it (default `./data/haiku.rag.lancedb`) | No |
+| `HAIKU_RAG_CONFIG_PATH` | The configuration file; the compose files set it to the mounted `/app/haiku.rag.yaml` | No |
+| `ANTHROPIC_API_KEY` | Anthropic API key | One LLM key required |
+| `OPENAI_API_KEY` | OpenAI API key | One LLM key required |
+| `OLLAMA_BASE_URL` | Ollama server URL (default: `http://host.docker.internal:11434`) | For local models |
+| `LOGFIRE_TOKEN` | Pydantic Logfire token for debugging | No |
+
+### haiku.rag.yaml
+
+Configure the LLM, embeddings, and search settings:
+
+```yaml
+qa:
+  model:
+    provider: anthropic  # or openai, ollama
+    name: claude-sonnet-4-20250514
+
+embeddings:
+  model:
+    provider: ollama
+    name: nomic-embed-text
+
+search:
+  limit: 10
+```
+
+See `haiku.rag.yaml.example` for all options.
+
+## Development
+
+For local development with hot reloading:
+
+```bash
+docker compose -f docker-compose.dev.yml up -d --build
+```
+
+- Backend code changes reload automatically
+- Frontend available at http://localhost:3000
+- Backend API at http://localhost:8001
+
+## Architecture
+
+```
+┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
+│    Frontend     │────▶│     Backend     │────▶│   haiku.rag     │
+│  (CopilotKit)   │     │  (pydantic-ai)  │     │   (LanceDB)     │
+│  localhost:3000 │     │  localhost:8001 │     │                 │
+└─────────────────┘     └─────────────────┘     └─────────────────┘
+```
+
+### Backend Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/v1/chat/stream` | POST | AG-UI chat streaming |
+| `/api/documents` | GET | List documents in database |
+| `/api/info` | GET | Database statistics |
+| `/api/visualize/{chunk_id}` | GET | Visual grounding for chunks |
+| `/health` | GET | Health check |
+
+## Chat Capabilities
+
+The chat can:
+
+- **Search** your documents with hybrid vector + full-text search
+- **Answer questions** with citations from your knowledge base
+- **Filter by document** when you ask about specific files
+- **Show visual grounding** for PDF/image sources

app/backend/Dockerfile

@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS builder
+
+WORKDIR /app
+
+ENV UV_COMPILE_BYTECODE=1 \
+    UV_LINK_MODE=copy
+
+# Install haiku.rag-slim from workspace
+COPY pyproject.toml uv.lock ./
+COPY haiku_rag_slim/pyproject.toml haiku_rag_slim/README.md haiku_rag_slim/LICENSE haiku_rag_slim/
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-install-project --no-dev --package haiku.rag-slim
+
+COPY haiku_rag_slim haiku_rag_slim/
+COPY README.md LICENSE ./
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-editable --no-dev --package haiku.rag-slim
+
+# Install app backend dependencies
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv pip install starlette uvicorn[standard] anthropic watchfiles
+
+# Final layer
+FROM python:3.13-slim
+WORKDIR /app
+
+COPY --from=builder /app/.venv /app/.venv
+COPY app/backend/*.py ./
+
+RUN mkdir -p /data
+ENV PATH="/app/.venv/bin:$PATH"
+
+EXPOSE 8000
+
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

app/backend/main.py

@@ -0,0 +1,296 @@
+import asyncio
+import logging
+from contextlib import asynccontextmanager
+from dataclasses import dataclass, field
+from typing import Any
+
+from ag_ui.core import EventType, StateSnapshotEvent
+from dotenv import find_dotenv, load_dotenv
+from pydantic_ai import Agent
+from pydantic_ai.ui import SSE_CONTENT_TYPE
+from pydantic_ai.ui.ag_ui import AGUIAdapter
+from starlette.applications import Starlette
+from starlette.middleware import Middleware
+from starlette.middleware.cors import CORSMiddleware
+from starlette.requests import Request
+from starlette.responses import JSONResponse, Response, StreamingResponse
+from starlette.routing import Route
+
+from haiku.rag.capabilities.compaction import (
+    create_capability as create_compaction,
+)
+from haiku.rag.capabilities.policy import (
+    create_capability as create_citation_policy,
+)
+from haiku.rag.capabilities.rag import AGENT_PREAMBLE, RAGState, create_capability
+from haiku.rag.client import HaikuRAG
+from haiku.rag.client.scope import DatabaseScope
+from haiku.rag.config import get_config
+from haiku.rag.telemetry import configure as configure_telemetry
+from haiku.rag.utils import get_model
+
+load_dotenv(find_dotenv(usecwd=True))
+
+configure_telemetry(service_name="haiku-rag-app")
+
+logging.basicConfig(
+    level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+)
+logger = logging.getLogger(__name__)
+
+# The configuration places the database. This app serves one.
+config = get_config()
+scope = DatabaseScope.resolve(config)
+if scope.covers_multiple:
+    raise SystemExit(
+        f"lancedb.databases names {', '.join(scope.names)}; this app serves one "
+        "database: configure exactly one entry"
+    )
+[database] = scope.databases
+
+
+def _database_exists() -> bool:
+    """A database behind a URI has no path to check."""
+    return database.db_path is None or database.db_path.exists()
+
+
+logger.info(f"Database: {database.name} at {database.location}")
+logger.info(f"QA Provider: {config.qa.model.provider}, Model: {config.qa.model.name}")
+
+# Only HaikuRAG client is a singleton (expensive to create)
+_client: HaikuRAG | None = None
+_client_lock = asyncio.Lock()
+
+
+async def get_client() -> HaikuRAG:
+    """Get or create the cached client.
+
+    Guarded by a lock because the first request after startup can race with
+    itself: two concurrent callers would both pass the None check, each build
+    and enter a HaikuRAG, and the loser would leak its LanceDB connection.
+    """
+    global _client
+    if _client is None:
+        async with _client_lock:
+            if _client is None:
+                client = HaikuRAG(config=config, create=True)
+                await client.__aenter__()
+                _client = client
+    return _client
+
+
+@dataclass
+class AppDeps:
+    state: dict[str, Any] = field(default_factory=dict)
+
+
+capability = create_capability(config=config, defer_loading=False)
+
+agent = Agent(
+    get_model(config.qa.model, config),
+    instructions=AGENT_PREAMBLE,
+    # Conversations here are multi-turn, so earlier questions are reduced to the
+    # evidence they cited rather than carried whole, and every answer declares
+    # what grounds it so the UI can show citations for all of them.
+    capabilities=[capability, create_compaction(), create_citation_policy()],
+    deps_type=AppDeps,
+)
+
+
+async def stream_chat(request: Request) -> Response:
+    """Chat streaming endpoint with AG-UI protocol."""
+    body = await request.body()
+    accept = request.headers.get("accept", SSE_CONTENT_TYPE)
+    run_input = AGUIAdapter.build_run_input(body)
+
+    adapter = AGUIAdapter(agent=agent, run_input=run_input, accept=accept)
+
+    incoming_state = run_input.state if isinstance(run_input.state, dict) else {}
+    incoming_state.setdefault("rag", RAGState().model_dump(mode="json"))
+    deps = AppDeps(state=incoming_state)
+
+    async def event_stream():
+        async def with_final_state():
+            async for event in adapter.run_stream(deps=deps):
+                if getattr(event, "type", None) == EventType.RUN_FINISHED:
+                    yield StateSnapshotEvent(
+                        type=EventType.STATE_SNAPSHOT,
+                        snapshot=deps.state,
+                    )
+                yield event
+
+        async for chunk in adapter.encode_stream(with_final_state()):
+            yield chunk
+
+    return StreamingResponse(
+        event_stream(),
+        media_type=accept,
+        headers={
+            "Cache-Control": "no-cache",
+            "Connection": "keep-alive",
+            "X-Accel-Buffering": "no",
+        },
+    )
+
+
+async def health_check(_: Request) -> JSONResponse:
+    """Health check endpoint."""
+    return JSONResponse(
+        {
+            "status": "healthy",
+            "qa_provider": config.qa.model.provider,
+            "qa_model": config.qa.model.name,
+            "db_path": str(database.location),
+            "db_exists": _database_exists(),
+        }
+    )
+
+
+async def list_documents(_: Request) -> JSONResponse:
+    """List all documents in the database."""
+    if not _database_exists():
+        return JSONResponse({"documents": [], "error": "Database not found"})
+
+    client = await get_client()
+    docs = await client.document_repository.list_all()
+    return JSONResponse(
+        {
+            "documents": [
+                {"id": doc.id, "title": doc.title, "uri": doc.uri} for doc in docs
+            ]
+        }
+    )
+
+
+async def db_info(_: Request) -> JSONResponse:
+    """Get database info and statistics."""
+    if not _database_exists():
+        return JSONResponse(
+            {
+                "exists": False,
+                "path": str(database.location),
+                "documents": 0,
+                "chunks": 0,
+            }
+        )
+
+    from haiku.rag.store.info import get_database_stats
+
+    client = await get_client()
+    stats = await get_database_stats(client.store.db)
+
+    return JSONResponse(
+        {
+            "exists": True,
+            "path": str(database.location),
+            "documents": stats["documents"].get("num_rows", 0),
+            "chunks": stats["chunks"].get("num_rows", 0),
+            "documents_bytes": stats["documents"].get("total_bytes", 0),
+            "chunks_bytes": stats["chunks"].get("total_bytes", 0),
+            "has_vector_index": stats["chunks"].get("has_vector_index", False),
+        }
+    )
+
+
+async def visualize_chunk(request: Request) -> JSONResponse:
+    """Return visual grounding images for one or more chunks as base64.
+
+    The path param accepts comma-separated chunk ids (a merged citation's
+    constituent chunks). The optional ``refs`` query param is a JSON-encoded
+    list of the citation's ``doc_item_refs`` — the exact items the model saw —
+    so the highlight matches the cited content instead of re-expanding.
+    """
+    import base64
+    import json
+    from io import BytesIO
+
+    chunk_id = request.path_params["chunk_id"]
+
+    refs: list[str] | None = None
+    refs_param = request.query_params.get("refs")
+    if refs_param:
+        try:
+            parsed = json.loads(refs_param)
+        except ValueError:
+            parsed = None
+        if isinstance(parsed, list):
+            refs = [str(x) for x in parsed]
+
+    if not _database_exists():
+        return JSONResponse({"error": "Database not found"}, status_code=404)
+
+    client = await get_client()
+
+    chunks = []
+    for cid in chunk_id.split(","):
+        chunk = await client.chunk_repository.get_by_id(cid)
+        if chunk:
+            chunks.append(chunk)
+    if not chunks:
+        return JSONResponse({"error": "Chunk not found"}, status_code=404)
+
+    images = await client.visualize_chunk(chunks, refs)
+    if not images:
+        return JSONResponse({"images": [], "message": "No visual grounding available"})
+
+    base64_images = []
+    for img in images:
+        buffer = BytesIO()
+        img.save(buffer, format="PNG")
+        buffer.seek(0)
+        base64_images.append(base64.b64encode(buffer.read()).decode("utf-8"))
+
+    return JSONResponse(
+        {
+            "images": base64_images,
+            "chunk_id": chunk_id,
+            "document_uri": chunks[0].document_uri,
+        }
+    )
+
+
+@asynccontextmanager
+async def lifespan(_app: Starlette):
+    """Shut down the cached HaikuRAG client cleanly on app exit.
+
+    Awaits any in-flight background vacuum tasks and closes the LanceDB
+    connection. Without this, vacuum tasks are cancelled abruptly and the
+    connection is never closed on process shutdown.
+    """
+    yield
+    global _client
+    if _client is not None:
+        await _client.__aexit__(None, None, None)
+        _client = None
+
+
+# Create Starlette app
+app = Starlette(
+    routes=[
+        Route("/v1/chat/stream", stream_chat, methods=["POST"]),
+        Route("/api/documents", list_documents, methods=["GET"]),
+        Route("/api/info", db_info, methods=["GET"]),
+        Route("/api/visualize/{chunk_id}", visualize_chunk, methods=["GET"]),
+        Route("/health", health_check, methods=["GET"]),
+    ],
+    middleware=[
+        Middleware(
+            CORSMiddleware,  # type: ignore[invalid-argument-type]
+            allow_origins=["http://localhost:3000", "http://frontend:3000"],
+            allow_credentials=True,
+            allow_methods=["*"],
+            allow_headers=["*"],
+        )
+    ],
+    lifespan=lifespan,
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(
+        "main:app",
+        host="0.0.0.0",
+        port=8000,
+        reload=True,
+    )

app/backend/pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "haiku-rag-app"
+version = "0.1.0"
+description = "Conversational RAG application with haiku.rag"
+requires-python = ">=3.12"
+dependencies = [
+    "starlette>=0.50.0",
+    "uvicorn[standard]>=0.40.0",
+    "pydantic-ai-slim[ag-ui,anthropic,openai]>=2.18.0,<3.0.0",
+    "python-dotenv>=1.2.1",
+    "haiku.rag-slim>=0.82.1",
+    "logfire[pydantic-ai]>=3.17.0",
+]
+
+[dependency-groups]
+dev = ["ty>=0.0.28", "ruff>=0.14.10"]
+
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["."]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"

app/docker-compose.dev.yml

@@ -0,0 +1,46 @@
+# Local development with hot reloading
+# Usage: docker compose -f docker-compose.dev.yml up --build
+services:
+  backend:
+    build:
+      context: ..
+      dockerfile: app/backend/Dockerfile
+    command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
+    working_dir: /app/src
+    # No authentication; bound to loopback. The frontend reaches it over the compose network.
+    ports:
+      - "127.0.0.1:8001:8000"
+    environment:
+      - HAIKU_RAG_CONFIG_PATH=/app/haiku.rag.yaml
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - OLLAMA_BASE_URL=${OLLAMA_BASE_URL:-http://host.docker.internal:11434}
+      - LOGFIRE_TOKEN=${LOGFIRE_TOKEN:-}
+    volumes:
+      # haiku.rag.yaml places the database at /data.
+      - ${DB_VOLUME:-./data/haiku.rag.lancedb}:/data
+      - ./backend:/app/src:ro
+      - ./haiku.rag.yaml:/app/haiku.rag.yaml:ro
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  frontend:
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+      target: base
+    command: sh -c "pnpm install && pnpm dev"
+    working_dir: /app
+    ports:
+      - "3000:3000"
+    environment:
+      - BACKEND_URL=http://backend:8000
+      - HOSTNAME=0.0.0.0
+    volumes:
+      - ./frontend:/app
+      - frontend_node_modules:/app/node_modules
+    depends_on:
+      - backend
+
+volumes:
+  frontend_node_modules:

app/docker-compose.yml

@@ -0,0 +1,31 @@
+services:
+  backend:
+    build:
+      context: ..
+      dockerfile: app/backend/Dockerfile
+    # No authentication; bound to loopback. The frontend reaches it over the compose network.
+    ports:
+      - "127.0.0.1:8001:8000"
+    environment:
+      - HAIKU_RAG_CONFIG_PATH=/app/haiku.rag.yaml
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - OLLAMA_BASE_URL=${OLLAMA_BASE_URL:-http://host.docker.internal:11434}
+      - LOGFIRE_TOKEN=${LOGFIRE_TOKEN:-}
+    volumes:
+      # haiku.rag.yaml places the database at /data.
+      - ${DB_VOLUME:-./data/haiku.rag.lancedb}:/data
+      - ./haiku.rag.yaml:/app/haiku.rag.yaml:ro
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  frontend:
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+    ports:
+      - "3000:3000"
+    environment:
+      - BACKEND_URL=http://backend:8000
+    depends_on:
+      - backend

app/frontend/.dockerignore

@@ -0,0 +1,4 @@
+node_modules
+.next
+.git
+*.log

app/frontend/.gitignore

@@ -0,0 +1,30 @@
+# Dependencies
+node_modules/
+.pnpm-store/
+
+# Next.js build
+.next/
+out/
+
+# Production
+build/
+
+# Misc
+.DS_Store
+*.pem
+
+# Debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# Env files
+.env*.local
+
+# Vercel
+.vercel
+
+# TypeScript
+*.tsbuildinfo
+next-env.d.ts

app/frontend/Dockerfile

@@ -0,0 +1,35 @@
+FROM node:22-alpine AS base
+
+RUN corepack enable && corepack prepare pnpm@10.15.1 --activate
+
+FROM base AS deps
+WORKDIR /app
+COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
+RUN pnpm install --frozen-lockfile
+
+FROM base AS builder
+WORKDIR /app
+COPY --from=deps /app/node_modules ./node_modules
+COPY . .
+RUN pnpm build
+
+FROM base AS runner
+WORKDIR /app
+
+ENV NODE_ENV=production
+
+RUN addgroup --system --gid 1001 nodejs && \
+    adduser --system --uid 1001 nextjs
+
+COPY --from=builder /app/public ./public
+COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
+COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
+
+USER nextjs
+
+EXPOSE 3000
+
+ENV PORT=3000
+ENV HOSTNAME="0.0.0.0"
+
+CMD ["node", "server.js"]

app/frontend/app/api/copilotkit/route.ts

@@ -6,22 +6,16 @@ import {
 } from "@copilotkit/runtime";
 import type { NextRequest } from "next/server";
 
-// Connect CopilotKit to PydanticAI via HttpAgent
-// The HttpAgent creates a bridge between the Next.js frontend and the Python backend
-// It communicates with the server created by agent.to_ag_ui()
 const runtime = new CopilotRuntime({
 	agents: {
-		// "research_agent" maps to the agent name used in useCoAgent() on the frontend
-		research_agent: new HttpAgent({
-			url: `${process.env.BACKEND_URL || "http://backend:8000"}/agent`,
+		chat_agent: new HttpAgent({
+			url: `${process.env.BACKEND_URL || "http://backend:8000"}/v1/chat/stream`,
 		}),
 	},
 });
 
-// Service adapter for multi-agent support (empty since we only have one agent)
 const serviceAdapter = new ExperimentalEmptyAdapter();
 
-// Next.js API route handler that proxies requests between frontend and backend
 export async function POST(request: NextRequest) {
 	const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
 		runtime,

app/frontend/app/api/documents/route.ts

@@ -0,0 +1,16 @@
+import { NextResponse } from "next/server";
+
+export async function GET() {
+	const backendUrl = process.env.BACKEND_URL || "http://backend:8000";
+
+	try {
+		const response = await fetch(`${backendUrl}/api/documents`);
+		const data = await response.json();
+		return NextResponse.json(data);
+	} catch {
+		return NextResponse.json(
+			{ documents: [], error: "Backend unavailable" },
+			{ status: 503 },
+		);
+	}
+}

app/frontend/app/api/info/route.ts

@@ -0,0 +1,16 @@
+import { NextResponse } from "next/server";
+
+export async function GET() {
+	const backendUrl = process.env.BACKEND_URL || "http://backend:8000";
+
+	try {
+		const response = await fetch(`${backendUrl}/api/info`);
+		const data = await response.json();
+		return NextResponse.json(data);
+	} catch {
+		return NextResponse.json(
+			{ exists: false, error: "Backend unavailable" },
+			{ status: 503 },
+		);
+	}
+}

app/frontend/app/api/visualize/[chunk_id]/route.ts

@@ -0,0 +1,26 @@
+import { NextResponse } from "next/server";
+
+export async function GET(
+	request: Request,
+	{ params }: { params: Promise<{ chunk_id: string }> },
+) {
+	const { chunk_id } = await params;
+	const backendUrl = process.env.BACKEND_URL || "http://backend:8000";
+	const refs = new URL(request.url).searchParams.get("refs");
+	const query = refs ? `?refs=${encodeURIComponent(refs)}` : "";
+
+	try {
+		const response = await fetch(
+			`${backendUrl}/api/visualize/${chunk_id}${query}`,
+		);
+		const data = await response.json();
+
+		if (!response.ok) {
+			return NextResponse.json(data, { status: response.status });
+		}
+
+		return NextResponse.json(data);
+	} catch {
+		return NextResponse.json({ error: "Backend unavailable" }, { status: 503 });
+	}
+}

app/frontend/app/layout.tsx

@@ -1,10 +1,10 @@
 import type { Metadata } from "next";
+import "@copilotkit/react-core/v2/styles.css";
 import "./globals.css";
 
 export const metadata: Metadata = {
-	title: "Haiku.rag Research Assistant",
-	description:
-		"Interactive research powered by Haiku.rag, Pydantic AI, and AG-UI",
+	title: "haiku.rag Chat",
+	description: "Conversational RAG powered by haiku.rag and AG-UI",
 };
 
 export default function RootLayout({

app/frontend/app/page.tsx

@@ -0,0 +1,5 @@
+import Chat from "@/components/Chat";
+
+export default function Home() {
+	return <Chat />;
+}

app/frontend/biome.json

@@ -1,9 +1,9 @@
 {
 	"$schema": "https://biomejs.dev/schemas/2.2.6/schema.json",
 	"vcs": {
-		"enabled": false,
+		"enabled": true,
 		"clientKind": "git",
-		"useIgnoreFile": false
+		"useIgnoreFile": true
 	},
 	"files": {
 		"ignoreUnknown": false
@@ -15,7 +15,11 @@
 	"linter": {
 		"enabled": true,
 		"rules": {
-			"recommended": true
+			"recommended": true,
+			"a11y": {
+				"noAutofocus": "off",
+				"noSvgWithoutTitle": "off"
+			}
 		}
 	},
 	"javascript": {

app/frontend/components/Chat.tsx

@@ -0,0 +1,482 @@
+"use client";
+
+import {
+	CopilotChatMessageView,
+	CopilotChatView,
+	CopilotKitProvider,
+	defineToolCallRenderer,
+	UseAgentUpdate,
+	useAgent,
+	useCopilotKit,
+} from "@copilotkit/react-core/v2";
+import {
+	createContext,
+	useCallback,
+	useContext,
+	useEffect,
+	useMemo,
+	useState,
+} from "react";
+import { FilterIcon } from "../lib/icons";
+import type { RAGState } from "../lib/sessionStorage";
+import {
+	AGUI_STATE_KEY,
+	agentStateOf,
+	createSession,
+	getActiveSessionId,
+	getLatestCitations,
+	getSession,
+	normalizeRAGState,
+	updateSessionMessages,
+} from "../lib/sessionStorage";
+import CitationBlock from "./CitationBlock";
+import DbInfo from "./DbInfo";
+import DocumentFilter from "./DocumentFilter";
+import SessionManager from "./SessionManager";
+
+// AG-UI state is namespaced under AGUI_STATE_KEY (see sessionStorage).
+interface AgentState {
+	[AGUI_STATE_KEY]?: RAGState;
+}
+
+// biome-ignore lint/suspicious/noExplicitAny: CopilotKit message objects vary at runtime
+function serializeMessages(messages: any[]): any[] {
+	return JSON.parse(JSON.stringify(messages));
+}
+
+function SpinnerIcon() {
+	return (
+		<svg
+			width="16"
+			height="16"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+			className="tool-spinner"
+		>
+			<path d="M21 12a9 9 0 1 1-6.219-8.56" />
+		</svg>
+	);
+}
+
+function CheckIcon() {
+	return (
+		<svg
+			width="16"
+			height="16"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2.5"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<polyline points="20 6 9 17 4 12" />
+		</svg>
+	);
+}
+
+function SearchIcon() {
+	return (
+		<svg
+			width="14"
+			height="14"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<circle cx="11" cy="11" r="8" />
+			<path d="m21 21-4.3-4.3" />
+		</svg>
+	);
+}
+
+function MessageIcon() {
+	return (
+		<svg
+			width="14"
+			height="14"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M7.9 20A9 9 0 1 0 4 16.1L2 22Z" />
+		</svg>
+	);
+}
+
+function ToolCallIndicator({
+	toolName,
+	status,
+	args,
+}: {
+	toolName: string;
+	status: string;
+	args: Record<string, unknown>;
+}) {
+	const isComplete = status === "complete";
+
+	const getToolIcon = () => {
+		switch (toolName) {
+			case "rag_search":
+				return <SearchIcon />;
+			case "rag_cite":
+				return <MessageIcon />;
+			default:
+				return <SearchIcon />;
+		}
+	};
+
+	const getToolLabel = () => {
+		switch (toolName) {
+			case "rag_search":
+				return "Search";
+			case "rag_cite":
+				return "Cite";
+			default:
+				return toolName;
+		}
+	};
+
+	const getDescription = () => {
+		switch (toolName) {
+			case "rag_search": {
+				const query = args.query as string;
+				return <span className="tool-query">{query}</span>;
+			}
+			case "rag_cite":
+				return <span className="tool-query">Registering citations</span>;
+			default:
+				return <span>Processing...</span>;
+		}
+	};
+
+	return (
+		<div className={`tool-call-card ${isComplete ? "complete" : "loading"}`}>
+			<div className="tool-status-icon">
+				{isComplete ? <CheckIcon /> : <SpinnerIcon />}
+			</div>
+			<div className="tool-content">
+				<div className="tool-header">
+					<span className="tool-badge">
+						{getToolIcon()}
+						{getToolLabel()}
+					</span>
+					<span className="tool-status-text">
+						{isComplete ? "Done" : "Working..."}
+					</span>
+				</div>
+				<div className="tool-description">{getDescription()}</div>
+			</div>
+		</div>
+	);
+}
+
+// Context for sharing chat state with the message view
+const ChatStateContext = createContext<RAGState | null>(null);
+
+// Wildcard tool call renderer for all server-side tools
+const toolCallRenderers = [
+	defineToolCallRenderer({
+		name: "*",
+		render: ({ name, args, result }) => (
+			<ToolCallIndicator
+				toolName={name}
+				status={result !== undefined ? "complete" : "loading"}
+				args={(args ?? {}) as Record<string, unknown>}
+			/>
+		),
+	}),
+];
+
+// Custom message view that injects CitationBlocks after assistant responses.
+// Uses CopilotChatMessageView's children render prop to post-process the
+// rendered message elements and inject citations at the right positions.
+function MessageViewWithCitations({
+	messages = [],
+	isRunning = false,
+}: {
+	// biome-ignore lint/suspicious/noExplicitAny: AG-UI Message type is a broad union
+	messages?: any[];
+	isRunning?: boolean;
+}) {
+	const ragState = useContext(ChatStateContext);
+	const latestCitations = ragState ? getLatestCitations(ragState) : [];
+
+	const cursor = isRunning ? (
+		<div key="cursor" className="streaming-cursor">
+			<span className="dot" />
+			<span className="dot" />
+			<span className="dot" />
+		</div>
+	) : null;
+
+	// CopilotChatMessageView renders one element per user/assistant message.
+	// Inject CitationBlocks after assistant responses that
+	// followed tool calls.
+	return (
+		<CopilotChatMessageView messages={messages} isRunning={isRunning}>
+			{({ messageElements }) => {
+				const result: React.ReactNode[] = [];
+				let elemIdx = 0;
+				let seenToolCalls = false;
+
+				for (const msg of messages) {
+					if (msg.role === "user") {
+						seenToolCalls = false;
+					}
+
+					if (
+						msg.role === "assistant" &&
+						Array.isArray(msg.toolCalls) &&
+						msg.toolCalls.length > 0
+					) {
+						seenToolCalls = true;
+					}
+
+					if (msg.role !== "user" && msg.role !== "assistant") continue;
+
+					if (elemIdx < messageElements.length) {
+						result.push(messageElements[elemIdx]);
+						elemIdx++;
+					}
+
+					// After an assistant text response that followed tool calls,
+					// show citations from the latest turn
+					if (msg.role === "assistant" && msg.content && seenToolCalls) {
+						if (latestCitations.length > 0) {
+							result.push(
+								<CitationBlock
+									key={`citations-${msg.id}`}
+									citations={latestCitations}
+								/>,
+							);
+						}
+						seenToolCalls = false;
+					}
+				}
+
+				while (elemIdx < messageElements.length) {
+					result.push(messageElements[elemIdx]);
+					elemIdx++;
+				}
+
+				return (
+					<>
+						{result}
+						{cursor}
+					</>
+				);
+			}}
+		</CopilotChatMessageView>
+	);
+}
+MessageViewWithCitations.Cursor = CopilotChatMessageView.Cursor;
+
+function ChatContentInner({
+	sessionId,
+	onSessionChange,
+}: {
+	sessionId: string;
+	onSessionChange: (id: string) => void;
+}) {
+	const [filterOpen, setFilterOpen] = useState(false);
+	// Track selected document names locally (frontend-only)
+	const [selectedDocuments, setSelectedDocuments] = useState<string[]>([]);
+
+	const { agent } = useAgent({
+		agentId: "chat_agent",
+		updates: [
+			UseAgentUpdate.OnMessagesChanged,
+			UseAgentUpdate.OnStateChanged,
+			UseAgentUpdate.OnRunStatusChanged,
+		],
+	});
+	const { copilotkit: ck } = useCopilotKit();
+
+	// Set threadId (CopilotChat normally does this in its connect effect)
+	useEffect(() => {
+		agent.threadId = sessionId;
+	}, [agent, sessionId]);
+
+	const ragState = normalizeRAGState(
+		(agent.state as AgentState)?.[AGUI_STATE_KEY],
+	);
+
+	// Restore session from localStorage when agent reference changes.
+	// useAgent returns a provisional agent initially, then the real agent
+	// after runtime connects — re-run restore each time so messages stick.
+	useEffect(() => {
+		if (agent.messages.length > 0) return;
+		const session = getSession(sessionId);
+		// Seed state for the capabilities; the backend replaces it after each run.
+		// The whole namespace map goes back, not just the fields this UI reads:
+		// compaction and the citation policy read what earlier questions recorded.
+		agent.setState({ ...agent.state, ...agentStateOf(session ?? undefined) });
+		if (session && session.messages.length > 0) {
+			// biome-ignore lint/suspicious/noExplicitAny: AG-UI Message type is a broad union
+			agent.setMessages(session.messages as any[]);
+		}
+	}, [agent, sessionId]);
+
+	// Persist messages and state to localStorage.
+	// Read ragState from agent.state at effect time (not render time) so that
+	// restore and persist effects in the same commit see consistent state.
+	// biome-ignore lint/correctness/useExhaustiveDependencies: JSON.stringify tracks content changes
+	useEffect(() => {
+		if (sessionId && agent.messages.length > 0) {
+			updateSessionMessages(
+				sessionId,
+				serializeMessages(agent.messages),
+				(agent.state ?? {}) as Record<string, unknown>,
+			);
+		}
+	}, [JSON.stringify(agent.messages), ragState, sessionId]);
+
+	// Deduplicate messages by id to avoid React duplicate key warnings.
+	// biome-ignore lint/correctness/useExhaustiveDependencies: stable identity via agent ref
+	const messages = useMemo(() => {
+		const seen = new Map<string, number>();
+		const msgs = agent.messages;
+		for (let i = 0; i < msgs.length; i++) {
+			const id = msgs[i].id;
+			if (id) seen.set(id, i);
+		}
+		return msgs.filter((msg, i) => !msg.id || seen.get(msg.id) === i);
+	}, [JSON.stringify(agent.messages)]);
+
+	const onSubmitMessage = useCallback(
+		async (text: string) => {
+			agent.addMessage({
+				id: crypto.randomUUID(),
+				role: "user",
+				content: text,
+			});
+			try {
+				await ck.runAgent({ agent });
+			} catch (error) {
+				console.error("runAgent failed", error);
+			}
+		},
+		[agent, ck],
+	);
+
+	const onStop = useCallback(() => {
+		try {
+			ck.stopAgent({ agent });
+		} catch {
+			agent.abortRun();
+		}
+	}, [agent, ck]);
+
+	const handleFilterApply = (selected: string[]) => {
+		setSelectedDocuments(selected);
+		// Convert selected document names to SQL filter for the backend
+		const filter =
+			selected.length > 0
+				? selected
+						.map(
+							(name) =>
+								`(title LIKE '%${name.replace(/'/g, "''")}%' OR uri LIKE '%${name.replace(/'/g, "''")}%')`,
+						)
+						.join(" OR ")
+				: null;
+		agent.setState({
+			...agent.state,
+			[AGUI_STATE_KEY]: {
+				...ragState,
+				document_filter: filter,
+			},
+		});
+	};
+
+	return (
+		<ChatStateContext.Provider value={ragState}>
+			<div className="chat-wrapper">
+				<div className="chat-container">
+					<div className="chat-header">
+						<SessionManager
+							activeSessionId={sessionId}
+							onSessionChange={onSessionChange}
+						/>
+						<button
+							type="button"
+							className={`header-btn ${selectedDocuments.length > 0 ? "has-content" : ""}`}
+							onClick={() => setFilterOpen(true)}
+							title={
+								selectedDocuments.length > 0
+									? `Filtering: ${selectedDocuments.length} document(s)`
+									: "Filter documents"
+							}
+						>
+							<FilterIcon />
+							{selectedDocuments.length > 0
+								? `Filter (${selectedDocuments.length})`
+								: "Filter"}
+						</button>
+					</div>
+					<div className="chat-content">
+						<CopilotChatView
+							messageView={MessageViewWithCitations}
+							messages={messages}
+							isRunning={agent.isRunning}
+							onSubmitMessage={onSubmitMessage}
+							onStop={onStop}
+						>
+							{({ scrollView, input }) => (
+								<div className="chat-layout">
+									<div className="chat-scroll-area">{scrollView}</div>
+									<div className="chat-input-area">{input}</div>
+								</div>
+							)}
+						</CopilotChatView>
+					</div>
+					<DbInfo />
+				</div>
+			</div>
+			<DocumentFilter
+				isOpen={filterOpen}
+				onClose={() => setFilterOpen(false)}
+				selected={selectedDocuments}
+				onApply={handleFilterApply}
+			/>
+		</ChatStateContext.Provider>
+	);
+}
+
+export default function Chat() {
+	const [activeSessionId, setActiveSessionId] = useState<string | null>(null);
+
+	useEffect(() => {
+		let id = getActiveSessionId();
+		if (!id) {
+			id = createSession().id;
+		}
+		setActiveSessionId(id);
+	}, []);
+
+	if (!activeSessionId) return null;
+
+	return (
+		<CopilotKitProvider
+			key={activeSessionId}
+			runtimeUrl="/api/copilotkit"
+			useSingleEndpoint
+			renderToolCalls={toolCallRenderers}
+		>
+			<ChatContentInner
+				sessionId={activeSessionId}
+				onSessionChange={setActiveSessionId}
+			/>
+		</CopilotKitProvider>
+	);
+}

app/frontend/components/CitationBlock.tsx

@@ -0,0 +1,226 @@
+"use client";
+
+import { useCallback, useEffect, useRef, useState } from "react";
+import type { Citation } from "../lib/sessionStorage";
+
+interface CitationBlockProps {
+	citations: Citation[];
+}
+
+interface VisualGroundingState {
+	isOpen: boolean;
+	chunkId: string | null;
+	images: string[];
+	loading: boolean;
+	error: string | null;
+}
+
+function CitationItem({
+	citation,
+	onViewInDocument,
+}: {
+	citation: Citation;
+	onViewInDocument: (chunkId: string, refs?: string[]) => void;
+}) {
+	const [expanded, setExpanded] = useState(false);
+
+	const title = citation.document_title || citation.document_uri || "Unknown";
+	const pageInfo =
+		citation.page_numbers.length > 0
+			? `p. ${citation.page_numbers.join(", ")}`
+			: null;
+
+	return (
+		<div className="citation-item">
+			<button
+				type="button"
+				className="citation-header"
+				onClick={() => setExpanded(!expanded)}
+			>
+				<span className="citation-index">[{citation.index}]</span>
+				<span className="citation-title">{title}</span>
+				{pageInfo && <span className="citation-page">{pageInfo}</span>}
+				<span className={`citation-chevron ${expanded ? "expanded" : ""}`}>
+					{expanded ? "▼" : "▶"}
+				</span>
+			</button>
+			{expanded && (
+				<div className="citation-content">
+					{citation.headings && citation.headings.length > 0 && (
+						<div className="citation-headings">
+							{citation.headings.join(" › ")}
+						</div>
+					)}
+					<div className="citation-text">{citation.content}</div>
+					<button
+						type="button"
+						className="citation-view-btn"
+						onClick={() =>
+							onViewInDocument(
+								citation.chunk_ids?.length
+									? citation.chunk_ids.join(",")
+									: citation.chunk_id,
+								citation.doc_item_refs,
+							)
+						}
+					>
+						View in Document
+					</button>
+				</div>
+			)}
+		</div>
+	);
+}
+
+export default function CitationBlock({ citations }: CitationBlockProps) {
+	const [visualGrounding, setVisualGrounding] = useState<VisualGroundingState>({
+		isOpen: false,
+		chunkId: null,
+		images: [],
+		loading: false,
+		error: null,
+	});
+	// AbortController for the in-flight visualize fetch so a rapid close/reopen
+	// doesn't let a stale response overwrite the new request's state.
+	const abortRef = useRef<AbortController | null>(null);
+
+	// Abort any in-flight request on unmount.
+	useEffect(() => {
+		return () => abortRef.current?.abort();
+	}, []);
+
+	const fetchVisualGrounding = useCallback(
+		async (chunkId: string, refs?: string[]) => {
+			abortRef.current?.abort();
+			const controller = new AbortController();
+			abortRef.current = controller;
+
+			setVisualGrounding({
+				isOpen: true,
+				chunkId,
+				images: [],
+				loading: true,
+				error: null,
+			});
+
+			const query = refs?.length
+				? `?refs=${encodeURIComponent(JSON.stringify(refs))}`
+				: "";
+			try {
+				const response = await fetch(
+					`/api/visualize/${encodeURIComponent(chunkId)}${query}`,
+					{
+						signal: controller.signal,
+					},
+				);
+				const data = await response.json();
+
+				if (controller.signal.aborted) return;
+				if (!response.ok) {
+					throw new Error(data.error || "Failed to fetch visual grounding");
+				}
+
+				setVisualGrounding((prev) => ({
+					...prev,
+					images: data.images || [],
+					loading: false,
+					error: data.images?.length === 0 ? data.message : null,
+				}));
+			} catch (err) {
+				if (controller.signal.aborted) return;
+				setVisualGrounding((prev) => ({
+					...prev,
+					loading: false,
+					error: err instanceof Error ? err.message : "Unknown error",
+				}));
+			}
+		},
+		[],
+	);
+
+	const closeVisualGrounding = useCallback(() => {
+		abortRef.current?.abort();
+		abortRef.current = null;
+		setVisualGrounding({
+			isOpen: false,
+			chunkId: null,
+			images: [],
+			loading: false,
+			error: null,
+		});
+	}, []);
+
+	if (!citations || citations.length === 0) {
+		return null;
+	}
+
+	return (
+		<>
+			<div className="citation-block">
+				<div className="citation-block-header">
+					Sources ({citations.length})
+				</div>
+				{citations.map((citation) => (
+					<CitationItem
+						key={citation.chunk_id}
+						citation={citation}
+						onViewInDocument={fetchVisualGrounding}
+					/>
+				))}
+			</div>
+
+			{visualGrounding.isOpen && (
+				<div
+					className="visual-modal-overlay"
+					onClick={closeVisualGrounding}
+					onKeyDown={(e) => e.key === "Escape" && closeVisualGrounding()}
+					role="dialog"
+					aria-modal="true"
+					aria-label="Visual grounding"
+				>
+					{/* biome-ignore lint/a11y/noStaticElementInteractions: modal content wrapper */}
+					<div
+						className="visual-modal"
+						onClick={(e) => e.stopPropagation()}
+						onKeyDown={(e) => e.stopPropagation()}
+					>
+						<button
+							type="button"
+							className="visual-modal-close"
+							onClick={closeVisualGrounding}
+						>
+							✕
+						</button>
+						<h3 className="visual-modal-title">Visual Grounding</h3>
+						{visualGrounding.loading && (
+							<div className="visual-modal-loading">Loading...</div>
+						)}
+						{visualGrounding.error && (
+							<div className="visual-modal-error">{visualGrounding.error}</div>
+						)}
+						{!visualGrounding.loading &&
+							!visualGrounding.error &&
+							visualGrounding.images.length > 0 && (
+								<div className="visual-modal-images">
+									{visualGrounding.images.map((img, idx) => (
+										// biome-ignore lint/suspicious/noArrayIndexKey: images have no stable id
+										<div key={idx}>
+											<div className="visual-modal-page-label">
+												Page {idx + 1} of {visualGrounding.images.length}
+											</div>
+											{/* biome-ignore lint/performance/noImgElement: base64 data URLs require img element */}
+											<img
+												src={`data:image/png;base64,${img}`}
+												alt={`Page ${idx + 1}`}
+												className="visual-modal-image"
+											/>
+										</div>
+									))}
+								</div>
+							)}
+					</div>
+				</div>
+			)}
+		</>
+	);
+}

app/frontend/components/DbInfo.tsx

@@ -0,0 +1,83 @@
+"use client";
+
+import { useEffect, useState } from "react";
+
+interface DbInfoData {
+	exists: boolean;
+	path: string;
+	documents: number;
+	chunks: number;
+	documents_bytes: number;
+	chunks_bytes: number;
+	has_vector_index: boolean;
+}
+
+function formatBytes(bytes: number): string {
+	if (bytes === 0) return "0 B";
+	const k = 1024;
+	const sizes = ["B", "KB", "MB", "GB"];
+	const i = Math.floor(Math.log(bytes) / Math.log(k));
+	return `${Number.parseFloat((bytes / k ** i).toFixed(1))} ${sizes[i]}`;
+}
+
+export default function DbInfo() {
+	const [info, setInfo] = useState<DbInfoData | null>(null);
+	const [error, setError] = useState<string | null>(null);
+
+	useEffect(() => {
+		fetch("/api/info")
+			.then((res) => res.json())
+			.then(setInfo)
+			.catch((err) => setError(err.message));
+	}, []);
+
+	if (error) {
+		return (
+			<div className="db-info db-info-error">
+				<span>Database unavailable</span>
+			</div>
+		);
+	}
+
+	if (!info) {
+		return (
+			<div className="db-info db-info-loading">
+				<span>Loading...</span>
+			</div>
+		);
+	}
+
+	if (!info.exists) {
+		return (
+			<div className="db-info db-info-empty">
+				<span>No database found</span>
+			</div>
+		);
+	}
+
+	return (
+		<div className="db-info">
+			<div className="db-stat">
+				<span className="db-stat-value">{info.documents}</span>
+				<span className="db-stat-label">documents</span>
+			</div>
+			<div className="db-stat">
+				<span className="db-stat-value">{info.chunks}</span>
+				<span className="db-stat-label">chunks</span>
+			</div>
+			<div className="db-stat">
+				<span className="db-stat-value">
+					{formatBytes(info.documents_bytes + info.chunks_bytes)}
+				</span>
+				<span className="db-stat-label">total</span>
+			</div>
+			<div className="db-stat">
+				<span
+					className={`db-index-badge ${info.has_vector_index ? "indexed" : "not-indexed"}`}
+				>
+					{info.has_vector_index ? "indexed" : "no index"}
+				</span>
+			</div>
+		</div>
+	);
+}

app/frontend/components/DocumentFilter.tsx

@@ -0,0 +1,205 @@
+"use client";
+
+import { useCallback, useEffect, useId, useState } from "react";
+import { FilterIcon } from "../lib/icons";
+
+interface Document {
+	id: string;
+	title: string | null;
+	uri: string | null;
+}
+
+interface DocumentFilterProps {
+	isOpen: boolean;
+	onClose: () => void;
+	selected: string[];
+	onApply: (selected: string[]) => void;
+}
+
+const getDisplayName = (doc: Document) => doc.title || doc.uri || doc.id;
+
+export default function DocumentFilter({
+	isOpen,
+	onClose,
+	selected,
+	onApply,
+}: DocumentFilterProps) {
+	const titleId = useId();
+	const [documents, setDocuments] = useState<Document[]>([]);
+	const [loading, setLoading] = useState(true);
+	const [searchTerm, setSearchTerm] = useState("");
+	// Track selection by document id — two docs can share a title, but ids
+	// are unique. Display names are only used for rendering and for the
+	// filter string returned to the parent.
+	const [localSelected, setLocalSelected] = useState<Set<string>>(new Set());
+
+	// Refetch on every open so newly-added or deleted documents show up.
+	useEffect(() => {
+		if (!isOpen) return;
+		setLoading(true);
+		fetch("/api/documents")
+			.then((res) => res.json())
+			.then((data) => {
+				setDocuments(data.documents || []);
+				setLoading(false);
+			})
+			.catch(() => {
+				setLoading(false);
+			});
+	}, [isOpen]);
+
+	// Seed local selection from the parent's display-name list once documents
+	// are available. Any doc whose display name is in `selected` starts checked.
+	useEffect(() => {
+		if (!isOpen) return;
+		const selectedNames = new Set(selected);
+		setLocalSelected(
+			new Set(
+				documents
+					.filter((d) => selectedNames.has(getDisplayName(d)))
+					.map((d) => d.id),
+			),
+		);
+		setSearchTerm("");
+	}, [isOpen, selected, documents]);
+
+	const handleKeyDown = useCallback(
+		(e: React.KeyboardEvent) => {
+			if (e.key === "Escape") {
+				onClose();
+			}
+		},
+		[onClose],
+	);
+
+	const toggleDocument = (docId: string) => {
+		setLocalSelected((prev) => {
+			const next = new Set(prev);
+			if (next.has(docId)) {
+				next.delete(docId);
+			} else {
+				next.add(docId);
+			}
+			return next;
+		});
+	};
+
+	const handleApply = () => {
+		const names = documents
+			.filter((d) => localSelected.has(d.id))
+			.map(getDisplayName);
+		// Dedupe: two selected docs sharing a title collapse to one filter term.
+		onApply(Array.from(new Set(names)));
+		onClose();
+	};
+
+	const handleClearAll = () => {
+		setLocalSelected(new Set());
+	};
+
+	const filteredDocuments = documents.filter((doc) => {
+		if (!searchTerm) return true;
+		const displayName = getDisplayName(doc).toLowerCase();
+		return displayName.includes(searchTerm.toLowerCase());
+	});
+
+	if (!isOpen) {
+		return null;
+	}
+
+	return (
+		<div
+			className="filter-modal-overlay"
+			onClick={onClose}
+			onKeyDown={handleKeyDown}
+			role="dialog"
+			aria-modal="true"
+			aria-labelledby={titleId}
+		>
+			{/* biome-ignore lint/a11y/noStaticElementInteractions: modal content wrapper */}
+			<div
+				className="filter-modal"
+				onClick={(e) => e.stopPropagation()}
+				onKeyDown={(e) => e.stopPropagation()}
+			>
+				<div className="filter-modal-header">
+					<div className="filter-modal-icon">
+						<FilterIcon size={24} strokeWidth={1.5} />
+					</div>
+					<h2 id={titleId} className="filter-modal-title">
+						Filter Documents
+					</h2>
+				</div>
+				<p className="filter-modal-description">
+					Select documents to restrict searches. When active, only selected
+					documents will be searched.
+				</p>
+				<input
+					type="text"
+					className="filter-search"
+					placeholder="Search documents..."
+					value={searchTerm}
+					onChange={(e) => setSearchTerm(e.target.value)}
+				/>
+				<div className="filter-list">
+					{loading ? (
+						<div className="filter-loading">Loading documents...</div>
+					) : filteredDocuments.length === 0 ? (
+						<div className="filter-empty">
+							{searchTerm ? "No matching documents" : "No documents found"}
+						</div>
+					) : (
+						filteredDocuments.map((doc) => {
+							const displayName = getDisplayName(doc);
+							return (
+								<label key={doc.id} className="filter-item">
+									<input
+										type="checkbox"
+										checked={localSelected.has(doc.id)}
+										onChange={() => toggleDocument(doc.id)}
+									/>
+									<span className="filter-item-label">{displayName}</span>
+								</label>
+							);
+						})
+					)}
+				</div>
+				<div className="filter-footer">
+					<div className="filter-count">
+						{localSelected.size > 0 ? (
+							<>
+								<strong>{localSelected.size}</strong> document
+								{localSelected.size === 1 ? "" : "s"} selected
+								<button
+									type="button"
+									className="filter-btn filter-btn-clear"
+									onClick={handleClearAll}
+								>
+									Clear all
+								</button>
+							</>
+						) : (
+							"No filter (all documents)"
+						)}
+					</div>
+					<div className="filter-buttons">
+						<button
+							type="button"
+							className="filter-btn filter-btn-secondary"
+							onClick={onClose}
+						>
+							Cancel
+						</button>
+						<button
+							type="button"
+							className="filter-btn filter-btn-primary"
+							onClick={handleApply}
+						>
+							Apply
+						</button>
+					</div>
+				</div>
+			</div>
+		</div>
+	);
+}

app/frontend/components/SessionManager.tsx

@@ -0,0 +1,268 @@
+"use client";
+
+import { useEffect, useRef, useState } from "react";
+import { formatRelativeTime } from "../lib/format";
+import {
+	createSession,
+	deleteSession,
+	exportSessionToMarkdown,
+	getAllSessions,
+	type StoredSession,
+	setActiveSessionId,
+} from "../lib/sessionStorage";
+
+interface SessionManagerProps {
+	activeSessionId: string | null;
+	onSessionChange: (sessionId: string) => void;
+}
+
+function HistoryIcon() {
+	return (
+		<svg
+			width="18"
+			height="18"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M3 12a9 9 0 1 0 9-9 9.75 9.75 0 0 0-6.74 2.74L3 8" />
+			<path d="M3 3v5h5" />
+			<path d="M12 7v5l4 2" />
+		</svg>
+	);
+}
+
+function PlusIcon() {
+	return (
+		<svg
+			width="14"
+			height="14"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2.5"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M12 5v14" />
+			<path d="M5 12h14" />
+		</svg>
+	);
+}
+
+function DownloadIcon() {
+	return (
+		<svg
+			width="14"
+			height="14"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4" />
+			<polyline points="7 10 12 15 17 10" />
+			<line x1="12" y1="15" x2="12" y2="3" />
+		</svg>
+	);
+}
+
+function TrashIcon() {
+	return (
+		<svg
+			width="14"
+			height="14"
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth="2"
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M3 6h18" />
+			<path d="M19 6v14c0 1-1 2-2 2H7c-1 0-2-1-2-2V6" />
+			<path d="M8 6V4c0-1 1-2 2-2h4c1 0 2 1 2 2v2" />
+		</svg>
+	);
+}
+
+export default function SessionManager({
+	activeSessionId,
+	onSessionChange,
+}: SessionManagerProps) {
+	const [isOpen, setIsOpen] = useState(false);
+	const [sessions, setSessions] = useState<StoredSession[]>([]);
+	const [confirmDelete, setConfirmDelete] = useState<string | null>(null);
+	const dropdownRef = useRef<HTMLDivElement>(null);
+
+	useEffect(() => {
+		if (isOpen) setSessions(getAllSessions());
+	}, [isOpen]);
+
+	useEffect(() => {
+		function handleClickOutside(e: MouseEvent) {
+			if (
+				dropdownRef.current &&
+				!dropdownRef.current.contains(e.target as Node)
+			) {
+				setIsOpen(false);
+				setConfirmDelete(null);
+			}
+		}
+		if (isOpen) document.addEventListener("mousedown", handleClickOutside);
+		return () => document.removeEventListener("mousedown", handleClickOutside);
+	}, [isOpen]);
+
+	const handleNewSession = () => {
+		const session = createSession();
+		setSessions(getAllSessions());
+		setIsOpen(false);
+		onSessionChange(session.id);
+	};
+
+	const handleSelectSession = (id: string) => {
+		setActiveSessionId(id);
+		setIsOpen(false);
+		onSessionChange(id);
+	};
+
+	const handleDelete = (id: string) => {
+		deleteSession(id);
+		const remaining = getAllSessions();
+		setSessions(remaining);
+		setConfirmDelete(null);
+		if (id === activeSessionId) {
+			if (remaining.length > 0) {
+				setActiveSessionId(remaining[0].id);
+				onSessionChange(remaining[0].id);
+			} else {
+				const session = createSession();
+				setSessions(getAllSessions());
+				onSessionChange(session.id);
+			}
+		}
+	};
+
+	const handleExport = (session: StoredSession) => {
+		exportSessionToMarkdown(session);
+	};
+
+	const activeTitle =
+		sessions.find((s) => s.id === activeSessionId)?.title ?? "Sessions";
+
+	return (
+		<div ref={dropdownRef} style={{ position: "relative" }}>
+			<button
+				type="button"
+				className="header-btn"
+				onClick={() => setIsOpen(!isOpen)}
+				title="Session history"
+			>
+				<HistoryIcon />
+				<span
+					style={{
+						maxWidth: 120,
+						overflow: "hidden",
+						textOverflow: "ellipsis",
+						whiteSpace: "nowrap",
+					}}
+				>
+					{activeTitle}
+				</span>
+			</button>
+
+			{isOpen && (
+				<div className="session-dropdown">
+					<div className="session-dropdown-header">
+						<span>Sessions</span>
+						<button
+							type="button"
+							className="new-session-btn"
+							onClick={handleNewSession}
+						>
+							<PlusIcon />
+							New
+						</button>
+					</div>
+					<div className="session-list">
+						{sessions.length === 0 && (
+							<div
+								style={{
+									padding: "16px",
+									textAlign: "center",
+									color: "#94a3b8",
+									fontSize: "13px",
+								}}
+							>
+								No sessions yet
+							</div>
+						)}
+						{sessions.map((session) => (
+							<div
+								key={session.id}
+								className={`session-item ${session.id === activeSessionId ? "active" : ""}`}
+							>
+								<button
+									type="button"
+									className="session-item-content"
+									onClick={() => handleSelectSession(session.id)}
+									onKeyDown={(e) => {
+										if (e.key === "Enter") handleSelectSession(session.id);
+									}}
+								>
+									<div className="session-item-title">{session.title}</div>
+									<div className="session-item-meta">
+										<span>{session.messages.length} messages</span>
+										<span>{formatRelativeTime(session.updatedAt, true)}</span>
+									</div>
+								</button>
+								{confirmDelete === session.id ? (
+									<div className="confirm-delete">
+										<button
+											type="button"
+											className="confirm-yes"
+											onClick={() => handleDelete(session.id)}
+										>
+											Delete
+										</button>
+										<button
+											type="button"
+											className="confirm-no"
+											onClick={() => setConfirmDelete(null)}
+										>
+											Cancel
+										</button>
+									</div>
+								) : (
+									<div className="session-actions">
+										<button
+											type="button"
+											className="session-action-btn"
+											onClick={() => handleExport(session)}
+											title="Export to markdown"
+										>
+											<DownloadIcon />
+										</button>
+										<button
+											type="button"
+											className="session-action-btn danger"
+											onClick={() => setConfirmDelete(session.id)}
+											title="Delete session"
+										>
+											<TrashIcon />
+										</button>
+									</div>
+								)}
+							</div>
+						))}
+					</div>
+				</div>
+			)}
+		</div>
+	);
+}

app/frontend/lib/format.ts

@@ -0,0 +1,18 @@
+export function formatRelativeTime(dateStr: string, compact = false): string {
+	const now = Date.now();
+	const then = new Date(dateStr).getTime();
+	const seconds = Math.floor((now - then) / 1000);
+	if (seconds < 60) return "just now";
+	const minutes = Math.floor(seconds / 60);
+	if (minutes < 60)
+		return compact
+			? `${minutes}m ago`
+			: `${minutes} minute${minutes === 1 ? "" : "s"} ago`;
+	const hours = Math.floor(minutes / 60);
+	if (hours < 24)
+		return compact
+			? `${hours}h ago`
+			: `${hours} hour${hours === 1 ? "" : "s"} ago`;
+	const days = Math.floor(hours / 24);
+	return compact ? `${days}d ago` : new Date(dateStr).toLocaleDateString();
+}

app/frontend/lib/icons.tsx

@@ -0,0 +1,46 @@
+interface IconProps {
+	size?: number;
+	strokeWidth?: number;
+}
+
+export function BrainIcon({ size = 18, strokeWidth = 2 }: IconProps) {
+	return (
+		<svg
+			width={size}
+			height={size}
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth={strokeWidth}
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<path d="M12 5a3 3 0 1 0-5.997.125 4 4 0 0 0-2.526 5.77 4 4 0 0 0 .556 6.588A4 4 0 1 0 12 18Z" />
+			<path d="M12 5a3 3 0 1 1 5.997.125 4 4 0 0 1 2.526 5.77 4 4 0 0 1-.556 6.588A4 4 0 1 1 12 18Z" />
+			<path d="M15 13a4.5 4.5 0 0 1-3-4 4.5 4.5 0 0 1-3 4" />
+			<path d="M17.599 6.5a3 3 0 0 0 .399-1.375" />
+			<path d="M6.003 5.125A3 3 0 0 0 6.401 6.5" />
+			<path d="M3.477 10.896a4 4 0 0 1 .585-.396" />
+			<path d="M19.938 10.5a4 4 0 0 1 .585.396" />
+			<path d="M6 18a4 4 0 0 1-1.967-.516" />
+			<path d="M19.967 17.484A4 4 0 0 1 18 18" />
+		</svg>
+	);
+}
+
+export function FilterIcon({ size = 18, strokeWidth = 2 }: IconProps) {
+	return (
+		<svg
+			width={size}
+			height={size}
+			viewBox="0 0 24 24"
+			fill="none"
+			stroke="currentColor"
+			strokeWidth={strokeWidth}
+			strokeLinecap="round"
+			strokeLinejoin="round"
+		>
+			<polygon points="22 3 2 3 10 12.46 10 19 14 21 14 12.46 22 3" />
+		</svg>
+	);
+}

app/frontend/lib/sessionStorage.ts

@@ -0,0 +1,191 @@
+export interface Citation {
+	index: number;
+	document_id: string;
+	chunk_id: string;
+	chunk_ids?: string[];
+	document_uri: string;
+	document_title: string | null;
+	page_numbers: number[];
+	headings: string[] | null;
+	content: string;
+	doc_item_refs?: string[];
+}
+
+// Matches RAGState from the backend capability. The fields named here are the
+// ones this UI reads; the capability owns the rest of its namespace, including
+// the evidence record that compaction builds its capsule from, so the state has
+// to round-trip whole rather than be rebuilt from known keys.
+export interface RAGState {
+	citation_index: Record<string, Citation>;
+	citations: string[];
+	document_filter: string | null;
+	searches: Record<string, unknown[]>;
+	[key: string]: unknown;
+}
+
+export interface StoredMessage {
+	id: string;
+	role?: string;
+	content?: string;
+	[key: string]: unknown;
+}
+
+export interface StoredSession {
+	id: string;
+	title: string;
+	messages: StoredMessage[];
+	// The whole AG-UI state. The rag namespace is not the only one a capability
+	// writes: the citation policy records violations beside it.
+	agentState: AgentState;
+	// Sessions stored before agentState existed.
+	ragState?: RAGState;
+	createdAt: string;
+	updatedAt: string;
+}
+
+export const AGUI_STATE_KEY = "rag";
+
+export type AgentState = Record<string, unknown>;
+
+// Reads the rag namespace out of a stored session, whichever way it was stored.
+export function ragStateOf(session?: StoredSession): RAGState {
+	const namespaced = session?.agentState?.[AGUI_STATE_KEY] as
+		| Partial<RAGState>
+		| undefined;
+	return normalizeRAGState(namespaced ?? session?.ragState);
+}
+
+// The state to seed an agent with when a session is resumed.
+export function agentStateOf(session?: StoredSession): AgentState {
+	return session?.agentState ?? { [AGUI_STATE_KEY]: ragStateOf(session) };
+}
+
+const SESSIONS_KEY = "haiku.rag.sessions";
+const ACTIVE_SESSION_KEY = "haiku.rag.activeSession";
+
+export function normalizeRAGState(state?: Partial<RAGState>): RAGState {
+	return {
+		...state,
+		citation_index: state?.citation_index ?? {},
+		citations: state?.citations ?? [],
+		document_filter: state?.document_filter ?? null,
+		searches: state?.searches ?? {},
+	};
+}
+
+export function getLatestCitations(state: RAGState): Citation[] {
+	return state.citations
+		.map((id) => state.citation_index[id])
+		.filter((c): c is Citation => c !== undefined);
+}
+
+export function getAllSessions(): StoredSession[] {
+	const raw = localStorage.getItem(SESSIONS_KEY);
+	if (!raw) return [];
+	try {
+		return JSON.parse(raw) as StoredSession[];
+	} catch {
+		return [];
+	}
+}
+
+export function getSession(id: string): StoredSession | null {
+	return getAllSessions().find((s) => s.id === id) ?? null;
+}
+
+export function getActiveSessionId(): string | null {
+	return localStorage.getItem(ACTIVE_SESSION_KEY);
+}
+
+export function setActiveSessionId(id: string): void {
+	localStorage.setItem(ACTIVE_SESSION_KEY, id);
+}
+
+export function createSession(): StoredSession {
+	const now = new Date().toISOString();
+	const session: StoredSession = {
+		id: crypto.randomUUID(),
+		title: "New Session",
+		messages: [],
+		agentState: { [AGUI_STATE_KEY]: normalizeRAGState() },
+		createdAt: now,
+		updatedAt: now,
+	};
+	const sessions = getAllSessions();
+	sessions.unshift(session);
+	localStorage.setItem(SESSIONS_KEY, JSON.stringify(sessions));
+	setActiveSessionId(session.id);
+	return session;
+}
+
+export function saveSession(session: StoredSession): void {
+	const sessions = getAllSessions();
+	const idx = sessions.findIndex((s) => s.id === session.id);
+	if (idx >= 0) {
+		sessions[idx] = session;
+	} else {
+		sessions.unshift(session);
+	}
+	localStorage.setItem(SESSIONS_KEY, JSON.stringify(sessions));
+}
+
+export function updateSessionMessages(
+	id: string,
+	messages: StoredMessage[],
+	agentState: AgentState,
+): void {
+	const sessions = getAllSessions();
+	const idx = sessions.findIndex((s) => s.id === id);
+	if (idx < 0) return;
+
+	const session = sessions[idx];
+	session.messages = messages;
+	session.agentState = agentState;
+	session.updatedAt = new Date().toISOString();
+
+	// Derive title from first user message
+	if (session.title === "New Session") {
+		const firstUserMsg = messages.find(
+			(m) =>
+				m.content &&
+				typeof m.role === "string" &&
+				m.role.toLowerCase() === "user",
+		);
+		if (firstUserMsg?.content) {
+			session.title =
+				firstUserMsg.content.length > 60
+					? `${firstUserMsg.content.slice(0, 57)}...`
+					: firstUserMsg.content;
+		}
+	}
+
+	sessions[idx] = session;
+	localStorage.setItem(SESSIONS_KEY, JSON.stringify(sessions));
+}
+
+export function deleteSession(id: string): void {
+	const sessions = getAllSessions().filter((s) => s.id !== id);
+	localStorage.setItem(SESSIONS_KEY, JSON.stringify(sessions));
+	if (getActiveSessionId() === id) {
+		localStorage.removeItem(ACTIVE_SESSION_KEY);
+	}
+}
+
+export function exportSessionToMarkdown(session: StoredSession): void {
+	const lines: string[] = [`# ${session.title}`, ""];
+	for (const msg of session.messages) {
+		const role = typeof msg.role === "string" ? msg.role.toLowerCase() : "";
+		if (role === "user" && msg.content) {
+			lines.push(`**User:** ${msg.content}`, "");
+		} else if (role === "assistant" && msg.content) {
+			lines.push(`**Assistant:** ${msg.content}`, "");
+		}
+	}
+	const blob = new Blob([lines.join("\n")], { type: "text/markdown" });
+	const url = URL.createObjectURL(blob);
+	const a = document.createElement("a");
+	a.href = url;
+	a.download = `${session.title.replace(/[^a-zA-Z0-9]/g, "_")}.md`;
+	a.click();
+	URL.revokeObjectURL(url);
+}

app/frontend/package.json

@@ -0,0 +1,30 @@
+{
+	"name": "haiku-rag-app",
+	"version": "0.1.0",
+	"private": true,
+	"scripts": {
+		"dev": "next dev",
+		"build": "next build",
+		"start": "next start",
+		"lint": "next lint",
+		"check": "biome check app components lib",
+		"format": "biome check --write app components lib"
+	},
+	"dependencies": {
+		"@ag-ui/client": "^0.0.57",
+		"@copilotkit/react-core": "^1.61.1",
+		"@copilotkit/runtime": "^1.61.1",
+		"next": "^16.2.11",
+		"openai": "^6",
+		"react": "^19.0.0",
+		"react-dom": "^19.0.0",
+		"zod": "^3.24.0"
+	},
+	"devDependencies": {
+		"@biomejs/biome": "2.4.2",
+		"@types/node": "^22",
+		"@types/react": "^19",
+		"@types/react-dom": "^19",
+		"typescript": "^5"
+	}
+}

app/frontend/pnpm-workspace.yaml

@@ -0,0 +1,5 @@
+allowBuilds:
+  '@scarf/scarf': false
+  sharp: true
+overrides:
+  sharp: '>=0.35.0'

app/frontend/tsconfig.json

@@ -0,0 +1,41 @@
+{
+  "compilerOptions": {
+    "target": "ES2017",
+    "lib": [
+      "dom",
+      "dom.iterable",
+      "esnext"
+    ],
+    "allowJs": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "module": "esnext",
+    "moduleResolution": "bundler",
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "incremental": true,
+    "plugins": [
+      {
+        "name": "next"
+      }
+    ],
+    "paths": {
+      "@/*": [
+        "./*"
+      ]
+    }
+  },
+  "include": [
+    "next-env.d.ts",
+    "**/*.ts",
+    "**/*.tsx",
+    ".next/types/**/*.ts",
+    ".next/dev/types/**/*.ts"
+  ],
+  "exclude": [
+    "node_modules"
+  ]
+}

app/haiku.rag.yaml.example

@@ -0,0 +1,47 @@
+# haiku.rag configuration for the chat app
+# Copy to haiku.rag.yaml and customize as needed
+
+# The database. The compose files mount DB_VOLUME (default
+# ./data/haiku.rag.lancedb) at /data.
+lancedb:
+  databases:
+    haiku.rag: /data
+
+# QA model configuration
+qa:
+  model:
+    provider: ollama
+    name: gpt-oss
+    # For Anthropic:
+    # provider: anthropic
+    # name: claude-sonnet-4-20250514
+    # For OpenAI:
+    # provider: openai
+    # name: gpt-4o
+
+# Embedding configuration
+embeddings:
+  model:
+    provider: ollama
+    name: qwen3-embedding:4b
+    vector_dim: 2560
+    # For OpenAI:
+    # provider: openai
+    # name: text-embedding-3-small
+    # vector_dim: 1536
+
+# Optional reranking
+# reranking:
+#   model:
+#     provider: cohere
+#     name: rerank-v3.5
+
+# Search settings
+search:
+  limit: 5
+
+# Provider settings
+providers:
+  ollama:
+    # Use host.docker.internal to reach Ollama running on the host machine
+    base_url: http://host.docker.internal:11434

docker/Dockerfile

@@ -13,13 +13,13 @@ ENV UV_COMPILE_BYTECODE=1 \
 COPY pyproject.toml uv.lock ./
 COPY haiku_rag_slim/pyproject.toml haiku_rag_slim/README.md haiku_rag_slim/LICENSE haiku_rag_slim/
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv sync --frozen --no-install-project --no-dev
+    uv sync --frozen --no-install-project --no-dev --extra ingester
 
-# Install the project itself
+# Install the project itself (with the ingester extra so haiku-ingester is on PATH)
 COPY haiku_rag_slim haiku_rag_slim/
 COPY README.md LICENSE ./
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv sync --frozen --no-editable --no-dev
+    uv sync --frozen --no-editable --no-dev --extra ingester
 
 # Final layer
 FROM python:3.13-slim
@@ -33,8 +33,11 @@ ENV DEFAULT_DATA_DIR=/data
 
 ENV PATH="/app/.venv/bin:$PATH"
 
-# Expose port for MCP server
-EXPOSE 8001
+# Expose MCP server (8001) and ingester control plane (8765) ports.
+# docker-compose overrides this image's default command to run either the
+# MCP server (read-only) or the ingester service.
+EXPOSE 8001 8765
 
-# Run all services (monitoring, MCP)
-CMD ["python", "-m", "haiku.rag.cli", "serve", "--monitor", "--mcp", "--mcp-port", "8001", "--db", "/data/haiku.rag.lancedb"]
+# Default command: read-only MCP server. The companion ingester service is
+# launched via docker-compose against the same image.
+CMD ["haiku-rag", "--config", "/app/haiku.rag.yaml", "mcp", "--port", "8001"]

docker/Dockerfile.slim

@@ -0,0 +1,42 @@
+# syntax=docker/dockerfile:1
+FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS builder
+
+WORKDIR /app
+
+# Enable bytecode compilation for faster startup
+ENV UV_COMPILE_BYTECODE=1 \
+    UV_LINK_MODE=copy
+
+# Install dependencies into a venv
+# Install only haiku.rag-slim (no docling extra - use docling-serve instead)
+COPY pyproject.toml uv.lock ./
+COPY haiku_rag_slim/pyproject.toml haiku_rag_slim/README.md haiku_rag_slim/LICENSE haiku_rag_slim/
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-install-project --no-dev --extra ingester --package haiku.rag-slim
+
+# Install the project itself (with the ingester extra so haiku-ingester is on PATH)
+COPY haiku_rag_slim haiku_rag_slim/
+COPY README.md LICENSE ./
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-editable --no-dev --extra ingester --package haiku.rag-slim
+
+# Final layer
+FROM python:3.13-slim
+WORKDIR /app
+COPY --from=builder /app/.venv /app/.venv
+COPY --from=builder /app /app
+
+# Set default data directory
+RUN mkdir -p /data
+ENV DEFAULT_DATA_DIR=/data
+
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Expose MCP server (8001) and ingester control plane (8765) ports.
+# docker-compose overrides this image's default command to run either the
+# MCP server (read-only) or the ingester service.
+EXPOSE 8001 8765
+
+# Default command: read-only MCP server. The companion ingester service is
+# launched via docker-compose against the same image.
+CMD ["haiku-rag", "--config", "/app/haiku.rag.yaml", "mcp", "--port", "8001"]

docker/README.md

@@ -1,13 +1,21 @@
 # haiku.rag Docker Image
 
-Pre-built images are available at `ghcr.io/ggozad/haiku.rag` with all extras (voyageai, mxbai).
+The full haiku.rag Docker image includes all features and extras (docling, voyageai, cross-encoder). You can build it locally using the provided Dockerfile.
 
-## Using Pre-built Image
+## Building the Image
+
+Build the full image with all features:
 
 ```bash
-docker pull ghcr.io/ggozad/haiku.rag:latest
+docker build -f docker/Dockerfile -t haiku-rag .
 ```
 
+This creates an image with:
+- All document processing capabilities (Docling)
+- VoyageAI embeddings
+- MixedBread AI reranking
+- Full feature set
+
 ## Configuration
 
 Create a configuration file `haiku.rag.yaml`:
@@ -17,13 +25,15 @@ Create a configuration file `haiku.rag.yaml`:
 environment: production
 
 embeddings:
-  provider: ollama
-  model: nomic-embed-text
-  vector_dim: 768
+  model:
+    provider: ollama
+    name: nomic-embed-text
+    vector_dim: 768
 
 qa:
-  provider: ollama
-  model: qwen3
+  model:
+    provider: ollama
+    name: qwen3
 ```
 
 See [Configuration docs](https://ggozad.github.io/haiku.rag/configuration/) for all available options.
@@ -34,27 +44,50 @@ Mount your config file and data directory:
 
 ```bash
 docker run -p 8001:8001 \
-  -v $(pwd)/haiku.rag.yaml:/app/haiku.rag.yaml \
-  -v $(pwd)/data:/data \
-  ghcr.io/ggozad/haiku.rag:latest
+  -v /path/to/haiku.rag.yaml:/app/haiku.rag.yaml \
+  -v /path/to/data:/data \
+  haiku-rag
 ```
 
-The container will automatically use the mounted `haiku.rag.yaml` configuration file.
+For continuous ingestion of a watched directory, run `haiku-ingester` in a
+separate container against the same data volume:
+
+```bash
+docker run \
+  -v /path/to/haiku.rag.yaml:/app/haiku.rag.yaml \
+  -v /path/to/data:/data \
+  -v /path/to/docs:/docs \
+  -p 8765:8765 \
+  haiku-rag haiku-ingester --config /app/haiku.rag.yaml serve
+```
+
+Configure the watched directory in `haiku.rag.yaml` using the **container
+path**:
+
+```yaml
+ingester:
+  queue:
+    path: /data/ingester.db   # persist queue in the data volume
+  sources:
+    - type: fs
+      id: docs
+      root: /docs             # container path, not host path
+      delete_orphans: true
+```
+
+The MCP server running in the first container must be started with
+`--read-only` when an ingester is writing to the same database — LanceDB
+allows one writer and N readers per URI. See
+`examples/docker/docker-compose.yml` for a working two-service setup.
 
 For API keys (OpenAI, Anthropic, etc.), pass them as environment variables:
 
 ```bash
 docker run -p 8001:8001 \
-  -v $(pwd)/haiku.rag.yaml:/app/haiku.rag.yaml \
-  -v $(pwd)/data:/data \
+  -v /path/to/haiku.rag.yaml:/app/haiku.rag.yaml \
+  -v /path/to/data:/data \
   -e OPENAI_API_KEY=your-key-here \
-  ghcr.io/ggozad/haiku.rag:latest
-```
-
-## Building Locally
-
-```bash
-docker build -f docker/Dockerfile -t haiku-rag .
+  haiku-rag
 ```
 
 ## Docker Compose

docs/agents.md

@@ -1,282 +0,0 @@
-## Agents
-
-Three agentic flows are provided by haiku.rag:
-
-- Simple QA Agent — a focused question answering agent
-- Deep QA Agent — multi-agent question decomposition for complex questions
-- Research Multi‑Agent — a multi‑step, analyzable research workflow
-
-For an interactive example using Pydantic AI and AG-UI, see the [Interactive Research Assistant](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research) example ([demo video](https://vimeo.com/1128874386)). The demo uses a knowledge base containing haiku.rag's code and documentation.
-
-
-### Simple QA Agent
-
-The simple QA agent answers a single question using the knowledge base. It retrieves relevant chunks, optionally expands context around them, and asks the model to answer strictly based on that context.
-
-Key points:
-
-- Uses a single `search_documents` tool to fetch relevant chunks
-- Can be run with or without inline citations in the prompt (citations prefer
-  document titles when present, otherwise URIs)
-- Returns a plain string answer
-
-Python usage:
-
-```python
-from haiku.rag.client import HaikuRAG
-from haiku.rag.qa.agent import QuestionAnswerAgent
-
-async with HaikuRAG(path_to_db) as client:
-    # Choose a provider and model (see Configuration for env defaults)
-    agent = QuestionAnswerAgent(
-        client=client,
-        provider="openai",  # or "ollama", "vllm", etc.
-        model="gpt-4o-mini",
-        use_citations=False,  # set True to bias prompt towards citing sources
-    )
-
-    answer = await agent.answer("What is climate change?")
-    print(answer)
-```
-
-### Deep QA Agent
-
-Deep QA is a multi-agent system that decomposes complex questions into sub-questions, answers them in batches, evaluates sufficiency, and iterates if needed before synthesizing a final answer. It's lighter than the full research workflow but more powerful than the simple QA agent.
-
-```mermaid
----
-title: Deep QA graph
----
-stateDiagram-v2
-  [*] --> plan
-  plan --> get_batch
-  get_batch --> search_one: Has questions (map)
-  get_batch --> synthesize: No questions
-  search_one --> collect_answers
-  collect_answers --> decide
-  decide --> get_batch: Continue QA
-  decide --> synthesize: Done with QA
-  synthesize --> [*]
-```
-
-Key nodes:
-
-- **plan**: Decomposes the question into focused sub-questions using a presearch tool
-- **get_batch**: Retrieves remaining sub-questions for the current iteration
-- **search_one**: Answers a single sub-question using the knowledge base (mapped in parallel)
-- **collect_answers**: Aggregates search results from parallel executions
-- **decide**: Evaluates if sufficient information has been gathered or if more iterations are needed
-- **synthesize**: Generates the final comprehensive answer from all gathered information
-
-Key differences from Research:
-
-- **Simpler evaluation**: Uses sufficiency check (not confidence + insight analysis)
-- **Direct answers**: Returns just the answer (not a full research report)
-- **Question-focused**: Optimized for answering specific questions, not open-ended research
-- **Supports citations**: Can include inline source citations like `[document.md]`
-- **Configurable iterations**: Control max_iterations (default: 2) and max_concurrency (default: 1)
-
-Note on parallel execution:
-- The `search_one` node is mapped over all questions in a batch
-- Parallelism is controlled via `max_concurrency`
-- All questions in an iteration are processed before evaluation
-
-CLI usage:
-
-```bash
-# Deep QA without citations
-haiku-rag ask "What are the main features of haiku.rag?" --deep
-
-# Deep QA with citations
-haiku-rag ask "What are the main features of haiku.rag?" --deep --cite
-```
-
-Python usage:
-
-```python
-from haiku.rag.client import HaikuRAG
-from haiku.rag.config import Config
-from haiku.rag.qa.deep.dependencies import DeepQAContext
-from haiku.rag.qa.deep.graph import build_deep_qa_graph
-from haiku.rag.qa.deep.state import DeepQADeps, DeepQAState
-
-async with HaikuRAG(path_to_db) as client:
-    # Use global config (recommended)
-    graph = build_deep_qa_graph(config=Config)
-    context = DeepQAContext(
-        original_question="What are the main features of haiku.rag?",
-        use_citations=True
-    )
-    state = DeepQAState.from_config(context=context, config=Config)
-    deps = DeepQADeps(client=client)
-
-    result = await graph.run(
-        state=state,
-        deps=deps
-    )
-
-    print(result.answer)
-    print(result.sources)
-```
-
-Alternative usage with custom config:
-
-```python
-# Create a custom config with different settings
-from haiku.rag.config.models import AppConfig, QAConfig
-
-custom_config = AppConfig(
-    qa=QAConfig(
-        provider="openai",
-        model="gpt-4o-mini",
-        max_sub_questions=5,
-        max_iterations=3,
-        max_concurrency=2,
-    )
-)
-
-graph = build_deep_qa_graph(config=custom_config)
-context = DeepQAContext(
-    original_question="What are the main features of haiku.rag?",
-    use_citations=True
-)
-state = DeepQAState.from_config(context=context, config=custom_config)
-deps = DeepQADeps(client=client)
-
-result = await graph.run(state=state, deps=deps)
-```
-
-### Research Graph
-
-The research workflow is implemented as a typed pydantic‑graph. It plans, searches (in parallel batches), evaluates, and synthesizes into a final report — with clear stop conditions and shared state.
-
-```mermaid
----
-title: Research graph
----
-stateDiagram-v2
-  [*] --> plan
-  plan --> get_batch
-  get_batch --> search_one: Has questions (map)
-  get_batch --> synthesize: No questions
-  search_one --> collect_answers
-  collect_answers --> analyze_insights
-  analyze_insights --> decide
-  decide --> get_batch: Continue research
-  decide --> synthesize: Done researching
-  synthesize --> [*]
-```
-
-Key nodes:
-
-- **plan**: Builds up to 3 standalone sub‑questions (uses an internal presearch tool)
-- **get_batch**: Retrieves remaining sub‑questions for the current iteration
-- **search_one**: Answers a single sub‑question using the KB with minimal, verbatim context (mapped in parallel)
-- **collect_answers**: Aggregates search results from parallel executions
-- **analyze_insights**: Synthesizes fresh insights, updates gaps, and suggests new sub-questions
-- **decide**: Checks sufficiency/confidence thresholds and determines whether to continue research
-- **synthesize**: Generates a final structured research report
-
-Primary models:
-
-- `SearchAnswer` — one per sub‑question (query, answer, context, sources)
-- `InsightRecord` / `GapRecord` — structured tracking of findings and open issues
-- `InsightAnalysis` — output of the analysis stage (insights, gaps, commentary)
-- `EvaluationResult` — insights, new questions, sufficiency, confidence
-- `ResearchReport` — final report (title, executive summary, findings, conclusions, …)
-
-Note on parallel execution:
-- The `search_one` node is mapped over all questions in a batch
-- Parallelism is controlled via `max_concurrency`
-- Analysis and decision nodes process results after each batch completes
-
-CLI usage:
-
-```bash
-# Basic usage (uses config from file or defaults)
-haiku-rag research "How does haiku.rag organize and query documents?" --verbose
-
-# With custom config file
-haiku-rag --config my-research-config.yaml research "How does haiku.rag organize and query documents?" --verbose
-```
-
-Python usage (blocking result):
-
-```python
-from haiku.rag.client import HaikuRAG
-from haiku.rag.config import Config
-from haiku.rag.research.dependencies import ResearchContext
-from haiku.rag.research.graph import build_research_graph
-from haiku.rag.research.state import ResearchDeps, ResearchState
-
-async with HaikuRAG(path_to_db) as client:
-    # Use global config (recommended)
-    graph = build_research_graph(config=Config)
-    question = "What are the main drivers and trends of global temperature anomalies since 1990?"
-    context = ResearchContext(original_question=question)
-    state = ResearchState.from_config(context=context, config=Config)
-    deps = ResearchDeps(client=client)
-
-    result = await graph.run(
-        state=state,
-        deps=deps,
-    )
-
-    report = result
-    print(report.title)
-    print(report.executive_summary)
-```
-
-Alternative usage with custom config:
-
-```python
-from haiku.rag.config.models import AppConfig, ResearchConfig
-
-custom_config = AppConfig(
-    research=ResearchConfig(
-        provider="openai",
-        model="gpt-4o-mini",
-        max_iterations=5,
-        confidence_threshold=0.85,
-        max_concurrency=3,
-    )
-)
-
-graph = build_research_graph(config=custom_config)
-context = ResearchContext(original_question=question)
-state = ResearchState.from_config(context=context, config=custom_config)
-deps = ResearchDeps(client=client)
-
-result = await graph.run(state=state, deps=deps)
-```
-
-Python usage (streamed events):
-
-```python
-from haiku.rag.client import HaikuRAG
-from haiku.rag.config import Config
-from haiku.rag.research.dependencies import ResearchContext
-from haiku.rag.research.graph import build_research_graph
-from haiku.rag.research.state import ResearchDeps, ResearchState
-from haiku.rag.research.stream import stream_research_graph
-
-async with HaikuRAG(path_to_db) as client:
-    graph = build_research_graph(config=Config)
-    question = "What are the main drivers and trends of global temperature anomalies since 1990?"
-    context = ResearchContext(original_question=question)
-    state = ResearchState.from_config(context=context, config=Config)
-    deps = ResearchDeps(client=client)
-
-    async for event in stream_research_graph(
-        graph,
-        state,
-        deps,
-    ):
-        if event.type == "log":
-            iteration = event.state.iterations if event.state else state.iterations
-            print(f"[{iteration}] {event.message}")
-        elif event.type == "report":
-            print("\nResearch complete!\n")
-            print(event.report.title)
-            print(event.report.executive_summary)
-```

docs/apps.md

@@ -0,0 +1,87 @@
+# Web application
+
+A browser-based reference implementation of conversational RAG, built on a Starlette backend with pydantic-ai's `AGUIAdapter` and a Next.js / CopilotKit frontend. It lives in the `app/` directory of the haiku.rag repository.
+
+This is a starting point for your own deployments, not the canonical haiku.rag UX. For the day-to-day terminal experience see [Chat](chat.md).
+
+!!! warning "No authentication"
+    An illustrative example meant as a starting point, with no authentication. The compose files bind the backend to `127.0.0.1`; don't expose it to an untrusted network.
+
+## Features
+
+- Streaming chat with real-time tool execution visibility.
+- Expandable citations with source documents, pages, and headings.
+- Visual grounding to view chunk source locations in documents.
+- Document filter to restrict searches to selected documents.
+- Session state view for inspecting citations and search results.
+
+## Quick start
+
+```bash
+cd app
+docker compose -f docker-compose.dev.yml up -d --build
+```
+
+- Frontend: `http://localhost:3000`
+- Backend: `http://localhost:8001`
+
+## Architecture
+
+- **Backend**: Starlette server with pydantic-ai `AGUIAdapter`.
+- **Frontend**: Next.js with CopilotKit.
+- **Protocol**: AG-UI for streaming chat.
+
+## Configuration
+
+Create a `.env` file in the `app/` directory:
+
+```bash
+# API Keys (at least one required)
+ANTHROPIC_API_KEY=your-anthropic-key
+OPENAI_API_KEY=your-openai-key
+
+# Host path of the LanceDB database, mounted at /data
+DB_VOLUME=/path/to/your/haiku.rag.lancedb
+
+# Optional: Ollama base URL (if using local models)
+OLLAMA_BASE_URL=http://localhost:11434
+
+# Optional: Logfire for observability
+LOGFIRE_TOKEN=your-logfire-token
+```
+
+The mounted `haiku.rag.yaml` places the database at `/data` and configures the models; the compose files point `HAIKU_RAG_CONFIG_PATH` at it:
+
+```yaml
+# app/haiku.rag.yaml
+lancedb:
+  databases:
+    haiku.rag: /data
+
+qa:
+  model:
+    provider: anthropic
+    name: claude-sonnet-4-20250514
+```
+
+Outside compose, the backend loads its configuration like the CLI: `HAIKU_RAG_CONFIG_PATH`, then `./haiku.rag.yaml`, then the platform directory.
+
+## API endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/v1/chat/stream` | POST | AG-UI chat streaming |
+| `/api/documents` | GET | List all documents |
+| `/api/info` | GET | Database statistics |
+| `/api/visualize/{chunk_id}` | GET | Visual grounding images (base64) |
+| `/health` | GET | Health check |
+
+## Development
+
+The backend reloads automatically on file changes. For frontend changes:
+
+```bash
+docker compose -f docker-compose.dev.yml up -d --build frontend
+```
+
+If `LOGFIRE_TOKEN` is set, LLM calls are traced and available in the Logfire dashboard.

docs/benchmarks.md

@@ -1,97 +1,291 @@
 # Benchmarks
 
-We use the [repliqa](https://huggingface.co/datasets/ServiceNow/repliqa) dataset for the evaluation of `haiku.rag`.
+We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, HotpotQA, FRAMES, and MTRAG are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the RAG and analysis capabilities.
 
-You can perform your own evaluations with the Typer CLI in
-`evaluations/evaluations/benchmark.py`, for example `python -m evaluations.benchmark repliqa`.
-The evaluation flow is orchestrated with
-[`pydantic-evals`](https://github.com/pydantic/pydantic-ai/tree/main/libs/pydantic-evals),
-which we leverage for dataset management, scoring, and report generation.
+## Current results
 
-## Configuration
+Numbers below were measured on a recent `haiku.rag` version. Most rows were judged by `Qwen3.6-35B-A3B-NVFP4`; the `Qwen3.8-27B` rows were judged by the currently pinned `qwen3.8`, as their footnote states. Rows are not re-judged when the pinned judge changes, so compare rows judged by the same judge and treat cross-judge differences as unmeasured.
 
-The benchmark script accepts a `--config` option to specify a custom `haiku.rag.yaml` configuration file:
+No benchmark database carries a vector index, so every number below reflects exact brute-force kNN rather than approximate search. A vector index is never built automatically. `haiku-rag create-index` builds one, and `haiku-rag doctor` reports whether a database has it. For the measured effect of indexing on retrieval, see [Vector Indexing](configuration/storage.md#vector-indexing).
+
+### OpenRAG Bench (ORB)
+
+[OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval and reasoning over visual content like figures, charts, and diagrams. Each query maps to one relevant document.
+
+Two approaches are benchmarked separately:
+
+- **Multimodal embedder** (`Qwen/Qwen3-VL-Embedding-8B`, served via vLLM): picture bytes and text live in a shared vector space, no VLM is run at ingest.
+- **Text embedder + VLM picture descriptions** (`qwen3-embedding:4b` + `ollama/ministral-3`): pictures are described at ingest and the descriptions are woven into chunk text. Retrieval runs over text only. See [Picture handling configuration](configuration/processing.md#picture-handling).
+
+#### Multimodal embedder
+
+##### Retrieval (MAP)
+
+| Embedding Model                          | Reranker                                             | Cases | MAP    |
+|------------------------------------------|------------------------------------------------------|------:|-------:|
+| `Qwen/Qwen3-VL-Embedding-8B`             | none                                                 |  3045 | 0.9774 |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | none                                                 |  3045 | 0.9798 |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `nvidia/llama-nemotron-rerank-vl-1b-v2` (multimodal) |  3045 | 0.9913 |
+
+*The nemotron row without a reranker is measured on this release. The reranked row uses `reranking.multimodal: true`: picture chunks reach the vision reranker as images alongside their description text, measured on haiku.rag main post-v0.67.3.*
+
+##### QA accuracy + citation retrieval
+
+| Embedding Model                          | Target          | Capability model                       | Cases | QA accuracy | Mean `cited_map` |
+|------------------------------------------|-----------------|-----------------------------------|------:|-------------|------------------|
+| `Qwen/Qwen3-VL-Embedding-8B`             | `rag-capability`     | `vllm:Gemma-4-26B-A4B-NVFP4`      |  1409 | 0.89        | —                |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `rag-capability`     | `vllm:Gemma-4-26B-A4B-NVFP4`      |  3039 | 0.9263      | 0.9761           |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-capability`| `vllm:Gemma-4-26B-A4B-NVFP4`      |  3040 | 0.9362      | 0.9343           |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-capability`| `vllm:Qwen3.6-35B-A3B-NVFP4`      |  3045 | 0.95        | 0.93             |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `rag-capability`     | `vllm:Muse-Glimmer-30B-NVFP4`     |  3045 | 0.9494      | 0.9771           |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-capability`| `vllm:Muse-Glimmer-30B-NVFP4`     |  3017 | 0.9718      | 0.9837           |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `rag-capability`     | `vllm:Qwen3.8-27B-NVFP4`          |  3045 | 0.9514      | 0.9817           |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-capability`| `vllm:Qwen3.8-27B-NVFP4`          |  3042 | 0.9629      | 0.9835           |
+
+*The `Muse-Glimmer-30B` rows run at `chat_template_kwargs.reasoning_strength: high`, no reranker, same judge.*
+
+*The `Qwen3.8-27B` rows run at `chat_template_kwargs.reasoning_effort: low`, no reranker, and are **judged by `Qwen3.8-27B` itself** — the pinned judge, and the same model that produced the answers. A 120-case cross-check by an independent judge agreed on 95% and was never stricter, but the incumbent `Qwen3.6` judge is no longer hosted, so the older rows cannot be re-judged for a like-for-like comparison. `rag-capability` cites on 99.34% of cases with 1.09 citations each; `analysis-capability` on 99.70% with 1.12, at 0.13 code executions per case. Case counts exclude 0 and 3 provider errors respectively.*
+
+*Both nemotron `Gemma-4` rows are measured on this release, no reranker, judged by `vllm:Qwen3.6-35B-A3B-NVFP4` with thinking on, and exclude the cases that errored (6 of 3045 for `rag-capability`, 5 for `analysis-capability`). The `rag-capability` row cites at 99.64% with a mean of 1.08 citations per case, at a median 4.7s per case against 5.0s for `analysis-capability`. Citation coverage is what moved on this release: 4.9% of analysis cases register no citation, against 26.3% before, at unchanged searches and code executions per case. The remaining rows are from haiku.rag v0.52.0, where Qwen3-VL covered 1409 / 3045 cases.*
+
+#### Text embedder + VLM picture descriptions
+
+##### Retrieval (MAP)
+
+| Embedding Model                          | VLM                  | Reranker               | Cases | MAP    |
+|------------------------------------------|----------------------|------------------------|------:|-------:|
+| `qwen3-embedding:4b`                     | Ollama / ministral-3 | `mxbai-rerank-base-v2` |  3045 | 0.9834 |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | Ollama / ministral-3 | `mxbai-rerank-base-v2` |  3045 | 0.9863 |
+
+*Measured on haiku.rag v0.50.0.*
+
+##### QA accuracy + citation retrieval
+
+| Embedding Model                          | VLM                  | Capability model                  | Cases | QA accuracy | Mean `cited_map` |
+|------------------------------------------|----------------------|------------------------------|------:|-------------|------------------|
+| `qwen3-embedding:4b`                     | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` |  3045 | 0.92        | 0.80             |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` |  2836 | 0.96        | 0.81             |
+
+*Measured on haiku.rag v0.50.0 with `mxbai-rerank-base-v2`, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. Nemotron covered 2836 / 3045 cases.*
+
+### T²-RAGBench (FinQA)
+
+[T²-RAGBench](https://huggingface.co/datasets/G4KMU/t2-ragbench) reformulates financial-report QA into context-independent questions with short numeric answers and a 1:1 gold document mapping. The FinQA subset is 2,789 single-page PDFs / 8,281 questions, ingested via docling. Unlike the other datasets, QA is scored deterministically with `NumberMatchEvaluator` (relative tolerance 0.01) instead of an LLM judge, so QA accuracy here is exact numeric match rather than a judged fraction.
+
+##### QA accuracy + citation retrieval
+
+| Embedding Model      | Reranker               | Target           | Capability model                  | Cases | QA accuracy | Mean `cited_map` |
+|----------------------|------------------------|------------------|------------------------------|------:|-------------|------------------|
+| `qwen3-embedding:4b` | `mxbai-rerank-base-v2` | `analysis-capability` | `vllm:Qwen3.6-35B-A3B-NVFP4` |  7939 | 0.77        | 0.78             |
+
+*Measured on haiku.rag v0.55.0, deterministic Number-Match scoring (ε=0.01), 2560-dim `qwen3-embedding:4b` (vLLM) with `mxbai-rerank-base-v2`. 341 / 8281 cases excluded as nulls (analysis spirals from the request limit and in-generation loops). Accuracy and `cited_map` are over the 7939 scored cases. Mean 16.0s/case.*
+
+### HotpotQA
+
+[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is multi-hop question answering over Wikipedia: each question requires combining facts from two supporting paragraphs, with distractor paragraphs in the corpus. We use the distractor validation split: 7,405 questions over ~66k unique paragraphs, each question mapping to two gold documents.
+
+##### Retrieval (MAP)
+
+| Embedding Model      | Reranker            | Cases | MAP    |
+|----------------------|---------------------|------:|-------:|
+| `qwen3-embedding:4b` | `Qwen3-Reranker-4B` |  7405 | 0.8202 |
+| `qwen3-embedding:4b` | none                |  7405 | 0.6995 |
+
+The reranker's contribution is larger here than on the single-doc datasets: hybrid search usually surfaces the first-hop document at rank 1, while the second-hop document often needs the reranker to climb into the result window.
+
+##### QA accuracy + citation retrieval
+
+| Skill model                  | Reranker            | QA accuracy | Mean `cited_map` |
+|------------------------------|---------------------|-------------|------------------|
+| `vllm:Gemma-4-26B-A4B-NVFP4` | `Qwen3-Reranker-4B` | 0.85        | 0.80             |
+| `vllm:Gemma-4-26B-A4B-NVFP4` | none                | 0.83        | 0.75             |
+
+*Measured on haiku.rag v0.66.0 with `qwen3-embedding:4b` (vLLM, dim 2560), judged by `vllm:Qwen3.6-35B-A3B-NVFP4`, 7,405 cases. The reranker lifts QA accuracy +2.7pts and `cited_map` +4.6pts. Without a reranker, `cited_map` (0.75) still exceeds the no-reranker retrieval MAP (0.70): the skill reformulates queries across search calls, partially recovering second-hop documents that a single query misses.*
+
+### FRAMES
+
+[FRAMES](https://huggingface.co/datasets/google/frames-benchmark) is Google's multi-hop QA benchmark: 824 questions, each grounded in 2–23 Wikipedia articles, exercising temporal, numerical, and tabular reasoning across documents. We evaluate 822 questions (2 excluded: a linked article was deleted from Wikipedia) over a fixed corpus of the 2,521 linked articles fetched at current revision. There is no official FRAMES evaluation setup; our protocol — fixed corpus, agentic retrieval, judged accuracy — corresponds to the paper's *multi-step retrieval* setting, where [the paper](https://arxiv.org/abs/2409.12941) reports 0.66 with Gemini-Pro-1.5 (0.729 in its oracle setting, with gold articles provided). Answers were authored against ~2024 revisions and may have drifted with article content.
+
+##### Retrieval (MAP)
+
+| Embedding Model      | Reranker            | Cases | MAP    |
+|----------------------|---------------------|------:|-------:|
+| `qwen3-embedding:4b` | `Qwen3-Reranker-4B` |   822 | 0.5631 |
+
+*Single-query retrieval is capped by FRAMES' indirection: in the zero-MAP queries the gold article's subject is never named in the question ("the year the Titanic sank" → `1912_Summer_Olympics`). The agentic targets recover these through iterative search, passing 55% of the very cases single-shot retrieval scores zero on.*
+
+##### QA accuracy + citation retrieval
+
+| Capability model | Target | QA accuracy | Mean `cited_map` |
+|------------------|--------|-------------|------------------|
+| `vllm:Muse-Glimmer-30B-NVFP4` | `analysis-capability` | 0.7506 | 0.5852 |
+| `vllm:Qwen3.8-27B-NVFP4` | `analysis-capability` | 0.8095 | 0.6847 |
+
+*Both rows use `qwen3-embedding:4b` (vLLM, dim 2560), `Qwen3-Reranker-4B`, and are judged by `Qwen3.8-27B`. QA accuracy is over judged cases; counting unanswered cases as failures, the floors are 0.7397 (`Muse-Glimmer`, 1.5% lost to provider errors) and 0.7701 (`Qwen3.8`, 4.87% lost to answers truncated at `max_tokens: 16384`). The `Qwen3.8` row is self-judged — a 100-case paired cross-judge (Glimmer as judge, difference-in-differences) measured the self-preference at +1.0pp with 99% judge agreement. Cite rates are 90.1% (`Muse-Glimmer`) and 99.3% (`Qwen3.8`); `cited_map` is structurally capped below 1 on FRAMES because gold sets span 2–23 articles while answering typically uses a subset. `Qwen3.8` reaches its score on substantially less tool traffic than `Muse-Glimmer` (4.0 searches and 3.9 code executions per case vs 7.0 and 9.5, measured identically from tool spans).*
+
+### MTRAG (ClapNQ)
+
+[MTRAG](https://github.com/IBM/mt-rag-benchmark) is IBM's multi-turn RAG benchmark (TACL 2025, SemEval-2026 Task 8): human-authored conversations with per-turn answerability labels and binary relevance judgments. We evaluate the ClapNQ (Wikipedia) domain: 183,408 passages, 29 conversations, 224 turns, 208 retrieval queries.
+
+Four dataset keys share one database. `mtrag_clapnq` retrieves with the raw last user turn and runs QA by replaying each task's reference conversation prefix as message history. `mtrag_clapnq_rewrite` retrieves with the human standalone rewrites. `mtrag_clapnq_live` replays whole conversations through a single capability session, carrying the model's own answers, tool history and capability state across turns, with `EvidenceCompactionCapability` registered. `mtrag_clapnq_live_uncompacted` is the same replay without compaction, isolating what compaction contributes. This is the only multi-turn evaluation, so it is the only one where compaction acts at all.
+
+##### Retrieval (Recall@k / nDCG@k)
+
+Directly comparable with [IBM's published results](https://github.com/IBM/mt-rag-benchmark/tree/main/mtrag-human/retrieval_tasks). Elser is IBM's strongest reported retriever.
+
+| Retriever | Queries | R@5 | R@10 | nDCG@5 | nDCG@10 |
+|-----------|---------|----:|-----:|-------:|--------:|
+| Elser (IBM) | lastturn | 0.49 | 0.58 | 0.45 | 0.49 |
+| `haiku.rag` | lastturn | 0.501 | 0.600 | 0.455 | 0.497 |
+| Elser (IBM) | rewrite | 0.52 | 0.64 | 0.48 | 0.54 |
+| `haiku.rag` | rewrite | 0.548 | 0.668 | 0.503 | 0.556 |
+
+##### QA accuracy + citation retrieval
+
+| Mode | Capability model | Turns | QA accuracy | Mean `cited_map` |
+|------|------------------|------:|-------------|------------------|
+| Gold-prefix (`mtrag_clapnq`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224 | 0.76 | 0.35 |
+| Live compacted (`mtrag_clapnq_live`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224/224 scored | 0.83 micro / 0.84 macro | 0.42 |
+| Live uncompacted (`mtrag_clapnq_live_uncompacted`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224/224 scored | 0.78 micro / 0.79 macro | 0.42 |
+
+*Measured on haiku.rag v0.74.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `Qwen3-Reranker-4B`, stock capability instructions with `reasoning_strength: high`, judged by the pinned `vllm:Qwen3.6-35B-A3B-NVFP4` (temperature 0.6, thinking). QA numbers are internal (our judge and rubric) and not comparable with IBM's published generation metrics. Gold-prefix and live rates answer different judge questions and are not comparable with each other. The dataset is text-only (ClapNQ passages), so it exercises no multimodal paths.*
+
+The two live arms replay the same 29 conversations (224 turns) and differ only in registering `EvidenceCompactionCapability`, so they are compared as paired observations:
+
+- Input tokens per model request, computed as total input tokens divided by model requests across the whole arm: 7,461 compacted (5,207,627 tokens over 698 requests) vs 13,539 uncompacted (9,707,530 over 717 requests). The uncompacted arm used 1.81x as many tokens per request, a 44.9% reduction under compaction.
+- Answer pass rate: 185/224 vs 175/224 turns. Of the 18 turns where the arms disagree, 14 pass only compacted and 4 only uncompacted. McNemar exact two-sided p = 0.031. The paired difference is +4.5pp with a Wald 95% CI of +0.8 to +8.1pp, so the honest claim is an improvement of roughly 1 to 8 points, not the point estimate.
+- Citation MAP, macro-averaged over conversations with 208 of 224 turns eligible (turns with gold passages) in each arm: 0.4174 compacted vs 0.4230 uncompacted. The gold-prefix 0.35 is over 208 of 224 eligible cases.
+- Refusal precision and recall against the answerability labels (16 UNANSWERABLE turns per arm): compacted 0.33 precision and 0.44 recall (21 refusals), uncompacted 0.23 and 0.31 (22 refusals). Gold-prefix: 0.24 and 0.44 (29 refusals).
+
+## Methodology
+
+### Retrieval Metrics
+
+**Mean Average Precision (MAP)** scores ranked retrieval results against the gold `expected_uris`.
+
+- For each relevant document at position k, calculate precision@k = (relevant docs in top k) / k
+- Average Precision (AP) = sum of these precision values / total relevant documents
+- MAP is the mean of AP scores across all queries
+- Range: 0 to 1. Rewards ranking relevant documents higher
+- For single-doc queries this collapses to `1/rank` (i.e. reciprocal rank)
+
+### QA Accuracy
+
+`pydantic-evals` coordinates an LLM judge to determine whether the capability's answer is correct. The default judge is `ollama:qwen3.8`, pinned so changes to the capability model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions.
+
+A dataset that brings its own deterministic evaluator is scored by that evaluator instead, and no judge runs. T²-RAGBench is the only such dataset today, scored by `NumberMatchEvaluator`.
+
+`qwen3.8` replaced `qwen3.6` after a 120-case calibration on ORB, stratified 60 pass / 60 fail: agreement 0.950, Cohen's κ 0.900, and in all 6 disagreements it matched or beat `qwen3.6` (4 were `qwen3.6` failing answers that were equivalent in different notation). It emits no reasoning content, so it avoids the thinking spirals that made `qwen3.6` exceed its output budget and drop verdicts. `reasoning_effort` changes its verdicts in 1 case per 120, so the cheaper `low` is pinned.
+
+Before that, we picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs.
+
+### Citation Retrieval
+
+Alongside QA accuracy, a second metric scores the URIs the capability registered via the `cite` tool against each dataset's gold `expected_uris`, using the same MAP math as raw retrieval. The score key is `cited_map`. Console output also includes the cite rate (% of cases with at least one citation) and the mean number of citations per case.
+
+This is computed alongside QA accuracy from the same capability run, no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the capability grounded its answer on it.
+
+## Running Evaluations
+
+You can run evaluations with the `evaluations` CLI:
 
 ```bash
-python -m evaluations.benchmark repliqa --config /path/to/haiku.rag.yaml
+evaluations run hotpotqa
+evaluations run orb_text
 ```
 
-If no config file is specified, the script will search for a config file in the standard locations:
-1. `./haiku.rag.yaml` (current directory)
-2. User config directory
-3. Falls back to default configuration
+The evaluation flow is orchestrated with [`pydantic-evals`](https://github.com/pydantic/pydantic-ai/tree/main/libs/pydantic-evals), which we leverage for dataset management, scoring, and report generation.
 
-You can also use command-line options:
+### Pre-built Databases
+
+Building evaluation databases from scratch can take a long time, especially for large datasets like OpenRAG Bench. Pre-built databases are available on HuggingFace:
+
+```bash
+# Download a specific dataset
+evaluations download hotpotqa
+
+# Download all datasets
+evaluations download all
+
+# Force re-download (overwrite existing)
+evaluations download hotpotqa --force
+```
+
+Active datasets:
+
+| Dataset | Size |
+|---------|------|
+| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
+| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
+| `orb_multimodal_nemotron` — OpenRAG Bench, multimodal embedder (`nvidia/llama-nemotron-embed-vl-1b-v2`), the embedder behind the published headline results | ~16 GB |
+| `t2_finqa` — T²-RAGBench (FinQA) financial QA, text embedder (`qwen3-embedding:4b`); scored by exact numeric match, run with `--target analysis-capability` | ~2 GB |
+| `hotpotqa` — HotpotQA multi-hop QA over Wikipedia paragraphs, text embedder (`qwen3-embedding:4b`) | ~1.5 GB |
+| `mtrag_clapnq` — MTRAG multi-turn RAG, ClapNQ (Wikipedia) passages, text embedder (`qwen3-embedding:4b`); also serves the `mtrag_clapnq_rewrite`, `mtrag_clapnq_live` and `mtrag_clapnq_live_uncompacted` keys | ~2.8 GB |
+
+After downloading, run benchmarks with `--skip-db`. Each database is built with a specific embedder, so pass its reference config from `evaluations/configs/` (a database only opens against a config whose embedder matches):
+
+```bash
+evaluations run orb_multimodal_nemotron --skip-db --config configs/orb_multimodal_nemotron.yaml
+```
+
+The configs use `vllm` as the model host. Point `base_url` at your own OpenAI-compatible endpoints to reproduce the numbers.
+
+### Configuration
+
+The benchmark script accepts several options:
+
+```bash
+evaluations run hotpotqa --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
+```
+
+**Options:**
+
+- `--config PATH` - Specify a custom `haiku.rag.yaml` configuration file
+- `--db PATH` - Override the database path (default: platform-specific user data directory)
 - `--skip-db` - Skip updating the evaluation database
 - `--skip-retrieval` - Skip retrieval benchmark
 - `--skip-qa` - Skip QA benchmark
-- `--qa-limit N` - Limit number of QA cases to evaluate
+- `--limit N` - Limit number of test cases
+- `--name NAME` - Override the evaluation name
+- `--target {rag-capability,analysis-capability}` - Choose which [capability](capabilities/index.md) to benchmark end-to-end (default: `rag-capability`). The target names remain stable dataset identifiers.
+- `--capability-model PROVIDER:NAME` - Override the capability model independently from the judge (default: `config.qa.model`, or `config.analysis.model` when set for `--target analysis-capability`).
+- `--filter CLAUSE` / `-f CLAUSE` - Restrict every benchmark search to a subset of the database (see [Restricting the corpus](#restricting-the-corpus)).
 
-## Recall
+If no config file is specified, the script searches standard locations: `./haiku.rag.yaml`, user config directory, then falls back to defaults.
 
-In order to calculate recall, we load the `News Stories` from `repliqa_3` (1035 documents) and index them. Subsequently, we run a search over the `question` field for each row of the dataset and check whether we match the document that answers the question. Questions for which the answer cannot be found in the documents are ignored.
+To pin the LLM judge in YAML (rather than the default `ollama:qwen3.8`). These are the recommended settings:
 
+```yaml
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://localhost:8000/v1   # optional, for OpenAI-compatible servers (vLLM, LM Studio, etc.)
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low   # qwen3.8: low | medium | xhigh (default)
+```
 
-The recall obtained is ~0.79 for matching in the top result, raising to ~0.91 for the top 3 results with the "bare" default settings (Ollama `qwen3`, `mxbai-embed-large` embeddings, no reranking).
+### Restricting the corpus
 
-| Embedding Model                       | Document in top 1 | Document in top 3 | Reranker               |
-|---------------------------------------|-------------------|-------------------|------------------------|
-| Ollama / `qwen3-embedding`            | 0.81              | 0.95              | None                   |
-| Ollama / `qwen3-embedding`            | 0.91              | 0.98              | `mxbai-rerank-base-v2` |
-| Ollama / `mxbai-embed-large`          | 0.79              | 0.91              | None                   |
-| Ollama / `mxbai-embed-large`          | 0.90              | 0.95              | `mxbai-rerank-base-v2` |
-| Ollama / `nomic-embed-text-v1.5`      | 0.74              | 0.90              | None                   |
+When a database holds documents from several corpora — only some of which a dataset's questions are drawn from — `--filter` restricts every benchmark search to a subset. It takes the same SQL `WHERE` clause as `haiku-rag search --filter`, over document columns (`id`, `uri`, `title`, `created_at`, `updated_at`, `metadata`). Each dataset writes its own URIs: `orb_text` uses bare arXiv ids such as `2407.01528v3`, `hotpotqa` uses page titles.
 
-## Question/Answer evaluation
+```bash
+evaluations run orb_text --skip-db --config haiku.rag.s3.yaml \
+  --filter "uri LIKE '2407%'"
+```
 
-Again using the same dataset, we use a QA agent to answer the question.
-`pydantic-evals` runs each case and coordinates an LLM judge (Ollama `qwen3`) to
-determine whether the answer is correct. The obtained accuracy is as follows:
+If the corpora are distinguished by a tag rather than by URI, attach it at ingest time as document metadata and match it with `LIKE`. `metadata` is stored as a `json.dumps` string, so there is no JSON subfield access — match the serialized key/value, including the space after the colon:
 
-| Embedding Model                    | QA Model                          | Accuracy  | Reranker               |
-|------------------------------------|-----------------------------------|-----------|------------------------|
-| Ollama / `qwen3-embedding. `       | Ollama / `gpt-oss`                | 0.93      | None                   |
-| Ollama / `mxbai-embed-large`       | Ollama / `qwen3`                  | 0.85      | None                   |
-| Ollama / `mxbai-embed-large`       | Ollama / `qwen3`                  | 0.87      | `mxbai-rerank-base-v2` |
-| Ollama / `mxbai-embed-large`       | Ollama / `qwen3:0.6b`             | 0.28      | None                   |
+```bash
+evaluations run orb_text --skip-db --filter "metadata LIKE '%\"corpus\": \"orb_text\"%'"
+```
 
-Note the significant degradation when very small models are used such as `qwen3:0.6b`.
+The clause applies to both benchmark phases — the retrieval benchmark's searches and every search the capability runs during QA — so the two score the same subset. It is recorded as `document_filter` in the run's experiment metadata, so a filtered run is never mistaken for an unfiltered one when comparing results.
 
-## Wix dataset
-
-We also track retrieval performance on [WixQA](https://huggingface.co/datasets/Wix/WixQA),
-a dataset of real customer support questions paired with curated answers from
-Wix. The benchmark follows the evaluation protocol described in the
-[WixQA paper](https://arxiv.org/abs/2505.08643) and gives us a view into how the
-system handles conversational, product-specific support queries.
-
-For retrieval evaluation, we index the reference answer passages shipped with the dataset and
-run retrieval against each user question. Each sample supplies one or more
-relevant passage URIs. We track two complementary metrics:
-
-- **Recall@K**: Fraction of relevant documents retrieved in top K results. Measures coverage.
-- **Success@K**: Fraction of queries with at least one relevant document in top K. Most relevant for RAG, where finding one good document is often sufficient.
-
-### Recall@K Results
-
-| Embedding Model            | Recall@1 | Recall@3 | Recall@5 | Reranker               |
-|----------------------------|----------|----------|----------|------------------------|
-| `qwen3-embedding`          | 0.31     | 0.48     | 0.54     | None                   |
-| `qwen3-embedding`          | 0.36     | 0.57     | 0.68     | `mxbai-rerank-base-v2` |
-| `qwen3-embedding`          | 0.36     | 0.58     | 0.67     | `zeroentropy`          |
-
-### Success@K Results
-
-| Embedding Model            | Success@1 | Success@3 | Success@5 | Reranker               |
-|----------------------------|-----------|-----------|-----------|------------------------|
-| `qwen3-embedding`          | 0.36      | 0.54      | 0.62      | None                   |
-| `qwen3-embedding`          | 0.42      | 0.66      | 0.76      | `mxbai-rerank-base-v2` |
-| `qwen3-embedding`          | 0.41      | 0.66      | 0.76      | `zeroentropy`          |
-
-## QA Accuracy
-
-And for QA accuracy,
-
-| Embedding Model            | QA Model  | Accuracy | Reranker               |
-|----------------------------|-----------|----------|------------------------|
-| `qwen3-embedding`          | `gpt-oss` | 0.75     | `mxbai-rerank-base-v2` |
+Filtering affects searches only — a run without `--skip-db` still populates the database with the dataset's full corpus.

docs/capabilities/analysis.md

@@ -0,0 +1,59 @@
+# Analysis Capability
+
+`AnalysisCapability` adds search, citations, and sandboxed Python computation over the document corpus. Use it for counts, aggregation, comparison, structural traversal, and section-scoped reading.
+
+It is deferred by default, keeping its substantial instructions and tool schemas out of context until the model chooses to load it.
+
+The default request limit is 30 model requests per question. Override it with `create_capability(request_limit=...)`, or set `request_limit=None` to disable it. As with the RAG capability, `create_capability(vision=...)` overrides the image-attachment gate, defaulting to the configured analysis model's `vision` flag. At the limit, `analysis_search` and `analysis_execute_code` are removed while `analysis_cite` remains for two further requests that call an analysis tool, so the model can register citations before answering from gathered evidence. Requests spent on other capabilities do not count against that window. Other agent and capability tools remain available, and the budget resets for every agent run.
+
+When `qa.max_searches` or `analysis.max_executions` runs out, the exhausted tool keeps failing rather than disappearing, and the instructions name it on every following request. Searching from inside `analysis_execute_code` does not count against `qa.max_searches`.
+
+## Tools
+
+| Tool | Purpose |
+|---|---|
+| `analysis_search(query, limit?)` | Search the corpus for evidence. |
+| `analysis_execute_code(code)` | Run Python against the virtual document filesystem. |
+| `analysis_cite(chunk_ids)` | Register retrieved or filesystem-derived chunk IDs. |
+
+The sandbox exposes documents under `/documents/{document_id}/` with `metadata.json`, `content.txt`, `items.jsonl`, `chunks.jsonl` (chunk ids with their metadata) and `toc.json`. In code, `await search()` results carry `chunk_meta` and `await list_documents()` rows carry `metadata`. The interpreter's limits and the per-call budgets are listed under [MCP, Code](../mcp.md#code).
+
+## Compose an agent
+
+Register it on its own, not alongside `RAGCapability`: it already searches and cites,
+and the two together give the model duplicate tools and separate budgets. See
+[Capabilities](index.md#compose-an-agent).
+
+```python
+from pydantic_ai import Agent
+from haiku.rag.capabilities.analysis import create_capability as analysis
+from haiku.rag.capabilities.compaction import create_capability as compaction
+from haiku.rag.capabilities.policy import create_capability as citation_policy
+
+agent = Agent(
+    "openai:gpt-5",
+    capabilities=[
+        analysis(db_path="my.lancedb"),
+        compaction(),
+        citation_policy(),
+    ],
+)
+```
+
+For the high-level convenience API:
+
+```python
+from haiku.rag.client import HaikuRAG
+
+async with HaikuRAG("my.lancedb") as client:
+    result = await client.analyze("Which quarter had the highest revenue?")
+    print(result.answer)
+```
+
+## State
+
+When dependencies expose a state dictionary, `AnalysisState` is stored under `"analysis"`. It contains the document filter, code execution log, searches, citations, and the `evidence` record of what was retrieved and cited per question. Searches and executions are cleared when a new question starts, and a resumed question keeps them; the filter, citation index and evidence record persist.
+
+This capability does not alter the message history either. Register the [compaction capability](compaction.md) to compact earlier questions.
+
+The capability lazily opens both LanceDB and the sandbox only after it is loaded and a tool requires them. Resources close at the end of the agent run.

docs/capabilities/compaction.md

@@ -0,0 +1,50 @@
+# Evidence compaction capability
+
+`EvidenceCompactionCapability` keeps a multi-turn conversation from carrying every
+search result it ever produced. Every question adds its evidence to the history, so
+requests grow turn after turn, which degrades answers and can exceed a provider's
+limits.
+
+Register it alongside an evidence capability:
+
+```python
+from pydantic_ai import Agent
+from haiku.rag.capabilities.compaction import create_capability as compaction
+from haiku.rag.capabilities.rag import create_capability as rag
+
+agent = Agent(
+    "openai:gpt-5",
+    capabilities=[rag(db_path="my.lancedb"), compaction()],
+)
+```
+
+It exposes no tools and takes no configuration. Registering it is the only switch:
+leave it out and the transcript reaches the model untouched.
+
+The host must carry the capability state between runs, alongside the message
+history: the capsule is built from what earlier questions recorded there. Given
+only a message history, every run starts from an empty record, and compaction
+refuses rather than replace evidence it cannot retain. See
+[Compose an agent](index.md#compose-an-agent) for the shape.
+
+## What it does
+
+On each request, evidence from earlier questions is replaced by the evidence those
+questions actually cited. Cited text and cited page images are kept in full, grouped by
+the question that cited them, and stay citable by the same chunk ids. Evidence spanning
+more than one collection carries a `Collection:` line naming the one it came from. Every
+other earlier evidence return becomes a short receipt. The current question is untouched.
+
+Compaction rewrites the request, never the stored history, so `all_messages()` still
+holds everything the run gathered.
+
+This reduces what a request carries. It does not bound it: retained evidence still
+grows with the conversation. A host that needs more aggressive pruning can compact its
+own requests further, on the wire only.
+
+## Resuming a question
+
+Resuming a question (deferred tool results, an interruption, a suspension) requires the
+host to carry the capability state from the run being resumed, alongside the message
+history. Without it the identity of the question in progress is unknowable, and the run
+fails rather than silently treating it as a new question.

docs/capabilities/index.md

@@ -0,0 +1,150 @@
+# Capabilities
+
+haiku.rag provides native [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/):
+
+| Capability | Use it for |
+|---|---|
+| [`RAGCapability`](rag.md) | Grounded document search and citations. |
+| [`AnalysisCapability`](analysis.md) | Corpus computation and structural analysis with sandboxed Python. |
+| [`EvidenceCompactionCapability`](compaction.md) | Optional. Shrinking a conversation's history to the evidence that was cited. |
+| [`CitationPolicyCapability`](policy.md) | Optional. Requiring every answer to declare what grounds it. |
+
+The two evidence capabilities are deferred by default. An agent initially sees only their descriptions and the standard `load_capability` tool. Instructions and tools enter the model context only when the model loads a capability.
+
+## Compose an agent
+
+Pick one evidence capability, and add both optional capabilities to it:
+
+```python
+from dataclasses import dataclass, field
+from typing import Any
+
+from pydantic_ai import Agent
+from pydantic_ai.messages import ModelMessage
+
+from haiku.rag.capabilities.compaction import create_capability as compaction
+from haiku.rag.capabilities.policy import create_capability as citation_policy
+from haiku.rag.capabilities.rag import create_capability as rag
+
+
+@dataclass
+class Deps:
+    state: dict[str, Any] = field(default_factory=dict)
+
+
+agent = Agent(
+    "openai:gpt-5",
+    capabilities=[
+        rag(db_path="my.lancedb"),
+        compaction(),
+        citation_policy(),
+    ],
+    deps_type=Deps,
+)
+
+# One Deps and one history for the conversation: the capabilities read both.
+deps = Deps()
+history: list[ModelMessage] = []
+
+result = await agent.run("What does the knowledge base say about X?", deps=deps, message_history=history)
+history = list(result.all_messages())
+print(result.output)
+```
+
+!!! warning "Both optional capabilities need the host to carry state"
+
+    They read what earlier questions retrieved and cited from the capability's
+    state, so the host must expose a `state` dict on its agent dependencies and
+    hand the same dict back on every run of a conversation, alongside the message
+    history. With only the message history, every run starts from an empty record:
+    compaction refuses rather than replace evidence it cannot retain, and the
+    citation policy cannot enforce a follow-up about evidence cited earlier.
+
+Swap `rag` for `analysis` for an analysis agent. Both optional capabilities work the
+same way with either one, and neither exposes tools or takes configuration.
+
+!!! note "Register one evidence capability, not both"
+
+    `RAGCapability` and `AnalysisCapability` overlap. Both search the same corpus and
+    both register citations, so an agent holding both must choose between two
+    near-identical search tools, and its citations land in whichever capability it
+    happened to call. Each also carries its own request limit and its own search
+    budget, so registering both doubles what a question may spend.
+
+    Choose by what the questions need. `RAGCapability` answers questions from retrieved
+    passages. `AnalysisCapability` adds a Python sandbox and a document filesystem, for
+    questions that compute over many documents or read their structure, and it can
+    search too. If you need computation, register the analysis capability alone rather
+    than adding it to the RAG one.
+
+## Agent specs
+
+The capabilities can be declared in a Pydantic AI [agent spec](https://ai.pydantic.dev/agent-spec/):
+
+```yaml title="agent.yaml"
+model: openai:gpt-5
+instructions: You are a research assistant with access to a document knowledge base.
+capabilities:
+  - RAGCapability:
+      db_path: /data/kb.lancedb
+      defer_loading: false
+  - EvidenceCompactionCapability
+  - CitationPolicyCapability
+```
+
+Pydantic AI does not discover third-party capabilities, so the caller names the classes:
+
+```python
+from pydantic_ai import Agent
+
+from haiku.rag.capabilities.compaction import EvidenceCompactionCapability
+from haiku.rag.capabilities.policy import CitationPolicyCapability
+from haiku.rag.capabilities.rag import RAGCapability
+
+agent = Agent.from_file(
+    "agent.yaml",
+    deps_type=Deps,
+    custom_capability_types=[
+        RAGCapability,
+        EvidenceCompactionCapability,
+        CitationPolicyCapability,
+    ],
+)
+```
+
+`deps_type` stays a Python argument, since the capabilities read and write their state
+through `deps.state` (see [State](#state)). `Agent.from_file` reads YAML, which needs
+`pydantic-ai-slim[spec]`; `Agent.from_spec` takes a dict and needs no YAML parser.
+
+Set `defer_loading: false` when the agent registers a single evidence capability, so its
+tools are visible immediately. Leave it at the default when the model should route among
+multiple capabilities.
+
+A `config:` block accepts a whole `AppConfig`, for agents in one process that need
+different databases or embedding models:
+
+```yaml
+capabilities:
+  - RAGCapability:
+      db_path: /data/kb.lancedb
+      config:
+        embeddings:
+          model: {provider: ollama, name: embeddinggemma, vector_dim: 2048}
+```
+
+The block is read like a `haiku.rag.yaml` file: keys it omits take `AppConfig` defaults
+rather than values from the configuration file on disk. The embedding model must match the
+database; a mismatch may prevent opening it or produce invalid retrieval. Write the block in
+full or omit it and let the [configuration file](../configuration/index.md) apply.
+
+## State
+
+Capabilities use a plain `state: dict[str, Any]` attribute on agent dependencies when one is available. RAG state lives under `"rag"`; analysis state lives under `"analysis"`. This keeps state independent of any transport or UI protocol.
+
+Applications serving AG-UI should adapt the agent with Pydantic AI's `AGUIAdapter`. Native model and tool events require no haiku.rag-specific bridge.
+
+## Database Selection
+
+RAG and analysis capabilities cover the databases the configuration places: [`lancedb.databases`](../configuration/storage.md#multiple-databases), or with nothing configured the default database `haiku.rag` under `storage.data_dir`. The `db_path` argument places one database where the configuration places none; beside `lancedb.databases` it raises `AmbiguousDatabaseError`.
+
+Passing a client through `rag=` bypasses this selection. The capability uses the databases covered by that client and does not close it.

docs/capabilities/policy.md

@@ -0,0 +1,54 @@
+# Citation policy capability
+
+`CitationPolicyCapability` requires every answer to declare what grounds it. Citing is
+always available and always recorded without it, but nothing makes the model do it.
+
+Register it alongside an evidence capability:
+
+```python
+from pydantic_ai import Agent
+from haiku.rag.capabilities.policy import create_capability as citation_policy
+from haiku.rag.capabilities.rag import create_capability as rag
+
+agent = Agent(
+    "openai:gpt-5",
+    capabilities=[rag(db_path="my.lancedb"), citation_policy()],
+)
+```
+
+It exposes no tools and takes no configuration. Exactly one policy capability makes
+the decision, however many evidence capabilities are registered, so two of them
+cannot each demand a citation for one answer.
+
+The host must carry the capability state between runs, alongside the message
+history. Enforcement reads what the conversation has already cited, so without it
+a follow-up about evidence cited earlier goes unenforced. See
+[Compose an agent](index.md#compose-an-agent) for the shape.
+
+## Declaring nothing is a valid answer
+
+A model that finds nothing relevant calls the cite tool with an empty list. That records
+the answer as *ungrounded*, which is distinct from an answer that declared nothing at
+all (*missing*). The distinction is what makes a declaration requirable without forcing
+the model to invent grounding.
+
+## What happens when a question ends undeclared
+
+The model is asked once to record what grounded the answer it already gave. It is not
+asked to change the answer. If the cite tool is no longer available by then, or the
+question finishes undeclared anyway, the question is recorded in
+`CitationPolicyState.violations` under the `"citation_policy"` state key. Pointing a
+model at a tool that is gone costs it retries, so the capability records the failure
+instead.
+
+## Which answers are enforced
+
+Every answer in a conversation that has something to declare: either this question
+retrieved evidence, or the conversation has already cited something, which stays
+available to later answers. A follow-up about evidence cited earlier is enforced even
+though it searched nothing, which is the case the capability exists for.
+
+Once anything has been cited, later turns are enforced too, a greeting included. The
+model satisfies the policy by citing an empty list, at the cost of one extra request. A
+conversation with neither a current-question evidence outcome nor any earlier citation
+is not enforced.

docs/capabilities/rag.md

@@ -0,0 +1,69 @@
+# RAG Capability
+
+`RAGCapability` adds grounded document search and citations to a Pydantic AI agent. It is deferred by default, so its instructions and tools do not consume model context until loaded.
+
+## Tools
+
+| Tool | Purpose |
+|---|---|
+| `rag_search(query, limit?)` | Hybrid vector and full-text search with context expansion. |
+| `rag_cite(chunk_ids)` | Register exact result chunk IDs as answer citations. |
+
+The distinct `rag_` prefix lets this capability coexist with analysis and other search providers.
+
+## Create and compose
+
+Register it on its own rather than alongside `AnalysisCapability`, which searches and
+cites as well. See [Capabilities](index.md#compose-an-agent).
+
+```python
+from pydantic_ai import Agent
+from haiku.rag.capabilities.compaction import create_capability as compaction
+from haiku.rag.capabilities.policy import create_capability as citation_policy
+from haiku.rag.capabilities.rag import create_capability as rag
+
+agent = Agent(
+    "openai:gpt-5",
+    capabilities=[
+        rag(db_path="my.lancedb"),
+        compaction(),
+        citation_policy(),
+    ],
+)
+
+result = await agent.run("What safety equipment does the manual require?")
+print(result.output)
+```
+
+`create_capability` accepts `db_path`, `config`, `defer_loading`, `request_limit`, and `vision`. Set `defer_loading=False` for a dedicated RAG agent where routing is unnecessary. The default request limit is 20 model requests per question; set `request_limit=None` to disable it. `vision` controls whether picture results are attached to search returns as images and should reflect the model the hosting agent runs; it defaults to the configured QA model's `vision` flag.
+
+When the limit is reached, `rag_search` is removed while `rag_cite` remains for two further requests that call a RAG tool, so the model can register citations before answering from evidence already gathered. Requests spent on other capabilities do not count against that window. Unrelated agent and capability tools remain available. A new agent run starts a fresh limit, so multi-turn chat does not consume one shared budget.
+
+## State
+
+When agent dependencies expose a `state` dictionary, the capability maintains a `RAGState` under `"rag"`:
+
+```python
+class RAGState(BaseModel):
+    citation_index: dict[str, Citation]
+    citations: list[str]
+    document_filter: str | None
+    evidence: CapabilityEvidenceRecord
+    searches: dict[str, list[SearchResult]]
+```
+
+`document_filter`, `citation_index` and `evidence` persist across runs. Citations and searches are cleared when a new question starts; a run that resumes a question keeps the evidence it is still answering from.
+
+`evidence` records which chunks this capability retrieved and cited, and in which question. `haiku.rag.capabilities.ledger.citation_status(records, question=...)` derives `missing`, `grounded` or `ungrounded` from it, across capabilities.
+
+State is ordinary application state; the capability does not depend on AG-UI. An AG-UI application can expose it using Pydantic AI's standard adapter.
+
+## Context management
+
+This capability does not alter the message history. To stop long conversations resending old retrieved content, register the [compaction capability](compaction.md) alongside it.
+
+## Domain context and vision
+
+`prompts.domain_preamble` is prepended to the packaged capability instructions. When the capability's `vision` gate is on (by default, when the configured QA model has `vision: true`), picture results are attached to search returns as `BinaryContent`.
+
+See [Search and question answering](../configuration/qa.md) and [picture processing](../configuration/processing.md#picture-handling).

docs/changelog.md

@@ -0,0 +1 @@
+--8<-- "CHANGELOG.md"

docs/chat.md

@@ -0,0 +1,87 @@
+# Chat
+
+The chat TUI runs conversational RAG against your database from the terminal. Streaming responses, expandable citations with visual grounding, multi-turn sessions, and a command palette for filtering and inspection.
+
+!!! note
+    Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in the full `haiku.rag` package).
+
+## Run it
+
+```bash
+haiku-rag chat
+haiku-rag chat --db /path/to/database.lancedb
+haiku-rag chat --model openai:gpt-4o
+```
+
+![Chat TUI session with the analysis capability](img/chat-qa.png)
+
+## How it works
+
+The chat is a Pydantic AI agent with the [RAG capability](capabilities/rag.md) attached by default. A single capability loads eagerly; when both RAG and analysis are enabled, they remain deferred until the model chooses which one to load. Each turn the agent decides which tool to call next, runs hybrid search against your documents, expands context around the hits, may issue further searches, and answers with citations. You see streaming text and native tool events directly from Pydantic AI.
+
+The session is in-memory for the lifetime of the TUI. Conversation history is kept across turns so follow-up questions reuse prior context. Citations are tracked per turn and inspectable via the command palette. Clearing the chat resets the session and the agent's memory.
+
+## Citations and visual grounding
+
+Each answer cites the chunks the agent used, with source document, page numbers, and section headings. Citations are expandable inline. Picture citations render the figure directly underneath the text snippet.
+
+![Expanded citation with an inline figure](img/chat-citation-figure.png)
+
+For visual grounding of a text chunk (the chunk highlighted on its source page image), open the command palette and pick "Show visual grounding". This requires:
+
+- Documents processed via Docling with page images (default for PDFs).
+- A terminal that supports inline images (iTerm2, WezTerm, Kitty).
+- A stored DoclingDocument on the document. Plain text added via `haiku-rag add` doesn't have it.
+
+You can also render visual grounding from the CLI without launching the TUI:
+
+```bash
+haiku-rag visualize <chunk_id>
+```
+
+## Attaching images
+
+Press `Ctrl+I` to open the image picker: a directory tree filtered to image files with a live preview. Selecting an image inserts an `[Image #N]` token at the cursor and attaches the image to your next message. Tokens delete as a unit with backspace or delete, and you can place them anywhere in the text to control where each image appears relative to your words.
+
+Retrieval stays text-based; the images are sent to the model alongside your message, so the driving model needs `vision: true` in its configuration.
+
+## Command palette
+
+`Ctrl+P` opens the palette.
+
+| Command | What it does |
+|---------|--------------|
+| Clear chat | Reset session memory |
+| Filter documents | Restrict searches to selected documents |
+| Show visual grounding | Visual grounding for a citation |
+| Database info | Document and chunk counts, storage stats |
+
+## Capabilities
+
+The default capability is `rag`. Enable `analysis` when the question needs computation, aggregation, comparison across documents, or section-scoped reading that a single search can't deliver:
+
+```bash
+# analysis instead of rag
+haiku-rag chat -c analysis
+
+# both, which gives the model duplicate search and cite tools
+haiku-rag chat -c rag -c analysis
+```
+
+Prefer one. `analysis` searches and cites as well as computing, so pairing it with `rag`
+duplicates tools and budgets. See [Capabilities](capabilities/index.md#compose-an-agent).
+
+The `analysis` capability mounts every document as a virtual filesystem at `/documents/{id}/` (with `metadata.json`, `content.txt`, `items.jsonl`, and `toc.json`) and runs Python in a sandboxed interpreter with `search` and `list_documents` as awaitable functions. It's the right choice for questions like:
+
+- "How many of these documents mention X?"
+- "Summarize Section 5 of paper Y."
+- "Compare the experimental sections across these three reports."
+- "Which section discusses the proof of Theorem 4.10?"
+
+For everyday Q&A, RAG alone is faster and cheaper. Attaching both lets the agent pick. See [Analysis capability](capabilities/analysis.md).
+
+## Document filter
+
+Run "Filter documents" from the command palette to restrict searches to a subset. The filter applies to every search the agent runs for the rest of the session.
+
+Chat also honors the global `--read-only` flag. See the [CLI reference](cli.md) for details.

docs/cli.md

@@ -6,46 +6,37 @@ The `haiku-rag` CLI provides complete document management functionality.
     Global options (must be specified before the command):
 
     - `--config` - Specify custom configuration file
+    - `--read-only` - Open database in read-only mode (blocks writes, skips upgrades)
+    - `--db-name` - Name of a database from `lancedb.databases` to work on
     - `--version` / `-v` - Show version and exit
 
     Per-command options:
 
-    - `--db` - Specify custom database path
+    - `--db` - Open the database at this path, named by its stem, whatever the configuration places
     - `-h` - Show help for specific command
 
     Example:
     ```bash
     haiku-rag --config /path/to/config.yaml list
     haiku-rag --config /path/to/config.yaml list --db /path/to/custom.db
+    haiku-rag --read-only search "query"
+    haiku-rag --db-name papers list
     haiku-rag add -h
     ```
 
+    With `lancedb.databases` configured, `search`, `ask`, `analyze`, `chat`, and `mcp` use the full set by default. Select one database for other commands with `--db-name` or `--db`. `settings`, `init-config`, and `download-models` do not open a database. See [Multiple Databases](configuration/storage.md#multiple-databases).
+
 ## Document Management
 
-### List Documents
-
-```bash
-haiku-rag list
-```
-
-Filter documents by properties:
-```bash
-# Filter by URI pattern
-haiku-rag list --filter "uri LIKE '%arxiv%'"
-
-# Filter by exact title
-haiku-rag list --filter "title = 'My Document'"
-
-# Combine multiple conditions
-haiku-rag list --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
-```
-
 ### Add Documents
 
 From text:
 ```bash
 haiku-rag add "Your document content here"
 
+# Set a title
+haiku-rag add "Your document content here" --title "My Document"
+
 # Attach metadata (repeat --meta for multiple entries)
 haiku-rag add "Your document content here" --meta author=alice --meta topic=notes
 ```
@@ -68,8 +59,19 @@ From directory (recursively adds all supported files):
 haiku-rag add-src /path/to/documents/
 ```
 
+From an S3 bucket (requires the `[s3]` extra, see the [ingester docs](ingester.md) for continuous S3 polling):
+```bash
+# AWS S3 with credentials in the default chain (env vars, IAM role, AWS profile)
+haiku-rag add-src s3://my-bucket/path/to/document.pdf
+
+# S3-compatible endpoint (SeaweedFS, MinIO, Cloudflare R2, etc.)
+AWS_ACCESS_KEY_ID=key AWS_SECRET_ACCESS_KEY=secret AWS_REGION=us-east-1 \
+  AWS_ENDPOINT_URL=http://localhost:8333 \
+  haiku-rag add-src s3://my-bucket/path/to/document.pdf
+```
+
 !!! note
-    When adding a directory, the same content filters configured for [file monitoring](configuration.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added.
+    When adding a directory, the converter's supported extensions filter applies. For pattern-based ignore/include filtering (e.g. `**/.git/**`), use the [ingester](ingester.md) with a filesystem source.
 
 !!! note
     As you add documents to `haiku.rag` the database keeps growing. By default, LanceDB supports versioning
@@ -77,6 +79,24 @@ haiku-rag add-src /path/to/documents/
     the database rolls back to the pre‑operation snapshot using LanceDB table versioning. You can optimize and
     compact the database by running the [vacuum](#vacuum-optimize-and-cleanup) command.
 
+### List Documents
+
+```bash
+haiku-rag list
+```
+
+Filter documents by properties:
+```bash
+# Filter by URI pattern (--filter or -f)
+haiku-rag list --filter "uri LIKE '%arxiv%'"
+
+# Filter by exact title
+haiku-rag list --filter "title = 'My Document'"
+
+# Combine multiple conditions
+haiku-rag list --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
+```
+
 ### Get Document
 
 ```bash
@@ -90,8 +110,6 @@ haiku-rag delete 3f4a...   # document ID
 haiku-rag rm 3f4a...       # alias
 ```
 
-Use this when you want to change things like the embedding model or chunk size for example.
-
 ## Search
 
 Basic search:
@@ -101,10 +119,22 @@ haiku-rag search "machine learning"
 
 With options:
 ```bash
-haiku-rag search "python programming" --limit 10
+haiku-rag search "python programming" --limit 10  # or -l 10
 ```
 
-With filters (filter by document properties):
+With search type:
+```bash
+# Hybrid search (the default)
+haiku-rag search "python programming" --search-type hybrid  # or -s hybrid
+
+# Full-text search only
+haiku-rag search "python programming" --search-type fts  # or -s fts
+
+# Vector search only
+haiku-rag search "python programming" --search-type vector  # or -s vector
+```
+
+With filters (filter by document properties, use `--filter` or `-f`):
 ```bash
 # Filter by URI pattern
 haiku-rag search "neural networks" --filter "uri LIKE '%arxiv%'"
@@ -116,6 +146,13 @@ haiku-rag search "transformers" --filter "title = 'Deep Learning Guide'"
 haiku-rag search "AI" --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
 ```
 
+Image-as-query (requires a multimodal embedder):
+```bash
+haiku-rag search --image path/to/figure.png --limit 5
+```
+
+When `--image` is used, the positional query is omitted. Pass one or the other, not both.
+
 ## Question Answering
 
 Ask questions about your documents:
@@ -123,78 +160,130 @@ Ask questions about your documents:
 haiku-rag ask "Who is the author of haiku.rag?"
 ```
 
-Ask questions with citations showing source documents:
+Filter to specific documents:
 ```bash
-haiku-rag ask "Who is the author of haiku.rag?" --cite
+haiku-rag ask "What are the main findings?" --filter "uri LIKE '%paper%'"
 ```
 
-Use deep QA for complex questions (multi-agent decomposition):
+Attach images to the question, for example to check an image against indexed documents:
 ```bash
-haiku-rag ask "What are the main features and architecture of haiku.rag?" --deep --cite
+haiku-rag ask "Does this photo satisfy the spec in the design document?" --image photo.jpg
 ```
 
-Show verbose output with deep QA:
+`ask` runs the [RAG capability](capabilities/rag.md) and always renders citations under the answer. When available, citations use the document title, otherwise they fall back to the URI.
+
+Citation text is truncated to a 300-character preview. To read the whole passage the model saw:
 ```bash
-haiku-rag ask "What are the main features and architecture of haiku.rag?" --deep --verbose
-```
-
-The QA agent will search your documents for relevant information and provide a comprehensive answer. With `--cite`, responses include citations showing which documents were used. With `--deep`, the question is decomposed into sub-questions that are answered in parallel before synthesizing a final answer. With `--verbose` (only with `--deep`), you'll see the planning, searching, evaluation, and synthesis steps as they happen.
-When available, citations use the document title; otherwise they fall back to the URI.
-
-## Research
-
-Run the multi-step research graph:
-
-```bash
-haiku-rag research "How does haiku.rag organize and query documents?" \
-  --max-iterations 2 \
-  --confidence-threshold 0.8 \
-  --max-concurrency 3 \
-  --verbose
+haiku-rag ask "What are the main findings?" --full-citations
 ```
 
 Flags:
-- `--max-iterations, -n`: maximum search/evaluate cycles (default: 3)
-- `--confidence-threshold`: stop once evaluation confidence meets/exceeds this (default: 0.8)
-- `--max-concurrency`: number of sub-questions searched in parallel each iteration (default: 3)
-- `--verbose`: show planning, searching previews, evaluation summary, and stop reason
 
-When `--verbose` is set the CLI also consumes the internal research stream, printing every `log` event as agents progress through planning, search, evaluation, and synthesis. If you build your own integration, call `stream_research_graph` to access the same `log`, `report`, and `error` events and render them however you like while the graph is running.
+- `--filter` / `-f`: Restrict searches to documents matching the filter (see [Filtering Search Results](python.md#filtering-search-results))
+- `--image`: Path to an image attached to the question (repeatable). Retrieval stays text-based; the model must have `vision: true` configured.
+- `--full-citations`: Show the full text of each citation instead of a truncated preview
 
-## Server
+## Analyze
+
+Answer complex analytical questions via code execution:
 
-Start services (requires at least one flag):
 ```bash
-# MCP server only (HTTP transport)
-haiku-rag serve --mcp
-
-# MCP server (stdio transport)
-haiku-rag serve --mcp --stdio
-
-# File monitoring only
-haiku-rag serve --monitor
-
-# Both services
-haiku-rag serve --monitor --mcp
-
-# Custom port
-haiku-rag serve --mcp --mcp-port 9000
+haiku-rag analyze "How many documents mention security?"
 ```
 
-See [Server Mode](server.md) for details on available services.
+Filter to specific documents:
 
-## Settings
-
-View current configuration settings:
 ```bash
-haiku-rag settings
+haiku-rag analyze "What is the total revenue?" --filter "title LIKE '%Financial%'"
 ```
 
-## Maintenance
+Flags:
 
-### Info (Read-only)
+- `--filter` / `-f`: SQL WHERE clause to restrict document access
+- `--image`: Path to an image attached to the question (repeatable). Requires `vision: true` on the analysis model.
+- `--full-citations`: Show the full text of each citation instead of a truncated preview
 
-Display database metadata without upgrading or modifying it:
+See [Analysis capability](capabilities/analysis.md) for details and configuration.
+
+## Chat
+
+Launch an interactive chat session for multi-turn conversations:
+
+```bash
+haiku-rag chat
+haiku-rag chat --db /path/to/database.lancedb
+
+# Enable the analysis capability (code execution)
+haiku-rag chat -c rag -c analysis
+```
+
+!!! note
+    Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in full `haiku.rag` package)
+
+Flags:
+
+- `--capability` / `-c`: Capabilities to enable. `rag` (default), `analysis`. Can be repeated.
+
+The chat interface provides:
+
+- Streaming responses with real-time tool execution
+- Expandable citations with source metadata
+- Session memory for context-aware follow-up questions
+- Visual grounding to inspect chunk source locations
+
+See [Chat](chat.md) for keyboard shortcuts and features.
+
+## Inspect
+
+Launch the interactive inspector TUI for browsing documents and chunks:
+
+```bash
+haiku-rag inspect
+haiku-rag inspect --db /path/to/database.lancedb
+```
+
+!!! note
+    Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in full `haiku.rag` package)
+
+The inspector provides:
+
+- Browse all documents in the database
+- View document metadata and content
+- Explore individual chunks
+- Search and filter results
+
+See [Tuning: Inspector](tuning.md#inspector) for the full keybindings and modal flows.
+
+## Visualize Chunk
+
+Display visual grounding for a chunk - shows page images with highlighted bounding boxes:
+
+```bash
+haiku-rag visualize <chunk_id>
+```
+
+This renders the source document pages with the chunk's location highlighted. The chunk itself draws in a strong highlight, while surrounding context swept in by expansion draws fainter. Useful for verifying chunk boundaries and understanding document structure.
+
+Pass `--no-expand` to highlight only the chunk itself, without its expanded context.
+
+!!! note
+    Requires a terminal with image support (iTerm2, Kitty, WezTerm, etc.) and documents processed with docling that have page images stored.
+
+## Database lifecycle
+
+### Initialize Database
+
+Create a new database:
+
+```bash
+haiku-rag init [--db /path/to/your.lancedb]
+```
+
+This creates the database with the configured settings. **All other commands require an existing database** - they will fail with an informative error if the database doesn't exist.
+
+### Info
+
+Display database metadata:
 
 ```bash
 haiku-rag info [--db /path/to/your.lancedb]
@@ -204,33 +293,84 @@ Shows:
 - path to the database
 - stored haiku.rag version (from settings)
 - embeddings provider/model and vector dimension
-- number of documents
-- table versions per table (documents, chunks)
+- per-table row counts and storage sizes (documents, document_meta, chunks, document_items)
+- vector index status (exists/not created, indexed/unindexed chunks)
+- table versions per table (documents, document_meta, chunks)
 
-At the end, a separate “Versions” section lists runtime package versions:
+At the end, a separate "Versions" section lists runtime package versions:
 - haiku.rag
 - lancedb
 - docling
 
-### Vacuum (Optimize and Cleanup)
+### Doctor
 
-Reduce disk usage by optimizing and pruning old table versions across all tables:
+Check the database for consistency problems and print a pass/warn/fail report:
 
 ```bash
-haiku-rag vacuum
+haiku-rag doctor [--db /path/to/your.lancedb] [--duplicates-out groups.yaml]
 ```
 
-**Automatic Cleanup:** Vacuum runs automatically in the background after document operations. By default, it removes versions older than 60 seconds (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately.
+While it runs, doctor shows a spinner naming the check currently in progress.
 
-### Rebuild Database
+`--duplicates-out PATH` additionally writes the near-duplicate document groups to a YAML file (one block per group with `keep` and a list of `documents`, each carrying `document_id`, `document`, `chunks`, `similarity`, and `keep_suggested`) for offline review.
 
-Rebuild the database by deleting all chunks & embeddings and re-indexing all documents. This is useful
-when want to switch embeddings provider or model:
+Checks include:
+
+- required tables are present
+- `documents` and `document_meta` are in 1:1 correspondence
+- chunks and document items reference documents that exist
+- documents with text content produced chunks (empty and heading/furniture-only documents are not flagged; image-only documents are flagged according to whether the embedder can index images)
+- chunked documents have document items (empty documents are not flagged)
+- chunk `doc_item_refs` resolve to existing document items
+- chunk vector size matches the stored embedding dimension
+- chunks are embedded (no all-zero vectors)
+- pictures in image/PDF documents carry their image data (external image references in text documents are not flagged)
+- exactly one settings row is present
+- the configured embedding identity matches the stored settings
+- no database migrations are pending
+- the vector index covers all chunks
+- the full-text index covers the chunks it searches
+- near-identical documents (by embedding-centroid similarity) are grouped and reported, with the largest member flagged as the likely one to keep (advisory only, never deleted, tuned via `doctor.duplicates` in config)
+- API keys are set for configured providers
+
+It also probes the external endpoints the config uses and reports them under a Providers section:
+
+- Ollama is reachable and the configured models are installed (`{base_url}/api/tags`)
+- docling-serve is reachable when used as the converter or chunker (`{base_url}/health`)
+- custom OpenAI-compatible and vLLM endpoints respond (`{base_url}/models`)
+
+SaaS providers (OpenAI, Anthropic, Cohere, Jina, ZeroEntropy, Voyage) are covered by the API-key check rather than a network probe. In-process local models (sentence-transformers, cross-encoder, jina-local) have no endpoint and are reported as such.
+
+Each failure prints the command that fixes it (`rebuild`, `create-index`, `vacuum`, `migrate`, `rebuild --set-embedder`). `doctor` makes no changes. It exits with status 1 when any check fails, so it can gate CI or monitoring.
+
+### Migrate Database
+
+Apply pending database migrations:
 
 ```bash
-haiku-rag rebuild
+haiku-rag migrate [--db /path/to/your.lancedb]
 ```
 
+When you upgrade haiku.rag to a new version that includes schema changes, the database requires migration. Opening a database with pending migrations will display an error:
+
+```
+Error: Database requires migration from 0.19.0 to 0.26.5. 3 migration(s) pending. Run 'haiku-rag migrate' to upgrade.
+```
+
+Run `haiku-rag migrate` to apply the pending migrations. The command shows which migrations were applied:
+
+```
+Applied 4 migration(s):
+  - 0.20.0: Add 'docling_document_json' and 'docling_version' columns
+  - 0.23.1: Add content_fts column for contextualized FTS search
+  - 0.25.0: Compress docling_document with gzip
+  - 0.38.0: Split docling_document pages into separate column and re-compress with zstd
+Migration completed successfully.
+```
+
+!!! tip
+    Back up your database before running migrations. While migrations are designed to be safe, having a backup provides peace of mind for production databases.
+
 ### Download Models
 
 Download required runtime models:
@@ -239,6 +379,201 @@ Download required runtime models:
 haiku-rag download-models
 ```
 
-This command:
-- Downloads Docling OCR/conversion models (no-op if already present).
-- Pulls Ollama models referenced in your configuration (embeddings, QA, research, rerank).
+This command downloads:
+
+- Docling OCR/conversion models
+- HuggingFace tokenizer (for chunking)
+- Ollama models referenced in your configuration (embeddings, QA, rerank)
+
+Progress is displayed in real-time with download status and progress bars for Ollama model pulls.
+
+## Maintenance
+
+### Create Vector Index
+
+Create a vector index on the chunks table for fast approximate nearest neighbor search:
+
+```bash
+haiku-rag create-index [--db /path/to/your.lancedb]
+```
+
+**Requirements:**
+- Minimum 256 chunks required for index creation (LanceDB training data requirement)
+- Creates an IVF_PQ index using the configured `search.vector_index_metric` (cosine/l2)
+
+**When to use:**
+- On a collection over 100,000 chunks (below that, brute-force kNN is exact and fast enough)
+- After substantial corpus growth, to retrain the centroids
+- Use `haiku-rag info` to check index status and see how many chunks are indexed/unindexed, or `haiku-rag doctor` for the same as a health check
+
+See [Vector Indexing](configuration/storage.md#vector-indexing) for the measured accuracy and build cost.
+
+**Search behavior:**
+- Without index: Brute-force kNN search (exact nearest neighbors, slower for large datasets)
+- With index: ANN (approximate nearest neighbors) using IVF_PQ, tuned by `search.vector_nprobes`
+- Between a write and the next `optimize()`: LanceDB combines ANN over indexed rows with brute-force kNN over the remainder
+
+### Rebuild Database
+
+Rebuild the database by re-indexing documents. Useful when switching embeddings provider/model or changing chunking settings:
+
+```bash
+# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
+haiku-rag rebuild
+
+# Re-chunk from stored content (no source file access)
+haiku-rag rebuild --rechunk
+
+# Only regenerate embeddings (fastest, keeps existing chunks)
+haiku-rag rebuild --embed-only
+
+# Only generate titles for untitled documents
+haiku-rag rebuild --title-only
+
+# Run the VLM over already-stored picture bytes and patch descriptions
+# into the docling blob. Skips the docling parse entirely.
+haiku-rag rebuild --descriptions
+
+# Adopt the current embedder identity without re-embedding (same vector dimension)
+haiku-rag rebuild --set-embedder
+```
+
+**Rebuild modes:**
+
+| Mode | Flag | Use case |
+|------|------|----------|
+| Full | (default) | Changed converter, source files updated |
+| Rechunk | `--rechunk` | Changed chunking strategy or chunk size |
+| Embed only | `--embed-only` | Changed embedding model or vector dimensions |
+| Title only | `--title-only` | Generate titles for documents without one |
+| Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database |
+| Set embedder | `--set-embedder` | Same model, different serving stack (e.g. Ollama to vLLM); vector dimension unchanged |
+
+**`--set-embedder` mode** updates the stored embedding provider/name to match the current config without re-embedding, valid only when the vector dimension is unchanged. Use it when the same model is served by a different stack so the recorded identity stops drifting from the config. A changed vector dimension is rejected; regenerate embeddings with `--embed-only` or a full rebuild instead.
+
+**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent: pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely. Only the VLM time is paid.
+
+### Vacuum (Optimize and Cleanup)
+
+Reduce disk usage by optimizing and pruning old table versions across all tables:
+
+```bash
+haiku-rag vacuum
+```
+
+**Automatic Cleanup:** Vacuum runs automatically in the background after document operations, throttled to at most once every 5 minutes so sustained ingestion does not trigger continuous compaction (a final vacuum runs when the client closes). By default, it removes versions older than 1 day (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately.
+
+## MCP Server
+
+```bash
+# HTTP transport on port 8001
+haiku-rag mcp
+
+# stdio transport (for Claude Desktop)
+haiku-rag mcp --stdio
+
+# Custom port
+haiku-rag mcp --port 9000
+
+# Bind to all interfaces (containers, trusted LAN)
+haiku-rag mcp --host 0.0.0.0
+```
+
+See [MCP](mcp.md) for details. For continuous document ingestion
+(filesystem watch, S3 polling, HTTP / WebDAV sources), use the
+[ingester](ingester.md).
+
+## Settings
+
+View current configuration settings:
+```bash
+haiku-rag settings
+```
+
+### Generate Configuration File
+
+Generate a YAML configuration file with defaults:
+```bash
+haiku-rag init-config [output_path]
+```
+
+If no path is specified, creates `haiku.rag.yaml` in the current directory.
+
+## Tags
+
+A tag names the current database state. It is a logical snapshot composed of one LanceDB tag on each of the five tables, created from a single version snapshot.
+
+```bash
+# Tag the current state, e.g. at deploy time or after an ingestion run
+haiku-rag tag create release-1
+
+# List tags with the versions they point to
+haiku-rag tag list
+
+# Delete a tag, releasing its versions for cleanup
+haiku-rag tag delete release-1
+```
+
+A tag present on every table is complete. A tag missing from some tables (created outside haiku.rag, or left behind by a failure) is partial. `tag list` marks partial tags. Partial tags can be listed and deleted but never restored.
+
+Create tags with other writers stopped. Tag creation coordinates writers within one process only; a writer in another process can commit between the per-table snapshot reads, and the tag then captures a mixed state.
+
+Tagged versions survive `vacuum`. Vacuum retains the oldest tagged version and every newer version; versions older than the oldest tag remain eligible for cleanup. Delete tags you no longer need so cleanup can advance.
+
+### Restore
+
+`tag restore` brings the database back to a tagged state:
+
+```bash
+haiku-rag tag restore release-1
+```
+
+Restore changes the live state. It is not a read-only view: each table gets a new latest version equal to the tagged one, and reads and writes continue from there. Versions written after the tag remain in history until vacuum removes them.
+
+Before changing anything, restore creates a complete safety tag (`before-restore-<timestamp>`) for the current state and reports it, so you always have a named path back:
+
+```bash
+haiku-rag tag create release-1 --db /path/to/db.lancedb
+# Stop all writers before either restore.
+haiku-rag tag restore release-1 --db /path/to/db.lancedb --yes
+haiku-rag tag list --db /path/to/db.lancedb
+haiku-rag tag restore before-restore-YYYYMMDDTHHMMSSZ --db /path/to/db.lancedb --yes
+```
+
+Restore is a maintenance operation:
+
+- Stop all ingestion and other writers before restoring and keep them stopped until it finishes.
+- The operation is coordinated but not transactionally atomic across tables. On failure it attempts to roll back to the pre-restore state and reports whether the rollback succeeded.
+- `--yes` only skips the confirmation prompt. It provides no locking and no concurrent-writer protection.
+- Restore never migrates. Restoring a tag from an older haiku.rag version completes normally, and the next open reports the required migration. Run `haiku-rag migrate` explicitly.
+
+### Version History
+
+View version history for database tables:
+
+```bash
+# Show history for all tables
+haiku-rag history
+
+# Show history for a specific table
+haiku-rag history --table documents
+
+# Limit number of versions shown
+haiku-rag history --limit 10
+```
+
+Output shows version numbers and timestamps, sorted newest first, with tags marked:
+
+```
+Version History
+
+documents
+  v5: 2025-01-15 14:30:00  <- release-1
+  v4: 2025-01-14 10:00:00
+  v3: 2025-01-13 09:15:00
+
+chunks
+  v8: 2025-01-15 14:30:00  <- release-1
+  v7: 2025-01-14 10:00:00
+  ...
+```

docs/configuration.md

@@ -1,574 +0,0 @@
-# Configuration
-
-Configuration is done through YAML configuration files.
-
-!!! note
-    If you create a db with certain settings and later change them, `haiku.rag` will detect incompatibilities (for example, if you change embedding provider) and will exit. You can **rebuild** the database to apply the new settings, see [Rebuild Database](./cli.md#rebuild-database).
-
-## Getting Started
-
-Generate a configuration file with defaults:
-
-```bash
-haiku-rag init-config
-```
-
-This creates a `haiku.rag.yaml` file in your current directory with all available settings.
-
-## Configuration File Locations
-
-`haiku.rag` searches for configuration files in this order:
-
-1. Path specified via `--config` flag: `haiku-rag --config /path/to/config.yaml <command>`
-2. `./haiku.rag.yaml` (current directory)
-3. Platform-specific user directory:
-    - **Linux**: `~/.local/share/haiku.rag/haiku.rag.yaml`
-    - **macOS**: `~/Library/Application Support/haiku.rag/haiku.rag.yaml`
-    - **Windows**: `C:/Users/<USER>/AppData/Roaming/haiku.rag/haiku.rag.yaml`
-
-## Minimal Configuration
-
-A minimal configuration file with defaults:
-
-```yaml
-# haiku.rag.yaml
-environment: production
-
-embeddings:
-  provider: ollama
-  model: qwen3-embedding
-  vector_dim: 4096
-
-qa:
-  provider: ollama
-  model: gpt-oss
-```
-
-## Complete Configuration Example
-
-```yaml
-# haiku.rag.yaml
-environment: production
-
-storage:
-  data_dir: ""  # Empty = use default platform location
-  disable_autocreate: false
-  vacuum_retention_seconds: 60
-
-monitor:
-  directories:
-    - /path/to/documents
-    - /another/path
-  ignore_patterns: []  # Gitignore-style patterns to exclude
-  include_patterns: []  # Gitignore-style patterns to include
-
-lancedb:
-  uri: ""  # Empty for local, or db://, s3://, az://, gs://
-  api_key: ""
-  region: ""
-
-embeddings:
-  provider: ollama
-  model: qwen3-embedding
-  vector_dim: 4096
-
-reranking:
-  provider: ""  # Empty to disable, or mxbai, cohere, zeroentropy, vllm
-  model: ""
-
-qa:
-  provider: ollama
-  model: gpt-oss
-
-research:
-  provider: ""  # Empty to use qa settings
-  model: ""
-
-processing:
-  chunk_size: 256
-  context_chunk_radius: 0
-  markdown_preprocessor: ""
-
-providers:
-  ollama:
-    base_url: http://localhost:11434
-
-  vllm:
-    embeddings_base_url: ""
-    rerank_base_url: ""
-    qa_base_url: ""
-    research_base_url: ""
-```
-
-## Programmatic Configuration
-
-When using haiku.rag as a Python library, you can pass configuration directly to the `HaikuRAG` client:
-
-```python
-from haiku.rag.config import AppConfig
-from haiku.rag.client import HaikuRAG
-
-# Create custom configuration
-custom_config = AppConfig(
-    qa={"provider": "openai", "model": "gpt-4o"},
-    embeddings={"provider": "ollama", "model": "qwen3-embedding"},
-    processing={"chunk_size": 512}
-)
-
-# Pass configuration to the client
-client = HaikuRAG(config=custom_config)
-```
-
-If you don't pass a config, the client uses the global configuration loaded from your YAML file or defaults.
-
-This is useful for:
-- Jupyter notebooks
-- Python scripts
-- Testing with different configurations
-- Applications that need multiple clients with different configurations
-
-## File Monitoring
-
-Set directories to monitor for automatic indexing:
-
-```yaml
-monitor:
-  directories:
-    - /path/to/documents
-    - /another_path/to/documents
-```
-
-### Filtering Monitored Files
-
-Use gitignore-style patterns to control which files are monitored:
-
-```yaml
-monitor:
-  directories:
-    - /path/to/documents
-
-  # Exclude specific files or directories
-  ignore_patterns:
-    - "*draft*"         # Ignore files with "draft" in the name
-    - "temp/"           # Ignore temp directory
-    - "**/archive/**"   # Ignore all archive directories
-    - "*.backup"        # Ignore backup files
-
-  # Only include specific files (whitelist mode)
-  include_patterns:
-    - "*.md"            # Only markdown files
-    - "*.pdf"           # Only PDF files
-    - "**/docs/**"      # Only files in docs directories
-```
-
-**How patterns work:**
-
-1. **Extension filtering** - Only supported file types are considered
-2. **Include patterns** - If specified, only matching files are included (whitelist)
-3. **Ignore patterns** - Matching files are excluded (blacklist)
-4. **Combining both** - Include patterns are applied first, then ignore patterns
-
-**Common patterns:**
-
-```yaml
-# Only monitor markdown documentation, but ignore drafts
-monitor:
-  include_patterns:
-    - "*.md"
-  ignore_patterns:
-    - "*draft*"
-    - "*WIP*"
-
-# Monitor all supported files except in specific directories
-monitor:
-  ignore_patterns:
-    - "node_modules/"
-    - ".git/"
-    - "**/test/**"
-    - "**/temp/**"
-```
-
-Patterns follow [gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_format):
-- `*` matches anything except `/`
-- `**` matches zero or more directories
-- `?` matches any single character
-- `[abc]` matches any character in the set
-
-## Embedding Providers
-
-If you use Ollama, you can use any pulled model that supports embeddings.
-
-### Ollama (Default)
-
-```yaml
-embeddings:
-  provider: ollama
-  model: mxbai-embed-large
-  vector_dim: 1024
-```
-
-The Ollama base URL can be configured in your config file or via environment variable:
-
-```yaml
-providers:
-  ollama:
-    base_url: http://localhost:11434
-```
-
-Or via environment variable:
-
-```bash
-export OLLAMA_BASE_URL=http://localhost:11434
-```
-
-If not configured, it defaults to `http://localhost:11434`.
-
-!!! note
-    You can use a `.env` file in your project directory to set environment variables like `OLLAMA_BASE_URL` and API keys (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). These will be automatically loaded when running `haiku-rag` commands.
-
-### VoyageAI
-
-If you installed `haiku.rag` (full package), VoyageAI is already included. If you installed `haiku.rag-slim`, install with VoyageAI extras:
-
-```bash
-uv pip install haiku.rag-slim[voyageai]
-```
-
-```yaml
-embeddings:
-  provider: voyageai
-  model: voyage-3.5
-  vector_dim: 1024
-```
-
-Set your API key via environment variable:
-
-```bash
-export VOYAGE_API_KEY=your-api-key
-```
-
-### OpenAI
-
-OpenAI embeddings are included in the default installation:
-
-```yaml
-embeddings:
-  provider: openai
-  model: text-embedding-3-small  # or text-embedding-3-large
-  vector_dim: 1536
-```
-
-Set your API key via environment variable:
-
-```bash
-export OPENAI_API_KEY=your-api-key
-```
-
-### vLLM
-
-For high-performance local inference, you can use vLLM to serve embedding models with OpenAI-compatible APIs:
-
-```yaml
-embeddings:
-  provider: vllm
-  model: mixedbread-ai/mxbai-embed-large-v1
-  vector_dim: 512
-
-providers:
-  vllm:
-    embeddings_base_url: http://localhost:8000
-```
-
-**Note:** You need to run a vLLM server separately with an embedding model loaded.
-
-## Question Answering Providers
-
-Configure which LLM provider to use for question answering. Any provider and model supported by [Pydantic AI](https://ai.pydantic.dev/models/) can be used.
-
-### Ollama (Default)
-
-```yaml
-qa:
-  provider: ollama
-  model: gpt-oss
-```
-
-The Ollama base URL can be configured via the `OLLAMA_BASE_URL` environment variable, config file, or defaults to `http://localhost:11434`:
-
-```bash
-export OLLAMA_BASE_URL=http://localhost:11434
-```
-
-Or in your config file:
-
-```yaml
-providers:
-  ollama:
-    base_url: http://localhost:11434
-```
-
-### OpenAI
-
-OpenAI QA is included in the default installation:
-
-```yaml
-qa:
-  provider: openai
-  model: gpt-4o-mini  # or gpt-4, gpt-3.5-turbo, etc.
-```
-
-Set your API key via environment variable:
-
-```bash
-export OPENAI_API_KEY=your-api-key
-```
-
-### Anthropic
-
-Anthropic QA is included in the default installation:
-
-```yaml
-qa:
-  provider: anthropic
-  model: claude-3-5-haiku-20241022  # or claude-3-5-sonnet-20241022, etc.
-```
-
-Set your API key via environment variable:
-
-```bash
-export ANTHROPIC_API_KEY=your-api-key
-```
-
-### vLLM
-
-For high-performance local inference:
-
-```yaml
-qa:
-  provider: vllm
-  model: Qwen/Qwen3-4B  # Any model with tool support in vLLM
-
-providers:
-  vllm:
-    qa_base_url: http://localhost:8002
-```
-
-**Note:** You need to run a vLLM server separately with a model that supports tool calling loaded. Consult the specific model's documentation for proper vLLM serving configuration.
-
-### Other Providers
-
-Any provider supported by Pydantic AI can be used. Examples:
-
-```yaml
-# Google Gemini
-qa:
-  provider: gemini
-  model: gemini-1.5-flash
-
-# Groq
-qa:
-  provider: groq
-  model: llama-3.3-70b-versatile
-
-# Mistral
-qa:
-  provider: mistral
-  model: mistral-small-latest
-```
-
-See the [Pydantic AI documentation](https://ai.pydantic.dev/models/) for the complete list of supported providers and models.
-
-## Reranking
-
-Reranking improves search quality by re-ordering the initial search results using specialized models. When enabled, the system retrieves more candidates (3x the requested limit) and then reranks them to return the most relevant results.
-
-Reranking is **disabled by default** (`provider: ""`) for faster searches. You can enable it by configuring one of the providers below.
-
-### MixedBread AI
-
-If you installed `haiku.rag` (full package), MxBAI is already included. If you installed `haiku.rag-slim`, add the mxbai extra:
-
-```bash
-uv pip install haiku.rag-slim[mxbai]
-```
-
-Then configure:
-
-```yaml
-reranking:
-  provider: mxbai
-  model: mixedbread-ai/mxbai-rerank-base-v2
-```
-
-### Cohere
-
-If you installed `haiku.rag` (full package), Cohere is already included. If you installed `haiku.rag-slim`, add the cohere extra:
-
-```bash
-uv pip install haiku.rag-slim[cohere]
-```
-
-Then configure:
-
-```yaml
-reranking:
-  provider: cohere
-  model: rerank-v3.5
-```
-
-Set your API key via environment variable:
-
-```bash
-export CO_API_KEY=your-api-key
-```
-
-### Zero Entropy
-
-If you installed `haiku.rag` (full package), Zero Entropy is already included. If you installed `haiku.rag-slim`, add the zeroentropy extra:
-
-```bash
-uv pip install haiku.rag-slim[zeroentropy]
-```
-
-Then configure:
-
-```yaml
-reranking:
-  provider: zeroentropy
-  model: zerank-1  # Currently the only available model
-```
-
-Set your API key via environment variable:
-
-```bash
-export ZEROENTROPY_API_KEY=your-api-key
-```
-
-### vLLM
-
-For high-performance local reranking using dedicated reranking models:
-
-```yaml
-reranking:
-  provider: vllm
-  model: mixedbread-ai/mxbai-rerank-base-v2
-
-providers:
-  vllm:
-    rerank_base_url: http://localhost:8001
-```
-
-**Note:** vLLM reranking uses the `/rerank` API endpoint. You need to run a vLLM server separately with a reranking model loaded. Consult the specific model's documentation for proper vLLM serving configuration.
-
-## Other Settings
-
-### Database and Storage
-
-By default, `haiku.rag` uses a local LanceDB database:
-
-```yaml
-storage:
-  data_dir: /path/to/data  # Empty = use default platform location
-```
-
-For remote storage, use the `lancedb` settings with various backends:
-
-```yaml
-# LanceDB Cloud
-lancedb:
-  uri: db://your-database-name
-  api_key: your-api-key
-  region: us-west-2  # optional
-
-# Amazon S3
-lancedb:
-  uri: s3://my-bucket/my-table
-# Use AWS credentials or IAM roles
-
-# Azure Blob Storage
-lancedb:
-  uri: az://my-container/my-table
-# Use Azure credentials
-
-# Google Cloud Storage
-lancedb:
-  uri: gs://my-bucket/my-table
-# Use GCP credentials
-
-# HDFS
-lancedb:
-  uri: hdfs://namenode:port/path/to/table
-```
-
-Authentication is handled through standard cloud provider credentials (AWS CLI, Azure CLI, gcloud, etc.) or by setting `api_key` for LanceDB Cloud.
-
-**Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization is still performed locally.
-
-#### Disable database auto-creation
-
-By default, haiku.rag creates the local LanceDB directory and required tables on first use. To prevent accidental database creation and fail fast if a database hasn't been set up yet:
-
-```yaml
-storage:
-  disable_autocreate: true
-```
-
-When enabled, for local paths, haiku.rag errors if the LanceDB directory does not exist, and it will not create parent directories.
-
-### Document Processing
-
-```yaml
-processing:
-  # Chunk size for document processing
-  chunk_size: 256
-
-  # Number of adjacent chunks to include before/after retrieved chunks for context
-  # 0 = no expansion (default), 1 = include 1 chunk before and after, etc.
-  # When expanded chunks overlap or are adjacent, they are automatically merged
-  # into single chunks with continuous content to eliminate duplication
-  context_chunk_radius: 0
-
-  # Optional dotted path or file path to a callable that preprocesses
-  # markdown content before chunking
-  markdown_preprocessor: ""
-
-storage:
-  # Vacuum retention threshold (seconds) for automatic cleanup
-  # When documents are added/updated, old table versions older than this are removed
-  # Default: 60 seconds (safe for concurrent connections)
-  # Set to 0 for aggressive cleanup (removes all old versions immediately)
-  vacuum_retention_seconds: 60
-```
-
-#### Markdown Preprocessor
-
-Optionally preprocess Markdown before chunking by pointing to a callable that receives and returns Markdown text. This is useful for normalizing content, stripping boilerplate, or applying custom transformations before chunk boundaries are computed.
-
-```yaml
-processing:
-  # A callable path in one of these formats:
-  # - package.module:func
-  # - package.module.func
-  # - /abs/or/relative/path/to/file.py:func
-  markdown_preprocessor: my_pkg.preprocess:clean_md
-```
-
-!!! note
-    - The function signature should be `def clean_md(text: str) -> str` or `async def clean_md(text: str) -> str`.
-    - If the function raises or returns a non-string, haiku.rag logs a warning and proceeds without preprocessing.
-    - The preprocessor affects only the chunking pipeline. The stored document content remains unchanged.
-
-Example implementation:
-
-```python
-# my_pkg/preprocess.py
-def clean_md(text: str) -> str:
-    # strip HTML comments and collapse multiple blank lines
-    lines = [line for line in text.splitlines() if not line.strip().startswith("<!--")]
-    out = []
-    for line in lines:
-        if line.strip() == "" and (out and out[-1] == ""):
-            continue
-        out.append(line)
-    return "\n".join(out)
-```

docs/configuration/index.md

@@ -0,0 +1,211 @@
+# Configuration
+
+Configuration is done through YAML configuration files.
+
+!!! note
+    haiku.rag enforces one hard rule on existing databases: the embedding `vector_dim` in your config must match the value stored in the db. A mismatch exits with `ConfigMismatchError` and you must **rebuild** to apply the change (see [Rebuild Database](../cli.md#rebuild-database)).
+
+    Opening a database never writes to it, so the stored embedding identity is left untouched. Changing only `provider` or `name` (e.g. switching from Ollama to vLLM serving the same model) is treated as soft drift: read-only opens log a warning and continue, while writable opens exit with `ConfigMismatchError`. Reconcile the stored identity with your config by running `haiku-rag rebuild --set-embedder` (see [Rebuild Database](../cli.md#rebuild-database)). If the change was unintentional, revert your config instead.
+
+## Getting Started
+
+Generate a configuration file with defaults:
+
+```bash
+haiku-rag init-config
+```
+
+This creates a `haiku.rag.yaml` file in your current directory with all available settings.
+
+## Configuration File Locations
+
+`haiku.rag` searches for configuration files in this order:
+
+1. Path specified via `--config` flag: `haiku-rag --config /path/to/config.yaml <command>`
+2. `./haiku.rag.yaml` (current directory)
+3. Platform-specific user directory:
+    - **Linux**: `~/.local/share/haiku.rag/haiku.rag.yaml`
+    - **macOS**: `~/Library/Application Support/haiku.rag/haiku.rag.yaml`
+    - **Windows**: `C:/Users/<USER>/AppData/Roaming/haiku.rag/haiku.rag.yaml`
+
+## Environment Variables
+
+Any string value can reference an environment variable, so secrets stay out of the file and one config can serve multiple deployments:
+
+```yaml
+ingester:
+  queue:
+    dburi: postgresql+asyncpg://haiku:${POSTGRES_PASSWORD}@db:5432/haiku_rag
+```
+
+- `${VAR}` is replaced with the value of `VAR`. If `VAR` is unset, loading fails with an error naming the variable.
+- `${VAR:-default}` uses `default` when `VAR` is unset or empty.
+- `$$` produces a literal `$`.
+
+Substitution happens after the YAML is parsed, so a value containing `:`, `@`, or `#` fills the string verbatim and never changes the document structure.
+
+## Minimal Configuration
+
+A minimal configuration file with defaults:
+
+```yaml
+# haiku.rag.yaml
+environment: production
+
+embeddings:
+  model:
+    provider: ollama
+    name: qwen3-embedding:4b
+    vector_dim: 2560
+
+qa:
+  model:
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: true
+```
+
+## Complete Configuration Example
+
+```yaml
+# haiku.rag.yaml
+environment: production
+
+storage:
+  data_dir: ""  # Empty = use default platform location
+  vacuum_retention_seconds: 86400
+
+ingester:
+  sources:
+    - type: fs
+      id: local-docs
+      root: /path/to/documents
+      ignore_patterns: []  # Gitignore-style patterns to exclude
+      include_patterns: []  # Gitignore-style patterns to include
+      delete_orphans: true
+
+lancedb:
+  databases: {}  # Name-to-location map; empty places haiku.rag under data_dir
+  api_key: ""  # LanceDB Cloud (db://) credentials
+  region: ""
+
+embeddings:
+  model:
+    provider: ollama
+    name: qwen3-embedding:4b
+    vector_dim: 2560
+
+reranking:
+  # Omit this section, or set `model: null`, to disable reranking.
+  model:
+    provider: cross-encoder  # cross-encoder, cohere, zeroentropy, vllm, jina, jina-local
+    name: cross-encoder/ms-marco-MiniLM-L-6-v2
+  multimodal: false  # vllm only: send picture chunks to the reranker as images
+
+qa:
+  model:
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: true
+    temperature: 0.3
+  max_searches: 5
+
+search:
+  limit: 5                     # Default number of results to return
+  max_context_chars: 5000     # Maximum characters in expanded context
+  vector_index_metric: cosine  # cosine or l2
+  vector_refine_factor: 30
+
+doctor:
+  duplicates:                    # Near-duplicate document detection (doctor command)
+    similarity_threshold: 0.97   # cosine cutoff on document embedding centroids
+    min_chunks: 3                # documents with fewer chunks are excluded
+
+prompts:
+  domain_preamble: ""  # Prepended to capability instructions
+
+processing:
+  converter: docling-local  # docling-local or docling-serve
+  chunker: docling-local    # docling-local or docling-serve
+  chunker_type: hybrid      # hybrid or hierarchical
+  chunk_size: 256
+  chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B"
+  chunking_merge_peers: true
+  chunking_use_markdown_tables: false
+  auto_title: false              # Auto-generate titles on ingestion
+  title_model:
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: false
+    temperature: 0.3
+    max_tokens: 100
+  conversion_options:
+    do_ocr: true
+    force_ocr: false
+    ocr_lang: []
+    do_table_structure: true
+    table_mode: accurate
+    table_cell_matching: true
+    images_scale: 2.0
+
+providers:
+  ollama:
+    base_url: http://localhost:11434
+
+  docling_serve:
+    base_url: http://localhost:5001
+    api_key: ""
+    timeout: 300
+```
+
+## Programmatic Configuration
+
+When using haiku.rag as a Python library, you can pass configuration directly to the `HaikuRAG` client:
+
+```python
+from haiku.rag.config import AppConfig
+from haiku.rag.config.models import EmbeddingModelConfig, ModelConfig, QAConfig, EmbeddingsConfig
+from haiku.rag.client import HaikuRAG
+
+# Create custom configuration
+custom_config = AppConfig(
+    qa=QAConfig(
+        model=ModelConfig(
+            provider="openai",
+            name="gpt-4o",
+            temperature=0.3
+        )
+    ),
+    embeddings=EmbeddingsConfig(
+        model=EmbeddingModelConfig(
+            provider="ollama",
+            name="qwen3-embedding:4b",
+            vector_dim=2560
+        )
+    ),
+    processing={"chunk_size": 512}
+)
+
+# Pass configuration to the client
+async with HaikuRAG(config=custom_config) as client:
+    ...
+```
+
+If you don't pass a config, the client uses the global configuration loaded from your YAML file or defaults.
+
+This is useful for:
+- Jupyter notebooks
+- Python scripts
+- Testing with different configurations
+- Applications that need multiple clients with different configurations
+
+## Configuration Topics
+
+For detailed configuration of specific topics, see:
+
+- **[Providers](providers.md)** - Model settings and provider-specific configuration (embeddings, reranking)
+- **[Search and Question Answering](qa.md)** - Search settings and question answering
+- **[Document Processing](processing.md)** - Document conversion and chunking
+- **[Ingester](../ingester.md)** - Continuous ingestion from filesystem, HTTP, S3, and WebDAV sources
+- **[Storage](storage.md)** - Database, remote storage, and vector indexing
+- **[Prompts](prompts.md)** - Customize agent prompts for your domain

docs/configuration/processing.md

@@ -0,0 +1,419 @@
+# Document Processing
+
+This guide covers how haiku.rag converts and chunks documents. Continuous
+ingestion (watching directories, polling HTTP / S3 / WebDAV sources) lives
+in the [ingester](../ingester.md) service.
+
+## Document Processing
+
+Configure how documents are converted and chunked:
+
+```yaml
+processing:
+  # Chunking configuration
+  chunk_size: 256                            # Maximum tokens per chunk
+
+  # Converter selection
+  converter: docling-local                   # docling-local or docling-serve
+
+  # Chunker selection and configuration
+  chunker: docling-local                     # docling-local or docling-serve
+  chunker_type: hybrid                       # hybrid or hierarchical
+  chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B"  # HuggingFace model for tokenization
+  chunking_merge_peers: true                 # Merge undersized successive chunks
+  chunking_use_markdown_tables: false        # Use markdown tables vs narrative format
+
+  # PDF /EmbeddedFiles attachments
+  extract_pdf_attachments: true              # Ingest embedded files as separate Documents
+
+  # Automatic title generation
+  auto_title: false                          # Auto-generate titles on ingestion
+  title_model:                               # LLM for title generation (fallback)
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: false
+
+  # Conversion options (works with both local and remote converters)
+  conversion_options:
+    # OCR settings
+    do_ocr: true                             # Enable OCR for bitmap content
+    force_ocr: false                         # Replace existing text with OCR
+    ocr_engine: auto                         # OCR engine: auto, easyocr, rapidocr, tesseract, tesserocr, ocrmac
+    ocr_lang: []                             # OCR languages (e.g., ["en", "fr", "de"])
+
+    # Table extraction
+    do_table_structure: true                 # Extract table structure
+    table_mode: accurate                     # fast or accurate
+    table_cell_matching: true                # Match table cells back to PDF cells
+
+    # Image settings
+    images_scale: 2.0                        # Image scale factor
+    generate_page_images: true               # Include rendered page images (for visualize_chunk)
+
+    # VLM settings used when processing.pictures == "description" (see "Picture Handling" below)
+    picture_description:
+      model:
+        provider: ollama
+        name: qwen3.8
+  pictures: image                            # none | description | image
+```
+
+### Local vs Remote Processing
+
+**Local processing** (default):
+
+- Uses `docling` library locally
+- No external dependencies
+- Good for development and small workloads
+
+**Remote processing** (docling-serve):
+
+- Offloads processing to docling-serve API
+- Better for heavy workloads and production
+- Requires docling-serve instance (see [Remote processing setup](../remote-processing.md))
+
+To use remote processing:
+
+```yaml
+processing:
+  converter: docling-serve
+  chunker: docling-serve
+
+providers:
+  docling_serve:
+    base_url: http://localhost:5001
+    api_key: "your-api-key"  # Optional
+```
+
+`base_url` also accepts a list — jobs round-robin across the entries, with
+each job's submit / poll / result pinned to one instance (task IDs are
+instance-local):
+
+```yaml
+providers:
+  docling_serve:
+    base_url:
+      - http://gpu-1:5001
+      - http://cpu-1:5001
+      - http://cpu-2:5001
+    max_attempts: 3
+    circuit_breaker:
+      failure_threshold: 3
+      cooldown_s: 30.0
+```
+
+The round-robin counter is per-process — multiple concurrent ingester or
+client processes pick independently, so the distribution evens out over many
+jobs without coordination. When a listed instance crashes or returns 5xx, the
+client fails the request over to another instance (up to `max_attempts`) and
+opens that instance's circuit breaker so subsequent jobs skip it until its
+`cooldown_s` elapses. An external load balancer can only front docling-serve in
+RQ mode (shared Redis task state); with the default standalone instances the
+submit / poll / result trio is instance-pinned, so the failover and health
+checks live in the client.
+
+**Tuning `ingester.workers.worker_count` for docling-serve users**: convert
+is usually the throughput ceiling — a default docling-serve instance
+processes one task at a time (configurable via `DOCLING_SERVE_ENG_LOC_NUM_WORKERS`
+if you've set it). A reasonable starting point for `worker_count` is **1–2 ×
+the number of `docling_serve.base_url` entries**: enough to overlap fetch /
+embed / store of one job with the convert of another, without piling jobs
+into docling-serve's internal queue beyond what its workers can chew through.
+The ingester logs the worker / source / docling-serve counts on startup so
+you can eyeball the ratio.
+
+Conversion options work identically for both local and remote processing.
+
+### Large PDFs and docling memory
+
+Docling's parser is memory-hungry and has confirmed leaks in current versions
+([docling #2209](https://github.com/docling-project/docling/issues/2209),
+[#1343](https://github.com/docling-project/docling/issues/1343),
+[#2954](https://github.com/docling-project/docling/issues/2954);
+[docling-serve #366](https://github.com/docling-project/docling-serve/issues/366),
+[#474](https://github.com/docling-project/docling-serve/issues/474)).
+Single-pass conversion of 400-page PDFs can OOM a workstation in local mode,
+and long-running docling-serve containers see RSS grow monotonically.
+
+Mitigation in haiku.rag — set `processing.split_pages`:
+
+```yaml
+processing:
+  split_pages: 10                  # 0 disables (default)
+```
+
+When `split_pages > 0`, PDFs are split at the byte level into N-page slices
+(using pypdfium2, already bundled), each slice converted independently, then
+merged back via `DoclingDocument.concatenate` — preserving page numbers and
+re-indexing `self_ref` values across slices. Peak memory per conversion is
+bounded by one slice's working set rather than the whole document; in
+docling-serve mode each slice is also an independent task that lets the
+server release task-local state between requests.
+
+Recommendation: `10` is a sensible starting point for any consistently-large
+PDF workload. Smaller slices reduce peak memory but multiply task overhead
+(per-slice docling startup + HTTP round-trips for docling-serve). Cross-page
+references (named destinations, multi-page link annotations) are dropped at
+the split — accepted loss; haiku.rag doesn't surface them downstream.
+
+**Operational note for long-running ingest**: even with `split_pages`,
+docling's per-process leak rate is non-zero. For deployments running
+continuously:
+
+- *docling-serve mode*: set `mem_limit` on the container in Compose
+  (or `resources.limits.memory` in Kubernetes) plus `restart: unless-stopped`
+  so the kernel OOM-kills and the runtime restarts. Run multiple
+  docling-serve replicas behind the round-robin `base_url` list above so a
+  restart of one doesn't stop ingest.
+- *docling-local mode*: the leak is inside the `haiku-ingester` process
+  itself. Apply the same `mem_limit` + restart policy to the ingester
+  container. Restarts are graceful — in-flight jobs land in the queue's
+  reaper window and resume on next start.
+
+**Note:** When using `chunker: docling-serve`, OCR options (`do_ocr`, `force_ocr`, `ocr_engine`, `ocr_lang`) from `conversion_options` are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail. Set `do_ocr: false` to disable OCR entirely.
+
+### Conversion Options
+
+The `conversion_options` section allows fine-grained control over document conversion. These options work with both `docling-local` and `docling-serve` converters.
+
+#### OCR Settings
+
+```yaml
+conversion_options:
+  do_ocr: true          # Enable OCR for bitmap/scanned content
+  force_ocr: false      # Replace all text with OCR output
+  ocr_engine: auto      # OCR engine selection
+  ocr_lang: []          # List of OCR languages, e.g., ["en", "fr", "de"]
+```
+
+- **do_ocr**: When `true`, applies OCR to images and scanned pages. Disable for faster processing if documents contain only native text.
+- **force_ocr**: When `true`, replaces existing text layers with OCR output. Useful for documents with poor text extraction.
+- **ocr_engine**: Select the OCR engine to use. Options:
+  - `auto` (default): Automatically select the best available engine
+  - `easyocr`: EasyOCR - supports many languages, good accuracy
+  - `rapidocr`: RapidOCR - fast processing
+  - `tesseract`: Tesseract OCR
+  - `tesserocr`: Tesseract via tesserocr Python binding
+  - `ocrmac`: macOS native OCR (macOS only)
+- **ocr_lang**: List of language codes for OCR. Empty list uses default language detection. Examples: `["en"]`, `["en", "fr", "de"]`.
+
+#### Table Extraction
+
+```yaml
+conversion_options:
+  do_table_structure: true    # Extract structured table data
+  table_mode: accurate        # fast or accurate
+  table_cell_matching: true   # Match cells back to PDF
+```
+
+- **do_table_structure**: When `true`, extracts table structure. Disable for faster processing if tables aren't important.
+- **table_mode**:
+  - `accurate`: Better table structure recognition (slower)
+  - `fast`: Faster processing with simpler table detection
+- **table_cell_matching**: When `true`, matches detected table cells back to PDF cells. Disable if tables have merged cells across columns.
+
+#### Image Settings
+
+```yaml
+conversion_options:
+  images_scale: 2.0               # Image resolution scale factor
+  generate_page_images: true      # Include rendered page images
+  fetch_remote_images: true       # Fetch external <img src> URLs in HTML/MD
+```
+
+- **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0.
+- **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size.
+- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `<img src="https://...">` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to `docling-local`. **docling-serve doesn't fetch external `<img>` URLs** (the `ConvertDocumentsOptions` API exposes no equivalent flag, and HTML falls through to docling's `fetch_images=False` default); HTML ingested via docling-serve produces picture items with `picture_data=NULL`. Use `converter: docling-local` if you need image bytes from HTML/Markdown.
+
+#### External image fetching
+
+For HTML and Markdown inputs, docling fetches images referenced by URL when `fetch_remote_images: true`. Pictures end up in `document_items.picture_data` alongside the ones extracted from PDF/DOCX/PPTX. Inherited from docling:
+
+- **SSRF guard**: hostnames must resolve to a global IP. Loopback, private (RFC1918), link-local, reserved, multicast, and unspecified addresses are rejected.
+- **Size cap**: 20 MB per image (sent as a `Range` header), enforced again when streaming the response body.
+- **Timeouts**: 5 s connect, 30 s read.
+- **SVGs are skipped** (PIL cannot rasterize them).
+- **`data:` URIs** are decoded inline (no network).
+- **`file://` URIs** are *not* fetched. `enable_local_fetch` stays off to keep the SSRF surface narrow for arbitrary HTML/MD content.
+
+Per-image failures (404, timeout, oversized, unreadable) leave that picture as a placeholder with `picture_data=NULL`. The rest of the document still ingests.
+
+**Scope of conversion options across formats:**
+
+| Input | OCR / table options | `images_scale` / `generate_page_images` | `pictures` | `fetch_remote_images` |
+|---|---|---|---|---|
+| `.pdf` | ✅ | ✅ | ✅ | n/a |
+| `.png` / `.jpg` / `.jpeg` / `.bmp` / `.tiff` / `.webp` | ✅ | ✅ | ✅ | n/a |
+| `.html` / `.xhtml` | n/a (markup-based) | n/a | ✅ on embedded pictures | ✅ |
+| `.md` / `.qmd` / `.rmd` | n/a | n/a | ✅ on embedded pictures | ✅ (only `<img>` HTML blocks; native `![alt](url)` syntax is not fetched by docling) |
+| `.docx` / `.pptx` | n/a | n/a | ✅ on embedded pictures | n/a |
+| Other (`.csv`, `.xlsx`, `.adoc`, `.tex`, `.xml`) | n/a | n/a | n/a | n/a |
+
+#### Picture Handling
+
+`processing.pictures` picks one of three modes:
+
+| Mode | Picture-image generation in docling | Bytes stored in `document_items.picture_data` | VLM runs at ingest |
+|---|---|---|---|
+| `none` | off | no | no |
+| `description` | on | yes | yes |
+| `image` (default) | on | yes | no |
+
+Not every picture becomes a picture chunk. Identical picture bytes within a document produce a single chunk, so a watermark or logo repeated on every page embeds once. Pictures smaller than `processing.min_picture_size` pixels on their smaller side (default 64, `0` disables) are skipped entirely. Filtered pictures keep their bytes in `document_items`, so context expansion and vision QA still see them.
+
+Use `none` when you don't need picture content (e.g. very large reference manuals where RAM is tight). Use `description` to weave VLM-generated text into chunk content and keep bytes for later. Use `image` (default) to keep bytes without paying the VLM cost. The prompt is configurable under `prompts.picture_description`. See [Prompts](prompts.md).
+
+```yaml
+processing:
+  pictures: description           # none | description | image
+  conversion_options:
+    picture_description:          # only consulted when pictures == "description"
+      model:
+        provider: ollama          # any OpenAI-compatible /v1/chat/completions provider
+        name: qwen3.8
+      timeout: 90
+      max_tokens: 200
+```
+
+!!! warning "Breaking change"
+    `processing.conversion_options.picture_description.enabled` is replaced by `processing.pictures`. Map `enabled: true` → `pictures: description`, `enabled: false` → `pictures: image`. The pre-April-30 `generate_picture_images` flag also no longer exists. Use `pictures: none` for the old opt-out.
+
+**Switching modes on an existing database** doesn't require reingesting when the bytes are already stored:
+
+- `image` → `description`: `haiku-rag rebuild --descriptions` runs the VLM over stored bytes and re-chunks. Skips the docling parse entirely.
+- `description` → `image`: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions.
+- Switching to/from `none`: a full reingest is needed since the bytes either weren't stored or need to be discarded.
+
+When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag. See [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve).
+
+#### Pictures × embedder × QA model: how the pieces compose
+
+Three independent settings drive ingest, retrieval, and QA:
+
+| Setting | Question it answers | Values |
+|---|---|---|
+| `processing.pictures` | Generate and/or describe pictures at ingest? | `none` / `description` / `image` (default) |
+| `embeddings.model.multimodal` | Can the embedder index image content? | `false` (default, text-only) / `true` (supported on `vllm`, `voyageai`, `cohere`) |
+| `qa.model.vision` | Can the QA model interpret images? | `false` / `true` (default) |
+
+The Embedder column below is driven by `embeddings.model.multimodal`, not the provider name — a vision-capable model under a text-only configuration still indexes no images, and an image-only document then produces zero chunks. See [Multimodal embedders](providers.md#multimodal-embedders).
+
+**What gets stored** by `pictures` × embedder:
+
+| `pictures` | Embedder | Text chunks | Synthetic picture chunks |
+|---|---|---|---|
+| `none` | any | text only (caption/surrounding) | none |
+| `image` | text-only | text only (caption/surrounding) | none |
+| `image` | multimodal | text only | one per distinct picture, vector = image embedding |
+| `description` | text-only | text + descriptions | none |
+| `description` | multimodal | text + descriptions | one per distinct picture, vector = image embedding |
+
+**What QA receives** at search time:
+
+- `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose).
+- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`. The model reads figures directly. Requires `pictures != none` so the bytes exist.
+
+`qa.model.vision` is independent of ingestion. Flipping it never requires reingesting. It declares what the model can read: the default `qwen3.8` is vision-capable, so the default is `true`. Set it `false` when pointing `qa.model` at a text-only model, where `true` causes silent acceptance and confabulation on Ollama and a 400 on OpenAI.
+
+**Recommended combinations:**
+
+| Use case | `processing.pictures` | Embedder | `qa.model.vision` |
+|---|---|---|---|
+| Pure text RAG, no figures, lowest RAM | `none` | text-only | `false` |
+| Text RAG, store figure bytes for later | `image` | text-only | `false` |
+| Text RAG, figures answered through descriptions | `description` | text-only | `false` |
+| Vision QA on figure-rich docs (no cross-modal search) | `image` or `description` | text-only | `true` |
+| Cross-modal search + vision QA | `image` or `description` | multimodal | `true` |
+| Cross-modal search, text QA only | `description` | multimodal | `false` |
+
+### Chunking Strategies
+
+**Hybrid chunking** (default):
+- Structure-aware chunking
+- Respects document boundaries
+- Best for most use cases
+
+**Hierarchical chunking**:
+- Creates hierarchical chunk structure
+- Preserves document hierarchy
+- Useful for complex documents
+
+### Chunk Size
+
+```yaml
+processing:
+  chunk_size: 256  # Maximum tokens per chunk
+```
+
+Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](qa.md#search-settings).
+
+### Table Serialization
+
+Control how tables are represented in chunks:
+
+```yaml
+processing:
+  chunking_use_markdown_tables: false  # Default: narrative format
+```
+
+- `false`: Tables as narrative text ("Value A, Column 2 = Value B")
+- `true`: Tables as markdown (preserves table structure)
+
+### Automatic Title Generation
+
+Enable automatic title generation during document ingestion:
+
+```yaml
+processing:
+  auto_title: true
+  title_model:
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: false
+```
+
+When `auto_title` is enabled, haiku.rag attempts to extract a title for each document during ingestion using a two-tier approach:
+
+1. **Structural extraction** (free, no model calls): Scans the DoclingDocument for semantic labels (HTML `<title>` tags, `<h1>` headings, PDF title blocks, and section headers)
+2. **LLM fallback**: When no structural title is found (e.g., plain text), generates a title using the configured `title_model`
+
+Priority order: HTML `<title>` (furniture layer) → h1/PDF title (body layer) → first section header → LLM generation.
+
+Explicit titles passed via `title=` parameter always take precedence and are never overridden. When updating documents, existing titles are preserved. Auto-generation only applies to untitled documents.
+
+To generate titles for existing untitled documents, use [`rebuild --title-only`](../cli.md#rebuild-database).
+
+### PDF Embedded Attachments
+
+A PDF can carry other files inside it via the `/EmbeddedFiles` table (signed memos, appendices, supporting documents). With `extract_pdf_attachments: true` (the default), each embedded file is ingested as a separate Document linked to the wrapper through `metadata.parent_uri`:
+
+```yaml
+processing:
+  extract_pdf_attachments: true
+```
+
+```python
+# After ingesting a PDF with two attachments:
+parent = await client.create_document_from_source("/path/to/parent.pdf")
+children = await client.list_documents(
+    filter=f"metadata LIKE '%\"parent_uri\": \"{parent.uri}\"%'"
+)
+# children: 2 Documents, each with parent.uri in metadata.parent_uri,
+# URIs like file:///path/to/parent.pdf#attachment=memo.pdf
+```
+
+Behavior:
+
+- Children inherit the standard ingest metadata (`content_type`, `md5`, `source_revision`) plus `parent_uri`.
+- Re-ingesting the wrapper reconciles its current attachment set against existing children: new files are added, changed bytes update in place, and dropped names are deleted.
+- `delete_document(parent_id)` cascades through `parent_uri` and removes all children.
+- Nested attachments (a PDF whose attachment is itself a PDF with attachments) recurse up to 3 levels. Deeper chains log a warning and skip.
+- Attachments whose extension or content type the converter does not support log a warning and are skipped without aborting the rest of the set.
+
+Set `extract_pdf_attachments: false` to ingest only the wrapper.
+
+## Continuous ingestion
+
+For automatic ingestion of local directories, S3 buckets, or HTTP
+sources (with filtering, retries, and a dead-letter queue), see the
+[Ingester](../ingester.md) page.

docs/configuration/prompts.md

@@ -0,0 +1,76 @@
+# Prompt Customization
+
+Customize the prompts used by haiku.rag's capabilities to match your domain.
+
+## Configuration
+
+```yaml
+prompts:
+  # Domain context prepended to capability instructions
+  domain_preamble: |
+    This knowledge base contains technical documentation for the Helios solar panel
+    system, including installation manuals, maintenance procedures, and safety guidelines.
+    Questions about "the system" or unqualified specs refer to the Helios panel.
+
+  # VLM prompt for image description during conversion.
+  # Omit the key to use the built-in prompt.
+  picture_description: |
+    Describe this figure in two sentences, naming any axis labels and units.
+```
+
+## Domain Preamble
+
+The `domain_preamble` field provides **domain context** prepended to the RAG and analysis capability instructions. Use this to:
+
+- Describe what the knowledge base contains
+- Clarify domain-specific terminology
+- Provide context that helps the model interpret ambiguous queries
+
+**Important:** `domain_preamble` is for domain context, not behavioral instructions. Descriptions of subject matter, terminology, and content scope belong here. Applications can add behavioral guidance through normal Pydantic AI agent instructions.
+
+**Example:**
+
+```yaml
+prompts:
+  domain_preamble: |
+    This knowledge base contains product documentation, API references,
+    and troubleshooting guides for Acme Corp's cloud platform.
+    "Deployment" refers to Acme's managed deployment service, not general CI/CD.
+```
+
+## Picture Description Prompt
+
+Customize the prompt used when generating VLM descriptions for embedded images during document conversion. This prompt is sent to the configured Vision Language Model for each image.
+
+**Default prompt:**
+
+```
+Describe this image for a blind user. State the image type (screenshot, chart, photo, etc.),
+what it depicts, any visible text, and key visual details. Be concise and accurate.
+```
+
+**Custom example:**
+
+```yaml
+prompts:
+  picture_description: |
+    Describe this image for a document search system.
+    Focus on: image type, main content, any text, key visual elements.
+    Be concise and factual.
+```
+
+The prompt is used when `processing.pictures` is `"description"`. See [Picture Handling](processing.md#picture-handling) for full configuration.
+
+## Programmatic Configuration
+
+```python
+from haiku.rag.config import AppConfig
+from haiku.rag.config.models import PromptsConfig
+
+config = AppConfig(
+    prompts=PromptsConfig(
+        domain_preamble="This knowledge base contains Acme Corp product documentation and API references.",
+        picture_description="Describe this image for search indexing.",
+    )
+)
+```

docs/configuration/providers.md

@@ -0,0 +1,554 @@
+# Providers
+
+haiku.rag supports multiple AI providers for embeddings, question answering, and reranking. This guide covers provider-specific configuration and setup.
+
+!!! note
+    You can use a `.env` file in your project directory to set environment variables like `OLLAMA_BASE_URL` and API keys (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). These will be automatically loaded when running `haiku-rag` commands.
+
+## Model Settings
+
+Configure model behavior for the `qa` and `analysis` capabilities. These settings apply to any provider that supports them.
+
+### Basic Settings
+
+```yaml
+qa:
+  model:
+    provider: ollama
+    name: qwen3.8
+    temperature: 0.3
+    max_tokens: 500
+```
+
+**Available options:**
+
+- **temperature**: Sampling temperature (0.0-1.0+). Defaults vary by task: 0.3 for QA and title generation, 0.0 for analysis and picture description.
+  - Lower (0.0-0.3): Deterministic, focused responses
+  - Medium (0.4-0.7): Balanced
+  - Higher (0.8-1.0+): Creative, varied responses
+- **max_tokens**: Maximum tokens in response. Default: unset (provider default), except title generation (100).
+- **enable_thinking**: Control reasoning behavior (see below)
+- **base_url**: Custom endpoint for OpenAI-compatible servers (vLLM, LM Studio, etc.)
+- **api_key**: Key for this endpoint, overriding the provider's environment variable (see [Per-endpoint API keys](#per-endpoint-api-keys))
+- **extra_body**: Raw dict forwarded to the model SDK (see [Raw Provider Pass-through](#raw-provider-pass-through))
+
+### Per-endpoint API keys
+
+The `openai` provider reads `OPENAI_API_KEY`, so several `openai`-compatible endpoints in one config would otherwise share a single key. Set `api_key` per model to give each its own, and keep the secret in the environment with [variable expansion](index.md#environment-variables):
+
+```yaml
+qa:
+  model:
+    provider: openai
+    name: some-model
+    base_url: https://vendor-a.example/v1
+    api_key: ${VENDOR_A_KEY}
+
+embeddings:
+  model:
+    provider: openai
+    name: some-embedding-model
+    vector_dim: 1024
+    base_url: https://vendor-b.example/v1
+    api_key: ${VENDOR_B_KEY}
+```
+
+`api_key` is honored on the `openai` and `ollama` providers, on `vllm` embedders and rerankers, and on the picture-description VLM endpoint (which otherwise falls back to `OPENAI_API_KEY` only for the public OpenAI endpoint, never for a custom `base_url`). Other providers (`anthropic`, `cohere`, `voyageai`, …) reach their vendor SDK by name and read their own environment variable; setting `api_key` there raises rather than being dropped silently.
+
+### Thinking Control
+
+The `enable_thinking` setting controls whether models use explicit reasoning steps before answering.
+
+```yaml
+qa:
+  model:
+    enable_thinking: true   # Better grounded answers
+```
+
+**Values:**
+- `false`: Disable reasoning for faster responses
+- `true`: Enable reasoning for complex tasks
+- Not set: Use model defaults
+
+**Provider support:**
+
+See the [Pydantic AI thinking documentation](https://ai.pydantic.dev/thinking/) for detailed provider support. haiku.rag supports thinking control for:
+
+- **OpenAI**: Reasoning models (o1, o3, gpt-oss)
+- **Anthropic**: All Claude models
+- **Google**: Gemini models with thinking support
+- **Groq**: Models with reasoning capabilities
+- **Bedrock**: Claude, Qwen, and `gpt-oss` models. Bedrock Converse does not serve the proprietary OpenAI models, so configuring one raises an error. Reach those through `provider: bedrock-mantle`.
+- **Ollama**: Any model with a thinking capability. `enable_thinking` maps to `reasoning_effort`: `false` sends `none` (`low` for `gpt-oss`, whose template has no `none` level), `true` sends `high`.
+- **vLLM**: Models with a pydantic-ai reasoning profile (gpt-oss). Qwen3, Gemma, and similar templates ignore the OpenAI `reasoning_effort` that `enable_thinking` translates to — use [`extra_body`](#raw-provider-pass-through) to drive them.
+- **LM Studio**: Models supporting reasoning (gpt-oss, etc.)
+
+**When to use:**
+- Enable for QA, complex reasoning, and mathematical problems
+- Disable for speed-critical applications, title generation, and simple tasks
+
+!!! note "Anthropic thinking and max_tokens"
+    Anthropic requires `max_tokens` to exceed the thinking budget, and `enable_thinking: true` requests Pydantic AI's default budget of 10000 tokens. Set `max_tokens` above 10000 on Claude models that use budget-based thinking, or leave it unset on Sonnet 4.6+ and Opus 4.6+, which use adaptive thinking instead of a budget.
+
+!!! note "vLLM-served models without a reasoning profile"
+    On `provider: openai` with a custom `base_url`, `enable_thinking` only takes effect for models whose pydantic-ai profile advertises reasoning support (o-series, gpt-5, gpt-oss). For other vLLM-served models (Qwen3, Gemma family, …) the field is a silent no-op. Reach the chat template's thinking switch directly via [`extra_body`](#raw-provider-pass-through).
+
+### Raw Provider Pass-through
+
+The `extra_body` setting takes a dict that haiku.rag forwards verbatim to the underlying model SDK as `ModelSettings.extra_body`. Use it to reach provider-specific keys that haiku.rag does not model with a dedicated field.
+
+**Example: disable Qwen3 thinking on vLLM:**
+
+```yaml
+qa:
+  model:
+    provider: openai
+    name: qwen3.6-35b
+    base_url: http://localhost:11430/v1
+    extra_body:
+      chat_template_kwargs:
+        enable_thinking: false
+```
+
+vLLM serves Qwen3 chat templates that read their thinking switch from `chat_template_kwargs.enable_thinking`. The high-level `enable_thinking` setting on the openai provider maps to vLLM's `reasoning_effort` parameter, which Qwen3 templates ignore, so the field is a no-op for this combination. `extra_body` reaches the chat template directly and disables thinking. With it off, Qwen3 returns the answer in `content` immediately instead of emitting a hidden reasoning trace first.
+
+**Example: enable Gemma-family thinking on vLLM:**
+
+```yaml
+qa:
+  model:
+    provider: openai
+    name: nvidia/Gemma-4-26B-A4B-NVFP4
+    base_url: http://localhost:11432/v1
+    extra_body:
+      chat_template_kwargs:
+        enable_thinking: true
+```
+
+Same mechanism, opposite direction. Without `extra_body` the Gemma-4 chat template defaults to non-thinking and dumps a verbose answer straight into `content`. With it on, vLLM (started with `--reasoning-parser`) populates the parsed `reasoning` field and leaves `content` as the concise final answer.
+
+**Provider support:** honored by openai, ollama, anthropic, and groq via pydantic-ai's `ModelSettings.extra_body`. Silently ignored by google and bedrock.
+
+## Embedding Providers
+
+Embedding models require three settings: `provider`, `name`, and `vector_dim`. Optionally, use `base_url` for OpenAI-compatible servers and [`api_key`](#per-endpoint-api-keys) for the key that endpoint expects.
+
+### Batch Size
+
+`embeddings.batch_size` (default `512`) sets how many text chunks are sent per `/v1/embeddings` call during ingest. Lower it if your provider caps total tokens per request. Picture embeddings are always sent one image per call and are unaffected.
+
+### Ollama (Default)
+
+```yaml
+embeddings:
+  model:
+    provider: ollama
+    name: mxbai-embed-large
+    vector_dim: 1024
+```
+
+The Ollama base URL can be configured in your config file or via environment variable:
+
+```yaml
+providers:
+  ollama:
+    base_url: http://localhost:11434
+```
+
+Or via environment variable:
+
+```bash
+export OLLAMA_BASE_URL=http://localhost:11434
+```
+
+If not configured, it defaults to `http://localhost:11434`.
+
+### VoyageAI
+
+If you installed `haiku.rag` (full package), VoyageAI is already included. If you installed `haiku.rag-slim`, install with VoyageAI extras:
+
+```bash
+uv pip install haiku.rag-slim[voyageai]
+```
+
+```yaml
+embeddings:
+  model:
+    provider: voyageai
+    name: voyage-3.5
+    vector_dim: 1024
+```
+
+Set your API key via environment variable:
+
+```bash
+export VOYAGE_API_KEY=your-api-key
+```
+
+### OpenAI
+
+OpenAI embeddings are included in the default installation:
+
+```yaml
+embeddings:
+  model:
+    provider: openai
+    name: text-embedding-3-small  # or text-embedding-3-large
+    vector_dim: 1536
+```
+
+Set your API key via environment variable:
+
+```bash
+export OPENAI_API_KEY=your-api-key
+```
+
+### Cohere
+
+Cohere embeddings are available via pydantic-ai:
+
+```yaml
+embeddings:
+  model:
+    provider: cohere
+    name: embed-v4.0
+    vector_dim: 1024
+```
+
+Set your API key via environment variable:
+
+```bash
+export CO_API_KEY=your-api-key
+```
+
+### SentenceTransformers
+
+For local embeddings using HuggingFace models:
+
+```yaml
+embeddings:
+  model:
+    provider: sentence-transformers
+    name: all-MiniLM-L6-v2
+    vector_dim: 384
+```
+
+### OpenAI-Compatible Servers (vLLM, LM Studio, etc.)
+
+For local inference servers with OpenAI-compatible APIs, use the `openai` provider with a custom `base_url`:
+
+```yaml
+# vLLM example
+embeddings:
+  model:
+    provider: openai
+    name: mixedbread-ai/mxbai-embed-large-v1
+    vector_dim: 512
+    base_url: http://localhost:8000/v1
+
+# LM Studio example
+embeddings:
+  model:
+    provider: openai
+    name: text-embedding-qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://localhost:1234/v1
+```
+
+**Note:** The `base_url` must include the `/v1` path for OpenAI-compatible endpoints. This path is text-only. For a vision-language model served by vLLM, use `provider: vllm` with `multimodal: true` (below), not `provider: openai`.
+
+### Multimodal embedders
+
+For cross-modal retrieval (text and pictures share a single vector space), set `embeddings.model.multimodal: true`. Capability is decided by this flag, not the provider name: each provider passes images in its own wire format, so multimodal is supported only on `vllm`, `voyageai`, and `cohere`. Setting it on any other provider raises at startup.
+
+A model produces picture chunks at ingest only when its embedder is multimodal. Without the flag, an image-only document produces zero chunks and is not retrievable. Switching `multimodal` on or off does not change the stored embedding identity, so it raises no drift error; re-ingest or `rebuild` to add or drop picture chunks.
+
+**vLLM** — a vLLM server hosting a multimodal embedding model. Text inputs use the standard OpenAI `input` field; image inputs use vLLM's `messages`-with-`image_url` superset. Tested with `Qwen/Qwen3-VL-Embedding-8B` (4096-dim) and `jinaai/jina-embeddings-v4` (2048-dim). Run vLLM separately; haiku.rag adds no Python ML dependencies for this path.
+
+```yaml
+embeddings:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-VL-Embedding-8B
+    vector_dim: 4096
+    base_url: http://localhost:8000/v1
+    multimodal: true
+```
+
+**VoyageAI** — `voyage-multimodal-3` (1024-dim) via the `voyageai` extra. Reads `VOYAGE_API_KEY` from the environment.
+
+```yaml
+embeddings:
+  model:
+    provider: voyageai
+    name: voyage-multimodal-3
+    vector_dim: 1024
+    multimodal: true
+```
+
+**Cohere** — `embed-v4.0` (configurable `vector_dim`, e.g. 1536) via the `cohere` extra. Reads `CO_API_KEY` from the environment.
+
+```yaml
+embeddings:
+  model:
+    provider: cohere
+    name: embed-v4.0
+    vector_dim: 1536
+    multimodal: true
+```
+
+A text-only model served by vLLM uses `provider: vllm` without the flag (or `provider: openai` with a `base_url`).
+
+Picture chunks for retrieval are emitted at ingest under any multimodal embedder. See [Picture Handling](processing.md#picture-handling).
+
+## Question Answering Providers
+
+Configure which LLM provider to use for question answering. Any provider and model supported by [Pydantic AI](https://ai.pydantic.dev/models/) can be used.
+
+### Ollama (Default)
+
+```yaml
+qa:
+  model:
+    provider: ollama
+    name: qwen3.8
+```
+
+The Ollama base URL can be configured via the `OLLAMA_BASE_URL` environment variable, config file, or defaults to `http://localhost:11434`:
+
+```bash
+export OLLAMA_BASE_URL=http://localhost:11434
+```
+
+Or in your config file:
+
+```yaml
+providers:
+  ollama:
+    base_url: http://localhost:11434
+```
+
+### OpenAI
+
+OpenAI QA is included in the default installation:
+
+```yaml
+qa:
+  model:
+    provider: openai
+    name: gpt-4o-mini  # or gpt-4, gpt-3.5-turbo, etc.
+```
+
+Set your API key via environment variable:
+
+```bash
+export OPENAI_API_KEY=your-api-key
+```
+
+### Anthropic
+
+Anthropic QA is included in the default installation:
+
+```yaml
+qa:
+  model:
+    provider: anthropic
+    name: claude-3-5-haiku-20241022  # or claude-3-5-sonnet-20241022, etc.
+```
+
+Set your API key via environment variable:
+
+```bash
+export ANTHROPIC_API_KEY=your-api-key
+```
+
+### OpenAI-Compatible Servers (vLLM, LM Studio, etc.)
+
+For local inference servers with OpenAI-compatible APIs, use the `openai` provider with a custom `base_url`:
+
+```yaml
+# vLLM example
+qa:
+  model:
+    provider: openai
+    name: Qwen/Qwen3-4B
+    base_url: http://localhost:8002/v1
+
+# LM Studio example
+qa:
+  model:
+    provider: openai
+    name: gpt-oss-20b
+    base_url: http://localhost:1234/v1
+    enable_thinking: false
+```
+
+**Note:** The server must be running with a model that supports tool calling. The `base_url` must include the `/v1` path.
+
+### Other Providers
+
+Any provider supported by Pydantic AI can be used. Examples:
+
+```yaml
+# Google Gemini
+qa:
+  model:
+    provider: google
+    name: gemini-1.5-flash
+
+# Groq
+qa:
+  model:
+    provider: groq
+    name: llama-3.3-70b-versatile
+
+# Mistral
+qa:
+  model:
+    provider: mistral
+    name: mistral-small-latest
+```
+
+See the [Pydantic AI documentation](https://ai.pydantic.dev/models/) for the complete list of supported providers and models.
+
+## Reranking Providers
+
+Reranking improves search quality by re-ordering the initial search results using specialized models. When enabled, the system retrieves more candidates (10x the requested limit) and then reranks them to return the most relevant results.
+
+Reranking is **disabled by default** for faster searches: there is no `reranking.model`. Enable it by configuring one of the providers below, and disable it again by removing the section or setting `model: null`.
+
+### Cohere
+
+If you installed `haiku.rag` (full package), Cohere is already included. If you installed `haiku.rag-slim`, add the cohere extra:
+
+```bash
+uv pip install haiku.rag-slim[cohere]
+```
+
+Then configure:
+
+```yaml
+reranking:
+  model:
+    provider: cohere
+    name: rerank-v3.5
+```
+
+Set your API key via environment variable:
+
+```bash
+export CO_API_KEY=your-api-key
+```
+
+### Zero Entropy
+
+If you installed `haiku.rag` (full package), Zero Entropy is already included. If you installed `haiku.rag-slim`, add the zeroentropy extra:
+
+```bash
+uv pip install haiku.rag-slim[zeroentropy]
+```
+
+Then configure:
+
+```yaml
+reranking:
+  model:
+    provider: zeroentropy
+    name: zerank-1  # Currently the only available model
+```
+
+Set your API key via environment variable:
+
+```bash
+export ZEROENTROPY_API_KEY=your-api-key
+```
+
+### vLLM
+
+For high-performance local reranking using dedicated reranking models:
+
+```yaml
+reranking:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://localhost:8001/v1
+```
+
+**Note:** vLLM reranking posts to the `/v1/rerank` endpoint. As with the embedder, `base_url` may be written with or without the `/v1` path. You need to run a vLLM server separately with a reranking model loaded.
+
+#### Multimodal reranking
+
+When serving a vision reranker (for example `nvidia/llama-nemotron-rerank-vl-1b-v2`), set `multimodal: true` to score picture chunks by their image bytes in addition to their description text:
+
+```yaml
+reranking:
+  multimodal: true
+  model:
+    provider: vllm
+    name: nvidia/llama-nemotron-rerank-vl-1b-v2
+    base_url: http://localhost:8001/v1
+```
+
+Picture chunks are sent as image documents (base64 data URIs) alongside plain text documents in the same rerank request. The flag is supported on the vllm provider only, and the served model must accept multimodal inputs.
+
+### Jina AI
+
+Jina provides high-quality reranking with two deployment options: API mode and local inference.
+
+#### API Mode
+
+Use the Jina Reranker API for cloud-based reranking:
+
+```yaml
+reranking:
+  model:
+    provider: jina
+    name: jina-reranker-v3
+```
+
+Set your API key via environment variable:
+
+```bash
+export JINA_API_KEY=your-api-key
+```
+
+#### Local Mode
+
+For local inference, install the jina extra:
+
+```bash
+uv pip install haiku.rag-slim[jina]
+```
+
+Then configure:
+
+```yaml
+reranking:
+  model:
+    provider: jina-local
+    name: jinaai/jina-reranker-v3
+```
+
+**Note:** The Jina Reranker v3 local model is licensed under CC BY-NC 4.0, which restricts commercial use. For commercial applications, use the API mode instead.
+
+### Cross-Encoder (sentence-transformers)
+
+Run any HuggingFace cross-encoder reranker in-process via `sentence-transformers`. No separate server required. Useful when you want a specific model (BGE, Qwen3-Reranker, MS-MARCO MiniLM, etc.) without running vLLM.
+
+Install the extra:
+
+```bash
+uv pip install haiku.rag-slim[cross-encoder]
+```
+
+Then configure with any HuggingFace model id:
+
+```yaml
+reranking:
+  model:
+    provider: cross-encoder
+    name: Qwen/Qwen3-Reranker-0.6B
+```
+
+Other tested models: `BAAI/bge-reranker-v2-m3`, `cross-encoder/ms-marco-MiniLM-L-6-v2`. Any model exposed as a `sentence_transformers.CrossEncoder` works.

docs/configuration/qa.md

@@ -0,0 +1,63 @@
+# Search and Question Answering
+
+## Search Settings
+
+Configure search behavior and context expansion:
+
+```yaml
+search:
+  limit: 5                     # Default number of results to return
+  max_context_chars: 5000     # Maximum characters in expanded context
+```
+
+- **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and QA. Default: 5
+- **max_context_chars**: Hard limit on total characters in expanded content. Default: 5000.
+
+Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers). This naturally crosses into adjacent sections until the budget is filled. Picture and table matches are exempt: they return their enclosing section as-is and never cross section boundaries. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded.
+
+!!! note "Reranking behavior"
+    When a reranker is configured, search automatically retrieves 10x the requested limit, then reranks to return the final count. This improves result quality without requiring you to adjust `limit`.
+
+## Question Answering Configuration
+
+Configure the RAG capability (used by `client.ask` and `haiku-rag ask`):
+
+```yaml
+qa:
+  model:
+    provider: ollama
+    name: qwen3.8
+    enable_thinking: true
+    temperature: 0.3          # Default: 0.3
+    vision: true              # Set false for text-only models
+  max_searches: 5       # Maximum search units per question
+```
+
+- **model**: LLM configuration (see [Providers](providers.md#model-settings))
+- **model.vision**: Set to `true` for vision-capable models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, …). The capability's `search` tool only attaches picture bytes (`BinaryContent`) to its `ToolReturn` when this is `true`, otherwise picture bytes are withheld. See [Pictures × embedder × QA model](processing.md#pictures-embedder-qa-model-how-the-pieces-compose) for the full matrix.
+- **max_searches**: Maximum number of search units a capability can spend per question (default: 5). Up to three searches emitted in the same model response share one unit, so a model that rephrases its query in one response spends one unit. A search in a later response starts a new unit, as does each further group of three within one response. Shared by the RAG and analysis capabilities. Searches in one response also deduplicate their returns: evidence a sibling search already showed collapses to a reference line, and each picture attaches once per response.
+
+!!! note "Thinking on vLLM"
+    `enable_thinking` only applies to models with a pydantic-ai reasoning profile (o-series, gpt-5, gpt-oss). For other vLLM-served models such as Qwen3 or the Gemma family, the field is a silent no-op — set the chat template switch via [`extra_body`](providers.md#raw-provider-pass-through) instead.
+
+## Analysis Configuration
+
+Configure the analysis capability:
+
+```yaml
+analysis:
+  model:
+    provider: anthropic
+    name: claude-sonnet-4-20250514
+    temperature: 0.0        # Default: 0.0 (deterministic for code generation)
+  code_timeout: 60.0      # Per call: compute stops, no read or search starts past it
+  max_output_chars: 50000 # Truncate output after this many chars
+  max_executions: 15      # Max execute_code calls per question
+```
+
+- **model**: LLM configuration (see [Providers](providers.md#model-settings)). When unset, falls back to `qa.model`.
+- **code_timeout**: Seconds a single `execute_code` call has (default: 60). Past it the sandbox starts no further host call, a document read or an in-code `search()` / `list_documents()`; one already running finishes. Code that computes without host calls is killed by the worker watchdog at the same limit. `code_timeout * max_executions` is the cumulative ceiling across all calls in one question.
+- **max_output_chars**: Truncate code output after this many characters (default: 50000)
+- **max_executions**: Maximum `execute_code` calls per question before the capability is told to answer from what it has (default: 15)
+
+See [Analysis capability](../capabilities/analysis.md) for usage details.

docs/configuration/storage.md

@@ -0,0 +1,360 @@
+# Database and Storage
+
+## Operational constraints
+
+Four things to know before deploying.
+
+**Run one writer per database.** This is a haiku.rag constraint, not a LanceDB
+one. A write that spans several tables is serialized by an in-process lock and
+rolled back by restoring each table to the version it had when the write started.
+Both are process-local: a second writing process can commit between that snapshot
+and the mutation, and a rollback would then revert its work along with ours. Run
+a single writer, either the [`haiku-ingester`](../ingester.md) service or your own
+application. Read-only consumers are unrestricted.
+
+**Readers lag by an interval.** A connection always sees its own writes. It sees
+another process's writes after `lancedb.read_consistency_interval_seconds`
+(default 30).
+
+**Migrate after an upgrade that changes the schema.** `haiku-rag migrate` applies
+pending migrations in place, and `haiku-rag info` lists what is pending. A
+release that needs it says so in the [changelog](../changelog.md).
+
+**The embedding dimension is fixed per database.** Every chunk vector has the
+dimension the database was created with. Changing `embeddings.model.vector_dim`
+raises `ConfigMismatchError` on open, because stored vectors cannot be compared
+against new ones. Changing the provider or model name while keeping the dimension
+warns on a read-only open and raises on a writable one. `haiku-rag rebuild
+--set-embedder` adopts the new identity without re-embedding, and `haiku-rag
+rebuild --embed-only` re-embeds against the new model.
+
+## Local Storage
+
+By default, `haiku.rag` uses a local LanceDB database:
+
+```yaml
+storage:
+  data_dir: /path/to/data  # Empty = use default platform location
+  auto_vacuum: true  # Enable automatic vacuuming after operations
+  vacuum_retention_seconds: 86400  # Cleanup threshold in seconds
+```
+
+- **data_dir**: Directory for local database storage. When empty, uses platform-specific default locations
+- **auto_vacuum**: When enabled (default), automatically runs vacuum after document create/update/delete operations and database rebuilds. Background vacuums are throttled to at most one every 5 minutes, so sustained ingestion does not trigger continuous compaction, and a final vacuum runs when the client closes. Set to `false` to disable automatic vacuuming and rely on manual `haiku-rag vacuum` commands only. Disabling can help avoid potential crashes in high-concurrency scenarios
+- **vacuum_retention_seconds**: When vacuum runs, old table versions older than this threshold are removed. Default: 86400 seconds (1 day). Set to 0 for aggressive cleanup (removes all old versions immediately)
+
+!!! warning "Vacuum Retention Threshold"
+    The `vacuum_retention_seconds` value should be larger than the typical time it takes to process and write a document. If a concurrent operation is in progress while vacuum runs, setting this value too low can cause race conditions where vacuum removes table versions that an in-flight operation still needs. The default of 86400 seconds (1 day) is conservative and safe for most use cases.
+
+### Vacuum Memory Requirements
+
+Vacuum compacts small data files into larger ones. LanceDB targets roughly one million rows per fragment, which a `documents` table holding multi-megabyte docling blobs never reaches, so each vacuum that follows new documents re-merges the whole existing fragment rather than only the new ones. Peak memory therefore scales with the total size of the `documents` table, not with how much was added.
+
+Measured peak resident memory is about 5x the size of the `documents` table's data files. An 8.8 GB table peaked at 48.7 GB. Plan for **6x the size of `documents/` on disk** as available RAM, or the vacuum will be killed by the OOM killer partway through.
+
+Check the current size with:
+
+```bash
+du -sh /path/to/database.lancedb/documents.lance
+```
+
+If that number times six exceeds available RAM, use one of:
+
+- Reduce `images_scale` (see [Image Settings](processing.md#image-settings)). Rendered page rasters dominate the size of `documents`, and their byte cost falls with the square of the scale factor.
+- Set `generate_page_images: false` if visual grounding through `visualize_chunk()` is not needed. This removes page rasters entirely.
+- Set `auto_vacuum: false` and run `haiku-rag vacuum` manually when the machine is otherwise idle, so the peak does not land alongside ingestion.
+
+Vacuum also folds new rows into the full-text index. Search stays correct without it but scans the uncovered rows on every query. `haiku-rag doctor` reports the coverage.
+
+This is an upstream limitation rather than a `haiku.rag` setting. Compaction bounds itself by row count instead of bytes, and LanceDB's async API exposes no batch size or fragment target to override it. Tracked at [lancedb/lancedb#2325](https://github.com/lancedb/lancedb/issues/2325). The requirement above will drop once compaction batches by bytes.
+
+### Placing the Database
+
+`lancedb.databases` maps a name to a location, a local path or a URI, and is the one way to place databases. With nothing configured, the database is the entry `haiku.rag` at `<storage.data_dir>/haiku.rag.lancedb`. To put one database somewhere else, name it:
+
+```yaml
+lancedb:
+  databases:
+    notes: /data/notes.lancedb
+```
+
+The name is what `source` carries in search results, citations and documents, and what `--db-name` and `sources` select. The default database answers to `haiku.rag`.
+
+An explicit `--db PATH` on the command line opens that database instead, named by the path's stem, whatever is configured. From Python, `db_path` places the database only where the configuration places none: beside `lancedb.databases` it raises `AmbiguousDatabaseError`.
+
+A value with no scheme is a local path wherever it is configured, so `haiku-rag init` creates it and every command that opens an existing database requires it to exist. A mistyped path fails rather than becoming a new empty database.
+
+## Database Creation
+
+Databases must be explicitly created before use:
+
+**CLI:**
+```bash
+# Create in default location (see Configuration File Locations below)
+haiku-rag init
+
+# Create at custom path
+haiku-rag init --db /path/to/database.lancedb
+```
+
+**Python:**
+```python
+# Create at custom path
+async with HaikuRAG("/path/to/database.lancedb", create=True) as client:
+    ...
+
+# Create in default location
+async with HaikuRAG(create=True) as client:
+    ...
+```
+
+The [default location](index.md#configuration-file-locations) is platform-specific (e.g., `~/Library/Application Support/haiku.rag/` on macOS).
+
+Opening a nonexistent local database given as a path raises `FileNotFoundError`, naming the path. This prevents accidental database creation from typos or misconfigured paths. A configured or default database raises `SourceUnavailableError` instead, naming the database and not its location.
+
+## Remote Storage
+
+For remote storage, give the database a URI as its location. Credentials and storage options are connection settings, shared by every database in the configuration:
+
+```yaml
+# LanceDB Cloud
+lancedb:
+  databases:
+    papers: db://your-database-name
+  api_key: your-api-key
+  region: us-west-2
+
+# Amazon S3
+lancedb:
+  databases:
+    papers: s3://my-bucket/my-table
+  storage_options:
+    region: us-east-1
+
+# Amazon S3 with explicit credentials
+lancedb:
+  databases:
+    papers: s3://my-bucket/my-table
+  storage_options:
+    aws_access_key_id: YOUR_ACCESS_KEY
+    aws_secret_access_key: YOUR_SECRET_KEY
+    region: us-east-1
+
+# S3-compatible (SeaweedFS, Tigris, etc.)
+lancedb:
+  databases:
+    papers: s3://my-bucket/my-table
+  storage_options:
+    endpoint: http://localhost:8333
+    aws_access_key_id: YOUR_ACCESS_KEY
+    aws_secret_access_key: YOUR_SECRET_KEY
+    region: us-east-1
+    allow_http: "true"
+
+# Azure Blob Storage
+lancedb:
+  databases:
+    papers: az://my-container/my-table
+
+# Google Cloud Storage
+lancedb:
+  databases:
+    papers: gs://my-bucket/my-table
+
+# HDFS
+lancedb:
+  databases:
+    papers: hdfs://namenode:port/path/to/table
+```
+
+- **LanceDB Cloud** (`db://`): Requires `api_key` and `region`. Table optimization and indexing are managed server-side.
+- **Object storage** (`s3://`, `gs://`, `az://`, `hdfs://`): Uses `storage_options` for credentials and endpoint configuration. Authentication can also be provided via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, etc.) or cloud provider SDK defaults (AWS CLI, Azure CLI, gcloud).
+- **S3-compatible stores** (MinIO, Tigris, etc.): Set `endpoint` in `storage_options`. When using `http://` endpoints, also set `allow_http: "true"`.
+- **Local path** (no scheme): a location without a scheme is a local path. See [Placing the Database](#placing-the-database).
+
+The `storage_options` keys are case-insensitive and passed directly to the underlying object store library. Available keys depend on the backend. See the [LanceDB storage docs](https://lancedb.com/docs/storage/) for details.
+
+**Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization and vector indexing are still performed normally.
+
+### Caching and Read Consistency
+
+```yaml
+lancedb:
+  read_consistency_interval_seconds: 30   # null to never re-check
+  index_cache_size_bytes: 536870912       # null for the LanceDB default
+  metadata_cache_size_bytes: 268435456
+```
+
+- **read_consistency_interval_seconds**: how often a connection checks for writes from another process. `null` never checks, so a long-lived reader never sees the ingester's writes. `0` checks on every read.
+- **index_cache_size_bytes** / **metadata_cache_size_bytes**: sizes for the caches held by the LanceDB session, which is shared across every connection in the process. The first vector query loads the index into it, so on object storage the cache is what stops the next connection refetching it. Size it for the total set of indexes a process keeps warm, against the memory available to it.
+
+### Deployment Pattern: One Writer, Many Readers
+
+The [one-writer constraint](#operational-constraints) shapes the deployment: one
+writing process per database URI, any number of read-only consumers.
+
+The recommended layout for production is "different buckets, same account, separate IAM roles per process":
+
+- **Ingestion process** — IAM role with `s3:Get/List` on the documents bucket and `s3:Get/Put/Delete` on the LanceDB bucket. Runs `haiku-ingester serve` (with `ingester.sources[type=s3]` pointing at the documents bucket). Exactly one such process per LanceDB URI.
+- **Consumer processes** (1..N) — IAM role with `s3:Get/List` on the LanceDB bucket only. Run `haiku-rag mcp`, the chat TUI, etc. They never see the documents bucket.
+
+Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
+
+`haiku-ingester` writes the database the configuration places, so a one-entry `lancedb.databases` needs no further option. `--db PATH` overrides it. When `lancedb.databases` contains more than one database the ingester has no way to name which it writes, and refuses to start with `AmbiguousDatabaseError`: give each database its own ingester process, each with a configuration naming a single database, or select one with `--db PATH`.
+
+## Multiple Databases
+
+Use `lancedb.databases` to name local or remote databases that should be searched together:
+
+```yaml
+lancedb:
+  databases:
+    papers: s3://my-bucket/papers.lancedb
+    wiki: s3://my-bucket/wiki.lancedb
+    notes: /data/notes.lancedb
+```
+
+A location can be a URI or local path.
+
+Results, documents, and citations use the configured name as `source`. An unavailable configured database raises `SourceUnavailableError`, which names the database and not its location, so a location never travels in an error a consumer might render or log. A migration, configuration or read-only failure keeps its own type, with the database named in the message. Commands that report on a database, such as `info`, still show where it is.
+
+Searches spanning multiple databases identify each result with a model-facing `Collection:` line. Searches over one database omit it. Structured `source` fields on results, documents, citations, and analysis dictionaries are unchanged.
+
+Embedding compatibility is checked against two different things.
+
+On open, each database is compared with the current configuration. A dimension mismatch raises `ConfigMismatchError`. A provider or model-name mismatch at the same dimension warns in read-only mode and raises in writable mode.
+
+Across a selection, the databases are compared with each other. Vector and hybrid search embed the query once, so every database answering it must record the same provider, model, and dimension. A disagreement raises `ConfigMismatchError` in read-only mode as well. Only the databases searched together have to agree, and full-text search embeds nothing, so it is unaffected.
+
+### Search and Provenance
+
+`search`, `ask`, and `analyze` use the full set by default. Pass `sources` to select a subset:
+
+```python
+results = await client.search("query")                     # every database
+results = await client.search("query", sources=["papers"])   # one of them
+```
+
+Candidates are combined into one ranked list with the configured reranker, or by cosine similarity to the query when reranking is disabled, with within-database rank breaking ties (full-text-only searches order by retrieval score). `SearchResult.source`, `Citation.source`, and `Document.source` carry the database name, for a set and for one database alike.
+
+The CLI labels results and citations only when the operation spans multiple databases. A command already narrowed with `--db-name` does not repeat the name on every result.
+
+#### Duplicate IDs
+
+IDs are unique within a database, not across databases. Copies of a database therefore retain the same IDs.
+
+Citation ambiguity is evaluated against evidence available to the run. A cited chunk ID is rejected with `AmbiguousCitationError` if search returned it from multiple databases, or it was previously cited from another database. If only one retrieved result has the ID, that result is cited. For an ID absent from search results, the fallback checks every selected database and rejects multiple holders. A shared ID that nothing cites is ignored.
+
+`get_document_by_id`, `get_chunk_by_id` and `get_picture_bytes` take an optional `source`, and ask that database alone. A name the client does not cover raises `UnknownDatabaseError`. Without one, the document and chunk lookups ask every covered database and answer from the first that holds the ID; `get_picture_bytes` requires one whenever the client covers a set.
+
+The analysis sandbox rejects shared document IDs because its mount path is `/documents/{id}/`.
+
+The chat document filter selects by document and database: the search is narrowed to the databases the selection names, and the ID filter applies within them. An ID that copies share still matches in every selected database that holds it.
+
+#### Ranking
+
+Without a reranker, the fused list is ordered by cosine similarity between the query vector and each candidate. The databases in a selection share an embedder, so similarity in that one space is comparable across databases, where retrieval scores are each database's own arithmetic. Ties resolve by the candidate's rank within its own database, and configured order decides only when both tie. Similarities rarely tie exactly, so declaration order decides almost nothing: on MTRAG retrieval benchmarks, reversing it left recall unchanged in every cell. Full-text-only searches have no query vector and order by retrieval score instead.
+
+Results are not guaranteed to spread across databases: a database with nothing relevant to a query contributes nothing, and a strong database can fill every slot. On MTRAG retrieval benchmarks over two to eight collections, cosine fusion holds recall roughly flat as collections are added, where position-based fusion lost up to half its recall at eight.
+
+A configured reranker scores the combined candidate set directly, ignoring which database each candidate came from, and remains the strongest option: roughly 6 to 8 recall points above cosine fusion on the same benchmarks. Its cost grows with the number of databases because each contributes candidates.
+
+Image queries are vector-only and skip the reranker: the reranker interface takes a text query, and multimodal reranking applies to pictures on the candidate side, not to image queries. Their fused list is ordered by cosine similarity like any other vector search.
+
+If a selected database is unavailable, the operation fails with `SourceUnavailableError`, which names that database.
+
+### Python Operations
+
+Creating, writing, rebuilding, and vacuuming require one database. Calling these operations on a client that covers multiple raises `AmbiguousDatabaseError`. Select one at creation time or obtain a single-database client:
+
+```python
+async with HaikuRAG(config=config, create=True, sources=["papers"]) as papers:
+    ...
+
+async with HaikuRAG(config=config) as client:
+    papers = (await client.clients_for(["papers"]))[0]
+```
+
+Conversion, chunking, and title generation do not access a database and remain available on a multi-database client.
+
+### CLI Commands
+
+Commands use database sets as follows:
+
+- **Set-capable**: `search`, `ask`, `analyze`, `chat`, and `mcp` use the full configured set, or the single database selected by `--db-name`.
+- **Config-only**: `settings`, `init-config`, and `download-models` do not open a database.
+- **Single-database**: everything else — document writes, `rebuild`, `vacuum`, `migrate`, `init`, `info`, `history`, `tag`, `doctor`, `list`, `inspect`, and `visualize` — works on one database, selected with the global `--db-name` option.
+
+```bash
+haiku-rag search "query"         # every configured database
+haiku-rag --db-name papers list  # one of them
+haiku-rag --db-name papers migrate
+```
+
+`--db-name` selects an entry from `lancedb.databases`, including remote entries, and `haiku.rag` when nothing is configured. `--db` opens a local path, named by its stem, whatever is configured. A single-database command requires one of these options when multiple databases are configured. A configured set of one is selected automatically.
+
+Each database is created, migrated and vacuumed on its own:
+
+```bash
+haiku-rag --db-name papers init
+haiku-rag --db-name wiki init
+```
+
+## Vector Indexing
+
+Configure vector search settings:
+
+```yaml
+search:
+  vector_index_metric: cosine  # cosine or l2
+  vector_refine_factor: 30     # Re-ranking factor for accuracy
+  vector_nprobes: 20           # IVF partitions searched per query
+```
+
+For search behavior settings (`limit`, `max_context_chars`), see [Search and Question Answering](qa.md#search-settings).
+
+- **vector_index_metric**: Distance metric for vector similarity:
+  - `cosine`: Cosine similarity (default, best for most embeddings)
+  - `l2`: Euclidean distance
+- **vector_refine_factor**: Improves accuracy when using a vector index by retrieving `refine_factor * limit` candidates (using approximate search) and re-ranking them with exact distances. Higher values increase accuracy but slow down queries. Default: 30
+  - **Only applies with a vector index** - has no effect on brute-force search, which already returns exact results
+- **vector_nprobes**: How many IVF partitions each query searches. Higher values increase recall and latency. A larger corpus holds more partitions, so the same value covers a smaller fraction of it. Default: 20
+  - **Only applies with a vector index** - ignored by brute-force search
+
+!!! note
+    Vector indexes are only necessary for large datasets with over 100,000 chunks. For smaller datasets, LanceDB's brute-force kNN search provides exact results with good performance. Only create an index if you notice search performance degradation on large datasets.
+
+Retrieval MAP with and without an index, measured on copies of the benchmark databases with no reranker:
+
+| Dataset | Chunks | Dim | Exact | Indexed | Delta | Build | Peak RSS |
+|---------|-------:|----:|------:|--------:|------:|------:|---------:|
+| `hotpotqa` | 70,527 | 2560 | 0.6978 | 0.6979 | +0.0001 | 29.3 s | 3.19 GB |
+| `orb_multimodal_nemotron` | 121,168 | 2048 | 0.9799 | 0.9800 | +0.0001 | 25.8 s | 3.38 GB |
+| `frames` | 425,940 | 2560 | 0.5431 | 0.5387 | -0.0044 | 34.1 s | 4.02 GB |
+
+An index costs no accuracy at 70k and 121k chunks and 0.0044 MAP at 426k. A larger corpus holds more IVF partitions, so the default number of probes covers a smaller fraction of the space, and `vector_refine_factor` can only re-score what those probes returned. Raise `vector_nprobes` to trade latency for recall on a large corpus. Build cost is near-flat in row count because training samples the data rather than scanning it, and vector dimension drives it more than corpus size.
+
+**Index creation:**
+
+Vector indexes are **not created automatically** during document ingestion to avoid slowing down the process. After you've added documents (at least 256 chunks required), create the index manually:
+
+```bash
+haiku-rag create-index
+```
+
+This command:
+- Checks if you have enough data (minimum 256 chunks)
+- Creates an IVF_PQ index for fast approximate nearest neighbor (ANN) search
+- Uses LanceDB's automatic parameter calculation based on your dataset size and vector dimensions
+
+**Re-indexing:**
+
+New chunks reach the index without a rebuild. `optimize()`, which runs after writes while `auto_vacuum` is on, adds them as a delta part. Between a write and the next optimize, LanceDB serves ANN over the indexed rows and a brute-force scan over the remainder, then combines the results.
+
+A rebuild retrains the centroids, which are fitted once at build time and never recomputed. As a corpus grows past the distribution it was trained on the partitioning fits it less well, and delta parts accumulate. Rebuild after substantial growth:
+
+```bash
+haiku-rag create-index
+```
+
+For datasets with fewer than 256 chunks, searches use brute-force kNN scans (exact nearest neighbors, 100% recall) which work well for small datasets but don't scale beyond a few hundred thousand vectors.

docs/custom-pipelines.md

@@ -0,0 +1,255 @@
+# Custom Processing Pipelines
+
+haiku.rag provides processing primitives that let you build custom document pipelines. Use these when you need control over conversion, chunking, or embedding (for example, to preprocess content, use external services, or implement custom chunking logic).
+
+## When to Use Custom Pipelines
+
+Use the primitives when you need to:
+
+- Preprocess or clean content before chunking
+- Filter or modify chunks before embedding
+- Use external embedding services
+- Implement custom chunking strategies
+- Debug or inspect intermediate processing steps
+
+For standard use cases, prefer the convenience methods:
+
+- `create_document()` - Create from text content
+- `create_document_from_source()` - Create from file or URL
+- `import_document()` - Store pre-processed documents with custom chunks
+
+## Processing Primitives
+
+The client exposes four primitives that can be composed into custom workflows:
+
+| Primitive | Input | Output | Purpose |
+|-----------|-------|--------|---------|
+| `convert()` | file, URL, or text | `DoclingDocument` | Convert source to structured document |
+| `chunk()` | `DoclingDocument` | `list[Chunk]` | Split document into chunks |
+| `embed_chunks()` | `list[Chunk]`, embedder | `list[Chunk]` | Generate embeddings for chunks (includes contextualization) |
+| `contextualize()` | `list[Chunk]` | `list[str]` | Get embedding-ready text (for custom embedders only) |
+
+## Basic Pipeline
+
+The standard pipeline mirrors what `create_document()` does internally:
+
+```python
+from haiku.rag.client import HaikuRAG
+from haiku.rag.embeddings import embed_chunks
+
+async with HaikuRAG("database.lancedb", create=True) as client:
+    # 1. Convert source to DoclingDocument
+    docling_doc = await client.convert("path/to/document.pdf")
+
+    # 2. Chunk the document
+    chunks = await client.chunk(docling_doc)
+
+    # 3. Generate embeddings
+    embedded_chunks = await embed_chunks(chunks, client.embedder)
+
+    # 4. Store the document with chunks
+    doc = await client.import_document(
+        docling_document=docling_doc,
+        chunks=embedded_chunks,
+        uri="file:///path/to/document.pdf",
+        title="My Document",
+    )
+```
+
+## Convert
+
+`convert()` accepts files, URLs, or plain text and returns a `DoclingDocument`:
+
+```python
+# From local file
+docling_doc = await client.convert("report.pdf")
+docling_doc = await client.convert(Path("/absolute/path/to/file.docx"))
+
+# From URL (downloads and converts)
+docling_doc = await client.convert("https://example.com/paper.pdf")
+
+# From plain text (parsed as markdown by default)
+docling_doc = await client.convert("# Title\n\nYour text content here")
+
+# From HTML text (use format parameter to preserve structure)
+html_content = "<h1>Title</h1><p>Paragraph</p><ul><li>Item</li></ul>"
+docling_doc = await client.convert(html_content, format="html")
+
+# From file:// URI
+docling_doc = await client.convert("file:///path/to/document.md")
+```
+
+The `format` parameter controls how text content is parsed:
+
+- `"md"` (default) - Parse as Markdown
+- `"html"` - Parse as HTML, preserving semantic structure (headings, lists, tables)
+
+!!! note
+    The `format` parameter only applies to text content. Files and URLs determine their format from the file extension or content-type header.
+
+Supported formats depend on your converter configuration (docling-local or docling-serve). Common formats include PDF, DOCX, HTML, Markdown, and images.
+
+## Chunk
+
+`chunk()` splits a `DoclingDocument` into `Chunk` objects with metadata:
+
+```python
+chunks = await client.chunk(docling_doc)
+
+for chunk in chunks:
+    print(f"Order: {chunk.order}")
+    print(f"Content: {chunk.content[:100]}...")
+
+    # Access structured metadata
+    meta = chunk.get_chunk_metadata()
+    print(f"Headings: {meta.headings}")
+    print(f"Page numbers: {meta.page_numbers}")
+    print(f"Labels: {meta.labels}")
+    # Access raw metadata (including headings, page_numbers and labels)
+    print(f"Raw metadata: {chunk.metadata}")
+```
+
+Chunks are returned with:
+
+- `content` - The chunk text
+- `order` - Position in document (0-indexed)
+- `metadata` - Dict with `doc_item_refs`, `headings`, `labels`, `page_numbers`
+- `embedding` - `None` (not yet embedded)
+- `document_id` - `None` (not yet stored)
+
+A custom `DocumentChunker` can provide other keys and values in `metadata`. They will be stored with the chunk and are accessible when it is returned in a search result or citation, within `SearchResult.chunk_meta` / `Citation.chunk_meta`.
+
+## Embed
+
+`embed_chunks()` generates embeddings for chunks using the client's embedder. It automatically contextualizes chunks (prepends section headings) before embedding for better semantic search, without modifying the stored content:
+
+```python
+from haiku.rag.embeddings import embed_chunks
+
+# Generate embeddings (returns new Chunk objects)
+embedded_chunks = await embed_chunks(chunks, client.embedder)
+
+# Original chunks unchanged
+assert chunks[0].embedding is None
+
+# New chunks have embeddings
+assert embedded_chunks[0].embedding is not None
+```
+
+`embed_chunks()` returns **new** `Chunk` objects with embeddings set. The original chunks are not modified.
+
+## Contextualize (for custom embedders)
+
+`contextualize()` is a lower-level utility that prepares chunk content for embedding by prepending section headings. You only need this when implementing custom embedding logic. `embed_chunks()` already calls it internally.
+
+```python
+from haiku.rag.embeddings import contextualize
+
+# Get embedding-ready text (only needed for custom embedders)
+texts = contextualize(chunks)
+# texts[0] might be: "Chapter 1\nIntroduction\nThe actual chunk content..."
+```
+
+See the [Custom Embeddings](#custom-embeddings) example below for when to use `contextualize()`.
+
+## Custom Processing Examples
+
+### Preprocessing Content
+
+Transform content before chunking:
+
+```python
+def clean_markdown(text: str) -> str:
+    """Remove HTML comments and normalize whitespace."""
+    import re
+    text = re.sub(r'<!--.*?-->', '', text, flags=re.DOTALL)
+    text = re.sub(r'\n{3,}', '\n\n', text)
+    return text.strip()
+
+async with HaikuRAG("database.lancedb", create=True) as client:
+    # Convert to get raw content
+    docling_doc = await client.convert("document.md")
+
+    # Extract and preprocess markdown
+    markdown = docling_doc.export_to_markdown()
+    cleaned = clean_markdown(markdown)
+
+    # Re-convert the cleaned content
+    processed_doc = await client.convert(cleaned)
+
+    # Continue with standard pipeline
+    chunks = await client.chunk(processed_doc)
+    embedded_chunks = await embed_chunks(chunks, client.embedder)
+
+    await client.import_document(
+        chunks=embedded_chunks,
+        content=cleaned,
+    )
+```
+
+### Filtering Chunks
+
+Remove unwanted chunks before embedding:
+
+```python
+async with HaikuRAG("database.lancedb", create=True) as client:
+    docling_doc = await client.convert("document.pdf")
+    chunks = await client.chunk(docling_doc)
+
+    # Filter out short chunks or boilerplate
+    filtered = [
+        c for c in chunks
+        if len(c.content) > 50
+        and "copyright" not in c.content.lower()
+    ]
+
+    # Re-number the order field after filtering
+    for i, chunk in enumerate(filtered):
+        chunk.order = i
+
+    embedded_chunks = await embed_chunks(filtered, client.embedder)
+
+    await client.import_document(
+        docling_document=docling_doc,
+        chunks=embedded_chunks,
+    )
+```
+
+### Custom Embeddings
+
+Use your own embedding service:
+
+```python
+async def my_embedder(texts: list[str]) -> list[list[float]]:
+    """Your custom embedding function."""
+    # Call your embedding API here
+    ...
+
+async with HaikuRAG("database.lancedb", create=True) as client:
+    docling_doc = await client.convert("document.pdf")
+    chunks = await client.chunk(docling_doc)
+
+    # Use contextualize for consistent embedding input
+    texts = contextualize(chunks)
+
+    # Generate embeddings with your service
+    embeddings = await my_embedder(texts)
+
+    # Create chunks with embeddings
+    from haiku.rag.store.models.chunk import Chunk
+
+    embedded_chunks = [
+        Chunk(
+            content=chunk.content,
+            metadata=chunk.metadata,
+            order=chunk.order,
+            embedding=embedding,
+        )
+        for chunk, embedding in zip(chunks, embeddings)
+    ]
+
+    await client.import_document(
+        docling_document=docling_doc,
+        chunks=embedded_chunks,
+    )
+```

docs/development.md

@@ -0,0 +1,123 @@
+# Development
+
+This guide covers setting up a development environment and running tests.
+
+## Setup
+
+Clone the repository and install dependencies:
+
+```bash
+git clone https://github.com/ggozad/haiku.rag.git
+cd haiku.rag
+uv sync
+```
+
+## Running Tests
+
+```bash
+uv run pytest
+```
+
+### Test Markers
+
+Tests use pytest markers to categorize them:
+
+- `@pytest.mark.integration` - Tests requiring local services (Docling models, etc.) that aren't available in CI
+- `@pytest.mark.asyncio` - Async tests (applied automatically via pytest-asyncio)
+- `@pytest.mark.vcr()` - Tests with HTTP call recording
+
+CI runs `pytest -m "not integration"` to skip integration tests.
+
+## HTTP Recording with VCR
+
+Tests use [pytest-recording](https://github.com/kiwicom/pytest-recording) (VCR.py) to record and replay HTTP calls. This allows tests to run without external services like Ollama or API providers.
+
+### How It Works
+
+1. Tests marked with `@pytest.mark.vcr()` record HTTP interactions to YAML cassettes
+2. On subsequent runs, HTTP calls are replayed from cassettes instead of hitting real services
+3. Cassettes are committed to the repository so CI can run tests without external dependencies
+
+### Recording New Cassettes
+
+When adding a new test that makes HTTP calls:
+
+1. Add the `@pytest.mark.vcr()` decorator to your test
+2. Run the test with the required services available (e.g., Ollama running)
+3. The cassette is automatically created on first run
+
+### Re-recording Cassettes
+
+To update an existing cassette, delete it and re-run the test, or use `--record-mode=rewrite`.
+
+### Running Without Cassettes (Live Mode)
+
+To run tests against real services instead of recorded cassettes:
+
+```bash
+uv run pytest --disable-recording
+```
+
+## Writing Tests
+
+### Common Fixtures
+
+Available fixtures from `tests/conftest.py`:
+
+- `temp_db_path` - Isolated temporary database
+- `temp_yaml_config` - Temporary config file
+- `allow_model_requests` - Enables pydantic-ai model calls
+
+### Example: Adding a New Test with VCR
+
+```python
+import pytest
+from haiku.rag.client import HaikuRAG
+
+
+@pytest.mark.asyncio
+@pytest.mark.vcr()
+async def test_my_feature(temp_db_path):
+    async with HaikuRAG(temp_db_path, create=True) as client:
+        doc = await client.create_document("Test content", uri="test://doc")
+        assert doc.id is not None
+```
+
+### Integration Tests
+
+For tests requiring local services that can't be mocked via VCR:
+
+```python
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_pdf_visualization(temp_db_path):
+    # Test code that needs local PDF processing
+    pass
+```
+
+Integration tests are skipped in CI but run locally when you have the required services.
+
+## Linting and Formatting
+
+```bash
+uv run ruff check
+uv run ruff format
+uv run ty check
+```
+
+## Mock API Keys
+
+Tests automatically set mock API keys for providers that require them during client initialization. When running with VCR playback, these mock keys are sufficient since no real API calls are made.
+
+Recording reaches the real service, so the recording command needs network
+access and the keys that service reads. Name the exact test and pass `-n0`:
+a module-wide `--record-mode=rewrite` re-records every cassette in it,
+including ones whose service you do not have running.
+
+```bash
+# Ollama-backed cassettes need no key, only a running Ollama
+uv run pytest tests/test_embedder.py::test_ollama_embedder -n0 --record-mode=rewrite
+
+# A keyed provider reads its own variable. Cohere's SDK reads CO_API_KEY
+CO_API_KEY=... uv run pytest tests/test_reranker.py::test_cohere_reranker -n0 --record-mode=rewrite
+```

docs/index.md

@@ -1,66 +1,38 @@
-# haiku.rag
+---
+title: haiku.rag
+description: Local-first agentic RAG. Index PDFs, web pages, and whole directories, then ask questions and get answers cited to page numbers and section headings. Hybrid search, reranking, and multimodal retrieval on embedded LanceDB.
+---
 
-`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama, MixedBread AI) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
-
-## Features
-
-- **Local LanceDB**: No need to run additional servers
-- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own
-- **Native Hybrid Search**: Vector search combined with full-text search using native LanceDB RRF reranking
-- **Reranking**: Optional result reranking with MixedBread AI or Cohere
-- **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic
-- **File monitoring**: Automatically index files when run as a server
-- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, code files and more. Or add a URL!
-- **MCP server**: Exposes functionality as MCP tools
-- **CLI commands**: Access all functionality from your terminal
-  - Add sources from text, files, or URLs, optionally with a human‑readable title
-- **Python client**: Call `haiku.rag` from your own python applications
-
-## Quick Start
-
-Install haiku.rag:
+haiku.rag indexes PDFs, web pages, and whole directories, retrieves with hybrid search, and answers with citations down to the page number and section heading. It runs on an embedded database with open models, so your documents stay on your machine and there is no server to operate.
 
 ```bash
 uv pip install haiku.rag
+
+haiku-rag init
+haiku-rag add-src ~/Documents/some-paper.pdf
+haiku-rag ask "what does it conclude?"
 ```
 
-Use from Python:
+[Quickstart](tutorial.md) covers provider setup and the first ingestion.
 
-```python
-from haiku.rag.client import HaikuRAG
+## Why haiku.rag
 
-async with HaikuRAG("database.lancedb") as client:
-    # Add a document
-    doc = await client.create_document("Your content here")
+**Answers you can check.** Every answer carries citations with page numbers and section headings. Visual grounding shows the cited chunk highlighted on the original page image. Optional capabilities require an answer to declare what grounds it, including declaring that nothing does.
 
-    # Search documents
-    results = await client.search("query")
+**Local-first, no server.** Embedded [LanceDB](https://lancedb.com/) and open models through [Ollama](https://ollama.com/) by default. No database to run and no API keys required. The same code runs against S3, GCS, Azure, LanceDB Cloud, or any provider Pydantic AI supports.
 
-    # Ask questions
-    answer = await client.ask("Who is the author of haiku.rag?")
-```
+**Built for agents.** Native [Pydantic AI](https://ai.pydantic.dev/) capabilities compose into your own agents. An [MCP server](mcp.md) exposes the same database to Claude Desktop and other assistants. The analysis capability runs sandboxed Python across documents for questions that need computation rather than retrieval.
 
-Or use the CLI:
+**Measured, not asserted.** Retrieval and answer quality are tracked against public benchmarks with runnable configs. See [Benchmarks](benchmarks.md).
 
-```bash
-haiku-rag add "Your document content"
-haiku-rag add "Your document content" --meta author=alice
-haiku-rag add-src /path/to/document.pdf --title "Q3 Financial Report" --meta source=manual
-haiku-rag search "query"
-haiku-rag ask "Who is the author of haiku.rag?"
-```
+## Start here
 
-## Documentation
+- [Quickstart](tutorial.md): install, index, chat.
+- [Installation](installation.md): packages and extras.
+- [Architecture](overview.md): how a document becomes a cited answer.
+- [Capabilities](capabilities/index.md): native RAG and analysis capabilities for Pydantic AI agents.
+- [Python API](python.md): use haiku.rag from code.
+- [MCP server](mcp.md): expose haiku.rag to Claude Desktop or other AI assistants.
+- [Configuration](configuration/index.md): every setting.
 
-- [Getting started](tutorial.md) - Tutorial
-- [Installation](installation.md) - Install haiku.rag with different providers
-- [Configuration](configuration.md) - Environment variables and settings
-- [CLI](cli.md) - Command line interface usage
-- [Server](server.md) - File monitoring and server mode
-- [MCP](mcp.md) - Model Context Protocol integration
-- [Python](python.md) - Python API reference
-- [Agents](agents.md) - QA agent and multi-agent research
-
-## License
-
-This project is licensed under the [MIT License](https://raw.githubusercontent.com/ggozad/haiku.rag/main/LICENSE).
+MIT licensed. Source on [GitHub](https://github.com/ggozad/haiku.rag).

docs/ingester.md

@@ -0,0 +1,684 @@
+# Ingester
+
+The ingester is a long-running service that watches sources for
+changes and feeds documents into haiku.rag's LanceDB. It runs as a
+separate process (`haiku-ingester serve`), owns its own job queue
+(SQLite by default, or a database server), and exposes a small HTTP
+control plane for operations.
+
+Use the ingester when:
+
+- you have a corpus you want to keep in sync continuously
+- documents arrive over time from filesystem, S3, or HTTP sources
+- you want retry + dead-letter behavior, not "fire and forget"
+
+For one-off ingestion, the `haiku-rag add-src` CLI is enough — see
+[CLI → Add Documents](cli.md).
+
+**On this page:**
+
+- [Install](#install)
+- [Configure sources](#configure-sources) (FS, S3, HTTP, WebDAV)
+- [Workers and retry](#workers-and-retry)
+- [Circuit breaker](#circuit-breaker)
+- [Run it](#run-it)
+- [HTTP control plane](#http-control-plane)
+- [Operating](#operating) (smoke test, queue inspection, logs, API)
+
+Single-writer constraint: only one ingester per LanceDB. See
+[Storage → Deployment Pattern](configuration/storage.md#deployment-pattern-one-writer-many-readers).
+
+## Install
+
+The ingester ships behind an optional extra:
+
+```bash
+pip install 'haiku.rag-slim[ingester]'
+# or, for the full package:
+pip install 'haiku.rag[ingester]'
+```
+
+That pulls `fastapi`, `uvicorn`, `sqlalchemy`, `aiosqlite`, `asyncpg`, and
+the `[s3]` extra. The production binary is `haiku-ingester`.
+
+## Configure sources
+
+Add an `ingester:` block to your `haiku.rag.yaml`. The minimum is a
+single source:
+
+```yaml
+ingester:
+  sources:
+    - type: fs
+      id: local-docs
+      root: /Users/you/docs
+      delete_orphans: true
+```
+
+### Filesystem
+
+```yaml
+ingester:
+  sources:
+    - type: fs
+      id: local-docs                          # optional; auto-derives from root
+      root: /Users/you/docs
+      poll_interval_s: 300
+      delete_orphans: true
+      ignore_patterns: ["**/.git/**", "**/node_modules/**"]
+      include_patterns: ["*.md", "*.pdf"]    # optional whitelist
+```
+
+Uses `watchfiles` for push events plus a periodic sweep that catches
+anything the OS dropped between starts. Patterns follow
+[gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_format).
+
+### S3 / object storage
+
+```yaml
+ingester:
+  sources:
+    - type: s3
+      id: corp-docs
+      uri: s3://my-bucket/incoming/
+      poll_interval_s: 300
+      delete_orphans: true
+      ignore_patterns: ["draft*"]
+      include_patterns: ["*.pdf", "*.md"]
+      storage_options:
+        endpoint: http://seaweed:8333         # omit for AWS default chain
+        aws_access_key_id: ${AWS_KEY}
+        aws_secret_access_key: ${AWS_SECRET}
+        region: us-east-1
+        allow_http: "true"
+```
+
+ETags are the cheap-skip key. Each sweep lists the prefix, compares
+the listed ETag against the document's stored `metadata["source_revision"]`,
+and only fetches keys whose ETag has changed. If the bytes turn out to
+match the stored MD5 (multipart re-upload landing a new ETag on the
+same content), only the revision is refreshed — no re-chunk.
+
+`storage_options` follows the same convention as `lancedb.storage_options` —
+the dict is passed straight to obstore (the Rust `object_store` library
+LanceDB uses internally), so credentials configured for the LanceDB
+backend can be copy-pasted here.
+
+### HTTP
+
+```yaml
+ingester:
+  sources:
+    - type: http
+      id: arxiv
+      urls:
+        - https://arxiv.org/pdf/2301.12345.pdf
+      headers:
+        Authorization: Bearer ${SOME_TOKEN}
+      poll_interval_s: 86400
+```
+
+HTTP is pull-based with HEAD-driven change detection. A `410 Gone`
+response from a configured URL triggers a delete event; other failure
+statuses fall through to UPSERT-with-no-revision so the worker can
+GET and decide.
+
+### WebDAV
+
+```yaml
+ingester:
+  sources:
+    - type: webdav
+      id: nextcloud
+      base_url: https://nextcloud.example.com/remote.php/dav/files/alice/Documents/
+      username: alice
+      password: ${NEXTCLOUD_APP_PASSWORD}
+      ignore_patterns: ["**/Trash/**"]
+      poll_interval_s: 600
+```
+
+Each sweep issues one `PROPFIND` with `Depth: infinity` against
+`base_url` and parses the multistatus response. Files (non-collection
+resources) are emitted as UPSERT / UNCHANGED based on the `getetag`
+property (falling back to `getlastmodified` if the server omits it);
+URIs that were in the previous snapshot but no longer appear under the
+collection are emitted as DELETE.
+
+Fetches are plain HTTP GETs — any WebDAV server already supports them.
+
+Redirects are followed for both `PROPFIND` and `GET`, so front-ended
+servers that 30x on trailing-slash normalisation or scheme upgrades (e.g.
+Plone) work without extra configuration. Discovered URIs stay anchored to
+`base_url` (the `GET` fetch follows redirects to the bytes). A same-host
+scheme upgrade (`http`→`https`) is transparent; a redirect that moves the
+collection to a different path or host makes discovery raise so you can
+point `base_url` at the new location rather than silently dropping every
+file. Credentials are never replayed to a different host on a redirect.
+
+Bearer-token auth can replace HTTP Basic via the standard `headers` map:
+
+```yaml
+    - type: webdav
+      id: kdrive
+      base_url: https://kdrive.infomaniak.com/app/drive/123/
+      headers:
+        Authorization: Bearer ${KDRIVE_TOKEN}
+```
+
+### File size limits
+
+Any source can set `max_file_size` (bytes) to reject oversized files
+before they are read into memory. Files exceeding the limit go
+straight to the DLQ without retrying.
+
+```yaml
+    - type: fs
+      root: /data/docs
+      max_file_size: 104857600        # 100 MB
+```
+
+FS and S3 sources know the size before downloading (`stat`, object
+metadata), so the limit is always enforced. For HTTP and WebDAV the check
+relies on a `Content-Length` response header; a server that omits it (for
+example a chunked response) is fetched in full and the limit does not
+apply.
+
+### Metadata providers
+
+A source can attach custom metadata to every document it ingests by
+naming a `metadata_provider`. The provider is a callable that an external
+package registers under the `haiku.rag.metadata_providers` entry-point
+group; when the document is fetched for ingestion, the ingester calls it
+with `(source_id, uri, result)`, where `result` is the source's
+`FetchResult`, and merges the returned dict into the document's metadata.
+
+```yaml
+    - type: webdav
+      id: handbook
+      base_url: https://dav.example.com/remote.php/dav/files/svc
+      metadata_provider: example-provider
+```
+
+The provider is a zero-argument callable returning the provider instance,
+so a class is its own factory:
+
+```python
+# example_pkg/__init__.py
+from urllib.parse import urlparse
+
+from haiku.rag.sources import FetchResult
+
+
+class Provider:
+    async def __call__(
+        self, source_id: str, uri: str, result: FetchResult
+    ) -> dict:
+        path = urlparse(uri).path
+        return {
+            "collection": source_id,
+            "folder": path.rsplit("/", 1)[0] or "/",
+            "bytes": str(len(result.body)),
+        }
+```
+
+```toml
+# in the provider package's pyproject.toml
+[project.entry-points."haiku.rag.metadata_providers"]
+example-provider = "example_pkg:Provider"
+```
+
+The provider is built once at startup, so it can hold a client or cache
+across calls. When a document's source revision is unchanged, the
+ingester keeps the existing cheap HEAD short-circuit and preserves the
+stored provider metadata; the provider runs again when the document is
+fetched for a new or changed revision. The source-derived keys (`md5`,
+`source_revision`, `content_type`) are stripped from provider output, so
+a provider cannot override them. A `metadata_provider` name with no
+installed entry point fails at startup. A provider exception is
+classified like any other ingestion error (network and timeout errors
+retry; others go to the DLQ).
+
+### Custom sources
+
+The four built-in source types (`fs`, `http`, `s3`, `webdav`) cover the
+common cases. To ingest from something else (a git host, a ticketing
+system, a bespoke API), an external package registers a source factory
+under the `haiku.rag.sources` entry-point group and a config references it
+with `type: plugin`.
+
+```yaml
+    - type: plugin
+      id: api-docs
+      plugin: git
+      options:
+        owner: acme
+        repo: api
+        branch: main
+        token: ${SCM_TOKEN}
+```
+
+`plugin` is the entry-point name. `options` is an opaque mapping passed
+straight to the factory, which validates it however it likes (for example
+with its own Pydantic model). The base fields on every source
+(`id`, `poll_interval_s`, `delete_orphans`, `max_file_size`, `retry`,
+`circuit_breaker`, `metadata_provider`) are handled by the ingester and
+are not part of `options`.
+
+The factory is called with the source id, the validated `options`, and the
+ambient extension and size limits, and returns a `Source`:
+
+```python
+def __call__(
+    self,
+    *,
+    source_id: str,
+    options: dict,
+    supported_extensions: list[str] | None,
+    max_file_size: int | None,
+) -> Source: ...
+```
+
+A `Source` implements this protocol:
+
+```python
+class Source(Protocol):
+    source_id: str
+
+    def supports(self, uri: str) -> bool: ...
+
+    # Current revision for `uri`, cheaply, or None if there is no cheap
+    # lookup. Lets the pipeline skip re-ingest when the revision is unchanged.
+    async def head(self, uri: str) -> str | None: ...
+
+    # Release resources (connection pools, etc.). Called once at shutdown.
+    async def aclose(self) -> None: ...
+
+    async def fetch(self, uri: str) -> FetchResult: ...
+
+    # Yield UPSERT / UNCHANGED / DELETE events. `since` is the uri -> revision
+    # snapshot from the previous sweep so the source can emit only deltas.
+    def discover(
+        self,
+        since: RevisionSnapshot | None = None,
+        *,
+        known_uris: set[str] | None = None,
+    ) -> AsyncIterator[SourceEvent]: ...
+```
+
+`FetchResult`, `SourceEvent`, `SourceEventKind`, and `RevisionSnapshot`
+live in `haiku.rag.sources`.
+
+```toml
+# in the source package's pyproject.toml
+[project.entry-points."haiku.rag.sources"]
+git = "example_pkg:build_git_source"
+```
+
+Only the plugin a source references is imported, so an unused plugin with a
+missing optional dependency does not break startup. A `plugin` name with no
+installed entry point fails at startup, as does a factory that returns
+something that is not a `Source`.
+
+Two limits to know:
+
+- Custom sources are reached through configured discovery and the job
+  queue, not through one-shot `haiku-rag add-src <uri>`, which only knows
+  the built-in URI schemes.
+- Change detection is per `(source, uri)`. A source that needs a single
+  per-source cursor (for example a git last-commit SHA) tracks it itself,
+  by encoding it in each URI's revision or stashing it under a sentinel URI.
+
+## Workers and retry
+
+```yaml
+ingester:
+  workers:
+    worker_count: 4
+    poll_idle_interval_s: 1.0
+    lease_ttl_s: 120
+    heartbeat_interval_s: 30
+    reaper_interval_s: 60
+    shutdown_grace_s: 60            # SIGTERM drains in-flight up to this long
+    retry:
+      max_attempts: 5
+      base_delay_s: 2.0
+      max_delay_s: 300.0
+      jitter: 0.25                  # ±25%
+```
+
+The worker pool runs `worker_count` async workers, each processing one
+job at a time. `worker_count` is therefore also the maximum number of
+concurrent in-flight jobs. Jobs that hit a `TransientError` are
+rescheduled with exponential backoff plus jitter, up to `max_attempts`,
+then land in the dead-letter queue. `PermanentError` (unsupported
+extension, 4xx HTTP except 408/429, object-store credential and
+configuration errors, etc.) skips retry entirely.
+
+While a worker processes a job it renews the job's lease every
+`heartbeat_interval_s`. A reaper task resets any claim whose lease has not
+been renewed within `lease_ttl_s` so a crashed worker doesn't strand its
+job. Because a live worker keeps renewing, `lease_ttl_s` need not exceed
+job duration — a slow job is not reaped while it is still running.
+
+**Backpressure.** Each poller skips its periodic sweep when its source
+already has queued or claimed jobs in the queue. The unique-index dedup
+would coalesce a re-sweep anyway; the skip saves the listing round-trip
+(`PROPFIND` / `S3 LIST` / FS walk). FS push events from `watchfiles`
+still flow during a skipped sweep, so new files aren't lost.
+
+**Graceful shutdown.** On `SIGINT` / `SIGTERM`, pollers stop immediately
+and workers are given `shutdown_grace_s` to finish in-flight jobs. Jobs
+still running after the grace window are cancelled and released back to
+`queued` for immediate re-claim; any release that doesn't land has its
+lease lapse and is reclaimed by the reaper after `lease_ttl_s`.
+
+**Tuning.**
+
+- `lease_ttl_s` bounds how long a crashed worker's job stays stuck before
+  another worker takes it over. It no longer needs to exceed job duration,
+  so it can be short; keep it well above `heartbeat_interval_s`.
+- `heartbeat_interval_s` must be at most `lease_ttl_s / 3` so scheduler
+  jitter or a slow DB round-trip can't let a live job's lease lapse.
+- `worker_count` should match downstream capacity. docling-serve
+  processes one task per instance, so `worker_count` above the number
+  of `providers.docling_serve.base_url` entries over-subscribes the
+  fleet — extra submissions queue inside docling-serve. They are not
+  reaped while queued because the worker keeps renewing the lease.
+- `poll_idle_interval_s`: lower = faster pickup, more SQLite churn.
+- `reaper_interval_s`: worst-case post-crash reclaim is
+  `lease_ttl_s + reaper_interval_s`.
+
+**Per-source override.** A source can opt out of the global retry
+policy:
+
+```yaml
+ingester:
+  sources:
+    - type: http
+      id: flaky-api
+      urls: [...]
+      retry:
+        max_attempts: 10
+        base_delay_s: 10
+```
+
+## Circuit breaker
+
+After N consecutive `discover()` failures, a source's circuit breaker
+opens and polling pauses for a cooldown. Other sources keep running.
+
+```yaml
+ingester:
+  sources:
+    - type: http
+      id: rate-limited
+      urls: [...]
+      circuit_breaker:
+        failure_threshold: 5
+        cooldown_s: 600
+```
+
+## Run it
+
+```bash
+haiku-ingester serve                          # workers + pollers + API
+haiku-ingester serve --no-api                 # workers + pollers only
+haiku-ingester serve --db /path.lancedb       # explicit DB
+haiku-ingester serve --host 0.0.0.0           # bind API on all interfaces
+haiku-ingester serve --port 9000              # override API port
+```
+
+`--host` and `--port` are CLI overrides for `ingester.api.host` and
+`ingester.api.port` in `haiku.rag.yaml`. Both default to the YAML value
+(which itself defaults to `127.0.0.1:8765` — loopback only).
+
+The service blocks until SIGINT or SIGTERM. Shutdown drains the API
+server, then pollers, then in-flight workers.
+
+### Single-writer constraint
+
+haiku.rag serializes multi-table writes with a process-local lock and rolls
+them back by restoring table versions, so a second writing process can
+commit inside another's transaction and be reverted by its rollback. Run
+exactly one `haiku-ingester serve` against a given LanceDB. Multiple
+MCP servers or read-only consumers against the same DB are fine. Sharing
+the Postgres queue across processes is safe (the claim/lease lifecycle is
+cross-process-correct) but does not relax this constraint — it governs the
+queue, not the LanceDB.
+
+## HTTP control plane
+
+By default the ingester exposes a FastAPI control plane on
+`127.0.0.1:8765`. Set `ingester.api.auth_token` to require a Bearer
+token; without one the API stays open and the service logs a warning.
+
+!!! warning "Non-loopback binds need a token"
+    Loopback (`127.0.0.1`) is local-only and safe to leave open. If
+    you bind to any other interface (`0.0.0.0`, a LAN IP, behind a
+    reverse proxy) **set `auth_token`** — the control plane can
+    cancel jobs, retry from the DLQ, and trigger source refreshes.
+    The startup warning is your only signal that you forgot.
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/` | browser dashboard (HTML; unauthenticated, the JS attaches the bearer on its own JSON fetches) |
+| `GET` | `/health` | liveness + queue counts + live worker/poller counts; `status` is `"ok"` or `"degraded"` |
+| `GET` | `/sources` | configured pollers + last-poll time + breaker state + last skip reason |
+| `POST` | `/sources/{id}/refresh` | force an out-of-band sweep |
+| `GET` | `/jobs` | filtered list (`status`, `source_id`, `uri`, `limit`, `offset`) |
+| `GET` | `/jobs/{id}` | one job |
+| `POST` | `/jobs/{id}/retry` | reset attempts to 0, status to queued |
+| `DELETE` | `/jobs/{id}` | cancel a queued/claimed job |
+| `GET` | `/dlq` | dead jobs |
+| `POST` | `/dlq/{id}/retry` | resurrect from DLQ |
+| `GET` | `/stats` | rolling throughput (5m / 30m / 1h succeeded), worker occupancy, oldest queued age, per-source DLQ + backlog |
+| `GET` | `/database` | LanceDB snapshot — stored version, embeddings, per-table row counts/sizes, vector index status, pending migrations, package versions (same data as `haiku-rag info`) |
+| `GET` | `/config` | full effective configuration (defaults filled in) as YAML, with secrets redacted |
+
+OpenAPI docs at `http://localhost:8765/docs`. The dashboard at `/` polls
+the JSON endpoints above every few seconds and surfaces the same data
+visually — queue depth chips, per-source health with a `queue busy` badge
+when sweeps are skipped, throughput counters, active jobs with a Cancel
+button, recent failures with a Retry button, and the last-completed
+feed. The Database and Configuration panels are collapsed and load on
+demand (the Database panel has a Refresh button) rather than on the poll
+loop.
+
+![Ingester dashboard mid-ingest: queue depth, per-source health, active and recent jobs](img/ingester-dashboard.png)
+
+```yaml
+ingester:
+  api:
+    enabled: true
+    host: 127.0.0.1
+    port: 8765
+    auth_token: secret                        # null → unauthenticated
+    root_path: ""                             # e.g. /ingester behind a proxy
+```
+
+### Behind a reverse proxy
+
+To serve the control plane under a sub-path (so a reverse proxy can front
+it alongside other services on one origin, e.g. `https://host/ingester/`),
+set `ingester.api.root_path` (or `serve --root-path /ingester`). It is
+forwarded to FastAPI/uvicorn as `root_path` — OpenAPI/`/docs` links become
+prefix-aware — and the dashboard is served with a matching `<base href>` so
+its JSON fetches resolve under the prefix. The value is normalized to a
+single leading slash with no trailing slash (`ingester`, `/ingester/` and
+`/` become `/ingester`, `/ingester` and `""`). Strip the prefix at the proxy
+before forwarding; for example, with nginx:
+
+```nginx
+# Redirect the bare prefix to the trailing-slash form so the dashboard's
+# <base href> resolves correctly.
+location = /ingester {
+    return 308 /ingester/;
+}
+
+location /ingester/ {
+    rewrite ^/ingester/?(.*)$ /$1 break;
+    proxy_pass http://127.0.0.1:8765;
+}
+```
+
+## Operating
+
+### One-shot batch build
+
+`run-batch` runs a single discover sweep across every configured source,
+drains the queue, then exits. New and changed resources are ingested,
+resources that vanished from a source are deleted. The periodic poller
+loops never start, so the run is deterministic and finishes as soon as the
+queue is empty. This is the mode for building a database in CI or on a
+schedule rather than running the service continuously.
+
+```bash
+haiku-ingester run-batch
+haiku-ingester run-batch --db rag.lancedb
+```
+
+To review a batch before it mutates the document store, use `--dry-run`.
+Dry-run performs the same discovery checks but writes no queue jobs and does
+not update `sync_state`. It writes a YAML manifest named
+`manifest-<datestamp>.yaml` by default:
+
+```bash
+haiku-ingester run-batch --dry-run
+haiku-ingester run-batch --dry-run --output manifest-20260622.yaml
+```
+
+The manifest records the `upsert` and `delete` changes discovered for each
+source. Replay it later to ingest exactly that changeset, without another
+discovery sweep:
+
+```bash
+haiku-ingester run-batch --manifest manifest-20260622.yaml
+```
+
+Manifest replay rejects sources with queued or claimed work, preserving the
+one-active-changeset-per-source pattern. Revisioned upserts are checked
+against the current upstream revision before fetch; if the resource changed
+after dry-run, that job dead-letters and the newer version waits for the next
+dry-run. Sources that provide no revision can freeze URI discovery but cannot
+prove byte identity at replay time.
+
+Orphan deletion compares each source against `sync_state` in the queue DB,
+so persist `ingester.db` between runs for deletions to be detected. It exits
+non-zero if any job dead-letters or a source's discovery sweep does not
+complete.
+
+### The queue
+
+The ingester's SQLite queue lives at
+`~/Library/Application Support/haiku.rag/ingester.db` on macOS
+(platform user data dir; configurable via `ingester.queue.path`). It's
+created automatically by `serve`.
+
+For ops setup you can pre-create it:
+
+```bash
+haiku-ingester queue init             # create the DB and schema
+haiku-ingester queue migrate          # apply pending schema changes
+```
+
+Terminal job rows (`succeeded` and `dead`) are kept for history and pruned by
+the reaper once they age past `retention_days`:
+
+```yaml
+ingester:
+  queue:
+    path: /var/lib/haiku-rag/ingester.db
+    retention_days: 30                # null disables pruning
+```
+
+The reaper deletes terminal rows whose `completed_at` is older than the window
+on its `reaper_interval_s` cadence. Set `retention_days: null` to keep all
+terminal rows.
+
+#### Using a database server
+
+If you already run a database server, point the queue at it with
+`ingester.queue.dburi`, a SQLAlchemy async URL. SQLite is used when `dburi` is
+unset.
+
+```yaml
+ingester:
+  queue:
+    dburi: postgresql+asyncpg://haiku:secret@db:5432/haiku_rag
+```
+
+Postgres (`postgresql+asyncpg://`) is supported alongside the default SQLite.
+The `asyncpg` driver ships with the `[ingester]` extra. `dburi` overrides
+`path`, and the `--queue` CLI flag is ignored while it is set. Create the schema
+the same way as for SQLite:
+
+```bash
+haiku-ingester queue init
+```
+
+Workers claim jobs with `FOR UPDATE SKIP LOCKED`, and the claim/lease lifecycle
+is cross-process-safe — claims are renewed and reaped correctly no matter which
+process owns them — so several `haiku-ingester serve` processes can share one
+Postgres queue without double-claiming or reaping each other's live jobs.
+
+This does not lift the LanceDB
+[single-writer constraint](#single-writer-constraint): each `serve` still owns
+its own LanceDB. A shared queue therefore spans processes writing distinct
+LanceDB URIs; it does not let several processes write one database.
+
+One caveat: idle workers wake on new work instantly only within their own
+process. Workers in other processes pick up enqueued jobs on their next
+`poll_idle_interval_s` tick rather than immediately.
+
+### Logs
+
+The service logs via Python `logging` to stderr through a Rich handler.
+A typical run looks like:
+
+```
+INFO     Ingester running: 4 worker(s), 1 source(s)
+INFO     API listening on 127.0.0.1:8765
+INFO     Swept local-docs: 142 upsert, 0 delete, 8 unchanged
+INFO     Processing upsert file:///.../a.md (job 5d9a...)
+INFO     Job 5d9a... succeeded in 0.34s: file:///.../a.md
+```
+
+When `LOGFIRE_TOKEN` is set, spans are also shipped to Logfire. Spans carry
+`service.name` (`haiku-ingester`) and `service.version`. To tell concurrent
+ingestions apart in Logfire, give each process a distinct name via the standard
+`OTEL_SERVICE_NAME` (or `LOGFIRE_SERVICE_NAME`) environment variable, which
+overrides the default:
+
+```bash
+OTEL_SERVICE_NAME=ingester-tenant-a haiku-ingester serve
+```
+
+The span tree is `ingester.poller.sweep` -> `ingester.job` (tagged with
+`source_id` and `uri`) -> `document.convert` / `document.chunk`. When a source
+uses docling-serve, each request emits a `docling_serve.request` span carrying
+the instance `url` and `attempt`, so a failed conversion can be traced to the
+exact instance that served it. A worker circuit breaker opening emits an
+`ingester.worker breaker opened` event with `source_id`, `threshold`, and
+`cooldown_s`.
+
+The `debug-ingestion` skill in `.claude/skills/` turns these spans into
+ready-made Logfire queries (failed jobs, docling-serve failover, per-source
+sweeps, breaker trips) for use from Claude Code.
+
+### Operating against the API
+
+```bash
+TOKEN=$INGESTER_TOKEN   # omit -H entirely if no token configured
+
+curl http://localhost:8765/health
+curl -H "Authorization: Bearer $TOKEN" http://localhost:8765/sources
+curl -H "Authorization: Bearer $TOKEN" 'http://localhost:8765/jobs?status=dead'
+
+# Force a poll now
+curl -H "Authorization: Bearer $TOKEN" -X POST \
+    http://localhost:8765/sources/local-docs/refresh
+
+# Resurrect a dead job
+curl -H "Authorization: Bearer $TOKEN" -X POST \
+    http://localhost:8765/jobs/<id>/retry
+```

docs/installation.md

@@ -10,40 +10,56 @@
 uv pip install haiku.rag
 ```
 
-The full package includes **all features and extras**:
-- **Document processing** (Docling) - PDF, DOCX, PPTX, images, and 40+ file formats
-- **All embedding providers** - VoyageAI
-- **All rerankers** - MixedBread AI, Cohere, Zero Entropy
+The full package pulls the `docling`, `voyageai`, `cohere`, `zeroentropy`,
+`cross-encoder`, `jina` and `tui` extras. It does not include `s3` or `ingester`:
 
-This is the easiest way to get started with all features enabled.
+```bash
+uv pip install 'haiku.rag[ingester]'   # the haiku-ingester service
+uv pip install 'haiku.rag[s3]'         # S3 and object storage
+```
 
 ### Slim Package (Minimal Dependencies)
 
 ```bash
-# Minimal installation (no document processing)
 uv pip install haiku.rag-slim
-
-# With document processing
-uv pip install haiku.rag-slim[docling]
-
-# With specific providers
-uv pip install haiku.rag-slim[docling,voyageai,mxbai]
+uv pip install 'haiku.rag-slim[docling]'
+uv pip install 'haiku.rag-slim[docling,voyageai,cross-encoder]'
 ```
 
-The slim package has minimal dependencies and lets you install only what you need:
+### Extras
 
-- `docling` - PDF, DOCX, PPTX, images, and other document formats
-- `voyageai` - VoyageAI embeddings
-- `mxbai` - MixedBread AI reranking
-- `cohere` - Cohere reranking
-- `zeroentropy` - Zero Entropy reranking
+Every extra `haiku.rag-slim` defines. The right-hand column marks the ones the
+full `haiku.rag` package already includes.
+
+| Extra | Provides | In `haiku.rag` |
+|---|---|---|
+| `docling` | PDF, DOCX, PPTX, images and 40+ formats, converted locally | yes |
+| `tui` | Terminal UI for `chat` and `inspect` | yes |
+| `voyageai` | VoyageAI embeddings | yes |
+| `cohere` | Cohere embeddings and reranking | yes |
+| `zeroentropy` | Zero Entropy reranking | yes |
+| `cross-encoder` | Local reranking via sentence-transformers | yes |
+| `jina` | Local Jina reranking (`provider: jina-local`) | yes |
+| `s3` | S3 and object-storage access | no |
+| `ingester` | The `haiku-ingester` service (also pulls `s3`) | no |
+| `anthropic` | Anthropic Claude models | no |
+| `google` | Google Gemini models | no |
+| `groq` | Groq models | no |
+| `mistral` | Mistral models | no |
+| `bedrock` | AWS Bedrock models | no |
+| `vertexai` | Google Vertex AI models | no |
+
+Ollama and any OpenAI-compatible endpoint work with no extra at all.
 
 **Built-in providers** (no extras needed):
 - **Ollama** (default embedding provider)
 - **OpenAI** (GPT models for QA and embeddings)
-- **Anthropic** (Claude models for QA)
+- **vLLM** and other OpenAI-compatible endpoints (embeddings, QA, reranking)
+- **Jina** reranking via `provider: jina`, which calls the Jina HTTP API
 
-See [Configuration](configuration.md) for configuring providers including advanced options like vLLM.
+Other providers come from the extras above, which pull the matching Pydantic AI extra. For Claude models, `uv pip install 'haiku.rag-slim[anthropic]'`.
+
+See [Configuration](configuration/index.md) for configuring providers including advanced options like vLLM.
 
 ## Requirements
 
@@ -58,18 +74,45 @@ You can prefetch all required runtime models before first use:
 haiku-rag download-models
 ```
 
-This will download Docling models and pull any Ollama models referenced by your current configuration.
+This will download:
+- Docling models for document processing
+- HuggingFace tokenizer models for chunking
+- Any Ollama models referenced by your current configuration
+
+## Remote Processing (Optional)
+
+When using `haiku.rag-slim`, you can skip installing the `docling` extra and instead use [docling-serve](https://github.com/docling-project/docling-serve) for remote document processing. This is useful for:
+
+- Keeping dependencies minimal
+- Offloading heavy document processing to a dedicated service
+- Production deployments with separate processing infrastructure
+
+See [Remote processing](remote-processing.md) for setup instructions and [Document Processing](configuration/processing.md) for configuration options.
 
 ## Docker
 
-```bash
-docker pull ghcr.io/ggozad/haiku.rag:latest
-```
+Only the slim image is published. Build the full image yourself:
 
-Run the container with all services:
+### Slim Image (Minimal)
+
+Pre-built slim image with minimal dependencies - use with external docling-serve for document processing:
 
 ```bash
-docker run -p 8000:8000 -p 8001:8001 -v $(pwd)/data:/data ghcr.io/ggozad/haiku.rag:latest
+docker pull ghcr.io/ggozad/haiku.rag-slim:latest
 ```
 
-This starts the MCP server on port 8001, with data persisted to `./data`.
+See `examples/docker/docker-compose.yml` for a complete setup with docling-serve.
+
+### Full Image (Self-contained)
+
+Build locally to include all features and document processing without docling-serve:
+
+```bash
+docker build -f docker/Dockerfile -t haiku-rag .
+docker run -p 8001:8001 \
+  -v /path/to/haiku.rag.yaml:/app/haiku.rag.yaml \
+  -v /path/to/data:/data \
+  haiku-rag
+```
+
+See `docker/README.md` for complete build and configuration instructions, including how to run the [ingester](ingester.md) service for continuous document ingestion.

docs/mcp.md

@@ -1,30 +1,210 @@
 # Model Context Protocol (MCP)
 
-The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients.
-
-## Available Tools
-
-### Document Management
-
-- `add_document_from_file` - Add documents from local file paths
-- `add_document_from_url` - Add documents from URLs
-- `add_document_from_text` - Add documents from raw text content
-- `get_document` - Retrieve specific documents by ID
-- `list_documents` - List all documents with pagination and optional filtering
-- `delete_document` - Delete documents by ID
-
-### Search
-
-- `search_documents` - Search documents using hybrid search (vector + full-text)
+The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like Claude Desktop.
 
 ## Starting MCP Server
 
-The MCP server starts automatically with the serve command and supports Streamable HTTP and stdio transports:
+The MCP server supports Streamable HTTP and stdio transports:
 
 ```bash
-# Default streamable HTTP transport
-haiku-rag serve
+# Default streamable HTTP transport on 127.0.0.1:8001
+haiku-rag mcp
+
+# Custom port
+haiku-rag mcp --port 9000
+
+# Bind to all interfaces (e.g. inside a container)
+haiku-rag mcp --host 0.0.0.0 --port 8001
 
 # stdio transport (for Claude Desktop)
-haiku-rag serve --stdio
+haiku-rag mcp --stdio
+
 ```
+
+`--host` defaults to `127.0.0.1` (loopback only). Bind to `0.0.0.0` only
+when you want the MCP server reachable from outside the local machine —
+e.g. inside a Docker container with port mapping, or on a trusted LAN.
+
+The server opens the database read-only. Ingestion goes through the CLI
+(`haiku-rag add`, `add-src`, `delete`) or [`haiku-ingester`](ingester.md).
+
+## Collections
+
+With several databases in `lancedb.databases`, the server covers all of
+them, as `haiku-rag search` does. Results and documents name theirs in
+`source`. `sources` on `search_documents`, `search_documents_by_image`
+and `execute_code` restricts a call to a subset; `source` on `get_document` names the database holding the
+document. A name the server does not cover is an error.
+`haiku-rag --db-name NAME mcp` serves one. See
+[Multiple Databases](configuration/storage.md#multiple-databases).
+
+## Claude Code
+
+The repository ships a plugin that registers the server and a skill telling
+Claude when and how to use it:
+
+```bash
+claude plugin marketplace add ggozad/haiku.rag
+claude plugin install haiku-rag
+```
+
+The plugin runs `haiku-rag mcp --stdio`, so `haiku-rag` must be on the PATH
+and the configuration decides the database. The skill pre-approves every tool
+and is also invocable as `/haiku-rag`. To register the server without the
+plugin:
+
+```bash
+claude mcp add haiku-rag -- haiku-rag mcp --stdio
+```
+
+The skill works with that registration too: copy `plugins/haiku-rag/skills/haiku-rag`
+into `~/.claude/skills/` and change the tool prefix in its `allowed-tools` from
+`mcp__plugin_haiku-rag_haiku-rag__` to `mcp__haiku-rag__`.
+
+## Codex
+
+The repository's Codex plugin registers the server and installs the same Agent
+Skill:
+
+```bash
+codex plugin marketplace add ggozad/haiku.rag
+codex plugin add haiku-rag@haiku-rag
+```
+
+The plugin runs `haiku-rag mcp --stdio`, so `haiku-rag` must be on the PATH.
+Invoke the skill as `$haiku-rag`. Codex can also select it automatically from
+its description. To register the server without the plugin:
+
+```bash
+codex mcp add haiku-rag -- haiku-rag mcp --stdio
+```
+
+The skill works with that registration too: copy
+`plugins/haiku-rag/skills/haiku-rag` into `~/.agents/skills/`.
+The `allowed-tools` field supplies Claude Code's tool pre-approval and may be
+ignored by other Agent Skills clients. Codex configures MCP tool approvals
+separately in `config.toml`.
+
+## Claude Desktop Integration
+
+Add to your Claude Desktop configuration (`claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["mcp", "--stdio"]
+    }
+  }
+}
+```
+
+With a custom database path:
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["mcp", "--stdio", "--db", "/path/to/database.lancedb"]
+    }
+  }
+}
+```
+
+After restarting Claude Desktop, you can ask Claude to search your documents or answer questions using your knowledge base.
+
+## Tools
+
+Every tool is read-only and says so in its annotations. Each parameter carries
+a description in the tool schema, so the listing below names them without
+repeating it.
+
+| Tool | Registered | Parameters |
+|---|---|---|
+| `search_documents` | always | `query`, `limit`, `include_images`, `filter`, `sources` |
+| `search_documents_by_image` | multimodal embedder only | `image_base64`, `limit`, `include_images`, `filter`, `sources` |
+| `get_document` | always | `document_id`, `source` |
+| `get_document_outline` | always | `document_id`, `source` |
+| `get_document_section` | always | `document_id`, `section_id`, `source` |
+| `list_documents` | always | `limit`, `offset`, `filter` |
+| `execute_code` | always | `code`, `filter`, `sources` |
+
+`search_documents` runs hybrid search, vector and full-text. Its text content
+is the rendering the in-process agents read: results best first, each with its
+rank, `Document ID`, `Collection` when the server covers several, the document
+title, section headings, the matched chunk's metadata when it has any, and the
+passage expanded to its section the way the agents get it
+(`search.max_context_chars` caps it). Pictures in the results follow as
+image blocks, one per distinct picture, each preceded by a line naming its
+result; `include_images: false` leaves them out. Search results carry no
+structured content, so every client shows the model the same text and
+images. Scores are not comparable across
+queries or search types, so rank is the signal. `search_documents_by_image`
+embeds the query image and searches by vector similarity alone.
+
+`get_document` returns a document whole, in reading order. For a long one,
+`get_document_outline` returns the heading tree with page numbers and
+`get_document_section` the text of one section, subsections included; a
+node's `id` in the outline is the `section_id`. A document without headings
+has an empty outline. `list_documents` returns titles, URIs and metadata,
+which is how a client learns what a filter can match.
+
+### Code
+
+`execute_code` runs a Python program in the sandbox of the
+[analysis capability](capabilities/analysis.md), over the documents `filter`
+and `sources` select, and returns what it printed. The program reads
+`/documents/{document_id}/` (`metadata.json`, `content.txt`, `items.jsonl`,
+`chunks.jsonl`, `toc.json`) and can `await search()` and
+`await list_documents()`; the tool description spells out the fields and the
+patterns that matter. Each call is one program: nothing carries over between
+calls, and the sandbox is created and closed per call. A failing program is a
+tool error carrying the interpreter's message and any output printed before
+it. No model runs on the server. Claude Code moves a call still running after
+about two minutes to a background task.
+
+The interpreter is [Monty](https://github.com/pydantic/monty), a Python subset.
+Useful modules include `json`, `re`, `math`, `pathlib`, `datetime`,
+`collections`, `itertools`, `functools` and `dataclasses`. Absent, and often
+reached for: `decimal` and `statistics`. No generator functions, class
+inheritance or `match` statements, and a file object cannot be iterated. Files are read-only, and
+there is no network and no filesystem
+beyond `/documents`. `analysis.code_timeout` is the call's budget: compute is
+stopped at it, and past it no further host call starts, a file read or an
+in-code search alike, though one already running finishes.
+`analysis.max_output_chars` bounds the output.
+
+### Filters
+
+`filter` is a SQL WHERE clause over the document columns `id`, `uri`, `title`,
+`metadata`, `created_at`, `updated_at`. `metadata` is a JSON string, so match
+its keys with LIKE:
+
+```sql
+metadata LIKE '%"author": "Smith"%'
+uri LIKE '%.pdf'
+title = 'Q3 report'
+```
+
+### Errors
+
+A failure is an MCP error carrying its message, never an empty result: a
+document or section id that matches nothing, a collection the server does not
+cover, a filter the query engine rejects, invalid base64, a program that fails
+in `execute_code` with the error it hit, and anything unexpected with its own
+message.
+
+### Instructions
+
+The server publishes `instructions` describing the knowledge base: what it
+holds, when to reach for it, the collection names when it covers several, and
+`prompts.domain_preamble` when set. Claude Code and Codex show them to the
+model. Claude Desktop does not, so every tool description stands on its own.
+
+## Continuous ingestion
+
+For continuous document ingestion (filesystem watch, S3 polling, HTTP
+sources, a job queue with retries), run [`haiku-ingester`](ingester.md)
+as a separate process against the same LanceDB.

docs/overview.md

@@ -0,0 +1,90 @@
+# Architecture
+
+haiku.rag ingests documents, retrieves from them with hybrid search, and answers
+with citations. This page follows the data through the system. For a working
+setup, start with the [Quickstart](tutorial.md).
+
+## Ingestion
+
+```text
+source adapter -> converter -> chunker -> embedder -> LanceDB
+```
+
+A **source adapter** owns the I/O and the identity of a document: it fetches
+bytes, reports the backend's revision (mtime for a file, ETag for S3 or HTTP),
+and computes the content hash. The same adapters serve one-shot ingestion
+(`haiku-rag add-src`, `HaikuRAG.create_document_from_source`) and the continuous
+[`haiku-ingester`](ingester.md) service, so both agree on what a document is and
+when it has changed.
+
+The **converter** turns those bytes into a `DoclingDocument`, the structured form
+that carries headings, tables, pictures and page provenance. It runs in-process
+with the `docling` extra, or against a [docling-serve](remote-processing.md)
+fleet.
+
+The **chunker** splits that structure into chunks, each keeping the headings it
+sits under, the page numbers it came from, and references to the document items
+it covers. With a multimodal embedder, pictures become chunks of their own.
+
+The **embedder** vectorizes them in batches. The document, its mutable metadata,
+its chunks and its structural items are written under one process-local
+transaction: it takes a version snapshot, and on failure restores each table to
+it. A rollback that cannot complete raises rather than reporting success, and the
+snapshot is only meaningful while this process is the only writer.
+
+## Storage
+
+LanceDB is embedded, so there is no server. The same code runs against a local
+directory, S3, GCS, Azure or LanceDB Cloud by changing a database's location in
+`lancedb.databases`.
+
+Tables are versioned. Vacuum collapses old versions on a retention window, and
+[tags](cli.md) name a state across all tables so a database can be restored to
+it later.
+
+One process writes at a time. Reads are unrestricted, and a reader sees another
+process's writes after `lancedb.read_consistency_interval_seconds`.
+
+## Retrieval
+
+```text
+query -> vector + full-text search -> fusion -> rerank -> context expansion
+```
+
+Search runs a vector query and a full-text query and fuses the rankings. With a
+reranker configured, it retrieves ten times the requested limit and reranks down
+to it, so quality improves without changing the caller's limit.
+
+Results then expand: a chunk is returned with the section it belongs to, bounded
+by `search.max_context_chars`. Sections that fit come back whole, larger ones
+grow outward from the match, and small ones grow across boundaries. Every result
+carries its page numbers and headings, which is what makes a citation checkable.
+
+## Answering
+
+Two [capabilities](capabilities/index.md) sit on top, both native Pydantic AI
+capabilities you can attach to your own agent:
+
+- The **RAG capability** searches and cites. Its citations carry page numbers and
+  headings, and `haiku-rag visualize` draws the cited chunk on the page image.
+- The **analysis capability** adds a sandboxed Python interpreter with the
+  documents mounted as a filesystem, for questions that need computation across
+  documents rather than retrieval.
+
+Two optional capabilities compose with them: evidence compaction replaces older
+turns' evidence with what was actually cited, and citation policy requires every
+answer to declare what grounds it.
+
+The same database is reachable from [Python](python.md), the [CLI](cli.md), and
+the [MCP server](mcp.md).
+
+## Running it
+
+A laptop needs nothing but the package and Ollama. Production adds the
+[`haiku-ingester`](ingester.md) service, which polls its sources, queues work in
+SQLite or Postgres, and retries with a circuit breaker per source.
+
+Before deploying, read the operational constraints in
+[Storage](configuration/storage.md): one writer per database, `haiku-rag migrate`
+after an upgrade that changes the schema, and a fixed embedding dimension per
+database.

docs/python.md

@@ -8,12 +8,33 @@ Use `haiku.rag` directly in your Python applications.
 from pathlib import Path
 from haiku.rag.client import HaikuRAG
 
-# Use as async context manager (recommended)
-async with HaikuRAG("Path(path/to/database.lancedb")) as client:
+# Create a new database
+async with HaikuRAG("path/to/database.lancedb", create=True) as client:
     # Your code here
     pass
+
+# Open an existing database (will fail if database doesn't exist)
+async with HaikuRAG("path/to/database.lancedb") as client:
+    # Your code here
+    pass
+
+# Open in read-only mode (blocks writes)
+async with HaikuRAG("path/to/database.lancedb", read_only=True) as client:
+    results = await client.search("query")  # Read operations work
+    # await client.create_document(...)  # Would raise ReadOnlyError
 ```
 
+`async with` is the lifecycle. A caller that owns the client some other way releases it with `await client.aclose()`, which does the same work for every client shape. `client.close()` closes the connection to one database and nothing else, since draining the background vacuum and releasing the embedder and reranker are awaitable; it refuses a client covering several.
+
+!!! note
+    Databases must be explicitly created with `create=True` or via `haiku-rag init` before use. Opening a nonexistent local database given as a path raises `FileNotFoundError`, naming the path; a configured or default database raises `SourceUnavailableError`, which names the database rather than its location. A path beside a configured `lancedb.databases` raises `AmbiguousDatabaseError`.
+
+!!! note
+    Read-only mode is useful for safely accessing databases without risk of modification. It blocks all write operations and downgrades an embedding provider/name mismatch to a warning instead of raising `ConfigMismatchError`.
+
+!!! warning "Database Migrations"
+    When upgrading haiku.rag to a version with schema changes, opening an existing database will raise `MigrationRequiredError`. Run `haiku-rag migrate` to apply pending migrations before using the database. See [CLI Database Management](cli.md#migrate-database) for details.
+
 ## Document Management
 
 ### Creating Documents
@@ -28,31 +49,25 @@ doc = await client.create_document(
 )
 ```
 
-With custom externally generated chunks:
+From HTML content (preserves document structure):
 ```python
-from haiku.rag.store.models.chunk import Chunk
-
-# Create custom chunks with optional embeddings
-chunks = [
-    Chunk(
-        content="This is the first chunk",
-        metadata={"section": "intro"}
-    ),
-    Chunk(
-        content="This is the second chunk",
-        metadata={"section": "body"},
-        embedding=[0.1] * 1024  # Optional pre-computed embedding
-    ),
-]
-
+html_content = "<h1>Title</h1><p>Paragraph</p><ul><li>Item 1</li></ul>"
 doc = await client.create_document(
-    content="Full document content",
-    uri="doc://custom",
-    metadata={"source": "manual"},
-    chunks=chunks  # Use provided chunks instead of auto-generating
+    content=html_content,
+    uri="doc://html-example",
+    format="html"  # parse as HTML instead of markdown
 )
 ```
 
+The `format` parameter controls how text content is parsed:
+
+- `"md"` (default) - Parse as Markdown
+- `"html"` - Parse as HTML, preserving semantic structure (headings, lists, tables)
+- `"plain"` - Plain text, no parsing (creates a simple text document)
+
+!!! note
+    The document's `content` field stores the markdown export of the parsed document for consistent display. The original DoclingDocument structure is preserved in the `docling_document` field (zstd-compressed, without page images). Page images are stored separately in `docling_pages`.
+
 From file:
 ```python
 doc = await client.create_document_from_source(
@@ -67,11 +82,15 @@ doc = await client.create_document_from_source(
 )
 ```
 
+PDFs that carry attachments via the `/EmbeddedFiles` table are split into one Document per attachment, linked to the wrapper through `metadata.parent_uri`. See [PDF Embedded Attachments](configuration/processing.md#pdf-embedded-attachments).
+
 ### Retrieving Documents
 
 By ID:
 ```python
-doc = await client.get_document_by_id(1)
+doc = await client.get_document_by_id("document-id-string")
+doc = await client.get_document_by_id("document-id-string", "papers")
+chunk = await client.get_chunk_by_id("chunk-id-string", "papers")
 ```
 
 By URI:
@@ -79,9 +98,21 @@ By URI:
 doc = await client.get_document_by_uri("file:///path/to/document.pdf")
 ```
 
+Both return content, uri, title and metadata. The multi-MB docling blobs are
+loaded separately:
+
+```python
+docling = await client.document_repository.get_docling_data(doc.id)
+pages = await client.document_repository.get_pages_data(doc.id)
+```
+
 List all documents:
 ```python
 docs = await client.list_documents(limit=10, offset=0)
+
+# Include the text content (not loaded by default). A listing never loads the
+# docling blobs.
+docs = await client.list_documents(include_content=True)
 ```
 
 Filter documents by properties:
@@ -99,43 +130,62 @@ docs = await client.list_documents(
 )
 ```
 
+Count documents:
+```python
+# Count all documents
+total = await client.count_documents()
+
+# Count with filter
+pdf_count = await client.count_documents(filter="uri LIKE '%.pdf'")
+```
+
 ### Updating Documents
 
 ```python
-doc.content = "Updated content"
-await client.update_document(doc)
+# Update content (triggers re-chunking)
+await client.update_document(document_id=doc.id, content="New content")
+
+# Update metadata only (no re-chunking)
+await client.update_document(
+    document_id=doc.id,
+    metadata={"version": "2.0", "updated_by": "admin"}
+)
+
+# Update title only (no re-chunking)
+await client.update_document(document_id=doc.id, title="New Title")
+
+# Update uri only (no re-chunking)
+await client.update_document(document_id=doc.id, uri="file:///new/path.txt")
+
+# Update multiple fields at once
+await client.update_document(
+    document_id=doc.id,
+    content="New content",
+    title="Updated Title",
+    metadata={"status": "final"}
+)
+
+# Use custom chunks (embeddings optional - will be generated if missing)
+custom_chunks = [
+    Chunk(content="Custom chunk 1"),
+    Chunk(content="Custom chunk 2", embedding=[...]),  # Pre-computed embedding
+]
+await client.update_document(document_id=doc.id, chunks=custom_chunks)
 ```
 
+**Notes:**
+
+- Updates to only `metadata` or `title` skip re-chunking
+- Updates to `content` trigger re-chunking and re-embedding
+- Custom `chunks` with embeddings are stored as-is. Missing embeddings are generated automatically
+
 ### Deleting Documents
 
 ```python
 await client.delete_document(doc.id)
 ```
 
-### Rebuilding the Database
-
-```python
-async for doc_id in client.rebuild_database():
-    print(f"Processed document {doc_id}")
-```
-
-## Maintenance
-
-Run maintenance to optimize storage and prune old table versions:
-
-```python
-await client.vacuum()
-```
-
-This compacts tables and removes historical versions to keep disk usage in check. It’s safe to run anytime, for example after bulk imports or periodically in long‑running apps.
-
-### Atomic Writes and Rollback
-
-Document create and update operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores both the `documents` and `chunks` tables to their pre‑operation state using LanceDB’s table versioning.
-
-- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, and internal rebuild/update flows.
-- Scope: Both document rows and all associated chunks are rolled back together.
-- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency; rollbacks occur immediately during the failing operation and are not impacted.
+Deleting a document also removes any child Documents linked to it via `metadata.parent_uri` (PDF attachment children, primarily). The cascade is transitive.
 
 ## Searching Documents
 
@@ -144,12 +194,14 @@ The search method performs native hybrid search (vector + full-text) using Lance
 Basic hybrid search (default):
 ```python
 results = await client.search("machine learning algorithms", limit=5)
-for chunk, score in results:
-    print(f"Score: {score:.3f}")
-    print(f"Content: {chunk.content}")
-    print(f"Document ID: {chunk.document_id}")
+for result in results:
+    print(f"Score: {result.score:.3f}")
+    print(f"Content: {result.content}")
+    print(f"Document ID: {result.document_id}")
 ```
 
+Each result carries the parent document's metadata in `result.document_meta` and the relevant chunk's verbatim metadata in `result.chunk_meta`. Neither is shown to the model during QA.
+
 Search with different search types:
 ```python
 # Vector search only
@@ -174,15 +226,68 @@ results = await client.search(
 )
 
 # Process results
-for chunk, relevance_score in results:
-    print(f"Relevance: {relevance_score:.3f}")
-    print(f"Content: {chunk.content}")
-    print(f"From document: {chunk.document_id}")
-    print(f"Document URI: {chunk.document_uri}")
-    print(f"Document Title: {chunk.document_title}")  # when available
-    print(f"Document metadata: {chunk.document_meta}")
+for result in results:
+    print(f"Relevance: {result.score:.3f}")
+    print(f"Content: {result.content}")
+    print(f"From document: {result.document_id}")
+    print(f"Document URI: {result.document_uri}")
+    print(f"Document Title: {result.document_title}")  # when available
 ```
 
+### Searching Multiple Databases
+
+With [`lancedb.databases`](configuration/storage.md#multiple-databases) configured, a client covers the full set. Use `sources` to select a subset. Each result includes its database name:
+
+```python
+results = await client.search("machine learning")                    # all of them
+results = await client.search("machine learning", sources=["papers"])  # one of them
+
+for result in results:
+    print(f"{result.source}: {result.content}")
+```
+
+`ask` and `analyze` also accept `sources`. Citations include the database name:
+
+```python
+answer, citations = await client.ask("What changed?", sources=["papers", "wiki"])
+for cite in citations:
+    print(f"[{cite.source}] {cite.document_title or cite.document_uri}")
+
+result = await client.analyze("How many documents mention it?", sources=["papers"])
+```
+
+A scoped question can cite only the selected databases. Analysis mounts only their documents.
+
+`sources=None` covers every database the client covers. `sources=[]` covers none: `search` returns no results, and `ask` and `analyze` run with no evidence from any database.
+
+A name no client covers raises `UnknownDatabaseError`, a `KeyError`, wherever it is given: at construction, per query, and when placing a citation.
+
+On the constructor `sources=[]` means something else. Passing `sources` alongside a database path raises `AmbiguousDatabaseError` immediately, since both say which database to open. Passing `sources=[]` alone raises `ValueError` on entering the client: a selection of nothing to search is a legitimate question, a client over no database is not.
+
+#### Inspecting the client scope
+
+```python
+client.covers_multiple      # whether the client covers more than one database
+client.source_names         # database names, in order; known before the client opens
+client.source               # the one database's name, or None for a set
+
+owner = await client.reader_for("papers")      # the client reading that database
+papers, wiki = await client.clients_for(["papers", "wiki"])
+```
+
+`reader_for` and `clients_for` open databases lazily and return borrowed clients. They remain valid while the covering client is open and inherit its read-only mode. The covering client owns and closes their database sessions.
+
+To learn what a configuration covers without opening anything, resolve it:
+
+```python
+from haiku.rag.client import DatabaseScope
+
+for ref in DatabaseScope.resolve(config).databases:
+    print(ref.name, ref.location)   # "haiku.rag", Path(".../haiku.rag.lancedb") when nothing is configured
+```
+
+`DatabaseScope.resolve` is pure: it reads the configuration and classifies each location as a local path or a URI.
+
 ### Filtering Search Results
 
 Filter search results to only include chunks from documents matching specific criteria:
@@ -225,60 +330,276 @@ results = await client.search(
 - `created_at`, `updated_at` - Timestamps
 - `metadata` - Document metadata (as string, use LIKE for pattern matching)
 
+### Image queries
+
+`client.search()` accepts an image instead of a text query when the configured embedder is multimodal (`embeddings.model.multimodal: true` on a vLLM, VoyageAI, or Cohere model). The image is embedded once and the chunks table is searched vector-only. Full-text search and reranking don't apply without a text query.
+
+```python
+from PIL import Image
+
+# Bytes
+results = await client.search(
+    open("figure.png", "rb").read(),
+    limit=5,
+)
+
+# PIL.Image works equivalently
+results = await client.search(
+    Image.open("figure.png"),
+    limit=5,
+)
+```
+
+Image queries surface picture chunks (synthetic per-figure chunks emitted at ingest under a multimodal embedder) and any text chunks whose vectors land near the image vector in the shared embedding space. Calling `client.search(bytes)` against a text-only embedder raises a `ValueError`.
+
 ### Expanding Search Context
 
-Expand search results with adjacent chunks for more complete context:
+Expand search results with surrounding content from the document:
 
 ```python
 # Get initial search results
 search_results = await client.search("machine learning", limit=3)
 
-# Expand with adjacent chunks using config setting
+# Expand with section-bounded context
 expanded_results = await client.expand_context(search_results)
 
-# Or specify a custom radius
-expanded_results = await client.expand_context(search_results, radius=2)
-
-# The expanded results contain chunks with combined content from adjacent chunks
-for chunk, score in expanded_results:
-    print(f"Expanded content: {chunk.content}")  # Now includes before/after chunks
+for result in expanded_results:
+    print(f"Expanded content: {result.content}")
 ```
 
-**Smart Merging**: When expanded chunks overlap or are adjacent within the same document, they are automatically merged into single chunks with continuous content. This eliminates duplication and provides coherent text blocks. The merged chunk uses the highest relevance score from the original chunks.
+Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers). This naturally crosses into adjacent sections until the budget is filled. Picture and table matches are exempt: they return their enclosing section as-is and never cross section boundaries. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded.
 
-This is automatically used by the QA system when `processing.context_chunk_radius > 0` (configured in `haiku.rag.yaml`) to provide better answers with more complete context.
+Configuration:
+
+- **search.max_context_chars**: Maximum characters in expanded context. Default: 5000.
+
+**Smart Merging**: When expanded results overlap within the same document, they are automatically merged into a single result with continuous content and the highest relevance score.
 
 ## Question Answering
 
 Ask questions about your documents:
 
 ```python
-answer = await client.ask("Who is the author of haiku.rag?")
+answer, citations = await client.ask("Who is the author of haiku.rag?")
 print(answer)
+for cite in citations:
+    print(f"  [{cite.chunk_id}] {cite.document_title or cite.document_uri}")
 ```
 
-Ask questions with citations showing source documents:
+Filter to specific documents:
 
 ```python
-answer = await client.ask("Who is the author of haiku.rag?", cite=True)
-print(answer)
-```
-
-Customize the QA agent's behavior with a custom system prompt:
-
-```python
-custom_prompt = """You are a technical support expert for WIX.
-Answer questions based on the knowledge base documents provided.
-Be concise and helpful."""
-
-answer = await client.ask(
-    "How do I create a blog?",
-    system_prompt=custom_prompt
+answer, citations = await client.ask(
+    "What are the main findings?",
+    filter="uri LIKE '%paper%'"
 )
 ```
 
-The QA agent will search your documents for relevant information and use the configured LLM to generate a comprehensive answer. With `cite=True`, responses include citations showing which documents were used as sources. Citations prefer the document title when present, otherwise they use the URI.
+Attach images to the question, for example to check an image against indexed documents:
 
-The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](configuration.md)).
+```python
+answer, citations = await client.ask(
+    "Does this image satisfy the requirements in the design spec?",
+    images=[Path("photo.jpg").read_bytes()],
+)
+```
 
-See also: [Agents](agents.md) for details on the QA agent and the multi‑agent research workflow.
+Images are passed to the model alongside the question. Retrieval stays text-based. The QA model must have `vision: true` in its configuration.
+
+`client.ask` runs the [RAG capability](capabilities/rag.md) and returns `(answer_text, list[Citation])`. Citations include page numbers, section headings, document references, the document's metadata (`document_meta`), and the cited chunk's raw, unparsed metadata (`chunk_meta`), so UIs can render metadata keys such as a public source URL alongside the citation.
+
+The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](configuration/index.md)).
+
+See also: [Capabilities](capabilities/index.md) for direct agent composition.
+
+## Analysis
+
+Answer complex analytical questions via code execution:
+
+```python
+# Aggregation across documents
+result = await client.analyze("Which quarter had the highest revenue?")
+print(result.answer)
+for citation in result.citations:
+    print(citation.uri, citation.title)
+
+# Computation within a document set
+result = await client.analyze(
+    "What is the average deal size mentioned in these contracts?",
+    filter="uri LIKE '%contracts%'"
+)
+```
+
+`client.analyze` runs the [analysis capability](capabilities/analysis.md), which writes and executes Python code in a sandboxed environment to solve problems that traditional RAG struggles with: aggregation, computation, and multi-document analysis.
+
+`client.analyze` also accepts `images=` like `client.ask`, requiring `vision: true` on the analysis model (or the QA model when no analysis model is configured).
+
+See [Analysis capability](capabilities/analysis.md) for details and configuration.
+
+## Building custom agents
+
+`client.ask` and `client.analyze` are convenience wrappers. To build your own Pydantic AI agent, attach the native RAG and analysis capabilities directly. See [Capabilities](capabilities/index.md).
+
+For the low-level toolset factories under `haiku.rag.tools` (one rung below the capability abstraction), see [Toolsets](tools.md).
+
+## Importing Pre-Processed Documents
+
+If you process documents externally or need custom processing, use `import_document()`:
+
+```python
+from haiku.rag.store.models.chunk import Chunk
+
+# Convert your source to a DoclingDocument
+docling_doc = await client.convert("path/to/document.pdf")
+
+# Create chunks (embeddings optional - will be generated if missing)
+chunks = [
+    Chunk(
+        content="This is the first chunk",
+        metadata={"section": "intro"},
+        order=0,
+    ),
+    Chunk(
+        content="This is the second chunk",
+        metadata={"section": "body"},
+        embedding=[0.1] * 1024,  # Optional: pre-computed embedding
+        order=1,
+    ),
+]
+
+# Import document with custom chunks
+doc = await client.import_document(
+    docling_document=docling_doc,
+    chunks=chunks,
+    uri="doc://custom",
+    title="Custom Document",
+    metadata={"source": "external-pipeline"},
+)
+```
+
+The `docling_document` provides rich metadata for visual grounding, page numbers, and section headings. Content is automatically extracted from the DoclingDocument.
+
+### Batch Import
+
+Each `create_document*` / `import_document` call writes new versions of the `documents`, `document_meta`, `chunks`, and `document_items` tables. Ingesting many documents in a loop therefore creates a table version per document. Use `import_documents()` to write the whole batch in a single version per table:
+
+```python
+from haiku.rag.client import DocumentImport
+
+imports = []
+for path in paths:  # paths: list[Path]
+    docling_doc = await client.convert(path)
+    chunks = await client.chunk(docling_doc)
+    imports.append(
+        DocumentImport(
+            docling_document=docling_doc,
+            chunks=chunks,
+            uri=path.absolute().as_uri(),
+            metadata={"source": "external-pipeline"},
+        )
+    )
+
+docs = await client.import_documents(imports)
+```
+
+Chunks without embeddings are embedded automatically. The import is all-or-nothing: if any document fails, all tables are restored to their pre-batch state.
+
+See [Custom Processing Pipelines](custom-pipelines.md) for building pipelines with `convert()`, `chunk()`, and `embed_chunks()`.
+
+## Maintenance
+
+Run maintenance to optimize storage and prune old table versions:
+
+```python
+await client.vacuum()
+```
+
+This compacts tables and removes historical versions to keep disk usage in check. It’s safe to run anytime, for example after bulk imports or periodically in long‑running apps.
+
+### Tags
+
+Tag the current database state and restore it later, for example after an ingestion run. A tag covers all five tables and is created from a single version snapshot. Create tags with other writers stopped: the snapshot is coordinated within one process only, and a writer in another process can commit between the per-table reads.
+
+```python
+await client.store.create_tag("release-1")
+
+tags = await client.store.list_tags()
+for name, info in tags.items():
+    print(name, info.tables, info.complete)
+```
+
+`restore_tag` brings the live database back to a tagged state. It creates a complete safety tag for the current state before changing any table and returns its name:
+
+```python
+safety_tag = await client.store.restore_tag("release-1")
+```
+
+Restore is a maintenance operation: stop all other writers first. A tag present on only some tables is partial; `list_tags` reports it via `missing_tables`, and partial tags can be deleted but never restored.
+
+Delete tags you no longer need. Vacuum retains the oldest tagged version and everything newer:
+
+```python
+await client.store.delete_tag("release-1")
+```
+
+### Rebuilding the Database
+
+```python
+from haiku.rag.client import RebuildMode
+
+# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
+async for doc_id in client.rebuild_database():
+    print(f"Processed document {doc_id}")
+
+# Re-chunk from stored content (no source file access)
+async for doc_id in client.rebuild_database(mode=RebuildMode.RECHUNK):
+    print(f"Processed document {doc_id}")
+
+# Only regenerate embeddings (fastest, keeps existing chunks)
+async for doc_id in client.rebuild_database(mode=RebuildMode.EMBED_ONLY):
+    print(f"Processed document {doc_id}")
+
+# Add VLM picture descriptions to an existing database. Runs the VLM
+# over already-stored picture bytes, patches descriptions into the
+# docling blob, then re-chunks + re-embeds. Requires
+# processing.pictures='description' in the config.
+async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
+    print(f"Described pictures in {doc_id}")
+```
+
+**Rebuild modes:**
+
+- `RebuildMode.FULL` - Re-convert from source files, re-chunk, re-embed (default)
+- `RebuildMode.RECHUNK` - Re-chunk from existing document content, re-embed
+- `RebuildMode.EMBED_ONLY` - Keep existing chunks, only regenerate embeddings
+- `RebuildMode.TITLE_ONLY` - Generate titles for untitled documents (no re-chunking or re-embedding)
+- `RebuildMode.DESCRIPTIONS` - Run the VLM over picture bytes already stored on `document_items.picture_data`, patch descriptions into the docling blob, re-chunk + re-embed. Skips the docling parse entirely. Idempotent: pictures already carrying `meta.description.text` are not re-described, so the operation is safe to re-run.
+
+### Generating Titles
+
+Generate a title for an existing document on demand:
+
+```python
+title = await client.generate_title(doc)
+if title:
+    await client.update_document(document_id=doc.id, title=title)
+```
+
+Uses the same two-tier approach as automatic ingestion: structural extraction from DoclingDocument metadata first, with LLM fallback via `processing.title_model`. Unlike ingestion, this method does not catch exceptions. If the LLM call fails, the error propagates.
+
+To batch-generate titles for all untitled documents, use `RebuildMode.TITLE_ONLY`:
+
+```python
+async for doc_id in client.rebuild_database(mode=RebuildMode.TITLE_ONLY):
+    print(f"Generated title for {doc_id}")
+```
+
+See [Automatic Title Generation](configuration/processing.md#automatic-title-generation) for configuration details.
+
+### Atomic Writes and Rollback
+
+Document create, update, and delete operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores the `documents`, `document_meta`, `chunks`, and `document_items` tables to their pre‑operation state using LanceDB’s table versioning. These writes are serialized under a single lock, so the rollback is safe under concurrent ingester workers.
+
+- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, `delete_document(...)` (including the `parent_uri` cascade), and internal rebuild/update flows.
+- Scope: Document rows, their mutable attributes, and all associated chunks and items are rolled back together.
+- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency. Rollbacks occur immediately during the failing operation and are not impacted.

docs/remote-processing.md

@@ -0,0 +1,130 @@
+# Remote Processing
+
+`haiku.rag` can use [docling-serve](https://github.com/docling-project/docling-serve) for remote document processing and chunking, offloading resource-intensive operations to a dedicated service.
+
+## Overview
+
+docling-serve is a REST API service that provides:
+
+- Document conversion (PDF, DOCX, PPTX, images, etc.)
+- Intelligent chunking with structure preservation
+- OCR capabilities for scanned documents
+- Table and figure extraction
+
+## When to Use docling-serve
+
+**Use local processing (default) when:**
+
+- Working with small to medium document volumes
+- Running on development machines
+- Want zero external dependencies
+- Processing simple document formats
+
+**Use docling-serve when:**
+
+- Processing large volumes of documents
+- Working with complex PDFs requiring OCR
+- Running in production environments
+- Separating compute-intensive tasks
+- Scaling document processing independently
+
+## Setup
+
+haiku.rag is tested against docling-serve 1.25.0.
+
+### Docker Compose (Recommended)
+
+The slim Docker image with docker-compose is the recommended setup. See `examples/docker/docker-compose.yml` for a complete configuration that includes both services.
+
+### Running docling-serve Manually
+
+See the [official docling-serve repository](https://github.com/docling-project/docling-serve) for installation options. The quickest way is using Docker:
+
+```bash
+docker run -p 5001:5001 quay.io/docling-project/docling-serve
+```
+
+To enable the web UI for debugging:
+
+```bash
+docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=true quay.io/docling-project/docling-serve
+```
+
+### Configuration
+
+Configure haiku.rag to use docling-serve. See the [Document Processing](configuration/processing.md) guide for all available options.
+
+```yaml
+# haiku.rag.yaml
+processing:
+  converter: docling-serve  # Use remote conversion
+  chunker: docling-serve    # Use remote chunking
+
+providers:
+  docling_serve:
+    base_url: http://localhost:5001
+    api_key: ""  # Optional API key for authentication
+    timeout: 300  # Per-request timeout in seconds
+```
+
+For converter / chunker config options (chunking strategy, tokenizer,
+OCR, table handling, picture description), see
+[Document Processing](configuration/processing.md). The configuration is
+identical between `docling-local` and `docling-serve` modes — this page
+covers only what's specific to running docling-serve as a separate
+service.
+
+## VLM picture description with docling-serve
+
+When `processing.pictures = "description"` and `converter: docling-serve`,
+the VLM API calls are made by the docling-serve container, not by
+haiku.rag. Two deployment caveats:
+
+### Enable remote services
+
+docling-serve blocks outbound calls by default. Enable them by setting
+`DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true` on the container:
+
+```bash
+docker run -p 5001:5001 \
+  -e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \
+  quay.io/docling-project/docling-serve
+```
+
+### Reach host services from inside the container
+
+If your VLM (e.g. Ollama) runs on the host while docling-serve runs in
+Docker, set the VLM's `base_url` in
+`processing.conversion_options.picture_description.model` to
+`http://host.docker.internal:11434` rather than `localhost`. See
+[Document Processing → Picture Handling](configuration/processing.md#picture-handling)
+for the full config snippet.
+
+## Operational notes
+
+Long-running docling-serve containers see CPU memory grow monotonically
+([docling-serve #366](https://github.com/docling-project/docling-serve/issues/366),
+[#474](https://github.com/docling-project/docling-serve/issues/474)). The
+underlying parser leaks are in core docling
+([#2209](https://github.com/docling-project/docling/issues/2209),
+[#1343](https://github.com/docling-project/docling/issues/1343)) and affect
+docling-local too.
+
+Recommended deployment shape:
+
+- Set `mem_limit` on the docling-serve container (or `resources.limits.memory`
+  in Kubernetes) at a value comfortably above your largest expected job.
+- Combine with `restart: unless-stopped` so the runtime restarts when the
+  kernel OOM-kills.
+- Run multiple docling-serve replicas behind haiku.rag's round-robin
+  `providers.docling_serve.base_url` list (see
+  [Document Processing](configuration/processing.md)). A restart of one
+  replica doesn't stop ingest.
+- In haiku.rag, set `processing.split_pages` for large-PDF workloads so each
+  slice is an independent docling-serve task and the per-task working set
+  stays bounded.
+
+## Resources
+
+- [docling-serve GitHub](https://github.com/docling-project/docling-serve)
+- [docling-serve Documentation](https://github.com/docling-project/docling-serve#readme)

docs/server.md

@@ -1,95 +0,0 @@
-# Server Mode
-
-The server provides automatic file monitoring and MCP functionality.
-
-## Starting the Server
-
-The `serve` command requires at least one service flag. You can enable file monitoring, MCP server, or both:
-
-### MCP Server Only
-
-```bash
-haiku-rag serve --mcp
-```
-
-Transport options:
-- Default - Streamable HTTP transport on port 8001
-- `--stdio` - Standard input/output transport
-- `--mcp-port` - Custom port (default: 8001)
-
-### File Monitoring Only
-
-```bash
-haiku-rag serve --monitor
-```
-
-### Both Services
-
-```bash
-haiku-rag serve --monitor --mcp
-```
-
-This will start file monitoring and MCP server on port 8001.
-
-## File Monitoring
-
-Configure directories to monitor in your `haiku.rag.yaml`:
-
-```yaml
-monitor:
-  directories:
-    - /path/to/documents
-    - /another/path
-```
-
-Then start the server:
-
-```bash
-haiku-rag serve --monitor
-```
-
-### Monitoring Features
-
-- **Startup**: Scans all monitored directories and adds new files
-- **File Added/Modified**: Automatically parses and updates documents
-- **File Deleted**: Removes corresponding documents from database
-
-### Filtering Files
-
-You can filter which files to monitor using gitignore-style patterns:
-
-```yaml
-monitor:
-  directories:
-    - /path/to/documents
-
-  # Ignore patterns (exclude files)
-  ignore_patterns:
-    - "*draft*"         # Ignore draft files
-    - "temp/"           # Ignore temp directory
-    - "**/archive/**"   # Ignore archive directories
-
-  # Include patterns (whitelist files)
-  include_patterns:
-    - "*.md"           # Only markdown files
-    - "**/docs/**"     # Files in docs directories
-```
-
-**Pattern behavior:**
-- Extension filtering is applied first (only supported file types)
-- Include patterns create a whitelist (if specified)
-- Ignore patterns exclude files
-- Both can be combined for fine-grained control
-
-### Supported Formats
-
-The server can parse 40+ file formats including:
-- PDF documents
-- Microsoft Office (DOCX, XLSX, PPTX)
-- HTML and Markdown
-- Plain text files
-- Code files (Python, JavaScript, etc.)
-- Images (processed via OCR)
-- And more...
-
-URLs are also supported for web content.

docs/stylesheets/extra.css

@@ -0,0 +1,53 @@
+.haiku-rag-hero {
+  padding: 2rem 0 2.5rem;
+}
+
+.haiku-rag-hero__inner {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  gap: 2.5rem;
+  align-items: center;
+}
+
+@media (max-width: 60em) {
+  .haiku-rag-hero__inner {
+    grid-template-columns: 1fr;
+    gap: 1.5rem;
+  }
+}
+
+.haiku-rag-hero__title {
+  font-family: var(--md-code-font, monospace);
+  font-size: 4.5rem;
+  font-weight: 700;
+  line-height: 1;
+  margin: 0 0 0.75rem;
+  color: var(--md-primary-fg-color);
+  letter-spacing: -0.02em;
+}
+
+.haiku-rag-hero__tagline {
+  font-size: 1.15rem;
+  line-height: 1.5;
+  color: var(--md-default-fg-color--light);
+  margin: 0 0 1.5rem;
+}
+
+.haiku-rag-hero__actions {
+  display: flex;
+  gap: 0.75rem;
+  flex-wrap: wrap;
+}
+
+.haiku-rag-hero__media img,
+.haiku-rag-hero__media video {
+  display: block;
+  width: 100%;
+  height: auto;
+  border-radius: 0.5rem;
+  box-shadow: 0 8px 32px rgba(0, 0, 0, 0.25);
+}
+
+body:has(.haiku-rag-hero) .md-main {
+  display: none;
+}

docs/tools.md

@@ -0,0 +1,66 @@
+# Toolsets
+
+For agent integrations, use the native Pydantic AI [capabilities](capabilities/index.md). This page documents the lower-level toolsets used by other haiku.rag surfaces.
+
+For lower-level access, `haiku.rag.tools` provides individual `FunctionToolset` factories used across haiku.rag.
+
+## Low-Level Toolsets
+
+For advanced use cases, individual toolset factories are available in `haiku.rag.tools` and can be reused to build custom agents.
+
+### RAGDeps Protocol
+
+All toolsets use the `RAGDeps` protocol for dependency injection:
+
+```python
+from haiku.rag.tools import RAGDeps
+
+class MyDeps:
+    def __init__(self, client: HaikuRAG):
+        self.client = client
+```
+
+### Search Toolset
+
+`create_search_toolset()` provides hybrid search with context expansion.
+
+```python
+from haiku.rag.tools import create_search_toolset
+
+search = create_search_toolset(config)
+```
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `config` | required | `AppConfig` |
+| `expand_context` | `True` | Expand results with surrounding chunks |
+| `base_filter` | `None` | SQL WHERE clause applied to all searches |
+| `tool_name` | `"search"` | Name of the tool exposed to the agent |
+| `on_results` | `None` | Callback `(list[SearchResult]) -> None` invoked with results |
+
+### Document Toolset
+
+`create_document_toolset()` provides document browsing and retrieval.
+
+```python
+from haiku.rag.tools import create_document_toolset
+
+docs = create_document_toolset(config)
+```
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `config` | required | `AppConfig` |
+| `base_filter` | `None` | SQL WHERE clause for list operations |
+
+**Tools:**
+
+- `list_documents(page?)` — Paginated document listing (50 per page).
+- `get_document(query)` — Retrieve a document by title or URI.
+- `summarize_document(query)` — Generate an LLM summary of a document's content.
+
+## Filter Helpers
+
+`haiku.rag.tools.filters` provides utilities for building SQL filters:
+
+- **`build_multi_document_filter(document_names)`** — Combines multiple document name filters with OR logic. Matches against both `uri` and `title`, case-insensitive.

docs/tuning.md

@@ -0,0 +1,124 @@
+# Tuning
+
+How to adjust haiku.rag's pipeline for better retrieval and answer quality. For individual setting definitions and defaults, see [Configuration](configuration/index.md).
+
+For ingester-side tuning (worker count, lease TTL and heartbeat, retry policy, backpressure, circuit breakers), see [Ingester → Workers and retry](ingester.md#workers-and-retry).
+
+## Pipeline Overview
+
+Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is the highest-leverage stage. If the LLM never sees the right chunks, no prompt or model change will help.
+
+## Tuning Retrieval
+
+### Chunking
+
+`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each. Larger chunks provide more surrounding information but dilute relevance signals. See [Processing](configuration/processing.md#chunk-size) for configuration.
+
+`chunker_type` selects between `hybrid` (default) and `hierarchical` chunking. Hierarchical chunking preserves the document's heading structure and works better for deeply nested or structured content. See [Chunking Strategies](configuration/processing.md#chunking-strategies).
+
+### Embedding Model
+
+Larger embedding models produce better representations at the cost of slower indexing and more storage. The choice of embedding model has a larger impact on retrieval quality than most other settings. See [Providers](configuration/providers.md) for available options and [Benchmarks](benchmarks.md) for real comparisons across models.
+
+### Reranking
+
+When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision. See [Search Settings](configuration/qa.md#search-settings) for how reranking integrates with search.
+
+### Search Settings
+
+`limit` controls how many results reach the LLM. More candidates improve recall but increase token usage. See [Search Settings](configuration/qa.md#search-settings).
+
+Context expansion is automatic and section-aware. Search results are expanded to include surrounding content from the same document section. For structured documents, expansion stays within section boundaries and filters noise (footnotes, page headers). For unstructured documents, expansion grows outward until the character budget is filled. `max_context_chars` caps expansion to prevent context bloat.
+
+## Tuning Generation
+
+Model and temperature selection affect answer quality directly. See [Providers](configuration/providers.md#model-settings) for options.
+
+`domain_preamble` prepends domain context to the RAG and analysis capability instructions. Use it to describe what the knowledge base contains and clarify domain-specific terminology. See [Prompt Customization](configuration/prompts.md).
+
+## What Requires a Rebuild
+
+| Change | Rebuild required? |
+|--------|:-:|
+| `chunk_size`, `chunker_type`, `chunking_merge_peers` | Yes (run `haiku-rag rebuild`) |
+| Embedding model | Yes (run `haiku-rag rebuild`) |
+| Search settings, reranking, prompts | No |
+
+## Inspector
+
+The inspector shows what your model would actually receive for a given query. Run it against your database and step through the same hybrid search, context expansion, and chunk previews the RAG capability uses at runtime. Press `c` on a chunk and you see the exact context the LLM would get back from a search hit.
+
+```bash
+haiku-rag inspect
+haiku-rag inspect --db /path/to/database.lancedb
+```
+
+!!! note
+    Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in the full `haiku.rag` package).
+
+### Layout
+
+Three panels:
+
+- **Documents** (left): every document in the database.
+- **Chunks** (top right): chunks for the selected document.
+- **Detail view** (bottom right): full content and metadata.
+
+![Inspector search](img/inspector-search.svg)
+
+### Keys
+
+| Key | Action |
+|-----|--------|
+| `Tab` | Cycle panels |
+| `↑` / `↓` | Navigate lists |
+| `/` | Search modal |
+| `c` | Context expansion modal (the chunk plus what the agent would see around it) |
+| `v` | Visual grounding modal (chunk highlighted on the page) |
+| `q` | Quit |
+
+Mouse: click to select, scroll to view content.
+
+### Search
+
+Press `/` to open the search modal. Type a query and press `Enter`. The left panel lists results with relevance scores like `[0.95] content preview`. The right panel shows the full chunk and its metadata. `↑` / `↓` navigates results, `Enter` jumps to the document and chunk, `Esc` closes the modal. Search uses the same hybrid (vector + full-text) retrieval the RAG capability uses.
+
+### Context expansion (`c`)
+
+Press `c` on a chunk to see the expanded context that would be fed to the RAG capability. This is where you find out whether your `chunk_size`, `chunker_type`, and `max_context_chars` settings actually deliver the surrounding content the model needs. The modal shows:
+
+- The expanded text. Section-aware expansion stays within section boundaries on structured documents and fills `max_context_chars` outward on unstructured ones.
+- Source document, content type, and relevance score.
+- Filtered noise. Footnotes, page headers and footers are excluded from structured documents.
+
+If `qa.model.vision = true` is set, the modal also renders the picture bytes attached to that chunk, so you see exactly what the vision model would receive.
+
+### Visual grounding (`v`)
+
+Press `v` to highlight the chunk's bounding box on its page image. Useful for verifying chunk boundaries and seeing how Docling carved up the document.
+
+- `←` / `→` to navigate pages when a chunk spans multiple pages.
+- `Esc` closes the modal.
+
+![Visual grounding modal](img/tui-visual-grounding.png)
+
+Requirements: documents must have page images (default for PDFs), and the terminal must support inline images (iTerm2, WezTerm, Kitty). Plain-text documents added via `haiku-rag add` don't have visual grounding.
+
+You can also visualize a chunk from the CLI without launching the TUI: `haiku-rag visualize <chunk_id>`.
+
+## Measuring Changes
+
+For systematic measurement, use the `evaluations/` workspace which provides retrieval metrics (MRR, MAP) and LLM-judged QA accuracy via `pydantic-evals`:
+
+```bash
+# Run retrieval + QA benchmarks
+evaluations run <dataset>
+
+# Skip database rebuild when only changing search/reranking/prompt settings
+evaluations run <dataset> --skip-db
+
+# Limit test cases for faster iteration
+evaluations run <dataset> --limit 50
+```
+
+See [Benchmarks](benchmarks.md) for dataset details, methodology, and baseline results.

docs/tutorial.md

@@ -1,218 +1,86 @@
-# Tutorial
+# Quickstart
 
-This tutorial provides quickstart instructions for getting familiar with `haiku.rag`. This tutorial is intended for people who are familiar with command line and Python, but not different AI ecosystem tools.
+Install haiku.rag, index a document, and chat with it.
 
-The tutorial covers:
+## Install
 
-- RAG and embeddings basics
-- Installing `haiku.rag` Python package
-- Configuring `haiku.rag` with YAML
-- Adding and retrieving items
-- Inspecting the database
-
-The tutorial uses OpenAI API service - no local installation needed and will work on computers with any amount of RAM and GPU. The OpenAI API is pay-as-you-go, so you need to top it up with at least ~$5 when creating the API key.
-
-## Introduction
-
-Retrieval-Augmented Generation (RAG) lets you give AI models access to your own documents and data. Instead of relying solely on the model's training data, RAG finds relevant information from your documents and includes it in the AI's responses.
-
-`haiku.rag` handles the mechanics: it converts your documents into searchable embeddings, stores them locally, and retrieves relevant chunks when you ask questions. You provide the documents and questions, and it coordinates between the embedding service (like OpenAI) and the AI model to give you accurate, grounded answers.
-
-## Setup
-
-First, [get an OpenAI API key](https://platform.openai.com/api-keys).
-
-Install `haiku.rag` Python package using [uv](https://docs.astral.sh/uv/getting-started/installation/) or your favourite Python package manager:
-
-```shell
-# Python 3.12+ needed
+```bash
 uv pip install haiku.rag
 ```
 
-Configure haiku.rag to use OpenAI. Create a `haiku.rag.yaml` file:
-
-```yaml
-embeddings:
-  provider: openai
-  model: text-embedding-3-small  # or text-embedding-3-large
-  vector_dim: 1536
-
-qa:
-  provider: openai
-  model: gpt-4o-mini  # or gpt-4o, gpt-4, etc.
-```
-
-Set your OpenAI API key as an environment variable (API keys should not be stored in the YAML file):
+You also need [Ollama](https://ollama.com/) for the default embedding and answering models:
 
 ```bash
-export OPENAI_API_KEY="<your OpenAI API key>"
+ollama pull qwen3-embedding:4b
+ollama pull qwen3.8
 ```
 
-For the list of available OpenAI models and their vector dimensions, see the [OpenAI documentation](https://platform.openai.com/docs/guides/embeddings).
+!!! note "Prefer OpenAI?"
+    Drop this into a `haiku.rag.yaml` next to where you'll run the CLI:
 
-See [Configuration](configuration.md) for all available options.
+    ```yaml
+    embeddings:
+      model:
+        provider: openai
+        name: text-embedding-3-small
+        vector_dim: 1536
 
-## Adding the first documents
+    qa:
+      model:
+        provider: openai
+        name: gpt-4o-mini
+    ```
 
-Now you can add some pieces of text in the database:
+    Then `export OPENAI_API_KEY="sk-..."` and continue with the rest of this page. Any provider Pydantic AI supports works the same way. See [Providers](configuration/providers.md).
 
-```shell
-haiku-rag add "Python is the best programming language in the world, because it is flexible, with robust ecosystem, open source licensing and thousands of contributors"
-haiku-rag add "JavaScript is a popular programming language, but has a lot of warts"
-haiku-rag add "PHP is a bad programming language, because of spotted security history, horrible syntax and declining popularity"
+## Initialize
+
+```bash
+haiku-rag init
 ```
 
-What will happen
+This creates a LanceDB database in your platform's user directory. Pass `--db` to any subcommand to use a different path:
 
-- The piece of text is send to OpenAI `/embeddings` API service
-- OpenAI translates the free form text to RAG embedding vectors needed for the retrieval
-- The vector values will be stored in a local database
-
-Now you can view your [LanceDB](https://lancedb.com/) database, and the embeddings it is configured for:
-
-```shell
-haiku-rag info
+```bash
+haiku-rag init --db /tmp/test.lancedb
 ```
 
-You should get the back the information:
+## Add a document
 
-```
-haiku.rag database info
-  path: /Users/moo/Library/Application Support/haiku.rag/haiku.rag.lancedb
-  haiku.rag version (db): 0.13.3
-  embeddings: openai/text-embedding-3-small (dim: 1536)
-  documents: 3
-  versions (documents): 3
-  versions (chunks): 3
-──────────────────────────────────────────────────────────────────────────────────
-Versions
-  haiku.rag: 0.13.3
-  lancedb: 0.25.2
-  docling: 2.58.0
+Add a file, a URL, or a whole folder:
+
+```bash
+haiku-rag add-src https://arxiv.org/pdf/2408.09134
+haiku-rag add-src ~/Documents/papers/
 ```
 
-## Asking questions and retrieving information
+Or paste text inline:
 
-Now we can use OpenAI LLMs to retrieve information from our embeddings database.
-
-In this example, we connect to a remote OpenAI API.
-
-Behind the scenes [pydantic-ai](https://ai.pydantic.dev/) query is created
-using `OpenAIChatModel.request()`.
-
-The easiest way to do this is `ask` CLI command:
-
-```shell
-haiku-rag ask "What is the best programming language in the world"
+```bash
+haiku-rag add "Yiorgis wrote haiku.rag in 2025."
 ```
 
-```
-Question: What is the best programming language in the world
+Each `add-src` call converts the file with Docling, splits it into chunks, embeds them, and writes everything to LanceDB. Run `haiku-rag list` to see what you've added, `haiku-rag info` for a database summary.
 
-Answer:
-According to the document, Python is considered the best programming language in the world due to its flexibility, robust ecosystem, open-source licensing, and thousands of contributors.
+## Chat
+
+```bash
+haiku-rag chat
 ```
 
-## Programmatic interaction in Python
+Ask a question. The agent searches your documents, expands context around the hits, and answers with citations pointing back to the source page and section. Citations are expandable, with visual grounding so you can see the chunk highlighted on the original page. Follow-ups continue within the same session. Start a new session when you switch topics.
 
-You can interact with Haiku RAG from Python in a similar manner as you can from the command line. Here we use Haiku RAG with the interactive Python command prompt (REPL).
+You can also ask a single question directly from the CLI without launching the TUI:
 
-First we need to install `ipython`, as built-in Python REPL does not support async blocks.
-
-```shell
-uv pip install ipython
+```bash
+haiku-rag ask "Who wrote haiku.rag?"
 ```
 
-Run IPython:
+## Where to go next
 
-```shell
-ipython
-```
-
-Then copy paste in the snippet (you can use [%cpaste](https://ipythonbook.com/magic/cpaste.html) command):
-
-```python
-import sys
-import logging
-from haiku.rag.client import HaikuRAG
-
-# Increase logging verbosity so we see what happens behind the scenes,
-# and check that the logger works
-logging.basicConfig(
-  stream=sys.stdout,
-  level=logging.DEBUG,
-  format="%(name)s - %(levelname)s - %(message)s",
-)
-logger = logging.getLogger()
-logger.setLevel(logging.DEBUG)
-logger.debug("AGI here we come")
-
-# Uses LanceDB database from default storage location
-async with HaikuRAG() as client:
-    answer = await client.ask("What is the best programming language in the world?")
-    print(answer)
-
-```
-
-You should see:
-
-```
-2025-10-18 17:05:49,611 - DEBUG - HTTP Response: POST https://api.openai.com/v1/chat/completions "200 OK" Headers({'date': 'Sat, 18 Oct 2025 14:05:49 GMT', 'content-type': 'application/json', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'access-control-expose-headers': 'X-Request-ID', 'openai-organization': 'xxx', 'openai-processing-ms': '788', 'openai-project': 'xxx', 'openai-version': '2020-10-01', 'x-envoy-upstream-service-time': '1050', 'x-ratelimit-limit-requests': '10000', 'x-ratelimit-limit-tokens': '200000', 'x-ratelimit-remaining-requests': '9998', 'x-ratelimit-remaining-tokens': '199603', 'x-ratelimit-reset-requests': '14.981s', 'x-ratelimit-reset-tokens': '119ms', 'x-request-id': 'req_9651a3691a144dd388e97066ad67a49c', 'x-openai-proxy-wasm': 'v0.1', 'cf-cache-status': 'DYNAMIC', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'server': 'cloudflare', 'cf-ray': '990897b6f8d270d7-ARN', 'content-encoding': 'gzip', 'alt-svc': 'h3=":443"; ma=86400'})
-2025-10-18 17:05:49,611 - DEBUG - request_id: req_9651a3691a144dd388e97066ad67a49c
-
-According to the document, Python is considered the best programming language in the world due to its flexibility, robust ecosystem, open-source licensing, and support from thousands of contributors.
-```
-
-## Complex documents
-
-Haiku RAG can also handle types beyond plain text, including PDF, DOCX, HTML, and 40+ other file formats.
-
-Here we add research papers about Python from [arxiv](https://arxiv.org/search/?query=python&searchtype=all&source=header) using URL retriever.
-
-```shell
-# Better Python Programming for all: With the focus on Maintainability
-haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.org/pdf/2408.09134"
-
-# Interoperability From OpenTelemetry to Kieker: Demonstrated as Export from the Astronomy Shop
-haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.org/pdf/2510.11179"
-```
-
-Then we can query this:
-
-```shell
-haiku-rag ask "Who wrote a paper about OpenTelemetry interoperability, and what was his take"
-```
-
-We should get something along the lines:
-
-```
-Answer:
-David Georg Reichelt from Lancaster University wrote a paper titled "Interoperability From OpenTelemetry to Kieker: Demonstrated as Export from the Astronomy Shop." In his work, he indicates that there is a structural difference between Kieker’s synchronous traces and OpenTelemetry’s asynchronous traces, leading to  limited compatibility between the two systems. This highlights the challenges of interoperability in observability frameworks.
-```
-
-We can also add offline files, like PDFs. Here we add a local file to ensure OpenAI does not cheat - a file we know that should not be very well known in Internet:
-
-```shell
-# This static file is supplied in haiku.rag repo
-haiku-rag add-src "examples/samples/PyCon Finland 2025 Schedule.html"
-```
-
-And then:
-
-```shell
-haiku-rag ask "Who were presenting talks in Pycon Finland 2025? Can you give at least five different people."
-```
-
-```
-The following people are presenting talks at PyCon Finland 2025:
-
- 1 Jeremy Mayeres - Talk: The Limits of Imagination: An Open Source Journey
- 2 Aroma Rodrigues - Talk: Python and Rust, a Perfect Pairing
- 3 Andreas Jung - Talk: Guillotina Volto: A New Backend for Volto
- 4 Daniel Vahla - Talk: Experiences with AI in Software Projects
- 5 Andreas Jung (also presenting another talk) - Talk: Debugging Python
-```
-
-## Configuration
-
-See [Configuration page](./configuration.md) for complete documentation on YAML configuration and all available options.
+- [Chat](chat.md): sessions, citations, and the full TUI.
+- [CLI reference](cli.md): every command.
+- [Python API](python.md): use haiku.rag in your own code.
+- [Capabilities](capabilities/index.md): native RAG and analysis components used by the client.
+- [Tuning](tuning.md): better retrieval.
+- [Configuration](configuration/index.md): every setting.

evaluations/README.md

@@ -1,4 +1,4 @@
-# Haiku RAG - Evaluations
+# haiku.rag - Evaluations
 
 Internal benchmarking and evaluation scripts for haiku.rag.
 
@@ -6,6 +6,99 @@ This package is not published to PyPI and is only used for development and testi
 
 ## Overview
 
-Contains evaluation scripts for benchmarking RAG performance using datasets like:
-- RepliQA
-- WiX
+Contains evaluation scripts for benchmarking RAG retrieval and QA performance. Available datasets:
+
+- HotpotQA (`hotpotqa`) — multi-hop QA over Wikipedia paragraphs (distractor validation split, 7,405 questions, two gold documents per question)
+- MTRAG ClapNQ (`mtrag_clapnq`, `mtrag_clapnq_rewrite`) — IBM's multi-turn RAG benchmark, ClapNQ (Wikipedia) domain: 183,408 passages, 208 retrieval queries with binary qrels, 224 generation tasks. The base key retrieves with the raw last user turn; the `_rewrite` variant uses the human standalone rewrites (both share one database). Retrieval reports Recall@5/@10, nDCG@5/@10, and MAP against IBM's published setup. QA replays each task's reference conversation prefix as message history and answers the final turn; the judge sees the conversation as a transcript, citation MAP is scored only on turns with gold passages, and refusal precision/recall is reported against the answerability labels. Generation scores are internal (our judge and rubric), not comparable with IBM's published generation numbers. The `mtrag_clapnq_live` key replays whole conversations (one case per conversation, `--limit` counts conversations) through a single capability session, carrying the model's own answers and tool history across turns; it reports the same outcomes per turn plus micro (per-turn) and macro (per-conversation) aggregates.
+- FRAMES (`frames`) — multi-hop QA (822 questions, 2-23 gold Wikipedia articles per question; 2 of the original 824 questions are excluded because a linked article has been deleted from Wikipedia). The corpus is the union of the 2,521 linked articles, fetched from the Wikipedia REST API at current revision (revision id and fetch date recorded in the article cache) with navigation chrome stripped. There is no official FRAMES evaluation setup; numbers here correspond to the paper's multi-step retrieval setting (fixed corpus, agentic retrieval, judged accuracy) and are not comparable to its closed-book, oracle-prompt, or web-search settings. Answers were authored against ~2024 revisions and may have drifted with article content.
+- OpenRAG Bench, two variants:
+  - `orb_text` — text embedder (`qwen3-embedding:4b`, 2560-dim) with VLM picture descriptions baked into chunk content at ingest. Use for text-only retrieval/QA against figure-rich corpora.
+  - `orb_multimodal` — multimodal embedder (`qwen3-vl-embedding-8b`, 4096-dim) with picture vectors in the same space as text. Use for cross-modal retrieval (text-as-query → figure hits, image-as-query) and vision QA where the figure itself is the answer.
+
+## Usage
+
+After installing the package, you can run evaluations using the `evaluations` command:
+
+```bash
+# Run retrieval + QA benchmarks
+evaluations run hotpotqa
+evaluations run orb_text
+
+# Use a custom config file
+evaluations run hotpotqa --config /path/to/haiku.rag.yaml
+
+# Override the database path
+evaluations run hotpotqa --db /path/to/custom.lancedb
+
+# Skip database population and run only benchmarks
+evaluations run hotpotqa --skip-db
+
+# Skip specific benchmarks
+evaluations run hotpotqa --skip-retrieval
+evaluations run hotpotqa --skip-qa
+
+# Limit the number of test cases
+evaluations run hotpotqa --limit 100
+```
+
+### Choosing the target
+
+`evaluations run` benchmarks `--target rag-capability` by default. Use
+`--target analysis-capability` to benchmark the analysis capability against the same
+datasets and judge:
+
+```bash
+evaluations run hotpotqa --target rag-capability
+evaluations run hotpotqa --target analysis-capability --capability-model ollama:qwen3.8
+```
+
+`--capability-model "provider:name"` overrides the capability model independently from
+the judge (defaults to `qa.model`, or `analysis.model` when set for the
+analysis-capability target). A citation retrieval metric (`cited_map`) is computed
+alongside QA accuracy from the URIs the capability registered via the `cite` tool.
+
+### Debugging runs in Logfire
+
+With `LOGFIRE_TOKEN` set, runs ship spans under `service_name = 'evals'`. The
+`debug-evals` skill in `.claude/skills/` turns these into ready-made Logfire
+queries (recent runs, per-case pass rate and `cited_map`, failing and slowest
+cases) for use from Claude Code.
+
+### Pre-built Databases
+
+Download pre-built evaluation databases from HuggingFace:
+
+```bash
+evaluations download hotpotqa
+evaluations download all
+evaluations download hotpotqa --force
+```
+
+Upload databases (maintainer only):
+
+```bash
+evaluations upload hotpotqa
+evaluations upload all
+```
+
+## Database Storage
+
+By default, evaluation databases are stored in the haiku.rag data directory:
+- **Linux**: `~/.local/share/haiku.rag/evaluations/dbs/`
+- **macOS**: `~/Library/Application Support/haiku.rag/evaluations/dbs/`
+- **Windows**: `C:/Users/<USER>/AppData/Roaming/haiku.rag/evaluations/dbs/`
+
+You can override this with the `--db` option.
+
+### Evaluating over Multiple Databases
+
+With [`lancedb.databases`](https://ggozad.github.io/haiku.rag/configuration/storage/#multiple-databases) configured, `evaluations run <dataset> --skip-db` benchmarks the full set. Retrieval, QA, and live conversations preserve the database name on results and citations. A configured set of one follows the same path and retains its name.
+
+Population writes one database and therefore requires `--db`:
+
+```bash
+evaluations run hotpotqa --db /path/to/one.lancedb   # populate, then benchmark
+evaluations run hotpotqa --skip-db                   # benchmark the configured set
+```
+
+`--db` overrides the configured set for both population and benchmarks.

evaluations/configs/frames.yaml

@@ -0,0 +1,52 @@
+# Reference config for the `frames` evaluation database.
+# FRAMES (google/frames-benchmark): 824 multi-hop questions over a corpus of
+# the ~2.5k Wikipedia articles linked per question, fetched at current
+# revision (revid + fetch date recorded in the article cache).
+# Run: evaluations run frames --config configs/frames.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: openai
+    name: qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://vllm:11431/v1
+
+reranking:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
+
+analysis:
+  # Bounds per-execution sandbox output so accumulated code returns cannot
+  # outgrow the model's input budget.
+  max_output_chars: 20000
+
+qa:
+  model:
+    provider: openai
+    name: gemma4-26b
+    base_url: http://vllm:11432/v1
+    # vLLM reserves max_tokens out of max_model_len; a large value starves
+    # the input budget and 400s long agentic contexts.
+    max_tokens: 8192
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/hotpotqa.yaml

@@ -0,0 +1,37 @@
+# Reference config for the `hotpotqa` pre-built evaluation database.
+# HotpotQA (distractor validation split) multi-hop QA over wiki paragraphs.
+# Run: evaluations run hotpotqa --config configs/hotpotqa.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: openai
+    name: qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://vllm:11431/v1
+
+qa:
+  model:
+    provider: openai
+    name: gemma4-26b
+    base_url: http://vllm:11432/v1
+    max_tokens: 49152
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/mtrag_clapnq.yaml

@@ -0,0 +1,56 @@
+# Reference config for the `mtrag_clapnq` pre-built evaluation database.
+# IBM MTRAG, ClapNQ (Wikipedia) domain: multi-turn retrieval and QA over
+# 183,408 passages. Also serves mtrag_clapnq_rewrite, mtrag_clapnq_live and
+# mtrag_clapnq_live_uncompacted.
+# Run: evaluations run mtrag_clapnq --config configs/mtrag_clapnq.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+# The corpus is text-only: no multimodal embedder, no vision paths. This eval
+# cannot exercise image or vision turn-boundary behavior.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: openai
+    name: qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://vllm:11431/v1
+
+reranking:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
+
+qa:
+  model:
+    provider: openai
+    name: RedHatAI/Muse-Glimmer-30B-NVFP4
+    base_url: http://vllm:11450/v1
+    # vLLM enforces input + max_tokens <= max_model_len, so a large output
+    # budget silently shrinks the input budget. MTRAG answers are sentences.
+    max_tokens: 8192
+    extra_body:
+      chat_template_kwargs:
+        # Part of the measured baseline. vLLM's reasoning parser consumes
+        # enable_thinking before the chat template sees it; reasoning_strength
+        # is the knob Muse Glimmer templates honour, and a template that
+        # defaults it to low silently changes search behavior.
+        reasoning_strength: high
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/orb_multimodal.yaml

@@ -0,0 +1,41 @@
+# Reference config for the `orb_multimodal` pre-built evaluation database.
+# OpenRAG Bench with a multimodal embedder; picture vectors share the text space.
+# Run: evaluations run orb_multimodal --skip-db --config configs/orb_multimodal.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: vllm
+    name: qwen3-embedding-v-8b
+    vector_dim: 4096
+    multimodal: true
+    base_url: http://vllm:11433/v1
+
+reranking:
+  model: null
+
+qa:
+  model:
+    provider: openai
+    name: gemma4-26b
+    base_url: http://vllm:11432/v1
+    vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/orb_multimodal_nemotron.yaml

@@ -0,0 +1,42 @@
+# Reference config for the `orb_multimodal_nemotron` pre-built evaluation database.
+# OpenRAG Bench with the nvidia/llama-nemotron-embed-vl-1b-v2 multimodal embedder,
+# the embedder behind the published headline benchmark numbers.
+# Run: evaluations run orb_multimodal_nemotron --skip-db --config configs/orb_multimodal_nemotron.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: vllm
+    name: nvidia/llama-nemotron-embed-vl-1b-v2
+    vector_dim: 2048
+    multimodal: true
+    base_url: http://vllm:11438/v1
+
+reranking:
+  model: null
+
+qa:
+  model:
+    provider: openai
+    name: gemma4-26b
+    base_url: http://vllm:11432/v1
+    vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/orb_text.yaml

@@ -0,0 +1,43 @@
+# Reference config for the `orb_text` pre-built evaluation database.
+# OpenRAG Bench with a text embedder and VLM picture descriptions baked into chunks.
+# Run: evaluations run orb_text --skip-db --config configs/orb_text.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: openai
+    name: qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://vllm:11431/v1
+
+reranking:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
+
+qa:
+  model:
+    provider: openai
+    name: gemma4-26b
+    base_url: http://vllm:11432/v1
+    vision: true
+
+evaluations:
+  judge:
+    provider: openai
+    name: Inferact/Qwen3.8-27B-NVFP4
+    base_url: http://vllm:11439/v1
+    temperature: 0.6
+    max_tokens: 16384
+    extra_body:
+      top_p: 0.95
+      top_k: 20
+      min_p: 0
+      chat_template_kwargs:
+        reasoning_effort: low

evaluations/configs/t2_finqa.yaml

@@ -0,0 +1,49 @@
+# Reference config for the `t2_finqa` pre-built evaluation database.
+# T²-RAGBench (FinQA) financial QA, scored by exact numeric match.
+# No `evaluations.judge` block: the spec sets `NumberMatchEvaluator`, which
+# replaces the evaluator list, so no LLM judge is constructed for this dataset.
+# Run: evaluations run t2_finqa --skip-db --target analysis-capability --config configs/t2_finqa.yaml
+# base_url uses the `vllm` host serving each model over an OpenAI-compatible API.
+
+environment: development
+
+storage:
+  auto_vacuum: false
+
+embeddings:
+  model:
+    provider: openai
+    name: qwen3-embedding-4b
+    vector_dim: 2560
+    base_url: http://vllm:11431/v1
+
+processing:
+  converter: docling-local
+  pictures: none  # ~4% of pages carry a figure/chart; dropped as non-essential to the numeric QA
+  chunking_use_markdown_tables: true
+
+reranking:
+  model:
+    provider: vllm
+    name: Qwen/Qwen3-Reranker-4B
+    base_url: http://vllm:11455
+
+qa:
+  model:
+    provider: openai
+    name: RedHatAI/Qwen3.6-35B-A3B-NVFP4
+    base_url: http://vllm:11430/v1
+    temperature: 0.3
+    max_tokens: 16384
+    extra_body:
+      chat_template_kwargs:
+        enable_thinking: true
+
+prompts:
+  domain_preamble: |
+    Use search() to find the relevant documents. Do not iterate over all of
+    /documents or read every document's content — that will time out.
+
+    For questions with a numeric answer, end your response with a final line
+    formatted exactly as `ANSWER: <number>`, containing a single number. Keep a
+    percent sign if the answer is a percentage.

evaluations/evaluations/artifacts.py

@@ -0,0 +1,106 @@
+"""Pre-built evaluation databases on HuggingFace."""
+
+import os
+import shutil
+import tempfile
+from pathlib import Path
+
+from huggingface_hub import HfApi, snapshot_download
+from rich.console import Console
+
+from evaluations.config import DatasetSpec
+
+console = Console()
+
+HF_REPO_ID = "ggozad/haiku-rag-eval-dbs"
+
+
+def download_dataset_db(spec: DatasetSpec, force: bool = False) -> None:
+    """Fetch one dataset's database from HuggingFace into its local path."""
+    db = spec.db_path()
+    if db.exists() and not force:
+        console.print(
+            f"[yellow]Skipping {spec.key}: database already exists at {db}[/yellow]"
+        )
+        console.print("Use --force to overwrite.")
+        return
+
+    console.print(f"[blue]Downloading {spec.key}...[/blue]")
+
+    try:
+        downloaded_path = snapshot_download(
+            repo_id=HF_REPO_ID,
+            repo_type="dataset",
+            allow_patterns=f"{spec.db_filename}/*",
+        )
+    except Exception as e:
+        console.print(f"[red]Failed to download {spec.key}: {e}[/red]")
+        return
+
+    source_path = Path(downloaded_path) / spec.db_filename
+    if not source_path.exists():
+        console.print(f"[red]Database {spec.key} not found in HuggingFace repo.[/red]")
+        console.print(
+            f"[yellow]The database may not have been uploaded yet. "
+            f"Try running 'evaluations build {spec.key}' to create it locally.[/yellow]"
+        )
+        return
+
+    if db.exists():
+        shutil.rmtree(db)
+
+    db.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copytree(source_path, db)
+
+    console.print(f"[green]Downloaded {spec.key} to {db}[/green]")
+
+
+def upload_dataset_db(spec: DatasetSpec) -> None:
+    """Push one dataset's database to HuggingFace (maintainer only).
+
+    Uses ``upload_large_folder`` for resumable, parallel transfer — important
+    for the multi-GB ORB databases which would otherwise abort on any transient
+    network failure under plain ``upload_folder``.
+
+    ``upload_large_folder`` has no ``path_in_repo`` — it ships the contents of
+    ``folder_path`` to the repo root. Stage the db under a temp parent with
+    hardlinks so the basename becomes the remote path, leaving everything else
+    at the root undisturbed.
+    """
+    db = spec.db_path()
+    if not db.exists():
+        console.print(f"[red]Database not found at {db}[/red]")
+        return
+
+    api = HfApi()
+
+    # Wipe the existing remote path so we don't accumulate orphaned files from
+    # prior uploads. upload_large_folder doesn't accept delete_patterns, so we
+    # do this as a separate commit. Safe to run if the path is missing.
+    try:
+        api.delete_folder(
+            path_in_repo=spec.db_filename,
+            repo_id=HF_REPO_ID,
+            repo_type="dataset",
+        )
+    except Exception:
+        pass
+
+    with tempfile.TemporaryDirectory() as staging:
+        target = Path(staging) / spec.db_filename
+        target.mkdir()
+        for src in db.rglob("*"):
+            if not src.is_file():
+                continue
+            dest = target / src.relative_to(db)
+            dest.parent.mkdir(parents=True, exist_ok=True)
+            os.link(src, dest)
+
+        console.print(f"[blue]Uploading {spec.key} ({db})...[/blue]")
+        api.upload_large_folder(
+            folder_path=staging,
+            repo_id=HF_REPO_ID,
+            repo_type="dataset",
+        )
+
+    console.print(f"[green]Uploaded {spec.key} to {HF_REPO_ID}[/green]")

evaluations/evaluations/benchmark.py

@@ -1,338 +1,237 @@
 import asyncio
-from collections.abc import Mapping
+
 from pathlib import Path
-from typing import Any, cast
+from typing import cast
 
-import logfire
 import typer
-from pydantic_ai.models.openai import OpenAIChatModel
-from pydantic_ai.providers.ollama import OllamaProvider
-from pydantic_evals import Dataset as EvalDataset
-from pydantic_evals.evaluators import IsInstance, LLMJudge
-from pydantic_evals.reporting import ReportCaseFailure
+from dotenv import find_dotenv, load_dotenv
 from rich.console import Console
-from rich.progress import Progress
 
+from evaluations.artifacts import download_dataset_db, upload_dataset_db
 from evaluations.config import DatasetSpec
+from evaluations.population import populate_db
+from evaluations.qa import TARGETS, Target, run_live_qa_benchmark, run_qa_benchmark
+from evaluations.retrieval import run_retrieval_benchmark
 from evaluations.datasets import DATASETS
-from evaluations.llm_judge import ANSWER_EQUIVALENCE_RUBRIC
-from evaluations.prompts import WIX_SUPPORT_PROMPT
-from haiku.rag.client import HaikuRAG
 from haiku.rag.config import AppConfig, find_config_file, load_yaml_config
+from haiku.rag.config.models import ModelConfig
 from haiku.rag.logging import configure_cli_logging
-from haiku.rag.qa import get_qa_agent
+from haiku.rag.telemetry import configure as configure_telemetry
+from haiku.rag.utils import parse_model_option
 
-QA_JUDGE_MODEL = "qwen3"
 
-logfire.configure(send_to_logfire="if-token-present", service_name="evals")
-logfire.instrument_pydantic_ai()
+load_dotenv(find_dotenv(usecwd=True))
+
+# Scrubbing off: eval outputs are financial answers with words like "authorized"
+# that trip Logfire's secret scrubber and redact the model's answer text.
+configure_telemetry(service_name="evals", scrubbing=False)
 configure_cli_logging()
 console = Console()
 
 
-async def populate_db(spec: DatasetSpec, config: AppConfig) -> None:
-    spec.db_path.parent.mkdir(parents=True, exist_ok=True)
-    corpus = spec.document_loader()
-    if spec.document_limit is not None:
-        corpus = corpus.select(range(min(spec.document_limit, len(corpus))))
-
-    with Progress() as progress:
-        task = progress.add_task("[green]Populating database...", total=len(corpus))
-        async with HaikuRAG(spec.db_path, config=config) as rag:
-            for doc in corpus:
-                doc_mapping = cast(Mapping[str, Any], doc)
-                payload = spec.document_mapper(doc_mapping)
-                if payload is None:
-                    progress.advance(task)
-                    continue
-
-                existing = await rag.get_document_by_uri(payload.uri)
-                if existing is not None:
-                    assert existing.id
-                    chunks = await rag.chunk_repository.get_by_document_id(existing.id)
-                    if chunks:
-                        progress.advance(task)
-                        continue
-                    await rag.document_repository.delete(existing.id)
-
-                await rag.create_document(
-                    content=payload.content,
-                    uri=payload.uri,
-                    title=payload.title,
-                    metadata=payload.metadata,
-                )
-                progress.advance(task)
-
-
-async def run_retrieval_benchmark(
-    spec: DatasetSpec, config: AppConfig
-) -> dict[str, float] | None:
-    if spec.retrieval_loader is None or spec.retrieval_mapper is None:
-        console.print("Skipping retrieval benchmark; no retrieval config.")
-        return None
-
-    corpus = spec.retrieval_loader()
-
-    recall_totals = {
-        1: 0.0,
-        3: 0.0,
-        5: 0.0,
-    }
-    success_totals = {
-        1: 0.0,
-        3: 0.0,
-        5: 0.0,
-    }
-    total_queries = 0
-
-    with Progress() as progress:
-        task = progress.add_task(
-            "[blue]Running retrieval benchmark...", total=len(corpus)
-        )
-        async with HaikuRAG(spec.db_path, config=config) as rag:
-            for doc in corpus:
-                doc_mapping = cast(Mapping[str, Any], doc)
-                sample = spec.retrieval_mapper(doc_mapping)
-                if sample is None or sample.skip:
-                    progress.advance(task)
-                    continue
-
-                matches = await rag.search(query=sample.question, limit=5)
-                if not matches:
-                    progress.advance(task)
-                    continue
-
-                total_queries += 1
-
-                retrieved_uris: list[str] = []
-                for chunk, _ in matches:
-                    if chunk.document_id is None:
-                        continue
-                    retrieved_doc = await rag.get_document_by_id(chunk.document_id)
-                    if retrieved_doc and retrieved_doc.uri:
-                        retrieved_uris.append(retrieved_doc.uri)
-
-                # Compute metrics for each cutoff
-                for cutoff in (1, 3, 5):
-                    top_k = set(retrieved_uris[:cutoff])
-                    relevant = set(sample.expected_uris)
-                    if relevant:
-                        matched = len(top_k & relevant)
-                        # Recall: fraction of relevant docs retrieved
-                        recall_totals[cutoff] += matched / len(relevant)
-                        # Success: binary - did we get at least one relevant doc?
-                        success_totals[cutoff] += 1.0 if matched > 0 else 0.0
-
-                progress.advance(task)
-
-    if total_queries == 0:
-        console.print("No retrieval cases to evaluate.")
-        return None
-
-    recall_at_1 = recall_totals[1] / total_queries
-    recall_at_3 = recall_totals[3] / total_queries
-    recall_at_5 = recall_totals[5] / total_queries
-
-    success_at_1 = success_totals[1] / total_queries
-    success_at_3 = success_totals[3] / total_queries
-    success_at_5 = success_totals[5] / total_queries
-
-    console.print("\n=== Retrieval Benchmark Results ===", style="bold cyan")
-    console.print(f"Total queries: {total_queries}")
-    console.print("\nRecall@K (fraction of relevant docs retrieved):")
-    console.print(f"  Recall@1: {recall_at_1:.4f}")
-    console.print(f"  Recall@3: {recall_at_3:.4f}")
-    console.print(f"  Recall@5: {recall_at_5:.4f}")
-    console.print("\nSuccess@K (queries with at least one relevant doc):")
-    console.print(f"  Success@1: {success_at_1:.4f} ({success_at_1 * 100:.1f}%)")
-    console.print(f"  Success@3: {success_at_3:.4f} ({success_at_3 * 100:.1f}%)")
-    console.print(f"  Success@5: {success_at_5:.4f} ({success_at_5 * 100:.1f}%)")
-
-    return {
-        "recall@1": recall_at_1,
-        "recall@3": recall_at_3,
-        "recall@5": recall_at_5,
-        "success@1": success_at_1,
-        "success@3": success_at_3,
-        "success@5": success_at_5,
-    }
-
-
-async def run_qa_benchmark(
-    spec: DatasetSpec, config: AppConfig, qa_limit: int | None = None
-) -> ReportCaseFailure[str, str, dict[str, str]] | None:
-    corpus = spec.qa_loader()
-    if qa_limit is not None:
-        corpus = corpus.select(range(min(qa_limit, len(corpus))))
-
-    cases = [
-        spec.qa_case_builder(index, cast(Mapping[str, Any], doc))
-        for index, doc in enumerate(corpus, start=1)
-    ]
-
-    judge_model = OpenAIChatModel(
-        model_name=QA_JUDGE_MODEL,
-        provider=OllamaProvider(base_url=f"{config.providers.ollama.base_url}/v1"),
-    )
-
-    evaluation_dataset = EvalDataset[str, str, dict[str, str]](
-        cases=cases,
-        evaluators=[
-            IsInstance(type_name="str"),
-            LLMJudge(
-                rubric=ANSWER_EQUIVALENCE_RUBRIC,
-                include_input=True,
-                include_expected_output=True,
-                model=judge_model,
-                assertion={
-                    "evaluation_name": "answer_equivalent",
-                    "include_reason": True,
-                },
-            ),
-        ],
-    )
-
-    total_processed = 0
-    passing_cases = 0
-    failures: list[ReportCaseFailure[str, str, dict[str, str]]] = []
-
-    with Progress(console=console) as progress:
-        qa_task = progress.add_task(
-            "[yellow]Evaluating QA cases...",
-            total=len(evaluation_dataset.cases),
-        )
-
-        async with HaikuRAG(spec.db_path, config=config) as rag:
-            system_prompt = WIX_SUPPORT_PROMPT if spec.key == "wix" else None
-            qa = get_qa_agent(rag, system_prompt=system_prompt)
-
-            async def answer_question(question: str) -> str:
-                return await qa.answer(question)
-
-            for case in evaluation_dataset.cases:
-                single_case_dataset = EvalDataset[str, str, dict[str, str]](
-                    cases=[case],
-                    evaluators=evaluation_dataset.evaluators,
-                )
-
-                report = await single_case_dataset.evaluate(
-                    answer_question,
-                    name="qa_answer",
-                    max_concurrency=1,
-                    progress=False,
-                )
-
-                total_processed += 1
-
-                if report.cases:
-                    result_case = report.cases[0]
-
-                    equivalence = result_case.assertions.get("answer_equivalent")
-                    if equivalence is not None:
-                        if equivalence.value:
-                            passing_cases += 1
-
-                if report.failures:
-                    failures.extend(report.failures)
-                    failure = report.failures[0]
-                    progress.console.print(
-                        "[red]Failure encountered during case evaluation:[/red]"
-                    )
-                    progress.console.print(f"Error: {failure.error_message}")
-                    progress.console.print("")
-
-                progress.update(
-                    qa_task,
-                    description="[yellow]Evaluating QA cases...[/yellow] "
-                    f"[green]Accuracy: {(passing_cases / total_processed):.2f} "
-                    f"{passing_cases}/{total_processed}[/green]",
-                )
-                progress.advance(qa_task)
-
-    total_cases = total_processed
-    accuracy = passing_cases / total_cases if total_cases > 0 else 0
-
-    console.print("\n=== QA Benchmark Results ===", style="bold cyan")
-    console.print(f"Total questions: {total_cases}")
-    console.print(f"Correct answers: {passing_cases}")
-    console.print(f"QA Accuracy: {accuracy:.4f} ({accuracy * 100:.2f}%)")
-
-    if failures:
-        console.print("[red]\nSummary of failures:[/red]")
-        for failure in failures:
-            console.print(f"Case: {failure.name}")
-            console.print(f"Question: {failure.inputs}")
-            console.print(f"Error: {failure.error_message}")
-            console.print("")
-
-    return failures[0] if failures else None
-
-
 async def evaluate_dataset(
     spec: DatasetSpec,
     config: AppConfig,
     skip_db: bool,
     skip_retrieval: bool,
     skip_qa: bool,
-    qa_limit: int | None,
+    limit: int | None,
+    name: str | None,
+    db_path: Path | None,
+    vacuum_interval: int = 100,
+    multimodal_only: bool = False,
+    judge_model: ModelConfig | None = None,
+    target: Target = "rag-capability",
+    capability_model: ModelConfig | None = None,
+    case_ids: set[str] | None = None,
+    document_filter: str | None = None,
 ) -> None:
+    if document_filter is not None:
+        console.print(f"Document filter: {document_filter}", style="dim")
+
+    if db_path is not None and config.lancedb.databases:
+        raise ValueError(
+            "--db PATH places the database where the configuration places none, "
+            f"and this configuration names {', '.join(config.lancedb.databases)} "
+            "in lancedb.databases. Drop --db to evaluate the configured set."
+        )
+
     if not skip_db:
+        if spec.uses_configured_databases(config, db_path):
+            raise ValueError(
+                "lancedb.databases places the databases this run reads, and "
+                "population writes to one, so it would ingest into a database "
+                "the run does not read. Pass --skip-db to evaluate the "
+                "configured set, or --db PATH to populate and evaluate one."
+            )
         console.print(f"Using dataset: {spec.key}", style="bold magenta")
-        await populate_db(spec, config)
+        await populate_db(
+            spec, config, db_path=db_path, vacuum_interval=vacuum_interval
+        )
 
     if not skip_retrieval:
         console.print("Running retrieval benchmarks...", style="bold blue")
-        await run_retrieval_benchmark(spec, config)
+        await run_retrieval_benchmark(
+            spec,
+            config,
+            limit=limit,
+            name=name,
+            db_path=db_path,
+            multimodal_only=multimodal_only,
+            document_filter=document_filter,
+        )
 
     if not skip_qa:
-        console.print("\nRunning QA benchmarks...", style="bold yellow")
-        await run_qa_benchmark(spec, config, qa_limit=qa_limit)
+        console.print(
+            f"\nRunning QA benchmarks (target={target})...", style="bold yellow"
+        )
+        qa_benchmark = run_live_qa_benchmark if spec.live else run_qa_benchmark
+        await qa_benchmark(
+            spec,
+            config,
+            limit=limit,
+            name=name,
+            db_path=db_path,
+            judge_model=judge_model,
+            target=target,
+            capability_model=capability_model,
+            case_ids=case_ids,
+            document_filter=document_filter,
+        )
 
 
 app = typer.Typer(help="Run retrieval and QA benchmarks for configured datasets.")
 
 
+def _load_config(config_path: Path | None) -> AppConfig:
+    """Load AppConfig from a file path or standard search path."""
+    if config_path:
+        if not config_path.exists():
+            raise typer.BadParameter(f"Config file not found: {config_path}")
+        console.print(f"Loading config from: {config_path}", style="dim")
+        yaml_data = load_yaml_config(config_path)
+        return AppConfig.model_validate(yaml_data)
+
+    found = find_config_file(None)
+    if found:
+        console.print(f"Loading config from: {found}", style="dim")
+        yaml_data = load_yaml_config(found)
+        return AppConfig.model_validate(yaml_data)
+
+    console.print("No config file found, using defaults", style="dim")
+    return AppConfig()
+
+
+def _load_case_ids(path: Path | None) -> set[str] | None:
+    """Read a newline-delimited case-id file into a set (None when no path)."""
+    if path is None:
+        return None
+    return {line.strip() for line in path.read_text().splitlines() if line.strip()}
+
+
+def _resolve_dataset(dataset: str) -> DatasetSpec:
+    """Resolve a dataset key to a DatasetSpec or raise BadParameter."""
+    spec = DATASETS.get(dataset.lower())
+    if spec is None:
+        valid_datasets = ", ".join(sorted(DATASETS))
+        raise typer.BadParameter(
+            f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}"
+        )
+    return spec
+
+
+def _resolve_datasets(dataset: str) -> list[DatasetSpec]:
+    """Resolve 'all' or a single dataset key to a list of DatasetSpecs.
+
+    'all' yields one spec per database: query variants sharing a db_filename
+    would otherwise be downloaded/uploaded twice.
+    """
+    if dataset.lower() == "all":
+        seen: set[str] = set()
+        specs: list[DatasetSpec] = []
+        for spec in DATASETS.values():
+            if spec.db_filename in seen:
+                continue
+            seen.add(spec.db_filename)
+            specs.append(spec)
+        return specs
+    return [_resolve_dataset(dataset)]
+
+
 @app.command()
 def run(
     dataset: str = typer.Argument(..., help="Dataset key to evaluate."),
     config: Path | None = typer.Option(
         None, "--config", help="Path to haiku.rag YAML config file."
     ),
+    db: Path | None = typer.Option(
+        None,
+        "--db",
+        help="Database path, where the configuration places no database.",
+    ),
     skip_db: bool = typer.Option(
-        False, "--skip-db", help="Skip updateing the evaluation db."
+        False, "--skip-db", help="Skip updating the evaluation db."
     ),
     skip_retrieval: bool = typer.Option(
         False, "--skip-retrieval", help="Skip retrieval benchmark."
     ),
     skip_qa: bool = typer.Option(False, "--skip-qa", help="Skip QA benchmark."),
-    qa_limit: int | None = typer.Option(
-        None, "--qa-limit", help="Limit number of QA cases."
+    limit: int | None = typer.Option(
+        None, "--limit", help="Limit number of test cases for both retrieval and QA."
+    ),
+    name: str | None = typer.Option(None, "--name", help="Override evaluation name."),
+    vacuum_interval: int = typer.Option(
+        100, "--vacuum-interval", help="Vacuum every N documents during DB population."
+    ),
+    multimodal_only: bool = typer.Option(
+        False,
+        "--multimodal-only",
+        help="Only evaluate queries requiring image understanding.",
+    ),
+    target: str = typer.Option(
+        "rag-capability",
+        "--target",
+        help="What to benchmark: rag-capability | analysis-capability.",
+    ),
+    capability_model: str | None = typer.Option(
+        None,
+        "--capability-model",
+        help=(
+            "Capability model as 'provider:name'. Defaults to qa.model (or "
+            "analysis.model when --target is analysis-capability) from the config."
+        ),
+    ),
+    document_filter: str | None = typer.Option(
+        None,
+        "--filter",
+        "-f",
+        help=(
+            "SQL WHERE clause over document columns (id, uri, title, "
+            "created_at, updated_at, metadata) restricting every benchmark "
+            "search, e.g. \"uri LIKE '%arxiv%'\". metadata is stored as a "
+            "string, so match it with LIKE."
+        ),
+    ),
+    filter_ids: Path | None = typer.Option(
+        None,
+        "--filter-ids",
+        help=(
+            "Path to a newline-delimited file of QA case ids to run "
+            "(failure-subset rerun). Filters QA only; retrieval is unaffected."
+        ),
     ),
 ) -> None:
-    spec = DATASETS.get(dataset.lower())
-    if spec is None:
-        valid_datasets = ", ".join(sorted(DATASETS))
+    spec = _resolve_dataset(dataset)
+    app_config = _load_config(config)
+    if target not in TARGETS:
         raise typer.BadParameter(
-            f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}"
+            f"Unknown target {target!r}. Choose from: {', '.join(TARGETS)}"
         )
-
-    # Load config from file or use defaults
-    if config:
-        if not config.exists():
-            raise typer.BadParameter(f"Config file not found: {config}")
-        console.print(f"Loading config from: {config}", style="dim")
-        yaml_data = load_yaml_config(config)
-        app_config = AppConfig.model_validate(yaml_data)
-    else:
-        # Try to find config file using standard search path
-        config_path = find_config_file(None)
-        if config_path:
-            console.print(f"Loading config from: {config_path}", style="dim")
-            yaml_data = load_yaml_config(config_path)
-            app_config = AppConfig.model_validate(yaml_data)
-        else:
-            console.print("No config file found, using defaults", style="dim")
-            app_config = AppConfig()
+    target_value = cast(Target, target)
+    judge_model_config = app_config.evaluations.judge
+    capability_model_config = (
+        parse_model_option(capability_model) if capability_model else None
+    )
 
     asyncio.run(
         evaluate_dataset(
@@ -341,10 +240,38 @@ def run(
             skip_db=skip_db,
             skip_retrieval=skip_retrieval,
             skip_qa=skip_qa,
-            qa_limit=qa_limit,
+            limit=limit,
+            name=name,
+            db_path=db,
+            vacuum_interval=vacuum_interval,
+            multimodal_only=multimodal_only,
+            judge_model=judge_model_config,
+            target=target_value,
+            capability_model=capability_model_config,
+            case_ids=_load_case_ids(filter_ids),
+            document_filter=document_filter,
         )
     )
 
 
+@app.command()
+def download(
+    dataset: str = typer.Argument(..., help="Dataset key or 'all' to download all."),
+    force: bool = typer.Option(False, "--force", help="Overwrite existing database."),
+) -> None:
+    """Download pre-built evaluation database from HuggingFace."""
+    for spec in _resolve_datasets(dataset):
+        download_dataset_db(spec, force=force)
+
+
+@app.command()
+def upload(
+    dataset: str = typer.Argument(..., help="Dataset key or 'all' to upload all."),
+) -> None:
+    """Upload evaluation database to HuggingFace (maintainer only)."""
+    for spec in _resolve_datasets(dataset):
+        upload_dataset_db(spec)
+
+
 if __name__ == "__main__":
     app()

evaluations/evaluations/capability_runner.py

@@ -0,0 +1,296 @@
+from collections.abc import Callable, Iterable
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, NamedTuple
+
+from pydantic_ai import Agent
+from pydantic_ai.messages import (
+    ModelMessage,
+    ModelRequest,
+    ModelResponse,
+    RetryPromptPart,
+    TextPart,
+    ToolCallPart,
+    ToolReturnPart,
+    UserPromptPart,
+)
+from pydantic_ai.models import Model
+
+from pydantic_ai.capabilities import AbstractCapability
+
+from evaluations.config import Turn
+from haiku.rag.capabilities import EvidenceState, RAGCapabilityBase
+from haiku.rag.capabilities.compaction import create_capability as create_compaction
+from haiku.rag.capabilities.ledger import citation_status
+from haiku.rag.config.models import AppConfig
+
+CapabilityFactory = Callable[..., RAGCapabilityBase[Any]]
+
+
+def prefix_to_messages(turns: Iterable[Turn]) -> list[ModelMessage]:
+    """Render a conversation prefix as pydantic-ai message history."""
+    messages: list[ModelMessage] = []
+    for turn in turns:
+        if turn.speaker == "user":
+            messages.append(ModelRequest(parts=[UserPromptPart(content=turn.text)]))
+        else:
+            messages.append(ModelResponse(parts=[TextPart(content=turn.text)]))
+    return messages
+
+
+@dataclass
+class CapabilityRunResult:
+    answer: str
+    cited_uris: list[str] = field(default_factory=list)
+    cited_chunk_ids: list[str] = field(default_factory=list)
+    # The database each cited chunk came from, in the order they were cited.
+    # Empty string for a citation built without a source.
+    cited_sources: list[str] = field(default_factory=list)
+    searched_uris: list[str] = field(default_factory=list)
+    n_searches: int = 0
+    n_executions: int = 0
+    n_search_calls: int = 0
+    n_rejected_searches: int = 0
+    n_failed_tools: int = 0
+    n_requests: int = 0
+    citation_status: str | None = None
+
+
+class ToolTraffic(NamedTuple):
+    n_search_calls: int
+    n_rejected_searches: int
+    n_failed_tools: int
+    n_requests: int
+
+
+def _count_tool_traffic(
+    messages: list[ModelMessage], namespace: str, tool_names: frozenset[str]
+) -> ToolTraffic:
+    """Count search calls, failed calls and model requests in a run.
+
+    The history is the only source: ``state.searches`` is keyed by query so it
+    hides repeats and refusals, and ``for_run`` hands the run a ``replace()``
+    copy, leaving the outer capability's counters at zero.
+
+    Only search failures mean an exhausted budget. A failed code call may be
+    either the execution budget or any error in model-written Python, so
+    ``n_failed_tools`` covers both without claiming to tell them apart. It counts
+    ``RetryPromptPart`` too, since ``_cite`` rejects with ``ModelRetry`` and only
+    ``ToolFailed`` sets ``outcome="failed"``. Both are restricted to
+    ``tool_names``, excluding host tools and output-validation retries.
+
+    ``n_requests`` counts the run's requests, which matches the capability's own
+    budget only while it stays loaded — a deferred capability skips hooks until
+    it loads.
+    """
+    search_tool = f"{namespace}_search"
+    search_calls = 0
+    rejected_searches = 0
+    failed_tools = 0
+    requests = 0
+    for message in messages:
+        if isinstance(message, ModelResponse):
+            requests += 1
+            search_calls += sum(
+                1
+                for part in message.parts
+                if isinstance(part, ToolCallPart) and part.tool_name == search_tool
+            )
+            continue
+        for part in message.parts:
+            if not isinstance(part, RetryPromptPart | ToolReturnPart):
+                continue
+            if part.tool_name not in tool_names:
+                continue
+            if isinstance(part, RetryPromptPart):
+                failed_tools += 1
+            elif part.outcome == "failed":
+                failed_tools += 1
+                if part.tool_name == search_tool:
+                    rejected_searches += 1
+    return ToolTraffic(
+        n_search_calls=search_calls,
+        n_rejected_searches=rejected_searches,
+        n_failed_tools=failed_tools,
+        n_requests=requests,
+    )
+
+
+@dataclass
+class _EvalDeps:
+    state: dict[str, Any] = field(default_factory=dict)
+
+
+def _prepare_agent(
+    capability_factory: CapabilityFactory,
+    db_path: Path | None,
+    config: AppConfig,
+    capability_model: str | Model,
+    document_filter: str | None,
+    request_limit: int | None,
+    compaction: bool = False,
+) -> tuple[RAGCapabilityBase[Any], _EvalDeps, Agent[_EvalDeps, str]]:
+    capability = capability_factory(
+        db_path=db_path,
+        config=config,
+        defer_loading=False,
+    )
+    if request_limit is not None:
+        capability.request_limit = request_limit
+    state = capability.state_type()
+    if document_filter is not None:
+        state.document_filter = document_filter
+
+    capabilities: list[AbstractCapability] = [capability]
+    if compaction:
+        capabilities.append(create_compaction())
+    deps = _EvalDeps(state={capability.state_namespace: state.model_dump(mode="json")})
+    agent = Agent(
+        capability_model,
+        deps_type=_EvalDeps,
+        capabilities=capabilities,
+    )
+    return capability, deps, agent
+
+
+def _state_after_run(
+    capability: RAGCapabilityBase[Any], deps: _EvalDeps
+) -> EvidenceState:
+    return capability.state_type.model_validate(deps.state[capability.state_namespace])
+
+
+async def run_capability_question(
+    capability_factory: CapabilityFactory,
+    db_path: Path | None,
+    config: AppConfig,
+    question: str,
+    capability_model: str | Model,
+    document_filter: str | None = None,
+    request_limit: int | None = None,
+    message_history: list[ModelMessage] | None = None,
+) -> CapabilityRunResult:
+    """Run a single question through a capability and return answer + retrieval data.
+
+    Builds a native capability via ``capability_factory(db_path=..., config=...)``.
+    After the run, citations and searched documents
+    are extracted from the state for downstream eval scoring.
+
+    The capability must produce a state with RAG-capability-shaped fields (citation
+    index, searches, optional document filter) — i.e. ``RAGState`` or
+    ``AnalysisState`` from ``haiku.rag.capabilities``.
+    """
+    capability, deps, agent = _prepare_agent(
+        capability_factory,
+        db_path,
+        config,
+        capability_model,
+        document_filter,
+        request_limit,
+    )
+    agent_result = await agent.run(question, deps=deps, message_history=message_history)
+    traffic = _count_tool_traffic(
+        agent_result.new_messages(), capability.state_namespace, capability.tool_names
+    )
+    return _result_from_run(
+        agent_result.output, _state_after_run(capability, deps), traffic
+    )
+
+
+async def run_capability_conversation(
+    capability_factory: CapabilityFactory,
+    db_path: Path | None,
+    config: AppConfig,
+    questions: list[str],
+    capability_model: str | Model,
+    document_filter: str | None = None,
+    compaction: bool = False,
+) -> list[CapabilityRunResult]:
+    """Run a conversation's user turns sequentially through one capability.
+
+    Each turn runs with the previous turn's full ``all_messages()`` as history
+    (tool calls and returns included) and the same state dict, which is what
+    lets ``EvidenceCompactionCapability`` (registered when ``compaction`` is
+    True) replace earlier questions' evidence on the request. Per-invocation
+    state (citations, searches) is cleared by the capability on every run, so
+    each returned result reflects only its turn.
+    """
+    capability, deps, agent = _prepare_agent(
+        capability_factory,
+        db_path,
+        config,
+        capability_model,
+        document_filter=document_filter,
+        request_limit=None,
+        compaction=compaction,
+    )
+    history: list[ModelMessage] | None = None
+    results: list[CapabilityRunResult] = []
+    for question in questions:
+        agent_result = await agent.run(question, deps=deps, message_history=history)
+        history = agent_result.all_messages()
+        traffic = _count_tool_traffic(
+            agent_result.new_messages(),
+            capability.state_namespace,
+            capability.tool_names,
+        )
+        results.append(
+            _result_from_run(
+                agent_result.output, _state_after_run(capability, deps), traffic
+            )
+        )
+    return results
+
+
+def _result_from_run(
+    answer: str, typed: EvidenceState, traffic: ToolTraffic
+) -> CapabilityRunResult:
+    cited_chunk_ids: list[str] = list(typed.citations)
+    seen_cited: set[str] = set()
+    cited_uris: list[str] = []
+    cited_sources: list[str] = []
+    for chunk_id in cited_chunk_ids:
+        citation = typed.citation_index.get(chunk_id)
+        if citation is None:
+            continue
+        cited_sources.append(citation.source or "")
+        if citation.document_uri not in seen_cited:
+            seen_cited.add(citation.document_uri)
+            cited_uris.append(citation.document_uri)
+
+    seen_searched: set[str] = set()
+    searched_uris: list[str] = []
+    for results in typed.searches.values():
+        for search_result in results:
+            uri = search_result.document_uri
+            if uri and uri not in seen_searched:
+                seen_searched.add(uri)
+                searched_uris.append(uri)
+
+    executions = getattr(typed, "executions", None)
+    n_executions = len(executions) if executions is not None else 0
+
+    record = typed.evidence
+    status = (
+        citation_status([record], question=record.question)
+        if record.question is not None
+        else None
+    )
+
+    return CapabilityRunResult(
+        answer=answer,
+        cited_uris=cited_uris,
+        cited_chunk_ids=cited_chunk_ids,
+        cited_sources=cited_sources,
+        searched_uris=searched_uris,
+        # Distinct search keys, not searches. Analysis files every in-code
+        # `search()` under one "_sandbox" key, so twenty sandbox searches read
+        # as one here; `n_search_calls` is the true count of search *tool*
+        # calls, and in-code searches are not counted anywhere.
+        n_searches=len(typed.searches),
+        n_executions=n_executions,
+        n_search_calls=traffic.n_search_calls,
+        n_rejected_searches=traffic.n_rejected_searches,
+        n_failed_tools=traffic.n_failed_tools,
+        n_requests=traffic.n_requests,
+        citation_status=status,
+    )