From 0ee00334c7733de56467202e2dd149e25dd66a77 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Wed, 20 May 2026 14:26:39 +0300 Subject: [PATCH] drop prose emdashes and semicolons; minor fixes --- docs/benchmarks.md | 10 +++++----- docs/cli.md | 8 ++++---- docs/configuration/processing.md | 26 +++++++++++++------------- docs/configuration/prompts.md | 2 +- docs/configuration/providers.md | 8 ++++---- docs/configuration/qa.md | 4 ++-- docs/configuration/storage.md | 4 ++-- docs/custom-pipelines.md | 4 ++-- docs/python.md | 14 +++++++------- docs/remote-processing.md | 6 +++--- docs/server.md | 6 +++--- docs/skills/analysis.md | 10 +++++----- docs/skills/custom.md | 10 +++++----- docs/skills/rag.md | 6 +++--- docs/tuning.md | 14 +++++++------- 15 files changed, 66 insertions(+), 66 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index dc758a69..16dab38e 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -98,11 +98,11 @@ evaluations: - For each relevant document at position k, calculate precision@k = (relevant docs in top k) / k - Average Precision (AP) = mean of these precision values / total relevant documents - MAP is the mean of AP scores across all queries -- Range: 0 to 1; rewards ranking relevant documents higher +- Range: 0 to 1. Rewards ranking relevant documents higher ### QA Accuracy -`pydantic-evals` coordinates an LLM judge to determine whether the skill's answer is correct. The default judge is `ollama:qwen3.6` — pinned so changes to the skill model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions. +`pydantic-evals` coordinates an LLM judge to determine whether the skill's answer is correct. The default judge is `ollama:qwen3.6`, pinned so changes to the skill model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions. We picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs. @@ -110,7 +110,7 @@ We picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibrat Alongside QA accuracy, a second metric scores the URIs the skill registered via the `cite` tool against each dataset's gold `expected_uris`, using the same MRR / MAP math as raw retrieval. The score key is `cited_mrr` for single-doc datasets and `cited_map` for multi-doc. Console output also includes the cite rate (% of cases with at least one citation) and the mean number of citations per case. -This is computed alongside QA accuracy from the same skill run — no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the skill grounded its answer on it. +This is computed alongside QA accuracy from the same skill run, no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the skill grounded its answer on it. ## Current results @@ -118,7 +118,7 @@ Numbers measured under the current pinned judge (`ollama:qwen3.6`) on a recent ` ### Wix -[WixQA](https://huggingface.co/datasets/Wix/WixQA) — real customer support questions paired with curated answers. 200 cases. +[WixQA](https://huggingface.co/datasets/Wix/WixQA) is real customer support questions paired with curated answers. 200 cases. `evaluations run wix --target rag-skill` runs the RAG skill end-to-end and produces both QA accuracy and a citation retrieval metric (`cited_map`) computed from the URIs the skill registered via the `cite` tool against the gold `expected_uris`. @@ -135,7 +135,7 @@ Numbers measured under the current pinned judge (`ollama:qwen3.6`) on a recent ` Two approaches are benchmarked separately: - **Multimodal embedder** (`Qwen/Qwen3-VL-Embedding-8B`, served via vLLM): picture bytes and text live in a shared vector space, no VLM is run at ingest. -- **Text embedder + VLM picture descriptions** (`qwen3-embedding:4b` + `ollama/ministral-3`): pictures are described at ingest and the descriptions are woven into chunk text; retrieval runs over text only. See [Picture handling configuration](configuration/processing.md#picture-handling). +- **Text embedder + VLM picture descriptions** (`qwen3-embedding:4b` + `ollama/ministral-3`): pictures are described at ingest and the descriptions are woven into chunk text. Retrieval runs over text only. See [Picture handling configuration](configuration/processing.md#picture-handling). #### Multimodal embedder diff --git a/docs/cli.md b/docs/cli.md index f709b7e6..0bbb23a5 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -57,7 +57,7 @@ From directory (recursively adds all supported files): haiku-rag add-src /path/to/documents/ ``` -From an S3 bucket (requires the `[s3]` extra — see [Server Mode → S3 / Object Storage Monitoring](server.md#s3-object-storage-monitoring)): +From an S3 bucket (requires the `[s3]` extra, see [Server Mode → S3 / Object Storage Monitoring](server.md#s3-object-storage-monitoring)): ```bash # AWS S3 with credentials in the default chain (env vars, IAM role, AWS profile) haiku-rag add-src s3://my-bucket/path/to/document.pdf @@ -163,7 +163,7 @@ Filter to specific documents: haiku-rag ask "What are the main findings?" --filter "uri LIKE '%paper%'" ``` -`ask` runs the [rag skill](skills/index.md) and always renders citations under the answer. When available, citations use the document title; otherwise they fall back to the URI. +`ask` runs the [rag skill](skills/index.md) and always renders citations under the answer. When available, citations use the document title, otherwise they fall back to the URI. Flags: @@ -206,7 +206,7 @@ haiku-rag chat -s rag -s analysis Flags: -- `--skill` / `-s`: Skills to enable — `rag` (default), `analysis`. Can be repeated for multiple skills. +- `--skill` / `-s`: Skills to enable. `rag` (default), `analysis`. Can be repeated for multiple skills. The chat interface provides: @@ -385,7 +385,7 @@ haiku-rag rebuild --descriptions | Title only | `--title-only` | Generate titles for documents without one | | Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database | -**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent — pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely; only the VLM time is paid. +**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent: pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely. Only the VLM time is paid. ### Vacuum (Optimize and Cleanup) diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index 48eb596a..42625d74 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -82,7 +82,7 @@ providers: Conversion options work identically for both local and remote processing. -**Note:** When using `chunker: docling-serve`, OCR options (`do_ocr`, `force_ocr`, `ocr_engine`, `ocr_lang`) from `conversion_options` are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail—set `do_ocr: false` to disable OCR entirely. +**Note:** When using `chunker: docling-serve`, OCR options (`do_ocr`, `force_ocr`, `ocr_engine`, `ocr_lang`) from `conversion_options` are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail. Set `do_ocr: false` to disable OCR entirely. ### Conversion Options @@ -135,7 +135,7 @@ conversion_options: - **images_scale**: Scale factor for extracted images. Higher values = better quality but larger size. Typical range: 1.0-3.0. - **generate_page_images**: When `true` (default), rendered images of each PDF page are included in the document. Required for `visualize_chunk()` to show visual grounding. When `false`, page images are excluded to reduce document size. -- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to docling-local; see [Remote processing](../remote-processing.md#html-image-fetching) for the docling-serve limitation. +- **fetch_remote_images**: When `true` (default), HTML and Markdown inputs have their external `` URLs fetched and stored as picture bytes. Set `false` for air-gapped ingest. Applies only to docling-local. See [Remote processing](../remote-processing.md#html-image-fetching) for the docling-serve limitation. #### External image fetching @@ -146,9 +146,9 @@ For HTML and Markdown inputs, docling fetches images referenced by URL when `fet - **Timeouts**: 5 s connect, 30 s read. - **SVGs are skipped** (PIL cannot rasterize them). - **`data:` URIs** are decoded inline (no network). -- **`file://` URIs** are *not* fetched — `enable_local_fetch` stays off to keep the SSRF surface narrow for arbitrary HTML/MD content. +- **`file://` URIs** are *not* fetched. `enable_local_fetch` stays off to keep the SSRF surface narrow for arbitrary HTML/MD content. -Per-image failures (404, timeout, oversized, unreadable) leave that picture as a placeholder with `picture_data=NULL` — the rest of the document still ingests. +Per-image failures (404, timeout, oversized, unreadable) leave that picture as a placeholder with `picture_data=NULL`. The rest of the document still ingests. **Scope of conversion options across formats:** @@ -171,7 +171,7 @@ Per-image failures (404, timeout, oversized, unreadable) leave that picture as a | `description` | on | yes | yes | | `image` (default) | on | yes | no | -Use `none` when you don't need picture content (e.g. very large reference manuals where RAM is tight); use `description` to weave VLM-generated text into chunk content and keep bytes for later; use `image` (default) to keep bytes without paying the VLM cost. The prompt is configurable under `prompts.picture_description` — see [Prompts](prompts.md). +Use `none` when you don't need picture content (e.g. very large reference manuals where RAM is tight). Use `description` to weave VLM-generated text into chunk content and keep bytes for later. Use `image` (default) to keep bytes without paying the VLM cost. The prompt is configurable under `prompts.picture_description`. See [Prompts](prompts.md). ```yaml processing: @@ -186,7 +186,7 @@ processing: ``` !!! warning "Breaking change" - `processing.conversion_options.picture_description.enabled` is replaced by `processing.pictures`. Map `enabled: true` → `pictures: description`, `enabled: false` → `pictures: image`. The pre-April-30 `generate_picture_images` flag also no longer exists; use `pictures: none` for the old opt-out. + `processing.conversion_options.picture_description.enabled` is replaced by `processing.pictures`. Map `enabled: true` → `pictures: description`, `enabled: false` → `pictures: image`. The pre-April-30 `generate_picture_images` flag also no longer exists. Use `pictures: none` for the old opt-out. **Switching modes on an existing database** doesn't require reingesting when the bytes are already stored: @@ -194,7 +194,7 @@ processing: - `description` → `image`: `haiku-rag rebuild --rechunk` recomposes chunk text from the stripped docling blob without descriptions. - Switching to/from `none`: a full reingest is needed since the bytes either weren't stored or need to be discarded. -When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag — see [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve). +When using `converter: docling-serve`, the VLM is invoked from docling-serve rather than haiku.rag. See [Remote processing](../remote-processing.md#vlm-picture-description-with-docling-serve). #### Pictures × embedder × QA model: how the pieces compose @@ -219,9 +219,9 @@ Three independent settings drive ingest, retrieval, and QA: **What QA receives** at search time: - `qa.model.vision: false` — text chunks only (descriptions, when present, answer figure questions in prose). -- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`; the model reads figures directly. Requires `pictures != none` so the bytes exist. +- `qa.model.vision: true` — text chunks + raw picture bytes via `BinaryContent`. The model reads figures directly. Requires `pictures != none` so the bytes exist. -`qa.model.vision` is independent of ingestion — flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI; default `false` is the safe choice. +`qa.model.vision` is independent of ingestion. Flipping it never requires reingesting. Setting `vision: true` against a text-only model causes silent acceptance and confabulation on Ollama and a 400 on OpenAI. Default `false` is the safe choice. **Recommended combinations:** @@ -282,12 +282,12 @@ processing: When `auto_title` is enabled, haiku.rag attempts to extract a title for each document during ingestion using a two-tier approach: -1. **Structural extraction** (free, no model calls): Scans the DoclingDocument for semantic labels — HTML `` tags, `<h1>` headings, PDF title blocks, and section headers +1. **Structural extraction** (free, no model calls): Scans the DoclingDocument for semantic labels (HTML `<title>` tags, `<h1>` headings, PDF title blocks, and section headers) 2. **LLM fallback**: When no structural title is found (e.g., plain text), generates a title using the configured `title_model` Priority order: HTML `<title>` (furniture layer) → h1/PDF title (body layer) → first section header → LLM generation. -Explicit titles passed via `title=` parameter always take precedence and are never overridden. When updating documents, existing titles are preserved — auto-generation only applies to untitled documents. +Explicit titles passed via `title=` parameter always take precedence and are never overridden. When updating documents, existing titles are preserved. Auto-generation only applies to untitled documents. To generate titles for existing untitled documents, use [`rebuild --title-only`](../cli.md#rebuild-database). @@ -379,8 +379,8 @@ monitor: allow_http: "true" ``` -Each entry is independent — own poll interval, own include/ignore patterns, own `delete_orphans` setting, own credentials. Omit `storage_options` to fall back to the AWS default credential chain (env vars, IAM role, AWS profile). +Each entry is independent: own poll interval, own include/ignore patterns, own `delete_orphans` setting, own credentials. Omit `storage_options` to fall back to the AWS default credential chain (env vars, IAM role, AWS profile). -The dict shape matches `lancedb.storage_options` — the same Rust `object_store` library is used by both, so credentials configured for the LanceDB backend can be copy-pasted here. +The dict shape matches `lancedb.storage_options`. The same Rust `object_store` library is used by both, so credentials configured for the LanceDB backend can be copy-pasted here. See [Server Mode → S3 / Object Storage Monitoring](../server.md#s3-object-storage-monitoring) for behaviour details (ETag-based change detection, orphan-deletion scope, CLI `add-src s3://…`). diff --git a/docs/configuration/prompts.md b/docs/configuration/prompts.md index 248e3d32..15152aee 100644 --- a/docs/configuration/prompts.md +++ b/docs/configuration/prompts.md @@ -24,7 +24,7 @@ The `domain_preamble` field provides **domain context** prepended to the rag and - Clarify domain-specific terminology - Provide context that helps the model interpret ambiguous queries -**Important:** `domain_preamble` is for domain context, not behavioral instructions. Descriptions of subject matter, terminology, and content scope belong here. Behavioral guidance (tone, response style, formatting rules) lives in the skill's SKILL.md — fork the skill via `haiku-rag create-skill` to customize behavior. +**Important:** `domain_preamble` is for domain context, not behavioral instructions. Descriptions of subject matter, terminology, and content scope belong here. Behavioral guidance (tone, response style, formatting rules) lives in the skill's SKILL.md. Fork the skill via `haiku-rag create-skill` to customize behavior. **Example:** diff --git a/docs/configuration/providers.md b/docs/configuration/providers.md index c3a903d9..0bd5ca31 100644 --- a/docs/configuration/providers.md +++ b/docs/configuration/providers.md @@ -22,7 +22,7 @@ qa: **Available options:** -- **temperature**: Sampling temperature (0.0-1.0+). Defaults vary by task: 0.3 for QA and title generation; 0.0 for analysis and picture description. +- **temperature**: Sampling temperature (0.0-1.0+). Defaults vary by task: 0.3 for QA and title generation, 0.0 for analysis and picture description. - Lower (0.0-0.3): Deterministic, focused responses - Medium (0.4-0.7): Balanced - Higher (0.8-1.0+): Creative, varied responses @@ -67,7 +67,7 @@ See the [Pydantic AI thinking documentation](https://ai.pydantic.dev/thinking/) The `extra_body` setting takes a dict that haiku.rag forwards verbatim to the underlying model SDK as `ModelSettings.extra_body`. Use it to reach provider-specific keys that haiku.rag does not model with a dedicated field. -**Example — disable Qwen3 thinking on vLLM:** +**Example: disable Qwen3 thinking on vLLM:** ```yaml qa: @@ -225,7 +225,7 @@ embeddings: base_url: http://localhost:8000/v1 ``` -Tested with `Qwen/Qwen3-VL-Embedding-8B` (4096-dim) and `jinaai/jina-embeddings-v4` (2048-dim). Run vLLM separately; haiku.rag adds no Python ML dependencies for this path. Text inputs use the standard OpenAI `input` field; image inputs use vLLM's `messages`-with-`image_url` superset, transparently to the caller. +Tested with `Qwen/Qwen3-VL-Embedding-8B` (4096-dim) and `jinaai/jina-embeddings-v4` (2048-dim). Run vLLM separately. haiku.rag adds no Python ML dependencies for this path. Text inputs use the standard OpenAI `input` field. Image inputs use vLLM's `messages`-with-`image_url` superset, transparently to the caller. Picture chunks for retrieval are emitted at ingest under any embedder reporting `supports_images=True`. See [Picture Handling](processing.md#picture-handling). @@ -464,7 +464,7 @@ reranking: ### Cross-Encoder (sentence-transformers) -Run any HuggingFace cross-encoder reranker in-process via `sentence-transformers` — no separate server required. Useful when you want a specific model (BGE, Qwen3-Reranker, MS-MARCO MiniLM, etc.) without running vLLM. +Run any HuggingFace cross-encoder reranker in-process via `sentence-transformers`. No separate server required. Useful when you want a specific model (BGE, Qwen3-Reranker, MS-MARCO MiniLM, etc.) without running vLLM. Install the extra: diff --git a/docs/configuration/qa.md b/docs/configuration/qa.md index 8d1b4899..e2da510e 100644 --- a/docs/configuration/qa.md +++ b/docs/configuration/qa.md @@ -13,7 +13,7 @@ search: - **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and QA. Default: 10 - **max_context_chars**: Hard limit on total characters in expanded content. Default: 10000. -Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers) — this naturally crosses into adjacent sections until the budget is filled. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded. +Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers). This naturally crosses into adjacent sections until the budget is filled. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded. !!! note "Reranking behavior" When a reranker is configured, search automatically retrieves 10x the requested limit, then reranks to return the final count. This improves result quality without requiring you to adjust `limit`. @@ -34,7 +34,7 @@ qa: ``` - **model**: LLM configuration (see [Providers](providers.md#model-settings)) -- **model.vision**: Set to `true` for vision-capable models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, …). The skill's `search` tool only attaches picture bytes (`BinaryContent`) to its `ToolReturn` when this is `true`; otherwise picture bytes are withheld. See [Pictures × embedder × QA model](processing.md#pictures-embedder-qa-model-how-the-pieces-compose) for the full matrix. +- **model.vision**: Set to `true` for vision-capable models (`qwen2.5vl`, `qwen3.6`, `gpt-4o`, `claude-sonnet`, …). The skill's `search` tool only attaches picture bytes (`BinaryContent`) to its `ToolReturn` when this is `true`, otherwise picture bytes are withheld. See [Pictures × embedder × QA model](processing.md#pictures-embedder-qa-model-how-the-pieces-compose) for the full matrix. - **max_searches**: Maximum number of search tool calls the rag skill can make per question (default: 3) ## Analysis Configuration diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index 849c76fc..43c3fbfe 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -98,13 +98,13 @@ lancedb: - **Object storage** (`s3://`, `gs://`, `az://`, `hdfs://`): Uses `storage_options` for credentials and endpoint configuration. Authentication can also be provided via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, etc.) or cloud provider SDK defaults (AWS CLI, Azure CLI, gcloud). - **S3-compatible stores** (MinIO, Tigris, etc.): Set `endpoint` in `storage_options`. When using `http://` endpoints, also set `allow_http: "true"`. -The `storage_options` keys are case-insensitive and passed directly to the underlying object store library. Available keys depend on the backend — see the [LanceDB storage docs](https://lancedb.com/docs/storage/) for details. +The `storage_options` keys are case-insensitive and passed directly to the underlying object store library. Available keys depend on the backend. See the [LanceDB storage docs](https://lancedb.com/docs/storage/) for details. **Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization and vector indexing are still performed normally. ### Deployment Pattern: One Writer, Many Readers -LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state — this is a LanceDB property, not something `haiku.rag` enforces. +LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state. This is a LanceDB property, not something `haiku.rag` enforces. The recommended layout for production is "different buckets, same account, separate IAM roles per process": diff --git a/docs/custom-pipelines.md b/docs/custom-pipelines.md index e3c48907..3321e4ed 100644 --- a/docs/custom-pipelines.md +++ b/docs/custom-pipelines.md @@ -1,6 +1,6 @@ # Custom Processing Pipelines -haiku.rag provides processing primitives that let you build custom document pipelines. Use these when you need control over conversion, chunking, or embedding—for example, to preprocess content, use external services, or implement custom chunking logic. +haiku.rag provides processing primitives that let you build custom document pipelines. Use these when you need control over conversion, chunking, or embedding (for example, to preprocess content, use external services, or implement custom chunking logic). ## When to Use Custom Pipelines @@ -136,7 +136,7 @@ assert embedded_chunks[0].embedding is not None ## Contextualize (for custom embedders) -`contextualize()` is a lower-level utility that prepares chunk content for embedding by prepending section headings. You only need this when implementing custom embedding logic—`embed_chunks()` already calls it internally. +`contextualize()` is a lower-level utility that prepares chunk content for embedding by prepending section headings. You only need this when implementing custom embedding logic. `embed_chunks()` already calls it internally. ```python from haiku.rag.embeddings import contextualize diff --git a/docs/python.md b/docs/python.md index 210e8d03..1380ed77 100644 --- a/docs/python.md +++ b/docs/python.md @@ -159,7 +159,7 @@ await client.update_document(document_id=doc.id, chunks=custom_chunks) - Updates to only `metadata` or `title` skip re-chunking - Updates to `content` trigger re-chunking and re-embedding -- Custom `chunks` with embeddings are stored as-is; missing embeddings are generated automatically +- Custom `chunks` with embeddings are stored as-is. Missing embeddings are generated automatically ### Deleting Documents @@ -256,7 +256,7 @@ results = await client.search( ### Image queries -`client.search()` accepts an image instead of a text query when the configured embedder is multimodal (e.g. `provider: vllm` against a vision-language embedding model). The image is embedded once and the chunks table is searched vector-only — full-text search and reranking don't apply without a text query. +`client.search()` accepts an image instead of a text query when the configured embedder is multimodal (e.g. `provider: vllm` against a vision-language embedding model). The image is embedded once and the chunks table is searched vector-only. Full-text search and reranking don't apply without a text query. ```python from PIL import Image @@ -291,7 +291,7 @@ for result in expanded_results: print(f"Expanded content: {result.content}") ``` -Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers) — this naturally crosses into adjacent sections until the budget is filled. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded. +Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers). This naturally crosses into adjacent sections until the budget is filled. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded. Configuration: @@ -419,7 +419,7 @@ async for doc_id in client.rebuild_database(mode=RebuildMode.RECHUNK): async for doc_id in client.rebuild_database(mode=RebuildMode.EMBED_ONLY): print(f"Processed document {doc_id}") -# Add VLM picture descriptions to an existing database — runs the VLM +# Add VLM picture descriptions to an existing database. Runs the VLM # over already-stored picture bytes, patches descriptions into the # docling blob, then re-chunks + re-embeds. Requires # processing.pictures='description' in the config. @@ -433,7 +433,7 @@ async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS): - `RebuildMode.RECHUNK` - Re-chunk from existing document content, re-embed - `RebuildMode.EMBED_ONLY` - Keep existing chunks, only regenerate embeddings - `RebuildMode.TITLE_ONLY` - Generate titles for untitled documents (no re-chunking or re-embedding) -- `RebuildMode.DESCRIPTIONS` - Run the VLM over picture bytes already stored on `document_items.picture_data`, patch descriptions into the docling blob, re-chunk + re-embed. Skips the docling parse entirely. Idempotent — pictures already carrying `meta.description.text` are not re-described, so the operation is safe to re-run. +- `RebuildMode.DESCRIPTIONS` - Run the VLM over picture bytes already stored on `document_items.picture_data`, patch descriptions into the docling blob, re-chunk + re-embed. Skips the docling parse entirely. Idempotent: pictures already carrying `meta.description.text` are not re-described, so the operation is safe to re-run. ### Generating Titles @@ -445,7 +445,7 @@ if title: await client.update_document(document_id=doc.id, title=title) ``` -Uses the same two-tier approach as automatic ingestion: structural extraction from DoclingDocument metadata first, with LLM fallback via `processing.title_model`. Unlike ingestion, this method does not catch exceptions — if the LLM call fails, the error propagates. +Uses the same two-tier approach as automatic ingestion: structural extraction from DoclingDocument metadata first, with LLM fallback via `processing.title_model`. Unlike ingestion, this method does not catch exceptions. If the LLM call fails, the error propagates. To batch-generate titles for all untitled documents, use `RebuildMode.TITLE_ONLY`: @@ -462,4 +462,4 @@ Document create and update operations take a snapshot of table versions before a - Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, and internal rebuild/update flows. - Scope: Both document rows and all associated chunks are rolled back together. -- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency; rollbacks occur immediately during the failing operation and are not impacted. +- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency. Rollbacks occur immediately during the failing operation and are not impacted. diff --git a/docs/remote-processing.md b/docs/remote-processing.md index 71d14783..be8a89b2 100644 --- a/docs/remote-processing.md +++ b/docs/remote-processing.md @@ -25,8 +25,8 @@ docling-serve is a REST API service that provides: - Processing large volumes of documents - Working with complex PDFs requiring OCR - Running in production environments -- Want to separate compute-intensive tasks -- Need to scale document processing independently +- Separating compute-intensive tasks +- Scaling document processing independently ## Setup @@ -136,7 +136,7 @@ processing: ## HTML Image Fetching -docling-serve does **not** fetch external `<img src="https://...">` URLs in HTML inputs. The `ConvertDocumentsOptions` API exposes no equivalent of docling-local's `HTMLBackendOptions.fetch_images` / `enable_remote_fetch`, and the server-side `DoclingConverterManager` registers `format_options` only for PDF and IMAGE — HTML falls through to docling's defaults (`fetch_images=False`). +docling-serve does **not** fetch external `<img src="https://...">` URLs in HTML inputs. The `ConvertDocumentsOptions` API exposes no equivalent of docling-local's `HTMLBackendOptions.fetch_images` / `enable_remote_fetch`, and the server-side `DoclingConverterManager` registers `format_options` only for PDF and IMAGE. HTML falls through to docling's defaults (`fetch_images=False`). Consequence: ingesting HTML with external image references through docling-serve produces picture items with `picture_data=NULL`. The same input through docling-local fetches the bytes (subject to the SSRF / size / timeout guards documented in [Configuration → External image fetching](configuration/processing.md#external-image-fetching)). diff --git a/docs/server.md b/docs/server.md index 90296ada..ecea6f56 100644 --- a/docs/server.md +++ b/docs/server.md @@ -151,7 +151,7 @@ monitor: allow_http: "true" ``` -Then start the server with `--monitor` — the same flag enables both local-directory and S3 watchers: +Then start the server with `--monitor` (the same flag enables both local-directory and S3 watchers): ```bash haiku-rag serve --monitor @@ -161,7 +161,7 @@ Each entry in `monitor.s3` runs as its own polling task. On every sweep the watc ### Credentials -`storage_options` follows the same convention as `lancedb.storage_options` — the dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile). +`storage_options` follows the same convention as `lancedb.storage_options`. The dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile). ### Orphan deletion scope @@ -169,4 +169,4 @@ Each entry in `monitor.s3` runs as its own polling task. On every sweep the watc ## One-off ingestion -`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool — see [CLI → Add Documents](cli.md#add-documents). +`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool. See [CLI → Add Documents](cli.md#add-documents). diff --git a/docs/skills/analysis.md b/docs/skills/analysis.md index 434b7e80..2012207a 100644 --- a/docs/skills/analysis.md +++ b/docs/skills/analysis.md @@ -2,7 +2,7 @@ Plain RAG (search → cite → answer) works for questions whose answer sits in a chunk or two: "Who wrote this?", "What does X say about Y?". It struggles when the answer requires touching the whole corpus, reading a specific section in full, or doing arithmetic on the data. -The analysis skill (`rag-analysis`) gives the agent a second tool — `execute_code` — that runs Python in a sandboxed interpreter against a structured view of your documents. The agent can search, read, count, slice, and compare without leaving the tool call. Citations work the same way as the rag skill. +The analysis skill (`rag-analysis`) gives the agent a second tool (`execute_code`) that runs Python in a sandboxed interpreter against a structured view of your documents. The agent can search, read, count, slice, and compare without leaving the tool call. Citations work the same way as the rag skill. `client.analyze`, `haiku-rag analyze`, the MCP `analyze` tool, and the chat TUI (when `-s analysis` is enabled) all run through this skill. @@ -14,7 +14,7 @@ Reach for the analysis skill when the question needs more than a search: - **Section-scoped reading.** "Summarize Section 5 of paper Y." - **Structural comparison.** "Do both papers have an Experimental Results section?" - **Computation on retrieved data.** "What's the average revenue across these quarterly reports?" -- **Multi-step chains.** Search, filter the results in Python, search again, aggregate — all in one tool call. +- **Multi-step chains.** Search, filter the results in Python, search again, aggregate, all in one tool call. For everyday Q&A, the [RAG skill](rag.md) is faster and cheaper. Attach both and the agent routes. @@ -22,9 +22,9 @@ For everyday Q&A, the [RAG skill](rag.md) is faster and cheaper. Attach both and Two things make the agent's programs short and the resulting analyses tractable: -1. **Search and document listing are awaitable inside the code.** `await search(query)` returns the same hits the rag skill sees — chunk IDs, text, source metadata, picture refs. The agent can immediately filter, sort, count, or follow up with another search without exiting the tool call. +1. **Search and document listing are awaitable inside the code.** `await search(query)` returns the same hits the rag skill sees: chunk IDs, text, source metadata, picture refs. The agent can immediately filter, sort, count, or follow up with another search without exiting the tool call. -2. **Every document is mounted as a virtual filesystem at `/documents/{id}/`.** The agent reads four files per document: identifiers and metadata, full text, a list of structured items (paragraphs, tables, figures, headings), and a section tree built from the document's headings. The structure exposes what search alone hides — the agent can navigate from a search hit to the section it lives in, slice a single section instead of pulling the whole document, or scan a document's text directly when keyword precision matters. +2. **Every document is mounted as a virtual filesystem at `/documents/{id}/`.** The agent reads four files per document: identifiers and metadata, full text, a list of structured items (paragraphs, tables, figures, headings), and a section tree built from the document's headings. The structure exposes what search alone hides. The agent can navigate from a search hit to the section it lives in, slice a single section instead of pulling the whole document, or scan a document's text directly when keyword precision matters. A search hit is always a starting point. The agent reads structure around it, drills into the right section, and cites the chunks it actually used. Chunk IDs from search results and chunk IDs surfaced through the VFS are both accepted by `cite`. @@ -114,7 +114,7 @@ The agent routes Q&A to the rag skill and computational questions to rag-analysi ## What the agent actually writes -You don't write these programs yourself — the agent does, inside `execute_code`. Seeing the shape helps when you tune prompts, debug a run via `AnalysisState.executions`, or design a custom skill. +You don't write these programs yourself. The agent does, inside `execute_code`. Seeing the shape helps when you tune prompts, debug a run via `AnalysisState.executions`, or design a custom skill. **Aggregate across the corpus.** *"How many documents mention security vulnerabilities?"* diff --git a/docs/skills/custom.md b/docs/skills/custom.md index f1d433ea..2ffd06f6 100644 --- a/docs/skills/custom.md +++ b/docs/skills/custom.md @@ -1,6 +1,6 @@ # Custom Skills -The two skills haiku.rag ships work against any LanceDB database. When you want a *domain-specific* skill that bundles its own data, prompt, and tool surface — for example, a "recipes" skill that knows about cooking and ships with a recipes database — generate one with `haiku-rag create-skill`. +The two skills haiku.rag ships work against any LanceDB database. When you want a *domain-specific* skill that bundles its own data, prompt, and tool surface (for example, a "recipes" skill that knows about cooking and ships with a recipes database), generate one with `haiku-rag create-skill`. The generated package is a regular pip-installable Python package that registers as a `haiku.skills` entry point. Any haiku.skills-aware host (haiku.skills CLI, your own agent, the AG-UI adapter) discovers it automatically. @@ -11,7 +11,7 @@ The generated package is a regular pip-installable Python package that registers - You want to ship a knowledge base plus its prompt as one distributable unit. - You're running multiple skills against different databases in the same agent. -If you just want to point a haiku.rag database at your own model and prompt, configure `haiku.rag.yaml` and use the built-in `rag` skill — no custom package needed. +If you just want to point a haiku.rag database at your own model and prompt, configure `haiku.rag.yaml` and use the built-in `rag` skill. No custom package needed. ## Generate @@ -51,7 +51,7 @@ haiku-skills chat --use-entrypoints --skill recipes `cite`, `execute_code`, `get_document`, `list_documents`, `search`. -Drop `execute_code` from `--tools` if the skill shouldn't run sandboxed Python — that gives you a search-and-cite-only skill with no analysis capabilities. +Drop `execute_code` from `--tools` if the skill shouldn't run sandboxed Python. That gives you a search-and-cite-only skill with no analysis capabilities. ## Anatomy of a generated skill @@ -66,7 +66,7 @@ Drop `execute_code` from `--tools` if the skill shouldn't run sandboxed Python └── haiku.rag.yaml # Optional config (only if --config-file passed) ``` -- **`SKILL.md`** carries the instruction prompt the agent will follow. The frontmatter includes the skill name and description; everything below is the prompt body. Edit this to change behavior. +- **`SKILL.md`** carries the instruction prompt the agent will follow. The frontmatter includes the skill name and description. Everything below is the prompt body. Edit this to change behavior. - **`__init__.py`** exposes `create_skill()` (the entry point) and `visualize_chunk()` for rendering visual grounding. - **`assets/{name}.lancedb/`** is the database, shipped inside the package. - **`assets/haiku.rag.yaml`** (optional) pins provider settings the skill needs. @@ -119,6 +119,6 @@ Each skill maintains state under its own namespace (`recipes`, `medic`, …), so ## Writing a skill from scratch -`create-skill` is the convenience path. If you need full control over the tools, state model, or instruction loading, write the skill against [haiku.skills](https://github.com/ggozad/haiku.skills) directly. The generated package in `{name}_skill/__init__.py` is a good reference — it composes haiku.rag's `_tools` factory with a `haiku.skills.Skill` and registers under the `haiku.skills` entry point group in `pyproject.toml`. +`create-skill` is the convenience path. If you need full control over the tools, state model, or instruction loading, write the skill against [haiku.skills](https://github.com/ggozad/haiku.skills) directly. The generated package in `{name}_skill/__init__.py` is a good reference. It composes haiku.rag's `_tools` factory with a `haiku.skills.Skill` and registers under the `haiku.skills` entry point group in `pyproject.toml`. See the haiku.skills repository for the full Skill contract. diff --git a/docs/skills/rag.md b/docs/skills/rag.md index b84a0c52..14ee8418 100644 --- a/docs/skills/rag.md +++ b/docs/skills/rag.md @@ -8,7 +8,7 @@ The `rag` skill answers questions over a knowledge base with hybrid search, stru - You want citations under every answer. - You're building a Q&A agent, a documentation chatbot, or any RAG-style integration. -If the question requires *computation* over the corpus (counts, aggregates, comparisons, section-scoped reading), reach for the [Analysis skill](analysis.md) instead — or attach both. +If the question requires *computation* over the corpus (counts, aggregates, comparisons, section-scoped reading), reach for the [Analysis skill](analysis.md) instead, or attach both. ## Tools @@ -32,7 +32,7 @@ class RAGState(BaseModel): ``` - **citation_index** — All citations indexed by chunk ID. Accumulates across invocations so historical chunk IDs stay resolvable in UI scrollback. -- **citations** — Chunk IDs registered via `cite` during the current invocation. Deduplicated; cleared at the start of each invocation. +- **citations** — Chunk IDs registered via `cite` during the current invocation. Deduplicated, cleared at the start of each invocation. - **document_filter** — SQL WHERE clause applied to `search` and `list_documents`. Persists across invocations. - **searches** — Search results keyed by query string. Cleared at the start of each invocation. @@ -51,7 +51,7 @@ skill = create_skill(db_path="my.lancedb") ## Examples -### Minimal QA agent +### Minimal agent ```python from haiku.rag.skills.rag import create_skill diff --git a/docs/tuning.md b/docs/tuning.md index 571af8ed..904ded76 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -4,13 +4,13 @@ How to adjust haiku.rag's pipeline for better retrieval and answer quality. For ## Pipeline Overview -Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is highest-leverage — if the LLM never sees the right chunks, no prompt or model change will help. +Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is the highest-leverage stage. If the LLM never sees the right chunks, no prompt or model change will help. ## Tuning Retrieval ### Chunking -`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each; larger chunks provide more surrounding information but dilute relevance signals. On the Wix benchmark, increasing from 256 to 512 tokens raised MAP from 0.43 to 0.45 on plain text — a modest gain that also increases token cost per result. See [Processing](configuration/processing.md#chunk-size) for configuration. +`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each. Larger chunks provide more surrounding information but dilute relevance signals. On the Wix benchmark, increasing from 256 to 512 tokens raised MAP from 0.43 to 0.45 on plain text, a modest gain that also increases token cost per result. See [Processing](configuration/processing.md#chunk-size) for configuration. `chunker_type` selects between `hybrid` (default) and `hierarchical` chunking. Hierarchical chunking preserves the document's heading structure and works better for deeply nested or structured content. See [Chunking Strategies](configuration/processing.md#chunking-strategies). @@ -20,17 +20,17 @@ Larger embedding models produce better representations at the cost of slower ind ### Reranking -When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision — on the Wix benchmark, adding `mxbai-rerank-base-v2` raised MAP from 0.34 to 0.39 on HTML content. See [Search Settings](configuration/qa.md#search-settings) for how reranking integrates with search. +When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision. On the Wix benchmark, adding `mxbai-rerank-base-v2` raised MAP from 0.34 to 0.39 on HTML content. See [Search Settings](configuration/qa.md#search-settings) for how reranking integrates with search. ### Search Settings `limit` controls how many results reach the LLM. More candidates improve recall but increase token usage. See [Search Settings](configuration/qa.md#search-settings). -Context expansion is automatic and section-aware — search results are expanded to include surrounding content from the same document section. For structured documents, expansion stays within section boundaries and filters noise (footnotes, page headers). For unstructured documents, expansion grows outward until the character budget is filled. `max_context_chars` caps expansion to prevent context bloat. +Context expansion is automatic and section-aware. Search results are expanded to include surrounding content from the same document section. For structured documents, expansion stays within section boundaries and filters noise (footnotes, page headers). For unstructured documents, expansion grows outward until the character budget is filled. `max_context_chars` caps expansion to prevent context bloat. ## Tuning Generation -Model and temperature selection affect answer quality directly — see [Providers](configuration/providers.md#model-settings) for options. +Model and temperature selection affect answer quality directly. See [Providers](configuration/providers.md#model-settings) for options. `domain_preamble` prepends domain context to the rag and rag-analysis skill instructions. Use it to describe what the knowledge base contains and clarify domain-specific terminology. See [Prompt Customization](configuration/prompts.md). @@ -38,8 +38,8 @@ Model and temperature selection affect answer quality directly — see [Provider | Change | Rebuild required? | |--------|:-:| -| `chunk_size`, `chunker_type`, `chunking_merge_peers` | Yes — `haiku-rag rebuild` | -| Embedding model | Yes — `haiku-rag rebuild` | +| `chunk_size`, `chunker_type`, `chunking_merge_peers` | Yes (run `haiku-rag rebuild`) | +| Embedding model | Yes (run `haiku-rag rebuild`) | | Search settings, reranking, prompts | No | ## Inspector