Update docs, include tuning document

2025-12-09 12:55:28 +02:00 · 2025-12-09 12:55:28 +02:00 · 52acdcb56a
commit 52acdcb56a
parent 5a593e928e
7 changed files with 361 additions and 53 deletions
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@ -101,7 +101,7 @@ research:
 search:
  limit: 5                     # Default number of results to return
  context_radius: 0            # DocItems before/after to include for text content
-  max_context_items: 25        # Maximum items in expanded context
+  max_context_items: 10        # Maximum items in expanded context
  max_context_chars: 10000     # Maximum characters in expanded context
  vector_index_metric: cosine  # cosine, l2, or dot
  vector_refine_factor: 30
@ -191,7 +191,7 @@ This is useful for:

 For detailed configuration of specific topics, see:

- **[Providers](providers.md)** - Model settings and provider-specific configuration (embeddings, QA, reranking)
- **[QA and Research](qa-research.md)** - Question answering and research workflow configuration
- **[Storage](storage.md)** - Database, remote storage, and vector indexing
+- **[Providers](providers.md)** - Model settings and provider-specific configuration (embeddings, reranking)
+- **[Search and Question Answering](qa-research.md)** - Search settings, question answering, and research workflows
 - **[Document Processing](processing.md)** - Document conversion, chunking, and file monitoring
+- **[Storage](storage.md)** - Database, remote storage, and vector indexing
--- a/docs/configuration/processing.md
+++ b/docs/configuration/processing.md
@ -139,7 +139,7 @@ processing:
  chunk_size: 256  # Maximum tokens per chunk
 ```

-Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](storage.md#search-settings).
+Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](qa-research.md#search-settings).

 ## File Monitoring

--- a/docs/configuration/qa-research.md
+++ b/docs/configuration/qa-research.md
@ -1,4 +1,26 @@
-# QA and Research Configuration
+# Search and Question Answering
+
+## Search Settings
+
+Configure search behavior and context expansion:
+
+```yaml
+search:
+  limit: 5                     # Default number of results to return
+  context_radius: 0            # DocItems before/after to include for text content
+  max_context_items: 10        # Maximum items in expanded context
+  max_context_chars: 10000     # Maximum characters in expanded context
+```
+
+- **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, QA, and research workflows. Default: 5
+- **context_radius**: For text content (paragraphs), includes N DocItems before and after. Set to 0 to disable expansion (default).
+- **max_context_items**: Limits how many document items (paragraphs, list items, etc.) can be included in expanded context. Default: 10.
+- **max_context_chars**: Hard limit on total characters in expanded content. Default: 10000.
+
+Structural content (tables, code blocks, lists) uses type-aware expansion that automatically includes the complete structure regardless of how it was chunked.
+
+!!! note "Reranking behavior"
+    When a reranker is configured, search automatically retrieves 10x the requested limit, then reranks to return the final count. This improves result quality without requiring you to adjust `limit`.

 ## Question Answering Configuration

@ -43,29 +65,3 @@ research:
 - **max_concurrency**: Sub-questions searched in parallel per iteration (default: 1)

 The research workflow plans sub-questions, searches in parallel batches, evaluates findings, and iterates until reaching the confidence threshold or max iterations.
-
-## AG-UI Server Configuration
-
-Configure the AG-UI HTTP server for streaming graph execution events:
-
-```yaml
-agui:
-  host: "0.0.0.0"
-  port: 8000
-  cors_origins: ["*"]
-  cors_credentials: true
-  cors_methods: ["GET", "POST", "OPTIONS"]
-  cors_headers: ["*"]
-```
-
-Start the AG-UI server with:
-
-```bash
-haiku-rag serve --agui
-```
-
-The server exposes:
- `GET /health` - Health check endpoint
- `POST /v1/agent/stream` - Research graph streaming endpoint (Server-Sent Events)
-
-See [Server Mode](../server.md) for more details.
--- a/docs/configuration/storage.md
+++ b/docs/configuration/storage.md
@ -57,33 +57,17 @@ haiku.rag intelligently handles database creation based on operation type:

 This prevents the common mistake where a search query accidentally creates an empty database. To initialize your database, simply add your first document using `haiku-rag add` or `haiku-rag add-src`.

-## Search Settings
+## Vector Indexing

-Configure search behavior and context expansion:
+Configure vector search settings:

 ```yaml
 search:
-  limit: 5                     # Default number of results to return
-  context_radius: 0            # DocItems before/after to include for text content
-  max_context_items: 25        # Maximum items in expanded context
-  max_context_chars: 10000     # Maximum characters in expanded context
  vector_index_metric: cosine  # cosine, l2, or dot
  vector_refine_factor: 30     # Re-ranking factor for accuracy
 ```

- **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and API. Default: 5
-
-### Context Expansion
-
-Context expansion enriches search results with surrounding content from the source document:
-
- **context_radius**: For text content (paragraphs), includes N DocItems before and after. Set to 0 to disable expansion (default).
- **max_context_items**: Limits how many document items (paragraphs, list items, etc.) can be included in expanded context.
- **max_context_chars**: Hard limit on total characters in expanded content.
-
-Structural content (tables, code blocks, lists) uses type-aware expansion that automatically includes the complete structure regardless of how it was chunked. For example, if a table was split across multiple chunks, expansion retrieves the complete table.
-
-### Vector Indexing
+For search behavior settings (`limit`, `context_radius`, `max_context_items`, `max_context_chars`), see [QA and Research](qa-research.md#search-settings).

 - **vector_index_metric**: Distance metric for vector similarity:
  - `cosine`: Cosine similarity (default, best for most embeddings)
--- a/docs/server.md
+++ b/docs/server.md
@ -143,9 +143,16 @@ agui:
  port: 8000
  cors_origins: ["*"]
  cors_credentials: true
+  cors_methods: ["GET", "POST", "OPTIONS"]
+  cors_headers: ["*"]
 ```

-See [Configuration](configuration/qa-research.md#ag-ui-server-configuration) for all available options.
+- **host**: Bind address (default: `0.0.0.0`)
+- **port**: Server port (default: `8000`)
+- **cors_origins**: Allowed CORS origins (default: `["*"]`)
+- **cors_credentials**: Allow credentials in CORS requests (default: `true`)
+- **cors_methods**: Allowed HTTP methods (default: `["GET", "POST", "OPTIONS"]`)
+- **cors_headers**: Allowed headers (default: `["*"]`)

 ### Using the Streaming Endpoints

--- a/docs/tuning.md
+++ b/docs/tuning.md
@ -0,0 +1,320 @@
+# Tuning haiku.rag for Your Corpus
+
+This guide explains how to tune haiku.rag settings based on your document corpus characteristics. The right settings depend on your document types, query patterns, and accuracy requirements.
+
+## Key Concepts
+
+### Retrieval vs Generation
+
+RAG has two phases:
+
+1. **Retrieval**: Finding relevant chunks from your corpus
+2. **Generation**: Using those chunks to answer questions
+
+Poor retrieval means the LLM never sees the relevant content, regardless of how good the model is. Tuning retrieval is usually more impactful than tuning generation.
+
+### Recall vs Precision
+
+- **Recall**: What fraction of relevant documents did we find?
+- **Precision**: What fraction of retrieved documents are relevant?
+
+For RAG, recall matters more than precision. Missing a relevant chunk means wrong answers. Including an extra irrelevant chunk just wastes context tokens.
+
+## Search Settings
+
+### `search.limit`
+
+Default number of chunks to retrieve.
+
+```yaml
+search:
+  limit: 5  # Default
+```
+
+**When to increase:**
+
+- Complex questions requiring information from multiple sources
+- Broad topics spread across many documents
+
+**When to decrease:**
+
+- Simple factual questions
+- Highly focused corpus where top results are usually correct
+- Cost-sensitive deployments (fewer chunks = fewer tokens)
+
+**Typical values:** 3-10
+
+### `search.context_radius`
+
+Number of adjacent DocItems to include when expanding search results. Only applies to text content (paragraphs). Tables, code blocks, and lists use structural expansion automatically.
+
+```yaml
+search:
+  context_radius: 0  # Default: no expansion
+```
+
+**When to increase:**
+
+- Answers require surrounding context (definitions, explanations)
+- Chunks are small and queries need more context
+- Documents have strong local coherence (adjacent paragraphs relate)
+
+**When to keep at 0:**
+
+- Large chunks that already contain sufficient context
+- Documents where adjacent content is often unrelated
+- When chunk boundaries align well with semantic units
+
+**Typical values:** 0-3
+
+### `search.max_context_items` and `search.max_context_chars`
+
+Safety limits on context expansion to prevent runaway expansion.
+
+```yaml
+search:
+  max_context_items: 10     # Max DocItems per expanded result
+  max_context_chars: 10000  # Max characters per expanded result
+```
+
+Increase if expansion is being truncated and you need more context. Decrease if expanded results are too long for your LLM context window.
+
+## Processing Settings
+
+### `processing.chunk_size`
+
+Maximum tokens per chunk (using the configured tokenizer).
+
+```yaml
+processing:
+  chunk_size: 256  # Default
+```
+
+**Trade-offs:**
+
+| Smaller chunks (128-256) | Larger chunks (512-1024) |
+|-------------------------|-------------------------|
+| More precise retrieval | Better context per chunk |
+| May miss spanning content | Better recall |
+| More chunks to search | Faster search |
+| Better for specific queries | Better for broad queries |
+
+**Guidance by corpus type:**
+
+- **Technical documentation**: 256-512 (specific lookups)
+- **Long-form articles**: 512-1024 (need context)
+- **FAQs/short answers**: 128-256 (discrete answers)
+- **Code documentation**: 256-512 (function-level)
+
+### `processing.chunker_type`
+
+Chunking strategy.
+
+```yaml
+processing:
+  chunker_type: hybrid  # Default
+```
+
+- **`hybrid`**: Structure-aware with token limits. Best for most documents.
+- **`hierarchical`**: Preserves document hierarchy strictly. Use for highly structured documents where hierarchy matters.
+
+
+### `processing.chunking_merge_peers`
+
+Whether to merge adjacent small chunks that share the same section.
+
+```yaml
+processing:
+  chunking_merge_peers: true  # Default
+```
+
+Keep `true` unless you specifically want very granular chunks. Merging improves embedding quality by ensuring chunks have sufficient context.
+
+## Embedding Settings
+
+### Model Selection
+
+Embedding model choice significantly impacts retrieval quality.
+
+```yaml
+embeddings:
+  model:
+    provider: ollama
+    name: qwen3-embedding:4b
+    vector_dim: 2560
+```
+
+**Considerations:**
+
+- Larger models generally produce better embeddings but are slower
+- Match `vector_dim` to your model's actual output dimension
+- Local models (Ollama) vs API models (OpenAI, VoyageAI) trade-off cost vs quality
+
+### Contextualizing Embeddings
+
+Chunks are embedded with section headings prepended (via `contextualize()`). This improves retrieval by including structural context in the embedding.
+
+If your documents lack clear headings, embeddings will be based on chunk content alone.
+
+## Reranking
+
+Reranking retrieves more candidates than needed, then uses a cross-encoder to re-score them.
+
+```yaml
+reranking:
+  model:
+    provider: mxbai  # or cohere, zeroentropy, vllm
+    name: mxbai-rerank-base-v1
+```
+
+**When to use reranking:**
+
+- Embedding model has limited accuracy
+- Queries are complex or ambiguous
+- You can afford the latency (adds ~100-500ms)
+
+**When to skip reranking:**
+
+- Simple, specific queries
+- High-quality embedding model
+- Latency-sensitive applications
+
+When reranking is enabled, haiku.rag automatically retrieves 10x the requested limit, then reranks to the final count. You don't need to adjust `search.limit` for reranking.
+
+## Tuning Workflow
+
+### 1. Use the Inspector
+
+The inspector is your best tool for understanding how your corpus is chunked and how search behaves:
+
+```bash
+haiku-rag inspect
+```
+
+**What to look for:**
+
+- Browse documents and their chunks to see how content is split
+- Use the search modal (`/`) to test queries and see which chunks are retrieved
+- Press `c` on a chunk to view expanded context - see what additional content would be included with `context_radius > 0`
+- Check chunk sizes - are they too small (fragmented) or too large (unfocused)?
+
+### 2. Test Search Manually
+
+Before changing settings, run searches from the CLI to understand current behavior:
+
+```bash
+# Search and see results
+haiku-rag search "your test query" --limit 10
+
+# Try the QA to see end-to-end behavior
+haiku-rag ask "your question"
+```
+
+### 3. Identify the Bottleneck
+
+- **Relevant chunks not retrieved**: Try larger `search.limit`, smaller `chunk_size`, or a different embedding model
+- **Too many irrelevant chunks**: Try reranking or larger `chunk_size`
+- **Chunks found but answers wrong**: Try `context_radius` expansion or a better QA model
+
+### 4. Test One Change at a Time
+
+```bash
+# After changing chunk_size, rebuild is required
+haiku-rag rebuild
+
+# After changing search settings, no rebuild needed - just test again
+haiku-rag search "your test query"
+```
+
+### 5. Build Dataset-Specific Evaluations
+
+For systematic tuning, create evaluations specific to your corpus. See the `evaluations/` directory in the repository for examples of how to:
+
+- Define test cases with questions and expected answers
+- Run retrieval benchmarks (MRR, MAP)
+- Run QA accuracy benchmarks with LLM judges
+
+Custom evaluations let you measure the impact of configuration changes objectively rather than relying on intuition.
+
+### 6. Consider Your Corpus
+
+| Corpus Type | Suggested Starting Point |
+|-------------|-------------------------|
+| Technical docs | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
+| Legal/contracts | `chunk_size: 512`, `limit: 5`, `context_radius: 2` |
+| News articles | `chunk_size: 512`, `limit: 5`, `context_radius: 0` |
+| Scientific papers | `chunk_size: 256`, `limit: 5`, reranking enabled |
+| FAQs | `chunk_size: 128`, `limit: 5`, `context_radius: 0` |
+| Code repos | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
+
+## Common Issues
+
+### "Relevant content not being retrieved"
+
+1. Check chunk boundaries - is the content split awkwardly?
+2. Try smaller chunks for more granular matching
+3. Increase `search.limit`
+4. Consider a different embedding model
+
+### "Retrieved chunks lack context"
+
+1. Increase `context_radius` for text content
+2. Increase `chunk_size` for more context per chunk
+3. Structural content (tables, code) expands automatically
+
+### "Search is slow"
+
+1. Create a vector index: `haiku-rag create-index`
+2. Reduce `search.limit`
+3. Consider a smaller embedding model
+
+### "QA answers are wrong despite good retrieval"
+
+1. Check if chunks are being truncated by LLM context limits
+2. Try a more capable QA model
+3. Reduce number of chunks or expansion to fit context window
+
+## Example Configurations
+
+### High-Precision Technical Documentation
+
+```yaml
+processing:
+  chunk_size: 256
+  chunker_type: hybrid
+
+search:
+  limit: 10
+  context_radius: 1
+  max_context_items: 15
+
+reranking:
+  model:
+    provider: mxbai
+    name: mxbai-rerank-base-v1
+```
+
+### Long-Form Content (Articles, Reports)
+
+```yaml
+processing:
+  chunk_size: 512
+  chunker_type: hybrid
+
+search:
+  limit: 5
+  context_radius: 2
+  max_context_items: 10
+```
+
+### FAQ/Knowledge Base
+
+```yaml
+processing:
+  chunk_size: 128
+  chunker_type: hybrid
+
+search:
+  limit: 5
+  context_radius: 0
+```
--- a/mkdocs.yml
+++ b/mkdocs.yml
@ -62,12 +62,13 @@ nav:
      - Configuration:
          - configuration/index.md
          - Providers: configuration/providers.md
-          - QA and Research: configuration/qa-research.md
+          - Search and Question Answering: configuration/qa-research.md
          - Document Processing: configuration/processing.md
          - Storage: configuration/storage.md
      - CLI: cli.md
      - Python: python.md
      - Custom Pipelines: custom-pipelines.md
+      - Tuning: tuning.md
      - Agents: agents.md
      - Server: server.md
      - Remote processing: remote-processing.md