320 lines
8.7 KiB
Markdown
320 lines
8.7 KiB
Markdown
# Tuning haiku.rag for Your Corpus
|
|
|
|
This guide explains how to tune haiku.rag settings based on your document corpus characteristics. The right settings depend on your document types, query patterns, and accuracy requirements.
|
|
|
|
## Key Concepts
|
|
|
|
### Retrieval vs Generation
|
|
|
|
RAG has two phases:
|
|
|
|
1. **Retrieval**: Finding relevant chunks from your corpus
|
|
2. **Generation**: Using those chunks to answer questions
|
|
|
|
Poor retrieval means the LLM never sees the relevant content, regardless of how good the model is. Tuning retrieval is usually more impactful than tuning generation.
|
|
|
|
### Recall vs Precision
|
|
|
|
- **Recall**: What fraction of relevant documents did we find?
|
|
- **Precision**: What fraction of retrieved documents are relevant?
|
|
|
|
For RAG, recall matters more than precision. Missing a relevant chunk means wrong answers. Including an extra irrelevant chunk just wastes context tokens.
|
|
|
|
## Search Settings
|
|
|
|
### `search.limit`
|
|
|
|
Default number of chunks to retrieve.
|
|
|
|
```yaml
|
|
search:
|
|
limit: 5 # Default
|
|
```
|
|
|
|
**When to increase:**
|
|
|
|
- Complex questions requiring information from multiple sources
|
|
- Broad topics spread across many documents
|
|
|
|
**When to decrease:**
|
|
|
|
- Simple factual questions
|
|
- Highly focused corpus where top results are usually correct
|
|
- Cost-sensitive deployments (fewer chunks = fewer tokens)
|
|
|
|
**Typical values:** 3-10
|
|
|
|
### `search.context_radius`
|
|
|
|
Number of adjacent DocItems to include when expanding search results. Only applies to text content (paragraphs). Tables, code blocks, and lists use structural expansion automatically.
|
|
|
|
```yaml
|
|
search:
|
|
context_radius: 0 # Default: no expansion
|
|
```
|
|
|
|
**When to increase:**
|
|
|
|
- Answers require surrounding context (definitions, explanations)
|
|
- Chunks are small and queries need more context
|
|
- Documents have strong local coherence (adjacent paragraphs relate)
|
|
|
|
**When to keep at 0:**
|
|
|
|
- Large chunks that already contain sufficient context
|
|
- Documents where adjacent content is often unrelated
|
|
- When chunk boundaries align well with semantic units
|
|
|
|
**Typical values:** 0-3
|
|
|
|
### `search.max_context_items` and `search.max_context_chars`
|
|
|
|
Safety limits on context expansion to prevent runaway expansion.
|
|
|
|
```yaml
|
|
search:
|
|
max_context_items: 10 # Max DocItems per expanded result
|
|
max_context_chars: 10000 # Max characters per expanded result
|
|
```
|
|
|
|
Increase if expansion is being truncated and you need more context. Decrease if expanded results are too long for your LLM context window.
|
|
|
|
## Processing Settings
|
|
|
|
### `processing.chunk_size`
|
|
|
|
Maximum tokens per chunk (using the configured tokenizer).
|
|
|
|
```yaml
|
|
processing:
|
|
chunk_size: 256 # Default
|
|
```
|
|
|
|
**Trade-offs:**
|
|
|
|
| Smaller chunks (128-256) | Larger chunks (512-1024) |
|
|
|-------------------------|-------------------------|
|
|
| More precise retrieval | Better context per chunk |
|
|
| May miss spanning content | Better recall |
|
|
| More chunks to search | Faster search |
|
|
| Better for specific queries | Better for broad queries |
|
|
|
|
**Guidance by corpus type:**
|
|
|
|
- **Technical documentation**: 256-512 (specific lookups)
|
|
- **Long-form articles**: 512-1024 (need context)
|
|
- **FAQs/short answers**: 128-256 (discrete answers)
|
|
- **Code documentation**: 256-512 (function-level)
|
|
|
|
### `processing.chunker_type`
|
|
|
|
Chunking strategy.
|
|
|
|
```yaml
|
|
processing:
|
|
chunker_type: hybrid # Default
|
|
```
|
|
|
|
- **`hybrid`**: Structure-aware with token limits. Best for most documents.
|
|
- **`hierarchical`**: Preserves document hierarchy strictly. Use for highly structured documents where hierarchy matters.
|
|
|
|
|
|
### `processing.chunking_merge_peers`
|
|
|
|
Whether to merge adjacent small chunks that share the same section.
|
|
|
|
```yaml
|
|
processing:
|
|
chunking_merge_peers: true # Default
|
|
```
|
|
|
|
Keep `true` unless you specifically want very granular chunks. Merging improves embedding quality by ensuring chunks have sufficient context.
|
|
|
|
## Embedding Settings
|
|
|
|
### Model Selection
|
|
|
|
Embedding model choice significantly impacts retrieval quality.
|
|
|
|
```yaml
|
|
embeddings:
|
|
model:
|
|
provider: ollama
|
|
name: qwen3-embedding:4b
|
|
vector_dim: 2560
|
|
```
|
|
|
|
**Considerations:**
|
|
|
|
- Larger models generally produce better embeddings but are slower
|
|
- Match `vector_dim` to your model's actual output dimension
|
|
- Local models (Ollama) vs API models (OpenAI, VoyageAI) trade-off cost vs quality
|
|
|
|
### Contextualizing Embeddings
|
|
|
|
Chunks are embedded with section headings prepended (via `contextualize()`). This improves retrieval by including structural context in the embedding.
|
|
|
|
If your documents lack clear headings, embeddings will be based on chunk content alone.
|
|
|
|
## Reranking
|
|
|
|
Reranking retrieves more candidates than needed, then uses a cross-encoder to re-score them.
|
|
|
|
```yaml
|
|
reranking:
|
|
model:
|
|
provider: mxbai # or cohere, zeroentropy, vllm
|
|
name: mxbai-rerank-base-v1
|
|
```
|
|
|
|
**When to use reranking:**
|
|
|
|
- Embedding model has limited accuracy
|
|
- Queries are complex or ambiguous
|
|
- You can afford the latency (adds ~100-500ms)
|
|
|
|
**When to skip reranking:**
|
|
|
|
- Simple, specific queries
|
|
- High-quality embedding model
|
|
- Latency-sensitive applications
|
|
|
|
When reranking is enabled, haiku.rag automatically retrieves 10x the requested limit, then reranks to the final count. You don't need to adjust `search.limit` for reranking.
|
|
|
|
## Tuning Workflow
|
|
|
|
### 1. Use the Inspector
|
|
|
|
The inspector is your best tool for understanding how your corpus is chunked and how search behaves:
|
|
|
|
```bash
|
|
haiku-rag inspect
|
|
```
|
|
|
|
**What to look for:**
|
|
|
|
- Browse documents and their chunks to see how content is split
|
|
- Use the search modal (`/`) to test queries and see which chunks are retrieved
|
|
- Press `c` on a chunk to view expanded context - see what additional content would be included with `context_radius > 0`
|
|
- Check chunk sizes - are they too small (fragmented) or too large (unfocused)?
|
|
|
|
### 2. Test Search Manually
|
|
|
|
Before changing settings, run searches from the CLI to understand current behavior:
|
|
|
|
```bash
|
|
# Search and see results
|
|
haiku-rag search "your test query" --limit 10
|
|
|
|
# Try the QA to see end-to-end behavior
|
|
haiku-rag ask "your question"
|
|
```
|
|
|
|
### 3. Identify the Bottleneck
|
|
|
|
- **Relevant chunks not retrieved**: Try larger `search.limit`, smaller `chunk_size`, or a different embedding model
|
|
- **Too many irrelevant chunks**: Try reranking or larger `chunk_size`
|
|
- **Chunks found but answers wrong**: Try `context_radius` expansion or a better QA model
|
|
|
|
### 4. Test One Change at a Time
|
|
|
|
```bash
|
|
# After changing chunk_size, rebuild is required
|
|
haiku-rag rebuild
|
|
|
|
# After changing search settings, no rebuild needed - just test again
|
|
haiku-rag search "your test query"
|
|
```
|
|
|
|
### 5. Build Dataset-Specific Evaluations
|
|
|
|
For systematic tuning, create evaluations specific to your corpus. See the `evaluations/` directory in the repository for examples of how to:
|
|
|
|
- Define test cases with questions and expected answers
|
|
- Run retrieval benchmarks (MRR, MAP)
|
|
- Run QA accuracy benchmarks with LLM judges
|
|
|
|
Custom evaluations let you measure the impact of configuration changes objectively rather than relying on intuition.
|
|
|
|
### 6. Consider Your Corpus
|
|
|
|
| Corpus Type | Suggested Starting Point |
|
|
|-------------|-------------------------|
|
|
| Technical docs | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
|
|
| Legal/contracts | `chunk_size: 512`, `limit: 5`, `context_radius: 2` |
|
|
| News articles | `chunk_size: 512`, `limit: 5`, `context_radius: 0` |
|
|
| Scientific papers | `chunk_size: 256`, `limit: 5`, reranking enabled |
|
|
| FAQs | `chunk_size: 128`, `limit: 5`, `context_radius: 0` |
|
|
| Code repos | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
|
|
|
|
## Common Issues
|
|
|
|
### "Relevant content not being retrieved"
|
|
|
|
1. Check chunk boundaries - is the content split awkwardly?
|
|
2. Try smaller chunks for more granular matching
|
|
3. Increase `search.limit`
|
|
4. Consider a different embedding model
|
|
|
|
### "Retrieved chunks lack context"
|
|
|
|
1. Increase `context_radius` for text content
|
|
2. Increase `chunk_size` for more context per chunk
|
|
3. Structural content (tables, code) expands automatically
|
|
|
|
### "Search is slow"
|
|
|
|
1. Create a vector index: `haiku-rag create-index`
|
|
2. Reduce `search.limit`
|
|
3. Consider a smaller embedding model
|
|
|
|
### "QA answers are wrong despite good retrieval"
|
|
|
|
1. Check if chunks are being truncated by LLM context limits
|
|
2. Try a more capable QA model
|
|
3. Reduce number of chunks or expansion to fit context window
|
|
|
|
## Example Configurations
|
|
|
|
### High-Precision Technical Documentation
|
|
|
|
```yaml
|
|
processing:
|
|
chunk_size: 256
|
|
chunker_type: hybrid
|
|
|
|
search:
|
|
limit: 10
|
|
context_radius: 1
|
|
max_context_items: 15
|
|
|
|
reranking:
|
|
model:
|
|
provider: mxbai
|
|
name: mxbai-rerank-base-v1
|
|
```
|
|
|
|
### Long-Form Content (Articles, Reports)
|
|
|
|
```yaml
|
|
processing:
|
|
chunk_size: 512
|
|
chunker_type: hybrid
|
|
|
|
search:
|
|
limit: 5
|
|
context_radius: 2
|
|
max_context_items: 10
|
|
```
|
|
|
|
### FAQ/Knowledge Base
|
|
|
|
```yaml
|
|
processing:
|
|
chunk_size: 128
|
|
chunker_type: hybrid
|
|
|
|
search:
|
|
limit: 5
|
|
context_radius: 0
|
|
```
|