docs: add chat page, restructure nav, reorder sections
This commit is contained in:
Yiorgis Gozadinos
parent
76804b682f
commit
19e2be003e
13 changed files with 715 additions and 714 deletions
177
docs/apps.md
177
docs/apps.md
|
|
@ -1,90 +1,34 @@
|
|||
# Applications
|
||||
# Web application
|
||||
|
||||
Three interactive applications for working with your knowledge base.
|
||||
A browser-based reference implementation of conversational RAG, built on a Starlette backend with pydantic-ai's `AGUIAdapter` and a Next.js / CopilotKit frontend. It lives in the `app/` directory of the haiku.rag repository.
|
||||
|
||||
## Chat TUI
|
||||
This is a starting point for your own deployments, not the canonical haiku.rag UX. For the day-to-day terminal experience see [Chat](chat.md).
|
||||
|
||||
Conversational RAG from the terminal with streaming responses and session memory.
|
||||
## Features
|
||||
|
||||
!!! note
|
||||
Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in full `haiku.rag` package)
|
||||
- Streaming chat with real-time tool execution visibility.
|
||||
- Expandable citations with source documents, pages, and headings.
|
||||
- Visual grounding to view chunk source locations in documents.
|
||||
- Document filter to restrict searches to selected documents.
|
||||
- Session state view for inspecting citations and search results.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
haiku-rag chat
|
||||
haiku-rag chat --db /path/to/database.lancedb
|
||||
|
||||
# Enable analysis skill (code execution)
|
||||
haiku-rag chat -s rag -s analysis
|
||||
|
||||
# Analysis only
|
||||
haiku-rag chat -s analysis
|
||||
```
|
||||
|
||||
### Interface
|
||||
|
||||
The chat interface provides:
|
||||
|
||||
- Streaming responses with real-time tool execution indicators
|
||||
- Expandable citations showing source document, pages, and headings
|
||||
- Session memory for context-aware follow-up questions
|
||||
- Visual grounding to inspect chunk source locations
|
||||
|
||||
|
||||
|
||||
<div style="padding:56.25% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1159658167?badge=0&autopause=0&player_id=0&app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="haiku.rag Chat TUI demo"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>
|
||||
|
||||
*Demo: Chatting with an agent over 1000 arXiv papers. Shows context building (3:00), citations with visual grounding (3:20), and document listing/retrieval.*
|
||||
|
||||
### Command Palette
|
||||
|
||||
Press `Ctrl+P` to open the command palette:
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| View state | View the current session state |
|
||||
| Filter documents | Select documents to restrict searches |
|
||||
| Show database info | View document/chunk counts and storage info |
|
||||
| Visual grounding | View chunk source location in document |
|
||||
| Clear chat | Clear chat history and reset session |
|
||||
|
||||
### Session Management
|
||||
|
||||
- Conversation history is maintained in memory for the session
|
||||
- Citations are tracked per response and can be inspected
|
||||
- Document filter restricts all searches to selected documents
|
||||
- Clearing chat resets session state
|
||||
|
||||
## Web Application
|
||||
|
||||
Browser-based conversational RAG with a CopilotKit frontend.
|
||||
|
||||
### Features
|
||||
|
||||
- Streaming chat with real-time tool execution visibility
|
||||
- Expandable citations with source documents, pages, and headings
|
||||
- Visual grounding to view chunk source locations in documents
|
||||
- Document filter to restrict searches to selected documents
|
||||
- Session state view for inspecting citations and search results
|
||||
|
||||
### Quick Start
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
cd app
|
||||
docker compose -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
- Frontend: http://localhost:3000
|
||||
- Backend: http://localhost:8001
|
||||
- Frontend: `http://localhost:3000`
|
||||
- Backend: `http://localhost:8001`
|
||||
|
||||
### Architecture
|
||||
## Architecture
|
||||
|
||||
- **Backend**: Starlette server with pydantic-ai `AGUIAdapter`
|
||||
- **Frontend**: Next.js with CopilotKit
|
||||
- **Protocol**: AG-UI for streaming chat
|
||||
- **Backend**: Starlette server with pydantic-ai `AGUIAdapter`.
|
||||
- **Frontend**: Next.js with CopilotKit.
|
||||
- **Protocol**: AG-UI for streaming chat.
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
Create a `.env` file in the `app/` directory:
|
||||
|
||||
|
|
@ -113,7 +57,7 @@ qa:
|
|||
name: claude-sonnet-4-20250514
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
## API endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
|
|
@ -123,91 +67,12 @@ qa:
|
|||
| `/api/visualize/{chunk_id}` | GET | Visual grounding images (base64) |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
### Development
|
||||
## Development
|
||||
|
||||
**Hot reload**: The backend reloads automatically on file changes. For frontend changes:
|
||||
The backend reloads automatically on file changes. For frontend changes:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml up -d --build frontend
|
||||
```
|
||||
|
||||
**Logfire debugging**: If `LOGFIRE_TOKEN` is set, LLM calls are traced and available in the Logfire dashboard.
|
||||
|
||||
## Inspector
|
||||
|
||||
TUI for browsing documents, chunks, and search results.
|
||||
|
||||
!!! note
|
||||
Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in full `haiku.rag` package)
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
haiku-rag inspect
|
||||
haiku-rag inspect --db /path/to/database.lancedb
|
||||
```
|
||||
|
||||
### Interface
|
||||
|
||||
Three panels display your data:
|
||||
|
||||
- **Documents** (left) - All documents in the database
|
||||
- **Chunks** (top right) - Chunks for the selected document
|
||||
- **Detail View** (bottom right) - Full content and metadata
|
||||
|
||||
|
||||
|
||||
### Navigation
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| `Tab` | Cycle between panels |
|
||||
| `↑` / `↓` | Navigate lists |
|
||||
| `/` | Open search modal |
|
||||
| `c` | Context expansion modal (when viewing a chunk) |
|
||||
| `v` | Visual grounding modal (when viewing a chunk) |
|
||||
| `q` | Quit |
|
||||
|
||||
**Mouse**: Click to select, scroll to view content.
|
||||
|
||||
### Search
|
||||
|
||||
Press `/` to open the full-screen search modal:
|
||||
|
||||
- Enter your query and press `Enter` to search
|
||||
- **Left panel**: Search results with relevance scores `[0.95] content preview`
|
||||
- **Right panel**: Full chunk content and metadata
|
||||
- Use `↑` / `↓` to navigate results
|
||||
- Press `Enter` on a result to navigate to that document/chunk
|
||||
- Press `Esc` to close search
|
||||
|
||||
Search uses hybrid (vector + full-text) search across all chunks.
|
||||
|
||||
### Context Expansion
|
||||
|
||||
Press `c` while viewing a chunk to see the expanded context that would be provided to the rag skill:
|
||||
|
||||
- Section-aware expansion: expands to fill the current document section
|
||||
- Noise filtering: footnotes, page headers/footers excluded from structured documents
|
||||
- Includes metadata like source document, content type, and relevance score
|
||||
|
||||
### Visual Grounding
|
||||
|
||||
Visual grounding shows exactly where a chunk appears in the original document by highlighting its bounding box on the page image. This helps verify chunk boundaries and understand how content was extracted.
|
||||
|
||||
Press `v` while viewing a chunk to see page images with the chunk's location highlighted:
|
||||
|
||||
- Bounding boxes show the exact region of the page that maps to the chunk
|
||||
- Use `←` / `→` arrow keys to navigate between pages when a chunk spans multiple pages
|
||||
- Press `Esc` to close the modal
|
||||
|
||||
|
||||
|
||||
#### Requirements
|
||||
|
||||
- **Page images**: Documents must be processed with Docling's page image extraction enabled (default for PDFs)
|
||||
- **Terminal image support**: Your terminal must support inline images (e.g., iTerm2, WezTerm, Kitty). Terminals without image support will show a placeholder message.
|
||||
- **DoclingDocument storage**: Text-only documents (plain text, markdown added via `add`) don't have visual grounding available
|
||||
|
||||
!!! tip
|
||||
You can also view visual grounding from the command line with `haiku-rag visualize <chunk_id>`. See [CLI documentation](cli.md#visualize-chunk) for details.
|
||||
If `LOGFIRE_TOKEN` is set, LLM calls are traced and available in the Logfire dashboard.
|
||||
|
|
|
|||
74
docs/chat.md
Normal file
74
docs/chat.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Chat
|
||||
|
||||
The chat TUI runs conversational RAG against your database from the terminal. Streaming responses, expandable citations with visual grounding, multi-turn sessions, and a command palette for filtering and inspection.
|
||||
|
||||
!!! note
|
||||
Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in the full `haiku.rag` package).
|
||||
|
||||
## Run it
|
||||
|
||||
```bash
|
||||
haiku-rag chat
|
||||
haiku-rag chat --db /path/to/database.lancedb
|
||||
haiku-rag chat --model openai:gpt-4o
|
||||
```
|
||||
|
||||
|
||||
|
||||
<div style="padding:56.25% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1159658167?badge=0&autopause=0&player_id=0&app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="haiku.rag Chat TUI demo"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>
|
||||
|
||||
*Demo: chatting with an agent over 1000 arXiv papers. Shows context building (3:00), citations with visual grounding (3:20), and document listing.*
|
||||
|
||||
## How it works
|
||||
|
||||
The chat is a Pydantic AI agent with the `rag` [skill](skills/rag.md) attached. Each turn the agent decides which tool to call next, runs hybrid search against your documents, expands context around the hits, may issue further searches, and answers with citations. You see streaming text and a live indicator of which tool is running.
|
||||
|
||||
The session is in-memory for the lifetime of the TUI. Conversation history is kept across turns so follow-up questions reuse prior context. Citations are tracked per turn and inspectable via the command palette. Clearing the chat resets the session and the agent's memory.
|
||||
|
||||
## Citations and visual grounding
|
||||
|
||||
Each answer cites the chunks the agent used, with source document, page numbers, and section headings. Citations are expandable inline.
|
||||
|
||||
For visual grounding (the chunk highlighted on its page image), open the command palette and pick "Show visual grounding". This requires:
|
||||
|
||||
- Documents processed via Docling with page images (default for PDFs).
|
||||
- A terminal that supports inline images (iTerm2, WezTerm, Kitty).
|
||||
- A stored DoclingDocument on the document. Plain text added via `haiku-rag add` doesn't have it.
|
||||
|
||||
You can also render visual grounding from the CLI without launching the TUI:
|
||||
|
||||
```bash
|
||||
haiku-rag visualize <chunk_id>
|
||||
```
|
||||
|
||||
## Command palette
|
||||
|
||||
`Ctrl+P` opens the palette.
|
||||
|
||||
| Command | What it does |
|
||||
|---------|--------------|
|
||||
| Clear chat | Reset session memory |
|
||||
| Filter documents | Restrict searches to selected documents |
|
||||
| Show visual grounding | Visual grounding for a citation |
|
||||
| Database info | Document and chunk counts, storage stats |
|
||||
| View state | Current session state, citations, and intermediate tool results |
|
||||
|
||||
## Skills
|
||||
|
||||
The default skill is `rag`. Add `analysis` for sandboxed Python execution over your documents:
|
||||
|
||||
```bash
|
||||
# both skills
|
||||
haiku-rag chat -s rag -s analysis
|
||||
|
||||
# analysis only
|
||||
haiku-rag chat -s analysis
|
||||
```
|
||||
|
||||
The `analysis` skill mounts a virtual filesystem under `/documents/{id}/` and runs Python code against it inside a sandbox. Useful for aggregation, computation, and multi-document analysis. See [Analysis skill](skills/analysis.md).
|
||||
|
||||
## Document filter
|
||||
|
||||
Run "Filter documents" from the command palette to restrict searches to a subset. The filter applies to every search the agent runs for the rest of the session.
|
||||
|
||||
Chat also honors the global `--read-only` and `--before` flags. See the [CLI reference](cli.md) for details.
|
||||
454
docs/cli.md
454
docs/cli.md
|
|
@ -26,24 +26,6 @@ The `haiku-rag` CLI provides complete document management functionality.
|
|||
|
||||
## Document Management
|
||||
|
||||
### List Documents
|
||||
|
||||
```bash
|
||||
haiku-rag list
|
||||
```
|
||||
|
||||
Filter documents by properties:
|
||||
```bash
|
||||
# Filter by URI pattern (--filter or -f)
|
||||
haiku-rag list --filter "uri LIKE '%arxiv%'"
|
||||
|
||||
# Filter by exact title
|
||||
haiku-rag list --filter "title = 'My Document'"
|
||||
|
||||
# Combine multiple conditions
|
||||
haiku-rag list --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
|
||||
```
|
||||
|
||||
### Add Documents
|
||||
|
||||
From text:
|
||||
|
|
@ -95,6 +77,24 @@ AWS_ACCESS_KEY_ID=key AWS_SECRET_ACCESS_KEY=secret AWS_REGION=us-east-1 \
|
|||
the database rolls back to the pre‑operation snapshot using LanceDB table versioning. You can optimize and
|
||||
compact the database by running the [vacuum](#vacuum-optimize-and-cleanup) command.
|
||||
|
||||
### List Documents
|
||||
|
||||
```bash
|
||||
haiku-rag list
|
||||
```
|
||||
|
||||
Filter documents by properties:
|
||||
```bash
|
||||
# Filter by URI pattern (--filter or -f)
|
||||
haiku-rag list --filter "uri LIKE '%arxiv%'"
|
||||
|
||||
# Filter by exact title
|
||||
haiku-rag list --filter "title = 'My Document'"
|
||||
|
||||
# Combine multiple conditions
|
||||
haiku-rag list --filter "uri LIKE '%.pdf' AND title LIKE '%paper%'"
|
||||
```
|
||||
|
||||
### Get Document
|
||||
|
||||
```bash
|
||||
|
|
@ -108,19 +108,6 @@ haiku-rag delete 3f4a... # document ID
|
|||
haiku-rag rm 3f4a... # alias
|
||||
```
|
||||
|
||||
## Visualize Chunk
|
||||
|
||||
Display visual grounding for a chunk - shows page images with highlighted bounding boxes:
|
||||
|
||||
```bash
|
||||
haiku-rag visualize <chunk_id>
|
||||
```
|
||||
|
||||
This renders the source document pages with the chunk's location highlighted. Useful for verifying chunk boundaries and understanding document structure.
|
||||
|
||||
!!! note
|
||||
Requires a terminal with image support (iTerm2, Kitty, WezTerm, etc.) and documents processed with docling that have page images stored.
|
||||
|
||||
## Search
|
||||
|
||||
Basic search:
|
||||
|
|
@ -182,6 +169,26 @@ Flags:
|
|||
|
||||
- `--filter` / `-f`: Restrict searches to documents matching the filter (see [Filtering Search Results](python.md#filtering-search-results))
|
||||
|
||||
## Analyze
|
||||
|
||||
Answer complex analytical questions via code execution:
|
||||
|
||||
```bash
|
||||
haiku-rag analyze "How many documents mention security?"
|
||||
```
|
||||
|
||||
Filter to specific documents:
|
||||
|
||||
```bash
|
||||
haiku-rag analyze "What is the total revenue?" --filter "title LIKE '%Financial%'"
|
||||
```
|
||||
|
||||
Flags:
|
||||
|
||||
- `--filter` / `-f`: SQL WHERE clause to restrict document access
|
||||
|
||||
See [Analysis skill](skills/analysis.md) for details on capabilities and configuration.
|
||||
|
||||
## Chat
|
||||
|
||||
Launch an interactive chat session for multi-turn conversations:
|
||||
|
|
@ -208,7 +215,7 @@ The chat interface provides:
|
|||
- Session memory for context-aware follow-up questions
|
||||
- Visual grounding to inspect chunk source locations
|
||||
|
||||
See [Applications](apps.md#chat-tui) for keyboard shortcuts and features.
|
||||
See [Chat](chat.md) for keyboard shortcuts and features.
|
||||
|
||||
## Inspect
|
||||
|
||||
|
|
@ -229,27 +236,207 @@ The inspector provides:
|
|||
- Explore individual chunks
|
||||
- Search and filter results
|
||||
|
||||
See [Applications](apps.md#inspector) for details.
|
||||
See [Tuning: Inspector](tuning.md#inspector) for the full keybindings and modal flows.
|
||||
|
||||
## Analyze
|
||||
## Visualize Chunk
|
||||
|
||||
Answer complex analytical questions via code execution:
|
||||
Display visual grounding for a chunk - shows page images with highlighted bounding boxes:
|
||||
|
||||
```bash
|
||||
haiku-rag analyze "How many documents mention security?"
|
||||
haiku-rag visualize <chunk_id>
|
||||
```
|
||||
|
||||
Filter to specific documents:
|
||||
This renders the source document pages with the chunk's location highlighted. Useful for verifying chunk boundaries and understanding document structure.
|
||||
|
||||
!!! note
|
||||
Requires a terminal with image support (iTerm2, Kitty, WezTerm, etc.) and documents processed with docling that have page images stored.
|
||||
|
||||
## Database lifecycle
|
||||
|
||||
### Initialize Database
|
||||
|
||||
Create a new database:
|
||||
|
||||
```bash
|
||||
haiku-rag analyze "What is the total revenue?" --filter "title LIKE '%Financial%'"
|
||||
haiku-rag init [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
Flags:
|
||||
This creates the database with the configured settings. **All other commands require an existing database** - they will fail with an informative error if the database doesn't exist.
|
||||
|
||||
- `--filter` / `-f`: SQL WHERE clause to restrict document access
|
||||
### Info
|
||||
|
||||
See [Analysis](agents/analysis.md) for details on capabilities and configuration.
|
||||
Display database metadata:
|
||||
|
||||
```bash
|
||||
haiku-rag info [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
Shows:
|
||||
- path to the database
|
||||
- stored haiku.rag version (from settings)
|
||||
- embeddings provider/model and vector dimension
|
||||
- number of documents and chunks (with storage sizes)
|
||||
- vector index status (exists/not created, indexed/unindexed chunks)
|
||||
- table versions per table (documents, chunks)
|
||||
|
||||
At the end, a separate "Versions" section lists runtime package versions:
|
||||
- haiku.rag
|
||||
- lancedb
|
||||
- docling
|
||||
|
||||
### Migrate Database
|
||||
|
||||
Apply pending database migrations:
|
||||
|
||||
```bash
|
||||
haiku-rag migrate [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
When you upgrade haiku.rag to a new version that includes schema changes, the database requires migration. Opening a database with pending migrations will display an error:
|
||||
|
||||
```
|
||||
Error: Database requires migration from 0.19.0 to 0.26.5. 3 migration(s) pending. Run 'haiku-rag migrate' to upgrade.
|
||||
```
|
||||
|
||||
Run `haiku-rag migrate` to apply the pending migrations. The command shows which migrations were applied:
|
||||
|
||||
```
|
||||
Applied 4 migration(s):
|
||||
- 0.20.0: Add 'docling_document_json' and 'docling_version' columns
|
||||
- 0.23.1: Add content_fts column for contextualized FTS search
|
||||
- 0.25.0: Compress docling_document with gzip
|
||||
- 0.38.0: Split docling_document pages into separate column and re-compress with zstd
|
||||
Migration completed successfully.
|
||||
```
|
||||
|
||||
!!! tip
|
||||
Back up your database before running migrations. While migrations are designed to be safe, having a backup provides peace of mind for production databases.
|
||||
|
||||
### Download Models
|
||||
|
||||
Download required runtime models:
|
||||
|
||||
```bash
|
||||
haiku-rag download-models
|
||||
```
|
||||
|
||||
This command downloads:
|
||||
|
||||
- Docling OCR/conversion models
|
||||
- HuggingFace tokenizer (for chunking)
|
||||
- Ollama models referenced in your configuration (embeddings, QA, rerank)
|
||||
|
||||
Progress is displayed in real-time with download status and progress bars for Ollama model pulls.
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Create Vector Index
|
||||
|
||||
Create a vector index on the chunks table for fast approximate nearest neighbor search:
|
||||
|
||||
```bash
|
||||
haiku-rag create-index [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- Minimum 256 chunks required for index creation (LanceDB training data requirement)
|
||||
- Creates an IVF_PQ index using the configured `search.vector_index_metric` (cosine/l2/dot)
|
||||
|
||||
**When to use:**
|
||||
- After ingesting documents (indexes are not created automatically)
|
||||
- After adding significant new data to rebuild the index
|
||||
- Use `haiku-rag info` to check index status and see how many chunks are indexed/unindexed
|
||||
|
||||
**Search behavior:**
|
||||
- Without index: Brute-force kNN search (exact nearest neighbors, slower for large datasets)
|
||||
- With index: Fast ANN (approximate nearest neighbors) using IVF_PQ
|
||||
- With stale index: LanceDB combines indexed results (fast ANN) + brute-force kNN on unindexed rows
|
||||
- Performance degrades as more unindexed data accumulates
|
||||
|
||||
### Rebuild Database
|
||||
|
||||
Rebuild the database by re-indexing documents. Useful when switching embeddings provider/model or changing chunking settings:
|
||||
|
||||
```bash
|
||||
# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
|
||||
haiku-rag rebuild
|
||||
|
||||
# Re-chunk from stored content (no source file access)
|
||||
haiku-rag rebuild --rechunk
|
||||
|
||||
# Only regenerate embeddings (fastest, keeps existing chunks)
|
||||
haiku-rag rebuild --embed-only
|
||||
|
||||
# Only generate titles for untitled documents
|
||||
haiku-rag rebuild --title-only
|
||||
|
||||
# Run the VLM over already-stored picture bytes and patch descriptions
|
||||
# into the docling blob. Skips the docling parse entirely.
|
||||
haiku-rag rebuild --descriptions
|
||||
```
|
||||
|
||||
**Rebuild modes:**
|
||||
|
||||
| Mode | Flag | Use case |
|
||||
|------|------|----------|
|
||||
| Full | (default) | Changed converter, source files updated |
|
||||
| Rechunk | `--rechunk` | Changed chunking strategy or chunk size |
|
||||
| Embed only | `--embed-only` | Changed embedding model or vector dimensions |
|
||||
| Title only | `--title-only` | Generate titles for documents without one |
|
||||
| Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database |
|
||||
|
||||
**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent — pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely; only the VLM time is paid.
|
||||
|
||||
### Vacuum (Optimize and Cleanup)
|
||||
|
||||
Reduce disk usage by optimizing and pruning old table versions across all tables:
|
||||
|
||||
```bash
|
||||
haiku-rag vacuum
|
||||
```
|
||||
|
||||
**Automatic Cleanup:** Vacuum runs automatically in the background after document operations. By default, it removes versions older than 1 day (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately.
|
||||
|
||||
## Server
|
||||
|
||||
Start services (requires at least one flag):
|
||||
```bash
|
||||
# MCP server only (HTTP transport)
|
||||
haiku-rag serve --mcp
|
||||
|
||||
# MCP server (stdio transport)
|
||||
haiku-rag serve --mcp --stdio
|
||||
|
||||
# File monitoring only
|
||||
haiku-rag serve --monitor
|
||||
|
||||
# Both services
|
||||
haiku-rag serve --monitor --mcp
|
||||
|
||||
# Custom MCP port
|
||||
haiku-rag serve --mcp --mcp-port 9000
|
||||
|
||||
# Read-only mode (excludes write MCP tools, disables monitor)
|
||||
haiku-rag --read-only serve --mcp
|
||||
```
|
||||
|
||||
See [Server Mode](server.md) for details on available services.
|
||||
|
||||
## Settings
|
||||
|
||||
View current configuration settings:
|
||||
```bash
|
||||
haiku-rag settings
|
||||
```
|
||||
|
||||
### Generate Configuration File
|
||||
|
||||
Generate a YAML configuration file with defaults:
|
||||
```bash
|
||||
haiku-rag init-config [output_path]
|
||||
```
|
||||
|
||||
If no path is specified, creates `haiku.rag.yaml` in the current directory.
|
||||
|
||||
## Create Skill
|
||||
|
||||
|
|
@ -309,191 +496,6 @@ haiku-skills chat --use-entrypoints --skill medic
|
|||
└── haiku.rag.yaml # Optional config
|
||||
```
|
||||
|
||||
## Server
|
||||
|
||||
Start services (requires at least one flag):
|
||||
```bash
|
||||
# MCP server only (HTTP transport)
|
||||
haiku-rag serve --mcp
|
||||
|
||||
# MCP server (stdio transport)
|
||||
haiku-rag serve --mcp --stdio
|
||||
|
||||
# File monitoring only
|
||||
haiku-rag serve --monitor
|
||||
|
||||
# Both services
|
||||
haiku-rag serve --monitor --mcp
|
||||
|
||||
# Custom MCP port
|
||||
haiku-rag serve --mcp --mcp-port 9000
|
||||
|
||||
# Read-only mode (excludes write MCP tools, disables monitor)
|
||||
haiku-rag --read-only serve --mcp
|
||||
```
|
||||
|
||||
See [Server Mode](server.md) for details on available services.
|
||||
|
||||
## Settings
|
||||
|
||||
View current configuration settings:
|
||||
```bash
|
||||
haiku-rag settings
|
||||
```
|
||||
|
||||
### Generate Configuration File
|
||||
|
||||
Generate a YAML configuration file with defaults:
|
||||
```bash
|
||||
haiku-rag init-config [output_path]
|
||||
```
|
||||
|
||||
If no path is specified, creates `haiku.rag.yaml` in the current directory.
|
||||
|
||||
## Database Management
|
||||
|
||||
### Initialize Database
|
||||
|
||||
Create a new database:
|
||||
|
||||
```bash
|
||||
haiku-rag init [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
This creates the database with the configured settings. **All other commands require an existing database** - they will fail with an informative error if the database doesn't exist.
|
||||
|
||||
### Migrate Database
|
||||
|
||||
Apply pending database migrations:
|
||||
|
||||
```bash
|
||||
haiku-rag migrate [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
When you upgrade haiku.rag to a new version that includes schema changes, the database requires migration. Opening a database with pending migrations will display an error:
|
||||
|
||||
```
|
||||
Error: Database requires migration from 0.19.0 to 0.26.5. 3 migration(s) pending. Run 'haiku-rag migrate' to upgrade.
|
||||
```
|
||||
|
||||
Run `haiku-rag migrate` to apply the pending migrations. The command shows which migrations were applied:
|
||||
|
||||
```
|
||||
Applied 4 migration(s):
|
||||
- 0.20.0: Add 'docling_document_json' and 'docling_version' columns
|
||||
- 0.23.1: Add content_fts column for contextualized FTS search
|
||||
- 0.25.0: Compress docling_document with gzip
|
||||
- 0.38.0: Split docling_document pages into separate column and re-compress with zstd
|
||||
Migration completed successfully.
|
||||
```
|
||||
|
||||
!!! tip
|
||||
Back up your database before running migrations. While migrations are designed to be safe, having a backup provides peace of mind for production databases.
|
||||
|
||||
### Info
|
||||
|
||||
Display database metadata:
|
||||
|
||||
```bash
|
||||
haiku-rag info [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
Shows:
|
||||
- path to the database
|
||||
- stored haiku.rag version (from settings)
|
||||
- embeddings provider/model and vector dimension
|
||||
- number of documents and chunks (with storage sizes)
|
||||
- vector index status (exists/not created, indexed/unindexed chunks)
|
||||
- table versions per table (documents, chunks)
|
||||
|
||||
At the end, a separate "Versions" section lists runtime package versions:
|
||||
- haiku.rag
|
||||
- lancedb
|
||||
- docling
|
||||
|
||||
### Create Vector Index
|
||||
|
||||
Create a vector index on the chunks table for fast approximate nearest neighbor search:
|
||||
|
||||
```bash
|
||||
haiku-rag create-index [--db /path/to/your.lancedb]
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- Minimum 256 chunks required for index creation (LanceDB training data requirement)
|
||||
- Creates an IVF_PQ index using the configured `search.vector_index_metric` (cosine/l2/dot)
|
||||
|
||||
**When to use:**
|
||||
- After ingesting documents (indexes are not created automatically)
|
||||
- After adding significant new data to rebuild the index
|
||||
- Use `haiku-rag info` to check index status and see how many chunks are indexed/unindexed
|
||||
|
||||
**Search behavior:**
|
||||
- Without index: Brute-force kNN search (exact nearest neighbors, slower for large datasets)
|
||||
- With index: Fast ANN (approximate nearest neighbors) using IVF_PQ
|
||||
- With stale index: LanceDB combines indexed results (fast ANN) + brute-force kNN on unindexed rows
|
||||
- Performance degrades as more unindexed data accumulates
|
||||
|
||||
### Vacuum (Optimize and Cleanup)
|
||||
|
||||
Reduce disk usage by optimizing and pruning old table versions across all tables:
|
||||
|
||||
```bash
|
||||
haiku-rag vacuum
|
||||
```
|
||||
|
||||
**Automatic Cleanup:** Vacuum runs automatically in the background after document operations. By default, it removes versions older than 1 day (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately.
|
||||
|
||||
### Rebuild Database
|
||||
|
||||
Rebuild the database by re-indexing documents. Useful when switching embeddings provider/model or changing chunking settings:
|
||||
|
||||
```bash
|
||||
# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
|
||||
haiku-rag rebuild
|
||||
|
||||
# Re-chunk from stored content (no source file access)
|
||||
haiku-rag rebuild --rechunk
|
||||
|
||||
# Only regenerate embeddings (fastest, keeps existing chunks)
|
||||
haiku-rag rebuild --embed-only
|
||||
|
||||
# Only generate titles for untitled documents
|
||||
haiku-rag rebuild --title-only
|
||||
|
||||
# Run the VLM over already-stored picture bytes and patch descriptions
|
||||
# into the docling blob. Skips the docling parse entirely.
|
||||
haiku-rag rebuild --descriptions
|
||||
```
|
||||
|
||||
**Rebuild modes:**
|
||||
|
||||
| Mode | Flag | Use case |
|
||||
|------|------|----------|
|
||||
| Full | (default) | Changed converter, source files updated |
|
||||
| Rechunk | `--rechunk` | Changed chunking strategy or chunk size |
|
||||
| Embed only | `--embed-only` | Changed embedding model or vector dimensions |
|
||||
| Title only | `--title-only` | Generate titles for documents without one |
|
||||
| Descriptions | `--descriptions` | Add VLM picture descriptions to an existing database |
|
||||
|
||||
**`--descriptions` mode** runs the configured VLM (`processing.conversion_options.picture_description.model`) over the picture bytes already stored in `document_items.picture_data`, patches each description into the stored docling blob's `pictures[i].meta.description.text`, and re-chunks + re-embeds so chunk text reflects the new descriptions. Requires `processing.pictures: description` in the config. Idempotent — pictures that already carry a description are skipped, so the operation is safe to re-run after a partial failure. The docling parse is skipped entirely; only the VLM time is paid.
|
||||
|
||||
### Download Models
|
||||
|
||||
Download required runtime models:
|
||||
|
||||
```bash
|
||||
haiku-rag download-models
|
||||
```
|
||||
|
||||
This command downloads:
|
||||
|
||||
- Docling OCR/conversion models
|
||||
- HuggingFace tokenizer (for chunking)
|
||||
- Ollama models referenced in your configuration (embeddings, QA, rerank)
|
||||
|
||||
Progress is displayed in real-time with download status and progress bars for Ollama model pulls.
|
||||
|
||||
## Time Travel
|
||||
|
||||
LanceDB maintains version history for tables, enabling you to query the database as it existed at a previous point in time. This is useful for:
|
||||
|
|
|
|||
|
|
@ -53,6 +53,37 @@ processing:
|
|||
pictures: image # none | description | image
|
||||
```
|
||||
|
||||
### Local vs Remote Processing
|
||||
|
||||
**Local processing** (default):
|
||||
|
||||
- Uses `docling` library locally
|
||||
- No external dependencies
|
||||
- Good for development and small workloads
|
||||
|
||||
**Remote processing** (docling-serve):
|
||||
|
||||
- Offloads processing to docling-serve API
|
||||
- Better for heavy workloads and production
|
||||
- Requires docling-serve instance (see [Remote processing setup](../remote-processing.md))
|
||||
|
||||
To use remote processing:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
converter: docling-serve
|
||||
chunker: docling-serve
|
||||
|
||||
providers:
|
||||
docling_serve:
|
||||
base_url: http://localhost:5001
|
||||
api_key: "your-api-key" # Optional
|
||||
```
|
||||
|
||||
Conversion options work identically for both local and remote processing.
|
||||
|
||||
**Note:** When using `chunker: docling-serve`, OCR options (`do_ocr`, `force_ocr`, `ocr_engine`, `ocr_lang`) from `conversion_options` are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail—set `do_ocr: false` to disable OCR entirely.
|
||||
|
||||
### Conversion Options
|
||||
|
||||
The `conversion_options` section allows fine-grained control over document conversion. These options work with both `docling-local` and `docling-serve` converters.
|
||||
|
|
@ -203,6 +234,39 @@ Three independent settings drive ingest, retrieval, and QA:
|
|||
| Cross-modal search + vision QA | `image` or `description` | multimodal | `true` |
|
||||
| Cross-modal search, text QA only | `description` | multimodal | `false` |
|
||||
|
||||
### Chunking Strategies
|
||||
|
||||
**Hybrid chunking** (default):
|
||||
- Structure-aware chunking
|
||||
- Respects document boundaries
|
||||
- Best for most use cases
|
||||
|
||||
**Hierarchical chunking**:
|
||||
- Creates hierarchical chunk structure
|
||||
- Preserves document hierarchy
|
||||
- Useful for complex documents
|
||||
|
||||
### Chunk Size
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 256 # Maximum tokens per chunk
|
||||
```
|
||||
|
||||
Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](qa-research.md#search-settings).
|
||||
|
||||
### Table Serialization
|
||||
|
||||
Control how tables are represented in chunks:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunking_use_markdown_tables: false # Default: narrative format
|
||||
```
|
||||
|
||||
- `false`: Tables as narrative text ("Value A, Column 2 = Value B")
|
||||
- `true`: Tables as markdown (preserves table structure)
|
||||
|
||||
### Automatic Title Generation
|
||||
|
||||
Enable automatic title generation during document ingestion:
|
||||
|
|
@ -227,70 +291,6 @@ Explicit titles passed via `title=` parameter always take precedence and are nev
|
|||
|
||||
To generate titles for existing untitled documents, use [`rebuild --title-only`](../cli.md#rebuild-database).
|
||||
|
||||
### Local vs Remote Processing
|
||||
|
||||
**Local processing** (default):
|
||||
|
||||
- Uses `docling` library locally
|
||||
- No external dependencies
|
||||
- Good for development and small workloads
|
||||
|
||||
**Remote processing** (docling-serve):
|
||||
|
||||
- Offloads processing to docling-serve API
|
||||
- Better for heavy workloads and production
|
||||
- Requires docling-serve instance (see [Remote processing setup](../remote-processing.md))
|
||||
|
||||
To use remote processing:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
converter: docling-serve
|
||||
chunker: docling-serve
|
||||
|
||||
providers:
|
||||
docling_serve:
|
||||
base_url: http://localhost:5001
|
||||
api_key: "your-api-key" # Optional
|
||||
```
|
||||
|
||||
Conversion options work identically for both local and remote processing.
|
||||
|
||||
**Note:** When using `chunker: docling-serve`, OCR options (`do_ocr`, `force_ocr`, `ocr_engine`, `ocr_lang`) from `conversion_options` are passed to the chunking API. This is useful when running docling-serve in a read-only container where OCR model downloads fail—set `do_ocr: false` to disable OCR entirely.
|
||||
|
||||
### Chunking Strategies
|
||||
|
||||
**Hybrid chunking** (default):
|
||||
- Structure-aware chunking
|
||||
- Respects document boundaries
|
||||
- Best for most use cases
|
||||
|
||||
**Hierarchical chunking**:
|
||||
- Creates hierarchical chunk structure
|
||||
- Preserves document hierarchy
|
||||
- Useful for complex documents
|
||||
|
||||
### Table Serialization
|
||||
|
||||
Control how tables are represented in chunks:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunking_use_markdown_tables: false # Default: narrative format
|
||||
```
|
||||
|
||||
- `false`: Tables as narrative text ("Value A, Column 2 = Value B")
|
||||
- `true`: Tables as markdown (preserves table structure)
|
||||
|
||||
### Chunk Size
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 256 # Maximum tokens per chunk
|
||||
```
|
||||
|
||||
Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](qa.md#search-settings).
|
||||
|
||||
## File Monitoring
|
||||
|
||||
Set directories to monitor for automatic indexing:
|
||||
|
|
|
|||
|
|
@ -18,6 +18,34 @@ storage:
|
|||
!!! warning "Vacuum Retention Threshold"
|
||||
The `vacuum_retention_seconds` value should be larger than the typical time it takes to process and write a document. If a concurrent operation is in progress while vacuum runs, setting this value too low can cause race conditions where vacuum removes table versions that an in-flight operation still needs. The default of 86400 seconds (1 day) is conservative and safe for most use cases.
|
||||
|
||||
## Database Creation
|
||||
|
||||
Databases must be explicitly created before use:
|
||||
|
||||
**CLI:**
|
||||
```bash
|
||||
# Create in default location (see Configuration File Locations below)
|
||||
haiku-rag init
|
||||
|
||||
# Create at custom path
|
||||
haiku-rag init --db /path/to/database.lancedb
|
||||
```
|
||||
|
||||
**Python:**
|
||||
```python
|
||||
# Create at custom path
|
||||
async with HaikuRAG("/path/to/database.lancedb", create=True) as client:
|
||||
...
|
||||
|
||||
# Create in default location
|
||||
async with HaikuRAG(create=True) as client:
|
||||
...
|
||||
```
|
||||
|
||||
The [default location](index.md#configuration-file-locations) is platform-specific (e.g., `~/Library/Application Support/haiku.rag/` on macOS).
|
||||
|
||||
Operations on non-existent databases raise `FileNotFoundError`. This prevents accidental database creation from typos or misconfigured paths.
|
||||
|
||||
## Remote Storage
|
||||
|
||||
For remote storage, use the `lancedb` settings with various backends:
|
||||
|
|
@ -85,34 +113,6 @@ The recommended layout for production is "different buckets, same account, separ
|
|||
|
||||
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
|
||||
|
||||
## Database Creation
|
||||
|
||||
Databases must be explicitly created before use:
|
||||
|
||||
**CLI:**
|
||||
```bash
|
||||
# Create in default location (see Configuration File Locations below)
|
||||
haiku-rag init
|
||||
|
||||
# Create at custom path
|
||||
haiku-rag init --db /path/to/database.lancedb
|
||||
```
|
||||
|
||||
**Python:**
|
||||
```python
|
||||
# Create at custom path
|
||||
async with HaikuRAG("/path/to/database.lancedb", create=True) as client:
|
||||
...
|
||||
|
||||
# Create in default location
|
||||
async with HaikuRAG(create=True) as client:
|
||||
...
|
||||
```
|
||||
|
||||
The [default location](index.md#configuration-file-locations) is platform-specific (e.g., `~/Library/Application Support/haiku.rag/` on macOS).
|
||||
|
||||
Operations on non-existent databases raise `FileNotFoundError`. This prevents accidental database creation from typos or misconfigured paths.
|
||||
|
||||
## Vector Indexing
|
||||
|
||||
Configure vector search settings:
|
||||
|
|
|
|||
|
|
@ -2,6 +2,22 @@
|
|||
|
||||
haiku.rag provides processing primitives that let you build custom document pipelines. Use these when you need control over conversion, chunking, or embedding—for example, to preprocess content, use external services, or implement custom chunking logic.
|
||||
|
||||
## When to Use Custom Pipelines
|
||||
|
||||
Use the primitives when you need to:
|
||||
|
||||
- Preprocess or clean content before chunking
|
||||
- Filter or modify chunks before embedding
|
||||
- Use external embedding services
|
||||
- Implement custom chunking strategies
|
||||
- Debug or inspect intermediate processing steps
|
||||
|
||||
For standard use cases, prefer the convenience methods:
|
||||
|
||||
- `create_document()` - Create from text content
|
||||
- `create_document_from_source()` - Create from file or URL
|
||||
- `import_document()` - Store pre-processed documents with custom chunks
|
||||
|
||||
## Processing Primitives
|
||||
|
||||
The client exposes four primitives that can be composed into custom workflows:
|
||||
|
|
@ -233,19 +249,3 @@ async with HaikuRAG("database.lancedb", create=True) as client:
|
|||
chunks=embedded_chunks,
|
||||
)
|
||||
```
|
||||
|
||||
## When to Use Custom Pipelines
|
||||
|
||||
Use the primitives when you need to:
|
||||
|
||||
- Preprocess or clean content before chunking
|
||||
- Filter or modify chunks before embedding
|
||||
- Use external embedding services
|
||||
- Implement custom chunking strategies
|
||||
- Debug or inspect intermediate processing steps
|
||||
|
||||
For standard use cases, prefer the convenience methods:
|
||||
|
||||
- `create_document()` - Create from text content
|
||||
- `create_document_from_source()` - Create from file or URL
|
||||
- `import_document()` - Store pre-processed documents with custom chunks
|
||||
|
|
|
|||
100
docs/mcp.md
100
docs/mcp.md
|
|
@ -2,6 +2,56 @@
|
|||
|
||||
The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like Claude Desktop.
|
||||
|
||||
## Starting MCP Server
|
||||
|
||||
The MCP server supports Streamable HTTP and stdio transports:
|
||||
|
||||
```bash
|
||||
# Default streamable HTTP transport on port 8001
|
||||
haiku-rag serve --mcp
|
||||
|
||||
# Custom port
|
||||
haiku-rag serve --mcp --mcp-port 9000
|
||||
|
||||
# stdio transport (for Claude Desktop)
|
||||
haiku-rag serve --mcp --stdio
|
||||
|
||||
# Read-only mode (excludes write tools)
|
||||
haiku-rag --read-only serve --mcp --stdio
|
||||
```
|
||||
|
||||
**Read-only mode:** When `--read-only` is specified, write tools (`add_document_from_file`, `add_document_from_url`, `add_document_from_text`, `delete_document`) are not registered. Only search and query tools remain available.
|
||||
|
||||
## Claude Desktop Integration
|
||||
|
||||
Add to your Claude Desktop configuration (`claude_desktop_config.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With a custom database path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio", "--db", "/path/to/database.lancedb"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After restarting Claude Desktop, you can ask Claude to search your documents, add new content, or answer questions using your knowledge base.
|
||||
|
||||
## Available Tools
|
||||
|
||||
### Document Management
|
||||
|
|
@ -58,56 +108,6 @@ The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like
|
|||
- `document` (optional): Document title/ID to pre-load (can repeat)
|
||||
- Best for aggregation, computation, and multi-document analysis
|
||||
|
||||
## Starting MCP Server
|
||||
|
||||
The MCP server supports Streamable HTTP and stdio transports:
|
||||
|
||||
```bash
|
||||
# Default streamable HTTP transport on port 8001
|
||||
haiku-rag serve --mcp
|
||||
|
||||
# Custom port
|
||||
haiku-rag serve --mcp --mcp-port 9000
|
||||
|
||||
# stdio transport (for Claude Desktop)
|
||||
haiku-rag serve --mcp --stdio
|
||||
|
||||
# Read-only mode (excludes write tools)
|
||||
haiku-rag --read-only serve --mcp --stdio
|
||||
```
|
||||
|
||||
**Read-only mode:** When `--read-only` is specified, write tools (`add_document_from_file`, `add_document_from_url`, `add_document_from_text`, `delete_document`) are not registered. Only search and query tools remain available.
|
||||
|
||||
## Claude Desktop Integration
|
||||
|
||||
Add to your Claude Desktop configuration (`claude_desktop_config.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With a custom database path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio", "--db", "/path/to/database.lancedb"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After restarting Claude Desktop, you can ask Claude to search your documents, add new content, or answer questions using your knowledge base.
|
||||
|
||||
## Running with Other Services
|
||||
|
||||
Combine MCP with file monitoring:
|
||||
|
|
|
|||
222
docs/python.md
222
docs/python.md
|
|
@ -80,45 +80,6 @@ doc = await client.create_document_from_source(
|
|||
)
|
||||
```
|
||||
|
||||
### Importing Pre-Processed Documents
|
||||
|
||||
If you process documents externally or need custom processing, use `import_document()`:
|
||||
|
||||
```python
|
||||
from haiku.rag.store.models.chunk import Chunk
|
||||
|
||||
# Convert your source to a DoclingDocument
|
||||
docling_doc = await client.convert("path/to/document.pdf")
|
||||
|
||||
# Create chunks (embeddings optional - will be generated if missing)
|
||||
chunks = [
|
||||
Chunk(
|
||||
content="This is the first chunk",
|
||||
metadata={"section": "intro"},
|
||||
order=0,
|
||||
),
|
||||
Chunk(
|
||||
content="This is the second chunk",
|
||||
metadata={"section": "body"},
|
||||
embedding=[0.1] * 1024, # Optional: pre-computed embedding
|
||||
order=1,
|
||||
),
|
||||
]
|
||||
|
||||
# Import document with custom chunks
|
||||
doc = await client.import_document(
|
||||
docling_document=docling_doc,
|
||||
chunks=chunks,
|
||||
uri="doc://custom",
|
||||
title="Custom Document",
|
||||
metadata={"source": "external-pipeline"},
|
||||
)
|
||||
```
|
||||
|
||||
The `docling_document` provides rich metadata for visual grounding, page numbers, and section headings. Content is automatically extracted from the DoclingDocument.
|
||||
|
||||
See [Custom Processing Pipelines](custom-pipelines.md) for building pipelines with `convert()`, `chunk()`, and `embed_chunks()`.
|
||||
|
||||
### Retrieving Documents
|
||||
|
||||
By ID:
|
||||
|
|
@ -206,78 +167,6 @@ await client.update_document(document_id=doc.id, chunks=custom_chunks)
|
|||
await client.delete_document(doc.id)
|
||||
```
|
||||
|
||||
### Rebuilding the Database
|
||||
|
||||
```python
|
||||
from haiku.rag.client import RebuildMode
|
||||
|
||||
# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
|
||||
async for doc_id in client.rebuild_database():
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Re-chunk from stored content (no source file access)
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.RECHUNK):
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Only regenerate embeddings (fastest, keeps existing chunks)
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.EMBED_ONLY):
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Add VLM picture descriptions to an existing database — runs the VLM
|
||||
# over already-stored picture bytes, patches descriptions into the
|
||||
# docling blob, then re-chunks + re-embeds. Requires
|
||||
# processing.pictures='description' in the config.
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
||||
print(f"Described pictures in {doc_id}")
|
||||
```
|
||||
|
||||
**Rebuild modes:**
|
||||
|
||||
- `RebuildMode.FULL` - Re-convert from source files, re-chunk, re-embed (default)
|
||||
- `RebuildMode.RECHUNK` - Re-chunk from existing document content, re-embed
|
||||
- `RebuildMode.EMBED_ONLY` - Keep existing chunks, only regenerate embeddings
|
||||
- `RebuildMode.TITLE_ONLY` - Generate titles for untitled documents (no re-chunking or re-embedding)
|
||||
- `RebuildMode.DESCRIPTIONS` - Run the VLM over picture bytes already stored on `document_items.picture_data`, patch descriptions into the docling blob, re-chunk + re-embed. Skips the docling parse entirely. Idempotent — pictures already carrying `meta.description.text` are not re-described, so the operation is safe to re-run.
|
||||
|
||||
### Generating Titles
|
||||
|
||||
Generate a title for an existing document on demand:
|
||||
|
||||
```python
|
||||
title = await client.generate_title(doc)
|
||||
if title:
|
||||
await client.update_document(document_id=doc.id, title=title)
|
||||
```
|
||||
|
||||
Uses the same two-tier approach as automatic ingestion: structural extraction from DoclingDocument metadata first, with LLM fallback via `processing.title_model`. Unlike ingestion, this method does not catch exceptions — if the LLM call fails, the error propagates.
|
||||
|
||||
To batch-generate titles for all untitled documents, use `RebuildMode.TITLE_ONLY`:
|
||||
|
||||
```python
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.TITLE_ONLY):
|
||||
print(f"Generated title for {doc_id}")
|
||||
```
|
||||
|
||||
See [Automatic Title Generation](configuration/processing.md#automatic-title-generation) for configuration details.
|
||||
|
||||
## Maintenance
|
||||
|
||||
Run maintenance to optimize storage and prune old table versions:
|
||||
|
||||
```python
|
||||
await client.vacuum()
|
||||
```
|
||||
|
||||
This compacts tables and removes historical versions to keep disk usage in check. It’s safe to run anytime, for example after bulk imports or periodically in long‑running apps.
|
||||
|
||||
### Atomic Writes and Rollback
|
||||
|
||||
Document create and update operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores both the `documents` and `chunks` tables to their pre‑operation state using LanceDB’s table versioning.
|
||||
|
||||
- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, and internal rebuild/update flows.
|
||||
- Scope: Both document rows and all associated chunks are rolled back together.
|
||||
- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency; rollbacks occur immediately during the failing operation and are not impacted.
|
||||
|
||||
## Searching Documents
|
||||
|
||||
The search method performs native hybrid search (vector + full-text) using LanceDB with optional reranking for improved relevance:
|
||||
|
|
@ -481,3 +370,114 @@ result = await agent.run("What are the main findings?")
|
|||
```
|
||||
|
||||
See [Toolsets](tools.md) for the full API reference.
|
||||
|
||||
## Importing Pre-Processed Documents
|
||||
|
||||
If you process documents externally or need custom processing, use `import_document()`:
|
||||
|
||||
```python
|
||||
from haiku.rag.store.models.chunk import Chunk
|
||||
|
||||
# Convert your source to a DoclingDocument
|
||||
docling_doc = await client.convert("path/to/document.pdf")
|
||||
|
||||
# Create chunks (embeddings optional - will be generated if missing)
|
||||
chunks = [
|
||||
Chunk(
|
||||
content="This is the first chunk",
|
||||
metadata={"section": "intro"},
|
||||
order=0,
|
||||
),
|
||||
Chunk(
|
||||
content="This is the second chunk",
|
||||
metadata={"section": "body"},
|
||||
embedding=[0.1] * 1024, # Optional: pre-computed embedding
|
||||
order=1,
|
||||
),
|
||||
]
|
||||
|
||||
# Import document with custom chunks
|
||||
doc = await client.import_document(
|
||||
docling_document=docling_doc,
|
||||
chunks=chunks,
|
||||
uri="doc://custom",
|
||||
title="Custom Document",
|
||||
metadata={"source": "external-pipeline"},
|
||||
)
|
||||
```
|
||||
|
||||
The `docling_document` provides rich metadata for visual grounding, page numbers, and section headings. Content is automatically extracted from the DoclingDocument.
|
||||
|
||||
See [Custom Processing Pipelines](custom-pipelines.md) for building pipelines with `convert()`, `chunk()`, and `embed_chunks()`.
|
||||
|
||||
## Maintenance
|
||||
|
||||
Run maintenance to optimize storage and prune old table versions:
|
||||
|
||||
```python
|
||||
await client.vacuum()
|
||||
```
|
||||
|
||||
This compacts tables and removes historical versions to keep disk usage in check. It’s safe to run anytime, for example after bulk imports or periodically in long‑running apps.
|
||||
|
||||
### Rebuilding the Database
|
||||
|
||||
```python
|
||||
from haiku.rag.client import RebuildMode
|
||||
|
||||
# Full rebuild (default) - re-converts from source files, re-chunks, re-embeds
|
||||
async for doc_id in client.rebuild_database():
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Re-chunk from stored content (no source file access)
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.RECHUNK):
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Only regenerate embeddings (fastest, keeps existing chunks)
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.EMBED_ONLY):
|
||||
print(f"Processed document {doc_id}")
|
||||
|
||||
# Add VLM picture descriptions to an existing database — runs the VLM
|
||||
# over already-stored picture bytes, patches descriptions into the
|
||||
# docling blob, then re-chunks + re-embeds. Requires
|
||||
# processing.pictures='description' in the config.
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.DESCRIPTIONS):
|
||||
print(f"Described pictures in {doc_id}")
|
||||
```
|
||||
|
||||
**Rebuild modes:**
|
||||
|
||||
- `RebuildMode.FULL` - Re-convert from source files, re-chunk, re-embed (default)
|
||||
- `RebuildMode.RECHUNK` - Re-chunk from existing document content, re-embed
|
||||
- `RebuildMode.EMBED_ONLY` - Keep existing chunks, only regenerate embeddings
|
||||
- `RebuildMode.TITLE_ONLY` - Generate titles for untitled documents (no re-chunking or re-embedding)
|
||||
- `RebuildMode.DESCRIPTIONS` - Run the VLM over picture bytes already stored on `document_items.picture_data`, patch descriptions into the docling blob, re-chunk + re-embed. Skips the docling parse entirely. Idempotent — pictures already carrying `meta.description.text` are not re-described, so the operation is safe to re-run.
|
||||
|
||||
### Generating Titles
|
||||
|
||||
Generate a title for an existing document on demand:
|
||||
|
||||
```python
|
||||
title = await client.generate_title(doc)
|
||||
if title:
|
||||
await client.update_document(document_id=doc.id, title=title)
|
||||
```
|
||||
|
||||
Uses the same two-tier approach as automatic ingestion: structural extraction from DoclingDocument metadata first, with LLM fallback via `processing.title_model`. Unlike ingestion, this method does not catch exceptions — if the LLM call fails, the error propagates.
|
||||
|
||||
To batch-generate titles for all untitled documents, use `RebuildMode.TITLE_ONLY`:
|
||||
|
||||
```python
|
||||
async for doc_id in client.rebuild_database(mode=RebuildMode.TITLE_ONLY):
|
||||
print(f"Generated title for {doc_id}")
|
||||
```
|
||||
|
||||
See [Automatic Title Generation](configuration/processing.md#automatic-title-generation) for configuration details.
|
||||
|
||||
### Atomic Writes and Rollback
|
||||
|
||||
Document create and update operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores both the `documents` and `chunks` tables to their pre‑operation state using LanceDB’s table versioning.
|
||||
|
||||
- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, and internal rebuild/update flows.
|
||||
- Scope: Both document rows and all associated chunks are rolled back together.
|
||||
- Vacuum: Running `vacuum()` later prunes old versions for disk efficiency; rollbacks occur immediately during the failing operation and are not impacted.
|
||||
|
|
|
|||
|
|
@ -167,6 +167,6 @@ Each entry in `monitor.s3` runs as its own polling task. On every sweep the watc
|
|||
|
||||
`delete_orphans: true` is per-entry: a watcher only removes documents whose URI starts with that entry's `s3://bucket/prefix/`. Documents from other buckets, prefixes, or local-file sources are never touched.
|
||||
|
||||
### One-off ingestion
|
||||
## One-off ingestion
|
||||
|
||||
`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool — see [CLI → Add Documents](cli.md#add-documents).
|
||||
|
|
|
|||
|
|
@ -9,16 +9,6 @@ haiku.rag exposes its RAG capabilities as [haiku.skills](https://github.com/ggoz
|
|||
| [`rag`](rag.md) | Search, retrieve, and answer questions from the knowledge base |
|
||||
| [`rag-analysis`](analysis.md) | Computational analysis via code execution |
|
||||
|
||||
## Discovery
|
||||
|
||||
Skills are registered as Python entrypoints under `haiku.skills`. They are discovered automatically by `haiku.skills`:
|
||||
|
||||
```bash
|
||||
haiku-skills list --use-entrypoints
|
||||
# rag — Search, retrieve and analyze documents using RAG.
|
||||
# rag-analysis — Analyze documents using code execution in a sandboxed interpreter.
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
|
|
@ -39,6 +29,49 @@ agent = Agent(
|
|||
result = await agent.run("What documents do we have?")
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
Each skill manages its own state under a dedicated namespace. State is automatically synced via the AG-UI protocol when using `AGUIAdapter`.
|
||||
|
||||
```python
|
||||
rag_state = toolset.get_namespace("rag")
|
||||
analysis_state = toolset.get_namespace("analysis")
|
||||
```
|
||||
|
||||
See the individual skill pages for state model details.
|
||||
|
||||
## Database Path Resolution
|
||||
|
||||
Both skills resolve the database path in the same order:
|
||||
|
||||
1. `db_path` argument passed to `create_skill()`
|
||||
2. `HAIKU_RAG_DB` environment variable
|
||||
3. Config default (`config.storage.data_dir / "haiku.rag.lancedb"`)
|
||||
|
||||
## AG-UI Streaming
|
||||
|
||||
For web applications, use pydantic-ai's `AGUIAdapter` to stream tool calls, text, and state deltas:
|
||||
|
||||
```python
|
||||
from pydantic_ai.ui.ag_ui import AGUIAdapter
|
||||
|
||||
adapter = AGUIAdapter(agent=agent, run_input=run_input)
|
||||
event_stream = adapter.run_stream()
|
||||
sse_event_stream = adapter.encode_stream(event_stream)
|
||||
```
|
||||
|
||||
See the [Web application](../apps.md) reference implementation for an end-to-end example.
|
||||
|
||||
## Discovery
|
||||
|
||||
Skills are registered as Python entrypoints under `haiku.skills`. They are discovered automatically by `haiku.skills`:
|
||||
|
||||
```bash
|
||||
haiku-skills list --use-entrypoints
|
||||
# rag — Search, retrieve and analyze documents using RAG.
|
||||
# rag-analysis — Analyze documents using code execution in a sandboxed interpreter.
|
||||
```
|
||||
|
||||
## Generating Custom Skills
|
||||
|
||||
Use `create-skill` to generate a standalone skill package with an embedded database:
|
||||
|
|
@ -73,36 +106,3 @@ images = await visualize_chunk(chunk_id)
|
|||
```
|
||||
|
||||
See [CLI: Create Skill](../cli.md#create-skill) for all options.
|
||||
|
||||
## Database Path Resolution
|
||||
|
||||
Both skills resolve the database path in the same order:
|
||||
|
||||
1. `db_path` argument passed to `create_skill()`
|
||||
2. `HAIKU_RAG_DB` environment variable
|
||||
3. Config default (`config.storage.data_dir / "haiku.rag.lancedb"`)
|
||||
|
||||
## State Management
|
||||
|
||||
Each skill manages its own state under a dedicated namespace. State is automatically synced via the AG-UI protocol when using `AGUIAdapter`.
|
||||
|
||||
```python
|
||||
rag_state = toolset.get_namespace("rag")
|
||||
analysis_state = toolset.get_namespace("analysis")
|
||||
```
|
||||
|
||||
See the individual skill pages for state model details.
|
||||
|
||||
## AG-UI Streaming
|
||||
|
||||
For web applications, use pydantic-ai's `AGUIAdapter` to stream tool calls, text, and state deltas:
|
||||
|
||||
```python
|
||||
from pydantic_ai.ui.ag_ui import AGUIAdapter
|
||||
|
||||
adapter = AGUIAdapter(agent=agent, run_input=run_input)
|
||||
event_stream = adapter.run_stream()
|
||||
sse_event_stream = adapter.encode_stream(event_stream)
|
||||
```
|
||||
|
||||
See the [Web Application](../apps.md#web-application) for a complete implementation.
|
||||
|
|
|
|||
|
|
@ -42,14 +42,70 @@ Model and temperature selection affect answer quality directly — see [Provider
|
|||
| Embedding model | Yes — `haiku-rag rebuild` |
|
||||
| Search settings, reranking, prompts | No |
|
||||
|
||||
## Measuring Changes
|
||||
## Inspector
|
||||
|
||||
Use the inspector for ad-hoc exploration:
|
||||
The inspector is the fastest way to see what your model would actually receive for a given query. Run it against your database and step through the same hybrid search, context expansion, and chunk previews the rag skill uses at runtime. Press `c` on a chunk and you see the exact context the LLM would get back from a search hit.
|
||||
|
||||
```bash
|
||||
haiku-rag inspect
|
||||
haiku-rag inspect --db /path/to/database.lancedb
|
||||
```
|
||||
|
||||
!!! note
|
||||
Requires the `tui` extra: `pip install haiku.rag-slim[tui]` (included in the full `haiku.rag` package).
|
||||
|
||||
### Layout
|
||||
|
||||
Three panels:
|
||||
|
||||
- **Documents** (left): every document in the database.
|
||||
- **Chunks** (top right): chunks for the selected document.
|
||||
- **Detail view** (bottom right): full content and metadata.
|
||||
|
||||
|
||||
|
||||
### Keys
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| `Tab` | Cycle panels |
|
||||
| `↑` / `↓` | Navigate lists |
|
||||
| `/` | Search modal |
|
||||
| `c` | Context expansion modal (the chunk plus what the agent would see around it) |
|
||||
| `v` | Visual grounding modal (chunk highlighted on the page) |
|
||||
| `q` | Quit |
|
||||
|
||||
Mouse: click to select, scroll to view content.
|
||||
|
||||
### Search
|
||||
|
||||
Press `/` to open the search modal. Type a query and press `Enter`. The left panel lists results with relevance scores like `[0.95] content preview`. The right panel shows the full chunk and its metadata. `↑` / `↓` navigates results, `Enter` jumps to the document and chunk, `Esc` closes the modal. Search uses the same hybrid (vector + full-text) retrieval the rag skill uses.
|
||||
|
||||
### Context expansion (`c`)
|
||||
|
||||
Press `c` on a chunk to see the expanded context that would be fed to the rag skill. This is where you find out whether your `chunk_size`, `chunker_type`, and `max_context_chars` settings actually deliver the surrounding content the model needs. The modal shows:
|
||||
|
||||
- The expanded text. Section-aware expansion stays within section boundaries on structured documents and fills `max_context_chars` outward on unstructured ones.
|
||||
- Source document, content type, and relevance score.
|
||||
- Filtered noise. Footnotes, page headers and footers are excluded from structured documents.
|
||||
|
||||
If `qa.model.vision = true` is set, the modal also renders the picture bytes attached to that chunk, so you see exactly what the vision model would receive.
|
||||
|
||||
### Visual grounding (`v`)
|
||||
|
||||
Press `v` to highlight the chunk's bounding box on its page image. Useful for verifying chunk boundaries and seeing how Docling carved up the document.
|
||||
|
||||
- `←` / `→` to navigate pages when a chunk spans multiple pages.
|
||||
- `Esc` closes the modal.
|
||||
|
||||
|
||||
|
||||
Requirements: documents must have page images (default for PDFs), and the terminal must support inline images (iTerm2, WezTerm, Kitty). Plain-text documents added via `haiku-rag add` don't have visual grounding.
|
||||
|
||||
You can also visualize a chunk from the CLI without launching the TUI: `haiku-rag visualize <chunk_id>`.
|
||||
|
||||
## Measuring Changes
|
||||
|
||||
For systematic measurement, use the `evaluations/` workspace which provides retrieval metrics (MRR, MAP) and LLM-judged QA accuracy via `pydantic-evals`:
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ haiku-rag ask "Who wrote haiku.rag?"
|
|||
|
||||
## Where to go next
|
||||
|
||||
- [Chat](apps.md#chat-tui): sessions, citations, and the full TUI.
|
||||
- [Chat](chat.md): sessions, citations, and the full TUI.
|
||||
- [CLI reference](cli.md): every command.
|
||||
- [Python API](python.md): use haiku.rag in your own code.
|
||||
- [Skills](skills/index.md): the rag and rag-analysis skills the client wraps.
|
||||
|
|
|
|||
36
mkdocs.yml
36
mkdocs.yml
|
|
@ -55,31 +55,35 @@ plugins:
|
|||
# Material for MkDocs
|
||||
search:
|
||||
nav:
|
||||
- haiku.rag:
|
||||
- Get started:
|
||||
- index.md
|
||||
- Getting started: tutorial.md
|
||||
- Quickstart: tutorial.md
|
||||
- Installation: installation.md
|
||||
- Configuration:
|
||||
- configuration/index.md
|
||||
- Providers: configuration/providers.md
|
||||
- Search and Question Answering: configuration/qa.md
|
||||
- Document Processing: configuration/processing.md
|
||||
- Storage: configuration/storage.md
|
||||
- Prompts: configuration/prompts.md
|
||||
- Use it:
|
||||
- CLI: cli.md
|
||||
- Python: python.md
|
||||
- Custom Pipelines: custom-pipelines.md
|
||||
- Chat: chat.md
|
||||
- Configure:
|
||||
- configuration/index.md
|
||||
- Providers: configuration/providers.md
|
||||
- Search and question answering: configuration/qa.md
|
||||
- Document processing: configuration/processing.md
|
||||
- Storage: configuration/storage.md
|
||||
- Prompts: configuration/prompts.md
|
||||
- Tuning: tuning.md
|
||||
- Analysis: agents/analysis.md
|
||||
- Production:
|
||||
- Server: server.md
|
||||
- MCP: mcp.md
|
||||
- Remote processing: remote-processing.md
|
||||
- Build with it:
|
||||
- Python: python.md
|
||||
- Custom pipelines: custom-pipelines.md
|
||||
- Skills:
|
||||
- skills/index.md
|
||||
- RAG: skills/rag.md
|
||||
- Analysis: skills/analysis.md
|
||||
- Toolsets: tools.md
|
||||
- Applications: apps.md
|
||||
- Server: server.md
|
||||
- Remote processing: remote-processing.md
|
||||
- MCP: mcp.md
|
||||
- Web app: apps.md
|
||||
- Reference:
|
||||
- Benchmarks: benchmarks.md
|
||||
- Development: development.md
|
||||
- Changelog: changelog.md
|
||||
|
|
|
|||
Loading…
Reference in a new issue