Improve documentation
This commit is contained in:
Yiorgis Gozadinos
parent
5dfa07cb2c
commit
e10b207b47
9 changed files with 184 additions and 53 deletions
26
README.md
26
README.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Retrieval-Augmented Generation (RAG) library built on LanceDB.
|
||||
|
||||
`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
|
||||
`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama, LM Studio, vLLM) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
|
||||
|
||||
## Features
|
||||
|
||||
|
|
@ -144,10 +144,23 @@ async with HaikuRAG("database.lancedb") as client:
|
|||
Use with AI assistants like Claude Desktop:
|
||||
|
||||
```bash
|
||||
haiku-rag serve --stdio
|
||||
haiku-rag serve --mcp --stdio
|
||||
```
|
||||
|
||||
Provides tools for document management and search directly in your AI assistant.
|
||||
Add to your Claude Desktop configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Provides tools for document management, search, QA, and research directly in your AI assistant.
|
||||
|
||||
## Examples
|
||||
|
||||
|
|
@ -166,7 +179,10 @@ Full documentation at: https://ggozad.github.io/haiku.rag/
|
|||
- [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
|
||||
- [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
|
||||
- [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research
|
||||
- [MCP Server](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
|
||||
- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance Benchmarks
|
||||
- [Server](https://ggozad.github.io/haiku.rag/server/) - File monitoring, MCP, and AG-UI
|
||||
- [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
|
||||
- [Inspector](https://ggozad.github.io/haiku.rag/inspector/) - Database browser TUI
|
||||
- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks
|
||||
- [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history
|
||||
|
||||
mcp-name: io.github.ggozad/haiku-rag
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
## Agents
|
||||
# Agents
|
||||
|
||||
Three agentic flows are provided by haiku.rag:
|
||||
|
||||
|
|
@ -8,6 +8,8 @@ Three agentic flows are provided by haiku.rag:
|
|||
|
||||
For an interactive example using Pydantic AI and AG-UI, see the [Interactive Research Assistant](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research) example ([demo video](https://vimeo.com/1128874386)). The demo uses a knowledge base containing haiku.rag's code and documentation.
|
||||
|
||||
See [QA and Research Configuration](configuration/qa-research.md) for configuring model, iterations, concurrency, and other settings.
|
||||
|
||||
|
||||
### Simple QA Agent
|
||||
|
||||
|
|
|
|||
1
docs/changelog.md
Normal file
1
docs/changelog.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
--8<-- "CHANGELOG.md"
|
||||
|
|
@ -4,18 +4,18 @@
|
|||
|
||||
## Features
|
||||
|
||||
- **Local LanceDB**: No need to run additional servers
|
||||
- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own
|
||||
- **Native Hybrid Search**: Vector search combined with full-text search using native LanceDB RRF reranking
|
||||
- **Reranking**: Optional result reranking with MixedBread AI or Cohere
|
||||
- **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic
|
||||
- **File monitoring**: Automatically index files when run as a server
|
||||
- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, code files and more. Or add a URL!
|
||||
- **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
|
||||
- **Multiple embedding providers**: Ollama, LM Studio, VoyageAI, OpenAI, vLLM
|
||||
- **Multiple QA providers**: Any provider/model supported by Pydantic AI (Ollama, LM Studio, OpenAI, Anthropic, etc.)
|
||||
- **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking
|
||||
- **Reranking**: Optional result reranking with MixedBread AI, Cohere, Zero Entropy, or vLLM
|
||||
- **Question answering**: Built-in QA agents on your documents
|
||||
- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI
|
||||
- **File monitoring**: Auto-index files when run as server
|
||||
- **Extended file format support**: Parse PDF, DOCX, HTML, Markdown, images, code files and more
|
||||
- **Flexible document processing**: Local processing with docling or remote with [docling-serve](remote-processing.md)
|
||||
- **MCP server**: Exposes functionality as MCP tools
|
||||
- **CLI commands**: Access all functionality from your terminal
|
||||
- Add sources from text, files, or URLs, optionally with a human‑readable title
|
||||
- **Python client**: Call `haiku.rag` from your own python applications
|
||||
- **MCP server**: Expose as tools for AI assistants
|
||||
- **CLI & Python API**: Use from command line or Python
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
|
|
|||
107
docs/mcp.md
107
docs/mcp.md
|
|
@ -1,30 +1,113 @@
|
|||
# Model Context Protocol (MCP)
|
||||
|
||||
The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients.
|
||||
The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like Claude Desktop.
|
||||
|
||||
## Available Tools
|
||||
|
||||
### Document Management
|
||||
|
||||
- `add_document_from_file` - Add documents from local file paths
|
||||
- `add_document_from_url` - Add documents from URLs
|
||||
- `add_document_from_text` - Add documents from raw text content
|
||||
- `get_document` - Retrieve specific documents by ID
|
||||
- `list_documents` - List all documents with pagination and optional filtering
|
||||
- `delete_document` - Delete documents by ID
|
||||
- **`add_document_from_file`** - Add documents from local file paths
|
||||
- `file_path` (required): Path to the file
|
||||
- `metadata` (optional): Key-value metadata
|
||||
- `title` (optional): Human-readable title
|
||||
|
||||
- **`add_document_from_url`** - Add documents from URLs
|
||||
- `url` (required): URL to fetch
|
||||
- `metadata` (optional): Key-value metadata
|
||||
- `title` (optional): Human-readable title
|
||||
|
||||
- **`add_document_from_text`** - Add documents from raw text content
|
||||
- `content` (required): Text content
|
||||
- `uri` (optional): URI identifier
|
||||
- `metadata` (optional): Key-value metadata
|
||||
- `title` (optional): Human-readable title
|
||||
|
||||
- **`get_document`** - Retrieve a document by ID
|
||||
- `document_id` (required): The document ID
|
||||
|
||||
- **`list_documents`** - List documents with pagination and filtering
|
||||
- `limit` (optional): Maximum number to return
|
||||
- `offset` (optional): Number to skip
|
||||
- `filter` (optional): SQL WHERE clause for filtering
|
||||
|
||||
- **`delete_document`** - Delete a document by ID
|
||||
- `document_id` (required): The document ID
|
||||
|
||||
### Search
|
||||
|
||||
- `search_documents` - Search documents using hybrid search (vector + full-text)
|
||||
- **`search_documents`** - Search using hybrid search (vector + full-text)
|
||||
- `query` (required): Search query
|
||||
- `limit` (optional): Maximum results (default: 5)
|
||||
|
||||
### Question Answering
|
||||
|
||||
- **`ask_question`** - Ask questions about your documents
|
||||
- `question` (required): The question to ask
|
||||
- `cite` (optional): Include source citations (default: false)
|
||||
- `deep` (optional): Use multi-agent deep QA for complex questions (default: false)
|
||||
|
||||
- **`research_question`** - Run multi-agent research on complex topics
|
||||
- `question` (required): The research question
|
||||
- Returns a structured research report with findings, conclusions, and sources
|
||||
|
||||
## Starting MCP Server
|
||||
|
||||
The MCP server starts automatically with the serve command and supports Streamable HTTP and stdio transports:
|
||||
The MCP server supports Streamable HTTP and stdio transports:
|
||||
|
||||
```bash
|
||||
# Default streamable HTTP transport
|
||||
haiku-rag serve
|
||||
# Default streamable HTTP transport on port 8001
|
||||
haiku-rag serve --mcp
|
||||
|
||||
# Custom port
|
||||
haiku-rag serve --mcp --mcp-port 9000
|
||||
|
||||
# stdio transport (for Claude Desktop)
|
||||
haiku-rag serve --stdio
|
||||
haiku-rag serve --mcp --stdio
|
||||
```
|
||||
|
||||
## Claude Desktop Integration
|
||||
|
||||
Add to your Claude Desktop configuration (`claude_desktop_config.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With a custom database path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"haiku-rag": {
|
||||
"command": "haiku-rag",
|
||||
"args": ["serve", "--mcp", "--stdio", "--db", "/path/to/database.lancedb"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After restarting Claude Desktop, you can ask Claude to search your documents, add new content, or answer questions using your knowledge base.
|
||||
|
||||
## Running with Other Services
|
||||
|
||||
Combine MCP with file monitoring or AG-UI:
|
||||
|
||||
```bash
|
||||
# MCP + file monitoring
|
||||
haiku-rag serve --mcp --monitor
|
||||
|
||||
# MCP + AG-UI streaming
|
||||
haiku-rag serve --mcp --agui
|
||||
|
||||
# All services
|
||||
haiku-rag serve --mcp --monitor --agui
|
||||
```
|
||||
|
||||
See [Server Mode](server.md) for details on file monitoring and AG-UI.
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ from pathlib import Path
|
|||
from haiku.rag.client import HaikuRAG
|
||||
|
||||
# Use as async context manager (recommended)
|
||||
async with HaikuRAG("Path(path/to/database.lancedb")) as client:
|
||||
async with HaikuRAG("path/to/database.lancedb") as client:
|
||||
# Your code here
|
||||
pass
|
||||
```
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ This will start file monitoring and MCP server on port 8001.
|
|||
|
||||
## File Monitoring
|
||||
|
||||
Configure directories to monitor in your `haiku.rag.yaml`:
|
||||
Configure directories to monitor in your `haiku.rag.yaml` (see [Document Processing](configuration/processing.md#file-monitoring) for all options):
|
||||
|
||||
```yaml
|
||||
monitor:
|
||||
|
|
@ -83,16 +83,39 @@ monitor:
|
|||
|
||||
### Supported Formats
|
||||
|
||||
The server can parse 40+ file formats including:
|
||||
- PDF documents
|
||||
- Microsoft Office (DOCX, XLSX, PPTX)
|
||||
- HTML and Markdown
|
||||
- Plain text files
|
||||
- Code files (Python, JavaScript, etc.)
|
||||
- Images (processed via OCR)
|
||||
- And more...
|
||||
The file monitor processes documents using [Docling](https://github.com/DS4SD/docling), which supports:
|
||||
|
||||
URLs are also supported for web content.
|
||||
**Documents:**
|
||||
- PDF (`.pdf`) - with OCR support for scanned documents
|
||||
- Microsoft Word (`.docx`)
|
||||
- Microsoft Excel (`.xlsx`)
|
||||
- Microsoft PowerPoint (`.pptx`)
|
||||
- HTML (`.html`, `.htm`)
|
||||
- Markdown (`.md`)
|
||||
- AsciiDoc (`.adoc`, `.asciidoc`)
|
||||
|
||||
**Data formats:**
|
||||
- CSV (`.csv`)
|
||||
- JSON (`.json`)
|
||||
- XML (`.xml`)
|
||||
|
||||
**Images (via OCR):**
|
||||
- PNG (`.png`)
|
||||
- JPEG (`.jpg`, `.jpeg`)
|
||||
- TIFF (`.tiff`, `.tif`)
|
||||
- BMP (`.bmp`)
|
||||
|
||||
**Code files:**
|
||||
- Python (`.py`)
|
||||
- JavaScript (`.js`)
|
||||
- TypeScript (`.ts`)
|
||||
- And other text-based code files
|
||||
|
||||
**Plain text:**
|
||||
- Text files (`.txt`)
|
||||
- RST (`.rst`)
|
||||
|
||||
URLs are also supported - the content is fetched and converted to markdown.
|
||||
|
||||
## AG-UI Server
|
||||
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ First, [get an OpenAI API key](https://platform.openai.com/api-keys).
|
|||
|
||||
Install `haiku.rag` Python package using [uv](https://docs.astral.sh/uv/getting-started/installation/) or your favourite Python package manager:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
# Python 3.12+ needed
|
||||
uv pip install haiku.rag
|
||||
```
|
||||
|
|
@ -57,7 +57,7 @@ See [Configuration](configuration/index.md) for all available options.
|
|||
|
||||
Now you can add some pieces of text in the database:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
haiku-rag add "Python is the best programming language in the world, because it is flexible, with robust ecosystem, open source licensing and thousands of contributors"
|
||||
haiku-rag add "JavaScript is a popular programming language, but has a lot of warts"
|
||||
haiku-rag add "PHP is a bad programming language, because of spotted security history, horrible syntax and declining popularity"
|
||||
|
|
@ -71,7 +71,7 @@ What will happen
|
|||
|
||||
Now you can view your [LanceDB](https://lancedb.com/) database, and the embeddings it is configured for:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
haiku-rag info
|
||||
```
|
||||
|
||||
|
|
@ -103,7 +103,7 @@ using `OpenAIChatModel.request()`.
|
|||
|
||||
The easiest way to do this is `ask` CLI command:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
haiku-rag ask "What is the best programming language in the world"
|
||||
```
|
||||
|
||||
|
|
@ -120,13 +120,13 @@ You can interact with Haiku RAG from Python in a similar manner as you can from
|
|||
|
||||
First we need to install `ipython`, as built-in Python REPL does not support async blocks.
|
||||
|
||||
```shell
|
||||
```bash
|
||||
uv pip install ipython
|
||||
```
|
||||
|
||||
Run IPython:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
ipython
|
||||
```
|
||||
|
||||
|
|
@ -170,7 +170,7 @@ Haiku RAG can also handle types beyond plain text, including PDF, DOCX, HTML, an
|
|||
|
||||
Here we add research papers about Python from [arxiv](https://arxiv.org/search/?query=python&searchtype=all&source=header) using URL retriever.
|
||||
|
||||
```shell
|
||||
```bash
|
||||
# Better Python Programming for all: With the focus on Maintainability
|
||||
haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.org/pdf/2408.09134"
|
||||
|
||||
|
|
@ -180,7 +180,7 @@ haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.o
|
|||
|
||||
Then we can query this:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
haiku-rag ask "Who wrote a paper about OpenTelemetry interoperability, and what was his take"
|
||||
```
|
||||
|
||||
|
|
@ -193,14 +193,14 @@ David Georg Reichelt from Lancaster University wrote a paper titled "Interoperab
|
|||
|
||||
We can also add offline files, like PDFs. Here we add a local file to ensure OpenAI does not cheat - a file we know that should not be very well known in Internet:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
# This static file is supplied in haiku.rag repo
|
||||
haiku-rag add-src "examples/samples/PyCon Finland 2025 Schedule.html"
|
||||
```
|
||||
|
||||
And then:
|
||||
|
||||
```shell
|
||||
```bash
|
||||
haiku-rag ask "Who were presenting talks in Pycon Finland 2025? Can you give at least five different people."
|
||||
```
|
||||
|
||||
|
|
@ -214,6 +214,10 @@ The following people are presenting talks at PyCon Finland 2025:
|
|||
5 Andreas Jung (also presenting another talk) - Talk: Debugging Python
|
||||
```
|
||||
|
||||
## Configuration
|
||||
## Next Steps
|
||||
|
||||
See [Configuration page](./configuration/index.md) for complete documentation on YAML configuration and all available options.
|
||||
- **[CLI Reference](cli.md)** - All available commands and options
|
||||
- **[Python API](python.md)** - Use haiku.rag in your Python applications
|
||||
- **[Agents](agents.md)** - Deep QA and multi-agent research workflows
|
||||
- **[Configuration](configuration/index.md)** - Complete YAML configuration reference
|
||||
- **[Server Mode](server.md)** - File monitoring, MCP server, and AG-UI streaming
|
||||
|
|
|
|||
|
|
@ -73,6 +73,7 @@ nav:
|
|||
- MCP: mcp.md
|
||||
- Inspector: inspector.md
|
||||
- Benchmarks: benchmarks.md
|
||||
- Changelog: changelog.md
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- attr_list
|
||||
|
|
@ -83,7 +84,8 @@ markdown_extensions:
|
|||
pygments_lang_class: true
|
||||
use_pygments: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.snippets
|
||||
- pymdownx.snippets:
|
||||
base_path: ['.']
|
||||
- pymdownx.superfences:
|
||||
custom_fences:
|
||||
- name: mermaid
|
||||
|
|
|
|||
Loading…
Reference in a new issue