diff --git a/README.md b/README.md index 0de7c536..98d21730 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB. -`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported. +`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama, LM Studio, vLLM) as well as commercial (OpenAI, VoyageAI) embedding providers are supported. ## Features @@ -144,10 +144,23 @@ async with HaikuRAG("database.lancedb") as client: Use with AI assistants like Claude Desktop: ```bash -haiku-rag serve --stdio +haiku-rag serve --mcp --stdio ``` -Provides tools for document management and search directly in your AI assistant. +Add to your Claude Desktop configuration: + +```json +{ + "mcpServers": { + "haiku-rag": { + "command": "haiku-rag", + "args": ["serve", "--mcp", "--stdio"] + } + } +} +``` + +Provides tools for document management, search, QA, and research directly in your AI assistant. ## Examples @@ -166,7 +179,10 @@ Full documentation at: https://ggozad.github.io/haiku.rag/ - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs - [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research -- [MCP Server](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration -- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance Benchmarks +- [Server](https://ggozad.github.io/haiku.rag/server/) - File monitoring, MCP, and AG-UI +- [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration +- [Inspector](https://ggozad.github.io/haiku.rag/inspector/) - Database browser TUI +- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks +- [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history mcp-name: io.github.ggozad/haiku-rag diff --git a/docs/agents.md b/docs/agents.md index ed38cd36..376ac14d 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -1,4 +1,4 @@ -## Agents +# Agents Three agentic flows are provided by haiku.rag: @@ -8,6 +8,8 @@ Three agentic flows are provided by haiku.rag: For an interactive example using Pydantic AI and AG-UI, see the [Interactive Research Assistant](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research) example ([demo video](https://vimeo.com/1128874386)). The demo uses a knowledge base containing haiku.rag's code and documentation. +See [QA and Research Configuration](configuration/qa-research.md) for configuring model, iterations, concurrency, and other settings. + ### Simple QA Agent diff --git a/docs/changelog.md b/docs/changelog.md new file mode 100644 index 00000000..786b75d5 --- /dev/null +++ b/docs/changelog.md @@ -0,0 +1 @@ +--8<-- "CHANGELOG.md" diff --git a/docs/index.md b/docs/index.md index 8f9403dc..0f64b0a2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,18 +4,18 @@ ## Features -- **Local LanceDB**: No need to run additional servers -- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own -- **Native Hybrid Search**: Vector search combined with full-text search using native LanceDB RRF reranking -- **Reranking**: Optional result reranking with MixedBread AI or Cohere -- **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic -- **File monitoring**: Automatically index files when run as a server -- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, code files and more. Or add a URL! +- **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure +- **Multiple embedding providers**: Ollama, LM Studio, VoyageAI, OpenAI, vLLM +- **Multiple QA providers**: Any provider/model supported by Pydantic AI (Ollama, LM Studio, OpenAI, Anthropic, etc.) +- **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking +- **Reranking**: Optional result reranking with MixedBread AI, Cohere, Zero Entropy, or vLLM +- **Question answering**: Built-in QA agents on your documents +- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI +- **File monitoring**: Auto-index files when run as server +- **Extended file format support**: Parse PDF, DOCX, HTML, Markdown, images, code files and more - **Flexible document processing**: Local processing with docling or remote with [docling-serve](remote-processing.md) -- **MCP server**: Exposes functionality as MCP tools -- **CLI commands**: Access all functionality from your terminal - - Add sources from text, files, or URLs, optionally with a human‑readable title -- **Python client**: Call `haiku.rag` from your own python applications +- **MCP server**: Expose as tools for AI assistants +- **CLI & Python API**: Use from command line or Python ## Quick Start diff --git a/docs/mcp.md b/docs/mcp.md index 1fe06f51..357ce383 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -1,30 +1,113 @@ # Model Context Protocol (MCP) -The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients. +The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like Claude Desktop. ## Available Tools ### Document Management -- `add_document_from_file` - Add documents from local file paths -- `add_document_from_url` - Add documents from URLs -- `add_document_from_text` - Add documents from raw text content -- `get_document` - Retrieve specific documents by ID -- `list_documents` - List all documents with pagination and optional filtering -- `delete_document` - Delete documents by ID +- **`add_document_from_file`** - Add documents from local file paths + - `file_path` (required): Path to the file + - `metadata` (optional): Key-value metadata + - `title` (optional): Human-readable title + +- **`add_document_from_url`** - Add documents from URLs + - `url` (required): URL to fetch + - `metadata` (optional): Key-value metadata + - `title` (optional): Human-readable title + +- **`add_document_from_text`** - Add documents from raw text content + - `content` (required): Text content + - `uri` (optional): URI identifier + - `metadata` (optional): Key-value metadata + - `title` (optional): Human-readable title + +- **`get_document`** - Retrieve a document by ID + - `document_id` (required): The document ID + +- **`list_documents`** - List documents with pagination and filtering + - `limit` (optional): Maximum number to return + - `offset` (optional): Number to skip + - `filter` (optional): SQL WHERE clause for filtering + +- **`delete_document`** - Delete a document by ID + - `document_id` (required): The document ID ### Search -- `search_documents` - Search documents using hybrid search (vector + full-text) +- **`search_documents`** - Search using hybrid search (vector + full-text) + - `query` (required): Search query + - `limit` (optional): Maximum results (default: 5) + +### Question Answering + +- **`ask_question`** - Ask questions about your documents + - `question` (required): The question to ask + - `cite` (optional): Include source citations (default: false) + - `deep` (optional): Use multi-agent deep QA for complex questions (default: false) + +- **`research_question`** - Run multi-agent research on complex topics + - `question` (required): The research question + - Returns a structured research report with findings, conclusions, and sources ## Starting MCP Server -The MCP server starts automatically with the serve command and supports Streamable HTTP and stdio transports: +The MCP server supports Streamable HTTP and stdio transports: ```bash -# Default streamable HTTP transport -haiku-rag serve +# Default streamable HTTP transport on port 8001 +haiku-rag serve --mcp + +# Custom port +haiku-rag serve --mcp --mcp-port 9000 # stdio transport (for Claude Desktop) -haiku-rag serve --stdio +haiku-rag serve --mcp --stdio ``` + +## Claude Desktop Integration + +Add to your Claude Desktop configuration (`claude_desktop_config.json`): + +```json +{ + "mcpServers": { + "haiku-rag": { + "command": "haiku-rag", + "args": ["serve", "--mcp", "--stdio"] + } + } +} +``` + +With a custom database path: + +```json +{ + "mcpServers": { + "haiku-rag": { + "command": "haiku-rag", + "args": ["serve", "--mcp", "--stdio", "--db", "/path/to/database.lancedb"] + } + } +} +``` + +After restarting Claude Desktop, you can ask Claude to search your documents, add new content, or answer questions using your knowledge base. + +## Running with Other Services + +Combine MCP with file monitoring or AG-UI: + +```bash +# MCP + file monitoring +haiku-rag serve --mcp --monitor + +# MCP + AG-UI streaming +haiku-rag serve --mcp --agui + +# All services +haiku-rag serve --mcp --monitor --agui +``` + +See [Server Mode](server.md) for details on file monitoring and AG-UI. diff --git a/docs/python.md b/docs/python.md index 12ac32ab..9413f0b9 100644 --- a/docs/python.md +++ b/docs/python.md @@ -9,7 +9,7 @@ from pathlib import Path from haiku.rag.client import HaikuRAG # Use as async context manager (recommended) -async with HaikuRAG("Path(path/to/database.lancedb")) as client: +async with HaikuRAG("path/to/database.lancedb") as client: # Your code here pass ``` diff --git a/docs/server.md b/docs/server.md index c4bab7d9..d08f7b3a 100644 --- a/docs/server.md +++ b/docs/server.md @@ -33,7 +33,7 @@ This will start file monitoring and MCP server on port 8001. ## File Monitoring -Configure directories to monitor in your `haiku.rag.yaml`: +Configure directories to monitor in your `haiku.rag.yaml` (see [Document Processing](configuration/processing.md#file-monitoring) for all options): ```yaml monitor: @@ -83,16 +83,39 @@ monitor: ### Supported Formats -The server can parse 40+ file formats including: -- PDF documents -- Microsoft Office (DOCX, XLSX, PPTX) -- HTML and Markdown -- Plain text files -- Code files (Python, JavaScript, etc.) -- Images (processed via OCR) -- And more... +The file monitor processes documents using [Docling](https://github.com/DS4SD/docling), which supports: -URLs are also supported for web content. +**Documents:** +- PDF (`.pdf`) - with OCR support for scanned documents +- Microsoft Word (`.docx`) +- Microsoft Excel (`.xlsx`) +- Microsoft PowerPoint (`.pptx`) +- HTML (`.html`, `.htm`) +- Markdown (`.md`) +- AsciiDoc (`.adoc`, `.asciidoc`) + +**Data formats:** +- CSV (`.csv`) +- JSON (`.json`) +- XML (`.xml`) + +**Images (via OCR):** +- PNG (`.png`) +- JPEG (`.jpg`, `.jpeg`) +- TIFF (`.tiff`, `.tif`) +- BMP (`.bmp`) + +**Code files:** +- Python (`.py`) +- JavaScript (`.js`) +- TypeScript (`.ts`) +- And other text-based code files + +**Plain text:** +- Text files (`.txt`) +- RST (`.rst`) + +URLs are also supported - the content is fetched and converted to markdown. ## AG-UI Server diff --git a/docs/tutorial.md b/docs/tutorial.md index 56bd9404..e46bc0a8 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -24,7 +24,7 @@ First, [get an OpenAI API key](https://platform.openai.com/api-keys). Install `haiku.rag` Python package using [uv](https://docs.astral.sh/uv/getting-started/installation/) or your favourite Python package manager: -```shell +```bash # Python 3.12+ needed uv pip install haiku.rag ``` @@ -57,7 +57,7 @@ See [Configuration](configuration/index.md) for all available options. Now you can add some pieces of text in the database: -```shell +```bash haiku-rag add "Python is the best programming language in the world, because it is flexible, with robust ecosystem, open source licensing and thousands of contributors" haiku-rag add "JavaScript is a popular programming language, but has a lot of warts" haiku-rag add "PHP is a bad programming language, because of spotted security history, horrible syntax and declining popularity" @@ -71,7 +71,7 @@ What will happen Now you can view your [LanceDB](https://lancedb.com/) database, and the embeddings it is configured for: -```shell +```bash haiku-rag info ``` @@ -103,7 +103,7 @@ using `OpenAIChatModel.request()`. The easiest way to do this is `ask` CLI command: -```shell +```bash haiku-rag ask "What is the best programming language in the world" ``` @@ -120,13 +120,13 @@ You can interact with Haiku RAG from Python in a similar manner as you can from First we need to install `ipython`, as built-in Python REPL does not support async blocks. -```shell +```bash uv pip install ipython ``` Run IPython: -```shell +```bash ipython ``` @@ -170,7 +170,7 @@ Haiku RAG can also handle types beyond plain text, including PDF, DOCX, HTML, an Here we add research papers about Python from [arxiv](https://arxiv.org/search/?query=python&searchtype=all&source=header) using URL retriever. -```shell +```bash # Better Python Programming for all: With the focus on Maintainability haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.org/pdf/2408.09134" @@ -180,7 +180,7 @@ haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.o Then we can query this: -```shell +```bash haiku-rag ask "Who wrote a paper about OpenTelemetry interoperability, and what was his take" ``` @@ -193,14 +193,14 @@ David Georg Reichelt from Lancaster University wrote a paper titled "Interoperab We can also add offline files, like PDFs. Here we add a local file to ensure OpenAI does not cheat - a file we know that should not be very well known in Internet: -```shell +```bash # This static file is supplied in haiku.rag repo haiku-rag add-src "examples/samples/PyCon Finland 2025 Schedule.html" ``` And then: -```shell +```bash haiku-rag ask "Who were presenting talks in Pycon Finland 2025? Can you give at least five different people." ``` @@ -214,6 +214,10 @@ The following people are presenting talks at PyCon Finland 2025: 5 Andreas Jung (also presenting another talk) - Talk: Debugging Python ``` -## Configuration +## Next Steps -See [Configuration page](./configuration/index.md) for complete documentation on YAML configuration and all available options. +- **[CLI Reference](cli.md)** - All available commands and options +- **[Python API](python.md)** - Use haiku.rag in your Python applications +- **[Agents](agents.md)** - Deep QA and multi-agent research workflows +- **[Configuration](configuration/index.md)** - Complete YAML configuration reference +- **[Server Mode](server.md)** - File monitoring, MCP server, and AG-UI streaming diff --git a/mkdocs.yml b/mkdocs.yml index 72aa2c9e..eb69281f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,6 +73,7 @@ nav: - MCP: mcp.md - Inspector: inspector.md - Benchmarks: benchmarks.md + - Changelog: changelog.md markdown_extensions: - admonition - attr_list @@ -83,7 +84,8 @@ markdown_extensions: pygments_lang_class: true use_pygments: true - pymdownx.inlinehilite - - pymdownx.snippets + - pymdownx.snippets: + base_path: ['.'] - pymdownx.superfences: custom_fences: - name: mermaid