Merge pull request #173 from ggozad/chore/improve-documentation

Improve documentation

README.md

@@ -2,7 +2,7 @@
 
 Retrieval-Augmented Generation (RAG) library built on LanceDB.
 
-`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
+`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama, LM Studio, vLLM) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
 
 ## Features
 
@@ -144,10 +144,23 @@ async with HaikuRAG("database.lancedb") as client:
 Use with AI assistants like Claude Desktop:
 
 ```bash
-haiku-rag serve --stdio
+haiku-rag serve --mcp --stdio
 ```
 
-Provides tools for document management and search directly in your AI assistant.
+Add to your Claude Desktop configuration:
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["serve", "--mcp", "--stdio"]
+    }
+  }
+}
+```
+
+Provides tools for document management, search, QA, and research directly in your AI assistant.
 
 ## Examples
 
@@ -166,7 +179,10 @@ Full documentation at: https://ggozad.github.io/haiku.rag/
 - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
 - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
 - [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research
-- [MCP Server](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
+- [Server](https://ggozad.github.io/haiku.rag/server/) - File monitoring, MCP, and AG-UI
-- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance Benchmarks
+- [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration
+- [Inspector](https://ggozad.github.io/haiku.rag/inspector/) - Database browser TUI
+- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance benchmarks
+- [Changelog](https://ggozad.github.io/haiku.rag/changelog/) - Version history
 
 mcp-name: io.github.ggozad/haiku-rag

docs/agents.md

@@ -1,4 +1,4 @@
-## Agents
+# Agents
 
 Three agentic flows are provided by haiku.rag:
 
@@ -8,6 +8,8 @@ Three agentic flows are provided by haiku.rag:
 
 For an interactive example using Pydantic AI and AG-UI, see the [Interactive Research Assistant](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research) example ([demo video](https://vimeo.com/1128874386)). The demo uses a knowledge base containing haiku.rag's code and documentation.
 
+See [QA and Research Configuration](configuration/qa-research.md) for configuring model, iterations, concurrency, and other settings.
+
 
 ### Simple QA Agent
 

docs/changelog.md

@@ -0,0 +1 @@
+--8<-- "CHANGELOG.md"

docs/index.md

@@ -4,18 +4,18 @@
 
 ## Features
 
-- **Local LanceDB**: No need to run additional servers
+- **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
-- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own
+- **Multiple embedding providers**: Ollama, LM Studio, VoyageAI, OpenAI, vLLM
-- **Native Hybrid Search**: Vector search combined with full-text search using native LanceDB RRF reranking
+- **Multiple QA providers**: Any provider/model supported by Pydantic AI (Ollama, LM Studio, OpenAI, Anthropic, etc.)
-- **Reranking**: Optional result reranking with MixedBread AI or Cohere
+- **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking
-- **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic
+- **Reranking**: Optional result reranking with MixedBread AI, Cohere, Zero Entropy, or vLLM
-- **File monitoring**: Automatically index files when run as a server
+- **Question answering**: Built-in QA agents on your documents
-- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, code files and more. Or add a URL!
+- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI
+- **File monitoring**: Auto-index files when run as server
+- **Extended file format support**: Parse PDF, DOCX, HTML, Markdown, images, code files and more
 - **Flexible document processing**: Local processing with docling or remote with [docling-serve](remote-processing.md)
-- **MCP server**: Exposes functionality as MCP tools
+- **MCP server**: Expose as tools for AI assistants
-- **CLI commands**: Access all functionality from your terminal
+- **CLI & Python API**: Use from command line or Python
-  - Add sources from text, files, or URLs, optionally with a human‑readable title
-- **Python client**: Call `haiku.rag` from your own python applications
 
 ## Quick Start
 

docs/mcp.md

@@ -1,30 +1,113 @@
 # Model Context Protocol (MCP)
 
-The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients.
+The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like Claude Desktop.
 
 ## Available Tools
 
 ### Document Management
 
-- `add_document_from_file` - Add documents from local file paths
+- **`add_document_from_file`** - Add documents from local file paths
-- `add_document_from_url` - Add documents from URLs
+  - `file_path` (required): Path to the file
-- `add_document_from_text` - Add documents from raw text content
+  - `metadata` (optional): Key-value metadata
-- `get_document` - Retrieve specific documents by ID
+  - `title` (optional): Human-readable title
-- `list_documents` - List all documents with pagination and optional filtering
+
-- `delete_document` - Delete documents by ID
+- **`add_document_from_url`** - Add documents from URLs
+  - `url` (required): URL to fetch
+  - `metadata` (optional): Key-value metadata
+  - `title` (optional): Human-readable title
+
+- **`add_document_from_text`** - Add documents from raw text content
+  - `content` (required): Text content
+  - `uri` (optional): URI identifier
+  - `metadata` (optional): Key-value metadata
+  - `title` (optional): Human-readable title
+
+- **`get_document`** - Retrieve a document by ID
+  - `document_id` (required): The document ID
+
+- **`list_documents`** - List documents with pagination and filtering
+  - `limit` (optional): Maximum number to return
+  - `offset` (optional): Number to skip
+  - `filter` (optional): SQL WHERE clause for filtering
+
+- **`delete_document`** - Delete a document by ID
+  - `document_id` (required): The document ID
 
 ### Search
 
-- `search_documents` - Search documents using hybrid search (vector + full-text)
+- **`search_documents`** - Search using hybrid search (vector + full-text)
+  - `query` (required): Search query
+  - `limit` (optional): Maximum results (default: 5)
+
+### Question Answering
+
+- **`ask_question`** - Ask questions about your documents
+  - `question` (required): The question to ask
+  - `cite` (optional): Include source citations (default: false)
+  - `deep` (optional): Use multi-agent deep QA for complex questions (default: false)
+
+- **`research_question`** - Run multi-agent research on complex topics
+  - `question` (required): The research question
+  - Returns a structured research report with findings, conclusions, and sources
 
 ## Starting MCP Server
 
-The MCP server starts automatically with the serve command and supports Streamable HTTP and stdio transports:
+The MCP server supports Streamable HTTP and stdio transports:
 
 ```bash
-# Default streamable HTTP transport
+# Default streamable HTTP transport on port 8001
-haiku-rag serve
+haiku-rag serve --mcp
+
+# Custom port
+haiku-rag serve --mcp --mcp-port 9000
 
 # stdio transport (for Claude Desktop)
-haiku-rag serve --stdio
+haiku-rag serve --mcp --stdio
 ```
+
+## Claude Desktop Integration
+
+Add to your Claude Desktop configuration (`claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["serve", "--mcp", "--stdio"]
+    }
+  }
+}
+```
+
+With a custom database path:
+
+```json
+{
+  "mcpServers": {
+    "haiku-rag": {
+      "command": "haiku-rag",
+      "args": ["serve", "--mcp", "--stdio", "--db", "/path/to/database.lancedb"]
+    }
+  }
+}
+```
+
+After restarting Claude Desktop, you can ask Claude to search your documents, add new content, or answer questions using your knowledge base.
+
+## Running with Other Services
+
+Combine MCP with file monitoring or AG-UI:
+
+```bash
+# MCP + file monitoring
+haiku-rag serve --mcp --monitor
+
+# MCP + AG-UI streaming
+haiku-rag serve --mcp --agui
+
+# All services
+haiku-rag serve --mcp --monitor --agui
+```
+
+See [Server Mode](server.md) for details on file monitoring and AG-UI.

docs/python.md

@@ -9,7 +9,7 @@ from pathlib import Path
 from haiku.rag.client import HaikuRAG
 
 # Use as async context manager (recommended)
-async with HaikuRAG("Path(path/to/database.lancedb")) as client:
+async with HaikuRAG("path/to/database.lancedb") as client:
     # Your code here
     pass
 ```

docs/server.md

@@ -33,7 +33,7 @@ This will start file monitoring and MCP server on port 8001.
 
 ## File Monitoring
 
-Configure directories to monitor in your `haiku.rag.yaml`:
+Configure directories to monitor in your `haiku.rag.yaml` (see [Document Processing](configuration/processing.md#file-monitoring) for all options):
 
 ```yaml
 monitor:
@@ -83,16 +83,39 @@ monitor:
 
 ### Supported Formats
 
-The server can parse 40+ file formats including:
+The file monitor processes documents using [Docling](https://github.com/DS4SD/docling), which supports:
-- PDF documents
-- Microsoft Office (DOCX, XLSX, PPTX)
-- HTML and Markdown
-- Plain text files
-- Code files (Python, JavaScript, etc.)
-- Images (processed via OCR)
-- And more...
 
-URLs are also supported for web content.
+**Documents:**
+- PDF (`.pdf`) - with OCR support for scanned documents
+- Microsoft Word (`.docx`)
+- Microsoft Excel (`.xlsx`)
+- Microsoft PowerPoint (`.pptx`)
+- HTML (`.html`, `.htm`)
+- Markdown (`.md`)
+- AsciiDoc (`.adoc`, `.asciidoc`)
+
+**Data formats:**
+- CSV (`.csv`)
+- JSON (`.json`)
+- XML (`.xml`)
+
+**Images (via OCR):**
+- PNG (`.png`)
+- JPEG (`.jpg`, `.jpeg`)
+- TIFF (`.tiff`, `.tif`)
+- BMP (`.bmp`)
+
+**Code files:**
+- Python (`.py`)
+- JavaScript (`.js`)
+- TypeScript (`.ts`)
+- And other text-based code files
+
+**Plain text:**
+- Text files (`.txt`)
+- RST (`.rst`)
+
+URLs are also supported - the content is fetched and converted to markdown.
 
 ## AG-UI Server
 

docs/tutorial.md

@@ -24,7 +24,7 @@ First, [get an OpenAI API key](https://platform.openai.com/api-keys).
 
 Install `haiku.rag` Python package using [uv](https://docs.astral.sh/uv/getting-started/installation/) or your favourite Python package manager:
 
-```shell
+```bash
 # Python 3.12+ needed
 uv pip install haiku.rag
 ```
@@ -57,7 +57,7 @@ See [Configuration](configuration/index.md) for all available options.
 
 Now you can add some pieces of text in the database:
 
-```shell
+```bash
 haiku-rag add "Python is the best programming language in the world, because it is flexible, with robust ecosystem, open source licensing and thousands of contributors"
 haiku-rag add "JavaScript is a popular programming language, but has a lot of warts"
 haiku-rag add "PHP is a bad programming language, because of spotted security history, horrible syntax and declining popularity"
@@ -71,7 +71,7 @@ What will happen
 
 Now you can view your [LanceDB](https://lancedb.com/) database, and the embeddings it is configured for:
 
-```shell
+```bash
 haiku-rag info
 ```
 
@@ -103,7 +103,7 @@ using `OpenAIChatModel.request()`.
 
 The easiest way to do this is `ask` CLI command:
 
-```shell
+```bash
 haiku-rag ask "What is the best programming language in the world"
 ```
 
@@ -120,13 +120,13 @@ You can interact with Haiku RAG from Python in a similar manner as you can from
 
 First we need to install `ipython`, as built-in Python REPL does not support async blocks.
 
-```shell
+```bash
 uv pip install ipython
 ```
 
 Run IPython:
 
-```shell
+```bash
 ipython
 ```
 
@@ -170,7 +170,7 @@ Haiku RAG can also handle types beyond plain text, including PDF, DOCX, HTML, an
 
 Here we add research papers about Python from [arxiv](https://arxiv.org/search/?query=python&searchtype=all&source=header) using URL retriever.
 
-```shell
+```bash
 # Better Python Programming for all: With the focus on Maintainability
 haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.org/pdf/2408.09134"
 
@@ -180,7 +180,7 @@ haiku-rag add-src --meta collection="Interesting Python papers" "https://arxiv.o
 
 Then we can query this:
 
-```shell
+```bash
 haiku-rag ask "Who wrote a paper about OpenTelemetry interoperability, and what was his take"
 ```
 
@@ -193,14 +193,14 @@ David Georg Reichelt from Lancaster University wrote a paper titled "Interoperab
 
 We can also add offline files, like PDFs. Here we add a local file to ensure OpenAI does not cheat - a file we know that should not be very well known in Internet:
 
-```shell
+```bash
 # This static file is supplied in haiku.rag repo
 haiku-rag add-src "examples/samples/PyCon Finland 2025 Schedule.html"
 ```
 
 And then:
 
-```shell
+```bash
 haiku-rag ask "Who were presenting talks in Pycon Finland 2025? Can you give at least five different people."
 ```
 
@@ -214,6 +214,10 @@ The following people are presenting talks at PyCon Finland 2025:
  5 Andreas Jung (also presenting another talk) - Talk: Debugging Python
 ```
 
-## Configuration
+## Next Steps
 
-See [Configuration page](./configuration/index.md) for complete documentation on YAML configuration and all available options.
+- **[CLI Reference](cli.md)** - All available commands and options
+- **[Python API](python.md)** - Use haiku.rag in your Python applications
+- **[Agents](agents.md)** - Deep QA and multi-agent research workflows
+- **[Configuration](configuration/index.md)** - Complete YAML configuration reference
+- **[Server Mode](server.md)** - File monitoring, MCP server, and AG-UI streaming

mkdocs.yml

@@ -73,6 +73,7 @@ nav:
       - MCP: mcp.md
       - Inspector: inspector.md
       - Benchmarks: benchmarks.md
+      - Changelog: changelog.md
 markdown_extensions:
   - admonition
   - attr_list
@@ -83,7 +84,8 @@ markdown_extensions:
       pygments_lang_class: true
       use_pygments: true
   - pymdownx.inlinehilite
-  - pymdownx.snippets
+  - pymdownx.snippets:
+      base_path: ['.']
   - pymdownx.superfences:
       custom_fences:
         - name: mermaid