From 56cf6ebfb6d0db918d849579038a52475f84633e Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Fri, 30 Jan 2026 12:08:39 +0200 Subject: [PATCH] Docs --- CHANGELOG.md | 9 ++ README.md | 5 + docs/agents.md | 3 +- docs/architecture.md | 18 ++- docs/cli.md | 27 ++++ docs/configuration/qa-research.md | 21 +++ docs/index.md | 2 + docs/mcp.md | 6 + docs/python.md | 25 ++++ docs/rlm.md | 204 ++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 11 files changed, 319 insertions(+), 2 deletions(-) create mode 100644 docs/rlm.md diff --git a/CHANGELOG.md b/CHANGELOG.md index bd0120ee..c8444f2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ - **docling-serve Chunker OCR Options**: The docling-serve chunker now respects OCR settings from `conversion_options` - Passes `do_ocr`, `force_ocr`, `ocr_engine`, and `ocr_lang` to the chunking API - Allows disabling OCR via config when running docling-serve in read-only containers +- **RLM Agent (Recursive Language Model)**: New agent for complex analytical tasks via sandboxed Python code execution + - Solves problems traditional RAG can't handle: aggregation, computation, multi-document analysis + - Sandboxed execution with safe builtins and allowed imports (json, re, math, statistics, etc.) + - Available functions: `search()`, `list_documents()`, `get_document()`, `get_docling_document()`, `llm()` + - Pre-loaded documents support via `documents` variable + - Context filter for scoping searches without LLM control + - New `client.rlm(question)` method on HaikuRAG client + - New `haiku-rag rlm` CLI command + - New `rlm_question` MCP tool ### Fixed diff --git a/README.md b/README.md index 2351b518..8a63aeac 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p - **Question answering** — QA agents with citations (page numbers, section headings) - **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM - **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize +- **RLM agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis) - **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory - **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion - **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI @@ -64,6 +65,9 @@ haiku-rag ask "How does the proposed method compare to the baseline on MMLU?" -- # Research mode — iterative planning and search haiku-rag research "What are the limitations of the approach?" +# RLM mode — complex analytical tasks via code execution +haiku-rag rlm "How many documents mention transformers?" + # Interactive chat — multi-turn conversations with memory haiku-rag chat @@ -137,6 +141,7 @@ Full documentation at: https://ggozad.github.io/haiku.rag/ - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs - [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA, chat, and research agents +- [RLM Agent](https://ggozad.github.io/haiku.rag/rlm/) - Complex analytical tasks via code execution - [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector - [Server](https://ggozad.github.io/haiku.rag/server/) - File monitoring and MCP - [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration diff --git a/docs/agents.md b/docs/agents.md index 7c946caf..71ec3658 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -1,10 +1,11 @@ # Agents -Three agentic flows are provided by haiku.rag: +Four agentic flows are provided by haiku.rag: - **Simple QA Agent** — a focused question answering agent - **Chat Agent** — multi-turn conversational RAG with session memory - **Research Graph** — a multi-step research workflow with question decomposition +- **RLM Agent** — complex analytical tasks via sandboxed Python code execution (see [RLM Agent](rlm.md)) See [QA and Research Configuration](configuration/qa-research.md) for configuring model, iterations, concurrency, and other settings. diff --git a/docs/architecture.md b/docs/architecture.md index 62a40bff..3de8ba9f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -26,6 +26,7 @@ flowchart TB QA[QA Agent] Chat[Chat Agent] Research[Research Graph] + RLM[RLM Agent] end subgraph Apps["Applications"] @@ -97,7 +98,7 @@ flowchart LR ### Agent Layer -Three agent types for different use cases: +Four agent types for different use cases: ```mermaid flowchart TB @@ -122,6 +123,14 @@ flowchart TB Evaluate -->|Continue| Batch Evaluate -->|Done| Synthesize[Synthesize] end + + subgraph RLM["RLM Agent"] + Q4[Question] --> Code[Write Code] + Code --> Execute[Execute] + Execute --> Examine[Examine Results] + Examine -->|Iterate| Code + Examine -->|Done| A4[Answer] + end ``` **QA Agent** - Single-turn question answering: @@ -144,6 +153,13 @@ flowchart TB - Iterative refinement based on confidence - Synthesizes structured research report +**RLM Agent** - Complex analytical tasks via code execution: + +- Writes Python code to explore the knowledge base +- Executes in sandboxed environment +- Handles aggregation, computation, multi-document analysis +- Iterates until answer is found + ### Applications | Application | Interface | Use Case | diff --git a/docs/cli.md b/docs/cli.md index 83cbead1..968d2b7d 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -257,6 +257,33 @@ Flags: Research parameters like `max_iterations` and `max_concurrency` are configured in your [configuration file](configuration/index.md) under the `research` section. +## RLM (Recursive Language Model) + +Answer complex analytical questions via code execution: + +```bash +haiku-rag rlm "How many documents mention security?" +``` + +Filter to specific documents: + +```bash +haiku-rag rlm "What is the total revenue?" --filter "title LIKE '%Financial%'" +``` + +Pre-load specific documents for comparison: + +```bash +haiku-rag rlm "Compare the conclusions" --document "Report A" --document "Report B" +``` + +Flags: + +- `--filter` / `-f`: SQL WHERE clause to restrict document access +- `--document` / `-d`: Pre-load a document by title or ID (can repeat) + +See [RLM Agent](rlm.md) for details on capabilities and configuration. + ## Server Start services (requires at least one flag): diff --git a/docs/configuration/qa-research.md b/docs/configuration/qa-research.md index a5810e7f..4d874652 100644 --- a/docs/configuration/qa-research.md +++ b/docs/configuration/qa-research.md @@ -61,3 +61,24 @@ research: - **max_concurrency**: Concurrent search operations (default: 1) The research workflow uses an iterative feedback loop: the planner proposes one question at a time, sees the answer, then decides whether to continue or synthesize. This continues until the planner marks research as complete or `max_iterations` is reached. + +## RLM Configuration + +Configure the RLM (Recursive Language Model) agent: + +```yaml +rlm: + model: + provider: anthropic + name: claude-sonnet-4-20250514 + code_timeout: 60.0 # Max seconds for code execution + max_tool_calls: 20 # Max execute_code calls per question + max_output_chars: 50000 # Truncate output after this many chars +``` + +- **model**: LLM configuration (see [Providers](providers.md#model-settings)) +- **code_timeout**: Maximum seconds for each code execution (default: 60) +- **max_tool_calls**: Maximum number of code execution calls per question (default: 20) +- **max_output_chars**: Truncate code output after this many characters (default: 50000) + +See [RLM Agent](../rlm.md) for usage details. diff --git a/docs/index.md b/docs/index.md index d3d90564..1de71954 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,6 +8,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p - **Question answering** — QA agents with citations (page numbers, section headings) - **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM - **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize +- **RLM agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis) - **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory - **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion - **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI @@ -64,6 +65,7 @@ haiku-rag chat # Interactive conversation mode - [Python](python.md) - Python API reference - [Custom Pipelines](custom-pipelines.md) - Build custom processing workflows - [Agents](agents.md) - QA, chat, and research agents +- [RLM Agent](rlm.md) - Complex analytical tasks via code execution - [Applications](apps.md) - Chat TUI, web app, and inspector - [Server](server.md) - File monitoring and server mode - [MCP](mcp.md) - Model Context Protocol integration diff --git a/docs/mcp.md b/docs/mcp.md index a3755d9f..219100da 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -50,6 +50,12 @@ The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like - `question` (required): The research question - Returns a structured research report with findings, conclusions, and sources +- **`rlm_question`** - Answer complex analytical questions via code execution + - `question` (required): The question to answer + - `filter` (optional): SQL WHERE clause to restrict document access + - `document` (optional): Document title/ID to pre-load (can repeat) + - Best for aggregation, computation, and multi-document analysis + ## Starting MCP Server The MCP server supports Streamable HTTP and stdio transports: diff --git a/docs/python.md b/docs/python.md index 33f336dd..fdf204aa 100644 --- a/docs/python.md +++ b/docs/python.md @@ -396,3 +396,28 @@ The QA agent searches your documents for relevant information and uses the confi The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](configuration/index.md)). See also: [Agents](agents.md) for details on the QA agent and the multi‑agent research workflow. + +## RLM (Recursive Language Model) + +Answer complex analytical questions via code execution: + +```python +# Aggregation across documents +answer = await client.rlm("Which quarter had the highest revenue?") + +# Computation within a document set +answer = await client.rlm( + "What is the average deal size mentioned in these contracts?", + filter="uri LIKE '%contracts%'" +) + +# Multi-document comparison +answer = await client.rlm( + "What changed between these two versions of the policy?", + documents=["Policy v1.0", "Policy v2.0"] +) +``` + +The RLM agent writes and executes Python code in a sandboxed environment to solve problems that traditional RAG struggles with: aggregation, computation, and multi-document analysis. + +See [RLM Agent](rlm.md) for details on capabilities and configuration. diff --git a/docs/rlm.md b/docs/rlm.md new file mode 100644 index 00000000..5cdd1e8d --- /dev/null +++ b/docs/rlm.md @@ -0,0 +1,204 @@ +# RLM Agent (Recursive Language Model) + +The RLM agent enables complex analytical tasks by writing and executing Python code in a sandboxed environment. It solves problems that traditional RAG struggles with: + +- **Aggregation**: "How many documents mention security vulnerabilities?" +- **Computation**: "What's the average revenue across all quarterly reports?" +- **Multi-document analysis**: "Compare the key findings between Report A and Report B" +- **Structured data extraction**: "Extract all tables from the document and summarize them" + +## How It Works + +1. The agent receives a question +2. It writes Python code to explore the knowledge base +3. Code executes in a sandboxed environment with access to haiku.rag functions +4. The agent iterates: run code, examine results, refine approach +5. Final answer is synthesized from the gathered data + +## CLI Usage + +```bash +# Basic usage +haiku-rag rlm "How many documents are in the database?" + +# With document filter (restricts what the agent can access) +haiku-rag rlm "Summarize the key points" --filter "uri LIKE '%report%'" + +# Pre-load specific documents +haiku-rag rlm "Compare these two reports" --document "Q1 Report" --document "Q2 Report" +``` + +## Python Usage + +```python +from haiku.rag.client import HaikuRAG + +async with HaikuRAG(path_to_db) as client: + # Basic question + answer = await client.rlm("How many documents mention 'security'?") + print(answer) + + # With filter (agent can only see filtered documents) + answer = await client.rlm( + "What is the total revenue?", + filter="title LIKE '%Financial%'" + ) + + # Pre-load specific documents + answer = await client.rlm( + "Compare the conclusions", + documents=["Report A", "Report B"] + ) +``` + +## Available Functions + +Inside the sandbox, these functions are available (no imports needed): + +### search(query, limit=10) + +Search the knowledge base using hybrid search (vector + full-text). + +```python +results = search("climate change impacts", limit=20) +for r in results: + print(r['document_title'], r['score']) + print(r['content'][:200]) +``` + +Returns list of dicts with keys: `chunk_id`, `content`, `document_id`, `document_title`, `document_uri`, `score`, `page_numbers`, `headings` + +### list_documents(limit=10, offset=0) + +List available documents in the knowledge base. + +```python +docs = list_documents(limit=100) +for doc in docs: + print(doc['id'], doc['title']) +``` + +Returns list of dicts with keys: `id`, `title`, `uri`, `created_at` + +### get_document(id_or_title) + +Get the full text content of a document by ID, title, or URI. + +```python +content = get_document("Q1 Report") +if content: + print(len(content), "characters") +``` + +Returns the document content as a string, or `None` if not found. + +### get_docling_document(id_or_title) + +Get the structured DoclingDocument object for advanced analysis of tables, figures, and document structure. + +```python +doc = get_docling_document("Technical Manual") +if doc: + print(f"Tables: {len(doc.tables)}") + print(f"Pictures: {len(doc.pictures)}") + + # Extract table data + for table in doc.tables: + for cell in table.data.table_cells: + print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}") +``` + +### llm(prompt) + +Call an LLM directly for classification, summarization, or extraction tasks. + +```python +content = get_document("Q1 Report") +sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}") +print(sentiment) +``` + +Use this when you have content and need LLM reasoning without RAG search. + +## Pre-loaded Documents + +When documents are pre-loaded via the `documents` parameter, they're available as a `documents` variable: + +```python +# Available when documents are pre-loaded +for doc in documents: + print(doc['title'], len(doc['content'])) +``` + +Each document dict has keys: `id`, `title`, `uri`, `content` + +## Allowed Imports + +The following standard library modules can be imported: + +- `json` - JSON encoding/decoding +- `re` - Regular expressions +- `math` - Mathematical functions +- `statistics` - Statistical functions +- `collections` - Specialized containers +- `itertools` - Iterator utilities +- `functools` - Higher-order functions +- `datetime` - Date and time handling +- `typing` - Type hints + +```python +import re +import json +from collections import Counter + +# Extract and count patterns +results = search("error", limit=50) +error_types = [] +for r in results: + matches = re.findall(r'Error: (\w+)', r['content']) + error_types.extend(matches) + +print(Counter(error_types).most_common(10)) +``` + +## Security + +The sandbox enforces several security measures: + +- **Blocked builtins**: `eval`, `exec`, `compile`, `open`, `input`, `__import__`, `globals`, `locals`, `getattr`, `setattr`, `delattr` +- **Blocked imports**: `os`, `sys`, `subprocess`, `shutil`, `socket`, `requests`, `builtins` +- **Private attribute access blocked**: Cannot access `__dunder__` attributes (except common ones like `__init__`, `__str__`) +- **Execution timeout**: Code execution times out after configurable limit (default 60s) +- **Output truncation**: Large outputs are truncated to prevent memory issues + +## Context Filter + +The `filter` parameter restricts what documents the agent can access. Unlike tool parameters, the filter is applied automatically and cannot be bypassed by the LLM: + +```python +# Agent can only see documents with "confidential" in the URI +answer = await client.rlm( + "Summarize all findings", + filter="uri LIKE '%confidential%'" +) +``` + +This is useful for: + +- Scoping to specific document sets +- Enforcing access control +- Limiting context for focused analysis + +## Configuration + +RLM settings can be configured in `haiku.rag.yaml`: + +```yaml +rlm: + model: + provider: anthropic + name: claude-sonnet-4-20250514 + code_timeout: 60.0 # Max seconds for code execution + max_tool_calls: 20 # Max execute_code calls per question + max_output_chars: 50000 # Truncate output after this many chars +``` diff --git a/mkdocs.yml b/mkdocs.yml index 9a4d04ce..0efc7e8a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -72,6 +72,7 @@ nav: - Custom Pipelines: custom-pipelines.md - Tuning: tuning.md - Agents: agents.md + - RLM Agent: rlm.md - Applications: apps.md - Server: server.md - Remote processing: remote-processing.md