Merge pull request #271 from ggozad/feat/recursive-llm

Add RLM agent for analytical tasks via sandboxed Python execution
This commit is contained in:
Yiorgis Gozadinos 2026-02-06 12:41:05 +01:00 committed by GitHub
commit 3c3b14958d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
46 changed files with 11941 additions and 7 deletions

View file

@ -74,3 +74,22 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: false
test-docker-sandbox:
needs: [lint]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
with:
enable-cache: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: "pyproject.toml"
- name: Install dependencies
run: uv sync --all-extras
- name: Build Docker image
run: docker build -t haiku-rag-slim:test -f docker/Dockerfile.slim .
- name: Run Docker integration tests
run: uv run pytest tests/agents/rlm/test_sandbox.py -v

View file

@ -6,6 +6,18 @@
- **docling-serve Chunker OCR Options**: The docling-serve chunker now respects OCR settings from `conversion_options`
- Passes `do_ocr`, `force_ocr`, `ocr_engine`, and `ocr_lang` to the chunking API
- Allows disabling OCR via config when running docling-serve in read-only containers
- **RLM Agent (Recursive Language Model)**: New agent for complex analytical tasks via sandboxed Python code execution
- Solves problems traditional RAG can't handle: aggregation, computation, multi-document analysis
- Docker-based sandbox with full Python environment (no import restrictions)
- Container reuse within a single `rlm()` call for reduced latency
- Available functions: `search()`, `list_documents()`, `get_document()`, `get_docling_document()`, `llm()`
- Pre-loaded documents support via `documents` variable
- Context filter for scoping searches without LLM control
- New `client.rlm(question)` method on HaikuRAG client
- New `haiku-rag rlm` CLI command
- New `rlm_question` MCP tool
- New config options: `docker_image`, `docker_memory_limit`
- **CI**: Docker sandbox integration tests run in GitHub Actions
### Fixed

View file

@ -11,6 +11,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p
- **Question answering** — QA agents with citations (page numbers, section headings)
- **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM
- **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize
- **RLM agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis)
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI
@ -64,6 +65,9 @@ haiku-rag ask "How does the proposed method compare to the baseline on MMLU?" --
# Research mode — iterative planning and search
haiku-rag research "What are the limitations of the approach?"
# RLM mode — complex analytical tasks via code execution
haiku-rag rlm "How many documents mention transformers?"
# Interactive chat — multi-turn conversations with memory
haiku-rag chat
@ -137,6 +141,7 @@ Full documentation at: https://ggozad.github.io/haiku.rag/
- [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
- [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
- [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA, chat, and research agents
- [RLM Agent](https://ggozad.github.io/haiku.rag/rlm/) - Complex analytical tasks via code execution
- [Applications](https://ggozad.github.io/haiku.rag/apps/) - Chat TUI, web app, and inspector
- [Server](https://ggozad.github.io/haiku.rag/server/) - File monitoring and MCP
- [MCP](https://ggozad.github.io/haiku.rag/mcp/) - Model Context Protocol integration

View file

@ -1,10 +1,11 @@
# Agents
Three agentic flows are provided by haiku.rag:
Four agentic flows are provided by haiku.rag:
- **Simple QA Agent** — a focused question answering agent
- **Chat Agent** — multi-turn conversational RAG with session memory
- **Research Graph** — a multi-step research workflow with question decomposition
- **RLM Agent** — complex analytical tasks via sandboxed Python code execution (see [RLM Agent](rlm.md))
See [QA and Research Configuration](configuration/qa-research.md) for configuring model, iterations, concurrency, and other settings.

View file

@ -26,6 +26,7 @@ flowchart TB
QA[QA Agent]
Chat[Chat Agent]
Research[Research Graph]
RLM[RLM Agent]
end
subgraph Apps["Applications"]
@ -97,7 +98,7 @@ flowchart LR
### Agent Layer
Three agent types for different use cases:
Four agent types for different use cases:
```mermaid
flowchart TB
@ -122,6 +123,14 @@ flowchart TB
Evaluate -->|Continue| Batch
Evaluate -->|Done| Synthesize[Synthesize]
end
subgraph RLM["RLM Agent"]
Q4[Question] --> Code[Write Code]
Code --> Execute[Execute]
Execute --> Examine[Examine Results]
Examine -->|Iterate| Code
Examine -->|Done| A4[Answer]
end
```
**QA Agent** - Single-turn question answering:
@ -144,6 +153,13 @@ flowchart TB
- Iterative refinement based on confidence
- Synthesizes structured research report
**RLM Agent** - Complex analytical tasks via code execution:
- Writes Python code to explore the knowledge base
- Executes in sandboxed environment
- Handles aggregation, computation, multi-document analysis
- Iterates until answer is found
### Applications
| Application | Interface | Use Case |

View file

@ -257,6 +257,33 @@ Flags:
Research parameters like `max_iterations` and `max_concurrency` are configured in your [configuration file](configuration/index.md) under the `research` section.
## RLM (Recursive Language Model)
Answer complex analytical questions via code execution:
```bash
haiku-rag rlm "How many documents mention security?"
```
Filter to specific documents:
```bash
haiku-rag rlm "What is the total revenue?" --filter "title LIKE '%Financial%'"
```
Pre-load specific documents for comparison:
```bash
haiku-rag rlm "Compare the conclusions" --document "Report A" --document "Report B"
```
Flags:
- `--filter` / `-f`: SQL WHERE clause to restrict document access
- `--document` / `-d`: Pre-load a document by title or ID (can repeat)
See [RLM Agent](rlm.md) for details on capabilities and configuration.
## Server
Start services (requires at least one flag):

View file

@ -61,3 +61,22 @@ research:
- **max_concurrency**: Concurrent search operations (default: 1)
The research workflow uses an iterative feedback loop: the planner proposes one question at a time, sees the answer, then decides whether to continue or synthesize. This continues until the planner marks research as complete or `max_iterations` is reached.
## RLM Configuration
Configure the RLM (Recursive Language Model) agent:
```yaml
rlm:
model:
provider: anthropic
name: claude-sonnet-4-20250514
code_timeout: 60.0 # Max seconds for code execution
max_output_chars: 50000 # Truncate output after this many chars
```
- **model**: LLM configuration (see [Providers](providers.md#model-settings))
- **code_timeout**: Maximum seconds for each code execution (default: 60)
- **max_output_chars**: Truncate code output after this many characters (default: 50000)
See [RLM Agent](../rlm.md) for usage details.

View file

@ -8,6 +8,7 @@ Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.p
- **Question answering** — QA agents with citations (page numbers, section headings)
- **Reranking** — MxBAI, Cohere, Zero Entropy, or vLLM
- **Research agents** — Multi-agent workflows via pydantic-graph: plan, search, evaluate, synthesize
- **RLM agent** — Complex analytical tasks via sandboxed Python code execution (aggregation, computation, multi-document analysis)
- **Conversational RAG** — Chat TUI and web application for multi-turn conversations with session memory
- **Document structure** — Stores full [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/), enabling structure-aware context expansion
- **Multiple providers** — Embeddings: Ollama, OpenAI, VoyageAI, LM Studio, vLLM. QA/Research: any model supported by Pydantic AI
@ -64,6 +65,7 @@ haiku-rag chat # Interactive conversation mode
- [Python](python.md) - Python API reference
- [Custom Pipelines](custom-pipelines.md) - Build custom processing workflows
- [Agents](agents.md) - QA, chat, and research agents
- [RLM Agent](rlm.md) - Complex analytical tasks via code execution
- [Applications](apps.md) - Chat TUI, web app, and inspector
- [Server](server.md) - File monitoring and server mode
- [MCP](mcp.md) - Model Context Protocol integration

View file

@ -50,6 +50,12 @@ The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients like
- `question` (required): The research question
- Returns a structured research report with findings, conclusions, and sources
- **`rlm_question`** - Answer complex analytical questions via code execution
- `question` (required): The question to answer
- `filter` (optional): SQL WHERE clause to restrict document access
- `document` (optional): Document title/ID to pre-load (can repeat)
- Best for aggregation, computation, and multi-document analysis
## Starting MCP Server
The MCP server supports Streamable HTTP and stdio transports:

View file

@ -396,3 +396,30 @@ The QA agent searches your documents for relevant information and uses the confi
The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](configuration/index.md)).
See also: [Agents](agents.md) for details on the QA agent and the multiagent research workflow.
## RLM (Recursive Language Model)
Answer complex analytical questions via code execution:
```python
# Aggregation across documents
result = await client.rlm("Which quarter had the highest revenue?")
print(result.answer) # The answer
print(result.program) # The final consolidated program
# Computation within a document set
result = await client.rlm(
"What is the average deal size mentioned in these contracts?",
filter="uri LIKE '%contracts%'"
)
# Multi-document comparison
result = await client.rlm(
"What changed between these two versions of the policy?",
documents=["Policy v1.0", "Policy v2.0"]
)
```
The RLM agent writes and executes Python code in a sandboxed environment to solve problems that traditional RAG struggles with: aggregation, computation, and multi-document analysis.
See [RLM Agent](rlm.md) for details on capabilities and configuration.

218
docs/rlm.md Normal file
View file

@ -0,0 +1,218 @@
# RLM Agent (Recursive Language Model)
The RLM agent enables complex analytical tasks by writing and executing Python code in a sandboxed environment. It solves problems that traditional RAG struggles with:
- **Aggregation**: "How many documents mention security vulnerabilities?"
- **Computation**: "What's the average revenue across all quarterly reports?"
- **Multi-document analysis**: "Compare the key findings between Report A and Report B"
- **Structured data extraction**: "Extract all tables from the document and summarize them"
## How It Works
1. The agent receives a question
2. It writes Python code to explore the knowledge base
3. Code executes in a sandboxed environment with access to haiku.rag functions
4. The agent iterates: run code, examine results, refine approach
5. Final answer is synthesized from the gathered data
## CLI Usage
```bash
# Basic usage
haiku-rag rlm "How many documents are in the database?"
# With document filter (restricts what the agent can access)
haiku-rag rlm "Summarize the key points" --filter "uri LIKE '%report%'"
# Pre-load specific documents
haiku-rag rlm "Compare these two reports" --document "Q1 Report" --document "Q2 Report"
```
## Python Usage
```python
from haiku.rag.client import HaikuRAG
async with HaikuRAG(path_to_db) as client:
# Basic question
result = await client.rlm("How many documents mention 'security'?")
print(result.answer) # The answer
print(result.program) # The final consolidated program
# With filter (agent can only see filtered documents)
result = await client.rlm(
"What is the total revenue?",
filter="title LIKE '%Financial%'"
)
# Pre-load specific documents
result = await client.rlm(
"Compare the conclusions",
documents=["Report A", "Report B"]
)
```
## Available Functions
Inside the sandbox, these functions are available (no imports needed):
### search(query, limit=10)
Search the knowledge base using hybrid search (vector + full-text).
```python
results = search("climate change impacts", limit=20)
for r in results:
print(r['document_title'], r['score'])
print(r['content'][:200])
```
Returns list of dicts with keys: `chunk_id`, `content`, `document_id`, `document_title`, `document_uri`, `score`, `page_numbers`, `headings`
### list_documents(limit=10, offset=0)
List available documents in the knowledge base.
```python
docs = list_documents(limit=100)
for doc in docs:
print(doc['id'], doc['title'])
```
Returns list of dicts with keys: `id`, `title`, `uri`, `created_at`
### get_document(id_or_title)
Get the full text content of a document by ID, title, or URI.
```python
content = get_document("Q1 Report")
if content:
print(len(content), "characters")
```
Returns the document content as a string, or `None` if not found.
### get_docling_document(id_or_title)
Get the structured DoclingDocument object for advanced analysis of tables, figures, and document structure.
```python
doc = get_docling_document("Technical Manual")
if doc:
print(f"Tables: {len(doc.tables)}")
print(f"Pictures: {len(doc.pictures)}")
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
### llm(prompt)
Call an LLM directly for classification, summarization, or extraction tasks.
```python
content = get_document("Q1 Report")
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
Use this when you have content and need LLM reasoning without RAG search.
## Pre-loaded Documents
When documents are pre-loaded via the `documents` parameter, they're available as a `documents` variable:
```python
# Available when documents are pre-loaded
for doc in documents:
print(doc['title'], len(doc['content']))
```
Each document dict has keys: `id`, `title`, `uri`, `content`
## Imports
The sandbox runs in a Docker container with full Python available. Any module installed in the container image can be imported:
```python
import re
import json
from collections import Counter
# Extract and count patterns
results = search("error", limit=50)
error_types = []
for r in results:
matches = re.findall(r'Error: (\w+)', r['content'])
error_types.extend(matches)
print(Counter(error_types).most_common(10))
```
The default image (`ghcr.io/ggozad/haiku.rag-slim`) includes the Python standard library. Custom images can add additional packages like `pandas` or `numpy`.
## Docker Sandbox
Code executes in an isolated Docker container with:
- **Read-only database**: The LanceDB database is mounted read-only
- **Memory limits**: Configurable memory limit (default 512MB)
- **Execution timeout**: Code times out after configurable limit (default 60s)
- **Output truncation**: Large outputs are truncated to prevent memory issues
- **Container reuse**: Within a single `rlm()` call, the container stays warm for multiple code executions
## Context Filter
The `filter` parameter restricts what documents the agent can access. Unlike tool parameters, the filter is applied automatically and cannot be bypassed by the LLM:
```python
# Agent can only see documents with "confidential" in the URI
result = await client.rlm(
"Summarize all findings",
filter="uri LIKE '%confidential%'"
)
```
This is useful for:
- Scoping to specific document sets
- Enforcing access control
- Limiting context for focused analysis
## Configuration
RLM settings can be configured in `haiku.rag.yaml`:
```yaml
rlm:
model:
provider: anthropic
name: claude-sonnet-4-20250514
code_timeout: 60.0 # Max seconds for code execution
max_output_chars: 50000 # Truncate output after this many chars
docker_image: "ghcr.io/ggozad/haiku.rag-slim:latest" # Container image
docker_memory_limit: "512m" # Container memory limit
```
### Custom Docker Image
To add additional Python packages, create a custom Dockerfile:
```dockerfile
FROM ghcr.io/ggozad/haiku.rag-slim:latest
RUN pip install pandas numpy
```
Build and configure:
```bash
docker build -t my-rlm-image .
```
```yaml
rlm:
docker_image: "my-rlm-image"
```

View file

@ -13,7 +13,7 @@ How to decide which tool to use:
- "list_documents" - Use when the user wants to browse or see what documents are available (e.g., "what documents are available?", "show me the documents", "list available docs").
- "summarize_document" - Use when the user wants an overview or summary of a specific document (e.g., "summarize document X", "what does Y cover?", "give me an overview of Z").
- "get_document" - Use when the user wants the FULL content of a specific document (e.g., "get the paper about Y", "fetch 2412.00566", "show me the full document").
- "ask" - Use for questions about topics in the knowledge base. It automatically finds relevant prior answers from conversation history and searches across documents to return answers with citations.
- "ask" - Use for CONTENT questions: "What does X say about Y?", "What are the main findings?", "Explain concept Z from the documents". This tool retrieves and synthesizes text from documents.
- "search" - Use when the user explicitly asks to search/find/explore documents. Call it ONCE. After calling search, copy the ENTIRE tool response to your output INCLUDING the content snippets. Do NOT shorten, summarize, or omit any part of the results.
IMPORTANT - When user mentions a document in search/ask:

View file

@ -0,0 +1,16 @@
from haiku.rag.agents.rlm.agent import create_rlm_agent
from haiku.rag.agents.rlm.dependencies import RLMContext, RLMDeps
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox, SandboxResult
from haiku.rag.agents.rlm.models import CodeExecution, RLMResult
from haiku.rag.agents.rlm.prompts import RLM_SYSTEM_PROMPT
__all__ = [
"CodeExecution",
"DockerSandbox",
"RLMContext",
"RLMDeps",
"RLMResult",
"RLM_SYSTEM_PROMPT",
"SandboxResult",
"create_rlm_agent",
]

View file

@ -0,0 +1,60 @@
from pydantic_ai import Agent, RunContext
from haiku.rag.agents.rlm.dependencies import RLMDeps
from haiku.rag.agents.rlm.models import CodeExecution, RLMResult
from haiku.rag.agents.rlm.prompts import RLM_SYSTEM_PROMPT
from haiku.rag.config.models import AppConfig
from haiku.rag.utils import get_model
def create_rlm_agent(config: AppConfig) -> Agent[RLMDeps, RLMResult]:
"""Create an RLM agent with code execution capability.
The RLM (Recursive Language Model) agent can write and execute Python code
in a sandboxed environment to solve problems that require computation,
aggregation, or complex traversal across documents.
Args:
config: Application configuration.
Returns:
A pydantic-ai Agent configured for RLM execution.
"""
model = get_model(config.rlm.model, config)
agent: Agent[RLMDeps, RLMResult] = Agent( # type: ignore[invalid-assignment]
model,
deps_type=RLMDeps,
output_type=RLMResult,
instructions=RLM_SYSTEM_PROMPT,
retries=3,
)
@agent.tool
async def execute_code(ctx: RunContext[RLMDeps], code: str) -> CodeExecution:
"""Execute Python code in a Docker-sandboxed environment.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
Use print() to output results.
Args:
code: Python code to execute.
Returns:
Structured result with success status, stdout, and stderr.
"""
result = await ctx.deps.sandbox.execute(code)
execution = CodeExecution(
code=code,
stdout=result.stdout,
stderr=result.stderr,
success=result.success,
)
return execution
return agent

View file

@ -0,0 +1,23 @@
from dataclasses import dataclass, field
from typing import TYPE_CHECKING
from haiku.rag.store.models import Document
if TYPE_CHECKING:
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox
@dataclass
class RLMContext:
"""Mutable context accumulating data during RLM execution."""
documents: list[Document] | None = None
filter: str | None = None
@dataclass
class RLMDeps:
"""Dependencies for RLM agent."""
sandbox: "DockerSandbox"
context: RLMContext = field(default_factory=RLMContext)

View file

@ -0,0 +1,216 @@
"""Docker-based sandboxed execution."""
import asyncio
import json
import os
import subprocess
import sys
from dataclasses import dataclass
from typing import TYPE_CHECKING
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.config.models import RLMConfig
if TYPE_CHECKING:
from haiku.rag.client import HaikuRAG
@dataclass
class SandboxResult:
"""Result of executing code in the sandbox."""
stdout: str
stderr: str
success: bool
class DockerSandbox: # pragma: no cover
"""Execute code in a persistent Docker container.
Use as an async context manager to manage container lifecycle:
async with DockerSandbox(client, config, context) as sandbox:
result = await sandbox.execute("print('hello')")
result = await sandbox.execute("print('world')")
"""
DEFAULT_IMAGE = "ghcr.io/ggozad/haiku.rag-slim:latest"
haiku_client: "HaikuRAG"
config: RLMConfig
context: RLMContext
image: str
_process: subprocess.Popen[bytes] | None
def __init__(
self,
client: "HaikuRAG",
config: RLMConfig,
context: RLMContext,
image: str | None = None,
):
self.haiku_client = client
self.config = config
self.context = context
self.image = image or self.DEFAULT_IMAGE
self._process = None
def _build_docker_cmd(self) -> list[str]:
"""Build the docker run command."""
db_path = str(self.haiku_client.store.db_path)
env_list = ["-e", "HAIKU_DB_PATH=/data/db.lancedb"]
if self.context.filter:
env_list.extend(["-e", f"HAIKU_FILTER={self.context.filter}"])
ollama_host = os.environ.get("OLLAMA_HOST", "")
ollama_base_url = os.environ.get("OLLAMA_BASE_URL", "")
if sys.platform == "darwin":
if not ollama_host or "localhost" in ollama_host:
ollama_host = "http://host.docker.internal:11434"
if not ollama_base_url or "localhost" in ollama_base_url:
ollama_base_url = "http://host.docker.internal:11434"
if ollama_host:
env_list.extend(["-e", f"OLLAMA_HOST={ollama_host}"])
if ollama_base_url:
env_list.extend(["-e", f"OLLAMA_BASE_URL={ollama_base_url}"])
for key in [
"ANTHROPIC_API_KEY",
"OPENAI_API_KEY",
"VOYAGE_API_KEY",
"COHERE_API_KEY",
]:
if value := os.environ.get(key):
env_list.extend(["-e", f"{key}={value}"])
return [
"docker",
"run",
"--rm",
"-i",
"-v",
f"{db_path}:/data/db.lancedb:ro",
f"--memory={self.config.docker_memory_limit}",
"--network=host",
*env_list,
self.image,
"python",
"-m",
"haiku.rag.agents.rlm.runner",
]
async def __aenter__(self) -> "DockerSandbox":
"""Start the container."""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self._start_container)
return self
async def __aexit__(
self, exc_type: object, exc_val: object, exc_tb: object
) -> None:
"""Stop the container."""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self._stop_container)
def _start_container(self) -> None:
"""Start the persistent container process."""
if self._process is not None:
return
cmd = self._build_docker_cmd()
self._process = subprocess.Popen(
cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
def _stop_container(self) -> None:
"""Stop the container process."""
if self._process is None:
return
try:
if self._process.stdin:
try:
self._process.stdin.close()
except BrokenPipeError:
pass
self._process.terminate()
self._process.wait(timeout=5)
except subprocess.TimeoutExpired:
self._process.kill()
self._process.wait()
finally:
self._process = None
async def execute(self, code: str) -> SandboxResult:
"""Execute code in the container."""
if self._process is None:
return SandboxResult(
stdout="",
stderr="Container not started. Use 'async with' context manager.",
success=False,
)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, self._execute_sync, code)
def _execute_sync(self, code: str) -> SandboxResult:
"""Send code to container and read result."""
assert self._process is not None and self._process.stdin is not None
try:
message = json.dumps({"code": code})
length_line = f"{len(message)}\n".encode()
self._process.stdin.write(length_line)
self._process.stdin.write(message.encode())
self._process.stdin.flush()
if self._process.stdout is None:
return SandboxResult(
stdout="", stderr="No stdout from container.", success=False
)
length_line = self._process.stdout.readline()
if not length_line:
stderr = ""
if self._process.stderr:
stderr = self._process.stderr.read().decode()
return SandboxResult(
stdout="",
stderr=stderr or "Container closed unexpectedly.",
success=False,
)
length = int(length_line.strip())
response = self._process.stdout.read(length).decode()
result_data = json.loads(response)
return SandboxResult(
stdout=result_data.get("stdout", ""),
stderr=result_data.get("stderr", ""),
success=result_data.get("success", False),
)
except subprocess.TimeoutExpired:
return SandboxResult(
stdout="",
stderr=f"Execution timed out after {self.config.code_timeout} seconds",
success=False,
)
except json.JSONDecodeError as e:
return SandboxResult(
stdout="",
stderr=f"Invalid response from container: {e}",
success=False,
)
except Exception as e:
return SandboxResult(
stdout="",
stderr=f"Execution error: {e}",
success=False,
)

View file

@ -0,0 +1,17 @@
from pydantic import BaseModel, Field
class CodeExecution(BaseModel):
"""Result of executing a code block in the RLM sandbox."""
code: str = Field(description="The Python code that was executed")
stdout: str = Field(description="Standard output captured during execution")
stderr: str = Field(description="Standard error captured during execution")
success: bool = Field(description="Whether execution completed without error")
class RLMResult(BaseModel):
"""Result from RLM agent execution."""
answer: str = Field(description="The answer to the user's question")
program: str = Field(description="The final consolidated program")

View file

@ -0,0 +1,153 @@
RLM_SYSTEM_PROMPT = """You are a Recursive Language Model (RLM) agent that solves complex research questions by writing and executing Python code.
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") CORRECT
- from haiku.rag import search WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
### search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
## Pre-loaded Documents Variable
If documents were pre-loaded for this session, a `documents` variable is available:
```python
# documents is a list of dicts with keys: id, title, uri, content
for doc in documents:
print(doc['title'], len(doc['content']))
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\\$([\\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
## Workflow
1. **ALWAYS start by using execute_code** to explore the knowledge base
2. Run multiple code blocks as needed to gather information
3. After collecting data, provide your final answer
## Output Format
CRITICAL: Your final response MUST be valid JSON matching this exact schema:
```json
{"answer": "Your complete answer here as a string", "program": "Your final consolidated program here as a string"}
```
- `answer`: A clear answer to the user's question with key findings and references to specific documents/chunks.
- `program`: A single, self-contained Python program that produces the answer. Consolidate your exploratory code executions into one clean script.
Do NOT return arbitrary JSON structures. Always use the exact format: {"answer": "...", "program": "..."}
CRITICAL: You MUST call execute_code at least once before providing your answer. Never give up without trying to execute code first."""

View file

@ -0,0 +1,190 @@
"""Entry point for sandboxed code execution in Docker container."""
import asyncio
import json
import sys
import traceback
from io import StringIO
from typing import Any
def build_namespace(
client: Any, config: Any, context: Any, loop: asyncio.AbstractEventLoop
) -> dict[str, Any]:
"""Build execution namespace with haiku.rag functions injected."""
def run_async(coro: Any) -> Any:
"""Run async coroutine from sync context using thread-safe scheduling."""
future = asyncio.run_coroutine_threadsafe(coro, loop)
return future.result(timeout=config.rlm.code_timeout)
def search(query: str, limit: int = 10) -> list[dict]:
async def _search() -> Any:
return await client.search(query, limit=limit, filter=context.filter)
results = run_async(_search())
return [
{
"chunk_id": r.chunk_id,
"content": r.content,
"document_id": r.document_id,
"document_title": r.document_title,
"document_uri": r.document_uri,
"score": r.score,
"page_numbers": r.page_numbers,
"headings": r.headings,
}
for r in results
]
def list_documents(limit: int = 10, offset: int = 0) -> list[dict]:
async def _list() -> Any:
return await client.list_documents(
limit=limit, offset=offset, filter=context.filter
)
docs = run_async(_list())
return [
{
"id": d.id,
"title": d.title,
"uri": d.uri,
"created_at": str(d.created_at),
}
for d in docs
]
def get_document(id_or_title: str) -> str | None:
async def _get() -> str | None:
doc = await client.resolve_document(id_or_title)
return doc.content if doc else None
return run_async(_get())
def get_docling_document(id_or_title: str) -> Any:
async def _get() -> Any:
doc = await client.resolve_document(id_or_title)
return doc.get_docling_document() if doc else None
return run_async(_get())
def llm(prompt: str) -> str:
async def _llm() -> str:
from pydantic_ai import Agent
from haiku.rag.utils import get_model
model = get_model(config.rlm.model, config)
agent: Agent[None, str] = Agent(model, output_type=str)
result = await agent.run(prompt)
return result.output
return run_async(_llm())
namespace: dict[str, Any] = {
"search": search,
"list_documents": list_documents,
"get_document": get_document,
"get_docling_document": get_docling_document,
"llm": llm,
}
if context.documents:
namespace["documents"] = [
{"id": d.id, "title": d.title, "uri": d.uri, "content": d.content}
for d in context.documents
]
return namespace
def execute_code(
code: str, namespace: dict[str, Any], max_output_chars: int
) -> dict[str, Any]:
"""Execute code and capture output."""
stdout_capture = StringIO()
original_stdout = sys.stdout
try:
sys.stdout = stdout_capture
exec(code, namespace)
stdout = stdout_capture.getvalue()
if len(stdout) > max_output_chars:
stdout = stdout[:max_output_chars] + "\n... (output truncated)"
return {
"success": True,
"stdout": stdout,
"stderr": "",
}
except Exception:
return {
"success": False,
"stdout": stdout_capture.getvalue(),
"stderr": traceback.format_exc(),
}
finally:
sys.stdout = original_stdout
def send_response(result: dict[str, Any]) -> None:
"""Send length-prefixed JSON response."""
response = json.dumps(result)
sys.stdout.write(f"{len(response)}\n")
sys.stdout.write(response)
sys.stdout.flush()
async def main() -> None:
"""Main entry point for container execution.
Runs a loop reading length-prefixed JSON messages and executing code.
"""
import concurrent.futures
import os
from pathlib import Path
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.client import HaikuRAG
from haiku.rag.config import get_config
config = get_config()
db_path = Path(os.environ.get("HAIKU_DB_PATH", "/data/db.lancedb"))
filter_expr = os.environ.get("HAIKU_FILTER")
context = RLMContext(filter=filter_expr)
max_output_chars = config.rlm.max_output_chars
loop = asyncio.get_running_loop()
async with HaikuRAG(db_path, config=config, read_only=True) as client:
namespace = build_namespace(client, config, context, loop)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
while True:
# Read length-prefixed message
length_line = sys.stdin.readline()
if not length_line:
break
try:
length = int(length_line.strip())
message = sys.stdin.read(length)
request = json.loads(message)
code = request.get("code", "")
result = await loop.run_in_executor(
executor, execute_code, code, namespace, max_output_chars
)
send_response(result)
except (ValueError, json.JSONDecodeError) as e:
send_response(
{
"success": False,
"stdout": "",
"stderr": f"Invalid request: {e}",
}
)
if __name__ == "__main__":
asyncio.run(main())

View file

@ -16,6 +16,7 @@ from rich.progress import (
TextColumn,
TransferSpeedColumn,
)
from rich.syntax import Syntax
from haiku.rag.agents.research.dependencies import ResearchContext
from haiku.rag.agents.research.graph import build_research_graph
@ -432,6 +433,40 @@ class HaikuRAGApp:
for renderable in format_citations_rich(citations):
self.console.print(renderable)
async def rlm(
self,
question: str,
document: str | None = None,
filter: str | None = None,
):
"""Answer a question using the RLM agent with code execution.
Args:
question: The question to answer
document: Optional document ID or title to pre-load
filter: SQL WHERE clause to filter documents
"""
async with HaikuRAG(
db_path=self.db_path,
config=self.config,
read_only=self.read_only,
before=self.before,
) as self.client:
documents = [document] if document else None
self.console.print(f"[bold blue]Question:[/bold blue] {question}")
self.console.print()
self.console.print("[dim]Running RLM agent with code execution...[/dim]")
self.console.print()
result = await self.client.rlm(question, documents=documents, filter=filter)
self.console.print("[bold yellow]Program:[/bold yellow]")
self.console.print(Syntax(result.program, "python"))
self.console.print()
self.console.print("[bold green]Answer:[/bold green]")
self.console.print(Markdown(result.answer))
async def research(
self,
question: str,

View file

@ -364,6 +364,39 @@ def ask(
)
@_cli.command("rlm", help="Answer questions using code execution (RLM agent)")
def rlm(
question: str = typer.Argument(
help="The question to answer",
),
db: Path | None = typer.Option(
None,
"--db",
help="Path to the LanceDB database file",
),
document: str | None = typer.Option(
None,
"--document",
"-d",
help="Document ID or title to pre-load for analysis",
),
filter: str | None = typer.Option(
None,
"--filter",
"-f",
help="SQL WHERE clause to filter documents (e.g., \"uri LIKE '%arxiv%'\")",
),
):
app = create_app(db)
asyncio.run(
app.rlm(
question=question,
document=document,
filter=filter,
)
)
@_cli.command("research", help="Run multi-agent research and output a concise report")
def research(
question: str = typer.Argument(..., help="The research question to investigate"),

View file

@ -22,13 +22,17 @@ from haiku.rag.store.engine import Store
from haiku.rag.store.models.chunk import Chunk, SearchResult
from haiku.rag.store.models.document import Document
from haiku.rag.store.repositories.chunk import ChunkRepository
from haiku.rag.store.repositories.document import DocumentRepository
from haiku.rag.store.repositories.document import (
DocumentRepository,
_escape_sql_string,
)
from haiku.rag.store.repositories.settings import SettingsRepository
if TYPE_CHECKING:
from docling_core.types.doc.document import DoclingDocument
from haiku.rag.agents.research.models import Citation
from haiku.rag.agents.rlm.models import RLMResult
logger = logging.getLogger(__name__)
@ -736,6 +740,30 @@ class HaikuRAG:
"""
return await self.document_repository.get_by_uri(uri)
async def resolve_document(self, id_or_title: str) -> Document | None:
"""Resolve a document by ID, title, or URI (in that order).
Args:
id_or_title: Document ID, title, or URI to look up.
Returns:
The Document instance if found, None otherwise.
"""
doc = await self.get_document_by_id(id_or_title)
if doc:
return doc
safe_input = _escape_sql_string(id_or_title)
docs = await self.list_documents(filter=f"title = '{safe_input}'")
if docs and docs[0].id:
return await self.get_document_by_id(docs[0].id)
docs = await self.list_documents(filter=f"uri = '{safe_input}'")
if docs and docs[0].id:
return await self.get_document_by_id(docs[0].id)
return None
async def update_document(
self,
document_id: str,
@ -1293,6 +1321,59 @@ class HaikuRAG:
qa_agent = get_qa_agent(self, config=self._config, system_prompt=system_prompt)
return await qa_agent.answer(question, filter=filter)
async def rlm(
self,
question: str,
documents: list[str] | None = None,
filter: str | None = None,
) -> "RLMResult":
"""Answer a question using the RLM agent with code execution.
The RLM (Recursive Language Model) agent can write and execute Python
code in a sandboxed environment to solve problems that require
computation, aggregation, or complex traversal across documents.
Args:
question: The question to answer.
documents: Optional list of document IDs or titles to pre-load.
filter: SQL WHERE clause to filter documents during searches.
Returns:
RLMResult with the answer and the final consolidated program.
"""
from haiku.rag.agents.rlm import (
DockerSandbox,
RLMContext,
RLMDeps,
create_rlm_agent,
)
context = RLMContext(filter=filter)
if documents:
loaded_docs = []
for doc_ref in documents:
doc = await self.resolve_document(doc_ref)
if doc:
loaded_docs.append(doc)
context.documents = loaded_docs if loaded_docs else None
async with DockerSandbox(
client=self,
config=self._config.rlm,
context=context,
image=self._config.rlm.docker_image,
) as sandbox:
deps = RLMDeps(
sandbox=sandbox,
context=context,
)
agent = create_rlm_agent(self._config)
result = await agent.run(question, deps=deps)
return result.output
async def visualize_chunk(self, chunk: Chunk) -> list:
"""Render page images with bounding box highlights for a chunk.

View file

@ -94,6 +94,20 @@ class ResearchConfig(BaseModel):
max_concurrency: int = 1
class RLMConfig(BaseModel):
model: ModelConfig = Field(
default_factory=lambda: ModelConfig(
provider="ollama",
name="gpt-oss",
enable_thinking=False,
)
)
code_timeout: float = 60.0
max_output_chars: int = 50_000
docker_image: str = "ghcr.io/ggozad/haiku.rag-slim:latest"
docker_memory_limit: str = "512m"
class PictureDescriptionConfig(BaseModel):
"""Configuration for VLM-based picture description."""
@ -194,6 +208,7 @@ class AppConfig(BaseModel):
reranking: RerankingConfig = Field(default_factory=RerankingConfig)
qa: QAConfig = Field(default_factory=QAConfig)
research: ResearchConfig = Field(default_factory=ResearchConfig)
rlm: RLMConfig = Field(default_factory=RLMConfig)
processing: ProcessingConfig = Field(default_factory=ProcessingConfig)
search: SearchConfig = Field(default_factory=SearchConfig)
providers: ProvidersConfig = Field(default_factory=ProvidersConfig)

View file

@ -245,4 +245,32 @@ def create_mcp_server(
except Exception:
return None
@mcp.tool()
async def rlm_question(
question: str,
document: str | None = None,
filter: str | None = None,
) -> str:
"""Answer complex questions using code execution (RLM agent).
Use this for questions requiring computation, aggregation, or
complex traversal across documents. The agent can write Python
code to search, analyze, and compute answers.
Args:
question: The question to answer.
document: Optional document ID or title to pre-load for analysis.
filter: Optional SQL WHERE clause to filter documents.
Returns:
The answer as a string.
"""
try:
async with HaikuRAG(db_path, config=config, read_only=read_only) as rag:
documents = [document] if document else None
result = await rag.rlm(question, documents=documents, filter=filter)
return result.answer
except Exception as e:
return f"Error running RLM agent: {e!s}"
return mcp

View file

@ -77,9 +77,10 @@ class DocumentRepository:
async def get_by_id(self, entity_id: str) -> Document | None:
"""Get a document by its ID."""
safe_id = _escape_sql_string(entity_id)
results = list(
self.store.documents_table.search()
.where(f"id = '{entity_id}'")
.where(f"id = '{safe_id}'")
.limit(1)
.to_pydantic(DocumentRecord)
)
@ -104,8 +105,9 @@ class DocumentRepository:
entity.updated_at = datetime.fromisoformat(now)
# Update the record
safe_id = _escape_sql_string(entity.id)
self.store.documents_table.update(
where=f"id = '{entity.id}'",
where=f"id = '{safe_id}'",
values={
"content": entity.content,
"uri": entity.uri,
@ -136,7 +138,8 @@ class DocumentRepository:
await self.chunk_repository.delete_by_document_id(entity_id)
# Delete the document
self.store.documents_table.delete(f"id = '{entity_id}'")
safe_id = _escape_sql_string(entity_id)
self.store.documents_table.delete(f"id = '{safe_id}'")
return True
async def list_all(

View file

@ -72,6 +72,7 @@ nav:
- Custom Pipelines: custom-pipelines.md
- Tuning: tuning.md
- Agents: agents.md
- RLM Agent: rlm.md
- Applications: apps.md
- Server: server.md
- Remote processing: remote-processing.md

View file

View file

@ -0,0 +1,54 @@
import os
import subprocess
from pathlib import Path
import pytest
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox
from haiku.rag.client import HaikuRAG
from haiku.rag.config.models import RLMConfig
TEST_DOCKER_IMAGE = os.environ.get("HAIKU_TEST_DOCKER_IMAGE", "haiku-rag-slim:test")
@pytest.fixture(scope="session")
def test_docker_image():
"""Build and return the Docker image for testing."""
if os.environ.get("CI"):
return TEST_DOCKER_IMAGE
project_root = Path(__file__).parent.parent.parent.parent
dockerfile = project_root / "docker" / "Dockerfile.slim"
if not dockerfile.exists():
pytest.skip(f"Dockerfile.slim not found at {dockerfile}")
result = subprocess.run(
["docker", "build", "-t", TEST_DOCKER_IMAGE, "-f", str(dockerfile), "."],
cwd=project_root,
capture_output=True,
text=True,
)
if result.returncode != 0:
pytest.fail(f"Failed to build Docker image:\n{result.stderr}")
return TEST_DOCKER_IMAGE
@pytest.fixture
async def empty_client(temp_db_path):
"""Create an empty HaikuRAG client without documents."""
async with HaikuRAG(temp_db_path, create=True) as client:
yield client
@pytest.fixture
async def docker_sandbox(empty_client, test_docker_image):
"""Create a Docker sandbox for testing."""
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=empty_client, config=config, context=context, image=test_docker_image
) as sandbox:
yield sandbox

View file

@ -0,0 +1,331 @@
from pathlib import Path
import pytest
from pydantic_ai import Agent
from haiku.rag.agents.rlm.agent import create_rlm_agent
from haiku.rag.agents.rlm.dependencies import RLMDeps
from haiku.rag.agents.rlm.models import CodeExecution, RLMResult
from haiku.rag.config import AppConfig, Config
@pytest.fixture(scope="module")
def vcr_cassette_dir():
return str(Path(__file__).parent.parent.parent / "cassettes" / "test_rlm")
class TestCreateRLMAgent:
def test_creates_agent_with_correct_types(self):
agent = create_rlm_agent(Config)
assert isinstance(agent, Agent)
assert agent.deps_type is RLMDeps
assert agent.output_type is RLMResult
def test_agent_has_execute_code_tool(self):
agent = create_rlm_agent(Config)
tool_names = list(agent._function_toolset.tools.keys())
assert "execute_code" in tool_names
class TestCodeExecutionModel:
def test_code_execution_has_correct_fields(self):
"""Test that CodeExecution has all expected fields."""
execution = CodeExecution(
code="print('hello')",
stdout="hello\n",
stderr="",
success=True,
)
assert execution.code == "print('hello')"
assert execution.stdout == "hello\n"
assert execution.stderr == ""
assert execution.success is True
class TestClientRLMIntegration:
"""Integration tests for client.rlm() method."""
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_count_documents(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can count documents.
Agent program:
docs = list_documents(limit=1000)
print(len(docs))
"""
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document("First document about cats.", title="Doc 1")
await client.create_document("Second document about dogs.", title="Doc 2")
await client.create_document("Third document about birds.", title="Doc 3")
result = await client.rlm("How many documents are in the database?")
assert "3" in result.answer
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_aggregation(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can perform aggregation across documents.
Agent program:
import re
revs = {}
for d in ['Q1 Report', 'Q2 Report', 'Q3 Report']:
content = get_document(d)
if content:
vals = re.findall(r'\\$([\\d,]+)', content)
if vals:
rev = sum(int(v.replace(',', '')) for v in vals)
else:
rev = None
else:
rev = None
revs[d] = rev
print(revs)
"""
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"Sales report Q1: Revenue was $100,000.", title="Q1 Report"
)
await client.create_document(
"Sales report Q2: Revenue was $150,000.", title="Q2 Report"
)
await client.create_document(
"Sales report Q3: Revenue was $200,000.", title="Q3 Report"
)
result = await client.rlm(
"What is the total revenue across all quarterly reports?"
)
assert "450" in result.answer or "450,000" in result.answer
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_with_filter(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent respects filter parameter.
Agent program:
docs = list_documents(limit=1000)
print(len(docs))
print(docs[:5])
The filter is applied via context, so list_documents() only sees "Cats".
"""
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document("Cat document.", title="Cats")
await client.create_document("Dog document.", title="Dogs")
await client.create_document("Bird document.", title="Birds")
result = await client.rlm(
"How many documents are available?",
filter="title = 'Cats'",
)
assert "1" in result.answer
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_docling_document_structure(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can analyze document structure using DoclingDocument.
Agent program:
docs = list_documents(limit=20)
print(docs)
doc = get_docling_document('<doc_id>')
print(doc.name)
print('tables:', len(doc.tables))
print('pictures:', len(doc.pictures))
"""
from haiku.rag.client import HaikuRAG
pdf_path = Path("tests/data/doclaynet.pdf")
config = AppConfig()
config.processing.conversion_options.do_ocr = False
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document_from_source(pdf_path)
result = await client.rlm(
"How many tables are in the document? "
"Also tell me how many pictures/figures it contains."
)
# The doclaynet.pdf has 1 table and 1 picture
assert "1" in result.answer
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_semantic_analysis_with_llm(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can use llm() for semantic analysis combined with computation.
Agent program:
docs = list_documents(limit=100)
print(len(docs))
print([d['title'] for d in docs[:20]])
sentiments = {}
for title in ['Q1 Update', 'Q2 Update', 'Q3 Update']:
content = get_document(title)
if content:
result = llm(f"Classify sentiment as positive/negative/mixed: {content}")
sentiments[title] = result
print(sentiments)
"""
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"The new product launch exceeded expectations. Sales grew 40% "
"and customer feedback has been overwhelmingly positive. "
"Team morale is at an all-time high.",
title="Q1 Update",
)
await client.create_document(
"We faced significant challenges this quarter. Supply chain issues "
"caused delays, and we missed our revenue target by 15%. "
"Several key employees left the company.",
title="Q2 Update",
)
await client.create_document(
"Mixed results this quarter. While product quality improved, "
"marketing campaigns underperformed. Revenue was flat compared "
"to last year but customer retention increased.",
title="Q3 Update",
)
result = await client.rlm(
"Analyze the sentiment of each quarterly update. "
"How many quarters were positive, negative, and mixed?"
)
# Should identify: Q1=positive, Q2=negative, Q3=mixed
assert "positive" in result.answer.lower()
assert "negative" in result.answer.lower()
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_search_and_extract(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can use search() to find content and extract information.
Agent program:
results = search("document element types", limit=20)
print(len(results))
for r in results[:5]:
print(r['document_title'], r['chunk_id'], r['score'])
print(r['content'][:200])
results = search("DocBank element types", limit=10)
...
"""
from haiku.rag.client import HaikuRAG
pdf_path = Path("tests/data/doclaynet.pdf")
config = AppConfig()
config.processing.conversion_options.do_ocr = False
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document_from_source(pdf_path)
result = await client.rlm(
"Search for content about document element types or labels. "
"What are all the different document element types mentioned? "
"List them all."
)
# The doclaynet.pdf defines exactly 11 class labels for document elements
# Normalize Unicode hyphens (U+2011 non-breaking hyphen) to regular hyphens
answer_lower = result.answer.lower().replace("\u2011", "-")
expected_labels = [
"caption",
"footnote",
"formula",
"list-item",
"page-footer",
"page-header",
"picture",
"section-header",
"table",
"text",
"title",
]
# Check that the agent found at least 6 of the 11 labels
# (LLM summaries may not always include all labels)
found_labels = [
label
for label in expected_labels
if label in answer_lower or label.replace("-", " ") in answer_lower
]
assert len(found_labels) >= 6, (
f"Expected at least 6 labels, found {len(found_labels)}: {found_labels}"
)
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_with_preloaded_documents(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can use pre-loaded documents variable.
Agent program:
if 'documents' in dir():
for doc in documents:
print(doc['title'], len(doc['content']))
else:
print('No preloaded documents')
"""
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"The company was founded in 1985 by Jane Smith.",
title="Company History",
)
await client.create_document(
"Our mission is to make technology accessible to everyone.",
title="Mission Statement",
)
result = await client.rlm(
"Using the pre-loaded documents variable, "
"tell me when was the company founded and what is their mission?",
documents=["Company History", "Mission Statement"],
)
assert "1985" in result.answer
assert (
"accessible" in result.answer.lower()
or "technology" in result.answer.lower()
)

View file

@ -0,0 +1,32 @@
from haiku.rag.agents.rlm.models import CodeExecution, RLMResult
class TestCodeExecution:
def test_create_successful_execution(self):
execution = CodeExecution(
code="print('hello')",
stdout="hello\n",
stderr="",
success=True,
)
assert execution.code == "print('hello')"
assert execution.stdout == "hello\n"
assert execution.stderr == ""
assert execution.success is True
def test_create_failed_execution(self):
execution = CodeExecution(
code="1/0",
stdout="",
stderr="ZeroDivisionError: division by zero",
success=False,
)
assert execution.success is False
assert "ZeroDivisionError" in execution.stderr
class TestRLMResult:
def test_create_result(self):
result = RLMResult(answer="The answer is 42", program="print(42)")
assert result.answer == "The answer is 42"
assert result.program == "print(42)"

View file

@ -0,0 +1,251 @@
import os
from pathlib import Path
import pytest
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox, SandboxResult
from haiku.rag.client import HaikuRAG
from haiku.rag.config.models import RLMConfig
@pytest.fixture(scope="module")
def vcr_cassette_dir():
return str(Path(__file__).parent.parent.parent / "cassettes" / "test_sandbox")
def is_docker_available() -> bool:
"""Check if Docker daemon is available."""
try:
import subprocess
result = subprocess.run(["docker", "info"], capture_output=True, timeout=5)
return result.returncode == 0
except Exception:
return False
docker_required = pytest.mark.skipif(
not is_docker_available(),
reason="Docker daemon not available",
)
@pytest.mark.integration
class TestDockerSandboxBasics:
"""Test basic Docker sandbox functionality."""
@docker_required
@pytest.mark.asyncio
async def test_execute_simple_code(self, docker_sandbox):
"""Test executing simple code in the sandbox."""
result = await docker_sandbox.execute("print('hello world')")
assert isinstance(result, SandboxResult)
assert result.success
assert "hello world" in result.stdout
assert result.stderr == ""
@pytest.mark.integration
class TestDockerSandboxErrors:
"""Test error handling in Docker sandbox."""
@docker_required
@pytest.mark.asyncio
async def test_syntax_error(self, docker_sandbox):
"""Test that syntax errors are reported."""
result = await docker_sandbox.execute("def foo(")
assert not result.success
assert "SyntaxError" in result.stderr
@docker_required
@pytest.mark.asyncio
async def test_runtime_error(self, docker_sandbox):
"""Test that runtime errors are reported."""
result = await docker_sandbox.execute("x = 1/0")
assert not result.success
assert "ZeroDivisionError" in result.stderr
@docker_required
@pytest.mark.asyncio
async def test_name_error(self, docker_sandbox):
"""Test that name errors are reported."""
result = await docker_sandbox.execute("print(undefined_variable)")
assert not result.success
assert "NameError" in result.stderr
@docker_required
@pytest.mark.asyncio
async def test_missing_image(self, temp_db_path):
"""Test error when Docker image is not found."""
async with HaikuRAG(temp_db_path, create=True) as client:
config = RLMConfig(docker_image="nonexistent-image:v999.999.999")
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=config.docker_image
) as sandbox:
result = await sandbox.execute("print('hello')")
assert not result.success
assert (
"not found" in result.stderr.lower()
or "error" in result.stderr.lower()
)
@pytest.mark.integration
class TestDockerSandboxHaikuRAG:
"""Test haiku.rag functions in Docker sandbox."""
@docker_required
@pytest.mark.asyncio
async def test_list_documents_empty(self, docker_sandbox):
"""Test list_documents returns empty list for empty database."""
result = await docker_sandbox.execute(
"docs = list_documents()\nprint(type(docs).__name__, len(docs))"
)
assert result.success
assert "list 0" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_list_documents_with_data(self, temp_db_path, test_docker_image):
"""Test list_documents returns documents when populated."""
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="Test content",
uri="test://doc1",
title="Test Document",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"docs = list_documents()\nprint(len(docs))\nprint(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Test Document" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
@pytest.mark.skipif(
os.environ.get("CI") == "true",
reason="Requires Ollama - VCR can't capture calls from inside Docker",
)
async def test_search_with_data(self, temp_db_path, test_docker_image):
"""Test search function works."""
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="The quick brown fox jumps over the lazy dog.",
uri="test://animals",
title="Animals",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"results = search('fox', limit=5)\n"
"print(len(results))\n"
"if results:\n"
" print('fox' in results[0]['content'].lower())"
)
assert result.success
# Search should return at least one result
assert "True" in result.stdout or "1" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_get_document(self, temp_db_path, test_docker_image):
"""Test get_document function."""
async with HaikuRAG(temp_db_path, create=True) as client:
doc = await client.create_document(
content="Content about foxes and dogs.",
uri="test://doc",
title="Fox Document",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
f"content = get_document('{doc.id}')\n"
"print('foxes' in content.lower() if content else 'None')"
)
assert result.success
assert "True" in result.stdout
@docker_required
@pytest.mark.asyncio
async def test_get_document_not_found(self, docker_sandbox):
"""Test get_document returns None for missing document."""
result = await docker_sandbox.execute(
"content = get_document('nonexistent-id')\nprint(content is None)"
)
assert result.success
assert "True" in result.stdout
@pytest.mark.integration
class TestDockerSandboxContextFilter:
"""Test context filter is applied."""
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_filter_applied_to_list_documents(
self, temp_db_path, test_docker_image
):
"""Test that context filter is passed to list_documents."""
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="Public content",
uri="public://doc1",
title="Public Doc",
)
await client.create_document(
content="Private content",
uri="private://doc2",
title="Private Doc",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext(filter="uri LIKE 'public://%'")
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"docs = list_documents()\n"
"print(len(docs))\n"
"if docs:\n"
" print(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Public Doc" in result.stdout
assert "Private Doc" not in result.stdout
@pytest.mark.integration
class TestDockerSandboxPreloadedDocuments:
"""Test pre-loaded documents context variable."""
@docker_required
@pytest.mark.asyncio
async def test_documents_variable_not_available_without_preload(
self, docker_sandbox
):
"""documents variable is not available when context.documents is None."""
result = await docker_sandbox.execute("print(documents)")
assert not result.success
assert "NameError" in result.stderr

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View file

@ -669,3 +669,57 @@ def test_migrate_closes_store_on_exception(tmp_path):
app.migrate()
mock_store.close.assert_called_once()
@pytest.mark.asyncio
async def test_rlm(app: HaikuRAGApp, monkeypatch):
"""Test rlm method calls client.rlm and prints results."""
from haiku.rag.agents.rlm.models import RLMResult
mock_result = RLMResult(
answer="The total is 42.",
program="result = sum(values)\nprint(result)",
)
mock_client = AsyncMock()
mock_client.rlm = AsyncMock(return_value=mock_result)
mock_client.__aenter__.return_value = mock_client
mock_print = MagicMock()
monkeypatch.setattr(app.console, "print", mock_print)
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
await app.rlm("What is the total?")
mock_client.rlm.assert_called_once_with(
"What is the total?", documents=None, filter=None
)
calls = [str(c) for c in mock_print.call_args_list]
assert any("Question" in c for c in calls)
assert any("Program" in c for c in calls)
assert any("Answer" in c for c in calls)
@pytest.mark.asyncio
async def test_rlm_with_document_and_filter(app: HaikuRAGApp, monkeypatch):
"""Test rlm method passes document and filter to client."""
from haiku.rag.agents.rlm.models import RLMResult
mock_result = RLMResult(
answer="Answer with filter",
program="print('filtered')",
)
mock_client = AsyncMock()
mock_client.rlm = AsyncMock(return_value=mock_result)
mock_client.__aenter__.return_value = mock_client
mock_print = MagicMock()
monkeypatch.setattr(app.console, "print", mock_print)
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
await app.rlm("What is it?", document="doc-123", filter="uri LIKE '%test%'")
mock_client.rlm.assert_called_once_with(
"What is it?", documents=["doc-123"], filter="uri LIKE '%test%'"
)

View file

@ -85,6 +85,42 @@ async def test_client_document_crud(qa_corpus: Dataset, temp_db_path):
assert deleted_again is False
async def test_client_resolve_document(temp_db_path):
"""Test resolve_document finds documents by ID, title, or URI."""
async with HaikuRAG(temp_db_path, create=True) as client:
# Insert document directly via repository (no embeddings needed)
doc = Document(
content="Test content",
uri="test://resolve-test",
title="Resolve Test Doc",
)
doc = await client.document_repository.create(doc)
# Resolve by ID
by_id = await client.resolve_document(doc.id)
assert by_id is not None
assert by_id.id == doc.id
# Resolve by title
by_title = await client.resolve_document("Resolve Test Doc")
assert by_title is not None
assert by_title.id == doc.id
# Resolve by URI
by_uri = await client.resolve_document("test://resolve-test")
assert by_uri is not None
assert by_uri.id == doc.id
# Not found returns None
not_found = await client.resolve_document("nonexistent")
assert not_found is None
# SQL injection is escaped
injection = "x' OR title LIKE '%"
injected = await client.resolve_document(injection)
assert injected is None
@pytest.mark.vcr()
async def test_client_update_document(qa_corpus: Dataset, temp_db_path):
"""Test updating document with individual parameters."""
@ -1384,3 +1420,50 @@ async def test_client_convert_with_html_format(temp_db_path):
labels = [str(getattr(item, "label", "")) for item, _ in items]
assert "title" in labels
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_sql_injection_is_blocked_with_escaping(temp_db_path):
"""SQL injection is blocked when using _escape_sql_string.
This test verifies that _escape_sql_string properly prevents SQL injection
by escaping single quotes in user input.
"""
from haiku.rag.store.repositories.document import _escape_sql_string
async with HaikuRAG(temp_db_path, create=True) as client:
# Create documents
await client.create_document(
content="Secret classified data XYZ",
uri="secret://doc",
title="Secret",
)
await client.create_document(
content="Public report about weather",
uri="public://report",
title="Weather Report",
)
# Without escaping, this injection would match all documents
# by breaking out of the string literal: title = 'x' OR title LIKE '%'
injection_payload = "x' OR title LIKE '%"
# With proper escaping, single quotes become double quotes
# so the filter becomes: title = 'x'' OR title LIKE ''%'
# which searches for a literal title containing the injection string
safe_payload = _escape_sql_string(injection_payload)
docs = await client.list_documents(filter=f"title = '{safe_payload}'")
# Should find 0 documents (injection is escaped, searching for literal string)
assert len(docs) == 0
# Verify the escaping works correctly
assert safe_payload == "x'' OR title LIKE ''%"
# Verify unescaped injection would have matched documents (for test validity)
# This demonstrates that the injection works without escaping
docs_unescaped = await client.list_documents(
filter=f"title = '{injection_payload}'"
)
assert len(docs_unescaped) == 2 # SQL injection succeeds without escaping

View file

@ -307,3 +307,71 @@ async def test_mcp_research_question():
assert result.title == "Research Title"
assert result.executive_summary == "Summary"
mock_graph.run.assert_called_once()
@pytest.mark.asyncio
async def test_mcp_rlm_question():
"""Test rlm_question tool is properly wired."""
with tempfile.TemporaryDirectory() as temp_dir:
db_path = Path(temp_dir) / "test.lancedb"
mcp = create_mcp_server(db_path)
from haiku.rag.agents.rlm.models import RLMResult
mock_result = RLMResult(
answer="The total is 42.",
program="result = sum(values)\nprint(result)",
)
with patch("haiku.rag.mcp.HaikuRAG") as mock_rag_class:
mock_rag = AsyncMock()
mock_rag.rlm = AsyncMock(return_value=mock_result)
mock_rag_class.return_value.__aenter__ = AsyncMock(return_value=mock_rag)
mock_rag_class.return_value.__aexit__ = AsyncMock(return_value=None)
tools = await mcp.get_tools()
rlm_tool = next(t for t in tools.values() if t.name == "rlm_question")
result = await rlm_tool.fn(question="What is the total?")
assert result == "The total is 42."
mock_rag.rlm.assert_called_once_with(
"What is the total?", documents=None, filter=None
)
@pytest.mark.asyncio
async def test_mcp_rlm_question_with_document_and_filter():
"""Test rlm_question tool with document and filter parameters."""
with tempfile.TemporaryDirectory() as temp_dir:
db_path = Path(temp_dir) / "test.lancedb"
mcp = create_mcp_server(db_path)
from haiku.rag.agents.rlm.models import RLMResult
mock_result = RLMResult(
answer="Filtered answer",
program="print('filtered')",
)
with patch("haiku.rag.mcp.HaikuRAG") as mock_rag_class:
mock_rag = AsyncMock()
mock_rag.rlm = AsyncMock(return_value=mock_result)
mock_rag_class.return_value.__aenter__ = AsyncMock(return_value=mock_rag)
mock_rag_class.return_value.__aexit__ = AsyncMock(return_value=None)
tools = await mcp.get_tools()
rlm_tool = next(t for t in tools.values() if t.name == "rlm_question")
result = await rlm_tool.fn(
question="Analyze this",
document="doc-123",
filter="uri LIKE '%test%'",
)
assert result == "Filtered answer"
mock_rag.rlm.assert_called_once_with(
"Analyze this",
documents=["doc-123"],
filter="uri LIKE '%test%'",
)