danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #280 from ggozad/feat/monty

Browse source 
Replace Docker sandbox with pydantic-monty
This commit is contained in:
Yiorgis Gozadinos 2026-02-24 13:28:04 +02:00 committed by GitHub
commit 0ed1e5289c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
41 changed files with 6208 additions and 7012 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

19
.github/workflows/test.yml vendored
View file

@ -76,22 +76,3 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: false
test-docker-sandbox:
needs: [lint]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
with:
enable-cache: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: "pyproject.toml"
- name: Install dependencies
run: uv sync --all-extras
- name: Build Docker image
run: docker build -t haiku-rag-slim:test -f docker/Dockerfile.slim .
- name: Run Docker integration tests
run: uv run pytest tests/agents/rlm/test_sandbox.py -v

15
CHANGELOG.md
View file

@ -1,6 +1,21 @@
# Changelog
## [Unreleased]
### Changed
- **RLM sandbox**: Replaced Docker-based code execution with [pydantic-monty](https://github.com/pydantic/monty), a minimal secure Python interpreter written in Rust. Eliminates Docker as a runtime dependency for RLM with sub-millisecond sandbox startup
- **RLM sandbox functions**: Added `get_chunk(chunk_id)` for retrieving chunk content and metadata from search results. `get_docling_document(document_id)` now returns the full document structure as a JSON dict. All sandbox functions now require `await`
- **`RLMConfig`**: Removed `docker_image` and `docker_memory_limit` fields
### Added
- **RLM sandbox regex functions**: `regex_findall`, `regex_sub`, `regex_search`, `regex_split` for pattern matching without LLM calls
- **`HaikuRAG.get_chunk_by_id()`**: Public method for chunk lookup by ID
### Removed
- **`docker_sandbox.py`**, **`runner.py`**: Docker container plumbing replaced by `sandbox.py`
## [0.31.1] - 2026-02-20
### Fixed

154
docs/agents/rlm.md
View file

@ -5,13 +5,13 @@ The RLM agent enables complex analytical tasks by writing and executing Python c
- **Aggregation**: "How many documents mention security vulnerabilities?"
- **Computation**: "What's the average revenue across all quarterly reports?"
- **Multi-document analysis**: "Compare the key findings between Report A and Report B"
- **Structured data extraction**: "Extract all tables from the document and summarize them"
- **Structured data extraction**: "Extract all dollar amounts and compute totals"
## How It Works
1. The agent receives a question
2. It writes Python code to explore the knowledge base
3. Code executes in a sandboxed environment with access to haiku.rag functions
3. Code executes in a sandboxed Python interpreter with access to knowledge base functions
4. The agent iterates: run code, examine results, refine approach
5. Final answer is synthesized from the gathered data
@ -52,117 +52,37 @@ async with HaikuRAG(path_to_db) as client:
)
```
## Available Functions
## Sandbox Capabilities
Inside the sandbox, these functions are available (no imports needed):
The agent's code runs in a sandboxed Python interpreter ([pydantic-monty](https://github.com/pydantic/monty)) with access to these knowledge base functions:
### search(query, limit=10)
| Function | Description |
|----------|-------------|
| `search(query, limit)` | Hybrid search (vector + full-text) returning matching chunks with scores |
| `list_documents(limit, offset)` | List documents in the knowledge base |
| `get_document(id_or_title)` | Get full text content of a document |
| `get_chunk(chunk_id)` | Get a chunk with metadata (headings, page numbers, labels) for citations |
| `get_docling_document(document_id)` | Get the full DoclingDocument structure as a dict (texts, tables, pictures, pages) |
| `llm(prompt)` | Call an LLM for classification, summarization, or extraction |
| `regex_findall(pattern, text)`, `regex_sub(pattern, repl, text)`, `regex_search(pattern, text)`, `regex_split(pattern, text)` | Regular expression matching via Python's `re` module |
Search the knowledge base using hybrid search (vector + full-text).
When documents are pre-loaded via the `documents` parameter, they are injected as a `documents` variable accessible in the sandbox code.
```python
results = search("climate change impacts", limit=20)
for r in results:
print(r['document_title'], r['score'])
print(r['content'][:200])
```
### Python Features
Returns list of dicts with keys: `chunk_id`, `content`, `document_id`, `document_title`, `document_uri`, `score`, `page_numbers`, `headings`
The interpreter supports a subset of Python: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, try/except, and the `json` module.
### list_documents(limit=10, offset=0)
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements. For pattern matching, the agent can use the `regex_*` functions, string methods, or the `llm()` function.
List available documents in the knowledge base.
### Security
```python
docs = list_documents(limit=100)
for doc in docs:
print(doc['id'], doc['title'])
```
Code executes in an isolated interpreter with:
Returns list of dicts with keys: `id`, `title`, `uri`, `created_at`
### get_document(id_or_title)
Get the full text content of a document by ID, title, or URI.
```python
content = get_document("Q1 Report")
if content:
print(len(content), "characters")
```
Returns the document content as a string, or `None` if not found.
### get_docling_document(id_or_title)
Get the structured DoclingDocument object for advanced analysis of tables, figures, and document structure.
```python
doc = get_docling_document("Technical Manual")
if doc:
print(f"Tables: {len(doc.tables)}")
print(f"Pictures: {len(doc.pictures)}")
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
### llm(prompt)
Call an LLM directly for classification, summarization, or extraction tasks.
```python
content = get_document("Q1 Report")
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
Use this when you have content and need LLM reasoning without RAG search.
## Pre-loaded Documents
When documents are pre-loaded via the `documents` parameter, they're available as a `documents` variable:
```python
# Available when documents are pre-loaded
for doc in documents:
print(doc['title'], len(doc['content']))
```
Each document dict has keys: `id`, `title`, `uri`, `content`
## Imports
The sandbox runs in a Docker container with full Python available. Any module installed in the container image can be imported:
```python
import re
import json
from collections import Counter
# Extract and count patterns
results = search("error", limit=50)
error_types = []
for r in results:
matches = re.findall(r'Error: (\w+)', r['content'])
error_types.extend(matches)
print(Counter(error_types).most_common(10))
```
The default image (`ghcr.io/ggozad/haiku.rag-slim`) includes the Python standard library. Custom images can add additional packages like `pandas` or `numpy`.
## Docker Sandbox
Code executes in an isolated Docker container with:
- **Read-only database**: The LanceDB database is mounted read-only
- **Memory limits**: Configurable memory limit (default 512MB)
- **Execution timeout**: Code times out after configurable limit (default 60s)
- **No filesystem access**: Code cannot read or write files
- **No network access**: Code cannot make HTTP requests or open sockets
- **No imports**: Only the `json` module is available
- **Execution timeout**: Configurable limit (default 60s)
- **Output truncation**: Large outputs are truncated to prevent memory issues
- **Container reuse**: Within a single `rlm()` call, the container stays warm for multiple code executions
## Context Filter
@ -176,11 +96,7 @@ result = await client.rlm(
)
```
This is useful for:
- Scoping to specific document sets
- Enforcing access control
- Limiting context for focused analysis
This is useful for scoping to specific document sets, enforcing access control, or limiting context for focused analysis.
## Configuration
@ -193,26 +109,4 @@ rlm:
name: claude-sonnet-4-20250514
code_timeout: 60.0 # Max seconds for code execution
max_output_chars: 50000 # Truncate output after this many chars
docker_image: "ghcr.io/ggozad/haiku.rag-slim:latest" # Container image
docker_memory_limit: "512m" # Container memory limit
```
### Custom Docker Image
To add additional Python packages, create a custom Dockerfile:
```dockerfile
FROM ghcr.io/ggozad/haiku.rag-slim:latest
RUN pip install pandas numpy
```
Build and configure:
```bash
docker build -t my-rlm-image .
```
```yaml
rlm:
docker_image: "my-rlm-image"
```

4
docs/skills/index.md
View file

@ -7,7 +7,7 @@ haiku.rag exposes its RAG capabilities as [haiku.skills](https://github.com/ggoz
| Skill | Description |
|-------|-------------|
| [`rag`](rag.md) | Search, retrieve, and answer questions from the knowledge base |
| [`rag-rlm`](rlm.md) | Computational analysis via code execution (requires Docker) |
| [`rag-rlm`](rlm.md) | Computational analysis via code execution |
## Discovery
@ -16,7 +16,7 @@ Skills are registered as Python entrypoints under `haiku.skills`. They are disco
```bash
haiku-skills list --use-entrypoints
# rag — Search, retrieve and analyze documents using RAG.
# rag-rlm — Analyze documents using code execution in a Docker sandbox.
# rag-rlm — Analyze documents using code execution in a sandboxed interpreter.
```
## Usage

5
docs/skills/rlm.md
View file

@ -1,9 +1,6 @@
# RLM Skill
The RLM (Reflexion Language Model) skill provides computational analysis via code execution. It writes and runs Python code in an isolated Docker sandbox to answer questions that require computation, aggregation, or data traversal.
!!! warning "Requires Docker"
The `analyze` tool executes code in a Docker sandbox. Docker must be running on the host machine. This skill is not suitable for Docker-deployed applications — use the [`rag`](rag.md) skill alone in those environments.
The RLM (Recursive Language Model) skill provides computational analysis via code execution. It writes and runs Python code in a sandboxed interpreter to answer questions that require computation, aggregation, or data traversal.
## `create_skill(db_path?, config?)`

2
docs/tools.md
View file

@ -61,7 +61,7 @@ docs = create_document_toolset(config)
### Analysis Toolset
`create_analysis_toolset()` provides computational analysis via the RLM agent (Docker sandbox).
`create_analysis_toolset()` provides computational analysis via the RLM agent.
```python
from haiku.rag.tools import create_analysis_toolset

4
haiku_rag_slim/haiku/rag/agents/rlm/__init__.py
View file

@ -1,16 +1,16 @@
from haiku.rag.agents.rlm.agent import create_rlm_agent
from haiku.rag.agents.rlm.dependencies import RLMContext, RLMDeps
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox, SandboxResult
from haiku.rag.agents.rlm.models import CodeExecution, RLMResult
from haiku.rag.agents.rlm.prompts import RLM_SYSTEM_PROMPT
from haiku.rag.agents.rlm.sandbox import Sandbox, SandboxResult
__all__ = [
"CodeExecution",
"DockerSandbox",
"RLMContext",
"RLMDeps",
"RLMResult",
"RLM_SYSTEM_PROMPT",
"Sandbox",
"SandboxResult",
"create_rlm_agent",
]

5
haiku_rag_slim/haiku/rag/agents/rlm/agent.py
View file

@ -32,11 +32,10 @@ def create_rlm_agent(config: AppConfig) -> Agent[RLMDeps, RLMResult]:
@agent.tool
async def execute_code(ctx: RunContext[RLMDeps], code: str) -> CodeExecution:
"""Execute Python code in a Docker-sandboxed environment.
"""Execute Python code in a sandboxed interpreter.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
get_document, get_chunk, llm).
Use print() to output results.

4
haiku_rag_slim/haiku/rag/agents/rlm/dependencies.py
View file

@ -4,7 +4,7 @@ from typing import TYPE_CHECKING
from haiku.rag.store.models import Document
if TYPE_CHECKING:
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox
from haiku.rag.agents.rlm.sandbox import Sandbox
@dataclass
@ -19,5 +19,5 @@ class RLMContext:
class RLMDeps:
"""Dependencies for RLM agent."""
sandbox: "DockerSandbox"
sandbox: "Sandbox"
context: RLMContext = field(default_factory=RLMContext)

216
haiku_rag_slim/haiku/rag/agents/rlm/docker_sandbox.py
View file

@ -1,216 +0,0 @@
"""Docker-based sandboxed execution."""
import asyncio
import json
import os
import subprocess
import sys
from dataclasses import dataclass
from typing import TYPE_CHECKING
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.config.models import RLMConfig
if TYPE_CHECKING:
from haiku.rag.client import HaikuRAG
@dataclass
class SandboxResult:
"""Result of executing code in the sandbox."""
stdout: str
stderr: str
success: bool
class DockerSandbox: # pragma: no cover
"""Execute code in a persistent Docker container.
Use as an async context manager to manage container lifecycle:
async with DockerSandbox(client, config, context) as sandbox:
result = await sandbox.execute("print('hello')")
result = await sandbox.execute("print('world')")
"""
DEFAULT_IMAGE = "ghcr.io/ggozad/haiku.rag-slim:latest"
haiku_client: "HaikuRAG"
config: RLMConfig
context: RLMContext
image: str
_process: subprocess.Popen[bytes] | None
def __init__(
self,
client: "HaikuRAG",
config: RLMConfig,
context: RLMContext,
image: str | None = None,
):
self.haiku_client = client
self.config = config
self.context = context
self.image = image or self.DEFAULT_IMAGE
self._process = None
def _build_docker_cmd(self) -> list[str]:
"""Build the docker run command."""
db_path = str(self.haiku_client.store.db_path)
env_list = ["-e", "HAIKU_DB_PATH=/data/db.lancedb"]
if self.context.filter:
env_list.extend(["-e", f"HAIKU_FILTER={self.context.filter}"])
ollama_host = os.environ.get("OLLAMA_HOST", "")
ollama_base_url = os.environ.get("OLLAMA_BASE_URL", "")
if sys.platform == "darwin":
if not ollama_host or "localhost" in ollama_host:
ollama_host = "http://host.docker.internal:11434"
if not ollama_base_url or "localhost" in ollama_base_url:
ollama_base_url = "http://host.docker.internal:11434"
if ollama_host:
env_list.extend(["-e", f"OLLAMA_HOST={ollama_host}"])
if ollama_base_url:
env_list.extend(["-e", f"OLLAMA_BASE_URL={ollama_base_url}"])
for key in [
"ANTHROPIC_API_KEY",
"OPENAI_API_KEY",
"VOYAGE_API_KEY",
"COHERE_API_KEY",
]:
if value := os.environ.get(key):
env_list.extend(["-e", f"{key}={value}"])
return [
"docker",
"run",
"--rm",
"-i",
"-v",
f"{db_path}:/data/db.lancedb:ro",
f"--memory={self.config.docker_memory_limit}",
"--network=host",
*env_list,
self.image,
"python",
"-m",
"haiku.rag.agents.rlm.runner",
]
async def __aenter__(self) -> "DockerSandbox":
"""Start the container."""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self._start_container)
return self
async def __aexit__(
self, exc_type: object, exc_val: object, exc_tb: object
) -> None:
"""Stop the container."""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self._stop_container)
def _start_container(self) -> None:
"""Start the persistent container process."""
if self._process is not None:
return
cmd = self._build_docker_cmd()
self._process = subprocess.Popen(
cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
def _stop_container(self) -> None:
"""Stop the container process."""
if self._process is None:
return
try:
if self._process.stdin:
try:
self._process.stdin.close()
except BrokenPipeError:
pass
self._process.terminate()
self._process.wait(timeout=5)
except subprocess.TimeoutExpired:
self._process.kill()
self._process.wait()
finally:
self._process = None
async def execute(self, code: str) -> SandboxResult:
"""Execute code in the container."""
if self._process is None:
return SandboxResult(
stdout="",
stderr="Container not started. Use 'async with' context manager.",
success=False,
)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, self._execute_sync, code)
def _execute_sync(self, code: str) -> SandboxResult:
"""Send code to container and read result."""
assert self._process is not None and self._process.stdin is not None
try:
message = json.dumps({"code": code})
length_line = f"{len(message)}\n".encode()
self._process.stdin.write(length_line)
self._process.stdin.write(message.encode())
self._process.stdin.flush()
if self._process.stdout is None:
return SandboxResult(
stdout="", stderr="No stdout from container.", success=False
)
length_line = self._process.stdout.readline()
if not length_line:
stderr = ""
if self._process.stderr:
stderr = self._process.stderr.read().decode()
return SandboxResult(
stdout="",
stderr=stderr or "Container closed unexpectedly.",
success=False,
)
length = int(length_line.strip())
response = self._process.stdout.read(length).decode()
result_data = json.loads(response)
return SandboxResult(
stdout=result_data.get("stdout", ""),
stderr=result_data.get("stderr", ""),
success=result_data.get("success", False),
)
except subprocess.TimeoutExpired:
return SandboxResult(
stdout="",
stderr=f"Execution timed out after {self.config.code_timeout} seconds",
success=False,
)
except json.JSONDecodeError as e:
return SandboxResult(
stdout="",
stderr=f"Invalid response from container: {e}",
success=False,
)
except Exception as e:
return SandboxResult(
stdout="",
stderr=f"Execution error: {e}",
success=False,
)

157
haiku_rag_slim/haiku/rag/agents/rlm/prompts.py
View file

@ -1,33 +1,52 @@
RLM_SYSTEM_PROMPT = """You are a Recursive Language Model (RLM) agent that solves complex research questions by writing and executing Python code.
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside execute_code. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just call them with `await`:
- results = await search("query") ✓ CORRECT
- import search ✗ WRONG - will fail
- results = search("query") ✗ WRONG - must use await
## Available Functions
### search(query, limit=10) -> list[dict]
### await search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
### await list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
### await get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### await get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
### await get_docling_document(document_id) -> dict | None
Get the full document structure as a dict (DoclingDocument format).
Use `list_documents()` or search results to get document IDs first.
- `texts`: list of text items, each with `text`, `label` (e.g. "title", "text", "section_header", "list_item"), and `prov` (provenance with page/bounding box)
- `tables`: list of tables, each with `data` containing `grid` (list of rows, each row a list of cells with `text`), `num_rows`, `num_cols`
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
@ -40,107 +59,69 @@ If documents were pre-loaded for this session, a `documents` variable is availab
for doc in documents:
print(doc['title'], len(doc['content']))
```
Check if it exists with: `if 'documents' in dir(): ...`
Check if it exists with: `try: documents ... except NameError: ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
1. **Explore First**: Start by listing documents or searching to understand what's available. Document `title` is often None — use `uri` or `id` to identify documents instead.
2. **If get_document returns None**: Use `await list_documents()` to see available documents (check `uri` and `id`), or `await search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
4. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use `await llm()`.
5. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
docs = await list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
content = await get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with regex
```python
import re
numbers = []
results = search("financial data", limit=20)
results = await search("financial data", limit=20)
for r in results:
matches = re.findall(r'\\$([\\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
amounts = await regex_findall(r'\\$([\\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using llm() for classification
### Extracting tables from a document
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
docs = await list_documents(limit=10)
for d in docs:
doc = await get_docling_document(d['id'])
if doc:
tables = doc.get('tables', [])
if tables:
print(f"{d['title']}: {len(tables)} table(s)")
for i, table in enumerate(tables):
grid = table.get('data', {}).get('grid', [])
for row in grid:
cells = [cell.get('text', '') for cell in row]
print(f" Table {i}: {cells}")
```
## Workflow
1. **ALWAYS start by using execute_code** to explore the knowledge base
2. Run multiple code blocks as needed to gather information
3. After collecting data, provide your final answer
## Output Format
CRITICAL: Your final response MUST be valid JSON matching this exact schema:
Your final response MUST be valid JSON matching this exact schema:
```json
{"answer": "Your complete answer here as a string", "program": "Your final consolidated program here as a string"}
```
@ -150,4 +131,4 @@ CRITICAL: Your final response MUST be valid JSON matching this exact schema:
Do NOT return arbitrary JSON structures. Always use the exact format: {"answer": "...", "program": "..."}
CRITICAL: You MUST call execute_code at least once before providing your answer. Never give up without trying to execute code first."""
You MUST call execute_code at least once before providing your answer. Never give up without trying to execute code first."""

190
haiku_rag_slim/haiku/rag/agents/rlm/runner.py
View file

@ -1,190 +0,0 @@
"""Entry point for sandboxed code execution in Docker container."""
import asyncio
import json
import sys
import traceback
from io import StringIO
from typing import Any
def build_namespace( # pragma: no cover
client: Any, config: Any, context: Any, loop: asyncio.AbstractEventLoop
) -> dict[str, Any]:
"""Build execution namespace with haiku.rag functions injected."""
def run_async(coro: Any) -> Any:
"""Run async coroutine from sync context using thread-safe scheduling."""
future = asyncio.run_coroutine_threadsafe(coro, loop)
return future.result(timeout=config.rlm.code_timeout)
def search(query: str, limit: int = 10) -> list[dict]:
async def _search() -> Any:
return await client.search(query, limit=limit, filter=context.filter)
results = run_async(_search())
return [
{
"chunk_id": r.chunk_id,
"content": r.content,
"document_id": r.document_id,
"document_title": r.document_title,
"document_uri": r.document_uri,
"score": r.score,
"page_numbers": r.page_numbers,
"headings": r.headings,
}
for r in results
]
def list_documents(limit: int = 10, offset: int = 0) -> list[dict]:
async def _list() -> Any:
return await client.list_documents(
limit=limit, offset=offset, filter=context.filter
)
docs = run_async(_list())
return [
{
"id": d.id,
"title": d.title,
"uri": d.uri,
"created_at": str(d.created_at),
}
for d in docs
]
def get_document(id_or_title: str) -> str | None:
async def _get() -> str | None:
doc = await client.resolve_document(id_or_title)
return doc.content if doc else None
return run_async(_get())
def get_docling_document(id_or_title: str) -> Any:
async def _get() -> Any:
doc = await client.resolve_document(id_or_title)
return doc.get_docling_document() if doc else None
return run_async(_get())
def llm(prompt: str) -> str:
async def _llm() -> str:
from pydantic_ai import Agent
from haiku.rag.utils import get_model
model = get_model(config.rlm.model, config)
agent: Agent[None, str] = Agent(model, output_type=str)
result = await agent.run(prompt)
return result.output
return run_async(_llm())
namespace: dict[str, Any] = {
"search": search,
"list_documents": list_documents,
"get_document": get_document,
"get_docling_document": get_docling_document,
"llm": llm,
}
if context.documents:
namespace["documents"] = [
{"id": d.id, "title": d.title, "uri": d.uri, "content": d.content}
for d in context.documents
]
return namespace
def execute_code(
code: str, namespace: dict[str, Any], max_output_chars: int
) -> dict[str, Any]:
"""Execute code and capture output."""
stdout_capture = StringIO()
original_stdout = sys.stdout
try:
sys.stdout = stdout_capture
exec(code, namespace)
stdout = stdout_capture.getvalue()
if len(stdout) > max_output_chars:
stdout = stdout[:max_output_chars] + "\n... (output truncated)"
return {
"success": True,
"stdout": stdout,
"stderr": "",
}
except Exception:
return {
"success": False,
"stdout": stdout_capture.getvalue(),
"stderr": traceback.format_exc(),
}
finally:
sys.stdout = original_stdout
def send_response(result: dict[str, Any]) -> None:
"""Send length-prefixed JSON response."""
response = json.dumps(result)
sys.stdout.write(f"{len(response)}\n")
sys.stdout.write(response)
sys.stdout.flush()
async def main() -> None: # pragma: no cover
"""Main entry point for container execution.
Runs a loop reading length-prefixed JSON messages and executing code.
"""
import concurrent.futures
import os
from pathlib import Path
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.client import HaikuRAG
from haiku.rag.config import get_config
config = get_config()
db_path = Path(os.environ.get("HAIKU_DB_PATH", "/data/db.lancedb"))
filter_expr = os.environ.get("HAIKU_FILTER")
context = RLMContext(filter=filter_expr)
max_output_chars = config.rlm.max_output_chars
loop = asyncio.get_running_loop()
async with HaikuRAG(db_path, config=config, read_only=True) as client:
namespace = build_namespace(client, config, context, loop)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
while True:
# Read length-prefixed message
length_line = sys.stdin.readline()
if not length_line:
break
try:
length = int(length_line.strip())
message = sys.stdin.read(length)
request = json.loads(message)
code = request.get("code", "")
result = await loop.run_in_executor(
executor, execute_code, code, namespace, max_output_chars
)
send_response(result)
except (ValueError, json.JSONDecodeError) as e:
send_response(
{
"success": False,
"stdout": "",
"stderr": f"Invalid request: {e}",
}
)
if __name__ == "__main__":
asyncio.run(main())

231
haiku_rag_slim/haiku/rag/agents/rlm/sandbox.py Normal file
View file

@ -0,0 +1,231 @@
import json
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Literal
import pydantic_monty
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.config.models import AppConfig
from haiku.rag.store.compression import decompress_json
if TYPE_CHECKING:
from haiku.rag.client import HaikuRAG
@dataclass
class SandboxResult:
"""Result of executing code in the sandbox."""
stdout: str
stderr: str
success: bool
class Sandbox:
"""Execute code in a sandboxed Python interpreter.
Uses pydantic-monty, a minimal secure Python interpreter written in Rust.
External functions (search, list_documents, etc.) are called by Monty code
using ``await`` and resolved asynchronously on the host.
sandbox = Sandbox(client, config, context)
result = await sandbox.execute("print('hello')")
"""
_client: "HaikuRAG"
_config: AppConfig
_context: RLMContext
def __init__(
self,
client: "HaikuRAG",
config: AppConfig,
context: RLMContext,
):
self._client = client
self._config = config
self._context = context
def _build_external_functions(self) -> dict[str, Any]:
"""Build async external functions for the Monty interpreter."""
client = self._client
config = self._config
context = self._context
async def search(query: str, limit: int = 10) -> list[dict[str, Any]]:
results = await client.search(query, limit=limit, filter=context.filter)
return [
{
"chunk_id": r.chunk_id,
"content": r.content,
"document_id": r.document_id,
"document_title": r.document_title,
"document_uri": r.document_uri,
"score": r.score,
"page_numbers": r.page_numbers,
"headings": r.headings,
}
for r in results
]
async def list_documents(
limit: int = 10, offset: int = 0
) -> list[dict[str, Any]]:
docs = await client.list_documents(
limit=limit, offset=offset, filter=context.filter
)
return [
{
"id": d.id,
"title": d.title,
"uri": d.uri,
"created_at": str(d.created_at),
}
for d in docs
]
async def get_document(id_or_title: str) -> str | None:
doc = await client.resolve_document(id_or_title)
return doc.content if doc else None
async def get_chunk(chunk_id: str) -> dict[str, Any] | None:
chunk = await client.get_chunk_by_id(chunk_id)
if not chunk:
return None
meta = chunk.get_chunk_metadata()
doc_title = chunk.document_title
if not doc_title and chunk.document_id:
doc = await client.get_document_by_id(chunk.document_id)
if doc:
doc_title = doc.title
return {
"chunk_id": chunk.id,
"content": chunk.content,
"document_id": chunk.document_id,
"document_title": doc_title,
"headings": meta.headings,
"page_numbers": meta.page_numbers,
"labels": meta.labels,
}
async def get_docling_document(
document_id: str,
) -> dict[str, Any] | None:
doc = await client.get_document_by_id(document_id)
if not doc or not doc.docling_document:
return None
json_str = decompress_json(doc.docling_document)
return json.loads(json_str)
async def llm(prompt: str) -> str:
from pydantic_ai import Agent
from haiku.rag.utils import get_model
model = get_model(config.rlm.model, config)
agent: Agent[None, str] = Agent(model, output_type=str)
result = await agent.run(prompt)
return result.output
async def regex_findall(pattern: str, text: str) -> list[str]:
return re.findall(pattern, text)
async def regex_sub(pattern: str, repl: str, text: str) -> str:
return re.sub(pattern, repl, text)
async def regex_search(pattern: str, text: str) -> dict[str, Any] | None:
m = re.search(pattern, text)
if m is None:
return None
return {
"group": m.group(),
"groups": list(m.groups()),
"start": m.start(),
"end": m.end(),
}
async def regex_split(pattern: str, text: str) -> list[str]:
return re.split(pattern, text)
return {
"search": search,
"list_documents": list_documents,
"get_document": get_document,
"get_chunk": get_chunk,
"get_docling_document": get_docling_document,
"llm": llm,
"regex_findall": regex_findall,
"regex_sub": regex_sub,
"regex_search": regex_search,
"regex_split": regex_split,
}
async def execute(self, code: str) -> SandboxResult:
"""Execute Python code in the Monty interpreter."""
external_fns = self._build_external_functions()
input_names: list[str] = []
inputs: dict[str, Any] | None = None
if self._context.documents:
input_names.append("documents")
inputs = {
"documents": [
{
"id": d.id,
"title": d.title,
"uri": d.uri,
"content": d.content,
}
for d in self._context.documents
]
}
try:
monty = pydantic_monty.Monty(
code,
inputs=input_names,
external_functions=list(external_fns.keys()),
)
except (
pydantic_monty.MontySyntaxError,
pydantic_monty.MontyRuntimeError,
) as e:
return SandboxResult(stdout="", stderr=str(e), success=False)
stdout_lines: list[str] = []
def print_callback(_stream: Literal["stdout"], text: str) -> None:
stdout_lines.append(text)
max_chars = self._config.rlm.max_output_chars
limits: pydantic_monty.ResourceLimits = {
"max_duration_secs": self._config.rlm.code_timeout,
}
try:
output = await pydantic_monty.run_monty_async(
monty,
inputs=inputs,
external_functions=external_fns,
limits=limits,
print_callback=print_callback,
)
except pydantic_monty.MontyRuntimeError as e:
stdout = "".join(stdout_lines)
if len(stdout) > max_chars:
stdout = stdout[:max_chars] + "\n... (output truncated)"
return SandboxResult(stdout=stdout, stderr=str(e), success=False)
stdout = "".join(stdout_lines)
if output is not None:
stdout_with_output = f"{stdout}{output}" if stdout else str(output)
else:
stdout_with_output = stdout
if len(stdout_with_output) > max_chars:
stdout_with_output = (
stdout_with_output[:max_chars] + "\n... (output truncated)"
)
return SandboxResult(stdout=stdout_with_output, stderr="", success=True)

2
haiku_rag_slim/haiku/rag/app.py
View file

@ -354,7 +354,7 @@ class HaikuRAGApp: # pragma: no cover
read_only=self.read_only,
before=self.before,
) as self.client:
chunk = await self.client.chunk_repository.get_by_id(chunk_id)
chunk = await self.client.get_chunk_by_id(chunk_id)
if not chunk:
self.console.print(f"[red]Chunk with id {chunk_id} not found.[/red]")
return

2
haiku_rag_slim/haiku/rag/chat/app.py
View file

@ -344,7 +344,7 @@ class ChatApp(App):
return
citation = selected_widgets[0].citation
chunk = await self.client.chunk_repository.get_by_id(citation.chunk_id)
chunk = await self.client.get_chunk_by_id(citation.chunk_id)
if not chunk:
return

34
haiku_rag_slim/haiku/rag/client.py
View file

@ -732,6 +732,17 @@ class HaikuRAG:
"""
return await self.document_repository.get_by_id(document_id)
async def get_chunk_by_id(self, chunk_id: str) -> Chunk | None:
"""Get a chunk by its ID.
Args:
chunk_id: The unique identifier of the chunk.
Returns:
The Chunk instance if found, None otherwise.
"""
return await self.chunk_repository.get_by_id(chunk_id)
async def get_document_by_uri(self, uri: str) -> Document | None:
"""Get a document by its URI.
@ -1379,9 +1390,9 @@ class HaikuRAG:
RLMResult with the answer and the final consolidated program.
"""
from haiku.rag.agents.rlm import (
DockerSandbox,
RLMContext,
RLMDeps,
Sandbox,
create_rlm_agent,
)
@ -1395,21 +1406,20 @@ class HaikuRAG:
loaded_docs.append(doc)
context.documents = loaded_docs if loaded_docs else None
async with DockerSandbox(
sandbox = Sandbox(
client=self,
config=self._config.rlm,
config=self._config,
context=context,
image=self._config.rlm.docker_image,
) as sandbox:
deps = RLMDeps(
sandbox=sandbox,
context=context,
)
)
deps = RLMDeps(
sandbox=sandbox,
context=context,
)
agent = create_rlm_agent(self._config)
result = await agent.run(question, deps=deps)
agent = create_rlm_agent(self._config)
result = await agent.run(question, deps=deps)
return result.output
return result.output
async def visualize_chunk(self, chunk: Chunk) -> list:
"""Render page images with bounding box highlights for a chunk.

2
haiku_rag_slim/haiku/rag/config/models.py
View file

@ -104,8 +104,6 @@ class RLMConfig(BaseModel):
)
code_timeout: float = 60.0
max_output_chars: int = 50_000
docker_image: str = "ghcr.io/ggozad/haiku.rag-slim:latest"
docker_memory_limit: str = "512m"
class PictureDescriptionConfig(BaseModel):

4
haiku_rag_slim/haiku/rag/inspector/widgets/search_modal.py
View file

@ -108,9 +108,7 @@ class SearchModal(Screen):
self.chunks = []
for result in self.search_results:
if result.chunk_id:
chunk = await self.client.chunk_repository.get_by_id(
result.chunk_id
)
chunk = await self.client.get_chunk_by_id(result.chunk_id)
if chunk:
self.chunks.append(chunk)

4
haiku_rag_slim/haiku/rag/skills/rag-rlm/SKILL.md
View file

@ -1,7 +1,7 @@
---
name: rag-rlm
description: >
Computational analysis of the knowledge base via code execution in a Docker sandbox.
Computational analysis of the knowledge base via code execution in a sandboxed Python interpreter.
Use for questions requiring counting, aggregation, statistics, data traversal,
comparison across documents, or any task best answered by writing Python code.
Examples: "how many pages?", "compare table 3 across documents",
@ -10,4 +10,4 @@ description: >
# RLM Analysis
Use the `analyze` tool for complex analytical questions. It writes and executes Python code against the knowledge base in an isolated Docker sandbox.
Use the `analyze` tool for complex analytical questions. It writes and executes Python code against the knowledge base in a sandboxed Python interpreter.

31
haiku_rag_slim/haiku/rag/tools/analysis.py
View file

@ -3,7 +3,7 @@ from pydantic_ai import FunctionToolset, RunContext
from haiku.rag.agents.rlm.agent import create_rlm_agent
from haiku.rag.agents.rlm.dependencies import RLMContext, RLMDeps
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox
from haiku.rag.agents.rlm.sandbox import Sandbox
from haiku.rag.config.models import AppConfig
from haiku.rag.tools.context import RAGDeps
from haiku.rag.tools.filters import (
@ -62,26 +62,25 @@ def create_analysis_toolset(
rlm_context = RLMContext(filter=effective_filter)
async with DockerSandbox(
sandbox = Sandbox(
client=client,
config=config.rlm,
config=config,
context=rlm_context,
image=config.rlm.docker_image,
) as sandbox:
deps = RLMDeps(
sandbox=sandbox,
context=rlm_context,
)
)
deps = RLMDeps(
sandbox=sandbox,
context=rlm_context,
)
rlm_agent = create_rlm_agent(config)
result = await rlm_agent.run(task, deps=deps)
rlm_agent = create_rlm_agent(config)
result = await rlm_agent.run(task, deps=deps)
program = result.output.program
program = result.output.program
return AnalysisResult(
answer=result.output.answer,
code_executed=bool(program),
)
return AnalysisResult(
answer=result.output.answer,
code_executed=bool(program),
)
toolset: FunctionToolset[RAGDeps] = FunctionToolset()
toolset.add_function(analyze, name=tool_name)

1
haiku_rag_slim/pyproject.toml
View file

@ -31,6 +31,7 @@ dependencies = [
"pathspec>=1.0.3",
"pydantic>=2.12.5",
"pydantic-ai-slim[openai,fastmcp,logfire,ag-ui]>=1.46.0",
"pydantic-monty>=0.0.7",
"python-dotenv>=1.2.1",
"pyyaml>=6.0.3",
"rich>=14.2.0",

45
tests/agents/rlm/conftest.py
View file

@ -1,39 +1,9 @@
import os
import subprocess
from pathlib import Path
import pytest
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox
from haiku.rag.agents.rlm.sandbox import Sandbox
from haiku.rag.client import HaikuRAG
from haiku.rag.config.models import RLMConfig
TEST_DOCKER_IMAGE = os.environ.get("HAIKU_TEST_DOCKER_IMAGE", "haiku-rag-slim:test")
@pytest.fixture(scope="session")
def test_docker_image():
"""Build and return the Docker image for testing."""
if os.environ.get("CI"):
return TEST_DOCKER_IMAGE
project_root = Path(__file__).parent.parent.parent.parent
dockerfile = project_root / "docker" / "Dockerfile.slim"
if not dockerfile.exists():
pytest.skip(f"Dockerfile.slim not found at {dockerfile}")
result = subprocess.run(
["docker", "build", "-t", TEST_DOCKER_IMAGE, "-f", str(dockerfile), "."],
cwd=project_root,
capture_output=True,
text=True,
)
if result.returncode != 0:
pytest.fail(f"Failed to build Docker image:\n{result.stderr}")
return TEST_DOCKER_IMAGE
from haiku.rag.config.models import AppConfig
@pytest.fixture
@ -44,11 +14,8 @@ async def empty_client(temp_db_path):
@pytest.fixture
async def docker_sandbox(empty_client, test_docker_image):
"""Create a Docker sandbox for testing."""
config = RLMConfig(docker_image=test_docker_image)
async def sandbox(empty_client):
"""Create a Monty sandbox for testing."""
config = AppConfig()
context = RLMContext()
async with DockerSandbox(
client=empty_client, config=config, context=context, image=test_docker_image
) as sandbox:
yield sandbox
return Sandbox(client=empty_client, config=config, context=context)

67
tests/agents/rlm/test_agent.py
View file

@ -47,9 +47,7 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_count_documents(
self, allow_model_requests, temp_db_path, test_docker_image
):
async def test_rlm_count_documents(self, allow_model_requests, temp_db_path):
"""Test RLM agent can count documents.
Agent program:
@ -59,7 +57,7 @@ class TestClientRLMIntegration:
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document("First document about cats.", title="Doc 1")
await client.create_document("Second document about dogs.", title="Doc 2")
@ -71,9 +69,7 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_aggregation(
self, allow_model_requests, temp_db_path, test_docker_image
):
async def test_rlm_aggregation(self, allow_model_requests, temp_db_path):
"""Test RLM agent can perform aggregation across documents.
Agent program:
@ -95,7 +91,7 @@ class TestClientRLMIntegration:
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"Sales report Q1: Revenue was $100,000.", title="Q1 Report"
@ -115,9 +111,7 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_with_filter(
self, allow_model_requests, temp_db_path, test_docker_image
):
async def test_rlm_with_filter(self, allow_model_requests, temp_db_path):
"""Test RLM agent respects filter parameter.
Agent program:
@ -130,7 +124,7 @@ class TestClientRLMIntegration:
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document("Cat document.", title="Cats")
await client.create_document("Dog document.", title="Dogs")
@ -145,42 +139,36 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_docling_document_structure(
self, allow_model_requests, temp_db_path, test_docker_image
):
"""Test RLM agent can analyze document structure using DoclingDocument.
async def test_rlm_search_and_get_chunk(self, allow_model_requests, temp_db_path):
"""Test RLM agent can search and use get_chunk for citations.
Agent program:
docs = list_documents(limit=20)
print(docs)
doc = get_docling_document('<doc_id>')
print(doc.name)
print('tables:', len(doc.tables))
print('pictures:', len(doc.pictures))
results = search("content", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(chunk['document_title'], chunk['chunk_id'])
"""
from haiku.rag.client import HaikuRAG
pdf_path = Path("tests/data/doclaynet.pdf")
config = AppConfig()
config.processing.conversion_options.do_ocr = False
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document_from_source(pdf_path)
result = await client.rlm(
"How many tables are in the document? "
"Also tell me how many pictures/figures it contains."
await client.create_document(
"The quick brown fox jumps over the lazy dog.",
title="Animal Facts",
)
# The doclaynet.pdf has 1 table and 1 picture
assert "1" in result.answer
result = await client.rlm(
"Search for content about animals and tell me "
"which document it came from."
)
assert "Animal Facts" in result.answer
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_semantic_analysis_with_llm(
self, allow_model_requests, temp_db_path, test_docker_image
self, allow_model_requests, temp_db_path
):
"""Test RLM agent can use llm() for semantic analysis combined with computation.
@ -200,7 +188,7 @@ class TestClientRLMIntegration:
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"The new product launch exceeded expectations. Sales grew 40% "
@ -232,9 +220,7 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_search_and_extract(
self, allow_model_requests, temp_db_path, test_docker_image
):
async def test_rlm_search_and_extract(self, allow_model_requests, temp_db_path):
"""Test RLM agent can use search() to find content and extract information.
Agent program:
@ -252,7 +238,6 @@ class TestClientRLMIntegration:
pdf_path = Path("tests/data/doclaynet.pdf")
config = AppConfig()
config.processing.conversion_options.do_ocr = False
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document_from_source(pdf_path)
@ -293,7 +278,7 @@ class TestClientRLMIntegration:
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_rlm_with_preloaded_documents(
self, allow_model_requests, temp_db_path, test_docker_image
self, allow_model_requests, temp_db_path
):
"""Test RLM agent can use pre-loaded documents variable.
@ -307,7 +292,7 @@ class TestClientRLMIntegration:
from haiku.rag.client import HaikuRAG
config = AppConfig()
config.rlm.docker_image = test_docker_image
async with HaikuRAG(temp_db_path, config=config, create=True) as client:
await client.create_document(
"The company was founded in 1985 by Jane Smith.",

50
tests/agents/rlm/test_runner.py
View file

@ -1,50 +0,0 @@
import json
from io import StringIO
from haiku.rag.agents.rlm.runner import execute_code, send_response
def test_execute_code_success():
namespace: dict = {}
result = execute_code("x = 1 + 1", namespace, max_output_chars=1000)
assert result["success"] is True
assert result["stderr"] == ""
def test_execute_code_stdout_capture():
namespace: dict = {}
result = execute_code("print('hello')", namespace, max_output_chars=1000)
assert result["success"] is True
assert "hello" in result["stdout"]
def test_execute_code_exception():
namespace: dict = {}
result = execute_code("raise ValueError('boom')", namespace, max_output_chars=1000)
assert result["success"] is False
assert "ValueError" in result["stderr"]
assert "boom" in result["stderr"]
def test_execute_code_output_truncation():
namespace: dict = {}
code = "print('x' * 100)"
result = execute_code(code, namespace, max_output_chars=10)
assert result["success"] is True
assert "truncated" in result["stdout"]
assert len(result["stdout"]) < 100
def test_send_response(monkeypatch):
buf = StringIO()
monkeypatch.setattr("sys.stdout", buf)
payload = {"success": True, "stdout": "hi", "stderr": ""}
send_response(payload)
output = buf.getvalue()
lines = output.split("\n", 1)
length = int(lines[0])
body = lines[1]
assert json.loads(body) == payload
assert length == len(json.dumps(payload))

486
tests/agents/rlm/test_sandbox.py
View file

@ -1,12 +1,12 @@
import os
from pathlib import Path
import pytest
from haiku.rag.agents.rlm.dependencies import RLMContext
from haiku.rag.agents.rlm.docker_sandbox import DockerSandbox, SandboxResult
from haiku.rag.agents.rlm.sandbox import Sandbox, SandboxResult
from haiku.rag.client import HaikuRAG
from haiku.rag.config.models import RLMConfig
from haiku.rag.config.models import AppConfig
from haiku.rag.store.models import Document
@pytest.fixture(scope="module")
@ -14,103 +14,83 @@ def vcr_cassette_dir():
return str(Path(__file__).parent.parent.parent / "cassettes" / "test_sandbox")
def is_docker_available() -> bool:
"""Check if Docker daemon is available."""
try:
import subprocess
class TestSandboxBasics:
"""Test basic sandbox functionality."""
result = subprocess.run(["docker", "info"], capture_output=True, timeout=5)
return result.returncode == 0
except Exception:
return False
docker_required = pytest.mark.skipif(
not is_docker_available(),
reason="Docker daemon not available",
)
@pytest.mark.integration
class TestDockerSandboxBasics:
"""Test basic Docker sandbox functionality."""
@docker_required
@pytest.mark.asyncio
async def test_execute_simple_code(self, docker_sandbox):
async def test_execute_simple_code(self, sandbox):
"""Test executing simple code in the sandbox."""
result = await docker_sandbox.execute("print('hello world')")
result = await sandbox.execute("print('hello world')")
assert isinstance(result, SandboxResult)
assert result.success
assert "hello world" in result.stdout
assert result.stderr == ""
@pytest.mark.integration
class TestDockerSandboxErrors:
"""Test error handling in Docker sandbox."""
@docker_required
@pytest.mark.asyncio
async def test_syntax_error(self, docker_sandbox):
async def test_execute_expression_output(self, sandbox):
"""Test that expression values are captured."""
result = await sandbox.execute("1 + 2")
assert result.success
assert "3" in result.stdout
@pytest.mark.asyncio
async def test_execute_print_and_expression(self, sandbox):
"""Test print output combined with expression value."""
result = await sandbox.execute("print('hello')\n42")
assert result.success
assert "hello" in result.stdout
assert "42" in result.stdout
class TestSandboxErrors:
"""Test error handling in sandbox."""
@pytest.mark.asyncio
async def test_syntax_error(self, sandbox):
"""Test that syntax errors are reported."""
result = await docker_sandbox.execute("def foo(")
result = await sandbox.execute("def foo(")
assert not result.success
assert "SyntaxError" in result.stderr
assert result.stderr != ""
@docker_required
@pytest.mark.asyncio
async def test_runtime_error(self, docker_sandbox):
async def test_runtime_error(self, sandbox):
"""Test that runtime errors are reported."""
result = await docker_sandbox.execute("x = 1/0")
result = await sandbox.execute("x = 1/0")
assert not result.success
assert "ZeroDivisionError" in result.stderr
@docker_required
@pytest.mark.asyncio
async def test_name_error(self, docker_sandbox):
async def test_name_error(self, sandbox):
"""Test that name errors are reported."""
result = await docker_sandbox.execute("print(undefined_variable)")
result = await sandbox.execute("print(undefined_variable)")
assert not result.success
assert "NameError" in result.stderr
@docker_required
@pytest.mark.asyncio
async def test_missing_image(self, temp_db_path):
"""Test error when Docker image is not found."""
async with HaikuRAG(temp_db_path, create=True) as client:
config = RLMConfig(docker_image="nonexistent-image:v999.999.999")
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=config.docker_image
) as sandbox:
result = await sandbox.execute("print('hello')")
assert not result.success
assert (
"not found" in result.stderr.lower()
or "error" in result.stderr.lower()
)
async def test_multi_module_import(self, sandbox):
"""Test that unsupported multi-module imports are caught gracefully."""
result = await sandbox.execute("import json, string")
assert not result.success
assert result.stderr != ""
@pytest.mark.integration
class TestDockerSandboxHaikuRAG:
"""Test haiku.rag functions in Docker sandbox."""
class TestSandboxHaikuRAG:
"""Test haiku.rag functions in sandbox."""
@docker_required
@pytest.mark.asyncio
async def test_list_documents_empty(self, docker_sandbox):
async def test_list_documents_empty(self, sandbox):
"""Test list_documents returns empty list for empty database."""
result = await docker_sandbox.execute(
"docs = list_documents()\nprint(type(docs).__name__, len(docs))"
result = await sandbox.execute(
"docs = await list_documents()\nprint(type(docs).__name__, len(docs))"
)
assert result.success
assert "list 0" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_list_documents_with_data(self, temp_db_path, test_docker_image):
async def test_list_documents_with_data(self, temp_db_path):
"""Test list_documents returns documents when populated."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="Test content",
@ -118,27 +98,22 @@ class TestDockerSandboxHaikuRAG:
title="Test Document",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"docs = list_documents()\nprint(len(docs))\nprint(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Test Document" in result.stdout
sb = Sandbox(client=client, config=config, context=context)
result = await sb.execute(
"docs = await list_documents()\n"
"print(len(docs))\n"
"print(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Test Document" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
@pytest.mark.skipif(
os.environ.get("CI") == "true",
reason="Requires Ollama - VCR can't capture calls from inside Docker",
)
async def test_search_with_data(self, temp_db_path, test_docker_image):
async def test_search_with_data(self, temp_db_path):
"""Test search function works."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="The quick brown fox jumps over the lazy dog.",
@ -146,26 +121,22 @@ class TestDockerSandboxHaikuRAG:
title="Animals",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"results = search('fox', limit=5)\n"
"print(len(results))\n"
"if results:\n"
" print('fox' in results[0]['content'].lower())"
)
assert result.success
# Search should return at least one result
assert "True" in result.stdout or "1" in result.stdout
sb = Sandbox(client=client, config=config, context=context)
result = await sb.execute(
"results = await search('fox', limit=5)\n"
"print(len(results))\n"
"if results:\n"
" print('fox' in results[0]['content'].lower())"
)
assert result.success
assert "True" in result.stdout or "1" in result.stdout
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_get_document(self, temp_db_path, test_docker_image):
async def test_get_document(self, temp_db_path):
"""Test get_document function."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
doc = await client.create_document(
content="Content about foxes and dogs.",
@ -173,40 +144,149 @@ class TestDockerSandboxHaikuRAG:
title="Fox Document",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext()
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
f"content = get_document('{doc.id}')\n"
"print('foxes' in content.lower() if content else 'None')"
)
assert result.success
assert "True" in result.stdout
sb = Sandbox(client=client, config=config, context=context)
result = await sb.execute(
f"content = await get_document('{doc.id}')\n"
"print('foxes' in content.lower() if content else 'None')"
)
assert result.success
assert "True" in result.stdout
@docker_required
@pytest.mark.asyncio
async def test_get_document_not_found(self, docker_sandbox):
async def test_get_document_not_found(self, sandbox):
"""Test get_document returns None for missing document."""
result = await docker_sandbox.execute(
"content = get_document('nonexistent-id')\nprint(content is None)"
result = await sandbox.execute(
"content = await get_document('nonexistent-id')\nprint(content is None)"
)
assert result.success
assert "True" in result.stdout
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_get_chunk(self, temp_db_path):
"""Test get_chunk function returns chunk with metadata."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="Content about foxes and dogs.",
uri="test://doc",
title="Fox Document",
)
context = RLMContext()
sb = Sandbox(client=client, config=config, context=context)
# First search to get a chunk_id
result = await sb.execute(
"results = await search('foxes', limit=1)\n"
"chunk_id = results[0]['chunk_id']\n"
"chunk = await get_chunk(chunk_id)\n"
"print(chunk['document_title'])\n"
"print('content' in chunk)"
)
assert result.success
assert "Fox Document" in result.stdout
assert "True" in result.stdout
@pytest.mark.asyncio
async def test_get_chunk_not_found(self, sandbox):
"""Test get_chunk returns None for missing chunk."""
result = await sandbox.execute(
"chunk = await get_chunk('nonexistent-id')\nprint(chunk is None)"
)
assert result.success
assert "True" in result.stdout
@pytest.mark.integration
class TestDockerSandboxContextFilter:
class TestSandboxExternalFunctionEdgeCases:
"""Test edge cases in external function dispatch."""
@pytest.mark.asyncio
async def test_unknown_external_function(self, sandbox):
"""Test that calling an unregistered external function resumes with KeyError."""
original_build = sandbox._build_external_functions
def patched_build():
fns = original_build()
fns["search"] = None
return fns
sandbox._build_external_functions = patched_build
result = await sandbox.execute(
"try:\n"
" await search('hello')\n"
"except:\n"
" print('caught')\n"
"print('done')"
)
assert result.success
assert "caught" in result.stdout
assert "done" in result.stdout
@pytest.mark.asyncio
async def test_external_function_raises_exception(self, sandbox):
"""Test that exceptions from async external functions surface as errors.
With run_monty_async, exceptions from async external functions
propagate as MontyRuntimeError rather than being catchable inside
Monty's try/except.
"""
original_build = sandbox._build_external_functions
def patched_build():
fns = original_build()
async def failing_search(*args, **kwargs):
raise ValueError("external error")
fns["search"] = failing_search
return fns
sandbox._build_external_functions = patched_build
result = await sandbox.execute("await search('hello')")
assert not result.success
assert "external error" in result.stderr
class TestSandboxOutputTruncation:
"""Test output truncation behavior."""
@pytest.mark.asyncio
async def test_truncate_stdout_on_runtime_error(self, empty_client):
"""Test stdout is truncated when a runtime error occurs after large output."""
config = AppConfig()
config.rlm.max_output_chars = 20
context = RLMContext()
sb = Sandbox(client=empty_client, config=config, context=context)
result = await sb.execute("print('a' * 100)\nx = 1/0")
assert not result.success
assert "ZeroDivisionError" in result.stderr
assert result.stdout.endswith("... (output truncated)")
assert len(result.stdout) < 100
@pytest.mark.asyncio
async def test_truncate_successful_output(self, empty_client):
"""Test output is truncated on successful execution with large output."""
config = AppConfig()
config.rlm.max_output_chars = 20
context = RLMContext()
sb = Sandbox(client=empty_client, config=config, context=context)
result = await sb.execute("print('b' * 100)")
assert result.success
assert result.stdout.endswith("... (output truncated)")
assert len(result.stdout) < 100
class TestSandboxContextFilter:
"""Test context filter is applied."""
@docker_required
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_filter_applied_to_list_documents(
self, temp_db_path, test_docker_image
):
async def test_filter_applied_to_list_documents(self, temp_db_path):
"""Test that context filter is passed to list_documents."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
await client.create_document(
content="Public content",
@ -219,33 +299,165 @@ class TestDockerSandboxContextFilter:
title="Private Doc",
)
config = RLMConfig(docker_image=test_docker_image)
context = RLMContext(filter="uri LIKE 'public://%'")
async with DockerSandbox(
client=client, config=config, context=context, image=test_docker_image
) as sandbox:
result = await sandbox.execute(
"docs = list_documents()\n"
"print(len(docs))\n"
"if docs:\n"
" print(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Public Doc" in result.stdout
assert "Private Doc" not in result.stdout
sb = Sandbox(client=client, config=config, context=context)
result = await sb.execute(
"docs = await list_documents()\n"
"print(len(docs))\n"
"if docs:\n"
" print(docs[0]['title'])"
)
assert result.success
assert "1" in result.stdout
assert "Public Doc" in result.stdout
assert "Private Doc" not in result.stdout
@pytest.mark.integration
class TestDockerSandboxPreloadedDocuments:
class TestSandboxPreloadedDocuments:
"""Test pre-loaded documents context variable."""
@docker_required
@pytest.mark.asyncio
async def test_documents_variable_not_available_without_preload(
self, docker_sandbox
):
async def test_documents_variable_not_available_without_preload(self, sandbox):
"""documents variable is not available when context.documents is None."""
result = await docker_sandbox.execute("print(documents)")
result = await sandbox.execute("print(documents)")
assert not result.success
assert "NameError" in result.stderr
@pytest.mark.asyncio
async def test_documents_variable_available_with_preload(self, empty_client):
"""documents variable is available when context.documents is set."""
config = AppConfig()
docs = [
Document(id="1", content="Content A", title="Doc A", uri="a://1"),
Document(id="2", content="Content B", title="Doc B", uri="b://2"),
]
context = RLMContext(documents=docs)
sb = Sandbox(client=empty_client, config=config, context=context)
result = await sb.execute(
"print(len(documents))\n"
"print(documents[0]['title'])\n"
"print(documents[1]['title'])"
)
assert result.success
assert "2" in result.stdout
assert "Doc A" in result.stdout
assert "Doc B" in result.stdout
class TestSandboxDoclingDocument:
"""Test get_docling_document() external function."""
@pytest.mark.asyncio
async def test_returns_none_for_missing_document(self, sandbox):
"""get_docling_document returns None for a non-existent document."""
result = await sandbox.execute(
"doc = await get_docling_document('nonexistent-id')\nprint(doc is None)"
)
assert result.success
assert "True" in result.stdout
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_returns_dict_for_document_with_docling_data(self, temp_db_path):
"""get_docling_document returns a dict for a document with docling data."""
config = AppConfig()
async with HaikuRAG(temp_db_path, create=True) as client:
doc = await client.create_document(
content="Docling processed content",
uri="test://docling",
title="Docling Doc",
)
context = RLMContext()
sb = Sandbox(client=client, config=config, context=context)
result = await sb.execute(
f"doc = await get_docling_document('{doc.id}')\n"
"print(type(doc).__name__)\n"
"print(doc['name'])\n"
"print('texts' in doc)"
)
assert result.success
assert "dict" in result.stdout
assert "True" in result.stdout
class TestSandboxRegex:
"""Test regex external functions."""
@pytest.mark.asyncio
async def test_regex_findall(self, sandbox):
"""regex_findall extracts all matches."""
result = await sandbox.execute(
r"matches = await regex_findall(r'\d+', 'abc 123 def 456')"
"\nprint(matches)"
)
assert result.success
assert "['123', '456']" in result.stdout
@pytest.mark.asyncio
async def test_regex_sub(self, sandbox):
"""regex_sub replaces matches."""
result = await sandbox.execute(
r"out = await regex_sub(r'\d+', 'X', 'abc 123 def 456')"
"\nprint(out)"
)
assert result.success
assert "abc X def X" in result.stdout
@pytest.mark.asyncio
async def test_regex_search_match(self, sandbox):
"""regex_search returns match dict when pattern matches."""
result = await sandbox.execute(
r"m = await regex_search(r'(\d+)', 'abc 123')"
"\nprint(m['group'])"
"\nprint(m['start'])"
"\nprint(m['end'])"
)
assert result.success
assert "123" in result.stdout
assert "4" in result.stdout
assert "7" in result.stdout
@pytest.mark.asyncio
async def test_regex_search_no_match(self, sandbox):
"""regex_search returns None when pattern doesn't match."""
result = await sandbox.execute(
r"m = await regex_search(r'\d+', 'abc')"
"\nprint(m is None)"
)
assert result.success
assert "True" in result.stdout
@pytest.mark.asyncio
async def test_regex_split(self, sandbox):
"""regex_split splits on pattern."""
result = await sandbox.execute(
"out = await regex_split(',', 'a,b,,c')\nprint(out)"
)
assert result.success
assert "['a', 'b', '', 'c']" in result.stdout
@pytest.mark.asyncio
async def test_regex_invalid_pattern(self, sandbox):
"""Invalid regex pattern surfaces as an error."""
result = await sandbox.execute("await regex_findall('[invalid', 'text')")
assert not result.success
assert result.stderr != ""
class TestSandboxLLM:
"""Test llm() external function."""
@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_llm_function(self, allow_model_requests, empty_client):
"""Test llm() calls the model and returns a string."""
config = AppConfig()
context = RLMContext()
sb = Sandbox(client=empty_client, config=config, context=context)
result = await sb.execute(
"answer = await llm('What is 2 + 2? Reply with just the number.')\n"
"print(answer)"
)
assert result.success
assert "4" in result.stdout

2300
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_aggregation.yaml
View file

File diff suppressed because one or more lines are too long

301
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_count_documents.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7774'
- '7325'
content-type:
- application/json
host:
@ -141,32 +141,33 @@ interactions:
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") ✓ CORRECT
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just call them with `await`:
- results = await search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
- results = search("query") ✗ WRONG - must use await
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly with `await`, no imports needed):
## Available Functions
### search(query, limit=10) -> list[dict]
### await search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
### await list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
### await get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### await get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
@ -181,93 +182,64 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
2. **If get_document returns None**: Use `await list_documents()` to see actual document titles, or `await search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The REPL captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use `await llm()`.
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
docs = await list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
content = await get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
results = await search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = await llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = await search("safety requirements", limit=5)
for r in results:
chunk = await get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
content = await get_document("Q1 Report")
sentiment = await llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -300,11 +272,10 @@ interactions:
tools:
- function:
description: |-
<summary>Execute Python code in a Docker-sandboxed environment.
<summary>Execute Python code in a sandboxed interpreter.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
get_document, get_chunk, llm).
Use print() to output results.</summary>
<returns>
@ -344,7 +315,7 @@ interactions:
response:
headers:
content-length:
- '523'
- '514'
content-type:
- application/json
parsed_body:
@ -353,24 +324,24 @@ interactions:
index: 0
message:
content: ''
reasoning: We need to list documents.
reasoning: Need to list docs.
role: assistant
tool_calls:
- function:
arguments: '{"code":"# list documents\nimport json\nprint(list_documents())\n"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
name: execute_code
id: call_d8xhmimu
id: call_stp0fimx
index: 0
type: function
created: 1770373335
id: chatcmpl-184
created: 1771924497
id: chatcmpl-750
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 42
prompt_tokens: 1747
total_tokens: 1789
completion_tokens: 44
prompt_tokens: 1623
total_tokens: 1667
status:
code: 200
message: OK
@ -383,7 +354,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8588'
- '7759'
content-type:
- application/json
host:
@ -396,32 +367,33 @@ interactions:
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") ✓ CORRECT
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just call them with `await`:
- results = await search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
- results = search("query") ✗ WRONG - must use await
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly with `await`, no imports needed):
## Available Functions
### search(query, limit=10) -> list[dict]
### await search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
### await list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
### await get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### await get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
@ -436,93 +408,64 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
2. **If get_document returns None**: Use `await list_documents()` to see actual document titles, or `await search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The REPL captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use `await llm()`.
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
docs = await list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
content = await get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
results = await search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = await llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = await search("safety requirements", limit=5)
for r in results:
chunk = await get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
content = await get_document("Q1 Report")
sentiment = await llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -548,23 +491,18 @@ interactions:
role: system
- content: How many documents are in the database?
role: user
- content: |-
<think>
We need to list documents.
</think>
- content: null
reasoning: Need to list docs.
role: assistant
tool_calls:
- function:
arguments: '{"code":"# list documents\nimport json\nprint(list_documents())\n"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
name: execute_code
id: call_d8xhmimu
id: call_stp0fimx
type: function
- content: '{"code":"# list documents\nimport json\nprint(list_documents())\n","stdout":"[{''id'': ''b73f8a17-4328-475c-84db-3d81ce52adce'',
''title'': ''Doc 1'', ''uri'': None, ''created_at'': ''2026-02-06 11:21:55.843558''}, {''id'': ''accb877b-f04e-4bf2-ba4c-2d90339fa875'',
''title'': ''Doc 2'', ''uri'': None, ''created_at'': ''2026-02-06 11:21:57.397026''}, {''id'': ''afdb966f-5e9d-4759-a08f-28eb5108c80f'',
''title'': ''Doc 3'', ''uri'': None, ''created_at'': ''2026-02-06 11:21:58.988378''}]\n","stderr":"","success":true}'
- content: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))","stdout":"3\n","stderr":"","success":true}'
role: tool
tool_call_id: call_d8xhmimu
tool_call_id: call_stp0fimx
model: gpt-oss
reasoning_effort: low
stream: false
@ -572,11 +510,10 @@ interactions:
tools:
- function:
description: |-
<summary>Execute Python code in a Docker-sandboxed environment.
<summary>Execute Python code in a sandboxed interpreter.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
get_document, get_chunk, llm).
Use print() to output results.</summary>
<returns>
@ -616,7 +553,7 @@ interactions:
response:
headers:
content-length:
- '523'
- '416'
content-type:
- application/json
parsed_body:
@ -624,19 +561,17 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There are 3 documents in the database.","program":"# List and count documents\nimport json\n\ndocs
= list_documents()\nprint(f\"Number of documents: {len(docs)}\")\n"}'
reasoning: Count is 3. Provide answer.
content: '{"answer":"There are 3 documents in the database.","program":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
role: assistant
created: 1770373336
id: chatcmpl-441
created: 1771924498
id: chatcmpl-945
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 68
prompt_tokens: 2019
total_tokens: 2087
completion_tokens: 38
prompt_tokens: 1709
total_tokens: 1747
status:
code: 200
message: OK

988
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_docling_document_structure.yaml
View file

File diff suppressed because one or more lines are too long

2908
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_extract.yaml
View file

File diff suppressed because one or more lines are too long

545
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_get_chunk.yaml Normal file
View file

File diff suppressed because one or more lines are too long

1738
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_semantic_analysis_with_llm.yaml
View file

File diff suppressed because one or more lines are too long

300
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_filter.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7768'
- '7319'
content-type:
- application/json
host:
@ -141,32 +141,33 @@ interactions:
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") ✓ CORRECT
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just call them with `await`:
- results = await search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
- results = search("query") ✗ WRONG - must use await
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly with `await`, no imports needed):
## Available Functions
### search(query, limit=10) -> list[dict]
### await search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
### await list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
### await get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### await get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
@ -181,93 +182,64 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
2. **If get_document returns None**: Use `await list_documents()` to see actual document titles, or `await search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The REPL captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use `await llm()`.
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
docs = await list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
content = await get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
results = await search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = await llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = await search("safety requirements", limit=5)
for r in results:
chunk = await get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
content = await get_document("Q1 Report")
sentiment = await llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -300,11 +272,10 @@ interactions:
tools:
- function:
description: |-
<summary>Execute Python code in a Docker-sandboxed environment.
<summary>Execute Python code in a sandboxed interpreter.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
get_document, get_chunk, llm).
Use print() to output results.</summary>
<returns>
@ -344,7 +315,7 @@ interactions:
response:
headers:
content-length:
- '517'
- '547'
content-type:
- application/json
parsed_body:
@ -353,24 +324,24 @@ interactions:
index: 0
message:
content: ''
reasoning: Need to get list_documents.
reasoning: Need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs = await list_documents(limit=1000, offset=0)\nprint(len(docs))\nprint(docs[:3])"}'
name: execute_code
id: call_ly3bn3y0
id: call_pu34e9fx
index: 0
type: function
created: 1770373376
id: chatcmpl-196
created: 1771924517
id: chatcmpl-236
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 44
prompt_tokens: 1745
total_tokens: 1789
completion_tokens: 56
prompt_tokens: 1621
total_tokens: 1677
status:
code: 200
message: OK
@ -383,7 +354,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8201'
- '7939'
content-type:
- application/json
host:
@ -396,32 +367,33 @@ interactions:
IMPORTANT: You MUST use the `execute_code` tool to run Python code. The functions described below are ONLY available inside the execute_code tool - you cannot access them any other way. Always execute code to answer questions; do not just describe what code would do.
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just use them directly:
- search("query") ✓ CORRECT
CRITICAL: Inside execute_code, these functions are ALREADY available in the namespace. Do NOT import them - just call them with `await`:
- results = await search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
- results = search("query") ✗ WRONG - must use await
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly with `await`, no imports needed):
## Available Functions
### search(query, limit=10) -> list[dict]
### await search(query, limit=10) -> list[dict]
Search the knowledge base using hybrid search (vector + full-text).
Returns list of dicts with keys: chunk_id, content, document_id, document_title, document_uri, score, page_numbers, headings
### list_documents(limit=10, offset=0) -> list[dict]
### await list_documents(limit=10, offset=0) -> list[dict]
List available documents in the knowledge base.
Returns list of dicts with keys: id, title, uri, created_at
### get_document(id_or_title) -> str | None
### await get_document(id_or_title) -> str | None
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### await get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
already have the content and just need LLM reasoning.
@ -436,93 +408,64 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
1. **Explore First**: Start by listing documents or searching to understand what's available. Document names may differ from filenames (e.g., "tbmed593.pdf" might be stored as "TB MED 593" or similar).
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
2. **If get_document returns None**: Use `await list_documents()` to see actual document titles, or `await search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The REPL captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use `await llm()`.
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
### Counting documents matching a condition
```python
docs = list_documents(limit=100)
docs = await list_documents(limit=100)
count = 0
for doc in docs:
content = get_document(doc['id'])
content = await get_document(doc['id'])
if content and 'keyword' in content.lower():
count += 1
print(f"Found in: {doc['title']}")
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
results = await search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = await llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = await search("safety requirements", limit=5)
for r in results:
chunk = await get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
content = await get_document("Q1 Report")
sentiment = await llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -548,20 +491,19 @@ interactions:
role: system
- content: How many documents are available?
role: user
- content: |-
<think>
Need to get list_documents.
</think>
- content: null
reasoning: Need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs = await list_documents(limit=1000, offset=0)\nprint(len(docs))\nprint(docs[:3])"}'
name: execute_code
id: call_ly3bn3y0
id: call_pu34e9fx
type: function
- content: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))","stdout":"1\n","stderr":"","success":true}'
- content: '{"code":"docs = await list_documents(limit=1000, offset=0)\nprint(len(docs))\nprint(docs[:3])","stdout":"1\n[{''id'':
''90fce4e5-0d68-4736-acaf-d075e3ed4822'', ''title'': ''Cats'', ''uri'': None, ''created_at'': ''2026-02-24 11:15:14.007278''}]\n","stderr":"","success":true}'
role: tool
tool_call_id: call_ly3bn3y0
tool_call_id: call_pu34e9fx
model: gpt-oss
reasoning_effort: low
stream: false
@ -569,11 +511,10 @@ interactions:
tools:
- function:
description: |-
<summary>Execute Python code in a Docker-sandboxed environment.
<summary>Execute Python code in a sandboxed interpreter.
The code has access to haiku.rag functions (search, list_documents,
get_document, get_docling_document, llm) and any Python standard
library module.
get_document, get_chunk, llm).
Use print() to output results.</summary>
<returns>
@ -613,7 +554,7 @@ interactions:
response:
headers:
content-length:
- '424'
- '704'
content-type:
- application/json
parsed_body:
@ -621,17 +562,20 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There is 1 document available in the knowledge base.","program":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
content: '{"answer":"There is 1 document available in the knowledge base.","program":"# List all documents\nimport
json\nasync def main():\n docs = await list_documents(limit=1000, offset=0)\n print(\"Number of documents:\",
len(docs))\n # Optional: print first few document titles for reference\n for d in docs[:5]:\n print(f\"ID:
{d[''id'']}, Title: {d[''title'']}\")\n\nawait main()"}'
role: assistant
created: 1770373377
id: chatcmpl-195
created: 1771924519
id: chatcmpl-487
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 39
prompt_tokens: 1836
total_tokens: 1875
completion_tokens: 117
prompt_tokens: 1793
total_tokens: 1910
status:
code: 200
message: OK

2035
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_preloaded_documents.yaml
View file

File diff suppressed because one or more lines are too long

0
tests/cassettes/test_sandbox/TestDockerSandboxContextFilter.test_filter_applied_to_list_documents.yaml → tests/cassettes/test_sandbox/TestSandboxContextFilter.test_filter_applied_to_list_documents.yaml
View file

42
tests/cassettes/test_sandbox/TestSandboxDoclingDocument.test_returns_dict_for_document_with_docling_data.yaml Normal file
View file

File diff suppressed because one or more lines are too long

82
tests/cassettes/test_sandbox/TestSandboxHaikuRAG.test_get_chunk.yaml Normal file
View file

File diff suppressed because one or more lines are too long

0
tests/cassettes/test_sandbox/TestDockerSandboxHaikuRAG.test_get_document.yaml → tests/cassettes/test_sandbox/TestSandboxHaikuRAG.test_get_document.yaml
View file

0
tests/cassettes/test_sandbox/TestDockerSandboxHaikuRAG.test_list_documents_with_data.yaml → tests/cassettes/test_sandbox/TestSandboxHaikuRAG.test_list_documents_with_data.yaml
View file

40
tests/cassettes/test_sandbox/TestDockerSandboxHaikuRAG.test_search_with_data.yaml → tests/cassettes/test_sandbox/TestSandboxHaikuRAG.test_search_with_data.yaml
View file

File diff suppressed because one or more lines are too long

51
tests/cassettes/test_sandbox/TestSandboxLLM.test_llm_function.yaml Normal file
View file

@ -0,0 +1,51 @@
interactions:
- request:
headers:
accept:
- application/json
accept-encoding:
- gzip, deflate, zstd
connection:
- keep-alive
content-length:
- '143'
content-type:
- application/json
host:
- localhost:11434
method: POST
parsed_body:
messages:
- content: What is 2 + 2? Reply with just the number.
role: user
model: gpt-oss
reasoning_effort: low
stream: false
uri: http://localhost:11434/v1/chat/completions
response:
headers:
content-length:
- '311'
content-type:
- application/json
parsed_body:
choices:
- finish_reason: stop
index: 0
message:
content: '4'
reasoning: Just reply 4.
role: assistant
created: 1771924616
id: chatcmpl-525
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 16
prompt_tokens: 81
total_tokens: 97
status:
code: 200
message: OK
version: 1

156
uv.lock
View file

@ -180,7 +180,7 @@ wheels = [
[[package]]
name = "anthropic"
version = "0.79.0"
version = "0.83.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@ -192,9 +192,9 @@ dependencies = [
{ name = "sniffio" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/15/b1/91aea3f8fd180d01d133d931a167a78a3737b3fd39ccef2ae8d6619c24fd/anthropic-0.79.0.tar.gz", hash = "sha256:8707aafb3b1176ed6c13e2b1c9fb3efddce90d17aee5d8b83a86c70dcdcca871", size = 509825, upload-time = "2026-02-07T18:06:18.388Z" }
sdist = { url = "https://files.pythonhosted.org/packages/db/e5/02cd2919ec327b24234abb73082e6ab84c451182cc3cc60681af700f4c63/anthropic-0.83.0.tar.gz", hash = "sha256:a8732c68b41869266c3034541a31a29d8be0f8cd0a714f9edce3128b351eceb4", size = 534058, upload-time = "2026-02-19T19:26:38.904Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/95/b2/cc0b8e874a18d7da50b0fda8c99e4ac123f23bf47b471827c5f6f3e4a767/anthropic-0.79.0-py3-none-any.whl", hash = "sha256:04cbd473b6bbda4ca2e41dd670fe2f829a911530f01697d0a1e37321eb75f3cf", size = 405918, upload-time = "2026-02-07T18:06:20.246Z" },
{ url = "https://files.pythonhosted.org/packages/5f/75/b9d58e4e2a4b1fc3e75ffbab978f999baf8b7c4ba9f96e60edb918ba386b/anthropic-0.83.0-py3-none-any.whl", hash = "sha256:f069ef508c73b8f9152e8850830d92bd5ef185645dbacf234bb213344a274810", size = 456991, upload-time = "2026-02-19T19:26:40.114Z" },
]
[[package]]
@ -1092,7 +1092,7 @@ name = "ffmpeg-python"
version = "0.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "future" },
{ name = "future", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/dd/5e/d5f9105d59c1325759d838af4e973695081fbbc97182baf73afc78dec266/ffmpeg-python-0.2.0.tar.gz", hash = "sha256:65225db34627c578ef0e11c8b1eb528bb35e024752f6f10b78c011f6f64c4127", size = 21543, upload-time = "2019-07-06T00:19:08.989Z" }
wheels = [
@ -1306,30 +1306,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c4/ab/09169d5a4612a5f92490806649ac8d41e3ec9129c636754575b3553f4ea4/googleapis_common_protos-1.72.0-py3-none-any.whl", hash = "sha256:4299c5a82d5ae1a9702ada957347726b167f9f8d1fc352477702a1e851ff4038", size = 297515, upload-time = "2025-11-06T18:29:13.14Z" },
]
[[package]]
name = "griffe"
version = "2.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "griffecli" },
{ name = "griffelib" },
]
wheels = [
{ url = "https://files.pythonhosted.org/packages/8b/94/ee21d41e7eb4f823b94603b9d40f86d3c7fde80eacc2c3c71845476dddaa/griffe-2.0.0-py3-none-any.whl", hash = "sha256:5418081135a391c3e6e757a7f3f156f1a1a746cc7b4023868ff7d5e2f9a980aa", size = 5214, upload-time = "2026-02-09T19:09:44.105Z" },
]
[[package]]
name = "griffecli"
version = "2.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama" },
{ name = "griffelib" },
]
wheels = [
{ url = "https://files.pythonhosted.org/packages/e6/ed/d93f7a447bbf7a935d8868e9617cbe1cadf9ee9ee6bd275d3040fbf93d60/griffecli-2.0.0-py3-none-any.whl", hash = "sha256:9f7cd9ee9b21d55e91689358978d2385ae65c22f307a63fb3269acf3f21e643d", size = 9345, upload-time = "2026-02-09T19:09:42.554Z" },
]
[[package]]
name = "griffelib"
version = "2.0.0"
@ -1456,6 +1432,7 @@ dependencies = [
{ name = "pathspec" },
{ name = "pydantic" },
{ name = "pydantic-ai-slim", extra = ["ag-ui", "fastmcp", "logfire", "openai"] },
{ name = "pydantic-monty" },
{ name = "python-dotenv" },
{ name = "pyyaml" },
{ name = "rich" },
@ -1529,6 +1506,7 @@ requires-dist = [
{ name = "pydantic-ai-slim", extras = ["openai", "fastmcp", "logfire", "ag-ui"], specifier = ">=1.46.0" },
{ name = "pydantic-ai-slim", extras = ["vertexai"], marker = "extra == 'vertexai'" },
{ name = "pydantic-ai-slim", extras = ["voyageai"], marker = "extra == 'voyageai'" },
{ name = "pydantic-monty", specifier = ">=0.0.7" },
{ name = "python-dotenv", specifier = ">=1.2.1" },
{ name = "pyyaml", specifier = ">=6.0.3" },
{ name = "rich", specifier = ">=14.2.0" },
@ -1976,14 +1954,14 @@ name = "langchain-core"
version = "1.2.13"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "jsonpatch" },
{ name = "langsmith" },
{ name = "packaging" },
{ name = "pydantic" },
{ name = "pyyaml" },
{ name = "tenacity" },
{ name = "typing-extensions" },
{ name = "uuid-utils" },
{ name = "jsonpatch", marker = "python_full_version < '3.14'" },
{ name = "langsmith", marker = "python_full_version < '3.14'" },
{ name = "packaging", marker = "python_full_version < '3.14'" },
{ name = "pydantic", marker = "python_full_version < '3.14'" },
{ name = "pyyaml", marker = "python_full_version < '3.14'" },
{ name = "tenacity", marker = "python_full_version < '3.14'" },
{ name = "typing-extensions", marker = "python_full_version < '3.14'" },
{ name = "uuid-utils", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fb/bb/c501ca60556c11ac80d1454bdcac63cb33583ce4e64fc4535ad5a7d5c6ba/langchain_core-1.2.13.tar.gz", hash = "sha256:d2773d0d0130a356378db9a858cfeef64c3d64bc03722f1d4d6c40eb46fdf01b", size = 831612, upload-time = "2026-02-15T07:45:57.014Z" }
wheels = [
@ -1995,7 +1973,7 @@ name = "langchain-text-splitters"
version = "1.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "langchain-core" },
{ name = "langchain-core", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/41/42/c178dcdc157b473330eb7cc30883ea69b8ec60078c7b85e2d521054c4831/langchain_text_splitters-1.1.0.tar.gz", hash = "sha256:75e58acb7585dc9508f3cd9d9809cb14751283226c2d6e21fb3a9ae57582ca22", size = 272230, upload-time = "2025-12-14T01:15:38.659Z" }
wheels = [
@ -2007,15 +1985,15 @@ name = "langsmith"
version = "0.7.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "httpx" },
{ name = "orjson", marker = "platform_python_implementation != 'PyPy'" },
{ name = "packaging" },
{ name = "pydantic" },
{ name = "requests" },
{ name = "requests-toolbelt" },
{ name = "uuid-utils" },
{ name = "xxhash" },
{ name = "zstandard" },
{ name = "httpx", marker = "python_full_version < '3.14'" },
{ name = "orjson", marker = "python_full_version < '3.14' and platform_python_implementation != 'PyPy'" },
{ name = "packaging", marker = "python_full_version < '3.14'" },
{ name = "pydantic", marker = "python_full_version < '3.14'" },
{ name = "requests", marker = "python_full_version < '3.14'" },
{ name = "requests-toolbelt", marker = "python_full_version < '3.14'" },
{ name = "uuid-utils", marker = "python_full_version < '3.14'" },
{ name = "xxhash", marker = "python_full_version < '3.14'" },
{ name = "zstandard", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/8d/bc/8172fefad4f2da888a6d564a27d1fb7d4dbf3c640899c2b40c46235cbe98/langsmith-0.7.3.tar.gz", hash = "sha256:0223b97021af62d2cf53c8a378a27bd22e90a7327e45b353e0069ae60d5d6f9e", size = 988575, upload-time = "2026-02-13T23:25:32.916Z" }
wheels = [
@ -3624,20 +3602,20 @@ email = [
[[package]]
name = "pydantic-ai-slim"
version = "1.60.0"
version = "1.63.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "genai-prices" },
{ name = "griffe" },
{ name = "griffelib" },
{ name = "httpx" },
{ name = "opentelemetry-api" },
{ name = "pydantic" },
{ name = "pydantic-graph" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/16/97/f73f439f3d415d43f38250f76852121188d0ef6114ec702e84e7d69c301d/pydantic_ai_slim-1.60.0.tar.gz", hash = "sha256:12ba3e6ef933fcb9fc6a307dbdaa43ca15bbc1b8ec77521afd1b7a526d12330f", size = 418839, upload-time = "2026-02-17T00:33:29.672Z" }
sdist = { url = "https://files.pythonhosted.org/packages/da/6d/2b5c0c60b42e6af49830f6a09b5d38fecdb1f20d9659152691eba95613b4/pydantic_ai_slim-1.63.0.tar.gz", hash = "sha256:9377afecdfe4bc17f5c9ed72c758e460703ac5876931aa2f18ace8ac0e69312a", size = 426862, upload-time = "2026-02-23T17:56:36.215Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e3/40/8cb494a4d2ba62b5f92ae8bc79a2abbbf8509cb692edd4bc841695187780/pydantic_ai_slim-1.60.0-py3-none-any.whl", hash = "sha256:6865188a225a2979c82bb022a299d438d805d258c6d3f9810f7fe4e3c86af80a", size = 546410, upload-time = "2026-02-17T00:33:21.901Z" },
{ url = "https://files.pythonhosted.org/packages/f2/ca/c4e39eec1cff5a294b64313a8a959b38d326819e0f0a41f48e61ce019a22/pydantic_ai_slim-1.63.0-py3-none-any.whl", hash = "sha256:ed393b0f871b748171f65bec5191c3025b5abb8a4fc616afee17eb9dc2dfa15d", size = 554190, upload-time = "2026-02-23T17:56:29.533Z" },
]
[package.optional-dependencies]
@ -3681,7 +3659,7 @@ vertexai = [
{ name = "requests" },
]
voyageai = [
{ name = "voyageai" },
{ name = "voyageai", marker = "python_full_version < '3.14'" },
]
[[package]]
@ -3757,7 +3735,7 @@ wheels = [
[[package]]
name = "pydantic-evals"
version = "1.60.0"
version = "1.63.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@ -3767,14 +3745,14 @@ dependencies = [
{ name = "pyyaml" },
{ name = "rich" },
]
sdist = { url = "https://files.pythonhosted.org/packages/ec/2c/bed606a726b09adc9ee414bdb919ffe499edf7f8c631ba03b0ff3aa34435/pydantic_evals-1.60.0.tar.gz", hash = "sha256:ae3edd6667075acd8ef04c0d6fffb1ebe72c37ff077295fdbd6319e59284580b", size = 54214, upload-time = "2026-02-17T00:33:31.697Z" }
sdist = { url = "https://files.pythonhosted.org/packages/99/43/21b6ddf65b56f7401c344f98e4e6258a02d2868c8a52a8b79c0e0e701029/pydantic_evals-1.63.0.tar.gz", hash = "sha256:eed56a7192e07c8be8cf16e53bb2ef652b4f7f7b8527650ac45fde865a4ecf9d", size = 56365, upload-time = "2026-02-23T17:56:37.71Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/fe/a2/60790b2c971f6ce78fea2db2b65800489982c341ad9aa6db750070a93dcd/pydantic_evals-1.60.0-py3-none-any.whl", hash = "sha256:7a7414535002cae63ba0d0d9b15c6e72c28252cf120290b18372c9852c91fcfe", size = 65278, upload-time = "2026-02-17T00:33:23.411Z" },
{ url = "https://files.pythonhosted.org/packages/9b/f2/7174ad6abca2457e35a1b902ca4fa78aa8ee72e4ec2e9cd5dc8904014ec9/pydantic_evals-1.63.0-py3-none-any.whl", hash = "sha256:2e92a3af579a5670b2babf2044081d0ef99ab5a9ef141972616d71fd7e5bfd0e", size = 67279, upload-time = "2026-02-23T17:56:31.008Z" },
]
[[package]]
name = "pydantic-graph"
version = "1.60.0"
version = "1.63.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "httpx" },
@ -3782,9 +3760,53 @@ dependencies = [
{ name = "pydantic" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f9/e6/1cae7cd39ab29f2eebc87c0a82c7ffcdfbe88492d2fb4afaaad91534e1ff/pydantic_graph-1.60.0.tar.gz", hash = "sha256:9710e457c2f8c113fd63629f05174e45bdca917d90c69ec8cf558649f995505f", size = 58492, upload-time = "2026-02-17T00:33:32.66Z" }
sdist = { url = "https://files.pythonhosted.org/packages/7a/c8/aa3cb56552562b799f31e9de291c8bd88306308cfc9647d220dfff2bea18/pydantic_graph-1.63.0.tar.gz", hash = "sha256:5fd98bb22fa6181f0357a6ffad38a3214af12868bd46492d6456c5db434466b4", size = 58528, upload-time = "2026-02-23T17:56:39.118Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d8/6c/ce1c0eca77c6efbf7c7168c05a244c2756bde876a52575f750471d522024/pydantic_graph-1.60.0-py3-none-any.whl", hash = "sha256:741fa1e48424b0def86079a01100ad0652e75882f0352cd157232b75ace468a5", size = 72345, upload-time = "2026-02-17T00:33:25.077Z" },
{ url = "https://files.pythonhosted.org/packages/a4/1c/8dcae24c824dd2690fbe7375083b369b10ed1ad773e2b9d1122bb6c0fcdc/pydantic_graph-1.63.0-py3-none-any.whl", hash = "sha256:d9b7a387116f358d470c042b07aa08125cadfcfa8c08ef01769746a489aef0d5", size = 72353, upload-time = "2026-02-23T17:56:32.304Z" },
]
[[package]]
name = "pydantic-monty"
version = "0.0.7"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/f5/e3/0d8b2b025628477c839f894e632f5197872b19df0a86b2ec30fac3b5960a/pydantic_monty-0.0.7.tar.gz", hash = "sha256:2189ea1d7aadab2f95374733d692f51d1206379a4fc7ce18ab46895512e88f92", size = 684705, upload-time = "2026-02-19T14:12:47.235Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ce/7e/ca0884108c3237bb15bb2a1b3f24ddd957b9c750f1ed3211801497941999/pydantic_monty-0.0.7-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:f35e18f284524d26d5f27084e2b93eb40139055bbf0cab6221a043eb5e9ce2dc", size = 6264252, upload-time = "2026-02-19T14:13:59.317Z" },
{ url = "https://files.pythonhosted.org/packages/5a/5b/31f70c7792a857bacbdce90b8aae4629c31a9fec35f0116d91a2fb53241b/pydantic_monty-0.0.7-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8e5d8924c65bb1ced60785a156e28c73f7f79f164b4f090dc26312c3917ffff7", size = 6133285, upload-time = "2026-02-19T14:14:46.577Z" },
{ url = "https://files.pythonhosted.org/packages/42/56/c92216c0427e8a10a01fa98f29252f6fabd8ca80ca193e0fd30fe28e65c1/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6279a468469d5a3b80d94dd0ab6110cd291a1dfbb057fa7d6dbad1f499be855d", size = 6059856, upload-time = "2026-02-19T14:13:02.594Z" },
{ url = "https://files.pythonhosted.org/packages/b5/a7/bc3e67b12d8a9da65f2677d9a48bc1e055a1d853d48572ba4845a64075cf/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:cb5feb69a5902d059db5dab269f90423b85a22163668422be47ccac8d7f7c44a", size = 6313780, upload-time = "2026-02-19T14:13:46.45Z" },
{ url = "https://files.pythonhosted.org/packages/c8/97/a9b856b17ee1e54892dafbb7ea29305520cae2dcd8aafe82a26a0edbc33c/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a2f1913e9729aa6711092ecbfce764df199a4787fe3e23a7ed74c78bf846579e", size = 6856827, upload-time = "2026-02-19T14:13:40.836Z" },
{ url = "https://files.pythonhosted.org/packages/98/57/2d8184b9f5a0b2b3bb47fdad7061c6a182699824efe6de9d8dd19ee68c0a/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ec528f9f4194e6298757ad99e25da47f06f49ae2bc176ee26f49da5eb1dd7849", size = 6870737, upload-time = "2026-02-19T14:13:53.602Z" },
{ url = "https://files.pythonhosted.org/packages/6b/b3/fe3d3eff82b41e517739841a492d7a48ea2daf8e7b822299b848b5d4c0aa/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:75405b9186a9acfa49cd66aa339b5a2450d733a2d59fae20cc8be45e45204d5f", size = 6611843, upload-time = "2026-02-19T14:13:49.957Z" },
{ url = "https://files.pythonhosted.org/packages/ff/d2/fdd8fe135ea14e30b40adadc896dda6c805596688998c9bcdfd8d85a16cf/pydantic_monty-0.0.7-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1917e42fce4733f92f5f6ad64ae4d0e87abbf9fb284ef52589ac3e292e928bfe", size = 6692856, upload-time = "2026-02-19T14:13:00.635Z" },
{ url = "https://files.pythonhosted.org/packages/0d/a6/fdde6f8d76aa0cff4b53060b63d7f09dbb19da61a967cb4b3dfd972acf1c/pydantic_monty-0.0.7-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:d72a5d4f3ee7f9d2630b0379e4cfb397e181eb0b16e8c49a03f80ba6471edb89", size = 6236587, upload-time = "2026-02-19T14:14:32.602Z" },
{ url = "https://files.pythonhosted.org/packages/1b/7e/0580bbc001a39252b2f7da4b7504ac10572e4ca0ec967aebc5a9d752b6f7/pydantic_monty-0.0.7-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:20028220981516912f130986354ef6c926b98778146ca349560cb852e44d9ca6", size = 6672260, upload-time = "2026-02-19T14:13:26.527Z" },
{ url = "https://files.pythonhosted.org/packages/d8/53/578a7b781a5714db5c4b1989c6e876d30caa0adf8a5a4caad89abc306667/pydantic_monty-0.0.7-cp312-cp312-win32.whl", hash = "sha256:e28b1c3ed52892f8ac12ee0f2b535402dfe1cb1e5c18128f1cb69eb8b66c285a", size = 6131085, upload-time = "2026-02-19T14:14:13.296Z" },
{ url = "https://files.pythonhosted.org/packages/56/98/20bd45fcd472937b1b3438b7587e209e3cfd447c30d02f654b86b44adaad/pydantic_monty-0.0.7-cp312-cp312-win_amd64.whl", hash = "sha256:031dfab63ff9d7acdc641852e0d822603038cef1c27c5060900b9fd51cc853d0", size = 6664431, upload-time = "2026-02-19T14:13:29.968Z" },
{ url = "https://files.pythonhosted.org/packages/ec/fe/d8cb6c30d9d7bcc7d3c8d2c349a227e2a83cd1fbe7182f4941896eb35443/pydantic_monty-0.0.7-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:55d36818f8e35872ed35e395b41df8acc460bcdbbfd471fe0c39e293a1d50db5", size = 6262596, upload-time = "2026-02-19T14:14:15.2Z" },
{ url = "https://files.pythonhosted.org/packages/81/7a/f6b4881ca9779bd87eb8d8c0823133b56c232cf09d765c07a7f91d641490/pydantic_monty-0.0.7-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:acb437458d93d54a9658656545fb6c9b396dbe66f68633b3c57bfd2f4aa1d400", size = 6133793, upload-time = "2026-02-19T14:14:06.498Z" },
{ url = "https://files.pythonhosted.org/packages/39/b9/dfcffd95ff233b8c98db9254242d9c10190989762016d18509aa04d43b1b/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b196345ffa1997041cb870ea693148feafe270575e2c2963532eedde0e84dedf", size = 6059400, upload-time = "2026-02-19T14:13:38.953Z" },
{ url = "https://files.pythonhosted.org/packages/6b/2e/d6ecef842024267ddf4128613342b8985a7444e74f3a4a312713c913a91a/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d471e3cfe77d62edaf43b7f0962b95270ee4243abea12cb8e8cf1ad972dc3612", size = 6312625, upload-time = "2026-02-19T14:13:36.901Z" },
{ url = "https://files.pythonhosted.org/packages/75/2e/e4a2a9fbc3640bcee15b80c2f8ba0f97bf989c58c01d6da187524f71d12b/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3a67771afd385579bf3f894ce933fb9e467fba9a632ccf27246e271d448f6f5f", size = 6859902, upload-time = "2026-02-19T14:14:26.67Z" },
{ url = "https://files.pythonhosted.org/packages/d2/4a/7aaf5c793f52e3403892a2de1f5dd18ae38234d82111cc9b7d92443e5b0d/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bf662e1bbee4ddd318d5b8bfa9233173045029be0f67f15f816955b246bb7ec0", size = 6870524, upload-time = "2026-02-19T14:13:28.208Z" },
{ url = "https://files.pythonhosted.org/packages/c3/a9/c16f078864a273460923f1371b769c2719e1ce1ad86bc9031e3ed7fb3eae/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0151ff59a8a0d9e29ddb448affa33943108121e6e324795646a2f5facaf1a5d8", size = 6611960, upload-time = "2026-02-19T14:14:38.812Z" },
{ url = "https://files.pythonhosted.org/packages/20/d3/b3ef3432558a8cc9551d8b80a028a0f51cd2a518275932e03359eac3dc39/pydantic_monty-0.0.7-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:45f17a65134d3a0031e1f54d143770699f6a0ce92a1e74f0ae4914e52370f058", size = 6691834, upload-time = "2026-02-19T14:13:34.661Z" },
{ url = "https://files.pythonhosted.org/packages/ef/56/1ab5d1cbc0edfb522f0c28c9f5a7fc74eea6355234f73833087524d034bf/pydantic_monty-0.0.7-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:ede6e68cb8a1f7216e26b0b2fb6cd0eae7a92104be8a49d1042e9e428de9262b", size = 6235704, upload-time = "2026-02-19T14:13:48.276Z" },
{ url = "https://files.pythonhosted.org/packages/c4/d1/cdebae67b0543f696ed7daff8587dc8a458e6552b52d5877cb8e55be74b4/pydantic_monty-0.0.7-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:4b2fd51eea05a0cc37bb91f326efdb1acbcd4b8262dac1c55aeb208e51254978", size = 6671530, upload-time = "2026-02-19T14:12:56.96Z" },
{ url = "https://files.pythonhosted.org/packages/9f/53/c0dacaec260b71050fd6b31d09570f9d74bbb2a2e9586032694e92b9fa59/pydantic_monty-0.0.7-cp313-cp313-win32.whl", hash = "sha256:40f2092970c5899ac2a2784d712a4c7e194b33cd0133315254e4baf141cd6c93", size = 6130341, upload-time = "2026-02-19T14:14:09.701Z" },
{ url = "https://files.pythonhosted.org/packages/3e/05/31490a7a899d8bbb2e513630ea6f591ceb8a111c91fe7573a96c9f6b6327/pydantic_monty-0.0.7-cp313-cp313-win_amd64.whl", hash = "sha256:42cee2646415bb9bd7da428d169783203618a418f960c6f75c7a74d6946d6b31", size = 6664341, upload-time = "2026-02-19T14:14:19.008Z" },
{ url = "https://files.pythonhosted.org/packages/99/15/64aff358df0b822dd22f212fec501e3944edffe978a4ab05530ea641dc68/pydantic_monty-0.0.7-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:c97b2e1dcd0126417892595c1da724a8c4348f7dcec26ba774117bd51bde46f8", size = 6266090, upload-time = "2026-02-19T14:13:12.364Z" },
{ url = "https://files.pythonhosted.org/packages/64/90/7b5a4292eb9993eb8be9d958b5a57764818eeda471e3e79be7da4e9b49ba/pydantic_monty-0.0.7-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:6f40e6e133b309ba733874f5980ab6cf867ec8cea2a6a389a641819cd8dcb7cd", size = 6152219, upload-time = "2026-02-19T14:13:20.734Z" },
{ url = "https://files.pythonhosted.org/packages/43/e4/2740af0157eb3c6f10c16b0d8376b8c9cf0b910720fe90229885bccb4a91/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:57f2a327b6aa7402a2b2c3ddb3964bd45f12597f8f950dd4c5905b843b353b73", size = 6060942, upload-time = "2026-02-19T14:12:45.601Z" },
{ url = "https://files.pythonhosted.org/packages/f6/26/1cf235c2cc8e219a94ed8b11151280ba89e8020a475b6280c89e62f7275f/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:771f1f158af0de2480ea2a8862e4c0c7f79e9a13cc3e17529002e1abfc077f95", size = 6315477, upload-time = "2026-02-19T14:14:11.314Z" },
{ url = "https://files.pythonhosted.org/packages/b8/41/0faca7b9d8868822b7177ae941f193f397479bb114d3a6396466167a3198/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a4182312ee8c26d8834e76375b2b5c766cb5d86d1dcf1515aa95e02704fcad83", size = 6862130, upload-time = "2026-02-19T14:13:22.375Z" },
{ url = "https://files.pythonhosted.org/packages/34/7b/0f2bd4105a285f50f17af721e83a76ebc1186a9f07a2a29d6d576490a232/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8751696fe66fb1bdd429d98fde3a7f4b7dce9cb45f22095e28b96b690a422a2c", size = 6872292, upload-time = "2026-02-19T14:14:08.078Z" },
{ url = "https://files.pythonhosted.org/packages/c9/18/4380820d62d348afb1355814ea674788e28db7a73108a486e0b8898987de/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:40b73cd1137bb4fd9bf95ed5f87e48d960f7d556c30eeafde6996f908abbf183", size = 6636567, upload-time = "2026-02-19T14:14:01.464Z" },
{ url = "https://files.pythonhosted.org/packages/3e/57/29e5f89a558a6409d514bb2790c72442ec65804cbf1df6a870bbc0038673/pydantic_monty-0.0.7-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c1fa45a66757de5ea45c0809e15643bc2521959dbe2bc231694b22fa189decc9", size = 6693896, upload-time = "2026-02-19T14:14:34.128Z" },
{ url = "https://files.pythonhosted.org/packages/f6/26/5886d0f57ddb5ddf766ee2d0a4b3032267be9efd49a7c95c4d87a0b4b6a9/pydantic_monty-0.0.7-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:ffe8122db9f0f64619a66f4cee2f577245ba59158b7d651a2eae08df691d34f9", size = 6236867, upload-time = "2026-02-19T14:14:20.704Z" },
{ url = "https://files.pythonhosted.org/packages/b5/72/1bb8741baf84f217d92291b862f8a8cb64d735fe4be20be2827fdf787593/pydantic_monty-0.0.7-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:2122a5b6df53843329af01f6671300747449608d9cdf88ae5f9cf9977794e7a2", size = 6673504, upload-time = "2026-02-19T14:14:24.517Z" },
{ url = "https://files.pythonhosted.org/packages/d1/14/a4ff2bfe46350ffde4b5edc1f293b252cff90063f1f4cece49affe5a6462/pydantic_monty-0.0.7-cp314-cp314-win32.whl", hash = "sha256:bfbea2eddb9eef186326a6dfb27d79f8de434d7a3979f36f03b04216234a0275", size = 6131872, upload-time = "2026-02-19T14:13:14.437Z" },
{ url = "https://files.pythonhosted.org/packages/60/1f/d873f280aae5cbd27021189843fbd5f77be4262a7654ef30445d984518ab/pydantic_monty-0.0.7-cp314-cp314-win_amd64.whl", hash = "sha256:1b750afceef78f5c5d3e3e3c32a8060b3a7e1b97e3a00ac2bede5bc5e87cde8f", size = 6687125, upload-time = "2026-02-19T14:13:04.317Z" },
]
[[package]]
@ -4352,7 +4374,7 @@ name = "requests-toolbelt"
version = "1.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "requests" },
{ name = "requests", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f3/61/d7545dafb7ac2230c70d38d31cbfe4cc64f7144dc41f6e4e4b78ecd9f5bb/requests-toolbelt-1.0.0.tar.gz", hash = "sha256:7681a0a3d047012b5bdc0ee37d7f8f07ebe76ab08caeccfc3921ce23c88d5bc6", size = 206888, upload-time = "2023-05-01T04:11:33.229Z" }
wheels = [
@ -5325,16 +5347,16 @@ name = "voyageai"
version = "0.3.7"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiohttp" },
{ name = "aiolimiter" },
{ name = "ffmpeg-python" },
{ name = "langchain-text-splitters" },
{ name = "aiohttp", marker = "python_full_version < '3.14'" },
{ name = "aiolimiter", marker = "python_full_version < '3.14'" },
{ name = "ffmpeg-python", marker = "python_full_version < '3.14'" },
{ name = "langchain-text-splitters", marker = "python_full_version < '3.14'" },
{ name = "numpy", marker = "python_full_version < '3.14'" },
{ name = "pillow" },
{ name = "pydantic" },
{ name = "requests" },
{ name = "tenacity" },
{ name = "tokenizers" },
{ name = "pillow", marker = "python_full_version < '3.14'" },
{ name = "pydantic", marker = "python_full_version < '3.14'" },
{ name = "requests", marker = "python_full_version < '3.14'" },
{ name = "tenacity", marker = "python_full_version < '3.14'" },
{ name = "tokenizers", marker = "python_full_version < '3.14'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/94/16/1b46b3cd401e1717a68197c1fe336d7bb4e0a1833f8105e1738f5b1add05/voyageai-0.3.7.tar.gz", hash = "sha256:826cd97f97223f42b5babc5c459c9c80f3a8215ce5c0e007b0b276550f790d24", size = 26485, upload-time = "2025-12-16T18:43:05.26Z" }
wheels = [