danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update prompts & docs. Remove docker sandbox workflow

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-02-17 16:05:29 +02:00
parent c61ded1271
commit 017712c9b1
No known key found for this signature in database
10 changed files with 2780 additions and 7213 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

19
.github/workflows/test.yml vendored
View file

@ -76,22 +76,3 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: false
test-docker-sandbox:
needs: [lint]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
with:
enable-cache: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: "pyproject.toml"
- name: Install dependencies
run: uv sync --all-extras
- name: Build Docker image
run: docker build -t haiku-rag-slim:test -f docker/Dockerfile.slim .
- name: Run Docker integration tests
run: uv run pytest tests/agents/rlm/test_sandbox.py -v

88
docs/agents/rlm.md
View file

@ -5,13 +5,13 @@ The RLM agent enables complex analytical tasks by writing and executing Python c
- **Aggregation**: "How many documents mention security vulnerabilities?"
- **Computation**: "What's the average revenue across all quarterly reports?"
- **Multi-document analysis**: "Compare the key findings between Report A and Report B"
- **Structured data extraction**: "Extract all tables from the document and summarize them"
- **Structured data extraction**: "Extract all dollar amounts and compute totals"
## How It Works
1. The agent receives a question
2. It writes Python code to explore the knowledge base
3. Code executes in a sandboxed environment with access to haiku.rag functions
3. Code executes in a sandboxed Python interpreter with access to haiku.rag functions
4. The agent iterates: run code, examine results, refine approach
5. Final answer is synthesized from the gathered data
@ -93,22 +93,19 @@ if content:
Returns the document content as a string, or `None` if not found.
### get_docling_document(id_or_title)
### get_chunk(chunk_id)
Get the structured DoclingDocument object for advanced analysis of tables, figures, and document structure.
Get a specific chunk by its ID (from search results). Use this to retrieve full chunk details and metadata for citations.
```python
doc = get_docling_document("Technical Manual")
if doc:
print(f"Tables: {len(doc.tables)}")
print(f"Pictures: {len(doc.pictures)}")
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
Returns dict with keys: `chunk_id`, `content`, `document_id`, `document_title`, `headings`, `page_numbers`, `labels`
### llm(prompt)
Call an LLM directly for classification, summarization, or extraction tasks.
@ -133,36 +130,39 @@ for doc in documents:
Each document dict has keys: `id`, `title`, `uri`, `content`
## Imports
## Python Features
The sandbox runs in a Docker container with full Python available. Any module installed in the container image can be imported:
The sandbox uses [pydantic-monty](https://github.com/pydantic/monty), a minimal secure Python interpreter written in Rust. It supports a subset of Python:
**Supported:** variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
**Not supported:** imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function:
```python
import re
import json
from collections import Counter
# Extract and count patterns
results = search("error", limit=50)
error_types = []
# Extract data with llm() instead of regex
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'Error: (\w+)', r['content'])
error_types.extend(matches)
print(Counter(error_types).most_common(10))
extracted = llm(f"Extract all dollar amounts as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
The default image (`ghcr.io/ggozad/haiku.rag-slim`) includes the Python standard library. Custom images can add additional packages like `pandas` or `numpy`.
## Sandboxed Execution
## Docker Sandbox
Code executes in an isolated interpreter with:
Code executes in an isolated Docker container with:
- **Read-only database**: The LanceDB database is mounted read-only
- **Memory limits**: Configurable memory limit (default 512MB)
- **No filesystem access**: Code cannot read or write files
- **No network access**: Code cannot make HTTP requests or open sockets
- **No imports**: Only the `json` module is available
- **Execution timeout**: Code times out after configurable limit (default 60s)
- **Output truncation**: Large outputs are truncated to prevent memory issues
- **Container reuse**: Within a single `rlm()` call, the container stays warm for multiple code executions
## Context Filter
@ -193,26 +193,4 @@ rlm:
name: claude-sonnet-4-20250514
code_timeout: 60.0 # Max seconds for code execution
max_output_chars: 50000 # Truncate output after this many chars
docker_image: "ghcr.io/ggozad/haiku.rag-slim:latest" # Container image
docker_memory_limit: "512m" # Container memory limit
```
### Custom Docker Image
To add additional Python packages, create a custom Dockerfile:
```dockerfile
FROM ghcr.io/ggozad/haiku.rag-slim:latest
RUN pip install pandas numpy
```
Build and configure:
```bash
docker build -t my-rlm-image .
```
```yaml
rlm:
docker_image: "my-rlm-image"
```

89
haiku_rag_slim/haiku/rag/agents/rlm/prompts.py
View file

@ -6,7 +6,7 @@ CRITICAL: Inside execute_code, these functions are ALREADY available in the name
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
@ -22,10 +22,10 @@ Returns list of dicts with keys: id, title, uri, created_at
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
@ -42,8 +42,13 @@ for doc in documents:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
@ -51,51 +56,9 @@ You can import any Python standard library module.
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
@ -111,23 +74,31 @@ for doc in docs:
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\\$([\\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```

2133
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_aggregation.yaml
View file

File diff suppressed because one or more lines are too long

227
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_count_documents.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7719'
- '7083'
content-type:
- application/json
host:
@ -145,7 +145,7 @@ interactions:
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
@ -161,10 +161,10 @@ interactions:
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
@ -181,8 +181,13 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
@ -190,51 +195,9 @@ interactions:
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
@ -250,23 +213,31 @@ interactions:
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -343,7 +314,7 @@ interactions:
response:
headers:
content-length:
- '585'
- '552'
content-type:
- application/json
parsed_body:
@ -352,24 +323,24 @@ interactions:
index: 0
message:
content: ''
reasoning: We need to list documents via list_documents to count.
reasoning: Need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])"}'
arguments: '{"code":"docs=list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])"}'
name: execute_code
id: call_1he6vvcy
id: call_tumky965
index: 0
type: function
created: 1771336260
id: chatcmpl-619
created: 1771336699
id: chatcmpl-910
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 63
prompt_tokens: 1734
total_tokens: 1797
completion_tokens: 56
prompt_tokens: 1562
total_tokens: 1618
status:
code: 200
message: OK
@ -382,7 +353,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8283'
- '7612'
content-type:
- application/json
host:
@ -399,7 +370,7 @@ interactions:
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
@ -415,10 +386,10 @@ interactions:
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
@ -435,8 +406,13 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
@ -444,51 +420,9 @@ interactions:
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
@ -504,23 +438,31 @@ interactions:
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -548,18 +490,18 @@ interactions:
- content: How many documents are in the database?
role: user
- content: null
reasoning: We need to list documents via list_documents to count.
reasoning: Need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])"}'
arguments: '{"code":"docs=list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])"}'
name: execute_code
id: call_1he6vvcy
id: call_tumky965
type: function
- content: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])","stdout":"3\nDoc
- content: '{"code":"docs=list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''title''])","stdout":"3\nDoc
1\nDoc 2\nDoc 3\n","stderr":"","success":true}'
role: tool
tool_call_id: call_1he6vvcy
tool_call_id: call_tumky965
model: gpt-oss
reasoning_effort: low
stream: false
@ -610,7 +552,7 @@ interactions:
response:
headers:
content-length:
- '577'
- '517'
content-type:
- application/json
parsed_body:
@ -618,19 +560,18 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There are 3 documents in the database. They are titled: Doc 1, Doc 2, and Doc 3.","program":"#
Count documents in the database\n\ndocs = list_documents(limit=1000)\nprint(f\"Number of documents: {len(docs)}\")\nfor
d in docs:\n print(f\"- {d[''title'']}\")"}'
content: '{"answer":"There are 3 documents in the database.","program":"docs = list_documents(limit=1000)\nprint(f\"Number
of documents: {len(docs)}\")\nfor doc in docs:\n print(f\"- {doc[''title'']} (ID: {doc[''id'']})\")"}'
role: assistant
created: 1771336262
id: chatcmpl-256
created: 1771336701
id: chatcmpl-57
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 91
prompt_tokens: 1864
total_tokens: 1955
completion_tokens: 82
prompt_tokens: 1684
total_tokens: 1766
status:
code: 200
message: OK

2604
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_extract.yaml
View file

File diff suppressed because one or more lines are too long

800
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_get_chunk.yaml
View file

File diff suppressed because one or more lines are too long

2971
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_semantic_analysis_with_llm.yaml
View file

File diff suppressed because it is too large Load diff

226
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_filter.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7713'
- '7077'
content-type:
- application/json
host:
@ -145,7 +145,7 @@ interactions:
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
@ -161,10 +161,10 @@ interactions:
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
@ -181,8 +181,13 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
@ -190,51 +195,9 @@ interactions:
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
@ -250,23 +213,31 @@ interactions:
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -343,7 +314,7 @@ interactions:
response:
headers:
content-length:
- '527'
- '526'
content-type:
- application/json
parsed_body:
@ -352,24 +323,24 @@ interactions:
index: 0
message:
content: ''
reasoning: Need to call list_documents to count.
reasoning: Need to list_documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs=list_documents(limit=1000);print(len(docs)); print(docs[:3])"}'
name: execute_code
id: call_9nd3m1g0
id: call_rpfcy176
index: 0
type: function
created: 1771336314
id: chatcmpl-255
created: 1771336720
id: chatcmpl-390
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 46
prompt_tokens: 1732
total_tokens: 1778
completion_tokens: 47
prompt_tokens: 1560
total_tokens: 1607
status:
code: 200
message: OK
@ -382,7 +353,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8154'
- '7655'
content-type:
- application/json
host:
@ -399,7 +370,7 @@ interactions:
- search("query") ✓ CORRECT
- from haiku.rag import search ✗ WRONG - will fail
You have access to a sandboxed Python environment with these haiku.rag functions (use them directly, no imports needed):
You have access to a sandboxed Python interpreter with these haiku.rag functions (use them directly, no imports needed):
## Available Functions
@ -415,10 +386,10 @@ interactions:
Get the full text content of a document by ID, title, or URI.
Returns the document content as a string, or None if not found.
### get_docling_document(id_or_title) -> DoclingDocument | None
Get the structured DoclingDocument object for advanced analysis.
Returns a DoclingDocument object, or None if not found.
See "DoclingDocument API" section below for how to use it.
### get_chunk(chunk_id) -> dict | None
Get a specific chunk by its ID (from search results).
Returns dict with keys: chunk_id, content, document_id, document_title, headings, page_numbers, labels
Use this to retrieve full chunk details and metadata for citation.
### llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
@ -435,8 +406,13 @@ interactions:
```
Check if it exists with: `if 'documents' in dir(): ...`
## Standard Library Modules
You can import any Python standard library module.
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, try/except, and the `json` module.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use string methods (`str.split`, `str.find`, `str.startswith`, `in` operator) or the `llm()` function.
## Strategy Guide
@ -444,51 +420,9 @@ interactions:
2. **If get_document returns None**: Use `list_documents()` to see actual document titles, or `search()` to find relevant content.
3. **Iterative Refinement**: Run code, examine results, adjust your approach based on what you find.
4. **Use print() Liberally**: The sandbox captures stdout - print intermediate results to see what you're working with.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and use collections.
5. **Aggregate with Code**: For counting, averaging, or comparing across documents, write loops and data structures.
6. **Use llm() for Classification/Extraction**: When you need to classify, summarize, or extract structured data from content you already have, use llm().
7. **Cite Your Sources**: Track which documents/chunks informed your answer for citation.
## DoclingDocument API
When you call `get_docling_document(id_or_title)`, you get a DoclingDocument object for structured document analysis.
### Properties
- `doc.texts` - List of all text items (paragraphs, headings, etc.)
- `doc.tables` - List of all tables
- `doc.pictures` - List of all pictures/figures
- `doc.name` - Document name
### Methods
- `doc.iterate_items(with_groups=False)` - Iterate all items with hierarchy level
Returns tuples of (item, level) where level is nesting depth
- `doc.export_to_markdown()` - Export entire document as markdown string
### Text Item Properties
- `item.text` - The text content
- `item.label` - Type: title, paragraph, section_header, list_item, etc. (lowercase enum values)
- `item.prov` - Provenance (page numbers, bounding boxes)
### Table Access
- `table.data.num_rows`, `table.data.num_cols` - Dimensions
- `table.data.table_cells` - List of TableCell objects
- `cell.text`, `cell.start_row_offset_idx`, `cell.start_col_offset_idx`
### Example Usage
```python
doc = get_docling_document("My Document")
# Get all headings
headings = [t.text for t in doc.texts if "header" in str(t.label)]
# Iterate with structure
for item, level in doc.iterate_items():
print(" " * level + item.text[:50])
# Extract table data
for table in doc.tables:
for cell in table.data.table_cells:
print(f"Row {cell.start_row_offset_idx}, Col {cell.start_col_offset_idx}: {cell.text}")
```
7. **Cite Your Sources**: Use get_chunk() to retrieve chunk metadata for citations. Track which documents/chunks informed your answer.
## Example Patterns
@ -504,23 +438,31 @@ interactions:
print(f"Total: {count}")
```
### Aggregating data across documents
### Extracting data with llm()
```python
import re
numbers = []
results = search("financial data", limit=20)
for r in results:
matches = re.findall(r'\$([\d,]+)', r['content'])
for m in matches:
numbers.append(int(m.replace(',', '')))
print(f"Average: ${sum(numbers)/len(numbers):,.2f}")
extracted = llm(f"Extract all dollar amounts from this text as a comma-separated list of numbers (no $ signs): {r['content']}")
for part in extracted.split(','):
part = part.strip().replace(',', '')
if part.isdigit():
numbers.append(int(part))
if numbers:
print(f"Average: {sum(numbers) / len(numbers)}")
```
### Using search results with get_chunk for citations
```python
results = search("safety requirements", limit=5)
for r in results:
chunk = get_chunk(r['chunk_id'])
print(f"From '{chunk['document_title']}', page {chunk['page_numbers']}: {chunk['content'][:100]}")
```
### Using llm() for classification
```python
# Get document content
content = get_document("Q1 Report")
# Use llm() to classify sentiment
sentiment = llm(f"Classify the sentiment as positive, negative, or mixed: {content}")
print(sentiment)
```
@ -548,17 +490,18 @@ interactions:
- content: How many documents are available?
role: user
- content: null
reasoning: Need to call list_documents to count.
reasoning: Need to list_documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs=list_documents(limit=1000);print(len(docs)); print(docs[:3])"}'
name: execute_code
id: call_9nd3m1g0
id: call_rpfcy176
type: function
- content: '{"code":"docs = list_documents(limit=1000)\nprint(len(docs))","stdout":"1\n","stderr":"","success":true}'
- content: '{"code":"docs=list_documents(limit=1000);print(len(docs)); print(docs[:3])","stdout":"1\n[{''id'': ''50c659ee-70c5-4854-a418-12ae2abdcf40'',
''title'': ''Cats'', ''uri'': None, ''created_at'': ''2026-02-17 15:58:36.831575''}]\n","stderr":"","success":true}'
role: tool
tool_call_id: call_9nd3m1g0
tool_call_id: call_rpfcy176
model: gpt-oss
reasoning_effort: low
stream: false
@ -609,7 +552,7 @@ interactions:
response:
headers:
content-length:
- '424'
- '453'
content-type:
- application/json
parsed_body:
@ -617,17 +560,18 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There is 1 document available in the knowledge base.","program":"docs = list_documents(limit=1000)\nprint(len(docs))"}'
content: '{"answer":"There is 1 document available in the knowledge base.","program":"docs=list_documents(limit=1000)\nprint(f\"Number
of documents: {len(docs)}\")"}'
role: assistant
created: 1771336315
id: chatcmpl-500
created: 1771336721
id: chatcmpl-97
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 39
prompt_tokens: 1819
total_tokens: 1858
completion_tokens: 46
prompt_tokens: 1713
total_tokens: 1759
status:
code: 200
message: OK

836
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_preloaded_documents.yaml
View file

File diff suppressed because it is too large Load diff