danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Re-record rlm tests

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-03-12 17:31:26 +02:00
parent 713e9087cc
commit 2c5128e5ba
No known key found for this signature in database
7 changed files with 2753 additions and 4540 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

515
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_aggregation.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7737'
- '7274'
content-type:
- application/json
host:
@ -173,18 +173,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -202,11 +190,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -232,10 +220,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -296,6 +285,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -325,7 +315,7 @@ interactions:
response:
headers:
content-length:
- '690'
- '699'
content-type:
- application/json
parsed_body:
@ -334,25 +324,26 @@ interactions:
index: 0
message:
content: ''
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
index: 0
type: function
created: 1772626955
id: chatcmpl-2
created: 1773329130
id: chatcmpl-407
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 85
prompt_tokens: 1688
total_tokens: 1773
prompt_tokens: 1594
total_tokens: 1679
status:
code: 200
message: OK
@ -405,7 +396,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8682'
- '8148'
content-type:
- application/json
host:
@ -450,18 +441,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -479,11 +458,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -509,10 +488,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -552,21 +532,20 @@ interactions:
- content: What is the total revenue across all quarterly reports?
role: user
- content: null
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''], r[''content''][:200])","stdout":"3\nQ1 Report 0.03226645663380623 Sales report Q1: Revenue was $100,000.\nQ2
Report 0.03226645663380623 Sales report Q2: Revenue was $150,000.\nQ3 Report 0.032258063554763794 Sales report Q3:
Revenue was $200,000.\n","stderr":"","success":true}'
- content: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly report
revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n","stdout":"3\nQ1
Report 0.03226645663380623\nQ2 Report 0.03226645663380623\nQ3 Report 0.032258063554763794\n","stderr":"","success":true}'
role: tool
tool_call_id: call_no1egdsi
tool_call_id: call_exbbulxp
model: gpt-oss
reasoning_effort: low
response_format:
@ -589,6 +568,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -618,7 +598,7 @@ interactions:
response:
headers:
content-length:
- '689'
- '680'
content-type:
- application/json
parsed_body:
@ -627,25 +607,25 @@ interactions:
index: 0
message:
content: ''
reasoning: Three reports with revenue numbers. Sum = 450k. Need citation chunks. Retrieve chunks.
reasoning: We need revenue numbers. Let's fetch chunks.
role: assistant
tool_calls:
- function:
arguments: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())"}'
arguments: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n"}'
name: execute_code
id: call_sg5vtxpb
id: call_2hd3mgon
index: 0
type: function
created: 1772626957
id: chatcmpl-37
created: 1773329146
id: chatcmpl-173
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 85
prompt_tokens: 1922
total_tokens: 2007
prompt_tokens: 1798
total_tokens: 1883
status:
code: 200
message: OK
@ -658,7 +638,7 @@ interactions:
connection:
- keep-alive
content-length:
- '9437'
- '8925'
content-type:
- application/json
host:
@ -703,18 +683,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -732,11 +700,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -762,10 +730,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -805,35 +774,34 @@ interactions:
- content: What is the total revenue across all quarterly reports?
role: user
- content: null
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''], r[''content''][:200])","stdout":"3\nQ1 Report 0.03226645663380623 Sales report Q1: Revenue was $100,000.\nQ2
Report 0.03226645663380623 Sales report Q2: Revenue was $150,000.\nQ3 Report 0.032258063554763794 Sales report Q3:
Revenue was $200,000.\n","stderr":"","success":true}'
- content: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly report
revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n","stdout":"3\nQ1
Report 0.03226645663380623\nQ2 Report 0.03226645663380623\nQ3 Report 0.032258063554763794\n","stderr":"","success":true}'
role: tool
tool_call_id: call_no1egdsi
tool_call_id: call_exbbulxp
- content: null
reasoning: Three reports with revenue numbers. Sum = 450k. Need citation chunks. Retrieve chunks.
reasoning: We need revenue numbers. Let's fetch chunks.
role: assistant
tool_calls:
- function:
arguments: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())"}'
arguments: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n"}'
name: execute_code
id: call_sg5vtxpb
id: call_2hd3mgon
type: function
- content: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
- content: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
role: tool
tool_call_id: call_sg5vtxpb
tool_call_id: call_2hd3mgon
model: gpt-oss
reasoning_effort: low
response_format:
@ -856,6 +824,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -885,7 +854,7 @@ interactions:
response:
headers:
content-length:
- '737'
- '713'
content-type:
- application/json
parsed_body:
@ -894,25 +863,25 @@ interactions:
index: 0
message:
content: ''
reasoning: Need to keep results variable. Use same scope. Let's re-run with storing results.
reasoning: Need to re-run search inside same code.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())"}'
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n"}'
name: execute_code
id: call_h6pfbenb
id: call_10n40kz9
index: 0
type: function
created: 1772626960
id: chatcmpl-20
created: 1773329159
id: chatcmpl-749
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 98
prompt_tokens: 2084
total_tokens: 2182
completion_tokens: 93
prompt_tokens: 1970
total_tokens: 2063
status:
code: 200
message: OK
@ -965,7 +934,7 @@ interactions:
connection:
- keep-alive
content-length:
- '10405'
- '9885'
content-type:
- application/json
host:
@ -1010,18 +979,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -1039,11 +996,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -1069,10 +1026,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -1112,51 +1070,50 @@ interactions:
- content: What is the total revenue across all quarterly reports?
role: user
- content: null
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''], r[''content''][:200])","stdout":"3\nQ1 Report 0.03226645663380623 Sales report Q1: Revenue was $100,000.\nQ2
Report 0.03226645663380623 Sales report Q2: Revenue was $150,000.\nQ3 Report 0.032258063554763794 Sales report Q3:
Revenue was $200,000.\n","stderr":"","success":true}'
- content: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly report
revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n","stdout":"3\nQ1
Report 0.03226645663380623\nQ2 Report 0.03226645663380623\nQ3 Report 0.032258063554763794\n","stderr":"","success":true}'
role: tool
tool_call_id: call_no1egdsi
tool_call_id: call_exbbulxp
- content: null
reasoning: Three reports with revenue numbers. Sum = 450k. Need citation chunks. Retrieve chunks.
reasoning: We need revenue numbers. Let's fetch chunks.
role: assistant
tool_calls:
- function:
arguments: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())"}'
arguments: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n"}'
name: execute_code
id: call_sg5vtxpb
id: call_2hd3mgon
type: function
- content: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
- content: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
role: tool
tool_call_id: call_sg5vtxpb
tool_call_id: call_2hd3mgon
- content: null
reasoning: Need to keep results variable. Use same scope. Let's re-run with storing results.
reasoning: Need to re-run search inside same code.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())"}'
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n"}'
name: execute_code
id: call_h6pfbenb
id: call_10n40kz9
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())","stdout":"Q1
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n","stdout":"Q1
Report Sales report Q1: Revenue was $100,000.\nQ2 Report Sales report Q2: Revenue was $150,000.\nQ3 Report Sales
report Q3: Revenue was $200,000.\n","stderr":"","success":true}'
role: tool
tool_call_id: call_h6pfbenb
tool_call_id: call_10n40kz9
model: gpt-oss
reasoning_effort: low
response_format:
@ -1179,6 +1136,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -1208,7 +1166,7 @@ interactions:
response:
headers:
content-length:
- '614'
- '643'
content-type:
- application/json
parsed_body:
@ -1216,25 +1174,26 @@ interactions:
- finish_reason: tool_calls
index: 0
message:
content: Now compute sum.
content: ''
reasoning: Sum revenues.
role: assistant
tool_calls:
- function:
arguments: '{"code":"total = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for
n in nums:\n total += int(n.replace('','', ''''))\nprint(total)"}'
arguments: '{"code":"total=0\nfor c in chunks:\n import re\n m=re.search(r''\\$([\\d,]+)'', c[''content''])\n if
m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n"}'
name: execute_code
id: call_rlqh271l
id: call_51jmnwfv
index: 0
type: function
created: 1772626962
id: chatcmpl-924
created: 1773329170
id: chatcmpl-461
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 83
prompt_tokens: 2312
total_tokens: 2395
completion_tokens: 88
prompt_tokens: 2196
total_tokens: 2284
status:
code: 200
message: OK
@ -1247,7 +1206,7 @@ interactions:
connection:
- keep-alive
content-length:
- '11090'
- '10618'
content-type:
- application/json
host:
@ -1292,18 +1251,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -1321,11 +1268,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -1351,10 +1298,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -1394,65 +1342,65 @@ interactions:
- content: What is the total revenue across all quarterly reports?
role: user
- content: null
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''], r[''content''][:200])","stdout":"3\nQ1 Report 0.03226645663380623 Sales report Q1: Revenue was $100,000.\nQ2
Report 0.03226645663380623 Sales report Q2: Revenue was $150,000.\nQ3 Report 0.032258063554763794 Sales report Q3:
Revenue was $200,000.\n","stderr":"","success":true}'
- content: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly report
revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n","stdout":"3\nQ1
Report 0.03226645663380623\nQ2 Report 0.03226645663380623\nQ3 Report 0.032258063554763794\n","stderr":"","success":true}'
role: tool
tool_call_id: call_no1egdsi
tool_call_id: call_exbbulxp
- content: null
reasoning: Three reports with revenue numbers. Sum = 450k. Need citation chunks. Retrieve chunks.
reasoning: We need revenue numbers. Let's fetch chunks.
role: assistant
tool_calls:
- function:
arguments: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())"}'
arguments: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n"}'
name: execute_code
id: call_sg5vtxpb
id: call_2hd3mgon
type: function
- content: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
- content: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
role: tool
tool_call_id: call_sg5vtxpb
tool_call_id: call_2hd3mgon
- content: null
reasoning: Need to keep results variable. Use same scope. Let's re-run with storing results.
reasoning: Need to re-run search inside same code.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())"}'
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n"}'
name: execute_code
id: call_h6pfbenb
id: call_10n40kz9
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())","stdout":"Q1
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n","stdout":"Q1
Report Sales report Q1: Revenue was $100,000.\nQ2 Report Sales report Q2: Revenue was $150,000.\nQ3 Report Sales
report Q3: Revenue was $200,000.\n","stderr":"","success":true}'
role: tool
tool_call_id: call_h6pfbenb
- content: Now compute sum.
tool_call_id: call_10n40kz9
- content: null
reasoning: Sum revenues.
role: assistant
tool_calls:
- function:
arguments: '{"code":"total = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for
n in nums:\n total += int(n.replace('','', ''''))\nprint(total)"}'
arguments: '{"code":"total=0\nfor c in chunks:\n import re\n m=re.search(r''\\$([\\d,]+)'', c[''content''])\n if
m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n"}'
name: execute_code
id: call_rlqh271l
id: call_51jmnwfv
type: function
- content: '{"code":"total = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for
n in nums:\n total += int(n.replace('','', ''''))\nprint(total)","stdout":"","stderr":"NameError: name ''chunks''
is not defined","success":false}'
- content: '{"code":"total=0\nfor c in chunks:\n import re\n m=re.search(r''\\$([\\d,]+)'', c[''content''])\n if
m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n","stdout":"","stderr":"NameError:
name ''chunks'' is not defined","success":false}'
role: tool
tool_call_id: call_rlqh271l
tool_call_id: call_51jmnwfv
model: gpt-oss
reasoning_effort: low
response_format:
@ -1475,6 +1423,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -1504,7 +1453,7 @@ interactions:
response:
headers:
content-length:
- '819'
- '848'
content-type:
- application/json
parsed_body:
@ -1513,27 +1462,26 @@ interactions:
index: 0
message:
content: ''
reasoning: Variable scope lost. Let's combine.
reasoning: Need to keep chunks variable. Let's combine.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n\ntotal = 0\nfor c in chunks:\n nums = await
regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for n in nums:\n total += int(n.replace('','',
''''))\nprint(total)"}'
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n\ntotal=0\nimport re\nfor c in chunks:\n m=re.search(r''\\$([\\d,]+)'',
c[''content''])\n if m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n"}'
name: execute_code
id: call_kbsx3dxm
id: call_axsiusi8
index: 0
type: function
created: 1772626965
id: chatcmpl-319
created: 1773329179
id: chatcmpl-56
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 132
prompt_tokens: 2483
total_tokens: 2615
completion_tokens: 140
prompt_tokens: 2380
total_tokens: 2520
status:
code: 200
message: OK
@ -1586,7 +1534,7 @@ interactions:
connection:
- keep-alive
content-length:
- '12120'
- '11699'
content-type:
- application/json
host:
@ -1631,18 +1579,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -1660,11 +1596,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -1690,10 +1626,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -1733,81 +1670,81 @@ interactions:
- content: What is the total revenue across all quarterly reports?
role: user
- content: null
reasoning: Need revenue from quarterly reports. Search for "quarterly report revenue".
reasoning: Need to search for quarterly reports revenue.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor
r in results[:5]:\n print(r[''document_title''], r[''score''], r[''content''][:200])"}'
arguments: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly
report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n"}'
name: execute_code
id: call_no1egdsi
id: call_exbbulxp
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''],
r[''score''], r[''content''][:200])","stdout":"3\nQ1 Report 0.03226645663380623 Sales report Q1: Revenue was $100,000.\nQ2
Report 0.03226645663380623 Sales report Q2: Revenue was $150,000.\nQ3 Report 0.032258063554763794 Sales report Q3:
Revenue was $200,000.\n","stderr":"","success":true}'
- content: '{"code":"import re\n# search for revenue in quarterly reports\nresults = await search(\"quarterly report
revenue\", limit=20)\nprint(len(results))\nfor r in results[:5]:\n print(r[''document_title''], r[''score''])\n","stdout":"3\nQ1
Report 0.03226645663380623\nQ2 Report 0.03226645663380623\nQ3 Report 0.032258063554763794\n","stderr":"","success":true}'
role: tool
tool_call_id: call_no1egdsi
tool_call_id: call_exbbulxp
- content: null
reasoning: Three reports with revenue numbers. Sum = 450k. Need citation chunks. Retrieve chunks.
reasoning: We need revenue numbers. Let's fetch chunks.
role: assistant
tool_calls:
- function:
arguments: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())"}'
arguments: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n"}'
name: execute_code
id: call_sg5vtxpb
id: call_2hd3mgon
type: function
- content: '{"code":"chunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''].strip())","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
- content: '{"code":"# get chunks for each result\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''],
chunk[''content''][:200])\n","stdout":"","stderr":"NameError: name ''results'' is not defined","success":false}'
role: tool
tool_call_id: call_sg5vtxpb
tool_call_id: call_2hd3mgon
- content: null
reasoning: Need to keep results variable. Use same scope. Let's re-run with storing results.
reasoning: Need to re-run search inside same code.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())"}'
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n"}'
name: execute_code
id: call_h6pfbenb
id: call_10n40kz9
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n print(c[''document_title''], c[''content''].strip())","stdout":"Q1
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n print(chunk[''document_title''], chunk[''content''][:200])\n","stdout":"Q1
Report Sales report Q1: Revenue was $100,000.\nQ2 Report Sales report Q2: Revenue was $150,000.\nQ3 Report Sales
report Q3: Revenue was $200,000.\n","stderr":"","success":true}'
role: tool
tool_call_id: call_h6pfbenb
- content: Now compute sum.
role: assistant
tool_calls:
- function:
arguments: '{"code":"total = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for
n in nums:\n total += int(n.replace('','', ''''))\nprint(total)"}'
name: execute_code
id: call_rlqh271l
type: function
- content: '{"code":"total = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for
n in nums:\n total += int(n.replace('','', ''''))\nprint(total)","stdout":"","stderr":"NameError: name ''chunks''
is not defined","success":false}'
role: tool
tool_call_id: call_rlqh271l
tool_call_id: call_10n40kz9
- content: null
reasoning: Variable scope lost. Let's combine.
reasoning: Sum revenues.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n\ntotal = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'',
c[''content''])\n for n in nums:\n total += int(n.replace('','', ''''))\nprint(total)"}'
arguments: '{"code":"total=0\nfor c in chunks:\n import re\n m=re.search(r''\\$([\\d,]+)'', c[''content''])\n if
m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n"}'
name: execute_code
id: call_kbsx3dxm
id: call_51jmnwfv
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c
= await get_chunk(r[''chunk_id''])\n chunks.append(c)\n\ntotal = 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'',
c[''content''])\n for n in nums:\n total += int(n.replace('','', ''''))\nprint(total)","stdout":"450000\n","stderr":"","success":true}'
- content: '{"code":"total=0\nfor c in chunks:\n import re\n m=re.search(r''\\$([\\d,]+)'', c[''content''])\n if
m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n","stdout":"","stderr":"NameError:
name ''chunks'' is not defined","success":false}'
role: tool
tool_call_id: call_kbsx3dxm
tool_call_id: call_51jmnwfv
- content: null
reasoning: Need to keep chunks variable. Let's combine.
role: assistant
tool_calls:
- function:
arguments: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n\ntotal=0\nimport re\nfor c in chunks:\n m=re.search(r''\\$([\\d,]+)'',
c[''content''])\n if m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n"}'
name: execute_code
id: call_axsiusi8
type: function
- content: '{"code":"results = await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk
= await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n\ntotal=0\nimport re\nfor c in chunks:\n m=re.search(r''\\$([\\d,]+)'',
c[''content''])\n if m:\n val=int(m.group(1).replace('','',''''))\n total+=val\nprint(total)\n","stdout":"450000\n","stderr":"","success":true}'
role: tool
tool_call_id: call_axsiusi8
model: gpt-oss
reasoning_effort: low
response_format:
@ -1830,6 +1767,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -1859,7 +1797,7 @@ interactions:
response:
headers:
content-length:
- '880'
- '980'
content-type:
- application/json
parsed_body:
@ -1867,21 +1805,22 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"The total revenue across all quarterly reports is $450,000. This figure comes from three reports:\n-
Q1 Report: $100,000\n- Q2 Report: $150,000\n- Q3 Report: $200,000\nSumming these gives the stated total.\n", "program":"results
= await search(\"quarterly report revenue\", limit=20)\nchunks = []\nfor r in results:\n c = await get_chunk(r[''chunk_id''])\n chunks.append(c)\n\ntotal
= 0\nfor c in chunks:\n nums = await regex_findall(r''\\$([\\d,]+)'', c[''content''])\n for n in nums:\n total
+= int(n.replace('','', ''''))\nprint(total)"}'
content: '{"answer":"The total revenue reported across all quarterly reports is $450,000. This figure is the sum
of $100,000 from Q1, $150,000 from Q2, and $200,000 from Q3, as extracted from the respective quarterly report
documents.","program":"import re\n# Search for quarterly report revenue chunks\nresults = await search(\"quarterly
report revenue\", limit=20)\nchunks = []\nfor r in results:\n chunk = await get_chunk(r[''chunk_id''])\n chunks.append(chunk)\n#
Sum revenue amounts\ntotal = 0\nfor c in chunks:\n m = re.search(r\"\\$([\\d,]+)\", c[''content''])\n if
m:\n val = int(m.group(1).replace('','', ''''))\n total += val\nprint(total)"}'
role: assistant
created: 1772626970
id: chatcmpl-772
created: 1773329189
id: chatcmpl-169
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 178
prompt_tokens: 2744
total_tokens: 2922
completion_tokens: 186
prompt_tokens: 2655
total_tokens: 2841
status:
code: 200
message: OK

87
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_count_documents.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7721'
- '7258'
content-type:
- application/json
host:
@ -173,18 +173,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -202,11 +190,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -232,10 +220,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -296,6 +285,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -325,7 +315,7 @@ interactions:
response:
headers:
content-length:
- '522'
- '525'
content-type:
- application/json
parsed_body:
@ -338,20 +328,20 @@ interactions:
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n"}'
name: execute_code
id: call_cthflnpr
id: call_04viywlj
index: 0
type: function
created: 1772626945
id: chatcmpl-979
created: 1773329125
id: chatcmpl-653
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 45
prompt_tokens: 1686
total_tokens: 1731
completion_tokens: 47
prompt_tokens: 1592
total_tokens: 1639
status:
code: 200
message: OK
@ -364,7 +354,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8163'
- '7706'
content-type:
- application/json
host:
@ -409,18 +399,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -438,11 +416,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -468,10 +446,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -515,13 +494,13 @@ interactions:
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n"}'
name: execute_code
id: call_cthflnpr
id: call_04viywlj
type: function
- content: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))","stdout":"3\n","stderr":"","success":true}'
- content: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n","stdout":"3\n","stderr":"","success":true}'
role: tool
tool_call_id: call_cthflnpr
tool_call_id: call_04viywlj
model: gpt-oss
reasoning_effort: low
response_format:
@ -544,6 +523,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -573,7 +553,7 @@ interactions:
response:
headers:
content-length:
- '459'
- '416'
content-type:
- application/json
parsed_body:
@ -581,18 +561,17 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There are 3 documents in the database.","program":"docs = await list_documents(limit=1000)\nprint(f''There
are {len(docs)} documents in the database.'')" }'
content: '{"answer":"There are 3 documents in the database.","program":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
role: assistant
created: 1772626947
id: chatcmpl-490
created: 1773329134
id: chatcmpl-151
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 49
prompt_tokens: 1773
total_tokens: 1822
completion_tokens: 38
prompt_tokens: 1683
total_tokens: 1721
status:
code: 200
message: OK

2618
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_extract.yaml
View file

File diff suppressed because one or more lines are too long

407
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_search_and_get_chunk.yaml
View file

File diff suppressed because one or more lines are too long

1771
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_semantic_analysis_with_llm.yaml
View file

File diff suppressed because one or more lines are too long

94
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_filter.yaml
View file

@ -128,7 +128,7 @@ interactions:
connection:
- keep-alive
content-length:
- '7715'
- '7252'
content-type:
- application/json
host:
@ -173,18 +173,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -202,11 +190,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -232,10 +220,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -296,6 +285,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -325,7 +315,7 @@ interactions:
response:
headers:
content-length:
- '522'
- '576'
content-type:
- application/json
parsed_body:
@ -334,24 +324,25 @@ interactions:
index: 0
message:
content: ''
reasoning: Need to list documents.
reasoning: We need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''id''],
d.get(''title''))"}'
name: execute_code
id: call_jsd3hga7
id: call_nap68rkx
index: 0
type: function
created: 1772626976
id: chatcmpl-987
created: 1773329124
id: chatcmpl-461
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 46
prompt_tokens: 1684
total_tokens: 1730
completion_tokens: 64
prompt_tokens: 1590
total_tokens: 1654
status:
code: 200
message: OK
@ -364,7 +355,7 @@ interactions:
connection:
- keep-alive
content-length:
- '8160'
- '7846'
content-type:
- application/json
host:
@ -409,18 +400,6 @@ interactions:
- `pictures`: list of figures/images with metadata
- `pages`: page dimensions and metadata
### await regex_findall(pattern, text) -> list[str]
Find all non-overlapping matches of a regular expression pattern in text.
### await regex_sub(pattern, repl, text) -> str
Replace all occurrences of a regular expression pattern with a replacement string.
### await regex_search(pattern, text) -> dict | None
Search for the first match of a pattern. Returns a dict with keys: group, groups, start, end — or None if no match.
### await regex_split(pattern, text) -> list[str]
Split text by a regular expression pattern.
### await llm(prompt) -> str
Call an LLM directly with the given prompt. Returns the response as a string.
Use this for classification, summarization, extraction, or any task where you
@ -438,11 +417,11 @@ interactions:
## Available Python Features
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `sorted()`/`.sort(key=...)`, try/except, and the `json` module.
The interpreter supports: variables, arithmetic, strings, f-strings, lists, dicts, tuples, sets, loops, conditionals, comprehensions, functions, async/await, `map()`, `filter()`, `getattr()`, `sorted()`/`.sort(key=...)`, try/except, and the `json`, `re`, `math` modules.
Not supported: imports (other than `json`), class definitions, generators/yield, match statements, decorators, `with` statements.
Not supported: most imports (only `json`, `re`, `math` are available), class definitions, generators/yield, match statements, decorators, `with` statements.
For pattern matching or text extraction, use the `regex_*` functions, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
For pattern matching or text extraction, use `import re`, string methods (`str.split`, `str.find`, `str.startswith`, `in` operator), or the `llm()` function.
## Strategy Guide
@ -468,10 +447,11 @@ interactions:
### Extracting data with regex
```python
import re
numbers = []
results = await search("financial data", limit=20)
for r in results:
amounts = await regex_findall(r'\$([\d,]+)', r['content'])
amounts = re.findall(r'\$([\d,]+)', r['content'])
for a in amounts:
numbers.append(int(a.replace(',', '')))
if numbers:
@ -511,17 +491,19 @@ interactions:
- content: How many documents are available?
role: user
- content: null
reasoning: Need to list documents.
reasoning: We need to list documents.
role: assistant
tool_calls:
- function:
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n"}'
arguments: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''id''],
d.get(''title''))"}'
name: execute_code
id: call_jsd3hga7
id: call_nap68rkx
type: function
- content: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\n","stdout":"1\n","stderr":"","success":true}'
- content: '{"code":"docs = await list_documents(limit=1000)\nprint(len(docs))\nfor d in docs:\n print(d[''id''],
d.get(''title''))","stdout":"1\n7b1434a1-bbe5-4ae1-bb3e-5400eed94d72 Cats\n","stderr":"","success":true}'
role: tool
tool_call_id: call_jsd3hga7
tool_call_id: call_nap68rkx
model: gpt-oss
reasoning_effort: low
response_format:
@ -544,6 +526,7 @@ interactions:
strict: true
type: json_schema
stream: false
temperature: 0.0
tool_choice: auto
tools:
- function:
@ -573,7 +556,7 @@ interactions:
response:
headers:
content-length:
- '430'
- '484'
content-type:
- application/json
parsed_body:
@ -581,17 +564,18 @@ interactions:
- finish_reason: stop
index: 0
message:
content: '{"answer":"There is 1 document available in the knowledge base.","program":"docs = await list_documents(limit=1000)\nprint(len(docs))"}'
content: '{"answer":"There is 1 document available in the knowledge base.","program":"docs = await list_documents(limit=1000)\nprint(len(docs))\nfor
d in docs:\n print(d[''id''], d.get(''title''))"}'
role: assistant
created: 1772626977
id: chatcmpl-959
created: 1773329133
id: chatcmpl-985
model: gpt-oss
object: chat.completion
system_fingerprint: fp_ollama
usage:
completion_tokens: 37
prompt_tokens: 1774
total_tokens: 1811
completion_tokens: 59
prompt_tokens: 1741
total_tokens: 1800
status:
code: 200
message: OK

1801
tests/cassettes/test_rlm/TestClientRLMIntegration.test_rlm_with_preloaded_documents.yaml
View file

File diff suppressed because it is too large Load diff