Commit graph

96 commits

Author SHA1 Message Date
Yiorgis Gozadinos
c137305468
Scope the limit notice and spend the cite window on own turns only
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.

The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.

Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
2026-07-30 19:14:14 +03:00
Yiorgis Gozadinos
27ed0b3bb3
Send analysis to the sandbox when only its search budget is spent
The notice told the model to answer from what it had the moment
qa.max_searches ran out, while up to 15 code executions remained and
in-code search() does not count against that budget. It now names the
spent tool and points at whichever evidence tool still has budget,
falling back to answer-and-cite only when none do.

Also count RetryPromptPart in n_failed_tools: _cite rejects with
ModelRetry, so a run whose every cite attempt was refused reported zero
failures. And note that n_requests is the run's request count, which
tracks a capability's own budget only while it stays loaded.
2026-07-30 18:17:26 +03:00
Yiorgis Gozadinos
b62d6e920b
Drop budget_spent from eval attributes
It was n_rejected_searches > 0 recorded next to the integer it derived
from, and the name overclaimed: analysis_execute_code also raises
ToolFailed when execute_count exceeds max_executions, which the flag
never saw. Callers can compare the counters directly.
2026-07-30 15:57:48 +03:00
Yiorgis Gozadinos
721acbcf38
Address review on PR #524
- _budget_notice no longer names the cite tool after prepare_tools has
  withdrawn it; the post-grace state gets the plain no-tools text back.
- Split search-budget rejections from any failed tool call: the code tool
  raises ToolFailed for every error in model-written Python, so
  budget_spent was true for a ZeroDivisionError.
- docs/capabilities/rag.md described the old single-turn removal.
- Drop the rationale clause from the CHANGELOG entry.
2026-07-30 15:50:06 +03:00
Yiorgis Gozadinos
a528ab912f
Record per-case retrieval diagnostics in evaluations
Count tool traffic from the message history, where refused and repeated
calls stay visible, unlike state.searches which is keyed by query.
2026-07-30 14:56:17 +03:00
Yiorgis Gozadinos
044da7ae99
Open the eval database read-only outside population
Retrieval and QA only read from the database, but the benchmark opened it
writable, where an embedder identity differing from the stored one aborts
instead of warning. Running a pre-built database against a different
serving stack then needed a `rebuild --set-embedder` first.

Correct the debug-evals skill alongside it: the pydantic-ai span names are
`execute_tool {tool_name}` and `invoke_agent agent`, targets are
`{rag,analysis}-capability`, and no `skill_model` metadata key exists.
2026-07-28 11:36:27 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation 2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
0e8fa551f3
Restore hotpotqa evaluation dataset 2026-07-17 16:25:27 +03:00
Yiorgis Gozadinos
a82673a900
Route eval scripts through the shared telemetry configure 2026-07-10 11:38:33 +03:00
Yiorgis Gozadinos
4d03b1e669
Add nemotron-vl multimodal eval database and reference configs 2026-06-29 10:14:53 +03:00
Yiorgis Gozadinos
295570797c
Disable Logfire scrubbing in evaluations 2026-06-08 11:51:13 +03:00
Yiorgis Gozadinos
e9bd56467c
Add T²-RAGBench leaderboard submission exporter 2026-06-08 11:51:02 +03:00
Yiorgis Gozadinos
a3a73f1331
Add --filter-ids to run QA on a case-id subset 2026-06-08 09:36:01 +03:00
Yiorgis Gozadinos
1d7d540270
Match the numeric scale convention in Number-Match 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
d5b0aafa2b
Add T²-RAGBench TAT-DQA subset; generalize subset layout 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
de9731d5f3
Score Number-Match on the declared ANSWER line, magnitude and ×100 scale 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
390deb4203
Normalize unicode signs 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
80594cd38c
Handle percent/decimal convention in Number-Match 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
9f1bd9940a
Add T²-RAGBench FinQA evaluation dataset 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
eb74525084
Lift analysis-skill citation rate via SKILL.md tightening 2026-06-01 18:58:52 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
00f2a40a60
Refresh benchmarks doc and remove unused eval datasets 2026-06-01 10:40:51 +03:00
Yiorgis Gozadinos
392c74039b
bump pydantic-ai-slim to 1.100 and haiku.skills to 0.17
Migrate off two APIs slated for removal in pydantic-ai 2.0:

- Agent(tool_retries=, output_retries=) -> Agent(retries={"tools": ...,
  "output": ...}) in the LLM-as-judge evaluator.
- Evaluator.evaluation_name class attribute -> overriding
  get_default_evaluation_name() on the citation MRR / MAP evaluators.

haiku.skills 0.17.0 already migrated its internal AGUIAdapter,
MCPToolset and ProcessEventStream usage; no further changes needed on
our side beyond the pin bumps.
2026-05-21 13:20:54 +03:00
Yiorgis Gozadinos
f9392cee46
evaluations: judge model moves to config.evaluations.judge 2026-05-20 12:56:22 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow 2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
ceb645564c
Drop GEPA prompt optimization and --target qa from evaluations 2026-05-19 11:01:07 +03:00
Yiorgis Gozadinos
74dea65f21
analysis.model inherits qa.model when unset; per-skill vision gate 2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
1df6530d03
Bump pydantic-ai, prepare for 2.* 2026-05-18 15:14:29 +03:00
Yiorgis Gozadinos
cc59dc7203
Use upload_large_folder for hf upload 2026-05-06 13:12:08 +03:00
Yiorgis Gozadinos
ce7201271f
split open_rag_bench dataset into orb_text and orb_multimodal variants 2026-05-06 12:49:03 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark 2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
5c3a864a0b
rename open_rag_bench eval DBs to text/multimodal variants 2026-05-05 12:55:06 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source 2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
8962987ff9
pin judge model to ollama:qwen3.6 2026-04-29 12:24:06 +03:00
Yiorgis Gozadinos
7bfb818d77
remove dataset-specific system prompts 2026-04-28 14:33:25 +03:00
Yiorgis Gozadinos
7d288c525e
citation retrieval scoring 2026-04-28 12:46:16 +03:00
Yiorgis Gozadinos
d00befd0c4
benchmark RAG and analysis skills via --target 2026-04-28 12:09:19 +03:00
Yiorgis Gozadinos
ff8ad0879c
fix context expansion: respect section boundaries, remove max_context_items 2026-04-20 15:18:16 +03:00
Yiorgis Gozadinos
4f16714430
prevent stale files from accumulating on HF during uploads. 2026-04-16 13:45:24 +03:00
Yiorgis Gozadinos
e92f4774fa
Improve judge prompt 2026-04-16 12:11:54 +03:00
Yiorgis Gozadinos
b6113bf8ab
replace fixed-radius expansion with section-bounded algorithm
Context expansion is now automatic and structure-aware. For structured
documents, expands within the section containing the match. For sections
that exceed the budget or are too small, expands item-by-item outward
skipping noise labels. Unstructured documents use budget-based outward
expansion. Results sorted by relevance score.
2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
e4dc60c4f1
Update core dependencies 2026-04-03 10:14:27 +03:00
Yiorgis Gozadinos
fd327996b8
Configurable judge and reflect models for evaluations 2026-03-18 17:14:11 +02:00
Yiorgis Gozadinos
6ac5f277d8
Rename iterations to num_candidates 2026-03-12 13:00:51 +02:00
Yiorgis Gozadinos
5b3ad9bae7
Refactor QAPromptAdapter to accept QA agent directly, rewrite mock-heavy tests 2026-03-12 12:35:39 +02:00
Yiorgis Gozadinos
24275d2ef9
Fixes 2026-03-12 12:19:21 +02:00
Yiorgis Gozadinos
c785a54f5f
Respect config.prompts.qa in evaluations benchmark and optimization 2026-03-12 12:05:51 +02:00
Yiorgis Gozadinos
6812435ba0
cleanup 2026-03-12 12:05:50 +02:00