danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Drop budget_spent from eval attributes

Browse source 
It was n_rejected_searches > 0 recorded next to the integer it derived
from, and the name overclaimed: analysis_execute_code also raises
ToolFailed when execute_count exceeds max_executions, which the flag
never saw. Callers can compare the counters directly.
This commit is contained in:
Yiorgis Gozadinos 2026-07-30 15:57:48 +03:00
parent 721acbcf38
commit b62d6e920b
No known key found for this signature in database
3 changed files with 10 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CHANGELOG.md
View file

@ -3,7 +3,7 @@
### Added ### Added
- `evaluations run` records `cited_chunk_ids`, `searched_uris`, `n_searches`, `n_search_calls`, `n_rejected_searches`, `n_failed_tools`, `n_executions`, `n_requests` and `budget_spent` as eval attributes alongside `cited_uris`. - `evaluations run` records `cited_chunk_ids`, `searched_uris`, `n_searches`, `n_search_calls`, `n_rejected_searches`, `n_failed_tools`, `n_executions` and `n_requests` as eval attributes alongside `cited_uris`.
### Changed ### Changed

1
evaluations/evaluations/benchmark.py
View file

@ -442,7 +442,6 @@ async def run_qa_benchmark(
set_eval_attribute("n_failed_tools", result.n_failed_tools) set_eval_attribute("n_failed_tools", result.n_failed_tools)
set_eval_attribute("n_executions", result.n_executions) set_eval_attribute("n_executions", result.n_executions)
set_eval_attribute("n_requests", result.n_requests) set_eval_attribute("n_requests", result.n_requests)
set_eval_attribute("budget_spent", result.budget_spent)
return result.answer return result.answer
report = await _evaluate(answer_question) report = await _evaluate(answer_question)

17
evaluations/evaluations/capability_runner.py
View file

@ -39,7 +39,6 @@ class CapabilityRunResult:
n_rejected_searches: int = 0 n_rejected_searches: int = 0
n_failed_tools: int = 0 n_failed_tools: int = 0
n_requests: int = 0 n_requests: int = 0
budget_spent: bool = False
def _count_tool_traffic( def _count_tool_traffic(
@ -48,13 +47,16 @@ def _count_tool_traffic(
"""Count search calls, failed calls and model requests in a run. """Count search calls, failed calls and model requests in a run.
``state.searches`` is keyed by query, so it collapses repeated queries and ``state.searches`` is keyed by query, so it collapses repeated queries and
never records a call the capability refused. Counting the message history never records a call the capability refused. The capability object cannot be
instead gives the real number of attempts. read instead: ``for_run`` hands the run a ``replace()`` copy, so the outer
instance's counters stay at zero. Counting the message history is the only
way to see the real number of attempts.
Failures are split by tool. Only the search tool fails for want of budget, Search failures are counted apart because the search tool fails only when
whereas the code tool raises ``ToolFailed`` for any error in model-written its budget is spent. A failed code call is ambiguous — an exhausted
Python, so counting every failure together would report a ``ZeroDivisionError`` execution budget and any error in model-written Python both surface as
as budget exhaustion. ``ToolFailed``, distinguishable in the history only by message text — so
``n_failed_tools`` covers both without claiming to tell them apart.
""" """
search_tool = f"{namespace}_search" search_tool = f"{namespace}_search"
search_calls = 0 search_calls = 0
@ -162,5 +164,4 @@ async def run_capability_question(
n_rejected_searches=n_rejected_searches, n_rejected_searches=n_rejected_searches,
n_failed_tools=n_failed_tools, n_failed_tools=n_failed_tools,
n_requests=n_requests, n_requests=n_requests,
budget_spent=n_rejected_searches > 0,
) )