haiku.rag/evaluations/tests
Yiorgis Gozadinos c137305468
Scope the limit notice and spend the cite window on own turns only
The request-limit notice said only the cite tool remained available, but
chat registers rag and analysis in one agent, so exhausting analysis
claimed rag_search was gone too. Scoped to the capability's own tools.

The cite window was counted over every model request once loaded, so
turns spent on another capability expired it before the model was ever
placed where citing was the obvious move. Count only requests whose
preceding response called one of this capability's tools; engagement is
also the only thing that can loop, which is all the bound guards against.

Also: _count_tool_traffic returns a named tuple rather than four bare
ints, and counts failures only for this capability's tools, so host-tool
retries and output-validation retries no longer read as its failures.
2026-07-30 19:14:14 +03:00
..
__init__.py Tests for gepa 2026-03-12 12:05:25 +02:00
test_benchmark.py replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
test_capability_runner.py Scope the limit notice and spend the cite window on own turns only 2026-07-30 19:14:14 +03:00
test_citation_evaluators.py Remove unecessary Mean Reciprocal Rank metric 2026-06-01 10:40:52 +03:00
test_config.py remove dataset-specific system prompts 2026-04-28 14:33:25 +03:00
test_datasets.py Restore hotpotqa evaluation dataset 2026-07-17 16:25:27 +03:00
test_evaluators.py Match the numeric scale convention in Number-Match 2026-06-06 14:52:05 +03:00
test_numbers.py Normalize unicode signs 2026-06-06 14:52:04 +03:00
test_submission.py Add T²-RAGBench leaderboard submission exporter 2026-06-08 11:51:02 +03:00