haiku.rag/evaluations/evaluations
Yiorgis Gozadinos cab510d0d5
Give the evidence capabilities one typed state
RAGState and AnalysisState each declared the same five fields, so the generic
base could not name them: StateT was bound to BaseModel, and every access
went through cast(Any, state), a getattr by string, or a loop clearing fields
by name so it could skip the one only AnalysisState has.

EvidenceState declares them once. RAGState adds nothing, AnalysisState adds
executions and overrides begin_invocation to clear them. StateT binds to
EvidenceState, which removes all ten casts and both state-shape getattrs; the
three getattr(ctx.deps, "state") probes stay, since those check a
host-supplied object rather than our own state.

discover_evidence reached into capability.state for two fields. It now asks
through evidence_record() and citation_index(), alongside the
evidence_tool_names() and cite_available accessors it already used. The eval
runner's _RagLikeState protocol and the chat app's getattr reads described
this shape from outside and are gone.

Compatibility is semantic JSON-object equivalence, not bytes: field names and
nesting are unchanged, so a dict stored by 0.75.0 loads and re-dumps equal,
but deriving from a shared base reorders AnalysisState's keys. Nothing
serializes, hashes or string-compares this state — every carry point
re-validates by key.
2026-08-19 17:08:44 +03:00
..
datasets Simplify the eval harness and share the embed-fill path. 2026-08-17 10:56:26 +03:00
evaluators Simplify the eval harness and share the embed-fill path. 2026-08-17 10:56:26 +03:00
__init__.py Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00
benchmark.py Pin the eval judge to qwen3.8 2026-08-18 14:28:16 +03:00
capability_runner.py Give the evidence capabilities one typed state 2026-08-19 17:08:44 +03:00
config.py Simplify the eval harness and share the embed-fill path. 2026-08-17 10:56:26 +03:00
numbers.py Normalize unicode signs 2026-06-06 14:52:04 +03:00
submission.py replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00