Yiorgis Gozadinos
044da7ae99
Open the eval database read-only outside population
...
Retrieval and QA only read from the database, but the benchmark opened it
writable, where an embedder identity differing from the stored one aborts
instead of warning. Running a pre-built database against a different
serving stack then needed a `rebuild --set-embedder` first.
Correct the debug-evals skill alongside it: the pydantic-ai span names are
`execute_tool {tool_name}` and `invoke_agent agent`, targets are
`{rag,analysis}-capability`, and no `skill_model` metadata key exists.
2026-07-28 11:36:27 +03:00
Yiorgis Gozadinos
94e03d777a
vb
2026-07-25 15:11:54 +03:00
Yiorgis Gozadinos
acbd66afbd
vb
2026-07-24 17:32:20 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation
2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
43c17a6777
fix capability execution limits and chat loading
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities
2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
ec78641cc4
vb
2026-07-24 15:15:20 +03:00
Yiorgis Gozadinos
9f49bdde6e
vb
2026-07-23 14:28:23 +03:00
Yiorgis Gozadinos
294ced57cd
vb
2026-07-23 10:45:48 +03:00
Yiorgis Gozadinos
4f0fb7322e
vb
2026-07-22 13:27:35 +03:00
Yiorgis Gozadinos
11303aa714
Document hotpotqa benchmark results and finalize the reference config
2026-07-17 16:28:16 +03:00
Yiorgis Gozadinos
0e8fa551f3
Restore hotpotqa evaluation dataset
2026-07-17 16:25:27 +03:00
Yiorgis Gozadinos
b5c0ea630c
vb
2026-07-16 13:50:40 +03:00
Yiorgis Gozadinos
5dd310a9b7
vb
2026-07-14 16:06:50 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider
2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
5c41f8a0ed
vb
2026-07-10 13:38:24 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event
2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
a82673a900
Route eval scripts through the shared telemetry configure
2026-07-10 11:38:33 +03:00
Yiorgis Gozadinos
da0748916b
vb
2026-07-09 15:58:58 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id
2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
9a9bea86d3
vb
2026-07-03 13:35:10 +03:00
Yiorgis Gozadinos
9a85632cf3
vb
2026-06-29 12:36:36 +03:00
Yiorgis Gozadinos
325f4517ba
Update t2_finqa config
2026-06-29 12:22:58 +03:00
Yiorgis Gozadinos
3cb229d2e0
Add t2_finqa pre-built evaluation database reference config and docs
2026-06-29 10:52:23 +03:00
Yiorgis Gozadinos
4d03b1e669
Add nemotron-vl multimodal eval database and reference configs
2026-06-29 10:14:53 +03:00
Yiorgis Gozadinos
b2abdfaffb
vb
2026-06-28 10:57:26 +03:00
Yiorgis Gozadinos
c7c3f75508
vb
2026-06-27 09:59:21 +03:00
Yiorgis Gozadinos
cd1a73b389
vb
2026-06-26 17:13:03 +03:00
Yiorgis Gozadinos
197d58a602
vb
2026-06-24 18:41:05 +03:00
Yiorgis Gozadinos
ab1775dbc6
vb
2026-06-24 16:45:19 +03:00
Yiorgis Gozadinos
65f1339aa0
vb
2026-06-23 16:59:00 +03:00
Yiorgis Gozadinos
0fcf91fcf7
vb
2026-06-22 15:18:35 +03:00
Yiorgis Gozadinos
5c4b37df89
vb
2026-06-19 11:12:45 +03:00
Yiorgis Gozadinos
99011755fb
vb
2026-06-16 16:48:36 +03:00
Yiorgis Gozadinos
df8af54298
rebase from main
2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
88696cc6f1
vb
2026-06-11 09:50:33 +03:00
Yiorgis Gozadinos
bdad908a60
vb
2026-06-09 15:39:14 +03:00
Yiorgis Gozadinos
46b8fa3fef
vb
2026-06-08 12:24:04 +03:00
Yiorgis Gozadinos
295570797c
Disable Logfire scrubbing in evaluations
2026-06-08 11:51:13 +03:00
Yiorgis Gozadinos
e9bd56467c
Add T²-RAGBench leaderboard submission exporter
2026-06-08 11:51:02 +03:00
Yiorgis Gozadinos
a3a73f1331
Add --filter-ids to run QA on a case-id subset
2026-06-08 09:36:01 +03:00
Yiorgis Gozadinos
1d7d540270
Match the numeric scale convention in Number-Match
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
d5b0aafa2b
Add T²-RAGBench TAT-DQA subset; generalize subset layout
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
de9731d5f3
Score Number-Match on the declared ANSWER line, magnitude and ×100 scale
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
390deb4203
Normalize unicode signs
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
80594cd38c
Handle percent/decimal convention in Number-Match
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
9f1bd9940a
Add T²-RAGBench FinQA evaluation dataset
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
916b84bff4
vb
2026-06-05 12:09:57 +03:00
Yiorgis Gozadinos
e70f271ac7
vb
2026-06-04 14:13:55 +03:00
