Commit graph

7 commits

Author SHA1 Message Date
Yiorgis Gozadinos
73d9d93db9
Add MTRAG ClapNQ multi-turn evaluation
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.

Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
2026-08-17 10:53:16 +03:00
Yiorgis Gozadinos
1d7d540270
Match the numeric scale convention in Number-Match 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
de9731d5f3
Score Number-Match on the declared ANSWER line, magnitude and ×100 scale 2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
80594cd38c
Handle percent/decimal convention in Number-Match 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
66f88f7872
Test evaluations 2026-03-12 12:05:50 +02:00