From f1d5918d43fba2fbd3039ae70ec3174ce1f9bd97 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Fri, 14 Aug 2026 15:11:41 +0300 Subject: [PATCH] Update MTRAG benchmark numbers to the glimmer baseline --- docs/benchmarks.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index dd73d320..7d603b89 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -248,7 +248,8 @@ Directly comparable with [IBM's published results](https://github.com/IBM/mt-rag | Mode | Capability model | Turns | QA accuracy | Mean `cited_map` | |------|------------------|------:|-------------|------------------| -| Gold-prefix (`mtrag_clapnq`) | `vllm:Gemma-4-26B-A4B-NVFP4` | 223 | 0.68 | 0.35 | -| Live (`mtrag_clapnq_live`) | `vllm:Gemma-4-26B-A4B-NVFP4` | 195/224 scored | 0.72 micro / 0.73 macro | 0.35 | +| Gold-prefix (`mtrag_clapnq`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224 | 0.76 | 0.35 | +| Live compacted (`mtrag_clapnq_live`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224/224 scored | 0.83 micro / 0.84 macro | 0.42 | +| Live uncompacted (`mtrag_clapnq_live_uncompacted`) | `vllm:Muse-Glimmer-30B-NVFP4` | 224/224 scored | 0.78 micro / 0.79 macro | 0.42 | -*Measured on haiku.rag v0.67.1 with `qwen3-embedding:4b` (vLLM, dim 2560) and `Qwen3-Reranker-4B`, stock capability instructions, judged by `vllm:Qwen3.6-35B-A3B-NVFP4` at temperature 0. Two changes since re-baseline these numbers: the judge sampling was re-pinned repo-wide (0.6 with thinking), and 0.74.0 rewrote the citation instructions (refusals now declare an empty citation list instead of being exempt). Retrieval is unaffected by both and remains the control. QA numbers are internal (our judge and rubric) and not comparable with IBM's published generation metrics. Live mode additionally reports refusal precision/recall against the per-turn answerability labels and per-turn pass rates; pass rate declines with conversation depth (93% at turn 1 to 38% at turn 9). The dataset is text-only (ClapNQ passages), so it exercises no multimodal paths.* +*Measured on haiku.rag v0.74.0 with `qwen3-embedding:4b` (vLLM, dim 2560), `Qwen3-Reranker-4B`, stock capability instructions with `reasoning_strength: high`, judged by the pinned `vllm:Qwen3.6-35B-A3B-NVFP4` (temperature 0.6, thinking). The two live arms differ only in registering `EvidenceCompactionCapability`. Compaction reduces input tokens per request 1.8x (7.5k vs 13.5k), leaves citation retrieval unchanged, and improves answer pass rate (paired McNemar p = 0.031, 95% CI +0.8 to +8.2pp). QA numbers are internal (our judge and rubric) and not comparable with IBM's published generation metrics. Gold-prefix and live rates answer different judge questions and are not comparable with each other. The dataset is text-only (ClapNQ passages), so it exercises no multimodal paths.*