danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update benchmarks for qwen3.8

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-08-20 11:52:40 +03:00
parent a896ac9eec
commit d51356f476
No known key found for this signature in database
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
docs/benchmarks.md
View file

@ -4,7 +4,7 @@ We evaluate `haiku.rag` on a small set of datasets that exercise different parts
## Current results
Numbers below were measured under `Qwen3.6-35B-A3B-NVFP4` as judge, on a recent `haiku.rag` version. The pinned judge is now `qwen3.8`; rows are not re-judged, so compare rows to each other rather than to runs judged by `qwen3.8`.
Numbers below were measured on a recent `haiku.rag` version. Most rows were judged by `Qwen3.6-35B-A3B-NVFP4`; the `Qwen3.8-27B` rows were judged by the currently pinned `qwen3.8`, as their footnote states. Rows are not re-judged when the pinned judge changes, so compare rows judged by the same judge and treat cross-judge differences as unmeasured.
### OpenRAG Bench (ORB)
@ -37,9 +37,13 @@ Two approaches are benchmarked separately:
| `nvidia/llama-nemotron-embed-vl-1b-v2` | `analysis-capability`| `vllm:Qwen3.6-35B-A3B-NVFP4` | 3045 | 0.95 | 0.93 |
| `nvidia/llama-nemotron-embed-vl-1b-v2` | `rag-capability` | `vllm:Muse-Glimmer-30B-NVFP4` | 3045 | 0.9494 | 0.9771 |
| `nvidia/llama-nemotron-embed-vl-1b-v2` | `analysis-capability`| `vllm:Muse-Glimmer-30B-NVFP4` | 3017 | 0.9718 | 0.9837 |
| `nvidia/llama-nemotron-embed-vl-1b-v2` | `rag-capability` | `vllm:Qwen3.8-27B-NVFP4` | 3045 | 0.9514 | 0.9817 |
| `nvidia/llama-nemotron-embed-vl-1b-v2` | `analysis-capability`| `vllm:Qwen3.8-27B-NVFP4` | 3042 | 0.9629 | 0.9835 |
*The `Muse-Glimmer-30B` rows run at `chat_template_kwargs.reasoning_strength: high`, no reranker, same judge.*
*The `Qwen3.8-27B` rows run at `chat_template_kwargs.reasoning_effort: low`, no reranker, and are **judged by `Qwen3.8-27B` itself** — the pinned judge, and the same model that produced the answers. A 120-case cross-check by an independent judge agreed on 95% and was never stricter, but the incumbent `Qwen3.6` judge is no longer hosted, so the older rows cannot be re-judged for a like-for-like comparison. `rag-capability` cites on 99.34% of cases with 1.09 citations each; `analysis-capability` on 99.70% with 1.12, at 0.13 code executions per case. Case counts exclude 0 and 3 provider errors respectively.*
*Both nemotron `Gemma-4` rows are measured on this release, no reranker, judged by `vllm:Qwen3.6-35B-A3B-NVFP4` with thinking on, and exclude the cases that errored (6 of 3045 for `rag-capability`, 5 for `analysis-capability`). The `rag-capability` row cites at 99.64% with a mean of 1.08 citations per case, at a median 4.7s per case against 5.0s for `analysis-capability`. Citation coverage is what moved on this release: 4.9% of analysis cases register no citation, against 26.3% before, at unchanged searches and code executions per case. The remaining rows are from haiku.rag v0.52.0, where Qwen3-VL covered 1409 / 3045 cases.*
#### Text embedder + VLM picture descriptions