From 50a1a6171c54e9ed204448085952bcbdbf2ee9f3 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Sun, 25 Jan 2026 16:27:46 +0200 Subject: [PATCH] Update orb qa accuracy --- docs/benchmarks.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 77521bd9..7649ef90 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -110,7 +110,6 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents. *Results from v0.20.2* -QA accuracy is evaluated over 2000 "hard" questions from the validation dataset. ### Retrieval (MAP) @@ -137,3 +136,9 @@ QA accuracy is evaluated over 2000 "hard" questions from the validation dataset. | Embedding Model | MAP | VLM | |----------------------|--------|----------------------| | `qwen3-embedding:4b` | 0.9626 | Ollama / ministral-3 | + +### QA Accuracy + +| Embedding Model | QA Model | Accuracy | VLM | +|----------------------|-----------------------------|----------|----------------------| +| `qwen3-embedding:4b` | `gpt-oss:20b` - no thinking | 0.912 | Ollama / ministral-3 |