danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update retrieval/qa benchmark for hotpotqa

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-12-16 14:23:04 +02:00
parent 7bcd205a37
commit d77bf89aad
No known key found for this signature in database
1 changed files with 9 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
docs/benchmarks.md
View file

@ -109,14 +109,17 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML
[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents. [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents.
*Results from v0.20.2*
QA accuracy is evaluated over 2000 "hard" questions from the validation dataset.
### Retrieval (MAP) ### Retrieval (MAP)
| Embedding Model | MAP | Reranker | Notes | | Embedding Model | MAP | Reranker |
|-----------------|-----|----------|-------| |----------------------|------|----------|
| | | | | | `qwen3-embedding:4b` | 0.69 | none |
### QA Accuracy ### QA Accuracy
| Embedding Model | QA Model | Accuracy | Notes | | Embedding Model | QA Model | Accuracy |
|-----------------|----------|----------|-------| |----------------------|--------------------------|----------|
| | | | | | `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 |