danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update retrieval/qa benchmark for hotpotqa

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-12-16 14:23:04 +02:00
parent 7bcd205a37
commit d77bf89aad
No known key found for this signature in database
1 changed files with 9 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
docs/benchmarks.md
View file

@ -109,14 +109,17 @@ We benchmark both the plain text version (HTML stripped, no structure) and HTML
[HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) is a multi-hop question answering dataset requiring reasoning over multiple Wikipedia paragraphs. Each question requires evidence from 2+ documents, making it ideal for testing retrieval and reasoning capabilities. We use MAP for retrieval evaluation since queries have multiple relevant documents.
*Results from v0.20.2*
QA accuracy is evaluated over 2000 "hard" questions from the validation dataset.
### Retrieval (MAP)
| Embedding Model | MAP | Reranker | Notes |
|-----------------|-----|----------|-------|
| | | | |
| Embedding Model | MAP | Reranker |
|----------------------|------|----------|
| `qwen3-embedding:4b` | 0.69 | none |
### QA Accuracy
| Embedding Model | QA Model | Accuracy | Notes |
|-----------------|----------|----------|-------|
| | | | |
| Embedding Model | QA Model | Accuracy |
|----------------------|--------------------------|----------|
| `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 |