danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add retrieval benchmarks for ORB

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-01-22 15:01:37 +02:00
parent de022639b6
commit 756285e91c
No known key found for this signature in database
1 changed files with 14 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
docs/benchmarks.md
View file

@ -123,3 +123,17 @@ QA accuracy is evaluated over 2000 "hard" questions from the validation dataset.
| Embedding Model | QA Model | Accuracy |
|----------------------|--------------------------|----------|
| `qwen3-embedding:4b` | `gpt-oss:20b` - thinking | 0.86 |
## OpenRAG Bench (ORB)
[OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval over visual content like figures, charts, and diagrams. We use MAP for retrieval evaluation since each query maps to one relevant document.
**Multimodal processing**: Picture descriptions are generated using a Vision Language Model (VLM) during document conversion, making embedded images searchable via text queries. See [Picture Description configuration](configuration/processing.md#picture-description-vlm).
*Results from v0.26.8*
### Retrieval (MAP)
| Embedding Model | MAP | VLM |
|----------------------|--------|----------------------|
| `qwen3-embedding:4b` | 0.9626 | Ollama / ministral-3 |