danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update benchmarks for orb for gemma4

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-05-14 14:03:19 +03:00
parent 862a209137
commit d76c645862
No known key found for this signature in database
1 changed files with 8 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/benchmarks.md
View file

@ -166,11 +166,15 @@ The text-vs-image gap on retrieval is small (0.81 pp) but on QA it widens to ~5.
##### QA Accuracy
| Embedding Model | VLM | QA Model | Accuracy |
|----------------------|----------------------|------------------|---------:|
| `qwen3-embedding:4b` | Ollama / ministral-3 | `ollama:qwen3.6` | 0.95 |
| Embedding Model | VLM | QA Model | Reranker | Accuracy |
|----------------------|----------------------|-------------------------------------|------------------------|---------:|
| `qwen3-embedding:4b` | Ollama / ministral-3 | `ollama:qwen3.6` | none | 0.95 |
| `qwen3-embedding:4b` | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` | none | 0.81 |
| `qwen3-embedding:4b` | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` | `mxbai-rerank-base-v2` | 0.92 |
*Measured on haiku.rag v0.45.0, judged by `ollama:qwen3.6` (current default).*
*Measured on haiku.rag v0.45.0, judged by `ollama:qwen3.6` (current default). T
On Gemma-4 the reranker is worth +11 percentage points. Inspecting the no-reranker failures, the dominant patterns are RRF score compression (the right chunk in top-8 but not top-1), FTS keyword bleed from unrelated papers, and References-section chunks competing with body sections. `mxbai-rerank-base-v2` resolves all three on most cases.
##### Skill QA + citation retrieval