danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update benchmarks

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-07-20 13:06:41 +03:00
parent ad9879802a
commit 4908ea9fcc
No known key found for this signature in database
1 changed files with 14 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
docs/benchmarks.md
View file

@ -12,19 +12,22 @@ In order to calculate recall, we load the `News Stories` from `repliqa_3` which
The recall obtained is ~0.73 for matching in the top result, raising to ~0.75 for the top 3 results.
| Model | Document in top 1 | Document in top 3 | Reranker |
|---------------------------------------|-------------------|-------------------|----------------------|
| Ollama / `mxbai-embed-large` | 0.77 | 0.89 | None |
| Ollama / `mxbai-embed-large` | 0.81 | 0.91 | mxbai-rerank-base-v2 |
| Ollama / `nomic-embed-text` | 0.74 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| Embedding Model | Document in top 1 | Document in top 3 | Reranker |
|---------------------------------------|-------------------|-------------------|------------------------|
| Ollama / `mxbai-embed-large` | 0.77 | 0.89 | None |
| Ollama / `mxbai-embed-large` | 0.81 | 0.91 | `mxbai-rerank-base-v2` |
| Ollama / `nomic-embed-text` | 0.74 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.83 | 0.90 | Cohere / `rerank-v3.5` |
## Question/Answer evaluation
Again using the same dataset, we use a QA agent to answer the question. In addition we use an LLM judge (using the Ollama `qwen3`) to evaluate whether the answer is correct or not. The obtained accuracy is as follows:
| Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------|-----------------------------------|-----------|----------------------|
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.64 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.72 | mxbai-rerank-base-v2 |
| Ollama / `mxbai-embed-large` | Anthropic / `Claude Sonnet 3.7` | 0.79 | None |
| Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------------|-----------------------------------|-----------|------------------------|
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.64 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.72 | `mxbai-rerank-base-v2` |
| Ollama / `mxbai-embed-large` | Anthropic / `Claude Sonnet 3.7` | 0.79 | None |
| OpenAI / `text-embeddings-3-small` | OpenAI / `gpt-4-turbo` | 0.62 | None |