danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update benchmarks

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-09-02 14:52:49 +03:00
parent ecf8894755
commit b73949c784
No known key found for this signature in database
2 changed files with 25 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
docs/benchmarks.md
View file

@ -28,6 +28,9 @@ Again using the same dataset, we use a QA agent to answer the question. In addit
| Embedding Model | QA Model | Accuracy | Reranker | | Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------------|-----------------------------------|-----------|------------------------| |------------------------------------|-----------------------------------|-----------|------------------------|
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.85 | None | | Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.85 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.72 | `mxbai-rerank-base-v2` | | Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.87 | `mxbai-rerank-base-v2` |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |
Note the significant degradation when very small models are used such as `qwen3:0.6b`.
<!-- | Ollama / `mxbai-embed-large` | Anthropic / `Claude Sonnet 3.7` | 0.79 | None | <!-- | Ollama / `mxbai-embed-large` | Anthropic / `Claude Sonnet 3.7` | 0.79 | None |
| OpenAI / `text-embeddings-3-small` | OpenAI / `gpt-4-turbo` | 0.62 | None | --> | OpenAI / `text-embeddings-3-small` | OpenAI / `gpt-4-turbo` | 0.62 | None | -->

35
tests/generate_benchmark_db.py
View file

@ -120,21 +120,28 @@ async def run_qa_benchmark(k: int | None = None):
question = doc["question"] # type: ignore question = doc["question"] # type: ignore
expected_answer = doc["answer"] # type: ignore expected_answer = doc["answer"] # type: ignore
generated_answer = await qa.answer(question) # Really small models might fail, let's account for that in try/except
is_equivalent = await judge.judge_answers( try:
question, generated_answer, expected_answer generated_answer = await qa.answer(question)
) is_equivalent = await judge.judge_answers(
console.print(f"Question: {question}") question, generated_answer, expected_answer
console.print(f"Expected: {expected_answer}") )
console.print(f"Generated: {generated_answer}") console.print(f"Question: {question}")
console.print(f"Equivalent: {is_equivalent}\n") console.print(f"Expected: {expected_answer}")
console.print(f"Generated: {generated_answer}")
console.print(f"Equivalent: {is_equivalent}\n")
if is_equivalent: if is_equivalent:
correct_answers += 1 correct_answers += 1
total_questions += 1 except Exception as e:
console.print("Current score:", correct_answers, "/", total_questions) console.print(f"[red]Error processing question: {question}[/red]")
console.print(f"[red]{e}[/red]")
progress.advance(task) finally:
total_questions += 1
console.print(
"Current score:", correct_answers, "/", total_questions
)
progress.advance(task)
accuracy = correct_answers / total_questions if total_questions > 0 else 0 accuracy = correct_answers / total_questions if total_questions > 0 else 0