From 357460da577cf13ca9a1a9ffb02fa5f95e162ae3 Mon Sep 17 00:00:00 2001
From: Yiorgis Gozadinos <ggozadinos@gmail.com>
Date: Thu, 26 Jun 2025 11:56:49 +0300
Subject: [PATCH] Update benchmarks

---
 BENCHMARKS.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index bc66cab3..977e1927 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -7,3 +7,7 @@ We use [repliqa](https://huggingface.co/datasets/ServiceNow/repliqa) for the eva
 We load the `News Stories` from `repliqa_3` which is 1035 documents, using `tests/generate_benchmark_db.py`, using the `mxbai-embed-large` Ollama embeddings.
 
 Subsequently, we run a search over the `question` for each row of the dataset and check whether we match the document that answers the question. The recall obtained is ~0.75 for matching in the top result, raising to ~0.75 for the top 3 results.
+
+* Question/Answer evaluation
+
+We use the  `News Stories` from `repliqa_3` using the `mxbai-embed-large` Ollama embeddings, with a QA agent also using Ollama with the `qwen3` model (8b). For each story we ask the `question` and use an LLM judge (also `qwen3`) to evaluate whether the answer is correct or not. Thus we obtain accuracy of ~0.54.